1.Scaling Uber’s Real-time Optimization with Flink Xingzhong Xu Engineer | Uber Marketplace xxu@uber.com Apr 10, 2018 

2.Scaling Uber’s Real-time Optimization with Flink Uber's mission is to bring transportation — for everyone, everywhere. — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0 

3.Scaling Uber’s Real-time Optimization with Flink Agenda — ● Uber Marketplace ● Geo/temporal event aggregation ● Online model update ● Streaming application — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0 

4.Scaling Uber’s Real-time Optimization with Flink Uber Marketplace — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0 

5.Scaling Uber’s Real-time Optimization with Flink Uber marketplace — Dynamic logistics network and decision engines at your fingertips. — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0 

6.Scaling Uber’s Real-time Optimization with Flink Marketplace dynamics — ● Supply ● Demand ● Forecast ● Trips ● Traffic — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0 

7.Scaling Uber’s Real-time Optimization with Flink Marketplace decision engines — ● Dispatch ● Pricing ● Driver Positioning ● Promotions — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0 

8.Scaling Uber’s Real-time Optimization with Flink Uber Marketplace — — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0 

9.Scaling Uber’s Real-time Optimization with Flink Geo-temporal event aggregation — — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0 

10.Scaling Uber’s Real-time Optimization with Flink Events in physical world drive marketplace dynamics Every second, marketplace ingesting millions events real-time real-world — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0 

11. Scaling Uber’s Real-time Optimization with Flink Real-time challenges ● Event time ordering ● Time sensitive — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0 

12.Scaling Uber’s Real-time Optimization with Flink Aggregate in real-time — ● Aggregation in windows bucket ● Results over event time ● As soon as possible ● As accurate as possible — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0 

13. Scaling Uber’s Real-time Optimization with Flink Real-world challenges ● Event spatial mapping ● Locality sensitive — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0 

14.Scaling Uber’s Real-time Optimization with Flink Aggregation in real-world — ● Influences its current and neighbours ● Apply geo func on related events — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0 

15.Scaling Uber’s Real-time Optimization with Flink How to aggregate geo-related events in real-time? — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0 

16.Scaling Uber’s Real-time Optimization with Flink Online analytical processing (OLAP) — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0 

17.Scaling Uber’s Real-time Optimization with Flink OLAP solution — — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0 

18.Scaling Uber’s Real-time Optimization with Flink OLAP solution — Periodical crontab — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0 

19.Scaling Uber’s Real-time Optimization with Flink OLAP solution — Periodical crontab Batch snapshot — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0 

20.Scaling Uber’s Real-time Optimization with Flink Event driven solution — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0 

21.Scaling Uber’s Real-time Optimization with Flink Event based solution (flatmap) Geo fanout first — — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0 

22.Scaling Uber’s Real-time Optimization with Flink Event based solution (reduce) Event time agg later — — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0 

23. Scaling Uber’s Real-time Optimization with Flink No more periodical queries ➔ Flexible windows and trigger strategy ➔ Compute triggered by events only ➔ Materialized result pushed to consumer — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0 

24. Scaling Uber’s Real-time Optimization with Flink No more bottleneck ➔ Avoid single point of bottleneck in dataflow ➔ Better isolation and scale independently — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0 

25.Scaling Uber’s Real-time Optimization with Flink Event driven design concern? — ● Excessive fanout vs shuffle-free ● Specific topology vs generic query — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0 

26.Scaling Uber’s Real-time Optimization with Flink Event driven design concern? in flink — ● Excessive fanout vs shuffle-free ○ Virtual key ○ Memory management ● Specific topology vs generic query — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0 

27.Scaling Uber’s Real-time Optimization with Flink Event driven design concern? in flink — ● Excessive fanout vs shuffle-free ○ Virtual key ○ Memory management ● Specific topology vs generic query ○ dataSteam API and dataflow language ○ Customized job per application — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0 

28.Scaling Uber’s Real-time Optimization with Flink Online model updates — — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0 

29.Scaling Uber’s Real-time Optimization with Flink In marketplace, there are the models that describe the world and the decision engines that act on those models — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0