展开查看详情
1.Scaling Uber’s Real-time Optimization with Flink Xingzhong Xu Engineer | Uber Marketplace xxu@uber.com Apr 10, 2018
2.Scaling Uber’s Real-time Optimization with Flink Uber's mission is to bring transportation — for everyone, everywhere. — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0
3.Scaling Uber’s Real-time Optimization with Flink Agenda — ● Uber Marketplace ● Geo/temporal event aggregation ● Online model update ● Streaming application — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0
4.Scaling Uber’s Real-time Optimization with Flink Uber Marketplace — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0
5.Scaling Uber’s Real-time Optimization with Flink Uber marketplace — Dynamic logistics network and decision engines at your fingertips. — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0
6.Scaling Uber’s Real-time Optimization with Flink Marketplace dynamics — ● Supply ● Demand ● Forecast ● Trips ● Traffic — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0
7.Scaling Uber’s Real-time Optimization with Flink Marketplace decision engines — ● Dispatch ● Pricing ● Driver Positioning ● Promotions — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0
8.Scaling Uber’s Real-time Optimization with Flink Uber Marketplace — — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0
9.Scaling Uber’s Real-time Optimization with Flink Geo-temporal event aggregation — — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0
10.Scaling Uber’s Real-time Optimization with Flink Events in physical world drive marketplace dynamics Every second, marketplace ingesting millions events real-time real-world — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0
11. Scaling Uber’s Real-time Optimization with Flink Real-time challenges ● Event time ordering ● Time sensitive — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0
12.Scaling Uber’s Real-time Optimization with Flink Aggregate in real-time — ● Aggregation in windows bucket ● Results over event time ● As soon as possible ● As accurate as possible — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0
13. Scaling Uber’s Real-time Optimization with Flink Real-world challenges ● Event spatial mapping ● Locality sensitive — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0
14.Scaling Uber’s Real-time Optimization with Flink Aggregation in real-world — ● Influences its current and neighbours ● Apply geo func on related events — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0
15.Scaling Uber’s Real-time Optimization with Flink How to aggregate geo-related events in real-time? — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0
16.Scaling Uber’s Real-time Optimization with Flink Online analytical processing (OLAP) — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0
17.Scaling Uber’s Real-time Optimization with Flink OLAP solution — — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0
18.Scaling Uber’s Real-time Optimization with Flink OLAP solution — Periodical crontab — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0
19.Scaling Uber’s Real-time Optimization with Flink OLAP solution — Periodical crontab Batch snapshot — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0
20.Scaling Uber’s Real-time Optimization with Flink Event driven solution — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0
21.Scaling Uber’s Real-time Optimization with Flink Event based solution (flatmap) Geo fanout first — — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0
22.Scaling Uber’s Real-time Optimization with Flink Event based solution (reduce) Event time agg later — — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0
23. Scaling Uber’s Real-time Optimization with Flink No more periodical queries ➔ Flexible windows and trigger strategy ➔ Compute triggered by events only ➔ Materialized result pushed to consumer — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0
24. Scaling Uber’s Real-time Optimization with Flink No more bottleneck ➔ Avoid single point of bottleneck in dataflow ➔ Better isolation and scale independently — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0
25.Scaling Uber’s Real-time Optimization with Flink Event driven design concern? — ● Excessive fanout vs shuffle-free ● Specific topology vs generic query — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0
26.Scaling Uber’s Real-time Optimization with Flink Event driven design concern? in flink — ● Excessive fanout vs shuffle-free ○ Virtual key ○ Memory management ● Specific topology vs generic query — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0
27.Scaling Uber’s Real-time Optimization with Flink Event driven design concern? in flink — ● Excessive fanout vs shuffle-free ○ Virtual key ○ Memory management ● Specific topology vs generic query ○ dataSteam API and dataflow language ○ Customized job per application — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0
28.Scaling Uber’s Real-time Optimization with Flink Online model updates — — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0
29.Scaling Uber’s Real-time Optimization with Flink In marketplace, there are the models that describe the world and the decision engines that act on those models — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0