Scaling Uber’s Realtime Optimization with Apache Flink

发布于

2530

人观看

Uber engineers leverage Apache Flink to build a platform that not only runs compute intensive optimization models, but also very quickly reacts to rapid changes in marketplace. In this talk, I will cover the compute platform that leverages Apache Flink to i.) aggregate billions of realtime and forecasted demand and supply level information across the globe. ii.) trigger on-demand optimization models to respond to changes in marketplace and iii.) scale both horizontally and vertically as we expand the platform to onboard new applications and experiences.

展开查看详情

1 .Scaling Uber’s Real-time Optimization with Flink Xingzhong Xu Engineer | Uber Marketplace xxu@uber.com Apr 10, 2018

2 .Scaling Uber’s Real-time Optimization with Flink Uber's mission is to bring transportation — for everyone, everywhere. — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

3 .Scaling Uber’s Real-time Optimization with Flink Agenda — ● Uber Marketplace ● Geo/temporal event aggregation ● Online model update ● Streaming application — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

4 .Scaling Uber’s Real-time Optimization with Flink Uber Marketplace — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

5 .Scaling Uber’s Real-time Optimization with Flink Uber marketplace — Dynamic logistics network and decision engines at your fingertips. — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

6 .Scaling Uber’s Real-time Optimization with Flink Marketplace dynamics — ● Supply ● Demand ● Forecast ● Trips ● Traffic — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

7 .Scaling Uber’s Real-time Optimization with Flink Marketplace decision engines — ● Dispatch ● Pricing ● Driver Positioning ● Promotions — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

8 .Scaling Uber’s Real-time Optimization with Flink Uber Marketplace — — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

9 .Scaling Uber’s Real-time Optimization with Flink Geo-temporal event aggregation — — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

10 .Scaling Uber’s Real-time Optimization with Flink Events in physical world drive marketplace dynamics Every second, marketplace ingesting millions events real-time real-world — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

11 . Scaling Uber’s Real-time Optimization with Flink Real-time challenges ● Event time ordering ● Time sensitive — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

12 .Scaling Uber’s Real-time Optimization with Flink Aggregate in real-time — ● Aggregation in windows bucket ● Results over event time ● As soon as possible ● As accurate as possible — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

13 . Scaling Uber’s Real-time Optimization with Flink Real-world challenges ● Event spatial mapping ● Locality sensitive — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

14 .Scaling Uber’s Real-time Optimization with Flink Aggregation in real-world — ● Influences its current and neighbours ● Apply geo func on related events — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

15 .Scaling Uber’s Real-time Optimization with Flink How to aggregate geo-related events in real-time? — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

16 .Scaling Uber’s Real-time Optimization with Flink Online analytical processing (OLAP) — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

17 .Scaling Uber’s Real-time Optimization with Flink OLAP solution — — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

18 .Scaling Uber’s Real-time Optimization with Flink OLAP solution — Periodical crontab — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

19 .Scaling Uber’s Real-time Optimization with Flink OLAP solution — Periodical crontab Batch snapshot — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

20 .Scaling Uber’s Real-time Optimization with Flink Event driven solution — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

21 .Scaling Uber’s Real-time Optimization with Flink Event based solution (flatmap) Geo fanout first — — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

22 .Scaling Uber’s Real-time Optimization with Flink Event based solution (reduce) Event time agg later — — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

23 . Scaling Uber’s Real-time Optimization with Flink No more periodical queries ➔ Flexible windows and trigger strategy ➔ Compute triggered by events only ➔ Materialized result pushed to consumer — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

24 . Scaling Uber’s Real-time Optimization with Flink No more bottleneck ➔ Avoid single point of bottleneck in dataflow ➔ Better isolation and scale independently — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

25 .Scaling Uber’s Real-time Optimization with Flink Event driven design concern? — ● Excessive fanout vs shuffle-free ● Specific topology vs generic query — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

26 .Scaling Uber’s Real-time Optimization with Flink Event driven design concern? in flink — ● Excessive fanout vs shuffle-free ○ Virtual key ○ Memory management ● Specific topology vs generic query — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

27 .Scaling Uber’s Real-time Optimization with Flink Event driven design concern? in flink — ● Excessive fanout vs shuffle-free ○ Virtual key ○ Memory management ● Specific topology vs generic query ○ dataSteam API and dataflow language ○ Customized job per application — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

28 .Scaling Uber’s Real-time Optimization with Flink Online model updates — — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

29 .Scaling Uber’s Real-time Optimization with Flink In marketplace, there are the models that describe the world and the decision engines that act on those models — — — Uber 2018 Prepare for Flink Forward 2018 Version 1.0

13 点赞

3 收藏

3下载