Apache Flink® - Redefining Computation

1 .Apache Flink - Redefining Computation ® ;LDRZHL -LDQJ 6HQLRU'LUHFWRU $OLEDED 

2 . About Me Stratify Inc. Facebook 2000 - 2002 2010 - 2014 Member of Technical Staff Software Engineer 2002 2010 2014 Now Microsoft Alibaba Group 2002 - 2010 2014 - Now Principal Software Engineer Senior Director 

3 . About Alibaba EB Total PB Everyday 1T Event/Day 1.7B Events/sec 

4 . Chapter.1 1. Stream Processing 

5 . 11.11 Dashboard Lots of Events Exactly-Once Web Tier DataHub DataPipeline HBase Dashboard DB Tier MQ Sub Second Latency Highly Available 

6 . / Online Service Online ML / 0- 0 0 : - - - 0 -0 : 1 0 :B : 1 0- 0: - 0 0 / - 0 1 / / 0 0- 0 - 0- 0 /0 , /- 0 Prediction Service 

7 . . . - . . . . OP OP . * - .. . . . . ) . . ) ,( OP OP State 

8 . Exactly-Once High Throughput Low Latency Fault Tolerant Flink Stream Process 

9 . Alibaba’s Improvements to Flink Flink Runtime Improvements Flink SQL Improvements Table API & SQL Relational Distributed Architecture Semantics Rework Cluster Management [FLIP-6/FLINK-4319] DataStream API DataSet API Stream Processing Batch Processing Fault Tolerance Functionality JobManager Failover [FLINK-4911] Agg/w Retraction Runtime Window Distributed Streaming DataFlow Region-based Task Failover [FLIP1/FLINK-4256] UDX Support DDL Support Local Cluster Cloud Performance Connector Support Single JVM Standalone/YARN ECS/EC2 Incremental Checkpoint [FLINK-5053] Async I/O [FLIP12/FLINK-4391] Credit-based Flow Control [FLINK-7282] Flink Architecture 

10 ./ / 

11 . Chapter.2 2. Unified Engine 

12 . Lambda Architecture Batch ETL Full Build Search Service / / / Stream ETL Incremental Build 

13 . Scenarios for Data Processing Low Latency Periodic/Continuous High Throughput Fixed Query Batch Jobs Flexible Query Stream Processing Progressive Processing Batch Analytics Streaming as the core abstraction, Batch as a special case of streaming 

14 . Batch as Special Case of Streaming Result of sorting 80GB/node (3.2TB) Unbounded Stream OP OP Unbounded Stream 2320 2200 2,157 2,171 2080 OP OP 1960 Unbounded Stream Unbounded Stream Flink is the fastest due to its 1840 1,887 pipelined execution 1720 Tez and Spark do not overlap 1st and 2nd stages 1600 MapReduce is slow despite overlapping stages 1480 1,480 Bounded Stream OP OP Bounded Stream 1360 1240 1120 Bounded Stream OP 1000 OP Bounded Stream Hadoop-2.7.1 Tez-0.7.0 Spark-1.5.1 Flink-0.9.1 A Comparative Performance Evaluation of Flink, Dongwon Kim, POSTECH, Flink Forward 2015 

15 . Flink Architecture: New Design Table API & SQL DataStream TableAPI SQL Relational API Relational Relational DataStream API DataSet API Query Processor Stream Processing Batch Processing Query Optimization & Query Execution Runtime Runtime Distributed Streaming DataFlow DAG API & Stream Operators Local Cluster Cloud Local Cluster Cloud Single JVM Standalone/YARN ECS/EC2 Single JVM Standalone/YARN ECS/EC2 Old Design New Design 

16 . Improvements in New Design Query Query Runtime Execution Optimizer New Operator Framework Expression Optimizations Cost Based Customizable Scheduling Performant Operators Advanced Rules Flexible Chaining Resource Optimizations Rich Stats 

17 . () ( TPC-DS Performance (the Lower, the Better) 40000s 20000s Spark 2000s Spark Spark Blink Blink Blink Spark 2.31 

18 . 4140 4 04 4 , 4 , , Online Services Matching LogParser Aggregate Accumulator Export Feature Service Search Engine Batch Data 4140 4:0 4 0 1 , 0, 4 Ranking 0, 4 10, 0 Stream Data Model Table Service , & 0 4 , 4 0 LogParser Build PullWeight Gradient W Update Weight Export Model 

19 . Flink Ecosystem Flink Hive Flink Zeppelin 

20 . Chapter. 3 3. Future 

21 . Model W’ = W - ηΔW Parameter Server Data Training Δ Data Training Δ Model Data Training Replicas Δ Data Training Δ Data Shards Delta Iteration 

22 . Vertex Values C A Graph ΔVertex Data Algorithm D Delta Iteration B 

23 . / 0 /0 2 / 1 0 10 0 - - - - & 0 0 / /0 1 - - / 0 0 / 1 - - 0 10 0 / / - - Shipping Order Inventory Payment 

24 .Stream Processing Stream & Batch Bigdata Engine Bigdata & App Engine Processing 

25 .