Flink 技术介绍与未来展望
展开查看详情
1. 1 8 ( 2 ) 9.. 0
2. L B . . F J 2015 2016 2017 2018 C B F B
3.
4.Flink
5.Stateful Computations Over Streams
6.Fast Scalable Event-time Fault-tolerant Large-state In-memory Exaclty-once Stateful Computations Over Streams
7.Checkpoint State Time Window Chandy-Lamport State API Watermark ValueState, ListState, MapState BroadcastState
8.
9.• M • > > • e e • F • C - • J S • EP S
10.Blink is an unified engine for large-scale data processing.
11.
12. - Flink 1.0.0 Flink 1.2.0 Flink 1.5.0 Table API v1.0 tumbling, sliding, session Join in Table API/SQL CEP library added group-windows for Table API/SQL SQL CLI Flink 1.1.0 Flink 1.3.0 Refactor Table API/SQL Retractions in Table API/SQL to base on Apache Calcite Rescalable CEP operators
13. Flink 1.0.0 Flink 1.2.0 Flink 1.5.0 State API (ValueState, ProcessFunction Broadcast State ReducingState, ListState) (low-level API) Flink 1.1.0 Flink 1.3.0 Flink 1.6.0 Session windows Side outputs State TTL Late-arriving events DataStream Interval Join
14. - - - ) ) - ) AB - )) - - )) )) - - ()- ) ) ()-
15. & Flink 1.0.0 Flink 1.2.0 Flink 1.5.0 RocksDB for out-of-core Rescalable keyed and Task local state recovery state support operator state Flink 1.1.0 Flink 1.3.0 Fully async RocksDB Incremental checkpoint snapshots Fine-grained recovery
16.
17. Flink 1.4.0 Flink 1.2.0 End-to-End exactly once Async I/O processing (with kafka, too) Flink 1.3.0 Flink 1.5.0 HistoryServer: FLIP-6: New Deployment and Process Model archive status and statistics Rewrite Network Stack of completed jobs
18.• • 2 )(1 -/ / 2 /- • 2
19.
20.0 0
21.
22.1 /-1 ) ( • • B • • •
23.
24.1 Spark 2 Stream SQL - Join TopN CEP on SQL 3 Stream SQL MicroBatch, Local-Global, CountDistinct 4 SQL Python UDF
25.