Flink 技术介绍与未来展望

展开查看详情

1. 1 8 ( 2 ) 9.. 0

2. L B . . F J 2015 2016 2017 2018 C B F B

3.

4.Flink

5.Stateful Computations Over Streams

6.Fast Scalable Event-time Fault-tolerant Large-state In-memory Exaclty-once Stateful Computations Over Streams

7.Checkpoint State Time Window Chandy-Lamport State API Watermark ValueState, ListState, MapState BroadcastState

8.

9.• M • > > • e e • F • C - • J S • EP S

10.Blink is an unified engine for large-scale data processing.

11.

12. - Flink 1.0.0 Flink 1.2.0 Flink 1.5.0 Table API v1.0 tumbling, sliding, session Join in Table API/SQL CEP library added group-windows for Table API/SQL SQL CLI Flink 1.1.0 Flink 1.3.0 Refactor Table API/SQL Retractions in Table API/SQL to base on Apache Calcite Rescalable CEP operators

13. Flink 1.0.0 Flink 1.2.0 Flink 1.5.0 State API (ValueState, ProcessFunction Broadcast State ReducingState, ListState) (low-level API) Flink 1.1.0 Flink 1.3.0 Flink 1.6.0 Session windows Side outputs State TTL Late-arriving events DataStream Interval Join

14. - - - ) ) - ) AB - )) - - )) )) - - ()- ) ) ()-

15. & Flink 1.0.0 Flink 1.2.0 Flink 1.5.0 RocksDB for out-of-core Rescalable keyed and Task local state recovery state support operator state Flink 1.1.0 Flink 1.3.0 Fully async RocksDB Incremental checkpoint snapshots Fine-grained recovery

16.

17. Flink 1.4.0 Flink 1.2.0 End-to-End exactly once Async I/O processing (with kafka, too) Flink 1.3.0 Flink 1.5.0 HistoryServer: FLIP-6: New Deployment and Process Model archive status and statistics Rewrite Network Stack of completed jobs

18.• • 2 )(1 -/ / 2 /- • 2

19.

20.0 0

21.

22.1 /-1 ) ( • • B • • •

23.

24.1 Spark 2 Stream SQL - Join TopN CEP on SQL 3 Stream SQL MicroBatch, Local-Global, CountDistinct 4 SQL Python UDF

25.