How to work with state and checkpoint in Flink data system

1 . Add your logo image Thanks ! 

2 . Add your logo image Apache Flink How to work with state and checkpoint in Flink data stream Thanks 唐云 Yun Tang ! Apache Flink contributor, Engineer at Alibaba 

3 . Add your logo image Apache Flink How to work with state and checkpoint in Flink data stream ●Monitoring system with the help of Flink State Thanks ●What could Flink State do ●What is Flink State ! ●How Flink achieve fault tolerance 

4 . Add your logo image Apache Flink • Monitoring system with the help of Flink State An implementation of monitoring system: Web Frontend Grafana plugin of Thanks our data source 1. Pull data from queue to influxDB with aggregated message. metrics 2. generate downsampled low gateway ! precision data and push to influxDB. metrics importer Messaging system (Streaming J ob) (open source version) Time Series DB (open source version) 

5 . Add your logo image Apache Flink • What could Flink State do • Stateful transformations Streaming analytics Thanks ! 

6 . Add your logo image Apache Flink • What could Flink State do • Stateful transformations Data pipeline Thanks ! 

7 . Add your logo image Apache Flink • What could Flink State do • Stateful transformations Thanks Event-driven application (use window state) ! 

8 . Add your logo image Apache Flink • What is Flink State Process records • local: Flink state is kept local to the one-at-a time machine that processes it, and can be accessed at memory speed. • vertically scalable: Flink state can Your Code Thanks be kept in embedded RocksDB instances that scale by adding more local disk. ! • horizontally scalable: Flink state is redistributed as your cluster grows and shrinks. embedded local state backend • queryable: Flink state can be queried via a REST API 

9 . Add your logo image Apache Flink • What is Flink State Thanks ! durable: Flink state is automatically checkpointed and restored. 

10 . Add your logo image Apache Flink • What is Flink State (a quick look at the code in user view) Word Count example code: Thanks Invoke ! Create StreamGroupedReduce KeyedStream 

11 . Add your logo image Apache Flink • What is Flink State (a quick look at the code in user view) Thanks ! 

12 . Add your logo image Apache Flink • What is Flink State (where to store state) Thanks ! https://training.ververica.com/state-backends.html 

13 . Add your logo image Apache Flink • How Flink achieve fault tolerance a variant of the Chandy-Lamport algorithm (asynchronous barrier snapshotting.) Thanks ! 

14 . Add your logo image Apache Flink • How Flink achieve fault tolerance Thanks 1. checkpoint coordinator notify source operator to ! trigger checkpoint 

15 . Add your logo image Apache Flink • How Flink achieve fault tolerance Thanks 1. checkpoint coordinator notify source operator to ! trigger checkpoint 2. Once task receive barrier, it would execute snapshot broadcast barrier downstream and persist state asynchronously. 

16 . Add your logo image Apache Flink • How Flink achieve fault tolerance Thanks 1. checkpoint coordinator notify source operator to trigger checkpoint 2. Once task receive barrier, ! it would execute snapshot broadcast barrier downstream and persist state asynchronously. 3. Once async phase completed, task would send state handle to coordinator. 

17 . Add your logo image Apache Flink • How Flink achieve fault tolerance Thanks 1. checkpoint coordinator notify source operator to ! trigger checkpoint 2. Once task receive barrier, it would execute snapshot broadcast barrier downstream and persist state asynchronously. 3. Once async phase completed, task would send state handle to coordinator. 

18 . Add your logo image Apache Flink • How Flink achieve fault tolerance Thanks 1. checkpoint coordinator notify source operator to trigger checkpoint 2. Once task receive barrier, ! it would execute snapshot broadcast barrier downstream and persist state asynchronously. 3. Once async phase completed, task would send state handle to coordinator. 

19 . Add your logo image Apache Flink • How Flink achieve fault tolerance Thanks 1. checkpoint coordinator notify source operator to trigger checkpoint 2. Once task receive barrier, ! it would execute snapshot broadcast barrier downstream and persist state asynchronously. 3. Once async phase completed, task would send state handle to coordinator. 4. Coordinator receives all handle, the checkpoint completed! 

20 . Add your logo image Apache Flink • How Flink achieve fault tolerance Barrier alignment Thanks ! 

21 . Add your logo image Apache Flink • How Flink achieve fault tolerance End-to-end Exactly once, Nothing is lost or duplicated • sources must be replayable. Thanks • Flink use exactly-once checkpoint (by default) • sinks must be transactional (or idempotent) ! https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/guarantees.html 

22 . Add your logo image Apache Flink • Some future work FLIP-50: Spill-able Heap Keyed State Backend Thanks Unaligned checkpoints ! 

23 . Add your logo image Apache Flink Thanks ! Thanks! 

24 . Add your logo image Apache Flink Apache Flink 社区微信公众号「 Ververica 」 Thanks ! Meetup 动态 / Release 发布信息 / Flink 应用实践 

25 . Add your logo image 全球最大的 Apache Flink 官方会议 预计 2000+ 参会人员， 2019 年 11 月 28-30 日 @ 北京国家会议中心 国内外一线厂商悉数参与 阿里巴巴、腾讯、字节跳动、 intel 、 Dell EMC 、 Uber 、美团点评、 Ververica ... Thanks 大会官网，查看更多 Flink Forward Asia !