1. Elastic Data Processing with Apache Pulsar and Apache Flink Kaixing Zhao, Apache Flink Committer Sijie Guo, Apache Pulsar Committer 2019-03-23 Hangzhou 

2. Agenda • What is Apache Flink? • Flink use cases and ecosystem • Pulsar - Segmented Stream Architecture • Pulsar - Tiered Storage • Pulsar + Flink Integration 

3.Pulsar == Segmented Streams 

4.Batch - HDFS 

5.Stream - Pub/Sub 

6.A Flink View on Computing “Batch processing is a special case of Stream processing” 

7.A Pulsar View on Data 

8. Topic Producers Topic Consumers Time 

9.Partitions P0 Producers P1 P2 Consumers P3 Time 

10. Segments P0 Segment 1 Segment 2 Segment 3 Producers P1 Segment 1 Segment 2 Segment 3 Segment 4 P2 Segment 1 Segment 2 Segment 3 Consumers P3 Segment 1 Segment 2 Segment 3 Time 

11. Stream P0 Segment 1 Segment 2 Segment 3 Producers P1 Segment 1 Segment 2 Segment 3 Segment 4 P2 Segment 1 Segment 2 Segment 3 Consumers P3 Segment 1 Segment 2 Segment 3 Time 

12. Stream Producers Stream Segment 1 Segment 2 Segment 3 Segment 4 Consumers Time 

13. Access Patterns ✓ Write
 写 Stream Segment 1 Segment 2 Segment 3 Segment 4 ✓ Tailing Read
 追尾 Time ✓ Catchup Read
 追赶读 

14. Access Patterns ✓ Write
 Write 写 Stream Segment 1 Segment 2 Segment 3 Segment 4 ✓ Tailing Read
 追尾 Time ✓ Catchup Read
 追赶读 

15. Access Patterns Tailing Read ✓ Write
 Write 写 Stream Segment 1 Segment 2 Segment 3 Segment 4 ✓ Tailing Read
 追尾 Time ✓ Catchup Read
 追赶读 

16. Access Patterns Tailing Read ✓ Write
 Write 写 Stream Segment 1 Segment 2 Segment 3 Segment 4 ✓ Tailing Read
 追尾 Time Catchup Read ✓ Catchup Read
 追赶读 

17. Write Kafka Pulsar Broker Broker Broker Broker Broker Broker Partition Partition Partition (Leader) (Follower) (Follower) 

18. Tailing Read Kafka Pulsar Broker Broker Broker Broker Broker Broker Partition Partition Partition (Leader) (Follower) (Follower) 

19. Catchup Read Kafka Pulsar Broker Broker Broker Broker Broker Broker Partition Partition Partition (Leader) (Follower) (Follower) 

20. Infinite Stream ✓ Write
 写 Bookies Brokers Stream Segment 1 Segment 2 Segment 3 Segment 4 ✓ Tailing Read
 追尾 Time ✓ Catchup Read
 追赶读 

21. Infinite Stream ✓ Write
 写 Tiered Storage Bookies Brokers Stream Segment 1 Segment 2 Segment 3 Segment 4 ✓ Tailing Read
 追尾 Time ✓ Catchup Read
 追赶读 

22. Tiered Storage • Offloader • When: size-based, time-based, or triggered by pulsar-admin • How: copy a segment to tiered storage, and delete it from bookkeeper • Access: broker knows how to read the data back, or bypass read 
 the offloaded segments directly • Available Offloaders • Cloud Offloder : AWS, GCS, Azure, … • HDFS, Ceph, … 

23. Stream as a Unified View on Data Segment Readers Producers Stream Segment 1 Segment 2 Segment 3 Segment 4 Segment 5 Segment 6 Consumers Time 

24. Data Processing on Pulsar Bounded Stream Bounded Stream Stream Segment 1 Segment 2 Segment 3 Segment 4 Segment 5 Segment 6 Time Unbounded Stream Unbounded Stream 

25.Pulsar + Flink 

26. Goals • Flink + Pulsar • Streaming Connectors • Source Connectors • PulsarCatalog: Schema Integration • PulsarStateBackend • Pulsar for the unified view of Data, Flink for the unified view of Computing 

27.Done 

28.Streaming Source -> Streaming Sink 

29.Streaming Source -> Streaming Table Sink