1.Database Streaming with Apache Pulsar Sijie Guo — twitter: @sijieg wechat: guosijie_ 

2.Who am I • Sijie Guo • Pulsar, BookKeeper, Hive, HBase, … • Streamlio Cofounder • Yahoo -> Twitter -> Streamlio • 华中科⼤ -> 中科院计算所 

3.Agenda 1. The Beauty of Change Data Capture 2. Streaming MySQL - Connectors/Functions, Schema 3. Interactive Queries - Presto SQL 

4.Agenda 1. The Beauty of Change Data Capture 2. Streaming MySQL - Connectors/Functions, Schema 3. Interactive Queries - Presto SQL 

5.Traditional Batch ETL Pipeline 

6.Traditional Batch ETL Pipeline High Latency Huge number of Jobs Error prone Manual Schema Updates 

7.Current Mess 

8.Pulsar as the backbone 

9.Getting data to Pulsar 

10.Option 1 - Dural Writes 

11.Option 1 - Dural Writes Data consistency??? :( 

12.Option 2 - Event Sourcing 

13.Option 2 - Event Sourcing Violates read-your-writes consistency :( 

14.Option 3 - Change Data Capture 

15. “In databases, a design pattern that captures individual database changes to a stream of changes” 

16.Option 3 - Change Data Capture Yeah ! :) 

17.Agenda 1. The Beauty of Change Data Capture 2. Streaming MySQL - Connectors/Functions, Schema 3. Interactive Queries - Presto SQL 

18.Pulsar IO Framework 

19.Mysql —> HDFS 

20.Mysql —> Pulsar 

21.Function Overview Light weight Compute Framework Language Native (Java, Python) Flexible deployment (Thread, Process, Container, K8S) Pulsar Functions Managed State ETL, Filtering, Routing, … 

22.Function in Action 

23.Function Example public class WordCountFunction implements Function<String, Void> { @Override public Void process(String input, Context context) { Arrays.asList(input.split(“\\s+”)) .forEach(word -> context.incrCounter(word, 1)); return null; } } 

24.Function Submission $ bin/pulsar-admin functions create \ --jar target/my-functions.jar \ --classname org.example.functions.MyFunction \ Submission --cpu 8 \ --ram 8589934592 \ --disk 10737418240 \ —-inputs input_topic —-output output_topic $ bin/pulsar-admin functions local run \ Local Run --jar target/my-functions.jar \ --classname org.example.functions.MyFunction \ —-inputs input_topic —-output output_topic 

25.Function Deployment Function Function Function wordcount-1 wordcount-1 transform-2 Function Function Pod 1 wordcount-1 transform-2 Worker Function dataroute-1 Worker Node 1 Pod 2 Broker 1 Broker 1 Broker 1 Node 1 Node 2 Pod 3 Thread Process K8S 

26.IO Connectors A light weight connector framework built on functions runtime 

27.Running Connectors $ ./bin/pulsar-admin source create \ --tenant test \ --namespace ns1 \ Source --name twitter-source \ --destinationTopicName twitter_data \ --sourceConfigFile examples/twitter.yml \ --source-type twitter $ ./bin/pulsar-admin sink create \ Sink --tenant public \ --namespace default \ --name cassandra-test-sink \ --sink-type cassandra \ --sinkConfigFile examples/cassandra-sink.yml \ --inputs test_cassandra 

28.Connector Config configs: roots: "localhost:9042" keyspace: "pulsar_test_keyspace" columnFamily: "pulsar_test_table" keyname: "key" columnName: "col" 

29.Debezium Overview Open source CDC Records row-level changes via WAL Supports MYSQL, MongoDB, PostgreSQL, Oracle, 
 Debezium SQL Server