Pulsar 2.3.0 Overview

1 .Pulsar 2.3.0 @sijieg 

2 . Agenda • Bookie & Broker • Schema • Security • Functions • Connectors • Clients 

3 .Bookie & Broker 

4 . Memory Management • Direct Memory & Netty Buffer Pool • Problems: • -Xms2g -Xmx2g -XX:MaxDirectMemorySize=4g • io.netty.util.internal.OutOfDirectMemoryError 

5 . MM - ByteBufAllocator • ByteBufAllocator • PoolingPolicy • UnpooledHeap, PooledDirect • OutOfMemoryPolicy • ThrowException, FallbackToHeap • LeakDetectionPolicy • Disabled, Simple, Advanced, Paranoid 

6 .ReadAhead Amplification 

7 .Sticky Read 

8 .Sticky Read 

9 . Performance & Metadata • Performance • CPU-Affinity • JCTools - Lock-free concurrent data structures • Metadata • BookKeeper: Support Etcd as a metadata store • Reverse Index : Ledger => Topic, Subscription, Cursor 

10 .Schema 

11 . New Schemas • KeyValue Schema • AUTO_PRODUCE Schema • AUTO_CONSUME Schema 

12 .Support in new language clients • C++ • Python • JSON • Avro • Primitives: str, bytes 

13 .Security 

14 . Before 2.3.0 • TLS • Athenz • Password 

15 . JWT Authentication • JSON Web Tokens 

16 .Functions 

17 . Pulsar Functions in K8S • Demo 

18 .Python Function State 

19 .Connectors 

20 . New Connectors • CDC • Debezium: MySQL, Postgres, MongoDB, and … • Canal: MySQL • MongoDB • HBase • Elastic Search • Local Filesystem 

21 . New Connectors • CDC • Debezium: MySQL, Postgres, MongoDB, and … • Canal: MySQL • MongoDB Thanks! • HBase • Elastic Search - @tuteng - @ambition119 • Local Filesystem - @murong00 

22 .Clients 

23 . Java • Multi-Hosts Service Url • pulsar://host1:6650,host2:6650,host3:6650/ • Auto Partitions Change Discovery • GetPartitionsForTopic • Consumer: pauseMessageListener, resumeMessageListener 

24 . C++ • Consumer: • SubscriptionInitialPosition • receiveAsync • Producer: Flush • Hostname verifications on TLS connections • Avro Schema 

25 . Go • Caught up a lot of features in CGo client 

26 . 2.4.0 and more • PIP-30: Kerberos Authentication • PIP-32: Go Function, More languages to support • PIP-31: Transactional Streaming • System Topic, Replicated Subscription, Key Ordered Dispatching, … • NodeJS Client, Native Go Client, … • More Connectors, Integrations and Larger ecosystem 

27 . Community ✓ Twitter: @apache_pulsar ✓ Wechat Subscription: ApachePulsar ✓ Mailing Lists
 dev@pulsar.apache.org, users@pulsar.apache.org ✓ Slack
 https://apache-pulsar.slack.com (#china) ✓ Localization
 https://crowdin.com/project/apache-pulsar ✓ Github
 https://github.com/apache/pulsar
 https://github.com/apache/bookkeeper 

28 .