Pulsar 2.3.0 Overview

Pulsar和核心概念介绍: • Bookie & Broker • Schema • Security • Functions • Connectors • Clients
展开查看详情

1.Pulsar 2.3.0 @sijieg

2. Agenda • Bookie & Broker • Schema • Security • Functions • Connectors • Clients

3.Bookie & Broker

4. Memory Management • Direct Memory & Netty Buffer Pool • Problems: • -Xms2g -Xmx2g -XX:MaxDirectMemorySize=4g • io.netty.util.internal.OutOfDirectMemoryError

5. MM - ByteBufAllocator • ByteBufAllocator • PoolingPolicy • UnpooledHeap, PooledDirect • OutOfMemoryPolicy • ThrowException, FallbackToHeap • LeakDetectionPolicy • Disabled, Simple, Advanced, Paranoid

6.ReadAhead Amplification

7.Sticky Read

8.Sticky Read

9. Performance & Metadata • Performance • CPU-Affinity • JCTools - Lock-free concurrent data structures • Metadata • BookKeeper: Support Etcd as a metadata store • Reverse Index : Ledger => Topic, Subscription, Cursor

10.Schema

11. New Schemas • KeyValue Schema • AUTO_PRODUCE Schema • AUTO_CONSUME Schema

12.Support in new language clients • C++ • Python • JSON • Avro • Primitives: str, bytes

13.Security

14. Before 2.3.0 • TLS • Athenz • Password

15. JWT Authentication • JSON Web Tokens

16.Functions

17. Pulsar Functions in K8S • Demo

18.Python Function State

19.Connectors

20. New Connectors • CDC • Debezium: MySQL, Postgres, MongoDB, and … • Canal: MySQL • MongoDB • HBase • Elastic Search • Local Filesystem

21. New Connectors • CDC • Debezium: MySQL, Postgres, MongoDB, and … • Canal: MySQL • MongoDB Thanks! • HBase • Elastic Search - @tuteng - @ambition119 • Local Filesystem - @murong00

22.Clients

23. Java • Multi-Hosts Service Url • pulsar://host1:6650,host2:6650,host3:6650/ • Auto Partitions Change Discovery • GetPartitionsForTopic • Consumer: pauseMessageListener, resumeMessageListener

24. C++ • Consumer: • SubscriptionInitialPosition • receiveAsync • Producer: Flush • Hostname verifications on TLS connections • Avro Schema

25. Go • Caught up a lot of features in CGo client

26. 2.4.0 and more • PIP-30: Kerberos Authentication • PIP-32: Go Function, More languages to support • PIP-31: Transactional Streaming • System Topic, Replicated Subscription, Key Ordered Dispatching, … • NodeJS Client, Native Go Client, … • More Connectors, Integrations and Larger ecosystem

27. Community ✓ Twitter: @apache_pulsar ✓ Wechat Subscription: ApachePulsar ✓ Mailing Lists
 dev@pulsar.apache.org, users@pulsar.apache.org ✓ Slack
 https://apache-pulsar.slack.com (#china) ✓ Localization
 https://crowdin.com/project/apache-pulsar ✓ Github
 https://github.com/apache/pulsar
 https://github.com/apache/bookkeeper

28.

StreamNative 是一家围绕 Apache Pulsar 和 Apache BookKeeper 打造下一代流数据平台的开源基础软件公司。秉承 Event Streaming 是大数据的未来基石、开源是基础软件的未来这两个理念,专注于开源生态和社区的构建,致力于前沿技术。