Apache Pulsar: 云原⽣消息系统简介

1 .Apache Pulsar: 云原⽣生消息系统简介 @sijieg 

2 . Who am I • Founder & CEO of StreamNative • Apache Pulsar PMC Member • Apache BookKeeper PMC Member • Ex-Twitter, Ex-Yahoo • Interested in technologies around Event Streaming 

3 .What is Apache Pulsar? 2003 2010 2012 2006 2011 

4 . “Flexible Pub/Sub messaging backed by durable stream storage” 

5 . A brief history of Pulsar • 2012 Pulsar idea started • 5+ years on production, 100+ applications, 10+ data centers • 2016/09 Yahoo open sourced Pulsar • 2017/06 Yahoo donated Pulsar to ASF • 2018/09 Pulsar graduated as a Top-Level project • 25 committers, 139 contributors, 843 forks, 3700+ stars 

6 . Why Pulsar? • Challenges • Multi-tenant, millions of topics, low latency, persistent, geo-replication • Existing systems have problems • Storage and serving/computing layers are coupled • Problems of page-cache • Problems of partition-centric design • IT Infrastructure is moving towards container and cloud 

7 .1.x - Pub/Sub Messaging 

8 .PubSub Messaging Multi language clients Messaging Stream Storage Low Latency Storage 

9 . PubSub Messaging Multi language clients Messaging Stream Storage Low Latency Storage 

10 . Pulsar Concepts Multi language clients Messaging Stream Storage Low Latency Storage 

11 . Multi Tenancy Multi language clients Messaging Stream Storage Low Latency Storage 

12 . Multi Tenancy Multi language clients • Admin Operations : REST API, `bin/pulsar-admin` Messaging • Namespace Policies • Storage quota • Rate limiting - dispatch/publish/replication rate Stream Storage • Retention, TTL • Isolation - Anti affinity groups Low Latency Storage 

13 . Multi Tenancy Multi language clients Messaging Stream Storage Low Latency Storage 

14 . Flexible Messaging Multi language clients Messaging Stream Storage Low Latency Storage 

15 . Exclusive Subscription Multi language clients Messaging Stream Storage Low Latency Storage 

16 . Failover Subscription Multi language clients Messaging Stream Storage Low Latency Storage 

17 . Shared Subscription Multi language clients Messaging Stream Storage Low Latency Storage 

18 . Flexible Messaging Multi language clients • One copy of data Messaging • Consume multiple times • Consume messages in different ways • Exclusive / Failover: partition based ordered Stream Storage consumption • Shared: unordered consumption, go beyond the limit of partitions Low Latency Storage 

19 . Use Case: zhaopin.com Multi language clients Messaging Stream Storage Low Latency Storage 

20 . Use Case: zhaopin.com Multi language clients Messaging Stream Storage Low Latency Storage 

21 .Flexible Messaging - Key_Shared Multi language clients Messaging Stream Storage Low Latency Storage 

22 .Flexible Messaging - Key_Shared Multi language clients Messaging • Key_Shared • Key based ordered consumption • Go beyond the limit of partitions Stream Storage • Scale the consumption up and down Low Latency Storage 

23 . Flexible Messaging Multi language clients • More notes about messaging Messaging • Ack individual vs cumulatively • Subscription based retention • Retention, TTL Stream Storage • Dead letter topic • Delayed / scheduled messages Low Latency Storage 

24 . Flexible Messaging Multi language clients Messaging Stream Storage Low Latency Storage 

25 . Cloud-native Architecture Multi language clients Messaging Stream Storage Low Latency Storage 

26 . Cloud-native Architecture Multi language clients • Layered Architecture Messaging • Brokers: messaging & serving • Bookies: storage • Independent scalability Stream Storage • Instant failover & fault tolerant • Easy to scale up and down Low Latency Storage 

27 . Cloud-native Architecture Multi language clients Messaging Stream Storage Low Latency Storage 

28 . Segment Centric Storage Multi language clients Messaging Stream Storage Low Latency Storage 

29 . Segment Centric Storage Multi language clients Messaging Stream Storage Low Latency Storage