Apache Pulsar: 云原⽣消息系统简介

展开查看详情

1.Apache Pulsar: 云原⽣生消息系统简介 @sijieg

2. Who am I • Founder & CEO of StreamNative • Apache Pulsar PMC Member • Apache BookKeeper PMC Member • Ex-Twitter, Ex-Yahoo • Interested in technologies around Event Streaming

3.What is Apache Pulsar? 2003 2010 2012 2006 2011

4. “Flexible Pub/Sub messaging backed by durable stream storage”

5. A brief history of Pulsar • 2012 Pulsar idea started • 5+ years on production, 100+ applications, 10+ data centers • 2016/09 Yahoo open sourced Pulsar • 2017/06 Yahoo donated Pulsar to ASF • 2018/09 Pulsar graduated as a Top-Level project • 25 committers, 139 contributors, 843 forks, 3700+ stars

6. Why Pulsar? • Challenges • Multi-tenant, millions of topics, low latency, persistent, geo-replication • Existing systems have problems • Storage and serving/computing layers are coupled • Problems of page-cache • Problems of partition-centric design • IT Infrastructure is moving towards container and cloud

7.1.x - Pub/Sub Messaging

8.PubSub Messaging Multi language clients Messaging Stream Storage Low Latency Storage

9. PubSub Messaging Multi language clients Messaging Stream Storage Low Latency Storage

10. Pulsar Concepts Multi language clients Messaging Stream Storage Low Latency Storage

11. Multi Tenancy Multi language clients Messaging Stream Storage Low Latency Storage

12. Multi Tenancy Multi language clients • Admin Operations : REST API, `bin/pulsar-admin` Messaging • Namespace Policies • Storage quota • Rate limiting - dispatch/publish/replication rate Stream Storage • Retention, TTL • Isolation - Anti affinity groups Low Latency Storage

13. Multi Tenancy Multi language clients Messaging Stream Storage Low Latency Storage

14. Flexible Messaging Multi language clients Messaging Stream Storage Low Latency Storage

15. Exclusive Subscription Multi language clients Messaging Stream Storage Low Latency Storage

16. Failover Subscription Multi language clients Messaging Stream Storage Low Latency Storage

17. Shared Subscription Multi language clients Messaging Stream Storage Low Latency Storage

18. Flexible Messaging Multi language clients • One copy of data Messaging • Consume multiple times • Consume messages in different ways • Exclusive / Failover: partition based ordered Stream Storage consumption • Shared: unordered consumption, go beyond the limit of partitions Low Latency Storage

19. Use Case: zhaopin.com Multi language clients Messaging Stream Storage Low Latency Storage

20. Use Case: zhaopin.com Multi language clients Messaging Stream Storage Low Latency Storage

21.Flexible Messaging - Key_Shared Multi language clients Messaging Stream Storage Low Latency Storage

22.Flexible Messaging - Key_Shared Multi language clients Messaging • Key_Shared • Key based ordered consumption • Go beyond the limit of partitions Stream Storage • Scale the consumption up and down Low Latency Storage

23. Flexible Messaging Multi language clients • More notes about messaging Messaging • Ack individual vs cumulatively • Subscription based retention • Retention, TTL Stream Storage • Dead letter topic • Delayed / scheduled messages Low Latency Storage

24. Flexible Messaging Multi language clients Messaging Stream Storage Low Latency Storage

25. Cloud-native Architecture Multi language clients Messaging Stream Storage Low Latency Storage

26. Cloud-native Architecture Multi language clients • Layered Architecture Messaging • Brokers: messaging & serving • Bookies: storage • Independent scalability Stream Storage • Instant failover & fault tolerant • Easy to scale up and down Low Latency Storage

27. Cloud-native Architecture Multi language clients Messaging Stream Storage Low Latency Storage

28. Segment Centric Storage Multi language clients Messaging Stream Storage Low Latency Storage

29. Segment Centric Storage Multi language clients Messaging Stream Storage Low Latency Storage