TGIP-001：Pulsar Basics（Sijie G）

1 .Pulsar Concepts TGIP-CN Episode 001 

2 .Pulsar with legos ❏ We will now demonstrate how Pulsar works with legos ❏ Concepts are shown ❏ Publish / Subscribe ❏ Topics ❏ Partitions ❏ Brokers ❏ Producers / Consumers / Readers 

3 .Producers Producer 0 Producer 1 Producers output the Producers send all events events to Pulsar. What as key/value pairs. each producer does is Pulsar can make a topic use case specific. conform to a schema. 

4 .Topics Producer 0 Producer 1 Topic Partition 0 Partition 1 Partition 2 Data is divided into All events are sent and partitions. Partitions received on topics. are both logical and Topic is the logic unit. physical divisions. 

5 .Brokers Producer 0 Producer 1 Topic Partition 0 Partition 1 Partition 2 Broker X Broker Y Broker Z The brokers are responsible for receiving and dispatching events. Data is stored elsewhere 

6 . Producer 0 Producer 1 Topic Partition 0 Partition 1 Partition 2 Broker X Broker Y Broker Z A consumer with an exclusive Consumer (P012) subscription receives events from all topic partitions it subscribes to, and connects to broker X, Y, Z Subscription A 

7 .Subscription Producer 0 Producer 1 Modes Topic Partition 0 Partition 1 Partition 2 Broker X Broker Y Broker Z Consumer (P012) Consumer (P012) Consumer (P012) Consumer (P) Consumer (P012) Consumer (P012) Failover Subscription Shared Subscription Key_Shared Subscription In failover, all partitions are In shared, messages are In key_shared, messages with consumed by one consumer and sent in a round robin the same key are consistently will fail over to hot spare on fail. way to all consumers. routed to the same consumer. 

8 .Topic (Partition) Event streams are append- only sequences of data. Streams can be truncated for space. ❏ Infinite Event Stream Event Stream Segment 1 Segment 2 Segment 3 Streams are made up of individual segments. New segments are Entry 1 Entry 2 Entry 3 Entry 4 rolled when events are appended to streams Msg 1 Msg 2 Msg 3 Msg 4 Segments are made up of individual entries. This is how pieces of data are stored. Entries are made up of individual messages. This is how Pulsar messages are stored. 

9 .Segment - Apache BookKeeper Event Stream Segment 1 Segment 2 Segment 3 Entry 1 Entry 2 Entry 3 Entry 4 Msg 1 Msg 2 Msg 3 Msg 4 

10 .Message ❏ Data*: bytes ❏ Key: string, optional ❏ Publish Time: long, timestamp generated by the producer ❏ Event Time: long, specified by the producer application ❏ Properties: Map<String, String>, a key/value property map specified by applications ❏ MessageId: the position of a message within a topic ❏ SchemaVersion: schema version of the message 

11 .Message ID Topic Partition 1 Partition 2 Partition 0 Segment 1 Segment 2 Segment 3 Entry 1 Entry 2 Entry 3 Entry 4 Msg 1 Msg 2 Msg 3 Msg 4 ledger-id, entry-id, batch-index, partition-index 

12 .Message ID Examples ❏ 100, 10, 2, 0: ptn 0, segment 100, entry 10, batch 2. ❏ 100, 10, -1, 0 ❏ 100, 10, -1, -1 ❏ 100, 10, 0, -1 ❏ ... 

13 .Cursor: Subscription State Subscription B (2, 2) Subscription C (3, 2) Subscription A (1, 1) Partition (Event Stream) 

14 .Acknowledge Cumulatively Subscription B (2, 2) acknowledgeCumulative (4, 3) Subscription C (3, 2) Subscription A (1, 1) Partition (Event Stream) 

15 .Acknowledge Cumulatively Subscription B (2, 2) acknowledgeCumulative (4, 3) Subscription C (3, 2) Subscription A Subscription A (1, 1) (4, 3) Partition (Event Stream) 

16 .Acknowledge moves cursors Subscription B (2, 2) acknowledgeCumulative (4, 3) Subscription C (3, 2) Subscription A (4, 3) Partition (Event Stream) 

17 .Seek / Reset-Cursor Reset Cursor Subscription B (1, 1) (2, 2) Subscription C (3, 2) Subscription A (4, 3) Partition (Event Stream) 

18 .Seek / Reset-Cursor Reset Cursor Subscription B (1, 1) (2, 2) Subscription C (3, 2) Subscription A Subscription A (1, 1) (4, 3) Partition (Event Stream) 

19 .Seek / Reset-Cursor Reset Cursor Subscription B (1, 1)c (2, 2) Subscription C (3, 2) Subscription A (1, 1) Partition (Event Stream) 

20 .Reader: Non-durable Cursor Subscription B (2, 2) Subscription A Subscription C (1, 1) (3, 2) Partition (Event Stream) Non-durable Exclusive Subscription (1, 100) Reader (1, 100) 

21 .Reader: Non-durable Cursor Subscription B (2, 2) Subscription A Subscription C (1, 1) (3, 2) Partition (Event Stream) Close Reader will trigger delete non-durable subscription at broker side Non-durable Exclusive Subscription (1, 100) Reader (1, 100) 

22 .Tenant & Namespace persistent :// my-tenant / my-namespace / my-topic Topic Domain Tenant Namespace Topic 

23 .Fully Qualified Name ❏ persistent://my-tenant/my-namespace/my-topic ❏ p://my-tenant/my-namespace/my-topic ❏ my-tenant/my-namespace/my-topic ❏ my-topic => persistent://public/default/my-topic