TGIP-001:Pulsar Basics(Sijie G)

第一期 TGIP 演讲文件

展开查看详情

1.Pulsar Concepts TGIP-CN Episode 001

2.Pulsar with legos ❏ We will now demonstrate how Pulsar works with legos ❏ Concepts are shown ❏ Publish / Subscribe ❏ Topics ❏ Partitions ❏ Brokers ❏ Producers / Consumers / Readers

3.Producers Producer 0 Producer 1 Producers output the Producers send all events events to Pulsar. What as key/value pairs. each producer does is Pulsar can make a topic use case specific. conform to a schema.

4.Topics Producer 0 Producer 1 Topic Partition 0 Partition 1 Partition 2 Data is divided into All events are sent and partitions. Partitions received on topics. are both logical and Topic is the logic unit. physical divisions.

5.Brokers Producer 0 Producer 1 Topic Partition 0 Partition 1 Partition 2 Broker X Broker Y Broker Z The brokers are responsible for receiving and dispatching events. Data is stored elsewhere

6. Producer 0 Producer 1 Topic Partition 0 Partition 1 Partition 2 Broker X Broker Y Broker Z A consumer with an exclusive Consumer (P012) subscription receives events from all topic partitions it subscribes to, and connects to broker X, Y, Z Subscription A

7.Subscription Producer 0 Producer 1 Modes Topic Partition 0 Partition 1 Partition 2 Broker X Broker Y Broker Z Consumer (P012) Consumer (P012) Consumer (P012) Consumer (P) Consumer (P012) Consumer (P012) Failover Subscription Shared Subscription Key_Shared Subscription In failover, all partitions are In shared, messages are In key_shared, messages with consumed by one consumer and sent in a round robin the same key are consistently will fail over to hot spare on fail. way to all consumers. routed to the same consumer.

8.Topic (Partition) Event streams are append- only sequences of data. Streams can be truncated for space. ❏ Infinite Event Stream Event Stream Segment 1 Segment 2 Segment 3 Streams are made up of individual segments. New segments are Entry 1 Entry 2 Entry 3 Entry 4 rolled when events are appended to streams Msg 1 Msg 2 Msg 3 Msg 4 Segments are made up of individual entries. This is how pieces of data are stored. Entries are made up of individual messages. This is how Pulsar messages are stored.

9.Segment - Apache BookKeeper Event Stream Segment 1 Segment 2 Segment 3 Entry 1 Entry 2 Entry 3 Entry 4 Msg 1 Msg 2 Msg 3 Msg 4

10.Message ❏ Data*: bytes ❏ Key: string, optional ❏ Publish Time: long, timestamp generated by the producer ❏ Event Time: long, specified by the producer application ❏ Properties: Map<String, String>, a key/value property map specified by applications ❏ MessageId: the position of a message within a topic ❏ SchemaVersion: schema version of the message

11.Message ID Topic Partition 1 Partition 2 Partition 0 Segment 1 Segment 2 Segment 3 Entry 1 Entry 2 Entry 3 Entry 4 Msg 1 Msg 2 Msg 3 Msg 4 ledger-id, entry-id, batch-index, partition-index

12.Message ID Examples ❏ 100, 10, 2, 0: ptn 0, segment 100, entry 10, batch 2. ❏ 100, 10, -1, 0 ❏ 100, 10, -1, -1 ❏ 100, 10, 0, -1 ❏ ...

13.Cursor: Subscription State Subscription B (2, 2) Subscription C (3, 2) Subscription A (1, 1) Partition (Event Stream)

14.Acknowledge Cumulatively Subscription B (2, 2) acknowledgeCumulative (4, 3) Subscription C (3, 2) Subscription A (1, 1) Partition (Event Stream)

15.Acknowledge Cumulatively Subscription B (2, 2) acknowledgeCumulative (4, 3) Subscription C (3, 2) Subscription A Subscription A (1, 1) (4, 3) Partition (Event Stream)

16.Acknowledge moves cursors Subscription B (2, 2) acknowledgeCumulative (4, 3) Subscription C (3, 2) Subscription A (4, 3) Partition (Event Stream)

17.Seek / Reset-Cursor Reset Cursor Subscription B (1, 1) (2, 2) Subscription C (3, 2) Subscription A (4, 3) Partition (Event Stream)

18.Seek / Reset-Cursor Reset Cursor Subscription B (1, 1) (2, 2) Subscription C (3, 2) Subscription A Subscription A (1, 1) (4, 3) Partition (Event Stream)

19.Seek / Reset-Cursor Reset Cursor Subscription B (1, 1)c (2, 2) Subscription C (3, 2) Subscription A (1, 1) Partition (Event Stream)

20.Reader: Non-durable Cursor Subscription B (2, 2) Subscription A Subscription C (1, 1) (3, 2) Partition (Event Stream) Non-durable Exclusive Subscription (1, 100) Reader (1, 100)

21.Reader: Non-durable Cursor Subscription B (2, 2) Subscription A Subscription C (1, 1) (3, 2) Partition (Event Stream) Close Reader will trigger delete non-durable subscription at broker side Non-durable Exclusive Subscription (1, 100) Reader (1, 100)

22.Tenant & Namespace persistent :// my-tenant / my-namespace / my-topic Topic Domain Tenant Namespace Topic

23.Fully Qualified Name ❏ persistent://my-tenant/my-namespace/my-topic ❏ p://my-tenant/my-namespace/my-topic ❏ my-tenant/my-namespace/my-topic ❏ my-topic => persistent://public/default/my-topic

StreamNative 是一家围绕 Apache Pulsar 和 Apache BookKeeper 打造下一代流数据平台的开源基础软件公司。秉承 Event Streaming 是大数据的未来基石、开源是基础软件的未来这两个理念,专注于开源生态和社区的构建,致力于前沿技术。
关注他