Transaction Preview

展开查看详情

1.

2.Transaction Support in Pulsar Sijie Guo Apache Pulsar / BookKeeper PMC Member Yong Zhang Apache Pulsar Contributor

3.Messaging Semantics • At-most once • At-least once • Exactly once

4.Messaging Semantics • At-most once Before 1.20.0-incubating • At-least once • Exactly once

5.Messaging Semantics • At-most once • At-least once PIP-6: Guaranteed • Exactly once Message Deduplication

6.Revisit Existing Semantics

7.Pulsar’s Existing Semantics send(m1) Log Producer Broker

8.Pulsar’s Existing Semantics append(m1) Log Producer Broker

9.Pulsar’s Existing Semantics m1 Log Producer Broker

10.Pulsar’s Existing Semantics ack(m1) m1 Log Producer Broker

11.Pulsar’s Existing Semantics ack(m1) m1 Log Producer Broker

12.Pulsar’s Existing Semantics send(m2) m1 Log Producer Broker

13.Pulsar’s Existing Semantics append(m2) m1m2 Log Producer Broker

14.Pulsar’s Existing Semantics ack(m2) m1m2 Log Producer Broker

15.Pulsar’s Existing Semantics What do we do now? ack(m2) m1m2 Log Producer Broker

16.At Least Once send(m2) m1m2 Log Producer Broker

17.At Least Once append(m2) m1m2m2 Log Producer Broker

18.At Least Once Duplicates ! ! append(m2) m1m2m2 Log Producer Broker

19.Why the duplicates are introduced? • Broker can fail • The request from Producer to Broker can fail • Producer or Consumer can fail

20.I want exactly-once

21.Message Deduplication • Producer: Idempotent Producer • Broker: Guaranteed Message Deduplication (PIP-6) • Consumer: Reader + Checkpoints (Flink / Spark)

22.Idempotent Producer • Producer Name - Identify who is producing the messages • Sequence ID - Identify the message • Producer Name + Sequence ID: The unique identifier for a message

23.Guaranteed Message Deduplication • Broker maintains a map between Producer Name and Last-Produced-Sequence-ID • Broker accepts messages if the sequence id of a new message is larger than its last produced sequence id • Broker treats messages whose sequence id are smaller • Broker keeps the map in a de-duplication cursor

24.Exactly Once send(1, m1) Log Producer Broker

25.Exactly Once append(1, m1) 1, m1 Log Producer Broker

26.Exactly Once append(2, m2) 2, m2 1, m1 Log Producer Broker

27.Exactly Once What do we do now? ack(2, m2) 2, m2 1, m1 Log Producer Broker

28.Exactly Once send(2, m2) 2, m2 1, m1 Log Producer Broker

29.Exactly Once append(2, m2) 2, m2 1, m1 Log Producer Broker