1.

2.Apache Pulsar BIGO | BIGO 

3.About Me - BIGO - Apache Pulsar Contributor - Apache Bookkeeper Contributor - StreamNative/Pulsar-Flink-Connector Contributor 

4.Motivation BIGO Kafka Kafka Ø o Ø o ISR VS. Ø o catch-up PageCache Ø o HDD Ø o Ø Apache Kafka Apache Pulsar 

5.Apache Pulsar BIGO 2019.12 2020.4 Apache Pulsar Apache Pulsar Apache Pulsar Apache Pulsar Apache Pulsar & Apache Kafka 2019.11 2020.4 2020.5 

6.BIGOer Apache Pulsar 

7.Apache Pulsar BIGO Baina (BIGO C++ ) KMM (Kafka Mirror Maker) Flink SQL … … PUB - SUB 

8. (Apache Pulsar 2.5.1) Ø Pulsar broker Ø Pulsar broker Cache bookie Ø broker OOM Ø Bookie direct memory OOM Ø Bookie Ø Journal HDD fsync bookie add entry 99th latency Ø bookie add entry latency Ø Pulsar client Lookup Timeout Exception” Ø ZooKeeper Pulsar Ø Reader API(eg. Pulsar Flink Connector) Pulsar topic Pulsar 2.5.2 

9.Apache Pulsar ü Bookie Journal/Ledger § OS: 1 ~ 2 GB ü Journal/Ledger HDD ZooKeeper § JVM: 1/2 dataDir/dataLogDir Journal/Ledger § heap: 1/3 § direct memory: 2/3 § PageCache: 1/2 o jvm heap/gc o bytes in per broker o message in per broker o loadbalance o broker Cache o bookie client quarantine ratio o bookie client request queue Broker 

10.Ø § Broker § Bookie Ø § Broker broker direct memory OOM § Broker consumer/reader consumer/reader GC Ø Cache Ø ZooKeeper Ø auto bundle split 

11. —Broker (PR-6772) new_avg = old_avg * factor + (1-factor) * avg new_avg: newest average resoruce usage old_avg: old average resource usge which is calculate in last round. factor: the decrease factor, default value is 0.9 avg: the average resource usage of the brokers Loadbalance : broker resource usage > average resource usage + threshold 0 avg avg+threshold 100 loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl. ThresholdShedder 

12.Ø § Broker § Bookie Ø § Broker broker direct memory OOM § Broker consumer/reader consumer/reader GC Ø Cache Ø ZooKeeper Ø auto bundle split 

13.—Bookie 

14.—Bookie Bookkeeper PR-2327 Bookie Client 

15.Ø § Broker § Bookie Ø § Broker broker direct memory OOM § Broker consumer/reader consumer/reader GC Ø Cache Ø ZooKeeper Ø auto bundle split 

16.– Broker 

17.Ø § Broker § Bookie Ø § Broker broker direct memory OOM § Broker consumer/reader consumer/reader GC Ø Cache Ø ZooKeeper Ø auto bundle split 

18.– Consumer Consumer N message broker M entry ==> Consumer ==> GC Broker Message request Message Push 

19. – Consumer (PR-6719) avgMessagesPerEntry = avgMessagePerEntry * avgPercent + (1 - avgPercent) * new Value Default: avgMessagePerEntry = 1000 avgPercent = 0.9 # Precise dispatcher flow control according to history message number of each entry preciseDispatcherFlowControl=true 

20.Ø § Broker § Bookie Ø § Broker broker direct memory OOM § Broker consumer/reader consumer/reader GC Ø Cache Ø ZooKeeper Ø auto bundle split 

21. Cache (PR-6769, PR-7894) Ø broker Cache Tailing Read Ø bookie write Cache(Memtable) Ø bookie read Cache Ø OS PageCache Has Active Cursor: § durable cursor § Cursor lag managedLedgerCursorBackloggedThreshold ( 1000 entry) 

22.Ø § Broker § Bookie Ø § Broker broker direct memory OOM § Broker consumer/reader consumer/reader GC Ø Cache Ø ZooKeeper Ø auto bundle split 

23. ZooKeeper § HDD ZooKeeper dataDir/dataLogDir IO bookie Journal/Ledger ) SSD § ZooKeeper dataDir dataLogDir SSD § broker/bookie ZooKeeper 

24.Ø § Broker § Bookie Ø § Broker broker direct memory OOM § Broker consumer/reader consumer/reader GC Ø Cache Ø ZooKeeper Ø auto bundle split 

25. auto bundle split Pulsar bundle split producer/consumer/reader namespace bundle auto bundle split 

26.Pulsar Pulsar 

27.Q&A 

28. ! Before I Get Old