展开查看详情
1.
2.Apache Pulsar BIGO | BIGO
3.About Me - BIGO - Apache Pulsar Contributor - Apache Bookkeeper Contributor - StreamNative/Pulsar-Flink-Connector Contributor
4.Motivation BIGO Kafka Kafka Ø o Ø o ISR VS. Ø o catch-up PageCache Ø o HDD Ø o Ø Apache Kafka Apache Pulsar
5.Apache Pulsar BIGO 2019.12 2020.4 Apache Pulsar Apache Pulsar Apache Pulsar Apache Pulsar Apache Pulsar & Apache Kafka 2019.11 2020.4 2020.5
6.BIGOer Apache Pulsar
7.Apache Pulsar BIGO Baina (BIGO C++ ) KMM (Kafka Mirror Maker) Flink SQL … … PUB - SUB
8. (Apache Pulsar 2.5.1) Ø Pulsar broker Ø Pulsar broker Cache bookie Ø broker OOM Ø Bookie direct memory OOM Ø Bookie Ø Journal HDD fsync bookie add entry 99th latency Ø bookie add entry latency Ø Pulsar client Lookup Timeout Exception” Ø ZooKeeper Pulsar Ø Reader API(eg. Pulsar Flink Connector) Pulsar topic Pulsar 2.5.2
9.Apache Pulsar ü Bookie Journal/Ledger § OS: 1 ~ 2 GB ü Journal/Ledger HDD ZooKeeper § JVM: 1/2 dataDir/dataLogDir Journal/Ledger § heap: 1/3 § direct memory: 2/3 § PageCache: 1/2 o jvm heap/gc o bytes in per broker o message in per broker o loadbalance o broker Cache o bookie client quarantine ratio o bookie client request queue Broker
10.Ø § Broker § Bookie Ø § Broker broker direct memory OOM § Broker consumer/reader consumer/reader GC Ø Cache Ø ZooKeeper Ø auto bundle split
11. —Broker (PR-6772) new_avg = old_avg * factor + (1-factor) * avg new_avg: newest average resoruce usage old_avg: old average resource usge which is calculate in last round. factor: the decrease factor, default value is 0.9 avg: the average resource usage of the brokers Loadbalance : broker resource usage > average resource usage + threshold 0 avg avg+threshold 100 loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl. ThresholdShedder
12.Ø § Broker § Bookie Ø § Broker broker direct memory OOM § Broker consumer/reader consumer/reader GC Ø Cache Ø ZooKeeper Ø auto bundle split
13.—Bookie
14.—Bookie Bookkeeper PR-2327 Bookie Client
15.Ø § Broker § Bookie Ø § Broker broker direct memory OOM § Broker consumer/reader consumer/reader GC Ø Cache Ø ZooKeeper Ø auto bundle split
16.– Broker
17.Ø § Broker § Bookie Ø § Broker broker direct memory OOM § Broker consumer/reader consumer/reader GC Ø Cache Ø ZooKeeper Ø auto bundle split
18.– Consumer Consumer N message broker M entry ==> Consumer ==> GC Broker Message request Message Push
19. – Consumer (PR-6719) avgMessagesPerEntry = avgMessagePerEntry * avgPercent + (1 - avgPercent) * new Value Default: avgMessagePerEntry = 1000 avgPercent = 0.9 # Precise dispatcher flow control according to history message number of each entry preciseDispatcherFlowControl=true
20.Ø § Broker § Bookie Ø § Broker broker direct memory OOM § Broker consumer/reader consumer/reader GC Ø Cache Ø ZooKeeper Ø auto bundle split
21. Cache (PR-6769, PR-7894) Ø broker Cache Tailing Read Ø bookie write Cache(Memtable) Ø bookie read Cache Ø OS PageCache Has Active Cursor: § durable cursor § Cursor lag managedLedgerCursorBackloggedThreshold ( 1000 entry)
22.Ø § Broker § Bookie Ø § Broker broker direct memory OOM § Broker consumer/reader consumer/reader GC Ø Cache Ø ZooKeeper Ø auto bundle split
23. ZooKeeper § HDD ZooKeeper dataDir/dataLogDir IO bookie Journal/Ledger ) SSD § ZooKeeper dataDir dataLogDir SSD § broker/bookie ZooKeeper
24.Ø § Broker § Bookie Ø § Broker broker direct memory OOM § Broker consumer/reader consumer/reader GC Ø Cache Ø ZooKeeper Ø auto bundle split
25. auto bundle split Pulsar bundle split producer/consumer/reader namespace bundle auto bundle split
26.Pulsar Pulsar
27.Q&A
28. ! Before I Get Old