How Apache Pulsar Helps Tencent Process Tens of Billions of Transactions Efficiently——Ningguo Ch

1 . How Apache Pulsar Helps Tencent Process Tens of Billions of Transactions Efficiently 

2 .Self-Introduction • Ningguo Chen, Lead Architect. • Joined Tencent in 2008. Lead Architect of Tencent Midas Billing Platform. Leads and participates in building Tencent QB E-shop, Tencent Midas Standard/Enterprise/Oversea Editions, etc, facilitating Tencent Midas becomes an all-round one-stop global billing platform. • Professional in Billing and Transaction system. Owner of 20+ patents. Mainly focused on providing billing services of high stability, efficiency and security for Tencent’s online and offline business. 

3 .Midas platform OverSeas Editon Enterprise Editon • 100+ Payment Channels WeChat Ads • 180+ Countries Cloud Up to 1,000,000 TPS • Full link Risk Control prise enter ov er Midas se ile a mob Standard Edition Games,Video,Club etc 米大师 y 30+ Billions of Accounts str op du Billions of Transactions en in Industry Edition Education，Travel，Traffic etc 

4 .Midas Architecture 

5 .Pain Points of Billing platform • High consistency of transaction, ensuring the consistency of payments and goods delivery. -Failure and timeout: A single payment of Midas often involves many internal and external systems. This leads to longer call chains and more exceptions, especially in network timeouts (e.g. overseas payment services), DB manipulation, goods delivery Failures. • High data reliability crossing regions • Oceans of timed processing - subscription & recurring billing, in-time account reconciliation • High Performance • High Scalability - automatically scales elastically according to the business scope 

6 .Our Solution • TDXA+TDMQ TDXA TDMQ TDXA, a framework solution for transaction control. Ensuring high consistency of transactions, high fault tolerance, high scalability. TDMQ，based on Pulsar，distributed Message Queue. EnablesTDXA to deal with messaging with high consistency and availability, and makes convergence of various exceptions with the state table. 

7 .Deployment of Tencent billing system App App App MQ’s Achievements: … … 1. Asynchronous data transmission between sytems 2. System Exception TDXA TDXA TDXA handling 2 3. High data 1 consistency MQ MQ 3 TDSQL TDSQL TDSQL Wechat Wechat Pay bank Pay Shenzhen Shanghai Hongkong 

8 .Our Requirements for MQ • Highly-consistency and highly-availability across regions • Massive storage, able to manage massive delayed messages and massive topics • Highly concurrent consumptions (the number of consumers is m uch larger than the number of queues) • High Scalability • Support message modification and more protocols, such as Restf ul, AMQP Pulsar perfectly supports the first 4 points. That’s why we choose Pulsar to be our infrastructure. 

9 .Geo Replication • Cross City • Cross Region/State Shanghai Toronto Shanghai Shenzhen (China) (Canada) broker bookie Benchmark Hardware: cpu 24cores memory 48G network 10G 2 replica up to 560k QPS Compare to Kafka： 3 replica up to 360k QPS high throughput（async）: up to 1million QPS 3 geo replica up to 280k QPS 2 replica(sync)： only 60k QPS 

10 .Practice of DB replication across cities n Base on database binlog, hash method n Batch compression n Conflict detection，ensure correctness Result： Data Replication 10w/s，Latency ~30ms, from Shenzhen to shanghai 

11 .Performance • Question: with one topic, can improve the consume performance? • Answer: Read-only Broker Pulsar Microservice A Broker (leader) B broker C Broker Could (read only?) Consumer? … 

12 .Real-time Interaction • Question: use MQ as a RPC? • Answer: add Request-Reply mode Product Broker Leader Consume Msg Msg Session Caller A Callee Rsp(+data) Session Rsp(+data) B Read-only Product(data) transmit ACK Msg Callee ACK 

13 .Batch-Process • Question: how to modify data when data errors happened? • Answer: Offset Shadow + Priority Queue Data eror, How to modify? product consume Offset Shadow Correct Data reproduct Priority Queue Offset Shadow, prevent incorrect data from being consumed Priority Queue, prevent revised data from being consumed too late 

14 .Protocol Support Question: how to support various protocol? Answer: add proxy layer broker proxy Native queue Restful topic Node,PHP,etc KOP channel AOP(AMQP partition ) session Fragment ZK Entry bookie 

15 .