NSQ-重塑之路
展开查看详情
1.NSQ重塑之路 Youzan消息队列在微服务架构中的实践 李⽂ 有赞技术专家
2. Agenda • Original Architecture Overview • Missing Features and the Demand ofYouzan • Redesigned Architecture • Jepsen test in the new NSQ • Compare with others • Usage inYouzan
3.What is MQ? What is NSQ?
4.Original Arch
5. Missing Features • Replication • HA • Auto-Balance • Delivery in Order • Tracing • Consume History Messages
6. Redesigned Topic Queue • Use go channel to store data • Not searchable, Not Stable • Redesigned: Use list of segment files
7. Topic Write Flow on NSQD Producer Topic Topic Leader Follower
8.Write Buffer Committed Queue Data Topic Leader
9. Group Commit Write Buffer Waiting Sync Committed Queue Data Data Topic Yes Leader Allow Commit?
10. Group Commit Write Buffer Committed Committed Queue Data Queue Data Success Topic Yes Leader Allow Commit?
11.Improve Write Performance
12.Group Commit In Go
13.
14. Redesigned Consume Channel • Copy all data from topic channel to consume channel • Waste disk space, lots of data copy • Redesigned: Use cursor only for consume
15. Consume Channel vEach Channel Hold the Endless Disk Queue Offset cursor Info vAll clients share same Channel A Channel B offset cursor info in the same channel
16. Consume in History • Move consume cursor Endless Disk Queue • Support timestamp, disk queue offset
17.Message Sub State Machine
18.Dispatcher in Go
19.Optimize channel timeout in Go
20.Redesigned Replication & HA
21.Redesigned Architecture
22. Architecture Detail • Etcd: meta data service • NSQLookup: lookup and placement service • NSQD: topic data storage service
23.Meta in etcd
24. Replication and HA • Green is ISR (In synced replicas) Topic Etcd • Yellow: Catching up Leader node • lookup will select the Try Lock Leader most newest ISR as new leader Topic Topic Topic Follower Follower Follower
25. Replication and HA • Green is ISR (In synced replicas) Topic Etcd • Yellow: Catching up Follower node • lookup will select the Try Lock Leader most newest ISR as Success new leader Topic Topic Topic Follower Follower Leader
26. Replication and HA • Green is ISR (In synced replicas) Topic Etcd • Yellow: Catching up Follower node • lookup will select the Try Lock Leader most newest ISR as Success new leader Topic Topic Topic Follower Follower Leader
27. Dynamic ISR • Avoid slow write due to failed replica • Add/Remove replica for balance
28.
29.