1.NSQ重塑之路 Youzan消息队列在微服务架构中的实践 李⽂ 有赞技术专家 

2. Agenda • Original Architecture Overview • Missing Features and the Demand ofYouzan • Redesigned Architecture • Jepsen test in the new NSQ • Compare with others • Usage inYouzan 

3.What is MQ? What is NSQ? 

4.Original Arch 

5. Missing Features • Replication • HA • Auto-Balance • Delivery in Order • Tracing • Consume History Messages 

6. Redesigned Topic Queue • Use go channel to store data • Not searchable, Not Stable • Redesigned: Use list of segment files 

7. Topic Write Flow on NSQD Producer Topic Topic Leader Follower 

8.Write Buffer Committed Queue Data Topic Leader 

9. Group Commit Write Buffer Waiting Sync Committed Queue Data Data Topic Yes Leader Allow Commit? 

10. Group Commit Write Buffer Committed Committed Queue Data Queue Data Success Topic Yes Leader Allow Commit? 

11.Improve Write Performance 

12.Group Commit In Go 

13.

14. Redesigned Consume Channel • Copy all data from topic channel to consume channel • Waste disk space, lots of data copy • Redesigned: Use cursor only for consume 

15. Consume Channel vEach Channel Hold the Endless Disk Queue Offset cursor Info vAll clients share same Channel A Channel B offset cursor info in the same channel 

16. Consume in History • Move consume cursor Endless Disk Queue • Support timestamp, disk queue offset 

17.Message Sub State Machine 

18.Dispatcher in Go 

19.Optimize channel timeout in Go 

20.Redesigned Replication & HA 

21.Redesigned Architecture 

22. Architecture Detail • Etcd: meta data service • NSQLookup: lookup and placement service • NSQD: topic data storage service 

23.Meta in etcd 

24. Replication and HA • Green is ISR (In synced replicas) Topic Etcd • Yellow: Catching up Leader node • lookup will select the Try Lock Leader most newest ISR as new leader Topic Topic Topic Follower Follower Follower 

25. Replication and HA • Green is ISR (In synced replicas) Topic Etcd • Yellow: Catching up Follower node • lookup will select the Try Lock Leader most newest ISR as Success new leader Topic Topic Topic Follower Follower Leader 

26. Replication and HA • Green is ISR (In synced replicas) Topic Etcd • Yellow: Catching up Follower node • lookup will select the Try Lock Leader most newest ISR as Success new leader Topic Topic Topic Follower Follower Leader 

27. Dynamic ISR • Avoid slow write due to failed replica • Add/Remove replica for balance 

28.

29.