NSQ-重塑之路

1 .NSQ重塑之路 Youzan消息队列在微服务架构中的实践 李⽂ 有赞技术专家 

2 . Agenda • Original Architecture Overview • Missing Features and the Demand ofYouzan • Redesigned Architecture • Jepsen test in the new NSQ • Compare with others • Usage inYouzan 

3 .What is MQ? What is NSQ? 

4 .Original Arch 

5 . Missing Features • Replication • HA • Auto-Balance • Delivery in Order • Tracing • Consume History Messages 

6 . Redesigned Topic Queue • Use go channel to store data • Not searchable, Not Stable • Redesigned: Use list of segment files 

7 . Topic Write Flow on NSQD Producer Topic Topic Leader Follower 

8 .Write Buffer Committed Queue Data Topic Leader 

9 . Group Commit Write Buffer Waiting Sync Committed Queue Data Data Topic Yes Leader Allow Commit? 

10 . Group Commit Write Buffer Committed Committed Queue Data Queue Data Success Topic Yes Leader Allow Commit? 

11 .Improve Write Performance 

12 .Group Commit In Go 

13 .

14 . Redesigned Consume Channel • Copy all data from topic channel to consume channel • Waste disk space, lots of data copy • Redesigned: Use cursor only for consume 

15 . Consume Channel vEach Channel Hold the Endless Disk Queue Offset cursor Info vAll clients share same Channel A Channel B offset cursor info in the same channel 

16 . Consume in History • Move consume cursor Endless Disk Queue • Support timestamp, disk queue offset 

17 .Message Sub State Machine 

18 .Dispatcher in Go 

19 .Optimize channel timeout in Go 

20 .Redesigned Replication & HA 

21 .Redesigned Architecture 

22 . Architecture Detail • Etcd: meta data service • NSQLookup: lookup and placement service • NSQD: topic data storage service 

23 .Meta in etcd 

24 . Replication and HA • Green is ISR (In synced replicas) Topic Etcd • Yellow: Catching up Leader node • lookup will select the Try Lock Leader most newest ISR as new leader Topic Topic Topic Follower Follower Follower 

25 . Replication and HA • Green is ISR (In synced replicas) Topic Etcd • Yellow: Catching up Follower node • lookup will select the Try Lock Leader most newest ISR as Success new leader Topic Topic Topic Follower Follower Leader 

26 . Replication and HA • Green is ISR (In synced replicas) Topic Etcd • Yellow: Catching up Follower node • lookup will select the Try Lock Leader most newest ISR as Success new leader Topic Topic Topic Follower Follower Leader 

27 . Dynamic ISR • Avoid slow write due to failed replica • Add/Remove replica for balance 

28 .

29 .