NSQ-重塑之路

• OriginalArchitecture Overview • Missing Features and the Demand ofYouzan • RedesignedArchitecture • Jepsen test in the new NSQ • Compare with others • Usage inYouzan
展开查看详情

1.NSQ重塑之路 Youzan消息队列在微服务架构中的实践 李⽂ 有赞技术专家

2. Agenda • Original Architecture Overview • Missing Features and the Demand ofYouzan • Redesigned Architecture • Jepsen test in the new NSQ • Compare with others • Usage inYouzan

3.What is MQ? What is NSQ?

4.Original Arch

5. Missing Features • Replication • HA • Auto-Balance • Delivery in Order • Tracing • Consume History Messages

6. Redesigned Topic Queue • Use go channel to store data • Not searchable, Not Stable • Redesigned: Use list of segment files

7. Topic Write Flow on NSQD Producer Topic Topic Leader Follower

8.Write Buffer Committed Queue Data Topic Leader

9. Group Commit Write Buffer Waiting Sync Committed Queue Data Data Topic Yes Leader Allow Commit?

10. Group Commit Write Buffer Committed Committed Queue Data Queue Data Success Topic Yes Leader Allow Commit?

11.Improve Write Performance

12.Group Commit In Go

13.

14. Redesigned Consume Channel • Copy all data from topic channel to consume channel • Waste disk space, lots of data copy • Redesigned: Use cursor only for consume

15. Consume Channel vEach Channel Hold the Endless Disk Queue Offset cursor Info vAll clients share same Channel A Channel B offset cursor info in the same channel

16. Consume in History • Move consume cursor Endless Disk Queue • Support timestamp, disk queue offset

17.Message Sub State Machine

18.Dispatcher in Go

19.Optimize channel timeout in Go

20.Redesigned Replication & HA

21.Redesigned Architecture

22. Architecture Detail • Etcd: meta data service • NSQLookup: lookup and placement service • NSQD: topic data storage service

23.Meta in etcd

24. Replication and HA • Green is ISR (In synced replicas) Topic Etcd • Yellow: Catching up Leader node • lookup will select the Try Lock Leader most newest ISR as new leader Topic Topic Topic Follower Follower Follower

25. Replication and HA • Green is ISR (In synced replicas) Topic Etcd • Yellow: Catching up Follower node • lookup will select the Try Lock Leader most newest ISR as Success new leader Topic Topic Topic Follower Follower Leader

26. Replication and HA • Green is ISR (In synced replicas) Topic Etcd • Yellow: Catching up Follower node • lookup will select the Try Lock Leader most newest ISR as Success new leader Topic Topic Topic Follower Follower Leader

27. Dynamic ISR • Avoid slow write due to failed replica • Add/Remove replica for balance

28.

29.