A Dynamic High-Speed Inter-Cluster Kafka Replicator
Propane replicates around 150 billion events per day cross colo across its clusters with an average latency of less than 2 seconds.
1585026
展开查看详情
1.Kafka Kloner
Michael Natkovich
2.Kafka Kloner
A Dynamic High-Speed Inter-Cluster Kafka Replicator
Propane replicates around 150 billion events per day cross colo across its
clusters with an average latency of less than 2 seconds.
3.Motivation
■ Demand for Yahoo Streaming in multiple data centers
■ Kafka MirrorMaker operability issues
Finally, we borrowed the MirrorMaker approach and implemented replication
using Storm.
Yahoo Confidential & Proprietary
4.MirrorMaker vs Kloner
MirrorMaker Kloner
Performance High High
Operability Low High
Fault Tolerance Low High
Topic Management Low High
Monitoring Low High
Yahoo Confidential & Proprietary
5.Overview
Yahoo Confidential & Proprietary
6.Features
Design
■ Storm based approach
■ One topology per topic being cloned
■ One worker per topic partition
■ One kafka consumer + producer per worker
Core Logic
■ Same as MirrorMaker
● Read; Commit Offsets; Send Message;
Yahoo Confidential & Proprietary
7.Features
Manager
■ Runs based on a DSL input with list of topics and their destinations
■ Polls Storm for list of topics already being replicated and number of workers
■ Polls Kafka clusters for topic metadata such as partitions per topic
Then,
1. Starts replication by starting new topologies
2. Scales replication by rebalancing topologies (Restarts with updated worker count)
3. Stops replication by stopping topologies
■ The Manager can be a cron or simply invoked after DSL update
Yahoo Confidential & Proprietary
9.Numbers
Clusters 2
Topics 15
Msgs/sec 1 million
Bytes/sec 1.5 GB
Replication Lag < 2 seconds
Yahoo Confidential & Proprietary
10.Limitations
■ No exact replication
● Offsets differ between clusters
■ No replay or spooling
● No message level failure handling
● But we are tracking number of failed events
● There is rate limiting
Yahoo Confidential & Proprietary
