Kafka quick start

1 .Kafka quick start Weida Cui

2 .

3 .Terminology Producer Topic Partition Offset

4 .Consumer Commit Poll Subscribe Group

5 .Cluster Broker Producer Consumer

6 .Mapping partitions and consumers https://www.safaribooksonline.com/library/view/kafka-the-definitive/9781491936153/ch04.html 1 Partition can be consumed by 1 Consumer per Group only

7 .Install & start kafka in Windows Download Kafka from https://kafka.apache.org/downloads Extract package use 7Zip or run cmdline in gitbash Start Zookeeper Start Kafka tar -xzf kafka_2.12-1.1.0.tgz cd C:\Code\kafka_2.12-1.1.0 bin\windows\zookeeper-server-start.bat config\ zookeeper.properties cd C:\Code\kafka_2.12-1.1.0 bin\windows\kafka-server-start.bat config\ server.properties

8 .Server settings other settings in early version: port=9092 host.name=kafka01 log.cleaner.enable =false delete.topic.enable =true

9 .Topic create a topic list topics describe a topic alter a topic delete a topic bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test bin\windows\kafka-topics.bat --list --zookeeper localhost:2181 bin\windows\kafka-topics.bat --describe --zookeeper localhost:2181 --topic test bin\windows\kafka-topics.bat --alter --zookeeper localhost:2181 --topic test --partitions 4 bin\windows\kafka-topics.bat --delete --zookeeper localhost:2181 --topic test

10 .Produce & consume with command line produce consume bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test bin\windows\kafka-console-consumer.bat --zookeeper localhost:2181 --topic test --from-beginning

11 .Produce & consume settings

12 .Confluent.Kafka Download confluent- kafka - dotnet from https://github.com/confluentinc/confluent-kafka-dotnet Then we have source code and example. Also we can import it as nuget package See some examples >>

13 .consumer  2 subscribe methods: consumer.Subscribe (topics); consumer.Assign

14 .3 modes for Clients consuming message Mode description development At-most-once Get message -> Commit offset -> C onsume message. Risk: if consuming step break down, this message would be lost. enable.auto.commit = true; Consumer.Subscribe (topics); consumer.Poll (); At-least-once Get message -> C onsume message -> Commit offset. Risk: if commit offset step break down, this message would be consumed twice. enable.auto.commit = false; consumer.Subscribe (topics); consumer.Consume (); consumer.CommitAsync (); Exactly-once Banding offset and message consuming result -> ensure they are updated together. Risk: Storing offsets in Consumer. Storing offsets in Consumer http://kafka.apache.org/0110/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html

15 .Kafka Cluster Make another server.properties // e.g : server1.properties Below properties must be set: broker.id=1 //should be different id listeners=PLAINTEXT://10.78.147.194:9093 //should use IP address log.dirs =/ tmp /kafka-logs1 //folder can be wrote Start another kafka server bin\windows\kafka-server-start.bat config\server1.properties

16 .References https://kafka.apache.org/intro Kafka: The Definitive Guide Kafka：The Definitive Guide 中文整理版