1.Kafka quick start Weida Cui

2.

3.Terminology Producer Topic Partition Offset

4.Consumer Commit Poll Subscribe Group

5.Cluster Broker Producer Consumer

6.Mapping partitions and consumers https://www.safaribooksonline.com/library/view/kafka-the-definitive/9781491936153/ch04.html 1 Partition can be consumed by 1 Consumer per Group only

7.Install & start kafka in Windows Download Kafka from https://kafka.apache.org/downloads Extract package use 7Zip or run cmdline in gitbash Start Zookeeper Start Kafka tar -xzf kafka_2.12-1.1.0.tgz cd C:\Code\kafka_2.12-1.1.0 bin\windows\zookeeper-server-start.bat config\ zookeeper.properties cd C:\Code\kafka_2.12-1.1.0 bin\windows\kafka-server-start.bat config\ server.properties

8.Server settings other settings in early version: port=9092 host.name=kafka01 log.cleaner.enable =false delete.topic.enable =true

9.Topic create a topic list topics describe a topic alter a topic delete a topic bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test bin\windows\kafka-topics.bat --list --zookeeper localhost:2181 bin\windows\kafka-topics.bat --describe --zookeeper localhost:2181 --topic test bin\windows\kafka-topics.bat --alter --zookeeper localhost:2181 --topic test --partitions 4 bin\windows\kafka-topics.bat --delete --zookeeper localhost:2181 --topic test

10.Produce & consume with command line produce consume bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test bin\windows\kafka-console-consumer.bat --zookeeper localhost:2181 --topic test --from-beginning

11.Produce & consume settings

12.Confluent.Kafka Download confluent- kafka - dotnet from https://github.com/confluentinc/confluent-kafka-dotnet Then we have source code and example. Also we can import it as nuget package See some examples >>

13.consumer  2 subscribe methods: consumer.Subscribe (topics); consumer.Assign

14.3 modes for Clients consuming message Mode description development At-most-once Get message -> Commit offset -> C onsume message. Risk: if consuming step break down, this message would be lost. enable.auto.commit = true; Consumer.Subscribe (topics); consumer.Poll (); At-least-once Get message -> C onsume message -> Commit offset. Risk: if commit offset step break down, this message would be consumed twice. enable.auto.commit = false; consumer.Subscribe (topics); consumer.Consume (); consumer.CommitAsync (); Exactly-once Banding offset and message consuming result -> ensure they are updated together. Risk: Storing offsets in Consumer. Storing offsets in Consumer http://kafka.apache.org/0110/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html

15.Kafka Cluster Make another server.properties // e.g : server1.properties Below properties must be set: broker.id=1 //should be different id listeners=PLAINTEXT://10.78.147.194:9093 //should use IP address log.dirs =/ tmp /kafka-logs1 //folder can be wrote Start another kafka server bin\windows\kafka-server-start.bat config\server1.properties

16.References https://kafka.apache.org/intro Kafka: The Definitive Guide Kafka：The Definitive Guide 中文整理版