Kafka quick start

Kafka quick start
展开查看详情

1.Kafka quick start Weida Cui

2.

3.Terminology Producer Topic Partition Offset

4.Consumer Commit Poll Subscribe Group

5.Cluster Broker Producer Consumer

6.Mapping partitions and consumers https://www.safaribooksonline.com/library/view/kafka-the-definitive/9781491936153/ch04.html 1 Partition can be consumed by 1 Consumer per Group only

7.Install & start kafka in Windows Download Kafka from https://kafka.apache.org/downloads Extract package use 7Zip or run cmdline in gitbash Start Zookeeper Start Kafka tar -xzf kafka_2.12-1.1.0.tgz cd C:\Code\kafka_2.12-1.1.0 bin\windows\zookeeper-server-start.bat config\ zookeeper.properties cd C:\Code\kafka_2.12-1.1.0 bin\windows\kafka-server-start.bat config\ server.properties

8.Server settings other settings in early version: port=9092 host.name=kafka01 log.cleaner.enable =false delete.topic.enable =true

9.Topic create a topic list topics describe a topic alter a topic delete a topic bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test bin\windows\kafka-topics.bat --list --zookeeper localhost:2181 bin\windows\kafka-topics.bat --describe --zookeeper localhost:2181 --topic test bin\windows\kafka-topics.bat --alter --zookeeper localhost:2181 --topic test --partitions 4 bin\windows\kafka-topics.bat --delete --zookeeper localhost:2181 --topic test

10.Produce & consume with command line produce consume bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test bin\windows\kafka-console-consumer.bat --zookeeper localhost:2181 --topic test --from-beginning

11.Produce & consume settings

12.Confluent.Kafka Download confluent- kafka - dotnet from https://github.com/confluentinc/confluent-kafka-dotnet Then we have source code and example. Also we can import it as nuget package See some examples >>

13.consumer  2 subscribe methods: consumer.Subscribe (topics); consumer.Assign

14.3 modes for Clients consuming message Mode description development At-most-once Get message -> Commit offset -> C onsume message. Risk: if consuming step break down, this message would be lost. enable.auto.commit = true; Consumer.Subscribe (topics); consumer.Poll (); At-least-once Get message -> C onsume message -> Commit offset. Risk: if commit offset step break down, this message would be consumed twice. enable.auto.commit = false; consumer.Subscribe (topics); consumer.Consume (); consumer.CommitAsync (); Exactly-once Banding offset and message consuming result -> ensure they are updated together. Risk: Storing offsets in Consumer. Storing offsets in Consumer http://kafka.apache.org/0110/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html

15.Kafka Cluster Make another server.properties // e.g : server1.properties Below properties must be set: broker.id=1 //should be different id listeners=PLAINTEXT://10.78.147.194:9093 //should use IP address log.dirs =/ tmp /kafka-logs1 //folder can be wrote Start another kafka server bin\windows\kafka-server-start.bat config\server1.properties

16.References https://kafka.apache.org/intro Kafka: The Definitive Guide Kafka:The Definitive Guide 中文整理版