Install

  1. Download and unzip
    1. https://www.apache.org/dyn/closer.cgi?path=/kafka/2.2.0/kafka_2.12-2.2.0.tgz
    2. unzip and unzip
    3. paste under C:\
  2. Create a folder data under kafka’s root dir.
  3. Create two folders: kafka and zookeeper under data folder.
  4. Change properties:
    1. config\zookeeper.properties: dataDir=C:/kafka_2.12-2.2.0/data/zookeeper
    2. config\server.properties: log.dirs=C:/kafka_2.12-2.2.0/data/kafka

 

Launch Kafka

  1. Add environment PATH: C:\kafka_2.12-2.2.0\bin\windows
  2. Launch Zookeeper FIRST, then start Kafka:
    1. zookeeper-server-start.bat  config\zookeeper.properties
    2. kafka-server-start.bat  config\server.properties

kafka-topics

1. Create

C:\kafka_2.12-2.2.0>kafka-topics --zookeeper 127.0.0.1:2181 --topic first_topic --create --partitions 3 --replication-factor 1
  1. when we see list of command with description, means something is wrong.
  2. when create a topic, we need to specify how many partitions we want.
  3. can not have more replication factor than broker

2. List Details

C:\kafka_2.12-2.2.0>kafka-topics --zookeeper 127.0.0.1:2181 --list
C:\kafka_2.12-2.2.0>kafka-topics --zookeeper 127.0.0.1:2181 --topic first_topic --describe
 

3. Delete (Windows user DO NOT do this)

C:\kafka_2.12-2.2.0>kafka-topics --zookeeper 127.0.0.1:2181 --topic second_topic --delete
 

kafka-console-producer

C:\kafka_2.12-2.2.0>kafka-console-producer --broker-list 127.0.0.1:9092 --topic first_topic --producer-property acks=all
  1. Press Ctrl + C stop message producer.
  2. --producer-property can be set in CLI
  3. --broker-list and --topic are REQUIRED in kafka-console-producer
  4. WARN because new_topic does not exist yet, but kafka will create it (with default property settings), and message created as well. LEADER_NOT_AVAILABLE because the new topic gets created, no leader election yet. 
  5. The default newly created new_topic will have only one partition, because server.properties is the default settings. So we can change default settings.
  6. We do recommend that create a topic first, before a producer use it.

kafka-console-consumer

C:\kafka_2.12-2.2.0>kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic first_topic
  1. --bootstrap-server and --topic are REQUIRED
  2. kafka-console-consumer ONLY  intercepter new messages, after launched.

 

kafka-console-consumer (in group)

# Launch 3 consumer in same group.
C:\kafka_2.12-2.2.0>kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic first_topic --group my-first-group
C:\kafka_2.12-2.2.0>kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic first_topic --group my-first-group
C:\kafka_2.12-2.2.0>kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic first_topic --group my-first-group
  1. Three consumers in SAME group --group my-first-group, same group id. Topic’s message would send to partitions randomly (load balanced). Each consumer in the same group, would subscribe one partition
  2. Read from beginning. --from-beginning would load from current offset. Due to my-second-group is a new group, so ‘–from-beginning’ should start from offset 0. But once all previous messages are read, offset would set to current.

kafka-consumer-groups

–list

C:\kafka_2.12-2.2.0>kafka-consumer-groups --bootstrap-server 127.0.0.1:9092 --list
  1. List all consumer groups.
  2. If a console-consumer created without given a group name, kafka will create a random group for it.
  3. kafka >= 2.1, ‘console-consumer-groups’ will not appear in this command anymore.

–describe

C:\kafka_2.12-2.2.0>kafka-consumer-groups --bootstrap-server 127.0.0.1:9092 --describe --group my-first-group
  1. List out each consumer’s offset and are there any unread (lag) messages or not.

–reset-offsets

C:\kafka_2.12-2.2.0>kafka-consumer-groups --bootstrap-server 127.0.0.1:9092 --group my-first-group --reset-offsets --to-earliest --execute --topic first_topic
  1. --reset-offsets can only modify inactive group.
  2. Once the consumer-group is inactive, then all new offsets in each partition are reset.
  3. There are many pre-set options for ‘–reset-offsets’

Other commonly used CLI

Producer

kafka-console-producer --broker-list 127.0.0.1:9092 --topic first_topic --property parse.key=true --property key.separator=,
> key,value
> another key,another value

Consumer

kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --topic first_topic --from-beginning --property print.key=true --property key.separator=,