Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.conduktor.io/llms.txt

Use this file to discover all available pages before exploring further.

Learn how Kafka consumer groups work through hands-on CLI practice in 15 minutes Consumer groups enable parallel message processing by distributing partitions among group members. This tutorial demonstrates consumer group behavior using the kafka-console-consumer CLI. What you’ll learn:
  • How to create multiple consumers in the same group
  • How partitions are distributed among consumers
  • What happens when consumers join or leave a group
  • How offsets are committed and resumed
Use CLI commands with appropriate extensions for your platform: for Windows - kafka-console-consumer.bat, for Mac and Linux - kafka-console-consumer.sh.
In addition, we will provide an optional consumer group parameter with --group flag.

​
How to create consumers in a Kafka consumer group?

To start consumers in a consumer group, do the following:
  1. Create a topic with at least 2 partitions and send data to it
  2. Create a first kafka-console-consumer and assign a group name with --group
  3. Open a new terminal / shell window
  4. Create a second kafka-console-consumer and use the same --group argument
  5. Send data to the topic and you will see consumers sharing the reads
If you need a refresh on how consumers in a consumer group work, have a read here.

​
Create consumer group example

You cannot have more consumers in a group than partitions in your Kafka topic, and therefore we first need to create a Kafka topic with a few partitions (in the example 3). 
kafka-topics --bootstrap-server localhost:9092 --topic first_topic --create --partitions 3 --replication-factor 1
Then launch a consumer in a consumer group, named my-first-application 
kafka-console-consumer --bootstrap-server localhost:9092 --topic first_topic --group my-first-application
Open a new terminal/shell window and launch a second consumer in the same consumer group my-first-application (note we’re using the exact same command) 
kafka-console-consumer --bootstrap-server localhost:9092 --topic first_topic --group my-first-application
Open a new terminal/shell window and launch a third consumer in the same consumer group my-first-application 
kafka-console-consumer --bootstrap-server localhost:9092 --topic first_topic --group my-first-application
Each consumer in the consumer group my-first-application will get assigned a partition. Produce a few string messages in the topic. 
$ kafka-console-producer --bootstrap-server localhost:9092 --topic first_topic
>first message
>second message
>third message
>fourth message
Each consumer will show only the messages produced on the partition that are assigned to it. If you stop a consumer, messages automatically get sent to the remaining consumers because consumers in a consumer group automatically perform a consumer rebalance.

​
Stop all consumers

And keep on producing to the topic 
>eigth message
>ninth message
>tenth message
Upon restart of a consumer in the group, the consumer will read from the latest committed offsets and read only the messages you’ve just produced 
$ kafka-console-consumer --bootstrap-server localhost:9092 --topic first_topic --group my-first-application

eigth message
ninth message
tenth message
You have seen how consumers work in consumer groups!

​
Gotchas

  • If you consume in a consumer groups using the --group command, then if you try using the --from-beginning option afterwards with the same group, it will be ignored. Instead, you need to reset your consumer groups as shown here.
  • If you don’t specify a —group option, the consumer group of the consumer will be a random consumer group such as console-consumer-11984
  • If you see one consumer getting all the messages, that probably means that your topic was only created with 1 partition, which you can verify with the kafka-topics --describe command

​
Consumer group behavior summary

ScenarioResult
3 consumers, 3 partitionsEach consumer gets 1 partition
2 consumers, 3 partitionsOne consumer gets 2 partitions
4 consumers, 3 partitionsOne consumer sits idle
Consumer leaves groupPartitions rebalanced to remaining consumers
Consumer joins groupRebalance distributes partitions
See it in practice with ConduktorConduktor Console displays consumer group membership, partition assignments, and lag metrics in real-time. Monitor rebalances and track consumer health visually.

​
Next steps