How to consume data from Kafka?
The Kafka console consumer CLI, kafka-console-consumer is used to read data from Kafka and output it to standard output. Make sure you have started Kafka beforehand.Use CLI commands with appropriate extensions for your platform: for Windows -
kafka-console-consumer.bat
, for Mac and Linux - kafka-console-consumer.sh
.How to consume data in a Kafka topic using the CLI?
To consume data from a Kafka topic, we need to provide the mandatory parameters:- Find your Kafka hostname and port e.g.,
localhost:9092
- Provide the mandatory parameters: topic name
- If you need to read historical data, use the
--from-beginning
option - Otherwise, you will only be reading future data
- Use the
kafka-console-consumer.sh
CLI
Important recommendation: do NOT use the
--zookeeper
optionSince Kafka v0.10, the consumer is leveraging a Kafka connection string, not Zookeeper. This is due to how consumer offsets are stored. Any tutorial you see using the --zookeeper
option should be considered outdated.Example
Make sure you have followed the Kafka producer CLI tutorial to create and send data to a Kafka topic first. Consuming only the future messages of a Kafka topic--from-beginning
option
Ctrl+C
.
Command output
String
).
By default, the Kafka console consumer does not show the key, or any partition information.
If you do not see any output but you know your Kafka topic has data in it, don’t forget to use the --from-beginning
option.
Important note on message orderingThe order of messages is not total, it is per partition. As a topic may be created with more than one partition, the order is only guaranteed at the partition level. If you try with only one partition, you will see total ordering.
Gotchas
Here are the common mistakes and caveats with thekafka-console-consumer.sh
command:
Messages by default will not display the key or metadata information (see below for how to do it).
- When you start a
kafka-console-consumer
, unless specifying the--from-beginning
option, only future messages will be displayed and read - If the topic does not exist, the console consumer will automatically create it with default settings
- You can consume multiple topics at a time with a comma-delimited list or a pattern
- If a consumer group id is not specified, the
kafka-console-consumer
generates a random consumer group - If messages do not appear in order, remember that the order is at the partition level, not at the topic level
Extra important options you can set (advanced)
--from-beginning
We won’t repeat this one enough. To read all historical messages
--formatter
To display messages in a particular format (example below to display keys)
--consumer-property
To pass in any consumer property, such as the allow.auto.create.topics
setting
--group
By default a random consumer group ID is chosen, but you can override it with this option. See the demo in the Kafka consumers in group CLI tutorial.
--max-messages
Number of messages to consume before exiting
--partition
If you want to only consume from a specific partition.
How to consume a Kafka topic and show both the key and value using the Kafka console consumer CLI?
By default, the console consumer will show only the value of the Kafka record. Using this command you can show both the key and value. Using theformatter
kafka.tools.DefaultMessageFormatter
and using the properties print.timestamp=true
print.key=true
print.value=true
:
print.partition
print.offset
print.headers
key.separator
line.separator
headers.separator