How to send data into Kafka?
The Kafka console producer CLI, kafka-console-producer is used to read data from standard input and publish it to Kafka. Make sure you have started Kafka beforehand.Use CLI commands with appropriate extensions for your platform: for Windows -
kafka-console-producer.bat
, for Mac and Linux - kafka-console-producer.sh
.How to produce a message into a Kafka topic using the CLI?
To produce to a Kafka topic, we need to provide the mandatory parameters:- Find your Kafka hostname and port e.g.,
localhost:9092
- If Kafka v2.5+, use the
--bootstrap-server
option - If older version of Kafka, use the
--broker-list
option - Provide the mandatory parameters: topic name
- Use the
kafka-console-producer.sh
CLI as outlined below
Example
Make sure the topic first_topic is already created with 3 partitions and a replication factor of 1:Ctrl+C
.
Command output
>
sign. Then any line of text you write afterwards will be sent to the Kafka topic (when pressing Enter
)
Gotchas
Here are the common mistakes and caveats with thekafka-console-producer.sh
command:
-
Messages are sent with the
null
key by default (see below for more options) - If the topic does not exist, it can be auto-created by Kafka:
config/server.properties
file), with the following defaults:
Extra important options you can set (advanced)
--compression-codec
To enable message compression, default gzip
, possible values 'none'
, 'gzip', 'snappy', 'lz4', or 'zstd'
--producer-property
To pass in any producer property, such as the acks=all
setting
--request-required-acks
An alternative to set the acks
setting directly
How to produce messages from a file with the Kafka console producer CLI?
Example filetopic-input.txt
(make sure each message is on a new line)
How to produce messages with key in the Kafka console producer CLI?
By default messages sent to a Kafka topic will result in messages withnull
keys.
We must use the properties parse.key
and key.separator
to send the key alongside messages.
In this example, the separator between the key and the value is: :