How to send data into Kafka?
The Kafka console producer CLI, kafka-console-producer is used to read data from standard input and publish it to Kafka. Make sure you have started Kafka beforehand.Use CLI commands with appropriate extensions for your platform: for Windows -
kafka-console-producer.bat, for Mac and Linux - kafka-console-producer.sh.How to produce a message into a Kafka topic using the CLI?
To produce to a Kafka topic, we need to provide the mandatory parameters:- Find your Kafka hostname and port e.g.,
localhost:9092 - If Kafka v2.5+, use the
--bootstrap-serveroption - If older version of Kafka, use the
--broker-listoption - Provide the mandatory parameters: topic name
- Use the
kafka-console-producer.shCLI as outlined below
Example
Make sure the topic first_topic is already created with 3 partitions and a replication factor of 1:Ctrl+C.
Command output
> sign. Then any line of text you write afterwards will be sent to the Kafka topic (when pressing Enter)
Gotchas
Here are the common mistakes and caveats with thekafka-console-producer.sh command:
-
Messages are sent with the
nullkey by default (see below for more options) - If the topic does not exist, it can be auto-created by Kafka:
config/server.properties file), with the following defaults:
Extra important options you can set (advanced)
--compression-codec
To enable message compression, default gzip, possible values 'none', 'gzip', 'snappy', 'lz4', or 'zstd'
--producer-property
To pass in any producer property, such as the acks=all setting
--request-required-acks
An alternative to set the acks setting directly
How to produce messages from a file with the Kafka console producer CLI?
Example filetopic-input.txt (make sure each message is on a new line)
How to produce messages with key in the Kafka console producer CLI?
By default messages sent to a Kafka topic will result in messages withnull keys.
We must use the properties parse.key and key.separator to send the key alongside messages.
In this example, the separator between the key and the value is: :