Idempotent producers ensure that messages are not duplicated even when retries occur, providing exactly-once semantics for producer operations.

​
What is producer idempotency?

Producer idempotency means that sending the same message multiple times will result in exactly one copy of the message being written to the Kafka topic, even in the presence of failures and retries. Without idempotency:
  • Producer sends message → Network failure → Producer retries → Duplicate message in topic
With idempotency:
  • Producer sends message → Network failure → Producer retries → Kafka deduplicates → Single message in topic

​
Enabling idempotent producers

Idempotent producers are enabled by default in Kafka 3.0+. For older versions, enable explicitly: 
enable.idempotence=true
When idempotency is enabled, Kafka automatically sets these configurations:
  • retries=Integer.MAX_VALUE
  • max.in.flight.requests.per.connection=5
  • acks=all

​
How Kafka achieves idempotency

Kafka uses two key mechanisms to ensure idempotency:

​
1. Producer ID (PID)

Each producer instance gets a unique Producer ID from the broker:
  • Assigned when producer starts up
  • Valid for the lifetime of the producer session
  • Used to track message sequences

​
2. Sequence numbers

Each message gets a sequence number per topic-partition:
  • Starts at 0 for each producer-topic-partition combination
  • Incremented for each message sent
  • Used by broker to detect duplicates
Kafka Idempotent Producer Sequence Numbers

​
How deduplication works

When a broker receives a message, it checks:
  1. Producer ID: Is this from a known producer?
  2. Sequence number: Is this the expected next sequence number?
Scenarios:
  • Expected sequence: Message is written normally
  • Duplicate sequence: Message is discarded, success response sent
  • Out-of-order sequence: OutOfOrderSequenceException thrown
Broker state: Producer 123, Partition 0, Last sequence: 42

Incoming message: Producer 123, Partition 0, Sequence 43 ✅ Accept
Incoming message: Producer 123, Partition 0, Sequence 42 ⚠️ Duplicate (ignore)
Incoming message: Producer 123, Partition 0, Sequence 45 ❌ Out of order (reject)

​
Configuration requirements

For idempotent producers to work correctly, certain configurations must be set:

​
Required settings

enable.idempotence=true
acks=all                                    # Automatically set
retries=Integer.MAX_VALUE                   # Automatically set
max.in.flight.requests.per.connection=5    # Max value for idempotency

​
Compatible settings

# Compression - works with idempotency
compression.type=snappy

# Batching - improves performance with idempotency
batch.size=16384
linger.ms=5

# Timeouts
request.timeout.ms=30000
delivery.timeout.ms=120000

​
Performance implications

​
Benefits

  • Exactly-once semantics: No duplicate messages
  • Simplified error handling: Less complex retry logic needed
  • Better reliability: Automatic handling of transient failures

​
Trade-offs

  • Memory overhead: Broker maintains sequence state
  • Slightly higher latency: Additional sequence checking
  • Connection limits: Max 5 in-flight requests per connection

​
Performance comparison

ConfigurationThroughputLatencyDuplicates
No idempotency, retries=0HighestLowestNone (data loss possible)
No idempotency, retries>0HighMediumPossible
Idempotent producerMedium-HighMediumNone

​
Error handling

​
Retriable errors

Idempotent producers automatically retry these errors:
  • TimeoutException
  • RetriableException
  • Network connectivity issues
  • Broker leadership changes

​
Non-retriable errors

These errors cause immediate failure:
  • OutOfOrderSequenceException
  • UnknownProducerIdException
  • InvalidProducerEpochException
  • Authorization failures

​
Error handling example

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("enable.idempotence", true);

Producer<String, String> producer = new KafkaProducer<>(props);

try {
    ProducerRecord<String, String> record = new ProducerRecord<>("my-topic", "key", "value");
    producer.send(record).get(); // Blocks until success or failure
} catch (Exception e) {
    // Handle non-retriable errors
    System.err.println("Failed to send message: " + e.getMessage());
} finally {
    producer.close();
}

​
Best practices

​
When to use idempotent producers

  • Always recommended for production applications
  • Essential when exactly-once semantics are required
  • Critical for financial, audit, or transactional data

​
Configuration recommendations

# Recommended production configuration
enable.idempotence=true
acks=all
retries=Integer.MAX_VALUE
max.in.flight.requests.per.connection=5
delivery.timeout.ms=120000
compression.type=snappy
batch.size=32768
linger.ms=5

​
Monitoring and observability

Key metrics to monitor:
  • duplicate-send-rate: Should be 0 with proper idempotency
  • retry-rate: Indicates network issues or broker problems
  • error-rate: Monitor for non-retriable errors
  • request-latency: Track impact on performance
Default in Kafka 3.0+Idempotent producers are enabled by default in Kafka 3.0 and later versions. This provides better out-of-the-box reliability without requiring explicit configuration.

​
Limitations

​
Session-based idempotency

  • Idempotency is only guaranteed within a single producer session
  • If producer restarts, it gets a new Producer ID
  • Messages sent across sessions may still be duplicated

​
Topic-partition scope

  • Sequence numbers are per topic-partition combination
  • No deduplication across different partitions
  • No deduplication across different topics

​
Memory requirements

  • Brokers maintain sequence state for each producer-topic-partition
  • State is cleaned up after producer session expires
  • May require additional heap memory for high-throughput scenarios
Producer restartsIf a producer application restarts, it will get a new Producer ID and sequence numbers reset to 0. This means potential duplicates across application restarts, even with idempotency enabled.