A consumer reading from a Kafka partition may choose when to commit offsets. That strategy impacts the behaviors if messages are skipped or read twice upon a consumer restart.

​
At most once delivery

In this case, offsets are committed as soon as a message batch is received after calling poll(). If the subsequent processing fails, the message will be lost. It will not be read again as the offsets of those messages have been committed already. Kafka Consumer Delivery Semantics At Most Once

​
At least once delivery (usually preferred)

In at-least-once delivery, every event from the source system will reach its destination, but sometimes retries will cause duplicates. Here, offsets are committed after the message is processed.
Idempotent ProcessingMake sure your processing is idempotent (i.e. processing again the messages won’t impact your systems)
Kafka Consumer Delivery Semantics At Least Once

​
Exactly once delivery

Some applications require exactly-once semantics. Each message is delivered exactly once. This may be achieved in certain situations if Kafka and the consumer application cooperate:
  • Achievable for Kafka topic to Kafka topic workflows using the transactions API
  • For Kafka topic to External System workflows, use an idempotent consumer

​
Summary

  • At most once: Offsets committed immediately on receipt. Message may be lost.
  • At least once: Offsets committed after processing. Potential for duplicate processing.
  • Exactly once: Achievable for Kafka => Kafka workflows using Kafka Streams API.
Bottom LineFor most applications, use ‘At Least Once’ processing and ensure transformations are idempotent.

​
Automatic offset committing strategy

By default, consumers are configured with enable.auto.commit=true which means that offsets will be committed automatically on a schedule. This provides at-least-once delivery semantics.

​
Manual offset committing strategy

You can also choose to control when offsets are committed by setting enable.auto.commit=false and using the commitSync() or commitAsync() methods to manually commit offsets. This gives you full control over when messages are considered “processed”.