Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.conduktor.io/llms.txt

Use this file to discover all available pages before exploring further.

Learn how to configure data retention policies in Kafka in 10 minutes Kafka log retention controls how long messages are stored before being deleted. Understanding retention configuration is essential for managing storage costs, compliance requirements, and consumer catch-up scenarios. What you’ll learn:
  • How Kafka’s delete cleanup policy works
  • How to configure retention by time and size
  • The relationship between broker and topic-level settings
  • Common retention patterns for different use cases

​
Log retention overview

Kafka stores messages for a set amount of time and purges messages older than the retention period. This expiration happens due to the log.cleanup.policy=delete policy (the default for user topics).

​
Retention configuration options

​
Retention by time

The most common configuration for how long Kafka will retain messages is by time.
SettingDefaultDescription
log.retention.hours168 (7 days)Retention time in hours
log.retention.minutes-Retention time in minutes
log.retention.ms-Retention time in milliseconds
If more than one is specified, the smaller unit size will take precedence.
Use milliseconds for consistencyBecause the Kafka CLI command only allows you to set the ms version of this parameter, we recommend using retention.ms across all your configurations.

​
Retention by size

Another way to expire messages is based on the total number of bytes of messages retained.
SettingDefaultDescription
log.retention.bytes-1 (unlimited)Maximum size per partition
The default is -1, meaning that there is no limit and only a time limit is applied. This parameter is useful to set a to positive value if you want to keep the size of a log under a threshold.
Broker-level vs Topic-levelKafka broker-level topic configurations are prefixed by log. and we can remove it to find the equivalent Kafka topic-level configuration. For example, log.retention.ms becomes retention.ms at the topic level.

​
How retention is applied

Diagram illustrating the process Kafka uses to delete old log segments based on the time or size rules configured for the Kafka Topic. Retention by time is performed by examining the last modified time on each log segment file on disk. This is the time that the log segment was closed, and represents the timestamp of the last message in the file. If you have specified a value for both log.retention.bytes and log.retention.hours, messages may be removed when either criteria is met.
Minimum guarantees, not hard limitsThese are minimum guarantees, not hard limits. The active segment does not count toward the byte limit, and the time limit can be much greater than expected if the segment is very big (few messages per day in a 1GB segment).

​
Common retention patterns

​
One week of retention (default)

retention.ms=604800000    # 7 days in milliseconds
retention.bytes=-1        # No size limit

​
Infinite time retention bounded by 500MB

retention.ms=-1           # No time limit
retention.bytes=524288000 # 500MB per partition

​
Short retention for high-volume topics

retention.ms=86400000     # 24 hours
retention.bytes=-1        # No size limit

​
Compliance: 90 days retention

retention.ms=7776000000   # 90 days
retention.bytes=-1        # No size limit

​
Configure retention

To set these configurations using the CLI: 
kafka-configs --bootstrap-server localhost:9092 \
  --alter --entity-type topics --entity-name configured-topic \
  --add-config retention.ms=-1,retention.bytes=524288000

​
Retention decision guide

See it in practice with ConduktorConduktor Console displays topic retention settings and current storage usage. Monitor disk space consumption to validate your retention configuration meets both storage and compliance needs.The Insights dashboard identifies empty, stale, and tiny topics that may be consuming unnecessary storage, helping you make data-driven decisions about retention policies and topic cleanup.

​
Next steps