Kafka consumers poll the Kafka broker to receive batches of data. Once the consumer is subscribed to Kafka topics, the poll loop handles all details of coordination, partition rebalances, heartbeats, and data fetching, leaving the developer with a clean API that simply returns available data from assigned partitions.
Internal code optimizationIf the consumer successfully fetched some data from Kafka, it will start sending the next fetch requests ahead of time, so that while processing the current batch, there will be less waiting on the next .poll() call.
Polling allows consumers to control:
- From where in the log they want to consume
- How fast they want to consume
- Ability to replay events
Internal poll thread & heartbeat thread
The way consumers maintain membership in a consumer group and ownership of partitions is by sending heartbeats to a Kafka broker designated as the group coordinator.
Kafka consumer heartbeat thread
Heartbeats help to determine consumer liveliness:
- As long as the consumer sends heartbeats at regular intervals, it is assumed to be alive and processing messages
- If the consumer stops sending heartbeats long enough, its session will time out and trigger a rebalance
Key configurations:
heartbeat.interval.ms (default: 3 seconds)session.timeout.ms (Kafka v3.0+: 45 seconds)
This mechanism detects consumer application downtime or network failures.
Kafka consumer poll thread
Consumers poll brokers periodically using the .poll() method.
Key configuration:
max.poll.interval.ms (default: 5 minutes)
This controls the maximum time between calls to poll(). If this interval is exceeded, the consumer is considered failed and triggers a rebalance.
Important consumer settings
Poll behavior settings
max.poll.records (default: 500)
- Maximum number of records returned in a single call to
poll()
- Lower values can improve latency but may reduce throughput
- Higher values improve throughput but may increase processing time per batch
fetch.min.bytes (default: 1)
- Minimum amount of data the server should return for a fetch request
- Setting higher values can improve throughput by reducing request overhead
- May increase latency as consumer waits for more data
fetch.max.wait.ms (default: 500ms)
- Maximum time server will block before answering fetch request if there isn’t sufficient data
- Works together with
fetch.min.bytes
Session and heartbeat settings
session.timeout.ms (default: 45 seconds in Kafka 3.0+)
- Timeout for detecting consumer failures
- Must be between
group.min.session.timeout.ms and group.max.session.timeout.ms
- Lower values enable faster failure detection but may cause false positives
heartbeat.interval.ms (default: 3 seconds)
- Expected time between heartbeats
- Must be lower than
session.timeout.ms
- Typically set to 1/3 of
session.timeout.ms
max.poll.interval.ms (default: 5 minutes)
- Maximum delay between invocations of
poll()
- If exceeded, consumer leaves group and triggers rebalance
- Should be set higher than your maximum expected processing time
For high throughput
max.poll.records=1000
fetch.min.bytes=1048576 # 1MB
fetch.max.wait.ms=1000
For low latency
max.poll.records=100
fetch.min.bytes=1
fetch.max.wait.ms=100
For long processing times
max.poll.interval.ms=600000 # 10 minutes
session.timeout.ms=60000 # 1 minute
heartbeat.interval.ms=20000 # 20 seconds
Best practices
- Tune
max.poll.records based on your processing time per message
- Set
max.poll.interval.ms higher than your worst-case processing time
- Monitor consumer lag to ensure settings are appropriate
- Test rebalance behavior under your expected load conditions
- Consider batch processing patterns when setting poll configurations
Avoid blocking in poll loopNever perform long-running operations in the thread that calls poll(). This can trigger unnecessary rebalances and degrade performance.
