Kafka consumers poll the Kafka broker to receive batches of data. Once the consumer is subscribed to Kafka topics, the poll loop handles all details of coordination, partition rebalances, heartbeats, and data fetching, leaving the developer with a clean API that simply returns available data from assigned partitions.
Internal code optimizationIf the consumer successfully fetched some data from Kafka, it will start sending the next fetch requests ahead of time, so that while processing the current batch, there will be less waiting on the next .poll() call.
Kafka Consumer Poll Behavior Polling allows consumers to control:
  • From where in the log they want to consume
  • How fast they want to consume
  • Ability to replay events

​
Internal poll thread & heartbeat thread

The way consumers maintain membership in a consumer group and ownership of partitions is by sending heartbeats to a Kafka broker designated as the group coordinator. Kafka Consumer Internal Threads

​
Kafka consumer heartbeat thread

Heartbeats help to determine consumer liveliness:
  • As long as the consumer sends heartbeats at regular intervals, it is assumed to be alive and processing messages
  • If the consumer stops sending heartbeats long enough, its session will time out and trigger a rebalance
Key configurations:
  • heartbeat.interval.ms (default: 3 seconds)
  • session.timeout.ms (Kafka v3.0+: 45 seconds)
This mechanism detects consumer application downtime or network failures.

​
Kafka consumer poll thread

Consumers poll brokers periodically using the .poll() method. Key configuration:
  • max.poll.interval.ms (default: 5 minutes)
This controls the maximum time between calls to poll(). If this interval is exceeded, the consumer is considered failed and triggers a rebalance.

​
Important consumer settings

​
Poll behavior settings

max.poll.records (default: 500)
  • Maximum number of records returned in a single call to poll()
  • Lower values can improve latency but may reduce throughput
  • Higher values improve throughput but may increase processing time per batch
fetch.min.bytes (default: 1)
  • Minimum amount of data the server should return for a fetch request
  • Setting higher values can improve throughput by reducing request overhead
  • May increase latency as consumer waits for more data
fetch.max.wait.ms (default: 500ms)
  • Maximum time server will block before answering fetch request if there isn’t sufficient data
  • Works together with fetch.min.bytes

​
Session and heartbeat settings

session.timeout.ms (default: 45 seconds in Kafka 3.0+)
  • Timeout for detecting consumer failures
  • Must be between group.min.session.timeout.ms and group.max.session.timeout.ms
  • Lower values enable faster failure detection but may cause false positives
heartbeat.interval.ms (default: 3 seconds)
  • Expected time between heartbeats
  • Must be lower than session.timeout.ms
  • Typically set to 1/3 of session.timeout.ms
max.poll.interval.ms (default: 5 minutes)
  • Maximum delay between invocations of poll()
  • If exceeded, consumer leaves group and triggers rebalance
  • Should be set higher than your maximum expected processing time

​
Performance tuning guidelines

​
For high throughput

max.poll.records=1000
fetch.min.bytes=1048576    # 1MB
fetch.max.wait.ms=1000

​
For low latency

max.poll.records=100
fetch.min.bytes=1
fetch.max.wait.ms=100

​
For long processing times

max.poll.interval.ms=600000  # 10 minutes
session.timeout.ms=60000     # 1 minute
heartbeat.interval.ms=20000  # 20 seconds

​
Best practices

  1. Tune max.poll.records based on your processing time per message
  2. Set max.poll.interval.ms higher than your worst-case processing time
  3. Monitor consumer lag to ensure settings are appropriate
  4. Test rebalance behavior under your expected load conditions
  5. Consider batch processing patterns when setting poll configurations
Avoid blocking in poll loopNever perform long-running operations in the thread that calls poll(). This can trigger unnecessary rebalances and degrade performance.