Skip to main content
Kafka’s incremental cooperative rebalancing and static group membership features reduce the disruption caused by consumer group rebalances, improving overall system stability and performance.

​
Traditional rebalancing problems

​
Eager rebalancing (pre-Kafka 2.4)

  • Stop-the-world: All consumers stop processing during rebalance
  • Complete reassignment: All partitions are revoked and reassigned
  • Processing downtime: No messages processed during rebalance period
  • Cascading rebalances: One consumer failure affects entire group

​
Performance impact

Consumer 1: [P0, P1, P2] → [ ] → [P0, P1]
Consumer 2: [P3, P4, P5] → [ ] → [P2, P3] 
Consumer 3: [P6, P7, P8] → [ ] → [P4, P5, P6, P7, P8]

All consumers stop processing during the transition

​
Incremental cooperative rebalancing

​
How it works (Kafka 2.4+)

  • Minimal disruption: Only affected partitions are reassigned
  • Continued processing: Unaffected partitions continue processing
  • Gradual transition: Rebalance happens in multiple phases
  • Reduced downtime: Significantly shorter processing interruptions

​
Rebalancing phases

Phase 1: Revoke only partitions that need to move
Consumer 1: [P0, P1, P2] → [P0, P1] (revoke P2)
Consumer 2: [P3, P4, P5] → [P3, P4, P5] (no change)
Consumer 3: [P6, P7, P8] → [P6, P7, P8] (no change)

Phase 2: Assign revoked partitions to new owners
Consumer 1: [P0, P1] → [P0, P1] (no change)
Consumer 2: [P3, P4, P5] → [P3, P4, P5] (no change)
Consumer 3: [P6, P7, P8] → [P6, P7, P8, P2] (assign P2)

​
Configuration

# Enable incremental cooperative rebalancing (default in Kafka 2.4+)
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

# Or use range assignor with cooperative rebalancing
partition.assignment.strategy=org.apache.kafka.clients.consumer.RangeAssignor

​
Static group membership

​
Concept

Static group membership allows consumers to maintain stable identities across restarts, preventing unnecessary rebalances during planned maintenance or brief outages.

​
Benefits

  • Fewer rebalances: Consumer restarts don’t trigger rebalances
  • Stable assignments: Partitions stay with the same consumer instance
  • Faster recovery: Consumers can resume processing from where they left off
  • Operational efficiency: Planned maintenance doesn’t disrupt other consumers

​
Configuration

# Assign static member ID to consumer
group.instance.id=consumer-instance-1

# Increase session timeout for planned restarts
session.timeout.ms=300000  # 5 minutes

# Adjust heartbeat interval accordingly
heartbeat.interval.ms=100000  # ~1.7 minutes

​
Consumer lifecycle

Properties props = new Properties();
props.put("group.id", "my-consumer-group");
props.put("group.instance.id", "consumer-1"); // Static member ID
props.put("session.timeout.ms", "300000");    // 5 minutes
props.put("heartbeat.interval.ms", "100000"); // ~1.7 minutes

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);

​
Use cases and benefits

​
High-availability applications

# Configuration for critical applications
group.instance.id=${hostname}-${process.id}
session.timeout.ms=300000
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

# Allows planned restarts without affecting other consumers

​
Containerized environments

# Kubernetes deployment example
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
      - name: kafka-consumer
        env:
        - name: GROUP_INSTANCE_ID
          value: "consumer-${POD_NAME}"
        # Consumer will maintain identity across pod restarts

​
Stream processing applications

  • State preservation: Local state stores remain associated with specific consumers
  • Reduced reprocessing: Avoid recomputing state after rebalances
  • Consistent partitioning: Same consumer always processes same partitions

​
Monitoring and observability

​
Key metrics

  • Rebalance frequency: Number of rebalances per time period
  • Rebalance duration: Time taken for rebalance completion
  • Partition assignment stability: How often partitions change owners
  • Consumer lag during rebalance: Processing delay during rebalances

​
JMX metrics

# Rebalance metrics
kafka.consumer:type=consumer-coordinator-metrics,client-id=*
- rebalance-rate-per-hour
- rebalance-latency-avg
- rebalance-latency-max

# Assignment metrics  
kafka.consumer:type=consumer-metrics,client-id=*
- assigned-partitions

​
Configuration best practices

​
For incremental rebalancing

# Use cooperative assignors
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

# Optimize for stability
session.timeout.ms=45000
heartbeat.interval.ms=15000
max.poll.interval.ms=300000

​
For static group membership

# Stable consumer identity
group.instance.id=unique-consumer-id

# Extended timeouts for planned restarts
session.timeout.ms=600000    # 10 minutes
heartbeat.interval.ms=200000 # ~3.3 minutes

# Prevent accidental timeouts
max.poll.interval.ms=900000  # 15 minutes

​
Combined configuration

# Best of both worlds
group.instance.id=consumer-${hostname}
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor
session.timeout.ms=300000
heartbeat.interval.ms=100000
max.poll.interval.ms=600000

​
Operational considerations

​
Deployment strategies

  1. Rolling updates: Use static group membership for zero-downtime deployments
  2. Blue-green: Static IDs help maintain partition assignments
  3. Canary releases: Incremental rebalancing minimizes impact on stable consumers

​
Maintenance windows

# Planned consumer restart with static membership
# 1. Consumer stops gracefully
# 2. Other consumers continue processing (no rebalance)
# 3. Consumer restarts with same group.instance.id
# 4. Resumes processing assigned partitions

​
Troubleshooting

Common issues and solutions:
  • Duplicate static IDs: Ensure unique group.instance.id per consumer
  • Long session timeouts: Balance between stability and failure detection
  • Assignment strategy conflicts: Ensure all consumers use compatible assignors
Migration strategyWhen migrating to incremental rebalancing and static membership:
  1. Start with incremental rebalancing first
  2. Monitor rebalance behavior and performance
  3. Gradually introduce static group membership
  4. Test failure scenarios thoroughly
Static member considerations
  • Static members that don’t restart within session.timeout.ms will be removed from the group
  • Ensure unique group.instance.id values to avoid conflicts
  • Plan for scaling scenarios where static IDs need management

​
Performance impact

​
Before (eager rebalancing)

Rebalance triggered → All consumers stop → Complete reassignment → Resume processing
Downtime: 10-30 seconds for entire consumer group

​
After (incremental + static)

Rebalance triggered → Only affected partitions stop → Minimal reassignment → Resume processing  
Downtime: 1-5 seconds for affected partitions only

​
Measurable improvements

  • 90% reduction in processing downtime during rebalances
  • 50% fewer unnecessary rebalances with static membership
  • Improved throughput due to reduced processing interruptions
  • Better consumer utilization with sticky partition assignments