Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.conduktor.io/llms.txt

Use this file to discover all available pages before exploring further.

Learn how to optimize consumer reads with rack awareness in 12 minutes Kafka consumers can be configured to read from the closest replica rather than always reading from the leader, which can improve performance and reduce cross-datacenter network traffic in geographically distributed deployments. What you’ll learn:
  • How replica reading works in Kafka
  • How to configure rack-aware consumers
  • Benefits and trade-offs of closest replica reading
  • Best practices for multi-region deployments

​
How replica reading works

By default, Kafka consumers always read from the partition leader. However, starting with Kafka 2.4, consumers can be configured to read from follower replicas that are “close” to the consumer.

​
Default behavior (leader-only reads)

  • All consumers read from partition leaders
  • Followers only replicate data, never serve reads
  • Simple and consistent behavior
  • May result in cross-datacenter traffic

​
Closest replica reading

  • Consumers can read from geographically closest replicas
  • Reduces network latency and cross-datacenter bandwidth
  • Requires proper rack awareness configuration
  • Maintains consistency guarantees

​
Configuration

​
Enable closest replica reading

# Consumer configuration
client.rack=us-west-2a

# This tells Kafka which rack/availability zone the consumer is in
# Kafka will prefer replicas in the same rack when available

​
Broker configuration for rack awareness

# Broker configuration (server.properties)
broker.rack=us-west-2a

# Each broker should be configured with its rack/AZ
# This enables Kafka to make intelligent replica placement decisions

​
Topic configuration

When creating topics, consider replica placement: 
# Create topic with rack-aware replica assignment
kafka-topics --bootstrap-server localhost:9092 \
  --create --topic my-topic \
  --partitions 6 \
  --replication-factor 3 \
  --config min.insync.replicas=2

​
Benefits

BenefitDescription
Reduced latencyConsumers read from local replicas instead of remote leaders
Cost savingsReduces expensive cross-region data transfer charges
Improved availabilityContinues reading even if cross-datacenter links are degraded

​
Consistency considerations

​
Read-after-write consistency

With closest replica reading, you may encounter scenarios where:
  • Producer writes to leader in datacenter A
  • Consumer reads from follower in datacenter B
  • Replication lag may cause temporary inconsistency

​
Mitigation strategies

# Ensure minimum in-sync replicas for writes
min.insync.replicas=2

# Use appropriate acks setting
acks=all

# Configure consumer to handle potential inconsistencies
Replication lag impactWhen reading from follower replicas, consumers may see slightly stale data due to replication lag. Ensure your application can tolerate this eventual consistency model.

​
Use cases

​
Multi-region deployments

Region US-West:
- Brokers with rack=us-west
- Consumers with client.rack=us-west
- Reads stay local to us-west replicas

Region EU-Central:
- Brokers with rack=eu-central
- Consumers with client.rack=eu-central
- Reads stay local to eu-central replicas

​
Availability zone optimization

AZ-1: broker.rack=az-1, client.rack=az-1
AZ-2: broker.rack=az-2, client.rack=az-2
AZ-3: broker.rack=az-3, client.rack=az-3

Each AZ reads from local replicas when possible

​
Configuration examples

​
Cloud deployment (AWS)

# Broker configuration
broker.rack=${aws.availability.zone}

# Consumer configuration
client.rack=us-west-2a

# Additional consumer settings for optimal performance
fetch.min.bytes=1048576
fetch.max.wait.ms=500

​
On-premises multi-datacenter

# Broker configuration
broker.rack=datacenter-1

# Consumer configuration
client.rack=datacenter-1

# Network optimization
socket.receive.buffer.bytes=65536
fetch.max.bytes=52428800

​
Decision guide

ScenarioRecommendation
Single datacenterNot needed
Multi-AZ within regionOptional, reduces AZ-to-AZ traffic
Multi-regionRecommended for cost and latency
Strong consistency requiredUse leader-only reads
Eventually consistent OKUse closest replica
Gradual rolloutConsider implementing closest replica reading gradually:
  1. Start with non-critical consumer groups
  2. Monitor metrics and consistency behavior
  3. Expand to more critical workloads as confidence builds

​
Troubleshooting

​
Common issues

IssueCauseSolution
Still reading from leaderMissing rack configurationConfigure client.rack on consumer
No local replicasInsufficient replicas in rackAdd brokers or increase replication factor
High replication lagFollower falling behindMonitor and tune replication

​
Verification steps

# Check broker rack configuration
kafka-configs --bootstrap-server localhost:9092 --describe --entity-type brokers

# Monitor consumer metrics
kafka-consumer-groups --bootstrap-server localhost:9092 --describe --group your-group

# Check topic replica distribution
kafka-topics --bootstrap-server localhost:9092 --describe --topic your-topic
See it in practice with ConduktorConduktor Console displays broker rack configuration and replica distribution across your cluster. Monitor which replicas consumers are reading from and track cross-datacenter traffic patterns.

​
Next steps