Skip to main content
Learn advanced producer configuration options in 15 minutes Beyond basic producer settings, advanced configurations provide fine-grained control over producer behavior, performance, and reliability for production deployments. What you’ll learn:
  • Buffer memory and timeout configurations
  • Connection and network settings
  • Monitoring and metrics configuration
  • Environment-specific tuning templates

​
Configuration categories

​
Buffer memory configuration

​
buffer.memory

Total memory allocated for producer buffering: 
# Default: 33554432 (32MB)
buffer.memory=67108864  # 64MB total buffer
Impact:
  • Too small: Producer blocks frequently when buffer fills
  • Too large: Excessive memory usage, potential GC pressure
  • Optimal size: Balance between memory efficiency and producer throughput

​
max.block.ms

Time to block when buffer is full: 
# Default: 60000 (1 minute)
max.block.ms=30000  # Block for 30 seconds maximum

​
Memory calculation

Required buffer = batch.size × active_partitions × safety_factor

Example:
- 100 active partitions
- 32KB batch size
- 2x safety factor
= 32KB × 100 × 2 = 6.4MB minimum

​
Request timeout settings

SettingDefaultDescription
request.timeout.ms30sTimeout for individual requests
delivery.timeout.ms2minOverall timeout including retries
metadata.max.age.ms5minHow long to cache topic metadata

​
Timeout relationship

If delivery.timeout.ms is shorter than the worst-case retry sequence, the producer throws a ConfigException at startup: 
delivery.timeout.ms >= request.timeout.ms + (retries × retry.backoff.ms)

​
Connection configurations

​
max.in.flight.requests.per.connection

Maximum unacknowledged requests per connection: 
# Default: 5
max.in.flight.requests.per.connection=5
Impact on ordering:
SettingOrderingThroughput
1StrictLower
5 (with idempotence)MaintainedHigher
> 5 (without idempotence)May reorderHighest

​
Configuration templates

​
High-throughput configuration

# Optimize for maximum throughput
batch.size=65536
linger.ms=20
buffer.memory=134217728
compression.type=lz4
max.in.flight.requests.per.connection=5
request.timeout.ms=30000
delivery.timeout.ms=120000
enable.idempotence=true

​
Low-latency configuration

# Optimize for minimum latency
batch.size=16384
linger.ms=0
buffer.memory=33554432
compression.type=none
max.in.flight.requests.per.connection=1
request.timeout.ms=10000
delivery.timeout.ms=30000

​
High-reliability configuration

# Optimize for maximum reliability
acks=all
retries=2147483647
enable.idempotence=true
max.in.flight.requests.per.connection=5
delivery.timeout.ms=600000
request.timeout.ms=30000
buffer.memory=67108864
batch.size=32768

​
Memory-constrained configuration

# Minimize memory usage
buffer.memory=16777216      # 16MB
batch.size=8192             # 8KB
max.request.size=1048576    # 1MB
connections.max.idle.ms=60000

​
Monitor your configuration

​
Essential metrics to track

MetricDescriptionAlert threshold
buffer-available-bytesAvailable buffer space< 10%
request-latency-avgAverage request latency> baseline
record-error-rateRate of failed records> 0
record-retry-rateRate of retried records> baseline

​
Client identification

# Always set for easier troubleshooting
client.id=order-service-producer-01

​
Common configuration mistakes

MistakeProblemSolution
Insufficient buffer.memoryFrequent blockingIncrease buffer or reduce batch size
Too high linger.msUnnecessary latencyReduce to 0-20ms
Mismatched timeoutsdelivery.timeout too smallEnsure delivery > request + retries
Missing client.idHard to troubleshootAlways set meaningful client ID
Wrong max.in.flightOrdering issuesUse 5 with idempotence
Some configurations are interdependent. Enabling enable.idempotence=true automatically sets retries=Integer.MAX_VALUE, max.in.flight.requests.per.connection=5, and acks=all. Begin with default configurations and adjust one parameter at a time based on monitoring data—most applications work well with minimal configuration changes.
See it in practice with ConduktorConduktor Console helps you test different producer configurations and observe their effects. Monitor producer metrics to validate your configuration changes work as expected.

​
Next steps