Documentation Index
Fetch the complete documentation index at: https://docs.conduktor.io/llms.txt
Use this file to discover all available pages before exploring further.
Learn advanced producer configuration options in 15 minutes
Beyond basic producer settings, advanced configurations provide fine-grained control over producer behavior, performance, and reliability for production deployments.
What you’ll learn:
- Buffer memory and timeout configurations
- Connection and network settings
- Monitoring and metrics configuration
- Environment-specific tuning templates
Configuration categories
Buffer memory configuration
buffer.memory
Total memory allocated for producer buffering:
# Default: 33554432 (32MB)
buffer.memory=67108864 # 64MB total buffer
- Too small: Producer blocks frequently when buffer fills
- Too large: Excessive memory usage, potential GC pressure
- Optimal size: Balance between memory efficiency and producer throughput
max.block.ms
Time to block when buffer is full:
# Default: 60000 (1 minute)
max.block.ms=30000 # Block for 30 seconds maximum
Memory calculation
Required buffer = batch.size × active_partitions × safety_factor
Example:
- 100 active partitions
- 32KB batch size
- 2x safety factor
= 32KB × 100 × 2 = 6.4MB minimum
Request timeout settings
| Setting | Default | Description |
|---|
request.timeout.ms | 30s | Timeout for individual requests |
delivery.timeout.ms | 2min | Overall timeout including retries |
metadata.max.age.ms | 5min | How long to cache topic metadata |
Timeout relationship
If delivery.timeout.ms is shorter than the worst-case retry sequence, the producer throws a ConfigException at startup:
delivery.timeout.ms >= request.timeout.ms + (retries × retry.backoff.ms)
Connection configurations
max.in.flight.requests.per.connection
Maximum unacknowledged requests per connection:
# Default: 5
max.in.flight.requests.per.connection=5
| Setting | Ordering | Throughput |
|---|
| 1 | Strict | Lower |
| 5 (with idempotence) | Maintained | Higher |
| > 5 (without idempotence) | May reorder | Highest |
Configuration templates
High-throughput configuration
# Optimize for maximum throughput
batch.size=65536
linger.ms=20
buffer.memory=134217728
compression.type=lz4
max.in.flight.requests.per.connection=5
request.timeout.ms=30000
delivery.timeout.ms=120000
enable.idempotence=true
Low-latency configuration
# Optimize for minimum latency
batch.size=16384
linger.ms=0
buffer.memory=33554432
compression.type=none
max.in.flight.requests.per.connection=1
request.timeout.ms=10000
delivery.timeout.ms=30000
High-reliability configuration
# Optimize for maximum reliability
acks=all
retries=2147483647
enable.idempotence=true
max.in.flight.requests.per.connection=5
delivery.timeout.ms=600000
request.timeout.ms=30000
buffer.memory=67108864
batch.size=32768
Memory-constrained configuration
# Minimize memory usage
buffer.memory=16777216 # 16MB
batch.size=8192 # 8KB
max.request.size=1048576 # 1MB
connections.max.idle.ms=60000
Monitor your configuration
Essential metrics to track
| Metric | Description | Alert threshold |
|---|
buffer-available-bytes | Available buffer space | < 10% |
request-latency-avg | Average request latency | > baseline |
record-error-rate | Rate of failed records | > 0 |
record-retry-rate | Rate of retried records | > baseline |
Client identification
# Always set for easier troubleshooting
client.id=order-service-producer-01
Common configuration mistakes
| Mistake | Problem | Solution |
|---|
Insufficient buffer.memory | Frequent blocking | Increase buffer or reduce batch size |
Too high linger.ms | Unnecessary latency | Reduce to 0-20ms |
| Mismatched timeouts | delivery.timeout too small | Ensure delivery > request + retries |
Missing client.id | Hard to troubleshoot | Always set meaningful client ID |
Wrong max.in.flight | Ordering issues | Use 5 with idempotence |
enable.idempotence=true automatically sets retries=Integer.MAX_VALUE, max.in.flight.requests.per.connection=5, and acks=all. Begin with default configurations and adjust one parameter at a time based on monitoring data—most applications work well with minimal configuration changes.
See it in practice with ConduktorConduktor Console helps you test different producer configurations and observe their effects. Monitor producer metrics to validate your configuration changes work as expected. Next steps
