Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.conduktor.io/llms.txt

Use this file to discover all available pages before exploring further.

Learn how to choose and configure Kafka compression algorithms in 12 minutes Kafka producer compression reduces the size of messages before sending them to brokers, improving throughput and reducing network bandwidth at the cost of CPU overhead. What you’ll learn:
  • How compression works in Kafka and when to use it
  • Characteristics of each compression algorithm (GZIP, LZ4, Snappy, ZSTD)
  • How to choose the right algorithm for your use case
  • How to monitor compression effectiveness

​
Why use compression?

Compression provides several benefits in Kafka deployments:
  • Reduced network usage: Smaller message sizes mean less data transmitted
  • Lower storage costs: Compressed messages take less disk space on brokers
  • Improved throughput: More messages can fit in each batch
  • Better performance: Particularly beneficial for text-heavy payloads like JSON or XML
Compression is most effective with:
  • Text-based formats (JSON, XML, CSV)
  • Repetitive data patterns
  • Large message payloads (> 1KB)

​
Compression algorithms

Kafka supports four compression algorithms, each with different performance characteristics:

​
GZIP

General-purpose compression with high compression ratios but higher CPU usage. Characteristics:
  • Highest compression ratio (smallest message size)
  • Slowest compression/decompression speed
  • Good for scenarios where network bandwidth is limited
  • CPU-intensive
Configuration: 
compression.type=gzip

​
LZ4

Fast compression with moderate compression ratios, good balance of speed and size reduction. Characteristics:
  • Very fast compression/decompression
  • Moderate compression ratio
  • Low CPU overhead
  • Good general-purpose choice
Configuration: 
compression.type=lz4

​
Snappy

Fast compression optimized for speed with reasonable compression ratios. Characteristics:
  • Fast compression/decompression (slightly slower than LZ4)
  • Moderate compression ratio
  • Good CPU efficiency
  • Popular choice for high-throughput scenarios
Configuration: 
compression.type=snappy

​
ZSTD (Zstandard)

Modern compression algorithm offering excellent compression ratios with good performance. Characteristics:
  • Excellent compression ratio (better than LZ4/Snappy, approaches GZIP)
  • Good compression/decompression speed
  • Relatively new (available since Kafka 2.1)
  • Tunable compression levels
Configuration: 
compression.type=zstd

​
Performance comparison

​
Compression ratio comparison

Based on typical JSON payloads:
AlgorithmCompression ratioCPU usageSpeed
GZIP~75%HighSlow
LZ4~65%LowFast
Snappy~68%LowFast
ZSTD~72%MediumMedium

​
Visual comparison: Compression trade-offs

This comparison shows how different algorithms balance compression ratio and speed:
AlgorithmCompression ratioSpeedCPU usageBest use case
GZIPHighest (~75%)SlowestHighNetwork-constrained environments
ZSTDHigh (~72%)MediumMediumBalanced choice (RECOMMENDED)
LZ4Medium (~65%)Very fastVery LowHigh-throughput scenarios
SnappyMedium (~68%)Very fastLowHigh-throughput scenarios
Key insights:
  • ZSTD offers the best balance: high compression with reasonable speed (ideal for most workloads)
  • LZ4 and snappy prioritize speed over compression (best for real-time/high-throughput)
  • GZIP maximizes compression at the cost of speed (best for bandwidth-limited scenarios)

​
Throughput impact

Compression affects producer throughput in different ways: Positive impact:
  • Faster network transmission due to smaller message sizes
  • Higher effective batch sizes (more messages per batch)
  • Reduced broker disk I/O
Negative impact:
  • CPU overhead for compression on producer side
  • Additional latency for compression process

​
Configuration and tuning

​
Basic compression configuration

# Enable compression for all messages
compression.type=lz4

# Compression happens at batch level
batch.size=16384
linger.ms=5

​
Compression at different levels

Compression can be configured at multiple levels: Producer level (affects all topics): 
Properties props = new Properties();
props.put("compression.type", "lz4");
Producer<String, String> producer = new KafkaProducer<>(props);
Topic level (broker-side configuration): 
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name my-topic \
  --alter --add-config compression.type=snappy

​
Interaction with batching

Compression works at the batch level, so batching configuration matters: 
# Larger batches = better compression ratios
batch.size=32768          # 32KB batches
linger.ms=10              # Wait up to 10ms to fill batch
compression.type=lz4      # Fast compression for batches

​
CPU vs network trade-offs

​
When to prioritize CPU savings

Choose faster algorithms (LZ4, Snappy) when:
  • CPU resources are limited
  • High message throughput is required
  • Network bandwidth is abundant
  • Low latency is critical

​
When to prioritize network savings

Choose higher compression (GZIP, ZSTD) when:
  • Network bandwidth is expensive or limited
  • Messages are stored for long periods
  • CPU resources are abundant
  • Storage costs are a concern

​
Choose the right algorithm

​
Decision matrix

For high-throughput, low-latency scenarios: 
compression.type=lz4  # Fast compression, low CPU overhead
For network-constrained environments: 
compression.type=gzip  # Maximum compression, reduce network usage
For balanced performance: 
compression.type=snappy  # Good compression ratio with reasonable speed
For modern deployments (Kafka 2.1+): 
compression.type=zstd  # Excellent compression with good performance

​
Decision tree: Choose your compression algorithm

Use this decision tree to select the optimal compression algorithm based on your requirements:
Quick recommendation: For most production workloads, use ZSTD (Kafka 2.1+) for excellent compression with good performance, or LZ4 if maximum throughput is critical.

​
Monitor compression effectiveness

​
Key metrics to track

Compression ratio: 
compression_ratio = uncompressed_size / compressed_size
Network savings: 
network_savings = (1 - compressed_size / uncompressed_size) × 100%
CPU impact: Monitor CPU utilization on producer machines before and after enabling compression.

​
JMX metrics

kafka.producer:type=producer-metrics,client-id=<client-id>
- compression-rate-avg: Average compression ratio
- record-size-avg: Average record size before compression
- batch-size-avg: Average batch size (after compression)

​
Best practices

​
Production recommendations

  1. Start with LZ4: Good balance of speed and compression for most use cases
  2. Test with your data: Compression effectiveness varies by payload type
  3. Monitor CPU usage: Ensure compression doesn’t become a bottleneck
  4. Combine with batching: Larger batches compress better
  5. Consider message format: JSON compresses better than binary formats

​
Configuration examples

High-throughput producer: 
compression.type=lz4
batch.size=65536          # Large batches for better compression
linger.ms=5               # Short linger time for low latency
buffer.memory=67108864    # 64MB buffer for batching
Network-optimized producer: 
compression.type=gzip
batch.size=32768          # Reasonable batch size
linger.ms=10              # Allow time for batching
buffer.memory=134217728   # 128MB buffer for larger compressed batches
Balanced producer (recommended starting point): 
compression.type=snappy
batch.size=16384          # Default batch size
linger.ms=5               # Low latency
buffer.memory=33554432    # 32MB buffer

​
Common pitfalls

​
Mistakes to avoid

  1. Over-compressing small messages: Compression overhead may exceed benefits
  2. Ignoring CPU monitoring: Compression can become a producer bottleneck
  3. Not testing with production data: Compression ratios vary significantly by content
  4. Using GZIP for high-throughput: May create CPU bottlenecks
  5. Forgetting about decompression: Consumers also pay CPU cost for decompression

​
Troubleshooting compression issues

Poor compression ratios:
  • Check message format (binary data compresses poorly)
  • Verify batch sizes are adequate
  • Consider if data is already compressed
High CPU usage:
  • Switch to faster algorithm (LZ4 instead of GZIP)
  • Monitor producer thread CPU utilization
  • Consider reducing compression level if using ZSTD
Increased latency:
  • Reduce linger.ms to decrease batching delay
  • Use faster compression algorithm
  • Monitor end-to-end message latency
Remember that consumers have to decompress messages, which also uses CPU. Consider the total system CPU cost, not just producer-side overhead. Always benchmark compression algorithms with your actual production data, as compression effectiveness varies significantly based on message format, size, and content patterns.
See it in practice with ConduktorConduktor Console monitors compression metrics in real-time, showing compression ratios, batch sizes, and CPU impact. View producer performance dashboards to validate your compression configuration choices.

​
Next steps