Skip to main content
Learn how to handle large messages in Kafka in 15 minutes Apache Kafka has default message size limits that can be configured to handle larger payloads, but there are important considerations and best practices to follow. What you’ll learn:
  • Default message size limits across Kafka components
  • How to configure producers, brokers, and consumers for large messages
  • Performance implications of large messages
  • Alternative patterns for handling large payloads

​
Default message size limits

By default, Kafka has the following message size limits:
ComponentConfigurationDefault
Producermax.request.size1 MB
Brokermessage.max.bytes1 MB
Topicmax.message.bytesInherits from broker
Consumermax.partition.fetch.bytes1 MB

​
Configure Kafka for large messages

To send messages larger than 1MB, you need to configure multiple components:

​
Producer configuration

# Set maximum request size for producer
max.request.size=10485760  # 10MB
# Increase buffer memory if needed
buffer.memory=67108864     # 64MB

​
Broker configuration

# Set maximum message size for broker
message.max.bytes=10485760          # 10MB
# Set maximum replica fetch size
replica.fetch.max.bytes=10485760    # 10MB
# Set maximum response size
socket.receive.buffer.bytes=1048576 # 1MB
socket.send.buffer.bytes=1048576    # 1MB

​
Topic configuration

kafka-configs --bootstrap-server localhost:9092 \
  --alter --entity-type topics --entity-name large-topic \
  --add-config max.message.bytes=10485760

​
Consumer configuration

# Set maximum fetch size for consumer
max.partition.fetch.bytes=10485760  # 10MB
fetch.max.bytes=52428800            # 50MB

​
Performance implications

Sending large messages in Kafka has several performance implications:
AreaImpactMitigation
MemoryHigher heap usage, GC pressureTune JVM heap sizes
NetworkMore bandwidth, potential timeoutsAdjust buffer sizes
Disk I/OMore operations, slower compactionUse faster storage
ThroughputLower overall message rateEnable compression

​
Alternative approaches

Instead of sending large messages directly, consider these alternatives:

​
1. External storage pattern

Store large payloads in external systems and send only references: 
{
  "id": "message-123",
  "timestamp": "2023-01-01T00:00:00Z",
  "data_location": "s3://bucket/path/to/large-file.json",
  "metadata": {
    "size": 50000000,
    "checksum": "abc123"
  }
}
Benefits:
  • Keeps Kafka messages small and fast
  • Allows for separate scaling of storage and messaging
  • Enables efficient caching strategies

​
2. Split messages

Break large messages into smaller chunks: 
{
  "message_id": "msg-123",
  "chunk_id": "chunk-1",
  "total_chunks": 5,
  "chunk_data": "...",
  "sequence": 1
}
Benefits:
  • Works within default Kafka limits
  • Allows for parallel processing
  • Provides better error recovery

​
3. Compression

Enable compression to reduce message sizes: 
# Producer compression
compression.type=snappy  # or gzip, lz4, zstd
Benefits:
  • Reduces network bandwidth usage
  • Decreases storage requirements
  • Often improves throughput

​
Best practices

Recommendations for large messages
  1. Avoid large messages when possible - Kafka is optimized for small, high-throughput messages
  2. Use external storage - Store large payloads externally and reference them in Kafka messages
  3. Enable compression - Always enable compression for large messages
  4. Monitor memory usage - Ensure adequate heap sizing for all components
  5. Test thoroughly - Verify performance impact in your specific environment

​
Configuration checklist

When configuring for large messages, ensure all these settings are aligned:
  • ✅ Producer max.request.size
  • ✅ Broker message.max.bytes
  • ✅ Topic max.message.bytes
  • ✅ Consumer max.partition.fetch.bytes
  • ✅ Consumer fetch.max.bytes
  • ✅ Broker replica.fetch.max.bytes

​
Monitor large messages

Monitor these metrics when working with large messages:
  • Memory usage on brokers, producers, and consumers
  • Network bandwidth utilization
  • Disk I/O patterns and latency
  • Garbage collection frequency and duration
  • Message throughput and latency

​
Large message decision guide

Large messages can significantly impact Kafka performance. Always test in a staging environment that mirrors your production setup before deploying large message configurations.
See it in practice with ConduktorConduktor Console lets you produce and consume messages while monitoring their sizes. Test your large message configurations and verify all component limits are aligned.

​
Next steps