Learn how to upgrade Kafka clusters with zero downtime in 15 minutes
Kafka’s versioned API calls enable bi-directional compatibility between clients and brokers. This means you can upgrade brokers and clients independently without coordinated downtime.
What you’ll learn:
- How Kafka client/broker version compatibility works
- The rolling upgrade process for brokers
- How to upgrade without data loss or downtime
- When to upgrade clients vs brokers
Kafka client version compatibility
Since Kafka 0.10.2, clients and brokers have bi-directional compatibility because API calls are versioned.
| Scenario | Compatible? |
|---|
| Old client (2.1) → New broker (3.0) | Yes |
| New client (3.0) → Old broker (2.1) | Yes |
Always use the latest client library version if you can
Kafka broker version upgrades
Upgrading Kafka brokers is by far the most important operation that we have to do on our cluster. It is recommended to upgrade very often in order to keep up with bug fixes, new features, and performance improvements. The more we wait to upgrade a broker, the more painful it will be as there will be many versions jump.
Instructions on how to upgrade Kafka are always outlined in the documentation. Regardless, the steps are usually pretty similar from version to version, as you’ll see below.
How to upgrade Kafka brokers without downtime?
It is fairly simple to upgrade Kafka. There are five steps to upgrade with a rolling restart required at each step. In this example we are giving the steps to upgrade from version Kafka v2.1 to v3.0.
-
Set inter broker and log version to current Kafka version. If you are using 2.1 now, just put the following broker properties:
inter.broker.protocol.version=2.1 and log.message.format.version=2.1
-
One broker at a time: stop the broker, upgrade the binaries to 3.0, and start the broker. Wait for the cluster to stabilize (all the partitions are in-sync replicas (ISR).
-
At this stage, all your brokers are running the newer binaries of 3.0, but running with
inter.broker.protocol.version=2.1 and log.message.format.version=2.1
-
Ensure the cluster is stable in this form. You still have the option to downgrade here if you want to because the log message format has not been upgraded yet.
-
One broker at a time: stop the broker, change the inter broker protocol version for that broker with
inter.broker.protocol.version=3.0 and start that broker. Wait for the cluster to stabilize (all the partitions are in-sync replicas (ISR).
-
At the end of this stage, all the brokers are running with the new inter broker protocol version, but they are not using the new log message format yet.
-
Upgrade Kafka clients if documentation specifies it (all or most of them to avoid up and down conversion - which can affect performance)
-
One broker at a time: stop the broker, upgrade message protocol version
log.message.format.version=3.0 and start the broker. Wait for the cluster to stabilize (all the partitions are in-sync replicas (ISR).
-
At this stage, all brokers are upgraded to the latest binary, are running the latest inter-broker protocol and are using the latest log message format version. Congratulations!
Monitor during upgradesWatch ISR counts, under-replicated partitions, and request latencies during rolling restarts. Wait for the cluster to stabilize before proceeding to the next broker.
Upgrade summary
| Phase | Action | Rollback possible? |
|---|
| 1 | Set version properties | Yes |
| 2 | Upgrade binaries | Yes |
| 3 | Upgrade inter.broker.protocol | Yes |
| 4 | Upgrade clients | Yes |
| 5 | Upgrade log.message.format | Limited |
See it in practice with ConduktorConduktor Console helps you monitor cluster health during upgrades. Track broker versions, ISR status, and partition health in real-time to ensure safe rolling restarts. Next steps
