Skip to main content
Quick navigation

What is cluster switching?

Conduktor Gateway's cluster switching allows hot-switch the backend Kafka cluster without having to change your client configuration, or restart Gateway.

This features enables you to build a seamless disaster recovery strategy for your Kafka cluster, when Gateway is deployed in combination with a replication solution (like MirrorMaker, Confluent replicator, Cluster Linking, etc.).

View the full demo in realtime

You can either follow all the steps manually, or watch the recording

Limitations to consider when designing a disaster recovery strategy

  • Cluster switching does not replicate data between clusters. You need to use a replication solution like MirrorMaker to replicate data between clusters
  • Because of their asynchronous nature, such replication solutions may lead to data loss in case of a disaster
  • Cluster switching is a manual process - automatic failover is not supported, yet
  • Concentrated topics offsets: Gateway stores client offsets of concentrated topics in a regular Kafka topic. When replicating this topic, there will be no adjustments of potential offsets shifts between the source and failover cluster
  • When switching, Kafka consumers will perform a group rebalance. They will not be able to commit their offset before the rebalance. This may lead to a some messages being consumed twice

Review the docker compose environment

As can be seen from docker-compose.yaml the demo environment consists of the following services:

  • failover-kafka1
  • failover-kafka2
  • failover-kafka3
  • gateway1
  • gateway2
  • kafka-client
  • kafka1
  • kafka2
  • kafka3
  • mirror-maker
  • schema-registry
  • zookeeper
cat docker-compose.yaml

Review the Gateway configuration

Review the Gateway configuration

cat clusters.yaml

Review the Mirror-Maker configuration

Review the Mirror-Maker configuration

cat mm2.properties

Starting the docker environment

Start all your docker processes, wait for them to be up and ready, then run in background

  • --wait: Wait for services to be running|healthy. Implies detached mode.
  • --detach: Detached mode: Run containers in the background
docker compose up --detach --wait

Creating virtual cluster teamA

Creating virtual cluster teamA on gateway gateway1 and reviewing the configuration file to access it

# Generate virtual cluster teamA with service account sa
token=$(curl \
--request POST "http://localhost:8888/admin/vclusters/v1/vcluster/teamA/username/sa" \
--header 'Content-Type: application/json' \
--user 'admin:conduktor' \
--silent \
--data-raw '{"lifeTimeSeconds": 7776000}' | jq -r ".token")

# Create access file
echo """
bootstrap.servers=localhost:6969
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='sa' password='$token';
""" > teamA-sa.properties

# Review file
cat teamA-sa.properties

Creating topic users on teamA

Creating on teamA:

  • Topic users with partitions:1 and replication-factor:1
kafka-topics \
--bootstrap-server localhost:6969 \
--command-config teamA-sa.properties \
--replication-factor 1 \
--partitions 1 \
--create --if-not-exists \
--topic users

Send tom and laura into topic users

Producing 2 messages in users in cluster teamA

Sending 2 events

{
"name" : "tom",
"username" : "tom@conduktor.io",
"password" : "motorhead",
"visa" : "#abc123",
"address" : "Chancery lane, London"
}
{
"name" : "laura",
"username" : "laura@conduktor.io",
"password" : "kitesurf",
"visa" : "#888999XZ",
"address" : "Dubai, UAE"
}

with

echo '{"name":"tom","username":"tom@conduktor.io","password":"motorhead","visa":"#abc123","address":"Chancery lane, London"}' | \
kafka-console-producer \
--bootstrap-server localhost:6969 \
--producer.config teamA-sa.properties \
--topic users

echo '{"name":"laura","username":"laura@conduktor.io","password":"kitesurf","visa":"#888999XZ","address":"Dubai, UAE"}' | \
kafka-console-producer \
--bootstrap-server localhost:6969 \
--producer.config teamA-sa.properties \
--topic users

Listing topics in kafka1

kafka-topics \
--bootstrap-server localhost:19092,localhost:19093,localhost:19094 \
--list

Wait for mirror maker to do its job on gateway internal topic

Wait for mirror maker to do its job on gateway internal topic in cluster failover-kafka1

kafka-console-consumer \
--bootstrap-server localhost:29092,localhost:29093,localhost:29094 \
--topic _topicMappings \
--from-beginning \
--max-messages 1 \
--timeout-ms 15000 | jq

returns

Processed a total of 1 messages
{
"users": {
"clusterId": "main",
"name": "teamAusers",
"isConcentrated": false,
"compactedName": "teamAusers",
"isCompacted": false,
"compactedAndDeletedName": "teamAusers",
"isCompactedAndDeleted": false,
"createdAt": [
2024,
2,
14,
0,
33,
23,
291
],
"isDeleted": false,
"configuration": {
"numPartitions": 1,
"replicationFactor": 1,
"properties": {}
},
"isVirtual": false
}
}

Wait for mirror maker to do its job on users topics

Wait for mirror maker to do its job on users topics in cluster failover-kafka1

kafka-console-consumer \
--bootstrap-server localhost:29092,localhost:29093,localhost:29094 \
--topic teamAusers \
--from-beginning \
--max-messages 1 \
--timeout-ms 15000 | jq

returns

Processed a total of 1 messages
{
"name": "tom",
"username": "tom@conduktor.io",
"password": "motorhead",
"visa": "#abc123",
"address": "Chancery lane, London"
}

Assert mirror maker did its job

kafka-topics \
--bootstrap-server localhost:29092,localhost:29093,localhost:29094 \
--list

Failing over from main to failover

Failing over from main to failover on gateway gateway1

curl \
--request POST 'http://localhost:8888/admin/pclusters/v1/pcluster/main/switch?to=failover' \
--user 'admin:conduktor' \
--silent | jq
    From now on `gateway1` the cluster with id `main` is pointing to the `failover cluster.

Failing over from main to failover

Failing over from main to failover on gateway gateway2

curl \
--request POST 'http://localhost:8889/admin/pclusters/v1/pcluster/main/switch?to=failover' \
--user 'admin:conduktor' \
--silent | jq
    From now on `gateway2` the cluster with id `main` is pointing to the `failover cluster.

Produce alice into users, it should hit only failover-kafka

Producing 1 message in users in cluster teamA

Sending 1 event

{
"name" : "alice",
"username" : "alice@conduktor.io",
"password" : "youpi",
"visa" : "#812SSS",
"address" : "Les ifs"
}

with

echo '{"name":"alice","username":"alice@conduktor.io","password":"youpi","visa":"#812SSS","address":"Les ifs"}' | \
kafka-console-producer \
--bootstrap-server localhost:6969 \
--producer.config teamA-sa.properties \
--topic users

Verify we can read laura (via mirror maker), tom (via mirror maker) and alice (via cluster switching)

Verify we can read laura (via mirror maker), tom (via mirror maker) and alice (via cluster switching) in cluster teamA

kafka-console-consumer \
--bootstrap-server localhost:6969 \
--consumer.config teamA-sa.properties \
--topic users \
--from-beginning \
--max-messages 3 \
--timeout-ms 10000 | jq

returns

Processed a total of 3 messages
{
"name": "tom",
"username": "tom@conduktor.io",
"password": "motorhead",
"visa": "#abc123",
"address": "Chancery lane, London"
}
{
"name": "laura",
"username": "laura@conduktor.io",
"password": "kitesurf",
"visa": "#888999XZ",
"address": "Dubai, UAE"
}
{
"name": "alice",
"username": "alice@conduktor.io",
"password": "youpi",
"visa": "#812SSS",
"address": "Les ifs"
}

Verify alice is not in main kafka

Verify alice is not in main kafka in cluster kafka1

kafka-console-consumer \
--bootstrap-server localhost:19092,localhost:19093,localhost:19094 \
--topic teamAusers \
--from-beginning \
--timeout-ms 10000 | jq

returns

Processed a total of 2 messages
{
"name": "tom",
"username": "tom@conduktor.io",
"password": "motorhead",
"visa": "#abc123",
"address": "Chancery lane, London"
}
{
"name": "laura",
"username": "laura@conduktor.io",
"password": "kitesurf",
"visa": "#888999XZ",
"address": "Dubai, UAE"
}

Verify alice is in failover

Verify alice is in failover in cluster failover-kafka1

kafka-console-consumer \
--bootstrap-server localhost:29092,localhost:29093,localhost:29094 \
--topic teamAusers \
--from-beginning \
--max-messages 3 \
--timeout-ms 15000 | jq

returns

Processed a total of 3 messages
{
"name": "tom",
"username": "tom@conduktor.io",
"password": "motorhead",
"visa": "#abc123",
"address": "Chancery lane, London"
}
{
"name": "laura",
"username": "laura@conduktor.io",
"password": "kitesurf",
"visa": "#888999XZ",
"address": "Dubai, UAE"
}
{
"name": "alice",
"username": "alice@conduktor.io",
"password": "youpi",
"visa": "#812SSS",
"address": "Les ifs"
}

Tearing down the docker environment

Remove all your docker processes and associated volumes

  • --volumes: Remove named volumes declared in the "volumes" section of the Compose file and anonymous volumes attached to containers.
docker compose down --volumes

Conclusion

Cluster switching help your seamlessly move from one cluster to another!