Skip to main content
Quick navigation

What is cluster switching?

Conduktor Gateway's cluster switching allows hot-switch the backend Kafka cluster without having to change your client configuration, or restart Gateway.

This features enables you to build a seamless disaster recovery strategy for your Kafka cluster, when Gateway is deployed in combination with a replication solution (like MirrorMaker, Confluent replicator, Cluster Linking, etc.).

View the full demo in realtime​

You can either follow all the steps manually, or watch the recording

Limitations to consider when designing a disaster recovery strategy​

  • Cluster switching does not replicate data between clusters. You need to use a replication solution like MirrorMaker to replicate data between clusters
  • Because of their asynchronous nature, such replication solutions may lead to data loss in case of a disaster
  • Cluster switching is a manual process - automatic failover is not supported, yet
  • Concentrated topics offsets: Gateway stores client offsets of concentrated topics in a regular Kafka topic. When replicating this topic, there will be no adjustments of potential offsets shifts between the source and failover cluster
  • When switching, Kafka consumers will perform a group rebalance. They will not be able to commit their offset before the rebalance. This may lead to a some messages being consumed twice

Review the docker compose environment​

As can be seen from docker-compose.yaml the demo environment consists of the following services:

  • failover-kafka1
  • failover-kafka2
  • failover-kafka3
  • gateway1
  • gateway2
  • kafka-client
  • kafka1
  • kafka2
  • kafka3
  • mirror-maker
  • schema-registry
  • zookeeper
cat docker-compose.yaml

Review the Gateway configuration​

Review the Gateway configuration

cat clusters.yaml

Review the Mirror-Maker configuration​

Review the Mirror-Maker configuration

cat mm2.properties

Starting the docker environment​

Start all your docker processes, wait for them to be up and ready, then run in background

  • --wait: Wait for services to be running|healthy. Implies detached mode.
  • --detach: Detached mode: Run containers in the background
docker compose up --detach --wait

Creating virtual cluster teamA​

Creating virtual cluster teamA on gateway gateway1 and reviewing the configuration file to access it

# Generate virtual cluster teamA with service account sa
token=$(curl \
    --request POST "http://localhost:8888/admin/vclusters/v1/vcluster/teamA/username/sa" \
    --header 'Content-Type: application/json' \
    --user 'admin:conduktor' \
    --silent \
    --data-raw '{"lifeTimeSeconds": 7776000}' | jq -r ".token")

# Create access file
echo  """
bootstrap.servers=localhost:6969
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='sa' password='$token';
""" > teamA-sa.properties

# Review file
cat teamA-sa.properties

Creating topic users on teamA​

Creating on teamA:

  • Topic users with partitions:1 and replication-factor:1
kafka-topics \
    --bootstrap-server localhost:6969 \
    --command-config teamA-sa.properties \
    --replication-factor 1 \
    --partitions 1 \
    --create --if-not-exists \
    --topic users

Send tom and laura into topic users​

Producing 2 messages in users in cluster teamA

Sending 2 events

{
  "name" : "tom",
  "username" : "tom@conduktor.io",
  "password" : "motorhead",
  "visa" : "#abc123",
  "address" : "Chancery lane, London"
}
{
  "name" : "laura",
  "username" : "laura@conduktor.io",
  "password" : "kitesurf",
  "visa" : "#888999XZ",
  "address" : "Dubai, UAE"
}

with

echo '{"name":"tom","username":"tom@conduktor.io","password":"motorhead","visa":"#abc123","address":"Chancery lane, London"}' | \
    kafka-console-producer \
        --bootstrap-server localhost:6969 \
        --producer.config teamA-sa.properties \
        --topic users

echo '{"name":"laura","username":"laura@conduktor.io","password":"kitesurf","visa":"#888999XZ","address":"Dubai, UAE"}' | \
    kafka-console-producer \
        --bootstrap-server localhost:6969 \
        --producer.config teamA-sa.properties \
        --topic users

Listing topics in kafka1​

kafka-topics \
    --bootstrap-server localhost:19092,localhost:19093,localhost:19094 \
    --list

Wait for mirror maker to do its job on gateway internal topic​

Wait for mirror maker to do its job on gateway internal topic in cluster failover-kafka1

kafka-console-consumer \
    --bootstrap-server localhost:29092,localhost:29093,localhost:29094 \
    --topic _topicMappings \
    --from-beginning \
    --max-messages 1 \
    --timeout-ms 15000 | jq

Wait for mirror maker to do its job on users topics​

Wait for mirror maker to do its job on users topics in cluster failover-kafka1

kafka-console-consumer \
    --bootstrap-server localhost:29092,localhost:29093,localhost:29094 \
    --topic teamAusers \
    --from-beginning \
    --max-messages 1 \
    --timeout-ms 15000 | jq

Assert mirror maker did its job​

kafka-topics \
    --bootstrap-server localhost:29092,localhost:29093,localhost:29094 \
    --list

Failing over from main to failover​

Failing over from main to failover on gateway gateway1

curl \
  --request POST 'http://localhost:8888/admin/pclusters/v1/pcluster/main/switch?to=failover' \
  --user 'admin:conduktor' \
  --silent | jq

Failing over from main to failover​

Failing over from main to failover on gateway gateway2

curl \
  --request POST 'http://localhost:8889/admin/pclusters/v1/pcluster/main/switch?to=failover' \
  --user 'admin:conduktor' \
  --silent | jq

Produce alice into users, it should hit only failover-kafka​

Producing 1 message in users in cluster teamA

Sending 1 event

{
  "name" : "alice",
  "username" : "alice@conduktor.io",
  "password" : "youpi",
  "visa" : "#812SSS",
  "address" : "Les ifs"
}

with

echo '{"name":"alice","username":"alice@conduktor.io","password":"youpi","visa":"#812SSS","address":"Les ifs"}' | \
    kafka-console-producer \
        --bootstrap-server localhost:6969 \
        --producer.config teamA-sa.properties \
        --topic users

Verify we can read laura (via mirror maker), tom (via mirror maker) and alice (via cluster switching)​

Verify we can read laura (via mirror maker), tom (via mirror maker) and alice (via cluster switching) in cluster teamA

kafka-console-consumer \
    --bootstrap-server localhost:6969 \
    --consumer.config teamA-sa.properties \
    --topic users \
    --from-beginning \
    --max-messages 3 \
    --timeout-ms 10000 | jq

returns 1 event

{
  "name" : "alice",
  "username" : "alice@conduktor.io",
  "password" : "youpi",
  "visa" : "#812SSS",
  "address" : "Les ifs"
}

Verify alice is not in main kafka​

Verify alice is not in main kafka in cluster kafka1

kafka-console-consumer \
    --bootstrap-server localhost:19092,localhost:19093,localhost:19094 \
    --topic teamAusers \
    --from-beginning \
    --timeout-ms 10000 | jq

returns 2 events

{
  "name" : "tom",
  "username" : "tom@conduktor.io",
  "password" : "motorhead",
  "visa" : "#abc123",
  "address" : "Chancery lane, London"
}
{
  "name" : "laura",
  "username" : "laura@conduktor.io",
  "password" : "kitesurf",
  "visa" : "#888999XZ",
  "address" : "Dubai, UAE"
}

Verify alice is in failover​

Verify alice is in failover in cluster failover-kafka1

kafka-console-consumer \
    --bootstrap-server localhost:29092,localhost:29093,localhost:29094 \
    --topic teamAusers \
    --from-beginning \
    --max-messages 3 \
    --timeout-ms 15000 | jq

returns 1 event

{
  "name" : "alice",
  "username" : "alice@conduktor.io",
  "password" : "youpi",
  "visa" : "#812SSS",
  "address" : "Les ifs"
}

Tearing down the docker environment​

Remove all your docker processes and associated volumes

  • --volumes: Remove named volumes declared in the "volumes" section of the Compose file and anonymous volumes attached to containers.
docker compose down --volumes

Conclusion

Cluster switching help your seamlessly move from one cluster to another!

Quick nav