Overview
Conduktor supports creating Partner Zones against a Gateway configured with multiple clusters. This guide will walk you through the process including the steps required to set up Gateway and Console.Prerequisites
To have Gateway set up with multi-cluster support, you will first need to create multiple underlying Kafka clusters. Once you have the bootstrap servers for each of the kafka clusters, you can then configure Gateway to point to them.1. Set up your Gateway with multi-cluster support
Let’s say that you have one main Gateway clustergateway-main
and one underlying Kafka cluster clusterA
.
You’ll first need to configure the Gateway to use the multiple clusters. Create a file cluster-config.yml
with the configuration for the clusters you have created, like so:
cluster-config.yaml
cluster-config.yaml
file in the Gateway container using the configuration GATEWAY_BACKEND_KAFKA_SELECTOR
:
2. Configure the underlying Kafka clusters in Console
Once you have the Gateway configured with multiple clusters, you can then configure the underlying Kafka clusters in Console. This step is necessary to ensure that the Kafka clusters are recognized by Console to be available when creating a Partner Zone.- Using Console UI
- Using Conduktor Configuration File
In our case, we have
clusterA
as the underlying Kafka cluster. You can configure it in Console:- Go to the Clusters page in Console, and click on Add Cluster.
- Fill in the details for
clusterA
, e.g.:- Technical Id:
clusterA
- Bootstrap Servers:
<clusterA_kafka_bootstrap_server>:9092
- Additionally, provide the security settings if required, such as:
- Security Protocol:
SASL_SSL
- SASL Mechanism:
PLAIN
- SASL JAAS Config:
org.apache.kafka.common.security.plain.PlainLoginModule required username="<clusterA_api_key>" password="<clusterA_api_secret>";
- Security Protocol:
- Technical Id:
3. Create a Partner Zone
Assuming that we have agateway-main
Gateway cluster and a clusterA
Kafka cluster, we can create a Partner Zone that connects to the clusterA
Kafka cluster as the backing cluster.
- Using Console UI
- Using Conduktor CLI
- Go to the Partner Zones page in Console.
- Click on Create Partner Zone.
- Select Gateway to choose the gateway to use i.e.
gateway-main
. - Select a cluster to choose an available underlying cluster i.e.
clusterA
. - Select the topics you want to include in the Partner Zone, and optionally create any topic aliases.
- Fill in and complete the remaining steps to create the Partner Zone.
- Finally click Create to create the Partner Zone.
4. Check Partner Zone status
Once the Partner Zone is created, you can check its status and manage it through the Console UI or the Conduktor CLI. With the Conduktor CLI, you can use the following command to check the status of the Partner Zone:metadata status
can have one of these values:
- Pending: the configuration isn’t deployed or refreshed yet
- Ready: the configuration is up to date on Gateway
- Failed: something unexpected happened during the creation