Skip to main content

​
Overview

Conduktor supports creating Partner Zones against a Gateway configured with multiple clusters. This guide will walk you through the process including the steps required to set up Gateway and Console.

​
Prerequisites

To have Gateway set up with multi-cluster support, you will first need to create multiple underlying Kafka clusters. Once you have the bootstrap servers for each of the kafka clusters, you can then configure Gateway to point to them.

​
1. Set up your Gateway with multi-cluster support

Let’s say that you have one main Gateway cluster gateway-main and one underlying Kafka cluster clusterA. You’ll first need to configure the Gateway to use the multiple clusters. Create a file cluster-config.yml with the configuration for the clusters you have created, like so:
cluster-config.yaml
config:
  main:
    bootstrap.servers: '<main_bootstrap_servers>:9092'
    security.protocol: 'SASL_SSL'
    sasl.mechanism: 'PLAIN'
    sasl.jaas.config: 'org.apache.kafka.common.security.plain.PlainLoginModule required username="<main_api_key>" password="<main_api_secret>";'
  clusterA:
    bootstrap.servers: '<clusterA_bootstrap_servers>:9092'
    security.protocol: 'SASL_SSL'
    sasl.mechanism: 'PLAIN'
    sasl.jaas.config: 'org.apache.kafka.common.security.plain.PlainLoginModule required username="<clusterA_api_key>" password="<clusterA_api_secret>";'
    gateway.roles: 'upstream'
Then, mount the cluster-config.yaml file in the Gateway container using the configuration GATEWAY_BACKEND_KAFKA_SELECTOR: 
GATEWAY_BACKEND_KAFKA_SELECTOR: 'file : { path: /cluster-config.yaml}'
Check out the following guide for additional information about Configuring Gateway with multi-clusters.

​
2. Configure the underlying Kafka clusters in Console

Once you have the Gateway configured with multiple clusters, you can then configure the underlying Kafka clusters in Console. This step is necessary to ensure that the Kafka clusters are recognized by Console to be available when creating a Partner Zone.
  • Using Console UI
  • Using Conduktor Configuration File
In our case, we have clusterA as the underlying Kafka cluster. You can configure it in Console:
  1. Go to the Clusters page in Console, and click on Add Cluster.
  2. Fill in the details for clusterA, e.g.:
    • Technical Id: clusterA
    • Bootstrap Servers: <clusterA_kafka_bootstrap_server>:9092
    • Additionally, provide the security settings if required, such as:
      • Security Protocol: SASL_SSL
      • SASL Mechanism: PLAIN
      • SASL JAAS Config: org.apache.kafka.common.security.plain.PlainLoginModule required username="<clusterA_api_key>" password="<clusterA_api_secret>";
Find out how to configure Kafka Clusters in Console.

​
3. Create a Partner Zone

Assuming that we have a gateway-main Gateway cluster and a clusterA Kafka cluster, we can create a Partner Zone that connects to the clusterA Kafka cluster as the backing cluster.
  • Using Console UI
  • Using Conduktor CLI
  1. Go to the Partner Zones page in Console.
  2. Click on Create Partner Zone.
  3. Select Gateway to choose the gateway to use i.e. gateway-main.
  4. Select a cluster to choose an available underlying cluster i.e. clusterA.
  5. Select the topics you want to include in the Partner Zone, and optionally create any topic aliases.
  6. Fill in and complete the remaining steps to create the Partner Zone.
  7. Finally click Create to create the Partner Zone.
Find out how to create a Partner Zone.

​
4. Check Partner Zone status

Once the Partner Zone is created, you can check its status and manage it through the Console UI or the Conduktor CLI. With the Conduktor CLI, you can use the following command to check the status of the Partner Zone: 
conduktor get PartnerZone
The metadata status can have one of these values:
  • PENDING: the configuration isn’t deployed or refreshed yet
  • READY: the configuration is up to date on Gateway
  • FAILED: something unexpected happened during the creation
  • TO_DELETE: the Partner Zone is being deleted asynchronously. This status is visible via CLI and API only.
Find out more about creating Partner Zones.
⌘I