Skip to main content
Quick navigation

Managing Clusters

The Admin section of Conduktor Platform enables you to add, update, and delete Kafka clusters. By default, only users with Admin permissions will be able to view and perform actions relating to Cluster management. You will not be able to add clusters beyond your current organization limit; please use the chat box below or contact us to discuss an increase to this limit.

Cluster admin

To create a new cluster, click on the "Create cluster" button in the top right-hand corner, then follow the instructions from the Get Started section.

To edit an existing cluster, select it from the list. You will then be able to adjust the cluster name & color, technical ID, bootstrap servers, and additional properties. You can also enable Schema Registry and Kafka Connect for the cluster.

Schema Registry​

To enable Schema Registry, you will need to have it setup for your cluster. This will depend on whether you are using open source Kafka or a managed Kafka service from the likes of Confluent, Amazon MSK, or Aiven.

You will then need to obtain the address of your Schema Registry, as well as any authentication details.

Kafka Connect​

Setting up Kafka Connect works similarly to Schema Registry; you will need to enable it for your clusters first, then locate the Kafka Connect URL and authentication for entry into Conduktor Platform. Below is an example using Confluent Cloud

Confluent Cloud Kafka Connect​

Generate an API key​

You need to generate an API key and secret to act as username and password specifically for Kafka Connect. Do not use the Kafka credentials you already have, these won't work. This can be done using the Confluent Cloud CLI tool: ccloud. Install it using the official documentation: https://docs.confluent.io/ccloud-cli/current/install.html

  • Login and select your environment if you have several of them:
$ ccloud login
$ ccloud environment
  • Generate a key: write down the key & secret:
$ ccloud api-key create --resource cloud
+---------+------------------------------------------------------------------+
| API Key | ABCDEFKZBF56666                                                  |
| Secret  | ToMaHaWkjQ1bt7BxvdyFjaJ8j3nSokaAd83Nhan739snAiufIAfdk7fFAAnBKxai |
+---------+------------------------------------------------------------------+
  • The API Key is the username, the Secret is the token

Kafka Connect URL​

  • Get the ID of the environment where your cluster is located:
$ ccloud environment list                                                                                                                                                                                   16:49:31
      Id      |    Name
+-------------+------------+
    env-1234 | staging
  * env-6789 | production
  • Get the ID of the Kafka cluster :
 ccloud environment use env-6789
 ccloud kafka cluster list                                                                    16:21:10
      Id      |         Name         | Type  | Provider |    Region    | Availability | Status
+-------------+----------------------+-------+----------+--------------+--------------+--------+
    lkc-8888 | analytics-production | BASIC | gcp      | europe-west4 | single-zone  | UP

  • Then build the Kafka connection URL for this env-id cluster-id couple
https://api.confluent.cloud/connect/v1/environments/env-6789/clusters/lkc-8888/

Configure your cluster using this API key​

From the Kafka Connect tab of your cluster, configure with these elements, selecting Basic Auth to add the username/password. Click "Test connection" to ensure that everything is working, and you're done!