Connect to a Kafka Cluster
The ability to reach your cluster will depend on where you have setup the Testing Agent.
Adding a cluster
To add a Kafka cluster, navigate to the Clusters tab from within your Workspace.
Select Create New and you will be presented with the cluster configuration modal.
You will be required to input the following information:
- Cluster name: This will enable you to reference your cluster when using it in test scenarios.
- Bootstrap servers: The list of host and port pairs used to establish the initial connection to the Kafka cluster
- Additional properties: Additional properties that you would usually provide your CLI or Java clients. This is especially important if you have a secure Kafka cluster.
Note it's possible to paste the list of additional properties by toggling Switch view.
When you have configured the cluster properties, click refresh to check the connection status.
When you configure a cluster, you also have the option to configure Schema Registry. Conduktor supports the Avro, JSON and Protobuf formats.
Select the Schema registry tab from within the cluster configuration modal to get started.
- Schema Registry URL: HTTP or HTTPS endpoint of your schema registry
- Authentication: Select your authentication method (None, Basic or Bearer token)
- Additional Properties: Any additional properties required for your connection. For example, properties that define your SSL truststore/keystore location and password.
Connect to a Local Kafka cluster
You are able to connect to a Kafka cluster that is running on your localhost. For example, a cluster that has been started via Docker, or via the Kafka binaries.
The only pre-requisite is that the Testing Agent is running on a host that has access to your cluster.
To connect to Kafka running on localhost:
- Ensure your agent is running, and selected via the dropdown in the navigation
- Provide the host and port like any other cluster, for example: localhost:9092
Connect to a secure Kafka cluster
Conduktor leverages the default Apache Kafka Java Clients, and therefore we use the same configuration properties.
When the Conduktor Testing application needs to connect to a secure Kafka cluster, you must specify the values from your
For more detail, visit the docs for our Conduktor Desktop product.
Connect to a Confluent cluster
Via your Confluent cluster dashboard, select the Clients tab within Data integration.
Select Java as the language.
Create the Kafka cluster API key. You also have the option to create the Schema Registry API Key if you are using Schema Registry.
Copy the configuration to your clipboard.
From within the cluster configuration modal of the Testing application, Paste your configuration as additional properties.
Refresh the connection and if successful, you will see the CONNECTED label turn green.
Click Create Cluster to save your cluster.
Connect to an Aiven cluster
To connect to an Aiven cluster, you must specify the Access Key, Access Certificate, and CA Certificate.
To obtain these files, go to:
https://console.aiven.io/project/\<project>/services/\<cluster> \ \ From within the cluster configuration modal, add the Bootstrap server. This is labelled Service URI within the Aiven console.
Using the additional properties, reference the absolute path of the files downloaded from the Aiven console.
ssl.keystore.location = /path/to/client.keystore.p12
It's important that the Testing Agent is running on a host with access to these files.
Connect to a Red Hat cluster
Select Connection from your Red Hat Kafka Instance
From within the slide-out menu, choose Create service account
Copy the credentials that are generated in the subsequent modal.
Create a cluster inside Conduktor Testing using the below additional properties.
Note you must replace the clientId and clientSecret with the service account credentials that were generated in the previous step.
sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required clientId="YOUR-CLIENT-ID" clientSecret="YOUR-CLIENT-SECRET" ssl.protocol="SSL" ;
Connect to a MSK cluster
For connecting to MSK, we recommend installing the Testing Agent on an EC2 instance with access to the cluster.
When you create an Agent inside the Testing UI, you will be provided the commands for downloading and running it.
Execute these commands on your EC2 instance, and you should see the message:
Agent <agent name> connected!
From within the Testing UI, you should now be able to see that the Agent is connected:
Navigate to the Clusters tab, and Create a new cluster.
- Add the bootstrap servers
- For MSK, you can find this by selecting the Cluster within AWS > MSK and clicking View client information
- Add any required configuration properties