Skip to main content
Quick navigation

Configuration Snippets

danger

Cluster configuration from YAML have several limitation you should be aware of:

  • This it not GitOps compatible (changes after the initial deployment will not be reflected)
  • No Audit log events send on cluster configuration
  • No SSL authentication support on Schema Registry and Kafka Connect
  • Limited AWS Glue support
  • Certificates defined in properties or with CDK_SSL_TRUSTSTORE_PATH are not stored in the Conduktor Platform Certificate Store and cannot be updated from the UI

The new recommended way to configure Kafka Cluster, Schema Registry and Kafka Connect is using Conduktor Platform UI.

The Manage Clusters page (/admin/clusters) has several advantages over the YAML configuration:

  • Intuitive interface with live update capabilities
  • Centralized and secured with RBAC and Audit Logs Events
  • Certificate store to help with your Custom certificates needs (no more JKS files and volume mounts)

Need to configure your Kafka Clusters using GitOps processes?
Contact our Customer Success or give us feedback on this feature.

If you absolutely need to configure your clusters using YAML, keep reading.

Below outlines snippets demonstrating different configuration options for the platform-config.yaml.

Jump to:

Complete Configuration Example

This demonstrates a complete configuration for Conduktor Enterprise consisting of two Kafka clusters with Schema Registry, SSO and license key.

organization:
name: conduktor

admin:
email: admin@company.io
password: admin

clusters:
- id: my-local-kafka-cluster
name: My Local Kafka Cluster
color: '#0013E7'
bootstrapServers: 'my-bootstrap-server:9092'
schemaRegistry:
id: my-schema-registry
url: 'http://my-schema-registry:8081'

- id: confluent-pkc
name: Confluent pkc-lq8v7
color: '#E70000'
bootstrapServers: 'pkc-lq8v7.eu-central-1.aws.confluent.cloud:9092'
properties: |
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<username>" password="<password>";
schemaRegistry:
id: confluent-sr
url: 'https://psrc-o268o.eu-central-1.aws.confluent.cloud'
security:
username: <username>
password: <password>
labels: {}

sso:
oauth2:
- name: 'auth0'
default: true
client-id: <client_id>
client-secret: <client_secret>
callback-uri: http://localhost/auth/oauth/callback/auth0
openid:
issuer: https://conduktor-staging.eu.auth0.com/

license: '<license_key>'

Plain Auth Example

Connect to a local cluster with no auth/encryption, for example a local development Kafka

clusters:
- id: local
name: My Local Kafka Cluster
color: '#0013E7'
ignoreUntrustedCertificate: false
bootstrapServers: 'localhost:9092'

Plain Auth With Schema Registry

Connect to a local cluster with schema registry

clusters:
- id: local
name: My Local Kafka Cluster
color: '#0013E7'
bootstrapServers: 'localhost:9092'
schemaRegistry:
id: Local SR
url: 'http://localhost:8081'

Amazon MSK with IAM Authentication Example

Connect to an MSK cluster with IAM Authentication using AWS Access Key and Secret

Billing note

Note that deploying this CloudFormation template into your environment will result in billable resources being consumed. See Amazon MSK pricing for more information.

clusters:
- id: amazon-msk-iam
name: Amazon MSK IAM
color: #FF9900
bootstrapServers: 'b-3-public.****.kafka.eu-west-1.amazonaws.com:9198'
properties: |
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=io.conduktor.aws.IAMClientCallbackHandler
aws_access_key_id=<access-key-id>
aws_secret_access_key=<secret-access-key>

Connect to an MSK cluster with IAM credentials inherited from environment

clusters:
- id: amazon-msk-iam
name: Amazon MSK IAM
color: #FF9900
bootstrapServers: 'b-3-public.****.kafka.eu-west-1.amazonaws.com:9198'
properties: |
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler

On top of that, you can override either the default profile or the role to assume. Override Profile

sasl.jaas.config = software.amazon.msk.auth.iam.IAMLoginModule required awsProfileName="other-profile";

Override Role

sasl.jaas.config = software.amazon.msk.auth.iam.IAMLoginModule required awsRoleArn="arn:aws:iam::123456789012:role/msk_client_role";

Amazon MSK with Glue Schema Registry

Connect to an MSK cluster with schema registry using credentials

clusters:
- id: amazon-msk-iam
name: Amazon MSK IAM
color: #FF9900
bootstrapServers: 'b-3-public.****.kafka.eu-west-1.amazonaws.com:9198'
properties: |
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=io.conduktor.aws.IAMClientCallbackHandler
aws_access_key_id=<access-key-id>
aws_secret_access_key=<secret-access-key>
schemaRegistry:
region: <aws-region>
security:
type: Credentials
accessKeyId: <access-key-id>
secretKey: <secret-key>

Connect to an MSK cluster with schema registry using the default chain of credentials providers

clusters:
- id: amazon-msk-iam
name: Amazon MSK IAM
color: #FF9900
bootstrapServers: 'b-3-public.****.kafka.eu-west-1.amazonaws.com:9198'
properties: |
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=io.conduktor.aws.IAMClientCallbackHandler
aws_access_key_id=<access-key-id>
aws_secret_access_key=<secret-access-key>
schemaRegistry:
region: <aws-region>
security:
type: FromContext
profile: <profile> # optional to use the default profile

Connect to an MSK cluster with schema registry using a specific role

clusters:
- id: amazon-msk-iam
name: Amazon MSK IAM
color: #FF9900
bootstrapServers: 'b-3-public.****.kafka.eu-west-1.amazonaws.com:9198'
properties: |
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=io.conduktor.aws.IAMClientCallbackHandler
aws_access_key_id=<access-key-id>
aws_secret_access_key=<secret-access-key>
schemaRegistry:
region: <aws-region>
security:
type: FromRole
role: <role>

On top of that, and for all these previous configuration example, you can add a registryName to the schemaRegistry section to use a specific registry for this cluster.

schemaRegistry:
region: <aws-region>
security:
type: Credentials
accessKeyId: <access-key-id>
secretKey: <secret-key>
registryName: <registry-name>

Confluent Cloud Example

Connect to a confluent cloud cluster using API keys

clusters:
- id: confluent-pkc
name: Confluent pkc-lzoyy
color: '#E70000'
bootstrapServers: 'pkc-lzoyy.eu-central-1.aws.confluent.cloud:9092'
properties: |
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<username>" password="<password>";

Confluent Cloud with Schema Registry

Connect to a confluent cloud cluster with schema registry using basic auth

- id: confluent-pkc
name: Confluent pkc-lq8v7
color: '#E70000'
bootstrapServers: 'pkc-lq8v7.eu-central-1.aws.confluent.cloud:9092'
properties: |
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<usernam>" password="<password>";
schemaRegistry:
id: confluent-sr
url: 'https://psrc-o268o.eu-central-1.aws.confluent.cloud'
security:
username: <username>
password: <password>

SSL Certificates Example - Aiven (truststore)

Keystore and truststore are not supported. But you can directly use the PEM formatted files (.pem or .cer)
Aiven example providing inline CA certificate
Please make sure the certificate is on one single line

- id: aiven-stg
name: My Aiven Cluster
bootstrapServers: 'kafka-09ba.aivencloud.com:21661'
properties: |
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="<username>" password="<password>";
ssl.truststore.type=PEM
ssl.truststore.certificates=-----BEGIN CERTIFICATE----- <YOUR CA CERTIFICATE> -----END CERTIFICATE-----

2 Way SSL (keystore + truststore)

You should have 3 files, and generally they are embedded in 2 files:

  • Your access key (in the keystore.jks file)
  • Your access certificate (in the keystore.jks file)
  • Your CA certificate (in the truststore.jks file) Please make sure to have the content is on a single line
- id: aiven-ssl
name: Aiven SSL
bootstrapServers: kafka-09ba.aivencloud.com:21650
properties: |
security.protocol=SSL
ssl.truststore.type=PEM
ssl.truststore.certificates=-----BEGIN CERTIFICATE----- <YOUR CA CERTIFICATE> -----END CERTIFICATE-----
ssl.keystore.type=PEM
ssl.keystore.key=-----BEGIN PRIVATE KEY----- <YOUR ACCES KEY> -----END PRIVATE KEY-----
ssl.keystore.certificate.chain=-----BEGIN CERTIFICATE----- <YOUR ACCESS CERTIFICATE> -----END CERTIFICATE-----

Kafka Connect

Cluster with Kafka Connect configured with Basic Auth

- id: cluster-connect
name: My Kafka With Connect
color: #C90000
bootstrapServers: '{Bootstrap Servers}'
properties:
kafkaConnects:
- url: '{Kafka Connect URL}'
id: kafka-connect
name: kafkConnect
security:
username: <username>
password: <password>