Configuration Snippets
Cluster configuration from YAML have several limitation you should be aware of:
- This it not GitOps compatible (changes after the initial deployment will not be reflected)
- No Audit log events send on cluster configuration
- No SSL authentication support on Schema Registry and Kafka Connect
- Limited AWS Glue support
- Certificates defined in properties or with CDK_SSL_TRUSTSTORE_PATH are not stored in the Conduktor Platform Certificate Store and cannot be updated from the UI
The new recommended way to configure Kafka Cluster, Schema Registry and Kafka Connect is using Conduktor Platform UI.
The Manage Clusters page (/admin/clusters
) has several advantages over the YAML configuration:
- Intuitive interface with live update capabilities
- Centralized and secured with RBAC and Audit Logs Events
- Certificate store to help with your Custom certificates needs (no more JKS files and volume mounts)
Need to configure your Kafka Clusters using GitOps processes?
Contact our Customer Success or give us feedback on this feature.
If you absolutely need to configure your clusters using YAML, keep reading.
Below outlines snippets demonstrating different configuration options for the platform-config.yaml
.
Jump to:
- Complete Configuration Example
- Plain Auth Example
- Plain Auth With Schema Registry
- Amazon MSK with IAM Authentication Example
- Amazon MSK with Glue Schema Registry
- Confluent Cloud Example
- Confluent Cloud with Schema Registry
- SSL Certificates Example - Aiven (truststore)
- 2 Way SSL (keystore + truststore)
- Kafka Connect
Complete Configuration Example
This demonstrates a complete configuration for Conduktor Enterprise consisting of two Kafka clusters with Schema Registry, SSO and license key.
organization:
name: conduktor
admin:
email: admin@company.io
password: admin
clusters:
- id: my-local-kafka-cluster
name: My Local Kafka Cluster
color: '#0013E7'
bootstrapServers: 'my-bootstrap-server:9092'
schemaRegistry:
id: my-schema-registry
url: 'http://my-schema-registry:8081'
- id: confluent-pkc
name: Confluent pkc-lq8v7
color: '#E70000'
bootstrapServers: 'pkc-lq8v7.eu-central-1.aws.confluent.cloud:9092'
properties: |
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<username>" password="<password>";
schemaRegistry:
id: confluent-sr
url: 'https://psrc-o268o.eu-central-1.aws.confluent.cloud'
security:
username: <username>
password: <password>
labels: {}
sso:
oauth2:
- name: 'auth0'
default: true
client-id: <client_id>
client-secret: <client_secret>
callback-uri: http://localhost/auth/oauth/callback/auth0
openid:
issuer: https://conduktor-staging.eu.auth0.com/
license: '<license_key>'
Plain Auth Example
Connect to a local cluster with no auth/encryption, for example a local development Kafka
clusters:
- id: local
name: My Local Kafka Cluster
color: '#0013E7'
ignoreUntrustedCertificate: false
bootstrapServers: 'localhost:9092'
Plain Auth With Schema Registry
Connect to a local cluster with schema registry
clusters:
- id: local
name: My Local Kafka Cluster
color: '#0013E7'
bootstrapServers: 'localhost:9092'
schemaRegistry:
id: Local SR
url: 'http://localhost:8081'
Amazon MSK with IAM Authentication Example
Connect to an MSK cluster with IAM Authentication using AWS Access Key and Secret
Billing note
Note that deploying this CloudFormation template into your environment will result in billable resources being consumed. See Amazon MSK pricing for more information.
clusters:
- id: amazon-msk-iam
name: Amazon MSK IAM
color: #FF9900
bootstrapServers: 'b-3-public.****.kafka.eu-west-1.amazonaws.com:9198'
properties: |
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=io.conduktor.aws.IAMClientCallbackHandler
aws_access_key_id=<access-key-id>
aws_secret_access_key=<secret-access-key>
Connect to an MSK cluster with IAM credentials inherited from environment
clusters:
- id: amazon-msk-iam
name: Amazon MSK IAM
color: #FF9900
bootstrapServers: 'b-3-public.****.kafka.eu-west-1.amazonaws.com:9198'
properties: |
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
On top of that, you can override either the default
profile or the role to assume.
Override Profile
sasl.jaas.config = software.amazon.msk.auth.iam.IAMLoginModule required awsProfileName="other-profile";
Override Role
sasl.jaas.config = software.amazon.msk.auth.iam.IAMLoginModule required awsRoleArn="arn:aws:iam::123456789012:role/msk_client_role";
Amazon MSK with Glue Schema Registry
Connect to an MSK cluster with schema registry using credentials
clusters:
- id: amazon-msk-iam
name: Amazon MSK IAM
color: #FF9900
bootstrapServers: 'b-3-public.****.kafka.eu-west-1.amazonaws.com:9198'
properties: |
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=io.conduktor.aws.IAMClientCallbackHandler
aws_access_key_id=<access-key-id>
aws_secret_access_key=<secret-access-key>
schemaRegistry:
region: <aws-region>
security:
type: Credentials
accessKeyId: <access-key-id>
secretKey: <secret-key>
Connect to an MSK cluster with schema registry using the default chain of credentials providers
clusters:
- id: amazon-msk-iam
name: Amazon MSK IAM
color: #FF9900
bootstrapServers: 'b-3-public.****.kafka.eu-west-1.amazonaws.com:9198'
properties: |
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=io.conduktor.aws.IAMClientCallbackHandler
aws_access_key_id=<access-key-id>
aws_secret_access_key=<secret-access-key>
schemaRegistry:
region: <aws-region>
security:
type: FromContext
profile: <profile> # optional to use the default profile
Connect to an MSK cluster with schema registry using a specific role
clusters:
- id: amazon-msk-iam
name: Amazon MSK IAM
color: #FF9900
bootstrapServers: 'b-3-public.****.kafka.eu-west-1.amazonaws.com:9198'
properties: |
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=io.conduktor.aws.IAMClientCallbackHandler
aws_access_key_id=<access-key-id>
aws_secret_access_key=<secret-access-key>
schemaRegistry:
region: <aws-region>
security:
type: FromRole
role: <role>
On top of that, and for all these previous configuration example,
you can add a registryName
to the schemaRegistry
section to use a specific registry for this cluster.
schemaRegistry:
region: <aws-region>
security:
type: Credentials
accessKeyId: <access-key-id>
secretKey: <secret-key>
registryName: <registry-name>
Confluent Cloud Example
Connect to a confluent cloud cluster using API keys
clusters:
- id: confluent-pkc
name: Confluent pkc-lzoyy
color: '#E70000'
bootstrapServers: 'pkc-lzoyy.eu-central-1.aws.confluent.cloud:9092'
properties: |
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<username>" password="<password>";
Confluent Cloud with Schema Registry
Connect to a confluent cloud cluster with schema registry using basic auth
- id: confluent-pkc
name: Confluent pkc-lq8v7
color: '#E70000'
bootstrapServers: 'pkc-lq8v7.eu-central-1.aws.confluent.cloud:9092'
properties: |
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<usernam>" password="<password>";
schemaRegistry:
id: confluent-sr
url: 'https://psrc-o268o.eu-central-1.aws.confluent.cloud'
security:
username: <username>
password: <password>
SSL Certificates Example - Aiven (truststore)
Keystore and truststore are not supported. But you can directly use the PEM formatted files (.pem or .cer)
Aiven example providing inline CA certificate
Please make sure the certificate is on one single line
- id: aiven-stg
name: My Aiven Cluster
bootstrapServers: 'kafka-09ba.aivencloud.com:21661'
properties: |
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="<username>" password="<password>";
ssl.truststore.type=PEM
ssl.truststore.certificates=-----BEGIN CERTIFICATE----- <YOUR CA CERTIFICATE> -----END CERTIFICATE-----
2 Way SSL (keystore + truststore)
You should have 3 files, and generally they are embedded in 2 files:
- Your access key (in the keystore.jks file)
- Your access certificate (in the keystore.jks file)
- Your CA certificate (in the truststore.jks file) Please make sure to have the content is on a single line
- id: aiven-ssl
name: Aiven SSL
bootstrapServers: kafka-09ba.aivencloud.com:21650
properties: |
security.protocol=SSL
ssl.truststore.type=PEM
ssl.truststore.certificates=-----BEGIN CERTIFICATE----- <YOUR CA CERTIFICATE> -----END CERTIFICATE-----
ssl.keystore.type=PEM
ssl.keystore.key=-----BEGIN PRIVATE KEY----- <YOUR ACCES KEY> -----END PRIVATE KEY-----
ssl.keystore.certificate.chain=-----BEGIN CERTIFICATE----- <YOUR ACCESS CERTIFICATE> -----END CERTIFICATE-----
Kafka Connect
Cluster with Kafka Connect configured with Basic Auth
- id: cluster-connect
name: My Kafka With Connect
color: #C90000
bootstrapServers: '{Bootstrap Servers}'
properties:
kafkaConnects:
- url: '{Kafka Connect URL}'
id: kafka-connect
name: kafkConnect
security:
username: <username>
password: <password>