platform-config.yaml or environment variables.
To set up your organization’s environment:
- Enter the organization’s name.
- Connect to an external database (required).
- Set up user authentication (basic or SSO).
- Apply your Console license.
We recommend using the Console UI (Settings > Clusters page) to configure Kafka cluster, schema registry and Kafka connect. This has several advantages over the YAML configuration:
- Intuitive interface with live update capabilities
- Centralized and secured with and audit logs events
- Certificate store to help with custom certificates configuration (no more JKS files and volume mounts)
Security considerations
- The configuration file should be protected by file system permissions.
- The database should have at-rest data encryption enabled on the data volume and have limited network connectivity.
Configuration file
platform-config.yaml
Bind file
Thedocker-compose below shows how to bind your platform-config.yaml file.
You can alternatively use environment variables. The CDK_IN_CONF_FILE variable is used to indicate that a configuration file is being used and the location to find it.
docker-compose.yaml
Environment override
Input configuration fields can also be provided using environment variables. Here’s an example ofdocker-compose that uses environment variables for configuration:
"docker-compose.yaml
Container user and permissions
Console is running as a non-root userconduktor-platform with UID 10001 and GID 0. All files inside the container volume /var/conduktor are owned by conduktor-platform user.
Configure memory usage
We rely on container CGroups limits and use up to 80% of the container memory limit for JVM max heap size.- Console helm
- Kubernetes
- Docker Compose
Configure SSL or TLS
Depending on the environment, Conduktor might need to access external services (such as Kafka clusters, SSO servers, databases or object storage) that require a custom certificate for SSL/TLS communication. You can configure this using:- Console UI (recommended) - you can manage your certificates in a dedicated screen and configure SSL authentication from the broker setup wizard.
- volume mount - this method is only required if you have LDAPS. Do not use it for Kafka or Kafka components.
Use the Conduktor certificate store
This option is recommended for Kafka, Kafka Connect and Schema Registry connections.
- .crt
- .pem
- .jks
- .p12
Upload certificates
You can add cluster configurations from Settings > Clusters page. When you add the bootstrap server to your configuration, a check will be made to validate if the certificate is issued by a valid authority. If the response indicates the certificate is not issued by a valid authority, you have two options:- Skip SSL Check: This will skip validation of the SSL certificate on your server. This is an easy option for development environments with self-signed certificates
- Upload Certificate: This option will enable you to upload the certificate (
.crt,.pem,.jksor.p12files), or paste the certificate as text


Add truststores
You can also manage organization truststores using the Settings > Certificates page. Simply add all of your certificates by uploading them or pasting them as text. In doing this, the SSL context will be derived when you configure Kafka, Kafka Connect and Schema Registry connections.
Mount custom truststore
This option is recommended for SSO, DB or other external services requiring SSL/TLS communication.
Create TrustStore (JKS) from certificate in PEM format
If you already have a truststore, you can ignore this step. You need akeytool program that is usually packaged on JDK distributions and a certificate in PEM format (.pem or .crt).
Configure custom truststore via Conduktor Console
Mount the truststore file into theconduktor-console container and pass the correct environment variables for locating truststore file inside the container (and password, if needed).
If the truststore file is truststore.jks with password changeit, mount truststore file into /opt/conduktor/certs/truststore.jks inside the container.
If run from Docker :
Client certificate authentication
This option is recommended for mTLS.
- Mutual SSL, Mutual TLS, mTLS
- Two-Way SSL, SSL Certificate Authentication
- Digital Certificate Authentication, Public Key Infrastructure (PKI) Authentication
Use the UI (keystore method)
Use the keystore file from your Kafka admin or provider (in .jks or .p12 format). Click the “Import from keystore” button to select a keystore file from your filesystem.


Use the UI (Access key and access certificate method)
Your Kafka admin or your Kafka provider gave you 2 files for authentication.- An Access key (
.keyfile) - An Access certificate (
.pemor.crtfile)

Use volume mount
You can mount the keystore file in theconduktor-console image:

Configure Postgres database
Conduktor Console requires a Postgres database to store its state.Postgres requirements
- Postgres version 13 or higher
- Provided connection role should have grant
ALL PRIVILEGESon the configured database. Console should be able to create/update/delete schemas and tables on the database.
- 1-2 vCPU, 1 GB of RAM, and 10 GB of disk
Database configuration properties
database: is a key/value configuration consisting of:database.url: database connection url in the format[jdbc:]postgresql://[user[:password]@][[netloc][:port],...][/dbname][?param1=value1&...]database.hosts[].host: Postgresql server hosts namedatabase.hosts[].port: Postgresql server portsdatabase.host: Postgresql server host name (Deprecated. Usedatabase.hostsinstead)database.port: Postgresql server port (Deprecated. Usedatabase.hostsinstead)database.name: Database namedatabase.username: Database login roledatabase.password: Database login passworddatabase.connection_timeout: Connection timeout option in seconds
URL format
Console supports both, the standard PostgreSQL URL and JDBC PostgreSQL . Connection username and password can be provided in the URL as basic authentication or as parameters.SSL support
By default, Console will try to connect to the database using SSL modeprefer. We plan to make this configurable in the future along with database certificate.
Setup
There are several options available when configuring an external database:-
From a single connection URL
- With the
CDK_DATABASE_URLenvironment variable. - With the
database.urlconfiguration field. In either case, this connection url is using a standard PostgreSQL url in the format[jdbc:]postgresql://[user[:password]@][[netloc][:port],...][/dbname][?param1=value1&...]
- With the
-
From decomposed configuration fields
- With the
CDK_DATABASE_*env vars. - With the
database.*on configuration file.
- With the
Example
- If all connection URLs and decomposed configuration fields are provided, the decomposed configuration fields take priority.
- If an invalid connection URL or a mandatory configuration field (
host,usernameorname) is missing, Conduktor will fail gracefully with a meaningful error message. - Before Console v1.2.0, the
EMBEDDED_POSTGRES=falsewas mandatory to enable external Postgresql configuration.
Multi-host configuration
If you have a multi-host setup, you can configure the database connection with a list of hosts. Conduktor uses a PostgreSQL JDBC driver to connect to the database that supports multiple hosts in the connection url . To configure a multi-host setup, you can use thedatabase.url configuration field with a list of hosts separated by commas:
targetServerType to specify the target server type for the connection:
targetServerType are supported: any, primary, master, slave, secondary, preferSlave, preferSecondary and preferPrimary.
Debug Console
Conduktor Console Docker image runs on Ubuntu Linux. It runs multiple services in a single Docker container. These services are supervised by supervisord . To troubleshoot Console:- Verify that Console is up and running.
- Check the logs and send them to our support team if necessary.
1. Verify that Conduktor is up and running
- From Docker
- Kubernetes
First, verify that all the components are running.If you’re using an external Kafka installation and external database, you will only need to verify that the You can save these logs in a file:
Get containers status
Output
conduktor-console container is showing healthy as the STATUS.If Console is showing an “exited” status, check the Docker logs by running the command (with the appropriate container name):Get container logs
Store logs in a file
2. Get the logs and send them to support
Logs are kept in/var/conduktor/log. You can see them using:
List log files
Output
Configuration properties and environment variables
For the complete list of variables, see the Console environment variables reference.Docker image environment variables
Console properties reference
You have multiple options to configure Console: via environment variables, or via a YAML configuration file. You can find a mapping of the configuration fields in theplatform-config.yaml to environment variables below.
Environment variables can be set on the container or imported from a file. When importing from a file, mount the file into the container and provide its path by setting the environment variable CDK_ENV_FILE. Use a .env file with key value pairs.
Sourcing environment variables from $CDK_ENV_FILE, or warn if set and the file is not found
Lists start at index 0 and are provided using
_idx_ syntax.YAML property cases
YAML configuration supports multiple case formats (camelCase/kebab-case/lowercase) for property fragments such as:
clusters[].schemaRegistry.ignoreUntrustedCertificateclusters[].schema-registry.ignore-untrusted-certificateclusters[].schemaregistry.ignoreuntrustedcertificate
Environment variable conversion
At startup, Conduktor Console will merge environment variables and YAML based configuration files into one unified configuration. The conversion rules are:- Filter for environment variables that start with
CDK_ - Remove the
CDK_prefix - Convert the variable name to lowercase
- Replace
_with.for nested properties - Replace
_[0-9]+_with[0-9].for list properties. (Lists start at index 0)
CDK_DATABASE_URL will be converted to database.url, or CDK_SSO_OAUTH2_0_OPENID_ISSUER will be converted into sso.oauth2[0].openid.issuer.
The YAML equivalent would be:
UPPER-KEBAB-CASE will be converted to kebab-case in the YAML configuration.
Conversion edge cases
Because of YAML multiple case formats support, the conversion rules have some edge cases when trying to mix environment variables and YAML configuration. Extra rules when mixing environment variables and YAML configuration:- Don’t use
camelCasein YAML configuration. Usekebab-caseorlowercase - Stick to one compatible case format for a given property fragment using the following compatibility matrix
For example,
CDK_CLUSTERS_0_SCHEMAREGISTRY_IGNOREUNTRUSTEDCERTIFICATE environment variable:
CDK_CLUSTERS_0_SCHEMA-REGISTRY_IGNORE-UNTRUSTED-CERTIFICATE, that’s why camelCase is not recommended in YAML configuration when mixing with environment variables.
Support of shell expansion in the YAML configuration file
Console supports shell expansion for environment variables and home tilde~. This is useful if you have to use custom environment variables in your configuration.
For example, you can use the following syntax:
YAML configuration file
This will be expanded to:
Expanded configuration
$$. For example, if you want admin.password to be secret$123, you should set admin.password: "secret$$123".
File path environment variables
When an environment variable ending with_FILE is set to a file path, its corresponding unprefixed environment variable will be replaced with the content of that file.
For example, if you set CDK_LICENSE_FILE=/run/secrets/license, the value of CDK_LICENSE will be overridden by the content of the file located at /run/secrets/license.
Global properties
Database properties
See database configuration for details.Session lifetime properties
Local users properties
Optional local account list used to log into Console.Monitoring properties
To see monitoring graphs and use alerts, you have to ensure that Cortex is also deployed.Monitoring Configuration for Console
First, we need to configure Console to connect to Cortex services. By default, Cortex ports are:- Query port: 9009
- Alert manager port: 9010
monitoring.use-aggregated-metrics and monitoring.enable-non-aggregated-metrics are temporary flags to help you transition to the new metrics collection system. They will be removed in a future release.Swap their default value if you experience performance issues when Console is connected with large Kafka clusters:Monitoring configuration for Cortex
See Cortex configuration for details.SSO properties
LDAP properties
OAuth2 properties
JWT auth properties
Kafka cluster properties
Kafka vendor specific properties
Note that you only need to set the Kafka cluster properties to use the core features of Console. However, you can get additional benefits by setting the flavor of your cluster. This corresponds to theProvider tab of your cluster configuration in Console.
Schema registry properties
Amazon Glue schema registry properties
Kafka Connect properties
ksqlDB properties
We support ksqlDB integration as of Conduktor Console v1.21.0.AuditLog export properties
The audit log can be exported to a Kafka topic, once configured in Console.Conduktor SQL properties
In order to use Conduktor SQL, you need to configure a second database to store the topics data. You can configure Conduktor SQL Database usingCDK_KAFKASQL_DATABASE_URL or set each value individually with CDK_KAFKASQL_DATABASE_*.
Configure SQL to get started.
Advanced properties:
Partner Zones properties
Advanced configuration for Partner Zones.Configure HTTP proxy
Specify the proxy settings for Conduktor to use when accessing Internet. The HTTP proxy works for both HTTP and HTTPS connection. There are five properties you can set to specify the proxy that will be used by the HTTP protocol handler:CDK_HTTP_PROXY_HOST: the host name of the proxy serverCDK_HTTP_PROXY_PORT: the port number. Default value is 80.CDK_HTTP_NON_PROXY_HOSTS: a list of hosts that should be reached directly, bypassing the proxy. This is a list of patterns separated by|. The patterns may start or end with a*for wildcards, we do not support/. Any host matching one of these patterns will be reached through a direct connection instead of through a proxy.CDK_HTTP_PROXY_USERNAME: the proxy usernameCDK_HTTP_PROXY_PASSWORD: the proxy password
Example
Configure HTTPS
To configure Conduktor Console to respond to HTTPS requests, you have to define a certificate and a private key. The server certificate is a public entity that’s sent to every client that connects to the server and it should be provided as a PEM file. Configuration properties are:platform.https.cert.pathor environment variableCDK_PLATFORM_HTTPS_CERT_PATH: the path to server certificate fileplatform.https.key.pathor environment variableCDK_PLATFORM_HTTPS_KEY_PATH: the path to server private key file
Both the certificate and private key files have to allow read from user
conduktor-platform (UID 10001 GID 0) but don’t need to be readable system-wide.Sample configuration using docker-compose
In this example, server certificate and key (server.crt and server.key) are stored in the same directory as thedocker-compose file.
conduktor/conduktor-console-cortex is running as well, you have to provide the CA public certificate to the monitoring image to allow metrics scraping on HTTPS.