Skip to main content
Conduktor can be deployed and configured using either a configuration file platform-config.yaml or environment variables. To set up your organization’s environment:
  1. Enter the organization’s name.
  2. Connect to an external database (required).
  3. Set up user authentication (basic or SSO).
  4. Apply your Console license.
We recommend using the Console UI (Settings > Clusters page) to configure Kafka cluster, schema registry and Kafka connect. This has several advantages over the YAML configuration:
  • Intuitive interface with live update capabilities
  • Centralized and secured with and audit logs events
  • Certificate store to help with custom certificates configuration (no more JKS files and volume mounts)
Check out the recommended deployment on GitHub .

Security considerations

  • The configuration file should be protected by file system permissions.
  • The database should have at-rest data encryption enabled on the data volume and have limited network connectivity.

Configuration file

platform-config.yaml

Bind file

The docker-compose below shows how to bind your platform-config.yaml file. You can alternatively use environment variables. The CDK_IN_CONF_FILE variable is used to indicate that a configuration file is being used and the location to find it.
docker-compose.yaml

Environment override

Input configuration fields can also be provided using environment variables. Here’s an example of docker-compose that uses environment variables for configuration:
"docker-compose.yaml

Container user and permissions

Console is running as a non-root user conduktor-platform with UID 10001 and GID 0. All files inside the container volume /var/conduktor are owned by conduktor-platform user.

Configure memory usage

We rely on container CGroups limits and use up to 80% of the container memory limit for JVM max heap size.
You only need to care about the limits that you set on your container.

Configure SSL or TLS

Depending on the environment, Conduktor might need to access external services (such as Kafka clusters, SSO servers, databases or object storage) that require a custom certificate for SSL/TLS communication. You can configure this using:
  • Console UI (recommended) - you can manage your certificates in a dedicated screen and configure SSL authentication from the broker setup wizard.
  • volume mount - this method is only required if you have LDAPS. Do not use it for Kafka or Kafka components.

Use the Conduktor certificate store

This option is recommended for Kafka, Kafka Connect and Schema Registry connections.
You can import and parse the certificates as text or files. The supported formats are:
  • .crt
  • .pem
  • .jks
  • .p12

Upload certificates

You can add cluster configurations from Settings > Clusters page. When you add the bootstrap server to your configuration, a check will be made to validate if the certificate is issued by a valid authority. If the response indicates the certificate is not issued by a valid authority, you have two options:
  • Skip SSL Check: This will skip validation of the SSL certificate on your server. This is an easy option for development environments with self-signed certificates
  • Upload Certificate: This option will enable you to upload the certificate (.crt, .pem, .jks or .p12 files), or paste the certificate as text
Cluster certificate Upon uploading the certificate, you should then see the green icon indicating the connection is secure. Cluster connection

Add truststores

You can also manage organization truststores using the Settings > Certificates page. Simply add all of your certificates by uploading them or pasting them as text. In doing this, the SSL context will be derived when you configure Kafka, Kafka Connect and Schema Registry connections. Certs

Mount custom truststore

This option is recommended for SSO, DB or other external services requiring SSL/TLS communication.
Conduktor supports SSL/TLS connections using Java truststore.

Create TrustStore (JKS) from certificate in PEM format

If you already have a truststore, you can ignore this step. You need a keytool program that is usually packaged on JDK distributions and a certificate in PEM format (.pem or .crt).

Configure custom truststore via Conduktor Console

Mount the truststore file into the conduktor-console container and pass the correct environment variables for locating truststore file inside the container (and password, if needed). If the truststore file is truststore.jks with password changeit, mount truststore file into /opt/conduktor/certs/truststore.jks inside the container. If run from Docker :
From docker-compose :

Client certificate authentication

This option is recommended for mTLS.
This mechanism uses TLS protocol to authenticate the client. Also known as:
  • Mutual SSL, Mutual TLS, mTLS
  • Two-Way SSL, SSL Certificate Authentication
  • Digital Certificate Authentication, Public Key Infrastructure (PKI) Authentication

Use the UI (keystore method)

Use the keystore file from your Kafka admin or provider (in .jks or .p12 format). Click the “Import from keystore” button to select a keystore file from your filesystem. Cluster keystore Fill in the required keystore password and key password and click “Import”. Import keystore You’ll get back to the cluster screen with the content of your keystore extracted into Access key and Access certificate. Cluster keystore import

Use the UI (Access key and access certificate method)

Your Kafka admin or your Kafka provider gave you 2 files for authentication.
  • An Access key (.key file)
  • An Access certificate (.pem or .crt file)
Here’s an example with Aiven: Aiven certs You can paste the contents of the two files into Conduktor or import from keystore.

Use volume mount

You can mount the keystore file in the conduktor-console image:
Then from the UI, choose the SSL Authentication method Keystore file is mounted on the volume and fill in the required fields

Configure Postgres database

Conduktor Console requires a Postgres database to store its state.

Postgres requirements

  • Postgres version 13 or higher
  • Provided connection role should have grant ALL PRIVILEGES on the configured database. Console should be able to create/update/delete schemas and tables on the database.
Minimum specifications for small or testing environments only:
  • 1-2 vCPU, 1 GB of RAM, and 10 GB of disk
For production deployments, please refer to our comprehensive PostgreSQL Sizing Guide which provides tier-based recommendations for AWS, GCP, and Azure to avoid performance issues like IOPS throttling, CPU spikes, and memory exhaustion.
If you want to use AWS RDS or AWS Aurora as a database with Console, consider the following: Console will not work with all PostgreSQL engines within RDS, it will only work with engine versions 14.8+ / 15.3+ (other versions are not fully supported).

Database configuration properties

  • database : is a key/value configuration consisting of:
    • database.url : database connection url in the format [jdbc:]postgresql://[user[:password]@][[netloc][:port],...][/dbname][?param1=value1&...]
    • database.hosts[].host : Postgresql server hosts name
    • database.hosts[].port : Postgresql server ports
    • database.host : Postgresql server host name (Deprecated. Use database.hosts instead)
    • database.port : Postgresql server port (Deprecated. Use database.hosts instead)
    • database.name : Database name
    • database.username : Database login role
    • database.password : Database login password
    • database.connection_timeout : Connection timeout option in seconds

URL format

Console supports both, the standard PostgreSQL URL and JDBC PostgreSQL . Connection username and password can be provided in the URL as basic authentication or as parameters.

SSL support

By default, Console will try to connect to the database using SSL mode prefer. We plan to make this configurable in the future along with database certificate.

Setup

There are several options available when configuring an external database:
  1. From a single connection URL
    • With the CDK_DATABASE_URL environment variable.
    • With the database.url configuration field. In either case, this connection url is using a standard PostgreSQL url in the format [jdbc:]postgresql://[user[:password]@][[netloc][:port],...][/dbname][?param1=value1&...]
  2. From decomposed configuration fields
    • With the CDK_DATABASE_* env vars.
    • With the database.* on configuration file.

Example

  • If all connection URLs and decomposed configuration fields are provided, the decomposed configuration fields take priority.
  • If an invalid connection URL or a mandatory configuration field (host, username or name) is missing, Conduktor will fail gracefully with a meaningful error message.
  • Before Console v1.2.0, the EMBEDDED_POSTGRES=false was mandatory to enable external Postgresql configuration.

Multi-host configuration

If you have a multi-host setup, you can configure the database connection with a list of hosts. Conduktor uses a PostgreSQL JDBC driver to connect to the database that supports multiple hosts in the connection url . To configure a multi-host setup, you can use the database.url configuration field with a list of hosts separated by commas:
or with decomposed configuration fields:
You can also provide JDBC connection parameter targetServerType to specify the target server type for the connection:
Nearly all targetServerType are supported: any, primary, master, slave, secondary, preferSlave, preferSecondary and preferPrimary.

Debug Console

Conduktor Console Docker image runs on Ubuntu Linux. It runs multiple services in a single Docker container. These services are supervised by supervisord . To troubleshoot Console:
  1. Verify that Console is up and running.
  2. Check the logs and send them to our support team if necessary.

1. Verify that Conduktor is up and running

First, verify that all the components are running.
Get containers status
Output
If you’re using an external Kafka installation and external database, you will only need to verify that the conduktor-console container is showing healthy as the STATUS.If Console is showing an “exited” status, check the Docker logs by running the command (with the appropriate container name):
Get container logs
You can save these logs in a file:
Store logs in a file

2. Get the logs and send them to support

Logs are kept in /var/conduktor/log. You can see them using:
List log files
Output
The best here is to simply bring all the logs to your local machine (in PWD) by running:
Then send these logs to oursupport team . If you’ve contacted us before, log into your account and create a ticket .

Configuration properties and environment variables

For the complete list of variables, see the Console environment variables reference.

Docker image environment variables

Console properties reference

You have multiple options to configure Console: via environment variables, or via a YAML configuration file. You can find a mapping of the configuration fields in the platform-config.yaml to environment variables below. Environment variables can be set on the container or imported from a file. When importing from a file, mount the file into the container and provide its path by setting the environment variable CDK_ENV_FILE. Use a .env file with key value pairs.
The logs will confirm, Sourcing environment variables from $CDK_ENV_FILE, or warn if set and the file is not found
In case you set both environment variable and YAML value for a specific field, the environment variable will take precedence.
Lists start at index 0 and are provided using _idx_ syntax.

YAML property cases

YAML configuration supports multiple case formats (camelCase/kebab-case/lowercase) for property fragments such as:
  • clusters[].schemaRegistry.ignoreUntrustedCertificate
  • clusters[].schema-registry.ignore-untrusted-certificate
  • clusters[].schemaregistry.ignoreuntrustedcertificate
All are valid and equivalent in YAML.

Environment variable conversion

At startup, Conduktor Console will merge environment variables and YAML based configuration files into one unified configuration. The conversion rules are:
  • Filter for environment variables that start with CDK_
  • Remove the CDK_ prefix
  • Convert the variable name to lowercase
  • Replace _ with . for nested properties
  • Replace _[0-9]+_ with [0-9]. for list properties. (Lists start at index 0)
For example, the environment variables CDK_DATABASE_URL will be converted to database.url, or CDK_SSO_OAUTH2_0_OPENID_ISSUER will be converted into sso.oauth2[0].openid.issuer. The YAML equivalent would be:
When converting environment variables to YAML configuration, environment variables in UPPER-KEBAB-CASE will be converted to kebab-case in the YAML configuration.

Conversion edge cases

Because of YAML multiple case formats support, the conversion rules have some edge cases when trying to mix environment variables and YAML configuration. Extra rules when mixing environment variables and YAML configuration:
  • Don’t use camelCase in YAML configuration. Use kebab-case or lowercase
  • Stick to one compatible case format for a given property fragment using the following compatibility matrix
Compatibility matrix: For example, CDK_CLUSTERS_0_SCHEMAREGISTRY_IGNOREUNTRUSTEDCERTIFICATE environment variable:
And CDK_CLUSTERS_0_SCHEMA-REGISTRY_IGNORE-UNTRUSTED-CERTIFICATE, that’s why camelCase is not recommended in YAML configuration when mixing with environment variables.

Support of shell expansion in the YAML configuration file

Console supports shell expansion for environment variables and home tilde ~. This is useful if you have to use custom environment variables in your configuration. For example, you can use the following syntax:
YAML configuration file
with the following environment variables: This will be expanded to:
Expanded configuration
If you want to escape the shell expansion, you can use the following syntax: $$. For example, if you want admin.password to be secret$123, you should set admin.password: "secret$$123".

File path environment variables

When an environment variable ending with _FILE is set to a file path, its corresponding unprefixed environment variable will be replaced with the content of that file. For example, if you set CDK_LICENSE_FILE=/run/secrets/license, the value of CDK_LICENSE will be overridden by the content of the file located at /run/secrets/license.
The CDK_IN_CONF_FILE is not supported.

Global properties

Database properties

See database configuration for details.

Session lifetime properties

Local users properties

Optional local account list used to log into Console.

Monitoring properties

To see monitoring graphs and use alerts, you have to ensure that Cortex is also deployed.

Monitoring Configuration for Console

First, we need to configure Console to connect to Cortex services. By default, Cortex ports are:
  • Query port: 9009
  • Alert manager port: 9010
monitoring.use-aggregated-metrics and monitoring.enable-non-aggregated-metrics are temporary flags to help you transition to the new metrics collection system. They will be removed in a future release.Swap their default value if you experience performance issues when Console is connected with large Kafka clusters:

Monitoring configuration for Cortex

See Cortex configuration for details.

SSO properties

LDAP properties

OAuth2 properties

JWT auth properties

Kafka cluster properties

The new recommended way to configure clusters is through the CLI and YAML manifests.

Kafka vendor specific properties

Note that you only need to set the Kafka cluster properties to use the core features of Console. However, you can get additional benefits by setting the flavor of your cluster. This corresponds to the Provider tab of your cluster configuration in Console.

Schema registry properties

Amazon Glue schema registry properties

Kafka Connect properties

ksqlDB properties

We support ksqlDB integration as of Conduktor Console v1.21.0.

AuditLog export properties

The audit log can be exported to a Kafka topic, once configured in Console.

Conduktor SQL properties

In order to use Conduktor SQL, you need to configure a second database to store the topics data. You can configure Conduktor SQL Database using CDK_KAFKASQL_DATABASE_URL or set each value individually with CDK_KAFKASQL_DATABASE_*. Configure SQL to get started. Advanced properties:

Partner Zones properties

Advanced configuration for Partner Zones.

Configure HTTP proxy

Specify the proxy settings for Conduktor to use when accessing Internet. The HTTP proxy works for both HTTP and HTTPS connection. There are five properties you can set to specify the proxy that will be used by the HTTP protocol handler:
  • CDK_HTTP_PROXY_HOST: the host name of the proxy server
  • CDK_HTTP_PROXY_PORT: the port number. Default value is 80.
  • CDK_HTTP_NON_PROXY_HOSTS: a list of hosts that should be reached directly, bypassing the proxy. This is a list of patterns separated by |. The patterns may start or end with a * for wildcards, we do not support /. Any host matching one of these patterns will be reached through a direct connection instead of through a proxy.
  • CDK_HTTP_PROXY_USERNAME: the proxy username
  • CDK_HTTP_PROXY_PASSWORD: the proxy password

Example

Configure HTTPS

To configure Conduktor Console to respond to HTTPS requests, you have to define a certificate and a private key. The server certificate is a public entity that’s sent to every client that connects to the server and it should be provided as a PEM file. Configuration properties are:
  • platform.https.cert.path or environment variable CDK_PLATFORM_HTTPS_CERT_PATH: the path to server certificate file
  • platform.https.key.path or environment variable CDK_PLATFORM_HTTPS_KEY_PATH: the path to server private key file
Both the certificate and private key files have to allow read from user conduktor-platform (UID 10001 GID 0) but don’t need to be readable system-wide.

Sample configuration using docker-compose

In this example, server certificate and key (server.crt and server.key) are stored in the same directory as the docker-compose file.
If the monitoring image conduktor/conduktor-console-cortex is running as well, you have to provide the CA public certificate to the monitoring image to allow metrics scraping on HTTPS.