Conduktor Console can be deployed and configured using either a configuration file platform-config.yaml or environment variables.To set up your organization’s environment:
Enter the organization’s name.
Connect to an external database (required).
Set up user authentication (basic or SSO).
Apply your Console license.
We recommend using the Console UI (Settings > Clusters page) to configure Kafka cluster, schema registry and Kafka connect. This has several advantages over the YAML configuration:
Intuitive interface with live update capabilities
Centralized and secured with RBAC and audit logs events
Certificate store to help with custom certificates configuration (no more JKS files and volume mounts)
The docker-compose below shows how to bind your platform-config.yaml file.You can alternatively use environment variables. The CDK_IN_CONF_FILE variable is used to indicate that a configuration file is being used and the location to find it.
Input configuration fields can also be provided using environment variables. Here’s an example of docker-compose that uses environment variables for configuration:
Console is running as a non-root user conduktor-platform with UID 10001 and GID 0. All files inside the container volume /var/conduktor are owned by conduktor-platform user.
Depending on the environment, Conduktor might need to access external services (such as Kafka clusters, SSO servers, databases or object storage) that require a custom certificate for SSL/TLS communication.You can configure this using:
Console UI (recommended) - you can manage your certificates in a dedicated screen and configure SSL authentication from the broker setup wizard.
volume mount - this method is only required if you have LDAPS. Do not use it for Kafka or Kafka components.
You can add cluster configurations from Settings > Clusters page. When you add the bootstrap server to your configuration, a check will be made to validate if the certificate is issued by a valid authority.If the response indicates the certificate is not issued by a valid authority, you have two options:
Skip SSL Check: This will skip validation of the SSL certificate on your server. This is an easy option for development environments with self-signed certificates
Upload Certificate: This option will enable you to upload the certificate (.crt, .pem, .jks or .p12 files), or paste the certificate as text
Upon uploading the certificate, you should then see the green icon indicating the connection is secure.
You can also manage organization truststores using the Settings > Certificates page. Simply add all of your certificates by uploading them or pasting them as text. In doing this, the SSL context will be derived when you configure Kafka, Kafka Connect and Schema Registry connections.
Create TrustStore (JKS) from certificate in PEM format
If you already have a truststore, you can ignore this step.You need a keytool program that is usually packaged on JDK distributions and a certificate in PEM format (.pem or .crt).
Mount the truststore file into the conduktor-console container and pass the correct environment variables for locating truststore file inside the container (and password, if needed).If the truststore file is truststore.jks with password changeit, mount truststore file into /opt/conduktor/certs/truststore.jks inside the container.If run from Docker :
Use the keystore file from your Kafka admin or provider (in .jks or .p12 format).Click the “Import from keystore” button to select a keystore file from your filesystem.Fill in the required keystore password and key password and click “Import”.You’ll get back to the cluster screen with the content of your keystore extracted into Access key and Access certificate.
Provided connection role should have grant ALL PRIVILEGES on the configured database. Console should be able to create/update/delete schemas and tables on the database.
For your Postgres deployment use at least 1-2 vCPU, 1 GB of Ram, and 10 GB of disk.
If you want to use AWS RDS or AWS Aurora as a database with Console, consider the following: Console will not work with all PostgreSQL engines within RDS, it will only work with engine versions 14.8+ / 15.3+ (other versions are not fully supported).
Console supports both, the standard PostgreSQL URL and JDBC PostgreSQL.Connection username and password can be provided in the URL as basic authentication or as parameters.
Copy
Ask AI
database: url: 'jdbc:postgresql://user:password@host:5432/database' # or 'postgresql://host:5432/database?user=user&password=password'
By default, Console will try to connect to the database using SSL mode prefer. We plan to make this configurable in the future along with database certificate.
There are several options available when configuring an external database:
From a single connection URL
With the CDK_DATABASE_URL environment variable.
With the database.url configuration field.
In either case, this connection url is using a standard PostgreSQL url in the format [jdbc:]postgresql://[user[:password]@][[netloc][:port],...][/dbname][?param1=value1&...]
If all connection URLs and decomposed configuration fields are provided, the decomposed configuration fields take priority.
If an invalid connection URL or a mandatory configuration field (host, username or name) is missing, Conduktor will fail gracefully with a meaningful error message.
Before Console v1.2.0, the EMBEDDED_POSTGRES=false was mandatory to enable external Postgresql configuration.
If you have a multi-host setup, you can configure the database connection with a list of hosts. Conduktor uses a PostgreSQL JDBC driver to connect to the database that supports multiple hosts in the connection url.To configure a multi-host setup, you can use the database.url configuration field with a list of hosts separated by commas:
Conduktor Console Docker image runs on Ubuntu Linux. It runs multiple services in a single Docker container. These services are supervised by supervisord.To troubleshoot Console:
Verify that Console is up and running.
Check the logs and send them to our support team if necessary.
First, verify that all the components are running.
Get containers status
Copy
Ask AI
docker ps
Output
Copy
Ask AI
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTSconduktor-console conduktor/conduktor-console:1.21.0 "/__cacert_entrypoin…" conduktor-console 10 minutes ago Up 9 minutes (healthy) 0.0.0.0:8080->8080/tcpconduktor-monitoring conduktor/conduktor-console-cortex:1.21.0 "/opt/conduktor/scri…" conduktor-monitoring 10 minutes ago Up 10 minutes (healthy) 0.0.0.0:9009-9010->9009-9010/tcp, 0.0.0.0:9090->9090/tcppostgres postgres:15.1 "docker-entrypoint.s…" postgres 10 minutes ago Up 10 minutes 0.0.0.0:5432->5432/tcp
If you’re using an external Kafka installation and external database, you will only need to verify that the conduktor-console container is showing healthy as the STATUS.If Console is showing an “exited” status, check the Docker logs by running the command (with the appropriate container name):
You have multiple options to configure Console: via environment variables, or via a YAML configuration file. You can find a mapping of the configuration fields in the platform-config.yaml to environment variables below.Environment variables can be set on the container or imported from a file. When importing from a file, mount the file into the container and provide its path by setting the environment variable CDK_ENV_FILE. Use a .env file with key value pairs.
Copy
Ask AI
MY_ENV_VAR1=valueMY_ENV_VAR2=otherValue
The logs will confirm, Sourcing environment variables from $CDK_ENV_FILE, or warn if set and the file is not found
Copy
Ask AI
Warning: CDK_ENV_FILE is set but the file does not exist or is not readable.
In case you set both environment variable and YAML value for a specific field, the environment variable will take precedence.
Lists start at index 0 and are provided using _idx_ syntax.
At startup, Conduktor Console will merge environment variables and YAML based configuration files into one unified configuration. The conversion rules are:
Filter for environment variables that start with CDK_
Remove the CDK_ prefix
Convert the variable name to lowercase
Replace _ with . for nested properties
Replace _[0-9]+_ with [0-9]. for list properties. (Lists start at index 0)
For example, the environment variables CDK_DATABASE_URL will be converted to database.url, or CDK_SSO_OAUTH2_0_OPENID_ISSUER will be converted into sso.oauth2[0].openid.issuer.The YAML equivalent would be:
When converting environment variables to YAML configuration, environment variables in UPPER-KEBAB-CASE will be converted to kebab-case in the YAML configuration.
Because of YAML multiple case formats support, the conversion rules have some edge cases when trying to mix environment variables and YAML configuration.Extra rules when mixing environment variables and YAML configuration:
Don’t use camelCase in YAML configuration. Use kebab-case or lowercase
Stick to one compatible case format for a given property fragment using the following compatibility matrix
Compatibility matrix:
YAML\Environment
UPPER-KEBAB-CASE
UPPERCASE
kebab-case
✅
🚫
lowercase
🚫
✅
camelCase
🚫
🚫
For example, CDK_CLUSTERS_0_SCHEMAREGISTRY_IGNOREUNTRUSTEDCERTIFICATE environment variable:
Copy
Ask AI
# Is equivalent to and compatible withclusters: - schemaregistry: ignoreuntrustedcertificate: true# but not withclusters: - schema-registry: ignore-untrusted-certificate: true
And CDK_CLUSTERS_0_SCHEMA-REGISTRY_IGNORE-UNTRUSTED-CERTIFICATE, that’s why camelCase is not recommended in YAML configuration when mixing with environment variables.
Support of shell expansion in the YAML configuration file
Console supports shell expansion for environment variables and home tilde ~. This is useful if you have to use custom environment variables in your configuration.For example, you can use the following syntax:
If you want to escape the shell expansion, you can use the following syntax: $$. For example, if you want admin.password to be secret$123, you should set admin.password: "secret$$123".
When an environment variable ending with _FILE is set to a file path, its corresponding unprefixed environment variable will be replaced with the content of that file.For example, if you set CDK_LICENSE_FILE=/run/secrets/license, the value of CDK_LICENSE will be overridden by the content of the file located at /run/secrets/license.
Your organization’s root administrator account email
CDK_ADMIN_EMAIL
true
string
∅
admin.password
Your organization’s root administrator account password. Must be at least 8 characters in length, and include at least 1 uppercase letter, 1 lowercase letter, 1 number, and 1 special symbol
CDK_ADMIN_PASSWORD
true
string
∅
license
Enterprise license key. If not provided, fallback to free plan.
CDK_LICENSE or LICENSE_KEY
false
string
∅
platform.external.url
Force Console external URL. Useful for SSO callback URL when using a reverse proxy. By default, Console will try to guess it automatically using X-Forwarded-* headers coming from upstream reverse proxy.
CDK_PLATFORM_EXTERNAL_URL
false
string
∅
platform.https.cert.path
Path to the SSL certificate file
CDK_PLATFORM_HTTPS_CERT_PATH
false
string
∅
platform.https.key.path
Path to the SSL private key file
CDK_PLATFORM_HTTPS_KEY_PATH
false
string
∅
enable_product_metrics
In order to improve Conduktor Console, we collect anonymous usage metrics. Set to false, this configuration disable all of our metrics collection.
First, we need to configure Console to connect to Cortex services. By default, Cortex ports are:
Query port: 9009
Alert manager port: 9010
Property
Description
Environment variable
Mandatory
Type
Default
monitoring.cortex-url
Cortex Search Query URL with port 9009
CDK_MONITORING_CORTEXURL
true
string
∅
monitoring.alert-manager-url
Cortex Alert Manager URL with port 9010
CDK_MONITORING_ALERTMANAGERURL
true
string
∅
monitoring.callback-url
Console API
CDK_MONITORING_CALLBACKURL
true
string
∅
monitoring.notifications-callback-url
Where the Slack notification should redirect
CDK_MONITORING_NOTIFICATIONCALLBACKURL
true
string
∅
monitoring.clusters-refresh-interval
Refresh rate in seconds for metrics
CDK_MONITORING_CLUSTERREFRESHINTERVAL
false
int
60
monitoring.use-aggregated-metrics
Defines whether use the new aggregated metrics in the Console graphs
CDK_MONITORING_USEAGGREGATEDMETRICS
No
Boolean
false
monitoring.enable-non-aggregated-metrics
Toggles the collection of obsolete granular metrics
CDK_MONITORING_ENABLENONAGGREGATEDMETRICS
No
Boolean
true
monitoring.use-aggregated-metrics and monitoring.enable-non-aggregated-metrics are temporary flags to help you transition to the new metrics collection system. They will be removed in a future release.Swap their default value if you experience performance issues when Console is connected with large Kafka clusters:
Sets the search filter. By default, the filter is set to (uid={0}) for users using class type InetOrgPerson.
CDK_SSO_LDAP_0_SEARCHFILTER
false
string
"(uid={0})"
sso.ldap[].search-attributes
Sets the attributes list to return. By default, all attributes are returned. Platform search for uid, cn, mail, email, givenName, sn, displayName attributes to map into user token.
CDK_SSO_LDAP_0_SEARCHATTRIBUTES
false
string array
[]
sso.ldap[].groups-enabled
Sets if group search is enabled.
CDK_SSO_LDAP_0_GROUPSENABLED
false
boolean
false
sso.ldap[].groups-subtree
Sets if the subtree should be searched.
CDK_SSO_LDAP_0_GROUPSSUBTREE
false
boolean
true
sso.ldap[].groups-base
Sets the base DN to search from.
CDK_SSO_LDAP_0_GROUPSBASE
true
string
∅
sso.ldap[].groups-filter
Sets the group search filter. If using group class type GroupOfUniqueNames use the filter "uniqueMember={0}". For group class GroupOfNames use "member={0}".
CDK_SSO_LDAP_0_GROUPSFILTER
false
string
"uniquemember={0}"
sso.ldap[].groups-filter-attribute
Sets the name of the user attribute to bind to the group search filter. Defaults to the user’s DN.
CDK_SSO_LDAP_0_GROUPSFILTERATTRIBUTE
false
string
∅
sso.ldap[].groups-attribute
Sets the group attribute name. Defaults to cn.
CDK_SSO_LDAP_0_GROUPSATTRIBUTE
false
string
"cn"
sso.ldap[].properties
Additional properties that will be passed to identity provider context.
Note that you only need to set the Kafka cluster properties to use the core features of Console.However, you can get additional benefits by setting the flavor of your cluster. This corresponds to the Provider tab of your cluster configuration in Console.
Property
Description
Environment variable
Mandatory
Type
Default
clusters[].kafkaFlavor.type
Kafka flavor type, one of Confluent, Aiven, Gateway
CDK_CLUSTERS_0_KAFKAFLAVOR_TYPE
false
string
∅
Flavor is Confluent
Manage Confluent Cloud service accounts, API keys, and ACLs
In order to use Conduktor SQL, you need to configure a second database to store the topics data.You can configure Conduktor SQL Database using CDK_KAFKASQL_DATABASE_URL or set each value individually with CDK_KAFKASQL_DATABASE_*.Configure SQL to get started.
Property
Description
Environment variable
Mandatory
Type
Default
kafka_sql.database.url
External PostgreSQL configuration URL in format [jdbc:]postgresql://[user[:password]@][[netloc][:port],...][/dbname][?param1=value1&...]
CDK_KAFKASQL_DATABASE_URL
false
string
∅
kafka_sql.database.hosts[].host
External PostgreSQL servers hostname
CDK_KAFKASQL_DATABASE_HOSTS_0_HOST
false
string
∅
kafka_sql.database.hosts[].port
External PostgreSQL servers port
CDK_KAFKASQL_DATABASE_HOSTS_0_PORT
false
int
∅
kafka_sql.database.host
External PostgreSQL server hostname (Deprecated, use kafka_sql.database.hosts instead)
CDK_KAFKASQL_DATABASE_HOST
false
string
∅
kafka_sql.database.port
External PostgreSQL server port (Deprecated, use kafka_sql.database.hosts instead)
CDK_KAFKASQL_DATABASE_PORT
false
int
∅
kafka_sql.database.name
External PostgreSQL database name
CDK_KAFKASQL_DATABASE_NAME
false
string
∅
kafka_sql.database.username
External PostgreSQL login role
CDK_KAFKASQL_DATABASE_USERNAME
false
string
∅
kafka_sql.database.password
External PostgreSQL login password
CDK_KAFKASQL_DATABASE_PASSWORD
false
string
∅
kafka_sql.database.connection_timeout
External PostgreSQL connection timeout in seconds
CDK_KAFKASQL_DATABASE_CONNECTIONTIMEOUT
false
int
∅
Advanced properties:
Property
Description
Environment variable
Mandatory
Type
Default
kafka_sql.commit_offset_every_in_sec
Frequency at which Conduktor SQL commits offsets into Kafka and flushes rows in the database
CDK_KAFKASQL_COMMITOFFSETEVERYINSEC
false
int
30 (seconds)
kafka_sql.clean_expired_record_every_in_hour
How often to check for expired records and delete them from the database
The interval at which Partner Zone’s state (that’s stored on Console) is synchronized with Gateway. A lower value results in faster alignment between the required state and the current state on Gateway.
Specify the proxy settings for Conduktor to use when accessing Internet. The HTTP proxy works for both HTTP and HTTPS connection.There are five properties you can set to specify the proxy that will be used by the HTTP protocol handler:
CDK_HTTP_PROXY_HOST: the host name of the proxy server
CDK_HTTP_PROXY_PORT: the port number. Default value is 80.
CDK_HTTP_NON_PROXY_HOSTS: a list of hosts that should be reached directly, bypassing the proxy. This is a list of patterns separated by |. The patterns may start or end with a * for wildcards, we do not support /. Any host matching one of these patterns will be reached through a direct connection instead of through a proxy.
To configure Conduktor Console to respond to HTTPS requests, you have to define a certificate and a private key.The server certificate is a public entity that’s sent to every client that connects to the server and it should be provided as a PEM file.Configuration properties are:
platform.https.cert.path or environment variable CDK_PLATFORM_HTTPS_CERT_PATH: the path to server certificate file
platform.https.key.path or environment variable CDK_PLATFORM_HTTPS_KEY_PATH: the path to server private key file
Both the certificate and private key files have to allow read from user conduktor-platform (UID 10001 GID 0) but don’t need to be readable system-wide.
If the monitoring image conduktor/conduktor-console-cortex is running as well, you have to provide the CA public certificate to the monitoring image to allow metrics scraping on HTTPS.
Copy
Ask AI
services: conduktor-console: image: conduktor/conduktor-console ports: - 8080:8080 volumes: - type: bind source: ./server.crt target: /opt/conduktor/certs/server.crt read_only: true - type: bind source: ./server.key target: /opt/conduktor/certs/server.key read_only: true environment: # HTTPS configuration CDK_PLATFORM_HTTPS_CERT_PATH: '/opt/conduktor/certs/server.crt' CDK_PLATFORM_HTTPS_KEY_PATH: '/opt/conduktor/certs/server.key' # monitoring configuration CDK_MONITORING_CORTEX-URL: http://conduktor-monitoring:9009/ CDK_MONITORING_ALERT-MANAGER-URL: http://conduktor-monitoring:9010/ CDK_MONITORING_CALLBACK-URL: https://conduktor-console:8080/monitoring/api/ CDK_MONITORING_NOTIFICATIONS-CALLBACK-URL: http://localhost:8080 conduktor-monitoring: image: conduktor/conduktor-console-cortex volumes: - type: bind source: ./server.crt target: /opt/conduktor/certs/server.crt read_only: true environment: CDK_CONSOLE-URL: "https://conduktor-console:8080" CDK_SCRAPER_SKIPSSLCHECK: "false" # can be set to true if you don't want to check the certificate CDK_SCRAPER_CAFILE: "/opt/conduktor/certs/server.crt"