Skip to main content
Quick navigation

Environment Variables

Configuring the environment variables is the recommended way of setting up Conduktor Gateway.

Jump to:

Kafka Environment Variables

Conduktor Gateway's connection to Kafka are configured by the KAFKA_ environment variables. When translating Kafka's properties, use upper case instead and replace the . with _.

For example;
When defining Gateway's Kafka property bootstrap.servers, declare it as the environment variable KAFKA_BOOTSTRAP_SERVERS.

Any variable prefixed with KAFKA_ will be treated as a connection parameter by Gateway.

Gateway Environment Variables

Default configurations for Conduktor Gateway can be overridden by environment variables.

Guidelines

There is no typical deployment of Gateway as every environment will be unique in it's design considerations and security requirements.

The below is an example including some variables we recommend you modify in any setup you do, but is by no means a guarantee of sufficient requirements in your setup.

We will support you in onboarding of Conduktor Gateway to help you get setup in the first place and for any ongoing issues or questions please contact support at support@conduktor.io.

Example Values

  -e KAFKA_BOOTSTRAP_SERVERS=kafka1:9092,kafka2:9092 \
-e KAFKA_SASL_MECHANISM=PLAIN \
-e KAFKA_SECURITY_PROTOCOL=SASL_PLAINTEXT \
-e KAFKA_SASL_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username='usr' password='pwd';"

Host/Port

Environment VariableDefault ValueDescription
GATEWAY_BIND_HOST0.0.0.0The host on which to bind the gateway
GATEWAY_ADVERTISED_HOSTdefaults to your hostnameThe gateway hostname that should be presented to clients
GATEWAY_PORT_START6969Port on which Gateway will start listening on
GATEWAY_PORT_COUNTdefaults to your number of brokers +2Number of ports to be used by the Gateway, each port will correspond to a broker in the Kafka cluster so it must be at least as large as the broker count of the Kafka cluster. In production, we recommend it is double the size of the Kafka cluster to allow for expansion and reassignment.

Load Balancing

Environment VariableDefault ValueDescription
GATEWAY_CLUSTER_IDconduktorGatewayA unique identifier for a given Gateway cluster, this is used to establish Gateway cluster membership for load balancing
GATEWAY_FEATURE_FLAGS_INTERNAL_LOAD_BALANCINGtrueWhether to use Conduktor Gateway's internal load balancer to balance connections between Gateway instances.
GATEWAY_RACK_IDnoneSimilar as broker.rack
  • KAFKA_SECURITY where your credentials and ACL handled your target Kafka cluster
  • GATEWAY_SECURITY where your credentials and ACL handled by Gateway
  • VCLUSTER where your virtual clusters, credentials and ACL handled by Gateway

Client to Gateway Authentication

Note: These configurations apply to authentication between clients and Conduktor Gateway. For authentication between Conduktor Gateway and Kafka see Kafka Environment Variables

Environment VariableDefault ValueDescription
GATEWAY_SECURITY_PROTOCOLdefaults to KAFKA_SECURITY_PROTOCOLThe type of authentication clients should use to connect to the gateway, valid values are PLAINTEXT, SASL_PLAINTEXT, SASL_SSL, SSL, DELEGATED_SASL_PLAINTEXT and DELEGATED_SASL_SSL
GATEWAY_FEATURE_FLAGS_MANDATORY_VCLUSTERdefault to falseSet if authenticated users are automatically assigned to passthrough vcluster if it's not configured. Reject authentication if set to true and vcluster is not configured for a principal
GATEWAY_ACL_STORE_ENABLEDdefault to falseWhen in non-delegated authentication mode, enable Kafka ACLs on the Gateway

SSL

Environment VariableDefault ValueDescription
GATEWAY_SSL_KEY_STORE_PATHconfig/kafka-proxy.keystore.jksPath to a keystore for SSL connections
GATEWAY_SSL_KEY_STORE_PASSWORD123456Password for the keystore defined above
GATEWAY_SSL_KEY_PASSWORD123456Password for the key contained in the store above
GATEWAY_SSL_KEY_TYPEjksWe currently only support jks
GATEWAY_SSL_UPDATE_INTERVAL_MS600000
GATEWAY_SSL_UPDATE_CONTEXT_INTERVAL_MINUTES5Interval in minutes to refresh SSL context
Environment VariableDefault ValueDescription
GATEWAY_SSL_TRUST_STORE_PATHconfig/kafka-proxy.truststore.jksPath to a keystore for SSL connections
GATEWAY_SSL_TRUST_STORE_PASSWORD123456Password for the keystore defined above
GATEWAY_SSL_TRUST_STORE_TYPEjksWe currently only support jks
GATEWAY_SSL_CLIENT_AUTHNONENONE will not request client authentication, OPTIONAL will request client authentication, REQUIRE will require client authentication

SSL Config

Environment VariableDefault ValueDescription
GATEWAY_AUTHENTICATION_CONNECTION_MAX_REAUTH_MS0Max Reauth
GATEWAY_AUTHENTICATION_TIMEOUT_MS1000Timeout in ms
GATEWAY_AUTHENTICATION_EXPONENTIAL_BACKOFF_MULTIPLIER2Backoff multiplier on reauth
GATEWAY_AUTHENTICATION_EXPONENTIAL_BACKOFF_MAX_MS5000Max backoff

MTLS

more context for MTLS here

Environment VariableDefault ValueDescription
GATEWAY_SSL_PRINCIPAL_MAPPING_RULESextracts the CNmTLS leverages SSL mutual authentication to identify a Kafka client. Principal for mTLS connection can be detected from the subject certificate using the same feature as in Apache Kafka, the SSL principal mapping

OAuthbearer

Some of these definitions are taken from the Kafka documentation, e.g. JKWS_REFRESH.

Environment VariableDefault ValueDescription
GATEWAY_OAUTH_JWKS_URLNULLThe OAuth/OIDC provider URL from which the provider's JWKS (JSON Web Key Set) can be retrieved. The URL can be HTTP(S)-based or file-based.
GATEWAY_OAUTH_EXPECTED_ISSUERNULLThe (optional) setting for the broker to use to verify that the JWT was created by the expected issuer. The JWT will be inspected for the standard OAuth iss claim and if this value is set, the broker will match it exactly against what is in the JWT's iss claim. If there is no match, the broker will reject the JWT and authentication will fail
GATEWAY_OAUTH_EXPECTED_AUDIENCESNULLThe (optional) comma-delimited setting for the broker to use to verify that the JWT was issued for one of the expected audiences. The JWT will be inspected for the standard OAuth aud claim and if this value is set, the broker will match the value from JWT's aud claim to see if there is an exact match. If there is no match, the broker will reject the JWT and authentication will fail.
GATEWAY_OAUTH_JWKS_REFRESHNULLThe (optional) value in milliseconds for the broker to wait between refreshing its JWKS (JSON Web Key Set) cache that contains the keys to verify the signature of the JWT.
GATEWAY_OAUTH_JWKS_RETRYNULLThe (optional) value in milliseconds for the initial wait between JWKS (JSON Web Key Set) retrieval attempts from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting
GATEWAY_OAUTH_JWKS_MAX_RETRYNULLThe (optional) value in milliseconds for the maximum wait between attempts to retrieve the JWKS (JSON Web Key Set) from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting
GATEWAY_OAUTH_SCOPE_CLAIM_NAMENULLThe OAuth claim for the scope is often named scope, but this (optional) setting can provide a different name to use for the scope included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim.
GATEWAY_OAUTH_SUB_CLAIM_NAMENULLThe OAuth claim for the subject is often named sub, but this (optional) setting can provide a different name to use for the subject included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim.

SECURITY PROVIDER

Environment VariableDefault ValueDescription
GATEWAY_SECURITY_PROVIDERDEFAULTSpecify your security provider, can be DEFAULT (from your JRE), BOUNCY_CASTLE, BOUNCY_CASTLE_FIPS and CONSCRYPT

Please note that CONSCRYPT does not support Mac OS with aarch64.

SECRET MANAGEMENT

Secrets may be passed from configuration to Gateway using environment variables. Some suggested examples are below that may be more common, but you are free to use your own and avoid any clashes with existing environment variables.

  • SCHEMA_REGISTRY_LOGIN
  • SCHEMA_REGISTRY_PASSWORD
  • AWS_ACCESS_KEY
  • AWS_SECRET_KEY
  • AWS_SESSION_TOKEN
  • AZURE_CLIENT_ID
  • AZURE_TENANT_ID
  • AZURE_CLIENT_SECRET
  • VAULT_TOKEN
  • VAULT_USERNAME
  • VAULT_PASSWORD
  • etc.

For a full list of security examples consider the marketplace plugin pages.

HTTP

Environment VariableDefault ValueDescription
GATEWAY_HTTP_PORT8888The port on which the gateway will present a HTTP management API
GATEWAY_SECURED_METRICStrueDoes the HTTP management API require users?
GATEWAY_ADMIN_API_USERS[{username: admin, password: conduktor, admin: true}]Users that can access the api, please note that admin is required to do any write

Internal State

Conduktor needs to save state, you can choose where:

Environment VariableDefault ValueDescription
GATEWAY_STORAGE_TYPEKAFKACan be IN_MEMORY or, KAFKA
GATEWAY_GROUP_IDnullSet the group name for internal topic if not defined
GATEWAY_STORE_TTL_MS604800000Time between full refresh

Topics Names

State is saved in different location based on GATEWAY_STORAGE_TYPE

When it is set

  • KAFKA they will be materialized as a topic.
  • IN_MEMORY they will be stored in memory.
Environment VariableDefault ValueDescription
GATEWAY_LICENSE_TOPIC_conduktor_gateway_licenseName of license topic
GATEWAY_TOPIC_MAPPINGS_TOPIC_conduktor_gateway_topicmappingsName of topicMappings topic
GATEWAY_USER_MAPPINGS_TOPIC_conduktor_gateway_usermappingsName of the user mapping topic
GATEWAY_CONSUMER_SUBSCRIPTIONS_TOPIC_conduktor_gateway_consumer_subscriptionsName of the subscriptions for concentrated topic consumption topic
GATEWAY_CONSUMER_OFFSETS_TOPIC_conduktor_gateway_consumer_offsetsName of the topic to store the offsets for concentrated topic consumption
GATEWAY_INTERCEPTOR_CONFIGS_TOPIC_conduktor_gateway_interceptor_configsName of interceptor config topic
GATEWAY_ENCRYPTION_CONFIGS_TOPIC_conduktor_gateway_encryption_configsName of encryption configuration topic
GATEWAY_ACLS_TOPIC_conduktor_gateway_aclsName of the acl topic
GATEWAY_AUDIT_LOG_TOPIC_conduktor_gateway_auditlogsName of audit topic

IN_MEMORY State Configurations

none

KAFKA State Configurations

Environment VariableDefault ValueDescription
GATEWAY_TOPIC_STORE_REAL_TOPIC_PARTITION_COUNT-1Defaults to the one defined in your cluster settings
GATEWAY_TOPIC_STORE_KCACHE_REPLICATION_FACTOR-1Defaults to the one defined in your cluster settings
GATEWAY_TOPIC_STORE_DISTRIBUTED_CATCHUP_TIMEOUT_IN_SECONDS1Duration for catchup

Internal Setup

Threading

Environment VariableDefault ValueDescription
GATEWAY_DOWNSTREAM_THREADnumber of coresThe number of threads dedicated to handling IO between clients and Conduktor Gateway
GATEWAY_UPSTREAM_THREADnumber of coresThe number of threads dedicated to handling IO between Kafka and Conduktor Gateway

Upstream Connection

Environment VariableDefault ValueDescription
GATEWAY_UPSTREAM_CONNECTION_POOL_TYPENONEUpstream connection pool type. Possible values are NONE (no connection pool), ROUND_ROBIN (Round robin selected connection pool)
GATEWAY_UPSTREAM_NUM_CONNECTION10The number of connections between Conduktor Gateway and Kafka per upstream thread. Used only when ROUND_ROBIN is enabled.

Feature Flags

Environment VariableDefault ValueDescription
GATEWAY_FEATURE_FLAGS_AUDITtrueWhether or not to enable the audit feature
GATEWAY_FEATURE_FLAGS_INTERNAL_LOAD_BALANCINGtrueWhether or not to enable we replicate kafka internal load balancing

Licensing

Environment VariableDefault ValueDescription
GATEWAY_LICENSE_KEYNoneLicense key

Audit

Environment VariableDefault ValueDescription
GATEWAY_AUDIT_LOG_CONFIG_SPEC_VERSION0.1.0Version
GATEWAY_AUDIT_LOG_SERVICE_BACKING_TOPIC_auditLogsTarget topic name
GATEWAY_AUDIT_LOG_REPLICATION_FACTOR_OF_TOPIC-1Replication factor to be used when creating the audit topic, defaults to the one defined in your cluster settings
GATEWAY_AUDIT_LOG_NUM_PARTITIONS_OF_TOPIC-1Number of partitions to be used when creating the audit topic, defaults to the one defined in your cluster settings

Logging

Environment VariableDefault ValueDescription
LOG4J2_APPENDER_LAYOUTpatternThe format to output console logging. Use json for json layout or pattern for pattern layout
LOG4J2_ROOT_LEVELinfoThe logging level for the root logger
LOG4J2_ORG_APACHE_KAFKA_LEVELwarnThe logging level for the package org.apache.kafka
LOG4J2_IO_KCACHE_LEVELwarnThe logging level for the package io.kcache
LOG4J2_IO_VERTX_LEVELwarnThe logging level for the package io.vertx
LOG4J2_IO_NETTY_LEVELerrorThe logging level for the package io.netty
LOG4J2_IO_CONDUKTOR_LEVELinfoThe logging level for the package io.conduktor
LOG4J2_IO_CONDUKTOR_PROXY_AUTHORIZATION_LEVELinfoThe logging level for the package io.conduktor.proxy.authorization
LOG4J2_IO_CONDUKTOR_PROXY_REBUILDER_COMPONENTS_LEVELinfoThe logging level for the package io.conduktor.proxy.rebuilder.components
LOG4J2_IO_CONDUKTOR_PROXY_SERVICE_LEVELinfoThe logging level for the package io.conduktor.proxy.service
LOG4J2_IO_CONDUKTOR_PROXY_NETWORK_LEVELinfoThe logging level for the package io.conduktor.proxy.network
LOG4J2_IO_MICROMETER_LEVELerrorThe logging level for the package io.micrometer

Product Analytics

Environment VariableDefault ValueDescription
GATEWAY_FEATURE_FLAGS_ANALYTICStrueConduktor collects basic user analytics to understand product usage and enhance product development and improvement, such as a Gateway Started event. This is not based on any of the underlying Kafka data which is never sent to Conduktor.