Quick navigation
Environment Variables
Configuring the environment variables is the recommended way of setting up Conduktor Gateway.
Jump to:
Kafka Environment Variables
Conduktor Gateway's connection to Kafka are configured by the KAFKA_ environment variables.
When translating Kafka's properties, use upper case instead and replace the . with _.
For example;
When defining Gateway's Kafka property bootstrap.servers, declare it as the environment variable KAFKA_BOOTSTRAP_SERVERS.
Any variable prefixed with KAFKA_ will be treated as a connection parameter by Gateway.
Gateway Environment Variables
Default configurations for Conduktor Gateway can be overridden by environment variables.
Guidelines
There is no typical deployment of Gateway as every environment will be unique in it's design considerations and security requirements.
The below is an example including some variables we recommend you modify in any setup you do, but is by no means a guarantee of sufficient requirements in your setup.
We will support you in onboarding of Conduktor Gateway to help you get setup in the first place and for any ongoing
issues or questions please contact support at support@conduktor.io.
Example Values
-e KAFKA_BOOTSTRAP_SERVERS=kafka1:9092,kafka2:9092 \
-e KAFKA_SECURITY_PROTOCOL=SASL_PLAINTEXT \
-e KAFKA_SASL_MECHANISM=PLAIN \
-e KAFKA_SASL_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username='usr' password='pwd';"
Host/Port
| Environment Variable | Default Value | Description |
|---|---|---|
GATEWAY_BIND_HOST | 0.0.0.0 | The host on which to bind the gateway |
GATEWAY_ADVERTISED_HOST | Your hostname | The gateway hostname that should be presented to clients |
GATEWAY_PORT_START | 6969 | Port on which Gateway will start listening on |
GATEWAY_PORT_COUNT | (maxBrokerId-minBrokerId)+3 | Number of ports to be used by the Gateway, see networking for more on host/port configuration. |
GATEWAY_MIN_BROKERID | 0 | The broker id associated to Gateway's first port (GATEWAY_PORT_START), should be the lowest broker.id (or node.id) defined in the Kafka cluster. |
GATEWAY_ROUTING_MECHANISM | port | port or host. |
| SNI Routing only | Check our dedicated guide on SNI Routing | |
GATEWAY_ADVERTISED_SNI_PORT | none | Port to be advertised to the client if routing mechanism is set to host for SNI routing. |
GATEWAY_ADVERTISED_HOST_PREFIX | broker | Set the host prefix when using SNI Routing |
GATEWAY_SNI_HOST_SEPARATOR | - | Set the host separator when using SNI Routing. The value is split on the first occurence of this separator. |
Load Balancing
| Environment Variable | Default Value | Description |
|---|---|---|
GATEWAY_CLUSTER_ID | conduktorGateway | A unique identifier for a given Gateway cluster, this is used to establish Gateway cluster membership for load balancing |
GATEWAY_FEATURE_FLAGS_INTERNAL_LOAD_BALANCING | true | Whether to use Conduktor Gateway's internal load balancer to balance connections between Gateway instances. |
GATEWAY_RACK_ID | none | Similar as broker.rack |
Client to Gateway Authentication
Note: These configurations apply to authentication between clients and Conduktor Gateway. For authentication between Conduktor Gateway and Kafka see Kafka Environment Variables
| Environment Variable | Default Value | Description |
|---|---|---|
GATEWAY_SECURITY_PROTOCOL | defaults to KAFKA_SECURITY_PROTOCOL | The type of authentication clients should use to connect to the gateway, valid values are PLAINTEXT, SASL_PLAINTEXT, SASL_SSL, SSL, DELEGATED_SASL_PLAINTEXT and DELEGATED_SASL_SSL |
GATEWAY_FEATURE_FLAGS_MANDATORY_VCLUSTER | default to false | If no virtual cluster was detected then user automatically falls back into the transparent virtual cluster, named passthrough. Reject authentication if set to true and vcluster is not configured for a principal |
GATEWAY_ACL_ENABLED | default to false | Enable / Disable ACLs support on the Gateway (not including Virtual Clusters) |
GATEWAY_SUPER_USERS | empty | Coma-separated list of service accounts that will be super users on the Gateway (excluding Virtual Clusters). Example: alice,bob. |
GATEWAY_ACL_STORE_ENABLED | default to false | Obsolete, use VirtualCluster resource now Enable / Disable ACLs support for Virtual Clusters only. |
SSL
| Environment Variable | Default Value | Description |
|---|---|---|
GATEWAY_SSL_KEY_STORE_PATH | config/kafka-proxy.keystore.jks | Path to a keystore for SSL connections |
GATEWAY_SSL_KEY_STORE_PASSWORD | 123456 | Password for the keystore defined above |
GATEWAY_SSL_KEY_PASSWORD | 123456 | Password for the key contained in the store above |
GATEWAY_SSL_KEY_TYPE | jks | We currently only support jks |
GATEWAY_SSL_UPDATE_INTERVAL_MS | 600000 | |
GATEWAY_SSL_UPDATE_CONTEXT_INTERVAL_MINUTES | 5 | Interval in minutes to refresh SSL context |
| Environment Variable | Default Value | Description |
|---|---|---|
GATEWAY_SSL_TRUST_STORE_PATH | config/kafka-proxy.truststore.jks | Path to a keystore for SSL connections |
GATEWAY_SSL_TRUST_STORE_PASSWORD | 123456 | Password for the keystore defined above |
GATEWAY_SSL_TRUST_STORE_TYPE | jks | We currently only support jks |
GATEWAY_SSL_CLIENT_AUTH | NONE | NONE will not request client authentication, OPTIONAL will request client authentication, REQUIRE will require client authentication |
SSL Config
| Environment Variable | Default Value | Description |
|---|---|---|
GATEWAY_AUTHENTICATION_CONNECTION_MAX_REAUTH_MS | 0 | Max Reauth |
GATEWAY_AUTHENTICATION_TIMEOUT_MS | 1000 | Timeout in ms |
GATEWAY_AUTHENTICATION_EXPONENTIAL_BACKOFF_MULTIPLIER | 2 | Backoff multiplier on reauth |
GATEWAY_AUTHENTICATION_EXPONENTIAL_BACKOFF_MAX_MS | 5000 | Max backoff |
MTLS
More context for mTLS here
| Environment Variable | Default Value | Description |
|---|---|---|
GATEWAY_SSL_PRINCIPAL_MAPPING_RULES | extracts the CN | mTLS leverages SSL mutual authentication to identify a Kafka client. Principal for mTLS connection can be detected from the subject certificate using the same feature as in Apache Kafka, the SSL principal mapping |
OAuthbearer
Some of these definitions are taken from the Kafka documentation, e.g. JKWS_REFRESH.
| Environment Variable | Default Value | Description |
|---|---|---|
GATEWAY_OAUTH_JWKS_URL | NULL | The OAuth/OIDC provider URL from which the provider's JWKS (JSON Web Key Set) can be retrieved. The URL can be HTTP(S)-based or file-based. |
GATEWAY_OAUTH_EXPECTED_ISSUER | NULL | The (optional) setting for the broker to use to verify that the JWT was created by the expected issuer. The JWT will be inspected for the standard OAuth iss claim and if this value is set, the broker will match it exactly against what is in the JWT's iss claim. If there is no match, the broker will reject the JWT and authentication will fail |
GATEWAY_OAUTH_EXPECTED_AUDIENCES | NULL | The (optional) comma-delimited setting for the broker to use to verify that the JWT was issued for one of the expected audiences. The JWT will be inspected for the standard OAuth aud claim and if this value is set, the broker will match the value from JWT's aud claim to see if there is an exact match. If there is no match, the broker will reject the JWT and authentication will fail. |
GATEWAY_OAUTH_JWKS_REFRESH | NULL | The (optional) value in milliseconds for the broker to wait between refreshing its JWKS (JSON Web Key Set) cache that contains the keys to verify the signature of the JWT. |
GATEWAY_OAUTH_JWKS_RETRY | NULL | The (optional) value in milliseconds for the initial wait between JWKS (JSON Web Key Set) retrieval attempts from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting |
GATEWAY_OAUTH_JWKS_MAX_RETRY | NULL | The (optional) value in milliseconds for the maximum wait between attempts to retrieve the JWKS (JSON Web Key Set) from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms setting |
GATEWAY_OAUTH_SCOPE_CLAIM_NAME | NULL | The OAuth claim for the scope is often named scope, but this (optional) setting can provide a different name to use for the scope included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim. |
GATEWAY_OAUTH_SUB_CLAIM_NAME | NULL | The OAuth claim for the subject is often named sub, but this (optional) setting can provide a different name to use for the subject included in the JWT payload's claims if the OAuth/OIDC provider uses a different name for that claim. |
PLAIN
These settings are used when credentials are managed on the Gateway, see Client Authentication for details.
| Environment Variable | Default Value | Description |
|---|---|---|
GATEWAY_USER_POOL_SECRET_KEY | A default value is used to sign tokens and must be changed. | Used for the PLAIN mechanism when generating JWT tokens for clients. You must set a random value which is at least 256 bit long to ensure tokens can't be forged. |
SECURITY PROVIDER
| Environment Variable | Default Value | Description |
|---|---|---|
GATEWAY_SECURITY_PROVIDER | DEFAULT | Specify your security provider, can be DEFAULT (from your JRE), BOUNCY_CASTLE, BOUNCY_CASTLE_FIPS and CONSCRYPT |
Please note that CONSCRYPT does not support Mac OS with aarch64.
SECRET MANAGEMENT
Secrets may be passed from configuration to Gateway using environment variables. Some suggested examples are below that may be more common, but you are free to use your own and avoid any clashes with existing environment variables.
SCHEMA_REGISTRY_LOGINSCHEMA_REGISTRY_PASSWORDAWS_ACCESS_KEYAWS_SECRET_KEYAWS_SESSION_TOKENAZURE_CLIENT_IDAZURE_TENANT_IDAZURE_CLIENT_SECRETVAULT_TOKENVAULT_USERNAMEVAULT_PASSWORD- etc.
HTTP
| Environment Variable | Default Value | Description |
|---|---|---|
GATEWAY_HTTP_PORT | 8888 | The port on which the gateway will present the HTTP management API |
GATEWAY_SECURED_METRICS | true | Does the HTTP management API require authentication? |
GATEWAY_ADMIN_API_USERS | [{username: admin, password: conduktor, admin: true}] | Users that can access the api, please note that admin is required to do any write |
GATEWAY_HTTPS_CLIENT_AUTH | NONE | Client auth configuration for HTTPS incoming connection. Possible values: NONE, REQUEST, REQUIRED |
GATEWAY_HTTPS_KEY_STORE_PATH | Activates HTTPS and defines the keystore to use for TLS connection | |
GATEWAY_HTTPS_KEY_STORE_PASSWORD | Configures the password of the keystore used for HTTPS TLS connection | |
GATEWAY_HTTPS_TRUST_STORE_PATH | Configure the truststore used for the HTTPS TLS connection | |
GATEWAY_HTTPS_TRUST_STORE_PASSWORD | Configures the password of the truststore used for HTTPS TLS connection |
Internal State
To keep the Gateway instances stateless, internal state is stored in Kafka topics.
| Environment Variable | Default Value | Description |
|---|---|---|
GATEWAY_GROUP_ID | null | Set the group name for internal topic if not defined |
GATEWAY_STORE_TTL_MS | 604800000 | Time between full refresh |
GATEWAY_TOPIC_STORE_REAL_TOPIC_PARTITION_COUNT | -1 | Defaults to the one defined in your cluster settings |
GATEWAY_TOPIC_STORE_KCACHE_REPLICATION_FACTOR | -1 | Defaults to the one defined in your cluster settings |
GATEWAY_TOPIC_STORE_DISTRIBUTED_CATCHUP_TIMEOUT_IN_SECONDS | 1 | Duration for catchup |
Topic Names
| Environment Variable | Default Value | Description |
|---|---|---|
GATEWAY_LICENSE_TOPIC | _conduktor_gateway_license | Name of license topic |
GATEWAY_TOPIC_MAPPINGS_TOPIC | _conduktor_gateway_topicmappings | Name of topicMappings topic |
GATEWAY_USER_MAPPINGS_TOPIC | _conduktor_gateway_usermappings | Name of the user mapping topic |
GATEWAY_CONSUMER_OFFSETS_TOPIC | _conduktor_gateway_consumer_offsets | Name of the topic to store the offsets for concentrated topic consumption |
GATEWAY_INTERCEPTOR_CONFIGS_TOPIC | _conduktor_gateway_interceptor_configs | Name of interceptor config topic |
GATEWAY_ENCRYPTION_CONFIGS_TOPIC | _conduktor_gateway_encryption_configs | Name of encryption configuration topic |
GATEWAY_ACLS_TOPIC | _conduktor_gateway_acls | Name of the acl topic |
GATEWAY_AUDIT_LOG_TOPIC | _conduktor_gateway_auditlogs | Name of audit topic |
GATEWAY_VCLUSTERS_TOPIC | _conduktor_gateway_vclusters | Name of vclusters topic |
GATEWAY_GROUPS_TOPIC | _conduktor_gateway_groups | Name of groups topic |
Cluster Switching / Failover
For a fuller description of the failover experience see the failover how-to. Setup of environment variables is similar to normally connecting to a Kafka cluster, but you provide two sets, one for your main cluster, one for your failover cluster. You can also load a cluster-config file if you prefer.
| Environment Variable | Default Value | Description |
|---|---|---|
GATEWAY_BACKEND_KAFKA_SELECTOR | Indicates use of a file for config, and provide path to it e.g. 'file : { path: /cluster-config.yaml}' | |
KAFKA_MAIN_BOOTSTRAP_SERVERS | Bootstrap server of the main cluster | |
KAFKA_MAIN_SECURITY_PROTOCOL | Security protocol of the main cluster | |
KAFKA_MAIN_SASL_MECHANISM | SASL mechanism of the main cluster | |
KAFKA_MAIN_SASL_JAAS_CONFIG | SASL jaas config of the main cluster | |
KAFKA_FAILOVER_BOOTSTRAP_SERVERS | Bootstrap server of the failover cluster | |
KAFKA_FAILOVER_SECURITY_PROTOCOL | Security protocol of the failover cluster | |
KAFKA_FAILOVER_SASL_MECHANISM | SASL mechanism of the main cluster | |
KAFKA_FAILOVER_SASL_JAAS_CONFIG | SASL jaas config of the main cluster | |
KAFKA_FAILOVER_GATEWAY_ROLES | Set the Gateway into failover mode, set this to failover for this scenario |
Internal Setup
Threading
| Environment Variable | Default Value | Description |
|---|---|---|
GATEWAY_DOWNSTREAM_THREAD | number of cores | The number of threads dedicated to handling IO between clients and Conduktor Gateway |
GATEWAY_UPSTREAM_THREAD | number of cores | The number of threads dedicated to handling IO between Kafka and Conduktor Gateway |
Upstream Connection
| Environment Variable | Default Value | Description |
|---|---|---|
GATEWAY_UPSTREAM_CONNECTION_POOL_TYPE | NONE | Upstream connection pool type. Possible values are NONE (no connection pool), ROUND_ROBIN (Round robin selected connection pool) |
GATEWAY_UPSTREAM_NUM_CONNECTION | 10 | The number of connections between Conduktor Gateway and Kafka per upstream thread. Used only when ROUND_ROBIN is enabled. |
Feature Flags
| Environment Variable | Default Value | Description |
|---|---|---|
GATEWAY_FEATURE_FLAGS_AUDIT | true | Whether or not to enable the audit feature |
GATEWAY_FEATURE_FLAGS_INTERNAL_LOAD_BALANCING | true | Whether or not to enable we replicate kafka internal load balancing |
Licensing
| Environment Variable | Default Value | Description |
|---|---|---|
GATEWAY_LICENSE_KEY | None | License key |
Audit
| Environment Variable | Default Value | Description |
|---|---|---|
GATEWAY_AUDIT_LOG_CONFIG_SPEC_VERSION | 0.1.0 | Version |
GATEWAY_AUDIT_LOG_SERVICE_BACKING_TOPIC | _auditLogs | Target topic name |
GATEWAY_AUDIT_LOG_REPLICATION_FACTOR_OF_TOPIC | -1 | Replication factor to be used when creating the audit topic, defaults to the one defined in your cluster settings |
GATEWAY_AUDIT_LOG_NUM_PARTITIONS_OF_TOPIC | -1 | Number of partitions to be used when creating the audit topic, defaults to the one defined in your cluster settings |
GATEWAY_AUDIT_LOG_KAFKA_ | Overrides Kafka Producer configuration for Audit Logs ie. GATEWAY_AUDIT_LOG_KAFKA_LINGER_MS=0 |
Logging
| Environment Variable | Default Value | Description | Package |
|---|---|---|---|
LOG4J2_APPENDER_LAYOUT | pattern | The format to output console logging. Use json for json layout or pattern for pattern layout | |
LOG4J2_IO_CONDUKTOR_PROXY_NETWORK_LEVEL | info | Low-level networking, connection mapping, authentication, authorization | io.conduktor.proxy.network |
LOG4J2_IO_CONDUKTOR_UPSTREAM_THREAD_LEVEL | info | Requests processing and forwarding. At trace, log requests sent | io.conduktor.proxy.thread.UpstreamThread |
LOG4J2_IO_CONDUKTOR_PROXY_REBUILDER_COMPONENTS_LEVEL | info | Requests and responses rewriting. Logs responses payload in debug (useful for checking METADATA) | io.conduktor.proxy.rebuilder.components |
LOG4J2_IO_CONDUKTOR_PROXY_SERVICE_LEVEL | info | Various. Logs ACL checks and interceptor targettings at debug. Logs post-interceptor requests/response payload at trace | io.conduktor.proxy.service |
LOG4J2_IO_CONDUKTOR_LEVEL | info | Get even more logs not covered by specific packages | io.conduktor |
LOG4J2_ORG_APACHE_KAFKA_LEVEL | warn | Kafka log level | org.apache.kafka |
LOG4J2_IO_KCACHE_LEVEL | warn | Kcache log level (our persistence library) | io.kcache |
LOG4J2_IO_VERTX_LEVEL | warn | Vertx log level (our HTTP API framework) | io.vertx |
LOG4J2_IO_NETTY_LEVEL | error | Netty log level (our network framework) | io.netty |
LOG4J2_IO_MICROMETER_LEVEL | error | Micrometer log level (our metrics framework) | io.micrometer |
LOG4J2_ROOT_LEVEL | info | Root logging level (applies to anything else) | (root) |
Product Analytics
| Environment Variable | Default Value | Description |
|---|---|---|
GATEWAY_FEATURE_FLAGS_ANALYTICS | true | Conduktor collects basic user analytics to understand product usage and enhance product development and improvement, such as a Gateway Started event. This is not based on any of the underlying Kafka data which is never sent to Conduktor. |