Skip to main content
To configure Conduktor Gateway, we recommend setting up environment variables. They can be set in the Gateway container or taken from a file. To make sure the values were set correctly, check the startup logs.

​
Use the Gateway container

  • Docker
  • Kubernetes
You can set the environment variables during the docker-run command with -e or --env:
docker run -d \
  -e KAFKA_BOOTSTRAP_SERVERS=kafka1:9092,kafka2:9092 \
  -e KAFKA_SECURITY_PROTOCOL=SASL_PLAINTEXT \
  -e KAFKA_SASL_MECHANISM=PLAIN \
  -e KAFKA_SASL_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username='usr' password='pwd';" \
  -p 6969:6969 \
  conduktor/conduktor-gateway:latest
Or in a docker-compose.yaml:
services:
  conduktor-gateway:
    image: conduktor/conduktor-gateway:latest
    ports:
      - 6969:6969
    environment:
      KAFKA_BOOTSTRAP_SERVERS: kafka1:9092,kafka2:9092
      KAFKA_SECURITY_PROTOCOL: SASL_PLAINTEXT
      KAFKA_SASL_MECHANISM: PLAIN
      KAFKA_SASL_JAAS_CONFIG: org.apache.kafka.common.security.plain.PlainLoginModule required username='usr' password='pwd';

​
Use a file

You can mount a file with the key-value pairs into the container and provide its path by setting the environment variable GATEWAY_ENV_FILE. Note these variables should be exported by the file as they are injected in the wrapper that starts the Gateway process.
Example
export KAFKA_BOOTSTRAP_SERVERS=kafka1:9092,kafka2:9092
export KAFKA_SECURITY_PROTOCOL=SASL_PLAINTEXT
export KAFKA_SASL_MECHANISM=PLAIN
export KAFKA_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username='usr' password='pwd';
You’ll get a confirmation in the logs: Sourcing environment variables from $GATEWAY_ENV_FILE (or a warning if the file isn’t found: Warning: GATEWAY_ENV_FILE is set but the file does not exist or is not readable.).

​
Networking

​
Port and SNI routing

Environment variableDescriptionDefault value
Common properties
GATEWAY_ADVERTISED_HOSTThe hostname returned in the Gateway’s metadata for clients to connect to.Your hostname
GATEWAY_ROUTING_MECHANISMDefines the routing method: port for port routing, host for SNI routing.port
GATEWAY_PORT_STARTThe first port the Gateway listens on.6969
GATEWAY_MIN_BROKERIDThe broker ID associated with the first port (GATEWAY_PORT_START). Should match the lowest broker.id (or node.id) in the Kafka cluster.0
GATEWAY_BIND_HOSTThe network interface the Gateway binds to.0.0.0.0
Port routing specific
GATEWAY_PORT_COUNTThe total number of ports used by Gateway.(maxBrokerId - minBrokerId) + 3
SNI routing specific
GATEWAY_ADVERTISED_SNI_PORTThe port returned in the Gateway’s metadata for clients to connect to when using SNI routing.GATEWAY_PORT_START
GATEWAY_ADVERTISED_HOST_PREFIXConfigures the advertised broker names.broker
GATEWAY_SECURITY_MODEDefines where authentication takes place, Gateway or Kafka. Valid values are: GATEWAY_MANAGED, KAFKA_MANAGED.The default value depends on combination of GATEWAY_SECURITY_PROTOCOL and KAFKA_SECURITY_PROTOCOL. See security defaults
GATEWAY_SECURITY_PROTOCOLDefines the security protocol that clients should use to connect to Gateway. **Has to be set to SSL or SASL_SSL when in GATEWAY_MANAGED security mode, or SASL_SSL when in KAFKA_MANAGED security mode, for SNI routing.The default value depends on combination of GATEWAY_SECURITY_PROTOCOL and KAFKA_SECURITY_PROTOCOL. See security defaults
GATEWAY_SNI_HOST_SEPARATORThe separator used to construct returned metadata.-
As of Gateway 3.10.0, the DELEGATED_XXX security protocols have been deprecated in favour of an additional environment variable, GATEWAY_SECURITY_MODE.The DELEGATED values remain supported for backward compatibility but are no longer recommended for new configurations.If you’re using DELEGATED security protocols, check out the security mode migration guide before proceeding.

​
Load balancing

Environment variableDescriptionDefault value
GATEWAY_CLUSTER_IDA unique identifier for a given Gateway cluster, used to establish Gateway cluster membership for load balancing.gateway
GATEWAY_FEATURE_FLAGS_INTERNAL_LOAD_BALANCINGWhether to use Conduktor Gateway’s internal load balancer to balance connections between Gateway instances.true
GATEWAY_RACK_IDSimilar to broker.rack.

​
HTTP API

Environment variableDescriptionDefault value
GATEWAY_HTTP_PORTThe port on which Gateway will present the HTTP management API.8888
GATEWAY_SECURED_METRICSDetermines whether the HTTP management API requires authentication.true
GATEWAY_ADMIN_API_USERSUsers that can access the API. Admin access is required for write operations. To grant read-only access, set admin: true.[{username: admin, password: conduktor, admin: true}]
HTTPS configuration
GATEWAY_HTTPS_KEY_STORE_PATHEnables HTTPS and specifies the keystore to use for TLS connections.
GATEWAY_HTTPS_KEY_STORE_PASSWORDSets the password for the keystore used in HTTPS TLS connections.
GATEWAY_HTTPS_CLIENT_AUTHClient authentication configuration for mTLS. Possible values: NONE, REQUEST, REQUIRED.NONE
GATEWAY_HTTPS_TRUST_STORE_PATHSpecifies the truststore used for mTLS.
GATEWAY_HTTPS_TRUST_STORE_PASSWORDPassword for the truststore defined above.

​
Upstream connection

Environment variableDescriptionDefault value
GATEWAY_UPSTREAM_CONNECTION_POOL_TYPEUpstream connection pool type. Possible values are NONE (no connection pool), ROUND_ROBIN (Round robin selected connection pool)NONE
GATEWAY_UPSTREAM_NUM_CONNECTIONThe number of connections between Conduktor Gateway and Kafka per upstream thread. Used only when ROUND_ROBIN is enabled.10

​
Licensing

Environment variableDescriptionDefault value
GATEWAY_LICENSE_KEYLicense keyNone

​
Connect from Gateway to Kafka

Conduktor Gateway’s connection to Kafka is configured by the KAFKA_ environment variables. When translating Kafka’s properties, use upper case instead and replace the . with _. For example: When defining Gateway’s Kafka property bootstrap.servers, declare it as the environment variable KAFKA_BOOTSTRAP_SERVERS. Any variable prefixed with KAFKA_ will be treated as a connection parameter by Gateway.

​
Connect from clients to Gateway

Environment variableDescriptionDefault value
GATEWAY_SECURITY_MODEDefine where authentication takes place, Gateway or Kafka. Valid values are: GATEWAY_MANAGED, KAFKA_MANAGED. Note that KAFKA_MANAGED mode is incompatible with PLAINTEXT or SSL Gateway security protocols.The default value depends on combination of GATEWAY_SECURITY_PROTOCOL and KAFKA_SECURITY_PROTOCOL. See security defaults
GATEWAY_SECURITY_PROTOCOLThe type of authentication clients should use to connect to Gateway. Valid values are: PLAINTEXT, SASL_PLAINTEXT, SASL_SSL, SSL.The default value depends on combination of GATEWAY_SECURITY_MODE and KAFKA_SECURITY_PROTOCOL. See security defaults
GATEWAY_FEATURE_FLAGS_MANDATORY_VCLUSTERIf no virtual cluster was detected, the user then automatically falls back into the transparent virtual cluster called passthrough. If set to true, reject authentication if the principal (the identifier the authentication process provides) is not mapped to a named virtual cluster with a service account. This must be set to true when using the Partner Zone feature of Conduktor Exchange.false
GATEWAY_AUTO_CREATE_TOPICS_ENABLEDEnable auto-create topics feature. When enabled, topics can be automatically created when producing or consuming through Gateway, leveraging the Kafka property auto.create.topics.enable. Authorization: When enabled, users require either CLUSTER resource with CREATE permission (allows creating any topic) or TOPIC resource with CREATE permission for specific topics. Warning: this feature doesn’t support concentrated topics. When auto-create topics is enabled, topics that would normally be concentrated not be, they’ll simply be created as regular physical topics instead. Take caution when enabling this setting.false
GATEWAY_ACL_ENABLEDEnable/disable ACL support on the Gateway transparent virtual cluster (passthrough) only.When GATEWAY SECURITY MODE is set to
- GATEWAY_MANAGED : true.
- KAFKA_MANAGED: false.
- Unset and undetermined: false.
GATEWAY_SUPER_USERSSemicolon-separated (;) list of service accounts that will be super users on Gateway (excluding virtual clusters).
Example: alice;bob.		Usernames from GATEWAY_ADMIN_API_USERS
GATEWAY_ACL_STORE_ENABLEDObsolete, use the VirtualCluster resource. Enable/disable ACLs support for Virtual Clusters only.false
GATEWAY_AUTHENTICATION_CONNECTION_MAX_REAUTH_MSForce the client re-authentication after this amount of time. If set to 0, we never force the client to re-authenticate until the next connection0
GATEWAY_SHUTDOWN_DELAY_BETWEEN_BROKERS_MSThe pause between disconnection of broker clients during shutdown process. Set to 1000 or higher for librdkafka clients.0
During shutdown, Gateway closes client connections in controlled manner to simulate rolling Kafka broker restart, with GATEWAY_SHUTDOWN_DELAY_BETWEEN_BROKERS_MS pause between brokers. For librdkafka clients, set GATEWAY_SHUTDOWN_DELAY_BETWEEN_BROKERS_MS to 1000 or higher to prevent them from crashing during shutdown because of ALL_BROKERS_DOWN error.
As of Gateway 3.10.0, the DELEGATED_SASL_SSL security protocol has been deprecated in favour of an additional environment variable GATEWAY_SECURITY_MODE.The default behavior of GATEWAY_ACL_ENABLED has changed, instead derived from security mode when left unset.The DELEGATED values remain supported for backward compatibility but are no longer recommended for new configurations.If you’re using DELEGATED security protocols, check out the migration guide before proceeding.

​
Security defaults

This decision tree explains how Gateway determines default values for GATEWAY_SECURITY_PROTOCOL and GATEWAY_SECURITY_MODE when one or both are not explicitly set.

​
Scenario 1: Both GATEWAY_SECURITY_PROTOCOL and GATEWAY_SECURITY_MODE are not set

  • Infer security values based on Kafka configuration
    Kafka Security ProtocolInferred GATEWAY_SECURITY_PROTOCOLInferred GATEWAY_SECURITY_MODE
    SASL_PLAINTEXTSASL_PLAINTEXTKAFKA_MANAGED
    SASL_SSLSASL_SSLKAFKA_MANAGED
    PLAINTEXTPLAINTEXTGATEWAY_MANAGED
    SSLSSLGATEWAY_MANAGED

​
Scenario 2: GATEWAY_SECURITY_PROTOCOL is not set but GATEWAY_SECURITY_MODE is

  • Infer security values based on Kafka configuration and Gateway security mode
    • Note we will not infer an invalid combination
GATEWAY_SECURITY_MODEKafka Security ProtocolInferred GATEWAY_SECURITY_PROTOCOL
KAFKA_MANAGEDSASL_PLAINTEXTSASL_PLAINTEXT
KAFKA_MANAGEDSASL_SSLSASL_SSL
GATEWAY_MANAGEDPLAINTEXTPLAINTEXT
GATEWAY_MANAGEDSSLSSL
Any other combinationNo default; we will ask you to set manually

​
Scenario 3: GATEWAY_SECURITY_PROTOCOL is set but GATEWAY_SECURITY_MODE is not

GATEWAY_SECURITY_PROTOCOLInferred GATEWAY_SECURITY_MODE
SASL_PLAINTEXTGATEWAY_MANAGED
SASL_SSLGATEWAY_MANAGED
PLAINTEXTGATEWAY_MANAGED
SSLGATEWAY_MANAGED
DELEGATED_SASL_PLAINTEXT (deprecated)KAFKA_MANAGED
DELEGATED_SASL_SSL (deprecated)KAFKA_MANAGED

​
SSL authentication

Environment variableDescriptionDefault value
Keystore
GATEWAY_SSL_KEY_STORE_PATHPath to a mounted keystore for SSL connections
GATEWAY_SSL_KEY_STORE_PASSWORDPassword for the keystore defined above
GATEWAY_SSL_KEY_PASSWORDPassword for the key contained in the store above
GATEWAY_SSL_KEY_TYPEjksor pkcs12jks
GATEWAY_SSL_UPDATE_CONTEXT_INTERVAL_MINUTESInterval in minutes to refresh SSL context5
Truststore (for mTLS)
GATEWAY_SSL_TRUST_STORE_PATHPath to a keystore for SSL connections
GATEWAY_SSL_TRUST_STORE_PASSWORDPassword for the truststore defined above
GATEWAY_SSL_TRUST_STORE_TYPEjks, pkcs12jks
GATEWAY_SSL_CLIENT_AUTHNONE will not request client authentication, OPTIONAL will request client authentication, REQUIRE will require client authenticationNONE
GATEWAY_SSL_PRINCIPAL_MAPPING_RULESmTLS leverages SSL mutual authentication to identify a Kafka client. Principal for mTLS connection can be detected from the subject certificate using the same feature as in Apache Kafka, the SSL principal mapping Extracts the subject
GATEWAY_SSL_ENABLED_PROTOCOLSComma-separated list of TLS protocol versions; restricts which protocols are offered during SSL/TLS handshake. If not provided it will fall back to the default protocols of security provider configured by GATEWAY_SECURITY_PROVIDER.
GATEWAY_SSL_CIPHER_SUITESComma-separated list of cipher suites; restricts cryptographic algorithms that are offered during SSL/TLS handshake. If not provided it will fall back to the default ciphers of security provider configured by GATEWAY_SECURITY_PROVIDER.

​
OAuthbearer

Some of these definitions (e.g. SASL_OAUTHBEARER_JWKS_ENDPOINT_REFRESH) are taken from Kafka documentation .
Environment variableDescription
GATEWAY_OAUTH_JWKS_URLThe OAuth/OIDC provider URL from which the provider’s JWKS (JSON Web Key Set) can be retrieved. The URL can be HTTP(S)-based or file-based.
GATEWAY_OAUTH_EXPECTED_ISSUERThe (optional) setting for the broker to use to verify that the JWT was created by the expected issuer. The JWT will be inspected for the standard OAuth iss claim and if this value is set, the broker will match it exactly against what is in the JWT’s iss claim. If there’s no match, the broker will reject the JWT and authentication will fail
GATEWAY_OAUTH_EXPECTED_AUDIENCESThe (optional) comma-delimited setting for the broker to use to verify that the JWT was issued for one of the expected audiences. The JWT will be inspected for the standard OAuth aud claim and if this value is set, the broker will match the value from JWT’s aud claim to see if there is an exact match. If there’s no match, the broker will reject the JWT and authentication will fail.
GATEWAY_OAUTH_JWKS_REFRESHThe (optional) value in milliseconds for the broker to wait between refreshing its JWKS (JSON Web Key Set) cache that contains the keys to verify the signature of the JWT.
GATEWAY_OAUTH_JWKS_RETRYThe (optional) value in milliseconds for the initial wait between JWKS (JSON Web Key Set) retrieval attempts from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts, up to a maximum wait length specified by sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms.
GATEWAY_OAUTH_JWKS_MAX_RETRYThe (optional) value in milliseconds for the maximum wait between attempts to retrieve the JWKS (JSON Web Key Set) from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the sasl.oauthbearer.jwks.endpoint.retry.backoff.ms setting and will double in wait length between attempts, up to a maximum wait length specified by sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms.
GATEWAY_OAUTH_SCOPE_CLAIM_NAMEThe OAuth claim for the scope is often named scope but this (optional) setting can provide a different name to use for the scope included in the JWT payload’s claims, if the OAuth/OIDC provider uses a different name for that claim.
GATEWAY_OAUTH_SUB_CLAIM_NAMEThe OAuth claim for the subject is often named sub, but this (optional) setting can provide a different name to use for the subject included in the JWT payload’s claims, if the OAuth/OIDC provider uses a different name for that claim.
GATEWAY_OAUTH_USE_CC_POOL_IDSet to true to use the Confluent Cloud pool ID as the principal name. This is useful for Confluent Cloud users in Delegated mode who want to use the pool ID as the principal name instead of the sub claim.

​
Principal Resolver

Environment variableDescriptionDefault value
GATEWAY_PRINCIPAL_RESOLVERThe principal resolver to use. Currently only supported is CONFLUENT_CLOUD.
GATEWAY_CONFLUENT_CLOUD_API_KEYThe Confluent Cloud API key (not the Kafka cluster API key).
GATEWAY_CONFLUENT_CLOUD_API_SECRETThe Confluent Cloud API secret.
GATEWAY_CONFLUENT_CLOUD_CACHE_SIZEThe number of principals to cache.1000
GATEWAY_CONFLUENT_CLOUD_CACHE_EXPIRY_MSThe cache expiry time in milliseconds.86400000 (1 day)
Find out more about the principal resolver.

​
Plain authentication

Environment variableDescriptionDefault value
GATEWAY_USER_POOL_SECRET_KEYBase64 encoded value of 256bits long (e.g. openssl rand -base64 32). If using SASL_PLAIN or SASL_SSL, you have the ability to create local service accounts on Gateway. These service accounts will have credentials generated by Gateway based on the GATEWAY_USER_POOL_SECRET_KEY.No default value is provided. You must provide this value for all deployments

​
Security provider

Environment variableDescriptionDefault value
GATEWAY_SECURITY_PROVIDERSpecify your security provider. It can be: DEFAULT (from your JRE), BOUNCY_CASTLE, BOUNCY_CASTLE_FIPS and CONSCRYPT. Please note that CONSCRYPT doesn’t support Mac OS with aarch64.DEFAULT
GATEWAY_USER_POOL_SERVICE_ACCOUNT_REQUIREDIf true, verify the existence of user mapping for the service account when the user connects in Gateway Managed SASL/PLAIN mode. Note that this variable is deprecated in Gateway v3.9.0. The behaviour is the same as if it was set to true.false

​
Cluster switching / failover

Setting up your Kafka clusters for failover is similar to the standard setup, but you need to provide two sets of properties: one for your main cluster and one for your failover cluster. You can define these properties as environment variables or load a cluster configuration file.
Environment variableDescription
GATEWAY_BACKEND_KAFKA_SELECTORIndicates the use of a configuration file and provides its path, e.g.: 'file: { path: /cluster-config.yaml }'.
KAFKA_FAILOVER_GATEWAY_ROLESTo turn Gateway into failover mode, set this to failover.
Main Cluster
KAFKA_MAIN_BOOTSTRAP_SERVERSBootstrap server.
KAFKA_MAIN_SECURITY_PROTOCOLSecurity protocol.
KAFKA_MAIN_SASL_MECHANISMSASL mechanism.
KAFKA_MAIN_SASL_JAAS_CONFIGSASL JAAS config.
Failover Cluster
KAFKA_FAILOVER_BOOTSTRAP_SERVERSBootstrap server.
KAFKA_FAILOVER_SECURITY_PROTOCOLSecurity protocol.
KAFKA_FAILOVER_SASL_MECHANISMSASL mechanism.
KAFKA_FAILOVER_SASL_JAAS_CONFIGSASL JAAS config.

​
Internal topics

As Gateway is stateless, it uses Kafka topics to store its internal state. Use the following environment variables to configure these internal topics. If missing, Gateway will automatically create the topics (if it has the permission to do so). You can also create the topics independently of Gateway, just make sure they are configured as described below.

​
Internal state

Firstly, there are some general configuration settings for Gateway internal state management which apply to all used topics.
Environment variableDescriptionDefault value
GATEWAY_GROUP_IDSet a consumer group name which will be used by Gateway to consume the internal license topic. This consumer group will be used by Gateways from the same cluster to recognize each other.conduktor_${GATEWAY_CLUSTER_ID}
GATEWAY_STORE_TTL_MSTime between full refresh.604800000
GATEWAY_TOPIC_STORE_KCACHE_REPLICATION_FACTORDefaults to the value defined in your cluster settings.-1

​
Topic names

Environment variableDescriptionDefault value
GATEWAY_LICENSE_TOPICTopic where the license is stored._conduktor_${GATEWAY_CLUSTER_ID}_license
GATEWAY_TOPIC_MAPPINGS_TOPICTopic where the topics aliases are stored._conduktor_${GATEWAY_CLUSTER_ID}_topicmappings
GATEWAY_USER_MAPPINGS_TOPICTopic where the service accounts are stored._conduktor_${GATEWAY_CLUSTER_ID}_usermappings
GATEWAY_CONSUMER_OFFSETS_TOPICTopic where the offsets for concentrated topic consumption are stored._conduktor_${GATEWAY_CLUSTER_ID}_consumer_offsets
GATEWAY_INTERCEPTOR_CONFIGS_TOPICTopic where the deployed Interceptors are stored._conduktor_${GATEWAY_CLUSTER_ID}_interceptor_configs
GATEWAY_ENCRYPTION_CONFIGS_TOPICTopic where the encryption configuration is stored, in specific cases._conduktor_${GATEWAY_CLUSTER_ID}_encryption_configs
GATEWAY_ACLS_TOPICTopic where the ACLs managed by Gateway are stored._conduktor_${GATEWAY_CLUSTER_ID}_acls
GATEWAY_AUDIT_LOG_TOPICTopic where Gateway audit log is stored._conduktor_${GATEWAY_CLUSTER_ID}_auditlogs
GATEWAY_VCLUSTERS_TOPICTopic where the virtual clusters are stored._conduktor_${GATEWAY_CLUSTER_ID}_vclusters
GATEWAY_GROUPS_TOPICTopic where the service account groups are stored._conduktor_${GATEWAY_CLUSTER_ID}_groups
GATEWAY_ENCRYPTION_KEYS_TOPICName of the topic for storing EDEKs when gateway KMS enabled in encryption Interceptors_conduktor_${GATEWAY_CLUSTER_ID}_encryption_keys
GATEWAY_DATA_QUALITY_TOPICTopic where the data quality violation are stored._conduktor_${GATEWAY_CLUSTER_ID}_data_quality_violation

​
Required topic configuration

The most important setting is log.cleanup.policy which defines the clean up policy for the topic. Most of the topics used by Gateway are compacted, but some use time-based retention. If this isn’t set up properly, Gateway will throw an error on startup. Set the following:
  • log.cleanup.policy=compact for compaction
  • log.cleanup.policy=delete for time based retention
If Gateway creates the topics for you, it will set the right values. The second vital setting is the replication factor. This should be set to at least 3 in production environments to ensure that the data is safe (Gateway will warn you on startup, if it’s set to less than three). When creating topics, Gateway uses the default value for your Kafka brokers for this setting. For partition count, most of the topics are low volume and can operate well with only a single partition. This isn’t enforced (Gateway will work with multi partition topics for internal state), however there is no need to have more than one partition. The exception to this is the audit log topic which can have a lot of events written to it, if enabled for a busy cluster. We recommend starting with 3 partitions for audit logs (this doesn’t affect Gateway performance as it is a writer, not a reader), but will impact any other consumers you may run reading from it.
TopicCleanup policyRecommended partitionsOther configuration
_conduktor_${GATEWAY_CLUSTER_ID}_licensecompact1
_conduktor_${GATEWAY_CLUSTER_ID}_topicmappingscompact1
_conduktor_${GATEWAY_CLUSTER_ID}_usermappingscompact1
_conduktor_${GATEWAY_CLUSTER_ID}_consumer_offsetscompact1
_conduktor_${GATEWAY_CLUSTER_ID}_interceptor_configscompact1
_conduktor_${GATEWAY_CLUSTER_ID}_encryption_configscompact1
_conduktor_${GATEWAY_CLUSTER_ID}_aclscompact1
_conduktor_${GATEWAY_CLUSTER_ID}_vclusterscompact1
_conduktor_${GATEWAY_CLUSTER_ID}_groupscompact1
_conduktor_${GATEWAY_CLUSTER_ID}_encryption_keyscompact1
_conduktor_${GATEWAY_CLUSTER_ID}_auditlogsdelete3We recommend a retention time of around 7 days for this topic due to its high volume.
_conduktor_${GATEWAY_CLUSTER_ID}_data_quality_violationdelete1

​
Internal setup

​
Threading

Environment variableDescriptionDefault value
GATEWAY_DOWNSTREAM_THREADThe number of threads dedicated to handling IO between clients and Conduktor Gateway.Number of cores
GATEWAY_UPSTREAM_THREADThe number of threads dedicated to handling IO between Kafka and Conduktor Gateway.Number of cores

​
Feature flags

Environment variableDescriptionDefault value
GATEWAY_FEATURE_FLAGS_AUDITWhether or not to enable the audit feature.true
GATEWAY_FEATURE_FLAGS_INTERNAL_LOAD_BALANCINGWhether or not to enable Gateway’s internal load balancing.true
GATEWAY_FEATURE_FLAGS_BLOCK_UNSUPPORTED_APISWhether to block Kafka APIs that aren’t explicitly supported by Gateway. See API blocking modes.false

​
API blocking modes

The GATEWAY_FEATURE_FLAGS_BLOCK_UNSUPPORTED_APIS environment variable controls how Gateway handles Kafka APIs that aren’t explicitly supported. Two modes:
  • Permissive mode (false, default): Gateway allows unsupported APIs to pass through - maintains backward compatibility
  • Restrictive mode (true): Gateway blocks unsupported APIs for enhanced security
When to use each:
  • Use permissive (default) when upgrading or if you have legacy applications
  • Use restrictive when you want maximum security and only allow known, supported APIs
Impact of each mode: Permissive mode (false):
  • ✅ Backward compatibility: Legacy applications continue to work
  • ✅ Smooth upgrades: No breaking changes during Gateway updates
  • ⚠️ Security risk: Unsupported APIs bypass Gateway’s security controls
Restrictive mode (true):
  • ✅ Enhanced security: Only explicitly supported APIs are allowed
  • ✅ Controlled environment: Prevents access to potentially problematic APIs
  • ❌ Breaking changes: Applications using unsupported APIs will fail
APIs not supported by Gateway: Gateway categorizes unsupported APIs into three groups:
  1. Stability-blocked APIs (always blocked):
  • DESCRIBE_TOPIC_PARTITIONS - New API for describing topic partitions
  • CONSUMER_GROUP_DESCRIBE - New consumer group protocol API (KIP-848)
  • CONSUMER_GROUP_HEARTBEAT - New consumer group protocol API (KIP-848)
  1. Security-blocked APIs (blocked when ACLs enabled):
  • SHARE_GROUP_DESCRIBE - Share consumer group API (KIP-932)
  • SHARE_GROUP_HEARTBEAT - Share consumer group API (KIP-932)
  • SHARE_FETCH - Share consumer fetch API (KIP-932)
  • SHARE_ACKNOWLEDGE - Share consumer acknowledge API (KIP-932)
  1. Unknown/Unsupported APIs (affected by this setting):
  • KIP-932 Share Group APIs (completely unsecured):
    • INITIALIZE_SHARE_GROUP_STATE - Initialize share group state. This is an inter-broker RPC authorized as a cluster action.
    • READ_SHARE_GROUP_STATE - Read share group state. This is an inter-broker RPC authorized as a cluster action.
    • WRITE_SHARE_GROUP_STATE - Write share group state. This is an inter-broker RPC authorized as a cluster action.
    • DELETE_SHARE_GROUP_STATE - Delete share group state. This is an inter-broker RPC authorized as a cluster action.
    • READ_SHARE_GROUP_STATE_SUMMARY - Read share group state. This is an inter-broker RPC authorized as a cluster action.
  • KIP-1000 Configuration APIs (bypass ACL checks):
    • LIST_CLIENT_METRICS_RESOURCES (KIP-1000) - List client metrics configuration resources without DESCRIBE_CONFIGS permission
  • Internal/Admin APIs (not meant for client use):
    • LEADER_AND_ISR - Internal leader election and ISR management
    • STOP_REPLICA - Internal replica management
    • UPDATE_METADATA - Internal metadata updates
    • CONTROLLED_SHUTDOWN - Internal broker shutdown coordination
    • WRITE_TXN_MARKERS - Internal transaction marker management
    • DESCRIBE_QUORUM - Internal quorum management
    • ENVELOPE - Internal request envelope handling
    • UNREGISTER_BROKER - Internal broker registration management
    • ADD_RAFT_VOTER - Internal Raft consensus management
    • REMOVE_RAFT_VOTER - Internal Raft consensus management
  • Future Kafka APIs not yet implemented in Gateway
  • Any other APIs that may bypass Gateway’s security controls or don’t work correctly with Gateway features
Security implications: When GATEWAY_FEATURE_FLAGS_BLOCK_UNSUPPORTED_APIS=false (permissive mode):
  • Unsupported APIs are passed through to Kafka without Gateway processing
  • Critical security risk: KIP-932 Share Group APIs bypass all ACL checks completely
  • Configuration exposure: KIP-1000 APIs allow access to sensitive configuration without proper permissions
  • Internal API access: Internal/Admin APIs meant for broker-to-broker communication may be accessible to clients
  • Topic mapping and other Gateway features may not work correctly
  • Potential for unauthorized access to restricted resources and information disclosure
When GATEWAY_FEATURE_FLAGS_BLOCK_UNSUPPORTED_APIS=true (restrictive mode):
  • All unsupported APIs are blocked at the Gateway level
  • Prevents security vulnerabilities: Blocks KIP-932 and KIP-1000 APIs that bypass ACL checks
  • Prevents internal API access: Blocks internal/Admin APIs that should not be accessible to clients
  • Enhanced security through explicit allow-list approach
  • Applications must use only Gateway-supported APIs
Example of problematic situation: ListClientMetricsResources API bypasses ACL checks: When GATEWAY_FEATURE_FLAGS_BLOCK_UNSUPPORTED_APIS=false (permissive mode), clients can call the ListClientMetricsResources API without proper authorization. This API is designed to list all client metrics configuration resources in the cluster. Attack scenario:
  1. Unauthorized client connects to Gateway without DESCRIBE_CONFIGS permission on CLUSTER resource
  2. Client calls ListClientMetricsResources API directly
  3. Gateway forwards the request to Kafka without ACL validation (since it’s not in the supported APIs list)
  4. Kafka responds with complete list of client metrics configuration resources
  5. Client receives sensitive configuration information including:
    • All configured client metrics resources
    • Internal cluster configuration details
    • Resource names and configurations that should be protected
Security impact:
  • Information disclosure: Unauthorized access to sensitive cluster configuration
  • ACL bypass: Complete circumvention of Gateway’s permission system
  • Compliance violation: Access to restricted data without proper authorization
  • Reconnaissance: Attackers can enumerate cluster resources for further attacks
Why this happens: The ListClientMetricsResources API is not included in Gateway’s supported APIs list, so when permissive mode is enabled, it bypasses all Gateway security controls and is forwarded directly to Kafka without ACL validation.
This setting only affects unsupported APIs. APIs that are explicitly blocked for security or stability reasons (like consumer group management) will always be blocked, regardless of this setting.

​
Gateway internal timeout

These timeout thresholds are used by Gateway and typically don’t need modification since they match Apache Kafka’s defaults. If your Kafka brokers or clients use non-default timeout values, you may want to adjust these.
Environment variableDescriptionDefault value
GATEWAY_INFLIGHT_REQUEST_EXPIRY_MSTimeout for in-flight requests. If Kafka doesn’t respond before this delay, Gateway will respond with a request timeout to the client. Default is 300000. It has to exceed the consumer’s max.poll.interval.ms and client’s request.timeout.ms because JOIN_GROUP or SYNC_GROUP requests can take up to that duration per KIP-62.330000
GATEWAY_UPSTREAM_MAX_IDLE_TIME_MSMaximum time Gateway connections can remain idle before being closed, in milliseconds. Default is 600000. It has to exceed the Kafka broker’s connections.max.idle.ms and should be set higher than GATEWAY_INFLIGHT_REQUEST_EXPIRY_MS.660000

​
Monitoring

​
Audit

Environment variableDescriptionDefault value
GATEWAY_AUDIT_LOG_CONFIG_SPEC_VERSIONVersion of the log.0.1.0
GATEWAY_AUDIT_LOG_SERVICE_BACKING_TOPICTarget topic name._auditLogs
GATEWAY_AUDIT_LOG_REPLICATION_FACTOR_OF_TOPICReplication factor to be used when creating the audit topic, defaults to the one defined in your cluster settings.-1
GATEWAY_AUDIT_LOG_NUM_PARTITIONS_OF_TOPICNumber of partitions to be used when creating the audit topic, defaults to the one defined in your cluster settings.-1
GATEWAY_AUDIT_LOG_KAFKA_Overrides Kafka Producer configuration for audit logs, i.e.: GATEWAY_AUDIT_LOG_KAFKA_LINGER_MS=0
GATEWAY_AUDIT_LOG_EVENT_TYPESComma-separated list of event types for audit log, defaults to ALL to generate audit log for all event types.ALL
GATEWAY_AUDIT_LOG_EVENT_TYPES valid value is ALL (default) or a comma-separated list of the following:
  • CONNECTION for initial connection event, i.e. receiving Kafka’s API_VERSIONS request
  • AUTHENTICATION for authentication success or failure event
  • REST_API for all REST v1 admin requests and all REST v2 requests (see API reference)
  • ACL_ADMIN for ACL management events done via Kafka Admin API, i.e. CREATE_ACLS and DELETE_ACLS requests
  • AUTHORIZATION for authorization failure events, i.e. when ACL permissions deny an operation on a resource
Find out more about Gateway audit event types.

​
Logging

Environment variableDescriptionDefault valuePackage
LOG4J2_APPENDER_LAYOUTThe format to output Console logging. Use json for json layout or pattern for pattern layout.pattern
LOG4J2_IO_CONDUKTOR_PROXY_NETWORK_LEVELLow-level networking, connection mapping, authentication, authorization.infoio.conduktor.proxy.network
LOG4J2_IO_CONDUKTOR_UPSTREAM_THREAD_LEVELRequests processing and forwarding. At trace, log requests sent.infoio.conduktor.proxy.thread.UpstreamThread
LOG4J2_IO_CONDUKTOR_PROXY_REBUILDER_COMPONENTS_LEVELRequests and responses rewriting. Logs responses payload in debug (useful for checking METADATA).infoio.conduktor.proxy.rebuilder.components
LOG4J2_IO_CONDUKTOR_PROXY_SERVICE_LEVELVarious. Logs ACL checks and Interceptor targeting at debug. Logs post-interceptor requests/response payload at trace.infoio.conduktor.proxy.service
LOG4J2_IO_CONDUKTOR_LEVELGet even more logs not covered by specific packages.infoio.conduktor
LOG4J2_ORG_APACHE_KAFKA_LEVELKafka log level.warnorg.apache.kafka
LOG4J2_IO_KCACHE_LEVELKcache log level (our persistence library).warnio.kcache
LOG4J2_IO_VERTX_LEVELVertx log level (our HTTP API framework).warnio.vertx
LOG4J2_IO_NETTY_LEVELNetty log level (our network framework).errorio.netty
LOG4J2_IO_MICROMETER_LEVELMicrometer log level (our metrics framework).errorio.micrometer
LOG4J2_ROOT_LEVELRoot logging level (applies to anything else that hasn’t been listed above).info(root)

​
Product analytics

Environment variableDescriptionDefault value
GATEWAY_FEATURE_FLAGS_ANALYTICSConduktor collects basic user analytics to understand product usage and enhance product development and improvement, such as a Gateway Started event. This is not based on any of the underlying Kafka data which is never sent to Conduktor.true

​
Data Quality topic configs

Environment variableDescriptionDefault value
GATEWAY_DATA_QUALITY_TOPICTarget topic name_conduktor_${GATEWAY_CLUSTER_ID}_data_quality_violation
GATEWAY_DATA_QUALITY_TOPIC_REPLICATION_FACTORReplication factor to be used when creating the data quality topic, defaults to the one defined in your cluster settingscluster default
GATEWAY_DATA_QUALITY_TOPIC_PARTITIONSNumber of partitions to be used when creating the data quality topic, defaults to the one defined in your cluster settingscluster default
GATEWAY_DATA_QUALITY_TOPIC_RETENTION_HOURRetention period (in hours) to be used when creating the data quality topic168 (7 days)