# Kafka proxy and governance
Source: https://docs.conduktor.io/gateway
Conduktor Gateway is a transparent Kafka proxy that enforces encryption, governance policies, traffic control, data masking, and multi-tenancy.
Conduktor Gateway is a network proxy for Apache Kafka, complete with an extensible and dynamic plugin mechanism that can be used to add technical and business logic on top of your existing Kafka deployment.
**Learn more:** [Conduktor Gateway](https://www.conduktor.io/gateway) The product overview: what Gateway does and the problems it solves.
## Gateway benefits
This can be used to provide functionality that is not available in Kafka natively, such as:
* **Centrally configure encryption** at the field-level or full payload.
* **Mask sensitive data** at the proxy layer.
* **Enforce governance and traffic control policies** — rate limits, data quality rules, topic creation constraints.
* **Leverage multi-tenancy** with Virtual Clusters and topic concentration.
* **Empower development teams** with self-service governance.
Supports all Kafka providers (Confluent, AWS MSK, Redpanda, Aiven, Apache Kafka).
Gateway is one of the main [Conduktor components](/guide/conduktor-in-production/deploy-artifacts) and together with [Console](/guide/conduktor-in-production/deploy-artifacts/deploy-console) make up the Conduktor platform:
## How it works
Gateway acts as a Kafka-compliant middle layer between clients and Kafka clusters.
### Authentication
Supports multiple security protocols:
* PLAINTEXT
* SSL
* SASL SSL
* mTLS
### Interceptors
Use Interceptors to add logic like encryption or filtering. [Find out about Interceptors](/guide/conduktor-concepts/interceptors).
## Related resources
* [Deploy Gateway](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway)
* [Gateway resource reference](/guide/reference/gateway-reference)
* [Gateway API](https://developers.conduktor.io/?product=gateway)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
* [Arrange a technical demo](https://www.conduktor.io/contact/demo)
# Kafka cost allocation and chargeback
Source: https://docs.conduktor.io/guide/conduktor-concepts/chargeback
Track and allocate Kafka costs to teams with Conduktor Chargeback. Measure storage, partitions, ingress and egress per cluster, application or label.
Chargeback is a cost allocation feature that helps you track, visualize and attribute Kafka infrastructure costs to clusters, teams or applications.
By measuring resource use over time, Chargeback gives you visibility into Kafka spending, supports cost accountability and helps you make informed decisions about resource optimization.
Conduktor measures five cost axes — **storage**, **partitions**, **ingress**, **egress** and **Kafka Connect** — and presents costs through three views: by cluster, by application or by label.
From Console v1.45.0, Chargeback unifies the previous **with Gateway** and **without Gateway** experiences into a single page. The previous V1 and V2 endpoints are deprecated but still respond to existing API clients.
**From our blog:** [Chargeback: attribute and map Kafka costs to your business](https://conduktor.io/blog/chargeback-attribute-map-kafka-costs-to-your-business) Practical strategies to tag, trace, and report Kafka spending by team and application.
## Cost axes
Each cluster has its own configurable unit costs across five axes. Set the costs that match your provider pricing or your internal infrastructure model.
| Cost axis | Unit | What it measures |
| ----------------- | ----------------- | -------------------------------------------- |
| **Storage** | \$/GB/hour | Bytes retained on disk per topic |
| **Partitions** | \$/partition/hour | Number of partitions per topic |
| **Ingress** | \$/GB | Bytes produced into Kafka |
| **Egress** | \$/GB | Bytes consumed from Kafka |
| **Kafka Connect** | \$/task/hour | Hours that Kafka Connect connector tasks run |
A cluster only needs the axes that match its pricing model. For an on-premises cluster, you might only set storage and partitions; for a Confluent Cloud cluster, you can set all five.
Kafka Connect costs use a default per-task-hour rate for the cluster. You can also set per-connector-class overrides — for example, a higher rate for an S3 sink connector than for a JDBC source connector.
Costs are calculated at query time, so updating a unit cost takes effect immediately across historical data.
## What you can measure per cluster type
Chargeback collects metrics from four sources: Console's metadata indexer for storage and partition counts, the Confluent Cloud Metrics API for Confluent Cloud throughput, the observability Interceptor for Gateway throughput, and Console's Kafka Connect integration for connector task hours.
| Cluster type | Storage | Partitions | Ingress / egress per topic | Ingress / egress per service account | Kafka Connect |
| --------------- | :-----: | :--------: | :------------------------: | :----------------------------------: | :-----------: |
| Vanilla Kafka | ✓ | ✓ | – | – | ✓ |
| Confluent Cloud | ✓ | ✓ | ✓ | ✓ (no topic breakdown) | ✓ |
| Gateway | ✓ | ✓ | ✓ | ✓ | ✓ |
Kafka Connect costs apply to any cluster type, but only when the cluster has one or more Kafka Connect clusters configured in Console. Clusters without Kafka Connect don't show Connect costs.
Storage and partition metrics are bucketed by UTC day. Confluent Cloud throughput is collected hourly. Gateway throughput is streamed continuously through the observability Interceptor and flushed at a configurable interval.
## Views
Chargeback presents data through three top-level views, each with the same cost columns and the ability to drill down into topics, service accounts and connectors.
* **Cluster view**: costs grouped by Kafka cluster
* **Application view**: costs grouped by application, with drill-down into application instances
* **Label view**: costs grouped by the values of a label key (for example, `env=production` or `team=payments`)
Label-based grouping requires the label key to first be selected as a Chargeback Label by an administrator.
## Chargeback Labels
Not every label attached to a topic or service account is meaningful for cost attribution. Administrators select a subset of label keys to expose in Chargeback.
Once a label key is enabled, you can group costs by its values. Topics or service accounts without that label are excluded from the grouping. Chargeback Labels are global — the same set of label keys applies across all clusters.
## Permissions
Access to Chargeback comes at two levels: platform-wide and per application.
### Platform-wide access
Two platform-level permissions grant access to every application's costs:
* **Can view Chargeback** (`PlatformChargebackView`): view cost metrics across all views and drill-downs
* **Can manage Chargeback** (`PlatformChargebackManage`): set per-cluster unit costs and select Chargeback Labels
By default, platform admins have both permissions. Console shows cost attribution for all clusters regardless of the cluster-level permissions assigned to the user.
### Per-application access
Application owners can see the costs of their own applications without platform-wide access. Assign the `applicationInstanceChargebackView` instance permission through an [ApplicationGroup](/guide/reference/self-service-reference#instance-permissions-reference) — Chargeback then appears for those members, scoped to the applications they hold the permission on.
This permission isn't granted by default, including to groups with full access on their application instances. Assign it explicitly to opt an application into cost visibility.
Because application teams manage their own ApplicationGroups, a team can assign this permission to itself. To keep cost data centralized, platform teams can block it with a ResourcePolicy — see [keeping Chargeback data centralized](/guide/reference/self-service-reference#keeping-chargeback-data-centralized).
## Related resources
* [Use Chargeback](/guide/use-cases/chargeback)
* [Configure Chargeback](/guide/tutorials/configure-chargeback)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Kafka data products and governance
Source: https://docs.conduktor.io/guide/conduktor-concepts/data-products
Build governed Kafka data products with Conduktor: combine self-service ownership, schema enforcement, topic policies, and resource metadata.
## Overview
Conduktor allows you to effectively *create* data products by combining multiple features and functionality.
**From our blog:** [Stop calling your Kafka topics data products](https://conduktor.io/blog/kafka-data-products) Why a raw topic is shared mutable state, and what it takes to wrap one into a real data product.
## Ownership and governance
[Self-service](/guide/conduktor-concepts/self-service) provides the foundation:
* Applications define ownership and business context
* Application Instances manage resources (topics, consumer groups, subjects) across environments
* Topic policies enforce configuration standards (retention, partitions)
* Resource labels add business metadata (data-criticality, descriptions, documentation URLs)
## Discovery
Topic Catalog and Application Catalog enable discovery:
* Search topics by name, labels or owning application
* View topic metadata and business context
* Find applications and their owned resources
* Request access to topics owned by other teams
## Quality and validation
[Data quality policies](/guide/conduktor-concepts/data-quality-policies) ensure data meets standards:
* CEL (Common Expression Language) expression rules for business logic validation
* JSON schema validation
* Built-in rules (for example, EnforceAvro for schema registry integration)
* Policy violations are tracked and can block or mark non-compliant data
## Access control and sharing
Control who can access your data products:
* Application Instance Permissions enable controlled sharing between internal teams
* [Partner Zones](/guide/conduktor-concepts/partner-zones) enable secure external data sharing with third parties
* Service accounts and ACLs (Access Control Lists) enforce access at the Kafka level
* RBAC (Role-Based Access Control) in Console controls who can manage resources
## Related resources
* [Use Self-service](/guide/use-cases/self-service)
* [Share data externally](/guide/use-cases/third-party-data)
* [Enforce data quality](/guide/use-cases/enforce-data-quality)
# Kafka data quality — validation rules and policies
Source: https://docs.conduktor.io/guide/conduktor-concepts/data-quality-policies
Define Kafka data quality rules using CEL expressions or JSON Schema. Attach policies to topics to observe violations or enforce blocking with Conduktor.
Conduktor helps you detect and enforce data quality standards for messages flowing through Kafka.
## How it works
You create **Rules** that define the expected format and content of messages, then attach them to **Policies** that target specific Kafka topics or topic prefixes.
Conduktor evaluates these Rules against every message produced on targeted topics after Policy creation and tracks the count of violating messages as well as the total number of messages evaluated.
## Observe vs enforce
The capabilities depend on whether you're using Gateway:
**Without Gateway (observe only)**
* Records are verified **after** they have been produced
* Track violations and monitor data quality issues
* No impact on message flow
**With Gateway (observe and enforce)**
* Records are verified **before** production
* See which service accounts produced the faulty records
* Take action on faulty records:
* **Block**: reject the message (and the entire record batch). The producer receives a non-retriable error. [Find out about handling blocked batches](/guide/conduktor-in-production/admin/gateway-policies#handle-blocked-batches)
* **Mark**: add a header to the record containing all violated rules for every Policy using the mark action
* Prevent bad data from entering your topics
### Key constraints
* Policies cannot mix different target cluster types (Gateway vs. non-Gateway)
* You can't target a Gateway-backing cluster directly to avoid validation conflicts
**From our blog:** [Kafka data contracts: a schema is not a contract](https://conduktor.io/blog/kafka-data-contracts) Why a schema alone won't stop a producer from breaking consumers, and where to enforce the contract.
## Data quality metrics
The quality overview dashboard provides a summary of your data quality governance across the entire ecosystem.
Use it to:
* **Track progress**: monitor how many topics are protected by Policies
* **Identify gaps**: find high-volume topics without coverage
* **Measure health**: see which producers need attention and how well your Policies are performing. The health score is calculated based on the number of topics that have Policies assigned, with a modifier that determines how effectively they're enforced (for example, whether violations are blocked).
* **Take action**: click through to add topics to Policies or investigate violations
In Console, access the overview by going to **Trust > Data quality overview**. You can export all data quality metrics to CSV for offline analysis or reporting.
To see metrics, you need to have at least one topic with a Policy assigned to it.
### Coverage
The coverage metrics show how many topics are protected by Policies.
* **Topics with at least one Policy**: percentage of topics with Policy coverage
* **Topics with multiple Policies**: percentage of topics with layered validation
* **VIP topics at risk**: highly utilized topics with no Policy coverage that should be prioritized.
VIP topics are those with activity within the last 24 hours, containing over 500 messages or topics that have more than 3 consumers.
Higher coverage means more of your data is being validated. Aim for 100% coverage of VIP topics first.
To take action and add a Policy to a topic at risk, click **Add to Policy** next to the topic name and select an existing Policy. Metrics are updated every 5 minutes by default and changes may not be reflected immediately.
### Health
The health metrics show how effectively your Rules are being enforced.
* **Health score**: overall data quality based on coverage and enforcement
* **Action distribution**: breakdown of no action, mark and block actions over time
* **Top violating producers**: producers with the highest violation rates to help prioritize remediation
Mark and block actions and producer identification require Gateway. Without Gateway, violations are reported but not enforced and you will not see information about the producer of violations on non-Gateway topics.
## Related resources
* [Observe data quality](/guide/use-cases/observe-data-quality)
* [Enforce data quality](/guide/use-cases/enforce-data-quality)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Gateway authentication and authorization
Source: https://docs.conduktor.io/guide/conduktor-concepts/gateway-authentication-authorization
Configure Kafka client authentication in Conduktor Gateway: Gateway-managed mode with local service accounts and Kafka-managed mode with delegated auth.
Conduktor Gateway provides flexible authentication and authorization for Kafka clients, allowing you to choose where and how clients are authenticated and what permissions they have.
## Two authentication modes
Gateway offers two distinct modes for managing client authentication and authorization:
**Gateway-managed mode**
* Gateway handles all authentication and authorization
* Service accounts and ACLs defined in Gateway
* Supports both local and external service accounts
* Local service accounts authenticate with `PLAIN` (token) or, with the service accounts feature flag enabled, `SCRAM-SHA-512` / `SCRAM-SHA-256` credentials managed by Gateway
* Full control over client access without touching Kafka configuration
* Enables Virtual Clusters and other Gateway-specific features
* Enables efficient connection pooling ([`GATEWAY_UPSTREAM_CONNECTION_POOL_TYPE=ROUND_ROBIN`](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables)) to multiplex client connections onto a small pool of connections per broker, cutting connection overhead on Kafka as the number of clients grows
**Kafka-managed mode**
* Authentication and authorization delegated to the backing Kafka cluster
* Existing Kafka service accounts and ACLs continue to work
* Useful for gradual migration to Gateway
* External service accounts can still be mapped for friendly names in Gateway
* Virtual resources (Virtual Clusters, alias topics) not available
* Connection pooling not available (each client authenticates on its own connection)
## Key decisions
When configuring Gateway authentication, you need to decide:
* **Where to authenticate**: at Gateway or delegate to Kafka
* **Authentication method**: SASL (PLAIN, SCRAM, OAUTHBEARER), mTLS, or anonymous
* **Service account type**: local (Gateway-managed) or external (identity provider)
* **Authorization location**: Gateway ACLs or Kafka ACLs
## Benefits
* **Gradual adoption**: start with Kafka-managed mode and migrate to Gateway-managed
* **Unified access control**: manage authentication across multiple clusters from one place
* **Flexible identity integration**: work with existing identity providers or use Gateway's built-in authentication
* **Enhanced security**: add Gateway policies and Interceptors without changing Kafka security
## Related resources
* [View supported authentication methods table](/guide/reference/gateway-reference#supported-authentication-methods)
* [Gateway service accounts](/guide/conduktor-concepts/gateway-service-accounts)
* [Manage service accounts using Gateway](/guide/tutorials/manage-gateway-service-accounts)
* [Manage service accounts and ACLs using Console](/guide/manage-kafka/kafka-resources/service-accounts-acls)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Gateway service accounts
Source: https://docs.conduktor.io/guide/conduktor-concepts/gateway-service-accounts
Manage Kafka client identities in Conduktor Gateway. Use local service accounts for isolated access or external accounts from OIDC, mTLS.
Gateway service accounts are identities used to authenticate clients connecting to Conduktor Gateway.
## Two types of service accounts
Gateway supports two types of service accounts, each suited for different use cases:
**Local service accounts**
* Created and managed entirely within Gateway using the admin API
* No external identity provider required
* Gateway generates credentials with configurable time-to-live
* Can also hold long-lived SCRAM credentials for `SCRAM-SHA-512` / `SCRAM-SHA-256` authentication when the service accounts feature flag is enabled
* Ideal for sharing data with external partners without managing a separate identity provider
* Only available in Gateway-managed authentication mode
**External service accounts**
* Managed by external identity providers (OIDC, mTLS certificates)
* Can be mapped to friendly names in Gateway for better readability
* Original identity from provider (like Azure UUID) is preserved but aliased
* Used when integrating with existing enterprise identity systems
* Works in both Gateway-managed and Kafka-managed modes
* In Kafka-managed mode, external service accounts can rename identities and be referenced in Interceptors
## Benefits
* **Flexible identity management**: choose between self-managed or provider-managed identities
* **Simplified access control**: apply ACLs and Interceptors using friendly names
* **Audit trail**: all service account activity logged in Gateway audit logs
* **Virtual Cluster association**: link service accounts to specific Virtual Clusters for multi-tenancy
## Related resources
* [View Gateway service account reference](/guide/reference/gateway-reference#gatewayserviceaccount)
* [Manage service accounts and ACLs using Console](/guide/manage-kafka/kafka-resources/service-accounts-acls)
* [Manage service accounts using Gateway](/guide/tutorials/manage-gateway-service-accounts)
* [Gateway authentication and authorization](/guide/conduktor-concepts/gateway-authentication-authorization)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Indexing
Source: https://docs.conduktor.io/guide/conduktor-concepts/indexing
Console Indexing collects Kafka metadata every 30 seconds into internal tables for topic sorting, filtering, and labelling.
In Console, many of the list pages (e.g. Topics, Consumer Groups, Schema Registry, Kafka Connect) are served from an **internal table** instead of making *AdminClient* calls to Kafka.
These tables are updated by a background process we call Indexing. The Indexing process collects changes to your Kafka every 30 seconds and stores that metadata in internal tables.
Detail pages are not using Indexing. As soon as you're on a page for a specific topic or consumer group, the data is fetched directly from the Kafka cluster.
## Benefits
Indexing improves user experience and provides functionality that's not available with out-of-the-box Kafka resources, such as:
* smart tables with **sorting and filtering** capabilities, allowing you to get message count, size and much more
* **search** and **labelling** that allows you to organize and find required resources
## Examples
These are just some examples of how Indexing can help you:
* identify **idle topics**: which topics (that have no active consumers and producers) haven't published data for more than a week?
* are there any **over-partitioned topics** - topics that have a large number of partitions while the biggest consumer group only has a few members consuming in parallel?
* prioritize **data at risk topics**: which topics are at risk of losing data, because of the replication factor or the min ISR being incorrect?
* find **outliers topics**: which topics contain bad or overridden configurations that they shouldn't have?
* remove **over-replicated applications**: which consumer groups have idle members? Typically, this is because the number of consumers exceeded the number of total partitions.
## Circuit breaker configuration
Each indexed task in Console includes configurable circuit breaker behavior that provides fine-grained control over indexing resilience and recovery. This configuration helps prevent prolonged indexing outages and gives administrators better control over system behavior.
### Configuration parameters
**`sequentialFailureThreshold`**
* Controls how many consecutive failures have to occur before the circuit breaker activates
* When the threshold is reached, the indexing task will temporarily stop attempting to collect data
* Prevents resource exhaustion during extended connectivity or permissions issues
**`blockingDuration`**
* Defines how long the circuit breaker remains active before attempting to resume indexing
* Provides a controlled recovery mechanism with predictable timing
* Ensures predictable recovery intervals for indexing operations
### Benefits
With these circuit breaker settings, you can:
* Adjust failure detection sensitivity based on your environment's reliability
* Control recovery timing to balance system stability with data freshness
* Maintain predictable indexing behavior even during temporary infrastructure issues
* Prevent resource exhaustion during extended connectivity problems
Circuit breaker configuration is applied per indexed tasks (e.g., topic metadata polling, consumer group indexing). For guidance on optimal settings for your specific deployment scenario, [contact support](https://support.conduktor.io/hc/en-gb/requests/new) .
## Troubleshoot
Topics created 'now' would not be indexed until the next Indexing cycle.
This means they wouldn't appear in Console for up to 30 seconds.
To mitigate this, we've come up with a counter-measure: any user request to the topic list will ALWAYS make one *AdminClient* call to Kafka: `listTopics`.
It’s cheap, simple and will only return the topic names. So, when topics are listed in Console, 99% of the time Indexing will serve all the topics with all the columns (name, partitions, count, size, etc.) and 1% of the time Indexing will serve most topics except for one or two **not-indexed yet** where only the name will be available.
# Kafka interceptors — Gateway plugins for security and governance
Source: https://docs.conduktor.io/guide/conduktor-concepts/interceptors
Conduktor Gateway interceptors enforce Kafka encryption, data quality, access control, and traffic control without changing client code.
Interceptors are Gateway plugins that intercept and modify Kafka requests and responses in real time.
They help enforce policies, protect data, and add functionality without changing your applications or Kafka configuration.
**Learn more:** [Conduktor Gateway](https://www.conduktor.io/gateway) How Gateway and its Interceptors add encryption, governance, and traffic control to Kafka.
## What Interceptors do
Interceptors can:
* Encrypt and decrypt message fields or entire payloads
* Block or skip records that don't meet data quality rules
* Enforce producer settings like acks or compression
* Control topic creation parameters like replication factor or partition count
* Transform message content or headers
## How they work
Each Interceptor has a `priority` field — a number where lowest runs first. The same numeric order applies on both produce and consume; ordering doesn't reverse between directions. Priorities are typically spaced apart (for example, `100`, `200`, `300`) so you can insert Interceptors between existing ones without renumbering.
Interceptors fire on different paths depending on what they do:
* **Produce-only** — for example, `EncryptPlugin` runs as records flow from client to broker.
* **Consume-only** — for example, `DecryptPlugin` runs as records flow from broker to client.
* **Both paths** — for example, `LargeMessageHandlingPlugin` runs as records flow from client to broker and broker to client.
For example, an `EncryptPlugin`, `LargeMessageHandlingPlugin`, and `DecryptPlugin` configured on the same topic share one priority order across both paths:
```yaml theme={null}
# Priority 100: encrypt on produce
apiVersion: gateway/v2
kind: Interceptor
metadata:
name: myEncryptPlugin
scope:
vCluster: passthrough
spec:
pluginClass: io.conduktor.gateway.interceptor.EncryptPlugin
priority: 100
config:
topic: "topic.*"
kmsConfig:
vault:
uri: http://vault:8200
token: ${VAULT_TOKEN}
recordValue:
payload:
keySecretId: vault-kms://vault:8200/transit/keys/payload-key
algorithm: AES128_GCM
---
# Priority 200: offload large messages to cloud storage
apiVersion: gateway/v2
kind: Interceptor
metadata:
name: myLargeMessageHandlingPlugin
scope:
vCluster: passthrough
spec:
pluginClass: io.conduktor.gateway.interceptor.LargeMessageHandlingPlugin
priority: 200
config:
topic: "topic.*"
minimumSizeInBytes: 1024
localDiskDirectory: myStorage/
s3Config:
bucketName: myBucketName
uri: http://myexampleuri
region: us-east-1
---
# Priority 300: decrypt on consume
apiVersion: gateway/v2
kind: Interceptor
metadata:
name: myDecryptPlugin
scope:
vCluster: passthrough
spec:
pluginClass: io.conduktor.gateway.interceptor.DecryptPlugin
priority: 300
config:
topic: "topic.*"
kmsConfig:
vault:
uri: http://vault:8200
token: ${VAULT_TOKEN}
```
For why this ordering matters when offloading encrypted records to cloud storage, see [Combine with encryption](/guide/use-cases/manage-large-messages#combine-with-encryption).
Interceptors can also be:
* **Scoped** — apply to specific service accounts, groups or Virtual Clusters
* **Overridden** — more specific scopes take precedence over broader ones
Gateway applies Interceptors dynamically based on which client connects, so different teams can have different policies without separate clusters.
## Related resources
* [Interceptor reference](/guide/reference/interceptor-reference)
* [Configure data quality policies](/guide/use-cases/enforce-data-quality)
* [Apply traffic control policies](/guide/use-cases/apply-traffic-control-policies)
# Logical topics
Source: https://docs.conduktor.io/guide/conduktor-concepts/logical-topics
Create logical Kafka topics in Conduktor Gateway: alias topics for renaming, concentrated topics to reduce partition costs, and SQL or CEL filter topics.
Conduktor logical topics are abstractions of real Kafka topics that provide additional functionality that's not available by default. We offer the following types of logical topics:
1. [Alias topics](#alias-topics) are topics that can be accessed with a name (alias), but point to another, *real* topic behind the scenes. Alias topics can be very useful when you want to share topics but have sensitive naming conventions; or in scenarios where underlying topics might be frequently renamed.
2. [Concentrated topics](#concentrated-topics) transparently co-locate multiple topics in the same physical topic behind the scenes, acting as pointers to reduce costs on low-volume topics with large partition counts. They are completely transparent to consumers and producers and allow you to emulate different partition counts irrespective of the backing physical topic's partition count.
3. [Topic views](#topic-views) filter records from physical topics using SQL WHERE clauses. This is the recommended approach for filtering JSON data.
4. [SQL topics](#sql-topics) **DEPRECATED** use SQL language to query and filter an existing topic.
5. [CEL topics](#filter-topics-with-cel) **DEPRECATED** another way to filter an existing topic using CEL expressions instead of SQL.
## Alias topics
Alias topics act as pointers that target a specific physical topic, defined with the [AliasTopic resource](/guide/reference/gateway-reference#aliastopic).
One of Kafka's limitations is that you can't rename topics - an issue that is solved with alias topics. You can have a number of alias topics pointing to the same physical topic.
### Use
Gateway manages an alias topic mapping in its internal configuration by registering a target physical topic. This topic will be presented to Kafka clients like a regular topic. However, all requests for this topic will be forwarded to the physical topic.
This means that consumer groups, fetch and produce are shared. Also, the alias topic does not replace the original one.
For example, if you create an alias topic `applicationB_orders` that's pointing to a physical topic `orders`, a client that can access the physical topic would be able to see both topics.
### Limitations
* ACLs using delegated Kafka security SASL delegated security protocols aren't supported.
* Alias topics can't reference another alias topic.
## Concentrated topics
Occasionally, topics have to be created for logical, rather than technical reasons (e.g. to differentiate between business units) which can result in considerable overuse of Kafka resources.
Conduktor's topic concentration allows data from a set of topics to be represented on a single underlying topic. Clients connecting through Conduktor Gateway can use concentrated topics as usual without any additional configuration.
For example, let's say we have the following topics:
* us\_east\_orders - 100 partitions
* us\_west\_orders - 100 partitions
* emea\_orders - 100 partitions
* latam\_orders - 100 partitions
The total Kafka resource requirement is 400 partitions.
With topic concentration, all of these topics can be concentrated to a single topic, using only 1/4 of resources:
* concentrated\_orders - 100 partitions
### Configuration
Configure concentrated topics with a `ConcentrationRule`. For all available fields, see the [ConcentrationRule resource reference](/guide/reference/gateway-reference#concentrationrule).
To create concentrated topics, first deploy `ConcentrationRule`:
```yaml theme={null}
---
kind: ConcentrationRule
metadata:
name: concentration1
spec:
pattern: concentrated.*
physicalTopics:
delete: physical.topic
```
Then topics that match the *ConcentrationRule* `spec.pattern`:
```bash theme={null}
kafka-topics \
--bootstrap-server conduktor-gateway:6969 \
--topic concentrated.topicA \
--partitions 3
kafka-topics \
--bootstrap-server conduktor-gateway:6969 \
--topic concentrated.topicB \
--partitions 4
```
We now have two concentrated topics (`concentrated.topicA` and `concentrated.topicB`) with partition counts of 3 and 4 respectively, mapped to a single physical topic (`physical.topic`) with three partitions.
To ensure that consumers don't consume messages from other partitions or from other concentrated topics, we store the concentrated partition and the concentrated topic name in the record headers. Gateway will automatically filter the messages that should be returned to the consumer.
```bash theme={null}
curl -X PUT "http://conduktor-gateway:8888/gateway/v2/concentration-rule" \
-H "Content-Type: application/json" \
-u "admin:conduktor" \
-d '{
"kind": "ConcentrationRule",
"apiVersion": "gateway/v2",
"metadata": {
"name": "concentration1"
},
"spec": {
"pattern": "concentrated.*",
"physicalTopics": {
"delete": "physical.topic"
}
}
}'
```
### Limitations
#### Consumer offsets
When consuming from a concentrated topic, messages and ordering is always preserved but any metadata calculations (primarily lag and message count) are unlikely to be as expected.
This is because the associated metadata is from the backing Kafka topic, rather than the concentrated topic seen from the perspective of the consumer. This is a known limitation.
#### Compact and delete+compact topics
You can create concentrated topics with any *cleanup.policy*, but your `ConcentrationRule` has to have a backing topic for each of them, otherwise it won't let you create the topic.
```yaml theme={null}
---
kind: ConcentrationRule
metadata:
name: concentration1
spec:
pattern: concentrated.*
physicalTopics:
delete: physical.topic-delete
compact: physical.topic-compact
# deleteCompact: physical.topic-deletecompact
```
In this example, since the config for `spec.deleteCompact` is commented out, trying to create this topic will fail:
```bash theme={null}
kafka-topics --create
--bootstrap-server conduktor-gateway:6969 \
--topic \
--partitions 3 \
--config cleanup.policy=compact,delete
Error while executing topic command : Cleanup Policy is invalid
```
Backing topic cleanup policies are checked when you deploy a new `ConcentrationRule`. This prevents you from declaring a backing topic with a *cleanup.policy* of delete on the *ConcentrationRule* `spec.physicalTopic.compact` field.
#### Restricted topic configurations
The following list of topic properties are the only allowed properties for concentrated topics:
* `partitions`
* `cleanup.policy`
* `retention.ms`
* `retention.bytes`
* `delete.retention.ms`
If any other configuration than the above is set, the topic creation will fail with an error.
`retention.ms` and `retention.bytes` can be set to values lower or equal to the backing topic. If a user tries to create a topic with a higher value, topic creation will fail with an error:
```
kafka-topics --create
--bootstrap-server conduktor-gateway:6969 \
--topic \
--partitions 3 \
--config retention.ms=704800000
Error while executing topic command : Value '704800000' for configuration 'retention.ms' is incompatible with physical topic value '604800000'.
```
This behavior can be altered with the flag `spec.autoManaged`.
With concentrated topics, the enforced retention policy is the physical topic's retention policy, and not the policy requested at the concentrated topic creation time. The `retention.ms` and `retention.bytes` are not cleanup but retention guarantees.
#### Auto-managed backing topics
When `autoManaged` is enabled:
* backing topics are automatically created with the default cluster configuration and partition count.
* concentrated topics created with higher `retention.ms` and `retention.bytes` are allowed. This automatically extends the configuration of the backing topic.
```yaml theme={null}
---
kind: ConcentrationRule
metadata:
name: concentration1
spec:
pattern: concentrated.*
physicalTopics:
delete: physical.topic
autoManaged: true
```
Let's check the backing topic retention on the physical cluster:
```bash theme={null}
kafka-configs --bootstrap-server kafka:9092 \
--entity-type topics --entity-name physical.topic \
--describe
Configs for topic 'physical.topic' are:
cleanup.policy=delete
retention.ms=604800000
retention.bytes=-1
```
Let's try to create a concentrated topic with a higher retention on Gateway:
```bash theme={null}
kafka-topics --create
--bootstrap-server conduktor-gateway:6969 \
--topic \
--partitions 3 \
--config retention.ms=704800000
```
Let's review the backing topic again:
```bash theme={null}
kafka-configs --bootstrap-server kafka:9092 \
--entity-type topics --entity-name physical.topic \
--describe
Configs for topic 'physical.topic' are:
cleanup.policy=delete
retention.ms=704800000
retention.bytes=-1
```
As we can see, the retention has been updated.
If one user requests a topic with infinite retention (`retention.ms = -1`), **all the topics** with the same cleanup policy associated with the rule **will also inherit** this extended configuration and have infinite retention.
#### Message count, lag and offset (in)correctness
By default, concentrated topic reports the offsets of their backing topics. This impacts the calculations of **Lag** and **Message Count** that relies on partition **EndOffset** and group **CommittedOffset**.
Any tooling will currently display the message count, and the lag relative to the `EndOffset` of the physical topic. This can create confusion for customers and applications that will see incorrect metrics.
Gateway has an `offsetCorrectness` setting on `ConcentrationRule` that makes concentrated topics report virtual offsets, so that lag and message count read correctly. It is deprecated, and we don't recommend enabling it.
`spec.offsetCorrectness` is deprecated as of v3.21.0 and is planned for removal in v3.24.0. Existing rules keep working, but Gateway now logs a single deprecation warning at startup listing the affected concentrated topics. Because of the limitations listed below, treat the physical-topic offsets described above as the supported behavior.
If you already have it enabled, the setting looks like this:
```yaml theme={null}
---
kind: ConcentrationRule
metadata:
name: concentration1
spec:
pattern: concentrated.*
physicalTopics:
delete: physical.topic
offsetCorrectness: true
```
* `spec.offsetCorrectness` only applies to concentrated topics with the `cleanup.policy=delete`
* `spec.offsetCorrectness` is not retroactive on previously created concentrated topics
#### Stop using offset correction
If you have `offsetCorrectness` enabled in production, [get in touch for support with this migration](https://support.conduktor.io/hc/en-gb/requests/new?ticket_form_id=17438363566609).
#### Known issues with offset correctness
There are three known issues with the offset correctness in concentrated topics:
**1. Performance**
On startup, Gateway has to read the concentrated topic entirely before it's available to consumers. The end-to-end latency is increased by up to 500 ms (or `fetch.max.wait.ms`, if non-default).
**2. Memory**
Gateway consumes about \~250MB of heap memory per million records it's read in concentrated topics. This value is not bound, so we don't recommend offset correctness on high-volume topics, and recommend to size your JVM accordingly.
**3. Unsupported Kafka API**
* `DeleteRecords` is not supported
* Transactions are not supported
* Only `IsolationLevel.READ_UNCOMMITTED` is supported (using `IsolationLevel.READ_COMMITTED` is undefined behavior)
* Partition truncation (upon `unclean.leader.election=true`) may not be detected by consumers
**Very slow consumer group edge case**
Do not enable offset correctness when your topic has extended periods of inactivity.
When using topic concentration with `offsetCorrectness` enabled, there's currently a limitation for consumer groups where the data in the topics is slow moving, and/or the consumer groups are not committing their offsets frequently.
If a consumer group with a committed offset waits for the backing physical topic longer than the retention time (without committing a new offset), there's a possibility for that consumer group to become blocked.
In this scenario, a consumer group whose last committed offset has been removed from the topic, the group becomes blocked only if Gateway restarted before the next offset commit. If this limitation happens, the offsets for the affected consumer group will need to be manually reset for it to continue.
## Topic views
Topic views provide non-materialized views of physical Kafka topics, similar to database views. They apply transformations to records as they're consumed, without modifying the underlying data. Topic views support SQL-based filtering and projection on schema-less JSON, and SQL-based filtering on Avro, Protobuf and JSON-Schema records read through a [schema registry connection](#read-schema-encoded-topics). Support for additional transformations, for example CEL, is planned.
See a topic view give one team a filtered, projected read of a topic:
Topic views are generally available from Gateway 3.20.0. Most topic views created in the 3.19.x tech preview keep working; those with invalid SQL are now rejected and have to be recreated with valid SQL.
Existing consumer groups won't work after the upgrade — recreate them with a `group.id` that follows the new convention (see [Set the consumer group ID](#set-the-consumer-group-id)). A fresh group starts with no committed offsets. If you use Console, you can [duplicate the existing group](/guide/manage-kafka/kafka-resources/consumer-groups#duplicate-consumer-group) into the new name to carry your position over.
Topic views are configured as [a dedicated Gateway resource](/guide/reference/gateway-reference#topicview), similar to [alias topics](#alias-topics).
Topic views are a replacement for [SQL topics](#sql-topics) and [CEL topics](#filter-topics-with-cel), which are deprecated. Topic views will be extended to cover all of their functionality, after which the plugins will be removed.
Prefer a hands-on walkthrough? [Migrate a Kafka Streams transformer to a Topic View](/guide/tutorials/migrate-kafka-streams-to-topic-view) runs the whole before → after migration — creating the view, seeding offsets, and cutting a downstream consumer over with no reprocessing.
### How it works
When a consumer fetches from a topic view, Gateway:
1. Reads records from the underlying physical topic
2. Applies the configured transformation to filter or modify each record
3. Returns the resulting records to the consumer
Schema-encoded records whose schema ID is stored in a record header, the format [introduced in Confluent Platform 8.2](https://www.confluent.io/blog/schema-id-kafka-headers-data-governance), require **Gateway 3.21.0** or later.
Consumer offsets are committed against the physical topic because the topic view doesn't exist in Kafka's `__consumer_offsets`. This means you may see gaps in offset numbers when records are filtered out. The committed offset is always the offset of the last record in the batch, regardless of whether that record was filtered out.
Topic views are read-only by design, but Gateway doesn't enforce this yet. Producing to a topic view isn't blocked, so avoid it until enforcement is added.
### Configuration
Configure topic views using the Gateway REST API or Conduktor CLI. For all available fields, see the [TopicView resource reference](/guide/reference/gateway-reference#topicview).
```bash theme={null}
curl -X PUT "http://conduktor-gateway:8888/gateway/v2/topic-view" \
-H "Content-Type: application/json" \
-u "admin:conduktor" \
-d '{
"kind": "TopicView",
"apiVersion": "gateway/v2",
"metadata": {
"name": "uk-customers",
"vCluster": "passthrough"
},
"spec": {
"transformation": {
"type": "sql",
"statement": "SELECT * FROM customers WHERE country='\''UK'\''"
},
"onError": {
"type": "DROP"
}
}
}'
```
```yaml theme={null}
apiVersion: gateway/v2
kind: TopicView
metadata:
name: uk-customers
vCluster: passthrough
spec:
transformation:
type: sql
statement: "SELECT * FROM customers WHERE country='UK'"
onError:
type: DROP
```
Apply with:
```bash theme={null}
conduktor apply -f topic-view.yaml
```
### SQL transformation
When using `transformation.type: sql`, topic views filter and transform records using SQL syntax. The transformation uses a subset of SQL that will be extended in future releases.
Topic names with dash `-` characters have to be double quoted, as the dash is not a valid character for a SQL name. For example, if you have a topic `our-orders`, use `SELECT * FROM "our-orders" WHERE ...`
#### Projection
Use the `SELECT` clause to reshape records by renaming fields, selecting a subset of fields, computing new values, or including record metadata.
```sql theme={null}
-- Rename fields
SELECT name AS employee_name, salary FROM employees
-- Select a subset of fields (others are dropped)
SELECT name, department FROM employees
-- Compute values with CASE expressions
SELECT name, CASE WHEN salary > 2000 THEN 'high' ELSE 'low' END AS salary_band FROM employees
-- Include record metadata alongside value fields
SELECT name, record.offset AS offset, record.partition AS partition FROM employees
```
Projection applies to views over schema-less JSON. A view that names a schema registry connection has to use `SELECT *` — see [read schema-encoded topics](#read-schema-encoded-topics).
#### `CASE` expressions
A `CASE` expression maps conditions to fixed values:
```sql theme={null}
SELECT name,
CASE
WHEN salary > 5000 THEN 'high'
WHEN salary > 2000 THEN 'medium'
ELSE 'low'
END AS salary_band
FROM employees
```
Gateway takes the first branch whose condition holds, and the `ELSE` value when none do. The following rules apply:
* Each `WHEN` condition takes the same predicates as a `WHERE` clause, listed under supported predicates below.
* `THEN` and `ELSE` values have to be literals: a string, number, boolean, date, timestamp or time. A field reference or a computed expression isn't supported.
* `ELSE` is required. Without it, a record that matches no branch has no value to project.
* The switch form `CASE department WHEN 'sales' THEN …` isn't supported. Write the comparison out in each `WHEN` instead.
* A `CASE` can't sit inside a `JSON_OBJECT` value.
#### Nested output with `JSON_OBJECT`
Projection is flat by default: `SELECT a AS x, b` produces `{"x": ..., "b": ...}`, so a dotted path lands under a flat key. Use `JSON_OBJECT` to emit **nested** JSON instead:
```sql theme={null}
SELECT JSON_OBJECT(
'id' VALUE customerId,
'address' VALUE JSON_OBJECT(
'city' VALUE address.city,
'country' VALUE address.country
)
) AS payload
FROM customers
```
For this record:
```json theme={null}
{
"customerId": "c1",
"address": {
"city": "London",
"country": "UK",
"zip": "EC1"
}
}
```
`JSON_OBJECT` projects:
```json theme={null}
{
"id": "c1",
"address": {
"city": "London",
"country": "UK"
}
}
```
* Each `JSON_OBJECT` value is a field reference or another `JSON_OBJECT`, nested to any depth.
* Keys within one object have to be unique.
* When the whole `SELECT` is a single top-level `JSON_OBJECT`, that object becomes the record value and its alias (`AS payload`) is ignored. Alongside other columns, it nests under its own alias instead.
* An absent or `null` field is emitted as `null` — the key is always kept.
* Available for schema-less JSON only.
#### Supported predicates
* `=`, `>`, `>=`, `<`, `<=`, `<>` and `!=` (standard comparison operators)
* `REGEXP` (regular expression matching)
* `IN` / `NOT IN` with a literal list, e.g. `country IN ('UK', 'US')`
* `AND` operator for combining multiple conditions (`OR` is not yet supported)
* Parentheses around a single condition, e.g. `(country = 'UK') AND (amount > 10)`
#### Regular expression matching
Use `REGEXP` for pattern matching. SQL `LIKE` is not supported, so use `REGEXP` in place of `LIKE` wildcards:
```sql theme={null}
-- Match email addresses
SELECT * FROM users WHERE email REGEXP '^[a-z]+@[a-z]+\.com$'
-- Match records where name starts with 'Bo'
SELECT * FROM customers WHERE name REGEXP 'Bo.*'
-- Match addresses containing 'Engineering'
SELECT * FROM employees WHERE education.major REGEXP '.*Engineering'
```
#### Date and time values
A record's date and time values are typically stored as strings, since JSON has no native date or time type. To compare them chronologically rather than lexically, write the comparison value as a JDBC escape literal so Gateway reads it as a date or time:
* Date: `{d 'YYYY-MM-DD'}`
* Time: `{t 'HH:MM:SS'}`
* Timestamp: `{ts 'YYYY-MM-DD HH:MM:SS'}`
Gateway parses the record's string field into the same type and compares the two chronologically. If the field value can't be parsed as that type, the record doesn't match:
```sql theme={null}
-- order_date holds an ISO-8601 date string, e.g. "2026-01-01"
SELECT * FROM orders WHERE order_date = {d '2026-01-01'}
-- event_at holds a timestamp string, space or T separator, fractional seconds optional:
-- "2026-01-01 12:34:56" or "2026-01-01T12:34:56.123"
SELECT * FROM events WHERE event_at > {ts '2026-01-01 12:34:56'}
-- alarm_at holds a time string, e.g. "12:00:00"
SELECT * FROM events WHERE alarm_at <= {t '12:00:00'}
```
Use the JDBC escape form for date and time comparisons. Two alternatives don't work:
* A plain string literal (such as `event_at > '2026-01-01 12:34:56'`) is compared lexicographically, not chronologically. Gateway doesn't error, but range results can be wrong across mixed formats.
* The ANSI SQL forms `DATE '2026-01-01'` and `TIMESTAMP '2026-01-01 12:34:56'` aren't supported.
#### Nested field access
Access nested JSON fields using dot notation:
```sql theme={null}
-- Simple nested field
SELECT * FROM orders WHERE address.city = 'London'
-- Deeply nested field
SELECT * FROM users WHERE profile.settings.theme = 'dark'
-- Numeric comparison on nested field
SELECT * FROM metrics WHERE metadata.stats.count > 100
```
#### Array index access
Access array elements using bracket notation with backticks:
```sql theme={null}
-- First element (index 0)
SELECT * FROM orders WHERE `items[0].name` = 'apple'
-- Last element (negative index)
SELECT * FROM orders WHERE `items[-1].name` = 'banana'
-- Numeric array access
SELECT * FROM metrics WHERE `values[1]` > 50
-- Nested array field
SELECT * FROM events WHERE `data.records[0].status` = 'active'
```
Backticks are required around field names with array index notation.
#### Record key, headers, and Kafka metadata
In addition to record value fields, you can reference the record key, headers, and Kafka metadata:
```sql theme={null}
-- Value field, using the $ prefix for the payload
SELECT * FROM events WHERE $.severity = 'ERROR'
-- Raw record key, compared as a string
SELECT * FROM users WHERE record.key = 'account-42'
-- Nested key field when the key is JSON
SELECT * FROM users WHERE record.key.user.name = 'alice'
-- Header value, compared as a UTF-8 string
SELECT * FROM orders WHERE record.header.tenant = 'prod'
-- Partition, offset, or timestamp, compared as numbers
SELECT * FROM events WHERE record.timestamp > 1700000000000
-- These also work in the SELECT projection, not just WHERE filters
SELECT record.key AS key, record.header.tenant AS tenant, record.offset AS offset FROM orders
```
`record.timestamp` is milliseconds since the epoch. Header names that contain `.` aren't supported in this syntax.
#### Virtual Cluster topic names
If you use Virtual Clusters, the topic name in the `FROM` clause has to be the full physical name on the backing cluster, including the Virtual Cluster prefix. This applies even when the topic view targets a topic in the same Virtual Cluster.
For example, a topic view in the `vc-alice` Virtual Cluster that targets the `orders` topic in that cluster reads from the physical name `vc-aliceorders`:
```sql theme={null}
SELECT * FROM "vc-aliceorders" WHERE country = 'UK'
```
#### Unsupported syntax
Topic views support only the SQL features documented above, and Gateway rejects anything else when you create or update the view. The documented set is the contract: the list below names what people reach for most often, not every rejected construct.
**Operators:**
* `OR` operator (planned)
* `NOT`, including `NOT REGEXP` and `NOT LIKE`
* `LIKE` with wildcards
* `BETWEEN`
* `IS NULL` / `IS NOT NULL`
* `EXISTS`
* String concatenation (`||`)
* Numerical operators (`+`, `-`, `*`, `/` etc)
**Functions:**
* String functions (`UPPER`, `LOWER`, `CONCAT`)
* `COALESCE`
* `CAST`
**Query features:**
* `JOIN`, and more than one topic in `FROM`
* Subqueries anywhere, including `IN (SELECT …)` and a subquery as the `FROM` item
* A topic alias or other `FROM` modifier, such as `FROM orders o` or `PIVOT`
* `SELECT DISTINCT`
* `LIMIT`, `OFFSET`, `FETCH` and `TOP`
* `ORDER BY`
* `GROUP BY` and `HAVING`
* `WITH` (common table expressions)
* `SELECT … INTO`
* Set operations (`UNION`, `INTERSECT`, `EXCEPT`, `MINUS`)
**JSON expressions** that resolve to more than one value:
* path expressions returning an array or an object
* Array wildcards: `items[*].name` or `items.*.name`
* JSONPath filter expressions: `items[?(@.price > 10)]`
* Slices (`items[0:2]`), index lists (`items[0,1]`), and recursive descent (`..status`)
If any of these SQL features would be useful for your use case, let your Conduktor support contact know.
### Read schema-encoded topics
Schema registry external connections are a preview feature and subject to change, so reading schema-encoded topics is too — a topic view has no other way to reach a schema registry.
By default a topic view reads schema-less JSON. To read Avro, Protobuf or JSON-Schema records, set `spec.schemaRegistryConnectionName` to the name of a [schema registry external connection](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/external-connections#connect-to-a-schema-registry):
```yaml theme={null}
apiVersion: gateway/v2
kind: TopicView
metadata:
name: uk-customers
vCluster: passthrough
spec:
transformation:
type: sql
statement: "SELECT * FROM customers WHERE country='UK'"
onError:
type: DROP
schemaRegistryConnectionName: DEFAULT_SR
```
Gateway decodes each record through that connection, applies the `WHERE` clause and returns the records that match.
A schema-backed topic view has to use `SELECT *`. A `WHERE` clause is still allowed, because it filters whole records rather than selecting columns. Gateway rejects a projection at create and update time:
`Column projection is not yet supported for schema-registry-backed topic views; use SELECT * with a WHERE clause.`
Views over schema-less JSON keep full projection support.
Gateway rejects an unknown connection name with `400 Bad Request` when you create or update the view. Remove a connection a view already names and the view stays saved but stops serving traffic. [See what happens at startup](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/external-connections#what-happens-at-startup).
Decoding still needs the registry while Gateway runs: it fetches any schema the connection hasn't already cached. If the registry becomes unavailable, Gateway withholds the affected records and the consumer retries, rather than applying the view's `onError` strategy — so a registry outage stalls the view instead of dropping records.
### Error handling
The `spec.onError` field declares what Gateway should do when a record can't be transformed (for example, the SQL doesn't match the record structure). The field is required on every topic view and takes one of two values:
* `DROP` — drop the failing record from the response and continue.
* `FAIL_FETCH` — fail the fetch for that partition.
```yaml theme={null}
spec:
transformation:
type: sql
statement: "SELECT * FROM customers WHERE country='UK'"
onError:
type: DROP
```
### Set the consumer group ID
To consume from a topic view, include the view's name in your consumer's `group.id`, delimited by `::`. The rest of the `group.id` is free-form, so you can keep your usual naming alongside the view name.
```properties theme={null}
# Consume the uk_customers topic view
group.id=uk_customers::analytics
```
To read from more than one topic view with a single consumer, add each view name as a token:
```properties theme={null}
group.id=uk_customers::recent_orders::analytics
```
Each view named in a `group.id` has to have a different backing topic. You can't name two views of the same backing topic in the same `group.id`.
This keeps consumers of different views in separate Kafka consumer groups. Without it, Kafka can split a backing topic's partitions across views in the same group, and each consumer drops the records meant for the other, causing silent data loss.
Gateway checks the `group.id` when a consumer joins a group or commits offsets, and rejects it with an `INVALID_GROUP_ID` error if it doesn't follow this convention. The check also covers a topic view's backing topic, whether read directly or through an alias, since that's equivalent to reading the view. Concentrated topics are exempt, and a consumer that touches no topic view or backing topic is unaffected.
Requiring a specific `group.id` format is a temporary limitation. We plan to remove it in a future release, once Gateway supports the [KIP-848](https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol) consumer group protocol.
For a worked example of migrating an existing consumer onto a view — including seeding its starting offsets — see the [Kafka Streams → Topic View tutorial](/guide/tutorials/migrate-kafka-streams-to-topic-view).
### Planned features
Topic views apply SQL transformations to schema-less JSON, and SQL filtering to schema-encoded records. We plan to add:
* **More transformation types**: [CEL (Common Expression Language)](https://cel.dev/) and [RSQL](https://github.com/jirutka/rsql-parser), alongside SQL.
* **Projection on schema-encoded records**: selecting a subset of fields from Avro, Protobuf and JSON-Schema records, which today requires `SELECT *`.
## SQL topics
SQL topics are deprecated as of Gateway v3.19.0. They will be removed once [topic views](#topic-views) support the remaining SQL topic functionality.
SQL topics are incompatible with [incremental fetch sessions](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables#incremental-fetch-sessions), and return incorrect results if you enable that feature. [Topic views](#topic-views) don't have this limitation.
Conduktor Gateway's SQL topic feature uses a SQL-like language to filter and project messages, based on a simple SQL statement:
```sql theme={null}
SELECT
type,
price as amount,
color,
CASE
WHEN color = 'red' AND price > 1000 THEN 'Exceptional'
WHEN price > 8000 THEN 'Luxury'
ELSE 'Regular'
END as quality,
record.offset as record_offset,
record.partition as record_partition
FROM cars
```
This supports `FetchResponse` only (i.e., resulting topic is read-only):
`SELECT [list of fields] FROM [topic name] WHERE [field filter criteria]`
Topic names with dash `-` characters have to be double quoted, as the dash is not a valid character for a SQL name. For example, if you have a topic `our-orders`, use `SELECT * FROM "our-orders" WHERE ...`
Other limitations:
* With filter records based on more than one condition, only `AND` operator is supported
* Supported predicates: `=`, `>`, `>=`, `<`, `<=`, `<>` and `REGEXP` (RegExp MySQL Operator)
* Case expression is supported
* Filtered by:
* Record key (It supports SR):
* Record key as string: - `.. WHERE record.key = 'some thing'`
* Record key as schema: `.. WHERE record.key.someValue.someChildValue = 'some thing'`
* Record value (It supports SR): `.. WHERE $.someValue.someChildValue = 'some thing'`
* Partition: `.. WHERE record.partition = 1`
* Timestamp: `.. WHERE record.timestamp = 98717823712`
* Header: `.. WHERE record.header.someHeaderKey = 'some thing'`
* Offset: `.. WHERE record.offset = 1`
### Schemas and projections
If your data uses a schema, then it's not possible to make use of the projection feature here because the resulting data will no longer match the original schema. For plain JSON topics, you can use the `SELECT` clause to alter the shape of the data returned; however, for schema'd data (Avro and Protobuf) you must not use a projection, i.e. the select should be in the form:
`SELECT * FROM ...`
Filtering with the where clause is still supported.
### Configuration
| Key | Type | Description |
| :------------------- | :-------------- | :------------------------------------------------------------------------------------------------------------------ |
| virtualTopic | String | if virtualTopic exists, fetch this topic will get the data from the statement without configure it's own statement. |
| statement | String | SQL Statement |
| schemaRegistryConfig | Schema registry | [Schema registry configuration](#schema-registry-configuration) |
### Example
```json theme={null}
{
"name": "mySqlTopicPlugin",
"pluginClass": "io.conduktor.gateway.interceptor.VirtualSqlTopicPlugin",
"priority": 100,
"config": {
"virtualTopic": "legal_user",
"statement": "SELECT * FROM users WHERE age > 18",
"schemaRegistryConfig": {
"host": "http://schema-registry:8081"
}
}
}
```
#### Schema registry with secured template
```json theme={null}
{
"name": "mySqlTopicPlugin",
"pluginClass": "io.conduktor.gateway.interceptor.VirtualSqlTopicPlugin",
"priority": 100,
"config": {
"virtualTopic": "legal_user",
"statement": "SELECT * FROM users WHERE age > 18",
"schemaRegistryConfig": {
"host": "http://schema-registry:8081",
"additionalConfigs": {
"basic.auth.credentials.source": "${SR_BASIC_AUTH_CRED_SRC}",
"basic.auth.user.info": "${SR_BASIC_AUTH_USER_INFO}"
}
}
}
}
```
```yaml theme={null}
apiVersion: gateway/v2
kind: Interceptor
metadata:
name: mySqlTopicPlugin
scope:
vCluster: passthrough
spec:
pluginClass: io.conduktor.gateway.interceptor.VirtualSqlTopicPlugin
priority: 100
config:
virtualTopic: legal_user
statement: "SELECT * FROM users WHERE age > 18"
schemaRegistryConfig:
host: http://schema-registry:8081
additionalConfigs:
basic.auth.credentials.source: $${SR_BASIC_AUTH_CRED_SRC}
basic.auth.user.info: $${SR_BASIC_AUTH_USER_INFO}
```
Apply with:
```bash theme={null}
conduktor apply -f sql-topic-secured-template.yaml
```
## Filter topics with CEL
CEL topics are deprecated as of Gateway v3.17.0. They will be removed once [topic views](#topic-views) support CEL transformations.
CEL topics are incompatible with [incremental fetch sessions](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables#incremental-fetch-sessions), and return incorrect results if you enable that feature. [Topic views](#topic-views) don't have this limitation.
Conduktor Gateway's CEL topic feature uses CEL (Common Expression Language) expression to filter messages, based on a simple CEL expression in the form.
Currently
* Filtered by:
* Record key (It supports SR):
* Record key as string: - `.. record.key == 'some thing'`
* Record key as schema: `.. record.key.someValue.someChildValue == 'some thing'`
* Record value (It supports SR): `.. record.value.someValue.someChildValue == 'some thing'`
* Partition: `.. record.partition == 1`
* Timestamp: `.. record.timestamp == 98717823712`
* Header: `.. record.header.someHeaderKey == 'some thing'`
* Offset: `.. record.offset == 1`
### Configuration
| Key | Type | Description |
| :------------------- | :-------------------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------- |
| virtualTopic | String | When accessed, this virtual topic retrieves filtered data from the specified client topic, applying a CEL expression for data filtering. |
| topic | String | Specifies the client topic from which data is fetched. |
| expression | String | A CEL expression that returns BOOLEAN to filter data. This determines which data from the topic is relevant based on the given criteria. |
| schemaRegistryConfig | [Schema registry configuration](#schema-registry-configuration) | Schema registry configuration |
| celCacheSize | int | In memory cache size for CEL expressions, balancing speed and resource use, optimize performance. |
### Example
```json theme={null}
{
"name": "myCelTopicPlugin",
"pluginClass": "io.conduktor.gateway.interceptor.CelTopicPlugin",
"priority": 100,
"config": {
"virtualTopic": "legal_user",
"topic": "users",
"expression": "record.value.age > 18",
"schemaRegistryConfig": {
"host": "http://schema-registry:8081"
},
"celCacheSize": 100
}
}
```
#### Schema registry with secured template
```json theme={null}
{
"name": "myCelTopicPlugin",
"pluginClass": "io.conduktor.gateway.interceptor.CelTopicPlugin",
"priority": 100,
"config": {
"virtualTopic": "legal_user",
"topic": "users",
"expression": "record.value.age > 18",
"schemaRegistryConfig": {
"host": "http://schema-registry:8081",
"additionalConfigs": {
"basic.auth.credentials.source": "${SR_BASIC_AUTH_CRED_SRC}",
"basic.auth.user.info": "${SR_BASIC_AUTH_USER_INFO}"
}
},
"celCacheSize": 100
}
}
```
```yaml theme={null}
apiVersion: gateway/v2
kind: Interceptor
metadata:
name: myCelTopicPlugin
scope:
vCluster: passthrough
spec:
pluginClass: io.conduktor.gateway.interceptor.CelTopicPlugin
priority: 100
config:
virtualTopic: legal_user
topic: users
expression: "record.value.age > 18"
schemaRegistryConfig:
host: http://schema-registry:8081
additionalConfigs:
basic.auth.credentials.source: $${SR_BASIC_AUTH_CRED_SRC}
basic.auth.user.info: $${SR_BASIC_AUTH_USER_INFO}
celCacheSize: 100
```
Apply with:
```bash theme={null}
conduktor apply -f cel-topic-secured-template.yaml
```
## Schema registry configuration
Schema-encoded records whose schema ID is stored in a record header, the format [introduced in Confluent Platform 8.2](https://www.confluent.io/blog/schema-id-kafka-headers-data-governance), require **Gateway 3.21.0** or later.
Gateway supports **Confluent-like** and **AWS Glue** schema registries.
| Key | Type | Default | Description |
| --------------------- | ------ | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `type` | string | `CONFLUENT` | The type of schema registry to use: choose `CONFLUENT` (for Confluent-like schema registries including OSS Kafka) or `AWS` for AWS Glue schema registries. |
| `additionalConfigs` | map | | Additional properties maps to specific security-related parameters. For enhanced security, you can hide the sensitive values using [environment variables as secrets](/guide/reference/data-security#environment-variables-as-secrets). |
| **Confluent-like** | | | **Configuration for Confluent-like schema registries** |
| `host` | string | | URL of your schema registry. |
| `cacheSize` | string | `50` | Number of schemas that can be cached locally by this Interceptor so that it doesn't have to query the schema registry every time. |
| **AWS Glue** | | | **Configuration for AWS Glue schema registries** |
| `region` | string | | The AWS region for the schema registry, e.g. `us-east-1`. |
| `registryName` | string | | The name of the schema registry in AWS (leave blank for the AWS default of `default-registry`). |
| `basicCredentials` | string | | Access credentials for AWS. |
| **AWS credentials** | | | **AWS credential configuration** |
| `accessKey` | string | | The access key for the connection to the schema registry. |
| `secretKey` | string | | The secret key for the connection to the schema registry. |
| `validateCredentials` | bool | `true` | Whether Gateway rejects a blank `accessKey` or `secretKey` when it builds the credentials. Set it to `false` to accept them and fail later instead. |
| `accountId` | string | | The Id for the AWS account to use. |
If you don't supply a `basicCredentials` section for the AWS Glue schema registry, the client will attempt to find the connection information it needs from the environment ([see AWS docs for details](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default) ) and the credentials required can be passed this way to Gateway as part of its core configuration.
[Read our blog about schema registry](https://www.conduktor.io/blog/what-is-the-schema-registry-and-why-do-you-need-to-use-it/) .
## Related resources
* [Manage Kafka topics](/guide/manage-kafka/kafka-resources/topics)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Kafka data sharing with partners
Source: https://docs.conduktor.io/guide/conduktor-concepts/partner-zones
Share Kafka topics with external partners without data duplication using Conduktor Partner Zones.
If a partner needs access to your Kafka topics, the usual approach is to replicate the data to a separate cluster. That works, but it means two copies to maintain and keep in sync.
Partner Zones skip the copy. You share specific topics directly through Gateway, with controls over what the partner can see and do:
* set up **dedicated zones** with **customized access** to Kafka topics
* create a **single source of truth** because data isn't duplicated
* **reduce operational costs**, since you don't have to keep data streams synchronized
* track consumed and produced data with **traffic analytics**
**From our blog:** [The Kafka data sharing ladder](https://www.conduktor.io/blog/the-data-sharing-ladder) Where Partner Zones sit on the four-level path from internal streaming to external data exchange.
## How it works
A Partner Zone is a filtered view of your Kafka cluster, served through Conduktor Gateway. Partners connect with their own credentials and see only what you've shared — topics appear under whatever names you choose to expose.
```mermaid theme={null}
flowchart TD
K[Your Kafka cluster] --> GW[Conduktor Gateway]
GW --> A[Partner Zone A]
GW --> B[Partner Zone B]
A -. "orders-public, inventory-summary" .-> PA([Partner A connects here])
B -. "shipment-events" .-> PB([Partner B connects here])
```
Partners get a dedicated Kafka bootstrap endpoint and credentials. From their perspective, they connect to a standard Kafka cluster — no special client libraries needed.
## Key capabilities
| Capability | Description |
| --------------------- | ------------------------------------------------------------------------ |
| Topic aliasing | Expose internal topic names under partner-facing aliases |
| Scoped access | Each zone sees only its assigned topics — no cross-zone visibility |
| Bidirectional sharing | Partners can produce to topics you allow, not just consume |
| Traffic analytics | Monitor bytes in/out, record counts, and consumer lag per zone |
| Credential isolation | Partner credentials are independent — rotating one doesn't affect others |
## Related resources
* [Get started with Partner Zones](/guide/use-cases/third-party-data)
* [Configure Partner Zone traffic analytics](/guide/tutorials/configure-partner-zone-traffic-analytics)
* [View resource reference](/guide/reference/console-reference#partner-zones)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Kafka self-service — topic requests and governed access
Source: https://docs.conduktor.io/guide/conduktor-concepts/self-service
Enable dev teams to create Kafka topics and manage access through a governed self-service portal. Conduktor enforces policies, ACLs, and ownership.
Conduktor Self-service lets application teams request Kafka topics and access without going through the platform team for every change. Platform teams set the rules upfront; everything else flows from there.
## How it works
Self-service introduces two roles:
* **Platform teams** — assign application's owner a set of resources and define the governance rules they will be restricted by.
* **Application teams** — create resources within their ownership scope, approve access requests, and subscribe to other teams' topics. Approved requests are applied automatically without platform team intervention for each change.
Topic creation is a direct API call that passes if the request is policy-compliant — no approval workflow needed. Access requests between teams go through the owning application team, not the platform team:
```mermaid theme={null}
flowchart LR
AT[Application team] -- "create topic (policy-compliant)" --> GW[Gateway / Kafka]
AT2[Other app team] -- "request access" --> AT
AT -- "approve" --> GW
```
Every change goes into Git. You get a full history and can review anything before it lands on Kafka.
## Key concepts
| Concept | Description |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Application | A logical grouping for a team or service — owns its topics and ACLs |
| Application team | Group of people interacting with an application's resources, organized into ApplicationGroups with explicit resource and instance permissions (e.g., support team with read access, devOps team with API key creation rights) |
| Access request | A request for read or write access to a topic owned by another application |
| Policy | Rules that constrain what teams can request (e.g., max partitions, naming conventions) |
| Ownership | Each resource has a declared owner. Access management (requesting and granting) is delegated to ApplicationGroup members with the appropriate instance permissions |
## Benefits
* **Less bottleneck** — teams don't wait on platform for routine requests
* **Guardrails, not gatekeeping** — policies constrain what can be requested, not who can work
* **Full audit trail** — every change is in Git, reviewable and revertible
* **Explicit ownership** — each resource has a declared owner, so access requests go to the right person
* **Data discovery** — Kafka is made to open access to data. Our data catalog is here for that and exposes topics metadata to add context
**From our blog:** [No Kafka data platform without ownership](https://www.conduktor.io/blog/no-kafka-data-platform-without-ownership) Why ownership is the foundation self-service, autonomy, and scale are built on.
## Related resources
* [Use Self-service](/guide/use-cases/self-service)
* [Check out the quick start tutorial](/guide/tutorials/get-started-with-self-service)
* [Start from the Self-service reference template](https://github.com/conduktor/self-service-template) — a production-ready GitHub repository layout with CODEOWNERS, scope-based CI/CD workflows, and example ResourcePolicies
* [View resource reference](/guide/reference/self-service-reference)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Stream lineage
Source: https://docs.conduktor.io/guide/conduktor-concepts/stream-lineage
Visualize how data flows between Kafka topics and the service accounts that produce or consume them.
## Overview
Stream lineage shows the flow of data through your Kafka estate, mapping the producers, consumers, and topics involved in each stream. Use it to:
* Trace how a topic is being read and written across applications
* Identify which service accounts depend on a given topic
* Investigate the downstream impact of a topic or application change
* Watch live produce and consume throughput on Gateway clusters
Lineage is derived from the Kafka ACLs granted to each service account. If you use Self-service, those ACLs are generated from your `ApplicationInstance` and `ApplicationInstancePermission` resources (including approved Topic Access Requests), so the resulting lineage reflects the applications and permissions you've modeled in Console.
On Confluent Cloud or Confluent Platform clusters configured to use role bindings instead of ACLs, the edges of the lineage graph are populated from those role bindings. See the guides for migrating to role bindings on [Confluent Cloud](/guide/tutorials/migrate-confluent-cloud-rbac) and [Confluent Platform](/guide/tutorials/migrate-confluent-platform-rbac).
## Lineage view entry point
Open **Lineage** from the Console sidebar to land on the entry point for the selected cluster. From here, choose which dimension of your platform's data flow you want to explore:
* **Application instance** — view the flow built from Self-service data, with `ApplicationInstance` resources as the nodes
* **Service account** — view the flow built from the underlying Kafka authorization data (ACLs, or role bindings on Confluent Cloud and Confluent Platform), with service accounts as the nodes
The search bar at the top lets you jump straight to a known Application instance or service account without browsing.
## Application instance view
The Application instance view shows how your Self-service applications connect to one another through the topics they produce to and consume from. Each card on the left lists an instance with its read and write counts; selecting one focuses the graph on that instance and its neighbours.
The **Topics** toggle in the top right controls how connections are rendered:
* **On** — every topic that links two applications is shown as a node on the graph, so you can see exactly which topics carry data between services
* **Off** — topics are hidden and the graph collapses to a compressed view showing only how applications are transitively connected through any shared topics
Switch to **Data flow** in the top right to see a left-to-right view of the same relationships: the topics an instance reads from on the left, the instance in the middle, and the topics it writes to on the right, alongside the service accounts producing the data.
## Service account view
The Service account view shows the flow derived directly from Kafka authorization data, with each service account as a node and edges drawn from the ACLs (or role bindings) granting read or write access to a topic. This is the view to use when you want to reason about access at the authorization layer rather than the Self-service model — for example, to spot service accounts that have permissions outside of any registered application.
The **Topics** toggle and the **Graph** / **Data flow** switch behave the same as in the Application instance view.
The data flow view lays the same relationships out left to right, with the topics a service account reads from on the left and the topics it writes to on the right:
## Live lineage
On Gateway clusters, the **Live** toggle at the top of the Lineage view overlays live throughput on the graph. Each connection shows its current rate in bytes per second, updated every few seconds: connections carrying more traffic are drawn thicker and animate in the direction the data flows, while connections with no traffic stay thin and grey. A badge on the canvas shows the state of the live feed: **Live**, **Connecting** or **Reconnecting**.
Live throughput works in the Graph and Data flow views and in the drill-down views. With the **Topics** toggle off, the collapsed application-to-application connections keep their topic counts but don't show a rate, because traffic can't be attributed to a single collapsed link.
To use live lineage:
* The cluster has to be a Gateway cluster. On other cluster types the **Live** toggle doesn't appear.
* The [observability Interceptor](/guide/tutorials/configure-chargeback#collect-gateway-throughput) has to be deployed on Gateway. This is the same Interceptor that Chargeback uses for Gateway throughput, so if you've already set that up, live lineage needs no extra configuration.
* Rates are averaged over the Interceptor's flush interval (five minutes by default), so set `flushIntervalInSecond` to a low value, such as 10 seconds, for numbers that respond quickly.
The live feed updates the numbers on the graph you're viewing. New topics or connections appear the next time the graph loads.
## Large graphs and drill-down views
For very large graphs, the view is trimmed to stay responsive: Console keeps the 5,000 most connected nodes and draws the most significant connections between them, with a banner telling you when the graph has been trimmed. To see everything around a specific resource, use the drill-down views: select an individual Application instance or service account from the sidebar list and click **Expand** to open a sub-graph that centers on that resource and shows only its transitive relationships to other services and topics.
The drill-down view focuses on the selected node, with read and write counts, the ACLs that produce each edge, and a **HOPS** control in the top right to widen or narrow how many relationship steps out from the node are included in the graph.
## Performance impact
Console caches each cluster's lineage graph in memory to keep the views responsive, which carries a small JVM heap cost. From our internal benchmarks, a cluster with a graph of around 15,000 nodes adds roughly 150 MB to the Console JVM heap while that cluster sits in the cache.
To keep the overhead bounded:
* Graphs are cached with a 15 minute TTL, after which Console rebuilds them on the next request
* At most three clusters' graphs live in the cache at any one time; cycling to a fourth cluster evicts the least recently used graph
To disable Stream lineage entirely or change the refresh frequency, set the relevant `CDK_STREAMLINEAGE_*` options listed in the [Console environment variables reference](/guide/conduktor-in-production/deploy-artifacts/deploy-console/environment-variables#stream-lineage-properties).
## Related resources
* [Service accounts and ACLs](/guide/manage-kafka/kafka-resources/service-accounts-acls)
* [Self-service](/guide/use-cases/self-service)
* [Deploy the observability Interceptor](/guide/tutorials/configure-chargeback#collect-gateway-throughput)
# Kafka rate limiting and traffic control
Source: https://docs.conduktor.io/guide/conduktor-concepts/traffic-control-policies
Enforce Kafka rate limits, producer quotas, and topic rules with Conduktor Gateway. Protect cluster infrastructure from misbehaving clients without application changes.
Traffic Control Policies let you enforce limits and rules on Kafka operations to protect your infrastructure and ensure compliance.
**From our blog:** [No Kafka data platform without ownership](https://www.conduktor.io/blog/no-kafka-data-platform-without-ownership) Why federated ownership underpins governance and policy enforcement at scale.
## What you can control
Traffic Control Policies work as GatewayInterceptors that validate and enforce rules on:
**Configuration policies**
* Broker and topic configuration changes
* Topic creation parameters (replication factor, partition count)
* Schema ID requirements
**Rate limiting policies**
* Producer throughput limits
* Consumer fetch rate limits
* Offset commit frequency
* Consumer group join frequency
* Connection attempt limits
**Access control policies**
* Read-only topics
* Required client IDs
* Consumer group restrictions
* Message header removal
## Benefits
* **Prevent misconfiguration**: stop teams from creating topics with dangerous settings
* **Protect cluster performance**: rate limit clients before they impact others
* **Enforce standards**: require schema IDs, client IDs, or other organizational policies
* **Gradual rollout**: apply policies per service account or Virtual Cluster
## Available policies
* [Alter broker config](/guide/use-cases/apply-traffic-control-policies#alter-broker-config)
* [Alter topic config](/guide/use-cases/apply-traffic-control-policies#alter-topic-config)
* [ClientId required](/guide/use-cases/apply-traffic-control-policies#clientid-required)
* [Consumer group policy](/guide/use-cases/apply-traffic-control-policies#consumer-group-policy)
* [Create topic policy](/guide/use-cases/apply-traffic-control-policies#create-topic-policy)
* [Fetch policy](/guide/use-cases/apply-traffic-control-policies#fetch-policy)
* [Limit commit offset](/guide/use-cases/apply-traffic-control-policies#limit-commit-offset-policy)
* [Limit connection attempts](/guide/use-cases/apply-traffic-control-policies#limit-connection-attempts-policy)
* [Limit join group](/guide/use-cases/apply-traffic-control-policies#limit-join-group-policy)
* [Limit producer rate](/guide/use-cases/apply-traffic-control-policies#producer-rate-limiting-policy)
* [Produce requests](/guide/use-cases/apply-traffic-control-policies#produce-policy)
* [Read-only topic](/guide/use-cases/apply-traffic-control-policies#read-only-topic-policy)
* [Remove message headers](/guide/use-cases/apply-traffic-control-policies#message-header-removal-policy)
* [Schema ID required](/guide/use-cases/apply-traffic-control-policies#topic-required-schema-id-policy)
## Related resources
* [Apply Traffic Control Policies](/guide/use-cases/apply-traffic-control-policies)
* [Interceptors concept](/guide/conduktor-concepts/interceptors)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Kafka multi-tenancy with Virtual Clusters — Gateway
Source: https://docs.conduktor.io/guide/conduktor-concepts/virtual-clusters
Create isolated Kafka Virtual Clusters for teams, environments, or partners with Conduktor Gateway.
A Virtual Cluster is a Conduktor Gateway concept that logically represents a Kafka cluster with strict resource segregation. Each Virtual Cluster has its own authentication and authorization layer, dedicated Topics and Consumer Groups that only mapped applications can access, and specific **Interceptors**.
By using Virtual Clusters, you can create multiple environments, or segregated secured namespaces for your partners or projects, while maintaining a single physical Kafka cluster.
The Virtual Clusters are only available on your Conduktor Gateway if it is configured with `GATEWAY_SECURITY_MODE: GATEWAY_MANAGED`. The Gateway needs to manage both the authentication and authorization of clients to work.
**From our blog:** [How to isolate non-prod Kafka with Virtual Clusters](https://conduktor.io/blog/how-to-isolate-non-prod-kafka-with-virtual-clusters) Replace VPC-per-environment sprawl with one physical cluster and virtual isolation that satisfies security teams.
## When to use Virtual Clusters?
### External data sharing
To best support external data sharing use cases, you will most likely need to protect from noisy neighbors, manage ACLs, and monetize the shared data. In such case, we highly recommend using [**Partner Zones**](/guide/use-cases/third-party-data). Partner Zones provide:
* a dedicated management UI
* resource metadata and labels for business tracking
* simplified UX (one API call manages ACLs, Topic Aliases, Service Accounts, and metadata)
* Chargeback capabilities based on throughput
A Virtual Cluster can be used as a **segregated secured namespace for your partners** or **external applications** to connect to. It will allow you to:
* **Isolate the Topics you want to expose** to specific applications, with its related set of **ACLs** to restrict their permissions
* **Prefix the Consumer Groups** they might use, helping with troubleshooting and auditing
* Use it as an **Interceptor** **scope** to easily manage Traffic Control Policy or Encryption at this level
* **Expose internal Topics** to external partners **with business-friendly names** while keeping your internal naming conventions private
* Potentially **expose Topics from multiple physical Kafka clusters**, while hiding this complexity from the client
In this context, we recommend setting the [`spec.type`](/guide/reference/gateway-reference#param-spec-type) to `Partner`.
### Internal environments segregation
A Virtual Cluster can be used to simulate multiple logical environments on the same physical cluster. That way, you can:
* **Merge your lower environments** (like dev & QA) into the same Kafka cluster, and separate the Topics from each environment in a dedicated Virtual Cluster
* **Strongly isolate the business units** that never interacts with each other and that should be completely separated from each other, like multiple subsidiaries, or different lines of business.
* **Ease the environment upgrade** as the Topics can have the same name in the different Virtual Cluster, as they will exist with different names on the physical Kafka Cluster
* Create as many **developers sandboxes** as you want, with a quick API call
* Use [Topic Concentration](/guide/conduktor-concepts/logical-topics#concentrated-topics) to allow your developers to **create as many topics as they want**, for testing purpose, while storing the data **on the same physical Topic**.
In this context, we recommend setting the [`spec.type`](/guide/reference/gateway-reference#param-spec-type) to `Standard`.
Virtual Clusters are **not** designed for **multi-tenancy** (one Virtual Cluster per internal team). For that, use our Console [**Self-service**](/guide/use-cases/self-service) feature instead.
## Get started with Virtual Clusters
The creation of a Virtual Cluster is done via our [API](/guide/conduktor-in-production/automate/api-automation#gateway-api) / [CLI](/guide/conduktor-in-production/automate/cli-automation) / [Terraform Provider](/guide/conduktor-in-production/automate/terraform-automation). It requires the Gateway to be up and running, and the Gateway API to be accessible.
You can refer to the [Virtual Cluster reference](/guide/reference/gateway-reference#virtualcluster) to customize its name, type, super users, and ACLs.
Once the Virtual Cluster is created, **all the Consumer Groups and Topics prefixed by its name** will be automatically visible in it.
To add Topics that **aren't prefixed by the Virtual Cluster name**, you need to define [Topics Aliases](/guide/reference/gateway-reference#aliastopic) and include them in the Virtual Cluster using the [`metadata.vCluster`](/guide/reference/gateway-reference#param-metadata-v-cluster) field.
An application can have its Service Account attached to only one Virtual Cluster. For that, you need to set its `metadata.vCluster` to the name of your Virtual Cluster.
Once your application has been attached to a Virtual Cluster, it will be able to interact with the Topics attached to this Virtual Cluster.
## Troubleshooting
* Make sure the Topics have been mapped in this Virtual Cluster, either using Topic Aliases or their prefix.
* Try to connect and list Topics using the Virtual Cluster super user defined, to confirm the Topic is in there.
* Confirm the Virtual Cluster has the right Kafka ACLs attached to it
## Related resources
* [Virtual Cluster reference](/guide/reference/gateway-reference#virtualcluster)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Conduktor deployment architecture
Source: https://docs.conduktor.io/guide/conduktor-in-production
Deploy Conduktor Console and Gateway on Kubernetes (Helm) or Docker. Production architecture overview with deployment options for each platform component.
Here's an architecture of Conduktor platform in a production ready environment:
[Check out the recommendations and requirements for a production environment](https://github.com/conduktor/conduktor-reference-architecture)
## Kubernetes (recommended)
The primary deployment target for the Conduktor platform is Kubernetes using Helm.
* [Console](/guide/conduktor-in-production/deploy-artifacts/deploy-console/kubernetes)
* [Gateway](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/kubernetes)
## Docker
* [Console](/guide/conduktor-in-production/deploy-artifacts/deploy-console)
* [Gateway](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway)
## AWS
* [Console](/guide/tutorials/deploy-aws)
# Admin configuration — Conduktor Console setup
Source: https://docs.conduktor.io/guide/conduktor-in-production/admin
Admin setup for Conduktor Console: connect Kafka clusters, configure RBAC, set up SSO, manage users and groups, enable audit logging, and configure data masking.
We recommend completing these tasks first.
## Core configuration
[Connect your Kafka infrastructure](/guide/conduktor-in-production/admin/configure-clusters) to start managing resources and topics.
[Enable health monitoring](/guide/conduktor-in-production/monitor) for your Gateway and Console deployments to track system health.
## Access and security
[Configure authentication](/guide/conduktor-in-production/admin/user-access) including [user groups](/guide/conduktor-in-production/admin/user-access/map-external-groups) for team access management.
[Implement role-based access control](/guide/conduktor-in-production/admin/set-up-rbac) to manage user permissions and security.
[Enable audit logging](/guide/conduktor-in-production/admin/audit-logs) for compliance and security event tracking.
# Kafka audit logging — track all actions
Source: https://docs.conduktor.io/guide/conduktor-in-production/admin/audit-logs
Configure Kafka audit logs in Conduktor Console to track every user action and resource event.
As you and your team interact with Conduktor, audit events are captured that give a detailed log of actions, providing a centralized visibility of user-related and resource-related events.
## Console overview
The Console audit log events can be **browsed, filtered and searched** directly via the UI or **exported from a Kafka topic** for further use (such as maintaining your own audit trail in other systems).
Click on an event in the audit log to expose event-specific metadata. Here's an example of an audit event for a new connector which can include metadata such as custom tags, the cluster, connector name and its ID.
Once configured with the correct [environment variables](/guide/conduktor-in-production/deploy-artifacts/deploy-console/environment-variables#audit-log-export-properties), audit log events are also exported to a Kafka topic, allowing you to leverage the benefits of Conduktor when finding a message.
## View and export audit logs
#### Export audit log events
You can export audit log events from a Kafka topic using the Console UI. The exportable events have more detail than the legacy events, providing additional information about the event that has taken place.
Learn how to configure audit events for export via [configuration properties](/guide/conduktor-in-production/deploy-artifacts/deploy-console/environment-variables#audit-log-export-properties).
The events conform to the [CloudEvents specification](https://github.com/cloudevents/spec/blob/main/cloudevents/spec.md) , a vendor-neutral format that follows the following structure:
```json theme={null}
{
"specversion" : "1.0",
"type" : "com.github.pull_request.opened",
"source" : "https://github.com/cloudevents/spec/pull",
"subject" : "123",
"id" : "A234-1234-1234",
"time" : "2018-04-05T17:31:00Z",
"comexampleextension1" : "value",
"comexampleothervalue" : 5,
"datacontenttype" : "text/xml",
"data" : ""
}
```
Here's an example of a Conduktor event:
```json theme={null}
{
"source": "//kafka/kafkacluster/production/topic/website-orders",
"data": {
"eventType": "Kafka.Topic.Create",
// Additional event specific data...
"metadata": {
"name": "website-orders",
"cluster": "production"
}
// Additional event specific metadata...
},
"datacontenttype": "application/json",
"id": "ad85122c-0041-421e-b04b-6bc2ec901e08",
"time": "2024-10-10T07:52:07.483140Z",
"type": "AuditLogEventType(Kafka,Topic,Create)",
"specversion": "1.0"
}
```
Here's a list of currently available audit log event types.
##### Kafka related events
| **Event type** | **Description** |
| --------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| **Kafka.Subject.Create** | Kafka subject is created. |
| **Kafka.Subject.Update** | Kafka subject is updated. |
| **Kafka.Subject.Delete** | Kafka subject is deleted. |
| **Kafka.Subject.ChangeCompatibility** | Kafka subject compatibility is changed |
| **Kafka.Topic.Create** | Kafka topic is created. |
| **Kafka.Topic.Update** | Kafka topic is updated. |
| **Kafka.Topic.Delete** | Kafka topic is deleted; `priorState` contains its previous metadata and configuration when available. |
| **Kafka.Topic.Empty** | Records are deleted from a topic by emptying it, emptying a partition or advancing low watermarks. |
| **Kafka.Topic.Browse** | Kafka topic is browsed. |
| **Kafka.Topic.ProduceRecord** | Kafka topic record is produced. |
| **Kafka.Topic.SqlQuery** | Kafka topic is requested through Console SQL. |
| **Kafka.Connector.Create** | Kafka connector is created. |
| **Kafka.Connector.Update** | Kafka connector is updated. |
| **Kafka.Connector.Delete** | Kafka connector is deleted. |
| **Kafka.Connector.Restart** | Kafka connector is restarted. |
| **Kafka.Connector.TaskRestart** | Kafka connector task is restarted. |
| **Kafka.Connector.Pause** | Kafka connector is paused. |
| **Kafka.Connector.Resume** | Kafka connector is resumed. |
| **Kafka.Connector.AutoRestartActivate** | Kafka connector auto-restart is activated. |
| **Kafka.Connector.AutoRestartStop** | Kafka connector auto-restart is stopped. |
| **Kafka.ConsumerGroup.Duplicate** | Kafka consumer group is duplicated. |
| **Kafka.ConsumerGroup.Update** | Kafka consumer group is updated, when the offset is reset. |
| **Kafka.ConsumerGroup.Delete** | Kafka consumer group is deleted. |
##### IAM related events
| **Event type** | **Description** |
| -------------------- | --------------------- |
| **Iam.User.Create** | IAM user is created. |
| **Iam.User.Update** | IAM user is updated. |
| **Iam.User.Delete** | IAM user is deleted. |
| **Iam.User.Login** | IAM user logs in. |
| **Iam.User.Logout** | IAM user logs out. |
| **Iam.Group.Create** | IAM group is created. |
| **Iam.Group.Update** | IAM group is updated. |
| **Iam.Group.Delete** | IAM group is deleted. |
##### Self-service related events
| **Event type** | **Description** |
| ---------------------------------------------------- | ----------------------------------------------------- |
| **SelfService.Application.Create** | Self-service application is created. |
| **SelfService.Application.Update** | Self-service application is updated. |
| **SelfService.Application.Upsert** | Self-service application is created or updated. |
| **SelfService.Application.Delete** | Self-service application is deleted. |
| **SelfService.ApplicationInstance.Create** | Self-service application instance is created. |
| **SelfService.ApplicationInstance.Update** | Self-service application instance is updated. |
| **SelfService.ApplicationInstance.Delete** | Self-service application instance is deleted. |
| **SelfService.ApplicationInstanceApiKey.Create** | Self-service application instance API key is created. |
| **SelfService.ApplicationInstanceApiKey.Delete** | Self-service application instance API key is deleted. |
| **SelfService.ApplicationGroup.Create** | Self-service application group is created. |
| **SelfService.ApplicationGroup.Update** | Self-service application group is updated. |
| **SelfService.ApplicationGroup.Delete** | Self-service application group is deleted. |
| **SelfService.ApplicationPolicy.Create** | Self-service application policy is created. |
| **SelfService.ApplicationPolicy.Update** | Self-service application policy is updated. |
| **SelfService.ApplicationPolicy.Delete** | Self-service application policy is deleted. |
| **SelfService.ApplicationInstancePermission.Create** | Permissions are created for an app instance. |
| **SelfService.ApplicationInstancePermission.Delete** | Permissions are deleted for an app instance. |
| **SelfService.ServiceAccount.Create** | Service account is created. |
| **SelfService.ServiceAccount.Update** | Service account is updated. |
| **SelfService.ServiceAccount.Delete** | Service account is deleted. |
##### Admin related events
| **Event type** | **Description** |
| -------------------------------------------- | ------------------------------------------------------------ |
| **Admin.KafkaConnect.Create** | A Kafka Connect instance is created. |
| **Admin.KafkaConnect.Update** | A Kafka Connect instance is updated |
| **Admin.KafkaConnect.Delete** | A Kafka Connect instance is deleted. |
| **Admin.KsqlDB.Create** | A ksqlDB instance is created. |
| **Admin.KsqlDB.Update** | A ksqlDB instance is updated. |
| **Admin.KsqlDB.Delete** | A ksqlDB instance is deleted. |
| **Admin.KafkaCluster.Create** | A Kafka cluster is created. |
| **Admin.KafkaCluster.Update** | A Kafka cluster is updated. |
| **Admin.KafkaCluster.Delete** | A Kafka cluster is deleted. |
| **Admin.SchemaRegistry.ChangeCompatibility** | The global compatibility of the schema registry is updated. |
| **Admin.Integration.Update** | The alert integration (Slack, MS Teams, Webhook) is updated. |
| **Admin.AdminApiKey.Create** | A new admin API key is created. |
| **Admin.AdminApiKey.Delete** | An admin API key is deleted. |
| **Admin.DataMaskingPolicy.Create** | A data masking policy is created. |
| **Admin.DataMaskingPolicy.Update** | A data masking policy is updated. |
| **Admin.DataMaskingPolicy.Delete** | A data masking policy is deleted. |
| **Admin.Certificate.Create** | A certificate is created. |
| **Admin.Certificate.Delete** | A certificate is deleted. |
| **Admin.ConfluentOrganization.Create** | A Confluent Cloud organization is registered. |
| **Admin.ConfluentOrganization.Update** | A Confluent Cloud organization is updated. |
| **Admin.ConfluentOrganization.Delete** | A Confluent Cloud organization is deleted. |
| **Admin.ApplicationGroupTemplate.Create** | An application group template is created. |
| **Admin.ApplicationGroupTemplate.Update** | An application group template is updated. |
| **Admin.ApplicationGroupTemplate.Delete** | An application group template is deleted. |
Confluent organization audit payloads exclude the API secret and redact the API key.
##### Alert related events
| **Event type** | **Description** |
| ----------------------- | ---------------------- |
| **Kafka.Alert.Create** | An alert is created. |
| **Kafka.Alert.Update** | An alert is updated. |
| **Kafka.Alert.Delete** | An alert is deleted. |
| **Kafka.Alert.Trigger** | An alert is triggered. |
##### Partner Zone related events
| **Event type** | **Description** |
| --------------------------------- | ------------------------------------------------ |
| **Admin.PartnerZone.Create** | A Partner Zone is created. |
| **Admin.PartnerZone.Update** | A Partner Zone is updated. |
| **Admin.PartnerZone.Delete** | A Partner Zone is deleted. |
| **Admin.PartnerZone.TokenCreate** | A token is created for accessing a Partner Zone. |
#### Legacy audit events
Here's a list of legacy audit events tracked by Conduktor:
* [Console](#console)
* [Data masking](#data-masking)
* [Self-service](#self-service)
* [Admin](#admin)
### Console
##### ResourceType: `Topic`
CRN: `kafka:/cluster//topic/`
* topic.ProducedRecord
* topic.Browsed
* topic.Deleted
* topic.Created
* topic.Updated
* topic.Emptied
* topic.Tagged
* topic.Untagged
##### ResourceType: `Subject`
CRN `kafka:/cluster//subject/`
* subject.Created
* subject.Updated
* subject.ChangedCompat
* subject.Deleted
* subject.DeletedVersions
* subject.DeletedVersion
##### ResourceType: `SchemaRegistry`
CRN `kafka:/cluster/`
* registry.ChangedGlobalCompat
##### ResourceType: `ConsumerGroup`
CRN `kafka:/cluster//group/`
* consumergroup.Created
* consumergroup.Updated (ResetOffsets)
* consumergroup.Deleted
##### ResourceType: `Connector`
CRN `kafka:/cluster//connect//`
* connector.Created
* connector.Updated
* connector.Deleted
* connector.Restarted
* connector.RestartedTask
* connector.Paused
* connector.Resumed
#### Data masking
##### ResourceType: `DatamaskingPolicy`
CRN `platform:/datamasking/`
* policy.Upserted
* policy.Deleted
#### Self-service
##### ResourceType: `Application`
CRN `platform:/application/`
* application.Created
* application.Deleted
* application.Updated
* application.access-request.Approved
from / to
* application.access-request.Rejected
#### Admin
##### ResourceType: `Cluster`
CRN `kafka:/cluster/`
* cluster.Created
* cluster.Updated
* cluster.Deleted
##### ResourceType: `Group`
CRN `platform:/group/`
* group.Created
* group.member.Added
* group.member.Deleted
* group.permission.Added
* group.permission.Deleted
##### ResourceType: `User`
CRN `platform:/user/`
* user.Login
* user.permission.Added
* user.permission.Deleted
* user.platform\_role.Updated
## Gateway overview
The Gateway audit log events are stored in an internal topic. The topic name is configured using `GATEWAY_AUDIT_LOG_TOPIC` and the feature can be enabled/disabled using `GATEWAY_FEATURE_FLAGS_AUDIT`. The audit events can be **browsed, filtered and searched** via Console UI or used to create custom alerts and notifications.
Here's an example of a Gateway audit log event in JSON format, showing a successful authentication attempt to Gateway:
```json theme={null}
{
"id": "a3b7355b-dbe6-4db4-90a8-f41e0111710e",
"source": null,
"type": "AUTHENTICATION",
"authenticationPrincipal": "",
"userName": "",
"connection": {
"localAddress": "/:",
"remoteAddress": "/:"
},
"specVersion": "0.1.0",
"time": "2025-07-18T16:07:06.570723674Z",
"eventData": "SUCCESS"
}
```
### Field details
**Key fields and their values:**
* `authenticationPrincipal`: The tenant name, or `null` on some authentication failures.
* `userName`: The attempted user name, or `null` on some authentication failures.
* `source`: Event source, always `null` for authentication events.
* `eventData`: Contains success/failure messages and error details.
* `connection`: The connection metadata for the client that triggered the event. See [connection block](#connection-block).
### Connection block
Every Gateway audit event type except authorization failure includes a `connection` block:
| **Field** | **Description** |
| :---------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `localAddress` | The Gateway address the client connected to. |
| `remoteAddress` | The socket peer address — the load balancer's address when Gateway sits behind one or the original client address if Kubernetes `externalTrafficPolicy: Local` is [configured](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/load-balancing#using-kubernetes-externaltrafficpolicy). |
| `proxyConnection` | (Optional) The original client address that comes with HAProxy Protocol header. Present only when Gateway captures the client IP address using [HAProxy Protocol](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/load-balancing#using-haproxy-protocol). |
For example:
```json theme={null}
"connection": {
"localAddress": "/:",
"remoteAddress": "/:",
"proxyConnection": {
"sourceAddress": ":",
"destinationAddress": ":"
}
}
```
## Gateway audit event types
Gateway supports the following audit event types:
### Connection event
The connection event tracks client connections to Gateway. It captures client information like remote address, which helps with monitoring connections. See [connection block](#connection-block) for the fields it reports, and [Capturing the client IP address](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/load-balancing#capturing-the-client-ip-address) for the original client address.
### Authentication event
The authentication event records the authentication attempts made by clients to Gateway. It records both successful and failed attempts, providing insights into who accessed Gateway. Its [connection block](#connection-block) includes the original client address — see [Capturing the client IP address](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/load-balancing#capturing-the-client-ip-address).
The `eventData` field contains different messages, depending on the Gateway security mode, Gateway security protocol and the authentication outcome. Here are some common examples:
| **Security mode** | **SASL mechanism** | **Outcome** | **Event data** |
| :---------------- | :----------------- | :---------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Gateway managed | PLAIN | Success | `"SUCCESS"` |
| Gateway managed | PLAIN | Failure | `"Authentication failed: credentials for user could not be verified. Cause: Invalid username or password"` |
| Gateway managed | PLAIN | Failure | `"Authentication failed: credentials for user could not be verified. Cause: Password token is valid but username is invalid"` |
| Gateway managed | PLAIN | Failure | `"Authentication failed: credentials for user could not be verified. Cause: Token has expired. Claims: {username=, vcluster=, exp=}"` |
| Gateway managed | PLAIN | Failure | `"Authentication failed: username not specified"` or `"Authentication failed: password not specified"` |
| Gateway managed | PLAIN | Failure | `"Authentication failed: credentials for user could not be verified. Cause: Password token is valid for username, but username is not an existing local user."` |
| Gateway managed | OAUTHBEARER | Failure | `"{\"status\":\"invalid_token\"}"` |
| Kafka managed | Any | Failure | `"SASL Authentication failed."` |
| Kafka managed | Any | Failure | `"Gateway can't detect principal from your authorization exchange. Please contact your administrator"` |
For the Gateway managed with OAUTHBEARER, authentication may also fail if the Kafka client is unable to fetch the token from the configured OAUTH provider. This occurs before reaching Gateway, so audit log event would not be created.
### Safeguard event
The safeguard event captures policy enforcement actions by safeguard Interceptors such as data quality producer policy and Traffic Control Policies. Its [connection block](#connection-block) includes the original client address — see [Capturing the client IP address](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/load-balancing#capturing-the-client-ip-address).
### API keys request event
The [Kafka protocol](https://kafka.apache.org/0101/protocol.html#protocol_api_keys) describes `API keys` that represent different kinds of requests that can be made to the broker.
On Gateway, you can configure an [audit Interceptor](/guide/reference/data-security#audit-interceptor) to track specific Kafka operations (such as `PRODUCE`, `FETCH`, `CREATE_TOPICS`, `DELETE_TOPICS` and `ALTER_CONFIGS`) to elevate audit logging on certain topics as needed. Its [connection block](#connection-block) includes the original client address — see [Capturing the client IP address](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/load-balancing#capturing-the-client-ip-address).
### REST API event
The REST API event logs calls made to the Gateway's REST endpoints.
### Interceptor event
The Interceptor event records encryption/decryption error messages from cryptography Interceptors. Its [connection block](#connection-block) includes the original client address — see [Capturing the client IP address](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/load-balancing#capturing-the-client-ip-address).
### ACL admin event
The ACL admin event tracks Access Control List operations done via Kafka Admin API, such as `CREATE_ACLS` and `DELETE_ACLS` requests. Its [connection block](#connection-block) includes the original client address — see [Capturing the client IP address](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/load-balancing#capturing-the-client-ip-address).
ACL operations done via REST API (for Virtual Cluster) are recorded as REST API events.
### Authorization failure event
The authorization failure event records failed attempts to access a resource when ACL permissions deny the operation. It doesn't report a [connection block](#connection-block).
ACL permissions are determined by [Kafka authorization primitives](https://kafka.apache.org/documentation/#security_authz_primitives) (i.e. Operation and Resource), resource name, user name, tenant (i.e. Virtual Cluster) name and host.
The value of `eventData` in the authorization failure audit log event depends on the value of `SECURITY_MODE` configuration.
| **Security mode** | **Information in event data** | **Example event data value** |
| :---------------- | :----------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Gateway managed | Kafka authorization primitives | `"DENY Operation.WRITE on Resource(name=test-topic, resourceType=TOPIC, patternType=LITERAL) for [Tenant: passthrough, User: test-user, Host: 127.0.0.1]"` |
| Kafka managed | Authorization failure type | `"Kafka error 'Topic authorization failed.' when processing 'PRODUCE' request."` |
## Related resources
* [Configure audit log topics](/guide/tutorials/configure-audit-log-topic)
* [Customize logs](/guide/tutorials/custom-logging)
* [Gateway audit Interceptor](/guide/reference/data-security#audit-interceptor)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Kafka cluster configuration
Source: https://docs.conduktor.io/guide/conduktor-in-production/admin/configure-clusters
Add, update, and manage Kafka cluster configurations in Conduktor Console. Connect to Kafka brokers, Schema Registry, and Kafka Connect from the Settings. Enjoy a better experience with Confluent, Aiven and Gateway special integrations.
## Overview
To add, update, and delete Kafka cluster configurations, go to **Settings** > **Clusters**. By default, only users belonging to the **Admin** group or having the `Can manage Cluster configurations` permission will be able to view and manage the clusters.
**To add a cluster:** Click **Add cluster** in the top right corner.
**To edit a cluster:** Select it from the list to modify the name, icon, bootstrap servers, and additional properties. You can also enable Schema Registry, Kafka Connect, ksqlDB, and Kafka providers.
## Network requirements
Console connects to Kafka clusters using the Apache Kafka Java client library. It uses standard Kafka client connections with the bootstrap servers you provide in the cluster configuration.
Console requires the same network access to Kafka brokers as any standard Kafka client application.
## Required Kafka permissions
Console connects to Kafka as a standard Kafka client using the service account credentials you provide in the cluster configuration. To function correctly, that service account has to have the following Kafka ACLs:
| Resource | Operations | Used for |
| ---------------------- | ------------------ | -------------------------------------------------------- |
| `CLUSTER` | `DESCRIBE` | Cluster metadata, broker list, log dirs, ACL enumeration |
| `CLUSTER` | `DESCRIBE_CONFIGS` | Broker configuration |
| `TOPIC` (wildcard `*`) | `DESCRIBE` | List and describe topics |
| `TOPIC` (wildcard `*`) | `DESCRIBE_CONFIGS` | Topic configuration |
| `TOPIC` (wildcard `*`) | `READ` | Message browser |
| `GROUP` (wildcard `*`) | `DESCRIBE` | List and describe consumer groups |
| `GROUP` (wildcard `*`) | `READ` | Consumer group offsets |
Console has no dedicated read-only mode. To limit Console's Kafka service account to read-only access, grant only the ACLs above and omit write operations (such as `CREATE`, `DELETE`, `ALTER`, `ALTER_CONFIGS`, and `WRITE`). Kafka will deny any write operation the service account is not authorized for.
Kafka ACLs only limit the Kafka operation. They do not control what Console users can do in the UI.
To control what Console users can do in the UI, use [Console RBAC](/guide/conduktor-in-production/admin/set-up-rbac) in addition to Kafka ACLs.
## Connect to a Conduktor Gateway cluster
Connect Conduktor Console to a Conduktor Gateway cluster the same way you would connect to any Kafka cluster by providing the Gateway bootstrap servers and security configurations. Console will simply act as a Kafka client connecting to Gateway.
The security protocol must match the listener Console connects to (`GATEWAY_LISTENER__SECURITY_PROTOCOL`) in your Gateway configuration.
To enable Gateway integration and access Interceptors, Chargeback with Gateway or Partner Zones, select the **Gateway provider** and enter your Gateway API key and secret.
## Connect to a secure Kafka cluster
Console uses the Apache Kafka Java Client [configuration properties](https://kafka.apache.org/documentation/#consumerconfigs) .
To connect to a secure Kafka cluster, specify the values from your `config.properties` file:
```sh theme={null}
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username='' password='';
```
## Connect to a Confluent Cloud cluster
1. In your Confluent cluster dashboard, go to **Data integration** > **Clients**
2. Select **Java** as the language
3. Create a **Kafka cluster API key**. Optionally, create a **Schema Registry API Key** if using Schema Registry
4. Copy the configuration to your clipboard.
5. In Console, add the bootstrap servers and paste the configuration as *advanced properties*.
6. Test the connection. A green **Connected** label appears on success.
7. Click **Create Configuration** to save
To manage Confluent service accounts, set up the **Confluent provider** in the [cluster configuration](/guide/manage-kafka/kafka-resources/service-accounts-acls#confluent-cloud).
## Connect to an Aiven cluster
Choose one of the following authentication methods:
### SSL authentication
Provide the **Access Key**, **Access Certificate**, and **CA Certificate** from your Aiven console. The **Service URI** is your bootstrap server.
In Console:
1. Enter the bootstrap server
2. Upload the CA certificate
3. Select **SSL** as the authentication method
4. Paste the **Access Key** and **Access Certificate**
To manage Aiven service accounts, set up the **Aiven provider** in the [cluster configuration](/guide/manage-kafka/kafka-resources/service-accounts-acls#aiven).
### SASL\_SSL authentication
Provide the **Service URI** (bootstrap server), **User**, and **Password** from your Aiven console.
The configuration in Console:
To manage Aiven service accounts, set up the **Aiven provider** in the [cluster configuration](/guide/manage-kafka/kafka-resources/service-accounts-acls#aiven).
## Connect to an AWS MSK cluster
1. Create an IAM user:
2. Attach permissions to the IAM user:
```json theme={null}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["kafka:*", "kafka-cluster:*"],
"Resource": "*"
}
]
}
```
3. Generate an access key:
4. In Console, enter the **Bootstrap server** and select **AWS IAM** as the authentication method
5. Choose one option:
* Inherit credentials from environment variables, or
* Enter the **Access Key** and **Secret Key**
## Connect to a Cloudera cluster
These instructions use **SASL\_SSL** protocol with **PLAIN** mechanism.
1. Create a workload user with **ownership of the Data Hub cluster**. Note the username and password:
2. Download certificates from Cloudera:
3. Convert CRT certificates to JKS format using Java keytool:
```bash theme={null}
keytool -import -keystore zeke-test2-cdp-env.jks -alias zeke-test2-cdp-env -file zeke-test2-cdp-env.crt
```
4. In Cloudera, open firewalls for Kafka brokers and Schema Registry
5. In Console, go to **Clusters**, select the Cloudera cluster, and upload the certificates
6. Configure the cluster using the workload username and password from step 1:
[Automated setup example](https://github.com/conduktor/conduktor-cloudera-quickstart-demo?tab=readme-ov-file#cloudera--conduktor)
## Connect to a Google Cloud cluster
Connect to Google Cloud Managed Service for Apache Kafka using **SASL\_SSL** protocol with **PLAIN** mechanism.
### Use a service account
[View Google Cloud documentation](https://cloud.google.com/managed-service-for-apache-kafka/docs/authentication-kafka#sasl-plain)
### Use an access token
1. Get an access token:
```bash theme={null}
gcloud auth login --no-launch-browser
gcloud auth print-access-token
```
2. Use the token with these parameters:
```bash theme={null}
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
username="PRINCIPAL_EMAIL_ADDRESS" \
password="ACCESS_TOKEN_VALUE";
```
Managed Service for Apache Kafka verifies that:
* The access token is valid and not expired
* The username matches the principal email associated with the access token
* The principal has the `managedkafka.clusters.connect` permission (included in `roles/managedkafka.client`)
## Related resources
* [Interactive cluster configuration guide](https://conduktor.navattic.com/cluster-configuration)
* [Feedback and feature requests](https://conduktor.io/roadmap)
# Kafka data masking and PII protection
Source: https://docs.conduktor.io/guide/conduktor-in-production/admin/data-masking
Define Kafka data masking policies in Conduktor Console to protect PII and sensitive fields. Control who sees unmasked values based on RBAC permissions.
## Overview
In order to meet **compliance regulations**, Conduktor Console provides a Data Masking feature that enables you to **obfuscate personal and sensitive data** within the Console.
As a Console administrator, you can **secure and govern** such data by creating Data Masking policies, so that users can't see them.
Data masking **does not impact how the underlying data is stored**. The data will only be masked **within Console** at runtime for specified users/groups only, the underlying Kafka data remains unchanged. To mask or encrypt the underlying Kafka data, use [Conduktor Gateway](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway).
The Flink SQL workbench (in preview since Console 1.47.0) reads topic data directly from Confluent Cloud, so data masking policies don't apply to Flink statement results: masked fields can appear unmasked. Before you grant users Flink access, check that they're allowed to see the unmasked data in the topics you give them access to.
Policies will be applied when **consuming Kafka messages in the Console**, as shown below. We can see that the phone number, the IBAN, and the card number, have been masked with some \*\*\*\*\*.
Here is the list of policies applied in this case.
## Create a data masking policy
In order to create a Data Masking policy and protect your data, go to **Settings** > **Data Policies**.
Click **New Policy** and fill in the required details:
| Policy detail | Description |
| ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Policy Name | Unique name for identifying your policy |
| Compliance | The compliance regulation the policy adheres to (e.g. GDPR, PCI-DSS) |
| Information Kind | The kind of information for obfuscation (e.g. PII, Financial) |
| Masking Rule | How the obfuscation should be implemented (e.g. hide-all, hide-last-3) |
| Risk Level | Categorization for the risk level associated with the policy |
| Mask all fields | When enabled, every field Console can find in the message is masked with the chosen masking rule. See [Mask all fields](#mask-all-fields). |
| Fields | List of fields that should be obfuscated, specified as dot-separated JSON paths. See [Field path syntax](#field-path-syntax) for details. If you want to hide multiple fields, you can click on **Add field**. Not available when **Mask all fields** is enabled. |
| Resources | List of resources where the policy must be applied, like clusters or topics. To add new resources, you can click on **Add resource**. |
| Exclude Users or Groups from policy | In case you want some users or groups to see the data, you can exclude them from the policy. |
In the case above, the policy will mask the field `credit_card`, for all the users **except people from the group "Order Owners"**, on the topic prefixed by `payment-` of the `Prod Kafka Cluster`.
## Mask all fields
When you don't know the message structure ahead of time, or want to mask everything by default and grant exceptions through RBAC, turn on **Mask all fields**. Console then applies the masking rule to every scalar value it finds in the message — strings, numbers, and booleans — at every nesting depth, including inside arrays.
A few things to know:
* **Type preservation.** Strings are replaced using the masking rule's string (for example `***`). Numbers are masked to `0` and booleans to `false`, so message schemas stay valid.
* **Per-field rules win.** If another policy already targets a specific field with its own masking rule, that rule still applies; **Mask all fields** only fills in the unmasked gaps.
* **Conflict detection.** A **Mask all fields** policy conflicts with any existing policy whose scope overlaps and that targets specific fields. Two **Mask all fields** policies can coexist as long as their resource scopes are disjoint.
* **Fields list.** You can't combine **Mask all fields** with an explicit fields list on the same policy — Console rejects the request if both are set.
## Field path syntax
Fields are specified using **dot-separated paths** that match the structure of your JSON messages. The masking engine traverses your message and applies the rule when the path matches.
### Examples
Given the following message:
```json theme={null}
{
"name": "Alice",
"address": {
"city": "Paris",
"zip": "75001"
},
"orders": [
{ "id": 1, "total": 99.90 },
{ "id": 2, "total": 45.00 }
],
"metadata": {
"tags": ["vip", "eu"],
"audit.source": "web"
}
}
```
| Field path | What gets masked | Explanation |
| ----------------------------- | ------------------- | ---------------------------------------------------- |
| `name` | `"Alice"` | Top-level field |
| `address.city` | `"Paris"` | Nested field using dot notation |
| `address.zip` | `"75001"` | Another nested field |
| `orders.total` | `99.90` and `45.00` | Field inside **every** element of the `orders` array |
| `orders.id` | `1` and `2` | Another field across all array elements |
| `metadata.tags` | `"vip"` and `"eu"` | Every element of a primitive array |
| `` metadata.`audit.source` `` | `"web"` | Field name containing a dot, escaped with backticks |
### Key rules
* Use **dot notation** to traverse nested objects (e.g. `address.city`).
* **Arrays are traversed automatically** — you do not need to specify indices. A path like `orders.total` applies to the `total` field in every element of the `orders` array.
* If a field name itself contains a dot, wrap it in **backticks** (e.g. `` `audit.source` `` or `` metadata.`audit.source` ``).
* Paths must match exactly — wildcard or prefix patterns (e.g. `data_order*`) are not supported.
## Validate a policy
Once you have created a policy, you should validate it through the Conduktor Console.
* Navigate to a topic that contains data where your policy should be applied
* Check that the expected fields are obfuscated using the appropriate masking rule
We can see that the name and the credit\_card are completely hidden, as we defined in the masking rules.
When the message key or value can't be transformed into a JSON-like structure, the whole message won't be displayed.
## Related resources
* [Configure field-level encryption in Gateway](/guide/tutorials/configure-encryption)
* [Set up RBAC](/guide/conduktor-in-production/admin/set-up-rbac)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Kafka data validation policies
Source: https://docs.conduktor.io/guide/conduktor-in-production/admin/gateway-policies
Validate Kafka producer records with Conduktor Gateway policies. Define CEL rules, JSON Schema checks, or Schema ID validation to block or flag invalid records.
## Validate using data quality rules and policies
The `DataQualityPlugin` Interceptor validates Kafka producer records using Policies made up of:
* custom Rules defined with [CEL (Common Expression Language)](https://github.com/google/cel-spec) expressions
* built-in Rules such as checking for Schema ID
* or a static [JSON schema](https://json-schema.org/) definition
It works by evaluating every record produced to matching topics and taking action on violations.
This is the recommended Interceptor to start with for data quality validation, and it is the same Interceptor that is deployed when you create [data quality Policies in Console](/guide/use-cases/enforce-data-quality). You should use Console when you want to reuse Rules and Policies, but you can also validate data quality using only Gateway. This guide explains how to deploy the Interceptor directly on Gateway without Console.
**From our blog:** [Kafka policy enforcement](https://conduktor.io/blog/kafka-policy-enforcement) How to move from 50-item policy checklists to automated, enforceable rules at the proxy layer.
### Examples
#### Mark and report with dead letter queue
Mark violating records on the `customer-transaction` topic and send them to a dead letter queue. Records are not blocked and still reach the original topic.
```bash theme={null}
curl \
--request PUT \
--url 'http://localhost:8888/gateway/v2/interceptor' \
--header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
--header 'Content-Type: application/json' \
--data-raw '{
"kind": "Interceptor",
"apiVersion": "gateway/v2",
"metadata": {
"name": "customer-policy"
},
"spec": {
"pluginClass": "io.conduktor.gateway.interceptor.dataquality.DataQualityPlugin",
"priority": 1,
"config": {
"policyName": "customer-policy",
"topicsRegex": [
"^customer-transaction.*$"
],
"block": false,
"mark": true,
"report": true,
"dlq": true,
"dlqTopic": "customer-transaction-dlq",
"maxNumberOfViolationReportPerSecond": 10,
"consoleDeploymentId": "standalone",
"rules": {
"email-rule": {
"type": "CEL",
"expression": "value.customer.email.matches(r\"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}\")"
},
"age-rule": {
"type": "CEL",
"expression": "type(value.customer.age) == int && value.customer.age >= 0 && value.customer.age <= 130"
},
"classification-header-rule": {
"type": "CEL",
"expression": "\"data-classification\" in headers && headers[\"data-classification\"] in [\"C0\", \"C1\", \"C2\", \"C3\"]"
}
}
}
}
}'
```
```yaml theme={null}
apiVersion: gateway/v2
kind: Interceptor
metadata:
name: customer-policy
scope:
vCluster: passthrough
spec:
pluginClass: io.conduktor.gateway.interceptor.dataquality.DataQualityPlugin
priority: 1
config:
policyName: customer-policy
topicsRegex:
- "^customer-transaction.*$"
block: false
mark: true
report: true
dlq: true
dlqTopic: customer-transaction-dlq
maxNumberOfViolationReportPerSecond: 10
consoleDeploymentId: standalone
rules:
email-rule:
type: CEL
expression: >
value.customer.email.matches(r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}")
age-rule:
type: CEL
expression: >
type(value.customer.age) == int
&& value.customer.age >= 0
&& value.customer.age <= 130
classification-header-rule:
type: CEL
expression: >
'data-classification' in headers
&& headers['data-classification'] in ['C0', 'C1', 'C2', 'C3']
```
Apply with:
```bash theme={null}
conduktor apply -f customer-policy.yaml
```
#### Enforce schema ID
If your schema registry requires authentication, pass credentials using `additionalConfigs`. Avoid exposing secrets in your configuration with secured templates. Gateway will use its local environment variables to resolve at runtime.
```bash theme={null}
curl \
--request PUT \
--url 'http://localhost:8888/gateway/v2/interceptor' \
--header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
--header 'Content-Type: application/json' \
--data-raw '{
"kind": "Interceptor",
"apiVersion": "gateway/v2",
"metadata": {
"name": "secured-dq-policy"
},
"spec": {
"pluginClass": "io.conduktor.gateway.interceptor.dataquality.DataQualityPlugin",
"priority": 1,
"config": {
"policyName": "secured-dq-policy",
"topicsRegex": [
"^sensitive-.*$"
],
"block": false,
"mark": true,
"report": true,
"maxNumberOfViolationReportPerSecond": 10,
"consoleDeploymentId": "standalone",
"rules": {
"enforce-schema": {
"type": "ENFORCE_SCHEMA_ID"
}
},
"schemaRegistryConfig": {
"host": "http://schema-registry:8081",
"type": "CONFLUENT",
"additionalConfigs": {
"basic.auth.credentials.source": "USER_INFO",
"basic.auth.user.info": "${SR_USER}:${SR_PASSWORD}",
"schema.registry.basic.auth.user.info": "${SR_USER}:${SR_PASSWORD}"
}
}
}
}
}'
```
```yaml theme={null}
apiVersion: gateway/v2
kind: Interceptor
metadata:
name: secured-dq-policy
scope:
vCluster: passthrough
spec:
pluginClass: io.conduktor.gateway.interceptor.dataquality.DataQualityPlugin
priority: 1
config:
policyName: secured-dq-policy
topicsRegex:
- ^sensitive-.*$
block: false
mark: true
report: true
maxNumberOfViolationReportPerSecond: 10
consoleDeploymentId: standalone
rules:
enforce-schema:
type: ENFORCE_SCHEMA_ID
schemaRegistryConfig:
host: http://schema-registry:8081
type: CONFLUENT
additionalConfigs:
basic.auth.credentials.source: USER_INFO
basic.auth.user.info: $${SR_USER}:$${SR_PASSWORD}
schema.registry.basic.auth.user.info: $${SR_USER}:$${SR_PASSWORD}
```
Apply with:
```bash theme={null}
conduktor apply -f secured-dq-policy.yaml
```
#### Block records that violate a JSON schema
Block any record on the `product` topic that does not match the expected product schema. This is useful for defending against poison pill records from errant CLI tools or misconfigured producers that write malformed data to a topic, which can cause downstream consumers to crash. No schema registry is needed since the JSON schema is defined inline in the Rule.
```bash theme={null}
curl \
--request PUT \
--url 'http://localhost:8888/gateway/v2/interceptor' \
--header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
--header 'Content-Type: application/json' \
--data-raw '{
"kind": "Interceptor",
"apiVersion": "gateway/v2",
"metadata": {
"name": "product-schema-policy"
},
"spec": {
"pluginClass": "io.conduktor.gateway.interceptor.dataquality.DataQualityPlugin",
"priority": 1,
"config": {
"policyName": "product-schema-policy",
"topicsRegex": [
"^product$"
],
"block": true,
"mark": false,
"report": true,
"dlq": true,
"dlqTopic": "product-dlq",
"maxNumberOfViolationReportPerSecond": 10,
"consoleDeploymentId": "standalone",
"rules": {
"valid-product-schema": {
"type": "JSON_SCHEMA",
"schema": "{\"$comment\":\"setting additionalProperties to false to ensure no extra fields are allowed\",\"$id\":\"https://example.com/product.schema.json\",\"$schema\":\"https://json-schema.org/draft/2020-12/schema\",\"additionalProperties\":false,\"description\":\"A product from Acme''s catalog\",\"properties\":{\"price\":{\"description\":\"The price of the product\",\"exclusiveMinimum\":0,\"type\":\"number\"},\"productId\":{\"description\":\"The unique identifier for a product\",\"type\":\"integer\"},\"productName\":{\"description\":\"Name of the product\",\"type\":\"string\"},\"tags\":{\"description\":\"Tags for the product\",\"items\":{\"type\":\"string\"},\"minItems\":1,\"type\":\"array\",\"uniqueItems\":true}},\"required\":[\"productId\",\"productName\",\"price\"],\"title\":\"Product\",\"type\":\"object\"}"
}
}
}
}
}'
```
```yaml theme={null}
apiVersion: gateway/v2
kind: Interceptor
metadata:
name: product-schema-policy
scope:
vCluster: passthrough
spec:
pluginClass: io.conduktor.gateway.interceptor.dataquality.DataQualityPlugin
priority: 1
config:
policyName: product-schema-policy
topicsRegex:
- ^product$
block: true
mark: false
report: true
dlq: true
dlqTopic: product-dlq
maxNumberOfViolationReportPerSecond: 10
consoleDeploymentId: standalone
rules:
valid-product-schema:
type: JSON_SCHEMA
schema: |
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://example.com/product.schema.json",
"title": "Product",
"description": "A product from Acme's catalog",
"type": "object",
"$comment": "setting additionalProperties to false to ensure no extra fields are allowed",
"additionalProperties": false,
"properties": {
"productId": {
"description": "The unique identifier for a product",
"type": "integer"
},
"productName": {
"description": "Name of the product",
"type": "string"
},
"price": {
"description": "The price of the product",
"type": "number",
"exclusiveMinimum": 0
},
"tags": {
"description": "Tags for the product",
"type": "array",
"items": {
"type": "string"
},
"minItems": 1,
"uniqueItems": true
}
},
"required": [ "productId", "productName", "price" ]
}
```
Apply with:
```bash theme={null}
conduktor apply -f product-schema-policy.yaml
```
### Configuration
| Key | Type | Required | Default | Description |
| :------------------------------------ | :------------------------------------------------------ | :------- | :------ | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `policyName` | String | Yes | | A name to identify this Policy. Violation reports and marking headers reference this name. |
| `topicsRegex` | List\ | Yes | | List of regex patterns matching the topics to enforce. For example, `["^orders-.*$", "^payments$"]`. |
| `rules` | Map\ | Yes | | A map of Rule names to Rule definitions. The Interceptor evaluates each Rule against every record. |
| `block` | Boolean | Yes | | If `true`, reject the entire batch when any record violates a Rule. The producer receives an `INVALID_RECORD` error. |
| `mark` | Boolean | No | `false` | If `true` (and `block` is `false`), add a `conduktor.dataquality.violations` header to records that violate Rules. |
| `report` | Boolean | Yes | | If `true`, write violation events to an internal Gateway topic for monitoring. See [violation reporting](#violation-reporting). |
| `maxNumberOfViolationReportPerSecond` | Integer | Yes | | Rate limit for violation reports per second. A value of `10` is a good starting point. |
| `dlq` | Boolean | No | `false` | If `true`, send violating records to a dead letter topic. |
| `dlqTopic` | String | No | | The topic name to use as a dead letter queue. Required if `dlq` is `true`. |
| `schemaRegistryConfig` | [Schema registry](#data-quality-policy-schema-registry) | No | | Schema registry configuration. Required if your Rules use the `ENFORCE_AVRO` or `ENFORCE_SCHEMA_ID` types. |
| `schemaRegistryConnectionName` | String | No | | Name of a [schema registry external connection](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/external-connections). An alternative to `schemaRegistryConfig` — set one or the other, not both. |
| `consoleDeploymentId` | String | Yes | | An identifier used to separate violation data across deployments. If you are not using Console, provide any placeholder value (for example, `"standalone"`). This field will be made optional in a future version. |
If you are deploying this Interceptor without Console, set `consoleDeploymentId` to any placeholder value such as `"standalone"`. This field will be made optional in a future version.
### Rules
Each Rule in the `rules` map has a `type` field that determines how the record is validated. The Interceptor supports four Rule types.
#### CEL expression
Evaluate a CEL expression against the record. The expression has to return a boolean: `true` means the record passes, `false` means it violates the Rule.
| Key | Type | Required | Description |
| :----------- | :----- | :------- | :------------------------------------------------------------------------------------------------------------------- |
| `type` | String | Yes | `CEL` |
| `expression` | String | Yes | CEL expression to evaluate. |
| `message` | String | No | Custom error message returned to the producer when blocking. Defaults to `Data quality Rule '' violated`. |
**Available variables in CEL expressions:**
* `value` — the deserialized record value (access nested fields with dot notation, for example `value.customer.email`)
* `key` — the record key
* `headers` — a map of header names to values
* `topic` — the topic name
* `partition` — the partition number
* `offset` — the record offset
**Example CEL expressions:**
| Use case | Expression |
| :------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Email format | value.customer.email.matches(r"\[a-zA-Z0-9.\_%+-]+@\[a-zA-Z0-9.-]+.\[a-zA-Z]\{2,}") |
| UUID format | value.id.matches(r"^\[0-9a-fA-F]\{8}-\[0-9a-fA-F]\{4}-\[0-9a-fA-F]\{4}-\[0-9a-fA-F]\{4}-\[0-9a-fA-F]\{12}\$") |
| Range check | `value.customer.age >= 0 && value.customer.age <= 130` |
| Header validation | `'data-classification' in headers && headers['data-classification'] in ['C0', 'C1', 'C2', 'C3']` |
| Conditional required field | !value.customer.preferences.newsletter \|\| (value.customer.preferences.preferred\_language != null && value.customer.preferences.preferred\_language != "") |
#### JSON schema
Validate the record value against a JSON schema definition. This is especially helpful when your producer team wants to enforce standards on produced data without breaking downstream consumers by introducing a schema registry.
| Key | Type | Required | Description |
| :-------- | :----- | :------- | :----------------------------------------------------------- |
| `type` | String | Yes | `JSON_SCHEMA` |
| `schema` | String | Yes | A JSON schema definition (as a string). |
| `message` | String | No | Custom error message returned to the producer when blocking. |
#### ENFORCE\_AVRO built-in Rule
Verify that the record value is Avro-encoded: it carries a schema ID, the schema ID exists in the schema registry and the schema is of type Avro. Requires `schemaRegistryConfig` to be set.
| Key | Type | Required | Description |
| :-------- | :----- | :------- | :-------------------- |
| `type` | String | Yes | `ENFORCE_AVRO` |
| `message` | String | No | Custom error message. |
#### ENFORCE\_SCHEMA\_ID built-in Rule
Verify that the record value carries a valid schema ID and that the schema exists in the schema registry. Requires `schemaRegistryConfig` to be set.
| Key | Type | Required | Description |
| :-------- | :----- | :------- | :-------------------- |
| `type` | String | Yes | `ENFORCE_SCHEMA_ID` |
| `message` | String | No | Custom error message. |
### Data quality policy schema registry
A schema registry is only required if your topics use schema-encoded data. If your topics contain plain JSON, you can omit the `schemaRegistryConfig` field entirely.
Gateway reads schemas either through an external connection or from configuration on the Interceptor itself.
Schema-encoded records whose schema ID is stored in a record header, the format [introduced in Confluent Platform 8.2](https://www.confluent.io/blog/schema-id-kafka-headers-data-governance), require **Gateway 3.21.0** or later.
External connections are a preview feature and subject to change. Inline `schemaRegistryConfig` remains fully supported.
Set `schemaRegistryConnectionName` to the name of a [schema registry external connection](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/external-connections#connect-to-a-schema-registry), declared once when Gateway starts with variables that follow this pattern:
```
GATEWAY_EXTERNAL_CONNECTION__CONFIG_
```
| Key | Type | Description |
| ------------------------------ | ------ | ----------------------------------------------------------------------------------------------------- |
| `schemaRegistryConnectionName` | string | The `` of a schema registry connection. Gateway rejects an unknown name with `400 Bad Request`. |
For a connection declared as `GATEWAY_EXTERNAL_CONNECTION_DEFAULT_SR_CONFIG_*`, the Interceptor reaches it with:
```json theme={null}
"schemaRegistryConnectionName": "DEFAULT_SR"
```
Set either `schemaRegistryConnectionName` or `schemaRegistryConfig`, never both. Gateway rejects a configuration that sets both with `400 Bad Request`, rather than silently picking one.
Remove a connection an Interceptor already names and the Interceptor stays visible through the API but stops applying to traffic. [See what happens at startup](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/external-connections#what-happens-at-startup).
Gateway supports **Confluent-like** and **AWS Glue** schema registries.
| Key | Type | Default | Description |
| --------------------- | ------ | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `type` | string | `CONFLUENT` | The type of schema registry to use: choose `CONFLUENT` (for Confluent-like schema registries including OSS Kafka) or `AWS` for AWS Glue schema registries. |
| `additionalConfigs` | map | | Additional properties maps to specific security-related parameters. For enhanced security, you can hide the sensitive values using [environment variables as secrets](/guide/reference/data-security#environment-variables-as-secrets). |
| **Confluent-like** | | | **Configuration for Confluent-like schema registries** |
| `host` | string | | URL of your schema registry. |
| `cacheSize` | string | `50` | Number of schemas that can be cached locally by this Interceptor so that it doesn't have to query the schema registry every time. |
| **AWS Glue** | | | **Configuration for AWS Glue schema registries** |
| `region` | string | | The AWS region for the schema registry, e.g. `us-east-1`. |
| `registryName` | string | | The name of the schema registry in AWS (leave blank for the AWS default of `default-registry`). |
| `basicCredentials` | string | | Access credentials for AWS. |
| **AWS credentials** | | | **AWS credential configuration** |
| `accessKey` | string | | The access key for the connection to the schema registry. |
| `secretKey` | string | | The secret key for the connection to the schema registry. |
| `validateCredentials` | bool | `true` | Whether Gateway rejects a blank `accessKey` or `secretKey` when it builds the credentials. Set it to `false` to accept them and fail later instead. |
| `accountId` | string | | The Id for the AWS account to use. |
If you don't supply a `basicCredentials` section for the AWS Glue schema registry, the client will attempt to find the connection information it needs from the environment ([see AWS docs for details](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default) ) and the credentials required can be passed this way to Gateway as part of its core configuration.
### Actions
The Interceptor supports three actions that can be combined:
| Action | Behavior |
| :-------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Block** (`block: true`) | Reject the **entire Kafka record batch** for a partition if **any** record in it violates a Rule. The producer receives an `INVALID_RECORD` error (Kafka error code 87) with a message identifying the violated Rules. See [handle blocked batches](#handle-blocked-batches). |
| **Mark** (`mark: true`) | Add a `conduktor.dataquality.violations` header to violating records. The header value is a JSON object mapping Policy names to arrays of violated Rule names. The Interceptor only applies marking when `block` is `false`. |
| **Report** (`report: true`) | Write violation events to an internal topic for monitoring. See [violation reporting](#violation-reporting). |
If both `block` and `mark` are enabled, only blocking is applied.
When blocking is enabled, a single invalid record causes the entire record batch for that partition to be rejected. This means valid records in the same record batch are also blocked. If you need to preserve valid records while flagging violations, use `mark` instead of `block`.
#### Mark header format
When marking is enabled, the `conduktor.dataquality.violations` header contains a JSON object:
```json theme={null}
{
"my-policy-name": ["email-rule", "age-rule"]
}
```
Multiple Interceptors targeting the same topic will each append their Policy to this header.
### Dead letter queue
When `dlq` is `true` and `dlqTopic` is set, the Interceptor sends violating records to the dead letter topic with the following headers:
| Header | Description |
| :----------------- | :---------------------------------------------------------------------------------------- |
| `X-ERROR-MSG` | Description of the violation including the Policy name, violated Rules and error details. |
| `X-TOPIC` | The topic the record was originally destined for. |
| `X-PARTITION` | The partition the record was intended for. |
| `X-POLICY` | The Policy name that was violated. |
| `X-VIOLATED-RULES` | Comma-separated list of violated Rule names. |
The Interceptor sends the violating records to the dead letter topic regardless of whether the action is `block` or `mark`. The `block` action will block the entire record batch that contains the violating record, but even so, only the violating record will be sent to the dead letter topic.
### Handle blocked batches
When `block` is enabled, the producer receives an `INVALID_RECORD` error (Kafka error code 87). This error is **non-retriable** and affects all records accumulated in the same record batch as the violating record. Because the Kafka producer client will not automatically retry the request, `block` should only be used when it is imperative that violating data not be written to disk.
The error message returned to the producer identifies the violated Rules, for example: `Data quality Rule 'email-rule' violated. Data quality Rule 'age-rule' violated`. If custom `message` fields are set on the Rules, those messages are used instead.
#### Handle producer errors
When testing in a development environment or favoring a "fail fast" approach to data quality validation, you should throw the `InvalidRecordException` in the record send callback to stop the application.
If the Kafka producer doesn't have an error callback or the callback is written to simply log the error and continue, then **all records in the same record batch as the violating record will be lost**.
If you decide a violating record must be blocked, but do not want to lose other passing records in the record batch, you must handle this error explicitly. One valid approach is:
1. Create a separate Kafka producer object with `batch.size = 1` and `linger.ms = 0` to only send records one at a time.
2. Catch `InvalidRecordException` so each record in the failed batch is passed to the separate producer in a different thread and retried. Use `.flush()` to ensure each record is sent by itself.
3. If this second send fails with the same exception, then we know this is a bad record and you can log the error message and continue.
The tradeoff to this approach is that it will **not preserve record ordering**. The record ordering is affected in two ways:
* The records in the failed batch will be retried out of order because the callback is executed asynchronously
* The main producer will continue to produce even while records in the failed batch are being retried
### Violation reporting
When `report` is `true`, the Interceptor writes Avro-encoded records to an internal Kafka topic named:
```
_conduktor_{gatewayClusterId}_data_quality_violation
```
All records on this topic are wrapped in a `DataQualityEvent` Avro envelope. Two event types are written:
#### Violation events
The Interceptor writes a `DataQualityViolation` event each time a record fails validation.
| Field | Type | Description |
| :-------------------- | :------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------- |
| `topic` | String | The topic where the violation occurred. |
| `partition` | Integer | The partition number. |
| `createdAt` | Instant | Timestamp when the violation was detected. |
| `policy` | String | The Policy name. |
| `violatedRules` | List\ | Rule names that were violated. |
| `gatewayClusterId` | String | ID of the Gateway cluster. |
| `vCluster` | String | Virtual Cluster name. |
| `serviceAccount` | String | The service account that produced the record. |
| `clientId` | String | Kafka client ID of the producer. |
| `errors` | Map\ | Error details per Rule (Rule name to error message). |
| `consoleDeploymentId` | String | The deployment identifier. |
| `offset` | Long or Object | The record offset. When the record was blocked, this contains batch metadata (batch size and offset within the batch) instead of the Kafka offset. |
| `actions` | Object | The actions that were applied (block, report, mark, dlq). |
The `maxNumberOfViolationReportPerSecond` configuration controls the rate at which the Interceptor writes violation events to this topic to prevent flooding under high violation volumes.
#### Evaluation count events
The Interceptor writes a `DataQualityChecksCount` event periodically (not per-record) with aggregated counts of all records evaluated, including records that passed validation. This provides the denominator for calculating violation rates.
Each event contains a list of counts grouped by cluster, topic, Policy and Rule:
| Count type | Fields | Description |
| :----------------- | :-------------------------------------------- | :--------------------------------------- |
| `PolicyCheckCount` | `cluster`, `topic`, `policy`, `value` | Total records evaluated for this Policy. |
| `RuleCheckCount` | `cluster`, `topic`, `policy`, `rule`, `value` | Total records evaluated for this Rule. |
This topic is also used by Conduktor Console to display violation metrics and history.
## Validate data using SQL-like checks
Conduktor Gateway offers a `DataQualityProducerPlugin` Interceptor that uses a SQL-like language to assert data quality before it's being produced.
Records in the topic from the FROM clause have to match the WHERE clause for the statement in order to be considered valid. This is particularly useful if your data is plain JSON with no schema but it can also be applied to AVRO, Protobuf data.
### Example
You have a topic for orders with records in this form:
```json theme={null}
{
"id" : "B2EE6886-7FFF-4CAB-9B2A-CF0A06C9E648",
"amount_cents": 12499,
"currency": "EUR",
"order_date": "2024-20-12T15:45:33Z"
}
```
You may want to ensure that the:
* `id` is a valid UUID format
* `amount_cents` is a positive integer and not too large
* `currency` is one of your accepted currencies
* `order_date` is in ISO 8601 format
This can be asserted with:
```sql theme={null}
SELECT
* -- ignored in this policy
FROM
orders -- topic to enforce the rule on
WHERE
id REGEXP '^[0-9A-F-]{36}$' -- 36 char UUID in hex with dash separators
AND amount_cents REGEXP '[0-9]+' -- amount must be an integer
AND amount_cents > 0 -- ... greater than zero
AND amount_cents < 1000000 -- ... and less than 1,000,000 cents
AND currency REGEXP 'EUR|GBP|USD' -- currency must be one of these three
AND order_date REGEXP '^20[2-9][0-9]-[0-1][0-9]-[0-3][0-9]T[0-2][0-9]:[0-5][0-9]:[0-5][0-9]Z$'
-- valid date, and after 2020
```
In the statement, the list of selected fields is actually ignored - the important parts are the FROM clause (specifying the topic that the policy is applied to) and the WHERE clause, which specifies the condition data must meet in order to be considered valid. i.e. If the select returns something, the record is valid. If it returns no results, the record is considered invalid.
`SELECT [ignored!] FROM [topic name] WHERE [field filter criteria]`
Only one topic can be specified in the FROM clause (joins will be ignored), and the topic name is matched explicitly (no regexp support). If a record does not match the WHERE clause, it will be rejected. There are a variety of options for this described in the actions below. Fields are assumed to be from the value of the record. The Interceptor currently supports values in JSON, AVRO and Protobuf formats.
Topic names with dash `-` characters in them must be double quoted, as the dash is not a valid character for a SQL name. E.g. for a topic `our-orders` you would need to use:
`SELECT * FROM "our-orders" WHERE ...`
Nested fields can be accessed as expected with dot notation in the WHERE clause, e.g.:
`address.street = 'Electric Avenue'`
#### WHERE clause
If you specify a field name in the WHERE clause that doesn't exist in the record, the condition will always fail and the record will always be considered invalid. Fields in the WHERE clause have to exist in a record for it to be considered valid.
The WHERE clause supports a subset of SQL operations:
* The operators `=, >, >=, <, <=, <>` and `REGEXP` (RegExp MySQL Operator)
* When providing more than one condition in the WHERE clause, only the `AND`
* The `IN` clause is not supported, but can be approximated with a RegExp
* By default, the fields in the WHERE clause are looked up from the value in the record. You can also filter by other parts of the record using the syntax below:
* Record key (it also supports encoded keys which require a schema registry lookup):
* Record key as string: - `.. WHERE record.key = 'some thing'`
* Record key as schema: `.. WHERE record.key.someValue.someChildValue = 'some thing'`
* Partition: `.. WHERE record.partition = 1`
* Timestamp: `.. WHERE record.timestamp = 98717823712`
* Header: `.. WHERE record.header.someHeaderKey = 'some thing'`
* Offset: `.. WHERE record.offset = 1`
#### Actions for invalid data
The policy acts on produce requests from Kafka clients which means it will often deal with a batch of multiple records spread over multiple topics and partitions. The policy can apply different effects to each request batch based on its configuration.
| Action | Description |
| :------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| BLOCK\_WHOLE\_BATCH | If any records in the batch are invalid, then block the whole batch. The produce request will fail for the client in this case. |
| AUDIT\_LOG\_ONLY | For any records in the produce request which are invalid, record this in the audit log only. All records still are saved in Kafka |
| THROTTLE | If any records in the produce request are invalid, throttle the producer for a certain amount of time (`throttleTimeMs`). All records are still saved in Kafka. |
#### Dead letter topic
If a dead letter topic service is configured for Gateway, you can optionally supply a topic name for this policy to use for any records which are considered invalid. This topic will be created with the default config for your Kafka setup.
Any record that the policy considers invalid, is written to the dead letter topic and has some headers added for audit purposes. Please note that this is also done in the `AUDIT_LOG_ONLY` mode, even though the records in this mode are still written to the "real" topic.
| Header | Message |
| :---------- | :------------------------------------------------------------------- |
| X-ERROR-MSG | Message does not match the statement \[ ...] |
| X-TOPIC | The topic that the message was intended to be written to |
| X-PARTITION | The partition of the topic the message was intended to be written to |
The generation of these headers can be disabled with the `addErrorHeader` configuration parameter (defaults to `true`).
If no `deadLetterTopic` is configured for the policy, no messages will be written out in this manner.
#### Audit log
Any policy violation is logged in the configured Gateway audit log. This is currently logged at the *batch* level for each topic in the produce request. There's no per record audit - it identifies that a policy breach occurred for the produce request and identifies the tenant, username and client IP for the request.
#### Configuration
| Key | Type | Description |
| :--------------------------- | :-------------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| statement | String | SQL statement |
| schemaRegistryConfig | [Schema registry](#sql-like-checks-schema-registry) | Schema registry config |
| schemaRegistryConnectionName | String | Name of a [schema registry external connection](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/external-connections). An alternative to `schemaRegistryConfig` — set one or the other, not both. |
| action | [Action](#action) | Data quality producer action |
| deadLetterTopic | String | Dead letter topic |
| addErrorHeader | boolean (default `true`) | Adds the error information headers into dead letter topic |
| throttleTimeMs | int (default: 100) | Value to throttle with (only applicable when action is set to `THROTTLE`). |
##### SQL-like checks schema registry
Gateway reads schemas either through an external connection or from configuration on the Interceptor itself.
External connections are a preview feature and subject to change. Inline `schemaRegistryConfig` remains fully supported.
Set `schemaRegistryConnectionName` to the name of a [schema registry external connection](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/external-connections#connect-to-a-schema-registry), declared once when Gateway starts with variables that follow this pattern:
```
GATEWAY_EXTERNAL_CONNECTION__CONFIG_
```
| Key | Type | Description |
| ------------------------------ | ------ | ----------------------------------------------------------------------------------------------------- |
| `schemaRegistryConnectionName` | string | The `` of a schema registry connection. Gateway rejects an unknown name with `400 Bad Request`. |
For a connection declared as `GATEWAY_EXTERNAL_CONNECTION_DEFAULT_SR_CONFIG_*`, the Interceptor reaches it with:
```json theme={null}
"schemaRegistryConnectionName": "DEFAULT_SR"
```
Set either `schemaRegistryConnectionName` or `schemaRegistryConfig`, never both. Gateway rejects a configuration that sets both with `400 Bad Request`, rather than silently picking one.
Remove a connection an Interceptor already names and the Interceptor stays visible through the API but stops applying to traffic. [See what happens at startup](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/external-connections#what-happens-at-startup).
Gateway supports **Confluent-like** and **AWS Glue** schema registries.
| Key | Type | Default | Description |
| --------------------- | ------ | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `type` | string | `CONFLUENT` | The type of schema registry to use: choose `CONFLUENT` (for Confluent-like schema registries including OSS Kafka) or `AWS` for AWS Glue schema registries. |
| `additionalConfigs` | map | | Additional properties maps to specific security-related parameters. For enhanced security, you can hide the sensitive values using [environment variables as secrets](/guide/reference/data-security#environment-variables-as-secrets). |
| **Confluent-like** | | | **Configuration for Confluent-like schema registries** |
| `host` | string | | URL of your schema registry. |
| `cacheSize` | string | `50` | Number of schemas that can be cached locally by this Interceptor so that it doesn't have to query the schema registry every time. |
| **AWS Glue** | | | **Configuration for AWS Glue schema registries** |
| `region` | string | | The AWS region for the schema registry, e.g. `us-east-1`. |
| `registryName` | string | | The name of the schema registry in AWS (leave blank for the AWS default of `default-registry`). |
| `basicCredentials` | string | | Access credentials for AWS. |
| **AWS credentials** | | | **AWS credential configuration** |
| `accessKey` | string | | The access key for the connection to the schema registry. |
| `secretKey` | string | | The secret key for the connection to the schema registry. |
| `validateCredentials` | bool | `true` | Whether Gateway rejects a blank `accessKey` or `secretKey` when it builds the credentials. Set it to `false` to accept them and fail later instead. |
| `accountId` | string | | The Id for the AWS account to use. |
If you don't supply a `basicCredentials` section for the AWS Glue schema registry, the client will attempt to find the connection information it needs from the environment ([see AWS docs for details](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default) ) and the credentials required can be passed this way to Gateway as part of its core configuration.
[Read our blog about schema registry](https://www.conduktor.io/blog/what-is-the-schema-registry-and-why-do-you-need-to-use-it/) .
#### Action
| Action | Description |
| :------------------ | :--------------------------------------------------------------------------------------------- |
| BLOCK\_WHOLE\_BATCH | If one message is invalid, block the whole batch |
| AUDIT\_LOG\_ONLY | If messages are invalid, audit log only (all messages still are saved in Kafka) |
| THROTTLE | If messages are invalid, throttle the producer for a certain amount of time (`throttleTimeMs`) |
#### Example
```json theme={null}
{
"name": "myDataQualityProducerPlugin",
"pluginClass": "io.conduktor.gateway.interceptor.safeguard.DataQualityProducerPlugin",
"priority": 100,
"config": {
"statement": "SELECT x FROM orders WHERE amount_cents > 0 AND amount_cents < 1000000",
"schemaRegistryConfig": {
"host": "http://schema-registry:8081"
},
"action": "BLOCK_WHOLE_BATCH",
"deadLetterTopic": "dead-letter-topic",
"addErrorHeader": false
}
}
```
## Validate schema payload
To enable your Kafka consumers to confidently and independently access data, ensure that all records sent through your Kafka system conform to an agreed structure. Records with missing or invalid schemas can cause application outages, as consumers may be unable to process the unexpected record format.
Moreover, the use of schemas can broadly only assert structural correctness of data and a level of compatibility for those structures. When it comes to the *values* in a record often all you can assert is a basic data type (integer, string, double etc.). This means that using a valid schema solves *some* concerns around data quality, there are other concerns to be dealt with in a bespoke or distributed manner across all the clients of a given data type.
Finally, while correct structure can be enforced in a Kafka ecosystem at a client level - each client needs to ensure that it knows and follows the expectations for the data. You cannot prevent one client correctly writing AVRO to a topic, while another one writes plain JSON to the same topic. If one client doesn't know the rules, it can't follow them.
### Enforce centralized policies
The schema validation Interceptor provides functionality that can be configured once in your Kafka system on the source for data (a topic), to ensure that:
* All records produced to Kafka have a schema set
* The record contents adhere to that schema
* The fields (values) in any given record comply to business validation rules you have set in the schema
This policy provides a centralized enforcement of the validation of these rules at the point of write to Kafka. This enforcement cannot be bypassed or ignored by a client, so provides a strong guarantee that data actually written in to Kafka does match your rule set.
[Read our blog about schema registry](https://www.conduktor.io/blog/what-is-the-schema-registry-and-why-do-you-need-to-use-it/) .
#### How does the Policy Work?
The policy operates on Produce Requests made to Kafka, and will inspect the entire batch of records in a request. Based on its setup, it performs various checks and then will take an action if it finds any problems.
The first important thing to note is that the Policy will do *nothing* if there is no Audit Log configured for Gateway (as it does not want to silently reject any data). So for the policy to work at all, you **must have the Audit Log configured**.
Next point of note is that the policy will only check the value for a Kafka record, and does not currently support validating the contents of the key or headers.
The core config values for the policy itself are:
* `topic` : the topic/s to apply the rule to
* `schemaIdRequired` : whether records must/must not have a schema assigned to them
* `validateSchema` : whether the policy should check if the data for the record matches the schema found for the record.
* `action` : what to do if a problem is found
Schema-encoded records whose schema ID is stored in a record header, the format [introduced in Confluent Platform 8.2](https://www.confluent.io/blog/schema-id-kafka-headers-data-governance), require **Gateway 3.21.0** or later.
There are three levels of check you can apply:
| Setup | Effect |
| :-------------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `schemaIdRequired = false` | Ensures that no records have a schema! |
| `schemaIdRequired = true`, `validateSchema = false` | Ensures that records have a valid schema set, and that schema exists in the schema registry. Does not check whether the value actually matches the schema though. |
| `schemaIdRequired = true`, `validateSchema = true` | Ensures that records have a valid schema set, that schema exists in the schema registry and that the value in the record matches the schema. This includes any data validation rules in the schema (see below) as well as a structural check. |
#### Action
If any problems are found, the policy will take an action as configured. The `action` can be set to one of:
* `BLOCK` → If any records in the batch fail the policy checks, record the problems in the audit log and then return an error to the client failing the entire batch. No records are written to Kafka at all if at least one of the records in the batch is considered invalid.
* `INFO` → In this mode the data is always written to Kafka whether it passes the checks or not - but any problems found recorded in the audit log.
* `THROTTLE` → If any records in the batch fail the policy checks, the data is still written to Kafka but the request will be throttled with time = `throttleTimeMs`, forcing the client to back off. Any problems found are recorded in the audit log.
#### Dead letter topic
If a dead letter topic service is configured for Gateway, then you can optionally supply a topic name for this policy to use for any records which are considered invalid. This topic will be created with the default config for your Kafka setup.
Any record which the policy considers invalid is written to the dead letter topic, and has some headers added for audit purposes. Please note that this is done in the `AUDIT_LOG_ONLY` mode also, even though the records in this mode are still written to the "real" topic.
| Header | Message |
| :---------- | :-------------------------------------------------------------------- |
| X-ERROR-MSG | Description of the reason for the policy failure |
| X-TOPIC | The topic the message was intended to be written to |
| X-PARTITION | The partition of that topic the message was intended to be written to |
The generation of these headers can be disabled if required, through the `addErrorHeader` configuration parameter (defaults to `true`).
If no `deadLetterTopic` is configured for the policy, then no messages will be written out in this manner.
#### Configuration
The full configuration topics for the policy are as below.
| Name | Type | Default | Description |
| :--------------------------- | :-------------------------- | :------ | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| topic | String | `.*` | Topics that match this regex will have the Interceptor applied |
| schemaIdRequired | Boolean | `false` | Records must/must not have schemaId |
| validateSchema | Boolean | `false` | If true, deserialize the record, validate the record structure and fields within the data itself. |
| action | `BLOCK`, `INFO`, `THROTTLE` | `BLOCK` | Action to take if the value is outside the specified range. |
| schemaRegistryConfig | Schema registry | N/A | Schema registry Config |
| schemaRegistryConnectionName | String | N/A | Name of a [schema registry external connection](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/external-connections). An alternative to `schemaRegistryConfig` — set one or the other, not both. |
| celCacheSize | int | 100 | In memory cache size for cel expressions, balancing speed and resource use, optimize performance. |
| deadLetterTopic | String | | Dead letter topic. Not used if this parameter is not set. |
| addErrorHeader | Boolean | `true` | Add or not add the error information headers into dead letter topic |
| throttleTimeMs | int | 100 | Value to throttle with (only applicable when action is set to `THROTTLE`). |
#### Schema payload validation schema registry
Gateway reads schemas either through an external connection or from configuration on the Interceptor itself.
External connections are a preview feature and subject to change. Inline `schemaRegistryConfig` remains fully supported.
Set `schemaRegistryConnectionName` to the name of a [schema registry external connection](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/external-connections#connect-to-a-schema-registry), declared once when Gateway starts with variables that follow this pattern:
```
GATEWAY_EXTERNAL_CONNECTION__CONFIG_
```
| Key | Type | Description |
| ------------------------------ | ------ | ----------------------------------------------------------------------------------------------------- |
| `schemaRegistryConnectionName` | string | The `` of a schema registry connection. Gateway rejects an unknown name with `400 Bad Request`. |
For a connection declared as `GATEWAY_EXTERNAL_CONNECTION_DEFAULT_SR_CONFIG_*`, the Interceptor reaches it with:
```json theme={null}
"schemaRegistryConnectionName": "DEFAULT_SR"
```
Set either `schemaRegistryConnectionName` or `schemaRegistryConfig`, never both. Gateway rejects a configuration that sets both with `400 Bad Request`, rather than silently picking one.
Remove a connection an Interceptor already names and the Interceptor stays visible through the API but stops applying to traffic. [See what happens at startup](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/external-connections#what-happens-at-startup).
Gateway supports **Confluent-like** and **AWS Glue** schema registries.
| Key | Type | Default | Description |
| --------------------- | ------ | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `type` | string | `CONFLUENT` | The type of schema registry to use: choose `CONFLUENT` (for Confluent-like schema registries including OSS Kafka) or `AWS` for AWS Glue schema registries. |
| `additionalConfigs` | map | | Additional properties maps to specific security-related parameters. For enhanced security, you can hide the sensitive values using [environment variables as secrets](/guide/reference/data-security#environment-variables-as-secrets). |
| **Confluent-like** | | | **Configuration for Confluent-like schema registries** |
| `host` | string | | URL of your schema registry. |
| `cacheSize` | string | `50` | Number of schemas that can be cached locally by this Interceptor so that it doesn't have to query the schema registry every time. |
| **AWS Glue** | | | **Configuration for AWS Glue schema registries** |
| `region` | string | | The AWS region for the schema registry, e.g. `us-east-1`. |
| `registryName` | string | | The name of the schema registry in AWS (leave blank for the AWS default of `default-registry`). |
| `basicCredentials` | string | | Access credentials for AWS. |
| **AWS credentials** | | | **AWS credential configuration** |
| `accessKey` | string | | The access key for the connection to the schema registry. |
| `secretKey` | string | | The secret key for the connection to the schema registry. |
| `validateCredentials` | bool | `true` | Whether Gateway rejects a blank `accessKey` or `secretKey` when it builds the credentials. Set it to `false` to accept them and fail later instead. |
| `accountId` | string | | The Id for the AWS account to use. |
If you don't supply a `basicCredentials` section for the AWS Glue schema registry, the client will attempt to find the connection information it needs from the environment ([see AWS docs for details](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default) ) and the credentials required can be passed this way to Gateway as part of its core configuration.
[Read our blog about schema registry](https://www.conduktor.io/blog/what-is-the-schema-registry-and-why-do-you-need-to-use-it/) .
#### Example
```json theme={null}
{
"name": "mySchemaIdValidationInterceptor",
"pluginClass": "io.conduktor.gateway.interceptor.safeguard.SchemaPayloadValidationPolicyPlugin",
"priority": 100,
"config": {
"topic": "topic_1.*",
"schemaIdRequired": true,
"validateSchema": true,
"schemaRegistryConfig": {
"host": "http://schema-registry:8081"
},
"action": "BLOCK",
"celCacheSize": 100
}
}
```
##### Schema registry with secured template
```json theme={null}
{
"name": "mySchemaIdValidationInterceptor",
"pluginClass": "io.conduktor.gateway.interceptor.safeguard.SchemaPayloadValidationPolicyPlugin",
"priority": 100,
"config": {
"topic": "topic_1.*",
"schemaIdRequired": true,
"validateSchema": true,
"schemaRegistryConfig": {
"host": "http://schema-registry:8081",
"additionalConfigs": {
"schema.registry.url": "${SR_URL}",
"basic.auth.credentials.source": "${SR_BASIC_AUTH_CRED_SRC}",
"basic.auth.user.info": "${SR_BASIC_AUTH_USER_INFO}"
}
},
"action": "BLOCK",
"celCacheSize": 100
}
}
```
```yaml theme={null}
apiVersion: gateway/v2
kind: Interceptor
metadata:
name: mySchemaIdValidationInterceptor
scope:
vCluster: passthrough
spec:
pluginClass: io.conduktor.gateway.interceptor.safeguard.SchemaPayloadValidationPolicyPlugin
priority: 100
config:
topic: "topic_1.*"
schemaIdRequired: true
validateSchema: true
schemaRegistryConfig:
host: http://schema-registry:8081
additionalConfigs:
schema.registry.url: $${SR_URL}
basic.auth.credentials.source: $${SR_BASIC_AUTH_CRED_SRC}
basic.auth.user.info: $${SR_BASIC_AUTH_USER_INFO}
action: BLOCK
celCacheSize: 100
```
Apply with:
```bash theme={null}
conduktor apply -f schema-validation-secured-template.yaml
```
#### Schema payload validations
When configured to do so, the schema validation Interceptor supports validating the value in a Kafka record against a specific set custom constraints for AvroSchema records. This is similar to the validations provided by JsonSchema, such as:
For fields in an Avro schema, you can specify specific constraints on what is considered a correct value. These rules operate on the specific fields value only.
* **INT**, **LONG**, **FLOAT**, **DOUBLE**: `minimum`, `maximum`, `exclusiveMinimum`, `exclusiveMaximum`, `multipleOf`
* **STRING**: `minLength`, `maxLength`, `pattern`, `format`
* **ARRAY**: `maxItems`, `minItems`
Current supported String `format` values:
* `byte`, `date`, `time`, `date-time`, `duration`, `uri`, `uri-reference`, `uri-template`, `uri`, `email`, `hostname`, `ipv4`, `ipv6`, `regex`, `uuid`, `json-pointer`, `json-pointer-uri-fragment`, `relative-json-pointer`
This Interceptor also supports further validating elements from the whole payload against specific custom constraints - or Metadata Rules - using an expression based on the CEL [Common Expression Language](https://github.com/google/cel-spec) format. This provides a means to define more advanced rules dependent on *multiple* values in a record.
#### Metadata rule
| Key | Type | Description |
| :--------- | :----- | :-------------------------------------------------------------------------------------------------------------- |
| name | string | Rule name |
| expression | string | CEL expression for validation, must return `BOOLEAN` |
| message | string | Error message if payload not matches the `expression` with namespace `message.` represents for produced message |
#### JSON schema example
In JSON schema, constraints and rules are defined directly in the schema. Here's an example that includes various validations:
```json theme={null}
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"name": {
"type": "string",
"minLength": 3,
"maxLength": 50,
"expression": "size(name) >= 3"
},
"age": {
"type": "integer",
"minimum": 0,
"maximum": 120,
"expression": "age >= 0 && age <= 120"
},
"email": {
"type": "string",
"format": "email",
"expression": "email.contains('foo')"
},
"address": {
"type": "object",
"properties": {
"street": {
"type": "string",
"minLength": 5,
"maxLength": 10,
"expression": "size(street) >= 5 && size(street) <= 10"
},
"city": {
"type": "string",
"minLength": 2,
"maxLength": 50
}
},
"expression": "size(address.street) > 1 && address.street.contains('paris') || address.city == 'paris'"
},
"hobbies": {
"type": "array",
"items": {
"type": "string"
},
"minItems": 3,
"expression": "size(hobbies) >= 3"
}
},
"metadata": {
"rules": [
{
"name": "check hobbies size and name",
"expression": "size(message.hobbies) == 3 && size(message.name) > 3",
"message": "hobbies must have 3 items"
},
{
"name": "checkAge",
"expression": "message.age >= 18",
"message": "age must be greater than or equal to 18"
},
{
"name": "check email",
"expression": "message.email.endsWith('yahoo.com')",
"message": "email should end with 'yahoo.com'"
},
{
"name": "check street",
"expression": "size(message.address.street) >= 3",
"message": "address.street length must be greater than equal to 3"
}
]
}
}
```
#### Avro schema example
In Avro, constraints and rules are defined directly in the schema. Here's an example that includes various validations:
```json theme={null}
{
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string", "minLength": 3, "maxLength": 50, "expression": "size(name) >= 3 && size(name) <= 50"},
{"name": "age", "type": "int", "minimum": 0, "maximum": 120, "expression": "age >= 0 && age <= 120"},
{"name": "email", "type": "string", "format": "email", "expression": "email.contains('foo')"},
{
"name": "address",
"type": {
"type": "record",
"name": "AddressRecord",
"fields": [
{"name": "street", "type": "string", "minLength": 5, "maxLength": 100, "expression": "size(street) >= 5 && size(street) <= 10"},
{"name": "city", "type": "string", "minLength": 2, "maxLength": 50}
]
},
"expression": "size(address.street) >= 5 && address.street.contains('paris') || address.city == 'paris'"
},
{"name": "hobbies", "type": {"type": "array", "items": "string"}, "minItems": 3, "expression": "size(hobbies) >= 3"},
{
"name": "friends",
"type": {
"type": "array",
"items": {
"type": "record",
"name": "Friend",
"fields": [
{"name": "name", "type": "string", "expression": "size(name) < 3"},
{"name": "age", "type": "int", "minimum": 2, "maximum": 10}
]
}
}
}
],
"metadata": {
"rules": [
{
"name": "check hobbies size and name",
"expression": "size(message.hobbies) == 3 && size(message.name) > 3",
"message": "hobbies must have 3 items"
},
{
"name": "checkAge",
"expression": "message.age >= 18",
"message": "age must be greater than or equal to 18"
},
{
"name": "check email",
"expression": "message.email.endsWith('yahoo.com')",
"message": "email should end with 'yahoo.com'"
},
{
"name": "check street",
"expression": "size(message.address.street) >= 3",
"message": "address.street length must be greater than equal to 3"
}
]
}
}
```
## Related resources
* [Find out more about Data Quality Policies](/guide/conduktor-concepts/data-quality-policies)
* [Enforce data quality](/guide/use-cases/enforce-data-quality)
* [Find out more about Interceptors](/guide/conduktor-concepts/interceptors)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Kafka RBAC — role-based access control
Source: https://docs.conduktor.io/guide/conduktor-in-production/admin/set-up-rbac
Set up Kafka RBAC in Conduktor Console to control access to topics, consumer groups, and resources.
The Conduktor RBAC (Role Based Access Control) system enables you to restrict access to resources and enforce permissions at **User** and **Group** granularity. This is a critical step in ensuring that you have control over your Apache Kafka data.
With Conduktor RBAC, you can:
* configure access to Conduktor services
* configure global permissions across **multiple clusters**
* administer permissions for Kafka resources (topics, consumer groups, clusters, subjects, connectors)
## Assign permissions
You can assign two types of permissions:
* [services](#manage-services-permissions): view/manage Console services
* [resources](#manage-resources-permissions): interact with Kafka resources
And you can assign those permissions to **users** or **groups**.
To assign user/group permissions, open Console and go to **Settings** > **Users** or **Groups**, as required. Click **...** next to the user/group you want to modify. Here's an example for a user:
If a user belongs to multiple groups, they will **inherit all the permissions** assigned to these groups. If they have restricted access to a topic but belong to a group that has full access, they will have full access to the topic.
## Manage services permissions
You can restrict access to Conduktor Console services such as settings or left menu items (like certificates). For example, you may want to limit the number of users who can generate API keys. By default, you all users can:
* access data masking policies
* view Self-service
## Manage resources permissions
The RBAC model is very granular and allows you to customize the permissions to Kafka resources based on your requirements:
| Resource | Permissions |
| ---------------- | ---------------------------------------- |
| Topics | View config |
| Topics | Consume |
| Topics | Produce |
| Topics | Create |
| Topics | Delete |
| Topics | Empty |
| Topics | Add partitions |
| Topics | Manage metadata (labels and description) |
| Consumer groups | Viewer |
| Consumer groups | Reset |
| Consumer groups | Create |
| Consumer groups | Delete |
| Consumer groups | Manage metadata (labels) |
| Subjects | View |
| Subjects | Edit compatibility |
| Subjects | Create / Update |
| Subjects | Delete |
| Subjects | Manage metadata (labels) |
| Kafka connectors | View task and status |
| Kafka connectors | View config |
| Kafka connectors | Edit config |
| Kafka connectors | Deploy |
| Kafka connectors | Delete |
| Kafka connectors | Restart |
| Kafka connectors | Pause / Resume |
| Kafka connectors | Manage metadata (labels and description) |
| Clusters | View ACL |
| Clusters | Manage ACL |
| Clusters | View broker |
| Clusters | Edit broker |
| Clusters | Edit SR compatibility |
All these permissions can be applied on one specific cluster, or all your clusters.
## Prefixes
When you define a permission, you might want it to be applied to:
* a specific topic, by typing `my-topic` for instance
* all the topics, by using a wildcard `*`
* a subset that starts with a certain prefix, by typing `my-prefix-*`
Here's an example of those three cases in Console:
## Govern who can edit labels and descriptions
By default, anyone who can view a Kafka resource can also edit its [labels](/guide/manage-kafka/kafka-resources/topics#manage-topic-labels) and, for topics, its description. Because labels and descriptions can carry governance metadata, you may want to restrict who can change them.
Metadata governance is an opt-in behavior that gates label and description editing on a dedicated permission, separate from the permission to view or edit the resource itself. It's **off by default**, so existing deployments keep their current behavior until you turn it on.
### Permissions and behavior
Metadata governance adds one **Manage metadata** permission per resource type. The permission covers labels for all four resource types, and also covers the description for topics and connectors (the two resources with an editable description).
| Resource | Manage metadata permission | Permission enforced when governance is **off** | Permission enforced when governance is **on** |
| -------------- | ------------------------------------------------------- | ---------------------------------------------- | --------------------------------------------- |
| Topic | `topicMetadataManage` (labels and description) | View config | `topicMetadataManage` |
| Subject | `subjectMetadataManage` (labels) | Create / Update | `subjectMetadataManage` |
| Connector | `kafkaConnectorMetadataManage` (labels and description) | Edit config | `kafkaConnectorMetadataManage` |
| Consumer group | `consumerGroupMetadataManage` (labels) | Create | `consumerGroupMetadataManage` |
This only changes the dedicated label and description endpoints. Setting labels or a description as part of a full resource create or update still uses the resource's existing edit permission.
When governance is on and a user lacks the **Manage metadata** permission, the Console UI hides the label and description edit controls for that resource. Existing labels and descriptions stay visible as read-only.
### Enable metadata governance
Set the `enable_metadata_governance` property (environment variable `CDK_ENABLE_METADATA_GOVERNANCE`) to `true`. See the [Console properties reference](/guide/conduktor-in-production/deploy-artifacts/deploy-console/environment-variables#global-properties).
```yaml theme={null}
enable_metadata_governance: true
```
Turning on metadata governance changes who can edit labels and descriptions. Grant the **Manage metadata** permissions before you enable it, so the right users keep their access. See the migration guidance below.
### Grant the Manage metadata permissions
You can grant each **Manage metadata** permission like any other resource permission — directly to a user, to a group, or through the API, CLI, or Terraform. For example, in a `Group` resource:
```yaml theme={null}
apiVersion: v2
kind: Group
metadata:
name: topic-governance
spec:
displayName: Topic governance
permissions:
- resourceType: TOPIC
cluster: "*"
patternType: PREFIXED
name: ""
permissions:
- topicMetadataManage
```
Application owner groups receive the matching permission for every resource type automatically. New applications get it when their owner group is created, and Conduktor backfills existing owner groups when you upgrade — so application owners keep their label and description edit access when you enable governance.
### Migration guidance
When you enable metadata governance, who can edit labels and descriptions changes:
* **Keeps access:** application owners (through their owner group), and any user or group that holds the resource's **Manage metadata** permission. Users with a full-access permission set on the resource type already hold it.
* **Loses access:** users who could edit labels or descriptions only because they could view or partially edit the resource, without the **Manage metadata** permission.
To avoid disruption, grant the **Manage metadata** permissions to the relevant users and groups first, then enable governance.
## Quick select
To save time during the permissions creation, you can use the `Quick select` to give a default set of permissions or set this up using the CLI, API or Terraform.
## User permissions example
Here's an example of a set of permissions given to Alice:
We can see that this is a recap of all the permissions this user has. In grey, we have the permissions Alice inherits from the group `Project A`, from the application `support-for-tracker` and in white the ones that are assigned to her directly.
This set of permissions gives her:
* Full access to the topic `alice-private-topic` on the cluster `test`
* Full access on all topics, that start with the prefix `app-a-`, across all clusters and that she inherits this from the group `Project A`
* Partial access to the topic `tracker-click-1` and `tracker-click-2` on the cluster `Cluster-A` and that she inherits this from the application `support-for-tracker`
## Related resources
* [Map external groups](/guide/conduktor-in-production/admin/user-access/map-external-groups)
* [Configure SSO](/guide/conduktor-in-production/admin/user-access/configure-sso)
* [Create data masking policies](/guide/conduktor-in-production/admin/data-masking)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Authentication options
Source: https://docs.conduktor.io/guide/conduktor-in-production/admin/user-access
Configure authentication for Conduktor Console: local admin users, LDAP, OIDC, or delegated JWT. Choose the method that fits your identity provider.
We recommend configuring SSO for production deployments.
## Authentication options
Conduktor supports various OIDC (OpenID Connect) providers, including:
* [Entra ID](/guide/conduktor-in-production/admin/user-access/configure-sso#configure-entra-id-as-sso)
* [Google](/guide/conduktor-in-production/admin/user-access/configure-sso#configure-google-as-sso)
* [Amazon Cognito](/guide/conduktor-in-production/admin/user-access/configure-sso#configure-amazon-cognito-as-sso)
* [Keycloak](/guide/conduktor-in-production/admin/user-access/configure-sso#configure-keycloak-as-sso)
* [Okta](/guide/conduktor-in-production/admin/user-access/configure-sso#configure-okta-as-sso)
* [Auth0](/guide/conduktor-in-production/admin/user-access/configure-sso#configure-auth0-as-sso)
* [WorkOS](/guide/conduktor-in-production/admin/user-access/configure-sso#configure-workos-as-sso)
* [Any other OIDC/OAuth2 provider](/guide/conduktor-in-production/admin/user-access/configure-sso#configure-an-oidcoauth2-provider-as-sso)
As well as:
* [LDAP](/guide/conduktor-in-production/admin/user-access/configure-sso#configure-ldap-as-sso)
* [JumpCloud](/guide/conduktor-in-production/admin/user-access/configure-sso#configure-jumpcloud-as-sso)
# SSO configuration in Conduktor Console — LDAP and OIDC
Source: https://docs.conduktor.io/guide/conduktor-in-production/admin/user-access/configure-sso
Set up SSO in Conduktor Console with LDAP or OIDC. Configuration examples for Okta, Azure AD, and external group mapping from your identity provider.
We offer several options when configuring SSO (Single Sign-On).
## Configure LDAP as SSO
[View the list of LDAP properties](#ldap-config-properties). Here's a sample snippet containing server, users and groups to put in your configuration file.
```yaml title="platform-config.yaml" theme={null}
sso:
ldap:
- name: "LDAP" # Custom name for LDAP connection
server: "ldap://openldap:1389" # LDAP server URI with port
managerDn: "cn=admin,dc=example,dc=org" # Bind DN
managerPassword: "adminpassword" # Bind password
search-subtree: true # Search subtree (default: true)
search-base: "ou=users,dc=example,dc=org" # Base DN to search for users
search-filter: "(uid={0})" # Search filter (default: "(uid={0})")
groups-enabled: true # Enable group membership (default: false)
groups-base: "ou=groups,dc=example,dc=org" # Base DN to search for groups
groups-filter: "(member={0})" # Filter on groups (default: "uniquemember={0}")
groups-attribute: "cn" # Group name entry (default: "cn")
```
```json title=".env" theme={null}
CDK_SSO_LDAP_0_NAME="LDAP"
CDK_SSO_LDAP_0_SERVER="ldap://openldap:1389"
CDK_SSO_LDAP_0_MANAGERDN="cn=admin,dc=example,dc=org"
CDK_SSO_LDAP_0_MANAGERPASSWORD="adminpassword"
CDK_SSO_LDAP_0_SEARCH-SUBTREE=true
CDK_SSO_LDAP_0_SEARCH-BASE="ou=users,dc=example,dc=org"
CDK_SSO_LDAP_0_SEARCH-FILTER="(uid={0})"
CDK_SSO_LDAP_0_GROUPS-ENABLED=true
CDK_SSO_LDAP_0_GROUPS-BASE="ou=groups,dc=example,dc=org"
CDK_SSO_LDAP_0_GROUPS-FILTER="(member={0})"
CDK_SSO_LDAP_0_GROUPS-ATTRIBUTE="cn"
```
If your LDAP server is **Active Directory** and you get an "invalid user" error when trying to log in, try setting your `search-filter` to `'(sAMAccountName={0})'`.
### User mapping
Here is the mapping between LDAP user's information and Conduktor Console:
| LDAP | Conduktor Console |
| -------------------------------------- | ----------------------------------------- |
| `uid` | User ID, used to log in |
| `mail`, `email` or `userPrincipalName` | User email (**The only mandatory field**) |
| `cn` | User name |
| `sn` | User family name |
| `givenName` | User first name |
| `displayName` | User display name |
#### Groups
To retrieve the groups each user belongs to, you have to set `groups-enabled` to `true`, and populate the attributes `groups-base` and `groups-filter`.
Note that depending on your LDAP `objectClass`, the attribute used to filter groups might be changed. For example:
| LDAP `objectClass` | Conduktor `groups-filter` |
| -------------------- | ------------------------- |
| `groupOfNames` | `"member={0}"` |
| `groupOfUniqueNames` | `"uniqueMember={0}"` |
##### Map to external groups
Now that your configuration is finished, you can set up a **mapping** between your LDAP groups and your Console groups. That way, when a user logs in, they will be automatically added to the corresponding Console groups, based on their LDAP groups.
To create this mapping, you have to create a group from Console, and mention the ID of the group on your LDAP (you should find it in the attribute you mentioned as `groups-base`).
After the user logged in, we can see they've been added to the group, without any action:
#### Configure LDAPS certificate
For LDAPs (LDAP over SSL) connection, you have to provide a trusted certificate using Java JKS TrustStore file.
LDAPS SSL certificate can also be passed as PEM encoded string using the property `sso.trustedCertificates`.
```yaml title="platform-config.yaml" theme={null}
sso:
ignoreUntrustedCertificate: false
trustedCertificates: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
```
## Configure Auth0 as SSO
[View the list of Auth0 properties](#oauth2-config-properties).
On Auth0 side, you'll have to create a new application:
1. Create a regular web application:
2. Get the `client ID`, `client secret` and `domain`:
3. Configure the callback URI. The redirect URI can be something like: `http(s)://(:)/oauth/callback/`. For example, if you deployed Console locally using the name `auth0` in your configuration file, you can use `http://localhost:8080/oauth/callback/auth0`.
Remember to specify how you want to connect using the **Connections** tab of your Auth0 application.
### Configure Console
On the Console side, you can add the snippet below to your configuration file. Replace the `client ID`, `client secret` and `domain` with values from step 2 above.
```yaml title="platform-config.yaml" theme={null}
sso:
oauth2:
- name: "auth0"
client-id: ""
client-secret: ""
openid:
issuer: "https://"
```
```json title=".env" theme={null}
CDK_SSO_OAUTH2_0_NAME="auth0"
CDK_SSO_OAUTH2_0_DEFAULT=true
CDK_SSO_OAUTH2_0_CLIENT-ID=""
CDK_SSO_OAUTH2_0_CLIENT-SECRET=""
CDK_SSO_OAUTH2_0_OPENID_ISSUER="https://"
```
## Configure Amazon Cognito as SSO
On Amazon Cognito side, you'll have to create a user pool with an application:
1. Create a new user pool
2. Configure the application client. You can select the name you want, shown here as `Conduktor Console` and enter the redirect URI as the following: `http(s)://(:)/oauth/callback/`. For example, if you deployed Console locally using the name `cognito` in your configuration file, you can use `http://localhost:8080/oauth/callback/cognito`.
Make sure that a client secret will be generated - select `Confidential client`.
3. Set the scopes `profile`, `email` and `openid` in the **Advanced app settings**:
4. Get the `user pool ID`, `client ID`, and `client secret`, that you'll use in the configuration file of Console
You can find the .well-known at: `https://cognito-idp..amazonaws.com//.well-known/openid-configuration`.
### Configure Console
On Console side, you can add the snippet below to your configuration file. You have to replace the `client ID`, `client secret`, `region`, and `pool ID`, with what you got during step 4.
```yaml title="platform-config.yaml" theme={null}
sso:
oauth2:
- name: "cognito"
client-id: ""
client-secret: ""
openid:
issuer: "https://cognito-idp..amazonaws.com/"
```
```json title=".env" theme={null}
CDK_SSO_OAUTH2_0_NAME="cognito"
CDK_SSO_OAUTH2_0_DEFAULT=true
CDK_SSO_OAUTH2_0_CLIENT-ID=""
CDK_SSO_OAUTH2_0_CLIENT-SECRET=""
CDK_SSO_OAUTH2_0_OPENID_ISSUER="https://cognito-idp..amazonaws.com/"
```
#### Configure groups
If you want to use the `external groups mapping` to map groups between your Conduktor Console instance and Amazon Cognito, you must set the property `groups-claim` to `"cognito:groups"` in Console configuration file. Below is the full snippet for your configuration file:
```yaml title="platform-config.yaml" theme={null}
sso:
oauth2:
- name: "cognito"
client-id: ""
client-secret: ""
groups-claim: "cognito:groups"
openid:
issuer: "https://cognito-idp..amazonaws.com/"
```
```json title=".env" theme={null}
CDK_SSO_OAUTH2_0_NAME="cognito"
CDK_SSO_OAUTH2_0_DEFAULT=true
CDK_SSO_OAUTH2_0_CLIENT-ID=""
CDK_SSO_OAUTH2_0_CLIENT-SECRET=""
CDK_SSO_OAUTH2_0_GROUPS-CLAIM="cognito:groups"
CDK_SSO_OAUTH2_0_OPENID_ISSUER="https://cognito-idp..amazonaws.com/"
```
##### Map to external groups
Now that your configuration is finished, you can set up the mapping between Amazon Cognito and Console groups. That way, when a user logs in, they will be automatically added to the corresponding Console groups, based on the groups they belong to in Amazon Cognito.
The value you need to put as an external group is the `Object ID` of the Amazon Cognito group.
## Configure Entra ID as SSO
On the Entra ID (formerly Azure Active Directory) side, you'll have to create a new application:
* **Step 1**: Create a new application in `App registrations` and set the callback URI
You can select the name you want, shown here as `Conduktor Console`, and enter the redirect URI as the following: `http(s)://(:)/oauth/callback/`.
For example, if you deployed Console locally using the name `azure` in your configuration file, you can use `http://localhost:8080/oauth/callback/azure`, like on the screenshot below.
* **Step 2**: Create a new client secret from the **Certificates and secrets** tab
You need to keep the `Value` somewhere safe, as you will not have access to it again.
* **Step 3**: Find the `client ID` and `tenant ID` in the **Overview** tab
You can find the .well-known at: `https://login.microsoftonline.com//v2.0/.well-known/openid-configuration`.
### Configure Console
On Console side, you can add the snippet below to your configuration file. You have to replace the client ID, client secret, and tenant ID, with what you got during steps 2 and 3.
```yaml title="platform-config.yaml" theme={null}
sso:
oauth2:
- name: "azure"
client-id: ""
client-secret: ""
openid:
issuer: "https://login.microsoftonline.com//v2.0"
```
```json title=".env" theme={null}
CDK_SSO_OAUTH2_0_NAME="azure"
CDK_SSO_OAUTH2_0_DEFAULT=true
CDK_SSO_OAUTH2_0_CLIENT-ID=""
CDK_SSO_OAUTH2_0_CLIENT-SECRET=""
CDK_SSO_OAUTH2_0_OPENID_ISSUER="https://login.microsoftonline.com//v2.0"
```
#### Configure groups
If you want to use the `external groups mapping` to map groups between your Conduktor Console instance and Azure, you must add this claim to your Azure application in the **Token configuration** tab:
The external groups mapping will not work for users who belong to [more than 200 groups](https://learn.microsoft.com/en-us/entra/identity/hybrid/connect/how-to-connect-fed-group-claims) . In this case, you will need to [assign some groups to the application](https://learn.microsoft.com/en-us/azure/active-directory/manage-apps/assign-user-or-group-access-portal?pivots=portal#assign-users-and-groups-to-an-application) and check the **Groups assigned to the application** box when creating the groups claim on Azure AD. This is to avoid exceeding the limit on the number of groups a token can contain.
Then, you must set the property `groups-claim` to `"groups"` in the Console configuration file. Below is the full snippet for your configuration file:
```yaml title="platform-config.yaml" theme={null}
sso:
oauth2:
- name: "azure"
client-id: ""
client-secret: ""
groups-claim: "groups"
openid:
issuer: "https://login.microsoftonline.com//v2.0"
```
```json title=".env" theme={null}
CDK_SSO_OAUTH2_0_NAME="azure"
CDK_SSO_OAUTH2_0_DEFAULT=true
CDK_SSO_OAUTH2_0_CLIENT-ID=""
CDK_SSO_OAUTH2_0_CLIENT-SECRET=""
CDK_SSO_OAUTH2_0_GROUPS-CLAIM="groups"
CDK_SSO_OAUTH2_0_OPENID_ISSUER="https://login.microsoftonline.com//v2.0"
```
##### Map to external groups
Now that your configuration is finished, you can set up the mapping between Azure AD and Console groups. That way, when a user will log in, they will be automatically added to the corresponding Console groups, based on the groups they belong to in Azure AD.
The value you need to put as an external group is the `Object ID` of the Azure AD group.
## Configure Google as SSO
On Google side:
* **Step 1**: Create an application on the **OAuth consent screen** tab
The scopes needed are `email`, `profile`, and `openid`. Optionally, you need `read only` for external group mapping.
* **Step 2**: Restrict access to your internal workspace by checking the `Internal` user type in the **OAuth consent screen**.
* **Step 3**: Create a new `OAuth client ID`
You can select the name you want, shown here as `Conduktor Console`, and enter the redirect URI like this: `http(s)://(:)/oauth/callback/`.
For example, if you deployed Console locally using the name `google` in your configuration file, you can use `http://localhost:8080/oauth/callback/google`, like on the screenshot below.
* **Step 4**: Get the `client ID` and the `secret ID`
After the creation, the pop-up below appears. You can save the client ID and secret as JSON if you want.
You can find the .well-known at: [`https://accounts.google.com/.well-known/openid-configuration`](https://accounts.google.com/.well-known/openid-configuration) .
If you need to add an **authorized domain** to your Google account, you can follow [this guide](https://support.google.com/cloud/answer/6158849?hl=en-GB#authorized-domains\&zippy=%2Cauthorized-domains) .
### Configure Console
On Console side, you can add the snippet below to your configuration file. You have to replace the client ID and secret with what you got during the step 4.
```yaml title="platform-config.yaml" theme={null}
sso:
oauth2:
- name: "google"
client-id: ""
client-secret: ""
scopes: "openid,email,profile"
openid:
issuer: "https://accounts.google.com"
```
```json title=".env" theme={null}
CDK_SSO_OAUTH2_0_NAME="google"
CDK_SSO_OAUTH2_0_CLIENT-ID=""
CDK_SSO_OAUTH2_0_CLIENT-SECRET=""
CDK_SSO_OAUTH2_0_SCOPES="openid,email,profile"
CDK_SSO_OAUTH2_0_OPENID_ISSUER="https://accounts.google.com"
```
#### Configure groups
An additional scope `https://www.googleapis.com/auth/cloud-identity.groups.readonly` is required if you want to sync Google Group with Conduktor Groups.
```yaml title="platform-config.yaml" theme={null}
sso:
oauth2:
- name: "google"
client-id: ""
client-secret: ""
scopes: "openid,email,profile,https://www.googleapis.com/auth/cloud-identity.groups.readonly"
openid:
issuer: "https://accounts.google.com"
```
```json title=".env" theme={null}
CDK_SSO_OAUTH2_0_NAME="google"
CDK_SSO_OAUTH2_0_CLIENT-ID=""
CDK_SSO_OAUTH2_0_CLIENT-SECRET=""
CDK_SSO_OAUTH2_0_SCOPES="openid,email,profile,https://www.googleapis.com/auth/cloud-identity.groups.readonly"
CDK_SSO_OAUTH2_0_OPENID_ISSUER="https://accounts.google.com"
```
##### Map to external groups
Now that your configuration is finished, you can set up the mapping between Google groups and Console groups.
That way, when a user logs in, they will be automatically added to the corresponding Console groups, based on the groups they belong to in Google.
The value you need to put as an external group is the `email` address of the Google Group.
## Configure JumpCloud as SSO
On the JumpCloud side, you'll have to create a new application:
* **Step 1**: Create a new application in `SSO Applications`.
* **Step 2**: Select a `Custom Application` as shown below.
Then ensure to select `Manage Single Sign-On (SSO)`, then `Configure SSO with OIDC` and `Export users to this app (Identity Management)` as seen in the screenshot below.
Following this, enter general information for your custom application, including the display label, such as `conduktor` as seen in the screenshot below and configure this application.
* **Step 3**: Configure OAuth settings including `Redirect URI(s)`, `Login URL` and the client authentication type.
The Redirect URI is where JumpCloud sends the authentication response and ID token for the user's sign-in request to. The Login URL is the URL users need to log into this application.
Enter the redirect URI in the following way:
`http(s)://:/oauth/callback/`.
For example, if you deployed Console locally using the name `jumpcloud` in your configuration file, you can use
`https://localhost:8080/oauth/callback/jumpcloud`, as seen in the screenshot below.
Enter the Login URL, which is the URL users need to log into this application. In the example below, this is `https://localhost:8080` .
For the **Client Authentication Type** make sure to select the **Client Secret Basic** option.
* **Step 4**: Find the `Client ID` and `Client Secret`.
After clicking activate during Step 3 you will be shown configurations for `Client ID` and `Client Secret`, be sure to save these somewhere safe.
You need to keep the `Client Secret` somewhere safe, as you will not have access to it again.
### Configure Console
On the Conduktor Console side, you can add the snippet below to your configuration file. You will have to replace the Client ID and Client Secret, as shown in steps 3 and 4.
You can find the opendid issuer at: `https://oauth.id.jumpcloud.com/` as shown [`here`](https://jumpcloud.com/support/sso-with-oidc)
```yaml title="platform-config.yaml" theme={null}
sso:
oauth2:
- name: "jumpcloud"
client-id: ""
client-secret: ""
groups-claim: "groups" #if wanting to use groups mapping
openid:
issuer: "https://oauth.id.jumpcloud.com/"
```
Or using environment variables:
```json theme={null}
CDK_SSO_OAUTH2_0_NAME="jumpcloud"
CDK_SSO_OAUTH2_0_DEFAULT=true
CDK_SSO_OAUTH2_0_CLIENT-ID=""
CDK_SSO_OAUTH2_0_CLIENT-SECRET=""
CDK_SSO_OAUTH2_0_GROUPS-CLAIM="groups"
CDK_SSO_OAUTH2_0_OPENID_ISSUER="https://oauth.id.jumpcloud.com/"
```
#### Configure groups
If you want to use the `external groups mapping` to map groups between your Conduktor Console instance and JumpCloud:
From the JumpCloud side, ensure you have:
* Checked `Email` and `Profile` under the standard scopes
* Set the `email_verified` to true
* The same value in `group attribute` as in the `groups-claim` or `CDK_SSO_OAUTH2_0_GROUPS-CLAIM` value of your Console's configuration
See the example screenshot shown below.
From the Conduktor Console side, you must set the property `groups-claim` to `"groups"` in the Console configuration file.
Below is the full snippet for your configuration file:
```yaml title="platform-config.yaml" theme={null}
sso:
oauth2:
- name: "jumpcloud"
client-id: ""
client-secret: ""
groups-claim: "groups"
openid:
issuer: "https://oauth.id.jumpcloud.com/"
```
Or using environment variables:
```json theme={null}
CDK_SSO_OAUTH2_0_NAME="jumpcloud"
CDK_SSO_OAUTH2_0_DEFAULT=true
CDK_SSO_OAUTH2_0_CLIENT-ID=""
CDK_SSO_OAUTH2_0_CLIENT-SECRET=""
CDK_SSO_OAUTH2_0_GROUPS-CLAIM="groups"
CDK_SSO_OAUTH2_0_OPENID_ISSUER="https://oauth.id.jumpcloud.com/"
```
##### Map to external groups
Now that your configuration is finished, you can [set up the mapping](https://docs.conduktor.io/platform/get-started/configuration/user-authentication/external-group-sync/#create-an-external-group-mapping) between JumpCloud and Conduktor Console groups. This way, when a user logs in, they will be automatically added to the corresponding Conduktor Console groups, based on the groups they belong to in JumpCloud.
The value you need to put as an external group is the name of the JumpCloud group.
## Configure Keycloak as SSO
On Keycloak side, you'll have to create a new application:
* **Step 1**: create a new OpenID Connect client, and set the `client ID`
* **Step 2**: Select the **Client authentication**
* **Step 3**: Configure the redirect URI
You can configure it as the following: `http(s)://(:)/oauth/callback/`
For example, if you deployed Console locally using the name `keycloak` in your configuration file, you can use `http://localhost:8080/oauth/callback/keycloak`, like in the screenshot below.
* **Step 4**: Get the `client secret` in the **Credentials** tab
You can find the .well-known at: `http://:/realms//.well-known/openid-configuration`.
### Configure Console
On Console side, you can add the snippet below to your configuration file. You have to replace the client ID, client secret, and tenant ID, with what you got during the previous steps.
```yaml title="platform-config.yaml" theme={null}
sso:
oauth2:
- name: "keycloak"
client-id: ""
client-secret: ""
openid:
issuer: "http://:/realms/"
```
```json title=".env" theme={null}
CDK_SSO_OAUTH2_0_NAME="keycloak"
CDK_SSO_OAUTH2_0_DEFAULT=true
CDK_SSO_OAUTH2_0_CLIENT-ID=""
CDK_SSO_OAUTH2_0_CLIENT-SECRET=""
CDK_SSO_OAUTH2_0_OPENID_ISSUER="http://:/realms/"
```
#### Configure groups
If you want to use the `external groups mapping` to map groups between your Conduktor Console instance and Keycloak, you must create a scope and add it to your Keycloak application:
* **Step 1**: Create the scope and configure the mapper to **Group Membership**
You can add the claim to the token you want. In this example, the **UserInfo**.
* **Step 2**: Add the scope to the application
Then, you must set the property `groups-claim` to `"groups"` in the Console configuration file. Below is the full snippet for your configuration file:
```yaml title="platform-config.yaml" theme={null}
sso:
oauth2:
- name: "keycloak"
client-id: ""
client-secret: ""
groups-claim: "groups"
openid:
issuer: "http://:/realms/"
```
```json title=".env" theme={null}
CDK_SSO_OAUTH2_0_NAME="keycloak"
CDK_SSO_OAUTH2_0_DEFAULT=true
CDK_SSO_OAUTH2_0_CLIENT-ID=""
CDK_SSO_OAUTH2_0_CLIENT-SECRET=""
CDK_SSO_OAUTH2_0_GROUPS-CLAIM="groups"
CDK_SSO_OAUTH2_0_OPENID_ISSUER="http://:/realms/"
```
##### Map to external groups
Now that your configuration is finished, you can setup the mapping between Keycloak and Console groups. That way, when a user logs in, they will be automatically added to the corresponding Console groups, based on the groups they belong to in Keycloak.
The value you need to put as an external group is the name of the Keycloak group.
If you've selected `Full group path` in the mapper details of the scope, you will need to use the full path instead of the name of the group.
## Configure Okta as SSO
On Okta side, create a new application:
* **Step 1**: Create an **OpenID Connect web application**
* **Step 2**: Configure the callback URI
The redirect URI can be like: `http(s)://(:)/oauth/callback/`.
For example, if you deployed Console locally using the name `okta` in your configuration file, you can use `http://localhost:8080/oauth/callback/okta`, like in the screenshot below.
* **Step 3**: Configure **app assignments**, and save changes
* **Step 4**: Get `client ID` and `client secret`, that you'll use in the configuration file of Console
* **Step 5**: Find the `issuer URL` in the **Sign On** tab of your application. It's made like `https://.okta.com`
You can find the .well-known at: `https://.okta.com/.well-known/openid-configuration`.
### Configure Console
On the Console side, you can add the snippet below to your configuration file. You have to replace the `client ID`, `client secret` and `domain`, with what you got during steps 4 and 5.
```yaml title="platform-config.yaml" theme={null}
sso:
oauth2:
- name: "okta"
client-id: ""
client-secret: ""
openid:
issuer: "https://.okta.com"
```
Please note that if you are using a custom auth server in Okta, the OPENID\_ISSUER should be in the form `https:///oauth2//` rather than `https://.okta.com`. [Find out more about token customization](https://developer.okta.com/docs/guide/customize-tokens-returned-from-okta/main/) .
```json title=".env" theme={null}
CDK_SSO_OAUTH2_0_NAME="okta"
CDK_SSO_OAUTH2_0_DEFAULT=true
CDK_SSO_OAUTH2_0_CLIENT-ID=""
CDK_SSO_OAUTH2_0_CLIENT-SECRET=""
CDK_SSO_OAUTH2_0_OPENID_ISSUER="https://.okta.com"
```
## Configure WorkOS as SSO
On the WorkOS side, you have to create an OAuth application:
* **Step 1**: In the WorkOS dashboard, go to **AuthKit → Connect** and create a new Connect OAuth application
WorkOS exposes two application types that both produce a `client_xxx` ID. Conduktor Console only works with a **Connect OAuth Application** (AuthKit → Connect panel). The **SSO Application** in the Applications panel is intended for server-side integrations using the WorkOS SDKs and isn't a standards-compliant OIDC client.
* **Step 2**: Add the redirect URI on the application's **Redirects** tab
The redirect URI is: `http(s)://(:)/oauth/callback/`. For example, if you deployed Console using the name `WorkOS` in your configuration file, the redirect URI is `https:///oauth/callback/WorkOS`.
The literal value of the OAuth2 config name is appended as a path segment, so the URI registered in WorkOS has to match it exactly, including casing. WorkOS doesn't accept wildcards or path prefixes.
* **Step 3**: Get the `client ID` and `client secret` from the Connect OAuth application
* **Step 4**: Find your AuthKit domain in **Authentication → Features → Hosted UI**
Each WorkOS environment has its own AuthKit domain, in the form `https://.authkit.app`. Use this as the OpenID issuer.
You can find the .well-known at: `https:///.well-known/openid-configuration`.
Don't use the `https://api.workos.com/user_management//.well-known/openid-configuration` endpoint. It returns an incomplete discovery document missing the `subject_types_supported` and `id_token_signing_alg_values_supported` fields, both required by [OIDC Discovery 1.0 §3](https://openid.net/specs/openid-connect-discovery-1_0.html#ProviderMetadata) . Console's OIDC client rejects it with `ParseException: Missing JSON object member with key subject_types_supported`.
### Configure Console
On the Console side, you can add the snippet below to your configuration file. You have to replace the `client ID`, `client secret`, and `authkit-domain` with what you got during the previous steps.
```yaml title="platform-config.yaml" theme={null}
sso:
oauth2:
- name: "WorkOS"
client-id: ""
client-secret: ""
scopes: "openid,email,profile"
openid:
issuer: "https://"
```
```json title=".env" theme={null}
CDK_SSO_OAUTH2_0_NAME="WorkOS"
CDK_SSO_OAUTH2_0_DEFAULT=true
CDK_SSO_OAUTH2_0_CLIENT-ID=""
CDK_SSO_OAUTH2_0_CLIENT-SECRET=""
CDK_SSO_OAUTH2_0_SCOPES="openid,email,profile"
CDK_SSO_OAUTH2_0_OPENID_ISSUER="https://"
```
## Configure an OIDC provider as SSO
Conduktor supports various OIDC (OpenID Connect)/OAuth2 providers, including:
* [Entra ID](#configure-entra-id-as-sso)
* [Google](#configure-google-as-sso)
* [Amazon Cognito](#configure-amazon-cognito-as-sso)
* [Keycloak](#configure-keycloak-as-sso)
* [Okta](#configure-okta-as-sso)
* [Auth0](#configure-auth0-as-sso)
* [WorkOS](#configure-workos-as-sso)
For others, follow these generic steps.
1. Create an OIDC (OpenID Connect) application in your chosen provider. This [application should use standard OAuth2/OIDC authorization code flow](https://openid.net/specs/openid-connect-core-1_0.html#CodeFlowAuth) with `CLIENT_SECRET_BASIC` authentication method.
2. Set OAuth2 authorized redirect URI in your application
For the OAuth2 authorization code flow to work, the OAuth2 provider needs to know; and authorize; where to redirect the user after the authentication process.
This is called the `redirect URI` or `callback URI`.
The redirect URI will look like this:\
`http(s)://(:)/oauth/callback/`
Where `` and `` depend on the Console external URL used and/or configured, and `` is the name of the OAuth2 configuration in your Console configuration file see [Console configuration](#console-configuration) step.
### More details on Console external URL
When Console initiate the OAuth2 authorization code flow, it tells the OIDC provider where to redirect the user after the authentication process.
But to forge this redirect URI, Console has several choices:
#### Console external URL is configured
If the Console external URL is configured using environment variable `CDK_PLATFORM_EXTERNAL_URL` or configuration `platform.external.url`, it will be used.
**However, the SSO will only work if Console is accessed using this URL.**
If you try to log in from the second URL, you'll be redirected to the first URL and then lose browser authentication cookies which means the SSO will not work.
##### Context path deployments
If you deploy Console with a context path using `CDK_PLATFORM_EXTERNAL_URL` with a context path (e.g., `https://mycompany.com/console`), your SSO callback URLs must include the base path.
For example, if Console is deployed at `https://mycompany.com/console` with an OAuth2 configuration named `okta`, the callback URL would be:
```
https://mycompany.com/console/oauth/callback/okta
```
Make sure to update your identity provider's callback/redirect URI configuration accordingly.
##### Console external URL is NOT configured
When no external Console URL is enforced, Console will use requests headers to resolve this external URL.
This is recommended if Console is accessed using multiple URLs (internal, external, etc) and have SSO on each of them.
The resolution strategy is the following:
1. Use the [`Forwarded`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded) header. This is the preferred method if you are using a **reverse proxy in front of Console**.
It uses the `host` and `proto` directives (if set) of `Forwarded` header to determine the external URL.
2. Use the `X-Forwarded-*` headers. Support for the non-standard forwarded headers coming from some **reverse proxy** implementations.
It uses the `X-Forwarded-Proto`, `X-Forwarded-Host` and `X-Forwarded-Port` headers to determine the external URL.
3. Use the `Host` header. Used if you access to Console directly, without a reverse proxy.
In this case, the `Host` header (generally set by the browser) will be used to determine the external URL.
**Port** will be guessed depending on the content of the `Host` header and fallback to Console configured port using environment variable `CDK_LISTENING_PORT` (default to `8080`).\
**Scheme** (http/https) will be guessed depending on the current TLS configuration of Console. (default to `http`).
4. Get the client ID and secret from application settings
#### Configure Console
On Console side, you need to configure several properties to enable OIDC SSO.
Required properties are:
* `sso.oauth2.name`: the name of the OAuth2 configuration. This name will be used in the redirect URI defined on your provider in the steps before. It must be unique.
* `sso.oauth2.client-id`: the client ID of your OAuth2 application.
* `sso.oauth2.client-secret`: the client secret of your OAuth2 application.
* `sso.oauth2.openid.issuer`: the issuer URL of your OpenID Connect provider. This url is used to discover the provider configuration using the `.well-known/openid-configuration` path.
Optionally, you can configure the following properties:
* `sso.oauth2.scopes`: the list of scopes to request during the authorization code flow. [See OAuth2 configuration for details](#configure-auth0-as-sso).
##### Example
The provider exposes its configuration using the well-known endpoint: `https:///.well-known/openid-configuration`. Here's an example of a configuration file for a generic OIDC provider.
```yaml title="platform-config.yaml" theme={null}
sso:
oauth2:
- name: "oidc-provider"
client-id: ""
client-secret: ""
openid:
issuer: "https:///"
```
```json title=".env" theme={null}
CDK_SSO_OAUTH2_0_NAME="oidc-provider"
CDK_SSO_OAUTH2_0_DEFAULT=true
CDK_SSO_OAUTH2_0_CLIENT-ID=""
CDK_SSO_OAUTH2_0_CLIENT-SECRET=""
CDK_SSO_OAUTH2_0_OPENID_ISSUER="https:///"
```
## SSO config properties
| Property | Description | Environment variable | Mandatory | Type | Default |
| -------------------------------- | ------------------------------------------------------------------------ | ------------------------------------ | --------- | ------- | ------- |
| `sso.ignoreUntrustedCertificate` | Disable SSL checks | `CDK_SSO_IGNOREUNTRUSTEDCERTIFICATE` | false | boolean | `false` |
| `sso.trustedCertificates` | SSL public certificates for SSO authentication (LDAPS and OAuth2) as PEM | `CDK_SSO_TRUSTEDCERTIFICATES` | false | string | ∅ |
### OAuth2 config properties
| Property | Description | Environment variable | Mandatory | Type | Default |
| --------------------------------------- | -------------------------------------------------------------------- | ---------------------------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| `sso.oauth2[].name` | OAuth2 connection name | `CDK_SSO_OAUTH2_0_NAME` | true | string | ∅ |
| `sso.oauth2[].default` | Use as default | `CDK_SSO_OAUTH2_0_DEFAULT` | true | boolean | ∅ |
| `sso.oauth2[].client-id` | OAuth2 client ID | `CDK_SSO_OAUTH2_0_CLIENTID` | true | string | ∅ |
| `sso.oauth2[].client-secret` | OAuth2 client secret | `CDK_SSO_OAUTH2_0_CLIENTSECRET` | true | string | ∅ |
| `sso.oauth2[].openid.issuer` | Issuer to check on token | `CDK_SSO_OAUTH2_0_OPENID_ISSUER` | true | string | ∅ |
| `sso.oauth2[].scopes` | Scopes to be requested in the client credentials request | `CDK_SSO_OAUTH2_0_SCOPES` | true | string | `[]` |
| `sso.oauth2[].groups-claim` | Group attribute from your identity provider | `CDK_SSO_OAUTH2_0_GROUPSCLAIM` | false | string | ∅ |
| `sso.oauth2[].username-claim` | Username attribute from your identity provider | `CDK_SSO_OAUTH2_0_USERNAMECLAIM` | false | string | `email` |
| `sso.oauth2[].email-claim` | Email attribute from your identity provider | `CDK_SSO_OAUTH2_0_EMAILCLAIM` | false | string | `email` |
| `sso.oauth2[].allow-unsigned-id-tokens` | Allow unsigned ID tokens | `CDK_SSO_OAUTH2_0_ALLOWUNSIGNEDIDTOKENS` | false | boolean | false |
| `sso.oauth2[].preferred-jws-algorithm` | Configure preferred JWS algorithm | `CDK_SSO_OAUTH2_0_PREFERREDJWSALGORITHM` | false | string one of: "HS256", "HS384", "HS512", "RS256", "RS384", "RS512", "ES256", "ES256K", "ES384", "ES512", "PS256", "PS384", "PS512", "EdDSA" | ∅ |
| `sso.oauth2[].pkce-method` | Configure PKCE method | `CDK_SSO_OAUTH2_0_PKCEMETHOD` | false | string one of: "plain", "S256" | ∅ |
| `sso.oauth2-logout` | Whether the central identity provider logout should be called or not | `CDK_SSO_OAUTH2LOGOUT` | false | boolean | true |
#### Nested group claim
`groups-claim` supports dot-notation to read groups from a nested JWT claim. Console first checks for an exact top-level key match (preserving backward compatibility with claim keys that contain dots), then treats the dot as a path separator.
**Flat claim** — groups returned as a top-level JWT claim:
```json theme={null}
{
"groups": ["admins", "developers"]
}
```
```yaml theme={null}
groups-claim: "groups"
```
**Nested claim** — groups inside a JSON object in the token:
```json theme={null}
{
"urn:spc:grant": {
"roles": ["admins", "developers"]
}
}
```
```yaml theme={null}
groups-claim: "urn:spc:grant.roles"
```
### LDAP config properties
| Property | Description | Environment variable | Mandatory | Type | Default |
| ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | --------- | ------------ | -------------------- |
| `sso.ldap[].name` | Ldap connection name | `CDK_SSO_LDAP_0_NAME` | true | string | ∅ |
| `sso.ldap[].server` | Ldap server host and port | `CDK_SSO_LDAP_0_SERVER` | true | string | ∅ |
| `sso.ldap[].managerDn` | Sets the manager DN | `CDK_SSO_LDAP_0_MANAGERDN` | true | string | ∅ |
| `sso.ldap[].managerPassword` | Sets the manager password | `CDK_SSO_LDAP_0_MANAGERPASSWORD` | true | string | ∅ |
| `sso.ldap[].search-subtree` | Sets if the subtree should be searched. | `CDK_SSO_LDAP_0_SEARCHSUBTREE` | false | boolean | `true` |
| `sso.ldap[].search-base` | Sets the base DN to search. | `CDK_SSO_LDAP_0_SEARCHBASE` | true | string | ∅ |
| `sso.ldap[].search-filter` | Sets the search filter. By default, the filter is set to `(uid={0})` for users using class type `InetOrgPerson`. | `CDK_SSO_LDAP_0_SEARCHFILTER` | false | string | `"(uid={0})"` |
| `sso.ldap[].search-attributes` | Sets the attributes list to return. By default, all attributes are returned. Platform search for `uid`, `cn`, `mail`, `email`, `givenName`, `sn`, `displayName` attributes to map into user token. | `CDK_SSO_LDAP_0_SEARCHATTRIBUTES` | false | string array | `[]` |
| `sso.ldap[].groups-enabled` | Sets if group search is enabled. | `CDK_SSO_LDAP_0_GROUPSENABLED` | false | boolean | `false` |
| `sso.ldap[].groups-subtree` | Sets if the subtree should be searched. | `CDK_SSO_LDAP_0_GROUPSSUBTREE` | false | boolean | `true` |
| `sso.ldap[].groups-base` | Sets the base DN to search from. | `CDK_SSO_LDAP_0_GROUPSBASE` | true | string | ∅ |
| `sso.ldap[].groups-filter` | Sets the group search filter. If using group class type `GroupOfUniqueNames` use the filter `"uniqueMember={0}"`. For group class `GroupOfNames` use `"member={0}"`. | `CDK_SSO_LDAP_0_GROUPSFILTER` | false | string | `"uniquemember={0}"` |
| `sso.ldap[].groups-filter-attribute` | Sets the name of the user attribute to bind to the group search filter. Defaults to the user’s DN. | `CDK_SSO_LDAP_0_GROUPSFILTERATTRIBUTE` | false | string | ∅ |
| `sso.ldap[].groups-attribute` | Sets the group attribute name. Defaults to `cn`. | `CDK_SSO_LDAP_0_GROUPSATTRIBUTE` | false | string | `"cn"` |
| `sso.ldap[].properties` | Additional properties that will be passed to identity provider context. | `CDK_SSO_LDAP_0_PROPERTIES` | false | dictionary | ∅ |
## Troubleshoot
Yes. We support any OIDC, Active Directory or LDAP standard identity provider. Moreover, this doesn't require a connection back to Conduktor - it integrates directly with your identify provider.
## Related resources
* [Map external groups](/guide/conduktor-in-production/admin/user-access/map-external-groups)
* [Set up RBAC](/guide/conduktor-in-production/admin/set-up-rbac)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Delegated JWT authentication
Source: https://docs.conduktor.io/guide/conduktor-in-production/admin/user-access/delegated-jwt-authentication
Configure delegated JWT authentication in Conduktor Console to validate tokens from an external identity provider and grant access via single sign-on.
## Overview
Console allows you to delegate authentication to an external Identity Provider (IP) using JSON web token (JWT).
Users authenticate through their organization's IP in their browser and get access Console via single sign-on (SSO).
Instead of managing authentication directly, Console validates tokens issued by a trusted IP.
You can protect Console by adding an API between Console and its clients.
The authentication flow of the client is performed by the API and the token from the identity provider is sent directly to Console.
In this case, no authentication is performed by Console, it only checks that the token is issued by a trusted identity provider and is valid.
When enabling the delegated authentication mode, tokens issued directly by Console are still accepted.
### How it works
In this authentication flow:
1. The user navigates to Console (e.g., `https://console.mydomain.com`).
2. The user is redirected by the API to their organization's IP for authentication.
3. After successful authentication, the IP issues a JWT token.
4. The API forwards the JWT token in all subsequent requests to Console.
5. Console validates the token against the configured trusted provider.
6. Console maps the token claims to a user account and applies appropriate permissions.
This approach allows organizations to:
* Automatically provision users on first login
* Use existing identity infrastructure (e.g., corporate IdP systems)
* Enforce centralized authentication policies
* Provide single sign-on experience across applications
## Prerequisites
To validate a token, Console will retrieve the issuer configuration and public keys. The issuer has to expose an OIDC discovery endpoint, such as `.well-know/openid-configuration`, to provide this information.
The token of the identity provider has to contain claims with either an API key or an email. These claims allow Console to map the token to a user or an API key and apply its permissions.
## Configuration example
In this example we configure Console to accept any token issued by `https://example.org/keycloak/realms/conduktor`. If a valid API key is defined in the `apikey` claim, it will be used. Otherwise, the email contained in the `email` claim will be mapped to a Console user. If the user doesn't exist, it will be created. The `groups` claim is optional and used for [external group mapping](/guide/conduktor-in-production/admin/user-access/map-external-groups).
```yaml title="platform-config.yaml" theme={null}
sso:
jwt-auth:
issuer: 'https://example.org/keycloak/realms/conduktor'
username-claim: 'email'
groups-claim: 'groups'
api-key-claim: 'apikey'
```
## Conduktor CLI
By default, the Conduktor CLI will try to log in to Console and generate a Conduktor token. With delegated authentication, we want to avoid the generation of a Conduktor token and directly use the one provided by the identity provider. To configure the CLI for this mode, set the `CDK_AUTH_MODE` environment variable to `external`.
If you already have a token from your identity provider, you can configure the CLI like this:
```shell theme={null}
# Bearer token (issued by the identity provider)
export CDK_AUTH_MODE="external"
export CDK_API_KEY=
```
If you use an API in front of Console, you can also send the client ID and secret as basic auth in the authentication header, like this:
```shell theme={null}
# Basic auth
export CDK_AUTH_MODE="external"
export CDK_USER=
export CDK_PASSWORD=
```
## Config properties
| Property | Description | Environment variable | Mandatory | Type | Default |
| ----------------------------- | --------------------------------------------- | ------------------------------- | --------- | ------ | -------- |
| `sso.jwt-auth.issuer` | Issuer of your identity provider | `CDK_SSO_JWTAUTH_ISSUER` | true | string | ∅ |
| `sso.jwt-auth.username-claim` | Email attribute from your identity provider | `CDK_SSO_JWTAUTH_USERNAMECLAIM` | false | string | `email` |
| `sso.jwt-auth.groups-claim` | Group attribute from your identity provider | `CDK_SSO_JWTAUTH_GROUPSCLAIM` | false | string | `groups` |
| `sso.jwt-auth.api-key-claim` | API key attribute from your identity provider | `CDK_SSO_JWTAUTH_APIKEYCLAIM` | false | string | `apikey` |
# Local admin and user management
Source: https://docs.conduktor.io/guide/conduktor-in-production/admin/user-access/local-admin-users
Create and manage local admin users in Conduktor Console. Add users via the UI, config file, or environment variables.
## Configure local admin and users
When you first start Conduktor Console, a **local admin** will be created using the credentials you provided. This admin account allows you to make some actions like creating users and groups, connecting clusters or giving permissions, in order to initialize your instance before onboarding users.
You can add new users via the Console UI, the configuration file or through environment variables.
### Add users in Console UI
In Console, go to **Settings** > **Users** page and click **Create members** at the top-right.
Select **SSO** or **Local**:
* SSO: if you already have SSO configured, see how to [manage user permissions before first login](#manage-user-permissions-before-first-login)
* Local: add users via basic email/password authentication
Enter user's email address and click **Add**.
You can later to add users from an IAC approach using the [ConsoleGroup component](/guide/reference/console-reference#consolegroup).
#### Configuration example
Here's an example of adding an administrator `admin@conduktor.io` with two local users `alice@conduktor.io` and `bob@conduktor.io`:
```yaml title="platform-config.yaml" theme={null}
admin:
email: admin@conduktor.io
password: admin-secret
auth:
local-users:
- email: alice@conduktor.io
password: alice-secret
- email: bob@conduktor.io
password: bob-secret
```
```json theme={null}
CDK_ADMIN_EMAIL="admin@conduktor.io"
CDK_ADMIN_PASSWORD="admin-secret"
CDK_AUTH_LOCAL-USERS_0_EMAIL="alice@conduktor.io"
CDK_AUTH_LOCAL-USERS_0_PASSWORD="alice-secret"
CDK_AUTH_LOCAL-USERS_1_EMAIL="bob@conduktor.io"
CDK_AUTH_LOCAL-USERS_1_PASSWORD="bob-secret"
```
### Configure SSO
In addition to these local admin and users, you can set up **Single Sign-On (SSO)** so your users can log in using your company **LDAP** or **OIDC** identity provider.
This feature only works after setting up Console to use your identity provider as SSO. To be guided through the steps, please select your identity provider.
### Manage user permissions before first login
For the user to appear in the users list in Console, they have to log in first. To ensure appropriate access, you can **configure permissions before their first login**. To do that:
* Go to the **Users** page and click **Create members**.
* Select **SSO** and enter the email of the user you want to add, then click **Add**.
* You can also add them to a group. Go to the **Groups** page, select the group and click **Add members**.
### Local account config properties
| Property | Description | Environment variable | Mandatory | Type | Default value |
| ----------------------------- | ------------- | -------------------------------- | --------- | ------ | ---------------------- |
| `auth.local-users[].email` | User login | `CDK_AUTH_LOCALUSERS_0_EMAIL` | true | string | `"admin@conduktor.io"` |
| `auth.local-users[].password` | User password | `CDK_AUTH_LOCALUSERS_0_PASSWORD` | true | string | `"admin"` |
# Map external groups
Source: https://docs.conduktor.io/guide/conduktor-in-production/admin/user-access/map-external-groups
Map LDAP or OIDC groups to Conduktor Console RBAC groups. Users inherit permissions at login and lose access when removed from the external group.
## Overview
External groups mapping allows you to integrate [**Conduktor's RBAC system**](/guide/conduktor-in-production/admin/set-up-rbac) with your LDAP or OIDC source of truth.
When you map an external group, the **user is added to the Conduktor Console group at login**. This ensures that users inherit necessary group permissions and that they are removed from Conduktor groups if their external membership changes.
## Prerequisites
You have to configure SSO to an LDAP or OAuth2.0 identity provider. In addition to the properties required for the default configuration, you have to also add the relevant group properties and create the scope in your IdP.
### LDAP
For LDAP, populate the `groups-base` and `groups-filter` attributes.
```yaml title="platform-config.yaml" theme={null}
sso:
ldap:
- name: "default"
server: "ldap://ldap.test.io:1389"
managerDn: "cn=admin,dc=conduktor,dc=io"
managerPassword: "pass"
search-base: "dc=conduktor,dc=io"
groups-base: "dc=conduktor,dc=io"
groups-filter: "member={0}"
```
### OIDC
For OIDC, populate the `groups-claim` attribute. You can find some examples by selecting your identity providers.
```yaml title="platform-config.yaml" theme={null}
sso:
oauth2:
- name: ""
client-id: ""
client-secret: ""
groups-claim: ""
openid:
issuer: ""
```
## Create an external group mapping
Once LDAP or OIDC are configured, you can create the mapping between external and Conduktor groups. You have two options, map an existing or a new Conduktor group to an external one.
### Map an existing group to an external one
In Console, go to **Groups** page and click **...** at the top-right and select **Map external groups**. The **External groups** tab will be open allowing you to manage this list.
The value set depends on the IdP you use. For example, for Azure, you have put the `Object ID` of your groups. For Keycloak, this is the name of the group.
Once you've mapped the external groups, use Conduktor's RBAC to enforce permissions.
### Create a new group with an external group mapped
In Console, go to **Groups** page and click **Create group** at the top-right. Provide a name and description.
To enable LDAP or OIDC users (who are members of certain teams/groups) to be automatically added, tick **Activate external groups mapping**.
Once you've mapped the external groups, use Conduktor's RBAC to enforce permissions.
# Session timeout configuration
Source: https://docs.conduktor.io/guide/conduktor-in-production/admin/user-access/session-lifetime
Configure session timeout and idle session time in Conduktor Console. Set maximum session lifetime and access token duration for user authentication.
## Session timeout overview
Conduktor uses long and short-lived tokens to verify authenticated users. By default, **the maximum session lifetime is three days**. This means that you can close Console and return within the three day window to continue your session without needing to re-authenticate.
You can also configure **the maximum idle session time** before a user becomes unauthenticated. The idle timeout represents the duration of the short-lived access token. This token will be refreshed periodically until either:
* the maximum session lifetime is reached or
* you close Conduktor for a period that extends the idle timeout value.
If the idle timeout value is not set, the same value as the maximum session time will be used. We recommend ensuring that **the idle timeout value is always less than or equal to the maximum session lifetime**.
### Configuration
Use the environment variables or configuration properties to configure your session lifetime requirements.
* `CDK_AUTH_SESSIONLIFETIME`: Max session lifetime in seconds. The default is 3 days.
* `CDK_AUTH_IDLETIMEOUT`: Max idle session time in seconds (access token lifetime). If this value is not set, the same value will be taken from `CDK_AUTH_SESSIONLIFETIME`. Should be lower than `CDK_AUTH_SESSIONLIFETIME`.
| Property | Description | Environment Variable | Mandatory | Type | Default value |
| ---------------------- | ----------------------------------------------------------------------------------------------------- | -------------------------- | --------- | ---- | ------------- |
| `auth.sessionLifetime` | Max session lifetime in seconds | `CDK_AUTH_SESSIONLIFETIME` | false | int | `259200` |
| `auth.idleTimeout` | Max idle session time in seconds (access token lifetime). Should be lower than `auth.sessionLifetime` | `CDK_AUTH_IDLETIMEOUT` | false | int | `259200` |
# Conduktor automation — CLI, Terraform, and REST API
Source: https://docs.conduktor.io/guide/conduktor-in-production/automate
Automate Conduktor Console and Gateway with the CLI, Terraform provider, or REST API. Manage users, topics, permissions.
Conduktor lets you automate various tasks, such as: managing users, groups, permissions, Kafka resources and Conduktor resources.
You can use:
* [API](/guide/conduktor-in-production/automate/api-automation)
* [Conduktor CLI](/guide/conduktor-in-production/automate/cli-automation)
* [Terraform](/guide/conduktor-in-production/automate/terraform-automation)
You can automate:
* [Console resources](/guide/reference/console-reference)
* [Gateway resources](/guide/reference/gateway-reference)
* [Kafka resources](/guide/reference/kafka-reference)
* [Self-service resources](/guide/reference/self-service-reference)
* [Metric resources](/guide/reference/console-metrics)
Supported resources:
| Resource | API/CLI | Terraform |
| ----------------------------------------------------------------- | :-----: | :-------: |
| [Console resources](/guide/reference/console-reference) | | |
| ConsoleUser | ✅ | ✅ |
| ConsoleGroup | ✅ | ✅ |
| KafkaCluster | ✅ | ✅ |
| KafkaConnectCluster | ✅ | ✅ |
| KsqlDBCluster | ✅ | ✅ |
| PartnerZone | ✅ | ✅ |
| DataQualityRule | ✅ | 🚫 |
| DataQualityPolicy | ✅ | 🚫 |
| IndexedTopic | ✅ | 🚫 |
| [Gateway resources](/guide/reference/gateway-reference) | | |
| VirtualCluster | ✅ | ✅ |
| AliasTopic | ✅ | 🚫 |
| ConcentrationRule | ✅ | 🚫 |
| Interceptor | ✅ | ✅ |
| GatewayServiceAccount | ✅ | ✅ |
| GatewayGroup | ✅ | 🚫 |
| Token | ✅ | ✅ |
| [Kafka resources](/guide/reference/kafka-reference) | | |
| Topic | ✅ | ✅ |
| Subject | ✅ | ✅ |
| Connector | ✅ | ✅ |
| [Self-service resources](/guide/reference/self-service-reference) | | |
| TopicPolicy | ✅ | ✅ |
| ResourcePolicy | ✅ | ✅ |
| TopicTemplate | ✅ | ✅ |
| ConnectorTemplate | ✅ | ✅ |
| ApplicationGroupTemplate | ✅ | ✅ |
| Application | ✅ | ✅ |
| ApplicationInstance | ✅ | ✅ |
| ApplicationInstancePermission | ✅ | ✅ |
| ApplicationGroup | ✅ | ✅ |
| ServiceAccount | ✅ | ✅ |
| [Monitoring resources](/guide/reference/console-metrics) | | |
| Alert | ✅ | 🚫 |
| Integration | ✅ | 🚫 |
# Conduktor REST API — Console and Gateway automation
Source: https://docs.conduktor.io/guide/conduktor-in-production/automate/api-automation
Automate Conduktor using the Console and Gateway REST APIs. Manage users, groups, permissions, clusters.
## Conduktor API overview
You can automate some of your tasks using our REST API. Conduktor offers the Console API and Gateway API, each used for managing different resources.
**From our blog:** [Kafka automation platform: self-serve by policy](https://conduktor.io/blog/kafka-automation-platform) Why manual provisioning turns platform teams into a ticket queue, and how policy-based automation gives them their time back.
The Conduktor HTTP API can be used to manage your organizations users, groups and their associated permissions. You can also use it to interact with Kafka resources such as clusters and certificates.
[Go to the API portal](https://developers.conduktor.io/?product=console) or `/docs` in your deployment host (e.g., `http://localhost:8080/docs`).
Use the API portal to download the OpenAPI specification. You can then import it to tools such as [Postman](https://www.postman.com/) .
### Manage API keys
In Console, go to **Settings** > **API Keys** to manage and create new keys. Only members of the admin group can generate API keys and access the API.
Give it a unique name and click **Generate Key**. You can then copy it and use as required.
Make sure to copy the key as it won't be accessible when this window is closed.
### Revoke API keys
To revoke a key, select it from the list and click the **trash can** icon. You'll be prompted to confirm as this can't be undone.
### Example
Here's a sample request listing the permissions associated with group 'project-a':
```bash theme={null}
curl \
--request GET \
--url http://localhost:8080/public/iam/v2/group/project-a \
--header "Authorization: Bearer $ADMIN_API_KEY" | jq .
{
"apiVersion": "v2",
"kind": "Group",
"metadata": {
"name": "project-a"
},
"spec": {
"displayName": "Project A",
"description": "Group for the Project A",
"externalGroups": [
"my-ldap-group"
],
"permissions": [
{
"resourceType": "TOPIC",
"cluster": "underlying-kafka",
"name": "project-a-",
"patternType": "PREFIXED",
"permissions": [
"topicConsume",
"topicProduce",
"topicViewConfig"
]
}
]
}
}
```
The Conduktor Gateway HTTP API can be used to manage your organizations Interceptors and other resources such as alias topics or topic concentration rules.
[Go to the API portal](https://developers.conduktor.io/?product=gateway) or the HTTP root page of your deployment host (e.g., `http://localhost:8888/`).
Use the API portal to download the OpenAPI specification. You can then import it to tools such as [Postman](https://www.postman.com/) .
## Related resources
* [Automate with CLI](/guide/conduktor-in-production/automate/cli-automation)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Integrate with Backstage
Source: https://docs.conduktor.io/guide/conduktor-in-production/automate/backstage-integration
[Backstage.io](https://backstage.io) is a popular open-source framework for building internal developer portals. In this guide, we show the minimal configuration to integrate Backstage with the Conduktor Console API. This integration requires a backend proxy configuration and a custom frontend Javascript/Typescript plugin in Backstage.
## Prerequisites
* Backstage v1.28 or later
* Conduktor Console with [delegated JWT authentication](/guide/conduktor-in-production/admin/user-access/delegated-jwt-authentication) configured
* Backstage and Conduktor Console use the same identity provider (IdP)
* The IdP uses OIDC to issue signed JWTs with email and groups claims (e.g. Keycloak, Okta, Auth0, Jumpcloud, etc.)
## How auth works
Authentication is split across three components:
* **Your IdP** handles user authentication (login, MFA, credential issuance)
* **Backstage** handles the interactive login UX and acquires the JWT from the IdP
* **Conduktor Console** validates the token and handles authorization
When a request arrives from Backstage, Conduktor Console verifies the JWT's signature against your IdP's JWKS public keys, checks the issuer, and confirms the token hasn't expired. Console does not authenticate the user directly; it trusts the IdP's assertion. Once the token is validated, Console extracts the email and groups claims, maps them to internal permissions, and filters the API response accordingly.
Requests without a valid JWT are rejected with a `401` unauthorized error code.
## Configure the Backstage proxy
Add the Conduktor proxy endpoint to your Backstage `app-config.yaml`:
```yaml theme={null}
proxy:
endpoints:
'/conduktor':
target: https://
credentials: dangerously-allow-unauthenticated
allowedHeaders:
- Authorization
changeOrigin: true
```
This forwards requests from `/api/proxy/conduktor/*` to your Conduktor Console, passing through the `Authorization` header containing the user's JWT. The proxy is needed because the Backstage frontend and Conduktor run on different origins, and browser CORS policies would block direct cross-origin requests.
The [`dangerously-allow-unauthenticated`](https://backstage.io/docs/plugins/proxying/#configuration) setting tells the Backstage proxy to skip its own token validation and forward requests as-is, allowing the `Authorization` header to pass through to Conduktor unchanged. This is safe because Conduktor validates the JWT and enforces permissions on every request. Requests without a valid token are rejected by Conduktor with a `401`.
## Call the Conduktor API from a frontend plugin
In your Backstage frontend plugin, get the user's access token from the existing sign-in session and call Conduktor through the proxy. The `authApiRef` below is whatever auth provider API ref your Backstage app uses for sign-in (for example, the ref you pass to the `SignInPage` component). If you use a different IdP, replace it with the corresponding ref (such as `microsoftAuthApiRef`, `oktaAuthApiRef` or a custom ref you created with `createApiRef`).
```typescript theme={null}
const authApi = useApi(authApiRef); // your app's auth provider API ref
const discoveryApi = useApi(discoveryApiRef);
const token = await authApi.getAccessToken();
const proxyBase = await discoveryApi.getBaseUrl('proxy');
const response = await fetch(
`${proxyBase}/conduktor/public/console/v2/kafka-cluster`,
{ headers: { Authorization: `Bearer ${token}` } }
);
const clusters = await response.json();
// Render the response and display as you see fit
```
Use the browser's native `fetch()`, not Backstage's `fetchApi.fetch()`. The Backstage `fetchApi` automatically injects its own identity token into the `Authorization` header, which would overwrite your IdP token.
For the full API reference and response structure, see the [API portal](https://developers.conduktor.io/?product=console) or access `/docs` on your Conduktor deployment.
## Optional: add Conduktor API to the Backstage catalog
You can register the Conduktor API spec as a catalog entity so developers can browse the API documentation directly in Backstage. Add this to your catalog:
```yaml theme={null}
apiVersion: backstage.io/v1alpha1
kind: API
metadata:
name: conduktor-console
description: Conduktor Console API for Kafka management
tags:
- kafka
- rest
spec:
type: openapi
lifecycle: production
owner: platform-team
definition:
$text: https:///public/docs/docs.yaml
```
This is purely for documentation browsing and doesn't affect authentication or permissions.
## Permissions
No permission logic is needed in Backstage. Conduktor enforces all authorization server-side.
When a request arrives with a valid JWT, Conduktor:
1. Validates the token (signature via JWKS, issuer, expiration)
2. Extracts the user's email from the configured claim
3. Maps the groups claim to internal Conduktor groups
4. Auto-provisions the user if they don't exist yet
5. Applies RBAC and filters the API response
If a user can see a resource in the Conduktor UI, they can see it through Backstage. If they can't, the API either omits it from the response or returns a `403`.
# Conduktor CLI — automate Kafka with command line
Source: https://docs.conduktor.io/guide/conduktor-in-production/automate/cli-automation
Manage Conduktor Console and Gateway resources from the CLI or CI/CD pipelines. Apply, get, and delete Kafka resources declaratively with the Conduktor CLI.
## Conduktor CLI overview
Conduktor CLI (Command Line Interface) helps you to manage your [Console and Gateway resources](/guide/reference) via commands or CI/CD pipelines.
**From our blog:** [Houston, we have 7 CLI tools and zero answers](https://conduktor.io/blog/houston-we-have-7-cli-tools-and-zero-answers) The CLI fragmentation problem for Kafka teams and how a unified toolchain changes the game.
## Install
You have three options to install Conduktor CLI:
* [Native binary](#native-binary) - for individual use and testing
* [Homebrew](#homebrew) - for individual use and testing on MacOS
* [Docker](#docker) - for integration in CI/CD pipelines
### Native binary
[Download the latest binary from GitHub](https://github.com/conduktor/ctl/releases) on Windows, Linux or MacOS.
In the **Assets** list, download the build that corresponds to your machine (for Apple Silicon use `darwin-arm64`).
### Homebrew
Use the [Homebrew package manager](https://brew.sh/) to install our CLI:
```bash theme={null}
brew install conduktor/brew/conduktor-cli
```
### Docker
Pull the [latest Dockerhub version](https://hub.docker.com/r/conduktor/conduktor-ctl) of the CLI using the following command line:
```bash theme={null}
docker pull conduktor/conduktor-ctl
```
## Upgrade
### Homebrew
To upgrade to the latest version of the CLI:
```bash theme={null}
brew upgrade conduktor/brew/conduktor-cli
```
### Native binary
[Download the latest binary from GitHub](https://github.com/conduktor/ctl/releases) and replace your existing installation.
### Docker
Pull the latest version:
```bash theme={null}
docker pull conduktor/conduktor-ctl
```
## Configure
To use the Conduktor CLI, you need to set:
* the URL of your Conduktor Console and/or Gateway API
* the authentication mechanism: either by using an API key, or user credentials
### Authenticate to Console using API keys
These are the necessary environment variables for this authentication method::
```bash theme={null}
export CDK_BASE_URL=http://localhost:8080
export CDK_API_KEY=
```
There are two types of API keys: **admin** and **Self-service application**.
Admin API keys grant the maximum permissions in Console.
```bash title="Generate an admin key named 'my-admin-key'" theme={null}
conduktor token create admin my-admin-key
```
Go to **Settings** > **API Keys** and select **New API Key**:
Self-service Application API key permissions are **limited to the scope of the ApplicationInstance** for which they have been generated. [Find out more about Self-service](/guide/use-cases/self-service).
```bash title="Generate an application API key named 'my-admin-key' for the application 'my-app-instance'" theme={null}
conduktor token create application-instance -i=my-app-instance my-app-instance-key
```
Go to **Applications**, pick your **Application** and **Application Instance**, then go to the **API Keys** tab and click **New API Key**:
### Authenticate to Console using user credentials
You can simply set the username and password of the **local** or **LDAP** user and use the CLI, or create an associated short-lived API key. This type of API key will have the permissions of the user who created it.
This kind of authentication will be valid for the same duration as the configured [session timeout](/guide/conduktor-in-production/admin/user-access/session-lifetime).
```bash theme={null}
export CDK_BASE_URL=http://localhost:8080
export CDK_USER=admin@company.io
export CDK_PASSWORD=adminP4ss!
```
```bash title="Generate a short-lived API key" theme={null}
conduktor login
```
### Authenticate to Console using external authentication
By default, the Conduktor CLI authenticates with Console using a Console API key, or generates one based on the given credentials. This is the `console` authentication mode (`CDK_AUTH_MODE` is `console` or empty).
Alternatively, you can use the external authentication mode to bypass token generation and use credentials from your identity provider or the API proxy instead. For that, set the `CDK_AUTH_MODE` environment variable to `external`:
```bash title="Set up external authentication with a bearer token" theme={null}
export CDK_AUTH_MODE="external"
export CDK_API_KEY=
```
```bash title="Set up external authentication with client credentials" theme={null}
export CDK_AUTH_MODE="external"
export CDK_USER=
export CDK_PASSWORD=
```
### Authenticate to Gateway using API credentials
To use the CLI to manage Gateway objects, you need to set the URL of the Gateway API and the API credentials.
```bash theme={null}
export CDK_GATEWAY_BASE_URL=http://localhost:8888
export CDK_GATEWAY_USER=admin
export CDK_GATEWAY_PASSWORD=conduktor
```
### Environment variables
This is the list of the supported environment variables:
| Environment variables | Description |
| ---------------------- | ------------------------------------------------------------------------ |
| **Console** | |
| `CDK_BASE_URL` | Console base URL, e.g. `http://localhost:8080` |
| `CDK_API_KEY` | Console API key (admin or application key) |
| `CDK_USER` | Console local or LDAP user username |
| `CDK_PASSWORD` | Console local or LDAP user password |
| `CDK_AUTH_MODE` | Authentication mode (`external`, `console` or empty (default)) |
| **Gateway** | |
| `CDK_GATEWAY_BASE_URL` | Gateway base URL, e.g. `http://localhost:8888` |
| `CDK_GATEWAY_USER` | Gateway user login email |
| `CDK_GATEWAY_PASSWORD` | Gateway user login password |
| **TLS and mTLS** | |
| `CDK_CACERT` | Root CA certificate to verify the Conduktor Console certificate (.pem) |
| `CDK_INSECURE` | Skip the TLS verification of the Conduktor Console. Defaults to `false`. |
| `CDK_CERT` | Certificate to authenticate the CLI (.pem) |
| `CDK_KEY` | Key to authenticate the CLI (.pem) |
## CLI commands
The CLI offers a set of commands to manage Conduktor resources. You can use it to create, update, delete and get resources.
```
Usage:
conduktor [flags]
conduktor [command]
Available commands:
apply Upsert a resource on Conduktor
completion Generate the autocompletion script for the specified shell
delete Delete resource of a given kind and name
get Get resource of a given kind
help Help about any command
login Login user using username password to get a JWT token
run run an action
sql
template Get a .yaml example for a given kind
token Manage Admin and Application Instance tokens
version Display the version of Conduktor
Flags:
-h, --help help for Conduktor
--permissive permissive mode, allow undefined environment variables
-v, --verbose show more information for debugging
Use "conduktor [command] --help" for more information about a command.
```
### Apply
The `apply` command allows you to deploy any resource.
```
Usage:
conduktor apply [flags]
Flags:
--dry-run Test potential changes without the effects being applied
-f, --file stringArray Specify the files or folders to apply. For folders, all .yaml or .yml files within the folder will be applied, while files in subfolders will be ignored.
-h, --help help for apply
--parallelism int Run each apply in parallel, useful when applying a large number of resources. Must be less than 100. (default 1)
-r, --recursive Apply all .yaml or .yml files in the specified folder and its subfolders. If not set, only files in the specified folder will be applied.
Global Flags:
--permissive Permissive mode, allow undefined environment variables
-v, --verbose Show more information for debugging
```
The `--parallelism` flag allows you to run multiple applies at the same time, in parallel. This is useful when applying a large number of resources. Be careful not to set it too high, as it may overload the Console/Gateway API.
Examples:
```bash title="Apply all resources in multiple directories" theme={null}
conduktor apply -f ./directory1 -f ./directory2
```
```bash title="Apply a specific file" theme={null}
conduktor apply -f file.yaml
```
```bash title="Apply all resources in the current directory and its subfolders" theme={null}
conduktor apply -f . -r
```
### Template
The `template` command allows you to get an example of a resource definition:
```bash theme={null}
conduktor template topic
---
apiVersion: v2
kind: Topic
metadata:
name: my-topic
cluster: my-cluster
labels:
conduktor.io/application: application-a
conduktor.io/application-instance: staging
user-labels: I am a user label
catalogVisibility: PUBLIC
descriptionIsEditable: true
description: This is a topic description
sqlStorage:
retentionTimeInSecond: 42
spec:
partitions: 1
replicationFactor: 1
configs:
cleanup.policy: delete
retention.ms: '86400000'
```
You can also save the output in a file, in order to edit it before applying it:
```bash theme={null}
conduktor template KafkaCluster -o definition.yml
conduktor template Topic >> definition.yml #Appending to an existing file
vim definition.yml
conduktor apply -f definition.yml
```
### Delete
The `delete` command allows you to delete a resource. It works the same way as the `apply`.
Resources are deleted instantly and can't be recovered. Any data or access associated with the resource is permanently lost.
Examples:
```bash title="Delete all resources in multiple directories" theme={null}
conduktor delete -f ./directory1 -f ./directory2
```
```bash title="Delete a specific file" theme={null}
conduktor delete -f file.yaml
```
```bash title="Delete all resources in the current directory and its subfolders" theme={null}
conduktor delete -f . -r
```
### Get
The `get` command allows you to retrieve resources. You can specify a kind, a name, or list all the resources.
```
conduktor get user --help
If name not provided it will list all resource
Usage:
conduktor get User [name] [flags]
Aliases:
User, user
Flags:
-h, --help help for User
Global Flags:
-o, --output output Output format. One of: json|yaml|name (default yaml)
--permissive Permissive mode, allow undefined environment variables
-v, --verbose Show more information for debugging
```
Examples:
```bash title="Get all users" theme={null}
conduktor get user
```
```bash title="Get a specific Kafka connect cluster" theme={null}
conduktor get KafkaConnectCluster --cluster my-kafka-cluster
```
```bash title="Get all the Console resources with a 'name' output format" theme={null}
conduktor get all --console -o name
```
### Token
To run your CI/CD pipelines, you can use the `token` command to manage your API keys.
```bash title="Create an admin API keys" theme={null}
conduktor token create admin my_first_admin_token
```
```bash title="List Console admin API keys" theme={null}
conduktor token list admin
```
```bash title="Create an application instance API key" theme={null}
conduktor token create application-instance -i=my_instance my_token_for_my_instance
```
```bash title="List API keys of an application instance" theme={null}
conduktor token list application-instance -i=my_instance
```
```bash title="Delete an API key" theme={null}
conduktor token delete 1233ff18-0e95-4638-b3fc-f55e20088b8d
```
### Run
The `run` command allows you to run a few actions:
```
Usage:
conduktor run [flags]
conduktor run [command]
Available Commands:
generateServiceAccountToken Generate a token for a service account on a virtual cluster
partnerZoneGenerateCredentials generate a token for a partner zone service account
whoami Return information about the credentials configured in the CLI
Flags:
-h, --help help for run
Global Flags:
--permissive Permissive mode, allow undefined environment variables
-v, --verbose Show more information for debugging
```
Examples:
```bash title="Generate a service account password for a virtual cluster" theme={null}
conduktor run generateServiceAccountToken --v-cluster my-vcluster --username my-service-account
```
```bash title="Generate a Partner Zone service account password" theme={null}
conduktor run partnerZoneGenerateCredentials --partner-zone-name my-partner-zone
```
```bash title="Get information about the credentials configured in the CLI" theme={null}
conduktor run whoami
```
### Manage state
State management is available in CLI v0.7.0 and above.
State management is an optional feature that tracks resources applied via the CLI. It helps detect orphaned resources (resources in state but no longer in your configuration files) and automatically removes them.
Enable state management with the `--enable-state` flag:
```bash title="Apply with local state tracking" theme={null}
conduktor apply -f resources.yaml --enable-state
```
#### Local state
By default, the state file is stored locally at:
| Platform | Default location |
| -------- | ------------------------------------------------------------ |
| Linux | `$XDG_DATA_HOME/.local/share/conduktor/cli-state.json` |
| macOS | `$HOME/Library/Application Support/conduktor/cli-state.json` |
| Windows | `$APPDATA/conduktor/cli-state.json` |
Use `--state-file` to specify a custom path:
```bash theme={null}
conduktor apply -f resources.yaml --enable-state --state-file ./my-state.json
```
The state file is a JSON document containing metadata about tracked resources:
```json theme={null}
{
"version": "v1",
"lastUpdated": "2024-12-05T10:30:00Z",
"resources": [
{
"apiVersion": "v1",
"kind": "Topic",
"metadata": { "name": "my-topic", "cluster": "my-cluster" }
}
]
}
```
#### Remote state
For team collaboration and CI/CD pipelines, store state in remote object storage using `--state-remote-uri`:
```bash theme={null}
# Set AWS credentials
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
# Apply with S3 remote state
conduktor apply -f resources.yaml \
--enable-state \
--state-remote-uri "s3://my-bucket/conduktor/state/?region=us-east-1"
```
Also supports S3-compatible services like MinIO or DigitalOcean Spaces.
See [S3 URI options](https://github.com/conduktor/ctl/blob/main/docs/state_management.md#s3-uri-query-parameters) and [authentication methods](https://github.com/conduktor/ctl/blob/main/docs/state_management.md#s3-authentication).
```bash theme={null}
# Set GCP credentials
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
# Apply with GCS remote state
conduktor apply -f resources.yaml \
--enable-state \
--state-remote-uri "gs://my-bucket/conduktor/state/"
```
See [GCS URI options](https://github.com/conduktor/ctl/blob/main/docs/state_management.md#gcs-uri-query-parameters) and [authentication methods](https://github.com/conduktor/ctl/blob/main/docs/state_management.md#gcs-authentication).
```bash theme={null}
# Set Azure credentials
export AZURE_STORAGE_ACCOUNT=
export AZURE_STORAGE_KEY=
# Apply with Azure Blob remote state
conduktor apply -f resources.yaml \
--enable-state \
--state-remote-uri "azblob://my-container/conduktor/state/"
```
See [Azure URI options](https://github.com/conduktor/ctl/blob/main/docs/state_management.md#azure-blob-uri-query-parameters) and [authentication methods](https://github.com/conduktor/ctl/blob/main/docs/state_management.md#azure-blob-authentication).
#### Environment variables
You can enable state management globally using environment variables:
| Environment variable | Description |
| ---------------------- | ---------------------------------------- |
| `CDK_STATE_ENABLED` | Enable state tracking globally |
| `CDK_STATE_FILE` | Custom local state file path |
| `CDK_STATE_REMOTE_URI` | Remote storage URI (S3, GCS, Azure Blob) |
[See detailed URI options, authentication methods and best practices](https://github.com/conduktor/ctl/blob/main/docs/state_management.md).
### Version
Check the current version of your CLI:
```bash title="Check CLI version" theme={null}
conduktor version
```
### CI/CD pipeline integration example
Conduktor CLI can be easily integrated to a CI/CD pipeline. This example presents two pipelines:
* The first one triggers on each new PR and launches the CLI using the `--dry-run` flag, generating a report confirming that the resources can be successfully created or modified.
* The second one triggers on a push to the `main` branch, making the changes live.
For a fuller, production-ready Self-service example — splitting platform and application resources, using scope-based tokens, isolating remote state per workflow, and enforcing review with CODEOWNERS — start from the [conduktor/self-service-template](https://github.com/conduktor/self-service-template) reference repository.
Consider the following folder structure:
```
├── resources/
├── topics.yml # Your topics are there
├── permissions.yml # Your permissions to other apps are there
```
```yaml title=".github/workflows/on-pr.yml" theme={null}
name: Check PR Validity
on:
pull_request:
branches: [ "main" ]
workflow_dispatch:
jobs:
build:
runs-on: ubuntu-latest
container: conduktor/conduktor-ctl
steps:
- uses: actions/checkout@v3
- run: /bin/conduktor apply -f resources/ --dry-run
env:
CDK_BASE_URL: https://conduktor.domain.com
CDK_API_KEY: ${{ secrets.CONDUKTOR_TOKEN }}
```
```yaml title=".github/workflows/on-push.yml" theme={null}
name: Execute committed changes
on:
push:
branches: [ "main" ]
jobs:
build:
runs-on: ubuntu-latest
container: conduktor/conduktor-ctl
steps:
- uses: actions/checkout@v3
- run: /bin/conduktor apply -f resources/
env:
CDK_BASE_URL: https://conduktor.domain.com
CDK_API_KEY: ${{ secrets.CONDUKTOR_TOKEN }}
```
```yaml title=".gitlab-ci.yml" theme={null}
conduktor-pr:
only:
- merge_requests
stage: deploy
image:
name: conduktor/conduktor-ctl
entrypoint: [""]
variables:
- export CDK_BASE_URL=https://conduktor.domain.com
- export CDK_API_KEY=${CONDUKTOR_TOKEN}
script:
- /bin/conduktor apply -f resources/ --dry-run
conduktor-main:
only:
refs:
- main
stage: deploy
image:
name: conduktor/conduktor-ctl
entrypoint: [""]
variables:
- export CDK_BASE_URL=https://conduktor.domain.com
- export CDK_API_KEY=${CONDUKTOR_TOKEN}
script:
- /bin/conduktor apply -f resources/
```
### Environment variables for secrets
The Conduktor CLI automatically substitutes environment variables in YAML manifests using `${ENV_VAR}` patterns. This keeps sensitive data like credentials out of configuration files.
You can set a default value using POSIX notation `${ENV:-default}`.
By default, the CLI fails if an environment variable is undefined or empty, ensuring all required variables are properly configured.
Use the `--permissive` flag to replace missing or empty variables with empty strings instead of failing.
#### Escape sequences for Interceptor configs
When you use environment variables in Gateway Interceptor configurations, you have to escape dollar signs with double dollar signs `$$` for secured templates.
This applies to:
* Encryption keys
* Passwords
* API tokens
* Any secret values in Interceptor config blocks
**Example:**
```yaml theme={null}
apiVersion: gateway/v2
kind: Interceptor
metadata:
name: my-encryption-interceptor
spec:
pluginClass: io.conduktor.gateway.interceptor.EncryptPlugin
config:
# Use $$ to escape environment variables in Interceptor configs
encryptionKey: $${ENCRYPTION_KEY}
apiToken: $${API_TOKEN}
```
Without the `$$` escape sequence, the CLI will not correctly substitute the environment variable in the Interceptor configuration.
#### Kafka cluster example
The example below shows a Conduktor cluster configuration with credentials secured using environment variables:
```yaml theme={null}
---
apiVersion: console/v2
kind: KafkaCluster
metadata:
name: my-cluster
spec:
displayName: "My Kafka Cluster"
icon: "kafka"
color: "#000000"
bootstrapServers: "localhost:9092"
ignoreUntrustedCertificate: false
properties:
sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username="${ENV_VAR_FOR_USER}" password="${ENV_VAR_FOR_PASSWORD}";
security.protocol: SASL_SSL
sasl.mechanism: PLAIN
schemaRegistry:
type: "ConfluentLike"
url: http://localhost:8080
security:
type: BasicAuth
username: ${ENV_VAR_FOR_USER}
password: ${ENV_VAR_FOR_PASSWORD}
ignoreUntrustedCertificate: false
kafkaFlavor:
type: "Confluent"
key: "${ENV_VAR_CONFLUENT_KEY}"
secret: "${ENV_VAR_CONFLUENT_SECRET}"
confluentEnvironmentId: "${ENV_VAR_CONFLUENT_ENV_ID:-dev}"
confluentClusterId: "${ENV_VAR_CONFLUENT_CLUSTER_ID:-main}"
```
## Related resources
* [Automate with API](/guide/conduktor-in-production/automate/api-automation)
* [View Console resource reference](/guide/reference/console-reference)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Kafka MCP server for AI assistants
Source: https://docs.conduktor.io/guide/conduktor-in-production/automate/mcp
Connect AI assistants to Kafka via Conduktor's MCP server. Query topics, consumer groups, and schema registry in natural language using Claude or compatible MCP clients.
To discover information about your Kafka resources, you can connect to Conduktor Console from an AI assistant via the Model Context Protocol (MCP). Use Console UI for setup instructions:
The MCP server uses your own permissions based on the token you generate. Make sure the token has the appropriate access rights for the resources you want to query.
Use AI tools for discovery and exploration only — results are open to interpretation and outside of Conduktor's control.
**From our blog:** [Conduktor MCP and skills for AI agents](https://conduktor.io/blog/conduktor-mcp-and-skills-for-ai-agents) How Conduktor's native MCP server connects AI assistants to Kafka with built-in RBAC and no separate server to deploy.
## MCP tools
All tools are read-only. They cover Kafka resources and [Insights](/guide/insights) data held by Console.
### Cluster management
* **list-clusters**: list all Kafka clusters available in Conduktor
* **get-cluster**: get details about a specific cluster by its slug
### Topic operations
* **list-topics-with-usage**: list topics with usage metrics (message count, size, rates) and filtering by name, cleanup policy, labels, tags, or to hide internal/stream topics
* **get-last-messages**: retrieve the last *N* messages from a topic (up to 100 messages, all partitions or a specific subset)
### Insights
* **insights-cluster**: cluster health summary and serialization format breakdown
* **insights-topics**: full topic insights — partition skew, replication issues, partition distribution, health overview, empty/stale/tiny topics, schema coverage, and Self-service coverage
### Schema registry
* **list-subject-names**: list subject names in a cluster, with filtering by schema type (AVRO, PROTOBUF, JSON)
### Consumer groups
* **list-consumer-groups**: list consumer groups with state, lag, and member count; filter by name, state, or labels; sort by lag, members, etc.
* **list-consumer-groups-by-topic**: list consumer groups consuming from a specific topic
### Gateway and Interceptors
* **list-interceptors**: list interceptors configured in a cluster (requires Conduktor Gateway)
# Kafka Terraform provider — Conduktor
Source: https://docs.conduktor.io/guide/conduktor-in-production/automate/terraform-automation
Manage Conduktor Console resources as code with the Conduktor Terraform provider. Declare topics, users, permissions, and Gateway configs in HCL.
## Conduktor Terraform overview
The Conduktor Terraform provider gives you the ability to perform some operations on Conduktor Console state directly from existing [Terraform Infrastructure-as-Code environment](https://developer.hashicorp.com/terraform/intro) .
The Conduktor Terraform provider may not include the latest features. [Check out all the supported resources](/guide/conduktor-in-production/automate).
## Install
The provider should be installed automatically with `terraform init`, however we recommend to pin a specific version or range of versions using the [required\_provider configurations](https://developer.hashicorp.com/terraform/language/providers/configuration) :
```hcl theme={null}
terraform {
required_providers {
conduktor = {
source = "conduktor/conduktor"
version = "~> X.Y" # where X.Y is the current major version and minor version
}
}
}
```
## Configure
To use the Conduktor Terraform Provider, you need to configure:
* the mode, since the provider can also be used for managing Conduktor Gateway resources,
* the URL of Conduktor Console,
* the authentication mechanism: either using an API key or the local user credentials (usually admin account)
[View the full configuration reference](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs) .
### Authenticate using API Keys
```hcl theme={null}
# configure provider
provider "conduktor" {
mode = "console"
base_url = "http://localhost:8080"
api_token = "your-api-key"
}
```
There are two types of API Keys: **admin** and **Self-service application**.
Admin API Keys grant the maximum permissions on Console.
```bash theme={null}
# Generate a key named my-admin-key
$ conduktor token create admin my-admin-key
AWpw1sZZC20=.29Qb9KbyeQTrewMtnVDYAprxmYo7MUQats2KHzVhx+B/kGOBuIoH8CMsjOcvolUjLKFqbQNSvY0/98wb8mqxU4NwQTSgbSSAlLxau3caByHR6/X9EeqQdj3Lhf0xCzh87/GxYK5JG2DI1VWj55A6xcH++ottyG909PwuGe/GIwgfxX3FKaopg8hxgUmPJNRSWqX+75a8eQi014J4YxuTD7w+723kOQBTXOysfGUaYnfwCCjPPmSWXEEqy5wkH2NS+jXi3S6+fH0ts8CoqvV6Z8YLmBupdMgCtJ9MVBYeDarIzQw6XY7yNuypUqer0dcd9B3KyVR8ecNpFiF7ybvP4g==
```
Go to **Settings** > **API Keys** and select **New API Key**:
Self-service Application API Key permissions are **limited to the scope of the ApplicationInstance** for which they have been generated. [Find out more about Self-service](/guide/use-cases/self-service).
```bash theme={null}
$ conduktor token create application-instance -i= my-app-instance-key
AWpw1sZZC20=.29Qb9KbyeQTrewMtnVDYAprxmYo7MUQats2KHzVhx+B/kGOBuIoH8CMsjOcvolUjLKFqbQNSvY0/98wb8mqxU4NwQTSgbSSAlLxau3caByHR6/X9EeqQdj3Lhf0xCzh87/GxYK5JG2DI1VWj55A6xcH++ottyG909PwuGe/GIwgfxX3FKaopg8hxgUmPJNRSWqX+75a8eQi014J4YxuTD7w+723kOQBTXOysfGUaYnfwCCjPPmSWXEEqy5wkH2NS+jXi3S6+fH0ts8CoqvV6Z8YLmBupdMgCtJ9MVBYeDarIzQw6XY7yNuypUqer0dcd9B3KyVR8ecNpFiF7ybvP4g==
```
Go to **Applications**, pick your **Application** and under the **Application Instances** tab click **New API Key**:
### Authenticate using short-lived user credentials
This type of API Key will have the permissions of the user who created it. It only works for **Local** and **LDAP** users.
Short-lived user API Keys will be valid for the same duration as the configured [session timeout](/guide/conduktor-in-production/admin/user-access/session-lifetime).
OIDC users can't be used.
```hcl theme={null}
# configure provider
provider "conduktor" {
mode = "console"
base_url = "http://localhost:8080"
admin_user = "console-admin@mycompany.io"
admin_password = "console-admin-password"
}
```
### Environment variables
The provider configuration also supports environment variables for all attributes except for `mode`.
| Environment variables | HCL value | Description |
| ---------------------- | ---------------- | ----------------------------------------------------------------------------- |
| n/a | `mode` | Terraform Provider mode: either `console` or `gateway` |
| `CDK_CONSOLE_BASE_URL` | `base_url` | Console base URL, e.g. `http://localhost:8080` |
| `CDK_API_KEY` | `api_token` | Console API key |
| `CDK_CONSOLE_USER` | `admin_user` | Console user login email |
| `CDK_CONSOLE_PASSWORD` | `admin_password` | Console user login password |
| `CDK_CONSOLE_CERT` | `cert` | Cert in PEM format to authenticate using client certificates |
| `CDK_CONSOLE_INSECURE` | `insecure` | Skip TLS verification flag. Defaults to `false` |
| `CDK_CONSOLE_CACERT` | `cacert` | Root CA certificate in PEM format to verify the Conduktor Console certificate |
| `CDK_CONSOLE_KEY` | `key` | Key in PEM format to authenticate using client certificates |
The configuration resolution is (in order of priority):
1. HCL values
2. Environment variables
3. Default values (if applicable)
## Provider usage
Once the provider is configured, all the [currently available resources](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs) can be used in the Terraform definition.
### Usage example
A full example of user and group creation using Conduktor Console locally is available on port `8080` with default admin account credentials `admin@mycompany.io` / `adminP4ss!`.
```hcl title="conduktor-iac.tf" theme={null}
# Provider installation configuration
terraform {
required_providers {
conduktor = {
source = "conduktor/conduktor"
version = ">= 0.5.0"
}
}
}
# Provider configuration
provider "conduktor" {
mode = "console"
base_url = "http://localhost:8080"
admin_user = "admin@mycompany.io"
admin_password = "adminP4ss!"
}
# Create example_user, Bob
resource "conduktor_console_user_v2" "bob" {
name = "bob@mycompany.io"
spec = {
firstname = "Bob"
lastname = "Smith"
permissions = [
{
resource_type = "PLATFORM"
permissions = ["userView", "datamaskingView", "auditLogView"]
}
]
}
}
# Create a group with Bob as a member
resource "conduktor_console_group_v2" "team-a" {
name = "team-a"
spec = {
display_name = "team-a"
description = "The group of team-a"
members = [conduktor_console_user_v2.bob.name]
permissions = []
}
}
```
Then on a terminal with Terraform installed and in a directory containing `conduktor-iac.tf` file:
```shell theme={null}
# Initialize terraform project
terraform init
# Plan to preview the changes Terraform will make to match your configuration.
terraform plan
# Apply to make the planned changes.
terraform apply
```
Now if you navigate to the Conduktor UI, you will see a new user, Bob, and team-a's group created. Log in using an external SSO (LDAP or OIDC) with email `bob@mycompany.io` and you'll be recognized by Conduktor as being in the `team-a` group.
To revert the Conduktor state, you can destroy the created resources using `terraform destroy`. [Find out more about Terraform CLI commands](https://developer.hashicorp.com/terraform/cli/commands) .
#### Case of the `generic` resource
The `generic` resource leverages the YAML format used by the CLI to be used as resource definition in Terraform.
This is an experimental resource that has several [limitations](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs/resources/generic#limitations) and is subject to breaking changes in future releases.
We recommend you don't use it in production workflows.
## Related resources
* [Conduktor Terraform provider](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs)
* [Terraform introduction](https://developer.hashicorp.com/terraform/intro)
* [Full list of Terraform resources](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs)
* [Terraform HCL docs](https://developer.hashicorp.com/terraform/language)
* [Terraform CLI docs](https://developer.hashicorp.com/terraform/cli/commands)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Conduktor component overview
Source: https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts
Conduktor platform components: Console for Kafka management UI, Gateway as Kafka proxy, Cortex for monitoring, and CLI for automation.
Conduktor Platform consists of these components:
* [Console](#console): The centralized UI-based control plane for managing Kafka.
* [Gateway](#gateway): A network proxy for Apache Kafka, complete with an extensible and dynamic plugin mechanism that can be used to add technical and business logic on top of your existing Kafka deployment.
* [Cortex](#cortex): The monitoring component for Conduktor Platform that runs in the background.
* [Conduktor CLI](#conduktor-cli): Our Command Line Interface that lets you automate operations.
## Gateway
Conduktor Gateway is deployed between your client applications and existing Kafka clusters. As it's Kafka protocol compliant, there are minimal adjustments required for clients other than pointing to a new bootstrap server.
### Gateway benefits
This can be used to provide functionality that is not available in Kafka natively, such as:
* **Centrally configure encryption** at the field-level or full payload.
* **Mask sensitive data** and define access control.
* **Set granular RBAC controls**.
* **Leverage multi-tenancy**.
* **Empower development teams** with self-service governance.
Supports all Kafka providers (Confluent, AWS MSK, Redpanda, Aiven, Apache Kafka).
### How it works
Gateway acts as a Kafka-compliant middle layer between clients and Kafka clusters.
#### Authentication
Supports multiple security protocols:
* PLAINTEXT
* SSL
* SASL SSL
* mTLS
#### Interceptors
Use Interceptors to add logic like encryption or filtering. Manage through [API](https://developers.conduktor.io/?product=gateway) . [More on Interceptors](/guide/conduktor-concepts/interceptors)
#### Processing flow
```mermaid theme={null}
flowchart LR
A[User App]
subgraph G [Gateway]
direction LR
Auth[Authentication & Authorization]
subgraph I [Dynamic interceptor pipeline]
direction LR
I1(Plugin priority: 1 interceptor)
I2(Plugin priority: 10 interceptor1 & interceptor2)
I3(Plugin priority: 42 interceptor)
I1 <--> I2 <--> I3
end
subgraph Core [Core features]
direction TB
LT(Logical Topics)
VC(Virtual clusters)
end
Auth <--> I
end
subgraph K [Main Kafka cluster]
B1(broker 1)
B2(broker 2)
B3(broker 3)
B1 === B2 === B3
end
A <--> Auth
I <--> Core
Core <--> K
```
## Related resources
* [Conduktor in production deployment overview](/guide/conduktor-in-production/)
* [Contact support](https://www.conduktor.io/contact/support)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Deploy Console with Docker, Kubernetes, Helm
Source: https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-console
Deploy Conduktor Console with Docker, Kubernetes (Helm), or docker-compose. Requires PostgreSQL. Covers TLS, environment variables, memory tuning, and proxy configuration.
Conduktor Console can be deployed and configured using either a configuration file `platform-config.yaml` or **environment variables**.
To set up your organization's environment:
1. Enter the organization's name.
2. Connect to an external database (**required**).
3. Set up user authentication (basic or SSO).
4. Apply your Console license.
We recommend using the Console UI (**Settings** > **Clusters** page) to configure Kafka cluster, schema registry and Kafka connect. This has several advantages over the YAML configuration:
* Intuitive interface with live update capabilities
* Centralized and secured with RBAC and audit logs events
* Certificate store to help with custom certificates configuration (no more JKS files and volume mounts)
[Check out the recommended deployment on GitHub](https://github.com/conduktor/conduktor-reference-architecture) .
### Security considerations
* The configuration file should be protected by file system permissions.
* The database should have at-rest data encryption enabled on the data volume and have limited network connectivity.
#### Configuration file
```yaml title="platform-config.yaml" theme={null}
organization:
name: demo
admin:
email: admin@company.io
password: admin
database:
url: postgresql://conduktor:change_me@host:5432/conduktor
# OR in a decomposed way
# host: "host"
# port: 5432
# name: "conduktor"
# username: "conduktor"
# password: "change_me"
# connection_timeout: 30 # in seconds
auth:
local-users:
- email: user@conduktor.io
password: user
license: ''
```
#### Bind file
The `docker-compose` below shows how to bind your **platform-config.yaml** file.
You can alternatively use environment variables. The `CDK_IN_CONF_FILE` variable is used to indicate that a configuration file is being used and the location to find it.
```yaml title="docker-compose.yaml" theme={null}
services:
postgresql:
image: postgres:14
hostname: postgresql
volumes:
- pg_data:/var/lib/postgresql/data
environment:
POSTGRES_DB: "conduktor"
POSTGRES_USER: "conduktor"
POSTGRES_PASSWORD: "change_me"
POSTGRES_HOST_AUTH_METHOD: "scram-sha-256"
conduktor-console:
image: conduktor/conduktor-console
depends_on:
- postgresql
ports:
- "8080:8080"
volumes:
- conduktor_data:/var/conduktor
- type: bind
source: "./platform-config.yaml"
target: /opt/conduktor/platform-config.yaml
read_only: true
environment:
CDK_IN_CONF_FILE: /opt/conduktor/platform-config.yaml
healthcheck:
test: curl -f http://localhost:8080/platform/api/modules/health/live || exit 1
interval: 10s
start_period: 10s
timeout: 5s
retries: 3
volumes:
pg_data: {}
conduktor_data: {}
```
#### Environment override
Input configuration fields can also be provided using environment variables. Here's an example of `docker-compose` that uses environment variables for configuration:
```yaml title="docker-compose.yaml theme={null}
services:
postgresql:
image: postgres:14
hostname: postgresql
volumes:
- pg_data:/var/lib/postgresql/data
environment:
POSTGRES_DB: "conduktor"
POSTGRES_USER: "conduktor"
POSTGRES_PASSWORD: "change_me"
POSTGRES_HOST_AUTH_METHOD: "scram-sha-256"
conduktor-console:
image: conduktor/conduktor-console
depends_on:
- postgresql
ports:
- "8080:8080"
volumes:
- conduktor_data:/var/conduktor
healthcheck:
test: curl -f http://localhost:8080/platform/api/modules/health/live || exit 1
interval: 10s
start_period: 10s
timeout: 5s
retries: 3
environment:
CDK_DATABASE_URL: "postgresql://conduktor:change_me@postgresql:5432/conduktor"
CDK_LICENSE: ""
CDK_ORGANIZATION_NAME: "demo"
CDK_ADMIN_EMAIL: "admin@company.io"
CDK_ADMIN_PASSWORD: "admin"
volumes:
pg_data: {}
conduktor_data: {}
```
### Container user and permissions
Console is running as a non-root user `conduktor-platform` with UID `10001` and GID `0`. All files inside the container volume `/var/conduktor` are owned by `conduktor-platform` user.
## Configure memory usage
We rely on container *CGroups limits* and use *up to 80%* of the container memory limit for JVM max heap size.
```bash theme={null}
-XX:+UseContainerSupport -XX:MaxRAMPercentage=80
```
You only need to care about the limits that you set on your container.
```yaml theme={null}
# Values.yaml
...
platform:
resources:
limits:
memory: 8Gi
...
```
```yaml theme={null}
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
...
template:
spec:
containers:
- name: console
image: conduktor/conduktor-console
resources:
limits:
memory: 8G
...
```
```yaml theme={null}
# docker-compose.yaml
...
conduktor-console:
image: conduktor/conduktor-console
deploy:
resources:
limits:
memory: 8G
...
```
## Configure SSL or TLS
Depending on the environment, Conduktor might need to access external services (such as Kafka clusters, SSO servers, databases or object storage) that require a custom certificate for SSL/TLS communication.
You can configure this using:
* Console UI (recommended) - you can manage your certificates in a dedicated screen and configure SSL authentication from the broker setup wizard.
* volume mount - this method is only required if you have LDAPS. Do not use it for Kafka or Kafka components.
| | Kafka clusters | Schema registry / Kafka Connect | LDAPS, OIDC |
| ------------------------------ | -------------- | ------------------------------- | ------------- |
| SSL to secure data in transit | UI | UI | UI |
| SSL to authenticate the client | UI | UI | Not supported |
### Use the Conduktor certificate store
This option is recommended for Kafka, Kafka Connect and Schema Registry connections.
You can import and parse the certificates as text or files. The supported formats are:
* .crt
* .pem
* .jks
* .p12
#### Upload certificates
You can add cluster configurations from **Settings** > **Clusters** page. When you add the bootstrap server to your configuration, a check will be made to validate if the certificate is issued by a valid authority.
If the response indicates the certificate is not issued by a valid authority, you have two options:
* **Skip SSL Check**: This will skip validation of the SSL certificate on your server. This is an easy option for development environments with self-signed certificates
* **Upload Certificate**: This option will enable you to upload the certificate (`.crt`, `.pem`, `.jks` or `.p12` files), or paste the certificate as text
Upon uploading the certificate, you should then see the green icon indicating the **connection is secure**.
#### Add truststores
You can also manage organization truststores using the **Settings** > **Certificates** page. Simply add all of your certificates by uploading them or pasting them as text. In doing this, the SSL context will be derived when you configure Kafka, Kafka Connect and Schema Registry connections.
#### Mount custom truststore
This option is recommended for SSO, DB or other external services requiring SSL/TLS communication.
Conduktor supports SSL/TLS connections using Java truststore.
#### Create TrustStore (JKS) from certificate in PEM format
If you already have a truststore, you can ignore this step.
You need a `keytool` program that is usually packaged on JDK distributions and a certificate in PEM format (`.pem` or `.crt`).
```bash theme={null}
keytool \
-importcert \
-noprompt \
-trustcacerts \
-keystore ./truststore.jks \ # Output truststore jks file
-alias "my-domain.com" \ # Certificate alias inside the truststore (usually the certificate subject)
-file ./my-certificate-file.pem \ # Input certificate file
-storepass changeit \ # Truststore password
-storetype JKS
```
#### Configure custom truststore via Conduktor Console
Mount the truststore file into the `conduktor-console` container and pass the correct environment variables for locating truststore file inside the container (and password, if needed).
If the truststore file is `truststore.jks` with password `changeit`, mount truststore file into `/opt/conduktor/certs/truststore.jks` inside the container.
If run from Docker :
```bash theme={null}
docker run --rm \
--mount "type=bind,source=$PWD/truststore.jks,target=/opt/conduktor/certs/truststore.jks" \
-e CDK_SSL_TRUSTSTORE_PATH="/opt/conduktor/certs/truststore.jks" \
-e CDK_SSL_TRUSTSTORE_PASSWORD="changeit" \
conduktor/conduktor-console
```
From docker-compose :
```yaml theme={null}
services:
conduktor-console:
image: conduktor/conduktor-console
ports:
- 8080:8080
volumes:
- type: bind
source: ./truststore.jks
target: /opt/conduktor/certs/truststore.jks
read_only: true
environment:
CDK_SSL_TRUSTSTORE_PATH: '/opt/conduktor/certs/truststore.jks'
CDK_SSL_TRUSTSTORE_PASSWORD: 'changeit'
```
#### Client certificate authentication
This option is recommended for mTLS.
This mechanism uses TLS protocol to authenticate the client. Also known as:
* Mutual SSL, Mutual TLS, mTLS
* Two-Way SSL, SSL Certificate Authentication
* Digital Certificate Authentication, Public Key Infrastructure (PKI) Authentication
#### Use the UI (keystore method)
Use the keystore file from your Kafka admin or provider (in **.jks** or **.p12** format).
Click the "Import from keystore" button to select a keystore file from your filesystem.
Fill in the required keystore password and key password and click "Import".
You'll get back to the cluster screen with the content of your keystore extracted into Access key and Access certificate.
#### Use the UI (Access key and access certificate method)
Your Kafka admin or your Kafka provider gave you 2 files for authentication.
* An Access key (`.key` file)
* An Access certificate (`.pem` or `.crt` file)
Here's an example with Aiven:
You can paste the contents of the two files into Conduktor or [import from keystore](#use-the-ui-keystore-method).
#### Use volume mount
You can mount the keystore file in the `conduktor-console` image:
```yaml theme={null}
services:
conduktor-console:
image: conduktor/conduktor-console
ports:
- 8080:8080
volumes:
- type: bind
source: ./keystore.jks
target: /opt/conduktor/certs/keystore.jks
read_only: true
```
Then from the UI, choose the SSL Authentication method **Keystore file is mounted on the volume** and fill in the required fields
## Configure Postgres database
Conduktor Console **requires a Postgres database to store its state**.
### Postgres requirements
* Postgres version 13 or higher
* Provided connection role should have grant `ALL PRIVILEGES` on the configured database. Console should be able to create/update/delete schemas and tables on the database.
**Minimum specifications for small or testing environments only:**
* 1-2 vCPU, 1 GB of RAM, and 10 GB of disk
For **production deployments**, please refer to our [comprehensive PostgreSQL Sizing Guide](/guide/conduktor-in-production/deploy-artifacts/deploy-console/postgres-sizing) which provides tier-based recommendations for AWS, GCP, and Azure to avoid performance issues like IOPS throttling, CPU spikes, and memory exhaustion.
If you want to use AWS RDS or AWS Aurora as a database with Console, consider the following: Console will not work with all PostgreSQL engines within RDS, it will only work with engine versions 14.8+ / 15.3+ (other versions are not fully supported).
### Database configuration properties
* `database` : is a key/value configuration consisting of:
* `database.url` : database connection url in the format `[jdbc:]postgresql://[user[:password]@][[netloc][:port],...][/dbname][?param1=value1&...]`
* `database.hosts[].host` : Postgresql server hosts name
* `database.hosts[].port` : Postgresql server ports
* `database.host` : Postgresql server host name (Deprecated. Use `database.hosts` instead)
* `database.port` : Postgresql server port (Deprecated. Use `database.hosts` instead)
* `database.name` : Database name
* `database.username` : Database login role
* `database.password` : Database login password
* `database.connection_timeout` : Connection timeout option in seconds
#### URL format
Console supports both, the standard [PostgreSQL URL](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING-URIS) and [JDBC PostgreSQL](https://jdbc.postgresql.org/documentation/use/#connecting-to-the-database) .
Connection username and password can be provided in the URL as basic authentication or as parameters.
```yaml theme={null}
database:
url: 'jdbc:postgresql://user:password@host:5432/database' # or 'postgresql://host:5432/database?user=user&password=password'
```
### SSL support
By default, Console will try to connect to the database using SSL mode `prefer`. We plan to make this configurable in the future along with database certificate.
### Setup
There are several options available when configuring an external database:
1. From a single connection URL
* With the `CDK_DATABASE_URL` environment variable.
* With the `database.url` configuration field.
In either case, this connection url is using a standard PostgreSQL url in the format `[jdbc:]postgresql://[user[:password]@][[netloc][:port],...][/dbname][?param1=value1&...]`
2. From decomposed configuration fields
* With the `CDK_DATABASE_*` env vars.
* With the `database.*` on configuration file.
```yaml theme={null}
database:
host: 'host'
port: 5432
name: 'database'
username: 'user'
password: 'password'
connection_timeout: 30 # in seconds
```
#### Example
```shell theme={null}
docker run --rm \
-p "8080:8080" \
-e CDK_DATABASE_URL="postgresql://user:password@host:5432/database" \
-e LICENSE_KEY="" \
conduktor/conduktor-console:latest
```
* If all connection URLs and decomposed configuration fields are provided, **the decomposed configuration fields take priority**.
* If an invalid connection URL or a mandatory configuration field (`host`, `username` or `name`) is missing, Conduktor will fail gracefully with a meaningful error message.
* **Before Console v1.2.0**, the `EMBEDDED_POSTGRES=false` was mandatory to enable external Postgresql configuration.
### Multi-host configuration
If you have a multi-host setup, you can configure the database connection with a list of hosts. Conduktor uses a PostgreSQL JDBC driver to connect to the database that supports [multiple hosts in the connection url](https://jdbc.postgresql.org/documentation/use/#connection-fail-over) .
To configure a multi-host setup, you can use the `database.url` configuration field with a list of hosts separated by commas:
```yaml theme={null}
database:
url: 'jdbc:postgresql://user:password@host1:5432,host2:5432/database'
```
or with decomposed configuration fields:
```yaml theme={null}
database:
hosts:
- host: 'host1'
port: 5432
- host: 'host2'
port: 5432
name: 'database'
username: 'user'
password: 'password'
connection_timeout: 30 # in seconds
```
You can also provide [JDBC connection parameter](https://jdbc.postgresql.org/documentation/use/#connection-parameters) `targetServerType` to specify the target server type for the connection:
```yaml theme={null}
database:
url: 'jdbc:postgresql://user:password@host1:5432,host2:5432/database?targetServerType=primary'
```
Nearly all `targetServerType` are supported: `any`, `primary`, `master`, `slave`, `secondary`, `preferSlave`, `preferSecondary` and `preferPrimary`.
## Debug Console
To troubleshoot Console:
1. Verify that Console is up and running.
2. Check the logs and send them to our support team if necessary.
### 1. Verify that Conduktor is up and running
First, verify that all the components are running.
```bash title="Get containers status" theme={null}
docker ps
```
```txt title="Output" theme={null}
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
conduktor-console conduktor/conduktor-console:1.21.0 "/__cacert_entrypoin…" conduktor-console 10 minutes ago Up 9 minutes (healthy) 0.0.0.0:8080->8080/tcp
conduktor-monitoring conduktor/conduktor-console-cortex:1.21.0 "/opt/conduktor/scri…" conduktor-monitoring 10 minutes ago Up 10 minutes (healthy) 0.0.0.0:9009->9009/tcp, 0.0.0.0:9090->9090/tcp
postgres postgres:15.1 "docker-entrypoint.s…" postgres 10 minutes ago Up 10 minutes 0.0.0.0:5432->5432/tcp
```
If you're using an external Kafka installation and external database, you will only need to verify that the `conduktor-console` container is showing `healthy` as the `STATUS`.
If Console is showing an "exited" status, check the Docker logs by running the command (with the appropriate container name):
```bash title="Get container logs" theme={null}
docker logs conduktor-console
```
You can save these logs in a file:
```bash title="Store logs in a file" theme={null}
docker logs conduktor-console >& docker-logs-output.txt
```
To get the status of the Conduktor Console pod in Kubernetes, you can run the following command (with the correct namespace, if any):
```bash title="Get containers status" theme={null}
kubectl get pod --namespace conduktor
```
```txt title="Output" theme={null}
NAME READY STATUS RESTARTS AGE
console-instance-cortex-5d85d5cfb4-qcxhs 1/1 Running 0 2m4s
console-instance-747d5ffc7b-gcpkx 1/1 Running 0 2m4s
```
The pod status is available in the **STATUS** column.
### 2. Get the logs and send them to support
Logs are kept in `/var/conduktor/log`. You can see them using:
```sh title="List log files" theme={null}
ls /var/conduktor/log/
```
```txt title="Output" theme={null}
console-stdout---supervisor-umscgn8w.log proxy proxy-stdout---supervisor-2gim6er7.log supervisord.log
platform_api-stdout---supervisor-cqvwnsqi.log proxy-stderr---supervisor-8i0bjkaz.log startup.log
```
The best here is to simply bring all the logs to your local machine (in PWD) by running:
```sh theme={null}
docker compose cp conduktor-console:/var/conduktor/log .
```
Then send these logs to our[support team](https://support.conduktor.io/hc/en-gb/requests/new) . If you've contacted us before, [log into your account and create a ticket](https://conduktor-support.zendesk.com/auth/v2/login/signin?return_to=https%3A%2F%2Fsupport.conduktor.io%2Fhc%2Fen-gb\&theme=hc\&locale=en-gb\&brand_id=15739556902161\&auth_origin=15739556902161%2Ctrue%2Ctrue) .
## Configuration properties and environment variables
For the complete list of variables, see the [Console environment variables reference](/guide/conduktor-in-production/deploy-artifacts/deploy-console/environment-variables).
## Configure HTTP proxy
Specify the proxy settings for Conduktor to use when accessing Internet. The HTTP proxy works for both HTTP and [HTTPS](#configure-https) connection.
There are five properties you can set to specify the proxy that will be used by the HTTP protocol handler:
* `CDK_HTTP_PROXY_HOST`: the host name of the proxy server
* `CDK_HTTP_PROXY_PORT`: the port number. Default value is 80.
* `CDK_HTTP_NON_PROXY_HOSTS`: a list of hosts that should be reached directly, bypassing the proxy. This is a list of patterns separated by `|`. The patterns may start or end with a `*` for wildcards, we do not support `/`. Any host matching one of these patterns will be reached through a direct connection instead of through a proxy.
* `CDK_HTTP_PROXY_USERNAME`: the proxy username
* `CDK_HTTP_PROXY_PASSWORD`: the proxy password
### Example
```yaml theme={null}
services:
conduktor-console:
image: conduktor/conduktor-console
ports:
- 8080:8080
environment:
CDK_HTTP_PROXY_HOST: "proxy.mydomain.com"
CDK_HTTP_PROXY_PORT: 8000
CDK_HTTP_NON_PROXY_HOSTS: "*.mydomain.com"
```
### Configure HTTPS
To configure Conduktor Console to respond to HTTPS requests, you have to define a certificate and a private key.
The server certificate is a public entity that's sent to every client that connects to the server and it should be provided as a PEM file.
Configuration properties are:
* `platform.https.cert.path` or environment variable `CDK_PLATFORM_HTTPS_CERT_PATH`: the path to server certificate file
* `platform.https.key.path` or environment variable `CDK_PLATFORM_HTTPS_KEY_PATH`: the path to server private key file
Both the certificate and private key files have to allow read from user `conduktor-platform` (UID 10001 GID 0) but don't need to be readable system-wide.
#### Sample configuration using docker-compose
In this example, server certificate and key (**server.crt** and **server.key**) are stored in the same directory as the `docker-compose` file.
```yaml theme={null}
services:
conduktor-console:
image: conduktor/conduktor-console
ports:
- 8080:8080
volumes:
- type: bind
source: ./server.crt
target: /opt/conduktor/certs/server.crt
read_only: true
- type: bind
source: ./server.key
target: /opt/conduktor/certs/server.key
read_only: true
environment:
CDK_PLATFORM_HTTPS_CERT_PATH: '/opt/conduktor/certs/server.crt'
CDK_PLATFORM_HTTPS_KEY_PATH: '/opt/conduktor/certs/server.key'
```
If the monitoring image `conduktor/conduktor-console-cortex` is running as well, you have to provide the CA public certificate to the monitoring image to allow metrics scraping on HTTPS.
```yaml theme={null}
services:
conduktor-console:
image: conduktor/conduktor-console
ports:
- 8080:8080
volumes:
- type: bind
source: ./server.crt
target: /opt/conduktor/certs/server.crt
read_only: true
- type: bind
source: ./server.key
target: /opt/conduktor/certs/server.key
read_only: true
environment:
# HTTPS configuration
CDK_PLATFORM_HTTPS_CERT_PATH: '/opt/conduktor/certs/server.crt'
CDK_PLATFORM_HTTPS_KEY_PATH: '/opt/conduktor/certs/server.key'
# monitoring configuration
CDK_MONITORING_CORTEX-URL: http://conduktor-monitoring:9009/
CDK_MONITORING_ALERT-MANAGER-URL: http://conduktor-monitoring:9009/
CDK_MONITORING_CALLBACK-URL: https://conduktor-console:8080/monitoring/api/
CDK_MONITORING_NOTIFICATIONS-CALLBACK-URL: http://localhost:8080
conduktor-monitoring:
image: conduktor/conduktor-console-cortex
volumes:
- type: bind
source: ./server.crt
target: /opt/conduktor/certs/server.crt
read_only: true
environment:
CDK_CONSOLE-URL: "https://conduktor-console:8080"
CDK_SCRAPER_SKIPSSLCHECK: "false" # can be set to true if you don't want to check the certificate
CDK_SCRAPER_CAFILE: "/opt/conduktor/certs/server.crt"
```
# Conduktor Console environment variables reference
Source: https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-console/environment-variables
Complete reference for Conduktor Console environment variables: logging, database, authentication, monitoring.
## Docker image environment variables
These are container-level variables for the Console Docker image, distinct from the `platform-config` properties documented below.
| Environment variable | Description | Default value | Since Console version |
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- | --------------------- |
| **Logs** | | | |
| `CDK_DEBUG` | Enable Console debug logs (equivalent to `CDK_ROOT_LOG_LEVEL=DEBUG`) | `false` | 1.0.0 |
| `CDK_ROOT_LOG_LEVEL` | Set the Console global log level (one of `DEBUG`, `INFO`, `WARN`, `ERROR`) | `INFO` | 1.11.0 |
| `CDK_ROOT_LOG_FORMAT` | Set logs format (one of `TEXT`, `JSON`) | `TEXT` | 1.26.0 |
| `CDK_ROOT_LOG_COLOR` | Enable ANSI colors in logs | `true` | 1.11.0 |
| `CDK_LOG_TIMEZONE` | Timezone for dates in logs (in Olson timezone ID format, e.g. `Europe/Paris`) | `TZ` environment variable or `UTC` if `TZ` is not defined | 1.28.0 |
| **Proxy settings** | | | |
| `CDK_HTTP_PROXY_HOST` | Proxy hostname | ∅ | 1.10.0 |
| `CDK_HTTP_PROXY_PORT` | Proxy port | `80` | 1.10.0 |
| `CDK_HTTP_NON_PROXY_HOSTS` | List of hosts that should be reached directly, bypassing the proxy. Hosts must be separated by `\|`, end with a `*` for wildcards, and not contain any `/`. | ∅ | 1.10.0 |
| `CDK_HTTP_PROXY_USERNAME` | Proxy username | ∅ | 1.10.0 |
| `CDK_HTTP_PROXY_PASSWORD` | Proxy password | ∅ | 1.10.0 |
| **SSL** | | | |
| `CDK_SSL_TRUSTSTORE_PATH` | Truststore file path used by Console for Kafka, SSO, S3,... clients SSL/TLS verification | ∅ | 1.5.0 |
| `CDK_SSL_TRUSTSTORE_PASSWORD` | Truststore password (optional) | ∅ | 1.5.0 |
| `CDK_SSL_TRUSTSTORE_TYPE` | Truststore type (optional) | `jks` | 1.5.0 |
| `CDK_SSL_DEBUG` | Enable SSL/TLS debug logs | `false` | 1.9.0 |
| **Java** | | | |
| `CDK_GLOBAL_JAVA_OPTS` | Custom JAVA\_OPTS parameters passed to Console | ∅ | 1.10.0 |
| `CONSOLE_MEMORY_OPTS` | Configure Java memory options | `-XX:+UseContainerSupport -XX:MaxRAMPercentage=80` | 1.18.0 |
| **Console** | | | |
| `CDK_LISTENING_PORT` | Console listening port | `8080` | 1.2.0 |
| `CDK_VOLUME_DIR` | Volume directory where Console stores data | `/var/conduktor` | 1.0.2 |
| `CDK_IN_CONF_FILE` | Console configuration file location | `/opt/conduktor/default-platform-config.yaml` | 1.0.2 |
| `CDK_PLUGINS_DIR` | Volume directory for Custom Deserializer plugins | `/opt/conduktor/plugins` | 1.22.0 |
| **Nginx** | | | |
| `PROXY_BUFFER_SIZE` | Tune internal Nginx `proxy_buffer_size` | `8k` | 1.16.0 |
## Configure Console properties
You can configure Console via environment variables or using the YAML file.
Environment variables can be set on the container or imported from a file. When importing from a file, mount the file into the container and provide its path by setting the environment variable `CDK_ENV_FILE`. Use the **.env** file with key value pairs.
```bash theme={null}
MY_ENV_VAR1=value
MY_ENV_VAR2=otherValue
```
The logs will confirm, `Sourcing environment variables from $CDK_ENV_FILE`, or warn if set and the file is not found
```bash theme={null}
Warning: CDK_ENV_FILE is set but the file does not exist or is not readable.
```
Lists start at index 0 and are provided using the `_idx_` syntax.
If you set *both* the environment variable and a YAML value for a specific field, **the environment variable will take precedence**.
### YAML property cases
YAML configuration supports multiple case formats (`camelCase`/`kebab-case`/`lowercase`) for property fragments such as:
* `clusters[].schemaRegistry.ignoreUntrustedCertificate`
* `clusters[].schema-registry.ignore-untrusted-certificate`
* `clusters[].schemaregistry.ignoreuntrustedcertificate`
All are valid and equivalent in YAML.
### Environment variable conversion
At startup, Conduktor Console will merge environment variables and YAML based configuration files into one unified configuration. The conversion rules are:
* Filter for environment variables that start with `CDK_`
* Remove the `CDK_` prefix
* Convert the variable name to lower case
* Replace `_` with `.` for nested properties
* Replace `_[0-9]+_` with `[0-9].` for list properties. Lists start at index 0.
For example, the environment variable `CDK_DATABASE_URL` will be converted to `database.url`; `CDK_SSO_OAUTH2_0_OPENID_ISSUER` will be converted to `sso.oauth2[0].openid.issuer`.
The YAML equivalent would be:
```yaml theme={null}
database:
url: "..."
sso:
oauth2:
- openid:
issuer: "..."
```
When converting environment variables to YAML configuration, environment variables in `UPPER-KEBAB-CASE` will be converted to `kebab-case` in the YAML configuration.
#### Conversion edge cases
Because of YAML multiple case formats support, the conversion rules have some edge cases when trying to mix environment variables and YAML configuration.
Extra rules when mixing environment variables and YAML configuration:
* Don't use `camelCase` in YAML configuration. Use `kebab-case` or `lowercase`
* Stick to one compatible case format for a given property fragment using the following compatibility matrix
Compatibility matrix:
| YAML\Environment | `UPPER-KEBAB-CASE` | `UPPERCASE` |
| ---------------- | ------------------ | ----------- |
| `kebab-case` | ✅ | 🚫 |
| `lowercase` | 🚫 | ✅ |
| `camelCase` | 🚫 | 🚫 |
For example, the `CDK_CLUSTERS_0_SCHEMAREGISTRY_IGNOREUNTRUSTEDCERTIFICATE` environment variable :
```yaml theme={null}
# Is equivalent to and compatible with
clusters:
- schemaregistry:
ignoreuntrustedcertificate: true
# but not with
clusters:
- schema-registry:
ignore-untrusted-certificate: true
```
And conversely, for `CDK_CLUSTERS_0_SCHEMA-REGISTRY_IGNORE-UNTRUSTED-CERTIFICATE` environment variable.
That's why camelCase is not recommended in YAML configuration when mixing with environment variables.
### Shell expansion in the YAML configuration file
Console supports shell expansion for environment variables and home tilde `~`. This is useful if you have to use custom environment variables in your configuration.
For example, you can use the following syntax:
```yaml title="YAML configuration file" theme={null}
database:
url: "jdbc:postgresql://${DB_LOGIN}:${DB_PWD}@${DB_HOST}:${DB_PORT:-5432}/${DB_NAME}"
```
with the following environment variables:
| Environment variable | Value |
| -------------------- | ----------- |
| `DB_LOGIN` | `usr` |
| `DB_PWD` | `pwd` |
| `DB_HOST` | `some_host` |
| `DB_NAME` | `cdk` |
This will be expanded to:
```yaml title="Expanded configuration" theme={null}
database:
url: "jdbc:postgresql://usr:pwd@some_host:5432/cdk"
```
If you want to escape the shell expansion, you can use the following syntax: `$$`. For example, if you want `admin.password` to be `secret$123`, set `admin.password: "secret$$123"`.
### Support for \*\_FILE environment variables
When an environment variable ending with `_FILE` is set to a file path, its corresponding unprefixed environment variable will be replaced with the content of that file.
For instance, if you set `CDK_LICENSE_FILE=/run/secrets/license`, the value of `CDK_LICENSE` will be overridden by the content of the file located at `/run/secrets/license`.
The `CDK_IN_CONF_FILE` is not supported.
## Core properties
### Global properties
| Property | Description | Environment variable | Mandatory | Type | Default |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------- | --------- | ------- | ------------- |
| `organization.name` | Your organization's name | `CDK_ORGANIZATION_NAME` | false | string | `"Conduktor"` |
| `admin.email` | Your organization's root administrator account email | `CDK_ADMIN_EMAIL` | true | string | ∅ |
| `admin.password` | Your organization's root administrator account password. Must be at least 8 characters in length, and include at least 1 uppercase letter, 1 lowercase letter, 1 number, and 1 special symbol | `CDK_ADMIN_PASSWORD` | true | string | ∅ |
| `license` | Enterprise license key. If not provided, fallback to free plan. | `CDK_LICENSE` or `LICENSE_KEY` | false | string | ∅ |
| `platform.external.url` | Force Console external URL. Useful for SSO callback URL when using a reverse proxy. Can include a context path (e.g., `https://mycompany.com/conduktor`) for context path deployments (since 1.41.0). By default, Console will try to guess it automatically using X-Forwarded-\* headers coming from upstream reverse proxy. | `CDK_PLATFORM_EXTERNAL_URL` | false | string | ∅ |
| `platform.https.cert.path` | Path to the SSL certificate file | `CDK_PLATFORM_HTTPS_CERT_PATH` | false | string | ∅ |
| `platform.https.key.path` | Path to the SSL private key file | `CDK_PLATFORM_HTTPS_KEY_PATH` | false | string | ∅ |
| `platform.http.requestTimeoutSeconds` | HTTP request timeout of the Console server, in seconds. Increase it to accommodate long-running requests behind slow upstream systems, or lower it to fail faster in environments with tight service-level objectives. Since 1.46.2. | `CDK_PLATFORM_HTTP_REQUESTTIMEOUTSECONDS` | false | long | `60` |
| `enable_product_metrics` | To improve Conduktor Console and support you proactively, we collect product usage metrics. [Find out what product telemetry contains](/guide/support/product-telemetry). With an enterprise license, if set to `false`, this configuration disables all of our metrics collection. With a community license this property cannot be disabled and product metrics will always be collected. | `CDK_ENABLE_PRODUCT_METRICS` | false | boolean | `true` |
| `enable_metadata_governance` | Enforce a dedicated permission for editing labels and descriptions on Kafka resources. When `true`, the **Manage metadata** permission is required to edit a resource's labels and description. See [Govern who can edit labels and descriptions](/guide/conduktor-in-production/admin/set-up-rbac#govern-who-can-edit-labels-and-descriptions). | `CDK_ENABLE_METADATA_GOVERNANCE` | false | boolean | `false` |
### Database properties
Console uses PostgreSQL to persist application state and operational data. The database maintains user accounts and their group memberships, along with associated permissions and role-based access controls.
It also stores your cluster definitions and connection details, indexed metadata from connected clusters (including topics, schemas, and connectors), alert configurations and audit log entries that track system activity.
| Property | Description | Environment variable | Mandatory | Type | Default |
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------- | --------- | ------ | ------- |
| `database.url` | External PostgreSQL configuration URL in format `[jdbc:]postgresql://[user[:password]@][[netloc][:port],...][/dbname][?param1=value1&...]` | `CDK_DATABASE_URL` | false | string | ∅ |
| `database.hosts[].host` | External PostgreSQL servers hostname | `CDK_DATABASE_HOSTS_0_HOST` | false | string | ∅ |
| `database.hosts[].port` | External PostgreSQL servers port | `CDK_DATABASE_HOSTS_0_PORT` | false | int | ∅ |
| `database.host` | External PostgreSQL server hostname (Deprecated, use `database.hosts` instead) | `CDK_DATABASE_HOST` | false | string | ∅ |
| `database.port` | External PostgreSQL server port (Deprecated, use `database.hosts` instead) | `CDK_DATABASE_PORT` | false | int | ∅ |
| `database.name` | External PostgreSQL database name | `CDK_DATABASE_NAME` | false | string | ∅ |
| `database.username` | External PostgreSQL login role | `CDK_DATABASE_USERNAME` | false | string | ∅ |
| `database.password` | External PostgreSQL login password | `CDK_DATABASE_PASSWORD` | false | string | ∅ |
| `database.connection_timeout` | External PostgreSQL connection timeout in seconds | `CDK_DATABASE_CONNECTIONTIMEOUT` | false | int | ∅ |
| `database.options` | External PostgreSQL options (available via YAML configuration only) | | false | map | ∅ |
| `CDK_DATABASE_CONNECTION_POOL_SIZE` | External PostgreSQL connection pool size (via environment variable only) | `CDK_DATABASE_CONNECTION_POOL_SIZE` | false | int | 15 |
Example configuration for PostgreSQL connection options:
```yaml theme={null}
database:
hosts:
host: localhost
port: 5432
name: postgres
username: postgres
password: password
options:
socketTimeout: "30"
tcpKeepAlive: "true"
keepaliveTime: "30000"
application_name: "myapp"
```
To pass options via environment variable, use `CDK_DATABASE_URL`. The equivalent of the above configuration is:
```bash theme={null}
CDK_DATABASE_URL="jdbc:postgresql://postgres:password@localhost:5432/database?socketTimeout=30&tcpKeepAlive=true&keepaliveTime=30000&application_name=myapp"
```
Console supports all valid [JDBC PostgreSQL configuration options](https://jdbc.postgresql.org/documentation/use/). For [Hikari configuration](https://github.com/brettwooldridge/HikariCP?tab=readme-ov-file#frequently-used), only the following options are supported:
* `keepaliveTime`
* `connectionTimeout`
* `idleTimeout`
* `maxLifetime`
### Session lifetime properties
| Property | Description | Environment variable | Mandatory | Type | Default value |
| ---------------------- | ----------------------------------------------------------------------------------------------------- | -------------------------- | --------- | ---- | ------------- |
| `auth.sessionLifetime` | Max session lifetime in seconds | `CDK_AUTH_SESSIONLIFETIME` | false | int | `259200` |
| `auth.idleTimeout` | Max idle session time in seconds (access token lifetime). Should be lower than `auth.sessionLifetime` | `CDK_AUTH_IDLETIMEOUT` | false | int | `259200` |
### Local user properties
Optional local accounts list, used to log into Console.
| Property | Description | Environment variable | Mandatory | Type | Default value |
| ----------------------------- | ------------- | -------------------------------- | --------- | ------ | ---------------------- |
| `auth.local-users[].email` | User login | `CDK_AUTH_LOCALUSERS_0_EMAIL` | true | string | `"admin@conduktor.io"` |
| `auth.local-users[].password` | User password | `CDK_AUTH_LOCALUSERS_0_PASSWORD` | true | string | `"admin"` |
## Monitoring and observability properties
### Monitoring properties
We provide a pre-configured monitoring solution based on Cortex (`conduktor/conduktor-console-cortex`). It's a custom implementation of Prometheus used in several production systems including Amazon Managed Service for Prometheus (AMP).
If using your own solution, set the `monitoring.backend` property. We support **Cortex**, **Mimir** and **Prometheus**.
You can choose to *not* deploy any monitoring solution. If none is deployed, you won't be able to see the monitoring graphs or configure alerts.
Depending on the chosen solution, you'll need to provide the corresponding URL and other properties. The mandatory column indicates whether the property is required for the chosen backend.
For example, if you choose `mimir`, you have to provide the `monitoring.mimir-url` property but if you select `conduktor`, you have to provide the `monitoring.cortex-url`.
If an alert manager URL isn't provided, Console will use its built-in alert system.
| Property | Description | Environment variable | Mandatory | Type | Default |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------- | ----------------------------------------------- | ----------------------------------------------------------- | ------------- |
| `monitoring.backend` | Type of monitoring solution installed | `CDK_MONITORING_BACKEND` | false | string one of: `conduktor`, `cortex`, `mimir`, `prometheus` | `conduktor` |
| `monitoring.cortex-url` | Cortex search query URL (default port is 9009 for `conduktor` backend) | `CDK_MONITORING_CORTEXURL` | if backend is `conduktor` or `cortex` | string | ∅ |
| `monitoring.mimir-url` | Mimir URL | `CDK_MONITORING_MIMIRURL` | If backend is `mimir` | string | ∅ |
| `monitoring.prometheus-url` | Prometheus URL | `CDK_MONITORING_PROMETHEUSURL` | If backend is `prometheus` | string | ∅ |
| `monitoring.prometheus-http-prefix` | Path to prometheus API for Cortex and Mimir | `CDK_MONITORING_PROMETHEUSHTTPPREFIX` | false | string | `/prometheus` |
| `monitoring.tenant-id` | Value of the X-Scope-OrgId header to authenticate to Cortex or Mimir when multi-tenancy is enabled | `CDK_MONITORING_TENANTID` | false | string | ∅ |
| `monitoring.alert-manager-url` | External Alert Manager URL (default port is 9009 for `conduktor` backend, [9010 before Console 1.46.0](/guide/release-notes#cortex-alert-manager-port-v1-46-0)) | `CDK_MONITORING_ALERTMANAGERURL` | false | string | ∅ |
| `monitoring.callback-url` | Console API | `CDK_MONITORING_CALLBACKURL` | If `monitoring.alert-manager-url` is configured | string | ∅ |
| `monitoring.rules-namespace` | Namespace for rules configured in Prometheus | `CDK_MONITORING_RULESNAMESPACE` | false | string | `conduktor` |
| `monitoring.notifications-callback-url` | Where the Slack notification should redirect | `CDK_MONITORING_NOTIFICATIONSCALLBACKURL` | true | string | ∅ |
| `monitoring.clusters-refresh-interval` | Refresh rate in seconds for metrics | `CDK_MONITORING_CLUSTERREFRESHINTERVAL` | false | int | `60` |
| `monitoring.use-aggregated-metrics` | Defines whether use the new aggregated metrics in the Console graphs | `CDK_MONITORING_USEAGGREGATEDMETRICS` | false | Boolean | `false` |
| `monitoring.enable-non-aggregated-metrics` | Toggles the collection of obsolete granular metrics | `CDK_MONITORING_ENABLENONAGGREGATEDMETRICS` | false | Boolean | `true` |
| `monitoring.repeat-alert-frequency-minutes` | Frequency in minutes of the repetition of firing alerts (built-in alert system) | `CDK_MONITORING_REPEATALERTFREQUENCYMINUTES` | false | int | `60` |
| `monitoring.check-alert-frequency-minutes` | Frequency in minutes of the verification of alert status (built-in alert system) | `CDK_MONITORING_CHECKALERTFREQUENCYMINUTES` | false | int | `1` |
| `monitoring.check-alert-parallelism` | How many alerts are verified in parallel (built-in alert system) | `CDK_MONITORING_CHECKALERTPARALLELISM` | false | int | `5` |
| `monitoring.prometheus_auth.username` | Username for basic authentication to Prometheus/Cortex/Mimir | `CDK_MONITORING_PROMETHEUSAUTH_USERNAME` | false | string | ∅ |
| `monitoring.prometheus_auth.password` | Password for basic authentication to Prometheus/Cortex/Mimir | `CDK_MONITORING_PROMETHEUSAUTH_PASSWORD` | false | string | ∅ |
| `monitoring.basic-auth.email` | Basic auth username for monitoring backend to scrape Console metrics. When using the bundled Cortex component, configure matching credentials using `CDK_SCRAPER_BASICAUTH_EMAIL`. | `CDK_MONITORING_BASICAUTH_EMAIL` | false | string | ∅ |
| `monitoring.basic-auth.password` | Basic auth password for monitoring backend to scrape Console metrics. When using the bundled Cortex component, configure matching credentials using `CDK_SCRAPER_BASICAUTH_PASSWORD`. Must contain at least 8 characters, including one uppercase letter, one lowercase letter, one number and one special symbol. | `CDK_MONITORING_BASICAUTH_PASSWORD` | false | string | ∅ |
`monitoring.use-aggregated-metrics` and `monitoring.enable-non-aggregated-metrics` are temporary flags to help you transition to the new metrics collection system. They will be removed in a future release.
Swap their default value if you experience performance issues when Console is connected with large Kafka clusters:
```yaml theme={null}
CDK_MONITORING_USEAGGREGATEDMETRICS: true
CDK_MONITORING_ENABLENONAGGREGATEDMETRICS: false
```
## Authentication (SSO) properties
| Property | Description | Environment variable | Mandatory | Type | Default |
| -------------------------------- | ------------------------------------------------------------------------ | ------------------------------------ | --------- | ------- | ------- |
| `sso.ignoreUntrustedCertificate` | Disable SSL checks | `CDK_SSO_IGNOREUNTRUSTEDCERTIFICATE` | false | boolean | `false` |
| `sso.trustedCertificates` | SSL public certificates for SSO authentication (LDAPS and OAuth2) as PEM | `CDK_SSO_TRUSTEDCERTIFICATES` | false | string | ∅ |
### LDAP properties
| Property | Description | Environment variable | Mandatory | Type | Default |
| ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | --------- | ------------ | -------------------- |
| `sso.ldap[].name` | Ldap connection name | `CDK_SSO_LDAP_0_NAME` | true | string | ∅ |
| `sso.ldap[].server` | Ldap server host and port | `CDK_SSO_LDAP_0_SERVER` | true | string | ∅ |
| `sso.ldap[].managerDn` | Sets the manager DN | `CDK_SSO_LDAP_0_MANAGERDN` | true | string | ∅ |
| `sso.ldap[].managerPassword` | Sets the manager password | `CDK_SSO_LDAP_0_MANAGERPASSWORD` | true | string | ∅ |
| `sso.ldap[].search-subtree` | Sets if the subtree should be searched. | `CDK_SSO_LDAP_0_SEARCHSUBTREE` | false | boolean | `true` |
| `sso.ldap[].search-base` | Sets the base DN to search. | `CDK_SSO_LDAP_0_SEARCHBASE` | true | string | ∅ |
| `sso.ldap[].search-filter` | Sets the search filter. By default, the filter is set to `(uid={0})` for users using class type `InetOrgPerson`. | `CDK_SSO_LDAP_0_SEARCHFILTER` | false | string | `"(uid={0})"` |
| `sso.ldap[].search-attributes` | Sets the attributes list to return. By default, all attributes are returned. Search for `uid`, `cn`, `mail`, `email`, `givenName`, `sn`, `displayName` attributes to map into user token. | `CDK_SSO_LDAP_0_SEARCHATTRIBUTES` | false | string array | `[]` |
| `sso.ldap[].groups-enabled` | Sets if group search is enabled. | `CDK_SSO_LDAP_0_GROUPSENABLED` | false | boolean | `false` |
| `sso.ldap[].groups-subtree` | Sets if the subtree should be searched. | `CDK_SSO_LDAP_0_GROUPSSUBTREE` | false | boolean | `true` |
| `sso.ldap[].groups-base` | Sets the base DN to search from. | `CDK_SSO_LDAP_0_GROUPSBASE` | true | string | ∅ |
| `sso.ldap[].groups-filter` | Sets the group search filter. If using group class type `GroupOfUniqueNames` use the filter `"uniqueMember={0}"`. For group class `GroupOfNames` use `"member={0}"`. | `CDK_SSO_LDAP_0_GROUPSFILTER` | false | string | `"uniquemember={0}"` |
| `sso.ldap[].groups-filter-attribute` | Sets the name of the user attribute to bind to the group search filter. Defaults to the user’s DN. | `CDK_SSO_LDAP_0_GROUPSFILTERATTRIBUTE` | false | string | ∅ |
| `sso.ldap[].groups-attribute` | Sets the group attribute name. Defaults to `cn`. | `CDK_SSO_LDAP_0_GROUPSATTRIBUTE` | false | string | `"cn"` |
| `sso.ldap[].properties` | Additional properties that will be passed to identity provider context. | `CDK_SSO_LDAP_0_PROPERTIES` | false | dictionary | ∅ |
### OAuth2 properties
| Property | Description | Environment variable | Mandatory | Type | Default | | |
| --------------------------------------- | -------------------------------------------------------------------- | ---------------------------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ------- | - | ------- |
| `sso.oauth2[].name` | OAuth2 connection name | `CDK_SSO_OAUTH2_0_NAME` | true | string | ∅ | | |
| `sso.oauth2[].default` | Use as default | `CDK_SSO_OAUTH2_0_DEFAULT` | true | boolean | ∅ | | |
| `sso.oauth2[].client-id` | OAuth2 client ID | `CDK_SSO_OAUTH2_0_CLIENTID` | true | string | ∅ | | |
| `sso.oauth2[].client-secret` | OAuth2 client secret | `CDK_SSO_OAUTH2_0_CLIENTSECRET` | true | string | ∅ | | |
| `sso.oauth2[].openid.issuer` | Issuer to check on token | `CDK_SSO_OAUTH2_0_OPENID_ISSUER` | true | string | ∅ | | |
| `sso.oauth2[].scopes` | Scopes to be requested in the client credentials request | `CDK_SSO_OAUTH2_0_SCOPES` | true | string | `[]` | | |
| `sso.oauth2[].groups-claim` | Group attribute from your identity provider | `CDK_SSO_OAUTH2_0_GROUPSCLAIM` | false | string | ∅ | | |
| `sso.oauth2[].username-claim` | Username attribute from your identity provider | `CDK_SSO_OAUTH2_0_USERNAMECLAIM` | false | string | `email` | | |
| `sso.oauth2[].email-claim` | Email attribute from your identity provider | `CDK_SSO_OAUTH2_0_EMAILCLAIM` | false | string | `email` | | `email` |
| `sso.oauth2[].allow-unsigned-id-tokens` | Allow unsigned ID tokens | `CDK_SSO_OAUTH2_0_ALLOWUNSIGNEDIDTOKENS` | false | boolean | false | | |
| `sso.oauth2[].preferred-jws-algorithm` | Configure preferred JWS algorithm | `CDK_SSO_OAUTH2_0_PREFERREDJWSALGORITHM` | false | string one of: "HS256", "HS384", "HS512", "RS256", "RS384", "RS512", "ES256", "ES256K", "ES384", "ES512", "PS256", "PS384", "PS512", "EdDSA" | ∅ | | |
| `sso.oauth2[].pkce-method` | Configure PKCE method | `CDK_SSO_OAUTH2_0_PKCEMETHOD` | false | string one of: "plain", "S256" | ∅ | | |
| `sso.oauth2-logout` | Whether the central identity provider logout should be called or not | `CDK_SSO_OAUTH2LOGOUT` | false | boolean | true | | |
### JWT auth properties
| Property | Description | Environment variable | Mandatory | Type | Default |
| ----------------------------- | --------------------------------------------- | ------------------------------- | --------- | ------ | -------- |
| `sso.jwt-auth.issuer` | Issuer of your identity provider | `CDK_SSO_JWTAUTH_ISSUER` | true | string | ∅ |
| `sso.jwt-auth.username-claim` | Email attribute from your identity provider | `CDK_SSO_JWTAUTH_USERNAMECLAIM` | false | string | `email` |
| `sso.jwt-auth.groups-claim` | Group attribute from your identity provider | `CDK_SSO_JWTAUTH_GROUPSCLAIM` | false | string | `groups` |
| `sso.jwt-auth.api-key-claim` | API key attribute from your identity provider | `CDK_SSO_JWTAUTH_APIKEYCLAIM` | false | string | `apikey` |
## Kafka properties
### Kafka cluster properties
| Property | Description | Environment variable | Mandatory | Type | Default |
| --------------------------------------- | -------------------------------------------------------------- | ------------------------------------------- | --------- | ---------------------------------------- | ------- |
| `clusters[].id` | String used to uniquely identify your Kafka cluster | `CDK_CLUSTERS_0_ID` | true | string | ∅ |
| `clusters[].name` | Alias or user-friendly name for your Kafka cluster | `CDK_CLUSTERS_0_NAME` | true | string | ∅ |
| `clusters[].color` | Attach a color to associate with your cluster in the UI | `CDK_CLUSTERS_0_COLOR` | false | string in hexadecimal format (`#FFFFFF`) | random |
| `clusters[].ignoreUntrustedCertificate` | Skip SSL certificate validation | `CDK_CLUSTERS_0_IGNOREUNTRUSTEDCERTIFICATE` | false | boolean | `false` |
| `clusters[].bootstrapServers` | List of host:port for your Kafka brokers separated by coma `,` | `CDK_CLUSTERS_0_BOOTSTRAPSERVERS` | true | string | ∅ |
| `clusters[].properties` | Any cluster configuration properties | `CDK_CLUSTERS_0_PROPERTIES` | false | string where each line is a property | ∅ |
### Kafka vendor-specific properties
Note that you only need to set the [Kafka cluster properties](#kafka-cluster-properties) to use the core features of Console. For additional benefits though, set the flavor of your cluster. To define that, go to **Settings** > **Clusters** and open the **Provider** tab.
| Property | Description | Environment variable | Mandatory | Type | Default |
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------- | --------- | ------- | ------- |
| `clusters[].kafkaFlavor.type` | Kafka flavor type, one of `Confluent`, `ConfluentPlatform`, `Aiven`, `Gateway` | `CDK_CLUSTERS_0_KAFKAFLAVOR_TYPE` | false | string | ∅ |
| **Flavor is `Confluent`** | Manage Confluent Cloud service accounts, API keys, and ACLs | | | | |
| `clusters[].kafkaFlavor.key` | Confluent Cloud API Key | `CDK_CLUSTERS_0_KAFKAFLAVOR_KEY` | true | string | ∅ |
| `clusters[].kafkaFlavor.secret` | Confluent Cloud API Secret | `CDK_CLUSTERS_0_KAFKAFLAVOR_SECRET` | true | string | ∅ |
| `clusters[].kafkaFlavor.confluentEnvironmentId` | Confluent Environment ID | `CDK_CLUSTERS_0_KAFKAFLAVOR_CONFLUENTENVIRONMENTID` | true | string | ∅ |
| `clusters[].kafkaFlavor.confluentClusterId` | Confluent Cluster ID | `CDK_CLUSTERS_0_KAFKAFLAVOR_CONFLUENTCLUSTERID` | true | string | ∅ |
| **Flavor is `ConfluentPlatform`** | Manage Confluent Platform service accounts and RBAC role bindings via the Metadata Service (MDS) | | | | |
| `clusters[].kafkaFlavor.kafkaClusterId` | Confluent Platform Kafka cluster ID | `CDK_CLUSTERS_0_KAFKAFLAVOR_KAFKACLUSTERID` | true | string | ∅ |
| `clusters[].kafkaFlavor.mdsUrl` | URL of the Confluent Platform Metadata Service (MDS) | `CDK_CLUSTERS_0_KAFKAFLAVOR_MDSURL` | false | string | ∅ |
| `clusters[].kafkaFlavor.authentication.username` | MDS username (LDAP user with `SystemAdmin` privileges) | `CDK_CLUSTERS_0_KAFKAFLAVOR_AUTHENTICATION_USERNAME` | false | string | ∅ |
| `clusters[].kafkaFlavor.authentication.password` | MDS password | `CDK_CLUSTERS_0_KAFKAFLAVOR_AUTHENTICATION_PASSWORD` | false | string | ∅ |
| `clusters[].kafkaFlavor.schemaRegistryClusterId` | Schema Registry cluster ID, required to provision subject role bindings | `CDK_CLUSTERS_0_KAFKAFLAVOR_SCHEMAREGISTRYCLUSTERID` | false | string | ∅ |
| `clusters[].kafkaFlavor.enableRbacRoleBindings` | Manage permissions through Confluent Platform RBAC role bindings instead of Kafka ACLs | `CDK_CLUSTERS_0_KAFKAFLAVOR_ENABLERBACROLEBINDINGS` | false | boolean | `false` |
| `clusters[].kafkaFlavor.ignoreUntrustedCertificate` | Skip SSL certificate validation when contacting MDS | `CDK_CLUSTERS_0_KAFKAFLAVOR_IGNOREUNTRUSTEDCERTIFICATE` | false | boolean | `false` |
| **Flavor is `Aiven`** | Manage Aiven service accounts and ACLs | | | | |
| `clusters[].kafkaFlavor.apiToken` | Aiven API token | `CDK_CLUSTERS_0_KAFKAFLAVOR_APITOKEN` | true | string | ∅ |
| `clusters[].kafkaFlavor.project` | Aiven project | `CDK_CLUSTERS_0_KAFKAFLAVOR_PROJECT` | true | string | ∅ |
| `clusters[].kafkaFlavor.serviceName` | Aiven service name | `CDK_CLUSTERS_0_KAFKAFLAVOR_SERVICENAME` | true | string | ∅ |
| **Flavor is `Gateway`** | Manage Conduktor Gateway interceptors | | | | |
| `clusters[].kafkaFlavor.url` | Gateway API endpoint URL | `CDK_CLUSTERS_0_KAFKAFLAVOR_URL` | true | string | ∅ |
| `clusters[].kafkaFlavor.user` | Gateway API username | `CDK_CLUSTERS_0_KAFKAFLAVOR_USER` | true | string | ∅ |
| `clusters[].kafkaFlavor.password` | Gateway API password | `CDK_CLUSTERS_0_KAFKAFLAVOR_PASSWORD` | true | string | ∅ |
| `clusters[].kafkaFlavor.virtualCluster` | Gateway virtual cluster | `CDK_CLUSTERS_0_KAFKAFLAVOR_VIRTUALCLUSTER` | true | string | ∅ |
### Schema registry properties
| Property | Description | Environment variable | Mandatory | Type | Default |
| ------------------------------------------------------ | -------------------------------------------- | ---------------------------------------------------------- | --------- | ------------------------------------ | ------- |
| `clusters[].schemaRegistry.url` | The schema registry URL | `CDK_CLUSTERS_0_SCHEMAREGISTRY_URL` | true | string | ∅ |
| `clusters[].schemaRegistry.ignoreUntrustedCertificate` | Skip SSL certificate validation | `CDK_CLUSTERS_0_SCHEMAREGISTRY_IGNOREUNTRUSTEDCERTIFICATE` | false | boolean | `false` |
| `clusters[].schemaRegistry.properties` | Any schema registry configuration parameters | `CDK_CLUSTERS_0_SCHEMAREGISTRY_PROPERTIES` | false | string where each line is a property | ∅ |
| **Basic authentication** | | | | | |
| `clusters[].schemaRegistry.security.username` | Basic auth username | `CDK_CLUSTERS_0_SCHEMAREGISTRY_SECURITY_USERNAME` | false | string | ∅ |
| `clusters[].schemaRegistry.security.password` | Basic auth password | `CDK_CLUSTERS_0_SCHEMAREGISTRY_SECURITY_PASSWORD` | false | string | ∅ |
| **Bearer token authentication** | | | | | |
| `clusters[].schemaRegistry.security.token` | Bearer auth token | `CDK_CLUSTERS_0_SCHEMAREGISTRY_SECURITY_TOKEN` | false | string | ∅ |
| **mTLS authentication** | | | | | |
| `clusters[].schemaRegistry.security.key` | Access Key | `CDK_CLUSTERS_0_SCHEMAREGISTRY_SECURITY_KEY` | false | string | ∅ |
| `clusters[].schemaRegistry.security.certificateChain` | Access certificate | `CDK_CLUSTERS_0_SCHEMAREGISTRY_SECURITY_CERTIFICATECHAIN` | false | string | ∅ |
#### Amazon Glue properties
| Property | Description | Environment variable | Mandatory | Type | Default |
| ------------------------------------------------------ | -------------------------------------------------------------------------------- | ---------------------------------------------------------- | --------- | ------ | ------- |
| `clusters[].schemaRegistry.region` | The Glue schema registry region | `CDK_CLUSTERS_0_SCHEMAREGISTRY_REGION` | true | string | ∅ |
| `clusters[].schemaRegistry.registryName` | The Glue schema registry name | `CDK_CLUSTERS_0_SCHEMAREGISTRY_REGISTRYNAME` | false | string | ∅ |
| `clusters[].schemaRegistry.amazonSecurity.type` | Authentication with credentials, one of `Credentials`, `FromContext`, `FromRole` | `CDK_CLUSTERS_0_SCHEMAREGISTRY_AMAZONSECURITY_TYPE` | true | string | ∅ |
| **Credentials Security** | | | | | |
| `clusters[].schemaRegistry.amazonSecurity.accessKeyId` | Credentials auth access key | `CDK_CLUSTERS_0_SCHEMAREGISTRY_AMAZONSECURITY_ACCESSKEYID` | true | string | ∅ |
| `clusters[].schemaRegistry.amazonSecurity.secretKey` | Credentials auth secret key | `CDK_CLUSTERS_0_SCHEMAREGISTRY_AMAZONSECURITY_SECRETKEY` | true | string | ∅ |
| **FromContext Security** | | | | | |
| `clusters[].schemaRegistry.amazonSecurity.profile` | Authentication profile | `CDK_CLUSTERS_0_SCHEMAREGISTRY_AMAZONSECURITY_PROFILE` | false | string | ∅ |
| **FromRole Security** | | | | | |
| `clusters[].schemaRegistry.amazonSecurity.role` | Authentication role | `CDK_CLUSTERS_0_SCHEMAREGISTRY_AMAZONSECURITY_ROLE` | true | string | ∅ |
### Kafka Connect properties
| Property | Description | Environment variable | Mandatory | Type | Default |
| ------------------------------------------------------- | --------------------------------------------------------------- | ----------------------------------------------------------- | --------- | ------- | ------- |
| `clusters[].kafkaConnects[].id` | String used to uniquely identify your Kafka Connect | `CDK_CLUSTERS_0_KAFKACONNECTS_0_ID` | true | string | ∅ |
| `clusters[].kafkaConnects[].name` | Name your Kafka Connect | `CDK_CLUSTERS_0_KAFKACONNECTS_0_NAME` | true | string | ∅ |
| `clusters[].kafkaConnects[].url` | The Kafka connect URL | `CDK_CLUSTERS_0_KAFKACONNECTS_0_URL` | true | string | ∅ |
| `clusters[].kafkaConnects[].headers` | Optional additional headers (ie: `X-API-Token=123,X-From=Test`) | `CDK_CLUSTERS_0_KAFKACONNECTS_0_HEADERS` | false | string | ∅ |
| `clusters[].kafkaConnects[].ignoreUntrustedCertificate` | Skip SSL certificate validation | `CDK_CLUSTERS_0_KAFKACONNECTS_0_IGNOREUNTRUSTEDCERTIFICATE` | false | boolean | `false` |
| **Basic authentication** | | | | | |
| `clusters[].kafkaConnects[].security.username` | Basic auth username | `CDK_CLUSTERS_0_KAFKACONNECTS_0_SECURITY_USERNAME` | false | string | ∅ |
| `clusters[].kafkaConnects[].security.password` | Basic auth password | `CDK_CLUSTERS_0_KAFKACONNECTS_0_SECURITY_PASSWORD` | false | string | ∅ |
| **Bearer token authentication** | | | | | |
| `clusters[].kafkaConnects[].security.token` | Bearer token | `CDK_CLUSTERS_0_KAFKACONNECTS_0_SECURITY_TOKEN` | false | string | ∅ |
| **mTLS authentication** | | | | | |
| `clusters[].kafkaConnects[].security.key` | Access key | `CDK_CLUSTERS_0_KAFKACONNECTS_0_SECURITY_KEY` | false | string | ∅ |
| `clusters[].kafkaConnects[].security.certificateChain` | Access certificate | `CDK_CLUSTERS_0_KAFKACONNECTS_0_SECURITY_CERTIFICATECHAIN` | false | string | ∅ |
### ksqlDB properties
| Property | Description | Environment variable | Mandatory | Type | Default |
| ------------------------------------------------- | ----------------------------------------------------------------- | ----------------------------------------------------- | --------- | ------- | ------- |
| `clusters[].ksqlDBs[].id` | String used to uniquely identify your ksqlDB Cluster | `CDK_CLUSTERS_0_KSQLDBS_0_ID` | true | string | ∅ |
| `clusters[].ksqlDBs[].name` | Name of your ksqlDB Cluster | `CDK_CLUSTERS_0_KSQLDBS_0_NAME` | true | string | ∅ |
| `clusters[].ksqlDBs[].url` | The ksqlDB API URL | `CDK_CLUSTERS_0_KSQLDBS_0_URL` | true | string | ∅ |
| `clusters[].ksqlDBs[].headers` | Optional additional headers (e.g., `X-API-Token=123,X-From=Test`) | `CDK_CLUSTERS_0_KSQLDBS_0_HEADERS` | false | string | ∅ |
| `clusters[].ksqlDBs[].ignoreUntrustedCertificate` | Skip SSL certificate validation | `CDK_CLUSTERS_0_KSQLDBS_0_IGNOREUNTRUSTEDCERTIFICATE` | false | boolean | `false` |
| **Basic Authentication** | | | | | |
| `clusters[].ksqlDBs[].security.username` | Basic auth username | `CDK_CLUSTERS_0_KSQLDBS_0_SECURITY_USERNAME` | false | string | ∅ |
| `clusters[].ksqlDBs[].security.password` | Basic auth password | `CDK_CLUSTERS_0_KSQLDBS_0_SECURITY_PASSWORD` | false | string | ∅ |
| **Bearer Token Authentication** | | | | | |
| `clusters[].ksqlDBs[].security.token` | Bearer token | `CDK_CLUSTERS_0_KSQLDBS_0_SECURITY_TOKEN` | false | string | ∅ |
| **mTLS Authentication** | | | | | |
| `clusters[].ksqlDBs[].security.key` | Access key | `CDK_CLUSTERS_0_KSQLDBS_0_SECURITY_KEY` | false | string | ∅ |
| `clusters[].ksqlDBs[].security.certificateChain` | Access certificate | `CDK_CLUSTERS_0_KSQLDBS_0_SECURITY_CERTIFICATECHAIN` | false | string | ∅ |
### Indexing properties
Indexing fetches metadata from your Kafka cluster (e.g. topics, consumer groups, subjects).
You should only modify these parameters if you see an issue with the performance.
Console 1.41.0 merged the monitoring indexer into the metadata indexer. The `CDK_MONITORINGCONFIG_*` variables no longer have any effect. Use the metadata indexer settings (`CDK_METADATAINDEXER_*`) below instead.
| Property | Description | Environment variable | Mandatory | Type | Default |
| ---------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | --------- | ------- | ----------------- |
| **Lag exporter** | | | | | |
| `lagexporter.frequency` | Frequency in seconds of the execution of the lag exporter | `CDK_LAGEXPORTER_FREQUENCY` | false | int | `30` |
| `lagexporter.clusterparallelism` | Number of clusters indexed in parallel for the lag exporter | `CDK_LAGEXPORTER_CLUSTERPARALLELISM` | false | int | `1` |
| `lagexporter.indexertimeout` | Lag exporter timeout in seconds | `CDK_LAGEXPORTER_INDEXERTIMEOUT` | false | int | `300` (5 minutes) |
| `lagexporter.blockingduration` | Circuit breaker blocking duration in seconds - defines how long the indexer will wait after a series of failed requests before reattempting indexing | `CDK_LAGEXPORTER_BLOCKINGDURATION` | false | int | `600` |
| `lagexporter.sequentialfailurethreshold` | Number of sequential failures before the circuit breaker activates | `CDK_LAGEXPORTER_SEQUENTIALFAILURETHRESHOLD` | false | int | `10` |
| **Metadata indexer** | | | | | |
| `metadataindexer.frequency` | Frequency in seconds of the execution of the metadata indexer | `CDK_METADATAINDEXER_FREQUENCY` | false | int | `30` |
| `metadataindexer.clusterparallelism` | Number of clusters indexed in parallel for the metadata indexer | `CDK_METADATAINDEXER_CLUSTERPARALLELISM` | false | int | `1` |
| `metadataindexer.indexertimeout` | Metadata indexer timeout in seconds | `CDK_METADATAINDEXER_INDEXERTIMEOUT` | false | int | `300` (5 minutes) |
| `metadataindexer.blockingduration` | Circuit breaker blocking duration in seconds - defines how long the indexer will wait after a series of failed requests before reattempting indexing | `CDK_METADATAINDEXER_BLOCKINGDURATION` | false | int | `600` |
| `metadataindexer.sequentialfailurethreshold` | Number of sequential failures before the circuit breaker activates | `CDK_METADATAINDEXER_SEQUENTIALFAILURETHRESHOLD` | false | int | `10` |
| **Schema registry indexer** | | | | | |
| `registryindexer.frequency` | Frequency in seconds of the execution of the schema registry indexer | `CDK_REGISTRYINDEXER_FREQUENCY` | false | int | `30` |
| `registryindexer.clusterparallelism` | Number of clusters indexed in parallel for the schema registry indexer | `CDK_REGISTRYINDEXER_CLUSTERPARALLELISM` | false | int | `1` |
| `registryindexer.indexertimeout` | Schema registry indexer timeout in seconds | `CDK_REGISTRYINDEXER_INDEXERTIMEOUT` | false | int | `300` (5 minutes) |
| `registryindexer.blockingduration` | Circuit breaker blocking duration in seconds - defines how long the indexer will wait after a series of failed requests before reattempting indexing | `CDK_REGISTRYINDEXER_BLOCKINGDURATION` | false | int | `600` |
| `registryindexer.sequentialfailurethreshold` | Number of sequential failures before the circuit breaker activates | `CDK_REGISTRYINDEXER_SEQUENTIALFAILURETHRESHOLD` | false | int | `10` |
| **Kafka connect indexer** | | | | | |
| `connectindexer.frequency` | Frequency in seconds of the execution of the kafka connect indexer | `CDK_CONNECTINDEXER_FREQUENCY` | false | int | `30` |
| `connectindexer.clusterparallelism` | Number of clusters indexed in parallel for the kafka connect indexer | `CDK_CONNECTINDEXER_CLUSTERPARALLELISM` | false | int | `1` |
| `connectindexer.indexertimeout` | Kafka connect indexer timeout in seconds | `CDK_CONNECTINDEXER_INDEXERTIMEOUT` | false | int | `300` (5 minutes) |
| `connectindexer.blockingduration` | Circuit breaker blocking duration in seconds - defines how long the indexer will wait after a series of failed requests before reattempting indexing | `CDK_CONNECTINDEXER_BLOCKINGDURATION` | false | int | `600` |
| `connectindexer.sequentialfailurethreshold` | Number of sequential failures before the circuit breaker activates | `CDK_CONNECTINDEXER_SEQUENTIALFAILURETHRESHOLD` | false | int | `10` |
| **Refresh analytics** | | | | | |
| `refreshanalytics.frequency` | Frequency in seconds for refreshing of analytics materialized views such as Insights. Note that there is a performance penalty to increasing this frequency | `CDK_REFRESHANALYTICS_FREQUENCY` | false | int | `900` |
| `refreshanalytics.timeout` | Timeout in seconds for refreshing of analytics materialized views such as Insights | `CDK_REFRESHANALYTICS_TIMEOUT` | false | int | `600` |
| `refreshanalytics.enabled` | Enable or disable refreshing of analytics materialized views such as Insights. Defaults to `true`. Only disable if you do not wish to refresh analytic views such as Insights | `CDK_REFRESHANALYTICS_ENABLED` | false | boolean | `true` |
| **Kafka admin client configuration** | | | | | |
| `kafka_admin.list_consumer_group_offsets_batch_size` | The number of consumer groups offset to fetch in a single query. Old versions of Kafka may time out when fetching too many offsets at once. | `CDK_KAFKAADMIN_LISTCONSUMERGROUPOFFSETSBATCHSIZE` | false | int | `100` |
| `kafka_admin.describe_consumer_groups_batch_size` | The number of consumer groups to be described in a single query. | `CDK_KAFKAADMIN_DESCRIBECONSUMERGROUPSBATCHSIZE` | false | int | `10000` |
| `kafka_admin.list_consumer_group_offsets_batch_size` | The number of consumer groups offset to fetch in a single query. Old versions of Kafka may time out when fetching too many offsets at once. | `CDK_KAFKAADMIN_LISTCONSUMERGROUPOFFSETSBATCHSIZE` | false | int | `100` |
| `kafka_admin.describe_consumer_groups_batch_size` | The number of consumer groups to be described in a single query. | `CDK_KAFKAADMIN_DESCRIBECONSUMERGROUPSBATCHSIZE` | false | int | `10000` |
| `kafka_admin.batch_parallel_size` | Maximum of batched requests that can be sent in parallel | `CDK_KAFKAADMIN_BATCHPARALLELSIZE` | false | int | `5` |
| `kafka_admin.record_size_limit` | Maximum size in bytes of a single message to display in the consume page. For larger messages, you'll get a link to open in a dedicated page. | `CDK_KAFKAADMIN_RECORDSIZELIMIT` | false | int | `102400` (bytes) |
### Indexer watchdog properties
Indexer tasks run continuously, but they can occasionally stop for no clear reason. To handle this, Console includes an indexer watchdog that periodically checks the state of indexer tasks. If an indexer's state is stale (no update within the configured threshold), the watchdog restarts all indexer tasks.
| Property | Description | Environment variable | Mandatory | Type | Default |
| -------------------------------------- | --------------------------------------------------------------------------- | ------------------------------------------ | --------- | ---- | ------- |
| `watchdog_staleness_threshold_seconds` | Indexer staleness threshold in seconds. Set to `0` to disable the watchdog. | `CDK_WATCHDOG_STALENESS_THRESHOLD_SECONDS` | false | int | `600` |
### Audit log export properties
The audit log can be exported to a Kafka topic, once configured in Console.
| Property | Description | Environment variable | Mandatory | Type | Default |
| --------------------------------------------------- | ---------------------------------------------------- | ----------------------------------------------------- | --------- | ------ | ------- |
| `audit_log_publisher.cluster` | The cluster ID where the audit logs will be exported | `CDK_AUDITLOGPUBLISHER_CLUSTER` | false | string | ∅ |
| `audit_log_publisher.topicName` | The topic name where the audit logs will be exported | `CDK_AUDITLOGPUBLISHER_TOPICNAME` | false | string | ∅ |
| `audit_log_publisher.topicConfig.partition` | The number of partitions for the audit log topic | `CDK_AUDITLOGPUBLISHER_TOPICCONFIG_PARTITION` | false | int | `1` |
| `audit_log_publisher.topicConfig.replicationFactor` | The replication factor for the audit log topic | `CDK_AUDITLOGPUBLISHER_TOPICCONFIG_REPLICATIONFACTOR` | false | int | `1` |
## Data properties
### Conduktor SQL properties
In order to use Conduktor SQL, you need to configure a second database to store the Topics data. You can configure Conduktor SQL Database using `CDK_KAFKASQL_DATABASE_URL` or alternatively, set each value individually `CDK_KAFKASQL_DATABASE_*`.
| Property | Description | Environment variable | Mandatory | Type | Default |
| --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------- | --------- | ------ | ------- |
| `kafka_sql.database.url` | External PostgreSQL configuration URL in format `[jdbc:]postgresql://[user[:password]@][[netloc][:port],...][/dbname][?param1=value1&...]` | `CDK_KAFKASQL_DATABASE_URL` | false | string | ∅ |
| `kafka_sql.database.hosts[].host` | External PostgreSQL servers hostname | `CDK_KAFKASQL_DATABASE_HOSTS_0_HOST` | false | string | ∅ |
| `kafka_sql.database.hosts[].port` | External PostgreSQL servers port | `CDK_KAFKASQL_DATABASE_HOSTS_0_PORT` | false | int | ∅ |
| `kafka_sql.database.host` | External PostgreSQL server hostname (Deprecated, use `kafka_sql.database.hosts` instead) | `CDK_KAFKASQL_DATABASE_HOST` | false | string | ∅ |
| `kafka_sql.database.port` | External PostgreSQL server port (Deprecated, use `kafka_sql.database.hosts` instead) | `CDK_KAFKASQL_DATABASE_PORT` | false | int | ∅ |
| `kafka_sql.database.name` | External PostgreSQL database name | `CDK_KAFKASQL_DATABASE_NAME` | false | string | ∅ |
| `kafka_sql.database.username` | External PostgreSQL login role | `CDK_KAFKASQL_DATABASE_USERNAME` | false | string | ∅ |
| `kafka_sql.database.password` | External PostgreSQL login password | `CDK_KAFKASQL_DATABASE_PASSWORD` | false | string | ∅ |
| `kafka_sql.database.connection_timeout` | External PostgreSQL connection timeout in seconds | `CDK_KAFKASQL_DATABASE_CONNECTIONTIMEOUT` | false | int | ∅ |
| `kafka_sql.database.options` | External PostgreSQL options (available via YAML configuration only) | | false | map | ∅ |
[Find out more about database properties](#database-properties).
These are advanced properties that typically won't need to be changed.
| Property | Description | Environment variable | Mandatory | Type | Default |
| ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | -------------------------------------------------- | --------- | ------ | --------------- |
| `kafka_sql.commit_offset_every_in_sec` | Frequency at which Conduktor SQL commits offsets into Kafka and flushes rows in the database | `CDK_KAFKASQL_COMMITOFFSETEVERYINSEC` | false | int | `30` (seconds) |
| `kafka_sql.clean_expired_record_every_in_hour` | How often to check for expired records and delete them from the Database | `CDK_KAFKASQL_CLEANEXPIREDRECORDEVERYINHOUR` | false | int | `1` (hour) |
| `kafka_sql.refresh_topic_configuration_every_in_sec` | Frequency at which Conduktor SQL looks for new topics to start indexing or stop indexing | `CDK_KAFKASQL_REFRESHTOPICCONFIGURATIONEVERYINSEC` | false | int | `30` (seconds) |
| `kafka_sql.consumer_group_id` | Consumer group used to identify Conduktor SQL | `CDK_KAFKASQL_CONSUMER-GROUP-ID` | false | string | `conduktor-sql` |
| `kafka_sql.refresh_user_permissions_every_in_sec` | Frequency at which Conduktor SQL refreshes the role permissions in the DB to match the RBAC setup in Console | `CDK_KAFKASQL_REFRESHUSERPERMISSIONSEVERYINSEC` | false | int | `30` (seconds) |
### Data quality properties
These are advanced properties that typically won't need to be changed.
| Property | Description | Environment variable | Mandatory | Type | Default |
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------- | --------- | ---- | ------------- |
| `data_quality.reconcile_every_seconds` | The interval at which the data quality's state on Console is synchronized with Gateway. A lower value results in faster alignment between the required state and the current state on Gateway. The default value is 5 seconds. | CDK\_DATAQUALITY\_RECONCILEEVERYSECONDS | false | int | `5` (seconds) |
| `data_quality.report_max_per_second` | The number of violations reported in data quality can be high. This value fixes the max number of violations that will be reported, per second. If more violations are produced, they won't be reported in Console. | CDK\_DATAQUALITY\_REPORTMAXPERSECOND | false | int | 10 (unit) |
| `data_quality.report_retention_days` | The number of days that the violation history logs will be retained. | CDK\_DATAQUALITY\_REPORTRETENTIONDAYS | false | int | `30` (days) |
### Data quality overview properties
These properties configure the data quality overview page which provides coverage and health metrics.
| Property | Description | Environment variable | Mandatory | Type | Default |
| ----------------------------------------- | --------------------------------------------------------------------------------- | --------------------------------------------- | --------- | ------- | ----------------- |
| `refresh_data_quality_overview.enabled` | Enable or disable the data quality overview feature. | `CDK_REFRESH-DATA-QUALITY-OVERVIEW_ENABLED` | false | boolean | `true` |
| `refresh_data_quality_overview.frequency` | The interval in seconds at which the data quality overview metrics are refreshed. | `CDK_REFRESH-DATA-QUALITY-OVERVIEW_FREQUENCY` | false | int | `300` (5 minutes) |
| `refresh_data_quality_overview.timeout` | The timeout in seconds for data quality overview computations. | `CDK_REFRESH-DATA-QUALITY-OVERVIEW_TIMEOUT` | false | int | `120` (2 minutes) |
### Data policy properties
These are advanced properties that typically won't need to be changed.
| Property | Description | Environment variable | Mandatory | Type | Default |
| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- | --------- | ------- | ------- |
| `data_policy.disable_key_masking` | Disables data masking policy for record keys. Set to `true` if your record keys don't use JSON or schema-based structures. | CDK\_DATAPOLICY\_DISABLEKEYMASKING | false | boolean | `false` |
### Stream lineage properties
These properties control [Stream lineage](/guide/conduktor-concepts/stream-lineage) graph caching and live throughput in Console.
| Property | Description | Environment variable | Mandatory | Type | Default |
| ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------- | --------- | ------- | -------- |
| `stream_lineage.enabled` | Enable Stream lineage. Set to `false` to disable the feature (including live throughput) and avoid the in-memory graph cache overhead. | `CDK_STREAMLINEAGE_ENABLED` | false | boolean | `true` |
| `stream_lineage.indexer_frequency_seconds` | How often Console rebuilds each cluster's Stream lineage graph, in seconds. Acts as the cache TTL. | `CDK_STREAMLINEAGE_INDEXERFREQUENCYSECONDS` | false | long | `900` |
| `stream_lineage.throughput_max_keys_per_cluster` | Upper bound on the number of service account and topic pairs tracked in memory per cluster for live throughput. Adjusting this has performance ramifications: each tracked pair holds memory on the Console instance, so raising the bound increases memory usage, while lowering it can evict throughput rates for less active pairs from the graph. Since 1.47.0. | `CDK_STREAMLINEAGE_THROUGHPUTMAXKEYSPERCLUSTER` | false | long | `250000` |
## Advanced properties
### Partner Zones properties
These are advanced properties that typically won't need to be changed.
| Property | Description | Environment variable | Mandatory | Type | Default |
| --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | --------- | ---- | ----------------- |
| `partner_zone.reconcile_with_gateway_every_seconds` | The interval at which the partner zone's state that is stored on Console, is synchronized with Gateway. A lower value results in faster alignment between the desired state and the current state on Gateway. The default is 3 minutes. | CDK\_PARTNERZONE\_RECONCILEWITHGATEWAYEVERYSECONDS | false | int | `180` (3 minutes) |
### Chargeback properties
These properties enable and configure Chargeback functionality, both with and without Gateway.
| Property | Description | Environment variable | Mandatory | Type | Default |
| --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------- | --------- | ------- | ------- |
| `chargebackv2.enabled` | Enable Chargeback functionality (both with and without Gateway). Enabled by default from Console v1.46.0; set to `false` to disable. | `CDK_CHARGEBACKV2_ENABLED` | false | boolean | `true` |
| `chargebackv2.metrics-update-frequency` | Frequency in seconds for metrics collection for Chargeback without Gateway (available in Console v1.41.0 and later). | `CDK_CHARGEBACKV2_METRICSUPDATEFREQUENCY` | false | long | `3600` |
### UI settings properties
Customize Console UI behavior for your organization, including default topic creation settings and custom support links.
| Property | Description | Environment variable | Mandatory | Type | Default |
| ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------ | --------- | ------- | ------- |
| `ui_settings.default_number_of_partitions` | Default number of partitions in the create-topic dialog. When set, overrides the Kafka cluster default for all topic creation in Console. | `CDK_UISETTINGS_DEFAULTNUMBEROFPARTITIONS` | false | integer | ∅ |
| `ui_settings.support_request_url` | URL for the **Support** link in the Console help menu. When set, replaces the default Conduktor support portal link. | `CDK_UISETTINGS_SUPPORTREQUESTURL` | false | string | ∅ |
| `ui_settings.feature_request_url` | URL for the **Feedback** link in the Console help menu. When set, replaces the default Conduktor roadmap link. | `CDK_UISETTINGS_FEATUREREQUESTURL` | false | string | ∅ |
# Conduktor Console on Kubernetes — Helm deployment guide
Source: https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-console/kubernetes
Deploy Conduktor Console on Kubernetes using the Helm chart. Covers Helm repo setup, PostgreSQL requirements, and reference architecture for production.
## Overview
Conduktor provides a [Helm repository](https://helm.conduktor.io) containing a chart that will deploy Conduktor Platform on your Kubernetes cluster. We don't provide any relational database dependency, you will have to provide your own database. See the [production requirements](#production-requirements) for details.
For a full reference of our recommended Conduktor platform deployment architecture, [see our sample GitHub repository](https://github.com/conduktor/conduktor-reference-architecture) .
[Check out the snippets section for more examples](#examples).
```shell theme={null}
# Setup Helm repository
helm repo add conduktor https://helm.conduktor.io
helm repo update
export ADMIN_EMAIL=""
export ADMIN_PASSWORD=""
export ORG_NAME=""
export NAMESPACE=""
# Deploy Helm chart
helm install console conduktor/console \
--create-namespace -n ${NAMESPACE} \
--set config.organization.name="${ORG_NAME}" \
--set config.admin.email="${ADMIN_EMAIL}" \
--set config.admin.password="${ADMIN_PASSWORD}" \
--set config.database.password="" \
--set config.database.username="" \
--set config.database.host="" \
--set config.database.port="5432" \
--set config.license="${LICENSE}" # can be omitted if deploying the free tier
# Port forward to access Conduktor
kubectl port-forward deployment/console -n ${NAMESPACE} 8080:8080
open http://localhost:8080
```
## Chart dependencies
All charts within this repository have one dependency which is `bitnami-common`. You can find the chart here: [https://github.com/bitnami/charts/tree/main/bitnami/common](https://github.com/bitnami/charts/tree/main/bitnami/common)
## Compatibility matrix
Find out which versions of Conduktor Platform work on which version of our Conduktor Platform Helm chart.
> We recommend you use the version of Platform that comes pre-configured with the Helm chart. You can adjust the version in your values property according to the supported Platform version, if required.
> Notes column only lists **chart-level changes**. [See release notes](https://docs.conduktor.io/changelog/) to determine whether there are breaking changes within the artifacts.
### Helm chart compatibility
Breaking changes:
🟡 - Breaks additional services (e.g. Grafana dashboard changes)
🔴 - Breaks overall deployment of the product (e.g. renaming variables in .values, major product releases)
| Chart version | Default App Version | Min App Version | Release date | Notes |
| -------------------------------------------------------------------------------------------------- | --------------------------------------------------------- | ----------------------------------------------------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------- |
| [console-1.32.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.32.1) | **1.47.1** | [1.25.1](/guide/release-notes/archive#console-1-25-1) | 2026-09-03 | |
| [console-1.32.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.32.0) | **[1.47.0](/guide/release-notes#console-1-47-0)** | [1.25.1](/guide/release-notes/archive#console-1-25-1) | 2026-08-20 | |
| [console-1.31.2](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.31.2) | **[1.46.2](/guide/release-notes#console-1-46-2)** | [1.25.1](/guide/release-notes/archive#console-1-25-1) | 2026-08-11 | |
| [console-1.31.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.31.1) | **[1.46.1](/guide/release-notes#console-1-46-1)** | [1.25.1](/guide/release-notes/archive#console-1-25-1) | 2026-06-29 | |
| [console-1.31.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.31.0) | **[1.46.0](/guide/release-notes#console-1-46-0)** | [1.25.1](/guide/release-notes/archive#console-1-25-1) | 2026-06-16 | |
| [console-1.30.2](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.30.2) | **[1.45.1](/guide/release-notes#console-1-45-1)** | [1.25.1](/guide/release-notes/archive#console-1-25-1) | 2026-05-27 | |
| [console-1.30.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.30.1) | **[1.45.1](/guide/release-notes#console-1-45-1)** | [1.25.1](/guide/release-notes/archive#console-1-25-1) | 2026-05-14 | |
| [console-1.30.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.30.0) | **[1.45.0](/guide/release-notes#console-1-45-0)** | [1.25.1](/guide/release-notes/archive#console-1-25-1) | 2026-05-05 | |
| [console-1.29.2](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.29.2) | **[1.44.2](/guide/release-notes#console-1-44-2)** | [1.25.1](/guide/release-notes/archive#console-1-25-1) | 2026-06-04 | |
| [console-1.29.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.29.1) | **[1.44.1](/guide/release-notes#console-1-44-1)** | [1.25.1](/guide/release-notes/archive#console-1-25-1) | 2026-03-27 | |
| [console-1.29.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.29.0) | **[1.44.0](/guide/release-notes#console-1-44-0)** | [1.25.1](/guide/release-notes/archive#console-1-25-1) | 2026-03-18 | |
| [console-1.28.3](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.28.3) | **[1.43.3](/guide/release-notes#console-1-43-3)** | [1.25.1](/guide/release-notes/archive#console-1-25-1) | 2026-03-11 | |
| [console-1.28.2](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.28.2) | **[1.43.2](/guide/release-notes#console-1-43-2)** | [1.25.1](/guide/release-notes/archive#console-1-25-1) | 2026-02-27 | |
| [console-1.28.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.28.1) | **[1.43.1](/guide/release-notes#console-1-43-1)** | [1.25.1](/guide/release-notes/archive#console-1-25-1) | 2026-02-16 | |
| [console-1.28.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.28.0) | **[1.43.0](/guide/release-notes#console-1-43-0)** | [1.25.1](/guide/release-notes/archive#console-1-25-1) | 2026-02-11 | |
| [console-1.27.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.27.1) | **[1.42.0](/guide/release-notes#console-1-42-0)** | [1.25.1](/guide/release-notes/archive#console-1-25-1) | 2026-02-02 | |
| [console-1.27.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.27.0) | **[1.42.0](/guide/release-notes#console-1-42-0)** | [1.25.1](/guide/release-notes/archive#console-1-25-1) | 2026-01-29 | |
| [console-1.26.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.26.1) | **[1.41.0](/guide/release-notes#console-1-41-0)** | [1.25.1](/guide/release-notes/archive#console-1-25-1) | 2026-01-08 | |
| [console-1.26.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.26.0) | **[1.41.0](/guide/release-notes#console-1-41-0)** | [1.25.1](/guide/release-notes/archive#console-1-25-1) | 2025-12-12 | |
| [console-1.25.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.25.0) | **[1.40.0](/guide/release-notes#console-1-40-0)** | [1.25.1](/guide/release-notes/archive#console-1-25-1) | 2025-11-17 | |
| [console-1.24.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.24.0) | **[1.39.0](/guide/release-notes#console-1-39-0)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2025-10-27 | |
| [console-1.23.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.23.0) | **[1.38.0](/guide/release-notes#console-1-38-0)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2025-10-01 | |
| [console-1.22.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.22.1) | **[1.37.1](/guide/release-notes#console-1-37-1)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2025-09-19 | |
| [console-1.22.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.22.0) | **[1.37.0](/guide/release-notes#console-1-37-0)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2025-08-21 | |
| [console-1.21.2](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.21.2) | **[1.36.2](/guide/release-notes#console-1-36-2)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2025-08-05 | |
| [console-1.21.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.21.1) | **[1.36.1](/guide/release-notes#console-1-36-1)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2025-07-22 | |
| [console-1.21.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.21.0) | **[1.36.0](/guide/release-notes#console-1-36-0)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2025-07-18 | |
| [console-1.20.3](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.20.3) | **[1.35.2](/guide/release-notes/archive#console-1-35-2)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2025-07-03 | |
| [console-1.20.2](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.20.2) | **[1.35.1](/guide/release-notes/archive#console-1-35-1)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2025-06-26 | |
| [console-1.20.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.20.1) | **[1.35.0](/guide/release-notes/archive#console-1-35-0)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2025-06-19 | |
| [console-1.20.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.20.0) | **[1.35.0](/guide/release-notes/archive#console-1-35-0)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2025-06-19 | |
| [console-1.19.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.19.1) | **[1.34.2](/guide/release-notes/archive#console-1-34-2)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2025-05-26 | |
| [console-1.19.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.19.0) | **[1.34.0](/guide/release-notes/archive#console-1-34-0)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2025-05-14 | |
| [console-1.18.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.18.0) | **[1.33.0](/guide/release-notes/archive#console-1-33-0)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2025-04-09 | |
| [console-1.17.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.17.1) | **[1.32.1](/guide/release-notes/archive#console-1-32-1)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2025-03-25 | |
| [console-1.17.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.17.0) | **[1.32.0](/guide/release-notes/archive#console-1-32-0)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2025-03-12 | |
| [console-1.16.2](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.16.2) | **[1.31.2](/guide/release-notes/archive#console-1-31-2)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2025-03-04 | |
| [console-1.16.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.16.1) | **[1.31.1](/guide/release-notes/archive#console-1-31-1)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2025-02-19 | |
| [console-1.16.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.16.0) | **[1.31.0](/guide/release-notes/archive#console-1-31-0)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2025-02-06 | |
| [console-1.15.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.15.1) | **[1.30.0](/guide/release-notes/archive#console-1-30-0)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2025-01-15 | |
| [console-1.15.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.15.0) | **[1.30.0](/guide/release-notes/archive#console-1-30-0)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2024-12-16 | |
| [console-1.14.2](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.14.2) | **[1.29.2](/guide/release-notes/archive#console-1-29-2)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2024-12-09 | |
| [console-1.14.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.14.1) | **[1.29.1](/guide/release-notes/archive#console-1-29-1)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2024-11-28 | |
| [console-1.14.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.14.0) | **[1.29.0](/guide/release-notes/archive#console-1-29-0)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2024-11-25 | |
| [console-1.13.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.13.0) | **[1.28.0](/guide/release-notes/archive#console-1-28-0)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2024-10-15 | |
| [console-1.12.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.12.1) | **[1.27.1](/guide/release-notes/archive#console-1-27-1)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2024-09-26 | |
| [console-1.12.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.12.0) | **[1.27.0](/guide/release-notes/archive#console-1-27-0)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2024-09-13 | |
| [console-1.11.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.11.0) | **[1.26.0](/guide/release-notes/archive#console-1-26-0)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2024-08-19 | |
| [console-1.10.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.10.0) | **[1.25.1](/guide/release-notes/archive#console-1-25-1)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2024-07-24 | |
| [console-1.9.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.9.1) | **[1.24.1](/guide/release-notes/archive#console-1-24-1)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2024-06-24 | |
| [console-1.9.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.9.0) | **[1.24.0](/guide/release-notes/archive#console-1-24-0)** | [1.24.0](/guide/release-notes/archive#console-1-24-0) | 2024-06-19 | 🔴 Changed liveness and readiness probe path [see here](https://github.com/conduktor/conduktor-public-charts/pull/80) |
| [console-1.8.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.8.1) | **[1.23.0](/guide/release-notes/archive#console-1-23-0)** | [1.22.0](/guide/release-notes/archive#console-1-22-0) | 2024-05-10 | |
| [console-1.7.2](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.7.2) | **[1.22.0](/guide/release-notes/archive#console-1-22-0)** | [1.22.0](/guide/release-notes/archive#console-1-22-0) | 2024-04-03 | 🔴 Service Monitor endpoint changes, Grafana template changes [see here](https://github.com/conduktor/conduktor-public-charts/pull/65) |
| [console-1.6.2](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.6.2) | **[1.21.1](/guide/release-notes/archive#console-1-21-1)** | [1.21.0](/guide/release-notes/archive#console-1-21-0) | 2024-03-18 | |
| [console-1.6.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.6.1) | **[1.21.1](/guide/release-notes/archive#console-1-21-1)** | [1.21.0](/guide/release-notes/archive#console-1-21-0) | 2024-03-05 | |
| [console-1.6.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.6.0) | **[1.21.0](/guide/release-notes/archive#console-1-21-0)** | [1.21.0](/guide/release-notes/archive#console-1-21-0) | 2024-02-26 | 🔴 Paths and folder changed [see here](https://github.com/conduktor/conduktor-public-charts/pull/54) |
| [console-1.5.5](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.5.5) | **[1.20.0](/guide/release-notes/archive#console-1-20-0)** | [1.17.3](/guide/release-notes/archive#console-1-17-3) | 2024-01-23 | |
| [console-1.5.4](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.5.4) | **[1.20.0](/guide/release-notes/archive#console-1-20-0)** | [1.17.3](/guide/release-notes/archive#console-1-17-3) | 2024-01-10 | 🟡 Updated Grafana template [see here](https://github.com/conduktor/conduktor-public-charts/pull/49) |
| [console-1.5.3](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.5.3) | **[1.20.0](/guide/release-notes/archive#console-1-20-0)** | [1.17.3](/guide/release-notes/archive#console-1-17-3) | 2024-01-05 | 🟡 Updated Grafana template [see here](https://github.com/conduktor/conduktor-public-charts/pull/47) |
| [console-1.5.2](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.5.2) | **[1.20.0](/guide/release-notes/archive#console-1-20-0)** | [1.17.3](/guide/release-notes/archive#console-1-17-3) | 2024-01-04 | 🟡 Updated Grafana template [see here](https://github.com/conduktor/conduktor-public-charts/pull/44) |
| [console-1.5.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.5.1) | **[1.20.0](/guide/release-notes/archive#console-1-20-0)** | [1.17.3](/guide/release-notes/archive#console-1-17-3) | 2023-12-19 | |
| [console-1.5.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.5.0) | **[1.20.0](/guide/release-notes/archive#console-1-20-0)** | [1.17.3](/guide/release-notes/archive#console-1-17-3) | 2023-12-18 | |
| [console-1.4.2](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.4.2) | **[1.19.2](/guide/release-notes/archive#console-1-19-2)** | [1.17.3](/guide/release-notes/archive#console-1-17-3) | 2023-11-24 | |
| [console-1.4.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.4.1) | **[1.19.1](/guide/release-notes/archive#console-1-19-1)** | [1.17.3](/guide/release-notes/archive#console-1-17-3) | 2023-11-15 | |
| [console-1.4.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.4.0) | **[1.19.0](/guide/release-notes/archive#console-1-19-0)** | [1.17.3](/guide/release-notes/archive#console-1-17-3) | 2023-11-02 | |
| [console-1.3.9](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.3.9) | **[1.18.4](/guide/release-notes/archive#console-1-18-4)** | [1.17.3](/guide/release-notes/archive#console-1-17-3) | 2023-11-02 | |
| [console-1.3.8](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.3.8) | **[1.18.4](/guide/release-notes/archive#console-1-18-4)** | [1.17.3](/guide/release-notes/archive#console-1-17-3) | 2023-10-19 | |
| [console-1.3.7](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.3.7) | **1.18.3** | [1.17.3](/guide/release-notes/archive#console-1-17-3) | 2023-10-10 | |
| [console-1.3.6](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.3.6) | **[1.18.2](/guide/release-notes/archive#console-1-18-2)** | [1.17.3](/guide/release-notes/archive#console-1-17-3) | 2023-10-10 | |
| [console-1.3.5](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.3.5) | **[1.18.1](/guide/release-notes/archive#console-1-18-1)** | [1.17.3](/guide/release-notes/archive#console-1-17-3) | 2023-10-03 | |
| [console-1.3.4](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.3.4) | **[1.18.1](/guide/release-notes/archive#console-1-18-1)** | [1.17.3](/guide/release-notes/archive#console-1-17-3) | 2023-09-29 | |
| [console-1.3.3](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.3.3) | **[1.18.1](/guide/release-notes/archive#console-1-18-1)** | [1.17.3](/guide/release-notes/archive#console-1-17-3) | 2023-09-27 | |
| [console-1.3.2](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.3.2) | **[1.18.0](/guide/release-notes/archive#console-1-18-0)** | [1.17.3](/guide/release-notes/archive#console-1-17-3) | 2023-09-27 | |
| [console-1.3.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.3.1) | **[1.18.0](/guide/release-notes/archive#console-1-18-0)** | [1.17.3](/guide/release-notes/archive#console-1-17-3) | 2023-09-27 | |
| [console-1.3.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.3.0) | **[1.18.0](/guide/release-notes/archive#console-1-18-0)** | [1.17.3](/guide/release-notes/archive#console-1-17-3) | 2023-09-19 | |
| [console-1.2.4](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.2.4) | **[1.17.3](/guide/release-notes/archive#console-1-17-3)** | [1.17.3](/guide/release-notes/archive#console-1-17-3) | 2023-09-08 | |
| [console-1.2.3](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.2.3) | **[1.17.3](/guide/release-notes/archive#console-1-17-3)** | [1.17.3](/guide/release-notes/archive#console-1-17-3) | 2023-09-08 | |
| [console-1.2.2](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.2.2) | **[1.17.3](/guide/release-notes/archive#console-1-17-3)** | [1.17.3](/guide/release-notes/archive#console-1-17-3) | 2023-09-08 | |
| [console-1.2.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.2.1) | **[1.17.3](/guide/release-notes/archive#console-1-17-3)** | [1.17.3](/guide/release-notes/archive#console-1-17-3) | 2023-09-08 | |
| [console-1.2.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.2.0) | **[1.17.3](/guide/release-notes/archive#console-1-17-3)** | [1.17.3](/guide/release-notes/archive#console-1-17-3) | 2023-09-08 | |
| [console-1.1.4](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.1.4) | **[1.17.3](/guide/release-notes/archive#console-1-17-3)** | [1.17.3](/guide/release-notes/archive#console-1-17-3) | 2023-09-07 | 🔴 Fixed issue with license checksum [see here](https://github.com/conduktor/conduktor-public-charts/pull/14) |
| [console-1.1.3](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.1.3) | **[1.17.3](/guide/release-notes/archive#console-1-17-3)** | [1.17.2](/guide/release-notes/archive#console-1-17-2) | 2023-09-07 | |
| [console-1.1.2](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.1.2) | **[1.17.3](/guide/release-notes/archive#console-1-17-3)** | [1.17.2](/guide/release-notes/archive#console-1-17-2) | 2023-09-07 | |
| [console-1.1.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.1.1) | **[1.17.3](/guide/release-notes/archive#console-1-17-3)** | [1.17.2](/guide/release-notes/archive#console-1-17-2) | 2023-09-06 | |
| [console-1.1.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.1.0) | **[1.17.3](/guide/release-notes/archive#console-1-17-3)** | [1.17.2](/guide/release-notes/archive#console-1-17-2) | 2023-09-01 | |
| [console-1.0.3](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.0.3) | **[1.17.3](/guide/release-notes/archive#console-1-17-3)** | [1.17.2](/guide/release-notes/archive#console-1-17-2) | 2023-08-30 | |
| [console-1.0.2](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.0.2) | **[1.17.3](/guide/release-notes/archive#console-1-17-3)** | [1.17.2](/guide/release-notes/archive#console-1-17-2) | 2023-08-29 | |
| [console-1.0.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.0.1) | **[1.17.3](/guide/release-notes/archive#console-1-17-3)** | [1.17.2](/guide/release-notes/archive#console-1-17-2) | 2023-08-21 | |
| [console-1.0.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/console-1.0.0) | **[1.17.2](/guide/release-notes/archive#console-1-17-2)** | [1.17.2](/guide/release-notes/archive#console-1-17-2) | 2023-08-17 | |
## General requirements
* Basic knowledge of Kubernetes
* Kubernetes cluster 1.19+ ([set up a local cluster](https://k3d.io/#installation) )\[^1]
* Kubectl ([install](https://kubernetes.io/docs/tasks/tools/#kubectl) ) with proper kube context configured
* Helm 3.1.0+ ([install](https://helm.sh/docs/intro/install/) )
## Production requirements
**Mandatory for production environments**:
* set up an external PostgreSQL (13+) database with an appropriate backup policy
* set up an external S3 Bucket
* allow enough resources to run Conduktor with the recommended configuration
### A note on TLS and URL forwarding
For production environments, we recommend to run with TLS enabled from your ingress controller and terminating on Platform. This creates a more secure connection, while also telling Platform that it should use TLS when forwarding on any URL requests (e.g. requests to SSO providers).
Without TLS terminating on Platform itself, requests between the ingress controller and Platform will be in plain text, as will URL forwarding to your SSO provider which can lead to rejection of the request for not being secure.
## 1. Set up Helm repository
```shell theme={null}
helm repo add conduktor https://helm.conduktor.io
helm repo update
```
## 2. Install the Console chart
Configure Console with the following values:
```yaml title="values.yaml" theme={null}
config:
organization:
name: ""
admin:
email: ""
password: ""
database:
host: ''
port: 5432
name: ''
username: ''
password: ''
# HERE you can paste the console configuration (under the config key)
```
Install the chart on your cluster:
```shell theme={null}
helm install console conduktor/console \
--create-namespace -n conduktor \
--values values.yaml \
--set config.license="${LICENSE}" # can be omitted if deploying the free tier
```
Once deployed, you will be able to access Conduktor on
[localhost:8080](localhost:8080) by using a port-forward. You can also configure an ingress to make Platform available externally, [check out snippets](#examples).
```bash theme={null}
kubectl port-forward deployment/console -n ${NAMESPACE} 8080:8080
```
## 3. Configure Console
### Fresh install
You can configure Platform by inserting it into the `config` section of the **values.yaml** file.
### Based on a Docker configuration
If you're already using a config file within Docker,you can use it by giving it to the Helm chart with the following command:
```yaml title="values.yaml" theme={null}
config:
organization:
name: ""
admin:
email: ""
password: ""
database:
host: ''
port: 5432
name: ''
username: ''
password: ''
# HERE you can paste the console configuration (under the config key)
```
### Configure with an enterprise license
```yaml title="values.yaml" theme={null}
config:
organization:
name: ""
admin:
email: ""
password: ""
database:
host: ''
port: 5432
name: ''
username: ''
password: ''
license: ""
# HERE you can paste the console configuration (under the config key)
```
## Examples
[Find sample code on our GitHub](https://github.com/conduktor/conduktor-public-charts/blob/main/charts/console/README.md#snippets) .
### Deploy Console with a context path
Console v1.41.0 introduces the option to configure an ingress with a context path, such as `mycompany.com/conduktor`.
To configure this, set the `CDK_PLATFORM_EXTERNAL_URL` environment variable or the `config.platform.external.url` property to include the full URL with the context path.
Here's an example using `/conduktor` as the context path with an **nginx** ingress controller:
```yaml title="values.yaml" theme={null}
config:
platform:
external:
url: "https://conduktor.my-domain.com/conduktor"
ingress:
enabled: true
hostname: conduktor.my-domain.com
ingressClassName: nginx
tls: true
path: /conduktor(/|$)(.*)
pathType: ImplementationSpecific
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
```
Check the requirements of your ingress controller - you may need to add specific annotations and patterns to the path. As shown in the example above, your ingress controller has to also remove the context path when redirecting requests to Console.
### Secure metric endpoints with basic auth
Starting with Console v1.42.0, you can protect the metric scraping endpoints (`/api/monitoring/metrics` and `/monitoring/metrics/`) with basic auth credentials.
```yaml title="values.yaml" theme={null}
config:
monitoring:
basicauth:
email: "metrics@example.com"
password: "Hello123!"
```
When configured, your monitoring backend (Prometheus, Cortex, etc.) has to provide these credentials when scraping Console metrics. [Learn more about configuring your monitoring backend](/guide/conduktor-in-production/deploy-artifacts/deploy-external-monitoring#basic-auth-for-metrics-scraping).
Basic auth is optional in Console v1.42.0 and no configuration changes are required for existing deployments.
## Related resources
* [View resource reference](/guide/reference/console-reference)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# PostgreSQL sizing for Conduktor Console production
Source: https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-console/postgres-sizing
Size PostgreSQL for Conduktor Console in production on AWS, GCP, or Azure. Covers CPU, RAM, disk IOPS.
## Overview
This guide provides production-ready recommendations for sizing and scaling your PostgreSQL database for Conduktor Console across AWS, GCP or Azure to ensure a performant and consistent user experience.
The minimum requirements (1-2 vCPU, 1 GB RAM, 10 GB disk) are only suitable for **proof-of-concept** or **test** environments.
For production deployments, follow this guide to avoid performance issues like IOPS throttling, CPU spikes and memory exhaustion.
## Considerations
Key considerations for sizing your database:
Select database specifications that match your production workload - avoid using proof-of-concept configurations in production environments.
Set up alerts for CPU, memory, IOPS and query latency before going live.
Provision a minimum of 3,000 IOPS (Input/Output Operations Per Second) - Background sync processes are write-heavy.
Use these recommendations as a **baseline**, then optimize based on actual usage.
## Usage level
To get started, choose the level that matches your expected initial usage:
| Level | Concurrent users | Kafka scale | Estimated Maximum DB size |
| ---------------- | ---------------- | ------------------------------------------------------------------------------ | ------------------------- |
| **Standard** | Up to 500 | 1-5 clusters, up to 1,000 topics / 10,000 partitions, \~500 consumer groups | up to 50 GB |
| **Mid scale** | 500-1,000 | 5-10 clusters, up to 5,000 topics / 50,000 partitions, \~1,000 consumer groups | up to 100 GB |
| **Fully scaled** | 1,000-5,000 | 10+ clusters, 5,000+ topics / 50,000+ partitions, 1,000+ consumer groups | up to 250 GB |
The estimated maximum DB size is based on an installation making maximum utilization of all Conduktor features over a year and caters for accumulated information such as audit logging.
## Target performance
Our recommendations are based on providing the optimal Conduktor experience while avoiding over-provisioning of the database. We've based the recommendations on:
* keeping P95 query latency under a level to provide the best user experience
* handling concurrent queries based on your expected user base size
* supporting background Conduktor metadata updates for your expected Kafka platform size
* avoiding IOPS throttling or similar cloud provider limitations during normal operations
Conduktor Console continuously syncs Kafka metadata to the database via a background task, which requires sufficient IOPS (Input/Output Operations Per Second) as specified in the recommendations below.
## Recommended specifications
We recommend that you use AWS RDS PostgreSQL. The Postgres version has to be 14.8+ or 15.3+.
**Instance type:**
* **Minimum**: `db.t4g.large` (2 vCPU, 8 GB RAM)
* **Recommended**: `db.m6g.large` (2 vCPU, 8 GB RAM) for consistent performance without CPU credits
**Storage:**
* **Type**: general purpose SSD (gp3)
* **Size**: 50 GB minimum (allows for growth)
* **IOPS**: 3,000 IOPS baseline (included free with gp3)
* **Throughput**: 125 MB/s (included free)
**Configuration:**
* PostgreSQL version: 14.8+ or 15.3+. [See version requirements](/guide/conduktor-in-production/deploy-artifacts/deploy-console#configure-postgres-database).
* Multi-AZ: recommended for production
* Automated backups: enable with 7-day retention minimum
**Instance type:**
* **Recommended**: `db.m6g.xlarge` (4 vCPU, 16 GB RAM)
**Storage:**
* **Type**: general purpose SSD (gp3)
* **Size**: 150 GB minimum
* **IOPS**: 5,000-8,000 IOPS (provision additional IOPS beyond baseline)
* **Throughput**: 250-500 MB/s
**Configuration:**
* Multi-AZ: strongly recommended
* Automated backups: 14-day retention
* Performance insights: enable for monitoring
**Instance type:**
* **Recommended**: `db.m6g.2xlarge` (8 vCPU, 32 GB RAM)
**Storage:**
* **Type**: general purpose SSD (gp3) or provisioned IOPS SSD (io1/io2) for >20,000 IOPS
* **Size**: 500 GB minimum
* **IOPS**: 15,000-30,000 IOPS
* **Throughput**: 500-1,000 MB/s
**Configuration:**
* Multi-AZ: required
* Automated backups: 30-day retention
* Performance insights: enable with extended retention
* Enhanced monitoring: enable
We recommend that you use Azure database for PostgreSQL. The Postgres version has to be 13 or higher.
**Compute tier:**
* **Tier**: general purpose
* **SKU**: `Standard_D2s_v3` (2 vCPU, 8 GB RAM)
**Storage:**
* **Type**: premium SSD
* **Size**: 64 GB minimum (32 GB is minimum for premium SSD)
* **IOPS**: depends on VM tier, typically 3,200+ IOPS for D2s\_v3
**Configuration:**
* High availability: zone-redundant recommended for production
* Automated backups: enable with 7-day retention minimum
**Compute tier:**
* **Tier**: general purpose
* **SKU**: `Standard_D4s_v3` (4 vCPU, 16 GB RAM)
**Storage:**
* **Type**: premium SSD
* **Size**: 256 GB
* **IOPS**: \~7,000-10,000 IOPS (depends on VM tier)
**Configuration:**
* High availability: zone-redundant strongly recommended
* Automated backups: 14-day retention
**Compute tier:**
* **Tier**: general purpose
* **SKU**: `Standard_D8s_v3` (8 vCPU, 32 GB RAM)
**Storage:**
* **Type**: premium SSD v2 (when available) or premium SSD
* **Size**: 1 TB minimum
* **IOPS**: 20,000-40,000 IOPS (customize with premium SSD v2)
**Configuration:**
* High availability: zone-redundant required
* Automated backups: 30-day retention
* Enable query performance insight
We recommend that you use GCP Cloud SQL for PostgreSQL. PostgreSQL has to be v13 or higher.
**Machine type:**
* **Recommended**: `db-custom-2-8192` (2 vCPU, 8 GB RAM)
* Or use `db-standard-2` (2 vCPU, 7.5 GB RAM) for slightly lower cost
**Storage:**
* **Type**: SSD
* **Size**: 50 GB minimum
* **IOPS**: 3,000 IOPS (50 GB × 60 IOPS/GB = 3,000 IOPS)
**Configuration:**
* High availability: recommended for production
* Automated backups: enable with 7-day retention minimum
**Machine type:**
* **Recommended**: `db-custom-4-16384` (4 vCPU, 16 GB RAM)
**Storage:**
* **Type**: SSD
* **Size**: 150 GB minimum
* **IOPS**: 9,000 IOPS (150 GB × 60 IOPS/GB)
**Configuration:**
* High availability: strongly recommended
* Automated backups: 14-day retention
* Enable query insights
**Machine type:**
* **Recommended**: `db-custom-8-32768` (8 vCPU, 32 GB RAM)
**Storage:**
* **Type**: SSD
* **Size**: 500 GB minimum
* **IOPS**: 30,000 IOPS (500 GB × 60 IOPS/GB)
**Configuration:**
* High availability: required
* Automated backups: 30-day retention
* Enable query insights and database flags monitoring
## Monitoring and observability
Regardless of your level of usage, we recommend that you implement these monitoring practices for the database:
### Critical metrics
| Metric | Warning threshold | Critical threshold | Action |
| --------------------------- | ----------------- | ------------------ | ----------------------------------------- |
| **CPU utilization** | >70% sustained | >85% sustained | Scale up compute |
| **Memory (Freeable)** | \<25% free | \<15% free | Scale up memory |
| **IOPS (read/write)** | >80% of limit | >95% of limit | Increase provisioned IOPS or storage size |
| **Disk space** | \<20% free | \<10% free | Increase storage size |
| **Replication lag** (if HA) | >30 seconds | >60 seconds | Check network, investigate load |
Conduktor support is available if you have any concerns on the metrics you are monitoring.
### Cloud-specific tools
* Enable **Performance Insights** (provides query-level analysis)
* Enable **Enhanced Monitoring** (OS-level metrics)
* Create CloudWatch alarms for critical metrics
* Monitor: `ReadIOPS`, `WriteIOPS`, `CPUUtilization`, `FreeableMemory`, `DatabaseConnections`
* [CloudWatch Metrics for RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/monitoring-cloudwatch.html)
* Enable **Query Insights** (slow query analysis)
* Use **Cloud Monitoring** dashboards
* Set up alerting policies in Cloud Monitoring
* Monitor: `database/cpu/utilization`, `database/memory/utilization`, `database/disk/read_ops_count`, `database/disk/write_ops_count`
* [Cloud SQL Monitoring](https://cloud.google.com/sql/docs/postgres/monitoring)
* Enable **Query Performance Insight**
* Use **Azure Monitor** for metrics and alerts
* Enable **Server Parameters** monitoring
* Monitor: `cpu_percent`, `memory_percent`, `io_consumption_percent`, `storage_percent`, `active_connections`
* [Azure PostgreSQL Monitoring](https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-monitoring)
## Scaling and performance
### When to scale up
1. **CPU consistently >70%** for more than 1 hour during business hours
2. **Memory (freeable) \<25%** sustained, indicating index and working set don't fit in RAM
3. **IOPS at >80% of limit** for more than 30 minutes, causing query slowdowns
### Connection pooling
Conduktor Console includes built-in connection pooling.
The default is 15 connections per instance but you can change this using the `CDK_DATABASE_CONNECTION_POOL_SIZE` parameter.
Cloud providers have default connection limits based on the provisioned database instance size.
Verify that your instance type supports your required connection count. As a general rule, you should also allow for a few more (\~10) connections on top of this.
```
max_connections = (Console instances × connections_per_instance) + 10
```
**Example for with 3 Console instances:**
```
max connections = (3 × 15) + 10 = 55
```
## Database maintenance
### Backup and recovery
Our suggested backup requirements are:
* **Automated backups**: enabled with 7-day retention minimum (14-30 days for production)
* **Backup window**: during low-usage periods (e.g., 2-4 AM local time)
* **Point-in-time recovery**: enabled (available on all cloud providers)
* **Cross-region backups**: for disaster recovery (if required by compliance)
* **Manual snapshots**: take a manual backup before upgrades of Conduktor Console
### Upgrade paths
**Scaling compute (vertical scaling):**
All providers support instance size changes with brief downtime (typically 5-15 minutes). Check your cloud provider documentation for full information.
**Scaling storage:**
* **AWS**: storage can be scaled up without downtime (gp3 volumes support online resizing)
* **GCP**: storage automatically scales up; can be manually increased without downtime
* **Azure**: storage can be scaled up without downtime
**IOPS scaling:**
* **AWS**: modify gp3 IOPS or switch to Provisioned IOPS (io1/io2) during a maintenance window
* **GCP**: IOPS scale automatically with storage size
* **Azure**: Premium SSD v2 allows online IOPS adjustment; Premium SSD requires storage tier change
## Cost optimization
1. **Use reserved instances/committed use discounts**: save 30-60% for predictable workloads
2. **Right-size early**: starting oversized and scaling down is difficult; start with recommendations and scale up as needed
3. **Use gp3 storage on AWS**: 20% cheaper than gp2 with better baseline performance, especially for IOPS
4. **Enable multi-AZ only for production**: dev/test environments can use single-AZ to save 50% on instance costs
5. **Monitor idle connections**: ensure connection pooling is working correctly to avoid over-provisioning
## Troubleshoot
Yes, Aurora PostgreSQL is compatible with Conduktor Console and is a good option for fully scaled deployments.
Aurora provides better scalability, automatic failover, and read replicas. Version requirements still apply (14.8+ / 15.3+).
IOPS throttling causes slow queries, timeouts, and potential user-facing errors.
The background metadata sync process is especially sensitive to IOPS limits. Monitor `ReadIOPS` and `WriteIOPS` metrics and scale up before hitting limits.
T-series (AWS) or Burstable tier (Azure) instances can work for standard level installs with low, consistent load.
However, once CPU credits are exhausted, performance degrades significantly. For production, we recommend general purpose instances (M-series on AWS, General Purpose on Azure/GCP) for predictable performance.
Database growth typically depends on:
* the number of Kafka topics, partitions, subjects (schemas), jobs (Kafka Connect) and consumer groups as well as
* number of users, the level of RBAC and the activity level of these users
Monitor your database size monthly to project growth.
Read replicas can help with read-heavy workloads but add complexity. They are typically not needed for Conduktor.
These levels are guidelines. If you have 150 users but 100,000 topics, use mid or fully scaled sizing.
The Kafka scale (topics, consumer groups) drives database size more than user count. When in doubt, start with the next level up and scale down if over-provisioned.
## Related resources
* [Deploy Conduktor Console](/guide/conduktor-in-production/deploy-artifacts/deploy-console)
* [Deploy Console with Kubernetes](/guide/conduktor-in-production/deploy-artifacts/deploy-console/kubernetes)
* [View Console environment variables](/guide/conduktor-in-production/deploy-artifacts/deploy-console/environment-variables)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Conduktor Console configuration examples
Source: https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-console/sample-configuration
Console configuration examples: use API/CLI/Terraform for GitOps production, Console UI for quick changes.
Console supports multiple configuration methods - choose based on your deployment needs:
* **Console API/CLI/Terraform** - recommended for **production and GitOps** environments requiring dynamic configuration management. Enables real-time updates without service interruption.
* **Console UI** - ideal for **development, testing, and quick configuration changes** through the Console interface. Changes are not version-controlled or easily repeatable.
* **YAML/Environment variables** - best for **initial Console setup and static configurations** that rarely change. Requires container restart to apply configuration changes.
This page focuses on **YAML and environment variable** configurations. For API/CLI/Terraform methods [see Console reference](/guide/reference/console-reference).
If you want to configure clusters with a GitOps approach, we recommend using [Console API](https://developers.conduktor.io/?product=console) .
**Clusters created through the UI are independent** and persist regardless of YAML configurations.
We recommend **not mixing configuration methods** to avoid confusion and unwanted overrides.
## Ready-to-use configurations
### Complete production-ready setup for Confluent Cloud
This demonstrates a complete configuration for Conduktor Console including database, monitoring, authentication and Confluent Cloud cluster connections with SASL\_SSL/PLAIN security, Schema Registry, and Kafka Connect.
```yaml theme={null}
database:
hosts:
- host: 'postgresql'
port: 5432
name: 'conduktor'
username: 'conduktor'
password: ''
connection_timeout: 30 # in seconds
monitoring:
cortex-url: 'http://conduktor-monitoring:9009/'
alert-manager-url: 'http://conduktor-monitoring:9009/'
callback-url: 'http://conduktor-console:8080/monitoring/api/'
notifications-callback-url: 'http://localhost:8080'
admin:
email: ''
password: '' # Must be at least 8 characters with mixed case, numbers, and symbols
sso:
oauth2:
- name: 'auth0'
client-id: ''
client-secret: ''
openid:
issuer: 'https://'
scopes: # Optional
- 'openid'
- 'profile'
- 'email'
groups-claim: 'groups' # Optional - Default: 'roles'
auth:
local-users: # Optional - Additional local users beyond admin
- email: 'user@example.com'
password: ''
- email: 'another@example.com'
password: ''
clusters:
- id: 'confluent-prod'
name: 'Confluent Production'
color: '#FF5733' # Optional
icon: 'kafka' # Optional
bootstrapServers: 'pkc-xxxxx.region.aws.confluent.cloud:9092'
properties: |
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="" password="";
kafkaFlavor:
type: "Confluent"
key: "" # Confluent Cloud API Key, NOT cluster API Key
secret: "" # Confluent Cloud API Secret, NOT cluster API Secret
confluentEnvironmentId: ""
confluentClusterId: ""
organizationId: "" # Optional - Required for RBAC role bindings
schemaRegistryId: "" # Optional - Required if managing Schema Registry via API
enableRbacRoleBindings: true # Optional - Default: false
schemaRegistry:
url: 'https://psrc-xxxxx.region.aws.confluent.cloud'
security:
username: ''
password: ''
kafkaConnects:
- id: 'kafka-connect'
name: 'My Kafka Connect'
url: 'http://localhost:8083'
security:
username: ''
password: ''
license: "" # Enterprise license key
```
```bash theme={null}
environment:
CDK_CLUSTERS_0_ID: 'kafka'
CDK_CLUSTERS_0_NAME: 'Kafka'
CDK_CLUSTERS_0_BOOTSTRAPSERVERS: 'localhost:9092'
CDK_CLUSTERS_0_KAFKACONNECTS_0_ID: 'kafka-connect'
CDK_CLUSTERS_0_KAFKACONNECTS_0_NAME: 'My Kafka Connect'
CDK_CLUSTERS_0_KAFKACONNECTS_0_URL: 'http://localhost:8083'
CDK_CLUSTERS_0_KAFKACONNECTS_0_SECURITY_USERNAME: ''
CDK_CLUSTERS_0_KAFKACONNECTS_0_SECURITY_PASSWORD: ''
```
### Amazon MSK with IAM authentication
Connect to an MSK cluster with IAM authentication. You can use explicit credentials or inherit them from the environment.
Deploying this **CloudFormation** template to your environment might result in billable resources being consumed. [See Amazon MSK pricing for details](https://aws.amazon.com/msk/pricing/) .
**Using explicit credentials:**
```yml theme={null}
clusters:
- id: 'amazon-msk-iam'
name: 'Amazon MSK IAM'
bootstrapServers: 'b-3-public.****.kafka.eu-west-1.amazonaws.com:9198'
properties: |
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=io.conduktor.aws.IAMClientCallbackHandler
aws_access_key_id=
aws_secret_access_key=
```
### Complete production-ready setup for Aiven
This demonstrates a complete configuration for Conduktor Console including database, monitoring, authentication and Aiven cluster connections using mTLS with the Aiven flavor.
You should have three files:
* Your access key (in the keystore.jks file).
* Your access certificate (in the keystore.jks file).
* Your CA certificate (in the truststore.jks file).
Make sure the content is on a single line.
```yaml theme={null}
database:
hosts:
- host: 'postgresql'
port: 5432
name: 'conduktor'
username: 'conduktor'
password: ''
connection_timeout: 30 # in seconds
monitoring:
cortex-url: 'http://conduktor-monitoring:9009/'
alert-manager-url: 'http://conduktor-monitoring:9009/'
callback-url: 'http://conduktor-console:8080/monitoring/api/'
notifications-callback-url: 'http://localhost:8080'
admin:
email: ''
password: '' # Must be at least 8 characters with mixed case, numbers, and symbols
sso:
oauth2:
- name: 'auth0'
client-id: ''
client-secret: ''
openid:
issuer: 'https://'
scopes: # Optional
- 'openid'
- 'profile'
- 'email'
groups-claim: 'groups' # Optional - Default: 'roles'
auth:
local-users: # Optional - Additional local users beyond admin
- email: 'user@example.com'
password: ''
- email: 'another@example.com'
password: ''
clusters:
- id: 'aiven-ssl'
name: 'Aiven SSL'
color: '#FF5733' # Optional
icon: 'kafka' # Optional
bootstrapServers: 'kafka-09ba.aivencloud.com:21650'
properties: |
security.protocol=SSL
ssl.truststore.type=PEM
ssl.truststore.certificates=-----BEGIN CERTIFICATE----- -----END CERTIFICATE-----
ssl.keystore.type=PEM
ssl.keystore.key=-----BEGIN PRIVATE KEY----- -----END PRIVATE KEY-----
ssl.keystore.certificate.chain=-----BEGIN CERTIFICATE----- -----END CERTIFICATE-----
kafkaFlavor:
type: "Aiven"
apiToken: ""
project: ""
serviceName: "kafka-xxxx" # kafka cluster id (service name)
license: "" # Enterprise license key
```
```bash theme={null}
environment:
# Enterprise license key
CDK_LICENSE: ''
# Database configuration
CDK_DATABASE_URL: 'postgresql://conduktor:@postgresql:5432/conduktor'
# Connection to Conduktor Cortex container
CDK_MONITORING_CORTEX-URL: 'http://conduktor-monitoring:9009/'
CDK_MONITORING_ALERT-MANAGER-URL: 'http://conduktor-monitoring:9009/'
CDK_MONITORING_CALLBACK-URL: 'http://conduktor-console:8080/monitoring/api/'
CDK_MONITORING_NOTIFICATIONS-CALLBACK-URL: 'http://localhost:8080'
# Admin username/password
CDK_ADMIN_EMAIL: ''
CDK_ADMIN_PASSWORD: '' # Must be at least 8 characters with mixed case, numbers, and symbols
# SSO configuration
CDK_SSO_OAUTH2_0_NAME: 'auth0'
CDK_SSO_OAUTH2_0_CLIENT-ID: ''
CDK_SSO_OAUTH2_0_CLIENT-SECRET: ''
CDK_SSO_OAUTH2_0_OPENID_ISSUER: 'https://'
CDK_SSO_OAUTH2_0_SCOPES: 'openid,profile,email' # Optional - Comma-separated list
CDK_SSO_OAUTH2_0_GROUPS-CLAIM: 'groups' # Optional
# Local users configuration (optional - additional users beyond admin)
CDK_AUTH_LOCAL-USERS_0_EMAIL: 'user@example.com'
CDK_AUTH_LOCAL-USERS_0_PASSWORD: ''
CDK_AUTH_LOCAL-USERS_1_EMAIL: 'another@example.com'
CDK_AUTH_LOCAL-USERS_1_PASSWORD: ''
# Kafka cluster configuration
CDK_CLUSTERS_0_ID: 'aiven-ssl'
CDK_CLUSTERS_0_NAME: 'Aiven SSL'
CDK_CLUSTERS_0_COLOR: '#FF5733' # Optional
CDK_CLUSTERS_0_ICON: 'kafka' # Optional
CDK_CLUSTERS_0_BOOTSTRAPSERVERS: 'kafka-09ba.aivencloud.com:21650'
CDK_CLUSTERS_0_PROPERTIES: "security.protocol=SSL\nssl.truststore.type=PEM\nssl.truststore.certificates=-----BEGIN CERTIFICATE----- -----END CERTIFICATE-----\nssl.keystore.type=PEM\nssl.keystore.key=-----BEGIN PRIVATE KEY----- -----END PRIVATE KEY-----\nssl.keystore.certificate.chain=-----BEGIN CERTIFICATE----- -----END CERTIFICATE-----"
# Aiven flavor configuration
CDK_CLUSTERS_0_KAFKAFLAVOR_TYPE: "Aiven"
CDK_CLUSTERS_0_KAFKAFLAVOR_APITOKEN: ""
CDK_CLUSTERS_0_KAFKAFLAVOR_PROJECT: ""
CDK_CLUSTERS_0_KAFKAFLAVOR_SERVICENAME: "kafka-xxxx"
```
## Kafka Cluster configuration
Basic connection without authentication or encryption.
```yaml theme={null}
clusters:
- id: 'local-kafka'
name: 'Local Development'
bootstrapServers: 'localhost:9092'
```
```bash theme={null}
CDK_CLUSTERS_0_ID: 'local-kafka'
CDK_CLUSTERS_0_NAME: 'Local Development'
CDK_CLUSTERS_0_BOOTSTRAPSERVERS: 'localhost:9092'
```
SASL can be used with PLAINTEXT or SSL transport, supporting multiple mechanisms: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, GSSAPI and OAUTHBEARER.
```yaml theme={null}
clusters:
- id: 'sasl-plain-plaintext'
name: 'SASL PLAIN (Plaintext)'
bootstrapServers: 'broker.example.com:9092'
properties: |
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="" password="";
```
```bash theme={null}
CDK_CLUSTERS_0_ID: 'sasl-plain-plaintext'
CDK_CLUSTERS_0_NAME: 'SASL PLAIN (Plaintext)'
CDK_CLUSTERS_0_BOOTSTRAPSERVERS: 'broker.example.com:9092'
CDK_CLUSTERS_0_PROPERTIES: "security.protocol=SASL_PLAINTEXT\nsasl.mechanism=PLAIN\nsasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username=\"\" password=\"\";"
```
```yaml theme={null}
clusters:
- id: 'sasl-scram-plaintext'
name: 'SASL SCRAM (Plaintext)'
bootstrapServers: 'broker.example.com:9092'
properties: |
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-256
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="" password="";
```
```bash theme={null}
CDK_CLUSTERS_0_ID: 'sasl-scram-plaintext'
CDK_CLUSTERS_0_NAME: 'SASL SCRAM (Plaintext)'
CDK_CLUSTERS_0_BOOTSTRAPSERVERS: 'broker.example.com:9092'
CDK_CLUSTERS_0_PROPERTIES: "security.protocol=SASL_PLAINTEXT\nsasl.mechanism=SCRAM-SHA-256\nsasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username=\"\" password=\"\";"
```
```yaml theme={null}
clusters:
- id: 'sasl-plain-ssl'
name: 'SASL PLAIN (SSL)'
bootstrapServers: 'broker.example.com:9093'
properties: |
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="" password="";
```
```bash theme={null}
CDK_CLUSTERS_0_ID: 'sasl-plain-ssl'
CDK_CLUSTERS_0_NAME: 'SASL PLAIN (SSL)'
CDK_CLUSTERS_0_BOOTSTRAPSERVERS: 'broker.example.com:9093'
CDK_CLUSTERS_0_PROPERTIES: "security.protocol=SASL_SSL\nsasl.mechanism=PLAIN\nsasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username=\"\" password=\"\";"
```
```yaml theme={null}
clusters:
- id: 'sasl-scram-ssl'
name: 'SASL SCRAM (SSL)'
bootstrapServers: 'broker.example.com:9093'
properties: |
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="" password="";
ssl.truststore.type=PEM
ssl.truststore.certificates=-----BEGIN CERTIFICATE----- -----END CERTIFICATE-----
```
```bash theme={null}
CDK_CLUSTERS_0_ID: 'sasl-scram-ssl'
CDK_CLUSTERS_0_NAME: 'SASL SCRAM (SSL)'
CDK_CLUSTERS_0_BOOTSTRAPSERVERS: 'broker.example.com:9093'
CDK_CLUSTERS_0_PROPERTIES: "security.protocol=SASL_SSL\nsasl.mechanism=SCRAM-SHA-512\nsasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username=\"\" password=\"\";\nssl.truststore.type=PEM\nssl.truststore.certificates=-----BEGIN CERTIFICATE----- -----END CERTIFICATE-----"
```
OIDC authentication (available since Kafka 3.1 - [KIP-768](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=186877575))
```yaml theme={null}
clusters:
- id: 'sasl-oauth-ssl'
name: 'SASL OAuth (SSL)'
bootstrapServers: 'broker.example.com:9093'
properties: |
security.protocol=SASL_SSL
sasl.mechanism=OAUTHBEARER
sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required clientId="" clientSecret="";
sasl.oauthbearer.token.endpoint.url=https://auth.example.com/oauth2/token
sasl.login.callback.handler.class=org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler
```
```bash theme={null}
CDK_CLUSTERS_0_ID: 'sasl-oauth-ssl'
CDK_CLUSTERS_0_NAME: 'SASL OAuth (SSL)'
CDK_CLUSTERS_0_BOOTSTRAPSERVERS: 'broker.example.com:9093'
CDK_CLUSTERS_0_PROPERTIES: "security.protocol=SASL_SSL\nsasl.mechanism=OAUTHBEARER\nsasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required clientId=\"\" clientSecret=\"\";\nsasl.oauthbearer.token.endpoint.url=https://auth.example.com/oauth2/token\nsasl.login.callback.handler.class=org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler"
```
Server authentication only.
```yaml theme={null}
clusters:
- id: 'ssl-oneway'
name: 'SSL One-way TLS'
bootstrapServers: 'broker.example.com:9093'
properties: |
security.protocol=SSL
ssl.truststore.type=PEM
ssl.truststore.certificates=-----BEGIN CERTIFICATE----- -----END CERTIFICATE-----
```
```bash theme={null}
CDK_CLUSTERS_0_ID: 'ssl-oneway'
CDK_CLUSTERS_0_NAME: 'SSL One-way TLS'
CDK_CLUSTERS_0_BOOTSTRAPSERVERS: 'broker.example.com:9093'
CDK_CLUSTERS_0_PROPERTIES: "security.protocol=SSL\nssl.truststore.type=PEM\nssl.truststore.certificates=-----BEGIN CERTIFICATE----- -----END CERTIFICATE-----"
```
Client and server authentication.
```yaml theme={null}
clusters:
- id: 'ssl-mtls'
name: 'SSL Mutual TLS'
bootstrapServers: 'broker.example.com:9093'
properties: |
security.protocol=SSL
ssl.truststore.type=PEM
ssl.truststore.certificates=-----BEGIN CERTIFICATE----- -----END CERTIFICATE-----
ssl.keystore.type=PEM
ssl.keystore.key=-----BEGIN PRIVATE KEY----- -----END PRIVATE KEY-----
ssl.keystore.certificate.chain=-----BEGIN CERTIFICATE----- -----END CERTIFICATE-----
```
```bash theme={null}
CDK_CLUSTERS_0_ID: 'ssl-mtls'
CDK_CLUSTERS_0_NAME: 'SSL Mutual TLS'
CDK_CLUSTERS_0_BOOTSTRAPSERVERS: 'broker.example.com:9093'
CDK_CLUSTERS_0_PROPERTIES: "security.protocol=SSL\nssl.truststore.type=PEM\nssl.truststore.certificates=-----BEGIN CERTIFICATE----- -----END CERTIFICATE-----\nssl.keystore.type=PEM\nssl.keystore.key=-----BEGIN PRIVATE KEY----- -----END PRIVATE KEY-----\nssl.keystore.certificate.chain=-----BEGIN CERTIFICATE----- -----END CERTIFICATE-----"
```
Credentials inherited from environment.
```yaml theme={null}
clusters:
- id: 'aws-msk-iam'
name: 'AWS MSK with IAM'
bootstrapServers: 'b-1.cluster.kafka.region.amazonaws.com:9098'
properties: |
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
```
```bash theme={null}
CDK_CLUSTERS_0_ID: 'aws-msk-iam'
CDK_CLUSTERS_0_NAME: 'AWS MSK with IAM'
CDK_CLUSTERS_0_BOOTSTRAPSERVERS: 'b-1.cluster.kafka.region.amazonaws.com:9098'
CDK_CLUSTERS_0_PROPERTIES: "security.protocol=SASL_SSL\nsasl.mechanism=AWS_MSK_IAM\nsasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;\nsasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler"
```
You can also override the `default` profile or role:
* Profile: `sasl.jaas.config = software.amazon.msk.auth.iam.IAMLoginModule required awsProfileName="";`
* Role: `sasl.jaas.config = software.amazon.msk.auth.iam.IAMLoginModule required awsRoleArn="";`
Credentials from explicit configuration.
```yaml theme={null}
clusters:
- id: 'aws-msk-iam'
name: 'AWS MSK with IAM'
bootstrapServers: 'b-1.cluster.kafka.region.amazonaws.com:9098'
properties: |
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=io.conduktor.aws.IAMClientCallbackHandler
aws_access_key_id=
aws_secret_access_key=
```
```bash theme={null}
CDK_CLUSTERS_0_ID: 'aws-msk-iam'
CDK_CLUSTERS_0_NAME: 'AWS MSK with IAM'
CDK_CLUSTERS_0_BOOTSTRAPSERVERS: 'b-1.cluster.kafka.region.amazonaws.com:9098'
CDK_CLUSTERS_0_PROPERTIES: "security.protocol=SASL_SSL\nsasl.mechanism=AWS_MSK_IAM\nsasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;\nsasl.client.callback.handler.class=io.conduktor.aws.IAMClientCallbackHandler\naws_access_key_id=\naws_secret_access_key="
```
## Schema Registry configuration
To enable Schema Registry support, attach these code examples to any of the cluster configurations above.
```yaml theme={null}
schemaRegistry:
url: 'https://psrc-xxxx.region.aws.confluent.cloud'
```
```bash theme={null}
CDK_CLUSTERS_0_SCHEMAREGISTRY_URL: 'https://psrc-xxxx.region.aws.confluent.cloud'
```
```yaml theme={null}
schemaRegistry:
url: 'https://psrc-xxxxx.region.aws.confluent.cloud'
security:
username: ''
password: ''
```
```bash theme={null}
CDK_CLUSTERS_0_SCHEMAREGISTRY_URL: 'https://psrc-xxxxx.region.aws.confluent.cloud'
CDK_CLUSTERS_0_SCHEMAREGISTRY_SECURITY_USERNAME: ''
CDK_CLUSTERS_0_SCHEMAREGISTRY_SECURITY_PASSWORD: ''
```
```yaml theme={null}
schemaRegistry:
url: 'https://psrc-xxxxx.region.aws.confluent.cloud'
security:
token: ''
```
```bash theme={null}
CDK_CLUSTERS_0_SCHEMAREGISTRY_URL: 'https://psrc-xxxxx.region.aws.confluent.cloud'
CDK_CLUSTERS_0_SCHEMAREGISTRY_SECURITY_TOKEN: ''
```
```yaml theme={null}
schemaRegistry:
url: 'https://psrc-xxxxx.region.aws.confluent.cloud'
security:
key: -----BEGIN PRIVATE KEY----- -----END PRIVATE KEY-----
certificateChain: -----BEGIN CERTIFICATE----- -----END CERTIFICATE-----
```
```bash theme={null}
CDK_CLUSTERS_0_SCHEMAREGISTRY_URL: 'https://psrc-xxxxx.region.aws.confluent.cloud'
CDK_CLUSTERS_0_SCHEMAREGISTRY_SECURITY_KEY: '-----BEGIN PRIVATE KEY----- -----END PRIVATE KEY-----'
CDK_CLUSTERS_0_SCHEMAREGISTRY_SECURITY_CERTIFICATECHAIN: '-----BEGIN CERTIFICATE----- -----END CERTIFICATE-----'
```
Connect MSK clusters with AWS Glue Schema Registry using different authentication methods.
```yaml theme={null}
schemaRegistry:
region: ''
registryName: '' # optional
amazonSecurity:
type: 'Credentials'
accessKeyId: ''
secretKey: ''
```
```bash theme={null}
CDK_CLUSTERS_0_SCHEMAREGISTRY_REGION: ''
CDK_CLUSTERS_0_SCHEMAREGISTRY_REGISTRYNAME: '' # optional
CDK_CLUSTERS_0_SCHEMAREGISTRY_AMAZONSECURITY_TYPE: 'Credentials'
CDK_CLUSTERS_0_SCHEMAREGISTRY_AMAZONSECURITY_ACCESSKEYID: ''
CDK_CLUSTERS_0_SCHEMAREGISTRY_AMAZONSECURITY_SECRETKEY: ''
```
```yaml theme={null}
schemaRegistry:
region: ''
registryName: '' # optional
amazonSecurity:
type: 'FromContext'
profile: '' # optional, inherited from environment by default
```
```bash theme={null}
CDK_CLUSTERS_0_SCHEMAREGISTRY_REGION: ''
CDK_CLUSTERS_0_SCHEMAREGISTRY_AMAZONSECURITY_TYPE: 'FromContext'
CDK_CLUSTERS_0_SCHEMAREGISTRY_AMAZONSECURITY_PROFILE: '' # optional, inherited from environment by default
```
```yaml theme={null}
schemaRegistry:
region: ''
registryName: '' # optional
amazonSecurity:
type: 'FromRole'
role: ''
```
```bash theme={null}
CDK_CLUSTERS_0_SCHEMAREGISTRY_REGION: ''
CDK_CLUSTERS_0_SCHEMAREGISTRY_AMAZONSECURITY_TYPE: 'FromRole'
CDK_CLUSTERS_0_SCHEMAREGISTRY_AMAZONSECURITY_ROLE: ''
```
## Kafka Connect configuration
To add Kafka Connect to your cluster configuration use the code examples below.
```yaml theme={null}
kafkaConnects:
- id: 'kafka-connect'
name: 'My Kafka Connect'
url: 'http://localhost:8083'
headers: 'myHeader=myValue'
ignoreUntrustedCertificate: false
```
```bash theme={null}
CDK_CLUSTERS_0_KAFKACONNECTS_0_ID: 'kafka-connect'
CDK_CLUSTERS_0_KAFKACONNECTS_0_NAME: 'My Kafka Connect'
CDK_CLUSTERS_0_KAFKACONNECTS_0_URL: 'http://localhost:8083'
CDK_CLUSTERS_0_KAFKACONNECTS_0_HEADERS: 'myHeader=myValue'
CDK_CLUSTERS_0_KAFKACONNECTS_0_IGNOREUNTRUSTEDCERTIFICATE: 'false'
```
```yaml theme={null}
kafkaConnects:
- id: 'kafka-connect'
name: 'My Kafka Connect'
url: 'http://localhost:8083'
headers: 'myHeader=myValue'
ignoreUntrustedCertificate: false
security:
username: ''
password: ''
```
```bash theme={null}
CDK_CLUSTERS_0_KAFKACONNECTS_0_ID: 'kafka-connect'
CDK_CLUSTERS_0_KAFKACONNECTS_0_NAME: 'My Kafka Connect'
CDK_CLUSTERS_0_KAFKACONNECTS_0_URL: 'http://localhost:8083'
CDK_CLUSTERS_0_KAFKACONNECTS_0_HEADERS: 'myHeader=myValue'
CDK_CLUSTERS_0_KAFKACONNECTS_0_IGNOREUNTRUSTEDCERTIFICATE: 'false'
CDK_CLUSTERS_0_KAFKACONNECTS_0_SECURITY_USERNAME: ''
CDK_CLUSTERS_0_KAFKACONNECTS_0_SECURITY_PASSWORD: ''
```
```yaml theme={null}
kafkaConnects:
- id: 'kafka-connect'
name: 'My Kafka Connect'
url: 'http://localhost:8083'
headers: 'myHeader=myValue'
ignoreUntrustedCertificate: false
security:
token: ''
```
```bash theme={null}
CDK_CLUSTERS_0_KAFKACONNECTS_0_ID: 'kafka-connect'
CDK_CLUSTERS_0_KAFKACONNECTS_0_NAME: 'My Kafka Connect'
CDK_CLUSTERS_0_KAFKACONNECTS_0_URL: 'http://localhost:8083'
CDK_CLUSTERS_0_KAFKACONNECTS_0_HEADERS: 'myHeader=myValue'
CDK_CLUSTERS_0_KAFKACONNECTS_0_IGNOREUNTRUSTEDCERTIFICATE: 'false'
CDK_CLUSTERS_0_KAFKACONNECTS_0_SECURITY_TOKEN: ''
```
```yaml theme={null}
kafkaConnects:
- id: 'kafka-connect'
name: 'My Kafka Connect'
url: 'http://localhost:8083'
headers: 'myHeader=myValue'
ignoreUntrustedCertificate: false
security:
key: -----BEGIN PRIVATE KEY----- -----END PRIVATE KEY-----
certificateChain: -----BEGIN CERTIFICATE----- -----END CERTIFICATE-----
```
```bash theme={null}
CDK_CLUSTERS_0_KAFKACONNECTS_0_ID: 'kafka-connect'
CDK_CLUSTERS_0_KAFKACONNECTS_0_NAME: 'My Kafka Connect'
CDK_CLUSTERS_0_KAFKACONNECTS_0_URL: 'http://localhost:8083'
CDK_CLUSTERS_0_KAFKACONNECTS_0_HEADERS: 'myHeader=myValue'
CDK_CLUSTERS_0_KAFKACONNECTS_0_IGNOREUNTRUSTEDCERTIFICATE: 'false'
CDK_CLUSTERS_0_KAFKACONNECTS_0_SECURITY_KEY: '-----BEGIN PRIVATE KEY----- -----END PRIVATE KEY-----'
CDK_CLUSTERS_0_KAFKACONNECTS_0_SECURITY_CERTIFICATECHAIN: '-----BEGIN CERTIFICATE----- -----END CERTIFICATE-----'
```
## ksqlDB configuration
OAUTHBEARER with OIDC Authentication is possible since Kafka 3.1 and [KIP-768](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=186877575) . To demonstrate OIDC authentication, you can connect to NASA's GCN Kafka cluster after you [sign up](https://gcn.nasa.gov/quickstart) . Here's a configuration example (adapt the values to your needs):
```yaml theme={null}
ksqlDBs:
- id: 'ksqldb-basic'
name: 'My ksqlDB Server'
url: 'http://localhost:8088'
ignoreUntrustedCertificate: false
headers: 'myHeader=myValue'
```
```bash theme={null}
CDK_CLUSTERS_0_KSQLDBS_0_ID: 'ksqldb-basic'
CDK_CLUSTERS_0_KSQLDBS_0_NAME: 'My ksqlDB Server'
CDK_CLUSTERS_0_KSQLDBS_0_URL: 'http://localhost:8088'
CDK_CLUSTERS_0_KSQLDBS_0_IGNOREUNTRUSTEDCERTIFICATE: 'false'
CDK_CLUSTERS_0_KSQLDBS_0_HEADERS: 'myHeader=myValue'
```
```yaml theme={null}
ksqlDBs:
- id: 'ksqldb-basic-auth'
name: 'My ksqlDB Server'
url: 'http://localhost:8088'
ignoreUntrustedCertificate: false
headers: 'myHeader=myValue'
security:
username: ''
password: ''
```
```bash theme={null}
CDK_CLUSTERS_0_KSQLDBS_0_ID: 'ksqldb-basic-auth'
CDK_CLUSTERS_0_KSQLDBS_0_NAME: 'My ksqlDB Server'
CDK_CLUSTERS_0_KSQLDBS_0_URL: 'http://localhost:8088'
CDK_CLUSTERS_0_KSQLDBS_0_IGNOREUNTRUSTEDCERTIFICATE: 'false'
CDK_CLUSTERS_0_KSQLDBS_0_HEADERS: 'myHeader=myValue'
CDK_CLUSTERS_0_KSQLDBS_0_SECURITY_USERNAME: ''
CDK_CLUSTERS_0_KSQLDBS_0_SECURITY_PASSWORD: ''
```
```yaml theme={null}
ksqlDBs:
- id: 'ksqldb-token-auth'
name: 'My ksqlDB Server'
url: 'http://localhost:8088'
ignoreUntrustedCertificate: false
headers: 'myHeader=myValue'
security:
token: ''
```
```bash theme={null}
CDK_CLUSTERS_0_KSQLDBS_0_ID: 'ksqldb-token-auth'
CDK_CLUSTERS_0_KSQLDBS_0_NAME: 'My ksqlDB Server'
CDK_CLUSTERS_0_KSQLDBS_0_URL: 'http://localhost:8088'
CDK_CLUSTERS_0_KSQLDBS_0_IGNOREUNTRUSTEDCERTIFICATE: 'false'
CDK_CLUSTERS_0_KSQLDBS_0_HEADERS: 'myHeader=myValue'
CDK_CLUSTERS_0_KSQLDBS_0_SECURITY_TOKEN: ''
```
```yaml theme={null}
ksqlDBs:
- id: 'ksqldb-ssl-auth'
name: 'My ksqlDB Server'
url: 'http://localhost:8088'
ignoreUntrustedCertificate: false
headers: 'myHeader=myValue'
security:
key: -----BEGIN PRIVATE KEY----- -----END PRIVATE KEY-----
certificateChain: -----BEGIN CERTIFICATE----- -----END CERTIFICATE-----
```
```bash theme={null}
CDK_CLUSTERS_0_KSQLDBS_0_ID: 'ksqldb-ssl-auth'
CDK_CLUSTERS_0_KSQLDBS_0_NAME: 'My ksqlDB Server'
CDK_CLUSTERS_0_KSQLDBS_0_URL: 'http://localhost:8088'
CDK_CLUSTERS_0_KSQLDBS_0_IGNOREUNTRUSTEDCERTIFICATE: 'false'
CDK_CLUSTERS_0_KSQLDBS_0_HEADERS: 'myHeader=myValue'
CDK_CLUSTERS_0_KSQLDBS_0_SECURITY_KEY: '-----BEGIN PRIVATE KEY----- -----END PRIVATE KEY-----'
CDK_CLUSTERS_0_KSQLDBS_0_SECURITY_CERTIFICATECHAIN: '-----BEGIN CERTIFICATE----- -----END CERTIFICATE-----'
```
## Provider configuration
To enable enhanced provider-specific capabilities, attach the following snippets to any of the above cluster configurations.
Connect to Confluent Cloud with enhanced management capabilities for service accounts, API keys, and ACLs.
```yaml theme={null}
kafkaFlavor:
type: "Confluent"
key: "" # Confluent Cloud API Key, NOT cluster API Key
secret: "" # Confluent Cloud API Secret, NOT cluster API Secret
confluentEnvironmentId: ""
confluentClusterId: ""
organizationId: "" # Optional - Required for RBAC role bindings
schemaRegistryId: "" # Optional - Required if managing Schema Registry via API
enableRbacRoleBindings: true # Optional - Default: false
```
```bash theme={null}
CDK_CLUSTERS_0_KAFKAFLAVOR_TYPE: "Confluent"
CDK_CLUSTERS_0_KAFKAFLAVOR_KEY: ""
CDK_CLUSTERS_0_KAFKAFLAVOR_SECRET: ""
CDK_CLUSTERS_0_KAFKAFLAVOR_CONFLUENTENVIRONMENTID: ""
CDK_CLUSTERS_0_KAFKAFLAVOR_CONFLUENTCLUSTERID: ""
CDK_CLUSTERS_0_KAFKAFLAVOR_ORGANIZATIONID: "" # Optional
CDK_CLUSTERS_0_KAFKAFLAVOR_SCHEMAREGISTRYID: "" # Optional
CDK_CLUSTERS_0_KAFKAFLAVOR_ENABLERBACROLEBINDINGS: "true" # Optional
```
Connect to Confluent Platform with enhanced management capabilities for service accounts and RBAC role bindings via the Metadata Service (MDS).
```yaml theme={null}
kafkaFlavor:
type: "ConfluentPlatform"
kafkaClusterId: ""
mdsUrl: "https://mds.example.com:8090" # Optional - Required for RBAC role bindings
authentication:
username: "" # LDAP user with SystemAdmin privileges
password: ""
schemaRegistryClusterId: "" # Optional - Required for subject role bindings
enableRbacRoleBindings: true # Optional - Default: false
ignoreUntrustedCertificate: false # Optional - Default: false
```
```bash theme={null}
CDK_CLUSTERS_0_KAFKAFLAVOR_TYPE: "ConfluentPlatform"
CDK_CLUSTERS_0_KAFKAFLAVOR_KAFKACLUSTERID: ""
CDK_CLUSTERS_0_KAFKAFLAVOR_MDSURL: "https://mds.example.com:8090" # Optional
CDK_CLUSTERS_0_KAFKAFLAVOR_AUTHENTICATION_USERNAME: ""
CDK_CLUSTERS_0_KAFKAFLAVOR_AUTHENTICATION_PASSWORD: ""
CDK_CLUSTERS_0_KAFKAFLAVOR_SCHEMAREGISTRYCLUSTERID: "" # Optional
CDK_CLUSTERS_0_KAFKAFLAVOR_ENABLERBACROLEBINDINGS: "true" # Optional
CDK_CLUSTERS_0_KAFKAFLAVOR_IGNOREUNTRUSTEDCERTIFICATE: "false" # Optional
```
Connect to Aiven Kafka with enhanced management capabilities for service accounts and ACLs.
```yaml theme={null}
kafkaFlavor:
type: "Aiven"
apiToken: ""
project: ""
serviceName: "kafka-18350d67" # kafka cluster id (service name)
```
```bash theme={null}
CDK_CLUSTERS_0_KAFKAFLAVOR_TYPE: "Aiven"
CDK_CLUSTERS_0_KAFKAFLAVOR_APITOKEN: ""
CDK_CLUSTERS_0_KAFKAFLAVOR_PROJECT: ""
CDK_CLUSTERS_0_KAFKAFLAVOR_SERVICENAME: "kafka-18350d67"
```
Connect Console to Conduktor Gateway for centralized Interceptor management through the Console UI.
```yaml theme={null}
kafkaFlavor:
type: "Gateway"
url: "http://conduktor-gateway:8888"
user: ""
password: ""
virtualCluster: "passthrough"
```
```bash theme={null}
CDK_CLUSTERS_0_KAFKAFLAVOR_TYPE: "Gateway"
CDK_CLUSTERS_0_KAFKAFLAVOR_URL: "http://conduktor-gateway:8888"
CDK_CLUSTERS_0_KAFKAFLAVOR_USER: ""
CDK_CLUSTERS_0_KAFKAFLAVOR_PASSWORD: ""
CDK_CLUSTERS_0_KAFKAFLAVOR_VIRTUALCLUSTER: "passthrough"
```
## Logging configuration
### Global log settings
Configure Console-wide logging behavior using these environment variables:
| Environment variable | Default value | Description |
| --------------------- | ------------- | ------------------------------------------------------------------------ |
| `CDK_ROOT_LOG_LEVEL` | `INFO` | Global Console log level, one of `OFF`, `ERROR`, `WARN`, `INFO`, `DEBUG` |
| `CDK_ROOT_LOG_FORMAT` | `TEXT` | Log format, one of `TEXT` or `JSON` |
| `CDK_ROOT_LOG_COLOR` | `true` | Enable color in logs when possible |
For backward compatibility, `CDK_DEBUG: true` is still supported and is equivalent to `CDK_ROOT_LOG_LEVEL: DEBUG`.
### Module-specific log settings
Configure logging levels for individual Console modules:
Possible values for all of them are: `OFF`, `ERROR`, `WARN`, `INFO`, `DEBUG`, and `TRACE`.
| Environment variable | Default value | Description |
| ----------------------------- | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `PLATFORM_STARTUP_LOG_LEVEL` | `INFO` | Set the setup/configuration process logs level. By default, it is set to `INFO`, but switches to `DEBUG` if `CDK_ROOT_LOG_LEVEL: DEBUG`. |
| `CONSOLE_ROOT_LOG_LEVEL` | `CDK_ROOT_LOG_LEVEL` | Logs related to any actions done in the Console UI |
| `PLATFORM_API_ROOT_LOG_LEVEL` | `CDK_ROOT_LOG_LEVEL` | Internal platform API logs (health endpoints) |
### Log level inheritance
If you don't explicitly set the log level for a module, it will inherit the `CDK_ROOT_LOG_LEVEL`.
For instance, if you only set
```yaml theme={null}
CDK_ROOT_LOG_LEVEL: DEBUG
# CONSOLE_ROOT_LOG_LEVEL isn't set
```
Then, `CONSOLE_ROOT_LOG_LEVEL` will be automatically set to `DEBUG`.
Similarly, if you set:
```yaml theme={null}
CDK_ROOT_LOG_LEVEL: INFO
CONSOLE_ROOT_LOG_LEVEL: DEBUG
```
Then, `CONSOLE_ROOT_LOG_LEVEL` will still be set to `DEBUG`, and isn't overridden.
If you want to further customize your logging at an individual logger-level, you can use a per-module logback configuration file.
By default, all logback configuration files are in **/opt/conduktor/loggers/** with `READ-ONLY` permissions.
At startup, Console will copy all (missing) logback configuration files from `/opt/conduktor/loggers/` to `/var/conduktor/configs/loggers/` directory with `READ-WRITE` permissions.
Because all logback configuration files are set to reload themselves every 15 seconds, you can then edit them inside the container volume in **/var/conduktor/configs/loggers/** to tune log level per logger.
All logback configuration files declare some expected appenders:
| Appender name | Description |
| -------------------- | ------------------------------------------------------------- |
| `STDOUT` | Appender that writes logs to stdout |
| `STDOUT_COLOR` | Appender that writes logs to stdout with color |
| `ASYNC_STDOUT` | Appender that writes logs to stdout asynchronously |
| `ASYNC_STDOUT_COLOR` | Appender that writes logs to stdout asynchronously with color |
### JSON structured logging
Enable structured logging by setting `CDK_ROOT_LOG_FORMAT=JSON`. Logs will use this JSON format:
```json theme={null}
{
"timestamp": "2024-06-14T10:09:25.802542476+00:00",
"level": "",
"message": "",
"logger": "",
"thread": "",
"stack_trace": "",
"mdc": {
"key": "value"
}
}
```
The log `timestamp` is encoded in [ISO-8601 format](https://en.wikipedia.org/wiki/ISO_8601) . When structured logging is enabled, `CDK_ROOT_LOG_COLOR` is always ignored.
### Runtime logger API
Console provides runtime log level management via [REST API](https://developers.conduktor.io/?product=console#tag/logging). **This requires an admin API key**.
The `loggerName` filter used to GET or SET a logger level uses a **contains** so you can either use the fully qualified cardinal name or just a part of it, meaning that the filter `authenticator` will match `io.conduktor.authenticator` or `io.conduktor.authenticator.ConduktorUserProfile` loggers, among others.
The `logLevel` is **case-insensitive** and can be: `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, `OFF`.
# Cortex deployment for Conduktor Console monitoring
Source: https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-cortex
Deploy conduktor-console-cortex for Kafka monitoring in Conduktor Console. Includes Prometheus and Alertmanager.
To enable monitoring via Console, you can deploy and configure the `conduktor-console-cortex` component or use your own compatible monitoring solution (since Console v1.38).
We support **Cortex**, **Mimir** and **Prometheus** as alternative solutions.
This guide focuses on deploying the `conduktor-console-cortex` component, which is a preconfigured Cortex that includes Prometheus and Alertmanager.
## Deploy and configure Cortex
The only required property is `CDK_CONSOLE-URL`, everything else is related to storage for the metrics.
By default, data will be stored in `/var/conduktor/monitoring` inside the running image. You can mount a volume on this folder to keep metrics data between updates. Alternatively, you can use the storage parameters described below to store the data using either `s3`, `gcs`, `azure` or `swift`.
| Environment variable | Description | Mandatory | Type | Default | Since |
| --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | ------ | ----------------------------------------- | -------- |
| `CDK_CONSOLEURL` | Console URL and port (example: `"http://conduktor-console:8080"`). | true | string | ∅ | `1.18.0` |
| `CDK_SCRAPER_SKIPSSLCHECK` | Disable TLS check when scraping metrics from Console. | false | bool | `false` | `1.18.2` |
| `CDK_SCRAPER_CAFILE` | Path to CA certificate file inside the container to perform TLS check when scraping metrics from Console. | false | string | ∅ | `1.18.2` |
| `CDK_SCRAPER_BASICAUTH_EMAIL` | Email for basic auth when scraping metrics from Console. Has to match `CDK_MONITORING_BASICAUTH_EMAIL` on Console. | false | string | ∅ | `1.42.0` |
| `CDK_SCRAPER_BASICAUTH_PASSWORD` | Password for basic auth when scraping metrics from Console. Has to match `CDK_MONITORING_BASICAUTH_PASSWORD` on Console. | false | string | ∅ | `1.42.0` |
| **S3** | | | | | |
| `CDK_MONITORING_STORAGE_S3_ENDPOINT` | S3 storage endpoint. | false | string | ∅ | `1.18.0` |
| `CDK_MONITORING_STORAGE_S3_REGION` | S3 storage region. | false | string | ∅ | `1.18.0` |
| `CDK_MONITORING_STORAGE_S3_BUCKET` | S3 storage bucket name. | true | string | ∅ | `1.18.0` |
| `CDK_MONITORING_STORAGE_S3_INSECURE` | S3 storage SSL/TLS check flag. | false | bool | `false` | `1.18.0` |
| `CDK_MONITORING_STORAGE_S3_ACCESSKEYID` | Access key ID of an AWS IAM identity for monitoring to upload logs to S3. If set to false, and if you are running on a Kubernetes deployment, Monitoring can attempt to leverage AWS IRSA for the Pod’s service account when connecting to S3. | false | string | ∅ | `1.18.0` |
| `CDK_MONITORING_STORAGE_S3_SECRETACCESSKEY` | Secret access key of an AWS IAM identity for monitoring to upload logs to S3. If set to false, and if you are running on a Kubernetes deployment, Monitoring can attempt to leverage AWS IRSA for the Pod’s service account when connecting to S3. | false | string | ∅ | `1.18.0` |
| **GCS** | | | | | |
| `CDK_MONITORING_STORAGE_GCS_BUCKETNAME` | GCS storage bucket name. | true | string | ∅ | `1.18.0` |
| `CDK_MONITORING_STORAGE_GCS_SERVICEACCOUNT` | GCS storage service account JSON content. | true | string | ∅ | `1.18.0` |
| **Azure** | | | | | |
| `CDK_MONITORING_STORAGE_AZURE_ACCOUNTNAME` | Azure storage account name. | true | string | ∅ | `1.18.0` |
| `CDK_MONITORING_STORAGE_AZURE_ACCOUNTKEY` | Azure storage account key. | true | string | ∅ | `1.18.0` |
| `CDK_MONITORING_STORAGE_AZURE_CONTAINERNAME` | Azure storage container name. | true | string | ∅ | `1.18.0` |
| `CDK_MONITORING_STORAGE_AZURE_ENDPOINTSUFFIX` | Azure storage endpoint suffix. | false | string | `blob.core.windows.net` | `1.18.0` |
| **Swift** | | | | | |
| `CDK_MONITORING_STORAGE_SWIFT_AUTHURL` | Swift storage authentication URL. | true | string | ∅ | `1.18.0` |
| `CDK_MONITORING_STORAGE_SWIFT_PASSWORD` | Swift storage user password. | true | string | ∅ | `1.18.0` |
| `CDK_MONITORING_STORAGE_SWIFT_CONTAINERNAME` | Swift storage container name. | true | string | ∅ | `1.18.0` |
| `CDK_MONITORING_STORAGE_SWIFT_USERID` | Swift storage user ID. | false | string | ∅ | `1.18.0` |
| `CDK_MONITORING_STORAGE_SWIFT_USERNAME` | Swift storage user name. | false | string | ∅ | `1.18.0` |
| `CDK_MONITORING_STORAGE_SWIFT_USERDOMAINNAME` | Swift storage user domain name. | false | string | ∅ | `1.18.0` |
| `CDK_MONITORING_STORAGE_SWIFT_USERDOMAINID` | Swift storage user domain ID. | false | string | ∅ | `1.18.0` |
| `CDK_MONITORING_STORAGE_SWIFT_DOMAINID` | Swift storage user domain ID. | false | string | ∅ | `1.18.0` |
| `CDK_MONITORING_STORAGE_SWIFT_DOMAINNAME` | Swift storage user domain name. | false | string | ∅ | `1.18.0` |
| `CDK_MONITORING_STORAGE_SWIFT_PROJECTID` | Swift storage project ID. | false | string | ∅ | `1.18.0` |
| `CDK_MONITORING_STORAGE_SWIFT_REGIONNAME` | Swift storage region name. | false | string | ∅ | `1.18.0` |
| **Logs** | | | | | |
| `CORTEX_ROOT_LOG_LEVEL` | Cortex log level. | false | string | `info` | `1.18.0` |
| `CORTEX_ALERT_ROOT_LOG_LEVEL` | Alert manager log level. | false | string | `info` | `1.18.0` |
| `PROMETHEUS_ROOT_LOG_LEVEL` | Prometheus log level. | false | string | `info` | `1.18.0` |
| **Retention configuration** | | | | | |
| `PROMETHEUS_RETENTION_SIZE` | Prometheus `storage.tsdb.retention.size` startup argument that defines retention size. | false | string | `5MB` | `1.37.0` |
| `PROMETHEUS_RETENTION_TIME` | Prometheus `storage.tsdb.retention.time` startup argument that defines retention time. | false | string | `30m` | `1.37.0` |
| `PROMETHEUS_BLOCK_DURATION` | Prometheus `storage.tsdb.min-block-duration` and `storage.tsdb.max-block-duration` startup argument format "min,max". | false | string | `5m,30m` | `1.37.0` |
| `PROMETHEUS_SCRAPE_INTERVAL` | Prometheus `global.scrape_interval` in configuration. | false | string | `15s` | `1.37.0` |
| `PROMETHEUS_EVAL_INTERVAL` | Prometheus `global.evaluation_interval` in configuration. | false | string | `15s` | `1.37.0` |
| `CORTEX_BLOCKS_RANGES` | Cortex block ranges configuration (`blocks_storage.tsdb.block_ranges_period`). | false | string | `1h` | `1.37.0` |
| `CORTEX_RETENTION_PERIOD` | Cortex local block retention before flushing to external storage (`blocks_storage.tsdb.retention_period`). | false | string | `3h` | `1.37.0` |
| `CORTEX_COMPACTOR_BLOCKS_RANGES` | Cortex compactor blocks sizes configuration (`compactor.block_ranges`). | false | string | `2h,12h,24h` | `1.37.0` |
| `CORTEX_COMPACTOR_INTERVAL` | Cortex compactor run interval configuration (`compactor.compaction_interval`). | false | string | `30m` | `1.37.0` |
| `CORTEX_COMPACTOR_DELETION_DELAY` | Cortex compactor block deletion delay configuration (`compactor.deletion_delay`). | false | string | `1h` | `1.37.0` |
| `CORTEX_COMPACTOR_RETENTION_PERIOD` | Cortex compactor block retention configuration (`limits_config.compactor_blocks_retention_period`). | false | string | `45d` | `1.46.0` |
| `CORTEX_CLEANER_INTERVAL` | Configure how often cleanup task is running (only use when no external object storage is configured). | false | string | `6h` | `1.37.0` |
| `CORTEX_CLEANER_RETENTION` | Configure the retention duration in the "file system bucket" (`/var/conduktor/monitoring/data/cortex/tsdb-backend/`). | false | string | `30d` | `1.37.0` |
| **Override configuration** | | | | | |
| `PROMETHEUS_OVERRIDE_CONFIG_FILE` | Location to yaml config file to **patch** Prometheus configuration with. | false | string | `/opt/override-configs/prometheus.yaml` | `1.19.0` |
| `CORTEX_OVERRIDE_CONFIG_FILE` | Location to yaml config file to **patch** Cortex configuration with. | false | string | `/opt/override-configs/cortex.yaml` | `1.19.0` |
| `ALERTMANAGER_OVERRIDE_CONFIG_FILE` | Location to yaml config file to **patch** Cortex alert manager configuration with. | false | string | `/opt/override-configs/alertmanager.yaml` | `1.37.0` |
| `PROMETHEUS_EXTRA_RUN_ARGS` | Extra run arguments for Prometheus. | false | string | `""` | `1.37.0` |
| `CORTEX_EXTRA_RUN_ARGS` | Extra run arguments for Cortex. | false | string | `""` | `1.37.0` |
| `CORTEX_ALERT_EXTRA_RUN_ARGS` | Extra run arguments for Cortex alert manager. | false | string | `""` | `1.37.0` |
## Sample configuration
In a Docker Compose it may look like the following:
```yaml theme={null}
version: '3.8'
services:
conduktor-console:
image: conduktor/conduktor-console
ports:
- "8080:8080"
environment:
CDK_MONITORING_CORTEX-URL: http://conduktor-monitoring:9009/
CDK_MONITORING_ALERT-MANAGER-URL: http://conduktor-monitoring:9009/
CDK_MONITORING_CALLBACK-URL: http://conduktor-console:8080/monitoring/api/
CDK_MONITORING_NOTIFICATIONS-CALLBACK-URL: http://localhost:8080
conduktor-monitoring:
image: conduktor/conduktor-console-cortex
ports:
- "9009:9009" # cortex and alertmanager api
- "9090:9090" # prometheus api
environment:
CDK_CONSOLE-URL: "http://conduktor-console:8080"
```
Since Console 1.46.0, the alert manager API is served on port `9009` alongside the Cortex API. Before Console 1.46.0, it was served on port `9010`, and `CDK_MONITORING_ALERT-MANAGER-URL` had to use that port. See the [Console 1.46.0 breaking change](/guide/release-notes#cortex-alert-manager-port-v1-46-0) for the upgrade steps.
## Configure basic auth for metric scraping
Since Console v1.42.0, you can configure basic auth credentials to protect access to Console metric endpoints.
When configured, the Cortex component has to provide these credentials when scraping metrics from the `/api/monitoring/metrics` and `/monitoring/metrics/` endpoints.
Basic auth is optional in Console v1.42.0 and no configuration changes are required for existing deployments.
### Enable basic auth on Console
Add these environment variables to Console:
```yaml theme={null}
environment:
CDK_MONITORING_BASICAUTH_EMAIL: "metrics@example.com"
CDK_MONITORING_BASICAUTH_PASSWORD: "Hello123!"
```
The password has to contain at least 8 characters, including one uppercase letter, one lowercase letter, one number and one special symbol.
### Configure Cortex to authenticate
When basic auth is enabled in Console, configure Cortex to authenticate when scraping metrics by adding the same credentials:
```yaml theme={null}
environment:
CDK_SCRAPER_BASICAUTH_EMAIL: "metrics@example.com"
CDK_SCRAPER_BASICAUTH_PASSWORD: "Hello123!"
```
```yaml theme={null}
services:
conduktor-console:
image: conduktor/conduktor-console
environment:
CDK_MONITORING_BASICAUTH_EMAIL: "metrics@example.com"
CDK_MONITORING_BASICAUTH_PASSWORD: "Hello123!"
# Other configuration...
conduktor-monitoring:
image: conduktor/conduktor-console-cortex
environment:
CDK_CONSOLE-URL: "http://conduktor-console:8080"
CDK_SCRAPER_BASICAUTH_EMAIL: "metrics@example.com"
CDK_SCRAPER_BASICAUTH_PASSWORD: "Hello123!"
```
Add the environment variables to your Helm chart values:
```yaml theme={null}
platformCortex:
env:
CDK_SCRAPER_BASICAUTH_EMAIL: "metrics@example.com"
CDK_SCRAPER_BASICAUTH_PASSWORD: "Hello123!"
```
For sensitive values, use a secret instead:
```yaml theme={null}
platformCortex:
envFrom:
- secretRef:
name: cortex-auth-credentials
```
## Metrics retention
### Local storage retention
Without external storage metrics are stored in `/var/conduktor/monitoring/data/cortex/tsdb-backend/`.
**Default retention lifecycle:**
* Prometheus creates blocks every 5-30 minutes (`PROMETHEUS_BLOCK_DURATION: 5m,30m`)
* Cortex keeps blocks for 3 hours (`CORTEX_RETENTION_PERIOD: 3h`)
* Cortex compacts blocks every 30 minutes (`CORTEX_COMPACTOR_INTERVAL: 30m`)
* Compacted blocks are deleted 1 hour after compaction (`CORTEX_COMPACTOR_DELETION_DELAY: 1h`)
* Cleanup runs every 6 hours (`CORTEX_CLEANER_INTERVAL: 6h`)
* Final retention: 30 days (`CORTEX_CLEANER_RETENTION: 30d`)
All metrics are lost when the container restarts or is recreated. Mount a persistent volume to `/var/conduktor/monitoring` or use external storage.
### External storage retention
With external storage (S3, GCS, Azure, PVC) configured, metrics persist across container restarts.
`CORTEX_RETENTION_PERIOD` (default: 3 hours) controls how long the ingester buffers data locally
before flushing it to external storage. `CORTEX_COMPACTOR_RETENTION_PERIOD` (default: 45 days)
controls how long the compactor retains blocks in external storage before deleting them.
The `CORTEX_CLEANER_*` variables are ignored when external storage is configured.
Use `CORTEX_COMPACTOR_RETENTION_PERIOD` to configure how long the compactor keeps
blocks in external storage, or configure your storage provider's lifecycle policy as a fallback.
**Example: Configure 7-day retention with S3:**
```yaml theme={null}
environment:
CDK_MONITORING_STORAGE_S3_BUCKET: my-metrics-bucket
CDK_MONITORING_STORAGE_S3_REGION: us-east-1
CORTEX_COMPACTOR_RETENTION_PERIOD: 7d
```
For production, external storage is required to avoid data loss.
[Configure external storage](/guide/conduktor-in-production/system-requirements#production-requirements)
using the storage environment variables in the table above.
Cortex maintains a cache in local storage for every block uploaded to external storage.
Without a retention period (`CORTEX_COMPACTOR_RETENTION_PERIOD`) set, blocks
accumulate indefinitely in both local and external storage. This is why it defaults to 45 days.
Therefore, do not set this to `0s` (disabled) or too high value (e.g. `365d`).
### Storage sizing
Storage requirements vary based on your cluster count, topic count, scrape interval and retention period.
**Example:** A deployment with 5 clusters and 500 topics using default settings (15s scrape interval) with 30-day retention might use approximately 85 GB.
Monitor actual usage to determine your requirements:
```bash theme={null}
# Local storage
docker exec conduktor-monitoring du -sh /var/conduktor/monitoring/data
# Kubernetes PVC
kubectl exec -it -- df -h /var/conduktor/monitoring
```
## Overriding configuration
### Overriding with YAML
#### Cortex
[Cortex configuration](https://cortexmetrics.io/docs/configuration/configuration-file/) can be **patched** by mounting a YAML file to path `/opt/override-configs/cortex.yaml`. To set an alternative path, use the `CORTEX_OVERRIDE_CONFIG_FILE` environment variable.
Cortex alert manager configuration can also be **patched** by mounting a YAML file to path `/opt/override-configs/alertmanager.yaml`. To set an alternative path, use the `ALERTMANAGER_OVERRIDE_CONFIG_FILE` environment variable.
For example, create a file `cortex.yaml` add in only your overrides:
```yaml theme={null}
limits:
ingestion_rate: 50000
max_series_per_metric: 100000
```
1. Mount to `/opt/override-configs/cortex.yaml`.
2. Spin up the container.
3. Exec into the container and confirm the contents, replace `2` with the number of lines of override you wish to see or remove grep to get the whole file:
`cat /var/conduktor/configs/monitoring-cortex.yaml | grep limits -A2`.
You should see a similar entry to the below in the opening logs:
```text theme={null}
INFO monitoring_entrypoint - Patch "/var/conduktor/configs/monitoring-cortex.yaml" configuration with "/opt/override-configs/cortex.yaml" fragment
```
#### Prometheus
Before Console v1.37.0, Prometheus configuration was overridden using the **replace** strategy. Since **v1.37.0** it's changed to the **patch** (YAML merging) strategy.
Prometheus configuration can be also **patched** by mounting a YAML file to path `/opt/override-configs/prometheus.yaml`. To set an alternative path, use the `PROMETHEUS_OVERRIDE_CONFIG_FILE` environment variable.
You should see a similar entry to the below in the opening logs:
```text theme={null}
INFO monitoring_entrypoint - Replace "/var/conduktor/configs/monitoring-prometheus.yaml" configuration with "/opt/override-configs/prometheus.yaml" content
```
### Overriding with ConfigMap
If you're deploying Cortex using our [Helm charts](https://github.com/conduktor/conduktor-public-charts/blob/main/charts/console/README.md#platform-cortex-parameters) you may expand the input with a custom ConfigMap for patching configuration such as retention time within Cortex.
You can also replace the Prometheus config as described below.
```yaml theme={null}
apiVersion: v1
kind: ConfigMap
metadata:
name: conduktor-console-cortex-config
labels:
app.kubernetes.io/name: console
app.kubernetes.io/instance: conduktor
app.kubernetes.io/component: conduktor-platform-cortex
data:
cortex.yaml: |
blocks_storage:
tsdb:
retention_period: 24h
prometheus.yaml: |
global:
scrape_interval: 15s
evaluation_interval: 15s
```
On chart `values.yaml` :
```yaml theme={null}
platformCortex:
extraVolumes:
- name: cortex-config-override
configMap:
name: conduktor-console-cortex-config
extraVolumeMounts:
- name: cortex-config-override
mountPath: /opt/override-configs/cortex.yaml
subPath: cortex.yaml
- name: cortex-config-override
mountPath: /opt/override-configs/prometheus.yaml
subPath: prometheus.yaml
```
## Monitoring endpoints
Monitoring is designed to be interacted with through the API endpoints by Console (not users). Therefore, **ingress isn't available externally** and you shouldn't set it up because an authentication mechanism doesn't exist.
### Cortex endpoint
`/ready` on port `9009`
If Cortex is running, returns the *200 status* with `ready`.
```shell title="cURL example" theme={null}
curl -s "http://localhost:9009/ready"
```
### Alertmanager endpoint
Since Console 1.46.0, the alert manager API is served on port `9009` alongside the Cortex API, so the Cortex readiness endpoint above also covers it. Before Console 1.46.0, the alert manager served `/ready` on its own port `9010`. See the [Console 1.46.0 breaking change](/guide/release-notes#cortex-alert-manager-port-v1-46-0).
### Prometheus endpoint
`/-/healthy` on port `9090`
If Prometheus is running, returns the *200 status* with `Prometheus Server is Healthy.`.
```shell title="cURL example" theme={null}
curl -s "http://localhost:9090/-/healthy"
```
## Troubleshoot
### No metrics in the monitoring page
Go to `http://localhost:9090/targets` to see Prometheus scraping target status.
If it fails, check that you can query metrics endpoint from `conduktor-console-cortex` container.
You might also have to configure `CDK_SCRAPER_SKIPSSLCHECK` or `CDK_SCRAPER_CAFILE` if `conduktor-console` is configured with TLS termination.
### Custom CA certificates for S3-like storage
The `CDK_SCRAPER_CAFILE` environment variable only configures the CA certificate for the Scraper component (which scrapes metrics from Console). It does **not** configure the CA certificate for S3-like storage backends.
If you use an S3-compatible service (like MinIO, Ceph or other S3-like services) with custom certificates, you have two options:
#### Option 1: Disable SSL verification
To skip TLS hostname verification with S3-like services, [override the Cortex configuration with YAML](#overriding-with-yaml) by creating a YAML file with these contents:
```yaml theme={null}
blocks_storage:
s3:
http:
insecure_skip_verify: true
ruler_storage:
s3:
http:
insecure_skip_verify: true
```
Mount this file to `/opt/override-configs/cortex.yaml` (or use the `CORTEX_OVERRIDE_CONFIG_FILE` environment variable to specify an alternative path).
Disabling SSL verification removes certificate validation and is **not recommended for production** environments. Use this only for testing or non-production deployments.
#### Option 2: Contact Conduktor support
For production environments requiring custom CA certificates with S3-compatible storage, [contact Conduktor support](https://www.conduktor.io/contact/demo/?utm_source=docs\&utm_medium=webpage) for guidance on configuring certificate trust for Cortex components.
## Related resources
* [Conduktor monitoring](/guide/monitor-brokers-apps)
* [Gateway and Console healthcheck monitoring](/guide/conduktor-in-production/monitor)
* [Set up alerts](/guide/monitor-brokers-apps/alerts)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# External monitoring setup
Source: https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-external-monitoring
Connect Conduktor Console to an existing Cortex, Mimir, or Prometheus stack. Available since Console v1.38 as alternative to deploying the Cortex.
Console v1.38 introduced support for external monitoring solutions including **Cortex**, **Mimir**, and **Prometheus**.
When using external monitoring solutions, Console connects to your existing monitoring infrastructure instead of deploying the `console-cortex` component.
[Check out monitoring-related resource reference](/guide/conduktor-in-production/deploy-artifacts/deploy-console/environment-variables#configure-console-properties).
## Configure
### Metrics collection
Your external monitoring solution has to be configured to scrape Console metrics from the `/api/monitoring/metrics` endpoint.
Example Prometheus scrape configuration:
```yaml theme={null}
scrape_configs:
- job_name: 'conduktor-console'
scheme: http
metrics_path: '/api/monitoring/metrics'
static_configs:
- targets: ['conduktor-console:8080']
```
### Basic auth for metrics scraping
Since Console v1.42.0, you can configure basic auth credentials to protect access to Console metrics. When configured, your monitoring backend must provide these credentials in order to scrape metrics from the `/api/monitoring/metrics` and `/monitoring/metrics/` endpoints.
Basic auth is optional in Console v1.42.0 and no configuration changes are required for existing deployments.
Configure basic auth on Console:
```yaml theme={null}
environment:
CDK_MONITORING_BASICAUTH_EMAIL: "metrics@example.com"
CDK_MONITORING_BASICAUTH_PASSWORD: "Hello123!"
```
The password must contain at least 8 characters, including one uppercase letter, one lowercase letter, one number and one special symbol.
Then configure your Prometheus scrape config to include these credentials:
```yaml theme={null}
scrape_configs:
- job_name: 'conduktor-console'
scheme: http
metrics_path: '/api/monitoring/metrics'
static_configs:
- targets: ['conduktor-console:8080']
basic_auth:
username: 'metrics@example.com'
password: 'Hello123!'
```
### Multi-tenancy support
If your Cortex or Mimir instance has multi-tenancy enabled, you can configure the tenant ID using:
```yaml theme={null}
environment:
CDK_MONITORING_TENANT-ID: "your-tenant-id"
```
This sets the `X-Scope-OrgId` header for authentication.
### Alerting system
If you configure a Cortex or Mimir integration, you can also set up Alertmanager for Console alerts. Alternatively, Console will use its own built-in alerting system.
Because Prometheus doesn't support configuration of alert rules via API, Console will always use its own built-in alerting system.
When configuring Alertmanager, Console will *override* the Alertmanager configuration. If you share the same Alertmanager instance with other applications, you have to ensure that multi-tenancy is enabled and that Console has a dedicated tenant.
## Configuration examples
### Cortex
```yaml theme={null}
services:
conduktor-console:
image: conduktor/conduktor-console
ports:
- "8080:8080"
environment:
# Connect to external Cortex instance
CDK_MONITORING_BACKEND: "cortex"
CDK_MONITORING_CORTEX-URL: https://cortex.your-domain.com:9009/
CDK_MONITORING_TENANT-ID: "conduktor" # Optional: Set tenant ID for multi-tenancy
CDK_MONITORING_ALERT-MANAGER-URL: https://alertmanager.your-domain.com:9093/
CDK_MONITORING_CALLBACK-URL: http://conduktor-console:8080/monitoring/api/
CDK_MONITORING_NOTIFICATIONS-CALLBACK-URL: http://conduktor-console:8080
# Other Console configuration...
```
### Mimir
```yaml theme={null}
services:
conduktor-console:
image: conduktor/conduktor-console
ports:
- "8080:8080"
environment:
# Connect to external Mimir instance
CDK_MONITORING_BACKEND: "mimir"
CDK_MONITORING_MIMIR-URL: https://mimir.your-domain.com:9009/
CDK_MONITORING_TENANT-ID: "conduktor" # Optional: Set tenant ID for multi-tenancy
CDK_MONITORING_ALERT-MANAGER-URL: https://alertmanager.your-domain.com:9093/
CDK_MONITORING_CALLBACK-URL: http://conduktor-console:8080/monitoring/api/
CDK_MONITORING_NOTIFICATIONS-CALLBACK-URL: http://conduktor-console:8080
# Other Console configuration...
```
### Prometheus
```yaml theme={null}
services:
conduktor-console:
image: conduktor/conduktor-console
ports:
- "8080:8080"
environment:
# Connect to external Prometheus instance
CDK_MONITORING_BACKEND: "prometheus"
CDK_MONITORING_PROMETHEUS-URL: https://prometheus.your-domain.com:9090/
CDK_MONITORING_NOTIFICATIONS-CALLBACK-URL: http://conduktor-console:8080
# Other Console configuration...
```
## Troubleshooting
### Monitoring debug page
Console provides a dedicated troubleshooting page for administrators to diagnose monitoring connectivity issues. Access this page at: `/settings/debug/monitoring`
This debug page shows:
* Connectivity status to your monitoring solution
* Current configuration settings
* Metrics collection statistics
# Deployment considerations
Source: https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-gateway
Deploy Conduktor Gateway: choose network routing, load balancing, Kafka connection, and client security before deploying to Docker or Kubernetes.
Deploying Conduktor Gateway involves making several decisions, based on your requirements and goals.
Gateway exposes Kafka clients through one or more **listeners**. Each listener has its own routing, network bindings and security protocol. Most deployments need a single listener.
Follow these steps to configure your Gateway deployment:
1. [Configure listeners](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/listeners) - set up the network endpoints clients use to reach Gateway
2. [Define load balancing](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/load-balancing) - select internal or external load balancing
3. [Connect Gateway to Kafka](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/connect-to-kafka) - configure the upstream Kafka connection
4. [Configure listener security](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/listener-security) - pick the security protocol and authentication mechanism for each listener
5. [Decide on Virtual Clusters](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/virtual-clusters) - enable logical cluster isolation (optional)
We also recommend that you [configure Gateway for failover](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/failover).
If applicable, [set up Gateway for multi-clusters](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/multi-clusters).
Gateway requires a license key to start which has to be [provided as an environment variable](/guide/conduktor-in-production/manage-licenses/gateway).
## Secure Gateway before going live
Before deploying Gateway to production:
Set up authentication mechanisms (SASL, mTLS, or OAuth) on each listener. [Configure listener security](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/listener-security).
Set `GATEWAY_USER_POOL_SECRET_KEY` to a random base64 encoded value of 256 bits when using local service accounts. Generate the key using `openssl rand -base64 32` to ensure tokens aren't forged.
Configure TLS certificates for encrypted communication between clients and Gateway. Set up keystores and truststores according to your security protocol.
Configure credentials in `GATEWAY_ADMIN_API_USERS` and set `GATEWAY_SECURED_METRICS` to require authentication for the HTTP management API. [See HTTP API configuration](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables#http-api).
## Troubleshoot
### Custom CA certificates for KMS
If you use a Key Management Service (KMS) with custom SSL/TLS certificates (for example, Vault KMS with self-signed certificates or internal certificate authorities), Gateway needs to trust the certificate authority.
Gateway is a Java application, so you need to configure a custom Java truststore that includes your CA certificate.
#### Steps to configure custom CA for KMS
1. **Create a truststore with your CA certificate**
Use the Java `keytool` command to create a truststore and import your CA certificate:
```bash theme={null}
keytool -import \
-trustcacerts \
-alias my-kms-ca \
-file /path/to/your/ca-certificate.crt \
-keystore /path/to/truststore.jks \
-storepass yourpassword \
-storetype JKS \
-noprompt
```
2. **Mount the truststore into the Gateway container**
Mount your truststore file to a location in the Gateway container (for example, `/opt/conduktor/truststore.jks`).
```yaml theme={null}
services:
conduktor-gateway:
image: conduktor/conduktor-gateway:latest
ports:
- 6969:6969
volumes:
- /path/on/host/truststore.jks:/opt/conduktor/truststore.jks:ro
environment:
KAFKA_BOOTSTRAP_SERVERS: kafka1:9092,kafka2:9092
JAVA_TOOL_OPTIONS: >-
-Djavax.net.ssl.trustStore=/opt/conduktor/truststore.jks
-Djavax.net.ssl.trustStorePassword=yourpassword
-Djavax.net.ssl.trustStoreType=JKS
```
```yaml theme={null}
apiVersion: v1
kind: ConfigMap
metadata:
name: gateway-truststore
binaryData:
truststore.jks:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: conduktor-gateway
spec:
template:
spec:
containers:
- name: gateway
image: conduktor/conduktor-gateway:latest
env:
- name: JAVA_TOOL_OPTIONS
value: >-
-Djavax.net.ssl.trustStore=/opt/conduktor/truststore.jks
-Djavax.net.ssl.trustStorePassword=yourpassword
-Djavax.net.ssl.trustStoreType=JKS
volumeMounts:
- name: truststore
mountPath: /opt/conduktor/truststore.jks
subPath: truststore.jks
readOnly: true
volumes:
- name: truststore
configMap:
name: gateway-truststore
```
3. **Set the `JAVA_TOOL_OPTIONS` environment variable**
Configure Gateway to use the custom truststore by setting:
```bash theme={null}
JAVA_TOOL_OPTIONS="-Djavax.net.ssl.trustStore=/opt/conduktor/truststore.jks -Djavax.net.ssl.trustStorePassword=yourpassword -Djavax.net.ssl.trustStoreType=JKS"
```
The `JAVA_TOOL_OPTIONS` environment variable is a standard Java mechanism that applies the truststore configuration to all Java SSL/TLS connections made by Gateway, including connections to KMS services.
#### Verify the configuration
After restarting Gateway, check the startup logs to confirm the truststore is being used:
```
Picked up JAVA_TOOL_OPTIONS: -Djavax.net.ssl.trustStore=/opt/conduktor/truststore.jks -Djavax.net.ssl.trustStorePassword=*** -Djavax.net.ssl.trustStoreType=JKS
```
If Gateway successfully connects to your KMS service, the configuration is correct.
## Related resources
* [Manage service accounts](/guide/tutorials/manage-gateway-service-accounts)
* [Resource reference](/guide/reference/gateway-reference)
* [Configure encryption and decryption](/guide/tutorials/configure-encryption)
* [Configure external connections](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/external-connections)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Connect Gateway to Kafka
Source: https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/connect-to-kafka
Configure the connection between Conduktor Gateway and your Kafka cluster. Set security protocols, authentication credentials.
Gateway depends on a 'backing' Kafka cluster for its operation.
Configuring the Gateway connection to the backing Kafka cluster closely resembles configuring a standard Kafka client's connection to a cluster.
Set the client to Gateway configuration variables first. Gateway has to know which [security mode](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/listener-security) you run, because that determines whether it reuses each client's credentials to reach Kafka or connects with its own service account.
The configuration is done via environment variables, as it is for the other aspects of a Gateway configuration.
All environment variables that start with `KAFKA_` are mapped to configuration properties for connecting Gateway to the Kafka cluster.
As Gateway is based on the Java-based Kafka-clients, it supports all configuration properties that Java-clients do.
Kafka configuration properties are mapped to Gateway environment variables as follows:
* Add a `KAFKA_` prefix
* Replace each dot, `.` , with an underscore, `_`
* Convert to uppercase
For example, `bootstrap.servers` is set by the `KAFKA_BOOTSTRAP_SERVERS` environment variable.
## Supported protocols
You can use all the Kafka security protocols to authenticate Gateway to the Kafka cluster; `PLAINTEXT`, `SASL_PLAINTEXT`, `SASL_SSL` and `SSL`.
These can be used with all SASL mechanisms supported by Apache Kafka: `PLAIN`, `SCRAM-SHA`, `OAuthBearer`, `Kerberos` etc. In addition, we support IAM authentication for AWS MSK clusters.
In the following examples, we provide blocks of environment variables which can be provided to Gateway, e.g. in a docker-compose file, or a `helm` deployment.
Information which should be customized is enclosed by `<` and `>`.
### PLAINTEXT
Kafka cluster without authentication or encryption in transit, `PLAINTEXT`.
In this case you just need the bootstrap servers:
```yaml theme={null}
KAFKA_BOOTSTRAP_SERVERS: ,
```
### SSL
```yaml theme={null}
KAFKA_BOOTSTRAP_SERVERS: ,
KAFKA_SECURITY_PROTOCOL: SSL
KAFKA_SSL_TRUSTSTORE_TYPE:
KAFKA_SSL_TRUSTSTORE_LOCATION:
KAFKA_SSL_TRUSTSTORE_PASSWORD:
KAFKA_SSL_KEYSTORE_TYPE:
KAFKA_SSL_KEYSTORE_LOCATION:
KAFKA_SSL_KEYSTORE_PASSWORD:
KAFKA_SSL_KEY_PASSWORD:
```
#### mTLS
Kafka cluster with mTLS client authentication.
```yaml theme={null}
KAFKA_BOOTSTRAP_SERVERS: ,,
KAFKA_SECURITY_PROTOCOL: SSL
KAFKA_SSL_TRUSTSTORE_LOCATION: /security/truststore.jks
KAFKA_SSL_TRUSTSTORE_PASSWORD: conduktor
KAFKA_SSL_KEYSTORE_LOCATION: /security/kafka.gw.keystore.jks
KAFKA_SSL_KEYSTORE_PASSWORD: conduktor
KAFKA_SSL_KEY_PASSWORD: conduktor
```
### SASL\_PLAINTEXT
Kafka cluster with SASL\_PLAINTEXT security protocol but no encryption in transit, supporting the following SASL\_MECHANISMs.
### SASL PLAIN
```yaml theme={null}
KAFKA_BOOTSTRAP_SERVERS: ,
KAFKA_SECURITY_PROTOCOL: SASL_PLAINTEXT
KAFKA_SASL_MECHANISM: PLAIN
KAFKA_SASL_JAAS_CONFIG: org.apache.kafka.common.security.plain.PlainLoginModule required username="" password="";
```
### SASL SCRAM
```yaml theme={null}
KAFKA_BOOTSTRAP_SERVERS: ,
KAFKA_SECURITY_PROTOCOL: SASL_PLAINTEXT
KAFKA_SASL_MECHANISM: SCRAM-SHA-256 # or SCRAM-SHA-512
KAFKA_SASL_JAAS_CONFIG: org.apache.kafka.common.security.scram.ScramLoginModule required username="" password="";
```
### OAuthBearer (client secret)
```yaml theme={null}
KAFKA_BOOTSTRAP_SERVERS: ,
KAFKA_SECURITY_PROTOCOL: SASL_PLAINTEXT
KAFKA_SASL_MECHANISM: OAUTHBEARER
KAFKA_SASL_OAUTHBEARER_TOKEN_ENDPOINT_URL: ""
KAFKA_SASL_LOGIN_CALLBACK_HANDLER_CLASS: org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginCallbackHandler
KAFKA_SASL_JAAS_CONFIG: org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required clientId="<>" clientSecret="<>" scope=".default";
```
### SASL\_SSL
Kafka cluster that uses SASL for authentication and TLS (formerly SSL) for encryption in transit.
### PLAIN
Kafka cluster with SASL\_SSL and PLAIN SASL mechanism.
### Confluent Cloud with API key/secret
This example can be seen as a special case of the above.
```yaml theme={null}
KAFKA_BOOTSTRAP_SERVERS:
KAFKA_SECURITY_PROTOCOL: SASL_SSL
KAFKA_SASL_MECHANISM: PLAIN
KAFKA_SASL_JAAS_CONFIG: org.apache.kafka.common.security.plain.PlainLoginModule required username="" password="";
```
As Confluent Cloud uses certificates signed by a well-known CA, you normally do not need to specify a trust-store.
### SASL SCRAM
```yaml theme={null}
KAFKA_BOOTSTRAP_SERVERS: ,
KAFKA_SECURITY_PROTOCOL: SASL_SSL
KAFKA_SASL_MECHANISM: SCRAM-SHA-256 # or SCRAM-SHA-512
KAFKA_SASL_JAAS_CONFIG: org.apache.kafka.common.security.scram.ScramLoginModule required username="" password="";
KAFKA_SSL_TRUSTSTORE_TYPE:
KAFKA_SSL_TRUSTSTORE_LOCATION:
KAFKA_SSL_TRUSTSTORE_PASSWORD:
```
### SASL GSSAPI (Kerberos)
```yaml theme={null}
KAFKA_BOOTSTRAP_SERVERS: ,
KAFKA_SECURITY_PROTOCOL: SASL_SSL
KAFKA_SASL_MECHANISM: GSSAPI
KAFKA_SASL_JAAS_CONFIG: com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true keyTab="<>>" principal="<>";
KAFKA_SASL_KERBEROS_SERVICE_NAME:
KAFKA_SSL_TRUSTSTORE_TYPE:
KAFKA_SSL_TRUSTSTORE_LOCATION:
KAFKA_SSL_TRUSTSTORE_PASSWORD:
```
### OAuthBearer (client secret)
```yaml theme={null}
KAFKA_BOOTSTRAP_SERVERS: ,
KAFKA_SECURITY_PROTOCOL: SASL_SSL
KAFKA_SASL_MECHANISM: OAUTHBEARER
KAFKA_SASL_OAUTHBEARER_TOKEN_ENDPOINT_URL: ""
KAFKA_SASL_LOGIN_CALLBACK_HANDLER_CLASS: org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginCallbackHandler
KAFKA_SASL_JAAS_CONFIG: org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required clientId="<>" clientSecret="<>" scope=".default";
```
### OAuthBearer (client assertion)
Rather than a shared client secret, the Gateway can authenticate to the Kafka cluster with a signed JWT assertion ([KIP-1258](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1258%3A+Add+Support+for+OAuth+Client+Assertion+to+client_credentials+Grant+Type)).
```yaml theme={null}
KAFKA_BOOTSTRAP_SERVERS: ,
KAFKA_SECURITY_PROTOCOL: SASL_SSL
KAFKA_SASL_MECHANISM: OAUTHBEARER
KAFKA_SASL_OAUTHBEARER_TOKEN_ENDPOINT_URL: ""
KAFKA_SASL_LOGIN_CALLBACK_HANDLER_CLASS: org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginCallbackHandler
KAFKA_SASL_JAAS_CONFIG: org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required clientId="";
KAFKA_SASL_OAUTHBEARER_ASSERTION_PRIVATE_KEY_FILE:
KAFKA_SASL_OAUTHBEARER_ASSERTION_CLAIM_ISS:
KAFKA_SASL_OAUTHBEARER_ASSERTION_CLAIM_SUB:
```
The private key file has to be added to the `allowed.files` list via `JAVA_OPTS`:
```yaml theme={null}
JAVA_OPTS: "-Dorg.apache.kafka.sasl.oauthbearer.allowed.files="
```
## AWS MSK cluster with IAM
```yaml theme={null}
KAFKA_BOOTSTRAP_SERVERS:
KAFKA_SECURITY_PROTOCOL: SASL_SSL
KAFKA_SASL_MECHANISM: AWS_MSK_IAM
KAFKA_SASL_JAAS_CONFIG: software.amazon.msk.auth.iam.IAMLoginModule required;
KAFKA_SASL_CLIENT_CALLBACK_HANDLER_CLASS: io.conduktor.aws.IAMClientCallbackHandler
KAFKA_AWS_ACCESS_KEY_ID:
KAFKA_AWS_SECRET_ACCESS_KEY:
```
## Service account and ACL requirements
Gateway connects to the backing Kafka with its own service account. What else that account needs depends on the [security mode](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/listener-security) you run:
* In [Kafka-managed](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/listener-security#kafka_managed) mode, clients present their own credentials and Gateway reuses them to reach Kafka, so each client inherits the ACLs of its own service account on the backing cluster
* In [Gateway-managed](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/listener-security#gateway_managed) mode, Gateway reaches Kafka with its own service account for every client, so that account also needs every permission your Gateway users rely on
In both modes, Gateway's own service account needs these ACLs to operate correctly:
* `Read`, `Write` and `DescribeConfigs` on Gateway's internal topics, which default to the `_conduktor_${GATEWAY_CLUSTER_ID}_` prefix
* `Create` on those topics, or on the cluster, unless you pre-create them all with `cleanup.policy=compact` on the state topics
* `Read` on the `conduktor_${GATEWAY_CLUSTER_ID}` consumer group, or on the group set in `GATEWAY_GROUP_ID`
* `Describe` on the physical topics behind alias topics and topic views
* `Write` and `Create` on the topics Gateway writes its own records to: the [audit log](/guide/conduktor-in-production/admin/audit-logs),
the [observability Interceptor](/guide/tutorials/configure-chargeback#collect-gateway-throughput) (chargeback) topic,
and the violation and dead letter topics of your [data validation policies](/guide/conduktor-in-production/admin/gateway-policies)
With topic concentration, it also needs:
* `Describe` and `DescribeConfigs` on the physical topics, plus `Create` and `AlterConfigs` for auto-managed rules
* `DescribeConfigs` on the cluster, if clients create concentrated topics without a partition count
* `Read` on the physical topics behind rules that still set the deprecated [`offsetCorrectness`](/guide/conduktor-concepts/logical-topics#message-count-lag-and-offset-incorrectness),
and a prefixed `Read` ACL on consumer group `conduktor_${GATEWAY_CLUSTER_ID}-logical-offsets-`,
or `${GATEWAY_GROUP_ID}-logical-offsets-` when you set that variable
## Related resources
* [Define Gateway environment variables](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables)
* [Configure listener security](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/listener-security)
* [Set up Gateway for multi-clusters](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/multi-clusters)
# Conduktor Gateway environment variables reference
Source: https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables
Complete reference for Conduktor Gateway environment variables: network routing, Kafka connection, security, Virtual Clusters.
To configure Conduktor Gateway, we recommend setting up environment variables. They can be **set in the Gateway container** or **taken from a file**. To make sure the values were set correctly, check the startup logs.
## Use the Gateway container
You can set the environment variables during the docker-run command with `-e` or `--env`:
```shell theme={null}
docker run -d \
-e KAFKA_BOOTSTRAP_SERVERS=kafka1:9092,kafka2:9092 \
-e KAFKA_SECURITY_PROTOCOL=SASL_PLAINTEXT \
-e KAFKA_SASL_MECHANISM=PLAIN \
-e KAFKA_SASL_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule required username='usr' password='pwd';" \
-p 6969:6969 \
conduktor/conduktor-gateway:latest
```
Or in a `docker-compose.yaml`:
```yaml theme={null}
services:
conduktor-gateway:
image: conduktor/conduktor-gateway:latest
ports:
- 6969:6969
environment:
KAFKA_BOOTSTRAP_SERVERS: kafka1:9092,kafka2:9092
KAFKA_SECURITY_PROTOCOL: SASL_PLAINTEXT
KAFKA_SASL_MECHANISM: PLAIN
KAFKA_SASL_JAAS_CONFIG: org.apache.kafka.common.security.plain.PlainLoginModule required username='usr' password='pwd';
```
You can set the environment variables in the `values.yaml` of our [Helm chart](https://github.com/conduktor/conduktor-public-charts/blob/main/charts/gateway/README.md) :
```yaml theme={null}
gateway:
env:
KAFKA_BOOTSTRAP_SERVERS: kafka1:9092,kafka2:9092
KAFKA_SECURITY_PROTOCOL: SASL_PLAINTEXT
KAFKA_SASL_MECHANISM: PLAIN
KAFKA_SASL_JAAS_CONFIG: org.apache.kafka.common.security.plain.PlainLoginModule required username='usr' password='pwd';
```
## Use a file
You can mount a file with the key-value pairs into the container and provide its path by setting the environment variable `GATEWAY_ENV_FILE`. Note these variables should be exported by the file as they are injected in the wrapper that starts the Gateway process.
```env title="Example" theme={null}
export KAFKA_BOOTSTRAP_SERVERS=kafka1:9092,kafka2:9092
export KAFKA_SECURITY_PROTOCOL=SASL_PLAINTEXT
export KAFKA_SASL_MECHANISM=PLAIN
export KAFKA_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username='usr' password='pwd';
```
You'll get a confirmation in the logs: `Sourcing environment variables from $GATEWAY_ENV_FILE` (or a warning if the file isn't found: `Warning: GATEWAY_ENV_FILE is set but the file does not exist or is not readable.`).
## Listeners
Per-listener configuration with `GATEWAY_LISTENER__*` variables is available from **Gateway 3.20.0**. On earlier versions, networking is configured with the global variables described in [Legacy network configuration](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/legacy-network-configuration). To move an existing deployment over, see [Migrate to listener configuration](/guide/tutorials/migrate-to-listener-configuration).
Configure listeners using the pattern `GATEWAY_LISTENER__`, where `` is a label you choose. See [Configure Gateway listeners](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/listeners) for details.
Listener names have to start with a letter and contain only alphanumeric characters. Underscores are reserved as delimiters in `GATEWAY_LISTENER__`, so a name like `MY_LISTENER` would be ambiguous. Gateway only recognizes variables whose `` matches this pattern — variables using any other name are ignored, and the listener is silently not created.
| Variable | Description | Default |
| ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| `GATEWAY_LISTENER__ROUTING` | Routing mechanism: `port` or `sni` | Required |
| `GATEWAY_LISTENER__PORTS` | Port specification. [See the format](#listeners-port-specification-format) | Required |
| `GATEWAY_LISTENER__SECURITY_PROTOCOL` | Security protocol: `PLAINTEXT`, `SSL`, `SASL_SSL`, or `SASL_PLAINTEXT` | Required |
| `GATEWAY_LISTENER__BIND_ADDRESS` | Network interface to bind to | `0.0.0.0` |
| `GATEWAY_LISTENER__ADVERTISED_HOST` | Hostname returned to clients | System hostname or `localhost` |
| `GATEWAY_LISTENER__ADVERTISED_HOST_PATTERN` | Template for per-broker hostnames. Only used when routing is set to `sni`. | `broker-{{physicalCluster}}-{{nodeId}}.{{advertisedHost}}` |
| `GATEWAY_LISTENER__BOOTSTRAP_HOST_PATTERN` | Template for bootstrap hostname. Only used when routing is set to `sni`. | Derived from `ADVERTISED_HOST_PATTERN`, or `bootstrap-{{physicalCluster}}.{{advertisedHost}}` if not set |
| `GATEWAY_LISTENER__SSL_CLIENT_AUTH` | [mTLS](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/listener-security#mtls) client authentication policy: `NONE`, `OPTIONAL`, or `REQUIRE`. [See the modes](#listeners-ssl-client-authentication-modes) | `NONE` |
The following global variable applies to all listeners that use port routing:
| Variable | Description | Default |
| ---------------------- | --------------------------------------------------------------------------------------------------------------------- | ------- |
| `GATEWAY_MIN_BROKERID` | The broker ID mapped to the first port. Set it to the lowest `broker.id` (or `node.id`) in the backing Kafka cluster. | `0` |
### Listeners: port specification format
`GATEWAY_LISTENER__PORTS` accepts ranges, individual values, or a combination of both, separated by commas:
```bash theme={null}
GATEWAY_LISTENER_EXTERNAL_PORTS=9092-9095
GATEWAY_LISTENER_EXTERNAL_PORTS=9092,9093,9094,9095
GATEWAY_LISTENER_EXTERNAL_PORTS=9092-9093,9100,9200
```
When the advertised ports differ from the local ports (for example, behind a NAT or in Docker), use the `advertised:local` format:
```bash theme={null}
GATEWAY_LISTENER_EXTERNAL_PORTS=29092-29095:9092-9095
```
You can mix ranges and individual ports with local mappings. For example, with `GATEWAY_MIN_BROKERID=0` and three brokers:
```bash theme={null}
# Broker 0 → advertised 29092, local 9092
# Broker 1 → advertised 29093, local 9093
# Broker 2 → advertised 29100, local 9100
GATEWAY_LISTENER_EXTERNAL_PORTS=29092-29093:9092-9093,29100:9100
```
For SNI routing, the listener uses a single port; the `advertised:local` format maps the advertised port (for example, a load balancer's `443`) to the local one:
```bash theme={null}
GATEWAY_LISTENER_EXTERNAL_PORTS=443:9192
```
### Listeners: SSL client authentication modes
The `GATEWAY_LISTENER__SSL_CLIENT_AUTH` setting controls whether the Gateway requests a client certificate during the TLS handshake for a specific listener:
| Mode | Behavior |
| ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `NONE` | No client certificate is requested. This is standard one-way TLS where only the server is authenticated. |
| `OPTIONAL` | The Gateway requests a client certificate but allows the connection if the client does not provide one. Useful when some clients support mTLS and others do not. |
| `REQUIRE` | The Gateway requires a valid client certificate. Connections without a certificate are rejected during the TLS handshake. |
When `SSL_CLIENT_AUTH` is set to `OPTIONAL` or `REQUIRE`, you must configure a truststore in `sslConfig.trustStore` containing the CA certificates used to verify client certificates.
### Listeners: zero-config mode
If you don't set any listener or networking environment variables, Gateway configures itself automatically. It creates a single listener named `DEFAULT` with port routing, assigning ports from `6969` (one per broker, plus two for headroom), and takes the listener's security protocol from the backing Kafka cluster. `GATEWAY_ACL_ENABLED` defaults to `false` in this mode.
When `GATEWAY_SECURITY_MODE` is not set, Gateway infers it from the backing cluster's security protocol:
| Backing Kafka security protocol | Inferred `GATEWAY_SECURITY_MODE` |
| ------------------------------- | -------------------------------- |
| `SASL_PLAINTEXT`, `SASL_SSL` | `KAFKA_MANAGED` |
| `PLAINTEXT`, `SSL` | `GATEWAY_MANAGED` |
Zero-config mode is for local development and testing only. Configure your listeners, security mode and ACLs explicitly for production deployments.
### Listeners: legacy configuration (deprecated)
These global variables are deprecated in favor of per-listener configuration. They remain supported for now. For how they behave, see [Legacy network configuration](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/legacy-network-configuration); to switch, see [Migrate to listener configuration](/guide/tutorials/migrate-to-listener-configuration).
| Environment variable | Description | Default |
| -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| **Common properties** | | |
| `GATEWAY_ADVERTISED_HOST` | The hostname returned in the Gateway's metadata for clients to connect to. | Your hostname |
| `GATEWAY_ROUTING_MECHANISM` | Defines the routing method: `port` for port routing, `host` for SNI routing. | `port` |
| `GATEWAY_PORT_START` | The first port the Gateway listens on. | `6969` |
| `GATEWAY_BIND_HOST` | The network interface the Gateway binds to. | `0.0.0.0` |
| **Port routing specific** | | |
| `GATEWAY_PORT_COUNT` | The total number of ports used by Gateway. | `(maxBrokerId - minBrokerId) + 3` |
| **SNI routing specific** | | |
| `GATEWAY_ADVERTISED_SNI_PORT` | The port returned in the Gateway's metadata when using SNI routing. | `GATEWAY_PORT_START` |
| `GATEWAY_ADVERTISED_HOST_PREFIX` | Configures the advertised broker names. | `broker` |
| `GATEWAY_SNI_HOST_SEPARATOR` | The separator used to construct returned metadata. | `-` |
| `GATEWAY_SECURITY_PROTOCOL` | The security protocol clients use to connect to Gateway. Has to be `SSL` or `SASL_SSL` when in `GATEWAY_MANAGED` security mode, or `SASL_SSL` when in `KAFKA_MANAGED` security mode, for SNI routing. | Inferred from `GATEWAY_SECURITY_MODE` and `KAFKA_SECURITY_PROTOCOL` |
| `GATEWAY_SSL_CLIENT_AUTH` | mTLS client authentication policy: `NONE`, `OPTIONAL` or `REQUIRE`. Replaced by the per-listener `GATEWAY_LISTENER__SSL_CLIENT_AUTH`. | `NONE` |
## External connections (preview)
This is a preview feature and is subject to change.
Declare connections to external systems using the pattern `GATEWAY_EXTERNAL_CONNECTION__CONFIG_`, where `` is a label you choose and is how a resource references the connection. See [Configure Gateway external connections](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/external-connections) for details.
Connection names have to start with a letter and can contain letters, numbers and underscores, so `DEFAULT_SR` is a valid name. A name can't contain `_CONFIG_`, because that's the delimiter between the name and the property. Gateway rejects such a variable at startup rather than guessing where the name ends. Names are case-insensitive and Gateway stores them in upper case, so `default_sr` and `DEFAULT_SR` are the same connection. We recommend you declare them in upper case to match the variable.
Every connection needs a `TYPE`:
| Variable | Description | Default |
| ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
| `GATEWAY_EXTERNAL_CONNECTION__CONFIG_TYPE` | System to connect to: [`CONFLUENT_SCHEMA_REGISTRY`](#external-connections-confluent-like-schema-registry), [`AWS_SCHEMA_REGISTRY`](#external-connections-aws-glue-schema-registry), [`HASHICORP_VAULT`](#external-connections-hashicorp-vault-kms) or [`AWS_KMS`](#external-connections-aws-kms). Accepts any case | Required |
### External connections: Confluent-like schema registry
Set `TYPE` to `CONFLUENT_SCHEMA_REGISTRY`.
| Variable suffix | Description | Default |
| --------------------- | ------------------------------------------------------------------------------------------------------- | -------- |
| `SCHEMA_REGISTRY_URL` | URL of the schema registry | Required |
| `CACHE_SIZE` | Number of schemas Gateway caches for this connection, so it doesn't query the registry for every record | `50` |
### External connections: AWS Glue schema registry
Set `TYPE` to `AWS_SCHEMA_REGISTRY`.
| Variable suffix | Description | Default |
| ---------------------------------------- | -------------------------------------------------------------------------------------------------- | -------------------------- |
| `REGION` | AWS region of the registry, for example `us-east-1` | Required |
| `ENDPOINT` | Custom endpoint for the Glue API | AWS default for the region |
| `REGISTRY_NAME` | Name of the registry in AWS | `default-registry` |
| `SCHEMA_NAME` | Name of the schema within the registry | |
| `BASIC_CREDENTIALS_ACCESS_KEY_ID` | AWS access key ID | |
| `BASIC_CREDENTIALS_SECRET_ACCESS_KEY` | AWS secret access key | |
| `BASIC_CREDENTIALS_VALIDATE_CREDENTIALS` | Whether Gateway validates the credentials when they're set. Accepts `true` or `false`, in any case | `true` |
| `BASIC_CREDENTIALS_PROVIDER_NAME` | Name of the credentials provider to use | |
| `BASIC_CREDENTIALS_ACCOUNT_ID` | AWS account ID | |
Omit every `BASIC_CREDENTIALS_` variable and Gateway falls back to the AWS SDK's default credentials provider. [See what it searches](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/external-connections#aws-glue-schema-registries).
### External connections: schema registry client properties
Both schema registry types also accept any property their client library understands, in the same namespace. [See the formats and precedence rules](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/external-connections#pass-additional-client-properties).
| Variable suffix | Description |
| --------------- | ------------------------------------------------------------------------------------------ |
| `` | A single `key=value` pair, for example `_CONFIG_0=basic.auth.credentials.source=USER_INFO` |
| `JSON` | A JSON object of properties |
### External connections: HashiCorp Vault KMS
Set `TYPE` to `HASHICORP_VAULT`. These are the connection's own properties. Its credentials depend on `AUTH_TYPE` and are listed in the three sections that follow.
| Variable suffix | Description | Default |
| --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- |
| `URI` | Base URI of the Vault server, for example `https://vault.internal:8200`. Can't contain credentials | Required |
| `ENGINE` | Secrets engine backing the connection. `TRANSIT` is the only supported value | Required |
| `MOUNT` | Mount path of the engine, for example `transit` | Required |
| `AUTH_TYPE` | Auth method the connection uses: [`TOKEN`](#external-connections-vault-token-authentication), [`APP_ROLE`](#external-connections-vault-approle-authentication) or [`USERNAME_PASSWORD`](#external-connections-vault-username-and-password-authentication) | Required |
| `NAMESPACE` | Vault namespace, for Vault Enterprise | |
| `KEY_STORE_PATH` | Keystore holding the client certificate Gateway presents for mutual TLS | |
| `KEY_STORE_PASSWORD` | Password for that keystore | |
| `TRUST_STORE_PATH` | Truststore Gateway verifies the Vault server against | |
| `TRUST_STORE_PASSWORD` | Password for that truststore | |
| `CONNECTION_BACKOFF_DELAY` | First delay before Gateway retries a failed Vault connection | `5` |
| `CONNECTION_BACKOFF_MAX_DELAY` | Longest delay between retries | `30` |
| `CONNECTION_BACKOFF_CHRONO_UNIT` | Unit the two delays are measured in: `MILLIS`, `SECONDS`, `MINUTES` or `HOURS` | `SECONDS` |
| `CONNECTION_BACKOFF_DELAY_FACTOR` | Multiplier applied to the delay after each failed attempt | `1.1` |
Write `ENGINE`, `AUTH_TYPE` and `CONNECTION_BACKOFF_CHRONO_UNIT` in upper case, as shown. Gateway matches these values exactly, so `transit` and `seconds` fail startup.
#### External connections: Vault token authentication
Set `AUTH_TYPE` to `TOKEN`.
| Variable suffix | Description | Default |
| --------------- | -------------------------------------- | -------- |
| `AUTH_TOKEN` | Vault token Gateway authenticates with | Required |
Gateway doesn't renew a token you supply — with `AUTH_TYPE` set to `TOKEN` it never calls Vault's renew endpoint, so the token stays in use until its TTL expires, and recovering means replacing the variable and restarting. AppRole and username/password avoid that: Vault issues a short-lived token and Gateway renews it in the background. If you do use a token, give it a TTL that outlives your deployment window, or keep it fresh outside Gateway. [See Vault's production hardening guidance](https://developer.hashicorp.com/vault/docs/concepts/production-hardening) .
#### External connections: Vault AppRole authentication
Set `AUTH_TYPE` to `APP_ROLE`.
| Variable suffix | Description | Default |
| ---------------- | ----------------------------- | --------- |
| `AUTH_ROLE_ID` | AppRole role ID | Required |
| `AUTH_SECRET_ID` | AppRole secret ID | Required |
| `AUTH_MOUNT` | Mount path of the auth method | `approle` |
#### External connections: Vault username and password authentication
Set `AUTH_TYPE` to `USERNAME_PASSWORD`.
| Variable suffix | Description | Default |
| --------------- | ----------------------------- | ---------- |
| `AUTH_USERNAME` | Username | Required |
| `AUTH_PASSWORD` | Password | Required |
| `AUTH_MOUNT` | Mount path of the auth method | `userpass` |
### External connections: AWS KMS
Set `TYPE` to `AWS_KMS`. These are the connection's own properties. Its credentials depend on `CREDENTIALS_TYPE` and are listed in the three sections that follow.
| Variable suffix | Description | Default |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------- |
| `REGION` | AWS region holding the keys, for example `eu-west-1`. Lowercase | Required |
| `ACCOUNT_ID` | 12-digit AWS account id owning the keys | Required |
| `CREDENTIALS_TYPE` | Credentials the connection authenticates with: [`BASIC`](#external-connections-aws-kms-access-keys), [`SESSION`](#external-connections-aws-kms-temporary-credentials) or [`DEFAULT`](#external-connections-aws-kms-default-provider-chain) | Required |
Write `CREDENTIALS_TYPE` in upper case, as shown. Gateway matches the value exactly, so `basic` fails startup.
#### External connections: AWS KMS access keys
Set `CREDENTIALS_TYPE` to `BASIC`.
| Variable suffix | Description | Default |
| ------------------------------- | --------------------- | -------- |
| `CREDENTIALS_ACCESS_KEY_ID` | AWS access key ID | Required |
| `CREDENTIALS_SECRET_ACCESS_KEY` | AWS secret access key | Required |
#### External connections: AWS KMS temporary credentials
Set `CREDENTIALS_TYPE` to `SESSION`. Gateway doesn't refresh these, so they stop working when they expire.
| Variable suffix | Description | Default |
| ------------------------------- | --------------------- | -------- |
| `CREDENTIALS_ACCESS_KEY_ID` | AWS access key ID | Required |
| `CREDENTIALS_SECRET_ACCESS_KEY` | AWS secret access key | Required |
| `CREDENTIALS_SESSION_TOKEN` | AWS session token | Required |
#### External connections: AWS KMS default provider chain
Set `CREDENTIALS_TYPE` to `DEFAULT` and pass no credentials. Gateway resolves them through the AWS SDK's default provider chain. [See what it searches](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/external-connections#authenticate-with-the-aws-default-provider-chain).
### External connections: startup checks
Applies to the connection types Gateway contacts at startup, which means the two schema registry types. Gateway doesn't contact a KMS at startup.
| Variable | Description | Default |
| --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| `GATEWAY_EXTERNAL_CONNECTIONS_INIT_TIMEOUT_SECONDS` | How long Gateway waits for each connection's startup check. Maximum 300. Applies to schema registry connections, which Gateway contacts at startup. | `30` |
## Load balancing
| **Environment variable** | **Description** | **Default value** |
| ----------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | --------------------------------- |
| `GATEWAY_FEATURE_FLAGS_INTERNAL_LOAD_BALANCING` | Whether to use Conduktor Gateway's internal load balancer to balance connections between Gateway instances. | `true` |
| `GATEWAY_CLUSTER_ID` | A unique identifier for a given Gateway cluster, used to establish Gateway cluster membership for load balancing. | `gateway` |
| `GATEWAY_GROUP_ID` | Consumer group name used by Gateways in the same cluster to recognize each other. | `conduktor_${GATEWAY_CLUSTER_ID}` |
| `GATEWAY_RACK_ID` | Similar to `broker.rack`. | |
## Connect Gateway to Kafka
Conduktor Gateway's connection to Kafka is configured by the `KAFKA_` environment variables.
When translating Kafka's properties, **use upper case instead** and replace the `.` with `_`. For example:
When defining Gateway's Kafka property `bootstrap.servers`, declare it as the environment variable `KAFKA_BOOTSTRAP_SERVERS`. Any variable prefixed with `KAFKA_` will be treated as a connection parameter by Gateway.
### Connection pooling
| **Environment variable** | **Description** | **Default value** |
| --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | ----------------- |
| `GATEWAY_UPSTREAM_CONNECTION_POOL_TYPE` | Upstream connection pool type. Possible values are `NONE` (no connection pool), `ROUND_ROBIN` (Round robin selected connection pool) | `NONE` |
| `GATEWAY_UPSTREAM_NUM_CONNECTION` | The number of connections between Conduktor Gateway and Kafka per upstream thread. Used only when `ROUND_ROBIN` is enabled. | `10` |
### Cluster switch failover
Setting up your Kafka clusters for failover is similar to the standard setup, but you need to provide two sets of properties: one for your main cluster and one for your failover cluster.
You can define these properties as environment variables or load a cluster configuration file.
| **Environment variable** | **Description** |
| ---------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| `GATEWAY_BACKEND_KAFKA_SELECTOR` | Indicates the use of a configuration file and provides its path, e.g.: `'file: { path: /cluster-config.yaml }'`. |
| `KAFKA_FAILOVER_GATEWAY_ROLES` | To turn Gateway into failover mode, set this to `failover`. |
| **Main Cluster** | |
| `KAFKA_MAIN_BOOTSTRAP_SERVERS` | Bootstrap server. |
| `KAFKA_MAIN_SECURITY_PROTOCOL` | Security protocol. |
| `KAFKA_MAIN_SASL_MECHANISM` | SASL mechanism. |
| `KAFKA_MAIN_SASL_JAAS_CONFIG` | SASL JAAS config. |
| **Failover Cluster** | |
| `KAFKA_FAILOVER_BOOTSTRAP_SERVERS` | Bootstrap server. |
| `KAFKA_FAILOVER_SECURITY_PROTOCOL` | Security protocol. |
| `KAFKA_FAILOVER_SASL_MECHANISM` | SASL mechanism. |
| `KAFKA_FAILOVER_SASL_JAAS_CONFIG` | SASL JAAS config. |
## Security and authentication
These global settings determine where authentication and authorization take place, and how clients authenticate. The security protocol itself is set per listener — see [Listeners](#listeners).
| **Environment variable** | **Description** | **Default value** |
| ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GATEWAY_SECURITY_MODE` | Define where authentication takes place, Gateway or Kafka. Valid values are: `GATEWAY_MANAGED`, `KAFKA_MANAGED`. Note that `KAFKA_MANAGED` mode is incompatible with `PLAINTEXT` or `SSL` security protocols. | Required (set explicitly). For inference with legacy network configuration, see [Legacy network configuration](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/legacy-network-configuration#security-mode-and-protocol-inference). |
| `GATEWAY_ACL_ENABLED` | Enable/disable ACL support on the Gateway transparent virtual cluster (`passthrough`) only. | Required (set explicitly). For inference with legacy network configuration, see [Legacy network configuration](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/legacy-network-configuration#acl-default). |
| `GATEWAY_SUPER_USERS` | Semicolon-separated (`;`) list of service accounts that will be super users on Gateway (**excluding virtual clusters**). Example: `alice;bob`. | Usernames from GATEWAY\_ADMIN\_API\_USERS |
| `GATEWAY_FEATURE_FLAGS_MANDATORY_VCLUSTER` | If no virtual cluster was detected, the user then automatically falls back into the transparent virtual cluster called `passthrough`. If set to true, reject authentication if the principal (the identifier the authentication process provides) is not mapped to a named virtual cluster with a service account. This must be set to true when using the [Partner Zones](/guide/use-cases/third-party-data#partner-zones) feature. | `false` |
| `GATEWAY_AUTHENTICATION_CONNECTION_MAX_REAUTH_MS` | Force the client re-authentication after this amount of time. If set to 0, we never force the client to re-authenticate until the next connection | `0` |
| `GATEWAY_ACL_STORE_ENABLED` | **Obsolete**, use the *VirtualCluster* resource. Enable/disable ACLs support for Virtual Clusters only. | `false` |
| `GATEWAY_USER_POOL_SERVICE_ACCOUNT_REQUIRED` | **Obsolete** Verified the existence of user mapping for the service account when the user connects in Gateway Managed SASL/PLAIN mode. As of Gateway v3.9.0, this variable no longer exists: the behavior is always on. | |
Set the security protocol clients use to connect to Gateway per listener, with `GATEWAY_LISTENER__SECURITY_PROTOCOL` (see [Listeners](#listeners)). The global `GATEWAY_SECURITY_PROTOCOL` and its deprecated `DELEGATED_*` values apply only to [legacy network configuration](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/legacy-network-configuration).
### SSL authentication
| **Environment variable** | **Description** | **Default value** |
| --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------- |
| **Keystore** | | |
| `GATEWAY_SSL_KEY_STORE_PATH` | Path to a mounted keystore for SSL connections | |
| `GATEWAY_SSL_KEY_STORE_PASSWORD` | Password for the keystore defined above | |
| `GATEWAY_SSL_KEY_PASSWORD` | Password for the key contained in the store above | |
| `GATEWAY_SSL_KEY_TYPE` | `jks`or `pkcs12` | `jks` |
| `GATEWAY_SSL_UPDATE_CONTEXT_INTERVAL_MINUTES` | Interval in minutes to refresh SSL context | `5` |
| **Truststore (for mTLS)** | | |
| `GATEWAY_SSL_TRUST_STORE_PATH` | Path to a keystore for SSL connections | |
| `GATEWAY_SSL_TRUST_STORE_PASSWORD` | Password for the truststore defined above | |
| `GATEWAY_SSL_TRUST_STORE_TYPE` | `jks`, `pkcs12` | `jks` |
| `GATEWAY_SSL_PRINCIPAL_MAPPING_RULES` | mTLS leverages SSL mutual authentication to identify a Kafka client. Principal for mTLS connection can be detected from the subject certificate using the same feature as in Apache Kafka, the [SSL principal mapping](https://docs.confluent.io/platform/current/kafka/configure-mds/mutual-tls-auth-rbac.html#principal-mapping-rules-for-tls-ssl-listeners-extract-a-principal-from-a-certificate) | Extracts the subject |
| `GATEWAY_SSL_ENABLED_PROTOCOLS` | Comma-separated list of TLS protocol versions; restricts which protocols are offered during SSL/TLS handshake. If not provided it will fall back to the default protocols of security provider configured by `GATEWAY_SECURITY_PROVIDER`. | |
| `GATEWAY_SSL_CIPHER_SUITES` | Comma-separated list of cipher suites; restricts cryptographic algorithms that are offered during SSL/TLS handshake. If not provided it will fall back to the default ciphers of security provider configured by `GATEWAY_SECURITY_PROVIDER`. | |
The keystore and truststore are shared across all listeners. mTLS client authentication is set per listener with [`GATEWAY_LISTENER__SSL_CLIENT_AUTH`](#listeners) — the global `GATEWAY_SSL_CLIENT_AUTH` applies only to [legacy network configuration](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/legacy-network-configuration).
Password values (`GATEWAY_SSL_KEY_STORE_PASSWORD`, `GATEWAY_SSL_KEY_PASSWORD`, `GATEWAY_SSL_TRUST_STORE_PASSWORD`) support any characters except a literal single-quote (`'`). Pattern: `[^']*`
### OAuthbearer
Some of these definitions (e.g. `SASL_OAUTHBEARER_JWKS_ENDPOINT_REFRESH`) are taken from [Kafka documentation](https://kafka.apache.org/35/javadoc/constant-values.html#org.apache.kafka.common.config.SaslConfigs.SASL_OAUTHBEARER_JWKS_ENDPOINT_REFRESH_MS_DOC) .
| **Environment variable** | **Description** |
| ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GATEWAY_OAUTH_JWKS_URL` | The OAuth/OIDC provider URL from which the provider's JWKS (JSON Web Key Set) can be retrieved. The URL can be HTTP(S)-based or file-based. |
| `GATEWAY_OAUTH_EXPECTED_ISSUER` | The (optional) setting for the broker to use to verify that the JWT was created by the expected issuer. The JWT will be inspected for the standard OAuth `iss` claim and if this value is set, the broker will match it exactly against what is in the JWT's `iss` claim. If there's no match, the broker will reject the JWT and authentication will fail |
| `GATEWAY_OAUTH_EXPECTED_AUDIENCES` | The (optional) comma-delimited setting for the broker to use to verify that the JWT was issued for one of the expected audiences. The JWT will be inspected for the standard OAuth `aud` claim and if this value is set, the broker will match the value from JWT's `aud` claim to see if there is an exact match. If there's no match, the broker will reject the JWT and authentication will fail. |
| `GATEWAY_OAUTH_JWKS_REFRESH` | The (optional) value in milliseconds for the broker to wait between refreshing its JWKS (JSON Web Key Set) cache that contains the keys to verify the signature of the JWT. |
| `GATEWAY_OAUTH_JWKS_RETRY` | The (optional) value in milliseconds for the initial wait between JWKS (JSON Web Key Set) retrieval attempts from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the *sasl.oauthbearer.jwks.endpoint.retry.backoff.ms* setting and will double in wait length between attempts, up to a maximum wait length specified by **sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms**. |
| `GATEWAY_OAUTH_JWKS_MAX_RETRY` | The (optional) value in milliseconds for the maximum wait between attempts to retrieve the JWKS (JSON Web Key Set) from the external authentication provider. JWKS retrieval uses an exponential backoff algorithm with an initial wait based on the *sasl.oauthbearer.jwks.endpoint.retry.backoff.ms* setting and will double in wait length between attempts, up to a maximum wait length specified by **sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms**. |
| `GATEWAY_OAUTH_SCOPE_CLAIM_NAME` | The OAuth claim for the scope is often named `scope` but this (optional) setting can provide a different name to use for the scope included in the JWT payload's claims, if the OAuth/OIDC provider uses a different name for that claim. |
| `GATEWAY_OAUTH_SUB_CLAIM_NAME` | The OAuth claim for the subject is often named `sub`, but this (optional) setting can provide a different name to use for the subject included in the JWT payload's claims, if the OAuth/OIDC provider uses a different name for that claim. |
| `GATEWAY_OAUTH_USE_CC_POOL_ID` | Set to `true` to use the Confluent Cloud pool ID as the principal name. This is useful for Confluent Cloud users in Delegated mode who want to use the pool ID as the principal name instead of the `sub` claim. |
### Principal resolver
| **Environment variable** | **Description** | **Default value** |
| ----------------------------------------- | ----------------------------------------------------------------------------- | ------------------ |
| `GATEWAY_PRINCIPAL_RESOLVER` | The principal resolver to use. Currently only supported is `CONFLUENT_CLOUD`. | |
| `GATEWAY_CONFLUENT_CLOUD_API_KEY` | The Confluent Cloud API key (not the Kafka cluster API key). | |
| `GATEWAY_CONFLUENT_CLOUD_API_SECRET` | The Confluent Cloud API secret. | |
| `GATEWAY_CONFLUENT_CLOUD_CACHE_SIZE` | The number of principals to cache. | `1000` |
| `GATEWAY_CONFLUENT_CLOUD_CACHE_EXPIRY_MS` | The cache expiry time in milliseconds. | `86400000` (1 day) |
[Find out more about the principal resolver](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/listener-security#principal-resolver).
### Plain authentication
| **Environment variable** | **Description** | **Default value** |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------- |
| `GATEWAY_USER_POOL_SECRET_KEY` | Base64 encoded value of 256bits long (e.g. `openssl rand -base64 32`). If using SASL\_PLAIN or SASL\_SSL, you have the ability to create local service accounts on Gateway. These service accounts will have credentials generated by Gateway based on the `GATEWAY_USER_POOL_SECRET_KEY`. | No default value is provided. **You must provide this value for all deployments** |
### Security provider
| **Environment variable** | **Description** | **Default value** |
| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- |
| `GATEWAY_SECURITY_PROVIDER` | Specify your security provider. It can be: `DEFAULT` (from your JRE), `BOUNCY_CASTLE`, `BOUNCY_CASTLE_FIPS` and `CONSCRYPT`. Please note that `CONSCRYPT` doesn't support Mac OS with aarch64. | `DEFAULT` |
## HTTP API
| **Environment variable** | **Description** | **Default value** |
| ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- |
| `GATEWAY_HTTP_PORT` | The port on which Gateway will present the HTTP management API. | `8888` |
| `GATEWAY_SECURED_METRICS` | Determines whether the HTTP management API requires authentication. | `true` |
| `GATEWAY_ADMIN_API_USERS` | Users that can access the API. Set `admin: true` for full access (required for *write* operations). Set `admin: false` (default) for *read-only* access. | `[{username: admin, password: conduktor, admin: true}]` |
| **HTTPS configuration** | | |
| `GATEWAY_HTTPS_KEY_STORE_PATH` | Enables HTTPS and specifies the keystore to use for TLS connections. | |
| `GATEWAY_HTTPS_KEY_STORE_PASSWORD` | Sets the password for the keystore used in HTTPS TLS connections. | |
| `GATEWAY_HTTPS_CLIENT_AUTH` | Client authentication configuration for mTLS. Possible values: `NONE`, `REQUEST`, `REQUIRED`. | `NONE` |
| `GATEWAY_HTTPS_TRUST_STORE_PATH` | Specifies the truststore used for mTLS. | |
| `GATEWAY_HTTPS_TRUST_STORE_PASSWORD` | Password for the truststore defined above. | |
Password values (`GATEWAY_HTTPS_KEY_STORE_PASSWORD`, `GATEWAY_HTTPS_TRUST_STORE_PASSWORD`) support any characters except a literal single-quote (`'`). Pattern: `[^']*`
## Feature flags
| **Environment variable** | **Description** | **Default value** |
| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- |
| `GATEWAY_FEATURE_FLAGS_AUDIT` | Whether or not to enable the audit feature. | `true` |
| `GATEWAY_FEATURE_FLAGS_INTERNAL_LOAD_BALANCING` | Whether or not to enable Gateway's internal load balancing. | `true` |
| `GATEWAY_FEATURE_FLAGS_BLOCK_UNSUPPORTED_APIS` | Whether to block Kafka APIs that aren't explicitly supported by Gateway. [See API blocking modes](#api-blocking-modes). | `false` |
| `GATEWAY_FEATURE_FLAGS_TEST_TOKENIZATION` | Whether to enable the `test-tokenization://` KMS in the encryption plugin family. [Find out more about KMS integration](/guide/reference/data-security#kms-integration). | `false` |
| `GATEWAY_FEATURE_FLAGS_SERVICE_ACCOUNTS` | Whether to enable SCRAM credentials for local service accounts. When enabled, Gateway-managed SASL listeners advertise the `SCRAM-SHA-512` and `SCRAM-SHA-256` mechanisms and the [SCRAM credential endpoints](/guide/tutorials/manage-gateway-service-accounts#authenticate-with-scram-credentials) become available. | `false` |
| `GATEWAY_FEATURE_FLAGS_HAPROXY_PROTOCOL` | Whether to enable HAProxy protocol support. [Find out more about HAProxy Protocol](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/load-balancing#capturing-the-client-ip-address). | `false` |
| `GATEWAY_FEATURE_FLAGS_TOPIC_ID_SUPPORT` | Whether to enable topic-ID based Kafka protocol handling (preview). [See topic ID support](#topic-id-support-preview). | `false` |
| `GATEWAY_FEATURE_FLAGS_INCREMENTAL_FETCH_SESSIONS_ENABLED` | Whether to support Apache Kafka incremental fetch sessions ([KIP-227](https://cwiki.apache.org/confluence/display/KAFKA/KIP-227%3A+Introduce+Incremental+FetchRequests+to+Increase+Partition+Scalability)) on the Fetch API. You can't enable this at the same time as `GATEWAY_FEATURE_FLAGS_TOPIC_ID_SUPPORT`. [See incremental fetch sessions](#incremental-fetch-sessions). | `true` |
| `GATEWAY_AUTO_CREATE_TOPICS_ENABLED` | Enable auto-create topics feature. When enabled, topics can be automatically created when producing or consuming through Gateway, leveraging the Kafka property *auto.create.topics.enable*. **Authorization**: When enabled, users require either `CLUSTER` resource with `CREATE` permission (allows creating any topic) or `TOPIC` resource with `CREATE` permission for specific topics. **Warning:** this feature **doesn't support concentrated topics**. When auto-create topics is enabled, topics that would normally be concentrated not be, they'll simply be created as regular physical topics instead. Take caution when enabling this setting. | `false` |
### Topic ID support (preview)
Gateway supports topic-ID based Kafka protocol handling. Initial coverage is limited to the Fetch API: when disabled (default), Gateway advertises Fetch up to v12 during `ApiVersions` negotiation to force topic-name fetches; when enabled, clients can negotiate Fetch v13 or higher. Additional APIs will land behind the same flag in upcoming releases.
Enable by setting `GATEWAY_FEATURE_FLAGS_TOPIC_ID_SUPPORT=true`. You also have to set `GATEWAY_FEATURE_FLAGS_INCREMENTAL_FETCH_SESSIONS_ENABLED=false`, because [incremental fetch sessions](#incremental-fetch-sessions) are enabled by default and Gateway doesn't start when both flags are set.
This is a preview feature. We recommend enabling it only in non-production environments.
### Incremental fetch sessions
Apache Kafka clients can use incremental fetch sessions ([KIP-227](https://cwiki.apache.org/confluence/display/KAFKA/KIP-227%3A+Introduce+Incremental+FetchRequests+to+Increase+Partition+Scalability)) to make Fetch requests smaller: a consumer opens a session with a first, full Fetch request, then sends only the partitions that changed on each later request while the broker tracks the rest. Gateway supports these sessions through the proxy and enables them by default.
To opt out, set `GATEWAY_FEATURE_FLAGS_INCREMENTAL_FETCH_SESSIONS_ENABLED=false`.
#### Incompatibilities
* **Topic ID support**: you can't combine incremental fetch sessions with [topic ID support](#topic-id-support-preview). Gateway doesn't start when both `GATEWAY_FEATURE_FLAGS_INCREMENTAL_FETCH_SESSIONS_ENABLED` and `GATEWAY_FEATURE_FLAGS_TOPIC_ID_SUPPORT` are set to `true`. Incremental fetch sessions are on by default, so to use the topic ID preview you have to set `GATEWAY_FEATURE_FLAGS_INCREMENTAL_FETCH_SESSIONS_ENABLED=false`.
* **SQL and CEL topics**: these deprecated [logical topics](/guide/conduktor-concepts/logical-topics) are incompatible with incremental fetch sessions and return incorrect results unless you opt out. [Topic views](/guide/conduktor-concepts/logical-topics#topic-views) aren't affected.
If you use the [audit Interceptor](/guide/reference/data-security#audit-interceptor) or a [fetch policy](/guide/use-cases/apply-traffic-control-policies#fetch-policy), incremental fetch sessions change how they behave. See those pages for details.
## Licensing
| **Environment variable** | **Description** | **Default value** |
| ------------------------ | -------------------------- | ----------------- |
| `GATEWAY_LICENSE_KEY` | License key (**required**) | |
## Monitoring
### Audit
| **Environment variable** | **Description** | **Default value** |
| ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | ----------------- |
| `GATEWAY_AUDIT_LOG_CONFIG_SPEC_VERSION` | Version of the log. | `0.1.0` |
| `GATEWAY_AUDIT_LOG_SERVICE_BACKING_TOPIC` | Target topic name. | `_auditLogs` |
| `GATEWAY_AUDIT_LOG_REPLICATION_FACTOR_OF_TOPIC` | Replication factor to be used when creating the audit topic, defaults to the one defined in your cluster settings. | `-1` |
| `GATEWAY_AUDIT_LOG_NUM_PARTITIONS_OF_TOPIC` | Number of partitions to be used when creating the audit topic, defaults to the one defined in your cluster settings. | `-1` |
| `GATEWAY_AUDIT_LOG_KAFKA_` | Overrides Kafka Producer configuration for audit logs, i.e.: `GATEWAY_AUDIT_LOG_KAFKA_LINGER_MS=0` | |
| `GATEWAY_AUDIT_LOG_EVENT_TYPES` | Comma-separated list of event types for audit log, defaults to `ALL` to generate audit log for all event types. | `ALL` |
`GATEWAY_AUDIT_LOG_EVENT_TYPES` valid value is `ALL` (default) or a comma-separated list of the following:
* `CONNECTION` for initial connection event, i.e. receiving Kafka's `API_VERSIONS` request
* `AUTHENTICATION` for authentication success or failure event
* `REST_API` for all REST v1 admin requests and all REST v2 requests ([see API reference](/guide/conduktor-in-production/automate/api-automation))
* `ACL_ADMIN` for ACL management events done via Kafka Admin API, i.e. `CREATE_ACLS` and `DELETE_ACLS` requests
* `AUTHORIZATION` for authorization failure events, i.e. when ACL permissions deny an operation on a resource
[Find out more about Gateway audit event types, including Kafka API key auditing](/guide/conduktor-in-production/admin/audit-logs#gateway-audit-event-types).
### Logging
| **Environment variable** | **Description** | **Default value** | **Package** |
| ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | ---------------------------------------- |
| `LOG4J2_APPENDER_LAYOUT` | The format to output Console logging. Use `json` for json layout or `pattern` for pattern layout. | `pattern` | |
| `LOG4J2_TIME_FORMAT` | Timestamp format in log output. | `yyyy-MM-dd'T'HH:mm:ss.SSSZ` | |
| `LOG4J2_IO_CONDUKTOR_PROXY_NETWORK_LEVEL` | Low-level networking, connection mapping, authentication, authorization. | `info` | io.conduktor.proxy.network |
| `LOG4J2_IO_CONDUKTOR_UPSTREAM_THREAD_LEVEL` | Requests processing and forwarding. At `trace`, log requests sent. | `warn` | io.conduktor.proxy.thread.UpstreamThread |
| `LOG4J2_IO_CONDUKTOR_PROXY_REBUILDER_COMPONENTS_LEVEL` | Requests and responses rewriting. Logs responses payload in `debug` (useful for checking METADATA). | `info` | io.conduktor.proxy.rebuilder.components |
| `LOG4J2_IO_CONDUKTOR_PROXY_SERVICE_LEVEL` | Various. Logs ACL checks and Interceptor targeting at `debug`. Logs post-interceptor requests/response payload at `trace`. | `info` | io.conduktor.proxy.service |
| `LOG4J2_IO_CONDUKTOR_PROXY_AUTHORIZATION_LEVEL` | Authorization decision logs. | `info` | io.conduktor.proxy.authorization |
| `LOG4J2_IO_CONDUKTOR_LEVEL` | Get even more logs not covered by specific packages. | `info` | io.conduktor |
| `LOG4J2_ORG_APACHE_KAFKA_LEVEL` | Kafka log level. | `warn` | org.apache.kafka |
| `LOG4J2_IO_KCACHE_LEVEL` | Kcache log level (our persistence library). | `warn` | io.kcache |
| `LOG4J2_IO_VERTX_LEVEL` | Vertx log level (our HTTP API framework). | `warn` | io.vertx |
| `LOG4J2_IO_NETTY_LEVEL` | Netty log level (our network framework). | `error` | io.netty |
| `LOG4J2_IO_MICROMETER_LEVEL` | Micrometer log level (our metrics framework). | `error` | io.micrometer |
| `LOG4J2_IO_CONFLUENT_LEVEL` | Confluent library logs. | `warn` | io.confluent |
| `LOG4J2_COM_FORTANIX_LEVEL` | Fortanix SDK logs (suppressed by default). | `off` | com.fortanix |
| `LOG4J2_ROOT_LEVEL` | Root logging level (applies to anything else that hasn't been listed above). | `info` | (root) |
### Product analytics
| **Environment variable** | **Description** | **Default value** |
| --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- |
| `GATEWAY_FEATURE_FLAGS_ANALYTICS` | Conduktor collects basic product analytics to understand usage and support you proactively, such as a Gateway Started event and an hourly configuration summary. This is not based on any of the underlying Kafka data, which is never sent to Conduktor. [Find out what product telemetry contains](/guide/support/product-telemetry). | `true` |
## Advanced configuration
These settings are for advanced tuning and debugging. Most deployments don't need to change them.
### Threading
| **Environment variable** | **Description** | **Default value** |
| --------------------------- | ------------------------------------------------------------------------------------- | ----------------- |
| `GATEWAY_DOWNSTREAM_THREAD` | The number of threads dedicated to handling IO between clients and Conduktor Gateway. | Number of cores |
| `GATEWAY_UPSTREAM_THREAD` | The number of threads dedicated to handling IO between Kafka and Conduktor Gateway. | Number of cores |
### Gateway internal timeout
These timeout thresholds are used by Gateway and typically don't need modification since they match Apache Kafka's defaults.
If your Kafka brokers or clients use non-default timeout values, you may want to adjust these.
| **Environment variable** | **Description** | **Default value** |
| ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- |
| `GATEWAY_INFLIGHT_REQUEST_EXPIRY_MS` | Timeout for in-flight requests. If Kafka doesn't respond before this delay, Gateway will respond with a request timeout to the client. Default is `300000`. It has to exceed the consumer's `max.poll.interval.ms` and client's `request.timeout.ms` because JOIN\_GROUP or SYNC\_GROUP requests can take up to that duration per KIP-62. | `330000` |
| `GATEWAY_UPSTREAM_MAX_IDLE_TIME_MS` | Maximum time Gateway connections can remain idle before being closed, in milliseconds. Default is `600000`. It has to exceed the Kafka broker's `connections.max.idle.ms` and should be set higher than `GATEWAY_INFLIGHT_REQUEST_EXPIRY_MS`. | `660000` |
### Graceful shutdown
| **Environment variable** | **Description** | **Default value** |
| ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | ----------------- |
| `GATEWAY_SHUTDOWN_DELAY_BETWEEN_BROKERS_MS` | The pause between disconnection of broker clients during shutdown process. Set to `1000` or higher for librdkafka clients. | `0` |
During shutdown, Gateway closes client connections in controlled manner to simulate rolling Kafka broker restart, with `GATEWAY_SHUTDOWN_DELAY_BETWEEN_BROKERS_MS` pause between brokers. For librdkafka clients, set `GATEWAY_SHUTDOWN_DELAY_BETWEEN_BROKERS_MS` to `1000` or higher to prevent them from crashing during shutdown because of `ALL_BROKERS_DOWN` error.
### Internal topics
As Gateway is stateless, it uses Kafka topics to store its internal state. Use the following environment variables to configure these internal topics.
If missing, Gateway will automatically create the topics (if it has the permission to do so). You can also create the topics independently of Gateway, just make sure they are configured as described below.
#### Internal state
Firstly, there are some general configuration settings for Gateway internal state management which apply to all used topics.
| **Environment variable** | **Description** | **Default value** |
| ----------------------------------------------- | ------------------------------------------------------- | ----------------- |
| `GATEWAY_STORE_TTL_MS` | Time between full refresh. | `604800000` |
| `GATEWAY_TOPIC_STORE_KCACHE_REPLICATION_FACTOR` | Defaults to the value defined in your cluster settings. | `-1` |
#### Topic names
| **Environment variable** | **Description** | **Default value** |
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------- |
| `GATEWAY_TOPIC_MAPPINGS_TOPIC` | Topic where the topics aliases are stored. | `_conduktor_${GATEWAY_CLUSTER_ID}_topicmappings` |
| `GATEWAY_USER_MAPPINGS_TOPIC` | Topic where the service accounts are stored. | `_conduktor_${GATEWAY_CLUSTER_ID}_usermappings` |
| `GATEWAY_SERVICE_ACCOUNTS_TOPIC` | Topic where local service account SCRAM credentials are stored. Used when `GATEWAY_FEATURE_FLAGS_SERVICE_ACCOUNTS` is enabled. | `_conduktor_${GATEWAY_CLUSTER_ID}_serviceaccounts` |
| `GATEWAY_CONSUMER_OFFSETS_TOPIC` | Topic where the offsets for concentrated topic consumption are stored. | `_conduktor_${GATEWAY_CLUSTER_ID}_consumer_offsets` |
| `GATEWAY_INTERCEPTOR_CONFIGS_TOPIC` | Topic where the deployed Interceptors are stored. | `_conduktor_${GATEWAY_CLUSTER_ID}_interceptor_configs` |
| `GATEWAY_ACLS_TOPIC` | Topic where the ACLs managed by Gateway are stored. | `_conduktor_${GATEWAY_CLUSTER_ID}_acls` |
| `GATEWAY_AUDIT_LOG_TOPIC` | Topic where Gateway audit log is stored. | `_conduktor_${GATEWAY_CLUSTER_ID}_auditlogs` |
| `GATEWAY_VCLUSTERS_TOPIC` | Topic where the virtual clusters are stored. | `_conduktor_${GATEWAY_CLUSTER_ID}_vclusters` |
| `GATEWAY_GROUPS_TOPIC` | Topic where the service account groups are stored. | `_conduktor_${GATEWAY_CLUSTER_ID}_groups` |
| `GATEWAY_ENCRYPTION_KEYS_TOPIC` | Name of the topic for storing EDEKs when `gateway` KMS enabled in encryption Interceptors | `_conduktor_${GATEWAY_CLUSTER_ID}_encryption_keys` |
| `GATEWAY_DATA_QUALITY_TOPIC` | Topic where the data quality violation are stored. | `_conduktor_${GATEWAY_CLUSTER_ID}_data_quality_violation` |
#### Required topic configuration
The most important setting is `log.cleanup.policy` which defines the clean up policy for the topic. Most of the topics used by Gateway are compacted, but some use time-based retention. If this isn't set up properly, Gateway will throw an error on startup. Set the following:
* `log.cleanup.policy=compact` for compaction
* `log.cleanup.policy=delete` for time based retention
If Gateway creates the topics for you, it will set the right values.
The second vital setting is the **replication factor**. This should be set to **at least 3 in production environments** to ensure that the data is safe (Gateway will warn you on startup, if it's set to less than three). When creating topics, Gateway uses the default value for your Kafka brokers for this setting.
For partition count, most of the topics are low volume and can operate well with only a single partition. This isn't enforced (Gateway will work with multi partition topics for internal state), however there is no need to have more than one partition.
The exception to this is the audit log topic which can have a lot of events written to it, if enabled for a busy cluster. We recommend starting with 3 partitions for audit logs (this doesn't affect Gateway performance as it is a writer, not a reader), but will impact any other consumers you may run reading from it.
| Topic | Cleanup policy | Recommended partitions | Other configuration |
| --------------------------------------------------------- | -------------- | ---------------------- | ------------------------------------------------------------------------------------- |
| `_conduktor_${GATEWAY_CLUSTER_ID}_topicmappings` | compact | 1 | |
| `_conduktor_${GATEWAY_CLUSTER_ID}_usermappings` | compact | 1 | |
| `_conduktor_${GATEWAY_CLUSTER_ID}_consumer_offsets` | compact | 1 | |
| `_conduktor_${GATEWAY_CLUSTER_ID}_interceptor_configs` | compact | 1 | |
| `_conduktor_${GATEWAY_CLUSTER_ID}_acls` | compact | 1 | |
| `_conduktor_${GATEWAY_CLUSTER_ID}_vclusters` | compact | 1 | |
| `_conduktor_${GATEWAY_CLUSTER_ID}_groups` | compact | 1 | |
| `_conduktor_${GATEWAY_CLUSTER_ID}_encryption_keys` | compact | 1 | |
| `_conduktor_${GATEWAY_CLUSTER_ID}_auditlogs` | delete | 3 | We recommend a retention time of around 7 days for this topic due to its high volume. |
| `_conduktor_${GATEWAY_CLUSTER_ID}_data_quality_violation` | delete | 1 | |
#### Data Quality topic configs
| **Environment variable** | **Description** | **Default value** |
| ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- |
| `GATEWAY_DATA_QUALITY_TOPIC` | Target topic name | `_conduktor_${GATEWAY_CLUSTER_ID}_data_quality_violation` |
| `GATEWAY_DATA_QUALITY_TOPIC_REPLICATION_FACTOR` | Replication factor to be used when creating the data quality topic, defaults to the one defined in your cluster settings | cluster default |
| `GATEWAY_DATA_QUALITY_TOPIC_PARTITIONS` | Number of partitions to be used when creating the data quality topic, defaults to the one defined in your cluster settings | cluster default |
| `GATEWAY_DATA_QUALITY_TOPIC_RETENTION_HOUR` | Retention period (in hours) to be used when creating the data quality topic | 168 (7 days) |
### API blocking modes
The `GATEWAY_FEATURE_FLAGS_BLOCK_UNSUPPORTED_APIS` environment variable controls how Gateway handles Kafka APIs that aren't explicitly supported.
**Two modes:**
* **Permissive mode** (`false`, default): Gateway allows unsupported APIs to pass through - maintains backward compatibility
* **Restrictive mode** (`true`): Gateway blocks unsupported APIs for enhanced security
**When to use each:**
* Use **permissive** (default) when upgrading or if you have legacy applications
* Use **restrictive** when you want maximum security and only allow known, supported APIs
**Impact of each mode:**
**Permissive mode (`false`):**
* ✅ **Backward compatibility**: Legacy applications continue to work
* ✅ **Smooth upgrades**: No breaking changes during Gateway updates
* ⚠️ **Security risk**: Unsupported APIs bypass Gateway's security controls
**Restrictive mode (`true`):**
* ✅ **Enhanced security**: Only explicitly supported APIs are allowed
* ✅ **Controlled environment**: Prevents access to potentially problematic APIs
* ❌ **Breaking changes**: Applications using unsupported APIs will fail
**APIs not supported by Gateway:**
Gateway categorizes unsupported APIs into two groups:
1. **Stability-blocked APIs** (always blocked):
* **Stability** (APIs not yet fully implemented or tested in Gateway):
* `DESCRIBE_TOPIC_PARTITIONS` - New API for describing topic partitions
* `CONSUMER_GROUP_DESCRIBE` - New consumer group protocol API (KIP-848)
* `CONSUMER_GROUP_HEARTBEAT` - New consumer group protocol API (KIP-848)
* **New Kafka 4 group protocols** (Gateway has no rebuilder for these, so it blocks them for every client):
* Share groups (KIP-932), also known as Kafka queues: `SHARE_GROUP_HEARTBEAT`, `SHARE_GROUP_DESCRIBE`, `SHARE_FETCH`, `SHARE_ACKNOWLEDGE`, `INITIALIZE_SHARE_GROUP_STATE`, `READ_SHARE_GROUP_STATE`, `WRITE_SHARE_GROUP_STATE`, `DELETE_SHARE_GROUP_STATE`, `READ_SHARE_GROUP_STATE_SUMMARY`, `DESCRIBE_SHARE_GROUP_OFFSETS`, `ALTER_SHARE_GROUP_OFFSETS`, `DELETE_SHARE_GROUP_OFFSETS`
* Streams groups (KIP-1071), the new Kafka Streams rebalance protocol: `STREAMS_GROUP_HEARTBEAT`, `STREAMS_GROUP_DESCRIBE`
* **Admin operations** (these operations should be performed directly on the underlying Kafka cluster rather than through Gateway):
* `ALTER_REPLICA_LOG_DIRS` - Admin operation for altering replica log directories
* `ELECT_LEADERS` - Admin operation for leader election
* `ALTER_PARTITION_REASSIGNMENTS` - Admin operation for partition reassignments
* `LIST_PARTITION_REASSIGNMENTS` - Admin operation for listing partition reassignments
* `ALTER_PARTITION` - Admin operation for altering partitions
* `ALLOCATE_PRODUCER_IDS` - Admin operation for producer ID allocation
* `CREATE_DELEGATION_TOKEN` - Admin operation for creating delegation tokens
* `RENEW_DELEGATION_TOKEN` - Admin operation for renewing delegation tokens
* `EXPIRE_DELEGATION_TOKEN` - Admin operation for expiring delegation tokens
* `DESCRIBE_DELEGATION_TOKEN` - Admin operation for describing delegation tokens
* `DESCRIBE_USER_SCRAM_CREDENTIALS` - Admin operation for describing SCRAM credentials
* `ALTER_USER_SCRAM_CREDENTIALS` - Admin operation for altering SCRAM credentials
* `DESCRIBE_CLIENT_QUOTAS` - Admin operation for describing client quotas
* `ALTER_CLIENT_QUOTAS` - Admin operation for altering client quotas
* `UPDATE_FEATURES` - Admin operation for updating cluster features
2. **Unknown/Unsupported APIs** (affected by this setting):
* **KIP-1000 Configuration APIs** (bypass ACL checks):
* `LIST_CLIENT_METRICS_RESOURCES` (KIP-1000) - List client metrics configuration resources without DESCRIBE\_CONFIGS permission
* **Internal/Admin APIs** (not meant for client use):
* `LEADER_AND_ISR` - Internal leader election and ISR management
* `STOP_REPLICA` - Internal replica management
* `UPDATE_METADATA` - Internal metadata updates
* `CONTROLLED_SHUTDOWN` - Internal broker shutdown coordination
* `WRITE_TXN_MARKERS` - Internal transaction marker management
* `DESCRIBE_QUORUM` - Internal quorum management
* `ENVELOPE` - Internal request envelope handling
* `UNREGISTER_BROKER` - Internal broker registration management
* `ADD_RAFT_VOTER` - Internal Raft consensus management
* `REMOVE_RAFT_VOTER` - Internal Raft consensus management
* **Future Kafka APIs** not yet implemented in Gateway
* **Any other APIs** that may bypass Gateway's security controls or don't work correctly with Gateway features
**Security implications:**
When `GATEWAY_FEATURE_FLAGS_BLOCK_UNSUPPORTED_APIS=false` (permissive mode):
* Unsupported APIs are passed through to Kafka without Gateway processing
* **Configuration exposure**: KIP-1000 APIs allow access to sensitive configuration without proper permissions
* **Internal API access**: Internal/Admin APIs meant for broker-to-broker communication may be accessible to clients
* Topic mapping and other Gateway features may not work correctly
* Potential for unauthorized access to restricted resources and information disclosure
The KIP-932 share-group and KIP-1071 streams-group APIs are **not** affected by this setting — Gateway always blocks them, in both modes.
When `GATEWAY_FEATURE_FLAGS_BLOCK_UNSUPPORTED_APIS=true` (restrictive mode):
* All unsupported APIs are blocked at the Gateway level
* **Prevents security vulnerabilities**: Blocks KIP-1000 APIs that bypass ACL checks
* **Prevents internal API access**: Blocks internal/Admin APIs that should not be accessible to clients
* Enhanced security through explicit allow-list approach
* Applications must use only Gateway-supported APIs
**Example of problematic situation:**
**`ListClientMetricsResources` API bypasses ACL checks:**
When `GATEWAY_FEATURE_FLAGS_BLOCK_UNSUPPORTED_APIS=false` (permissive mode), clients can call the `ListClientMetricsResources` API without proper authorization. This API is designed to list all client metrics configuration resources in the cluster.
**Attack scenario:**
1. **Unauthorized client** connects to Gateway without `DESCRIBE_CONFIGS` permission on `CLUSTER` resource
2. **Client calls** `ListClientMetricsResources` API directly
3. **Gateway forwards** the request to Kafka without ACL validation (since it's not in the supported APIs list)
4. **Kafka responds** with complete list of client metrics configuration resources
5. **Client receives** sensitive configuration information including:
* All configured client metrics resources
* Internal cluster configuration details
* Resource names and configurations that should be protected
**Security impact:**
* **Information disclosure**: Unauthorized access to sensitive cluster configuration
* **ACL bypass**: Complete circumvention of Gateway's permission system
* **Compliance violation**: Access to restricted data without proper authorization
* **Reconnaissance**: Attackers can enumerate cluster resources for further attacks
**Why this happens:**
The `ListClientMetricsResources` API is not included in Gateway's supported APIs list, so when permissive mode is enabled, it bypasses all Gateway security controls and is forwarded directly to Kafka without ACL validation.
This setting only affects unsupported APIs. APIs that are explicitly blocked for security or stability reasons (like consumer group management) will always be blocked, regardless of this setting.
### Low-level network configuration
These variables should not be changed without consulting the [Conduktor support team](https://support.conduktor.io/hc/en-gb/requests/new). They allow tuning and debugging uncommon network behavior.
| **Environment variable** | **Description** | **Default value** |
| --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- |
| `GATEWAY_NETWORK_CONNECTION_BACKLOG` | TCP accept queue size (SO\_BACKLOG). Limits pending new connections requests, between handshake and accept(). | `1024` |
| `GATEWAY_NETWORK_MAX_MESSAGES_PER_READ` | Messages processed per event loop iteration. Prevents single channel from monopolizing resources. | `16` |
| `GATEWAY_NETWORK_MAX_FRAME_SIZE` | Maximum size of a single Kafka request/response frame. Prevents memory exhaustion from malformed/malicious large frames. | `104857600` (100 MB) |
| `GATEWAY_NETWORK_MAX_CONNECTIONS` | Maximum total concurrent established connections. | `2147483647` |
| `GATEWAY_NETWORK_MAX_PENDING_REQUESTS` | Maximum in-flight requests in the queue. | `5` |
| `GATEWAY_NETWORK_USE_POOLED_ALLOCATOR` | Whether to use pooled ByteBuf allocators for Netty I/O operations. Pooled allocators reuse buffers to reduce allocation overhead and GC pressure. | `true` |
| `GATEWAY_NETWORK_USE_DIRECT_BUFFERS` | Whether to use direct buffers for Netty I/O operations. | `true` |
# Configure Gateway external connections (preview)
Source: https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/external-connections
Declare a connection to an external system once in Conduktor Gateway, then reference it by name from Interceptors and topic views instead of repeating it.
An external connection is a named, reusable declaration of one external system, made when Gateway starts. Interceptors and topic views then reference it by name, instead of each carrying its own copy of the address and credentials.
**Preview functionality**. This is a preview feature and is subject to change. Inline configuration remains fully supported.
For topic views this is the only way to configure a schema registry.
For Interceptors, prefer an external connection when several resources use the same registry: the address and credentials live in one place instead of being repeated in every Interceptor, so rotating a credential or moving a registry is a single edit. Inline configuration keeps working and is not deprecated.
## Declare a connection
Declare each connection with environment variables that follow this pattern:
```
GATEWAY_EXTERNAL_CONNECTION__CONFIG_
```
`` is a label you choose, and is how a resource references the connection. `` is one of the properties the connection's type accepts.
Connection names have to start with a letter and can contain letters, numbers and underscores, so `DEFAULT_SR` is a valid name. A name can't contain `_CONFIG_`, because that's the delimiter between the name and the property. Gateway rejects such a variable at startup rather than guessing where the name ends. Names are case-insensitive and Gateway stores them in upper case, so `default_sr` and `DEFAULT_SR` are the same connection. We recommend you declare them in upper case to match the variable.
Every connection needs a `TYPE`, which selects the system it connects to:
| `TYPE` | System | Properties |
| --------------------------- | --------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| `CONFLUENT_SCHEMA_REGISTRY` | Confluent Schema Registry and compatible registries, including Redpanda Schema Registry | [Confluent-like schema registries](#confluent-like-schema-registries) |
| `AWS_SCHEMA_REGISTRY` | AWS Glue Schema Registry | [AWS Glue schema registries](#aws-glue-schema-registries) |
| `HASHICORP_VAULT` | HashiCorp Vault, for encryption keys | [HashiCorp Vault](#hashicorp-vault) |
| `AWS_KMS` | AWS Key Management Service, for encryption keys | [AWS KMS](#aws-kms) |
Every connection follows the same shape. This one is a Confluent-like schema registry named `DEFAULT_SR`, which needs only a URL:
```bash theme={null}
GATEWAY_EXTERNAL_CONNECTION_DEFAULT_SR_CONFIG_TYPE=CONFLUENT_SCHEMA_REGISTRY
GATEWAY_EXTERNAL_CONNECTION_DEFAULT_SR_CONFIG_SCHEMA_REGISTRY_URL=http://schema-registry:8081
```
Gateway rejects an unrecognized property at startup, naming the variable and listing the properties the connection's type accepts, so a typo never becomes a silently ignored setting.
## Reference a connection
A resource reaches a connection through a field in its own configuration:
| Resource | Field |
| ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------- |
| Interceptors that read schemas — [data security](/guide/reference/data-security) and [data quality](/guide/conduktor-in-production/admin/gateway-policies) | `schemaRegistryConnectionName` |
| [Topic views](/guide/conduktor-concepts/logical-topics#read-schema-encoded-topics) | `spec.schemaRegistryConnectionName` |
| [Encryption Interceptors](/guide/reference/data-security#name-a-kms-external-connection) | `kmsConnectionName` |
Set the field to the connection's ``. The schema registry connection declared above is named `DEFAULT_SR`, so an Interceptor or topic view reaches it with:
```json theme={null}
"schemaRegistryConnectionName": "DEFAULT_SR"
```
Gateway checks the name when you create or update the resource, and rejects an unknown one with `400 Bad Request`. Remove a connection that a resource already names and the resource stays saved and visible through the API, but stops serving traffic.
An Interceptor can name a schema registry connection, or configure a registry inline in its own `schemaRegistryConfig`. Gateway rejects a configuration that sets both rather than silently picking one.
## Connect to a schema registry
Gateway needs a schema registry to decode Avro, Protobuf and JSON-Schema records. Both Interceptors and topic views can reference a schema registry connection. Gateway contacts the registry at startup and fails to start if it can't be reached — see [what happens at startup](#what-happens-at-startup) for the timeout, the permissions the connection needs and what the log shows.
### Confluent-like schema registries
Set `TYPE` to `CONFLUENT_SCHEMA_REGISTRY` for Confluent Schema Registry and compatible registries, including Redpanda Schema Registry:
```bash theme={null}
GATEWAY_EXTERNAL_CONNECTION_DEFAULT_SR_CONFIG_TYPE=CONFLUENT_SCHEMA_REGISTRY
GATEWAY_EXTERNAL_CONNECTION_DEFAULT_SR_CONFIG_SCHEMA_REGISTRY_URL=http://schema-registry:8081
```
For every property this type accepts, see the [environment variables reference](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables#external-connections-confluent-like-schema-registry).
To reach a registry that needs authentication, pass the registry client's own properties as [additional client properties](#pass-additional-client-properties).
Don't put credentials in `SCHEMA_REGISTRY_URL`. Gateway rejects a URL carrying user information before an `@` and names the connection, so pass them as additional client properties instead.
### AWS Glue schema registries
Set `TYPE` to `AWS_SCHEMA_REGISTRY` for AWS Glue Schema Registry:
```bash theme={null}
GATEWAY_EXTERNAL_CONNECTION_PROD_SR_CONFIG_TYPE=AWS_SCHEMA_REGISTRY
GATEWAY_EXTERNAL_CONNECTION_PROD_SR_CONFIG_REGION=us-east-1
GATEWAY_EXTERNAL_CONNECTION_PROD_SR_CONFIG_BASIC_CREDENTIALS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
GATEWAY_EXTERNAL_CONNECTION_PROD_SR_CONFIG_BASIC_CREDENTIALS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
```
For every property this type accepts, see the [environment variables reference](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables#external-connections-aws-glue-schema-registry).
Leave every `BASIC_CREDENTIALS_` property unset and Gateway falls back to the AWS SDK's default credentials provider, which searches environment variables, Java system properties, the shared credentials and config files, and container or instance IAM roles. [See how the AWS SDK resolves default credentials](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials-chain.html).
### Pass additional client properties
A schema registry connection also accepts any property its registry's own client understands. Gateway doesn't interpret these properties. It collects them and hands the map to the client, so a key the client doesn't recognize passes startup rather than failing it.
Use indexed entries, where each value is a `key=value` pair split at the first `=`:
```bash theme={null}
GATEWAY_EXTERNAL_CONNECTION_DEFAULT_SR_CONFIG_0=basic.auth.credentials.source=USER_INFO
GATEWAY_EXTERNAL_CONNECTION_DEFAULT_SR_CONFIG_1=basic.auth.user.info=user:password
```
Or pass a JSON object, which suits keys containing `=` and migration from existing JSON configuration:
```bash theme={null}
GATEWAY_EXTERNAL_CONNECTION_DEFAULT_SR_CONFIG_JSON={"basic.auth.credentials.source":"USER_INFO","basic.auth.user.info":"user:password"}
```
Both formats work together, under two rules:
* **Indexed entries win.** An indexed entry overrides the same key in the JSON object, so you can patch one value without rewriting the object.
* **Two indexed entries can't share a key.** There's no patch order between them, so Gateway rejects the pair and names the key.
A client property can address the same setting as a typed one — `_CONFIG_0=schema.registry.url=…` beside `_CONFIG_SCHEMA_REGISTRY_URL`, for example. Gateway doesn't check for the overlap, and which of the two it connects with is left to the registry's client.
Which keys to use depends on the registry:
| Connection type | Keys are | Reference |
| --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `CONFLUENT_SCHEMA_REGISTRY` | Confluent Schema Registry client configuration. This is also how the connection authenticates, since it has no typed credential properties: use `basic.auth.credentials.source` and `basic.auth.user.info`, as above | [Schema Registry security](https://docs.confluent.io/platform/current/schema-registry/security/index.html) |
| `AWS_SCHEMA_REGISTRY` | AWS Glue Schema Registry serde configuration, merged over the typed properties. Credentials come from the `BASIC_CREDENTIALS_` properties or the AWS SDK's default chain, so these keys cover other client settings | [AWS Glue Schema Registry library](https://github.com/awslabs/aws-glue-schema-registry) |
## Connect to a KMS
Gateway encrypts data in Kafka records with keys held in a key management service. An external connection holds the provider's address, credentials and account details, so an Interceptor names a key rather than a full key URI.
Not every provider is available as an external connection yet. Encryption Interceptors reach [more of them](/guide/reference/data-security#choose-your-kms-provider) through their own inline `kmsConfig`, so configure a provider inline until it arrives here.
### HashiCorp Vault
Set `TYPE` to `HASHICORP_VAULT`. This example authenticates with a token, one of three auth methods covered below:
```bash theme={null}
GATEWAY_EXTERNAL_CONNECTION_PROD_VAULT_CONFIG_TYPE=HASHICORP_VAULT
GATEWAY_EXTERNAL_CONNECTION_PROD_VAULT_CONFIG_URI=https://vault.internal:8200
GATEWAY_EXTERNAL_CONNECTION_PROD_VAULT_CONFIG_ENGINE=TRANSIT
GATEWAY_EXTERNAL_CONNECTION_PROD_VAULT_CONFIG_MOUNT=transit
GATEWAY_EXTERNAL_CONNECTION_PROD_VAULT_CONFIG_AUTH_TYPE=TOKEN
GATEWAY_EXTERNAL_CONNECTION_PROD_VAULT_CONFIG_AUTH_TOKEN=hvs.CAESIExampleToken
```
For every property a Vault connection accepts, including mutual TLS and connection retry, see the [environment variables reference](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables#external-connections-hashicorp-vault-kms).
An external connection reaches less of Vault than inline `kmsConfig` does. We're widening it in future releases, and until then inline configuration is the way to use anything missing here:
* **Secrets engines.** An external connection supports the [Transit secrets engine](https://developer.hashicorp.com/vault/docs/secrets/transit) (`MOUNT=transit` above). Inline configuration also supports [Transform](/guide/reference/data-security#vault-kms), which needs Vault Enterprise.
* **Auth methods.** An external connection supports the three below, out of the [eleven inline configuration offers](/guide/reference/data-security#vault-kms).
Don't put credentials in `URI`. Gateway rejects a URI carrying user information before an `@` and names the connection, so authenticate with `AUTH_TYPE` and its credentials instead.
#### Authenticate with a token
Set `AUTH_TYPE` to `TOKEN` and `AUTH_TOKEN` to the Vault token Gateway authenticates with:
```bash theme={null}
GATEWAY_EXTERNAL_CONNECTION_PROD_VAULT_CONFIG_AUTH_TYPE=TOKEN
GATEWAY_EXTERNAL_CONNECTION_PROD_VAULT_CONFIG_AUTH_TOKEN=hvs.CAESIExampleToken
```
For every property this method accepts, see the [environment variables reference](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables#external-connections-vault-token-authentication).
Gateway doesn't renew a token you supply — with `AUTH_TYPE` set to `TOKEN` it never calls Vault's renew endpoint, so the token stays in use until its TTL expires, and recovering means replacing the variable and restarting. AppRole and username/password avoid that: Vault issues a short-lived token and Gateway renews it in the background. If you do use a token, give it a TTL that outlives your deployment window, or keep it fresh outside Gateway. [See Vault's production hardening guidance](https://developer.hashicorp.com/vault/docs/concepts/production-hardening) .
#### Authenticate with AppRole
Set `AUTH_TYPE` to `APP_ROLE` and pass the role ID and secret ID. Vault exchanges them for a short-lived token, which Gateway [renews in the background](/guide/reference/data-security#vault-kms):
```bash theme={null}
GATEWAY_EXTERNAL_CONNECTION_PROD_VAULT_CONFIG_AUTH_TYPE=APP_ROLE
GATEWAY_EXTERNAL_CONNECTION_PROD_VAULT_CONFIG_AUTH_ROLE_ID=59d6d1ca-47bb-4e7e-a40b-8be3bc5a0ba8
GATEWAY_EXTERNAL_CONNECTION_PROD_VAULT_CONFIG_AUTH_SECRET_ID=84896a0c-1347-aa90-a4f6-aca8b7558780
```
For every property this method accepts, including `AUTH_MOUNT` when you mounted it somewhere other than `approle`, see the [environment variables reference](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables#external-connections-vault-approle-authentication).
#### Authenticate with a username and password
Set `AUTH_TYPE` to `USERNAME_PASSWORD` and pass the userpass login. Vault exchanges it for a short-lived token, which Gateway [renews in the background](/guide/reference/data-security#vault-kms):
```bash theme={null}
GATEWAY_EXTERNAL_CONNECTION_PROD_VAULT_CONFIG_AUTH_TYPE=USERNAME_PASSWORD
GATEWAY_EXTERNAL_CONNECTION_PROD_VAULT_CONFIG_AUTH_USERNAME=gateway
GATEWAY_EXTERNAL_CONNECTION_PROD_VAULT_CONFIG_AUTH_PASSWORD=s3cr3t
```
For every property this method accepts, including `AUTH_MOUNT` when you mounted it somewhere other than `userpass`, see the [environment variables reference](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables#external-connections-vault-username-and-password-authentication).
### AWS KMS
Set `TYPE` to `AWS_KMS`. A connection covers one AWS account in one region. This example authenticates with access keys, one of three credential types covered below:
```bash theme={null}
GATEWAY_EXTERNAL_CONNECTION_PROD_AWS_CONFIG_TYPE=AWS_KMS
GATEWAY_EXTERNAL_CONNECTION_PROD_AWS_CONFIG_REGION=eu-west-1
GATEWAY_EXTERNAL_CONNECTION_PROD_AWS_CONFIG_ACCOUNT_ID=123456789012
GATEWAY_EXTERNAL_CONNECTION_PROD_AWS_CONFIG_CREDENTIALS_TYPE=BASIC
GATEWAY_EXTERNAL_CONNECTION_PROD_AWS_CONFIG_CREDENTIALS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
GATEWAY_EXTERNAL_CONNECTION_PROD_AWS_CONFIG_CREDENTIALS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
```
For every property an AWS KMS connection accepts, see the [environment variables reference](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables#external-connections-aws-kms).
`REGION` and `ACCOUNT_ID` are both required, because Gateway builds each key's full ARN from them — so an Interceptor names a key as a bare key id or `alias/`, never an ARN. [See how the ARN is built and what's rejected](/guide/reference/data-security#aws-kms).
The connection needs `kms:Encrypt` and `kms:Decrypt` on every key it uses, allowed by the key policy and by the identity's IAM policy.
#### Authenticate with access keys
Set `CREDENTIALS_TYPE` to `BASIC` and pass a long-lived access key pair:
```bash theme={null}
GATEWAY_EXTERNAL_CONNECTION_PROD_AWS_CONFIG_CREDENTIALS_TYPE=BASIC
GATEWAY_EXTERNAL_CONNECTION_PROD_AWS_CONFIG_CREDENTIALS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
GATEWAY_EXTERNAL_CONNECTION_PROD_AWS_CONFIG_CREDENTIALS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
```
For every property this type accepts, see the [environment variables reference](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables#external-connections-aws-kms-access-keys).
#### Authenticate with temporary credentials
Set `CREDENTIALS_TYPE` to `SESSION` and pass the session token alongside the key pair. Gateway doesn't refresh these, so they stop working when they expire:
```bash theme={null}
GATEWAY_EXTERNAL_CONNECTION_PROD_AWS_CONFIG_CREDENTIALS_TYPE=SESSION
GATEWAY_EXTERNAL_CONNECTION_PROD_AWS_CONFIG_CREDENTIALS_ACCESS_KEY_ID=ASIAIOSFODNN7EXAMPLE
GATEWAY_EXTERNAL_CONNECTION_PROD_AWS_CONFIG_CREDENTIALS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
GATEWAY_EXTERNAL_CONNECTION_PROD_AWS_CONFIG_CREDENTIALS_SESSION_TOKEN=IQoJb3JpZ2luX2VjEXAMPLE
```
For every property this type accepts, see the [environment variables reference](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables#external-connections-aws-kms-temporary-credentials).
#### Authenticate with the AWS default provider chain
Set `CREDENTIALS_TYPE` to `DEFAULT` and pass no credentials at all. Gateway uses the AWS SDK's default provider chain, which searches the container or instance role, IAM Roles for Service Accounts, the shared credentials file and the `AWS_` environment variables. This suits a Gateway that already runs with an AWS identity, and it's the only type where the credentials rotate without a restart:
```bash theme={null}
GATEWAY_EXTERNAL_CONNECTION_PROD_AWS_CONFIG_CREDENTIALS_TYPE=DEFAULT
```
[See how the AWS SDK resolves default credentials](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials-chain.html) .
## What happens at startup
Connections are read once, when Gateway starts, and this is true of every connection type:
* **Configuration is validated.** An unrecognized property, or a malformed value such as a JSON object of [additional client properties](#pass-additional-client-properties), fails startup and names the variable.
* **Connections load before persisted resources.** Interceptors and topic views are activated from storage after the connections exist, so their names resolve.
* **A missing connection doesn't stop Gateway.** If an Interceptor or topic view names a connection that's no longer declared, that resource stays saved and visible through the API but doesn't activate, while everything else starts normally. Gateway logs an error naming the missing connection — [check the startup log](#verify-the-configuration).
* **Secrets are masked** in the startup log. Additional client property values are masked whichever format set them, and their keys stay visible.
Whether Gateway also *contacts* the external system at startup depends on the connection type:
| Connection type | Contacted at startup |
| -------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| Schema registry — `CONFLUENT_SCHEMA_REGISTRY`, `AWS_SCHEMA_REGISTRY` | Yes. Gateway fails to start if a registry can't be reached. |
| KMS — `HASHICORP_VAULT`, `AWS_KMS` | No. Gateway validates the connection's properties but doesn't contact the provider. |
Gateway checks schema registry connections in parallel and waits for all of them to finish, so one broken connection doesn't hide another — you see every failure in one startup attempt. Each attempt is bounded by `GATEWAY_EXTERNAL_CONNECTIONS_INIT_TIMEOUT_SECONDS`, which defaults to 30 seconds and caps at 300. [See the variable reference](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables#external-connections-startup-checks).
The check, and the permission it needs, differ by registry:
* **Confluent-like** — Gateway lists the registry's subjects, so the connection needs permission to list them even if your Interceptors only read schemas by ID.
* **AWS Glue** — Gateway requests a schema version that deliberately doesn't exist, so the connection needs `glue:GetSchemaVersion` and a not-found answer counts as success.
A KMS connection is established when an Interceptor that names it loads, so a wrong token or an unreachable Vault server surfaces then rather than at boot, and Gateway starts normally either way. Check the Interceptor's own status after deploying it, not the startup log.
## Verify the configuration
Gateway logs each schema registry connection's startup check. Look for one line per connection:
```
External connection DEFAULT_SR initialized in 84ms
```
A failure names the connection and the cause, and Gateway then reports how many failed before it stops:
```
External connection PROD_SR failed: ...
1 of 2 external connections failed: PROD_SR
```
These lines cover schema registry connections only, since those are the ones Gateway contacts. With none declared — including when you've declared only KMS connections — Gateway logs `No external connections configured` and starts normally.
Gateway reports KMS connections separately, one line each:
```
Loaded 2 KMS connection(s):
PROD_VAULT [HASHICORP_VAULT] engine=TRANSIT mount=transit uri=https://vault.internal:8200
PROD_AWS [AWS_KMS] region=eu-west-1 accountId=123456789012 credentials=BASIC keyUri=aws-kms://arn:aws:kms:eu-west-1:123456789012:
```
With none declared, that becomes `No KMS connections configured`. Neither summary includes a credential.
## Current limitations
* **Connections are read and checked once, at startup.** Changing one takes a restart. Gateway doesn't monitor a system that becomes unreachable afterwards, so that failure surfaces on traffic needing an uncached schema.
* **A schema-backed topic view has to use `SELECT *`.** A `WHERE` clause is still allowed. See [read schema-encoded topics](/guide/conduktor-concepts/logical-topics#read-schema-encoded-topics).
* **Gateway doesn't contact a KMS at startup.** It validates the connection's properties, so a wrong credential or an unreachable provider surfaces when an Interceptor that names it loads, not at boot.
* **An Interceptor that names a removed connection stops applying, rather than failing.** It stays saved and visible through the API, but after a restart the traffic it covered passes through unprocessed. An upcoming release will reject that traffic instead.
* **There's no API that lists configured connections.** Use the startup log to confirm what Gateway loaded.
## Related resources
* [Gateway environment variables](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables#external-connections-preview)
* [Read schema-encoded topics with a topic view](/guide/conduktor-concepts/logical-topics#read-schema-encoded-topics)
* [Data security Interceptors](/guide/reference/data-security)
* [Kafka data validation policies](/guide/conduktor-in-production/admin/gateway-policies)
* [Configure Gateway listeners](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/listeners)
# Kafka disaster recovery with Gateway failover
Source: https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/failover
Configure Conduktor Gateway to redirect Kafka clients to a secondary cluster during disaster recovery, without restarting or reconfiguring Kafka clients.
In a disaster recovery or business continuity scenario, we want to be able to switch clients from one Kafka cluster (the primary) to another one (the secondary) without having to reconfigure the clients.
Reconfiguring clients would at least involve changing the bootstrap servers of the clients, forcing the clients to refresh the metadata and retry all messages in flight. It might also involve distributing new credentials to the clients. For example, API keys and secrets in Confluent Cloud are tied to a specific cluster. Other Kafka providers might have different restrictions.
Essentially, this implies that the central operations/Kafka team (who would be responsible for initiating the failover-process) would have knowledge about all clients, which in practice is not feasible. The failover capability of Gateway solves this by redirecting client connections from the primary to the secondary cluster. This can be initiated at a central location using the Gateway HTTP API without having to reconfigure/restart each Kafka client individually.
**From our blog:** [How Gateway reduces Kafka DR from hours to minutes](https://conduktor.io/blog/how-gateway-reduces-kafka-dr-from-hours-to-minutes) Gateway's failover in action: redirect Kafka clients without touching bootstrap servers or credentials.
## Prerequisites
### Data replication is already in place
Gateway does not currently provide any mechanism to replicate already written data from the primary to the secondary cluster. Therefore, to make use of our solution, you should already have this mechanism in place. Common solutions for this include:
* MirrorMaker 2
* Confluent Replicator
* Confluent Cluster Linking
Note that none of these solutions (and therefore neither Conduktor's failover solution) can guarantee the absence of data loss during a disaster scenario.
### Kafka client configuration
No specific client configuration is necessary, besides ensuring that clients have configured enough retries (or that the `delivery.timeout.ms` for JVM-based clients) setting is large enough to cover the time necessary for the operations team to discover failure of the primary cluster and initiate a failover procedure. Especially for JVM-based clients, the default delivery timeout of 2 minutes might be too short.
### System requirements
* Gateway version `3.3.0`+
* Kafka brokers version `2.8.2`+
Note that due to a current limitation in Kafka clients, the primary and secondary Kafka clusters must have some broker id's in common (see [KIP-899](https://cwiki.apache.org/confluence/display/KAFKA/KIP-899%3A+Allow+producer+and+consumer+clients+to+rebootstrap) ). This ensures clients can recognize the secondary cluster as a legitimate continuation of the primary one.
## How it works
Conduktor Gateway acts as a 'hot-switch' to the secondary Kafka cluster, eliminating the need to change any client configurations in a disaster scenario. This is achievable because Gateway de-couples authentication between clients and the backing Kafka cluster(s).
Note that to initiate failover, it must be triggered through an API request to every Gateway instance. The Conduktor team can support you in finding the best solution for initiating failover, depending on your deployment specificities.
## Set up Gateway
To set up Gateway for failover, you should configure the primary and secondary clusters along with their configuration properties. This can be achieved through a **cluster-config file**, or through **environment variables**.
### Configuring through a cluster-config file
Specify your primary and secondary cluster configurations, along with a `gateway.roles` entry to mark the failover cluster - note that the API keys differ in the Confluent Cloud example below:
```yaml theme={null}
config:
main:
bootstrap.servers: :9092
security.protocol: SASL_SSL
sasl.mechanism: PLAIN
sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username="" password="";
failover:
bootstrap.servers: :9092
security.protocol: SASL_SSL
sasl.mechanism: PLAIN
sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username="" password="";
gateway.roles: failover
```
Mount the cluster config file in the Gateway container using the configuration `GATEWAY_BACKEND_KAFKA_SELECTOR`:
```yaml theme={null}
GATEWAY_BACKEND_KAFKA_SELECTOR: 'file : { path: /cluster-config.yaml}'
```
### Configuring through environment variables
Alternatively, you can configure your primary and secondary cluster through environment variables:
```bash theme={null}
KAFKA_MAIN_BOOTSTRAP_SERVERS=':9092'
KAFKA_MAIN_SECURITY_PROTOCOL='SASL_SSL'
KAFKA_MAIN_SASL_MECHANISM='PLAIN'
KAFKA_MAIN_SASL_JAAS_CONFIG='org.apache.kafka.common.security.plain.PlainLoginModule required username="" password="";'
KAFKA_FAILOVER_BOOTSTRAP_SERVERS=':9092'
KAFKA_FAILOVER_SECURITY_PROTOCOL='SASL_SSL'
KAFKA_FAILOVER_SASL_MECHANISM='PLAIN'
KAFKA_FAILOVER_SASL_JAAS_CONFIG='org.apache.kafka.common.security.plain.PlainLoginModule required username="" password="";'
KAFKA_FAILOVER_GATEWAY_ROLES='failover'
```
## Initiating failover
The cluster switch API updates which Kafka cluster the main connection points to.
* **fromPhysicalCluster**: The physical cluster whose connection is being updated. In a failover setup, client traffic always goes through the main cluster slot, so this is always `main`.
* **toPhysicalCluster**: The physical cluster whose connection configuration is used (either `main` or `failover`).
To fail over from the primary to the secondary cluster, make the following request to all Gateway instances:
```bash theme={null}
curl \
--request POST 'http://localhost:8888/gateway/v2/cluster-switching' \
--header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
--header 'Content-Type: application/json' \
--data-raw '{
"fromPhysicalCluster": "main",
"toPhysicalCluster": "failover"
}
```
This updates the main cluster's connection to use the failover cluster's configuration, so client traffic is redirected to the secondary cluster.
## Switching back
To switch back from the secondary cluster to the primary cluster, set both parameters to `main` and make the following request to all Gateway instances:
```bash theme={null}
curl \
--request POST 'http://localhost:8888/gateway/v2/cluster-switching' \
--header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
--header 'Content-Type: application/json' \
--data-raw '{
"fromPhysicalCluster": "main",
"toPhysicalCluster": "main"
}
```
### Alternative solutions to switchover
Note that Conduktor can recommend alternative solutions for initiating the switchover that does not involve making an API call to every Gateway instance. These alternatives are dependent on your deployment configuration, therefore we recommend [contacting us](https://www.conduktor.io/contact/demo/?utm_source=docs\&utm_medium=webpage) to discuss this.
### Failover limitation
During a failover event, Chargeback will only collect data for the original cluster. During a failover event data is not collected but would resume if failed back to the original cluster.
## Related resources
* [Connect Gateway to Kafka](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/connect-to-kafka)
* [Resource reference](/guide/reference/gateway-reference)
* [Gateway system requirements](/guide/conduktor-in-production/system-requirements#scaling-gateway)
# Conduktor Gateway on Kubernetes — Helm deployment
Source: https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/kubernetes
Deploy Conduktor Gateway on Kubernetes using the official Helm chart.
We recommend deploying Conduktor Gateway on Kubernetes using the official [Helm chart](https://helm.conduktor.io) for the best experience and supportability. Kubernetes keeps stateless services like Gateway highly available with built-in routing, scaling and self-healing.
## Local example
We recommend running the [local-stack example](https://github.com/conduktor/conduktor-reference-architecture/tree/main/local-stack) from our Conduktor Reference Architecture repository.
This stack creates a production-like local deployment of the entire Conduktor Platform using [k3d](https://k3d.io/) as a local Kubernetes cluster. It includes:
| Component | Required for Gateway? | Details |
| -------------------------- | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Conduktor Gateway** | Yes | Deployed with [security mode](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/listener-security#security-mode) `GATEWAY_MANAGED` and SNI routing. |
| **Kafka** | Yes | Required by both Gateway and Console. |
| **Conduktor Console** | No | UI and API for managing Kafka resources. |
| **PostgreSQL** | No | Stores Console state. Required for Console |
| **Conduktor Cortex** | No | Metrics collection for Console. |
| **Schema Registry** | No | Schema management for Gateway and Console. |
| **HashiCorp Vault** | No | Secret storage for Gateway encryption features. |
| **MinIO S3** | No | S3-compatible storage for metrics. |
| **Prometheus and Grafana** | No | Monitoring dashboards for Gateway and Console. |
Gateway will only start if a valid license is provided (version 3.18+). If you’d like to evaluate Gateway, [contact us](https://www.conduktor.io/contact).
### Helm values
Inspect the [helm values for Gateway](https://github.com/conduktor/conduktor-reference-architecture/blob/main/local-stack/gateway-values.yaml).
The sensitive configurations are provided by reference to a [Kubernetes Secret](https://github.com/conduktor/conduktor-reference-architecture/blob/main/local-stack/gateway-secrets.yaml).
In production, manage secrets with a dedicated secret manager. Don't store them unencrypted in a git repository.
## Preserving client IP address
By default, the Kubernetes load balancer changes the client IP address to its own. See [Capturing the client IP address](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/load-balancing#capturing-the-client-ip-address) for how to preserve it, using `externalTrafficPolicy` on the Gateway Helm chart or HAProxy Protocol.
## Next steps
* [Configure environment variables](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables) to tune Gateway for your environment
* Inspect the full Helm [values.yaml](https://github.com/conduktor/conduktor-public-charts/blob/main/charts/gateway/values.yaml)
* [Set up Interceptors](/guide/reference/interceptor-reference) to apply policies to your traffic
* [Configure listener security](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/listener-security) to define how Kafka clients authenticate to Gateway
* [Hands-on tutorial: Deploy Conduktor Gateway with Kubernetes and Host-based Routing](https://github.com/conduktor/conduktor-gateway-kubernetes-tutorial/) with walkthrough video
## Chart dependencies
All charts in this repository depend on [`bitnami-common`](https://github.com/bitnami/charts/tree/main/bitnami/common) .
## Compatibility matrix
This compatibility matrix is a resource to help you find which versions of Conduktor Gateway work on which version of our Conduktor Gateway Helm chart.
> We recommend you use the version of Gateway that comes pre-configured with the Helm chart. You can adjust the version in your values property according to the supported Gateway version, if required.
> Notes column only lists **chart-level changes**. See Conduktor [release notes](https://docs.conduktor.io/changelog/) to determine whether there are breaking changes within the artifacts.
### Helm chart compatibility
Breaking changes:
🟡 - breaks additional services / small behavior changes (e.g. Grafana dashboard changes)
🔴 - breaks overall deployment of the product (e.g. renaming variables in .values, major product releases)
| Chart version | Default App Version | Min App Version | Release date | Notes |
| ---------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- | --------------------------------------------------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [conduktor-gateway-3.23.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.23.0) | **[3.21.0](/guide/release-notes#gateway-3-21-0)** | [3.18.0](/guide/release-notes#gateway-3-18-0) | 2026-08-18 | |
| [conduktor-gateway-3.22.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.22.0) | **[3.20.0](/guide/release-notes#gateway-3-20-0)** | [3.18.0](/guide/release-notes#gateway-3-18-0) | 2026-08-03 | |
| [conduktor-gateway-3.21.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.21.1) | **[3.20.0](/guide/release-notes#gateway-3-20-0)** | [3.18.0](/guide/release-notes#gateway-3-18-0) | 2026-06-29 | |
| [conduktor-gateway-3.21.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.21.0) | **[3.20.0](/guide/release-notes#gateway-3-20-0)** | [3.18.0](/guide/release-notes#gateway-3-18-0) | 2026-06-17 | 🔴 [Deprecated gateway.portRange in favour of gateway.listeners - old gateway.portRange needs gateway.portRange.enable = true to work](https://github.com/conduktor/conduktor-public-charts/pull/283) |
| [conduktor-gateway-3.20.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.20.1) | **[3.19.1](/guide/release-notes#gateway-3-19-1)** | [3.18.0](/guide/release-notes#gateway-3-18-0) | 2026-06-11 | |
| [conduktor-gateway-3.20.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.20.0) | **[3.19.1](/guide/release-notes#gateway-3-19-1)** | [3.18.0](/guide/release-notes#gateway-3-18-0) | 2026-05-27 | |
| [conduktor-gateway-3.19.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.19.1) | **[3.19.1](/guide/release-notes#gateway-3-19-1)** | [3.12.0](/guide/release-notes#gateway-3-12-0) | 2026-05-26 | |
| [conduktor-gateway-3.19.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.19.0) | **[3.19.0](/guide/release-notes#gateway-3-19-0)** | [3.12.0](/guide/release-notes#gateway-3-12-0) | 2026-05-05 | |
| [conduktor-gateway-3.18.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.18.0) | **[3.18.0](/guide/release-notes#gateway-3-18-0)** | [3.12.0](/guide/release-notes#gateway-3-12-0) | 2026-03-18 | |
| [conduktor-gateway-3.17.2](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.17.2) | **[3.17.2](/guide/release-notes#gateway-3-17-2)** | [3.12.0](/guide/release-notes#gateway-3-12-0) | 2026-02-19 | |
| [conduktor-gateway-3.17.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.17.1) | **[3.17.1](/guide/release-notes#gateway-3-17-1)** | [3.12.0](/guide/release-notes#gateway-3-12-0) | 2026-02-10 | |
| [conduktor-gateway-3.17.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.17.0) | **[3.17.0](/guide/release-notes#gateway-3-17-0)** | [3.12.0](/guide/release-notes#gateway-3-12-0) | 2026-01-29 | |
| [conduktor-gateway-3.16.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.16.1) | **[3.16.0](/guide/release-notes#gateway-3-16-0)** | [3.12.0](/guide/release-notes#gateway-3-12-0) | 2026-01-08 | |
| [conduktor-gateway-3.16.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.16.0) | **[3.16.0](/guide/release-notes#gateway-3-16-0)** | [3.12.0](/guide/release-notes#gateway-3-12-0) | 2025-12-12 | |
| [conduktor-gateway-3.15.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.15.0) | **[3.15.0](/guide/release-notes#gateway-3-15-0)** | [3.12.0](/guide/release-notes#gateway-3-12-0) | 2025-11-17 | |
| [conduktor-gateway-3.14.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.14.0) | **[3.14.0](/guide/release-notes#gateway-3-14-0)** | [3.12.0](/guide/release-notes#gateway-3-12-0) | 2025-10-27 | |
| [conduktor-gateway-3.13.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.13.0) | **[3.13.0](/guide/release-notes#gateway-3-13-0)** | [3.12.0](/guide/release-notes#gateway-3-12-0) | 2025-10-01 | 🟡 [Added podAntiAffinity default preset](https://github.com/conduktor/conduktor-public-charts/pull/226) |
| [conduktor-gateway-3.12.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.12.1) | **[3.12.0](/guide/release-notes#gateway-3-12-0)** | [3.12.0](/guide/release-notes#gateway-3-12-0) | 2025-08-22 | 🔴 [Updated startupProbe, livenessProbe, readinessProbe endpoints](https://github.com/conduktor/conduktor-public-charts/pull/214) |
| [conduktor-gateway-3.12.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.12.0) | **[3.12.0](/guide/release-notes#gateway-3-12-0)** | [3.1.0](/guide/release-notes/archive#gateway-3-1-0) | 2025-08-22 | |
| [conduktor-gateway-3.11.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.11.0) | **[3.11.0](/guide/release-notes#gateway-3-11-0)** | [3.1.0](/guide/release-notes/archive#gateway-3-1-0) | 2025-07-22 | |
| [conduktor-gateway-3.10.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.10.0) | **[3.10.0](/guide/release-notes/archive#gateway-3-10-0)** | [3.1.0](/guide/release-notes/archive#gateway-3-1-0) | 2025-06-19 | 🔴 The default value of the environment variable `GATEWAY_ACL_ENABLED` has been switched to `true` 🔴 Deprecated `DELEGATED_SASL_PLAINTEXT` and `DELEGATED_SASL_SSL` security protocols (they remain supported for backward compatibility) [Find out more from release notes](https://docs.conduktor.io/changelog/#gateway-3100). |
| [conduktor-gateway-3.9.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.9.1) | **[3.9.0](/guide/release-notes/archive#gateway-3-9-0)** | [3.1.0](/guide/release-notes/archive#gateway-3-1-0) | 2025-06-02 | |
| [conduktor-gateway-3.9.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.9.0) | **[3.9.0](/guide/release-notes/archive#gateway-3-9-0)** | [3.1.0](/guide/release-notes/archive#gateway-3-1-0) | 2025-05-14 | 🔴 When using PLAIN tokens, Gateway service accounts are now always required 🔴 When using PLAIN tokens, Gateway JWT signing key must always be set |
| [conduktor-gateway-3.8.2](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.8.2) | **[3.8.1](/guide/release-notes/archive#gateway-3-8-1)** | [3.1.0](/guide/release-notes/archive#gateway-3-1-0) | 2025-04-24 | |
| [conduktor-gateway-3.8.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.8.1) | **[3.8.0](/guide/release-notes/archive#gateway-3-8-0)** | [3.1.0](/guide/release-notes/archive#gateway-3-1-0) | 2025-04-10 | |
| [conduktor-gateway-3.8.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.8.0) | **[3.8.0](/guide/release-notes/archive#gateway-3-8-0)** | [3.1.0](/guide/release-notes/archive#gateway-3-1-0) | 2025-04-09 | |
| [conduktor-gateway-3.7.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.7.1) | **[3.7.0](/guide/release-notes/archive#gateway-3-7-0)** | [3.1.0](/guide/release-notes/archive#gateway-3-1-0) | 2025-03-28 | |
| [conduktor-gateway-3.7.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.7.0) | **[3.7.0](/guide/release-notes/archive#gateway-3-7-0)** | [3.1.0](/guide/release-notes/archive#gateway-3-1-0) | 2025-03-21 | 🟡 [Updated Grafana template](https://github.com/conduktor/conduktor-public-charts/pull/160) 🟡 [Removed dependency on in-built Kafka cluster](https://github.com/conduktor/conduktor-public-charts/pull/154) . Now the chart checks that the `KAFKA_BOOTSTRAP_SERVERS` environment variable is set in `gateway.extraSecretEnvVars` or `gateway.env` before deploying the chart. |
| [conduktor-gateway-3.6.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.6.1) | **[3.6.1](/guide/release-notes/archive#gateway-3-6-1)** | [3.1.0](/guide/release-notes/archive#gateway-3-1-0) | 2025-03-06 | |
| [conduktor-gateway-3.6.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.6.0) | **[3.6.0](/guide/release-notes/archive#gateway-3-6-0)** | [3.1.0](/guide/release-notes/archive#gateway-3-1-0) | 2025-02-17 | |
| [conduktor-gateway-3.5.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.5.1) | **[3.5.2](/guide/release-notes/archive#gateway-3-5-2)** | [3.1.0](/guide/release-notes/archive#gateway-3-1-0) | 2025-01-23 | |
| [conduktor-gateway-3.5.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.5.0) | **[3.5.0](/guide/release-notes/archive#gateway-3-5-0)** | [3.1.0](/guide/release-notes/archive#gateway-3-1-0) | 2024-12-16 | |
| [conduktor-gateway-3.4.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.4.1) | **[3.4.0](/guide/release-notes/archive#gateway-3-4-0)** | [3.1.0](/guide/release-notes/archive#gateway-3-1-0) | 2024-11-22 | |
| [conduktor-gateway-3.4.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.4.0) | **[3.4.0](/guide/release-notes/archive#gateway-3-4-0)** | [3.1.0](/guide/release-notes/archive#gateway-3-1-0) | 2024-11-22 | 🔴 [Changed service account creation behavior](https://github.com/conduktor/conduktor-public-charts/pull/125) 🟡 [Updated Grafana template](https://github.com/conduktor/conduktor-public-charts/pull/123) |
| [conduktor-gateway-3.3.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.3.1) | **[3.3.1](/guide/release-notes/archive#gateway-3-3-1)** | [3.1.0](/guide/release-notes/archive#gateway-3-1-0) | 2024-09-27 | |
| [conduktor-gateway-3.3.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.3.0) | **[3.3.0](/guide/release-notes/archive#gateway-3-3-0)** | [3.1.0](/guide/release-notes/archive#gateway-3-1-0) | 2024-09-05 | |
| [conduktor-gateway-3.2.2](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.2.2) | **[3.2.2](/guide/release-notes/archive#gateway-3-2-2)** | [3.1.0](/guide/release-notes/archive#gateway-3-1-0) | 2024-08-28 | 🟡 [Updated Grafana template](https://github.com/conduktor/conduktor-public-charts/pull/98) |
| [conduktor-gateway-3.2.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.2.1) | **[3.2.1](/guide/release-notes/archive#gateway-3-2-1)** | [3.1.0](/guide/release-notes/archive#gateway-3-1-0) | 2024-08-01 | |
| [conduktor-gateway-3.2.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.2.0) | **[3.2.0](/guide/release-notes/archive#gateway-3-2-0)** | [3.1.0](/guide/release-notes/archive#gateway-3-1-0) | 2024-07-19 | |
| [conduktor-gateway-3.1.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.1.1) | **[3.1.1](/guide/release-notes/archive#gateway-3-1-1)** | [3.1.0](/guide/release-notes/archive#gateway-3-1-0) | 2024-06-21 | |
| [conduktor-gateway-3.1.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.1.0) | **[3.0.3](/guide/release-notes/archive#gateway-3-0-3)** | [3.0.0](/guide/release-notes/archive#gateway-3-0-0) | 2024-06-12 | 🟡 [Updated Grafana template](https://github.com/conduktor/conduktor-public-charts/pull/81) |
| [conduktor-gateway-3.0.1](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.0.1) | **[3.0.1](/guide/release-notes/archive#gateway-3-0-1)** | [3.0.0](/guide/release-notes/archive#gateway-3-0-0) | 2024-04-15 | |
| [conduktor-gateway-3.0.0](https://github.com/conduktor/conduktor-public-charts/releases/tag/conduktor-gateway-3.0.0) | **[3.0.0](/guide/release-notes/archive#gateway-3-0-0)** | [3.0.0](/guide/release-notes/archive#gateway-3-0-0) | 2024-04-05 | 🔴 [Major product update](https://github.com/conduktor/conduktor-public-charts/pull/56) |
# Legacy network configuration
Source: https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/legacy-network-configuration
Reference for the deprecated global Conduktor Gateway networking variables: how to detect the mode and how it behaves.
Before [Gateway 3.20.0](/snippets/changelog/Gateway-3.20.0), Gateway was configured with global `GATEWAY_PORT_*`, `GATEWAY_ADVERTISED_HOST`, `GATEWAY_ROUTING_MECHANISM` and `GATEWAY_SECURITY_PROTOCOL` environment variables. This **legacy network configuration** is deprecated and will be removed in Gateway 3.23. New deployments should use [listener configuration](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/listeners).
This page is for deployments still running on legacy network configuration. It documents how the deprecated mode behaves. To switch to listener configuration, see [Migrate to listener configuration](/guide/tutorials/migrate-to-listener-configuration).
The legacy network configuration remains supported for now to give existing deployments time to migrate. Plan a migration; new functionality will not be backported to this mode.
## Detect which mode you're running
Gateway detects the configuration mode automatically at startup based on the environment variables you set:
| Mode | Trigger | Status |
| ---------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | ---------------------------------- |
| **Explicit** | Any `GATEWAY_LISTENER_*` env var is set | Recommended |
| **Legacy network** | Legacy env vars like `GATEWAY_PORT_START` are set | Deprecated, still supported |
| [**Zero-config**](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables#listeners-zero-config-mode) | No listener or network env vars are set | Local development and testing only |
From Gateway 3.20.0, legacy network configuration logs a deprecation warning at startup. Its presence in the logs confirms the mode is active:
```
[main] [WARN] [SecurityParametersInference] - Legacy environment variables detected (GATEWAY_PORT_START, etc.). This configuration method is deprecated and will be removed in version 3.23. Please migrate to the new listener configuration format (GATEWAY_LISTENER__*): https://docs.conduktor.io/guide/tutorials/migrate-to-listener-configuration
[main] [INFO] [SecurityParametersInference] - Legacy config: securityMode=GATEWAY_MANAGED, aclEnabled=false
```
Internally, Gateway converts the legacy variables into a single listener named `DEFAULT`, so the `Computed configuration` block lists a `DEFAULT` listener in both legacy and listener configuration. The deprecation warning is what distinguishes the two.
## How the legacy mode behaves
In legacy network configuration, Gateway infers missing security settings from the backing Kafka cluster.
### Security mode and protocol inference
How Gateway determines default values for `GATEWAY_SECURITY_PROTOCOL` and `GATEWAY_SECURITY_MODE` depends on which of the two are explicitly set.
**Both `GATEWAY_SECURITY_PROTOCOL` and `GATEWAY_SECURITY_MODE` unset:** Gateway infers both from the Kafka cluster's security protocol.
| Kafka cluster security protocol | Inferred `GATEWAY_SECURITY_PROTOCOL` | Inferred `GATEWAY_SECURITY_MODE` |
| ------------------------------- | ------------------------------------ | -------------------------------- |
| `SASL_PLAINTEXT` | `SASL_PLAINTEXT` | `KAFKA_MANAGED` |
| `SASL_SSL` | `SASL_SSL` | `KAFKA_MANAGED` |
| `PLAINTEXT` | `PLAINTEXT` | `GATEWAY_MANAGED` |
| `SSL` | `SSL` | `GATEWAY_MANAGED` |
If no security protocol is configured on the backing Kafka cluster, Gateway defaults to `PLAINTEXT` with `GATEWAY_MANAGED` mode.
**Only `GATEWAY_SECURITY_MODE` set:** Gateway infers the protocol from the combination.
| `GATEWAY_SECURITY_MODE` | Kafka cluster security protocol | Inferred `GATEWAY_SECURITY_PROTOCOL` |
| ----------------------- | ------------------------------- | --------------------------------------------- |
| `KAFKA_MANAGED` | `SASL_PLAINTEXT` | `SASL_PLAINTEXT` |
| `KAFKA_MANAGED` | `SASL_SSL` | `SASL_SSL` |
| `GATEWAY_MANAGED` | `PLAINTEXT` | `PLAINTEXT` |
| `GATEWAY_MANAGED` | `SSL` | `SSL` |
| *Any other combination* | | *No default; Gateway requires manual setting* |
**Only `GATEWAY_SECURITY_PROTOCOL` set:** Gateway infers the mode from the protocol.
| `GATEWAY_SECURITY_PROTOCOL` | Inferred `GATEWAY_SECURITY_MODE` |
| ----------------------------------------- | -------------------------------- |
| `SASL_PLAINTEXT` | `GATEWAY_MANAGED` |
| `SASL_SSL` | `GATEWAY_MANAGED` |
| `PLAINTEXT` | `GATEWAY_MANAGED` |
| `SSL` | `GATEWAY_MANAGED` |
| `DELEGATED_SASL_PLAINTEXT` *(deprecated)* | `KAFKA_MANAGED` |
| `DELEGATED_SASL_SSL` *(deprecated)* | `KAFKA_MANAGED` |
The `DELEGATED_SASL_PLAINTEXT` and `DELEGATED_SASL_SSL` values were deprecated in Gateway 3.10.0 in favor of `GATEWAY_SECURITY_MODE`. They remain supported but are no longer recommended. [See the security mode migration guide](/guide/tutorials/migrate-gateway-security).
### ACL default
`GATEWAY_ACL_ENABLED` is inferred from the security mode:
* `GATEWAY_MANAGED` → `true`
* `KAFKA_MANAGED` → `false`
You can override the inferred value by setting `GATEWAY_ACL_ENABLED` explicitly.
## Legacy environment variable reference
For reference, these are the environment variables that listener configuration replaces. Use them only on deployments that have not yet migrated.
| Environment variable | Description | Default |
| -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| **Common properties** | | |
| `GATEWAY_ADVERTISED_HOST` | The hostname returned in the Gateway's metadata for clients to connect to. | Your hostname |
| `GATEWAY_ROUTING_MECHANISM` | Defines the routing method: `port` for port routing, `host` for SNI routing. | `port` |
| `GATEWAY_PORT_START` | The first port the Gateway listens on. | `6969` |
| `GATEWAY_BIND_HOST` | The network interface the Gateway binds to. | `0.0.0.0` |
| **Port routing specific** | | |
| `GATEWAY_PORT_COUNT` | The total number of ports used by Gateway. | `(maxBrokerId - minBrokerId) + 3` |
| **SNI routing specific** | | |
| `GATEWAY_ADVERTISED_SNI_PORT` | The port returned in the Gateway's metadata when using SNI routing. | `GATEWAY_PORT_START` |
| `GATEWAY_ADVERTISED_HOST_PREFIX` | Configures the advertised broker names. | `broker` |
| `GATEWAY_SNI_HOST_SEPARATOR` | The separator used to construct returned metadata. | `-` |
| `GATEWAY_SECURITY_PROTOCOL` | The security protocol clients use to connect to Gateway. Has to be `SSL` or `SASL_SSL` when in `GATEWAY_MANAGED` security mode, or `SASL_SSL` when in `KAFKA_MANAGED` security mode, for SNI routing. | Inferred from `GATEWAY_SECURITY_MODE` and `KAFKA_SECURITY_PROTOCOL` |
| `GATEWAY_SSL_CLIENT_AUTH` | mTLS client authentication policy: `NONE`, `OPTIONAL` or `REQUIRE`. Replaced by the per-listener `GATEWAY_LISTENER__SSL_CLIENT_AUTH`. | `NONE` |
`GATEWAY_MIN_BROKERID` is unchanged and still applies to port routing under listener configuration. See [Gateway environment variables](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables#listeners).
## Related resources
* [Migrate to listener configuration](/guide/tutorials/migrate-to-listener-configuration) — variable mapping, a worked example, and behavior changes
* [Configure Gateway listeners](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/listeners)
* [Configure Gateway listener security](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/listener-security)
* [Define Gateway environment variables](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables)
* [Migrate security mode (v3.10–3.19)](/guide/tutorials/migrate-gateway-security) — migrate from the deprecated `DELEGATED_*` security protocols
# Configure Gateway listener security
Source: https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/listener-security
Pick the security protocol and authentication mechanism for a Conduktor Gateway listener. Choose between PLAINTEXT, SSL, SASL_PLAINTEXT or SASL_SSL, and decide whether authentication is managed by Gateway or by Kafka.
Per-listener configuration with `GATEWAY_LISTENER__*` variables is available from **Gateway 3.20.0**. On earlier versions, networking is configured with the global variables described in [Legacy network configuration](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/legacy-network-configuration). To move an existing deployment over, see [Migrate to listener configuration](/guide/tutorials/migrate-to-listener-configuration).
Once you've [configured a listener](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/listeners), the next step is to choose how clients authenticate against it. This page covers the security protocol and authentication options available for each listener.
All open connections in Gateway result in a `Principal` that represents the authenticated identity of the Kafka client.
Configuring listener security involves two key decisions:
1. **Security mode** (`GATEWAY_SECURITY_MODE`): defines where authentication and authorization take place — Gateway or Kafka. This applies globally to all listeners.
2. **Security protocol** (`GATEWAY_LISTENER__SECURITY_PROTOCOL`): defines how clients communicate and authenticate with a given listener. Set this per listener.
For local development and testing, you can skip listener configuration entirely. With no listener or networking variables set, Gateway runs in [zero-config mode](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables#listeners-zero-config-mode): it creates a default port-routing listener and infers the security mode and protocol from the backing Kafka cluster (with ACLs disabled). Don't use it in production.
The examples below configure a single listener named `DEFAULT`. To expose Gateway through multiple endpoints with different security, see [Set up multiple listeners](/guide/tutorials/multi-listener).
## Security mode
The `GATEWAY_SECURITY_MODE` environment variable determines where authentication and authorization take place. Gateway supports two modes:
### GATEWAY\_MANAGED
Gateway handles authentication and authorization. Use this mode when:
* You want Gateway to manage clients' credentials using local service accounts, or external ones from your identity provider (for OIDC) or certificates (for mTLS)
* You need Gateway to enforce ACLs
* You want the Gateway to be an abstraction layer that manages its own identities and ACLs
**Compatible security protocols:** `PLAINTEXT`, `SSL`, `SASL_PLAINTEXT`, `SASL_SSL`
**ACL behavior:** Set `GATEWAY_ACL_ENABLED` to control ACL enforcement on the passthrough virtual cluster — typically `true` for this mode. Use `GATEWAY_SUPER_USERS` to define super users.
Example configuration:
```yaml theme={null}
GATEWAY_SECURITY_MODE: GATEWAY_MANAGED
GATEWAY_ACL_ENABLED: true
GATEWAY_SUPER_USERS: alice;bob
GATEWAY_LISTENER_DEFAULT_SECURITY_PROTOCOL: SASL_SSL
GATEWAY_LISTENER_DEFAULT_ROUTING: sni
GATEWAY_LISTENER_DEFAULT_PORTS: 9092
```
### KAFKA\_MANAGED
Kafka handles authentication and authorization. Gateway forwards client credentials to the backing Kafka cluster. Use this mode when:
* Your backing Kafka cluster manages all credentials
* You want clients to use their existing Kafka credentials through Gateway
* You need Gateway to respect Kafka's ACLs and permissions
**Compatible security protocols:** `SASL_PLAINTEXT`, `SASL_SSL` only
`KAFKA_MANAGED` mode is incompatible with `PLAINTEXT` or `SSL` security protocols because these protocols don't support credential forwarding to Kafka.
**ACL behavior:** Gateway doesn't enforce ACLs on the passthrough virtual cluster in this mode — Kafka manages authorization, so `GATEWAY_ACL_ENABLED` has to be `false`. Setting it to `true` causes a startup error.
Example configuration:
```yaml theme={null}
GATEWAY_SECURITY_MODE: KAFKA_MANAGED
GATEWAY_ACL_ENABLED: false
GATEWAY_LISTENER_DEFAULT_SECURITY_PROTOCOL: SASL_SSL
GATEWAY_LISTENER_DEFAULT_ROUTING: sni
GATEWAY_LISTENER_DEFAULT_PORTS: 9092
```
## Security protocols
Gateway supports the following security protocols for client connections:
In the table below, "security protocol" refers to the per-listener `GATEWAY_LISTENER__SECURITY_PROTOCOL` setting.
| | ***Clients ⟶ GW transit in plaintext*** | ***Clients ⟶ GW transit is encrypted*** |
| --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ***Anonymous access only*** | Security mode: `GATEWAY_MANAGED` Security protocol: `PLAINTEXT` Authentication mechanism: `None` | Security mode: `GATEWAY_MANAGED` Security protocol: `SSL` Authentication mechanism: `None` |
| ***Credentials managed by Gateway (token)*** | Security mode: `GATEWAY_MANAGED` Security protocol: `SASL_PLAINTEXT` Authentication mechanism: `PLAIN` | Security mode: `GATEWAY_MANAGED` Security protocol: `SASL_SSL` Authentication mechanism: `PLAIN` |
| ***Credentials managed by Gateway (SCRAM)*** | Security mode: `GATEWAY_MANAGED` Security protocol: `SASL_PLAINTEXT` Authentication mechanism: `SCRAM-SHA-512` or `SCRAM-SHA-256` | Security mode: `GATEWAY_MANAGED` Security protocol: `SASL_SSL` Authentication mechanism: `SCRAM-SHA-512` or `SCRAM-SHA-256` |
| ***Gateway configured with OAuth*** | Security mode: `GATEWAY_MANAGED` Security protocol: `SASL_PLAINTEXT` Authentication mechanism: `OAUTHBEARER` | Security mode: `GATEWAY_MANAGED` Security protocol: `SASL_SSL` Authentication mechanism: `OAUTHBEARER` |
| ***Clients are identified by certificates (mTLS)*** | Not possible (mTLS requires encryption) | Security mode: `GATEWAY_MANAGED` Security protocol: `SSL` Authentication mechanism: `MTLS` |
| ***Credentials managed by Kafka*** | Security mode: `KAFKA_MANAGED` Security protocol: `SASL_PLAINTEXT` Authentication mechanism: `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`, `OAUTHBEARER` or `AWS_MSK_IAM` | Security mode: `KAFKA_MANAGED` Security protocol: `SASL_SSL` Authentication mechanism: `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`, `OAUTHBEARER` or `AWS_MSK_IAM` |
Gateway-managed SCRAM requires Gateway v3.21.0 or later, with the service accounts feature flag enabled (`GATEWAY_FEATURE_FLAGS_SERVICE_ACCOUNTS: true`).
## Configure security protocol
The `GATEWAY_LISTENER__SECURITY_PROTOCOL` environment variable defines how clients communicate with a given Gateway listener. In the examples below, the listener is named `DEFAULT`.
The global `GATEWAY_SECURITY_PROTOCOL` is deprecated in favor of the per-listener `GATEWAY_LISTENER__SECURITY_PROTOCOL`. If your deployment still sets it, see [Legacy network configuration](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/legacy-network-configuration).
Gateway supports all standard Kafka security protocols. The following sections show configuration examples for each protocol with both security modes where applicable.
### PLAINTEXT
There is no client authentication to Gateway and all communication is exchanged without any network security.
**Security mode:** `GATEWAY_MANAGED` only
Gateway configuration:
```yaml theme={null}
GATEWAY_SECURITY_MODE: GATEWAY_MANAGED
GATEWAY_ACL_ENABLED: true
GATEWAY_LISTENER_DEFAULT_SECURITY_PROTOCOL: PLAINTEXT
GATEWAY_LISTENER_DEFAULT_ROUTING: port
GATEWAY_LISTENER_DEFAULT_PORTS: 9092-9095
```
Client configuration:
```properties theme={null}
bootstrap.servers=your.gateway.hostname:9092
security.protocol=PLAINTEXT
```
### SSL
With SSL only, there is no client authentication, but communication between the client and Gateway broker will be encrypted.
**Security mode:** `GATEWAY_MANAGED` only
Gateway configuration:
```yaml theme={null}
GATEWAY_SECURITY_MODE: GATEWAY_MANAGED
GATEWAY_ACL_ENABLED: true
GATEWAY_LISTENER_DEFAULT_SECURITY_PROTOCOL: SSL
GATEWAY_LISTENER_DEFAULT_ROUTING: port
GATEWAY_LISTENER_DEFAULT_PORTS: 9092-9095
GATEWAY_SSL_KEY_STORE_PATH: /path/to/your/keystore.jks
GATEWAY_SSL_KEY_STORE_PASSWORD: yourKeystorePassword
GATEWAY_SSL_KEY_PASSWORD: yourKeyPassword
```
Client configuration:
```properties theme={null}
bootstrap.servers=your.gateway.hostname:9092
security.protocol=SSL
ssl.truststore.location=/path/to/your/truststore.jks
ssl.truststore.password=yourTruststorePassword
ssl.protocol=TLSv1.3
```
The truststore contains certificates from trusted Certificate Authorities (CAs) used to verify Gateway's TLS certificate, which is stored in the keystore. [Find out more about JKS truststores](https://docs.oracle.com/cd/E19509-01/820-3503/6nf1il6er/index.html).
### mTLS
Mutual TLS leverages client side certificates to authenticate a Kafka client.
`Principal` for an mTLS connection can be detected from the subject of the certificate using the same feature as Apache Kafka, the [SSL principal mapping](https://docs.confluent.io/platform/current/kafka/configure-mds/mutual-tls-auth-rbac.html#principal-mapping-rules-for-tls-ssl-listeners-extract-a-principal-from-a-certificate).
**Security mode:** `GATEWAY_MANAGED` only
Gateway configuration:
```yaml theme={null}
GATEWAY_SECURITY_MODE: GATEWAY_MANAGED
GATEWAY_ACL_ENABLED: true
GATEWAY_LISTENER_DEFAULT_SECURITY_PROTOCOL: SSL
GATEWAY_LISTENER_DEFAULT_ROUTING: port
GATEWAY_LISTENER_DEFAULT_PORTS: 9092-9095
GATEWAY_LISTENER_DEFAULT_SSL_CLIENT_AUTH: REQUIRE
GATEWAY_SSL_KEY_STORE_PATH: /path/to/your/keystore.jks
GATEWAY_SSL_KEY_STORE_PASSWORD: yourKeystorePassword
GATEWAY_SSL_KEY_PASSWORD: yourKeyPassword
GATEWAY_SSL_TRUST_STORE_PATH: /path/to/your/truststore.jks
GATEWAY_SSL_TRUST_STORE_PASSWORD: yourTrustStorePassword
```
Client configuration:
```properties theme={null}
bootstrap.servers=your.gateway.hostname:9093
security.protocol=SSL
ssl.keystore.type=PEM
ssl.keystore.key=/path/to/your/client.key
ssl.keystore.certificate.chain=/path/to/your/client.crt
ssl.truststore.type=PEM
ssl.truststore.certificates=/path/to/your/ca.crt
ssl.protocol=TLSv1.3
ssl.client.auth=required
```
The server CA certificate here is provided as a PEM file as well as the client's certificates (*ssl.keystore.xx* keys). Jks could also be used for both client and server side authentication.
### SASL\_PLAINTEXT
Authentication from the client is mandatory but all communications are exchanged without any network security.
**Security mode:** `GATEWAY_MANAGED` or `KAFKA_MANAGED`
You don't set `GATEWAY_SASL_MECHANISM` — this environment variable **does not exist**. Gateway automatically detects the authentication mechanism based on how the client presents itself. For example, if a client uses `OAUTHBEARER`, Gateway uses the OAuth configuration. If a client uses `PLAIN`, Gateway validates credentials based on the security mode.
#### Gateway-managed (PLAIN mechanism)
Plain mechanism uses username/password credentials to authenticate against Gateway. Credentials take the form of a JWT token managed in Gateway using the Admin (HTTP) API.
Gateway configuration:
```yaml theme={null}
GATEWAY_SECURITY_MODE: GATEWAY_MANAGED
GATEWAY_ACL_ENABLED: true
GATEWAY_LISTENER_DEFAULT_SECURITY_PROTOCOL: SASL_PLAINTEXT
GATEWAY_LISTENER_DEFAULT_ROUTING: port
GATEWAY_LISTENER_DEFAULT_PORTS: 9092-9095
GATEWAY_USER_POOL_SECRET_KEY: yourRandom256bitKeyUsedToSignTokens
```
The `GATEWAY_USER_POOL_SECRET_KEY` **has to be** set to a random base64-encoded value of 256 bits to ensure that tokens aren't forged. For example: `openssl rand -base64 32`. This value is required — Gateway has no default.
Client configuration:
```properties theme={null}
bootstrap.servers=your.gateway.hostname:9092
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
username="yourUsername" \
password="yourToken";
```
*Note on the password :*
It has to be a token that's obtained by a Gateway admin via the Admin (HTTP) API, as follows:
1. Create the service account, the username
Request:
```bash theme={null}
curl \
--request PUT \
--url 'http://localhost:8888/gateway/v2/service-account' \
--user admin:conduktor \
--header 'Content-Type: application/json' \
--data-raw '{
"kind" : "GatewayServiceAccount",
"apiVersion" : "gateway/v2",
"metadata" : {
"name" : "jdoe",
"vCluster" : "passthrough"
},
"spec" : { "type" : "LOCAL" }'
```
Response:
```json theme={null}
{
"resource" : {
"kind" : "GatewayServiceAccount",
"apiVersion" : "gateway/v2",
"metadata" : {
"name" : "jdoe",
"vCluster" : "passthrough"
},
"spec" : {
"type" : "LOCAL"
}
},
"upsertResult" : "CREATED"
}
```
1. Generate a token for the service account, the password
Request:
```bash theme={null}
curl \
--silent \
--request POST \
--url 'http://localhost:8888/gateway/v2/token' \
--header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
--header 'Content-Type: application/json' \
--data-raw '{
"username": "jdoe",
"vCluster": "passthrough",
"lifeTimeSeconds": 3600000
}'
```
```json theme={null}
{"token":"eyJhbGciOiJIUzI1NiJ9.eyJ1c2VybmFtZSI6Impkb2UiLCJ2Y2x1c3RlciI6InBhc3N0aHJvdWdoIiwiZXhwIjoxNzQ1MzY1OTcxfQ.zPPiD17MiRnXyHJw07Cx4SKPySDi_ErJrXmi5BycR04"}
```
The token conforms to the JWT token specification.
The JWT payload contains the username, the vCluster and the expiration date:
```bash theme={null}
jwt decode eyJhbGciOiJIUzI1NiJ9.eyJ1c2VybmFtZSI6Impkb2UiLCJ2Y2x1c3RlciI6InBhc3N0aHJvdWdoIiwiZXhwIjoxNzQ1MzY1OTcxfQ.zPPiD17MiRnXyHJw07Cx4SKPySDi_ErJrXmi5BycR04
Token claims
------------
{
"exp": 1745365971,
"username": "jdoe",
"vcluster": "passthrough"
}
```
#### Gateway-managed (SCRAM mechanisms)
Salted Challenge Response Authentication Mechanism (SCRAM) uses username/password credentials that you set yourself, rather than a generated token. Gateway derives a verifier from the password you supply and stores only that verifier, never the clear password. SCRAM credentials don't expire, which suits long-running clients.
Set `GATEWAY_FEATURE_FLAGS_SERVICE_ACCOUNTS` to `true` to enable them. Gateway-managed SASL listeners then advertise `SCRAM-SHA-512` and `SCRAM-SHA-256` alongside `PLAIN`, and you can add, list and revoke credentials for a local service account through the Admin (HTTP) API.
Gateway configuration:
```yaml theme={null}
GATEWAY_SECURITY_MODE: GATEWAY_MANAGED
GATEWAY_ACL_ENABLED: true
GATEWAY_LISTENER_DEFAULT_SECURITY_PROTOCOL: SASL_PLAINTEXT
GATEWAY_LISTENER_DEFAULT_ROUTING: port
GATEWAY_LISTENER_DEFAULT_PORTS: 9092-9095
GATEWAY_USER_POOL_SECRET_KEY: yourRandom256bitKeyUsedToSignTokens
GATEWAY_FEATURE_FLAGS_SERVICE_ACCOUNTS: true
```
Client configuration:
```properties theme={null}
bootstrap.servers=your.gateway.hostname:9092
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
username="yourLogin" \
password="yourPassword";
```
Use `SCRAM-SHA-256` as the `sasl.mechanism` if you created the credential with the `SHA-256` algorithm.
[Find out how to add SCRAM credentials to a service account](/guide/tutorials/manage-gateway-service-accounts#authenticate-with-scram-credentials).
#### Gateway-managed (OAUTHBEARER mechanism)
OAuthbearer uses an OAuth2/OIDC security provider to authenticate a token in Gateway. The OAuth credentials base is managed in the configured provider.
This mechanism also allows you to verify claims from your OIDC provider (`audience` and `issuer`).
Gateway configuration:
```yaml theme={null}
GATEWAY_SECURITY_MODE: GATEWAY_MANAGED
GATEWAY_ACL_ENABLED: true
GATEWAY_LISTENER_DEFAULT_SECURITY_PROTOCOL: SASL_PLAINTEXT
GATEWAY_LISTENER_DEFAULT_ROUTING: port
GATEWAY_LISTENER_DEFAULT_PORTS: 9092-9095
GATEWAY_OAUTH_JWKS_URL: https://login.microsoftonline.com/common/discovery/keys
GATEWAY_OAUTH_EXPECTED_ISSUER: https://sts.windows.net/xxxxxxxx-df00-48cd-805b-1ebe914e8b11/
GATEWAY_OAUTH_EXPECTED_AUDIENCES: "[00000002-0000-0000-c000-000000000000]"
```
Client configuration:
```properties theme={null}
bootstrap.servers=your.gateway.hostname:9092
security.protocol=SASL_PLAINTEXT
sasl.mechanism=OAUTHBEARER
sasl.login.callback.handler.class=org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler
sasl.oauthbearer.token.endpoint.url=https://login.microsoftonline.com/xxxxxxxx-df00-48cd-805b-1ebe914e8b11/oauth2/token
sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required \
clientId="yourClientID" \
clientSecret="yourClientSecret" \
scope=".default";
```
#### Kafka-managed
Gateway forwards client credentials to the backing Kafka cluster for authentication. Credentials are managed by your backing Kafka cluster.
Supported SASL mechanisms: `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`, `OAUTHBEARER`, `AWS_MSK_IAM`
Gateway configuration:
```yaml theme={null}
GATEWAY_SECURITY_MODE: KAFKA_MANAGED
GATEWAY_ACL_ENABLED: false
GATEWAY_LISTENER_DEFAULT_SECURITY_PROTOCOL: SASL_PLAINTEXT
GATEWAY_LISTENER_DEFAULT_ROUTING: port
GATEWAY_LISTENER_DEFAULT_PORTS: 9092-9095
```
Client configuration (using PLAIN mechanism as an example):
```properties theme={null}
bootstrap.servers=your.gateway.hostname:9092
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="yourKafkaUser" password="yourKafkaPassword";
```
### SASL\_SSL
Authentication from client is mandatory and communication will be encrypted using TLS.
**Security mode:** `GATEWAY_MANAGED` or `KAFKA_MANAGED`
You don't set `GATEWAY_SASL_MECHANISM` — this environment variable **does not exist**. Gateway automatically detects the authentication mechanism based on how the client presents itself. For example, if a client uses `OAUTHBEARER`, Gateway uses the OAuth configuration. If a client uses `PLAIN`, Gateway validates credentials based on the security mode.
#### Gateway-managed (PLAIN mechanism)
Plain mechanism uses username/password credentials to authenticate against Gateway. Credentials are managed in Gateway using the HTTP API.
Gateway configuration:
```yaml theme={null}
GATEWAY_SECURITY_MODE: GATEWAY_MANAGED
GATEWAY_ACL_ENABLED: true
GATEWAY_LISTENER_DEFAULT_SECURITY_PROTOCOL: SASL_SSL
GATEWAY_LISTENER_DEFAULT_ROUTING: port
GATEWAY_LISTENER_DEFAULT_PORTS: 9092-9095
GATEWAY_USER_POOL_SECRET_KEY: yourRandom256bitKeyUsedToSignTokens
GATEWAY_SSL_KEY_STORE_PATH: /path/to/your/keystore.jks
GATEWAY_SSL_KEY_STORE_PASSWORD: yourKeystorePassword
GATEWAY_SSL_KEY_PASSWORD: yourKeyPassword
```
You have to set `GATEWAY_USER_POOL_SECRET_KEY` to a random value to ensure that tokens cannot be forged. This value is required — Gateway has no default.
Client configuration:
```properties theme={null}
bootstrap.servers=your.gateway.hostname:9093
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
ssl.truststore.location=/path/to/your/truststore.jks
ssl.truststore.password=yourTruststorePassword
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
username="yourUsername" \
password="yourToken";
```
See the above section for requirements on how to create tokens using the Admin (HTTP) API.
#### Gateway-managed (SCRAM mechanisms)
Salted Challenge Response Authentication Mechanism (SCRAM) uses username/password credentials that you set yourself, rather than a generated token. Gateway derives a verifier from the password you supply and stores only that verifier, never the clear password. SCRAM credentials don't expire, which suits long-running clients.
Set `GATEWAY_FEATURE_FLAGS_SERVICE_ACCOUNTS` to `true` to enable them. Gateway-managed SASL listeners then advertise `SCRAM-SHA-512` and `SCRAM-SHA-256` alongside `PLAIN`, and you can add, list and revoke credentials for a local service account through the Admin (HTTP) API.
Gateway configuration:
```yaml theme={null}
GATEWAY_SECURITY_MODE: GATEWAY_MANAGED
GATEWAY_ACL_ENABLED: true
GATEWAY_LISTENER_DEFAULT_SECURITY_PROTOCOL: SASL_SSL
GATEWAY_LISTENER_DEFAULT_ROUTING: port
GATEWAY_LISTENER_DEFAULT_PORTS: 9092-9095
GATEWAY_USER_POOL_SECRET_KEY: yourRandom256bitKeyUsedToSignTokens
GATEWAY_FEATURE_FLAGS_SERVICE_ACCOUNTS: true
GATEWAY_SSL_KEY_STORE_PATH: /path/to/your/keystore.jks
GATEWAY_SSL_KEY_STORE_PASSWORD: yourKeystorePassword
GATEWAY_SSL_KEY_PASSWORD: yourKeyPassword
```
Client configuration:
```properties theme={null}
bootstrap.servers=your.gateway.hostname:9093
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
ssl.truststore.location=/path/to/your/truststore.jks
ssl.truststore.password=yourTruststorePassword
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
username="yourLogin" \
password="yourPassword";
```
Use `SCRAM-SHA-256` as the `sasl.mechanism` if you created the credential with the `SHA-256` algorithm.
[Find out how to add SCRAM credentials to a service account](/guide/tutorials/manage-gateway-service-accounts#authenticate-with-scram-credentials).
#### Gateway-managed (OAUTHBEARER mechanism)
OAuthbearer uses an OAuth2/OIDC security provider to authenticate a token in Gateway. The OAuth credentials base is managed in the configured provider.
This mechanism also allows you to verify claims from your OIDC provider (`audience` and `issuer`).
Gateway configuration:
```yaml theme={null}
GATEWAY_SECURITY_MODE: GATEWAY_MANAGED
GATEWAY_ACL_ENABLED: true
GATEWAY_LISTENER_DEFAULT_SECURITY_PROTOCOL: SASL_SSL
GATEWAY_LISTENER_DEFAULT_ROUTING: port
GATEWAY_LISTENER_DEFAULT_PORTS: 9092-9095
GATEWAY_OAUTH_JWKS_URL: https://login.microsoftonline.com/common/discovery/keys
GATEWAY_OAUTH_EXPECTED_ISSUER: https://sts.windows.net/xxxxxxxx-df00-48cd-805b-1ebe914e8b11/
GATEWAY_OAUTH_EXPECTED_AUDIENCES: "[00000002-0000-0000-c000-000000000000]"
GATEWAY_SSL_KEY_STORE_PATH: /path/to/your/keystore.jks
GATEWAY_SSL_KEY_STORE_PASSWORD: yourKeystorePassword
GATEWAY_SSL_KEY_PASSWORD: yourKeyPassword
```
Client configuration:
```properties theme={null}
bootstrap.servers=your.gateway.hostname:9092
security.protocol=SASL_SSL
sasl.mechanism=OAUTHBEARER
ssl.truststore.location=/path/to/your/truststore.jks
ssl.truststore.password=yourTruststorePassword
sasl.login.callback.handler.class=org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler
sasl.oauthbearer.token.endpoint.url=https://login.microsoftonline.com/xxxxxxxx-df00-48cd-805b-1ebe914e8b11/oauth2/token
sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required \
clientId="yourClientID" \
clientSecret="yourClientSecret" \
scope=".default";
```
#### Kafka-managed
Gateway forwards client credentials to the backing Kafka cluster for authentication. Credentials are managed by your backing Kafka cluster.
Supported SASL mechanisms: `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512`, `OAUTHBEARER`, `AWS_MSK_IAM`
Gateway configuration:
```yaml theme={null}
GATEWAY_SECURITY_MODE: KAFKA_MANAGED
GATEWAY_ACL_ENABLED: false
GATEWAY_LISTENER_DEFAULT_SECURITY_PROTOCOL: SASL_SSL
GATEWAY_LISTENER_DEFAULT_ROUTING: port
GATEWAY_LISTENER_DEFAULT_PORTS: 9092-9095
GATEWAY_SSL_KEY_STORE_PATH: /path/to/your/keystore.jks
GATEWAY_SSL_KEY_STORE_PASSWORD: yourKeystorePassword
GATEWAY_SSL_KEY_PASSWORD: yourKeyPassword
```
Client configuration (using PLAIN mechanism as an example):
```properties theme={null}
bootstrap.servers=your.gateway.hostname:9092
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
ssl.truststore.location=/path/to/your/truststore.jks
ssl.truststore.password=yourTruststorePassword
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="yourKafkaUser" password="yourKafkaPassword";
```
## Principal resolver
When using Confluent Cloud authentication with Kafka-managed mode, Gateway supports automatically resolving API keys to their associated service account.
This feature enhances security and improves usability by working with the service account principals instead of raw API keys.
[See the principal resolver environment variables](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables#principal-resolver).
Gateway configuration using environment variables:
```yaml theme={null}
GATEWAY_PRINCIPAL_RESOLVER: CONFLUENT_CLOUD
GATEWAY_CONFLUENT_CLOUD_API_KEY: your-api-key
GATEWAY_CONFLUENT_CLOUD_API_SECRET: your-api-secret
GATEWAY_CONFLUENT_CLOUD_CACHE_SIZE: 1000 # default
GATEWAY_CONFLUENT_CLOUD_CACHE_EXPIRY_MS: 86400000 # 1 day default
```
Gateway configuration using a configuration file
```yaml theme={null}
authenticationConfig:
principalResolver: CONFLUENT_CLOUD
confluentCloud:
apiKey: ${GATEWAY_CONFLUENT_CLOUD_API_KEY}
apiSecret: ${GATEWAY_CONFLUENT_CLOUD_API_SECRET}
cacheConfig:
maxSize: ${GATEWAY_CONFLUENT_CLOUD_CACHE_SIZE|1000}
ttlMs: ${GATEWAY_CONFLUENT_CLOUD_CACHE_EXPIRY_MS|86400000} # 1 day
```
The value of `GATEWAY_CONFLUENT_CLOUD_API_KEY` has to be a Confluent Cloud API key because it will be used to access the Confluent Cloud API. [Find out more about Confluent Cloud API keys](https://docs.confluent.io/cloud/current/api.html#section/Authentication/API-keys).
When the principal resolver is configured, Gateway will automatically resolve Kafka cluster API keys (like `XIGMNERQXOUKXDQU`) to their associated owner, which will be either the user (e.g. `u-12345`) or the service account (e.g. `sa-72839j`).
### Authentication flow
```mermaid theme={null}
sequenceDiagram
autonumber
participant A as Application
box "Gateway cluster"
participant GW as Gateway
participant GPR as Gateway Principal Resolver
end
participant K as Backing Kafka Cluster
participant CC as Confluent Cloud
Note over A,GW: Application uses credentials `apiKey:apiSecret`
A ->> GW: Connect to Gateway using application credentials
GW ->> K : Gateway forward authentication to Kafka
alt Authentication failed
K -->> GW: AUTH_FAILED
GW -->> A : AUTH_FAILED
else Authentication successful (principalToken)
K -->> GW: AUTH_OK(principalToken)
GW ->> GW: extractPrincipal(principalToken)
GW ->> GPR: isConfluentCloudResolverEnabled()
alt Opt‑in=true
GW ->> CC: GET /iam/v2/api-keys/{apiKeyId}
CC -->> GW: { service_account: "sa-72839j", ... }
GW ->> GW: cacheMapping(apiKeyId -> serviceAccount)
Note over GW,K: Principal resolved to Service Account sa‑72839j
else Opt‑in = false
Note over GW,K: Principal remains apiKeyId
end
GW -->> A : CONNECTED
end
```
## Re-authentication support
We support Apache Kafka re-authentication as Kafka brokers. [Find out more about KIP-368](https://cwiki.apache.org/confluence/display/KAFKA/KIP-368%3A+Allow+SASL+Connections+to+Periodically+Re-Authenticate).
## Related resources
* [Manage service accounts](/guide/tutorials/manage-gateway-service-accounts)
* [Connect Gateway to Kafka](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/connect-to-kafka)
* [Define Gateway environment variables](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables)
# Configure Gateway listeners
Source: https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/listeners
Configure Conduktor Gateway listeners — the network endpoints clients use to reach Gateway. Choose port or SNI routing, set bindings, and link to listener security and load balancing.
Per-listener configuration with `GATEWAY_LISTENER__*` variables is available from **Gateway 3.20.0**. On earlier versions, networking is configured with the global variables described in [Legacy network configuration](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/legacy-network-configuration). To move an existing deployment over, see [Migrate to listener configuration](/guide/tutorials/migrate-to-listener-configuration).
Kafka clients connect to Conduktor Gateway through **listeners**. A listener is a network endpoint with its own [routing strategy](#choose-a-routing-strategy), [network bindings](#set-up-a-listener-with-port-routing) and [security protocol](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/listener-security). A simple deployment uses a single listener; containerized environments often expose two (internal and external).
## Choose a routing strategy
A listener routes clients to brokers in one of two ways:
Choose [port routing](#set-up-a-listener-with-port-routing) if your environment:
* doesn't require TLS encryption
* has flexible network port management capabilities
* prefers a simpler, straightforward configuration without DNS complexities
Choose [SNI routing](#set-up-a-listener-with-sni-routing) if your environment:
* requires TLS-encrypted connections for secure communication
* faces challenges managing multiple network ports
* seeks a scalable solution with easier routing management through DNS and host names
For local development and testing, you can skip listener configuration entirely. With no listener or networking variables set, Gateway runs in [zero-config mode](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables#listeners-zero-config-mode): it creates a default port-routing listener and infers the security mode and protocol from the backing Kafka cluster (with ACLs disabled). Don't use it in production.
## Define a listener
Define a listener using environment variables with the pattern `GATEWAY_LISTENER__`, where `` is a label you choose (for example, `DEFAULT`, `EXTERNAL`, `INTERNAL`).
Listener names have to start with a letter and contain only alphanumeric characters. Underscores are reserved as delimiters in `GATEWAY_LISTENER__`, so a name like `MY_LISTENER` would be ambiguous. Gateway only recognizes variables whose `` matches this pattern — variables using any other name are ignored, and the listener is silently not created.
## Set up a listener with port routing
With port routing, Gateway assigns one port per Kafka broker. Clients connect to the appropriate port to reach the broker they need. The recommended number of ports in production is **double the number of brokers** to cover cluster growth.
An example single-listener configuration:
```bash theme={null}
GATEWAY_LISTENER_DEFAULT_ROUTING=port
GATEWAY_LISTENER_DEFAULT_PORTS=9092-9097
GATEWAY_LISTENER_DEFAULT_ADVERTISED_HOST=gateway.example.com
GATEWAY_MIN_BROKERID=1
```
Gateway maps ports to broker IDs in order from `GATEWAY_MIN_BROKERID`. Set it to the lowest broker ID in your Kafka cluster and provide enough ports for every broker. Above, brokers 1, 2, 3 map to ports 9092, 9093, 9094, with 9095-9097 as headroom for cluster growth.
We recommend SNI routing when broker IDs are not sequential and stable. For example, a three-broker cluster with IDs 100, 200, 300 and `GATEWAY_MIN_BROKERID=100` would need a port range covering 100 through 300, and would fail if broker ID 400 were introduced.
For the full list of listener environment variables, see the [environment variables reference](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables#listeners).
## Set up a listener with SNI routing
With SNI routing, Gateway listens on a single port and routes traffic based on the hostname in the TLS handshake. SNI routing requires a TLS-based security protocol (`SSL` or `SASL_SSL`) and proper DNS and TLS certificates for each broker hostname.
[Follow the SNI routing tutorial](/guide/tutorials/sni-routing) for the step-by-step setup, including certificate preparation and DNS configuration.
## Advertised host and load balancing
A listener's advertised host is what Gateway returns to clients in metadata responses. When Gateway sits behind a load balancer, the advertised host has to point at the load balancer rather than at the Gateway pod, otherwise clients will try to bypass it.
[Configure Gateway load balancing](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/load-balancing) before finalizing the listener's advertised host.
## Related resources
* [Configure listener security](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/listener-security) — pick the security protocol and authentication mechanism for the listener
* [Set up multiple listeners](/guide/tutorials/multi-listener) — expose Gateway through more than one endpoint
* [Gateway environment variables](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables#listeners)
* [Set up Gateway for multi-clusters](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/multi-clusters)
* [Legacy network configuration](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/legacy-network-configuration) — reference for the deprecated global variables
* [Migrate to listener configuration](/guide/tutorials/migrate-to-listener-configuration) — move an existing deployment off the deprecated global variables
# Configure Gateway load balancing
Source: https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/load-balancing
Configure load balancing for Conduktor Gateway nodes. Choose between internal Gateway load balancing or an external load balancer for Kafka client connections.
To map the different Gateway nodes sharing the same cluster and your Kafka brokers, you can either use:
* Gateway **internal load balancing** or
* an **external load balancing**
These options are only for the Kafka protocol, as any of the Gateway nodes can manage HTTP calls to their API.
## Internal load balancing
Gateway's ability to distribute the client connections between the different Gateway nodes in the same cluster is what we refer to as internal load balancing. This is done automatically by Gateway and is the default behavior.
To deploy multiple Gateway nodes as part of the same Gateway cluster, you have to set the same `GATEWAY_CLUSTER_ID` in each node's deployment configuration. This configuration ensures that all nodes join the same consumer group, enabling them to consume an internal topic from your Kafka cluster. This is how the nodes recognize each other as members of the same Gateway cluster.
When a client connects to one of the Gateway nodes to request metadata, the following process occurs (assuming `GATEWAY_FEATURE_FLAGS_INTERNAL_LOAD_BALANCING` is set to `true`, which is the default setting):
1. The client chooses one of the bootstrap servers to ask for metadata.
2. The Gateway node generates a mapping between its cluster nodes and the Kafka brokers.
3. The Gateway node returns this mapping to the client.
4. With the mapping in hand, the client can efficiently route its requests. For instance, if the client needs to produce to a partition where broker 3 is the leader, it knows to forward the request to Gateway 2 on port 9094.
If multiple Gateway cluster nodes are mapped to one Kafka broker, the nodes are selected in a round-robin fashion for each metadata request. This is to ensure the load is balanced among Gateway cluster nodes.
For example, you have a Gateway cluster composed of two Gateway nodes, connected to a Kafka cluster with three brokers. The client's metadata discovery process might look like this:
This mapping will be made again for every client asking for metadata, and will be made again as soon as a Gateway node is added or removed from the Gateway cluster.
If you have specified a `GATEWAY_RACK_ID`, then the mapping will take this into consideration and a Gateway node in the same rack as the Kafka broker will be assigned.
Here's the same example but with multiple clients:
The process is repeated but will likely result in a different mapping compared to that of Client 1 due to the round-robin node assignment of Gateway nodes for the client.
### Internal load balancing limitations
In a Kubernetes environment, your ingress must point at a single service, which could be an external load balancer as detailed below.
## External load balancing
Alternatively, you can disable the internal load balancing by setting `GATEWAY_FEATURE_FLAGS_INTERNAL_LOAD_BALANCING: false`.
Here's an example where:
1. All client requests are directed to the external load balancer which acts as the entry point to your Gateway cluster.
2. The load balancer forwards each request to one of the Gateway nodes, regardless of the port.
3. The selected Gateway node, which knows which broker is the leader of each partition, forwards the request to the appropriate Kafka broker.
When using an external load balancer, configure each listener's advertised host (`GATEWAY_LISTENER__ADVERTISED_HOST` or `GATEWAY_LISTENER__ADVERTISED_HOST_PATTERN` for SNI routing) to the load balancer's hostname. If this isn't done, applications will attempt to connect directly to Gateway, bypassing the load balancer.
The advertised host is set per listener from Gateway 3.20.0. On earlier versions, the global `GATEWAY_ADVERTISED_HOST` serves the same purpose — see [Legacy network configuration](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/legacy-network-configuration).
### Capturing the client IP address
When running Conduktor Gateway behind a load balancer, the load balancer replaces the client IP address with its own, therefore audit log and error messages do not report the actual client IP address. This makes it difficult to identify the actual client IP address if there is an issue, for example, when a client fails during SSL handshake.
Conduktor Gateway supports two ways to capture the original client IP address, depending on how your load balancer is set up:
* You configure the load balancer yourself (HA Proxy, Nginx, an AWS or Azure load balancer, etc.) — use [HAProxy Protocol](#using-haproxy-protocol).
* Kubernetes provisions the load balancer for you as a `LoadBalancer`-type Service — use [Kubernetes `externalTrafficPolicy`](#using-kubernetes-externaltrafficpolicy).
#### Using HAProxy Protocol
In order to capture the original client IP address behind an external load balancer, Conduktor Gateway supports [HAProxy Protocol](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt).
To enable HAProxy Protocol support, set [`GATEWAY_FEATURE_FLAGS_HAPROXY_PROTOCOL`](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables#feature-flags) to `true`. At the same time, you must also configure the external load balancer to send HAProxy Protocol header to Conduktor Gateway.
If HAProxy Protocol is enabled, Conduktor Gateway reads the client IP address from the HAProxy Protocol header and reports it as `connection.proxyConnection` on every audit event type that carries a connection block, alongside the load balancer's address. See the [audit event connection block](/guide/conduktor-in-production/admin/audit-logs#connection-block) for the field details.
Connection-path log messages, such as a TLS handshake failure, also include the original client address next to the load balancer's. For example:
```
[WARN ] [ProxyChannelInitializer:109] - TLS handshake failed [fecd5b8b]: SslHandshakeCompletionEvent(java.nio.channels.ClosedChannelException) on [socket_peer=/198.51.100.7:55512 proxied_client=/172.16.160.44:59321]
```
Rejections that happen before Gateway reads any bytes from the connection (such as connections rejection due to connection limit or shutdown) only show the load balancer's address (`socket_peer`), even with HAProxy Protocol enabled because the PROXY header has not arrived yet at that point.
Enabling HAProxy Protocol also makes the [`{{userIp}}` header injection variable](/guide/reference/data-security#context-variables) resolve to the real client address instead of the load balancer's.
For reference, this is the external documentation on enabling HAProxy Protocol header:
* [HAProxy](https://www.haproxy.com/documentation/haproxy-configuration-tutorials/proxying-essentials/client-ip-preservation/enable-proxy-protocol/)
* [Nginx](https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html#proxy_protocol)
* [AWS load balancer](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/enable-proxy-protocol.html)
* [Azure private link](https://learn.microsoft.com/en-us/azure/private-link/private-link-service-overview#getting-connection-information-using-tcp-proxy-v2)
#### Using Kubernetes `externalTrafficPolicy`
By default, the Kubernetes load balancer changes the client IP address to its own.
If you need Gateway to log the actual client IP address, add [`externalTrafficPolicy`](https://kubernetes.io/docs/reference/networking/virtual-ips/#traffic-policies) to the Gateway Helm chart:
```yaml theme={null}
service:
external:
extraSpecs:
externalTrafficPolicy: Local
```
This is needed if you need to identify a client that failed to connect during handshake. For example, in this log message:
```
[WARN ] [ProxyChannelInitializer:109] - TLS handshake failed [fecd5b8b]: SslHandshakeCompletionEvent(java.nio.channels.ClosedChannelException) on [socket_peer=/172.16.160.44:59321]
```
`externalTrafficPolicy` preserves the client's address at the socket level as the connection's source IP address. It doesn't send a PROXY protocol header, so it doesn't populate `connection.proxyConnection` on audit events. The client-IP-based policies use the source IP address as the client IP.
### External load balancing limitations
This requires you to handle load balancing manually, as you won't have the advantage of the automatic load balancing offered by Gateway's internal load balancing feature.
No. Gateway is not a Single Point of Failure (SPOF).
It works the same way that Kafka does: deploy many instances, spread the load, restart them - clients continue without impact.
**Stateless**
Gateway doesn't store local data, so a restart never affects traffic. The backing Kafka cluster stores all the states.
**Kafka-aware**
Clients use their normal retry and metadata refresh flows (same as with brokers). When a Gateway instance becomes unavailable, clients re-connect to another instance using standard Kafka client behavior.
**Scale horizontally**
Run several pods to handle load and keep latency steady. You can add or remove instances based on throughput needs as [Gateway scales horizontally](/guide/conduktor-in-production/system-requirements#scaling-gateway).
**Safe restarts**
Roll restart any pod and clients re-connect on their own. Standard Kafka client retry mechanisms handle temporary unavailability, which makes Gateway restarts transparent to applications.
For production deployments, we recommend at least three Gateway instances to ensure an effective load distribution handling and availability during restarts.
## Related resources
* [Gateway system requirements](/guide/conduktor-in-production/system-requirements#scaling-gateway)
* [Deploy Gateway on Kubernetes](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/kubernetes)
* [Define Gateway environment variables](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables)
# Conduktor Gateway multi-cluster setup
Source: https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/multi-clusters
Configure Conduktor Gateway to proxy multiple Kafka clusters from a single endpoint. Route clients to topics across clusters using topic aliases and access controls.
Gateway can be configured to communicate with multiple Kafka clusters and expose their topics to your partners. You can:
* direct partners to a **single endpoint**,
* provide them with **access to topics in multiple Kafka clusters** and
* expose topics using **aliases** that can be different from actual topic names.
To set up Gateway to support multi-clusters, you should:
1. Configure **one** main cluster which will be used by Gateway to store its internal state.
2. Set up any number of upstream physical Kafka clusters that you want to expose through Gateway.
If you're using partner virtual clusters to share data with external third parties, be aware that cluster IDs (e.g., `clusterA`, `clusterB`) may appear in the bootstrap server address or client logs.
To prevent unintended exposure, **avoid using sensitive names/information in cluster IDs**.
## Choose a routing strategy
For production deployments with multiple clusters, use [SNI routing](/guide/tutorials/sni-routing) on your listeners. With port routing, Gateway assigns a fixed port to every broker across all clusters, so adding or removing a cluster shifts those port assignments and can break existing client connections. SNI routing instead distinguishes brokers by hostname, so adding or removing a cluster leaves other clients unaffected.
Port routing remains a simpler choice for development and test environments — or anywhere you can update client bootstrap servers and restart clients when the set of clusters changes. [Configure listeners](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/listeners).
## Configure the clusters
Define your main and upstream clusters using either a configuration file or environment variables.
Specify your main and upstream cluster configurations, along with a `gateway.roles` entry to mark the upstream clusters.
```yaml title="cluster-config.yaml" theme={null}
config:
main:
bootstrap.servers: ':9092'
security.protocol: 'SASL_SSL'
sasl.mechanism: 'PLAIN'
sasl.jaas.config: 'org.apache.kafka.common.security.plain.PlainLoginModule required username="" password="";'
clusterA:
bootstrap.servers: ':9092'
security.protocol: 'SASL_SSL'
sasl.mechanism: 'PLAIN'
sasl.jaas.config: 'org.apache.kafka.common.security.plain.PlainLoginModule required username="" password="";'
gateway.roles: 'upstream' # Note: may be omitted as upstream is the default (used to differentiate from failover clusters)
clusterB:
bootstrap.servers: ':9092'
security.protocol: 'SASL_SSL'
sasl.mechanism: 'PLAIN'
sasl.jaas.config: 'org.apache.kafka.common.security.plain.PlainLoginModule required username="" password="";'
gateway.roles: 'upstream' # Note: may be omitted as upstream is the default (used to differentiate from failover clusters)
```
Then, mount the cluster config file in the Gateway container using the configuration `GATEWAY_BACKEND_KAFKA_SELECTOR`:
```yaml theme={null}
GATEWAY_BACKEND_KAFKA_SELECTOR: 'file : { path: /cluster-config.yaml}'
```
Configure your main and upstream clusters through environment variables, defined in the Gateway container:
```yaml theme={null}
KAFKA_MAIN_BOOTSTRAP_SERVERS: ':9092'
KAFKA_MAIN_SECURITY_PROTOCOL: 'SASL_SSL'
KAFKA_MAIN_SASL_MECHANISM: 'PLAIN'
KAFKA_MAIN_SASL_JAAS_CONFIG: 'org.apache.kafka.common.security.plain.PlainLoginModule required username="" password="";'
KAFKA_CLUSTERA_BOOTSTRAP_SERVERS: ':9092'
KAFKA_CLUSTERA_SECURITY_PROTOCOL: 'SASL_SSL'
KAFKA_CLUSTERA_SASL_MECHANISM: 'PLAIN'
KAFKA_CLUSTERA_SASL_JAAS_CONFIG: 'org.apache.kafka.common.security.plain.PlainLoginModule required username="" password="";'
KAFKA_CLUSTERA_GATEWAY_ROLES: 'upstream' # Note: may be omitted as upstream is the default (used to differentiate from failover clusters)
KAFKA_CLUSTERB_BOOTSTRAP_SERVERS: ':9092'
KAFKA_CLUSTERB_SECURITY_PROTOCOL: 'SASL_SSL'
KAFKA_CLUSTERB_SASL_MECHANISM: 'PLAIN'
KAFKA_CLUSTERB_SASL_JAAS_CONFIG: 'org.apache.kafka.common.security.plain.PlainLoginModule required username="" password="";'
KAFKA_CLUSTERB_GATEWAY_ROLES: 'upstream' # Note: may be omitted as upstream is the default (used to differentiate from failover clusters)
```
## Partner virtual clusters
These steps are for setting up a partner virtual cluster manually.
Alternatively, to simplify the process, we recommend creating a [Partner Zone](/guide/use-cases/third-party-data#create-a-partner-zone) which supports multi-clusters.
You can also [Check out the tutorial on creating Partner Zones with multi-cluster Gateway](/guide/tutorials/partner-zone-multi-cluster).
### 1. Create a partner virtual cluster
First, create a new partner virtual cluster.
For partner virtual clusters, `aclEnabled` has to be `true` and `superUsers` must not be empty.
Create this YAML file:
```yaml title="mypartner.yaml" theme={null}
---
kind: VirtualCluster
apiVersion: gateway/v2
metadata:
name: mypartner
spec:
aclEnabled: true
superUsers:
- super-user
type: Partner
```
Then, apply it:
```shell theme={null}
conduktor apply -f mypartner.yaml
```
```sh theme={null}
curl \
--request PUT \
--url 'http://localhost:8888/gateway/v2/virtual-cluster' \
--header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
--header 'Content-Type: application/json' \
--data-raw '{
"kind": "VirtualCluster",
"apiVersion": "gateway/v2",
"metadata": {
"name": "mypartner"
},
"spec": {
"aclEnabled": true,
"superUsers": [ "super-user" ],
"type": "Partner"
}
}'
```
### 2. Alias your topics
Finally, create aliases for existing topics in the partner virtual cluster.
Alias topics within a partner virtual cluster **can only point to topics from the same physical cluster**.
Create this YAML file:
```yaml title="alias-topics.yaml" theme={null}
---
kind: AliasTopic
apiVersion: gateway/v2
metadata:
name: topic1
vCluster: mypartner
spec:
physicalName: internal-topic-name1
physicalCluster: clusterA
---
kind: AliasTopic
apiVersion: gateway/v2
metadata:
name: topic2
vCluster: mypartner
spec:
physicalName: internal-topic-name2
physicalCluster: clusterA
```
Then, apply it:
```sh theme={null}
conduktor apply -f alias-topics.yaml
```
```sh theme={null}
curl \
--request PUT \
--url 'http://localhost:8888/gateway/v2/alias-topic' \
--header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
--header 'Content-Type: application/json' \
--data-raw '{
"kind": "AliasTopic",
"apiVersion": "gateway/v2",
"metadata": {
"name": "topic1",
"vCluster": "mypartner"
},
"spec": {
"physicalName": "internal-topic-name1",
"physicalCluster": "clusterA"
}
}'
curl \
--request PUT \
--url 'http://localhost:8888/gateway/v2/alias-topic' \
--header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
--header 'Content-Type: application/json' \
--data-raw '{
"kind": "AliasTopic",
"apiVersion": "gateway/v2",
"metadata": {
"name": "topic2",
"vCluster": "mypartner"
},
"spec": {
"physicalName": "internal-topic-name2",
"physicalCluster": "clusterA"
}
}'
```
### 3. Create service accounts
Once the virtual cluster is created and contains the topics to expose to your partners, you'll need to create service accounts and configure ACLs (Access Control Lists).
Create two service accounts for the partner virtual cluster: **one super user and one partner user**.
The super user will manage ACLs and grant permissions to the partner user, who will use their account to access the exposed topics.
```yaml title="service-accounts.yaml" theme={null}
---
kind: GatewayServiceAccount
apiVersion: gateway/v2
metadata:
name: super-user
vCluster: mypartner
spec:
type: LOCAL
---
kind: GatewayServiceAccount
apiVersion: gateway/v2
metadata:
name: partner-user
vCluster: mypartner
spec:
type: LOCAL
```
Then, apply it:
```shell theme={null}
conduktor apply -f service-accounts.yaml
```
In order to connect to Gateway using these service accounts, you need to get their associated password.
```shell theme={null}
conduktor run generateServiceAccountToken \
--username super-user \
--v-cluster mypartner \
--life-time-seconds 100000000
conduktor run generateServiceAccountToken \
--username partner-user \
--v-cluster mypartner \
--life-time-seconds 100000000
```
```sh theme={null}
curl \
--request PUT \
--url 'http://localhost:8888/gateway/v2/service-account' \
--header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
--header 'Content-Type: application/json' \
--data-raw '{
"kind": "GatewayServiceAccount",
"apiVersion": "gateway/v2",
"metadata": {
"name": "super-user",
"vCluster": "mypartner"
},
"spec": {
"type": "LOCAL"
}
}'
curl \
--request PUT \
--url 'http://localhost:8888/gateway/v2/service-account' \
--header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
--header 'Content-Type: application/json' \
--data-raw '{
"kind": "GatewayServiceAccount",
"apiVersion": "gateway/v2",
"metadata": {
"name": "partner-user",
"vCluster": "mypartner"
},
"spec": {
"type": "LOCAL"
}
}'
```
To connect to Gateway using these service accounts, you need to get the associated password.
```sh theme={null}
curl \
--request POST \
--url 'http://localhost:8888/gateway/v2/token' \
--header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
--header 'Content-Type: application/json' \
--data-raw '{
"username": "super-user",
"vCluster": "mypartner",
"lifeTimeSeconds": 3600000
}'
curl \
--request POST \
--url 'http://localhost:8888/gateway/v2/token' \
--header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
--header 'Content-Type: application/json' \
--data-raw '{
"username": "partner-user",
"vCluster": "mypartner",
"lifeTimeSeconds": 3600000
}'
```
Put the admin credentials in a file called `mypartner-super-user.properties`:
```properties title="mypartner-super-user.properties" theme={null}
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="super-user" password="$SUPER-USER-PASSWORD";
```
And the partner credentials in a file called `mypartner-partner-user.properties`:
```properties title="mypartner-partner-user.properties" theme={null}
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="partner-user" password="$PARTNER-USER-PASSWORD";
```
### 4. Create ACLs for the service accounts
Before creating ACLs, you need to know how to reach this partner virtual cluster.
For that, make the following request:
```sh theme={null}
curl \
--silent \
--user "admin:conduktor" \
"http://localhost:8888/gateway/v2/virtual-cluster/mypartner"
```
This will return something like this, with the bootstrap address and client properties:
```json theme={null}
{
"kind": "VirtualCluster",
"apiVersion": "gateway/v2",
"metadata": {
"name": "mypartner"
},
"spec": {
"aclEnabled": true,
"superUsers": [ "super-user" ],
"type": "Partner",
"bootstrapServers": "",
"clientProperties": {
"security.protocol": "SASL_PLAINTEXT",
"sasl.mechanism": "PLAIN",
"sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username={{username}} password={{password}};"
}
}
}
```
From there, you have everything you need to create ACLs for the partner service accounts:
```sh theme={null}
# The partner can consume all the topics of this partner virtual cluster
kafka-acls --bootstrap-server \
--command-config mypartner-super-user.properties \
--add \
--allow-principal User:partner-user \
--consumer \
--topic "*" \
--group partner-app
# The partner can produce and consume from the topic1 alias topic
kafka-acls --bootstrap-server \
--command-config mypartner-super-user.properties \
--add \
--allow-principal User:partner-user \
--producer \
--topic topic1
```
### 5. Test partner virtual cluster access
Now that the partner user has the correct ACLs, you can use their credentials to interact with the alias topics and verify that the permissions are correctly set.
```sh theme={null}
kafka-console-producer --bootstrap-server localhost:6974 \
--topic topic1 \
--producer.config mypartner-partner-user.properties
kafka-console-consumer --bootstrap-server localhost:6974 \
--topic topic2 \
--consumer.config mypartner-partner-user.properties \
--group partner-app \
--from-beginning
```
Once confirmed, simply share the `mypartner-partner-user.properties` file and the correct bootstrap server details with your partner.
## Related resources
* [Share data with third parties](/guide/use-cases/third-party-data)
* [Create Partner Zones with multi-cluster Gateway](/guide/tutorials/partner-zone-multi-cluster)
* [Connect Gateway to Kafka](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/connect-to-kafka)
* [Decide on Virtual Clusters](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/virtual-clusters)
* [Manage service accounts](/guide/tutorials/manage-gateway-service-accounts)
* [Resource reference](/guide/reference/gateway-reference)
# Enable Virtual Clusters in Gateway
Source: https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/virtual-clusters
Enable Virtual Clusters in Conduktor Gateway to create isolated logical Kafka environments on a single physical cluster.
A Virtual Cluster in Conduktor Gateway is a logical representation of a Kafka cluster.
This allows you to create multiple virtual clusters while maintaining a single physical Kafka cluster, enabling the simulation of multiple Kafka environments on a single physical infrastructure.
Virtual Clusters are entirely optional. If you choose to not configure any, Conduktor Gateway will act as a transparent proxy for your backing Kafka Cluster. This is the default mode and all topics/resources will be visible and accessible as usual, without any additional configuration.
**From our blog:** [How to isolate non-prod Kafka with Virtual Clusters](https://conduktor.io/blog/how-to-isolate-non-prod-kafka-with-virtual-clusters) Replace VPC-per-environment sprawl with one physical cluster and virtual isolation that satisfies security teams.
## Virtual cluster benefits
**Flexibility and scalability**: Virtual Clusters provide the flexibility to simulate multiple independent Kafka clusters without the need for additional physical resources. This is particularly useful for environments where different teams or applications require separate Kafka instances but maintaining multiple physical clusters would be cost-prohibitive or complex.
**Isolation and multi-tenancy**: By using Virtual Clusters, you can ensure isolation between different logical clusters, similar to enabling multitenancy in Kafka. Each Virtual Cluster can have its own set of topics and consumer groups, and these are managed independently even though they reside on the same physical cluster.
**Resource efficiency**: Instead of deploying and managing multiple physical clusters, which can be resource-intensive and expensive, Virtual Clusters allow you to maximize the utilization of a single physical Kafka cluster. This leads to better resource management and operational efficiency.
## Example
When you create a Virtual Cluster in Conduktor Gateway, it prefixes all resources (such as topics and consumer groups) associated with that Virtual Cluster on the backing physical Kafka cluster.
This prefixing ensures that there's no overlap or conflict between resources belonging to different Virtual Clusters, thereby maintaining their isolation.
In the example below, we assume a topic `order` has been created on Virtual Cluster `vc-alice`. Let's see how other Virtual Clusters and Backing cluster perceive this:
```shell theme={null}
# Listing topics on Virtual Cluster vc-alice
$ kafka-topics --bootstrap-server=gateway:6969 --command-config vc-alice.properties --list
orders
# As expected, Alice sees its topic normally.
# Now let's check what happens if we list the topics with Bob
$ kafka-topics --bootstrap-server=gateway:6969 --command-config vc-bob.properties --list
[]
# Bob doesn't see Alice's topics.
# Let's try to create the same topic for Bob
$ kafka-topics --bootstrap-server=gateway:6969 --command-config vc-bob.properties --create --topic orders
$ kafka-topics --bootstrap-server=gateway:6969 --command-config vc-bob.properties --list
orders
# If we contact directly the backing cluster instead of Gateway,
# we can see both topics under a different name. This is the actual topic name on the Kafka cluster, which is observed when not interacting through Gateway.
$ kafka-topics.sh --bootstrap-server=backing-kafka:9092 --list
vc-alice.orders
vc-bob.orders
```
## Related resources
* [Resource reference](/guide/reference/gateway-reference)
* [Manage service accounts](/guide/tutorials/manage-gateway-service-accounts)
* [Set up Gateway for multi-clusters](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/multi-clusters)
# Conduktor Schema Registry Proxy configuration examples
Source: https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-schema-registry-proxy/configuration
Deployment examples for Conduktor Schema Registry Proxy: production and multi-node docker-compose, Confluent Cloud and Keycloak integration, monitoring.
Ready-to-adapt deployment examples and operational guidance for Schema Registry Proxy (SRP). For the full list of configuration options, see the [Schema Registry Proxy environment variables reference](/guide/reference/schema-registry-proxy-environment-variables).
## Deployment Examples
## Production Deployment with Full Security
```yaml theme={null}
version: "3.8"
services:
schema-registry-proxy:
image: conduktor/conduktor-schema-registry-proxy:0.1.0-rc1
hostname: srp-prod-1
environment:
# Core Configuration
SCHEMA_REGISTRY_PROXY_APP_ID: srp-prod-cluster
PORT: "8080"
# SSL/TLS Server Configuration
SSL_KEYSTORE_PATH: /opt/srp/certs/server.jks
SSL_KEYSTORE_PASSWORD: ${SSL_KEYSTORE_PASSWORD}
SSL_TRUSTSTORE_PATH: /opt/srp/certs/truststore.jks
SSL_TRUSTSTORE_PASSWORD: ${SSL_TRUSTSTORE_PASSWORD}
SSL_SECURE_PORT: "8443"
SSL_INSECURE_ENABLED: "false"
# Kafka Configuration with SASL/SSL
KAFKA_BOOTSTRAP_SERVERS: kafka1:9093,kafka2:9093,kafka3:9093
KAFKA_SECURITY_PROTOCOL: SASL_SSL
KAFKA_SASL_MECHANISM: SCRAM-SHA-512
KAFKA_SASL_JAAS_CONFIG: 'org.apache.kafka.common.security.scram.ScramLoginModule required username="${KAFKA_USER}" password="${KAFKA_PASSWORD}";'
KAFKA_SSL_TRUSTSTORE_LOCATION: /opt/srp/certs/kafka-truststore.jks
KAFKA_SSL_TRUSTSTORE_PASSWORD: ${KAFKA_TRUSTSTORE_PASSWORD}
# Schema Registry Backend with mTLS
CONFLUENT_SCHEMA_REGISTRY_URL:
SCHEMA_REGISTRY_SSL_TRUSTSTORE_LOCATION: /opt/srp/certs/sr-truststore.jks
SCHEMA_REGISTRY_SSL_TRUSTSTORE_PASSWORD: ${SR_TRUSTSTORE_PASSWORD}
SCHEMA_REGISTRY_SSL_KEYSTORE_LOCATION: /opt/srp/certs/sr-client.jks
SCHEMA_REGISTRY_SSL_KEYSTORE_PASSWORD: ${SR_KEYSTORE_PASSWORD}
# JWT Authentication
AUTH_PROVIDER: jwt
JWT_JWKS_URL:
JWT_VALIDATE_EXPIRATION: "true"
JWT_SUBJECT_CLAIM_NAME: preferred_username
# Observability
OTEL_EXPORTER_OTLP_ENDPOINT:
OTEL_TRACES_ENABLED: "true"
OTEL_METRICS_ENABLED: "true"
LOG_LEVEL: INFO
volumes:
- ./certs:/opt/srp/certs:ro
ports:
- "8443:8443"
- "9464:9464"
deploy:
replicas: 3
resources:
limits:
memory: 2G
cpus: "2"
reservations:
memory: 1G
cpus: "1"
```
## Multi-Node Deployment Configuration
For high availability deployments with multiple SRP instances:
```yaml theme={null}
version: "3.8"
services:
schema-registry-proxy-1:
image: conduktor/conduktor-schema-registry-proxy:0.1.0-rc1
environment:
SCHEMA_REGISTRY_PROXY_APP_ID: srp-cluster
CLUSTER_NODE_INDEX: "0"
CLUSTER_MAX_NODE_INDEX: "2"
# ... other configuration ...
schema-registry-proxy-2:
image: conduktor/conduktor-schema-registry-proxy:0.1.0-rc1
environment:
SCHEMA_REGISTRY_PROXY_APP_ID: srp-cluster
CLUSTER_NODE_INDEX: "1"
CLUSTER_MAX_NODE_INDEX: "2"
# ... other configuration ...
schema-registry-proxy-3:
image: conduktor/conduktor-schema-registry-proxy:0.1.0-rc1
environment:
SCHEMA_REGISTRY_PROXY_APP_ID: srp-cluster
CLUSTER_NODE_INDEX: "2"
CLUSTER_MAX_NODE_INDEX: "2"
# ... other configuration ...
```
## Security Configurations
### Example: Confluent Cloud Integration
```yaml theme={null}
environment:
# Kafka connection to Confluent Cloud
KAFKA_BOOTSTRAP_SERVERS:
KAFKA_SECURITY_PROTOCOL: SASL_SSL
KAFKA_SASL_MECHANISM: PLAIN
KAFKA_SASL_JAAS_CONFIG: 'org.apache.kafka.common.security.plain.PlainLoginModule required username="${CONFLUENT_API_KEY}" password="${CONFLUENT_API_SECRET}";'
# Schema Registry connection
CONFLUENT_SCHEMA_REGISTRY_URL:
SCHEMA_REGISTRY_BASIC_AUTH_CREDENTIALS_SOURCE: USER_INFO
SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO: "${SR_API_KEY}:${SR_API_SECRET}"
```
### Example: Keycloak Integration
```yaml theme={null}
environment:
AUTH_PROVIDER: jwt
JWT_JWKS_URL:
JWT_VALIDATE_EXPIRATION: "true"
JWT_SUBJECT_CLAIM_NAME: preferred_username
```
## Monitoring and Operations
### Health Checks
The service exposes health endpoints for monitoring:
* **Health Check**: `GET https://host:PORT/health`
* **Prometheus Metrics**: `GET https://host:PORT/metrics`
### Metrics
Key metrics exposed via Prometheus:
* `http_server_requests_total`: Total HTTP requests
* `http_server_request_duration_seconds`: Request latency histogram
* `kafka_consumer_lag`: Consumer lag for configuration topics
* `schema_registry_backend_requests_total`: Backend request counts
* `schema_registry_backend_request_duration_seconds`: Backend request latency
* `srp_heartbeat_published_total`: Count of published heartbeats
* `srp_permissions_updated_total`: Count of permission updates received
### Logging
Structured JSON logging with configurable levels:
```yaml theme={null}
environment:
LOG_LEVEL: INFO # DEBUG for troubleshooting
JAVALIN_DEV_LOGGING_ENABLED: "false" # Never enable in production
```
### Distributed Tracing
When configured with OpenTelemetry, the service automatically creates spans for:
* HTTP requests (parent spans)
* Backend Schema Registry calls (child spans)
* Kafka operations (consuming permissions, publishing heartbeats)
* Authentication/authorization operations
# Deploy Schema Registry Proxy
Source: https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-schema-registry-proxy/index
Deploy the Conduktor Schema Registry Proxy for authentication, authorization and observability on top of Confluent Schema Registry.
The Schema Registry Proxy (SRP) sits between your Kafka clients and Confluent Schema Registry, adding authentication, authorization and observability. It integrates with Console for Self-service permission management, and automatically synchronizes and enforces access rights for schemas.
Schema Registry Proxy is in early access. Only Confluent-compatible schema registries are supported.
**From our blog:** [Introducing Schema Registry Proxy](https://conduktor.io/blog/introducing-schema-registry-proxy) How SRP adds authentication, authorization, and observability on top of Confluent Schema Registry.
SRP is early access — talk to our team to get artifact access, deployment help and a working setup on your stack.
## Prerequisites
| Requirement | Details |
| ----------------------------- | ---------------------------------------------------------------------------------- |
| **Apache Kafka** | Version 3.8+. Used for permission distribution and heartbeat discovery. |
| **Confluent Schema Registry** | Version 7.0+. The backend SRP proxies requests to. |
| **Authentication provider** | OIDC-compliant provider with a JWKS endpoint (for example, Keycloak, Auth0, Okta). |
| **Conduktor Console** | For Self-service permission management. |
## Enable SRP discovery in Console
Set the following environment variable on your Console deployment:
```yaml theme={null}
environment:
CDK_SCHEMAREGISTRYPROXY_ENABLED: "true"
```
Once SRP is running, the proxy instance appears in Console:
1. Go to the Kafka cluster configured as the SRP backing cluster.
2. Open the **Schema Registry** tab for a Schema Registry with ConfluentLike flavor.
3. Select the discovered SRP instance from the dropdown.
## Deploy with Docker Compose
```yaml theme={null}
services:
schema-registry-proxy:
image: conduktor/conduktor-schema-registry-proxy:0.1.0-rc1
hostname: schema-registry-proxy
depends_on:
kafka:
condition: service_healthy
schema-registry:
condition: service_healthy
environment:
PORT: "7070"
KAFKA_BOOTSTRAP_SERVERS:
CONFLUENT_SCHEMA_REGISTRY_URL:
AUTH_PROVIDER: jwt
JWT_JWKS_URL:
SCHEMA_REGISTRY_PROXY_APP_ID: srp-cluster-1
ports:
- "7070:7070"
- "9464:9464"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:7070/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
```
## Verify deployment
Check that SRP is healthy:
```bash theme={null}
curl http://host:PORT/health
```
Confirm Prometheus metrics are available:
```bash theme={null}
curl http://host:PORT/metrics
```
In Console, verify the SRP instance appears in the **Schema Registry** tab dropdown for the relevant cluster.
## Next steps
* [Configure SRP environment variables](/guide/reference/schema-registry-proxy-environment-variables)
* [Troubleshoot common issues](/guide/conduktor-in-production/deploy-artifacts/deploy-schema-registry-proxy/troubleshooting)
# Troubleshoot Schema Registry Proxy
Source: https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-schema-registry-proxy/troubleshooting
Common issues and fixes for Schema Registry Proxy deployment and operation.
### SRP not discovered by Console
**Symptom**: The SRP instance does not appear in the Console Schema Registry dropdown.
**Cause**: Console is not receiving heartbeats from SRP.
**Fix**:
1. Verify `CDK_SCHEMAREGISTRYPROXY_ENABLED=true` is set on Console.
2. Check that `KAFKA_HEARTBEAT_ENABLED=true` on SRP (this is the default).
3. Confirm SRP and Console connect to the same Kafka cluster.
4. Verify the `_conduktor_srp_events` topic exists and is accessible.
### Permissions not applied
**Symptom**: Users receive authorization errors even after permissions are granted in Console.
**Cause**: SRP is not consuming permission updates from Kafka.
**Fix**:
1. Check that `AUTH_USE_REACTIVE_CONFIG=true` on SRP (this is the default).
2. Verify the `_conduktor_srp_commands` topic exists with the expected replication factor.
3. Confirm Console is connected to the same Kafka cluster as SRP.
4. Review SRP logs for permission consumption messages.
### Authentication failures
**Symptom**: Clients receive `401 Unauthorized` responses.
**Cause**: JWT token validation is failing.
**Fix**:
1. Verify the JWKS endpoint is accessible from the SRP container.
2. Confirm `JWT_SUBJECT_CLAIM_NAME` matches the claim used in your tokens.
3. Check token expiration — set `JWT_VALIDATE_EXPIRATION=false` temporarily to isolate the issue.
# Manage your Console license
Source: https://docs.conduktor.io/guide/conduktor-in-production/manage-licenses/console
Obtain, apply, verify, renew and monitor usage of your Conduktor Console license.
## Obtain a license
How you obtain a license depends on the type:
* **Enterprise** — for production deployments running Console, Gateway or both. The same key applies to both products. [Contact Conduktor sales](https://conduktor.io/contact) to start the process.
* **Console Community Edition** — free, no license key needed. Console runs in Community Edition when no `CDK_LICENSE` environment variable is set. [See what's included in Console Community Edition](/guide/conduktor-in-production/manage-licenses/console-community-edition).
## Apply your license
We recommend applying your license to the environment variables. Alternatively, you can add it to your YAML config file.
```js title=".env" theme={null}
CDK_LICENSE=''
```
```yaml title="platform-config.yaml" theme={null}
license: ''
```
## Verify your license
Use Console to check that your license has been applied correctly. You can do this via the UI or by checking the logs.
### Use the UI
Log into Conduktor Console then go to **Settings** > **Plan**:
### Use the logs
Run Console then check the top of the startup logs: `License Enterprise is valid until 2026-01-01 00:00:00`.
## Renew or update your license
To renew an existing or apply a new license, change the `CDK_LICENSE` value in the environment variables (or the `license` value in the YAML file, depending on your setup) and re-deploy the Conduktor Console container.
## Monitor license expiration
Console exposes a Prometheus metric to track the number of days remaining before license expiration: `console_license_remaining_days`.
You can set up monitoring alerts on this metric to proactively manage license renewals. [Check out the metrics reference](/guide/reference/console-metrics).
## Track Console usage
Usage data allows you to monitor Console usage and license utilization.
You can see all active users, view Self-service applications and instance usage, view access requests and see the total number of Partner Zones and shared topics.
Usage data is accessible to admin users only.
### View usage data
In Console, go to **Settings** > **Plan** to view your current license details.
Click **View usage** to open the usage dashboard.
### Tracked metrics
| Metric | What it counts |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Active users** | Users who logged in to Console (via local authentication or SSO) within the selected time period or three months prior, plus owners of a [Self-service](/guide/conduktor-concepts/self-service) application. |
| **Applications** | Total number of Self-service applications across all clusters. |
| **Application instances** | Total number of Self-service application instances across all clusters. |
| **Access requests** | Total number of application instance permission requests — pending, approved and denied — across all clusters. |
| **Partner Zones** | Total number of [Partner Zones](/guide/conduktor-concepts/partner-zones) created for sharing Kafka topics with external partners. |
| **Shared topics** | Total number of topics shared across Partner Zones. |
The active users graph displays monthly trends, showing how usage patterns change over time. When your license limit has been reached or surpassed, Console displays a warning indicator.
### Time periods
The graph provides a visual overview of how Console usage has changed over time, helping you identify trends and plan for license capacity.
* **Last 30 days**: rolling 30-day window from today
* **Current month**: month-to-date metrics
* **Last 12 months**: monthly aggregated data for the past year
Each data point represents the total count for that month, with the current month showing data up to today.
### Export usage data
Click **Export CSV** to download usage data for further analysis or reporting. The export includes metric names, values, time period information and a timestamp.
## Troubleshoot
Conduktor considers the user as being a distinct entry that's stored in the users database, regardless of whether they've ever logged in/are attached to any existing groups or not. Entries are created in the users database when: the first admin account(s) are created, users are added via basic authentication or users log in via SSO (or are added explicitly to the Conduktor UI prior to their first login via SSO). To manage users, go to **Settings** > **Users**.
Conduktor employs a soft limit to ensure that your service is not disrupted in cases where you onboard more users than expected. We understand it can be difficult to plan in advance and ask you get in touch with your customer success team if you exceed your user threshold. Please note this limit is a contractual agreement between Conduktor and your company, and may be subject to auditing from time to time.
In Console, go to **Settings** > **Users** to review the last login time of each user.
# Console Community Edition
Source: https://docs.conduktor.io/guide/conduktor-in-production/manage-licenses/console-community-edition
Console Community Edition is Conduktor's free self-hosted tier. Manage up to 3 Kafka clusters and 50 users with core Console features and no time limit.
No license key is needed: Console runs in Community Edition mode when no `CDK_LICENSE` environment variable is set.
If you're currently using the Community license, these limits will apply only when you upgrade to Console v1.43.0 and above.
## What's included
Community Edition gives access to Console for **up to 50 users** and **3 clusters**.
| Feature | Included |
| ---------------------------------- | -------- |
| Topic management (CRUD) | ✓ |
| Topic activity metrics | ✓ |
| Schema Registry management | ✓ |
| Consumer group management | ✓ |
| Kafka Connect management | ✓ |
| ksqlDB management | ✓ |
| ACL and service account management | ✓ |
| Labels | ✓ |
| Multi-cluster management (up to 3) | ✓ |
| Feature | Included |
| -------------------------- | -------- |
| Search and filter messages | ✓ |
| JavaScript/JQ filters | ✓ |
| SQL queries on topics | ✓ |
| Produce messages | ✓ |
| Message export | ✓ |
| Live tail | ✓ |
| JSON/Avro/Protobuf support | ✓ |
| Custom serializers | ✓ |
| Message sharing | ✓ |
| DLQ/Reprocess messages | ✓ |
Monitoring in the UI is restricted to showing data for to the last hour only.
| Feature | Included |
| -------------------------- | -------- |
| Real-time cluster metrics | ✓ |
| Real-time broker stats | ✓ |
| Real-time consumer lag | ✓ |
| Consumer lag visualization | ✓ |
| Feature | Included |
| --------------------- | -------- |
| Basic user RBAC | ✓ |
| SSO via OIDC/SAML | ✓ |
| LDAP/Active Directory | ✓ |
| Feature | Included |
| --------------------------- | -------- |
| REST API (OpenAPI) | ✓ |
| API keys/token management | ✓ |
| CLI tool | ✓ |
| Email notifications | ✓ |
| Slack integration | ✓ |
| Microsoft Teams integration | ✓ |
| Webhook support | ✓ |
| MCP | ✓ |
| Feature | Included |
| ------------ | -------- |
| Topics | ✓ |
| Cost control | ✓ |
| VIP topics | ✓ |
## What's not included
The following capabilities require a paid plan:
| Category | Features |
| ------------ | ------------------------------------------------------------------------- |
| Monitoring | Historical metrics, alerting, Prometheus integration |
| Security | Data masking, audit logs, external group sync, user inactivity management |
| Governance | Self-service catalogs, approval workflows, topic ownership, data lineage |
| Automation | Terraform provider, GitOps integration, auto connector restart |
| Intelligence | Chargeback, cost optimization, cluster utilization analytics |
| Enterprise | Multi-tenancy, disaster recovery, dedicated support |
## When to upgrade
**Your team is growing**
* More than 50 users need access to Conduktor
* You need to manage more than 3 Kafka clusters
* Multiple teams need isolated access with group-based permissions
**You're running in production**
* You need historical metrics to troubleshoot issues that happened hours or days ago
* You want alerts for consumer lag, broker health or custom conditions
* You need Prometheus integration for centralized monitoring
* Automatic connector restart is required for reliability
**You need governance and compliance**
* Audit logs are required for compliance or security reviews
* Data masking is needed to protect sensitive information
* You want Self-service catalogs so teams can request resources without waiting
* Approval workflows are needed to control access to production topics
**You're scaling automation**
* Infrastructure as code with Terraform is part of your workflow
* You need GitOps integration for declarative resource management
* Cost allocation and Chargeback reporting is required for internal billing
**You need enterprise support**
* Dedicated support with SLAs for faster issue resolution
* Disaster recovery and multi-tenancy for large-scale deployments
[Contact us](https://conduktor.io/contact) to discuss your requirements.
# Manage your Gateway license
Source: https://docs.conduktor.io/guide/conduktor-in-production/manage-licenses/gateway
Obtain, apply, verify and update your Conduktor Gateway license.
## Obtain a license
How you obtain a license depends on the type:
* **Enterprise** — for production deployments running Console, Gateway or both. The same key applies to both products. [Contact Conduktor sales](https://conduktor.io/contact) to start the process.
* **Gateway Community Edition** — a free license for Gateway as a Kafka-managed passthrough proxy, without Interceptors, Virtual Clusters or other Gateway-managed features. Applied via `GATEWAY_LICENSE_KEY`. [Find out more about Gateway Community Edition](/guide/conduktor-in-production/manage-licenses/gateway-community-edition).
## Apply your license
Set the license as an environment variable in your Gateway configuration:
```js title=".env" theme={null}
GATEWAY_LICENSE_KEY=""
```
## Verify your license
Check the expiry of the license used to launch Gateway by looking at the startup logs:
```md theme={null}
2026-05-20T14:38:09.961+0000 [main] [INFO] [ConfiguredLicenseService:57] - Loaded license with expiry: 2027-01-01T00:00:00.000Z
```
## Update your license
To update your license, set the `GATEWAY_LICENSE_KEY` environment variable to the new value and re-deploy the Conduktor Gateway container.
## Monitor license expiration
Gateway exposes a Prometheus metric to track the number of days remaining before license expiration: `gateway_license_remaining_days`.
You can set up monitoring alerts on this metric to proactively manage license renewals. [Check out the metrics reference](/guide/reference/gateway-metrics).
# Gateway Community Edition
Source: https://docs.conduktor.io/guide/conduktor-in-production/manage-licenses/gateway-community-edition
Gateway Community Edition runs Conduktor Gateway as a Kafka-managed proxy that translates broker addresses, without storing credentials or exposing Gateway-managed features.
Gateway Community Edition is a license-restricted mode of Conduktor Gateway. Gateway forwards Kafka client connections 1:1 to brokers and translates broker addresses, without storing credentials or exposing Gateway-managed features.
Use Gateway Community Edition when you need to:
* Make privately accessible brokers (such as Confluent Cloud dedicated clusters or Google Cloud managed Kafka) reachable from outside their private network segment.
* Add or change advertised listener addresses without reconfiguring broker listeners.
Gateway Community Edition requires a separate license from a standard Conduktor Gateway license. See [Obtain a Gateway Community Edition license](#obtain-a-gateway-community-edition-license).
## What's included
Conduktor Gateway runs in [Kafka-managed mode](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/listener-security#kafka_managed). Gateway forwards client connections 1:1 to brokers, and clients authenticate using their SASL credentials, which the Kafka broker validates. Gateway does not store or validate credentials.
## What's not included
The following Gateway capabilities are unavailable under a Gateway Community Edition license:
* [Interceptors](/guide/conduktor-concepts/interceptors)
* [Topic views](/guide/conduktor-concepts/logical-topics#topic-views)
* [Alias topics](/guide/conduktor-concepts/logical-topics#alias-topics)
* [Virtual Clusters](/guide/conduktor-concepts/virtual-clusters)
* [Gateway service accounts](/guide/conduktor-concepts/gateway-service-accounts)
* [Gateway groups](/guide/reference/gateway-reference#gatewaygroup)
* [Topic concentration](/guide/conduktor-concepts/logical-topics#concentrated-topics)
* [Failover](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/failover)
## Configure Gateway Community Edition
Deploy Gateway as a [Kafka-managed proxy](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/listener-security#kafka_managed).
Gateway Community Edition supports Kafka-managed security only, via `SASL_PLAINTEXT` or `SASL_SSL`.
Apply the license the same way as any other Gateway license:
```js title=".env" theme={null}
GATEWAY_LICENSE_KEY=""
```
[See Licensing overview](/guide/conduktor-in-production/manage-licenses) for verification and renewal.
## Restrictions
A Gateway Community Edition license enforces three customer-visible restrictions:
* **Disallowed configuration prevents startup.** If Gateway starts with configuration that requires an excluded feature (such as Interceptors or Virtual Clusters), it refuses to start and logs the offending configuration.
* **Disallowed Admin API calls return 403.** Admin API calls that create or modify excluded features return `403 Forbidden`. The response identifies the disallowed feature.
* **Product-usage metrics cannot be disabled.** Gateway Community Edition deployments report anonymous product-usage metrics to Conduktor. The reporting cannot be disabled from within Gateway.
## Obtain a Gateway Community Edition license
To request a Gateway Community Edition license, see [conduktor.io/gateway/community-edition](https://www.conduktor.io/gateway/community-edition).
# Monitoring setup for Conduktor Gateway and Console
Source: https://docs.conduktor.io/guide/conduktor-in-production/monitor
Set up health monitoring for Conduktor Gateway and Console. Configure liveness and readiness endpoints, Prometheus metrics.
## Gateway health and monitoring
### Liveness endpoint
`/health/live`
Returns a status HTTP 200 when Gateway is up.
```shell title="cURL example" theme={null}
curl -s http://localhost:8888/health/live
```
Could be used to set up probes on Kubernetes.
#### Kubernetes startup probe
```yaml title="Port configuration" theme={null}
ports:
- containerPort: 8888
protocol: TCP
name: httpprobe
```
```yaml title="Probe configuration" theme={null}
startupProbe:
httpGet:
path: /health/live
port: httpprobe
enabled: true
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 1
failureThreshold: 5
successThreshold: 1
```
#### Kubernetes liveness probe
```yaml title="Port configuration" theme={null}
ports:
- containerPort: 8888
protocol: TCP
name: httpprobe
```
```yaml title="Probe configuration" theme={null}
livenessProbe:
httpGet:
path: /health/live
port: httpprobe
enabled: true
initialDelaySeconds: 0
periodSeconds: 5
timeoutSeconds: 1
failureThreshold: 3
successThreshold: 1
```
### Readiness endpoint
`/health/ready`
Returns readiness of Gateway. Modules status:
* `NOTREADY` (initial state)
* `READY`
This endpoint returns a 200 status code if Gateway is in a `READY` state. Otherwise, it returns a 503 status code if Gateway is not ready to accept traffic yet.
```shell title="cURL example" theme={null}
curl -s http://localhost:8888/health/ready
# READY
```
Could be used to set up probes on [docker-compose](#docker-compose-probe-setup) or [Kubernetes](#kubernetes-readiness-probe).
#### Docker-compose probe setup
The `healthcheck` configuration below is optional, as it's already backed into the Conduktor image and not required unless you're experiencing issues.
```yaml theme={null}
healthcheck:
test:
[
'CMD-SHELL',
'curl --fail http://localhost:${CDK_LISTENING_PORT:-8888}/health/ready',
]
interval: 5s
retries: 25
```
#### Kubernetes readiness probe
```yaml title="Port configuration" theme={null}
ports:
- containerPort: 8888
protocol: TCP
name: httpprobe
```
```yaml title="Probe configuration" theme={null}
readinessProbe:
httpGet:
path: /health/ready
port: httpprobe
enabled: true
initialDelaySeconds: 0
periodSeconds: 5
timeoutSeconds: 1
failureThreshold: 3
successThreshold: 1
```
### Gateway versions
`/versions`
Returns the Gateway version.
```shell title="cURL example" theme={null}
curl -s http://localhost:8888/versions | jq .
# {
# "gateway": "3.12.0-SNAPSHOT"
# }
```
### Access Prometheus metrics from Gateway
The Prometheus endpoint is `:/metrics`. For example:
```bash theme={null}
localhost:8888/metrics
```
Please be aware that if `GATEWAY_SECURED_METRICS` is enabled (which is the default setting), you will need to use the credentials specified in `GATEWAY_ADMIN_API_USERS` to access it.
For example, using the default credentials, you can access the metrics with the following command:
```bash title='Retrieve Gateway Metrics' theme={null}
curl conduktor-gateway:8888/metrics --user "admin:conduktor"
```
### Available metrics for Prometheus
[Find out more about Gateway metrics](/guide/reference/gateway-metrics).
### Audit log events
| **Event type** | **Description** |
| -------------------------------------------- | ------------------------------------------------------------ |
| **Admin.KafkaConnect.Create** | A Kafka Connect instance is created. |
| **Admin.KafkaConnect.Update** | A Kafka Connect instance is updated |
| **Admin.KafkaConnect.Delete** | A Kafka Connect instance is deleted. |
| **Admin.KsqlDB.Create** | A ksqlDB instance is created. |
| **Admin.KsqlDB.Update** | A ksqlDB instance is updated. |
| **Admin.KsqlDB.Delete** | A ksqlDB instance is deleted. |
| **Admin.KafkaCluster.Create** | A Kafka cluster is created. |
| **Admin.KafkaCluster.Update** | A Kafka cluster is updated. |
| **Admin.KafkaCluster.Delete** | A Kafka cluster is deleted. |
| **Admin.SchemaRegistry.ChangeCompatibility** | The global compatibility of the schema registry is updated. |
| **Admin.Integration.Update** | The alert integration (Slack, MS Teams, Webhook) is updated. |
| **Admin.AdminApiKey.Create** | A new admin API key is created. |
| **Admin.AdminApiKey.Delete** | An admin API key is deleted. |
| **Admin.DataMaskingPolicy.Create** | A data masking policy is created. |
| **Admin.DataMaskingPolicy.Update** | A data masking policy is updated. |
| **Admin.DataMaskingPolicy.Delete** | A data masking policy is deleted. |
| **Admin.Certificate.Create** | A certificate is created. |
| **Admin.Certificate.Delete** | A certificate is deleted. |
| **Iam.User.Create** | IAM user is created. |
| **Iam.User.Update** | IAM user is updated. |
| **Iam.User.Delete** | IAM user is deleted. |
| **Iam.User.Login** | IAM user logs in. |
| **Iam.User.Logout** | IAM user logs out. |
| **Iam.Group.Create** | IAM group is created. |
| **Iam.Group.Update** | IAM group is updated. |
| **Iam.Group.Delete** | IAM group is deleted. |
## Console endpoints
### Liveness endpoint
`/api/health/live`
Returns a status HTTP 200 when Console is up.
```shell title="cURL example" theme={null}
curl -s http://localhost:8080/api/health/live
```
Could be used to set up probes on Kubernetes.
#### Kubernetes startup probe
```yaml title="Port configuration" theme={null}
ports:
- containerPort: 8080
protocol: TCP
name: httpprobe
```
```yaml title="Probe configuration" theme={null}
startupProbe:
httpGet:
path: /api/health/live
port: httpprobe
enabled: false
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 10
successThreshold: 1
```
#### Kubernetes liveness probe
```yaml title="Port configuration" theme={null}
ports:
- containerPort: 8080
protocol: TCP
name: httpprobe
```
```yaml title="Probe configuration" theme={null}
livenessProbe:
httpGet:
path: /api/health/live
port: httpprobe
enabled: true
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
successThreshold: 1
```
### Readiness/startup endpoint
`/api/health/ready`
Returns readiness of the Console.
Modules status :
* `NOTREADY` (initial state)
* `READY`
This endpoint returns a 200 status code if Console is in a `READY` state. Otherwise, it returns a 503 status code if Console fails to start.
```shell title="cURL example" theme={null}
curl -s http://localhost:8080/api/health/ready
# READY
```
Could be used to set up probes on docker-compose or Kubernetes.
#### Docker-compose probe setup
The `healthcheck` configuration below is optional, as it's already backed into the Conduktor image and not required unless you're experiencing issues.
```yaml theme={null}
healthcheck:
test:
[
'CMD-SHELL',
'curl --fail http://localhost:${CDK_LISTENING_PORT:-8080}/api/health/ready',
]
interval: 10s
start_period: 120s # Leave time for the psql init scripts to run
timeout: 5s
retries: 3
```
#### Kubernetes readiness probe
```yaml title="Port configuration" theme={null}
ports:
- containerPort: 8080
protocol: TCP
name: httpprobe
```
```yaml title="Probe configuration" theme={null}
readinessProbe:
httpGet:
path: /api/health/ready
port: httpprobe
enabled: true
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
successThreshold: 1
```
### Console versions
`/api/versions`
This endpoint exposes module versions used to build the Console along with the overall Console version.
```shell title="cURL example" theme={null}
curl -s http://localhost:8080/api/versions | jq .
# {
# "platform": "1.27.0",
# "platformCommit": "ed849cbd545bb4711985ce0d0c93ca8588a6b31f",
# "console": "f97704187a7122f78ddc9110c09abdd1a9f9d470",
# "console_web": "05dea2124c01dfd9479bc0eb22d9f7d8aed6911b"
# }
```
### Access Prometheus metrics from Console
The Prometheus metrics endpoints are:
* `/api/monitoring/metrics` - Console and Kafka metrics
* `/monitoring/metrics` - Kafka metrics
Since Console v1.42.0, you can protect these endpoints with basic auth by configuring the `CDK_MONITORING_BASICAUTH_EMAIL` and `CDK_MONITORING_BASICAUTH_PASSWORD` environment variables.
When basic auth is configured, you need to provide credentials to access the metrics:
```bash title='Retrieve Console metrics with basic auth' theme={null}
curl http://conduktor-console:8080/api/monitoring/metrics --user "metrics@example.com:secure-password"
```
Basic auth is optional in v1.42.0 to avoid breaking existing deployments. In a future release, basic auth will become mandatory for metrics scraping.
[Find out more about Console metrics](/guide/reference/console-metrics#console-metrics).
## Cortex monitoring endpoints
### Cortex endpoint
`/ready` on port `9009`
Returns a status 200 with response `ready` if Cortex is running
```shell title="cURL example" theme={null}
curl -s "http://localhost:9009/ready"
```
### Alertmanager endpoint
Since Console 1.46.0, the alert manager API is served on port `9009` alongside the Cortex API, so the Cortex readiness endpoint above also covers it. Before Console 1.46.0, the alert manager served `/ready` on its own port `9010`. See the [Console 1.46.0 breaking change](/guide/release-notes#cortex-alert-manager-port-v1-46-0).
### Prometheus endpoint
`/-/healthy` on port `9090`
Returns a status 200 with response `Prometheus Server is Healthy.` if Prometheus is running
```shell title="cURL example" theme={null}
curl -s "http://localhost:9090/-/healthy"
```
# Gateway monitoring & alerting recommendations
Source: https://docs.conduktor.io/guide/conduktor-in-production/monitor/gateway_jmx_recommendations
This page provides best practices and recommendations for monitoring and alerting on Conduktor Gateway deployments. Use these as starting points and tune thresholds to match your environment's baseline.
Before you start, make sure you have [set up monitoring](/guide/conduktor-in-production/monitor) and can access the Gateway Prometheus endpoint.
Example Grafana dashboards are available on Github, in the Conduktor Helm package at [charts/gateway/grafana-dashboards](https://github.com/conduktor/conduktor-public-charts/tree/main/charts/gateway/grafana-dashboards).
## Metrics endpoint
Gateway exposes a Prometheus-compatible `/metrics` endpoint on its HTTP API port. By default, this endpoint is unauthenticated, meaning anyone with network access to the port can scrape metrics.
For production deployments, we recommend configuring credentials in [`GATEWAY_ADMIN_API_USERS`](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/env-variables#http-api) and enabling authentication on the metrics endpoint by setting the [`GATEWAY_SECURED_METRICS`](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/env-variables#http-api) environment variable to `true`. When enabled, requests to the `/metrics` endpoint require the same credentials as other Gateway HTTP API calls.
## Request flow overview
The diagram below shows how traffic flows through Gateway, and where key metrics are captured. Use it as a reference when choosing what to monitor.
```mermaid theme={null}
flowchart LR
C[Client] -->|"upstream (toward Kafka)"| GW[Gateway]
GW -->|upstream| K[Kafka Broker]
K -->|downstream| GW
GW -->|"downstream (toward client)"| C
style C fill:#e8f4f8,stroke:#333
style GW fill:#fff3cd,stroke:#333
style K fill:#d4edda,stroke:#333
```
The `direction` label on byte metrics refers to the direction of data flow: `upstream` means bytes flowing from clients toward Kafka, and `downstream` means bytes flowing from Kafka toward clients.
**Where metrics are captured:**
| Boundary | Key metrics |
| ----------------- | ----------------------------------------------------------------------------------------------------------- |
| Client to Gateway | `gateway_active_connections_vcluster`, `gateway_bytes_exchanged{direction="upstream"}` |
| Gateway to Kafka | `gateway_upstream_connections_upstream_connected`, `gateway_bytes_exchanged_vcluster{direction="upstream"}` |
| Kafka to Gateway | `gateway_bytes_exchanged_vcluster{direction="downstream"}` |
| Gateway to Client | `gateway_bytes_exchanged{direction="downstream"}` |
| Round-trip | `gateway_latency_request_response` |
| Inside Gateway | `gateway_current_inflight_apiKeys`, `gateway_thread_tasks` |
## Availability and license
### Gateway down
Alert when the Gateway instance is unreachable.
```yaml theme={null}
alert: GatewayDown
expr: up{job="conduktor-gateway"} == 0
for: 1m
labels:
severity: critical
annotations:
summary: "Gateway instance {{ $labels.instance }} is down"
```
### License expiring
Track [`gateway_license_remaining_days`](/guide/reference/metric-reference#gateway-reference) and alert at two thresholds to give time for renewal.
```yaml theme={null}
alert: GatewayLicenseExpiringSoon
expr: gateway_license_remaining_days < 14
for: 1h
labels:
severity: warning
annotations:
summary: "Gateway license expires in {{ $value }} days"
```
```yaml theme={null}
alert: GatewayLicenseCritical
expr: gateway_license_remaining_days < 3
for: 5m
labels:
severity: critical
annotations:
summary: "Gateway license expires in {{ $value }} days - renew immediately"
```
### Kafka node loss
Track [`gateway_upstream_io_nodes`](/guide/reference/metric-reference#gateway-reference) to detect when Gateway loses visibility of Kafka brokers.
```yaml theme={null}
alert: GatewayKafkaNodeLoss
expr: gateway_upstream_io_nodes < 3
for: 2m
labels:
severity: critical
annotations:
summary: "Gateway sees only {{ $value }} Kafka nodes (expected 3+)"
```
Adjust the threshold to match your Kafka cluster size.
## Connections
Kafka connections are long-lived. In a stable environment, the active connection count should be relatively constant. Establish a baseline for your deployment and alert on unusual deviations.
### Client connections
Monitor [`gateway_active_connections_vcluster`](/guide/reference/metric-reference#gateway-reference) for sudden drops (client disconnects) or spikes (connection storms).
```yaml theme={null}
alert: GatewayConnectionsDrop
expr: delta(gateway_active_connections_vcluster[5m]) < -50
for: 2m
labels:
severity: warning
annotations:
summary: "Sudden drop in client connections on vcluster {{ $labels.vcluster }}"
```
```yaml theme={null}
alert: GatewayConnectionsHigh
expr: gateway_active_connections_vcluster > 1000
for: 5m
labels:
severity: warning
annotations:
summary: "High client connection count ({{ $value }}) on vcluster {{ $labels.vcluster }}"
```
### Upstream connections
[`gateway_upstream_connections_upstream_connected`](/guide/reference/metric-reference#gateway-reference) tracks the number of connections from Gateway to the backing Kafka cluster. This should be stable in an established environment. A sudden change could indicate Kafka broker issues or Gateway restarts.
```yaml theme={null}
alert: GatewayUpstreamConnectionsDrop
expr: delta(gateway_upstream_connections_upstream_connected[5m]) < -5
for: 2m
labels:
severity: warning
annotations:
summary: "Upstream connections dropped by {{ $value }} in 5 minutes"
```
[`gateway_upstream_connection_close_rate`](/guide/reference/metric-reference#gateway-reference) tracks how frequently upstream connections are closed. A high close rate relative to creation rate signals connection churn, which degrades performance.
```yaml theme={null}
alert: GatewayUpstreamConnectionChurn
expr: gateway_upstream_connection_close_rate > 5
for: 5m
labels:
severity: warning
annotations:
summary: "High upstream connection churn ({{ $value }} closes/s)"
```
### Authentication failures
Track [`gateway_failed_authentications`](/guide/reference/metric-reference#gateway-reference) to detect clients stuck in authentication loops or brute-force attempts.
```yaml theme={null}
alert: GatewayAuthFailuresHigh
expr: rate(gateway_failed_authentications_total[5m]) > 10
for: 2m
labels:
severity: warning
annotations:
summary: "High authentication failure rate ({{ $value }}/s) for user {{ $labels.user }}"
```
## Throughput
### Overall data flow
[`gateway_bytes_exchanged`](/guide/reference/metric-reference#gateway-reference) tracks the total bytes exchanged between clients and Gateway, tagged by direction (`upstream` from clients toward Kafka, `downstream` from Kafka toward clients). Use this to monitor overall traffic volume and detect anomalies.
```yaml theme={null}
alert: GatewayTrafficDrop
expr: rate(gateway_bytes_exchanged[10m]) < 0.5 * rate(gateway_bytes_exchanged[1h] offset 1h)
for: 5m
labels:
severity: warning
annotations:
summary: "Gateway traffic dropped by more than 50% compared to 1 hour ago"
```
### Per-Virtual Cluster throughput
[`gateway_bytes_exchanged_vcluster`](/guide/reference/metric-reference#gateway-reference) breaks down bytes exchanged per Virtual Cluster. This is useful for:
* Identifying which tenants generate the most traffic
* Capacity planning per Virtual Cluster
* Detecting unexpected traffic spikes from specific tenants
```yaml theme={null}
alert: GatewayVClusterTrafficSpike
expr: rate(gateway_bytes_exchanged_vcluster[5m]) > 2 * rate(gateway_bytes_exchanged_vcluster[1h] offset 1h)
for: 5m
labels:
severity: warning
annotations:
summary: "Traffic spike on vcluster {{ $labels.vcluster }} — {{ $value | humanize }}B/s (2x above baseline)"
```
For per-topic granularity, use [`gateway_bytes_exchanged_topic_total`](/guide/reference/metric-reference#gateway-reference) to identify hot topics that may need partitioning or throttling.
## Latency and performance
### Round-trip latency
[`gateway_latency_request_response`](/guide/reference/metric-reference#gateway-reference) measures the round-trip time from Gateway to Kafka and back. This includes the time for Gateway to send a request to the Kafka broker, receive the response, and process it.
Monitor percentiles (p50, p95, p99) rather than averages. A rising p99 often signals the need to scale before the average shows any degradation.
```yaml theme={null}
alert: GatewayLatencyHigh
expr: histogram_quantile(0.99, rate(gateway_latency_request_response_bucket[5m])) > 0.5
for: 5m
labels:
severity: warning
annotations:
summary: "Gateway p99 latency is {{ $value }}s"
```
For per-operation granularity, use [`gateway_apiKeys_latency_request_response`](/guide/reference/metric-reference#gateway-reference) to break down latency by API key (Produce, Fetch, Metadata, etc.) and isolate which operations are slow.
### Throttling
[`gateway_apiKeys_throttle_ms`](/guide/reference/metric-reference#gateway-reference) reports the `throttleTimeMs` value from Kafka broker responses, broken down by API key. This is the throttle time imposed by Kafka itself (for example, due to quota violations), not by Gateway. Non-zero values mean Kafka is asking clients to back off.
```yaml theme={null}
alert: GatewayKafkaThrottling
expr: rate(gateway_apiKeys_throttle_ms_total[5m]) > 0
for: 5m
labels:
severity: warning
annotations:
summary: "Kafka is throttling {{ $labels.apiKey }} requests on vcluster {{ $labels.vcluster }}"
```
### Inflight requests
[`gateway_current_inflight_apiKeys`](/guide/reference/metric-reference#gateway-reference) tracks the number of requests currently in-flight for each Virtual Cluster, user, and API key combination. It increments when Gateway forwards a request to Kafka and decrements when the response is sent back to the client.
The request pipeline between clients and Kafka is decoupled through an internal buffer. If a request stays in the buffer too long, Gateway expires it and sends a timeout error to the client (tracked by [`gateway_request_expired`](/guide/reference/metric-reference#gateway-reference) below). The buffer capacity is controlled by the [`GATEWAY_NETWORK_MAX_PENDING_REQUESTS`](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/env-variables#low-level-network-configuration) configuration parameter.
Watch this metric alongside latency — a rising inflight count with rising latency suggests Gateway is becoming a bottleneck and may need scaling.
```yaml theme={null}
alert: GatewayInflightRequestsHigh
expr: gateway_current_inflight_apiKeys > 500
for: 5m
labels:
severity: warning
annotations:
summary: "Sustained high inflight requests ({{ $value }}) for {{ $labels.apiKey }} on vcluster {{ $labels.vcluster }}"
```
### Processing backlog
[`gateway_thread_tasks`](/guide/reference/metric-reference#gateway-reference) tracks pending tasks on the Gateway thread where request/response rebuilding happens. A sustained high value indicates a processing bottleneck.
```yaml theme={null}
alert: GatewayProcessingBacklog
expr: gateway_thread_tasks > 100
for: 5m
labels:
severity: warning
annotations:
summary: "Processing backlog of {{ $value }} pending tasks on thread {{ $labels.thread }}"
```
To check load distribution across threads, use [`gateway_thread_request_received`](/guide/reference/metric-reference#gateway-reference) to verify requests are spread evenly.
## Errors and timeouts
### Error rate
[`gateway_error_per_apiKeys`](/guide/reference/metric-reference#gateway-reference) counts processing exceptions per API key for a given Virtual Cluster and user. Alert when the error rate exceeds a percentage of total traffic.
```yaml theme={null}
alert: GatewayHighErrorRate
expr: rate(gateway_error_per_apiKeys_total[5m]) / rate(gateway_current_inflight_apiKeys_total[5m]) > 0.05
for: 5m
labels:
severity: critical
annotations:
summary: "Error rate above 5% for API key {{ $labels.apiKey }} on vcluster {{ $labels.vcluster }}"
```
### Request timeouts
[`gateway_request_expired`](/guide/reference/metric-reference#gateway-reference) counts client requests that timed out waiting for a response from Kafka. Non-zero values usually indicate connectivity problems between Gateway and the Kafka cluster.
```yaml theme={null}
alert: GatewayRequestTimeouts
expr: rate(gateway_request_expired_total[5m]) > 0
for: 2m
labels:
severity: critical
annotations:
summary: "Client requests are timing out waiting for Kafka responses"
```
## Consumer lag
These metrics are populated for topics using [topic concentration](/guide/conduktor-concepts/logical-topics), where multiple logical topics share a single physical Kafka topic. They reflect Gateway-managed consumer groups, not the underlying Kafka offsets. Standard Kafka consumer lag tools cannot disaggregate lag per logical topic in this scenario, so Gateway provides these metrics as the only way to monitor consumer lag per concentrated topic.
* **[`gateway_topic_log_end_offset`](/guide/reference/metric-reference#gateway-reference)** — the latest offset in each partition of a logical topic, representing the most recent message written. Labeled by `vcluster`, `topic`, and `partition`.
* **[`gateway_topic_current_offset`](/guide/reference/metric-reference#gateway-reference)** — the last committed offset for a consumer group on a logical topic, representing how far it has read. Labeled by `vcluster`, `topic`, `partition`, and `group`.
The difference between them is the consumer lag: how many messages a group has yet to process. Because the two metrics have different label sets, use explicit label matching in PromQL:
```yaml theme={null}
alert: GatewayConsumerLagHigh
expr: >
gateway_topic_log_end_offset
- on(vcluster, topic, partition) group_right(group)
gateway_topic_current_offset > 10000
for: 5m
labels:
severity: warning
annotations:
summary: "Consumer lag of {{ $value }} on topic {{ $labels.topic }} for group {{ $labels.group }}"
```
Tune the threshold based on your expected throughput and processing speed.
## Cache health
[`gateway_kcache_size`](/guide/reference/metric-reference#gateway-reference) reflects the number of key-value pairs in Gateway's internal cache, broken down by the `type` label:
| Type | What it tracks |
| ------------------------ | ------------------------- |
| `topic` | Topic mappings |
| `topicConcentrationRule` | Topic concentration rules |
| `offsetTracking` | Offset tracking entries |
| `aclRules` | ACL rules |
| `encryptionCache` | Encryption cache entries |
| `testTokenization` | Test tokenization entries |
Monitor for unexpected growth in any of these types, which could indicate memory pressure or a configuration issue.
```yaml theme={null}
alert: GatewayCacheGrowth
expr: delta(gateway_kcache_size[1h]) > 1000
for: 10m
labels:
severity: warning
annotations:
summary: "Cache type {{ $labels.type }} grew by {{ $value }} entries in the last hour"
```
## Quick reference
| Metric | Category | What to watch for | Suggested alert |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | -------------------------- | -------------------------------- |
| `up{job="conduktor-gateway"}` | Availability | Instance unreachable | `== 0` for 1m |
| [`gateway_license_remaining_days`](/guide/reference/metric-reference#gateway-reference) | Availability | License approaching expiry | `< 14` warning, `< 3` critical |
| [`gateway_upstream_io_nodes`](/guide/reference/metric-reference#gateway-reference) | Availability | Kafka node loss | Below expected cluster size |
| [`gateway_active_connections_vcluster`](/guide/reference/metric-reference#gateway-reference) | Connections | Sudden drops or spikes | `delta < -50` in 5m, or `> 1000` |
| [`gateway_upstream_connections_upstream_connected`](/guide/reference/metric-reference#gateway-reference) | Connections | Connection instability | `delta < -5` in 5m |
| [`gateway_upstream_connection_close_rate`](/guide/reference/metric-reference#gateway-reference) | Connections | Connection churn | `> 5` closes/s |
| [`gateway_failed_authentications`](/guide/reference/metric-reference#gateway-reference) | Connections | Auth loops or brute force | `rate > 10/s` |
| [`gateway_bytes_exchanged`](/guide/reference/metric-reference#gateway-reference) | Throughput | Traffic anomalies | `> 50%` drop vs. baseline |
| [`gateway_bytes_exchanged_vcluster`](/guide/reference/metric-reference#gateway-reference) | Throughput | Per-tenant traffic spikes | `> 2x` above baseline |
| [`gateway_latency_request_response`](/guide/reference/metric-reference#gateway-reference) | Latency | Rising response times | p99 > 500ms |
| [`gateway_apiKeys_throttle_ms`](/guide/reference/metric-reference#gateway-reference) | Latency | Kafka-imposed throttling | `rate > 0` |
| [`gateway_current_inflight_apiKeys`](/guide/reference/metric-reference#gateway-reference) | Latency | Request backlog | `> 500` sustained |
| [`gateway_thread_tasks`](/guide/reference/metric-reference#gateway-reference) | Latency | Processing bottleneck | `> 100` sustained |
| [`gateway_error_per_apiKeys`](/guide/reference/metric-reference#gateway-reference) | Errors | Processing failures | `> 5%` of total traffic |
| [`gateway_request_expired`](/guide/reference/metric-reference#gateway-reference) | Errors | Kafka connectivity | `rate > 0` |
| [`gateway_topic_log_end_offset`](/guide/reference/metric-reference#gateway-reference) - [`gateway_topic_current_offset`](/guide/reference/metric-reference#gateway-reference) | Consumer lag | Growing lag | `> 10000` |
| [`gateway_kcache_size`](/guide/reference/metric-reference#gateway-reference) | Cache | Unexpected growth | `delta > 1000` in 1h |
## Related resources
* [Gateway metric reference](/guide/reference/metric-reference#gateway-reference)
* [Set up monitoring](/guide/conduktor-in-production/monitor)
* [Grafana dashboards in Helm charts](https://github.com/conduktor/conduktor-public-charts/tree/main/charts/gateway/grafana-dashboards)
# Conduktor technical requirements — Console and Gateway
Source: https://docs.conduktor.io/guide/conduktor-in-production/system-requirements
Hardware, software, and dependency requirements for Conduktor Console and Gateway. Covers CPU, RAM, PostgreSQL versions, and supported Kafka providers.
## Min hardware spec
| | Gateway | Console |
| :--------- | :------ | :------ |
| RAM | 4 GB | 3 GB |
| CPU cores | 2 | 2 |
| Disk space | N/A | 5 GB |
## Console requirements
Conduktor Console is provided as a single Docker container.
### Production requirements
For **production deployments**, ensure:
* An external [PostgreSQL (13+)](/guide/conduktor-in-production/deploy-artifacts/deploy-console/#configure-postgres-database) with backups and [high availability](#database-connection-fail-over)
* [Blob storage](/guide/conduktor-in-production/deploy-artifacts/deploy-console/#monitoring-properties) (S3, GCS, Azure) for monitoring
* Proper hardware allocation (see below)
* Kafka version **2.7.0+**
If deploying via [Helm](/guide/conduktor-in-production/deploy-artifacts/deploy-console/kubernetes), check the install guide for detailed requirements.
### Hardware requirements
Use container CGroup limits for resource control.
#### Minimum
* 2 CPU cores
* 3 GB RAM
* 5 GB disk
#### Recommended
* 4+ CPU cores
* 4+ GB RAM
* 10+ GB disk
[Configure env variables](/guide/conduktor-in-production/deploy-artifacts/deploy-console/environment-variables) or [get started with Docker](/guide/get-started).
### Memory configuration
> **Note**: `RUN_MODE` is deprecated. JVM now respects container limits (up to 80%).
```bash theme={null}
-XX:+UseContainerSupport -XX:MaxRAMPercentage=80
```
## Gateway requirements
Conduktor Gateway is provided as a [Docker image](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway) and a [Helm chart](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/kubernetes).
### Minimum setup
For light usage per Gateway node:
* 2 CPU cores
* 4 GB of RAM
> Each Gateway instance should support around *20–30 MB/s* of sustained throughput with minimal latency impact.
### Recommended starting setup
For medium to high usage per Gateway node:
* 4 CPU cores
* 8 GB of RAM
> This setup supports \~40–50 MB/s throughput.
> For production, run **at least 3 Gateway instances**. Scale horizontally as needed.
### Local storage requirements
While Gateway itself doesn't use local storage directly, some interceptors do. The [large message handling](/guide/use-cases/manage-large-messages#local-disk-cache) interceptor maintains a local disk cache that must be sized for the expected payload volume; see its [Local disk cache](/guide/use-cases/manage-large-messages#local-disk-cache) section for sizing and cleanup behavior.
### Scaling Gateway
Gateway is designed to **scale horizontally or vertically**. It's mostly CPU-bound unless caching is enabled. Monitor metrics to scale effectively.
### Interceptor impact
Interceptors add **CPU load**. Interceptors doing heavy inspection (e.g. encryption) require:
* More CPU
* More memory (recommend **4 GB RAM per CPU**)
### Kafka requirements
* Kafka version **2.7.0 or higher**
* Gateway must connect as an **admin user** with rights to:
* Manage topics, groups, and cluster info
* Commit offsets
## Troubleshoot
No, you can run a Docker image in your environment without internet access.
Yes, the external database is used to store configuration such as users and permissions.
# Conduktor quick start — deploy Console with Docker Compose
Source: https://docs.conduktor.io/guide/get-started
Get Conduktor Console running in minutes with Docker Compose. Includes a full Kafka stack (Redpanda, PostgreSQL) for testing features before a production.
Pre-requisite [Docker Compose](https://docs.docker.com/compose/install)
[Get started with Conduktor](https://www.conduktor.io/get-started) in just a few minutes.
This get started bundle builds a **complete Conduktor deployment** consisting of a modern web UI (Console), a PostgreSQL database, a Redpanda cluster and other supporting services.
## Overview
Once deployed, you’ll have access to a fully functional Kafka stack integrated with Conduktor’s UI and backend services. This environment is ideal for trying out Conduktor’s features before deploying to a production environment.
The **data generation service** publishes a continuous stream of synthetic events that *simulate an e-commerce business*.
The generated **data lands in a Redpanda Kafka cluster** with a *schema registry* enforcing structure and compatibility across topics.
Conduktor [Console](/guide/conduktor-in-production/deploy-artifacts/deploy-console) provides a **centralized UI-based control plane** to manage the Redpanda Kafka cluster, allowing you to:
* monitor topics, schemas, consumer groups and connectors
* trace message flows
* audit activity across the entire stack
## Run Conduktor
```bash theme={null}
curl -L https://releases.conduktor.io/quick-start -o docker-compose.yml && docker compose up -d --wait && echo "Conduktor started on http://localhost:8080"
```
## Explore Console
Console runs on `http://localhost:8080`. This walkthrough covers cluster health, topic management, live tail, access workflows and Insights.
## Reset and start fresh
To reset your local Conduktor environment and remove all data:
```bash theme={null}
docker compose down -v
```
This removes all containers and volumes, including the PostgreSQL database. When you restart Conduktor, it will be a fresh installation with default credentials.
## Related resources
* [Arrange a technical demo](https://www.conduktor.io/contact/demo)
* [Conduktor in production deployment overview](/guide/conduktor-in-production)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Conduktor — get started
Source: https://docs.conduktor.io/guide/index
Conduktor guide: deploy Console and Gateway, configure Kafka clusters, set up RBAC and SSO, manage resources, and automate with CLI, API.
Conduktor is the data hub for operational data and AI that integrates with your existing infrastructure, providing centralized visibility and control.
It's a fully self-hosted solution that depends on Docker to run. You can [get started for free](https://www.conduktor.io/get-started) with [Console Community Edition](/guide/conduktor-in-production/manage-licenses/console-community-edition) or [Gateway Community Edition](/guide/conduktor-in-production/manage-licenses/gateway-community-edition).
Conduktor has two products — **Console** and **Gateway**. Each works on its own, or together.
* **Conduktor Console** provides a powerful UI that lets you manage essential Kafka tasks and beyond:
* **Conduktor Gateway** is a vendor-agnostic Kafka proxy that provides control over traffic management, data quality and security, authorization and much more.
## Related resources
* [Get started with Conduktor](/guide/get-started)
* [Arrange a technical demo](https://www.conduktor.io/contact/demo)
* [Contact support](https://www.conduktor.io/contact/support)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
* [Check out release notes](/guide/release-notes/)
* [Conduktor pricing](https://conduktor.io/pricing)
# Kafka Insights dashboard
Source: https://docs.conduktor.io/guide/insights
The Conduktor Insights dashboard identifies Kafka configuration risks, storage costs, governance gaps, and business-critical topics across your clusters.
The Insights dashboard helps platform teams:
* identify configuration issues before they become outages,
* optimize storage costs,
* track governance metrics,
* monitor business-critical topics.
To access the dashboard, click **Insights** on the left menu in Conduktor Console UI.
## Overview
Watch how Insights surfaces risk, cost and governance gaps across your clusters:
Insights covers four key areas:
Identify topics at risk of data loss, poor cluster efficiency or load imbalance.
Find empty, stale or tiny topics.
Track usage of highly utilized topics.
Track schema adoption, serialization formats and Self-service coverage for your VIP topics.
## Access to Insights
Access to the Insight information is managed by the RBAC system in Console. The insight information a user sees is based on the permission to view or edit the configuration for a topic. All insight views are filtered to only show data aggregated across topics where a user has these permissions.
## Metrics summary
At the top of the dashboard you'll see a cluster-wide summary of all the metrics:
* **Topics** - the number of topics in the cluster
* **Partitions** - an aggregate partition count across all topics
* **Consumed** - the total number of times topics are consumed by consumer groups
* **Topic distribution** - a breakdown of topics by type (internal, streams or user)
* **Health score** - the cluster health indicator based on our risk analysis
These metrics update dynamically, based on applied filters.
## Filter data
Use the filter at the top of the Insights dashboard to narrow down data across all sections. Filters apply globally to the summary metrics, risk analysis, cost control and VIP topics sections.
Filter topics by their classification:
* **Internal** - internal topics (e.g., `__consumer_offsets`)
* **Streams** - Kafka Streams internal topics (e.g., changelog, repartition topics)
* **User** - application topics created by you
Filter topics by naming patterns:
* **prefix** - `my-topics*` or just `my-topics` will filter to all topics whose names starts with "my-topics"
* **postfix** - `*my-topics` will filter to all topics whose name ends with "my-topics"
* **contains** - `*my-topics*` will filter to all topics whose name contains "my-topics"
Filter topics by one or more owning Applications (Self-service) to only see the insight data for topics owned by one or a set of applications.
You can also filter topics by labels. Select one or more labels to only see topics with matching labels across all the Insights sections.
Clicking a label in any table also applies it as a filter, making it easy to explore related topics.
Click **Clear all** to remove all applied filters and return to the full cluster view.
## Export data
You can export data as a .zip file containing CSV files. Choose whether to export all available Insights data or only section-specific metrics.
When filters are applied, the export includes only the filtered data and file names include a `-filtered` suffix (e.g., `risk-analysis-filtered.csv`) to indicate the data is a subset of the full cluster.
Share the export with stakeholders, use it for offline analysis, or archive it for audit purposes.
## Related resources
* [Analyze risk ](/guide/insights/risk-analysis)
* [Review cost insights](/guide/insights/cost-control)
* [Check out business-critical topic insights](/guide/insights/vip-topics)
* [Measure data governance maturity](/guide/insights/governance)
* [Configure topic settings](/guide/manage-kafka/kafka-resources/topics)
* [Set up alerts](/guide/monitor-brokers-apps)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Kafka topic cost control
Source: https://docs.conduktor.io/guide/insights/cost-control
Identify Kafka topics consuming unnecessary storage with Conduktor Insights. Surface empty, stale, and tiny topics to reduce costs.
Cost control is one of the sections in the [Insights dashboard](/guide/insights). It helps platform teams identify topics consuming unnecessary storage and make informed cleanup and retention decisions.
**Free ebook:** [Where Kafka costs hide: a field guide](https://www.conduktor.io/resources/ebooks/where-kafka-costs-hide-a-field-guide) Where Kafka spend actually accumulates, and how to find and cut it.
## Overview
The Cost Control section displays three categories of topics that may be wasting storage:
* **Empty topics** - topics with no messages that consume metadata overhead
* **Stale topics** - topics with no writes in a configured time period (default: 7+ days)
* **Tiny topics** - topics with minimal data volume relative to their partition overhead
Each category shows the number of affected topics. Individual topics are listed with idle time in days and partition count.
You can switch the view of data on the page between a graphs view and a table view using the "Graphs" toggle found at the top right of the page.
Before taking action on any topic, verify it is truly unused by checking consumer groups, producer configurations, ownership and documentation.
Deleting a topic is permanent and cannot be undone. All partition metadata and configuration will be lost.
## What the dashboard shows
### Empty topics
Topics containing zero messages. These topics have been created but never received data, or have had all messages deleted through retention policies.
For each topic: name, idle time in days, and partition count.
Empty topics consume cluster resources even without data: metadata storage in ZooKeeper or KRaft, memory for buffers, and often per-partition charges on cloud Kafka services.
Empty topics are common in development and staging environments where topics are created for testing but not cleaned up afterward.
**Good candidates for deletion:**
* topics created for abandoned projects or experiments
* temporary topics used for testing or debugging
* topics created by mistake with typos in the name
* legacy topics from deprecated applications
* topics created automatically by frameworks but never used
**Reasons to keep empty topics:**
* topics pre-created for upcoming features or applications
* topics serving as placeholders for future data pipelines
* topics referenced in consumer group configurations awaiting data
* topics used as dead letter queues that only receive messages in error scenarios
Click the topic name to go to the **Topics** page. Check:
* **Consumer Groups** tab for active consumers or unconsumed data (lag > 0)
* **Monitoring** tab graphs to confirm zero produce activity
* Application references in codebases and deployment configs
* Dependencies - verify no streaming jobs or external systems depend on the topic
In the Cost Control section, click the **...** menu next to the topic and select **Delete topic**.
Type **DELETE** to confirm and click the delete button.
Topic deletion can take up to 5 minutes to be reflected in the dashboard. Consumer groups that were subscribed will need to be updated to remove the topic from their subscriptions.
### Stale topics
Topics with no write activity for **7+ days** and no active consumer groups.
For each topic: name, idle time in days, and partition count.
Stale topics often indicate abandoned data pipelines from shut down applications, failed producers or cancelled projects. They continue consuming storage through retention policies and add operational overhead.
A topic with no writes may still have active consumers reading historical data. Stale write activity does not necessarily mean the topic is unused or safe to delete.
**Investigation checklist:**
1. **Check consumer groups** - Are any consumers actively reading from the topic?
2. **Review consumer lag** - Is there unconsumed data that applications still need?
3. **Verify business requirements** - Is the historical data needed for compliance or analytics?
4. **Contact topic owners** - Do the application teams still need this topic?
5. **Check data lineage** - Do downstream systems depend on this data?
**Topics that are safe to delete:**
* No active consumer groups subscribed
* All consumers show zero lag (all data has been consumed)
* Application owners confirm the topic is no longer needed
* No compliance requirements for data retention
* Historical data has been archived to long-term storage
**Topics to keep:**
* Active consumers are still reading historical data
* Data is required for compliance or audit purposes
* Topic serves as a source of truth for event sourcing
* Downstream analytics or reporting systems depend on the data
* Occasional writes expected (seasonal or infrequent events)
Deleting a topic with unconsumed messages will result in permanent data loss for consumers that have not yet processed those messages.
For topics containing valuable historical data, export and archive the data before deletion.
Navigate to the topic, click the **Consume** tab, configure the consumer to read from **Beginning** offset, and export data to a file or external storage.
For large topics, use a dedicated archival tool or Kafka Connect to export data to object storage (S3, Azure Blob Storage) or a data lake.
Confirm all messages were exported successfully, validate the data format and completeness, and store in long-term storage with documentation of location and format.
Maintain a record of archived topics including the archive location, date, original message count and retention period for the archived data.
Follow the deletion steps above.
If a topic might receive future writes but historical data is no longer needed, reduce the retention period instead of deleting.
Navigate to **Topics**, select the topic, click **Configuration**, and edit:
* `retention.ms` - Time-based retention (e.g., `86400000` for 1 day)
* `retention.bytes` - Size-based retention per partition (e.g., `1073741824` for 1 GB)
Kafka automatically deletes segments older than the retention period during its next cleanup cycle (typically runs every few minutes based on broker configuration).
Monitor the topic size in the **Overview** tab to verify that old data is being cleaned up as expected.
### Tiny topics
Topics meeting all of the following: more than 1 partition, less than 10 MB of data OR fewer than 1000 messages, and name does not end with `_repartition`.
For each topic: name, idle time in days, and partition count.
Topics with many partitions but little data represent inefficient resource allocation. Cloud Kafka services often charge per partition, and high partition counts increase controller overhead.
A common cause is using default partition counts (often 6-12) for topics that only receive occasional messages, such as configuration topics or low-volume event streams.
**How to evaluate:**
Keep the topic as-is if it expects significant growth, requires high parallelism, is part of a standardized policy or would require complex migration. Consider optimization if the topic has been tiny for months without growth, has arbitrary partition counts or handles low-frequency data without parallelism requirements.
Since Kafka does not support reducing partition count, you have three options:
If you have multiple tiny topics serving similar purposes, consolidate them into a single topic with categorization through message headers or keys. This reduces total partition count while maintaining logical separation.
**Considerations:**
* Identify candidates from the same application with similar schemas
* Design consolidated topic structure with category or type fields
* Update producers and consumers accordingly
Topic consolidation changes the partition key distribution and may affect message ordering. Ensure consumers can handle messages from multiple original topics in a single stream.
Determine the optimal partition count:
* **1 partition** - Low-volume topics where ordering is critical
* **3 partitions** - Topics needing some parallelism and broker distribution
* **Match consumer count** - For parallelism requirements
For topics receiving fewer than 1000 messages per day, a single partition is often sufficient unless you have specific parallelism requirements.
If historical data must be preserved, export all messages from the topic using the Consume tab and store in a file or external system.
Navigate to **Topics** and click **Create topic**. Use the same topic name with a suffix (for example, `topic-name-v2`), set the optimized partition count and copy all relevant configurations from the original topic.
Create the new topic with a different name initially to allow for parallel operation during migration.
Update producer and consumer configurations to use the new topic name. Deploy consumer updates first, then producers, to ensure no messages are lost during the transition.
If historical data must be available in the new topic, use producer applications or data migration tools to restore the archived data.
After verifying the new topic is working correctly, monitor for a grace period (24-48 hours), confirm no applications are producing to the old topic, then delete the tiny topic.
Verify consumer lag on the new topic is being processed normally and no errors are occurring in producer or consumer logs.
For some topics, optimization effort may not justify savings:
* System internals (`__consumer_offsets` has 50 partitions by default)
* Framework defaults (Kafka Streams state stores)
* Topics where migration risk outweighs storage savings
## Our recommendations
Prevent cost control issues through governance and automated ownership tracking rather than manual remediation.
**Use Self-service for ownership and governance** - it brings a GitOps approach to topic lifecycle management where applications represent streaming apps or data pipelines and dictate ownership of Kafka resources.
Benefits for cost control:
* **Automatic ownership tracking** - topics created through Self-service have clear ownership from creation
* **Policy enforcement** - application instance policies can prevent creation of tiny topics by enforcing minimum data volume or maximum partition count requirements
* **Naming conventions** - policies enforce consistent naming that includes environment and ownership information
* **Lifecycle management** - clear application ownership reduces abandoned topics through better tracking and accountability
[Learn about Self-service](/guide/use-cases/self-service) | [Self-service concepts](/guide/conduktor-concepts/self-service)
**Configure RBAC** - Use role-based access control to enforce naming conventions, limit partition counts and require labeling at topic creation. [Learn more about RBAC](/guide/conduktor-in-production/admin/set-up-rbac)
**Implement cleanup policies** - Document organization-wide policies for retention periods, deletion criteria and approval processes. Schedule regular reviews of cost control reports (weekly for new topics, monthly for trends).
## Troubleshoot
Producer health checks, monitoring systems or automated tests may send periodic messages that are immediately consumed and deleted. Kafka's log cleaner activity can also appear as writes in some monitoring systems.
Check the **Messages in per second** graph over 7+ days to distinguish sustained activity from sporadic checks. Review the **Consume** tab to see actual message content.
No, topic deletion is permanent. All partition data and topic metadata are removed. Consumer group offsets are retained but the topic cannot be recreated with the same history.
Always archive important data before deletion. Implement a grace period policy where topics are first marked for deletion (using labels) before actual removal. To recreate a deleted topic, create a new topic with the same name in Console and restore data from archives if available.
Kafka creates internal topics automatically: `__consumer_offsets` (consumer group offsets), `__transaction_state` (transactional state), `_schemas` (Schema Registry metadata) and topics matching `.*-changelog` or `.*-repartition` (Kafka Streams internals).
Never delete `__consumer_offsets` or `__transaction_state` as this causes cluster-wide failures. Only delete Kafka Streams topics when the application is permanently decommissioned and all instances are shut down.
Kafka's log cleanup process runs periodically (default: every 5 minutes). Data is deleted at segment boundaries, not individual messages, and the current active segment is never deleted even if older than retention.
Wait 15-30 minutes after reducing retention, then check topic size in the **Overview** tab.
Check topic naming patterns, review consumer groups in the **Consumer Groups** tab, search code repositories and deployment configs for references, and consult internal documentation.
If ownership remains unclear, add a `scheduled-for-deletion` label with a future date (for example, 30 days) and send announcements asking owners to identify themselves. After the grace period expires with no claims, proceed with deletion.
Implement a policy requiring application labels at topic creation time. Use [RBAC permissions](/guide/conduktor-in-production/admin/set-up-rbac) to enforce labeling as a prerequisite for topic creation.
This is normal for seasonal data (holiday sales, tax season), emergency systems (disaster recovery), compliance events (audit logs) or legacy integrations (backup systems).
Verify the producer is authorized, add a `usage-pattern: seasonal` or `usage-pattern: emergency` label, and document the activity pattern. Do not delete topics showing this pattern without understanding the business requirement.
## Related resources
* [View Insights overview](/guide/insights)
* [Configure and manage topics](/guide/manage-kafka/kafka-resources/topics)
* [Set up monitoring and alerts](/guide/monitor-brokers-apps)
* [Monitor and manage brokers](/guide/manage-kafka/kafka-resources/brokers)
* [Set up RBAC](/guide/conduktor-in-production/admin/set-up-rbac)
* [Learn about Self-service topic management](/guide/conduktor-concepts/self-service)
* [Give us feedback or request a feature](https://conduktor.io/roadmap)
# Governance for VIP
Source: https://docs.conduktor.io/guide/insights/governance
Governance is one of the sections in the [VIP topics dashboard](/guide/insights/vip-topics).
It helps platform teams measure and improve data governance practices across the organization by tracking schema adoption, serialization format usage and Self-service workflow coverage for the most critical topics.
Strong governance reduces operational risk and improves data quality. Use these metrics to track adoption progress.
**From our blog:** [No Kafka data platform without ownership](https://www.conduktor.io/blog/no-kafka-data-platform-without-ownership) Why ownership is the foundation of a governed Kafka platform: accountability, autonomy, scale.
## Overview
The governance section displays three graphs that measure how well your critical Kafka infrastructure is governed:
* **Topics with schema**: the percentage and count of VIP topics using schema registry for data contracts
* **Serialization formats**: the distribution of serialization formats in use across VIP topics
* **Self-service coverage**: the percentage of VIP topics managed through governed Self-service workflows
Each graph shows where governance is strong and where gaps remain.
## What the graphs show
### Topics with schema
A schema registry enforces data contracts between producers and consumers. Schemas validate data at produce time, rejecting invalid messages before they enter topics. Compatibility modes (backward, forward, full) allow schema evolution without breaking existing consumers.
Topics without schemas lack the enforcement of a data contract. Producers can send any format, causing consumer failures and data quality issues.
The **topics with schema** graph shows the percentage of topics that have schemas registered in schema registry:
* **Percentage of VIP topics with schemas** - [VIP topics](/guide/insights/vip-topics) that have registered schemas and enforce data contracts
* **Percentage of all topics with schemas** - all topics that have registered schemas and enforce data contracts
Click **Show full list** to view detailed information about topics with and without schemas. The table displays topic names with their topic type label (internal, streams, or user) and any custom labels. Use the search box to filter topics by name, and click column headers to sort.
Clicking a topic type or label in the table applies it as a global filter across all Insights sections.
#### How to interpret schema coverage
**High (80-100%)**: strong governance with defined data contracts, controlled evolution and established standards
**Medium (50-79%)**: partial adoption with gaps in coverage across teams or topic categories
**Low (\<50%)**: significant gaps with high risk of breaking changes and data quality issues
Prioritize schema adoption for [VIP topics](/guide/insights/vip-topics) first and test in lower environments before deploying to production.
1. **Register schema in Console** - go to **Schema Registry** and click **New Subject**. Provide schema definition (Avro, Protobuf or JSON Schema), strategy, and other required settings.
2. **Update producers** - configure Schema Registry URL and use appropriate serializers (KafkaAvroSerializer, KafkaProtobufSerializer or KafkaJsonSchemaSerializer) in producer applications.
3. **Verify usage** - check the topic's **Overview** tab to confirm schema association.
Data is now validated at produce time. Invalid messages are rejected before entering the topic.
**Alternative: auto-registration** - producers can automatically register schemas on first produce. Requires schema registry URL configuration and `create` permissions in schema registry.
[Configure and use schema registry](/guide/manage-kafka/kafka-resources/schema-registry)
### Serialization formats
Using multiple serialization formats increases complexity — each requires different serializers, deserializers, tooling and expertise. Standardizing on one format (or two at most) simplifies operations and makes governance easier to enforce.
The **serialization formats** graph shows the distribution of serialization formats across topics that have schemas registered in schema registry:
* **Avro** - compact binary format with rich schema features
* **Protobuf** - protocol Buffers format with cross-language support
* **JSON schema** - JSON with schema validation and documentation
The graph shows what percentage of schema-registered topics use each format.
Topics without schemas or using plain JSON/String formats are not shown in this graph.
#### How to interpret format distribution
**Standardized** - one dominant format represents 90%+ of schema-registered topics, indicating strong format consistency and simplified operations
**Mixed** - multiple formats each represent significant percentages, suggesting inconsistent practices across teams or ongoing migration efforts
Having multiple formats isn't necessarily bad but it increases operational complexity.
Consider standardizing on one primary format unless there are strong technical reasons to support multiple formats.
Conduktor enforces serialization formats using Rules (validation logic) attached to Policies (applied to topics with actions).
1. **Create validation Rules** - go to **Rules** under the Trust section and click **+New Rule**. Choose the appropriate rule type:
* **EnforceAvro** (built-in) - Ensures messages have a schema ID, the ID exists in Schema Registry, and the schema type is Avro
* **JSON schema** - Validates JSON messages against a JSON schema definition with required fields and structure
Rules define validation logic but do nothing on their own until attached to a Policy.
2. **Create and configure Policy** - go to **Policies** under the Trust section and click **+New Policy**. Attach your Rules, select target topics (specific topics or prefixes like `production-*`), and assign to a user group with "Manage data quality" permission.
Enable Policy actions based on your governance requirements:
* **Report** - log violations in Policy history for monitoring
* **Block** - reject non-compliant messages entirely
* **Mark** - add violation header for downstream handling
Block action prevents message delivery. Communicate format requirements and provide migration support before enabling.
3. **Monitor enforcement** - track violations in Policy detail pages and support teams adopting required formats through Schema Registry integration.
[Enforce data quality with Policies and Rules](/guide/use-cases/enforce-data-quality).
### Self-service coverage
Self-service workflows enforce organizational standards at topic creation time through templates that mandate naming conventions, required configurations (replication factor, partition count), schema requirements and ownership labels.
This provides an audit trail, enables approval workflows and reduces ad-hoc creation that bypasses governance.
The **Self-service coverage** graph shows the percentage of topics managed through governed self-service workflows:
* **VIP topic Self-service coverage** - percentage of business-critical topics under Self-service governance
* **Overall Self-service coverage** - percentage of all topics created and managed via Self-service
Click **Show full list** to view topics and their Self-service status. The table displays topic names with their topic type label (internal, streams, or user) and any custom labels. Use the search box to filter topics by name, and click column headers to sort.
#### How to interpret Self-service coverage
**High (80-100%)** - strong adoption with most topics created through proper governance channels
**Medium (50-79%)** - partial adoption, possibly indicating rollout in progress or legacy topics predating self-service
**Low (\<50%)** - limited adoption requiring establishment and promotion of workflows
Prioritize Self-service coverage for [VIP topics](/guide/insights/vip-topics) first. Business-critical topics benefit most from governed creation and change management.
Self-service uses a GitOps approach where platform teams define applications and policies using YAML resources managed through the Conduktor CLI. Application teams then create and manage their own Kafka resources within defined boundaries.
1. **Define applications and instances** - create Application resources representing streaming apps or data pipelines, and ApplicationInstance resources linking each application to specific Kafka clusters with service accounts.
2. **Establish topic policies** - define TopicPolicy resources that enforce standards for resource creation: replication factors, partition limits, retention settings, schema requirements and naming conventions.
Create multiple policies for different environments to balance governance with flexibility.
3. **Enable application team autonomy** - Application teams use the Conduktor CLI with their application context to create Topics, Subjects, Connectors and ApplicationInstancePermissions following defined policies.
Console provides read-only catalog views in Application Catalog and Topic Catalog pages for discoverability.
Teams can now manage their own Kafka resources within governance boundaries and collaborate through permission grants.
**Bring existing topics under self-service** - review unmanaged topics in the governance graph, contact owners to explain ownership benefits, verify compliance with standards and import topics into application instances using Console's import functionality.
## Troubleshoot
**Process:**
1. Design Avro schema matching JSON structure and register in Schema Registry
2. Update producers to use KafkaAvroSerializer with Schema Registry URL
3. Update consumers to use KafkaAvroDeserializer with Schema Registry URL
**Migration approaches:**
* **Dual-format (recommended)** - Create new Avro topic, write to both topics temporarily, migrate consumers, then decommission old topic
* **Big-bang** - Schedule downtime, deploy all updates simultaneously with rollback plan ready
Avro messages can't be read by plain JSON consumers. Coordinate carefully and test thoroughly in non-production environments first.
**Address common objections:**
* Demonstrate self-service is faster than manual ticketing
* Show how templates save time with proven configurations
* Explain governance benefits (reduced errors, better visibility)
* Streamline approval processes to minimize wait times
**Make adoption easier:**
* Provide excellent documentation and training
* Create templates for common use cases
* Assign self-service champions within teams
* Provide dedicated support during initial adoption
No, 100% coverage is not necessary. Prioritize schemas for:
* Production topics with business data
* Topics with multiple consumers or teams
* [VIP topics](/guide/insights/vip-topics) with high consumer counts
* Topics requiring data evolution and compatibility
**Topics that may not need schemas:**
* Configuration or control topics
* Internal framework topics (Kafka Streams, Kafka Connect)
* Development or testing topics
**Recommended targets:** 90-100% of production business data topics, 80%+ overall coverage
Set realistic targets based on organizational maturity. Focus on production and business-critical topics while allowing pragmatic exceptions.
**Highest priority:**
* [VIP topics](/guide/insights/vip-topics) without schemas (many consumers, breaking changes affect multiple teams)
* Topics with frequent schema evolution (high risk without compatibility enforcement)
* Topics with multiple producing teams (schema enforces consistency)
**Medium priority:**
* High-throughput topics (debugging costs increase at scale)
* Topics in critical data pipelines (quality issues cascade to business decisions)
**Lower priority:**
* Low-volume internal topics with single consumers
**Approach**: export governance data as a .zip file containing CSV files, cross-reference with VIP topics graph, review recent data quality incidents and create prioritized migration plan.
Use Insights to identify topics that are both VIP and lack schemas. These represent the highest-priority governance gaps.
## Related resources
* [View Insights overview](/guide/insights)
* [Configure and manage topics](/guide/manage-kafka/kafka-resources/topics)
* [Set up monitoring and alerts](/guide/monitor-brokers-apps)
* [Monitor and manage brokers](/guide/manage-kafka/kafka-resources/brokers)
* [Set up RBAC](/guide/conduktor-in-production/admin/set-up-rbac)
* [Learn about Self-service topic management](/guide/conduktor-concepts/self-service)
* [Configure and use Schema Registry](/guide/manage-kafka/kafka-resources/schema-registry)
* [Configure data quality policies to enforce standards](/guide/use-cases/enforce-data-quality)
* [Give us feedback or request a feature](https://conduktor.io/roadmap)
# Kafka configuration risk analysis
Source: https://docs.conduktor.io/guide/insights/risk-analysis
Identify Kafka topic configuration risks that impact reliability. Conduktor Insights flags replication, partition, and retention problems before outages.
Risk analysis is one of the sections in the [Insights dashboard](/guide/insights).
It identifies **topics with configuration issues** that could impact your Kafka cluster's reliability and performance.
## Overview
Risk analysis monitors three critical aspects of topic configuration:
* **Data loss risk**: topics with insufficient data redundancy based on replication factor (RF) and min in-sync replicas
* **Cluster efficiency**: topics with sub-optimal partition allocation across brokers
* **Load imbalance risk**: topics with uneven data distribution across partitions
Each graph uses color coding to indicate severity level:
* **Red**: high risk requiring immediate attention
* **Orange/Yellow**: medium risk that should be addressed
* **Green**: low risk or healthy configuration
Below the graphs, a **Topic health overview** table shows topics with recommendations, displaying key metrics and warning indicators to help you prioritize remediation.
You can switch the view of data on the page between a graphs view and a table view using the "Graphs" toggle found at the top right of the page.
## What the graphs show
### Data loss risk
The data loss risk graph displays topics categorized by their vulnerability to data loss based on a combination of the replication factor (RF) and min in-sync replicas configuration.
Topics are classified into three risk levels:
* **High risk (red)**: topics with RF = 1 (no data redundancy). If the broker hosting the topic fails, all data becomes unavailable permanently.
* **Medium risk (orange)**: topics with RF = 2, or topics with RF > 2 but min.insync.replicas \< RF-1. Limited fault tolerance or insufficient write durability guarantees.
* **Low risk (green)**: topics with RF ≥ 3 and min.insync.replicas = RF-1. Provides adequate fault tolerance and write durability for production environments.
**RF = 3** with **min.insync.replicas = 2** is recommended for production environments. This provides the right balance between data safety and storage overhead, tolerating one broker failure without data loss while ensuring writes are acknowledged by multiple replicas.
#### Resolve data loss risk
Kafka does not allow changing the replication factor of an existing topic through configuration updates.
You have to use partition reassignment to add replicas or recreate the topic with the required replication factor.
**Identify the topic with low replication factor:**
1. **Go to the topic** - go to **Topics** from the main menu and select the topic shown in the graph.
2. **Review current configuration** - click the **Configuration** tab and note the replication factor shown at the top of the page to confirm the current value.
Replication factor is a topic-level setting that can't be changed after topic creation through normal configuration updates.
Partition reassignment allows you to add replicas to existing topics without recreating them. This operation requires Kafka administrative tools external to Console:
1. **Document current partition assignment** - in Console, go to the topic and click the **Partitions** tab. Document the current replica assignments for all partitions.
2. **Perform partition reassignment** - use Kafka administrative tools (such as `kafka-reassign-partitions`) to add additional replicas to the topic. This process replicates data across additional brokers in the background.
Partition reassignment requires creating a JSON file specifying new replica assignments and executing the reassignment using Kafka CLI tools. Set throttling limits to avoid impacting cluster performance during the operation.
3. **Verify completion in Console** - return to the **Partitions** tab in Console and verify all partitions now show the increased replication factor.
Works without downtime and preserves existing data. Best for production environments and topics with significant data.
For smaller topics or non-production environments, recreating the topic with the correct replication factor may be simpler:
1. **Create a new topic with RF = 3** - in Console, go to **Topics** and click **Create topic**. Set the replication factor to 3.
2. **Migrate data** - use Kafka Connect, Kafka Streams or a migration tool to copy data from the old topic to the new topic.
3. **Update applications** - update producer and consumer configurations to use the new topic name.
4. **Verify and clean up** - after confirming the migration is successful, delete the old topic.
Simpler approach for small topics or non-production environments. Requires data migration and application updates.
Set `default.replication.factor=3` in broker configuration and configure `min.insync.replicas=2` to ensure writes are acknowledged by at least two replicas.
Use [RBAC permissions](/guide/conduktor-in-production/admin/set-up-rbac) to prevent users from creating topics with RF \< 3.
### Cluster efficiency
The cluster efficiency graph displays topics categorized by partition allocation patterns that affect broker load distribution and resource utilization.
Topics are classified into two categories:
* **Ideal (green)**: topics with fewer partitions than the number of brokers in the cluster, or topics with partition counts which are a multiple of the number of brokers. These partition counts distribute evenly across typical broker configurations.
* **Poor (orange)**: topics with more partitions than there are brokers, and where the partition count is not divisible by the broker count. These configurations may create uneven distribution across brokers, leading to hotspots and suboptimal resource utilization.
Optimal partition distribution spreads partitions evenly across all brokers with balanced leadership. Ideal partition counts facilitate even distribution in your Kafka clusters.
#### Resolve cluster efficiency issues
**Analyze current distribution:**
1. **Go to the topic** - go to **Topics** and select the affected topic.
2. **Review partition distribution** - click the **Partitions** tab and examine the distribution across brokers.
3. **Switch views** - toggle between **Per partition** and **Per broker** views to understand the distribution pattern.
**Per broker view shows:**
* Which brokers lead which partitions
* Which brokers hold follower replicas
* Imbalances in partition leadership
4. **Identify rebalancing needs** - look for:
* Brokers with significantly more leader partitions than others
* Brokers with no partitions for critical topics
* Uneven distribution patterns that could cause hotspots
Use Kafka administrative tools to trigger preferred leader election, which reassigns leadership to each partition's preferred leader without moving data.
This lightweight operation is safe for production and should be run regularly.
Preferred leader election only changes which broker is the leader for each partition. It does not move data or change replica assignments.
For significant distribution issues, use Kafka administrative tools to reassign partition replicas.
Generate a reassignment plan that spreads partitions evenly across brokers, then execute with throttling to limit impact. Monitor progress until completion.
Partition reassignment moves data across brokers. Use throttling to limit replication bandwidth and schedule reassignments during low-traffic periods.
Add partitions directly in Console to improve distribution:
Choose partition counts that are multiples of your broker count (for example, 12 partitions for 3 brokers = 4 per broker). Monitor broker-level metrics in the [Monitoring](/guide/monitor-brokers-apps) dashboard to track distribution.
1. **Go to Topics in Console** - go to **Topics** and select the topic you want to modify.
2. **Open the partition menu** - click the **+ Add partitions** button.
3. **Enter new partition count** - specify the new total partition count. Choose a number that's a multiple of your broker count for optimal distribution.
4. **Confirm the operation** - review the changes and confirm. New partitions will be created and automatically distributed across available brokers.
Adding partitions changes partition assignment for keyed messages. New messages with the same key may be assigned to different partitions, impacting consumers that rely on partition-level ordering.
### Load imbalance risk
The load imbalance risk graph displays topics with uneven data distribution across partitions, measured by a partition skew ratio, shown as a percentage.
High partition skew causes performance problems (hot partitions, consumer lag), resource inefficiency (wasted parallelism, uneven disk usage) and may indicate poor partition key selection or producer misconfiguration.
#### How skew is calculated
Console calculates partition skew as the difference between the largest and smallest partition sizes, divided by the average partition size, then divided again by the partition count so topics with different partition counts can be compared. The formula is:
```
partition_skew = (largest_partition_size - smallest_partition_size) / average_partition_size / number_of_partitions * 100%
```
The value is between 0 and 100%. A skew of 0% means all partitions are the same size (no skew), while a skew of 100% means all records are in one partition and the rest are empty (maximum skew).
Topics are categorized by skew percentage:
* **\< 25% (green)**: slight imbalance with acceptable variation in partition sizes
* **25% - 75% (orange)**: moderate imbalance that should be monitored and investigated
* **> 75% (red)**: severe imbalance requiring immediate attention to prevent performance issues
#### Possible causes
**Poor partition key selection** - keys with uneven distribution, too few unique keys, or clustering around certain values.
**Producer configuration issues** - manual partition assignment, custom partitioner with flawed logic, or null keys.
**Data model problems** - business logic creating natural hotspots, temporal patterns, or geographic clustering.
#### Resolve load imbalance
**Diagnose the skew:**
1. **Go to the topic** - go to **Topics** and select the affected topic.
2. **Review partition details** - click the **Partitions** tab and select the **Per partition** view.
3. **Identify imbalanced partitions** - compare the following columns across all partitions:
* **Total number of records** - Shows message count per partition
* **Partition size** - Shows disk space consumed
* **Begin offset** and **End offset** - Shows the range of messages
Look for partitions with significantly higher values than others.
4. **Document the pattern** - note which partitions are oversized and by how much. This will help identify the root cause.
**Analyze message keys:**
1. **Go to the Consume tab** - click the **Consume** tab for the topic.
2. **Configure consumer settings** - configure the consumer to read from **All partitions** to see the full data distribution.
3. **Review message keys** - examine the keys in the consumed data. Look for patterns:
* Are certain keys appearing far more frequently than others?
* Are many messages using null keys?
* Is there visible clustering in key values?
4. **Filter by partition** - use the partition filter to consume from specific partitions (especially the largest and smallest) to compare key distributions.
Kafka can't rebalance existing data in partitions. Focus on preventing future skew:
Choose keys with high cardinality and even distribution:
**Good choices:**
* User ID, Order ID, Transaction ID, Device ID
* Composite keys like `${region}-${customerId}`
* Any identifier with naturally even distribution
**Poor choices:**
* Status fields (limited values)
* Boolean values (only two values)
* Small enums (limited set of values)
* Dates without time component
* Null keys
Changing the partition key requires updating producer applications. Coordinate with your development team to implement the new key strategy.
Recommended approach. Choose better partition keys with high cardinality and even distribution to prevent future skew.
Add more partitions to spread the load. In Console, go to **Topics**, select the skewed topic, and click **+ Add partitions**.
Increase the partition count substantially (for example, 3x current count) to dilute the effect of skewed keys.
Adding partitions improves distribution for new messages only. Existing skewed data remains in its original partitions until it expires based on retention settings.
Adding partitions changes partition assignment for keyed messages. New messages with the same key may be assigned to different partitions, impacting consumers that rely on partition-level ordering.
Dilute the effect of skewed keys by adding more partitions. Improves distribution for new messages only.
For severe skew issues, create a replacement topic with more partitions and corrected partition key strategy.
Migrate producers and consumers to the new topic, then delete the old topic after verification.
1. **Create replacement topic** - in Console, create a new topic with:
* Substantially more partitions (2-3x current count)
* Same replication factor and retention settings
* Descriptive name indicating it's the replacement
2. **Update producers** - modify producer applications to:
* Use the new partition key strategy
* Write to the new topic
* Deploy and verify message production
3. **Migrate consumers** - update consumer applications to:
* Read from the new topic
* Handle any schema or key changes
* Deploy and verify consumption
4. **Verify and clean up** - after confirming successful migration:
* Monitor the new topic for skew in risk analysis
* Wait for retention period to expire on old topic
* Delete the old topic when no longer needed
Set up [alerts](/guide/monitor-brokers-apps/alerts) for partition size differences and review the Risk Analysis dashboard regularly. Monitor consumer lag by partition to identify performance impacts.
## Table view
The topic health overview table, seen when you toggle graphs off with the control at the top of the page, provides a detailed view of topics that need attention. It displays key configuration metrics and warning indicators.
* **Topic**: topic name with topic type label (internal, streams, or user) and any custom labels
* **Min in-sync replicas**: minimum number of replicas that have to acknowledge writes
* **Partitions**: total partition count for the topic
* **Avg. msg size**: average message size in bytes
* **Replication factor**: number of replicas for each partition
* **Skew**: partition imbalance percentage showing data distribution — see [how skew is calculated](#how-skew-is-calculated)
### Sort and search
Use the search box to filter topics by name. Click any column header to sort the table by that column in ascending or descending order.
Clicking a topic type or label in the table applies it as a global filter across all Insights sections.
### Warning indicators
Topics display warning indicators when they meet the following thresholds:
* **Average message size > 600KB** - Large messages that may cause performance issues
* **Replication factor \< 3** - Insufficient redundancy for production environments
* **Skew > 75%** with at least 1000 messages per partition - Severe partition imbalance requiring attention
Use this table to identify which topics need immediate remediation and understand the specific configuration issues affecting each topic.
## Troubleshoot
Several factors can cause imbalance even with well-designed partition keys:
* **Time-based patterns**: Temporal clustering (business hours vs. night) creates natural imbalance based on when data was produced
* **Compaction**: Log compacted topics retain more messages in partitions with higher key diversity
* **Retention**: Uneven produce rates over time mean partitions contain data from different periods
* **Producer failures**: Restarts or errors may temporarily cluster messages on specific partitions
* **Natural data distribution**: Some business scenarios naturally create imbalance (one customer generating 80% of orders)
Analyze imbalance trends over time (days or weeks) rather than point-in-time snapshots. If imbalance is transient and self-correcting, monitor but don't take action. Use retention-based cleanup to eventually age out historical imbalanced data.
Yes, use partition reassignment to add replicas to existing topics through Kafka administrative tools:
1. View current replica assignments in Console's **Partitions** tab
2. Create a reassignment plan specifying new replica assignments with additional broker IDs
3. Execute the reassignment using Kafka CLI tools (data replicates in the background)
4. Monitor progress and verify completion in Console
This approach increases replication without downtime or data loss.
**Key considerations:**
* **Throughput**: More partitions = more parallelism and higher potential throughput
* **Consumer count**: You need at least as many partitions as consumers for full parallelism
* **Broker count**: Choose a partition count that's a multiple of broker count for even distribution
* **Message ordering**: Ordering is only guaranteed within a single partition
* **Overhead**: Each partition adds metadata overhead. Tens of thousands of partitions can cause performance issues
**General guideline**: Use 4 or fewer partitions for low-volume topics. For topics requiring more partitions, use multiples of 6 (6, 12, 18, 24, etc.).
**Examples:**
* Low-volume topic: 1-4 partitions
* Standard topic: 6 or 12 partitions
* High-throughput topic: 18, 24, 30 or more partitions (multiples of 6)
This is normal and expected for newly created topics. Initial messages create imbalance as partitions receive different amounts of data before distribution stabilizes.
**Expected behavior:**
* First 100-1000 messages: High imbalance is normal
* After 1000+ messages: Imbalance should normalize if partition keys are well-distributed
* After 24-48 hours: Imbalance percentages should stabilize
Monitor load imbalance over 24-48 hours or after at least 10,000 messages before taking corrective action. If imbalance remains high (> 75%) after this period, investigate partition key selection.
Partition reassignment impacts network traffic, disk I/O, client latency, and broker CPU. Duration depends on data volume: small topics (\< 1 GB) complete in minutes; large topics (> 1 TB) can take hours or days.
**Best practices to minimize impact:**
* Schedule during low-traffic periods
* Use throttling to prevent saturating network bandwidth
* Monitor cluster metrics (CPU, disk I/O, network throughput, client latency) using Console
* For very large topics, reassign partitions in batches
* Adjust throttle dynamically based on traffic patterns
* Remove throttle after completion
Always set a throttle value when performing partition reassignment in production. Un-throttled reassignment can impact client operations and cause outages.
These are related but different conditions:
**Under-replicated partitions:**
* A partition has fewer in-sync replicas than its configured replication factor
* Caused by broker failures, network issues or replicas falling behind
* Reduces fault tolerance but doesn't immediately block writes
* Example: Topic with RF=3 but only 2 replicas are in-sync
**Under min ISR partitions:**
* The number of in-sync replicas falls below the `min.insync.replicas` setting
* Blocks producers configured with `acks=all` from writing
* More severe condition indicating immediate data durability risk
* Example: `min.insync.replicas=2` but only 1 replica is in-sync
A partition can be under-replicated without being under min ISR (if enough replicas remain in-sync to meet the minimum). However, under min ISR partitions are always also under-replicated.
## Related resources
* [View Insights overview](/guide/insights)
* [Configure and manage topics](/guide/manage-kafka/kafka-resources/topics)
* [Set up monitoring and alerts](/guide/monitor-brokers-apps)
* [Monitor and manage brokers](/guide/manage-kafka/kafka-resources/brokers)
* [Set up RBAC](/guide/conduktor-in-production/admin/set-up-rbac)
* [Learn about Self-service topic management](/guide/conduktor-concepts/self-service)
* [Give us feedback or request a feature](https://conduktor.io/roadmap)
# Monitor VIP Kafka topics
Source: https://docs.conduktor.io/guide/insights/vip-topics
Identify business-critical Kafka topics in Conduktor Insights VIP Topics. Surface your highest-traffic topics by consumer group count and message volume.
VIP topics is one of the sections in the [Insights dashboard](/guide/insights). It helps you focus attention and resources on the most utilized topics for maximum impact.
## Overview
The VIP topics section displays a health overview graph showing topics identified as important to your infrastructure. VIP topics are determined by two key metrics:
* **Consumer group count** - topics with many subscribing consumer groups (shown by bar height)
* **Message volume** - topics with high throughput and data volume (shown by color intensity)
The combination of these metrics identifies topics that are both widely used and heavily trafficked, indicating business criticality. Topics with many consumers and high message volumes represent critical data pipelines that multiple applications depend on, making them prime candidates for elevated monitoring and careful management.
High traffic makes VIP topics susceptible to performance problems, while configuration issues create data loss risk. Changes require coordination across teams and careful testing.
Issues with VIP topics have multiplied impact. A single misconfiguration or outage can affect dozens of applications and business processes simultaneously.
VIP topics are automatically identified based on cluster-wide analysis of consumer patterns and message throughput. Use this data-driven approach to prioritize operational focus and monitoring resources.
You can switch the view of data on the page between a graphs view and a table view using the "Graphs" toggle found at the top right of the page.
## What the page shows
Firstly, an overview of [how well governed](/guide/insights/governance) the VIP topics are is shown.
The bar graph visualizes your most important topics using two dimensions.
**Bar height** represents the number of consumer groups subscribed to each topic:
* Taller bars indicate more consumer groups depend on the topic
* Many consumers suggest the topic provides data critical to multiple applications
* Wide usage indicates potential for widespread impact if issues occur
**Color intensity** represents message volume and throughput:
* Darker blue indicates higher message volume and throughput
* Lighter blue indicates lower message volume
* Message volume combined with consumer count identifies truly critical topics
Hover over any bar to see detailed metrics including the exact message count for the topic.
#### How to interpret the graph
**Tall, dark bars** - topics with many consumers and high message volume are your most business-critical data pipelines requiring highest priority for monitoring and careful change management.
**Tall, light bars** - topics with many consumers but lower message volume may represent configuration or control topics that still require careful management despite lower throughput.
**Short, dark bars** - topics with high message volume but fewer consumers represent specialized high-throughput pipelines requiring performance optimization and capacity planning.
## What the table shows
Below the graph, an expandable **VIP topic health overview** table provides detailed metrics for topics with recommendations. The table header shows the percentage of VIP topics that have recommendations requiring attention.
* **Topic**: topic name with topic type label (internal, streams, or user) and any custom labels
* **Fan-out**: number of consumer groups subscribed to the topic
* **Messages**: total message count showing volume
* **Replication factor**: number of replicas for fault tolerance
* **Skew**: partition imbalance percentage
Use the search box to filter topics by name. Click any column header to sort the table by that column in ascending or descending order.
Clicking a topic type or label in the table applies it as a global filter across all Insights sections.
## Our recommendations
For each VIP topic identified in the dashboard, verify and optimize configurations across multiple areas. Expand each section to review specific actions and guidance.
Validate that VIP topics have production-grade settings:
Go to **Topics** from the main menu and select a VIP topic identified from the Insights dashboard.
Click the **Configuration** tab to review current settings.
Check the following configurations:
**Replication factor:**
* Has to be at least **3** for production VIP topics
* Provides fault tolerance for broker failures
* Ensures data durability and availability
* Check the data loss risk graph in risk analysis for topics with insufficient replication
**Retention policy:**
* `retention.ms` - Time-based retention appropriate for business needs
* `retention.bytes` - Size-based retention per partition if applicable
* Consider longer retention for VIP topics to support late-arriving consumers
**Partition count:**
* Sufficient partitions for current and projected throughput
* Ideally a multiple of broker count for even distribution
* Adequate parallelism for all consumer groups
**Cleanup policy:**
* `delete` - For time-series or event data
* `compact` - For state or changelog topics
* Appropriate for the data model and consumption patterns
VIP topics should never have replication factor of 1 or 2 in production environments. If the [Risk Analysis](/guide/insights/risk-analysis) section identifies data loss risk for VIP topics, address these immediately.
Implement proactive monitoring to detect issues before they impact consuming applications:
Go to **Topics** and select the VIP topic you want to monitor. Click the **Alerts** tab on the topic detail page.
Set up alerts for consumer group lag thresholds:
* Define acceptable lag limits based on business requirements
* Use stricter thresholds for VIP topics than standard topics
* Alert on both absolute lag (message count) and time-based lag
For VIP topics, consider alerting when any consumer group exceeds 1000 messages of lag or 5 minutes of time-based lag, rather than the default thresholds used for standard topics.
Configure alerts for replication issues:
* Alert immediately if any partitions become under-replicated
* Under-replicated partitions indicate broker issues or failures
* Critical for VIP topics where data loss risk is unacceptable
Monitor storage consumption for VIP topics:
* Alert on rapid growth that could cause disk space issues
* Track retention effectiveness
* Plan capacity expansions before reaching limits
Track unusual traffic patterns:
* Alert on sudden drops in produce rate (possible producer failure)
* Alert on unexpected spikes that could cause performance issues
* Baseline normal throughput to detect anomalies
[Set up alerts for topic monitoring](/guide/monitor-brokers-apps/alerts)
Ensure VIP topics have clear ownership and governance from the start through self-service workflows rather than manual tracking.
**Use self-service for automatic ownership**
Conduktor's self-service framework provides a GitOps approach to topic lifecycle management where Applications dictate ownership of Kafka resources. When VIP topics are managed through self-service:
* **Automatic ownership tracking** - applications define owners and business context at topic creation
* **Clear accountability** - Console automatically assigns ownership to application teams
* **Governance enforcement** - policies ensure VIP topics meet configuration standards (RF=3, appropriate retention)
* **Business context preserved** - application definitions maintain documentation about purpose, dependencies and SLAs
**Benefits for VIP topics:**
* When issues occur, the right teams are contacted immediately through defined ownership
* Configuration changes follow approval workflows specific to business-critical topics
* Governance policies prevent VIP topics from being created with suboptimal settings
* Topic purpose and dependencies are documented in application definitions
[Implement self-service for governed topic creation](/guide/use-cases/self-service)
[Learn about self-service concepts and applications](/guide/conduktor-concepts/self-service)
Ensure VIP topics have appropriate security configurations:
Navigate to **Settings** > **RBAC** and review permissions for the VIP topic:
* Limit producer permissions to authorized applications only
* Restrict consumer access to approved teams and services
* Require elevated permissions for configuration changes
* Audit permissions regularly for VIP topics
[Learn more about RBAC](/guide/conduktor-in-production/admin/set-up-rbac)
For VIP topics containing sensitive data:
* Verify encryption in transit (SSL/TLS) is enforced
* Confirm ACLs or RBAC rules restrict access appropriately
* Check for data masking or encryption requirements
* Ensure compliance with organizational security policies
Track all consumer groups subscribed to VIP topics:
Go to **Topics** and select the VIP topic you want to analyze. Click the **Consumer Groups** tab on the topic detail page.
For each consumer group, examine:
* **Lag** - Current lag per partition and total lag
* **State** - Active consumers or empty groups
* **Members** - Number of active consumer instances
* **Commit frequency** - How often consumers commit offsets
Consumer lag on VIP topics requires immediate investigation. High lag indicates consumers cannot keep up with message volume, which may lead to processing delays, memory issues or timeout errors.
Look for warning signs:
* Consistently high or growing lag
* Consumers that frequently rebalance
* Groups with zero active members but uncommitted messages
* Uneven lag distribution across partitions
Contact the owning teams for consumer groups with persistent issues.
Ensure optimal partition allocation across brokers:
Go to **Topics** and select the VIP topic you want to review. Click the **Partitions** tab on the topic detail page.
Review the **Per broker** view to verify:
* Partitions are evenly distributed across all brokers
* Leadership is balanced (no single broker leads most partitions)
* No brokers are excluded from the topic
* Replica assignments provide proper fault tolerance
The cluster efficiency graph in [risk analysis](/guide/insights/risk-analysis) identifies partition distribution issues. For VIP topics, prioritize resolving distribution problems to avoid broker hotspots and performance bottlenecks.
Switch to **Per partition** view and compare:
* Partition sizes across all partitions
* Message counts per partition
* Offset ranges (begin offset to end offset)
Significant differences indicate load imbalance, which can cause uneven consumer load and processing delays.
[Learn how to address load imbalance](/guide/insights/risk-analysis#load-imbalance-risk)
Monitor VIP topic performance over time:
Go to **Topics** and select the VIP topic you want to monitor. Click the **Monitoring** tab on the topic detail page.
Analyze produce patterns:
* **Messages in per second** - Produce rate over time
* **Bytes in per second** - Data volume throughput
* Look for unusual spikes, drops or patterns
* Establish baseline performance for capacity planning
Analyze consume patterns:
* **Messages out per second** - Consume rate across all consumer groups
* **Bytes out per second** - Data volume being consumed
* Compare consume rate to produce rate to identify accumulation
Use the metrics to:
* Detect gradual throughput increases requiring capacity planning
* Identify time-of-day or day-of-week patterns
* Spot sudden changes that may indicate application issues
* Plan for peak traffic periods and scaling needs
## Troubleshoot
VIP topic identification is based on both message volume (over 500) and consumer count (more than 3). A topic may appear as VIP due to high consumer count (configuration topics read by many applications, control plane topics, event notification topics) or because it's business-critical despite lower volume (compliance, regulatory or SLA requirements).
Review the specific topic context to verify the VIP designation is appropriate for your organization's needs.
No. VIP topics should share minimum standards (replication factor of 3, configured alerting, documented ownership, proper RBAC permissions) but configurations should be tailored to each topic's requirements.
Partition count, retention, cleanup policy, compression and security settings should be based on specific throughput needs, data lifecycle requirements and sensitivity classification. For example, a high-volume transaction topic may need 50 partitions and 7-day retention, while a configuration topic may need 3 partitions and 90-day retention.
Document organization-wide minimum standards for VIP topics while allowing flexibility for topic-specific optimizations.
A poor health score indicates configuration or operational issues. Address systematically:
1. **Identify issues** - check all recommendations in the VIP topics section and review the [risk analysis](/guide/insights/risk-analysis).
2. **Prioritize by impact** - critical (data loss risk with RF \< 3); high (under-replicated partitions); medium (load imbalance); low (cluster efficiency issues).
3. **Take action** - for risk of data loss, [follow the replication remediation steps](/guide/insights/risk-analysis#data-loss-risk). For cluster efficiency, [follow partition troubleshooting steps](/guide/insights/risk-analysis#cluster-efficiency). For load imbalance, [resolve issues with partition skew](/guide/insights/risk-analysis#load-imbalance-risk).
4. **Verify improvement** - Monitor the health score after remediation to confirm resolution
Poor health scores on VIP topics require immediate investigation and remediation.
## Related resources
* [View Insights overview](/guide/insights)
* [Configure and manage topics](/guide/manage-kafka/kafka-resources/topics)
* [Set up monitoring and alerts](/guide/monitor-brokers-apps)
* [Monitor and manage brokers](/guide/manage-kafka/kafka-resources/brokers)
* [Set up RBAC](/guide/conduktor-in-production/admin/set-up-rbac)
* [Learn about Self-service topic management](/guide/conduktor-concepts/self-service)
* [Give us feedback or request a feature](https://conduktor.io/roadmap)
# Kafka management with Conduktor Console
Source: https://docs.conduktor.io/guide/manage-kafka
Manage all Kafka resources with Conduktor Console: topics, consumer groups, brokers, Schema Registry, connectors.
Use Conduktor Console to manage, monitor and govern all your Kafka resources - through our UI or through automation.
## Insights
* [Overview](/guide/insights)
* [Risk analysis](/guide/insights/risk-analysis)
* [Cost control](/guide/insights/cost-control)
* [VIP topics](/guide/insights/vip-topics)
* [Governance](/guide/insights/governance)
## Conduktor MCP
* [Configuration overview](/guide/conduktor-in-production/automate/mcp)
## Automate tasks
* [Overview](/guide/conduktor-in-production/automate)
* [API](/guide/conduktor-in-production/automate/api-automation)
* [CLI](/guide/conduktor-in-production/automate/cli-automation)
* [Terraform](/guide/conduktor-in-production/automate/terraform-automation)
## Manage Kafka resources
* [Topics](/guide/manage-kafka/kafka-resources/topics)
* [Brokers](/guide/manage-kafka/kafka-resources/brokers)
* [Consumer groups](/guide/manage-kafka/kafka-resources/consumer-groups)
* [Schema registry](/guide/manage-kafka/kafka-resources/schema-registry)
* [Kafka connect](/guide/manage-kafka/kafka-resources/kafka-connect)
* [Service accounts and ACLs](/guide/manage-kafka/kafka-resources/service-accounts-acls)
* [ksqlDB](/guide/manage-kafka/kafka-resources/ksql-db)
## Monitor systems
* [Overview](/guide/monitor-brokers-apps)
* [Alerts](/guide/monitor-brokers-apps/alerts)
* [Console license](/guide/conduktor-in-production/manage-licenses/console)
* [Gateway license](/guide/conduktor-in-production/manage-licenses/gateway)
## Govern data
* [Create data masking policies](/guide/conduktor-in-production/admin/data-masking)
* [Define data validation policies](/guide/conduktor-in-production/admin/gateway-policies)
# Kafka broker monitoring — view cluster health in Console
Source: https://docs.conduktor.io/guide/manage-kafka/kafka-resources/brokers
View Kafka broker status, cluster version, partition distribution, and configuration drift in Conduktor Console.
In Console, the **Brokers** page provides the broker list and information about the current state of your Kafka cluster, including:
* total number of brokers
* controller status (shows whether the selected broker is the controller or not)
* the version of your Kafka Cluster
* whether all brokers are configured the same or not (yes/no)
### Brokers tab
This is the default view that shows a key metrics graph and a table listing all your brokers. You can sort the view or search for a specific broker.
Click on a broker in the table to see its details:
* Hostname path
* Number of partitions
* Number of replicas
* Number of URPs (Under-Replicated Partitions)
* Configuration tab - search and view the list properties/values. Click **Raw view** to view or copy the raw broker data.
* Logs tab - view topics, partitions, offset lag and more
* Graphs tab - see disk usage and partition count graphs
### Graphs tab
This tab lets you visualize the current and past state of your Kafka clusters on metrics like:
* Produce and consume rate
* Disk usage
* Total partition count
* Offline, under replicated and under min ISR partition count
Each graph can be adjusted to show data for the past 24 hours, or one/seven/thirty day periods.
You can get notified of changes in any of these metrics as each graph can alerts set up.
To add an alert, click the **alert icon**. A side panel will open, allowing you to set parameters configure the triggers for your alert.
### Alerts tab
Manage existing and create new alerts for the selected broker.
## Related resources
* [Find broker risks with Insights](/guide/insights/risk-analysis)
* [Set up broker alerts](/guide/monitor-brokers-apps/alerts)
* [View metrics reference](/guide/reference/console-metrics)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Kafka consumer group monitoring
Source: https://docs.conduktor.io/guide/manage-kafka/kafka-resources/consumer-groups
Monitor Kafka consumer groups, track consumer lag per partition, reset offsets, and manage group membership with Conduktor Console.
Conduktor Console provides a visual interface to manage Kafka consumer groups. You can view consumer group details, monitor lag, manage offsets, and perform administrative operations.
Configure [**RBAC**](/guide/conduktor-in-production/admin/set-up-rbac) to restrict your users to view, manage, or perform operations only on certain consumer groups.
## View consumer groups
The Consumer Groups page lets you search for any consumer group on your currently selected Kafka cluster.
Multiple search capabilities can be combined to help you find the consumer group you want faster.
You can **filter** by:
* Consumer group name
* State (Stable, Empty, Dead, Rebalancing, Unknown)
* Labels (click on a label to add/remove it from the filters)
You can **sort** by:
* Name
* State
* Overall lag
* Max lag time
* Number of members
Pick **active columns** from a list of available columns using the **Edit columns** option.
Your current **filters**, active **sort**, and **visible columns** are stored in your browser's local storage for each Kafka cluster and persist across sessions.
## View consumer group details
Click on a consumer group from the list to view its details. The details page shows:
* **State**: The current state of the consumer group (Stable, Empty, Dead, Rebalancing, etc.)
* **Members**: List of active members with their client IDs, hosts, and assigned partitions
* **Topics**: Topics the consumer group is subscribed to
* **Lag**: Current lag per partition and overall lag
### View members
The **Members** tab displays all active members of the consumer group:
* Member ID and Client ID
* Host information
* Assigned topic-partitions
* Per-member lag statistics
### View topics
The **Topics** tab shows all topics the consumer group consumes from:
* Topic name
* Number of partitions assigned
* Current offset, end offset, and lag per partition
* Consume rate statistics
### View alerts
The **Alerts** tab shows all alerts configured for the consumer group. You can create alerts to monitor lag, consume rate, or other metrics.
### View statistics
The statistics bar shows an overview of the consumer group:
* **State**: Current state of the consumer group
* **Consume rate**: Messages consumed per second
* **Offset lag**: Total lag across all partitions
* **Time lag**: Estimated time to catch up based on consume rate
## Operations
Several actions are available for consumer groups.
### Create consumer group
You can create a new consumer group with initial offsets for specific topics.
1. Click the **Create consumer group** button
2. Provide a consumer group name
3. Select the topics to subscribe to
4. Choose the initial offset position:
* **Earliest**: Start from the beginning of each partition
* **Latest**: Start from the end of each partition (only consume new messages)
Create a consumer group with initial offsets when you want to pre-configure offset positions before your consumer application starts.
### Duplicate consumer group
You can duplicate an existing consumer group to create a new one with the same offset positions.
1. Select the consumer group you want to duplicate
2. Click on the **Duplicate** action
3. Provide a name for the new consumer group
This is useful for test scenarios or to create backup consumer groups.
### Reset offsets
Reset offsets to change the position from which a consumer group reads messages.
The consumer group has to be **inactive** (Empty or Dead state) to reset offsets. Stop all consumer instances before performing this operation.
To reset offsets:
1. Select the consumer group
2. Click on **Reset offsets**
3. Choose which partitions to reset:
* All partitions
* Specific topics
* Specific topic-partitions
4. Select the reset strategy:
* **To earliest**: Reset to the beginning of each partition
* **To latest**: Reset to the end of each partition
* **To specific offset**: Reset to a specific offset value
* **To timestamp**: Reset to the offset at a specific point in time
* **Shift by**: Add or subtract a number from current offsets
#### Preview changes
Before applying offset changes, use the **Preview** feature to see what the new offsets will be for each partition. This helps you verify the changes before committing them.
The preview shows:
* Topic and partition
* Current offset
* New offset after reset
### Delete consumer group
You can delete a consumer group from Kafka.
The consumer group has to be **inactive** (Empty or Dead state) to be deleted. Stop all consumer instances before performing this operation. This action is permanent and irreversible.
1. Select the consumer group
2. Click on the **Delete** action
3. Confirm the deletion
### Remove members and rebalance
Force all members to leave the consumer group and trigger a rebalance.
This removes all members from the group. Live members will re-join automatically. This is helpful to remove dead static members.
## Manage consumer group labels
You can categorize your consumer groups using key-value pairs called labels.
To manage labels:
1. Select the consumer group
2. Click on the **Edit labels** action
3. Add, modify, or remove label key-value pairs
Labels can be used to:
* Filter consumer groups in the list view
* Organize consumer groups by team, environment, or application
* Apply RBAC policies based on labels
## View linked resources
### View from topics
You can view consumer groups that consume from a specific topic directly from the Topic details page under the **Consumer Groups** tab.
This shows all consumer groups that have committed offsets for the topic, along with their lag information.
## Understand consumer group states
Understand consumer group states to manage them effectively:
| State | Description |
| ----------------------- | ---------------------------------------------------------------------------- |
| **Stable** | The group has active members and partition assignments are complete |
| **Empty** | The group exists but has no active members |
| **Dead** | The group has no members and no metadata (offsets may still exist) |
| **PreparingRebalance** | The group is preparing to rebalance partition assignments |
| **CompletingRebalance** | The group is completing a rebalance operation |
| **Rebalancing** | The group is actively rebalancing (combines preparing and completing states) |
| **Assigning** | The group is assigning partitions to members |
| **Reconciling** | The group is reconciling partition assignments |
| **Unknown** | The group state could not be determined |
Offset modifications (reset, delete) are only allowed when the consumer group is in **Empty** or **Dead** state.
## Related resources
* [Set up alerts for consumer lag](/guide/monitor-brokers-apps/alerts)
* [Manage Kafka topics](/guide/manage-kafka/kafka-resources/topics)
* [Set up RBAC](/guide/conduktor-in-production/admin/set-up-rbac)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Kafka Connect management
Source: https://docs.conduktor.io/guide/manage-kafka/kafka-resources/kafka-connect
Deploy, monitor, and manage Kafka Connect source and sink connectors from a single UI. Restart failed tasks, configure RBAC, and track connector status.
In Console, the Kafka Connect page lets you stream data between Apache Kafka and other data systems in a reliable and scalable way. Conduktor provides a simple interface for managing all of your source and sink connectors in one place.
## Kafka Connect cluster list
The Connect Cluster List is a summary view of all your configured Kafka Connect Clusters attached to your currently selected Kafka Cluster.
You can filter by Connect cluster name and order by:
* number of Connectors
* number of tasks
* connect cluster name
The list is warning you that some tasks have failed and gives you a high level overview by hovering over "Connectors".
## Connector
The Connector list page lets you search for any Connector on your currently selected Kafka Connect cluster.
[Configure RBAC](/guide/conduktor-in-production/admin/set-up-rbac) to restrict user access to viewing, browsing or performing any operation on selected topics.
Multiple search capabilities can be combined to help you find the Connector you want faster.
**Filtering** is possible on:
* Connector name
* Connector class
* Connector type (source / sink)
* Connector status
**Sorting** is possible on all columns.
**Active columns** can be picked from a list of Available columns from the side button « ⚙️ Edit columns »
The round arrow icon next to the Connector name indicates whether the connector will be auto-restarted by Conduktor: (Grey: disabled, Green: enabled)
Clicking a Connector in the list brings you to the Connector overview page where you can perform further actions on the selected Connector:
* Review the Connector Task details and status
* View and Edit the Connector configuration
* Create and manage alerts for this Connector
* Toggle auto-restart feature
Several actions are also directly available from the Connectors list:
* Add a Connector
* Pause/Resume and Restart
* Delete Connector
All these operations can be applied either on a single Connector, or on multiple Connectors at once:
## Add a Connector
Click **Add a Connector** to deploy a new one. You'll see all the connector plugin classes installed on this connect cluster.
If your platform team has defined [connector templates](/guide/reference/self-service-reference#connectortemplate) for the class you select, pick one from **Start from template** to pre-fill the wizard with a vetted configuration. You can change any value before you deploy the connector, and a template may include `{{placeholder}}` values to replace before you submit.
See a connector template pre-fill the wizard:
This is an excerpt from the [full templates walkthrough](/guide/use-cases/self-service#resource-templates).
Next, you'll get to the config wizard for Kafka connect which takes full advantage of the [Kafka Connect Validate API](https://docs.confluent.io/platform/current/connect/references/restapi.html#put--connector-plugins-\(string-name\)-config-validate) :
* A form is generated with structured configuration groups to be filled out
* Supportive error handling is included with each individual field
* Embedded documentation helps you understand which fields are required and what their expected, and default, values are
* Toggle advanced configuration to visualize only the most important fields
* Switch seamlessly between Form View and JSON View at any time
Configure your Connector to your convenience and use the **Validate** button to verify that your configuration is valid.
This will highlight the parts of the configuration that are invalid, and give you precise information on how to correct your Connector configuration.
While Kafka Connect Validate API generally checks for most configuration inconsistencies, there are some limits:
* It usually doesn't check for external configuration such as URL and user / passwords.
* Some Kafka Connect Plugins classes are notoriously badly implemented and don't take full advantage of Kafka Connect Validate API
When errors happen outside the nominal scope of Kafka Connect Validate API, you will see the errors as toasts:
At any point in time, you can switch to JSON view and edit the JSON payload directly. You can switch back and forth between JSON and Form view at your convenience.
When you're done, click "Next" and you'll be presented with a Review screen where you will be able to copy the YAML associated to your Kafka Connect configuration.
This YAML will help you automate your deployment with the help of Conduktor CLI.\
This is entirely optional and you can just deploy your Connector from the UI by clicking "Submit".
## Connector overview
The Connector overview page is split into three sections:
* Statistics
* Graphs and alerts
* Task list
### Statistics
This section gives you relevant information about the current state of your Connector that will stay visible across all tabs:
* Connector Status
* Connector Class
* Whether Auto-restart is enabled
* How many Topics are associated. You can click on the icon next to it to visualize their names
### Operations
A few operations are available:
* **Pause/Resume** uses the Kafka Connect API to Pause or Resume the Connector. A paused Connector keeps its tasks assigned, so it resumes without rebalancing
* **Stop** uses the Kafka Connect API to stop the Connector and release its tasks and resources. Unlike pausing, stopping frees the resources the Connector held. Stop a Connector before you edit or reset its offsets
* **Restart Connector** restarts the Connector (**not the tasks**)
* **Remove Connector** deletes the Connector from the Kafka Connect Cluster
* You can also **Reset Topics** to clear and refresh the list of topics used by that Connector. This is useful if your Connector is configured with `topics.regex` and new topics were added recently, or if stale topics need to be removed from the list
* This makes use of the Kafka Connect [Topics Reset API](https://docs.confluent.io/platform/current/connect/references/restapi.html#topics)
Stopping a Connector needs a Connect cluster running Kafka Connect 3.5 or later. On older Connect clusters, or where the managed Connect service doesn't support it (such as some Confluent Cloud connectors), the stop and offset operations aren't available.
After clicking **Reset Topics**, the topic count may temporarily show 0 or an incorrect number before displaying the accurate count. This is because Kafka Connect needs a few seconds to rebuild the topic list internally.
### Graphs and alerts
The Graph section lets you visualize the number of Failed Tasks of your Connector over the past 24 hours (Line chart on the left) or over the past 3 months (Heatmap on the right). This can help you understand if there is a specific pattern associated with your Kafka Connect Tasks failures.
#### Create an alert
To create an Alert, click on the **+** button inside the graph to open the **Create Alert** page. There, define the threshold for your alert as well as a name, and click **Save**.
You can get a summary of all alerts associated to this Connector in the dedicated **Alerts** tab.
### Task list
The task list displays all the information related to the tasks associated to the Connector:
* The task ID
* The Kafka Connect worker on which the task is running
* The current task state
* An option to **Restart task**
For Confluent Cloud Managed Connectors, the restart task will hit Confluent Cloud's [restart connector API](https://docs.confluent.io/cloud/current/api.html#tag/Lifecycle-\(connectv1\)) . This will restart the entire connector and all associated tasks.
If a task is in a Failed state, you can see the detailed logs about the failed task.
## Connector configuration
The **Connector configuration** tab lets you view or edit your Connector configuration with a wizard which takes full advantage of the [Kafka Connect Validate API](https://docs.confluent.io/platform/current/connect/references/restapi.html#put--connector-plugins-\(string-name\)-config-validate) :
* A form is generated with structured configuration groups to be filled out
* Supportive error handling is included with each individual field
* Embedded documentation helps you understand which fields are required and what their expected, and default, values are
* Toggle advanced configuration to visualize only the most important fields
* Switch seamlessly between Form View and JSON View at any time
The same configuration as a JSON:
## Connector offsets
The **Offsets** tab shows the current offsets the Connector has committed:
* For a source Connector, the source partition and source offset that track its position in the upstream system
* For a sink Connector, the Kafka topic-partitions and offsets it has consumed
The Offsets tab is read-only. To edit or reset offsets, use the [public API](/guide/conduktor-in-production/automate/api-automation). Stop the Connector first — you can't change the offsets of a running or paused Connector. Editing offsets controls where the Connector resumes from, and resetting clears all committed offsets so the Connector starts over.
Editing or resetting offsets changes where the Connector resumes from and can cause it to reprocess or skip data. Verify the new offsets before you restart the Connector.
## Connector auto-restart
Sometimes, Kafka Connect tasks experience failures due to transient issues, database which isn't available for a few minutes or a networking issue, etc. When this happens, Kafka Connect tasks shut down and you need to manually start the tasks again.
Watch how Console monitors and recovers failed connectors:
### Enable Auto-restart
Conduktor lets you enable **Auto-restart** on any connector instance. Once enabled, you can additionally configure the **delay** (default 10 minutes) between 2 consecutive restart attempts.
For Confluent Cloud Managed Connectors, the auto-restart functionality will hit Confluent Cloud's [restart connector API](https://docs.confluent.io/cloud/current/api.html#tag/Lifecycle-\(connectv1\)) . This will restart the entire connector and all its associated tasks.
### How Auto-restart works
**Every minute**, Console will check for failed tasks for this Connector. For failed tasks:
* If the Auto-restart process hasn't tried to restart the task in the past **10 minutes** (configurable):
* Capture the task error message
* Restart the failed task
* Otherwise, do nothing
#### Auto-restart history
You can review all the occurrences when Auto-restart triggered and the error message of the failed task before restarting.
### Connector alerts
The Connector Alert tab lets you visualize all active alerts associated to this Connector. You can edit them or, toggle them on or off.
## Enable Confluent Cloud managed connectors
### Generate an API key
In order for Conduktor to access a Kafka Connect, you need to generate an API key and secret to act as a username and password. This can be achieved through the Confluent Cloud CLI tool: `ccloud`. To install it, you can follow the [official Confluent documentation](https://docs.confluent.io/ccloud-cli/current/install.html) .
Once installed, complete the following steps:
* Login and select the appropriate environment:
```text theme={null}
$ ccloud login
$ ccloud environment
```
* Generate an API key and secret and write them down
```text theme={null}
$ ccloud api-key create --resource cloud
+---------+------------------------------------------------------------------+
| API Key | ABCDEFKZBF56666 |
| Secret | ToMaHaWkjQ1bt7BxvdyFjaJ8j3nSokaAd83Nhan739snAiufIAfdk7fFAAnBKxai |
+---------+------------------------------------------------------------------+
```
* The API key is the username, the secret is the token. Store these somewhere as we will need them later in the process.
### Obtaining your environment and cluster identifiers
Now we need to obtain the identifiers of the environment and cluster.
* Retrieving the ID of your environment can be achieved through the `ccloud environment list` command.
```text theme={null}
$ ccloud environment list
Id | Name
+-------------+------------+
env-1234 | staging
* env-6789 | production
```
In this scenario, the environment ID for the production cluster would be `env-6789`.
* Retrieving the ID of your Kafka cluster can be achieved from the `ccloud kafka cluster list` command.
```text theme={null}
ccloud environment use env-6789
ccloud kafka cluster list 16:21:10
Id | Name | Type | Provider | Region | Availability | Status
+-------------+----------------------+-------+----------+--------------+--------------+--------+
lkc-8888 | analytics-production | BASIC | gcp | europe-west4 | single-zone | UP
```
In this scenario, the ID of the cluster would be `lkc-8888`.
### Building the Kafka connection URL
With your obtained environment and cluster IDs, build the URL as so: `https://api.confluent.cloud/connect/v1/environments/${ENVIRONMENT_ID}/clusters/${CLUSTER_ID}/`
### Populating the configuration information in Conduktor console
Navigate to cluster settings and go to the 'Kafka connect' tab. Once there, fill in the form with the following information:
* Technical ID: The cluster ID you have obtained
* URL: The Kafka connection URL you have built
* Authentication: Basic auth
* Username: The API key you have generated
* Password: The token you have generated.
## Related resources
* [Set up connector alerts](/guide/monitor-brokers-apps/alerts)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# ksqlDB management — Conduktor Console
Source: https://docs.conduktor.io/guide/manage-kafka/kafka-resources/ksql-db
Manage ksqlDB clusters from Conduktor Console. Run streaming SQL queries, browse streams and tables, and control access with RBAC.
In Console, the ksqlDB home page shows the list of ksqlDB clusters, if any have been configured.
Configure [RBAC](/guide/conduktor-in-production/admin/set-up-rbac) to restrict access to certain clusters.
Click the required ksqlDB entry to see the details, such as:
* version
* number of streams
* amount of tables
* number of queries
The page consists of several tabs that let you visualize all the currently running queries and write your own or executes statements.
* [Streams and tables tabs](#streams-and-tables-tabs) let you view and manage the running streams and tables.
* [Queries tab](#queries-tab) shows all the queries currently running on the selected ksqlDB cluster.
* [Editor tab](#editor-tab) lets you run queries and execute statements.
By default, the **Streams** tab will be open, allowing you to view and manage all the streams and related topics, key/value formats as well as the number of read/write queries. Click on a row to see the details for each stream.
## Streams and tables tabs
These two tabs are similar but show different resources. The **Streams** one shows the result of `CREATE STREAM` statements and the **Tables** one is about `CREATE TABLE` statements.
The details include:
* name of the stream/table
* topic that this stream/table write data into
* key and value formats
* number of ksqlDB queries that write into this stream/table
* number of queries that read from this stream/table
You can **click an item to view its details** or delete the selected stream/table by clicking on the **trash can**.
### Streams and tables details page
This page consists of three tabs:
* **Fields** shows the structure (name and type) of the selected stream/table.
* **kSQL** displays the exact kSQL statement running behind the scenes.
* **Metrics** provides runtime statistics for the topic backing the selected stream/table.
Click **Terminate** at the top of the page to delete the stream/table. You'll be prompted to confirm your choice as it can't be undone.
## Queries tab
This tab shows all the currently running queries the selected ksqlDB cluster. The table shows:
* Query ID.
* Output topic: the topic into which the query is writing to.
* Type: either **Persistent** for the queries originating from `CREATE` statements or **Push** for queries executed by users in the query editor or via calls to `/query-stream` endpoint.
Click on the query to see field details (name and type) and the source KSQL code. Click **Terminate** at the top of the page to delete the stream/table. You'll be prompted to confirm your choice as it can't be undone.
## Editor tab
This tab lets you run queries and execute statements.
### Run a query
To run a `Push` or `Pull` query, select **Run a query**, paste or type the required commands and click **Run**. This will send requests to the `/query-stream` endpoint of ksqlDB.
You can change the **start from** to be **earliest** or **latest**.
The records will appear at the bottom in the **Results** section.
### Execute a statement
To run a sequence of SQL statements, select **Execute a statement**, paste or type the required scripts and click **Run**.
The `/ksql` endpoint of ksqlDB will be used.
Look out for a success or fail alert at the bottom of the screen.
# Kafka Schema Registry management
Source: https://docs.conduktor.io/guide/manage-kafka/kafka-resources/schema-registry
Manage Avro, Protobuf, and JSON schemas in your Kafka Schema Registry with Conduktor Console. Create subjects, browse versions.
Conduktor provides a visual interface for schema registry, allowing you to create and manage Subjects and schema with ease. Having schema registry ensures there is an automated way of ensuring data verification, schema evolution, and ability for new consumers to emerge without breaking downstream.
Conduktor supports **Confluent**, **Confluent like** (e.g. Karapace) and **AWS Glue** schema registry types.
**From our blog:** [Kafka schema governance: from chaos to confidence](https://conduktor.io/blog/kafka-schema-governance-from-chaos-to-confidence) Move from reactive schema firefighting to proactive governance across your Kafka ecosystem.
## Get started
Schema registry needs to be enabled on a per-cluster basis before any schemas can be added. Add your schema registry details on the tab available on an existing cluster configuration.
To see an example of setting up schema registry, head to [Part 3 of this blog post](https://www.conduktor.io/blog/what-is-the-schema-registry-and-why-do-you-need-to-use-it) . This example uses an Aiven cluster and Karapace schema registry.
## Create subject
Once schema registry is setup, you can explore your subjects in the **Schema registry** tab from the left-hand menu. From there, click on the **New subject** button in the top right corner. This will open a side panel where you can configure your new subject.
### Subject formats
Conduktor Console supports the [Avro](https://avro.apache.org/docs/current/spec.html) , [JSON](https://json-schema.org/) and [Protobuf](https://developers.google.com/protocol-buffers/) formats for schema registry.
### Strategy
Subjects can be applied in several ways: To topics, to records, to both topics and records, or through a custom method.
* **Topic Name**: This strategy associates your Subject with a specific topic of your choosing. Simply enter the topic name that you want the Subject to apply to. You will also need to specify if the Subject will apply to the key or value of a message. The topic name field will auto-complete for you, so no need to remember specific spellings!
* **Record Name**: This is set within a message. The Subject will only apply to messages with the record name that you define, but it will apply globally across all topics. Note that the field which this will apply to varies based on the format you choose. For Avro, this is set by `{name}`. For JSON, it is `{title}`. For Protobuf, it will be `{message}`.
* **Topic + Record Name**: Choose this option if you want your Subject to apply to both a specific topic, and messages with a specific name within that topic.
* **Custom**: For more advanced options, use the Custom strategy. You can define your strategy in the Custom name field.
### Subject content
Once you have decided on your format and strategy, you will need to enter your subject structure in the form provided.
Once it is ready, click **Create** and the Subject will be complete.
## Manage subject and schemas
You can manage your subjects and schemas from the **Schema Registry** tab. This includes:
**For subjects**:
* Change their compatibility
* Delete them
**For schemas**:
* Update a schema and create a new version
* Compare schemas from the same subject
* Delete them
### Change subject compatibility
Each subject has a compatibility type that determines how they should deal with evolution over time. As mentioned above, updating a Subject can cause issues upstream or downstream on applications, and choosing the right compatibility type can prevent this from occurring.
There are seven possible compatibility types:
1. Backward
2. Backward Transitive
3. Forward
4. Forward Transitive
5. Full
6. Full Transitive
7. None
[Find out what each type does from Confluent docs](https://docs.confluent.io//current/schema-registry/avro.html#summary) .
Compatibility can be changed **per subject** or **globally**.
### Delete subject
From the dropdown shown above, you can also delete a subject with the bin icon at the bottom.
This will be a **soft-delete**, meaning that the schema versions under this subject will be removed, but their metadata such as their schema IDs will remain for lookup.
### Update schema
To update a schema, select the relevant subject and click **Update Schema**. This will open a side panel where you can add or remove fields from your schema.
Before updating, use the **Check compatibility** button to understand if this change will break anything upstream or downstream on our applications.
### Compare schemas
If you have more than two schemas associated to your subject, you will be able to compare them. For that, simply select the two versions you'd like to compare.
### Delete schema
To delete a schema, select the right version in the dropdown and click on the `...`. From there, click on **Delete version**.
This will be a **soft-delete**, meaning that the schema version will be removed, but its metadata such as its schema ID will remain for lookup.
If you only have one schema for this subject, the subject will be soft-deleted too.
## Related resources
* [View schema adoption with governance Insights](/guide/insights/governance)
* [Enforce data quality with policies](/guide/use-cases/enforce-data-quality)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Kafka ACLs and service accounts
Source: https://docs.conduktor.io/guide/manage-kafka/kafka-resources/service-accounts-acls
Manage Kafka service accounts and ACLs in Conduktor Console. View permissions per principal, create ACLs, and integrate with Confluent, Aiven.
Console lets you manage service accounts and ACLs (Access Control Lists).
What you see on the **Service Accounts** page will depend on whether you have [configured a Kafka provider, such as Confluent or Aiven](#provider-integrations).
## View service accounts and manage ACLs
The **Service Accounts** page in Console offers a powerful design that aggregates ACLs to principal granularity. This lets you easily understand all the resource ACLs assigned to each service account.
After selecting a service account, you can **edit** existing ACLs and **add new** resource ACLs. When you create new ACLs for a resource, you can:
* Use pre-configured permissions (e.g. Consumer, Producer, Admin)
* Manually define permissions
To manually edit permissions, select the operation and choose **Allow**, **Deny** or **Not Set**.
## Provider integrations
Connect your Kafka provider to manage their benefits directly in Conduktor Console. The additional functionality you get from connecting your provider is made possible via their APIs.
To use Kafka provider service account functionality, you have to configure the provider. Go to **Settings** > **Clusters**, select a cluster and use the **Provider** tab.
### Confluent Cloud
When configuring Confluent Cloud as your provider, ensure you use [Confluent Cloud API keys](https://docs.confluent.io/cloud/current/access-management/authenticate/api-keys/api-keys.html) scoped to `Cloud resource management`.
To enable visibility of topic size metrics, ensure that [the API key is scoped](https://docs.confluent.io/cloud/current/monitoring/metrics-api.html#create-an-api-key-to-authenticate-to-the-metrics-api) to the `MetricsViewer` role.
Once you have configured Confluent Cloud as your [provider](#provider-integrations), you can:
* Manage service accounts and ACLs
* Manage API keys
To manage Confluent Cloud resources when connected to a Confluent Cloud Kafka cluster, go to the **Service Accounts** page.
You can **list** and **create API Keys** associated with a Confluent Cloud service account:
### Aiven
When configuring Aiven as your [provider](#provider-integrations), you can generate your API token following [this guide](https://aiven.io/docs/platform/howto/create_authentication_token) , and get the **project name** and **service name** fields in the below locations from within the Aiven console.
Once you have configured Aiven as your [provider](#provider-integrations), you can:
* Manage service accounts
* Manage ACLs
To manage Aiven resources when connected to an Aiven Kafka cluster, go to the **Service Accounts**.
You'll see the list of your Aiven users and their associated Aiven ACLs.
## Related resources
* [Gateway service accounts](/guide/conduktor-concepts/gateway-service-accounts)
* [Gateway authentication and authorization](/guide/conduktor-concepts/gateway-authentication-authorization)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Kafka topics — browse, produce, and consume in Console
Source: https://docs.conduktor.io/guide/manage-kafka/kafka-resources/topics
Browse, create, and manage Kafka topics in Conduktor Console. Filter by cleanup policy, view partition details, and produce or consume messages.
## Topic list
The Topic page lets you search for any topic on your currently selected Kafka cluster.
To restrict user/group access to view or manage certain topics, [configure RBAC](/guide/conduktor-in-production/admin/set-up-rbac).
Multiple search capabilities can be combined to help you find to the topic you want faster.
**Filtering** is possible on:
* Topic name
* Show/Hide Internal topics (starts with `_`)
* Show/Hide Kafka stream topics (ends with `-repartition` or `-changelog`)
* Cleanup policy
* Labels (click on a label to add/remove it from the filters. [Find out about managing topic labels](#manage-topic-labels)).
**Sorting** is possible on all columns.
**Active columns** can be picked from a list of available columns from the **Edit columns** option.
Your current **filters**, active **sort**, and **visible columns** are stored in your browser's local storage for each Kafka cluster and persist across sessions.
## Operations
Several actions are also available from the topic List: create, empty or delete topics and add partitions.
### Create topic
In the **Create Topic** page, you can provide all the necessary information to create a topic.
The default choices made by Console are generally safe for most typical Kafka production deployments. [Find out about choosing the replication factor and partition count](https://www.conduktor.io/kafka/kafka-topics-choosing-the-replication-factor-and-partitions-count/) and [Kafka cleanup policies](https://www.conduktor.io/kafka/kafka-topic-configuration-log-compaction/) .
**Start from template**
If your platform team has defined [topic templates](/guide/reference/self-service-reference#topictemplate), pick one from **Start from template** to pre-fill the form with a vetted set of defaults. You can change any value before you create the topic. A template may include `{{placeholder}}` values, such as a team name — replace them before you submit.
See a topic template pre-fill the Create Topic form:
This is an excerpt from the [full templates walkthrough](/guide/use-cases/self-service#resource-templates).
**Topic name**
As per [Kafka specification](https://github.com/apache/kafka/blob/08c437d25e74d63f11ae3f184dfb3889d4639ece/clients/src/main/java/org/apache/kafka/common/internals/Topic.java#L49-L62) , topic name must only contain the following characters `[a-zA-Z0-9._-]` and not exceed 249 characters.
**Partitions**
This lets you define how scalable your topic will be for your consumers. In general you want a multiple of your number of brokers.
Default: `3`
**Replication factor**
This configuration helps prevent data loss by writing the same data to more than one broker.
Default: `min (3, number of brokers)`
Use the [Insights dashboard](/guide/insights/risk-analysis) to identify topics with low replication factors or partition distribution issues that may pose risks to your infrastructure.
**Cleanup policy**
The cleanup policy (along with its associated advanced configurations) controls how the retention of your messages is done.
**Labels**
Use labels to organize your topics and facilitate searching them in Console. Each label is a key-value pair.
**Advanced configuration**
Enable this option to use all the available options.
[Find out more about the Apache Kafka topic configuration](https://kafka.apache.org/documentation/#topicconfigs)
### Add partitions
Increase the number of partitions for your topic. Number of partitions cannot be decreased.
Adding partitions reshuffles the target partition of messages with a given key. Existing data will stay on the previous partition. Consumers that rely on partition ordering could be impacted.
### Empty topic
This lets you delete all records from a topic. This operation is permanent and irreversible.
If you want to only delete all records from given partition, there's a dedicated operation on the partitions tab of the topic detail.
### Delete topic
This lets you delete the topic from Kafka. This operation is permanent and irreversible.
### Manage topic labels
You can help categorize your topics further using key-value pairs called labels.
To manage your topic's labels via the UI, click on the topic and **Edit** from the topic details view.
A side bar will appear with the current tags associated with the topic and a button to add more. To remove a label, click on the trash icon.
Anyone who can view a topic can edit its labels and description by default. To restrict editing to users with a dedicated permission, enable [metadata governance](/guide/conduktor-in-production/admin/set-up-rbac#govern-who-can-edit-labels-and-descriptions).
# Topic produce overview
* [Configure producer](#configure-producer)
* [Produced messages panel](#produced-messages-panel)
* [Operations](#operations)
The produce page lets you configure all the details necessary to produce a record in your current Kafka topic.
It is already configured with [sensible defaults](#sensible-defaults) that you can customize if necessary.
## Configure producer
Each section from the accordion menu will allow you to configure the Kafka producer further: key, value, headers, flow and additional options.
### Key and value
This section is similar for both key and value. The menu lets you choose your serializer to encode your message.
The default serializer is **String**, unless you have a matching subject name in your schema registry (with the **TopicNameStrategy**).
In that case, the serializer will be set automatically to the proper registry type (Avro, Proto, Json Schema) with the subject name of `-key` for the key and `-value` for the value.
### Confluent subject strategies
When producing messages with a schema registry, Conduktor supports Confluent subject strategies, providing control over how schemas are referenced. This gives users flexibility in schema organization and naming conventions, enabling support for different strategies based on your use case (TopicName, RecordName, TopicRecordName strategies).
You can configure these strategies directly from the produce page when working with topics that use schema registry.
#### Random data generator
Click **Generate once** to generate a message that conforms to the selected serializer.
This works with schema registry serializers as well:
### Headers
This section lets you add headers to your message.
Header key and header value both expect valid UTF8 string.
### Flow
Using the flow mode, you can produce multiple records in one go or configure a live producer that will produce records at a configurable rate.
**Send N records at a time**
Define how many messages should be produced every time you click **Produce**.
Default: 1
Range: \[1, 10]
**Generate random key/value**
Enable this option to generate a different message each time you click **Produce**.
When enabled, it will override the key or value configured above and will rely on the random data generator to produce messages (same as if you clicked **Generate once** before producing a record).
When disabled, the producer will use the Key/Value configured above
Default: disabled
**Producer mode**
Manual mode starts a single Kafka produce each time you click the **Produce**.
Automatic mode starts a long running process that batches several Kafka produce.
**Interval (ms):**
The interval between each produce batch in milliseconds.
Range: \[1000, 60000]
Stop conditions: The first met condition stops the producer
* **Number of records produced**
* Stops the producer after that many records have been produced.
* Range: \[0, 1000000]
* **Elapsed time (ms)**
* Stops the producer after a set period of time.
* Range: \[0, 60000]
Here's an example of a produce flow that will generate a batch of 10 records, every second for a minute. Uses the same key but a random value (based on the Avro schema linked to the topic) for every record:
### Additional options
**Force partition**
This option lets you choose the partition where to produce your record.
If set to **all**, it will use the [DefaultPartitioner](https://github.com/apache/kafka/blob/3.7/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java) from KafkaClient 3.6+
* StickyPartitioner when Key is null or `Utils.murmur2(serializedKey) % numPartitions` otherwise.
Default: all
**Compression type**
This option lets you compress your record(s) using any of the available [CompressionType](https://github.com/apache/kafka/blob/3.7/clients/src/main/java/org/apache/kafka/common/record/CompressionType.java) from the KafkaClient
Default: none
**Acks**
This lets you change the `acks` property of the producer.
[Learn more about Kafka producer acks](https://www.conduktor.io/kafka/kafka-producer-acks-deep-dive/) .
Default: all
### Sensible defaults
The following are pre-configured by default:
* If you have connected a schema registry and there's a subject named `-key` and `-value`, the serializers will be populated automatically to the right type (Avro/Protobuf/JsonSchema); otherwise, the `StringSerializer` will be picked.
* A single header `app.name=Conduktor` will be added
## Produced messages panel
Kafka records produced through this screen will be available from the produced message panel, which acts similarly as the consume page, allowing you to review your produced record and check the key, value, headers and metadata.
Click on a record to see its content and metadata:
## Operations
### Import CSV
This feature lets you produce a batch of Kafka records based on a CSV file.
The three required inputs are: the key and value serializer and the input file itself.
The file has to have this structure:
* named headers `key` and `value` must be present. Additional columns will be ignored.
* the separator must be `;`
* double-quoting a field `"` is optional unless it contains either `"` or `;`, then it's mandatory
* escape double-quotes by doubling them `""`
Examples
```undefined theme={null}
# Null key (empty)
key;value
;value without key
# Unused headers
topic;partition;key;value
test;0;my-key;my-value
# Mandatory Double quoting
key;value
order-123;"item1;item2"
# Json data
key;value
order-123;"{""item1"": ""value1"", ""item2"":""value2""}"
```
Click **Import** to start the process. While in progress, you'll see a loading state. Once the import is finished, you'll get a summary message.
### Save and load producer templates
If you are regularly using the same set of producer configuration, you can save your current settings as a template for reuse.
At the bottom of the produce page, click **Save** and enter the name for this template. Click **Load** to see the available templates.
Use this list to add or rename your templates.
## Topic partitions
The **Partitions** tab shows all the partition information associated with the topic. You can switch from the default **Per partition** view to the **Per broker** view.
The **Per partition** view show data available for each partition:
* Total number of records (estimated using EndOffset - BeginOffset)
* Partition size
* Begin and end offsets
* Broker Ids of the partition leader (green) and followers (grey)
The **Per broker** view pivots the data to show for each broker:
* partitions where the broker is Leader
* partitions where the broker is Follower
### Empty Partition
In the **Per partition** view, you can click on the trash icon to remove the Kafka records from this specific partition.
If you need to delete all records from all partitions, click `...` above the per partition/per broker switch and select **Empty topic**.
## Topic linked resources
If you need to find related resources, your can use one of the following tabs to display all the Kafka resources associated to this specific topic.
### Linked consumer groups
The topic consumer groups tab displays the consumer groups associated to the current topic.
### Linked schema registry subjects
The **Schema** tab shows the key subject and value subject associated to your topic, assuming you've defined them using [TopicNameStrategy](https://docs.confluent.io/cloud/current/sr/fundamentals/serdes-develop/index.html#how-the-naming-strategies-work) .
### Linked ACLs
The **ACL** tab displays the list of Kafka permissions associated with the current topic.
Only the permissions from the default Kafka authorizer implementation AclAuthorizer available using AdminClient are listed. If you're using one of Console's other supported ACLs (Aiven, Confluent Cloud), we recommend using the dedicated service account page instead.
### Linked role bindings
The **Role Bindings** tab lists the Confluent Cloud and Confluent Platform role bindings that grant access to the current topic. Each row shows the principal, the role, and the resource pattern that produced the binding.
The tab appears on Confluent Cloud and Confluent Platform clusters. If provider settings are incomplete, Console tells you what to set rather than hiding the tab.
Find out how to enable role bindings on [Confluent Cloud](/guide/tutorials/migrate-confluent-cloud-rbac) or [Confluent Platform](/guide/tutorials/migrate-confluent-platform-rbac).
## Topic graphs
When you browse any topic details page, you'll see the associated graphs for this topic:
* produce and consume rate
* number of records
* disk usage
Graphs can be visualized over 24 hours, 7 or 30 day periods.
You can create an alert for each metric.
## Alerts tab
The topic **Alert** tab lets you see all the active alerts associated to this Kafka cluster. You can edit or toggle them on/off.
## Topic consume
The topic consume page lets you read records as they arrive, narrow them down with filters and open any individual record to inspect its key, value, headers and metadata.
Watch how to tail a topic in real time and filter records as they arrive:
### View data quality policy violations
Available for Gateway clusters with Console v1.42 or later.
When viewing topics on Gateway clusters with data quality Policies and the [mark action applied](/guide/use-cases/enforce-data-quality#mark-action), Console will display the Policy violation information:
* A Policy badge appears in the topic header (for example, "Data Quality" or "Encryption")
* The consume table includes a **Marking violations** column showing Policy and Rule violations for each message
* Hover over Policy names to see the Policy ID and violated Rules (deleted Rules appear greyed out)
* Click Policy names to navigate to Policy details
* Open any message to see full violation details in the **Metadata** tab's **Violations** section, including Policy IDs, Rule IDs, and links to Policy details
[Find out about creating and managing data quality Policies](/guide/use-cases/enforce-data-quality)
## Configure the Kafka consumer
When you access a topic from the topic list page for the first time, a consumer is automatically triggered with default settings:
* show from: `Most recent`
* limit: `500 records`
* partitions: `All`
This default setup lets you quickly browse through the 500 most recent messages produced in the topic.
### Show from
`Show From` defines the starting point for the Kafka consumer in your topic.
Possible values:
* `Most recent` option works differently depending on the **Limit** that you select
* with **Number of records** limit (let's say 500), it sets the starting point in your topic backward relative to **Now**, in order to get your the 500 *most recent* records.
* with **None (live consume)**, it simply set the starting point to **Now**. This lets you consume only the messages produced **after** the consumer was started.
* `Latest hour`, `Today`, `Yesterday` to start the consumer, respectively:
* 60 minutes ago
* at the beginning of the day at 00:00:00 (local timezone based on your browser)
* at the beginning of the day before at 00:00:00 (local timezone)
* `Beginning` to start the consumer from the very beginning of the topic.
* `Date` and `Timestamp` to start from a specific point in time datetime or an epoch
* Date: ISO 8601 DateTime format with offset `2024-12-21T00:00:00+00:00`
* Timestamp: Unix timestamp in **milliseconds** `1734949697000`
* `Offset` to start the consumer at a specific offset, ideal for use with a single **Partition** setting.
* `Consumer Group` to start the consumer from the last offsets committed by a consumer group on this topic.
### Limit
`Limit` defines when your consumer has to stop.
Available options:
* `Latest offset` - stop the consumer upon reaching the end of the topic. The end offsets are calculated when you trigger the search. Records produced after that point **will not appear** in the search results.
* `None (live consume)` - start a live consumer that will look for messages indefinitely.
* `Number of records` - stop the consumer after having sent a certain number of records to the browser. When you have active filters, non-matching records will not count toward this limit.
* `Date` - stop the consumer after reaching the configured date. The ISO 8601 DateTime format with offset `2023-12-21T00:00:00+00:00`.
### Partitions
`Partitions` lets you restrict the consumer to only consume from certain partitions of your topic. By default, records from **all partitions** are consumed.
### Key and value format
`Key format` and `Value format` lets you force the deserializer for your topic.
#### Automatic deserializer
This is the default deserializer. **Automatic** infers the correct deserializer in the following order:
* schema registry deserializers (Avro, Protobuf, Json Schema)
* JsonDeserializer
* StringDeserializer
* ByteDeserializer (fallback)
Automatic deserializer applies to all the records within a topic, based on the one that matches the first record it encounters.
#### Schema IDs in record headers
Confluent Platform 8.2 introduced the option to reference a schema with a globally unique identifier (GUID) in the record headers, instead of prepending a schema ID to the payload. Producers opt in to this by setting `value.schema.id.serializer` to `io.confluent.kafka.serializers.schema.id.HeaderSchemaIdSerializer`.
Console deserializes these records with the **Automatic** deserializer and the schema registry deserializers, so consume and filter them as you would any other record.
Console reads records that carry a schema GUID in their headers, but doesn't yet write them. When you produce or re-process a record, Console prepends the schema ID to the payload. Full GUID support across Console comes in a future release.
#### Custom deserializer
If you have installed them, your custom deserializers will appear here. Optionally, configure them using the `Properties` text and your messages will show as expected.
[Check out the tutorial on installing and configuring custom deserializers in Console](/guide/tutorials/custom-deserializers).
#### JSON deserializer
JSON deserializer will explicitly fail on records that doesn't match a JSON type.
#### Bytes deserializer
Bytes deserializer helps you visualize your records by printing the non-ASCII characters as hexadecimal escape sequences. For instance, the following sequence of bytes:
```undefined theme={null}
00 00 00 00 07 10 49 27 6D 20 41 56 52 4F
```
corresponding to the [wire format of a Schema Registry AVRO message](https://docs.confluent.io/cloud/current/sr/fundamentals/serdes-develop/index.html#wire-format) :
```undefined theme={null}
00 0 Magic Byte (0)
00 00 00 07 1-4 Schema ID (7)
10 49 27 6D 20 41 56 52 4F 5+ serialized AVRO data
```
will be represented like this:
```undefined theme={null}
\x00\x00\x00\x00\x07\x10I'm AVRO
```
## Filter records
Console provides three methods to define filters that will be executed on the server and will only return the records that match.
This is a very powerful feature that allows you to quickly see the records that matter to you, especially in large topics.
### Global search
Global search is the most simple type of filter you can use.
1. Specify whether to look in the Key or in the value.
2. Pick an operator (contains, not contains, equals, not equals).
3. Type your search term.
Internally, the global will treat the record Key or Value as text to apply the operation (contains, equals, ...). This might not be the preferred approach if your record is JSON-ish
### Search in a specific field
You can make your search more fine-grained by activating "Search in a specific field".
Console will generate an autocomplete list by looking at the most recent 50 messages in the topic. If the key you're looking for is not here, you can type it manually. Examples:
`data.event.name`\
`data.event["correlation-id"]`\
`data.clientAddress[0].ip`
### JS search
If you need to construct more advanced filters, you can switch to the advanced view and use plain Javascript to build your filter.
While it is the option that can potentially address the most complex use-cases, it is not the recommended or the fastest one.
### Statistics pop-up
While the consumer is processing, you'll see the following:
## Browse records
### From the main table
Once the search starts, you'll see messages appearing in the main table with three columns: timestamp, key and value.
The timestamp column uses the **local timezone** of the user. For example, if you're producing a message from Dublin, Ireland (UTC+1) at `14:57:38 local time` and you then consume this message from your browser (in Dublin), you'll see `14:57:38`.
However, if another user consumes the same message in Console but from Paris, France (UTC+2), they'll see `15:57:38`.
### Individual records
Click on a record from the list to see the entire record. Use the up/down arrow keys to navigate between messages.
There are three tabs at the top, displaying different elements of your record: data, headers and metadata.
Click **Share** to open a standalone page for the record. From there, you can share it with colleagues or click **Reprocess** to send the record to the same or another topic.
### Data tab
The **Data** tab lets you visualize your record's key and value.
If your record value is serialized with JSON or using a schema registry, it's presented in the table view by default. You can also switch to the JSON view if necessary.
The two views offer different features.
* **Table** view lets you visualize your message field by field and allows you to restrict your search further by applying more filters on individual fields.
Filter types are include and exclude and are available for: string, number and boolean fields (disabled for null and fields contained within lists).
* **JSON** view lets you visualize your message and the **Enable JQ** toggle allows you to create a different projection of your record value.
The basic syntax lets you focus on sub-elements of your record:
```undefined theme={null}
{ foo: .bar } // Renders {"foo": "value of .bar"}
.meta.domain // Renders a single String
{ id, meta } // Renders a new JSON with both elements
```
[Find out more about JQ syntax reference and object construction for advanced use cases](https://jqlang.github.io/jq/manual/#object-construction)
### Headers tab
The **Headers** tab shows all the headers of your Kafka record and lets you find more messages with the same header value. Use the funnel icon to filter the view:
A filter will be created:
While Kafka header values are internally stored as `byte[]`, Console uses *StringDeserializer* to display and filter them. If your producer doesn't write header values as UTF8 strings, this tab might not render properly and the header filter might not work as expected.
### Metadata tab
The **Metadata** tab provides all other information regarding your record that could be useful:
* record partition
* record offset
* record timestamp
* the key and value serializer inferred by the [automatic deserializer](#automatic-deserializer)
* key size and value size (how it's serialized on the broker)
* compression type
* schema ID, if any
## Operations
### Export records in CSV and JSON
You can export records in either JSON or CSV format.
CSV is particularly useful because you can use Console to re-import the records either in a new topic or in the same topic after modifications (if required).
Click on the three dots to see the options:
The resulting files will look like this:
### Reprocess records
This feature lets you pick a record from the list and reprocess it either in the same or in a different topic, while letting you change its content beforehand. You can start from the consumed-message drawer or the standalone page opened with **Share**.
Click **Reprocess** (1), pick a destination topic (2) which takes you to the **Produce** tab (3) with your message pre-filled.
From there you can either produce the message directly or make adjustments:
See [Reprocess messages](/guide/tutorials/reprocess-kafka-messages) for the complete workflow.
### Save and load views
If you're regularly using the same set of consume configuration (e.g. show from, limit) and the filters or if you'd like to share the views with others, you can save your current view as a template.
To create a view, click **Save**. This will save your current view as a template:
Give your view a name add a description and select whether it's **private** or if you want everyone in your **organization** to be able to see it and use it.
To see all views available to you, click the **folder**:
## Most recent 500 messages
When you first land on a topic consume page, the default search is configured with **Most Recent 500 messages**.
The intention is to show you the most relevant messages, split across the partitions.
This algorithm guarantees to return some messages irrespective of **when** the records were produced, which we believe is a good starting point when browsing a topic for the first time.
In most cases, it will give you `500 / num_partitions` messages, per partition. If your topic has:\
**10** partitions, Most Recent **500** will give you **50** messages per partition.\
**2** partitions, Most Recent **500** will give you **250** messages per partition.
Edge cases might occur and the algorithm will account for it seamlessly.
Most recent N messages doesn't work well with filters. This is because the filters will only be applied to those 500 messages instead of a large number of records. We recommend switching to a time-based *ShowFrom* when using filters
## JS filter syntax
JS filter is used to filter Kafka records and is evaluated on each record on the server. It's powerful and can handle complex filters but requires writing JavaScript code.
We recommend that you use the simpler and more performant filters: the global search in a specific field.
The code has to return a boolean.
If your code returns `true`, the record will be included in the results, otherwise it will be skipped.
return value.totalPrice >= 30;
// Selects all the orders having a total price superior to or equal to 30
### Record attributes
When creating JavaScript filters, you may want to access message data or the metadata. See the parameters in the table below for accessing different message attributes.
| Attribute | Types |
| ------------------- | ------ |
| key | Object |
| value | Object |
| headers | Object |
| serializedKeySize | Number |
| serializedValueSize | Number |
| keySchemaId | Number |
| valueSchemaId | Number |
| offset | Number |
| partition | Number |
### Example filters
In this example, we have these two records in our topic:
**Record 1:**
```
{
"key": "order",
"value": {
"orderId": 12345,
"paid": true,
"totalPrice": 50,
"items": [
{
"id": "9cb5cb81-b678-4f96-84dc-70096038eca9",
"name": "beers pack"
},
{
"id": "507b5045-eafd-41a6-afb5-1890f08cfd8e",
"name": "baby diapers pack"
}
]
},
"headers": {
"app": "orders-microservice",
"trace-id": "9f0f004a-70c5-4301-9d28-bf5d7ebf238d"
}
}
```
**Record 2:**
```
{
"key": "order",
"value": {
"orderId": 12346,
"paid": false,
"totalPrice": 10,
"items": [
{
"id": "7f55662e-5ba2-4ab4-9546-45fdd1ca60ca",
"name": "shampoo bottle"
}
]
},
"headers": {
"app": "orders-microservice",
"trace-id": "1076c6dc-bd6c-4d5e-8a11-933b10bd77f5"
}
}
```
Here are some sample filters related to these records:
```
return value.totalPrice >= 30;
// Selects all the orders having a total price superior to or equal to 30
return value.items.length > 1;
// Selects all the orders containing more than one 1 item
return value.orderId == 12345;
// Finds a specific order based in its ID
return !value.paid;
// Selects all the orders that aren't paid
return !headers.includesKey("trace-id");
// Selects all the records not having a trace-id header
const isHighPrice = value.totalPrice >= 30
const moreThanOneItem = value.items.length > 1
return isHighPrice && moreThanOneItem
// Selects all the orders having a total price superior to or equal to 30 and having more than one 1 item
```
## The configuration tab
The topic **Configuration** tab lets you visualize and edit your topic configuration.
On top of the table you have different fields to help you:
* **Search**: filters out the configurations by name. In the example below, we display only configuration with the term `retention`.
* **Raw**/**Friendly**: formats the information either in its original form or in a more human readable way. For instance, `retention.ms` can be represented as raw (`43200000`) or friendly(`12h`). Default is `Friendly`.
* **Show overrides only**: displays only the configurations that are set at topic level, as opposed to the broker level. Default is `true`.
* **Raw view**: shows all the topic configurations as key value pairs.
* **`...`**: lets you do the same three operations that are available on the topic list page: add partitions, empty topic and delete topic.
To edit the configurations, click the **pen** button will open a new screen showing the name and description of the topic configuration, the Kafka default value and the current value:
You can either update the value or revert to the Kafka default which will apply the change directly on the Kafka cluster, similar to the `kafka-configs` commands below.
```shell-session theme={null}
# Update value
$ kafka-configs --bootstrap-server broker_host:port \
--entity-type topics --entity-name my_topic_name \
--alter --add-config cleanup.policy=
# Reset to default
$ kafka-configs --bootstrap-server broker_host:port \
--entity-type topics --entity-name my_topic_name \
--alter --delete-config cleanup.policy
```
## Related resources
* [Find topic risks with Insights](/guide/insights/risk-analysis)
* [Set up topic alerts](/guide/monitor-brokers-apps/alerts)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Kafka monitoring and observability
Source: https://docs.conduktor.io/guide/monitor-brokers-apps
Monitor Kafka broker health, consumer lag, partition status, and throughput with Conduktor Console.
Conduktor offers real-time statistics that provide insights into the most important Kafka metrics.
You can then [set up alerts](/guide/monitor-brokers-apps/alerts) to get notified about the [metrics](#monitoring-metrics) that matter to you.
Use the [Insights dashboard](/guide/insights) to identify infrastructure risks related to replication factors, partition distribution, and data skew across your topics.
## Prerequisite
[Deploy and configure Cortex](/guide/conduktor-in-production/deploy-artifacts/deploy-cortex) to enable monitoring and seamlessly integrate it with your existing systems.
## Ops monitoring
Operations monitoring enhances understanding of your Kafka infrastructure health, allowing you to monitor:
* cluster health state,
* partitions health state,
* topic activity, storage,
* and more.
## Application monitoring
Application monitoring enhances your understanding of your Kafka applications, by monitoring:
* consumer group states and
* consumer group lag
## Monitoring metrics
| Context | Metric | Description |
| ---------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Apps monitoring | Consumer group status | Indicates healthy or critical status based on lag. Critical if max lag/s exceeds 180. |
| Apps monitoring | Lag message count | Number of messages each consumer group is behind per partition. |
| Apps monitoring | Lag(s) | Estimated number of seconds that each consumer group is behind in the topic. |
| Cluster health | Messages count per broker (s) | This metric gives you the ability to gauge how active your producers are. Given batching and other factors this metric will change over time. |
| Cluster health | Messages in per broker (B/s) | This metric provides the amount of bandwidth per broker that's been taken up by producers as well as replication from partitions the broker leads in your cluster. This is useful for planning well distributed leader placement. |
| Cluster health | Messages out per broker (B/s) | This metric indicates how much bandwidth per broker is being utilized by consumers, as well as for replication to the broker. This is useful for planning replica and leader placements. |
| Cluster health | Offline partitions count | Offline partitions can be caused by lingering capacity issues, crashed brokers or cluster-wide faults. This is a critical factor in the health of your cluster - an offline partition can't be produced to or consumed from. If the controller believes a partition is offline, it may not re-assign or bring online a leader. |
| Cluster health | Under-replicated partitions count | Partitions that are under-replicated are a risk to data durability and availability. Under-replicated partitions can happen for various reasons, including an inability for replicas to keep up or network splits. |
| Cluster health | Under min ISR partitions count | Under minimum ISR partitions don't meet the durability requirements to be produced to. If producers that try to produce messages to a partition that's under the specified minimum, ISR will reject the messages and will be forced to handle the exception. |
| Cluster health | Disk - FS usage | If a Kafka broker fills up, its disk durability and availability means that data is at risk. Producers will also be unable to produce to that broker. Filling a broker's disk is also a hard incident to recover from and often involves loss of data. |
| Cluster health | Partitions count | Total number of partitions (including replicas) across the selected Kafka cluster. |
| Cluster health | Active brokers count | Number of active brokers on the selected Kafka cluster. |
| Cluster health | Active partitions count | Total number of partitions active on the selected Kafka cluster. |
| Cluster health | Active controllers count | Total number of active controllers on the selected Kafka cluster. |
| Topic monitoring | Messages count per topic (/s) | Number of messages produced per second, per broker at a topic granularity. |
| Topic monitoring | Topic traffic in (B/s) | Byte rate per second of messages produced, per broker at a topic granularity. |
| Topic monitoring | Topic traffic out (B/s) | Byte rate per second of messages consumed, per broker at a topic granularity. |
| Topic monitoring | Total size of messages | Total size of messages in the topic. |
## Troubleshoot
This happens because of automatic time interval increases for performance optimization:
* **30 day** views use eight hour step intervals
* **7 day** views use two hour step intervals
* Shorter views use smaller step intervals
When brief (lasting a few minutes) spikes occur within these larger time intervals, they are averaged and become invisible in the graph visualization. For example, a five minute data spike will be averaged across a two or eight hour window, making it appear as zero or negligible in the graph.
**Workaround**: use Grafana with a custom PromQL query to get higher resolution:
```promql theme={null}
sum(kafka_topic_produce_rate{cluster_id="gateway", topic="test"})[30d:1m]
```
This shows data over a **30 day** range with **one minute** step intervals, ensuring that brief spikes are not averaged.
Using high-resolution queries may impact performance. Assess the performance trade-offs in your own Grafana instance and adjust the range/step size accordingly.
## Related resources
* [Deploy Cortex](/guide/conduktor-in-production/deploy-artifacts/deploy-cortex)
* [See all available metrics](/guide/reference/console-metrics)
* [Set up alerts](/guide/monitor-brokers-apps/alerts)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Kafka alerts and notifications
Source: https://docs.conduktor.io/guide/monitor-brokers-apps/alerts
Set up Kafka alerts in Conduktor Console for infrastructure and application changes. Integrate with Slack, MS Teams, email, or custom webhooks.
## Overview
Alerts allow you to be notified and react to Kafka infrastructure or application changes as soon as they happen.
You can integrate alerts with **Slack**, **MS Teams** and **email** to receive real-time notifications. Alternatively, set up arbitrary **webhook** destinations with any URL and custom headers. [See how to configure integrations](#configure-integrations).
Alerts are periodically checked for **every minute**.
## Alert metrics
When creating alerts in Console, you can choose from several metrics depending on the resource type. These metrics are calculated using Prometheus queries that run against the data Console collects from your Kafka clusters.
### Topic metrics
| Metric | Description | Prometheus metric | PromQL query |
| ---------------- | ----------------------------------------------------------- | --------------------------- | -------------------------------------------------------------- |
| **MessageCount** | Total number of messages in the topic across all partitions | `kafka_topic_message_count` | `sum(kafka_topic_message_count{cluster_id="...",topic="..."})` |
| **MessageIn** | Rate of messages being produced to the topic per second | `kafka_topic_produce_rate` | `sum(kafka_topic_produce_rate{cluster_id="...",topic="..."})` |
| **MessageOut** | Rate of messages being consumed from the topic per second | `kafka_group_consume_rate` | `sum(kafka_group_consume_rate{cluster_id="...",topic="..."})` |
| **MessageSize** | Total size of the topic in bytes across all partitions | `kafka_topic_size` | `sum(kafka_topic_size{cluster_id="...",topic="..."})` |
### Broker metrics
| Metric | Description | Prometheus metric | PromQL query |
| --------------------------------- | -------------------------------------------------------------------------- | ------------------------------------------------------------ | -------------------------------------------------------------------------------- |
| **MessageIn** | Rate of messages being produced across all topics on the broker per second | `kafka_cluster_produce_rate` | `sum(kafka_cluster_produce_rate{cluster_id="..."})` |
| **MessageOut** | Rate of messages being consumed from the broker per second | `kafka_cluster_consume_rate` | `sum(kafka_cluster_consume_rate{cluster_id="..."})` |
| **MessageSize** | Total size of the broker in bytes | `kafka_broker_size` | `sum(kafka_broker_size{cluster_id="..."})` |
| **PartitionCount** | Number of partition leaders on the broker | `kafka_server_replicamanager_leadercount` | `sum(kafka_server_replicamanager_leadercount{cluster_id="..."})` |
| **OfflinePartitionCount** | Number of partitions with no active leader | `kafka_controller_kafkacontroller_offlinepartitionscount` | `sum(kafka_controller_kafkacontroller_offlinepartitionscount{cluster_id="..."})` |
| **UnderMinIsrPartitionCount** | Number of partitions under minimum ISR | `kafka_server_replicamanager_underminisrpartitioncount` | `kafka_server_replicamanager_underminisrpartitioncount{cluster_id="..."}` |
| **UnderReplicatedPartitionCount** | Number of under-replicated partitions | `kafka_controller_kafkacontroller_underreplicatedpartitions` | `kafka_controller_kafkacontroller_underreplicatedpartitions{cluster_id="..."}` |
### Consumer group metrics
| Metric | Description | Prometheus metric | PromQL query |
| ------------- | ---------------------------------------------- | ------------------------------------------- | ------------------------------------------------------------------------------ |
| **OffsetLag** | Current lag of the consumer group (all topics) | `kafka_consumergroup_group_sum_lag` | `sum(kafka_consumergroup_group_sum_lag{cluster_id="...",group="..."})` |
| **TimeLag** | Lag in seconds (time-based lag) | `kafka_consumergroup_group_max_lag_seconds` | `max(kafka_consumergroup_group_max_lag_seconds{cluster_id="...",group="..."})` |
### Kafka Connect metrics
| Metric | Description | Prometheus metric | PromQL query |
| ------------------- | ----------------------------------------- | ---------------------------- | --------------------------------------------------------------------------------------- |
| **FailedTaskCount** | Number of failed tasks for this Connector | `kafka_connect_failed_tasks` | `kafka_connect_failed_tasks{cluster_id="...",connect_cluster_id="...",connector="..."}` |
[Check out all the available metrics in the resource reference](/guide/reference/console-metrics).
## Create alerts
Alerts can be created for most resources (e.g. brokers or topics) in Conduktor Console.
To create an alert:
1. Navigate to the required resource.
2. Click on the **Alerts** tab.
3. Click the **+** button to create an alert.
In the **New Alert** pane enter the required details. Each alert has to have:
* a unique name
* a metric you want to measure (e.g. MessageCount, MessageIn, MessageSize). [See available metrics](#alert-metrics).
* an operator and value for the threshold
* an owner. This can be a group, an application instance or an individual user. The ownership will determine who can edit the alert. You can't change the owner after you create the alert.
If the owner of an alert is deleted, the associated alerts will also be deleted.
To let application team members work with alerts owned by an application instance, assign the `alertManage` [instance permission](/guide/reference/self-service-reference#instance-permissions-reference) through an ApplicationGroup. Data quality alerts can only be owned by a group or a user.
* (optional) a description explaining what the alert does or why you created it. This is useful if you're creating an external alert.
* a destination: internal (Conduktor Console) or external (Slack, Teams, email or Webhook). Each alert can have **only one external destination**. To verify that it works, click **Send test**.
## Alert destinations
Once you've configured your integrations, you can choose from several alert destinations:
### Slack
* **Destination channel**: select from available Slack channels (your Slack app has to be invited to the channel first).
### Microsoft Teams
* **Webhook URL**: use the channel webhook URL from your Teams workflow configuration.
### Email
* **Destination email**: specify the recipient's email - this is where the notifications will be sent.
* **Subject**: customize the email subject line to make them easily identifiable.
* **Body**: the body of the email. You can use handlebars syntax (e.g., `{{clusterName}}`, `{{threshold}}`, for url because the template engine can escape the whole url, you will have to use `{{{url}}}` to not escape the url) to embed alert variables dynamically.
Email alerts will be sent from the sender address configured in your email integration settings.
### Webhook
* **Method**: select the HTTP method to use for the webhook request (POST, PUT).
* **URL**: any webhook endpoint that accepts POST requests.
* **Custom Headers**: add custom headers, as needed.
* **Body**: receives structured JSON payload with alert details and metadata. You can use handlebars syntax (e.g., `{{clusterName}}`, `{{threshold}}`, for url because the template engine can escape the whole url, you will have to use `{{{url}}}` to not escape the url) to embed alert variables dynamically.
* **Authentication**: configure authentication, if required by your webhook endpoint (basic auth or bearer token).
#### Using environment variables in webhooks
You can reference environment variables in your webhook URL, headers, body, and authentication fields using the Mustache syntax `{{env.VAR_NAME}}`. **Only environment variables prefixed with `CDK_WEBHOOK_`** are available. The prefix is stripped when referencing them in templates.
For example, if you set `CDK_WEBHOOK_API_KEY=my-secret-key` as an environment variable on the Console container, you can reference it in your webhook configuration:
* **Header**: `Authorization: Bearer {{env.API_KEY}}`
* **Body**: `{"token": "{{env.API_KEY}}", "alert": "{{alertName}}"}`
This allows you to keep sensitive credentials like API keys out of the webhook configuration stored in the database.
If an environment variable is not found, it will be replaced with an empty string.
Click **Send test** for any external destinations to verify your configuration before saving the alert.
## Alert processing
Alert processing involves multiple components with different timing intervals:
* **Console metrics collection**: Console's indexer collects metrics every **30 seconds** by default from your Kafka clusters
* **Prometheus evaluation**: alert conditions are evaluated every **15 seconds** by Prometheus
* **Alert manager notification**: waits **30 seconds** before sending notifications after an alert triggers
* **Firing alert notifications**: alerts with `firing` status will trigger a notification every **hour**
### Alert resolution criteria
Alerts will resolve (stop firing) when the metric value no longer meets the configured threshold condition.
The actual timing between metric changes and alert notifications can vary based on when each interval aligns. In the worst case, it could take up to 75 seconds (30s metrics collection + 15s Prometheus evaluation + 30s Alert Manager delay) for a new condition to trigger a notification.
## Manage alerts
You can **deactivate an alert without deleting** it. Deactivated alerts won't send notifications or record history/status until reactivated.
You can also test, duplicate or delete the alert at any point.
Select the event and click the three dots in the top-right corner.
### Alert history
Each alert will keep a history of when it was triggered and the status over time. This gives you an overview of successful or failed deliveries. Alerts with the `firing` status will trigger a notification every hour.
### Alert list
Go to **Settings > Alerts** to see all the alerts, grouped by owner. You can sort the view by name, status or destination and activate/deactivate as required. You can also customize this view to show/hide columns relevant to you.
## Configure integrations
Conduktor Console can integrate with **Slack**, **MS Teams** and any system that receives webhooks.
Enabling the **Slack** integration requires creating a Slack application, installing this to your Slack workspace, inviting your app to the channel and adding the application token to Conduktor.
After enabling the **MS Teams** integration you will need a Teams webhook URL to create alerts with this destination.
Disabling an integration used by existing alerts will not delete the alerts, but will prevent them from sending notifications externally.
### Slack integration
As part of configuring the Slack integration in Conduktor, you will find steps for [creating a Slack application](https://api.slack.com/apps) (a bot) using the app manifest template (provided in Conduktor). This app needs to be installed to your Slack workspace and the OAuth token added to the Conduktor integration.
Once configured you'll need to add the application to the Slack channels you wish to send alerts too. You can do this by typing `/invite` in the channel and choose the application you have just created., or through managing the app within Slack.
Channels that have not had the application invited cannot be set as destinations for alerts. You'll get a `not_in_channel` error. Once the applications is invited to the channel, you'll be able to send alerts to that Slack channel.
### Email integration
Email integration allows you to receive notifications directly to your inbox. Setting up email alerts requires configuring your SMTP server details and authentication credentials.
To set up email alerts, you have to configure:
1. **Server settings**: configure your SMTP server details to enable email delivery
* SMTP server: enter your email server address with a port (e.g., `mail.company.com:587`)
* TLS encryption: (recommended) enable TLS encryption for secure email transmission
* SSL configuration: you can skip SSL checks or upload custom certificates for enhanced security
2. **Authentication**: provide credentials for authenticating with your email server
* Sender email: the email address that will appear as the sender for all alerts (e.g., `conduktor-alerts@company.com`)
* Username and password: your email server authentication credentials
* Test configuration: click **Test configuration** to validate your configuration before saving changes
Once your email integration is configured, you can create email alerts with customizable destination addresses, subjects and content. You can use handlebars syntax (e.g., `{{clusterName}}`, `{{threshold}}`) to embed alert variables dynamically in the body of the email.
#### Email alerts used for Teams
We directly integrate with Microsoft Teams using Workflows. If you're not allowed to use this feature, you can use the email integration to send alerts to Microsoft Teams channels.
For this, first set up the [email integration](#email-integration) and get the channel's email address. You can find this in the channel settings under **Get email address**.
From there, you can select which email addresses is authorized to send emails to this channel and copy it to use in your Conduktor alert.
### Microsoft Teams integration using Workflows
This guide focusses on the Workflows integration, rather than webhooks which are used with Microsoft teams to handle alerting from Conduktor's monitoring.
You can send notifications to different rooms for different alerts. However, this room must be a Microsoft Teams *standard room* and not a *shared* room.
#### Workflows setup
1. Open the Workflows app within the chat or channel by right-clicking on the conversation or by clicking **...** then selecting **Workflows**.
2. In the Workflow app, click **+ New flow** or select the Create tab and choose **Post to a channel when a webhook request is received** template.
3. Choose a name for this flow (or use the default **Post to a channel when a webhook request is received**) and choose who will own this workflow. It should default to the user logged into Teams. Click **Next**.
4. Pick the team and channel you want to post to, then click **Create flow**. This page may take some time to load.
5. Once you click on **Create flow** in the previous step, the next page should show you the URL for the webhook associated with the workflow you just created. *Copy and paste this somewhere* as it will be used in a later step and click **Done**.
6. Go back to the Home tab in Workflows and edit the newly created workflow by clicking **...** then **Edit**.
7. On this page:
1. Expand **When a Teams webhook request is received** to see the URL endpoint again, if you've not saved it from the previous step. You can also change *who can trigger the flow*. In this example we'll use *Anyone*.
2. Expand the second step and for **Select an output from previous steps**, pick **Attachments**. For **Adoptive Card** select **content**. The rest of the form should be filled in but you can change the channel and team here.
8. Open Console and [create a new alert](#create-alerts) or go to an existing one. In the **Alert destination** page select **Teams**. Paste the webhook URL and click **Save**. You can test your connection by clicking **Send test**.
9. Fire some alerts and you should now see alert messages being posted in Microsoft Teams under the specified channel:
### Webhook integration
You can use a tool like [webhook.site](https://webhook.site) to check the payload sent for webhook notifications.
Here's an example:
```json theme={null}
{
"data": {
"metatadata": {
"name": "high produce rate",
"appInstance": null,
"group": "menu-team",
"user": null,
"updatedAt": "2025-01-21T14:13:34.729783423Z",
"updatedBy": "mary@example.com",
"lastTriggeredAt": null,
"status": "Pending"
},
"spec": {
"cluster": "prod-internal",
"threshold": 1000,
"operator": "GreaterThan",
"metric": "MessageIn",
"promQl": "sum(rate(kafka_partition_latest_offset{cluster_id=\"shadow-julien\"}[1m])) > 1000",
"description": "Miguel knows how to fix this",
"displayName": null,
"destination": {
"url": "https://example.com/webhook/f0c608e3-aca3-4b07-8d4d-7226f629ade9",
"method": "POST",
"headers": {
"example": "123"
},
"authentification": null,
"type": "Webhook"
},
"disable": null,
"type": "BrokerAlert"
}
},
"status": "resolved"
}
```
### Audit log events
| **Event type** | **Description** |
| ----------------------- | ---------------------- |
| **Kafka.Alert.Create** | An alert is created. |
| **Kafka.Alert.Update** | An alert is updated. |
| **Kafka.Alert.Delete** | An alert is deleted. |
| **Kafka.Alert.Trigger** | An alert is triggered. |
## Related resources
* [View metrics reference](/guide/reference/console-metrics)
* [Find risks with Insights](/guide/insights/risk-analysis)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Advanced pattern support Interceptors
Source: https://docs.conduktor.io/guide/reference/advanced-pattern-support
Advanced patterns in Conduktor Gateway Interceptors: cache data, handle large messages, filter and project topic records with topic views, and concentrate topic partitions.
Conduktor Gateway advanced pattern Interceptors let you do things standard Kafka can't: cache repeated fetches, send oversized messages, filter and reshape topic content server-side, and collapse many virtual topics onto fewer physical ones. Producers and consumers keep using standard Kafka clients — Gateway handles it at the proxy layer.
## Available patterns
| Pattern | Description |
| ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| [Data caching](/guide/use-cases/cache-kafka-data) | Cache fetch responses at the proxy layer to reduce broker load and repeated consumer reads |
| [Large messages](/guide/use-cases/manage-large-messages#large-messages) | Split oversized messages at produce time and reassemble them transparently at consume time |
| [Topic views](/guide/conduktor-concepts/logical-topics#topic-views) | Present a non-materialized, filtered and projected view of a physical topic using SQL, without duplicating the underlying data |
| [Topic concentration](/guide/conduktor-concepts/logical-topics#concentrated-topics) | Map multiple virtual topics onto a single physical Kafka topic to reduce partition count and broker overhead |
[Topic views](/guide/conduktor-concepts/logical-topics#topic-views) replace [SQL topics](/guide/conduktor-concepts/logical-topics#sql-topics) and [CEL topics](/guide/conduktor-concepts/logical-topics#filter-topics-with-cel), which are deprecated. Use topic views for new work — the deprecated plugins will be removed once topic views cover the remaining functionality.
## When to use advanced patterns
* **Reduce broker load** — use caching to serve repeated fetch requests without hitting the broker every time
* **Work around Kafka limits** — large message handling lets you produce and consume messages that exceed `message.max.bytes` without reconfiguring brokers
* **Control data exposure** — topic views let you expose a filtered, projected subset of a topic's records to specific consumers without duplicating the underlying data
* **Optimize partition usage** — topic concentration consolidates logical namespaces onto fewer physical partitions, reducing overhead in multi-tenant environments
## Related resources
* [Logical topics overview](/guide/conduktor-concepts/logical-topics)
* [Topic views](/guide/conduktor-concepts/logical-topics#topic-views)
* [TopicView resource reference](/guide/reference/gateway-reference#topicview)
* [Migrate a Kafka Streams transformer to a Topic View](/guide/tutorials/migrate-kafka-streams-to-topic-view)
* [Large message use case](/guide/use-cases/manage-large-messages)
* [Cache data use case](/guide/use-cases/cache-kafka-data)
* [Data quality Interceptors](/guide/reference/data-quality)
# Client metrics Interceptor
Source: https://docs.conduktor.io/guide/reference/client-metrics
Collect client-side Kafka metrics through KIP-714 and forward them to an OpenTelemetry collector, enriched with Gateway identity labels.
The client metrics Interceptor terminates [Kafka Improvement Proposal 714 (KIP-714)](https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability) client telemetry in Gateway and forwards it to an OpenTelemetry (OTLP) collector. Kafka clients from 3.7.0 onward push metrics by default (`enable.metrics.push=true`), so you collect them without changing client code or configuration. Older clients don't support KIP-714 and never push.
Since Gateway handles the telemetry itself, your Kafka broker doesn't need any KIP-714 setup: no `CLIENT_METRICS` resources and no client telemetry reporter plugin. This also works against brokers that don't support KIP-714 at all.
Use this Interceptor to observe producer and consumer health from a single place, even when clients run in networks you don't control or can't instrument directly.
## How it works
Gateway tells each client which metrics to collect and how often to send them. Clients push their metrics to Gateway, which enriches each one with identity labels and forwards it to your collector. None of this reaches your Kafka broker.
The identity labels let you group and filter metrics by client and by Virtual Cluster:
| Label | Description |
| ----------------------- | ---------------------------------------------------- |
| `client_id` | The client's configured `client.id` |
| `client_instance_id` | Unique per client instance, assigned through KIP-714 |
| `client_source_address` | Source IP address of the client connection |
| `client_source_port` | Source port of the client connection |
| `principal` | Authenticated principal (user) of the client |
| `vcluster` | Virtual Cluster the client connected to |
## Configure the client metrics Interceptor
| Name | Type | Default | Description |
| :----------------- | :------------ | :--------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `requestedMetrics` | List\[String] | `["*"]` | Metric-name prefixes served to clients. Use `["*"]` for all metrics, an empty list for none, or a list of literal prefixes to filter. |
| `pushIntervalMs` | Integer | `300000` | Interval in milliseconds clients wait between pushes. Bounded to Kafka's accepted range of `100` to `3600000` (1 hour). |
| `maxBytes` | Integer | `1048576` | Maximum size in bytes of a client push accepted for export. Matches KIP-714's `telemetry.max.bytes`. |
| `exporterClass` | String | Built-in OTLP exporter | Fully-qualified name of the exporter class. Defaults to the built-in OTLP exporter, `io.conduktor.gateway.interceptor.clientmetrics.exporter.otlp.OtlpClientTelemetryExporter`. See [Choose an exporter](#choose-an-exporter). |
| `exporterConfig` | Map | `{}` | Configuration passed to the exporter selected by `exporterClass`. For the default OTLP exporter, see [OTLP exporter configuration](#otlp-exporter-configuration). |
Gateway serves one subscription to all clients. Per-client subscriptions aren't supported yet.
### Choose an exporter
An exporter forwards each metric out of Gateway. It's a standard Kafka telemetry plugin — a class implementing `org.apache.kafka.server.telemetry.ClientTelemetryExporterProvider`, loaded the same way Kafka loads `metric.reporters`. Gateway ships an OTLP exporter and uses it by default.
To forward metrics somewhere else, set `exporterClass` to your own implementation of that interface and put its settings under `exporterConfig`. The settings a given exporter accepts are defined by that exporter — the ones below are specific to the default OTLP exporter.
Add a custom exporter to Gateway's classpath by bind-mounting its JAR into the container's `/app/lib/` directory.
### OTLP exporter configuration
These settings apply to the default OTLP exporter. Set them under `exporterConfig`. Any value can reference a Gateway environment variable with the `${MY_ENV_VAR}` format.
| Name | Type | Default | Description |
| :------------ | :------ | :------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------- |
| `endpoint` | String | `${OTEL_EXPORTER_OTLP_ENDPOINT}` | `host:port` of the OTLP gRPC collector. Required. Configuration fails if neither this nor the environment variable is set. |
| `insecure` | Boolean | `false` | Use a plaintext (non-TLS) gRPC channel. Falls back to `OTEL_EXPORTER_OTLP_INSECURE`. A `http`/`https` scheme on `endpoint` takes precedence. |
| `compression` | Enum | `gzip` | Outbound compression: `none` or `gzip`. Falls back to `OTEL_EXPORTER_OTLP_COMPRESSION`. |
| `truststore` | Object | JVM default | JKS truststore to verify the collector's TLS certificate. Ignored on a plaintext connection. |
| `keystore` | Object | | JKS keystore providing a client certificate for mutual TLS (mTLS). Ignored on a plaintext connection. |
The `truststore` and `keystore` objects take these fields:
| Object | Fields | Environment variable fallback |
| :----------- | :------------------------------------- | :------------------------------------------------------------------------------- |
| `truststore` | `trustStorePath`, `trustStorePassword` | `OTEL_EXPORTER_OTLP_TRUST_STORE_PATH`, `OTEL_EXPORTER_OTLP_TRUST_STORE_PASSWORD` |
| `keystore` | `keyStorePath`, `keyStorePassword` | `OTEL_EXPORTER_OTLP_KEY_STORE_PATH`, `OTEL_EXPORTER_OTLP_KEY_STORE_PASSWORD` |
## Client metrics Interceptor example
This example collects all metrics, tells clients to push every 5 minutes, and forwards to a plaintext collector.
```bash theme={null}
curl \
--request PUT \
--url 'http://localhost:8888/gateway/v2/interceptor' \
--header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
--header 'Content-Type: application/json' \
--data-raw '{
"kind": "Interceptor",
"apiVersion": "gateway/v2",
"metadata": {
"name": "client-metrics"
},
"spec": {
"pluginClass": "io.conduktor.gateway.interceptor.clientmetrics.ClientMetricsPlugin",
"priority": 100,
"config": {
"requestedMetrics": [
"*"
],
"pushIntervalMs": 300000,
"exporterConfig": {
"endpoint": ":4317",
"insecure": true,
"compression": "gzip"
}
}
}
}'
```
```yaml theme={null}
apiVersion: gateway/v2
kind: Interceptor
metadata:
name: client-metrics
scope:
vCluster: passthrough
spec:
pluginClass: io.conduktor.gateway.interceptor.clientmetrics.ClientMetricsPlugin
priority: 100
config:
requestedMetrics:
- "*"
pushIntervalMs: 300000
exporterConfig:
endpoint: ":4317"
insecure: true
compression: gzip
```
To forward over TLS, drop `insecure` and set the collector endpoint to an `https` address or supply a `truststore` (and a `keystore` for mutual TLS).
## Related resources
* [Use and configure Interceptors](/guide/conduktor-concepts/interceptors)
* [View the full Interceptor list](/guide/reference/interceptor-reference)
* [KIP-714: Client metrics and observability](https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability)
# Console metrics reference
Source: https://docs.conduktor.io/guide/reference/console-metrics
Prometheus metrics exposed by Conduktor Console for Kafka resources and Console health. Use with Grafana, Cortex, Mimir.
Console exposes metrics using the Prometheus format for your Kafka resources and Console health that you can scrape and send to your external log management system.
Since Console v1.42.0, you can protect access to the `/monitoring/metrics` and `/api/monitoring/metrics` endpoints with basic auth. When configured, your monitoring backend must provide credentials to scrape these metrics. [Learn how to configure basic auth for metrics](/guide/conduktor-in-production/deploy-artifacts/deploy-external-monitoring#basic-auth-for-metrics-scraping).
## Kafka metrics reference
The metrics endpoint is located on `/monitoring/metrics` of your deployed Console instance. Data points are refreshed every 30 seconds.
| Metric name | Metric type | Labels | Description |
| ----------------------------------------------------------------------- | ----------- | ------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------- |
| **Consumer group metrics** | | | |
| `kafka_consumergroup_group_lag` | gauge | `cluster_id`, `cluster_name`, `group`, `topic`, `partition` | Absolute Lag of the consumer group on this topic-partition |
| `kafka_consumergroup_group_lag_seconds` | gauge | `cluster_id`, `cluster_name`, `group`, `topic`, `partition` | Lag in seconds of the consumer group on this topic-partition |
| `kafka_consumergroup_group_offset` | gauge | `cluster_id`, `cluster_name`, `group`, `topic`, `partition` | Last committed offset of the consumer group on this topic-partition |
| **Consumer group metrics (aggregated)** | | | |
| `kafka_consumergroup_group_max_lag` | gauge | `cluster_id`, `cluster_name`, `group` | Max group offset lag |
| `kafka_consumergroup_group_max_lag_seconds` | gauge | `cluster_id`, `cluster_name`, `group` | Lag in seconds of the consumer group (all topics) |
| `kafka_consumergroup_group_sum_lag` | gauge | `cluster_id`, `cluster_name`, `group` | Sum of Absolute Lag of the consumer group (all topics) |
| `kafka_consumergroup_group_topic_sum_lag` | gauge | `cluster_id`, `cluster_name`, `group`, `topic` | Sum of Absolute Lag of the consumer group on this topic (all partitions) |
| `kafka_consumergroup_group_topic_max_lag_seconds` | gauge | `cluster_id`, `cluster_name`, `group`, `topic` | Max lag in seconds of the consumer group on this topic |
| `kafka_group_consume_rate` | gauge | `cluster_id`, `cluster_name`, `topic` | Group consume rate |
| **Kafka cluster and broker metrics** | | | |
| `kafka_cluster_consume_rate` | gauge | `cluster_id`, `cluster_name` | Cluster consume rate |
| `kafka_cluster_produce_rate` | gauge | `cluster_id`, `cluster_name` | Number of messages produced per second |
| `kafka_controller_kafkacontroller_activebrokercount` | gauge | `cluster_name`, `cluster_id` | Number of active brokers on the Kafka cluster |
| `kafka_controller_kafkacontroller_activecontrollercount` | gauge | `cluster_name`, `cluster_id` | Number of active controllers on the Kafka cluster |
| `kafka_controller_kafkacontroller_globalpartitioncount` | gauge | `cluster_name`, `cluster_id` | Total number of partitions on the Kafka cluster |
| `kafka_controller_kafkacontroller_offlinepartitionscount` | gauge | `cluster_name`, `cluster_id` | Number of partitions with no active Leader |
| `kafka_broker_size` | gauge | `cluster_id`, `cluster_name`, `broker_id`, `broker_host` | Size in bytes of the current broker |
| `kafka_server_replicamanager_leadercount` | gauge | `cluster_name`, `cluster_id`, `broker_id`, `broker_host` | Number of partition Leaders on this Broker |
| `kafka_server_replicamanager_partitioncount` | gauge | `cluster_name`, `cluster_id`, `broker_id`, `broker_host` | Number of partitions on this Broker |
| `kafka_server_replicamanager_underminisrpartitioncount` | gauge | `cluster_name`, `cluster_id` | Number of partitions with replicas that don't meet their minimum ISR |
| `kafka_server_replicamanager_underreplicatedpartitions` | gauge | `cluster_name`, `cluster_id` | Number of partitions with replicas that don't meet their replication factor |
| **Kafka connect metrics** | | | |
| `kafka_connect_failed_tasks` | gauge | `cluster_name`, `cluster_id`, `connect_cluster_id`, `connector` | Number of failed tasks for this Connector |
| `kafka_connect_total_tasks` | gauge | `cluster_name`, `cluster_id`, `connect_cluster_id`, `connector` | Total number of tasks configured for this Connector |
| **Kafka topic and partition metrics** | | | |
| `kafka_topic_message_count` | gauge | `cluster_id`, `cluster_name`, `topic` | Message count of the current topic |
| `kafka_topic_produce_rate` | gauge | `cluster_id`, `cluster_name`, `topic` | Number of messages produced per second |
| `kafka_topic_size` | gauge | `cluster_id`, `cluster_name`, `topic` | Size in bytes of the current topic |
| `kafka_cluster_partition_underminisr` | gauge | `cluster_name`, `cluster_id`, `topic`, `partition` | `1` when the partition is under min ISR |
| `kafka_cluster_partition_underreplicated` | gauge | `cluster_name`, `cluster_id`, `topic`, `partition` | `1` when the partition is under-replicated |
| `kafka_partition_earliest_offset` | gauge | `cluster_id`, `cluster_name`, `topic`, `partition` | Earliest offset of the current topic-partition |
| `kafka_partition_latest_offset` | gauge | `cluster_id`, `cluster_name`, `topic`, `partition` | Latest offset of the current topic-partition |
| **Others** | | | |
| `kafka_consumergroup_poll_time_ms` | gauge | `cluster_id`, `cluster_name` | Group time poll time |
| `kafka_controller_controllerstats_uncleanleaderelectionspersec_created` | gauge | `cluster_name`, `cluster_id` | Counter for kafka\_controller\_controllerstats\_uncleanleaderelectionspersec metric |
| `kafka_controller_controllerstats_uncleanleaderelectionspersec_total` | counter | `cluster_name`, `cluster_id` | Counter for kafka\_controller\_controllerstats\_uncleanleaderelectionspersec metric |
| `kafka_log_log_size` | gauge | `broker_host`, `partition`, `broker_id`, `topic`, `cluster_name`, `cluster_id` | Size in bytes of the current topic-partition |
| **Data quality metrics** | | | |
| `data_quality_policy_msg_evaluated_total` | counter | `policy_name` | Number of messages evaluated per data quality policy |
| `data_quality_policy_msg_evaluated_created` | gauge | `policy_name` | Timestamp when data\_quality\_policy\_msg\_evaluated metric was created |
| `data_quality_rule_msg_evaluated_total` | counter | `rule_name`, `policy_name` | Number of messages evaluated per data quality rule |
| `data_quality_violations_per_policy_total` | counter | `policy_name` | Number of violations per data quality policy |
| `data_quality_violations_per_policy_created` | gauge | `policy_name` | Timestamp when data\_quality\_violations\_per\_policy metric was created |
| `data_quality_violations_per_rule_total` | counter | `rule_name`, `policy_name` | Number of violations per data quality rule |
| `data_quality_violations_per_rule_created` | gauge | `rule_name`, `policy_name` | Timestamp when data\_quality\_violations\_per\_rule metric was created |
Note that the label `cluster_name` is obsolete and will be removed soon. You should use `cluster_id` instead.
## Console metrics
The metrics endpoint is located on `/api/metrics` of your deployed Console instance.
| Metric name | Metric type | Labels | Description |
| -------------------------------------------------------------- | ----------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------- |
| **Console API** | | | |
| `console_api_request_active` | gauge | `path`, `method` | |
| `console_api_request_duration_seconds` | histogram | | |
| `console_api_request_duration_seconds_count` | | `path`, `method`, `status`, `phase` | |
| `console_api_request_duration_seconds_max` | | `path`, `method`, `status`, `phase` | |
| `console_api_request_duration_seconds_min` | | `path`, `method`, `status`, `phase` | |
| `console_api_request_duration_seconds_sum` | | `path`, `method`, `status`, `phase` | |
| `console_api_request_total` | counter | `path`, `method`, `status` | |
| **Console cache** | | | |
| `console_cache_cluster_reachable_hits_total` | counter | | Number of cache hits for cluster\_reachable |
| `console_cache_cluster_reachable_misses_total` | counter | | Number of cache miss for cluster\_reachable |
| `console_cache_cluster_reachable_size` | gauge | | Total cache size for cluster\_reachable |
| `console_cache_kafka_admin_hits_total` | counter | | Number of cache hits for kafka\_admin |
| `console_cache_kafka_admin_misses_total` | counter | | Number of cache miss for kafka\_admin |
| `console_cache_kafka_admin_size` | gauge | | Total cache size for kafka\_admin |
| `console_cache_schema_registry_clients_hits_total` | counter | | Number of cache hits for schema\_registry\_clients |
| `console_cache_schema_registry_clients_misses_total` | counter | | Number of cache miss for schema\_registry\_clients |
| `console_cache_schema_registry_clients_size` | gauge | | Total cache size for schema\_registry\_clients |
| **Kafka cluster indexing** | | | |
| `console_indexer_kafka_cluster_duration` | histogram | | Duration in milliseconds to index Kafka cluster |
| `console_indexer_kafka_cluster_duration_count` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_cluster_duration_max` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_cluster_duration_min` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_cluster_duration_sum` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_cluster_failed_count` | gauge | | Number of failed Kafka cluster indexing tasks |
| `console_indexer_kafka_cluster_skipped_count` | gauge | | Number of skipped Kafka cluster indexing tasks |
| `console_indexer_kafka_cluster_succeeded_count` | gauge | | Number of succeeded Kafka cluster indexing tasks |
| `console_indexer_kafka_cluster_timeout_count` | gauge | | Number of timed out Kafka cluster indexing tasks |
| `console_indexer_kafka_describe_topics_duration` | histogram | | Duration in milliseconds to describes all topics per Kafka cluster |
| `console_indexer_kafka_describe_topics_duration_count` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_describe_topics_duration_max` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_describe_topics_duration_min` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_describe_topics_duration_sum` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_get_consumers_groups_duration` | histogram | | Duration in milliseconds to get all consumers groups state per Kafka cluster |
| `console_indexer_kafka_get_consumers_groups_duration_count` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_get_consumers_groups_duration_max` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_get_consumers_groups_duration_min` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_get_consumers_groups_duration_sum` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_get_deployed_connectors_duration` | histogram | | Duration in milliseconds to list all deployed connector per Kafka cluster |
| `console_indexer_kafka_get_deployed_connectors_duration_count` | | `time_unit`, `technical_id`, `cluster_type`, `kafka_connect` | |
| `console_indexer_kafka_get_deployed_connectors_duration_max` | | `time_unit`, `technical_id`, `cluster_type`, `kafka_connect` | |
| `console_indexer_kafka_get_deployed_connectors_duration_min` | | `time_unit`, `technical_id`, `cluster_type`, `kafka_connect` | |
| `console_indexer_kafka_get_deployed_connectors_duration_sum` | | `time_unit`, `technical_id`, `cluster_type`, `kafka_connect` | |
| `console_indexer_kafka_get_topics_configs_duration` | histogram | | Duration in milliseconds to get all topics configuration per Kafka cluster |
| `console_indexer_kafka_get_topics_configs_duration_count` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_get_topics_configs_duration_max` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_get_topics_configs_duration_min` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_get_topics_configs_duration_sum` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_get_topics_partitions_duration` | histogram | | Duration in milliseconds to get all topics partitions state per Kafka cluster |
| `console_indexer_kafka_get_topics_partitions_duration_count` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_get_topics_partitions_duration_max` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_get_topics_partitions_duration_min` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_get_topics_partitions_duration_sum` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_init_cluster_connection_duration` | histogram | | Duration in milliseconds to initialize the connection per Kafka cluster |
| `console_indexer_kafka_init_cluster_connection_duration_count` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_init_cluster_connection_duration_max` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_init_cluster_connection_duration_min` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_init_cluster_connection_duration_sum` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_list_topics_duration` | histogram | | Duration in milliseconds to list topics per Kafka cluster |
| `console_indexer_kafka_list_topics_duration_count` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_list_topics_duration_max` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_list_topics_duration_min` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_list_topics_duration_sum` | | `time_unit`, `technical_id`, `cluster_type` | |
| `console_indexer_kafka_total_clusters_count` | gauge | | Total number of Kafka clusters |
| `console_indexer_kafka_total_duration` | histogram | | Total duration in milliseconds to index Kafka cluster |
| `console_indexer_kafka_total_duration_count` | | `time_unit` | |
| `console_indexer_kafka_total_duration_max` | | `time_unit` | |
| `console_indexer_kafka_total_duration_min` | | `time_unit` | |
| `console_indexer_kafka_total_duration_sum` | | `time_unit` | |
| **Kafka connect indexing** | | | |
| `console_indexer_kafka_connect_cluster_failed_count` | gauge | | Number of failed Kafka Connect server indexing tasks |
| `console_indexer_kafka_connect_cluster_skipped_count` | gauge | | Number of skipped Kafka Connect server indexing tasks |
| `console_indexer_kafka_connect_cluster_succeeded_count` | gauge | | Number of succeeded Kafka Connect server indexing tasks |
| `console_indexer_kafka_connect_cluster_timeout_count` | gauge | | Number of timed out Kafka Connect server indexing tasks |
| `console_indexer_kafka_connect_total_clusters_count` | gauge | | Total number of Kafka Connect server |
| `console_indexer_kafka_connect_total_duration` | histogram | | Total duration in milliseconds to index Kafka Connect server |
| `console_indexer_kafka_connect_total_duration_count` | | `time_unit` | |
| `console_indexer_kafka_connect_total_duration_max` | | `time_unit` | |
| `console_indexer_kafka_connect_total_duration_min` | | `time_unit` | |
| `console_indexer_kafka_connect_total_duration_sum` | | `time_unit` | |
| **Schema registry indexing** | | | |
| `console_indexer_schema_registry_cluster_failed_count` | gauge | | Number of failed schema registry server indexing tasks |
| `console_indexer_schema_registry_cluster_skipped_count` | gauge | | Number of skipped schema registry server indexing tasks |
| `console_indexer_schema_registry_cluster_succeeded_count` | gauge | | Number of succeeded schema registry server indexing tasks |
| `console_indexer_schema_registry_cluster_timeout_count` | gauge | | Number of timed out schema registry server indexing tasks |
| `console_indexer_schema_registry_total_clusters_count` | gauge | | Total number of schema registry server |
| `console_indexer_schema_registry_total_duration` | histogram | | Total duration in milliseconds to index schema registry server |
| `console_indexer_schema_registry_total_duration_count` | | `time_unit` | |
| `console_indexer_schema_registry_total_duration_max` | | `time_unit` | |
| `console_indexer_schema_registry_total_duration_min` | | `time_unit` | |
| `console_indexer_schema_registry_total_duration_sum` | | `time_unit` | |
| **License metrics** | | | |
| `console_license_remaining_days` | gauge | | The number of days remaining on your Console license |
## Related resources
* [Set up alerts and notifications](/guide/monitor-brokers-apps/alerts)
* [Deploy Cortex](/guide/conduktor-in-production/deploy-artifacts/deploy-cortex)
* [Give us feedback/request a feature](https://conduktor.io/roadmap)
# Console resources
Source: https://docs.conduktor.io/guide/reference/console-reference
YAML resource definitions for Conduktor Console: users, groups, permissions, RBAC bindings, cluster configurations, and certificate management schemas.
## ConsoleGroup
* **API key(s):** AdminToken
* **Managed with:** UI, CLI, API, TF
* **Labels support:** Missing
Creates a group with members and permissions in Console.
```yaml theme={null}
---
apiVersion: iam/v2
kind: Group
metadata:
name: developers-a
spec:
displayName: "Developers Team A"
description: "Members of the Team A - Developers"
externalGroups:
- "LDAP-GRP-A-DEV"
externalGroupRegex:
- "LDAP*"
members:
- member1@company.org
- member2@company.org
permissions:
- resourceType: TOPIC
cluster: shadow-it
patternType: PREFIXED
name: toto-
permissions:
- topicViewConfig
- topicConsume
- topicProduce
```
**Groups checks:**
* `spec.description` is **optional**
* `spec.externalGroups` is a list of LDAP or OIDC groups to sync with this Console Group
* Members added this way will not appear in `spec.members` but `spec.membersFromExternalGroups` instead
* `spec.externalGroupRegex` is a list of regex patterns that can match to a series of LDAP or OIDC groups to sync with this Console group. Members added this way will not appear in `spec.members` list.
* Supports regex patterns for dynamic group matching (e.g., `^TEAM-.*` to match all groups starting with "TEAM-")
* `spec.membersFromExternalGroups` is a **read-only** list of members added through `spec.externalGroups` or `spec.externalGroupRegex`
* `spec.members` must be email addresses of members you wish to add to this group
* `spec.permissions` are valid permissions as defined in [Permissions](#permissions)
[View Terraform documentation](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs/resources/console_group_v2)
**Side effects in Console and Kafka:**
* Console
* Members of the Group are given the associated permissions in the UI over the resources
* Members of the LDAP or OIDC groups will be automatically added or removed upon login
* Kafka
* No side effects
## ConsoleUser
* **API key(s):** AdminToken
* **Managed with:** UI, CLI, API, TF
* **Labels support:** Missing
Creates a user with Console permissions.
```yaml theme={null}
---
apiVersion: iam/v2
kind: User
metadata:
name: john.doe@company.org
spec:
firstName: "John"
lastName: "Doe"
permissions:
- resourceType: PLATFORM
permissions:
- taasView
- datamaskingView
- resourceType: TOPIC
cluster: shadow-it
patternType: PREFIXED
name: toto-
permissions:
- topicViewConfig
- topicConsume
- topicProduce
```
Make sure you set permissions for this user, otherwise it won't have access to Console functionality (such as **Application Catalog** or Kafka resources.
**User checks:**
* `spec.permissions` are valid permissions as defined in [Permissions](#permissions)
[View Terraform documentation](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs/resources/console_user_v2)
**Side effects in Console and Kafka:**
* Console
* User is given the associated permissions in the UI over the resources
* Kafka
* No side effects
## KafkaCluster
Creates a Kafka cluster definition in Console.
* **API key(s):** AdminToken
* **Managed with:** UI, CLI, API, TF
* **Labels support:** Partial
```yaml theme={null}
---
apiVersion: console/v2
kind: KafkaCluster
metadata:
name: my-dev-cluster
spec:
displayName: "My Dev Cluster"
icon: "kafka"
color: "#000000"
bootstrapServers: "localhost:9092"
ignoreUntrustedCertificate: false
properties:
sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret";
security.protocol: SASL_SSL
sasl.mechanism: PLAIN
schemaRegistry:
type: "ConfluentLike"
url: http://localhost:8080
security:
type: BasicAuth
username: some_user
password: some_password
ignoreUntrustedCertificate: false
kafkaFlavor:
type: "Confluent"
key: "string"
secret: "string"
confluentEnvironmentId: "string"
confluentClusterId: "string"
policiesRef:
- "generic-dev-topic"
- "connector-naming-rule"
```
`metadata.name`, `spec.displayName`, `spec.icon` and `spec.color` are combined to create the visual identity of the KafkaCluster within Console.
**KafkaCluster checks:**
* `spec.icon` (optional, default `kafka`) is a valid entry from our [Icon Sets](#icon-sets)
* `spec.color` (optional, default `#000000`) is a HEX color for `spec.icon`
* `spec.ignoreUntrustedCertificate` (optional, default `false`) must be one of \[`true`, `false`]
* `spec.schemaRegistry.type` (optional) must be one of \[`ConfluentLike`, `Glue`]
* See [Schema Registry Properties](#schema-registry) for the detailed list of options
* `spec.kafkaFlavor.type` (optional) must be one of \[`Confluent`, `ConfluentPlatform`, `Aiven`, `Gateway`]
* See [Kafka Provider Properties](#kafka-provider) for the detailed list of options
* `spec.policiesRef` (optional), if set, has to be a valid list of [ResourcePolicy](/guide/reference/self-service-reference#resourcepolicy).
Conduktor CLI does not verify that your Kafka configuration (`spec.bootstrapServers`, `spec.properties`, etc.) is valid. You need to check that in Console directly.
[View Terraform documentation](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs/resources/console_kafka_cluster_v2)
### Schema registry
This section lets you associate a schema registry to your KafkaCluster.
#### Confluent or Confluent-like Registry
```yaml theme={null}
spec:
schemaRegistry:
type: "ConfluentLike"
url: http://localhost:8080
ignoreUntrustedCertificate: false
security:
type: BasicAuth
username: some_user
password: some_password
```
**Confluent schema registry checks:**
* `spec.schemaRegistry.urls` must be a single URL of a Kafka Connect cluster
* **Multiple URLs are not supported for now. Coming soon**
* `spec.schemaRegistry.ignoreUntrustedCertificate` (optional, default `false`) must be one of \[`true`, `false`]
* `spec.schemaRegistry.properties` (optional) is Java Properties formatted key values to further configure the SchemaRegistry
* `spec.security.type` (optional) must be one of \[`BasicAuth`, `BearerToken`, `SSLAuth`]
* See [HTTP Security Properties](#http-security-properties) for the detailed list of options
[View Terraform documentation](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs/resources/console_kafka_cluster_v2#nested-schema-for-specschema_registryconfluent_like)
#### AWS Glue registry
```yaml theme={null}
spec:
schemaRegistry:
type: "Glue"
region: eu-west-1
registryName: default
security:
type: Credentials
accessKeyId: accessKey
secretKey: secretKey
```
**AWS Glue registry checks:**
* `spec.schemaRegistry.region` must be a valid AWS region
* `spec.schemaRegistry.registryName` must be a valid AWS Glue Registry in this region
* `spec.schemaRegistry.security.type` must be one of \[`Credentials`, `FromContext`, `FromRole`]
[View Terraform documentation](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs/resources/console_kafka_cluster_v2#nested-schema-for-specschema_registryglue)
**Credentials**\
Use AWS API Key/Secret to connect to the Glue registry.
```yaml theme={null}
security:
type: Credentials
accessKeyId: AKIAIOSFODNN7EXAMPLE
secretKey: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
```
[View Terraform documentation](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs/resources/console_kafka_cluster_v2#nested-schema-for-specschema_registrygluesecuritycredentials)
**FromContext**
```yaml theme={null}
security:
type: FromContext
profile: default
```
[View Terraform documentation](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs/resources/console_kafka_cluster_v2#nested-schema-for-specschema_registrygluesecurityfrom_context)
**FromRole**
```yaml theme={null}
security:
type: FromRole
role: arn:aws:iam::123456789012:role/example-role
```
[View Terraform documentation](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs/resources/console_kafka_cluster_v2#nested-schema-for-specschema_registrygluesecurityfrom_role)
### Kafka provider
This section lets you configure the Kafka provider for this KafkaCluster.
**Confluent Cloud**
Provide your Confluent Cloud details to get additional features in Console:
* Confluent Cloud service account support
* Confluent Cloud API key support
```yaml theme={null}
spec:
kafkaFlavor:
type: "Confluent"
key: "yourApiKey123456"
secret: "yourApiSecret123456"
confluentEnvironmentId: "env-12345"
confluentClusterId: "lkc-67890"
```
[View Terraform documentation](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs/resources/console_kafka_cluster_v2#confluent-kafka-cluster-with-schema-registry)
**Confluent Platform**
Provide your Confluent Platform details to get additional features in Console:
* Confluent Platform service account support
* Confluent Platform RBAC role bindings via the Metadata Service (MDS)
```yaml theme={null}
spec:
kafkaFlavor:
type: "ConfluentPlatform"
kafkaClusterId: "abcDEF1234ghIJKL567mno"
mdsUrl: "https://mds.example.com:8090"
authentication:
username: "mds-admin"
password: "mds-admin-password"
schemaRegistryClusterId: "schema-registry"
enableRbacRoleBindings: true
ignoreUntrustedCertificate: false
```
**Aiven**
Provide your Aiven Cloud details to get additional features in Console:
* Aiven service accounts support
* Aiven ACLs support
```yaml theme={null}
spec:
kafkaFlavor:
type: "Aiven"
apiToken: "a1b2c3d4e5f6g7h8i9j0"
project: "my-kafka-project"
serviceName: "my-kafka-service"
```
[View Terraform documentation](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs/resources/console_kafka_cluster_v2#aiven-kafka-cluster-with-schema-registry)
**Gateway**
Provide your Gateway details to get additional features in Console:
* Interceptor support
```yaml theme={null}
spec:
kafkaFlavor:
type: "Gateway"
url: "http://gateway:8888"
user: "admin"
password: "admin"
virtualCluster: passthrough
```
[View Terraform documentation](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs/resources/console_kafka_cluster_v2#conduktor-gateway-kafka-cluster-with-schema-registry)
### Icon sets
| | | | |
| :-----------------: | :-----------------: | :-----------------: | :-----------------: |
| `cloudBolt` | `cloudRainbow` | `cloud` | `snowflake` |
| | | | |
| | | | |
| `pooStorm` | `poop` | `bolt` | `umbrella` |
| | | | |
| | | | |
| `tennisBall` | `rugbyBall` | `trafficCone` | `faucet` |
| | | | |
| | | | |
| `basketShopping` | `box` | `scaleBalanced` | `sunglasses` |
| | | | |
| | | | |
| `sword` | `axeBattle` | `vial` | `featherPointed` |
| | | | |
| | | | |
| `bomb` | `flag` | `heart` | `key` |
| | | | |
| | | | |
| `fireExtinguisher` | `fireFlameCurved` | `alien` | `helmetBattle` |
| | | | |
| | | | |
| `ghost` | `robot` | `dog` | `elephant` |
| | | | |
| | | | |
| `bird` | `crab` | `catSpace` | `planetRinged` |
| | | | |
| | | | |
| `meteor` | `moon` | `spaceStation` | `rocketLaunch` |
| | | | |
| | | | |
| `paperPlane` | `carSide` | `buildingColumns` | `castle` |
| | | | |
| | | | |
| `acorn` | `burgerLettuce` | `croissant` | `mug` |
| | | | |
| | | | |
| `cactus` | `clover` | `cameraCctv` | `calendar` |
| | | | |
| | | | |
| `alarmClock` | `compass` | `gamepadModern` | `server` |
| | | | |
| | | | |
| `shieldBlank` | `computerClassic` | `dharmachakra` | `kafka` |
## KafkaConnectCluster
Creates a Kafka Connect cluster definition in Console.
* **API key(s):** AdminToken
* **Managed with:** API, CLI, UI, TF
* **Labels support:** Partial
```yaml theme={null}
---
apiVersion: console/v2
kind: KafkaConnectCluster
metadata:
cluster: my-dev-cluster
name: connect-1
spec:
displayName: "Connect 1"
urls: "http://localhost:8083"
headers:
X-PROJECT-HEADER: value
AnotherHeader: test
ignoreUntrustedCertificate: false
security:
type: "BasicAuth"
username: "toto"
password: "my-secret"
policiesRef:
- "limit-connector-tasks"
- "approved-connector-classes"
```
**KafkaConnectCluster checks:**
* `metadata.cluster` has to be a valid KafkaCluster name.
* `spec.urls` has to be a single URL of a Kafka Connect cluster. Multiple URLs are not currently supported.
* `spec.ignoreUntrustedCertificate` (optional, default `false`). Has to be `true` or `false`.
* `spec.headers` (optional) has to be key-value pairs of HTTP headers.
* `spec.security.type` (optional) has to be `BasicAuth`, `BearerToken` or `SSLAuth`. [Find out more](#http-security-properties).
* `spec.policiesRef` (optional), if set, has to be a valid list of [ResourcePolicy](/guide/reference/self-service-reference#resourcepolicy).
[View Terraform documentation](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs/resources/console_kafka_connect_v2)
## KsqlDBCluster
* **API key(s):** AdminToken
* **Managed with:** UI, CLI, API
* **Labels support:** Missing
Creates a ksqlDB cluster definition in Console.
```yaml theme={null}
---
apiVersion: console/v2
kind: KsqlDBCluster
metadata:
cluster: my-dev-cluster
name: ksql-1
spec:
displayName: "KSQL 1"
url: "http://localhost:8088"
ignoreUntrustedCertificate: false
security:
type: "BasicAuth"
username: "toto"
password: "my-secret"
```
**KafkaConnectCluster checks:**
* `metadata.cluster` has to be a valid KafkaCluster name.
* `spec.url` has to be a single URL of a KsqlDB cluster.
* `spec.ignoreUntrustedCertificate` (optional), default is `false`. Has to be `true` or `false`.
* `spec.headers` (optional) has to be key-value pairs of HTTP headers.
* `spec.security.type` (optional) has to be `BasicAuth`, `BearerToken` or `SSLAuth`. [Find out more](#http-security-properties).
## Integrations
* **API key(s):** AdminToken
* **Managed with:** UI, CLI, API
* **Labels support:** Missing
Configures alert integrations in Console. Integrations enable external notification destinations for [Alerts](#alerts).
```yaml theme={null}
---
apiVersion: v3
kind: Integration
metadata:
name: Slack
spec:
config:
type: Slack
token: "xoxb-your-slack-bot-token"
```
```yaml theme={null}
---
apiVersion: v3
kind: Integration
metadata:
name: Email
spec:
config:
type: Email
smtpServerUrl: "smtp.company.com:587"
enableTls: true
ignoreUntrustedCertificate: false
senderEmail: "conduktor-alerts@company.com"
username: "smtp-user"
password: "smtp-password"
tlsMode: StartTls
```
```yaml theme={null}
---
apiVersion: v3
kind: Integration
metadata:
name: Teams
spec: {}
```
```yaml theme={null}
---
apiVersion: v3
kind: Integration
metadata:
name: Webhook
spec: {}
```
**Integration checks:**
* `metadata.name` has to be one of `Slack`, `Email`, `Teams` or `Webhook`.
* `spec.config` is required for `Slack` and `Email` integrations. For `Teams` and `Webhook`, the spec can be empty as they are configured per-alert.
### Slack config
| Field | Type | Required | Description |
| ------- | ------ | -------- | --------------------- |
| `type` | string | Yes | Has to be `Slack` |
| `token` | string | Yes | Slack Bot OAuth token |
### Email config
| Field | Type | Required | Description |
| ---------------------------- | ------- | -------- | ----------------------------------------------------------------- |
| `type` | string | Yes | Has to be `Email` |
| `smtpServerUrl` | string | Yes | SMTP server address with port (e.g., `smtp.company.com:587`) |
| `enableTls` | boolean | Yes | Enable TLS encryption for SMTP connection |
| `ignoreUntrustedCertificate` | boolean | Yes | Skip SSL certificate validation (not recommended for production) |
| `senderEmail` | string | Yes | Email address that appears as the sender |
| `username` | string | Yes | SMTP authentication username |
| `password` | string | Yes | SMTP authentication password |
| `tlsMode` | string | No | TLS mode: `StartTls` (upgrade connection) or `Tls` (implicit TLS) |
For more details on configuring email integration through the UI, see [Email integration](/guide/monitor-brokers-apps/alerts#email-integration).
## Alerts
* **API key(s):** AdminToken, AppToken
* **Managed with:** UI, CLI, API
* **Labels support:** Missing
Creates an alert in Console.
```yaml theme={null}
---
apiVersion: console/v3
kind: Alert
metadata:
name: messages-in-dead-letter-queue
group: support-team # will be the owner of the alert, can be either a user, a group or an appInstance
# user: user@company.org
# appInstance: my-app-instance
spec:
cluster: my-dev-cluster
type: TopicAlert
topicName: wikipedia-parsed-DLQ
metric: MessageCount
operator: GreaterThan
threshold: 0
description: "Alert for monitoring messages in dead letter queue"
displayName: "DLQ message count alert"
destination:
type: Slack
channel: "alerts-p1"
```
**Alert checks:**
* `metadata.user`|`metadata.group`|`metadata.appInstance` has to be a valid user, group or appInstance.
* `metadata.destination.type` can be either `Slack`, `Teams` or `Webhook`. When set to:
* `Slack`: has to be a valid Slack channel ID
* `Teams`: has to be a valid Teams webhook URL
* `Webhook`:
* `spec.destination.url` has to be a valid URL
* `spec.destination.method` has to be `GET`, `POST`, `PUT` or `DELETE`
* `spec.destination.headers` (optional) has to be key-value pairs of HTTP headers
* `spec.destination.authentication.type` (optional) has to be `BasicAuth` (define `spec.destination.authentication.username` and `spec.destination.authentication.password`) or `BearerToken` (define `spec.destination.authentication.token`).
* `spec.cluster` has to be a valid KafkaCluster name.
* `spec.type` has to be `BrokerAlert`,`TopicAlert`, `KafkaConnectAlert` or `ConsumerGroupAlert`. When set to:
* `BrokerAlert`: `spec.metric` has to be `MessageIn`, `MessageOut`, `MessageSize`, `OfflinePartitionCount`, `PartitionCount`, `UnderMinIsrPartitionCount` or `UnderReplicatedPartitionCount`.
* `TopicAlert`: `spec.metric` has to be `MessageCount`, `MessageIn`, `MessageOut` or `MessageSize` and the `spec.topicName` has to be a Kafka topic that the owner can access.
* `KafkaConnectAlert`:`spec.metric` has to be `FailedTaskCount`;`spec.connectName` has to be a valid KafkaConnect cluster associated to this `spec.cluster` Kafka cluster and `spec.connectorName` has to be a Kafka Connect Connector that the owner can access.
* `ConsumerGroupAlert`:`spec.metric` has to be `OffsetLag` or `TimeLag` and `spec.consumerGroupName` has to be a Kafka Consumer group that the owner can access.
* `spec.metric` depends on the `spec.type`.
* `spec.operator` has to be `GreaterThan`, `GreaterThanOrEqual`, `LessThan`, `LessThanOrEqual` or `NotEqual`.
* `spec.threshold` has to be a number.
* `spec.description` (optional) provides a text description of the alert.
* `spec.displayName` (optional) provides a display name for the alert.
* `spec.disable` (optional), default is `false`. Has to be `true` or `false`.
**Alert resolution:** Alerts resolve (stop firing) when the metric value no longer meets the threshold condition defined by the comparison operator (GreaterThan, LessThan, etc.).
## Partner Zones
* **API key(s):** AdminToken
* **Managed with:** UI, CLI, API
* **Labels support:** Partial
Create or update a [Partner Zone](/guide/conduktor-concepts/partner-zones).
```yaml theme={null}
---
apiVersion: console/v2
kind: PartnerZone
metadata:
name: external-partner-zone
spec:
displayName: External Partner Zone
description: An external partner to exchange data with.
url: https://partner1.com
partner:
name: John Doe
role: Data analyst
email: johndoe@partner.io
phone: 07827 837 177
cluster: cdk-gateway
underlyingCluster: cluster1
authenticationMode:
serviceAccount: partner-external-partner
type: PLAIN
vclusterName: custom-vcluster-name
topics:
- name: topic-a
backingTopic: kafka-topic-a
permission: WRITE
- name: topic-b
backingTopic: kafka-topic-a
permission: READ
trafficControlPolicies:
maxProduceRate: 1e+06
maxConsumeRate: 1e+06
limitCommitOffset: 30
headers:
addOnProduce:
- key: partner-name
value: external-analytics-partner
overrideIfExists: false
- key: client-info
value: "Client:{{clientId}}, from IP:{{userIp}}"
overrideIfExists: true
- key: kafka-api
value: "Kafka API Key:{{apiKey}}, version {{apiKeyVersion}}"
overrideIfExists: true
- key: produce-metadata
value: "User:{{user}}, via Gateway:{{gatewayHost}}, at timestampMillis:{{timestampMillis}}"
overrideIfExists: true
removeOnConsume:
- keyRegex: my_team_prefix.*
```
**Partner Zone checks:**
* `spec.displayName` is mandatory.
* `spec.description`, `spec.url` and `spec.partner` (optional), useful for context information.
* `spec.cluster` has to be a valid Console cluster technical ID with the **Provider configured as *Gateway***.
* `spec.underlyingCluster` has to be a valid Console cluster technical ID where the cluster has to be defined under the Gateway defined by the `spec.gatewayClusterId`.
* When not specified, the value is inferred to be equal to the `spec.gatewayClusterId` field, hence selecting the main cluster behind Gateway by default.
* `spec.authenticationMode.type` must be one of \[`PLAIN`, `OAUTHBEARER`, `MTLS`]. See [authentication examples](/guide/use-cases/third-party-data#authentication-modes) for detailed configuration.
* `spec.authenticationMode.serviceAccount` requirements depend on the authentication type:
* **PLAIN**: Any unique identifier for your partner (e.g., `partner-external-partner`). This will be created as a local Gateway service account automatically if it doesn't exist.
* **OAUTHBEARER**: Must match the "sub" claim in the partner's OAuth/OIDC token (e.g., `oauth-partner-service-account`). The partner needs to authenticate using their OAuth provider.
* **MTLS**: Must match the client's Distinguished Name (DN) from their certificate, unless you've modified `GATEWAY_SSL_PRINCIPAL_MAPPING_RULES` (e.g., `CN=partner-client,OU=Engineering,O=PartnerCorp,C=US`).
* `spec.vclusterName` (optional), custom name for the Virtual Cluster. If not provided, it will be auto-generated.
* `topics[].name` is the name of the topic as it should appear to your external partner. This can be different from `backingTopic`.
* `topics[].backingTopic` is the internal name of the topic that you want to share.
* `topics[].permission` has to be set to either `READ` or `WRITE` (which includes `READ`).
* `trafficControlPolicies.maxProduceRate` (optional), sets the maximum rate (in bytes/s) at which the partner can produce messages to the topics per Gateway node.
* `trafficControlPolicies.maxConsumeRate` (optional), sets the maximum rate (in bytes/s) at which the partner can consume messages from the topics per Gateway node.
* `trafficControlPolicies.limitCommitOffset` (optional), sets the maximum number of commit requests (in requests/minute) that the partner can make per Gateway node.
* `headers.addOnProduce` (optional), list of headers to inject when producing messages. The `value` field supports special variables: `{{user}}`, `{{userIp}}`, `{{clientId}}`, `{{apiKey}}`, `{{apiKeyVersion}}`, `{{gatewayHost}}`, `{{timestampMillis}}`.
* `headers.removeOnConsume` (optional), list of header key patterns (regex) to remove when consuming messages.
**Side effects in Console and Kafka:**
Once created or updated, the following fields will be made available:
* `metadata.updatedAt` (by consecutive *get* from the CLI/API).
* `metadata.status` (by consecutive *get* from the CLI/API.) Possible values are `PENDING`, `READY` or `FAILED`.
* `metadata.failedReason` will be populated in case of `FAILED` status.
* The service account will be created if it doesn't exist and will be granted the permissions as declared in `spec.topics`.
* The traffic control policies will be applied to the service account.
## DataQualityRule
* **API key(s):** AdminToken
* **Managed with:** UI, CLI, API
* **Labels support:** Yes
Create or update a data quality rule that validates data quality for Kafka topics.
```yaml theme={null}
---
apiVersion: v1
kind: DataQualityRule
metadata:
name: email-validation-rule
labels:
category: validation
team: data-quality
spec:
type: Cel
displayName: Email validation rule
description: Validates that email field contains a valid email address
celExpression: 'record.email.matches("^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$")'
customErrorMessage: Email address is not valid
```
```yaml theme={null}
---
apiVersion: v1
kind: DataQualityRule
metadata:
name: user-schema-rule
labels:
category: schema
spec:
type: JsonSchema
displayName: User schema validation
description: Validates user records against JSON schema
schema:
type: object
required:
- userId
- email
properties:
userId:
type: string
email:
type: string
format: email
age:
type: integer
minimum: 0
customErrorMessage: Record does not match user schema
```
**DataQualityRule checks:**
* `metadata.name` is mandatory and must be unique.
* `metadata.labels` is optional and can be used to organize rules.
* `spec.type` has to be one of: `Cel`, `JsonSchema`, `EnforceAvro`, `EnforceSchemaId`.
* `spec.displayName` is mandatory and is shown in the Console UI.
* `spec.description` is optional but recommended to explain what the rule validates.
* `spec.customErrorMessage` is optional and provides a custom error message when validation fails.
* For `Cel` type:
* `spec.celExpression` is mandatory and contains the CEL (Common Expression Language) expression to validate records.
* For `JsonSchema` type:
* `spec.schema` is mandatory and contains the JSON schema object to validate records against.
* For `EnforceAvro` type:
* Validates that records conform to an Avro schema.
* For `EnforceSchemaId` type:
* Validates that records use a specific schema ID from Schema Registry.
**Side effects in Console and Kafka:**
Once created or updated, the following fields will be made available:
* `metadata.createdAt` (read-only timestamp).
* `metadata.updatedAt` (read-only timestamp).
* `metadata.createdBy` (read-only user identifier).
* `metadata.updatedBy` (read-only user identifier).
* `metadata.attachedPolicies` (read-only list of policies using this rule).
## DataQualityPolicy
* **API key(s):** AdminToken
* **Managed with:** UI, CLI, API
* **Labels support:** No
Create or update a data quality policy that applies rules to specific Kafka topics.
```yaml theme={null}
---
apiVersion: v1
kind: DataQualityPolicy
metadata:
name: user-data-validation-policy
spec:
displayName: User data validation policy
description: Validates all user data before it is produced to topics
rules:
- email-validation-rule
- user-schema-rule
targets:
- cluster: main-cluster
topic: users
patternType: LITERAL
- cluster: main-cluster
topic: user-events-
patternType: PREFIXED
actions:
block:
enabled: true
mark:
enabled: false
```
**DataQualityPolicy checks:**
* `metadata.name` is mandatory and must be unique.
* `spec.displayName` is mandatory and is shown in the Console UI.
* `spec.description` is optional but recommended to explain the policy's purpose.
* `spec.rules` is a list of DataQualityRule names to apply (references to rules created separately).
* `spec.targets` is mandatory and defines which topics the policy applies to:
* `cluster` is a valid Kafka cluster technical ID.
* `topic` is the topic name or prefix.
* `patternType` has to be either `LITERAL` (exact match) or `PREFIXED` (prefix match).
* `spec.actions` defines what happens when a rule violation occurs:
* `block.enabled` (optional, defaults to false) - when true, blocks messages that violate rules from being produced. Only available if the configured cluster is a Conduktor Gateway cluster with the appropriate license.
* `mark.enabled` (optional, defaults to false) - when true, marks messages that violate rules with a header but allows production.
**Side effects in Console and Kafka:**
Once created or updated, the following fields will be made available:
* `metadata.nameForMetrics` (read-only identifier used in metrics).
* `metadata.group` (read-only group identifier).
* `metadata.createdAt` (read-only timestamp).
* `metadata.updatedAt` (read-only timestamp).
* `metadata.createdBy` (read-only user identifier).
* `metadata.updatedBy` (read-only user identifier).
* `metadata.status` (read-only status: `failed`, `pending`, or `ready`).
* `metadata.statuses` (read-only per-cluster deployment status).
* `metadata.metrics` (read-only metrics showing evaluation and violation counts).
* `metadata.attachedRules` (read-only list of rules attached to this policy).
* Data quality rules are enforced on the specified topics.
* Messages violating rules are blocked or marked based on the configured actions.
## HTTP security properties
HTTP security properties are used in KafkaCluster ([schema registry](#confluent-or-confluent-like-registry)), [KafkaConnect](#kafkaconnectcluster) and [KsqlDBCluster](#ksqldbcluster).
### Basic authentication
```yaml theme={null}
security:
type: "BasicAuth"
username: "toto"
password: "my-secret"
```
### Bearer token
```yaml theme={null}
security:
type: "BearerToken"
token: "toto"
```
### mTLS/client certificate
```yaml theme={null}
security:
type: "SSLAuth"
key: |
-----BEGIN PRIVATE KEY-----
MIIOXzCCDUegAwIBAgIRAPRytMVYJNUgCbhnA+eYumgwDQYJKoZIhvcNAQELBQAw
...
IFyCs+xkcgvHFtBjjel4pnIET0agtbGJbGDEQBNxX+i4MDA=
-----END PRIVATE KEY-----
certificateChain: |
-----BEGIN CERTIFICATE-----
MIIOXzCCDUegAwIBAgIRAPRytMVYJNUgCbhnA+eYumgwDQYJKoZIhvcNAQELBQAw
RjELMAkGA1UEBhMCVVMxIjAgBgNVBAoTGUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBM
...
8/s+YDKveNdoeQoAmGQpUmxhvJ9rbNYj+4jiaujkfxT/6WtFN8N95r+k3W/1K4hs
IFyCs+xkcgvHFtBjjel4pnIET0agtbGJbGDEQBNxX+i4MDA=
-----END CERTIFICATE-----
```
## Permissions
Permissions are used in [groups](#consolegroup) and [users](#consoleuser) and lets you configure access to any Kafka resource or Console feature.
A permission applies to a certain `resourceType` which affects the required fields.
* [Topic permissions](#topic-permissions)
* [Subject permissions](#subject-permissions)
* [ConsumerGroup permissions](#consumergroup-permissions)
* [Cluster permissions](#cluster-permissions)
* [KafkaConnect permissions](#kafkaconnect-permissions)
* [KsqlDB permissions](#ksqldb-permissions)
* [Console permissions](#console-permissions)
### Topic permissions
```yaml theme={null}
# Grants consume, produce and view config to all topics toto-* on shadow-it cluster
- resourceType: TOPIC
cluster: shadow-it
patternType: PREFIXED
name: toto-
permissions:
- topicViewConfig
- topicConsume
- topicProduce
```
* `resourceType`: `TOPIC`
* `cluster` is a valid Kafka cluster
* `patternType` is either `PREFIXED` or `LITERAL`
* `name` is the name of the topic or topic prefix to apply the permissions to
* `permissions` is a list of valid topic permissions
| Available topic permissions | Description |
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `topicConsume` | Permission to consume messages from the topic. |
| `topicProduce` | Permission to produce (write) messages to the topic. |
| `topicViewConfig` | Permission to view the topic configuration. |
| `topicEditConfig` | Permission to edit the topic configuration. |
| `topicCreate` | Permission to create a new topic. |
| `topicDelete` | Permission to delete the topic. |
| `topicAddPartition` | Permission to add partitions to the topic. |
| `topicEmpty` | Permission to empty (delete all messages from) the topic. |
| `topicDataQualityManage` | Permission to manage data quality policies for the topic. |
| `topicMetadataManage` | Permission to edit the topic's labels and description. Enforced only when [metadata governance](/guide/conduktor-in-production/admin/set-up-rbac#govern-who-can-edit-labels-and-descriptions) is enabled. |
### Subject permissions
```yaml theme={null}
# Grants view and edit compatibility to all subjects starting with sub-* on shadow-it cluster
- resourceType: SUBJECT
cluster: shadow-it
patternType: PREFIXED
name: sub-
permissions:
- subjectView
- subjectEditCompatibility
```
* `resourceType`: `SUBJECT`
* `cluster` is a valid Kafka cluster
* `patternType` is either `PREFIXED` or `LITERAL`
* `name` is the name of the subject or subject prefix to apply the permissions to
* `permissions` is a list of valid subject permissions
| Available subject permissions | Description |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `subjectCreateUpdate` | Permission to create or update the subject. |
| `subjectDelete` | Permission to delete the subject. |
| `subjectEditCompatibility` | Permission to edit the subject compatibility settings. |
| `subjectView` | Permission to view the subject details. |
| `subjectMetadataManage` | Permission to edit the subject's labels. Enforced only when [metadata governance](/guide/conduktor-in-production/admin/set-up-rbac#govern-who-can-edit-labels-and-descriptions) is enabled. |
### ConsumerGroup permissions
```yaml theme={null}
# Grants view and reset on all consumer groups starting with group-* on shadow-it cluster
- resourceType: CONSUMER_GROUP
cluster: shadow-it
patternType: PREFIXED
name: group-
permissions:
- consumerGroupView
- consumerGroupReset
```
* `resourceType`: `CONSUMER_GROUP`
* `cluster` is a valid Kafka cluster
* `patternType` is either `PREFIXED` or `LITERAL`
* `name` is the name of the consumer group or consumer group prefix to apply the permissions to
* `permissions` is a list of valid consumer group permissions
| Available ConsumerGroup permissions | Description |
| ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `consumerGroupCreate` | Permission to create a new consumer group. |
| `consumerGroupReset` | Permission to reset the consumer group. |
| `consumerGroupDelete` | Permission to delete the consumer group. |
| `consumerGroupView` | Permission to view the consumer group details. |
| `consumerGroupMetadataManage` | Permission to edit the consumer group's labels. Enforced only when [metadata governance](/guide/conduktor-in-production/admin/set-up-rbac#govern-who-can-edit-labels-and-descriptions) is enabled. |
### Cluster permissions
```yaml theme={null}
# Grants view and edit broker, edit schema registry compatibility, view ACL and manage ACL on shadow-it cluster
- resourceType: CLUSTER
name: shadow-it
permissions:
- clusterViewBroker
- clusterEditSRCompatibility
- clusterEditBroker
- clusterViewACL
- clusterManageACL
```
* `resourceType`: `CLUSTER`
* `name` is the name of the cluster to apply the permissions to
* Use `*` for all clusters
* `permissions` is a list of valid cluster permissions
| Available cluster permissions | Description |
| ----------------------------- | ------------------------------------------------------------------------------------------------- |
| `clusterViewBroker` | Permission to view broker details. |
| `clusterEditSRCompatibility` | Permission to edit schema registry compatibility settings. |
| `clusterEditBroker` | Permission to edit broker configuration. |
| `clusterViewACL` | Permission to view ACLs for the cluster. Also grants ACL viewing on every topic in the cluster. |
| `clusterManageACL` | Permission to manage ACLs for the cluster. Also grants ACL viewing on every topic in the cluster. |
### KafkaConnect permissions
```yaml theme={null}
# Grants create and delete on all connectors starting with connector-* on shadow-it cluster and kafka-connect-cluster
- resourceType: KAFKA_CONNECT
cluster: shadow-it
kafkaConnect: kafka-connect-cluster
patternType: PREFIXED
name: connector-
permissions:
- kafkaConnectorCreate
- kafkaConnectorDelete
```
* `resourceType`: `KAFKA_CONNECT`
* `cluster` is a valid Kafka cluster
* `kafkaConnect` is a valid Kafka Connect cluster
* `patternType` is either `PREFIXED` or `LITERAL`
* `name` is the name of the connector or connector prefix to apply the permissions to
* `permissions` is a list of valid Kafka Connect permissions
| Available KafkaConnect permission | Description |
| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `kafkaConnectorViewConfig` | Permission to view the Kafka Connect configuration. |
| `kafkaConnectorStatus` | Permission to view the status of Kafka Connect connectors. |
| `kafkaConnectorEditConfig` | Permission to edit the Kafka Connect configuration. |
| `kafkaConnectorDelete` | Permission to delete connectors. |
| `kafkaConnectorCreate` | Permission to create new connectors. |
| `kafkaConnectPauseResume` | Permission to pause and resume connectors. |
| `kafkaConnectRestart` | Permission to restart connectors. |
| `kafkaConnectStop` | Permission to stop connectors. Distinct from pause and resume, as stopping a connector releases its tasks and resources. |
| `kafkaConnectOffsetsView` | Permission to view connector offsets. |
| `kafkaConnectOffsetsManage` | Permission to edit and reset connector offsets. The connector has to be stopped first. |
| `kafkaConnectorMetadataManage` | Permission to edit the connector's labels and description. Enforced only when [metadata governance](/guide/conduktor-in-production/admin/set-up-rbac#govern-who-can-edit-labels-and-descriptions) is enabled. |
### KsqlDB permissions
```yaml theme={null}
# Grants all permissions on KsqlDB cluster ksql-cluster
- resourceType: KSQLDB
cluster: shadow-it
ksqlDB: ksql-cluster
permissions:
- ksqldbAccess
```
* `resourceType`: `KSQLDB`
* `cluster` is a valid Kafka cluster
* `ksqlDB` is a valid Kafka Connect cluster
* `permissions` is a list of valid KsqlDB permissions
| Available KafkaConnect permissions | Description |
| ---------------------------------- | --------------------------------------------- |
| `ksqldbAccess` | Grants all permissions on the KsqlDB cluster. |
### Console permissions
```yaml theme={null}
# Grants Console permissions
- resourceType: PLATFORM
permissions:
- userView
- datamaskingView
```
* `resourceType`: `PLATFORM`
* `permissions` is a list of valid Console permissions
| Available Console permissions | Description |
| ----------------------------- | -------------------------------------------------------------- |
| `clusterConnectionsManage` | Permission to add / edit / remove Kafka clusters on Console. |
| `certificateManage` | Permission to add / edit / remove TLS certificates on Console. |
| `userManage` | Permission to manage Console users, groups and permissions. |
| `userView` | Permission to view Console users, groups and permissions. |
| `datamaskingManage` | Permission to manage data policies (masking rules). |
| `datamaskingView` | Permission to view data policies. |
| `notificationChannelManage` | Permission to manage integration channels. |
| `auditLogView` | Permission to browse audit log. |
| `taasView` | Permission to view Application Catalog. |
| `chargebackManage` | Permission to view Chargeback and manage its settings. |
| `sqlManage` | Permission to view indexed topics and create SQL queries. |
# Data quality Interceptors
Source: https://docs.conduktor.io/guide/reference/data-quality
Data quality Interceptor reference for Conduktor Gateway: CEL expression rules, JSON Schema validation, and Schema ID checks for enforcing Kafka message quality.
Conduktor Gateway data quality Interceptors validate Kafka messages at the proxy layer — before they hit the broker. Rules run per record and have three outcomes: **block** the message (error returned to the producer), **mark** it with a header so consumers can decide what to do, or **redirect** it to a dead-letter topic.
## When to use data quality interceptors
Use data quality interceptors when you need to:
* **Enforce schema contracts** — reject records that don't match an expected Avro, Protobuf, or JSON Schema
* **Apply business rules** — validate field values using CEL expressions (e.g., age > 0, email not null)
* **Prevent bad data propagation** — stop invalid records before they reach consumers and cause processing failures
* **Observe without blocking** — audit violations in real time without interrupting producer traffic
**From our blog:** [The hidden pitfalls of Kafka's schemaless data](https://conduktor.io/blog/the-hidden-pitfalls-of-kafka-s-schemaless-data) Why schemaless topics let bad data through, and how CEL and JSON Schema rules catch it at the proxy.
## Available policies
| Policy | Use case |
| -------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| [Data quality policy](/guide/conduktor-in-production/admin/gateway-policies#validate-data-quality-policy) | CEL expression and JSON Schema validation on topic messages |
| [Data quality producer policy](/guide/conduktor-in-production/admin/gateway-policies#validate-producer-data-quality) | Validate records at produce time before they are written to the topic |
| [Schema payload validation policy](/guide/conduktor-in-production/admin/gateway-policies#validate-schema-payload) | Enforce that records use a registered Schema Registry schema ID |
## Related resources
* [Data quality policies concept](/guide/conduktor-concepts/data-quality-policies)
* [Enforce data quality use case](/guide/use-cases/enforce-data-quality)
* [Observe data quality violations](/guide/use-cases/observe-data-quality)
# Data security Interceptors
Source: https://docs.conduktor.io/guide/reference/data-security
Data security Interceptor reference for Conduktor Gateway: field-level encryption, full payload encryption, data masking.
Configure audit logging, data masking, header injection, the message integrity Interceptor and encryption Interceptors to secure and track data in your Kafka topics.
| Job to do | Interceptor to use |
| -------------------------------------------------- | --------------------------------------------------------------- |
| Track the APIs used by an application | [Audit](#audit-interceptor) |
| Hide sensitive fields from consumers | [Data masking](#data-masking-interceptor) |
| Add metadata to messages | [Header injection](#dynamic-header-injection-interceptor) |
| Sign and verify message integrity | [Message integrity Interceptor](#message-integrity-interceptor) |
| Encrypt data | [Encryption](#encryption-interceptors) |
| Keep data usable for analytics while protecting it | [Tokenization](#tokenization) |
| Delete a single customer's data from Kafka | [Crypto shredding](/guide/tutorials/crypto-shredding) |
**From our blog:** [FedRAMP High for Kafka without replatforming](https://www.conduktor.io/blog/fedramp-high-for-kafka-without-replatforming) Meet FedRAMP High encryption requirements on your existing Kafka, no replatforming.
**[Kafka and HIPAA 2026](https://www.conduktor.io/blog/kafka-hipaa-compliance)** What the 2026 HIPAA Security Rule changes for Kafka, and how to close the gaps.
## Common configuration
The following configurations are shared across multiple Interceptors:
### Environment variables as secrets
To ensure your secrets don't appear in your Interceptors, you can refer to the environment variables set in your Gateway container.
Use the format `${MY_ENV_VAR}`.
We recommend using this for schema registry or Vault secrets and any other values you'd like to hide in the configuration.
### Schema registry
Gateway reads schemas either through an external connection or from configuration on the Interceptor itself.
Schema-encoded records whose schema ID is stored in a record header, the format [introduced in Confluent Platform 8.2](https://www.confluent.io/blog/schema-id-kafka-headers-data-governance), require **Gateway 3.21.0** or later.
External connections are a preview feature and subject to change. Inline `schemaRegistryConfig` remains fully supported.
Set `schemaRegistryConnectionName` to the name of a [schema registry external connection](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/external-connections#connect-to-a-schema-registry), declared once when Gateway starts with variables that follow this pattern:
```
GATEWAY_EXTERNAL_CONNECTION__CONFIG_
```
| Key | Type | Description |
| ------------------------------ | ------ | ----------------------------------------------------------------------------------------------------- |
| `schemaRegistryConnectionName` | string | The `` of a schema registry connection. Gateway rejects an unknown name with `400 Bad Request`. |
For a connection declared as `GATEWAY_EXTERNAL_CONNECTION_DEFAULT_SR_CONFIG_*`, the Interceptor reaches it with:
```json theme={null}
"schemaRegistryConnectionName": "DEFAULT_SR"
```
Set either `schemaRegistryConnectionName` or `schemaRegistryConfig`, never both. Gateway rejects a configuration that sets both with `400 Bad Request`, rather than silently picking one.
Remove a connection an Interceptor already names and the Interceptor stays visible through the API but stops applying to traffic. [See what happens at startup](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/external-connections#what-happens-at-startup).
Gateway supports **Confluent-like** and **AWS Glue** schema registries.
| Key | Type | Default | Description |
| --------------------- | ------ | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `type` | string | `CONFLUENT` | The type of schema registry to use: choose `CONFLUENT` (for Confluent-like schema registries including OSS Kafka) or `AWS` for AWS Glue schema registries. |
| `additionalConfigs` | map | | Additional properties maps to specific security-related parameters. For enhanced security, you can hide the sensitive values using [environment variables as secrets](/guide/reference/data-security#environment-variables-as-secrets). |
| **Confluent-like** | | | **Configuration for Confluent-like schema registries** |
| `host` | string | | URL of your schema registry. |
| `cacheSize` | string | `50` | Number of schemas that can be cached locally by this Interceptor so that it doesn't have to query the schema registry every time. |
| **AWS Glue** | | | **Configuration for AWS Glue schema registries** |
| `region` | string | | The AWS region for the schema registry, e.g. `us-east-1`. |
| `registryName` | string | | The name of the schema registry in AWS (leave blank for the AWS default of `default-registry`). |
| `basicCredentials` | string | | Access credentials for AWS. |
| **AWS credentials** | | | **AWS credential configuration** |
| `accessKey` | string | | The access key for the connection to the schema registry. |
| `secretKey` | string | | The secret key for the connection to the schema registry. |
| `validateCredentials` | bool | `true` | Whether Gateway rejects a blank `accessKey` or `secretKey` when it builds the credentials. Set it to `false` to accept them and fail later instead. |
| `accountId` | string | | The Id for the AWS account to use. |
If you don't supply a `basicCredentials` section for the AWS Glue schema registry, the client will attempt to find the connection information it needs from the environment ([see AWS docs for details](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html#credentials-default) ) and the credentials required can be passed this way to Gateway as part of its core configuration.
[Read our blog about schema registry](https://www.conduktor.io/blog/what-is-the-schema-registry-and-why-do-you-need-to-use-it/).
## Audit Interceptor
This Interceptor logs information from API key requests. To use it, inject it and implement `ApiKeyAuditLog` interface for audit.
The currently supported Kafka API requests are:
* ProduceRequest (PRODUCE)
* FetchRequest (FETCH)
* CreateTopicRequest (CREATE\_TOPICS)
* DeleteTopicRequest (DELETE\_TOPICS)
* AlterConfigRequest (ALTER\_CONFIGS)
When [incremental fetch sessions](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables#incremental-fetch-sessions) are enabled, Gateway records a fetch only when the request lists the topics being read, which a consumer stops doing once it settles into steady-state reads. A long-lived consumer can appear idle in audit logs while it keeps consuming. This affects audit visibility only — it doesn't change the data a consumer receives.
### Configure audit Interceptor
| Name | Type | Default | Description |
| :-------------- | :------------ | :------ | :------------------------------------------------------------------------ |
| topic | String | `.*` | Topics that match this regex will have the Interceptor applied |
| apiKeys | Set\[string] | | Set of Kafka API keys to be audited |
| vcluster | String | `.*` | vcluster that matches this regex will have the Interceptor applied |
| username | String | `.*` | username that matches this regex will have the Interceptor applied |
| consumerGroupId | String | `.*` | consumerGroupId that matches this regex will have the Interceptor applied |
| topicPartitions | Set\[Integer] | | Set of topic partitions to be audited |
### Audit Interceptor example
```bash theme={null}
curl \
--request PUT \
--url 'http://localhost:8888/gateway/v2/interceptor' \
--header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
--header 'Content-Type: application/json' \
--data-raw '{
"kind": "Interceptor",
"apiVersion": "gateway/v2",
"metadata": {
"name": "myAuditInterceptorPlugin"
},
"spec": {
"pluginClass": "io.conduktor.gateway.interceptor.AuditPlugin",
"priority": 100,
"config": {
"topic": ".*",
"apiKeys": [
"PRODUCE",
"FETCH"
],
"vcluster": ".*",
"username": ".*",
"consumerGroupId": ".*",
"topicPartitions": [
1,
2
]
}
}
}'
```
```yaml theme={null}
apiVersion: gateway/v2
kind: Interceptor
metadata:
name: myAuditInterceptorPlugin
scope:
vCluster: passthrough
spec:
pluginClass: io.conduktor.gateway.interceptor.AuditPlugin
priority: 100
config:
topic: ".*"
apiKeys:
- PRODUCE
- FETCH
vcluster: ".*"
username: ".*"
consumerGroupId: ".*"
topicPartitions:
- 1
- 2
```
Apply with:
```bash theme={null}
conduktor apply -f audit-interceptor.yaml
```
## Data masking Interceptor
Field level data masking Interceptor masks sensitive fields within messages as they are consumed.
### Configure data masking Interceptor
The policies will be applied when consuming messages.
| Key | Type | Default | Description |
| :--------------------------- | :---------------------------------- | :----------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| topic | String | `.*` | Topics that match this regex will have the Interceptor applied. |
| policies | Policy list | | List of your masking policies. |
| errorPolicy | String | `fail_fetch` | Determines the plugin behavior when it can't parse a fetched message without an associated schema: `fail_fetch` or `skip_masking`. |
| schemaRegistryConfig | [Schema registry](#schema-registry) | | The schema registry in use. Required for Avro, JSON Schema or Protobuf data. |
| schemaRegistryConnectionName | String | | Name of a [schema registry external connection](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/external-connections). An alternative to `schemaRegistryConfig` — set one or the other, not both. |
#### Data masking policy
| Key | Type | Description |
| :----- | :------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| name | String | Unique name to identify your policy. |
| fields | Set of String | Set of fields that should be obfuscated with the masking rule. Fields can be in a nested structure with dot `.`. For example: `education.account.username`, `banks[0].accountNo` or `banks[*].accountNo`. |
| rule | Rule | Masking rule to apply. |
#### Data masking rule
| Key | Type | Default | Description |
| :------------ | :----------- | :--------- | :----------------------------------------------------------- |
| type | Masking type | `MASK_ALL` | The type of masking (see below). |
| maskingChar | String | `*` | The character(s) used for masking data. |
| numberOfChars | number | | Number of masked characters, required if `type != MASK_ALL`. |
#### Masking type
* `MASK_ALL`: all data will be masked
* `MASK_FIRST_N`: the first `n` characters will be masked
* `MASK_LAST_N`: the last `n` characters will be masked
### Error policy
You can control the plugin behavior when it can't parse a fetched message through its `errorPolicy` which can be set to `fail_fetch` or `skip_masking`.
The error policy only applies to messages that do not have an associated schema. When a message has a schema (Avro, JSON Schema or Protobuf), the plugin uses the schema to parse the message and the error policy is not triggered.
The default is `fail_fetch`. In this mode, the plugin will return a failure to read the batch which the fetch record is part of, effectively blocking any consumer.
In `skip_masking` mode, if there's a failure to parse a message being fetched (e.g. an encrypted record or a schemaless message that can't be parsed), then that record is skipped and returned un-masked.
### Full payload encryption compatibility
Data masking is compatible with full payload encryption. When both Interceptors are applied to the same topic, data masking automatically detects **records with full payload encryption headers** and skips them, preventing deserialization errors that would otherwise occur when attempting to mask encrypted content.
[Check out the encryption configuration for details](#encryption-configuration).
Field level encryption is not affected by this behavior.
### Data masking Interceptor example
```bash theme={null}
curl \
--request PUT \
--url 'http://localhost:8888/gateway/v2/interceptor' \
--header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
--header 'Content-Type: application/json' \
--data-raw '{
"kind": "Interceptor",
"apiVersion": "gateway/v2",
"metadata": {
"name": "myFieldLevelDataMaskingPlugin"
},
"spec": {
"pluginClass": "io.conduktor.gateway.interceptor.FieldLevelDataMaskingPlugin",
"priority": 100,
"config": {
"schemaRegistryConfig": {
"host": "http://schema-registry:8081"
},
"policies": [
{
"name": "Mask password",
"rule": {
"type": "MASK_ALL"
},
"fields": [
"password"
]
},
{
"name": "Mask visa",
"rule": {
"type": "MASK_LAST_N",
"maskingChar": "X",
"numberOfChars": 4
},
"fields": [
"visa"
]
}
]
}
}
}'
```
```yaml theme={null}
apiVersion: gateway/v2
kind: Interceptor
metadata:
name: myFieldLevelDataMaskingPlugin
scope:
vCluster: passthrough
spec:
pluginClass: io.conduktor.gateway.interceptor.FieldLevelDataMaskingPlugin
priority: 100
config:
schemaRegistryConfig:
host: http://schema-registry:8081
policies:
- name: Mask password
rule:
type: MASK_ALL
fields:
- password
- name: Mask visa
rule:
type: MASK_LAST_N
maskingChar: X
numberOfChars: 4
fields:
- visa
```
Apply with:
```bash theme={null}
conduktor apply -f data-masking-interceptor.yaml
```
#### Secured schema registry
```bash theme={null}
curl \
--request PUT \
--url 'http://localhost:8888/gateway/v2/interceptor' \
--header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
--header 'Content-Type: application/json' \
--data-raw '{
"kind": "Interceptor",
"apiVersion": "gateway/v2",
"metadata": {
"name": "myFieldLevelDataMaskingPlugin"
},
"spec": {
"pluginClass": "io.conduktor.gateway.interceptor.FieldLevelDataMaskingPlugin",
"priority": 100,
"config": {
"schemaRegistryConfig": {
"host": "http://schema-registry:8081",
"additionalConfigs": {
"schema.registry.url": "${SR_URL}",
"basic.auth.credentials.source": "${SR_BASIC_AUTH_CRED_SRC}",
"basic.auth.user.info": "${SR_BASIC_AUTH_USER_INFO}"
}
},
"policies": [
{
"name": "Mask password",
"rule": {
"type": "MASK_ALL"
},
"fields": [
"password"
]
},
{
"name": "Mask visa",
"rule": {
"type": "MASK_LAST_N",
"maskingChar": "X",
"numberOfChars": 4
},
"fields": [
"visa"
]
}
]
}
}
}'
```
```yaml theme={null}
apiVersion: gateway/v2
kind: Interceptor
metadata:
name: myFieldLevelDataMaskingPlugin
scope:
vCluster: passthrough
spec:
pluginClass: io.conduktor.gateway.interceptor.FieldLevelDataMaskingPlugin
priority: 100
config:
schemaRegistryConfig:
host: http://schema-registry:8081
additionalConfigs:
schema.registry.url: $${SR_URL}
basic.auth.credentials.source: $${SR_BASIC_AUTH_CRED_SRC}
basic.auth.user.info: $${SR_BASIC_AUTH_USER_INFO}
policies:
- name: Mask password
rule:
type: MASK_ALL
fields:
- password
- name: Mask visa
rule:
type: MASK_LAST_N
maskingChar: X
numberOfChars: 4
fields:
- visa
```
Apply with:
```bash theme={null}
conduktor apply -f data-masking-secured-sr-interceptor.yaml
```
## Dynamic header injection Interceptor
This Interceptor injects headers (such as user IP) into records as they pass through Gateway. Use `target` to choose whether Gateway injects them when clients produce, when they consume, or both. We support templating in this format: `X-CLIENT_IP: "{{userIp}} testing"`.
### Choose the produce or consume path
The `target` setting controls which path the Interceptor runs on.
| `target` | What Gateway does | Use it when |
| :------------------ | :----------------------------------------------------------- | :------------------------------------------------------------------------------------------------ |
| `PRODUCE` (default) | Injects headers when clients produce through Gateway | producers go through Gateway and you want the header stored in the topic |
| `CONSUME` | Injects headers into the fetch response when clients consume | producers bypass Gateway, so consume is the only path Gateway sees |
| `BOTH` | Injects on both paths | some producers go through Gateway and some don't, and you want every record tagged when it's read |
In a future version we will stop defaulting `target` to `PRODUCE` and so we log a warning when it is not set. We advise you to set it explicitly.
On consume, Gateway adds the headers to the fetch response but does **not** write them to the topic, so the record in Kafka stays unchanged.
With `BOTH`, Gateway skips a header on consume if the record already has it, so records aren't tagged twice. Set `overrideIfExists` to `true` to inject anyway.
### Context variables
These values are available as template variables:
* uuid
* userIp
* vcluster
* user
* clientId
* gatewayIp
* gatewayHost
* gatewayVersion
* apiKey
* apiKeyVersion
* timestampMillis
Context variables resolve relative to the path the Interceptor runs on. With `target: CONSUME` or `BOTH`, `clientId`, `user`, `userIp` and `vcluster` describe the **consumer** rather than the producer, and `apiKey` is `Fetch` instead of `Produce`. Gateway regenerates `uuid` and `timestampMillis` on every fetch, so each read of the same record gets different values.
When Gateway sits behind a load balancer, `userIp` resolves to the original client address only when Gateway [captures it](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/load-balancing#capturing-the-client-ip-address). Otherwise, it's the load balancer's address.
#### Record extraction templates
You can also extract fields from the record key or value:
* `{{record.key}}` - extract the entire key payload as a string
* `{{record.value}}` - extract the entire value payload as a string
* `{{record.key.fieldName}}` - extract a specific field from the record key
* `{{record.value.fieldName}}` - extract a specific field from the record value
For example, if your record has a key with a field named "id", you can use `{{record.key.id}}` to extract that value and inject it as a header.
To use field extraction (`record.key.fieldName` or `record.value.fieldName`) with Avro, JSON Schema or Protobuf data, give the Interceptor a schema registry so Gateway can deserialize the records — either `schemaRegistryConnectionName` or `schemaRegistryConfig`. For plain JSON data, no schema registry is needed.
### Configure header injection Interceptor
| Config | Type | Default | Description |
| :--------------------------- | :---------------------------------- | :-------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| topic | String | | Regular expression that matches the topics the Interceptor applies to. |
| target | Enum | `PRODUCE` | Which path to inject on: `PRODUCE`, `CONSUME` or `BOTH`. See [Choose the produce or consume path](#choose-the-produce-or-consume-path). |
| headers | Map | | Map of header key and header value to inject. Values can use template variables like `{{userIp}}` or record extraction patterns. |
| overrideIfExists | boolean | `false` | Whether to override headers that already exist on the record. |
| failOnError | boolean | `false` | Whether to reject the request when header injection fails. When `false`, Gateway logs the error and continues. See [Error handling](#error-handling). |
| schemaRegistryConfig | [Schema registry](#schema-registry) | | Schema Registry configuration. Required when using `record.key.field` or `record.value.field` templates with schema-encoded data. |
| schemaRegistryConnectionName | String | | Name of a [schema registry external connection](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/external-connections). An alternative to `schemaRegistryConfig` — set one or the other, not both. |
#### Error handling
The `failOnError` setting controls how the Interceptor handles errors during header injection, such as a missing field or a payload it can't deserialize.
| `failOnError` | On produce | On consume |
| :---------------- | :------------------------------------------------------ | :----------------------------------------------------------------------------------------------------------------------------------------- |
| `false` (default) | Gateway logs a warning, skips that header and continues | Same as produce |
| `true` | The request fails and the producer gets an error | Gateway fails the whole fetch response. The consumer gets no records and no error, so it stalls while Gateway logs an error on each retry. |
If `target` is `CONSUME` or `BOTH`, we recommend leaving `failOnError` at `false`. To fail produce requests but only log on consume, use two Interceptors with different names: one with `target: PRODUCE` and `failOnError: true`, and one with `target: CONSUME` and `failOnError: false`.
### Header injection Interceptor example
```bash theme={null}
curl \
--request PUT \
--url 'http://localhost:8888/gateway/v2/interceptor' \
--header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
--header 'Content-Type: application/json' \
--data-raw '{
"kind": "Interceptor",
"apiVersion": "gateway/v2",
"metadata": {
"name": "myDynamicHeaderInjectionInterceptor"
},
"spec": {
"pluginClass": "io.conduktor.gateway.interceptor.DynamicHeaderInjectionPlugin",
"priority": 100,
"config": {
"topic": "topic.*",
"target": "PRODUCE",
"headers": {
"X-CLIENT_IP": "{{userIp}} testing",
"X-USER-ID": "{{record.key.id}}",
"X-USER-EMAIL": "{{record.value.email}}"
},
"overrideIfExists": true,
"failOnError": false,
"schemaRegistryConfig": {
"host": "http://schema-registry:8081"
}
}
}
}'
```
```yaml theme={null}
apiVersion: gateway/v2
kind: Interceptor
metadata:
name: myDynamicHeaderInjectionInterceptor
scope:
vCluster: passthrough
spec:
pluginClass: io.conduktor.gateway.interceptor.DynamicHeaderInjectionPlugin
priority: 100
config:
topic: topic.*
target: PRODUCE
headers:
X-CLIENT_IP: "{{userIp}} testing"
X-USER-ID: "{{record.key.id}}"
X-USER-EMAIL: "{{record.value.email}}"
overrideIfExists: true
failOnError: false
schemaRegistryConfig:
host: http://schema-registry:8081
```
Apply with:
```bash theme={null}
conduktor apply -f header-injection-interceptor.yaml
```
Let's produce a simple record to the `injectHeaderTopic` topic.
```bash theme={null}
echo 'inject_header' | docker-compose exec -T kafka-client \
kafka-console-producer \
--bootstrap-server conduktor-gateway:6969 \
--producer.config /clientConfig/gateway.properties \
--topic injectHeaderTopic
```
Let's consume from our `injectHeaderTopic`.
```bash theme={null}
docker-compose exec kafka-client \
kafka-console-consumer \
--bootstrap-server conduktor-gateway:6969 \
--consumer.config /clientConfig/gateway.properties \
--topic injectHeaderTopic \
--from-beginning \
--max-messages 1 \
--property print.headers=true
```
You should see the message with headers as below
```md theme={null}
X-USER_IP:172.19.0.3 testing inject_header
```
## Message integrity Interceptor
The message integrity Interceptor signs Kafka records on produce and verifies them on fetch, letting consumers detect whether a record changed after it was produced. Two plugins work together:
* **ProduceIntegrityPolicyPlugin** signs records using HMAC-SHA256 through Google Tink.
* **FetchIntegrityPolicyPlugin** verifies signatures and drops or allows records based on your policy.
You store signing keys in your HashiCorp Vault Key-Value (KV) v2 instance. Gateway reads and caches them locally.
### Ordering with other Interceptors
The message integrity Interceptor is always the outermost layer — signing runs last on produce (after all other Interceptors have transformed the record) and verification runs first on fetch (before any normal Interceptor runs). This ensures the signature covers the final produced payload and is verified before any transformation on consume.
Gateway enforces this ordering automatically. You don't have to set specific priority values for integrity Interceptors — Gateway places them in fixed pipeline positions regardless of their configured priority:
* **Produce:** the sign plugin always runs after all other Interceptors.
* **Fetch:** the verify plugin always runs before all other Interceptors.
Gateway also validates that two integrity Interceptors of the same type don't have overlapping scopes (Virtual Cluster, group or username). If they do, Gateway rejects the configuration.
The Interceptor reads Vault credentials **only from its config**. You have to specify credential fields (such as `token`, `roleId` or `secretId`) in the config. You can set their values with placeholders like `token: "${VAULT_TOKEN}"`, which Gateway resolves when it loads the config. Gateway does not fall back to environment variable names when you omit a credential field (for example, there is no built-in fallback to `VAULT_TOKEN` if you leave out `token`).
When Vault is unreachable (for example, on a cache miss), Gateway propagates the error to the Kafka client: producers receive the failure on produce and consumers receive it on fetch. Gateway does not silently drop records in these cases.
### Configure the secretKeyUri
`secretKeyUri` points to a specific field in a KV v2 secret. Use the format `/data/#`.
Examples: `secret/data/signing-key#key`, `secret/data/app/keys/signing#hmacKey`.
### Manage key versions in Vault KV v2
Vault KV v2 versions secrets: each write to the same path creates a new version. Gateway handles versions as follows:
* **Produce (sign):** Gateway uses the **latest** version of the secret at the path you configured. When you write a new value to that path in Vault, Gateway picks it up once the cache entry expires (see `cache.ttlMs`). Each signed record stores the key version in its signature header.
* **Fetch (verify):** The signature on each record identifies the key version used to sign it. Gateway fetches that exact version from Vault to verify, so records signed with an older version still verify correctly after you rotate to a newer version.
* **Older versions:** Keep older secret versions readable in Vault until you no longer need to verify records signed with them (for example, until they are consumed or past your retention period).
### Configure produce (sign) plugin
If a record already has the signature header, the Interceptor throws `PolicyViolationException` and does not re-sign.
| Key | Type | Default | Description |
| ----------------- | ------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| topic | String | `.*` | Topics matching this regex have the Interceptor applied |
| signatureHeader | String | `conduktor.integrity.signature` | Header name for the signature |
| secretKeyUri | String | — | Vault KV v2 path and field (for example, `secret/data/signing-key#key`) |
| keyProviderConfig | Object | — | Key provider (Vault); see [Authenticate with Vault](#authenticate-with-vault-for-message-integrity) below |
| cache.ttlMs | Long | 300000 | How long (in milliseconds) the signing key is cached before Gateway fetches it again from Vault. A shorter TTL means rotated keys are picked up faster (default: five minutes) |
| cache.maxSize | Integer | 100 | Maximum number of keys to cache |
### Configure fetch (verify) plugin
Gateway drops records that fail verification (missing signature, malformed header or invalid MAC (Message Authentication Code)) and emits an audit event. After successful verification, Gateway removes the signature header from the record before returning it to the consumer. For other errors (such as Vault being unreachable on a cache miss), Gateway propagates the error to the Kafka client and does not silently drop the record.
When a record is dropped or allowed with missing signature, the fetch (verify) plugin emits a **fetch response audit event** (error level). To receive these events, enable the audit feature with `GATEWAY_FEATURE_FLAGS_AUDIT` (see [Audit logs](/guide/conduktor-in-production/admin/audit-logs) and [Environment variables](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables)).
**Audit event details:**
* **Event type:** fetch response audit event (`level`: `error`)
* **Information included:** topic, partition, offset, Interceptor name, plugin name (`FetchIntegrityPolicyPlugin`) and a message describing the reason
* **Reason values:** `missing_signature` (no signature header), `malformed_signature` (header could not be decoded), `verification_failed:INVALID_SIGNATURE` (MAC does not match) or `verification_failed:UNKNOWN_KEY` (key version not found in Vault)
| Key | Type | Default | Description |
| ---------------------- | ------- | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| topic | String | `.*` | Topics matching this regex have the Interceptor applied |
| signatureHeader | String | `conduktor.integrity.signature` | Header name where Gateway stores the signature |
| missingSignaturePolicy | String | `SKIP` | When a record has no signature header: `SKIP` (drop and audit) or `ALLOW` (audit and keep in response) |
| keyProviderConfig | Object | — | Key provider (Vault); see [Authenticate with Vault](#authenticate-with-vault-for-message-integrity) below |
| cache.ttlMs | Long | 300000 | How long (in milliseconds) a verification key stays in cache before Gateway fetches it again from Vault. This controls how long previously seen keys are reused without a Vault lookup |
| cache.maxSize | Integer | 100 | Maximum number of keys to cache |
### Authenticate with Vault for message integrity
All auth types use the common fields: `uri` (required) and optionally `namespace`, `openTimeoutSeconds` (default five), `readTimeoutSeconds` (default 30), `keyStore`, `trustStore` and `connectionBackoff`. Set `type` to one of the following and add the corresponding fields.
You can also configure TLS for the Vault connection:
* **keyStore:** set `keyStorePath` and `keyStorePassword` for client certificate authentication
* **trustStore:** set `trustStorePath` and `trustStorePassword` to verify the Vault server certificate
| type | Required fields | Optional fields |
| ---------------------- | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------- |
| **TOKEN** | token | — |
| **APP\_ROLE** | roleId, secretId | mount (default `approle`) |
| **KUBERNETES** | role | tokenPath (default `/var/run/secrets/kubernetes.io/serviceaccount/token`), mount (default `kubernetes`) |
| **USERNAME\_PASSWORD** | username, password | mount (default `userpass`) |
| **GITHUB** | token | mount (default `github`) |
| **LDAP** | username, password | mount (default `ldap`) |
| **GCP** | role, jwt | — |
| **AWS\_EC2\_PKCS7** | pkcs7 | role, nonce, mount (default `aws`) |
| **AWS\_EC2** | identity, signature | role, nonce, mount (default `aws`) |
| **AWS\_IAM** | iamRequestUrl, iamRequestBody, iamRequestHeaders | role, mount (default `aws`) |
| **JWT** | provider, role, jwt | — |
All auth types except TOKEN support automatic token renewal. Gateway renews Vault tokens in the background so your Interceptor continues to work without interruption.
Optional `connectionBackoff` (for transient Vault failures): `backoffDelay` (default five), `backoffMaxDelay` (default 30), `backoffChronoUnit` (default SECONDS), `backoffDelayFactor` (default 1.1).
### Set up Vault for message integrity
1. Enable KV v2: `vault secrets enable -version=2 kv`
2. **Create a signing key** for HMAC-SHA256:
* The key material has to be **at least 32 bytes (256 bits)** after decoding. Gateway enforces this per NIST SP 800-107 Rev 1 and rejects shorter keys with an error.
* Store the key in a KV v2 secret as a **base64-encoded string**. Gateway decodes the Base64 value and uses the resulting bytes, so the decoded length has to be at least 32 bytes.
* The **field name** in the secret has to match the `` in your `secretKeyUri` (for example, `secret/data/signing-key#key` uses field name `key`).
* Example: generate 32 random bytes, base64-encode them for storage, then write to Vault: `KEY=$(openssl rand -base64 32)` then `vault kv put -mount=secret signing-key key="$KEY"`
3. Create a policy for Gateway with read access on the secret path (for example, `path "secret/data/signing-key" { capabilities = ["read"] }`)
### Understand the signature format
Gateway stores each signature in a Kafka header as JSON with two fields: `k` = the secretKeyUri with its version (for example, `secret/data/signing-key#key@1`) and `s` = the base64-encoded HMAC-SHA256 MAC.
### Message integrity Interceptor examples
```yaml theme={null}
apiVersion: gateway/v2
kind: Interceptor
metadata:
name: integrity-sign
scope:
vCluster: passthrough
spec:
pluginClass: io.conduktor.gateway.interceptor.integrity.ProduceIntegrityPolicyPlugin
config:
topic: ".*"
signatureHeader: "conduktor.integrity.signature"
secretKeyUri: "secret/data/signing-key#key"
keyProviderConfig:
vault:
uri: "http://localhost:8200"
type: TOKEN
token: "${VAULT_TOKEN}"
cache:
ttlMs: 3600000
maxSize: 100
```
```yaml theme={null}
apiVersion: gateway/v2
kind: Interceptor
metadata:
name: integrity-verify
scope:
vCluster: passthrough
spec:
pluginClass: io.conduktor.gateway.interceptor.integrity.FetchIntegrityPolicyPlugin
config:
topic: ".*"
signatureHeader: "conduktor.integrity.signature"
missingSignaturePolicy: SKIP
keyProviderConfig:
vault:
uri: "http://localhost:8200"
type: TOKEN
token: "${VAULT_TOKEN}"
cache:
ttlMs: 3600000
maxSize: 100
```
Apply with: `conduktor apply -f integrity-sign-interceptor.yaml` and `conduktor apply -f integrity-verify-interceptor.yaml`.
## Encryption Interceptors
Gateway encrypts your Kafka data as it passes through the proxy, before it reaches the broker.
Unlike TLS (Transport-Level Encryption), Gateway encryption ensures data remains encrypted when stored on Kafka brokers.
The section covers all of the configuration options available for every encryption Interceptor.
You can also check out other resources:
* [encryption use cases](/guide/use-cases/kafka-encryption)
* [encryption examples](/guide/tutorials/configure-encryption)
### Encryption configuration
The properties detailed in this section work for the following plugins:
| | Plugin |
| ------------ | -------------------------- |
| List-based | `EncryptPlugin` |
| Schema-based | `EncryptSchemaBasedPlugin` |
Both schema-based and list-based encryption plugins have their configuration, but some properties are common to both of them.
| Key | Type | Default | Description |
| -------------------------------------------- | ---------------------------------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Common properties** | | | |
| `topic` | String | `.*` | Topics matching this regex will have the interceptor applied. |
| `schemaRegistryConfig` | [SchemaRegistry](#schema-registry) | | Configuration of your schema registry. Required if you want to encrypt data produced using Avro, JSON or Protobuf schemas. |
| `schemaRegistryConnectionName` | String | | Name of a [schema registry external connection](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/external-connections). An alternative to `schemaRegistryConfig` — set one or the other, not both. |
| `schemaDataMode` | String | `preserve_avro` | Controls the format Gateway writes when it encrypts an Avro record. `preserve_avro` keeps the record in Avro, so it stays linked to its schema in the backing topic. This is the default and the recommended value since Gateway v3.3. **`convert_json` is deprecated and will be removed in Gateway 3.24.0.** It converts the record to JSON and breaks the link to its schema in the backing topic. Existing configurations continue to work, and Gateway logs a deprecation warning when it loads them. |
| `kmsConfig` | [KMS](#kms-configuration) | | Configuration of one or multiple KMS. |
| `kmsConnectionName` | String | | Name of a [KMS external connection](#name-a-kms-external-connection). An alternative to `kmsConfig` — set one or the other, not both |
| `enableAuditLogOnError` | Boolean | true | The audit log will be enabled when an error occurs during encryption/decryption |
| `compressionType` | Enum | none | The data is compressed before encryption (only for data configured with full payload encryption). Available values are: `none`, `gzip`, `snappy`, `lz4` or `zstd`. |
| `throttleTimeMs` | Integer | 0 | When encryption fails, apply client throttling for the specified time in milliseconds. This helps prevent overwhelming the system during encryption failures. [Find out more about client throttling](#client-throttling) |
| `errorPolicy` | [String](#encryption-error-policy) | `fail_on_encrypted` | Determines the plugin behavior when it encounters a record that's already encrypted. Possible values: `fail_on_encrypted`, `skip_already_encrypted` |
| [**List-based properties**](#list-based) | | | |
| `recordValue` | Value and key encryption | | Configuration to encrypt the record value. |
| `recordKey` | Value and key encryption | | Configuration to encrypt the record key. |
| `recordHeader` | Headers encryption | | Configuration to encrypt the record headers. |
| [**Schema-based properties**](#schema-based) | | | |
| `defaultKeySecretId` | [Key reference](#kms-integration) | | Default key URI to use if none is set in the schema, when the Interceptor configures its KMS inline. Can be a template for Crypto Shredding use cases. |
| `defaultKeyId` | [Key reference](#kms-integration) | | Bare key name for every encrypted field, when the Interceptor names a KMS connection. |
| `defaultAlgorithm` | [Algorithm](#supported-algorithms) | `AES128_GCM` | Default `algorithm` to use if no algorithm is set in the schema. |
| `tags` | List\[String] | | List of tags to search for in the schema to encrypt the specified fields. |
| `namespace` | String | `conduktor.` | Prefix of custom schema constraints for encryption. |
#### List-based
Decide what you want to encrypt:
* **Record value and record key**:
* Encrypt a set of fields
* Encrypt the full payload
* or **header keys**:
* Encrypt a set of fields
* Encrypt the full payload
* Encrypt a set of headers that match a regex
**Record values and record keys**
Set the following properties for `recordValue` (value encryption) and/or `recordKey` (key encryption):
| Key | Type | Default | Description |
| --------------------------- | ---------------------------------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------- |
| **Full-payload encryption** | | | |
| `payload.keySecretId` | [Key reference](#kms-integration) | | Key URI, when the Interceptor configures its KMS inline. Can be a template for Crypto Shredding use cases. |
| `payload.keyId` | [Key reference](#kms-integration) | | Bare key name, when the Interceptor names a KMS connection. Set this or `keySecretId`, never both. |
| `payload.algorithm` | [Algorithm](#supported-algorithms) | `AES128_GCM` | Algorithm to leverage. |
| **Field-Level encryption** | | | |
| `fields[].fieldName` | String | | Name of the field to encrypt. It can be a nested structure with a dot `.` such as `education.account.username` or `banks[0].accountNo`. |
| `fields[].keySecretId` | [Key reference](#kms-integration) | | Key URI, when the Interceptor configures its KMS inline. Can be a template for crypto shredding use cases. |
| `fields[].keyId` | [Key reference](#kms-integration) | | Bare key name, when the Interceptor names a KMS connection. Set this or `keySecretId`, never both. |
| `fields[].algorithm` | [Algorithm](#supported-algorithms) | | Algorithm to use to encrypt this field. |
[Check out the encryption examples](/guide/tutorials/configure-encryption#encryption-examples).
**Header keys**
Set the following properties for `recordHeader`:
| Key | Type | Default | Description |
| --------------------------- | ---------------------------------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------- |
| **Full-payload encryption** | | | Configuration to encrypt the full payload. |
| `payload.keySecretId` | [Key reference](#kms-integration) | | Key URI, when the Interceptor configures its KMS inline. Can be a template for crypto shredding use cases. |
| `payload.keyId` | [Key reference](#kms-integration) | | Bare key name, when the Interceptor names a KMS connection. Set this or `keySecretId`, never both. |
| `payload.algorithm` | [Algorithm](#supported-algorithms) | `AES128_GCM` | Algorithm to leverage. |
| **Field-level encryption** | | | |
| `fields[].fieldName` | String | | Name of the field to encrypt. It can be a nested structure with a dot `.` such as `education.account.username` or `banks[0].accountNo`. |
| `fields[].keySecretId` | [Key reference](#kms-integration) | | Key URI, when the Interceptor configures its KMS inline. Can be a template for crypto shredding use cases. |
| `fields[].keyId` | [Key reference](#kms-integration) | | Bare key name, when the Interceptor names a KMS connection. Set this or `keySecretId`, never both. |
| `fields[].algorithm` | [Algorithm](#supported-algorithms) | | |
| **Headers encryption** | | | |
| `header` | String | | Headers that match this regex will be encrypted.It can encrypt all headers including Gateway headers. |
[Check out the encryption example](/guide/tutorials/configure-encryption#encryption-examples).
#### Schema-based
In order to encrypt your data, you can set a few constraints in your schema. These constraints are detailed below, assuming you're using the default `namespace` value which is `conduktor.`. If you have changed the `namespace` value in the Interceptor configuration, please change the key name in your schema accordingly.
| Key | Type | Default | Description |
| ----------------------- | ---------------------------------- | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `conduktor.keySecretId` | [Key reference](#kms-integration) | | Key URI, when the Interceptor configures its KMS inline. Can be a template for crypto shredding use cases. |
| `conduktor.algorithm` | [Algorithm](#supported-algorithms) | `AES128_GCM` | Algorithm to use to encrypt this field. |
| `conduktor.tags` | List\[String] | | Fields tagged with a matching tag from your Interceptor will be encrypted using the key and `algorithm` specified in the schema. If these are not defined in the schema, the `defaultKeySecretId` (or `defaultKeyId`) and `defaultAlgorithm` from the Interceptor configuration will be used. |
If your field meets one of these three conditions, then it will be encrypted:
1. This field has a `keySecretId` set in the schema
2. This field has a `algorithm` set in the schema
3. This field has a set of `tags` set in the schema, and one of them is part of the `tags` list specified in the Interceptors.
Configure the KMS inline for schema-based encryption. A [KMS external connection](#name-a-kms-external-connection) doesn't yet honor per-field keys from a schema.
[Check out the encryption example](/guide/tutorials/configure-encryption#encryption-examples).
#### Secret keys
**Mustache template**
In all the encryption plugins, you can use mustache templates for the key an entry names — `keySecretId` inline, or `keyId` when the Interceptor names a [KMS external connection](#name-a-kms-external-connection). That way, your keys are dynamic.
The value of a field will be replaced with the encrypted value; so using the keyId as the encryption value isn't allowed.
| Pattern | Replaced by |
| --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `{{record.topic}}` | Name of the topic you're encrypting the data of. |
| `{{record.key}}` | Key of the encrypted record. |
| `{{record.value.someValueFieldName}}` | Value of the field called `someValueFieldName` If you're doing field-level encryption, please ensure that `someValueFieldName` is not included in the fields to encrypt. Otherwise, you will not be able to decrypt it. |
| `{{record.value.someList[0].someValueField}}` | Value of the field called `someValueFieldName`, in the first element of the list `someList` |
| `{{record.header.someHeader}}` | Value of the header called `someHeader` |
Here's a record example:
```md theme={null}
# Header
someHeader=myHeader
# Key
myKey
# Value
{
"someValueFieldName": "I",
"someList": [{ "someValueField": "love" }, { "someValueField": "Kafka" }]
}
```
Both fields take the same templates. Where the resulting key lives depends on how the Interceptor reaches its KMS:
```json theme={null}
{
"kmsConnectionName": "PROD_VAULT",
"recordValue": {
"fields": [{ "fieldName": "email", "keyId": "{{record.topic}}-{{record.header.someHeader}}-{{record.key}}" }]
}
}
```
The connection already knows the provider, so this creates a key called `myTopic-myHeader-myKey` in that KMS.
```json theme={null}
{
"kmsConfig": { "vault": { "uri": "https://vault:8200", "token": "${VAULT_TOKEN}" } },
"recordValue": {
"fields": [{ "fieldName": "email", "keySecretId": "vault-kms://https://vault:8200/transit/keys/{{record.topic}}-{{record.header.someHeader}}-{{record.key}}" }]
}
}
```
The `vault-kms://` prefix is what puts the key in Vault. Drop it — `"keySecretId": "{{record.topic}}-{{record.header.someHeader}}-{{record.key}}"` — and the key is created in memory instead.
#### KMS integration
An encryption entry names its key in one of two ways, depending on how its Interceptor reaches the KMS:
| The Interceptor sets | Name each key with | Format | Example |
| ---------------------------------------------------------------------------------- | ------------------ | -------------------------------------------------------- | -------------------------------------------------------- |
| `kmsConnectionName` — a [KMS external connection](#name-a-kms-external-connection) | `keyId` | Bare key name; the connection already knows the provider | `email-key`, or `alias/email-key` for AWS KMS |
| `kmsConfig` — [inline configuration](#configure-a-kms-inline) | `keySecretId` | Full key URI, prefixed with the KMS type | `vault-kms://vault.internal:8200/transit/keys/email-key` |
Set one or the other on an entry, never both. Both accept [mustache templates](#secret-keys).
```json theme={null}
{
"kmsConnectionName": "PROD_VAULT",
"recordValue": {
"fields": [{ "fieldName": "email", "keyId": "email-key" }]
}
}
```
```json theme={null}
{
"kmsConfig": { "vault": { "uri": "https://vault.internal:8200", "token": "${VAULT_TOKEN}" } },
"recordValue": {
"fields": [{ "fieldName": "email", "keySecretId": "vault-kms://vault.internal:8200/transit/keys/email-key" }]
}
}
```
For the `kmsType://` prefixes and their URI formats, see [configure a KMS inline](#configure-a-kms-inline).
Every entry has to carry the field matching the Interceptor's own choice, and carry it non-blank. Gateway rejects the rest when you create or update the Interceptor, naming which field to use.
#### Supported algorithms
* `AES128_GCM` (default)
* `AES128_EAX`
* `AES256_EAX`
* `AES128_CTR_HMAC_SHA256`
* `AES256_CTR_HMAC_SHA256`
* `CHACHA20_POLY1305`
* `XCHACHA20_POLY1305`
* `AES256_GCM`
#### Choosing an encryption algorithm
Gateway supports multiple encryption algorithms, with `AES128_GCM` as the default. When selecting an algorithm, consider your security requirements, performance needs, and message volume.
**Default algorithm: AES128\_GCM**
`AES128_GCM` is the default algorithm and is suitable for most use cases. However, it has an important security limitation:
**Security consideration**: When the same DEK (Data Encryption Key) is used to encrypt approximately 4 billion (2³²) messages, AES-GCM's security guarantees degrade due to nonce collision risks. According to [NIST Special Publication 800-38D (Section 8.3)](https://csrc.nist.gov/pubs/sp/800/38/d/final), nonce collisions may expose encryption keys, compromising the confidentiality and integrity of data encrypted with that key.
For high-traffic scenarios, this threshold can be reached quickly. For example:
* At 10-50MB/s with 1KB message sizes, the 4-billion-message threshold can be reached in just over 24 hours
* Each unique `keySecretId` uses its own DEK, so the limit applies per key, not globally
If you expect a single DEK to encrypt more than \~2³² messages, consider:
* Using a different algorithm (see recommendations below)
* Implementing DEK rotation before reaching the threshold
* Using multiple `keySecretId` values to distribute the message count across multiple DEKs
**When to keep the default (AES128\_GCM)**
* Low to moderate message volume per DEK (well below 2³² messages per key)
* Need for compatibility with existing AES-GCM implementations
* Hardware acceleration (AES-NI) is available, providing good performance
If you expect a single DEK to encrypt more than \~2³² messages, consider using alternative algorithms or implementing DEK rotation. Consult your cryptographic library documentation and security requirements to choose the appropriate algorithm for your use case.
#### Key rotation
Gateway uses envelope encryption with two types of keys: **DEK** (Data Encryption Key) and **KEK** (Key Encryption Key). Understanding how and when to rotate these keys is important for maintaining security.
**KEK rotation**
**Customer responsibility**: KEK (Key Encryption Key) rotation must be performed by the customer and is **not handled automatically by Gateway**. This is the customer's responsibility and must be done through your KMS provider (AWS KMS, Azure Key Vault, HashiCorp Vault, GCP KMS, etc.).
**When to rotate KEK**
You should rotate your KEK based on:
* **Security policies**: Follow your organization's key rotation policies and compliance requirements
* **DEK encryption frequency**: If DEKs are being encrypted frequently (high message volume), consider more frequent KEK rotation
* **Security incidents**: Rotate immediately if a KEK is suspected to be compromised
* **Best practices**: Many organizations rotate KEKs annually or quarterly, but the frequency should match your security requirements
After rotating a KEK, Gateway will automatically use the new KEK version for encrypting new DEKs. However, existing EDEKs encrypted with the old KEK version will still be de-cryptable as long as the old KEK version remains available in your KMS.
Most KMS providers retain old key versions for backward compatibility, allowing you to decrypt historical data while new data uses the rotated key.
#### Supported compression types
* `none`
* `gzip`
* `snappy`
* `lz4`
* `zstd`
#### Encryption error policy
This policy determines the actions when an encryption Interceptor encounters a record that's already encrypted.
| **Error policy** | **Description** |
| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `fail_on_encrypted` | The encryption operation will fail with an exception when encountering already encrypted records. This is the default behavior and maintains backward compatibility. |
| `skip_already_encrypted` | The encryption Interceptor will skip already encrypted records and pass them through unchanged. This enables chaining multiple encryption Interceptors together. |
**Example configuration with error policy:**
```json theme={null}
{
"name": "mySchemaBasedEncryptPlugin",
"pluginClass": "io.conduktor.gateway.interceptor.EncryptSchemaBasedPlugin",
"config": {
"topic": "sensitive-data",
"defaultKeySecretId": "vault-kms://vault:8200/transit/keys/default-key",
"defaultAlgorithm": "AES256_GCM",
"tags": ["PII", "ENCRYPT"],
"errorPolicy": "skip_already_encrypted",
"kmsConfig": {
"vault": {
"uri": "http://vault:8200",
"token": "${VAULT_TOKEN}"
}
}
}
}
```
### Tokenization
Tokenization replaces a sensitive value with an unrelated token, and keeps the mapping in Vault rather than in your Kafka record. A team can work with the tokenized field without the values ever leaving Vault.
Tokenization only works with the [Transform Secrets Engine](https://developer.hashicorp.com/vault/docs/secrets/transform), available in Vault Enterprise. No other KMS provider supports it for now, but we would love to hear from you.
Gateway calls the Transform role named in your `keySecretId` and writes the token it gets back into the record. Vault owns the behavior, so create the role with the `tokenization` transformation, which is the configuration Gateway is tested against:
```bash theme={null}
vault write transform/transformations/tokenization/email allowed_roles=email
vault write transform/role/email transformations=email
```
With this transformation, Vault stores a mapping of the token to the encrypted plaintext, and the token carries no link to the original value, so it isn't format-preserving.
#### When to use tokenization
Tokenization fits when a team has to work with the data rather than just move it: reporting on unique customer counts, or joining datasets on a customer reference.
It's a weaker fit when the same clear text can reach the same place as the tokens. Tokens are deterministic, so anyone holding a clear value and its token learns every other record carrying that value. If you can't guarantee clear text stays out, use non-deterministic [field level encryption](#encryption-configuration) instead. See [choose how to protect your data](/guide/use-cases/kafka-encryption) for the full comparison.
#### Configure tokenization
Tokenization uses the existing encryption Interceptors (`EncryptPlugin`, `DecryptPlugin` and so on). To tokenize data, use the encryption Interceptors with a `vault-transform://` prefix in your `keySecretId`. To de-tokenize and retrieve the original values, use the decryption Interceptors with the same configuration. See [how to configure Vault](#vault-kms).
Here's a sample field configured for tokenization:
```yaml theme={null}
fields:
- fieldName: "email"
keySecretId: "vault-transform://email-key"
```
Configure the KMS inline for tokenization. A [KMS external connection](#name-a-kms-external-connection) only reaches Vault's Transit engine, so it can't use the Transform engine tokenization relies on. Set `kmsConfig` on the Interceptor rather than `kmsConnectionName`.
### Decryption configuration
Now that your fields or payload are encrypted, you can decrypt them using the Interceptor `DecryptPlugin`.
| Key | Type | Default | Description |
| ------------------------------ | ---------------------------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `topic` | String | `.*` | Topics matching this regex will have the Interceptor applied. |
| `schemaRegistryConfig` | [SchemaRegistry](#schema-registry) | | Configuration of your schema registry, is needed if you want to decrypt into Avro, JSON or Protobuf schemas. |
| `schemaRegistryConnectionName` | String | | Name of a [schema registry external connection](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/external-connections). An alternative to `schemaRegistryConfig` — set one or the other, not both. |
| `kmsConfig` | [KMS](#kms-configuration) | | Configuration of one or multiple KMS |
| `kmsConnectionName` | String | | Name of a [KMS external connection](#name-a-kms-external-connection). An alternative to `kmsConfig` — set one or the other, not both |
| `recordValueFields` | List\[String] | | **Only for field-level encryption** - List of fields to decrypt in the value. If empty, we decrypt all the encrypted fields. |
| `recordKeyFields` | List\[String] | | **Only for field-level encryption** - List of fields to decrypt in the key. If empty, we decrypt all the encrypted fields. |
| `recordHeaderFields` | List\[String] | | **Only for field-level encryption** - List of headers to decrypt. If empty, we decrypt all the encrypted headers. |
| `enableAuditLogOnError` | Boolean | true | The audit log will be enabled when an error occurs during encryption/decryption |
| `errorPolicy` | String | `return_encrypted` | Determines the action if there is an error during decryption. The options are `return_encrypted`, `fail_fetch` and `crypto_shred_safe_fail_fetch` See [Decryption Error Policy](#decryption-error-policy) section for more details. |
| `throttleTimeMs` | Integer | 0 | When decryption fails, apply client throttling for the specified time in milliseconds. This helps prevent the overwhelming of the system during decryption failures. [Find out more about client throttling](#client-throttling) |
#### Decryption error policy
This policy determines the action if there is an error during decryption.
| **Error policy** | **Description** |
| ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `return_encrypted` | The encrypted data is returned to the client. |
| `fail_fetch` | The client will receive an error for the fetch and no data. When using the [Gateway KMS](#gateway-kms), data that fails to decrypt because no key was found will use the policy `return_encrypted` instead (supports crypto shredding for just this KMS system). |
| `crypto_shred_safe_fail_fetch` | The client will receive an error for the fetch and no data. Data that fails to decrypt because no key was found will use the policy `return_encrypted` instead (supports crypto shredding for all KMS systems). |
### KMS configuration
An encrypt or decrypt Interceptor reaches its KMS in one of two ways:
| Path | The Interceptor sets | Where the provider is configured | Keys are named with |
| ----------------------------------------------------------------- | -------------------- | -------------------------------- | ------------------- |
| [Name a KMS external connection](#name-a-kms-external-connection) | `kmsConnectionName` | Once, in Gateway's environment | `keyId` |
| [Configure a KMS inline](#configure-a-kms-inline) | `kmsConfig` | On each Interceptor | `keySecretId` |
Set one or the other, never both — Gateway rejects a configuration with both rather than silently picking one. [See how each names its keys](#kms-integration).
#### Configuration properties
| Property | Type | Default | Description |
| -------- | ---- | ------- | ------------------------------------------------------------------------------------------------ |
| keyTtlMs | long | 3600000 | Key's time-to-live in milliseconds. The default is 1 hour. Disable the cache by setting it to 0. |
#### Name a KMS external connection
KMS external connections are a preview feature and subject to change. Inline `kmsConfig` supports more providers, keeps working and isn't deprecated — [see which providers are available as an external connection](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/external-connections#connect-to-a-kms).
Instead of a `kmsConfig` block, an encryption or decryption Interceptor can name a [KMS external connection](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/external-connections#connect-to-a-kms) declared once when Gateway starts. The address, credentials and engine live in one place, so rotating a Vault credential is a single edit rather than one per Interceptor.
| Key | Type | Description |
| ------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `kmsConnectionName` | String | The `` of a KMS connection, declared with `GATEWAY_EXTERNAL_CONNECTION__CONFIG_` variables. Set at the top level of the Interceptor configuration, not inside `kmsConfig` |
#### Configure a KMS inline
Set a `kmsConfig` block on the Interceptor, holding one or more providers from the catalogue below. Each Interceptor carries its own copy of the address and credentials.
Each key is named with a full URI in `keySecretId`. Any that doesn't match one of the schemas detailed below will be **rejected** and the encryption operation will **fail**.
Keys are strings that start with a letter followed by a combination of letters, underscores (\_), hyphens (-) and numbers. Special characters are not allowed.
They also work with the mustache pattern.
To make sure the key is created in your KMS, configure the provider in `kmsConfig` and use the matching format:
| KMS | KMS identifier prefix | Key URI format | Example |
| ----------------- | --------------------- | ---------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| In-Memory | in-memory-kms\:// | `in-memory-kms://` | `in-memory-kms://my-password-key-id` |
| Azure | azure-kms\:// | `azure-kms://://.vault.azure.net/keys//` | `azure-kms://https://my-key-vault.vault.azure.net/keys/conduktor-gateway/4ceb7a4d1f3e4738b23bea870ae8745d` |
| AWS | aws-kms\:// | `aws-kms://arn:aws:kms:::key/` | `aws-kms://arn:aws:kms:us-east-1:123456789012:key/password-key-id` |
| GCP | gcp-kms\:// | `gcp-kms://projects//locations//keyRings//cryptoKeys/` | `gcp-kms://projects/my-project/locations/us-east1/keyRings/my-key-ring/cryptoKeys/password-key-id` |
| Fortanix | fortanix-kms\:// | `fortanix-kms://` | `fortanix-kms://password-key-id` |
| Gateway | gateway-kms\:// | `gateway-kms://` (uses master key from KMS above) | `gateway-kms://user-{{record.key}}` |
| Vault (Transit) | vault-kms\:// | `vault-kms://[scheme://]/transit/keys/` | `vault-kms://https://vault:8200/transit/keys/password-key-id` |
| Vault Transform | vault-transform:// | `vault-transform://` | `vault-transform://password-role` |
| Test tokenization | test-tokenization:// | `test-tokenization://` | `test-tokenization://password-role` |
Vault appears twice because encryption and tokenization use two different HashiCorp modules. `vault-kms://` uses the Transit Engine and encrypts. `vault-transform://` uses the Transform Secrets Engine, available in Vault Enterprise only, and tokenizes. Having one doesn't give you the other.
In-memory and test-tokenization modes are for testing and development purposes only. Test tokenization also requires `GATEWAY_FEATURE_FLAGS_TEST_TOKENIZATION` to be set to `TRUE`
If not specified, vault KMS scheme defaults to `https`. This means that `vault-kms://https://vault:8200/transit/keys/password-key-id` and `vault-kms://vault:8200/transit/keys/password-key-id` are identical.
Keys are strings that start with a letter followed by a combination of letters, underscores (\_), hyphens (-) and numbers. Special characters are not allowed. Keys also work with the Mustache pattern described above.
#### Choose your KMS provider
| Provider | When to use | Key consideration |
| ------------------------------------- | ------------------------------------------------- | ----------------------------------------------------------------------------- |
| [In-memory](#in-memory-kms) | Local development and testing only | Keys don't persist - data becomes unreadable after Gateway restart |
| [Gateway](#gateway-kms) | Crypto shredding with high-volume per-record keys | Stores encrypted keys locally, reducing KMS costs for millions of unique keys |
| [AWS KMS](#aws-kms) | AWS infrastructure | Native IAM integration, no credential management needed |
| [Azure Key Vault](#azure-kms) | Azure infrastructure | Managed identity support, integrates with Azure services |
| [Vault KMS](#vault-kms) | Multi-cloud or on-premises | Most flexible authentication (11 methods), works anywhere |
| [GCP KMS](#google-cloud-platform-kms) | Google Cloud infrastructure | Service account integration, follows GCP security model |
| [Fortanix KMS](#fortanix-kms) | FIPS 140-2 compliance required | Hardware security module (HSM) backed, meets strict regulatory requirements |
#### In-memory KMS
This should not be used on production data.
Keys in in-memory KMS are not persisted, this means that if you do one of the following, you won't be able to decrypt old records, losing the data.
* Use a Gateway cluster with more than a single node or
* restart Gateway or
* change the Interceptor configuration
#### Gateway KMS
This KMS type is effectively a delegated storage model and is designed to support encryption use cases which generate unique secret Ids per record or even field (typically via the Mustache template support for a secret Id). This technique is used in crypto-shredding type scenarios e.g. encrypting records per user with their own key.
It provides the option to leverage your KMS for security via a single master key, but efficiently and securely store many per-record level encryption keys (DEKs) in the Gateway managed store. For some architectures this can provide performance and cost savings for encryption use cases which generate a high volume of secret key Ids.
| Key | Type | Description |
| ------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `masterKeyId` | String | The master key secret Id used to encrypt any keys stored in the Gateway managed storage. This is in the same format as the `keySecretId` that's used for encryption and the valid values are the same. |
| `maxKeys` | Number | The maximum number of secret Id references to be cached in memory for re-use. To avoid creating new encryption keys (DEKs), this needs to be larger than the total number of expected secret Ids. By default, it's the same as `maxKeys` in cache config or 1000,000, if `maxKeys` isn't set. |
The `masterKeyId` is used to secure every key for this configuration, stored by Gateway. [Find out more about the secret key formats](#kms-integration). You have to also supply a valid configuration for the KMS type referenced by the master key so this can be used.
If this key is dropped from the backing KMS, then all keys stored by Gateway for that master key will become unreadable.
**Gateway KMS encryption example**
Here's a sample configuration for the Gateway KMS using a Vault-based master key:
```json theme={null}
"kmsConfig": {
"gateway": {
"masterKeyId": "vault-kms://vault:8200/transit/keys/applicants-1-master-key".
"maxKeys" : 10000000
},
"vault": {
"uri": "https://vault:8200",
"token": "my-vault-token",
"trustStore": {
"trustStorePath": "/security/truststore.jks"
}
}
}
```
This can then be used to encrypt a field using `gateway-kms://` as the secret key type:
```json theme={null}
"recordValue": {
"fields": [
{
"fieldName": "name",
"keySecretId": "gateway-kms://fieldKeySecret-name-{{record.key}}"
}
]
}
```
When processing a record for the first time using this configuration, Gateway will:
1. generate a DEK to encrypt the field data,
2. turn it into an EDEK by encrypting with the `masterKeyId` secret from vault and
3. store the EDEK in Gateway storage.
If a record key was `123456`, the associated EDEK would be stored on a kafka record with the following key:
```json theme={null}
{"algorithm":"AES128_GCM","keyId":"gateway-kms://fieldKeySecret-name-123456","uuid":""}
```
Multiple records produced against this config would cause multiple EDEKs to be saved in the Gateway storage (due to the `{{record.key}}` template giving a unique key for each Kafka record key).
If there are multiple Gateway nodes running, it's also possible for multiple DEKs/EDEKs to be generated for the same record key. Two nodes processing different records with the same record key at the same time could both assume they were generating a DEK/EDEK for the first time. In this scenario, there would be two EDEKs in the Gateway storage with the same `keyId` but they would each have a different `UUID`.
For example:
```json theme={null}
{"algorithm":"AES128_GCM","keyId":"gateway-kms://fieldKeySecret-name-123456","uuid":"2cd8125a-b55f-4214-a528-be3c9b47519b"}
{"algorithm":"AES128_GCM","keyId":"gateway-kms://fieldKeySecret-name-123456","uuid":"d8fcccf3-8480-4634-879a-48deed4e0e72"}
```
Nonetheless, there will **only ever be one master key stored in the vault KMS**, which is used to encrypt every DEK.
This feature provides flexibility for your KMS storage and key management setups - and is particularly useful for high volume crypto shredding.
**Decryption using Gateway KMS**
When using the `gateway-kms` secret key Id type, the decryption configuration used to decrypt the data has to also specify the `masterKeyId`, so that it can securely decrypt the keys stored in the local Gateway storage.
Here's a sample setup:
```json theme={null}
"config": {
"topic": "secure-topic",
"kmsConfig": {
"gateway": {
"masterKeyId": "vault-kms://vault:8200/transit/keys/secure-topic-master-key"
},
"vault": {
"uri": "https://vault:8200",
"token": "my-token-for-vault",
"trustStore": {
"trustStorePath": "/security/truststore.jks"
}
}
}
}
```
**Crypto shredding**
When using the `gateway-kms` secret key Id type, you can efficiently crypto shred EDEKs in the Gateway storage, so that anyone using the decryption plugin will immediately lose access to the associated encrypted data.
To do this, scan the Gateway storage Kafka topic (by default, `_conduktor_gateway_encryption_keys`) for every message matching the associated qualified secret Id.
For example, a qualified secretId of `gateway-kms://fieldKeySecret-name-123456` might have the following keys:
```json theme={null}
{"algorithm":"AES128_GCM","keyId":"gateway-kms://fieldKeySecret-name-123456","uuid":"2cd8125a-b55f-4214-a528-be3c9b47519b"}
{"algorithm":"AES128_GCM","keyId":"gateway-kms://fieldKeySecret-name-123456","uuid":"d8fcccf3-8480-4634-879a-48deed4e0e72"}
```
Publishing a message for each of these keys back to the same topic with a value of `null` (i.e. a tombstone) will effectively perform Crypto Shredding.
This process **won't prevent the creation of new keys** if new messages are sent using the same record key; it only ensures that messages using the crypto shredded keys remain unrecoverable.
#### AWS KMS
To set your AWS KMS, include this section in your Interceptor config, below `aws`.
You can use one of these two authentication methods:
* basic authentication or
* session.
Make sure to follow the right method and provided the correct properties. Omit both and Gateway falls back to the AWS SDK's default credentials provider.
**Naming keys through an external connection.** An [AWS KMS external connection](/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/external-connections#aws-kms) carries the region and account, so each `keyId` is a bare key name rather than an ARN. Gateway builds the ARN for you:
```
arn::kms:::key/
```
* **The partition comes from the region**, so GovCloud, China and the ISO regions work without naming one.
* **An `alias/` id is used as the ARN resource as-is**, so it becomes `…::alias/` rather than gaining a `key/` prefix.
* **Anything else is rejected** when you create or update the Interceptor, with `AWS keyId '…' must be a key id or 'alias/