Conduktor release notes

Console 1.32.1​

Release date: 2025-03-24

Fixes​

• Fixed dependencies vulnerable to the following CVEs:
  • CVE-2025-22228
  • CVE-2025-30204
• Fixed an issue where web browsers would try to autofill Kafka Connect configuration form fields with saved passwords

Gateway 3.7.0​

Release date: 2025-03-21

Breaking changes​

New backing topic required for Gateway​

The local KMS feature introduced in this release requires a new backing topic to store the keys.

When you upgrade to Gateway 3.7.0, a new topic _conduktor_gateway_encryption_keys will be created.

To change this default topic name, use the GATEWAY_ENCRYPTION_KEYS_TOPIC variable.

Find out more about environment variables.

Separator for super users​

Super users in Gateway (specified in the GATEWAY_SUPER_USERS environment variable) are now separated by a semicolon ; instead of a comma ,.

This change is to allow super users identified with mTLS using their full DN form (CN=writeuser,OU=Unknown,O=Unknown,L=Unknown,ST=Unknown,C=Unknown), and makes Gateway aligned with the Kafka configuration.

Deprecating V1 APIs​

V1 APIs are now deprecated in favor of the V2 APIs introduced in Gateway 3.3.0 in September 2024.
If you are using the Conduktor CLI to operate the Gateway, you are not impacted. Check the following link to understand which APIs are deprecated: Gateway API Doc. We plan to remove the V1 APIs from the Gateway in three releases time, in Gateway 3.10.0.
If you are using the V1 APIs, please migrate to the V2 APIs as soon as possible.
If you need support with this migration, please let us know.

Preview feature: introducing cost-effective Crypto Shredding with Gateway KMS​

This release introduces a a preview feature that significantly reduces the cost and complexity of implementing crypto shredding at scale. The new 'gateway' KMS type allows you to manage granular encryption keys for individual users or records without the prohibitive costs of storing each key in AWS KMS (which costs approximately $1 per key).

With this feature, you can maintain regulatory compliance and honor user deletion requests more efficiently by:

1. Storing only a single master key in your external KMS
2. Securely managing thousands of individual encryption keys in Gateway's internal key store
3. Deleting specific user keys when needed, rendering their data permanently inaccessible

This approach is particularly valuable for organizations that need to implement crypto shredding across large user bases or high-volume data sets, offering both substantial cost savings and improved performance compared to managing individual keys directly in AWS KMS.

The keys stored by Gateway are all encrypted themselves via a configured master key externally held in your KMS - ensuring they remain secure and useless without access to the external KMS.

Find out how to configure the Gateway KMS.

Support for AWS Glue Schema Registry​

This release extends the support in Gateway for schema registries to include AWS Glue schema registry. The default choice is Confluent like schema registries, and is backwards compatible with previous gateway configurations. For AWS Glue schema registry, different settings are required to connect, and this is covered in the plugin documentation.

Feature changes​

• Added support for . in the name of the Virtual Cluster APIs
• More detailed errors unrelated to interceptor validation added

Bug fixes​

• Added aws-java-sdk-sts dependency to allow assume role profiles when using AWS KMS
• Added jcl-over-slf4j dependency to see logs from AWS SDK

Console 1.32.0​

Release date: 2025-03-12

• Breaking changes
  • Partner Zones
• Scale
  • Alert history
  • Audit logs
  • Service account labels
  • RBAC-aware menu
• Exchange
  • Introducing Partner Zones UI
• Quality of life improvements
• Fixes
• Known issues

Breaking changes​

Partner Zones​

Improvements to Partner Zones require Gateway 3.6.1 to be deployed with this release of Console. Find out more about Partner Zones and required environment variables.

Scale​

Alert history​

In the alert details page you can now also see the history of an alert's status and notifications which may have failed to send. Find out more about alerts.

Audit logs​

You can now view the new CloudEvents-based audit log events with enhanced filtering capabilities for the new resource and event types, including Conduktor platform triggered events.

See the full list of all the audited events.

While legacy audit log events will stop being captured in this release, existing legacy events will remain accessible through a dedicated page until a future release.

Service account labels​

You can now edit labels on service accounts in plain Kafka clusters through the UI, in addition to the existing CLI & API support.

Support for labels on Aiven and Confluent Cloud service accounts is coming soon.

RBAC-aware menu​

The left-hand menu is now RBAC (Role Based Access Control) aware, dynamically showing/hiding menu items based on users' permissions. Hiding functionality that users don't have access to makes onboarding easier and reduces confusion.

In Console, menu items are shown based on the user's Resource access permissions on individual clusters, while the Settings menu items are shown according to the Service access permissions. Find out more about RBAC.

Exchange​

Exchange is a new Conduktor product that enables you to share data securely with your external partners. Find out more about the Exchange product.

Introducing Partner Zones UI​

Partner Zones enable you to securely share your Kafka streaming data with external partners, without the need to replicate that data into a second, physical Kafka cluster.

In this release we're introducing the option to create Partner Zones using the Console UI in just few steps, including the ability to set traffic control policies.

In upcoming releases we'll be adding further enhancements, such as:

• an ability to edit Partner Zone configurations
• the option to rename shared topics, securing your data even further by ensuring that no internal information is shared

Find out more about Partner Zones.

Quality of life improvements​

• Enabled the confirmation of resource deletion using the Enter key.
• Updated the connector restart button labels and toast messages to accurately reflect their behavior for Confluent Cloud connectors.
• Removed a legacy option to disable monitoring.

Fixes​

• Fixed an issue where editing a schema registry subject would overwrite its compatibility mode with the global compatibility setting.
• Fixed an issue where creating an ACL for a service account with a duplicate name could override the existing ACL.
• Fixed an issue where you could not remove the last ACL and Save in the Service Account UI.
• Fixed an issue where the Kafka Connect failed task heatmap didn't display data for days in 2025.
• Fixed an issue where the CLI would report incorrect actions taken (although the correct actions were shown when the --dry-run flag was used).
• When Azure Active Directory is used as an LDAP server, the userPrincipalName field can now be set as the field containing the email address.
• Fixed an issue where very large numbers would show rounded in the details view of a topic message (e.g. 7777705807840271771 would display as 7777705807840271000).
• Fixed the UI issue where the option to reset a consumer group offset would disappear off the screen, if the partition count was too large.

Known issues​

In the Topic Consume view, equality filters (==) on JSON number fields isn't working correctly when the number exceeds JavaScript's safe integer limit of 2^53-1. Note that while range operators (>, <, >=, <=) still work with large numbers, there's currently no workaround for exact equality filtering. We'll address this in a future release.

Conduktor CLI 0.5.0​

Release date: 2025-03-10

Changes​

• Included Gateway resources in get all
• Added cause to ApiError responses
• Fixed apply template comment in YAML file
• Added option to edit and apply immediately to template command
• Standardized flag descriptions

Fixes​

• Fixed verbose mode in single client configuration

Find out more.

Gateway 3.6.1​

Release date: 2025-03-05

New features​

• New metric gateway.apiKeys.throttle_ms : sets the throttling time in Kafka responses per apiKey in milliseconds
• Updated existing metric gateway.apiKeys.latency.request_response : sets the latency to process a request and generate a response for each API key
  • It now tracks latency for all verbs (eg CONNECTION) not just FETCH/PRODUCE

Feature changes​

• Changes to Limit Commit Offset Plugin:
  • accuracy of rate limiting has been improved
  • action/throttleTimeInMs properties did not work correctly and are now ignored
• Changes to cluster ACLs:
  • when creating cluster ACLs using a programmatic API, only allow kafka-cluster for the name part of the resource. This makes Gateway consistent with Kafka.

Fixes​

• Fixed a problem with the Create Topic Policy plugin which would not apply overrides to default configurations from the underlying Kafka setup.
• Fixed a problem with CreateTopics ACLs in Gateway which previously also required the Create cluster permission enabled.
• Addressed a problem with Non Delegated SASL/PLAIN token credentials, where Gateway would continue to work after service account has been deleted. To enable this feature set the environment variable GATEWAY_USER_POOL_SERVICE_ACCOUNT_REQUIRED to true (it currently defaults to false).
• Fixed a problem in AddPartitionsToTxnRequest where ACLs on transactionIds in new location were not being checked when Kafka API version was >= 4.
• An un-authorized idempotent producer will now throw a ClusterAuthorizationException instead of a TransactionalIdAuthorizationException, making Conduktor Gateway consistent with Kafka.

Console 1.31.2​

Release date: 2025-03-03

Fixes​

• Fixed prefixed ACLs not displaying correctly in the Service Account UI
• Fixed an issue where Service Accounts with no ACL were incorrectly shown in the UI
• Fixed an issue allowing creation of Service Accounts without ACLs

Known Issues​

• When removing ACLs in the Service Account UI, you cannot remove the last ACL
• As a workaround, you can remove that last ACL, then create a new ACL against a resource name that does not exist.
• We will address this issue in the next release

Console 1.31.1​

Release date: 2025-02-18

Console metrics performance and configuration​

To address issues related to monitoring graph timeouts and OutOfMemory issues when Console is connected with large Kafka clusters, we've introduced a change to how metrics are collected in Console.

This feature is currently optional but will be enabled by default in the next release.

If you're experiencing graph timeouts or OutOfMemory issues, upgrade to 1.31.1 and configure the following additional environment variables:

CDK_MONITORING_ENABLENONAGGREGATEDMETRICS: false
CDK_MONITORING_USEAGGREGATEDMETRICS: true

This configuration will disable the collection of obsolete granular metrics and use the new aggregated metrics in the Console graphs.

See metric configuration for details.

Fixes​

• Fixed dependencies vulnerable to the following CVEs:
  • CVE-2024-57699
  • CVE-2025-24970
  • CVE-2024-45337
  • CVE-2024-45338
• Resolved an issue related to SSL checks between the Console and Cortex where bad certificates caused communication issues
• Fixed an issue with user email addresses containing a ' character that blocked database migrations
• Reduced the memory consumption and improved the performance of metrics under heavy load that prevented them from displaying
• Partner Zones: configuration updates are now applied correctly
• Partner Zones: partners can now utilize consumer groups

Gateway 3.6.0​

Release date: 2025-02-12

Features​

Kafka Cluster connection management​

This release includes a set of enhancements to how Gateway manages connectivity to a Kafka Cluster. This provides greater stability and flexibility for how Gateway can be configured with the Kafka Clusters it's connected to, and is a precursor change for future product releases.

Encryption improvements​

Encryption Secret Id Mustache Templates​

The encryption now allows multiple mustache substitutions in a key secret Id configuration. Previously, only a single substitution was supported. E.g. This is now allowed:

"keySecretId": "vault-kms://my-vault:8200/transit/keys/{{record.key}}-{{record.header.someHeader}}"

Decryption failure modes​

This release adds a new optional configuration to the decryption plugin to allow different modes of handling errors. There are two supported modes:

• errorPolicy: "return_encrypted" Previous - if there's an error during decryption, then the encrypted data is returned.
• errorPolicy: "fail_fetch" New - if there's an error during decryption, then the fetch that was reading the data is failed and the client will receive an error.

In both cases, we have enhanced the logging, so issues during decryption are more fully reported.

Schema Registry access improvements​

Read only Schema Registry access​

Some of our Gateway plugins will deserialize and re-serialize messages in order to perform their functions. A side effect of this is that the serializer code would unnecessarily require write access to the Schema Registry. While there was no situation where Gateway would actually cause any updated or additional schema to appear, - we've altered the Schema Registry access to be read only. This avoids having to unnecessarily assign write permission for our Gateway Schema Registry connections if you're using ACLs on your Schema Registry.

Plugin validation of Schema Registry access​

All Gateway plugins which access the Schema Registry will now validate that the configuration for the Schema Registry is correct when it's added or updated in Gateway. Previously, this behavior was inconsistent and a few of our plugins would only detect incorrect configurations when they processed a message rather than when they were setup.

Quality of life improvements​

• Added a new CLI command conduktor run generateServiceAccountToken to generate the JWT for local service accounts. Update your CLI to version 0.4.0 or higher.

Fixes​

• Fix: CreateTopicPolicy and AlterTopicPolicy overrides. There were some edge cases where the desired overrides from the plugin config would not be applied. These plugins now behave consistently in all situations.
• Improvement: Removed some verbose logs and updated logging to be clearer. A general set of improvements has been made to Gateway logging, making some errors clearer and reducing repetition.

Conduktor CLI 0.4.0​

Release date: 2025-02-07

Changes​

• Environment variable references can now be passed to Gateway or Console, allowing you to store references to secret variables used by the host within your configuration.
• Partner Zones are now available, allowing you to securely share your streaming Kafka data with external partners without the need to replicate the data.
• More informative error responses in certain situations
• Console API schema updated
• Added run
• Schema code reorg
• Ops 630 pass external environment variable reference
• Introduced dev mode for hidden command
• Panic replaced with graceful exit
• Included Partner Zones Gateway API changes

Fixes​

• buildAlias duplication fixed
• Fixed ServiceAccount check when defining commands
• Release Action fixed
• Various doc fixes
• Fixed duplicate printout statements

Find out more.

Console 1.31.0​

Release date: 2025-02-05

• Breaking changes
  • Removed V1 Alerts
  • Changes to V2 Alerts
  • Id of Certificates
• Scale
  • Enhanced Alerting with Added Webhooks Support
  • API / CLI support for Service Accounts
  • Labels support for Service Accounts
  • Self-Service support for Application Managed Service Accounts
  • Application Group permissions now available on Users Permissions page
• Exchange
  • Introducing Partner Zones for Third-Party Data Sharing
• Quality of Life improvements
• Fixes
• Known issues

Breaking changes​

Removed V1 Alerts​

Original alerts created in the Monitoring/Alerts section are no longer available.

Changes to V2 Alerts​

V2 Alerts, that can be created since Console 1.28 on the dedicated resource page (Topics, Brokers, etc.) are still available and active, but have been migrated with the following rules:

• Alerts have been automatically configured with the previously globally configured channel (Teams or Slack).
• Alerts have been assigned to the individual who created them.

Read below for more information about the new alerting functionality.

Id of Certificates​

The Id of certificates in the public/v1/certificates API endpoints were modified to represent the fingerprint of the certificate. It brings a more stable way to identify certificates in audit log and prevent multiple uploads of the same certificate.

Scale​

Enhanced Alerting with Added Webhooks Support​

We have made significant improvements to the alerting system in Console.
Here are some of the changes:

• Alerts are now owned by individuals, groups, or applications
• We added Webhook destination to alerts notifications
• Destinations are now configurable per-alert
• API / CLI support for Alerts is now available

apiVersion: console/v3
kind: Alert
metadata:
  name: messages-in-dead-letter-queue
  group: support-team
spec:
  cluster: my-dev-cluster
  type: TopicAlert
  topicName: wikipedia-parsed-DLQ
  metric: MessageCount
  operator: GreaterThan
  threshold: 0
  destination:
    type: Slack
    channel: "alerts-p1"

Alert creation workflow has been updated to allow you to configure the alert destination and ownership in the UI.

Read the alerting section of our documentation for more information about the new alert functionality.

API / CLI support for Service Accounts​

We have added support for Service Accounts in the API and CLI.
Declaring ServiceAccount resource lets you manage the ACLs of a service account in Kafka.
At the moment we only support Kafka ACLs (calls to Kafka APIs) but we plan to add support for Aiven ACLs in ServiceAccount resource in the future.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  cluster: shadow-it
  name: clickstream-sa
spec:
  authorization:
    type: KAFKA_ACL
    acls:
      - type: TOPIC
        name: click.event-stream.avro
        patternType: PREFIXED
        operations:
          - Write
      - type: CLUSTER
        name: kafka-cluster
        patternType: LITERAL
        operations:
          - DescribeConfigs
      - type: CONSUMER_GROUP
        name: cg-name
        patternType: LITERAL
        operations:
          - Read

Labels support for Service Accounts​

We have added support for labels in the ServiceAccount resource.
For now you can only edit labels through ServiceAccount resource in the API and CLI.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  cluster: shadow-it
  name: clickstream-sa
  labels:
    domain: payment
    region: EMEA
    application: clickstream
spec:
  ...

The labels are used to filter the Service Accounts in the UI. Editing labels in the UI will be available in the next release.

Self-Service support for Application Managed Service Accounts​

We have added a new mode for ApplicationInstance that allows Application Teams to have full control over their Service Accounts.
This mode can be enabled in the ApplicationInstance with the following flag spec.applicationManagedServiceAccount set to true.
When enabled, Self-Service will not synchronize the Service Account with the ApplicationInstance and will let the Application Team manage the Service Account directly. Application Managed Service Accounts can be declared in the API and CLI using the Application API Key.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  appInstance: "clickstream-app-dev" # Mandatory to link the Service Account to the ApplicationInstance
  cluster: shadow-it
  name: clickstream-sa
spec:
  authorization:
    type: KAFKA_ACL
    acls:
...

Application Group permissions now available on Users Permissions page​

The users permissions page has been updated to show the permissions inherited when they belong to an ApplicationGroup.

Exchange​

Exchange is a new Conduktor Product aimed at helping you share your data securely with your external partners.
Check the associated Exchange Product page for more information.

Introducing Partner Zones for Third-Party Data Sharing​

Partner Zones enable you to securely share your streaming data with external partners, without needing to replicate the data into a second, physical Kafka cluster.

In the upcoming releases, we will be adding the following:

• Dedicated pages that allows you to manage Partner Zones completely from the UI
• Support for Traffic Control Policies to limit the amount of data that can be consumed or produced by your partners
• Topic renaming capability to avoid leaking internal topic names to your partners

For more information, check out the Partner Zones documentation.

Quality of Life improvements​

• Added a "Groups" tab in the Application page which shows all of the Application Groups created via Self-service
• Improved the license plan page to show the start and end date of the license, as well as the packages included in the license
• Added the remaining days left in the sidebar when the license is expiring in less than 30 days
• Improved how a connector's configuration is displayed in the raw JSON view by sorting the properties alphabetically

Fixes​

• Fixed several issues Confluent Cloud Managed Connectors
  • Fixed Pause/Resume connector
  • Fixed Connector and Task Restart
  • Fixed Connector Status (Running, Paused, etc.), previously displayed as "Unknown"
• Fixed a permission check issue when adding partitions to a topic
• Improved the serialization of String and com.fasterxml.jackson.databind.JsonNode types returned by custom deserializers
• Fixed an issue parsing masked data when choosing the String format on data that cannot be parsed as JSON
• Added topics ending with -subscription-registration-topic and -subscription-response-topic to the Kafka Stream filter
• Fixed the edition of ownership mode of application instances
• Fixed the form for saving producer templates
• Fixed the navigation to go back to the home page of connectors when switching clusters

Known issues​

• We are aware of more inconsistencies with Confluent Cloud Managed Connector support in Console. We are working on it.
  • Task status is not always correctly displayed
  • Various UI responsiveness issues

Gateway 3.5.2​

Release date: 2025-01-17

Features​

• Add support for sourcing environment variables from a file

Fixes​

• Add automatic refresh of HashiCorp Vault authentication tokens

Conduktor CLI 0.3.4​

Release date: 2025-01-10

Changes​

• Added support of query params for parent
• Added template

Fixes​

• Fixed the issue when deleting virtual clusters does nothing

Gateway 3.5.1​

Release date: 2025-01-08

Features​

• Added support for Hashicorp Vault Enterprise namespaces

Conduktor CLI 0.3.3​

Release date: 2024-12-16

Changes​

• Updated default kind for 1.30
• Added basic auth for Gateway API support

Fixes​

• Fixed default vCluster set to null for delete interceptors

Console 1.30.0​

Release date: 2024-12-16

• Features ✨
  • RBAC support for Conduktor SQL
  • Add support for multi-hosts database configuration
  • Delegating authentication to an identity provider
  • More Audit Log CloudEvents into Kafka
• Quality of Life improvements
• Fixes 🔨

Features ✨​

RBAC support for Conduktor SQL​

In a previous release, we introduced Conduktor SQL. It was restricted to Admins because it did not apply any permission model.

This new update brings full RBAC support on Conduktor SQL:

• Users & Groups can only see and query the tables for which they have an associated Topic permission in Console
• Data masking policies are applied (with limitations)

You can now bring SQL to all users within your organization.

For more detailed information, check out the SQL security section.

Add support for multi-hosts database configuration​

You can now setup Console's backing database for high availability(HA). If you have a PostgreSQL HA setup with multiple hosts, you can now configure a Console to JDBC connection to the database using a list of hosts.

CDK_DATABASE_URL: jdbc:postgresql://user:password@host1:5432,host2:5433/console_database
CDK_KAFKASQL_DATABASE_URL: jdbc:postgresql://user:password@host1:5432,host2:5433/kafka_sql_database

For more information, check out the Multi-host configuration section in the Database configuration documentation.

Delegating authentication to an identity provider​

Console can now be configured to accept a JWT token from an external identity provider.
It allows you to directly use your identity provider for managing access to Console.
A common use case of this feature is to delegate authentication to your API gateway.

For the full configuration details, check out the documentation.

More Audit Log CloudEvents into Kafka​

We have made more events available for the Audit Log Publisher:

• Kafka.Subject.ChangeCompatibility
• Kafka.Topic.Browse
• Kafka.Topic.ProduceRecord
• Kafka.Topic.SqlQuery
• Kafka.Connector.Restart
• Kafka.Connector.Pause
• Kafka.Connector.Resume
• Kafka.Connector.RestartTask
• Kafka.Connector.AutoRestartActivate
• Kafka.Connector.AutoRestartStop

A full list of all the exported audit log event types is published on the Audit Log page.

Quality of Life improvements ✨​

• Alert lists in the resource pages have been updated to show the metric and condition, the alert state and a new column "Last Triggered"
• Chargeback data can now be exported into a CSV file to enable easier integration with existing organization cost management data.
• The User permission page provides a clearer distinction between inherited and user-specific permissions.
• Topic policy validation errors message are easier to read when using the CLI
• Added support for Array and Boolean types in Conduktor SQL
• Added Kafka Key column and other metadata in Conduktor SQL Topics (Full list)

Fixes 🔨​

• Fixed a pagination issue in the SQL Indexed Topics view
• Fixed several instances where the CLI would not report the expected state change (Updated vs. Not Changed) on apply

Gateway 3.5.0​

Release date: 2024-12-16

Breaking changes​

Breaking change: Local Users 💣​

From this release, we will now strictly enforce that the username and the token matches in requests made to the Gateway where local service accounts are used. This will help reduce inconsistencies and avoid unexpected behaviors. If they do not match, requests will fail to authenticate.

Breaking change: Default SNI Host Separator​

In this release we have changed the default value for the separator used in the SNI routing configuration from a period . to a dash -. This is in order to better allow the use of wild card certificates when certificates are in use.

The format of the SNI routing host names is now as below:

 <host_prefix><cluster_id><broker_id>-<advertised_host>

The previous behaviour of Gateway can be configured by simply adding this to your configuration:

GATEWAY_SNI_HOST_SEPARATOR=.

For more information on SNI routing, see its documentation.

Features ✨​

Use of In-memory KMS for Encryption​

Gateway has always supported the use of an in memory KMS for encryption in order to provide an easy-to-use setting for testing and developing your encryption config. This mode is not however meant for production use as the state of the KMS is lost when Gateway restarts, rendering any data encrypted with it unrecoverable.

Before this release, the in memory mode was the default setting and would be used as a fallback if no valid external KMS was detected in the encyrption setup.

From this release, you must now explicitly opt-in to the in-memory mode for encryption using the prefix:

in-memory-kms://

If this, or any other valid KMS identifier, is not present the encryption plugin will now fail. This change is a precaution to prevent accidental misconfigurations resulting in the use of in memory mode and subsequent data loss.

See the encryption configuration docs for more information.

License Expiry​

We have altered the behaviour of the Gateway when your license expires to provide a better experience. The behaviour is now as below:

• We have added new metric gateway.license.remaining_days which you can monitor to track the time left on your license
• If the Gateway is currently running, do not automatically exit on license expiry. Rather, Gateway will now log a warning every hour that your license is expired:

License has expired! You need to add a valid license to continue using Conduktor Gateway. Checkout our documentation if unsure how to set the license

• These warnings will start 1 week before expiry occurs as a notification, in the format:

License will expire in less than {N} day(s)! You need to renew your license to continue using Conduktor Gateway

• Finally, we now check your license earlier in the bootstrap sequence for Gateway, so it will fail fast with a clear message when your license is expired.

The key change here is that if your license does expire, Gateway will not exit automatically anymore. It will continue running, logging warnings. Should you restart the Gateway in this state, it will then fail to start up - but there is no automatic shutdown.

Quality of Life Improvements ✨​

• Support for Kafka Clients up to 3.9
• Improved compatibility and logging for dealing with kafka-client versions and version negotiations
• Added support for multiple authentication mechanisms against Vault (AppRole, LDAP, ...)
• Introduced a new configuration enableAuditLogOnError (default: true) which enhances the errors which are logged when encryption/decryption fails
• Improved error logging for expired tokens on authentication, to replace large stack traces with concise information
• Performance improvements for TLS handshakes, in particular to prevent repeated failed attempts overloading the gateway
• Several improvements to data quality and encryption config validation to provide better error reporting and feedback in the case of problems

General fixes 🔨​

• Fixed an issue in ACL handling which caused an error if no topics were passed for an offset fetch request (being the case where the caller wants to retrieve offsets for all topics).
• Fixed an issue in Virtual Clusters which in some cases meant the ACLs for the physical Kafka clusters where exposed in error.
• Fixed an issue in the regular expression application in the data quality and SQL plugins, where .* would not always match the entire value for a field
• Fix an issue when creating both a service account and a service account group through the cli to ensure the order of operations is always correct, preventing intermittent failures in this case.

Known issues​

• We are aware of an issue with kcat when the new environment variable GATEWAY_MIN_BROKERID is not aligned with the first BrokerId of your Kafka cluster
  • As a workaround, you can either define GATEWAY_MIN_BROKERID to your first Kafka BrokerId or use kcat with the -E flag

Console 1.29.2​

Release date: 2024-12-09

Fixes 🔨​

• Fixes a bug where SSO and LDAP sign ups were being rejected for users with upper-case letters in their email

Console 1.29.1​

Release date: 2024-11-28

Fixes 🔨​

• Fixed an issue where configuring Google OIDC without a scope was throwing an Exception.

Console 1.29.0​

Release date: 2024-11-25

• Breaking Changes 💣
  • Changes to Conduktor.io Labels
  • Local Users Password policy update
• Features ✨
  • Conduktor Chargeback
  • Console Homepage
  • Consumer Group pages overhaul
  • Self-Service Topic Catalog visibility
  • Self-Service New Topic Policy Allowed Keys
  • More Audit Log CloudEvents into Kafka
  • Expanded Terraform Provider
• Quality of Life improvements
• Fixes 🔨

Breaking Changes 💣​

Changes to Conduktor.io Labels​

We have moved the conduktor.io labels previously available on Connector and Topic resources to new locations.

We recognize this change breaches the API contract and have carefully considered its implications. We remain committed to minimizing breaking changes and will continue to do our best to ensure that such changes are rare and well-justified.

As we expand the number of Conduktor-related features, this separation reduces the risk of conflicts, simplifies resource management, and provides flexibility for future improvements:

• Labels used for sorting and filtering throughout the product
• Conduktor-specific annotations used to drive behaviors on the resources

Topic Resource

• metadata.labels.'conduktor.io/description' → metadata.description
• metadata.labels.'conduktor.io/description.editable' → metadata.descriptionIsEditable

Connector Resource

• metadata.labels.'conduktor.io/auto-restart-enabled' → metadata.autoRestart.enabled
• metadata.labels.'conduktor.io/auto-restart-frequency' → metadata.autoRestart.frequencySeconds

Their associated values have been automatically migrated under the new names.

Important Note for CLI Users​

Applying YAML files with old conduktor.io labels will fail in Conduktor Console 1.29. Be sure to update your YAML files to reflect the new labels.

Example error for outdated YAML:

$ conduktor apply -f topic.yaml
Could not apply resource Topic/click.event-stream.avro: Invalid value for: body (Couldn't decode key. at 'metadata.labels.conduktor.io/description')

Local Users Password policy update​

Passwords for console Local Users configured through YAML and environment variables must comply with a new password policy. This change enforces the following password requirements:

• At least 8 characters in length
• Includes at least 1 uppercase letter, 1 lowercase letter, 1 number, and 1 special symbol

Passwords set in existing installations that do not meet these requirements will prevent Console from starting, throwing a startup error in the logs like this:

2024-11-21T14:25:47,434Z [console] ERROR zio-slf4j-logger - zio.Config$Error$InvalidData: (Invalid data at admin: Password must contain at least 8 characters including 1 uppercase letter, 1 lowercase letter, 1 number and 1 special symbol)

Local Users previously created with the UI are not impacted.
Update the passwords in your YAML or environment variables to meet the new policy before upgrading.

Features ✨​

Conduktor Chargeback: Track and Allocate Costs & Resource Consumption​

Conduktor Chargeback allows organizations to track and allocate costs & usage associated with Kafka resources to different teams or departments based on their data consumption and processing, facilitating cost accountability and management.

Check the dedicated Quickstart to get started with Chargeback.

Console Homepage​

The cluster homepage have been redesigned to present you with the most useful information in one single view:

• The health of your Kafka Cluster with a few key metrics and graphs
• The state of Console Indexing modules for this Kafka Cluster
• Quick access to your most recently viewed resources

Consumer Group pages overhaul​

Consumer group details page is now organized in a way that helps understand the status of your Consumer Group more easily:

• Topics tab shows the Consumer Group info grouped by its subscribed Topics
• Members tab shows the Consumer Group info grouped by its active members

Both Topics and Members lists can be explored further down to individual member/topic-partition assignments.

On top of that graphs are now directly available in the resource page for Lag and Time Lag, as well as a dedicated tab to manage Alerts.

Self-Service Topic Catalog visibility​

You can now choose which Topics should be visible in the Topic Catalog by annotating their YAML.

---
apiVersion: kafka/v2
kind: Topic
metadata:
  cluster: shadow-it
  name: click.event-stream.avro
  catalogVisibility: PUBLIC # or PRIVATE
spec:
  ...

It is also possible to change the default Topic Catalog visibility of all Topics of an Application Instance directly
Check the associated documentation.

Self-Service New Topic Policy Allowed Keys​

We have added a new constraint AllowedKeys to our Self-Service Topic Policy that restricts the properties that can be configured on a Topic.

---
# Limits the Topic spec.configs to only have retention.ms and cleanup.policy keys
apiVersion: self-service/v1
kind: TopicPolicy
metadata:
  name: "generic-dev-topic"
spec:
  policies:
    spec.configs:
      constraint: AllowedKeys
      keys:
        - retention.ms
        - cleanup.policy

This works in conjunction with existing constraints and ensures your Application Teams will only define properties that are allowed by the Central Team.
Read more about our Topic Policy constraints.

More Audit Log CloudEvents into Kafka​

We have made more events available for the Audit Log Publisher:

• IAM.User.Logout
• IAM.User.Login
• Kafka.ConsumerGroup.Duplicate
• Kafka.ConsumerGroup.Delete
• Kafka.ConsumerGroup.Update ( when we reset the offset of the consumer group )

A full list of all the exported audit log event types is published on the Audit Log page.

Expanded Terraform Provider: Kafka Cluster, Schema Registry, Kafka Connect​

We've expanded the scope of our Terraform provider, you can now create additional resources: Kafka cluster with schema registry, and Kafka connect clusters using Terraform. With this version also comes some additional small fixes as requested by the community, see the dedicated provider releases page for the full list.

All examples are available in our provider repo such as the below snippet for a Confluent Kafka cluster and schema registry (with mTLS) definition.

resource "conduktor_kafka_cluster_v2" "confluent" {
  name = "confluent-cluster"
  labels = {
    "env" = "staging"
  }
  spec {
    display_name      = "Confluent Cluster"
    bootstrap_servers = "aaa-aaaa.us-west4.gcp.confluent.cloud:9092"
    properties = {
      "sasl.jaas.config"  = "org.apache.kafka.common.security.plain.PlainLoginModule required username='admin' password='admin-secret';"
      "security.protocol" = "SASL_PLAINTEXT"
      "sasl.mechanism"    = "PLAIN"
    }
    icon                         = "kafka"
    ignore_untrusted_certificate = false
    kafka_flavor = {
      type                     = "Confluent"
      key                      = "yourApiKey123456"
      secret                   = "yourApiSecret123456"
      confluent_environment_id = "env-12345"
      confluent_cluster_id     = "lkc-67890"
    }
    schema_registry = {
      type                         = "ConfluentLike"
      url                          = "https://bbb-bbbb.us-west4.gcp.confluent.cloud:8081"
      ignore_untrusted_certificate = false
      security = {
        type              = "SSLAuth"
        key               = <<EOT
-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----
EOT
        certificate_chain = <<EOT
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
EOT
      }
    }
  }
}

Quality of Life improvements​

• Improved the performance of the Automatic deserializer
• Improved the performance of the Schema Registry indexing process
• Added support for Google Cloud Identity group claims
• Added License expiry warning in the UI when there is less than 30 days left

Fixes 🔨​

• Fixed an issue where Custom Deserializers weren't working as expected
• Fixed an issue where the ManageClusters permission wasn't working as expected
• Fixed an issue that prevented creating a KafkaCluster and a Topic on that newly declared KafkaCluster in a single CLI apply command
• Fixed /health/readiness endpoint to return HTTP 503 when the Postgres DB is down
• Fixed an issue where the Message Count wasn't updated to 0 when emptying a topic
• Fixed an issue where the Pause/Resume button wasn't visible when a connector was in the Failed state
• Fixed an issue where the Topic creation failure reason wasn't shown in the UI
  • This helps understand why Topic Creation is rejected (useful for Gateway and Self-Service Topic Policies), and how to modify the topic create request to meet the policy requirements

Conduktor CLI 0.3.2​

Release date: 2024-11-25

Changes​

• Updated Console schema with latest version

Fixes​

• Changed ordering between groups and service accounts in Gateway entities

Gateway 3.4.0​

Release date: 2024-11-15

Upcoming Breaking change: Local Users 💣​

Today, the token as the password for local Gateway service accounts contains all the necessary information. As a result, the SASL username is not used during the authentication phase.

In release 3.5.0, we will strictly enforce that the username and the token matches. This will help reduce inconsistencies and avoid unexpected behavior.

For this release 3.4.0, we'll only raise the following warning in the logs:

2024-08-27T18:15:29 [WARN] - Inconsistency detected for plain authentication. Username applicationA is not consistent with validated token created for application-A. SASL configuration should be changed accordingly.

Features ✨​

• Correct Offsets on Concentrated Topics

Correct Offsets on Concentrated Topics​

Concentrated Topics were reporting the offsets of the underlying backing topic. This caused Lag and Message Count metrics to be invalid.

Correct offsets can now be enabled per ConcentrationRule.

---
kind: ConcentrationRule
metadata:
  name: myapp-concentrated
spec:
  pattern: myapp-.*
  physicalTopics:
    delete: myapp-concentrated
  autoManaged: false
  offsetCorrectness: true

This feature is experimental and comes with a number of limitations that are important to understand beforehand.

General fixes 🔨​

• Fixed an issue impacting live consumption from concentrated topics within Console
• Fixed an issue with upserts in API V2 relating to service accounts (reporting updated when the status should be not changed)
• Fixed an issue related to Kafka 3.7 client support, ensuring topic id's for alias and concentrated topics are distinct from their underlying topics
• Fixed an issue whereby audit logs were not being captured during cluster switching
• Fixed an issue with SQL topics when parsing topic names containing "-"

Known issues​

• We are aware of an issue with kcat when the new environment variable GATEWAY_MIN_BROKERID is not aligned with the first BrokerId of your Kafka cluster
  • As a workaround, you can either define GATEWAY_MIN_BROKERID to your first Kafka BrokerId or use kcat with the -E flag

Console 1.28.0​

Release date: 2024-10-14

• Features ✨
  • Conduktor SQL
  • Monitoring improvements
  • New CLI/API resource Alert
  • Shareable Filters
  • Tags becomes Labels
  • Publish Audit Log CloudEvents into Kafka
  • Logging API
• Quality of Life improvements
• Fixes 🔨

Features ✨​

Conduktor SQL​

Index data from Kafka topics in a database to enable data to be queried from the UI, API or CLI using SQL.

This allows you to troubleshoot, sample, analyze, aggregate and join data through:

• Querying Kafka message data
• Querying Kafka metadata (such as the offset, partition and timestamp)

Read the dedicated guide on configuring SQL.

Query through the UI

Query through the API & CLI

CLI (upgrade to v0.3.1):

conduktor sql 'select * from "kafka-cluster-dev_customer_orders"' -n 2

API:

curl \
 --header "Authorization: $token" \
 --request POST 'localhost:8080/api/public/sql/v1/execute?maxLine=2' \
 --data 'select * from "kafka-cluster-dev_customer_orders"'

Important information regarding SQL

To use the feature there is a dependency on provisioning a new database. As a user, you have the choice of which topics you wish to index. Learn more about this and how to configure SQL using the dedicated guide.

We encourage you to use this feature in non-production environments and give us feedback.

Monitoring improvements​

We are migrating our Monitoring dashboards into their respective resource pages for a more integrated experience.

This migration will happen over the next few releases with our objective to remove the existing, generic Monitoring pages:

• Overview will be refactored into Home page
• Cluster Health dashboards and alerts will move under Brokers page
• Topic monitoring will be integrated with Topics page
• Apps monitoring will be integrated with Consumer Groups pages
• Alerts will be integrated as tabs in all the resource pages, similar to the recent changes for Kafka Connect

For this 1.28.0 release we are migrating the Topic monitoring and Cluster Health pages.

Topic Monitoring​

The 3 existing graphs have been moved to the Topic details. We have also added a new graph to track the number of records in the topic.

• Produce Rate and Consume Rate
• Disk Usage
• Records (new)

Cluster Health​

The charts and alerts are now available under the Brokers page with cleaner graphs.

• Produce Rate and Consume Rate
• Disk Usage
• Partitions Count
• Offline, Under Replicated and Under Min ISR Partitions

We have removed two metrics that were not always calculated correctly since the removal of the JMX integration back in release 1.15 (May 2023).

• Active Controller Count
• Unclean Leader Election

New CLI/API resource Alert​

Alerts can now be created via the API or CLI in addition to the UI.
See below for example config, and check the Alerts documentation for more details.

---
apiVersion: console/v2
kind: Alert
metadata:
  cluster: local-julien
  name: my-alert
spec:
  type: TopicAlert
  topicName: wikipedia-parsed-DLQ
  metric: MessageCount
  operator: GreaterThan
  threshold: 0

Starting today, we recommend you use the new alerts available under Brokers and Topics pages.

Shareable Filters​

Filters in the Topic Consume view are now shareable. This brings a number of benefits:

• Improved collaboration: Share pre-defined views to ensure users are looking at the same subset of data
• Time savings: Speed up troubleshooting and analysis with repeatable filters that share the same or similar criteria
• Consistency and accuracy: Standardized filters across teams and departments reduce the risk of errors or discrepancies that can occur when individuals manually create filters

After you've finished configuring filters on a topic, you now have an option to save the filter either as a Private or an Organization filter.

Anyone can then load Organization filters from the dedicated section.

Tags Become Labels​

With the introduction of the Self-service resource manifests, we brought customers a means to annotate all their resources with labels. Labels are more structured than the existing Conduktor tags, thereby allowing for more precise filtering capabilities, as can be seen in the Topic Catalog.

In this release, we'll perform an automatic migration from Tags to Labels.

Tags written with the naming convention <key>/<value> will automatically be added as similar labels:

• <key>: <value>

If there is a conflict such as; a topic containing tags with the same key, that already has the target label, or is not written with this naming convention, then they will be created with a tag- prefix as follows:

tag-<value>: true

Here's an example of how tags will be migrated into labels:

# Tags defined on topic:
- format/avro
- project/supplychain
- team/delivery
- color/blue
- color/red
- wikipedia
- non-prod

# Resulting topic labels after migration
labels:
  format: avro
  project: supplychain
  team: delivery
  tag-color/blue: true # Because conflict on "color"
  tag-color/red: true # Because conflict on "color"
  tag-wikipedia: true # Because wikipedia is not a key value pair
  tag-non-prod: true # Becuase non-prod is not a key value pair

The Topic list and Topic details page have been modified to use labels instead of tags.

We plan to bring this capability on all resources (Connectors, Service Accounts, Consumer Groups, ...) over the next few releases.
Let us know which resource you would like to see covered first.

Publish AuditLog CloudEvents into Kafka​

It is now possible to publish Console Audit Log events into a Kafka topic directly for any further use you may have for them, such as maintaining your own audit trail in other systems.

The exportable audit log events have more detail compared to the current UI events, providing additional information about the event that has taken place.

The events conform to the CloudEvents specification, a vendor-neutral format that follows the following structure:

{
    "specversion" : "1.0",
    "type" : "com.github.pull_request.opened",
    "source" : "https://github.com/cloudevents/spec/pull",
    "subject" : "123",
    "id" : "A234-1234-1234",
    "time" : "2018-04-05T17:31:00Z",
    "comexampleextension1" : "value",
    "comexampleothervalue" : 5,
    "datacontenttype" : "text/xml",
    "data" : "<much wow=\"xml\"/>"
}

An example Conduktor event would look like:

{
	"source": "//kafka/kafkacluster/production/topic/website-orders",
	"data": {
		"eventType": "Kafka.Topic.Create",
		// Additional event specific data...
		"metadata": {
			"name": "website-orders",
			"cluster": "production"
		}
		// Additional event specific metadata...
	},
	"datacontenttype": "application/json",
	"id": "ad85122c-0041-421e-b04b-6bc2ec901e08",
	"time": "2024-10-10T07:52:07.483140Z",
	"type": "AuditLogEventType(Kafka,Topic,Create)",
	"specversion": "1.0"
}

Specify the target Kafka cluster and topic using the environment variables CDK_AUDITLOGPUBLISHER_CLUSTER & CDK_AUDITLOGPUBLISHER_TOPICNAME and events will start being produced to the destination topic.

A full list of all the exported audit log event types is published on the Audit Log page.

Logging API​

Adjust the log level of Console without requiring a restart. We've added a new API endpoint to support targeted changes to log levels dynamically. Check the associated documentation for the full list of capabilities.

curl -X PUT 'http://localhost:8080/api/public/debug/v1/loggers/io.conduktor.authenticator/DEBUG' \
  -H "Authorization: Bearer $API_KEY"

Quality of Life improvements​

• Updated design and color theme
• Added navigation breadcrumb
• Enhanced error messages throughout the product
• Improved the connector 90 days heatmap
• Declaring an ApplicationInstance with resources ending in * will now fail with this error message
  • Could not apply resource ApplicationInstance/my-app-inst: resource name 'appA-*' is not allowed. Use name 'appA-' with patternType PREFIXED instead

Fixes 🔨​

• Fixed an issue with Topic Policy constraint Range where max value wasn't inclusive and min could greater than max
• Fixed an issue where Topic Policies were not enforced on Topic configuration changes in Console
• Added an error message when using the copy to clipboard button (for API Keys for instance) fails
• Added checks on local user creation emails
• Added new optional environment variable CDK_SSO_OAUTH2_0_OPENID_METADATADOCUMENT to modify the default discovery .well-known end-point
• Fixed an issue where Avro messages using logical type UUID couldn't be deserialized properly
• Fixed an issue with Cluster configuration page requiring platform.certificates.create permission to perform the TLS check
• Fixed an issue with "Remove user from group" button which is now disabled for users added by external group mapping
• Prevented the deletion of a group when it is owner of an Application
• Fixed an issue with the "New version" button in the banner that was still showing despite being on the latest version
• Fixed an issue where connections to the AWS glue schema registry would disconnect after a certain time and struggle to reconnect
• Fixed an issue where the "Reprocess message" feature was converting empty string headers to null value
• Fixed all critical and high CVE in console-cortex image
• Fixed an issue with the metric under_replicated_partitions when topics have confluent.placement.constraints property

Conduktor CLI 0.3.1​

Release date: 2024-10-14

Changes​

• Added handling of environment variables for YAML
• Added description for token CMD
• Updated README.md
• Added run sql
• Updated with latest Gateway API changes
• Added all

Fixes​

• Fixed release version check
• Fixed version bump in Homebrew PRs
• Fixed releases prefix to prevent errors

Gateway 3.3.2​

Release date: 2024-10-07

Fixes 🔨​

• Fixed an issue where the Producer Rate Limiting Policy would not honor its configuration.

Console 1.27.1​

Release date: 2024-09-25

Fixes 🔨​

• Improved performance of RBAC resolution when there is a large number of permissions

Gateway 3.3.1​

Release date: 2024-09-25

General fixes 🔨​

• Fixed an issue where Gateway would close the client connection upon receiving certain API Keys in parallel of the initial Metadata Request

Console 1.27.0​

Release date: 2024-09-14

• Features ✨
  • Kafka Connect Configuration Wizard
  • Alerts for Kafka Connect
  • Self-service: Limited Ownership mode
  • Quality of Life improvements
• Fixes 🔨
• Deprecation Warning: Upcoming migration from Tags to Labels

Features ✨​

Kafka Connect Configuration Wizard​

We are very excited to introduce our Configuration Wizard for Kafka Connect which is taking full advantage of the Kafka Connect Validate API:

• Form is generated with structured configuration groups
• Much nicer error handling, attached to each individual field
• Embedded documentation that helps you understand which fields are required and what are their expected and default values
• Ability to toggle advanced configuration to visualize only the most important fields
• Ability to switch seamlessly between Form View and JSON View at any time.

Give it a try and let us know what you think!

Alerts for Kafka Connect​

On top of the Kafka Connect Graphs we shipped last release, we now have added the ability to create alerts on them.

Self-service: Limited Ownership mode​

To help organizations transition to Self-service more easily, we have added a new attribute on ApplicationInstance to let Platform Teams decide the level of autonomy to give to Application Teams.

• ApplicationInstance resources configured with ownershipMode: ALL, which is the default, delegates all permissions related to that resource to the Application Team
• ApplicationInstance resources configured with ownershipMode: LIMITED delegates only a subset of the available permissions to the Application Team

This is especially useful if you already have a centralized repository and existing workflow for Topic creation. You may want to provide Self-service capabilities while still forcing your Application Teams to go through your pipeline for Topic Creation, instead of Self-service.

• Read more about Limited Ownership mode
• Get Started with Self-service today!

Quality of Life improvements​

• Self-service: External Group Mapping is now available for ApplicationGroup
• The Login page now steers users towards their OIDC provider rather than basic auth when OIDC is enabled

Fixes 🔨​

• Fixed an issue on Consumer group reset offset with the ToDatetime strategy
• Fixed an issue with Console indexing that could occur when deleting and recreating subject
• Fixed a recent regression with default replication factor when creating a topic
• Fixed a recent regression with Broker feature "Similar config" calculation
• Fixed a UI issue when Application Instance was created without any resources
• Fixed several issues around Microsoft Teams Integration to support Teams Workflow webhooks (Step by step guide)
• Fixed Kafka Connect client to use HTTP Proxy JVM configuration
• Switching Kafka cluster from the Topic details page now redirects to the Topic List
• Console doesn't override the client.id property anymore

Deprecation Warning: Upcoming migration from Tags to Labels 💣​

With the introduction of the Self-service resource manifests, we brought customers a means to annotate all their resources with labels. Labels are more structured than the existing Conduktor tags, thereby allowing for more precise filtering capabilities, as can be seen in the Topic Catalog.

In an upcoming release, we'll perform an automatic migration from Tags to Labels.

Tags written with the naming convention <key>/<value> will automatically be added as similar labels:

• <key>: <value>

If there is a conflict such as; a topic containing tags with the same key, that already has the target label, or is not written with this naming convention, then they will be created as follows:

tag-<value>: true

Here's an example of how tags will be migrated into labels:

# Tags:
- format/avro
- project/supplychain
- team/delivery
- color/blue
- color/red
- wikipedia
- non-prod

# Result
labels:
  format: avro
  project: supplychain
  team: delivery
  tag-color/blue: true # Because conflict on "color"
  tag-color/red: true # Because conflict on "color"
  tag-wikipedia: true
  tag-non-prod: true

⚠️ Conduktor can help you rename tags through Customer Support
Between now and the migration, we can help you rename your tags for a smooth transition to labels.

Contact us as soon as possible if you would like support.

Conduktor CLI 0.3.0​

Release date: 2024-09-17

Changes​

• Added support for -o json and -o name on get
• Updated to latest Gateway API
• Added support for Gateway API v2
• Clarified version with a v

Fixes​

• Fixed the release tag
• Fixed missing key retrieval from environment

Find out more.

Gateway 3.3.0​

Release date: 2024-09-05

Upcoming Breaking change 💣​

Today, the token as the password for local Gateway service accounts contains all the necessary information. As a result, the SASL username is not used during the authentication phase.

In an upcoming release, we will strictly enforce that the username and the token matches. This will help reduce inconsistencies and avoid unexpected behaviors.

This breaking change is due for release 3.5.0.
For this release 3.3.0, and next product release 3.4.0, we'll only raise the following warning in the logs:

2024-08-27T18:15:29 [WARN] - Inconsistency detected for plain authentication. Username applicationA is not consistent with validated token created for application-A. SASL configuration should be changed accordingly.

Features ✨​

• New V2 APIs and CLI support
• Support for HTTPS APIs
• Better UX for ACLs and superUsers
• Encryption Enhancements and Support Clarification

New V2 APIs and CLI support​

We’re excited to introduce our new Gateway API, designed for seamless integration with our CLI. This update allows you to deploy Gateway resources using infrastructure-as-code with straightforward, clearly defined concepts:

• Interceptor
• GatewayServiceAccount
• GatewayGroup
• ConcentrationRule
• AliasTopic
• VirtualCluster

Check the CLI reference to get started, and the resources reference for more information on each concept.

---
apiVersion: gateway/v2
kind: GatewayGroup
metadata:
  name: groupB
spec:
  members:
    - name: user1
    - name: user2
---
apiVersion: gateway/v2
kind: Interceptor
metadata:
  name: enforce-partition-limit
  scope:
    group: groupB
spec:
  pluginClass: io.conduktor.gateway.interceptor.safeguard.CreateTopicPolicyPlugin
  priority: 100
  config:
    numPartition:
      action: BLOCK
      max: 9
      min: 9
    topic: .*

$ conduktor apply -f gateway.yml
GatewayGroup/groupB: Created
Interceptor/enforce-partition-limit: Created

$ conduktor delete GatewayGroup groupB
The group groupB is still used by the following interceptor(s): enforce-partition-limit

Note: API V1 is still available, but we recommend that new users and those with simple Gateway configurations begin using the V2 API as soon as possible. We will announce a deprecation plan in the coming weeks and notify you in advance of which Gateway version will be the last to support the V1 APIs.

Support for HTTPS APIs​

It is now possible to configure HTTPS and mTLS authentication on the Gateway HTTP APIs. Check the HTTP section of the Environment Variables page for more details.

Better UX for ACLs and superUsers​

To coincide with the clearly defined concepts established in API V2, we are making changes to ACLs management in Gateway.

• ACLs and Super Users on the Gateway (excluding Virtual Clusters) must be configured through Environment Variables.
• ACLs and Super Users on Virtual Clusters must now be driven explicitly through API/CLI.

Enable ACLs for Gateway (excl. Virtual Clusters)​

Configure both environment variables:

GATEWAY_ACL_ENABLED=true # default false
GATEWAY_SUPER_USERS=alice,bob

If GATEWAY_SUPER_USERS is not set, it will default to GATEWAY_ADMIN_API_USERS for backward compatibility.

Enable ACLs for Virtual Clusters​

Example configuration:

---
apiVersion: gateway/v2
kind: VirtualCluster
metadata:
  name: "mon-app-A"
spec:
  aclEnabled: "true" # defaults to false
  superUsers:
  - username1
  - username2

Encryption Enhancements and Support Clarification​

Field-Level Encryption: Preserving Message Format to Enhance Usability​

When applying field-level encryption prior to 3.3.0, the encryption plugin would convert the message to JSON, and re-apply the schema format when the message was read back through the decryption plugin.

In Gateway 3.3.0, we now preserve the schema format for Avro messages - meaning the same schema is used in the backing topic, and the data can be read directly from Kafka or without the decryption plugin at all.

Read more about this change to the default behaviour, and how to configure it.

Fields which cannot be encrypted in-place (effectively any non-string field) have their encrypted value placed in the headers, and the field itself is given a default masking value. The default values are clarified below:

Note that the same default values are now used across all relevant plugins when manipulating a non-string field - Data Masking, Partial Decrypt, and Encrypt on Fetch.

Attempt to apply encryption to a message more than once will now fail​

If any of the encryption headers are detected in a message when encryption is about to be applied, then the encryption operation will fail. This is because applying encryption twice (or more) is currently not reversible.

Deprecated support for Schema Based (tag) encryption with Protobuf​

Note this is no longer supported, and the Gateway will now throw an exception if the encryption plugin attempts to apply schema (tag) based processing to a Protobuf message.

Note that any data previously written in this mode can still be read back - as the decrypt does not use the schemas at all, rather it uses the message header to know what was encrypted.

General fixes 🔨​

• Large double values (where > Float Max) are now supported in field-level encryption for Avro and Protobuf
• Bytes and fixed fields now properly supported in field-level encryption for Avro
• Avro unions of two or more values (rather than just a value and a null) are now supported in field-level encryption for Avro
• Schema (tag) based encryption now checks and fails if its config is invalid
• It is not possible to encrypt the headers which the encryption plugin uses to manage its decryption process (as this would render the data unrecoverable)
• Improved log messages for Interceptors that reject actions, such as TopicPolicyPlugin
• Several improvements to the LargeMessage & LargeBatch Interceptors
• Fixed an issue where KCache topic initialization would fail silently and leave Gateway in an unusable state
• Added a new Environment Variable GATEWAY_MIN_BROKERID (default 0) that allows for determinist mapping of brokers and ports
• Improved network stability during Gateway scaling or Kafka topology changes
• Added support for overriding Kafka Producer properties used for Audit Log topic with GATEWAY_AUDIT_LOG_KAFKA_ environment variables
• Removed metric gateway.brokered_active_connections. This was equal to portCount with port mapping and always 1 in host mapping
• Changed metric gateway.request_expired tags: nodeHost/nodePort are replaced by nodeId/clusterId
• Fix default value for GATEWAY_UPSTREAM_THREAD config. The new intended default (number of CPU) previously was (2 x number of CPU).
• Fixed an issue with GATEWAY_ADVERTISED_SNI_PORT that wasn't working properly
• Add log level for io.confluent packages in default log configuration
• Add default value to non mandatory configruation value for min and max bytes in FetchPolicyInterceptor
• Fix an issue with Concentrated Topics creation with Redpanda

Known issues​

• We are aware of an issue with kcat when the new environment variable GATEWAY_MIN_BROKERID is not aligned with the first BrokerId of your Kafka cluster.
  • As a workaround, you can either define GATEWAY_MIN_BROKERID to your first Kafka BrokerId or use kcat with the -E flag
• It is not possible to add Service Accounts to GatewayGroups using API V2 unless they are previously declared as GatewayServiceAccount.
  • This is not a wanted behavior, especially for OAuth or Delegated Kafka Authentication where declaring a GatewayServiceAccount should not be needed. We'll address this issue in a follow-up release
  • API V1 (user-mapping) is not impacted
• If you perform a rolling upgrade to 3.3.0, Gateway nodes in earlier versions will show the following error in the logs: [ERROR] [KafkaCache:1007] - Failed to deserialize a value org.apache.avro.AvroTypeException: Expected field name not found: clusterId
  • This is fine and will not cause any further problems
• If you use Virtual Clusters and ACLs: After updating to 3.3.0, you must manage VirtualCluster's ACL and superUsers through V2 API.

Gateway 3.2.2​

Release date: 2024-08-28

Upcoming Breaking change 💣​

Today, the token as the password for local Gateway service accounts contains all the necessary information. As a result, the SASL username is not used during the authentication phase.
In an upcoming release, we will strictly enforce that the username and the token matches. This will help reduce inconsistencies and avoid unexpected behaviors.

This breaking change is due for release 3.5.0.
For this hotfix release 3.2.2, and next product releases 3.3.0 and 3.4.0s, we'll only raise the following warning in the logs:

2024-08-27T18:15:29 [WARN] - Inconsistency detected for plain authentication. Username applicationA is not consistent with validated token created for application-A. SASL configuration should be changed accordingly.

General fixes 🔨​

• Fixed a severe authentication issue with Gateway generated tokens that could lead to a different user being authenticated, effectively causing elevated privileges under certain conditions.
• Fixed an issue where GATEWAY_SNI_HOST_SEPARATOR couldn't be set to the value -
• Fixed an issue where GATEWAY_SNI_HOST_SEPARATOR wasn't properly taken in account
• Fixed an issue with GATEWAY_ADVERTISED_SNI_PORT that wasn't working properly

Console 1.26.0​

Release date: 2024-08-14

Features ✨​

• Manage Connectors using the CLI
• Self-service support for Connectors
• Enhanced UI & Alerts for Kafka Connect
• Quality of Life improvements
• Deprecation Warning: Upcoming migration from Tags to Labels

Manage Connectors using the CLI​

Continuing with the Infra-as-code approach, we are happy to introduce CLI support for Connectors, providing an efficient and automated way to manage your Kafka Connect resources.

---
apiVersion: kafka/v2
kind: Connector
metadata:
  connectCluster: kafka-connect
  name: click.my-connector
  labels:
    conduktor.io/auto-restart-enabled: true
    conduktor.io/auto-restart-frequency: 600
spec:
  config:
    connector.class: 'org.apache.kafka.connect.tools.MockSourceConnector'
    tasks.max: '1'
    topic: click.pageviews

Self-service support for Connectors​

Application Teams can now manage their Connectors with Self-service.
From now on, you can grant ownership to connectors on Self-service Application Instance.

---
apiVersion: self-service/v1
kind: ApplicationInstance
metadata:
  application: "clickstream-app"
  name: "clickstream-dev"
spec:
  cluster: "shadow-it"
  serviceAccount: "sa-clicko"
  resources:
    - type: CONNECTOR
      connectCluster: shadow-connect
      patternType: PREFIXED
      name: "click."

Enhanced UI & Graphs for Kafka Connect​

We have revisited the Kafka Connect UI in multiple ways to improve your experience:

• Connect Cluster selection screen with a preview of Connector status
• New graphs demonstrating the state of your Connector over time

Support for High Availability (HA) Console​

Multiple Console instances can now be deployed in parallel to achieve high availability.

This applies to the deployment of conduktor-console, while conduktor-console-cortex is currently limited to a single instance. The design ensures minimal impact on the cluster by assigning only one instance to handle the indexing of Kafka data used for performance monitoring.

Quality of Life improvements​

• The checkbox to skip TLS verification is now always visible
• The YAML for Topic object now allows number in spec.configs. Previously it was mandatory to quote all numbers.
• Self-service Topic Policies are now visible in the UI

Fixes 🔨​

• Topic Policies from Self-service are now properly enforced from the UI, as well as both the API and CLI
• Fix Kafka Connect Cluster list showing invalid number of running tasks

Deprecation Warning: Upcoming migration from Tags to Labels 💣​

With the introduction of the Self-service resource manifests, we brought customers a means to annotate all their resources with labels. Labels are more structured than the existing Conduktor tags, thereby allowing for more precise filtering capabilities, as can be seen in the Topic Catalog.

In an upcoming release, we'll perform an automatic migration from Tags to Labels.

Tags written with the naming convention <key>/<value> will automatically be added as similar labels:

• <key>: <value>

If there is a conflict such as; a topic containing tags with the same key, that already has the target label, or is not written with this naming convention, then they will be created as follows:

tag-<value>: true

Here's an example of how tags will be migrated into labels:

# Tags:
- format/avro
- project/supplychain
- team/delivery
- color/blue
- color/red
- wikipedia
- non-prod

# Result
labels:
  format: avro
  project: supplychain
  team: delivery
  tag-color/blue: true # Because conflict on "color"
  tag-color/red: true # Because conflict on "color"
  tag-wikipedia: true
  tag-non-prod: true

⚠️ Conduktor can help you rename tags through Customer Support
Between now and the migration, we can help you rename your tags for a smooth transition to labels.

Contact us as soon as possible if you would like support.

Gateway 3.2.1​

Release date: 2024-07-31

General fixes 🔨​

• Fixed an issue when either GATEWAY_ACL_ENABLED or GATEWAY_ACL_STORE_ENABLED was enabled resulting in ACLs being enforced in undesirable scenarios.

Gateway 3.2.0​

Release date: 2024-07-19

Breaking Changes 💣​

Two new backing topics are required for Gateway​

In the next release (3.3), we'll bring a new API as well as support in the Conduktor CLI to manage Gateway concepts using infra-as-code approach.

In preparation for this upcoming release, we are replacing some weakly-defined concepts in favor of strongly-defined concepts. The following are now clearly captured in the topics mentioned below:

• Virtual Clusters that existed only through creation of UserMappings or Interceptors targeting
• GatewayGroups that existed only through UserMappings

As a result, 2 new topics will now be created once you upgrade to Gateway 3.2:

• _conduktor_gateway_vclusters
• _conduktor_gateway_groups

If you are happy with the default names, you have nothing to do. If you want to control the name of those topics, use the 2 new environment variables:

• GATEWAY_VCLUSTERS_TOPIC
• GATEWAY_GROUPS_TOPIC

Check the associated Documentation for more information.

Changes to ACL support on Gateway​

With Gateway 3.1 we removed our dedicated ACL interceptor in favor of a new environment variable GATEWAY_ACL_STORE_ENABLED. This variable was enabling ACLs in all scenarios, whether you used Virtual Clusters or not.

Changes for Gateway 3.2​

With Gateway 3.2, we are adding a new environment variable GATEWAY_ACL_ENABLED and modifying the behavior of the existing variable GATEWAY_ACL_STORE_ENABLED.

From now on, the variables works as follow:

Preview for Gateway 3.3​

In the next release, we will enhance ACLs to restore and expand the full set of features available before version 3.1. This will be achieved through the introduction of a CLI and APIs, making concepts like VirtualCluster first-class citizens.

Enable ACLs for Gateway (excl. Virtual Clusters)​

Configure both environment variables:

GATEWAY_ACL_ENABLED=true
GATEWAY_SUPER_USERS=alice,bob

Enable ACLs for Virtual Clusters​

---
apiVersion: gateway/v2
kind: VirtualCluster
metadata:
  name: "mon-app-A"
spec:
  aclEnabled: true # defaults to false
  superUsers:
  - username1
  - username2

This will effectively render GATEWAY_ACL_STORE_ENABLED obsolete.

General fixes 🔨​

• Fixed an issue with Field-level Avro encryption/decryption relating to numeric fields:
  • When using partial decryption with Avro schema registry, any numeric values (int, long, float, double) that are not being decrypted will instead be masked with the minimum (most negative) value for the numeric type
  • This is to ensure the field is compliant with the original type in the Avro schema
• Fixed an issue with the ClientIdRequired Policy that wasn't properly overriding the ClientId
• Fixed an issue to ensure all active connections are closed, and clients transition quicker to the new cluster during cluster switching

Console 1.25.1​

Release date: 2024-07-23

Breaking Changes 💣​

New docker image name​

We have renamed the Console docker image to conduktor/conduktor-console to clarify our product naming.
Please modify your installation to reflect this change as we will now stop publishing a conduktor/conduktor-platform image.

docker pull conduktor/conduktor-console:1.25.1

Features ✨​

• Conduktor Console IaC Compatible
  • Manage Cluster Connections
  • Short lived token generation on startup
  • Admin and Application Tokens
• Shareable Message Page
• Large Messages Support
• Topic Catalog Details Page
• Audit Last Activity of Users
• Quality of Life improvements

Conduktor Console IaC Compatible​

Console is now able to be fully deployed through an IaC approach with the following additions to Console 1.25 and CLI 0.2.7.

Manage Cluster Connections​

Manage your Console resource lifecycle with the addition of the KafkaCluster, KafkaConnectCluster and KsqlDBCluster objects to our IaC approach using the Conduktor CLI.

Checkout the example below and find the full definition at Console Resources Reference documentation.

---
apiVersion: console/v2
kind: KafkaCluster
metadata:
  name: cloud-kafka
spec:
  displayName: "Cloud Kafka"
  icon: "kafka"
  color: "#000000"
  bootstrapServers: "localhost:9092"
  properties:
    sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret";
    security.protocol: SASL_SSL
    sasl.mechanism: PLAIN
  schemaRegistry:
    url: http://localhost:8080
    security:
      type: BasicAuth
      username: some_user
      password: some_user

Short lived token generation on startup​

When spinning up Console, a token is needed to access the API. Previoulsy this had to be done in the UI which would not allow full IaC. Now, we have the conduktor login command which leverages the admin credentials to generate an API token, and allow the rest of the commands you may need to startup. This is expanded upon in the docs.

Admin and Application Tokens​

In addition to the startup token, you can now generate tokens for the appropriate scope, for admin and application level tokens. The docs will walk you through this.

Shareable Message Page​

Individual messages can now be accessed from a unique URL! Now you can link directly to a specific Kafka message for review or investigation, be that for sharing with a teammate, or commenting on a Jira ticket.

From within the Consume page, select a message and use the 'Share' button to navigate to the standalone page. The standalone message page shows the key, value, metadata and headers in a single view. Switch between the JSON view or table view, and utilize jq for additional filtering of the value.

Large Messages Support​

We have put a limit on the message sizes that are sent to the browser in the Consume page (100Kb). From now on, when a message is larger than this size, we'll provide you with a link to access the individual message - this mitigates performance issues and still provides a path for troubleshooting, and sharing, large messages.

Topic Catalog Details Page​

Expose contextual documentation about your Kafka Topics that exist in your organization with the Topic Details page. This helps democratize data to enhance its understanding and usage, and facilitate collaboration through a shared knowledge base.

You can choose to open or lock editing of descriptions within the UI using specific annotations. Check the Topic Resource documentation for more information.

Audit Last Activity of Users​

You can now audit the last activity date of users in Console.

From within the Settings > Users page, you will see a new column 'Last login'. Note that the user login event is also captured in the Audit Log.

Quality of Life improvements​

• Introduced an intermediate screen for Kafka Connect, allowing you to segment Connectors by each Connect cluster
• Within a Connect cluster, introduced an icon for each connector that clarifies if auto-restart is enabled
• Topic Catalog Search is now case-insensitive
• Improved error message when trying to delete an ApplicationInstance that is referenced elsewhere
• Improved error message when assign ownership on resources already owned by another ApplicationInstance
• CLI delete command can now be applied at the file level, simliar to resource creation through apply -f you can now delete -f

Fixes 🔨​

• Fixed an error that occurred when configuring a KsqlDBCluster in the UI
• Fixed a UI issue that caused several dropdowns components to look wrong
• Fixed an error message where expected and actual topic replication factor were inverted in the CLI
• When deleting a Kafka Cluster from Console, the Indexed data is now properly deleted as well
• Upgrade dependencies vulnerable to CVE-2024-21634

Conduktor CLI 0.2.7​

Release date: 2024-07-22

Changes​

• Made /api not mandatory when setting base URL
• Updated offline schema for future
• Added support for description from file in topic resources
• Added support for delete -f
• Improved client interface
• Improved CLI API for terraform provider POC
• Added a command to create token
• Added a login command

Fixes​

• Fixed auto login in client

Conduktor CLI 0.2.6​

Release date: 2024-06-25

Changes​

• Started with v to solve issue with go package
• Use resource priority from default catalog if catalog doesn't have any
• Added resource priority
• Keep order from API response
• Updated offline kind

Gateway 3.1.1​

Release date: 2024-06-20

General fixes 🔨​

• Performance is improved when using a large number of interceptors (backported in 3.0.5)
• Pre-create folders when using RocksDB as a cache backend
• Moved the Schema Id to the headers when using field level encryption with Avro

Console 1.24.1​

Release date: 2024-06-24

Fixes 🔨​

• Fixed a UI issue on Self-service Application Catalog and Topic Policies pages
• Fixed a UI issue on Topic Catalog when listing topics created with empty configs
• Fixed an issue with KqlDB connection test button
• Fixed an issue with the new delete users from group endpoint definition in OpenAPI spec

Conduktor CLI 0.2.5​

Release date: 2024-06-18

Changes​

• Expended Subject schemaFile on apply

Fixes​

• Fixed an issue on apiVersion where apiVersion is put at the end

Gateway 3.1.0​

Release date: 2024-06-05

AclsInterceptorPlugin removed​

Kafka ACLs are now fully integrated in the Core Features of Conduktor Gateway.
If you were using the AclsInterceptorPlugin, make sure to enable ACLs while upgrading the Gateway to 3.1.0.
To enable ACLs set the environment variable GATEWAY_ACL_STORE_ENABLED=true.

Features​

• Concentrated Topics can now be created with auto-managed flag. Backing topics will be automatically created and extended.
• Added support for Azure Managed Identity for Kafka authentication
• Added an optional configuration for SNI routing to define the separator to use when building host domain for brokers
• Added more context relative to interceptors in Audit logs
• Added the client & version (kafka-client, librdkafka, ...) of the client in the Audit logs on CONNECTED event

General fixes 🔨​

• Added Schema Registry validation on encryption plugins
• Fixed an issue where the KMS Key would not be created if it didn't exist
• Fixed an issue with logger API

Gateway 3.0.5​

Release date: 2024-06-04

Performance improvements 🚀​

• Performance is improved when using a large number of interceptors

Console 1.24.0​

Release date: 2024-06-19

Breaking Changes 💣​

New docker image name​

We have renamed the Console docker image to conduktor/conduktor-console to clarify our product naming. Final warning! This is the last version where we publish our images using both names. Please modify your installation to reflect this change in advance of us deprecating the name conduktor/conduktor-platform.

docker pull conduktor/conduktor-console:1.24.0

Change in ApplicationInstance Resource Type from GROUP to CONSUMER_GROUP​

We have renamed the resource type in ApplicationInstance from GROUP to CONSUMER_GROUP. This change is intended to prevent confusion with the newly introduced resources ApplicationGroup and Group.

---
kind: ApplicationInstance
spec:
  resources:
    - type: CONSUMER_GROUP        # Previously: GROUP
      name: "click."
      patternType: PREFIXED

Features ✨​

• Self-service
  • Subject
  • ApplicationGroup
  • Topic Catalog
• Manage Groups and Users using the CLI
• Topic list columns Produce Rate and Last Activity
• Active Data Policies in Topic Consume page
• Quality of Life improvements
• Fixes 🔨

Self-service​

There's a host of new functionality available providing a truly powerful self-service release. This comes from the addition of two new resources: Subject and ApplicationGroup

Application Teams can now manage their Subject resource lifecycle through IaC with the addition of the Subject object.

A new concept, ApplicationGroup lets Application Teams fully organize themselves within their Application scope to restrict who can do what over their resources within Console UI. It's a form of delegated RBAC.

Checkout the definitions below and find the full list of resource definitions via the Resource Reference documentation.

Subject​

This creates a Subject in the Schema Registry.

---
apiVersion: v1
kind: Subject
metadata:
  cluster: shadow-it
  name: myPrefix.topic-value
spec:
  schemaFile: schemas/topic.avsc # relative to conduktor CLI execution context
  format: AVRO
  compatibility: FORWARD_TRANSITIVE

ApplicationGroup​

Create an Application Group to directly reflect how your Application operates. You can create as many Application Groups as required to restrict or represent the different teams that use Console on your Application, e.g.:

• Support Team with only Read Access in Production
• DevOps Team with extended access across all environments
• Developers with higher permissions in Dev

# Permissions granted to Console users in the Application
---
apiVersion: v1
kind: ApplicationGroup
metadata:
  application: "clickstream-app"
  name: "clickstream-support"
spec:
  displayName: Support Clickstream
  description: |
    Members of the Support Group are allowed:
      Read access on all the resources
      Can reset offsets
  permissions:
    - appInstance: clickstream-app-dev
      resourceType: TOPIC
      patternType: "LITERAL"
      name: "*" # All owned & subscribed topics
      permissions: ["topicViewConfig", "topicConsume"]
    - appInstance: clickstream-app-dev
      resourceType: GROUP
      patternType: "LITERAL"
      name: "*" # All owned consumer groups
      permissions: ["consumerGroupReset", "consumerGroupView"]
  members:
    - alice@company.org
    - bob@company.org

Topic Catalog​

We're expanding on the Topic Catalog, to help teams discover Kafka Topics within your organization. You can now filter on all the topics based on user-defined, business metadata. Looking to request access to another applications resources? You can now generate the required ApplicationInstancePermission snippet that grants the necessary access to Topics belonging to another Application.

Manage Groups and Users using the CLI​

Manage your Console Group and Permissions lifecycle through IaC with the addition of the Group and User objects.
Checkout the example below and find the full definition via the Resource Reference documentation.

---
apiVersion: v2
kind: Group
metadata:
  name: developers-a
spec:
  displayName: "Developers Team A"
  description: "Members of the Team A - Developers"
  externalGroups: 
    - "LDAP-GRP-A-DEV"
  members:
    - member1@company.org
    - member2@company.org
  permissions:
    - resourceType: TOPIC
      cluster: shadow-it
      patternType: PREFIXED
      name: toto-
      permissions:
        - topicViewConfig
        - topicConsume
        - topicProduce

Topic list columns Produce Rate and Last Activity​

We added two new columns to the Topic List to help you troubleshoot and understand Kafka better: Produce Rate & Last Activity.

Values are computed once per Indexing (i.e. every 30s);

• Produce Rate is calculated from the two most recent offset values provided by our indexer.

• Last Activity is set to Datetime.now() if the latest offsets have changed since the last Indexing

Active Data Policies in Topic Consume page​

When exploring topics, fields masked by active Data Policies are now displayed in a different color, while the policy name is also now visible on hover.

Quality of Life improvements​

Topic pages

• You can now see all subjects associated to the Schema Id of the current message from the Message Viewer panel
• Added message Compression Type metadata in the Message Viewer panel
• Added buttons to navigate to previous and next message in the Message Viewer panel. Also works with the arrow keys
• The "Generate once" feature in the Produce tab now generates much more realistic, randomized messages, especially for Registry schemas and JSON

Other pages

• Added a button to force re-balance active Consumer Groups in the Consumer Group details page
• Added a "Test connection" button when adding a KsqlDB cluster in Settings
• Added KsqlDB query Start From selector, equivalent to the SET 'auto.offset.reset' command
• Added an icon in the Kafka Connect list to inform that auto-restart feature is active

API

• When returning a Forbidden error, the missing permissions are listed in the error message
• New endpoint to add user to a group by email

Conduktor CLI
Update your Conduktor CLI to 0.2.5

• Env Variable changed from CDK_TOKEN to CDK_API_KEY to set your Admin or Application API Key
• Added support for Subject field spec.schemaFile. Previous versions of the CLI will only accept spec.schema inlined.

Fixes 🔨​

• Clean monitoring metrics related to brokers that are unreachable
• Fix support of Avro byte arrays encoded as base64 when producing messages
• Fix bulk import of users in case a user already exist
• Fix user creation when the user is not admin but has the right permissions
• Fix class name selector when navigating from one interceptor to another
• External group mapping: support extraction of roles from both string array and comma separated string
• Fix preview of consumer group offset reset when selecting a specific offset
• Data masking: trim name of policy and fix encoding for URL
• Monitoring: show error in UI if cortex is unreachable
• Fix schema that disappeared from the form input when schema was invalid
• Prevent the creation of an application instance with resources that overlaps
• Fix permissions when 2 application instances define resources on the same cluster
• Fixed an issue where apiVersion was displayed at the end using the CLI

Gateway 3.0.4​

Release date: 2024-05-22

Performance improvements 🚀​

• Consumer group membership is no longer loaded synchronously
• Optimize hostname resolution for ACL

General fixes 🔨​

• GATEWAY_DOWNSTREAM_THREAD and GATEWAY_UPSTREAM_THREAD are now correctly gathering the number of cores
• in LargeMessageHandlingPlugin plugin, honor correctly the localCacheExpireAfterWriteInSeconds property

Gateway 3.0.3​

Release date: 2024-05-09

General fixes 🔨​

• Fixed an issue impacting the vault configuration key uri when special characters (i.e -) are present in the hostname.

Gateway 3.0.2​

Release date: 2024-05-07

General fixes 🔨​

• Fixed a race condition when closing connections (i.e. when Gateway detects a broker is removed from the cluster) that was causing restarts/timeouts
• Fix duplicated key exception when rebuilding fetch request with duplicated topics
• FIX NPE when handling expired ApiVersions requests
• Added a check to validate schema registry connection and provide more meaningful errors for schema-based encryption
• Added a check against defaultAlgorithm used in the encryption interceptor to ensure it's a valid enum value, and avoid overriding with defaults
• Fixed an issue with externalStorage set to true in the encryption interceptor that was failing to store headers in a separate internal topic
• Ensure that if the encryption algorithm is changed, a new entry appears in the internal topic used to store headers
• Default namespace is now applied properly on schema-based encryption
• Added support encryption/decryption of AVRO bytes and enums types

Console 1.23.0​

Release date: 2024-05-03

Future Breaking Changes 💣​

New docker image name​

We have renamed the Console docker image to conduktor/conduktor-console to clarify our product naming.

We will publish newer versions using both names for this release and the next release only. Please modify your installation to reflect this change in advance of us deprecating the name conduktor-platform.

docker pull conduktor/conduktor-console:1.23.0

Features ✨​

• Conduktor Console
  • Future Breaking Changes 💣
    • New docker image name
  • Features ✨
    • Self-service
      • Topic
      • TopicPolicy
      • Topic Catalog
      • Application API Keys
    • Editable columns on the Consume Page
    • Quality of Life improvements
  • Fixes 🔨

Self-service​

There's a host of new functionality available providing our first truly powerful self-service release. This comes from the addition of two new resources (Topic, TopicPolicy), application tokens, a topic catalog and service account synchronization.

Application Teams can now manage their Topic resource lifecycle through IaC with the addition of the Topic object, and they can do this safely with Platform Teams putting in place a Topic Policy to restrict expensive configurations and enforce naming standards.

Checkout the definitions below and find the full list of resource definitions via the Resource Reference documentation.

Topic​

This creates a Topic in the defined cluster.

---
apiVersion: v2
kind: Topic
metadata:
  cluster: shadow-it
  name: click.event-stream.avro
spec:
  replicationFactor: 3
  partitions: 3
  configs:
    min.insync.replicas: '2'
    cleanup.policy: delete
    retention.ms: '60000'

TopicPolicy​

TopicPolicy lets Platform Team define governance rules to restrict Application Teams to create Topics with misconfigurations. This is also useful to enforce naming convention or metadata annotation by Application Teams.

---
apiVersion: "v1"
kind: "TopicPolicy"
metadata:
  name: "click-naming-rule"
spec:
  policies:
    metadata.name:
      constraint: Match
      pattern: ^click\.(?<event>[a-z0-9-]+)\.(avro|json)$
    spec.replication.factor:
      constraint: OneOf
      values: ["3"]
    spec.configs.retention.ms:
      constraint: Range
      max: 604800000  # 7d 
      min: 3600000    # 1h

Topic Catalog​

We've introduced the Topic Catalog, to help teams discover Kafka Topics within your organization. Quickly get visbility on ownership and business metadata on your choice for topics.

Add topics to applications to see them appear within the catalog across all your clusters, searchable by name and labels.

Application API Keys​

Generate ApplicationInstance API Keys to create any ApplicationInstance scoped resources. Only ApplicationInstancePermission and Topic are supported at the moment.

Use this Key with the CLI to use it manually or within CI/CD pipelines.

In addition, Service Account's ACLs are now synchronized with the permissions from ApplicationInstance and ApplicationInstancePermission resources.

Read More about Self-service

Editable columns on the Consume Page​

You can now customise the columns you want to display in the Consume Page. Let us know if there's any additional metadata you want to see!

Quality of Life improvements​

Topic pages

• SchemaId is now displayed from the Message Viewer panel
• Header count is now displayed from the Message Viewer panel
• The More Options "..." button has been moved so that it's available from every Topic details tab
• Added a check to prevent producing empty keys to a compacted topic
• Added an "Add partitions" button in Partitions tab

Schema Registry pages

• The current schema is now inside a read-only area
• Increased the width of the side panel when creating/updating schemas
• Full height is used in the panel to show/edit the schema

Kafka Connect pages

• Kafka Connect List can now be sorted by the number of Tasks
• Removing a Connector now properly redirects the user to the Connector list instead of the Configuration tab of the deleted Connector
• Topics column is now sourced from more configuration keys (kafka.topic, kafka.topics, topic, topics)

Settings

• Permissions on KafkaConnect and ksqlDB now properly display the name instead of the UUID
• Adding Users to Groups can now be done from the User details page directly
• Added the Group name in the UI to be used in the API or CLI

Other

• Added Gateway version on the Interceptor List page
• Added a configuration option to toggle OIDC logout when logging out from Console
• Searching in screens now trims whitespace from the text supplied

Fixes 🔨​

• Fixed an issue with the Test Connection button that didn't work after a successful response
• Fixed an issue with the indexing of Confluent Cloud Managed Connect
• Fixed an issue with the Kafka Connect List where filter by Connect Cluster wouldn't work in some cases
• Fixed an issue with the Schema Registry indexer not properly handling a retriable HTTP error (GOAWAY)
• Fixed an issue with the timezone selector scrolling when resetting offsets for a Consumer Group by timestamp
• Fixed an issue with SSO in Azure environments for users who are members of a large amount of Azure groups
• The following fixes have also been back-ported in 1.22.1
  • Fixed an issue where two ACLs of the same name but with different pattern types (PREFIXED and LITERAL) were merged to the same group within the UI
  • Fixed an issue with OIDC login that could cause an expired session to become stuck and prevent login attempts
  • Fixed an issue with ksqlDB caused by not escaping the Stream or Table name in the query

Conduktor CLI 0.2.4​

Release date: 2024-05-02

Breaking change​

• CDK_TOKEN is now CDK_API_KEY

This also includes Conduktor CLI v 0.2.3 wih a fix for

• automation update brew formula

This also includes Conduktor CLI v 0.2.2 wih a fix for

• improved docs

Conduktor CLI 0.2.1​

Release date: 2024-04-29

Features​

• Added CA cert
• This release is only compatible with Console 1.23 or above

Fixes​

• Fixed an issue with insecure

Console 1.22.1​

Release date: 2024-04-18

Features ✨​

• Added support for Azure Managed Identity for Kafka authentication
• Implement OIDC logout. You may need to update your OIDC configuration to allow the root page of Console as a possible redirect URI

Fixes 🔨​

• Fixed an issue where two ACLs with the same name but with different pattern type (PREFIXED and LITERAL) were merged in the same group in the UI.
• Fixed an issue with OIDC login that could cause an expired sessions to become stuck and prevent login in again.
• Fixed an issue with ksqlDB caused by not escaping the Stream or Table name in the query.

Console 1.21.3​

Release date: 2024-04-18

Features ✨​

• Added support for Azure Managed Identity for Kafka authentication
• Implement OIDC logout. You may need to update your OIDC configuration to allow the root page of Console as a possible redirect URI

Fixes 🔨​

• Fixed an issue where two ACLs with the same name but with different pattern type (PREFIXED and LITERAL) were merged in the same group in the UI.
• Fixed an issue with OIDC login that could cause an expired sessions to become stuck and prevent login in again.

Gateway 3.0.1​

Release date: 2024-04-15

General fixes 🔨​

• Fixed some issues with Encryption when the value is a tombstone.
• Fixed some inconsistencies between the OpenAPI Spec and the actual implementation.
• Fixed a memory leak when using the default GATEWAY_UPSTREAM_CONNECTION_POOL_TYPE.
• Added a startup check to prevent an incompatible configuration: GATEWAY_UPSTREAM_CONNECTION_POOL_TYPE=ROUND_ROBIN with delegated authentication.

Console 1.22.0​

Release date: 2024-04-03

Future Breaking Changes 💣​

New docker image name​

We have renamed the Console docker image to conduktor/conduktor-console to clarify our product naming.

We will publish newer versions using both names for the next two releases only. Please modify your installation to reflect this change in advance of us deprecating the name conduktor-platform.

docker pull conduktor/conduktor-console:1.22.0

Features ✨​

• Conduktor Console
  • Future Breaking Changes 💣
    • New docker image name
  • Features ✨
    • Topic as a Service becomes Self-service
    • Conduktor CLI
    • Custom Deserializers
  • Fixes 🔨

Topic as a Service becomes Self-service​

Self-service is a replacement for Topic as a Service. It is more centered towards a GitOps way of working, though we have performed a migration for existing TaaS applications to ensure a seamless transition to the new model:

• Applications + Environments are migrated to Application and ApplicationInstance
• Cross Application accesses are migrated to ApplicationInstancePermission
• The Application list becomes Application Catalog
• At the moment, we decided that we should control everything from the CLI only. The UI will remain Read-Only for now, but the intention is to bring back UI-driven operations in a future release.

Read More about Self-service

To start using Self-service, you must download our Conduktor CLI which lets you deploy resource files in Console.

Conduktor CLI​

Console now has a CLI! Get Started with it today.

For now, we only support the following resources:

• Application
• ApplicationInstance
• ApplicationInstancePermission

Our objective is to let Application Teams and Central Teams manage both Console resources (Clusters, Groups, Permission, Self-service resources, DataPolicies, Alerts, ...) and Kafka resources (Topics, Subjects, Connectors, ...) using a common definition mechanism.

More to come, automate everything!

---
apiVersion: "v1"
kind: "ApplicationInstance"
metadata:
  application: "clickstream-app"
  name: "clickstream-app-dev"
spec:
  cluster: "shadow-it"
  service-account: "sa-clickstream-dev"
  resources:
    - type: TOPIC
      name: "click."
      patternType: PREFIXED
    - type: GROUP
      name: "click."
      patternType: PREFIXED

Custom Deserializers​

Console's support for Custom Deserializers is finally here!
A picture is worth a thousand words:

Check our dedicated How-To guide Installing and Configuring Custom Deserializers.

Fixes 🔨​

• Fixed an issue with controller metrics in Monitoring when the Kafka cluster is using KRaft
• Added support for Broker, Connect, and ksqlDB id field and TLS authentication in the YAML configuration file and Environment variables. This implies you might have a duplicate Connect instance if you use a YAML file with an ID for your Connect cluster. Check the Environment Variables page for more details
• Added new configurations to tune indexing batching and parallelization.s
• Fixed an issue with Azure PostgreSQL preventing the Schema Registry page from displaying properly

Conduktor CLI 0.2.0​

Release date: 2024-04-22

Features​

• This release is only compatible with Console 1.23 or above
• Made CTL work with complex path
• Set CDK-CLIENT header to improve analytics

Fixes​

• Fix for adding prefix X- to custom header

Find out more.

Gateway 3.0.0​

Release date: 2024-03-20

This major release of the Gateway brings functionality around targeting your interceptors more specifically, adding additional data quality & filtering tools and a host of rework under the hood for improved reliability & robustness. This can be seen in the form of reworked authorization to more closely align with what you're used to in the existing Kafka world and a more intuitive experience when working with the enhanced functionality Conduktor brings in concentrated and alias topics.

Features​

Interceptor Targeting​

Interceptors can now target more specifically than the previous scopes of vcluster and username. They can now be targeted at the global, vcluster, group(new), or service account level. Below are some areas and examples where targeting interceptors brings great power in their flexibility.

Apply interceptors on groups, across several service accounts, without duplicating the interceptor​

On a given Kafka cluster, each application may have different policy requirements.

Applications could be part of an organization's domain (finance, HR, Sales, etc.) or grouped by another dimension, such as data sensitivity. Platform teams will want to manage rules that apply to multiple applications at a "group" level.

Override behavior at a more specific service account, or group level​

Rather than apply interceptors across a wider domain, they may want to zoom in and target a specific application to override the wider defaults.

Examples:

A project from a domain is more advanced and doesn't need the safeguarding protections applied to the wider group.

• They know how to size topics correctly and are allowed a higher limit on partitions for topic creation, than the rest of the group
• Everyone is required to have compression enforced by default, but for this specific team they are allowed to remove it to meet a low latency requirement

Data quality validation rules across fields, using CEL​

Validate data across fields using Common Expression Language. Before we could define rules for fields within a schema, a great way to ensure data quality catching the data before it hits the cluster. Now, we can relate fields to each other. We can bring together data quality and business rules within our schema.

An example for age and email checks in our schema:

{
  "fields": [
    {
      "name": "age",
      "type": "int",
      "minimum": 18
    },
    {
      "name": "email",
      "type": "string",
      "format": "email"
    }
  ],
  "metatadata": {
    "rules": [
      {
        "name": "old people",
        "expression": "age >= 40 && email.endsWith('yahoo.com')",
        "message": "yahoo.com emails are allow only for people older that 40"
      }
    ]
  }
}

Filter messages on topics, using CEL​

Topic filtering can now be done with a simple plugin rather than building yet another pipeline. Effortlessly tailor message filtering rules to your use cases, ensuring only the most relevant data reaches your consumers.

Similar to how we allow you to filter data using SQL, you can now use CEL. By leveraging CEL expressions, you gain the flexibility to filter messages based on various attributes such as record key, value, partition, timestamp, header, and offset, offering unparalleled control over your data consumption.

Suppose you want to filter messages where the timestamp is greater than a certain threshold and the record key matches a specific pattern. With the enhanced CEL topic filtering feature, achieving this becomes straightforward as posting a plugin with the config:

{
  "virtualTopic": "your-topic-name",
  "expression": "record.timestamp > 1616400000000 && record.key.startsWith('prefix_')"
}

Topic multiplexing enhancements​

Several enhancements have been made when working with concentrated topics for topic multiplexing. Concentration can now be achieved on the default vcluster, passthrough. UX has been adjusted from using patterns only in favor of concentration rules, which have a dedicated part of the API.

Alias topic enhancements​

Alias topics (dedicated to referencing another topic within your cluster, see the docs for more) have been reworked for a more intuitive experience. Alias topics no longer replace the physical topic during interactions, but are seen as another topic. This will help in use cases related to migration, when applications use different topic names, and when exposing more topics within vclusters.

Default vcluster rework​

The default vcluster, passthrough, now has users associated with it by default rather than being rejected. This behavior can be reverted through configuration; see the docs for more.

General fixes 🔨​

• Fixed an issue that was prefixing consumer group names with Gateway in certain virtual clusters
• Simplified the security protocol experience, dropping the need for GATEWAY_MODE(s) to be defined, instead using Kafka standard security protocols or DELEGATED security protocols. Refer to the docs for more
• Less noisy metrics
• Configuration topics are now prefixed with the clusterID to prevent unintentional
• The PUT HTTP verb is now supported throughout the API
• ARM build is now available for the distro and distro-less images, to provide more flexibility to your deployment machine choices

Conduktor CLI 0.1.1​

Release date: 2024-04-15

Features​

• Add uniform CLI help commands/flags descriptions
• Use alpine as base Docker image with custom user
• Added support for multiple failing apply to get a detailed summary

Conduktor CLI 0.1.0​

Release date: 2024-03-26

Conduktor CLI allows you to perform operations directly from your command line or a CI/CD pipeline to a Conduktor Console instance.

Features​

• Support for get, apply (upsert) and delete commands for the following Conduktor Console resources:
  • Application
  • ApplicationInstance
  • ApplicationInstancePermission
• Support for --dry-run on apply and delete
• Support completion that generates the autocompletion script for the specified shell
• Support for proxy auth using certificate and key
• Support ignore untrusted certificates environment variable
• Configurable environment variables

Find out more.

Console 1.21.1​

Release date: 2024-03-05

Fixes 🔨​

• Resolved a problem causing a blank screen after login in certain situations, preventing users from accessing Console.

Console 1.21.0​

Release date: 2024-02-26

Future Breaking Changes 💣​

New docker image name​

To clarify our product naming we have renamed the Console docker image to conduktor/conduktor-console.

We will publish newer versions using both names for the next three releases only. Please modify your installation to reflect this change in advance of us deprecating the name conduktor-platform.

docker pull conduktor/conduktor-console:1.21.0

Features ✨​

• ksqlDB
• Metrics available via Prometheus
• Smart tables for Connect and Schema Registry
• Add Local Users from the UI
• Fixes

ksqlDB​

Say hello to seamless integration with ksqlDB for you and your team on Conduktor Console.
Grant permission to whom can access the interface to create queries, setup new connections and visualise the existing connections.

Now you can:

• Browse ksqlDB clusters that are connected to Console
• Visualize all the currently running queries as well as write your own queries or executes statements
• Visualize and act on the running Streams (resulting from CREATE STREAM statements) with the Streams tab
• Visualize and act on the running Streams (resulting from CREATE TABLE statements) with the Tables tab
• Show all the Persistent and Push queries currently running on the ksqlDB Cluster with the Queries tab
• Execute Pull and Pull Queries (SELECT) and Statements (CREATE, DESCRIBE, DROP, ...) with the Editor tab

More info about kSQL is available on their website.

For more information checkout the docs.

Subscribe to metrics via the Prometheus endpoint​

Gain deeper insights into your system's performance with metrics now readily available via the Prometheus endpoint. No need for yet another system to monitor, seamlessly integrate metrics directly into your external log system in the Prometheus format, allowing for effortless monitoring and optimization of Conduktor within your systems.

You can monitor metrics such as under replicated partitions, total & failed connector tasks and consumer group lags. For the full list of available metrics checkout the docs.

Smart tables for Kafka Connect and Schema Registry subjects​

Get the answers you need quicker with the new tables. Sort by what matters, be that subject name, version count, latest version and more! For Connect there's all the usual suspects: source/sink, topics, the connect cluster and importantly the state (e.g. Failed, Paused). Quickly find which connectors are failing with just one click.

Choose what columns to hide the noise. Filter on name, tags and other resource metatdata such as consumer group state.

Add Local Users from the UI​

Don't have SSO ? Now you can add Users directly from the Users & Groups page in Settings, instead of modifying the config file and restarting the Console.

Fixes 🔨​

• Added support for complex union-type avro messages in Console Producer
• Fixed a blank screen issue after login due to case-sensitivity bug with email address
• Fixed an issue where Message Reprocessing didn't work after refreshing the page
• Resolved an issue with MSK role assumption
• Fixed an issue with custom certificates for Schema Registry and Kafka Connect
• Fixed several issues improving indexing performance on large clusters
• Increased cortex ingestion limits for large clusters
• Fixed an issue that occurs when Group: ACLs are present

Archive​

clusters:
  - id: ccloud
    name: Confluent Cloud ABCD
    bootstrapServers: lkc-abcd.europe-west1.gcp.confluent.cloud:9092
    properties: |
      security.protocol=SASL_SSL
      sasl.mechanism=PLAIN
      sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
        username='<CLUSTER-API-KEY>' \ 
        password='<CLUSTER-API-PASSWORD>';
    kafkaflavor: # Optional, for Service Accounts & Api Key support
      type: Confluent
      key: "<GLOBAL-API-KEY>"
      secret: "<GLOBAL-API-SECRET>"
      confluentEnvironmentId: "env-abcd"
      confluentClusterId: "lkc-abcd"
    schemaRegistry:
      url: http://abcd.europe-west1.gcp.confluent.cloud/
      security:
        username: <REGISTRY-API-KEY>
        password: <REGISTRY-API-PASSWORD>
    kafkaConnects:
      - id: kafka-connect-1
        name: Connect 1
        url: http://kafka-connect-1:8083/
        security:
          username: username
          password: password
    

-XX:+UseContainerSupport -XX:MaxRAMPercentage=80

curl -X GET http://localhost:8080/public/v1/groups/project-a/permissions -H "Authorization: Bearer {token}"
[
    {
        "resourceType": "Topic",
        "clusterId": "local",
        "topicPattern": "projectA-*",
        "permissions": [
            "topicConsume",
            "topicViewConfig"
        ]
    },
    {
        "resourceType": "ConsumerGroup",
        "clusterId": "local",
        "consumerGroupPattern": "projectA-*",
        "permissions": [
            "consumerGroupView"
        ]
    }
]

clusters:
  - id: eastern-horizon
    name: My Local Kafka Cluster
    color: '#0013E7'
    bootstrapServers: kafka:9093