Changelog
- Visit our Get Started page to test the latest version of Conduktor
- Submit your feedback and feature requests
- Receive updates on our future Releases via our Support Portal and select follow
Console 1.29.1
Release date: 2024-11-28
Fixes 🔨
- Fixed an issue where configuring Google OIDC without a
scope
was throwing an Exception.
Console 1.29.0
Release date: 2024-11-25
Breaking Changes 💣
Changes to Conduktor.io Labels
We have moved the conduktor.io
labels previously available on Connector and Topic resources to new locations.
This change impacts you if you:
- Update the Topic Catalog description of Topic resources via CLI/API
- Configure Connect automatic restart with the CLI/API
You are not impacted if you perform these actions through the UI.
We recognize this change breaches the API contract and have carefully considered its implications. We remain committed to minimizing breaking changes and will continue to do our best to ensure that such changes are rare and well-justified.
As we expand the number of Conduktor-related features, this separation reduces the risk of conflicts, simplifies resource management, and provides flexibility for future improvements:
- Labels used for sorting and filtering throughout the product
- Conduktor-specific annotations used to drive behaviors on the resources
Topic Resource
metadata.labels.'conduktor.io/description'
→metadata.description
metadata.labels.'conduktor.io/description.editable'
→metadata.descriptionIsEditable
Connector Resource
metadata.labels.'conduktor.io/auto-restart-enabled'
→metadata.autoRestart.enabled
metadata.labels.'conduktor.io/auto-restart-frequency'
→metadata.autoRestart.frequencySeconds
Their associated values have been automatically migrated under the new names.
Important Note for CLI Users
Applying YAML files with old conduktor.io
labels will fail in Conduktor Console 1.29. Be sure to update your YAML files to reflect the new labels.
Example error for outdated YAML:
$ conduktor apply -f topic.yaml
Could not apply resource Topic/click.event-stream.avro: Invalid value for: body (Couldn't decode key. at 'metadata.labels.conduktor.io/description')
Local Users Password policy update
Passwords for console Local Users configured through YAML and environment variables must comply with a new password policy. This change enforces the following password requirements:
- At least 8 characters in length
- Includes at least 1 uppercase letter, 1 lowercase letter, 1 number, and 1 special symbol
This change impacts you if:
- Your Admin password doesn't comply with the new password policy (
CDK_ADMIN_PASSWORD
) - You have local user configured through YAML or Env variables (
CDK_AUTH_LOCALUSERS_0_PASSWORD
)
Passwords set in existing installations that do not meet these requirements will prevent Console from starting, throwing a startup error in the logs like this:
2024-11-21T14:25:47,434Z [console] ERROR zio-slf4j-logger - zio.Config$Error$InvalidData: (Invalid data at admin: Password must contain at least 8 characters including 1 uppercase letter, 1 lowercase letter, 1 number and 1 special symbol)
Local Users previously created with the UI are not impacted.
Update the passwords in your YAML or environment variables to meet the new policy before upgrading.
Features ✨
Conduktor Chargeback: Track and Allocate Costs & Resource Consumption
This feature is in Beta and is subject to change as we enhance it further.
Conduktor Chargeback allows organizations to track and allocate costs & usage associated with Kafka resources to different teams or departments based on their data consumption and processing, facilitating cost accountability and management.
Check the dedicated Quickstart to get started with Chargeback.
Console Homepage
The cluster homepage have been redesigned to present you with the most useful information in one single view:
- The health of your Kafka Cluster with a few key metrics and graphs
- The state of Console Indexing modules for this Kafka Cluster
- Quick access to your most recently viewed resources
Consumer Group pages overhaul
Consumer group details page is now organized in a way that helps understand the status of your Consumer Group more easily:
- Topics tab shows the Consumer Group info grouped by its subscribed Topics
- Members tab shows the Consumer Group info grouped by its active members
Both Topics and Members lists can be explored further down to individual member/topic-partition assignments.
On top of that graphs are now directly available in the resource page for Lag and Time Lag, as well as a dedicated tab to manage Alerts.
Self-Service Topic Catalog visibility
You can now choose which Topics should be visible in the Topic Catalog by annotating their YAML.
---
apiVersion: kafka/v2
kind: Topic
metadata:
cluster: shadow-it
name: click.event-stream.avro
catalogVisibility: PUBLIC # or PRIVATE
spec:
...
It is also possible to change the default Topic Catalog visibility of all Topics of an Application Instance directly
Check the associated documentation.
Self-Service New Topic Policy Allowed Keys
We have added a new constraint AllowedKeys
to our Self-Service Topic Policy that restricts the properties that can be configured on a Topic.
---
# Limits the Topic spec.configs to only have retention.ms and cleanup.policy keys
apiVersion: self-service/v1
kind: TopicPolicy
metadata:
name: "generic-dev-topic"
spec:
policies:
spec.configs:
constraint: AllowedKeys
keys:
- retention.ms
- cleanup.policy
This works in conjunction with existing constraints and ensures your Application Teams will only define properties that are allowed by the Central Team.
Read more about our Topic Policy constraints.
More Audit Log CloudEvents into Kafka
We have made more events available for the Audit Log Publisher:
- IAM.User.Logout
- IAM.User.Login
- Kafka.ConsumerGroup.Duplicate
- Kafka.ConsumerGroup.Delete
- Kafka.ConsumerGroup.Update ( when we reset the offset of the consumer group )
A full list of all the exported audit log event types is published on the Audit Log page.
Expanded Terraform Provider: Kafka Cluster, Schema Registry, Kafka Connect
We've expanded the scope of our Terraform provider, you can now create additional resources: Kafka cluster with schema registry, and Kafka connect clusters using Terraform. With this version also comes some additional small fixes as requested by the community, see the dedicated provider releases page for the full list.
All examples are available in our provider repo such as the below snippet for a Confluent Kafka cluster and schema registry (with mTLS) definition.
resource "conduktor_kafka_cluster_v2" "confluent" {
name = "confluent-cluster"
labels = {
"env" = "staging"
}
spec {
display_name = "Confluent Cluster"
bootstrap_servers = "aaa-aaaa.us-west4.gcp.confluent.cloud:9092"
properties = {
"sasl.jaas.config" = "org.apache.kafka.common.security.plain.PlainLoginModule required username='admin' password='admin-secret';"
"security.protocol" = "SASL_PLAINTEXT"
"sasl.mechanism" = "PLAIN"
}
icon = "kafka"
ignore_untrusted_certificate = false
kafka_flavor = {
type = "Confluent"
key = "yourApiKey123456"
secret = "yourApiSecret123456"
confluent_environment_id = "env-12345"
confluent_cluster_id = "lkc-67890"
}
schema_registry = {
type = "ConfluentLike"
url = "https://bbb-bbbb.us-west4.gcp.confluent.cloud:8081"
ignore_untrusted_certificate = false
security = {
type = "SSLAuth"
key = <<EOT
-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----
EOT
certificate_chain = <<EOT
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
EOT
}
}
}
}
Quality of Life improvements
- Improved the performance of the Automatic deserializer
- Improved the performance of the Schema Registry indexing process
- Added support for Google Cloud Identity group claims
- Added License expiry warning in the UI when there is less than 30 days left
Fixes 🔨
- Fixed an issue where Custom Deserializers weren't working as expected
- Fixed an issue where the ManageClusters permission wasn't working as expected
- Fixed an issue that prevented creating a KafkaCluster and a Topic on that newly declared KafkaCluster in a single CLI apply command
- Fixed
/health/readiness
endpoint to return HTTP 503 when the Postgres DB is down - Fixed an issue where the Message Count wasn't updated to 0 when emptying a topic
- Fixed an issue where the Pause/Resume button wasn't visible when a connector was in the Failed state
- Fixed an issue where the Topic creation failure reason wasn't shown in the UI
- This helps understand why Topic Creation is rejected (useful for Gateway and Self-Service Topic Policies), and how to modify the topic create request to meet the policy requirements
Gateway 3.4.0
Release date: 2024-11-15
Upcoming Breaking change: Local Users 💣
This breaking change only impacts Local Gateway service accounts generated through our token endpoints:
POST /admin/username/{username}
POST /admin/vclusters/v1/vcluster/{vcluster}/username/{username}
If you are using Gateway services accounts from OIDC, mTLS, Delegated Kafka, you are not impacted.
Today, the token as the password for local Gateway service accounts contains all the necessary information. As a result, the SASL username is not used during the authentication phase.
In release 3.5.0, we will strictly enforce that the username and the token matches. This will help reduce inconsistencies and avoid unexpected behavior.
For this release 3.4.0, we'll only raise the following warning in the logs:
2024-08-27T18:15:29 [WARN] - Inconsistency detected for plain authentication. Username applicationA is not consistent with validated token created for application-A. SASL configuration should be changed accordingly.
Features ✨
Correct Offsets on Concentrated Topics
Concentrated Topics were reporting the offsets of the underlying backing topic. This caused Lag and Message Count metrics to be invalid.
Correct offsets can now be enabled per ConcentrationRule.
---
kind: ConcentrationRule
metadata:
name: myapp-concentrated
spec:
pattern: myapp-.*
physicalTopics:
delete: myapp-concentrated
autoManaged: false
offsetCorrectness: true
This feature is experimental and comes with a number of limitations that are important to understand beforehand.
General fixes 🔨
- Fixed an issue impacting live consumption from concentrated topics within Console
- Fixed an issue with upserts in API V2 relating to service accounts (reporting updated when the status should be not changed)
- Fixed an issue related to Kafka 3.7 client support, ensuring topic id's for alias and concentrated topics are distinct from their underlying topics
- Fixed an issue whereby audit logs were not being captured during cluster switching
- Fixed an issue with SQL topics when parsing topic names containing "-"
Known issues
- We are aware of an issue with
kcat
when the new environment variableGATEWAY_MIN_BROKERID
is not aligned with the first BrokerId of your Kafka cluster- As a workaround, you can either define
GATEWAY_MIN_BROKERID
to your first Kafka BrokerId or usekcat
with the-E
flag
- As a workaround, you can either define
Console 1.28.0
Release date: 2024-10-14
Features ✨
Conduktor SQL
This feature is in Beta and is subject to change as we enhance it further.
It's currently only available to Console Admins, and will be made available for all users when integrated with our security model (i.e. RBAC, Data Masking).
Index data from Kafka topics in a database to enable data to be queried from the UI, API or CLI using SQL.
This allows you to troubleshoot, sample, analyze, aggregate and join data through:
- Querying Kafka message data
- Querying Kafka metadata (such as the offset, partition and timestamp)
Read the dedicated guide on configuring SQL.
Query through the UI
Query through the API & CLI
CLI (upgrade to v0.3.1):
conduktor sql 'select * from "kafka-cluster-dev_customer_orders"' -n 2
API:
curl \
--header "Authorization: $token" \
--request POST 'localhost:8080/api/public/sql/v1/execute?maxLine=2' \
--data 'select * from "kafka-cluster-dev_customer_orders"'
Important information regarding SQL
To use the feature there is a dependency on provisioning a new database. As a user, you have the choice of which topics you wish to index. Learn more about this and how to configure SQL using the dedicated guide.
We encourage you to use this feature in non-production environments and give us feedback.
Monitoring improvements
We are migrating our Monitoring dashboards into their respective resource pages for a more integrated experience.
This migration will happen over the next few releases with our objective to remove the existing, generic Monitoring pages:
- Overview will be refactored into Home page
- Cluster Health dashboards and alerts will move under Brokers page
- Topic monitoring will be integrated with Topics page
- Apps monitoring will be integrated with Consumer Groups pages
- Alerts will be integrated as tabs in all the resource pages, similar to the recent changes for Kafka Connect
For this 1.28.0 release we are migrating the Topic monitoring and Cluster Health pages.
Topic Monitoring
The 3 existing graphs have been moved to the Topic details. We have also added a new graph to track the number of records in the topic.
- Produce Rate and Consume Rate
- Disk Usage
- Records (new)
Cluster Health
The charts and alerts are now available under the Brokers page with cleaner graphs.
- Produce Rate and Consume Rate
- Disk Usage
- Partitions Count
- Offline, Under Replicated and Under Min ISR Partitions
We have removed two metrics that were not always calculated correctly since the removal of the JMX integration back in release 1.15 (May 2023).
- Active Controller Count
- Unclean Leader Election
New CLI/API resource Alert
Alerts can now be created via the API or CLI in addition to the UI.
See below for example config, and check the Alerts documentation for more details.
---
apiVersion: console/v2
kind: Alert
metadata:
cluster: local-julien
name: my-alert
spec:
type: TopicAlert
topicName: wikipedia-parsed-DLQ
metric: MessageCount
operator: GreaterThan
threshold: 0
Starting today, we recommend you use the new alerts available under Brokers and Topics pages.
We do not plan to migrate existing alerts to the new Alert model.
Original alerts will be removed in the near future in favor of the new ones.
We'll let you know a few releases in advance.
If you have a large number of alerts configured and need some help, we're happy to help, please get in touch with our support.
Shareable Filters
Filters in the Topic Consume view are now shareable. This brings a number of benefits:
- Improved collaboration: Share pre-defined views to ensure users are looking at the same subset of data
- Time savings: Speed up troubleshooting and analysis with repeatable filters that share the same or similar criteria
- Consistency and accuracy: Standardized filters across teams and departments reduce the risk of errors or discrepancies that can occur when individuals manually create filters
After you've finished configuring filters on a topic, you now have an option to save the filter either as a Private or an Organization filter.
Anyone can then load Organization filters from the dedicated section.
Tags Become Labels
With the introduction of the Self-service resource manifests, we brought customers a means to annotate all their resources with labels. Labels are more structured than the existing Conduktor tags, thereby allowing for more precise filtering capabilities, as can be seen in the Topic Catalog.
In this release, we'll perform an automatic migration from Tags to Labels.
Tags written with the naming convention <key>/<value>
will automatically be added as similar labels:
<key>: <value>
If there is a conflict such as; a topic containing tags with the same key, that already has the target label, or is not written with this naming convention, then they will be created with a tag-
prefix as follows:
tag-<value>: true
Here's an example of how tags will be migrated into labels:
# Tags defined on topic:
- format/avro
- project/supplychain
- team/delivery
- color/blue
- color/red
- wikipedia
- non-prod
# Resulting topic labels after migration
labels:
format: avro
project: supplychain
team: delivery
tag-color/blue: true # Because conflict on "color"
tag-color/red: true # Because conflict on "color"
tag-wikipedia: true # Because wikipedia is not a key value pair
tag-non-prod: true # Becuase non-prod is not a key value pair
The Topic list and Topic details page have been modified to use labels instead of tags.
We plan to bring this capability on all resources (Connectors, Service Accounts, Consumer Groups, ...) over the next few releases.
Let us know which resource you would like to see covered first.
Publish AuditLog CloudEvents into Kafka
It is now possible to publish Console Audit Log events into a Kafka topic directly for any further use you may have for them, such as maintaining your own audit trail in other systems.
The exportable audit log events have more detail compared to the current UI events, providing additional information about the event that has taken place.
The events conform to the CloudEvents specification, a vendor-neutral format that follows the following structure:
{
"specversion" : "1.0",
"type" : "com.github.pull_request.opened",
"source" : "https://github.com/cloudevents/spec/pull",
"subject" : "123",
"id" : "A234-1234-1234",
"time" : "2018-04-05T17:31:00Z",
"comexampleextension1" : "value",
"comexampleothervalue" : 5,
"datacontenttype" : "text/xml",
"data" : "<much wow=\"xml\"/>"
}
An example Conduktor event would look like:
{
"source": "//kafka/kafkacluster/production/topic/website-orders",
"data": {
"eventType": "Kafka.Topic.Create",
// Additional event specific data...
"metadata": {
"name": "website-orders",
"cluster": "production"
}
// Additional event specific metadata...
},
"datacontenttype": "application/json",
"id": "ad85122c-0041-421e-b04b-6bc2ec901e08",
"time": "2024-10-10T07:52:07.483140Z",
"type": "AuditLogEventType(Kafka,Topic,Create)",
"specversion": "1.0"
}
Specify the target Kafka cluster and topic using the environment variables CDK_AUDITLOGPUBLISHER_CLUSTER
& CDK_AUDITLOGPUBLISHER_TOPICNAME
and events will start being produced to the destination topic.
A full list of all the exported audit log event types is published on the Audit Log page.
Logging API
Adjust the log level of Console without requiring a restart. We've added a new API endpoint to support targeted changes to log levels dynamically. Check the associated documentation for the full list of capabilities.
curl -X PUT 'http://localhost:8080/api/public/debug/v1/loggers/io.conduktor.authenticator/DEBUG' \
-H "Authorization: Bearer $API_KEY"
Quality of Life improvements
- Updated design and color theme
- Added navigation breadcrumb
- Enhanced error messages throughout the product
- Improved the connector 90 days heatmap
- Declaring an ApplicationInstance with resources ending in
*
will now fail with this error messageCould not apply resource ApplicationInstance/my-app-inst: resource name 'appA-*' is not allowed. Use name 'appA-' with patternType PREFIXED instead
Fixes 🔨
- Fixed an issue with Topic Policy constraint Range where
max
value wasn't inclusive andmin
could greater thanmax
- Fixed an issue where Topic Policies were not enforced on Topic configuration changes in Console
- Added an error message when using the copy to clipboard button (for API Keys for instance) fails
- Added checks on local user creation emails
- Added new optional environment variable
CDK_SSO_OAUTH2_0_OPENID_METADATADOCUMENT
to modify the default discovery .well-known end-point - Fixed an issue where Avro messages using logical type UUID couldn't be deserialized properly
- Fixed an issue with Cluster configuration page requiring
platform.certificates.create
permission to perform the TLS check - Fixed an issue with "Remove user from group" button which is now disabled for users added by external group mapping
- Prevented the deletion of a group when it is owner of an Application
- Fixed an issue with the "New version" button in the banner that was still showing despite being on the latest version
- Fixed an issue where connections to the AWS glue schema registry would disconnect after a certain time and struggle to reconnect
- Fixed an issue where the "Reprocess message" feature was converting empty string headers to null value
- Fixed all critical and high CVE in
console-cortex
image - Fixed an issue with the metric
under_replicated_partitions
when topics haveconfluent.placement.constraints
property
Gateway 3.3.2
Release date: 2024-10-07
Fixes 🔨
- Fixed an issue where the Producer Rate Limiting Policy would not honor its configuration.
Console 1.27.1
Release date: 2024-09-25
Fixes 🔨
- Improved performance of RBAC resolution when there is a large number of permissions
Gateway 3.3.1
Release date: 2024-09-25
General fixes 🔨
- Fixed an issue where Gateway would close the client connection upon receiving certain API Keys in parallel of the initial Metadata Request
Console 1.27.0
Release date: 2024-09-14
Features ✨
Kafka Connect Configuration Wizard
We are very excited to introduce our Configuration Wizard for Kafka Connect which is taking full advantage of the Kafka Connect Validate API:
- Form is generated with structured configuration groups
- Much nicer error handling, attached to each individual field
- Embedded documentation that helps you understand which fields are required and what are their expected and default values
- Ability to toggle advanced configuration to visualize only the most important fields
- Ability to switch seamlessly between Form View and JSON View at any time.
Give it a try and let us know what you think!
Alerts for Kafka Connect
On top of the Kafka Connect Graphs we shipped last release, we now have added the ability to create alerts on them.
Self-service: Limited Ownership mode
To help organizations transition to Self-service more easily, we have added a new attribute on ApplicationInstance to let Platform Teams decide the level of autonomy to give to Application Teams.
- ApplicationInstance resources configured with
ownershipMode: ALL
, which is the default, delegates all permissions related to that resource to the Application Team - ApplicationInstance resources configured with
ownershipMode: LIMITED
delegates only a subset of the available permissions to the Application Team
This is especially useful if you already have a centralized repository and existing workflow for Topic creation. You may want to provide Self-service capabilities while still forcing your Application Teams to go through your pipeline for Topic Creation, instead of Self-service.
Quality of Life improvements
- Self-service: External Group Mapping is now available for ApplicationGroup
- The Login page now steers users towards their OIDC provider rather than basic auth when OIDC is enabled
Fixes 🔨
- Fixed an issue on Consumer group reset offset with the ToDatetime strategy
- Fixed an issue with Console indexing that could occur when deleting and recreating subject
- Fixed a recent regression with default replication factor when creating a topic
- Fixed a recent regression with Broker feature "Similar config" calculation
- Fixed a UI issue when Application Instance was created without any resources
- Fixed several issues around Microsoft Teams Integration to support Teams Workflow webhooks (Step by step guide)
- Fixed Kafka Connect client to use HTTP Proxy JVM configuration
- Switching Kafka cluster from the Topic details page now redirects to the Topic List
- Console doesn't override the client.id property anymore
Deprecation Warning: Upcoming migration from Tags to Labels 💣
With the introduction of the Self-service resource manifests, we brought customers a means to annotate all their resources with labels. Labels are more structured than the existing Conduktor tags, thereby allowing for more precise filtering capabilities, as can be seen in the Topic Catalog.
In an upcoming release, we'll perform an automatic migration from Tags to Labels.
Tags written with the naming convention <key>/<value>
will automatically be added as similar labels:
<key>: <value>
If there is a conflict such as; a topic containing tags with the same key, that already has the target label, or is not written with this naming convention, then they will be created as follows:
tag-<value>: true
Here's an example of how tags will be migrated into labels:
# Tags:
- format/avro
- project/supplychain
- team/delivery
- color/blue
- color/red
- wikipedia
- non-prod
# Result
labels:
format: avro
project: supplychain
team: delivery
tag-color/blue: true # Because conflict on "color"
tag-color/red: true # Because conflict on "color"
tag-wikipedia: true
tag-non-prod: true
⚠️ Conduktor can help you rename tags through Customer Support
Between now and the migration, we can help you rename your tags for a smooth transition to labels.
Contact us as soon as possible if you would like support.
We are aware of a critical CVE - CVE-2024-41110 - coming from a dependency of prometheus on the console-cortex
image. This CVE is related to prometheus docker metric scraping, which is not used by Conduktor.
Regardless, as soon as the prometheus team fix this issue, it will be patched immediately by Conduktor.
Gateway 3.3.0
Release date: 2024-09-05
Upcoming Breaking change 💣
This breaking change only impacts Local Gateway service accounts generated through our token endpoints:
POST /admin/username/{username}
POST /admin/vclusters/v1/vcluster/{vcluster}/username/{username}
If you are not using Local Gateway services accounts (OIDC, mTLS, Delegated Kafka), you are not impacted.
Today, the token as the password for local Gateway service accounts contains all the necessary information. As a result, the SASL username is not used during the authentication phase.
In an upcoming release, we will strictly enforce that the username and the token matches. This will help reduce inconsistencies and avoid unexpected behaviors.
This breaking change is due for release 3.5.0.
For this release 3.3.0, and next product release 3.4.0, we'll only raise the following warning in the logs:
2024-08-27T18:15:29 [WARN] - Inconsistency detected for plain authentication. Username applicationA is not consistent with validated token created for application-A. SASL configuration should be changed accordingly.
Features ✨
- New V2 APIs and CLI support
- Support for HTTPS APIs
- Better UX for ACLs and superUsers
- Encryption Enhancements and Support Clarification
New V2 APIs and CLI support
We’re excited to introduce our new Gateway API, designed for seamless integration with our CLI. This update allows you to deploy Gateway resources using infrastructure-as-code with straightforward, clearly defined concepts:
- Interceptor
- GatewayServiceAccount
- GatewayGroup
- ConcentrationRule
- AliasTopic
- VirtualCluster
Check the CLI reference to get started, and the resources reference for more information on each concept.
---
apiVersion: gateway/v2
kind: GatewayGroup
metadata:
name: groupB
spec:
members:
- name: user1
- name: user2
---
apiVersion: gateway/v2
kind: Interceptor
metadata:
name: enforce-partition-limit
scope:
group: groupB
spec:
pluginClass: io.conduktor.gateway.interceptor.safeguard.CreateTopicPolicyPlugin
priority: 100
config:
numPartition:
action: BLOCK
max: 9
min: 9
topic: .*
$ conduktor apply -f gateway.yml
GatewayGroup/groupB: Created
Interceptor/enforce-partition-limit: Created
$ conduktor delete GatewayGroup groupB
The group groupB is still used by the following interceptor(s): enforce-partition-limit
Note: API V1 is still available, but we recommend that new users and those with simple Gateway configurations begin using the V2 API as soon as possible. We will announce a deprecation plan in the coming weeks and notify you in advance of which Gateway version will be the last to support the V1 APIs.
Support for HTTPS APIs
It is now possible to configure HTTPS and mTLS authentication on the Gateway HTTP APIs. Check the HTTP section of the Environment Variables page for more details.
Better UX for ACLs and superUsers
To coincide with the clearly defined concepts established in API V2, we are making changes to ACLs management in Gateway.
- ACLs and Super Users on the Gateway (excluding Virtual Clusters) must be configured through Environment Variables.
- ACLs and Super Users on Virtual Clusters must now be driven explicitly through API/CLI.
Enable ACLs for Gateway (excl. Virtual Clusters)
Configure both environment variables:
GATEWAY_ACL_ENABLED=true # default false
GATEWAY_SUPER_USERS=alice,bob
If GATEWAY_SUPER_USERS
is not set, it will default to GATEWAY_ADMIN_API_USERS
for backward compatibility.
Enable ACLs for Virtual Clusters
Note that if you are migrating from an older version of Gateway, the migration will automatically generate existing Virtual Clusters as configuration.
- The automation will derive the boolean value
aclEnabled
from the previously usedGATEWAY_ACL_STORE_ENABLED
variable. - The migration will not populate the
superUsers
list automatically, so this must be addressed as part of your migration.
Example configuration:
---
apiVersion: gateway/v2
kind: VirtualCluster
metadata:
name: "mon-app-A"
spec:
aclEnabled: "true" # defaults to false
superUsers:
- username1
- username2
Encryption Enhancements and Support Clarification
Field-Level Encryption: Preserving Message Format to Enhance Usability
When applying field-level encryption prior to 3.3.0
, the encryption plugin would convert the message to JSON, and re-apply the schema format when the message was read back through the decryption plugin.
In Gateway 3.3.0
, we now preserve the schema format for Avro messages - meaning the same schema is used in the backing topic, and the data can be read directly from Kafka or without the decryption plugin at all.
Read more about this change to the default behaviour, and how to configure it.
Fields which cannot be encrypted in-place (effectively any non-string field) have their encrypted value placed in the headers, and the field itself is given a default masking value. The default values are clarified below:
Field Type | Default Value in 3.3.0 |
---|---|
Integer | Int MIN_VALUE |
Long | Long MIN_VALUE |
Float | Float, MIN_VALUE |
Double | Float MIN_VALUE (float again here due to some serdes behaviour) |
byte[] | "********" as bytes |
fixed[] | every byte filled with charater "*" |
boolean | false |
Note that the same default values are now used across all relevant plugins when manipulating a non-string field - Data Masking, Partial Decrypt, and Encrypt on Fetch.
Attempt to apply encryption to a message more than once will now fail
If any of the encryption headers are detected in a message when encryption is about to be applied, then the encryption operation will fail. This is because applying encryption twice (or more) is currently not reversible.
Deprecated support for Schema Based (tag) encryption with Protobuf
Note this is no longer supported, and the Gateway will now throw an exception if the encryption plugin attempts to apply schema (tag) based processing to a Protobuf message.
Note that any data previously written in this mode can still be read back - as the decrypt does not use the schemas at all, rather it uses the message header to know what was encrypted.
General fixes 🔨
- Large double values (where > Float Max) are now supported in field-level encryption for Avro and Protobuf
- Bytes and fixed fields now properly supported in field-level encryption for Avro
- Avro unions of two or more values (rather than just a value and a null) are now supported in field-level encryption for Avro
- Schema (tag) based encryption now checks and fails if its config is invalid
- It is not possible to encrypt the headers which the encryption plugin uses to manage its decryption process (as this would render the data unrecoverable)
- Improved log messages for Interceptors that reject actions, such as TopicPolicyPlugin
- Several improvements to the LargeMessage & LargeBatch Interceptors
- Fixed an issue where KCache topic initialization would fail silently and leave Gateway in an unusable state
- Added a new Environment Variable
GATEWAY_MIN_BROKERID
(default 0) that allows for determinist mapping of brokers and ports - Improved network stability during Gateway scaling or Kafka topology changes
- Added support for overriding Kafka Producer properties used for Audit Log topic with
GATEWAY_AUDIT_LOG_KAFKA_
environment variables - Removed metric
gateway.brokered_active_connections
. This was equal to portCount with port mapping and always 1 in host mapping - Changed metric
gateway.request_expired
tags: nodeHost/nodePort are replaced by nodeId/clusterId - Fix default value for
GATEWAY_UPSTREAM_THREAD
config. The new intended default (number of CPU) previously was (2 x number of CPU). - Fixed an issue with
GATEWAY_ADVERTISED_SNI_PORT
that wasn't working properly - Add log level for io.confluent packages in default log configuration
- Add default value to non mandatory configruation value for min and max bytes in FetchPolicyInterceptor
- Fix an issue with Concentrated Topics creation with Redpanda
Known issues
- We are aware of an issue with
kcat
when the new environment variableGATEWAY_MIN_BROKERID
is not aligned with the first BrokerId of your Kafka cluster.- As a workaround, you can either define
GATEWAY_MIN_BROKERID
to your first Kafka BrokerId or usekcat
with the-E
flag
- As a workaround, you can either define
- It is not possible to add Service Accounts to GatewayGroups using API V2 unless they are previously declared as GatewayServiceAccount.
- This is not a wanted behavior, especially for OAuth or Delegated Kafka Authentication where declaring a GatewayServiceAccount should not be needed. We'll address this issue in a follow-up release
- API V1 (user-mapping) is not impacted
- If you perform a rolling upgrade to 3.3.0, Gateway nodes in earlier versions will show the following error in the logs:
[ERROR] [KafkaCache:1007] - Failed to deserialize a value org.apache.avro.AvroTypeException: Expected field name not found: clusterId
- This is fine and will not cause any further problems
- If you use Virtual Clusters and ACLs: After updating to 3.3.0, you must manage VirtualCluster's ACL and superUsers through V2 API.
Gateway 3.2.2
Release date: 2024-08-28
Upcoming Breaking change 💣
This breaking change only impacts Local Gateway service accounts generated through our token endpoints:
POST /admin/username/{username}
POST /admin/vclusters/v1/vcluster/{vcluster}/username/{username}
If you are not using Local Gateway services accounts (OIDC, mTLS, Delegated Kafka), you are not impacted.
Today, the token as the password for local Gateway service accounts contains all the necessary information. As a result, the SASL username is not used during the authentication phase.
In an upcoming release, we will strictly enforce that the username and the token matches. This will help reduce inconsistencies and avoid unexpected behaviors.
This breaking change is due for release 3.5.0.
For this hotfix release 3.2.2, and next product releases 3.3.0 and 3.4.0s, we'll only raise the following warning in the logs:
2024-08-27T18:15:29 [WARN] - Inconsistency detected for plain authentication. Username applicationA is not consistent with validated token created for application-A. SASL configuration should be changed accordingly.
General fixes 🔨
- Fixed a severe authentication issue with Gateway generated tokens that could lead to a different user being authenticated, effectively causing elevated privileges under certain conditions.
- Fixed an issue where
GATEWAY_SNI_HOST_SEPARATOR
couldn't be set to the value-
- Fixed an issue where
GATEWAY_SNI_HOST_SEPARATOR
wasn't properly taken in account - Fixed an issue with
GATEWAY_ADVERTISED_SNI_PORT
that wasn't working properly
Console 1.26.0
Release date: 2024-08-14
We are aware of a critical CVE - CVE-2024-41110 - coming from a dependency of prometheus on the console-cortex
image. This CVE is related to prometheus docker metric scraping, which is not used by Conduktor.
Regardless, as soon as the prometheus team fix this issue, it will be patched immediately by Conduktor.
Features ✨
- Manage Connectors using the CLI
- Self-service support for Connectors
- Enhanced UI & Alerts for Kafka Connect
- Quality of Life improvements
- Deprecation Warning: Upcoming migration from Tags to Labels
Manage Connectors using the CLI
Continuing with the Infra-as-code approach, we are happy to introduce CLI support for Connectors, providing an efficient and automated way to manage your Kafka Connect resources.
---
apiVersion: kafka/v2
kind: Connector
metadata:
connectCluster: kafka-connect
name: click.my-connector
labels:
conduktor.io/auto-restart-enabled: true
conduktor.io/auto-restart-frequency: 600
spec:
config:
connector.class: 'org.apache.kafka.connect.tools.MockSourceConnector'
tasks.max: '1'
topic: click.pageviews
Self-service support for Connectors
Application Teams can now manage their Connectors with Self-service.
From now on, you can grant ownership to connectors on Self-service Application Instance.
---
apiVersion: self-service/v1
kind: ApplicationInstance
metadata:
application: "clickstream-app"
name: "clickstream-dev"
spec:
cluster: "shadow-it"
serviceAccount: "sa-clicko"
resources:
- type: CONNECTOR
connectCluster: shadow-connect
patternType: PREFIXED
name: "click."
Enhanced UI & Graphs for Kafka Connect
We have revisited the Kafka Connect UI in multiple ways to improve your experience:
- Connect Cluster selection screen with a preview of Connector status
- New graphs demonstrating the state of your Connector over time
Support for High Availability (HA) Console
Multiple Console instances can now be deployed in parallel to achieve high availability.
This applies to the deployment of conduktor-console
, while conduktor-console-cortex
is currently limited to a single instance. The design ensures minimal impact on the cluster by assigning only one instance to handle the indexing of Kafka data used for performance monitoring.
Quality of Life improvements
- The checkbox to skip TLS verification is now always visible
- The YAML for Topic object now allows number in
spec.configs
. Previously it was mandatory to quote all numbers. - Self-service Topic Policies are now visible in the UI
Fixes 🔨
- Topic Policies from Self-service are now properly enforced from the UI, as well as both the API and CLI
- Fix Kafka Connect Cluster list showing invalid number of running tasks
Deprecation Warning: Upcoming migration from Tags to Labels 💣
With the introduction of the Self-service resource manifests, we brought customers a means to annotate all their resources with labels. Labels are more structured than the existing Conduktor tags, thereby allowing for more precise filtering capabilities, as can be seen in the Topic Catalog.
In an upcoming release, we'll perform an automatic migration from Tags to Labels.
Tags written with the naming convention <key>/<value>
will automatically be added as similar labels:
<key>: <value>
If there is a conflict such as; a topic containing tags with the same key, that already has the target label, or is not written with this naming convention, then they will be created as follows:
tag-<value>: true
Here's an example of how tags will be migrated into labels:
# Tags:
- format/avro
- project/supplychain
- team/delivery
- color/blue
- color/red
- wikipedia
- non-prod
# Result
labels:
format: avro
project: supplychain
team: delivery
tag-color/blue: true # Because conflict on "color"
tag-color/red: true # Because conflict on "color"
tag-wikipedia: true
tag-non-prod: true
⚠️ Conduktor can help you rename tags through Customer Support
Between now and the migration, we can help you rename your tags for a smooth transition to labels.
Contact us as soon as possible if you would like support.
Gateway 3.2.1
Release date: 2024-07-31
General fixes 🔨
- Fixed an issue when either
GATEWAY_ACL_ENABLED
orGATEWAY_ACL_STORE_ENABLED
was enabled resulting in ACLs being enforced in undesirable scenarios.
Gateway 3.2.0
Release date: 2024-07-19
Breaking Changes 💣
Two new backing topics are required for Gateway
In the next release (3.3), we'll bring a new API as well as support in the Conduktor CLI to manage Gateway concepts using infra-as-code approach.
In preparation for this upcoming release, we are replacing some weakly-defined concepts in favor of strongly-defined concepts. The following are now clearly captured in the topics mentioned below:
- Virtual Clusters that existed only through creation of UserMappings or Interceptors targeting
- GatewayGroups that existed only through UserMappings
As a result, 2 new topics will now be created once you upgrade to Gateway 3.2:
_conduktor_gateway_vclusters
_conduktor_gateway_groups
If you are happy with the default names, you have nothing to do. If you want to control the name of those topics, use the 2 new environment variables:
GATEWAY_VCLUSTERS_TOPIC
GATEWAY_GROUPS_TOPIC
Check the associated Documentation for more information.
Changes to ACL support on Gateway
With Gateway 3.1 we removed our dedicated ACL interceptor in favor of a new environment variable GATEWAY_ACL_STORE_ENABLED
. This variable was enabling ACLs in all scenarios, whether you used Virtual Clusters or not.
Changes for Gateway 3.2
With Gateway 3.2, we are adding a new environment variable GATEWAY_ACL_ENABLED
and modifying the behavior of the existing variable GATEWAY_ACL_STORE_ENABLED
.
From now on, the variables works as follow:
Environment Variable | Description | Default |
---|---|---|
GATEWAY_ACL_ENABLED | Enable ACLs on the Gateway excluding Virtual Clusters | "false" |
GATEWAY_ACL_STORE_ENABLED | Enable ACLs on Virtual Clusters only | "false" |
Preview for Gateway 3.3
In the next release, we will enhance ACLs to restore and expand the full set of features available before version 3.1. This will be achieved through the introduction of a CLI and APIs, making concepts like VirtualCluster first-class citizens.
Enable ACLs for Gateway (excl. Virtual Clusters)
Configure both environment variables:
GATEWAY_ACL_ENABLED=true
GATEWAY_SUPER_USERS=alice,bob
Enable ACLs for Virtual Clusters
---
apiVersion: gateway/v2
kind: VirtualCluster
metadata:
name: "mon-app-A"
spec:
aclEnabled: true # defaults to false
superUsers:
- username1
- username2
This will effectively render GATEWAY_ACL_STORE_ENABLED
obsolete.
General fixes 🔨
- Fixed an issue with Field-level Avro encryption/decryption relating to numeric fields:
- When using partial decryption with Avro schema registry, any numeric values (int, long, float, double) that are not being decrypted will instead be masked with the minimum (most negative) value for the numeric type
- This is to ensure the field is compliant with the original type in the Avro schema
- Fixed an issue with the ClientIdRequired Policy that wasn't properly overriding the ClientId
- Fixed an issue to ensure all active connections are closed, and clients transition quicker to the new cluster during cluster switching
Console 1.25.1
Release date: 2024-07-23
Breaking Changes 💣
New docker image name
We have renamed the Console docker image to conduktor/conduktor-console
to clarify our product naming.
Please modify your installation to reflect this change as we will now stop publishing a conduktor/conduktor-platform
image.
docker pull conduktor/conduktor-console:1.25.1
Features ✨
- Conduktor Console IaC Compatible
- Shareable Message Page
- Large Messages Support
- Topic Catalog Details Page
- Audit Last Activity of Users
- Quality of Life improvements
Conduktor Console IaC Compatible
Console is now able to be fully deployed through an IaC approach with the following additions to Console 1.25 and CLI 0.2.7.
Manage Cluster Connections
Manage your Console resource lifecycle with the addition of the KafkaCluster, KafkaConnectCluster and KsqlDBCluster objects to our IaC approach using the Conduktor CLI.
Checkout the example below and find the full definition at Console Resources Reference documentation.
---
apiVersion: console/v2
kind: KafkaCluster
metadata:
name: cloud-kafka
spec:
displayName: "Cloud Kafka"
icon: "kafka"
color: "#000000"
bootstrapServers: "localhost:9092"
properties:
sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret";
security.protocol: SASL_SSL
sasl.mechanism: PLAIN
schemaRegistry:
url: http://localhost:8080
security:
type: BasicAuth
username: some_user
password: some_user
Short lived token generation on startup
When spinning up Console, a token is needed to access the API. Previoulsy this had to be done in the UI which would not allow full IaC. Now, we have the conduktor login
command which leverages the admin credentials to generate an API token, and allow the rest of the commands you may need to startup. This is expanded upon in the docs.
Admin and Application Tokens
In addition to the startup token, you can now generate tokens for the appropriate scope, for admin and application level tokens. The docs will walk you through this.
Shareable Message Page
Individual messages can now be accessed from a unique URL! Now you can link directly to a specific Kafka message for review or investigation, be that for sharing with a teammate, or commenting on a Jira ticket.
From within the Consume page, select a message and use the 'Share' button to navigate to the standalone page. The standalone message page shows the key, value, metadata and headers in a single view. Switch between the JSON view or table view, and utilize jq for additional filtering of the value.
Large Messages Support
We have put a limit on the message sizes that are sent to the browser in the Consume page (100Kb). From now on, when a message is larger than this size, we'll provide you with a link to access the individual message - this mitigates performance issues and still provides a path for troubleshooting, and sharing, large messages.
Topic Catalog Details Page
Expose contextual documentation about your Kafka Topics that exist in your organization with the Topic Details page. This helps democratize data to enhance its understanding and usage, and facilitate collaboration through a shared knowledge base.
You can choose to open or lock editing of descriptions within the UI using specific annotations. Check the Topic Resource documentation for more information.
Audit Last Activity of Users
You can now audit the last activity date of users in Console.
From within the Settings > Users page, you will see a new column 'Last login'. Note that the user login event is also captured in the Audit Log.
Quality of Life improvements
- Introduced an intermediate screen for Kafka Connect, allowing you to segment Connectors by each Connect cluster
- Within a Connect cluster, introduced an icon for each connector that clarifies if auto-restart is enabled
- Topic Catalog Search is now case-insensitive
- Improved error message when trying to delete an ApplicationInstance that is referenced elsewhere
- Improved error message when assign ownership on resources already owned by another ApplicationInstance
- CLI delete command can now be applied at the file level, simliar to resource creation through
apply -f
you can nowdelete -f
Fixes 🔨
- Fixed an error that occurred when configuring a KsqlDBCluster in the UI
- Fixed a UI issue that caused several dropdowns components to look wrong
- Fixed an error message where expected and actual topic replication factor were inverted in the CLI
- When deleting a Kafka Cluster from Console, the Indexed data is now properly deleted as well
- Upgrade dependencies vulnerable to CVE-2024-21634
Gateway 3.1.1
Release date: 2024-06-20
General fixes 🔨
- Performance is improved when using a large number of interceptors (backported in 3.0.5)
- Pre-create folders when using RocksDB as a cache backend
- Moved the Schema Id to the headers when using field level encryption with Avro
Console 1.24.1
Release date: 2024-06-24
Fixes 🔨
- Fixed a UI issue on Self-service Application Catalog and Topic Policies pages
- Fixed a UI issue on Topic Catalog when listing topics created with empty configs
- Fixed an issue with KqlDB connection test button
- Fixed an issue with the new delete users from group endpoint definition in OpenAPI spec
Gateway 3.1.0
Release date: 2024-06-05
AclsInterceptorPlugin removed
Kafka ACLs are now fully integrated in the Core Features of Conduktor Gateway.
If you were using the AclsInterceptorPlugin, make sure to enable ACLs while upgrading the Gateway to 3.1.0.
To enable ACLs set the environment variable GATEWAY_ACL_STORE_ENABLED=true
.
Features
- Concentrated Topics can now be created with auto-managed flag. Backing topics will be automatically created and extended.
- Added support for Azure Managed Identity for Kafka authentication
- Added an optional configuration for SNI routing to define the separator to use when building host domain for brokers
- Added more context relative to interceptors in Audit logs
- Added the client & version (kafka-client, librdkafka, ...) of the client in the Audit logs on CONNECTED event
General fixes 🔨
- Added Schema Registry validation on encryption plugins
- Fixed an issue where the KMS Key would not be created if it didn't exist
- Fixed an issue with logger API
Gateway 3.0.5
Release date: 2024-06-04