Changelog
- Visit our Get Started page to test the latest version of Conduktor
- Submit your feedback and ideas via our public roadmap
- Receive updates on our future Releases via our Support Portal and select follow
Gateway 3.3.0β
Release date: 2024-09-05
Upcoming Breaking change π£β
This breaking change only impacts Local Gateway service accounts generated through our token endpoints:
POST /admin/username/{username}
POST /admin/vclusters/v1/vcluster/{vcluster}/username/{username}
If you are not using Local Gateway services accounts (OIDC, mTLS, Delegated Kafka), you are not impacted.
Today, the token as the password for local Gateway service accounts contains all the necessary information. As a result, the SASL username is not used during the authentication phase.
In an upcoming release, we will strictly enforce that the username and the token matches. This will help reduce inconsistencies and avoid unexpected behaviors.
This breaking change is due for release 3.5.0.
For this release 3.3.0, and next product release 3.4.0, we'll only raise the following warning in the logs:
2024-08-27T18:15:29 [WARN] - Inconsistency detected for plain authentication. Username applicationA is not consistent with validated token created for application-A. SASL configuration should be changed accordingly.
Features β¨β
- New V2 APIs and CLI support
- Support for HTTPS APIs
- Better UX for ACLs and superUsers
- Encryption Enhancements and Support Clarification
New V2 APIs and CLI supportβ
Weβre excited to introduce our new Gateway API, designed for seamless integration with our CLI. This update allows you to deploy Gateway resources using infrastructure-as-code with straightforward, clearly defined concepts:
- Interceptor
- GatewayServiceAccount
- GatewayGroup
- ConcentrationRule
- AliasTopic
- VirtualCluster
Check the CLI reference to get started, and the resources reference for more information on each concept.
---
apiVersion: gateway/v2
kind: GatewayGroup
metadata:
name: groupB
spec:
members:
- name: user1
- name: user2
---
apiVersion: gateway/v2
kind: Interceptor
metadata:
name: enforce-partition-limit
scope:
group: groupB
spec:
pluginClass: io.conduktor.gateway.interceptor.safeguard.CreateTopicPolicyPlugin
priority: 100
config:
numPartition:
action: BLOCK
max: 9
min: 9
topic: .*
$ conduktor apply -f gateway.yml
GatewayGroup/groupB: Created
Interceptor/enforce-partition-limit: Created
$ conduktor delete GatewayGroup groupB
The group groupB is still used by the following interceptor(s): enforce-partition-limit
Note: API V1 is still available, but we recommend that new users and those with simple Gateway configurations begin using the V2 API as soon as possible. We will announce a deprecation plan in the coming weeks and notify you in advance of which Gateway version will be the last to support the V1 APIs.
Support for HTTPS APIsβ
It is now possible to configure HTTPS and mTLS authentication on the Gateway HTTP APIs. Check the HTTP section of the Environment Variables page for more details.
Better UX for ACLs and superUsersβ
To coincide with the clearly defined concepts established in API V2, we are making changes to ACLs management in Gateway.
- ACLs and Super Users on the Gateway (excluding Virtual Clusters) must be configured through Environment Variables.
- ACLs and Super Users on Virtual Clusters must now be driven explicitly through API/CLI.
Enable ACLs for Gateway (excl. Virtual Clusters)β
Configure both environment variables:
GATEWAY_ACL_ENABLED=true # default false
GATEWAY_SUPER_USERS=alice,bob
If GATEWAY_SUPER_USERS
is not set, it will default to GATEWAY_ADMIN_API_USERS
for backward compatibility.
Enable ACLs for Virtual Clustersβ
Note that if you are migrating from an older version of Gateway, the migration will automatically generate existing Virtual Clusters as configuration.
- The automation will derive the boolean value
aclEnabled
from the previously usedGATEWAY_ACL_STORE_ENABLED
variable. - The migration will not populate the
superUsers
list automatically, so this must be addressed as part of your migration.
Example configuration:
---
apiVersion: gateway/v2
kind: VirtualCluster
metadata:
name: "mon-app-A"
spec:
aclEnabled: "true" # defaults to false
superUsers:
- username1
- username2
Encryption Enhancements and Support Clarificationβ
Field-Level Encryption: Preserving Message Format to Enhance Usabilityβ
When applying field-level encryption prior to 3.3.0
, the encryption plugin would convert the message to JSON, and re-apply the schema format when the message was read back through the decryption plugin.
In Gateway 3.3.0
, we now preserve the schema format for Avro messages - meaning the same schema is used in the backing topic, and the data can be read directly from Kafka or without the decryption plugin at all.
Read more about this change to the default behaviour, and how to configure it.
Fields which cannot be encrypted in-place (effectively any non-string field) have their encrypted value placed in the headers, and the field itself is given a default masking value. The default values are clarified below:
Field Type | Default Value in 3.3.0 |
---|---|
Integer | Int MIN_VALUE |
Long | Long MIN_VALUE |
Float | Float, MIN_VALUE |
Double | Float MIN_VALUE (float again here due to some serdes behaviour) |
byte[] | "********" as bytes |
fixed[] | every byte filled with charater "*" |
boolean | false |
Note that the same default values are now used across all relevant plugins when manipulating a non-string field - Data Masking, Partial Decrypt, and Encrypt on Fetch.
Attempt to apply encryption to a message more than once will now failβ
If any of the encryption headers are detected in a message when encryption is about to be applied, then the encryption operation will fail. This is because applying encryption twice (or more) is currently not reversible.
Deprecated support for Schema Based (tag) encryption with Protobufβ
Note this is no longer supported, and the Gateway will now throw an exception if the encryption plugin attempts to apply schema (tag) based processing to a Protobuf message.
Note that any data previously written in this mode can still be read back - as the decrypt does not use the schemas at all, rather it uses the message header to know what was encrypted.
General fixes π¨β
- Large double values (where > Float Max) are now supported in field-level encryption for Avro and Protobuf
- Bytes and fixed fields now properly supported in field-level encryption for Avro
- Avro unions of two or more values (rather than just a value and a null) are now supported in field-level encryption for Avro
- Schema (tag) based encryption now checks and fails if its config is invalid
- It is not possible to encrypt the headers which the encryption plugin uses to manage its decryption process (as this would render the data unrecoverable)
- Improved log messages for Interceptors that reject actions, such as TopicPolicyPlugin
- Several improvements to the LargeMessage & LargeBatch Interceptors
- Fixed an issue where KCache topic initialization would fail silently and leave Gateway in an unusable state
- Added a new Environment Variable
GATEWAY_MIN_BROKERID
(default 0) that allows for determinist mapping of brokers and ports - Improved network stability during Gateway scaling or Kafka topology changes
- Added support for overriding Kafka Producer properties used for Audit Log topic with
GATEWAY_AUDIT_LOG_KAFKA_
environment variables - Removed metric
gateway.brokered_active_connections
. This was equal to portCount with port mapping and always 1 in host mapping - Changed metric
gateway.request_expired
tags: nodeHost/nodePort are replaced by nodeId/clusterId - Fix default value for
GATEWAY_UPSTREAM_THREAD
config. The new intended default (number of CPU) previously was (2 x number of CPU). - Fixed an issue with
GATEWAY_ADVERTISED_SNI_PORT
that wasn't working properly - Add log level for io.confluent packages in default log configuration
- Add default value to non mandatory configruation value for min and max bytes in FetchPolicyInterceptor
- Fix an issue with Concentrated Topics creation with Redpanda
Known issuesβ
- We are aware of an issue with
kcat
when the new environment variableGATEWAY_MIN_BROKERID
is not aligned with the first BrokerId of your Kafka cluster.- As a workaround, you can either define
GATEWAY_MIN_BROKERID
to your first Kafka BrokerId or usekcat
with the-E
flag
- As a workaround, you can either define
- It is not possible to add Service Accounts to GatewayGroups using API V2 unless they are previously declared as GatewayServiceAccount.
- This is not a wanted behavior, especially for OAuth or Delegated Kafka Authentication where declaring a GatewayServiceAccount should not be needed. We'll address this issue in a follow-up release
- API V1 (user-mapping) is not impacted
- If you perform a rolling upgrade to 3.3.0, Gateway nodes in earlier versions will show the following error in the logs:
[ERROR] [KafkaCache:1007] - Failed to deserialize a value org.apache.avro.AvroTypeException: Expected field name not found: clusterId
- This is fine and will not cause any further problems
- If you use Virtual Clusters and ACLs: After updating to 3.3.0, you must manage VirtualCluster's ACL and superUsers through V2 API.
Gateway 3.2.2β
Release date: 2024-08-28
Upcoming Breaking change π£β
This breaking change only impacts Local Gateway service accounts generated through our token endpoints:
POST /admin/username/{username}
POST /admin/vclusters/v1/vcluster/{vcluster}/username/{username}
If you are not using Local Gateway services accounts (OIDC, mTLS, Delegated Kafka), you are not impacted.
Today, the token as the password for local Gateway service accounts contains all the necessary information. As a result, the SASL username is not used during the authentication phase.
In an upcoming release, we will strictly enforce that the username and the token matches. This will help reduce inconsistencies and avoid unexpected behaviors.
This breaking change is due for release 3.5.0.
For this hotfix release 3.2.2, and next product releases 3.3.0 and 3.4.0s, we'll only raise the following warning in the logs:
2024-08-27T18:15:29 [WARN] - Inconsistency detected for plain authentication. Username applicationA is not consistent with validated token created for application-A. SASL configuration should be changed accordingly.
General fixes π¨β
- Fixed a severe authentication issue with Gateway generated tokens that could lead to a different user being authenticated, effectively causing elevated privileges under certain conditions.
- Fixed an issue where
GATEWAY_SNI_HOST_SEPARATOR
couldn't be set to the value-
- Fixed an issue where
GATEWAY_SNI_HOST_SEPARATOR
wasn't properly taken in account - Fixed an issue with
GATEWAY_ADVERTISED_SNI_PORT
that wasn't working properly
Console 1.26.0β
Release date: 2024-08-14
We are aware of a critical CVE - CVE-2024-41110 - coming from a dependency of prometheus on the console-cortex
image. This CVE is related to prometheus docker metric scraping, which is not used by Conduktor.
Regardless, as soon as the prometheus team fix this issue, it will be patched immediately by Conduktor.
Features β¨β
- Manage Connectors using the CLI
- Self-service support for Connectors
- Enhanced UI & Alerts for Kafka Connect
- Quality of Life improvements
- Deprecation Warning: Upcoming migration from Tags to Labels
Manage Connectors using the CLIβ
Continuing with the Infra-as-code approach, we are happy to introduce CLI support for Connectors, providing an efficient and automated way to manage your Kafka Connect resources.
---
apiVersion: kafka/v2
kind: Connector
metadata:
connectCluster: kafka-connect
name: click.my-connector
labels:
conduktor.io/auto-restart-enabled: true
conduktor.io/auto-restart-frequency: 600
spec:
config:
connector.class: 'org.apache.kafka.connect.tools.MockSourceConnector'
tasks.max: '1'
topic: click.pageviews
Self-service support for Connectorsβ
Application Teams can now manage their Connectors with Self-service.
From now on, you can grant ownership to connectors on Self-service Application Instance.
---
apiVersion: self-service/v1
kind: ApplicationInstance
metadata:
application: "clickstream-app"
name: "clickstream-dev"
spec:
cluster: "shadow-it"
serviceAccount: "sa-clicko"
resources:
- type: CONNECTOR
connectCluster: shadow-connect
patternType: PREFIXED
name: "click."
Enhanced UI & Graphs for Kafka Connectβ
We have revisited the Kafka Connect UI in multiple ways to improve your experience:
- Connect Cluster selection screen with a preview of Connector status
- New graphs demonstrating the state of your Connector over time
Support for High Availability (HA) Consoleβ
Multiple Console instances can now be deployed in parallel to achieve high availability.
This applies to the deployment of conduktor-console
, while conduktor-console-cortex
is currently limited to a single instance. The design ensures minimal impact on the cluster by assigning only one instance to handle the indexing of Kafka data used for performance monitoring.
Quality of Life improvementsβ
- The checkbox to skip TLS verification is now always visible
- The YAML for Topic object now allows number in
spec.configs
. Previously it was mandatory to quote all numbers. - Self-service Topic Policies are now visible in the UI
Fixes π¨β
- Topic Policies from Self-service are now properly enforced from the UI, as well as both the API and CLI
- Fix Kafka Connect Cluster list showing invalid number of running tasks
Deprecation Warning: Upcoming migration from Tags to Labels π£β
With the introduction of the Self-service resource manifests, we brought customers a means to annotate all their resources with labels. Labels are more structured than the existing Conduktor tags, thereby allowing for more precise filtering capabilities, as can be seen in the Topic Catalog.
In an upcoming release, we'll perform an automatic migration from Tags to Labels.
Tags written with the naming convention <key>/<value>
will automatically be added as similar labels:
<key>: <value>
If there is a conflict such as; a topic containing tags with the same key, that already has the target label, or is not written with this naming convention, then they will be created as follows:
tag-<value>: true
Here's an example of how tags will be migrated into labels:
# Tags:
- format/avro
- project/supplychain
- team/delivery
- color/blue
- color/red
- wikipedia
- non-prod
# Result
labels:
format: avro
project: supplychain
team: delivery
tag-color/blue: true # Because conflict on "color"
tag-color/red: true # Because conflict on "color"
tag-wikipedia: true
tag-non-prod: true
β οΈ Conduktor can help you rename tags through Customer Support
Between now and the migration, we can help you rename your tags for a smooth transition to labels.
Contact us as soon as possible if you would like support.
Gateway 3.2.1β
Release date: 2024-07-31
General fixes π¨β
- Fixed an issue when either
GATEWAY_ACL_ENABLED
orGATEWAY_ACL_STORE_ENABLED
was enabled resulting in ACLs being enforced in undesirable scenarios.
Gateway 3.2.0β
Release date: 2024-07-19
Breaking Changes π£β
Two new backing topics are required for Gatewayβ
In the next release (3.3), we'll bring a new API as well as support in the Conduktor CLI to manage Gateway concepts using infra-as-code approach.
In preparation for this upcoming release, we are replacing some weakly-defined concepts in favor of strongly-defined concepts. The following are now clearly captured in the topics mentioned below:
- Virtual Clusters that existed only through creation of UserMappings or Interceptors targeting
- GatewayGroups that existed only through UserMappings
As a result, 2 new topics will now be created once you upgrade to Gateway 3.2:
_conduktor_gateway_vclusters
_conduktor_gateway_groups
If you are happy with the default names, you have nothing to do. If you want to control the name of those topics, use the 2 new environment variables:
GATEWAY_VCLUSTERS_TOPIC
GATEWAY_GROUPS_TOPIC
Check the associated Documentation for more information.
Changes to ACL support on Gatewayβ
With Gateway 3.1 we removed our dedicated ACL interceptor in favor of a new environment variable GATEWAY_ACL_STORE_ENABLED
. This variable was enabling ACLs in all scenarios, whether you used Virtual Clusters or not.
Changes for Gateway 3.2β
With Gateway 3.2, we are adding a new environment variable GATEWAY_ACL_ENABLED
and modifying the behavior of the existing variable GATEWAY_ACL_STORE_ENABLED
.
From now on, the variables works as follow:
Environment Variable | Description | Default |
---|---|---|
GATEWAY_ACL_ENABLED | Enable ACLs on the Gateway excluding Virtual Clusters | "false" |
GATEWAY_ACL_STORE_ENABLED | Enable ACLs on Virtual Clusters only | "false" |
Preview for Gateway 3.3β
In the next release, we will enhance ACLs to restore and expand the full set of features available before version 3.1. This will be achieved through the introduction of a CLI and APIs, making concepts like VirtualCluster first-class citizens.
Enable ACLs for Gateway (excl. Virtual Clusters)β
Configure both environment variables:
GATEWAY_ACL_ENABLED=true
GATEWAY_SUPER_USERS=alice,bob
Enable ACLs for Virtual Clustersβ
---
apiVersion: gateway/v2
kind: VirtualCluster
metadata:
name: "mon-app-A"
spec:
aclEnabled: "true" # defaults to false
superUsers:
- username1
- username2
This will effectively render GATEWAY_ACL_STORE_ENABLED
obsolete.
General fixes π¨β
- Fixed an issue with Field-level Avro encryption/decryption relating to numeric fields:
- When using partial decryption with Avro schema registry, any numeric values (int, long, float, double) that are not being decrypted will instead be masked with the minimum (most negative) value for the numeric type
- This is to ensure the field is compliant with the original type in the Avro schema
- Fixed an issue with the ClientIdRequired Policy that wasn't properly overriding the ClientId
- Fixed an issue to ensure all active connections are closed, and clients transition quicker to the new cluster during cluster switching
Console 1.25.1β
Release date: 2024-07-23
Breaking Changes π£β
New docker image nameβ
We have renamed the Console docker image to conduktor/conduktor-console
to clarify our product naming.
Please modify your installation to reflect this change as we will now stop publishing a conduktor/conduktor-platform
image.
docker pull conduktor/conduktor-console:1.25.1
Features β¨β
- Conduktor Console IaC Compatible
- Shareable Message Page
- Large Messages Support
- Topic Catalog Details Page
- Audit Last Activity of Users
- Quality of Life improvements
Conduktor Console IaC Compatibleβ
Console is now able to be fully deployed through an IaC approach with the following additions to Console 1.25 and CLI 0.2.7.
Manage Cluster Connectionsβ
Manage your Console resource lifecycle with the addition of the KafkaCluster, KafkaConnectCluster and KsqlDBCluster objects to our IaC approach using the Conduktor CLI.
Checkout the example below and find the full definition at Console Resources Reference documentation.
---
apiVersion: console/v2
kind: KafkaCluster
metadata:
name: cloud-kafka
spec:
displayName: "Cloud Kafka"
icon: "kafka"
color: "#000000"
bootstrapServers: "34.140.204.135:12092"
properties:
sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret";
security.protocol: SASL_SSL
sasl.mechanism: PLAIN
schemaRegistry:
url: http://34.140.204.135/registry/
security:
type: BasicAuth
username: superUser
password: superUser
Short lived token generation on startupβ
When spinning up Console, a token is needed to access the API. Previoulsy this had to be done in the UI which would not allow full IaC. Now, we have the conduktor login
command which leverages the admin credentials to generate an API token, and allow the rest of the commands you may need to startup. This is expanded upon in the docs.
Admin and Application Tokensβ
In addition to the startup token, you can now generate tokens for the appropriate scope, for admin and application level tokens. The docs will walk you through this.
Shareable Message Pageβ
Individual messages can now be accessed from a unique URL! Now you can link directly to a specific Kafka message for review or investigation, be that for sharing with a teammate, or commenting on a Jira ticket.
From within the Consume page, select a message and use the 'Share' button to navigate to the standalone page. The standalone message page shows the key, value, metadata and headers in a single view. Switch between the JSON view or table view, and utilize jq for additional filtering of the value.
Large Messages Supportβ
We have put a limit on the message sizes that are sent to the browser in the Consume page (100Kb). From now on, when a message is larger than this size, we'll provide you with a link to access the individual message - this mitigates performance issues and still provides a path for troubleshooting, and sharing, large messages.
Topic Catalog Details Pageβ
Expose contextual documentation about your Kafka Topics that exist in your organization with the Topic Details page. This helps democratize data to enhance its understanding and usage, and facilitate collaboration through a shared knowledge base.
You can choose to open or lock editing of descriptions within the UI using specific annotations. Check the Topic Resource documentation for more information.
Audit Last Activity of Usersβ
You can now audit the last activity date of users in Console.
From within the Settings > Users page, you will see a new column 'Last login'. Note that the user login event is also captured in the Audit Log.
Quality of Life improvementsβ
- Introduced an intermediate screen for Kafka Connect, allowing you to segment Connectors by each Connect cluster
- Within a Connect cluster, introduced an icon for each connector that clarifies if auto-restart is enabled
- Topic Catalog Search is now case-insensitive
- Improved error message when trying to delete an ApplicationInstance that is referenced elsewhere
- Improved error message when assign ownership on resources already owned by another ApplicationInstance
- CLI delete command can now be applied at the file level, simliar to resource creation through
apply -f
you can nowdelete -f
Fixes π¨β
- Fixed an error that occurred when configuring a KsqlDBCluster in the UI
- Fixed a UI issue that caused several dropdowns components to look wrong
- Fixed an error message where expected and actual topic replication factor were inverted in the CLI
- When deleting a Kafka Cluster from Console, the Indexed data is now properly deleted as well
- Upgrade dependencies vulnerable to CVE-2024-21634
Gateway 3.1.1β
Release date: 2024-06-20
General fixes π¨β
- Performance is improved when using a large number of interceptors (backported in 3.0.5)
- Pre-create folders when using RocksDB as a cache backend
- Moved the Schema Id to the headers when using field level encryption with Avro
Console 1.24.1β
Release date: 2024-06-24
Fixes π¨β
- Fixed a UI issue on Self-service Application Catalog and Topic Policies pages
- Fixed a UI issue on Topic Catalog when listing topics created with empty configs
- Fixed an issue with KqlDB connection test button
- Fixed an issue with the new delete users from group endpoint definition in OpenAPI spec
Gateway 3.1.0β
Release date: 2024-06-05
AclsInterceptorPlugin removedβ
Kafka ACLs are now fully integrated in the Core Features of Conduktor Gateway.
If you were using the AclsInterceptorPlugin, make sure to enable ACLs while upgrading the Gateway to 3.1.0.
To enable ACLs set the environment variable GATEWAY_ACL_STORE_ENABLED=true
.
Featuresβ
- Concentrated Topics can now be created with auto-managed flag. Backing topics will be automatically created and extended.
- Added support for Azure Managed Identity for Kafka authentication
- Added an optional configuration for SNI routing to define the separator to use when building host domain for brokers
- Added more context relative to interceptors in Audit logs
- Added the client & version (kafka-client, librdkafka, ...) of the client in the Audit logs on CONNECTED event
General fixes π¨β
- Added Schema Registry validation on encryption plugins
- Fixed an issue where the KMS Key would not be created if it didn't exist
- Fixed an issue with logger API
Gateway 3.0.5β
Release date: 2024-06-04
Performance improvements πβ
- Performance is improved when using a large number of interceptors
Console 1.24.0β
Release date: 2024-06-19
Breaking Changes π£β
New docker image nameβ
We have renamed the Console docker image to conduktor/conduktor-console
to clarify our product naming.
Final warning! This is the last version where we publish our images using both names.
Please modify your installation to reflect this change in advance of us deprecating the name conduktor/conduktor-platform
.
docker pull conduktor/conduktor-console:1.24.0
Change in ApplicationInstance Resource Type from GROUP to CONSUMER_GROUPβ
We have renamed the resource type in ApplicationInstance
from GROUP
to CONSUMER_GROUP
. This change is intended to prevent confusion with the newly introduced resources ApplicationGroup
and Group
.
---
kind: ApplicationInstance
spec:
resources:
- type: CONSUMER_GROUP # Previously: GROUP
name: "click."
patternType: PREFIXED
Features β¨β
- Self-service
- Manage Groups and Users using the CLI
- Topic list columns Produce Rate and Last Activity
- Active Data Policies in Topic Consume page
- Quality of Life improvements
- Fixes π¨
Self-serviceβ
There's a host of new functionality available providing a truly powerful self-service release. This comes from the addition of two new resources: Subject and ApplicationGroup
Application Teams can now manage their Subject resource lifecycle through IaC with the addition of the Subject object.
A new concept, ApplicationGroup lets Application Teams fully organize themselves within their Application scope to restrict who can do what over their resources within Console UI. It's a form of delegated RBAC.
Checkout the definitions below and find the full list of resource definitions via the Resource Reference documentation.
Subjectβ
This creates a Subject in the Schema Registry.
---
apiVersion: v1
kind: Subject
metadata:
cluster: shadow-it
name: myPrefix.topic-value
spec:
schemaFile: schemas/topic.avsc # relative to conduktor CLI execution context
format: AVRO
compatibility: FORWARD_TRANSITIVE
ApplicationGroupβ
Create an Application Group to directly reflect how your Application operates. You can create as many Application Groups as required to restrict or represent the different teams that use Console on your Application, e.g.:
- Support Team with only Read Access in Production
- DevOps Team with extended access across all environments
- Developers with higher permissions in Dev
# Permissions granted to Console users in the Application
---
apiVersion: v1
kind: ApplicationGroup
metadata:
application: "clickstream-app"
name: "clickstream-support"
spec:
displayName: Support Clickstream
description: |
Members of the Support Group are allowed:
Read access on all the resources
Can reset offsets
permissions:
- appInstance: clickstream-app-dev
resourceType: TOPIC
patternType: "LITERAL"
name: "*" # All owned & subscribed topics
permissions: ["topicViewConfig", "topicConsume"]
- appInstance: clickstream-app-dev
resourceType: GROUP
patternType: "LITERAL"
name: "*" # All owned consumer groups
permissions: ["consumerGroupReset", "consumerGroupView"]
members:
- alice@company.org
- bob@company.org
Topic Catalogβ
We're expanding on the Topic Catalog, to help teams discover Kafka Topics within your organization.
You can now filter on all the topics based on user-defined, business metadata.
Looking to request access to another applications resources? You can now generate the required ApplicationInstancePermission
snippet that grants the necessary access to Topics belonging to another Application.
Manage Groups and Users using the CLIβ
Manage your Console Group and Permissions lifecycle through IaC with the addition of the Group and User objects.
Checkout the example below and find the full definition via the Resource Reference documentation.
---
apiVersion: v2
kind: Group
metadata:
name: developers-a
spec:
displayName: "Developers Team A"
description: "Members of the Team A - Developers"
externalGroups:
- "LDAP-GRP-A-DEV"
members:
- member1@company.org
- member2@company.org
permissions:
- resourceType: TOPIC
cluster: shadow-it
patternType: PREFIXED
name: toto-
permissions:
- topicViewConfig
- topicConsume
- topicProduce
Topic list columns Produce Rate and Last Activityβ
We added two new columns to the Topic List to help you troubleshoot and understand Kafka better: Produce Rate & Last Activity.
Values are computed once per Indexing (i.e. every 30s);
-
Produce Rate is calculated from the two most recent offset values provided by our indexer.
-
Last Activity is set to
Datetime.now()
if the latest offsets have changed since the last Indexing
Active Data Policies in Topic Consume pageβ
When exploring topics, fields masked by active Data Policies are now displayed in a different color, while the policy name is also now visible on hover.
Quality of Life improvementsβ
Topic pages
- You can now see all subjects associated to the Schema Id of the current message from the Message Viewer panel
- Added message Compression Type metadata in the Message Viewer panel
- Added buttons to navigate to previous and next message in the Message Viewer panel. Also works with the arrow keys
- The "Generate once" feature in the Produce tab now generates much more realistic, randomized messages, especially for Registry schemas and JSON
Other pages
- Added a button to force re-balance active Consumer Groups in the Consumer Group details page
- Added a "Test connection" button when adding a KsqlDB cluster in Settings
- Added KsqlDB query Start From selector, equivalent to the
SET 'auto.offset.reset'
command - Added an icon in the Kafka Connect list to inform that auto-restart feature is active
API
- When returning a Forbidden error, the missing permissions are listed in the error message
- New endpoint to add user to a group by email
Conduktor CLI
Update your Conduktor CLI to 0.2.5
- Env Variable changed from
CDK_TOKEN
toCDK_API_KEY
to set your Admin or Application API Key - Added support for Subject field
spec.schemaFile
. Previous versions of the CLI will only acceptspec.schema
inlined.
Fixes π¨β
- Clean monitoring metrics related to brokers that are unreachable
- Fix support of Avro byte arrays encoded as base64 when producing messages
- Fix bulk import of users in case a user already exist
- Fix user creation when the user is not admin but has the right permissions
- Fix class name selector when navigating from one interceptor to another
- External group mapping: support extraction of roles from both string array and comma separated string
- Fix preview of consumer group offset reset when selecting a specific offset
- Data masking: trim name of policy and fix encoding for URL
- Monitoring: show error in UI if cortex is unreachable
- Fix schema that disappeared from the form input when schema was invalid
- Prevent the creation of an application instance with resources that overlaps
- Fix permissions when 2 application instances define resources on the same cluster
- Fixed an issue where apiVersion was displayed at the end using the CLI
Gateway 3.0.4β
Release date: 2024-05-22
Performance improvements πβ
- Consumer group membership is no longer loaded synchronously
- Optimize hostname resolution for ACL
General fixes π¨β
GATEWAY_DOWNSTREAM_THREAD
andGATEWAY_UPSTREAM_THREAD
are now correctly gathering the number of cores- in
LargeMessageHandlingPlugin
plugin, honor correctly thelocalCacheExpireAfterWriteInSeconds
property
Gateway 3.0.3β
Release date: 2024-05-09
General fixes π¨β
- Fixed an issue impacting the vault configuration key
uri
when special characters (i.e-
) are present in the hostname.
Gateway 3.0.2β
Release date: 2024-05-07
General fixes π¨β
- Fixed a race condition when closing connections (i.e. when Gateway detects a broker is removed from the cluster) that was causing restarts/timeouts
- Fix duplicated key exception when rebuilding fetch request with duplicated topics
- FIX NPE when handling expired ApiVersions requests
- Added a check to validate schema registry connection and provide more meaningful errors for schema-based encryption
- Added a check against
defaultAlgorithm
used in the encryption interceptor to ensure it's a valid enum value, and avoid overriding with defaults - Fixed an issue with
externalStorage
set totrue
in the encryption interceptor that was failing to store headers in a separate internal topic - Ensure that if the encryption algorithm is changed, a new entry appears in the internal topic used to store headers
- Default namespace is now applied properly on schema-based encryption
- Added support encryption/decryption of AVRO
bytes
andenums
types
Console 1.23.0β
Release date: 2024-05-03
Future Breaking Changes π£β
New docker image nameβ
We have renamed the Console docker image to conduktor/conduktor-console
to clarify our product naming.
We will publish newer versions using both names for this release and the next release only. Please modify your installation to reflect this change in advance of us deprecating the name conduktor-platform
.
docker pull conduktor/conduktor-console:1.23.0
Features β¨β
Self-serviceβ
There's a host of new functionality available providing our first truly powerful self-service release. This comes from the addition of two new resources (Topic, TopicPolicy), application tokens, a topic catalog and service account synchronization.
Application Teams can now manage their Topic resource lifecycle through IaC with the addition of the Topic object, and they can do this safely with Platform Teams putting in place a Topic Policy to restrict expensive configurations and enforce naming standards.
Checkout the definitions below and find the full list of resource definitions via the Resource Reference documentation.
Topicβ
This creates a Topic in the defined cluster.
---
apiVersion: v2
kind: Topic
metadata:
cluster: shadow-it
name: click.event-stream.avro
spec:
replicationFactor: 3
partitions: 3
configs:
min.insync.replicas: '2'
cleanup.policy: delete
retention.ms: '60000'
TopicPolicyβ
TopicPolicy lets Platform Team define governance rules to restrict Application Teams to create Topics with misconfigurations. This is also useful to enforce naming convention or metadata annotation by Application Teams.
---
apiVersion: "v1"
kind: "TopicPolicy"
metadata:
name: "click-naming-rule"
spec:
policies:
metadata.name:
constraint: Match
pattern: ^click\.(?<event>[a-z0-9-]+)\.(avro|json)$
spec.replication.factor:
constraint: OneOf
values: ["3"]
spec.configs.retention.ms:
constraint: Range
max: 604800000 # 7d
min: 3600000 # 1h
Topic Catalogβ
We've introduced the Topic Catalog, to help teams discover Kafka Topics within your organization. Quickly get visbility on ownership and business metadata on your choice for topics.
Add topics to applications to see them appear within the catalog across all your clusters, searchable by name and labels.
Application API Keysβ
Generate ApplicationInstance API Keys to create any ApplicationInstance scoped resources. Only ApplicationInstancePermission and Topic are supported at the moment.
Use this Key with the CLI to use it manually or within CI/CD pipelines.
In addition, Service Account's ACLs are now synchronized with the permissions from ApplicationInstance and ApplicationInstancePermission resources.
Editable columns on the Consume Pageβ
You can now customise the columns you want to display in the Consume Page. Let us know if there's any additional metadata you want to see!
Quality of Life improvementsβ
Topic pages
- SchemaId is now displayed from the Message Viewer panel
- Header count is now displayed from the Message Viewer panel
- The More Options "..." button has been moved so that it's available from every Topic details tab
- Added a check to prevent producing empty keys to a compacted topic
- Added an "Add partitions" button in Partitions tab
Schema Registry pages
- The current schema is now inside a read-only area
- Increased the width of the side panel when creating/updating schemas
- Full height is used in the panel to show/edit the schema
Kafka Connect pages
- Kafka Connect List can now be sorted by the number of Tasks
- Removing a Connector now properly redirects the user to the Connector list instead of the Configuration tab of the deleted Connector
- Topics column is now sourced from more configuration keys (
kafka.topic
,kafka.topics
,topic
,topics
)
Settings
- Permissions on KafkaConnect and ksqlDB now properly display the name instead of the UUID
- Adding Users to Groups can now be done from the User details page directly
- Added the Group name in the UI to be used in the API or CLI
Other
- Added Gateway version on the Interceptor List page
- Added a configuration option to toggle OIDC logout when logging out from Console
- Searching in screens now trims whitespace from the text supplied
Fixes π¨β
- Fixed an issue with the Test Connection button that didn't work after a successful response
- Fixed an issue with the indexing of Confluent Cloud Managed Connect
- Fixed an issue with the Kafka Connect List where filter by Connect Cluster wouldn't work in some cases
- Fixed an issue with the Schema Registry indexer not properly handling a retriable HTTP error (GOAWAY)
- Fixed an issue with the timezone selector scrolling when resetting offsets for a Consumer Group by timestamp
- Fixed an issue with SSO in Azure environments for users who are members of a large amount of Azure groups
- The following fixes have also been back-ported in 1.22.1
- Fixed an issue where two ACLs of the same name but with different pattern types (PREFIXED and LITERAL) were merged to the same group within the UI
- Fixed an issue with OIDC login that could cause an expired session to become stuck and prevent login attempts
- Fixed an issue with ksqlDB caused by not escaping the Stream or Table name in the query
Console 1.22.1β
Release date: 2024-04-18
Features β¨β
- Added support for Azure Managed Identity for Kafka authentication
- Implement OIDC logout. You may need to update your OIDC configuration to allow the root page of Console as a possible redirect URI
Fixes π¨β
- Fixed an issue where two ACLs with the same name but with different pattern type (PREFIXED and LITERAL) were merged in the same group in the UI.
- Fixed an issue with OIDC login that could cause an expired sessions to become stuck and prevent login in again.
- Fixed an issue with ksqlDB caused by not escaping the Stream or Table name in the query.
Console 1.21.3β
Release date: 2024-04-18
Features β¨β
- Added support for Azure Managed Identity for Kafka authentication
- Implement OIDC logout. You may need to update your OIDC configuration to allow the root page of Console as a possible redirect URI
Fixes π¨β
- Fixed an issue where two ACLs with the same name but with different pattern type (PREFIXED and LITERAL) were merged in the same group in the UI.
- Fixed an issue with OIDC login that could cause an expired sessions to become stuck and prevent login in again.
Gateway 3.0.1β
Release date: 2024-04-15
General fixes π¨β
- Fixed some issues with Encryption when the value is a tombstone.
- Fixed some inconsistencies between the OpenAPI Spec and the actual implementation.
- Fixed a memory leak when using the default
GATEWAY_UPSTREAM_CONNECTION_POOL_TYPE
. - Added a startup check to prevent an incompatible configuration:
GATEWAY_UPSTREAM_CONNECTION_POOL_TYPE=ROUND_ROBIN
with delegated authentication.
Console 1.22.0β
Release date: 2024-04-03
Future Breaking Changes π£β
New docker image nameβ
We have renamed the Console docker image to conduktor/conduktor-console
to clarify our product naming.
We will publish newer versions using both names for the next two releases only. Please modify your installation to reflect this change in advance of us deprecating the name conduktor-platform
.
docker pull conduktor/conduktor-console:1.22.0
Features β¨β
Topic as a Service becomes Self-serviceβ
Self-service is a replacement for Topic as a Service. It is more centered towards a GitOps way of working, though we have performed a migration for existing TaaS applications to ensure a seamless transition to the new model:
- Applications + Environments are migrated to
Application
andApplicationInstance
- Cross Application accesses are migrated to
ApplicationInstancePermission
- The Application list becomes Application Catalog
- At the moment, we decided that we should control everything from the CLI only. The UI will remain Read-Only for now, but the intention is to bring back UI-driven operations in a future release.
To start using Self-service, you must download our Conduktor CLI which lets you deploy resource files in Console.
Conduktor CLIβ
Console now has a CLI! Get Started with it today.
For now, we only support the following resources:
- Application
- ApplicationInstance
- ApplicationInstancePermission
Our objective is to let Application Teams and Central Teams manage both Console resources (Clusters, Groups, Permission, Self-service resources, DataPolicies, Alerts, ...) and Kafka resources (Topics, Subjects, Connectors, ...) using a common definition mechanism.
More to come, automate everything!
---
apiVersion: "v1"
kind: "ApplicationInstance"
metadata:
application: "clickstream-app"
name: "clickstream-app-dev"
spec:
cluster: "shadow-it"
service-account: "sa-clickstream-dev"
resources:
- type: TOPIC
name: "click."
patternType: PREFIXED
- type: GROUP
name: "click."
patternType: PREFIXED
Custom Deserializersβ
Console's support for Custom Deserializers is finally here!
A picture is worth a thousand words:
Check our dedicated How-To guide Installing and Configuring Custom Deserializers.
Fixes π¨β
- Fixed an issue with controller metrics in Monitoring when the Kafka cluster is using KRaft
- Added support for Broker, Connect, and ksqlDB
id
field and TLS authentication in the YAML configuration file and Environment variables. This implies you might have a duplicate Connect instance if you use a YAML file with an ID for your Connect cluster. Check the Environment Variables page for more details - Added new configurations to tune indexing batching and parallelization.s
- Fixed an issue with Azure PostgreSQL preventing the Schema Registry page from displaying properly
Gateway 3.0.0β
Release date: 2024-03-20
This major release of the Gateway brings functionality around targeting your interceptors more specifically, adding additional data quality & filtering tools and a host of rework under the hood for improved reliability & robustness. This can be seen in the form of reworked authorization to more closely align with what you're used to in the existing Kafka world and a more intuitive experience when working with the enhanced functionality Conduktor brings in concentrated and alias topics.
Featuresβ
Interceptor Targetingβ
Interceptors can now target more specifically than the previous scopes of vcluster and username. They can now be targeted at the global, vcluster, group(new), or service account level. Below are some areas and examples where targeting interceptors brings great power in their flexibility.
Apply interceptors on groups, across several service accounts, without duplicating the interceptorβ
On a given Kafka cluster, each application may have different policy requirements.
Applications could be part of an organization's domain (finance, HR, Sales, etc.) or grouped by another dimension, such as data sensitivity. Platform teams will want to manage rules that apply to multiple applications at a "group" level.
Override behavior at a more specific service account, or group levelβ
Rather than apply interceptors across a wider domain, they may want to zoom in and target a specific application to override the wider defaults.
Examples:
A project from a domain is more advanced and doesn't need the safeguarding protections applied to the wider group.
- They know how to size topics correctly and are allowed a higher limit on partitions for topic creation, than the rest of the group
- Everyone is required to have compression enforced by default, but for this specific team they are allowed to remove it to meet a low latency requirement
Data quality validation rules across fields, using CELβ
Validate data across fields using Common Expression Language. Before we could define rules for fields within a schema, a great way to ensure data quality catching the data before it hits the cluster. Now, we can relate fields to each other. We can bring together data quality and business rules within our schema.
An example for age and email checks in our schema:
{
"fields": [
{
"name": "age",
"type": "int",
"minimum": 18
},
{
"name": "email",
"type": "string",
"format": "email"
}
],
"metatadata": {
"rules": [
{
"name": "old people",
"expression": "age >= 40 && email.endsWith('yahoo.com')",
"message": "yahoo.com emails are allow only for people older that 40"
}
]
}
}
Filter messages on topics, using CELβ
Topic filtering can now be done with a simple plugin rather than building yet another pipeline. Effortlessly tailor message filtering rules to your use cases, ensuring only the most relevant data reaches your consumers.
Similar to how we allow you to filter data using SQL, you can now use CEL. By leveraging CEL expressions, you gain the flexibility to filter messages based on various attributes such as record key, value, partition, timestamp, header, and offset, offering unparalleled control over your data consumption.
Suppose you want to filter messages where the timestamp is greater than a certain threshold and the record key matches a specific pattern. With the enhanced CEL topic filtering feature, achieving this becomes straightforward as posting a plugin with the config:
{
"virtualTopic": "your-topic-name",
"expression": "record.timestamp > 1616400000000 && record.key.startsWith('prefix_')"
}
Topic multiplexing enhancementsβ
Several enhancements have been made when working with concentrated topics for topic multiplexing.
Concentration can now be achieved on the default vcluster, passthrough
.
UX has been adjusted from using patterns only in favor of concentration rules, which have a dedicated part of the API.
Alias topic enhancementsβ
Alias topics (dedicated to referencing another topic within your cluster, see the docs for more) have been reworked for a more intuitive experience. Alias topics no longer replace the physical topic during interactions, but are seen as another topic. This will help in use cases related to migration, when applications use different topic names, and when exposing more topics within vclusters.
Default vcluster reworkβ
The default vcluster, passthrough
, now has users associated with it by default rather than being rejected. This behavior can be reverted through configuration; see the docs for more.
General fixes π¨β
- Fixed an issue that was prefixing consumer group names with Gateway in certain virtual clusters
- Simplified the security protocol experience, dropping the need for
GATEWAY_MODE
(s) to be defined, instead using Kafka standard security protocols orDELEGATED
security protocols. Refer to the docs for more - Less noisy metrics
- Configuration topics are now prefixed with the clusterID to prevent unintentional
- The PUT HTTP verb is now supported throughout the API
- ARM build is now available for the distro and distro-less images, to provide more flexibility to your deployment machine choices
Console 1.21.1β
Release date: 2024-03-05
Fixes π¨β
- Resolved a problem causing a blank screen after login in certain situations, preventing users from accessing Console.
Console 1.21.0β
Release date: 2024-02-26
Future Breaking Changes π£β
New docker image nameβ
To clarify our product naming we have renamed the Console docker image to conduktor/conduktor-console
.
We will publish newer versions using both names for the next three releases only. Please modify your installation to reflect this change in advance of us deprecating the name conduktor-platform
.
docker pull conduktor/conduktor-console:1.21.0
Features β¨β
- ksqlDB
- Metrics available via Prometheus
- Smart tables for Connect and Schema Registry
- Add Local Users from the UI
- Fixes
ksqlDBβ
Say hello to seamless integration with ksqlDB for you and your team on Conduktor Console.
Grant permission to whom can access the interface to create queries, setup new connections and visualise the existing connections.
Now you can:
- Browse ksqlDB clusters that are connected to Console
- Visualize all the currently running queries as well as write your own queries or executes statements
- Visualize and act on the running Streams (resulting from CREATE STREAM statements) with the Streams tab
- Visualize and act on the running Streams (resulting from CREATE TABLE statements) with the Tables tab
- Show all the Persistent and Push queries currently running on the ksqlDB Cluster with the Queries tab
- Execute Pull and Pull Queries (SELECT) and Statements (CREATE, DESCRIBE, DROP, ...) with the Editor tab
More info about kSQL is available on their website.
For more information checkout the docs.
Subscribe to metrics via the Prometheus endpointβ
Gain deeper insights into your system's performance with metrics now readily available via the Prometheus endpoint. No need for yet another system to monitor, seamlessly integrate metrics directly into your external log system in the Prometheus format, allowing for effortless monitoring and optimization of Conduktor within your systems.
You can monitor metrics such as under replicated partitions, total & failed connector tasks and consumer group lags. For the full list of available metrics checkout the docs.
Smart tables for Kafka Connect and Schema Registry subjectsβ
Get the answers you need quicker with the new tables. Sort by what matters, be that subject name, version count, latest version and more! For Connect there's all the usual suspects: source/sink, topics, the connect cluster and importantly the state (e.g. Failed, Paused). Quickly find which connectors are failing with just one click.
Choose what columns to hide the noise. Filter on name, tags and other resource metatdata such as consumer group state.
Add Local Users from the UIβ
Don't have SSO ? Now you can add Users directly from the Users & Groups page in Settings, instead of modifying the config file and restarting the Console.
Fixes π¨β
- Added support for complex union-type avro messages in Console Producer
- Fixed a blank screen issue after login due to case-sensitivity bug with email address
- Fixed an issue where Message Reprocessing didn't work after refreshing the page
- Resolved an issue with MSK role assumption
- Fixed an issue with custom certificates for Schema Registry and Kafka Connect
- Fixed several issues improving indexing performance on large clusters
- Increased cortex ingestion limits for large clusters
- Fixed an issue that occurs when
Group:
ACLs are present
Gateway 2.6.1β
General fixes π¨β
GATEWAY_SECURED_METRICS
=false
would not allow to access the prometheus metrics without security. This is now fixed.
Gateway 2.6.0β
Schema based encryptionβ
You can now define your encryption requirement directly within your Schemas.
Here is an example using json schema where we specify that we want to encrypt the password
and visa
fields, with their corresponding keySecretId
.
We also tag the location
field as PII
and GDPR
.
{
"$id": "https://example.com/person.schema.json",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "Customer",
"type": "object",
"properties": {
"name": { "type": "string" },
"username": { "type": "string" },
"password": { "type": "string", "conduktor.keySecretId": "password-secret"},
"visa": { "type": "string", "conduktor.keySecretId": "visa-secret" },
"address": {
"type": "object",
"properties": {
"location": { "type": "string", "conduktor.tags": ["PII", "GDPR"] },
"town": { "type": "string" },
"country": { "type": "string" }
}
}
}
}
The encryption configuration now supports defaults to simplify your setups
{
"defaultKeySecretId": "myDefaultKeySecret",
"defaultAlgorithm": {
"type": "TINK/AES128_EAX",
"kms": "IN_MEMORY"
},
"tags": [ "PII", "ENCRYPTION", "GDPR" ]
}
KMS now use cloud managed identities by defaultβ
To prevent setting up manual connectivity, KMS are now using cloud managed identity by default
Cache KMS Time to Liveβ
You can now cache the KMS keys for a certain amount of time. This is useful to reduce the number of calls to your KMS.
keyTtlMs
: The key's time to live in milliseconds. Default is 1 hour, disable the cache by setting it to 0
Override Header Injectionsβ
Header config can now be further enforced with overrides, the plugin now supports overrideIfExists
with default set to false
. When set to true
, the plugin will override the header if it already exists in the request. This can be useful for if a required piece of metadata is missing in the header, you can add something automatically whilst ignoring the ones that have set the value.
SSL Principal Extractionβ
The SSL principal extraction is now configurable with GATEWAY_SSL_PRINCIPAL_MAPPING_RULES
it will follow the same rules as Kafka.
General Fixes π¨β
- Quieter responses to Prometheus by not publishing HTTP quantiles in the response
- Topic configuration is now returned in all Gateway modes
- Additional tools have been added to the base image to help with setup and debug: lsof, iotop, htop, iftop
Gateway 2.5.2β
General fixes π¨β
PUT
can be used in both the User API and the Interceptor API to create resources when they don't already exist
Gateway 2.5.1β
General fixes π¨β
- Append data quality error reporting to the the header produced in the Dead Letter Queue
- Added a swagger endpoint at
/swagger
Gateway 2.5.0β
Schema-based data contract validationβ
Check and enforce data quality at the schema level, preventing outages from missing or badly formatted records.
Gateway can now validate payloads against specific constraints for AvroSchema and Protobuf using the same validations provided by JsonSchema, such as:
- Number:
minimum
,maximum
,exclusiveMinimum
,exclusiveMaximum
,multipleOf
- String:
minLength
,maxLength
,pattern
,format
- Collections:
maxItems
,minItems
If criteria are not met then informative feedback is provided such as, name is too short (1 < 3)
, hobbies has too few items (3 < 5)
as well as the topic and field level information.
Example: Without validation
{
"namespace": "schema.avro",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "age", "type": "int"},
{"name": "email", "type": "string"}
}
Example: With validation
{
"namespace": "schema.avro",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string", "minLength": 3, "maxLength": 50},
{"name": "age", "type": "int", "minimum": 18},
{"name": "email", "type": "string", "format": "email"}
}
Sounds interesting, try it out for yourself with this demo or come chat to us for proper evaluation.
This can be combined with the SQL data quality producer plugin described below, or standalone.
SQL data quality checks on produceβ
Check data quality with a SQL statement before it hits the cluster, ensure the data produced is valid.
If we want our cars
topic only to allow messages where the cars are red AND younger than 2020, we can write this out as a SQL statement in the plugin's config, and post it to the Gateway, e.g.
{
"statement": "SELECT * FROM cars WHERE color = 'red' and record.key.year > 2020",
"action": "BLOCK_WHOLE_BATCH",
"deadLetterTopic": "dead-letter-topic"
}
Messages not meeting this criteria should have the whole batch blocked, however, we also have the option to block only the bad messages, or allow them in and log the action in the audit log.
Rejected messages will throw up informative feedback on why the data quality is insufficient such as record is not produced because year is not > 2020
, or because color is not red
. These error messages will also added to the message header.
We also have a demo for you to try yourself.
Set config fields using environment variablesβ
Be able to alias all interceptor config fields using environment variables.
Set client ID on actionβ
ClientId is an optional field that helps identify applications within your Kafka. Requiring this is set presents opportunities such as speedier debugging by narrowing down which applications affect which messages, or quota management.
Rather than simply block messages that don't have it set, you can instead choose to override the message metadata to set one. This can be done on all Kafka verbs i.e. produce, consume, admin actions and more.
ARM buildβ
Conduktor Gateway is now available in an ARM build, not just AMD, to provide more flexibility to your deployment machine choices.
Interceptor API Upsert Supportβ
The interceptor API will now upsert (create if doesn't exist, update if exists) when making PUT
calls.
General fixes π¨β
- Fixed support for additional Kafka topic configuration properties such as
retention.ms = -1
Gateway 2.4.0β
User management unificationβ
We refactored all authentication process to unify some concern accross all authentication mechanism in Gateway.
Previously, the gateway user was defined differently depending of your authentication choice (mTls, SASL plain, SASK Oauthbearer, Delegated). This made user management and interceptor targeting complex.
Today we've uncoupled the authentication process from the user identification to leverage the UserMapping
added in Gateway 2.2.0
, for all authentication processes.
All sucessfull authentication process will result in a Principal
based on exhanged credentials (See information below to know how principal is detected based on your choice). This principal will be used to search in UserMapping
the associated gateway user and it's information.
You can now easily create and manage Gateway users, associate them to your authentication provider and define gateway information in the same fashion.
New authentication flow :
API changeβ
The following UserMapping
http API's now provide an optional body field principal
to be able to define the principal for a mapping.
/userMappings/v1
/userMappings/v1/vcluster/{vcluster}
FAQβ
- Do I need to recreate all my users?
No, you don't. This new unified flow is compatible with existing credentials and if a mapping doesn't exist it will detect the Gateway User based on authentication information. User mapping will always have priority over authentication specific information like claims for SASL with Oauthbearer. That means that you can start managing your users with UserMapping
while keeping your existing credentials.
- What is the principal extracted for my authentication ?
Here is the list by authentication mechanism that will be used as source for the principal:
Authentication | Principal |
---|---|
mTls | Certificate subject (ex: CN=myuser ) |
Sasl (Plain) | username |
Sasl (Oauthbearer) | Token subject (sub claim by default) |
Sasl (Delegated) | username |
- Is a principal and username the same thing?
No, depending on your authentication provider, the principal (the authorization id exchanged and validated) could be complex to manage. To solve this issue, a UserMapping
defines a username
and a principal
.
principal
is the field used to search for a user based on an authenticationusername
is the field to identify and target a user for Gateway processes (ACL, Interceptor targeting, ...)
By default if a principal
is not defined when creating a UserMapping
we define that the principal
is the same as username
.
Concentrated topicsβ
Logical partitions are no longer automatically remapped to another partition when their backing partitions are deleted.
Virtual clustersβ
Virtual clusters are currently under rework and this release introduces some deprecations:
- We removed flags for Virtual topics no longer used
- We remove the ability to mark a real topic as physically deleted (a legacy product feature)
- We removed some string interpolation in the real topic concentration that were unused
- We deprecated some fields in the response of the internal APIs (these APIs will probably go away in the next release)
General fixes π¨β
- Concentration: Fix transaction support (but the support is still a bit experimental)
- Concentration: Fix retention not emulated on
policy=compacted,deleted
- Concentration: Fix retention emulation issue that could cause some message to reappear after decreasing retention
- Disabled auto topic creation in more scenarios
Gateway 2.3.2β
General fixes π¨β
- ACL would not be applied in
GATEWAY_SECURITY
mode.
Gateway 2.3.1β
Enhancementsβ
Gateway modeβ
You can now choose to setup your gateway in one of the following mode
KAFKA_SECURITY
where your credentials and ACL handled your target Kafka clusterGATEWAY_SECURITY
where your credentials and ACL handled by GatewayVCLUSTER
where your virtual clusters, credentials and ACL handled by Gateway
Previous GATEWAY_FEATURE_FLAGS_SINGLE_TENANT
and GATEWAY_FEATURE_FLAGS_MULTI_TENANCY
are migrated automatically.
ACL fixesβ
Few ACL ordering/conflict patterns are now resolved. We are now following KIP679
Better error managementβ
Upon duplicate UserMapping creation we are now returning a STATUS_CODE_CONFLICT
error code (or 409).
Gateway and Broker configβ
Gateway properties are now visible in the broker properties page
General fixes π¨β
- We renamed
DuplicateResourcesPlugin
intoDuplicateMessagesPlugin
to better reflect the intent of the plugin.
Console 1.20.0β
Release date: 2023-12-12
Features β¨β
- UI Navigation Overhaul
- Sort, Filter and Customize the Topic List & Consumer Group List
- Import records from CSV
- Other features and improvements
- Fixes
UI Navigation Overhaulβ
Transition more seamlessly between different areas of the Console with a new UI experience. Everything Kafka related can now be accessed from the side bar, making it easier to navigate directly to your desired location in even fewer clicks. This change consolidates the app experience and puts all of Conduktor's power at your fingertips!
Sort, Filter and Customize the Topic List & Consumer Group Listβ
It probably looks underwhelming on the surface, but trust us... it's big! π
On the menu:
- Sortable columns (size, messages count, partitions, ...). You can now sort by what matters to you. For example, the highest message count or the most partitions.
- Instant results. We are not querying Kafka, so it's fast whether you have 10 topics or 1,000 topics.
- Choose the columns that matter and hide away the ones that don't.
- Filters on name, tags and other resource metadata such as consumer group state.
Need more? Give us feedback on smart tables.
Import records from CSVβ
Need to dump previously exported data back into Kafka? See how your application responsds to pre-prepared test data? We (finally) got you covered! From within a topic, navigate to the producer tab to utilize the new import CSV functionality.
Give us feedback on this feature.
Other features and improvementsβ
When adding a filter to search in a specific field, the input is now an autocomplete text field instead of a dropdown list. It's now even quicker to create your own filters on the fly!
The Table view quick filter buttons now generates Simple Filters instead of advanced JS filters.
Say goodbye to internal UUIDs! URLs are now taking advantage of the user-defined technical id making sharable links all the more readable with colleagues.
Fixes π¨β
- Hide Provider tab secrets on the Cluster Configuration page
- Adding an ACL in the Service Account page now adds the entry at the bottom of the list
- Modifying filters in Topic Consume page or Topic/ConsumerGroup Lists page now resets the view to page 1
- Removed labels (
member_host
,consumer_id
andclient_id
) in monitoring metrics to limit data points duplication generated during consumer group re-balances. This could cause ingestion limit issues in Cortex for large deployments - Fixed an issue where wrong data could be displayed when recreating a different cluster with the same technical id
- Added support for Console certificates when making calls to Gateway API
- Optimized Memory configuration
-XX:+UseContainerSupport -XX:MaxRAMPercentage=70 -XX:MaxDirectMemorySize=100m
- And many more UI fixes throughout the product
Gateway 2.3.0β
Features β¨β
Passthrough enhanced API UXβ
New API paths have been added to the API for when GW is in the default Passthrough
mode. This simplifies some of the interceptor paths by removing the need to include /vcluster/passthrough/
.
Simple secret managementβ
Previously secrets had to be defined in the configuration of interceptors. Now, secrets can be stored on the client side setup in an environment variable which can be used by the interceptor.
Before
{
...
"additionalConfigs": {
"schema.registry.url": "yourUrl",
"basic.auth.credentials.source": "some_source",
"keySecretId": "password-secret"
}
...
}
New option
{
...
"additionalConfigs": {
"schema.registry.url": "${SR_URL}",
"basic.auth.credentials.source": "${SR_BASIC_AUTH_CRED_SRC}",
"basic.auth.user.info": "${SR_BASIC_AUTH_USER_INFO}"
}
...
}
Further details are provided in the supporting documentation on the plugin page.
Enhancementsβ
Encryptionβ
You now have the option of storing encryption configuration within a topic, rather than in the headers of the messages. This is a design option to be considered. Storing in the topic requires less storage, but now messages are no longer self-sufficient and will depend on this topic data. Set the environment variable for the name of the topic to be used and you're good to go. See the docs for more.
Audit log filteringβ
Finding an issue or a specific event in your Kafka isn't always straightforward, especially on many physical, or virtual, clusters. With enriched properties of the audit log we can now filter on;
- Topic names
- APIKeys
- VCluster names
- Usernames
- Consumer Group Ids
- topicPartitions
General fixes π¨β
- Changed behaviour for field level actions to be more lenient. Gateway will ignore fields that don't match the encryption or masking interceptor configuration, rather than throwing an exception
- Changed behaviour for Oauth authorisation to be more secure. Gateway will prioritise Conduktor user-mappings when looking for which vcluster to connect to, if no user-mapping exists it fallback on claims, if no claim exist, the authorisation fails.
- Fixed some typos in logs
- Bumped some internal versions to reduce CVE risk
- Fixed an issue that prevented the use of large message support on MSK
- Reduced image size through some optimisations
- Product analytics startup event restored, minimal Gateway start data is collected on launch to enhance product development. This can be deactivated with the environemnt variable
- Removed stack trace dumps on Safeguard catches
- Fixed SQL topics in single-tenant Gateway mode
- Further fixes for our Helm chart deployment options
Gateway 2.2.2β
Features β¨β
Full message encryptionβ
Conduktor Gateway gives you the flexibility and power of field-level encryption. However, sometimes you just want the simplicity of having the whole message encrypted. With the latest release you can now use the encryption plugin to encrypt the entire message at once.
Config for the interceptor requires connection to your KMS, or built-in, as well as how you wish to encrypt.
{
"name": "fullMessage",
"priority": 1,
"pluginClass": "io.conduktor.gateway.interceptor.EncryptPlugin",
"config": {
"topic": ".*",
"payload": {
"keySecretId": "myKeySecretId",
"algorithm": {
"type": "AES_GCM",
"kms": "IN_MEMORY"
}
}
}
}
With a simple interceptor added we can acheive full message encryption.
Details are provided in the supporting documentation on the plugin page.