Release archive

• Breaking changes
  • Changes to Conduktor.io Labels
  • Important Note for CLI Users
  • Local Users Password policy update
• Features
  • Conduktor Chargeback - Track and Allocate Costs and Resource Consumption
  • Console Homepage
  • Consumer Group pages overhaul
  • Self-Service Topic Catalog visibility
  • Self-Service New Topic Policy Allowed Keys
  • More Audit Log CloudEvents into Kafka
  • Expanded Terraform Provider - Kafka Cluster, Schema Registry, Kafka Connect
• Quality of Life improvements
• Fixes

​

Breaking changes v1.29.0

​

Changes to Conduktor.io Labels

• Labels used for sorting and filtering throughout the product
• Conduktor-specific annotations used to drive behaviors on the resources

• metadata.labels.'conduktor.io/description' → metadata.description
• metadata.labels.'conduktor.io/description.editable' → metadata.descriptionIsEditable

• metadata.labels.'conduktor.io/auto-restart-enabled' → metadata.autoRestart.enabled
• metadata.labels.'conduktor.io/auto-restart-frequency' → metadata.autoRestart.frequencySeconds

​

Important Note for CLI Users

$ conduktor apply -f topic.yaml
Could not apply resource Topic/click.event-stream.avro: Invalid value for: body (Couldn't decode key. at 'metadata.labels.conduktor.io/description')

​

Local Users Password policy update

• At least 8 characters in length
• Includes at least 1 uppercase letter, 1 lowercase letter, 1 number, and 1 special symbol

2024-11-21T14:25:47,434Z [console] ERROR zio-slf4j-logger - zio.Config$Error$InvalidData: (Invalid data at admin: Password must contain at least 8 characters including 1 uppercase letter, 1 lowercase letter, 1 number and 1 special symbol)

​

Features v1.29.0

​

Conduktor Chargeback - Track and Allocate Costs and Resource Consumption

​

Console Homepage

• The health of your Kafka Cluster with a few key metrics and graphs
• The state of Console Indexing modules for this Kafka Cluster
• Quick access to your most recently viewed resources

​

Consumer Group pages overhaul

• Topics tab shows the Consumer Group info grouped by its subscribed Topics
• Members tab shows the Consumer Group info grouped by its active members

​

Self-Service Topic Catalog visibility

---
apiVersion: kafka/v2
kind: Topic
metadata:
  cluster: shadow-it
  name: click.event-stream.avro
  catalogVisibility: PUBLIC # or PRIVATE
spec:
  ...

​

Self-Service New Topic Policy Allowed Keys

---
# Limits the Topic spec.configs to only have retention.ms and cleanup.policy keys
apiVersion: self-service/v1
kind: TopicPolicy
metadata:
  name: "generic-dev-topic"
spec:
  policies:
    spec.configs:
      constraint: AllowedKeys
      keys:
        - retention.ms
        - cleanup.policy

​

More Audit Log CloudEvents into Kafka

• IAM.User.Logout
• IAM.User.Login
• Kafka.ConsumerGroup.Duplicate
• Kafka.ConsumerGroup.Delete
• Kafka.ConsumerGroup.Update ( when we reset the offset of the consumer group )

​

Expanded Terraform Provider - Kafka Cluster, Schema Registry, Kafka Connect

resource "conduktor_kafka_cluster_v2" "confluent" {
  name = "confluent-cluster"
  labels = {
    "env" = "staging"
  }
  spec {
    display_name      = "Confluent Cluster"
    bootstrap_servers = "aaa-aaaa.us-west4.gcp.confluent.cloud:9092"
    properties = {
      "sasl.jaas.config"  = "org.apache.kafka.common.security.plain.PlainLoginModule required username='admin' password='admin-secret';"
      "security.protocol" = "SASL_PLAINTEXT"
      "sasl.mechanism"    = "PLAIN"
    }
    icon                         = "kafka"
    ignore_untrusted_certificate = false
    kafka_flavor = {
      type                     = "Confluent"
      key                      = "yourApiKey123456"
      secret                   = "yourApiSecret123456"
      confluent_environment_id = "env-12345"
      confluent_cluster_id     = "lkc-67890"
    }
    schema_registry = {
      type                         = "ConfluentLike"
      url                          = "https://bbb-bbbb.us-west4.gcp.confluent.cloud:8081"
      ignore_untrusted_certificate = false
      security = {
        type              = "SSLAuth"
        key               = <<EOT
-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----
EOT
        certificate_chain = <<EOT
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
EOT
      }
    }
  }
}

​

Quality of life improvements v1.29.0

• Improved the performance of the Automatic deserializer
• Improved the performance of the Schema Registry indexing process
• Added support for Google Cloud Identity group claims
• Added License expiry warning in the UI when there is less than 30 days left

​

Fixes v1.29.0

• Fixed an issue where Custom Deserializers weren’t working as expected
• Fixed an issue where the ManageClusters permission wasn’t working as expected
• Fixed an issue that prevented creating a KafkaCluster and a Topic on that newly declared KafkaCluster in a single CLI apply command
• Fixed /health/readiness endpoint to return HTTP 503 when the Postgres DB is down
• Fixed an issue where the Message Count wasn’t updated to 0 when emptying a topic
• Fixed an issue where the Pause/Resume button wasn’t visible when a connector was in the Failed state
• Fixed an issue where the Topic creation failure reason wasn’t shown in the UI
  • This helps understand why Topic Creation is rejected (useful for Gateway and Self-Service Topic Policies), and how to modify the topic create request to meet the policy requirements

​

Features

• Conduktor SQL
• Monitoring improvements
• Topic Monitoring
• Cluster Health
• New CLI/API resource Alert
• Shareable Filters
• Tags Become Labels
• Publish AuditLog CloudEvents into Kafka
• Logging API

​

Conduktor SQL

• Querying Kafka message data
• Querying Kafka metadata (such as the offset, partition and timestamp)

conduktor sql 'select * from "kafka-cluster-dev_customer_orders"' -n 2

curl \
 --header "Authorization: $token" \
 --request POST 'localhost:8080/api/public/sql/v1/execute?maxLine=2' \
 --data 'select * from "kafka-cluster-dev_customer_orders"'

​

Monitoring improvements

• Overview will be refactored into Home page
• Cluster Health dashboards and alerts will move under Brokers page
• Topic monitoring will be integrated with Topics page
• Apps monitoring will be integrated with Consumer Groups pages
• Alerts will be integrated as tabs in all the resource pages, similar to the recent changes for Kafka Connect

​

Topic Monitoring

• Produce Rate and Consume Rate
• Disk Usage
• Records (new)

​

Cluster Health

• Produce Rate and Consume Rate
• Disk Usage
• Partitions Count
• Offline, Under Replicated and Under Min ISR Partitions

• Active Controller Count
• Unclean Leader Election

​

New CLI/API resource Alert

---
apiVersion: console/v2
kind: Alert
metadata:
  cluster: local-julien
  name: my-alert
spec:
  type: TopicAlert
  topicName: wikipedia-parsed-DLQ
  metric: MessageCount
  operator: GreaterThan
  threshold: 0

​

Shareable Filters

• Improved collaboration: Share pre-defined views to ensure users are looking at the same subset of data
• Time savings: Speed up troubleshooting and analysis with repeatable filters that share the same or similar criteria
• Consistency and accuracy: Standardized filters across teams and departments reduce the risk of errors or discrepancies that can occur when individuals manually create filters

​

Tags Become Labels

• <key>: <value>

tag-<value>: true

# Tags defined on topic:
- format/avro
- project/supplychain
- team/delivery
- color/blue
- color/red
- wikipedia
- non-prod

# Resulting topic labels after migration
labels:
  format: avro
  project: supplychain
  team: delivery
  tag-color/blue: true # Because conflict on "color"
  tag-color/red: true # Because conflict on "color"
  tag-wikipedia: true # Because wikipedia is not a key value pair
  tag-non-prod: true # Becuase non-prod is not a key value pair

​

Publish AuditLog CloudEvents into Kafka

{
    "specversion" : "1.0",
    "type" : "com.github.pull_request.opened",
    "source" : "https://github.com/cloudevents/spec/pull",
    "subject" : "123",
    "id" : "A234-1234-1234",
    "time" : "2018-04-05T17:31:00Z",
    "comexampleextension1" : "value",
    "comexampleothervalue" : 5,
    "datacontenttype" : "text/xml",
    "data" : "<much wow=\"xml\"/>"
}

{
 "source": "//kafka/kafkacluster/production/topic/website-orders",
 "data": {
  "eventType": "Kafka.Topic.Create",
  // Additional event specific data...
  "metadata": {
   "name": "website-orders",
   "cluster": "production"
  }
  // Additional event specific metadata...
 },
 "datacontenttype": "application/json",
 "id": "ad85122c-0041-421e-b04b-6bc2ec901e08",
 "time": "2024-10-10T07:52:07.483140Z",
 "type": "AuditLogEventType(Kafka,Topic,Create)",
 "specversion": "1.0"
}

​

Logging API

curl -X PUT 'http://localhost:8080/api/public/debug/v1/loggers/io.conduktor.authenticator/DEBUG' \
  -H "Authorization: Bearer $API_KEY"

​

Quality of life improvements v1.28.0

• Updated design and color theme
• Added navigation breadcrumb
• Enhanced error messages throughout the product
• Improved the connector 90 days heatmap
• Declaring an ApplicationInstance with resources ending in * will now fail with this error message
  • Could not apply resource ApplicationInstance/my-app-inst: resource name 'appA-*' is not allowed. Use name 'appA-' with patternType PREFIXED instead

​

Fixes v1.28.0

• Fixed an issue with Topic Policy constraint Range where max value wasn’t inclusive and min could greater than max
• Fixed an issue where Topic Policies were not enforced on Topic configuration changes in Console
• Added an error message when using the copy to clipboard button (for API Keys for instance) fails
• Added checks on local user creation emails
• Added new optional environment variable CDK_SSO_OAUTH2_0_OPENID_METADATADOCUMENT to modify the default discovery .well-known end-point
• Fixed an issue where Avro messages using logical type UUID couldn’t be deserialized properly
• Fixed an issue with Cluster configuration page requiring platform.certificates.create permission to perform the TLS check
• Fixed an issue with “Remove user from group” button which is now disabled for users added by external group mapping
• Prevented the deletion of a group when it is owner of an Application
• Fixed an issue with the “New version” button in the banner that was still showing despite being on the latest version
• Fixed an issue where connections to the AWS glue schema registry would disconnect after a certain time and struggle to reconnect
• Fixed an issue where the “Reprocess message” feature was converting empty string headers to null value
• Fixed all critical and high CVE in console-cortex image
• Fixed an issue with the metric under_replicated_partitions when topics have confluent.placement.constraints property

​

Features

• Kafka Connect Configuration Wizard
• Alerts for Kafka Connect
• Self-service: Limited Ownership mode

​

Kafka Connect Configuration Wizard

• Form is generated with structured configuration groups
• Much nicer error handling, attached to each individual field
• Embedded documentation that helps you understand which fields are required and what are their expected and default values
• Ability to toggle advanced configuration to visualize only the most important fields
• Ability to switch seamlessly between Form View and JSON View at any time.

​

Alerts for Kafka Connect

​

Self-service - Limited Ownership mode

• ApplicationInstance resources configured with ownershipMode: ALL, which is the default, delegates all permissions related to that resource to the Application Team
• ApplicationInstance resources configured with ownershipMode: LIMITED delegates only a subset of the available permissions to the Application Team

​

Quality of Life improvements v1.27.0

• Self-service: External Group Mapping is now available for ApplicationGroup
• The Login page now steers users towards their OIDC provider rather than basic auth when OIDC is enabled

​

Fixes v1.27.0

• Fixed an issue on Consumer group reset offset with the ToDatetime strategy
• Fixed an issue with Console indexing that could occur when deleting and recreating subject
• Fixed a recent regression with default replication factor when creating a topic
• Fixed a recent regression with Broker feature “Similar config” calculation
• Fixed a UI issue when Application Instance was created without any resources
• Fixed several issues around Microsoft Teams Integration to support Teams Workflow webhooks (Step by step guide)
• Fixed Kafka Connect client to use HTTP Proxy JVM configuration
• Switching Kafka cluster from the Topic details page now redirects to the Topic List
• Console doesn’t override the client.id property anymore

​

Deprecation warning for upcoming migration from Tags to Labels

• <key>: <value>

tag-<value>: true

# Tags:
- format/avro
- project/supplychain
- team/delivery
- color/blue
- color/red
- wikipedia
- non-prod

# Result
labels:
  format: avro
  project: supplychain
  team: delivery
  tag-color/blue: true # Because conflict on "color"
  tag-color/red: true # Because conflict on "color"
  tag-wikipedia: true
  tag-non-prod: true

​

Features

• Manage Connectors using the CLI
• Self-service support for Connectors
• Enhanced UI and graphs for Kafka Connect

​

Manage Connectors using the CLI

---
apiVersion: kafka/v2
kind: Connector
metadata:
  connectCluster: kafka-connect
  name: click.my-connector
  labels:
    conduktor.io/auto-restart-enabled: true
    conduktor.io/auto-restart-frequency: 600
spec:
  config:
    connector.class: 'org.apache.kafka.connect.tools.MockSourceConnector'
    tasks.max: '1'
    topic: click.pageviews

​

Self-service support for Connectors

---
apiVersion: self-service/v1
kind: ApplicationInstance
metadata:
  application: "clickstream-app"
  name: "clickstream-dev"
spec:
  cluster: "shadow-it"
  serviceAccount: "sa-clicko"
  resources:
    - type: CONNECTOR
      connectCluster: shadow-connect
      patternType: PREFIXED
      name: "click."

​

Enhanced UI and graphs for Kafka Connect

• Connect Cluster selection screen with a preview of Connector status
• New graphs demonstrating the state of your Connector over time

​

Support for High Availability (HA) Console

​

Quality of life improvements v1.26.0

• The checkbox to skip TLS verification is now always visible
• The YAML for Topic object now allows number in spec.configs. Previously it was mandatory to quote all numbers.
• Self-service Topic Policies are now visible in the UI

​

Fixes v1.26.0

• Topic Policies from Self-service are now properly enforced from the UI, as well as both the API and CLI
• Fix Kafka Connect Cluster list showing invalid number of running tasks

​

Upcoming deprecation warning

• <key>: <value>

tag-<value>: true

# Tags:
- format/avro
- project/supplychain
- team/delivery
- color/blue
- color/red
- wikipedia
- non-prod

# Result
labels:
  format: avro
  project: supplychain
  team: delivery
  tag-color/blue: true # Because conflict on "color"
  tag-color/red: true # Because conflict on "color"
  tag-wikipedia: true
  tag-non-prod: true

​

Breaking changes

​

New Docker image name

docker pull conduktor/conduktor-console:1.25.1

​

Features

• Conduktor Console IaC compatible
  • Manage Cluster connections
  • Short lived token generation on startup
  • Admin and Application Tokens
• Shareable Message Page
• Large Messages Support
• Topic Catalog Details Page
• Audit Last Activity of Users

​

Conduktor Console IaC compatible

​

Manage Cluster Connections

---
apiVersion: console/v2
kind: KafkaCluster
metadata:
  name: cloud-kafka
spec:
  displayName: "Cloud Kafka"
  icon: "kafka"
  color: "#000000"
  bootstrapServers: "localhost:9092"
  properties:
    sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret";
    security.protocol: SASL_SSL
    sasl.mechanism: PLAIN
  schemaRegistry:
    url: http://localhost:8080
    security:
      type: BasicAuth
      username: some_user
      password: some_user

​

Short lived token generation on startup

​

Admin and Application Tokens

​

Shareable Message Page

​

Large Messages Support

​

Topic Catalog Details Page

​

Audit Last Activity of Users

​

Quality of life improvements v1.25.1

• Introduced an intermediate screen for Kafka Connect, allowing you to segment Connectors by each Connect cluster
• Within a Connect cluster, introduced an icon for each connector that clarifies if auto-restart is enabled
• Topic Catalog Search is now case-insensitive
• Improved error message when trying to delete an ApplicationInstance that is referenced elsewhere
• Improved error message when assign ownership on resources already owned by another ApplicationInstance
• CLI delete command can now be applied at the file level, similar to resource creation through apply -f you can now delete -f

​

Fixes v1.25.1

• Fixed an error that occurred when configuring a KsqlDBCluster in the UI
• Fixed a UI issue that caused several dropdowns components to look wrong
• Fixed an error message where expected and actual topic replication factor were inverted in the CLI
• When deleting a Kafka Cluster from Console, the Indexed data is now properly deleted as well
• Upgrade dependencies vulnerable to CVE-2024-21634

​

Fixes

• Fixed a UI issue on Self-service Application Catalog and Topic Policies pages
• Fixed a UI issue on Topic Catalog when listing topics created with empty configs
• Fixed an issue with KqlDB connection test button
• Fixed an issue with the new delete users from group endpoint definition in OpenAPI spec

​

Breaking changes

​

New Docker image name

docker pull conduktor/conduktor-console:1.24.0

​

Change in ApplicationInstance Resource Type from GROUP to CONSUMER_GROUP

---
kind: ApplicationInstance
spec:
  resources:
    - type: CONSUMER_GROUP        # Previously: GROUP
      name: "click."
      patternType: PREFIXED

​

Features

• Self-service
• Subject
• ApplicationGroup
• Topic Catalog
• Manage Groups and Users using the CLI
• Topic list columns Produce Rate and Last Activity
• Active Data Policies in Topic Consume page

​

Self-service

​

Subject

---
apiVersion: v1
kind: Subject
metadata:
  cluster: shadow-it
  name: myPrefix.topic-value
spec:
  schemaFile: schemas/topic.avsc # relative to conduktor CLI execution context
  format: AVRO
  compatibility: FORWARD_TRANSITIVE

​

ApplicationGroup

• Support Team with only Read Access in Production
• DevOps Team with extended access across all environments
• Developers with higher permissions in Dev

# Permissions granted to Console users in the Application
---
apiVersion: v1
kind: ApplicationGroup
metadata:
  application: "clickstream-app"
  name: "clickstream-support"
spec:
  displayName: Support Clickstream
  description: |
    Members of the Support Group are allowed:
      Read access on all the resources
      Can reset offsets
  permissions:
    - appInstance: clickstream-app-dev
      resourceType: TOPIC
      patternType: "LITERAL"
      name: "*" # All owned & subscribed topics
      permissions: ["topicViewConfig", "topicConsume"]
    - appInstance: clickstream-app-dev
      resourceType: GROUP
      patternType: "LITERAL"
      name: "*" # All owned consumer groups
      permissions: ["consumerGroupReset", "consumerGroupView"]
  members:
    - [email protected]
    - [email protected]

​

Topic Catalog

​

Manage Groups and Users using the CLI

---
apiVersion: v2
kind: Group
metadata:
  name: developers-a
spec:
  displayName: "Developers Team A"
  description: "Members of the Team A - Developers"
  externalGroups: 
    - "LDAP-GRP-A-DEV"
  members:
    - [email protected]
    - [email protected]
  permissions:
    - resourceType: TOPIC
      cluster: shadow-it
      patternType: PREFIXED
      name: toto-
      permissions:
        - topicViewConfig
        - topicConsume
        - topicProduce

​

Topic list columns Produce Rate and Last Activity

• Produce Rate is calculated from the two most recent offset values provided by our indexer.
• Last Activity is set to Datetime.now() if the latest offsets have changed since the last Indexing

​

Active Data Policies in Topic Consume page

​

Quality of life improvements v1.24.0

• You can now see all subjects associated to the Schema Id of the current message from the Message Viewer panel
• Added message Compression Type metadata in the Message Viewer panel
• Added buttons to navigate to previous and next message in the Message Viewer panel. Also works with the arrow keys
• The “Generate once” feature in the Produce tab now generates much more realistic, randomized messages, especially for Registry schemas and JSON

• Added a button to force re-balance active Consumer Groups in the Consumer Group details page
• Added a “Test connection” button when adding a KsqlDB cluster in Settings
• Added KsqlDB query Start From selector, equivalent to the SET 'auto.offset.reset' command
• Added an icon in the Kafka Connect list to inform that auto-restart feature is active

• When returning a Forbidden error, the missing permissions are listed in the error message
• New endpoint to add user to a group by email

• Env Variable changed from CDK_TOKEN to CDK_API_KEY to set your Admin or Application API Key
• Added support for Subject field spec.schemaFile. Previous versions of the CLI will only accept spec.schema inlined.

​

Fixes v1.24.0

• Clean monitoring metrics related to brokers that are unreachable
• Fix support of Avro byte arrays encoded as base64 when producing messages
• Fix bulk import of users in case a user already exist
• Fix user creation when the user is not admin but has the right permissions
• Fix class name selector when navigating from one Interceptor to another
• External group mapping: support extraction of roles from both string array and comma separated string
• Fix preview of consumer group offset reset when selecting a specific offset
• Data masking: trim name of policy and fix encoding for URL
• Monitoring: show error in UI if cortex is unreachable
• Fix schema that disappeared from the form input when schema was invalid
• Prevent the creation of an application instance with resources that overlaps
• Fix permissions when 2 application instances define resources on the same cluster
• Fixed an issue where apiVersion was displayed at the end using the CLI

​

Breaking changes coming up

​

New Docker image name

docker pull conduktor/conduktor-console:1.23.0

​

Features

• Self-service
• Topic
• TopicPolicy
• Topic Catalog
• Application API Keys
• Editable columns on the Consume Page

​

Self-service

​

Topic

---
apiVersion: v2
kind: Topic
metadata:
  cluster: shadow-it
  name: click.event-stream.avro
spec:
  replicationFactor: 3
  partitions: 3
  configs:
    min.insync.replicas: '2'
    cleanup.policy: delete
    retention.ms: '60000'

​

TopicPolicy

---
apiVersion: "v1"
kind: "TopicPolicy"
metadata:
  name: "click-naming-rule"
spec:
  policies:
    metadata.name:
      constraint: Match
      pattern: ^click\.(?<event>[a-z0-9-]+)\.(avro|json)$
    spec.replication.factor:
      constraint: OneOf
      values: ["3"]
    spec.configs.retention.ms:
      constraint: Range
      max: 604800000  # 7d
      min: 3600000    # 1h

​

Topic Catalog

​

Application API Keys

​

Editable columns on the Consume Page

​

Quality of life improvements v1.23.0

• SchemaId is now displayed from the Message Viewer panel
• Header count is now displayed from the Message Viewer panel
• The More Options ”…” button has been moved so that it’s available from every Topic details tab
• Added a check to prevent producing empty keys to a compacted topic
• Added an “Add partitions” button in Partitions tab

• The current schema is now inside a read-only area
• Increased the width of the side panel when creating/updating schemas
• Full height is used in the panel to show/edit the schema

• Kafka Connect List can now be sorted by the number of Tasks
• Removing a Connector now properly redirects the user to the Connector list instead of the Configuration tab of the deleted Connector
• Topics column is now sourced from more configuration keys (kafka.topic, kafka.topics, topic, topics)

• Permissions on KafkaConnect and ksqlDB now properly display the name instead of the UUID
• Adding Users to Groups can now be done from the User details page directly
• Added the Group name in the UI to be used in the API or CLI

• Added Gateway version on the Interceptor List page
• Added a configuration option to toggle OIDC logout when logging out from Console
• Searching in screens now trims whitespace from the text supplied

​

Fixes v1.23.0

• Fixed an issue with the Test Connection button that didn’t work after a successful response
• Fixed an issue with the indexing of Confluent Cloud Managed Connect
• Fixed an issue with the Kafka Connect List where filter by Connect Cluster wouldn’t work in some cases
• Fixed an issue with the Schema Registry indexer not properly handling a retriable HTTP error (GOAWAY)
• Fixed an issue with the timezone selector scrolling when resetting offsets for a Consumer Group by timestamp
• Fixed an issue with SSO in Azure environments for users who are members of a large amount of Azure groups
• The following fixes have also been back-ported in 1.22.1
  • Fixed an issue where two ACLs of the same name but with different pattern types (PREFIXED and LITERAL) were merged to the same group within the UI
  • Fixed an issue with OIDC login that could cause an expired session to become stuck and prevent login attempts
  • Fixed an issue with ksqlDB caused by not escaping the Stream or Table name in the query

​

Features

• Added support for Azure Managed Identity for Kafka authentication
• Implement OIDC logout. You may need to update your OIDC configuration to allow the root page of Console as a possible redirect URI

​

Fixes

• Fixed an issue where two ACLs with the same name but with different pattern type (PREFIXED and LITERAL) were merged in the same group in the UI.
• Fixed an issue with OIDC login that could cause an expired sessions to become stuck and prevent login in again.
• Fixed an issue with ksqlDB caused by not escaping the Stream or Table name in the query.

​

Breaking changes coming up

​

New Docker image name

docker pull conduktor/conduktor-console:1.22.0

​

Features

• Topic as a Service becomes Self-service
• Conduktor CLI
• Custom Deserializers

​

Topic as a Service becomes Self-service

• Applications + Environments are migrated to Application and ApplicationInstance
• Cross Application accesses are migrated to ApplicationInstancePermission
• The Application list becomes Application Catalog
• At the moment, we decided that we should control everything from the CLI only. The UI will remain Read-Only for now, but the intention is to bring back UI-driven operations in a future release.

​

Conduktor CLI

• Application
• ApplicationInstance
• ApplicationInstancePermission

---
apiVersion: "v1"
kind: "ApplicationInstance"
metadata:
  application: "clickstream-app"
  name: "clickstream-app-dev"
spec:
  cluster: "shadow-it"
  service-account: "sa-clickstream-dev"
  resources:
    - type: TOPIC
      name: "click."
      patternType: PREFIXED
    - type: GROUP
      name: "click."
      patternType: PREFIXED

​

Custom Deserializers

​

Fixes v1.22.0

• Fixed an issue with controller metrics in Monitoring when the Kafka cluster is using KRaft
• Added support for Broker, Connect, and ksqlDB id field and TLS authentication in the YAML configuration file and Environment variables. This implies you might have a duplicate Connect instance if you use a YAML file with an ID for your Connect cluster. Check the Environment Variables page for more details
• Added new configurations to tune indexing batching and parallelization.s
• Fixed an issue with Azure PostgreSQL preventing the Schema Registry page from displaying properly

​

Features

• Added support for Azure Managed Identity for Kafka authentication
• Implement OIDC logout. You may need to update your OIDC configuration to allow the root page of Console as a possible redirect URI

​

Fixes

• Fixed an issue where two ACLs with the same name but with different pattern type (PREFIXED and LITERAL) were merged in the same group in the UI.
• Fixed an issue with OIDC login that could cause an expired sessions to become stuck and prevent login in again.

​

Fix

• Resolved a problem causing a blank screen after login in certain situations, preventing users from accessing Console.

​

Breaking changes coming up

​

New docker image name

docker pull conduktor/conduktor-console:1.21.0

​

Features

• ksqlDB
• Metrics available via Prometheus
• Smart tables for Connect and Schema Registry
• Add Local Users from the UI
• Fixes

​

ksqlDB

• Browse ksqlDB clusters that are connected to Console
• Visualize all the currently running queries as well as write your own queries or executes statements
• Visualize and act on the running Streams (resulting from CREATE STREAM statements) with the Streams tab
• Visualize and act on the running Streams (resulting from CREATE TABLE statements) with the Tables tab
• Show all the Persistent and Push queries currently running on the ksqlDB Cluster with the Queries tab
• Execute Pull and Pull Queries (SELECT) and Statements (CREATE, DESCRIBE, DROP, …) with the Editor tab

​

Subscribe to metrics via the Prometheus endpoint

​

Smart tables for Kafka Connect and Schema Registry subjects

​

Add Local Users from the UI

​

Fixes v1.21.0

• Added support for complex union-type avro messages in Console Producer
• Fixed a blank screen issue after login due to case-sensitivity bug with email address
• Fixed an issue where Message Reprocessing didn’t work after refreshing the page
• Resolved an issue with MSK role assumption
• Fixed an issue with custom certificates for Schema Registry and Kafka Connect
• Fixed several issues improving indexing performance on large clusters
• Increased cortex ingestion limits for large clusters
• Fixed an issue that occurs when Group: ACLs are present

• Features
  • UI Navigation Overhaul
  • Sort, filter and customize the Topic and Consumer Group lists
  • Import records from CSV
  • Other features and improvements
• Fixes

​

Features

​

UI Navigation Overhaul

​

Sort, filter and customize the Topic and Consumer Group lists

• Sortable columns (size, messages count, partitions, …). You can now sort by what matters to you. For example, the highest message count or the most partitions.
• Instant results. We are not querying Kafka, so it’s fast whether you have 10 topics or 1,000 topics.
• Choose the columns that matter and hide away the ones that don’t.
• Filters on name, tags and other resource metadata such as consumer group state.

​

Import records from CSV

​

Other features and improvements

​

Fixes

• Hide Provider tab secrets on the Cluster Configuration page
• Adding an ACL in the Service Account page now adds the entry at the bottom of the list
• Modifying filters in Topic Consume page or Topic/ConsumerGroup Lists page now resets the view to page 1
• Removed labels (member_host, consumer_id and client_id) in monitoring metrics to limit data points duplication generated during consumer group re-balances. This could cause ingestion limit issues in Cortex for large deployments
• Fixed an issue where wrong data could be displayed when recreating a different cluster with the same technical id
• Added support for Console certificates when making calls to Gateway API
• Optimized Memory configuration -XX:+UseContainerSupport -XX:MaxRAMPercentage=70 -XX:MaxDirectMemorySize=100m
• And many more UI fixes throughout the product

​

Fixes

• Fixed an issue with the Reset Offset Preview in the Consumer Groups details showing incorrect target offsets
• Fixed an issue with Service Account page where prefix ACLs were saved incorrectly (prefix-* instead of prefix-)
• Fixed an issue with Service Account page where forcing an operation to “Not Set” caused an error on save
• Fixed an issue with Audit Log after upgrading from 1.18 and previous versions
• Fixed a UI issue where ACLs could not be deleted by non-Admin group members
• Fixed multiple UI issues with Topic As a Service
• Added new configurations for better OIDC support (allow-unsigned-id-tokens and preferred-jws-algorithm)

​

Fixes

• Massive improvement over the indexing time of your cluster. This is especially notable if you have a large number of consumer groups (> 200) on your Kafka cluster.
• Fixed an issue where Gateway Interceptors could be created/deleted by all users instead of Admin group members.
• Fixed an issue where Monitoring didn’t work for non-Admin group members.
• Fixed an issue that occurred when deleting a user from Console.
• Fixed a UI issue where ACLs were not editable by non-Admin group members.
• Fixed a UI issue where user could not pick the Kafka Connect cluster when creating a new Connector.

​

Features

• Features
  • Consume live messages
  • Consume messages infinitely
  • Consume between dates (and obtain count statistics)
  • Provider Integrations within Conduktor Console
  • Conduktor Gateway Integration
  • Aiven Cloud Integration
  • Confluent Cloud Integration
  • YAML and Environment Variables for Clusters
  • Better memory configuration
• Fixes

​

Consume live messages

​

Consume messages infinitely

​

Consume between dates (and obtain count statistics)

• Show from: {datetime}
• Limit: {datetime}

​

Provider Integrations within Conduktor Console

​

Conduktor Gateway Integration

• Configure end-to-end encryption on Kafka topics
• Enforcing governance on configurations when creating topics

​

Aiven Cloud Integration

• Manage Service Accounts
• Manage ACLs

​

Confluent Cloud Integration

• Manage Service Accounts & ACLs
• Manage API Keys

​

YAML and Environment Variables for Clusters

clusters:
  - id: ccloud
    name: Confluent Cloud ABCD
    bootstrapServers: lkc-abcd.europe-west1.gcp.confluent.cloud:9092
    properties: |
      security.protocol=SASL_SSL
      sasl.mechanism=PLAIN
      sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
        username='<CLUSTER-API-KEY>' \
        password='<CLUSTER-API-PASSWORD>';
    kafkaflavor: # Optional, for Service Accounts & Api Key support
      type: Confluent
      key: "<GLOBAL-API-KEY>"
      secret: "<GLOBAL-API-SECRET>"
      confluentEnvironmentId: "env-abcd"
      confluentClusterId: "lkc-abcd"
    schemaRegistry:
      url: http://abcd.europe-west1.gcp.confluent.cloud/
      security:
        username: <REGISTRY-API-KEY>
        password: <REGISTRY-API-PASSWORD>
    kafkaConnects:
      - id: kafka-connect-1
        name: Connect 1
        url: http://kafka-connect-1:8083/
        security:
          username: username
          password: password

​

Better memory configuration

-XX:+UseContainerSupport -XX:MaxRAMPercentage=80

​

Fixes

• Optimized the AdminClient instances usage across Console, by reusing a shared instance as much as possible. This will hugely decrease the number of AdminClient authentications.
• Fixed an issue on Consumer Group details where unassigned partitions were showing as part of an active member
• Fixed an issue on Consumer Group Page where pagination and search didn’t work together properly
• Fixed an issue with JS filters when keys contain special characters like ’.’ or ’-’ (Use value["custom.key"] instead of value.custom.key)
• Fixed an issue where Console Group description could not be updated
• Added a message and link to documentation when browsing monitoring while it’s not configured
• Fixed an issue that sometimes forced users to cleanup their browser cache after restarting Console
• Removed output logs that were not useful and added others
• Added flushing metrics on AWS S3 or Azure Blob Storage when cortex shuts down, preventing monitoring data loss during restarts or updates
• Added support for array in Environments Variables (ie. CDK_SSO_OAUTH2_0_SCOPES=scope1,scope2)
• Added support for Cortex configuration overrides (mounting a file in /opt/override-configs/cortex.yaml)

​

Fixes

• Fixed an issue that prevented Console to function properly with PgBouncer
• Fixed an issue where properties (default.api.timeout.ms & request.timeout.ms) were mistakenly overridden, leading to possible timeouts on large Kafka Clusters
• (1.18.3) Fixed a performance issue on the Topic List page, leading to possible timeouts on large Kafka Clusters

​

Fix

• Fixed an issue on the Consumer Group List page that caused the page to timeout with a high number of consumer groups.

​

Fixes

• Increased timeout from 30s to 5m when indexing snapshots (used for caching list pages) of Kafka clusters
• Fixed a DB issue that prevented authentication to succeed when the generated token was longer than 255 characters
• Fixed an issue with cortex (conduktor/conduktor-platform-cortex) when console (conduktor/conduktor-platform) is configured with HTTPS
• Fixed an issue where LDAP users could not see or configure Alerts under certain circumstances

​

Important changes to our internal architecture

• We have extracted the ‘Storage & Alerting’ aspect of the Monitoring Solution from the base Console image and moved it to a dependency image
  • If you didn’t use Monitoring, no further action is required from your part
• We have removed the embedded Postgres Database from the Docker image
• We have removed Testing from the base image
• We have rewritten our authentication module. If you have any issue with authentication using LDAP or OIDC with this new release Contact Conduktor Support

​

Features

​

User Friendly Filters

​

Simple Filters

Search for any text in your messages either in your Key, or Value, using basic operators like “equals” or “contains”.

​

Field Filters

If your message is JSON, Avro or Protobuf, find values from specific fields. Operators can match the selected field type.

​

Advanced Filters

For when you require more complex rules, Advanced Filters will give you the full power of JS to construct your own custom filters.

​

New Design for Service Accounts & Kafka ACLs

We’ve overhauled Kafka ACLs and service account management. Visualize, create and edit your Kafka ACLs right within the UI with a simple but powerful design. Pair this with our market leading RBAC capability to empower only those that need it .\

​

Reduced Memory Footprint

​

Fixes

• Fixed a UI issue where Add Partitions wasn’t possible when using Firefox
• Fixed some UI issues with Kafka Connect and RBAC
• Added the “Last Indexed Date” info in the Consumer Groups List, to inform the user about the snapshot age
• Fixed an issue where Consumer Groups Details was populated from the snapshot, instead of AdminClient calls
• Fixed an issue where the Delete Consumer Group modal remained opened after deleting the Group
• Fixed an issue where users with Empty Topic permission were also allowed to Create Topics on the same scope
• Fixed an issue where users were logged out of Console when trying to access Datamasking without permission

​

Features

• Replace the alert threshold column with the full underlying query in the ‘Alerts’ table.
• When creating alerts, you can now type in the dropdown selection of the resource (e.g. topic name, consumer group name) to improve usability when creating/editing alerts.

​

Fixes

• Fix a case whereby saving alert did not preserve the consumer group name filter. Instead, saving or editing an existing alert would default to ‘all consumer groups’.
• Fix N/A representation of delay (lag/s) for consumer groups in ‘Apps Monitoring’ to be more helpful. Lag is now labelled as ‘infinite’ in cases whereby lag exists in the group, but no new data is being received and no members exist in the consumer group.
• Remove the default alert ‘No consumer group should be more than 3 minutes behind’ for new clusters added to your environment. This was causing false positives in customers environments (e.g. non-production groups) and therefore it’s advised to set alerts on explicit groups you wish to be alerted on. For existing clusters, if the alert is causing you to be notified on groups you do not wish for, then it’s recommended to deactivate this alert.

​

Known issue

• Upon adding a new Kafka Cluster, Built-in Alerts & Custom Alerts are not working. You must restart Conduktor after adding the Kafka Cluster for changes to take effect.

​

Fix

• Fixed an issue on the consumer group list screen which could inflate the number of members in a group. This was a result of migrating to the internal ‘snapshot’ cache for improved performance, and missing a case whereby we did not clean up old consumer group members.

​

Known issue

• Upon adding a new Kafka Cluster, Built-in Alerts & Custom Alerts are not working. You must restart Conduktor after adding the Kafka Cluster for changes to take effect.

​

Features

​

Overhauled RBAC (Role-Based Access Control)

​

Public APIs & Support for Automation

• Manage Clusters
• Manage Users, Groups and their respective Permissions (linked with our new RBAC!)

curl -X GET http://localhost:8080/public/v1/groups/project-a/permissions -H "Authorization: Bearer {token}"
[
    {
        "resourceType": "Topic",
        "clusterId": "local",
        "topicPattern": "projectA-*",
        "permissions": [
            "topicConsume",
            "topicViewConfig"
        ]
    },
    {
        "resourceType": "ConsumerGroup",
        "clusterId": "local",
        "consumerGroupPattern": "projectA-*",
        "permissions": [
            "consumerGroupView"
        ]
    }
]

​

Major performance boost to both the Topic and Consumer Group List pages

​

Service Accounts Page

​

Table View in Consume with Quick Filters

​

Precision regarding the timestamp timezone

​

Fixes

• Fixed a UI issue where Clusters configurations that use Kerberos or Oauth did not render properly
• Fixed a regression where Kafka password was displayed in the logs
• Disabled IPV6 listener in the docker image
• Fixed a bug preventing to access Consumer Group Details page from the Consumer Groups tab in Topics Details page
• Added Connection Pooling to reduce the number of active connections to the database by 25%
• Removed an error message in Broker page when there was no partition on the cluster
• Added support for Amazon RDS Proxy

​

Known issues

• Upon adding a new Kafka Cluster, Built-in Alerts & Custom Alerts are not working. You must restart Conduktor after adding the Kafka Cluster for changes to take effect.

​

Features

​

Config ID maps to technicalID

clusters:
  - id: eastern-horizon
    name: My Local Kafka Cluster
    color: '#0013E7'
    bootstrapServers: kafka:9093

​

Set idle session timeout

​

Fixes

• Large topic list handling. Some of our latest and greatest work wasn’t playing nicely when the number of topics was high, we’ve fixed that
• Improved offline environments. Some customers were experiencing issues when using completely offline environments, we’ve made some improvements to error handling, the error messages displayed and how we work with timeouts
• Connector restart history. One of our latest features for automatic connector restarts gave you all the lovely history so you can see what’s going on, perhaps too much ;). We’ve refined this so that each connector only displays it’s own history
• Connect first cluster improvements. When connecting your first cluster there were a few too many buttons available to click, which didn’t take you where you wanted to go. We have disabled more of those to make it even clearer on how to get connected
• App version label restored. Some of you have noticed the app version label in the UI was removed and want it back, we hear you. You’ll now find the current version of Console back where it was before, in the bottom left corner of the UI
• Squashed a bug with trailing slashes getting stuck on blank pages
• Improved error handling for tokens expiring
• Fixed a bug that prevented you to browse consumer groups with an empty name

​

Features

• Automatic connector restarts
• New home experience
• Configurable max session lifetime
• New ‘by broker’ view
• Show offsets in consumer group details

​

Automatic connector restarts (Preview feature)

​

New Home experience

​

Configurable max session lifetime

​

New ‘by broker view’

​

Show offsets in Consumer Group details

​

Fixes

• Fix the JQ filter in Console
• Fix showing metrics for consumer groups with blank name
• Slack channel alerting now supports 500 channels, previously it was restricted at 100

​

Monitoring

​

Features

​

Object storage support for external storage

​

Fixes 🔨

​

Alert channel list increased to 500

• Fixed an issue when settting up alerts you could only pick from the first 100 channels that were available, now the list allows selection from up to 500.

​

Important changes

• Note that in 1.15.0, the Testing application will no longer be started by default in the on-premise deployment. This provides a performance optimization for those customers that are not currently using Testing. If you wish to start Testing alongside Conduktor, you should set an explicit environment variable.

​

Features

• Admin
  • Audit Log has been completely revamped and enriched. Events now contain relevant contextual information (upon click), and logs can now be filtered by a range of criteria (user, resource type, resource name, event, cluster and date).
• Console
  • Tags are now organization-wide, meaning visibility of this additional metadata is shared amongst colleagues in your organization.
  • You can now export data in either CSV or JSON format via the Consumer view.
  • When using multiple JS filters in the Consumer view, these are now combined with logical AND instead of OR.
  • New experience for showing statistics on each resource that enables more of the screen to be used for displaying data.
  • Cluster selection RBAC: Visibility of clusters that can be selected now depends on the users data permissions applied through RBAC.
• Testing
  • Testing is now disabled by default when starting Conduktor. To start Testing alongside Conduktor, you should set an explicit environment variable.
  • Note that in Conduktor 1.15.0, the Testing application will no longer be started by default on our on-premise deployment. It’s still available on our Cloud version.

• Monitoring
  • All monitoring metrics are now provided without any agent dependencies

​

Fix 🔨

• Platform
  • CVEs fixes
• Console
  • Support negative lag in consumer groups (cases whereby the last committed offset is greater than the latest offset)
  • Security fix to stop exposing cluster configuration inside the UI.
• Monitoring
  • Computation of URP metric is set to 0 if we find confluent.log.placement.constraints in the cluster config or confluent.placement.constraints in the topic config. This avoids raising a false positive alert in case of a multi-region Confluent cluster. Note this logic will also be applied in Console in the next release.
• Governance
  • Do not show error toast in Data Masking when a user/group no longer exists in a policy.
• Admin
  • Accept ”/” character in consumer group name

​

Important changes

• This version requires a configuration update to ensure you have root administrator defined. This is mandatory for initialization of the platform, therefore you MUST update your configuration prior to upgrading.

​

Features

• Platform
  • See important changes above. You must ensure you have a root administrator defined in your platform configuration.
  • New onboarding wizard to help setup your environment for a new installation. This can be used to configure license, organization, database and SSO (Oauth2/LDAP) from inside the Conduktor UI. See docs.
  • Added a memory available check at startup to prevent platform running in cases whereby the memory available is less than the memory required by the RUN_MODE being used.
• Console
  • Long resource names are now wrapped to ensure the full string is always visible in the table.
  • We have improved the performance of loading topic names and associated metadata in the topic list screen.
  • We have improved the performance for loading subjects from schema registry.
• Admin
  • It’s now possible to add an SSO user to your organization before their first login. This allows you to allocate appropriate permissions prior to their first login.
  • Notifications screen is now ‘Integrations’, with support added for Microsoft Teams.
• Monitoring
  • Request confirmation from users when deleting alerts.
  • Reduced verbosity of logs from the topic scanner.
  • Integration of Microsoft Teams notifications for alerting.

​

Fixes

• Platform
  • Clean some embedded PostgreSQL temporary files on container restart to prevent database startup error in some edge cases.
  • Remove Content-Length header on web-socket requests, mitigating web-socket issues with some reverse proxy like Envoy.
• Console
  • Fix dropdown selectors in dark mode theme.
  • Consumer: pretty print Protobuf records
• Testing
  • Fix variable definition export.
  • Fix deletion of linked tasks.
• Admin
  • Every admin of the platform is now able to delete a user.
  • Initialize organization during platform startup.
  • For users in Trial, the upgrade license button is no longer crashing the UI.

​

Fixes

• Platform
  • Fix regression on platform database connection default schema used if changed from PostgreSQL default
  • Now fetch OIDC user infos endpoint for user email claim if not present in ID Token during login.
• Console
  • Support references in Schemas
  • Dark mode: fix color of cluster name in cluster dropdown
• Admin
  • Logs: reduce verbosity and avoid cluster configuration leakage

​

Fixes

• Platform
  • Fix an issue whereby platform does not take hostname from platform.external.url. Now the configured external url is always used for SSO callback url resolution.
  • Fix an issue whereby CDK_HTTP_PROXY_* configurations were not being used for OIDC SSO requests. Also set CDK_HTTP_NON_PROXY_HOSTS default value to localhost|127.* when HTTP proxy is configured. See our Http Proxy configuration documentation for more details.
• Testing
  • Fix migration for HTTP certs and some test scenarios

​

Important changes

• This version replaces platform.fqdn configuration with platform.external.url (used for SSO callback URL when using a reverse proxy).

​

Features

• Platform
  • External group mapping for LDAP/OIDC - See docs
  • DB support for TLS - See docs
  • Allow to force platform URL with platform.external.url configuration. This can fix SSO callback URL errors if upstream proxy sends the wrong X-Forwarded-* HTTP headers to the platform.
• Console
  • Support mTLS on Kafka Connect
  • Improve the performance of the consumer and topic lists screen
  • New design for the cluster dropdown has been implemented
• Admin
  • Configure Kafka Connect connection with mTLS
  • Configure an icon to distinguish your clusters
• Testing
  • Add ‘Skip SSL’ functionality to HTTP tasks

​

Fix

• Console
  • Fix connectivity test with RedPanda clusters

​

Important changes

• This version will delete existing user-defined alerts as the data in Cortex changed significantly with the switch towards agent-less monitoring

​

Features

• Platform
  • Remove the default ‘localhost:9092’ cluster from Admin when starting Platform
  • Remove extra local user [email protected] from default platform configuration
  • Print configuration resolution at platform startup if CDK_ROOT_LOG_LEVEL or PLATFORM_STARTUP_LOG_LEVEL are set to DEBUG. See configuration properties and environment variables
  • Added missing EXPOSE directive with default platform port 8080 to the Dockerfile
• Console
  • RBAC: Support for Kafka Connect. See docs
  • Schema registry: Support mTLS connection for Confluent schema registry. See docs
  • Schema registry: Support SSL truststore configured via advanced properties
  • Indicate if configuration entries of brokers are read-only
• Monitoring
  • Monitoring is now moving to agent-less. Once you add a cluster in Admin, it will be monitored without any additional configuration.
• Admin
  • RBAC: Support specifying permissions for Kafka Connect
  • Cluster: Add mTLS configuration for confluent schema registry
  • Cluster: Configure the SSL auth by importing JKS

​

Fixes

• Console
  • Fix plain text connection with RedPanda clusters and Conduktor proxy
  • Fix interaction between uploaded certificates and clusters configured with JKS truststore
  • AWS Glue: Warn when schema creation fails due to incompatibly with previous version
  • If the last used cluster no longer exists, we fallback to the first cluster in the list
  • When creating a topic, we now validate the topic name directly in the UI
  • Consumer: Fix blank screen when trying to open an Avro message

​

Fixes

• Console
  • Fix out of memory issues with misconfigured SSL clusters
  • Performance improvements for cluster selector and consumer groups page
• Testing
  • Restore Embedded Agent on restart, and prevent deleting/regenerating the token
  • Fix Avro uuid logicalType support in consumer
  • Fix Checks UI losing the operator on edit

​

Features

• Console
  • Support for AWS Glue Schema Registry.
  • Topics: a new tab is available to explore the partitions details.
  • Schema Registry: add the ability to review a schema even if it is malformed or erroneous.
  • Kafka Connect: you can now select and reset multiple connectors at once.
  • Kafka Connect: the connectors view have been redesigned to improve its readability
• Monitoring
  • New Overview screen with global health check based on URP, Offline Partitions, Active Controllers, Unclean Elections and Min ISR
  • Support for metrics without agent dependency: Message Count / s, Partitions count, Leader/Partition Skew, Active brokers, Active partitions, Active controllers, Total size of messages
  • Log alerting failures as errors and enable retry on transient failures
• Platform
  • Improve platform logs format and configuration by introducing new environment variable CDK_ROOT_LOG_LEVEL and CDK_ROOT_LOG_COLOR . More on log configuration on documentation.
  • Add support for LDAP client context extra properties configuration.
• Testing
  • Improve agent logs, now displaying the server it connects to (SaaS or on-premise)
  • Parsing errors in checks are now more verbose
• Topic as a Service
  • Teams around applications can be configured with RBAC roles.

​

Fix 🔨

• Admin
  • Prevent error on cluster update if the cluster was configured without properties in input platform configuration.
  • Remove transitive permissions on topic, subjects and consumer groups when a permission is given on a cluster
• Console
  • Consume: Fix WASM execution errors when using jq filters.
  • Consume: Fix parsing of DLQ headers from Spring Boot Cloud Stream
  • Consume: Immediately fail when ACL prevents from consuming a topic instead of hanging forever
  • ACLs: Support ’:’ character in ACL principal names
  • Improve performances of navigation and cluster connection status
  • Consume/Produce: Fix an error related to schema registry when consuming or producing on MSK clusters configured without schema registry
• Platform
  • Fix input clusters configuration properties jmx_scrape_port and node_scrape_port that was not used for cluster monitoring.
  • Test HTTPS certificate and key files permission at platform startup. see documentation
  • Update internal JVM GraalVM to 22.3.1.
  • Upgrade dependencies vulnerable to CVEs
    • CVE-2021-35515
    • CVE-2021-35516
    • CVE-2021-35517
    • CVE-2021-36090
    • CVE-2022-42003
    • CVE-2022-25857
• Testing
  • Fix schemas loading.

​

Fix

• Now allow RBAC permission to be set on schema subjects that contain a / in their name

​

Fix

• When you create a cluster from Admin interface, monitoring is now enabled automatically

​

Major changes

• migration from 1.9.1 or earlier to 1.10.x is not possible with the embedded database mode

​

Fixes

• Platform
  • Improve migration when using an embedded database (compatible with 1.10.2+ versions only)
  • Resolve an issue with platform state in embedded mode
  • Resolved an issue with some monitoring metrics which where not displayed anymore.

​

Major changes

• migration from 1.9.1 or earlier to 1.10.x is not possible with the embedded database mode

​

Fix

• Platform
  • Resolve an issue with forwarded HTTP header in authentication callback

Extended RBAC for Enterprise customers

​

Major changes

• migration from 1.9.1 or earlier to 1.10.x is not possible with the embedded database mode
• Depreciation of internal AUTHENTICATOR_JAVA_OPS enironment variable in favor of AUTHENTICATOR_JAVA_OPTS

​

Features

• Admin
  • Improve the UX of removing users from organization
  • Add audit logs for groups and cluster operations
  • Enhanced support for certificate checks on Schema Registry and Kafka Connect
• Console
  • Consumer: You can now apply a jq filter to better visualize message data
  • RBACs are now available for Consumer Groups, Subjects and ACLs
  • New Brokers pages:
    • Brokers list page: You can see the list of your Cluster’s Brokers
    • Broker details page: You can explore the details of one of your Brokers
  • Reset Offsets: New “Specific Offset” strategy to choose when to reset the offsets to
  • Consume: Add support for Avro logical types
  • ACLs: You can now create and delete ACLs in Console
  • Schema details: The “Structure” tab displaying the structure of Protobuf Schema
• Monitoring
  • Dynamic cluster definition: You can now edit them in Admin and monitoring will pick them up on save
  • Graphs are now updated smoothly as time passes
  • Graphs correctly display time values on axis
  • Improved handling of empty values
• Platform
  • Add support for HTTPS
  • Support for HTTP proxies

​

Fixes

• Admin
  • The “Test Connection” button on the Schema Registry configuration page timeouts faster (5 seconds)
• Console
  • Fixed Subject Updating: When a Subject was created with a different strategy than “Topic Name”, updating the schema would create a new Subject instead of updating it
  • Produce: We now produce null instead of an empty String if the Key and/or Value input is empty
  • Confluent Cloud:
    • The “Reset to default” feature on the Topic’s configuration was not working on Confluent Cloud
    • Our Kafka Connect page was not incorrectly showing errors when some connectors were stopped
    • Create new Connector: no error message was displayed when the Connector configuration was invalid
  • Schema Registry configuration: The “Test Connection” return now a human readable error message explains what the issue is.
• Monitoring
  • Selecting custom dates is correctly applied
  • Alerts update now and appropriately update the interface
  • Better handling of NaN and undefined values for graphs
  • Refreshing the graphs no longer execute when given fixed dates
• Testing
  • Increase defaults timeouts to solve flakiness issue with some providers
  • Fix data masking behavior with topic preview

​

Features

• Platform
  • File based environment variable
    • An environment variable with the *_FILE suffix will be replace by the file content

​

Fix 🔨

• Platform
  • Use business name for relevant service in audit log listing
  • Improve error support for TLS checks on Schema Registry and Kafka Connect
  • Prevent the last admin in an organization from removing themselves
  • Fix a rolling update issue due to DB pool size
• Monitoring
  • Fix the consumer lag graphs from the Apps Monitoring dashboard
• Testing
  • Fix agent error when running on Java 11
  • Fix cryptographic error preventing access from insecure origins

​

Important changes

• Conduktor container now runs as a non-root user conduktor-platform with UID 10001 and GID 0

​

Features

• Platform
  • Openshift support
  • Add CDK_SSL_DEBUG flag to enable SSL debug logs
• Topic as a Service
  • Adds the capability to delete applications
  • Adds the capability to delete an environment

​

Fixes

• Platform
  • Fix modules dependencies. Data masking is now enabled even if governance is disabled
• Topic as a service
  • Select owner after creation in the application form
  • Improve service accounts fields in the application form

​

Features

• Platform
  • Add ldapsearch tool to debug LDAP connections
• Console
  • Improve the UX when there are no clusters configured or when the cluster is unreachable
  • Schema Registry - Schema details: new “Structure” tab displaying the structure of the Schema (only Avro is supported for now)
  • Consumer Groups: You can now create new Consumer Groups from the Consumer Groups page
  • New Audit Log events for Topics,Schema Registry,Consumer Groups and Connectors
• Admin
  • Organization wide certificates management - See Docs
  • Cluster Configuration improvements
    • Add the TLS auto discovery on the schema registry and kafka connectors
    • Upload certificate ability if the TLS is discovered
    • UI improvements for easier auth configuration with SASL, SSL and AWS IAM connections
    • The authentication test now considers the “ignore untrusted certificate flag”
    • The authentication test displays the error message in a panel.
• Monitoring
  • New UI graphing library being used with more detailed data point tracing (tooltip)
• Topic as a Service
  • Applications
    • Access requests tab: You can now inspect if you have any topic access requests pending
    • When you have pending requests - it will be indicated in the application list and in the “Access requests” tab
    • Subscribed applications tab: you can now inspect which application has access to your application resources [by environment/by resource]
• Testing
  • New way to integrate into CI, through a Run Configuration that you can update in the UI
  • When creating a run configuration, we now provide helpers for major CI/CD platforms (Github Action, Circle CI, Gitlab, Jenkins)

​

Fix 🔨

• Platform
  • Ignore SSL validation is now working for LDAPS connections when setting sso.ignoreUntrustedCertificate: true
• Console
  • Performance improvements
  • Kafka Connect: fix the metadata displayed
  • Consumer Groups: display correctly newly created Consumer Groups
  • Consumer Group View: Fix the sort on “Overall lag” column
  • Schema Registry - Subjects list: We now display a “⚠️” icon when a Subject has an issue instead
• Data masking
  • Fixed caching issues
• Monitoring
  • Fix rendering for topic analysis metrics
• Testing
  • Fix missing Testing audit log events
  • Allow overriding Content-type in HTTP headers
  • Ignore orphan tasks
  • Updated docs links to point to the new documentation site
• Topic as a service
  • Fix ‘External topic access’ table behavior

​

Important change when updating to Conduktor 1.18.0 (September 14th, 2023)

​

Monitoring

• If you didn’t use Monitoring: No further action is required on your part. You won’t be able to access the Monitoring module anymore.
• If you used Monitoring: read the documentation to understand how to deploy and configure the dependency image conduktor/conduktor-console-cortex:1.18.0.

​

Embedded database

​

Testing

​

Authentication

​

Features

• Data Masking
  • Scoped masking policies on topic, users and groups.

​

Fix

• Testing
  • Fix audit logs events for Testing
  • Fix links to external docs from inside application
• Admin
  • Fixed timestamps in Admin’s Audit that were incorrect
  • Fix the notifications page

​

Features

• Platform
  • If you face an issue with the platform, we introduce a diagnostic tool. Take a look at the documentation
• Console
  • Consumer Groups: new “Preview Table” in the “Reset Offsets” feature allowing you to see the effect your request will have before executing it
  • Schema Registry - Create Schema form: We now support all the possible naming strategies:
    • Topic Name
    • Record Name
    • Topic + Record Name
    • Custom
• Topic as a service
  • We added a new concept of application that are linked to topics.
• Admin
  • When the “Test Connection” test fails, we now display an error notification

​

Fix 🔨

• Platform
  • Fix LDAP group mapping
• Console
  • Consume: The graph now displays time in a 24h format
  • Produce: It was not possible to change the Charset when producing Strings
  • Topics List: The table was needlessly refreshing the data when changing the Topic name truncation
  • Schema Registry - Subjects list: We now display a “⚠️” icon when a Subject has an issue instead of showing a less useful and noisy error notification
• Admin
  • The “Test Connection” button on the Kafka Cluster & Kafka connect configuration page timeouts faster (5 seconds)

​

Additional Notes 📝

• Testing
  • There are a number of broken links that still point to our old documentation site (docs.testing.conduktor.io).

​

Features

• Platform
  • Add support for LDAPS
• Console
  • New “ACL” page listing for all the ACLs of your Cluster (read-only for now)
  • Topics: You can now add new partitions to a topic
  • Topics list table: We removed the replication factor column from the table
  • Topics list table: We changed how we truncate long topic names
    • We’re now, by default, truncating the beginning of long topic names
    • There’s a new option in the table menu to switch to the previous behaviour (whereby we were truncating the end of long topic names)
• Monitoring
  • It’s now possible to copy/paste any link and share them
• Admin
  • Add the TLS check on the cluster’s bootstrap servers

​

Fix 🔨

• Platform
  • Remove unused configuration fields (auth.local-users[].groups and slack-token )
• Console
  • The “details” part of an error message was not scrollable
  • Create Topic form: when an error was happening, the error was not always displayed in the UI
  • Create Connector form: The error message displayed was not correct
  • Create Connector form: The “validate” feature was not reporting the correct error
  • Produce: Producing with random Avro was sometimes not working
• Testing
  • Fix agent connectivity hanging when using multiple instances of an agent
  • Fix menu tooltips being displayed behind the canvas
  • Fix loader not being centered
  • Fix Text4Shell CVE from org.apache.commons.commons-text-1.9
• Monitoring
  • You can navigate back via your browser without encountering issues
• Admin
  • RBAC: changing a role did not remove old permissions
  • As user with ‘read’ but without ‘edit’ permissions, I can browse the clusters as readonly

​

Features

• Platform
  • Add support of external S3 storage for monitoring data. see documentation
  • Add support of custom Truststore configuration for SSL/TLS connections. see documentation
  • SSO : Add more configuration parameters for LDAP connection. see documentation
• Console
  • Consumer Groups - Reset Offsets: New “datetime” strategy to choose when to reset the offsets to
  • Consumer Groups: You can now duplicate a Consumer Group
  • Topics configuration: You can now update your Topics configuration
  • Topics configuration: You can now reset your Topics configuration
  • IAM support: Our io.conduktor.aws.IAMClientCallbackHandler class used to configure IAM in the Platform now complies with the “credentials provider chain” mechanism of AWS.
    It’ll first try to find your credentials/role on your machine, as software.amazon.msk.auth.iam.IAMClientCallbackHandler would do. If nothing is found, then it’ll use our mechanism. For more info, see documentation.
    To summarize, our io.conduktor.aws.IAMClientCallbackHandler class can now be used as a drop-in replacement of software.amazon.msk.auth.iam.IAMClientCallbackHandler in your Kafka properties:

properties: |
  security.protocol=SASL_SSL
  sasl.mechanism=AWS_MSK_IAM
  sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
  sasl.client.callback.handler.class=io.conduktor.aws.IAMClientCallbackHandler

• Testing
  • Added a structured summary in console at the end of executions
  • AWS IAM : default credential chain compliance
• Monitoring
  • It’s now possible to select a custom date/time range via the calender selection
  • Number of messages in / s is now available even without configuring a jmx agent
• Admin
  • The clusters UI now supports read/write/delete (cluster, schema registry, kafka connect)
  • The clusters configuration int the configuration file can be used as a 1st initialization but is not mandatory anymore to create clusters

​

Fix 🔨

• Platform
  • Remove unused configuration fields (auth.local-users[].groups and slack-token )
• Console
  • Create Topic form: The replication factor was not aligned with the Cluster configuration
  • Create Topic form: Improve error handling
  • CTRL+F is now working in the data viewers
  • Consumer Group: When “Overall Lag” and/or “Members” values were 0, they were displayed as N/A
  • Sometimes, in the top-level bar of the app, your Clusters were reported as “Not connected” while they were connected
  • Sometimes, when producing Avro data, an “Invalid JSON returned. Please try again” error was incorrectly displayed, and the produced data was not correctly displayed
• Testing
  • Fix agent connectivity hanging when using multiple instances of an agent
  • Fix menu tooltips being displayed behind the canvas
  • Fix loader not being centered
  • Fix Text4Shell CVE from org.apache.commons.commons-text-1.9
• Data Masking
  • Fix the creation rule form when a lot of field are added. The “create button” stay accessible now.

​

Features

• Platform
  • Notify when a new Platform version is available
  • Environment variable generation from configuration file https://conduktor.github.io/yaml-to-env/
• Console
  • Create Topic form - Advanced configuration: The user can now configure all the settings of the topic
• Data Masking
  • Apply data masking rules before filters are applied in Console and Testing
• Admin
  • Clusters now have technical id. You can observe this on the cluster configuration screen inside admin/clusters. Soon, URLs inside the Platform solutions will use this technical id, enhancing shareability of URLs.

​

Fix 🔨

• Platform
  • Fix authentication with SSO when no local user definition is provided
  • Fix clusters[].schemaRegistry parsing from env
• Console
  • Producer was producing too many messages when it was configured in automatic produce mode with an elapsed time stop condition
• Testing
  • Prevent connecting two nodes at once
  • Quota error on some executions
• Data Masking
  • Add scroll to Policy form panel
• Admin
  • Fix the cluster creation capacity for enterprise license

​

Reset Offsets

​

Connect: Create Connector

​

Features

• Platform
  • Support ignoreUntrustedCertificate for SSO
• Console
  • Improve error messages displayed to the user on Kafka errors
  • Consume: The Bytes deserializer now uses the Kafka BytesDeserializer instead of returning a base64 version of the raw bytes
  • Consumer Groups - Reset Offsets:
    • It’s now possible to choose which Topic and which partition to reset
    • New “shift by” option
  • Consumer Group: You can now delete a Consumer Group
  • Kafka Connect - Create new connector: It’s now possible to select on which Kafka Connect instance the connector will be created
  • Kafka Connect - It’s now possible to select on which Kafka Connect instance the connector will be paused/resumed/restarted/deleted
• Testing
  • Improve agent version tracking and related warnings
  • Improve dark mode support
• Data Masking
  • Implement scope filtering in datamasking
  • UI improvements
• Admin
  • New sidebar design
  • Clusters have a human friendly “technical id” to ease url sharing

​

Fix 🔨

• Console
  • Fix: Kafka Connect - The connectors data table is now refreshed when an action is performed on a connector
• Testing
  • Fix: Screen/Modals freezing on some actions
  • Fix: Can’t connect multiple edges to some nodes
  • Fix: Reading large CSV files
  • Fix: Text4Shell CVE from org.apache.commons.commons-text-1.9
• Monitoring
  • Fix cluster discovery that failed in some cases
• Data Masking
  • Fix: Number fields are not masked
• Admin
  • Fix: Missing auth events in Audit log

​

Load CSVs from S3 Compatible Storage

​

Features

• Platform
  • Platform configurations can be overridden from environment variables.
• Testing
  • Simplify source selection in checks. Rather than remember the full jq prefix, we now provide a helper for the different data attributes you might want to test (message key, value, headers etc.)
  • Add support of S3 in LoadCSV task
  • Add a comparison data node that allows you to compare data across records consumed from 2 or more distinct topics.

​

Fix 🔨

• Platform
  • Fix links in the help menu
  • Enable solutions in the solution switcher to open in new tab
  • Use the intercom widget on feedback tab
• Console
  • Accept certificate chains in Kafka properties
  • Quick search in data tables is now case insensitive
  • Produce - partition selection dropdown was not working
  • Kafka Connect - Improve connector definition validation
• Testing
  • Accept certificate chains in Kafka properties
• Monitoring
  • Lag metrics now work with TLS
  • Clusters are now displayed as within other apps
• Data masking
  • Fix masking delete policy
  • Align policies table with Platform UI standards

​

Features

• Platform
  • Customize Platform port with PLATFORM_LISTENING_PORT
  • Support for MSK with IAM auth in Console and Testing - See documentation
• Testing
  • Re-generate the agent token after it’s been created

​

Features

• Platform
  • Support for external PostgreSQL - see configuration documentation
• Console
  • Consumer Groups - Reset offsets
  • Kafka Connect - Create, Update, Delete, Pause, Restart operations
  • Schema Registry - Compare schema versions
• Admin
  • Test the connection on Schema Registry

​

Fix 🔨

• Console
  • Group search was using “startWith”, now it uses “contains”
• Admin
  • Fix clusters sort by date

​

Upcoming breaking changes for local users

2024-08-27T18:15:29 [WARN] - Inconsistency detected for plain authentication. Username applicationA is not consistent with validated token created for application-A. SASL configuration should be changed accordingly.

​

New feature: correct offsets on concentrated topics

---
kind: ConcentrationRule
metadata:
  name: myapp-concentrated
spec:
  pattern: myapp-.*
  physicalTopics:
    delete: myapp-concentrated
  autoManaged: false
  offsetCorrectness: true

​

General fixes v3.4.0

• Fixed an issue impacting live consumption from concentrated topics within Console
• Fixed an issue with upserts in API V2 relating to service accounts (reporting updated when the status should be not changed)
• Fixed an issue related to Kafka 3.7 client support, ensuring topic id’s for alias and concentrated topics are distinct from their underlying topics
• Fixed an issue whereby audit logs were not being captured during cluster switching
• Fixed an issue with SQL topics when parsing topic names containing ”-“

​

Known issues v3.4.0

• We are aware of an issue with kcat when the new environment variable GATEWAY_MIN_BROKERID is not aligned with the first BrokerId of your Kafka cluster
  • As a workaround, you can either define GATEWAY_MIN_BROKERID to your first Kafka BrokerId or use kcat with the -E flag

​

Fix v3.3.2

​

General fix v3.3.1

​

Upcoming breaking changes v3.3.0

2024-08-27T18:15:29 [WARN] - Inconsistency detected for plain authentication. Username applicationA is not consistent with validated token created for application-A. SASL configuration should be changed accordingly.

​

New features v3.3.0

• New V2 APIs and CLI support
• Support for HTTPS APIs
• Better UX for ACLs and superUsers
• Enable ACLs for Gateway (excl. Virtual Clusters)
• Enable ACLs for Virtual Clusters
• Encryption enhancements and support clarification
• Field-Level encryption: preserving message format to enhance usability
• Attempt to apply encryption to a message more than once will now fail
• Deprecated support for schema based (tag) encryption with Protobuf

​

New V2 APIs and CLI support

• Interceptor
• GatewayServiceAccount
• GatewayGroup
• ConcentrationRule
• AliasTopic
• VirtualCluster

---
apiVersion: gateway/v2
kind: GatewayGroup
metadata:
  name: groupB
spec:
  members:
    - name: user1
    - name: user2
---
apiVersion: gateway/v2
kind: interceptor
metadata:
  name: enforce-partition-limit
  scope:
    group: groupB
spec:
  pluginClass: io.conduktor.gateway.interceptor.safeguard.CreateTopicPolicyPlugin
  priority: 100
  config:
    numPartition:
      action: BLOCK
      max: 9
      min: 9
    topic: .*

$ conduktor apply -f gateway.yml
GatewayGroup/groupB: Created
Interceptor/enforce-partition-limit: Created

$ conduktor delete GatewayGroup groupB
The group groupB is still used by the following Interceptor(s): enforce-partition-limit

​

Support for HTTPS APIs

​

Better UX for ACLs and superUsers

• ACLs and Super Users on the Gateway (excluding Virtual Clusters) must be configured through Environment Variables.
• ACLs and Super Users on Virtual Clusters must now be driven explicitly through API/CLI.

​

Enable ACLs for Gateway (excl. Virtual Clusters)

GATEWAY_ACL_ENABLED=true # default false
GATEWAY_SUPER_USERS=alice,bob

​

Enable ACLs for Virtual Clusters

---
apiVersion: gateway/v2
kind: VirtualCluster
metadata:
  name: "mon-app-A"
spec:
  aclEnabled: "true" # defaults to false
  superUsers:
  - username1
  - username2

​

Encryption Enhancements and Support Clarification

​

Field-Level Encryption: Preserving Message Format to Enhance Usability

​

Attempt to apply encryption to a message more than once will now fail

​

Deprecated support for Schema Based (tag) encryption with Protobuf

​

General fixes v3.3.0

• Large double values (where > Float Max) are now supported in field-level encryption for Avro and Protobuf
• Bytes and fixed fields now properly supported in field-level encryption for Avro
• Avro unions of two or more values (rather than just a value and a null) are now supported in field-level encryption for Avro
• Schema (tag) based encryption now checks and fails if its config is invalid
• It is not possible to encrypt the headers which the encryption plugin uses to manage its decryption process (as this would render the data unrecoverable)
• Improved log messages for Interceptors that reject actions, such as TopicPolicyPlugin
• Several improvements to the LargeMessage & LargeBatch Interceptors
• Fixed an issue where KCache topic initialization would fail silently and leave Gateway in an unusable state
• Added a new Environment Variable GATEWAY_MIN_BROKERID (default 0) that allows for determinist mapping of brokers and ports
• Improved network stability during Gateway scaling or Kafka topology changes
• Added support for overriding Kafka Producer properties used for Audit Log topic with GATEWAY_AUDIT_LOG_KAFKA_ environment variables
• Removed metric gateway.brokered_active_connections. This was equal to portCount with port mapping and always 1 in host mapping
• Changed metric gateway.request_expired tags: nodeHost/nodePort are replaced by nodeId/clusterId
• Fix default value for GATEWAY_UPSTREAM_THREAD config. The new intended default (number of CPU) previously was (2 x number of CPU).
• Fixed an issue with GATEWAY_ADVERTISED_SNI_PORT that wasn’t working properly
• Add log level for io.confluent packages in default log configuration
• Add default value to non mandatory configuration value for min and max bytes in FetchPolicyInterceptor
• Fix an issue with Concentrated Topics creation with Redpanda

​

Known issues v3.3.0

• We are aware of an issue with kcat when the new environment variable GATEWAY_MIN_BROKERID is not aligned with the first BrokerId of your Kafka cluster.
  • As a workaround, you can either define GATEWAY_MIN_BROKERID to your first Kafka BrokerId or use kcat with the -E flag
• It is not possible to add Service Accounts to GatewayGroups using API V2 unless they are previously declared as GatewayServiceAccount.
  • This is not a wanted behavior, especially for OAuth or Delegated Kafka Authentication where declaring a GatewayServiceAccount should not be needed. We’ll address this issue in a follow-up release
  • API V1 (user-mapping) is not impacted
• If you perform a rolling upgrade to 3.3.0, Gateway nodes in earlier versions will show the following error in the logs: [ERROR] [KafkaCache:1007] - Failed to deserialize a value org.apache.avro.AvroTypeException: Expected field name not found: clusterId
  • This is fine and will not cause any further problems
• If you use Virtual Clusters and ACLs: After updating to 3.3.0, you must manage VirtualCluster’s ACL and superUsers through V2 API.

​

Upcoming breaking changes v3.2.2

2024-08-27T18:15:29 [WARN] - Inconsistency detected for plain authentication. Username applicationA is not consistent with validated token created for application-A. SASL configuration should be changed accordingly.

​

General fixes v3.2.2

• Fixed a severe authentication issue with Gateway generated tokens that could lead to a different user being authenticated, effectively causing elevated privileges under certain conditions.
• Fixed an issue where GATEWAY_SNI_HOST_SEPARATOR couldn’t be set to the value -
• Fixed an issue where GATEWAY_SNI_HOST_SEPARATOR wasn’t properly taken in account
• Fixed an issue with GATEWAY_ADVERTISED_SNI_PORT that wasn’t working properly

​

General fix v3.2.1

​

Breaking changes v3.2.0

​

Two new backing topics are required for Gateway

• Virtual Clusters that existed only through creation of UserMappings or Interceptors targeting
• GatewayGroups that existed only through UserMappings

• _conduktor_gateway_vclusters
• _conduktor_gateway_groups

• GATEWAY_VCLUSTERS_TOPIC
• GATEWAY_GROUPS_TOPIC

​

Changes to ACL support on Gateway

​

Changes for Gateway v3.2.0

​

Preview for Gateway v3.3

​

Enable ACLs for Gateway (excl. Virtual Clusters)

GATEWAY_ACL_ENABLED=true
GATEWAY_SUPER_USERS=alice,bob

​

Enable ACLs for Virtual Clusters

---
apiVersion: gateway/v2
kind: VirtualCluster
metadata:
  name: "mon-app-A"
spec:
  aclEnabled: true # defaults to false
  superUsers:
  - username1
  - username2

​

General fixes v3.2.0

• Fixed an issue with Field-level Avro encryption/decryption relating to numeric fields:
  • When using partial decryption with Avro schema registry, any numeric values (int, long, float, double) that are not being decrypted will instead be masked with the minimum (most negative) value for the numeric type
  • This is to ensure the field is compliant with the original type in the Avro schema
• Fixed an issue with the ClientIdRequired Policy that wasn’t properly overriding the ClientId
• Fixed an issue to ensure all active connections are closed, and clients transition quicker to the new cluster during cluster switching

​

General fixes v3.1.1

• Performance is improved when using a large number of interceptors (backported in 3.0.5)
• Pre-create folders when using RocksDB as a cache backend
• Moved the Schema Id to the headers when using field level encryption with Avro

​

AclsInterceptorPlugin removed

​

Features v3.1.0

• Concentrated Topics can now be created with auto-managed flag. Backing topics will be automatically created and extended.
• Added support for Azure Managed Identity for Kafka authentication
• Added an optional configuration for SNI routing to define the separator to use when building host domain for brokers
• Added more context relative to interceptors in Audit logs
• Added the client & version (kafka-client, librdkafka, …) of the client in the Audit logs on CONNECTED event

​

General fixes v3.1.0

• Added Schema Registry validation on encryption plugins
• Fixed an issue where the KMS Key would not be created if it didn’t exist
• Fixed an issue with logger API

​

Performance improvement v3.0.5

​

Performance improvements v3.0.4

• Consumer group membership is no longer loaded synchronously
• Optimize hostname resolution for ACL

​

General fixes v3.0.4

• GATEWAY_DOWNSTREAM_THREAD and GATEWAY_UPSTREAM_THREAD are now correctly gathering the number of cores
• in LargeMessageHandlingPlugin plugin, honor correctly the localCacheExpireAfterWriteInSeconds property

​

General fixes v3.0.2

• Fixed a race condition when closing connections (i.e. when Gateway detects a broker is removed from the cluster) that was causing restarts/timeouts
• Fix duplicated key exception when rebuilding fetch request with duplicated topics
• FIX NPE when handling expired ApiVersions requests
• Added a check to validate schema registry connection and provide more meaningful errors for schema-based encryption
• Added a check against defaultAlgorithm used in the encryption Interceptor to ensure it’s a valid enum value, and avoid overriding with defaults
• Fixed an issue with externalStorage set to true in the encryption Interceptor that was failing to store headers in a separate internal topic
• Ensure that if the encryption algorithm is changed, a new entry appears in the internal topic used to store headers
• Default namespace is now applied properly on schema-based encryption
• Added support encryption/decryption of AVRO bytes and enums types

​

General fixes v3.0.1

• Fixed some issues with Encryption when the value is a tombstone.
• Fixed some inconsistencies between the OpenAPI Spec and the actual implementation.
• Fixed a memory leak when using the default GATEWAY_UPSTREAM_CONNECTION_POOL_TYPE.
• Added a startup check to prevent an incompatible configuration: GATEWAY_UPSTREAM_CONNECTION_POOL_TYPE=ROUND_ROBIN with delegated authentication.

​

Features v3.0.0

​

Interceptor targeting