Release date: 2024-11-25

• Breaking Changes 💣
  • Changes to Conduktor.io Labels
  • Local Users Password policy update
• Features ✨
  • Conduktor Chargeback
  • Console Homepage
  • Consumer Group pages overhaul
  • Self-Service Topic Catalog visibility
  • Self-Service New Topic Policy Allowed Keys
  • More Audit Log CloudEvents into Kafka
  • Expanded Terraform Provider
• Quality of Life improvements
• Fixes 🔨

Breaking Changes 💣​

Changes to Conduktor.io Labels​

We have moved the conduktor.io labels previously available on Connector and Topic resources to new locations.

We recognize this change breaches the API contract and have carefully considered its implications. We remain committed to minimizing breaking changes and will continue to do our best to ensure that such changes are rare and well-justified.

As we expand the number of Conduktor-related features, this separation reduces the risk of conflicts, simplifies resource management, and provides flexibility for future improvements:

• Labels used for sorting and filtering throughout the product
• Conduktor-specific annotations used to drive behaviors on the resources

Topic Resource

• metadata.labels.'conduktor.io/description' → metadata.description
• metadata.labels.'conduktor.io/description.editable' → metadata.descriptionIsEditable

Connector Resource

• metadata.labels.'conduktor.io/auto-restart-enabled' → metadata.autoRestart.enabled
• metadata.labels.'conduktor.io/auto-restart-frequency' → metadata.autoRestart.frequencySeconds

Their associated values have been automatically migrated under the new names.

Important Note for CLI Users​

Applying YAML files with old conduktor.io labels will fail in Conduktor Console 1.29. Be sure to update your YAML files to reflect the new labels.

Example error for outdated YAML:

$ conduktor apply -f topic.yaml
Could not apply resource Topic/click.event-stream.avro: Invalid value for: body (Couldn't decode key. at 'metadata.labels.conduktor.io/description')

Local Users Password policy update​

Passwords for console Local Users configured through YAML and environment variables must comply with a new password policy. This change enforces the following password requirements:

• At least 8 characters in length
• Includes at least 1 uppercase letter, 1 lowercase letter, 1 number, and 1 special symbol

Passwords set in existing installations that do not meet these requirements will prevent Console from starting, throwing a startup error in the logs like this:

2024-11-21T14:25:47,434Z [console] ERROR zio-slf4j-logger - zio.Config$Error$InvalidData: (Invalid data at admin: Password must contain at least 8 characters including 1 uppercase letter, 1 lowercase letter, 1 number and 1 special symbol)

Local Users previously created with the UI are not impacted.
Update the passwords in your YAML or environment variables to meet the new policy before upgrading.

Features ✨​

Conduktor Chargeback: Track and Allocate Costs & Resource Consumption​

Conduktor Chargeback allows organizations to track and allocate costs & usage associated with Kafka resources to different teams or departments based on their data consumption and processing, facilitating cost accountability and management.

Check the dedicated Quickstart to get started with Chargeback.

Console Homepage​

The cluster homepage have been redesigned to present you with the most useful information in one single view:

• The health of your Kafka Cluster with a few key metrics and graphs
• The state of Console Indexing modules for this Kafka Cluster
• Quick access to your most recently viewed resources

Consumer Group pages overhaul​

Consumer group details page is now organized in a way that helps understand the status of your Consumer Group more easily:

• Topics tab shows the Consumer Group info grouped by its subscribed Topics
• Members tab shows the Consumer Group info grouped by its active members

Both Topics and Members lists can be explored further down to individual member/topic-partition assignments.

On top of that graphs are now directly available in the resource page for Lag and Time Lag, as well as a dedicated tab to manage Alerts.

Self-Service Topic Catalog visibility​

You can now choose which Topics should be visible in the Topic Catalog by annotating their YAML.

---
apiVersion: kafka/v2
kind: Topic
metadata:
  cluster: shadow-it
  name: click.event-stream.avro
  catalogVisibility: PUBLIC # or PRIVATE
spec:
  ...

It is also possible to change the default Topic Catalog visibility of all Topics of an Application Instance directly
Check the associated documentation.

Self-Service New Topic Policy Allowed Keys​

We have added a new constraint AllowedKeys to our Self-Service Topic Policy that restricts the properties that can be configured on a Topic.

---
# Limits the Topic spec.configs to only have retention.ms and cleanup.policy keys
apiVersion: self-service/v1
kind: TopicPolicy
metadata:
  name: "generic-dev-topic"
spec:
  policies:
    spec.configs:
      constraint: AllowedKeys
      keys:
        - retention.ms
        - cleanup.policy

This works in conjunction with existing constraints and ensures your Application Teams will only define properties that are allowed by the Central Team.
Read more about our Topic Policy constraints.

More Audit Log CloudEvents into Kafka​

We have made more events available for the Audit Log Publisher:

• IAM.User.Logout
• IAM.User.Login
• Kafka.ConsumerGroup.Duplicate
• Kafka.ConsumerGroup.Delete
• Kafka.ConsumerGroup.Update ( when we reset the offset of the consumer group )

A full list of all the exported audit log event types is published on the Audit Log page.

Expanded Terraform Provider: Kafka Cluster, Schema Registry, Kafka Connect​

We've expanded the scope of our Terraform provider, you can now create additional resources: Kafka cluster with schema registry, and Kafka connect clusters using Terraform. With this version also comes some additional small fixes as requested by the community, see the dedicated provider releases page for the full list.

All examples are available in our provider repo such as the below snippet for a Confluent Kafka cluster and schema registry (with mTLS) definition.

resource "conduktor_kafka_cluster_v2" "confluent" {
  name = "confluent-cluster"
  labels = {
    "env" = "staging"
  }
  spec {
    display_name      = "Confluent Cluster"
    bootstrap_servers = "aaa-aaaa.us-west4.gcp.confluent.cloud:9092"
    properties = {
      "sasl.jaas.config"  = "org.apache.kafka.common.security.plain.PlainLoginModule required username='admin' password='admin-secret';"
      "security.protocol" = "SASL_PLAINTEXT"
      "sasl.mechanism"    = "PLAIN"
    }
    icon                         = "kafka"
    ignore_untrusted_certificate = false
    kafka_flavor = {
      type                     = "Confluent"
      key                      = "yourApiKey123456"
      secret                   = "yourApiSecret123456"
      confluent_environment_id = "env-12345"
      confluent_cluster_id     = "lkc-67890"
    }
    schema_registry = {
      type                         = "ConfluentLike"
      url                          = "https://bbb-bbbb.us-west4.gcp.confluent.cloud:8081"
      ignore_untrusted_certificate = false
      security = {
        type              = "SSLAuth"
        key               = <<EOT
-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----
EOT
        certificate_chain = <<EOT
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
EOT
      }
    }
  }
}

Quality of Life improvements​

• Improved the performance of the Automatic deserializer
• Improved the performance of the Schema Registry indexing process
• Added support for Google Cloud Identity group claims
• Added License expiry warning in the UI when there is less than 30 days left

Fixes 🔨​

• Fixed an issue where Custom Deserializers weren't working as expected
• Fixed an issue where the ManageClusters permission wasn't working as expected
• Fixed an issue that prevented creating a KafkaCluster and a Topic on that newly declared KafkaCluster in a single CLI apply command
• Fixed /health/readiness endpoint to return HTTP 503 when the Postgres DB is down
• Fixed an issue where the Message Count wasn't updated to 0 when emptying a topic
• Fixed an issue where the Pause/Resume button wasn't visible when a connector was in the Failed state
• Fixed an issue where the Topic creation failure reason wasn't shown in the UI
  • This helps understand why Topic Creation is rejected (useful for Gateway and Self-Service Topic Policies), and how to modify the topic create request to meet the policy requirements