Kafka Self-service — topic requests and governance

Enterprise

Overview

Self-service helps you scale Kafka usage in your organization by facilitating collaboration between the central platform team and application teams. It simplifies and automates processes, establishes clear rules/ways of working and standardizes the creation and management of Kafka resources. This approach brings governance into your enterprise through concepts like ownership and applications, delegating operations to the application teams rather than the central platform team.

Benefits for central platform teams

Define general rules of the game
Enforce naming conventions
Safeguard from invalid or expensive configurations (e.g. high partition numbers, etc.)
Declare applications and their rights

Benefits for applications teams

Autonomy and ownership over your resources
Isolation using application namespaces
Collaboration through delegating permissions without the need to interact with the central platform team
Discoverability through topic catalog in

Find out more about Self-service | View resource reference.

Start from a reference template. The conduktor/self-service-template repository shows a production-ready GitHub layout for Self-service: a platform/ directory owned by the central team (clusters, applications, groups, ResourcePolicies), an applications/ directory owned by application teams (topics, subjects, connectors, permissions), CODEOWNERS-based review, and three scope-based GitHub Actions workflows that run conduktor apply --dry-run on pull requests and apply on merge.

Concepts

Self-service relies on a central concept, the Application, which dictates ownership of Kafka resources. Here are the relationships between Self-service resources.

Central platform team resources

Application

An application represents a streaming app or data pipeline that is responsible for producing, consuming or processing data from Kafka. Applications give business context to Kafka resources (topics, consumer groups and subjects) that directly relate to the functioning of that application or pipeline. Example

---
apiVersion: "self-serve/v1"
kind: "Application"
metadata:
  name: "clickstream-app"
spec:
  title: "Clickstream App"
  description: "Clickstream records user’s clicks through their journey on our website"
  owner: "clickstream-group"            # technical-id of the Console Group

Application instance

Applications are generally deployed to one or more Kafka clusters, typically to align with the organization’s development cycle or environments. We call this concept the application instance. Each application instance:

is linked to a Kafka cluster and a
has ownership of Kafka resources (topics, consumer groups, subjects)
grants permissions for
- the service account, using Kafka (Read and Write on Topics, Read on ConsumerGroups)
- the application owner group to create ApplicationGroups in Console
- ApplicationGroup members, based on their assigned resource permissions and instance permissions

Example

# Application Instance (dev environment)
---
apiVersion: "self-serve/v1"
kind: "ApplicationInstance"
metadata:
  application: "clickstream-app"
  name: "clickstream-app-dev"
spec:
  cluster: "shadow-it"
  serviceAccount: "sa-clickstream-dev"
  resources:
    - type: TOPIC
      name: "click."
      patternType: PREFIXED
    - type: CONSUMER_GROUP
      name: "click."
      patternType: PREFIXED
    - type: TRANSACTIONAL_ID
      name: "clickstream-prod-txn"
      patternType: LITERAL

Application instance policies

Application instance policies restrict the creation of resources following certain rules. These rules can be related to Kafka configs but can also apply to metadata. Example

# Policies that restrict the application to a certain range of configurations
# on topic configs, but also on topic metadata
---
apiVersion: "self-serve/v1"
kind: "TopicPolicy"
metadata:
  name: "generic-dev-topic"
spec:
  policies:
    metadata.labels.data-criticality:
      constraint: OneOf
      values: ["C0", "C1", "C2"]
    spec.configs.retention.ms:
      constraint: "Range"
      max: 42
      min: 3

Application team resources

Once an application and application instance are defined, application teams can organize and structure their applications as they see fit. There are two groups of resources where application teams are given autonomy:

Kafka-related resources: Topic, Subject, Connector, ApplicationInstancePermission.
Console-related resources: in particular ApplicationGroup, allowing to define internally who can do what within the team.

Kafka resources

This is how application teams can create Kafka resources they need for their applications.

# Topic example
---
apiVersion: kafka/v2
kind: "Topic"
metadata:
  cluster: "shadow-it"
  name: "click.screen-events"
spec:
  replicationFactor: 3
  partitions: 6
  configs:
    min.insync.replicas: "2"
    cleanup.policy: "delete"
    retention.ms: "60000"

Application instance permissions

Application instance permissions lets teams collaborate with each other. Deploying this object will grant permission to the grantedTo application instance to:

its service account (Kafka ACL)
the application team members in Console

Example

# Read permission granted to the Heatmap Application on click.screen-events topic
---
apiVersion: self-serve/v1
kind: "ApplicationInstancePermission"
metadata:
  application: "clickstream-app"
  appInstance: "clickstream-app-dev"
  name: "clickstream-app-dev-to-heatmap"
spec:
  resource:
    type: TOPIC
    name: "click.screen-events"
    patternType: LITERAL
  permission: READ
  grantedTo: heatmap-app-dev

Application group

Create an application group to directly reflect how your application operates. You can create as many application groups as required to restrict or represent the different teams that use Console on your application. For example:

support team with only Read access in production
devOps team with extended access across all environments
engineering team with higher permissions in dev environment

Application groups support two types of permissions:

Resource permissions (permissions): control what members can do with specific Kafka resources (topics, consumer groups, subjects, connectors).
Instance permissions (instancePermissions): control application-instance-level actions like requesting access, granting access, creating API keys, and managing service accounts.

Upgrading from a previous version? On upgrade to Console 1.45.0, a mirror ApplicationGroup is automatically created for each existing application owner group, preserving all previous permissions. See the Console 1.45.0 release notes for details.

Example

# Permissions granted to Console users in the group, GP-COMPANY-CLICKSTREAM-SUPPORT, for the clickstream-app application
---
apiVersion: self-serve/v1
kind: "ApplicationGroup"
metadata:
  application: "clickstream-app"
  name: "clickstream-support"
spec:
  displayName: Support Clickstream
  description: |
    Members of the Support Group are allowed:
      Read access on all the resources
      Can restart owned connectors
      Can reset offsets
      Can request and grant access
  instancePermissions:
    - appInstance: clickstream-app-dev
      permissions: ["applicationInstancePermissionRequestAccess", "applicationInstancePermissionGrantAccess", "applicationInstanceApiKeyManage"]
  permissions:
    - appInstance: clickstream-app-dev
      resourceType: TOPIC
      patternType: "LITERAL"
      name: "*" # All owned and subscribed topics
      permissions: ["topicViewConfig", "topicConsume"]
    - appInstance: clickstream-app-dev
      resourceType: CONSUMER_GROUP
      patternType: "LITERAL"
      name: "*" # All owned consumer groups
      permissions: ["consumerGroupCreate", "consumerGroupReset", "consumerGroupDelete", "consumerGroupView"]
    - appInstance: clickstream-app-dev
      resourceType: CONNECTOR
      patternType: "LITERAL"
      name: "*" # All owned connectors
      permissions: ["kafkaConnectorViewConfig", "kafkaConnectorStatus", "kafkaConnectPauseResume", "kafkaConnectRestart"]
  members:
    - user1@company.org
    - user2@company.org
  externalGroups:
    - GP-COMPANY-CLICKSTREAM-SUPPORT

Enforce restrictions on ApplicationGroups

After the Console 1.45.0 owner group migration, application teams manage their own ApplicationGroups and decide which permissions and instancePermissions to grant. Platform teams can lock down what teams are allowed to assign themselves with a ResourcePolicy of targetKind: ApplicationGroup, linked to the relevant Application or ApplicationInstance through spec.policyRef. A common pattern is GitOps-only approval: restrict ApplicationGroups to a read-only resource permission set and to safe instance permissions, so any write operation (topicDelete, topicEditConfig, applicationInstancePermissionGrantAccess, serviceAccountManage) has to go through a peer-reviewed pull request rather than the Console UI:

---
apiVersion: self-serve/v1
kind: ResourcePolicy
metadata:
  name: applicationgroup-gitops-only
spec:
  targetKind: ApplicationGroup
  description: Restrict ApplicationGroups to read-only resource permissions and GitOps-only access approval
  rules:
    - condition: spec.permissions.all(p, p.permissions.all(x, x in ["topicViewConfig", "consumerGroupView", "subjectView", "kafkaConnectorStatus"]))
      errorMessage: resource permissions are limited to topicViewConfig, consumerGroupView, subjectView and kafkaConnectorStatus. Permissions like topicDelete and topicEditConfig have to go through the CLI
    - condition: '!has(spec.instancePermissions) || spec.instancePermissions.all(p, p.permissions.all(x, x in ["applicationInstancePermissionRequestAccess", "applicationInstancePermissionProposeAccess", "applicationInstanceApiKeyManage"]))'
      errorMessage: applicationInstancePermissionGrantAccess and serviceAccountManage are not allowed. Use applicationInstancePermissionProposeAccess and approve access via GitOps
    - condition: size(metadata.members) == 0
      errorMessage: spec.members not allowed. Use external group mapping instead

Once linked, any ApplicationGroup the application team creates is limited to the permissions the policy allows — members can view resources, request and propose access, and manage API keys, then use the Topic Catalog to copy the CLI snippet for incoming requests and submit it through their GitOps workflow for approval. Find out more about ApplicationGroup resource policies.

Resource labels

Labels are key value pairs with no constraints that help you organize and surface business metadata in Console. We recommend that all resources that can be created using the Conduktor CLI are annotated with metadata in the form of labels. Find out which resources are currently supported. Example

# Topic annotated with useful metadata
---
apiVersion: kafka/v2
kind: "Topic"
metadata:
  cluster: "shadow-it"
  name: clickstream.events
  labels:
    description: "A description for what kind of data this topic contains."
    business-data-classification: C2
    business-doc-url: "https://confluence.company.org/display/CLICK/Kafka"
    application-code: CLK
    environment-code: dev
spec:
  replicationFactor: 3
  partitions: 6
  configs:
    min.insync.replicas: "2"
    cleanup.policy: "delete"
    retention.ms: "60000"

Limited ownership mode

To help organizations transition to Self-service more easily, we’ve added a new attribute to ApplicationInstance to let platform teams decide the level of autonomy to provide to application teams.

ownershipMode: ALL (default value): delegates all permissions related to that resource to the application team.
ownershipMode: LIMITED: delegates only a subset of the available permissions to the application team.

This is especially useful if the central team have a centralized repository and an existing workflow for a topic (or other resource) creation and wants to own that part of the process. You can provide Self-service while still having application teams go through their pipeline for topic creation, instead of Self-service.

Permissions with `ownershipMode: LIMITED`	Description
Topic
`topicEditConfig`	Permission to edit the topic configuration.
`topicCreate`	Permission to create a new topic.
`topicDelete`	Permission to delete the topic.
`topicAddPartition`	Permission to add partitions to the topic.
Subject
`subjectCreateUpdate`	Permission to create or update the subject.
`subjectDelete`	Permission to delete the subject.
`subjectEditCompatibility`	Permission to edit the subject compatibility settings.
Consumer Group
`consumerGroupCreate`	Permission to create a new consumer group.
`consumerGroupDelete`	Permission to delete the consumer group.
Kafka Connect
`kafkaConnectorEditConfig`	Permission to edit the Kafka Connect configuration.
`kafkaConnectorDelete`	Permission to delete connectors.
`kafkaConnectorCreate`	Permission to create new connectors.

Self-service UI

Self-service is currently principally managed with the Conduktor CLI. The Console UI reconciles actions executed via the CLI to provide read-only views in the Application Catalog and Topic Catalog pages in Console. This promotes discoverability of Kafka resources with business context within your organization.

Topic Catalog page

In Console, the Topic Catalog page lets you search through the topics marked as public deployed in your organization. You can filter by application, Kafka cluster and the topic metadata.

Topic details page

The Topic details page summarizes all the information related to this topic:

General topic information.
Topic access information: displays all applications and application instances with access to this topic:
- You can remove access for your own applications.
- If you have the applicationInstancePermissionGrantAccess instance permission for this application instance, you can revoke access for other applications.

Members with the applicationInstancePermissionGrantAccess instance permission can grant access to any application instance for their application’s topics. If you have the applicationInstancePermissionRequestAccess instance permission, you can request access for your application instance directly through this page:

Check out this tutorial on the topic access requests:

Application Catalog page

In Console, the Application Catalog page lets you search through applications deployed in your organization. You can search by any available element such as owner and click on a required application.

Application Catalog list

The list page shows all applications deployed in your organization. You can search by any element available in the list: name, description, owner.

Click on an application to get to its details page.

Application details

The application details page summarizes all the information that relates to that application:

general information (like name)
application instances and ownership
subscribed topics
shared topics
application groups

If you have the applicationInstanceApiKeyManage instance permission (assigned through an ApplicationGroup), you can generate application instance API keys to be used with the Conduktor CLI to create resources:

Check out this tutorial on the controls:

Application instance

The application instance details page summarizes all the information relating to the specific application instance:

ownership and resource policies
access information
resources owned by the application instance
alerts related to the application instance
related API keys

If you have the applicationInstancePermissionGrantAccess instance permission, you can also manage access to topics owned by your application instance — approving, granting or revoking requests from other applications.

Application groups

The application groups details page offers access management options:

Members tab: manage group membership, making it easier to control who has access to application resources.
Resource access tab: granular control over topic, consumer group, subject and connector permissions for precise access management.
Instance permissions tab: manage application-instance-level permissions like requesting access, granting access, creating API keys, and managing service accounts.
External groups tab: application groups can map to external groups for streamlined access management.

Audit log events

Event type	Description
SelfService.Application.Create	Self-service application is created.
SelfService.Application.Update	Self-service application is updated.
SelfService.Application.Upsert	Self-service application is created or updated.
SelfService.Application.Delete	Self-service application is deleted.
SelfService.ApplicationInstance.Create	Self-service application instance is created.
SelfService.ApplicationInstance.Update	Self-service application instance is updated.
SelfService.ApplicationInstance.Delete	Self-service application instance is deleted.
SelfService.ApplicationInstanceApiKey.Create	Self-service application instance API key is created.
SelfService.ApplicationInstanceApiKey.Delete	Self-service application instance API key is deleted.
SelfService.ApplicationGroup.Create	Self-service application group is created.
SelfService.ApplicationGroup.Update	Self-service application group is updated.
SelfService.ApplicationGroup.Delete	Self-service application group is deleted.
SelfService.ApplicationPolicy.Create	Self-service application policy is created.
SelfService.ApplicationPolicy.Update	Self-service application policy is updated.
SelfService.ApplicationPolicy.Delete	Self-service application policy is deleted.
SelfService.ApplicationInstancePermission.Create	Permissions are created for an app instance.
SelfService.ApplicationInstancePermission.Delete	Permissions are deleted for an app instance.
SelfService.ServiceAccount.Create	Service account is created.
SelfService.ServiceAccount.Update	Service account is updated.
SelfService.ServiceAccount.Delete	Service account is deleted.

Documentation Index

​Overview

​Benefits for central platform teams

​Benefits for applications teams

​Concepts

​Central platform team resources

​Application

​Application instance

​Application instance policies

​Application team resources

​Kafka resources

​Application instance permissions

​Application group

​Enforce restrictions on ApplicationGroups

​Resource labels

​Limited ownership mode

​Self-service UI

​Topic Catalog page

​Topic details page

​Application Catalog page

​Application Catalog list

​Application details

​Application instance

​Application groups

​Audit log events

Overview

Benefits for central platform teams

Benefits for applications teams

Concepts

Central platform team resources

Application

Application instance

Application instance policies

Application team resources

Kafka resources

Application instance permissions

Application group

Enforce restrictions on ApplicationGroups

Resource labels

Limited ownership mode

Self-service UI

Topic Catalog page

Topic details page

Application Catalog page

Application Catalog list

Application details

Application instance

Application groups

Audit log events