This feature is available with Conduktor Scale Plus only.
Overview
Self-service helps you scale Kafka usage in your organization by facilitating collaboration between the central platform team and application teams. It simplifies and automates processes, establishes clear rules/ways of working and standardizes the creation and management of Kafka resources. This approach brings governance into your enterprise through concepts like ownership and applications, delegating operations to the application teams rather than the central platform team.Benefits for central platform teams
- Define general rules of the game
- Enforce naming conventions
- Safeguard from invalid or expensive configurations (e.g. high partition numbers, etc.)
- Declare applications and their rights
Benefits for applications teams
- Autonomy and ownership over your resources
- Isolation using application namespaces
- Collaboration through delegating permissions without the need to interact with the central platform team
- Discoverability through topic catalog in Console
Concepts
Self-service relies on a central concept, the Application, which dictates ownership of Kafka resources. Here are the relationships between Self-service resources.
Central platform team resources
Application
An application represents a streaming app or data pipeline that is responsible for producing, consuming or processing data from Kafka. Applications give business context to Kafka resources (topics, consumer groups and subjects) that directly relate to the functioning of that application or pipeline. ExampleApplication instance
Applications are generally deployed to one or more Kafka clusters, typically to align with the organization’s development cycle or environments. We call this concept the application instance. Each application instance:- is linked to a Kafka cluster and a service account
- has ownership of Kafka resources (topics, consumer groups, subjects)
- grants permissions for
- the service account, using Kafka ACLs (
Read
andWrite
on Topics,Read
on ConsumerGroups) and - the application owner group, using RBAC (
Admin
permission) in Console
- the service account, using Kafka ACLs (
Application instance policies
Application instance policies restrict the creation of resources following certain rules. These rules can be related to Kafka configs but can also apply to metadata. ExampleApplication team resources
Once an application and application instance are defined, application teams can organize and structure their applications as they see fit. There are two groups of resources where application teams are given autonomy:- Kafka-related resources:
Topic
,Subject
,Connector
,ApplicationInstancePermission
. - Console-related resources: in particular
ApplicationGroup
, allowing to define internally who can do what within the team.
Kafka resources
This is how application teams can create Kafka resources they need for their applications.Application instance permissions
Application instance permissions lets teams collaborate with each other. Deploying this object will grant permission to thegrantedTo
application instance to:
- its service account (Kafka ACL)
- the application team members in Console
Application group
Create an application group to directly reflect how your application operates. You can create as many application groups as required to restrict or represent the different teams that use Console on your application. For example:- support team with only
Read
Access in production - devOps team with extended access across all environments
- engineering team with higher permissions in dev environment
Resource labels
Labels are key value pairs with no constraints that help you organize and surface business metadata in Console. We recommend that all resources that can be created using the Conduktor CLI are annotated with metadata in the form of labels. Find out which resources are currently supported. ExampleLimited ownership mode
To help organizations transition to Self-service more easily, we’ve added a new attribute to ApplicationInstance to let platform teams decide the level of autonomy to provide to application teams.ownershipMode: ALL
(default value): delegates all permissions related to that resource to the application team.ownershipMode: LIMITED
: delegates only a subset of the available permissions to the application team.
Permissions with ownershipMode: LIMITED | Description |
---|---|
Topic | |
topicEditConfig | Permission to edit the topic configuration. |
topicCreate | Permission to create a new topic. |
topicDelete | Permission to delete the topic. |
topicAddPartition | Permission to add partitions to the topic. |
Subject | |
subjectCreateUpdate | Permission to create or update the subject. |
subjectDelete | Permission to delete the subject. |
subjectEditCompatibility | Permission to edit the subject compatibility settings. |
Consumer Group | |
consumerGroupCreate | Permission to create a new consumer group. |
consumerGroupDelete | Permission to delete the consumer group. |
Kafka Connect | |
kafkaConnectorEditConfig | Permission to edit the Kafka Connect configuration. |
kafkaConnectorDelete | Permission to delete connectors. |
kafkaConnectorCreate | Permission to create new connectors. |
Self-service UI
Self-service is currently principally managed with the Conduktor CLI. The Console UI reconciles actions executed via the CLI to provide read-only views in the Application Catalog and Topic Catalog pages in Console. This promotes discoverability of Kafka resources with business context within your organization.
Self-service UI is almost entirely read-only for now. Use the CLI to interact with the product.
Topic Catalog page
In Console, the Topic Catalog page lets you search through the topics marked as public deployed in your organization. You can filter by application, Kafka cluster and the topic metadata.
Application Catalog page
In Console, the Application Catalog page lets you search through applications deployed in your organization. You can search by any available element such as owner and click on a required application.Application Catalog list
The list page shows all applications deployed in your organization. You can search by any element available in the list: name, description, owner.
Application details
The application details page summarizes all the information that relates to that application:- general information (like name)
- application instances and ownership
- subscribed topics
- shared topics
- application groups

Application instance
The application instance details page summarizes all the information relating to the specific application instance:- ownership and resource policies
- external access information
- resources owned by the application instance
- alerts related to the application instance
- related API keys
Application groups
The application groups details page offers access management options:- Members tab: manage group membership, making it easier to control who has access to application resources.
- Resource access tab: granular control over topic, consumer group, subject and connector permissions for precise access management.
- External groups tab: application groups can map to external groups for streamlined access management.

Audit log events
Event type | Description |
---|---|
SelfService.Application.Create | Self-service application is created. |
SelfService.Application.Update | Self-service application is updated. |
SelfService.Application.Upsert | Self-service application is created or updated. |
SelfService.Application.Delete | Self-service application is deleted. |
SelfService.ApplicationInstance.Create | Self-service application instance is created. |
SelfService.ApplicationInstance.Update | Self-service application instance is updated. |
SelfService.ApplicationInstance.Delete | Self-service application instance is deleted. |
SelfService.ApplicationInstanceApiKey.Create | Self-service application instance API key is created. |
SelfService.ApplicationInstanceApiKey.Delete | Self-service application instance API key is deleted. |
SelfService.ApplicationGroup.Create | Self-service application group is created. |
SelfService.ApplicationGroup.Update | Self-service application group is updated. |
SelfService.ApplicationGroup.Delete | Self-service application group is deleted. |
SelfService.ApplicationPolicy.Create | Self-service application policy is created. |
SelfService.ApplicationPolicy.Update | Self-service application policy is updated. |
SelfService.ApplicationPolicy.Delete | Self-service application policy is deleted. |
SelfService.ApplicationInstancePermission.Create | Permissions are created for an app instance. |
SelfService.ApplicationInstancePermission.Delete | Permissions are deleted for an app instance. |
SelfService.ServiceAccount.Create | Service account is created. |
SelfService.ServiceAccount.Update | Service account is updated. |
SelfService.ServiceAccount.Delete | Service account is deleted. |