Self-service quick start

This guide gives an overview of Conduktor’s Self-service offering through a worked example where you will:

Define an Application for a team, and delegate a slice of Kafka to it
Set up ResourcePolicy guardrails that enforce governance on topics, schemas and permissions
Let the team manage their own topics inside those boundaries
Watch a policy reject a topic that breaks the rules, then grant an approved exception

For the full definition of each resource, see Self-service resource reference.

From our blog: No Kafka data platform without ownership Why ownership is the first domino of a real Kafka data platform, ahead of self-service and scale.

Prefer to watch instead of reading? Check out the full demo in video (24 min):

Why Self-service

As Kafka adoption grows, central teams hit a tradeoff: they review every resource request properly and become the bottleneck, or they skim and let misconfigurations reach production. Access requests are even harder because the central team has to approve them without knowing whether the data is sensitive or who should see it. Approvals go through anyway, periodic reviews slip, and “who can read this topic?” turns into a multi-day search through ACLs, tickets and spreadsheets when an auditor or an incident asks. Teams that want isolation ask for their own cluster, so cluster count grows faster than the workload does. With Conduktor Self-service, responsibility is federated. The platform team declares each application’s owner, what resources it owns, and what rules its resources have to follow. Application teams create, change, share, or delete their own resources inside those boundaries. Conduktor Self-service validates every change at apply time against the rules, with custom error messages to tell them what to fix. Ops protects the infrastructure while the developers with business context make decisions about their data. Moreover, having a proper ownership database turns out to be handy elsewhere in Console:

Stream lineage resolves raw service account principals into named applications, so a graph of sa-7f3a and svc-prod-2 becomes a graph of teams, with a view that hides everything Self-service doesn’t manage.
Chargeback rolls infrastructure cost up by application and by application instance, because it can trace usage back through the service account to the application that produced it.
Alerts belong to an application instance, so the team that owns a topic sees and manages the alerts on it.
The Topic Catalog shows each topic’s owner, schema, and documentation, allowing teams to maintain proper data products other teams can discover and use.

The example repo

Clone the repo that accompanies this guide:

git clone https://github.com/conduktor/self-service-getting-started
cd self-service-getting-started

It’s laid out the way a real Self-service repo is — split by who owns what, not by resource type:

platform/                    # Platform team resources
  clusters/                  #   KafkaCluster definitions
  groups/                    #   Console Groups, mapped from your IdP
  policies/                  #   ResourcePolicy guardrails
  applications/              #   Applications and ApplicationInstances
  exceptions/                #   Agreed policy exceptions
applications/                # Application team resources
  website-analytics/
    dev/                     #   topics, subjects, connectors, groups, permissions
    prod/

Everything under platform/ is managed exclusively by the platform team. Everything under applications/<app>/<instance>/ belongs to the application team that owns it. This delegation is the whole point. Because the platform team created the Application and its instances in advance, the website-analytics team can create, modify and approve changes to their own resources without asking anyone.

In a real setup these two directories are protected by CODEOWNERS so the right people review the right changes, and CI applies them on merge. The self-service-template repo has that wiring. This quick start keeps things manual so you can watch each step.

Key resources

Resource	Owner	What it does
KafkaCluster	Platform team	The Kafka endpoint applications bind to
Group	Platform team	A Console Group, mapped from an external IdP group
Application	Platform team	The unit of ownership, assigned to a Group
ApplicationInstance	Platform team	Binds an Application to a cluster, and sets its resource boundary and policies
ResourcePolicy	Platform team	CEL validation rules enforced at apply time
Topic, Subject, Connector	Application team	The Kafka resources the team manages day to day
ApplicationGroup	Application team	Console UI permissions for team members
ApplicationInstancePermission	Application team	Grants another application access to your topics

Running the example

Set up the demo

Docker is required to run this demo. So is a Conduktor license — Self-service is a licensed feature, and without a license Console runs on the Free plan and rejects every Self-service API call. Book a demo to get one.

Start the local stack — Console, a three-broker Kafka cluster, Schema Registry, and Gateway:
export CDK_LICENSE=<your-license-key> ./start.sh
This returns once Console is ready. You can log in at http://localhost:8080 as admin@conduktor.io : adminP4ss! to watch resources appear as you create them.
Mint an admin token. conduktor token create admin exists for exactly this — it logs in with CDK_USER and CDK_PASSWORD to create your first token, so there’s nothing to copy out of the UI. Pass the credentials to this one command rather than storing them anywhere:
ADMIN_TOKEN=$(docker compose exec -T \ -e CDK_USER=admin@conduktor.io \ -e CDK_PASSWORD='adminP4ss!' \ conduktor-ctl conduktor token create admin quickstart)
-T matters here: without it Docker allocates a TTY and control characters end up inside the token.
Every command below passes its token with -e CDK_API_KEY=.... Don’t set CDK_USER on the container alongside CDK_API_KEY — whatever the auth mode, the CLI refuses to run when both are present and exits with Can't set both CDK_USER and CDK_API_KEY.

The Conduktor CLI runs as a container here for convenience. You can also install it locally.

Apply the platform team’s resources

This is the platform team’s job: set up the cluster, the groups, the guardrails, and the application boundaries. Order matters — each step references the one before it.

Register the Kafka clusters — kafka-local, which the rest of this guide uses, and the Gateway that fronts it. Notice a cluster is itself a managed resource, not something configured by hand in the UI:
docker compose exec -e CDK_API_KEY=$ADMIN_TOKEN conduktor-ctl \ conduktor apply -f platform/clusters/

Create the Console Groups. website-analytics-owners will own the Application, so it has to exist first:

docker compose exec -e CDK_API_KEY=$ADMIN_TOKEN conduktor-ctl \
  conduktor apply -f platform/groups/

Create the policies:

docker compose exec -e CDK_API_KEY=$ADMIN_TOKEN conduktor-ctl \
  conduktor apply -f platform/policies/

Seven ResourcePolicies are applied:

Policy	Target	What it enforces
`topic-naming`	Topic	`<app>.<descriptive-name>` naming
`topic-labels`	Topic	`instance`, `business-unit`, `confidentiality` and `team` labels
`topic-rules-dev`	Topic	RF = 3, partitions 1-3
`topic-rules-prod`	Topic	RF = 3, partitions 1-12, retention >= 1h, ISR >= 2
`subject-rules`	Subject	`-key` or `-value` suffix, explicit compatibility
`connector-rules`	Connector	Approved plugin classes only, `tasks.max` up to 8
`appgroup-restrictions`	ApplicationGroup	No direct members; on prod instances, only consume, view config and manage data quality

Find them under Resource Policies in Console. Expand a policy to see the rules it enforces:

Policies do nothing on their own. A ResourcePolicy only takes effect once something references it — spec.policyRef on an Application or ApplicationInstance, or spec.policiesRef on a KafkaCluster or KafkaConnectCluster.

Create the Application and its two instances:

docker compose exec -e CDK_API_KEY=$ADMIN_TOKEN conduktor-ctl \
  conduktor apply -f platform/applications/website-analytics/

Open platform/applications/website-analytics/prod.yml and look at what an ApplicationInstance actually declares:

apiVersion: self-serve/v1
kind: ApplicationInstance
metadata:
  application: website-analytics
  name: website-analytics-prod
spec:
  cluster: "kafka-local"
  serviceAccount: "sa-website-analytics-prod"
  policyRef:                  # <-- the guardrails
    - topic-naming
    - topic-labels
    - topic-rules-prod
    - subject-rules
    - connector-rules
  defaultCatalogVisibility: PUBLIC
  resources:                  # <-- the boundary
    - type: TOPIC
      patternType: PREFIXED
      name: "website-analytics.prod."
    - type: CONSUMER_GROUP
      patternType: PREFIXED
      name: "website-analytics.prod."
    - type: SUBJECT
      patternType: PREFIXED
      name: "website-analytics.prod."

resources is the boundary: this application instance can do what it likes to any topic, consumer group or subject matching website-analytics.prod., and nothing at all outside it. policyRef is the guardrail: every resource it applies is validated against those policies first. The dev instance (dev.yml) has the same shape, with its own name, service account and website-analytics.dev. prefixes, and it references topic-rules-dev in place of topic-rules-prod. That single line is the difference between a permissive dev environment and a strict production one. The seventh policy, appgroup-restrictions, is referenced from application.yml rather than from either instance:

apiVersion: self-serve/v1
kind: Application
metadata:
  name: website-analytics
spec:
  title: "Website Analytics"
  description: "Clickstream and page-view events from the company website"
  owner: "website-analytics-owners"
  policyRef:
    - appgroup-restrictions

An ApplicationInstance’s spec.policyRef accepts only Topic, Connector and Subject policies. Referencing an ApplicationGroup policy there fails with Policy with name '<name>' has ApplicationGroup but only [Connector, Topic, Subject] are allowed. Attach those at the Application level instead, where they apply to every instance.

Applications and their instances appear in the Application Catalog:

Delegation is now complete. Everything from here is the application team’s work.

Apply the application team’s resources

Create the team’s topics and schemas in both environments:

docker compose exec -e CDK_API_KEY=$ADMIN_TOKEN conduktor-ctl \
  conduktor apply -f applications/website-analytics/dev/

docker compose exec -e CDK_API_KEY=$ADMIN_TOKEN conduktor-ctl \
  conduktor apply -f applications/website-analytics/prod/

Because these topics belong to an Application Instance, Console links them to their owner in the Topic Catalog. That’s how another team discovers who owns a topic — and requests access to it as part of a pull request, rather than by asking around. It can take up to 30 seconds for new topics to appear, depending on when the indexer last polled.

Note what the team did not have to do: no ticket, no central approval, no waiting. They also couldn’t have gone outside their lane if they’d tried, which is what the next section demonstrates.

Attempt to create a topic out of bounds

Everything so far used an Admin API key, which bypasses policy validation entirely — platform administrators are exempt by design. To see what the application team experiences, you need a key scoped to their application instance. Mint one with your admin token, and keep it in a separate variable so you can switch between the two roles:

APP_TOKEN=$(docker compose exec -T -e CDK_API_KEY=$ADMIN_TOKEN conduktor-ctl \
  conduktor token create application-instance -i=website-analytics-prod prod-key)

From here, a command passed -e CDK_API_KEY=$APP_TOKEN acts as the website-analytics prod application; one passed -e CDK_API_KEY=$ADMIN_TOKEN still acts as an administrator. You can also create the key from the UI, under Application Catalog > Website Analytics > the website-analytics-prod instance > API Keys > New API Key:

The team wants a topic for a replay job, and they want 24 partitions for it. Create applications/website-analytics/prod/replay.yml:

apiVersion: kafka/v2
kind: Topic
metadata:
  cluster: "kafka-local"
  name: website-analytics.prod.replay
  labels:
    instance: prod
    business-unit: marketing
    confidentiality: internal
    team: website-analytics
spec:
  replicationFactor: 3
  partitions: 24
  configs:
    cleanup.policy: delete
    retention.ms: "604800000"
    min.insync.replicas: "2"

Everything about that topic is fine except the partition count. topic-rules-prod (check out the file ) caps production topics at 12:

apiVersion: self-serve/v1
kind: ResourcePolicy
metadata:
  name: topic-rules-prod
spec:
  targetKind: Topic
  description: "Strict topic rules for production environments"
  rules:
    - condition: spec.replicationFactor == 3
      errorMessage: "Replication factor has to be exactly 3 in production"
    - condition: spec.partitions >= 1 && spec.partitions <= 12
      errorMessage: "Production topics need less than or equal to 12 partitions. If you need an exception, plead your case to the platform team."
    - condition: "\"retention.ms\" in spec.configs && int(string(spec.configs[\"retention.ms\"])) >= 3600000"
      errorMessage: "Retention has to be explicitly set and at least 1 hour in production"
    - condition: "\"min.insync.replicas\" in spec.configs && int(string(spec.configs[\"min.insync.replicas\"])) >= 2"
      errorMessage: "min.insync.replicas has to be at least 2 in production"

Each condition is a CEL expression evaluated against the resource’s metadata and spec. Try to apply it:

docker compose exec -e CDK_API_KEY=$APP_TOKEN conduktor-ctl \
  conduktor apply -f applications/website-analytics/prod/replay.yml

Applying resources
Could not apply resource Topic/website-analytics.prod.replay: Policies check failed:
- topic-rules-prod: Production topics need less than or equal to 12 partitions. If you need an exception, plead your case to the platform team.
Error: one or more resources could not be applied

The CLI exits non-zero, so a pipeline stops here. The rejection carries the errorMessage the platform team wrote, so the developer knows both what broke and what to do next. Failures are grouped by policy; if a resource breaks several rules, you get all of them at once rather than one per round trip.

In a real repo this happens on the pull request, not on your laptop. The CI workflow runs conduktor apply --dry-run, and the policy violation shows up as a failed check before anyone reviews the change.

Try a few variations to get a feel for the guardrails:

Rename the topic to payments.replay — website-analytics-prod doesn’t own that prefix, so the boundary rejects it before any policy runs.
Drop the confidentiality label — topic-labels rejects it.
Move the same topic to the dev instance, keeping 24 partitions: rename it to website-analytics.dev.replay, set the instance label to dev, and mint a token for website-analytics-dev. topic-rules-dev caps dev at three. Keep the prod name or the prod token and you hit the ownership boundary instead, before any policy runs.
Re-apply applications/website-analytics/<instance>/ with an application instance token rather than the admin one. The earlier step used the admin token, so none of those resources were policy-checked; an application token runs them through the guardrails, including appgroup-restrictions on the ApplicationGroups.

Grant a policy exception

The team genuinely does need 24 partitions, and the platform team agrees. Policies are strict rather than absolute — the escape hatch is a deliberate, auditable exception. The resource doesn’t change — its ownership does. It moves out of the team’s folder and into platform/exceptions/, where only the platform team can write. That file is already in the repo as platform/exceptions/website-analytics/prod/high-partition-topic.yml, with a label recording the ticket that authorized it:

  labels:
    ...
    exception-ticket: "PLAT-1234" # why this exists, and who approved it
spec:
  replicationFactor: 3
  partitions: 24 # exceeds the 12-partition cap in topic-rules-prod

Delete the copy you made in the team’s folder, and apply the exception with your admin token instead of the application’s:

rm applications/website-analytics/prod/replay.yml

docker compose exec -e CDK_API_KEY=$ADMIN_TOKEN conduktor-ctl \
  conduktor apply -f platform/exceptions/website-analytics/prod/

This time it succeeds. Nothing was configured to allow it — an AdminToken simply bypasses ResourcePolicy validation, which is exactly why the exception has to live in a platform-owned directory. In a GitOps repo that directory is CODEOWNED by the platform team, so granting an exception means a reviewed pull request with the reason recorded in the diff, not a quietly loosened policy that every other team inherits.

Conclusion

You’ve seen the whole loop: the platform team declares boundaries and guardrails, application teams move fast inside them, violations are caught at apply time with an actionable message, and genuine exceptions are granted deliberately rather than by weakening the rule for everyone. This demo doesn’t cover everything. Explore how to grant permissions to other teams on your owned resources, or how to set up teams within your application — both have commented examples in the repo under applications/website-analytics/<instance>/.

Ready to take this to production? The conduktor/self-service-template repository is the recommended starting point for a real Self-service rollout. It uses the same platform/ and applications/ layout as this quick start and adds the production wiring: CODEOWNERS-based review, per-instance state isolation, OIDC-federated credentials, and three scope-based GitHub Actions workflows (apply-platform.yml, apply-clusters.yml, apply-apps.yml) that run conduktor apply --dry-run on pull requests and apply on merge.

To close everything down, run ./stop.sh. To discuss your requirements or find out more about Self-service, book a demo .

Find out more about Self-service
Use Self-service
View Self-service resource reference
Self-service reference template on GitHub — production-ready GitHub layout with CI/CD workflows
Give us feedback/request a feature

Tutorials

Deploy

Use

Configure

Migrate

Why Self-service

The example repo

Key resources

Running the example

Set up the demo

Apply the platform team’s resources

Apply the application team’s resources

Attempt to create a topic out of bounds

Grant a policy exception

Conclusion

​Why Self-service

​The example repo

​Key resources

​Running the example

​Set up the demo

​Apply the platform team’s resources

​Apply the application team’s resources

​Attempt to create a topic out of bounds

​Grant a policy exception

​Conclusion

​Related resources

Why Self-service

The example repo

Key resources

Running the example

Set up the demo

Apply the platform team’s resources

Apply the application team’s resources

Attempt to create a topic out of bounds

Grant a policy exception

Conclusion

Related resources