This feature is available with Conduktor Scale Plus only.
- Define Applications for your teams
- Set up policies to enforce governance over topic usage
- Expose topics in the topic catalog
- Experience trying to create resources that breach policies
To follow-along this demo you’ll need to clone our repository (repo):
central-team-repo
and application-team-repo
.
Central team’s repo
This repo is where the central team would make, or approve, changes. It should contain definitions of infrastructure e.g. the cluster configuration and definitions of the Applications, Application Instances and Application Instance policies.Application team’s repo
This repo would exist and be owned by an application team. As the application team is the owner, they make or approve changes, without requiring the central team’s involvement. This model works because the central team has delegated appropriate scope by creating the Application and Application Instance(s) in advance. This delegation is key to Conduktor’s Self-service. Application teams can then create, modify and approve changes on their own resources without having to request further action from the central team. The application team’s repo will have sections for the different types of resource, Kafka resources, Application Instance Permissions and Application Groups.Worked example
The Central team repo has created a cluster configuration in/clusters
. Here the cluster state is assumed to be maintained by a combination of some infrastructure as code (IaC) and container management provider e.g. a Terraform and Kubernetes type setup. No need to change this file as you’ll see below a full example is provided for you to test. They may have groups defined in code here too.
The central team has defined three teams, clickstream, wikipedia and website-analytics each with their own Application, as can be seen by their respective yaml files in /applications
.
Each of these files contains the definition of both their Application and Application Instances. The central team also have folders for their topic policies and user groups.
This example will focus on the website-analytics team.
The website analytics team has their own “repo” for their resource, which for this example will be the application-team-repo
directory. Within they have defined their Kafka resources (topics, schema registry subjects, connectors), application group(s) for permissions within their team and an application instance permission for granting a different team access to their resource.
Running the example
Set up the demo
Docker is required to run this demo.
-
Spin up local resources: Conduktor and Kafka. A Docker compose file is provided - simply start it by navigating with your shell to the cloned repo and running the command below (you may need to download the images, if you’ve not run them before):
-
Log into Console at http://localhost:8080, with the credentials provided in the docker-compose,
admin@conduktor.io
:adminP4ss!
-
Generate an admin API key for the Conduktor CLI. Go to Settings > API Keys > New API Key then copy this value.
-
Note: This can be done from the CLI by setting the following variables and running the command below, but for this demo we’ll be using the UI.
-
Note: This can be done from the CLI by setting the following variables and running the command below, but for this demo we’ll be using the UI.
-
Open the
docker-compose.yml
file and assign the value you just copied to theCDK_API_KEY
environment variable for theconduktor-ctl
service. Save the file and re-rundocker compose up -d
to create the Conduktor CLI container: -
Open a Conduktor CLI by executing into the container:
Apply the central team’s resources
With setup complete we’re now ready to create the Conduktor Applications, so we can delegate responsibility to our application teams. This is from the perspective of the central team.- Create a Group,
website-analytics-team
. We are going to assign ownership of our Application to this group, so the group must exist before creating this Application creation.
- Create policies that some application instances might leverage using the Conduktor CLI

- Create the team resources, the Application and the Application Instances

Apply the application team’s resources
With our protective policies in place and the website analytics team’s Application created, let’s add some topics.- Create topics on our cluster:

Attempt to create topics out of bounds
We’ve demoed successful topic creation, but now let’s attempt to make a topic that doesn’t fit the criteria set by the central team. The API key we’ve been using has been an Admin API key, and we needed it to create topics beyond the scope of a single Application Instance. Remember, we’ve created two application instances in this demo: prod and dev, so we wouldn’t want to use an application level token. However, to properly recreate the application team experience, for this failure, we need to use a key that is scoped to the Application Instance level. Let’s swap in the correct key now and assume the role of the website analytics prod application:- In Console, go to the Application Catalog, our application Website Analytics, the prod instance of the application and click
New API Key
. Click Copy this value.

- Leave the CLI container:
- Open the
docker-compose.yml
again to swap in the new value forCDK_API_KEY
and restart docker compose again
- When the container is restarted, step back in the CLI container
generic-prod-topic-policy
(check out the file) which specifies the following rules:
- a
data-criticality
label must be provided with one of the specified values - the number of partitions must be in a certain range
- the replication factor must be in a certain range
- the retention must be in a certain range
---
as part of appending this block, to indicate a new resource block.
topics.yaml
used in this example contains topics for both the prod and the dev cluster; this application instance’s key can’t create topics on the dev cluster, as expected. We didn’t get this earlier as we were using the admin API key.
The last message is the one of interest, an appropriate error from the policy we just reviewed:
Conclusion
You can copy the example repos and start mapping your own team structure. This demo doesn’t cover everything, so do explore how to grant permissions to other teams on your owned resources or how to set up teams within your application. To close down the resources, exit the CLI container withCtrl + D
and run docker compose down -v
.
To discuss your requirements or find out more about Self-service, book a demo.