​
Manage a local service account

In this tutorial, you’ll learn how to manage service accounts on Gateway. We’ll cover the how to create both local and external service accounts and how to assign ACLs to them. In our scenario:
  • The SASL_PLAINTEXT security protocol is used for communication between Clients > Gateway > Kafka
  • The ACLs are enabled on the passthrough Virtual Cluster (GATEWAY_ACL_ENABLED: true)
  • The ACLs super-user is called GATEWAY_SUPER_USERS: local-acl-admin
  • The Gateway API admin credentials are the defaults
In this guide we’ll use the Gateway API to create and manage service accounts but these instructions will also work with Conduktor CLI.
For local deployments, the Gateway API documentation is available at http://localhost:8888. In this guide, we’ll use the service-account and the token endpoints.
In the service-account section of the Gateway API documentation, you’ll notice that to create a service account on Gateway, you have to chose between a local or external service account. A local service account is managed by Gateway itself, while an external service account is managed by an external OIDC identity provider. Find out more about Gateway authentication and authorization.

​
Prerequisites

​
1. Start the local deployment

Here’s the docker-compose file to start a local Gateway with the above configuration. 
docker compose -f docker-compose.yaml up -d

​
2. Export Java security manager config (optional)

Depending on your version of Java, you may have to run the below command in your shell session. Newer versions of Java don’t support security manager and current versions of Kafka CLI commands will fail without this being set. If you get errors when running the later commands with authentication, run this command: 
export KAFKA_OPTS="-Djava.security.manager=allow"

​
Create a few topics on Kafka

Let’s create a few topics in Kafka by running the following command: 
kafka-topics --create --bootstrap-server localhost:19092 --topic finance-data
kafka-topics --create --bootstrap-server localhost:19092 --topic finance-report

​
3. Create a local service account

A local service account is managed by Gateway itself. This means that we have to ask Gateway to create it for us by giving it a name. The first step is to create a reference to this new service account. In our case, we’ll call this service account local-app-finance-dev and we want it to exist in the passthrough Virtual Cluster:
Create the service account
curl \
  --request PUT \
  --url 'http://localhost:8888/gateway/v2/service-account' \
  --header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "kind" : "GatewayServiceAccount",
    "apiVersion" : "gateway/v2",
    "metadata" : {
      "name" : "local-app-finance-dev",
      "vCluster" : "passthrough"
    },
    "spec" : { "type" : "LOCAL" }
  }'
Then, we need to get the secret key of this service account, that has a limited lifetime:
Get the secret key
curl \
  --request POST \
  --url 'http://localhost:8888/gateway/v2/token' \
  --header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "vCluster": "passthrough",
    "username": "local-app-finance-dev",
    "lifeTimeSeconds": 3600000
  }'
This will return a JSON object with the token field containing the secret key.
Response
{
  "token": "eyJhbGciOiJIUzI1NiJ9.eyJ1c2VybmFtZSI6ImxvY2FsLWFwcC1maW5hbmNlLWRldiIsInZjbHVzdGVyIjoicGFzc3Rocm91Z2giLCJleHAiOjE3MzIwOTUzNjN9.-rivmwcI-zvTTqLPeO_0l3xUALz5mKtopp1YMaTswFk"
}
This means that, we can now connect to Gateway passthrough Virtual Cluster using the local-app-finance-dev service account and its secret key.

​
4. Connect to Gateway

Create a properties file local-client.properties with the credentials we’ve just generated to connect to Gateway:
local-client.properties
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="local-app-finance-dev" password="eyJhbGciOiJIUzI1NiJ9.eyJ1c2VybmFtZSI6ImxvY2FsLWFwcC1maW5hbmNlLWRldiIsInZjbHVzdGVyIjoicGFzc3Rocm91Z2giLCJleHAiOjE3MzIwOTUzNjN9.-rivmwcI-zvTTqLPeO_0l3xUALz5mKtopp1YMaTswFk";
List topics using the Kafka CLI, authenticating using our service account:
List topics
kafka-topics --list --bootstrap-server localhost:6969 --command-config local-client.properties
In this case, the command doesn’t return anything because we have enabled ACLs on this passthrough Virtual Cluster (GATEWAY_ACL_ENABLED: true). It means that my local service account doesn’t have the right permissions to see any resources, it’s not authorized. Let’s modify the ACLs, so that this service account can list topics.

​
5. Create an ACL admin

To modify the ACLs, we recommend that you define a dedicated ACL admin service account. This is a privileged service account and has to be defined in the Gateway configuration using the environment variable GATEWAY_SUPER_USERS in the case of the passthrough Virtual Cluster. In our example, we’ve called it local-acl-admin. Repeat the steps as before, using the name local-acl-admin. Create the service account, get its credentials and save them to file. 
curl \
  --request PUT \
  --url 'http://localhost:8888/gateway/v2/service-account' \
  --header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "kind" : "GatewayServiceAccount",
    "apiVersion" : "gateway/v2",
    "metadata" : {
      "name" : "local-acl-admin",
      "vCluster" : "passthrough"
    },
    "spec" : { "type" : "LOCAL" }
  }'
Store the generated credentials in a new file, local-acl-admin.properties.
local-acl-admin.properties
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="local-acl-admin" password="eyJhbGciOiJIUzI1NiJ9.eyJ1c2VybmFtZSI6ImxvY2FsLWFjbC1hZG1pbiIsInZjbHVzdGVyIjoicGFzc3Rocm91Z2giLCJleHAiOjE3MzIxNjEwOTB9.m8U_DVv4MTOY9mKiKY2tHeUGjxsUvhC9ssE6iAI3eJc";
As this user is an ACLs admin, they have access to all the Gateway topics and can create and modify ACLs for the other service accounts.

​
6. Create ACLs for other local service accounts

Use the ACL admin service account to create ACLs for other local service accounts. In order for the local-app-finance-dev service account to be able to interact with its topics, we have to give it the WRITE permission on its prefix. Run the following command to do so:
Give ACLs to local-app-finance-dev
kafka-acls --bootstrap-server localhost:6969 \
  --command-config local-acl-admin.properties \
  --add \
  --allow-principal User:local-app-finance-dev \
  --operation write \
  --topic finance- \
  --resource-pattern-type prefixed
Response
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=finance-, patternType=PREFIXED)`:
    (principal=User:local-app-finance-dev, host=*, operation=WRITE, permissionType=ALLOW)

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=finance-, patternType=PREFIXED)`:
    (principal=User:local-app-finance-dev, host=*, operation=WRITE, permissionType=ALLOW)
Finally, let’s list the topics using the local-app-finance-dev service account:
List topics as local-app-finance-dev
kafka-topics --list --bootstrap-server localhost:6969 --command-config local-client.properties
Response
finance-data
finance-report

​
Manage an external service account

An external service account is managed by an external OIDC identity provider. This means we only have to make Gateway aware of this external service account by giving it its OIDC principal (this is the externalNames). The credentials that will be used by this application are already defined in the OIDC identity provider. To follow these steps on your machine, you’ll need to have an OAUTHBEARER provider connected in the config of the Docker Compose you’re using, otherwise use it as a reference. To create this external service account reference on Gateway, you can run the following command to create a Gateway service account called azure-app-billing-dev which is recognized by it’s OIDC principal ("externalNames" : [ "TO_FILL" ]):
Create the service account
curl \
  --request PUT \
  --url 'http://localhost:8888/gateway/v2/service-account?dryMode=false' \
  --header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "kind" : "GatewayServiceAccount",
    "apiVersion" : "gateway/v2",
    "metadata" : {
      "name" : "azure-app-billing-dev",
      "vCluster" : "passthrough"
    },
    "spec" : {
      "type" : "EXTERNAL",
      "externalNames" : [ "TO_FILL" ]
    }
  }'
Now you can apply some interceptors to this service account, by referring to the service account name, azure-app-billing-dev.

​
1. Connect to Gateway

You can now connect to Gateway using the azure-app-billing-dev service account. Here’s the type of properties file you may use to connect to Gateway using OAUTHBEARER:
external-client.properties
security.protocol=SASL_PLAINTEXT
sasl.mechanism=OAUTHBEARER
sasl.login.callback.handler.class=org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler
sasl.oauthbearer.token.endpoint.url="TO_FILL"
sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required clientId="TO_FILL" clientSecret="TO_FILL" scope=".default";
And here’s an example of using the Kafka CLI to list the topics, using this service account:
List topics
kafka-topics --list --bootstrap-server localhost:6969 --command-config external-client.properties
In this case, the command wouldn’t return anything because we have enabled the ACLs on this passthrough virtual cluster (GATEWAY_ACL_ENABLED: true). It means that my local service account doesn’t have the right permissions to see any resources, it’s not authorized. The next step is then to give it some ACLs so it can list topics.

​
2. Create ACLs for an external service account

The steps here are exactly the same as the ones for the local service account. Follow those instructions but use the azure-app-billing-dev service account (instead of local-app-finance-dev).
Give ACLs to azure-app-billing-dev
kafka-acls --bootstrap-server localhost:6969 \
  --command-config local-acl-admin.properties \
  --add \
  --allow-principal User:azure-app-billing-dev \
  --operation write \
  --topic finance- \
  --resource-pattern-type prefixed

​
Differences if using Virtual Clusters

The example above is using a default passthrough Virtual Cluster. If you are using your own Virtual Clusters, you need to make a few changes. First, let’s see how to create a Virtual Cluster with the ACLs enabled, and a super user declared. Then, we’ll see how to create the super user credentials, in order to give permissions to the applications service account.

​
3. Create the Virtual Cluster with an ACL admin

The below creates a Virtual Cluster called my-vcluster that will have ACLs enabled and a super user named local-acl-admin.
Create a Virtual Cluster with ACLs Enabled
curl \
  --request PUT \
  --url 'http://localhost:8888/gateway/v2/virtual-cluster?dryMode=false' \
  --header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
  --header 'Content-Type: application/json' \
  --data-raw '{
  "kind" : "VirtualCluster",
  "apiVersion" : "gateway/v2",
  "metadata" : { "name" : "my-vcluster" },
  "spec" : {
    "aclEnabled" : true,
    "superUsers" : [ "local-acl-admin" ]
  }
}'

​
4. Creating a service account in a Virtual Cluster

Now that the Virtual Cluster my-vcluster exists, create the local service account for the super user:
Create the Service Account
curl \
  --request PUT \
  --url 'http://localhost:8888/gateway/v2/service-account' \
  --header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "kind" : "GatewayServiceAccount",
    "apiVersion" : "gateway/v2",
    "metadata" : {
      "name" : "local-acl-admin",
      "vCluster" : "my-vcluster"
    },
    "spec" : { "type" : "LOCAL" }
  }'
Finally, get its secret key:
Get the secret key
curl \
  --request POST \
  --url 'http://localhost:8888/gateway/v2/token' \
  --header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "vCluster": "my-vcluster",
    "username": "local-acl-admin",
    "lifeTimeSeconds": 3600000
  }'
The same modification applies for external service accounts. You can now create a properties file local-acl-admin.properties using the credentials you’ve just generated.

​
Auto-create topic authorization

When GATEWAY_AUTO_CREATE_TOPICS_ENABLED is set to true, users require specific ACL permissions to automatically create topics when producing or consuming through Gateway.

​
Required permissions

You have to have one of the following ACL permissions to create topics automatically:
  1. CLUSTER resource with CREATE operation - allows creating any topic
  2. TOPIC resource with CREATE operation - allows creating specific topics

​
Example ACLs for auto-create topics

​
Allow creating any topic (CLUSTER level)

Allow creating any topic
kafka-acls --bootstrap-server localhost:6969 \
  --command-config local-acl-admin.properties \
  --add \
  --allow-principal User:local-app-finance-dev \
  --operation create \
  --cluster

​
Allow creating specific topics (TOPIC level)

Allow creating specific topics
kafka-acls --bootstrap-server localhost:6969 \
  --command-config local-acl-admin.properties \
  --add \
  --allow-principal User:local-app-finance-dev \
  --operation create \
  --topic finance- \
  --resource-pattern-type prefixed
When auto-create topics is enabled, concentrated topics are not supported. Topics that would normally be concentrated will be created as physical topics instead.Auto-create topics is disabled by default (GATEWAY_AUTO_CREATE_TOPICS_ENABLED=false).