Skip to main content
Quick navigation

Manage Service Accounts & ACLs

Introduction​

In this how-to guide, you will learn how to manage service accounts on the Gateway. We will cover the creation of both local and external service accounts, and how to assign ACLs to them.

In this scenario:

  • The SASL_PLAINTEXT security protocol is used for the communication between Clients > Gateway > Kafka
  • The ACLs are enabled on the passthrough virtual cluster (GATEWAY_ACL_ENABLED: true)
  • The ACLs super-user is named GATEWAY_SUPER_USERS: local-acl-admin
  • The Gateway API admin credentials are the defaults

We will use the Gateway API to create and manage service accounts, but the following guide works with the CLI as well.

tip

For local deployments, the Gateway API documentation is available at http://localhost:8888. In this guide, we will use the service-account and the token endpoints.

In the service-account section of the Gateway API documentation, you'll notice that to create a service account on the Gateway, you have to chose between a local or external service account.

TL;DR

A local service account is managed by the Gateway itself, while an external service account is managed by an external OIDC identity provider.

For more information refer to the Service Accounts, Authentication & Authorization concept page if you have not done so already.

Prerequisites​

Start the local deployment​

You can find below the docker-compose file to start a local Gateway with the configuration described above.

docker compose -f docker-compose.yaml up -d

Export Java security manager config (optional)​

Depending on your version of Java you may need to run the below command in your shell session. Newer versions of Java have dropped support for security manager and current versions of Kafka CLI commands will fail without this being set. If you get errors when running the later commands with authentication, run this command.

export KAFKA_OPTS="-Djava.security.manager=allow"

Create a few topics on Kafka​

Let's create a few topics on Kafka by running the following command from your local machine:

kafka-topics --create --bootstrap-server localhost:19092 --topic finance-data
kafka-topics --create --bootstrap-server localhost:19092 --topic finance-report

Manage a Local Service Account​

Create a Local Service Account​

A local service account is managed by the Gateway itself. This means we have to ask the Gateway to create it for us, by giving it a name.

The first step is to create a reference to this new service account. In our case, we will call this service account local-app-finance-dev, and we want it to exist in the passthrough virtual cluster:

Create the service account
curl \
  --request PUT \
  --url 'http://localhost:8888/gateway/v2/service-account' \
  --header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "kind" : "GatewayServiceAccount",
    "apiVersion" : "gateway/v2",
    "metadata" : {
      "name" : "local-app-finance-dev",
      "vCluster" : "passthrough"
    },
    "spec" : { "type" : "LOCAL" }
  }'

Then, we need to get the secret key of this service account, that has a limited lifetime:

Get the secret key
curl \
  --request POST \
  --url 'http://localhost:8888/gateway/v2/token' \
  --header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "vCluster": "passthrough",
    "username": "local-app-finance-dev",
    "lifeTimeSeconds": 3600000
  }'

This will return a JSON object with the token field containing the secret key.

Response
{
  "token": "eyJhbGciOiJIUzI1NiJ9.eyJ1c2VybmFtZSI6ImxvY2FsLWFwcC1maW5hbmNlLWRldiIsInZjbHVzdGVyIjoicGFzc3Rocm91Z2giLCJleHAiOjE3MzIwOTUzNjN9.-rivmwcI-zvTTqLPeO_0l3xUALz5mKtopp1YMaTswFk"
}

This means that, we can now connect to the Gateway passthrough virtual cluster using the local-app-finance-dev service account and its secret key. Let's do so.

Connect to the Gateway with a Local Service Account​

Create a properties file, local-client.properties with the credentials we just generated to connect to the Gateway:

local-client.properties
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="local-app-finance-dev" password="eyJhbGciOiJIUzI1NiJ9.eyJ1c2VybmFtZSI6ImxvY2FsLWFwcC1maW5hbmNlLWRldiIsInZjbHVzdGVyIjoicGFzc3Rocm91Z2giLCJleHAiOjE3MzIwOTUzNjN9.-rivmwcI-zvTTqLPeO_0l3xUALz5mKtopp1YMaTswFk";

List topics using the Kafka CLI, authenticating using our service account:

List topics
kafka-topics --list --bootstrap-server localhost:6969 --command-config local-client.properties

In this case, the command doesn't return anything because we have enabled ACLs on this passthrough virtual cluster (GATEWAY_ACL_ENABLED: true). It means that my local service account doesn't have the right permissions to see any resources, it's not authorized. Let's modify the ACLs so this service account can list topics.

Create ACLs for a Local Service Account​

Create an ACL Admin Local Service Account​

In order to modify the ACLs, we recommend you define a dedicated ACL admin service account.
This is a privileged service account and must be defined in the Gateway configuration using the environment variable GATEWAY_SUPER_USERS, in the case of the passthrough Virtual Cluster. In our example, we have named this local-acl-admin.

Repeat the steps, as before, using the name local-acl-admin. Create the service account, get its credentials, save them to file.

curl \
  --request PUT \
  --url 'http://localhost:8888/gateway/v2/service-account' \
  --header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "kind" : "GatewayServiceAccount",
    "apiVersion" : "gateway/v2",
    "metadata" : {
      "name" : "local-acl-admin",
      "vCluster" : "passthrough"
    },
    "spec" : { "type" : "LOCAL" }
  }'

Store the generated credentials in a new file, local-acl-admin.properties.

local-acl-admin.properties
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="local-acl-admin" password="eyJhbGciOiJIUzI1NiJ9.eyJ1c2VybmFtZSI6ImxvY2FsLWFjbC1hZG1pbiIsInZjbHVzdGVyIjoicGFzc3Rocm91Z2giLCJleHAiOjE3MzIxNjEwOTB9.m8U_DVv4MTOY9mKiKY2tHeUGjxsUvhC9ssE6iAI3eJc";

As this user is an ACLs admin, it has access to all the Gateway topics and can create & modify ACLs for the other service accounts.

Create ACLs for another Local Service Account, using the ACL Admin Service Account​

In order for the local-app-finance-dev service account to be able to interact with its topics, we need to give it the WRITE permission on its prefix. Run the following command to do so:

Give ACLs to local-app-finance-dev
kafka-acls --bootstrap-server localhost:6969 \
  --command-config local-acl-admin.properties \
  --add \
  --allow-principal User:local-app-finance-dev \
  --operation write \
  --topic finance- \
  --resource-pattern-type prefixed
Response
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=finance-, patternType=PREFIXED)`:
 	(principal=User:local-app-finance-dev, host=*, operation=WRITE, permissionType=ALLOW)

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=finance-, patternType=PREFIXED)`:
 	(principal=User:local-app-finance-dev, host=*, operation=WRITE, permissionType=ALLOW)

Finally, let's list the topics using the local-app-finance-dev service account:

List topics as local-app-finance-dev
kafka-topics --list --bootstrap-server localhost:6969 --command-config local-client.properties
Response
finance-data
finance-report

Manage an External Service Account​

An external service account is managed by an external OIDC identity provider. This means we only have to make the Gateway aware of this external service account by giving it its OIDC principal (this is the externalNames). The credentials that will be used by this application are already defined in the OIDC identity provider.

To follow these steps on your machine, you will need to have connected an OAUTHBEARER provider in the config of the docker compose you are using, otherwise please use as reference.

In order to create this external service account reference on the Gateway, you can run the following command to:

  • Create a Gateway service account
  • Names azure-app-billing-dev
  • Which is recognized by it's OIDC principal ("externalNames" : [ "TO_FILL" ])
Create the service account
curl \
  --request PUT \
  --url 'http://localhost:8888/gateway/v2/service-account?dryMode=false' \
  --header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "kind" : "GatewayServiceAccount",
    "apiVersion" : "gateway/v2",
    "metadata" : {
      "name" : "azure-app-billing-dev",
      "vCluster" : "passthrough"
    },
    "spec" : {
      "type" : "EXTERNAL",
      "externalNames" : [ "TO_FILL" ]
    }
  }'

Now you can apply some interceptors to this service account, by referring to the service account name, azure-app-billing-dev.

Connect to the Gateway with an External Service Account​

You can now connect to the Gateway using the azure-app-billing-dev service account.

Here is the type of properties file you may use to connect to the Gateway using OAUTHBEARER:

external-client.properties
security.protocol=SASL_PLAINTEXT
sasl.mechanism=OAUTHBEARER
sasl.login.callback.handler.class=org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler
sasl.oauthbearer.token.endpoint.url="TO_FILL"
sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required clientId="TO_FILL" clientSecret="TO_FILL" scope=".default";

And here is an example of using the Kafka CLI to list the topics, using this service account:

List topics
kafka-topics --list --bootstrap-server localhost:6969 --command-config external-client.properties

In this case, the command wouldn't return anything because we have enabled the ACLs on this passthrough virtual cluster (GATEWAY_ACL_ENABLED: true). It means that my local service account doesn't have the right permissions to see any resources, it's not authorized. The next step is then to give it some ACLs so it can list topics.

Create ACLs for an External Service Account​

The steps here are exactly the same as the ones for the local service account. Please follow them but using the azure-app-billing-dev service account instead of local-app-finance-dev.

Give ACLs to azure-app-billing-dev
kafka-acls --bootstrap-server localhost:6969 \
  --command-config local-acl-admin.properties \
  --add \
  --allow-principal User:azure-app-billing-dev \
  --operation write \
  --topic finance- \
  --resource-pattern-type prefixed

Differences if using Virtual Clusters​

The example above is using a default passthrough Virtual Cluster. If you are using your own Virtual Clusters, you need to make a few changes.

First, let's see how to create a Virtual Cluster with the ACLs enabled, and a super user declared. Then, we'll see how to create the super user credentials, in order to give permissions to the applications service account.

Create the Virtual Cluster with an ACL Admin​

The below creates a Virtual Cluster called my-vcluster, that will have ACLs enabled, and a super user named local-acl-admin.

Create a Virtual Cluster with ACLs Enabled
curl \
  --request PUT \
  --url 'http://localhost:8888/gateway/v2/virtual-cluster?dryMode=false' \
  --header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
  --header 'Content-Type: application/json' \
  --data-raw '{
  "kind" : "VirtualCluster",
  "apiVersion" : "gateway/v2",
  "metadata" : { "name" : "my-vcluster" },
  "spec" : {
    "aclEnabled" : true,
    "superUsers" : [ "local-acl-admin" ]
  }
}'

Service Account Creation in a Virtual Cluster​

Now that the Virtual Cluster my-vcluster exists, create the local Service Account for the super user:

Create the Service Account
curl \
  --request PUT \
  --url 'http://localhost:8888/gateway/v2/service-account' \
  --header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "kind" : "GatewayServiceAccount",
    "apiVersion" : "gateway/v2",
    "metadata" : {
      "name" : "local-acl-admin",
      "vCluster" : "my-vcluster"
    },
    "spec" : { "type" : "LOCAL" }
  }'

Finally, get its secret key:

Get the secret key
curl \
  --request POST \
  --url 'http://localhost:8888/gateway/v2/token' \
  --header 'Authorization: Basic YWRtaW46Y29uZHVrdG9y' \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "vCluster": "my-vcluster",
    "username": "local-acl-admin",
    "lifeTimeSeconds": 3600000
  }'

Note that the same modification applies for external Service Accounts.

Now you can create a properties file, local-acl-admin.properties using the credentials you just generated. Refer to the previous sections for creating ACLs for local & external Service Accounts.