Skip to main content
Quick navigation

Using Gateway ACL on top of your Kafka

View the full demo in realtime

You can either follow all the steps manually, or watch the recording

Review the docker compose environment

As can be seen from docker-compose.yaml the demo environment consists of the following services:

  • gateway1
  • gateway2
  • kafka-client
  • kafka1
  • kafka2
  • kafka3
  • schema-registry
  • zookeeper
cat docker-compose.yaml

Starting the docker environment

Start all your docker processes, wait for them to be up and ready, then run in background

  • --wait: Wait for services to be running|healthy. Implies detached mode.
  • --detach: Detached mode: Run containers in the background
docker compose up --detach --wait

Creating virtual cluster aclCluster

Creating virtual cluster aclCluster on gateway gateway1 and reviewing the configuration file to access it

# Generate virtual cluster aclCluster with service account admin
token=$(curl \
--request POST "http://localhost:8888/admin/vclusters/v1/vcluster/aclCluster/username/admin" \
--header 'Content-Type: application/json' \
--user 'admin:conduktor' \
--silent \
--data-raw '{"lifeTimeSeconds": 7776000}' | jq -r ".token")

# Create access file
echo """
bootstrap.servers=localhost:6969
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='admin' password='$token';
""" > aclCluster-admin.properties

# Review file
cat aclCluster-admin.properties

Creating virtual cluster aclCluster

Creating virtual cluster aclCluster on gateway gateway1 and reviewing the configuration file to access it

# Generate virtual cluster aclCluster with service account producer
token=$(curl \
--request POST "http://localhost:8888/admin/vclusters/v1/vcluster/aclCluster/username/producer" \
--header 'Content-Type: application/json' \
--user 'admin:conduktor' \
--silent \
--data-raw '{"lifeTimeSeconds": 7776000}' | jq -r ".token")

# Create access file
echo """
bootstrap.servers=localhost:6969
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='producer' password='$token';
""" > aclCluster-producer.properties

# Review file
cat aclCluster-producer.properties

Creating virtual cluster aclCluster

Creating virtual cluster aclCluster on gateway gateway1 and reviewing the configuration file to access it

# Generate virtual cluster aclCluster with service account consumer
token=$(curl \
--request POST "http://localhost:8888/admin/vclusters/v1/vcluster/aclCluster/username/consumer" \
--header 'Content-Type: application/json' \
--user 'admin:conduktor' \
--silent \
--data-raw '{"lifeTimeSeconds": 7776000}' | jq -r ".token")

# Create access file
echo """
bootstrap.servers=localhost:6969
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='consumer' password='$token';
""" > aclCluster-consumer.properties

# Review file
cat aclCluster-consumer.properties

Adding interceptor acl

Add ACL interceptor

Creating the interceptor named acl of the plugin io.conduktor.gateway.interceptor.AclsInterceptorPlugin using the following payload

{
"pluginClass" : "io.conduktor.gateway.interceptor.AclsInterceptorPlugin",
"priority" : 100,
"config" : { }
}

Here's how to send it:

curl \
--request POST "http://localhost:8888/admin/interceptors/v1/vcluster/aclCluster/interceptor/acl" \
--header 'Content-Type: application/json' \
--user 'admin:conduktor' \
--silent \
--data @step-08-acl.json | jq

Try to create a topic as a consumer

Creating on aclCluster:

  • Topic restricted-topic with partitions:1 and replication-factor:1
kafka-topics \
--bootstrap-server localhost:6969 \
--command-config aclCluster-consumer.properties \
--replication-factor 1 \
--partitions 1 \
--create --if-not-exists \
--topic restricted-topic

[!IMPORTANT] We get the following exception

org.apache.kafka.common.errors.ClusterAuthorizationException:
> Cluster not authorized

Creating topic restricted-topic on aclCluster

Creating on aclCluster:

  • Topic restricted-topic with partitions:1 and replication-factor:1
kafka-topics \
--bootstrap-server localhost:6969 \
--command-config aclCluster-admin.properties \
--replication-factor 1 \
--partitions 1 \
--create --if-not-exists \
--topic restricted-topic

List topics with aclCluster-sa does not throw error but gets no topic

kafka-topics \
--bootstrap-server localhost:6969 \
--command-config aclCluster-consumer.properties \
--list

Let's give read-access to test-topic for consumer SA

kafka-acls \
--bootstrap-server localhost:6969 \
--command-config aclCluster-admin.properties \
--add \
--allow-principal User:consumer \
--operation read \
--topic restricted-topic

Consuming from _acls

Consuming from _acls in cluster kafka1

kafka-console-consumer \
--bootstrap-server localhost:19092,localhost:19093,localhost:19094 \
--topic _acls \
--from-beginning \
--timeout-ms 10000 \
--property print.key=true | jq

returns

Processed a total of 1 messages
{
"tenant": "aclCluster",
"principal": "User:consumer",
"host": "*",
"resource": {
"name": "restricted-topic",
"resourceType": "TOPIC",
"patternType": "LITERAL"
},
"operation": "READ"
}
true

Let's give read-access to fixed console-consumer for consumer SA

kafka-acls \
--bootstrap-server localhost:6969 \
--command-config aclCluster-admin.properties \
--add \
--allow-principal User:consumer \
--operation read \
--group console-consumer \
--resource-pattern-type prefixed

Listing topics in aclCluster

kafka-topics \
--bootstrap-server localhost:6969 \
--command-config aclCluster-consumer.properties \
--list

Give read/write access to test-topic to producer SA

kafka-acls \
--bootstrap-server localhost:6969 \
--command-config aclCluster-admin.properties \
--add \
--allow-principal User:producer \
--operation write \
--topic restricted-topic

Listing topics in aclCluster

kafka-topics \
--bootstrap-server localhost:6969 \
--command-config aclCluster-producer.properties \
--list

Let's write into test-topic (producer)

Producing 1 message in restricted-topic in cluster aclCluster

Sending 1 event

{
"msg" : "test message"
}

with

echo '{"msg":"test message"}' | \
kafka-console-producer \
--bootstrap-server localhost:6969 \
--producer.config aclCluster-producer.properties \
--topic restricted-topic

Let's consume from test-topic (consumer)

Let's consume from test-topic (consumer) in cluster aclCluster

kafka-console-consumer \
--bootstrap-server localhost:6969 \
--consumer.config aclCluster-consumer.properties \
--topic restricted-topic \
--from-beginning \
--timeout-ms 10000 | jq

returns

Processed a total of 1 messages
{
"msg": "test message"
}

Consumer-sa cannot write into the test-topic

Producing 1 message in restricted-topic in cluster aclCluster

Sending 1 event

{
"msg" : "I would be surprised if it would work!"
}

with

echo '{"msg":"I would be surprised if it would work!"}' | \
kafka-console-producer \
--bootstrap-server localhost:6969 \
--producer.config aclCluster-consumer.properties \
--topic restricted-topic

[!IMPORTANT] We get the following exception

org.apache.kafka.common.errors.TopicAuthorizationException:
> Not authorized to access topics: [restricted-topic]

Tearing down the docker environment

Remove all your docker processes and associated volumes

  • --volumes: Remove named volumes declared in the "volumes" section of the Compose file and anonymous volumes attached to containers.
docker compose down --volumes