Skip to main content
Quick navigation

A full field level with Vault and secret management

View the full demo in realtime​

You can either follow all the steps manually, or watch the recording

Review the docker compose environment​

As can be seen from docker-compose.yaml the demo environment consists of the following services:

  • gateway1
  • gateway2
  • kafka-client
  • kafka1
  • kafka2
  • kafka3
  • schema-registry
  • vault
  • zookeeper
cat docker-compose.yaml

Starting the docker environment​

Start all your docker processes, wait for them to be up and ready, then run in background

  • --wait: Wait for services to be running|healthy. Implies detached mode.
  • --detach: Detached mode: Run containers in the background
docker compose up --detach --wait

Creating virtual cluster teamA​

Creating virtual cluster teamA on gateway gateway1 and reviewing the configuration file to access it

# Generate virtual cluster teamA with service account sa
token=$(curl \
    --request POST "http://localhost:8888/admin/vclusters/v1/vcluster/teamA/username/sa" \
    --header 'Content-Type: application/json' \
    --user 'admin:conduktor' \
    --silent \
    --data-raw '{"lifeTimeSeconds": 7776000}' | jq -r ".token")

# Create access file
echo  """
bootstrap.servers=localhost:6969
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='sa' password='$token';
""" > teamA-sa.properties

# Review file
cat teamA-sa.properties

Creating topic customers on teamA​

Creating on teamA:

  • Topic customers with partitions:1 and replication-factor:1
kafka-topics \
    --bootstrap-server localhost:6969 \
    --command-config teamA-sa.properties \
    --replication-factor 1 \
    --partitions 1 \
    --create --if-not-exists \
    --topic customers

Listing topics in teamA​

kafka-topics \
    --bootstrap-server localhost:6969 \
    --command-config teamA-sa.properties \
    --list

Adding interceptor crypto-shredding-encrypt​

Let's ask gateway to encrypt messages using vault and secret management. The vault token is retrieved from your env variable ${VAULT_TOKEN}.

cat step-08-crypto-shredding-encrypt.json | jq

curl \
    --request POST "http://localhost:8888/admin/interceptors/v1/vcluster/teamA/interceptor/crypto-shredding-encrypt" \
    --header 'Content-Type: application/json' \
    --user 'admin:conduktor' \
    --silent \
    --data @step-08-crypto-shredding-encrypt.json | jq

Listing interceptors for teamA​

Listing interceptors on gateway1 for virtual cluster teamA

curl \
    --request GET 'http://localhost:8888/admin/interceptors/v1/vcluster/teamA' \
    --header 'Content-Type: application/json' \
    --user 'admin:conduktor' \
    --silent | jq

Let's produce sample data for tom and laura​

Producing 2 messages in customers in cluster teamA

Sending 2 events

{
  "name" : "laura",
  "username" : "laura@conduktor.io",
  "password" : "kitesurf",
  "visa" : "#888999XZ",
  "address" : "Dubai, UAE"
}
{
  "name" : "tom",
  "username" : "tom@conduktor.io",
  "password" : "motorhead",
  "visa" : "#abc123",
  "address" : "Chancery lane, London"
}

with

echo '{"name":"laura","username":"laura@conduktor.io","password":"kitesurf","visa":"#888999XZ","address":"Dubai, UAE"}' | \
    kafka-console-producer \
        --bootstrap-server localhost:6969 \
        --producer.config teamA-sa.properties \
        --topic customers

echo '{"name":"tom","username":"tom@conduktor.io","password":"motorhead","visa":"#abc123","address":"Chancery lane, London"}' | \
    kafka-console-producer \
        --bootstrap-server localhost:6969 \
        --producer.config teamA-sa.properties \
        --topic customers

Let's consume the message, and confirm tom and laura are encrypted​

Let's consume the message, and confirm tom and laura are encrypted in cluster teamA

kafka-console-consumer \
    --bootstrap-server localhost:6969 \
    --consumer.config teamA-sa.properties \
    --topic customers \
    --from-beginning \
    --timeout-ms 10000 | jq

returns 2 events

{
  "name" : "laura",
  "username" : "laura@conduktor.io",
  "password" : "AAAABQAAAAEAAABJdmF1bHQ6djE6LzN5WTVNVzJZdDVEMkNIZ2JNRjA4akRXL0wwS2VVL2JXZGV6Ryt5UkFyR3FCWFB0bXNqZVV1b0ZUbUxjRGc9PZ6QsOyTVE0JFze6hCkZN6/ZvnNn+vjMEtxlDcSB2LouMiYlbpF/hk38EPw=",
  "visa" : "AAAABQAAAAEAAABJdmF1bHQ6djE6LzN5WTVNVzJZdDVEMkNIZ2JNRjA4akRXL0wwS2VVL2JXZGV6Ryt5UkFyR3FCWFB0bXNqZVV1b0ZUbUxjRGc9PfwlVRpVwL6k9MEyHSG6WJXSSSve2ubnINjYQFM52HXR10AFL+k8IuBuQ2CR",
  "address" : "Dubai, UAE"
}
{
  "name" : "tom",
  "username" : "tom@conduktor.io",
  "password" : "AAAABQAAAAEAAABJdmF1bHQ6djE6MEpTMEJEV2sybUN1OWg5cDgzNElkeVVBZVVFV0NDTXZlNlI0WFA1TUk2Q1VyakNiYkp3a1haeld3bml2dFE9PU/0AZYmzS0Mg4MDuCv9zrBOHXjHqv/buw7ifBs1+hXe2sfKokQOeaEsJmrH",
  "visa" : "AAAABQAAAAEAAABJdmF1bHQ6djE6MEpTMEJEV2sybUN1OWg5cDgzNElkeVVBZVVFV0NDTXZlNlI0WFA1TUk2Q1VyakNiYkp3a1haeld3bml2dFE9Pc5xlMwSiszb2SujkrWVCPemgoqlDTL/qnFx1IdDz20Xlb9UdZgrfoAdtg==",
  "address" : "Chancery lane, London"
}

Adding interceptor crypto-shredding-decrypt​

Let's add the decrypt interceptor to decipher messages. The vault token is retrieved from your env variable ${VAULT_TOKEN}.

cat step-12-crypto-shredding-decrypt.json | jq

curl \
    --request POST "http://localhost:8888/admin/interceptors/v1/vcluster/teamA/interceptor/crypto-shredding-decrypt" \
    --header 'Content-Type: application/json' \
    --user 'admin:conduktor' \
    --silent \
    --data @step-12-crypto-shredding-decrypt.json | jq

Listing interceptors for teamA​

Listing interceptors on gateway1 for virtual cluster teamA

curl \
    --request GET 'http://localhost:8888/admin/interceptors/v1/vcluster/teamA' \
    --header 'Content-Type: application/json' \
    --user 'admin:conduktor' \
    --silent | jq

Confirm message from tom and laura are encrypted​

Confirm message from tom and laura are encrypted in cluster teamA

kafka-console-consumer \
    --bootstrap-server localhost:6969 \
    --consumer.config teamA-sa.properties \
    --topic customers \
    --from-beginning \
    --timeout-ms 10000 | jq

returns 2 events

{
  "name" : "laura",
  "username" : "laura@conduktor.io",
  "password" : "kitesurf",
  "visa" : "#888999XZ",
  "address" : "Dubai, UAE"
}
{
  "name" : "tom",
  "username" : "tom@conduktor.io",
  "password" : "motorhead",
  "visa" : "#abc123",
  "address" : "Chancery lane, London"
}

Tearing down the docker environment​

Remove all your docker processes and associated volumes

  • --volumes: Remove named volumes declared in the "volumes" section of the Compose file and anonymous volumes attached to containers.
docker compose down --volumes

Conclusion

Crypto shredding help you protect your most precious information