Skip to main content
Quick navigation

A full field level crypto shredding walkthrough

View the full demo in realtime

You can either follow all the steps manually, or watch the recording

Review the docker compose environment

As can be seen from docker-compose.yaml the demo environment consists of the following services:

  • gateway1
  • gateway2
  • kafka-client
  • kafka1
  • kafka2
  • kafka3
  • schema-registry
  • vault
  • zookeeper
cat docker-compose.yaml

Starting the docker environment

Start all your docker processes, wait for them to be up and ready, then run in background

  • --wait: Wait for services to be running|healthy. Implies detached mode.
  • --detach: Detached mode: Run containers in the background
docker compose up --detach --wait

Creating virtual cluster teamA

Creating virtual cluster teamA on gateway gateway1 and reviewing the configuration file to access it

# Generate virtual cluster teamA with service account sa
token=$(curl \
--request POST "http://localhost:8888/admin/vclusters/v1/vcluster/teamA/username/sa" \
--header 'Content-Type: application/json' \
--user 'admin:conduktor' \
--silent \
--data-raw '{"lifeTimeSeconds": 7776000}' | jq -r ".token")

# Create access file
echo """
bootstrap.servers=localhost:6969
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='sa' password='$token';
""" > teamA-sa.properties

# Review file
cat teamA-sa.properties

Creating topic customers-shredding on teamA

Creating on teamA:

  • Topic customers-shredding with partitions:1 and replication-factor:1
kafka-topics \
--bootstrap-server localhost:6969 \
--command-config teamA-sa.properties \
--replication-factor 1 \
--partitions 1 \
--create --if-not-exists \
--topic customers-shredding

Listing topics in teamA

kafka-topics \
--bootstrap-server localhost:6969 \
--command-config teamA-sa.properties \
--list

Adding interceptor crypto-shredding-encrypt

Let's ask gateway to encrypt messages using vault and dynamic keys

cat step-08-crypto-shredding-encrypt.json | jq

curl \
--request POST "http://localhost:8888/admin/interceptors/v1/vcluster/teamA/interceptor/crypto-shredding-encrypt" \
--header 'Content-Type: application/json' \
--user 'admin:conduktor' \
--silent \
--data @step-08-crypto-shredding-encrypt.json | jq

Listing interceptors for teamA

Listing interceptors on gateway1 for virtual cluster teamA

curl \
--request GET 'http://localhost:8888/admin/interceptors/v1/vcluster/teamA' \
--header 'Content-Type: application/json' \
--user 'admin:conduktor' \
--silent | jq

Let's produce sample data for tom and laura

Producing 2 messages in customers-shredding in cluster teamA

Sending 2 events

{
"name" : "laura",
"username" : "laura@conduktor.io",
"password" : "kitesurf",
"visa" : "#888999XZ",
"address" : "Dubai, UAE"
}
{
"name" : "tom",
"username" : "tom@conduktor.io",
"password" : "motorhead",
"visa" : "#abc123",
"address" : "Chancery lane, London"
}

with

echo '{"name":"laura","username":"laura@conduktor.io","password":"kitesurf","visa":"#888999XZ","address":"Dubai, UAE"}' | \
kafka-console-producer \
--bootstrap-server localhost:6969 \
--producer.config teamA-sa.properties \
--topic customers-shredding

echo '{"name":"tom","username":"tom@conduktor.io","password":"motorhead","visa":"#abc123","address":"Chancery lane, London"}' | \
kafka-console-producer \
--bootstrap-server localhost:6969 \
--producer.config teamA-sa.properties \
--topic customers-shredding

Let's consume the message, and confirm tom and laura are encrypted

Let's consume the message, and confirm tom and laura are encrypted in cluster teamA

kafka-console-consumer \
--bootstrap-server localhost:6969 \
--consumer.config teamA-sa.properties \
--topic customers-shredding \
--from-beginning \
--timeout-ms 10000 | jq

returns 2 events

{
"name" : "laura",
"username" : "laura@conduktor.io",
"password" : "AAAABQAAAAEAAABJdmF1bHQ6djE6WXVmMUp5cTM2cUtFUUNDaVpQQ09vT1ZTZWhSTUxsZ1Rvd051WEpiK3lIOXMxT3RBcGU0Y0xPSHVNMzJZQ3c9PUW1O9hJBFOdHDjV6Z10leHvd2E22QitKoM02D+SgjeLXlC1Y+VBnHlIIkY=",
"visa" : "AAAABQAAAAEAAABJdmF1bHQ6djE6WXVmMUp5cTM2cUtFUUNDaVpQQ09vT1ZTZWhSTUxsZ1Rvd051WEpiK3lIOXMxT3RBcGU0Y0xPSHVNMzJZQ3c9PQU3oYki9GuW9j3Vq7+Lj1fVfnKJrtZKNpVzE5YzIuEhq0vY3wT41xl2L0pS",
"address" : "Dubai, UAE"
}
{
"name" : "tom",
"username" : "tom@conduktor.io",
"password" : "AAAABQAAAAEAAABJdmF1bHQ6djE6VGNtdGcwdEU4bkNucU9kV0xGQWRjQXZzVXJ3bUxSOXNMVm51TCtjTGpkSSt0NXJOaTFQYzhKZU9veU9CS3c9PXK5SEikrCnhMUTImgVWa8OisKzZIHjw2cRqUJrcE0WXRc2/+ny+UQYOB1Cd",
"visa" : "AAAABQAAAAEAAABJdmF1bHQ6djE6VGNtdGcwdEU4bkNucU9kV0xGQWRjQXZzVXJ3bUxSOXNMVm51TCtjTGpkSSt0NXJOaTFQYzhKZU9veU9CS3c9PT5PT6k9G2kkJB05CLSqPb5FjSJtecYf5DcVEPpIEhKnkc7acF9cz5/7vQ==",
"address" : "Chancery lane, London"
}

Adding interceptor crypto-shredding-decrypt

Let's add the decrypt interceptor to decipher messages

cat step-12-crypto-shredding-decrypt.json | jq

curl \
--request POST "http://localhost:8888/admin/interceptors/v1/vcluster/teamA/interceptor/crypto-shredding-decrypt" \
--header 'Content-Type: application/json' \
--user 'admin:conduktor' \
--silent \
--data @step-12-crypto-shredding-decrypt.json | jq

Listing interceptors for teamA

Listing interceptors on gateway1 for virtual cluster teamA

curl \
--request GET 'http://localhost:8888/admin/interceptors/v1/vcluster/teamA' \
--header 'Content-Type: application/json' \
--user 'admin:conduktor' \
--silent | jq

Confirm message from tom and laura are encrypted

Confirm message from tom and laura are encrypted in cluster teamA

kafka-console-consumer \
--bootstrap-server localhost:6969 \
--consumer.config teamA-sa.properties \
--topic customers-shredding \
--from-beginning \
--timeout-ms 10000 | jq

returns 2 events

{
"name" : "laura",
"username" : "laura@conduktor.io",
"password" : "kitesurf",
"visa" : "#888999XZ",
"address" : "Dubai, UAE"
}
{
"name" : "tom",
"username" : "tom@conduktor.io",
"password" : "motorhead",
"visa" : "#abc123",
"address" : "Chancery lane, London"
}

Listing keys created in Vault

curl \
--request GET 'http://localhost:8200/v1/transit/keys/?list=true' \
--silent \
--header "X-Vault-Token: vault-plaintext-root-token" | jq -r ".data.keys"
curl \
--request POST 'http://localhost:8200/v1/transit/keys/secret-for-laura/config' \
--silent \
--header "X-Vault-Token: vault-plaintext-root-token" \
--header "content-type: application/json" \
--data-raw '{"min_decryption_version":"1","min_encryption_version":1,"deletion_allowed":true,"auto_rotate_period":0}' > /dev/null

curl \
--request DELETE http://localhost:8200/v1/transit/keys/secret-for-laura \
--silent \
--header "X-Vault-Token: vault-plaintext-root-token"

Let's make sure laura data are no more readable!

Let's make sure laura data are no more readable! in cluster teamA

kafka-console-consumer \
--bootstrap-server localhost:6969 \
--consumer.config teamA-sa.properties \
--topic customers-shredding \
--from-beginning \
--timeout-ms 10000 | jq

returns 2 events

{
"name" : "laura",
"username" : "laura@conduktor.io",
"password" : "AAAABQAAAAEAAABJdmF1bHQ6djE6WXVmMUp5cTM2cUtFUUNDaVpQQ09vT1ZTZWhSTUxsZ1Rvd051WEpiK3lIOXMxT3RBcGU0Y0xPSHVNMzJZQ3c9PUW1O9hJBFOdHDjV6Z10leHvd2E22QitKoM02D+SgjeLXlC1Y+VBnHlIIkY=",
"visa" : "AAAABQAAAAEAAABJdmF1bHQ6djE6WXVmMUp5cTM2cUtFUUNDaVpQQ09vT1ZTZWhSTUxsZ1Rvd051WEpiK3lIOXMxT3RBcGU0Y0xPSHVNMzJZQ3c9PQU3oYki9GuW9j3Vq7+Lj1fVfnKJrtZKNpVzE5YzIuEhq0vY3wT41xl2L0pS",
"address" : "Dubai, UAE"
}
{
"name" : "tom",
"username" : "tom@conduktor.io",
"password" : "motorhead",
"visa" : "#abc123",
"address" : "Chancery lane, London"
}

Tearing down the docker environment

Remove all your docker processes and associated volumes

  • --volumes: Remove named volumes declared in the "volumes" section of the Compose file and anonymous volumes attached to containers.
docker compose down --volumes

Conclusion

Crypto shredding help you protect your most precious information