Skip to main content
Quick navigation

A full field level crypto shredding walkthrough

View the full demo in realtime​

You can either follow all the steps manually, or watch the recording

Review the docker compose environment​

As can be seen from docker-compose.yaml the demo environment consists of the following services:

  • gateway1
  • gateway2
  • kafka-client
  • kafka1
  • kafka2
  • kafka3
  • schema-registry
  • vault
cat docker-compose.yaml

Starting the docker environment​

Start all your docker processes, wait for them to be up and ready, then run in background

  • --wait: Wait for services to be running|healthy. Implies detached mode.
  • --detach: Detached mode: Run containers in the background
docker compose up --detach --wait

Creating topic customers-shredding on gateway1​

Creating on gateway1:

  • Topic customers-shredding with partitions:1 and replication-factor:1
kafka-topics \
    --bootstrap-server localhost:6969 \
    --replication-factor 1 \
    --partitions 1 \
    --create --if-not-exists \
    --topic customers-shredding

Listing topics in gateway1​

kafka-topics \
    --bootstrap-server localhost:6969 \
    --list

Adding interceptor crypto-shredding-encrypt​

Let's ask gateway to encrypt messages using vault and dynamic keys

step-07-crypto-shredding-encrypt-interceptor.json:

{
  "kind" : "Interceptor",
  "apiVersion" : "gateway/v2",
  "metadata" : {
    "name" : "crypto-shredding-encrypt"
  },
  "spec" : {
    "comment" : "Adding interceptor: crypto-shredding-encrypt",
    "pluginClass" : "io.conduktor.gateway.interceptor.EncryptPlugin",
    "priority" : 100,
    "config" : {
      "topic" : "customers-shredding",
      "kmsConfig" : {
        "vault" : {
          "uri" : "http://vault:8200",
          "token" : "vault-plaintext-root-token",
          "version" : 1
        }
      },
      "fields" : [ {
        "fieldName" : "password",
        "keySecretId" : "vault-kms://vault:8200/transit/keys/secret-for-{{record.value.name}}",
        "algorithm" : "AES128_GCM"
      }, {
        "fieldName" : "visa",
        "keySecretId" : "vault-kms://vault:8200/transit/keys/secret-for-{{record.value.name}}",
        "algorithm" : "AES128_GCM"
      } ]
    }
  }
}
curl \
    --silent \
    --request PUT "http://localhost:8888/gateway/v2/interceptor" \
    --header "Content-Type: application/json" \
    --user "admin:conduktor" \
    --data @step-07-crypto-shredding-encrypt-interceptor.json | jq

Listing interceptors​

Listing interceptors on gateway1

curl \
    --silent \
    --request GET "http://localhost:8888/gateway/v2/interceptor" \
    --user "admin:conduktor" | jq

Let's produce sample data for tom and laura​

Producing 2 messages in customers-shredding in cluster gateway1

Sending 2 events

{
  "name" : "laura",
  "username" : "laura@conduktor.io",
  "password" : "kitesurf",
  "visa" : "#888999XZ",
  "address" : "Dubai, UAE"
}
{
  "name" : "tom",
  "username" : "tom@conduktor.io",
  "password" : "motorhead",
  "visa" : "#abc123",
  "address" : "Chancery lane, London"
}
echo '{"name":"laura","username":"laura@conduktor.io","password":"kitesurf","visa":"#888999XZ","address":"Dubai, UAE"}' | \
    kafka-console-producer \
        --bootstrap-server localhost:6969 \
        --topic customers-shredding

echo '{"name":"tom","username":"tom@conduktor.io","password":"motorhead","visa":"#abc123","address":"Chancery lane, London"}' | \
    kafka-console-producer \
        --bootstrap-server localhost:6969 \
        --topic customers-shredding

Let's consume the message, and confirm tom and laura are encrypted​

Let's consume the message, and confirm tom and laura are encrypted in cluster gateway1

kafka-console-consumer \
    --bootstrap-server localhost:6969 \
    --topic customers-shredding \
    --from-beginning \
    --max-messages 3 \
    --timeout-ms 3000 | jq

returns 2 events

{
  "name" : "laura",
  "username" : "laura@conduktor.io",
  "password" : "AAAABQAAAAEAAABJdmF1bHQ6djE6a29zcVRKWm5vNlJrd3pteFBnaENjaTdiNE05RHA4UExad3RIaG5SMUlvcEdmSHNWYTVXUGJQNnpyWmg1WFE9PTT+lvH6WRwOhbSK1zuXemP/zgru12rku7Tfl5gIKOwQAWa8nBV9P1BVJiE=",
  "visa" : "AAAABQAAAAEAAABJdmF1bHQ6djE6a29zcVRKWm5vNlJrd3pteFBnaENjaTdiNE05RHA4UExad3RIaG5SMUlvcEdmSHNWYTVXUGJQNnpyWmg1WFE9Pa5usmFlb8ibw4+n70I1pT09syLa2yqOt1XSf9MC0IH7Tv+9Zq+aKfv49x7X",
  "address" : "Dubai, UAE"
}
{
  "name" : "tom",
  "username" : "tom@conduktor.io",
  "password" : "AAAABQAAAAEAAABJdmF1bHQ6djE6WmVvTkh0eTFSQVdUODdZUm5DU3doMURFRnJJMnphaHFqZ0x3ajE3Um9FUUNVNjltS2tnaUZhMTgyUG5aVFE9PWdVu2uhmxW3cq1WDs6Xd77wbB1WQt2i4Lp3qFjKLTRWE13gttjbBS9dGdhY",
  "visa" : "AAAABQAAAAEAAABJdmF1bHQ6djE6WmVvTkh0eTFSQVdUODdZUm5DU3doMURFRnJJMnphaHFqZ0x3ajE3Um9FUUNVNjltS2tnaUZhMTgyUG5aVFE9PX+m2RpInN+f7nYJ4i+QhnbKwyVZ1e/uDpZufudchZXh23vMdyg8v/CYJA==",
  "address" : "Chancery lane, London"
}

Adding interceptor crypto-shredding-decrypt​

Let's add the decrypt interceptor to decipher messages

step-11-crypto-shredding-decrypt-interceptor.json:

{
  "kind" : "Interceptor",
  "apiVersion" : "gateway/v2",
  "metadata" : {
    "name" : "crypto-shredding-decrypt"
  },
  "spec" : {
    "comment" : "Adding interceptor: crypto-shredding-decrypt",
    "pluginClass" : "io.conduktor.gateway.interceptor.DecryptPlugin",
    "priority" : 100,
    "config" : {
      "topic" : "customers-shredding",
      "kmsConfig" : {
        "keyTtlMs" : 200,
        "vault" : {
          "uri" : "http://vault:8200",
          "token" : "vault-plaintext-root-token",
          "version" : 1
        }
      }
    }
  }
}
curl \
    --silent \
    --request PUT "http://localhost:8888/gateway/v2/interceptor" \
    --header "Content-Type: application/json" \
    --user "admin:conduktor" \
    --data @step-11-crypto-shredding-decrypt-interceptor.json | jq

Listing interceptors​

Listing interceptors on gateway1

curl \
    --silent \
    --request GET "http://localhost:8888/gateway/v2/interceptor" \
    --user "admin:conduktor" | jq

Confirm message from tom and laura are encrypted​

Confirm message from tom and laura are encrypted in cluster gateway1

kafka-console-consumer \
    --bootstrap-server localhost:6969 \
    --topic customers-shredding \
    --from-beginning \
    --max-messages 3 \
    --timeout-ms 3000 | jq

returns 2 events

{
  "name" : "laura",
  "username" : "laura@conduktor.io",
  "password" : "kitesurf",
  "visa" : "#888999XZ",
  "address" : "Dubai, UAE"
}
{
  "name" : "tom",
  "username" : "tom@conduktor.io",
  "password" : "motorhead",
  "visa" : "#abc123",
  "address" : "Chancery lane, London"
}

Listing keys created in Vault​

curl \
  --request GET 'http://localhost:8200/v1/transit/keys/?list=true' \
  --silent \
  --header "X-Vault-Token: vault-plaintext-root-token" | jq -r ".data.keys"
curl \
  --request POST 'http://localhost:8200/v1/transit/keys/secret-for-laura/config' \
  --silent \
  --header "X-Vault-Token: vault-plaintext-root-token" \
  --header "content-type: application/json" \
  --data-raw '{"min_decryption_version":"1","min_encryption_version":1,"deletion_allowed":true,"auto_rotate_period":0}' > /dev/null

curl \
  --request DELETE http://localhost:8200/v1/transit/keys/secret-for-laura \
  --silent \
  --header "X-Vault-Token: vault-plaintext-root-token"

Let's make sure laura data are no more readable!​

Let's make sure laura data are no more readable! in cluster gateway1

kafka-console-consumer \
    --bootstrap-server localhost:6969 \
    --topic customers-shredding \
    --from-beginning \
    --max-messages 3 \
    --timeout-ms 3000 | jq

returns 2 events

{
  "name" : "laura",
  "username" : "laura@conduktor.io",
  "password" : "AAAABQAAAAEAAABJdmF1bHQ6djE6a29zcVRKWm5vNlJrd3pteFBnaENjaTdiNE05RHA4UExad3RIaG5SMUlvcEdmSHNWYTVXUGJQNnpyWmg1WFE9PTT+lvH6WRwOhbSK1zuXemP/zgru12rku7Tfl5gIKOwQAWa8nBV9P1BVJiE=",
  "visa" : "AAAABQAAAAEAAABJdmF1bHQ6djE6a29zcVRKWm5vNlJrd3pteFBnaENjaTdiNE05RHA4UExad3RIaG5SMUlvcEdmSHNWYTVXUGJQNnpyWmg1WFE9Pa5usmFlb8ibw4+n70I1pT09syLa2yqOt1XSf9MC0IH7Tv+9Zq+aKfv49x7X",
  "address" : "Dubai, UAE"
}
{
  "name" : "tom",
  "username" : "tom@conduktor.io",
  "password" : "motorhead",
  "visa" : "#abc123",
  "address" : "Chancery lane, London"
}

Tearing down the docker environment​

Remove all your docker processes and associated volumes

  • --volumes: Remove named volumes declared in the "volumes" section of the Compose file and anonymous volumes attached to containers.
docker compose down --volumes

Conclusion

Crypto shredding help you protect your most precious information