Configure encryption and decryption - Conduktor documentation

This feature is available with Conduktor Shield only.

Configure field-level encryption

The properties detailed in this section work for the following plugins:

	On produce	On consume
List-based	`EncryptPlugin`	`FetchEncryptPlugin`
Schema-based	`EncryptSchemaBasedPlugin`	`FetchEncryptSchemaBasedPlugin`

Both schema-based and list-based encryption plugins have their configuration, but some properties are common to both of them.

Key	Type	Default	Description
Common properties
`topic`	String	`.*`	Topics matching this regex will have the interceptor applied.
`schemaRegistryConfig`	SchemaRegistry		Configuration of your schema registry, is needed if you want to encrypt data produced using Avro, JSON or Protobuf schemas.
`schemaDataMode`	String	`preserve_avro`	As of 3.3, you can decide to preserve the inbound message format when it encrypts the data IF the incoming data is Avro, rather than converting the message to JSON (as per current behavior). To convert the record to JSON and break the link to its schema in the backing topic, you can set this field to `convert_json` (default until 3.3).
`externalStorage`	Boolean	`false`	Choose where to store your encryption settings. `false` - Encryption settings will be stored within message headers. `true` - Encryption settings will be stored in a topic called `_conduktor_gateway_encryption_configs` by default, this can be renamed using the environment variable `GATEWAY_ENCRYPTION_CONFIGS_TOPIC`.
`kmsConfig`	KMS		Configuration of one or multiple KMS.
`enableAuditLogOnError`	Boolean	true	The audit log will be enabled when an error occurs during encryption/decryption
`compressionType`	Enum	none	The data is compressed before encryption (only for data configured with full payload encryption). Available values are: `none`, `gzip`, `snappy`, `lz4` or `zstd`.
List-based properties
`recordValue`	Value and key encryption		Configuration to encrypt the record value.
`recordKey`	Value and key encryption		Configuration to encrypt the record key.
`recordHeader`	Headers encryption		Configuration to encrypt the record headers.
Schema-based properties
`defaultKeySecretId`	Secret key template		Default `keySecretId` to use if none is set in the schema. It must be a unique identifier for the secret key, and can be a template for Crypto Shredding use cases.
`defaultAlgorithm`	Algorithm	`AES128_GCM`	Default `algorithm` to use if no algorithm is set in the schema.
`tags`	List[String]		List of tags to search for in the schema to encrypt the specified fields.
`namespace`	String	`conduktor.`	Prefix of custom schema constraints for encryption.

List-based encryption

To define what you want to encrypt, the following options are expanded upon in their respective sections below:

Value and key:
- Encrypt a set of fields
- Encrypt the full payload
Headers:
- Encrypt a set of fields
- Encrypt the full payload
- Encrypt a set of headers that match a regex

Value and key encryption

Set the following properties for recordValue (value encryption) and/or recordKey (key encryption):

Key	Type	Default	Description
Full-payload encryption
`payload.keySecretId`	Secret key template		Secret key, can be a template for Crypto Shredding use cases.
`payload.algorithm`	Algorithm	`AES128_GCM`	Algorithm to leverage.
Field-Level encryption
`fields[].fieldName`	String		Name of the field to encrypt. It can be a nested structure with a dot `.` such as `education.account.username` or `banks[0].accountNo`.
`fields[].keySecretId`	Secret Key template	`AES128_GCM`	Unique identifier for the secret key. You can store this key in your KMS by using the KMS key templates. It can be a template for crypto shredding use cases.
`fields[].algorithm`	Algorithm		Algorithm to use to encrypt this field.

Check out the encryption examples.

Headers encryption

Set the following properties below for recordHeader.

Key	Type	Default	Description
Full-payload encryption			Configuration to encrypt the full payload.
`payload.keySecretId`	Secret key template		Secret key, can be a template for crypto shredding use cases.
`payload.algorithm`	Algorithm	`AES128_GCM`	Algorithm to leverage.
Field-level encryption
`fields[].fieldName`	String		Name of the field to encrypt. It can be a nested structure with a dot `.` such as `education.account.username` or `banks[0].accountNo`.
`fields[].keySecretId`	Secret key template	`AES128_GCM`	Unique identifier for the secret key. It can be a template for crypto shredding use cases.
`fields[].algorithm`	Algorithm
Headers encryption
`header`	String		Headers that match this regex will be encrypted. Warning: it can encrypt all headers including gateway headers

Check out the encryption example.

Schema-based encryption - schema configuration

In order to encrypt your data, you can set a few constraints in your schema. These constraints are detailed below, assuming you’re using the default namespace value which is conduktor.. If you have changed the namespace value in the Interceptor configuration, please change the key name in your schema accordingly.

Key	Type	Default	Description
`conduktor.keySecretId`	Secret key template		Unique identifier for the secret key, and can be a template for crypto shredding use cases.
`conduktor.algorithm`	Algorithm	`AES128_GCM`	Algorithm to use to encrypt this field.
`conduktor.tags`	List[String]		Fields tagged with a matching tag from your Interceptor will be encrypted using the `keySecretId` and `algorithm` specified in the schema. If these are not defined in the schema, the `defaultKeySecretId` and `defaultAlgorithm` from the Interceptor configuration will be used.

If your field meets one of these three conditions, then it will be encrypted:

This field has a keySecretId set in the schema
This field has a algorithm set in the schema
This field has a set of tags set in the schema, and one of them is part of the tags list specified in the Interceptors.

Check out the encryption example.

Secret key templates

Mustache template

In all the encryption plugins, you can use mustache templates for the keySecretId. That way, your secret keys will be dynamic.

The value of a field will be replaced with the encrypted value. So it is not allowed to use the encryption field as the keyId.

Pattern	Replaced by
`{{record.topic}}`	Name of the topic you’re encrypting the data of.
`{{record.key}}`	Key of the encrypted record.
`{{record.value.someValueFieldName}}`	Value of the field called `someValueFieldName` If you’re doing field-level encryption, please ensure that `someValueFieldName` is not included in the fields to encrypt. Otherwise, you will not be able to decrypt it.
`{{record.value.someList[0].someValueField}}`	Value of the field called `someValueFieldName`, in the first element of the list `someList`
`{{record.header.someHeader}}`	Value of the header called `someHeader`

Here’s a record example:

# Header
someHeader=myHeader

# Key
myKey

# Value
{
  "someValueFieldName": "I",
  "someList": [{ "someValueField": "love" }, { "someValueField": "Kafka" }]
}

If you want, you can set something like "keySecretId": "{{record.topic}}-{{record.header.someHeader}}-{{record.key}}". This will create an encryption key called myTopic-myHeader-myKey in memory. If you want this key to be stored in your Vault KMS for instance, you can set: "keySecretId": "vault-kms://vault:8200/transit/keys/{{record.topic}}-{{record.header.someHeader}}-{{record.key}}".

Key Stored in KMS

Any keySecretId that doesn’t match one of the schemas detailed below will be rejected and the encryption operation will fail. If you want to make sure the key is well created in your KMS, you have to (1) make sure you have configured the connection to the KMS and (2) use the following format as keySecretId:

KMS	KMS identifier prefix	Key URI format	Example
In-Memory	in-memory-kms://	`in-memory-kms://<key-id>`	`in-memory-kms://my-password-key-id`
Vault	vault-kms://	`vault-kms://<vault-host>/transit/keys/<key-id>`	`vault-kms://vault:8200/transit/keys/password-key-id`
Azure	azure-kms://	`azure-kms://<key-vault-name>.vault.azure.net/keys/<object-name>/<object-version>`	`azure-kms://my-key-vault.vault.azure.net/keys/conduktor-gateway/4ceb7a4d1f3e4738b23bea870ae8745d`
AWS	aws-kms://	`aws-kms://arn:aws:kms:<region>:<account-id>:key/<key-id>`	`aws-kms://arn:aws:kms:us-east-1:123456789012:key/password-key-id`
GCP	gcp-kms://	`gcp-kms://projects/<project-id>/locations/<location-id>/keyRings/<key-ring-id>/cryptoKeys/<key-id>`	`gcp-kms://projects/my-project/locations/us-east1/keyRings/my-key-ring/cryptoKeys/password-key-id`

Note that In-memory mode is present for testing and development purposes only - keys stored in this manner do not persist between Gateway restarts.

Keys are strings that start with a letter followed by a combination of letters, underscores (_), hyphens (-), and numbers. Special characters are not allowed. They also works with the upper mustache pattern.

Supported algorithms

AES128_GCM (default)
AES128_EAX
AES256_EAX
AES128_CTR_HMAC_SHA256
AES256_CTR_HMAC_SHA256
CHACHA20_POLY1305
XCHACHA20_POLY1305
AES256_GCM

Supported compression types

none
gzip
snappy
lz4
zstd

Configure decryption

Now that your fields or payload are encrypted, you can decrypt them using the Interceptor DecryptPlugin.

Key	Type	Default	Description
`topic`	String	`.*`	Topics matching this regex will have the Interceptor applied
`schemaRegistryConfig`	SchemaRegistry		Configuration of your schema registry, is needed if you want to decrypt into Avro, JSON or Protobuf schemas.
`kmsConfig`	KMS		Configuration of one or multiple KMS
`recordValueFields`	List[String]		Only for field-level encryption - List of fields to decrypt in the value. If empty, we decrypt all the encrypted fields.
`recordKeyFields`	List[String]		Only for field-level encryption - List of fields to decrypt in the key. If empty, we decrypt all the encrypted fields.
`recordHeaderFields`	List[String]		Only for field-level encryption - List of headers to decrypt. If empty, we decrypt all the encrypted headers.
`enableAuditLogOnError`	Boolean	true	The audit log will be enabled when an error occurs during encryption/decryption
`errorPolicy`	String	`return_encrypted`	Determines the action if there is an error during decryption. The options are `return_encrypted` (the encrypted payload is returned to the client) or `fail_fetch` (the client will receive an error for the fetch and no data). For Crypto Shredding, the policy should always be `return_encrypted`, otherwise the consumer will become permanently blocked by messages that have been deliberately been made un-decryptable.

Configure schema registry

Once your records are produced using a schema, you have to configure these properties in your encryption/decryption Interceptors after schemaRegistryConfig in order to (de)serialize them. Gateway supports Confluent-like and AWS Glue schema registries.

Key	Type	Default	Description
`type`	string	`CONFLUENT`	The type of schema registry to use: choose `CONFLUENT` (for Confluent-like schema registries including OSS Kafka) or `AWS` for AWS Glue schema registries.
`additionalConfigs`	map		Additional properties maps to specific security-related parameters. For enhanced security, you can hide the sensitive values using environment variables as secrets.
Confluent-like			Configuration for Confluent-like schema registries
`host`	string		URL of your schema registry.
`cacheSize`	string	`50`	Number of schemas that can be cached locally by this Interceptor so that it doesn’t have to query the schema registry every time.
AWS Glue			Configuration for AWS Glue schema registries
`region`	string		The AWS region for the schema registry, e.g. `us-east-1`
`registryName`	string		The name of the schema registry in AWS (leave blank for the AWS default of `default-registry`)
`basicCredentials`	string		Access credentials for AWS (see below section for structure)
AWS credentials			AWS credential configuration
`accessKey`	string		The access key for the connection to the schema registry.
`secretKey`	string		The secret key for the connection to the schema registry.
`validateCredentials`	bool	`true`	`true` / `false` flag to determine whether the credentials provided should be validated when set.
`accountId`	string		The Id for the AWS account to use.

If you do not supply a basicCredentials section for the AWS Glue schema registry, the client we use to connect will instead attempt to find the connection information is needs from the environment, and the credentials required can be passed this way to Gateway as part of its core configuration. More information on the setup for this is found in the AWS documentation.

Use environment variables as secrets

You probably don’t want your secrets to appear in your Interceptors. In order to make sure this doesn’t happen, you can refer to the environment variables you have set in your Gateway container. For that, you can simply use the format ${MY_ENV_VAR}. We recommend you use this for Schema Registry or Vault secrets, and any other values you’d like to hide in the configuration.

Configure KMS

Find out how to configure the different KMS within your encrypt and decrypt Interceptors.

Key	Type	Description
keyTtlMs	long	Key’s time-to-live in milliseconds. The default is 1 hour. Disable the cache by setting it to 0.
in-memory	In-memory	In Memory KMS that is not persistent, internal to Gateway, for demo purposes only.
gateway	Gateway	Key storage managed by Gateway but with secret management still delegated to an external KMS
vault	Vault KMS	HashiCorp Vault KMS
azure	Azure KMS	Azure Key Vault
aws	AWS KMS	AWS KMS
gcp	GCP KMS	Google Key Management

In-memory KMS

This should not be used on production data.

Keys in in-memory KMS are not persisted, this means that if you do one of the following, you won’t be able to decrypt old records, loosing the data.

Use a Gateway cluster with more than a single node or
restart Gateway or
change the Interceptor configuration

Gateway KMS

This KMS type is effectively a delegated storage model and is designed to support encryption use cases which generate unique secret Ids per record or even field (typically via the Mustache template support for a secret Id). This technique is used in crypto-shredding type scenarios e.g. encrypting records per user with their own key. It provides the option to leverage your KMS for security via a single master key, but efficiently and securely store many per-record level encryption keys (DEKs) in the Gateway managed store. For some architectures this can provide performance and cost savings for encryption use cases which generate a high volume of secret key Ids.

Key	Type	Description
`masterKeyId`	String	The master key secret Id used to encrypt any keys stored in the Gateway managed storage. This is in the same format as the `keySecretId` that’s used for encryption and the valid values are the same.
`maxKeys`	Number	The maximum number of secret Id references to be cached in memory for re-use. To avoid creating new encryption keys (DEKs), this needs to be larger than the total number of expected secret Ids. By default, it’s the same as `maxKeys` in cache config or 1000,000, if `maxKeys` isn’t set.

The masterKeyId is used to secure every key for this configuration, stored by Gateway. Find out more about the secret key formats. You have to also supply a valid configuration for the KMS type referenced by the master key so this can be used. If this key is dropped from the backing KMS, then all keys stored by Gateway for that master key will become unreadable.

Gateway KMS encryption example

Here’s a sample configuration for the Gateway KMS using a Vault-based master key:

"kmsConfig": {
   "gateway": {
      "masterKeyId": "vault-kms://vault:8200/transit/keys/applicants-1-master-key".
      "maxKeys" : 10000000
   },
   "vault": {
      "uri": "http://vault:8200",
      "token": "my-vault-token",
      "version": 1
   }
}

This can then be used to encrypt a field using gateway-kms:// as the secret key type:

"recordValue": {
   "fields": [
      {
         "fieldName": "name",
         "keySecretId": "gateway-kms://fieldKeySecret-name-{{record.key}}"
      }
   ]
}

When processing a record for the first time using this configuration, Gateway will:

generate a DEK to encrypt the field data,
turn it into an EDEK by encrypting with the masterKeyId secret from vault and
store the EDEK in Gateway storage.

If a record key was 123456, the associated EDEK would be stored on a kafka record with the following key:

{"algorithm":"AES128_GCM","keyId":"gateway-kms://fieldKeySecret-name-123456","uuid":"<UNIQUE_PER_EDEK_GENERATED>"}

Multiple records produced against this config would cause multiple EDEKs to be saved in the Gateway storage (due to the {{record.key}} template giving a unique key for each Kafka record key). If there are multiple Gateway nodes running, it’s also possible for multiple DEKs/EDEKs to be generated for the same record key. Two nodes processing different records with the same record key at the same time could both assume they were generating a DEK/EDEK for the first time. In this scenario, there would be two EDEKs in the Gateway storage with the same keyId but they would each have a different UUID. For example:

{"algorithm":"AES128_GCM","keyId":"gateway-kms://fieldKeySecret-name-123456","uuid":"2cd8125a-b55f-4214-a528-be3c9b47519b"}
{"algorithm":"AES128_GCM","keyId":"gateway-kms://fieldKeySecret-name-123456","uuid":"d8fcccf3-8480-4634-879a-48deed4e0e72"}

Nonetheless, there will only ever be one master key stored in the vault KMS, which is used to encrypt every DEK. This feature provides flexibility for your KMS storage and key management setups - and is particularly useful for high volume crypto shredding.

Decrypt using Gateway KMS

When using the gateway-kms secret key Id type, the decryption configuration used to decrypt the data has to also specify the masterKeyId, so that it can securely decrypt the keys stored in the local Gateway storage. Here’s a sample setup:

"config": {
   "topic": "secure-topic",
   "kmsConfig": {
      "gateway": {
         "masterKeyId": "vault-kms://vault:8200/transit/keys/secure-topic-master-key"
      },
      "vault": {
         "uri": "http://vault:8200",
         "token": "my-token-for-vault",
         "version": 1
      }
   }
}

Crypto Shredding

When using the gateway-kms secret key Id type, you can efficiently crypto shred EDEKs in the Gateway storage, so that anyone using the decryption plugin will immediately lose access to the associated encrypted data. To do this, scan the Gateway storage Kafka topic (by default, _conduktor_gateway_encryption_keys) for every message matching the associated qualified secret Id. For example, a qualified secretId of gateway-kms://fieldKeySecret-name-123456 might have the following keys:

{"algorithm":"AES128_GCM","keyId":"gateway-kms://fieldKeySecret-name-123456","uuid":"2cd8125a-b55f-4214-a528-be3c9b47519b"}
{"algorithm":"AES128_GCM","keyId":"gateway-kms://fieldKeySecret-name-123456","uuid":"d8fcccf3-8480-4634-879a-48deed4e0e72"}

Publishing a message for each of these keys back to the same topic with a value of null (i.e. a tombstone) will effectively perform Crypto Shredding. This process won’t prevent the creation of new keys if new messages are sent using the same record key; it only ensures that messages using the crypto shredded keys remain unrecoverable.

Vault KMS

To set your Vault KMS, include this section in your Interceptor config after vault: You can use one of these two authentication methods:

Token
Username and password

Make sure you’ve followed the right method, and that you’ve provided the correct properties. For enhanced security, you can hide the sensitive values using environment variables as secrets.

Key	Type	Description
`uri`	String	Vault URI.
`version`	String	Version.
`namespace`	String	Namespace.
Managed identity		Load authentication information from the below environment variables.
`VAULT_URI`		Vault server base URI.
`VAULT_ENGINE_VERSION`		Vault KV Secrets Engine version.
`VAULT_NAMESPACE`		Vault namespace.
`type`	String	Required for all types of VaultKMSConfig. Determines the type of authentication to use. Supported types: -`TOKEN` -`USERNAME_PASSWORD` -`GITHUB` -`LDAP` -`APP_ROLE` -`KUBERNETES` -`GCP` -`AWS_EC2_PKCS7` -`AWS_EC2` -`AWS_IAM` -`JWT`

Vault authentication types

Key	Type	Description
Token authentication		Use token authentication.
`type`	String	Must be `TOKEN`. Indicates the type of authentication.
`token`	String	Security token for accessing Vault.
Managed identity		Load authentication information from the below environment variables.
`VAULT_AUTH_TYPE`	String	Must be `TOKEN`. Indicates the type of authentication.
`VAULT_TOKEN`		Token to use for accessing Vault.
Username and password		Use username and password authentication.
`type`	String	Must be `USERNAME_PASSWORD`. Indicates the type of authentication.
`username`	String	Username for accessing Vault.
`password`	String	Password for accessing Vault.
`userpassAuthMount`	String	(Optional) Mount path for the userpass auth method.
Managed identity		Load authentication information from the below environment variables.
`VAULT_AUTH_TYPE`	String	Must be `USERNAME_PASSWORD`. Indicates the type of authentication.
`VAULT_USERNAME`		Username for accessing Vault.
`VAULT_PASSWORD`		Password for accessing Vault.
`VAULT_AUTH_MOUNT`		(Optional) Mount path for the userpass auth method.
GitHub authentication		Use GitHub token authentication.
`type`	String	Must be `GITHUB`. Indicates the type of authentication.
`token`	String	GitHub personal access token.
`githubAuthMount`	String	(Optional) Mount path for the GitHub auth method.
Managed identity		Load authentication information from the below environment variables.
`VAULT_AUTH_TYPE`	String	Must be `GITHUB`. Indicates the type of authentication.
`VAULT_GITHUB_TOKEN`		GitHub token for accessing Vault.
`VAULT_AUTH_MOUNT`		(Optional) Mount path for the GitHub auth method.
LDAP authentication		Use LDAP authentication.
`type`	String	Must be `LDAP`. Indicates the type of authentication.
`username`	String	LDAP username.
`password`	String	LDAP password.
`ldapAuthMount`	String	(Optional) Mount path for the LDAP auth method.
Managed identity		Load authentication information from the below environment variables.
`VAULT_AUTH_TYPE`	String	Must be `LDAP`. Indicates the type of authentication.
`VAULT_LDAP_USERNAME`		LDAP username.
`VAULT_LDAP_PASSWORD`		LDAP password.
`VAULT_AUTH_MOUNT`		(Optional) Mount path for the LDAP auth method.
AppRole authentication		Use AppRole authentication.
`type`	String	Must be `APP_ROLE`. Indicates the type of authentication.
`roleId`	String	Role ID for AppRole authentication.
`secretId`	String	Secret ID for AppRole authentication.
`path`	String	(Optional) Mount path for the AppRole auth method.
Managed identity		Load authentication information from the below environment variables.
`VAULT_AUTH_TYPE`	String	Must be `APP_ROLE`. Indicates the type of authentication.
`VAULT_APP_ROLE_ID`		Role ID for AppRole authentication.
`VAULT_APP_SECRET_ID`		Secret ID for AppRole authentication.
`VAULT_APP_PATH`		(Optional) Mount path for the AppRole auth method.
Kubernetes authentication		Use Kubernetes authentication.
`type`	String	Must be `KUBERNETES`. Indicates the type of authentication.
`role`	String	Kubernetes role.
`jwt`	String	Kubernetes JWT token.
Managed identity		Load authentication information from the below environment variables.
`VAULT_AUTH_TYPE`	String	Must be `KUBERNETES`. Indicates the type of authentication.
`VAULT_KUBERNETES_ROLE`		Kubernetes role.
`VAULT_KUBERNETES_JWT`		Kubernetes JWT token.
GCP authentication		Use Google Cloud Platform authentication.
`type`	String	Must be `GCP`. Indicates the type of authentication.
`role`	String	GCP role for authentication.
`jwt`	String	JWT token issued by Google Cloud Platform.
Managed identity		Load authentication information from the below environment variables.
`VAULT_AUTH_TYPE`	String	Must be `GCP`. Indicates the type of authentication.
`VAULT_GCP_ROLE`		GCP role for authentication.
`VAULT_GCP_JWT`		JWT token for accessing Vault.
AWS EC2 authentication (PKCS7)		Use AWS EC2 PKCS7 authentication.
`type`	String	Must be `AWS_EC2_PKCS7`. Indicates the type of authentication.
`role`	String	AWS role for EC2 authentication.
`pkcs7`	String	PKCS7 identity document.
`nonce`	String	(Optional) Nonce value for EC2 authentication.
Managed identity		Load authentication information from the below environment variables.
`VAULT_AUTH_TYPE`	String	Must be `AWS_EC2_PKCS7`. Indicates the type of authentication.
`VAULT_AWS_ROLE`		AWS role for EC2 authentication.
`VAULT_AWS_PKCS7`		PKCS7 identity document.
`VAULT_AWS_NONCE`		(Optional) Nonce value for EC2 authentication.
`VAULT_AUTH_MOUNT`		(Optional) Mount path for the AWS EC2 PKCS7 auth method.
AWS EC2 authentication		Use AWS EC2 identity authentication.
`type`	String	Must be `AWS_EC2`. Indicates the type of authentication.
`role`	String	AWS role for EC2 authentication.
`identity`	String	AWS identity document.
`signature`	String	AWS signature for authentication.
`nonce`	String	Nonce value for EC2 authentication.
Managed identity		Load authentication information from the below environment variables.
`VAULT_AUTH_TYPE`	String	Must be `AWS_EC2`. Indicates the type of authentication.
`VAULT_AWS_ROLE`		AWS role for EC2 authentication.
`VAULT_AWS_IDENTITY`		AWS identity document.
`VAULT_AWS_SIGNATURE`		AWS signature for authentication.
`VAULT_AWS_NONCE`		(Optional) Nonce value for EC2 authentication.
`VAULT_AUTH_MOUNT`		(Optional) Mount path for the AWS EC2 auth method.
AWS IAM authentication		Use AWS IAM Authentication.
`type`	String	Must be `AWS_IAM`. Indicates the type of authentication.
`role`	String	AWS role for IAM authentication.
`iamRequestUrl`	String	IAM request URL for authentication.
`iamRequestBody`	String	IAM request body for authentication.
`iamRequestHeaders`	String	IAM request headers for authentication.
Managed identity		Load authentication information from the below environment variables.
`VAULT_AUTH_TYPE`	String	Must be `AWS_IAM`. Indicates the type of authentication.
`VAULT_AWS_ROLE`		AWS role for IAM authentication.
`VAULT_AWS_IAM_REQUEST_URL`		IAM request URL for authentication.
`VAULT_AWS_IAM_REQUEST_BODY`		IAM request body for authentication.
`VAULT_AWS_IAM_REQUEST_HEADERS`		IAM request headers for authentication.
JWT authentication		Use JWT authentication.
`type`	String	Must be `JWT`. Indicates the type of authentication.
`jwt`	String	JWT token for authentication.
`provider`	String	JWT provider for authentication.
`role`	String	JWT role for authentication.
Managed identity		Load authentication information from the below environment variables.
`VAULT_AUTH_TYPE`	String	Must be `JWT`. Indicates the type of authentication.
`VAULT_JWT`		JWT token for authentication.
`VAULT_JWT_PROVIDER`		JWT provider for authentication.

Example:

{
  "type": "APP_ROLE",
  "uri": "http://vault.example.com",
  "version": "1",
  "roleId": "my-role-id",
  "secretId": "my-secret-id"
}

Azure KMS

To set your Azure KMS, include this section in your Interceptor config, below azure. You can use one of these two authentication methods:

token or
username and password.

Make sure you’ve followed the right method, and that you’ve provided the correct properties. For enhanced security, you can hide the sensitive values using environment variables as secrets.

Key	Type	Description
Token
`tokenCredential.clientId`	string	Client ID.
`tokenCredential.tenantId`	string	Tenant ID.
`tokenCredential.clientSecret`	string	Client secret.
Username and password
`usernamePasswordCredential.clientId`	string	Client ID.
`usernamePasswordCredential.tenantId`	string	Tenant ID.
`usernamePasswordCredential.username`	string	Username.
`usernamePasswordCredential.password`	string	Password.
Managed identity		Configure the KMS from the context, and not using variables. This will be overwritten if a specific KMS is configured within the Interceptor.

AWS KMS

To set your AWS KMS, include this section in your Interceptor config, below aws. You can use one of these two authentication methods:

basic authentication or
session.

Make sure to follow the right method and provided the correct properties. For enhanced security, you can hide the sensitive values using environment variables as secrets.

Key	Type	Description
Basic authentication
`basicCredentials.accessKey`	String	Access key.
`basicCredentials.secretKey`	String	Secret key.
Session
`sessionCredentials.accessKey`	String	Access key.
`sessionCredentials.secretKey`	String	Secret key.
`sessionCredentials.sessionToken`	String	Session token.
Managed identity		Configure the KMS from the context, not using variables. This will be overwritten if a specific KMS is configured within the Interceptor.

Google Cloud Platform KMS

To set your Google Cloud Platform (GCP) KMS, include this section in your Interceptor config, under gcp: You must first configure the service account key file. For enhanced security, you can hide the sensitive values using environment variables as secrets.

Key	Type	Description
`serviceAccountCredentialsFilePath`	String	Service account key file in GCP.
Managed identity		Configure the KMS from the context, and not using variables. This will be overwritten if a specific KMS is configured within the Interceptor.

Encryption examples

Schema based encryption examples

Fields containing specific information with (keySecretId, algorithm, tags match) will be encrypted.
Field would be encrypted with the associated keySecretId, algorithm, if any missed, would be encrypted with the associated default ones in the Interceptor configuration.
Field would be encrypted with defaultSecret, defaultAlgorithm when tags has element with is in the Interceptor configuration.

Example for json schema, assuming we’re using default namespace (conduktor.):

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Customer",
  "type": "object",
  "properties": {
    "name": { "type": "string" },
    "username": { "type": "string" },
    "password": { "type": "string", "conduktor.keySecretId": "password-secret", "conduktor.algorithm": "AES128_GCM"},
    "visa": { "type": "string", "conduktor.keySecretId": "password-visa"},
    "address": {
      "type": "object",
      "properties": {
        "location": { "type": "string", "conduktor.tags": ["MY_TAG", "PII", "GDPR", "MY_OTHER_TAG"]},
        "town": { "type": "string" },
        "country": { "type": "string" }
      }
    }
  }
}

Explanation:

password would be encrypted with the associated keySecretId, algorithm etc.
visa would be encryption with the associated keySecretId and the default algorithm provided in the Interceptor configuration.
location would be encrypted with defaultSecret, defaultAlgorithm because tags has PII with is in the Interceptor configuration.
fields containing no specific information (keySecretId, algorithm, tags without match) are left untouched.

Example for Avro schema, assuming we’re using default namespace (conduktor.):

{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "name", "type": "string", "conduktor.algorithm": "AES128_GCM"},
    {"name": "age", "type": "int", "conduktor.keySecretId": "age-secret"},
    {"name": "email", "type": "string"},
    {
      "name": "address",
      "type": {
        "type": "record",
        "name": "AddressRecord",
        "fields": [
          {"name": "street", "type": "string", "conduktor.keySecretId": "street-secret"},
          {"name": "city", "type": "string", "conduktor.keySecretId": "city-secret", "conduktor.algorithm": "AES128_GCM"}
        ]
      }
    },
    {"name": "hobbies", "type": {"type": "array", "items": "string"}},
    {
      "name": "friends",
      "type": {
        "type": "array",
        "items": {
          "type": "record",
          "name": "Friend",
          "fields": [
            {"name": "name", "type": "string", "conduktor.tags": ["MY_TAG", "PII", "GDPR", "MY_OTHER_TAG"]},
            {"name": "age", "type": "int"}
          ]
        }
      }
    }
  ]
}

Example for Protobuf schema, assuming we’re using default namespace (conduktor.): In Protobuf, since we are using the Confluent schema registry, we use the (confluent.field_meta).params (with type map<string, string) for field options. Here’s how it can be defined:

  syntax = "proto3";
                 
  option java_package = "schema.protobuf";
  option java_outer_classname = "User";
  
  message Student {
    string name = 1 [(confluent.field_meta).params = {conduktor.keySecretId: "name-secret", conduktor.algorithm: "AES128_GCM"}];
    int32 age = 2 [(confluent.field_meta).params = {conduktor.keySecretId: "age-secret"}];
    string email = 3 [(confluent.field_meta).params = {conduktor.keySecretId: "email-secret"}];
    Address address = 4;
    repeated string hobbies = 5;
    repeated Friend friends = 6;
  
    message Address {
      string street = 1 [(confluent.field_meta).params = {conduktor.keySecretId: "street-secret", conduktor.algorithm: "AES128_GCM"}];
      string city = 2 [(confluent.field_meta).params = {conduktor.keySecretId: "city-secret"}];
    }
  
    message Friend {
      string name = 1 [(confluent.field_meta).params = {conduktor.tags: "[\"PII\", \"MY_TAG\"]"}];
      int32 age = 2 [(confluent.field_meta).params = {conduktor.keySecretId: "friend-age-secret"}];
    }
  }

Simple encrypt on produce

{
  "name": "myEncryptionPlugin",
  "pluginClass": "io.conduktor.gateway.interceptor.EncryptPlugin",
  "priority": 100,
  "config": {
    "topic": ".*",
    "recordValue": {
      "fields": [
        {
          "fieldName": "password",
          "keySecretId": "password-secret",
          "algorithm": "AES128_GCM"
        }
      ]
    }
  }
}

Field level encryption on produce

{
  "name": "myEncryptPlugin",
  "pluginClass": "io.conduktor.gateway.interceptor.EncryptPlugin",
  "priority": 100,
  "config": {
    "topic": ".*",
    "schemaRegistryConfig": {
      "host": "http://schema-registry:8081"
    },
    "kmsConfig": {
      "vault": {
        "uri": "http://vault:8200",
        "token": "vault-plaintext-root-token",
        "version": 1
      },
      "azure": {
        "tokenCredential": {
          "clientId": "azure_client_id",
          "tenantId": "azure_tenant_id",
          "clientSecret": "azure_client_secret"
        }
      }
    },
    "recordValue": {
      "fields": [
        {
          "fieldName": "password",
          "keySecretId": "vault-kms://vault:8200/transit/keys/password-secret",
          "algorithm": "AES128_GCM"
        },
        {
          "fieldName": "visa",
          "keySecretId": "vault-kms://vault:8200/transit/keys/{{record.header.test-header}}-visa-secret-{{record.key}}-{{record.value.username}}-{{record.value.education.account.accountId}}",
          "algorithm": "AES128_GCM"
        },
        {
          "fieldName": "education.account.username",
          "keySecretId": "azure-kms://my-key-vault.vault.azure.net/keys/conduktor-gateway/4ceb7a4d1f3e4738b23bea870ae8745d",
          "algorithm": "AES128_GCM"
        }
      ]
    }
  }
}

Field-level encryption on produce with secured template

{
  "name": "myEncryptPlugin",
  "pluginClass": "io.conduktor.gateway.interceptor.EncryptPlugin",
  "priority": 100,
  "config": {
    "topic": ".*",
    "schemaRegistryConfig": {
      "host": "http://schema-registry:8081",
      "additionalConfigs": {
        "schema.registry.url": "${SR_URL}",
        "basic.auth.credentials.source": "${SR_BASIC_AUTH_CRED_SRC}",
        "basic.auth.user.info": "${SR_BASIC_AUTH_USER_INFO}"
      }
    },
    "kmsConfig": {
      "vault": {
        "uri": "http://vault:8200",
        "token": "${VAULT_TOKEN}",
        "version": 1
      }
      }
    },
    "recordValue": {
      "fields": [
        {
          "fieldName": "password",
          "keySecretId": "vault-kms://vault:8200/transit/keys/password-secret",
          "algorithm": "AES128_GCM"
        },
        {
          "fieldName": "visa",
          "keySecretId": "vault-kms://vault:8200/transit/keys/{{record.header.test-header}}-visa-secret-{{record.key}}-{{record.value.username}}-{{record.value.education.account.accountId}}",
          "algorithm": "AES128_GCM"
        },
        {
          "fieldName": "education.account.username",
          "keySecretId": "vault-kms://vault:8200/transit/keys/{{record.header.test-header}}-secret-key-account-username-{{record.topic}}",
          "algorithm": "AES128_GCM"
        }
      ]
    }
  }
}

Schema-based field level encryption on produce

{
  "name": "mySchemaBasedEncryptPlugin",
  "pluginClass": "io.conduktor.gateway.interceptor.EncryptSchemaBasedPlugin",
  "priority": 100,
  "config": {
    "topic": ".*",
    "schemaRegistryConfig": {
      "host": "http://schema-registry:8081"
    },
    "kmsConfig": {
      "vault": {
        "uri": "http://vault:8200",
        "token": "vault-plaintext-root-token",
        "version": 1
      }
      }
    },
    "defaultKeySecretId": "vault-kms://vault:8200/transit/keys/myDefaultKeySecret",
    "defaultAlgorithm": "AES128_EAX",
    "tags": ["PII", "ENCRYPTION"],
    "namespace": "conduktor."
  }
}

Full message level encryption on produce

{
  "name": "myEncryptPlugin",
  "pluginClass": "io.conduktor.gateway.interceptor.EncryptPlugin",
  "priority": 100,
  "config": {
    "topic": ".*",
    "schemaRegistryConfig": {
      "host": "http://schema-registry:8081"
    },
    "kmsConfig": {
      "vault": {
        "uri": "http://vault:8200",
        "token": "vault-plaintext-root-token",
        "version": 1
      }
      }
    },
    "recordValue": {
      "payload": {
        "keySecretId": "vault-kms://vault:8200/transit/keys/{{record.header.test-header}}-secret-key-account-username-{{record.topic}}",
        "algorithm": "AES128_GCM"
      }
    }
  }
}

Full message level encryption on produce with secured template

{
  "name": "myEncryptPlugin",
  "pluginClass": "io.conduktor.gateway.interceptor.EncryptPlugin",
  "priority": 100,
  "config": {
    "topic": ".*",
    "schemaRegistryConfig": {
      "host": "http://schema-registry:8081",
      "additionalConfigs": {
        "schema.registry.url": "${SR_URL}",
        "basic.auth.credentials.source": "${SR_BASIC_AUTH_CRED_SRC}",
        "basic.auth.user.info": "${SR_BASIC_AUTH_USER_INFO}"
      }
    },
    "kmsConfig": {
      "vault": {
        "uri": "http://vault:8200",
        "token": "${VAULT_TOKEN}",
        "version": 1
      }
      }
    },
    "recordValue": {
      "payload": {
        "keySecretId": "vault-kms://vault:8200/transit/keys/{{record.header.test-header}}-secret-key-account-username-{{record.topic}}",
        "algorithm": "AES128_GCM"
      }
    }
  }
}

Encryption on consume

{
  "name": "myEncryptPlugin",
  "pluginClass": "io.conduktor.gateway.interceptor.FetchEncryptPlugin",
  "priority": 100,
  "config": {
    "topic": ".*",
    "schemaRegistryConfig": {
      "host": "http://schema-registry:8081"
    },
    "kmsConfig": {
      "vault": {
        "uri": "http://vault:8200",
        "token": "vault-plaintext-root-token",
        "version": 1
      }
    },
    "recordValue": {
      "fields": [
        {
          "fieldName": "password",
          "keySecretId": "vault-kms://vault:8200/transit/keys/password-secret",
          "algorithm": "AES128_GCM"
        },
        {
          "fieldName": "visa",
          "keySecretId": "vault-kms://vault:8200/transit/keys/{{record.header.test-header}}-visa-secret-{{record.key}}-{{record.value.username}}-{{record.value.education.account.accountId}}",
          "algorithm": "AES128_GCM"
        },
        {
          "fieldName": "education.account.username",
          "keySecretId": "vault-kms://vault:8200/transit/keys/{{record.header.test-header}}-secret-key-account-username-{{record.topic}}",
          "algorithm": "AES128_GCM"
        }
      ]
    }
  }
}

Schema-based field level encryption on consume

{
  "name": "mySchemaBasedEncryptPlugin",
  "pluginClass": "io.conduktor.gateway.interceptor.FetchEncryptSchemaBasedPlugin",
  "priority": 100,
  "config": {
    "topic": ".*",
    "schemaRegistryConfig": {
      "host": "http://schema-registry:8081"
    },
    "kmsConfig": {
      "vault": {
        "uri": "http://vault:8200",
        "token": "vault-plaintext-root-token",
        "version": 1
      }
      }
    },
    "defaultKeySecretId": "vault-kms://vault:8200/transit/keys/myDefaultKeySecret",
    "defaultAlgorithm": "AES128_EAX",
    "tags": ["PII", "ENCRYPTION"],
    "namespace": "conduktor."
  }
}

Encryption on consume with secured template

{
  "name": "myEncryptPlugin",
  "pluginClass": "io.conduktor.gateway.interceptor.FetchEncryptPlugin",
  "priority": 100,
  "config": {
    "topic": ".*",
    "schemaRegistryConfig": {
      "host": "http://schema-registry:8081",
      "additionalConfigs": {
        "schema.registry.url": "${SR_URL}",
        "basic.auth.credentials.source": "${SR_BASIC_AUTH_CRED_SRC}",
        "basic.auth.user.info": "${SR_BASIC_AUTH_USER_INFO}"
      }
    },
    "kmsConfig": {
      "vault": {
        "uri": "http://vault:8200",
        "token": "${VAULT_TOKEN}",
        "version": 1
      }
    },
    "recordValue": {
      "fields": [
        {
          "fieldName": "password",
          "keySecretId": "vault-kms://vault:8200/transit/keys/password-secret",
          "algorithm": "AES128_GCM"
        },
        {
          "fieldName": "visa",
          "keySecretId": "vault-kms://vault:8200/transit/keys/{{record.header.test-header}}-visa-secret-{{record.key}}-{{record.value.username}}-{{record.value.education.account.accountId}}",
          "algorithm": "AES128_GCM"
        },
        {
          "fieldName": "education.account.username",
          "keySecretId": "vault-kms://vault:8200/transit/keys/{{record.header.test-header}}-secret-key-account-username-{{record.topic}}",
          "algorithm": "AES128_GCM"
        }
      ]
    }
  }
}

Decryption examples

Decrypt all fields

{
  "name": "myDecryptPlugin",
  "pluginClass": "io.conduktor.gateway.interceptor.DecryptPlugin",
  "priority": 100,
  "config": {
    "topic": ".*",
    "schemaRegistryConfig": {
      "host": "http://schema-registry:8081"
    },
    "kmsConfig": {
      "vault": {
        "uri": "http://vault:8200",
        "token": "vault-plaintext-root-token",
        "version": 1
      }
    }
  }
}

Decrypt all fields with secured template

{
  "name": "myDecryptPlugin",
  "pluginClass": "io.conduktor.gateway.interceptor.DecryptPlugin",
  "priority": 100,
  "config": {
    "topic": ".*",
    "schemaRegistryConfig": {
      "host": "http://schema-registry:8081",
      "additionalConfigs": {
        "schema.registry.url": "${SR_URL}",
        "basic.auth.credentials.source": "${SR_BASIC_AUTH_CRED_SRC}",
        "basic.auth.user.info": "${SR_BASIC_AUTH_USER_INFO}"
      }
    },
    "kmsConfig": {
      "vault": {
        "uri": "http://vault:8200",
        "token": "${VAULT_TOKEN}",
        "version": 1
      }
    }
  }
}

Decrypt specific fields

{
  "name": "myDecryptPlugin",
  "pluginClass": "io.conduktor.gateway.interceptor.DecryptPlugin",
  "priority": 100,
  "config": {
    "topic": ".*",
    "schemaRegistryConfig": {
      "host": "http://schema-registry:8081"
    },
    "kmsConfig": {
      "vault": {
        "uri": "http://vault:8200",
        "token": "vault-plaintext-root-token",
        "version": 1
      }
    },
    "recordValueFields": [
      "visa",
      "education.account.username"
    ],
    "recordKeyFields": [
      "bank.accountNo"
    ],
    "recordHeaderFields": [
      "account.username"
    ]
  }
}

Decrypt specific fields with secured template

{
  "name": "myDecryptPlugin",
  "pluginClass": "io.conduktor.gateway.interceptor.DecryptPlugin",
  "priority": 100,
  "config": {
    "topic": ".*",
    "schemaRegistryConfig": {
      "host": "http://schema-registry:8081",
      "additionalConfigs": {
        "schema.registry.url": "${SR_URL}",
        "basic.auth.credentials.source": "${SR_BASIC_AUTH_CRED_SRC}",
        "basic.auth.user.info": "${SR_BASIC_AUTH_USER_INFO}"
      }
    },
    "kmsConfig": {
      "vault": {
        "uri": "http://vault:8200",
        "token": "${VAULT_TOKEN}",
        "version": 1
      }
    },
    "recordValueFields": [
      "visa",
      "education.account.username"
    ],
    "recordKeyFields": [
      "bank.accountNo"
    ],
    "recordHeaderFields": [
      "account.username"
    ]
  }
}

Decrypt full message

{
  "name": "myDecryptPlugin",
  "pluginClass": "io.conduktor.gateway.interceptor.DecryptPlugin",
  "priority": 100,
  "config": {
    "topic": ".*",
    "kmsConfig": {
      "vault": {
        "uri": "http://vault:8200",
        "token": "vault-plaintext-root-token",
        "version": 1
      }
    }
  }
}

Decrypt full message with secured template

{
  "name": "myDecryptPlugin",
  "pluginClass": "io.conduktor.gateway.interceptor.DecryptPlugin",
  "priority": 100,
  "config": {
    "topic": ".*",
    "kmsConfig": {
      "vault": {
        "uri": "http://vault:8200",
        "token": "${VAULT_TOKEN}",
        "version": 1
      }
    }
  }
}

Conduktor tutorials

​Configure field-level encryption

​List-based encryption

​Value and key encryption

​Headers encryption

​Schema-based encryption - schema configuration

​Secret key templates

​Mustache template

​Key Stored in KMS

​Supported algorithms

​Supported compression types

​Configure decryption

​Configure schema registry

​Use environment variables as secrets

​Configure KMS

​In-memory KMS

​Gateway KMS

​Gateway KMS encryption example

​Decrypt using Gateway KMS

​Crypto Shredding

​Vault KMS

​Vault authentication types

​Azure KMS

​AWS KMS

​Google Cloud Platform KMS

​Encryption examples

​Schema based encryption examples

​Simple encrypt on produce

​Field level encryption on produce

​Field-level encryption on produce with secured template

​Schema-based field level encryption on produce

​Full message level encryption on produce

​Full message level encryption on produce with secured template

​Encryption on consume

​Schema-based field level encryption on consume

​Encryption on consume with secured template

​Decryption examples

​Decrypt all fields

​Decrypt all fields with secured template

​Decrypt specific fields

​Decrypt specific fields with secured template

​Decrypt full message

​Decrypt full message with secured template

Configure field-level encryption

List-based encryption

Value and key encryption

Headers encryption

Schema-based encryption - schema configuration

Secret key templates

Mustache template

Key Stored in KMS

Supported algorithms

Supported compression types

Configure decryption

Configure schema registry

Use environment variables as secrets

Configure KMS

In-memory KMS

Gateway KMS

Gateway KMS encryption example

Decrypt using Gateway KMS

Crypto Shredding

Vault KMS

Vault authentication types

Azure KMS

AWS KMS

Google Cloud Platform KMS

Encryption examples

Schema based encryption examples

Simple encrypt on produce

Field level encryption on produce

Field-level encryption on produce with secured template

Schema-based field level encryption on produce

Full message level encryption on produce

Full message level encryption on produce with secured template

Encryption on consume

Schema-based field level encryption on consume

Encryption on consume with secured template

Decryption examples

Decrypt all fields

Decrypt all fields with secured template

Decrypt specific fields

Decrypt specific fields with secured template

Decrypt full message

Decrypt full message with secured template