On produce | On consume | |
---|---|---|
List-based | EncryptPlugin | FetchEncryptPlugin |
Schema-based | EncryptSchemaBasedPlugin | FetchEncryptSchemaBasedPlugin |
Key | Type | Default | Description |
---|---|---|---|
Common properties | |||
topic | String | .* | Topics matching this regex will have the interceptor applied. |
schemaRegistryConfig | SchemaRegistry | Configuration of your schema registry, is needed if you want to encrypt data produced using Avro, JSON or Protobuf schemas. | |
schemaDataMode | String | preserve_avro | As of 3.3, you can decide to preserve the inbound message format when it encrypts the data IF the incoming data is Avro, rather than converting the message to JSON (as per current behavior). To convert the record to JSON and break the link to its schema in the backing topic, you can set this field to convert_json (default until 3.3). |
externalStorage | Boolean | false | Choose where to store your encryption settings.false - Encryption settings will be stored within message headers.true - Encryption settings will be stored in a topic called _conduktor_gateway_encryption_configs by default, this can be renamed using the environment variable GATEWAY_ENCRYPTION_CONFIGS_TOPIC . |
kmsConfig | KMS | Configuration of one or multiple KMS. | |
enableAuditLogOnError | Boolean | true | The audit log will be enabled when an error occurs during encryption/decryption |
compressionType | Enum | none | The data is compressed before encryption (only for data configured with full payload encryption). Available values are: none , gzip , snappy , lz4 or zstd . |
List-based properties | |||
recordValue | Value and key encryption | Configuration to encrypt the record value. | |
recordKey | Value and key encryption | Configuration to encrypt the record key. | |
recordHeader | Headers encryption | Configuration to encrypt the record headers. | |
Schema-based properties | |||
defaultKeySecretId | Secret key template | Default keySecretId to use if none is set in the schema. It must be a unique identifier for the secret key, and can be a template for Crypto Shredding use cases. | |
defaultAlgorithm | Algorithm | AES128_GCM | Default algorithm to use if no algorithm is set in the schema. |
tags | List[String] | List of tags to search for in the schema to encrypt the specified fields. | |
namespace | String | conduktor. | Prefix of custom schema constraints for encryption. |
recordValue
(value encryption) and/or recordKey
(key encryption):
Key | Type | Default | Description |
---|---|---|---|
Full-payload encryption | |||
payload.keySecretId | Secret key template | Secret key, can be a template for Crypto Shredding use cases. | |
payload.algorithm | Algorithm | AES128_GCM | Algorithm to leverage. |
Field-Level encryption | |||
fields[].fieldName | String | Name of the field to encrypt. It can be a nested structure with a dot . such as education.account.username or banks[0].accountNo . | |
fields[].keySecretId | Secret Key template | AES128_GCM | Unique identifier for the secret key. You can store this key in your KMS by using the KMS key templates. It can be a template for crypto shredding use cases. |
fields[].algorithm | Algorithm | Algorithm to use to encrypt this field. |
recordHeader
.
Key | Type | Default | Description |
---|---|---|---|
Full-payload encryption | Configuration to encrypt the full payload. | ||
payload.keySecretId | Secret key template | Secret key, can be a template for crypto shredding use cases. | |
payload.algorithm | Algorithm | AES128_GCM | Algorithm to leverage. |
Field-level encryption | |||
fields[].fieldName | String | Name of the field to encrypt. It can be a nested structure with a dot . such as education.account.username or banks[0].accountNo . | |
fields[].keySecretId | Secret key template | AES128_GCM | Unique identifier for the secret key. It can be a template for crypto shredding use cases. |
fields[].algorithm | Algorithm | ||
Headers encryption | |||
header | String | Headers that match this regex will be encrypted. Warning: it can encrypt all headers including gateway headers |
namespace
value which is conduktor.
. If you have changed the namespace
value in the Interceptor configuration, please change the key name in your schema accordingly.
Key | Type | Default | Description |
---|---|---|---|
conduktor.keySecretId | Secret key template | Unique identifier for the secret key, and can be a template for crypto shredding use cases. | |
conduktor.algorithm | Algorithm | AES128_GCM | Algorithm to use to encrypt this field. |
conduktor.tags | List[String] | Fields tagged with a matching tag from your Interceptor will be encrypted using the keySecretId and algorithm specified in the schema.If these are not defined in the schema, the defaultKeySecretId and defaultAlgorithm from the Interceptor configuration will be used. |
keySecretId
set in the schemaalgorithm
set in the schematags
set in the schema, and one of them is part of the tags
list specified in the Interceptors.keySecretId
. That way, your secret keys will be dynamic.
Pattern | Replaced by |
---|---|
{{record.topic}} | Name of the topic you’re encrypting the data of. |
{{record.key}} | Key of the encrypted record. |
{{record.value.someValueFieldName}} | Value of the field called someValueFieldName If you’re doing field-level encryption, please ensure that someValueFieldName is not included in the fields to encrypt. Otherwise, you will not be able to decrypt it. |
{{record.value.someList[0].someValueField}} | Value of the field called someValueFieldName , in the first element of the list someList |
{{record.header.someHeader}} | Value of the header called someHeader |
"keySecretId": "{{record.topic}}-{{record.header.someHeader}}-{{record.key}}"
. This will create an encryption key called myTopic-myHeader-myKey
in memory.
If you want this key to be stored in your Vault KMS for instance, you can set: "keySecretId": "vault-kms://vault:8200/transit/keys/{{record.topic}}-{{record.header.someHeader}}-{{record.key}}"
.
keySecretId
that doesn’t match one of the schemas detailed below will be rejected and the encryption operation will fail.
If you want to make sure the key is well created in your KMS, you have to (1) make sure you have configured the connection to the KMS and (2) use the following format as keySecretId
:
KMS | KMS identifier prefix | Key URI format | Example |
---|---|---|---|
In-Memory | in-memory-kms:// | in-memory-kms://<key-id> | in-memory-kms://my-password-key-id |
Vault | vault-kms:// | vault-kms://<vault-host>/transit/keys/<key-id> | vault-kms://vault:8200/transit/keys/password-key-id |
Azure | azure-kms:// | azure-kms://<key-vault-name>.vault.azure.net/keys/<object-name>/<object-version> | azure-kms://my-key-vault.vault.azure.net/keys/conduktor-gateway/4ceb7a4d1f3e4738b23bea870ae8745d |
AWS | aws-kms:// | aws-kms://arn:aws:kms:<region>:<account-id>:key/<key-id> | aws-kms://arn:aws:kms:us-east-1:123456789012:key/password-key-id |
GCP | gcp-kms:// | gcp-kms://projects/<project-id>/locations/<location-id>/keyRings/<key-ring-id>/cryptoKeys/<key-id> | gcp-kms://projects/my-project/locations/us-east1/keyRings/my-key-ring/cryptoKeys/password-key-id |
AES128_GCM
(default)AES128_EAX
AES256_EAX
AES128_CTR_HMAC_SHA256
AES256_CTR_HMAC_SHA256
CHACHA20_POLY1305
XCHACHA20_POLY1305
AES256_GCM
none
gzip
snappy
lz4
zstd
DecryptPlugin
.
Key | Type | Default | Description |
---|---|---|---|
topic | String | .* | Topics matching this regex will have the Interceptor applied |
schemaRegistryConfig | SchemaRegistry | Configuration of your schema registry, is needed if you want to decrypt into Avro, JSON or Protobuf schemas. | |
kmsConfig | KMS | Configuration of one or multiple KMS | |
recordValueFields | List[String] | Only for field-level encryption - List of fields to decrypt in the value. If empty, we decrypt all the encrypted fields. | |
recordKeyFields | List[String] | Only for field-level encryption - List of fields to decrypt in the key. If empty, we decrypt all the encrypted fields. | |
recordHeaderFields | List[String] | Only for field-level encryption - List of headers to decrypt. If empty, we decrypt all the encrypted headers. | |
enableAuditLogOnError | Boolean | true | The audit log will be enabled when an error occurs during encryption/decryption |
errorPolicy | String | return_encrypted | Determines the action if there is an error during decryption. The options are return_encrypted (the encrypted payload is returned to the client) or fail_fetch (the client will receive an error for the fetch and no data). For Crypto Shredding, the policy should always be return_encrypted , otherwise the consumer will become permanently blocked by messages that have been deliberately been made un-decryptable. |
schemaRegistryConfig
in order to (de)serialize them.
Gateway supports Confluent-like and AWS Glue schema registries.
Key | Type | Default | Description |
---|---|---|---|
type | string | CONFLUENT | The type of schema registry to use: choose CONFLUENT (for Confluent-like schema registries including OSS Kafka) or AWS for AWS Glue schema registries. |
additionalConfigs | map | Additional properties maps to specific security-related parameters. For enhanced security, you can hide the sensitive values using environment variables as secrets. | |
Confluent-like | Configuration for Confluent-like schema registries | ||
host | string | URL of your schema registry. | |
cacheSize | string | 50 | Number of schemas that can be cached locally by this Interceptor so that it doesn’t have to query the schema registry every time. |
AWS Glue | Configuration for AWS Glue schema registries | ||
region | string | The AWS region for the schema registry, e.g. us-east-1 | |
registryName | string | The name of the schema registry in AWS (leave blank for the AWS default of default-registry ) | |
basicCredentials | string | Access credentials for AWS (see below section for structure) | |
AWS credentials | AWS credential configuration | ||
accessKey | string | The access key for the connection to the schema registry. | |
secretKey | string | The secret key for the connection to the schema registry. | |
validateCredentials | bool | true | true / false flag to determine whether the credentials provided should be validated when set. |
accountId | string | The Id for the AWS account to use. |
basicCredentials
section for the AWS Glue schema registry, the client we use to connect will instead attempt to find the connection information is needs from the environment, and the credentials required can be passed this way to Gateway as part of its core configuration. More information on the setup for this is found in the AWS documentation.
${MY_ENV_VAR}
.
We recommend you use this for Schema Registry or Vault secrets, and any other values you’d like to hide in the configuration.
Key | Type | Description |
---|---|---|
keyTtlMs | long | Key’s time-to-live in milliseconds. The default is 1 hour. Disable the cache by setting it to 0. |
in-memory | In-memory | In Memory KMS that is not persistent, internal to Gateway, for demo purposes only. |
gateway | Gateway | Key storage managed by Gateway but with secret management still delegated to an external KMS |
vault | Vault KMS | HashiCorp Vault KMS |
azure | Azure KMS | Azure Key Vault |
aws | AWS KMS | AWS KMS |
gcp | GCP KMS | Google Key Management |
Key | Type | Description |
---|---|---|
masterKeyId | String | The master key secret Id used to encrypt any keys stored in the Gateway managed storage. This is in the same format as the keySecretId that’s used for encryption and the valid values are the same. |
maxKeys | Number | The maximum number of secret Id references to be cached in memory for re-use. To avoid creating new encryption keys (DEKs), this needs to be larger than the total number of expected secret Ids. By default, it’s the same as maxKeys in cache config or 1000,000, if maxKeys isn’t set. |
masterKeyId
is used to secure every key for this configuration, stored by Gateway. Find out more about the secret key formats. You have to also supply a valid configuration for the KMS type referenced by the master key so this can be used.
If this key is dropped from the backing KMS, then all keys stored by Gateway for that master key will become unreadable.
gateway-kms://
as the secret key type:
masterKeyId
secret from vault and123456
, the associated EDEK would be stored on a kafka record with the following key:
{{record.key}}
template giving a unique key for each Kafka record key).
If there are multiple Gateway nodes running, it’s also possible for multiple DEKs/EDEKs to be generated for the same record key. Two nodes processing different records with the same record key at the same time could both assume they were generating a DEK/EDEK for the first time. In this scenario, there would be two EDEKs in the Gateway storage with the same keyId
but they would each have a different UUID
.
For example:
gateway-kms
secret key Id type, the decryption configuration used to decrypt the data has to also specify the masterKeyId
, so that it can securely decrypt the keys stored in the local Gateway storage.
Here’s a sample setup:
gateway-kms
secret key Id type, you can efficiently crypto shred EDEKs in the Gateway storage, so that anyone using the decryption plugin will immediately lose access to the associated encrypted data.
To do this, scan the Gateway storage Kafka topic (by default, _conduktor_gateway_encryption_keys
) for every message matching the associated qualified secret Id.
For example, a qualified secretId of gateway-kms://fieldKeySecret-name-123456
might have the following keys:
null
(i.e. a tombstone) will effectively perform Crypto Shredding.
This process won’t prevent the creation of new keys if new messages are sent using the same record key; it only ensures that messages using the crypto shredded keys remain unrecoverable.
vault
:
You can use one of these two authentication methods:
Key | Type | Description |
---|---|---|
uri | String | Vault URI. |
version | String | Version. |
namespace | String | Namespace. |
Managed identity | Load authentication information from the below environment variables. | |
VAULT_URI | Vault server base URI. | |
VAULT_ENGINE_VERSION | Vault KV Secrets Engine version. | |
VAULT_NAMESPACE | Vault namespace. | |
type | String | Required for all types of VaultKMSConfig. Determines the type of authentication to use. Supported types: - TOKEN - USERNAME_PASSWORD - GITHUB - LDAP - APP_ROLE - KUBERNETES - GCP - AWS_EC2_PKCS7 - AWS_EC2 - AWS_IAM - JWT |
Key | Type | Description |
---|---|---|
Token authentication | Use token authentication. | |
type | String | Must be TOKEN . Indicates the type of authentication. |
token | String | Security token for accessing Vault. |
Managed identity | Load authentication information from the below environment variables. | |
VAULT_AUTH_TYPE | String | Must be TOKEN . Indicates the type of authentication. |
VAULT_TOKEN | Token to use for accessing Vault. | |
Username and password | Use username and password authentication. | |
type | String | Must be USERNAME_PASSWORD . Indicates the type of authentication. |
username | String | Username for accessing Vault. |
password | String | Password for accessing Vault. |
userpassAuthMount | String | (Optional) Mount path for the userpass auth method. |
Managed identity | Load authentication information from the below environment variables. | |
VAULT_AUTH_TYPE | String | Must be USERNAME_PASSWORD . Indicates the type of authentication. |
VAULT_USERNAME | Username for accessing Vault. | |
VAULT_PASSWORD | Password for accessing Vault. | |
VAULT_AUTH_MOUNT | (Optional) Mount path for the userpass auth method. | |
GitHub authentication | Use GitHub token authentication. | |
type | String | Must be GITHUB . Indicates the type of authentication. |
token | String | GitHub personal access token. |
githubAuthMount | String | (Optional) Mount path for the GitHub auth method. |
Managed identity | Load authentication information from the below environment variables. | |
VAULT_AUTH_TYPE | String | Must be GITHUB . Indicates the type of authentication. |
VAULT_GITHUB_TOKEN | GitHub token for accessing Vault. | |
VAULT_AUTH_MOUNT | (Optional) Mount path for the GitHub auth method. | |
LDAP authentication | Use LDAP authentication. | |
type | String | Must be LDAP . Indicates the type of authentication. |
username | String | LDAP username. |
password | String | LDAP password. |
ldapAuthMount | String | (Optional) Mount path for the LDAP auth method. |
Managed identity | Load authentication information from the below environment variables. | |
VAULT_AUTH_TYPE | String | Must be LDAP . Indicates the type of authentication. |
VAULT_LDAP_USERNAME | LDAP username. | |
VAULT_LDAP_PASSWORD | LDAP password. | |
VAULT_AUTH_MOUNT | (Optional) Mount path for the LDAP auth method. | |
AppRole authentication | Use AppRole authentication. | |
type | String | Must be APP_ROLE . Indicates the type of authentication. |
roleId | String | Role ID for AppRole authentication. |
secretId | String | Secret ID for AppRole authentication. |
path | String | (Optional) Mount path for the AppRole auth method. |
Managed identity | Load authentication information from the below environment variables. | |
VAULT_AUTH_TYPE | String | Must be APP_ROLE . Indicates the type of authentication. |
VAULT_APP_ROLE_ID | Role ID for AppRole authentication. | |
VAULT_APP_SECRET_ID | Secret ID for AppRole authentication. | |
VAULT_APP_PATH | (Optional) Mount path for the AppRole auth method. | |
Kubernetes authentication | Use Kubernetes authentication. | |
type | String | Must be KUBERNETES . Indicates the type of authentication. |
role | String | Kubernetes role. |
jwt | String | Kubernetes JWT token. |
Managed identity | Load authentication information from the below environment variables. | |
VAULT_AUTH_TYPE | String | Must be KUBERNETES . Indicates the type of authentication. |
VAULT_KUBERNETES_ROLE | Kubernetes role. | |
VAULT_KUBERNETES_JWT | Kubernetes JWT token. | |
GCP authentication | Use Google Cloud Platform authentication. | |
type | String | Must be GCP . Indicates the type of authentication. |
role | String | GCP role for authentication. |
jwt | String | JWT token issued by Google Cloud Platform. |
Managed identity | Load authentication information from the below environment variables. | |
VAULT_AUTH_TYPE | String | Must be GCP . Indicates the type of authentication. |
VAULT_GCP_ROLE | GCP role for authentication. | |
VAULT_GCP_JWT | JWT token for accessing Vault. | |
AWS EC2 authentication (PKCS7) | Use AWS EC2 PKCS7 authentication. | |
type | String | Must be AWS_EC2_PKCS7 . Indicates the type of authentication. |
role | String | AWS role for EC2 authentication. |
pkcs7 | String | PKCS7 identity document. |
nonce | String | (Optional) Nonce value for EC2 authentication. |
Managed identity | Load authentication information from the below environment variables. | |
VAULT_AUTH_TYPE | String | Must be AWS_EC2_PKCS7 . Indicates the type of authentication. |
VAULT_AWS_ROLE | AWS role for EC2 authentication. | |
VAULT_AWS_PKCS7 | PKCS7 identity document. | |
VAULT_AWS_NONCE | (Optional) Nonce value for EC2 authentication. | |
VAULT_AUTH_MOUNT | (Optional) Mount path for the AWS EC2 PKCS7 auth method. | |
AWS EC2 authentication | Use AWS EC2 identity authentication. | |
type | String | Must be AWS_EC2 . Indicates the type of authentication. |
role | String | AWS role for EC2 authentication. |
identity | String | AWS identity document. |
signature | String | AWS signature for authentication. |
nonce | String | Nonce value for EC2 authentication. |
Managed identity | Load authentication information from the below environment variables. | |
VAULT_AUTH_TYPE | String | Must be AWS_EC2 . Indicates the type of authentication. |
VAULT_AWS_ROLE | AWS role for EC2 authentication. | |
VAULT_AWS_IDENTITY | AWS identity document. | |
VAULT_AWS_SIGNATURE | AWS signature for authentication. | |
VAULT_AWS_NONCE | (Optional) Nonce value for EC2 authentication. | |
VAULT_AUTH_MOUNT | (Optional) Mount path for the AWS EC2 auth method. | |
AWS IAM authentication | Use AWS IAM Authentication. | |
type | String | Must be AWS_IAM . Indicates the type of authentication. |
role | String | AWS role for IAM authentication. |
iamRequestUrl | String | IAM request URL for authentication. |
iamRequestBody | String | IAM request body for authentication. |
iamRequestHeaders | String | IAM request headers for authentication. |
Managed identity | Load authentication information from the below environment variables. | |
VAULT_AUTH_TYPE | String | Must be AWS_IAM . Indicates the type of authentication. |
VAULT_AWS_ROLE | AWS role for IAM authentication. | |
VAULT_AWS_IAM_REQUEST_URL | IAM request URL for authentication. | |
VAULT_AWS_IAM_REQUEST_BODY | IAM request body for authentication. | |
VAULT_AWS_IAM_REQUEST_HEADERS | IAM request headers for authentication. | |
JWT authentication | Use JWT authentication. | |
type | String | Must be JWT . Indicates the type of authentication. |
jwt | String | JWT token for authentication. |
provider | String | JWT provider for authentication. |
role | String | JWT role for authentication. |
Managed identity | Load authentication information from the below environment variables. | |
VAULT_AUTH_TYPE | String | Must be JWT . Indicates the type of authentication. |
VAULT_JWT | JWT token for authentication. | |
VAULT_JWT_PROVIDER | JWT provider for authentication. |
azure
.
You can use one of these two authentication methods:
Key | Type | Description |
---|---|---|
Token | ||
tokenCredential.clientId | string | Client ID. |
tokenCredential.tenantId | string | Tenant ID. |
tokenCredential.clientSecret | string | Client secret. |
Username and password | ||
usernamePasswordCredential.clientId | string | Client ID. |
usernamePasswordCredential.tenantId | string | Tenant ID. |
usernamePasswordCredential.username | string | Username. |
usernamePasswordCredential.password | string | Password. |
Managed identity | Configure the KMS from the context, and not using variables. This will be overwritten if a specific KMS is configured within the Interceptor. |
aws
.
You can use one of these two authentication methods:
Key | Type | Description |
---|---|---|
Basic authentication | ||
basicCredentials.accessKey | String | Access key. |
basicCredentials.secretKey | String | Secret key. |
Session | ||
sessionCredentials.accessKey | String | Access key. |
sessionCredentials.secretKey | String | Secret key. |
sessionCredentials.sessionToken | String | Session token. |
Managed identity | Configure the KMS from the context, not using variables. This will be overwritten if a specific KMS is configured within the Interceptor. |
gcp
:
You must first configure the service account key file.
For enhanced security, you can hide the sensitive values using environment variables as secrets.
Key | Type | Description |
---|---|---|
serviceAccountCredentialsFilePath | String | Service account key file in GCP. |
Managed identity | Configure the KMS from the context, and not using variables. This will be overwritten if a specific KMS is configured within the Interceptor. |
keySecretId
, algorithm
, tags
match) will be encrypted.keySecretId
, algorithm
,
if any missed, would be encrypted with the associated default ones in the Interceptor configuration.tags
has element with is in the Interceptor configuration.conduktor.
):
password
would be encrypted with the associated keySecretId, algorithm etc.visa
would be encryption with the associated keySecretId and the default algorithm provided in the Interceptor configuration.location
would be encrypted with defaultSecret, defaultAlgorithm because tags has PII
with is in the Interceptor configuration.keySecretId
, algorithm
, tags
without match) are left untouched.conduktor.
):
conduktor.
):
In Protobuf, since we are using the Confluent schema registry, we use the (confluent.field_meta).params
(with type map<string, string
) for field options. Here’s how it can be defined: