Overview
This page gives you copy-paste examples to encrypt and decrypt Kafka data with Conduktor Gateway. Each example is an end-to-end scenario: a message arrives, an encryption Interceptor protects it on produce, and aDecryptPlugin returns it on consume.
Find out more about:
From our blog: Stop building Kafka encryption libraries Why custom encryption code creates tech debt and how a proxy-based approach eliminates it.
On-consume encryption plugins (
FetchEncryptPlugin and FetchEncryptSchemaBasedPlugin) were removed in Gateway v3.19.0. To protect data on consume, use produce-side encryption or data masking. See removed Interceptors for their previous documentation.How to read these examples
The examples share one sample record, a customer with Personally Identifiable Information (PII):- Field-level examples encrypt the
email,ssnandphonefields of this record. - Apply a CLI example with
conduktor apply -f <file>.yaml. - The examples keep secrets (Vault tokens, Schema Registry credentials) out of the configuration by referencing environment variables set in the Gateway container. Use
$${VAR}in the CLI examples and${VAR}in the curl examples: both resolve in Gateway. Find out more about environment variables for secrets.
Full payload encryption
Use full payload encryption to protect the entire record value without looking inside it. Gateway treats the value as one blob, so you don’t need a Schema Registry. Scenario: any message produced to a topic has its whole value encrypted on produce, and decrypted on consume.Encrypt the full payload
- CLI
- curl
Decrypt the full payload
On consume, aDecryptPlugin with access to the same KMS returns the original value. You don’t list any fields, so Gateway decrypts the whole payload.
- CLI
- curl
Field-level encryption
Use field-level encryption to protect named fields and leave the rest of the record readable. Each field can use its own key and algorithm, so you can grant different consumers access to different fields. Scenario: your producers send the sample customer record, and you want to encryptemail, ssn and phone, each with its own key.
The field path syntax is the same whether the value is plain JSON or Avro. The one difference is that Avro records go through the Schema Registry, so the Avro example adds a schemaRegistryConfig block. Plain JSON needs no Schema Registry.
Encrypt selected fields in a JSON message
- CLI
- curl
Encrypt selected fields in an Avro message
The config is identical, plus aschemaRegistryConfig block so Gateway can deserialize the Avro record and find the fields. This example also secures the Schema Registry with environment variables.
- CLI
- curl
Decrypt all fields
To return every encrypted field to the consumer, deploy aDecryptPlugin with no field list. Gateway decrypts all the fields it can, using the same KMS.
- CLI
- curl
To decrypt Avro, JSON Schema or Protobuf records, add a
schemaRegistryConfig block to the DecryptPlugin config, as shown in the Avro encryption example above.Decrypt specific fields for a user
To grant different consumers access to different fields, scope theDecryptPlugin and list only the fields that consumer may read with recordValueFields. Set the scope with metadata.scope (username, group or vCluster) in both the CLI and the API.
Scenario: support agents (user support-agent) may read phone only, and the compliance team (group compliance) may read email and ssn.
- CLI
- curl
phone field in clear text while email and ssn stay encrypted. The compliance team reads email and ssn while phone stays encrypted.
Schema-based field-level encryption
Schema-based encryption is field-level encryption where you mark the fields to encrypt in the schema itself, instead of listing them in the Interceptor. Use it when you want the schema to own the encryption rules. It needs a Schema Registry and uses theEncryptSchemaBasedPlugin.
Tag fields in your schema
Add Conduktor constraints to the fields you want to encrypt, using the defaultconduktor. namespace. A field is encrypted when it has a conduktor.keySecretId, a conduktor.algorithm, or a conduktor.tags value that matches a tag in the Interceptor.
- JSON Schema
- Avro
email uses the key and algorithm set in the schema. ssn and phone carry the PII tag, so they use the defaultKeySecretId and defaultAlgorithm from the Interceptor. For the Protobuf syntax, see the encryption reference.
Encrypt fields defined in the schema
Deploy theEncryptSchemaBasedPlugin with the tags to look for and the defaults to apply.
- CLI
- curl
Decrypt schema-based fields
Decryption is the same for every field-level Interceptor, so you use the sameDecryptPlugin. Because the records are Avro or JSON Schema, add a schemaRegistryConfig block. With no field list, Gateway decrypts every encrypted field.
- CLI
- curl
Combine Interceptors
You can stack encryption Interceptors to apply different strategies to different parts of your data. A common pattern is field-level encryption for known sensitive fields, with full payload encryption as a fallback for messages that don’t match. This relies on two settings:priority: Interceptors run in priority order, lowest first.errorPolicy: skip_already_encrypted: an encryption Interceptor skips records a previous Interceptor already encrypted.
Encrypt fields with full payload fallback
The field-level Interceptor runs first (priority 1) and encryptsemail, ssn and phone. The full payload Interceptor runs second (priority 2) and encrypts anything left, skipping records that were already encrypted.
- CLI
- curl
- A message arrives on a
sensitive-.*topic. - The field-level Interceptor (priority 1) runs first. If the message has
email,ssnorphone, it encrypts those fields and flags the record. - The full payload Interceptor (priority 2) runs second. Because
errorPolicyisskip_already_encrypted, it skips records the field-level Interceptor already handled. Records with no matching fields get full payload encryption instead. - On consume, a single
DecryptPluginhandles both encryption types.
If you also offload large messages to cloud storage, see Combine with encryption for the recommended priority order so ciphertext, not plaintext, lands in your cloud storage.