Overview
This feature is available with Conduktor Shield only.
- On produce: data will be encrypted before it gets to the broker, i.e. encrypted before it enters Kafka. Gateway will intercept the records, encrypt them following your definition within the Interceptor configuration and then pass it to Kafka.
- On consume: the original data is already in Kafka. The Gateway will encrypt the original data as the consumer consumes it.
- Full payload: you want to encrypt the key, the value, or the headers of your records, which can be done on structured or unstructured messages. This is particularly useful when the entire message is sensitive.
- Field-level: define which fields (Avro, Protobuf and JSON) in your payload need to be encrypted. This is very useful when only certain parts of a message contain sensitive data (e.g. passwords or PII (Personally Identifiable Information)). You can choose the fields with a schema-based solution, or not. The choice will depend on how you produce the messages and what you want to encrypt:
- Schema-based - fields are encrypted based on tags you include in the record schema itself or
- List-based - fields are encrypted based on the list you specify in the Interceptor configuration.
How it works
Once configured, the encryption and decryption processes are seamlessly managed by the Interceptor. Encryption: The Interceptor identifies the data that needs to be encrypted and the KMS details to share the encryption key, Gateway will then encrypt and produce the message. Decryption: Similar to encryption, the Interceptor can decrypt either the entire message, specific fields or all the fields, based on your configuration. Flexibility and Compatibility You can refine how the encryption is configured with a choice of algorithm and KMS provider. Multiple Encryption Algorithms: The Interceptor supports a variety of encryption algorithms, allowing you to choose the one that best meets your security requirements. KMS Integration: It integrates with various Key Management Services (KMS), providing flexibility in how you manage and store encryption keys.Encryption process
-
Identify data. The Interceptor first determines, based on its configuration, what data needs to be encrypted. This may include the entire message, specific fields or all the fields within the message. For example, if you have configured the Interceptor to encrypt a
password
field, it will target this field within the incoming Kafka record for encryption. -
Retrieve key. The Interceptor then generates a key and shares it with the the configured KMS (Key Management Service) or retrieves it, if it exists. Supported KMS options include Vault, Azure, AWS, GCP or an in-memory service for local development only. The key is fetched using the
keySecretId
specified in your configuration to ensure the correct key is utilized. - Encrypt. Once the key is generated/retrieved, the Interceptor encrypts the identified data using the specified encryption algorithm. The original data within the message is now replaced with the encrypted version.
- Transmit. Finally, the encrypted data is either stored as is if it is an Avro record or converted into a JSON format and is then transmitted as a string to the designated destination.
Decryption process
- Identify data. The Interceptor first determines, based on its configuration, which data needs to be decrypted. This may include the entire message, specific fields, or all the fields within the message.
-
Retrieve key. Next, the Interceptor retrieves the decryption key from the KMS. Typically, this is the same key that was used during encryption. The correct key is obtained using the
keySecretId
provided in your Interceptor configuration and that’s stored in the header of the record, on the backing Kafka. - Decrypt. The Interceptor then decrypts the identified data using the retrieved key and the specified encryption algorithm. The decrypted data replaces the encrypted data within the message.
- Consume. Once decrypted, the message is ready for consumption by the end-user or application. The Interceptor ensures that the decrypted data is correctly formatted and fully compatible with the Kafka record structure.
These encryption and decryption processes are fully transparent to the end-user or application. The Interceptor manages all these operations, allowing you to concentrate on the core business logic.
Key management
The Interceptor uses theenvelope encryption
technique to encrypt data. Here are some key terms we’ll use:
Term | Definition |
---|---|
KMS | Key Management Service: A system responsible for managing and storing cryptographic keys, including the KEK. |
KEK | Key Encryption Key: A key stored in the KMS, used to encrypt the DEK. Notably, the KEK is never exposed to or known by the Interceptor. |
DEK | Data Encryption Key: A key generated by the Interceptor, used to encrypt the actual data. |
EDEK | Encrypted Data Encryption Key: The DEK that has been encrypted by the KEK, ensuring that the DEK remains secure when stored or transmitted. |
- Generates a DEK that is used to encrypt the data.
- Sends the DEK to the KMS, so it encrypts it using its KEK and returns the EDEK to Gateway.
- Caches the DEK and EDEK in memory for a configurable Time to Live (TTL).
- Encrypts the data using the DEK.
- Stores the EDEK alongside the encrypted data and both are sent to the backing Kafka.
- Retrieves the EDEK that’s stored with the encrypted data.
- Sends the EDEK to the KMS which decrypts it (using the KEK) and returns the DEK to Gateway.
- Decrypts the data using the DEK.
