Overview
What is Conduktor Gateway? A Kafka Proxy
Conduktor Gateway is a network proxy for Apache Kafka, complete with an extensible and dynamic plugin mechanism that can be used to add technical and business logic on top of your existing Kafka deployment.
This can be used to provide functionality that is not available in Kafka natively, such as:
- Centrally configure encryption at the field-level or full payload, to secure your data during transit and at rest, before the cluster
- Mask sensitive data across topics and set access rules, so users only see what they’re authorized to
- Set granular RBAC controls to manage access and permissions for data at the cluster, team or individual level
- Leverage multi-tenancy with virtual clusters to optimize resources and reduce operational overheads
- Empower development teams to manage their data within a federated control framework, accelerating project delivery
Conduktor Gateway is vendor-agnostic, meaning it supports all Kafka providers (Confluent, AWS MSK, Redpanda, Aiven, Apache Kafka), both cloud and on-premise.
To discover all functionality, see Gateway demos.
How it works
Conduktor Gateway is deployed between your client applications and existing Kafka clusters. As it's Kafka protocol compliant, there are minimal adjustments required for clients other than pointing to a new bootstrap server.
Authentication
Much like Kafka brokers, Gateway supports multiple security protocols for Kafka client to Gateway authentication, such as PLAINTEXT, SSL, SASL SSL, mTLS. Equally, Gateway supports all Kafka security protocols for Gateway to Kafka authentication. Read more about Gateway to Kafka Authentication or Client to Gateway Authentication.
Interceptors
Once Gateway is deployed, interceptors are used to add technical and business logic, such as message encryption, inside your Gateway deployment. Interceptors can be deployed and managed through the HTTP API, and targeted at different granularities (Global, Virtual Cluster, Group, Username). Read more about Interceptors
Processing Flow
Kafka protocol requests, such as Produce requests, pass sequentially through each of the components in the pipeline, before being forwarded to the broker. When the broker returns a response, such as a Produce response, the components in the pipeline are invoked in reverse order, each having the opportunity to inspect and/or manipulate the response. Eventually, a response is returned to the client.
What about scaling?
The Gateway is stateless and can be scaled horizontally by adding more instances and distributing the incoming traffic using a load balancer.
What about resilience?
Much like Kafka, if a broker dies it can be restarted whilst Gateway keeps running. As the Gateway is Kafka protocol compliant, your applications remain available.
What about latency?
By default, the Gateway operates with minimal impact on performance, typically adding only milliseconds of latency. However, if you begin implementing more resource-intensive features, such as encryption utilizing a Key Management Service (KMS), there will naturally be a slight increase in overhead.