GATEWAY_PORT_COUNT
. The recommended number of ports in production is double the amount of the Kafka brokers (to cover the growth of the Kafka cluster).
Configure port-based routing using these environment variables:
GATEWAY_ADVERTISED_HOST
GATEWAY_PORT_START
GATEWAY_PORT_COUNT
GATEWAY_MIN_BROKERID
GATEWAY_MIN_BROKERID
. E.g., in a three broker cluster with IDs 1, 2, 3, GATEWAY_MIN_BROKERID
should be set to 1 and the default port count will be 5.
We recommend SNI routing when not using a sequential and stable broker IDs range to avoid excessive port assignment. E.g., a three broker cluster with IDs 100, 200, 300 with GATEWAY_MIN_BROKERID
=100 will default to 203 ports and would fail if broker ID 400 is introduced.
GATEWAY_CLUSTER_ID
in each node’s deployment configuration. This configuration ensures that all nodes join the same consumer group, enabling them to consume the internal license topic from your Kafka cluster. This is how the nodes recognize each other as members of the same Gateway cluster.
When a client connects to one of the Gateway nodes to request metadata, the following process occurs (assuming GATEWAY_FEATURE_FLAGS_INTERNAL_LOAD_BALANCING
is set to true
, which is the default setting):
GATEWAY_RACK_ID
, then the mapping will take this into consideration and a Gateway node in the same rack as the Kafka broker will be assigned.GATEWAY_FEATURE_FLAGS_INTERNAL_LOAD_BALANCING: false
.
In this case, you would deploy your own load balancer, such as HAProxy, to manage traffic distribution. This would allow you to configure the stickiness of the load balancer as required.
Here’s an example where:
GATEWAY_ADVERTISED_LISTENER
of the Gateway nodes to the Load Balancer’s hostname. If this isn’t done, applications will attempt to connect directly to Gateway, bypassing the Load Balancer.KAFKA_
are mapped to configuration properties for connecting Gateway to the Kafka cluster.
As Gateway is based on the Java-based Kafka-clients, it supports all configuration properties that Java-clients do.
Kafka configuration properties are mapped to Gateway environment variables as follows:
KAFKA_
prefix.
, with an underscore, _
bootstrap.servers
is set by the KAFKA_BOOTSTRAP_SERVERS
environment variable.
PLAINTEXT
, SASL_PLAINTEXT
, SASL_SSL
and SSL
.
These can be used with all SASL mechanisms supported by Apache Kafka: PLAIN
, SCRAM-SHA
, OAuthBearer
, Kerberos
etc. In addition, we support IAM authentication for AWS MSK clusters.
In the following examples, we provide blocks of environment variables which can be provided to Gateway, e.g. in a docker-compose file, or a helm
deployment.
Information which should be customized is enclosed by <
and >
.
PLAINTEXT
.
In this case you just need the bootstrap servers:
GATEWAY_SECURITY_PROTOCOL
to DELEGATED_SASL_PLAINTEXT
. Clients will then be able to authenticate using their own API keys/secrets.
Read
on internal topics and they should existRead
on internal topics and they should existGATEWAY_ACL_STORE_ENABLED=true
and then you can use AdminClient to maintain ACLs with any service account declared in GATEWAY_ADMIN_API_USERS
.
Principal
that represents the authenticated identity of the Kafka client.
We can split this authentication and security configuration into two aspects:
Principal
for mTLS connection can be detected from the subject certificate using the same feature as in Apache Kafka, the SSL principal mapping.Clients ⟶ GW transit in plaintext | Clients ⟶ GW transit is encrypted | |
---|---|---|
Anonymous access only | Security protocol: PLAINTEXT Authentication mechanism: None | Security protocol: SSL Authentication mechanism: None |
Credentials managed by Gateway | Security protocol: SASL_PLAINTEXT Authentication mechanism: PLAIN | Security protocol: SASL_SSL Authentication mechanism: PLAIN |
Gateway configured with OAuth | Security protocol: SASL_PLAINTEXT Authentication mechanism: OAUTHBEARER | Security protocol: SASL_SSL Authentication mechanism: OAUTHBEARER |
Clients are identified by certificates (mTLS) | Not possible (mTLS means encryption) | Security protocol: SSL Authentication mechanism: MTLS |
Credentials managed by Kafka | Security protocol: DELEGATED_SASL_PLAINTEXT Authentication mechanism: PLAIN , SCRAM-SHA-256 , SCRAM-SHA-512 , OAUTHBEARER orAWS_MSK_IAM | Security protocol: DELEGATED_SASL_SSL Authentication mechanism: PLAIN , SCRAM-SHA-256 , SCRAM-SHA-512 , OAUTHBEARER orAWS_MSK_IAM |
GATEWAY_SECURITY_PROTOCOL
configuration.
Note that you don’t set an authentication mechanism on the client to Gateway side of the proxy, i.e. GATEWAY_SASL_MECHANISM
does not exist and is never configured by the user. Instead, Gateway will try to authenticate the client as it presents itself.
For example, if a client is using OAUTHBEARER
, Gateway will use the OAuth configuration to try authenticate it. If a client arrives using PLAIN
, Gateway will try use either the SSL configuration or validate the token itself, depending on the security protocol.
In addition to all the security protocols that Apache Kafka supports, Gateway adds two new protocols:DELEGATED_SASL_PLAINTEXT
and DELEGATED_SASL_SSL
for delegating to Kafka.
Principal
for an mTLS connection can be detected from the subject of the certificate using the same feature as Apache Kafka, the SSL principal mapping .
Gateway configuration:
GATEWAY_USER_POOL_SECRET_KEY
has to be set to a random base64 encoded value of 256bits long to ensure that tokens aren’t forged. For example: openssl rand -base64 32
. Otherwise, a default value for signing tokens will be used.
Client configuration:
audience
and issuer
).
Gateway configuration:
GATEWAY_USER_POOL_SECRET_KEY
to a random value to ensure that tokens cannot be forged. Otherwise, a default value for signing tokens will be used.
Client configuration:
audience
and issuer
).
Gateway configuration:
PLAINTEXT
by default.
Here’s our mapping from the Kafka cluster’s defined protocol:
Kafka cluster security protocol | Gateway cluster inferred security protocol |
---|---|
SASL_SSL | DELEGATED_SASL_SSL |
SASL_PLAINTEXT | DELEGATED_SASL_PLAINTEXT |
SSL | SSL |
PLAINTEXT | PLAINTEXT |
order
has been created on Virtual Cluster vc-alice
. Let’s see how other Virtual Clusters and Backing cluster perceive this:
delivery.timeout.ms
for JVM-based clients) setting is large enough to cover the time necessary for the operations team to discover failure of the primary cluster and initiate a failover procedure. Especially for JVM-based clients, the default delivery timeout of 2 minutes might be too short.
3.3.0
+2.8.2
+gateway.roles
entry to mark the failover cluster - note that the API keys differ in the Confluent Cloud example below:
GATEWAY_BACKEND_KAFKA_SELECTOR
:
clusterA
, clusterB
) may appear in the bootstrap server address or client logs. To prevent unintended exposure, avoid using sensitive names or information in cluster IDs.gateway.roles
entry to mark the upstream clusters.GATEWAY_BACKEND_KAFKA_SELECTOR
:aclEnabled
must be true
and superUsers
must not be empty.mypartner-super-user.properties
:
mypartner-partner-user.properties
:
mypartner-partner-user.properties
file and the correct bootstrap server details with your partner.