Skip to main content
Quick navigation

Docker

Conduktor Gateway is provided as a Docker image and Helm chart.

It should be deployed and managed how is best for your organisation and use case(s). This could be a single container, or more likely a number of Gateway instances should be deployed and scaled to meet your needs. Optionally, the instances could be deployed behind a load balancer.

Use this quick start guide to help you get started.

Jump to:

Running the Gateway

info

For a fully self-contained quick-start, see the Docker Compose.

In its simplest form, Gateway can be run from a simple Docker run command with the credentials to connect to your existing Kafka cluster.

Start a local Kafka

Create a new directory (note the docker network will be derived from the directory name):

mkdir gateway-quick-start && cd gateway-quick-start

Run the below command to start a single node Kafka and ZooKeeper:

curl -L https://releases.conduktor.io/single-kafka -o docker-compose.yml && docker compose up -d 

Start Conduktor Gateway

Run the below command to start Conduktor Gateway and configure Docker networking between the two containers:

  docker run \
--network gateway-quick-start_default \
-e KAFKA_BOOTSTRAP_SERVERS=kafka1:29092 \
-e GATEWAY_ADVERTISED_HOST=localhost \
-e GATEWAY_PORT_START=9099 \
-e GATEWAY_PORT_COUNT=1 \
-p 9099:9099 \
-d \
conduktor/conduktor-gateway:3.0.5

By default, the Gateway uses port-based routing and listens on as many ports as there are Kafka brokers. In this case, we started a single-node Kafka cluster and opened 1 port.

At this stage you have:

  • Kafka running and its brokers available on localhost:9092
  • Gateway acting as a proxy to the backing Kafka cluster, accessible at loalhost:9099

Connecting your clients

Your clients can now interact with Conduktor Gateway like any other Kafka cluster.

Example: creating a topic via Gateway using the Apache Kafka command line client:

bin/kafka-topics.sh --create --topic orders --bootstrap-server localhost:9099

Next Steps

This quick start shows the basics, demonstrating Conduktor Gateway acting as a network proxy for Kafka. However, the real value comes with configuring interceptors, which are pluggable components that augment Kafka by intercepting specific requests of the Kafka protocol and applying operations to it.

View demos that demonstrate how interceptors are used to satisfy specific use cases such as encryption, data quality and safeguarding your cluster with technical and business rules.

Connecting to secured Kafka

Your Kafka's bootstrap server, along with its authentication method should be configured using environment variables.

Conduktor Gateway connects to Kafka just like any other client.

Security configurations are provided using this scheme. For example:

ssl.truststore.location

becomes:

KAFKA_SSL_TRUSTSTORE_LOCATION

Confluent Cloud Example

Below shows the most simple way to get started with Confluent Cloud.

info

By default, Gateway assumes you want the same security protocol between your clients and Gateway, as between your Gateway and Kafka.

However, this example uses DELEGATED_SASL_PLAINTEXT for the GATEWAY_SECURITY_PROTOCOL. For quick start purposes, this avoids needing to configure SSL certificates when connecting to Conduktor Gateway.

  docker run \
-e KAFKA_BOOTSTRAP_SERVERS=$CONFLUENT_CLOUD_KAFKA_BOOTSTRAP_SERVER \
-e KAFKA_SASL_MECHANISM=PLAIN \
-e KAFKA_SECURITY_PROTOCOL=SASL_SSL \
-e KAFKA_SASL_JAAS_CONFIG='org.apache.kafka.common.security.plain.PlainLoginModule required username="$CONFLUENT_CLOUD_API_KEY" password="$CONFLUENT_CLOUD_API_SECRET' \
-e GATEWAY_SECURITY_PROTOCOL=DELEGATED_SASL_PLAINTEXT \
-e GATEWAY_ADVERTISED_HOST=localhost \
-e GATEWAY_CLUSTER_ID=test \
-p 6969-6999:6969-6999 \
-d \
conduktor/conduktor-gateway:3.0.5

Note that if you wish to maintain the SSL/TLS connection between clients and Conduktor Gateway, see Client to Gateway Configuration.

By default, the Gateway uses port-based routing and listens on as many ports as there are Kafka brokers. In this case, we open 30 ports to account for Confluent Cloud clusters with many brokers. However, you may need to open more ports depending on the size of your cluster.

If you need support with your configuration, please contact us for support.

Docker Compose

The below example demonstrates an environment consisting of:

cat docker-compose.yaml

1. Start the Docker environment

Start all your docker services, wait for them to be up and ready, then run in background:

  • --wait: Wait for services to be running|healthy. Implies detached mode.
  • --detach: Detached mode: Run containers in the background
docker compose up --detach --wait

2. Create a topic via Conduktor Gateway

kafka-topics \
--bootstrap-server localhost:6969 \
--replication-factor 1 \
--partitions 1 \
--create --if-not-exists \
--topic orders

3. Produce a message to your topic

echo '{"orderId":"12345","customerId":"67890","price":10000}' | \
kafka-console-producer \
--bootstrap-server localhost:6969 \
--topic orders

4. Consume a message from your topic

kafka-console-consumer \
--bootstrap-server localhost:6969 \
--topic cars \
--from-beginning \
--max-messages 1 \
--timeout-ms 10000 | jq

5. Next Steps: Configure an interceptor

This quick start shows the basics, demonstrating Conduktor Gateway can be interacted with like any other Kafka cluster.

However, the real value comes with configuring interceptors, which are pluggable components that augment Kafka by intercepting specific requests of the Kafka protocol and applying operations to it.

View demos that demonstrate how interceptors are used to satisfy specific use cases such as encryption, data quality and safeguarding your cluster with technical and business rules.