Skip to main content
Quick navigation

Docker

Conduktor Gateway is provided as a Docker image and Helm chart.

It should be deployed and managed in the best way for your organization and use case(s). This could be a single container, or more likely, multiple Gateway instances should be deployed and scaled to meet your needs. Optionally, the instances could be deployed behind a load balancer.

Use this quick start guide to help you get started.

Jump to:

Running the Gateway​

info

For a fully self-contained quick-start, see the Docker Compose.

In its simplest form, Gateway can be run from a simple Docker run command with the credentials to connect to your existing Kafka cluster.

Start a local Kafka​

Create a new directory (note the docker network will be derived from the directory name):

mkdir gateway-quick-start && cd gateway-quick-start

Run the below command to start a single node Kafka and ZooKeeper:

curl -L https://releases.conduktor.io/single-kafka -o docker-compose.yml && docker compose up -d

Start Conduktor Gateway​

Run the below command to start Conduktor Gateway and configure Docker networking between the two containers:

  docker run \
  --network gateway-quick-start_default \
  -e KAFKA_BOOTSTRAP_SERVERS=kafka1:29092 \
  -e GATEWAY_ADVERTISED_HOST=localhost \
  -e GATEWAY_PORT_START=9099 \
  -e GATEWAY_PORT_COUNT=1 \
  -p 9099:9099 \
  -d \
  conduktor/conduktor-gateway:3.0.5

By default, the Gateway uses port-based routing and listens on as many ports as there are Kafka brokers. In this case, we started a single-node Kafka cluster and opened 1 port.

At this stage you have:

  • Kafka running and its brokers available on localhost:9092
  • Gateway acting as a proxy to the backing Kafka cluster, accessible at loalhost:9099

Connecting your clients​

Your clients can now interact with Conduktor Gateway like any other Kafka cluster.

Example: creating a topic via Gateway using the Apache Kafka command line client:

bin/kafka-topics.sh --create --topic orders --bootstrap-server localhost:9099

Next Steps​

This quick start shows the basics, demonstrating Conduktor Gateway acting as a network proxy for Kafka. However, the real value comes with configuring interceptors, which are pluggable components that augment Kafka by intercepting specific requests of the Kafka protocol and applying operations to it.

View demos that demonstrate how interceptors are used to satisfy specific use cases such as encryption, data quality and safeguarding your cluster with technical and business rules.

Connecting to secured Kafka​

Your Kafka's bootstrap server, along with its authentication method should be configured using environment variables.

Conduktor Gateway connects to Kafka just like any other client.

Security configurations are provided using this scheme. For example:

ssl.truststore.location

becomes:

KAFKA_SSL_TRUSTSTORE_LOCATION

Confluent Cloud Example​

Below shows the most simple way to get started with Confluent Cloud.

info

By default, Gateway assumes you want the same security protocol between your clients and Gateway, as between your Gateway and Kafka.

However, this example uses DELEGATED_SASL_PLAINTEXT for the GATEWAY_SECURITY_PROTOCOL. For quick start purposes, this avoids needing to configure SSL certificates when connecting to Conduktor Gateway.

  docker run \
  -e KAFKA_BOOTSTRAP_SERVERS=$CONFLUENT_CLOUD_KAFKA_BOOTSTRAP_SERVER \
  -e KAFKA_SASL_MECHANISM=PLAIN \
  -e KAFKA_SECURITY_PROTOCOL=SASL_SSL \
  -e KAFKA_SASL_JAAS_CONFIG='org.apache.kafka.common.security.plain.PlainLoginModule required username="$CONFLUENT_CLOUD_API_KEY" password="$CONFLUENT_CLOUD_API_SECRET' \
  -e GATEWAY_SECURITY_PROTOCOL=DELEGATED_SASL_PLAINTEXT \
  -e GATEWAY_ADVERTISED_HOST=localhost \
  -e GATEWAY_CLUSTER_ID=test \
  -p 6969-6999:6969-6999 \
  -d \
  conduktor/conduktor-gateway:3.0.5

Note that if you wish to maintain the SSL/TLS connection between clients and Conduktor Gateway, see Client to Gateway Configuration.

By default, the Gateway uses port-based routing and listens on as many ports as there are Kafka brokers. In this case, we open 30 ports to account for Confluent Cloud clusters with many brokers. However, you may need to open more ports depending on the size of your cluster.

If you need support with your configuration, please contact us for support.

Docker Compose​

The below example demonstrates an environment consisting of:

cat docker-compose.yaml

1. Start the Docker environment​

Start all your docker services, wait for them to be up and ready, then run in background:

  • --wait: Wait for services to be running|healthy. Implies detached mode.
  • --detach: Detached mode: Run containers in the background
docker compose up --detach --wait

2. Create a topic via Conduktor Gateway​

kafka-topics \
    --bootstrap-server localhost:6969 \
    --replication-factor 1 \
    --partitions 1 \
    --create --if-not-exists \
    --topic orders

3. Produce a message to your topic​

echo '{"orderId":"12345","customerId":"67890","price":10000}' | \
    kafka-console-producer \
        --bootstrap-server localhost:6969 \
        --topic orders

4. Consume a message from your topic​

kafka-console-consumer \
    --bootstrap-server localhost:6969 \
    --topic cars \
    --from-beginning \
    --max-messages 1 \
    --timeout-ms 10000 | jq

5. Next Steps: Configure an interceptor​

This quick start shows the basics, demonstrating Conduktor Gateway can be interacted with like any other Kafka cluster.

However, the real value comes with configuring interceptors, which are pluggable components that augment Kafka by intercepting specific requests of the Kafka protocol and applying operations to it.

View demos that demonstrate how interceptors are used to satisfy specific use cases such as encryption, data quality and safeguarding your cluster with technical and business rules.