Skip to main content
Quick navigation

How about the latency impact?

View the full demo in realtime

You can either follow all the steps manually, or watch the recording

Review the docker compose environment

As can be seen from docker-compose.yaml the demo environment consists of the following services:

  • gateway1
  • gateway2
  • kafka-client
  • kafka1
  • kafka2
  • kafka3
  • schema-registry
  • zookeeper
cat docker-compose.yaml

Starting the docker environment

Start all your docker processes, wait for them to be up and ready, then run in background

  • --wait: Wait for services to be running|healthy. Implies detached mode.
  • --detach: Detached mode: Run containers in the background
docker compose up --detach --wait

Creating topic physical-kafka on kafka1

Creating on kafka1:

  • Topic physical-kafka with partitions:10 and replication-factor:1
kafka-topics \
--bootstrap-server localhost:19092,localhost:19093,localhost:19094 \
--replication-factor 1 \
--partitions 10 \
--create --if-not-exists \
--topic physical-kafka

Let's use EndToEndLatency that comes bundled with Kafka

  • arg1: broker_list
  • arg2: topic
  • arg3: num_messages
  • arg4: producer_acks
  • arg5: message_size_bytes
kafka-run-class kafka.tools.EndToEndLatency \
localhost:19092,localhost:19093,localhost:19094 \
physical-kafka 10000 all 255

Creating virtual cluster teamA

Creating virtual cluster teamA on gateway gateway1 and reviewing the configuration file to access it

# Generate virtual cluster teamA with service account sa
token=$(curl \
--request POST "http://localhost:8888/admin/vclusters/v1/vcluster/teamA/username/sa" \
--header 'Content-Type: application/json' \
--user 'admin:conduktor' \
--silent \
--data-raw '{"lifeTimeSeconds": 7776000}' | jq -r ".token")

# Create access file
echo """
bootstrap.servers=localhost:6969
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='sa' password='$token';
""" > teamA-sa.properties

# Review file
cat teamA-sa.properties

Creating topic via-gateway on teamA

Creating on teamA:

  • Topic via-gateway with partitions:10 and replication-factor:1
kafka-topics \
--bootstrap-server localhost:6969 \
--command-config teamA-sa.properties \
--replication-factor 1 \
--partitions 10 \
--create --if-not-exists \
--topic via-gateway

Let's use kafka-producer-perf-test that comes bundled with Kafka

arg1: broker_list arg2: topic arg3: num_messages arg4: producer_acks arg5: message_size_bytes arg6: property file

kafka-run-class kafka.tools.EndToEndLatency \
localhost:6969 \
via-gateway 10000 all 255 \
teamA-sa.properties

Tearing down the docker environment

Remove all your docker processes and associated volumes

  • --volumes: Remove named volumes declared in the "volumes" section of the Compose file and anonymous volumes attached to containers.
docker compose down --volumes

Conclusion

Gateway's end to end latency is enough for all use cases!