Skip to main content
Quick navigation

Infinite Partitions with Topic Concentration

Conduktor Gateway's topic concentration feature allows you to store multiple topics's data on a single underlying Kafka topic.

To clients, it appears that there are multiple topics and these can be read from as normal but in the underlying Kafka cluster there is a lot less resource required.

In this demo we are going to create a concentrated topic for powering several virtual topics.

Create the virtual topics, produce and consume data to them, and explore how this works.

View the full demo in realtime

You can either follow all the steps manually, or watch the recording

Review the docker compose environment

As can be seen from docker-compose.yaml the demo environment consists of the following services:

  • gateway1
  • gateway2
  • kafka-client
  • kafka1
  • kafka2
  • kafka3
  • schema-registry
  • zookeeper
cat docker-compose.yaml

Starting the docker environment

Start all your docker processes, wait for them to be up and ready, then run in background

  • --wait: Wait for services to be running|healthy. Implies detached mode.
  • --detach: Detached mode: Run containers in the background
docker compose up --detach --wait

Creating virtual cluster teamA

Creating virtual cluster teamA on gateway gateway1 and reviewing the configuration file to access it

# Generate virtual cluster teamA with service account sa
token=$(curl \
--request POST "http://localhost:8888/admin/vclusters/v1/vcluster/teamA/username/sa" \
--header 'Content-Type: application/json' \
--user 'admin:conduktor' \
--silent \
--data-raw '{"lifeTimeSeconds": 7776000}' | jq -r ".token")

# Create access file
echo """
bootstrap.servers=localhost:6969
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='sa' password='$token';
""" > teamA-sa.properties

# Review file
cat teamA-sa.properties

Create the topic that will hold virtual topics

Creating on kafka1:

  • Topic hold-many-concentrated-topics with partitions:5 and replication-factor:1
kafka-topics \
--bootstrap-server localhost:19092,localhost:19093,localhost:19094 \
--replication-factor 1 \
--partitions 5 \
--create --if-not-exists \
--topic hold-many-concentrated-topics

Let's tell gateway1 that topics matching the pattern concentrated-.* need to be concentrated into the underlying hold-many-concentrated-topics physical topic.

[!NOTE] You don���t need to create the physical topic that backs the concentrated topics, it will automatically be created when a client topic starts using the concentrated topic.

{
"concentrated" : true,
"readOnly" : false,
"physicalTopicName" : "hold-many-concentrated-topics"
}
curl \
--request POST 'http://localhost:8888/admin/vclusters/v1/vcluster/teamA/topics/concentrated-.%2A' \
--header 'Content-Type: application/json' \
--user 'admin:conduktor' \
--silent \
--data "@step-07-mapping.json" | jq

Let's create multiple topics with many partitions

Creating on teamA:

  • Topic concentrated-topic-with-10-partitions with partitions:10 and replication-factor:1
  • Topic concentrated-topic-with-20-partitions with partitions:20 and replication-factor:1
  • Topic concentrated-topic-with-50-partitions with partitions:50 and replication-factor:1
  • Topic concentrated-topic-with-100-partitions with partitions:100 and replication-factor:1
kafka-topics \
--bootstrap-server localhost:6969 \
--command-config teamA-sa.properties \
--replication-factor 1 \
--partitions 10 \
--create --if-not-exists \
--topic concentrated-topic-with-10-partitions
kafka-topics \
--bootstrap-server localhost:6969 \
--command-config teamA-sa.properties \
--replication-factor 1 \
--partitions 20 \
--create --if-not-exists \
--topic concentrated-topic-with-20-partitions
kafka-topics \
--bootstrap-server localhost:6969 \
--command-config teamA-sa.properties \
--replication-factor 1 \
--partitions 50 \
--create --if-not-exists \
--topic concentrated-topic-with-50-partitions
kafka-topics \
--bootstrap-server localhost:6969 \
--command-config teamA-sa.properties \
--replication-factor 1 \
--partitions 100 \
--create --if-not-exists \
--topic concentrated-topic-with-100-partitions

Assert they exist in teamA cluster

kafka-topics \
--bootstrap-server localhost:6969 \
--command-config teamA-sa.properties \
--list

Producing 1 message in concentrated-topic-with-10-partitions

Producing 1 message in concentrated-topic-with-10-partitions in cluster teamA

Sending 1 event

{
"message" : "10 partitions"
}

with

echo '{"message": "10 partitions"}' | \
kafka-console-producer \
--bootstrap-server localhost:6969 \
--producer.config teamA-sa.properties \
--topic concentrated-topic-with-10-partitions

Producing 1 message in concentrated-topic-with-20-partitions

Producing 1 message in concentrated-topic-with-20-partitions in cluster teamA

Sending 1 event

{
"message" : "20 partitions"
}

with

echo '{"message": "20 partitions"}' | \
kafka-console-producer \
--bootstrap-server localhost:6969 \
--producer.config teamA-sa.properties \
--topic concentrated-topic-with-20-partitions

Producing 1 message in concentrated-topic-with-50-partitions

Producing 1 message in concentrated-topic-with-50-partitions in cluster teamA

Sending 1 event

{
"message" : "50 partitions"
}

with

echo '{"message": "50 partitions"}' | \
kafka-console-producer \
--bootstrap-server localhost:6969 \
--producer.config teamA-sa.properties \
--topic concentrated-topic-with-50-partitions

Producing 1 message in concentrated-topic-with-100-partitions

Producing 1 message in concentrated-topic-with-100-partitions in cluster teamA

Sending 1 event

{
"message" : "100 partitions"
}

with

echo '{"message": "100 partitions"}' | \
kafka-console-producer \
--bootstrap-server localhost:6969 \
--producer.config teamA-sa.properties \
--topic concentrated-topic-with-100-partitions

Consuming from concentrated-topic-with-10-partitions

Consuming from concentrated-topic-with-10-partitions in cluster teamA

kafka-console-consumer \
--bootstrap-server localhost:6969 \
--consumer.config teamA-sa.properties \
--topic concentrated-topic-with-10-partitions \
--from-beginning \
--timeout-ms 10000 | jq

returns

Processed a total of 1 messages
{
"message": "10 partitions"
}

Consuming from concentrated-topic-with-20-partitions

Consuming from concentrated-topic-with-20-partitions in cluster teamA

kafka-console-consumer \
--bootstrap-server localhost:6969 \
--consumer.config teamA-sa.properties \
--topic concentrated-topic-with-20-partitions \
--from-beginning \
--timeout-ms 10000 | jq

returns

Processed a total of 1 messages
{
"message": "20 partitions"
}

Consuming from concentrated-topic-with-50-partitions

Consuming from concentrated-topic-with-50-partitions in cluster teamA

kafka-console-consumer \
--bootstrap-server localhost:6969 \
--consumer.config teamA-sa.properties \
--topic concentrated-topic-with-50-partitions \
--from-beginning \
--timeout-ms 10000 | jq

returns

Processed a total of 1 messages
{
"message": "50 partitions"
}

Consuming from concentrated-topic-with-100-partitions

Consuming from concentrated-topic-with-100-partitions in cluster teamA

kafka-console-consumer \
--bootstrap-server localhost:6969 \
--consumer.config teamA-sa.properties \
--topic concentrated-topic-with-100-partitions \
--from-beginning \
--timeout-ms 10000 | jq

returns

Processed a total of 1 messages
{
"message": "100 partitions"
}

Revealing the magic

Revealing the magic in cluster kafka1

kafka-console-consumer \
--bootstrap-server localhost:19092,localhost:19093,localhost:19094 \
--topic hold-many-concentrated-topics \
--from-beginning \
--timeout-ms 10000 \
--property print.headers=true

returns

PDK_originalPartition:5,PDK_originalTopic:concentrated-topic-with-10-partitions {"message": "10 partitions"}
PDK_originalPartition:11,PDK_originalTopic:concentrated-topic-with-20-partitions {"message": "20 partitions"}
PDK_originalPartition:10,PDK_originalTopic:concentrated-topic-with-50-partitions {"message": "50 partitions"}
PDK_originalPartition:93,PDK_originalTopic:concentrated-topic-with-100-partitions {"message": "100 partitions"}
Processed a total of 4 messages

Tearing down the docker environment

Remove all your docker processes and associated volumes

  • --volumes: Remove named volumes declared in the "volumes" section of the Compose file and anonymous volumes attached to containers.
docker compose down --volumes

Conclusion

Infinite partitions with topic concentration is really a game changer!