Unlimited Topics
Why
There are situations that call for the creation of a number of topics for logical rather than technical reasons (e.g. separation of business units), this may result in considerable overuse of Kafka resources in such cases. Topic Concentration allows the data from a set of topics to be represented on a single underlying topic.
Clients connecting through Conduktor Gateway can use concentrated topics as they would any other topic and require no extra configuration to use this feature.
For example, let's say we have the following topics:
- us_east_orders - 100 partitions
- us_west_orders - 100 partitions
- emea_orders - 100 partitions
- latam_orders - 100 partitions
The total Kafka resource requirement is 400 partitions. With topic concentration all of these topics can be concentrated to a single topic:
- concentrated_orders - 100 partitions
Using 1/4 of the Kafka resources.
Configuration
Topic concentration is configured via Conduktor Gateway Enterprise Edition's admin HTTP API.
Property Name | Type | Description |
---|---|---|
logicalTopicRegex | string | A regular expression to match all topic names that will be concentrated, specified as a URL parameter |
topicName | string | The name of the underlying topic to concentrate to |
For an example configuration please see the Conduktor Gateway Demos