group.protocol=consumer. The broker computes the partition assignment instead of a client leader, and members heartbeat with ConsumerGroupHeartbeat instead of JoinGroup and SyncGroup. Find out more in KIP-848.
Because the broker computes the assignment, it can only assign the physical topics it knows. Gateway translates the protocol so that consumers can also read Gateway topics Kafka doesn’t know about: topics in a Virtual Cluster, aliases and topic views.
The protocol is off by default. While it’s off, Gateway doesn’t advertise ConsumerGroupHeartbeat and ConsumerGroupDescribe, so a consumer with group.protocol=consumer fails with UnsupportedVersionException. Use group.protocol=classic instead.
Prerequisites
Enable the protocol
1
Set the feature flag
Set this environment variable on every Gateway node, then restart them:
After you restart the nodes, each classic consumer group that reads a topic in a Virtual Cluster, an alias or a topic view rebalances once, even if none of its consumers switches protocol. Gateway then stores the group’s assignment with physical topic names, so that Kafka keeps track of which partitions each member owns if the group later converts to the new protocol.
2
Configure your consumers
Once every node has restarted, set
group.protocol=consumer on the consumers that should use the new protocol. A consumer that reaches a node with the flag off fails with UnsupportedVersionException.Check that it works
Start a consumer withgroup.protocol=consumer through Gateway. If it fails with UnsupportedVersionException, check that GATEWAY_FEATURE_FLAGS_CONSUMER_GROUP_PROTOCOL is set to true on every Gateway node and that topic ID support isn’t turned off.
Then describe its group through Gateway:
What Gateway handles
Gateway topics
Gateway sends Kafka the physical topics, then hands each consumer its assignment in the topic names it subscribed to.Topic views
Topic views apply the samegroup.id convention to both protocols. Gateway rejects a consumer whose group.id doesn’t follow it with INVALID_GROUP_ID.
Regular expression subscriptions
A regular expression subscription (subscribe(SubscriptionPattern) in the Java client) matches the topic names the consumer sees through Gateway. It skips, without an error, topic views the group.id convention keeps the consumer out of.
Gateway ACLs
When Gateway ACLs are enabled, they apply to the new protocol as they do on Kafka:Groups that mix both protocols
Classic consumers and consumers using the new protocol can share a group, so you can migrate a group one consumer at a time.Gateway restarts
When a member using the new protocol reaches a Gateway node that doesn’t know it, for example after a restart, the node asks it to send its full subscription again. The member keeps its partitions, without a rebalance.Describe consumer groups
With topic ID support enabled, admin clients describe a group withConsumerGroupDescribe first and fall back to DescribeGroups for a classic group. Gateway reports a group using the new protocol with each member’s subscription and assignment in the topic names it reads.
Limitations
- Consumer group policies: the consumer group policy and limit join group policy Interceptors support classic consumers only.
- Concentrated topics: Gateway rejects a subscription to a concentrated topic with
INVALID_REQUEST. Usegroup.protocol=classicfor consumers that read concentrated topics. - Alias changes: don’t change the topic an alias points to while a group using the new protocol reads it, or its members can read the same records twice while they switch. To move an alias, delete the AliasTopic resource, wait until the groups that read it report an empty assignment, then create it again with the new physical topic.
- Failover: consumers using the new protocol can fail after a failover to another cluster. Use
group.protocol=classicfor consumers that have to survive a failover. - Describe on another node: a Gateway node describing a group whose members are connected to other nodes lists each member with every topic backed by the same physical topic, such as an alias and its physical topic. With Gateway ACLs, the group is refused unless the caller can describe all of those topics.
Disable the protocol
1
Switch your consumers back
Set
group.protocol=classic on every consumer that uses the new protocol.2
Turn off the flag
Set
GATEWAY_FEATURE_FLAGS_CONSUMER_GROUP_PROTOCOL=false on every Gateway node, then restart them.