Skip to main content
Kafka Improvement Proposal 848 (KIP-848) introduces a new consumer group protocol, which consumers select with group.protocol=consumer. The broker computes the partition assignment instead of a client leader, and members heartbeat with ConsumerGroupHeartbeat instead of JoinGroup and SyncGroup. Find out more in KIP-848. Because the broker computes the assignment, it can only assign the physical topics it knows. Gateway translates the protocol so that consumers can also read Gateway topics Kafka doesn’t know about: topics in a Virtual Cluster, aliases and topic views. The protocol is off by default. While it’s off, Gateway doesn’t advertise ConsumerGroupHeartbeat and ConsumerGroupDescribe, so a consumer with group.protocol=consumer fails with UnsupportedVersionException. Use group.protocol=classic instead.

Prerequisites

Enable the protocol

1

Set the feature flag

Set this environment variable on every Gateway node, then restart them:
After you restart the nodes, each classic consumer group that reads a topic in a Virtual Cluster, an alias or a topic view rebalances once, even if none of its consumers switches protocol. Gateway then stores the group’s assignment with physical topic names, so that Kafka keeps track of which partitions each member owns if the group later converts to the new protocol.
2

Configure your consumers

Once every node has restarted, set group.protocol=consumer on the consumers that should use the new protocol. A consumer that reaches a node with the flag off fails with UnsupportedVersionException.

Check that it works

Start a consumer with group.protocol=consumer through Gateway. If it fails with UnsupportedVersionException, check that GATEWAY_FEATURE_FLAGS_CONSUMER_GROUP_PROTOCOL is set to true on every Gateway node and that topic ID support isn’t turned off. Then describe its group through Gateway:
The members and their assigned partitions are listed with the topic names the consumer subscribed to, not the physical topic names.

What Gateway handles

Gateway topics

Gateway sends Kafka the physical topics, then hands each consumer its assignment in the topic names it subscribed to.

Topic views

Topic views apply the same group.id convention to both protocols. Gateway rejects a consumer whose group.id doesn’t follow it with INVALID_GROUP_ID.

Regular expression subscriptions

A regular expression subscription (subscribe(SubscriptionPattern) in the Java client) matches the topic names the consumer sees through Gateway. It skips, without an error, topic views the group.id convention keeps the consumer out of.

Gateway ACLs

When Gateway ACLs are enabled, they apply to the new protocol as they do on Kafka:

Groups that mix both protocols

Classic consumers and consumers using the new protocol can share a group, so you can migrate a group one consumer at a time.

Gateway restarts

When a member using the new protocol reaches a Gateway node that doesn’t know it, for example after a restart, the node asks it to send its full subscription again. The member keeps its partitions, without a rebalance.

Describe consumer groups

With topic ID support enabled, admin clients describe a group with ConsumerGroupDescribe first and fall back to DescribeGroups for a classic group. Gateway reports a group using the new protocol with each member’s subscription and assignment in the topic names it reads.

Limitations

  • Consumer group policies: the consumer group policy and limit join group policy Interceptors support classic consumers only.
  • Concentrated topics: Gateway rejects a subscription to a concentrated topic with INVALID_REQUEST. Use group.protocol=classic for consumers that read concentrated topics.
  • Alias changes: don’t change the topic an alias points to while a group using the new protocol reads it, or its members can read the same records twice while they switch. To move an alias, delete the AliasTopic resource, wait until the groups that read it report an empty assignment, then create it again with the new physical topic.
  • Failover: consumers using the new protocol can fail after a failover to another cluster. Use group.protocol=classic for consumers that have to survive a failover.
  • Describe on another node: a Gateway node describing a group whose members are connected to other nodes lists each member with every topic backed by the same physical topic, such as an alias and its physical topic. With Gateway ACLs, the group is refused unless the caller can describe all of those topics.

Disable the protocol

1

Switch your consumers back

Set group.protocol=classic on every consumer that uses the new protocol.
2

Turn off the flag

Set GATEWAY_FEATURE_FLAGS_CONSUMER_GROUP_PROTOCOL=false on every Gateway node, then restart them.