> ## Documentation Index
> Fetch the complete documentation index at: https://docs.conduktor.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Use the new consumer group protocol

> Let Kafka consumers use the KIP-848 consumer group protocol through Conduktor Gateway, including on Virtual Clusters, aliases and topic views.

[Preview](https://docs.conduktor.io/guide/reference/feature-labels.md#preview) (On its way to general availability (GA). Try it now. It may change before GA and isn't recommended for production yet.)

Kafka Improvement Proposal 848 (KIP-848) introduces a new consumer group protocol, which consumers select with `group.protocol=consumer`. The broker computes the partition assignment instead of a client leader, and members heartbeat with `ConsumerGroupHeartbeat` instead of `JoinGroup` and `SyncGroup`. Find out more in [KIP-848](https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol).

Because the broker computes the assignment, it can only assign the physical topics it knows. Gateway translates the protocol so that consumers can also read Gateway topics Kafka doesn't know about: topics in a [Virtual Cluster](https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/virtual-clusters.md), aliases and [topic views](https://docs.conduktor.io/guide/conduktor-concepts/logical-topics.md#topic-views).

The protocol is off by default. While it's off, Gateway doesn't advertise `ConsumerGroupHeartbeat` and `ConsumerGroupDescribe`, so a consumer with `group.protocol=consumer` fails with `UnsupportedVersionException`. Use `group.protocol=classic` instead.

## Prerequisites

| Component | Details |
| - | - |
| **Gateway** | Version 3.23.0 or later, on every node. |
| **Topic ID support** | On by default. Don't set `GATEWAY_FEATURE_FLAGS_TOPIC_ID_SUPPORT` to `false`: Gateway needs topic IDs to translate the protocol. [Find out more about topic ID support](https://docs.conduktor.io/guide/reference/kafka-protocol-support.md#topic-id-support). |
| **Kafka cluster** | Apache Kafka 4.0 or later, with the consumer group protocol available on the brokers. |
| **Kafka clients** | Clients that support `group.protocol=consumer`, such as the Apache Kafka Java client 4.0 or later. |

## Enable the protocol

**1. Set the feature flag**

Set this environment variable on every Gateway node, then restart them:

```bash theme={null}
GATEWAY_FEATURE_FLAGS_CONSUMER_GROUP_PROTOCOL=true
```

> [!NOTE]
>
> After you restart the nodes, each classic consumer group that reads a topic in a Virtual Cluster, an alias or a topic view rebalances once, even if none of its consumers switches protocol. Gateway then stores the group's assignment with physical topic names, so that Kafka keeps track of which partitions each member owns if the group later converts to the new protocol.

**2. Configure your consumers**

Once every node has restarted, set `group.protocol=consumer` on the consumers that should use the new protocol. A consumer that reaches a node with the flag off fails with `UnsupportedVersionException`.

```properties theme={null}
group.protocol=consumer
```

### Check that it works

Start a consumer with `group.protocol=consumer` through Gateway. If it fails with `UnsupportedVersionException`, check that `GATEWAY_FEATURE_FLAGS_CONSUMER_GROUP_PROTOCOL` is set to `true` on every Gateway node and that topic ID support isn't turned off.

Then describe its group through Gateway:

```bash theme={null}
kafka-consumer-groups --bootstrap-server <GATEWAY_BOOTSTRAP> \
  --command-config <CLIENT_PROPERTIES> \
  --describe --group <GROUP_ID> --members
```

The members and their assigned partitions are listed with the topic names the consumer subscribed to, not the physical topic names.

## What Gateway handles

### Gateway topics

Gateway sends Kafka the physical topics, then hands each consumer its assignment in the topic names it subscribed to.

### Topic views

[Topic views](https://docs.conduktor.io/guide/conduktor-concepts/logical-topics.md#set-the-consumer-group-id) apply the same `group.id` convention to both protocols. Gateway rejects a consumer whose `group.id` doesn't follow it with `INVALID_GROUP_ID`.

### Regular expression subscriptions

A regular expression subscription (`subscribe(SubscriptionPattern)` in the Java client) matches the topic names the consumer sees through Gateway. It skips, without an error, topic views the `group.id` convention keeps the consumer out of.

### Gateway ACLs

When [Gateway ACLs](https://docs.conduktor.io/guide/conduktor-concepts/gateway-authentication-authorization.md) are enabled, they apply to the new protocol as they do on Kafka:

| Request | Required permissions |
| - | - |
| `ConsumerGroupHeartbeat` | `READ` on the group, and `DESCRIBE` on every subscribed topic |
| `ConsumerGroupDescribe` | `DESCRIBE` on the group, and on every topic assigned to its members |

### Groups that mix both protocols

Classic consumers and consumers using the new protocol can share a group, so you can migrate a group one consumer at a time.

### Gateway restarts

When a member using the new protocol reaches a Gateway node that doesn't know it, for example after a restart, the node asks it to send its full subscription again. The member keeps its partitions, without a rebalance.

### Describe consumer groups

With topic ID support enabled, admin clients describe a group with `ConsumerGroupDescribe` first and fall back to `DescribeGroups` for a classic group. Gateway reports a group using the new protocol with each member's subscription and assignment in the topic names it reads.

## Limitations

* **Consumer group policies**: the [consumer group policy](https://docs.conduktor.io/guide/use-cases/apply-traffic-control-policies.md#consumer-group-policy) and [limit join group policy](https://docs.conduktor.io/guide/use-cases/apply-traffic-control-policies.md#limit-join-group-policy) Interceptors support classic consumers only.
* **Concentrated topics**: Gateway rejects a subscription to a [concentrated topic](https://docs.conduktor.io/guide/conduktor-concepts/logical-topics.md#concentrated-topics) with `INVALID_REQUEST`. Use `group.protocol=classic` for consumers that read concentrated topics.
* **Alias changes**: don't change the topic an alias points to while a group using the new protocol reads it, or its members can read the same records twice while they switch. To move an alias, delete the AliasTopic resource, wait until the groups that read it report an empty assignment, then create it again with the new physical topic.
* **Failover**: consumers using the new protocol can fail after a [failover](https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/failover.md) to another cluster. Use `group.protocol=classic` for consumers that have to survive a failover.
* **Describe on another node**: a Gateway node describing a group whose members are connected to other nodes lists each member with every topic backed by the same physical topic, such as an alias and its physical topic. With Gateway ACLs, the group is refused unless the caller can describe all of those topics.

## Disable the protocol

**1. Switch your consumers back**

Set `group.protocol=classic` on every consumer that uses the new protocol.

**2. Turn off the flag**

Set `GATEWAY_FEATURE_FLAGS_CONSUMER_GROUP_PROTOCOL=false` on every Gateway node, then restart them.

## Related resources

* [Kafka protocol support](https://docs.conduktor.io/guide/reference/kafka-protocol-support.md)
* [Gateway environment variables](https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-gateway/environment-variables.md#feature-flags)
* [Logical topics](https://docs.conduktor.io/guide/conduktor-concepts/logical-topics.md)
* [Feature labels](https://docs.conduktor.io/guide/reference/feature-labels.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.