From our blog: No Kafka data platform without ownership Why federated ownership underpins governance and policy enforcement at scale.
What you can control
Traffic Control Policies work as that validate and enforce rules on: Configuration policies- Broker and topic configuration changes
- Topic creation parameters (replication factor, partition count)
- Schema ID requirements
- Producer throughput limits
- Consumer fetch rate limits
- Offset commit frequency
- Consumer group join frequency
- Connection attempt limits
- Read-only topics
- Required client IDs
- Consumer group restrictions
- Message header removal
Benefits
- Prevent misconfiguration: stop teams from creating topics with dangerous settings
- Protect cluster performance: rate limit clients before they impact others
- Enforce standards: require schema IDs, client IDs, or other organizational policies
- Gradual rollout: apply policies per or
Available policies
- Alter broker config
- Alter topic config
- ClientId required
- Consumer group policy
- Create topic policy
- Fetch policy
- Limit commit offset
- Limit connection attempts
- Limit join group
- Limit producer rate
- Produce requests
- Read-only topic
- Remove message headers
- Schema ID required
How limits apply
- A limit is shared by every client the Interceptor targets. A producer rate limit applied to a Virtual Cluster caps the combined throughput of all its clients. To limit one service account, target the Interceptor at that service account: all clients using it share the limit.
- Commit offset and join group limits apply per consumer group.
- Each Gateway node enforces its own limits. All nodes apply the same Interceptor configuration, but each node counts traffic separately. When the clients of a service account or Virtual Cluster connect to three Gateway nodes, their combined traffic can reach up to three times the configured limit.
- Gateway doesn’t limit per client IP address. The limit connection attempts policy caps how many connections per second can start sending requests, and a connection idle for more than five seconds counts as new again. It doesn’t cap the number of open connections. To cap the total number of open connections on a Gateway node, set
GATEWAY_NETWORK_MAX_CONNECTIONS.