Client metrics is a preview feature, available from Console 1.49.0. It’s turned off by default and isn’t available on the free plan.
- Find misbehaving clients: spot producers that retry silently or send one record per request, unstable consumer groups and clients that open connections constantly.
- Attribute traffic: see which service account, application and client library is behind each client instance.
- Check ownership: find clients that no Self-service application declares, and topics an application uses without declaring them.
- Compare with peers: see where a service account stands against the rest of the cluster.
- Tune clients: get suggested values for settings such as
linger.ms,batch.sizeandbuffer.memory, inferred from what the clients do.
How it works
- Gateway, or your Kafka brokers, tell each client which metrics to send and how often.
- Clients push their metrics. Gateway or the brokers attach the client’s identity (principal and client instance ID) and forward the metrics to an OpenTelemetry (OTel) Collector.
- The collector serves the metrics on a Prometheus endpoint.
- Console scrapes that endpoint every 60 seconds by default, groups the values by service account and runs its detection rules on them.
Prerequisites
Set up client metrics
1
Send client metrics to a collector
Clients only push metrics when Gateway or the brokers ask for them. Choose the path that matches your deployment.
- With Gateway
- Without Gateway
Deploy the client metrics Interceptor on Gateway and point its OTLP exporter at your collector. Gateway handles KIP-714 itself, so your brokers don’t need any setup.Set
pushIntervalMs to Console’s scrape interval, 60000 by default:2
Expose the metrics to Console
Configure the collector’s Console reads the metrics at
prometheus exporter so Console can attribute each value to a client:http://<YOUR_COLLECTOR_HOST>:8889/metrics.3
Turn on client metrics in Console
Set The other
client_metrics.enabled to true in the Console configuration and restart Console:- YAML
- Environment variable
client_metrics properties control the scrape interval, the window size and memory limits. See Client metrics properties.4
Declare the collectors on each cluster
Add the collectors that serve each cluster’s metrics. A cluster can list one collector, or one per broker: Console reads them all and adds up the clients. Declare collectors in the Console UI, in the Console configuration, with environment variables, with the Conduktor CLI or with the API.
- Console UI
- YAML
- Environment variables
- Conduktor CLI
- Go to Settings > Clusters and select the cluster.
- Open the Client metrics tab. It only shows once client metrics is turned on.
- Click Add a collector and enter the collector’s URL, for example
http://<YOUR_COLLECTOR_HOST>:8889/metrics. - If the collector requires it, pick a bearer token or basic authentication under Authentication.
- Click Save.
When the cluster has a Gateway provider, Console only keeps the metrics of the provider’s Virtual Cluster, so several Virtual Clusters of one Gateway can share a collector. Console also keeps the metrics sent by your brokers, which don’t belong to any Virtual Cluster.
5
Check that data arrives
In Console, select the cluster and go to Service accounts. Until the first metrics arrive, the page shows Waiting for the first client push. Clients ask for their subscription on their own schedule, which can take up to five minutes. Once data arrives, the client health panel appears above the list of service accounts.
View client health
Select a cluster and go to Service accounts. The client health panel at the top of the page summarizes every service account whose clients push metrics:- Fleet health: a score out of 100%, with a level from Excellent to Critical. Each finding lowers its service account’s health: a critical finding the most, an info finding the least. One service account can’t lower the score by more than its own share.
- Service accounts without ACLs: the number of connected service accounts that no ACL mentions. They’re superusers, anonymous, or let in because no ACL matched, so no ACL restricts what they do.
- Categories: the share of service accounts with at least one finding in each category, and the number of recommendations.
- Reliability risk: applications losing throughput to Kafka or to their own code.
- Efficiency and waste: applications that work, but use more cluster capacity than they need to.
- Governance coverage: applications nobody owns, that connect without authentication or that run old client libraries.
- Findings: every finding on the cluster, most severe first.
Investigate a service account
Click a service account to open its details. Next to the existing tabs, client metrics adds four tabs. The page opens on the tab that holds the most severe item.
Console can’t read a client’s configuration. Suggested settings are inferred from what the clients do, and only appear when the related problem is detected.
Findings reference
Thresholds are conservative: a healthy cluster shows no findings at all.
Permissions
Limitations
- Supported clients: only clients that support KIP-714 push metrics. librdkafka-based clients send fewer metrics than the Java client, so some findings, such as chatty producer, can’t be detected for them. The stale library check only applies to the Java client.
- History: Console keeps a sliding window only, 30 minutes with the default settings.
- Leader changes: Console scrapes collectors from a single instance. After that instance changes, one interval of counter values is lost.
- Failed scrapes: if any collector of a cluster fails, Console skips the whole scrape for that cluster rather than storing a partial view.
- Limits: by default, Console tracks up to 500 service accounts per cluster, keeps the 20 largest series (for example, topics) per metric and service account, and tracks up to one million counters per cluster. See Client metrics properties to change them.
Troubleshoot
Console still shows Waiting for the first client push
Symptom: after more than five minutes, the Service accounts page still shows Waiting for the first client push. Cause: the clients don’t push metrics, or Console can’t attribute them. Fix:- Check that the clients support KIP-714: Java client 3.7 or later, or librdkafka 2.5 or later. Check also that
enable.metrics.pushisn’t set tofalse. - Without Gateway, check that a client metrics subscription exists:
kafka-client-metrics.sh --bootstrap-server <YOUR_BOOTSTRAP_SERVERS> --describe. - Open the collector endpoint,
http://<YOUR_COLLECTOR_HOST>:8889/metrics, and check that series starting withorg_apache_kafka_carryprincipalandclient_instance_idlabels. If they don’t, enableresource_to_telemetry_conversionon theprometheusexporter.
Values jump and drop to zero
Symptom: charts alternate between a spike and several zero values, or clients appear and disappear. Cause: clients push less often than Console scrapes. Fix: set the push interval to Console’s scrape interval:pushIntervalMs on the Gateway Interceptor, or --interval on the broker subscription.