Skip to main content
Client metrics is a preview feature, available from Console 1.49.0. It’s turned off by default and isn’t available on the free plan.
Kafka clients from version 3.7 onward push their own metrics to the cluster, as defined in Kafka Improvement Proposal 714 (KIP-714) . Console reads these metrics, groups them by service account and tells you which applications need attention and what to change. Use client metrics to:
  • Find misbehaving clients: spot producers that retry silently or send one record per request, unstable consumer groups and clients that open connections constantly.
  • Attribute traffic: see which service account, application and client library is behind each client instance.
  • Check ownership: find clients that no Self-service application declares, and topics an application uses without declaring them.
  • Compare with peers: see where a service account stands against the rest of the cluster.
  • Tune clients: get suggested values for settings such as linger.ms, batch.size and buffer.memory, inferred from what the clients do.

How it works

  1. Gateway, or your Kafka brokers, tell each client which metrics to send and how often.
  2. Clients push their metrics. Gateway or the brokers attach the client’s identity (principal and client instance ID) and forward the metrics to an OpenTelemetry (OTel) Collector.
  3. The collector serves the metrics on a Prometheus endpoint.
  4. Console scrapes that endpoint every 60 seconds by default, groups the values by service account and runs its detection rules on them.
Console keeps a short sliding window of values per service account: 30 points, or 30 minutes with the default settings. It doesn’t keep long-term history. To keep history, also export the metrics from your collector to your own observability stack.

Prerequisites

Set up client metrics

1

Send client metrics to a collector

Clients only push metrics when Gateway or the brokers ask for them. Choose the path that matches your deployment.
Deploy the client metrics Interceptor on Gateway and point its OTLP exporter at your collector. Gateway handles KIP-714 itself, so your brokers don’t need any setup.Set pushIntervalMs to Console’s scrape interval, 60000 by default:
Kafka and Gateway default to a push every five minutes. Leave it there and Console sees each client only once every five minutes: series expire from the collector between pushes and counters show as a spike followed by zeros. Set the push interval to Console’s scrape interval.
2

Expose the metrics to Console

Configure the collector’s prometheus exporter so Console can attribute each value to a client:
Console reads the metrics at http://<YOUR_COLLECTOR_HOST>:8889/metrics.
3

Turn on client metrics in Console

Set client_metrics.enabled to true in the Console configuration and restart Console:
The other client_metrics properties control the scrape interval, the window size and memory limits. See Client metrics properties.
4

Declare the collectors on each cluster

Add the collectors that serve each cluster’s metrics. A cluster can list one collector, or one per broker: Console reads them all and adds up the clients. Declare collectors in the Console UI, in the Console configuration, with environment variables, with the Conduktor CLI or with the API.
  1. Go to Settings > Clusters and select the cluster.
  2. Open the Client metrics tab. It only shows once client metrics is turned on.
  3. Click Add a collector and enter the collector’s URL, for example http://<YOUR_COLLECTOR_HOST>:8889/metrics.
  4. If the collector requires it, pick a bearer token or basic authentication under Authentication.
  5. Click Save.
When the cluster has a Gateway provider, Console only keeps the metrics of the provider’s Virtual Cluster, so several Virtual Clusters of one Gateway can share a collector. Console also keeps the metrics sent by your brokers, which don’t belong to any Virtual Cluster.
5

Check that data arrives

In Console, select the cluster and go to Service accounts. Until the first metrics arrive, the page shows Waiting for the first client push. Clients ask for their subscription on their own schedule, which can take up to five minutes. Once data arrives, the client health panel appears above the list of service accounts.

View client health

Select a cluster and go to Service accounts. The client health panel at the top of the page summarizes every service account whose clients push metrics:
  • Fleet health: a score out of 100%, with a level from Excellent to Critical. Each finding lowers its service account’s health: a critical finding the most, an info finding the least. One service account can’t lower the score by more than its own share.
  • Service accounts without ACLs: the number of connected service accounts that no ACL mentions. They’re superusers, anonymous, or let in because no ACL matched, so no ACL restricts what they do.
  • Categories: the share of service accounts with at least one finding in each category, and the number of recommendations.
    • Reliability risk: applications losing throughput to Kafka or to their own code.
    • Efficiency and waste: applications that work, but use more cluster capacity than they need to.
    • Governance coverage: applications nobody owns, that connect without authentication or that run old client libraries.
  • Findings: every finding on the cluster, most severe first.
The list of service accounts below the panel also shows, for each account, its Self-service Application, the number of Clients connected, the client Library, Records sent /s, consumer Lag and a Status that reflects its most severe finding.

Investigate a service account

Click a service account to open its details. Next to the existing tabs, client metrics adds four tabs. The page opens on the tab that holds the most severe item. Console can’t read a client’s configuration. Suggested settings are inferred from what the clients do, and only appear when the related problem is detected.

Findings reference

Thresholds are conservative: a healthy cluster shows no findings at all.

Permissions

Limitations

  • Supported clients: only clients that support KIP-714 push metrics. librdkafka-based clients send fewer metrics than the Java client, so some findings, such as chatty producer, can’t be detected for them. The stale library check only applies to the Java client.
  • History: Console keeps a sliding window only, 30 minutes with the default settings.
  • Leader changes: Console scrapes collectors from a single instance. After that instance changes, one interval of counter values is lost.
  • Failed scrapes: if any collector of a cluster fails, Console skips the whole scrape for that cluster rather than storing a partial view.
  • Limits: by default, Console tracks up to 500 service accounts per cluster, keeps the 20 largest series (for example, topics) per metric and service account, and tracks up to one million counters per cluster. See Client metrics properties to change them.

Troubleshoot

Console still shows Waiting for the first client push

Symptom: after more than five minutes, the Service accounts page still shows Waiting for the first client push. Cause: the clients don’t push metrics, or Console can’t attribute them. Fix:
  1. Check that the clients support KIP-714: Java client 3.7 or later, or librdkafka 2.5 or later. Check also that enable.metrics.push isn’t set to false.
  2. Without Gateway, check that a client metrics subscription exists: kafka-client-metrics.sh --bootstrap-server <YOUR_BOOTSTRAP_SERVERS> --describe.
  3. Open the collector endpoint, http://<YOUR_COLLECTOR_HOST>:8889/metrics, and check that series starting with org_apache_kafka_ carry principal and client_instance_id labels. If they don’t, enable resource_to_telemetry_conversion on the prometheus exporter.

Values jump and drop to zero

Symptom: charts alternate between a spike and several zero values, or clients appear and disappear. Cause: clients push less often than Console scrapes. Fix: set the push interval to Console’s scrape interval: pushIntervalMs on the Gateway Interceptor, or --interval on the broker subscription.