> ## Documentation Index
> Fetch the complete documentation index at: https://docs.conduktor.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitor Kafka clients with client metrics

> Use KIP-714 client metrics in Console to find misbehaving producers and consumers, see who owns them and get suggested client settings.

> [!NOTE]
>
> Client metrics is a preview feature, available from Console 1.49.0. It's turned off by default and isn't available on the free plan.

Kafka clients from version 3.7 onward push their own metrics to the cluster, as defined in [Kafka Improvement Proposal 714 (KIP-714)](https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability) . Console reads these metrics, groups them by service account and tells you which applications need attention and what to change.

Use client metrics to:

* **Find misbehaving clients**: spot producers that retry silently or send one record per request, unstable consumer groups and clients that open connections constantly.
* **Attribute traffic**: see which service account, application and client library is behind each client instance.
* **Check ownership**: find clients that no Self-service application declares, and topics an application uses without declaring them.
* **Compare with peers**: see where a service account stands against the rest of the cluster.
* **Tune clients**: get suggested values for settings such as `linger.ms`, `batch.size` and `buffer.memory`, inferred from what the clients do.

## How it works

```mermaid theme={null}
flowchart LR
  C[Kafka clients] -- KIP-714 push --> G[Gateway or Kafka brokers]
  G -- OTLP --> O[OpenTelemetry Collector]
  Con[Console] -- scrapes Prometheus endpoint --> O
```

1. Gateway, or your Kafka brokers, tell each client which metrics to send and how often.
2. Clients push their metrics. Gateway or the brokers attach the client's identity (principal and client instance ID) and forward the metrics to an OpenTelemetry (OTel) Collector.
3. The collector serves the metrics on a Prometheus endpoint.
4. Console scrapes that endpoint every 60 seconds by default, groups the values by service account and runs its detection rules on them.

Console keeps a short sliding window of values per service account: 30 points, or 30 minutes with the default settings. It doesn't keep long-term history. To keep history, also export the metrics from your collector to your own observability stack.

## Prerequisites

| Component | Required | Details |
| - | - | - |
| **Kafka clients** | Yes | Java client 3.7 or later, or librdkafka 2.5 or later. Older clients can't push metrics and don't appear. |
| **Gateway or Kafka brokers** | Yes | Either Gateway 3.21.0 or later with the client metrics Interceptor, or Kafka brokers with a client telemetry plugin. See [Send client metrics to a collector](#send-client-metrics-to-a-collector). |
| **OpenTelemetry Collector** | Yes | Receives the metrics over OTLP and exposes them with the `prometheus` exporter. |
| **Console license** | Yes | Client metrics isn't available on the free plan. |

## Set up client metrics

**1. Send client metrics to a collector**

Clients only push metrics when Gateway or the brokers ask for them. Choose the path that matches your deployment.

> [tab] With Gateway
>
> Deploy the [client metrics Interceptor](https://docs.conduktor.io/guide/reference/client-metrics.md) on Gateway and point its OTLP exporter at your collector. Gateway handles KIP-714 itself, so your brokers don't need any setup.
>
> Set `pushIntervalMs` to Console's scrape interval, `60000` by default:
>
> ```yaml theme={null}
> apiVersion: gateway/v2
> kind: Interceptor
> metadata:
>   name: client-metrics
> spec:
>   pluginClass: io.conduktor.gateway.interceptor.clientmetrics.ClientMetricsPlugin
>   priority: 100
>   config:
>     requestedMetrics:
>       - "*"
>     # Match Console's scrape interval (client_metrics.scrape_interval_seconds)
>     pushIntervalMs: 60000
>     exporterConfig:
>       endpoint: "<YOUR_OTLP_COLLECTOR_HOST>:4317"
>       insecure: true
> ```

> [tab] Without Gateway
>
> Your Kafka brokers have to support KIP-714 (Kafka 3.7 or later) and run a client telemetry plugin: a metrics reporter, registered in `metric.reporters`, that implements Kafka's client telemetry interface and forwards the metrics to your collector over OTLP. Apache Kafka doesn't ship one. Without a plugin, brokers don't advertise the telemetry APIs and clients never push.
>
> The plugin has to attach these attributes to every metric. Console ignores metrics without `principal` and `client_instance_id`.
>
> | Attribute | Required | Holds |
> | - | - | - |
> | `principal` | Yes | The client's authenticated principal, for example `User:payments` |
> | `client_instance_id` | Yes | The client instance ID assigned through KIP-714 |
> | `client_id` | No | The client's `client.id` |
> | `client_software_name` | No | The client library name, used for the **Library** column and stale library findings |
> | `client_software_version` | No | The client library version |
>
> Then create a client metrics subscription so clients know what to send. Set the interval to Console's scrape interval, `60000` milliseconds by default:
>
> ```bash theme={null}
> kafka-client-metrics.sh --bootstrap-server <YOUR_BOOTSTRAP_SERVERS> \
>   --alter --name all-clients \
>   --metrics org.apache.kafka.producer.,org.apache.kafka.consumer. \
>   --interval 60000
> ```

> [!WARNING]
>
> Kafka and Gateway default to a push every five minutes. Leave it there and Console sees each client only once every five minutes: series expire from the collector between pushes and counters show as a spike followed by zeros. Set the push interval to Console's scrape interval.

**2. Expose the metrics to Console**

Configure the collector's `prometheus` exporter so Console can attribute each value to a client:

```yaml theme={null}
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889
    # Puts principal and client_instance_id on every series. Without it, Console can't attribute anything
    resource_to_telemetry_conversion:
      enabled: true
    # Lets Console tell a client that sent nothing new from a counter reset
    send_timestamps: true
    add_metric_suffixes: true
    # Longer than the push interval, so a stopped client disappears without leaving gaps for running ones
    metric_expiration: 2m

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheus]
```

Console reads the metrics at `http://<YOUR_COLLECTOR_HOST>:8889/metrics`.

**3. Turn on client metrics in Console**

Set `client_metrics.enabled` to `true` in the Console configuration and restart Console:

> [tab] YAML
>
> ```yaml theme={null}
> client_metrics:
>   enabled: true
> ```

> [tab] Environment variable
>
> ```bash theme={null}
> CDK_CLIENTMETRICS_ENABLED=true
> ```

The other `client_metrics` properties control the scrape interval, the window size and memory limits. See [Client metrics properties](https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-console/environment-variables.md#client-metrics-properties).

**4. Declare the collectors on each cluster**

Add the collectors that serve each cluster's metrics. A cluster can list one collector, or one per broker: Console reads them all and adds up the clients. Declare collectors in the Console UI, in the Console configuration, with environment variables, with the [Conduktor CLI](https://docs.conduktor.io/guide/reference/console-reference.md#kafkacluster) or with the API.

> [tab] Console UI
>
> 1. Go to **Settings** > **Clusters** and select the cluster.
> 2. Open the **Client metrics** tab. It only shows once client metrics is turned on.
> 3. Click **Add a collector** and enter the collector's URL, for example `http://<YOUR_COLLECTOR_HOST>:8889/metrics`.
> 4. If the collector requires it, pick a bearer token or basic authentication under **Authentication**.
> 5. Click **Save**.

> [tab] YAML
>
> ```yaml theme={null}
> clusters:
>   - id: prod
>     name: Production
>     bootstrapServers: <YOUR_BOOTSTRAP_SERVERS>
>     clientMetricsCollectors:
>       - url: http://<YOUR_COLLECTOR_HOST>:8889/metrics
>         # Optional: either a bearer token, or a username and password
>         security:
>           token: <YOUR_COLLECTOR_TOKEN>
>         ignoreUntrustedCertificate: false
> ```

> [tab] Environment variables
>
> ```bash theme={null}
> CDK_CLUSTERS_0_CLIENTMETRICSCOLLECTORS_0_URL=http://<YOUR_COLLECTOR_HOST>:8889/metrics
> CDK_CLUSTERS_0_CLIENTMETRICSCOLLECTORS_0_SECURITY_TOKEN=<YOUR_COLLECTOR_TOKEN>
> ```

> [tab] Conduktor CLI
>
> ```yaml theme={null}
> apiVersion: console/v2
> kind: KafkaCluster
> metadata:
>   name: prod
> spec:
>   displayName: Production
>   bootstrapServers: <YOUR_BOOTSTRAP_SERVERS>
>   clientMetricsCollectors:
>     - url: http://<YOUR_COLLECTOR_HOST>:8889/metrics
>       security:
>         type: BearerToken
>         token: <YOUR_COLLECTOR_TOKEN>
> ```

| Property | Required | Description |
| - | - | - |
| `url` | Yes | The collector's Prometheus endpoint. Starts with `http://` or `https://`. |
| `security` | No | A bearer token (`token`) or basic authentication (`username` and `password`). With the CLI and the API, also set `type` to `BearerToken` or `BasicAuth`. Mutual TLS isn't supported. |
| `ignoreUntrustedCertificate` | No | Skip TLS certificate validation of the collector. Default `false`. |

When the cluster has a Gateway provider, Console only keeps the metrics of the provider's Virtual Cluster, so several Virtual Clusters of one Gateway can share a collector. Console also keeps the metrics sent by your brokers, which don't belong to any Virtual Cluster.

**5. Check that data arrives**

In Console, select the cluster and go to **Service accounts**. Until the first metrics arrive, the page shows **Waiting for the first client push**. Clients ask for their subscription on their own schedule, which can take up to five minutes. Once data arrives, the client health panel appears above the list of service accounts.

## View client health

Select a cluster and go to **Service accounts**. The client health panel at the top of the page summarizes every service account whose clients push metrics:

* **Fleet health**: a score out of 100%, with a level from Excellent to Critical. Each finding lowers its service account's health: a critical finding the most, an info finding the least. One service account can't lower the score by more than its own share.
* **Service accounts without ACLs**: the number of connected service accounts that no ACL mentions. They're superusers, anonymous, or let in because no ACL matched, so no ACL restricts what they do.
* **Categories**: the share of service accounts with at least one finding in each category, and the number of recommendations.
  * **Reliability risk**: applications losing throughput to Kafka or to their own code.
  * **Efficiency and waste**: applications that work, but use more cluster capacity than they need to.
  * **Governance coverage**: applications nobody owns, that connect without authentication or that run old client libraries.
* **Findings**: every finding on the cluster, most severe first.

The list of service accounts below the panel also shows, for each account, its Self-service **Application**, the number of **Clients** connected, the client **Library**, **Records sent /s**, consumer **Lag** and a **Status** that reflects its most severe finding.

## Investigate a service account

Click a service account to open its details. Next to the existing tabs, client metrics adds four tabs. The page opens on the tab that holds the most severe item.

| Tab | Shows |
| - | - |
| **Findings** | What the detection rules found for this account, with the metrics each finding is based on and what to do about it. |
| **Scope** | Topics the account's clients use but its Self-service application never declared, and declared topics that saw no traffic over the window. |
| **Instances** | Each connected client instance, with its client ID and library. Instances that run a stale library are flagged. |
| **Tuning** | How the account's key values compare to the median of all service accounts on the cluster, and client settings worth changing. |

Console can't read a client's configuration. Suggested settings are inferred from what the clients do, and only appear when the related problem is detected.

## Findings reference

| Finding | Category | Severity | What it means | What to do |
| - | - | - | - | - |
| Behind on its own code | Reliability | Critical | The consumer is lagging while it spends most of its time processing records rather than waiting for them. | Add parallelism inside the consumer or make record handling faster. Adding partitions won't help. |
| Producer backpressure | Reliability | Critical | `send()` blocks application threads while it waits for buffer space. | Raise `buffer.memory` to absorb bursts, or find what slows the sends down. |
| Retrying silently | Reliability | Warning | More than 1% of sends are retries, with nothing visibly failing. | Check broker health, request timeouts and the quotas of this service account. |
| Throttled by quotas | Reliability | Warning | The broker throttles this producer: a quota is the limit, not cluster capacity. | Review the quota of this principal. |
| Unstable consumer group | Reliability | Warning | The consumer group rebalances more than six times an hour. | Look for members that exceed `max.poll.interval.ms` or instances that restart. Static group membership stops routine restarts from triggering rebalances. |
| Chatty producer | Efficiency | Warning | A busy producer sends about one record per request, so every record pays a full request's cost. | Raise `linger.ms` and `batch.size` so records group into batches. |
| Ineffective compression | Efficiency | Warning | Compression barely shrinks the payloads, or makes them larger. | Try another codec, or turn compression off for this producer. |
| Connection churn | Efficiency | Info | Each client opens more than one new connection per second, which usually means a client is created per request. | Create the producer or consumer once and reuse it. |
| Unauthenticated client | Governance | Critical | Clients connect without authenticating. | Require authentication on the listener that accepts unauthenticated connections. |
| Unowned client | Governance | Info | No Self-service application declares this service account. | Declare the service account on an application instance, so a team owns its traffic. |
| Stale client library | Governance | Info | Clients run a Java client older than 4.0. | Upgrade the client library. |

Thresholds are conservative: a healthy cluster shows no findings at all.

## Permissions

| To see | You need |
| - | - |
| The client health panel of a cluster | The **View ACL** permission on the cluster |
| The client metrics of a service account | The **View ACL** permission on the cluster, or membership of the Self-service application that declares the service account |

## Limitations

* **Supported clients**: only clients that support KIP-714 push metrics. librdkafka-based clients send fewer metrics than the Java client, so some findings, such as chatty producer, can't be detected for them. The stale library check only applies to the Java client.
* **History**: Console keeps a sliding window only, 30 minutes with the default settings.
* **Leader changes**: Console scrapes collectors from a single instance. After that instance changes, one interval of counter values is lost.
* **Failed scrapes**: if any collector of a cluster fails, Console skips the whole scrape for that cluster rather than storing a partial view.
* **Limits**: by default, Console tracks up to 500 service accounts per cluster, keeps the 20 largest series (for example, topics) per metric and service account, and tracks up to one million counters per cluster. See [Client metrics properties](https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-console/environment-variables.md#client-metrics-properties) to change them.

## Troubleshoot

### Console still shows Waiting for the first client push

**Symptom**: after more than five minutes, the **Service accounts** page still shows **Waiting for the first client push**.

**Cause**: the clients don't push metrics, or Console can't attribute them.

**Fix**:

1. Check that the clients support KIP-714: Java client 3.7 or later, or librdkafka 2.5 or later. Check also that `enable.metrics.push` isn't set to `false`.
2. Without Gateway, check that a client metrics subscription exists: `kafka-client-metrics.sh --bootstrap-server <YOUR_BOOTSTRAP_SERVERS> --describe`.
3. Open the collector endpoint, `http://<YOUR_COLLECTOR_HOST>:8889/metrics`, and check that series starting with `org_apache_kafka_` carry `principal` and `client_instance_id` labels. If they don't, enable `resource_to_telemetry_conversion` on the `prometheus` exporter.

### Values jump and drop to zero

**Symptom**: charts alternate between a spike and several zero values, or clients appear and disappear.

**Cause**: clients push less often than Console scrapes.

**Fix**: set the push interval to Console's scrape interval: `pushIntervalMs` on the Gateway Interceptor, or `--interval` on the broker subscription.

## Related resources

* [Client metrics Interceptor](https://docs.conduktor.io/guide/reference/client-metrics.md)
* [Client metrics properties](https://docs.conduktor.io/guide/conduktor-in-production/deploy-artifacts/deploy-console/environment-variables.md#client-metrics-properties)
* [Self-service](https://docs.conduktor.io/guide/conduktor-concepts/self-service.md)
* [KIP-714: Client metrics and observability](https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability)&#x20;


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.