Kafka metrics reference
The metrics endpoint is located on/monitoring/metrics of your deployed Console instance. Data points are refreshed every 30 seconds.
| Metric name | Metric type | Labels | Description |
|---|---|---|---|
| Consumer group metrics | |||
kafka_consumergroup_group_lag | gauge | cluster_id, cluster_name, group, topic, partition | Absolute Lag of the consumer group on this topic-partition |
kafka_consumergroup_group_lag_seconds | gauge | cluster_id, cluster_name, group, topic, partition | Lag in seconds of the consumer group on this topic-partition |
kafka_consumergroup_group_offset | gauge | cluster_id, cluster_name, group, topic, partition | Last committed offset of the consumer group on this topic-partition |
| Consumer group metrics (aggregated) | |||
kafka_consumergroup_group_max_lag | gauge | cluster_id, cluster_name, group | Max group offset lag |
kafka_consumergroup_group_max_lag_seconds | gauge | cluster_id, cluster_name, group | Lag in seconds of the consumer group (all topics) |
kafka_consumergroup_group_sum_lag | gauge | cluster_id, cluster_name, group | Sum of Absolute Lag of the consumer group (all topics) |
kafka_consumergroup_group_topic_sum_lag | gauge | cluster_id, cluster_name, group, topic | Sum of Absolute Lag of the consumer group on this topic (all partitions) |
kafka_consumergroup_group_topic_max_lag_seconds | gauge | cluster_id, cluster_name, group, topic | Max lag in seconds of the consumer group on this topic |
kafka_group_consume_rate | gauge | cluster_id, cluster_name, topic | Group consume rate |
| Kafka cluster and broker metrics | |||
kafka_cluster_consume_rate | gauge | cluster_id, cluster_name | Cluster consume rate |
kafka_cluster_produce_rate | gauge | cluster_id, cluster_name | Number of messages produced per second |
kafka_controller_kafkacontroller_activebrokercount | gauge | cluster_name, cluster_id | Number of active brokers on the Kafka cluster |
kafka_controller_kafkacontroller_activecontrollercount | gauge | cluster_name, cluster_id | Number of active controllers on the Kafka cluster |
kafka_controller_kafkacontroller_globalpartitioncount | gauge | cluster_name, cluster_id | Total number of partitions on the Kafka cluster |
kafka_controller_kafkacontroller_offlinepartitionscount | gauge | cluster_name, cluster_id | Number of partitions with no active Leader |
kafka_broker_size | gauge | cluster_id, cluster_name, broker_id, broker_host | Size in bytes of the current broker |
kafka_server_replicamanager_leadercount | gauge | cluster_name, cluster_id, broker_id, broker_host | Number of partition Leaders on this Broker |
kafka_server_replicamanager_partitioncount | gauge | cluster_name, cluster_id, broker_id, broker_host | Number of partitions on this Broker |
kafka_server_replicamanager_underminisrpartitioncount | gauge | cluster_name, cluster_id | Number of partitions with replicas that don’t meet their minimum ISR |
kafka_server_replicamanager_underreplicatedpartitions | gauge | cluster_name, cluster_id | Number of partitions with replicas that don’t meet their replication factor |
| Kafka connect metrics | |||
kafka_connect_failed_tasks | gauge | cluster_name, cluster_id, connect_cluster_id, connector | Number of failed tasks for this Connector |
kafka_connect_total_tasks | gauge | cluster_name, cluster_id, connect_cluster_id, connector | Total number of tasks configured for this Connector |
| Kafka topic and partition metrics | |||
kafka_topic_message_count | gauge | cluster_id, cluster_name, topic | Message count of the current topic |
kafka_topic_produce_rate | gauge | cluster_id, cluster_name, topic | Number of messages produced per second |
kafka_topic_size | gauge | cluster_id, cluster_name, topic | Size in bytes of the current topic |
kafka_cluster_partition_underminisr | gauge | cluster_name, cluster_id, topic, partition | 1 when the partition is under min ISR |
kafka_cluster_partition_underreplicated | gauge | cluster_name, cluster_id, topic, partition | 1 when the partition is under-replicated |
kafka_partition_earliest_offset | gauge | cluster_id, cluster_name, topic, partition | Earliest offset of the current topic-partition |
kafka_partition_latest_offset | gauge | cluster_id, cluster_name, topic, partition | Latest offset of the current topic-partition |
| Others | |||
kafka_consumergroup_poll_time_ms | gauge | cluster_id, cluster_name | Group time poll time |
kafka_controller_controllerstats_uncleanleaderelectionspersec_created | gauge | cluster_name, cluster_id | Counter for kafka_controller_controllerstats_uncleanleaderelectionspersec metric |
kafka_controller_controllerstats_uncleanleaderelectionspersec_total | counter | cluster_name, cluster_id | Counter for kafka_controller_controllerstats_uncleanleaderelectionspersec metric |
kafka_log_log_size | gauge | broker_host, partition, broker_id, topic, cluster_name, cluster_id | Size in bytes of the current topic-partition |
| Data quality metrics | |||
data_quality_policy_msg_evaluated_total | counter | policy_name | Number of messages evaluated per data quality policy |
data_quality_policy_msg_evaluated_created | gauge | policy_name | Timestamp when data_quality_policy_msg_evaluated metric was created |
data_quality_rule_msg_evaluated_total | counter | rule_name, policy_name | Number of messages evaluated per data quality rule |
data_quality_violations_per_policy_total | counter | policy_name | Number of violations per data quality policy |
data_quality_violations_per_policy_created | gauge | policy_name | Timestamp when data_quality_violations_per_policy metric was created |
data_quality_violations_per_rule_total | counter | rule_name, policy_name | Number of violations per data quality rule |
data_quality_violations_per_rule_created | gauge | rule_name, policy_name | Timestamp when data_quality_violations_per_rule metric was created |
Note that the label
cluster_name is obsolete and will be removed soon. You should use cluster_id instead.Console reference
The metrics endpoint is located on/api/metrics of your deployed Console instance.
| Metric name | Metric type | Labels | Description |
|---|---|---|---|
| Console API | |||
console_api_request_active | gauge | path, method | |
console_api_request_duration_seconds | histogram | ||
console_api_request_duration_seconds_count | path, method, status, phase | ||
console_api_request_duration_seconds_max | path, method, status, phase | ||
console_api_request_duration_seconds_min | path, method, status, phase | ||
console_api_request_duration_seconds_sum | path, method, status, phase | ||
console_api_request_total | counter | path, method, status | |
| Console cache | |||
console_cache_cluster_reachable_hits_total | counter | Number of cache hits for cluster_reachable | |
console_cache_cluster_reachable_misses_total | counter | Number of cache miss for cluster_reachable | |
console_cache_cluster_reachable_size | gauge | Total cache size for cluster_reachable | |
console_cache_kafka_admin_hits_total | counter | Number of cache hits for kafka_admin | |
console_cache_kafka_admin_misses_total | counter | Number of cache miss for kafka_admin | |
console_cache_kafka_admin_size | gauge | Total cache size for kafka_admin | |
console_cache_schema_registry_clients_hits_total | counter | Number of cache hits for schema_registry_clients | |
console_cache_schema_registry_clients_misses_total | counter | Number of cache miss for schema_registry_clients | |
console_cache_schema_registry_clients_size | gauge | Total cache size for schema_registry_clients | |
| Kafka cluster indexing | |||
console_indexer_kafka_cluster_duration | histogram | Duration in milliseconds to index Kafka cluster | |
console_indexer_kafka_cluster_duration_count | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_cluster_duration_max | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_cluster_duration_min | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_cluster_duration_sum | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_cluster_failed_count | gauge | Number of failed Kafka cluster indexing tasks | |
console_indexer_kafka_cluster_skipped_count | gauge | Number of skipped Kafka cluster indexing tasks | |
console_indexer_kafka_cluster_succeeded_count | gauge | Number of succeeded Kafka cluster indexing tasks | |
console_indexer_kafka_cluster_timeout_count | gauge | Number of timed out Kafka cluster indexing tasks | |
console_indexer_kafka_describe_topics_duration | histogram | Duration in milliseconds to describes all topics per Kafka cluster | |
console_indexer_kafka_describe_topics_duration_count | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_describe_topics_duration_max | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_describe_topics_duration_min | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_describe_topics_duration_sum | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_get_consumers_groups_duration | histogram | Duration in milliseconds to get all consumers groups state per Kafka cluster | |
console_indexer_kafka_get_consumers_groups_duration_count | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_get_consumers_groups_duration_max | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_get_consumers_groups_duration_min | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_get_consumers_groups_duration_sum | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_get_deployed_connectors_duration | histogram | Duration in milliseconds to list all deployed connector per Kafka cluster | |
console_indexer_kafka_get_deployed_connectors_duration_count | time_unit, technical_id, cluster_type, kafka_connect | ||
console_indexer_kafka_get_deployed_connectors_duration_max | time_unit, technical_id, cluster_type, kafka_connect | ||
console_indexer_kafka_get_deployed_connectors_duration_min | time_unit, technical_id, cluster_type, kafka_connect | ||
console_indexer_kafka_get_deployed_connectors_duration_sum | time_unit, technical_id, cluster_type, kafka_connect | ||
console_indexer_kafka_get_topics_configs_duration | histogram | Duration in milliseconds to get all topics topics configuration per Kafka cluster | |
console_indexer_kafka_get_topics_configs_duration_count | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_get_topics_configs_duration_max | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_get_topics_configs_duration_min | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_get_topics_configs_duration_sum | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_get_topics_partitions_duration | histogram | Duration in milliseconds to get all topics partitions state per Kafka cluster | |
console_indexer_kafka_get_topics_partitions_duration_count | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_get_topics_partitions_duration_max | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_get_topics_partitions_duration_min | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_get_topics_partitions_duration_sum | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_init_cluster_connection_duration | histogram | Duration in milliseconds to initialize the connection per Kafka cluster | |
console_indexer_kafka_init_cluster_connection_duration_count | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_init_cluster_connection_duration_max | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_init_cluster_connection_duration_min | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_init_cluster_connection_duration_sum | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_list_topics_duration | histogram | Duration in milliseconds to list topics per Kafka cluster | |
console_indexer_kafka_list_topics_duration_count | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_list_topics_duration_max | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_list_topics_duration_min | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_list_topics_duration_sum | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_total_clusters_count | gauge | Total number of Kafka clusters | |
console_indexer_kafka_total_duration | histogram | Total duration in milliseconds to index Kafka cluster | |
console_indexer_kafka_total_duration_count | time_unit | ||
console_indexer_kafka_total_duration_max | time_unit | ||
console_indexer_kafka_total_duration_min | time_unit | ||
console_indexer_kafka_total_duration_sum | time_unit | ||
| Kafka connect indexing | |||
console_indexer_kafka_connect_cluster_failed_count | gauge | Number of failed Kafka Connect server indexing tasks | |
console_indexer_kafka_connect_cluster_skipped_count | gauge | Number of skipped Kafka Connect server indexing tasks | |
console_indexer_kafka_connect_cluster_succeeded_count | gauge | Number of succeeded Kafka Connect server indexing tasks | |
console_indexer_kafka_connect_cluster_timeout_count | gauge | Number of timed out Kafka Connect server indexing tasks | |
console_indexer_kafka_connect_total_clusters_count | gauge | Total number of Kafka Connect server | |
console_indexer_kafka_connect_total_duration | histogram | Total duration in milliseconds to index Kafka Connect server | |
console_indexer_kafka_connect_total_duration_count | time_unit | ||
console_indexer_kafka_connect_total_duration_max | time_unit | ||
console_indexer_kafka_connect_total_duration_min | time_unit | ||
console_indexer_kafka_connect_total_duration_sum | time_unit | ||
| Schema registry indexing | |||
console_indexer_schema_registry_cluster_failed_count | gauge | Number of failed schema registry server indexing tasks | |
console_indexer_schema_registry_cluster_skipped_count | gauge | Number of skipped schema registry server indexing tasks | |
console_indexer_schema_registry_cluster_succeeded_count | gauge | Number of succeeded schema registry server indexing tasks | |
console_indexer_schema_registry_cluster_timeout_count | gauge | Number of timed out schema registry server indexing tasks | |
console_indexer_schema_registry_total_clusters_count | gauge | Total number of schema registry server | |
console_indexer_schema_registry_total_duration | histogram | Total duration in milliseconds to index schema registry server | |
console_indexer_schema_registry_total_duration_count | time_unit | ||
console_indexer_schema_registry_total_duration_max | time_unit | ||
console_indexer_schema_registry_total_duration_min | time_unit | ||
console_indexer_schema_registry_total_duration_sum | time_unit |
Gateway reference
The metrics endpoint is<gateway_host>:<gateway_port>/metrics of your deployed Gateway instance. For example: localhost:8888/metrics.
If
GATEWAY_SECURED_METRICS is enabled (which is the default), you’ll have to use the credentials specified in GATEWAY_ADMIN_API_USERS to access the metrics.| Metric name | Description |
|---|---|
gateway_upstream_connection_close_rate | The number of connections closed per second |
gateway_upstream_connection_close_total | The total number of connections closed |
gateway_upstream_connection_creation_rate | The number of new connections established per second |
gateway_upstream_connection_creation_total | The total number of new connections established |
gateway_upstream_select_rate | The number of times the I/O layer checked for new I/O to perform per second |
gateway_upstream_select_total | The total number of times the I/O layer checked for new I/O to perform |
gateway_upstream_io_wait_rate | The number of time the I/O thread spent waiting |
gateway_upstream_io_wait_total | The total time the I/O thread spent waiting |
gateway_brokered_active_connections | The number of active broker connections of the connection pool |
gateway_active_connections_vcluster | The number of active connections per vcluster |
gateway_latency_request_response | The latency to process a request and generate a response |
gateway_apiKeys_latency_request_response | The latency to process a request and generate a response for each API key |
gateway_bytes_exchanged | The total number of bytes exchanged through Gateway |
gateway_bytes_exchanged_vcluster | The total bytes exchanged within the context of the specified Virtual Cluster |
gateway_thread_request_rebuild | A counter on number of rebuilding Kafka request |
gateway_thread_tasks | The number of pending tasks on our Gateway thread (where all rebuilding request/response happen) |
gateway_upstream_connections_upstream_connected | The number of connections from Gateway to the backing Kafka cluster |
gateway_upstream_connections_downstream | The number of connections from clients to Gateway |
gateway_upstream_io_nodes | The number of Kafka nodes |
gateway_kcache_size | The size of the kcache (equal to the number of key-value pairs we have in the cache) |
gateway_backend_brokered_active_connections | The number of active backend brokers |
gateway_failed_authentications | The number of authentication attempts that failed for each user |
gateway_topic_log_end_offset | The log end offset of client topics |
gateway_topic_current_offset | The current offset of consumer group on client topic |
gateway_bytes_exchanged_topic_total | The total bytes exchanged within the context of the specified topic |
gateway_error_per_apiKeys | The total errors per API key for the specified virtual cluster and username |
gateway_current_inflight_apiKeys | The current inflight API keys of the specified virtual cluster and username |