Since Console v1.42.0, you can protect access to the
/monitoring/metrics and /api/monitoring/metrics endpoints with basic auth. When configured, your monitoring backend must provide credentials to scrape these metrics. Learn how to configure basic auth for metrics.Kafka metrics reference
The metrics endpoint is located on/monitoring/metrics of your deployed Console instance. Data points are refreshed every 30 seconds.
| Metric name | Metric type | Labels | Description |
|---|---|---|---|
| Consumer group metrics | |||
kafka_consumergroup_group_lag | gauge | cluster_id, cluster_name, group, topic, partition | Absolute Lag of the consumer group on this topic-partition |
kafka_consumergroup_group_lag_seconds | gauge | cluster_id, cluster_name, group, topic, partition | Lag in seconds of the consumer group on this topic-partition |
kafka_consumergroup_group_offset | gauge | cluster_id, cluster_name, group, topic, partition | Last committed offset of the consumer group on this topic-partition |
| Consumer group metrics (aggregated) | |||
kafka_consumergroup_group_max_lag | gauge | cluster_id, cluster_name, group | Max group offset lag |
kafka_consumergroup_group_max_lag_seconds | gauge | cluster_id, cluster_name, group | Lag in seconds of the consumer group (all topics) |
kafka_consumergroup_group_sum_lag | gauge | cluster_id, cluster_name, group | Sum of Absolute Lag of the consumer group (all topics) |
kafka_consumergroup_group_topic_sum_lag | gauge | cluster_id, cluster_name, group, topic | Sum of Absolute Lag of the consumer group on this topic (all partitions) |
kafka_consumergroup_group_topic_max_lag_seconds | gauge | cluster_id, cluster_name, group, topic | Max lag in seconds of the consumer group on this topic |
kafka_group_consume_rate | gauge | cluster_id, cluster_name, topic | Group consume rate |
| Kafka cluster and broker metrics | |||
kafka_cluster_consume_rate | gauge | cluster_id, cluster_name | Cluster consume rate |
kafka_cluster_produce_rate | gauge | cluster_id, cluster_name | Number of messages produced per second |
kafka_controller_kafkacontroller_activebrokercount | gauge | cluster_name, cluster_id | Number of active brokers on the Kafka cluster |
kafka_controller_kafkacontroller_activecontrollercount | gauge | cluster_name, cluster_id | Number of active controllers on the Kafka cluster |
kafka_controller_kafkacontroller_globalpartitioncount | gauge | cluster_name, cluster_id | Total number of partitions on the Kafka cluster |
kafka_controller_kafkacontroller_offlinepartitionscount | gauge | cluster_name, cluster_id | Number of partitions with no active Leader |
kafka_broker_size | gauge | cluster_id, cluster_name, broker_id, broker_host | Size in bytes of the current broker |
kafka_server_replicamanager_leadercount | gauge | cluster_name, cluster_id, broker_id, broker_host | Number of partition Leaders on this Broker |
kafka_server_replicamanager_partitioncount | gauge | cluster_name, cluster_id, broker_id, broker_host | Number of partitions on this Broker |
kafka_server_replicamanager_underminisrpartitioncount | gauge | cluster_name, cluster_id | Number of partitions with replicas that don’t meet their minimum ISR |
kafka_server_replicamanager_underreplicatedpartitions | gauge | cluster_name, cluster_id | Number of partitions with replicas that don’t meet their replication factor |
| Kafka connect metrics | |||
kafka_connect_failed_tasks | gauge | cluster_name, cluster_id, connect_cluster_id, connector | Number of failed tasks for this Connector |
kafka_connect_total_tasks | gauge | cluster_name, cluster_id, connect_cluster_id, connector | Total number of tasks configured for this Connector |
| Kafka topic and partition metrics | |||
kafka_topic_message_count | gauge | cluster_id, cluster_name, topic | Message count of the current topic |
kafka_topic_produce_rate | gauge | cluster_id, cluster_name, topic | Number of messages produced per second |
kafka_topic_size | gauge | cluster_id, cluster_name, topic | Size in bytes of the current topic |
kafka_cluster_partition_underminisr | gauge | cluster_name, cluster_id, topic, partition | 1 when the partition is under min ISR |
kafka_cluster_partition_underreplicated | gauge | cluster_name, cluster_id, topic, partition | 1 when the partition is under-replicated |
kafka_partition_earliest_offset | gauge | cluster_id, cluster_name, topic, partition | Earliest offset of the current topic-partition |
kafka_partition_latest_offset | gauge | cluster_id, cluster_name, topic, partition | Latest offset of the current topic-partition |
| Others | |||
kafka_consumergroup_poll_time_ms | gauge | cluster_id, cluster_name | Group time poll time |
kafka_controller_controllerstats_uncleanleaderelectionspersec_created | gauge | cluster_name, cluster_id | Counter for kafka_controller_controllerstats_uncleanleaderelectionspersec metric |
kafka_controller_controllerstats_uncleanleaderelectionspersec_total | counter | cluster_name, cluster_id | Counter for kafka_controller_controllerstats_uncleanleaderelectionspersec metric |
kafka_log_log_size | gauge | broker_host, partition, broker_id, topic, cluster_name, cluster_id | Size in bytes of the current topic-partition |
| Data quality metrics | |||
data_quality_policy_msg_evaluated_total | counter | policy_name | Number of messages evaluated per data quality policy |
data_quality_policy_msg_evaluated_created | gauge | policy_name | Timestamp when data_quality_policy_msg_evaluated metric was created |
data_quality_rule_msg_evaluated_total | counter | rule_name, policy_name | Number of messages evaluated per data quality rule |
data_quality_violations_per_policy_total | counter | policy_name | Number of violations per data quality policy |
data_quality_violations_per_policy_created | gauge | policy_name | Timestamp when data_quality_violations_per_policy metric was created |
data_quality_violations_per_rule_total | counter | rule_name, policy_name | Number of violations per data quality rule |
data_quality_violations_per_rule_created | gauge | rule_name, policy_name | Timestamp when data_quality_violations_per_rule metric was created |
Note that the label
cluster_name is obsolete and will be removed soon. You should use cluster_id instead.Console metrics
The metrics endpoint is located on/api/metrics of your deployed Console instance.
| Metric name | Metric type | Labels | Description |
|---|---|---|---|
| Console API | |||
console_api_request_active | gauge | path, method | |
console_api_request_duration_seconds | histogram | ||
console_api_request_duration_seconds_count | path, method, status, phase | ||
console_api_request_duration_seconds_max | path, method, status, phase | ||
console_api_request_duration_seconds_min | path, method, status, phase | ||
console_api_request_duration_seconds_sum | path, method, status, phase | ||
console_api_request_total | counter | path, method, status | |
| Console cache | |||
console_cache_cluster_reachable_hits_total | counter | Number of cache hits for cluster_reachable | |
console_cache_cluster_reachable_misses_total | counter | Number of cache miss for cluster_reachable | |
console_cache_cluster_reachable_size | gauge | Total cache size for cluster_reachable | |
console_cache_kafka_admin_hits_total | counter | Number of cache hits for kafka_admin | |
console_cache_kafka_admin_misses_total | counter | Number of cache miss for kafka_admin | |
console_cache_kafka_admin_size | gauge | Total cache size for kafka_admin | |
console_cache_schema_registry_clients_hits_total | counter | Number of cache hits for schema_registry_clients | |
console_cache_schema_registry_clients_misses_total | counter | Number of cache miss for schema_registry_clients | |
console_cache_schema_registry_clients_size | gauge | Total cache size for schema_registry_clients | |
| Kafka cluster indexing | |||
console_indexer_kafka_cluster_duration | histogram | Duration in milliseconds to index Kafka cluster | |
console_indexer_kafka_cluster_duration_count | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_cluster_duration_max | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_cluster_duration_min | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_cluster_duration_sum | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_cluster_failed_count | gauge | Number of failed Kafka cluster indexing tasks | |
console_indexer_kafka_cluster_skipped_count | gauge | Number of skipped Kafka cluster indexing tasks | |
console_indexer_kafka_cluster_succeeded_count | gauge | Number of succeeded Kafka cluster indexing tasks | |
console_indexer_kafka_cluster_timeout_count | gauge | Number of timed out Kafka cluster indexing tasks | |
console_indexer_kafka_describe_topics_duration | histogram | Duration in milliseconds to describes all topics per Kafka cluster | |
console_indexer_kafka_describe_topics_duration_count | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_describe_topics_duration_max | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_describe_topics_duration_min | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_describe_topics_duration_sum | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_get_consumers_groups_duration | histogram | Duration in milliseconds to get all consumers groups state per Kafka cluster | |
console_indexer_kafka_get_consumers_groups_duration_count | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_get_consumers_groups_duration_max | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_get_consumers_groups_duration_min | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_get_consumers_groups_duration_sum | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_get_deployed_connectors_duration | histogram | Duration in milliseconds to list all deployed connector per Kafka cluster | |
console_indexer_kafka_get_deployed_connectors_duration_count | time_unit, technical_id, cluster_type, kafka_connect | ||
console_indexer_kafka_get_deployed_connectors_duration_max | time_unit, technical_id, cluster_type, kafka_connect | ||
console_indexer_kafka_get_deployed_connectors_duration_min | time_unit, technical_id, cluster_type, kafka_connect | ||
console_indexer_kafka_get_deployed_connectors_duration_sum | time_unit, technical_id, cluster_type, kafka_connect | ||
console_indexer_kafka_get_topics_configs_duration | histogram | Duration in milliseconds to get all topics configuration per Kafka cluster | |
console_indexer_kafka_get_topics_configs_duration_count | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_get_topics_configs_duration_max | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_get_topics_configs_duration_min | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_get_topics_configs_duration_sum | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_get_topics_partitions_duration | histogram | Duration in milliseconds to get all topics partitions state per Kafka cluster | |
console_indexer_kafka_get_topics_partitions_duration_count | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_get_topics_partitions_duration_max | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_get_topics_partitions_duration_min | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_get_topics_partitions_duration_sum | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_init_cluster_connection_duration | histogram | Duration in milliseconds to initialize the connection per Kafka cluster | |
console_indexer_kafka_init_cluster_connection_duration_count | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_init_cluster_connection_duration_max | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_init_cluster_connection_duration_min | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_init_cluster_connection_duration_sum | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_list_topics_duration | histogram | Duration in milliseconds to list topics per Kafka cluster | |
console_indexer_kafka_list_topics_duration_count | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_list_topics_duration_max | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_list_topics_duration_min | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_list_topics_duration_sum | time_unit, technical_id, cluster_type | ||
console_indexer_kafka_total_clusters_count | gauge | Total number of Kafka clusters | |
console_indexer_kafka_total_duration | histogram | Total duration in milliseconds to index Kafka cluster | |
console_indexer_kafka_total_duration_count | time_unit | ||
console_indexer_kafka_total_duration_max | time_unit | ||
console_indexer_kafka_total_duration_min | time_unit | ||
console_indexer_kafka_total_duration_sum | time_unit | ||
| Kafka connect indexing | |||
console_indexer_kafka_connect_cluster_failed_count | gauge | Number of failed Kafka Connect server indexing tasks | |
console_indexer_kafka_connect_cluster_skipped_count | gauge | Number of skipped Kafka Connect server indexing tasks | |
console_indexer_kafka_connect_cluster_succeeded_count | gauge | Number of succeeded Kafka Connect server indexing tasks | |
console_indexer_kafka_connect_cluster_timeout_count | gauge | Number of timed out Kafka Connect server indexing tasks | |
console_indexer_kafka_connect_total_clusters_count | gauge | Total number of Kafka Connect server | |
console_indexer_kafka_connect_total_duration | histogram | Total duration in milliseconds to index Kafka Connect server | |
console_indexer_kafka_connect_total_duration_count | time_unit | ||
console_indexer_kafka_connect_total_duration_max | time_unit | ||
console_indexer_kafka_connect_total_duration_min | time_unit | ||
console_indexer_kafka_connect_total_duration_sum | time_unit | ||
| Schema registry indexing | |||
console_indexer_schema_registry_cluster_failed_count | gauge | Number of failed schema registry server indexing tasks | |
console_indexer_schema_registry_cluster_skipped_count | gauge | Number of skipped schema registry server indexing tasks | |
console_indexer_schema_registry_cluster_succeeded_count | gauge | Number of succeeded schema registry server indexing tasks | |
console_indexer_schema_registry_cluster_timeout_count | gauge | Number of timed out schema registry server indexing tasks | |
console_indexer_schema_registry_total_clusters_count | gauge | Total number of schema registry server | |
console_indexer_schema_registry_total_duration | histogram | Total duration in milliseconds to index schema registry server | |
console_indexer_schema_registry_total_duration_count | time_unit | ||
console_indexer_schema_registry_total_duration_max | time_unit | ||
console_indexer_schema_registry_total_duration_min | time_unit | ||
console_indexer_schema_registry_total_duration_sum | time_unit | ||
| License metrics | |||
console_license_remaining_days | gauge | The number of days remaining on your Console license |