Skip to main content
Quick navigation

Important Notices

Below outlines important notices relating to Conduktor Platform.

Monitoring is changing (January 27, 2023)

On January 27, 2023, we announced that as of April 3rd, 2023 there will be changes made to improve the ease of setup and usability of Monitoring.

Since launching in September 2022, we have listened carefully to customer feedback. One area that has been prone to misconfiguration is setting up the Prometheus Node/JMX exporter. Conduktor’s goal has always been to make our users lives easier, not complicate them.

As a result, we will focus our immediate attention on providing an excellent Monitoring experience without the need for an Agent. This allows us to maximize support for all Kafka vendors out-of-the-box, rather than investing in a wide variety of bespoke setups and integrations. This means you will get the benefits of Monitoring without any additional configuration. Simply add your cluster to your organisation, and we will start collecting metrics immediately.

Though it will not be possible to derive every metric without any agent, we will continue to provide the most critical metrics for ensuring observability of your Kafka applications. See below on the metrics we will continue to collect.

ContextMetricDefinition
Apps MonitoringConsumer group statusIndication of healthy or critical status based on lag(s).
Critical if max lag/s exceeds 180
Apps MonitoringLag messages countNumber of messages each consumer group is behind per partition.
Apps MonitoringLag(s)Estimated number of seconds each consumer group is behind in the topic.
Cluster HealthMessages count
per cluster (s)
This metric gives you the ability to gauge how active your producers are. Given batching and other factors this metric will change over time.
Cluster HealthOffline partitions countOffline partitions can be caused by lingering capacity issues, crashed brokers or cluster wide faults. This is a critical factor in the healthiness of your cluster because an offline partition can not be produced to or consumed from. The view here is that of the controller, if the controller believes a partition is offline it may not reassign or bring online a leader.
Cluster HealthUnder replicated partitions countUnder replicated partitions are a risk to data durability as well as availability. Under replicated partitions can happen for various reasons including, inability for replicas to keep up or network splits.
Cluster HealthUnder min ISR partitions countUnder minimum ISR partitions do not meet the durability requirements to be produced to. Producers that try to produce messages to a partition that is under the specified minimum isr will have the messages rejected and be forced to handle the exception.
Cluster HealthPartitions countTotal number of partitions(including replicas) across selected Kafka cluster.
Cluster HealthActive brokers countNumber of active brokers on selected Kafka cluster.
Cluster HealthActive partitions countTotal number of partitions active on selected Kafka cluster.
Cluster HealthActive controllers countTotal number of active controllers on selected Kafka cluster.
Topic MonitoringMessages in per topic (/s)Number of messages produced per second, per broker at a topic granularity.
Topic MonitoringMessages out per topic (/s)Number of messages consumed per second, per broker at a topic granularity.
Topic MonitoringMessages out per topic (/s)Number of messages consumed per second, per broker at a topic granularity.
Topic MonitoringTotal size of messagesTotal size of messages in the topic.
Topic MonitoringDuplicates countDuplicate message count over the last N minutes, per topic.
Note this depends on Topic Analyzer being enabled.
Topic MonitoringDistinct duplicates countNumber of distinct, duplicated messages over the last N minutes, per topic.
Note this depends on Topic Analyzer being enabled.
Topic MonitoringTransaction abort countNumber of transactions aborted per topic.
Note this depends on Topic Analyzer being enabled.
Topic MonitoringBatch sizeAverage size of batches produced to a topic.
Note this depends on Topic Analyzer being enabled.
Topic MonitoringMessages per batchAverage number of messages produced per Batch to a topic.
Note this depends on Topic Analyzer being enabled.