> ## Documentation Index
> Fetch the complete documentation index at: https://docs.conduktor.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Indexing

> Console Indexing collects Kafka metadata every 30 seconds into internal tables for topic sorting, filtering, and labelling.

In <Tooltip tip="A user interface for managing and monitoring Kafka infrastructure, supporting Kubernetes deployment.">Console</Tooltip>, many of the list pages (e.g. Topics, Consumer Groups, Schema Registry, Kafka Connect) are served from an **internal table** instead of making *AdminClient* calls to Kafka.

Theses tables are updated by a background process we call Indexing. The Indexing process collects changes to your Kafka every 30 seconds and stores that metadata in internal tables.

<img src="https://mintcdn.com/conduktor/Kbwb2r5-pp_mR-Oq/images/indexing-explained.png?fit=max&auto=format&n=Kbwb2r5-pp_mR-Oq&q=85&s=14ae90e6c62f62faa1cd49c7543d517f" alt="Indexing explained" width="4137" height="1902" data-path="images/indexing-explained.png" />

<Info>
  Detail pages are not using Indexing. As soon as you're on a page for a specific topic or consumer group, the data is fetched directly from the Kafka cluster.
</Info>

## Benefits

Indexing improves user experience and provides functionality that's not available with out-of-the-box Kafka resources, such as:

* smart tables with **sorting and filtering** capabilities, allowing you to get message count, size and much more
* **search** and **labelling** that allows you to organize and find required resources

## Examples

These are just some examples of how Indexing can help you:

* identify **idle topics**: which topics (that have no active consumers and producers) haven't published data for more than a week?
* are there any **over-partitioned topics** - topics that have a large number of partitions while the biggest consumer group only has a few members consuming in parallel?
* prioritize **data at risk topics**: which topics are at risk of losing data, because of the replication factor or the min ISR being incorrect?
* find **outliers topics**: which topics contain bad or overridden configurations that they shouldn't have?
* remove **over-replicated applications**: which consumer groups have idle members? Typically, this is because the number of consumers exceeded the number of total partitions.

## Circuit breaker configuration

Each indexed task in Console includes configurable circuit breaker behavior that provides fine-grained control over indexing resilience and recovery. This configuration helps prevent prolonged indexing outages and gives administrators better control over system behavior.

### Configuration parameters

**`sequentialFailureThreshold`**

* Controls how many consecutive failures have to occur before the circuit breaker activates
* When the threshold is reached, the indexing task will temporarily stop attempting to collect data
* Prevents resource exhaustion during extended connectivity or permissions issues

**`blockingDuration`**

* Defines how long the circuit breaker remains active before attempting to resume indexing
* Provides a controlled recovery mechanism with predictable timing
* Ensures predictable recovery intervals for indexing operations

### Benefits

With these circuit breaker settings, you can:

* Adjust failure detection sensitivity based on your environment's reliability
* Control recovery timing to balance system stability with data freshness
* Maintain predictable indexing behavior even during temporary infrastructure issues
* Prevent resource exhaustion during extended connectivity problems

<Info>
  Circuit breaker configuration is applied per indexed tasks (e.g., topic metadata polling, consumer group indexing). For guidance on optimal settings for your specific deployment scenario, [contact support](https://support.conduktor.io/hc/en-gb/requests/new) <Icon icon="up-right-from-square" />.
</Info>

## Troubleshoot

<Accordion title="I've just created a topic, but get 'Indexing in progress'/can't see the metadata">
  Topics created 'now' would not be indexed until the next Indexing cycle.

  This means they wouldn't appear in Console for up to 30 seconds.

  To mitigate this, we've come up with a counter-measure: any user request to the topic list will ALWAYS make one *AdminClient* call to Kafka: `listTopics`.

  It’s cheap, simple and will only return the topic names. So, when topics are listed in Console, 99% of the time Indexing will serve all the topics with all the columns (name, partitions, count, size, etc.) and 1% of the time Indexing will serve most topics except for one or two **not-indexed yet** where only the name will be available.
</Accordion>