Skip to main content
Preview functionality: Insights is currently a preview feature and is subject to change as we continue working on it.
Cost control is one of the sections in the Insights dashboard. It helps platform teams identify topics that may be consuming unnecessary storage costs, enabling data-driven decisions about topic cleanup and retention policies.

​
Overview

The Cost Control section displays three categories of topics that may represent storage optimization opportunities:
  • Empty topics - Topics with no messages that consume metadata overhead
  • Stale topics - Topics with no writes in a configured time period (default: 7+ days)
  • Tiny topics - Topics with minimal data volume relative to their partition overhead
Each category shows the number of affected topics, allowing you to quickly assess the scope of potential cost savings. Individual topics are listed with key metadata including idle time in days, and partition count. The dashboard includes an Export button to download the complete list as CSV for stakeholder review, and a Full list toggle to expand or collapse the detailed view of all topics in each category.

​
Empty topics

​
What it shows

The Empty topics category displays topics that contain zero messages. These topics have been created but never received any data, or have had all their messages deleted through retention policies or manual deletion. For each empty topic, the dashboard shows:
  • Topic name - The full name of the empty topic
  • Idle time in days - How long the topic has remained empty
  • Partition count - Number of partitions allocated to the topic

​
Why it matters

Empty topics consume cluster resources even without data:
  • Each topic requires metadata storage in ZooKeeper or KRaft controller
  • Cloud-managed Kafka services often charge per partition regardless of data volume
  • Each partition consumes memory for metadata and buffers
  • Empty topics clutter management interfaces and complicate operations
Empty topics are common in development and staging environments where topics are frequently created for testing purposes but not cleaned up afterward.

​
When to delete empty topics

Good candidates for deletion:
  • Topics created for abandoned projects or experiments
  • Temporary topics used for testing or debugging
  • Topics created by mistake with typos in the name
  • Legacy topics from deprecated applications
  • Topics created automatically by frameworks but never used
Reasons to keep empty topics:
  • Topics pre-created for upcoming features or applications
  • Topics used for manual testing that will receive data soon
  • Topics serving as placeholders for future data pipelines
  • Topics referenced in consumer group configurations awaiting data
  • Topics used as dead letter queues that only receive messages in error scenarios

​
Actions to take

Before taking any action, verify the topic is truly unused by checking consumer groups, producer configurations, ownership and documentation.
Deleting a topic is permanent and cannot be undone. All partition metadata and configuration will be lost.
To delete an empty topic in Conduktor Console:
1

Delete from the list

In the Cost Control section, click the … menu next to the topic in the empty topics list and select Delete topic.
2

Confirm deletion

In the confirmation modal, type DELETE to confirm and click the delete button.
Topic deletion can take up to 5 minutes to be reflected in the dashboard. After deletion, the topic will disappear from the topics list. Consumer groups that were subscribed will need to be updated to remove the topic from their subscriptions.
To verify topic details before deletion (consumer groups, active consumers, monitoring graphs), click the topic name to navigate to the Topics page where you can review the Overview, Consumer groups and Monitoring tabs.

​
Stale topics

​
What it shows

The Stale topics category displays topics that have received no write activity for 7+ days. For each stale topic, the dashboard shows:
  • Topic name - The full name of the inactive topic
  • Idle time in days - Number of days since the last message was produced
  • Partition count - Number of partitions consuming resources

​
Why it matters

Stale topics with no recent writes often indicate abandoned data pipelines from shut down applications, failed producers or cancelled projects. These topics continue consuming storage through retention policies, occupy memory buffers and add operational overhead for monitoring, backups and capacity planning.
A topic with no writes may still have active consumers reading historical data. Stale write activity does not necessarily mean the topic is unused or safe to delete.

​
How to evaluate stale topics

Investigation checklist:
  1. Check consumer groups - Are any consumers actively reading from the topic?
  2. Review consumer lag - Is there unconsumed data that applications still need?
  3. Verify business requirements - Is the historical data needed for compliance or analytics?
  4. Contact topic owners - Do the application teams still need this topic?
  5. Check data lineage - Do downstream systems depend on this data?
Topics that are safe to delete:
  • No active consumer groups subscribed
  • All consumers show zero lag (all data has been consumed)
  • Application owners confirm the topic is no longer needed
  • No compliance requirements for data retention
  • Historical data has been archived to long-term storage
Topics to keep:
  • Active consumers are still reading historical data
  • Data is required for compliance or audit purposes
  • Topic serves as a source of truth for event sourcing
  • Downstream analytics or reporting systems depend on the data
  • Occasional writes expected (for example, infrequent events)

​
Actions to take

  • Archive before deletion
  • Reduce retention
  • Implement tombstones
For topics containing valuable historical data, export and archive the data before deletion.
1

Export the topic data

  1. Navigate to the topic in Console
  2. Click the Consume tab
  3. Configure the consumer to read from Beginning offset
  4. Export data to a file or external storage system
For large topics, use a dedicated archival tool or Kafka Connect to export data to object storage (S3, Azure Blob Storage) or a data lake.
2

Verify the archive

  • Confirm all messages were exported successfully
  • Validate the data format and completeness
  • Store the archive in long-term storage
  • Document the archive location and format
3

Delete the topic

Once archived, follow the deletion steps outlined in the Empty Topics section above.
Maintain a record of archived topics including the archive location, date, original message count and retention period for the archived data.

​
Tiny topics

​
What it shows

The Tiny topics category displays topics that meet all of the following criteria:
  • More than 1 partition
  • Less than 10 MB of data OR fewer than 1000 messages
  • Topic name does not end with _repartition
For each tiny topic, the dashboard shows:
  • Topic name - The full name of the undersized topic
  • Idle time in days - Age of the topic or time since creation
  • Partition count - Number of partitions (often high relative to data volume)

​
Why it matters

Topics with many partitions but little data represent inefficient resource allocation. Each partition requires memory buffers and metadata replication regardless of data volume. Cloud Kafka services often charge per partition, meaning a tiny topic with 50 partitions costs the same as a large topic with 50 partitions. High partition counts also increase controller overhead and may exhaust cluster-wide partition limits prematurely.
A common cause of tiny topics is using default partition counts (often 6-12) for topics that only receive occasional messages, such as configuration topics or low-volume event streams.

​
How to evaluate tiny topics

Keep the topic as-is if it expects significant growth, requires high parallelism, is part of a standardized policy or would require complex migration. Consider optimization if the topic has been tiny for months without growth, has arbitrary partition counts or handles low-frequency data without parallelism requirements.

​
Optimization strategies

  • Consolidate topics
  • Recreate with fewer partitions
  • Accept current state
If you have multiple tiny topics serving similar purposes, consider consolidating them into a single topic with proper categorization through message headers or keys. This reduces total partition count while maintaining logical separation.Implementation considerations:
  • Identify consolidation candidates from the same application with similar schemas
  • Design a consolidated topic structure with category or type fields
  • Update producers to use the consolidated topic with appropriate categorization
  • Update consumers to filter by category or type field
Topic consolidation changes the partition key distribution and may affect message ordering. Ensure consumers can handle messages from multiple original topics in a single stream.

​
Safety checks and best practices

Before deleting any topic, verify:
  1. Consumer groups - Check the Consumer Groups tab for active consumers or unconsumed data (lag > 0)
  2. Producer activity - Review Monitoring tab graphs to confirm zero produce activity
  3. Application references - Search codebases and deployment configs for topic references
  4. Dependencies - Verify no streaming jobs or external systems depend on the topic
Deleting a topic with unconsumed messages will result in permanent data loss for consumers that have not yet processed those messages.

​
Topic lifecycle best practices

Prevent cost control issues from the start through governance and automated ownership tracking rather than manual remediation.
Self-service is available with Conduktor Scale Plus only.
Use self-service for ownership and governance: it brings a GitOps approach to topic lifecycle management where Applications represent streaming apps or data pipelines and dictate ownership of Kafka resources. Each application has an owner (a Console Group) and provides business context to topics, making it clear who is responsible for resources from the moment of creation. Benefits for cost control:
  • Automatic ownership tracking - Topics created through self-service have clear ownership from creation, eliminating the need for manual labeling
  • Policy enforcement - Application instance policies can prevent creation of tiny topics by enforcing minimum data volume or maximum partition count requirements
  • Naming conventions - Policies enforce consistent naming that includes environment and ownership information, making it easy to identify topic owners when reviewing cost control reports
  • Lifecycle management - Clear application ownership reduces abandoned topics through better lifecycle tracking and accountability
Learn about self-service for application teams and explore self-service concepts Implement cleanup policies: Document organization-wide policies for retention periods, deletion criteria and approval processes. Schedule regular reviews of Cost Control reports (weekly for new topics, monthly for trends). Configure RBAC: Use role-based access control to enforce naming conventions, limit partition counts and require labeling at topic creation. Learn more about RBAC configuration Archive before deletion: For valuable historical data, export topics using the Consume tab or configure automated archival with Kafka Connect to S3, Azure Blob Storage or your data lake. Maintain metadata about archived topics for compliance and recovery needs.

​
Troubleshooting

Producer health checks, monitoring systems or automated tests may send periodic messages that are immediately consumed and deleted. Kafka’s log cleaner activity can also appear as writes in some monitoring systems.Check the Messages in per second graph over 7+ days to distinguish sustained activity from sporadic checks. Review the Consume tab to see actual message content.
No, topic deletion is permanent. All partition data and topic metadata are removed. Consumer group offsets are retained but the topic cannot be recreated with the same history.Always archive important data before deletion. Implement a grace period policy where topics are first marked for deletion (using labels) before actual removal. To recreate a deleted topic, create a new topic with the same name in Console and restore data from archives if available.
Kafka creates internal topics automatically: __consumer_offsets (consumer group offsets), __transaction_state (transactional state), _schemas (Schema Registry metadata) and topics matching .*-changelog or .*-repartition (Kafka Streams internals).Never delete __consumer_offsets or __transaction_state as this causes cluster-wide failures. Only delete Kafka Streams topics when the application is permanently decommissioned and all instances are shut down.
Kafka’s log cleanup process runs periodically (default: every 5 minutes). Data is deleted at segment boundaries, not individual messages, and the current active segment is never deleted even if older than retention.Wait 15-30 minutes after reducing retention, then check topic size in the Overview tab.
Check topic naming patterns, review consumer groups in the Consumer Groups tab, search code repositories and deployment configs for references, review platform logs and consult internal documentation.If ownership remains unclear, add a scheduled-for-deletion label with a future date (for example, 30 days out) and send announcements asking owners to identify themselves. After the grace period expires with no claims, proceed with deletion.Implement a policy requiring application labels at topic creation time. Use RBAC permissions to enforce labeling as a prerequisite for topic creation.
This is normal for seasonal data (holiday sales, tax season), emergency systems (disaster recovery), compliance events (audit logs) or legacy integrations (backup systems).Verify the producer is authorized, add a usage-pattern: seasonal or usage-pattern: emergency label, adjust the stale threshold (for example, 365 days) and document the activity pattern. Do not delete topics showing this pattern without understanding the business requirement.