Skip to main content
Preview functionality: Insights is currently a preview feature and is subject to change as we continue working on it.
VIP topics is one of the sections in the Insights dashboard. It identifies your most business-critical topics based on consumer group count and message volume, helping you focus attention and resources where they have maximum impact.

​
Overview

The VIP Topics section displays a health overview graph showing topics identified as important to your infrastructure. VIP topics are determined by two key metrics:
  • Consumer group count - Topics with many subscribing consumer groups (shown by bar height)
  • Message volume - Topics with high throughput and data volume (shown by color intensity)
The combination of these metrics identifies topics that are both widely used and heavily trafficked, indicating business criticality. Topics with many consumers and high message volumes represent critical data pipelines that multiple applications depend on, making them prime candidates for elevated monitoring and careful management.
VIP topics are automatically identified based on cluster-wide analysis of consumer patterns and message throughput. You can use this data-driven approach to prioritize operational focus and monitoring resources.

​
What the graph shows

​
VIP topics health overview

The bar graph visualizes your most important topics using two dimensions: Bar height represents the number of consumer groups subscribed to each topic:
  • Taller bars indicate more consumer groups depend on the topic
  • Many consumers suggest the topic provides data critical to multiple applications
  • Wide usage indicates potential for widespread impact if issues occur
Color intensity represents message volume and throughput:
  • Darker blue indicates higher message volume and throughput
  • Lighter blue indicates lower message volume
  • Message volume combined with consumer count identifies truly critical topics

​
Interpreting the visualization

Tall, dark bars - Topics with many consumers and high message volume are your most business-critical data pipelines requiring highest priority for monitoring and careful change management. Tall, light bars - Topics with many consumers but lower message volume may represent configuration or control topics that still require careful management despite lower throughput. Short, dark bars - Topics with high message volume but fewer consumers represent specialized high-throughput pipelines requiring performance optimization and capacity planning.

​
Why VIP topics matter

VIP topics represent the backbone of your data infrastructure. Multiple applications and consumer groups depend on these topics for critical business processes. Issues with VIP topics have cascading impact across downstream applications and services. High traffic makes VIP topics susceptible to performance problems, while configuration issues create data loss risk. Changes to VIP topics require coordination across teams and careful testing. Poor replication factors, insufficient partitions or inadequate retention can cause widespread application failures.
Issues with VIP topics have multiplied impact. A single misconfiguration or outage can affect dozens of applications and business processes simultaneously.
VIP topics drive infrastructure capacity planning and justify investment in comprehensive monitoring, stricter alerting thresholds and dedicated operational procedures.

​
Actions to take in Console

For each VIP topic identified in the dashboard, verify and optimize configurations across multiple areas. Expand each section to review specific actions and guidance.
Validate that VIP topics have production-grade settings:
1

Navigate to the topic

Go to Topics from the main menu and select a VIP topic identified from the Insights dashboard.
2

Open the Configuration tab

Click the Configuration tab to review current settings.
3

Verify critical settings

Check the following configurations:Replication factor:
  • Has to be at least 3 for production VIP topics
  • Provides fault tolerance for up to one broker failure
  • Ensures data durability and availability
Retention policy:
  • retention.ms - Time-based retention appropriate for business needs
  • retention.bytes - Size-based retention per partition if applicable
  • Consider longer retention for VIP topics to support late-arriving consumers
Partition count:
  • Sufficient partitions for current and projected throughput
  • Ideally a multiple of broker count for even distribution
  • Adequate parallelism for all consumer groups
Cleanup policy:
  • delete - For time-series or event data
  • compact - For state or changelog topics
  • Appropriate for the data model and consumption patterns
VIP topics should never have replication factor of 1 or 2 in production environments. If the Risk Analysis section identifies replication issues for VIP topics, address these immediately.
Implement proactive monitoring to detect issues before they impact consuming applications:
1

Navigate to the VIP topic

Go to Topics and select the VIP topic you want to monitor. Click the Alerts tab on the topic detail page.
2

Configure consumer lag alerts

Set up alerts for consumer group lag thresholds:
  • Define acceptable lag limits based on business requirements
  • Use stricter thresholds for VIP topics than standard topics
  • Alert on both absolute lag (message count) and time-based lag
For VIP topics, consider alerting when any consumer group exceeds 1000 messages of lag or 5 minutes of time-based lag, rather than the default thresholds used for standard topics.
3

Create under-replicated partition alerts

Configure alerts for replication issues:
  • Alert immediately if any partitions become under-replicated
  • Under-replicated partitions indicate broker issues or failures
  • Critical for VIP topics where data loss risk is unacceptable
4

Set disk usage alerts

Monitor storage consumption for VIP topics:
  • Alert on rapid growth that could cause disk space issues
  • Track retention effectiveness
  • Plan capacity expansions before reaching limits
5

Configure throughput alerts

Track unusual traffic patterns:
  • Alert on sudden drops in produce rate (possible producer failure)
  • Alert on unexpected spikes that could cause performance issues
  • Baseline normal throughput to detect anomalies
Find out about setting up alerts.
Ensure VIP topics have clear ownership and governance from the start through self-service workflows rather than manual tracking.
Self-service is available with Conduktor Scale Plus only.
Use self-service for automatic ownershipConduktor’s self-service framework provides a GitOps approach to topic lifecycle management where Applications dictate ownership of Kafka resources. When VIP topics are managed through self-service:
  • Automatic ownership tracking - applications define owners and business context at topic creation
  • Clear accountability - Console automatically assigns ownership to application teams
  • Governance enforcement - policies ensure VIP topics meet configuration standards (RF=3, appropriate retention)
  • Business context preserved - application definitions maintain documentation about purpose, dependencies and SLAs
Benefits for VIP topics:
  • When issues occur, the right teams are contacted immediately through defined ownership
  • Configuration changes follow approval workflows specific to business-critical topics
  • Governance policies prevent VIP topics from being created with suboptimal settings
  • Topic purpose and dependencies are documented in application definitions
Implement self-service for governed topic creationLearn about self-service concepts and applications
Ensure VIP topics have appropriate security configurations:
1

Verify RBAC permissions

Navigate to Settings > RBAC and review permissions for the VIP topic:
  • Limit producer permissions to authorized applications only
  • Restrict consumer access to approved teams and services
  • Require elevated permissions for configuration changes
  • Audit permissions regularly for VIP topics
Learn more about RBAC
2

Review security policies

For VIP topics containing sensitive data:
  • Verify encryption in transit (SSL/TLS) is enforced
  • Confirm ACLs or RBAC rules restrict access appropriately
  • Check for data masking or encryption requirements
  • Ensure compliance with organizational security policies
Track all consumer groups subscribed to VIP topics:
1

Navigate to the VIP topic

Go to Topics and select the VIP topic you want to analyze. Click the Consumer Groups tab on the topic detail page.
2

Review consumer group metrics

For each consumer group, examine:
  • Lag - Current lag per partition and total lag
  • State - Active consumers or empty groups
  • Members - Number of active consumer instances
  • Commit frequency - How often consumers commit offsets
Consumer lag on VIP topics requires immediate investigation. High lag indicates consumers cannot keep up with message volume, which may lead to processing delays, memory issues or timeout errors.
3

Identify problematic consumers

Look for warning signs:
  • Consistently high or growing lag
  • Consumers that frequently rebalance
  • Groups with zero active members but uncommitted messages
  • Uneven lag distribution across partitions
Contact the owning teams for consumer groups with persistent issues.
Ensure optimal partition allocation across brokers:
1

Navigate to the VIP topic

Go to Topics and select the VIP topic you want to review. Click the Partitions tab on the topic detail page.
2

Analyze partition distribution

Review the Per broker view to verify:
  • Partitions are evenly distributed across all brokers
  • Leadership is balanced (no single broker leads most partitions)
  • No brokers are excluded from the topic
  • Replica assignments provide proper fault tolerance
The Risk analysis section identifies partition distribution issues. For VIP topics, prioritize resolving distribution problems to avoid broker hotspots and performance bottlenecks.
3

Check for partition skew

Switch to Per partition view and compare:
  • Partition sizes across all partitions
  • Message counts per partition
  • Offset ranges (begin offset to end offset)
Significant differences indicate partition skew, which can cause uneven consumer load and processing delays.Learn how to address partition skew
Monitor VIP topic performance over time:
1

Navigate to the VIP topic

Go to Topics and select the VIP topic you want to monitor. Click the Monitoring tab on the topic detail page.
2

Review produce metrics

Analyze produce patterns:
  • Messages in per second - Produce rate over time
  • Bytes in per second - Data volume throughput
  • Look for unusual spikes, drops or patterns
  • Establish baseline performance for capacity planning
3

Review consume metrics

Analyze consume patterns:
  • Messages out per second - Consume rate across all consumer groups
  • Bytes out per second - Data volume being consumed
  • Compare consume rate to produce rate to identify accumulation
4

Identify trends and anomalies

Use the metrics to:
  • Detect gradual throughput increases requiring capacity planning
  • Identify time-of-day or day-of-week patterns
  • Spot sudden changes that may indicate application issues
  • Plan for peak traffic periods and scaling needs

​
Troubleshooting

VIP topic identification is based on both message volume and consumer count. A topic may appear as VIP due to high consumer count (configuration topics read by many applications, control plane topics, event notification topics) or because it’s business-critical despite lower volume (compliance, regulatory or SLA requirements). Review the specific topic context to verify the VIP designation is appropriate for your organization’s needs.
No. VIP topics should share minimum standards (replication factor of 3, configured alerting, documented ownership, proper RBAC permissions) but configurations should be tailored to each topic’s requirements.Partition count, retention, cleanup policy, compression and security settings should be based on specific throughput needs, data lifecycle requirements and sensitivity classification. For example, a high-volume transaction topic may need 50 partitions and 7-day retention, while a configuration topic may need 3 partitions and 90-day retention.
Document organization-wide minimum standards for VIP topics while allowing flexibility for topic-specific optimizations.
A poor health score indicates configuration or operational issues. Address systematically:
  1. Identify issues - Check All recommendations in VIP topics section and review the Risk analysis section
  2. Prioritize by impact - Critical (replication factor less than 3), High (under-replicated partitions), Medium (partition skew), Low (suboptimal configurations)
  3. Take action - For replication issues, follow replication remediation steps. For partition problems, follow partition troubleshooting steps. For performance issues, review consumer lag and throughput metrics
  4. Verify improvement - Monitor the health score after remediation to confirm resolution
Poor health scores on VIP topics require immediate investigation and remediation.