Cortex deployment for Conduktor Console monitoring

To enable monitoring via , you can deploy and configure the conduktor-console-cortex component or use your own compatible monitoring solution (since Console v1.38). We support , Mimir and Prometheus as alternative solutions. This guide focuses on deploying the conduktor-console-cortex component, which is a preconfigured Cortex that includes Prometheus and Alertmanager.

Deploy and configure Cortex

The only required property is CDK_CONSOLE-URL, everything else is related to storage for the metrics. By default, data will be stored in /var/conduktor/monitoring inside the running image. You can mount a volume on this folder to keep metrics data between updates. Alternatively, you can use the storage parameters described below to store the data using either s3, gcs, azure or swift.

Environment variable	Description	Mandatory	Type	Default	Since
`CDK_CONSOLEURL`	Console URL and port (example: `"http://conduktor-console:8080"`).	true	string	∅	`1.18.0`
`CDK_SCRAPER_SKIPSSLCHECK`	Disable TLS check when scraping metrics from Console.	false	bool	`false`	`1.18.2`
`CDK_SCRAPER_CAFILE`	Path to CA certificate file inside the container to perform TLS check when scraping metrics from Console.	false	string	∅	`1.18.2`
`CDK_SCRAPER_BASICAUTH_EMAIL`	Email for basic auth when scraping metrics from Console. Has to match `CDK_MONITORING_BASICAUTH_EMAIL` on Console.	false	string	∅	`1.42.0`
`CDK_SCRAPER_BASICAUTH_PASSWORD`	Password for basic auth when scraping metrics from Console. Has to match `CDK_MONITORING_BASICAUTH_PASSWORD` on Console.	false	string	∅	`1.42.0`
S3
`CDK_MONITORING_STORAGE_S3_ENDPOINT`	S3 storage endpoint.	false	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_S3_REGION`	S3 storage region.	false	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_S3_BUCKET`	S3 storage bucket name.	true	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_S3_INSECURE`	S3 storage SSL/TLS check flag.	false	bool	`false`	`1.18.0`
`CDK_MONITORING_STORAGE_S3_ACCESSKEYID`	Access key ID of an AWS IAM identity for monitoring to upload logs to S3. If set to false, and if you are running on a Kubernetes deployment, Monitoring can attempt to leverage AWS IRSA for the Pod’s service account when connecting to S3.	false	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_S3_SECRETACCESSKEY`	Secret access key of an AWS IAM identity for monitoring to upload logs to S3. If set to false, and if you are running on a Kubernetes deployment, Monitoring can attempt to leverage AWS IRSA for the Pod’s service account when connecting to S3.	false	string	∅	`1.18.0`
GCS
`CDK_MONITORING_STORAGE_GCS_BUCKETNAME`	GCS storage bucket name.	true	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_GCS_SERVICEACCOUNT`	GCS storage service account JSON content.	true	string	∅	`1.18.0`
Azure
`CDK_MONITORING_STORAGE_AZURE_ACCOUNTNAME`	Azure storage account name.	true	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_AZURE_ACCOUNTKEY`	Azure storage account key.	true	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_AZURE_CONTAINERNAME`	Azure storage container name.	true	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_AZURE_ENDPOINTSUFFIX`	Azure storage endpoint suffix.	false	string	`blob.core.windows.net`	`1.18.0`
Swift
`CDK_MONITORING_STORAGE_SWIFT_AUTHURL`	Swift storage authentication URL.	true	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_SWIFT_PASSWORD`	Swift storage user password.	true	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_SWIFT_CONTAINERNAME`	Swift storage container name.	true	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_SWIFT_USERID`	Swift storage user ID.	false	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_SWIFT_USERNAME`	Swift storage user name.	false	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_SWIFT_USERDOMAINNAME`	Swift storage user domain name.	false	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_SWIFT_USERDOMAINID`	Swift storage user domain ID.	false	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_SWIFT_DOMAINID`	Swift storage user domain ID.	false	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_SWIFT_DOMAINNAME`	Swift storage user domain name.	false	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_SWIFT_PROJECTID`	Swift storage project ID.	false	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_SWIFT_REGIONNAME`	Swift storage region name.	false	string	∅	`1.18.0`
Logs
`CORTEX_ROOT_LOG_LEVEL`	Cortex log level.	false	string	`info`	`1.18.0`
`CORTEX_ALERT_ROOT_LOG_LEVEL`	Alert manager log level.	false	string	`info`	`1.18.0`
`PROMETHEUS_ROOT_LOG_LEVEL`	Prometheus log level.	false	string	`info`	`1.18.0`
Retention configuration
`PROMETHEUS_RETENTION_SIZE`	Prometheus `storage.tsdb.retention.size` startup argument that defines retention size.	false	string	`5MB`	`1.37.0`
`PROMETHEUS_RETENTION_TIME`	Prometheus `storage.tsdb.retention.time` startup argument that defines retention time.	false	string	`30m`	`1.37.0`
`PROMETHEUS_BLOCK_DURATION`	Prometheus `storage.tsdb.min-block-duration` and `storage.tsdb.max-block-duration` startup argument format “min,max”.	false	string	`5m,30m`	`1.37.0`
`PROMETHEUS_SCRAPE_INTERVAL`	Prometheus `global.scrape_interval` in configuration.	false	string	`15s`	`1.37.0`
`PROMETHEUS_EVAL_INTERVAL`	Prometheus `global.evaluation_interval` in configuration.	false	string	`15s`	`1.37.0`
`CORTEX_BLOCKS_RANGES`	Cortex block ranges configuration (`blocks_storage.tsdb.block_ranges_period`).	false	string	`1h`	`1.37.0`
`CORTEX_RETENTION_PERIOD`	Cortex local block retention before flushing to external storage (`blocks_storage.tsdb.retention_period`).	false	string	`3h`	`1.37.0`
`CORTEX_COMPACTOR_BLOCKS_RANGES`	Cortex compactor blocks sizes configuration (`compactor.block_ranges`).	false	string	`2h,12h,24h`	`1.37.0`
`CORTEX_COMPACTOR_INTERVAL`	Cortex compactor run interval configuration (`compactor.compaction_interval`).	false	string	`30m`	`1.37.0`
`CORTEX_COMPACTOR_DELETION_DELAY`	Cortex compactor block deletion delay configuration (`compactor.deletion_delay`).	false	string	`1h`	`1.37.0`
`CORTEX_COMPACTOR_RETENTION_PERIOD`	Cortex compactor block retention configuration (`limits_config.compactor_blocks_retention_period`).	false	string	`45d`	`1.46.0`
`CORTEX_CLEANER_INTERVAL`	Configure how often cleanup task is running (only use when no external object storage is configured).	false	string	`6h`	`1.37.0`
`CORTEX_CLEANER_RETENTION`	Configure the retention duration in the “file system bucket” (`/var/conduktor/monitoring/data/cortex/tsdb-backend/`).	false	string	`30d`	`1.37.0`
Override configuration
`PROMETHEUS_OVERRIDE_CONFIG_FILE`	Location to yaml config file to patch Prometheus configuration with.	false	string	`/opt/override-configs/prometheus.yaml`	`1.19.0`
`CORTEX_OVERRIDE_CONFIG_FILE`	Location to yaml config file to patch Cortex configuration with.	false	string	`/opt/override-configs/cortex.yaml`	`1.19.0`
`ALERTMANAGER_OVERRIDE_CONFIG_FILE`	Location to yaml config file to patch Cortex alert manager configuration with.	false	string	`/opt/override-configs/alertmanager.yaml`	`1.37.0`
`PROMETHEUS_EXTRA_RUN_ARGS`	Extra run arguments for Prometheus.	false	string	`""`	`1.37.0`
`CORTEX_EXTRA_RUN_ARGS`	Extra run arguments for Cortex.	false	string	`""`	`1.37.0`
`CORTEX_ALERT_EXTRA_RUN_ARGS`	Extra run arguments for Cortex alert manager.	false	string	`""`	`1.37.0`

Sample configuration

In a Docker Compose it may look like the following:

version: '3.8'
services:
  conduktor-console:
    image: conduktor/conduktor-console
    ports:
      - "8080:8080"
    environment:
      CDK_MONITORING_CORTEX-URL: http://conduktor-monitoring:9009/
      CDK_MONITORING_ALERT-MANAGER-URL: http://conduktor-monitoring:9010/
      CDK_MONITORING_CALLBACK-URL: http://conduktor-console:8080/monitoring/api/
      CDK_MONITORING_NOTIFICATIONS-CALLBACK-URL: http://localhost:8080
  conduktor-monitoring:
    image: conduktor/conduktor-console-cortex
    ports:
      - "9009:9009" # cortex api
      - "9010:9010" # alertmanager api
      - "9090:9090" # prometheus api
    environment:
      CDK_CONSOLE-URL: "http://conduktor-console:8080"

Configure basic auth for metric scraping

Since Console v1.42.0, you can configure basic auth credentials to protect access to Console metric endpoints. When configured, the Cortex component has to provide these credentials when scraping metrics from the /api/monitoring/metrics and /monitoring/metrics/ endpoints.

Basic auth is optional in Console v1.42.0 and no configuration changes are required for existing deployments.

Enable basic auth on Console

Add these environment variables to Console:

environment:
  CDK_MONITORING_BASICAUTH_EMAIL: "metrics@example.com"
  CDK_MONITORING_BASICAUTH_PASSWORD: "Hello123!"

The password has to contain at least 8 characters, including one uppercase letter, one lowercase letter, one number and one special symbol.

Configure Cortex to authenticate

When basic auth is enabled in Console, configure Cortex to authenticate when scraping metrics by adding the same credentials:

environment:
  CDK_SCRAPER_BASICAUTH_EMAIL: "metrics@example.com"
  CDK_SCRAPER_BASICAUTH_PASSWORD: "Hello123!"

Docker Compose
Kubernetes

services:
  conduktor-console:
    image: conduktor/conduktor-console
    environment:
      CDK_MONITORING_BASICAUTH_EMAIL: "metrics@example.com"
      CDK_MONITORING_BASICAUTH_PASSWORD: "Hello123!"
      # Other configuration...

  conduktor-monitoring:
    image: conduktor/conduktor-console-cortex
    environment:
      CDK_CONSOLE-URL: "http://conduktor-console:8080"
      CDK_SCRAPER_BASICAUTH_EMAIL: "metrics@example.com"
      CDK_SCRAPER_BASICAUTH_PASSWORD: "Hello123!"

Add the environment variables to your Helm chart values:

platformCortex:
  env:
    CDK_SCRAPER_BASICAUTH_EMAIL: "metrics@example.com"
    CDK_SCRAPER_BASICAUTH_PASSWORD: "Hello123!"

For sensitive values, use a secret instead:

platformCortex:
  envFrom:
    - secretRef:
        name: cortex-auth-credentials

Metrics retention

Local storage retention

Without external storage metrics are stored in /var/conduktor/monitoring/data/cortex/tsdb-backend/. Default retention lifecycle:

Prometheus creates blocks every 5-30 minutes (PROMETHEUS_BLOCK_DURATION: 5m,30m)
Cortex keeps blocks for 3 hours (CORTEX_RETENTION_PERIOD: 3h)
Cortex compacts blocks every 30 minutes (CORTEX_COMPACTOR_INTERVAL: 30m)
Compacted blocks are deleted 1 hour after compaction (CORTEX_COMPACTOR_DELETION_DELAY: 1h)
Cleanup runs every 6 hours (CORTEX_CLEANER_INTERVAL: 6h)
Final retention: 30 days (CORTEX_CLEANER_RETENTION: 30d)

All metrics are lost when the container restarts or is recreated. Mount a persistent volume to /var/conduktor/monitoring or use external storage.

External storage retention

With external storage (S3, GCS, Azure, PVC) configured, metrics persist across container restarts. CORTEX_RETENTION_PERIOD (default: 3 hours) controls how long the ingester buffers data locally before flushing it to external storage. CORTEX_COMPACTOR_RETENTION_PERIOD (default: 45 days) controls how long the compactor retains blocks in external storage before deleting them. The CORTEX_CLEANER_* variables are ignored when external storage is configured. Use CORTEX_COMPACTOR_RETENTION_PERIOD to configure how long the compactor keeps blocks in external storage, or configure your storage provider’s lifecycle policy as a fallback. Example: Configure 7-day retention with S3:

environment:
  CDK_MONITORING_STORAGE_S3_BUCKET: my-metrics-bucket
  CDK_MONITORING_STORAGE_S3_REGION: us-east-1
  CORTEX_COMPACTOR_RETENTION_PERIOD: 7d

For production, external storage is required to avoid data loss. Configure external storage using the storage environment variables in the table above.

Cortex maintains a cache in local storage for every block uploaded to external storage. Without a retention period (CORTEX_COMPACTOR_RETENTION_PERIOD) set, blocks accumulate indefinitely in both local and external storage. This is why it defaults to 45 days. Therefore, do not set this to 0s (disabled) or too high value (e.g. 365d).

Storage sizing

Storage requirements vary based on your cluster count, topic count, scrape interval and retention period. Example: A deployment with 5 clusters and 500 topics using default settings (15s scrape interval) with 30-day retention might use approximately 85 GB. Monitor actual usage to determine your requirements:

# Local storage
docker exec conduktor-monitoring du -sh /var/conduktor/monitoring/data

# Kubernetes PVC
kubectl exec -it <cortex-pod> -- df -h /var/conduktor/monitoring

Overriding configuration

Overriding with YAML

Cortex

Cortex configuration can be patched by mounting a YAML file to path /opt/override-configs/cortex.yaml. To set an alternative path, use the CORTEX_OVERRIDE_CONFIG_FILE environment variable. Cortex alert manager configuration can also be patched by mounting a YAML file to path /opt/override-configs/alertmanager.yaml. To set an alternative path, use the ALERTMANAGER_OVERRIDE_CONFIG_FILE environment variable. For example, create a file cortex.yaml add in only your overrides:

limits:
  ingestion_rate: 50000
  max_series_per_metric: 100000

Mount to /opt/override-configs/cortex.yaml.
Spin up the container.
Exec into the container and confirm the contents, replace 2 with the number of lines of override you wish to see or remove grep to get the whole file: cat /var/conduktor/configs/monitoring-cortex.yaml | grep limits -A2.

You should see a similar entry to the below in the opening logs:

INFO monitoring_entrypoint - Patch "/var/conduktor/configs/monitoring-cortex.yaml" configuration with "/opt/override-configs/cortex.yaml" fragment

Prometheus

Before Console v1.37.0, Prometheus configuration was overridden using the replace strategy. Since v1.37.0 it’s changed to the patch (YAML merging) strategy.

Prometheus configuration can be also patched by mounting a YAML file to path /opt/override-configs/prometheus.yaml. To set an alternative path, use the PROMETHEUS_OVERRIDE_CONFIG_FILE environment variable. You should see a similar entry to the below in the opening logs:

INFO monitoring_entrypoint - Replace "/var/conduktor/configs/monitoring-prometheus.yaml" configuration with "/opt/override-configs/prometheus.yaml" content

Overriding with ConfigMap

If you’re deploying Cortex using our Helm charts you may expand the input with a custom ConfigMap for patching configuration such as retention time within Cortex. You can also replace the Prometheus config as described below.

apiVersion: v1
kind: ConfigMap
metadata:
  name: conduktor-console-cortex-config
  labels:
    app.kubernetes.io/name: console
    app.kubernetes.io/instance: conduktor
    app.kubernetes.io/component: conduktor-platform-cortex
data:
  cortex.yaml: |
    blocks_storage:
      tsdb:
        retention_period: 24h
  prometheus.yaml: |
    global:
      scrape_interval: 15s
      evaluation_interval: 15s

On chart values.yaml :

platformCortex:
  extraVolumes:
    - name: cortex-config-override
      configMap:
        name: conduktor-console-cortex-config
  extraVolumeMounts:
        - name: cortex-config-override
          mountPath: /opt/override-configs/cortex.yaml
          subPath: cortex.yaml
        - name: cortex-config-override
          mountPath: /opt/override-configs/prometheus.yaml
          subPath: prometheus.yaml

Monitoring endpoints

Monitoring is designed to be interacted with through the API endpoints by Console (not users). Therefore, ingress isn’t available externally and you shouldn’t set it up because an authentication mechanism doesn’t exist.

Cortex endpoint

/ready on port 9009 If Cortex is running, returns the 200 status with ready.

cURL example

curl -s "http://localhost:9009/ready"

Alertmanager endpoint

/ready on port 9010 If Alertmanager is running, returns the 200 status with ready.

cURL example

curl -s "http://localhost:9010/ready"

Prometheus endpoint

/-/healthy on port 9090 If Prometheus is running, returns the 200 status with Prometheus Server is Healthy..

cURL example

curl -s "http://localhost:9090/-/healthy"

Troubleshoot

No metrics in the monitoring page

Go to http://localhost:9090/targets to see Prometheus scraping target status. If it fails, check that you can query metrics endpoint from conduktor-console-cortex container. You might also have to configure CDK_SCRAPER_SKIPSSLCHECK or CDK_SCRAPER_CAFILE if conduktor-console is configured with TLS termination.

Custom CA certificates for S3-like storage

The CDK_SCRAPER_CAFILE environment variable only configures the CA certificate for the Scraper component (which scrapes metrics from Console). It does not configure the CA certificate for S3-like storage backends.

If you use an S3-compatible service (like MinIO, Ceph or other S3-like services) with custom certificates, you have two options:

Option 1: Disable SSL verification

To skip TLS hostname verification with S3-like services, override the Cortex configuration with YAML by creating a YAML file with these contents:

blocks_storage:
  s3:
    http:
      insecure_skip_verify: true
ruler_storage:
  s3:
    http:
      insecure_skip_verify: true

Mount this file to /opt/override-configs/cortex.yaml (or use the CORTEX_OVERRIDE_CONFIG_FILE environment variable to specify an alternative path).

Disabling SSL verification removes certificate validation and is not recommended for production environments. Use this only for testing or non-production deployments.

Option 2: Contact Conduktor support

For production environments requiring custom CA certificates with S3-compatible storage, contact Conduktor support for guidance on configuring certificate trust for Cortex components.

Conduktor in production

Deploy Console

Deploy Schema Registry Proxy

Deploy Gateway

Initial setup

Cortex deployment for Conduktor Console monitoring

Deploy and configure Cortex

Sample configuration

Configure basic auth for metric scraping

Enable basic auth on Console

Configure Cortex to authenticate

Metrics retention

Local storage retention

External storage retention

Storage sizing

Overriding configuration

Overriding with YAML

Cortex

Prometheus

Overriding with ConfigMap

Monitoring endpoints

Cortex endpoint

Alertmanager endpoint

Prometheus endpoint

Troubleshoot

No metrics in the monitoring page

Custom CA certificates for S3-like storage

Option 1: Disable SSL verification

Option 2: Contact Conduktor support

​Deploy and configure Cortex

​Sample configuration

​Configure basic auth for metric scraping

​Enable basic auth on Console

​Configure Cortex to authenticate

​Metrics retention

​Local storage retention

​External storage retention

​Storage sizing

​Overriding configuration

​Overriding with YAML

​Cortex

​Prometheus

​Overriding with ConfigMap

​Monitoring endpoints

​Cortex endpoint

​Alertmanager endpoint

​Prometheus endpoint

​Troubleshoot

​No metrics in the monitoring page

​Custom CA certificates for S3-like storage

​Option 1: Disable SSL verification

​Option 2: Contact Conduktor support

​Related resources

Deploy and configure Cortex

Sample configuration

Configure basic auth for metric scraping

Enable basic auth on Console

Configure Cortex to authenticate

Metrics retention

Local storage retention

External storage retention

Storage sizing

Overriding configuration

Overriding with YAML

Cortex

Prometheus

Overriding with ConfigMap

Monitoring endpoints

Cortex endpoint

Alertmanager endpoint

Prometheus endpoint

Troubleshoot

No metrics in the monitoring page

Custom CA certificates for S3-like storage

Option 1: Disable SSL verification

Option 2: Contact Conduktor support

Related resources