> ## Documentation Index > Fetch the complete documentation index at: https://docs.conduktor.io/llms.txt > Use this file to discover all available pages before exploring further. # Cortex deployment for Conduktor Console monitoring > Deploy conduktor-console-cortex for Kafka monitoring in Conduktor Console. Includes Prometheus and Alertmanager. To enable monitoring via Console, you can deploy and configure the `conduktor-console-cortex` component or use your own compatible monitoring solution (since Console v1.38). We support **Cortex**, **Mimir** and **Prometheus** as alternative solutions. This guide focuses on deploying the `conduktor-console-cortex` component, which is a preconfigured Cortex that includes Prometheus and Alertmanager. ## Deploy and configure Cortex The only required property is `CDK_CONSOLE-URL`, everything else is related to storage for the metrics. By default, data will be stored in `/var/conduktor/monitoring` inside the running image. You can mount a volume on this folder to keep metrics data between updates. Alternatively, you can use the storage parameters described below to store the data using either `s3`, `gcs`, `azure` or `swift`. | Environment variable | Description | Mandatory | Type | Default | Since | | --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | ------ | ----------------------------------------- | -------- | | `CDK_CONSOLEURL` | Console URL and port (example: `"http://conduktor-console:8080"`). | true | string | ∅ | `1.18.0` | | `CDK_SCRAPER_SKIPSSLCHECK` | Disable TLS check when scraping metrics from Console. | false | bool | `false` | `1.18.2` | | `CDK_SCRAPER_CAFILE` | Path to CA certificate file inside the container to perform TLS check when scraping metrics from Console. | false | string | ∅ | `1.18.2` | | `CDK_SCRAPER_BASICAUTH_EMAIL` | Email for basic auth when scraping metrics from Console. Has to match `CDK_MONITORING_BASICAUTH_EMAIL` on Console. | false | string | ∅ | `1.42.0` | | `CDK_SCRAPER_BASICAUTH_PASSWORD` | Password for basic auth when scraping metrics from Console. Has to match `CDK_MONITORING_BASICAUTH_PASSWORD` on Console. | false | string | ∅ | `1.42.0` | | **S3** | | | | | | | `CDK_MONITORING_STORAGE_S3_ENDPOINT` | S3 storage endpoint. | false | string | ∅ | `1.18.0` | | `CDK_MONITORING_STORAGE_S3_REGION` | S3 storage region. | false | string | ∅ | `1.18.0` | | `CDK_MONITORING_STORAGE_S3_BUCKET` | S3 storage bucket name. | true | string | ∅ | `1.18.0` | | `CDK_MONITORING_STORAGE_S3_INSECURE` | S3 storage SSL/TLS check flag. | false | bool | `false` | `1.18.0` | | `CDK_MONITORING_STORAGE_S3_ACCESSKEYID` | Access key ID of an AWS IAM identity for monitoring to upload logs to S3. If set to false, and if you are running on a Kubernetes deployment, Monitoring can attempt to leverage AWS IRSA for the Pod’s service account when connecting to S3. | false | string | ∅ | `1.18.0` | | `CDK_MONITORING_STORAGE_S3_SECRETACCESSKEY` | Secret access key of an AWS IAM identity for monitoring to upload logs to S3. If set to false, and if you are running on a Kubernetes deployment, Monitoring can attempt to leverage AWS IRSA for the Pod’s service account when connecting to S3. | false | string | ∅ | `1.18.0` | | **GCS** | | | | | | | `CDK_MONITORING_STORAGE_GCS_BUCKETNAME` | GCS storage bucket name. | true | string | ∅ | `1.18.0` | | `CDK_MONITORING_STORAGE_GCS_SERVICEACCOUNT` | GCS storage service account JSON content. | true | string | ∅ | `1.18.0` | | **Azure** | | | | | | | `CDK_MONITORING_STORAGE_AZURE_ACCOUNTNAME` | Azure storage account name. | true | string | ∅ | `1.18.0` | | `CDK_MONITORING_STORAGE_AZURE_ACCOUNTKEY` | Azure storage account key. | true | string | ∅ | `1.18.0` | | `CDK_MONITORING_STORAGE_AZURE_CONTAINERNAME` | Azure storage container name. | true | string | ∅ | `1.18.0` | | `CDK_MONITORING_STORAGE_AZURE_ENDPOINTSUFFIX` | Azure storage endpoint suffix. | false | string | `blob.core.windows.net` | `1.18.0` | | **Swift** | | | | | | | `CDK_MONITORING_STORAGE_SWIFT_AUTHURL` | Swift storage authentication URL. | true | string | ∅ | `1.18.0` | | `CDK_MONITORING_STORAGE_SWIFT_PASSWORD` | Swift storage user password. | true | string | ∅ | `1.18.0` | | `CDK_MONITORING_STORAGE_SWIFT_CONTAINERNAME` | Swift storage container name. | true | string | ∅ | `1.18.0` | | `CDK_MONITORING_STORAGE_SWIFT_USERID` | Swift storage user ID. | false | string | ∅ | `1.18.0` | | `CDK_MONITORING_STORAGE_SWIFT_USERNAME` | Swift storage user name. | false | string | ∅ | `1.18.0` | | `CDK_MONITORING_STORAGE_SWIFT_USERDOMAINNAME` | Swift storage user domain name. | false | string | ∅ | `1.18.0` | | `CDK_MONITORING_STORAGE_SWIFT_USERDOMAINID` | Swift storage user domain ID. | false | string | ∅ | `1.18.0` | | `CDK_MONITORING_STORAGE_SWIFT_DOMAINID` | Swift storage user domain ID. | false | string | ∅ | `1.18.0` | | `CDK_MONITORING_STORAGE_SWIFT_DOMAINNAME` | Swift storage user domain name. | false | string | ∅ | `1.18.0` | | `CDK_MONITORING_STORAGE_SWIFT_PROJECTID` | Swift storage project ID. | false | string | ∅ | `1.18.0` | | `CDK_MONITORING_STORAGE_SWIFT_REGIONNAME` | Swift storage region name. | false | string | ∅ | `1.18.0` | | **Logs** | | | | | | | `CORTEX_ROOT_LOG_LEVEL` | Cortex log level. | false | string | `info` | `1.18.0` | | `CORTEX_ALERT_ROOT_LOG_LEVEL` | Alert manager log level. | false | string | `info` | `1.18.0` | | `PROMETHEUS_ROOT_LOG_LEVEL` | Prometheus log level. | false | string | `info` | `1.18.0` | | **Retention configuration** | | | | | | | `PROMETHEUS_RETENTION_SIZE` | Prometheus `storage.tsdb.retention.size` startup argument that defines retention size. | false | string | `5MB` | `1.37.0` | | `PROMETHEUS_RETENTION_TIME` | Prometheus `storage.tsdb.retention.time` startup argument that defines retention time. | false | string | `30m` | `1.37.0` | | `PROMETHEUS_BLOCK_DURATION` | Prometheus `storage.tsdb.min-block-duration` and `storage.tsdb.max-block-duration` startup argument format "min,max". | false | string | `5m,30m` | `1.37.0` | | `PROMETHEUS_SCRAPE_INTERVAL` | Prometheus `global.scrape_interval` in configuration. | false | string | `15s` | `1.37.0` | | `PROMETHEUS_EVAL_INTERVAL` | Prometheus `global.evaluation_interval` in configuration. | false | string | `15s` | `1.37.0` | | `CORTEX_BLOCKS_RANGES` | Cortex block ranges configuration (`blocks_storage.tsdb.block_ranges_period`). | false | string | `1h` | `1.37.0` | | `CORTEX_RETENTION_PERIOD` | Cortex local block retention before flushing to external storage (`blocks_storage.tsdb.retention_period`). | false | string | `3h` | `1.37.0` | | `CORTEX_COMPACTOR_BLOCKS_RANGES` | Cortex compactor blocks sizes configuration (`compactor.block_ranges`). | false | string | `2h,12h,24h` | `1.37.0` | | `CORTEX_COMPACTOR_INTERVAL` | Cortex compactor run interval configuration (`compactor.compaction_interval`). | false | string | `30m` | `1.37.0` | | `CORTEX_COMPACTOR_DELETION_DELAY` | Cortex compactor block deletion delay configuration (`compactor.deletion_delay`). | false | string | `1h` | `1.37.0` | | `CORTEX_COMPACTOR_RETENTION_PERIOD` | Cortex compactor block retention configuration (`limits_config.compactor_blocks_retention_period`). | false | string | `45d` | `1.46.0` | | `CORTEX_CLEANER_INTERVAL` | Configure how often cleanup task is running (only use when no external object storage is configured). | false | string | `6h` | `1.37.0` | | `CORTEX_CLEANER_RETENTION` | Configure the retention duration in the "file system bucket" (`/var/conduktor/monitoring/data/cortex/tsdb-backend/`). | false | string | `30d` | `1.37.0` | | **Override configuration** | | | | | | | `PROMETHEUS_OVERRIDE_CONFIG_FILE` | Location to yaml config file to **patch** Prometheus configuration with. | false | string | `/opt/override-configs/prometheus.yaml` | `1.19.0` | | `CORTEX_OVERRIDE_CONFIG_FILE` | Location to yaml config file to **patch** Cortex configuration with. | false | string | `/opt/override-configs/cortex.yaml` | `1.19.0` | | `ALERTMANAGER_OVERRIDE_CONFIG_FILE` | Location to yaml config file to **patch** Cortex alert manager configuration with. | false | string | `/opt/override-configs/alertmanager.yaml` | `1.37.0` | | `PROMETHEUS_EXTRA_RUN_ARGS` | Extra run arguments for Prometheus. | false | string | `""` | `1.37.0` | | `CORTEX_EXTRA_RUN_ARGS` | Extra run arguments for Cortex. | false | string | `""` | `1.37.0` | | `CORTEX_ALERT_EXTRA_RUN_ARGS` | Extra run arguments for Cortex alert manager. | false | string | `""` | `1.37.0` | ## Sample configuration In a Docker Compose it may look like the following: ```yaml theme={null} version: '3.8' services: conduktor-console: image: conduktor/conduktor-console ports: - "8080:8080" environment: CDK_MONITORING_CORTEX-URL: http://conduktor-monitoring:9009/ CDK_MONITORING_ALERT-MANAGER-URL: http://conduktor-monitoring:9010/ CDK_MONITORING_CALLBACK-URL: http://conduktor-console:8080/monitoring/api/ CDK_MONITORING_NOTIFICATIONS-CALLBACK-URL: http://localhost:8080 conduktor-monitoring: image: conduktor/conduktor-console-cortex ports: - "9009:9009" # cortex api - "9010:9010" # alertmanager api - "9090:9090" # prometheus api environment: CDK_CONSOLE-URL: "http://conduktor-console:8080" ``` ## Configure basic auth for metric scraping Since Console v1.42.0, you can configure basic auth credentials to protect access to Console metric endpoints. When configured, the Cortex component has to provide these credentials when scraping metrics from the `/api/monitoring/metrics` and `/monitoring/metrics/` endpoints. Basic auth is optional in Console v1.42.0 and no configuration changes are required for existing deployments. ### Enable basic auth on Console Add these environment variables to Console: ```yaml theme={null} environment: CDK_MONITORING_BASICAUTH_EMAIL: "metrics@example.com" CDK_MONITORING_BASICAUTH_PASSWORD: "Hello123!" ``` The password has to contain at least 8 characters, including one uppercase letter, one lowercase letter, one number and one special symbol. ### Configure Cortex to authenticate When basic auth is enabled in Console, configure Cortex to authenticate when scraping metrics by adding the same credentials: ```yaml theme={null} environment: CDK_SCRAPER_BASICAUTH_EMAIL: "metrics@example.com" CDK_SCRAPER_BASICAUTH_PASSWORD: "Hello123!" ``` ```yaml theme={null} services: conduktor-console: image: conduktor/conduktor-console environment: CDK_MONITORING_BASICAUTH_EMAIL: "metrics@example.com" CDK_MONITORING_BASICAUTH_PASSWORD: "Hello123!" # Other configuration... conduktor-monitoring: image: conduktor/conduktor-console-cortex environment: CDK_CONSOLE-URL: "http://conduktor-console:8080" CDK_SCRAPER_BASICAUTH_EMAIL: "metrics@example.com" CDK_SCRAPER_BASICAUTH_PASSWORD: "Hello123!" ``` Add the environment variables to your Helm chart values: ```yaml theme={null} platformCortex: env: CDK_SCRAPER_BASICAUTH_EMAIL: "metrics@example.com" CDK_SCRAPER_BASICAUTH_PASSWORD: "Hello123!" ``` For sensitive values, use a secret instead: ```yaml theme={null} platformCortex: envFrom: - secretRef: name: cortex-auth-credentials ``` ## Metrics retention ### Local storage retention Without external storage metrics are stored in `/var/conduktor/monitoring/data/cortex/tsdb-backend/`. **Default retention lifecycle:** * Prometheus creates blocks every 5-30 minutes (`PROMETHEUS_BLOCK_DURATION: 5m,30m`) * Cortex keeps blocks for 3 hours (`CORTEX_RETENTION_PERIOD: 3h`) * Cortex compacts blocks every 30 minutes (`CORTEX_COMPACTOR_INTERVAL: 30m`) * Compacted blocks are deleted 1 hour after compaction (`CORTEX_COMPACTOR_DELETION_DELAY: 1h`) * Cleanup runs every 6 hours (`CORTEX_CLEANER_INTERVAL: 6h`) * Final retention: 30 days (`CORTEX_CLEANER_RETENTION: 30d`) All metrics are lost when the container restarts or is recreated. Mount a persistent volume to `/var/conduktor/monitoring` or use external storage. ### External storage retention With external storage (S3, GCS, Azure, PVC) configured, metrics persist across container restarts. `CORTEX_RETENTION_PERIOD` (default: 3 hours) controls how long the ingester buffers data locally before flushing it to external storage. `CORTEX_COMPACTOR_RETENTION_PERIOD` (default: 45 days) controls how long the compactor retains blocks in external storage before deleting them. The `CORTEX_CLEANER_*` variables are ignored when external storage is configured. Use `CORTEX_COMPACTOR_RETENTION_PERIOD` to configure how long the compactor keeps blocks in external storage, or configure your storage provider's lifecycle policy as a fallback. **Example: Configure 7-day retention with S3:** ```yaml theme={null} environment: CDK_MONITORING_STORAGE_S3_BUCKET: my-metrics-bucket CDK_MONITORING_STORAGE_S3_REGION: us-east-1 CORTEX_COMPACTOR_RETENTION_PERIOD: 7d ``` For production, external storage is required to avoid data loss. [Configure external storage](/guide/conduktor-in-production/system-requirements#production-requirements) using the storage environment variables in the table above. Cortex maintains a cache in local storage for every block uploaded to external storage. Without a retention period (`CORTEX_COMPACTOR_RETENTION_PERIOD`) set, blocks accumulate indefinitely in both local and external storage. This is why it defaults to 45 days. Therefore, do not set this to `0s` (disabled) or too high value (e.g. `365d`). ### Storage sizing Storage requirements vary based on your cluster count, topic count, scrape interval and retention period. **Example:** A deployment with 5 clusters and 500 topics using default settings (15s scrape interval) with 30-day retention might use approximately 85 GB. Monitor actual usage to determine your requirements: ```bash theme={null} # Local storage docker exec conduktor-monitoring du -sh /var/conduktor/monitoring/data # Kubernetes PVC kubectl exec -it -- df -h /var/conduktor/monitoring ``` ## Overriding configuration ### Overriding with YAML #### Cortex [Cortex configuration](https://cortexmetrics.io/docs/configuration/configuration-file/) can be **patched** by mounting a YAML file to path `/opt/override-configs/cortex.yaml`. To set an alternative path, use the `CORTEX_OVERRIDE_CONFIG_FILE` environment variable. Cortex alert manager configuration can also be **patched** by mounting a YAML file to path `/opt/override-configs/alertmanager.yaml`. To set an alternative path, use the `ALERTMANAGER_OVERRIDE_CONFIG_FILE` environment variable. For example, create a file `cortex.yaml` add in only your overrides: ```yaml theme={null} limits: ingestion_rate: 50000 max_series_per_metric: 100000 ``` 1. Mount to `/opt/override-configs/cortex.yaml`. 2. Spin up the container. 3. Exec into the container and confirm the contents, replace `2` with the number of lines of override you wish to see or remove grep to get the whole file: `cat /var/conduktor/configs/monitoring-cortex.yaml | grep limits -A2`. You should see a similar entry to the below in the opening logs: ```text theme={null} INFO monitoring_entrypoint - Patch "/var/conduktor/configs/monitoring-cortex.yaml" configuration with "/opt/override-configs/cortex.yaml" fragment ``` #### Prometheus Before Console v1.37.0, Prometheus configuration was overridden using the **replace** strategy. Since **v1.37.0** it's changed to the **patch** (YAML merging) strategy. Prometheus configuration can be also **patched** by mounting a YAML file to path `/opt/override-configs/prometheus.yaml`. To set an alternative path, use the `PROMETHEUS_OVERRIDE_CONFIG_FILE` environment variable. You should see a similar entry to the below in the opening logs: ```text theme={null} INFO monitoring_entrypoint - Replace "/var/conduktor/configs/monitoring-prometheus.yaml" configuration with "/opt/override-configs/prometheus.yaml" content ``` ### Overriding with ConfigMap If you're deploying Cortex using our [Helm charts](https://github.com/conduktor/conduktor-public-charts/blob/main/charts/console/README.md#platform-cortex-parameters) you may expand the input with a custom ConfigMap for patching configuration such as retention time within Cortex. You can also replace the Prometheus config as described below. ```yaml theme={null} apiVersion: v1 kind: ConfigMap metadata: name: conduktor-console-cortex-config labels: app.kubernetes.io/name: console app.kubernetes.io/instance: conduktor app.kubernetes.io/component: conduktor-platform-cortex data: cortex.yaml: | blocks_storage: tsdb: retention_period: 24h prometheus.yaml: | global: scrape_interval: 15s evaluation_interval: 15s ``` On chart `values.yaml` : ```yaml theme={null} platformCortex: extraVolumes: - name: cortex-config-override configMap: name: conduktor-console-cortex-config extraVolumeMounts: - name: cortex-config-override mountPath: /opt/override-configs/cortex.yaml subPath: cortex.yaml - name: cortex-config-override mountPath: /opt/override-configs/prometheus.yaml subPath: prometheus.yaml ``` ## Monitoring endpoints Monitoring is designed to be interacted with through the API endpoints by Console (not users). Therefore, **ingress isn't available externally** and you shouldn't set it up because an authentication mechanism doesn't exist. ### Cortex endpoint `/ready` on port `9009` If Cortex is running, returns the *200 status* with `ready`. ```shell title="cURL example" theme={null} curl -s "http://localhost:9009/ready" ``` ### Alertmanager endpoint `/ready` on port `9010` If Alertmanager is running, returns the *200 status* with `ready`. ```shell title="cURL example" theme={null} curl -s "http://localhost:9010/ready" ``` ### Prometheus endpoint `/-/healthy` on port `9090` If Prometheus is running, returns the *200 status* with `Prometheus Server is Healthy.`. ```shell title="cURL example" theme={null} curl -s "http://localhost:9090/-/healthy" ``` ## Troubleshoot ### No metrics in the monitoring page Go to `http://localhost:9090/targets` to see Prometheus scraping target status. If it fails, check that you can query metrics endpoint from `conduktor-console-cortex` container. You might also have to configure `CDK_SCRAPER_SKIPSSLCHECK` or `CDK_SCRAPER_CAFILE` if `conduktor-console` is configured with TLS termination. ### Custom CA certificates for S3-like storage The `CDK_SCRAPER_CAFILE` environment variable only configures the CA certificate for the Scraper component (which scrapes metrics from Console). It does **not** configure the CA certificate for S3-like storage backends. If you use an S3-compatible service (like MinIO, Ceph or other S3-like services) with custom certificates, you have two options: #### Option 1: Disable SSL verification To skip TLS hostname verification with S3-like services, [override the Cortex configuration with YAML](#overriding-with-yaml) by creating a YAML file with these contents: ```yaml theme={null} blocks_storage: s3: http: insecure_skip_verify: true ruler_storage: s3: http: insecure_skip_verify: true ``` Mount this file to `/opt/override-configs/cortex.yaml` (or use the `CORTEX_OVERRIDE_CONFIG_FILE` environment variable to specify an alternative path). Disabling SSL verification removes certificate validation and is **not recommended for production** environments. Use this only for testing or non-production deployments. #### Option 2: Contact Conduktor support For production environments requiring custom CA certificates with S3-compatible storage, [contact Conduktor support](https://www.conduktor.io/contact/demo/?utm_source=docs\&utm_medium=webpage) for guidance on configuring certificate trust for Cortex components. ## Related resources * [Conduktor monitoring](/guide/monitor-brokers-apps) * [Gateway and Console healthcheck monitoring](/guide/conduktor-in-production/monitor) * [Set up alerts](/guide/monitor-brokers-apps/alerts) * [Give us feedback/request a feature](https://conduktor.io/roadmap)