To enable monitoring via Console, you have to deploy and configure the console-cortex artifact.

​
Deploy and configure Cortex

The only required property is CDK_CONSOLE-URL, everything else is related to storage for the metrics. By default, data will be stored in /var/conduktor/monitoring inside the running image. You can mount a volume on this folder to keep metrics data between updates. Alternatively, you can use the storage parameters described below to store the data using either s3, gcs, azure or swift.
Environment variableDescriptionMandatoryTypeDefaultSince
CDK_CONSOLEURLConsole URL and port (example: "http://conduktor-console:8080").truestring∅1.18.0
CDK_SCRAPER_SKIPSSLCHECKDisable TLS check when scraping metrics from Console.falseboolfalse1.18.2
CDK_SCRAPER_CAFILEPath to CA certificate file inside the container to perform TLS check when scraping metrics from Console.falsestring∅1.18.2
S3
CDK_MONITORING_STORAGE_S3_ENDPOINTS3 storage endpoint.falsestring∅1.18.0
CDK_MONITORING_STORAGE_S3_REGIONS3 storage region.falsestring∅1.18.0
CDK_MONITORING_STORAGE_S3_BUCKETS3 storage bucket name.truestring∅1.18.0
CDK_MONITORING_STORAGE_S3_INSECURES3 storage SSL/TLS check flag.falseboolfalse1.18.0
CDK_MONITORING_STORAGE_S3_ACCESSKEYIDAccess key ID of an AWS IAM identity for monitoring to upload logs to S3. If set to false, and if you are running on a Kubernetes deployment, Monitoring can attempt to leverage AWS IRSA for the Pod’s service account when connecting to S3.falsestring∅1.18.0
CDK_MONITORING_STORAGE_S3_SECRETACCESSKEYSecret access key of an AWS IAM identity for monitoring to upload logs to S3. If set to false, and if you are running on a Kubernetes deployment, Monitoring can attempt to leverage AWS IRSA for the Pod’s service account when connecting to S3.falsestring∅1.18.0
GCS
CDK_MONITORING_STORAGE_GCS_BUCKETNAMEGCS storage bucket name.truestring∅1.18.0
CDK_MONITORING_STORAGE_GCS_SERVICEACCOUNTGCS storage service account JSON content.truestring∅1.18.0
Azure
CDK_MONITORING_STORAGE_AZURE_ACCOUNTNAMEAzure storage account name.truestring∅1.18.0
CDK_MONITORING_STORAGE_AZURE_ACCOUNTKEYAzure storage account key.truestring∅1.18.0
CDK_MONITORING_STORAGE_AZURE_CONTAINERNAMEAzure storage container name.truestring∅1.18.0
CDK_MONITORING_STORAGE_AZURE_ENDPOINTSUFFIXAzure storage endpoint suffix.falsestringblob.core.windows.net1.18.0
Swift
CDK_MONITORING_STORAGE_SWIFT_AUTHURLSwift storage authentication URL.truestring∅1.18.0
CDK_MONITORING_STORAGE_SWIFT_PASSWORDSwift storage user password.truestring∅1.18.0
CDK_MONITORING_STORAGE_SWIFT_CONTAINERNAMESwift storage container name.truestring∅1.18.0
CDK_MONITORING_STORAGE_SWIFT_USERIDSwift storage user ID.falsestring∅1.18.0
CDK_MONITORING_STORAGE_SWIFT_USERNAMESwift storage user name.falsestring∅1.18.0
CDK_MONITORING_STORAGE_SWIFT_USERDOMAINNAMESwift storage user domain name.falsestring∅1.18.0
CDK_MONITORING_STORAGE_SWIFT_USERDOMAINIDSwift storage user domain ID.falsestring∅1.18.0
CDK_MONITORING_STORAGE_SWIFT_DOMAINIDSwift storage user domain ID.falsestring∅1.18.0
CDK_MONITORING_STORAGE_SWIFT_DOMAINNAMESwift storage user domain name.falsestring∅1.18.0
CDK_MONITORING_STORAGE_SWIFT_PROJECTIDSwift storage project ID.falsestring∅1.18.0
CDK_MONITORING_STORAGE_SWIFT_REGIONNAMESwift storage region name.falsestring∅1.18.0
Logs
CORTEX_ROOT_LOG_LEVELCortex log level.falsestringinfo1.18.0
CORTEX_ALERT_ROOT_LOG_LEVELAlert manager log level.falsestringinfo1.18.0
PROMETHEUS_ROOT_LOG_LEVELPrometheus log level.falsestringinfo1.18.0
Retention configuration
PROMETHEUS_RETENTION_SIZEPrometheus storage.tsdb.retention.size startup argument that defines retention size.falsestring5MB1.37.0
PROMETHEUS_RETENTION_TIMEPrometheus storage.tsdb.retention.time startup argument that defines retention time.falsestring30m1.37.0
PROMETHEUS_BLOCK_DURATIONPrometheus storage.tsdb.min-block-duration and storage.tsdb.max-block-duration startup argument format “min,max”.falsestring5m,30m1.37.0
PROMETHEUS_SCRAPE_INTERVALPrometheus global.scrape_interval in configuration.falsestring15s1.37.0
PROMETHEUS_EVAL_INTERVALPrometheus global.evaluation_interval in configuration.falsestring15s1.37.0
CORTEX_BLOCKS_RANGESCortex block ranges configuration (blocks_storage.tsdb.block_ranges_period).falsestring1h1.37.0
CORTEX_RETENTION_PERIODCortex block local retention configuration (blocks_storage.tsdb.retention_period).falsestring3h1.37.0
CORTEX_COMPACTOR_BLOCKS_RANGESCortex compactor blocks sizes configuration (compactor.block_ranges).falsestring2h,12h,24h1.37.0
CORTEX_COMPACTOR_INTERVALCortex compactor run interval configuration (compactor.compaction_interval).falsestring30m1.37.0
CORTEX_COMPACTOR_DELETION_DELAYCortex compactor block deletion delay configuration (compactor.deletion_delay).falsestring1h1.37.0
CORTEX_CLEANER_INTERVALConfigure how often cleanup task is running (only use when no external object storage is configured).falsestring6h1.37.0
CORTEX_CLEANER_RETENTIONConfigure the retention duration in the “file system bucket” (/var/conduktor/monitoring/data/cortex/tsdb-backend/).falsestring30d1.37.0
Override configuration
PROMETHEUS_OVERRIDE_CONFIG_FILELocation to yaml config file to patch Prometheus configuration with.falsestring/opt/override-configs/prometheus.yaml1.19.0
CORTEX_OVERRIDE_CONFIG_FILELocation to yaml config file to patch Cortex configuration with.falsestring/opt/override-configs/cortex.yaml1.19.0
ALERTMANAGER_OVERRIDE_CONFIG_FILELocation to yaml config file to patch Cortex alert manager configuration with.falsestring/opt/override-configs/alertmanager.yaml1.37.0
PROMETHEUS_EXTRA_RUN_ARGSExtra run arguments for Prometheus.falsestring""1.37.0
CORTEX_EXTRA_RUN_ARGSExtra run arguments for Cortex.falsestring""1.37.0
CORTEX_ALERT_EXTRA_RUN_ARGSExtra run arguments for Cortex alert manager.falsestring""1.37.0

​
Sample configuration

In a Docker Compose it may look like the following: 
version: '3.8'
services:
  conduktor-console:
    image: conduktor/conduktor-console
    ports:
      - "8080:8080"
    environment:
      CDK_MONITORING_CORTEX-URL: http://conduktor-monitoring:9009/
      CDK_MONITORING_ALERT-MANAGER-URL: http://conduktor-monitoring:9010/
      CDK_MONITORING_CALLBACK-URL: http://conduktor-console:8080/monitoring/api/
      CDK_MONITORING_NOTIFICATIONS-CALLBACK-URL: http://localhost:8080
  conduktor-monitoring:
    image: conduktor/conduktor-console-cortex
    ports:
      - "9009:9009" # cortex api
      - "9010:9010" # alertmanager api
      - "9090:9090" # prometheus api
    environment:
      CDK_CONSOLE-URL: "http://conduktor-console:8080"

​
Overriding configuration

​
Overriding with YAML

​
Cortex

Cortex configuration can be patched by mounting a YAML file to path /opt/override-configs/cortex.yaml. To set an alternative path, use the CORTEX_OVERRIDE_CONFIG_FILE environment variable. Cortex alert manager configuration can also be patched by mounting a YAML file to path /opt/override-configs/alertmanager.yaml. To set an alternative path, use the ALERTMANAGER_OVERRIDE_CONFIG_FILE environment variable. For example, create a file cortex.yaml add in only your overrides: 
limits:
  ingestion_rate: 50000
  max_series_per_metric: 100000
  1. Mount to /opt/override-configs/cortex.yaml.
  2. Spin up the container.
  3. Exec into the container and confirm the contents, replace 2 with the number of lines of override you wish to see or remove grep to get the whole file: cat /var/conduktor/configs/monitoring-cortex.yaml | grep limits -A2.
You should see a similar entry to the below in the opening logs: 
INFO monitoring_entrypoint - Patch "/var/conduktor/configs/monitoring-cortex.yaml" configuration with "/opt/override-configs/cortex.yaml" fragment

​
Prometheus

Before Console v1.37.0, Prometheus configuration was overridden using the replace strategy. Since v1.37.0 it’s changed to the patch (YAML merging) strategy.
Prometheus configuration can be also patched by mounting a YAML file to path /opt/override-configs/prometheus.yaml. To set an alternative path, use the PROMETHEUS_OVERRIDE_CONFIG_FILE environment variable. You should see a similar entry to the below in the opening logs: 
INFO monitoring_entrypoint - Replace "/var/conduktor/configs/monitoring-prometheus.yaml" configuration with "/opt/override-configs/prometheus.yaml" content

​
Overriding with ConfigMap

If you’re deploying Cortex using our Helm charts you may expand the input with a custom ConfigMap for patching configuration such as retention time within Cortex. You can also replace the Prometheus config as described below. 
apiVersion: v1
kind: ConfigMap
metadata:
  name: conduktor-console-cortex-config
  labels:
    app.kubernetes.io/name: console
    app.kubernetes.io/instance: conduktor
    app.kubernetes.io/component: conduktor-platform-cortex
data:
  cortex.yaml: |
    blocks_storage:
      tsdb:
        retention_period: 24h
  prometheus.yaml: |
    global:
      scrape_interval: 15s
      evaluation_interval: 15s
On chart values.yaml : 
platformCortex:
  extraVolumes:
    - name: cortex-config-override
      configMap:
        name: conduktor-console-cortex-config
  extraVolumeMounts:
        - name: cortex-config-override
          mountPath: /opt/override-configs/cortex.yaml
          subPath: cortex.yaml
        - name: cortex-config-override
          mountPath: /opt/override-configs/prometheus.yaml
          subPath: prometheus.yaml

​
Monitoring endpoints

Monitoring is designed to be interacted with through the API endpoints by Console (not users). Therefore, ingress isn’t available externally and you shouldn’t set it up because an authentication mechanism doesn’t exist.

​
Cortex endpoint

/ready on port 9009 If Cortex is running, returns the 200 status with ready.
cURL example
curl -s "http://localhost:9009/ready"

​
Alertmanager endpoint

/ready on port 9010 If Alertmanager is running, returns the 200 status with ready.
cURL example
curl -s "http://localhost:9010/ready"

​
Prometheus endpoint

/-/healthy on port 9090 If Prometheus is running, returns the 200 status with Prometheus Server is Healthy..
cURL example
curl -s "http://localhost:9090/-/healthy"

​
Troubleshoot

​
No metrics in the monitoring page

Go to http://localhost:9090/targets to see Prometheus scraping target status. If it fails, check that you can query metrics endpoint from conduktor-console-cortex container. You might also have to configure CDK_SCRAPER_SKIPSSLCHECK or CDK_SCRAPER_CAFILE if conduktor-console is configured with TLS termination.