Deploy Cortex

To enable monitoring via Console, you have to deploy and configure the console-cortex component.

Deploy and configure Cortex

The only required property is CDK_CONSOLE-URL, everything else is related to storage for the metrics. By default, data will be stored in /var/conduktor/monitoring inside the running image. You can mount a volume on this folder to keep metrics data between updates. Alternatively, you can use the storage parameters described below to store the data using either s3, gcs, azure or swift.

Environment variable	Description	Mandatory	Type	Default	Since
`CDK_CONSOLEURL`	Console URL and port (example: `"http://conduktor-console:8080"`).	true	string	∅	`1.18.0`
`CDK_SCRAPER_SKIPSSLCHECK`	Disable TLS check when scraping metrics from Console.	false	bool	`false`	`1.18.2`
`CDK_SCRAPER_CAFILE`	Path to CA certificate file inside the container to perform TLS check when scraping metrics from Console.	false	string	∅	`1.18.2`
S3
`CDK_MONITORING_STORAGE_S3_ENDPOINT`	S3 storage endpoint.	false	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_S3_REGION`	S3 storage region.	false	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_S3_BUCKET`	S3 storage bucket name.	true	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_S3_INSECURE`	S3 storage SSL/TLS check flag.	false	bool	`false`	`1.18.0`
`CDK_MONITORING_STORAGE_S3_ACCESSKEYID`	Access key ID of an AWS IAM identity for monitoring to upload logs to S3. If set to false, and if you are running on a Kubernetes deployment, Monitoring can attempt to leverage AWS IRSA for the Pod’s service account when connecting to S3.	false	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_S3_SECRETACCESSKEY`	Secret access key of an AWS IAM identity for monitoring to upload logs to S3. If set to false, and if you are running on a Kubernetes deployment, Monitoring can attempt to leverage AWS IRSA for the Pod’s service account when connecting to S3.	false	string	∅	`1.18.0`
GCS
`CDK_MONITORING_STORAGE_GCS_BUCKETNAME`	GCS storage bucket name.	true	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_GCS_SERVICEACCOUNT`	GCS storage service account JSON content.	true	string	∅	`1.18.0`
Azure
`CDK_MONITORING_STORAGE_AZURE_ACCOUNTNAME`	Azure storage account name.	true	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_AZURE_ACCOUNTKEY`	Azure storage account key.	true	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_AZURE_CONTAINERNAME`	Azure storage container name.	true	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_AZURE_ENDPOINTSUFFIX`	Azure storage endpoint suffix.	false	string	`blob.core.windows.net`	`1.18.0`
Swift
`CDK_MONITORING_STORAGE_SWIFT_AUTHURL`	Swift storage authentication URL.	true	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_SWIFT_PASSWORD`	Swift storage user password.	true	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_SWIFT_CONTAINERNAME`	Swift storage container name.	true	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_SWIFT_USERID`	Swift storage user ID.	false	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_SWIFT_USERNAME`	Swift storage user name.	false	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_SWIFT_USERDOMAINNAME`	Swift storage user domain name.	false	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_SWIFT_USERDOMAINID`	Swift storage user domain ID.	false	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_SWIFT_DOMAINID`	Swift storage user domain ID.	false	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_SWIFT_DOMAINNAME`	Swift storage user domain name.	false	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_SWIFT_PROJECTID`	Swift storage project ID.	false	string	∅	`1.18.0`
`CDK_MONITORING_STORAGE_SWIFT_REGIONNAME`	Swift storage region name.	false	string	∅	`1.18.0`
Logs
`CORTEX_ROOT_LOG_LEVEL`	Cortex log level.	false	string	`info`	`1.18.0`
`CORTEX_ALERT_ROOT_LOG_LEVEL`	Alert manager log level.	false	string	`info`	`1.18.0`
`PROMETHEUS_ROOT_LOG_LEVEL`	Prometheus log level.	false	string	`info`	`1.18.0`
Retention configuration
`PROMETHEUS_RETENTION_SIZE`	Prometheus `storage.tsdb.retention.size` startup argument that defines retention size.	false	string	`5MB`	`1.37.0`
`PROMETHEUS_RETENTION_TIME`	Prometheus `storage.tsdb.retention.time` startup argument that defines retention time.	false	string	`30m`	`1.37.0`
`PROMETHEUS_BLOCK_DURATION`	Prometheus `storage.tsdb.min-block-duration` and `storage.tsdb.max-block-duration` startup argument format “min,max”.	false	string	`5m,30m`	`1.37.0`
`PROMETHEUS_SCRAPE_INTERVAL`	Prometheus `global.scrape_interval` in configuration.	false	string	`15s`	`1.37.0`
`PROMETHEUS_EVAL_INTERVAL`	Prometheus `global.evaluation_interval` in configuration.	false	string	`15s`	`1.37.0`
`CORTEX_BLOCKS_RANGES`	Cortex block ranges configuration (`blocks_storage.tsdb.block_ranges_period`).	false	string	`1h`	`1.37.0`
`CORTEX_RETENTION_PERIOD`	Cortex block local retention configuration (`blocks_storage.tsdb.retention_period`).	false	string	`3h`	`1.37.0`
`CORTEX_COMPACTOR_BLOCKS_RANGES`	Cortex compactor blocks sizes configuration (`compactor.block_ranges`).	false	string	`2h,12h,24h`	`1.37.0`
`CORTEX_COMPACTOR_INTERVAL`	Cortex compactor run interval configuration (`compactor.compaction_interval`).	false	string	`30m`	`1.37.0`
`CORTEX_COMPACTOR_DELETION_DELAY`	Cortex compactor block deletion delay configuration (`compactor.deletion_delay`).	false	string	`1h`	`1.37.0`
`CORTEX_CLEANER_INTERVAL`	Configure how often cleanup task is running (only use when no external object storage is configured).	false	string	`6h`	`1.37.0`
`CORTEX_CLEANER_RETENTION`	Configure the retention duration in the “file system bucket” (`/var/conduktor/monitoring/data/cortex/tsdb-backend/`).	false	string	`30d`	`1.37.0`
Override configuration
`PROMETHEUS_OVERRIDE_CONFIG_FILE`	Location to yaml config file to patch Prometheus configuration with.	false	string	`/opt/override-configs/prometheus.yaml`	`1.19.0`
`CORTEX_OVERRIDE_CONFIG_FILE`	Location to yaml config file to patch Cortex configuration with.	false	string	`/opt/override-configs/cortex.yaml`	`1.19.0`
`ALERTMANAGER_OVERRIDE_CONFIG_FILE`	Location to yaml config file to patch Cortex alert manager configuration with.	false	string	`/opt/override-configs/alertmanager.yaml`	`1.37.0`
`PROMETHEUS_EXTRA_RUN_ARGS`	Extra run arguments for Prometheus.	false	string	`""`	`1.37.0`
`CORTEX_EXTRA_RUN_ARGS`	Extra run arguments for Cortex.	false	string	`""`	`1.37.0`
`CORTEX_ALERT_EXTRA_RUN_ARGS`	Extra run arguments for Cortex alert manager.	false	string	`""`	`1.37.0`

Sample configuration

In a Docker Compose it may look like the following:

version: '3.8'
services:
  conduktor-console:
    image: conduktor/conduktor-console
    ports:
      - "8080:8080"
    environment:
      CDK_MONITORING_CORTEX-URL: http://conduktor-monitoring:9009/
      CDK_MONITORING_ALERT-MANAGER-URL: http://conduktor-monitoring:9010/
      CDK_MONITORING_CALLBACK-URL: http://conduktor-console:8080/monitoring/api/
      CDK_MONITORING_NOTIFICATIONS-CALLBACK-URL: http://localhost:8080
  conduktor-monitoring:
    image: conduktor/conduktor-console-cortex
    ports:
      - "9009:9009" # cortex api
      - "9010:9010" # alertmanager api
      - "9090:9090" # prometheus api
    environment:
      CDK_CONSOLE-URL: "http://conduktor-console:8080"

Overriding configuration

Overriding with YAML

Cortex

Cortex configuration can be patched by mounting a YAML file to path /opt/override-configs/cortex.yaml. To set an alternative path, use the CORTEX_OVERRIDE_CONFIG_FILE environment variable. Cortex alert manager configuration can also be patched by mounting a YAML file to path /opt/override-configs/alertmanager.yaml. To set an alternative path, use the ALERTMANAGER_OVERRIDE_CONFIG_FILE environment variable. For example, create a file cortex.yaml add in only your overrides:

limits:
  ingestion_rate: 50000
  max_series_per_metric: 100000

Mount to /opt/override-configs/cortex.yaml.
Spin up the container.
Exec into the container and confirm the contents, replace 2 with the number of lines of override you wish to see or remove grep to get the whole file: cat /var/conduktor/configs/monitoring-cortex.yaml | grep limits -A2.

You should see a similar entry to the below in the opening logs:

INFO monitoring_entrypoint - Patch "/var/conduktor/configs/monitoring-cortex.yaml" configuration with "/opt/override-configs/cortex.yaml" fragment

Prometheus

Before Console v1.37.0, Prometheus configuration was overridden using the replace strategy. Since v1.37.0 it’s changed to the patch (YAML merging) strategy.

Prometheus configuration can be also patched by mounting a YAML file to path /opt/override-configs/prometheus.yaml. To set an alternative path, use the PROMETHEUS_OVERRIDE_CONFIG_FILE environment variable. You should see a similar entry to the below in the opening logs:

INFO monitoring_entrypoint - Replace "/var/conduktor/configs/monitoring-prometheus.yaml" configuration with "/opt/override-configs/prometheus.yaml" content

Overriding with ConfigMap

If you’re deploying Cortex using our Helm charts you may expand the input with a custom ConfigMap for patching configuration such as retention time within Cortex. You can also replace the Prometheus config as described below.

apiVersion: v1
kind: ConfigMap
metadata:
  name: conduktor-console-cortex-config
  labels:
    app.kubernetes.io/name: console
    app.kubernetes.io/instance: conduktor
    app.kubernetes.io/component: conduktor-platform-cortex
data:
  cortex.yaml: |
    blocks_storage:
      tsdb:
        retention_period: 24h
  prometheus.yaml: |
    global:
      scrape_interval: 15s
      evaluation_interval: 15s

On chart values.yaml :

platformCortex:
  extraVolumes:
    - name: cortex-config-override
      configMap:
        name: conduktor-console-cortex-config
  extraVolumeMounts:
        - name: cortex-config-override
          mountPath: /opt/override-configs/cortex.yaml
          subPath: cortex.yaml
        - name: cortex-config-override
          mountPath: /opt/override-configs/prometheus.yaml
          subPath: prometheus.yaml

Monitoring endpoints

Monitoring is designed to be interacted with through the API endpoints by Console (not users). Therefore, ingress isn’t available externally and you shouldn’t set it up because an authentication mechanism doesn’t exist.

Cortex endpoint

/ready on port 9009 If Cortex is running, returns the 200 status with ready.

cURL example

curl -s "http://localhost:9009/ready"

Alertmanager endpoint

/ready on port 9010 If Alertmanager is running, returns the 200 status with ready.

cURL example

curl -s "http://localhost:9010/ready"

Prometheus endpoint

/-/healthy on port 9090 If Prometheus is running, returns the 200 status with Prometheus Server is Healthy..

cURL example

curl -s "http://localhost:9090/-/healthy"

Troubleshoot

No metrics in the monitoring page

Go to http://localhost:9090/targets to see Prometheus scraping target status. If it fails, check that you can query metrics endpoint from conduktor-console-cortex container. You might also have to configure CDK_SCRAPER_SKIPSSLCHECK or CDK_SCRAPER_CAFILE if conduktor-console is configured with TLS termination.

Conduktor in production

Deploy Gateway

Deploy Console

Initial setup

Deploy and configure Cortex

Sample configuration

Overriding configuration

Overriding with YAML

Cortex

Prometheus

Overriding with ConfigMap

Monitoring endpoints

Cortex endpoint

Alertmanager endpoint

Prometheus endpoint

Troubleshoot

No metrics in the monitoring page

Conduktor in production

Deploy Gateway

Deploy Console

Initial setup

​Deploy and configure Cortex

​Sample configuration

​Overriding configuration

​Overriding with YAML

​Cortex

​Prometheus

​Overriding with ConfigMap

​Monitoring endpoints

​Cortex endpoint

​Alertmanager endpoint

​Prometheus endpoint

​Troubleshoot

​No metrics in the monitoring page

​Related resources

Deploy and configure Cortex

Sample configuration

Overriding configuration

Overriding with YAML

Cortex

Prometheus

Overriding with ConfigMap

Monitoring endpoints

Cortex endpoint

Alertmanager endpoint

Prometheus endpoint

Troubleshoot

No metrics in the monitoring page

Related resources