This feature is available with Conduktor Trust only.

​
Overview

Bad data breaks customer experiences, drives churn and slows growth. Conduktor Trust helps teams catch and fix data quality issues before they impact your business. You define the Rules and we’ll enforce them at the streaming layer.

​
Prerequisites

Before creating data quality Rules and Policies, you have to:
  • use Conduktor Console 1.34 or later,
  • use Conduktor Gateway 3.9 or later,
  • be logged in as an admin to Console UI or use an admin token for the Conduktor CLI,
  • configure your Gateway cluster in Console and fill in the Provider tab with Gateway API credentials.

​
Rules

You can create Rules with the CEL expressions which capture business logic for your data. For example: value.customerId.matches("[0-9]{8}").
Rules do nothing on their own - you have to to attach them to a Policy.
The Rules page lists your Rules with a preview of their CEL expressions. Open Rule’s detail page to see its description, full CEL expression and attached Policies.

​
Create a Rule

You can create a data quality Rule using the Console UI or the Conduktor CLI.
To create a Rule using the Console UI:
  1. In the Trust section of the sidebar of Conduktor Console go to Rules and click +New Rule.
  2. Define the Rule details:
    • Add a descriptive name for the Rule.
    • The Technical ID will be auto-populated as you type in the name. This is used to identify this Rule in CLI/API.
    • (Optional) Enter a Description to explain your Rule.
  3. Define the CEL expression for your Rule:
    • Common Expression Language (CEL) is an expression language supporting common operators like == and > as well as macros like has() to check for the presence of fields. See the CEL language definition for details.
  4. Click Create.

​
Built-in Rules

We provide built-in validation Rules that can’t be achieved with CEL.
We currently only support Confluent and Confluent like (e.g. Redpanda) schema registries.

​
EnforceAvro

EnforceAvro ensures that:
  • Your messages have a schema ID prepended to the message content.
  • The schema ID exists within your schema registry.
  • The schema it references is of type avro.

​
Sample Rules

Here are some examples of data quality rules.
Make sure you amend these examples to use your values.

​
Policies

A Policy is made up of Rules that are applied to topics/prefixes. Once created, Policies can be assigned actions to take effect when certain criteria is met (e.g., a Rule in the Policy is violated). The Policy’s detail page shows its description, linked Rules, targets, the number of messages evaluated and the number of violations since the Policy was created. You can also enable (and disable) actions for the Policy from this page. The Policies page lists your Policies with their actions and targets.

​
Actions

The available actions to enable for a Policy are:
  • Report: when violations occur, log these as events in the Policy’s history
  • Block: when a violation occurs, prevent data from being processed or transmitted
By default, Policies created using the Console UI don’t have any actions enabled. You have to complete the Policy creation first and then enable the required actions. If there are any additional actions you’d like to see, get in touch.

​
Create a Policy

You can create a data quality Policy using the Console UI or the Conduktor CLI.
To can create a Policy using the Console UI:
  1. Under the Trust section of the main menu, go to Policies and click +New Policy.
  2. Define the Policy details:
    • Add a descriptive name for the Policy.
    • The Technical ID will be auto-populated as you type in the name. This is used to identify this Policy in CLI/API.
    • Select a group to own the Policy. This controls who can view and manage the Policy, and which resources can be targeted.
    • (Optional) Enter a Description to explain your Rule.
  3. Select Rules to be used in the Policy:
    • Every Policy must have at least one Rule
    • You can also create new Rules from this page
    • Click Continue to move to the next step.
  4. Select targets for the Policy:
    • Every Policy must have at least one target
    • A target consists of one or more topics on a specified Gateway
    • You can either select specific topics, or specify a prefix like orders-*
    • Click Continue to move to the next step.
  5. Review the Policy, and if you are happy, confirm by clicking Create.

​
Manage a Policy

Once a Policy is created, you are able to view the linked Rule(s), the target(s) of the Policy and change the actions of the Policy. You can also view the violations as they have occurred if you have reporting enabled, otherwise you will only have the counts available.
Since the block action has the ability to stop data from being sent to the requested topic, you have to confirm this by entering ‘BLOCK’ when prompted. Conversely, to disable the blocking, enter ‘UNBLOCK’ when prompted.

​
Assign permissions

Policies are owned by user groups and can be created by admin users or groups with the Manage data quality permission enabled. To apply this permission to a group, go to Settings > Groups and in the Resource access tab tick the Manage data quality checkbox for the relevant resources, as required.
Make sure to enable this on the Gateway (and not the underlying Kafka) cluster. Modifying group permissions won’t affect any Policies associated with the group.
The 'manage data quality' permission

​
Set up Policy violation alerts

You can create alerts that are triggered when a Policy violation happens. Data quality alerts can be owned by groups or individual users.
To create a data quality policy alert via the UI, go to the details page of a Policy (click on the button next to the violations graph) or from the alert tab on the Policies list page.A data quality policy alert needs to specify a Policy and a threshold: trigger after X violations within Y minutes/hours/days. This threshold replaces the combination of metric, operator and value used in other alerts.Find out more about alerts.

​
Using multiple Policies

When multiple Policies target the same topic, there are two scenarios that can occur when a record is produced:
  • None of the Policies block the record and all are evaluated
    • The evaluation count is increased for all of the Policies.
    • The violation count is increased for each violated Policy.
    • A report is generated for each violated Policy (that has reporting enabled).
  • One or more of the Policies block the record production. In this scenario, one of the Policies blocks the record first and then hides it from others
    • For the first blocking Policy, both the violation and evaluation counts are increased. If reporting is enabled for that policy, a report is generated.
    • For the other Policies: no counts are increased and no reports are generated.

​
Troubleshoot