Skip to main content
With Conduktor you can detect data quality issues in messages flowing through Kafka. You configure your own Rules to specify the expected format of messages and attach them to Policies targeting specific Kafka topics/prefixes. The power of these Policies depends on whether you’re targeting Conduktor Gateway clusters or Kafka clusters directly. In both cases, Conduktor will evaluate the rules of your Policy against every message produced on targeted topics after Policy creation and keep track of the count of violating messages as well as the total number of messages evaluated. This grants you observability into the data quality of your topics with Gateway offering an additional step of enforcement through actions. Without Gateway, records are verified after they have been produced whereas with Gateway, records are verified before production. This means that with Gateway, you can take specific actions on faulty records:
  • block: the message (and the entire batch) is rejected. The producer sending it will fail with an appropriate error message.
  • mark: a header, containing all violated rules for every policy using the mark action, is added to the record. The header key is , and the value is a JSON object in the following format: 
    {
  "policy_name_1": ["rule_name_1", "rule_name_2"],
  "policy_name_2": ["rule_name_3"]
}
Since the target cluster type determines the capabilities of data quality Policies, you can’t create a policy that mixes different target types. You also can’t target a Gateway-backing cluster directly, as this would create conflicts between Gateway record validation and Console validation.