Skip to main content
Gateway validates records when producers send them, with four Interceptors.
From our blog: The hidden pitfalls of Kafka’s schemaless data Why schemaless topics let bad data through, and how CEL and JSON Schema rules catch it at the proxy.

Common configuration

Schema registry

The DataQualityPlugin, SQL data quality and schema payload validation Interceptors read schema-encoded records through a schema registry. Gateway reads schemas either through an external connection or from configuration on the Interceptor itself.
Schema-encoded records whose schema ID is stored in a record header, the format introduced in Confluent Platform 8.2, require Gateway 3.21.0 or later.
Set schemaRegistryConnectionName to the name of a schema registry external connection, declared once when Gateway starts with variables that follow this pattern:
For a connection declared as GATEWAY_EXTERNAL_CONNECTION_DEFAULT_SR_CONFIG_*, the Interceptor reaches it with:
Set either schemaRegistryConnectionName or schemaRegistryConfig, never both. Gateway rejects a configuration that sets both with 400 Bad Request, rather than silently picking one.Remove a connection an Interceptor already names and the Interceptor stays visible through the API but stops applying to traffic. See what happens at startup.

Dead letter topics

When an Interceptor writes to a dead letter topic for the first time, Gateway checks whether the topic exists on the backing Kafka cluster. If it doesn’t, Gateway creates it with the broker default partition count and replication factor. Each dead letter record keeps the original key, value and headers. The Interceptor also adds the headers listed in its own section.

DataQualityPlugin Interceptor

The DataQualityPlugin Interceptor evaluates every record produced to matching topics against a set of Rules, then blocks or marks the records that violate them. Console also deploys it for data quality Policies: see the DataQualityPolicy reference for how Policy fields map to this configuration.
  • Plugin class: io.conduktor.gateway.interceptor.dataquality.DataQualityPlugin
  • License: requires a Gateway license that includes data quality. Without it, Gateway rejects the Interceptor with 403 Feature 'Data Quality' is not included in your license.

Configure DataQualityPlugin

Gateway ignores unknown keys in this configuration.

Rules

Each Rule in the rules map has a type that sets how the Interceptor validates the record value. Gateway compiles every Rule when you create or update the Interceptor, and rejects invalid CEL expressions and JSON schemas. message is the error message that the producer receives when the Interceptor blocks a record for this Rule. When you don’t set it, the producer receives Data quality rule '<rule-name>' violated.

CEL Rules

A CEL expression has to return a boolean: true when the record is valid, false when it violates the Rule. Expressions can use these variables: The Interceptor parses value and key only when the expression uses them:
  • With a schema registry, it deserializes Avro, Protobuf and JSON schema records. Avro records become maps.
  • Without a schema registry, or when deserialization fails, it parses the bytes as JSON and skips a schema registry wire-format prefix.
Expressions can use the standard CEL operators, macros and functions, such as matches(), contains(), startsWith(), size(), has() and all(). The Interceptor doesn’t load CEL extension libraries, so functions such as lowerAscii() or isEmail() aren’t available.

JSON schema Rules

schema holds a JSON schema definition as a string. The Interceptor parses the raw record value as JSON, so records that start with a schema registry wire-format prefix violate this Rule. Use ENFORCE_AVRO or ENFORCE_SCHEMA_ID for schema-encoded records.

Built-in schema Rules

ENFORCE_AVRO and ENFORCE_SCHEMA_ID need a schema registry. Without schemaRegistryConfig or schemaRegistryConnectionName, Gateway rejects the Interceptor with No SchemaRegistryClient or AWSSchemaRegistryClient provided. With a Confluent-like schema registry, the Interceptor reads the schema ID from the __value_schema_id header first, then from the wire-format prefix of the value. With AWS Glue, it reads the value prefix only.

Evaluation

The Interceptor evaluates every record of every partition of the matching topics against all of its Rules. A record that the Interceptor can’t evaluate counts as a violation. For example, a value that isn’t valid JSON, a CEL expression that reads a missing field or an unreachable schema registry all count as violations, and they block the record when block is true. When block is true, the Interceptor stops evaluating a batch at its first violating record, because it rejects the whole batch anyway. Violation events and evaluation counts for a blocked batch only include the records up to and including that first violation.

Actions

If both block and mark are true, the Interceptor only blocks. With neither action, the Interceptor only tracks violation events.
A single invalid record rejects its whole record batch, including the valid records. If you need to keep valid records while you flag violations, use mark instead of block.

Producer error

When the Interceptor blocks a batch, the producer receives an INVALID_RECORD error for every record in it. The error is non-retriable. Its message lists the Rules that the first invalid record violates, joined with . : the Rule message if set, otherwise Data quality rule '<rule-name>' violated.

Mark header

The conduktor.dataquality.violations header holds a UTF-8 JSON object that maps Policy names to the Rules that the record violates:
When several Interceptors mark the same topic, each one adds or replaces its own Policy entry in the header. On a valid record, the Interceptor removes the entry of its Policy.

Dead letter topic

When dlq is true, the Interceptor copies each record that violates a Rule to dlqTopic, with these headers: Use the dead letter topic with mark or with no action, so that every violating record gets a copy in the dead letter topic.
Don’t combine block with a dead letter topic. When the Interceptor blocks a batch, the dead letter topic doesn’t capture the blocked records, so you can’t use it to find or replay them.

Violation events

The Interceptor always writes violation events to the violation audit topic of Gateway:
Gateway creates this topic at startup. To change its name, partitions, replication factor or retention, see the Gateway environment variables. Each record on the topic has:
  • a value encoded with the Avro JSON encoding of the DataQualityEvent envelope, holding either a violation event or an evaluation count event
  • a consoleDeploymentId header with the value of consoleDeploymentId
  • the violated topic as its key for violation events, and no key for evaluation count events

Violation event

The Interceptor writes a DataQualityViolation event for each record that violates a Rule, within the maxNumberOfViolationReportPerSecond limit.

Evaluation count event

The Interceptor periodically writes a DataQualityChecksCount event with the number of records it evaluated, valid or not. Use it to compute violation rates. Gateway writes it every 15 seconds by default. To change the interval, see the Gateway environment variables. cluster holds the Virtual Cluster name.

Minimum Gateway versions

DataQualityPlugin example

Mark the records of the customer-transaction topics that violate a Rule and copy them to a dead letter topic. The records still reach their topic.
Apply with:

SQL data quality Interceptor

The SQL data quality Interceptor validates the records of one topic against the WHERE clause of a SQL-like statement. A record is valid when it matches the clause.
  • Plugin class: io.conduktor.gateway.interceptor.safeguard.DataQualityProducerPlugin
  • License: requires a Gateway license that includes data quality.

Configure SQL data quality

Gateway rejects unknown keys in this configuration.

Statement

  • The Interceptor ignores the selected fields.
  • FROM takes a single topic name. The Interceptor matches it exactly, without regex. Put topic names that contain a dash in double quotes, for example FROM "our-orders".
  • WHERE takes conditions joined with AND. The Interceptor doesn’t support OR or IN. Approximate IN with REGEXP.
  • Supported operators: =, >, >=, <, <=, <> and REGEXP. REGEXP has to match the whole value.
  • The Interceptor compares numbers when one side is a number and strings otherwise.
Gateway rejects a statement it can’t parse with Cannot parse statement <statement>, and an unsupported operator with Comparison operator <operator> is not supported. By default, field names point to fields of the record value. Use dot notation for nested fields, for example address.street = 'Electric Avenue'. Fields in the WHERE clause have to exist in the record for the record to be valid. The Interceptor reads JSON values, and Avro and Protobuf values through a schema registry.

SQL actions

The Interceptor logs violations in the Gateway audit log for every action.

SQL dead letter topic

When you set deadLetterTopic, the Interceptor writes the invalid records to it for every action, including AUDIT_LOG_ONLY and THROTTLE. With BLOCK_WHOLE_BATCH, it writes the whole blocked batch, including its valid records. When addErrorHeader is true, it adds these headers:

SQL data quality example

Make sure that every order has a valid ID, a positive amount below one million cents, a supported currency and an ISO 8601 date. The Interceptor blocks the batches that contain an invalid order.
Apply with:

Schema payload validation Interceptor

The schema payload validation Interceptor checks that record values carry a schema ID registered in the schema registry and, optionally, that the values match their schema, including the constraints and CEL Rules defined in the schema.
  • Plugin class: io.conduktor.gateway.interceptor.safeguard.SchemaPayloadValidationPolicyPlugin
  • License: doesn’t require the data quality license feature.
The Interceptor only runs when Gateway has an audit log configured. It checks the record value only, not the key or the headers.

Configure schema payload validation

Without a schema registry, Gateway rejects the Interceptor with schemaRegistryConfig or schemaRegistryConnectionName is required. Gateway also rejects unknown keys in this configuration.

Schema payload checks

With schemaIdRequired: true, records with an empty value, such as tombstones, are invalid.
Schema-encoded records whose schema ID is stored in a record header, the format introduced in Confluent Platform 8.2, require Gateway 3.21.0 or later.

Schema constraints

With validateSchema: true, the Interceptor validates JSON schemas with standard JSON schema validation. In Avro schemas, it applies these field properties: Supported format values: byte, date, time, date-time, duration, uri, uri-reference, uri-template, url, email, hostname, ipv4, ipv6, regex, uuid, json-pointer, json-pointer-uri-fragment, relative-json-pointer. Avro and JSON schemas also accept CEL Rules:
  • An expression property on a field validates that field. The expression uses the field name, for example "expression": "age >= 0 && age <= 120".
  • A metadata.rules list at the root of the schema validates the whole value. Each Rule has a name, an expression that returns a boolean and a message that becomes the error message. Expressions use message for the value, for example message.age >= 18.
The Interceptor doesn’t apply constraints or CEL Rules to Protobuf schemas. CEL Rules in schemas support the standard CEL functions, the CEL strings extension and these functions: isEmail(), isHostname(), isIpv4(), isIpv6(), isUri(), isUriRef() and isUuid().

Schema payload actions

With BLOCK, the producer receives an error whose message starts with Request parameters do not satisfy the configured policy., followed by one of:
  • Topic '<topic>' with schemaId is required.
  • Topic '<topic>' with schemaId is not allowed.
  • Topic '<topic>' has invalid <avro|json|protobuf> schema payload: <errors>

Schema payload dead letter topic

When you set deadLetterTopic, the Interceptor writes each invalid record to it for every action, including INFO and THROTTLE. When addErrorHeader is true, it adds these headers:

Schema payload validation example

Block the records of the topic_1 topics that don’t carry a registered schema ID or don’t match their schema.
Apply with:

Topic required schema ID Interceptor

The topic required schema ID Interceptor checks that record values carry a schema ID, or that they don’t. Unlike the other schema checks on this page, it doesn’t call the schema registry, so it doesn’t check that the schema exists.
  • Plugin class: io.conduktor.gateway.interceptor.safeguard.TopicRequiredSchemaIdPolicyPlugin
  • License: doesn’t require the data quality license feature.

Configure topic required schema ID

Gateway rejects unknown keys in this configuration.

Schema ID detection

The Interceptor reads the schema ID from the __value_schema_id header first, then from the wire-format prefix of the value. It skips records with a null value, such as tombstones.
Schema-encoded records whose schema ID is stored in a record header, the format introduced in Confluent Platform 8.2, require Gateway 3.21.0 or later.

Topic required schema ID actions

With BLOCK, the producer receives one of these errors:

Topic required schema ID example

Apply with: