From our blog: The hidden pitfalls of Kafka’s schemaless data Why schemaless topics let bad data through, and how CEL and JSON Schema rules catch it at the proxy.
Common configuration
Schema registry
The DataQualityPlugin, SQL data quality and schema payload validation Interceptors read schema-encoded records through a schema registry. Gateway reads schemas either through an external connection or from configuration on the Interceptor itself.Schema-encoded records whose schema ID is stored in a record header, the format introduced in Confluent Platform 8.2, require Gateway 3.21.0 or later.
- External connection
- Inline configuration
Set Set either
schemaRegistryConnectionName to the name of a schema registry external connection, declared once when Gateway starts with variables that follow this pattern:For a connection declared as
GATEWAY_EXTERNAL_CONNECTION_DEFAULT_SR_CONFIG_*, the Interceptor reaches it with:schemaRegistryConnectionName or schemaRegistryConfig, never both. Gateway rejects a configuration that sets both with 400 Bad Request, rather than silently picking one.Remove a connection an Interceptor already names and the Interceptor stays visible through the API but stops applying to traffic. See what happens at startup.Dead letter topics
When an Interceptor writes to a dead letter topic for the first time, Gateway checks whether the topic exists on the backing Kafka cluster. If it doesn’t, Gateway creates it with the broker default partition count and replication factor. Each dead letter record keeps the original key, value and headers. The Interceptor also adds the headers listed in its own section.DataQualityPlugin Interceptor
The DataQualityPlugin Interceptor evaluates every record produced to matching topics against a set of Rules, then blocks or marks the records that violate them. Console also deploys it for data quality Policies: see the DataQualityPolicy reference for how Policy fields map to this configuration.- Plugin class:
io.conduktor.gateway.interceptor.dataquality.DataQualityPlugin - License: requires a Gateway license that includes data quality. Without it, Gateway rejects the Interceptor with
403 Feature 'Data Quality' is not included in your license.
Configure DataQualityPlugin
Gateway ignores unknown keys in this configuration.
Rules
Each Rule in therules map has a type that sets how the Interceptor validates the record value. Gateway compiles every Rule when you create or update the Interceptor, and rejects invalid CEL expressions and JSON schemas.
message is the error message that the producer receives when the Interceptor blocks a record for this Rule. When you don’t set it, the producer receives Data quality rule '<rule-name>' violated.
CEL Rules
A CEL expression has to return a boolean:true when the record is valid, false when it violates the Rule. Expressions can use these variables:
The Interceptor parses
value and key only when the expression uses them:
- With a schema registry, it deserializes Avro, Protobuf and JSON schema records. Avro records become maps.
- Without a schema registry, or when deserialization fails, it parses the bytes as JSON and skips a schema registry wire-format prefix.
matches(), contains(), startsWith(), size(), has() and all(). The Interceptor doesn’t load CEL extension libraries, so functions such as lowerAscii() or isEmail() aren’t available.
JSON schema Rules
schema holds a JSON schema definition as a string. The Interceptor parses the raw record value as JSON, so records that start with a schema registry wire-format prefix violate this Rule. Use ENFORCE_AVRO or ENFORCE_SCHEMA_ID for schema-encoded records.
Built-in schema Rules
ENFORCE_AVRO and ENFORCE_SCHEMA_ID need a schema registry. Without schemaRegistryConfig or schemaRegistryConnectionName, Gateway rejects the Interceptor with No SchemaRegistryClient or AWSSchemaRegistryClient provided.
With a Confluent-like schema registry, the Interceptor reads the schema ID from the __value_schema_id header first, then from the wire-format prefix of the value. With AWS Glue, it reads the value prefix only.
Evaluation
The Interceptor evaluates every record of every partition of the matching topics against all of its Rules. A record that the Interceptor can’t evaluate counts as a violation. For example, a value that isn’t valid JSON, a CEL expression that reads a missing field or an unreachable schema registry all count as violations, and they block the record whenblock is true.
When block is true, the Interceptor stops evaluating a batch at its first violating record, because it rejects the whole batch anyway. Violation events and evaluation counts for a blocked batch only include the records up to and including that first violation.
Actions
If both
block and mark are true, the Interceptor only blocks. With neither action, the Interceptor only tracks violation events.
Producer error
When the Interceptor blocks a batch, the producer receives anINVALID_RECORD error for every record in it. The error is non-retriable. Its message lists the Rules that the first invalid record violates, joined with . : the Rule message if set, otherwise Data quality rule '<rule-name>' violated.
Mark header
Theconduktor.dataquality.violations header holds a UTF-8 JSON object that maps Policy names to the Rules that the record violates:
Dead letter topic
Whendlq is true, the Interceptor copies each record that violates a Rule to dlqTopic, with these headers:
Use the dead letter topic with
mark or with no action, so that every violating record gets a copy in the dead letter topic.
Violation events
The Interceptor always writes violation events to the violation audit topic of Gateway:- a value encoded with the Avro JSON encoding of the
DataQualityEventenvelope, holding either a violation event or an evaluation count event - a
consoleDeploymentIdheader with the value ofconsoleDeploymentId - the violated topic as its key for violation events, and no key for evaluation count events
Violation event
The Interceptor writes aDataQualityViolation event for each record that violates a Rule, within the maxNumberOfViolationReportPerSecond limit.
Evaluation count event
The Interceptor periodically writes aDataQualityChecksCount event with the number of records it evaluated, valid or not. Use it to compute violation rates. Gateway writes it every 15 seconds by default. To change the interval, see the Gateway environment variables.
cluster holds the Virtual Cluster name.
Minimum Gateway versions
DataQualityPlugin example
Mark the records of thecustomer-transaction topics that violate a Rule and copy them to a dead letter topic. The records still reach their topic.
- Conduktor CLI
- curl
SQL data quality Interceptor
The SQL data quality Interceptor validates the records of one topic against theWHERE clause of a SQL-like statement. A record is valid when it matches the clause.
- Plugin class:
io.conduktor.gateway.interceptor.safeguard.DataQualityProducerPlugin - License: requires a Gateway license that includes data quality.
Configure SQL data quality
Gateway rejects unknown keys in this configuration.
Statement
- The Interceptor ignores the selected fields.
FROMtakes a single topic name. The Interceptor matches it exactly, without regex. Put topic names that contain a dash in double quotes, for exampleFROM "our-orders".WHEREtakes conditions joined withAND. The Interceptor doesn’t supportORorIN. ApproximateINwithREGEXP.- Supported operators:
=,>,>=,<,<=,<>andREGEXP.REGEXPhas to match the whole value. - The Interceptor compares numbers when one side is a number and strings otherwise.
Cannot parse statement <statement>, and an unsupported operator with Comparison operator <operator> is not supported.
By default, field names point to fields of the record value. Use dot notation for nested fields, for example address.street = 'Electric Avenue'. Fields in the WHERE clause have to exist in the record for the record to be valid. The Interceptor reads JSON values, and Avro and Protobuf values through a schema registry.
SQL actions
The Interceptor logs violations in the Gateway audit log for every action.
SQL dead letter topic
When you setdeadLetterTopic, the Interceptor writes the invalid records to it for every action, including AUDIT_LOG_ONLY and THROTTLE. With BLOCK_WHOLE_BATCH, it writes the whole blocked batch, including its valid records.
When addErrorHeader is true, it adds these headers:
SQL data quality example
Make sure that every order has a valid ID, a positive amount below one million cents, a supported currency and an ISO 8601 date. The Interceptor blocks the batches that contain an invalid order.- Conduktor CLI
- curl
Schema payload validation Interceptor
The schema payload validation Interceptor checks that record values carry a schema ID registered in the schema registry and, optionally, that the values match their schema, including the constraints and CEL Rules defined in the schema.- Plugin class:
io.conduktor.gateway.interceptor.safeguard.SchemaPayloadValidationPolicyPlugin - License: doesn’t require the data quality license feature.
Configure schema payload validation
Without a schema registry, Gateway rejects the Interceptor with
schemaRegistryConfig or schemaRegistryConnectionName is required. Gateway also rejects unknown keys in this configuration.
Schema payload checks
With
schemaIdRequired: true, records with an empty value, such as tombstones, are invalid.
Schema-encoded records whose schema ID is stored in a record header, the format introduced in Confluent Platform 8.2, require Gateway 3.21.0 or later.
Schema constraints
WithvalidateSchema: true, the Interceptor validates JSON schemas with standard JSON schema validation. In Avro schemas, it applies these field properties:
Supported
format values: byte, date, time, date-time, duration, uri, uri-reference, uri-template, url, email, hostname, ipv4, ipv6, regex, uuid, json-pointer, json-pointer-uri-fragment, relative-json-pointer.
Avro and JSON schemas also accept CEL Rules:
- An
expressionproperty on a field validates that field. The expression uses the field name, for example"expression": "age >= 0 && age <= 120". - A
metadata.ruleslist at the root of the schema validates the whole value. Each Rule has aname, anexpressionthat returns a boolean and amessagethat becomes the error message. Expressions usemessagefor the value, for examplemessage.age >= 18.
isEmail(), isHostname(), isIpv4(), isIpv6(), isUri(), isUriRef() and isUuid().
JSON schema with constraints and CEL Rules
JSON schema with constraints and CEL Rules
Avro schema with constraints and CEL Rules
Avro schema with constraints and CEL Rules
Schema payload actions
With
BLOCK, the producer receives an error whose message starts with Request parameters do not satisfy the configured policy., followed by one of:
Topic '<topic>' with schemaId is required.Topic '<topic>' with schemaId is not allowed.Topic '<topic>' has invalid <avro|json|protobuf> schema payload: <errors>
Schema payload dead letter topic
When you setdeadLetterTopic, the Interceptor writes each invalid record to it for every action, including INFO and THROTTLE. When addErrorHeader is true, it adds these headers:
Schema payload validation example
Block the records of thetopic_1 topics that don’t carry a registered schema ID or don’t match their schema.
- Conduktor CLI
- curl
Topic required schema ID Interceptor
The topic required schema ID Interceptor checks that record values carry a schema ID, or that they don’t. Unlike the other schema checks on this page, it doesn’t call the schema registry, so it doesn’t check that the schema exists.- Plugin class:
io.conduktor.gateway.interceptor.safeguard.TopicRequiredSchemaIdPolicyPlugin - License: doesn’t require the data quality license feature.
Configure topic required schema ID
Gateway rejects unknown keys in this configuration.
Schema ID detection
The Interceptor reads the schema ID from the__value_schema_id header first, then from the wire-format prefix of the value. It skips records with a null value, such as tombstones.
Schema-encoded records whose schema ID is stored in a record header, the format introduced in Confluent Platform 8.2, require Gateway 3.21.0 or later.
Topic required schema ID actions
With
BLOCK, the producer receives one of these errors:
Topic required schema ID example
- Conduktor CLI
- curl