Skip to main content
Quick navigation

Yes, SQL topic works with Avro/ProtoBuf

View the full demo in realtime​

You can either follow all the steps manually, or watch the recording

Review the docker compose environment​

As can be seen from docker-compose.yaml the demo environment consists of the following services:

  • gateway1
  • gateway2
  • kafka-client
  • kafka1
  • kafka2
  • kafka3
  • schema-registry
cat docker-compose.yaml

Starting the docker environment​

Start all your docker processes, wait for them to be up and ready, then run in background

  • --wait: Wait for services to be running|healthy. Implies detached mode.
  • --detach: Detached mode: Run containers in the background
docker compose up --detach --wait

Creating topic cars on gateway1​

Creating on gateway1:

  • Topic cars with partitions:1 and replication-factor:1
kafka-topics \
    --bootstrap-server localhost:6969 \
    --replication-factor 1 \
    --partitions 1 \
    --create --if-not-exists \
    --topic cars

Listing topics in gateway1​

kafka-topics \
    --bootstrap-server localhost:6969 \
    --list

Produce avro payload​

schema='{
            "type": "record",
            "name": "car",
            "fields": [
              {"name": "type", "type": "string"},
              {"name": "price", "type": "long"},
              {"name": "color", "type": "string"}
            ]
          }'
echo '{"type":"Sports","price":75,"color":"blue"}' | \
    kafka-avro-console-producer  \
        --bootstrap-server localhost:6969 \
        --topic cars \
        --property schema.registry.url=http://localhost:8081 \
        --property "value.schema=$schema"

echo '{"type":"SUV","price":55,"color":"red"}' | \
    kafka-avro-console-producer  \
        --bootstrap-server localhost:6969 \
        --topic cars \
        --property schema.registry.url=http://localhost:8081 \
        --property "value.schema=$schema"

Consume the avro payload back​

kafka-avro-console-consumer  \
    --bootstrap-server localhost:6969 \
    --topic cars \
    --property schema.registry.url=http://localhost:8081 \
    --from-beginning \
    --max-messages 2 2>&1 | grep "{" | jq

Creating topic red-cars on gateway1​

Creating on gateway1:

  • Topic red-cars with partitions:1 and replication-factor:1
kafka-topics \
    --bootstrap-server localhost:6969 \
    --replication-factor 1 \
    --partitions 1 \
    --create --if-not-exists \
    --topic red-cars

Adding interceptor red-cars​

Let's create the interceptor to filter out the red cars from the cars topic.

step-10-red-cars-interceptor.json:

{
  "kind" : "Interceptor",
  "apiVersion" : "gateway/v2",
  "metadata" : {
    "name" : "red-cars"
  },
  "spec" : {
    "comment" : "Adding interceptor: red-cars",
    "pluginClass" : "io.conduktor.gateway.interceptor.VirtualSqlTopicPlugin",
    "priority" : 100,
    "config" : {
      "virtualTopic" : "red-cars",
      "statement" : "SELECT * FROM cars WHERE color = 'red'",
      "schemaRegistryConfig" : {
        "host" : "http://schema-registry:8081"
      }
    }
  }
}
curl \
    --silent \
    --request PUT "http://localhost:8888/gateway/v2/interceptor" \
    --header "Content-Type: application/json" \
    --user "admin:conduktor" \
    --data @step-10-red-cars-interceptor.json | jq

Listing interceptors​

Listing interceptors on gateway1

curl \
    --silent \
    --request GET "http://localhost:8888/gateway/v2/interceptor" \
    --user "admin:conduktor" | jq

kafka-avro-console-consumer  \
    --bootstrap-server localhost:6969 \
    --topic red-cars \
    --property schema.registry.url=http://localhost:8081 \
    --from-beginning \
    --max-messages 1 2>&1 | grep "{" | jq

Tearing down the docker environment​

Remove all your docker processes and associated volumes

  • --volumes: Remove named volumes declared in the "volumes" section of the Compose file and anonymous volumes attached to containers.
docker compose down --volumes

Conclusion

SQL virtual topics can also work on data serialized with Schema Registry!