Skip to main content
Quick navigation

Kafka Resources

Kafka Resources

Topic

Creates a Topic in Kafka.

API Keys: Admin API Key Application API Key
Managed with: CLI API Console UI

---
apiVersion: v2
kind: Topic
metadata:
cluster: shadow-it
name: click.event-stream.avro
spec:
replicationFactor: 3
partitions: 3
configs:
min.insync.replicas: '2'
cleanup.policy: delete
retention.ms: '60000'

Topic checks:

  • spec.cluster is a valid Kafka Cluster
  • metadata.name must belong to the Application Instances
  • spec.replicationFactor and spec.partitions are immutable and cannot be modified once the topic is created
  • All properties are validated against TopicPolicies attached to the Application Instance

Side effect in Console & Kafka:

Subject

Not implemented yet

This concept will be available in a future version

Creates a Subject in the Schema Registry.

API Keys: Admin API Key Application API Key
Managed with: CLI API Console UI

Local file

---
apiVersion: v1
kind: Subject
metadata:
cluster: shadow-it
name: myPrefix.topic-value
spec:
schemaFile: schemas/topic.avsc # relative to conduktor CLI execution context
format: AVRO
compatibility: BACKWARD

Inline

---
apiVersion: v1
kind: Subject
metadata:
cluster: shadow-it
name: myPrefix.topic-value
spec:
schema: |
{
"type": "long"
}

Schema Reference

---
apiVersion: v1
kind: Subject
metadata:
cluster: shadow-it
name: myPrefix.topic-value
spec:
schema: |
{
"type": "record",
"namespace": "com.schema.avro",
"name": "Client",
"fields": [
{
"name": "name",
"type": "string"
},
{
"name": "address",
"type": "com.schema.avro.Address"
}
]
}
references:
- name: com.schema.avro.Address
subject: commons.address-value
version: 1

Subject checks:

  • metadata.cluster is a valid Kafka Cluster
  • metadata.name must belong to the Application Instance
  • One of spec.schema or spec.schemaFile must be present. schema requires an inline schema, schemaFile requires a path to a file that contains the schema (relative to the CLI execution path)
  • spec.format is optional. Defines the schema format: AVRO (default), PROTOBUF, JSON
  • spec.compatibility is optional. Defines the subject compatibility mode: GLOBAL (default), BACKWARD, BACKWARD_TRANSITIVE, FORWARD, FORWARD_TRANSITIVE, FULL, FULL_TRANSITIVE, NONE
    • Use GLOBAL if you want the compatibility mode to be the one defined at the Schema Registry global level
  • spec.references is optional. It specifies the names of referenced schemas

Side effect in Console & Kafka:

Connector

Not implemented yet

This concept will be available in a future version

API Keys: Admin API Key Application API Key
Managed with: CLI API Console UI

---
apiVersion: v1
kind: Connector
metadata:
name: myPrefix.myConnector
spec:
connectCluster: myConnectCluster
config:
connector.class: io.connect.jdbc.JdbcSourceConnector
tasks.max: '1'
topics: myPrefix.myTopic
connection.url: "jdbc:mysql://127.0.0.1:3306/sample?verifyServerCertificate=false&useSSL=true&requireSSL=true"
consumer.override.sasl.jaas.config: o.a.k.s.s.ScramLoginModule required username="<user>" password="<password>";