Console resources

​

ConsoleGroup

• API key(s): AdminToken
• Managed with: UI, CLI, API, TF
• Labels support: Missing

---
apiVersion: iam/v2
kind: Group
metadata:
  name: developers-a
spec:
  displayName: "Developers Team A"
  description: "Members of the Team A - Developers"
  externalGroups:
    - "LDAP-GRP-A-DEV"
  externalGroupRegex:
    - "LDAP*"
  members:
    - [email protected]
    - [email protected]
  permissions:
    - resourceType: TOPIC
      cluster: shadow-it
      patternType: PREFIXED
      name: toto-
      permissions:
        - topicViewConfig
        - topicConsume
        - topicProduce

• Console
  • Members of the Group are given the associated permissions in the UI over the resources
  • Members of the LDAP or OIDC groups will be automatically added or removed upon login
• Kafka
  • No side effects

​

ConsoleUser

• API key(s): AdminToken
• Managed with: UI, CLI, API, TF
• Labels support: Missing

---
apiVersion: iam/v2
kind: User
metadata:
  name: [email protected]
spec:
  firstName: "John"
  lastName: "Doe"
  permissions:
    - resourceType: PLATFORM
      permissions:
        - taasView
        - datamaskingView
    - resourceType: TOPIC
      cluster: shadow-it
      patternType: PREFIXED
      name: toto-
      permissions:
        - topicViewConfig
        - topicConsume
        - topicProduce

• Console
  • User is given the associated permissions in the UI over the resources
• Kafka
  • No side effects

​

KafkaCluster

• API key(s): AdminToken
• Managed with: UI, CLI, API, TF
• Labels support: Partial

---
apiVersion: console/v2
kind: KafkaCluster
metadata:
  name: my-dev-cluster
spec:
  displayName: "My Dev Cluster"
  icon: "kafka"
  color: "#000000"
  bootstrapServers: "localhost:9092"
  ignoreUntrustedCertificate: false
  properties:
    sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret";
    security.protocol: SASL_SSL
    sasl.mechanism: PLAIN
  schemaRegistry:
    type: "ConfluentLike"
    url: http://localhost:8080
    security:
      type: BasicAuth
      username: some_user
      password: some_password
    ignoreUntrustedCertificate: false
  kafkaFlavor:
    type: "Confluent"
    key: "string"
    secret: "string"
    confluentEnvironmentId: "string"
    confluentClusterId: "string"

​

Schema registry

​

Confluent or Confluent-like Registry

spec:
  schemaRegistry:
    type: "ConfluentLike"
    url: http://localhost:8080
    ignoreUntrustedCertificate: false
    security:
      type: BasicAuth
      username: some_user
      password: some_password

​

AWS Glue registry

spec:
  schemaRegistry:
    type: "Glue"
    region: eu-west-1
    registryName: default
    security:
      type: Credentials
      accessKeyId: accessKey
      secretKey: secretKey

    security:
      type: Credentials
      accessKeyId: AKIAIOSFODNN7EXAMPLE
      secretKey: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

    security:
      type: FromContext
      profile: default

    security:
      type: FromRole
      role: arn:aws:iam::123456789012:role/example-role

​

Kafka provider

• Confluent Cloud service account support
• Confluent Cloud API key support

spec:
  kafkaFlavor:
    type: "Confluent"
    key: "yourApiKey123456"
    secret: "yourApiSecret123456"
    confluentEnvironmentId: "env-12345"
    confluentClusterId: "lkc-67890"

• Aiven service accounts support
• Aiven ACLs support

spec:
  kafkaFlavor:
    type: "Aiven"
    apiToken: "a1b2c3d4e5f6g7h8i9j0"
    project: "my-kafka-project"
    serviceName: "my-kafka-service"

• Interceptor support

spec:
  kafkaFlavor:
    type: "Gateway"
    url: "http://gateway:8888"
    user: "admin"
    password: "admin"
    virtualCluster: passthrough

​

Icon sets

​

KafkaConnectCluster

• API key(s): AdminToken
• Managed with: API, CLI, UI, TF
• Labels support: Partial

---
apiVersion: console/v2
kind: KafkaConnectCluster
metadata:
  cluster: my-dev-cluster
  name: connect-1
spec:
  displayName: "Connect 1"
  urls: "http://localhost:8083"
  headers:
    X-PROJECT-HEADER: value
    AnotherHeader: test
  ignoreUntrustedCertificate: false
  security:
    type: "BasicAuth"
    username: "toto"
    password: "my-secret"

​

KsqlDBCluster

• API key(s): AdminToken
• Managed with: UI, CLI, API
• Labels support: Missing

---
apiVersion: console/v2
kind: KsqlDBCluster
metadata:
  cluster: my-dev-cluster
  name: ksql-1
spec:
  displayName: "KSQL 1"
  url: "http://localhost:8088"
  ignoreUntrustedCertificate: false
  security:
    type: "BasicAuth"
    username: "toto"
    password: "my-secret"

• metadata.cluster has to be a valid KafkaCluster name.
• spec.url has to be a single URL of a KsqlDB cluster.
• spec.ignoreUntrustedCertificate (optional), default is false. Has to be true or false.
• spec.headers (optional) has to be key-value pairs of HTTP headers.
• spec.security.type (optional) has to be BasicAuth, BearerToken or SSLAuth. Find out more.

​

Integrations

• API key(s): AdminToken
• Managed with: UI, CLI, API
• **Labels support: Missing

---
apiVersion: v3
kind: Integration
metadata:
    name: Teams
spec: {}

---
apiVersion: v3
kind: Integration
metadata:
    name: Slack
spec:
    config:
        token: myToken
        type: Slack
---

• metadata.name has to be a valid integration name, here is the currently available integration: [Teams, Slack, Webhook, Email]
• spec is equal to an empty object when you don’t have any configuration attached to a specific integration: [Teams, Webhook]
• spec.config should be a valid configuration for [Slack, Email], the spec.config.type should match the name of the integration

​

Alerts

• API key(s): AdminToken, AppToken
• Managed with: UI, CLI, API
• Labels support: Missing

---
apiVersion: console/v3
kind: Alert
metadata:
  name: messages-in-dead-letter-queue
  group: support-team # will be the owner of the alert, can be either a user, a group or an appInstance
  # user: [email protected]
  # appInstance: my-app-instance
spec:
  cluster: my-dev-cluster
  type: TopicAlert
  topicName: wikipedia-parsed-DLQ
  metric: MessageCount
  operator: GreaterThan
  threshold: 0
  description: "Alert for monitoring messages in dead letter queue"
  displayName: "DLQ message count alert"
  destination:
    type: Slack
    channel: "alerts-p1"

• metadata.user|metadata.group|metadata.appInstance has to be a valid user, group or appInstance.
• metadata.destination.type can be either Slack, Teams or Webhook. When set to:
  • Slack: has to be a valid Slack channel ID
  • Teams: has to be a valid Teams webhook URL
  • Webhook:
    • spec.destination.url has to be a valid URL
    • spec.destination.method has to be GET, POST, PUT or DELETE
    • spec.destination.headers (optional) has to be key-value pairs of HTTP headers
    • spec.destination.authentication.type (optional) has to be BasicAuth (define spec.destination.authentication.username and spec.destination.authentication.password) or BearerToken (define spec.destination.authentication.token).
• spec.cluster has to be a valid KafkaCluster name.
• spec.type has to be BrokerAlert,TopicAlert, KafkaConnectAlert or ConsumerGroupAlert. When set to:
  • BrokerAlert: spec.metric has to be MessageIn, MessageOut, MessageSize, OfflinePartitionCount, PartitionCount, UnderMinIsrPartitionCount or UnderReplicatedPartitionCount.
  • TopicAlert: spec.metric has to be MessageCount, MessageIn, MessageOut or MessageSize and the spec.topicName has to be a Kafka topic that the owner can access.
  • KafkaConnectAlert:spec.metric has to be FailedTaskCount;spec.connectName has to be a valid KafkaConnect cluster associated to this spec.cluster Kafka cluster and spec.connectorName has to be a Kafka Connect Connector that the owner can access.
  • ConsumerGroupAlert:spec.metric has to be OffsetLag or TimeLag and spec.consumerGroupName has to be a Kafka Consumer group that the owner can access.
• spec.metric depends on the spec.type.
• spec.operator has to be GreaterThan, GreaterThanOrEqual, LessThan, LessThanOrEqual or NotEqual.
• spec.threshold has to be a number.
• spec.description (optional) provides a text description of the alert.
• spec.displayName (optional) provides a display name for the alert.
• spec.disable (optional), default is false. Has to be true or false.

​

Partner Zones

• API key(s): AdminToken
• Managed with: UI, CLI, API
• Labels support: Partial

---
apiVersion: console/v2
kind: PartnerZone
metadata:
  name: external-partner-zone
spec:
  displayName: External Partner Zone
  description: An external partner to exchange data with.
  url: https://partner1.com
  partner:
    name: John Doe
    role: Data analyst
    email: [email protected]
    phone: 07827 837 177
  cluster: cdk-gateway
  underlyingCluster: cluster1
  authenticationMode:
    serviceAccount: partner-external-partner
    type: PLAIN
  vclusterName: custom-vcluster-name
  topics:
    - name: topic-a
      backingTopic: kafka-topic-a
      permission: WRITE
    - name: topic-b
      backingTopic: kafka-topic-a
      permission: READ
  trafficControlPolicies:
    maxProduceRate: 1e+06
    maxConsumeRate: 1e+06
    limitCommitOffset: 30
  headers:
    addOnProduce:
        - key: partner-name
          value: external-analytics-partner
          overrideIfExists: false
        - key: client-info
          value: "Client:{{clientId}}, from IP:{{userIp}}"
          overrideIfExists: true
        - key: kafka-api
          value: "Kafka API Key:{{apiKey}}, version {{apiKeyVersion}}"
          overrideIfExists: true
        - key: produce-metadata
          value: "User:{{user}}, via Gateway:{{gatewayHost}}, at timestampMillis:{{timestampMillis}}"
          overrideIfExists: true
    removeOnConsume:
        - keyRegex: my_team_prefix.*

• spec.displayName is mandatory.
• spec.description, spec.url and spec.partner (optional), useful for context information.
• spec.cluster has to be a valid Console cluster technical ID with the Provider configured as Gateway.
• spec.underlyingCluster has to be a valid Console cluster technical ID where the cluster has to be defined under the Gateway defined by the spec.gatewayClusterId.
  • When not specified, the value is inferred to be equal to the spec.gatewayClusterId field, hence selecting the main cluster behind Gateway by default.
• spec.authenticationMode.type must be one of [PLAIN, OAUTHBEARER, MTLS]. See authentication examples for detailed configuration.
• spec.authenticationMode.serviceAccount requirements depend on the authentication type:
  • PLAIN: Any unique identifier for your partner (e.g., partner-external-partner). This will be created as a local Gateway service account automatically if it doesn’t exist.
  • OAUTHBEARER: Must match the “sub” claim in the partner’s OAuth/OIDC token (e.g., oauth-partner-service-account). The partner needs to authenticate using their OAuth provider.
  • MTLS: Must match the client’s Distinguished Name (DN) from their certificate, unless you’ve modified GATEWAY_SSL_PRINCIPAL_MAPPING_RULES (e.g., CN=partner-client,OU=Engineering,O=PartnerCorp,C=US).
• spec.vclusterName (optional), custom name for the Virtual Cluster. If not provided, it will be auto-generated.
• topics[].name is the name of the topic as it should appear to your external partner. This can be different from backingTopic.
• topics[].backingTopic is the internal name of the topic that you want to share.
• topics[].permission has to be set to either READ or WRITE (which includes READ).
• trafficControlPolicies.maxProduceRate (optional), sets the maximum rate (in bytes/s) at which the partner can produce messages to the topics per Gateway node.
• trafficControlPolicies.maxConsumeRate (optional), sets the maximum rate (in bytes/s) at which the partner can consume messages from the topics per Gateway node.
• trafficControlPolicies.limitCommitOffset (optional), sets the maximum number of commit requests (in requests/minute) that the partner can make per Gateway node.
• headers.addOnProduce (optional), list of headers to inject when producing messages. The value field supports special variables: {{user}}, {{userIp}}, {{clientId}}, {{apiKey}}, {{apiKeyVersion}}, {{gatewayHost}}, {{timestampMillis}}.
• headers.removeOnConsume (optional), list of header key patterns (regex) to remove when consuming messages.

• metadata.updatedAt (by consecutive get from the CLI/API).
• metadata.status (by consecutive get from the CLI/API.) Possible values are PENDING, READY or FAILED.
• metadata.failedReason will be populated in case of FAILED status.
• The service account will be created if it doesn’t exist and will be granted the permissions as declared in spec.topics.
• The traffic control policies will be applied to the service account.

​

DataQualityRule

• API key(s): AdminToken
• Managed with: UI, CLI, API
• Labels support: Yes

---
apiVersion: v1
kind: DataQualityRule
metadata:
  name: email-validation-rule
  labels:
    category: validation
    team: data-quality
spec:
  type: Cel
  displayName: Email validation rule
  description: Validates that email field contains a valid email address
  celExpression: 'record.email.matches("^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$")'
  customErrorMessage: Email address is not valid

---
apiVersion: v1
kind: DataQualityRule
metadata:
  name: user-schema-rule
  labels:
    category: schema
spec:
  type: JsonSchema
  displayName: User schema validation
  description: Validates user records against JSON schema
  schema:
    type: object
    required:
      - userId
      - email
    properties:
      userId:
        type: string
      email:
        type: string
        format: email
      age:
        type: integer
        minimum: 0
  customErrorMessage: Record does not match user schema

• metadata.name is mandatory and must be unique.
• metadata.labels is optional and can be used to organize rules.
• spec.type has to be one of: Cel, JsonSchema, EnforceAvro, EnforceSchemaId.
• spec.displayName is mandatory and is shown in the Console UI.
• spec.description is optional but recommended to explain what the rule validates.
• spec.customErrorMessage is optional and provides a custom error message when validation fails.
• For Cel type:
  • spec.celExpression is mandatory and contains the CEL (Common Expression Language) expression to validate records.
• For JsonSchema type:
  • spec.schema is mandatory and contains the JSON schema object to validate records against.
• For EnforceAvro type:
  • Validates that records conform to an Avro schema.
• For EnforceSchemaId type:
  • Validates that records use a specific schema ID from Schema Registry.

• metadata.createdAt (read-only timestamp).
• metadata.updatedAt (read-only timestamp).
• metadata.createdBy (read-only user identifier).
• metadata.updatedBy (read-only user identifier).
• metadata.attachedPolicies (read-only list of policies using this rule).

​

DataQualityPolicy

• API key(s): AdminToken
• Managed with: UI, CLI, API
• Labels support: No

---
apiVersion: v1
kind: DataQualityPolicy
metadata:
  name: user-data-validation-policy
spec:
  displayName: User data validation policy
  description: Validates all user data before it is produced to topics
  rules:
    - email-validation-rule
    - user-schema-rule
  targets:
    - cluster: main-cluster
      topic: users
      patternType: LITERAL
    - cluster: main-cluster
      topic: user-events-
      patternType: PREFIXED
  actions:
    block:
      enabled: true
    mark:
      enabled: false

• metadata.name is mandatory and must be unique.
• spec.displayName is mandatory and is shown in the Console UI.
• spec.description is optional but recommended to explain the policy’s purpose.
• spec.rules is a list of DataQualityRule names to apply (references to rules created separately).
• spec.targets is mandatory and defines which topics the policy applies to:
  • cluster is a valid Kafka cluster technical ID.
  • topic is the topic name or prefix.
  • patternType has to be either LITERAL (exact match) or PREFIXED (prefix match).
• spec.actions defines what happens when a rule violation occurs:
  • block.enabled (optional, defaults to false) - when true, blocks messages that violate rules from being produced. Only available if the configured cluster is a Conduktor Gateway cluster with the appropriate license.
  • mark.enabled (optional, defaults to false) - when true, marks messages that violate rules with a header but allows production.

• metadata.nameForMetrics (read-only identifier used in metrics).
• metadata.group (read-only group identifier).
• metadata.createdAt (read-only timestamp).
• metadata.updatedAt (read-only timestamp).
• metadata.createdBy (read-only user identifier).
• metadata.updatedBy (read-only user identifier).
• metadata.status (read-only status: failed, pending, or ready).
• metadata.statuses (read-only per-cluster deployment status).
• metadata.metrics (read-only metrics showing evaluation and violation counts).
• metadata.attachedRules (read-only list of rules attached to this policy).
• Data quality rules are enforced on the specified topics.
• Messages violating rules are blocked or marked based on the configured actions.

​

HTTP security properties

​

Basic authentication

  security:
    type: "BasicAuth"
    username: "toto"
    password: "my-secret"

​

Bearer token

  security:
    type: "BearerToken"
    token: "toto"

​

mTLS/client certificate

  security:
    type: "SSLAuth"
    key: |
      -----BEGIN PRIVATE KEY-----
      MIIOXzCCDUegAwIBAgIRAPRytMVYJNUgCbhnA+eYumgwDQYJKoZIhvcNAQELBQAw
      ...
      IFyCs+xkcgvHFtBjjel4pnIET0agtbGJbGDEQBNxX+i4MDA=
      -----END PRIVATE KEY-----
    certificateChain: |
      -----BEGIN CERTIFICATE-----
      MIIOXzCCDUegAwIBAgIRAPRytMVYJNUgCbhnA+eYumgwDQYJKoZIhvcNAQELBQAw
      RjELMAkGA1UEBhMCVVMxIjAgBgNVBAoTGUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBM
      ...
      8/s+YDKveNdoeQoAmGQpUmxhvJ9rbNYj+4jiaujkfxT/6WtFN8N95r+k3W/1K4hs
      IFyCs+xkcgvHFtBjjel4pnIET0agtbGJbGDEQBNxX+i4MDA=
      -----END CERTIFICATE-----

​

Permissions

• Topic permissions
• Subject permissions
• ConsumerGroup permissions
• Cluster permissions
• KafkaConnect permissions
• KsqlDB permissions
• Console permissions

​

Topic permissions

# Grants consume, produce and view config to all topics toto-* on shadow-it cluster
- resourceType: TOPIC
  cluster: shadow-it
  patternType: PREFIXED
  name: toto-
  permissions:
    - topicViewConfig
    - topicConsume
    - topicProduce

• resourceType: TOPIC
• cluster is a valid Kafka cluster
• patternType is either PREFIXED or LITERAL
• name is the name of the topic or topic prefix to apply the permissions to
• permissions is a list of valid topic permissions

​

Subject permissions

# Grants view and edit compatibility to all subjects starting with sub-* on shadow-it cluster
- resourceType: SUBJECT
  cluster: shadow-it
  patternType: PREFIXED
  name: sub-
  permissions:
    - subjectView
    - subjectEditCompatibility

• resourceType: SUBJECT
• cluster is a valid Kafka cluster
• patternType is either PREFIXED or LITERAL
• name is the name of the subject or subject prefix to apply the permissions to
• permissions is a list of valid subject permissions

​

ConsumerGroup permissions

# Grants view and reset on all consumer groups starting with group-* on shadow-it cluster
- resourceType: CONSUMER_GROUP
  cluster: shadow-it
  patternType: PREFIXED
  name: group-
  permissions:
    - consumerGroupView
    - consumerGroupReset

• resourceType: CONSUMER_GROUP
• cluster is a valid Kafka cluster
• patternType is either PREFIXED or LITERAL
• name is the name of the consumer group or consumer group prefix to apply the permissions to
• permissions is a list of valid consumer group permissions

​

Cluster permissions

# Grants view and edit broker, edit schema registry compatibility, view ACL and manage ACL on shadow-it cluster
- resourceType: CLUSTER
  name: shadow-it
  permissions:
    - clusterViewBroker
    - clusterEditSRCompatibility
    - clusterEditBroker
    - clusterViewACL
    - clusterManageACL

• resourceType: CLUSTER
• name is the name of the cluster to apply the permissions to
  • Use * for all clusters
• permissions is a list of valid cluster permissions

​

KafkaConnect permissions

# Grants create and delete on all connectors starting with connector-* on shadow-it cluster and kafka-connect-cluster
- resourceType: KAFKA_CONNECT
  cluster: shadow-it
  kafkaConnect: kafka-connect-cluster
  patternType: PREFIXED
  name: connector-
  permissions:
    - kafkaConnectorCreate
    - kafkaConnectorDelete

• resourceType: KAFKA_CONNECT
• cluster is a valid Kafka cluster
• kafkaConnect is a valid Kafka Connect cluster
• patternType is either PREFIXED or LITERAL
• name is the name of the connector or connector prefix to apply the permissions to
• permissions is a list of valid Kafka Connect permissions

​

KsqlDB permissions

# Grants all permissions on KsqlDB cluster ksql-cluster
- resourceType: KSQLDB
  cluster: shadow-it
  ksqlDB: ksql-cluster
  permissions:
    - ksqldbAccess

• resourceType: KSQLDB
• cluster is a valid Kafka cluster
• ksqlDB is a valid Kafka Connect cluster
• permissions is a list of valid KsqlDB permissions

​

Console permissions

# Grants Console permissions
- resourceType: PLATFORM
  permissions:
    - userView
    - datamaskingView

• resourceType: PLATFORM
• permissions is a list of valid Console permissions