Console Resources

ConsoleGroup

API Keys: Admin API Key
Managed with: API CLI Terraform Console UI

Creates a Group with members and permissions in Console

CLI

---
apiVersion: iam/v2
kind: Group
metadata:
  name: developers-a
spec:
  displayName: "Developers Team A"
  description: "Members of the Team A - Developers"
  externalGroups: 
    - "LDAP-GRP-A-DEV"
  members:
    - member1@company.org
    - member2@company.org
  permissions:
    - resourceType: TOPIC
      cluster: shadow-it
      patternType: PREFIXED
      name: toto-
      permissions:
        - topicViewConfig
        - topicConsume
        - topicProduce

Terraform

resource "conduktor_group_v2" "developers-a" {
  name = "developers-a"
  spec {
    display_name = "Developers Team A"
    description  = "Members of the Team A - Developers"
    externalGroups = [ "LDAP-GRP-A-DEV" ]
    members      = [ "member1@company.org", "member1@company.org" ]
    permissions  = [
     {
        resource_type = "TOPIC"
        cluster       = "shadow-it"
        patternType   = "PREFIXED"
        name          = "toto-"
        permissions   = ["topicViewConfig", "datamaskingView", "auditLogView"]
      }
    ]
  }
}

Groups checks:

• spec.description is optional
• spec.externalGroups is a list of LDAP or OIDC groups to sync with this Console Group
  • Members added this way will not appear in spec.members but spec.membersFromExternalGroups instead
• spec.membersFromExternalGroups is a read-only list of members added through spec.externalGroups
• spec.members must be email addresses of members you wish to add to this group
• spec.permissions are valid permissions as defined in Permissions

Side effect in Console & Kafka:

• Console
  • Members of the Group are given the associated permissions in the UI over the resources
  • Members of the LDAP or OIDC groups will be automatically added or removed upon login
• Kafka
  • No side effect

ConsoleUser

API Keys: Admin API Key
Managed with: API CLI Terraform Console UI

Sets a User with permissions in Console

CLI

---
apiVersion: iam/v2
kind: User
metadata:
  name: john.doe@company.org
spec:
  firstName: "John"
  lastName: "Doe"
  permissions:
    - resourceType: TOPIC
      cluster: shadow-it
      patternType: PREFIXED
      name: toto-
      permissions:
        - topicViewConfig
        - topicConsume
        - topicProduce

Terraform

resource "conduktor_group_v2" "john.doe@company.org" {
  name = "john.doe@company.org"
  spec {
    firstname = "John"
    lastname  = "Doe"
    permissions  = [
     {
        resource_type = "TOPIC"
        cluster       = "shadow-it"
        patternType   = "PREFIXED"
        name          = "toto-"
        permissions   = ["topicViewConfig", "datamaskingView", "auditLogView"]
      }
    ]
  }
}

Users checks:

• spec.permissions are valid permissions as defined in Permissions

Side effect in Console & Kafka:

• Console
  • User is given the associated permissions in the UI over the resources
• Kafka
  • No side effect

KafkaCluster

Creates a Kafka Cluster Definition in Console.

API Keys: Admin API Key
Managed with: API CLI Terraform Console UI

CLI

---
apiVersion: console/v2
kind: KafkaCluster
metadata:
  name: my-dev-cluster
spec:
  displayName: "My Dev Cluster"
  icon: "kafka"
  color: "#000000"
  bootstrapServers: "localhost:9092"
  ignoreUntrustedCertificate: false
  properties:
    sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret";
    security.protocol: SASL_SSL
    sasl.mechanism: PLAIN
  schemaRegistry:
    type: "ConfluentLike"
    url: http://localhost:8080
    security:
      type: BasicAuth
      username: some_user
      password: some_password
    ignoreUntrustedCertificate: false
  kafkaFlavor:
    type: "Confluent"
    key: "string"
    secret: "string"
    confluentEnvironmentId: "string"
    confluentClusterId: "string"

Terraform

resource "conduktor_kafka_cluster_v2" "my-dev-cluster" {
  name = "my-dev-cluster"
  spec {
    display_name                 = "My Dev Cluster"
    icon                         = "kafka"
    color                        = "#000000"
    bootstrap_servers            = "localhost:9092"
    ignore_untrusted_certificate = false
    properties = {
      "sasl.jaas.config"  = "org.apache.kafka.common.security.plain.PlainLoginModule required username='admin' password='admin-secret';"
      "security.protocol" = "SASL_SSL"
      "sasl.mechanism"    = "PLAIN"
    }
    schema_registry = {
      type     = "ConfluentLike"
      url      = "http://localhost:8080"
      security = {
        type     = "BasicAuth"
        username = "some_user"
        password = "some_password"
      }
      ignore_untrusted_certificate = false
    }
    kafka_flavor = {
      type   = "Confluent"
      key    = "string"
      secret = "string"
      confluent_environment_id = "string"
      confluent_cluster_id     = "string"
    }
  }
}

info

metadata.name, spec.displayName, spec.icon and spec.color work together to build the visual identity of the KafkaCluster throughout Console.

KafkaCluster checks:

• spec.icon (optional, default kafka) is a valid entry from our Icon Sets
• spec.color (optional, default #000000) is a HEX color for spec.icon
• spec.ignoreUntrustedCertificate (optional, default false) must be one of [true, false]
• spec.schemaRegistry.type (optional) must be one of [ConfluentLike, Glue]
  • See Schema Registry Properties for the detailed list of options
• spec.kafkaFlavor.type (optional) must be one of [Confluent, Aiven, Gateway]
  • See Kafka Provider Properties for the detailed list of options

Important

Conduktor CLI does not verify that your Kafka configuration (spec.bootstrapServers, spec.properties, ...) is valid.
You need to check that in Console directly.

Schema Registry

This section lets you associate a Schema Registry to your KafkaCluster

Confluent or Confluent-like Registry

CLI

spec:
  schemaRegistry:
    type: "ConfluentLike"
    url: http://localhost:8080
    ignoreUntrustedCertificate: false
    security:
      type: BasicAuth
      username: some_user
      password: some_password

Terraform

spec {
  schema_registry = {
    type = "ConfluentLike"
    url = "http://localhost:8080
    ignore_untrusted_certificate = false
    security = {
      type     = "BasicAuth"
      username = "some_user"
      password = "some_password"
    }
  }
}

Confluent Schema Registry checks:

• spec.schemaRegistry.urls must be a single URL of a Kafka Connect cluster
  • Multiple URLs are not supported for now. Coming soon
• spec.schemaRegistry.ignoreUntrustedCertificate (optional, default false) must be one of [true, false]
• spec.schemaRegistry.properties (optional) is Java Properties formatted key values to further configure the SchemaRegistry
• spec.security.type (optional) must be one of [BasicAuth, BearerToken, SSLAuth]
  • See HTTP Security Properties for the detailed list of options

AWS Glue Registry

CLI

spec:
  schemaRegistry:
    type: "Glue"
    region: eu-west-1
    registryName: default
    security:
      type: Credentials
      accessKeyId: accessKey
      secretKey: secretKey

Terraform

spec {
  schema_registry = {
    type          = "Glue"
    region        = "eu-west-1"
    registry_name = "default"
    security = {
      type          = "Credentials"
      access_key_id = "accessKey"
      secret_key    = "secretKey"
    }
  }
}

AWS Glue Registry checks:

• spec.schemaRegistry.region must be a valid AWS region
• spec.schemaRegistry.registryName must be a valid AWS Glue Registry in this region
• spec.schemaRegistry.security.type must be one of [Credentials, FromContext, FromRole]

Credentials
Use AWS API Key/Secret to connect to the Glue Registry

CLI

    security:
      type: Credentials
      accessKeyId: AKIAIOSFODNN7EXAMPLE
      secretKey: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Terraform

    security = {
      type          = "Credentials"
      access_key_id = "AKIAIOSFODNN7EXAMPLE"
      secret_key    = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
    }

FromContext

CLI

    security:
      type: FromContext
      profile: default

Terraform

    security = {
      type    = "FromContext"
      profile = "default"
    }

FromRole

CLI

    security:
      type: FromRole
      role: arn:aws:iam::123456789012:role/example-role

Terraform

    security = {
      type    = "FromRole"
      profile = "arn:aws:iam::123456789012:role/example-role"
    }

Kafka Provider

This section lets you configure the Kafka Provider for this KafkaCluster.

Confluent Cloud
Provide your Confluent Cloud details to get additional features in Console:

• Confluent Cloud Service Accounts support
• Confluent Cloud API Keys support

CLI

spec:
  kafkaFlavor:
    type: "Confluent"
    key: "yourApiKey123456"
    secret: "yourApiSecret123456"
    confluentEnvironmentId: "env-12345"
    confluentClusterId: "lkc-67890"

Terraform

spec {
  kafka_flavor = {
    type                     = "Confluent"
    key                      = "yourApiKey123456"
    secret                   = "yourApiSecret123456"
    confluent_environment_id = "env-12345"
    confluent_cluster_id     = "lkc-67890"
  }
}

Aiven
Provide your Aiven Cloud details to get additional features in Console:

• Aiven Service Accounts support
• Aiven ACLs support

CLI

spec:
  kafkaFlavor:
    type: "Aiven"
    apiToken: "a1b2c3d4e5f6g7h8i9j0"
    project: "my-kafka-project"
    serviceName: "my-kafka-service"

Terraform

spec {
  kafka_flavor = {
    type         = "Aiven"
    api_token    = "a1b2c3d4e5f6g7h8i9j0"
    project      = "my-kafka-project"
    service_name = "my-kafka-service"
  }
}

Gateway
Provide your Gateway details to get additional features in Console:

• Interceptors support

CLI

spec:
  kafkaFlavor:
    type: "Gateway"
    url: "http://gateway:8888"
    user: "admin"
    password: "admin"
    virtualCluster: passthrough

Terraform

spec {
  kafka_flavor = {
    type            = "Gateway"
    url             = "http://gateway:8888"
    user            = "admin"
    password        = "admin"
    virtual_cluster = "passthrough"
    ignore_untrusted_certificate = false
  }
}

Icon Sets

cloud-bolt	cloud-rainbow	cloud	snowflake
poo-storm	poop	bolt	umbrella
tennis-ball	rugby-ball	traffic-cone	faucet
basket-shopping	box	scale-balanced	sunglasses
sword	axe-battle	vial	feather-pointed
bomb	flag	heart	key
fire-extinguisher	fire-flame-curved	alien	helmet-battle
ghost	robot	dog	elephant
bird	crab	cat-space	planet-ringed
meteor	moon	space-station	rocket-launch
paper-plane	car-side	building-columns	castle
acorn	burger-lettuce	croissant	mug
cactus	clover	camera-cctv	calendar
alarm-clock	compass	gamepad-modern	server
shield-blank	computer-classic	dharmachakra	kafka

KafkaConnectCluster

Creates a Kafka Connect Cluster Definition in Console.

API Keys: Admin API Key
Managed with: API CLI Terraform Console UI

CLI

---
apiVersion: console/v2
kind: KafkaConnectCluster
metadata:
  cluster: my-dev-cluster
  name: connect-1
spec:
  displayName: "Connect 1"
  urls: "http://localhost:8083"
  headers:
    X-PROJECT-HEADER: value
    AnotherHeader: test
  ignoreUntrustedCertificate: false
  security:
    type: "BasicAuth"
    username: "toto"
    password: "my-secret"

Terraform

resource "conduktor_kafka_connect_v2" "connect-1" {
  name    = "connect-1"
  cluster = "my-dev-kafka-cluster"
  spec {
    display_name = "Connect 1"
    urls         = "http://localhost:8083"
    headers = {
      X-PROJECT-HEADER = "value"
      Cache-Control : "no-cache"
    }
    ignore_untrusted_certificate = false
    security = {
      type     = "BasicAuth"
      username = "toto"
      password = "my-secret"
    }
  }
}

KafkaConnectCluster checks:

• metadata.cluster must be a valid KafkaCluster name
• spec.urls must be a single URL of a Kafka Connect cluster
  • Multiple URLs are not supported for now. Coming soon
• spec.ignoreUntrustedCertificate (optional, default false) must be one of [true, false]
• spec.headers (optional) must be key-value pairs of HTTP Headers
• spec.security.type (optional) must be one of [BasicAuth, BearerToken, SSLAuth]
  • See HTTP Security Properties for the detailed list of options

KsqlDBCluster

API Keys: Admin API Key
Managed with: API CLI Console UI

Creates a ksqlDB Cluster Definition in Console.

---
apiVersion: console/v2
kind: KsqlDBCluster
metadata:
  cluster: my-dev-cluster
  name: ksql-1
spec:
  displayName: "KSQL 1"
  url: "http://localhost:8088"
  ignoreUntrustedCertificate: false
  security:
    type: "BasicAuth"
    username: "toto"
    password: "my-secret"

KafkaConnectCluster checks:

• metadata.cluster must be a valid KafkaCluster name
• spec.url must be a single URL of a KsqlDB cluster
• spec.ignoreUntrustedCertificate (optional, default false) must be one of [true, false]
• spec.headers (optional) must be key-value pairs of HTTP Headers
• spec.security.type (optional) must be one of [BasicAuth, BearerToken, SSLAuth]
  • See HTTP Security Properties for the detailed list of options

Alert

API Keys: Admin API Key
Managed with: API CLI Console UI

Creates an Alert in Console.

---
apiVersion: console/v2
kind: Alert
metadata:
  cluster: my-dev-cluster
  name: my-alert
spec:
  type: TopicAlert
  topicName: wikipedia-parsed-DLQ
  metric: MessageCount
  operator: GreaterThan
  threshold: 0
  disable: false

Alert checks:

• metadata.cluster must be a valid KafkaCluster name
• spec.type must be one of [BrokerAlert,TopicAlert,KafkaConnectAlert]
  • Check the section below for the additional mandatory fields needed for each spec.type
• spec.metric is depending on the spec.type
  • Check section below
• spec.operator must be one of [GreaterThan, GreaterThanOrEqual, LessThan, LessThanOrEqual, NotEqual]
• spec.threshold must be a number
• spec.disable (optional, default false) must be one of [true, false]

When spec.type is BrokerAlert

• spec.metric must be one of [MessageIn, MessageOut, MessageSize, OfflinePartitionCount, PartitionCount, UnderMinIsrPartitionCount, UnderReplicatedPartitionCount]

When spec.type is TopicAlert

• spec.metric must be one of [MessageCount, MessageIn, MessageOut, MessageSize]
• spec.topicName must be a Kafka Topic

When spec.type is KafkaConnectAlert

• spec.metric must be FailedTaskCount
• spec.connectName must be a valid KafkaConnect Cluster associated to this meta.cluster Kafka Cluster
• spec.connectorName must be a Kafka Connect Connector

DataMaskingPolicy

Not implemented yet

This concept will be available in a future version

HTTP Security Properties

HTTP Security Properties are used in KafkaCluster (Schema Registry), KafkaConnect, KsqlDBCluster

Basic Authentication

  security:
    type: "BasicAuth"
    username: "toto"
    password: "my-secret"

Bearer Token

  security:
    type: "BearerToken"
    token: "toto"

mTLS / Client Certificate

  security:
    type: "SSLAuth"
    key: |
      -----BEGIN PRIVATE KEY-----
      MIIOXzCCDUegAwIBAgIRAPRytMVYJNUgCbhnA+eYumgwDQYJKoZIhvcNAQELBQAw
      ...
      IFyCs+xkcgvHFtBjjel4pnIET0agtbGJbGDEQBNxX+i4MDA=
      -----END PRIVATE KEY-----
    certificateChain: |
      -----BEGIN CERTIFICATE-----
      MIIOXzCCDUegAwIBAgIRAPRytMVYJNUgCbhnA+eYumgwDQYJKoZIhvcNAQELBQAw
      RjELMAkGA1UEBhMCVVMxIjAgBgNVBAoTGUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBM
      ...
      8/s+YDKveNdoeQoAmGQpUmxhvJ9rbNYj+4jiaujkfxT/6WtFN8N95r+k3W/1K4hs
      IFyCs+xkcgvHFtBjjel4pnIET0agtbGJbGDEQBNxX+i4MDA=
      -----END CERTIFICATE-----

Permissions

Permissions are used in Groups and Users and lets you configure all the access to any Kafka resource or Console feature.

A permission applies to a certain resourceType, which affect the necessary fields as detailed below.

• Topic Permissions
• Subject Permissions
• ConsumerGroup Permissions
• Cluster Permissions
• KafkaConnect Permissions
• KsqlDB Permissions
• Platform Permissions

Topic Permissions

# Grants Consume, Produce and View Config to all topics toto-* on shadow-it cluster
- resourceType: TOPIC
  cluster: shadow-it
  patternType: PREFIXED
  name: toto-
  permissions:
    - topicViewConfig
    - topicConsume
    - topicProduce

• resourceType: TOPIC
• cluster is a valid Kafka cluster
• patternType is either PREFIXED or LITERAL
• name is the name of the topic or topic prefix to apply the permissions to
• permissions is a list of valid topic permissions (See Table)

Available Topic Permissions	Description
topicConsume	Permission to consume messages from the topic.
topicProduce	Permission to produce (write) messages to the topic.
topicViewConfig	Permission to view the topic configuration.
topicEditConfig	Permission to edit the topic configuration.
topicCreate	Permission to create a new topic.
topicDelete	Permission to delete the topic.
topicAddPartition	Permission to add partitions to the topic.
topicEmpty	Permission to empty (delete all messages from) the topic.

Subject Permissions

# Grants View and Edit Compatibility to all subjects starting with sub-* on shadow-it cluster
- resourceType: SUBJECT
  cluster: shadow-it
  patternType: PREFIXED
  name: sub-
  permissions:
    - subjectView
    - subjectEditCompatibility

• resourceType: SUBJECT
• cluster is a valid Kafka cluster
• patternType is either PREFIXED or LITERAL
• name is the name of the subject or subject prefix to apply the permissions to
• permissions is a list of valid subject permissions (See Table)

Available Subject Permissions	Description
subjectCreateUpdate	Permission to create or update the subject.
subjectDelete	Permission to delete the subject.
subjectEditCompatibility	Permission to edit the subject compatibility settings.
subjectView	Permission to view the subject details.

ConsumerGroup Permissions

# Grants View and Reset on all consumer groups starting with group-* on shadow-it cluster
- resourceType: CONSUMER_GROUP
  cluster: shadow-it
  patternType: PREFIXED
  name: group-
  permissions:
    - consumerGroupView
    - consumerGroupReset

• resourceType: CONSUMER_GROUP
• cluster is a valid Kafka cluster
• patternType is either PREFIXED or LITERAL
• name is the name of the consumer group or consumer group prefix to apply the permissions to
• permissions is a list of valid consumer group permissions (See Table)

Available ConsumerGroup Permissions	Description
consumerGroupCreate	Permission to create a new consumer group.
consumerGroupReset	Permission to reset the consumer group.
consumerGroupDelete	Permission to delete the consumer group.
consumerGroupView	Permission to view the consumer group details.

Cluster Permissions

# Grants View Broker, Edit Schema Registry Compatibility, Edit Broker, View ACL, and Manage ACL on shadow-it cluster
- resourceType: CLUSTER
  name: shadow-it
  permissions:
    - clusterViewBroker
    - clusterEditSRCompatibility
    - clusterEditBroker
    - clusterViewACL
    - clusterManageACL

• resourceType: CLUSTER
• name is the name of the cluster to apply the permissions to
  • Use * for all clusters
• permissions is a list of valid cluster permissions (See Table)

Available Cluster Permissions	Description
clusterViewBroker	Permission to view broker details.
clusterEditSRCompatibility	Permission to edit Schema Registry compatibility settings.
clusterEditBroker	Permission to edit broker configuration.
clusterViewACL	Permission to view Access Control Lists (ACLs) for the cluster.
clusterManageACL	Permission to manage Access Control Lists (ACLs) for the cluster.

KafkaConnect Permissions

# Grants Create and Delete on all connectors starting with connector-* on shadow-it cluster and kafka-connect-cluster
- resourceType: KAFKA_CONNECT
  cluster: shadow-it
  kafkaConnect: kafka-connect-cluster
  patternType: PREFIXED
  name: connector-
  permissions:
    - kafkaConnectorCreate
    - kafkaConnectorDelete

• resourceType: KAFKA_CONNECT
• cluster is a valid Kafka cluster
• kafkaConnect is a valid Kafka Connect cluster
• patternType is either PREFIXED or LITERAL
• name is the name of the connector or connector prefix to apply the permissions to
• permissions is a list of valid Kafka Connect permissions (See Table)

Available KafkaConnect Permissions	Description
kafkaConnectorViewConfig	Permission to view the Kafka Connect configuration.
kafkaConnectorStatus	Permission to view the status of Kafka Connect connectors.
kafkaConnectorEditConfig	Permission to edit the Kafka Connect configuration.
kafkaConnectorDelete	Permission to delete connectors.
kafkaConnectorCreate	Permission to create new connectors.
kafkaConnectPauseResume	Permission to pause and resume connectors.
kafkaConnectRestart	Permission to restart connectors.

KsqlDB Permissions

# Grants all permissions on KsqlDB cluster ksql-cluster
- resourceType: KSQLDB
  cluster: shadow-it
  ksqlDB: ksql-cluster
  permissions:
    - ksqldbAccess

• resourceType: KSQLDB
• cluster is a valid Kafka cluster
• ksqlDB is a valid Kafka Connect cluster
• permissions is a list of valid KsqlDB permissions (See Table)

Available KafkaConnect Permissions	Description
ksqldbAccess	Grants all permissions on the KsqlDB Cluster.

Platform Permissions

# Grants Platform permissions
- resourceType: PLATFORM
  permissions:
    - userView
    - datamaskingView

• resourceType: PLATFORM
• permissions is a list of valid Platform permissions (See Table)

Available Platform Permissions	Description
clusterConnectionsManage	Permission to add / edit / remove Kafka clusters on Console
certificateManage	Permission to add / edit / remove TLS Certificates on Console
userManage	Permission to manage Console users, groups & permissions
userView	Permission to view Console users, groups & permissions
datamaskingManage	Permission to manage Data policies (masking rules)
datamaskingView	Permission to view Data policies
notificationChannelManage	Permission to manage Integration channels
notificationChannelView	Permission to view Integration channels
auditLogView	Permission to browse audit log