> ## Documentation Index > Fetch the complete documentation index at: https://docs.conduktor.io/llms.txt > Use this file to discover all available pages before exploring further. # Console resources > YAML resource definitions for Conduktor Console: users, groups, permissions, RBAC bindings, cluster configurations, and certificate management schemas. ## ConsoleGroup * **API key(s):** AdminToken * **Managed with:** UI, CLI, API, TF * **Labels support:** Missing Creates a group with members and permissions in Console. ```yaml theme={null} --- apiVersion: iam/v2 kind: Group metadata: name: developers-a spec: displayName: "Developers Team A" description: "Members of the Team A - Developers" externalGroups: - "LDAP-GRP-A-DEV" externalGroupRegex: - "LDAP*" members: - member1@company.org - member2@company.org permissions: - resourceType: TOPIC cluster: shadow-it patternType: PREFIXED name: toto- permissions: - topicViewConfig - topicConsume - topicProduce ``` **Groups checks:** * `spec.description` is **optional** * `spec.externalGroups` is a list of LDAP or OIDC groups to sync with this Console Group * Members added this way will not appear in `spec.members` but `spec.membersFromExternalGroups` instead * `spec.externalGroupRegex` is a list of regex patterns that can match to a series of LDAP or OIDC groups to sync with this Console group. Members added this way will not appear in `spec.members` list. * Supports regex patterns for dynamic group matching (e.g., `^TEAM-.*` to match all groups starting with "TEAM-") * `spec.membersFromExternalGroups` is a **read-only** list of members added through `spec.externalGroups` or `spec.externalGroupRegex` * `spec.members` must be email addresses of members you wish to add to this group * `spec.permissions` are valid permissions as defined in [Permissions](#permissions) [View Terraform documentation](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs/resources/console_group_v2) **Side effects in Console and Kafka:** * Console * Members of the Group are given the associated permissions in the UI over the resources * Members of the LDAP or OIDC groups will be automatically added or removed upon login * Kafka * No side effects ## ConsoleUser * **API key(s):** AdminToken * **Managed with:** UI, CLI, API, TF * **Labels support:** Missing Creates a user with Console permissions. ```yaml theme={null} --- apiVersion: iam/v2 kind: User metadata: name: john.doe@company.org spec: firstName: "John" lastName: "Doe" permissions: - resourceType: PLATFORM permissions: - taasView - datamaskingView - resourceType: TOPIC cluster: shadow-it patternType: PREFIXED name: toto- permissions: - topicViewConfig - topicConsume - topicProduce ``` Make sure you set permissions for this user, otherwise it won't have access to Console functionality (such as **Application Catalog** or Kafka resources. **User checks:** * `spec.permissions` are valid permissions as defined in [Permissions](#permissions) [View Terraform documentation](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs/resources/console_user_v2) **Side effects in Console and Kafka:** * Console * User is given the associated permissions in the UI over the resources * Kafka * No side effects ## KafkaCluster Creates a Kafka cluster definition in Console. * **API key(s):** AdminToken * **Managed with:** UI, CLI, API, TF * **Labels support:** Partial ```yaml theme={null} --- apiVersion: console/v2 kind: KafkaCluster metadata: name: my-dev-cluster spec: displayName: "My Dev Cluster" icon: "kafka" color: "#000000" bootstrapServers: "localhost:9092" ignoreUntrustedCertificate: false properties: sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret"; security.protocol: SASL_SSL sasl.mechanism: PLAIN schemaRegistry: type: "ConfluentLike" url: http://localhost:8080 security: type: BasicAuth username: some_user password: some_password ignoreUntrustedCertificate: false kafkaFlavor: type: "Confluent" key: "string" secret: "string" confluentEnvironmentId: "string" confluentClusterId: "string" policiesRef: - "generic-dev-topic" - "connector-naming-rule" ``` `metadata.name`, `spec.displayName`, `spec.icon` and `spec.color` are combined to create the visual identity of the KafkaCluster within Console. **KafkaCluster checks:** * `spec.icon` (optional, default `kafka`) is a valid entry from our [Icon Sets](#icon-sets) * `spec.color` (optional, default `#000000`) is a HEX color for `spec.icon` * `spec.ignoreUntrustedCertificate` (optional, default `false`) must be one of \[`true`, `false`] * `spec.schemaRegistry.type` (optional) must be one of \[`ConfluentLike`, `Glue`] * See [Schema Registry Properties](#schema-registry) for the detailed list of options * `spec.kafkaFlavor.type` (optional) must be one of \[`Confluent`, `ConfluentPlatform`, `Aiven`, `Gateway`] * See [Kafka Provider Properties](#kafka-provider) for the detailed list of options * `spec.policiesRef` (optional), if set, has to be a valid list of [ResourcePolicy](/guide/reference/self-service-reference#resourcepolicy). Conduktor CLI does not verify that your Kafka configuration (`spec.bootstrapServers`, `spec.properties`, etc.) is valid. You need to check that in Console directly. [View Terraform documentation](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs/resources/console_kafka_cluster_v2) ### Schema registry This section lets you associate a schema registry to your KafkaCluster. #### Confluent or Confluent-like Registry ```yaml theme={null} spec: schemaRegistry: type: "ConfluentLike" url: http://localhost:8080 ignoreUntrustedCertificate: false security: type: BasicAuth username: some_user password: some_password ``` **Confluent schema registry checks:** * `spec.schemaRegistry.urls` must be a single URL of a Kafka Connect cluster * **Multiple URLs are not supported for now. Coming soon** * `spec.schemaRegistry.ignoreUntrustedCertificate` (optional, default `false`) must be one of \[`true`, `false`] * `spec.schemaRegistry.properties` (optional) is Java Properties formatted key values to further configure the SchemaRegistry * `spec.security.type` (optional) must be one of \[`BasicAuth`, `BearerToken`, `SSLAuth`] * See [HTTP Security Properties](#http-security-properties) for the detailed list of options [View Terraform documentation](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs/resources/console_kafka_cluster_v2#nested-schema-for-specschema_registryconfluent_like) #### AWS Glue registry ```yaml theme={null} spec: schemaRegistry: type: "Glue" region: eu-west-1 registryName: default security: type: Credentials accessKeyId: accessKey secretKey: secretKey ``` **AWS Glue registry checks:** * `spec.schemaRegistry.region` must be a valid AWS region * `spec.schemaRegistry.registryName` must be a valid AWS Glue Registry in this region * `spec.schemaRegistry.security.type` must be one of \[`Credentials`, `FromContext`, `FromRole`] [View Terraform documentation](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs/resources/console_kafka_cluster_v2#nested-schema-for-specschema_registryglue) **Credentials**\ Use AWS API Key/Secret to connect to the Glue registry. ```yaml theme={null} security: type: Credentials accessKeyId: AKIAIOSFODNN7EXAMPLE secretKey: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY ``` [View Terraform documentation](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs/resources/console_kafka_cluster_v2#nested-schema-for-specschema_registrygluesecuritycredentials) **FromContext** ```yaml theme={null} security: type: FromContext profile: default ``` [View Terraform documentation](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs/resources/console_kafka_cluster_v2#nested-schema-for-specschema_registrygluesecurityfrom_context) **FromRole** ```yaml theme={null} security: type: FromRole role: arn:aws:iam::123456789012:role/example-role ``` [View Terraform documentation](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs/resources/console_kafka_cluster_v2#nested-schema-for-specschema_registrygluesecurityfrom_role) ### Kafka provider This section lets you configure the Kafka provider for this KafkaCluster. **Confluent Cloud** Provide your Confluent Cloud details to get additional features in Console: * Confluent Cloud service account support * Confluent Cloud API key support ```yaml theme={null} spec: kafkaFlavor: type: "Confluent" key: "yourApiKey123456" secret: "yourApiSecret123456" confluentEnvironmentId: "env-12345" confluentClusterId: "lkc-67890" ``` [View Terraform documentation](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs/resources/console_kafka_cluster_v2#confluent-kafka-cluster-with-schema-registry) **Confluent Platform** Provide your Confluent Platform details to get additional features in Console: * Confluent Platform service account support * Confluent Platform RBAC role bindings via the Metadata Service (MDS) ```yaml theme={null} spec: kafkaFlavor: type: "ConfluentPlatform" kafkaClusterId: "abcDEF1234ghIJKL567mno" mdsUrl: "https://mds.example.com:8090" authentication: username: "mds-admin" password: "mds-admin-password" schemaRegistryClusterId: "schema-registry" enableRbacRoleBindings: true ignoreUntrustedCertificate: false ``` **Aiven** Provide your Aiven Cloud details to get additional features in Console: * Aiven service accounts support * Aiven ACLs support ```yaml theme={null} spec: kafkaFlavor: type: "Aiven" apiToken: "a1b2c3d4e5f6g7h8i9j0" project: "my-kafka-project" serviceName: "my-kafka-service" ``` [View Terraform documentation](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs/resources/console_kafka_cluster_v2#aiven-kafka-cluster-with-schema-registry) **Gateway** Provide your Gateway details to get additional features in Console: * Interceptor support ```yaml theme={null} spec: kafkaFlavor: type: "Gateway" url: "http://gateway:8888" user: "admin" password: "admin" virtualCluster: passthrough ``` [View Terraform documentation](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs/resources/console_kafka_cluster_v2#conduktor-gateway-kafka-cluster-with-schema-registry) ### Icon sets | | | | | | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | | `cloudBolt` | `cloudRainbow` | `cloud` | `snowflake` | | | | | | | | | | | | `pooStorm` | `poop` | `bolt` | `umbrella` | | | | | | | | | | | | `tennisBall` | `rugbyBall` | `trafficCone` | `faucet` | | | | | | | | | | | | `basketShopping` | `box` | `scaleBalanced` | `sunglasses` | | | | | | | | | | | | `sword` | `axeBattle` | `vial` | `featherPointed` | | | | | | | | | | | | `bomb` | `flag` | `heart` | `key` | | | | | | | | | | | | `fireExtinguisher` | `fireFlameCurved` | `alien` | `helmetBattle` | | | | | | | | | | | | `ghost` | `robot` | `dog` | `elephant` | | | | | | | | | | | | `bird` | `crab` | `catSpace` | `planetRinged` | | | | | | | | | | | | `meteor` | `moon` | `spaceStation` | `rocketLaunch` | | | | | | | | | | | | `paperPlane` | `carSide` | `buildingColumns` | `castle` | | | | | | | | | | | | `acorn` | `burgerLettuce` | `croissant` | `mug` | | | | | | | | | | | | `cactus` | `clover` | `cameraCctv` | `calendar` | | | | | | | | | | | | `alarmClock` | `compass` | `gamepadModern` | `server` | | | | | | | | | | | | `shieldBlank` | `computerClassic` | `dharmachakra` | `kafka` | ## KafkaConnectCluster Creates a Kafka Connect cluster definition in Console. * **API key(s):** AdminToken * **Managed with:** API, CLI, UI, TF * **Labels support:** Partial ```yaml theme={null} --- apiVersion: console/v2 kind: KafkaConnectCluster metadata: cluster: my-dev-cluster name: connect-1 spec: displayName: "Connect 1" urls: "http://localhost:8083" headers: X-PROJECT-HEADER: value AnotherHeader: test ignoreUntrustedCertificate: false security: type: "BasicAuth" username: "toto" password: "my-secret" policiesRef: - "limit-connector-tasks" - "approved-connector-classes" ``` **KafkaConnectCluster checks:** * `metadata.cluster` has to be a valid KafkaCluster name. * `spec.urls` has to be a single URL of a Kafka Connect cluster. Multiple URLs are not currently supported. * `spec.ignoreUntrustedCertificate` (optional, default `false`). Has to be `true` or `false`. * `spec.headers` (optional) has to be key-value pairs of HTTP headers. * `spec.security.type` (optional) has to be `BasicAuth`, `BearerToken` or `SSLAuth`. [Find out more](#http-security-properties). * `spec.policiesRef` (optional), if set, has to be a valid list of [ResourcePolicy](/guide/reference/self-service-reference#resourcepolicy). [View Terraform documentation](https://registry.terraform.io/providers/conduktor/conduktor/latest/docs/resources/console_kafka_connect_v2) ## KsqlDBCluster * **API key(s):** AdminToken * **Managed with:** UI, CLI, API * **Labels support:** Missing Creates a ksqlDB cluster definition in Console. ```yaml theme={null} --- apiVersion: console/v2 kind: KsqlDBCluster metadata: cluster: my-dev-cluster name: ksql-1 spec: displayName: "KSQL 1" url: "http://localhost:8088" ignoreUntrustedCertificate: false security: type: "BasicAuth" username: "toto" password: "my-secret" ``` **KafkaConnectCluster checks:** * `metadata.cluster` has to be a valid KafkaCluster name. * `spec.url` has to be a single URL of a KsqlDB cluster. * `spec.ignoreUntrustedCertificate` (optional), default is `false`. Has to be `true` or `false`. * `spec.headers` (optional) has to be key-value pairs of HTTP headers. * `spec.security.type` (optional) has to be `BasicAuth`, `BearerToken` or `SSLAuth`. [Find out more](#http-security-properties). ## Integrations * **API key(s):** AdminToken * **Managed with:** UI, CLI, API * **Labels support:** Missing Configures alert integrations in Console. Integrations enable external notification destinations for [Alerts](#alerts). ```yaml theme={null} --- apiVersion: v3 kind: Integration metadata: name: Slack spec: config: type: Slack token: "xoxb-your-slack-bot-token" ``` ```yaml theme={null} --- apiVersion: v3 kind: Integration metadata: name: Email spec: config: type: Email smtpServerUrl: "smtp.company.com:587" enableTls: true ignoreUntrustedCertificate: false senderEmail: "conduktor-alerts@company.com" username: "smtp-user" password: "smtp-password" tlsMode: StartTls ``` ```yaml theme={null} --- apiVersion: v3 kind: Integration metadata: name: Teams spec: {} ``` ```yaml theme={null} --- apiVersion: v3 kind: Integration metadata: name: Webhook spec: {} ``` **Integration checks:** * `metadata.name` has to be one of `Slack`, `Email`, `Teams` or `Webhook`. * `spec.config` is required for `Slack` and `Email` integrations. For `Teams` and `Webhook`, the spec can be empty as they are configured per-alert. ### Slack config | Field | Type | Required | Description | | ------- | ------ | -------- | --------------------- | | `type` | string | Yes | Has to be `Slack` | | `token` | string | Yes | Slack Bot OAuth token | ### Email config | Field | Type | Required | Description | | ---------------------------- | ------- | -------- | ----------------------------------------------------------------- | | `type` | string | Yes | Has to be `Email` | | `smtpServerUrl` | string | Yes | SMTP server address with port (e.g., `smtp.company.com:587`) | | `enableTls` | boolean | Yes | Enable TLS encryption for SMTP connection | | `ignoreUntrustedCertificate` | boolean | Yes | Skip SSL certificate validation (not recommended for production) | | `senderEmail` | string | Yes | Email address that appears as the sender | | `username` | string | Yes | SMTP authentication username | | `password` | string | Yes | SMTP authentication password | | `tlsMode` | string | No | TLS mode: `StartTls` (upgrade connection) or `Tls` (implicit TLS) | For more details on configuring email integration through the UI, see [Email integration](/guide/monitor-brokers-apps/alerts#email-integration). ## Alerts * **API key(s):** AdminToken, AppToken * **Managed with:** UI, CLI, API * **Labels support:** Missing Creates an alert in Console. ```yaml theme={null} --- apiVersion: console/v3 kind: Alert metadata: name: messages-in-dead-letter-queue group: support-team # will be the owner of the alert, can be either a user, a group or an appInstance # user: user@company.org # appInstance: my-app-instance spec: cluster: my-dev-cluster type: TopicAlert topicName: wikipedia-parsed-DLQ metric: MessageCount operator: GreaterThan threshold: 0 description: "Alert for monitoring messages in dead letter queue" displayName: "DLQ message count alert" destination: type: Slack channel: "alerts-p1" ``` **Alert checks:** * `metadata.user`|`metadata.group`|`metadata.appInstance` has to be a valid user, group or appInstance. * `metadata.destination.type` can be either `Slack`, `Teams` or `Webhook`. When set to: * `Slack`: has to be a valid Slack channel ID * `Teams`: has to be a valid Teams webhook URL * `Webhook`: * `spec.destination.url` has to be a valid URL * `spec.destination.method` has to be `GET`, `POST`, `PUT` or `DELETE` * `spec.destination.headers` (optional) has to be key-value pairs of HTTP headers * `spec.destination.authentication.type` (optional) has to be `BasicAuth` (define `spec.destination.authentication.username` and `spec.destination.authentication.password`) or `BearerToken` (define `spec.destination.authentication.token`). * `spec.cluster` has to be a valid KafkaCluster name. * `spec.type` has to be `BrokerAlert`,`TopicAlert`, `KafkaConnectAlert` or `ConsumerGroupAlert`. When set to: * `BrokerAlert`: `spec.metric` has to be `MessageIn`, `MessageOut`, `MessageSize`, `OfflinePartitionCount`, `PartitionCount`, `UnderMinIsrPartitionCount` or `UnderReplicatedPartitionCount`. * `TopicAlert`: `spec.metric` has to be `MessageCount`, `MessageIn`, `MessageOut` or `MessageSize` and the `spec.topicName` has to be a Kafka topic that the owner can access. * `KafkaConnectAlert`:`spec.metric` has to be `FailedTaskCount`;`spec.connectName` has to be a valid KafkaConnect cluster associated to this `spec.cluster` Kafka cluster and `spec.connectorName` has to be a Kafka Connect Connector that the owner can access. * `ConsumerGroupAlert`:`spec.metric` has to be `OffsetLag` or `TimeLag` and `spec.consumerGroupName` has to be a Kafka Consumer group that the owner can access. * `spec.metric` depends on the `spec.type`. * `spec.operator` has to be `GreaterThan`, `GreaterThanOrEqual`, `LessThan`, `LessThanOrEqual` or `NotEqual`. * `spec.threshold` has to be a number. * `spec.description` (optional) provides a text description of the alert. * `spec.displayName` (optional) provides a display name for the alert. * `spec.disable` (optional), default is `false`. Has to be `true` or `false`. **Alert resolution:** Alerts resolve (stop firing) when the metric value no longer meets the threshold condition defined by the comparison operator (GreaterThan, LessThan, etc.). ## Partner Zones * **API key(s):** AdminToken * **Managed with:** UI, CLI, API * **Labels support:** Partial Create or update a [Partner Zone](/guide/conduktor-concepts/partner-zones). ```yaml theme={null} --- apiVersion: console/v2 kind: PartnerZone metadata: name: external-partner-zone spec: displayName: External Partner Zone description: An external partner to exchange data with. url: https://partner1.com partner: name: John Doe role: Data analyst email: johndoe@partner.io phone: 07827 837 177 cluster: cdk-gateway underlyingCluster: cluster1 authenticationMode: serviceAccount: partner-external-partner type: PLAIN vclusterName: custom-vcluster-name topics: - name: topic-a backingTopic: kafka-topic-a permission: WRITE - name: topic-b backingTopic: kafka-topic-a permission: READ trafficControlPolicies: maxProduceRate: 1e+06 maxConsumeRate: 1e+06 limitCommitOffset: 30 headers: addOnProduce: - key: partner-name value: external-analytics-partner overrideIfExists: false - key: client-info value: "Client:{{clientId}}, from IP:{{userIp}}" overrideIfExists: true - key: kafka-api value: "Kafka API Key:{{apiKey}}, version {{apiKeyVersion}}" overrideIfExists: true - key: produce-metadata value: "User:{{user}}, via Gateway:{{gatewayHost}}, at timestampMillis:{{timestampMillis}}" overrideIfExists: true removeOnConsume: - keyRegex: my_team_prefix.* ``` **Partner Zone checks:** * `spec.displayName` is mandatory. * `spec.description`, `spec.url` and `spec.partner` (optional), useful for context information. * `spec.cluster` has to be a valid Console cluster technical ID with the **Provider configured as *Gateway***. * `spec.underlyingCluster` has to be a valid Console cluster technical ID where the cluster has to be defined under the Gateway defined by the `spec.gatewayClusterId`. * When not specified, the value is inferred to be equal to the `spec.gatewayClusterId` field, hence selecting the main cluster behind Gateway by default. * `spec.authenticationMode.type` must be one of \[`PLAIN`, `OAUTHBEARER`, `MTLS`]. See [authentication examples](/guide/use-cases/third-party-data#authentication-modes) for detailed configuration. * `spec.authenticationMode.serviceAccount` requirements depend on the authentication type: * **PLAIN**: Any unique identifier for your partner (e.g., `partner-external-partner`). This will be created as a local Gateway service account automatically if it doesn't exist. * **OAUTHBEARER**: Must match the "sub" claim in the partner's OAuth/OIDC token (e.g., `oauth-partner-service-account`). The partner needs to authenticate using their OAuth provider. * **MTLS**: Must match the client's Distinguished Name (DN) from their certificate, unless you've modified `GATEWAY_SSL_PRINCIPAL_MAPPING_RULES` (e.g., `CN=partner-client,OU=Engineering,O=PartnerCorp,C=US`). * `spec.vclusterName` (optional), custom name for the Virtual Cluster. If not provided, it will be auto-generated. * `topics[].name` is the name of the topic as it should appear to your external partner. This can be different from `backingTopic`. * `topics[].backingTopic` is the internal name of the topic that you want to share. * `topics[].permission` has to be set to either `READ` or `WRITE` (which includes `READ`). * `trafficControlPolicies.maxProduceRate` (optional), sets the maximum rate (in bytes/s) at which the partner can produce messages to the topics per Gateway node. * `trafficControlPolicies.maxConsumeRate` (optional), sets the maximum rate (in bytes/s) at which the partner can consume messages from the topics per Gateway node. * `trafficControlPolicies.limitCommitOffset` (optional), sets the maximum number of commit requests (in requests/minute) that the partner can make per Gateway node. * `headers.addOnProduce` (optional), list of headers to inject when producing messages. The `value` field supports special variables: `{{user}}`, `{{userIp}}`, `{{clientId}}`, `{{apiKey}}`, `{{apiKeyVersion}}`, `{{gatewayHost}}`, `{{timestampMillis}}`. * `headers.removeOnConsume` (optional), list of header key patterns (regex) to remove when consuming messages. **Side effects in Console and Kafka:** Once created or updated, the following fields will be made available: * `metadata.updatedAt` (by consecutive *get* from the CLI/API). * `metadata.status` (by consecutive *get* from the CLI/API.) Possible values are `PENDING`, `READY` or `FAILED`. * `metadata.failedReason` will be populated in case of `FAILED` status. * The service account will be created if it doesn't exist and will be granted the permissions as declared in `spec.topics`. * The traffic control policies will be applied to the service account. ## DataQualityRule * **API key(s):** AdminToken * **Managed with:** UI, CLI, API * **Labels support:** Yes Create or update a data quality rule that validates data quality for Kafka topics. ```yaml theme={null} --- apiVersion: v1 kind: DataQualityRule metadata: name: email-validation-rule labels: category: validation team: data-quality spec: type: Cel displayName: Email validation rule description: Validates that email field contains a valid email address celExpression: 'record.email.matches("^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$")' customErrorMessage: Email address is not valid ``` ```yaml theme={null} --- apiVersion: v1 kind: DataQualityRule metadata: name: user-schema-rule labels: category: schema spec: type: JsonSchema displayName: User schema validation description: Validates user records against JSON schema schema: type: object required: - userId - email properties: userId: type: string email: type: string format: email age: type: integer minimum: 0 customErrorMessage: Record does not match user schema ``` **DataQualityRule checks:** * `metadata.name` is mandatory and must be unique. * `metadata.labels` is optional and can be used to organize rules. * `spec.type` has to be one of: `Cel`, `JsonSchema`, `EnforceAvro`, `EnforceSchemaId`. * `spec.displayName` is mandatory and is shown in the Console UI. * `spec.description` is optional but recommended to explain what the rule validates. * `spec.customErrorMessage` is optional and provides a custom error message when validation fails. * For `Cel` type: * `spec.celExpression` is mandatory and contains the CEL (Common Expression Language) expression to validate records. * For `JsonSchema` type: * `spec.schema` is mandatory and contains the JSON schema object to validate records against. * For `EnforceAvro` type: * Validates that records conform to an Avro schema. * For `EnforceSchemaId` type: * Validates that records use a specific schema ID from Schema Registry. **Side effects in Console and Kafka:** Once created or updated, the following fields will be made available: * `metadata.createdAt` (read-only timestamp). * `metadata.updatedAt` (read-only timestamp). * `metadata.createdBy` (read-only user identifier). * `metadata.updatedBy` (read-only user identifier). * `metadata.attachedPolicies` (read-only list of policies using this rule). ## DataQualityPolicy * **API key(s):** AdminToken * **Managed with:** UI, CLI, API * **Labels support:** No Create or update a data quality policy that applies rules to specific Kafka topics. ```yaml theme={null} --- apiVersion: v1 kind: DataQualityPolicy metadata: name: user-data-validation-policy spec: displayName: User data validation policy description: Validates all user data before it is produced to topics rules: - email-validation-rule - user-schema-rule targets: - cluster: main-cluster topic: users patternType: LITERAL - cluster: main-cluster topic: user-events- patternType: PREFIXED actions: block: enabled: true mark: enabled: false ``` **DataQualityPolicy checks:** * `metadata.name` is mandatory and must be unique. * `spec.displayName` is mandatory and is shown in the Console UI. * `spec.description` is optional but recommended to explain the policy's purpose. * `spec.rules` is a list of DataQualityRule names to apply (references to rules created separately). * `spec.targets` is mandatory and defines which topics the policy applies to: * `cluster` is a valid Kafka cluster technical ID. * `topic` is the topic name or prefix. * `patternType` has to be either `LITERAL` (exact match) or `PREFIXED` (prefix match). * `spec.actions` defines what happens when a rule violation occurs: * `block.enabled` (optional, defaults to false) - when true, blocks messages that violate rules from being produced. Only available if the configured cluster is a Conduktor Gateway cluster with the appropriate license. * `mark.enabled` (optional, defaults to false) - when true, marks messages that violate rules with a header but allows production. **Side effects in Console and Kafka:** Once created or updated, the following fields will be made available: * `metadata.nameForMetrics` (read-only identifier used in metrics). * `metadata.group` (read-only group identifier). * `metadata.createdAt` (read-only timestamp). * `metadata.updatedAt` (read-only timestamp). * `metadata.createdBy` (read-only user identifier). * `metadata.updatedBy` (read-only user identifier). * `metadata.status` (read-only status: `failed`, `pending`, or `ready`). * `metadata.statuses` (read-only per-cluster deployment status). * `metadata.metrics` (read-only metrics showing evaluation and violation counts). * `metadata.attachedRules` (read-only list of rules attached to this policy). * Data quality rules are enforced on the specified topics. * Messages violating rules are blocked or marked based on the configured actions. ## HTTP security properties HTTP security properties are used in KafkaCluster ([schema registry](#confluent-or-confluent-like-registry)), [KafkaConnect](#kafkaconnectcluster) and [KsqlDBCluster](#ksqldbcluster). ### Basic authentication ```yaml theme={null} security: type: "BasicAuth" username: "toto" password: "my-secret" ``` ### Bearer token ```yaml theme={null} security: type: "BearerToken" token: "toto" ``` ### mTLS/client certificate ```yaml theme={null} security: type: "SSLAuth" key: | -----BEGIN PRIVATE KEY----- MIIOXzCCDUegAwIBAgIRAPRytMVYJNUgCbhnA+eYumgwDQYJKoZIhvcNAQELBQAw ... IFyCs+xkcgvHFtBjjel4pnIET0agtbGJbGDEQBNxX+i4MDA= -----END PRIVATE KEY----- certificateChain: | -----BEGIN CERTIFICATE----- MIIOXzCCDUegAwIBAgIRAPRytMVYJNUgCbhnA+eYumgwDQYJKoZIhvcNAQELBQAw RjELMAkGA1UEBhMCVVMxIjAgBgNVBAoTGUdvb2dsZSBUcnVzdCBTZXJ2aWNlcyBM ... 8/s+YDKveNdoeQoAmGQpUmxhvJ9rbNYj+4jiaujkfxT/6WtFN8N95r+k3W/1K4hs IFyCs+xkcgvHFtBjjel4pnIET0agtbGJbGDEQBNxX+i4MDA= -----END CERTIFICATE----- ``` ## Permissions Permissions are used in [groups](#consolegroup) and [users](#consoleuser) and lets you configure access to any Kafka resource or Console feature. A permission applies to a certain `resourceType` which affects the required fields. * [Topic permissions](#topic-permissions) * [Subject permissions](#subject-permissions) * [ConsumerGroup permissions](#consumergroup-permissions) * [Cluster permissions](#cluster-permissions) * [KafkaConnect permissions](#kafkaconnect-permissions) * [KsqlDB permissions](#ksqldb-permissions) * [Console permissions](#console-permissions) ### Topic permissions ```yaml theme={null} # Grants consume, produce and view config to all topics toto-* on shadow-it cluster - resourceType: TOPIC cluster: shadow-it patternType: PREFIXED name: toto- permissions: - topicViewConfig - topicConsume - topicProduce ``` * `resourceType`: `TOPIC` * `cluster` is a valid Kafka cluster * `patternType` is either `PREFIXED` or `LITERAL` * `name` is the name of the topic or topic prefix to apply the permissions to * `permissions` is a list of valid topic permissions | Available topic permissions | Description | | --------------------------- | --------------------------------------------------------- | | `topicConsume` | Permission to consume messages from the topic. | | `topicProduce` | Permission to produce (write) messages to the topic. | | `topicViewConfig` | Permission to view the topic configuration. | | `topicEditConfig` | Permission to edit the topic configuration. | | `topicCreate` | Permission to create a new topic. | | `topicDelete` | Permission to delete the topic. | | `topicAddPartition` | Permission to add partitions to the topic. | | `topicEmpty` | Permission to empty (delete all messages from) the topic. | | `topicDataQualityManage` | Permission to manage data quality policies for the topic. | ### Subject permissions ```yaml theme={null} # Grants view and edit compatibility to all subjects starting with sub-* on shadow-it cluster - resourceType: SUBJECT cluster: shadow-it patternType: PREFIXED name: sub- permissions: - subjectView - subjectEditCompatibility ``` * `resourceType`: `SUBJECT` * `cluster` is a valid Kafka cluster * `patternType` is either `PREFIXED` or `LITERAL` * `name` is the name of the subject or subject prefix to apply the permissions to * `permissions` is a list of valid subject permissions | Available subject permissions | Description | | ----------------------------- | ------------------------------------------------------ | | `subjectCreateUpdate` | Permission to create or update the subject. | | `subjectDelete` | Permission to delete the subject. | | `subjectEditCompatibility` | Permission to edit the subject compatibility settings. | | `subjectView` | Permission to view the subject details. | ### ConsumerGroup permissions ```yaml theme={null} # Grants view and reset on all consumer groups starting with group-* on shadow-it cluster - resourceType: CONSUMER_GROUP cluster: shadow-it patternType: PREFIXED name: group- permissions: - consumerGroupView - consumerGroupReset ``` * `resourceType`: `CONSUMER_GROUP` * `cluster` is a valid Kafka cluster * `patternType` is either `PREFIXED` or `LITERAL` * `name` is the name of the consumer group or consumer group prefix to apply the permissions to * `permissions` is a list of valid consumer group permissions | Available ConsumerGroup permissions | Description | | ----------------------------------- | ---------------------------------------------- | | `consumerGroupCreate` | Permission to create a new consumer group. | | `consumerGroupReset` | Permission to reset the consumer group. | | `consumerGroupDelete` | Permission to delete the consumer group. | | `consumerGroupView` | Permission to view the consumer group details. | ### Cluster permissions ```yaml theme={null} # Grants view and edit broker, edit schema registry compatibility, view ACL and manage ACL on shadow-it cluster - resourceType: CLUSTER name: shadow-it permissions: - clusterViewBroker - clusterEditSRCompatibility - clusterEditBroker - clusterViewACL - clusterManageACL ``` * `resourceType`: `CLUSTER` * `name` is the name of the cluster to apply the permissions to * Use `*` for all clusters * `permissions` is a list of valid cluster permissions | Available cluster permissions | Description | | ----------------------------- | ------------------------------------------------------------------------------------------------- | | `clusterViewBroker` | Permission to view broker details. | | `clusterEditSRCompatibility` | Permission to edit schema registry compatibility settings. | | `clusterEditBroker` | Permission to edit broker configuration. | | `clusterViewACL` | Permission to view ACLs for the cluster. Also grants ACL viewing on every topic in the cluster. | | `clusterManageACL` | Permission to manage ACLs for the cluster. Also grants ACL viewing on every topic in the cluster. | ### KafkaConnect permissions ```yaml theme={null} # Grants create and delete on all connectors starting with connector-* on shadow-it cluster and kafka-connect-cluster - resourceType: KAFKA_CONNECT cluster: shadow-it kafkaConnect: kafka-connect-cluster patternType: PREFIXED name: connector- permissions: - kafkaConnectorCreate - kafkaConnectorDelete ``` * `resourceType`: `KAFKA_CONNECT` * `cluster` is a valid Kafka cluster * `kafkaConnect` is a valid Kafka Connect cluster * `patternType` is either `PREFIXED` or `LITERAL` * `name` is the name of the connector or connector prefix to apply the permissions to * `permissions` is a list of valid Kafka Connect permissions | Available KafkaConnect permission | Description | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | `kafkaConnectorViewConfig` | Permission to view the Kafka Connect configuration. | | `kafkaConnectorStatus` | Permission to view the status of Kafka Connect connectors. | | `kafkaConnectorEditConfig` | Permission to edit the Kafka Connect configuration. | | `kafkaConnectorDelete` | Permission to delete connectors. | | `kafkaConnectorCreate` | Permission to create new connectors. | | `kafkaConnectPauseResume` | Permission to pause and resume connectors. | | `kafkaConnectRestart` | Permission to restart connectors. | | `kafkaConnectStop` | Permission to stop connectors. Distinct from pause and resume, as stopping a connector releases its tasks and resources. | | `kafkaConnectOffsetsView` | Permission to view connector offsets. | | `kafkaConnectOffsetsManage` | Permission to edit and reset connector offsets. The connector has to be stopped first. | ### KsqlDB permissions ```yaml theme={null} # Grants all permissions on KsqlDB cluster ksql-cluster - resourceType: KSQLDB cluster: shadow-it ksqlDB: ksql-cluster permissions: - ksqldbAccess ``` * `resourceType`: `KSQLDB` * `cluster` is a valid Kafka cluster * `ksqlDB` is a valid Kafka Connect cluster * `permissions` is a list of valid KsqlDB permissions | Available KafkaConnect permissions | Description | | ---------------------------------- | --------------------------------------------- | | `ksqldbAccess` | Grants all permissions on the KsqlDB cluster. | ### Console permissions ```yaml theme={null} # Grants Console permissions - resourceType: PLATFORM permissions: - userView - datamaskingView ``` * `resourceType`: `PLATFORM` * `permissions` is a list of valid Console permissions | Available Console permissions | Description | | ----------------------------- | -------------------------------------------------------------- | | `clusterConnectionsManage` | Permission to add / edit / remove Kafka clusters on Console. | | `certificateManage` | Permission to add / edit / remove TLS certificates on Console. | | `userManage` | Permission to manage Console users, groups and permissions. | | `userView` | Permission to view Console users, groups and permissions. | | `datamaskingManage` | Permission to manage data policies (masking rules). | | `datamaskingView` | Permission to view data policies. | | `notificationChannelManage` | Permission to manage integration channels. | | `auditLogView` | Permission to browse audit log. | | `taasView` | Permission to view Application Catalog. | | `chargebackManage` | Permission to view Chargeback and manage its settings. | | `sqlManage` | Permission to view indexed topics and create SQL queries. |