Automate with CLI

Conduktor CLI overview

Conduktor CLI (Command Line Interface) helps you to manage your Console and Gateway resources via commands or CI/CD pipelines.

Install

You have three options to install Conduktor CLI:

Native binary - for individual use and testing
Homebrew - for individual use and testing on MacOS
Docker - for integration in CI/CD pipelines

Native binary

Download the latest binary from GitHub on Windows, Linux or MacOS. In the Assets list, download the build that corresponds to your machine (for Apple Silicon use darwin-arm64).

Homebrew

Use the Homebrew package manager to install our CLI:

brew install conduktor/brew/conduktor-cli

Docker

Pull the latest Dockerhub version of the CLI using the following command line:

docker pull conduktor/conduktor-ctl

Configure

To use the Conduktor CLI, you need to set:

the URL of your Conduktor Console and/or Gateway API
the authentication mechanism: either by using an API key, or user credentials

Authenticate to Console using API keys

These are the necessary environment variables for this authentication method::

export CDK_BASE_URL=http://localhost:8080
export CDK_API_KEY=<your-API-key>

There are two types of API keys: admin and Self-service application.

Admin API keys grant the maximum permissions in Console.

Generate an admin key named 'my-admin-key'

conduktor token create admin my-admin-key

Authenticate to Console using user credentials

You can simply set the username and password of the local or LDAP user and use the CLI, or create an associated short-lived API key. This type of API key will have the permissions of the user who created it. This kind of authentication will be valid for the same duration as the configured session timeout.

export CDK_BASE_URL=http://localhost:8080
export CDK_USER=admin@company.io
export CDK_PASSWORD=adminP4ss!

Generate a short-lived API key

conduktor login

Authenticate to Console using external authentication

By default, the Conduktor CLI authenticates with Console using a Console API key, or generates one based on the given credentials. This is the console authentication mode (CDK_AUTH_MODE is console or empty). Alternatively, you can use the external authentication mode to bypass token generation and use credentials from your identity provider or the API proxy instead. For that, set the CDK_AUTH_MODE environment variable to external:

Set up external authentication with a bearer token

export CDK_AUTH_MODE="external"
export CDK_API_KEY=<bearer_token>

Set up external authentication with client credentials

export CDK_AUTH_MODE="external"
export CDK_USER=<client_id>
export CDK_PASSWORD=<client_secret>

Authenticate to Gateway using API credentials

To use the CLI to manage Gateway objects, you need to set the URL of the Gateway API and the API credentials.

export CDK_GATEWAY_BASE_URL=http://localhost:8888
export CDK_GATEWAY_USER=admin
export CDK_GATEWAY_PASSWORD=conduktor

Environment variables

This is the list of the supported environment variables:

Environment variables	Description
Console
`CDK_BASE_URL`	Console base URL, e.g. `http://localhost:8080`
`CDK_API_KEY`	Console API key (admin or application key)
`CDK_USER`	Console local or LDAP user username
`CDK_PASSWORD`	Console local or LDAP user password
`CDK_AUTH_MODE`	Authentication mode (`external`, `console` or empty (default))
Gateway
`CDK_GATEWAY_BASE_URL`	Gateway base URL, e.g. `http://localhost:8888`
`CDK_GATEWAY_USER`	Gateway user login email
`CDK_GATEWAY_PASSWORD`	Gateway user login password
TLS and mTLS
`CDK_CACERT`	Root CA certificate to verify the Conduktor Console certificate (.pem)
`CDK_INSECURE`	Skip the TLS verification of the Conduktor Console. Defaults to `false`.
`CDK_CERT`	Certificate to authenticate the CLI (.pem)
`CDK_KEY`	Key to authenticate the CLI (.pem)

CLI commands

The CLI offers a set of commands to manage Conduktor resources. You can use it to create, update, delete and get resources.

Usage:
  conduktor [flags]
  conduktor [command]

Available commands:
  apply              Upsert a resource on Conduktor
  completion         Generate the autocompletion script for the specified shell
  delete             Delete resource of a given kind and name
  get                Get resource of a given kind
  help               Help about any command
  login              Login user using username password to get a JWT token
  run                run an action
  sql
  template           Get a .yaml example for a given kind
  token              Manage Admin and Application Instance tokens
  version            Display the version of Conduktor

Flags:
  -h, --help         help for  Conduktor
      --permissive   permissive mode, allow undefined environment variables
  -v, --verbose      show more information for debugging

Use "conduktor [command] --help" for more information about a command.

Apply

The apply command allows you to deploy any resource.

Usage:
  conduktor apply [flags]

Flags:
      --dry-run            Test potential changes without the effects being applied
  -f, --file stringArray   Specify the files or folders to apply. For folders, all .yaml or .yml files within the folder will be applied, while files in subfolders will be ignored.
  -h, --help               help for apply
      --parallelism int    Run each apply in parallel, useful when applying a large number of resources. Must be less than 100. (default 1)
  -r, --recursive          Apply all .yaml or .yml files in the specified folder and its subfolders. If not set, only files in the specified folder will be applied.

Global Flags:
      --permissive   Permissive mode, allow undefined environment variables
  -v, --verbose      Show more information for debugging

The --parallelism flag allows you to run multiple applies at the same time, in parallel. This is useful when applying a large number of resources. Be careful not to set it too high, as it may overload the Console/Gateway API.

Examples:

Apply all resources in multiple directories

conduktor apply -f ./directory1 -f ./directory2

Apply a specific file

conduktor apply -f file.yaml

Apply all resources in the current directory and its subfolders

conduktor apply -f . -r

Template

The template command allows you to get an example of a resource definition:

conduktor template topic
---
apiVersion: v2
kind: Topic
metadata:
    name: my-topic
    cluster: my-cluster
    labels:
        conduktor.io/application: application-a
        conduktor.io/application-instance: staging
        user-labels: I am a user label
    catalogVisibility: PUBLIC
    descriptionIsEditable: true
    description: This is a topic description
    sqlStorage:
        retentionTimeInSecond: 42
spec:
    partitions: 1
    replicationFactor: 1
    configs:
        cleanup.policy: delete
        retention.ms: '86400000'

You can also save the output in a file, in order to edit it before applying it:

conduktor template KafkaCluster -o definition.yml
conduktor template Topic >> definition.yml   #Appending to an existing file
vim definition.yml
conduktor apply -f definition.yml

Delete

The delete command allows you to delete a resource. It works the same way as the apply.

Resources are deleted instantly and can’t be recovered. Any data or access associated with the resource is permanently lost.

Examples:

Delete all resources in multiple directories

conduktor delete -f ./directory1 -f ./directory2

Delete a specific file

conduktor delete -f file.yaml

Delete all resources in the current directory and its subfolders

conduktor delete -f . -r

Get

The get command allows you to retrieve resources. You can specify a kind, a name, or list all the resources.

conduktor get user --help
If name not provided it will list all resource

Usage:
  conduktor get User [name] [flags]

Aliases:
  User, user

Flags:
  -h, --help   help for User

Global Flags:
  -o, --output output   Output format. One of: json|yaml|name (default yaml)
      --permissive      Permissive mode, allow undefined environment variables
  -v, --verbose         Show more information for debugging

Examples:

Get all users

conduktor get user

Get a specific Kafka connect cluster

conduktor get KafkaConnectCluster --cluster my-kafka-cluster

Get all the Console resources with a 'name' output format

conduktor get all --console -o name

Token

To run your CI/CD pipelines, you can use the token command to manage your API keys.

Create an admin API keys

conduktor token create admin my_first_admin_token

List Console admin API keys

conduktor token list admin

Create an application instance API key

conduktor token create application-instance -i=my_instance my_token_for_my_instance

List API keys of an application instance

conduktor token list application-instance -i=my_instance

Delete an API key

conduktor token delete 1233ff18-0e95-4638-b3fc-f55e20088b8d

Run

The run command allows you to run a few actions:

Usage:
  conduktor run [flags]
  conduktor run [command]

Available Commands:
  generateServiceAccountToken    Generate a token for a service account on a virtual cluster
  partnerZoneGenerateCredentials generate a token for a partner zone service account
  whoami                         Return information about the credentials configured in the CLI

Flags:
  -h, --help   help for run

Global Flags:
      --permissive   Permissive mode, allow undefined environment variables
  -v, --verbose      Show more information for debugging

Examples:

Generate a service account password for a virtual cluster

conduktor run generateServiceAccountToken --v-cluster my-vcluster --username my-service-account

Generate a Partner Zone service account password

conduktor run partnerZoneGenerateCredentials --partner-zone-name my-partner-zone

Get information about the credentials configured in the CLI

conduktor run whoami

Version

Check the current version of your CLI:

Check CLI version

conduktor version

CI/CD pipeline integration example

Conduktor CLI can be easily integrated to a CI/CD pipeline. This example presents two pipelines:

The first one triggers on each new PR and launches the CLI using the --dry-run flag, generating a report confirming that the resources can be successfully created or modified.
The second one triggers on a push to the main branch, making the changes live.

Consider the following folder structure:

├── resources/
     ├── topics.yml          # Your topics are there
     ├── permissions.yml     # Your permissions to other apps are there

.github/workflows/on-pr.yml


name: Check PR Validity
on:
  pull_request:
    branches: [ "main" ]
  workflow_dispatch:
jobs:
  build:
    runs-on: ubuntu-latest
    container: conduktor/conduktor-ctl
    steps:
      - uses: actions/checkout@v3
      - run: /bin/conduktor apply -f resources/ --dry-run
        env:
          CDK_BASE_URL: https://conduktor.domain.com
          CDK_API_KEY: ${{ secrets.CONDUKTOR_TOKEN }}

.github/workflows/on-push.yml

name: Execute committed changes
on:
  push:
    branches: [ "main" ]
jobs:
  build:
    runs-on: ubuntu-latest
    container: conduktor/conduktor-ctl
    steps:
      - uses: actions/checkout@v3
      - run: /bin/conduktor apply -f resources/
        env:
          CDK_BASE_URL: https://conduktor.domain.com
          CDK_API_KEY: ${{ secrets.CONDUKTOR_TOKEN }}

Environment variables for secrets

The Conduktor CLI automatically substitutes environment variables in YAML manifests using ${ENV_VAR} patterns. This keeps sensitive data like credentials out of configuration files. You can set a default value using POSIX notation ${ENV:-default}. By default, the CLI fails if an environment variable is undefined or empty, ensuring all required variables are properly configured. Use the --permissive flag to replace missing or empty variables with empty strings instead of failing. The example below shows a Conduktor cluster configuration with credentials secured using environment variables:

---
apiVersion: console/v2
kind: KafkaCluster
metadata:
  name: my-cluster
spec:
  displayName: "My Kafka Cluster"
  icon: "kafka"
  color: "#000000"
  bootstrapServers: "localhost:9092"
  ignoreUntrustedCertificate: false
  properties:
    sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username="${ENV_VAR_FOR_USER}" password="${ENV_VAR_FOR_PASSWORD}";
    security.protocol: SASL_SSL
    sasl.mechanism: PLAIN
  schemaRegistry:
    type: "ConfluentLike"
    url: http://localhost:8080
    security:
      type: BasicAuth
      username: ${ENV_VAR_FOR_USER}
      password: ${ENV_VAR_FOR_PASSWORD}
    ignoreUntrustedCertificate: false
  kafkaFlavor:
    type: "Confluent"
    key: "${ENV_VAR_CONFLUENT_KEY}"
    secret: "${ENV_VAR_CONFLUENT_SECRET}"
    confluentEnvironmentId: "${ENV_VAR_CONFLUENT_ENV_ID:-dev}"
    confluentClusterId: "${ENV_VAR_CONFLUENT_CLUSTER_ID:-main}"

Manage Kafka resources

Monitor systems

Govern data

Automate tasks

Conduktor CLI overview

Install

Native binary

Homebrew

Docker

Configure

Authenticate to Console using API keys

Authenticate to Console using user credentials

Authenticate to Console using external authentication

Authenticate to Gateway using API credentials

Environment variables

CLI commands

Apply

Template

Delete

Get

Token

Run

Version

CI/CD pipeline integration example

Environment variables for secrets

Manage Kafka resources

Monitor systems

Govern data

Automate tasks

​Conduktor CLI overview

​Install

​Native binary

​Homebrew

​Docker

​Configure

​Authenticate to Console using API keys

​Authenticate to Console using user credentials

​Authenticate to Console using external authentication

​Authenticate to Gateway using API credentials

​Environment variables

​CLI commands

​Apply

​Template

​Delete

​Get

​Token

​Run

​Version

​CI/CD pipeline integration example

​Environment variables for secrets

Conduktor CLI overview

Install

Native binary

Homebrew

Docker

Configure

Authenticate to Console using API keys

Authenticate to Console using user credentials

Authenticate to Console using external authentication

Authenticate to Gateway using API credentials

Environment variables

CLI commands

Apply

Template

Delete

Get

Token

Run

Version

CI/CD pipeline integration example

Environment variables for secrets