Conduktor provides many ways when it comes to filtering and projecting data. This is an important use-case when using Conduktor:

• you're looking for a needle in your whole topic history

• you'd like to trace the events of your entity "abc" from today only

• you'd like to have in the same view how a specific field changed across time

• you're looking for what happened between 5.05pm and 5.10pm yesterday

• ...

Projecting data

You can choose to display only a subset of fields from a message. This works for JSON and Avro data (because they are converted to JSON). A section "Value projection" will appear where you can set a "jq" expression (documentation).

For instance, here, we only care about .state.version in:

{
  "id" : "84504272-8473-48f6-8ad5-a7e836e58dc7",
  "type" : "ItemAdded",
  "delta" : {
    "itemId" : "1abcdcc1-cc36-4ae4-a6eb-3582df6f8e65"
  },
  "state" : {
    "id" : "570ccfea-6c8a-44a3-83a9-ab54797d104f",
    "version" : 6, // we care only about this! <<<<<<<<<<
    ...
}

We setup this as our filter, and we'll only see the value of version:

Typical use-cases:

• .property.subprop.value : just nested objects

• .items[].object: access objects inside an array

• .items[].object.property: access fields inside objects inside an array

• .items | length: returns the length of an array

• .x.obj | keys : list the keys "obj"

• contains({"state":{"version": 1}}): "true" if the element with this value exists

• ...

• ​

• There are many functions availables: check the jq documentation​

If your jq filter is invalid, or only fails for some records, Conduktor will display the jq error in place of the value:

Filtering data

It's possible to combine multiple filters on multiple dimensions: time, quantity, fields.

Choose where to start from

• time based: a hour, a date

• offset based:

  • an existing consumer group: useful to see why an application is failing there, like a deserialization error

  • latest minux X offsets: just the latest data published, no matter when

  • an offset: this will be the starting offset for all the partitions consumed. You generally use this and select only one partition (see Advanced).

Choose until when to consume

Sometimes, you don't want to continuously consume incoming data and just want to consume a few messages and automatically stop.

• forever is the default: never stop

• number of messages: when we just want to take a peek

• max size: rare use-case, when the size matters

• timestamp: consume data up until a particular date.

  • also useful when combined to a start date

Filter messages by their key/value/headers

We offer many ways to select only the interesting key/value you're looking for.

• contains: a simple string we're looking for (fast)

• equals: the whole key or value must match, often used for keys (where it's generally a simple id)

• regex: for more complex case when you know regexes 🤓. Can be quite slow when the payloads are large.

• json field: it's the same format as in "Projecting data" (above), except the extracted value must match a value you'll expect

Most works with key, value and headers.

Filter: Regex

Not familiar with regexes? Let's present a few use-case to understand their power:

• Looking for keys ending in 200 ? The default "contains" won't work, because it will match "200133" "343200343" and so on. With regex, it's possible:

  • 200$: means "ends with 200"

  • ^200: means "starts with 200"

• Looking for "london" or "paris" at the same time?

  • london|paris: this will match values containing either london or paris

  • [wz] it's the "same" but for single characters only ('w' or 'z')

Filter: Json Field

Regexes can be complicated (look the above comment). This is why we also offer a simpler way to work with JSON-ish payload: json field filter!

• Define the property you want to filter is

• Define the expected value you want

Here, we want to see only the Avro records where .state.version equals to 227:

Refer to Projecting data to know what's possible, it's the same syntax.