Filtering & Projecting data

Search…
Conduktor provides many ways when it comes to filtering and projecting data. This is an important use-case when using Conduktor:
you're looking for a needle in your whole topic history
you'd like to trace the events of your entity "abc" from today only
you'd like to have in the same view how a specific field changed across time
you're looking for what happened between 5.05pm and 5.10pm yesterday
...
Projecting data
You can choose to display only a subset of fields from a message. This works for JSON and Avro data (because they are converted to JSON). A section "Value projection" will appear where you can set a "jq" expression (documentation).
For instance, here, we only care about .state.version in:
1
{
2
  "id" : "84504272-8473-48f6-8ad5-a7e836e58dc7",
3
  "type" : "ItemAdded",
4
  "delta" : {
5
    "itemId" : "1abcdcc1-cc36-4ae4-a6eb-3582df6f8e65"
6
  },
7
  "state" : {
8
    "id" : "570ccfea-6c8a-44a3-83a9-ab54797d104f",
9
    "version" : 6, // we care only about this! <<<<<<<<<<
10
    ...
11
}
Copied!
We setup this as our filter, and we'll only see the value of version:
Typical use-cases:
.property.subprop.value : just nested objects
.items[].object: access objects inside an array
.items[].object.property: access fields inside objects inside an array
.items | length: returns the length of an array
.x.obj | keys : list the keys "obj"
contains({"state":{"version": 1}}): "true" if the element with this value exists
...
​
There are many functions availables: check the jq documentation​
If your jq filter is invalid, or only fails for some records, Conduktor will display the jq error in place of the value:
If you're using Avro, it's also possible to provide a custom schema (with less fields) by using the format "Avro (Custom Schema)", see Avro without Confluent?​
Filtering data
It's possible to combine multiple filters on multiple dimensions: time, quantity, fields.
Choose where to start from
By default, Conduktor uses "now (latest)": it means that only new incoming data will appear in Conduktor, not the past.
time based: a hour, a date
offset based:
an existing consumer group: useful to see why an application is failing there, like a deserialization error
latest minux X offsets: just the latest data published, no matter when
an offset: this will be the starting offset for all the partitions consumed. You generally use this and select only one partition (see Advanced).
Choose until when to consume
Sometimes, you don't want to continuously consume incoming data and just want to consume a few messages and automatically stop.
forever is the default: never stop
number of messages: when we just want to take a peek
max size: rare use-case, when the size matters
timestamp: consume data up until a particular date.
also useful when combined to a start date
Filter messages by their key/value/headers
We offer many ways to select only the interesting key/value you're looking for.
contains: a simple string we're looking for (fast)
equals: the whole key or value must match, often used for keys (where it's generally a simple id)
regex: for more complex case when you know regexes 🤓. Can be quite slow when the payloads are large.
json field: it's the same format as in "Projecting data" (above), except the extracted value must match a value you'll expect
Most works with key, value and headers.
If you set several filters, it's treated like an "AND".
We don't support "OR" operand; we don't have any complex UI to tell "(this or (that and that)) or this"
The different filters options: contains, regex, json
Filter: Regex
Not familiar with regexes? Let's present a few use-case to understand their power:
Looking for keys ending in 200 ? The default "contains" won't work, because it will match "200133" "343200343" and so on. With regex, it's possible:
200$: means "ends with 200"
^200: means "starts with 200"
Looking for "london" or "paris" at the same time?
london|paris: this will match values containing either london or paris
[wz] it's the "same" but for single characters only ('w' or 'z')
Beware of "special" characters like "(" "{" or ".", they need to be "escaped".
If you want to match "1.2.2" or "1.2.3", use "1\.2\.[23]" and NOT "1.2.[23]" or you will match "10243": "." means "any character"
You can use regex with a json-ish syntax if you want to match the json payload, and not use the Json Field filter.
"state"\s:\s{\s"id"\s:\s".",\s"version"\s:\s*227
This is the "equivalent" of the json field filter: .state.version = 227
Filter: Json Field
Regexes can be complicated (look the above comment). This is why we also offer a simpler way to work with JSON-ish payload: json field filter!
Define the property you want to filter is
Define the expected value you want
Here, we want to see only the Avro records where .state.version equals to 227:
Refer to Projecting data to know what's possible, it's the same syntax.
Composed Filter
introduced in 2.18.2
Composed Filter allow to use multiple filter as one. Two kind of composition are available
All, [formally AND]. this composer will check that records match all the filtering rules
At least one, [formally OR]. this composer will check that records match at least one filtering rules
All kinds of existing filters can be used in composed filter.
To add or remove a filter use the + & - buttons.