Search for messages in a topic using JavaScript

Whether it is Json or Avro serialized message you can search for it using JavaScript query referencing any combination of the message fields, headers, and metadata. You can even declare JavaScript functions to handle a complicated case. Full support for ECMAScript 5.1 standard.

Kafka Magic: Query Topics using JavaScript

The query references the Context object to get required properties and returns boolean value.

Context object

When writing a JavaScript query you are provided with the Context object holding message content and metadata.

Here is the definition for the Context:

{
    Message: any;
    Topic: string;
    Partition: number;
    Offset: number;
    Timestamp: Date;
    Headers: any;
    Key: number[];
}

Context properties

  • Message - exact representation of the message, an object having all properties of the deserialized message.
  • Topic - the topic name.
  • Partition - partition id.
  • Offset - message’s offset in the partition.
  • Timestamp - the time either set by the producer on message creation time, or by the broker on message insertion time (depending on cluster configuration).
  • Headers - an object which properties represent message headers.
  • Key - binary representation of the Key - sequence of numbers each representing a byte.

Feel free to use all these values in your query.

Kafka Magic Search Query

Your query can be pretty much any JavaScript statement, or a set of statements. The only requirements is that at some point it must return a boolean value specifying if this is the message you are looking for.

Query can be as simple as a single return true; statement, or a complex code inspecting the Context object, making calculations, declaring variables and functions. Just don’t forget to return true if you want this particular message to be included in the results, or false otherwise.

Example

In this example you want to find messages which satisfy three conditions:

  • the header ‘myHeader’ has value ‘myHeaderValue’
  • the message field ‘title’ has value ‘Title value’
  • the message field ‘price’ has value greater than 12.34

In this case the query should return true when all these conditions are true:

return Context.Headers.myHeader === 'myHeaderValue' && Context.Message.title === 'Title value' && Context.Message.price > 12.34;

Filter Messages

When you enter a JavaScript query to find particular messages in a Kafka topic, the system applies this query to each and every message in the topic. For large topic this can take a lot of time.

Filtering by timestamp or partition/offset allows you to significantly reduce time system needs to find a particular message.

Filtering by Timestamp

Since version 0.10.0 Apache Kafka has an index based on message timestamps, which makes this operation very efficient. So, while using Kafka Magic to search for messages in a topic you can limit the scope of the search by specifying Start and Stop timestamps.

Kafka Magic: Filtering by Timestamp

Filtering by Partition/Offset

If you know approximate offset values for the Kafka message you are trying to find, filtering by partition/offset can be a very efficient way of reducing the search time.

Kafka Magic: Filtering by Partition/Offset