Usage scenario
- Actor: A developer of an application that makes use of the Apache Kafka platform
- Context: Design, Build, and Test stages of the Software Development Life Cycle
- Goal: Quickly validate assumptions, verify messaging flow, test software modules exchanging messages with Apache Kafka topics.
While developing an application utilizing Kafka producers or consumers, there is often a need to quickly run a debugging session to validate a piece of code dealing with a Kafka topic. Kafka Magic provides a convenient solution, allowing to quickly publish message to test a consumer, or to search for a message in a topic when validating a producer.
Kafka Magic allows searching for Json or Avro serialized messages using familiar JavaScript language. You can write powerful JavaScript queries referencing any combination of the message fields, headers, and metadata. You can even declare JavaScript functions for your query to handle even more complicated cases. Kafka Magic provides full support for ECMAScript 5.1 standard.
Let’s consider an example where you are developing an application processing customer purchase events it receives from a Kafka topic. You have created a consumer and a message processing code. To verify that the application achieves expected results, you need to make sure that the topic actually contains exactly the message you want to process with all appropriate headers and correct metadata.
Here is how Kafka Magic allows you to search for that message.
Expected message:
{
"orderId": "a-12345",
"orderDate": "2020-01-11T16:46:38.192+00:00",
"items": [{
"sku": "i-1111",
"name": "Item 1",
"quantity": 1
}, {
"sku": "i-2222",
"name": "Item 2",
"quantity": 2
}
]
}
You want to find a message containing an order with orderId == 'a-12345'.
To do that you would run this search query:
return Context.Message.orderId == 'a-12345';
If instead you want to find an order containing item with the sku == 'i-2222', you would write a little more complex query, checking all items in every order:
function findItem(items) {
if (!items) return false;
for (i=0; i < items.length; i++) {
if (items[i].sku == 'i-2222') return true;
}
return false;
}
return findItem(Context.Message.items);
When Kafka Magic finds the message, you can see it deserialized into JSON format with all relevant metadata.
You can publish (produce) a JSON or Avro serialized messages to a Kafka topic using User Interface or Automation Script. In both cases you have options to provide a message content or putting a message in the Context, containing content, headers and a key. You can publish a single message or an array of messages in a single step. You can also set message compression type and level.