Filter and map operations in Kafka - Mini Project: Build & Apply

Filter and map operations

📖 Scenario: You are working with a stream of messages representing orders in a Kafka topic. Each message contains an order ID and the order amount. You want to process this stream to find only the orders with amounts greater than 100 and then create a new stream with just the order IDs of these large orders.

🎯 Goal: Build a Kafka Streams application that filters orders with amounts greater than 100 and maps them to their order IDs.

📋 What You'll Learn

Create a KStream named orders with sample order data

Create a variable threshold set to 100

Use filter on orders to keep only orders with amount greater than threshold

Use map to transform filtered orders to just their order IDs

Print the resulting stream to the console

💡 Why This Matters

🌍 Real World

Filtering and mapping streams is common in real-time data processing, such as monitoring orders, transactions, or sensor data.

💼 Career

Kafka Streams skills are valuable for backend developers, data engineers, and anyone working with real-time data pipelines.

1

Create the initial orders stream

Create a KStream called orders with these exact entries: ("order1", 50), ("order2", 150), ("order3", 200), ("order4", 80).

Kafka

# Create a KStream named orders with the given order data
# Your code here

Need a hint?

Use builder.stream("orders-topic") to create the stream. The actual data is assumed to be in the topic.

2

Set the threshold value

Create an integer variable called threshold and set it to 100.

Kafka

KStream orders = builder.stream("orders-topic");
// For this exercise, assume orders stream contains: ("order1", 50), ("order2", 150), ("order3", 200), ("order4", 80)

// Create threshold variable
# Your code here

Need a hint?

Just create a simple integer variable named threshold and assign 100.

3

Filter and map the orders stream

Use filter on orders to keep only entries where the value (order amount) is greater than threshold. Then use map to transform the filtered stream to only the order IDs (the keys). Store the result in a KStream called largeOrderIds.

Kafka

KStream orders = builder.stream("orders-topic");
// For this exercise, assume orders stream contains: ("order1", 50), ("order2", 150), ("order3", 200), ("order4", 80)

int threshold = 100;

// Filter orders and map to order IDs
# Your code here

Need a hint?

Use filter with a lambda checking amount > threshold. Then use map with a lambda returning KeyValue.pair(orderId, orderId).

4

Print the filtered order IDs

Use foreach on largeOrderIds to print each order ID with the exact format: "Large order ID: <orderId>".

Kafka

KStream orders = builder.stream("orders-topic");
// For this exercise, assume orders stream contains: ("order1", 50), ("order2", 150), ("order3", 200), ("order4", 80)

int threshold = 100;

KStream largeOrderIds = orders
    .filter((orderId, amount) -> amount > threshold)
    .map((orderId, amount) -> KeyValue.pair(orderId, orderId));

// Print each large order ID
# Your code here

Need a hint?

Use foreach with a lambda that prints the string exactly as shown.