Why stream processing transforms data in Kafka - See It in Action

Why Stream Processing Transforms Data

📖 Scenario: Imagine you run a small online store. You get many orders every minute. You want to quickly know which products are popular right now. To do this, you use stream processing to transform the live order data as it flows in.

🎯 Goal: Build a simple Kafka stream processing program that reads order data, transforms it to count products sold, and outputs the counts.

📋 What You'll Learn

Create a Kafka topic with sample order data

Set up a stream processing configuration variable

Write a stream processing logic to transform orders into product counts

Print the final product counts

💡 Why This Matters

🌍 Real World

Online stores and many apps use stream processing to analyze live data instantly, like counting popular products or tracking user actions.

💼 Career

Understanding stream processing basics helps in roles like data engineer, backend developer, or any job working with real-time data pipelines.

1

Create the Kafka topic with sample order data

Create a Kafka topic called orders with these exact messages: {"order_id": 1, "product": "apple"}, {"order_id": 2, "product": "banana"}, {"order_id": 3, "product": "apple"}.

Kafka

# Your code here to create the 'orders' topic and add messages

Need a hint?

Use a list called orders with dictionaries for each order.

2

Set up the stream processing configuration

Create a variable called product_counts and set it to an empty dictionary {} to hold counts of each product.

Kafka

orders = [
    {"order_id": 1, "product": "apple"},
    {"order_id": 2, "product": "banana"},
    {"order_id": 3, "product": "apple"}
]
# Your code here to create product_counts dictionary

Need a hint?

Use an empty dictionary to store counts.

3

Transform orders to count products

Use a for loop with variable order to iterate over orders. Inside the loop, get the product name from order["product"]. Increase the count for that product in product_counts by 1. If the product is not yet in product_counts, add it with count 1.

Kafka

orders = [
    {"order_id": 1, "product": "apple"},
    {"order_id": 2, "product": "banana"},
    {"order_id": 3, "product": "apple"}
]
product_counts = {}
# Your code here to count products from orders

Need a hint?

Use a loop and check if product is already counted.

4

Print the product counts

Write print(product_counts) to display the final counts of products sold.

Kafka

orders = [
    {"order_id": 1, "product": "apple"},
    {"order_id": 2, "product": "banana"},
    {"order_id": 3, "product": "apple"}
]
product_counts = {}
for order in orders:
    product = order["product"]
    if product in product_counts:
        product_counts[product] += 1
    else:
        product_counts[product] = 1
# Your code here to print product_counts

Need a hint?

Use print to show the dictionary with counts.