Lambda architecture (batch + streaming) in Hadoop - Mini Project: Build & Apply

Lambda Architecture with Batch and Streaming Data

📖 Scenario: You work for a retail company that collects sales data from stores. The data comes in two ways: large files uploaded daily (batch data) and live sales events streaming in real-time (streaming data). You want to combine both to get a complete view of sales.

🎯 Goal: Build a simple Lambda architecture example that processes batch sales data and streaming sales data separately, then combines their results to get total sales per product.

📋 What You'll Learn

Create a batch dataset of sales with product names and quantities

Create a streaming dataset of sales events with product names and quantities

Write batch processing logic to sum quantities per product

Write streaming processing logic to sum quantities per product

Combine batch and streaming results to get total sales per product

Print the combined total sales

💡 Why This Matters

🌍 Real World

Retail companies often collect sales data in batches (daily reports) and streams (live transactions). Combining both helps get up-to-date sales insights.

💼 Career

Data engineers and data scientists use Lambda architecture to handle large-scale data processing combining batch and real-time data for analytics and reporting.

1

Create batch sales data

Create a dictionary called batch_sales with these exact entries: 'apple': 100, 'banana': 150, 'orange': 120 representing quantities sold in batch data.

Hadoop

# Create the batch_sales dictionary with product quantities
# Your code here

Need a hint?

Use curly braces to create a dictionary with keys as product names and values as quantities.

2

Create streaming sales data

Create a list called streaming_sales with these exact tuples representing live sales events: ('apple', 20), ('banana', 30), ('orange', 25).

Hadoop

batch_sales = {'apple': 100, 'banana': 150, 'orange': 120}
# Create the streaming_sales list with live sales tuples
# Your code here

Need a hint?

Use square brackets to create a list of tuples with product names and quantities.

3

Process batch and streaming data

Create a dictionary called streaming_totals that sums quantities per product from streaming_sales using a for loop with variables product and qty. Then create a dictionary called total_sales that adds batch and streaming quantities per product.

Hadoop

batch_sales = {'apple': 100, 'banana': 150, 'orange': 120}
streaming_sales = [('apple', 20), ('banana', 30), ('orange', 25)]
# Sum streaming sales quantities per product into streaming_totals
# Then add batch and streaming quantities into total_sales
# Your code here

Need a hint?

Use a for loop to sum streaming sales. Use dict.get(key, 0) to handle missing keys.

4

Print total sales per product

Write a for loop with variables product and qty to iterate over total_sales.items() and print the product and total quantity in the format: Product: apple, Total Sold: 120.

Hadoop

batch_sales = {'apple': 100, 'banana': 150, 'orange': 120}
streaming_sales = [('apple', 20), ('banana', 30), ('orange', 25)]

streaming_totals = {}
for product, qty in streaming_sales:
    streaming_totals[product] = streaming_totals.get(product, 0) + qty

total_sales = {}
for product in batch_sales:
    total_sales[product] = batch_sales[product] + streaming_totals.get(product, 0)

# Print total sales per product
# Your code here

Need a hint?

Use a for loop over total_sales.items() and an f-string to format the print output.