Lambda architecture (batch + streaming) in Hadoop - Step-by-Step Execution

Concept Flow - Lambda architecture (batch + streaming)

Raw Data Input

↓

Batch Layer

↓

Batch Views

↓

Serving Layer

↓

User Queries

Data flows from raw input into two paths: batch for large-scale processing and speed for real-time updates. Both results combine in the serving layer to answer user queries.

Execution Sample

Hadoop

1. Collect raw data continuously
2. Batch layer processes data in large chunks
3. Speed layer processes data in real-time
4. Serving layer merges batch and speed views
5. User queries get combined results

This shows how data moves through batch and speed layers, then merges for user queries.

Execution Table

Step	Layer	Action	Data Processed	Output Produced
1	Raw Data Input	Collect data stream	Events 1-1000	Raw data stored
2	Batch Layer	Process batch data	Events 1-1000	Batch view updated
3	Speed Layer	Process real-time data	Events 1001-1100	Real-time view updated
4	Serving Layer	Merge batch and speed views	Batch + Real-time views	Unified view ready
5	User Queries	Query unified view	Unified view	Query results returned
6	Batch Layer	Next batch processing	Events 1-2000	Batch view updated
7	Speed Layer	Process new real-time data	Events 2001-2100	Real-time view updated
8	Serving Layer	Merge updated views	Batch + Real-time views	Unified view refreshed
9	User Queries	Query refreshed view	Unified view	Updated query results
10	End	No more data or queries	-	-

💡 Execution stops when no new data arrives or queries are made.

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 6	After Step 7	After Step 8
Raw Data	Empty	Events 1-1000	Events 1-1100	Events 1-1100	Events 1-2000	Events 1-2100	Events 1-2100
Batch View	Empty	Processed Events 1-1000	Processed Events 1-1000	Processed Events 1-1000	Processed Events 1-2000	Processed Events 1-2000	Processed Events 1-2000
Speed View	Empty	Empty	Processed Events 1001-1100	Processed Events 1001-1100	Processed Events 1001-1100	Processed Events 2001-2100	Processed Events 2001-2100
Unified View	Empty	Empty	Empty	Batch + Speed Views merged	Empty	Empty	Batch + Speed Views merged

Key Moments - 3 Insights

Why do we need both batch and speed layers instead of just one?

Batch layer handles large data accurately but slowly (see Step 2 and 6). Speed layer handles recent data quickly but less accurately (see Step 3 and 7). Combining both gives fast and accurate results (Step 4 and 8).

How does the serving layer combine data from batch and speed layers?

Serving layer merges batch views (historical data) and speed views (real-time data) to create a unified view for queries (Step 4 and 8 in execution_table).

What happens if the speed layer misses some data?

The batch layer will eventually process all data in large chunks, correcting any misses from the speed layer, ensuring accuracy over time (compare Step 2 and 6).

Visual Quiz - 3 Questions

Look at the execution table, what data does the speed layer process at Step 3?

A Events 1001-1100

B Events 1-1000

C Events 1-1100

D Events 2001-2100

💡 Hint

Check the 'Data Processed' column for Step 3 in the execution_table.

At which step does the serving layer first merge batch and speed views?

A Step 2

B Step 3

C Step 4

D Step 6

💡 Hint

Look for 'Serving Layer' and 'Merge batch and speed views' in the execution_table.

If the batch layer processes data more frequently, how would the batch view change in variable_tracker?

A Batch view updates less often

B Batch view updates more often with more data

C Speed view updates more often

D Unified view stops updating

💡 Hint

Refer to the 'Batch View' row in variable_tracker and think about batch processing frequency.

Concept Snapshot

Lambda Architecture combines batch and streaming data processing.
Batch layer processes large data sets slowly but accurately.
Speed layer processes recent data quickly but less accurately.
Serving layer merges both views for fast and accurate queries.
This design balances latency and accuracy in big data systems.

Full Transcript

Lambda architecture splits data processing into batch and speed layers. Raw data flows into both layers. The batch layer processes large chunks of data to create accurate batch views. The speed layer processes data in real-time to create fast but approximate views. The serving layer merges these views to provide users with up-to-date and accurate query results. This approach ensures low latency and high accuracy by combining the strengths of both batch and streaming processing.

Practice

1. What is the main purpose of the batch layer in Lambda architecture?

easy

A. To process all data for accurate and complete results

B. To handle real-time data streams quickly

C. To store raw data without processing

D. To visualize data for end users

Solution

1. Step 1: Understand batch layer role

   The batch layer processes all accumulated data to provide accurate and comprehensive views.

2. Step 2: Compare with other layers

   The speed layer handles real-time data, not the batch layer.

3. Final Answer:

   To process all data for accurate and complete results -> Option A

4. Quick Check:

   Batch layer = accurate full data processing [OK]

Hint: Batch layer = full data, accurate results [OK]

Common Mistakes:

• Confusing batch layer with speed layer
• Thinking batch layer processes data in real-time
• Assuming batch layer only stores raw data

2. Which of the following is the correct component of Lambda architecture for handling real-time data?

easy

A. Batch layer

B. Speed layer

C. Serving layer

D. Storage layer

Solution

1. Step 1: Identify real-time processing component

   The speed layer is designed to process data streams in real-time for quick insights.

2. Step 2: Eliminate other options

   Batch layer processes full data, serving layer provides query results, storage layer is not a standard Lambda component.

3. Final Answer:

   Speed layer -> Option B

4. Quick Check:

   Speed layer = real-time data processing [OK]

Hint: Speed layer = real-time data [OK]

Common Mistakes:

• Choosing batch layer for real-time data
• Confusing serving layer with speed layer
• Thinking storage layer is a Lambda component

3. Consider this simplified pseudocode for Lambda architecture layers:

batch_result = batch_layer(data)
speed_result = speed_layer(stream)
final_result = merge(batch_result, speed_result)
print(final_result)

If batch_layer returns {'count': 100} and speed_layer returns {'count': 5}, what will final_result be after merging by summing counts?

medium

A. {'count': 105}

B. {'count': 100}

C. {'count': 5}

D. Error: Cannot merge dictionaries

Solution

1. Step 1: Understand merging logic

   Merging sums the 'count' values from batch and speed results: 100 + 5 = 105.

2. Step 2: Confirm final result

   The final dictionary after merging is {'count': 105}.

3. Final Answer:

   {'count': 105} -> Option A

4. Quick Check:

   100 + 5 = 105 [OK]

Hint: Sum batch and speed counts for final result [OK]

Common Mistakes:

• Ignoring speed layer result
• Assuming merge replaces batch with speed
• Thinking merge causes error

4. This code snippet tries to implement the speed layer but has an error:

def speed_layer(stream):
    for record in stream:
        process(record)
    return processed_data

stream = get_stream()
result = speed_layer(stream)
print(result)

What is the likely error here?

medium

A. process() function is missing

B. stream is not iterable

C. processed_data is not defined before return

D. print() syntax error

Solution

1. Step 1: Check variable initialization

   The variable processed_data is returned but never defined or updated inside the function.

2. Step 2: Verify other parts

   Assuming stream is iterable and process() exists, the main error is missing processed_data definition.

3. Final Answer:

   processed_data is not defined before return -> Option C

4. Quick Check:

   Undefined variable causes error [OK]

Hint: Check all variables are defined before return [OK]

Common Mistakes:

• Assuming process() is missing without checking
• Thinking stream is not iterable
• Ignoring variable initialization

5. In a Lambda architecture, if the batch layer updates data every 24 hours and the speed layer processes data every second, what is the best way to ensure the serving layer shows the most accurate and up-to-date results?

hard

A. Use the speed layer results immediately and replace them with batch results after 24 hours

B. Only show batch layer results to avoid inconsistency

C. Ignore speed layer and rely on batch layer for all queries

D. Merge speed and batch results continuously, updating serving layer with combined data

Solution

1. Step 1: Understand serving layer role

   The serving layer combines batch and speed layer outputs to provide accurate and timely results.

2. Step 2: Evaluate options

   Merge speed and batch results continuously, updating serving layer with combined data merges both results continuously, ensuring the serving layer is both up-to-date and accurate.

3. Final Answer:

   Merge speed and batch results continuously, updating serving layer with combined data -> Option D

4. Quick Check:

   Serving layer = combined batch + speed data [OK]

Hint: Serving layer merges batch and speed data continuously [OK]

Common Mistakes:

• Using only speed or batch layer results alone
• Replacing speed results abruptly without merging
• Ignoring serving layer's combining role