Lambda architecture (batch + streaming) in Hadoop - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Core components of Lambda Architecture

Which of the following best describes the three main layers of Lambda Architecture?

A Batch layer for storing master data, speed layer for real-time views, and serving layer for query responses

B Data ingestion layer, machine learning layer, and visualization layer

C Storage layer, processing layer, and presentation layer

D Data cleaning layer, batch processing layer, and streaming layer

💡 Hint

Think about how Lambda Architecture handles both historical and real-time data.

The batch layer stores the master dataset and precomputes batch views. The speed layer handles real-time data for low latency. The serving layer merges results for queries.

❓ Predict Output

intermediate

Output of batch and speed layer data merge

Given the batch view {'user1': 100, 'user2': 150} and speed view {'user2': 20, 'user3': 30}, what is the merged output in the serving layer summing values per user?

Hadoop

batch_view = {'user1': 100, 'user2': 150}
speed_view = {'user2': 20, 'user3': 30}
merged = {}
for user in set(batch_view) | set(speed_view):
    merged[user] = batch_view.get(user, 0) + speed_view.get(user, 0)
print(merged)

A {'user1': 100, 'user2': 170, 'user3': 30}

B {'user1': 100, 'user2': 150, 'user3': 30}

C {'user1': 100, 'user2': 20, 'user3': 30}

D {'user1': 0, 'user2': 170, 'user3': 30}

💡 Hint

Remember to add values from both batch and speed views for each user.

The serving layer merges batch and speed views by summing values for each user key.

🔧 Debug

advanced

Identify the error in streaming data processing code

What error will the following Hadoop streaming code produce?

from pyspark.streaming import StreamingContext
ssc = StreamingContext(sc, 1)
lines = ssc.socketTextStream('localhost', 9999)
words = lines.flatMap(lambda line: line.split())
wordCounts = words.map(lambda word: (word, 1)).reduceByKey(lambda a, b: a + b)
wordCounts.pprint()
ssc.start()
ssc.awaitTermination()

A TypeError because flatMap expects a list but gets a string

B NameError because 'sc' (SparkContext) is not defined

C RuntimeError because socketTextStream port is closed

D SyntaxError due to missing colon in lambda

💡 Hint

Check if all required objects are initialized before use.

The code uses 'sc' (SparkContext) without defining or initializing it, causing a NameError.

❓ data_output

advanced

Result of batch processing with Hadoop MapReduce

Given a text file with lines:
"apple banana apple"
"banana orange apple"
What is the output count of words after a Hadoop MapReduce word count job?

Hadoop

Input lines:
"apple banana apple"
"banana orange apple"

Map step: emits (word, 1) for each word
Reduce step: sums counts per word

Expected output format: (word, total_count)

A (apple, 3)\n(banana, 3)\n(orange, 1)

B (apple, 2)\n(banana, 2)\n(orange, 1)

C (apple, 3)\n(banana, 2)\n(orange, 1)

D (apple, 3)\n(banana, 2)\n(orange, 2)

💡 Hint

Count how many times each word appears in all lines combined.

"apple" appears 3 times, "banana" 2 times, "orange" 1 time.

🚀 Application

expert

Choosing Lambda Architecture for a use case

You have a system that needs to process large historical data and also provide real-time analytics with low latency. Which scenario best justifies using Lambda Architecture?

A A simple calculator app performing local computations without data storage

B A static website serving fixed content without updates

C A batch-only system processing monthly sales reports without real-time needs

D A social media platform analyzing user posts historically and showing live trending topics instantly

💡 Hint

Think about when both batch and streaming processing are needed together.

Lambda Architecture fits systems requiring both batch processing of large data and real-time streaming for immediate insights.

Practice

1. What is the main purpose of the batch layer in Lambda architecture?

easy

A. To process all data for accurate and complete results

B. To handle real-time data streams quickly

C. To store raw data without processing

D. To visualize data for end users

Solution

1. Step 1: Understand batch layer role

   The batch layer processes all accumulated data to provide accurate and comprehensive views.

2. Step 2: Compare with other layers

   The speed layer handles real-time data, not the batch layer.

3. Final Answer:

   To process all data for accurate and complete results -> Option A

4. Quick Check:

   Batch layer = accurate full data processing [OK]

Hint: Batch layer = full data, accurate results [OK]

Common Mistakes:

• Confusing batch layer with speed layer
• Thinking batch layer processes data in real-time
• Assuming batch layer only stores raw data

2. Which of the following is the correct component of Lambda architecture for handling real-time data?

easy

A. Batch layer

B. Speed layer

C. Serving layer

D. Storage layer

Solution

1. Step 1: Identify real-time processing component

   The speed layer is designed to process data streams in real-time for quick insights.

2. Step 2: Eliminate other options

   Batch layer processes full data, serving layer provides query results, storage layer is not a standard Lambda component.

3. Final Answer:

   Speed layer -> Option B

4. Quick Check:

   Speed layer = real-time data processing [OK]

Hint: Speed layer = real-time data [OK]

Common Mistakes:

• Choosing batch layer for real-time data
• Confusing serving layer with speed layer
• Thinking storage layer is a Lambda component

3. Consider this simplified pseudocode for Lambda architecture layers:

batch_result = batch_layer(data)
speed_result = speed_layer(stream)
final_result = merge(batch_result, speed_result)
print(final_result)

If batch_layer returns {'count': 100} and speed_layer returns {'count': 5}, what will final_result be after merging by summing counts?

medium

A. {'count': 105}

B. {'count': 100}

C. {'count': 5}

D. Error: Cannot merge dictionaries

Solution

1. Step 1: Understand merging logic

   Merging sums the 'count' values from batch and speed results: 100 + 5 = 105.

2. Step 2: Confirm final result

   The final dictionary after merging is {'count': 105}.

3. Final Answer:

   {'count': 105} -> Option A

4. Quick Check:

   100 + 5 = 105 [OK]

Hint: Sum batch and speed counts for final result [OK]

Common Mistakes:

• Ignoring speed layer result
• Assuming merge replaces batch with speed
• Thinking merge causes error

4. This code snippet tries to implement the speed layer but has an error:

def speed_layer(stream):
    for record in stream:
        process(record)
    return processed_data

stream = get_stream()
result = speed_layer(stream)
print(result)

What is the likely error here?

medium

A. process() function is missing

B. stream is not iterable

C. processed_data is not defined before return

D. print() syntax error

Solution

1. Step 1: Check variable initialization

   The variable processed_data is returned but never defined or updated inside the function.

2. Step 2: Verify other parts

   Assuming stream is iterable and process() exists, the main error is missing processed_data definition.

3. Final Answer:

   processed_data is not defined before return -> Option C

4. Quick Check:

   Undefined variable causes error [OK]

Hint: Check all variables are defined before return [OK]

Common Mistakes:

• Assuming process() is missing without checking
• Thinking stream is not iterable
• Ignoring variable initialization

5. In a Lambda architecture, if the batch layer updates data every 24 hours and the speed layer processes data every second, what is the best way to ensure the serving layer shows the most accurate and up-to-date results?

hard

A. Use the speed layer results immediately and replace them with batch results after 24 hours

B. Only show batch layer results to avoid inconsistency

C. Ignore speed layer and rely on batch layer for all queries

D. Merge speed and batch results continuously, updating serving layer with combined data

Solution

1. Step 1: Understand serving layer role

   The serving layer combines batch and speed layer outputs to provide accurate and timely results.

2. Step 2: Evaluate options

   Merge speed and batch results continuously, updating serving layer with combined data merges both results continuously, ensuring the serving layer is both up-to-date and accurate.

3. Final Answer:

   Merge speed and batch results continuously, updating serving layer with combined data -> Option D

4. Quick Check:

   Serving layer = combined batch + speed data [OK]

Hint: Serving layer merges batch and speed data continuously [OK]

Common Mistakes:

• Using only speed or batch layer results alone
• Replacing speed results abruptly without merging
• Ignoring serving layer's combining role