Lambda architecture (batch + streaming) in Hadoop - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Core components of Lambda Architecture

Which of the following best describes the three main layers of Lambda Architecture?

A Batch layer for storing master data, speed layer for real-time views, and serving layer for query responses

B Data ingestion layer, machine learning layer, and visualization layer

C Storage layer, processing layer, and presentation layer

D Data cleaning layer, batch processing layer, and streaming layer

💡 Hint

Think about how Lambda Architecture handles both historical and real-time data.

The batch layer stores the master dataset and precomputes batch views. The speed layer handles real-time data for low latency. The serving layer merges results for queries.

❓ Predict Output

intermediate

Output of batch and speed layer data merge

Given the batch view {'user1': 100, 'user2': 150} and speed view {'user2': 20, 'user3': 30}, what is the merged output in the serving layer summing values per user?

Hadoop

batch_view = {'user1': 100, 'user2': 150}
speed_view = {'user2': 20, 'user3': 30}
merged = {}
for user in set(batch_view) | set(speed_view):
    merged[user] = batch_view.get(user, 0) + speed_view.get(user, 0)
print(merged)

A {'user1': 100, 'user2': 170, 'user3': 30}

B {'user1': 100, 'user2': 150, 'user3': 30}

C {'user1': 100, 'user2': 20, 'user3': 30}

D {'user1': 0, 'user2': 170, 'user3': 30}

💡 Hint

Remember to add values from both batch and speed views for each user.

The serving layer merges batch and speed views by summing values for each user key.

🔧 Debug

advanced

Identify the error in streaming data processing code

What error will the following Hadoop streaming code produce?

from pyspark.streaming import StreamingContext
ssc = StreamingContext(sc, 1)
lines = ssc.socketTextStream('localhost', 9999)
words = lines.flatMap(lambda line: line.split())
wordCounts = words.map(lambda word: (word, 1)).reduceByKey(lambda a, b: a + b)
wordCounts.pprint()
ssc.start()
ssc.awaitTermination()

A TypeError because flatMap expects a list but gets a string

B NameError because 'sc' (SparkContext) is not defined

C RuntimeError because socketTextStream port is closed

D SyntaxError due to missing colon in lambda

💡 Hint

Check if all required objects are initialized before use.

The code uses 'sc' (SparkContext) without defining or initializing it, causing a NameError.

❓ data_output

advanced

Result of batch processing with Hadoop MapReduce

Given a text file with lines:
"apple banana apple"
"banana orange apple"
What is the output count of words after a Hadoop MapReduce word count job?

Hadoop

Input lines:
"apple banana apple"
"banana orange apple"

Map step: emits (word, 1) for each word
Reduce step: sums counts per word

Expected output format: (word, total_count)

A (apple, 3)\n(banana, 3)\n(orange, 1)

B (apple, 2)\n(banana, 2)\n(orange, 1)

C (apple, 3)\n(banana, 2)\n(orange, 1)

D (apple, 3)\n(banana, 2)\n(orange, 2)

💡 Hint

Count how many times each word appears in all lines combined.

"apple" appears 3 times, "banana" 2 times, "orange" 1 time.

🚀 Application

expert

Choosing Lambda Architecture for a use case

You have a system that needs to process large historical data and also provide real-time analytics with low latency. Which scenario best justifies using Lambda Architecture?

A A simple calculator app performing local computations without data storage

B A static website serving fixed content without updates

C A batch-only system processing monthly sales reports without real-time needs

D A social media platform analyzing user posts historically and showing live trending topics instantly

💡 Hint

Think about when both batch and streaming processing are needed together.

Lambda Architecture fits systems requiring both batch processing of large data and real-time streaming for immediate insights.