Kappa architecture (streaming only) in Hadoop - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Core principle of Kappa architecture

Which statement best describes the core principle of the Kappa architecture in streaming data processing?

A It processes data only once through a single stream processing pipeline without separate batch layers.

B It requires separate batch and speed layers to handle data processing.

C It stores all data in a data lake before processing in batches.

D It uses multiple batch jobs to update the data warehouse periodically.

💡 Hint

Think about how Kappa architecture simplifies data processing by avoiding multiple layers.

Kappa architecture processes data only once through a single streaming pipeline, eliminating the need for separate batch layers.

❓ Predict Output

intermediate

Output of a streaming filter in Kappa architecture

Given the following Hadoop streaming code snippet that filters events with value > 50, what is the output for input events [30, 60, 45, 80]?

Hadoop

input_events = [30, 60, 45, 80]
filtered_events = list(filter(lambda x: x > 50, input_events))
print(filtered_events)

A [60, 80]

B [30, 45]

C [30, 60, 45, 80]

D []

💡 Hint

Filter keeps only values greater than 50.

The filter function keeps only values greater than 50, so 60 and 80 remain.

❓ data_output

advanced

Result of windowed aggregation in streaming

In a Kappa streaming pipeline, a 5-minute tumbling window sums event values arriving every minute: [2, 3, 5, 7, 1]. What is the sum output for the window?

A 7

B 5

C 10

D 18

💡 Hint

Sum all values in the 5-minute window.

Sum of 2 + 3 + 5 + 7 + 1 equals 18.

🔧 Debug

advanced

Identify error in streaming data processing code

What error will this Hadoop streaming Python code raise?

events = [10, 20, 30]
result = sum(events.filter(lambda x: x > 15))
print(result)

A SyntaxError: invalid syntax

B TypeError: 'int' object is not callable

C AttributeError: 'list' object has no attribute 'filter'

D No error, output is 50

💡 Hint

Check if 'filter' is a method of list objects in Python.

Lists in Python do not have a 'filter' method; 'filter' is a built-in function.

🚀 Application

expert

Choosing Kappa architecture for a use case

You have a system that requires real-time fraud detection on transaction data with minimal latency and no need for batch reprocessing. Which architecture is best suited?

A Data lake architecture, because it stores raw data for later analysis.

B Kappa architecture, because it processes data only once in a streaming pipeline.

C Batch processing architecture, because it handles large volumes efficiently.

D Lambda architecture, because it combines batch and speed layers for accuracy.

💡 Hint

Consider the need for low latency and no batch reprocessing.

Kappa architecture suits real-time needs with a single streaming pipeline, avoiding batch layers.

Practice

1.

What is the main idea behind the Kappa architecture in data processing?

easy

A. Store data in batches and process later

B. Process data only once as a continuous stream

C. Use separate systems for batch and streaming data

D. Process data multiple times for accuracy

Solution

1. Step 1: Understand Kappa architecture concept

   Kappa architecture focuses on processing data as a stream only once, avoiding batch processing.

2. Step 2: Compare with other architectures

   Unlike Lambda architecture, Kappa does not separate batch and streaming layers.

3. Final Answer:

   Process data only once as a continuous stream -> Option B

4. Quick Check:

   Kappa = single stream processing [OK]

Hint: Kappa means one stream, one processing path [OK]

Common Mistakes:

• Confusing Kappa with Lambda architecture
• Thinking Kappa uses batch processing
• Assuming multiple processing passes

2.

Which of the following is a correct component used in a typical Kappa architecture setup?

A) Separate batch and speed layers

B) Hadoop MapReduce for batch jobs

C) HDFS for storing batch data

D) Kafka for streaming data

easy

A. Separate batch and speed layers

B. Hadoop MapReduce for batch jobs

C. HDFS for storing batch data

D. Kafka for streaming data

Solution

1. Step 1: Identify streaming components in Kappa

   Kappa architecture uses streaming tools like Kafka to handle real-time data.

2. Step 2: Exclude batch-specific components

   Batch tools like MapReduce or HDFS batch storage are not part of Kappa's streaming-only design.

3. Final Answer:

   Kafka for streaming data -> Option D

4. Quick Check:

   Kappa uses Kafka, not batch tools [OK]

Hint: Kappa = streaming tools like Kafka only [OK]

Common Mistakes:

• Choosing batch tools like MapReduce
• Selecting batch storage like HDFS
• Thinking Kappa has separate batch layers

3.

Consider this simplified streaming code snippet using Kafka and Spark in a Kappa architecture:

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('KappaExample').getOrCreate()

# Read stream from Kafka topic
stream_df = spark.readStream.format('kafka') \
  .option('kafka.bootstrap.servers', 'localhost:9092') \
  .option('subscribe', 'events') \
  .load()

# Select and cast value to string
value_df = stream_df.selectExpr('CAST(value AS STRING)')

# Write stream to console
query = value_df.writeStream.outputMode('append').format('console').start()

query.awaitTermination()

What will this code do?

medium

A. Continuously print new Kafka messages from 'events' topic to console

B. Read all past Kafka messages once and stop

C. Write data back to Kafka topic 'events'

D. Throw an error due to missing schema

Solution

1. Step 1: Understand the streaming read

   The code reads a continuous stream from Kafka topic 'events' using Spark Structured Streaming.

2. Step 2: Analyze output mode and sink

   Output mode 'append' with console sink means new messages print continuously to console.

3. Final Answer:

   Continuously print new Kafka messages from 'events' topic to console -> Option A

4. Quick Check:

   Streaming read + console output = continuous print [OK]

Hint: Streaming read + console output = live print [OK]

Common Mistakes:

• Thinking it reads all past messages once
• Assuming it writes back to Kafka
• Expecting schema error without explicit schema

4.

Identify the error in this Kappa architecture streaming code snippet using Kafka and Spark:

stream_df = spark.readStream.format('kafka') \
  .option('kafka.bootstrap.servers', 'localhost:9092') \
  .option('subscribe', 'events') \
  .load()

value_df = stream_df.selectExpr('CAST(value AS STRING)')

query = value_df.writeStream.outputMode('complete').format('console').start()

query.awaitTermination()

What is the problem?

medium

A. Output mode 'complete' is invalid for streaming without aggregation

B. Missing schema definition causes failure

C. Kafka server address is incorrect

D. No error, code runs fine

Solution

1. Step 1: Check output mode usage

   Output mode 'complete' requires aggregation queries to output full results.

2. Step 2: Analyze query type

   This query only selects and casts values without aggregation, so 'complete' mode is invalid.

3. Final Answer:

   Output mode 'complete' is invalid for streaming without aggregation -> Option A

4. Quick Check:

   'complete' mode needs aggregation [OK]

Hint: 'complete' mode only with aggregation queries [OK]

Common Mistakes:

• Assuming missing schema causes error
• Thinking Kafka address is wrong
• Believing code runs without error

5.

You want to build a real-time dashboard using Kappa architecture that shows the count of events per user from a Kafka stream. Which approach fits best?

A) Use two separate systems: one for batch counts and one for streaming counts, then merge results.

B) Store Kafka data in HDFS, run daily batch jobs to count events per user, then update dashboard.

C) Use Spark Structured Streaming to read Kafka, group by user, count events, and update dashboard continuously.

D) Read Kafka stream once and save raw data only, no processing.

hard

A. Use two separate systems: one for batch counts and one for streaming counts, then merge results.

B. Store Kafka data in HDFS, run daily batch jobs to count events per user, then update dashboard.

C. Use Spark Structured Streaming to read Kafka, group by user, count events, and update dashboard continuously.

D. Read Kafka stream once and save raw data only, no processing.

Solution

1. Step 1: Understand Kappa architecture goal

   Kappa architecture processes data as a stream only, avoiding batch layers.

2. Step 2: Choose streaming aggregation for real-time dashboard

   Using Spark Structured Streaming to group and count events per user fits Kappa's streaming-only design and real-time needs.

3. Step 3: Exclude batch or hybrid approaches

   Batch jobs or separate batch/stream systems contradict Kappa principles.

4. Final Answer:

   Use Spark Structured Streaming to read Kafka, group by user, count events, and update dashboard continuously. -> Option C

5. Quick Check:

   Real-time + streaming aggregation = Use Spark Structured Streaming to read Kafka, group by user, count events, and update dashboard continuously. [OK]

Hint: Real-time dashboard needs streaming aggregation [OK]

Common Mistakes:

• Choosing batch processing for real-time needs
• Using hybrid batch and streaming systems
• Not processing data, just storing raw stream