Kappa architecture (streaming only) in Hadoop - Step-by-Step Execution

Concept Flow - Kappa architecture (streaming only)

Data Ingested from Source

↓

Stream Processing Layer

↓

Real-time Processing & Analytics

↓

Serving Layer / Output

↓

Feedback / Monitoring

↩Back to Stream Processing Layer

Data flows continuously from source through a single streaming layer for real-time processing and output, with feedback looping back for monitoring.

Execution Sample

Hadoop

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('KappaExample').getOrCreate()
streamingDF = spark.readStream.format('kafka').option('subscribe', 'topic').load()
processedDF = streamingDF.selectExpr('CAST(value AS STRING)')
query = processedDF.writeStream.format('console').start()
query.awaitTermination()

This code reads streaming data from Kafka, processes it by casting values to strings, and outputs results to the console in real-time.

Execution Table

Step	Action	Input Data	Processing	Output / State
1	Start Spark Session	None	Initialize Spark streaming context	Spark session ready
2	Read Stream	Kafka topic messages	Connect to Kafka, read raw bytes	Streaming DataFrame with raw data
3	Select & Cast	Raw bytes	Convert bytes to string	Streaming DataFrame with string values
4	Write Stream	Processed data	Output to console sink	Streaming query started
5	Await Termination	Streaming query active	Keep process running	Continuous output of streaming data
6	Stop	User interrupt	Stop streaming query	Streaming stopped

💡 Streaming stops when user interrupts or error occurs; otherwise runs continuously.

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	Final
spark	None	SparkSession object	SparkSession object	SparkSession object	SparkSession object
streamingDF	None	DataFrame with raw Kafka data	DataFrame with string values	DataFrame with string values	DataFrame with string values
processedDF	None	None	DataFrame with string values	DataFrame with string values	DataFrame with string values
query	None	None	None	StreamingQuery object	StreamingQuery object or stopped

Key Moments - 3 Insights

Why does the streaming query keep running instead of stopping after processing one batch?

Because streaming queries in Kappa architecture run continuously to process data in real-time, as shown in execution_table step 5 where awaitTermination keeps the process alive.

What happens if new data arrives after the initial processing?

The streaming layer processes new data continuously, updating the output in real-time, demonstrated by the loop in concept_flow and continuous output in execution_table step 5.

Why is there only one processing layer instead of batch and streaming layers?

Kappa architecture uses a single streaming layer for simplicity and real-time processing, unlike Lambda which has separate batch and streaming layers, as described in concept_flow.

Visual Quiz - 3 Questions

Look at the execution_table at step 3, what transformation is applied to the input data?

A Casting raw bytes to string

B Filtering data by condition

C Aggregating data by key

D Writing data to storage

💡 Hint

Check the 'Processing' column in step 3 of execution_table.

At which step does the streaming query start writing output to the console?

A Step 3

B Step 4

C Step 2

D Step 5

💡 Hint

Look at the 'Action' and 'Output / State' columns in execution_table.

If the streaming query is stopped by the user, which step in the execution_table corresponds to this?

A Step 1

B Step 4

C Step 6

D Step 5

💡 Hint

Refer to the 'Action' column describing stopping the streaming query.

Concept Snapshot

Kappa architecture uses a single streaming layer for real-time data processing.
Data flows continuously from source to output.
No separate batch layer is used.
Streaming queries run continuously until stopped.
Ideal for simple, real-time analytics pipelines.

Full Transcript

Kappa architecture focuses on processing data as a continuous stream. Data is ingested from sources like Kafka and processed in a single streaming layer. This layer transforms and outputs data in real-time, without a separate batch layer. The streaming query runs continuously, processing new data as it arrives. The example code shows reading from Kafka, casting data to strings, and outputting to the console. The execution table traces each step from starting Spark to stopping the stream. Variables like streamingDF and query change state as the stream runs. Key moments clarify why streaming runs continuously and why only one processing layer is used. The visual quiz tests understanding of data transformation, output start, and stopping the stream. The snapshot summarizes the core idea: one streaming layer for real-time processing in Kappa architecture.

Practice

1.

What is the main idea behind the Kappa architecture in data processing?

easy

A. Store data in batches and process later

B. Process data only once as a continuous stream

C. Use separate systems for batch and streaming data

D. Process data multiple times for accuracy

Solution

1. Step 1: Understand Kappa architecture concept

   Kappa architecture focuses on processing data as a stream only once, avoiding batch processing.

2. Step 2: Compare with other architectures

   Unlike Lambda architecture, Kappa does not separate batch and streaming layers.

3. Final Answer:

   Process data only once as a continuous stream -> Option B

4. Quick Check:

   Kappa = single stream processing [OK]

Hint: Kappa means one stream, one processing path [OK]

Common Mistakes:

• Confusing Kappa with Lambda architecture
• Thinking Kappa uses batch processing
• Assuming multiple processing passes

2.

Which of the following is a correct component used in a typical Kappa architecture setup?

A) Separate batch and speed layers

B) Hadoop MapReduce for batch jobs

C) HDFS for storing batch data

D) Kafka for streaming data

easy

A. Separate batch and speed layers

B. Hadoop MapReduce for batch jobs

C. HDFS for storing batch data

D. Kafka for streaming data

Solution

1. Step 1: Identify streaming components in Kappa

   Kappa architecture uses streaming tools like Kafka to handle real-time data.

2. Step 2: Exclude batch-specific components

   Batch tools like MapReduce or HDFS batch storage are not part of Kappa's streaming-only design.

3. Final Answer:

   Kafka for streaming data -> Option D

4. Quick Check:

   Kappa uses Kafka, not batch tools [OK]

Hint: Kappa = streaming tools like Kafka only [OK]

Common Mistakes:

• Choosing batch tools like MapReduce
• Selecting batch storage like HDFS
• Thinking Kappa has separate batch layers

3.

Consider this simplified streaming code snippet using Kafka and Spark in a Kappa architecture:

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('KappaExample').getOrCreate()

# Read stream from Kafka topic
stream_df = spark.readStream.format('kafka') \
  .option('kafka.bootstrap.servers', 'localhost:9092') \
  .option('subscribe', 'events') \
  .load()

# Select and cast value to string
value_df = stream_df.selectExpr('CAST(value AS STRING)')

# Write stream to console
query = value_df.writeStream.outputMode('append').format('console').start()

query.awaitTermination()

What will this code do?

medium

A. Continuously print new Kafka messages from 'events' topic to console

B. Read all past Kafka messages once and stop

C. Write data back to Kafka topic 'events'

D. Throw an error due to missing schema

Solution

1. Step 1: Understand the streaming read

   The code reads a continuous stream from Kafka topic 'events' using Spark Structured Streaming.

2. Step 2: Analyze output mode and sink

   Output mode 'append' with console sink means new messages print continuously to console.

3. Final Answer:

   Continuously print new Kafka messages from 'events' topic to console -> Option A

4. Quick Check:

   Streaming read + console output = continuous print [OK]

Hint: Streaming read + console output = live print [OK]

Common Mistakes:

• Thinking it reads all past messages once
• Assuming it writes back to Kafka
• Expecting schema error without explicit schema

4.

Identify the error in this Kappa architecture streaming code snippet using Kafka and Spark:

stream_df = spark.readStream.format('kafka') \
  .option('kafka.bootstrap.servers', 'localhost:9092') \
  .option('subscribe', 'events') \
  .load()

value_df = stream_df.selectExpr('CAST(value AS STRING)')

query = value_df.writeStream.outputMode('complete').format('console').start()

query.awaitTermination()

What is the problem?

medium

A. Output mode 'complete' is invalid for streaming without aggregation

B. Missing schema definition causes failure

C. Kafka server address is incorrect

D. No error, code runs fine

Solution

1. Step 1: Check output mode usage

   Output mode 'complete' requires aggregation queries to output full results.

2. Step 2: Analyze query type

   This query only selects and casts values without aggregation, so 'complete' mode is invalid.

3. Final Answer:

   Output mode 'complete' is invalid for streaming without aggregation -> Option A

4. Quick Check:

   'complete' mode needs aggregation [OK]

Hint: 'complete' mode only with aggregation queries [OK]

Common Mistakes:

• Assuming missing schema causes error
• Thinking Kafka address is wrong
• Believing code runs without error

5.

You want to build a real-time dashboard using Kappa architecture that shows the count of events per user from a Kafka stream. Which approach fits best?

A) Use two separate systems: one for batch counts and one for streaming counts, then merge results.

B) Store Kafka data in HDFS, run daily batch jobs to count events per user, then update dashboard.

C) Use Spark Structured Streaming to read Kafka, group by user, count events, and update dashboard continuously.

D) Read Kafka stream once and save raw data only, no processing.

hard

A. Use two separate systems: one for batch counts and one for streaming counts, then merge results.

B. Store Kafka data in HDFS, run daily batch jobs to count events per user, then update dashboard.

C. Use Spark Structured Streaming to read Kafka, group by user, count events, and update dashboard continuously.

D. Read Kafka stream once and save raw data only, no processing.

Solution

1. Step 1: Understand Kappa architecture goal

   Kappa architecture processes data as a stream only, avoiding batch layers.

2. Step 2: Choose streaming aggregation for real-time dashboard

   Using Spark Structured Streaming to group and count events per user fits Kappa's streaming-only design and real-time needs.

3. Step 3: Exclude batch or hybrid approaches

   Batch jobs or separate batch/stream systems contradict Kappa principles.

4. Final Answer:

   Use Spark Structured Streaming to read Kafka, group by user, count events, and update dashboard continuously. -> Option C

5. Quick Check:

   Real-time + streaming aggregation = Use Spark Structured Streaming to read Kafka, group by user, count events, and update dashboard continuously. [OK]

Hint: Real-time dashboard needs streaming aggregation [OK]

Common Mistakes:

• Choosing batch processing for real-time needs
• Using hybrid batch and streaming systems
• Not processing data, just storing raw stream