Kappa architecture (streaming only) in Hadoop - Deep Dive

Overview - Kappa architecture (streaming only)

What is it?

Kappa architecture is a way to process data streams in real time. It focuses on handling data as a continuous flow instead of batches. This means data is processed once as it arrives, making systems simpler and faster. It is often used when you want to analyze or react to data immediately.

Why it matters

Before Kappa architecture, many systems used batch processing which delayed insights and actions. Without it, businesses would miss chances to respond quickly to events like fraud detection or live recommendations. Kappa architecture solves this by making data processing continuous and real-time, improving decision speed and accuracy.

Where it fits

Learners should first understand basic data processing concepts and batch processing architectures like Lambda. After Kappa, they can explore advanced streaming tools, real-time analytics, and event-driven systems.

Mental Model

Core Idea

Kappa architecture processes all data as a single continuous stream, avoiding separate batch layers to simplify and speed up real-time data handling.

Think of it like...

Imagine a river carrying logs downstream. Instead of collecting logs in batches and processing them later, you process each log as it floats by, one after another, without stopping the flow.

┌───────────────┐
│ Data Sources  │
└──────┬────────┘
       │ Stream of data
       ▼
┌───────────────┐
│ Streaming     │
│ Processing    │
│ Engine        │
└──────┬────────┘
       │ Processed results
       ▼
┌───────────────┐
│ Output /      │
│ Storage       │
└───────────────┘

Build-Up - 6 Steps

1

Foundation

Understanding Data Streams

Concept: Data streams are continuous flows of data generated over time.

Data streams come from sources like sensors, user clicks, or logs. Unlike static files, streams keep producing new data. Processing streams means handling data as it arrives, not waiting for all data to be collected.

Result

You can see data as a never-ending flow that needs immediate attention.

Understanding streams is key because Kappa architecture treats all data as a continuous flow, not batches.

2

Foundation

Basics of Stream Processing Engines

Concept: Stream processing engines handle data in motion, processing each event quickly.

Examples include Apache Kafka and Apache Flink. They read data streams, apply transformations or computations, and output results in real time. They differ from batch engines that wait for full datasets.

Result

You know how tools can process data instantly as it arrives.

Knowing stream engines helps you grasp how Kappa architecture runs its processing continuously.

3

Intermediate

Kappa vs Lambda Architecture

🤔Before reading on: do you think Kappa architecture uses both batch and streaming layers like Lambda, or just one? Commit to your answer.

Concept: Kappa architecture simplifies Lambda by using only a streaming layer.

Lambda architecture has two layers: batch for accuracy and streaming for speed. Kappa removes the batch layer, processing all data as a stream. This reduces complexity and maintenance.

Result

You understand Kappa is a simpler, streaming-only approach.

Knowing the difference clarifies why Kappa is preferred for simpler, real-time systems.

4

Intermediate

Reprocessing Data in Kappa Architecture

🤔Before reading on: do you think Kappa architecture can reprocess old data? How might it do that?

Concept: Kappa uses the same streaming pipeline to reprocess data by replaying stored streams.

Instead of a batch layer, Kappa stores raw data streams (like Kafka topics). To fix bugs or update logic, you replay these streams through the processing engine again. This keeps one code path for all processing.

Result

You see how Kappa handles corrections without batch jobs.

Understanding replaying streams avoids confusion about how Kappa manages data updates.

5

Advanced

Handling State in Streaming Systems

🤔Before reading on: do you think streaming systems can remember past events, or only process current data? Commit to your answer.

Concept: Streaming engines maintain state to track information across events.

State means remembering past data, like counts or user sessions. Kappa architecture uses stateful stream processing to enable complex analytics, like running totals or pattern detection, in real time.

Result

You understand how streaming can do more than simple event processing.

Knowing state handling is crucial for building powerful real-time applications with Kappa.

6

Expert

Scaling and Fault Tolerance in Kappa

🤔Before reading on: do you think Kappa architecture can handle failures without losing data? How might it do that?

Concept: Kappa architecture relies on distributed logs and checkpointing for fault tolerance and scaling.

Systems like Kafka store data durably and allow replay. Stream processors checkpoint their state periodically. If a failure occurs, processing restarts from the last checkpoint, ensuring no data loss and consistent results.

Result

You see how Kappa systems remain reliable and scalable in production.

Understanding these mechanisms explains why Kappa is trusted for critical real-time systems.

Under the Hood

Kappa architecture uses a distributed log system to store all incoming data as an immutable sequence. The stream processing engine reads from this log, processes events in order, and maintains state as needed. When reprocessing is required, the engine resets its state and replays the log from the beginning or a checkpoint. This design ensures a single source of truth and consistent processing.

Why designed this way?

Kappa was designed to simplify the complex Lambda architecture by removing the batch layer, which often caused code duplication and maintenance overhead. Using a single streaming pipeline reduces errors and speeds up development. The immutable log provides durability and replayability, which were challenges in earlier streaming systems.

┌───────────────┐
│ Data Sources  │
└──────┬────────┘
       │
       ▼
┌─────────────────────┐
│ Distributed Log      │
│ (Immutable, Replay)  │
└──────┬───────────────┘
       │
       ▼
┌─────────────────────┐
│ Stream Processing    │
│ Engine (Stateful)    │
│ Checkpointing        │
└──────┬───────────────┘
       │
       ▼
┌───────────────┐
│ Output /      │
│ Storage       │
└───────────────┘

Myth Busters - 3 Common Misconceptions

Quick: Does Kappa architecture completely eliminate the need for batch processing? Commit yes or no.

Common Belief: Kappa architecture means no batch processing is ever needed.

Reality: Kappa removes batch layers from the architecture but may still use batch jobs for certain offline analytics or heavy computations.

Why it matters: Believing batch is never needed can lead to trying to force all workloads into streaming, causing inefficiency or complexity.

Quick: Do you think Kappa architecture requires two separate codebases for batch and streaming? Commit yes or no.

Common Belief: Kappa architecture still needs separate code for batch and streaming processing.

Reality: Kappa uses a single streaming pipeline for all processing, avoiding code duplication.

Why it matters: Misunderstanding this leads to unnecessary complexity and maintenance effort.

Quick: Is it true that Kappa architecture cannot handle late or out-of-order data? Commit yes or no.

Common Belief: Kappa architecture struggles with late or out-of-order events because it processes data only once in order.

Reality: Modern streaming engines in Kappa handle late and out-of-order data using event-time processing and watermarks.

Why it matters: Ignoring this can cause incorrect assumptions about Kappa's capabilities and limit its use.

Expert Zone

1

Kappa architecture's reliance on immutable logs means storage costs can grow quickly; managing retention policies is critical.

2

Replaying streams for reprocessing can be resource-intensive; optimizing checkpoints and incremental updates is a subtle but important practice.

3

State management in streaming engines requires careful design to avoid inconsistent results during failures or scaling.

When NOT to use

Kappa architecture is less suitable when batch processing of large historical datasets with complex transformations is needed. In such cases, Lambda architecture or pure batch systems may be better. Also, if the streaming infrastructure is immature or data ordering is highly irregular, alternatives should be considered.

Production Patterns

In production, Kappa is often implemented with Apache Kafka as the log and Apache Flink or Kafka Streams for processing. Teams use compacted topics for state recovery and design pipelines to allow easy replay for bug fixes. Monitoring and alerting on lag and processing delays are standard practices.

Connections

Event Sourcing (Software Engineering)

Kappa architecture uses an immutable log similar to event sourcing's event log.

Understanding event sourcing helps grasp how Kappa treats data as a sequence of immutable events for reliable state reconstruction.

Real-Time Analytics

Kappa architecture is a foundation for real-time analytics systems.

Knowing Kappa clarifies how continuous data processing enables instant insights and decision-making.

Supply Chain Management

Both Kappa architecture and supply chains rely on continuous flow and tracking of items/events.

Seeing data as a flow like goods in a supply chain helps understand the importance of order, state, and replay in Kappa.

Common Pitfalls

#1 Trying to process data only once without storing it for replay.

Wrong approach:

StreamProcessor.process(event) # no log storage or replay capability

Correct approach:

DistributedLog.store(event)
StreamProcessor.processFromLog() # allows replay

Root cause:

Misunderstanding that Kappa requires an immutable log to enable reprocessing and fault tolerance.

#2 Using separate code for batch and streaming processing.

Wrong approach:

# Batch code
processBatch(data)

# Streaming code
processStream(event)

Correct approach:

def process(event):
# Unified processing logic for all data
...

Root cause:

Confusing Kappa with Lambda architecture, leading to duplicated effort and bugs.

#3 Ignoring late-arriving or out-of-order events in stream processing.

Wrong approach:

processEventsInArrivalOrderOnly(events)

Correct approach:

processEventsUsingEventTimeAndWatermarks(events)

Root cause:

Not leveraging streaming engine features that handle event-time semantics.

Key Takeaways

Kappa architecture simplifies data processing by using a single streaming pipeline for all data.

It relies on an immutable log to store data, enabling replay and fault tolerance.

This approach reduces complexity compared to architectures with separate batch and streaming layers.

Stateful stream processing allows Kappa to handle complex real-time analytics.

Understanding Kappa helps build scalable, reliable systems that react instantly to data.

Practice

1.

What is the main idea behind the Kappa architecture in data processing?

easy

A. Store data in batches and process later

B. Process data only once as a continuous stream

C. Use separate systems for batch and streaming data

D. Process data multiple times for accuracy

Solution

1. Step 1: Understand Kappa architecture concept

   Kappa architecture focuses on processing data as a stream only once, avoiding batch processing.

2. Step 2: Compare with other architectures

   Unlike Lambda architecture, Kappa does not separate batch and streaming layers.

3. Final Answer:

   Process data only once as a continuous stream -> Option B

4. Quick Check:

   Kappa = single stream processing [OK]

Hint: Kappa means one stream, one processing path [OK]

Common Mistakes:

• Confusing Kappa with Lambda architecture
• Thinking Kappa uses batch processing
• Assuming multiple processing passes

2.

Which of the following is a correct component used in a typical Kappa architecture setup?

A) Separate batch and speed layers

B) Hadoop MapReduce for batch jobs

C) HDFS for storing batch data

D) Kafka for streaming data

easy

A. Separate batch and speed layers

B. Hadoop MapReduce for batch jobs

C. HDFS for storing batch data

D. Kafka for streaming data

Solution

1. Step 1: Identify streaming components in Kappa

   Kappa architecture uses streaming tools like Kafka to handle real-time data.

2. Step 2: Exclude batch-specific components

   Batch tools like MapReduce or HDFS batch storage are not part of Kappa's streaming-only design.

3. Final Answer:

   Kafka for streaming data -> Option D

4. Quick Check:

   Kappa uses Kafka, not batch tools [OK]

Hint: Kappa = streaming tools like Kafka only [OK]

Common Mistakes:

• Choosing batch tools like MapReduce
• Selecting batch storage like HDFS
• Thinking Kappa has separate batch layers

3.

Consider this simplified streaming code snippet using Kafka and Spark in a Kappa architecture:

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('KappaExample').getOrCreate()

# Read stream from Kafka topic
stream_df = spark.readStream.format('kafka') \
  .option('kafka.bootstrap.servers', 'localhost:9092') \
  .option('subscribe', 'events') \
  .load()

# Select and cast value to string
value_df = stream_df.selectExpr('CAST(value AS STRING)')

# Write stream to console
query = value_df.writeStream.outputMode('append').format('console').start()

query.awaitTermination()

What will this code do?

medium

A. Continuously print new Kafka messages from 'events' topic to console

B. Read all past Kafka messages once and stop

C. Write data back to Kafka topic 'events'

D. Throw an error due to missing schema

Solution

1. Step 1: Understand the streaming read

   The code reads a continuous stream from Kafka topic 'events' using Spark Structured Streaming.

2. Step 2: Analyze output mode and sink

   Output mode 'append' with console sink means new messages print continuously to console.

3. Final Answer:

   Continuously print new Kafka messages from 'events' topic to console -> Option A

4. Quick Check:

   Streaming read + console output = continuous print [OK]

Hint: Streaming read + console output = live print [OK]

Common Mistakes:

• Thinking it reads all past messages once
• Assuming it writes back to Kafka
• Expecting schema error without explicit schema

4.

Identify the error in this Kappa architecture streaming code snippet using Kafka and Spark:

stream_df = spark.readStream.format('kafka') \
  .option('kafka.bootstrap.servers', 'localhost:9092') \
  .option('subscribe', 'events') \
  .load()

value_df = stream_df.selectExpr('CAST(value AS STRING)')

query = value_df.writeStream.outputMode('complete').format('console').start()

query.awaitTermination()

What is the problem?

medium

A. Output mode 'complete' is invalid for streaming without aggregation

B. Missing schema definition causes failure

C. Kafka server address is incorrect

D. No error, code runs fine

Solution

1. Step 1: Check output mode usage

   Output mode 'complete' requires aggregation queries to output full results.

2. Step 2: Analyze query type

   This query only selects and casts values without aggregation, so 'complete' mode is invalid.

3. Final Answer:

   Output mode 'complete' is invalid for streaming without aggregation -> Option A

4. Quick Check:

   'complete' mode needs aggregation [OK]

Hint: 'complete' mode only with aggregation queries [OK]

Common Mistakes:

• Assuming missing schema causes error
• Thinking Kafka address is wrong
• Believing code runs without error

5.

You want to build a real-time dashboard using Kappa architecture that shows the count of events per user from a Kafka stream. Which approach fits best?

A) Use two separate systems: one for batch counts and one for streaming counts, then merge results.

B) Store Kafka data in HDFS, run daily batch jobs to count events per user, then update dashboard.

C) Use Spark Structured Streaming to read Kafka, group by user, count events, and update dashboard continuously.

D) Read Kafka stream once and save raw data only, no processing.

hard

A. Use two separate systems: one for batch counts and one for streaming counts, then merge results.

B. Store Kafka data in HDFS, run daily batch jobs to count events per user, then update dashboard.

C. Use Spark Structured Streaming to read Kafka, group by user, count events, and update dashboard continuously.

D. Read Kafka stream once and save raw data only, no processing.

Solution

1. Step 1: Understand Kappa architecture goal

   Kappa architecture processes data as a stream only, avoiding batch layers.

2. Step 2: Choose streaming aggregation for real-time dashboard

   Using Spark Structured Streaming to group and count events per user fits Kappa's streaming-only design and real-time needs.

3. Step 3: Exclude batch or hybrid approaches

   Batch jobs or separate batch/stream systems contradict Kappa principles.

4. Final Answer:

   Use Spark Structured Streaming to read Kafka, group by user, count events, and update dashboard continuously. -> Option C

5. Quick Check:

   Real-time + streaming aggregation = Use Spark Structured Streaming to read Kafka, group by user, count events, and update dashboard continuously. [OK]

Hint: Real-time dashboard needs streaming aggregation [OK]

Common Mistakes:

• Choosing batch processing for real-time needs
• Using hybrid batch and streaming systems
• Not processing data, just storing raw stream