Kappa architecture (streaming only) in Hadoop - Deep Dive

Overview - Kappa architecture (streaming only)

What is it?

Kappa architecture is a way to process data streams in real time. It focuses on handling data as a continuous flow instead of batches. This means data is processed once as it arrives, making systems simpler and faster. It is often used when you want to analyze or react to data immediately.

Why it matters

Before Kappa architecture, many systems used batch processing which delayed insights and actions. Without it, businesses would miss chances to respond quickly to events like fraud detection or live recommendations. Kappa architecture solves this by making data processing continuous and real-time, improving decision speed and accuracy.

Where it fits

Learners should first understand basic data processing concepts and batch processing architectures like Lambda. After Kappa, they can explore advanced streaming tools, real-time analytics, and event-driven systems.

Mental Model

Core Idea

Kappa architecture processes all data as a single continuous stream, avoiding separate batch layers to simplify and speed up real-time data handling.

Think of it like...

Imagine a river carrying logs downstream. Instead of collecting logs in batches and processing them later, you process each log as it floats by, one after another, without stopping the flow.

┌───────────────┐
│ Data Sources  │
└──────┬────────┘
       │ Stream of data
       ▼
┌───────────────┐
│ Streaming     │
│ Processing    │
│ Engine        │
└──────┬────────┘
       │ Processed results
       ▼
┌───────────────┐
│ Output /      │
│ Storage       │
└───────────────┘

Build-Up - 6 Steps

1

Foundation

Understanding Data Streams

Concept: Data streams are continuous flows of data generated over time.

Data streams come from sources like sensors, user clicks, or logs. Unlike static files, streams keep producing new data. Processing streams means handling data as it arrives, not waiting for all data to be collected.

Result

You can see data as a never-ending flow that needs immediate attention.

Understanding streams is key because Kappa architecture treats all data as a continuous flow, not batches.

2

Foundation

Basics of Stream Processing Engines

Concept: Stream processing engines handle data in motion, processing each event quickly.

Examples include Apache Kafka and Apache Flink. They read data streams, apply transformations or computations, and output results in real time. They differ from batch engines that wait for full datasets.

Result

You know how tools can process data instantly as it arrives.

Knowing stream engines helps you grasp how Kappa architecture runs its processing continuously.

3

Intermediate

Kappa vs Lambda Architecture

🤔Before reading on: do you think Kappa architecture uses both batch and streaming layers like Lambda, or just one? Commit to your answer.

Concept: Kappa architecture simplifies Lambda by using only a streaming layer.

Lambda architecture has two layers: batch for accuracy and streaming for speed. Kappa removes the batch layer, processing all data as a stream. This reduces complexity and maintenance.

Result

You understand Kappa is a simpler, streaming-only approach.

Knowing the difference clarifies why Kappa is preferred for simpler, real-time systems.

4

Intermediate

Reprocessing Data in Kappa Architecture

🤔Before reading on: do you think Kappa architecture can reprocess old data? How might it do that?

Concept: Kappa uses the same streaming pipeline to reprocess data by replaying stored streams.

Instead of a batch layer, Kappa stores raw data streams (like Kafka topics). To fix bugs or update logic, you replay these streams through the processing engine again. This keeps one code path for all processing.

Result

You see how Kappa handles corrections without batch jobs.

Understanding replaying streams avoids confusion about how Kappa manages data updates.

5

Advanced

Handling State in Streaming Systems

🤔Before reading on: do you think streaming systems can remember past events, or only process current data? Commit to your answer.

Concept: Streaming engines maintain state to track information across events.

State means remembering past data, like counts or user sessions. Kappa architecture uses stateful stream processing to enable complex analytics, like running totals or pattern detection, in real time.

Result

You understand how streaming can do more than simple event processing.

Knowing state handling is crucial for building powerful real-time applications with Kappa.

6

Expert

Scaling and Fault Tolerance in Kappa

🤔Before reading on: do you think Kappa architecture can handle failures without losing data? How might it do that?

Concept: Kappa architecture relies on distributed logs and checkpointing for fault tolerance and scaling.

Systems like Kafka store data durably and allow replay. Stream processors checkpoint their state periodically. If a failure occurs, processing restarts from the last checkpoint, ensuring no data loss and consistent results.

Result

You see how Kappa systems remain reliable and scalable in production.

Understanding these mechanisms explains why Kappa is trusted for critical real-time systems.

Under the Hood

Kappa architecture uses a distributed log system to store all incoming data as an immutable sequence. The stream processing engine reads from this log, processes events in order, and maintains state as needed. When reprocessing is required, the engine resets its state and replays the log from the beginning or a checkpoint. This design ensures a single source of truth and consistent processing.

Why designed this way?

Kappa was designed to simplify the complex Lambda architecture by removing the batch layer, which often caused code duplication and maintenance overhead. Using a single streaming pipeline reduces errors and speeds up development. The immutable log provides durability and replayability, which were challenges in earlier streaming systems.

┌───────────────┐
│ Data Sources  │
└──────┬────────┘
       │
       ▼
┌─────────────────────┐
│ Distributed Log      │
│ (Immutable, Replay)  │
└──────┬───────────────┘
       │
       ▼
┌─────────────────────┐
│ Stream Processing    │
│ Engine (Stateful)    │
│ Checkpointing        │
└──────┬───────────────┘
       │
       ▼
┌───────────────┐
│ Output /      │
│ Storage       │
└───────────────┘

Myth Busters - 3 Common Misconceptions

Quick: Does Kappa architecture completely eliminate the need for batch processing? Commit yes or no.

Common Belief: Kappa architecture means no batch processing is ever needed.

Reality: Kappa removes batch layers from the architecture but may still use batch jobs for certain offline analytics or heavy computations.

Why it matters: Believing batch is never needed can lead to trying to force all workloads into streaming, causing inefficiency or complexity.

Quick: Do you think Kappa architecture requires two separate codebases for batch and streaming? Commit yes or no.

Common Belief: Kappa architecture still needs separate code for batch and streaming processing.

Reality: Kappa uses a single streaming pipeline for all processing, avoiding code duplication.

Why it matters: Misunderstanding this leads to unnecessary complexity and maintenance effort.

Quick: Is it true that Kappa architecture cannot handle late or out-of-order data? Commit yes or no.

Common Belief: Kappa architecture struggles with late or out-of-order events because it processes data only once in order.

Reality: Modern streaming engines in Kappa handle late and out-of-order data using event-time processing and watermarks.

Why it matters: Ignoring this can cause incorrect assumptions about Kappa's capabilities and limit its use.

Expert Zone

1

Kappa architecture's reliance on immutable logs means storage costs can grow quickly; managing retention policies is critical.

2

Replaying streams for reprocessing can be resource-intensive; optimizing checkpoints and incremental updates is a subtle but important practice.

3

State management in streaming engines requires careful design to avoid inconsistent results during failures or scaling.

When NOT to use

Kappa architecture is less suitable when batch processing of large historical datasets with complex transformations is needed. In such cases, Lambda architecture or pure batch systems may be better. Also, if the streaming infrastructure is immature or data ordering is highly irregular, alternatives should be considered.

Production Patterns

In production, Kappa is often implemented with Apache Kafka as the log and Apache Flink or Kafka Streams for processing. Teams use compacted topics for state recovery and design pipelines to allow easy replay for bug fixes. Monitoring and alerting on lag and processing delays are standard practices.

Connections

Event Sourcing (Software Engineering)

Kappa architecture uses an immutable log similar to event sourcing's event log.

Understanding event sourcing helps grasp how Kappa treats data as a sequence of immutable events for reliable state reconstruction.

Real-Time Analytics

Kappa architecture is a foundation for real-time analytics systems.

Knowing Kappa clarifies how continuous data processing enables instant insights and decision-making.

Supply Chain Management

Both Kappa architecture and supply chains rely on continuous flow and tracking of items/events.

Seeing data as a flow like goods in a supply chain helps understand the importance of order, state, and replay in Kappa.

Common Pitfalls

#1 Trying to process data only once without storing it for replay.

Wrong approach:

StreamProcessor.process(event) # no log storage or replay capability

Correct approach:

DistributedLog.store(event)
StreamProcessor.processFromLog() # allows replay

Root cause:

Misunderstanding that Kappa requires an immutable log to enable reprocessing and fault tolerance.

#2 Using separate code for batch and streaming processing.

Wrong approach:

# Batch code
processBatch(data)

# Streaming code
processStream(event)

Correct approach:

def process(event):
# Unified processing logic for all data
...

Root cause:

Confusing Kappa with Lambda architecture, leading to duplicated effort and bugs.

#3 Ignoring late-arriving or out-of-order events in stream processing.

Wrong approach:

processEventsInArrivalOrderOnly(events)

Correct approach:

processEventsUsingEventTimeAndWatermarks(events)

Root cause:

Not leveraging streaming engine features that handle event-time semantics.

Key Takeaways

Kappa architecture simplifies data processing by using a single streaming pipeline for all data.

It relies on an immutable log to store data, enabling replay and fault tolerance.

This approach reduces complexity compared to architectures with separate batch and streaming layers.

Stateful stream processing allows Kappa to handle complex real-time analytics.

Understanding Kappa helps build scalable, reliable systems that react instantly to data.