Output modes (append, complete, update) in Apache Spark - Deep Dive

Overview - Output modes (append, complete, update)

What is it?

Output modes in Apache Spark define how the results of streaming computations are written to the output sink. There are three main modes: append, complete, and update. Each mode controls whether new rows, all rows, or only changed rows are written after each trigger. This helps manage how data is saved or displayed during continuous streaming.

Why it matters

Without output modes, streaming systems would struggle to efficiently and correctly update results as new data arrives. Output modes solve the problem of how to handle changing data in real time, ensuring that outputs reflect the latest state without unnecessary duplication or loss. This is crucial for real-time dashboards, alerts, and data pipelines that rely on accurate, timely information.

Where it fits

Learners should first understand basic Spark Structured Streaming concepts like streams, triggers, and sinks. After mastering output modes, they can explore advanced topics like stateful aggregations, watermarking, and fault tolerance in streaming.

Mental Model

Core Idea

Output modes control how streaming results are written by deciding whether to add only new data, rewrite all data, or update changed data after each batch.

Think of it like...

Imagine a live scoreboard at a sports game: append mode is like adding new scores as they happen, complete mode is like rewriting the entire scoreboard every time, and update mode is like changing only the scores that have changed since the last update.

┌───────────────┐
│ Streaming Job │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Output Modes  │
├───────────────┤
│ Append       │
│ Complete     │
│ Update       │
└─────┬─────────┘
      │
      ▼
┌───────────────┐
│ Output Sink   │
└───────────────┘

Build-Up - 8 Steps

1

Foundation

Understanding Streaming Outputs

Concept: Streaming outputs are the results generated continuously as data flows through Spark Structured Streaming.

In Spark Structured Streaming, data arrives in small chunks called micro-batches. After processing each batch, Spark writes the results to an output destination like a file, database, or console. How these results are written depends on the output mode chosen.

Result

Learners understand that streaming produces ongoing results that need to be saved or displayed.

Knowing that streaming outputs are continuous helps grasp why special modes are needed to handle changing data efficiently.

2

Foundation

What Are Output Modes?

Concept: Output modes define how Spark writes streaming results after each batch.

There are three output modes: - Append: Only new rows since the last batch are written. - Complete: All rows are rewritten every time. - Update: Only rows that changed since the last batch are written. Each mode suits different use cases depending on the type of query and sink.

Result

Learners can name and describe the three output modes.

Understanding output modes is key to controlling how streaming data is saved or displayed.

3

Intermediate

Append Mode in Detail

🤔Before reading on: do you think append mode can update existing rows or only add new ones? Commit to your answer.

Concept: Append mode writes only new rows that were added in the latest batch, never changing previous output.

In append mode, Spark outputs only the new data that arrived since the last trigger. This mode works well for queries that only add new rows, like simple selects or filters without aggregations. It cannot update or remove existing rows.

Result

Output contains only new rows appended after each batch.

Knowing append mode only adds new data prevents errors when using it with queries that require updates.

4

Intermediate

Complete Mode Explained

🤔Before reading on: do you think complete mode rewrites all data or just changes? Commit to your answer.

Concept: Complete mode rewrites the entire result table after every batch, showing the full current state.

Complete mode outputs the full result of the query every time, replacing previous output. This is useful for aggregation queries where the entire result changes over time, like counts or sums. It requires sinks that can handle full overwrites.

Result

Output shows the entire updated result after each batch.

Understanding complete mode helps handle queries where the full state must be visible at all times.

5

Intermediate

Update Mode Explained

🤔Before reading on: do you think update mode outputs all rows or only changed rows? Commit to your answer.

Concept: Update mode outputs only rows that have changed since the last batch, including new and updated rows.

Update mode writes only the rows that changed since the last trigger. It supports queries with aggregations and updates but does not rewrite the entire result. This mode requires sinks that can update existing rows, like databases.

Result

Output contains only changed rows, making updates efficient.

Knowing update mode balances efficiency and accuracy for stateful streaming queries.

6

Advanced

Choosing Output Modes by Query Type

🤔Before reading on: which output mode would you pick for a streaming count aggregation? Commit to your answer.

Concept: Different query types require specific output modes to work correctly and efficiently.

Simple append-only queries use append mode. Aggregations without updates use complete mode. Aggregations with updates use update mode. Choosing the wrong mode causes errors or incorrect results. For example, append mode cannot be used with aggregations that update existing rows.

Result

Learners can match query types to appropriate output modes.

Understanding this prevents common runtime errors and ensures correct streaming results.

7

Advanced

Output Modes and Sink Compatibility

🤔Before reading on: do all sinks support all output modes? Commit to your answer.

Concept: Not all output modes work with every sink; compatibility depends on sink capabilities.

File sinks support append and complete modes but not update mode because files cannot be updated easily. Console sink supports all modes for display. Database sinks often support update mode to modify existing rows. Choosing the right sink and mode combination is critical for production.

Result

Learners understand how sink choice limits output mode options.

Knowing sink compatibility avoids deployment failures and data inconsistencies.

8

Expert

Internal State Management and Output Modes

🤔Before reading on: does Spark keep all past data in memory for output modes? Commit to your answer.

Concept: Spark manages internal state differently depending on output mode to track changes efficiently.

In complete mode, Spark maintains the full aggregation state and rewrites all results. In update mode, it tracks only changed rows to output updates. Append mode requires minimal state since it only adds new rows. This internal state management affects memory use and performance. Understanding this helps optimize streaming jobs.

Result

Learners grasp how output modes impact resource use and latency.

Knowing internal state behavior guides tuning and troubleshooting of streaming applications.

Under the Hood

Spark Structured Streaming processes data in micro-batches. After each batch, it computes the query result and writes output based on the mode. Append mode outputs only new rows detected by comparing current batch with previous state. Complete mode outputs the entire aggregated result by maintaining full state. Update mode tracks changes in state and outputs only updated rows. Internally, Spark uses state stores and checkpoints to manage this state and ensure fault tolerance.

Why designed this way?

These modes were designed to balance correctness, efficiency, and sink capabilities. Append mode is simple and efficient for append-only data. Complete mode ensures correctness for full aggregations but can be costly. Update mode offers a middle ground for incremental updates. Alternatives like rewriting entire datasets every time would be inefficient, and no output modes would cause inconsistent or incorrect streaming results.

┌───────────────┐
│ Micro-batch   │
│ Processing   │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ State Store   │<─────────────┐
│ (Aggregations)│              │
└──────┬────────┘              │
       │                       │
       ▼                       │
┌───────────────┐              │
│ Output Mode   │──────────────┤
│ Logic         │              │
└──────┬────────┘              │
       │                       │
       ▼                       │
┌───────────────┐              │
│ Output Sink   │              │
└───────────────┘              │
                              │
       ┌──────────────────────┘
       ▼
┌───────────────┐
│ Checkpointing │
│ & Fault Toler.│
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does append mode allow updating existing rows? Commit yes or no.

Common Belief: Append mode can update existing rows in the output.

Reality: Append mode only adds new rows and never modifies or deletes existing output rows.

Why it matters: Using append mode with queries that update rows causes incorrect or incomplete results.

Quick: Does complete mode output only changed rows? Commit yes or no.

Common Belief: Complete mode outputs only the rows that changed since the last batch.

Reality: Complete mode rewrites the entire result table every time, not just changes.

Why it matters: Misunderstanding this can lead to performance issues due to unnecessary full rewrites.

Quick: Can all sinks support update mode? Commit yes or no.

Common Belief: All output sinks support update mode for streaming.

Reality: Many sinks, like file systems, do not support update mode because they cannot modify existing data easily.

Why it matters: Choosing incompatible sinks causes runtime errors or data loss.

Quick: Does Spark keep all past data in memory for append mode? Commit yes or no.

Common Belief: Spark stores all past data in memory regardless of output mode.

Reality: Append mode requires minimal state and does not keep all past data in memory.

Why it matters: Overestimating memory needs can lead to inefficient resource allocation.

Expert Zone

1

Update mode requires sinks that support idempotent writes or upserts to avoid data duplication or corruption.

2

Complete mode can cause high latency and resource use for large stateful aggregations, so it is often avoided in production.

3

Append mode is the only mode supported for streaming queries with event-time watermarks and late data handling without aggregation.

When NOT to use

Avoid append mode for queries with aggregations that update existing rows; use update or complete instead. Avoid complete mode for large stateful queries due to performance costs; consider update mode or incremental processing. If the sink does not support updates, do not use update mode; instead, use append or complete with compatible sinks.

Production Patterns

In production, append mode is common for simple event logging pipelines. Update mode is used with databases like Cassandra or Kafka sinks that support upserts. Complete mode is used for small or medium-sized aggregations powering dashboards where full state visibility is needed. Combining output modes with watermarking and state cleanup is a common pattern to manage resource use.

Connections

Event-time Watermarking

Builds-on

Understanding output modes helps grasp how watermarking controls late data handling and state eviction in streaming.

Database Upserts

Same pattern

Update mode in streaming is similar to database upserts, where only changed rows are updated, improving efficiency.

Real-time Dashboarding

Application domain

Choosing the right output mode directly impacts how real-time dashboards reflect live data accurately and efficiently.

Common Pitfalls

#1 Using append mode with aggregation queries that update existing rows.

Wrong approach:

streamingQuery.writeStream.outputMode("append").format("console").start()

Correct approach:

streamingQuery.writeStream.outputMode("update").format("console").start()

Root cause:

Misunderstanding that append mode cannot handle updates causes incorrect or failed streaming jobs.

#2 Selecting update mode with a file sink that does not support updates.

Wrong approach:

streamingQuery.writeStream.outputMode("update").format("parquet").start()

Correct approach:

streamingQuery.writeStream.outputMode("append").format("parquet").start()

Root cause:

Not knowing sink limitations leads to runtime errors or data corruption.

#3 Using complete mode for very large aggregations causing high latency.

Wrong approach:

streamingQuery.writeStream.outputMode("complete").format("console").start()

Correct approach:

streamingQuery.writeStream.outputMode("update").format("console").start()

Root cause:

Ignoring performance impact of rewriting full results causes slow streaming.

Key Takeaways

Output modes in Spark Structured Streaming control how results are written after each batch: append adds new rows, complete rewrites all rows, and update writes only changed rows.

Choosing the correct output mode depends on the query type and sink capabilities to ensure correct and efficient streaming.

Append mode is simple and efficient for new data only, complete mode suits full aggregations but can be costly, and update mode balances efficiency with incremental updates.

Understanding sink compatibility with output modes prevents runtime errors and data inconsistencies.

Knowing internal state management behind output modes helps optimize resource use and streaming performance.