Output modes (append, complete, update) in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output mode behavior with streaming aggregation

Given a streaming DataFrame that counts words in a stream, what will be the output after the first batch if the output mode is set to 'append'?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import split, explode

spark = SparkSession.builder.appName('Test').getOrCreate()

# Simulate streaming input
input_data = [("hello world hello",)]

# Create static DataFrame to simulate streaming
df = spark.createDataFrame(input_data, ['line'])

words = df.select(explode(split(df.line, ' ')).alias('word'))

word_counts = words.groupBy('word').count()

query = word_counts.writeStream.outputMode('append').format('console').start()

query.processAllAvailable()
query.stop()

A

[word=hello, count=2]
[word=world, count=1]

B No output, because 'append' mode does not support aggregation queries

C

[word=hello, count=1]
[word=world, count=1]

D

[word=hello, count=2]
[word=world, count=1]
[word=hello, count=3]

💡 Hint

Think about which output modes support aggregation in streaming queries.

The 'append' output mode only outputs new rows added to the result table. Aggregations require 'complete' or 'update' modes because the counts update existing rows. Therefore, 'append' mode with aggregation causes no output.

❓ data_output

intermediate

Result of 'complete' output mode on streaming aggregation

What is the output of a streaming word count query after processing the first batch when using 'complete' output mode?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import split, explode

spark = SparkSession.builder.appName('Test').getOrCreate()

input_data = [("apple banana apple",)]

df = spark.createDataFrame(input_data, ['line'])

words = df.select(explode(split(df.line, ' ')).alias('word'))

word_counts = words.groupBy('word').count()

query = word_counts.writeStream.outputMode('complete').format('console').start()

query.processAllAvailable()
query.stop()

A

[word=apple, count=2]
[word=banana, count=1]

B

[word=apple, count=1]
[word=banana, count=1]

C

[word=apple, count=2]
[word=banana, count=2]

D No output because 'complete' mode is not supported for aggregation

💡 Hint

Remember that 'complete' mode outputs the full aggregated result table after each batch.

'Complete' mode outputs the entire aggregation result after each batch. Since 'apple' appears twice and 'banana' once, the output shows counts 2 and 1 respectively.

🧠 Conceptual

advanced

Understanding 'update' output mode in streaming

Which statement correctly describes the 'update' output mode in Apache Spark Structured Streaming?

A 'Update' mode outputs the entire result table after every trigger, regardless of changes.

B 'Update' mode outputs only new rows appended to the result table, ignoring updates to existing rows.

C 'Update' mode outputs only the rows that have changed since the last trigger, including new and updated rows, but not the entire result table.

D 'Update' mode is not supported for streaming aggregations.

💡 Hint

Think about how 'update' differs from 'append' and 'complete' modes.

'Update' mode outputs only rows that have changed since the last trigger. It includes new rows and updates to existing rows but does not output the full table like 'complete' mode.

🔧 Debug

advanced

Error caused by using 'append' mode with aggregation

What error will this code raise when running a streaming aggregation query with output mode set to 'append'?

Apache Spark

from pyspark.sql import SparkSession
from pyspark.sql.functions import split, explode

spark = SparkSession.builder.appName('Test').getOrCreate()

input_data = [("cat dog cat",)]

df = spark.createDataFrame(input_data, ['line'])

words = df.select(explode(split(df.line, ' ')).alias('word'))

word_counts = words.groupBy('word').count()

query = word_counts.writeStream.outputMode('append').format('console').start()

query.processAllAvailable()
query.stop()

A org.apache.spark.sql.AnalysisException: Append output mode not supported with aggregations

B No error, outputs counts correctly

C SyntaxError: invalid syntax in outputMode parameter

D RuntimeError: Streaming source not found

💡 Hint

Consider which output modes support aggregation in streaming queries.

Using 'append' mode with aggregation causes an AnalysisException because 'append' mode does not support streaming aggregations.

🚀 Application

expert

Choosing output mode for a streaming deduplication job

You have a streaming job that removes duplicate records based on a unique key and outputs the latest unique records. Which output mode should you use to ensure only updated unique records are output after each batch?

A append

B complete

C overwrite

D update

💡 Hint

Think about which mode outputs only changed rows without the full table.

'Update' mode outputs only the rows that have changed since the last trigger, which is ideal for deduplication where only updated unique records should be output.

Practice

1. Which output mode in Apache Spark streaming adds only new rows to the sink without updating existing rows?

easy

A. Update

B. Complete

C. Append

D. Overwrite

Solution

1. Step 1: Understand output modes

   Append mode adds only new rows to the output sink without changing existing data.

2. Step 2: Compare with other modes

   Complete mode outputs the entire result table, and Update mode outputs only changed rows.

3. Final Answer:

   Append -> Option C

4. Quick Check:

   Append = adds new rows only [OK]

Hint: Append means add new rows only, no updates [OK]

Common Mistakes:

• Confusing append with complete mode
• Thinking update mode adds only new rows
• Assuming overwrite is an output mode

2. Which of the following is the correct syntax to set the output mode to 'complete' in a Spark Structured Streaming query?

easy

A. query.writeStream.outputMode("complete")

B. query.writeStream.mode("complete")

C. query.writeStream.setOutputMode("complete")

D. query.writeStream.output_mode("complete")

Solution

1. Step 1: Recall Spark streaming syntax

   The correct method to set output mode is outputMode("mode_name") on writeStream.

2. Step 2: Check options

   query.writeStream.outputMode("complete") uses outputMode("complete"), which is correct syntax. Others use incorrect method names or casing.

3. Final Answer:

   query.writeStream.outputMode("complete") -> Option A

4. Quick Check:

   outputMode("complete") is correct method [OK]

Hint: Use outputMode() method exactly as shown [OK]

Common Mistakes:

• Using mode() instead of outputMode()
• Using underscores instead of camelCase
• Trying setOutputMode() which doesn't exist

3. Given the following code snippet, what will be the output mode behavior?

val query = df.writeStream
  .format("console")
  .outputMode("update")
  .start()

What does the 'update' mode do here?

medium

A. Overwrites the entire sink with new data

B. Outputs the entire result table every time

C. Outputs only new rows added since last trigger

D. Outputs only rows that have changed since last trigger

Solution

1. Step 1: Understand 'update' mode meaning

   Update mode outputs only rows that have changed since the last trigger, not the full table.

2. Step 2: Compare with other modes

   Append outputs only new rows, complete outputs full table, overwrite is not a streaming mode.

3. Final Answer:

   Outputs only rows that have changed since last trigger -> Option D

4. Quick Check:

   Update = changed rows only [OK]

Hint: Update mode shows changed rows only, not full or just new [OK]

Common Mistakes:

• Confusing update with append mode
• Thinking update outputs full table
• Assuming overwrite is a streaming output mode

4. You wrote this Spark streaming code but get an error:

df.writeStream
  .format("console")
  .outputMode("append")
  .start()

The query fails because the DataFrame has aggregations. What is the error and how to fix it?

medium

A. Append mode not allowed with aggregations; use complete mode instead

B. Output mode 'append' is misspelled; fix spelling

C. Format 'console' does not support append mode; use 'memory' format

D. Start method missing awaitTermination(); add it

Solution

1. Step 1: Identify cause of error

   Append mode cannot be used with aggregations because new rows alone can't be identified.

2. Step 2: Fix by changing output mode

   Use complete mode to output full aggregation results instead of append.

3. Final Answer:

   Append mode not allowed with aggregations; use complete mode instead -> Option A

4. Quick Check:

   Aggregations require complete mode, not append [OK]

Hint: Aggregations need complete mode, not append [OK]

Common Mistakes:

• Trying to fix by changing format instead of mode
• Ignoring aggregation restrictions
• Assuming spelling or method call errors

5. You have a streaming query with a stateful aggregation that updates counts per key. You want to output only the changed counts to a database sink that supports updates. Which output mode should you choose and why?

hard

A. Append mode, because it adds only new counts

B. Update mode, because it outputs only changed counts for efficient updates

C. Complete mode, because it outputs all counts every time

D. Overwrite mode, because it replaces all data in the sink

Solution

1. Step 1: Understand the use case

   Stateful aggregation updates counts per key, so some counts change over time.

2. Step 2: Choose output mode for efficient updates

   Update mode outputs only changed rows, ideal for sinks supporting updates to avoid full rewrites.

3. Step 3: Exclude other modes

   Append mode can't handle updates, complete mode outputs full data (less efficient), overwrite replaces all data (not streaming mode).

4. Final Answer:

   Update mode, because it outputs only changed counts for efficient updates -> Option B

5. Quick Check:

   Update mode = changed rows only, best for stateful updates [OK]

Hint: Use update mode for stateful aggregation with update-capable sinks [OK]

Common Mistakes:

• Choosing append mode for updates
• Using complete mode causing full data writes
• Confusing overwrite with streaming output mode