Output modes (append, complete, update) in Apache Spark - Time & Space Complexity

Time Complexity: Output modes (append, complete, update)

O(n)

Understanding Time Complexity

When working with streaming data in Apache Spark, it's important to know how the output mode affects processing time.

We want to understand how the time to write results grows as the data size increases for different output modes.

Scenario Under Consideration

Analyze the time complexity of this streaming output code snippet.

streamingDF.writeStream
  .outputMode("append")
  .format("console")
  .start()

streamingDF.writeStream
  .outputMode("complete")
  .format("console")
  .start()

streamingDF.writeStream
  .outputMode("update")
  .format("console")
  .start()

This code writes streaming data to the console using three output modes: append, complete, and update.

Identify Repeating Operations

Look at what happens each time new data arrives in the stream.

• Primary operation: Writing output rows to the sink (console here).
• How many times: For each batch of data, the output mode decides how many rows are written.

How Execution Grows With Input

As the total data processed grows, the amount of output work changes by mode.

Input Size (n rows)	Append Mode	Complete Mode	Update Mode
10	Writes ~10 new rows	Writes all 10 rows	Writes changed rows only
100	Writes ~100 new rows	Writes all 100 rows	Writes changed rows only
1000	Writes ~1000 new rows	Writes all 1000 rows	Writes changed rows only

Pattern observation: Append mode writes only new rows, so work grows with new data size. Complete mode writes all rows every time, so work grows with total data size. Update mode writes only changed rows, which can vary but often less than total data.

Final Time Complexity

Time Complexity: O(n)

This means the time to write output grows linearly with the number of rows processed or updated, depending on the mode.

Common Mistake

[X] Wrong: "All output modes take the same time regardless of data size."

[OK] Correct: Different modes write different amounts of data each time, so their time grows differently with input size.

Interview Connect

Understanding how output modes affect processing time helps you explain trade-offs in streaming applications clearly and confidently.

Self-Check

"What if we changed the output sink from console to a database? How might that affect the time complexity for each output mode?"