Output modes (append, complete, update) in Apache Spark - Cheat Sheet & Quick Revision

Recall & Review

beginner

What does the append output mode do in Apache Spark Structured Streaming?

The append mode adds only new rows to the output sink. It does not update or remove existing rows. This is useful when you want to keep adding new data without changing old data.

beginner

Explain the complete output mode in Apache Spark Structured Streaming.

The complete mode writes the entire updated result table to the output sink every time there is new data. It replaces the previous output with the full current state.

intermediate

What is the purpose of the update output mode in Apache Spark Structured Streaming?

The update mode outputs only the rows that have changed since the last trigger. It updates existing rows and adds new rows but does not output the entire table.

beginner

Which output mode should you use if you want to keep a running total and overwrite the entire output each time?

You should use the complete mode because it writes the full updated result every time, replacing the previous output.

intermediate

Can the append mode be used with aggregations that update previous results?

No, append mode only adds new rows and does not support updating previous aggregation results. For aggregations, update or complete modes are needed.

Which output mode in Apache Spark Structured Streaming outputs only new rows without modifying existing data?

A overwrite

B complete

C append

D update

The append mode adds only new rows to the output sink without changing existing data.

In which output mode does Spark write the entire updated result table to the sink every time?

A complete

B append

C update

D incremental

The complete mode writes the full updated result table each time there is new data.

Which output mode outputs only rows that have changed since the last trigger?

A append

B update

C complete

D replace

The update mode outputs only changed rows, updating existing ones and adding new ones.

If you want to keep a running aggregation and update the output incrementally, which mode should you choose?

A update

B complete

C append

D delete

Update mode is suitable for incremental updates of aggregations.

Can append mode be used with aggregations that update previous results?

A Only if output is to memory sink

B Yes, always

C Only with windowed aggregations

D No, append mode does not support updating aggregations

Append mode cannot update previous aggregation results; update or complete modes are needed.

Describe the differences between append, complete, and update output modes in Apache Spark Structured Streaming.

Think about how each mode handles new and existing data.

When would you choose the complete output mode over append or update modes?

Consider scenarios where partial updates are not enough.