Structured Streaming basics in Apache Spark - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is Structured Streaming in Apache Spark?

Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine. It allows you to process live data streams using the same APIs as batch data.

beginner

How does Structured Streaming treat streaming data internally?

Structured Streaming treats streaming data as an unbounded table that is continuously appended. Queries on this table produce incremental results as new data arrives.

intermediate

What is a 'trigger' in Structured Streaming?

A trigger defines how often the streaming query processes new data. For example, a trigger can run the query every 1 second or process data as soon as it arrives.

intermediate

Name two common output modes in Structured Streaming.

1. Append mode: Only new rows are added to the output.<br>2. Complete mode: The entire result table is output every time.<br>There is also Update mode which outputs only rows that changed.

intermediate

What is checkpointing in Structured Streaming and why is it important?

Checkpointing saves the state and progress of a streaming query to reliable storage. It helps recover from failures without losing data or processing progress.

What does Structured Streaming treat streaming data as?

A An unbounded table

B A static file

C A batch job

D A database

Structured Streaming treats streaming data as an unbounded table that keeps growing as new data arrives.

Which output mode outputs only new rows added since the last trigger?

A Overwrite mode

B Complete mode

C Update mode

D Append mode

Append mode outputs only the new rows added since the last trigger.

What is the purpose of checkpointing in Structured Streaming?

A To speed up queries

B To format output

C To save query progress for recovery

D To delete old data

Checkpointing saves the state and progress so the query can recover after failures.

Which API does Structured Streaming use to process data?

A Spark SQL DataFrame API

B RDD API

C MapReduce API

D Hadoop API

Structured Streaming uses the Spark SQL DataFrame API for processing streaming data.

What does a trigger control in Structured Streaming?

A How data is stored

B How often the query runs

C The output format

D The data source

A trigger controls how often the streaming query processes new data.

Explain how Structured Streaming processes live data using the concept of an unbounded table.

Think about how new data keeps adding rows to a table that never ends.

Describe the role of checkpointing and triggers in ensuring reliable and timely streaming data processing.

Checkpointing helps recover from crashes; triggers decide when to run the query.