Structured Streaming basics in Apache Spark - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is Structured Streaming in Apache Spark?

Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine. It allows you to process live data streams using the same APIs as batch data.

beginner

How does Structured Streaming treat streaming data internally?

Structured Streaming treats streaming data as an unbounded table that is continuously appended. Queries on this table produce incremental results as new data arrives.

intermediate

What is a 'trigger' in Structured Streaming?

A trigger defines how often the streaming query processes new data. For example, a trigger can run the query every 1 second or process data as soon as it arrives.

intermediate

Name two common output modes in Structured Streaming.

1. Append mode: Only new rows are added to the output.
2. Complete mode: The entire result table is output every time.
There is also Update mode which outputs only rows that changed.

intermediate

What is checkpointing in Structured Streaming and why is it important?

Checkpointing saves the state and progress of a streaming query to reliable storage. It helps recover from failures without losing data or processing progress.

What does Structured Streaming treat streaming data as?

A An unbounded table

B A static file

C A batch job

D A database

Structured Streaming treats streaming data as an unbounded table that keeps growing as new data arrives.

Which output mode outputs only new rows added since the last trigger?

A Overwrite mode

B Complete mode

C Update mode

D Append mode

Append mode outputs only the new rows added since the last trigger.

What is the purpose of checkpointing in Structured Streaming?

A To speed up queries

B To format output

C To save query progress for recovery

D To delete old data

Checkpointing saves the state and progress so the query can recover after failures.

Which API does Structured Streaming use to process data?

A Spark SQL DataFrame API

B RDD API

C MapReduce API

D Hadoop API

Structured Streaming uses the Spark SQL DataFrame API for processing streaming data.

What does a trigger control in Structured Streaming?

A How data is stored

B How often the query runs

C The output format

D The data source

A trigger controls how often the streaming query processes new data.

Explain how Structured Streaming processes live data using the concept of an unbounded table.

Think about how new data keeps adding rows to a table that never ends.

Describe the role of checkpointing and triggers in ensuring reliable and timely streaming data processing.

Checkpointing helps recover from crashes; triggers decide when to run the query.

Practice

1. What is the main purpose of Structured Streaming in Apache Spark?

easy

A. To visualize data using charts and graphs

B. To batch process large static datasets

C. To store data permanently in a database

D. To process live data continuously as it arrives

Solution

1. Step 1: Understand Structured Streaming concept

   Structured Streaming is designed to handle data streams, meaning data that arrives continuously over time.

2. Step 2: Compare with other options

   Batch processing handles static data, storing data is not the main function, and visualization is separate from streaming.

3. Final Answer:

   To process live data continuously as it arrives -> Option D

4. Quick Check:

   Structured Streaming = continuous live data processing [OK]

Hint: Streaming means live, continuous data processing [OK]

Common Mistakes:

• Confusing streaming with batch processing
• Thinking streaming stores data permanently
• Assuming streaming is for visualization

2. Which of the following is the correct way to start reading a streaming DataFrame in Spark?

easy

A. spark.read.format("json").load("path")

B. spark.readStream.format("json").load("path")

C. spark.writeStream.format("json").load("path")

D. spark.write.format("json").load("path")

Solution

1. Step 1: Identify streaming read syntax

   To read streaming data, use readStream with format and load.

2. Step 2: Eliminate non-streaming options

   read is for batch, writeStream is for output, so only readStream with load is correct.

3. Final Answer:

   spark.readStream.format("json").load("path") -> Option B

4. Quick Check:

   Streaming read uses readStream [OK]

Hint: Use readStream to read streaming data [OK]

Common Mistakes:

• Using read instead of readStream for streaming data
• Confusing writeStream with readStream
• Using writeStream with load instead of start

3. Given the code below, what will be the output schema of the streaming DataFrame?

df = spark.readStream.format("csv").option("header", "true").load("/data/clicks")
df.printSchema()

medium

A. Schema with columns inferred as integers only

B. Schema with no columns because streaming can't infer schema

C. Schema with columns inferred from CSV header as strings

D. Error because schema must be specified explicitly

Solution

1. Step 1: Understand schema inference in streaming CSV

   When reading CSV with header option true, Spark infers schema as strings by default for streaming.

2. Step 2: Check if explicit schema is required

   Explicit schema is recommended but not mandatory; Spark infers schema as string types if not provided.

3. Final Answer:

   Schema with columns inferred from CSV header as strings -> Option C

4. Quick Check:

   CSV header + streaming = inferred string schema [OK]

Hint: CSV header enables schema inference as strings [OK]

Common Mistakes:

• Assuming streaming can't infer schema
• Expecting integer types without schema
• Thinking explicit schema is always required

4. Identify the error in the following Structured Streaming code:

streamDF = spark.readStream.format("json").load("/data/events")
query = streamDF.writeStream.format("console").start()
query.awaitTermination(10)

medium

A. Missing option to specify output mode in writeStream

B. awaitTermination does not take any arguments

C. Incorrect format 'console' should be 'Console'

D. start() must be called after awaitTermination()

Solution

1. Step 1: Check writeStream requirements

   In Structured Streaming, writeStream requires outputMode() (e.g., "append") to be specified before format() for sinks like console.

2. Step 2: Verify other parts

   awaitTermination(10) accepts a timeout argument, format 'console' is correct lowercase, and start() must be called before awaitTermination().

3. Final Answer:

   Missing option to specify output mode in writeStream -> Option A

4. Quick Check:

   writeStream requires outputMode [OK]

Hint: writeStream requires outputMode [OK]

Common Mistakes:

• Missing outputMode in writeStream
• Thinking awaitTermination does not take timeout
• Misusing start and awaitTermination order

5. You want to count the number of words in a live stream of text lines from a socket source. Which sequence of operations correctly applies Structured Streaming to achieve this?

hard

A. Read stream from socket, split lines into words, group by word, count, write stream to console

B. Read batch from socket, split lines into words, group by word, count, write batch to file

C. Read stream from socket, count lines directly, write stream to console

D. Read stream from socket, split lines into words, filter empty words, write batch to console

Solution

1. Step 1: Understand streaming word count steps

   Streaming word count requires reading live data, splitting lines into words, grouping by word, and counting occurrences continuously.

2. Step 2: Check options for correctness

   Read stream from socket, split lines into words, group by word, count, write stream to console follows the correct streaming steps with writeStream to console. Read batch from socket, split lines into words, group by word, count, write batch to file uses batch, not streaming. Read stream from socket, count lines directly, write stream to console counts lines, not words. Read stream from socket, split lines into words, filter empty words, write batch to console writes batch, not stream, and misses counting.

3. Final Answer:

   Read stream from socket, split lines into words, group by word, count, write stream to console -> Option A

4. Quick Check:

   Streaming word count = readStream + split + groupBy + count + writeStream [OK]

Hint: For streaming word count, use readStream + groupBy + count + writeStream [OK]

Common Mistakes:

• Using batch read instead of streaming
• Counting lines instead of words
• Writing batch output instead of streaming