Structured Streaming basics in Apache Spark - Step-by-Step Execution

Concept Flow - Structured Streaming basics

Start Streaming Query

↓

Read Input Stream

↓

Apply Transformations

↓

Write Output Stream

↓

Continuous Processing

↓

Stop Streaming Query

Structured Streaming reads data continuously, applies transformations, writes output, and keeps running until stopped.

Execution Sample

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('stream').getOrCreate()
input_stream = spark.readStream.format('socket').option('host', 'localhost').option('port', 9999).load()
words = input_stream.selectExpr('explode(split(value, " ")) as word')
query = words.writeStream.format('console').start()
query.awaitTermination()

This code reads text data from a socket, splits lines into words, and prints words to the console continuously.

Execution Table

Step	Action	Input Data	Transformation	Output Data	Status
1	Start Streaming Query	No data yet	Initialize streaming context	No output	Running
2	Read Input Stream	Line: 'hello world'	Read line from socket	DataFrame with 1 row: 'hello world'	Running
3	Apply Transformation	DataFrame with 'hello world'	Split line into words	DataFrame with rows: 'hello', 'world'	Running
4	Write Output Stream	DataFrame with words	Print to console	Console shows 'hello' and 'world'	Running
5	Read Input Stream	Line: 'spark streaming'	Read line from socket	DataFrame with 1 row: 'spark streaming'	Running
6	Apply Transformation	DataFrame with 'spark streaming'	Split line into words	DataFrame with rows: 'spark', 'streaming'	Running
7	Write Output Stream	DataFrame with words	Print to console	Console shows 'spark' and 'streaming'	Running
8	Stop Streaming Query	N/A	Stop query	No output	Stopped

💡 Streaming query stopped by user or program

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 5	After Step 6	After Step 7	Final
input_stream	empty	'hello world'	'hello world'	'hello world'	'spark streaming'	'spark streaming'	'spark streaming'	final state
words	empty	empty	['hello', 'world']	['hello', 'world']	['hello', 'world']	['spark', 'streaming']	['spark', 'streaming']	final state
query.status	not started	running	running	running	running	running	running	stopped

Key Moments - 3 Insights

Why does the streaming query keep running after processing some data?

Because Structured Streaming is designed to run continuously, it waits for new data until explicitly stopped, as shown in steps 1 to 7 in the execution_table.

What happens if no new data arrives in the input stream?

The streaming query stays active but does not output new data until new input arrives, as the status remains 'Running' without new output rows.

Why do we use 'awaitTermination()' at the end?

'awaitTermination()' keeps the program running to allow continuous processing, preventing the program from exiting immediately after starting the stream.

Visual Quiz - 3 Questions

Look at the execution_table at Step 3, what transformation is applied to the input data?

A Writing data to console

B Reading data from socket

C Splitting the line into words

D Stopping the query

💡 Hint

Check the 'Transformation' column at Step 3 in the execution_table.

At which step does the streaming query stop running?

A Step 4

B Step 8

C Step 7

D Step 2

💡 Hint

Look at the 'Status' column in the execution_table for the step where status changes to 'Stopped'.

If the input stream receives no new data after Step 7, what will be the query status?

A Running but no new output

B Stopped

C Error

D Idle and closed

💡 Hint

Refer to the key moment about query behavior when no new data arrives.

Concept Snapshot

Structured Streaming reads data continuously from a source.
It applies transformations like splitting or filtering.
Results are written to a sink like console or file.
The query runs continuously until stopped.
Use 'awaitTermination()' to keep the program running.
Ideal for real-time data processing.

Full Transcript

Structured Streaming in Apache Spark allows you to process data continuously as it arrives. You start a streaming query that reads data from a source, applies transformations such as splitting lines into words, and writes the results to an output sink like the console. The streaming query keeps running, waiting for new data, until you stop it manually. The method 'awaitTermination()' is used to keep the program running so the streaming can continue. This approach is useful for real-time data processing where data flows in continuously.