Structured Streaming basics in Apache Spark - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to read a streaming DataFrame from a socket source.

Apache Spark

streamingDF = spark.readStream.format([1]).option("host", "localhost").option("port", 9999).load()

Drag options to blanks, or click blank then click option'

A"csv"

B"parquet"

C"json"

D"socket"

Attempts:

3 left

💡 Hint

Common Mistakes

Using file formats like csv or json instead of socket for streaming from a socket.

Forgetting to specify the format as a string.

The format "socket" is used to read streaming data from a socket source.

2fill in blank

medium

Complete the code to start the streaming query that writes output to the console.

Apache Spark

query = streamingDF.writeStream.format([1]).start()

Drag options to blanks, or click blank then click option'

A"console"

B"memory"

C"parquet"

D"csv"

Attempts:

3 left

💡 Hint

Common Mistakes

Using file formats like parquet or csv as output format without specifying a path.

Confusing memory sink with console sink.

The "console" format writes streaming output to the console for easy viewing.

3fill in blank

hard

Fix the error in the code to specify the output mode for the streaming query.

Apache Spark

query = streamingDF.writeStream.outputMode([1]).format("console").start()

Drag options to blanks, or click blank then click option'

A"complete"

B"overwrite"

C"append"

D"update"

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'overwrite' which is not a valid output mode for streaming.

Confusing 'complete' mode which outputs the entire result table.

The "append" output mode outputs only new rows added to the result table since the last trigger.

4fill in blank

hard

Fill both blanks to create a streaming aggregation that counts words from a streaming DataFrame.

Apache Spark

wordCounts = streamingDF.selectExpr("explode(split(value, ' ')) as word").groupBy([1]).count().writeStream.outputMode([2]).format("console").start()

Drag options to blanks, or click blank then click option'

A"word"

B"append"

C"complete"

D"value"

Attempts:

3 left

💡 Hint

Common Mistakes

Grouping by the wrong column like 'value' instead of 'word'.

Using 'append' mode which does not work with aggregations without watermark.

Grouping by "word" counts each word, and "complete" mode outputs the full aggregation result.

5fill in blank

hard

Fill all three blanks to define a streaming query that reads JSON files, selects a column, and writes to memory sink.

Apache Spark

streamingDF = spark.readStream.format([1]).load("/path/to/json")
query = streamingDF.select([2]).writeStream.format([3]).start()

Drag options to blanks, or click blank then click option'

A"json"

B"name"

C"memory"

D"csv"

Attempts:

3 left

💡 Hint

Common Mistakes

Using wrong format like 'csv' for JSON files.

Selecting a column name that does not exist.

Using console sink instead of memory sink when intending to query data.

Use "json" format to read JSON files, select the "name" column, and write to "memory" sink for querying.

Practice

1. What is the main purpose of Structured Streaming in Apache Spark?

easy

A. To visualize data using charts and graphs

B. To batch process large static datasets

C. To store data permanently in a database

D. To process live data continuously as it arrives

Solution

1. Step 1: Understand Structured Streaming concept

   Structured Streaming is designed to handle data streams, meaning data that arrives continuously over time.

2. Step 2: Compare with other options

   Batch processing handles static data, storing data is not the main function, and visualization is separate from streaming.

3. Final Answer:

   To process live data continuously as it arrives -> Option D

4. Quick Check:

   Structured Streaming = continuous live data processing [OK]

Hint: Streaming means live, continuous data processing [OK]

Common Mistakes:

• Confusing streaming with batch processing
• Thinking streaming stores data permanently
• Assuming streaming is for visualization

2. Which of the following is the correct way to start reading a streaming DataFrame in Spark?

easy

A. spark.read.format("json").load("path")

B. spark.readStream.format("json").load("path")

C. spark.writeStream.format("json").load("path")

D. spark.write.format("json").load("path")

Solution

1. Step 1: Identify streaming read syntax

   To read streaming data, use readStream with format and load.

2. Step 2: Eliminate non-streaming options

   read is for batch, writeStream is for output, so only readStream with load is correct.

3. Final Answer:

   spark.readStream.format("json").load("path") -> Option B

4. Quick Check:

   Streaming read uses readStream [OK]

Hint: Use readStream to read streaming data [OK]

Common Mistakes:

• Using read instead of readStream for streaming data
• Confusing writeStream with readStream
• Using writeStream with load instead of start

3. Given the code below, what will be the output schema of the streaming DataFrame?

df = spark.readStream.format("csv").option("header", "true").load("/data/clicks")
df.printSchema()

medium

A. Schema with columns inferred as integers only

B. Schema with no columns because streaming can't infer schema

C. Schema with columns inferred from CSV header as strings

D. Error because schema must be specified explicitly

Solution

1. Step 1: Understand schema inference in streaming CSV

   When reading CSV with header option true, Spark infers schema as strings by default for streaming.

2. Step 2: Check if explicit schema is required

   Explicit schema is recommended but not mandatory; Spark infers schema as string types if not provided.

3. Final Answer:

   Schema with columns inferred from CSV header as strings -> Option C

4. Quick Check:

   CSV header + streaming = inferred string schema [OK]

Hint: CSV header enables schema inference as strings [OK]

Common Mistakes:

• Assuming streaming can't infer schema
• Expecting integer types without schema
• Thinking explicit schema is always required

4. Identify the error in the following Structured Streaming code:

streamDF = spark.readStream.format("json").load("/data/events")
query = streamDF.writeStream.format("console").start()
query.awaitTermination(10)

medium

A. Missing option to specify output mode in writeStream

B. awaitTermination does not take any arguments

C. Incorrect format 'console' should be 'Console'

D. start() must be called after awaitTermination()

Solution

1. Step 1: Check writeStream requirements

   In Structured Streaming, writeStream requires outputMode() (e.g., "append") to be specified before format() for sinks like console.

2. Step 2: Verify other parts

   awaitTermination(10) accepts a timeout argument, format 'console' is correct lowercase, and start() must be called before awaitTermination().

3. Final Answer:

   Missing option to specify output mode in writeStream -> Option A

4. Quick Check:

   writeStream requires outputMode [OK]

Hint: writeStream requires outputMode [OK]

Common Mistakes:

• Missing outputMode in writeStream
• Thinking awaitTermination does not take timeout
• Misusing start and awaitTermination order

5. You want to count the number of words in a live stream of text lines from a socket source. Which sequence of operations correctly applies Structured Streaming to achieve this?

hard

A. Read stream from socket, split lines into words, group by word, count, write stream to console

B. Read batch from socket, split lines into words, group by word, count, write batch to file

C. Read stream from socket, count lines directly, write stream to console

D. Read stream from socket, split lines into words, filter empty words, write batch to console

Solution

1. Step 1: Understand streaming word count steps

   Streaming word count requires reading live data, splitting lines into words, grouping by word, and counting occurrences continuously.

2. Step 2: Check options for correctness

   Read stream from socket, split lines into words, group by word, count, write stream to console follows the correct streaming steps with writeStream to console. Read batch from socket, split lines into words, group by word, count, write batch to file uses batch, not streaming. Read stream from socket, count lines directly, write stream to console counts lines, not words. Read stream from socket, split lines into words, filter empty words, write batch to console writes batch, not stream, and misses counting.

3. Final Answer:

   Read stream from socket, split lines into words, group by word, count, write stream to console -> Option A

4. Quick Check:

   Streaming word count = readStream + split + groupBy + count + writeStream [OK]

Hint: For streaming word count, use readStream + groupBy + count + writeStream [OK]

Common Mistakes:

• Using batch read instead of streaming
• Counting lines instead of words
• Writing batch output instead of streaming