Reading CSV files with options in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this Spark code reading a CSV with header option?

Consider the following Apache Spark code snippet that reads a CSV file with a header row:

df = spark.read.option("header", "true").csv("data.csv")
df.show()

The CSV file data.csv contains:

name,age
Alice,30
Bob,25

What will be the output of df.show()?

Apache Spark

df = spark.read.option("header", "true").csv("data.csv")
df.show()

A

+-----+---+
| name|age|
+-----+---+
| name|age|
|Alice| 30|
|  Bob| 25|
+-----+---+

B

+-----+---+
| _c0 |_c1|
+-----+---+
| name|age|
|Alice| 30|
|  Bob| 25|
+-----+---+

C

+-----+---+
| name|age|
+-----+---+
|Alice| 30|
|  Bob| 25|
+-----+---+

D

+-----+---+
| _c0 |_c1|
+-----+---+
|Alice| 30|
|  Bob| 25|
+-----+---+

💡 Hint

When header option is set to true, Spark uses the first row as column names.

Setting header=true tells Spark to treat the first row as column names. So the columns are name and age. The data rows follow.

❓ Predict Output

intermediate

What happens if you read a CSV with wrong delimiter option?

Given a CSV file data.csv with content:

name;age
Alice;30
Bob;25

What will be the output of this Spark code?

df = spark.read.option("header", "true").option("delimiter", ",").csv("data.csv")
df.show()

Apache Spark

df = spark.read.option("header", "true").option("delimiter", ",").csv("data.csv")
df.show()

A AnalysisException: CSV parsing error due to wrong delimiter

B

+-----+---+
| _c0 |_c1|
+-----+---+
|name;age|null|
|Alice;30|null|
|Bob;25  |null|
+-----+---+

C

+-----+---+
| name|age|
+-----+---+
|Alice| 30|
|  Bob| 25|
+-----+---+

D

+----------------+
|name;age         |
+----------------+
|Alice;30        |
|Bob;25          |
+----------------+

💡 Hint

If the delimiter option does not match the actual delimiter in the file, Spark treats the whole line as one column.

The CSV file uses semicolon ; as delimiter but the code sets delimiter to comma ,. Spark reads each line as a single string column named 'name;age'. So the output shows one column with the whole line.

❓ data_output

advanced

How many rows are in the DataFrame after reading CSV with inferSchema option?

Consider a CSV file data.csv with content:

name,age
Alice,30
Bob,25
Charlie,35

What is the number of rows in the DataFrame after running:

df = spark.read.option("header", "true").option("inferSchema", "true").csv("data.csv")
row_count = df.count()

Apache Spark

df = spark.read.option("header", "true").option("inferSchema", "true").csv("data.csv")
row_count = df.count()

A 2

B 3

C 4

D 0

💡 Hint

The CSV has 3 data rows after the header.

The header row is used as column names and not counted as data. The file has 3 data rows, so count() returns 3.

🔧 Debug

advanced

What error does this Spark CSV read code raise?

What error will this code raise?

df = spark.read.option("header", True).csv("data.csv")
df.show()

Note: The header option value is a boolean True, not a string.

Apache Spark

df = spark.read.option("header", True).csv("data.csv")
df.show()

A No error, runs correctly showing data with header

B TypeError: option value must be a string

C AnalysisException: CSV file not found

D ValueError: invalid option value for header

💡 Hint

Spark's option method accepts boolean values for boolean options like header.

The option method accepts both strings and booleans for options like header. Passing a boolean True works correctly.

🚀 Application

expert

Which option correctly reads a CSV with multiline fields and tab delimiter?

You have a CSV file data.csv where fields can contain new lines and the delimiter is a tab character.

Which Spark code snippet correctly reads this CSV preserving multiline fields?

A df = spark.read.option("header", "true").option("delimiter", "\t").option("multiLine", "true").csv("data.csv")

B df = spark.read.option("header", true).option("delimiter", "\t").csv("data.csv")

C df = spark.read.option("header", "true").option("delimiter", "tab").option("multiLine", "true").csv("data.csv")

D df = spark.read.option("header", "true").option("delimiter", "\t").csv("data.csv")

💡 Hint

Remember to set multiLine option to true as a string and use the correct tab character for delimiter.

Option A correctly sets header and delimiter as tab character \t, and enables multiLine as string true. Option A misses multiLine option, so multiline fields won't be handled properly. Option A uses string "tab" which is invalid delimiter. Option A misses multiLine option.