Reading CSV files with options in Apache Spark - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this Spark code reading a CSV with header option?

Consider the following Apache Spark code snippet that reads a CSV file with a header row:

df = spark.read.option("header", "true").csv("data.csv")
df.show()

The CSV file data.csv contains:

name,age
Alice,30
Bob,25

What will be the output of df.show()?

Apache Spark

df = spark.read.option("header", "true").csv("data.csv")
df.show()

A

+-----+---+
| name|age|
+-----+---+
| name|age|
|Alice| 30|
|  Bob| 25|
+-----+---+

B

+-----+---+
| _c0 |_c1|
+-----+---+
| name|age|
|Alice| 30|
|  Bob| 25|
+-----+---+

C

+-----+---+
| name|age|
+-----+---+
|Alice| 30|
|  Bob| 25|
+-----+---+

D

+-----+---+
| _c0 |_c1|
+-----+---+
|Alice| 30|
|  Bob| 25|
+-----+---+

💡 Hint

When header option is set to true, Spark uses the first row as column names.

Setting header=true tells Spark to treat the first row as column names. So the columns are name and age. The data rows follow.

❓ Predict Output

intermediate

What happens if you read a CSV with wrong delimiter option?

Given a CSV file data.csv with content:

name;age
Alice;30
Bob;25

What will be the output of this Spark code?

df = spark.read.option("header", "true").option("delimiter", ",").csv("data.csv")
df.show()

Apache Spark

df = spark.read.option("header", "true").option("delimiter", ",").csv("data.csv")
df.show()

A AnalysisException: CSV parsing error due to wrong delimiter

B

+-----+---+
| _c0 |_c1|
+-----+---+
|name;age|null|
|Alice;30|null|
|Bob;25  |null|
+-----+---+

C

+-----+---+
| name|age|
+-----+---+
|Alice| 30|
|  Bob| 25|
+-----+---+

D

+----------------+
|name;age         |
+----------------+
|Alice;30        |
|Bob;25          |
+----------------+

💡 Hint

If the delimiter option does not match the actual delimiter in the file, Spark treats the whole line as one column.

The CSV file uses semicolon ; as delimiter but the code sets delimiter to comma ,. Spark reads each line as a single string column named 'name;age'. So the output shows one column with the whole line.

❓ data_output

advanced

How many rows are in the DataFrame after reading CSV with inferSchema option?

Consider a CSV file data.csv with content:

name,age
Alice,30
Bob,25
Charlie,35

What is the number of rows in the DataFrame after running:

df = spark.read.option("header", "true").option("inferSchema", "true").csv("data.csv")
row_count = df.count()

Apache Spark

df = spark.read.option("header", "true").option("inferSchema", "true").csv("data.csv")
row_count = df.count()

A 2

B 3

C 4

D 0

💡 Hint

The CSV has 3 data rows after the header.

The header row is used as column names and not counted as data. The file has 3 data rows, so count() returns 3.

🔧 Debug

advanced

What error does this Spark CSV read code raise?

What error will this code raise?

df = spark.read.option("header", True).csv("data.csv")
df.show()

Note: The header option value is a boolean True, not a string.

Apache Spark

df = spark.read.option("header", True).csv("data.csv")
df.show()

A No error, runs correctly showing data with header

B TypeError: option value must be a string

C AnalysisException: CSV file not found

D ValueError: invalid option value for header

💡 Hint

Spark's option method accepts boolean values for boolean options like header.

The option method accepts both strings and booleans for options like header. Passing a boolean True works correctly.

🚀 Application

expert

Which option correctly reads a CSV with multiline fields and tab delimiter?

You have a CSV file data.csv where fields can contain new lines and the delimiter is a tab character.

Which Spark code snippet correctly reads this CSV preserving multiline fields?

A df = spark.read.option("header", "true").option("delimiter", "\t").option("multiLine", "true").csv("data.csv")

B df = spark.read.option("header", true).option("delimiter", "\t").csv("data.csv")

C df = spark.read.option("header", "true").option("delimiter", "tab").option("multiLine", "true").csv("data.csv")

D df = spark.read.option("header", "true").option("delimiter", "\t").csv("data.csv")

💡 Hint

Remember to set multiLine option to true as a string and use the correct tab character for delimiter.

Option A correctly sets header and delimiter as tab character \t, and enables multiLine as string true. Option A misses multiLine option, so multiline fields won't be handled properly. Option A uses string "tab" which is invalid delimiter. Option A misses multiLine option.

Practice

1. What does the header option do when reading a CSV file in Apache Spark?

easy

A. It tells Spark to treat the first line as column names.

B. It specifies the character used to separate columns.

C. It infers the data types of each column automatically.

D. It sets the file reading mode to permissive.

Solution

1. Step 1: Understand the header option

   The header option tells Spark whether the first line of the CSV file contains column names or data.

2. Step 2: Identify its effect

   If header is set to true, Spark uses the first line as column names; otherwise, it treats all lines as data.

3. Final Answer:

   It tells Spark to treat the first line as column names. -> Option A

4. Quick Check:

   header = first line as columns [OK]

Hint: Remember: header=true means first line is column names [OK]

Common Mistakes:

• Confusing header with separator option
• Thinking header infers data types
• Assuming header controls error handling

2. Which of the following is the correct way to read a CSV file with a semicolon separator and header in Spark?

easy

A. spark.read.csv(options={'sep': ';', 'header': 'true'}, 'file.csv')

B. spark.read.csv(sep=';', 'file.csv', header=True)

C. spark.read.option('sep', ';').option('header', 'true').csv('file.csv')

D. spark.read.csv(header=True, sep=';', 'file.csv')

Solution

1. Step 1: Recall Spark CSV reading syntax

   Spark uses option() method to set options before calling csv().

2. Step 2: Check each option usage

   spark.read.option('sep', ';').option('header', 'true').csv('file.csv') correctly chains option('sep', ';') and option('header', 'true') before reading the file.

3. Final Answer:

   spark.read.option('sep', ';').option('header', 'true').csv('file.csv') -> Option C

4. Quick Check:

   Use option() chaining for CSV options [OK]

Hint: Use .option() chaining to set CSV read options [OK]

Common Mistakes:

• Passing options directly inside csv() as kwargs
• Using Python booleans instead of strings for options
• Incorrect method chaining order

3. Given the code below, what will be the schema of the DataFrame?

df = spark.read.option('header', 'true').option('inferSchema', 'true').csv('data.csv')

Assuming data.csv has columns: id (integer), name (string), age (integer).

medium

A. All columns will be read as strings.

B. id and age as integers, name as string.

C. All columns will be inferred as integers.

D. Schema will be empty because no schema is provided.

Solution

1. Step 1: Understand inferSchema option

   When inferSchema is true, Spark tries to detect the data type of each column automatically.

2. Step 2: Apply to given columns

   Since id and age contain integers, they will be inferred as integer types; name remains string.

3. Final Answer:

   id and age as integers, name as string. -> Option B

4. Quick Check:

   inferSchema=true means detect types [OK]

Hint: inferSchema=true detects column data types automatically [OK]

Common Mistakes:

• Assuming all columns are strings by default
• Thinking inferSchema converts all to integers
• Believing schema is empty without explicit definition

4. What is wrong with the following code snippet for reading a CSV file with a header and comma separator?

df = spark.read.csv(header=True, sep=',', 'file.csv')

medium

A. The header option must be set to 'false' to read headers.

B. The separator should be a semicolon, not a comma.

C. The file path must be an absolute path.

D. The options should be set using the option() method, not as parameters in csv().

Solution

1. Step 1: Check how options are passed in Spark CSV reading

   Spark requires options like header and sep to be set using option() method before calling csv().

2. Step 2: Identify the error in the code

   The code incorrectly passes options as parameters inside csv(), which is not supported and causes an error.

3. Final Answer:

   The options should be set using the option() method, not as parameters in csv(). -> Option D

4. Quick Check:

   Use .option() for CSV options, not csv() params [OK]

Hint: Set CSV options with .option(), not inside csv() call [OK]

Common Mistakes:

• Passing options directly inside csv()
• Using Python booleans instead of strings for options
• Confusing separator character

5. You want to read a CSV file that may have some corrupted rows. Which option should you use to skip corrupted rows and still read the rest of the data?

hard

A. mode='dropmalformed'

B. mode='failfast'

C. mode='permissive'

D. mode='ignore'

Solution

1. Step 1: Understand CSV reading modes in Spark

   Spark supports modes like 'permissive', 'failfast', and 'dropmalformed' to handle corrupted rows.

2. Step 2: Identify mode to skip corrupted rows

   The 'dropmalformed' mode skips corrupted rows and reads the rest without error.

3. Final Answer:

   mode='dropmalformed' -> Option A

4. Quick Check:

   Use dropmalformed to skip bad rows [OK]

Hint: Use mode='dropmalformed' to skip bad rows safely [OK]

Common Mistakes:

• Using failfast which stops on error
• Assuming permissive skips corrupted rows
• Using ignore which is not a valid mode