Reading CSV files with options in Apache Spark - Step-by-Step Execution

Concept Flow - Reading CSV files with options

Start Spark Session

↓

Specify CSV File Path

↓

Set Read Options

↓

Call spark.read.csv() with options

↓

Load DataFrame

↓

Use DataFrame for Analysis

↓

End

This flow shows how Spark reads a CSV file by starting a session, setting options, loading the file into a DataFrame, and then using it.

Execution Sample

Apache Spark

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
df = spark.read.option("header", "true").option("inferSchema", "true").csv("data.csv")
df.show(3)

This code reads a CSV file with header and schema inference options, then shows the first 3 rows.

Execution Table

Step	Action	Option Set	Resulting DataFrame Schema	Output Preview
1	Start Spark Session	N/A	N/A	Session started
2	Set option header=true	header=true	Schema will use first row as column names	N/A
3	Set option inferSchema=true	inferSchema=true	Schema types inferred from data	N/A
4	Read CSV file 'data.csv'	header=true, inferSchema=true	Columns: id:int, name:string, age:int	First 3 rows displayed
5	Show DataFrame	N/A	N/A	[{id:1, name:'Alice', age:30}, {id:2, name:'Bob', age:25}, {id:3, name:'Cathy', age:28}]
6	End	N/A	N/A	DataFrame ready for analysis

💡 DataFrame loaded with options header and inferSchema, ready for use.

Variable Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	Final
spark	None	SparkSession active	SparkSession active	SparkSession active	SparkSession active
df	None	None	None	DataFrame with schema inferred	DataFrame with data loaded

Key Moments - 3 Insights

Why do we set option header=true when reading a CSV?

Setting header=true tells Spark to use the first row as column names instead of data, as shown in execution_table step 2 and 4.

What does inferSchema=true do?

It makes Spark guess the data types of each column from the data, so columns like 'age' become integers, as seen in execution_table step 3 and 4.

What happens if we don't set header=true?

Spark treats the first row as data, not column names, so column names become default like _c0, _c1, which is different from step 4 in execution_table.

Visual Quiz - 3 Questions

Look at the execution_table, what option is set at step 3?

A header=true

B inferSchema=true

C delimiter=','

D mode=PERMISSIVE

💡 Hint

Check the 'Option Set' column at step 3 in execution_table.

At which step does the DataFrame get its schema inferred?

A Step 2

B Step 3

C Step 4

D Step 5

💡 Hint

Look at the 'Resulting DataFrame Schema' column in execution_table.

If we remove header=true option, what changes in the output preview?

A Column names become default like _c0, _c1

B Schema is inferred correctly

C First row becomes column names

D DataFrame will be empty

💡 Hint

Refer to key_moments explanation about header option.

Concept Snapshot

spark.read.csv(path)
  .option('header', 'true')  # Use first row as column names
  .option('inferSchema', 'true')  # Detect data types automatically
Returns a DataFrame ready for analysis.
Without header option, columns get default names like _c0.

Full Transcript

This visual execution shows how to read CSV files in Apache Spark using options. First, a Spark session starts. Then, options like header=true and inferSchema=true are set to tell Spark to use the first row as column names and to guess data types. The CSV file is read into a DataFrame with these options applied. The DataFrame schema reflects the inferred types and column names. Finally, the first few rows are shown to confirm the data loaded correctly. Key points include why header option is important and how inferSchema works. The visual quiz tests understanding of these steps.

Practice

1. What does the header option do when reading a CSV file in Apache Spark?

easy

A. It tells Spark to treat the first line as column names.

B. It specifies the character used to separate columns.

C. It infers the data types of each column automatically.

D. It sets the file reading mode to permissive.

Solution

1. Step 1: Understand the header option

   The header option tells Spark whether the first line of the CSV file contains column names or data.

2. Step 2: Identify its effect

   If header is set to true, Spark uses the first line as column names; otherwise, it treats all lines as data.

3. Final Answer:

   It tells Spark to treat the first line as column names. -> Option A

4. Quick Check:

   header = first line as columns [OK]

Hint: Remember: header=true means first line is column names [OK]

Common Mistakes:

• Confusing header with separator option
• Thinking header infers data types
• Assuming header controls error handling

2. Which of the following is the correct way to read a CSV file with a semicolon separator and header in Spark?

easy

A. spark.read.csv(options={'sep': ';', 'header': 'true'}, 'file.csv')

B. spark.read.csv(sep=';', 'file.csv', header=True)

C. spark.read.option('sep', ';').option('header', 'true').csv('file.csv')

D. spark.read.csv(header=True, sep=';', 'file.csv')

Solution

1. Step 1: Recall Spark CSV reading syntax

   Spark uses option() method to set options before calling csv().

2. Step 2: Check each option usage

   spark.read.option('sep', ';').option('header', 'true').csv('file.csv') correctly chains option('sep', ';') and option('header', 'true') before reading the file.

3. Final Answer:

   spark.read.option('sep', ';').option('header', 'true').csv('file.csv') -> Option C

4. Quick Check:

   Use option() chaining for CSV options [OK]

Hint: Use .option() chaining to set CSV read options [OK]

Common Mistakes:

• Passing options directly inside csv() as kwargs
• Using Python booleans instead of strings for options
• Incorrect method chaining order

3. Given the code below, what will be the schema of the DataFrame?

df = spark.read.option('header', 'true').option('inferSchema', 'true').csv('data.csv')

Assuming data.csv has columns: id (integer), name (string), age (integer).

medium

A. All columns will be read as strings.

B. id and age as integers, name as string.

C. All columns will be inferred as integers.

D. Schema will be empty because no schema is provided.

Solution

1. Step 1: Understand inferSchema option

   When inferSchema is true, Spark tries to detect the data type of each column automatically.

2. Step 2: Apply to given columns

   Since id and age contain integers, they will be inferred as integer types; name remains string.

3. Final Answer:

   id and age as integers, name as string. -> Option B

4. Quick Check:

   inferSchema=true means detect types [OK]

Hint: inferSchema=true detects column data types automatically [OK]

Common Mistakes:

• Assuming all columns are strings by default
• Thinking inferSchema converts all to integers
• Believing schema is empty without explicit definition

4. What is wrong with the following code snippet for reading a CSV file with a header and comma separator?

df = spark.read.csv(header=True, sep=',', 'file.csv')

medium

A. The header option must be set to 'false' to read headers.

B. The separator should be a semicolon, not a comma.

C. The file path must be an absolute path.

D. The options should be set using the option() method, not as parameters in csv().

Solution

1. Step 1: Check how options are passed in Spark CSV reading

   Spark requires options like header and sep to be set using option() method before calling csv().

2. Step 2: Identify the error in the code

   The code incorrectly passes options as parameters inside csv(), which is not supported and causes an error.

3. Final Answer:

   The options should be set using the option() method, not as parameters in csv(). -> Option D

4. Quick Check:

   Use .option() for CSV options, not csv() params [OK]

Hint: Set CSV options with .option(), not inside csv() call [OK]

Common Mistakes:

• Passing options directly inside csv()
• Using Python booleans instead of strings for options
• Confusing separator character

5. You want to read a CSV file that may have some corrupted rows. Which option should you use to skip corrupted rows and still read the rest of the data?

hard

A. mode='dropmalformed'

B. mode='failfast'

C. mode='permissive'

D. mode='ignore'

Solution

1. Step 1: Understand CSV reading modes in Spark

   Spark supports modes like 'permissive', 'failfast', and 'dropmalformed' to handle corrupted rows.

2. Step 2: Identify mode to skip corrupted rows

   The 'dropmalformed' mode skips corrupted rows and reads the rest without error.

3. Final Answer:

   mode='dropmalformed' -> Option A

4. Quick Check:

   Use dropmalformed to skip bad rows [OK]

Hint: Use mode='dropmalformed' to skip bad rows safely [OK]

Common Mistakes:

• Using failfast which stops on error
• Assuming permissive skips corrupted rows
• Using ignore which is not a valid mode