Reading CSV files with options in Apache Spark - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is the purpose of using options when reading CSV files in Apache Spark?

Options help customize how Spark reads CSV files, such as setting the delimiter, header presence, and handling null values. This makes data reading flexible and accurate.

beginner

How do you tell Spark that the CSV file has a header row?

You use the option "header" set to "true" like this: .option("header", "true"). This tells Spark to treat the first row as column names.

intermediate

What does the option "inferSchema" do when reading a CSV file?

Setting .option("inferSchema", "true") tells Spark to automatically detect the data types of each column instead of treating all data as strings.

beginner

How can you change the delimiter used in a CSV file when reading it with Spark?

Use the option "sep" with the delimiter character, for example: .option("sep", ";") to use semicolon as the separator instead of comma.

beginner

What happens if you do not specify the "header" option when reading a CSV file in Spark?

Spark treats all rows as data, and column names will be default names like _c0, _c1, etc. The first row will not be used as headers.

Which option tells Spark to treat the first row of a CSV file as column headers?

A header = true

B inferSchema = true

C sep = ","

D nullValue = "null"

The option header = true tells Spark to use the first row as column names.

What does setting inferSchema = true do when reading a CSV file?

A It skips the first row

B It changes the delimiter

C It automatically detects column data types

D It removes null values

inferSchema = true makes Spark detect the data types of columns automatically.

How do you specify a semicolon as the delimiter in a CSV file read by Spark?

A option("delimiter", ";")

B option("sep", ";")

C option("header", ";")

D option("inferSchema", ";")

The option sep sets the delimiter character, so sep = ";" uses semicolon.

If you do not set header = true, what will Spark do with the first row?

A Treat it as data

B Skip it

C Use it as column names

D Throw an error

Without header = true, Spark treats the first row as normal data.

Which option can you use to specify a string to treat as null in CSV data?

A sep

B header

C inferSchema

D nullValue

nullValue option lets you specify which string should be treated as null.

Explain how to read a CSV file in Spark that has a header row, uses semicolon as delimiter, and requires automatic schema detection.

Think about the options needed for header, delimiter, and schema.

What are the effects of not setting the header option when reading CSV files in Spark?

Consider how Spark names columns and treats rows by default.