Creating DataFrames from files (CSV, JSON, Parquet) in Apache Spark - Quick Revision & Summary

Recall & Review

beginner

What is a DataFrame in Apache Spark?

A DataFrame is a table-like data structure in Spark that holds data in rows and columns, similar to a spreadsheet or SQL table. It allows easy data manipulation and analysis.

beginner

How do you read a CSV file into a Spark DataFrame?

Use spark.read.csv('file_path', header=True, inferSchema=True) to load a CSV file. 'header=True' means the first row is column names, and 'inferSchema=True' lets Spark guess data types.

intermediate

What is the difference between reading JSON and Parquet files in Spark?

JSON files are text-based and semi-structured, read with spark.read.json('file_path'). Parquet files are binary, columnar, and optimized for speed, read with spark.read.parquet('file_path').

intermediate

Why use Parquet files instead of CSV or JSON in Spark?

Parquet files are faster to read and use less storage because they store data in a compressed, column-based format. This makes them better for big data processing.

beginner

What does the 'inferSchema' option do when reading CSV files in Spark?

'inferSchema=True' tells Spark to automatically detect the data type of each column instead of treating all columns as strings.

Which Spark method reads a JSON file into a DataFrame?

A spark.read.parquet('file_path')

B spark.read.json('file_path')

C spark.read.csv('file_path')

D spark.read.text('file_path')

Use spark.read.json() to load JSON files into a DataFrame.

What option should you set to True to use the first row as column names when reading a CSV?

A inferSchema=True

B delimiter=','

C header=True

D schema=True

Setting header=True tells Spark the first row contains column names.

Which file format is columnar and optimized for fast reading in Spark?

A Parquet

B JSON

C CSV

D TXT

Parquet is a columnar storage format designed for fast data processing.

What does 'inferSchema=True' do when reading CSV files?

A Automatically detects column data types

B Uses the first row as data

C Skips empty rows

D Reads the file faster

It lets Spark guess the correct data types for each column.

Which method reads Parquet files in Spark?

A spark.read.text()

B spark.read.json()

C spark.read.csv()

D spark.read.parquet()

spark.read.parquet() loads Parquet files into DataFrames.

Explain how to create a Spark DataFrame from a CSV file including important options.

Think about telling Spark where the file is and how to treat the first row and data types.

Describe the advantages of using Parquet files over CSV or JSON in Spark.

Focus on speed and storage benefits.