Data serialization (Avro, Parquet, ORC) in Hadoop - Step-by-Step Execution

Concept Flow - Data serialization (Avro, Parquet, ORC)

Raw Data

↓

Choose Format

↓

Avro

↓

Serialize Data

↓

Store in Hadoop

↓

Read & Deserialize

↓

Use Data for Analysis

Data flows from raw form to serialization in Avro, Parquet, or ORC formats, then stored and later read back for analysis.

Execution Sample

Hadoop

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame([(1, 'Alice'), (2, 'Bob')], ['id', 'name'])
df.write.parquet('data.parquet')
read_df = spark.read.parquet('data.parquet')
read_df.show()

This code saves a small table in Parquet format and reads it back to show the data.

Execution Table

Step	Action	Input Data	Format Used	Output / Result
1	Create DataFrame	[(1, 'Alice'), (2, 'Bob')]	N/A	DataFrame with 2 rows
2	Write DataFrame to file	DataFrame	Parquet	File 'data.parquet' created
3	Read file back	File 'data.parquet'	Parquet	DataFrame with 2 rows
4	Show DataFrame	DataFrame	N/A	Displays rows: (1, Alice), (2, Bob)
5	End	N/A	N/A	Process complete

💡 All steps completed successfully; data serialized and deserialized using Parquet.

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
df	None	DataFrame with 2 rows	DataFrame unchanged	DataFrame unchanged	DataFrame unchanged	DataFrame unchanged
read_df	None	None	None	DataFrame with 2 rows	DataFrame unchanged	DataFrame unchanged

Key Moments - 2 Insights

Why do we need to write data in a special format like Parquet instead of just saving as text?

Parquet stores data in a compact, efficient way with schema info, making reading faster and saving space, as shown in step 2 where the DataFrame is saved in Parquet format.

What happens if we try to read the data without specifying the correct format?

The system may fail or read data incorrectly because it expects the format used during writing, as seen in step 3 where reading uses Parquet format to correctly deserialize.

Visual Quiz - 3 Questions

Look at the execution table, what is the content of 'df' after step 1?

A DataFrame with 2 rows

B File 'data.parquet'

C Empty DataFrame

D None

💡 Hint

Check the 'df' variable state in variable_tracker after step 1.

At which step is the data actually saved to disk in Parquet format?

A Step 1

B Step 2

C Step 3

D Step 4

💡 Hint

Look at the 'Action' and 'Output / Result' columns in execution_table.

If we changed the format from Parquet to Avro in step 2, what would change in step 3?

A DataFrame would have more rows

B No change, still read as Parquet

C We would read the file using Avro format instead of Parquet

D DataFrame would be empty

💡 Hint

Refer to the 'Format Used' column in execution_table for steps 2 and 3.

Concept Snapshot

Data serialization saves data in formats like Avro, Parquet, or ORC.
These formats store data efficiently with schema info.
Use write() to save and read() to load data.
Choose format based on use case: Avro for row-based, Parquet/ORC for column-based.
Serialization improves storage and query speed.

Full Transcript

This visual execution shows how raw data is converted into serialized formats like Avro, Parquet, or ORC for storage in Hadoop. The flow starts with raw data, choosing a format, serializing, storing, then reading back for analysis. The sample code creates a small table, saves it as Parquet, reads it back, and displays the data. The execution table traces each step, showing data creation, writing to Parquet, reading from Parquet, and displaying results. Variable tracking shows how the DataFrame variables change over steps. Key moments clarify why special formats are used and the importance of matching read and write formats. The quiz tests understanding of variable states, steps of saving, and format changes. The snapshot summarizes key points about data serialization formats and their use in Hadoop environments.

Practice

1. Which data serialization format stores data in a row-wise manner and includes a schema for each record?

easy

A. Avro

B. Parquet

C. ORC

D. CSV

Solution

1. Step 1: Understand Avro's storage method

   Avro stores data row-wise, meaning it saves one record at a time along with its schema.

2. Step 2: Compare with other formats

   Parquet and ORC store data column-wise, optimizing for query speed, while CSV has no schema.

3. Final Answer:

   Avro -> Option A

4. Quick Check:

   Row-wise with schema = Avro [OK]

Hint: Row-wise + schema means Avro format [OK]

Common Mistakes:

• Confusing Parquet or ORC as row-wise
• Thinking CSV has schema
• Mixing storage orientation concepts

2. Which of the following is the correct way to specify a Parquet file format in a Hadoop job configuration?

easy

A. job.setOutputFormatClass(TextOutputFormat.class);

B. job.setOutputFormatClass(AvroOutputFormat.class);

C. job.setOutputFormatClass(OrcOutputFormat.class);

D. job.setOutputFormatClass(ParquetOutputFormat.class);

Solution

1. Step 1: Identify the output format class for Parquet

   Parquet files require setting the output format to ParquetOutputFormat in Hadoop jobs.

2. Step 2: Eliminate other options

   TextOutputFormat is for text files, AvroOutputFormat for Avro files, and OrcOutputFormat for ORC files.

3. Final Answer:

   job.setOutputFormatClass(ParquetOutputFormat.class); -> Option D

4. Quick Check:

   Parquet output format = ParquetOutputFormat [OK]

Hint: Parquet uses ParquetOutputFormat class in Hadoop [OK]

Common Mistakes:

• Using TextOutputFormat for Parquet files
• Confusing AvroOutputFormat with ParquetOutputFormat
• Mixing ORC and Parquet output formats

3. Given the following PySpark code snippet, what will be the output schema of the saved file?

df.write.format('orc').save('/data/output')

medium

A. Data saved in ORC format with schema preserved

B. Data saved in CSV format without schema

C. Data saved in Parquet format with schema preserved

D. Data saved in Avro format with schema preserved

Solution

1. Step 1: Identify the format used in the code

   The code uses df.write.format('orc'), which means data is saved in ORC format.

2. Step 2: Understand ORC properties

   ORC stores data column-wise and preserves schema for efficient queries.

3. Final Answer:

   Data saved in ORC format with schema preserved -> Option A

4. Quick Check:

   ORC format with schema = Data saved in ORC format with schema preserved [OK]

Hint: format('orc') means ORC with schema saved [OK]

Common Mistakes:

• Assuming default is CSV
• Confusing ORC with Parquet or Avro
• Ignoring schema preservation in ORC

4. You try to read a Parquet file in Spark using this code:

spark.read.format('parquet').load('/data/file.orc')

What is the main issue here?

medium

A. File extension does not match format specified

B. Parquet format cannot read ORC files

C. Spark cannot read Parquet files

D. No issue, code will run fine

Solution

1. Step 1: Check the file format and extension

   The code tries to read a file with .orc extension using Parquet format.

2. Step 2: Understand format compatibility

   Parquet reader cannot read ORC files because they have different internal structures.

3. Final Answer:

   Parquet format cannot read ORC files -> Option B

4. Quick Check:

   Parquet reader ≠ ORC file [OK]

Hint: Parquet reader can't read ORC files [OK]

Common Mistakes:

• Ignoring file extension mismatch
• Assuming Spark auto-detects format
• Thinking Parquet and ORC are interchangeable

5. You have a large dataset with many columns but only a few are needed for analysis. Which data serialization format is best to optimize query speed and storage, and why?

hard

A. CSV, because it is simple and easy to read

B. Avro, because it stores data row-wise with schema

C. Parquet, because it stores data column-wise allowing fast column queries

D. JSON, because it supports nested data

Solution

1. Step 1: Understand the dataset and query needs

   Many columns exist but only a few are needed, so reading only those columns is important.

2. Step 2: Compare serialization formats for columnar access

   Parquet stores data column-wise, enabling fast queries on selected columns and saving storage.

3. Step 3: Eliminate other options

   Avro is row-wise, so it reads all columns; CSV and JSON lack efficient columnar storage and schema.

4. Final Answer:

   Parquet, because it stores data column-wise allowing fast column queries -> Option C

5. Quick Check:

   Column-wise storage = Parquet best for selective queries [OK]

Hint: Column-wise format like Parquet speeds up column queries [OK]

Common Mistakes:

• Choosing Avro for column queries
• Picking CSV or JSON for big data efficiency
• Ignoring storage and query speed trade-offs