Data serialization (Avro, Parquet, ORC) in Hadoop - Mini Project: Build & Apply

Working with Data Serialization Formats: Avro, Parquet, and ORC

📖 Scenario: You work in a company that collects sales data daily. The data is stored in different formats to save space and speed up processing. You want to practice reading and writing data using popular serialization formats: Avro, Parquet, and ORC.

🎯 Goal: Learn how to create a simple dataset, configure the output format, write the data in the chosen serialization format, and then read it back to see the stored data.

📋 What You'll Learn

Create a sample dataset as a list of dictionaries

Set a variable to choose the serialization format

Write the dataset to a file in the chosen format

Read the file back and print the data

💡 Why This Matters

🌍 Real World

Data serialization formats like Avro, Parquet, and ORC are used in big data systems to store and transfer data efficiently.

💼 Career

Knowing how to read and write these formats is important for data engineers and data scientists working with Hadoop and big data tools.

1

Create the sample sales data

Create a variable called sales_data that is a list of dictionaries. Each dictionary should have these exact keys and values: {'date': '2024-04-01', 'product': 'apple', 'quantity': 10}, {'date': '2024-04-01', 'product': 'banana', 'quantity': 5}, and {'date': '2024-04-02', 'product': 'apple', 'quantity': 7}.

Hadoop

# Create the sales_data list with the exact dictionaries
# Your code here

Hint

Use a list with three dictionaries exactly as shown.

2

Set the serialization format

Create a variable called format_choice and set it to the string 'parquet'. This variable will decide which serialization format to use.

Hadoop

sales_data = [
    {'date': '2024-04-01', 'product': 'apple', 'quantity': 10},
    {'date': '2024-04-01', 'product': 'banana', 'quantity': 5},
    {'date': '2024-04-02', 'product': 'apple', 'quantity': 7}
]
# Set format_choice to 'parquet'
# Your code here

Hint

Set the variable exactly as format_choice = 'parquet'.

3

Write the data to a file in the chosen format

Use the format_choice variable to write sales_data to a file named sales_data with the correct extension: .parquet for 'parquet', .avro for 'avro', and .orc for 'orc'. Use PyArrow or fastavro libraries to write the data. Write the code to handle the 'parquet' case only.

Hadoop

sales_data = [
    {'date': '2024-04-01', 'product': 'apple', 'quantity': 10},
    {'date': '2024-04-01', 'product': 'banana', 'quantity': 5},
    {'date': '2024-04-02', 'product': 'apple', 'quantity': 7}
]
format_choice = 'parquet'

# Write sales_data to 'sales_data.parquet' if format_choice is 'parquet'
# Your code here

Hint

Use PyArrow's Table.from_pylist and write_table functions for parquet.

4

Read the data back and print it

Read the data from the sales_data.parquet file using PyArrow and print the result as a list of dictionaries.

Hadoop

import pyarrow as pa
import pyarrow.parquet as pq

sales_data = [
    {'date': '2024-04-01', 'product': 'apple', 'quantity': 10},
    {'date': '2024-04-01', 'product': 'banana', 'quantity': 5},
    {'date': '2024-04-02', 'product': 'apple', 'quantity': 7}
]
format_choice = 'parquet'

if format_choice == 'parquet':
    table = pa.Table.from_pylist(sales_data)
    pq.write_table(table, 'sales_data.parquet')

# Read the parquet file and print the data
# Your code here

Hint

Use pq.read_table and to_pylist() to read and convert the data.

Practice

1. Which data serialization format stores data in a row-wise manner and includes a schema for each record?

easy

A. Avro

B. Parquet

C. ORC

D. CSV

Solution

1. Step 1: Understand Avro's storage method

   Avro stores data row-wise, meaning it saves one record at a time along with its schema.

2. Step 2: Compare with other formats

   Parquet and ORC store data column-wise, optimizing for query speed, while CSV has no schema.

3. Final Answer:

   Avro -> Option A

4. Quick Check:

   Row-wise with schema = Avro [OK]

Hint: Row-wise + schema means Avro format [OK]

Common Mistakes:

• Confusing Parquet or ORC as row-wise
• Thinking CSV has schema
• Mixing storage orientation concepts

2. Which of the following is the correct way to specify a Parquet file format in a Hadoop job configuration?

easy

A. job.setOutputFormatClass(TextOutputFormat.class);

B. job.setOutputFormatClass(AvroOutputFormat.class);

C. job.setOutputFormatClass(OrcOutputFormat.class);

D. job.setOutputFormatClass(ParquetOutputFormat.class);

Solution

1. Step 1: Identify the output format class for Parquet

   Parquet files require setting the output format to ParquetOutputFormat in Hadoop jobs.

2. Step 2: Eliminate other options

   TextOutputFormat is for text files, AvroOutputFormat for Avro files, and OrcOutputFormat for ORC files.

3. Final Answer:

   job.setOutputFormatClass(ParquetOutputFormat.class); -> Option D

4. Quick Check:

   Parquet output format = ParquetOutputFormat [OK]

Hint: Parquet uses ParquetOutputFormat class in Hadoop [OK]

Common Mistakes:

• Using TextOutputFormat for Parquet files
• Confusing AvroOutputFormat with ParquetOutputFormat
• Mixing ORC and Parquet output formats

3. Given the following PySpark code snippet, what will be the output schema of the saved file?

df.write.format('orc').save('/data/output')

medium

A. Data saved in ORC format with schema preserved

B. Data saved in CSV format without schema

C. Data saved in Parquet format with schema preserved

D. Data saved in Avro format with schema preserved

Solution

1. Step 1: Identify the format used in the code

   The code uses df.write.format('orc'), which means data is saved in ORC format.

2. Step 2: Understand ORC properties

   ORC stores data column-wise and preserves schema for efficient queries.

3. Final Answer:

   Data saved in ORC format with schema preserved -> Option A

4. Quick Check:

   ORC format with schema = Data saved in ORC format with schema preserved [OK]

Hint: format('orc') means ORC with schema saved [OK]

Common Mistakes:

• Assuming default is CSV
• Confusing ORC with Parquet or Avro
• Ignoring schema preservation in ORC

4. You try to read a Parquet file in Spark using this code:

spark.read.format('parquet').load('/data/file.orc')

What is the main issue here?

medium

A. File extension does not match format specified

B. Parquet format cannot read ORC files

C. Spark cannot read Parquet files

D. No issue, code will run fine

Solution

1. Step 1: Check the file format and extension

   The code tries to read a file with .orc extension using Parquet format.

2. Step 2: Understand format compatibility

   Parquet reader cannot read ORC files because they have different internal structures.

3. Final Answer:

   Parquet format cannot read ORC files -> Option B

4. Quick Check:

   Parquet reader ≠ ORC file [OK]

Hint: Parquet reader can't read ORC files [OK]

Common Mistakes:

• Ignoring file extension mismatch
• Assuming Spark auto-detects format
• Thinking Parquet and ORC are interchangeable

5. You have a large dataset with many columns but only a few are needed for analysis. Which data serialization format is best to optimize query speed and storage, and why?

hard

A. CSV, because it is simple and easy to read

B. Avro, because it stores data row-wise with schema

C. Parquet, because it stores data column-wise allowing fast column queries

D. JSON, because it supports nested data

Solution

1. Step 1: Understand the dataset and query needs

   Many columns exist but only a few are needed, so reading only those columns is important.

2. Step 2: Compare serialization formats for columnar access

   Parquet stores data column-wise, enabling fast queries on selected columns and saving storage.

3. Step 3: Eliminate other options

   Avro is row-wise, so it reads all columns; CSV and JSON lack efficient columnar storage and schema.

4. Final Answer:

   Parquet, because it stores data column-wise allowing fast column queries -> Option C

5. Quick Check:

   Column-wise storage = Parquet best for selective queries [OK]

Hint: Column-wise format like Parquet speeds up column queries [OK]

Common Mistakes:

• Choosing Avro for column queries
• Picking CSV or JSON for big data efficiency
• Ignoring storage and query speed trade-offs