Data serialization (Avro, Parquet, ORC) in Hadoop

Introduction

Data serialization helps save and share data efficiently. It makes data smaller and faster to read or write.

When you want to store large amounts of data in a compact way.

When you need to transfer data between different systems quickly.

When you want to read data fast for analysis or reporting.

When you want to keep data organized with a schema.

When you want to save data in a format that works well with big data tools like Hadoop.

Syntax

Hadoop

Use tools or libraries to write/read data in formats like Avro, Parquet, or ORC.
Example: df.write.format('parquet').save('path')

Avro stores data row-wise and includes a schema for easy reading and writing.

Parquet and ORC store data column-wise, which is faster for reading specific columns.

Examples

Read data stored in Avro format using Spark.

Hadoop

spark.read.format('avro').load('data.avro')

Save a DataFrame in Parquet format for efficient storage.

Hadoop

df.write.format('parquet').save('output/path')

Load data stored in ORC format for fast columnar access.

Hadoop

spark.read.format('orc').load('data.orc')

Sample Program

This program creates a small table with three people. It saves the data in Avro, Parquet, and ORC formats. Then it reads each format back and shows the data. This helps see how serialization works in Hadoop with Spark.

Hadoop

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('SerializationExample').getOrCreate()

# Create sample data
data = [(1, 'Alice', 29), (2, 'Bob', 31), (3, 'Cathy', 25)]
columns = ['id', 'name', 'age']
df = spark.createDataFrame(data, columns)

# Save data in Avro format
avro_path = '/tmp/data_avro'
df.write.format('avro').mode('overwrite').save(avro_path)

# Read back Avro data
df_avro = spark.read.format('avro').load(avro_path)
print('Avro Data:')
df_avro.show()

# Save data in Parquet format
parquet_path = '/tmp/data_parquet'
df.write.format('parquet').mode('overwrite').save(parquet_path)

# Read back Parquet data
df_parquet = spark.read.format('parquet').load(parquet_path)
print('Parquet Data:')
df_parquet.show()

# Save data in ORC format
orc_path = '/tmp/data_orc'
df.write.format('orc').mode('overwrite').save(orc_path)

# Read back ORC data
df_orc = spark.read.format('orc').load(orc_path)
print('ORC Data:')
df_orc.show()

spark.stop()

Important Notes

Avro is good when you want to keep the schema with the data for easy reading and writing.

Parquet and ORC are better for big data queries because they store data by columns, making reading faster.

Make sure the storage path exists and you have permission to write when saving data.

Summary

Data serialization saves data in formats like Avro, Parquet, and ORC to make storage and reading efficient.

Avro stores data row-wise with schema, while Parquet and ORC store data column-wise for faster queries.

These formats are widely used in big data tools like Hadoop and Spark.

Practice

1. Which data serialization format stores data in a row-wise manner and includes a schema for each record?

easy

A. Avro

B. Parquet

C. ORC

D. CSV

Solution

1. Step 1: Understand Avro's storage method

   Avro stores data row-wise, meaning it saves one record at a time along with its schema.

2. Step 2: Compare with other formats

   Parquet and ORC store data column-wise, optimizing for query speed, while CSV has no schema.

3. Final Answer:

   Avro -> Option A

4. Quick Check:

   Row-wise with schema = Avro [OK]

Hint: Row-wise + schema means Avro format [OK]

Common Mistakes:

• Confusing Parquet or ORC as row-wise
• Thinking CSV has schema
• Mixing storage orientation concepts

2. Which of the following is the correct way to specify a Parquet file format in a Hadoop job configuration?

easy

A. job.setOutputFormatClass(TextOutputFormat.class);

B. job.setOutputFormatClass(AvroOutputFormat.class);

C. job.setOutputFormatClass(OrcOutputFormat.class);

D. job.setOutputFormatClass(ParquetOutputFormat.class);

Solution

1. Step 1: Identify the output format class for Parquet

   Parquet files require setting the output format to ParquetOutputFormat in Hadoop jobs.

2. Step 2: Eliminate other options

   TextOutputFormat is for text files, AvroOutputFormat for Avro files, and OrcOutputFormat for ORC files.

3. Final Answer:

   job.setOutputFormatClass(ParquetOutputFormat.class); -> Option D

4. Quick Check:

   Parquet output format = ParquetOutputFormat [OK]

Hint: Parquet uses ParquetOutputFormat class in Hadoop [OK]

Common Mistakes:

• Using TextOutputFormat for Parquet files
• Confusing AvroOutputFormat with ParquetOutputFormat
• Mixing ORC and Parquet output formats

3. Given the following PySpark code snippet, what will be the output schema of the saved file?

df.write.format('orc').save('/data/output')

medium

A. Data saved in ORC format with schema preserved

B. Data saved in CSV format without schema

C. Data saved in Parquet format with schema preserved

D. Data saved in Avro format with schema preserved

Solution

1. Step 1: Identify the format used in the code

   The code uses df.write.format('orc'), which means data is saved in ORC format.

2. Step 2: Understand ORC properties

   ORC stores data column-wise and preserves schema for efficient queries.

3. Final Answer:

   Data saved in ORC format with schema preserved -> Option A

4. Quick Check:

   ORC format with schema = Data saved in ORC format with schema preserved [OK]

Hint: format('orc') means ORC with schema saved [OK]

Common Mistakes:

• Assuming default is CSV
• Confusing ORC with Parquet or Avro
• Ignoring schema preservation in ORC

4. You try to read a Parquet file in Spark using this code:

spark.read.format('parquet').load('/data/file.orc')

What is the main issue here?

medium

A. File extension does not match format specified

B. Parquet format cannot read ORC files

C. Spark cannot read Parquet files

D. No issue, code will run fine

Solution

1. Step 1: Check the file format and extension

   The code tries to read a file with .orc extension using Parquet format.

2. Step 2: Understand format compatibility

   Parquet reader cannot read ORC files because they have different internal structures.

3. Final Answer:

   Parquet format cannot read ORC files -> Option B

4. Quick Check:

   Parquet reader ≠ ORC file [OK]

Hint: Parquet reader can't read ORC files [OK]

Common Mistakes:

• Ignoring file extension mismatch
• Assuming Spark auto-detects format
• Thinking Parquet and ORC are interchangeable

5. You have a large dataset with many columns but only a few are needed for analysis. Which data serialization format is best to optimize query speed and storage, and why?

hard

A. CSV, because it is simple and easy to read

B. Avro, because it stores data row-wise with schema

C. Parquet, because it stores data column-wise allowing fast column queries

D. JSON, because it supports nested data

Solution

1. Step 1: Understand the dataset and query needs

   Many columns exist but only a few are needed, so reading only those columns is important.

2. Step 2: Compare serialization formats for columnar access

   Parquet stores data column-wise, enabling fast queries on selected columns and saving storage.

3. Step 3: Eliminate other options

   Avro is row-wise, so it reads all columns; CSV and JSON lack efficient columnar storage and schema.

4. Final Answer:

   Parquet, because it stores data column-wise allowing fast column queries -> Option C

5. Quick Check:

   Column-wise storage = Parquet best for selective queries [OK]

Hint: Column-wise format like Parquet speeds up column queries [OK]

Common Mistakes:

• Choosing Avro for column queries
• Picking CSV or JSON for big data efficiency
• Ignoring storage and query speed trade-offs