Parquet format and columnar storage in Apache Spark - Deep Dive

Overview - Parquet format and columnar storage

What is it?

Parquet is a file format designed to store data in a column-oriented way. Instead of saving data row by row, it saves data column by column. This helps programs read only the data they need, making data processing faster and more efficient. It is widely used in big data tools like Apache Spark.

Why it matters

Without columnar storage like Parquet, data processing systems would have to read entire rows even if only a few columns are needed. This wastes time and computing power, especially with large datasets. Parquet helps save storage space and speeds up queries, making data analysis faster and cheaper.

Where it fits

Before learning Parquet, you should understand basic data storage formats like CSV and JSON and how data is organized in rows and columns. After mastering Parquet, you can explore advanced data processing techniques in Apache Spark, such as partitioning, predicate pushdown, and optimization strategies.

Mental Model

Core Idea

Parquet stores data by columns, not rows, so you can read only the parts you need, making data processing faster and smaller in size.

Think of it like...

Imagine a library where books are stored by chapters instead of whole books. If you only want one chapter, you don’t have to carry the entire book around.

┌───────────────┐
│   Parquet     │
├───────────────┤
│ Column 1 data │
│ Column 1 data │
│ Column 1 data │
├───────────────┤
│ Column 2 data │
│ Column 2 data │
│ Column 2 data │
├───────────────┤
│ Column 3 data │
│ Column 3 data │
│ Column 3 data │
└───────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding Row vs Column Storage

Concept: Difference between storing data by rows and by columns.

Traditional data files like CSV store data row by row. Each row contains all columns for one record. Columnar storage saves all values of one column together, then moves to the next column. This changes how data is read and stored.

Result

You see that row storage reads full records, while column storage reads only needed columns.

Understanding this difference is key to grasping why columnar formats like Parquet improve speed and storage.

2

Foundation

Basics of Parquet File Format

Concept: Parquet is a columnar file format with metadata and compression.

Parquet files store data in columns with metadata describing the schema and data types. It uses compression to reduce file size. This format is optimized for big data tools like Apache Spark.

Result

You get smaller files that keep track of data types and structure for efficient reading.

Knowing Parquet stores metadata helps understand how it speeds up data queries.

3

Intermediate

How Columnar Storage Speeds Up Queries

🤔Before reading on: do you think reading column data is always faster than row data? Commit to your answer.

Concept: Columnar storage allows reading only relevant columns, reducing data read and improving speed.

When a query needs only a few columns, Parquet reads just those columns from disk. This avoids loading unnecessary data. It also enables better compression because similar data types are stored together.

Result

Queries run faster and use less memory because less data is loaded.

Understanding selective reading explains why columnar formats are preferred for analytics.

4

Intermediate

Role of Metadata and Schema in Parquet

Concept: Parquet files include metadata that describes the data schema and statistics.

Metadata stores information like column names, data types, and min/max values. This helps query engines skip reading data blocks that don’t match query filters, a technique called predicate pushdown.

Result

Queries can skip irrelevant data blocks, speeding up processing.

Knowing metadata enables advanced optimizations that reduce data scanning.

5

Intermediate

Compression Benefits in Columnar Storage

Concept: Storing similar data types together improves compression efficiency.

Because each column stores similar data types, compression algorithms work better. For example, a column of integers compresses more than mixed data in rows. This reduces storage size and speeds up data transfer.

Result

Files are smaller and faster to read from disk or network.

Recognizing compression advantages explains why columnar formats save storage costs.

6

Advanced

Parquet in Apache Spark Workflows

🤔Before reading on: do you think Spark treats Parquet files differently than CSV? Commit to your answer.

Concept: Spark uses Parquet’s metadata and columnar layout to optimize query execution.

Spark reads Parquet files using schema information and can push filters down to skip data. It also reads only needed columns, reducing memory use. This integration makes Spark jobs faster and more scalable.

Result

Spark jobs on Parquet run faster and use fewer resources than on row-based files.

Understanding Spark’s use of Parquet explains why it is the preferred format in big data.

7

Expert

Internal Structure and Page Encoding in Parquet

🤔Before reading on: do you think Parquet stores all column data in one big block or smaller pages? Commit to your answer.

Concept: Parquet divides columns into pages with encoding and compression for efficient access.

Each column chunk is split into pages. Pages store data with encoding schemes like dictionary or run-length encoding. This allows fast decoding and skipping of pages during queries. Page-level metadata helps skip irrelevant data quickly.

Result

Parquet achieves high performance by combining columnar layout with page-level encoding and metadata.

Knowing about pages and encoding reveals how Parquet balances speed and compression in production.

Under the Hood

Parquet files organize data into row groups, each containing column chunks. Each column chunk is split into pages that store encoded and compressed data. Metadata at file, row group, and page levels describe schema, statistics, and encoding. When reading, systems use metadata to skip irrelevant data and read only needed columns and pages, reducing I/O and CPU work.

Why designed this way?

Parquet was designed to optimize big data analytics by minimizing disk I/O and memory use. Columnar storage was chosen because analytical queries often access few columns. Page-level encoding and metadata enable fine-grained skipping and compression. Alternatives like row-based formats were slower and larger for analytics workloads.

┌─────────────────────────────┐
│          Parquet File       │
├──────────────┬──────────────┤
│  Metadata    │  Row Groups  │
│ (Schema,     │ ┌──────────┐ │
│  Stats)     │ │ Row Group │ │
│              │ │ 1        │ │
│              │ │ ┌──────┐ │ │
│              │ │ │Column│ │ │
│              │ │ │Chunk │ │ │
│              │ │ │ 1    │ │ │
│              │ │ │ ┌───┐│ │ │
│              │ │ │ │Page││ │ │
│              │ │ │ │ 1  ││ │ │
│              │ │ │ └───┘│ │ │
│              │ │ │ ┌───┐│ │ │
│              │ │ │ │Page││ │ │
│              │ │ │ │ 2  ││ │ │
│              │ │ │ └───┘│ │ │
│              │ │ └──────┘ │
│              │ └──────────┘
└──────────────┴──────────────┘

Myth Busters - 3 Common Misconceptions

Quick: Does Parquet always make queries faster regardless of data size? Commit to yes or no.

Common Belief: Parquet always makes queries faster no matter what.

Reality: Parquet improves performance mostly on large datasets and analytical queries. For small datasets or queries needing most columns, row-based formats can be as fast or faster.

Why it matters: Assuming Parquet is always faster can lead to unnecessary complexity and slower performance on small or transactional workloads.

Quick: Do you think Parquet stores data exactly as it appears in the source? Commit to yes or no.

Common Belief: Parquet stores data exactly as in the original source without changes.

Reality: Parquet applies encoding and compression, changing how data is stored internally to save space and speed up reading.

Why it matters: Expecting raw data storage can confuse debugging and data inspection, leading to wrong assumptions about file contents.

Quick: Is Parquet suitable for all types of data, including unstructured text? Commit to yes or no.

Common Belief: Parquet is good for any data type, including unstructured text and images.

Reality: Parquet is optimized for structured, tabular data. It is not ideal for unstructured data like images or free text blobs.

Why it matters: Using Parquet for unstructured data wastes space and reduces performance; other formats like Avro or ORC may be better.

Expert Zone

1

Parquet’s page-level encoding allows skipping data within columns, not just whole columns, improving query speed on filtered data.

2

Predicate pushdown works best when filters match column statistics stored in metadata, but complex filters may not benefit.

3

Parquet files can be split and read in parallel by distributed systems, but improper row group sizing can hurt performance.

When NOT to use

Avoid Parquet for small datasets, real-time transactional systems, or unstructured data like images or logs. Use row-based formats like JSON or CSV for simple, small data or formats like Avro for schema evolution and streaming.

Production Patterns

In production, Parquet is used with partitioning by columns like date to speed up queries. It is combined with Spark’s Catalyst optimizer to push filters and select columns. Data lakes store large Parquet files on cloud storage for scalable analytics.

Connections

Database Indexing

Both use metadata to speed up data retrieval by skipping irrelevant data.

Understanding Parquet’s metadata is like understanding database indexes, which help avoid scanning all data.

Video Compression

Both use encoding and compression to reduce size while preserving essential information.

Knowing how Parquet encodes data pages is similar to how video codecs compress frames, balancing quality and size.

Library Cataloging Systems

Both organize large collections to quickly find needed items without scanning everything.

Parquet’s columnar layout and metadata act like a library’s catalog, helping find data fast.

Common Pitfalls

#1 Reading entire Parquet files when only a few columns are needed.

Wrong approach:

df = spark.read.parquet('data.parquet').select('*')

Correct approach:

df = spark.read.parquet('data.parquet').select('needed_column1', 'needed_column2')

Root cause:

Not leveraging column pruning causes unnecessary data loading and slows queries.

#2 Writing very small Parquet files without partitioning.

Wrong approach:

df.write.parquet('output_path') # no partitioning, small files

Correct approach:

df.write.partitionBy('date').parquet('output_path') # partitions by date

Root cause:

Ignoring partitioning leads to many small files, hurting read performance and increasing overhead.

#3 Using Parquet for streaming unstructured logs.

Wrong approach:

streaming_df.writeStream.format('parquet').start()

Correct approach:

streaming_df.writeStream.format('json').start() # better for unstructured logs

Root cause:

Parquet is not designed for unstructured streaming data, causing inefficiency and errors.

Key Takeaways

Parquet stores data by columns, enabling faster queries by reading only needed data.

Metadata and encoding in Parquet allow skipping irrelevant data and compressing efficiently.

Parquet is ideal for large, structured datasets and analytical workloads, especially with tools like Apache Spark.

Misusing Parquet for small or unstructured data can reduce performance and increase complexity.

Understanding Parquet’s internal structure helps optimize big data pipelines and storage.