Partitioning for query performance in Hadoop - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is partitioning in the context of Hadoop data storage?

Partitioning means dividing a large dataset into smaller, manageable parts based on a column's values. This helps Hadoop process queries faster by scanning only relevant parts.

beginner

How does partitioning improve query performance in Hadoop?

Partitioning reduces the amount of data scanned during a query by filtering partitions based on query conditions. This means less data to read and faster results.

beginner

What is a common column type used for partitioning in Hadoop tables?

Columns with discrete values like dates, regions, or categories are commonly used for partitioning because they split data into meaningful groups.

intermediate

What happens if you over-partition your data in Hadoop?

Over-partitioning creates many small files, which can slow down query performance due to overhead in managing many partitions and files.

intermediate

Explain the difference between partitioning and bucketing in Hadoop.

Partitioning divides data into folders based on column values, while bucketing splits data into a fixed number of files within partitions for better sampling and joins.

What is the main benefit of partitioning data in Hadoop?

A Encrypting data for security

B Compressing data to save space

C Backing up data automatically

D Faster query by scanning only relevant partitions

Partitioning helps queries run faster by scanning only the partitions that match the query filter.

Which type of column is best suited for partitioning?

A Continuous numeric values like temperature

B Unique IDs like user IDs

C Categorical values like dates or regions

D Text descriptions

Categorical columns with limited distinct values are best for partitioning to create meaningful data groups.

What is a downside of having too many partitions?

A Queries become slower due to overhead

B Data gets deleted automatically

C Partitions merge into one big file

D Data becomes unsearchable

Too many small partitions cause overhead in managing files, slowing down queries.

Partitioning in Hadoop physically stores data in:

A Random locations

B Separate folders based on partition column values

C Encrypted blocks

D Single large file

Partitioning stores data in separate folders named after partition column values.

Which technique complements partitioning for better query performance?

A Bucketing

B Indexing

C Compression

D Replication

Bucketing splits data inside partitions into fixed files, improving joins and sampling.

Describe how partitioning helps improve query performance in Hadoop.

Think about how filtering works on partitioned data.

What are the risks of over-partitioning data and how can it affect Hadoop queries?

Too much splitting can cause problems too.