HBase data model (column families) in Hadoop - Deep Dive

Overview - HBase data model (column families)

What is it?

HBase is a database designed to store very large amounts of data in a way that is fast to read and write. Its data model organizes data into tables, but unlike traditional databases, it groups columns into sets called column families. Each column family stores related data together on disk, which helps HBase manage and access data efficiently. This structure is key to how HBase handles big data in distributed systems.

Why it matters

Without column families, HBase would not be able to efficiently store and retrieve data at scale. Column families let HBase group related data physically, reducing the time and resources needed to access it. This means faster queries and better performance for big data applications like real-time analytics or large-scale web services. Without this, handling huge datasets would be slower and more costly.

Where it fits

Before learning about HBase column families, you should understand basic database concepts like tables, rows, and columns, as well as the idea of NoSQL databases. After mastering column families, you can explore HBase data operations, schema design, and performance tuning. This topic fits early in learning HBase architecture and data modeling.

Mental Model

Core Idea

Column families in HBase group related columns together physically to optimize storage and access.

Think of it like...

Think of a column family like a folder in a filing cabinet where you keep related documents together. Instead of scattering papers everywhere, you organize them by topic so you can find and manage them quickly.

┌─────────────┐
│   HBase     │
│   Table     │
│ ┌─────────┐ │
│ │ColFam A │ │
│ │ Col1    │ │
│ │ Col2    │ │
│ └─────────┘ │
│ ┌─────────┐ │
│ │ColFam B │ │
│ │ Col3    │ │
│ │ Col4    │ │
│ └─────────┘ │
└─────────────┘

Build-Up - 6 Steps

1

Foundation

Basics of HBase Tables and Rows

Concept: Introduce the basic structure of HBase tables and rows to set the stage for column families.

HBase stores data in tables, similar to spreadsheets. Each table has rows identified by a unique key. Unlike traditional tables, HBase rows can have many columns, and these columns can vary between rows. This flexible structure allows storing different data types and amounts per row.

Result

You understand that HBase tables are collections of rows with unique keys and flexible columns.

Understanding the flexible row and column structure is essential before learning how column families organize these columns.

2

Foundation

Introduction to Columns and Column Families

Concept: Explain what columns and column families are in HBase and how they differ from traditional databases.

In HBase, columns are grouped into column families. Each column family contains one or more columns that are stored together physically. This grouping is defined when creating the table and cannot be changed easily later. Columns outside these families do not exist; all columns belong to a family.

Result

You know that columns are not standalone but belong to fixed column families that group related data.

Recognizing that column families are fixed groups helps understand HBase's storage and retrieval efficiency.

3

Intermediate

Physical Storage of Column Families

🤔Before reading on: Do you think all columns in a row are stored together physically or separately by family? Commit to your answer.

Concept: Show how column families are stored separately on disk, affecting performance and data access.

Each column family is stored in its own file set called an HFile on disk. This means data in one family is physically separated from data in another. When you read or write data, HBase accesses only the relevant column family's files, making operations faster if you only need some families.

Result

You see that column families control physical data layout, impacting speed and resource use.

Knowing that column families separate data physically explains why careful family design improves performance.

4

Intermediate

Impact of Column Families on Performance

🤔Before reading on: Does adding more column families always improve performance? Commit to your answer.

Concept: Explore how the number and size of column families affect HBase's speed and resource consumption.

While column families help organize data, having too many can slow down HBase because each family requires separate storage and memory resources. Large families with many columns can also impact read/write speed. Balancing the number and size of families is key to good performance.

Result

You understand that more column families are not always better and that design affects efficiency.

Realizing the tradeoff between organization and resource use helps avoid common performance pitfalls.

5

Advanced

Schema Design with Column Families

🤔Before reading on: Should you put frequently accessed columns in the same family or separate families? Commit to your answer.

Concept: Teach how to design HBase schemas by grouping columns into families based on access patterns and data characteristics.

Good schema design groups columns accessed together into the same family to minimize disk reads. Columns updated together should also be in the same family to reduce write overhead. Rarely accessed or large columns can be placed in separate families to avoid slowing common queries.

Result

You can design column families that optimize read/write patterns and storage.

Understanding access patterns guides family grouping, which directly impacts application speed and cost.

6

Expert

Advanced Internals of Column Family Storage

🤔Before reading on: Do you think HBase compresses data per column family or per table? Commit to your answer.

Concept: Reveal how HBase applies compression, caching, and versioning at the column family level for efficiency.

HBase applies compression and caching settings per column family, not per table. Each family manages its own versions of data cells, allowing fine control over storage and retrieval. This design lets administrators tune families differently based on data type and usage, improving overall system performance.

Result

You learn that column families are the unit of storage optimization and version control in HBase.

Knowing that compression and caching happen per family explains why family design affects storage size and speed.

Under the Hood

HBase stores each column family in separate files called HFiles on distributed storage. When data is written, it goes to an in-memory store (MemStore) per family, then flushed to disk files. Reads access only the relevant family's files, reducing I/O. Each family manages its own versions and compression settings, enabling efficient storage and retrieval tailored to data characteristics.

Why designed this way?

Column families were designed to group related data physically to optimize access patterns common in big data workloads. Separating data by family allows HBase to minimize disk reads and writes, apply different compression, and manage versions independently. Alternatives like storing all columns together would reduce flexibility and performance at scale.

┌───────────────┐
│   HBase Table │
├───────────────┤
│ Row Key       │
│ ┌───────────┐ │
│ │ColFam A   │ │
│ │ MemStore  │ │
│ │  HFile(s) │ │
│ └───────────┘ │
│ ┌───────────┐ │
│ │ColFam B   │ │
│ │ MemStore  │ │
│ │  HFile(s) │ │
│ └───────────┘ │
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do you think you can add or remove column families anytime without downtime? Commit to yes or no.

Common Belief: You can add or remove column families from an HBase table anytime without any impact.

Reality: Column families must be defined when creating the table and cannot be removed later; adding families requires table modification and may cause downtime or complexity.

Why it matters: Misunderstanding this leads to poor schema design upfront and costly changes or downtime later.

Quick: Do you think all columns in HBase are stored together physically? Commit to yes or no.

Common Belief: All columns in a row are stored together physically on disk.

Reality: Columns are stored separately by their column family, not all together, affecting how data is accessed and stored.

Why it matters: Assuming all columns are stored together can cause inefficient schema design and slow queries.

Quick: Does having more column families always improve performance? Commit to yes or no.

Common Belief: More column families always improve data organization and performance.

Reality: Too many column families increase resource use and can degrade performance due to overhead in managing each family separately.

Why it matters: Overusing column families can cause slower reads/writes and higher memory consumption.

Quick: Is compression applied at the table level in HBase? Commit to yes or no.

Common Belief: Compression settings apply to the entire HBase table uniformly.

Reality: Compression is configured per column family, allowing different settings for different data groups.

Why it matters: Ignoring this can lead to inefficient storage and missed optimization opportunities.

Expert Zone

1

Column families are the smallest unit for HBase's storage optimizations like compression and caching, so their design directly affects resource use.

2

Versioning and TTL (time-to-live) settings are applied per column family, enabling fine-grained data lifecycle management.

3

The physical separation of column families means that cross-family scans can be slower, so schema design must consider query patterns carefully.

When NOT to use

Avoid using many small column families for unrelated data as it increases overhead; instead, use fewer families with logically grouped columns. For relational data with strict schema and joins, use traditional RDBMS instead of HBase.

Production Patterns

In production, teams design column families based on access frequency, grouping hot data in one family and cold or large data in another. Compression and caching settings are tuned per family to balance speed and storage. Monitoring tools track family-level metrics to optimize performance.

Connections

Relational Database Normalization

Opposite approach; normalization splits data into tables, while HBase column families group related columns together.

Understanding normalization helps appreciate why HBase groups columns physically to optimize big data access rather than enforcing strict schema.

Filesystem Directory Structure

Similar pattern; column families act like directories grouping files (columns) for organized storage.

Knowing how filesystems organize data helps understand HBase's physical grouping of columns for efficient retrieval.

Cache Partitioning in Computer Architecture

Builds-on concept; just as caches are partitioned to optimize access, column families partition data storage to speed up reads and writes.

Recognizing this connection clarifies why separating data into families reduces unnecessary data scanning and improves performance.

Common Pitfalls

#1 Creating too many column families thinking it improves organization.

Wrong approach:

create 'mytable', {NAME => 'cf1'}, {NAME => 'cf2'}, {NAME => 'cf3'}, {NAME => 'cf4'}, {NAME => 'cf5'}, {NAME => 'cf6'}

Correct approach:

create 'mytable', {NAME => 'cf1'}, {NAME => 'cf2'}

Root cause:

Misunderstanding that each column family adds overhead; too many families increase resource use and slow performance.

#2 Trying to add a new column family after table creation without proper steps.

Wrong approach:

alter 'mytable', {NAME => 'newcf'} # expecting instant addition

Correct approach:

disable 'mytable'; alter 'mytable', {NAME => 'newcf'}; enable 'mytable'

Root cause:

Not knowing that table must be disabled before altering column families causes errors or downtime.

#3 Placing frequently accessed and rarely accessed columns in the same family.

Wrong approach:

create 'mytable', {NAME => 'cf', VERSIONS => 3, COMPRESSION => 'SNAPPY'} # all columns in one family

Correct approach:

create 'mytable', {NAME => 'hotdata', VERSIONS => 3, COMPRESSION => 'SNAPPY'}, {NAME => 'colddate', VERSIONS => 1, COMPRESSION => 'GZIP'}

Root cause:

Ignoring access patterns leads to inefficient reads and wasted resources.

Key Takeaways

HBase column families group related columns physically to optimize storage and access.

Each column family is stored separately on disk, allowing fine-tuned compression, caching, and versioning.

Designing column families based on data access patterns improves performance and resource use.

Too many column families increase overhead and can degrade system speed.

Column families are fixed at table creation and require careful planning to avoid costly changes.