CRUD operations in HBase in Hadoop - Deep Dive

Overview - CRUD operations in HBase

What is it?

CRUD operations in HBase refer to the basic actions of Creating, Reading, Updating, and Deleting data in HBase tables. HBase is a distributed, scalable database built on top of Hadoop that stores data in a column-oriented way. These operations allow users to manage data efficiently in large datasets spread across many machines. Understanding CRUD in HBase helps you interact with big data stored in a fast and flexible manner.

Why it matters

Without CRUD operations, you cannot manage or manipulate data in HBase, making it impossible to build applications or analyze data stored there. CRUD operations solve the problem of handling huge amounts of data in a distributed system, allowing real-time access and updates. Without them, big data systems would be static and unusable for dynamic tasks like tracking user activity or updating records.

Where it fits

Before learning CRUD in HBase, you should understand basic Hadoop concepts and how HBase stores data in tables with rows and column families. After mastering CRUD, you can explore advanced topics like HBase filters, scans, and integration with MapReduce or Spark for big data processing.

Mental Model

Core Idea

CRUD operations in HBase are the fundamental ways to add, get, change, and remove data from a large, distributed table organized by rows and columns.

Think of it like...

Think of HBase like a giant library where each book is a row, and each chapter inside the book is a column family. CRUD operations are like adding new books, reading chapters, updating pages, or removing books from the library.

┌─────────────┐
│   HBase    │
│  Table     │
│────────────│
│ Row Key 1  │
│ ┌────────┐ │
│ │ColFam1 │ │
│ │ Col1   │ │
│ │ Col2   │ │
│ └────────┘ │
│ Row Key 2  │
│ ...        │
└────────────┘

Operations:
Create -> Add new row or column
Read   -> Get data by row key
Update -> Change existing data
Delete -> Remove row or column

Build-Up - 7 Steps

1

Foundation

Understanding HBase Table Structure

Concept: Learn how data is organized in HBase tables using row keys and column families.

HBase stores data in tables made of rows and column families. Each row has a unique key. Column families group related columns. Data is stored as cells identified by row key, column family, and column qualifier. This structure allows fast lookups by row key.

Result

You can visualize HBase data as a table with rows identified by keys and columns grouped in families.

Understanding the table structure is essential because all CRUD operations depend on knowing how to locate and organize data in HBase.

2

Foundation

Basic Create and Read Operations

Concept: Learn how to add new data and retrieve existing data using HBase commands or API.

To create data, you insert a new row with a unique key and specify column family and qualifier with values. To read data, you query by row key to get all or specific columns. For example, using HBase shell: Create: put 'table', 'row1', 'cf1:col1', 'value1' Read: get 'table', 'row1' This adds a value and retrieves it.

Result

Data is stored and can be retrieved by row key and column details.

Knowing how to create and read data is the foundation for interacting with HBase and enables you to verify data storage.

3

Intermediate

Updating Data in HBase

🤔Before reading on: Do you think updating data in HBase replaces the entire row or just specific columns? Commit to your answer.

Concept: Updates in HBase are done by putting new values for existing columns or adding new columns in a row.

To update data, you use the same 'put' command with the row key and column details. HBase overwrites the old value with the new one for that cell. For example: put 'table', 'row1', 'cf1:col1', 'new_value' This changes the value of 'col1' in 'row1'. HBase keeps versions by timestamp but the latest is returned by default.

Result

The specified cell value is updated without affecting other columns or rows.

Understanding that updates are cell-level and overwrite existing values helps avoid accidental data loss and enables precise data management.

4

Intermediate

Deleting Data from HBase

🤔Before reading on: Does deleting a row in HBase immediately remove all data or mark it for later removal? Commit to your answer.

Concept: Deletion in HBase marks data as deleted (tombstones) which are cleaned up later during compaction.

You can delete specific cells, entire columns, or whole rows. For example: delete 'table', 'row1', 'cf1:col1' # deletes one cell deleteall 'table', 'row1' # deletes entire row HBase does not immediately erase data but marks it. Actual removal happens during background cleanup.

Result

Data is marked deleted and becomes invisible to reads, but physical removal is deferred.

Knowing that deletion is logical first prevents confusion about why deleted data might still occupy storage temporarily.

5

Intermediate

Using HBase API for CRUD Operations

🤔Before reading on: Do you think HBase API operations are synchronous or asynchronous by default? Commit to your answer.

Concept: HBase provides Java API to perform CRUD operations programmatically with control over sync or async behavior.

Using HBase Java API, you create Put, Get, Delete objects and send them to the table. For example: Put put = new Put(Bytes.toBytes("row1")); put.addColumn(Bytes.toBytes("cf1"), Bytes.toBytes("col1"), Bytes.toBytes("value1")); table.put(put); Get get = new Get(Bytes.toBytes("row1")); Result result = table.get(get); This allows integration with applications.

Result

You can perform CRUD operations inside Java programs with fine control.

Using the API unlocks automation and integration, essential for real-world big data applications.

6

Advanced

Handling Versions and Timestamps in CRUD

🤔Before reading on: Does HBase keep multiple versions of a cell by default or only the latest? Commit to your answer.

Concept: HBase stores multiple versions of data cells identified by timestamps, allowing access to historical data.

Each cell in HBase can have multiple versions. When you put data, you can specify a timestamp or let HBase assign one. Reads can request specific versions. For example: Get get = new Get(Bytes.toBytes("row1")); get.readVersions(3); // get last 3 versions This feature supports auditing and rollback scenarios.

Result

You can access past values of data, not just the latest.

Understanding versions helps in designing systems that need history tracking or undo capabilities.

7

Expert

Optimizing CRUD with Batch and Atomic Operations

🤔Before reading on: Do you think batch operations in HBase improve performance by reducing network calls or do they add overhead? Commit to your answer.

Concept: Batching multiple CRUD operations reduces network overhead and improves throughput; atomic operations ensure consistency.

HBase supports batch operations where multiple puts, gets, or deletes are sent together: List batch = new ArrayList<>(); batch.add(put1); batch.add(delete1); table.batch(batch, results); Atomic operations like CheckAndPut ensure updates happen only if conditions match, preventing race conditions. This is critical in high-load production environments.

Result

CRUD operations become faster and safer under concurrent access.

Knowing how to batch and use atomic operations is key to building scalable, reliable big data applications.

Under the Hood

HBase stores data in HDFS as files called HFiles. When you perform a put, data is first written to a write-ahead log (WAL) for durability, then to an in-memory store called MemStore. Once MemStore fills, data is flushed to disk as an HFile. Reads merge data from MemStore and HFiles. Deletes create tombstones marking data as deleted, which are cleaned during compaction. This design ensures fast writes and consistent reads in a distributed environment.

Why designed this way?

HBase was designed to handle massive data with low latency on commodity hardware. Using WAL and MemStore balances durability and speed. The column-family model optimizes sparse data storage. Tombstones allow efficient deletes without immediate costly disk operations. Alternatives like relational databases couldn't scale horizontally as easily.

┌───────────────┐
│ Client CRUD   │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Write-Ahead   │
│ Log (WAL)     │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ MemStore      │
│ (In-memory)   │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ HFiles on HDFS│
└───────────────┘

Reads combine MemStore + HFiles
Deletes add tombstones
Compaction cleans tombstones and merges files

Myth Busters - 4 Common Misconceptions

Quick: Does deleting a row in HBase immediately free all disk space? Commit to yes or no.

Common Belief: Deleting a row instantly removes all its data from disk.

Reality: Deletion only marks data with tombstones; actual disk space is freed later during compaction.

Why it matters: Assuming immediate deletion can lead to confusion about storage usage and delays in reclaiming space.

Quick: When updating a cell, does HBase create a new version or overwrite the old one? Commit to your answer.

Common Belief: Updating a cell overwrites the old value and deletes previous versions.

Reality: HBase stores multiple versions of a cell; updates add new versions with timestamps.

Why it matters: Ignoring versions can cause unexpected data retrieval results and storage growth.

Quick: Are HBase CRUD operations synchronous by default? Commit to yes or no.

Common Belief: All CRUD operations block until completion before returning.

Reality: Some API calls can be asynchronous or buffered, affecting when data is actually written.

Why it matters: Misunderstanding sync behavior can cause bugs in data consistency and timing.

Quick: Does HBase support transactions across multiple rows? Commit to yes or no.

Common Belief: HBase supports multi-row transactions like relational databases.

Reality: HBase only supports atomic operations at single row level, not multi-row transactions.

Why it matters: Expecting multi-row transactions can lead to data consistency issues in distributed applications.

Expert Zone

1

HBase's use of timestamps for versions means that clock synchronization across clients can affect data visibility and ordering.

2

Batch operations reduce network overhead but require careful error handling because partial failures can occur.

3

Tombstones improve delete performance but can cause read amplification if compaction is delayed, impacting latency.

When NOT to use

HBase CRUD is not suitable when strong multi-row transactional guarantees are needed; in such cases, use distributed SQL databases like Google Spanner or Apache Phoenix on top of HBase. Also, for small datasets or simple key-value needs, lighter databases like RocksDB or Cassandra might be better.

Production Patterns

In production, CRUD operations are often combined with filters and scans to efficiently query subsets of data. Batch puts and deletes are used to optimize throughput. Atomic CheckAndPut operations enforce conditional updates. Monitoring compaction and tuning MemStore size are common to maintain performance.

Connections

Relational Database CRUD

Similar pattern of Create, Read, Update, Delete but with different storage and consistency models.

Understanding relational CRUD helps grasp the purpose of HBase CRUD, but HBase differs by being distributed and column-oriented.

Distributed Systems Consistency Models

HBase CRUD operations reflect eventual consistency and atomicity at row level in distributed storage.

Knowing distributed consistency concepts clarifies why HBase handles deletes with tombstones and limits transactions.

Version Control Systems

HBase cell versions are like commits in version control, storing history with timestamps.

Recognizing this connection helps understand how HBase manages multiple data versions and supports historical queries.

Common Pitfalls

#1 Trying to delete data and expecting immediate disk space recovery.

Wrong approach:

deleteall 'table', 'row1'
# Then checking disk space immediately expecting it to be freed

Correct approach:

deleteall 'table', 'row1'
# Understand that compaction will free space later

Root cause:

Misunderstanding that HBase uses tombstones and deferred cleanup.

#2 Updating a cell without specifying the correct column family or qualifier.

Wrong approach:

put 'table', 'row1', 'col1', 'value'
# Missing column family causes error or wrong data placement

Correct approach:

put 'table', 'row1', 'cf1:col1', 'value'
# Correctly specifying column family and qualifier

Root cause:

Confusing HBase's column family:qualifier syntax.

#3 Assuming batch operations are atomic across all rows.

Wrong approach:

table.batch([put1, put2, delete1], results)
# Expecting all succeed or all fail together

Correct approach:

Use CheckAndPut for atomicity on single rows; handle partial failures in batch manually

Root cause:

Misunderstanding batch operation guarantees.

Key Takeaways

CRUD operations in HBase allow you to create, read, update, and delete data in a distributed, column-oriented database.

Data is organized by row keys and column families, and operations target specific cells identified by these keys.

Updates overwrite cell values but keep older versions accessible by timestamp.

Deletes mark data with tombstones and actual removal happens later during compaction.

Batch and atomic operations optimize performance and consistency in real-world big data applications.