Data lake design patterns in Hadoop

Introduction

A data lake stores lots of raw data in one place. Design patterns help organize this data so it is easy to find and use.

When collecting data from many sources like apps, sensors, and databases.

When you want to keep data in its original form for future use.

When you need to support different types of data like text, images, and logs.

When you want to separate raw data from cleaned and processed data.

When you want to control who can access different parts of the data.

Syntax

Hadoop

Data Lake Design Patterns:

1. Raw Zone (Landing Zone): Store original data as-is.
2. Processed Zone (Cleansed Zone): Store cleaned and transformed data.
3. Curated Zone: Store data ready for analysis.
4. Metadata Layer: Store information about data (like source, format).
5. Access Layer: Manage how users and tools access data.

Use folders or tables to separate zones in Hadoop HDFS or Hive.

Zones help keep data organized and easy to manage.

Metadata helps users understand what data is available and how to use it.

Examples

Use separate folders in HDFS to organize data by its processing stage.

Hadoop

/data_lake/raw/  # Raw Zone folder for original files
/data_lake/processed/  # Processed Zone for cleaned data
/data_lake/curated/  # Curated Zone for analysis-ready data

Create Hive tables pointing to raw data files for easy querying.

Hadoop

CREATE EXTERNAL TABLE raw_events (
  event_id STRING,
  event_time STRING,
  event_data STRING
)
STORED AS PARQUET
LOCATION '/data_lake/raw/events/';

Curated tables hold cleaned and structured data ready for reports.

Hadoop

CREATE EXTERNAL TABLE curated_sales (
  sale_id STRING,
  sale_date DATE,
  amount DOUBLE
)
STORED AS PARQUET
LOCATION '/data_lake/curated/sales/';

Sample Program

This code reads raw JSON event data from the raw zone, cleans it by selecting columns and filtering, then saves it to the processed zone. Finally, it reads and shows the processed data.

Hadoop

from pyspark.sql import SparkSession

# Start Spark session
spark = SparkSession.builder.appName('DataLakeDesign').getOrCreate()

# Load raw data from HDFS
raw_df = spark.read.json('/data_lake/raw/events/')

# Show raw data
print('Raw Data:')
raw_df.show(3)

# Clean data: select needed columns and filter
processed_df = raw_df.select('event_id', 'event_time').filter(raw_df.event_id.isNotNull())

# Save processed data back to HDFS
processed_df.write.mode('overwrite').parquet('/data_lake/processed/events/')

# Load processed data
curated_df = spark.read.parquet('/data_lake/processed/events/')

# Show processed data
print('Processed Data:')
curated_df.show(3)

spark.stop()

Important Notes

Keep raw data unchanged to allow reprocessing if needed.

Use metadata to track data source, format, and update times.

Design zones to match your team's workflow and tools.

Summary

Data lake design patterns organize data into zones: raw, processed, and curated.

This helps keep data clean, easy to find, and ready for analysis.

Use folders and tables in Hadoop to separate and manage these zones.

Practice

1. What is the main purpose of organizing data into zones like raw, processed, and curated in a data lake?

easy

A. To increase the size of the data lake

B. To keep data clean, easy to find, and ready for analysis

C. To make data harder to access

D. To reduce the number of users accessing data

Solution

1. Step 1: Understand the role of data zones

   Raw, processed, and curated zones help organize data by its state and quality.

2. Step 2: Identify the benefit of this organization

   This organization keeps data clean, easy to find, and ready for analysis.

3. Final Answer:

   To keep data clean, easy to find, and ready for analysis -> Option B

4. Quick Check:

   Data zones improve data usability = A [OK]

Hint: Think about why clean data helps analysis fast [OK]

Common Mistakes:

• Confusing size increase with organization benefits
• Assuming data zones restrict access
• Thinking zones reduce user numbers

2. Which of the following is the correct way to represent the 'raw' data zone in Hadoop folder structure?

easy

A. /data/curated/

B. /data/processed/

C. /data/raw/

D. /data/analytics/

Solution

1. Step 1: Recall standard folder naming for data zones

   The raw data zone is usually stored in a folder named 'raw' to indicate unprocessed data.

2. Step 2: Match folder names to zones

   /data/raw/ matches the raw data zone, while others represent different zones or purposes.

3. Final Answer:

   /data/raw/ -> Option C

4. Quick Check:

   Raw data folder = /data/raw/ [OK]

Hint: Raw data folder always named 'raw' [OK]

Common Mistakes:

• Choosing processed or curated folder names for raw data
• Using unrelated folder names like analytics
• Confusing folder names with data types

3. Given the following Hadoop folder structure:

/data/raw/sales.csv
/data/processed/sales_clean.csv
/data/curated/sales_summary.csv

Which file would most likely contain cleaned and transformed sales data ready for analysis?

medium

A. /data/processed/sales_clean.csv

B. /data/raw/sales_summary.csv

C. /data/curated/sales_summary.csv

D. /data/raw/sales.csv

Solution

1. Step 1: Understand the meaning of each zone

   Raw zone has original data, processed zone has cleaned/transformed data, curated zone has aggregated or summarized data.

2. Step 2: Identify the file with cleaned and transformed data

   /data/processed/sales_clean.csv indicates cleaned sales data ready for further use.

3. Final Answer:

   /data/processed/sales_clean.csv -> Option A

4. Quick Check:

   Processed zone = cleaned data = D [OK]

Hint: Processed folder means cleaned and transformed data [OK]

Common Mistakes:

• Choosing raw data file as cleaned data
• Confusing curated summary with cleaned data
• Selecting non-existent file paths

4. You have a data lake with folders: /data/raw/, /data/processed/, and /data/curated/. You accidentally saved uncleaned data in /data/curated/. What is the best fix?

medium

A. Move the uncleaned data to /data/raw/ and reprocess it

B. Leave the data as is in /data/curated/

C. Delete the /data/raw/ folder

D. Rename /data/curated/ to /data/raw/

Solution

1. Step 1: Identify the problem with data placement

   Uncleaned data should not be in the curated zone, which is for clean, ready data.

2. Step 2: Choose the correct fix

   Moving uncleaned data back to raw zone and reprocessing it ensures proper data flow and quality.

3. Final Answer:

   Move the uncleaned data to /data/raw/ and reprocess it -> Option A

4. Quick Check:

   Raw zone = uncleaned data fix = B [OK]

Hint: Unclean data belongs in raw zone, not curated [OK]

Common Mistakes:

• Ignoring data quality rules
• Deleting unrelated folders
• Renaming folders without moving data

5. You want to design a data lake in Hadoop that supports fast analytics and easy data governance. Which design pattern should you follow?

hard

A. Store all data in a single folder without zones

B. Keep only raw data and process it on-demand

C. Use random folder names for each data type

D. Organize data into raw, processed, and curated zones with clear folder structures

Solution

1. Step 1: Understand the goal of fast analytics and governance

   Fast analytics needs clean, organized data; governance needs clear structure and control.

2. Step 2: Identify the best design pattern

   Organizing data into raw, processed, and curated zones with clear folders supports both goals effectively.

3. Final Answer:

   Organize data into raw, processed, and curated zones with clear folder structures -> Option D

4. Quick Check:

   Zones + clear folders = fast analytics + governance = C [OK]

Hint: Zones and clear folders improve analytics and governance [OK]

Common Mistakes:

• Storing all data together causing confusion
• Processing data only on-demand slowing queries
• Using unclear folder names causing management issues