Data lake design patterns in Hadoop - Step-by-Step Execution

Concept Flow - Data lake design patterns

Raw Data Ingested

↓

Landing Zone: Store Raw Data

↓

Cleansing & Transformation

↓

Curated Zone: Clean Data

↓

Data Serving Layer: Analytics & BI

↓

Users Access Data

↓

Feedback & Monitoring

↩Back to Cleansing & Transformation (Iterate)

Data flows from raw ingestion to landing zone, then cleansed and transformed into curated data, finally served for analytics, with feedback loops for improvement.

Execution Sample

Hadoop

1. Ingest raw data into landing zone
2. Clean and transform data
3. Store clean data in curated zone
4. Serve data for analytics
5. Users query and analyze data

This sequence shows the main steps in a data lake design pattern from raw data ingestion to user analytics.

Execution Table

Step	Action	Data State	Storage Zone	Purpose
1	Ingest raw data	Unprocessed, original format	Landing Zone	Capture all incoming data as-is
2	Clean and transform	Filtered, structured, enriched	Processing Layer	Prepare data for analysis
3	Store clean data	Validated and organized	Curated Zone	Reliable data for users
4	Serve data	Ready for queries	Serving Layer	Support analytics and BI tools
5	User access	Data consumed	Serving Layer	Enable insights and decisions
6	Feedback & monitor	Identify issues or improvements	Monitoring System	Improve data quality and processes
7	Iterate cleansing	Refined data	Processing Layer	Continuous improvement loop

💡 Process repeats with feedback to improve data quality and usability

Variable Tracker

Data State	Start	After Step 1	After Step 2	After Step 3	After Step 4	After Step 5	After Step 6
Raw Data	None	Raw ingested	Raw ingested	Raw ingested	Raw ingested	Raw ingested	Raw ingested
Clean Data	None	None	Cleaned & transformed	Stored curated	Available for queries	Queried by users	Refined after feedback
User Access	None	None	None	None	Data served	Data consumed	Data consumed

Key Moments - 3 Insights

Why do we keep raw data in the landing zone instead of cleaning it immediately?

Keeping raw data preserves the original source, allowing reprocessing if needed. See execution_table step 1 where raw data is stored as-is before cleaning.

What is the difference between the curated zone and the serving layer?

The curated zone stores clean, validated data, while the serving layer prepares data specifically for fast queries and analytics. Refer to execution_table steps 3 and 4.

How does feedback improve the data lake?

Feedback identifies data quality issues or process gaps, triggering reprocessing to refine data. This is shown in execution_table steps 6 and 7.

Visual Quiz - 3 Questions

Look at the execution_table, at which step is data first cleaned and transformed?

A Step 2

B Step 3

C Step 1

D Step 4

💡 Hint

Check the 'Action' column for cleaning and transformation in execution_table

According to variable_tracker, what is the state of 'Clean Data' after Step 3?

A None

B Stored curated

C Cleaned & transformed

D Raw ingested

💡 Hint

Look at the 'Clean Data' row under 'After Step 3' in variable_tracker

If feedback is ignored, which step in execution_table would be skipped?

A Step 5

B Step 6

C Step 7

D Step 4

💡 Hint

Feedback triggers iteration shown in Step 7 in execution_table

Concept Snapshot

Data lake design patterns:
1. Ingest raw data into landing zone (store as-is)
2. Clean and transform data in processing layer
3. Store clean data in curated zone
4. Serve data for analytics in serving layer
5. Use feedback loops to improve data quality
Keep raw data for reprocessing and separate zones for clarity.

Full Transcript

Data lake design patterns organize data flow from raw ingestion to user analytics. First, raw data is ingested and stored in the landing zone without changes. Then, data is cleaned and transformed in the processing layer. Clean data is stored in the curated zone for reliability. The serving layer prepares data for fast queries and analytics. Users access data here to gain insights. Feedback and monitoring identify issues and trigger reprocessing to improve data quality continuously. This pattern helps manage large data sets efficiently and keeps original data safe for future use.

Practice

1. What is the main purpose of organizing data into zones like raw, processed, and curated in a data lake?

easy

A. To increase the size of the data lake

B. To keep data clean, easy to find, and ready for analysis

C. To make data harder to access

D. To reduce the number of users accessing data

Solution

1. Step 1: Understand the role of data zones

   Raw, processed, and curated zones help organize data by its state and quality.

2. Step 2: Identify the benefit of this organization

   This organization keeps data clean, easy to find, and ready for analysis.

3. Final Answer:

   To keep data clean, easy to find, and ready for analysis -> Option B

4. Quick Check:

   Data zones improve data usability = A [OK]

Hint: Think about why clean data helps analysis fast [OK]

Common Mistakes:

• Confusing size increase with organization benefits
• Assuming data zones restrict access
• Thinking zones reduce user numbers

2. Which of the following is the correct way to represent the 'raw' data zone in Hadoop folder structure?

easy

A. /data/curated/

B. /data/processed/

C. /data/raw/

D. /data/analytics/

Solution

1. Step 1: Recall standard folder naming for data zones

   The raw data zone is usually stored in a folder named 'raw' to indicate unprocessed data.

2. Step 2: Match folder names to zones

   /data/raw/ matches the raw data zone, while others represent different zones or purposes.

3. Final Answer:

   /data/raw/ -> Option C

4. Quick Check:

   Raw data folder = /data/raw/ [OK]

Hint: Raw data folder always named 'raw' [OK]

Common Mistakes:

• Choosing processed or curated folder names for raw data
• Using unrelated folder names like analytics
• Confusing folder names with data types

3. Given the following Hadoop folder structure:

/data/raw/sales.csv
/data/processed/sales_clean.csv
/data/curated/sales_summary.csv

Which file would most likely contain cleaned and transformed sales data ready for analysis?

medium

A. /data/processed/sales_clean.csv

B. /data/raw/sales_summary.csv

C. /data/curated/sales_summary.csv

D. /data/raw/sales.csv

Solution

1. Step 1: Understand the meaning of each zone

   Raw zone has original data, processed zone has cleaned/transformed data, curated zone has aggregated or summarized data.

2. Step 2: Identify the file with cleaned and transformed data

   /data/processed/sales_clean.csv indicates cleaned sales data ready for further use.

3. Final Answer:

   /data/processed/sales_clean.csv -> Option A

4. Quick Check:

   Processed zone = cleaned data = D [OK]

Hint: Processed folder means cleaned and transformed data [OK]

Common Mistakes:

• Choosing raw data file as cleaned data
• Confusing curated summary with cleaned data
• Selecting non-existent file paths

4. You have a data lake with folders: /data/raw/, /data/processed/, and /data/curated/. You accidentally saved uncleaned data in /data/curated/. What is the best fix?

medium

A. Move the uncleaned data to /data/raw/ and reprocess it

B. Leave the data as is in /data/curated/

C. Delete the /data/raw/ folder

D. Rename /data/curated/ to /data/raw/

Solution

1. Step 1: Identify the problem with data placement

   Uncleaned data should not be in the curated zone, which is for clean, ready data.

2. Step 2: Choose the correct fix

   Moving uncleaned data back to raw zone and reprocessing it ensures proper data flow and quality.

3. Final Answer:

   Move the uncleaned data to /data/raw/ and reprocess it -> Option A

4. Quick Check:

   Raw zone = uncleaned data fix = B [OK]

Hint: Unclean data belongs in raw zone, not curated [OK]

Common Mistakes:

• Ignoring data quality rules
• Deleting unrelated folders
• Renaming folders without moving data

5. You want to design a data lake in Hadoop that supports fast analytics and easy data governance. Which design pattern should you follow?

hard

A. Store all data in a single folder without zones

B. Keep only raw data and process it on-demand

C. Use random folder names for each data type

D. Organize data into raw, processed, and curated zones with clear folder structures

Solution

1. Step 1: Understand the goal of fast analytics and governance

   Fast analytics needs clean, organized data; governance needs clear structure and control.

2. Step 2: Identify the best design pattern

   Organizing data into raw, processed, and curated zones with clear folders supports both goals effectively.

3. Final Answer:

   Organize data into raw, processed, and curated zones with clear folder structures -> Option D

4. Quick Check:

   Zones + clear folders = fast analytics + governance = C [OK]

Hint: Zones and clear folders improve analytics and governance [OK]

Common Mistakes:

• Storing all data together causing confusion
• Processing data only on-demand slowing queries
• Using unclear folder names causing management issues