Data lake design patterns in Hadoop - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is a Data Lake in simple terms?

A Data Lake is a big storage place where you keep all kinds of data, like raw files, logs, and databases, without organizing it first. It's like a big lake where all water flows in without filtering.

beginner

What is the 'Raw Zone' in a Data Lake design pattern?

The Raw Zone is where data is stored exactly as it arrives, without any changes. Think of it as a storage room where you keep all your stuff before sorting it.

beginner

Explain the 'Cleansed Zone' in Data Lake design.

The Cleansed Zone holds data that has been cleaned and organized. It's like a kitchen where you prepare and tidy up ingredients before cooking.

intermediate

What is the purpose of the 'Curated Zone' in a Data Lake?

The Curated Zone contains data that is ready for analysis and reporting. It's like a neatly arranged library where you can easily find and use books.

intermediate

Why use a layered design pattern in Data Lakes?

Layered design helps keep data organized by stages: raw, cleansed, and curated. This makes it easier to manage, find, and trust data, just like sorting mail into inbox, processing, and archive.

Which zone in a Data Lake stores data exactly as it arrives?

A Archive Zone

B Cleansed Zone

C Curated Zone

D Raw Zone

The Raw Zone stores data in its original form without changes.

What is the main purpose of the Curated Zone?

A Prepare data for analysis

B Clean data

C Store raw data

D Delete old data

The Curated Zone holds data ready for analysis and reporting.

Why is layered design important in Data Lakes?

A To organize data by processing stages

B To make data messy

C To delete data faster

D To store only one type of data

Layered design organizes data by stages like raw, cleansed, and curated.

Which Hadoop tool is commonly used to store Data Lake data?

A Hive

B HDFS

C Spark

D YARN

HDFS is the Hadoop Distributed File System used to store large data sets.

What happens in the Cleansed Zone?

A Data is deleted

B Data is stored raw

C Data is cleaned and organized

D Data is archived

The Cleansed Zone contains data that has been cleaned and organized.

Describe the main zones in a Data Lake design pattern and their roles.

Think about how data flows from arrival to analysis.

Explain why a layered approach is useful when designing a Data Lake.

Consider how sorting helps in daily life.

Practice

1. What is the main purpose of organizing data into zones like raw, processed, and curated in a data lake?

easy

A. To increase the size of the data lake

B. To keep data clean, easy to find, and ready for analysis

C. To make data harder to access

D. To reduce the number of users accessing data

Solution

1. Step 1: Understand the role of data zones

   Raw, processed, and curated zones help organize data by its state and quality.

2. Step 2: Identify the benefit of this organization

   This organization keeps data clean, easy to find, and ready for analysis.

3. Final Answer:

   To keep data clean, easy to find, and ready for analysis -> Option B

4. Quick Check:

   Data zones improve data usability = A [OK]

Hint: Think about why clean data helps analysis fast [OK]

Common Mistakes:

• Confusing size increase with organization benefits
• Assuming data zones restrict access
• Thinking zones reduce user numbers

2. Which of the following is the correct way to represent the 'raw' data zone in Hadoop folder structure?

easy

A. /data/curated/

B. /data/processed/

C. /data/raw/

D. /data/analytics/

Solution

1. Step 1: Recall standard folder naming for data zones

   The raw data zone is usually stored in a folder named 'raw' to indicate unprocessed data.

2. Step 2: Match folder names to zones

   /data/raw/ matches the raw data zone, while others represent different zones or purposes.

3. Final Answer:

   /data/raw/ -> Option C

4. Quick Check:

   Raw data folder = /data/raw/ [OK]

Hint: Raw data folder always named 'raw' [OK]

Common Mistakes:

• Choosing processed or curated folder names for raw data
• Using unrelated folder names like analytics
• Confusing folder names with data types

3. Given the following Hadoop folder structure:

/data/raw/sales.csv
/data/processed/sales_clean.csv
/data/curated/sales_summary.csv

Which file would most likely contain cleaned and transformed sales data ready for analysis?

medium

A. /data/processed/sales_clean.csv

B. /data/raw/sales_summary.csv

C. /data/curated/sales_summary.csv

D. /data/raw/sales.csv

Solution

1. Step 1: Understand the meaning of each zone

   Raw zone has original data, processed zone has cleaned/transformed data, curated zone has aggregated or summarized data.

2. Step 2: Identify the file with cleaned and transformed data

   /data/processed/sales_clean.csv indicates cleaned sales data ready for further use.

3. Final Answer:

   /data/processed/sales_clean.csv -> Option A

4. Quick Check:

   Processed zone = cleaned data = D [OK]

Hint: Processed folder means cleaned and transformed data [OK]

Common Mistakes:

• Choosing raw data file as cleaned data
• Confusing curated summary with cleaned data
• Selecting non-existent file paths

4. You have a data lake with folders: /data/raw/, /data/processed/, and /data/curated/. You accidentally saved uncleaned data in /data/curated/. What is the best fix?

medium

A. Move the uncleaned data to /data/raw/ and reprocess it

B. Leave the data as is in /data/curated/

C. Delete the /data/raw/ folder

D. Rename /data/curated/ to /data/raw/

Solution

1. Step 1: Identify the problem with data placement

   Uncleaned data should not be in the curated zone, which is for clean, ready data.

2. Step 2: Choose the correct fix

   Moving uncleaned data back to raw zone and reprocessing it ensures proper data flow and quality.

3. Final Answer:

   Move the uncleaned data to /data/raw/ and reprocess it -> Option A

4. Quick Check:

   Raw zone = uncleaned data fix = B [OK]

Hint: Unclean data belongs in raw zone, not curated [OK]

Common Mistakes:

• Ignoring data quality rules
• Deleting unrelated folders
• Renaming folders without moving data

5. You want to design a data lake in Hadoop that supports fast analytics and easy data governance. Which design pattern should you follow?

hard

A. Store all data in a single folder without zones

B. Keep only raw data and process it on-demand

C. Use random folder names for each data type

D. Organize data into raw, processed, and curated zones with clear folder structures

Solution

1. Step 1: Understand the goal of fast analytics and governance

   Fast analytics needs clean, organized data; governance needs clear structure and control.

2. Step 2: Identify the best design pattern

   Organizing data into raw, processed, and curated zones with clear folders supports both goals effectively.

3. Final Answer:

   Organize data into raw, processed, and curated zones with clear folder structures -> Option D

4. Quick Check:

   Zones + clear folders = fast analytics + governance = C [OK]

Hint: Zones and clear folders improve analytics and governance [OK]

Common Mistakes:

• Storing all data together causing confusion
• Processing data only on-demand slowing queries
• Using unclear folder names causing management issues