Data lake design patterns in Hadoop - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding the Zone Architecture in Data Lakes

In a typical data lake design, data is organized into different zones such as raw, cleansed, and curated. What is the primary purpose of the raw zone?

A Store metadata and data catalog information.

B Store only cleaned and validated data ready for business use.

C Store aggregated and summarized data for reporting.

D Store data exactly as ingested without any transformation or filtering.

💡 Hint

Think about where data first lands in the data lake before any processing.

The raw zone holds data exactly as it arrives from source systems. It is unprocessed and serves as the foundation for further cleansing and transformation.

❓ data_output

intermediate

Data Partitioning Effect on Query Performance

Given a Hadoop data lake storing logs partitioned by date, what is the expected effect on query performance when filtering by a specific date?

A Query performance is unaffected by partitioning.

B Query performance worsens because all partitions must be scanned.

C Query performance improves because only the relevant partition is scanned.

D Query performance depends only on file size, not partitioning.

💡 Hint

Consider how partition pruning works in Hadoop query engines.

Partitioning data by date allows query engines to scan only the relevant partition, reducing data scanned and improving performance.

🔧 Debug

advanced

Identifying the Cause of Data Duplication in a Data Lake

A data lake ingestion pipeline writes data daily to the curated zone. After some time, duplicate records appear in the curated data. Which of the following is the most likely cause?

A The ingestion job appends data without checking for existing records.

B The raw zone data is corrupted causing duplicates.

C Partitioning by date was not applied in the raw zone.

D The data lake storage is running out of space.

💡 Hint

Think about how data is merged or appended during ingestion.

If the ingestion job appends data daily without deduplication or overwrite logic, duplicates accumulate in the curated zone.

🚀 Application

advanced

Choosing the Best Data Lake Design Pattern for Streaming Data

You need to design a data lake solution for real-time streaming data ingestion and analytics. Which design pattern is most suitable?

A Using a traditional data warehouse instead of a data lake.

B Lambda architecture combining batch and speed layers.

C Storing all data in the raw zone without transformation.

D Only batch processing with nightly ingestion jobs.

💡 Hint

Consider architectures that handle both real-time and batch data.

Lambda architecture supports both real-time streaming (speed layer) and batch processing (batch layer), making it ideal for streaming data analytics.

❓ Predict Output

expert

Output of Hive Query on Partitioned Data

Consider a Hive table partitioned by year and month. What is the output of the following query?

SELECT year, month, COUNT(*) AS cnt FROM sales WHERE year = 2023 AND month = 5 GROUP BY year, month;

Assume the table has 1000 records for May 2023 and 5000 records for other months.

A [{'year': 2023, 'month': 5, 'cnt': 1000}]

B [{'year': 2023, 'month': 5, 'cnt': 6000}]

C [{'year': 2023, 'month': 5, 'cnt': 5000}]

D Empty result set

💡 Hint

Focus on the filter conditions and grouping in the query.

The query filters data for year 2023 and month 5, then counts records. Since there are 1000 records for May 2023, the count is 1000.

Practice

1. What is the main purpose of organizing data into zones like raw, processed, and curated in a data lake?

easy

A. To increase the size of the data lake

B. To keep data clean, easy to find, and ready for analysis

C. To make data harder to access

D. To reduce the number of users accessing data

Solution

1. Step 1: Understand the role of data zones

   Raw, processed, and curated zones help organize data by its state and quality.

2. Step 2: Identify the benefit of this organization

   This organization keeps data clean, easy to find, and ready for analysis.

3. Final Answer:

   To keep data clean, easy to find, and ready for analysis -> Option B

4. Quick Check:

   Data zones improve data usability = A [OK]

Hint: Think about why clean data helps analysis fast [OK]

Common Mistakes:

• Confusing size increase with organization benefits
• Assuming data zones restrict access
• Thinking zones reduce user numbers

2. Which of the following is the correct way to represent the 'raw' data zone in Hadoop folder structure?

easy

A. /data/curated/

B. /data/processed/

C. /data/raw/

D. /data/analytics/

Solution

1. Step 1: Recall standard folder naming for data zones

   The raw data zone is usually stored in a folder named 'raw' to indicate unprocessed data.

2. Step 2: Match folder names to zones

   /data/raw/ matches the raw data zone, while others represent different zones or purposes.

3. Final Answer:

   /data/raw/ -> Option C

4. Quick Check:

   Raw data folder = /data/raw/ [OK]

Hint: Raw data folder always named 'raw' [OK]

Common Mistakes:

• Choosing processed or curated folder names for raw data
• Using unrelated folder names like analytics
• Confusing folder names with data types

3. Given the following Hadoop folder structure:

/data/raw/sales.csv
/data/processed/sales_clean.csv
/data/curated/sales_summary.csv

Which file would most likely contain cleaned and transformed sales data ready for analysis?

medium

A. /data/processed/sales_clean.csv

B. /data/raw/sales_summary.csv

C. /data/curated/sales_summary.csv

D. /data/raw/sales.csv

Solution

1. Step 1: Understand the meaning of each zone

   Raw zone has original data, processed zone has cleaned/transformed data, curated zone has aggregated or summarized data.

2. Step 2: Identify the file with cleaned and transformed data

   /data/processed/sales_clean.csv indicates cleaned sales data ready for further use.

3. Final Answer:

   /data/processed/sales_clean.csv -> Option A

4. Quick Check:

   Processed zone = cleaned data = D [OK]

Hint: Processed folder means cleaned and transformed data [OK]

Common Mistakes:

• Choosing raw data file as cleaned data
• Confusing curated summary with cleaned data
• Selecting non-existent file paths

4. You have a data lake with folders: /data/raw/, /data/processed/, and /data/curated/. You accidentally saved uncleaned data in /data/curated/. What is the best fix?

medium

A. Move the uncleaned data to /data/raw/ and reprocess it

B. Leave the data as is in /data/curated/

C. Delete the /data/raw/ folder

D. Rename /data/curated/ to /data/raw/

Solution

1. Step 1: Identify the problem with data placement

   Uncleaned data should not be in the curated zone, which is for clean, ready data.

2. Step 2: Choose the correct fix

   Moving uncleaned data back to raw zone and reprocessing it ensures proper data flow and quality.

3. Final Answer:

   Move the uncleaned data to /data/raw/ and reprocess it -> Option A

4. Quick Check:

   Raw zone = uncleaned data fix = B [OK]

Hint: Unclean data belongs in raw zone, not curated [OK]

Common Mistakes:

• Ignoring data quality rules
• Deleting unrelated folders
• Renaming folders without moving data

5. You want to design a data lake in Hadoop that supports fast analytics and easy data governance. Which design pattern should you follow?

hard

A. Store all data in a single folder without zones

B. Keep only raw data and process it on-demand

C. Use random folder names for each data type

D. Organize data into raw, processed, and curated zones with clear folder structures

Solution

1. Step 1: Understand the goal of fast analytics and governance

   Fast analytics needs clean, organized data; governance needs clear structure and control.

2. Step 2: Identify the best design pattern

   Organizing data into raw, processed, and curated zones with clear folders supports both goals effectively.

3. Final Answer:

   Organize data into raw, processed, and curated zones with clear folder structures -> Option D

4. Quick Check:

   Zones + clear folders = fast analytics + governance = C [OK]

Hint: Zones and clear folders improve analytics and governance [OK]

Common Mistakes:

• Storing all data together causing confusion
• Processing data only on-demand slowing queries
• Using unclear folder names causing management issues