Data lake design patterns in Hadoop - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to create a raw zone directory in HDFS for the data lake.

Hadoop

hdfs dfs -mkdir /data_lake/[1]

Drag options to blanks, or click blank then click option'

Aanalytics

Braw_zone

Cprocessed_data

Dbackup

Attempts:

3 left

💡 Hint

Common Mistakes

Using processed_data instead of raw_zone

Creating backup folder instead of raw zone

The raw zone is where data is first ingested in its original form.

2fill in blank

medium

Complete the code to move data from the raw zone to the curated zone in HDFS.

Hadoop

hdfs dfs -mv /data_lake/raw_zone/[1] /data_lake/curated_zone/

Drag options to blanks, or click blank then click option'

Atemp_data.csv

Bbackup_data.csv

Cold_data.csv

Dlog_data.csv

Attempts:

3 left

💡 Hint

Common Mistakes

Moving backup_data.csv instead of temp_data.csv

Using wrong source or destination paths

temp_data.csv is the file being moved from raw to curated zone for processing.

3fill in blank

hard

Fix the error in the Spark code to read data from the curated zone in the data lake.

Hadoop

df = spark.read.format('parquet').load('/data_lake/[1]/2024/06/01')

Drag options to blanks, or click blank then click option'

Acurated_zone

Btemp_zone

Carchive_zone

Draw_zone

Attempts:

3 left

💡 Hint

Common Mistakes

Reading from raw_zone instead of curated_zone

Using archive_zone which is for old data

Data should be read from the curated zone after processing, not raw or archive zones.

4fill in blank

hard

Fill both blanks to create a partitioned table in Hive for the data lake's curated data.

Hadoop

CREATE EXTERNAL TABLE curated_data (id INT, name STRING) PARTITIONED BY ([1] STRING, [2] STRING) STORED AS PARQUET LOCATION '/data_lake/curated_zone/';

Drag options to blanks, or click blank then click option'

Ayear

Bmonth

Cday

Dhour

Attempts:

3 left

💡 Hint

Common Mistakes

Using day or hour as first partition instead of year

Not partitioning by time fields

Partitioning by year and month helps organize data efficiently for queries.

5fill in blank

hard

Fill all three blanks to create a dictionary comprehension that maps file names to their sizes for files larger than 100MB in the raw zone.

Hadoop

file_sizes = { [1]: [2] for [1] in files if files[[1]] > 100 }

Drag options to blanks, or click blank then click option'

Afile

Bfiles[file]

Csize

Dfiles

Attempts:

3 left

💡 Hint

Common Mistakes

Using size as loop variable which is undefined

Using files instead of files[file] to get size

The comprehension iterates over file names, maps each file to its size, and filters sizes greater than 100MB.

Practice

1. What is the main purpose of organizing data into zones like raw, processed, and curated in a data lake?

easy

A. To increase the size of the data lake

B. To keep data clean, easy to find, and ready for analysis

C. To make data harder to access

D. To reduce the number of users accessing data

Solution

1. Step 1: Understand the role of data zones

   Raw, processed, and curated zones help organize data by its state and quality.

2. Step 2: Identify the benefit of this organization

   This organization keeps data clean, easy to find, and ready for analysis.

3. Final Answer:

   To keep data clean, easy to find, and ready for analysis -> Option B

4. Quick Check:

   Data zones improve data usability = A [OK]

Hint: Think about why clean data helps analysis fast [OK]

Common Mistakes:

• Confusing size increase with organization benefits
• Assuming data zones restrict access
• Thinking zones reduce user numbers

2. Which of the following is the correct way to represent the 'raw' data zone in Hadoop folder structure?

easy

A. /data/curated/

B. /data/processed/

C. /data/raw/

D. /data/analytics/

Solution

1. Step 1: Recall standard folder naming for data zones

   The raw data zone is usually stored in a folder named 'raw' to indicate unprocessed data.

2. Step 2: Match folder names to zones

   /data/raw/ matches the raw data zone, while others represent different zones or purposes.

3. Final Answer:

   /data/raw/ -> Option C

4. Quick Check:

   Raw data folder = /data/raw/ [OK]

Hint: Raw data folder always named 'raw' [OK]

Common Mistakes:

• Choosing processed or curated folder names for raw data
• Using unrelated folder names like analytics
• Confusing folder names with data types

3. Given the following Hadoop folder structure:

/data/raw/sales.csv
/data/processed/sales_clean.csv
/data/curated/sales_summary.csv

Which file would most likely contain cleaned and transformed sales data ready for analysis?

medium

A. /data/processed/sales_clean.csv

B. /data/raw/sales_summary.csv

C. /data/curated/sales_summary.csv

D. /data/raw/sales.csv

Solution

1. Step 1: Understand the meaning of each zone

   Raw zone has original data, processed zone has cleaned/transformed data, curated zone has aggregated or summarized data.

2. Step 2: Identify the file with cleaned and transformed data

   /data/processed/sales_clean.csv indicates cleaned sales data ready for further use.

3. Final Answer:

   /data/processed/sales_clean.csv -> Option A

4. Quick Check:

   Processed zone = cleaned data = D [OK]

Hint: Processed folder means cleaned and transformed data [OK]

Common Mistakes:

• Choosing raw data file as cleaned data
• Confusing curated summary with cleaned data
• Selecting non-existent file paths

4. You have a data lake with folders: /data/raw/, /data/processed/, and /data/curated/. You accidentally saved uncleaned data in /data/curated/. What is the best fix?

medium

A. Move the uncleaned data to /data/raw/ and reprocess it

B. Leave the data as is in /data/curated/

C. Delete the /data/raw/ folder

D. Rename /data/curated/ to /data/raw/

Solution

1. Step 1: Identify the problem with data placement

   Uncleaned data should not be in the curated zone, which is for clean, ready data.

2. Step 2: Choose the correct fix

   Moving uncleaned data back to raw zone and reprocessing it ensures proper data flow and quality.

3. Final Answer:

   Move the uncleaned data to /data/raw/ and reprocess it -> Option A

4. Quick Check:

   Raw zone = uncleaned data fix = B [OK]

Hint: Unclean data belongs in raw zone, not curated [OK]

Common Mistakes:

• Ignoring data quality rules
• Deleting unrelated folders
• Renaming folders without moving data

5. You want to design a data lake in Hadoop that supports fast analytics and easy data governance. Which design pattern should you follow?

hard

A. Store all data in a single folder without zones

B. Keep only raw data and process it on-demand

C. Use random folder names for each data type

D. Organize data into raw, processed, and curated zones with clear folder structures

Solution

1. Step 1: Understand the goal of fast analytics and governance

   Fast analytics needs clean, organized data; governance needs clear structure and control.

2. Step 2: Identify the best design pattern

   Organizing data into raw, processed, and curated zones with clear folders supports both goals effectively.

3. Final Answer:

   Organize data into raw, processed, and curated zones with clear folder structures -> Option D

4. Quick Check:

   Zones + clear folders = fast analytics + governance = C [OK]

Hint: Zones and clear folders improve analytics and governance [OK]

Common Mistakes:

• Storing all data together causing confusion
• Processing data only on-demand slowing queries
• Using unclear folder names causing management issues