Why data lake architecture centralizes data in Hadoop - See It in Action

Why Data Lake Architecture Centralizes Data

📖 Scenario: Imagine a large company that collects data from many sources like sales, customer feedback, and website logs. They want to keep all this data in one place so everyone can use it easily.

🎯 Goal: You will create a simple example to show how data from different sources can be stored together in a data lake architecture using Python dictionaries. This will help you understand why data lakes centralize data.

📋 What You'll Learn

Create a dictionary called data_sources with three keys: 'sales', 'feedback', and 'logs'.

Each key should have a list of sample data strings as its value.

Create a variable called central_data_lake and set it to an empty list.

Use a for loop with variables source and records to iterate over data_sources.items().

Inside the loop, extend central_data_lake with the records.

Print the central_data_lake to show all data combined.

💡 Why This Matters

🌍 Real World

Companies collect data from many places like sales, customer feedback, and logs. A data lake stores all this data in one place so teams can analyze it easily.

💼 Career

Understanding data lake architecture helps you work with big data platforms like Hadoop and prepare data for analysis or machine learning.

1

Create the data sources dictionary

Create a dictionary called data_sources with these exact keys and values: 'sales' with ["sale1", "sale2"], 'feedback' with ["good", "bad"], and 'logs' with ["log1", "log2"].

Hadoop

# Create the data_sources dictionary with sales, feedback, and logs
# Your code here

Hint

Use curly braces to create a dictionary. Each key should have a list of strings as its value.

2

Create the central data lake list

Create a variable called central_data_lake and set it to an empty list [].

Hadoop

data_sources = {
    'sales': ["sale1", "sale2"],
    'feedback': ["good", "bad"],
    'logs': ["log1", "log2"]
}
# Create an empty list called central_data_lake
# Your code here

Hint

Use square brackets to create an empty list.

3

Combine all data into the central data lake

Use a for loop with variables source and records to iterate over data_sources.items(). Inside the loop, extend central_data_lake with the records.

Hadoop

data_sources = {
    'sales': ["sale1", "sale2"],
    'feedback': ["good", "bad"],
    'logs': ["log1", "log2"]
}
central_data_lake = []
# Use a for loop to add all records to central_data_lake
# Your code here

Hint

Use for source, records in data_sources.items(): and inside the loop use central_data_lake.extend(records).

4

Print the combined data

Write print(central_data_lake) to display all data combined in the central data lake.

Hadoop

data_sources = {
    'sales': ["sale1", "sale2"],
    'feedback': ["good", "bad"],
    'logs': ["log1", "log2"]
}
central_data_lake = []
for source, records in data_sources.items():
    central_data_lake.extend(records)
# Print the central_data_lake
# Your code here

Hint

Use the print function to show the combined list.

Practice

1. Why does data lake architecture centralize data in one place?

easy

A. To limit data access to only one user

B. To separate data by type in different locations

C. To store all types of data together for easy access and analysis

D. To reduce the amount of data stored

Solution

1. Step 1: Understand the purpose of data lake centralization

   Data lakes centralize data to keep all types of data in one place, making it easier to manage and analyze.

2. Step 2: Compare options with this purpose

   Only To store all types of data together for easy access and analysis correctly states that centralization is for storing all data types together for easy access and analysis.

3. Final Answer:

   To store all types of data together for easy access and analysis -> Option C

4. Quick Check:

   Centralization means one place for all data = A [OK]

Hint: Centralized means all data in one place for easy use [OK]

Common Mistakes:

• Thinking data lakes separate data by type
• Believing centralization limits access
• Assuming centralization reduces data volume

2. Which of the following is a correct statement about data lakes in Hadoop?

easy

A. Data lakes centralize all data types using Hadoop HDFS

B. Data lakes separate data into many small isolated files

C. Data lakes require data to be pre-processed before storage

D. Data lakes store only structured data in Hadoop HDFS

Solution

1. Step 1: Recall Hadoop HDFS role in data lakes

   Hadoop HDFS stores all types of data (structured, unstructured) centrally in data lakes.

2. Step 2: Evaluate each option

   Data lakes centralize all data types using Hadoop HDFS correctly states that data lakes centralize all data types using Hadoop HDFS. Other options are incorrect because data lakes store more than structured data, do not require pre-processing before storage, and do not isolate data into many small files.

3. Final Answer:

   Data lakes centralize all data types using Hadoop HDFS -> Option A

4. Quick Check:

   Hadoop HDFS stores all data types centrally = D [OK]

Hint: Hadoop HDFS stores all data types centrally in data lakes [OK]

Common Mistakes:

• Thinking data lakes store only structured data
• Believing data must be pre-processed before storage
• Assuming data lakes isolate data into many files

3. Given this code snippet using Hadoop HDFS API to list files in a data lake folder:

from pyarrow import fs
hdfs = fs.HadoopFileSystem('default')
files = hdfs.get_file_info(fs.FileSelector('/data_lake'))
print([file.path for file in files])

What will this code output?

medium

A. An empty list because '/data_lake' is empty

B. A list of file paths stored in the '/data_lake' directory

C. An error because 'get_file_info' does not exist

D. A list of file sizes instead of paths

Solution

1. Step 1: Understand the code functionality

   The code uses HadoopFileSystem to access HDFS, then lists file info in '/data_lake' directory, and prints their paths.

2. Step 2: Predict the output

   The output will be a list of file paths inside '/data_lake'. If the directory has files, their paths will be printed as a list.

3. Final Answer:

   A list of file paths stored in the '/data_lake' directory -> Option B

4. Quick Check:

   Listing files returns paths list = C [OK]

Hint: Listing files returns their paths as a list [OK]

Common Mistakes:

• Assuming method does not exist causing error
• Expecting file sizes instead of paths
• Thinking output is empty without checking directory contents

4. This code tries to read a file from a Hadoop data lake but causes an error:

from pyarrow import fs
hdfs = fs.HadoopFileSystem('default')
with hdfs.open_input_stream('/data_lake/file.txt') as f:
    data = f.read()
print(data)

What is the likely cause of the error?

medium

A. The 'read()' method is missing parentheses

B. The method 'open_input_stream' is not valid for HadoopFileSystem

C. The file is too large to read at once

D. The file path '/data_lake/file.txt' does not exist in HDFS

Solution

1. Step 1: Check method correctness

   'open_input_stream' is a valid method to open files in HadoopFileSystem, and 'read()' is correctly called.

2. Step 2: Identify common error cause

   Most errors here happen if the file path does not exist in HDFS, causing a file not found error.

3. Final Answer:

   The file path '/data_lake/file.txt' does not exist in HDFS -> Option D

4. Quick Check:

   File not found causes error = A [OK]

Hint: File not found is most common read error [OK]

Common Mistakes:

• Thinking method 'open_input_stream' is invalid
• Assuming file size causes immediate error
• Forgetting to use parentheses with read()

5. You want to design a data lake architecture using Hadoop that centralizes data from multiple sources including CSV files, images, and logs. Which approach best supports this goal?

hard

A. Store all data types in Hadoop HDFS without converting formats, enabling flexible access

B. Convert all data to a single format before storing in separate folders in HDFS

C. Store only structured data in HDFS and keep unstructured data outside the lake

D. Use multiple storage systems for each data type and link them with Hadoop

Solution

1. Step 1: Understand data lake centralization goal

   Data lakes centralize all data types in one place to allow flexible storage and easy access.

2. Step 2: Evaluate storage approaches

   Store all data types in Hadoop HDFS without converting formats, enabling flexible access stores all data types in Hadoop HDFS without forcing format conversion, supporting centralization and flexibility. Other options separate data or limit types, which breaks centralization.

3. Final Answer:

   Store all data types in Hadoop HDFS without converting formats, enabling flexible access -> Option A

4. Quick Check:

   Centralize all data types in one system = B [OK]

Hint: Keep all data types together in HDFS for true centralization [OK]

Common Mistakes:

• Converting all data to one format losing flexibility
• Separating unstructured data outside the lake
• Using multiple storage systems breaking centralization