Why ingestion pipelines feed the data lake in Hadoop - Challenge Your Understanding

Challenge - 5 Problems

🧠 Conceptual

intermediate

Purpose of Ingestion Pipelines in Data Lakes

Why do ingestion pipelines feed data lakes in big data systems?

A To convert all data into a single format before storing it

B To immediately clean and transform data before storing it in a database

C To delete old data from the data lake to save storage space

D To collect and store raw data from multiple sources for flexible analysis later

💡 Hint

Think about why raw data storage is useful for future analysis.

Ingestion pipelines feed data lakes to gather raw data from many sources. This raw data is stored as-is, allowing flexible and varied analysis later without losing original details.

🧠 Conceptual

intermediate

Data Lake vs Data Warehouse Feeding

What is a key reason ingestion pipelines feed data lakes instead of data warehouses?

A Data lakes are slower to access than data warehouses

B Data lakes only store images, while data warehouses store text data

C Data lakes store raw data, while data warehouses require cleaned and structured data

D Data lakes automatically analyze data, data warehouses do not

💡 Hint

Consider the difference in data format requirements between lakes and warehouses.

Data lakes accept raw, unstructured data from ingestion pipelines, while data warehouses need data to be cleaned and structured before loading.

❓ data_output

advanced

Output of a Hadoop Ingestion Pipeline

Given a Hadoop ingestion pipeline that collects JSON logs from multiple servers and stores them in HDFS, what is the typical output format stored in the data lake?

Hadoop

import json
logs = ['{"user":"alice","action":"login"}', '{"user":"bob","action":"logout"}']
parsed_logs = [json.loads(log) for log in logs]
# What is stored in HDFS?

A A single CSV file with columns user and action

B [{"user": "alice", "action": "login"}, {"user": "bob", "action": "logout"}] as raw JSON files

C A SQL database table with user and action columns

D An Excel spreadsheet with user and action data

💡 Hint

Think about how Hadoop stores raw data from ingestion pipelines.

Hadoop ingestion pipelines typically store raw JSON files directly in HDFS, preserving original data format for flexible processing later.

🔧 Debug

advanced

Identify the Error in a Hadoop Ingestion Script

What error will this Hadoop ingestion Python script raise?

Hadoop

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('Ingest').getOrCreate()
data = spark.read.json('hdfs://logs/*.json')
data.write.csv('hdfs://output/logs.csv')
spark.stop()

A TypeError because write.csv expects a directory, not a file path

B AnalysisException because output path 'hdfs://output/logs.csv' already exists

C FileNotFoundError because 'hdfs://logs/*.json' path is invalid

D No error; script runs successfully and writes CSV files

💡 Hint

Check the expected argument type for write.csv in Spark.

In Spark, write.csv expects a directory path, not a file path. Giving a file path causes an AnalysisException, not a TypeError.

🚀 Application

expert

Designing an Efficient Ingestion Pipeline for a Data Lake

You need to design an ingestion pipeline that feeds a data lake with streaming sensor data. Which approach best ensures data is stored quickly and can be processed later in different ways?

A Use a streaming system like Apache Kafka to collect raw sensor data and store it directly in the data lake as JSON files

B Transform sensor data into a fixed schema and load it immediately into a relational database

C Aggregate sensor data in real-time and store only summary statistics in the data lake

D Store sensor data temporarily in local files and batch upload to the data lake once a day

💡 Hint

Consider speed and flexibility of raw data storage for future analysis.

Using a streaming system like Kafka to ingest raw data into the data lake allows fast storage and flexible later processing without losing detail.

Practice

1. Why do ingestion pipelines feed data into a data lake?

easy

A. To collect raw data from multiple sources in one place

B. To delete old data automatically

C. To convert all data into a single format immediately

D. To prevent data from being accessed by users

Solution

1. Step 1: Understand the role of ingestion pipelines

   Ingestion pipelines gather data from many different sources.

2. Step 2: Understand the purpose of a data lake

   A data lake stores raw data centrally for easy access and analysis later.

3. Final Answer:

   To collect raw data from multiple sources in one place -> Option A

4. Quick Check:

   Ingestion pipelines feed raw data to data lakes = A [OK]

Hint: Remember: pipelines bring raw data together [OK]

Common Mistakes:

• Thinking pipelines delete data
• Assuming data is converted immediately
• Believing data lakes block user access

2. Which of the following is a correct tool used in ingestion pipelines for feeding data lakes?

easy

A. WordPress

B. Photoshop

C. Excel

D. Hadoop

Solution

1. Step 1: Identify tools related to data ingestion

   Hadoop is a big data tool used for storing and processing large datasets.

2. Step 2: Check other options

   Photoshop, Excel, and WordPress are not designed for data ingestion pipelines.

3. Final Answer:

   Hadoop -> Option D

4. Quick Check:

   Hadoop is a data ingestion tool = A [OK]

Hint: Pick big data tools, not office or design apps [OK]

Common Mistakes:

• Confusing data tools with office software
• Choosing popular but unrelated software
• Ignoring Hadoop's role in data pipelines

3. Consider this code snippet using Apache NiFi to ingest data:

flowFile = session.get()
if flowFile is None:
    session.commit()
    exit()
flowFile = session.putAttribute(flowFile, "source", "sensor1")
session.transfer(flowFile, REL_SUCCESS)

What does this pipeline step do?

medium

A. It deletes the data from the pipeline

B. It adds a source attribute and sends data forward

C. It converts data to CSV format

D. It stops the pipeline permanently

Solution

1. Step 1: Analyze the code actions

   The code gets a data flow file, adds an attribute named 'source' with value 'sensor1'.

2. Step 2: Understand the transfer step

   It then transfers the flow file to the success relationship, moving data forward.

3. Final Answer:

   It adds a source attribute and sends data forward -> Option B

4. Quick Check:

   Attribute added + transfer = C [OK]

Hint: Look for attribute changes and transfer commands [OK]

Common Mistakes:

• Thinking data is deleted
• Assuming format conversion happens here
• Believing pipeline stops permanently

4. This ingestion pipeline code snippet has an error:

data = kafka_consumer.poll()
if data:
print('Data received:', data)
process(data)

What is the error and how to fix it?

medium

A. Indentation error; indent the print and process lines

B. Syntax error; replace poll() with consume()

C. Runtime error; data variable is undefined

D. Logical error; should check data length instead of truthiness

Solution

1. Step 1: Identify the error in code structure

   The print and process lines are not indented under the if statement, causing an indentation error.

2. Step 2: Fix the indentation

   Indent these lines to be inside the if block to run only when data exists.

3. Final Answer:

   Indentation error; indent the print and process lines -> Option A

4. Quick Check:

   Indentation fixes code error = B [OK]

Hint: Check indentation after if statements [OK]

Common Mistakes:

• Confusing method names
• Assuming variable is undefined
• Overthinking data checks

5. You want to build an ingestion pipeline that feeds a data lake with streaming data from multiple sensors. Which approach best ensures data is stored raw and ready for future analysis?

hard

A. Use batch uploads once a day to reduce pipeline complexity

B. Transform data into summaries before storing to reduce storage needs

C. Use Kafka to collect streams, then store raw data in Hadoop HDFS without transformation

D. Store data only after converting to JSON format and filtering out duplicates

Solution

1. Step 1: Understand raw data storage needs

   Storing raw data means saving original data without changes for flexible analysis later.

2. Step 2: Evaluate options for streaming ingestion

   Kafka is designed for streaming data collection; Hadoop HDFS stores large raw datasets efficiently.

3. Step 3: Compare other options

   Transforming or filtering data early loses raw details; batch uploads delay data availability.

4. Final Answer:

   Use Kafka to collect streams, then store raw data in Hadoop HDFS without transformation -> Option C

5. Quick Check:

   Raw streaming data + Hadoop storage = D [OK]

Hint: Keep data raw and streaming for best analysis [OK]

Common Mistakes:

• Filtering data too early
• Using batch instead of streaming
• Converting formats before storage