Why ingestion pipelines feed the data lake in Hadoop - Challenge Your Understanding

Challenge - 5 Problems

🧠 Conceptual

intermediate

Purpose of Ingestion Pipelines in Data Lakes

Why do ingestion pipelines feed data lakes in big data systems?

A To convert all data into a single format before storing it

B To immediately clean and transform data before storing it in a database

C To delete old data from the data lake to save storage space

D To collect and store raw data from multiple sources for flexible analysis later

💡 Hint

Think about why raw data storage is useful for future analysis.

Ingestion pipelines feed data lakes to gather raw data from many sources. This raw data is stored as-is, allowing flexible and varied analysis later without losing original details.

🧠 Conceptual

intermediate

Data Lake vs Data Warehouse Feeding

What is a key reason ingestion pipelines feed data lakes instead of data warehouses?

A Data lakes are slower to access than data warehouses

B Data lakes only store images, while data warehouses store text data

C Data lakes store raw data, while data warehouses require cleaned and structured data

D Data lakes automatically analyze data, data warehouses do not

💡 Hint

Consider the difference in data format requirements between lakes and warehouses.

Data lakes accept raw, unstructured data from ingestion pipelines, while data warehouses need data to be cleaned and structured before loading.

❓ data_output

advanced

Output of a Hadoop Ingestion Pipeline

Given a Hadoop ingestion pipeline that collects JSON logs from multiple servers and stores them in HDFS, what is the typical output format stored in the data lake?

Hadoop

import json
logs = ['{"user":"alice","action":"login"}', '{"user":"bob","action":"logout"}']
parsed_logs = [json.loads(log) for log in logs]
# What is stored in HDFS?

A A single CSV file with columns user and action

B [{"user": "alice", "action": "login"}, {"user": "bob", "action": "logout"}] as raw JSON files

C A SQL database table with user and action columns

D An Excel spreadsheet with user and action data

💡 Hint

Think about how Hadoop stores raw data from ingestion pipelines.

Hadoop ingestion pipelines typically store raw JSON files directly in HDFS, preserving original data format for flexible processing later.

🔧 Debug

advanced

Identify the Error in a Hadoop Ingestion Script

What error will this Hadoop ingestion Python script raise?

Hadoop

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('Ingest').getOrCreate()
data = spark.read.json('hdfs://logs/*.json')
data.write.csv('hdfs://output/logs.csv')
spark.stop()

A TypeError because write.csv expects a directory, not a file path

B AnalysisException because output path 'hdfs://output/logs.csv' already exists

C FileNotFoundError because 'hdfs://logs/*.json' path is invalid

D No error; script runs successfully and writes CSV files

💡 Hint

Check the expected argument type for write.csv in Spark.

In Spark, write.csv expects a directory path, not a file path. Giving a file path causes an AnalysisException, not a TypeError.

🚀 Application

expert

Designing an Efficient Ingestion Pipeline for a Data Lake

You need to design an ingestion pipeline that feeds a data lake with streaming sensor data. Which approach best ensures data is stored quickly and can be processed later in different ways?

A Use a streaming system like Apache Kafka to collect raw sensor data and store it directly in the data lake as JSON files

B Transform sensor data into a fixed schema and load it immediately into a relational database

C Aggregate sensor data in real-time and store only summary statistics in the data lake

D Store sensor data temporarily in local files and batch upload to the data lake once a day

💡 Hint

Consider speed and flexibility of raw data storage for future analysis.

Using a streaming system like Kafka to ingest raw data into the data lake allows fast storage and flexible later processing without losing detail.