LOAD, FILTER, and STORE operations in Hadoop - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

What is the output of this Pig Latin script after filtering?

Given the following Pig Latin script, what will be the content of filtered_data?

data = LOAD 'input.txt' USING PigStorage(',') AS (name:chararray, age:int, city:chararray);
filtered_data = FILTER data BY age > 30;
DUMP filtered_data;

Assume input.txt contains:
John,25,New York
Mary,35,Chicago
Bob,40,Seattle

Hadoop

data = LOAD 'input.txt' USING PigStorage(',') AS (name:chararray, age:int, city:chararray);
filtered_data = FILTER data BY age > 30;
DUMP filtered_data;

A (John,25,New York)

B

(John,25,New York)
(Mary,35,Chicago)
(Bob,40,Seattle)

C

(Mary,35,Chicago)
(Bob,40,Seattle)

D Empty output

💡 Hint

Think about the filter condition age > 30 and which rows satisfy it.

The filter keeps only rows where age is greater than 30. John is 25, so excluded. Mary and Bob are included.

❓ data_output

intermediate

How many records remain after filtering?

Consider this Pig Latin script:

records = LOAD 'data.csv' USING PigStorage(',') AS (id:int, score:int);
passed = FILTER records BY score >= 50;
STORE passed INTO 'passed_output';

If data.csv has 5 records with scores: 45, 50, 60, 30, 55, how many records will be stored in passed_output?

Hadoop

records = LOAD 'data.csv' USING PigStorage(',') AS (id:int, score:int);
passed = FILTER records BY score >= 50;
STORE passed INTO 'passed_output';

A 3

B 2

C 5

D 0

💡 Hint

Count how many scores are 50 or more.

Scores 50, 60, and 55 meet the condition, so 3 records remain.

🔧 Debug

advanced

Identify the error in this Pig Latin script

What error will this Pig Latin script produce?

data = LOAD 'file.txt' USING PigStorage(',') AS (name:chararray, age:int);
filtered = FILTER data BY age > '30';
DUMP filtered;

Hadoop

data = LOAD 'file.txt' USING PigStorage(',') AS (name:chararray, age:int);
filtered = FILTER data BY age > '30';
DUMP filtered;

A SyntaxError: Invalid comparison operator

B RuntimeError: File not found

C No error, runs successfully

D TypeError: Cannot compare int with chararray

💡 Hint

Check the data types used in the filter condition.

The filter compares an integer field age with a string '30', causing a type mismatch error.

🚀 Application

advanced

Which Pig Latin command stores filtered data correctly?

You want to load a CSV file, filter rows where status is 'active', and save the result. Which command correctly stores the filtered data?

data = LOAD 'users.csv' USING PigStorage(',') AS (user_id:int, status:chararray);
active_users = FILTER data BY status == 'active';

Hadoop

data = LOAD 'users.csv' USING PigStorage(',') AS (user_id:int, status:chararray);
active_users = FILTER data BY status == 'active';

A STORE active_users INTO 'active_output' USING PigStorage();

B STORE data INTO 'active_output' USING PigStorage();

C STORE active_users INTO 'active_output';

D STORE active_users INTO 'active_output' USING JsonStorage();

💡 Hint

Remember to specify the storage format when using STORE.

Option A correctly stores the filtered relation active_users using PigStorage.

🧠 Conceptual

expert

What happens if you STORE a relation without filtering after LOAD?

Consider this Pig Latin script:

data = LOAD 'records.txt' USING PigStorage(',') AS (id:int, value:int);
STORE data INTO 'output_dir' USING PigStorage();

What will be the content of output_dir?

Hadoop

data = LOAD 'records.txt' USING PigStorage(',') AS (id:int, value:int);
STORE data INTO 'output_dir' USING PigStorage();

A Only records with non-null values will be stored.

B All records from 'records.txt' will be stored in 'output_dir'.

C No records will be stored because no FILTER was applied.

D An error will occur because STORE requires a FILTER operation first.

💡 Hint

Think about what STORE does with a loaded relation.

STORE saves all data in the relation. Filtering is optional. Without filtering, all loaded records are stored.

Practice

1. What does the LOAD operation do in Hadoop data processing?

easy

A. Reads data from files into the system

B. Saves data back to storage

C. Filters rows based on a condition

D. Deletes data from storage

Solution

1. Step 1: Understand the purpose of LOAD

   The LOAD operation is used to bring data from external files into the Hadoop processing environment.

2. Step 2: Differentiate from FILTER and STORE

   FILTER selects data rows, and STORE saves data back, so LOAD must be the reading step.

3. Final Answer:

   Reads data from files into the system -> Option A

4. Quick Check:

   LOAD = Read data [OK]

Hint: LOAD means bring data in from files [OK]

Common Mistakes:

• Confusing LOAD with STORE operation
• Thinking LOAD filters data
• Assuming LOAD deletes data

2. Which of the following is the correct syntax to filter rows where age is greater than 30 in Hadoop Pig Latin?

easy

A. filtered_data = LOAD data FILTER age > 30;

B. filtered_data = FILTER data BY age > 30;

C. filtered_data = SELECT * FROM data WHERE age > 30;

D. filtered_data = FILTER data WHERE age > 30;

Solution

1. Step 1: Recall FILTER syntax in Pig Latin

   The correct syntax uses FILTER ... BY to select rows based on a condition.

2. Step 2: Check each option

   A mixes LOAD and FILTER incorrectly; B uses WHERE which is invalid in Pig; C is SQL syntax; D uses FILTER ... BY correctly.

3. Final Answer:

   filtered_data = FILTER data BY age > 30; -> Option B

4. Quick Check:

   FILTER ... BY is correct syntax [OK]

Hint: FILTER uses BY, not WHERE, in Pig Latin [OK]

Common Mistakes:

• Using WHERE instead of BY in FILTER
• Confusing Pig syntax with SQL
• Trying to filter during LOAD

3. Given the following Pig Latin code:

data = LOAD 'users.txt' USING PigStorage(',') AS (name:chararray, age:int);
adults = FILTER data BY age >= 18;
STORE adults INTO 'adult_users';

What will be the content of the 'adult_users' folder?

medium

A. All users from 'users.txt' regardless of age

B. An empty folder because STORE is missing a format

C. Only users with age less than 18

D. Only users with age 18 or older

Solution

1. Step 1: Understand the FILTER condition

   The FILTER selects rows where age is 18 or more, so only adults remain.

2. Step 2: STORE saves filtered data

   The STORE command saves the filtered adults data into 'adult_users' folder.

3. Final Answer:

   Only users with age 18 or older -> Option D

4. Quick Check:

   FILTER age >= 18 then STORE = adults only [OK]

Hint: FILTER keeps only matching rows before STORE [OK]

Common Mistakes:

• Assuming STORE saves original data
• Thinking STORE needs format specified always
• Confusing FILTER condition direction

4. Identify the error in this Pig Latin code snippet:

data = LOAD 'data.txt' USING PigStorage(',');
filtered = FILTER data WHERE score > 50;
STORE filtered INTO 'high_scores';

medium

A. Missing AS clause in LOAD

B. STORE command syntax is incorrect

C. FILTER uses WHERE instead of BY

D. PigStorage should not have parameters

Solution

1. Step 1: Check FILTER syntax

   FILTER in Pig Latin requires BY keyword, not WHERE.

2. Step 2: Verify other parts

   LOAD can omit AS but then schema is unknown; STORE syntax is correct; PigStorage accepts parameters.

3. Final Answer:

   FILTER uses WHERE instead of BY -> Option C

4. Quick Check:

   FILTER ... BY is correct, not WHERE [OK]

Hint: FILTER always uses BY, never WHERE [OK]

Common Mistakes:

• Using WHERE in FILTER
• Thinking AS is mandatory in LOAD
• Misunderstanding PigStorage parameters

5. You have a dataset 'sales.csv' with columns (product, quantity, price). You want to load this data, filter only rows where quantity is greater than 10 and price less than 100, then store the result in 'filtered_sales'. Which Pig Latin code correctly performs this?

hard

A. sales = LOAD 'sales.csv' USING PigStorage(',') AS (product:chararray, quantity:int, price:int); filtered = FILTER sales BY quantity > 10 AND price < 100; STORE filtered INTO 'filtered_sales';

B. sales = LOAD 'sales.csv' USING PigStorage(','); filtered = FILTER sales WHERE quantity > 10 AND price < 100; STORE filtered INTO 'filtered_sales';

C. sales = LOAD 'sales.csv' USING PigStorage(',') AS (product, quantity, price); filtered = FILTER sales BY quantity > 10 OR price < 100; STORE filtered INTO 'filtered_sales';

D. sales = LOAD 'sales.csv' USING PigStorage(',') AS (product:chararray, quantity:int, price:int); filtered = FILTER sales BY quantity > 10 OR price < 100; STORE filtered INTO 'filtered_sales';

Solution

1. Step 1: Load with schema and correct syntax

   sales = LOAD 'sales.csv' USING PigStorage(',') AS (product:chararray, quantity:int, price:int); filtered = FILTER sales BY quantity > 10 AND price < 100; STORE filtered INTO 'filtered_sales'; loads with schema and uses PigStorage correctly.

2. Step 2: Filter with correct condition and operator

   We want quantity > 10 AND price < 100, so AND is correct, not OR.

3. Step 3: STORE command is correct

   sales = LOAD 'sales.csv' USING PigStorage(',') AS (product:chararray, quantity:int, price:int); filtered = FILTER sales BY quantity > 10 AND price < 100; STORE filtered INTO 'filtered_sales'; uses STORE correctly to save filtered data.

4. Final Answer:

   Correct load, filter with AND, and store commands -> Option A

5. Quick Check:

   LOAD with schema + FILTER with AND + STORE = sales = LOAD 'sales.csv' USING PigStorage(',') AS (product:chararray, quantity:int, price:int); filtered = FILTER sales BY quantity > 10 AND price < 100; STORE filtered INTO 'filtered_sales'; [OK]

Hint: Use AND for both conditions, FILTER ... BY, and STORE properly [OK]

Common Mistakes:

• Using WHERE instead of BY in FILTER
• Using OR instead of AND for filtering
• Omitting schema in LOAD causing errors