Pig Latin basics in Hadoop - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of a simple LOAD and FILTER operation

Given the following Pig Latin script, what is the output relation after the FILTER operation?

data = LOAD 'input.txt' USING PigStorage(',') AS (name:chararray, age:int);
adults = FILTER data BY age >= 18;

Assume input.txt contains:

John,17
Mary,22
Bob,18

Hadoop

data = LOAD 'input.txt' USING PigStorage(',') AS (name:chararray, age:int);
adults = FILTER data BY age >= 18;
DUMP adults;

A (Mary,22)

B

(John,17)
(Mary,22)
(Bob,18)

C (John,17)

D

(Mary,22)
(Bob,18)

💡 Hint

FILTER keeps only rows where the condition is true.

The FILTER keeps only rows where age is 18 or more, so John (17) is excluded.

❓ data_output

intermediate

Result of a GROUP and COUNT operation

What is the output of the following Pig Latin script?

data = LOAD 'input.txt' USING PigStorage(',') AS (category:chararray, value:int);
grp = GROUP data BY category;
counted = FOREACH grp GENERATE group, COUNT(data);
DUMP counted;

Assume input.txt contains:

A,10
B,20
A,30
B,40
A,50

Hadoop

data = LOAD 'input.txt' USING PigStorage(',') AS (category:chararray, value:int);
grp = GROUP data BY category;
counted = FOREACH grp GENERATE group, COUNT(data);
DUMP counted;

A

(A,1)
(B,1)

B

(A,3)
(B,2)

C

(A,5)
(B,5)

D

(A,2)
(B,3)

💡 Hint

GROUP collects rows by category, COUNT counts rows per group.

Category A appears 3 times, B appears 2 times, so counts are (A,3) and (B,2).

🔧 Debug

advanced

Identify the error in a JOIN operation

What error will this Pig Latin script produce?

users = LOAD 'users.txt' USING PigStorage(',') AS (user_id:int, name:chararray);
orders = LOAD 'orders.txt' USING PigStorage(',') AS (order_id:int, user:int, amount:float);
joined = JOIN users BY user_id, orders BY user_id;
DUMP joined;

Hadoop

users = LOAD 'users.txt' USING PigStorage(',') AS (user_id:int, name:chararray);
orders = LOAD 'orders.txt' USING PigStorage(',') AS (order_id:int, user:int, amount:float);
joined = JOIN users BY user_id, orders BY user_id;
DUMP joined;

A Error: Field 'user_id' not found in relation 'orders'

B No error, outputs joined data

C Error: Syntax error near JOIN statement

D Error: Duplicate field names in output

💡 Hint

Check field names used in JOIN keys.

The 'orders' relation has a field named 'user', not 'user_id', so JOIN fails with field not found error.

❓ visualization

advanced

Visualize the result of a FOREACH GENERATE with arithmetic

Given this Pig Latin script, what is the output after the FOREACH GENERATE?

data = LOAD 'input.txt' USING PigStorage(',') AS (item:chararray, price:float, quantity:int);
total = FOREACH data GENERATE item, price * quantity AS total_cost;
DUMP total;

Assume input.txt contains:

Pen,1.5,10
Notebook,2.0,5
Eraser,0.5,20

Hadoop

data = LOAD 'input.txt' USING PigStorage(',') AS (item:chararray, price:float, quantity:int);
total = FOREACH data GENERATE item, price * quantity AS total_cost;
DUMP total;

A

(Pen,15.0)
(Notebook,10.0)
(Eraser,10.0)

B

(Pen,1.5)
(Notebook,2.0)
(Eraser,0.5)

C

(Pen,10)
(Notebook,5)
(Eraser,20)

D

(Pen,150)
(Notebook,200)
(Eraser,100)

💡 Hint

Multiply price by quantity for each item.

Pen: 1.5*10=15.0, Notebook: 2.0*5=10.0, Eraser: 0.5*20=10.0

🚀 Application

expert

Determine the number of records after a COGROUP and FILTER

Consider these Pig Latin commands:

students = LOAD 'students.txt' USING PigStorage(',') AS (student_id:int, name:chararray);
grades = LOAD 'grades.txt' USING PigStorage(',') AS (student_id:int, grade:int);
grouped = COGROUP students BY student_id, grades BY student_id;
passed = FILTER grouped BY COUNT(grades) > 0 AND AVG(grades.grade) >= 60;
DUMP passed;

Assuming:

students.txt:
1,Alice
2,Bob
3,Charlie
grades.txt:
1,70
1,80
2,50
3,90
3,55

How many records will passed contain?

Hadoop

students = LOAD 'students.txt' USING PigStorage(',') AS (student_id:int, name:chararray);
grades = LOAD 'grades.txt' USING PigStorage(',') AS (student_id:int, grade:int);
grouped = COGROUP students BY student_id, grades BY student_id;
passed = FILTER grouped BY COUNT(grades) > 0 AND AVG(grades.grade) >= 60;
DUMP passed;

A 3

B 1

C 2

D 0

💡 Hint

Check which students have grades and average grade >= 60.

Student 1: grades 70,80 avg=75 (passes); Student 2: grade 50 avg=50 (fails); Student 3: grades 90,55 avg=72.5 (passes). So 2 records pass.

Practice

1. What is the main purpose of the FILTER statement in Pig Latin?

easy

A. To select rows that meet a condition

B. To combine two datasets

C. To load data from a file

D. To save the results to storage

Solution

1. Step 1: Understand the role of FILTER

   The FILTER statement is used to keep only the rows that satisfy a given condition.

2. Step 2: Compare with other statements

   LOAD reads data, GROUP groups data, STORE saves data, so FILTER is for selecting rows.

3. Final Answer:

   To select rows that meet a condition -> Option A

4. Quick Check:

   FILTER = select rows [OK]

Hint: FILTER keeps rows matching condition, like a sieve [OK]

Common Mistakes:

• Confusing FILTER with GROUP
• Thinking FILTER saves data
• Mixing FILTER with LOAD

2. Which of the following is the correct syntax to load a file named data.txt in Pig Latin?

easy

A. LOAD 'data.txt' USING PigStorage();

B. LOAD data.txt USING PigStorage();

C. LOAD 'data.txt' AS PigStorage();

D. LOAD 'data.txt' USING PigStorage

Solution

1. Step 1: Check the LOAD syntax

   LOAD requires the file path as a string in quotes and the loader function with parentheses.

2. Step 2: Validate each option

   LOAD 'data.txt' USING PigStorage(); uses quotes and parentheses correctly. LOAD data.txt USING PigStorage(); misses quotes. LOAD 'data.txt' AS PigStorage(); uses AS instead of USING. LOAD 'data.txt' USING PigStorage misses parentheses.

3. Final Answer:

   LOAD 'data.txt' USING PigStorage(); -> Option A

4. Quick Check:

   LOAD 'file' USING loader(); [OK]

Hint: LOAD needs quotes and parentheses for loader [OK]

Common Mistakes:

• Forgetting quotes around filename
• Using AS instead of USING
• Omitting parentheses after loader

3. Given the following Pig Latin code:

A = LOAD 'numbers.txt' USING PigStorage(',');
B = FILTER A BY $0 > 10;
C = FOREACH B GENERATE $0;
DUMP C;

What will be the output if numbers.txt contains:
5,abc
15,def
8,ghi
20,jkl

medium

A. (5,abc)
(8,ghi)

B. (5)
(8)

C. (15,def)
(20,jkl)

D. (15)
(20)

Solution

1. Step 1: Understand the FILTER condition

   FILTER keeps rows where first column ($0) is greater than 10, so rows with 15 and 20 remain.

2. Step 2: Understand FOREACH GENERATE

   FOREACH selects only the first column ($0), so output shows only 15 and 20 as single values.

3. Final Answer:

   (15) (20) -> Option D

4. Quick Check:

   FILTER $0 > 10 then select $0 [OK]

Hint: FILTER then FOREACH select column shows filtered values [OK]

Common Mistakes:

• Including second column in output
• Selecting rows less than or equal to 10
• Confusing FILTER condition

4. Identify the error in this Pig Latin script:

data = LOAD 'input.txt' USING PigStorage(',');
filtered = FILTER data BY $1 == 'USA';
grouped = GROUP filtered BY $2;
result = FOREACH grouped GENERATE group, COUNT(filtered);
STORE result INTO 'output';

medium

A. COUNT should use filtered instead of group

B. STORE command missing USING clause

C. FILTER condition uses wrong column index

D. GROUP should use filtered instead of data

Solution

1. Step 1: Check STORE syntax

   STORE requires a USING clause to specify the storage format, e.g., PigStorage().

2. Step 2: Verify other statements

   GROUP uses filtered correctly, COUNT uses filtered correctly, FILTER condition is valid if $1 is country column.

3. Final Answer:

   STORE command missing USING clause -> Option B

4. Quick Check:

   STORE ... INTO 'path' USING loader() [OK]

Hint: STORE needs USING clause to save data [OK]

Common Mistakes:

• Omitting USING in STORE
• Confusing GROUP input
• Misunderstanding COUNT argument

5. You have a dataset of sales with columns: product_id, region, amount. You want to find total sales per region but only for sales above 100. Which Pig Latin sequence correctly achieves this?

hard

A. A = LOAD 'sales.txt' USING PigStorage(',');
B = FILTER A BY amount > 100;
C = GROUP B BY region;
D = FOREACH C GENERATE group, SUM(B.amount);
STORE D INTO 'result' USING PigStorage();

B. A = LOAD 'sales.txt' USING PigStorage(',');
B = FILTER A BY $2 > 100;
C = GROUP B BY $1;
D = FOREACH C GENERATE group, SUM(B.$2);
STORE D INTO 'result';

C. A = LOAD 'sales.txt' USING PigStorage(',');
B = FILTER A BY $2 > 100;
C = GROUP B BY $1;
D = FOREACH C GENERATE group, SUM(B.$2);
STORE D INTO 'result' USING PigStorage();

D. A = LOAD 'sales.txt' USING PigStorage(',');
B = FILTER A BY $1 > 100;
C = GROUP B BY $2;
D = FOREACH C GENERATE group, SUM(B.$3);
STORE D INTO 'result' USING PigStorage();

Solution

1. Step 1: Identify correct column indexes

   Columns: product_id ($0), region ($1), amount ($2). Filter by amount > 100 means FILTER BY $2 > 100.

2. Step 2: Group by region and sum amount

   Group by $1 (region), then sum B.$2 (amount) in FOREACH. STORE needs USING clause.

3. Step 3: Check each option

   A = LOAD 'sales.txt' USING PigStorage(','); B = FILTER A BY $2 > 100; C = GROUP B BY $1; D = FOREACH C GENERATE group, SUM(B.$2); STORE D INTO 'result' USING PigStorage(); matches all requirements: correct filter, group, sum, and STORE syntax.

4. Final Answer:

   Option C code sequence -> Option C

5. Quick Check:

   Filter $2>100, group $1, sum $2, STORE with USING [OK]

Hint: Use correct column indexes and STORE with USING [OK]

Common Mistakes:

• Using wrong column indexes for filter or group
• Omitting USING in STORE
• Summing wrong column