Pig Latin basics in Hadoop - Step-by-Step Execution

Concept Flow - Pig Latin basics

Load Data

↓

Apply Transformations

↓

Filter or Group Data

↓

Generate Output

↓

Store or Dump Results

Pig Latin scripts start by loading data, then apply transformations like filtering or grouping, and finally output the results.

Execution Sample

Hadoop

A = LOAD 'data.txt' USING PigStorage(',');
B = FILTER A BY $0 > 10;
C = GROUP B BY $1;
D = FOREACH C GENERATE group, COUNT(B);
DUMP D;

This script loads data, filters rows where first column > 10, groups by second column, counts rows per group, and shows results.

Execution Table

Step	Action	Input Data Snapshot	Result	Notes
1	LOAD data.txt	Raw file with rows like (12,apple), (6,banana)	Relation A with all rows	Data loaded into relation A
2	FILTER A BY $0 > 10	Relation A with rows (12,apple), (6,banana)	Relation B with rows (12,apple)	Only rows with first column > 10 kept
3	GROUP B BY $1	Relation B with (12,apple)	Relation C grouped by second column 'apple'	Rows grouped by second column
4	FOREACH C GENERATE group, COUNT(B)	Grouped relation C	Relation D with ('apple', 1)	Count of rows per group calculated
5	DUMP D	Relation D	Output: ('apple', 1)	Results displayed on screen

💡 All steps completed, final output dumped

Variable Tracker

Relation	Start	After Step 1	After Step 2	After Step 3	After Step 4	Final
A	empty	All rows from data.txt	All rows from data.txt	All rows from data.txt	All rows from data.txt	All rows from data.txt
B	empty	empty	Rows with $0 > 10	Rows with $0 > 10	Rows with $0 > 10	Rows with $0 > 10
C	empty	empty	empty	Grouped by $1	Grouped by $1	Grouped by $1
D	empty	empty	empty	empty	Group counts	Group counts

Key Moments - 3 Insights

Why does FILTER keep only some rows and not all?

FILTER keeps rows where the condition is true. In the execution_table step 2, only rows with first column > 10 remain.

What does GROUP BY do to the data?

GROUP BY collects rows sharing the same key into one group. See execution_table step 3 where rows are grouped by the second column.

Why do we use FOREACH after GROUP BY?

FOREACH lets us process each group separately, like counting rows per group as shown in step 4 of the execution_table.

Visual Quiz - 3 Questions

Look at the execution_table at step 2. What rows does relation B contain?

A Rows where second column is 'apple'

B Rows where first column is greater than 10

C All rows from the file

D Empty relation

💡 Hint

Check the 'Action' and 'Result' columns at step 2 in execution_table

At which step does the data get grouped by the second column?

A Step 1

B Step 2

C Step 3

D Step 4

💡 Hint

Look for 'GROUP BY' action in execution_table

If the FILTER condition changed to $0 > 5, how would relation B change after step 2?

A It would contain more rows

B It would contain fewer rows

C It would be empty

D It would be unchanged

💡 Hint

Compare the FILTER condition and its effect on rows in variable_tracker

Concept Snapshot

Pig Latin basics:
- LOAD reads data into a relation
- FILTER keeps rows matching a condition
- GROUP BY collects rows by key
- FOREACH processes each group
- DUMP shows results
Simple steps to transform big data

Full Transcript

This visual trace shows how a Pig Latin script runs step-by-step. First, data is loaded from a file into a relation named A. Then, a FILTER keeps only rows where the first column is greater than 10, creating relation B. Next, the data in B is grouped by the second column, forming relation C. After grouping, FOREACH generates a new relation D with the group key and count of rows in each group. Finally, DUMP outputs the results. Variables change as data moves through these steps. Beginners often wonder why FILTER removes rows or how GROUP BY works; this trace clarifies those by showing exact data changes. The quiz tests understanding of these steps and their effects on data.

Practice

1. What is the main purpose of the FILTER statement in Pig Latin?

easy

A. To select rows that meet a condition

B. To combine two datasets

C. To load data from a file

D. To save the results to storage

Solution

1. Step 1: Understand the role of FILTER

   The FILTER statement is used to keep only the rows that satisfy a given condition.

2. Step 2: Compare with other statements

   LOAD reads data, GROUP groups data, STORE saves data, so FILTER is for selecting rows.

3. Final Answer:

   To select rows that meet a condition -> Option A

4. Quick Check:

   FILTER = select rows [OK]

Hint: FILTER keeps rows matching condition, like a sieve [OK]

Common Mistakes:

• Confusing FILTER with GROUP
• Thinking FILTER saves data
• Mixing FILTER with LOAD

2. Which of the following is the correct syntax to load a file named data.txt in Pig Latin?

easy

A. LOAD 'data.txt' USING PigStorage();

B. LOAD data.txt USING PigStorage();

C. LOAD 'data.txt' AS PigStorage();

D. LOAD 'data.txt' USING PigStorage

Solution

1. Step 1: Check the LOAD syntax

   LOAD requires the file path as a string in quotes and the loader function with parentheses.

2. Step 2: Validate each option

   LOAD 'data.txt' USING PigStorage(); uses quotes and parentheses correctly. LOAD data.txt USING PigStorage(); misses quotes. LOAD 'data.txt' AS PigStorage(); uses AS instead of USING. LOAD 'data.txt' USING PigStorage misses parentheses.

3. Final Answer:

   LOAD 'data.txt' USING PigStorage(); -> Option A

4. Quick Check:

   LOAD 'file' USING loader(); [OK]

Hint: LOAD needs quotes and parentheses for loader [OK]

Common Mistakes:

• Forgetting quotes around filename
• Using AS instead of USING
• Omitting parentheses after loader

3. Given the following Pig Latin code:

A = LOAD 'numbers.txt' USING PigStorage(',');
B = FILTER A BY $0 > 10;
C = FOREACH B GENERATE $0;
DUMP C;

What will be the output if numbers.txt contains:
5,abc
15,def
8,ghi
20,jkl

medium

A. (5,abc)
(8,ghi)

B. (5)
(8)

C. (15,def)
(20,jkl)

D. (15)
(20)

Solution

1. Step 1: Understand the FILTER condition

   FILTER keeps rows where first column ($0) is greater than 10, so rows with 15 and 20 remain.

2. Step 2: Understand FOREACH GENERATE

   FOREACH selects only the first column ($0), so output shows only 15 and 20 as single values.

3. Final Answer:

   (15) (20) -> Option D

4. Quick Check:

   FILTER $0 > 10 then select $0 [OK]

Hint: FILTER then FOREACH select column shows filtered values [OK]

Common Mistakes:

• Including second column in output
• Selecting rows less than or equal to 10
• Confusing FILTER condition

4. Identify the error in this Pig Latin script:

data = LOAD 'input.txt' USING PigStorage(',');
filtered = FILTER data BY $1 == 'USA';
grouped = GROUP filtered BY $2;
result = FOREACH grouped GENERATE group, COUNT(filtered);
STORE result INTO 'output';

medium

A. COUNT should use filtered instead of group

B. STORE command missing USING clause

C. FILTER condition uses wrong column index

D. GROUP should use filtered instead of data

Solution

1. Step 1: Check STORE syntax

   STORE requires a USING clause to specify the storage format, e.g., PigStorage().

2. Step 2: Verify other statements

   GROUP uses filtered correctly, COUNT uses filtered correctly, FILTER condition is valid if $1 is country column.

3. Final Answer:

   STORE command missing USING clause -> Option B

4. Quick Check:

   STORE ... INTO 'path' USING loader() [OK]

Hint: STORE needs USING clause to save data [OK]

Common Mistakes:

• Omitting USING in STORE
• Confusing GROUP input
• Misunderstanding COUNT argument

5. You have a dataset of sales with columns: product_id, region, amount. You want to find total sales per region but only for sales above 100. Which Pig Latin sequence correctly achieves this?

hard

A. A = LOAD 'sales.txt' USING PigStorage(',');
B = FILTER A BY amount > 100;
C = GROUP B BY region;
D = FOREACH C GENERATE group, SUM(B.amount);
STORE D INTO 'result' USING PigStorage();

B. A = LOAD 'sales.txt' USING PigStorage(',');
B = FILTER A BY $2 > 100;
C = GROUP B BY $1;
D = FOREACH C GENERATE group, SUM(B.$2);
STORE D INTO 'result';

C. A = LOAD 'sales.txt' USING PigStorage(',');
B = FILTER A BY $2 > 100;
C = GROUP B BY $1;
D = FOREACH C GENERATE group, SUM(B.$2);
STORE D INTO 'result' USING PigStorage();

D. A = LOAD 'sales.txt' USING PigStorage(',');
B = FILTER A BY $1 > 100;
C = GROUP B BY $2;
D = FOREACH C GENERATE group, SUM(B.$3);
STORE D INTO 'result' USING PigStorage();

Solution

1. Step 1: Identify correct column indexes

   Columns: product_id ($0), region ($1), amount ($2). Filter by amount > 100 means FILTER BY $2 > 100.

2. Step 2: Group by region and sum amount

   Group by $1 (region), then sum B.$2 (amount) in FOREACH. STORE needs USING clause.

3. Step 3: Check each option

   A = LOAD 'sales.txt' USING PigStorage(','); B = FILTER A BY $2 > 100; C = GROUP B BY $1; D = FOREACH C GENERATE group, SUM(B.$2); STORE D INTO 'result' USING PigStorage(); matches all requirements: correct filter, group, sum, and STORE syntax.

4. Final Answer:

   Option C code sequence -> Option C

5. Quick Check:

   Filter $2>100, group $1, sum $2, STORE with USING [OK]

Hint: Use correct column indexes and STORE with USING [OK]

Common Mistakes:

• Using wrong column indexes for filter or group
• Omitting USING in STORE
• Summing wrong column