Why Pig simplifies data transformation in Hadoop - See It in Action

Why Pig Simplifies Data Transformation

📖 Scenario: Imagine you work at a company that collects large amounts of sales data every day. You need to clean and summarize this data to find out which products sell the most. Doing this directly with raw data can be very complex and slow.

🎯 Goal: Learn how to use Apache Pig to simplify data transformation tasks like filtering, grouping, and summarizing big data easily with simple scripts.

📋 What You'll Learn

Create a Pig relation to load sales data

Filter sales records for a specific product category

Group sales by product to calculate total sales

Display the summarized sales results

💡 Why This Matters

🌍 Real World

Companies use Pig to quickly process and analyze large datasets like sales, logs, or user data without writing complex code.

💼 Career

Knowing Pig helps data engineers and analysts handle big data transformations efficiently in Hadoop environments.

1

Load the sales data

Write a Pig Latin statement to load the sales data from the file 'sales_data.csv' into a relation called sales. Assume the data has three fields: product (chararray), category (chararray), and amount (int). Use LOAD with PigStorage(',') and define the schema.

Hadoop

-- Load sales data from 'sales_data.csv' with schema (product:chararray, category:chararray, amount:int)
-- Your code here

Need a hint?

Use LOAD with PigStorage and specify the schema with AS.

2

Filter sales for the 'Electronics' category

Create a new relation called electronics_sales by filtering sales to keep only records where category equals 'Electronics'. Use the FILTER statement.

Hadoop

sales = LOAD 'sales_data.csv' USING PigStorage(',') AS (product:chararray, category:chararray, amount:int);
-- Filter sales where category is 'Electronics'
-- Your code here

Need a hint?

Use FILTER with the condition category == 'Electronics'.

3

Group sales by product and calculate total amount

Group electronics_sales by product into a relation called grouped_sales. Then, create a relation called total_sales that calculates the sum of amount for each product using the FOREACH and GENERATE statements.

Hadoop

sales = LOAD 'sales_data.csv' USING PigStorage(',') AS (product:chararray, category:chararray, amount:int);
electronics_sales = FILTER sales BY category == 'Electronics';
-- Group electronics_sales by product
-- Calculate total amount per product
-- Your code here

Need a hint?

Use GROUP to group by product, then FOREACH ... GENERATE with SUM to calculate totals.

4

Display the total sales per product

Use the DUMP statement to display the contents of total_sales.

Hadoop

sales = LOAD 'sales_data.csv' USING PigStorage(',') AS (product:chararray, category:chararray, amount:int);
electronics_sales = FILTER sales BY category == 'Electronics';
grouped_sales = GROUP electronics_sales BY product;
total_sales = FOREACH grouped_sales GENERATE group AS product, SUM(electronics_sales.amount) AS total_amount;
-- Display the total_sales relation
-- Your code here

Need a hint?

Use DUMP to print the relation contents.