Pig vs Hive comparison in Hadoop - Visual Side-by-Side Comparison

Concept Flow - Pig vs Hive comparison

Start: Data in Hadoop

↓

Pig Latin Script

↓

Pig Execution Engine

↓

MapReduce Jobs

↓

Data Processed

↓

Results Returned

Shows how Pig and Hive take different scripts or queries, convert them to MapReduce jobs, and process data in Hadoop.

Execution Sample

Hadoop

Pig: A = LOAD 'data' AS (name, age);
B = FILTER A BY age > 30;
DUMP B;

Hive: SELECT name, age FROM data WHERE age > 30;

Pig script loads data, filters by age > 30, and shows results; Hive query selects same data with condition.

Execution Table

Step	Tool	Action	Input	Output	Notes
1	Pig	LOAD data	'data' file	Relation A with (name, age)	Reads raw data into Pig relation
2	Pig	FILTER A BY age > 30	Relation A	Relation B with filtered rows	Filters rows where age > 30
3	Pig	DUMP B	Relation B	Print filtered rows	Outputs filtered data to console
4	Hive	Parse Query	SELECT name, age FROM data WHERE age > 30	Logical plan	Hive parses SQL-like query
5	Hive	Compile to MapReduce	Logical plan	MapReduce jobs	Hive converts query to MapReduce
6	Hive	Execute MapReduce	MapReduce jobs	Filtered data output	Runs jobs and returns results
7	Both	MapReduce Execution	Jobs from Pig or Hive	Processed data	Both use MapReduce under the hood
8	End	Process complete	Processed data	Results returned	Data ready for user

💡 Both Pig and Hive finish after MapReduce jobs process data and return results.

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	After Step 5	After Step 6	Final
Pig Relation A	empty	Loaded data	Loaded data	Loaded data	N/A	N/A	N/A	N/A
Pig Relation B	N/A	N/A	Filtered data (age > 30)	Filtered data	N/A	N/A	N/A	N/A
Hive Query	Raw SQL	Raw SQL	Raw SQL	Raw SQL	Logical plan	MapReduce jobs	Results	Results

Key Moments - 3 Insights

Why does Pig use a script while Hive uses SQL-like queries?

Pig uses Pig Latin, a procedural script language good for step-by-step data flow (see execution_table steps 1-3). Hive uses HiveQL, a declarative SQL-like language that describes what data to get (see steps 4-6).

Do Pig and Hive run on different engines?

No, both Pig and Hive convert their scripts or queries into MapReduce jobs that run on the same Hadoop engine (see execution_table step 7).

Which tool is better for complex data transformations?

Pig is often better for complex procedural data flows because you write step-by-step scripts (see Pig steps 1-3). Hive is better for SQL users and simpler queries (see Hive steps 4-6).

Visual Quiz - 3 Questions

Look at the execution table, what is the output after Pig's FILTER step?

A Relation A with all data

B Relation B with filtered rows where age > 30

C MapReduce jobs

D Parsed Hive query

💡 Hint

Check execution_table row 2 under Output column.

At which step does Hive convert the query into MapReduce jobs?

A Step 5

B Step 4

C Step 6

D Step 3

💡 Hint

Look at execution_table row 5 under Action column.

If Pig did not filter data, what would be the output after step 2?

A Filtered data with age > 30

B Empty relation

C Relation A with all loaded data

D MapReduce jobs

💡 Hint

Refer to variable_tracker for Pig Relation B after step 2.

Concept Snapshot

Pig vs Hive Comparison:
- Pig uses Pig Latin scripts (procedural).
- Hive uses HiveQL queries (declarative SQL).
- Both convert code to MapReduce jobs.
- Pig is good for complex data flows.
- Hive is good for SQL users and simple queries.
- Both run on Hadoop's MapReduce engine.

Full Transcript

This visual compares Pig and Hive in Hadoop. Pig uses scripts called Pig Latin to load, filter, and output data step-by-step. Hive uses SQL-like queries to select data. Both tools convert their code into MapReduce jobs that run on Hadoop to process data. Pig is procedural and good for complex flows. Hive is declarative and suits SQL users. The execution table shows each step Pig and Hive take, from loading data to returning results. Variable tracking shows how data changes in Pig relations and Hive queries. Key moments clarify why Pig uses scripts and Hive uses SQL, and that both run on the same engine. The quiz tests understanding of outputs and steps in the process.

Practice

1. Which statement best describes the main difference between Pig and Hive in Hadoop?

easy

A. Pig stores data, Hive processes data.

B. Pig is declarative, and Hive is procedural.

C. Pig is used only for batch processing, Hive only for real-time.

D. Pig uses a scripting language, while Hive uses SQL-like queries.

Solution

1. Step 1: Understand Pig and Hive languages

   Pig uses a scripting language called Pig Latin, which is procedural. Hive uses a SQL-like declarative language called HiveQL.

2. Step 2: Compare their query styles

   Pig scripts describe how to process data step-by-step (procedural), while Hive queries describe what data to retrieve (declarative).

3. Final Answer:

   Pig uses a scripting language, while Hive uses SQL-like queries. -> Option D

4. Quick Check:

   Language style difference = Pig scripting, Hive SQL [OK]

Hint: Pig = scripting; Hive = SQL-like queries [OK]

Common Mistakes:

• Confusing procedural and declarative terms
• Thinking Pig uses SQL
• Assuming Hive is procedural

2. Which of the following is the correct way to write a simple query in Hive to select all records from a table named users?

easy

A. SELECT * FROM users;

B. LOAD users;

C. FETCH ALL FROM users;

D. GET * FROM users;

Solution

1. Step 1: Recall Hive query syntax

   Hive uses SQL-like syntax. The standard way to select all records is SELECT * FROM table_name;.

2. Step 2: Check each option

   SELECT * FROM users; matches the correct SQL syntax. Options A, C, and D are not valid Hive commands.

3. Final Answer:

   SELECT * FROM users; -> Option A

4. Quick Check:

   Hive uses SQL syntax = SELECT * FROM [table] [OK]

Hint: Hive queries look like standard SQL SELECT statements [OK]

Common Mistakes:

• Using non-SQL keywords like LOAD or FETCH for queries
• Forgetting the FROM keyword
• Using GET instead of SELECT

3. Given the following Pig script:

data = LOAD 'input' AS (name:chararray, age:int);
filtered = FILTER data BY age > 25;
DUMP filtered;

What will be the output if the input data contains:

('Alice', 30)
('Bob', 22)
('Carol', 27)

medium

A. All three records

B. ('Bob', 22) only

C. ('Alice', 30) and ('Carol', 27)

D. No output

Solution

1. Step 1: Understand the FILTER condition

   The script filters records where age is greater than 25.

2. Step 2: Apply filter to input data

   'Alice' (30) and 'Carol' (27) satisfy age > 25; 'Bob' (22) does not.

3. Final Answer:

   ('Alice', 30) and ('Carol', 27) -> Option C

4. Quick Check:

   Filter age > 25 = Alice and Carol [OK]

Hint: Filter keeps only records matching condition [OK]

Common Mistakes:

• Including records with age equal to 25
• Confusing FILTER with LOAD
• Assuming all records are output

4. Identify the error in this Hive query:

SELECT name age FROM users WHERE age > 20;

medium

A. Missing comma between selected columns.

B. Incorrect WHERE clause syntax.

C. Table name should be in quotes.

D. SELECT keyword is misspelled.

Solution

1. Step 1: Analyze the SELECT clause

   The query selects two columns: name and age. They must be separated by a comma.

2. Step 2: Check the query syntax

   The query misses a comma between 'name' and 'age'. The WHERE clause and table name are correct.

3. Final Answer:

   Missing comma between selected columns. -> Option A

4. Quick Check:

   Columns in SELECT need commas [OK]

Hint: Separate columns with commas in SELECT [OK]

Common Mistakes:

• Omitting commas between columns
• Adding quotes around table names unnecessarily
• Miswriting WHERE clause

5. You want to process a complex data flow involving multiple steps of filtering, grouping, and joining large datasets in Hadoop. Which tool is better suited and why?

hard

A. Hive, because it uses SQL and is easier for complex flows.

B. Pig, because it supports procedural scripting for complex data flows.

C. Hive, because it is procedural and faster for joins.

D. Pig, because it only supports simple queries.

Solution

1. Step 1: Understand tool strengths

   Pig is procedural and designed for complex data flows with multiple steps. Hive is declarative and better for simple querying.

2. Step 2: Match tool to task

   For complex filtering, grouping, and joining in multiple steps, Pig's scripting is more flexible and suitable.

3. Final Answer:

   Pig, because it supports procedural scripting for complex data flows. -> Option B

4. Quick Check:

   Complex flows = Pig scripting [OK]

Hint: Use Pig for complex multi-step data processing [OK]

Common Mistakes:

• Choosing Hive for complex procedural flows
• Thinking Pig only supports simple queries
• Confusing procedural and declarative