GROUP and JOIN operations in Hadoop - Step-by-Step Execution

Concept Flow - GROUP and JOIN operations

Input Data

↓

Map Phase

↓

Shuffle & Sort

↓

Group by Key

↓

Join Keys

→Reduce Phase for JOIN

↓

Output Data

Data flows from input through map, then shuffle groups data by keys, then reduce performs GROUP aggregation or JOIN combining.

Execution Sample

Hadoop

map(key, value):
  emit(key, value)

reduce(key, values):
  if GROUP:
    emit(key, aggregate(values))
  if JOIN:
    emit(key, combine(values_from_both_datasets))

Map emits key-value pairs; reduce aggregates values for GROUP or combines values for JOIN by key.

Execution Table

Step	Phase	Input	Action	Output
1	Map	Dataset1: (A,1), (B,2)	Emit key-value pairs	(A,1), (B,2)
2	Map	Dataset2: (A,3), (C,4)	Emit key-value pairs	(A,3), (C,4)
3	Shuffle & Sort	(A,1), (B,2), (A,3), (C,4)	Group by key	A: [1,3], B: [2], C: [4]
4	Reduce (GROUP)	A: [1,3]	Sum values	(A,4)
5	Reduce (GROUP)	B: [2]	Sum values	(B,2)
6	Reduce (GROUP)	C: [4]	Sum values	(C,4)
7	Reduce (JOIN)	A: values from both datasets	Combine values	(A,(1,3))
8	Reduce (JOIN)	B: values only from Dataset1	No match in Dataset2	(B,(2,null))
9	Reduce (JOIN)	C: values only from Dataset2	No match in Dataset1	(C,(null,4))
10	End	All keys processed	Output final grouped or joined data	GROUP output: (A,4), (B,2), (C,4) JOIN output: (A,(1,3)), (B,(2,null)), (C,(null,4))

💡 All keys processed, reduce phase completes GROUP aggregation or JOIN combination.

Variable Tracker

Variable	Start	After Map 1	After Map 2	After Shuffle	After Reduce GROUP	After Reduce JOIN
Dataset1	[(A,1), (B,2)]	[(A,1), (B,2)]	[(A,1), (B,2)]	Grouped by key	Aggregated sums	Used for join
Dataset2	[(A,3), (C,4)]	[(A,3), (C,4)]	[(A,3), (C,4)]	Grouped by key	Aggregated sums	Used for join
Grouped Data	{}	{}	{}	{A:[1,3], B:[2], C:[4]}	{A:4, B:2, C:4}	{A:(1,3), B:(2,null), C:(null,4)}

Key Moments - 3 Insights

Why do keys appear multiple times before the reduce phase?

Because the map phase emits key-value pairs for each record, keys repeat until shuffle groups them together as shown in execution_table step 3.

How does the reduce phase know whether to do GROUP or JOIN?

The reduce function logic distinguishes GROUP by aggregating values or JOIN by combining values from multiple datasets, as seen in steps 4-9.

What happens if a key exists in only one dataset during JOIN?

The reduce phase outputs the key with null for missing dataset values, shown in steps 8 and 9.

Visual Quiz - 3 Questions

Look at the execution table at step 3. What is the grouped output for key 'A'?

A [2]

B [1,3]

C [4]

D [1,2,3]

💡 Hint

Check the 'Shuffle & Sort' phase output in execution_table row 3.

At which step does the reduce phase output the sum for key 'B' in GROUP operation?

A Step 7

B Step 3

C Step 5

D Step 9

💡 Hint

Look for 'Reduce (GROUP)' phase output for key 'B' in execution_table.

If Dataset2 had no key 'C', how would the JOIN output for 'C' change?

A No output for 'C'

B (C,(null,4))

C (C,(null,null))

D (C,(null,null)) with empty values

💡 Hint

Refer to execution_table steps 8 and 9 for keys missing in one dataset.

Concept Snapshot

GROUP and JOIN in Hadoop:
- Map emits key-value pairs
- Shuffle groups data by keys
- Reduce aggregates for GROUP (e.g., sum)
- Reduce combines datasets for JOIN
- Keys missing in one dataset get nulls in JOIN
- Output is grouped or joined data by key

Full Transcript

This visual execution shows how GROUP and JOIN operations work in Hadoop. First, the map phase emits key-value pairs from input datasets. Then, shuffle and sort group these pairs by key. In the reduce phase, for GROUP operations, values for each key are aggregated, such as summing numbers. For JOIN operations, values from both datasets are combined by key, with nulls used when a key is missing in one dataset. The execution table traces each step, showing inputs, actions, and outputs. Variable tracking shows how data changes from start to finish. Key moments clarify common confusions about repeated keys, reduce logic, and handling missing keys. The visual quiz tests understanding of grouping, reduce outputs, and join behavior with missing keys. This step-by-step trace helps beginners see exactly how Hadoop processes GROUP and JOIN operations.