Shuffle and sort phase in Hadoop - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to start the shuffle and sort phase in a MapReduce job.

Hadoop

job.setPartitionerClass([1]);

Drag options to blanks, or click blank then click option'

AHashPartitioner

BSortPartitioner

CShuffleSortPartitioner

DReducePartitioner

Attempts:

3 left

💡 Hint

Common Mistakes

Using a non-existent partitioner class.

Confusing partitioner with comparator.

The HashPartitioner is used by default to partition keys during shuffle and sort.

2fill in blank

medium

Complete the code to set the sort comparator class for the shuffle and sort phase.

Hadoop

job.setSortComparatorClass([1]);

Drag options to blanks, or click blank then click option'

AShuffleComparator

BWritableComparator

CText.Comparator

DIntWritable.Comparator

Attempts:

3 left

💡 Hint

Common Mistakes

Using a comparator that does not implement RawComparator.

Confusing comparator with partitioner.

The WritableComparator is commonly used to define custom sorting during shuffle and sort.

3fill in blank

hard

Fix the error in the code that merges sorted map outputs during shuffle.

Hadoop

MergeManager mergeManager = new MergeManager([1], mapOutputs);

Drag options to blanks, or click blank then click option'

Acontext

Bconfiguration

Cjob

Dconf

Attempts:

3 left

💡 Hint

Common Mistakes

Passing job or context instead of configuration.

Using undefined variables.

The conf variable holds the Hadoop configuration needed by MergeManager.

4fill in blank

hard

Fill both blanks to create a map output key-value pair and add it to the context during shuffle.

Hadoop

context.write(new [1](key), new [2](value));

Drag options to blanks, or click blank then click option'

AText

BIntWritable

CLongWritable

DFloatWritable

Attempts:

3 left

💡 Hint

Common Mistakes

Mixing up key and value types.

Using non-writable types.

Keys are often Text and values IntWritable in MapReduce shuffle phase.

5fill in blank

hard

Fill all three blanks to define a custom comparator for sorting keys in shuffle and sort phase.

Hadoop

public class CustomKeyComparator extends WritableComparator {
    protected CustomKeyComparator() {
        super([1].class, true);
    }

    @Override
    public int compare(WritableComparable w1, WritableComparable w2) {
        [2] key1 = ([3]) w1;
        [2] key2 = ([3]) w2;
        return key1.compareTo(key2);
    }
}

Drag options to blanks, or click blank then click option'

AText

CWritableComparable

DIntWritable

Attempts:

3 left

💡 Hint

Common Mistakes

Using different types in constructor and cast.

Casting to non-writable types.

The comparator extends WritableComparator for Text keys and casts accordingly.