Compression codecs (Snappy, LZO, Gzip) in Hadoop - Mini Project: Build & Apply

Using Compression Codecs in Hadoop

📖 Scenario: You work with large data files in Hadoop. To save space and speed up processing, you want to compress files using different codecs: Snappy, LZO, and Gzip.

🎯 Goal: Learn how to set up and use compression codecs Snappy, LZO, and Gzip in Hadoop to compress and decompress data files.

📋 What You'll Learn

Create a sample text file in Hadoop filesystem

Set a variable for the compression codec

Write a Hadoop command to compress the file using the chosen codec

Display the compressed file size

💡 Why This Matters

🌍 Real World

Compression codecs help save storage space and speed up data processing in big data systems like Hadoop.

💼 Career

Knowing how to use compression codecs is important for data engineers and data scientists working with large datasets in Hadoop environments.

1

Create a sample text file in Hadoop filesystem

Use the Hadoop command hdfs dfs -put to upload a local file named sample.txt with the exact content:
Hello Hadoop Compression Codecs
to the Hadoop directory /user/hadoop/input/.

Hadoop

# Use the command to upload sample.txt to /user/hadoop/input/
# Your code here

Hint

First create the directory, then upload the file using hdfs dfs -put.

2

Set the compression codec variable

Create a shell variable called codec and set it to the exact value snappy to specify the compression codec.

Hadoop

hdfs dfs -mkdir -p /user/hadoop/input/
echo "Hello Hadoop Compression Codecs" > sample.txt
hdfs dfs -put sample.txt /user/hadoop/input/
# Set the codec variable to snappy
# Your code here

Hint

Use codec=snappy to set the variable.

3

Compress the file using the chosen codec

Use the hadoop jar command with the variable $codec to compress the file /user/hadoop/input/sample.txt into /user/hadoop/output/sample.txt.$codec (a directory containing the compressed file). Use the appropriate codec option.

Hadoop

hdfs dfs -mkdir -p /user/hadoop/input/
echo "Hello Hadoop Compression Codecs" > sample.txt
hdfs dfs -put sample.txt /user/hadoop/input/
codec=snappy
# Compress the file using the $codec variable
# Your code here

Hint

Use -D mapreduce.output.fileoutputformat.compress=true and set the codec class for Snappy.

4

Display the compressed file size

Use the Hadoop command hdfs dfs -du -h to display the size of the compressed file /user/hadoop/output/sample.txt.$codec.

Hadoop

hdfs dfs -mkdir -p /user/hadoop/input/
echo "Hello Hadoop Compression Codecs" > sample.txt
hdfs dfs -put sample.txt /user/hadoop/input/
codec=snappy
hadoop jar /usr/lib/hadoop-mapreduce/hadoop-streaming.jar \
  -D mapreduce.output.fileoutputformat.compress=true \
  -D mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.SnappyCodec \
  -input /user/hadoop/input/sample.txt \
  -output /user/hadoop/output/sample.txt.$codec \
  -mapper cat -reducer cat
# Show the compressed file size
# Your code here

Hint

Use hdfs dfs -du -h to check file size.

Practice

1. Which compression codec is known for the fastest compression and decompression speed in Hadoop?

easy

A. LZO

B. Gzip

C. Snappy

D. Bzip2

Solution

1. Step 1: Understand codec speed characteristics

   Snappy is designed for speed, offering fast compression and decompression.

2. Step 2: Compare with other codecs

   Gzip compresses better but slower; LZO is fast but usually slower than Snappy; Bzip2 is slowest.

3. Final Answer:

   Snappy -> Option C

4. Quick Check:

   Fastest codec = Snappy [OK]

Hint: Snappy is fastest codec for speed, not best compression [OK]

Common Mistakes:

• Confusing Gzip as fastest due to better compression
• Thinking LZO is faster than Snappy
• Assuming Bzip2 is fast

2. Which of the following is the correct way to enable Snappy compression in a Hadoop job configuration XML?

easy

A. mapreduce.output.fileoutputformat.compress.codecorg.apache.hadoop.io.compress.GzipCodec

B. mapreduce.output.fileoutputformat.compress.codecorg.apache.hadoop.io.compress.SnappyCodec

C. mapreduce.output.fileoutputformat.compress.codecorg.apache.hadoop.io.compress.Lz4Codec

D. mapreduce.output.fileoutputformat.compress.codecorg.apache.hadoop.io.compress.Bzip2Codec

Solution

1. Step 1: Identify the property for compression codec

   The property key is 'mapreduce.output.fileoutputformat.compress.codec' to set codec class.

2. Step 2: Match the codec class for Snappy

   Snappy codec class is 'org.apache.hadoop.io.compress.SnappyCodec'.

3. Final Answer:

   Property with SnappyCodec class -> Option B

4. Quick Check:

   Snappy codec class = org.apache.hadoop.io.compress.SnappyCodec [OK]

Hint: Snappy codec class ends with SnappyCodec [OK]

Common Mistakes:

• Using GzipCodec or other codec classes by mistake
• Misspelling the property name
• Confusing Lz4Codec with LZO

3. Given the following Hadoop job configuration snippet, what compression codec will be used?

conf.set("mapreduce.output.fileoutputformat.compress", "true");
conf.set("mapreduce.output.fileoutputformat.compress.codec", "org.apache.hadoop.io.compress.GzipCodec");

medium

A. Snappy

B. LZO

C. No compression

D. Gzip

Solution

1. Step 1: Check if compression is enabled

   Property 'mapreduce.output.fileoutputformat.compress' is set to 'true', so compression is enabled.

2. Step 2: Identify the codec class used

   Codec class is 'org.apache.hadoop.io.compress.GzipCodec', which means Gzip compression is used.

3. Final Answer:

   Gzip -> Option D

4. Quick Check:

   Compression enabled with Gzip codec [OK]

Hint: Check compress property true + codec class name [OK]

Common Mistakes:

• Ignoring the compress flag and assuming no compression
• Confusing codec class with Snappy or LZO
• Assuming default codec if not checked

4. You configured LZO compression in your Hadoop job, but the job fails with a codec not found error. What is the most likely cause?

medium

A. LZO native libraries are not installed or not in the library path

B. The job configuration property name is misspelled

C. Snappy codec is enabled instead of LZO

D. Gzip compression is incompatible with Hadoop

Solution

1. Step 1: Understand LZO codec requirements

   LZO requires native libraries installed on all cluster nodes to work properly.

2. Step 2: Analyze error cause

   Codec not found error usually means native LZO libraries are missing or not accessible in the library path.

3. Final Answer:

   LZO native libraries missing or not in path -> Option A

4. Quick Check:

   LZO needs native libs installed [OK]

Hint: LZO needs native libs installed on cluster nodes [OK]

Common Mistakes:

• Assuming misspelled property causes codec not found
• Confusing Snappy and LZO codec errors
• Thinking Gzip is incompatible with Hadoop

5. You want to compress large Hadoop output files with the best compression ratio but can tolerate slower speed. Which codec should you choose and why?

hard

A. Gzip, because it offers the best compression ratio despite slower speed

B. LZO, because it balances speed and compression

C. Snappy, because it is fastest and good compression

D. No compression, to avoid overhead

Solution

1. Step 1: Identify compression needs

   Best compression ratio means smallest file size, speed is less important.

2. Step 2: Compare codecs by compression ratio and speed

   Gzip compresses better but slower; Snappy is fastest but less compression; LZO is in between.

3. Step 3: Choose codec matching needs

   Gzip fits best for high compression ratio with slower speed tolerance.

4. Final Answer:

   Gzip, best compression ratio with slower speed -> Option A

5. Quick Check:

   Best compression ratio = Gzip [OK]

Hint: Best compression ratio usually means Gzip codec [OK]

Common Mistakes:

• Choosing Snappy for best compression ratio
• Thinking LZO compresses better than Gzip
• Avoiding compression despite need to save space