Compression codecs (Snappy, LZO, Gzip) in Hadoop - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Which compression codec is best suited for fast decompression in Hadoop?

In Hadoop, you want to quickly read compressed data with minimal CPU overhead. Which compression codec is generally best for fast decompression?

A Snappy, because it is optimized for speed over compression ratio.

B Gzip, because it has the highest compression ratio.

C LZO, because it uses the least memory during decompression.

D No compression, to avoid decompression overhead.

💡 Hint

Think about which codec is designed for speed rather than maximum compression.

Snappy is designed to provide fast compression and decompression speeds, making it ideal for scenarios where speed is more important than maximum compression ratio.

❓ data_output

intermediate

What is the output size difference when compressing a 100MB text file with Gzip vs Snappy?

You compress the same 100MB text file using Gzip and Snappy codecs in Hadoop. Which statement correctly describes the expected compressed file sizes?

A Both files will be about the same size because they use similar algorithms.

B Gzip file will be smaller than Snappy file due to better compression ratio.

C Snappy file will be smaller than Gzip file because it compresses better.

D Snappy file will be larger than the original 100MB file.

💡 Hint

Consider which codec compresses better but is slower.

Gzip typically achieves a better compression ratio than Snappy, so the compressed file size will be smaller with Gzip.

🔧 Debug

advanced

Why does Hadoop fail to read LZO compressed files by default?

You try to read LZO compressed files in Hadoop but get errors. What is the most likely cause?

A LZO codec is not installed or native libraries are missing on the Hadoop nodes.

B LZO files are corrupted and cannot be decompressed.

C Hadoop does not support LZO compression at all.

D The input files are not compressed with LZO but with Gzip.

💡 Hint

Think about Hadoop's support for LZO and what is required to use it.

Hadoop requires native LZO libraries to be installed on all nodes to read LZO compressed files. Without these, decompression fails.

🚀 Application

advanced

Choosing the best codec for a Hadoop job with large binary files needing moderate compression and fast read times

You have large binary files to process in Hadoop. You want moderate compression to save space but also fast read times for analysis. Which codec should you choose?

A Gzip, for maximum compression despite slower reads.

B No compression, to avoid any decompression overhead.

C LZO, for a balance of compression and speed but requires setup.

D Snappy, for fast reads with moderate compression.

💡 Hint

Consider codecs that balance speed and compression but may need extra setup.

LZO offers a good balance between compression ratio and speed, making it suitable for large binary files where moderate compression and fast reads are needed. However, it requires native libraries installed.

❓ Predict Output

expert

What is the output of this Hadoop compression codec configuration snippet?

Given the following Hadoop configuration snippet, what compression codec will be used for output files?

Hadoop

conf.set("mapreduce.output.fileoutputformat.compress", "true")
conf.set("mapreduce.output.fileoutputformat.compress.codec", "org.apache.hadoop.io.compress.SnappyCodec")
conf.set("mapreduce.output.fileoutputformat.compress.type", "BLOCK")

A Output files will be compressed using LZO codec with record compression.

B Output files will be compressed using Gzip codec with block compression.

C Output files will not be compressed because compression type is invalid.

D Output files will be compressed using Snappy codec with block compression.

💡 Hint

Look carefully at the codec class name and compression type.

The configuration explicitly sets compression to true, uses SnappyCodec class, and block compression type, so output files will be compressed with Snappy in block mode.

Practice

1. Which compression codec is known for the fastest compression and decompression speed in Hadoop?

easy

A. LZO

B. Gzip

C. Snappy

D. Bzip2

Solution

1. Step 1: Understand codec speed characteristics

   Snappy is designed for speed, offering fast compression and decompression.

2. Step 2: Compare with other codecs

   Gzip compresses better but slower; LZO is fast but usually slower than Snappy; Bzip2 is slowest.

3. Final Answer:

   Snappy -> Option C

4. Quick Check:

   Fastest codec = Snappy [OK]

Hint: Snappy is fastest codec for speed, not best compression [OK]

Common Mistakes:

• Confusing Gzip as fastest due to better compression
• Thinking LZO is faster than Snappy
• Assuming Bzip2 is fast

2. Which of the following is the correct way to enable Snappy compression in a Hadoop job configuration XML?

easy

A. mapreduce.output.fileoutputformat.compress.codecorg.apache.hadoop.io.compress.GzipCodec

B. mapreduce.output.fileoutputformat.compress.codecorg.apache.hadoop.io.compress.SnappyCodec

C. mapreduce.output.fileoutputformat.compress.codecorg.apache.hadoop.io.compress.Lz4Codec

D. mapreduce.output.fileoutputformat.compress.codecorg.apache.hadoop.io.compress.Bzip2Codec

Solution

1. Step 1: Identify the property for compression codec

   The property key is 'mapreduce.output.fileoutputformat.compress.codec' to set codec class.

2. Step 2: Match the codec class for Snappy

   Snappy codec class is 'org.apache.hadoop.io.compress.SnappyCodec'.

3. Final Answer:

   Property with SnappyCodec class -> Option B

4. Quick Check:

   Snappy codec class = org.apache.hadoop.io.compress.SnappyCodec [OK]

Hint: Snappy codec class ends with SnappyCodec [OK]

Common Mistakes:

• Using GzipCodec or other codec classes by mistake
• Misspelling the property name
• Confusing Lz4Codec with LZO

3. Given the following Hadoop job configuration snippet, what compression codec will be used?

conf.set("mapreduce.output.fileoutputformat.compress", "true");
conf.set("mapreduce.output.fileoutputformat.compress.codec", "org.apache.hadoop.io.compress.GzipCodec");

medium

A. Snappy

B. LZO

C. No compression

D. Gzip

Solution

1. Step 1: Check if compression is enabled

   Property 'mapreduce.output.fileoutputformat.compress' is set to 'true', so compression is enabled.

2. Step 2: Identify the codec class used

   Codec class is 'org.apache.hadoop.io.compress.GzipCodec', which means Gzip compression is used.

3. Final Answer:

   Gzip -> Option D

4. Quick Check:

   Compression enabled with Gzip codec [OK]

Hint: Check compress property true + codec class name [OK]

Common Mistakes:

• Ignoring the compress flag and assuming no compression
• Confusing codec class with Snappy or LZO
• Assuming default codec if not checked

4. You configured LZO compression in your Hadoop job, but the job fails with a codec not found error. What is the most likely cause?

medium

A. LZO native libraries are not installed or not in the library path

B. The job configuration property name is misspelled

C. Snappy codec is enabled instead of LZO

D. Gzip compression is incompatible with Hadoop

Solution

1. Step 1: Understand LZO codec requirements

   LZO requires native libraries installed on all cluster nodes to work properly.

2. Step 2: Analyze error cause

   Codec not found error usually means native LZO libraries are missing or not accessible in the library path.

3. Final Answer:

   LZO native libraries missing or not in path -> Option A

4. Quick Check:

   LZO needs native libs installed [OK]

Hint: LZO needs native libs installed on cluster nodes [OK]

Common Mistakes:

• Assuming misspelled property causes codec not found
• Confusing Snappy and LZO codec errors
• Thinking Gzip is incompatible with Hadoop

5. You want to compress large Hadoop output files with the best compression ratio but can tolerate slower speed. Which codec should you choose and why?

hard

A. Gzip, because it offers the best compression ratio despite slower speed

B. LZO, because it balances speed and compression

C. Snappy, because it is fastest and good compression

D. No compression, to avoid overhead

Solution

1. Step 1: Identify compression needs

   Best compression ratio means smallest file size, speed is less important.

2. Step 2: Compare codecs by compression ratio and speed

   Gzip compresses better but slower; Snappy is fastest but less compression; LZO is in between.

3. Step 3: Choose codec matching needs

   Gzip fits best for high compression ratio with slower speed tolerance.

4. Final Answer:

   Gzip, best compression ratio with slower speed -> Option A

5. Quick Check:

   Best compression ratio = Gzip [OK]

Hint: Best compression ratio usually means Gzip codec [OK]

Common Mistakes:

• Choosing Snappy for best compression ratio
• Thinking LZO compresses better than Gzip
• Avoiding compression despite need to save space