Bucketing for sampling in Hadoop - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Bucketing Purpose in Hadoop Sampling

Why is bucketing used in Hadoop sampling when working with large datasets?

A To encrypt data buckets for secure sampling

B To compress data files to save storage space during sampling

C To divide data into fixed-size files for parallel processing and enable efficient sampling

D To sort data alphabetically before sampling

💡 Hint

Think about how dividing data helps in processing and sampling.

Bucketing splits data into fixed buckets based on a hash of a column, which helps parallelize processing and allows sampling by selecting specific buckets efficiently.

❓ data_output

intermediate

Output of Sampling Using Buckets in Hive

Given a Hive table bucketed by user_id into 4 buckets, what is the output count when sampling bucket number 2 only?

Table data (user_id): [101, 102, 103, 104, 105, 106, 107, 108]

Hash function: bucket = hash(user_id) % 4

Hadoop

SELECT COUNT(*) FROM table WHERE input__file__name LIKE '%bucket_2%';

A 4

B 0

C 1

D 2

💡 Hint

Calculate hash(user_id) % 4 for each user_id and count how many fall into bucket 2.

Applying hash(user_id) % 4 to each user_id, only two user_ids fall into bucket 2, so the count is 2.

🔧 Debug

advanced

Identify the Error in Bucketing Sampling Query

What error will this Hive query produce?

SELECT * FROM users TABLESAMPLE(BUCKET 3 OUT OF 5) WHERE age > 30;

A SyntaxError: TABLESAMPLE clause must come after WHERE clause

B No error, query runs successfully

C RuntimeError: Sampling on non-bucketed table

D SemanticError: BUCKET number cannot be greater than total buckets

💡 Hint

Check the order of clauses and bucket numbers.

The query is valid if the table is bucketed into 5 buckets. TABLESAMPLE clause can appear before WHERE. No error occurs.

❓ Predict Output

advanced

Result of Sampling Buckets in Hadoop MapReduce Job

What is the output of this pseudocode for sampling bucket 1 from a bucketed dataset with 3 buckets?

for record in dataset:
    if hash(record.key) % 3 == 1:
        print(record.key)

Dataset keys: [10, 11, 12, 13, 14]

A 11, 14

B 10, 13

C None

D 12, 15

💡 Hint

Calculate hash(key) % 3 for each key and select those equal to 1.

Assuming hash(key) = key, keys 10 and 13 satisfy hash(key) % 3 == 1, so they are printed.

🚀 Application

expert

Choosing Buckets for Stratified Sampling in Hadoop

You have a large dataset bucketed by 'region' into 10 buckets. You want to perform stratified sampling to get a representative sample from each region. Which approach is best?

A Sample equal number of records from each bucket by reading all buckets and filtering by region

B Randomly select 3 buckets and use all data from those buckets as sample

C Use TABLESAMPLE(BUCKET 1 OUT OF 10) to get a sample from all regions

D Sample only from the bucket with the largest region to reduce data size

💡 Hint

Think about how to get balanced samples from all regions.

Sampling equal records from each bucket ensures representation from all regions, as buckets correspond to regions. Other options risk bias or incomplete coverage.

Practice

1. What is the main purpose of bucketing in Hadoop when working with big data?

easy

A. To split data into fixed parts based on a column for efficient sampling

B. To encrypt data for security

C. To compress data to save storage space

D. To sort data alphabetically

Solution

1. Step 1: Understand bucketing concept

   Bucketing divides data into fixed parts or buckets based on a column value.

2. Step 2: Identify purpose in big data

   This helps in sampling and faster query processing by limiting data scanned.

3. Final Answer:

   To split data into fixed parts based on a column for efficient sampling -> Option A

4. Quick Check:

   Bucketing = Fixed parts for sampling [OK]

Hint: Bucketing splits data into parts for easier sampling [OK]

Common Mistakes:

• Confusing bucketing with data encryption
• Thinking bucketing compresses data
• Assuming bucketing sorts data alphabetically

2. Which of the following is the correct syntax to create a bucketed table in Hive with 4 buckets on column user_id?

easy

A. CREATE TABLE users_bucketed (user_id INT, name STRING) DISTRIBUTED BY (user_id) INTO 4 BUCKETS;

B. CREATE TABLE users_bucketed (user_id INT, name STRING) PARTITIONED BY (user_id) INTO 4 BUCKETS;

C. CREATE TABLE users_bucketed (user_id INT, name STRING) CLUSTERED BY (user_id) INTO 4 BUCKETS;

D. CREATE TABLE users_bucketed (user_id INT, name STRING) SORTED BY (user_id) INTO 4 BUCKETS;

Solution

1. Step 1: Recall Hive bucketing syntax

   Hive uses CLUSTERED BY (column) INTO n BUCKETS to create buckets.

2. Step 2: Match syntax with options

   CREATE TABLE users_bucketed (user_id INT, name STRING) CLUSTERED BY (user_id) INTO 4 BUCKETS; matches the correct syntax for bucketing on user_id with 4 buckets.

3. Final Answer:

   CREATE TABLE users_bucketed (user_id INT, name STRING) CLUSTERED BY (user_id) INTO 4 BUCKETS; -> Option C

4. Quick Check:

   CLUSTERED BY + INTO BUCKETS = correct syntax [OK]

Hint: Use CLUSTERED BY with INTO BUCKETS for bucketing syntax [OK]

Common Mistakes:

• Using PARTITIONED BY instead of CLUSTERED BY
• Confusing DISTRIBUTED BY with bucketing syntax
• Using SORTED BY which is for sorting, not bucketing

3. Given a Hive table sales bucketed by region into 3 buckets, what will be the output of the query SELECT COUNT(*) FROM sales TABLESAMPLE(BUCKET 2 OUT OF 3) if the total rows are 3000 evenly distributed?

medium

A. 1000

B. 2000

C. 3000

D. 0

Solution

1. Step 1: Understand TABLESAMPLE BUCKET syntax

   TABLESAMPLE(BUCKET 2 OUT OF 3) samples 1/3 fraction of the data by selecting bucket number 2 out of 3 buckets.

2. Step 2: Calculate expected rows

   Total rows 3000 evenly split into 3 buckets = 1000 rows per bucket. Sampling bucket 2 out of 3 returns 1000 rows.

3. Final Answer:

   1000 -> Option A

4. Quick Check:

   Sampling bucket 2 out of 3 returns 1 bucket = 1000 rows [OK]

Hint: BUCKET x OUT OF y samples bucket number x only [OK]

Common Mistakes:

• Assuming sampling returns all rows
• Thinking BUCKET 2 OUT OF 3 selects 2 buckets (2000 rows)
• Ignoring even distribution assumption

4. Identify the error in the following Hive query for bucketing:

CREATE TABLE employees (id INT, name STRING) CLUSTERED BY id INTO BUCKETS 5;

medium

A. The number of buckets should come before the keyword BUCKETS

B. The keyword BUCKETS should come before the number 5

C. The keyword BUCKETS is missing

D. The syntax should be CLUSTERED BY (id) INTO 5 BUCKETS

Solution

1. Step 1: Review correct bucketing syntax

   Correct syntax is CLUSTERED BY (column) INTO n BUCKETS.

2. Step 2: Compare with given query

   Given query has CLUSTERED BY id INTO BUCKETS 5, which is incorrect order and missing parentheses.

3. Final Answer:

   The syntax should be CLUSTERED BY (id) INTO 5 BUCKETS -> Option D

4. Quick Check:

   Correct order: INTO number BUCKETS and parentheses around column [OK]

Hint: Use INTO number BUCKETS, not INTO BUCKETS number [OK]

Common Mistakes:

• Placing BUCKETS before the number
• Omitting parentheses around column
• Using wrong keyword order

5. You have a large table transactions bucketed by customer_id into 10 buckets. You want to sample approximately 30% of the data using bucketing. Which TABLESAMPLE clause should you use to get close to this percentage?

hard

A. TABLESAMPLE(BUCKET 4 OUT OF 10)

B. TABLESAMPLE(BUCKET 3 OUT OF 10)

C. TABLESAMPLE(BUCKET 5 OUT OF 10)

D. TABLESAMPLE(BUCKET 7 OUT OF 10)

Solution

1. Step 1: Understand sampling with buckets

   Sampling bucket n out of 10 selects only bucket number n, which is 1/10 of the data.

2. Step 2: Find bucket count for ~30%

   To sample approximately 30%, you need to select multiple buckets, but TABLESAMPLE(BUCKET x OUT OF y) selects only one bucket.

3. Step 3: Conclusion

   None of the options correctly sample ~30% because each selects only one bucket (10%).

4. Final Answer:

   None of the above options correctly sample ~30% -> Option B

Hint: TABLESAMPLE(BUCKET x OUT OF y) selects only one bucket, so cannot sample 30% with one bucket out of 10 [OK]

Common Mistakes:

• Choosing 3 or 4 buckets assuming it samples multiple buckets
• Ignoring that TABLESAMPLE(BUCKET x OUT OF y) selects only one bucket
• Assuming bucket numbers are percentages