Bucketing for sampling in Hadoop - Step-by-Step Execution

Concept Flow - Bucketing for sampling

Start with large dataset

↓

Define number of buckets

↓

Apply hash function on key

↓

Assign each record to a bucket

↓

Select specific bucket(s) for sampling

↓

Use sampled bucket data for analysis

↓

End

Data is split into fixed buckets using a hash on a key. Sampling is done by selecting one or more buckets.

Execution Sample

Hadoop

CREATE TABLE logs_bucketed(
  user_id STRING,
  action STRING
)
CLUSTERED BY (user_id) INTO 4 BUCKETS;

SELECT * FROM logs_bucketed TABLESAMPLE(BUCKET 2 OUT OF 4);

Create a bucketed table by user_id into 4 buckets, then sample bucket number 2 for analysis.

Execution Table

Step	Action	Input Data Example	Hash(user_id) % 4	Bucket Assigned	Sampling Decision
1	Read record	{user_id: 'alice', action: 'click'}	hash('alice') % 4 = 1	Bucket 1	No (sampling bucket is 2)
2	Read record	{user_id: 'bob', action: 'view'}	hash('bob') % 4 = 2	Bucket 2	Yes (bucket 2 selected)
3	Read record	{user_id: 'carol', action: 'click'}	hash('carol') % 4 = 3	Bucket 3	No
4	Read record	{user_id: 'dave', action: 'view'}	hash('dave') % 4 = 0	Bucket 0	No
5	Read record	{user_id: 'eve', action: 'click'}	hash('eve') % 4 = 2	Bucket 2	Yes
6	End of data	-	-	-	-

💡 All records processed; only records in bucket 2 are sampled.

Variable Tracker

Variable	Start	After 1	After 2	After 3	After 4	After 5	Final
Current Record	None	{alice, click}	{bob, view}	{carol, click}	{dave, view}	{eve, click}	None
Bucket Number	None	1	2	3	0	2	None
Sampled Records Count	0	0	1	1	1	2	2

Key Moments - 3 Insights

Why do only some records appear in the sample?

Because only records assigned to the selected bucket (bucket 2) are included, as shown in execution_table rows 2 and 5.

How is the bucket number decided for each record?

By applying a hash function on the key (user_id) and taking modulo with the number of buckets, as shown in the 'Hash(user_id) % 4' column.

What happens if we change the number of buckets?

The bucket assignment changes because modulo divisor changes, affecting which records fall into which bucket.

Visual Quiz - 3 Questions

Look at the execution table, which bucket does the record with user_id 'bob' belong to?

A Bucket 2

B Bucket 1

C Bucket 3

D Bucket 0

💡 Hint

Check the row with user_id 'bob' in the execution_table under 'Bucket Assigned'.

At which step does the sampled records count increase to 2?

A After step 3

B After step 4

C After step 5

D After step 2

💡 Hint

Look at 'Sampled Records Count' in variable_tracker after each step.

If we select bucket 3 instead of bucket 2, which record would be included in the sample?

A Records with user_id 'bob' and 'eve'

B Record with user_id 'carol'

C Record with user_id 'dave'

D Record with user_id 'alice'

💡 Hint

Check 'Bucket Assigned' column in execution_table for bucket 3.

Concept Snapshot

Bucketing splits data into fixed parts using a hash on a key.
Each record is assigned a bucket by hash(key) % number_of_buckets.
Sampling is done by selecting one or more buckets.
This method ensures consistent sampling of related records.
Useful for scalable sampling in big data systems like Hadoop.

Full Transcript

Bucketing for sampling means dividing a big dataset into smaller parts called buckets. We do this by using a hash function on a key, like user_id, and then taking the remainder when divided by the number of buckets. Each record goes into one bucket. When we want to sample data, we pick one or more buckets and use only the records inside them. This way, sampling is consistent and easy to repeat. For example, if we have 4 buckets and pick bucket 2, only records assigned to bucket 2 are used for analysis. This helps handle big data efficiently in Hadoop.