Bucket pattern for time-series data in MongoDB - Mini Project: Build & Apply

Bucket Pattern for Time-Series Data in MongoDB

📖 Scenario: You are managing a temperature sensor system that records temperature readings every minute. To efficiently store and query this time-series data, you want to group readings into daily buckets in MongoDB.

🎯 Goal: Build a MongoDB collection that uses the bucket pattern to store temperature readings grouped by day. You will create the initial data, configure the bucket size, write the aggregation query to group data by day, and finalize the bucketed collection structure.

📋 What You'll Learn

Create a collection called temperature_readings with sample documents containing timestamp and temperature fields.

Define a variable bucketSize representing the number of milliseconds in one day.

Write an aggregation pipeline that groups readings by day using the bucket pattern.

Add the final stage to output the bucketed data with day and readings fields.

💡 Why This Matters

🌍 Real World

Time-series data like sensor readings, logs, or financial data often need efficient storage and querying. The bucket pattern groups data into time intervals to reduce document count and improve query speed.

💼 Career

Understanding how to organize and query time-series data is valuable for roles in data engineering, backend development, and database administration.

1

Create sample temperature readings

Insert three documents into the temperature_readings collection with these exact fields and values: { timestamp: new Date('2024-06-01T10:00:00Z'), temperature: 22 }, { timestamp: new Date('2024-06-01T15:00:00Z'), temperature: 24 }, and { timestamp: new Date('2024-06-02T09:00:00Z'), temperature: 20 }.

MongoDB

// Insert the three temperature reading documents into temperature_readings
// Your code here

Hint

Use insertMany on db.temperature_readings with an array of objects.

2

Define the bucket size for one day

Create a variable called bucketSize and set it to the number of milliseconds in one day (24 hours).

MongoDB

db.temperature_readings.insertMany([
  { timestamp: new Date('2024-06-01T10:00:00Z'), temperature: 22 },
  { timestamp: new Date('2024-06-01T15:00:00Z'), temperature: 24 },
  { timestamp: new Date('2024-06-02T09:00:00Z'), temperature: 20 }
])

// Define bucketSize as milliseconds in one day
// Your code here

Hint

Calculate milliseconds by multiplying hours, minutes, seconds, and 1000.

3

Write aggregation to group readings by day

Write an aggregation pipeline on temperature_readings that groups documents by day using $subtract and $mod on timestamp with bucketSize. Use $group with _id as the bucket start time and collect readings in an array called readings.

MongoDB

db.temperature_readings.insertMany([
  { timestamp: new Date('2024-06-01T10:00:00Z'), temperature: 22 },
  { timestamp: new Date('2024-06-01T15:00:00Z'), temperature: 24 },
  { timestamp: new Date('2024-06-02T09:00:00Z'), temperature: 20 }
])

const bucketSize = 24 * 60 * 60 * 1000;

// Write aggregation pipeline to group by day using bucket pattern
// Your code here

Hint

Use $group with _id as the bucket start timestamp calculated by subtracting the remainder of timestamp modulo bucketSize.

4

Add final stage to format bucketed data

Add a $project stage to the aggregation pipeline that outputs day as a Date from _id and includes the readings array. Assign the complete pipeline to a variable called bucketedData using db.temperature_readings.aggregate(pipeline).

MongoDB

db.temperature_readings.insertMany([
  { timestamp: new Date('2024-06-01T10:00:00Z'), temperature: 22 },
  { timestamp: new Date('2024-06-01T15:00:00Z'), temperature: 24 },
  { timestamp: new Date('2024-06-02T09:00:00Z'), temperature: 20 }
])

const bucketSize = 24 * 60 * 60 * 1000;

const pipeline = [
  {
    $group: {
      _id: {
        $subtract: [
          { $toLong: "$timestamp" },
          { $mod: [{ $toLong: "$timestamp" }, bucketSize] }
        ]
      },
      readings: { $push: "$temperature" }
    }
  }
];

// Add $project stage to convert _id to day and include readings
// Assign aggregation result to bucketedData
// Your code here

Hint

Use $project to rename _id to day as a date and keep readings. Then assign the aggregation cursor to bucketedData.

Practice

1.

What is the main purpose of using the bucket pattern in MongoDB for time-series data?

easy

A. To group many time-series records into fewer documents for efficiency

B. To delete old time-series data automatically

C. To create indexes on every single time record

D. To convert time-series data into text format

Solution

1. Step 1: Understand the bucket pattern concept

   The bucket pattern groups multiple time-series records into fewer documents to reduce storage and improve query speed.

2. Step 2: Compare options with the bucket pattern purpose

   Only To group many time-series records into fewer documents for efficiency correctly describes grouping records into fewer documents for efficiency.

3. Final Answer:

   To group many time-series records into fewer documents for efficiency -> Option A

4. Quick Check:

   Bucket pattern groups records = A [OK]

Hint: Bucket pattern groups many records into fewer documents [OK]

Common Mistakes:

• Thinking bucket pattern deletes data automatically
• Confusing bucket pattern with indexing every record
• Believing bucket pattern changes data format

2.

Which of the following is the correct way to use $bucket to group documents by time ranges in MongoDB?

{ $bucket: { groupBy: "$timestamp", boundaries: [ISODate("2024-01-01"), ISODate("2024-02-01"), ISODate("2024-03-01")], default: "Other" } }

easy

A. Using $bucket without groupBy field

B. Using $bucket without boundaries array

C. Using $bucket with string boundaries instead of dates

D. Using $bucket with groupBy on timestamp and specifying boundaries as dates

Solution

1. Step 1: Check the required fields for $bucket

   The $bucket stage requires a groupBy field and a boundaries array to define bucket ranges.

2. Step 2: Validate the boundaries and groupBy usage

   Using $bucket with groupBy on timestamp and specifying boundaries as dates correctly uses groupBy on timestamp and boundaries as ISODate objects, which is valid for time-series bucketing.

3. Final Answer:

   Using $bucket with groupBy on timestamp and specifying boundaries as dates -> Option D

4. Quick Check:

   $bucket needs groupBy and boundaries = A [OK]

Hint: Always include groupBy and boundaries in $bucket stage [OK]

Common Mistakes:

• Omitting boundaries array causes error
• Using strings instead of date objects for time boundaries
• Missing groupBy field in $bucket stage

3.

Given the following aggregation pipeline using $bucket on a collection with timestamps:

[{ $bucket: { groupBy: "$timestamp", boundaries: [ISODate("2024-01-01"), ISODate("2024-02-01"), ISODate("2024-03-01")], default: "Other", output: { count: { $sum: 1 } } } }]

What will be the output if the collection has 5 documents with timestamps in January 2024 and 3 documents in March 2024?

medium

A. [{_id: ISODate("2024-01-01"), count: 5}, {_id: ISODate("2024-02-01"), count: 0}, {_id: "Other", count: 3}]

B. [{_id: ISODate("2024-01-01"), count: 5}, {_id: ISODate("2024-02-01"), count: 3}]

C. [{_id: ISODate("2024-01-01"), count: 8}]

D. [{_id: "Other", count: 8}]

Solution

1. Step 1: Understand bucket boundaries and grouping

   Documents with timestamps from 2024-01-01 to before 2024-02-01 go to first bucket; 2024-02-01 to before 2024-03-01 to second; others to default "Other".

2. Step 2: Assign documents to buckets based on timestamps

   5 docs in January fall in first bucket; 3 docs in March fall outside boundaries, so go to "Other"; second bucket has 0 docs.

3. Final Answer:

   [{_id: ISODate("2024-01-01"), count: 5}, {_id: ISODate("2024-02-01"), count: 0}, {_id: "Other", count: 3}] -> Option A

4. Quick Check:

   Docs in Jan = 5, in Other = 3, Feb bucket empty = C [OK]

Hint: Check which bucket each timestamp fits, others go to default [OK]

Common Mistakes:

• Ignoring the default bucket for out-of-bound timestamps
• Assuming March docs fall in February bucket
• Summing all docs into one bucket

4.

Consider this aggregation pipeline snippet using $bucket:

{ $bucket: { groupBy: "$time", boundaries: [ISODate("2024-01-01")], output: { count: { $sum: 1 } } } }

What is the error in this pipeline?

medium

A. The groupBy field name is incorrect; it should be 'timestamp'

B. The boundaries array is missing the upper boundary for the last bucket

C. The boundaries array must have at least two elements defining ranges

D. The output field 'count' is not allowed in $bucket stage

Solution

1. Step 1: Check boundaries array requirements

   The boundaries array must have at least two elements to define bucket ranges properly, including an upper boundary for the last bucket.

2. Step 2: Identify missing upper boundary

   The boundaries array has only one date, so it cannot define any bucket ranges, causing an error.

3. Final Answer:

   The boundaries array must have at least two elements defining ranges -> Option C

4. Quick Check:

   Boundaries must have at least 2 elements = A [OK]

Hint: Boundaries array must have one more element than buckets [OK]

Common Mistakes:

• Using boundaries array with fewer than two elements
• Assuming groupBy field name must be 'timestamp'
• Thinking output fields are not allowed in $bucket

5.

You want to store sensor readings every minute for a month using the bucket pattern. Each bucket should hold data for one hour. Which boundaries array correctly defines the buckets for January 1, 2024?

hard

A. [ISODate("2024-01-01T00:00:00Z"), ISODate("2024-01-01T00:59:59Z"), ISODate("2024-01-01T01:59:59Z")]

B. [ISODate("2024-01-01T00:00:00Z"), ISODate("2024-01-01T01:00:00Z"), ISODate("2024-01-01T02:00:00Z"), ..., ISODate("2024-01-01T23:00:00Z"), ISODate("2024-01-02T00:00:00Z")]

C. [ISODate("2024-01-01"), ISODate("2024-01-02")]

D. [ISODate("2024-01-01T00:00:00Z"), ISODate("2024-01-01T00:30:00Z"), ISODate("2024-01-01T01:00:00Z")]

Solution

1. Step 1: Understand bucket boundaries for hourly buckets

   Each bucket covers one hour, so boundaries must be every hour mark from midnight to next day midnight.

2. Step 2: Check options for correct hourly boundaries

   [ISODate("2024-01-01T00:00:00Z"), ISODate("2024-01-01T01:00:00Z"), ISODate("2024-01-01T02:00:00Z"), ..., ISODate("2024-01-01T23:00:00Z"), ISODate("2024-01-02T00:00:00Z")] lists hourly ISODate boundaries from 00:00 to 24:00, correctly defining 24 one-hour buckets.

3. Final Answer:

   [ISODate("2024-01-01T00:00:00Z"), ISODate("2024-01-01T01:00:00Z"), ISODate("2024-01-01T02:00:00Z"), ..., ISODate("2024-01-01T23:00:00Z"), ISODate("2024-01-02T00:00:00Z")] -> Option B

4. Quick Check:

   Hourly boundaries every hour for 24 hours = D [OK]

Hint: Boundaries must mark start of each hour plus next day start [OK]

Common Mistakes:

• Using half-hour or irregular intervals for hourly buckets
• Using only start and end of day as boundaries
• Using end times like 00:59:59 instead of next hour start