Bucket pattern for time-series data in MongoDB - Step-by-Step Execution

Concept Flow - Bucket pattern for time-series data

Receive time-series data

↓

Group data by time intervals

↓

Create bucket document for each interval

↓

Store multiple data points inside bucket

↓

Query buckets for time range

↓

Extract individual data points from buckets

Data points are grouped into time buckets to store many points in one document, improving write and query efficiency.

Execution Sample

MongoDB

db.sensorData.insertOne({
  bucketStart: ISODate("2024-06-01T00:00:00Z"),
  readings: [
    {time: ISODate("2024-06-01T00:01:00Z"), value: 20},
    {time: ISODate("2024-06-01T00:02:00Z"), value: 21}
  ]
})

Insert a bucket document containing multiple sensor readings within a 1-hour interval.

Execution Table

Step	Action	Data State	Result
1	Receive new data point at 00:01	No buckets yet	Prepare to create bucket for 00:00-01:00
2	Check if bucket for 00:00 exists	No bucket found	Create new bucket document with bucketStart 00:00
3	Insert data point into readings array	Bucket readings empty	Readings array now has 1 data point
4	Receive new data point at 00:02	Bucket for 00:00 exists with 1 reading	Add new data point to readings array
5	Query data for 00:00-01:00	One bucket with 2 readings	Return bucket document with both readings
6	Extract individual readings from bucket	Bucket readings array	Two separate data points available for analysis
7	Receive data point at 01:05	Bucket for 00:00 full, no bucket for 01:00	Create new bucket for 01:00 interval
8	Insert data point into new bucket	New bucket readings empty	Readings array now has 1 data point
9	Query data for 01:00-02:00	One bucket with 1 reading	Return bucket document with that reading
10	End of demonstration	Buckets created and queried	Efficient storage and retrieval achieved

💡 No more data points to process; buckets created and queried successfully.

Variable Tracker

Variable	Start	After Step 3	After Step 4	After Step 7	After Step 8	Final
buckets	{}	{"00:00": [{time: "00:01", value: 20}]}	{"00:00": [{time: "00:01", value: 20}, {time: "00:02", value: 21}]}	{"00:00": [...], "01:00": []}	{"00:00": [...], "01:00": [{time: "01:05", value: "X"}]}	{"00:00": [...], "01:00": [...] }

Key Moments - 3 Insights

Why do we group multiple time-series points into one bucket document?

Grouping reduces the number of documents MongoDB must handle, improving write speed and query efficiency, as shown in steps 2-4 where multiple points go into one bucket.

How do we know which bucket a new data point belongs to?

We calculate the bucket interval from the data point's timestamp, like 00:00-01:00 for points at 00:01 and 00:02, as seen in step 1 and 2.

What happens when a data point falls outside existing buckets?

A new bucket document is created for that time interval, shown in step 7 when a point at 01:05 triggers a new bucket creation.

Visual Quiz - 3 Questions

Look at the execution table, at which step is the first bucket document created?

A Step 3

B Step 1

C Step 2

D Step 4

💡 Hint

Check the 'Action' column for bucket creation in the execution_table.

According to variable_tracker, how many readings are in the bucket after Step 4?

A 1 reading

B 2 readings

C 0 readings

D 3 readings

💡 Hint

Look at the 'buckets' variable state after Step 4 in variable_tracker.

If a new data point at 00:30 is received, which bucket will it be added to?

A Bucket starting at 00:00

B Bucket starting at 01:00

C A new bucket starting at 00:30

D No bucket, data discarded

💡 Hint

Refer to the concept_flow and execution_table steps about bucket intervals.

Concept Snapshot

Bucket pattern groups many time-series points into one document per time interval.
Each bucket document has a start time and an array of readings.
New data points go into the bucket matching their time interval.
Queries retrieve buckets for a time range, then extract points.
This reduces document count and improves performance.

Full Transcript

The bucket pattern for time-series data stores many data points in one document per time interval. When new data arrives, the system checks if a bucket for that time interval exists. If not, it creates one. Then it adds the data point to the bucket's readings array. Queries fetch buckets for a time range and extract individual points. This method improves write and query efficiency by reducing the number of documents handled. The execution table shows steps from receiving data points, creating buckets, inserting readings, to querying buckets. The variable tracker shows how the buckets variable changes as points are added. Key moments clarify why grouping helps, how buckets are chosen, and what happens when data falls outside existing buckets. The visual quiz tests understanding of bucket creation, readings count, and bucket assignment.

Practice

1.

What is the main purpose of using the bucket pattern in MongoDB for time-series data?

easy

A. To group many time-series records into fewer documents for efficiency

B. To delete old time-series data automatically

C. To create indexes on every single time record

D. To convert time-series data into text format

Solution

1. Step 1: Understand the bucket pattern concept

   The bucket pattern groups multiple time-series records into fewer documents to reduce storage and improve query speed.

2. Step 2: Compare options with the bucket pattern purpose

   Only To group many time-series records into fewer documents for efficiency correctly describes grouping records into fewer documents for efficiency.

3. Final Answer:

   To group many time-series records into fewer documents for efficiency -> Option A

4. Quick Check:

   Bucket pattern groups records = A [OK]

Hint: Bucket pattern groups many records into fewer documents [OK]

Common Mistakes:

• Thinking bucket pattern deletes data automatically
• Confusing bucket pattern with indexing every record
• Believing bucket pattern changes data format

2.

Which of the following is the correct way to use $bucket to group documents by time ranges in MongoDB?

{ $bucket: { groupBy: "$timestamp", boundaries: [ISODate("2024-01-01"), ISODate("2024-02-01"), ISODate("2024-03-01")], default: "Other" } }

easy

A. Using $bucket without groupBy field

B. Using $bucket without boundaries array

C. Using $bucket with string boundaries instead of dates

D. Using $bucket with groupBy on timestamp and specifying boundaries as dates

Solution

1. Step 1: Check the required fields for $bucket

   The $bucket stage requires a groupBy field and a boundaries array to define bucket ranges.

2. Step 2: Validate the boundaries and groupBy usage

   Using $bucket with groupBy on timestamp and specifying boundaries as dates correctly uses groupBy on timestamp and boundaries as ISODate objects, which is valid for time-series bucketing.

3. Final Answer:

   Using $bucket with groupBy on timestamp and specifying boundaries as dates -> Option D

4. Quick Check:

   $bucket needs groupBy and boundaries = A [OK]

Hint: Always include groupBy and boundaries in $bucket stage [OK]

Common Mistakes:

• Omitting boundaries array causes error
• Using strings instead of date objects for time boundaries
• Missing groupBy field in $bucket stage

3.

Given the following aggregation pipeline using $bucket on a collection with timestamps:

[{ $bucket: { groupBy: "$timestamp", boundaries: [ISODate("2024-01-01"), ISODate("2024-02-01"), ISODate("2024-03-01")], default: "Other", output: { count: { $sum: 1 } } } }]

What will be the output if the collection has 5 documents with timestamps in January 2024 and 3 documents in March 2024?

medium

A. [{_id: ISODate("2024-01-01"), count: 5}, {_id: ISODate("2024-02-01"), count: 0}, {_id: "Other", count: 3}]

B. [{_id: ISODate("2024-01-01"), count: 5}, {_id: ISODate("2024-02-01"), count: 3}]

C. [{_id: ISODate("2024-01-01"), count: 8}]

D. [{_id: "Other", count: 8}]

Solution

1. Step 1: Understand bucket boundaries and grouping

   Documents with timestamps from 2024-01-01 to before 2024-02-01 go to first bucket; 2024-02-01 to before 2024-03-01 to second; others to default "Other".

2. Step 2: Assign documents to buckets based on timestamps

   5 docs in January fall in first bucket; 3 docs in March fall outside boundaries, so go to "Other"; second bucket has 0 docs.

3. Final Answer:

   [{_id: ISODate("2024-01-01"), count: 5}, {_id: ISODate("2024-02-01"), count: 0}, {_id: "Other", count: 3}] -> Option A

4. Quick Check:

   Docs in Jan = 5, in Other = 3, Feb bucket empty = C [OK]

Hint: Check which bucket each timestamp fits, others go to default [OK]

Common Mistakes:

• Ignoring the default bucket for out-of-bound timestamps
• Assuming March docs fall in February bucket
• Summing all docs into one bucket

4.

Consider this aggregation pipeline snippet using $bucket:

{ $bucket: { groupBy: "$time", boundaries: [ISODate("2024-01-01")], output: { count: { $sum: 1 } } } }

What is the error in this pipeline?

medium

A. The groupBy field name is incorrect; it should be 'timestamp'

B. The boundaries array is missing the upper boundary for the last bucket

C. The boundaries array must have at least two elements defining ranges

D. The output field 'count' is not allowed in $bucket stage

Solution

1. Step 1: Check boundaries array requirements

   The boundaries array must have at least two elements to define bucket ranges properly, including an upper boundary for the last bucket.

2. Step 2: Identify missing upper boundary

   The boundaries array has only one date, so it cannot define any bucket ranges, causing an error.

3. Final Answer:

   The boundaries array must have at least two elements defining ranges -> Option C

4. Quick Check:

   Boundaries must have at least 2 elements = A [OK]

Hint: Boundaries array must have one more element than buckets [OK]

Common Mistakes:

• Using boundaries array with fewer than two elements
• Assuming groupBy field name must be 'timestamp'
• Thinking output fields are not allowed in $bucket

5.

You want to store sensor readings every minute for a month using the bucket pattern. Each bucket should hold data for one hour. Which boundaries array correctly defines the buckets for January 1, 2024?

hard

A. [ISODate("2024-01-01T00:00:00Z"), ISODate("2024-01-01T00:59:59Z"), ISODate("2024-01-01T01:59:59Z")]

B. [ISODate("2024-01-01T00:00:00Z"), ISODate("2024-01-01T01:00:00Z"), ISODate("2024-01-01T02:00:00Z"), ..., ISODate("2024-01-01T23:00:00Z"), ISODate("2024-01-02T00:00:00Z")]

C. [ISODate("2024-01-01"), ISODate("2024-01-02")]

D. [ISODate("2024-01-01T00:00:00Z"), ISODate("2024-01-01T00:30:00Z"), ISODate("2024-01-01T01:00:00Z")]

Solution

1. Step 1: Understand bucket boundaries for hourly buckets

   Each bucket covers one hour, so boundaries must be every hour mark from midnight to next day midnight.

2. Step 2: Check options for correct hourly boundaries

   [ISODate("2024-01-01T00:00:00Z"), ISODate("2024-01-01T01:00:00Z"), ISODate("2024-01-01T02:00:00Z"), ..., ISODate("2024-01-01T23:00:00Z"), ISODate("2024-01-02T00:00:00Z")] lists hourly ISODate boundaries from 00:00 to 24:00, correctly defining 24 one-hour buckets.

3. Final Answer:

   [ISODate("2024-01-01T00:00:00Z"), ISODate("2024-01-01T01:00:00Z"), ISODate("2024-01-01T02:00:00Z"), ..., ISODate("2024-01-01T23:00:00Z"), ISODate("2024-01-02T00:00:00Z")] -> Option B

4. Quick Check:

   Hourly boundaries every hour for 24 hours = D [OK]

Hint: Boundaries must mark start of each hour plus next day start [OK]

Common Mistakes:

• Using half-hour or irregular intervals for hourly buckets
• Using only start and end of day as boundaries
• Using end times like 00:59:59 instead of next hour start