Point-in-time correctness in MLOps - Commands & Configuration

Introduction

Point-in-time correctness means making sure your machine learning model and data match exactly at the same moment. This helps avoid mistakes when you check or use your model later.

When you want to compare model results with the exact data used to train it.

When you need to reproduce a model's prediction exactly as it was made before.

When you want to audit or debug a model's behavior at a specific time.

When you deploy a model and want to ensure it uses the same data snapshot as during training.

When you track experiments and want to keep data and model versions aligned.

Commands

This command runs an MLflow project specifying the exact data version and model version to ensure point-in-time correctness.

Terminal

mlflow run . -P data_version=2024-06-01 -P model_version=1.0

Expected Output

2024/06/01 12:00:00 INFO mlflow.projects: === Run (ID='123abc') succeeded ===

→

-P - Passes parameters to specify exact data and model versions

Downloads the exact model artifacts from the run to verify or use the model matching the data snapshot.

Terminal

mlflow artifacts download -r 123abc -d ./downloaded_model

Expected Output

Successfully downloaded artifacts to: ./downloaded_model

Starts a local server to serve the downloaded model for testing or deployment, ensuring the model matches the point-in-time data.

Terminal

mlflow models serve -m ./downloaded_model --no-conda

Expected Output

2024/06/01 12:01:00 INFO mlflow.models: Serving model at http://127.0.0.1:5000

→

--no-conda - Avoids creating a new environment, using current setup

Key Concept

If you remember nothing else from this pattern, remember: always link your model version with the exact data snapshot to avoid mismatches.

Code Example

MLOps

import mlflow

# Log data version and model version as tags
with mlflow.start_run() as run:
    mlflow.set_tag("data_version", "2024-06-01")
    mlflow.set_tag("model_version", "1.0")
    # Log a simple metric
    mlflow.log_metric("accuracy", 0.95)
    print(f"Run ID: {run.info.run_id} logged with point-in-time correctness tags")

Common Mistakes

Using the latest model without specifying the data version

This causes the model to be tested or deployed with data it was not trained on, leading to wrong results.

Always specify both model and data versions together when running or deploying.

Not downloading the exact model artifacts before serving

Serving a different or outdated model can cause inconsistent predictions.

Download and serve the model artifacts from the exact run that used the matching data.

Summary

Run MLflow projects specifying exact data and model versions to keep them aligned.

Download model artifacts from the specific run to verify or deploy the correct model.

Serve the downloaded model to ensure predictions match the data snapshot used during training.

Practice

1.

What does point-in-time correctness ensure in MLOps?

easy

A. Using all available data including future data for better accuracy

B. Ignoring timestamps in data processing

C. Using only data available up to a specific moment to avoid future data leaks

D. Using random data samples without time consideration

Solution

1. Step 1: Understand the concept of point-in-time correctness

   It means using data only up to a certain moment to avoid using future information.

2. Step 2: Identify the correct practice

   Using future data can cause wrong model results, so only past and present data should be used.

3. Final Answer:

   Using only data available up to a specific moment to avoid future data leaks -> Option C

4. Quick Check:

   Point-in-time correctness = Use past data only [OK]

Hint: Remember: no peeking into future data for training [OK]

Common Mistakes:

• Using future data accidentally
• Ignoring timestamps in data
• Assuming all data is valid regardless of time

2.

Which of the following is the correct way to filter data for point-in-time correctness using SQL?

SELECT * FROM sales WHERE sale_date <= '2023-01-01'

easy

A. SELECT * FROM sales WHERE sale_date <= '2023-01-01'

B. SELECT * FROM sales WHERE sale_date > '2023-01-01'

C. SELECT * FROM sales WHERE sale_date = '2023-01-01'

D. SELECT * FROM sales WHERE sale_date >= '2023-01-01'

Solution

1. Step 1: Understand filtering for point-in-time correctness

   We want data up to and including the date '2023-01-01'.

2. Step 2: Choose the correct SQL condition

   The condition should be sale_date less than or equal to '2023-01-01' to include all past data.

3. Final Answer:

   SELECT * FROM sales WHERE sale_date <= '2023-01-01' -> Option A

4. Quick Check:

   Use <= for up to a date [OK]

Hint: Use <= to include data up to the cutoff date [OK]

Common Mistakes:

• Using > instead of <=
• Filtering only exact date instead of all past data
• Using >= which includes future data

3.

Given the following Python code snippet for filtering data by timestamp, what will be the output?

data = [
  {'id': 1, 'timestamp': '2023-01-01'},
  {'id': 2, 'timestamp': '2023-02-01'},
  {'id': 3, 'timestamp': '2022-12-31'}
]
cutoff = '2023-01-01'
filtered = [d['id'] for d in data if d['timestamp'] <= cutoff]
print(filtered)

medium

A. [3]

B. [1, 2, 3]

C. [2]

D. [1, 3]

Solution

1. Step 1: Analyze the filtering condition

   We keep items where timestamp is less than or equal to '2023-01-01'.

2. Step 2: Check each item

   Item 1: '2023-01-01' <= '2023-01-01' (True), Item 2: '2023-02-01' <= '2023-01-01' (False), Item 3: '2022-12-31' <= '2023-01-01' (True).

3. Final Answer:

   [1, 3] -> Option D

4. Quick Check:

   Filter by <= cutoff date = [1, 3] [OK]

Hint: Compare timestamps as strings for ISO format dates [OK]

Common Mistakes:

• Including future dates mistakenly
• Confusing < and <=
• Ignoring date format in comparison

4.

Identify the error in this code snippet that tries to enforce point-in-time correctness:

def filter_data(data, cutoff):
    return [d for d in data if d['timestamp'] > cutoff]

# cutoff = '2023-01-01'

medium

A. The list comprehension syntax is incorrect

B. The comparison should be <= cutoff, not > cutoff

C. The cutoff variable is not defined

D. The function should return all data without filtering

Solution

1. Step 1: Understand the filtering logic

   Point-in-time correctness requires data up to the cutoff date, so timestamps should be less than or equal to cutoff.

2. Step 2: Identify the error in comparison

   The code uses > cutoff, which selects future data instead of past data.

3. Final Answer:

   The comparison should be <= cutoff, not > cutoff -> Option B

4. Quick Check:

   Use <= cutoff to filter past data [OK]

Hint: Filter with <= cutoff, not > cutoff [OK]

Common Mistakes:

• Using > instead of <=
• Ignoring cutoff definition
• Incorrect list comprehension syntax

5.

You have a dataset with multiple features collected over time. You want to create a feature store snapshot that guarantees point-in-time correctness for model training on 2023-03-01. Which approach is best?

hard

A. Filter all features to include only data with timestamps <= '2023-03-01' and save as snapshot

B. Include data with timestamps > '2023-03-01' to improve model accuracy

C. Use the latest data available regardless of timestamp

D. Randomly sample data without considering timestamps

Solution

1. Step 1: Understand snapshot purpose

   A snapshot should represent data exactly as it was up to the training date to avoid future data leaks.

2. Step 2: Choose filtering strategy

   Filtering all features with timestamps less than or equal to '2023-03-01' ensures point-in-time correctness.

3. Step 3: Save filtered data as snapshot

   This snapshot can be used safely for training without future data contamination.

4. Final Answer:

   Filter all features to include only data with timestamps <= '2023-03-01' and save as snapshot -> Option A

5. Quick Check:

   Snapshot = Filter by cutoff date [OK]

Hint: Snapshot = data filtered by cutoff timestamp [OK]

Common Mistakes:

• Using future data in snapshot
• Ignoring timestamp filtering
• Random sampling without time consideration