Audit trails for model decisions in MLOps - Mini Project: Build & Apply

Audit Trails for Model Decisions

📖 Scenario: You work in a team that builds machine learning models. Your team wants to keep a clear record of every decision the model makes. This helps to check and understand the model's behavior later, like keeping a diary of its choices.

🎯 Goal: Build a simple program that logs each model decision with details like input data, prediction, and timestamp. This log acts as an audit trail to track model decisions over time.

📋 What You'll Learn

Create a list to store audit trail entries

Add a configuration variable for the model name

Write a function to log decisions with input, prediction, and timestamp

Print the audit trail entries

💡 Why This Matters

🌍 Real World

Audit trails help teams track and review machine learning model decisions. This is important for debugging, compliance, and improving trust in AI systems.

💼 Career

Knowing how to implement audit trails is useful for roles in MLOps, data science, and software engineering where model transparency and accountability are required.

1

Create the audit trail list

Create a list called audit_trail that will hold all the model decision records.

MLOps

# Create a list called audit_trail
# Your code here

Hint

Use square brackets [] to create an empty list named audit_trail.

2

Add model name configuration

Create a variable called model_name and set it to the string "SimpleClassifierV1".

MLOps

audit_trail = []
# Create a variable model_name with value "SimpleClassifierV1"
# Your code here

Hint

Use an equals sign = to assign the string "SimpleClassifierV1" to model_name.

3

Write the logging function

Write a function called log_decision that takes input_data and prediction as parameters. Inside the function, import datetime, get the current time with datetime.datetime.now(), and append a dictionary with keys model, input, prediction, and timestamp to the audit_trail list.

MLOps

audit_trail = []
model_name = "SimpleClassifierV1"
# Define function log_decision(input_data, prediction)
# Your code here

Hint

Remember to import datetime inside the function and use audit_trail.append() to add the record.

4

Log a decision and print the audit trail

Call the log_decision function with input_data set to {"feature1": 5, "feature2": 3} and prediction set to "ClassA". Then print the audit_trail list.

MLOps

audit_trail = []
model_name = "SimpleClassifierV1"

def log_decision(input_data, prediction):
    import datetime
    now = datetime.datetime.now()
    audit_trail.append({
        "model": model_name,
        "input": input_data,
        "prediction": prediction,
        "timestamp": now
    })

# Call log_decision with input_data and prediction
# Print audit_trail
# Your code here

Hint

The printed list should contain one dictionary with the model name, input data, prediction, and a timestamp starting with datetime.datetime.

Practice

1. What is the main purpose of audit trails in machine learning model decisions?

easy

A. To encrypt the model data for security

B. To speed up the model training process

C. To reduce the size of the model

D. To record inputs, outputs, and context for each model decision

Solution

1. Step 1: Understand audit trail purpose

   Audit trails are used to keep a record of what data was input, what output was produced, and the context around the decision.

2. Step 2: Compare options

   Only To record inputs, outputs, and context for each model decision describes this purpose correctly. Other options describe unrelated tasks.

3. Final Answer:

   To record inputs, outputs, and context for each model decision -> Option D

4. Quick Check:

   Audit trails = record inputs and outputs [OK]

Hint: Audit trails track what goes in and out of models [OK]

Common Mistakes:

• Confusing audit trails with model optimization
• Thinking audit trails speed up training
• Believing audit trails encrypt data

2. Which of the following is the correct way to log a model decision with timestamp in Python?

easy

A. log_entry = f"{datetime.now()} - Input: {input_data}, Output: {output}"

B. log_entry = datetime.now() + input_data + output

C. log_entry = "Input: input_data, Output: output"

D. log_entry = f"Input: {input_data} Output: {output}"

Solution

1. Step 1: Check correct string formatting with timestamp

   log_entry = f"{datetime.now()} - Input: {input_data}, Output: {output}" uses f-string with datetime.now() to include timestamp and variables properly.

2. Step 2: Identify errors in other options

   log_entry = datetime.now() + input_data + output tries to add incompatible types, causing error. Options C and D miss timestamp or variable interpolation.

3. Final Answer:

   log_entry = f"{datetime.now()} - Input: {input_data}, Output: {output}" -> Option A

4. Quick Check:

   Use f-string with datetime.now() for logging [OK]

Hint: Use f-strings and datetime.now() for timestamped logs [OK]

Common Mistakes:

• Concatenating incompatible types without conversion
• Forgetting to include timestamp
• Not using variable interpolation in strings

3. Given the following Python code snippet for logging model decisions, what will be the output?

from datetime import datetime
input_data = {'age': 30}
output = 'approved'
log_entry = f"{datetime(2024, 6, 1, 12, 0)} - Input: {input_data}, Output: {output}"
print(log_entry)

medium

A. 2024/06/01 12:00 - Input: {'age': 30}, Output: approved

B. datetime.datetime(2024, 6, 1, 12, 0) - Input: {'age': 30}, Output: approved

C. 2024-06-01 12:00:00 - Input: {'age': 30}, Output: approved

D. Error: datetime object cannot be formatted in f-string

Solution

1. Step 1: Understand datetime object formatting in f-string

   Using datetime(2024, 6, 1, 12, 0) in f-string calls its __str__ method, which outputs '2024-06-01 12:00:00'.

2. Step 2: Combine string parts

   The rest of the string includes input_data and output as expected, so the full string prints correctly.

3. Final Answer:

   2024-06-01 12:00:00 - Input: {'age': 30}, Output: approved -> Option C

4. Quick Check:

   Datetime __str__ = 'YYYY-MM-DD HH:MM:SS' [OK]

Hint: Datetime prints as 'YYYY-MM-DD HH:MM:SS' in f-strings [OK]

Common Mistakes:

• Expecting datetime object to print as constructor call
• Confusing date formats
• Thinking f-string cannot handle datetime objects

4. You have this code snippet to log model decisions but it raises an error:

log_entry = f"{datetime.now()} - Input: {input_data}, Output: {output}"

What is the most likely cause of the error?

medium

A. datetime module is not imported

B. input_data is not defined

C. f-string syntax is incorrect

D. output variable is a number, not a string

Solution

1. Step 1: Check for datetime usage

   Using datetime.now() requires importing datetime module or class. If missing, NameError occurs.

2. Step 2: Verify other variables and syntax

   input_data and output can be any type; f-string handles them. Syntax is correct.

3. Final Answer:

   datetime module is not imported -> Option A

4. Quick Check:

   Missing import datetime causes NameError [OK]

Hint: Always import datetime before using datetime.now() [OK]

Common Mistakes:

• Assuming variables cause error without checking imports
• Thinking f-string syntax is wrong
• Believing numbers cause f-string errors

5. You want to create an audit trail that records model version, input data, output, and timestamp in JSON format for each decision. Which Python code snippet correctly creates this audit trail entry?

hard

A. import json, datetime
audit_entry = json.dumps({"model_version": "v1.2", "input": input_data, "output": output, "timestamp": datetime.datetime.now.isoformat()})

B. import json, datetime
audit_entry = json.dumps({"model_version": "v1.2", "input": input_data, "output": output, "timestamp": datetime.now().isoformat()})

C. import json, datetime
audit_entry = json.dumps({"model_version": "v1.2", "input": input_data, "output": output, "timestamp": datetime.now().str()})

D. import json, datetime
audit_entry = json.dumps({"model_version": "v1.2", "input": input_data, "output": output, "timestamp": datetime.now()})

Solution

1. Step 1: Check correct import and datetime usage

   import json, datetime audit_entry = json.dumps({"model_version": "v1.2", "input": input_data, "output": output, "timestamp": datetime.now().isoformat()}) correctly imports datetime and uses datetime.now().isoformat() to get a string timestamp.

2. Step 2: Validate JSON serialization

   datetime.now() returns a datetime object which is not JSON serializable directly, so isoformat() converts it to string. import json, datetime audit_entry = json.dumps({"model_version": "v1.2", "input": input_data, "output": output, "timestamp": datetime.now()}) fails here.

3. Step 3: Check other options

   import json, datetime audit_entry = json.dumps({"model_version": "v1.2", "input": input_data, "output": output, "timestamp": datetime.datetime.now.isoformat()}) tries to call isoformat on the now method object (missing () after now), causing AttributeError. import json, datetime audit_entry = json.dumps({"model_version": "v1.2", "input": input_data, "output": output, "timestamp": datetime.now().str()}) tries to call .str() on datetime object, causing AttributeError.

4. Final Answer:

   import json, datetime audit_entry = json.dumps({"model_version": "v1.2", "input": input_data, "output": output, "timestamp": datetime.now().isoformat()}) -> Option B

5. Quick Check:

   Use datetime.now().isoformat() for JSON timestamp [OK]

Hint: Use datetime.now().isoformat() for JSON-friendly timestamps [OK]

Common Mistakes:

• Missing () after now() leading to method object error
• Trying to serialize datetime object directly
• Using non-existent .str() method on datetime