OpenAI fine-tuning API in Prompt Engineering / GenAI - Model Metrics & Evaluation

Metrics & Evaluation - OpenAI fine-tuning API

Which metric matters for OpenAI fine-tuning API and WHY

When fine-tuning a language model with OpenAI's API, the key metric to watch is loss. Loss tells us how well the model predicts the next word during training. A lower loss means the model is learning the patterns in your data better.

Besides loss, if you have labeled data for tasks like classification, you can check accuracy, precision, and recall to see how well the model performs on your specific task.

Why loss? Because fine-tuning adjusts the model weights to reduce prediction errors. Watching loss helps you know if training is improving the model or if it's stuck.

Confusion matrix example for classification tasks

If your fine-tuned model does classification, you can use a confusion matrix to understand errors:

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP)  | False Positive (FP) |
      | False Negative (FN) | True Negative (TN)  |
    

For example, if your model predicts spam emails, TP means correctly flagged spam, FP means good emails wrongly flagged, FN means spam missed, and TN means good emails correctly allowed.

Precision vs Recall tradeoff with OpenAI fine-tuning

Imagine you fine-tune a model to detect spam. You want to avoid marking good emails as spam (high precision) but also want to catch most spam (high recall).

If you set the model to be very strict, it catches almost all spam (high recall) but may mark many good emails as spam (low precision).

If you set it to be very careful, it marks fewer good emails as spam (high precision) but misses some spam (low recall).

Fine-tuning lets you adjust this balance by changing training data or thresholds.

What good vs bad metric values look like for OpenAI fine-tuning

Good:

• Loss steadily decreases during training, showing learning progress.
• Accuracy, precision, and recall are balanced and high for your task (e.g., above 85%).
• Confusion matrix shows few false positives and false negatives.

Bad:

• Loss stays high or fluctuates wildly, meaning no learning.
• Accuracy is low or precision and recall are very unbalanced (e.g., 95% precision but 10% recall).
• Confusion matrix shows many errors, indicating poor predictions.

Common pitfalls in metrics for OpenAI fine-tuning

• Overfitting: Loss on training data goes down but validation loss goes up. Model memorizes instead of learning.
• Data leakage: Training data accidentally includes test examples, inflating metrics falsely.
• Ignoring class imbalance: High accuracy can be misleading if one class dominates.
• Using only accuracy: For imbalanced tasks, accuracy hides poor performance on minority classes.

Self-check question

Your fine-tuned model has 98% accuracy but only 12% recall on fraud detection. Is it good for production? Why or why not?

Answer: No, it is not good. The low recall means the model misses most fraud cases, which is dangerous. Even with high accuracy, missing fraud is costly. You should improve recall before using it in production.

Key Result

Loss is the key metric during fine-tuning; precision and recall matter for task-specific performance.

Practice

1. What is the main purpose of using the OpenAI fine-tuning API?

easy

A. To customize a base AI model with your own training data

B. To create a new AI model from scratch without any data

C. To delete existing AI models permanently

D. To convert AI models into images

Solution

1. Step 1: Understand fine-tuning concept

   Fine-tuning means adjusting a pre-trained AI model using your own data to make it better for your specific task.

2. Step 2: Identify the API's role

   The OpenAI fine-tuning API helps you upload your data and create a customized version of an existing model.

3. Final Answer:

   To customize a base AI model with your own training data -> Option A

4. Quick Check:

   Fine-tuning = Customize model with your data [OK]

Hint: Fine-tuning means customizing existing models with your data [OK]

Common Mistakes:

• Thinking fine-tuning creates models from scratch
• Confusing fine-tuning with deleting models
• Assuming fine-tuning changes model type (like image conversion)

2. Which of the following is the correct way to start a fine-tuning job using the OpenAI API in Python?

easy

A. openai.createFineTune(training='file-abc123')

B. openai.FineTune.create(training_file='file-abc123')

C. openai.fine_tune.start(file='file-abc123')

D. openai.finetune.upload(file='file-abc123')

Solution

1. Step 1: Recall OpenAI fine-tuning syntax

   The official OpenAI Python client uses openai.FineTune.create() to start fine-tuning jobs.

2. Step 2: Check parameter names

   The parameter for training data file is training_file, matching openai.FineTune.create(training_file='file-abc123') exactly.

3. Final Answer:

   openai.FineTune.create(training_file='file-abc123') -> Option B

4. Quick Check:

   Correct method and parameter = openai.FineTune.create(training_file='file-abc123') [OK]

Hint: Use openai.FineTune.create with training_file parameter [OK]

Common Mistakes:

• Using incorrect method names like fine_tune.start
• Wrong parameter names like 'file' instead of 'training_file'
• Mixing upload and create methods

3. Given this Python code snippet using OpenAI API, what will be the output?

response = openai.FineTune.create(training_file='file-xyz789')
print(response['status'])

medium

A. 'pending'

B. 'completed'

C. 'error'

D. 'unknown'

Solution

1. Step 1: Understand fine-tuning job lifecycle

   When a fine-tuning job is created, its initial status is usually 'pending' as it waits to start processing.

2. Step 2: Analyze code output

   The code prints the 'status' field from the response, which will be 'pending' immediately after creation.

3. Final Answer:

   'pending' -> Option A

4. Quick Check:

   New fine-tune job status = 'pending' [OK]

Hint: New fine-tune jobs start with status 'pending' [OK]

Common Mistakes:

• Assuming status is 'completed' right after creation
• Expecting 'error' without any failure
• Confusing status with model name

4. You wrote this code to fine-tune a model but get an error:

openai.FineTune.create(training_file='file-123')

What is the most likely cause of the error?

medium

A. The API key is missing from the code

B. The method name should be 'fine_tune.create' instead

C. You must specify the model parameter in create()

D. The training file ID 'file-123' is invalid or not uploaded

Solution

1. Step 1: Check common fine-tuning errors

   Errors often happen if the training file ID is wrong or the file was not uploaded properly.

2. Step 2: Validate method and parameters

   The method name and parameters are correct; model parameter is optional for fine-tuning base models.

3. Step 3: Consider API key

   Missing API key causes authentication errors, not file ID errors.

4. Final Answer:

   The training file ID 'file-123' is invalid or not uploaded -> Option D

5. Quick Check:

   Invalid file ID causes error [OK]

Hint: Check if training file ID is correct and uploaded [OK]

Common Mistakes:

• Using wrong method name with underscores
• Forgetting to upload training file before fine-tuning
• Assuming model parameter is always required

5. You want to fine-tune a model to improve chatbot responses for customer support. Which steps should you follow using the OpenAI fine-tuning API?

hard

A. Train a model locally without using OpenAI API

B. Directly call chat completions with the base model without uploading data

C. Upload a JSONL training file, create a fine-tune job with it, then use the new model for chat

D. Upload any text file and call openai.ChatCompletion.create without fine-tuning

Solution

1. Step 1: Prepare training data

   Fine-tuning requires a JSONL file with prompt-completion pairs relevant to customer support.

2. Step 2: Use OpenAI API to create fine-tune job

   Upload the file, then call openai.FineTune.create() with the training file ID.

3. Step 3: Use the fine-tuned model

   After training completes, use the new model for chat completions to get improved responses.

4. Final Answer:

   Upload a JSONL training file, create a fine-tune job with it, then use the new model for chat -> Option C

5. Quick Check:

   Fine-tune with data, then use new model [OK]

Hint: Upload data, fine-tune, then use new model for better chat [OK]

Common Mistakes:

• Skipping data upload and fine-tuning steps
• Trying to train models locally without OpenAI API
• Using base model without fine-tuning for custom tasks