Model evaluation (TFMA) in TensorFlow - Model Metrics & Evaluation

Metrics & Evaluation - Model evaluation (TFMA)

Which metric matters for Model evaluation (TFMA) and WHY

When using TensorFlow Model Analysis (TFMA), the key metrics depend on your model's goal. For classification, common metrics are accuracy, precision, recall, and AUC. TFMA helps you see these metrics across different slices of data, like age groups or regions. This helps find if the model works well everywhere or only on some parts. For regression, metrics like mean squared error (MSE) or mean absolute error (MAE) matter. TFMA shows these metrics clearly so you can trust your model's performance in real life.

Confusion matrix or equivalent visualization

Confusion Matrix Example (Binary Classification):

          Predicted
          0     1
Actual 0  TN    FP
       1  FN    TP

Example numbers:

          Predicted
          0     1
Actual 0  85    15
       1  10    90

Total samples = 85 + 15 + 10 + 90 = 200

From this:
- Precision = TP / (TP + FP) = 90 / (90 + 15) = 0.857
- Recall = TP / (TP + FN) = 90 / (90 + 10) = 0.9
- Accuracy = (TP + TN) / Total = (90 + 85) / 200 = 0.875
    

Precision vs Recall tradeoff with concrete examples

Precision and recall often pull in opposite directions. Imagine a spam email filter:

• High precision: Few good emails are wrongly marked as spam. This means when the filter says "spam," it is usually right.
• High recall: Most spam emails are caught. This means very few spam emails get through.

If you want to avoid missing spam, focus on recall. If you want to avoid losing good emails, focus on precision.

TFMA helps you see these metrics clearly for different data slices, so you can decide the best balance for your use case.

What "good" vs "bad" metric values look like for this use case

Good metrics mean your model works well and fairly:

• Accuracy: Above 80% is often good, but depends on data balance.
• Precision and Recall: Both above 70% is good; if one is low, check why.
• AUC: Above 0.8 means strong ability to separate classes.
• Consistent metrics across slices: No big drops in any group means fair model.

Bad metrics show problems:

• Accuracy near random guess (e.g., 50% for balanced binary).
• Very low recall means many positives missed.
• Very low precision means many false alarms.
• Big metric differences across slices mean bias or unfairness.

Metrics pitfalls

• Accuracy paradox: High accuracy can be misleading if data is imbalanced. For example, 95% accuracy on 95% negative data means model ignores positives.
• Data leakage: If test data leaks info from training, metrics look too good but model fails in real life.
• Overfitting indicators: Very high training metrics but low test metrics mean model memorizes training data, not generalizing.
• Ignoring slices: Overall good metrics can hide poor performance on important groups. TFMA helps avoid this by slicing data.

Self-check question

Your model has 98% accuracy but 12% recall on fraud cases. Is it good for production? Why or why not?

Answer: No, it is not good. The model misses 88% of fraud cases (low recall), which is dangerous. High accuracy is misleading because fraud is rare. You want high recall to catch most fraud.

Key Result

TFMA helps evaluate key metrics like precision, recall, and AUC across data slices to ensure fair and reliable model performance.

Practice

1. What is the main purpose of TensorFlow Model Analysis (TFMA)?

easy

A. To preprocess data before training

B. To train machine learning models faster

C. To visualize neural network architectures

D. To evaluate machine learning models with detailed metrics and data slices

Solution

1. Step 1: Understand TFMA's role

   TFMA is designed to evaluate models, not train or preprocess them.

2. Step 2: Identify TFMA's key feature

   It provides detailed metrics and supports evaluation on data slices.

3. Final Answer:

   To evaluate machine learning models with detailed metrics and data slices -> Option D

4. Quick Check:

   TFMA = Model evaluation [OK]

Hint: TFMA is for evaluation, not training or preprocessing [OK]

Common Mistakes:

• Confusing TFMA with training tools
• Thinking TFMA is for data preprocessing
• Assuming TFMA visualizes model structure

2. Which of the following is the correct way to specify metrics in an EvalConfig for TFMA?

easy

A. eval_config = tfma.EvalConfig(metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='ExampleCount')])])

B. eval_config = tfma.EvalConfig(metrics='accuracy')

C. eval_config = tfma.EvalConfig(metrics=['loss', 'accuracy'])

D. eval_config = tfma.EvalConfig(metrics=tf.keras.metrics.Accuracy())

Solution

1. Step 1: Check TFMA metric config syntax

   TFMA expects metrics_specs as a list of MetricsSpec objects containing MetricConfig, not strings or Keras metrics.

2. Step 2: Validate each option

   eval_config = tfma.EvalConfig(metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='ExampleCount')])]) uses the correct structure; others use incorrect types or formats.

3. Final Answer:

   eval_config = tfma.EvalConfig(metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='ExampleCount')])]) -> Option A

4. Quick Check:

   TFMA metrics = metrics_specs with MetricsSpec and MetricConfig [OK]

Hint: Use metrics_specs=[MetricsSpec(metrics=[MetricConfig(...)])] for TFMA metrics [OK]

Common Mistakes:

• Passing metric names as strings directly
• Using Keras metric objects instead of MetricConfig
• Not wrapping metrics in a list

3. Given the following TFMA code snippet, what will be the output type of eval_result?

import tensorflow_model_analysis as tfma

eval_result = tfma.run_model_analysis(
    model_location='saved_model_dir',
    data_location='eval_data.tfrecord',
    eval_config=tfma.EvalConfig(
        metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])]
    )
)

medium

A. A dictionary containing evaluation metrics and slicing results

B. A TensorFlow model object

C. A list of raw prediction tensors

D. A string path to saved evaluation results

Solution

1. Step 1: Understand tfma.run_model_analysis output

   This function returns a dictionary with metrics and slice keys describing evaluation results.

2. Step 2: Eliminate other options

   It does not return a model object, raw tensors, or just a path string.

3. Final Answer:

   A dictionary containing evaluation metrics and slicing results -> Option A

4. Quick Check:

   tfma.run_model_analysis returns dict [OK]

Hint: tfma.run_model_analysis returns dict with metrics [OK]

Common Mistakes:

• Expecting a model object instead of results
• Thinking output is raw tensors
• Assuming output is a file path string

4. You run TFMA evaluation but get an error: AttributeError: 'EvalConfig' object has no attribute 'metrics'. What is the likely cause?

medium

A. You used a wrong metric class name

B. You forgot to import tensorflow_model_analysis

C. You passed metrics as a parameter directly to EvalConfig constructor instead of inside a 'metrics_specs' field

D. Your saved model path is incorrect

Solution

1. Step 1: Understand EvalConfig structure

   EvalConfig expects metrics inside the metrics_specs field, not directly as a constructor argument named 'metrics'.

2. Step 2: Identify error cause

   Passing 'metrics' directly causes AttributeError because EvalConfig has no such direct attribute.

3. Final Answer:

   You passed metrics as a parameter directly to EvalConfig constructor instead of inside a 'metrics_specs' field -> Option C

4. Quick Check:

   EvalConfig metrics must be inside metrics_specs [OK]

Hint: Put metrics inside metrics_specs, not directly in EvalConfig [OK]

Common Mistakes:

• Passing metrics directly to EvalConfig constructor
• Ignoring correct EvalConfig structure
• Misreading error as import or path issue

5. You want to evaluate your model's accuracy separately for male and female users using TFMA. Which EvalConfig snippet correctly sets up slicing by a feature named gender?

hard

A. eval_config = tfma.EvalConfig(
model_specs=[tfma.ModelSpec()],
slicing_specs=[tfma.SlicingSpec(columns=['gender'])],
metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])]
)

B. eval_config = tfma.EvalConfig(
model_specs=[tfma.ModelSpec()],
slicing_specs=[tfma.SlicingSpec(feature_keys=['gender'])],
metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])]
)

C. eval_config = tfma.EvalConfig(
model_specs=[tfma.ModelSpec()],
slicing_specs=[tfma.SlicingSpec(feature_keys=['gender', 'age'])],
metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])]
)

D. eval_config = tfma.EvalConfig(
model_specs=[tfma.ModelSpec()],
slicing_specs=[tfma.SlicingSpec(feature_keys=['gender'])],
metrics_specs=[tfma.MetricsSpec(metrics=['Accuracy'])]
)

Solution

1. Step 1: Identify correct slicing key

   TFMA uses 'feature_keys' in SlicingSpec to specify features for slicing.

2. Step 2: Check metrics specification

   Metrics must be wrapped in MetricConfig inside MetricsSpec; eval_config = tfma.EvalConfig( model_specs=[tfma.ModelSpec()], slicing_specs=[tfma.SlicingSpec(feature_keys=['gender'])], metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])] ) does this correctly.

3. Step 3: Validate other options

   eval_config = tfma.EvalConfig( model_specs=[tfma.ModelSpec()], slicing_specs=[tfma.SlicingSpec(columns=['gender'])], metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])] ) uses 'columns' which is incorrect; C slices by two features, not just gender; D passes metrics as strings, which is invalid.

4. Final Answer:

   EvalConfig with slicing_specs using feature_keys=['gender'] and metrics_specs with MetricConfig metrics -> Option B

5. Quick Check:

   Slicing by feature_keys and MetricConfig metrics [OK]

Hint: Use feature_keys for slicing and MetricConfig for metrics [OK]

Common Mistakes:

• Using 'columns' instead of 'feature_keys' for slicing
• Passing metrics as strings instead of MetricConfig
• Adding extra features in slicing unintentionally