Model evaluation (TFMA) in TensorFlow - ML Experiment: Train & Evaluate

Experiment - Model evaluation (TFMA)

Problem:You have trained a TensorFlow model for binary classification. The training accuracy is 95%, but you want to evaluate the model fairly on a test dataset using TensorFlow Model Analysis (TFMA) to understand its performance across different slices of data.

Current Metrics:Training accuracy: 95%, Validation accuracy: 92%, No TFMA evaluation done yet.

Issue:Without TFMA, you cannot see detailed evaluation metrics or how the model performs on different data slices, which limits understanding of model fairness and robustness.

Your Task

Use TensorFlow Model Analysis (TFMA) to evaluate the trained model on the test dataset and compute overall and slice-based metrics such as accuracy and AUC.

Use TFMA for evaluation only; do not retrain the model.

Evaluate on the provided test dataset.

Compute at least accuracy and AUC metrics.

Create at least one slice based on a feature (e.g., 'gender').

Hint 1

Hint 2

Hint 3

Hint 4

Solution

TensorFlow

import tensorflow as tf
import tensorflow_model_analysis as tfma

# Assume the model is saved in 'saved_model_dir'
saved_model_dir = 'saved_model'

# Prepare test data as a list of dictionaries
# Example test data with features and labels
raw_test_data = [
    {'features': {'age': 25, 'gender': 'male'}, 'label': 1},
    {'features': {'age': 30, 'gender': 'female'}, 'label': 0},
    {'features': {'age': 22, 'gender': 'female'}, 'label': 1},
    {'features': {'age': 35, 'gender': 'male'}, 'label': 0}
]

# Convert raw data to tf.Example format
from tensorflow.train import Example, Features, Feature, BytesList, Int64List

def create_tf_example(data_point):
    feature = {}
    for key, value in data_point['features'].items():
        if isinstance(value, int):
            feature[key] = Feature(int64_list=Int64List(value=[value]))
        else:
            feature[key] = Feature(bytes_list=BytesList(value=[value.encode()]))
    feature['label'] = Feature(int64_list=Int64List(value=[data_point['label']]))
    example_proto = Example(features=Features(feature=feature))
    return example_proto.SerializeToString()

serialized_examples = [create_tf_example(dp) for dp in raw_test_data]

# Write serialized examples to a TFRecord file
import tempfile

test_tfrecord = tempfile.NamedTemporaryFile(delete=False, suffix='.tfrecord').name
with tf.io.TFRecordWriter(test_tfrecord) as writer:
    for ex in serialized_examples:
        writer.write(ex)

# Define feature specs for parsing
feature_spec = {
    'age': tf.io.FixedLenFeature([], tf.int64),
    'gender': tf.io.FixedLenFeature([], tf.string),
    'label': tf.io.FixedLenFeature([], tf.int64)
}

# Create EvalConfig with slicing on 'gender'
eval_config = tfma.EvalConfig(
    model_specs=[tfma.ModelSpec(label_key='label')],
    slicing_specs=[
        tfma.SlicingSpec(),  # Overall slice
        tfma.SlicingSpec(feature_keys=['gender'])  # Slice by gender
    ],
    metrics_specs=[
        tfma.MetricsSpec(metrics=[
            tfma.MetricConfig(class_name='Accuracy'),
            tfma.MetricConfig(class_name='AUC')
        ])
    ]
)

# Run TFMA evaluation
eval_result = tfma.run_model_analysis(
    eval_shared_model=tfma.default_eval_shared_model(eval_saved_model_path=saved_model_dir),
    data_location=test_tfrecord,
    file_format='tfrecord',
    eval_config=eval_config
)

# Print overall and slice metrics
for slice_key, metrics in eval_result.slicing_metrics.items():
    print(f"Slice: {slice_key}")
    for metric_name, metric_value in metrics.items():
        print(f"  {metric_name}: {metric_value}")

Added TFMA evaluation code to load the saved model.

Prepared test data as TFRecord format compatible with TFMA.

Defined slicing specs to evaluate metrics overall and by 'gender'.

Configured metrics to compute accuracy and AUC.

Printed evaluation metrics for each slice.

Results Interpretation

Before TFMA evaluation, only training and validation accuracy were known (95% and 92%). After TFMA evaluation, detailed metrics including accuracy and AUC were computed overall and for slices by gender.

Overall accuracy dropped to 75% on test data, showing realistic performance. The male slice had lower accuracy (50%) and AUC (60%), while the female slice had perfect scores (100%).

Using TFMA helps you understand model performance not just overall but also on specific groups of data. This reveals if the model works better or worse for certain slices, which is important for fairness and trust.

Bonus Experiment

Try adding a new slice based on 'age' groups (e.g., <30 and >=30) and evaluate metrics for these slices.

💡 Hint

Define a custom slicing function or use TFMA's slicing specs with feature transformations to create age groups.

Practice

1. What is the main purpose of TensorFlow Model Analysis (TFMA)?

easy

A. To preprocess data before training

B. To train machine learning models faster

C. To visualize neural network architectures

D. To evaluate machine learning models with detailed metrics and data slices

Solution

1. Step 1: Understand TFMA's role

   TFMA is designed to evaluate models, not train or preprocess them.

2. Step 2: Identify TFMA's key feature

   It provides detailed metrics and supports evaluation on data slices.

3. Final Answer:

   To evaluate machine learning models with detailed metrics and data slices -> Option D

4. Quick Check:

   TFMA = Model evaluation [OK]

Hint: TFMA is for evaluation, not training or preprocessing [OK]

Common Mistakes:

• Confusing TFMA with training tools
• Thinking TFMA is for data preprocessing
• Assuming TFMA visualizes model structure

2. Which of the following is the correct way to specify metrics in an EvalConfig for TFMA?

easy

A. eval_config = tfma.EvalConfig(metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='ExampleCount')])])

B. eval_config = tfma.EvalConfig(metrics='accuracy')

C. eval_config = tfma.EvalConfig(metrics=['loss', 'accuracy'])

D. eval_config = tfma.EvalConfig(metrics=tf.keras.metrics.Accuracy())

Solution

1. Step 1: Check TFMA metric config syntax

   TFMA expects metrics_specs as a list of MetricsSpec objects containing MetricConfig, not strings or Keras metrics.

2. Step 2: Validate each option

   eval_config = tfma.EvalConfig(metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='ExampleCount')])]) uses the correct structure; others use incorrect types or formats.

3. Final Answer:

   eval_config = tfma.EvalConfig(metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='ExampleCount')])]) -> Option A

4. Quick Check:

   TFMA metrics = metrics_specs with MetricsSpec and MetricConfig [OK]

Hint: Use metrics_specs=[MetricsSpec(metrics=[MetricConfig(...)])] for TFMA metrics [OK]

Common Mistakes:

• Passing metric names as strings directly
• Using Keras metric objects instead of MetricConfig
• Not wrapping metrics in a list

3. Given the following TFMA code snippet, what will be the output type of eval_result?

import tensorflow_model_analysis as tfma

eval_result = tfma.run_model_analysis(
    model_location='saved_model_dir',
    data_location='eval_data.tfrecord',
    eval_config=tfma.EvalConfig(
        metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])]
    )
)

medium

A. A dictionary containing evaluation metrics and slicing results

B. A TensorFlow model object

C. A list of raw prediction tensors

D. A string path to saved evaluation results

Solution

1. Step 1: Understand tfma.run_model_analysis output

   This function returns a dictionary with metrics and slice keys describing evaluation results.

2. Step 2: Eliminate other options

   It does not return a model object, raw tensors, or just a path string.

3. Final Answer:

   A dictionary containing evaluation metrics and slicing results -> Option A

4. Quick Check:

   tfma.run_model_analysis returns dict [OK]

Hint: tfma.run_model_analysis returns dict with metrics [OK]

Common Mistakes:

• Expecting a model object instead of results
• Thinking output is raw tensors
• Assuming output is a file path string

4. You run TFMA evaluation but get an error: AttributeError: 'EvalConfig' object has no attribute 'metrics'. What is the likely cause?

medium

A. You used a wrong metric class name

B. You forgot to import tensorflow_model_analysis

C. You passed metrics as a parameter directly to EvalConfig constructor instead of inside a 'metrics_specs' field

D. Your saved model path is incorrect

Solution

1. Step 1: Understand EvalConfig structure

   EvalConfig expects metrics inside the metrics_specs field, not directly as a constructor argument named 'metrics'.

2. Step 2: Identify error cause

   Passing 'metrics' directly causes AttributeError because EvalConfig has no such direct attribute.

3. Final Answer:

   You passed metrics as a parameter directly to EvalConfig constructor instead of inside a 'metrics_specs' field -> Option C

4. Quick Check:

   EvalConfig metrics must be inside metrics_specs [OK]

Hint: Put metrics inside metrics_specs, not directly in EvalConfig [OK]

Common Mistakes:

• Passing metrics directly to EvalConfig constructor
• Ignoring correct EvalConfig structure
• Misreading error as import or path issue

5. You want to evaluate your model's accuracy separately for male and female users using TFMA. Which EvalConfig snippet correctly sets up slicing by a feature named gender?

hard

A. eval_config = tfma.EvalConfig(
model_specs=[tfma.ModelSpec()],
slicing_specs=[tfma.SlicingSpec(columns=['gender'])],
metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])]
)

B. eval_config = tfma.EvalConfig(
model_specs=[tfma.ModelSpec()],
slicing_specs=[tfma.SlicingSpec(feature_keys=['gender'])],
metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])]
)

C. eval_config = tfma.EvalConfig(
model_specs=[tfma.ModelSpec()],
slicing_specs=[tfma.SlicingSpec(feature_keys=['gender', 'age'])],
metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])]
)

D. eval_config = tfma.EvalConfig(
model_specs=[tfma.ModelSpec()],
slicing_specs=[tfma.SlicingSpec(feature_keys=['gender'])],
metrics_specs=[tfma.MetricsSpec(metrics=['Accuracy'])]
)

Solution

1. Step 1: Identify correct slicing key

   TFMA uses 'feature_keys' in SlicingSpec to specify features for slicing.

2. Step 2: Check metrics specification

   Metrics must be wrapped in MetricConfig inside MetricsSpec; eval_config = tfma.EvalConfig( model_specs=[tfma.ModelSpec()], slicing_specs=[tfma.SlicingSpec(feature_keys=['gender'])], metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])] ) does this correctly.

3. Step 3: Validate other options

   eval_config = tfma.EvalConfig( model_specs=[tfma.ModelSpec()], slicing_specs=[tfma.SlicingSpec(columns=['gender'])], metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])] ) uses 'columns' which is incorrect; C slices by two features, not just gender; D passes metrics as strings, which is invalid.

4. Final Answer:

   EvalConfig with slicing_specs using feature_keys=['gender'] and metrics_specs with MetricConfig metrics -> Option B

5. Quick Check:

   Slicing by feature_keys and MetricConfig metrics [OK]

Hint: Use feature_keys for slicing and MetricConfig for metrics [OK]

Common Mistakes:

• Using 'columns' instead of 'feature_keys' for slicing
• Passing metrics as strings instead of MetricConfig
• Adding extra features in slicing unintentionally