Model evaluation (TFMA) in TensorFlow

Introduction

We use model evaluation to check how well our machine learning model works. TFMA helps us see detailed results easily.

After training a model to see how accurate it is on new data.

When you want to compare different models to pick the best one.

To check if the model works well for different groups of data.

Before deploying a model to make sure it meets quality needs.

Syntax

TensorFlow

import tensorflow_model_analysis as tfma

# Define evaluation config
eval_config = tfma.EvalConfig(
    model_specs=[tfma.ModelSpec(label_key='label')],
    metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='ExampleCount'), tfma.MetricConfig(class_name='Accuracy')])],
    slicing_specs=[tfma.SlicingSpec()]  # Evaluate on whole dataset
)

# Run evaluation
evaluation = tfma.run_model_analysis(
    eval_config=eval_config,
    eval_shared_model=tfma.default_eval_shared_model(
        eval_saved_model_path='path_to_saved_model'
    ),
    input_data='path_to_eval_data'
)

TFMA uses an EvalConfig to set what metrics and slices to compute.

You need a saved model and evaluation data to run TFMA.

Examples

Basic config to evaluate accuracy on the whole dataset.

TensorFlow

eval_config = tfma.EvalConfig(
    model_specs=[tfma.ModelSpec(label_key='target')],
    metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])],
    slicing_specs=[tfma.SlicingSpec()]  # Whole dataset
)

Config to evaluate AUC metric separately for each gender group.

TensorFlow

eval_config = tfma.EvalConfig(
    model_specs=[tfma.ModelSpec(label_key='label')],
    metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='AUC')])],
    slicing_specs=[tfma.SlicingSpec(feature_keys=['gender'])]  # Evaluate by gender
)

Sample Model

This program trains a simple model, saves it, creates evaluation data, and runs TFMA to get accuracy on the whole dataset.

TensorFlow

import tensorflow as tf
import tensorflow_model_analysis as tfma
import tempfile
import os

# Create simple dataset
features = {'feature': [0, 1, 0, 1]}
labels = [0, 1, 0, 1]

# Build a simple model
model = tf.keras.Sequential([
    tf.keras.layers.Dense(1, activation='sigmoid', input_shape=(1,))
])
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# Train model
model.fit(x=features['feature'], y=labels, epochs=5, verbose=0)

# Save model
model_dir = tempfile.mkdtemp()
model.save(model_dir, save_format='tf')

# Prepare eval data as TF Example records
from tensorflow.train import Example, Features, Feature, Int64List, FloatList

def create_tf_example(feature, label):
    return Example(features=Features(feature={
        'feature': Feature(float_list=FloatList(value=[feature])),
        'label': Feature(int64_list=Int64List(value=[label]))
    }))

examples = [create_tf_example(f, l) for f, l in zip(features['feature'], labels)]

# Write eval data to TFRecord
eval_data_path = os.path.join(model_dir, 'eval_data.tfrecord')
with tf.io.TFRecordWriter(eval_data_path) as writer:
    for ex in examples:
        writer.write(ex.SerializeToString())

# Define EvalConfig
eval_config = tfma.EvalConfig(
    model_specs=[tfma.ModelSpec(label_key='label')],
    metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])],
    slicing_specs=[tfma.SlicingSpec()]  # Whole dataset
)

# Run TFMA evaluation
eval_shared_model = tfma.default_eval_shared_model(
    eval_saved_model_path=model_dir
)

result = tfma.run_model_analysis(
    eval_config=eval_config,
    eval_shared_model=eval_shared_model,
    input_data=eval_data_path
)

# Print accuracy metric
for slice_result in result.slicing_metrics:
    slice_key = slice_result['slice']
    metrics = slice_result['metrics']
    accuracy = metrics['accuracy']
    print(f"Slice: {slice_key}, Accuracy: {accuracy}")

Important Notes

TFMA works best with saved TensorFlow models and TFRecord evaluation data.

You can slice data by features to see model performance on different groups.

TFMA supports many metrics like accuracy, AUC, precision, recall, etc.

Summary

TFMA helps check how well your model works using detailed metrics.

You set up an EvalConfig to choose metrics and data slices.

Run TFMA with your saved model and evaluation data to get results.

Practice

1. What is the main purpose of TensorFlow Model Analysis (TFMA)?

easy

A. To preprocess data before training

B. To train machine learning models faster

C. To visualize neural network architectures

D. To evaluate machine learning models with detailed metrics and data slices

Solution

1. Step 1: Understand TFMA's role

   TFMA is designed to evaluate models, not train or preprocess them.

2. Step 2: Identify TFMA's key feature

   It provides detailed metrics and supports evaluation on data slices.

3. Final Answer:

   To evaluate machine learning models with detailed metrics and data slices -> Option D

4. Quick Check:

   TFMA = Model evaluation [OK]

Hint: TFMA is for evaluation, not training or preprocessing [OK]

Common Mistakes:

• Confusing TFMA with training tools
• Thinking TFMA is for data preprocessing
• Assuming TFMA visualizes model structure

2. Which of the following is the correct way to specify metrics in an EvalConfig for TFMA?

easy

A. eval_config = tfma.EvalConfig(metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='ExampleCount')])])

B. eval_config = tfma.EvalConfig(metrics='accuracy')

C. eval_config = tfma.EvalConfig(metrics=['loss', 'accuracy'])

D. eval_config = tfma.EvalConfig(metrics=tf.keras.metrics.Accuracy())

Solution

1. Step 1: Check TFMA metric config syntax

   TFMA expects metrics_specs as a list of MetricsSpec objects containing MetricConfig, not strings or Keras metrics.

2. Step 2: Validate each option

   eval_config = tfma.EvalConfig(metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='ExampleCount')])]) uses the correct structure; others use incorrect types or formats.

3. Final Answer:

   eval_config = tfma.EvalConfig(metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='ExampleCount')])]) -> Option A

4. Quick Check:

   TFMA metrics = metrics_specs with MetricsSpec and MetricConfig [OK]

Hint: Use metrics_specs=[MetricsSpec(metrics=[MetricConfig(...)])] for TFMA metrics [OK]

Common Mistakes:

• Passing metric names as strings directly
• Using Keras metric objects instead of MetricConfig
• Not wrapping metrics in a list

3. Given the following TFMA code snippet, what will be the output type of eval_result?

import tensorflow_model_analysis as tfma

eval_result = tfma.run_model_analysis(
    model_location='saved_model_dir',
    data_location='eval_data.tfrecord',
    eval_config=tfma.EvalConfig(
        metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])]
    )
)

medium

A. A dictionary containing evaluation metrics and slicing results

B. A TensorFlow model object

C. A list of raw prediction tensors

D. A string path to saved evaluation results

Solution

1. Step 1: Understand tfma.run_model_analysis output

   This function returns a dictionary with metrics and slice keys describing evaluation results.

2. Step 2: Eliminate other options

   It does not return a model object, raw tensors, or just a path string.

3. Final Answer:

   A dictionary containing evaluation metrics and slicing results -> Option A

4. Quick Check:

   tfma.run_model_analysis returns dict [OK]

Hint: tfma.run_model_analysis returns dict with metrics [OK]

Common Mistakes:

• Expecting a model object instead of results
• Thinking output is raw tensors
• Assuming output is a file path string

4. You run TFMA evaluation but get an error: AttributeError: 'EvalConfig' object has no attribute 'metrics'. What is the likely cause?

medium

A. You used a wrong metric class name

B. You forgot to import tensorflow_model_analysis

C. You passed metrics as a parameter directly to EvalConfig constructor instead of inside a 'metrics_specs' field

D. Your saved model path is incorrect

Solution

1. Step 1: Understand EvalConfig structure

   EvalConfig expects metrics inside the metrics_specs field, not directly as a constructor argument named 'metrics'.

2. Step 2: Identify error cause

   Passing 'metrics' directly causes AttributeError because EvalConfig has no such direct attribute.

3. Final Answer:

   You passed metrics as a parameter directly to EvalConfig constructor instead of inside a 'metrics_specs' field -> Option C

4. Quick Check:

   EvalConfig metrics must be inside metrics_specs [OK]

Hint: Put metrics inside metrics_specs, not directly in EvalConfig [OK]

Common Mistakes:

• Passing metrics directly to EvalConfig constructor
• Ignoring correct EvalConfig structure
• Misreading error as import or path issue

5. You want to evaluate your model's accuracy separately for male and female users using TFMA. Which EvalConfig snippet correctly sets up slicing by a feature named gender?

hard

A. eval_config = tfma.EvalConfig(
model_specs=[tfma.ModelSpec()],
slicing_specs=[tfma.SlicingSpec(columns=['gender'])],
metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])]
)

B. eval_config = tfma.EvalConfig(
model_specs=[tfma.ModelSpec()],
slicing_specs=[tfma.SlicingSpec(feature_keys=['gender'])],
metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])]
)

C. eval_config = tfma.EvalConfig(
model_specs=[tfma.ModelSpec()],
slicing_specs=[tfma.SlicingSpec(feature_keys=['gender', 'age'])],
metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])]
)

D. eval_config = tfma.EvalConfig(
model_specs=[tfma.ModelSpec()],
slicing_specs=[tfma.SlicingSpec(feature_keys=['gender'])],
metrics_specs=[tfma.MetricsSpec(metrics=['Accuracy'])]
)

Solution

1. Step 1: Identify correct slicing key

   TFMA uses 'feature_keys' in SlicingSpec to specify features for slicing.

2. Step 2: Check metrics specification

   Metrics must be wrapped in MetricConfig inside MetricsSpec; eval_config = tfma.EvalConfig( model_specs=[tfma.ModelSpec()], slicing_specs=[tfma.SlicingSpec(feature_keys=['gender'])], metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])] ) does this correctly.

3. Step 3: Validate other options

   eval_config = tfma.EvalConfig( model_specs=[tfma.ModelSpec()], slicing_specs=[tfma.SlicingSpec(columns=['gender'])], metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])] ) uses 'columns' which is incorrect; C slices by two features, not just gender; D passes metrics as strings, which is invalid.

4. Final Answer:

   EvalConfig with slicing_specs using feature_keys=['gender'] and metrics_specs with MetricConfig metrics -> Option B

5. Quick Check:

   Slicing by feature_keys and MetricConfig metrics [OK]

Hint: Use feature_keys for slicing and MetricConfig for metrics [OK]

Common Mistakes:

• Using 'columns' instead of 'feature_keys' for slicing
• Passing metrics as strings instead of MetricConfig
• Adding extra features in slicing unintentionally