Model evaluation (TFMA) in TensorFlow - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is TFMA in the context of machine learning?

TFMA stands for TensorFlow Model Analysis. It is a tool to evaluate and analyze machine learning models, especially TensorFlow models, on large datasets.

intermediate

How does TFMA help in understanding model performance?

TFMA provides detailed metrics and visualizations for model performance across different slices of data, helping to identify strengths and weaknesses in specific groups.

beginner

What is a 'slice' in TFMA?

A slice is a subset of the data defined by feature values, like all examples where 'gender' is 'female'. TFMA evaluates model metrics separately on each slice.

intermediate

Which metrics can TFMA compute for classification models?

TFMA can compute metrics like accuracy, precision, recall, AUC (Area Under Curve), and confusion matrices for classification models.

intermediate

How do you run TFMA to evaluate a TensorFlow model?

You export your model and evaluation data, then use TFMA's API to compute metrics and visualize results, often integrating with TensorBoard for easy viewing.

What does TFMA primarily help with?

A Deploying models to production

B Training new machine learning models

C Cleaning data before training

D Evaluating machine learning models on large datasets

TFMA is designed to evaluate and analyze models, not to train or deploy them.

In TFMA, what is a 'slice'?

A A subset of data defined by feature values

B A training batch

C A type of loss function

D A part of the model architecture

Slices let you evaluate model performance on specific groups within your data.

Which metric is NOT typically computed by TFMA for classification?

A Mean Squared Error

B Accuracy

C Recall

D AUC

Mean Squared Error is a regression metric, not classification.

How can TFMA results be visualized?

A Using command line only

B Using Excel spreadsheets

C Using TensorBoard

D Using image editing software

TFMA integrates with TensorBoard for interactive visualization.

What is required before running TFMA evaluation?

A Converting data to images

B Exporting the trained model and evaluation data

C Re-training the model from scratch

D Deploying the model to cloud

TFMA needs the saved model and data to compute evaluation metrics.

Explain how TFMA helps in analyzing model performance across different groups in data.

Think about how you can check if a model works better for some groups than others.

Describe the steps to evaluate a TensorFlow model using TFMA.

Consider what you need before and after running TFMA.

Practice

1. What is the main purpose of TensorFlow Model Analysis (TFMA)?

easy

A. To preprocess data before training

B. To train machine learning models faster

C. To visualize neural network architectures

D. To evaluate machine learning models with detailed metrics and data slices

Solution

1. Step 1: Understand TFMA's role

   TFMA is designed to evaluate models, not train or preprocess them.

2. Step 2: Identify TFMA's key feature

   It provides detailed metrics and supports evaluation on data slices.

3. Final Answer:

   To evaluate machine learning models with detailed metrics and data slices -> Option D

4. Quick Check:

   TFMA = Model evaluation [OK]

Hint: TFMA is for evaluation, not training or preprocessing [OK]

Common Mistakes:

• Confusing TFMA with training tools
• Thinking TFMA is for data preprocessing
• Assuming TFMA visualizes model structure

2. Which of the following is the correct way to specify metrics in an EvalConfig for TFMA?

easy

A. eval_config = tfma.EvalConfig(metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='ExampleCount')])])

B. eval_config = tfma.EvalConfig(metrics='accuracy')

C. eval_config = tfma.EvalConfig(metrics=['loss', 'accuracy'])

D. eval_config = tfma.EvalConfig(metrics=tf.keras.metrics.Accuracy())

Solution

1. Step 1: Check TFMA metric config syntax

   TFMA expects metrics_specs as a list of MetricsSpec objects containing MetricConfig, not strings or Keras metrics.

2. Step 2: Validate each option

   eval_config = tfma.EvalConfig(metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='ExampleCount')])]) uses the correct structure; others use incorrect types or formats.

3. Final Answer:

   eval_config = tfma.EvalConfig(metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='ExampleCount')])]) -> Option A

4. Quick Check:

   TFMA metrics = metrics_specs with MetricsSpec and MetricConfig [OK]

Hint: Use metrics_specs=[MetricsSpec(metrics=[MetricConfig(...)])] for TFMA metrics [OK]

Common Mistakes:

• Passing metric names as strings directly
• Using Keras metric objects instead of MetricConfig
• Not wrapping metrics in a list

3. Given the following TFMA code snippet, what will be the output type of eval_result?

import tensorflow_model_analysis as tfma

eval_result = tfma.run_model_analysis(
    model_location='saved_model_dir',
    data_location='eval_data.tfrecord',
    eval_config=tfma.EvalConfig(
        metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])]
    )
)

medium

A. A dictionary containing evaluation metrics and slicing results

B. A TensorFlow model object

C. A list of raw prediction tensors

D. A string path to saved evaluation results

Solution

1. Step 1: Understand tfma.run_model_analysis output

   This function returns a dictionary with metrics and slice keys describing evaluation results.

2. Step 2: Eliminate other options

   It does not return a model object, raw tensors, or just a path string.

3. Final Answer:

   A dictionary containing evaluation metrics and slicing results -> Option A

4. Quick Check:

   tfma.run_model_analysis returns dict [OK]

Hint: tfma.run_model_analysis returns dict with metrics [OK]

Common Mistakes:

• Expecting a model object instead of results
• Thinking output is raw tensors
• Assuming output is a file path string

4. You run TFMA evaluation but get an error: AttributeError: 'EvalConfig' object has no attribute 'metrics'. What is the likely cause?

medium

A. You used a wrong metric class name

B. You forgot to import tensorflow_model_analysis

C. You passed metrics as a parameter directly to EvalConfig constructor instead of inside a 'metrics_specs' field

D. Your saved model path is incorrect

Solution

1. Step 1: Understand EvalConfig structure

   EvalConfig expects metrics inside the metrics_specs field, not directly as a constructor argument named 'metrics'.

2. Step 2: Identify error cause

   Passing 'metrics' directly causes AttributeError because EvalConfig has no such direct attribute.

3. Final Answer:

   You passed metrics as a parameter directly to EvalConfig constructor instead of inside a 'metrics_specs' field -> Option C

4. Quick Check:

   EvalConfig metrics must be inside metrics_specs [OK]

Hint: Put metrics inside metrics_specs, not directly in EvalConfig [OK]

Common Mistakes:

• Passing metrics directly to EvalConfig constructor
• Ignoring correct EvalConfig structure
• Misreading error as import or path issue

5. You want to evaluate your model's accuracy separately for male and female users using TFMA. Which EvalConfig snippet correctly sets up slicing by a feature named gender?

hard

A. eval_config = tfma.EvalConfig(
model_specs=[tfma.ModelSpec()],
slicing_specs=[tfma.SlicingSpec(columns=['gender'])],
metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])]
)

B. eval_config = tfma.EvalConfig(
model_specs=[tfma.ModelSpec()],
slicing_specs=[tfma.SlicingSpec(feature_keys=['gender'])],
metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])]
)

C. eval_config = tfma.EvalConfig(
model_specs=[tfma.ModelSpec()],
slicing_specs=[tfma.SlicingSpec(feature_keys=['gender', 'age'])],
metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])]
)

D. eval_config = tfma.EvalConfig(
model_specs=[tfma.ModelSpec()],
slicing_specs=[tfma.SlicingSpec(feature_keys=['gender'])],
metrics_specs=[tfma.MetricsSpec(metrics=['Accuracy'])]
)

Solution

1. Step 1: Identify correct slicing key

   TFMA uses 'feature_keys' in SlicingSpec to specify features for slicing.

2. Step 2: Check metrics specification

   Metrics must be wrapped in MetricConfig inside MetricsSpec; eval_config = tfma.EvalConfig( model_specs=[tfma.ModelSpec()], slicing_specs=[tfma.SlicingSpec(feature_keys=['gender'])], metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])] ) does this correctly.

3. Step 3: Validate other options

   eval_config = tfma.EvalConfig( model_specs=[tfma.ModelSpec()], slicing_specs=[tfma.SlicingSpec(columns=['gender'])], metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])] ) uses 'columns' which is incorrect; C slices by two features, not just gender; D passes metrics as strings, which is invalid.

4. Final Answer:

   EvalConfig with slicing_specs using feature_keys=['gender'] and metrics_specs with MetricConfig metrics -> Option B

5. Quick Check:

   Slicing by feature_keys and MetricConfig metrics [OK]

Hint: Use feature_keys for slicing and MetricConfig for metrics [OK]

Common Mistakes:

• Using 'columns' instead of 'feature_keys' for slicing
• Passing metrics as strings instead of MetricConfig
• Adding extra features in slicing unintentionally