Model evaluation (TFMA) in TensorFlow - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to import the TensorFlow Model Analysis library.

TensorFlow

import tensorflow_model_analysis as [1]

Drag options to blanks, or click blank then click option'

Atfae

Btfa

Ctfm

Dtfma

Attempts:

3 left

💡 Hint

Common Mistakes

Using incorrect aliases like 'tfa' which is for TensorFlow Addons.

Misspelling the alias causing import errors.

The standard alias for TensorFlow Model Analysis is tfma. This helps us use its functions easily.

2fill in blank

medium

Complete the code to create an evaluation config with accuracy metric.

TensorFlow

eval_config = tfma.EvalConfig(metrics_specs=[tfma.MetricsSpec(metrics=[tfma.Metric('[1]')])])

Drag options to blanks, or click blank then click option'

Aloss

Baccuracy

Cprecision

Drecall

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'loss' as a metric name here, which is not a metric but a loss function.

Confusing precision or recall with accuracy.

The accuracy metric measures how often the model predicts correctly. It's commonly used for classification tasks.

3fill in blank

hard

Fix the error in the code to run the evaluation on the model and data.

TensorFlow

eval_result = tfma.evaluate_model(model_location='[1]', data_location='gs://my_bucket/data')

Drag options to blanks, or click blank then click option'

A/models/my_model

Bmy_model

Cgs://my_bucket/models/my_model

Dmodel/my_model

Attempts:

3 left

💡 Hint

Common Mistakes

Using local or relative paths instead of cloud storage paths.

Omitting the 'gs://' prefix.

The model location must be a full path accessible by TFMA, usually a Google Cloud Storage path starting with gs://.

4fill in blank

hard

Fill both blanks to define slicing specs for evaluation by 'age' and 'gender'.

TensorFlow

eval_config = tfma.EvalConfig(slicing_specs=[tfma.SlicingSpec(feature_keys=[[1]]), tfma.SlicingSpec(feature_keys=[[2]])])

Drag options to blanks, or click blank then click option'

A'age'

B'gender'

C'income'

D'location'

Attempts:

3 left

💡 Hint

Common Mistakes

Using unrelated features like 'income' or 'location' when not specified.

Mixing up the order of features.

Slicing specs define how to split data for detailed evaluation. Here, we slice by 'age' and 'gender' features.

5fill in blank

hard

Fill all three blanks to extract accuracy from evaluation results for the first slice.

TensorFlow

accuracy = eval_result.slicing_metrics[0].metrics['[1]'].double_value
slice_key = eval_result.slicing_metrics[0].slice.[2]
feature_value = slice_key['[3]']

Drag options to blanks, or click blank then click option'

Aaccuracy

Bage

Cgender

Dloss

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'loss' instead of 'accuracy' for the metric.

Confusing feature keys in the slice dictionary.

We get the 'accuracy' metric value, then access the slice key for 'age', and finally get the 'gender' feature value from that slice.

Practice

1. What is the main purpose of TensorFlow Model Analysis (TFMA)?

easy

A. To preprocess data before training

B. To train machine learning models faster

C. To visualize neural network architectures

D. To evaluate machine learning models with detailed metrics and data slices

Solution

1. Step 1: Understand TFMA's role

   TFMA is designed to evaluate models, not train or preprocess them.

2. Step 2: Identify TFMA's key feature

   It provides detailed metrics and supports evaluation on data slices.

3. Final Answer:

   To evaluate machine learning models with detailed metrics and data slices -> Option D

4. Quick Check:

   TFMA = Model evaluation [OK]

Hint: TFMA is for evaluation, not training or preprocessing [OK]

Common Mistakes:

• Confusing TFMA with training tools
• Thinking TFMA is for data preprocessing
• Assuming TFMA visualizes model structure

2. Which of the following is the correct way to specify metrics in an EvalConfig for TFMA?

easy

A. eval_config = tfma.EvalConfig(metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='ExampleCount')])])

B. eval_config = tfma.EvalConfig(metrics='accuracy')

C. eval_config = tfma.EvalConfig(metrics=['loss', 'accuracy'])

D. eval_config = tfma.EvalConfig(metrics=tf.keras.metrics.Accuracy())

Solution

1. Step 1: Check TFMA metric config syntax

   TFMA expects metrics_specs as a list of MetricsSpec objects containing MetricConfig, not strings or Keras metrics.

2. Step 2: Validate each option

   eval_config = tfma.EvalConfig(metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='ExampleCount')])]) uses the correct structure; others use incorrect types or formats.

3. Final Answer:

   eval_config = tfma.EvalConfig(metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='ExampleCount')])]) -> Option A

4. Quick Check:

   TFMA metrics = metrics_specs with MetricsSpec and MetricConfig [OK]

Hint: Use metrics_specs=[MetricsSpec(metrics=[MetricConfig(...)])] for TFMA metrics [OK]

Common Mistakes:

• Passing metric names as strings directly
• Using Keras metric objects instead of MetricConfig
• Not wrapping metrics in a list

3. Given the following TFMA code snippet, what will be the output type of eval_result?

import tensorflow_model_analysis as tfma

eval_result = tfma.run_model_analysis(
    model_location='saved_model_dir',
    data_location='eval_data.tfrecord',
    eval_config=tfma.EvalConfig(
        metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])]
    )
)

medium

A. A dictionary containing evaluation metrics and slicing results

B. A TensorFlow model object

C. A list of raw prediction tensors

D. A string path to saved evaluation results

Solution

1. Step 1: Understand tfma.run_model_analysis output

   This function returns a dictionary with metrics and slice keys describing evaluation results.

2. Step 2: Eliminate other options

   It does not return a model object, raw tensors, or just a path string.

3. Final Answer:

   A dictionary containing evaluation metrics and slicing results -> Option A

4. Quick Check:

   tfma.run_model_analysis returns dict [OK]

Hint: tfma.run_model_analysis returns dict with metrics [OK]

Common Mistakes:

• Expecting a model object instead of results
• Thinking output is raw tensors
• Assuming output is a file path string

4. You run TFMA evaluation but get an error: AttributeError: 'EvalConfig' object has no attribute 'metrics'. What is the likely cause?

medium

A. You used a wrong metric class name

B. You forgot to import tensorflow_model_analysis

C. You passed metrics as a parameter directly to EvalConfig constructor instead of inside a 'metrics_specs' field

D. Your saved model path is incorrect

Solution

1. Step 1: Understand EvalConfig structure

   EvalConfig expects metrics inside the metrics_specs field, not directly as a constructor argument named 'metrics'.

2. Step 2: Identify error cause

   Passing 'metrics' directly causes AttributeError because EvalConfig has no such direct attribute.

3. Final Answer:

   You passed metrics as a parameter directly to EvalConfig constructor instead of inside a 'metrics_specs' field -> Option C

4. Quick Check:

   EvalConfig metrics must be inside metrics_specs [OK]

Hint: Put metrics inside metrics_specs, not directly in EvalConfig [OK]

Common Mistakes:

• Passing metrics directly to EvalConfig constructor
• Ignoring correct EvalConfig structure
• Misreading error as import or path issue

5. You want to evaluate your model's accuracy separately for male and female users using TFMA. Which EvalConfig snippet correctly sets up slicing by a feature named gender?

hard

A. eval_config = tfma.EvalConfig(
model_specs=[tfma.ModelSpec()],
slicing_specs=[tfma.SlicingSpec(columns=['gender'])],
metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])]
)

B. eval_config = tfma.EvalConfig(
model_specs=[tfma.ModelSpec()],
slicing_specs=[tfma.SlicingSpec(feature_keys=['gender'])],
metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])]
)

C. eval_config = tfma.EvalConfig(
model_specs=[tfma.ModelSpec()],
slicing_specs=[tfma.SlicingSpec(feature_keys=['gender', 'age'])],
metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])]
)

D. eval_config = tfma.EvalConfig(
model_specs=[tfma.ModelSpec()],
slicing_specs=[tfma.SlicingSpec(feature_keys=['gender'])],
metrics_specs=[tfma.MetricsSpec(metrics=['Accuracy'])]
)

Solution

1. Step 1: Identify correct slicing key

   TFMA uses 'feature_keys' in SlicingSpec to specify features for slicing.

2. Step 2: Check metrics specification

   Metrics must be wrapped in MetricConfig inside MetricsSpec; eval_config = tfma.EvalConfig( model_specs=[tfma.ModelSpec()], slicing_specs=[tfma.SlicingSpec(feature_keys=['gender'])], metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])] ) does this correctly.

3. Step 3: Validate other options

   eval_config = tfma.EvalConfig( model_specs=[tfma.ModelSpec()], slicing_specs=[tfma.SlicingSpec(columns=['gender'])], metrics_specs=[tfma.MetricsSpec(metrics=[tfma.MetricConfig(class_name='Accuracy')])] ) uses 'columns' which is incorrect; C slices by two features, not just gender; D passes metrics as strings, which is invalid.

4. Final Answer:

   EvalConfig with slicing_specs using feature_keys=['gender'] and metrics_specs with MetricConfig metrics -> Option B

5. Quick Check:

   Slicing by feature_keys and MetricConfig metrics [OK]

Hint: Use feature_keys for slicing and MetricConfig for metrics [OK]

Common Mistakes:

• Using 'columns' instead of 'feature_keys' for slicing
• Passing metrics as strings instead of MetricConfig
• Adding extra features in slicing unintentionally