Data validation (TFDV) in TensorFlow - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to generate a statistics summary from a dataset using TFDV.

TensorFlow

import tensorflow_data_validation as tfdv

stats = tfdv.generate_statistics_from_dataframe([1])
print(stats)

Drag options to blanks, or click blank then click option'

Adata

Bdataframe

Cdataset

Ddf

Attempts:

3 left

💡 Hint

Common Mistakes

Passing a variable that is not a DataFrame.

Using a variable name that does not exist.

The function generate_statistics_from_dataframe requires a pandas DataFrame as input. Here, df is the variable holding the dataset.

2fill in blank

medium

Complete the code to infer a schema from the statistics using TFDV.

TensorFlow

schema = tfdv.infer_schema(statistics=[1])
print(schema)

Drag options to blanks, or click blank then click option'

Adf

Bstats

Cdata

Dschema

Attempts:

3 left

💡 Hint

Common Mistakes

Passing the schema variable instead of statistics.

Passing the raw data instead of statistics.

The infer_schema function takes the statistics object as input to create a schema.

3fill in blank

hard

Fix the error in the code to validate the dataset against the schema using TFDV.

TensorFlow

anomalies = tfdv.validate_statistics(statistics=[1], schema=schema)
print(anomalies)

Drag options to blanks, or click blank then click option'

Adata

Bschema

Cstats

Ddf

Attempts:

3 left

💡 Hint

Common Mistakes

Passing the schema instead of statistics as the first argument.

Passing raw data instead of statistics.

The validate_statistics function requires the statistics object as the first argument to check for anomalies against the schema.

4fill in blank

hard

Fill both blanks to generate statistics and infer schema from a CSV file using TFDV.

TensorFlow

import tensorflow_data_validation as tfdv

stats = tfdv.generate_statistics_from_csv(data_location=[1])
schema = tfdv.infer_schema(statistics=[2])
print(schema)

Drag options to blanks, or click blank then click option'

A'data.csv'

Bstats

C'dataset.csv'

Dschema

Attempts:

3 left

💡 Hint

Common Mistakes

Passing the schema variable instead of statistics to infer_schema.

Not using quotes around the CSV file path.

Use the CSV file path as a string for data_location. Then pass the statistics variable stats to infer the schema.

5fill in blank

hard

Fill all three blanks to generate statistics, infer schema, and validate statistics from a pandas DataFrame using TFDV.

TensorFlow

import tensorflow_data_validation as tfdv

stats = tfdv.generate_statistics_from_dataframe([1])
schema = tfdv.infer_schema(statistics=[2])
anomalies = tfdv.validate_statistics(statistics=[3], schema=schema)
print(anomalies)

Drag options to blanks, or click blank then click option'

Adf

Bstats

Dschema

Attempts:

3 left

💡 Hint

Common Mistakes

Passing the schema variable instead of statistics to validation.

Using inconsistent variable names.

First, pass the DataFrame df to generate statistics. Then use stats to infer the schema and validate statistics.

Practice

1. What is the main purpose of TensorFlow Data Validation (TFDV) in a machine learning workflow?

easy

A. To find and report errors or anomalies in the dataset before training

B. To train the machine learning model faster

C. To visualize the model's predictions

D. To optimize the model's hyperparameters automatically

Solution

1. Step 1: Understand TFDV's role

   TFDV is designed to analyze datasets and detect issues before training.

2. Step 2: Compare options

   Only To find and report errors or anomalies in the dataset before training describes finding errors or anomalies in data, which is TFDV's main use.

3. Final Answer:

   To find and report errors or anomalies in the dataset before training -> Option A

4. Quick Check:

   TFDV purpose = Data validation [OK]

Hint: TFDV checks data quality before training [OK]

Common Mistakes:

• Confusing TFDV with model training tools
• Thinking TFDV optimizes model parameters
• Assuming TFDV visualizes predictions

2. Which of the following is the correct way to import TensorFlow Data Validation in Python?

easy

A. import tensorflow.data.validation as tfdv

B. import tfdv.tensorflow as tf

C. import tensorflow_data_validation as tfdv

D. import tfdv

Solution

1. Step 1: Recall the official import statement

   The standard import for TensorFlow Data Validation is 'import tensorflow_data_validation as tfdv'.

2. Step 2: Check other options

   Options B, C, and D are incorrect because they use wrong module paths or incomplete imports.

3. Final Answer:

   import tensorflow_data_validation as tfdv -> Option C

4. Quick Check:

   Correct import = import tensorflow_data_validation as tfdv [OK]

Hint: TFDV import starts with 'tensorflow_data_validation' [OK]

Common Mistakes:

• Using incorrect module paths
• Trying to import as 'tfdv' without full name
• Confusing with TensorFlow core imports

3. Given the following code snippet, what will be the output type of stats?

import tensorflow_data_validation as tfdv
stats = tfdv.generate_statistics_from_dataframe(df)

medium

A. A TensorFlow Dataset object

B. A DatasetFeatureStatisticsList protocol buffer

C. A Pandas DataFrame with statistics

D. A JSON string with data summary

Solution

1. Step 1: Understand the function output

   The function generate_statistics_from_dataframe returns a DatasetFeatureStatisticsList proto message.

2. Step 2: Match output types

   A DatasetFeatureStatisticsList protocol buffer correctly identifies the output as a protocol buffer object representing statistics.

3. Final Answer:

   A DatasetFeatureStatisticsList protocol buffer -> Option B

4. Quick Check:

   TFDV stats output = proto message [OK]

Hint: TFDV stats are protocol buffers, not DataFrames [OK]

Common Mistakes:

• Assuming output is a DataFrame
• Thinking output is JSON string
• Confusing with TensorFlow Dataset object

4. You run this code but get an error:

import tensorflow_data_validation as tfdv
stats = tfdv.generate_statistics_from_csv('data.csv')

What is the likely cause of the error?

medium

A. The function name is incorrect; it should be generate_statistics_from_dataframe

B. TFDV does not support CSV files directly

C. You forgot to import pandas before using TFDV

D. The CSV file path is invalid or file does not exist

Solution

1. Step 1: Check function correctness

   The function generate_statistics_from_csv exists in TFDV and supports CSV input.

2. Step 2: Identify common error

   Most errors come from invalid file paths or missing files, causing file not found errors.

3. Final Answer:

   The CSV file path is invalid or file does not exist -> Option D

4. Quick Check:

   File path error = The CSV file path is invalid or file does not exist [OK]

Hint: Check file path and existence first [OK]

Common Mistakes:

• Assuming function name is wrong
• Thinking TFDV can't read CSV
• Forgetting to check file location

5. You want to detect if new data has unexpected values compared to your training data. Which TFDV step should you perform after generating statistics for both datasets?

hard

A. Use tfdv.validate_statistics() with the training data schema

B. Retrain the model with new data statistics

C. Manually compare CSV files line by line

D. Use tfdv.generate_statistics_from_dataframe() again on new data

Solution

1. Step 1: Understand anomaly detection process

   After generating statistics, TFDV uses validate_statistics() to compare new data stats against a schema.

2. Step 2: Evaluate options

   Use tfdv.validate_statistics() with the training data schema correctly describes using validation with the schema to find anomalies.

3. Step 3: Exclude incorrect options

   Retraining or manual comparison is not part of TFDV's validation step; generating stats again alone doesn't detect anomalies.

4. Final Answer:

   Use tfdv.validate_statistics() with the training data schema -> Option A

5. Quick Check:

   Validate new stats with schema = Use tfdv.validate_statistics() with the training data schema [OK]

Hint: Validate new stats against schema to find anomalies [OK]

Common Mistakes:

• Skipping validation step
• Thinking retraining detects anomalies
• Confusing stats generation with validation