Data transformation (TF Transform) in TensorFlow

Introduction

Data transformation helps prepare raw data into a clean and useful format for machine learning models. TF Transform makes this process easy and consistent.

When you want to scale or normalize numeric data before training a model.

When you need to convert text data into numbers, like turning words into IDs.

When you want to apply the same data changes during training and serving.

When you want to handle missing values or fill them with default numbers.

When you want to create new features from existing data, like combining columns.

Syntax

TensorFlow

import tensorflow_transform as tft

def preprocessing_fn(inputs):
    outputs = {}
    outputs['scaled_feature'] = tft.scale_to_z_score(inputs['feature'])
    outputs['bucketized_feature'] = tft.bucketize(inputs['feature'], num_buckets=5)
    return outputs

The preprocessing_fn defines how to change your data.

Use tft functions to transform features, like scaling or bucketizing.

Examples

This example scales the 'age' feature to a 0 to 1 range.

TensorFlow

def preprocessing_fn(inputs):
    outputs = {}
    outputs['normalized_age'] = tft.scale_to_0_1(inputs['age'])
    return outputs

This example converts text into numeric IDs using a vocabulary.

TensorFlow

def preprocessing_fn(inputs):
    outputs = {}
    outputs['word_ids'] = tft.compute_and_apply_vocabulary(inputs['text'])
    return outputs

This example fills missing income values with 0.0.

TensorFlow

def preprocessing_fn(inputs):
    outputs = {}
    outputs['filled_income'] = tf.where(tf.math.is_nan(inputs['income']), 0.0, inputs['income'])
    return outputs

Sample Model

This program uses TF Transform to scale and bucketize a numeric feature. It runs a small pipeline that analyzes the data, applies transformations, and prints the results.

TensorFlow

import tensorflow as tf
import tensorflow_transform as tft
import tensorflow_transform.beam.impl as tft_beam
from tensorflow_transform.tf_metadata import dataset_metadata, schema_utils
from tensorflow_metadata.proto.v0 import schema_pb2
import apache_beam as beam

# Define preprocessing function

def preprocessing_fn(inputs):
    outputs = {}
    outputs['scaled_feature'] = tft.scale_to_z_score(inputs['feature'])
    outputs['bucketized_feature'] = tft.bucketize(inputs['feature'], num_buckets=3)
    return outputs

# Sample raw data
raw_data = [
    {'feature': 10.0},
    {'feature': 20.0},
    {'feature': 30.0},
    {'feature': 40.0},
    {'feature': 50.0}
]

# Define schema
feature_spec = {'feature': tf.io.FixedLenFeature([], tf.float32)}
raw_metadata = dataset_metadata.DatasetMetadata(schema_utils.schema_from_feature_spec(feature_spec))

# Run TF Transform pipeline
with beam.Pipeline() as pipeline:
    with tft_beam.Context(temp_dir='/tmp'):
        transformed_dataset, transform_fn = (
            (raw_data, raw_metadata) |
            tft_beam.AnalyzeAndTransformDataset(preprocessing_fn)
        )
        transformed_data, transformed_metadata = transformed_dataset

        # Print transformed data
        _ = transformed_data | 'Print' >> beam.Map(print)

        result = pipeline.run()
        result.wait_until_finish()

Important Notes

TF Transform runs transformations on the whole dataset to learn statistics like mean and variance.

It ensures the same transformations apply during training and when the model is used later.

You need Apache Beam to run TF Transform pipelines.

Summary

TF Transform helps prepare data by applying consistent changes like scaling and bucketizing.

It learns from the whole dataset to apply accurate transformations.

Use it to make sure your model sees data in the same way during training and serving.

Practice

1. What is the main purpose of TF Transform in machine learning pipelines?

easy

A. To visualize the dataset statistics

B. To train the machine learning model faster

C. To apply consistent data transformations during both training and serving

D. To generate synthetic data samples

Solution

1. Step 1: Understand TF Transform's role

   TF Transform is designed to apply the same data transformations consistently during training and serving.

2. Step 2: Compare options

   Options A, B, and D describe other tasks unrelated to TF Transform's main purpose.

3. Final Answer:

   To apply consistent data transformations during both training and serving -> Option C

4. Quick Check:

   Consistent transformations = To apply consistent data transformations during both training and serving [OK]

Hint: TF Transform ensures same data changes in training and serving [OK]

Common Mistakes:

• Confusing TF Transform with model training
• Thinking TF Transform generates data
• Assuming it visualizes data statistics

2. Which of the following is the correct way to define a TF Transform preprocessing function that scales a numeric feature named age by its mean and variance?

easy

A. def preprocessing_fn(inputs):
age = tft.scale_to_z_score(inputs['age'])
return {'age_scaled': age}

B. def preprocessing_fn(inputs):
return {'age_scaled': inputs['age'] / 100}

C. def preprocessing_fn(inputs):
return {'age_scaled': inputs['age'] + 10}

D. def preprocessing_fn(inputs):
age = inputs['age'] * 2
return {'age_scaled': age}

Solution

1. Step 1: Identify correct TF Transform scaling function

   TF Transform provides tft.scale_to_z_score to scale features by mean and variance.

2. Step 2: Check code correctness

   def preprocessing_fn(inputs): age = tft.scale_to_z_score(inputs['age']) return {'age_scaled': age} uses tft.scale_to_z_score correctly; others use incorrect or arbitrary scaling.

3. Final Answer:

   def preprocessing_fn(inputs): age = tft.scale_to_z_score(inputs['age']) return {'age_scaled': age} -> Option A

4. Quick Check:

   Use tft.scale_to_z_score for mean-variance scaling [OK]

Hint: Use tft.scale_to_z_score for standard scaling [OK]

Common Mistakes:

• Using simple division instead of TF Transform scaling
• Adding arbitrary constants instead of scaling
• Multiplying without normalization

3. Given the following TF Transform preprocessing function, what will be the output for input {'income': 50000} if the dataset mean income is 40000 and stddev is 10000?

def preprocessing_fn(inputs):
    income_scaled = tft.scale_to_z_score(inputs['income'])
    return {'income_scaled': income_scaled}

medium

A. {'income_scaled': 0.0}

B. {'income_scaled': 0.5}

C. {'income_scaled': -1.0}

D. {'income_scaled': 1.0}

Solution

1. Step 1: Calculate z-score for income

   Z-score = (value - mean) / stddev = (50000 - 40000) / 10000 = 1.0

2. Step 2: Match output format

   The function returns {'income_scaled': z-score}, so output is {'income_scaled': 1.0}

3. Final Answer:

   {'income_scaled': 1.0} -> Option D

4. Quick Check:

   Z-score calculation = 1.0 [OK]

Hint: Z-score = (value - mean) / stddev for scaling [OK]

Common Mistakes:

• Forgetting to subtract mean
• Dividing by mean instead of stddev
• Mixing up positive and negative signs

4. You wrote this TF Transform preprocessing function but get an error during analysis:

def preprocessing_fn(inputs):
    age_bucketized = tft.bucketize(inputs['age'], num_buckets=5)
    return {'age_bucketized': age_bucketized}

What is the likely cause of the error?

medium

A. The input 'age' is not a numeric tensor

B. The function tft.bucketize requires 'boundaries' instead of 'num_buckets'

C. Missing import statement for tensorflow_transform as tft

D. The return value must be a list, not a dictionary

Solution

1. Step 1: Check tft.bucketize function signature

   tft.bucketize requires 'boundaries' argument, not 'num_buckets'.

2. Step 2: Identify error cause

   Using 'num_buckets' causes an error because it's not a valid parameter.

3. Final Answer:

   The function tft.bucketize requires 'boundaries' instead of 'num_buckets' -> Option B

4. Quick Check:

   tft.bucketize needs boundaries, not num_buckets [OK]

Hint: Use boundaries list for tft.bucketize, not num_buckets [OK]

Common Mistakes:

• Passing num_buckets instead of boundaries
• Forgetting to import tensorflow_transform
• Returning wrong data type

5. You want to apply TF Transform to prepare a dataset with numeric feature height and categorical feature color. You want to scale height and one-hot encode color. Which preprocessing function correctly applies these transformations?

hard

A. def preprocessing_fn(inputs):
height_scaled = tft.scale_to_z_score(inputs['height'])
color_vocab = tft.compute_and_apply_vocabulary(inputs['color'])
color_one_hot = tf.one_hot(color_vocab, depth=3)
return {'height_scaled': height_scaled, 'color_one_hot': color_one_hot}

B. def preprocessing_fn(inputs):
height_scaled = tft.scale_to_z_score(inputs['height'])
color_one_hot = tft.compute_and_apply_vocabulary(inputs['color'])
return {'height_scaled': height_scaled, 'color_one_hot': color_one_hot}

C. def preprocessing_fn(inputs):
height_scaled = inputs['height'] / 100
color_one_hot = tf.one_hot(inputs['color'], depth=3)
return {'height_scaled': height_scaled, 'color_one_hot': color_one_hot}

D. def preprocessing_fn(inputs):
height_scaled = tft.scale_to_z_score(inputs['height'])
color_one_hot = tf.one_hot(inputs['color'], depth=3)
return {'height_scaled': height_scaled, 'color_one_hot': color_one_hot}

Solution

1. Step 1: Scale numeric feature 'height'

   Use tft.scale_to_z_score to scale 'height' correctly.

2. Step 2: One-hot encode categorical 'color'

   First, map 'color' to integer indices using tft.compute_and_apply_vocabulary, then apply tf.one_hot with correct depth.

3. Step 3: Verify options

   def preprocessing_fn(inputs): height_scaled = tft.scale_to_z_score(inputs['height']) color_vocab = tft.compute_and_apply_vocabulary(inputs['color']) color_one_hot = tf.one_hot(color_vocab, depth=3) return {'height_scaled': height_scaled, 'color_one_hot': color_one_hot} correctly chains vocabulary computation and one-hot encoding; others miss steps or misuse functions.

4. Final Answer:

   def preprocessing_fn(inputs): height_scaled = tft.scale_to_z_score(inputs['height']) color_vocab = tft.compute_and_apply_vocabulary(inputs['color']) color_one_hot = tf.one_hot(color_vocab, depth=3) return {'height_scaled': height_scaled, 'color_one_hot': color_one_hot} -> Option A

5. Quick Check:

   Vocabulary then one-hot encoding = def preprocessing_fn(inputs): height_scaled = tft.scale_to_z_score(inputs['height']) color_vocab = tft.compute_and_apply_vocabulary(inputs['color']) color_one_hot = tf.one_hot(color_vocab, depth=3) return {'height_scaled': height_scaled, 'color_one_hot': color_one_hot} [OK]

Hint: Use vocabulary lookup before one-hot encoding categorical data [OK]

Common Mistakes:

• Applying one-hot directly on strings
• Skipping vocabulary computation
• Scaling numeric features incorrectly