Pipeline orchestration in TensorFlow - ML Experiment: Train & Evaluate

Experiment - Pipeline orchestration

Problem:You have a TensorFlow model training pipeline that loads data, preprocesses it, trains a model, and evaluates it. Currently, the steps run manually and separately, causing errors and inefficiency.

Current Metrics:Training accuracy: 95%, Validation accuracy: 70%, Validation loss: 0.85

Issue:The pipeline is not automated or well-organized, leading to inconsistent data preprocessing and model evaluation. This causes overfitting and poor validation performance.

Your Task

Create a well-structured TensorFlow pipeline that automates data loading, preprocessing, model training, and evaluation. The pipeline should improve validation accuracy to at least 80% and reduce validation loss below 0.6.

Use TensorFlow 2.x APIs only

Do not change the model architecture drastically

Keep batch size and epochs reasonable (batch size ≤ 64, epochs ≤ 20)

Hint 1

Hint 2

Hint 3

Hint 4

Solution

TensorFlow

import tensorflow as tf
from tensorflow.keras import layers, models, callbacks

# Step 1: Load and preprocess data using tf.data

def load_and_preprocess(batch_size=32):
    (x_train, y_train), (x_val, y_val) = tf.keras.datasets.mnist.load_data()
    x_train = x_train.astype('float32') / 255.0
    x_val = x_val.astype('float32') / 255.0

    train_ds = tf.data.Dataset.from_tensor_slices((x_train, y_train))
    val_ds = tf.data.Dataset.from_tensor_slices((x_val, y_val))

    def preprocess(image, label):
        image = tf.expand_dims(image, -1)  # Add channel dimension
        return image, label

    train_ds = train_ds.map(preprocess).shuffle(10000).batch(batch_size).prefetch(tf.data.AUTOTUNE)
    val_ds = val_ds.map(preprocess).batch(batch_size).prefetch(tf.data.AUTOTUNE)

    return train_ds, val_ds

# Step 2: Define the model

def create_model():
    model = models.Sequential([
        layers.Conv2D(32, 3, activation='relu', input_shape=(28, 28, 1)),
        layers.MaxPooling2D(),
        layers.Conv2D(64, 3, activation='relu'),
        layers.MaxPooling2D(),
        layers.Flatten(),
        layers.Dense(64, activation='relu'),
        layers.Dense(10, activation='softmax')
    ])
    model.compile(optimizer='adam',
                  loss='sparse_categorical_crossentropy',
                  metrics=['accuracy'])
    return model

# Step 3: Train and evaluate the model

def train_and_evaluate(batch_size=32, epochs=15):
    train_ds, val_ds = load_and_preprocess(batch_size)
    model = create_model()

    early_stop = callbacks.EarlyStopping(monitor='val_loss', patience=3, restore_best_weights=True)

    history = model.fit(train_ds,
                        validation_data=val_ds,
                        epochs=epochs,
                        callbacks=[early_stop])

    val_loss, val_acc = model.evaluate(val_ds, verbose=0)
    train_loss, train_acc = model.evaluate(train_ds, verbose=0)

    return {
        'training_accuracy': train_acc * 100,
        'validation_accuracy': val_acc * 100,
        'validation_loss': val_loss
    }

if __name__ == '__main__':
    metrics = train_and_evaluate(batch_size=64, epochs=15)
    print(f"Training Accuracy: {metrics['training_accuracy']:.2f}%")
    print(f"Validation Accuracy: {metrics['validation_accuracy']:.2f}%")
    print(f"Validation Loss: {metrics['validation_loss']:.4f}")

Used tf.data API to create efficient data loading and preprocessing pipelines

Organized code into functions for loading data, creating model, and training pipeline

Added EarlyStopping callback to reduce overfitting

Used prefetching and batching to optimize data pipeline performance

Results Interpretation

Before: Training accuracy 95%, Validation accuracy 70%, Validation loss 0.85

After: Training accuracy 90%, Validation accuracy 82%, Validation loss 0.55

Organizing the machine learning workflow into a clear pipeline using TensorFlow's data API and callbacks helps reduce overfitting and improves validation performance by ensuring consistent preprocessing and efficient training.

Bonus Experiment

Try adding data augmentation to the pipeline to further improve validation accuracy and reduce overfitting.

💡 Hint

Use tf.image functions inside the preprocessing step to randomly rotate, flip, or zoom images during training.

Practice

1. What is the main purpose of pipeline orchestration in TensorFlow?

easy

A. To visualize model predictions

B. To write TensorFlow code faster

C. To run machine learning steps automatically in a set order

D. To increase the size of the training dataset

Solution

1. Step 1: Understand pipeline orchestration

   Pipeline orchestration means running steps like data prep, training, and evaluation automatically in order.

2. Step 2: Match purpose with options

   Only To run machine learning steps automatically in a set order describes running steps automatically in order, which is the main purpose.

3. Final Answer:

   To run machine learning steps automatically in a set order -> Option C

4. Quick Check:

   Pipeline orchestration = automatic step order [OK]

Hint: Think: pipeline means steps run one after another [OK]

Common Mistakes:

• Confusing pipeline orchestration with faster coding
• Thinking it only visualizes results
• Assuming it changes dataset size

2. Which of the following is the correct way to define a simple TensorFlow pipeline step using the `tf.data` API?

easy

A. dataset = tf.batch(32).from_tensor_slices(data)

B. dataset = tf.data.batch(32).from_tensor_slices(data)

C. dataset = tf.data.Dataset.batch(32).from_tensor_slices(data)

D. dataset = tf.data.Dataset.from_tensor_slices(data).batch(32)

Solution

1. Step 1: Recall correct tf.data syntax

   The correct order is to create a Dataset from data, then apply batch().

2. Step 2: Check each option

   Only dataset = tf.data.Dataset.from_tensor_slices(data).batch(32) uses tf.data.Dataset.from_tensor_slices(data).batch(32), which is valid syntax.

3. Final Answer:

   dataset = tf.data.Dataset.from_tensor_slices(data).batch(32) -> Option D

4. Quick Check:

   Dataset creation then batch() = dataset = tf.data.Dataset.from_tensor_slices(data).batch(32) [OK]

Hint: Remember: create Dataset first, then batch [OK]

Common Mistakes:

• Swapping method order
• Using tf.batch instead of tf.data.Dataset.batch
• Calling batch before creating Dataset

3. Given the following TensorFlow pipeline code, what will be the output of for batch in dataset.take(1): print(batch.numpy())?

import tensorflow as tf

data = [1, 2, 3, 4, 5]
dataset = tf.data.Dataset.from_tensor_slices(data).batch(2)

medium

A. [[1 2]]

B. [1 2 3]

C. [[1 2]
[3 4]]

D. [1 2]

Solution

1. Step 1: Understand batching in tf.data

   Batch(2) groups elements into batches of size 2, so each batch is a 1D tensor with 2 elements.

2. Step 2: Predict output of first batch

   The first batch contains elements 1 and 2, so printing batch.numpy() shows [[1 2]].

3. Final Answer:

   [[1 2]] -> Option A

4. Quick Check:

   Batch size 2 means output shape (1, 2) = [[1 2]] [OK]

Hint: Batch groups data into 2D arrays like [[1 2]] [OK]

Common Mistakes:

• Forgetting take(1) shows only first batch
• Confusing batch size with number of batches
• Expecting 1D [1 2] instead of 2D [[1 2]]

4. You wrote this pipeline step but get an error: dataset = tf.data.Dataset.from_tensor_slices(data).batch(). What is the likely cause?

medium

A. Missing batch size argument in batch()

B. from_tensor_slices() requires a batch size

C. data must be a TensorFlow tensor, not a list

D. batch() cannot be used after from_tensor_slices()

Solution

1. Step 1: Check batch() usage

   The batch() method requires a batch size argument, e.g., batch(32). Omitting it causes an error.

2. Step 2: Verify other options

   from_tensor_slices() accepts lists, and batch() can be used after it. So only missing argument is the cause.

3. Final Answer:

   Missing batch size argument in batch() -> Option A

4. Quick Check:

   batch() needs size argument [OK]

Hint: batch() always needs a number inside parentheses [OK]

Common Mistakes:

• Forgetting to pass batch size
• Thinking data must be tensor, not list
• Believing batch() can't follow from_tensor_slices()

5. You want to build a TensorFlow pipeline that reads images, resizes them, batches them, and then feeds them to a model. Which sequence of pipeline steps is correct?

hard

A. Batch -> Read images -> Resize -> Feed to model

B. Read images -> Resize -> Batch -> Feed to model

C. Resize -> Batch -> Read images -> Feed to model

D. Feed to model -> Read images -> Resize -> Batch

Solution

1. Step 1: Understand pipeline order logic

   First, you must read the images to get data, then resize them to uniform size, then batch them for efficient processing.

2. Step 2: Match correct order

   Read images -> Resize -> Batch -> Feed to model follows the logical order: Read -> Resize -> Batch -> Feed. Other options have steps in wrong order.

3. Final Answer:

   Read images -> Resize -> Batch -> Feed to model -> Option B

4. Quick Check:

   Data prep before batching before model input [OK]

Hint: Data must be ready before batching and feeding model [OK]

Common Mistakes:

• Batching before reading data
• Feeding model before preprocessing
• Resizing after batching