Broadcasting rules in TensorFlow

Introduction

Broadcasting lets TensorFlow do math on arrays of different shapes easily, like stretching smaller arrays to match bigger ones without copying data.

Adding a vector to each row of a matrix without writing loops.

Multiplying a tensor by a scalar value to scale all elements.

Applying a bias vector to each example in a batch during neural network training.

Combining images and filters of different shapes in convolution operations.

Performing element-wise operations on tensors with compatible but different shapes.

Syntax

TensorFlow

result = tensor1 + tensor2

TensorFlow automatically applies broadcasting when shapes differ but are compatible.

Broadcasting follows specific rules to align shapes from the right side.

Examples

Adds a vector of shape (4,) to each row of a matrix shape (3,4) by broadcasting the vector.

TensorFlow

import tensorflow as tf

# tensor1 shape: (3, 4)
tensor1 = tf.constant([[1, 2, 3, 4],
                       [5, 6, 7, 8],
                       [9, 10, 11, 12]])

# tensor2 shape: (4,)
tensor2 = tf.constant([1, 0, 1, 0])

result = tensor1 + tensor2
print(result)

Broadcasts tensor2 shape (3,) to (1,3) and then to (2,3,3) to add to tensor1 shape (2,3,1).

TensorFlow

import tensorflow as tf

# tensor1 shape: (2, 3, 1)
tensor1 = tf.constant([[[1], [2], [3]],
                       [[4], [5], [6]]])

# tensor2 shape: (3,)
tensor2 = tf.constant([10, 20, 30])

result = tensor1 + tensor2
print(result)

Sample Model

This program shows how TensorFlow adds a vector to each row of a matrix by broadcasting the vector shape (3,) to match the matrix shape (2,3). It prints the shapes, the result, and the mean value of the result tensor.

TensorFlow

import tensorflow as tf

# Define a 2D tensor (matrix) of shape (2, 3)
matrix = tf.constant([[1, 2, 3],
                      [4, 5, 6]])

# Define a 1D tensor (vector) of shape (3,)
vector = tf.constant([10, 20, 30])

# Add vector to each row of matrix using broadcasting
result = matrix + vector

# Print shapes and result
print(f"matrix shape: {matrix.shape}")
print(f"vector shape: {vector.shape}")
print("result:")
print(result)

# Calculate mean of result tensor
mean_value = tf.reduce_mean(result)
print(f"mean of result: {mean_value.numpy():.2f}")

Important Notes

Broadcasting compares shapes from right to left and matches dimensions if they are equal or one of them is 1.

If shapes are not compatible, TensorFlow will raise an error.

Broadcasting avoids copying data and makes code simpler and faster.

Summary

Broadcasting lets you do math on tensors with different but compatible shapes.

TensorFlow automatically stretches smaller tensors to match bigger ones following simple rules.

This helps write clean and efficient code without loops.