Categorical cross-entropy loss in TensorFlow

Introduction

Categorical cross-entropy loss helps measure how well a model predicts categories by comparing predicted probabilities to the true category labels.

When training a model to classify images into multiple classes like cats, dogs, and birds.

When building a text classifier that assigns sentences to topics such as sports, politics, or technology.

When predicting the type of fruit from pictures where each fruit is a separate category.

When you have one correct category per example and want the model to learn to pick it.

When your labels are one-hot encoded vectors representing categories.

Syntax

TensorFlow

tf.keras.losses.CategoricalCrossentropy(from_logits=False, label_smoothing=0, axis=-1)

from_logits=False means the model outputs probabilities (values between 0 and 1). Set True if outputs are raw scores.

axis=-1 means the last dimension holds the category probabilities.

Examples

Basic use with default settings where y_pred contains probabilities.

TensorFlow

loss = tf.keras.losses.CategoricalCrossentropy()
loss_value = loss(y_true, y_pred)

Use when model outputs raw scores (logits) instead of probabilities.

TensorFlow

loss = tf.keras.losses.CategoricalCrossentropy(from_logits=True)
loss_value = loss(y_true, logits)

Adds label smoothing to make the model less confident and improve generalization.

TensorFlow

loss = tf.keras.losses.CategoricalCrossentropy(label_smoothing=0.1)
loss_value = loss(y_true, y_pred)

Sample Model

This example calculates the categorical cross-entropy loss for three samples with three classes each. The true labels are one-hot vectors, and predictions are probabilities. The loss shows how far predictions are from the true labels.

TensorFlow

import tensorflow as tf
import numpy as np

# True labels (one-hot encoded for 3 classes)
y_true = np.array([[0, 1, 0], [1, 0, 0], [0, 0, 1]], dtype=np.float32)

# Predicted probabilities from model
# Each row sums to 1 and shows predicted chance for each class
y_pred = np.array([[0.05, 0.9, 0.05], [0.8, 0.1, 0.1], [0.1, 0.2, 0.7]], dtype=np.float32)

# Create loss function
loss_fn = tf.keras.losses.CategoricalCrossentropy()

# Calculate loss
loss_value = loss_fn(y_true, y_pred).numpy()

print(f"Categorical cross-entropy loss: {loss_value:.4f}")

Important Notes

Make sure your true labels are one-hot encoded vectors matching the number of classes.

If your model outputs raw scores (logits), set from_logits=True to apply softmax internally.

Label smoothing can help prevent the model from becoming too confident and improve accuracy on new data.

Summary

Categorical cross-entropy loss measures the difference between true categories and predicted probabilities.

Use it when you have multiple classes and one correct class per example.

Set from_logits=True if your model outputs raw scores instead of probabilities.