Activation functions (ReLU, Sigmoid, Softmax) in PyTorch - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of ReLU activation on a tensor

What is the output of the following PyTorch code applying ReLU activation?

PyTorch

import torch
import torch.nn.functional as F
x = torch.tensor([-2.0, 0.0, 3.0, -1.5, 2.5])
output = F.relu(x)
print(output.tolist())

A [-2.0, 0.0, 0.0, -1.5, 0.0]

B [-2.0, 0.0, 3.0, -1.5, 2.5]

C [2.0, 0.0, 3.0, 1.5, 2.5]

D [0.0, 0.0, 3.0, 0.0, 2.5]

💡 Hint

ReLU replaces negative values with zero and keeps positive values unchanged.

ReLU (Rectified Linear Unit) sets all negative values to zero and keeps positive values as they are. So, -2.0, -1.5 become 0.0, while 0.0, 3.0, 2.5 remain the same.

❓ Predict Output

intermediate

Output of Sigmoid activation on a tensor

What is the output of the following PyTorch code applying Sigmoid activation?

PyTorch

import torch
import torch.nn.functional as F
x = torch.tensor([-1.0, 0.0, 1.0])
output = torch.sigmoid(x)
print([round(v.item(), 3) for v in output])

A [-0.731, 0.0, 0.731]

B [0.0, 0.5, 1.0]

C [0.269, 0.5, 0.731]

D [0.5, 0.5, 0.5]

💡 Hint

Sigmoid outputs values between 0 and 1, with 0.5 at input zero.

Sigmoid function formula is 1 / (1 + exp(-x)). For -1, 0, 1 inputs, outputs are approximately 0.269, 0.5, and 0.731 respectively.

❓ Model Choice

advanced

Choosing activation function for multi-class classification

Which activation function is most appropriate for the output layer of a neural network performing multi-class classification?

A Sigmoid

B Softmax

C ReLU

D Tanh

💡 Hint

The output should represent probabilities that sum to 1 across classes.

Softmax converts raw scores into probabilities that sum to 1, which is ideal for multi-class classification tasks.

❓ Metrics

advanced

Effect of activation function on model output range

Which activation function restricts the output values strictly between 0 and 1, making it suitable for binary classification output?

A Sigmoid

B ReLU

C Softmax

D Linear

💡 Hint

This function squashes input values into a probability-like range.

Sigmoid outputs values strictly between 0 and 1, which can be interpreted as probabilities for binary classification.

🔧 Debug

expert

Identifying error in Softmax usage

What error will the following PyTorch code raise when applying Softmax incorrectly?

PyTorch

import torch
import torch.nn.functional as F
x = torch.tensor([[1.0, 2.0, 3.0]])
output = F.softmax(x)
print(output)

A TypeError: softmax() missing 1 required positional argument: 'dim'

B RuntimeError: input tensor must be 1D or 2D

C ValueError: invalid shape for softmax input

D No error, outputs softmax probabilities

💡 Hint

Softmax requires the dimension along which to apply the function.

PyTorch's softmax function requires the 'dim' argument to specify the axis for normalization. Omitting it causes a TypeError.