Activation functions (ReLU, sigmoid, softmax) in TensorFlow - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of ReLU activation on a tensor

What is the output of the following TensorFlow code applying ReLU activation?

TensorFlow

import tensorflow as tf
x = tf.constant([-3.0, 0.0, 2.0, -1.0, 5.0])
output = tf.nn.relu(x)
print(output.numpy())

A [0. 0. 0. 0. 0.]

B [-3. 0. 2. -1. 5.]

C [3. 0. 2. 1. 5.]

D [0. 0. 2. 0. 5.]

💡 Hint

ReLU sets all negative values to zero and keeps positive values unchanged.

ReLU (Rectified Linear Unit) replaces negative values with zero and leaves positive values as they are.

❓ Model Choice

intermediate

Choosing activation for binary classification output layer

Which activation function is most appropriate for the output layer of a binary classification model?

A Sigmoid

B Tanh

C ReLU

D Softmax

💡 Hint

Binary classification outputs a probability between 0 and 1.

Sigmoid outputs values between 0 and 1, suitable for binary classification probabilities.

❓ Metrics

advanced

Effect of softmax on output probabilities

Given logits [2.0, 1.0, 0.1], what is the output of applying softmax activation?

TensorFlow

import tensorflow as tf
logits = tf.constant([2.0, 1.0, 0.1])
probabilities = tf.nn.softmax(logits)
print(probabilities.numpy())

A [0.7310586 0.2689414 0.0000000]

B [0.500000 0.300000 0.200000]

C [0.65900114 0.24243297 0.09856589]

D [0.3333333 0.3333333 0.3333333]

💡 Hint

Softmax converts logits into probabilities that sum to 1.

Softmax exponentiates each logit and normalizes by the sum of all exponentials, producing probabilities.

🔧 Debug

advanced

Identifying error in sigmoid activation usage

What error will this TensorFlow code raise?

TensorFlow

import tensorflow as tf
x = tf.constant('string')
output = tf.nn.sigmoid(x)
print(output.numpy())

A TypeError: Input must be a numeric tensor

B ValueError: Cannot convert string to float

C AttributeError: 'str' object has no attribute 'numpy'

D No error, outputs sigmoid of string

💡 Hint

Sigmoid expects numeric input, not strings.

TensorFlow tries to convert string tensor to float for sigmoid but fails, raising ValueError.

🧠 Conceptual

expert

Why softmax is preferred over sigmoid for multi-class classification

Why is softmax activation preferred over sigmoid activation for the output layer in multi-class classification with mutually exclusive classes?

A Softmax outputs probabilities that sum to 1, sigmoid outputs independent probabilities that do not sum to 1

B Sigmoid is computationally more expensive than softmax

C Softmax outputs independent probabilities for each class, sigmoid outputs dependent probabilities

D Softmax can only be used with binary classification

💡 Hint

Consider the sum of output probabilities for all classes.

Softmax normalizes outputs so all class probabilities sum to 1, reflecting mutually exclusive classes. Sigmoid treats each class independently.

Practice

1. Which activation function is best suited for hidden layers in a neural network to keep only positive signals?

easy

A. ReLU

B. Sigmoid

C. Softmax

D. Linear

Solution

1. Step 1: Understand the role of activation functions in hidden layers

   Hidden layers need non-linear functions that allow positive values to pass and block negative ones to help learning complex patterns.

2. Step 2: Identify which function keeps positive signals

   ReLU (Rectified Linear Unit) outputs zero for negative inputs and passes positive inputs unchanged, making it ideal for hidden layers.

3. Final Answer:

   ReLU -> Option A

4. Quick Check:

   Hidden layers use ReLU = C [OK]

Hint: ReLU blocks negatives, perfect for hidden layers [OK]

Common Mistakes:

• Confusing sigmoid as best for hidden layers
• Thinking softmax works for hidden layers
• Assuming linear activation adds non-linearity

2. Which of the following is the correct way to apply the sigmoid activation function in TensorFlow?

easy

A. tf.nn.relu(x)

B. tf.nn.sigmoid(x)

C. tf.sigmoid(x)

D. tf.activation.sigmoid(x)

Solution

1. Step 1: Recall TensorFlow activation function syntax

   TensorFlow provides activation functions under tf.nn module, so sigmoid is tf.nn.sigmoid.

2. Step 2: Check each option for correct syntax

   tf.nn.sigmoid(x) uses tf.nn.sigmoid(x), which is the correct function call. Others are invalid or do not exist.

3. Final Answer:

   tf.nn.sigmoid(x) -> Option B

4. Quick Check:

   Sigmoid in TensorFlow = tf.nn.sigmoid(x) [OK]

Hint: TensorFlow activations are in tf.nn module [OK]

Common Mistakes:

• Using tf.sigmoid instead of tf.nn.sigmoid
• Confusing ReLU with sigmoid function
• Trying to call activation from tf.activation

3. What will be the output of the following code snippet?

import tensorflow as tf
x = tf.constant([-1.0, 0.0, 1.0, 2.0])
output = tf.nn.relu(x)
print(output.numpy())

medium

A. [0.5 0.5 0.5 0.5]

B. [-1. 0. 1. 2.]

C. [1. 1. 1. 1.]

D. [0. 0. 1. 2.]

Solution

1. Step 1: Understand ReLU behavior on input tensor

   ReLU outputs zero for negative inputs and passes positive inputs unchanged.

2. Step 2: Apply ReLU to each element in x

   -1.0 becomes 0.0, 0.0 stays 0.0, 1.0 stays 1.0, 2.0 stays 2.0.

3. Final Answer:

   [0. 0. 1. 2.] -> Option D

4. Quick Check:

   ReLU([-1,0,1,2]) = [0,0,1,2] [OK]

Hint: ReLU clips negatives to zero, keeps positives [OK]

Common Mistakes:

• Expecting negative values to remain
• Confusing ReLU with sigmoid output
• Assuming output is all ones

4. Identify the error in the following TensorFlow code that applies softmax activation:

import tensorflow as tf
x = tf.constant([2.0, 1.0, 0.1])
output = tf.nn.softmax(x, axis=1)
print(output.numpy())

medium

A. The axis parameter should be 0 or -1 for this tensor

B. Softmax cannot be applied to 1D tensors

C. The axis parameter should be omitted

D. The axis parameter should be 0 instead of 1

Solution

1. Step 1: Check the shape of input tensor x

   x is a 1D tensor with shape (3,), so valid axis values are 0 or -1.

2. Step 2: Understand axis parameter in softmax

   Axis=1 is invalid for 1D tensor because axis 1 does not exist; axis must be 0 or -1.

3. Final Answer:

   The axis parameter should be 0 or -1 for this tensor -> Option A

4. Quick Check:

   Softmax axis for 1D tensor = 0 or -1 [OK]

Hint: Axis must exist in tensor shape for softmax [OK]

Common Mistakes:

• Using axis=1 on 1D tensor causes error
• Thinking softmax can't apply to 1D tensors
• Omitting axis but expecting default to work

5. You want to build a neural network for multi-class classification with 4 classes. Which activation function should you use in the output layer to get probabilities for each class?

hard

A. ReLU

B. Sigmoid

C. Softmax

D. Tanh

Solution

1. Step 1: Understand output layer needs for multi-class classification

   Output layer must produce probabilities that sum to 1 across all classes.

2. Step 2: Identify activation function that outputs class probabilities

   Softmax converts raw scores into probabilities summing to 1, perfect for multi-class outputs.

3. Final Answer:

   Softmax -> Option C

4. Quick Check:

   Multi-class output uses Softmax = B [OK]

Hint: Softmax outputs probabilities summing to 1 [OK]

Common Mistakes:

• Using sigmoid for multi-class instead of softmax
• Choosing ReLU which doesn't output probabilities
• Confusing tanh with probability output