Activation functions (ReLU, sigmoid, softmax) in TensorFlow - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to apply the ReLU activation function to the tensor.

TensorFlow

import tensorflow as tf

x = tf.constant([-3.0, 0.0, 5.0])
relu_output = tf.nn.[1](x)
print(relu_output.numpy())

Drag options to blanks, or click blank then click option'

Arelu

Bsigmoid

Csoftmax

Dtanh

Attempts:

3 left

💡 Hint

Common Mistakes

Using sigmoid or softmax instead of relu for this task.

Trying to call activation as a method on the tensor.

The ReLU activation function is applied using tf.nn.relu. It outputs zero for negative inputs and the input itself if positive.

2fill in blank

medium

Complete the code to apply the sigmoid activation function to the tensor.

TensorFlow

import tensorflow as tf

x = tf.constant([-1.0, 0.0, 1.0])
sigmoid_output = tf.nn.[1](x)
print(sigmoid_output.numpy())

Drag options to blanks, or click blank then click option'

Asoftmax

Brelu

Csigmoid

Delu

Attempts:

3 left

💡 Hint

Common Mistakes

Using relu or softmax instead of sigmoid.

Confusing sigmoid with tanh.

The sigmoid activation function is applied using tf.nn.sigmoid. It outputs values between 0 and 1, useful for probabilities.

3fill in blank

hard

Fix the error in the code to correctly apply the softmax activation function along the last axis.

TensorFlow

import tensorflow as tf

logits = tf.constant([[1.0, 2.0, 3.0]])
softmax_output = tf.nn.softmax(logits, axis=[1])
print(softmax_output.numpy())

Drag options to blanks, or click blank then click option'

A0

B-1

C1

D2

Attempts:

3 left

💡 Hint

Common Mistakes

Using axis=0 or axis=1 which may not be the last axis.

Omitting the axis argument causing default behavior that may be incorrect.

The softmax function should be applied along the last axis of the logits tensor. Using axis=-1 ensures this regardless of tensor shape.

4fill in blank

hard

Fill both blanks to create a dictionary comprehension that maps each word to its sigmoid activation applied to its length.

TensorFlow

words = ['hi', 'hello', 'hey']
import tensorflow as tf
result = {word: tf.nn.[1](tf.constant(len(word), dtype=tf.float32)) for word in [2]
print(result)

Drag options to blanks, or click blank then click option'

Asigmoid

Brelu

Cwords

Drange(3)

Attempts:

3 left

💡 Hint

Common Mistakes

Using relu instead of sigmoid for activation.

Iterating over range instead of the list of words.

The sigmoid activation is applied to the length of each word. The dictionary comprehension iterates over the list words.

5fill in blank

hard

Fill all three blanks to create a dictionary comprehension that maps each word to its softmax probability over lengths greater than 2.

TensorFlow

words = ['hi', 'hello', 'hey', 'greetings']
import tensorflow as tf
lengths = [len(word) for word in words if len(word) [1] 2]
lengths_tensor = tf.constant(lengths, dtype=tf.float32)
softmax_vals = tf.nn.[2](lengths_tensor, axis=[3])
result = {word: softmax_vals[i].numpy() for i, word in enumerate(words) if len(word) > 2}
print(result)

Drag options to blanks, or click blank then click option'

A>

Bsigmoid

C-1

Dsoftmax

Attempts:

3 left

💡 Hint

Common Mistakes

Using sigmoid instead of softmax for activation.

Using wrong axis value for softmax.

Using '<' instead of '>' in the filter condition.

The dictionary comprehension maps words with length greater than 2 to their softmax probabilities. The softmax is applied along the last axis (-1).

Practice

1. Which activation function is best suited for hidden layers in a neural network to keep only positive signals?

easy

A. ReLU

B. Sigmoid

C. Softmax

D. Linear

Solution

1. Step 1: Understand the role of activation functions in hidden layers

   Hidden layers need non-linear functions that allow positive values to pass and block negative ones to help learning complex patterns.

2. Step 2: Identify which function keeps positive signals

   ReLU (Rectified Linear Unit) outputs zero for negative inputs and passes positive inputs unchanged, making it ideal for hidden layers.

3. Final Answer:

   ReLU -> Option A

4. Quick Check:

   Hidden layers use ReLU = C [OK]

Hint: ReLU blocks negatives, perfect for hidden layers [OK]

Common Mistakes:

• Confusing sigmoid as best for hidden layers
• Thinking softmax works for hidden layers
• Assuming linear activation adds non-linearity

2. Which of the following is the correct way to apply the sigmoid activation function in TensorFlow?

easy

A. tf.nn.relu(x)

B. tf.nn.sigmoid(x)

C. tf.sigmoid(x)

D. tf.activation.sigmoid(x)

Solution

1. Step 1: Recall TensorFlow activation function syntax

   TensorFlow provides activation functions under tf.nn module, so sigmoid is tf.nn.sigmoid.

2. Step 2: Check each option for correct syntax

   tf.nn.sigmoid(x) uses tf.nn.sigmoid(x), which is the correct function call. Others are invalid or do not exist.

3. Final Answer:

   tf.nn.sigmoid(x) -> Option B

4. Quick Check:

   Sigmoid in TensorFlow = tf.nn.sigmoid(x) [OK]

Hint: TensorFlow activations are in tf.nn module [OK]

Common Mistakes:

• Using tf.sigmoid instead of tf.nn.sigmoid
• Confusing ReLU with sigmoid function
• Trying to call activation from tf.activation

3. What will be the output of the following code snippet?

import tensorflow as tf
x = tf.constant([-1.0, 0.0, 1.0, 2.0])
output = tf.nn.relu(x)
print(output.numpy())

medium

A. [0.5 0.5 0.5 0.5]

B. [-1. 0. 1. 2.]

C. [1. 1. 1. 1.]

D. [0. 0. 1. 2.]

Solution

1. Step 1: Understand ReLU behavior on input tensor

   ReLU outputs zero for negative inputs and passes positive inputs unchanged.

2. Step 2: Apply ReLU to each element in x

   -1.0 becomes 0.0, 0.0 stays 0.0, 1.0 stays 1.0, 2.0 stays 2.0.

3. Final Answer:

   [0. 0. 1. 2.] -> Option D

4. Quick Check:

   ReLU([-1,0,1,2]) = [0,0,1,2] [OK]

Hint: ReLU clips negatives to zero, keeps positives [OK]

Common Mistakes:

• Expecting negative values to remain
• Confusing ReLU with sigmoid output
• Assuming output is all ones

4. Identify the error in the following TensorFlow code that applies softmax activation:

import tensorflow as tf
x = tf.constant([2.0, 1.0, 0.1])
output = tf.nn.softmax(x, axis=1)
print(output.numpy())

medium

A. The axis parameter should be 0 or -1 for this tensor

B. Softmax cannot be applied to 1D tensors

C. The axis parameter should be omitted

D. The axis parameter should be 0 instead of 1

Solution

1. Step 1: Check the shape of input tensor x

   x is a 1D tensor with shape (3,), so valid axis values are 0 or -1.

2. Step 2: Understand axis parameter in softmax

   Axis=1 is invalid for 1D tensor because axis 1 does not exist; axis must be 0 or -1.

3. Final Answer:

   The axis parameter should be 0 or -1 for this tensor -> Option A

4. Quick Check:

   Softmax axis for 1D tensor = 0 or -1 [OK]

Hint: Axis must exist in tensor shape for softmax [OK]

Common Mistakes:

• Using axis=1 on 1D tensor causes error
• Thinking softmax can't apply to 1D tensors
• Omitting axis but expecting default to work

5. You want to build a neural network for multi-class classification with 4 classes. Which activation function should you use in the output layer to get probabilities for each class?

hard

A. ReLU

B. Sigmoid

C. Softmax

D. Tanh

Solution

1. Step 1: Understand output layer needs for multi-class classification

   Output layer must produce probabilities that sum to 1 across all classes.

2. Step 2: Identify activation function that outputs class probabilities

   Softmax converts raw scores into probabilities summing to 1, perfect for multi-class outputs.

3. Final Answer:

   Softmax -> Option C

4. Quick Check:

   Multi-class output uses Softmax = B [OK]

Hint: Softmax outputs probabilities summing to 1 [OK]

Common Mistakes:

• Using sigmoid for multi-class instead of softmax
• Choosing ReLU which doesn't output probabilities
• Confusing tanh with probability output