Activation functions in ML Python - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to apply the ReLU activation function to the input tensor.

ML Python

import torch
input_tensor = torch.tensor([-1.0, 0.0, 2.0, -3.0])
output = torch.nn.functional.[1](input_tensor)

Drag options to blanks, or click blank then click option'

Atanh

Bsigmoid

Csoftmax

Drelu

Attempts:

3 left

💡 Hint

Common Mistakes

Using sigmoid instead of relu changes output range.

Using softmax applies normalization, not element-wise activation.

The ReLU function replaces negative values with zero, keeping positive values unchanged.

2fill in blank

medium

Complete the code to apply the sigmoid activation function to the input tensor.

ML Python

import torch
input_tensor = torch.tensor([-2.0, 0.0, 2.0])
output = torch.nn.functional.[1](input_tensor)

Drag options to blanks, or click blank then click option'

Asigmoid

Brelu

Csoftmax

Dtanh

Attempts:

3 left

💡 Hint

Common Mistakes

Using relu instead of sigmoid changes output range.

Using softmax applies normalization across vector, not element-wise.

The sigmoid function outputs values between 0 and 1, useful for probabilities.

3fill in blank

hard

Fix the error in the code by choosing the correct activation function to apply softmax over dimension 1.

ML Python

import torch
input_tensor = torch.tensor([[1.0, 2.0, 3.0], [1.0, 2.0, 3.0]])
output = torch.nn.functional.[1](input_tensor, dim=1)

Drag options to blanks, or click blank then click option'

Asigmoid

Bsoftmax

Crelu

Dtanh

Attempts:

3 left

💡 Hint

Common Mistakes

Using relu or sigmoid does not normalize across dimension.

Forgetting to specify dim parameter causes errors.

Softmax converts logits to probabilities across the specified dimension.

4fill in blank

hard

Fill both blanks to define a custom activation function that applies tanh and then scales the output by 2.

ML Python

import torch
input_tensor = torch.tensor([-1.0, 0.0, 1.0])
output = 2 * torch.nn.functional.[1](input_tensor) [2] 0

Drag options to blanks, or click blank then click option'

Atanh

B+

C-

D*

Attempts:

3 left

💡 Hint

Common Mistakes

Using relu instead of tanh changes output range.

Using multiplication instead of addition after scaling.

The tanh function outputs between -1 and 1. Adding 0 keeps the output unchanged after scaling.

5fill in blank

hard

Fill all three blanks to create a dictionary comprehension that maps each word to its sigmoid activation applied to its length.

ML Python

import torch
words = ['hi', 'hello', 'hey']
result = {word: torch.nn.functional.[1](torch.tensor(len(word))) for word in words if len(word) [2] 2 and len(word) [3] 5}

Drag options to blanks, or click blank then click option'

Asigmoid

B>

C<

D==

Attempts:

3 left

💡 Hint

Common Mistakes

Using relu instead of sigmoid changes output range.

Using equality instead of range conditions.

This comprehension applies sigmoid to the length of words longer than 2 and less than 5 characters.

Practice

1. What is the main purpose of an activation function in a neural network?

easy

A. To split data into training and testing sets

B. To initialize the weights of the network

C. To reduce the size of the dataset

D. To add non-linearity so the network can learn complex patterns

Solution

1. Step 1: Understand the role of activation functions

   Activation functions introduce non-linearity to the network, allowing it to learn complex patterns beyond simple straight lines.

2. Step 2: Compare with other options

   Weight initialization, data splitting, and dataset size reduction are unrelated to activation functions.

3. Final Answer:

   To add non-linearity so the network can learn complex patterns -> Option D

4. Quick Check:

   Activation function purpose = Add non-linearity [OK]

Hint: Activation functions make models learn complex patterns [OK]

Common Mistakes:

• Confusing activation functions with weight initialization
• Thinking activation functions reduce dataset size
• Believing activation functions split data

2. Which of the following is the correct Python syntax to apply the ReLU activation function on a variable x using NumPy?

easy

A. relu_x = np.maximum(0, x)

B. relu_x = np.min(0, x)

C. relu_x = np.exp(x)

D. relu_x = np.log(x)

Solution

1. Step 1: Recall ReLU function definition

   ReLU returns the maximum between 0 and the input value, so np.maximum(0, x) is correct.

2. Step 2: Check other options

   np.min(0, x) returns the minimum, which is incorrect. np.exp(x) and np.log(x) are exponential and logarithm functions, not ReLU.

3. Final Answer:

   relu_x = np.maximum(0, x) -> Option A

4. Quick Check:

   ReLU = max(0, x) [OK]

Hint: ReLU = max between zero and input [OK]

Common Mistakes:

• Using np.min instead of np.maximum
• Confusing ReLU with exponential or log functions
• Forgetting to import numpy as np

3. Given the following Python code using NumPy, what is the output of print(output)?

import numpy as np
x = np.array([-2, 0, 3])
output = 1 / (1 + np.exp(-x))
print(output)

medium

A. [0 0 3]

B. [0.11920292 0.5 0.95257413]

C. [-2 0 3]

D. [1 1 1]

Solution

1. Step 1: Identify the activation function used

   The code applies the sigmoid function: 1 / (1 + exp(-x)) on each element of x.

2. Step 2: Calculate sigmoid values for each element

   For x = -2: sigmoid ≈ 0.1192; for 0: sigmoid = 0.5; for 3: sigmoid ≈ 0.9526.

3. Final Answer:

   [0.11920292 0.5 0.95257413] -> Option B

4. Quick Check:

   Sigmoid([-2,0,3]) = [0.119,0.5,0.953] [OK]

Hint: Sigmoid outputs values between 0 and 1 [OK]

Common Mistakes:

• Confusing sigmoid with identity function
• Expecting original input as output
• Miscomputing exponential values

4. The following code attempts to implement the Tanh activation function but has an error. What is the error?

import numpy as np
def tanh(x):
    return (np.exp(x) - np.exp(-x)) / (np.exp(x) + np.exp(x))

print(tanh(1))

medium

A. The denominator should be (np.exp(x) + np.exp(-x)) instead of (np.exp(x) + np.exp(x))

B. The numerator should be (np.exp(x) + np.exp(-x))

C. The function should use np.tanh(x) directly

D. The function is missing a return statement

Solution

1. Step 1: Review the Tanh formula

   Tanh(x) = (exp(x) - exp(-x)) / (exp(x) + exp(-x)). The denominator in the code is incorrect.

2. Step 2: Identify the error in denominator

   The code uses (np.exp(x) + np.exp(x)) which doubles exp(x) instead of adding exp(-x).

3. Final Answer:

   The denominator should be (np.exp(x) + np.exp(-x)) instead of (np.exp(x) + np.exp(x)) -> Option A

4. Quick Check:

   Tanh denominator = exp(x)+exp(-x) [OK]

Hint: Check denominator carefully for plus and minus signs [OK]

Common Mistakes:

• Copying exp(x) twice in denominator
• Swapping numerator and denominator terms
• Forgetting to return the value

5. You want to design a neural network for a binary classification task. Which activation function should you use in the output layer to get a probability between 0 and 1, and why?

hard

A. Tanh, because it outputs values between -1 and 1

B. ReLU, because it outputs only positive values

C. Sigmoid, because it outputs values between 0 and 1 suitable for probabilities

D. Linear, because it outputs raw scores

Solution

1. Step 1: Understand output layer needs for binary classification

   The output should represent a probability, so it must be between 0 and 1.

2. Step 2: Match activation functions to output range

   Sigmoid outputs between 0 and 1, perfect for probabilities. ReLU outputs 0 or more but unbounded. Tanh outputs between -1 and 1, not suitable for probabilities. Linear outputs any real number.

3. Final Answer:

   Sigmoid, because it outputs values between 0 and 1 suitable for probabilities -> Option C

4. Quick Check:

   Binary output needs sigmoid for probability [OK]

Hint: Use sigmoid for binary output probabilities [OK]

Common Mistakes:

• Using ReLU or Tanh in output layer for probabilities
• Choosing linear activation for classification output
• Not considering output range needed