Activation functions in ML Python - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Why use activation functions in neural networks?

What is the main reason to use activation functions in neural networks?

A To convert outputs into probabilities only

B To reduce the size of the input data

C To speed up the training by skipping some layers

D To introduce non-linearity so the network can learn complex patterns

💡 Hint

Think about what would happen if the network was just a chain of linear operations.

Activation functions add non-linearity, allowing the network to learn complex patterns beyond simple linear relationships.

❓ Predict Output

intermediate

Output of ReLU activation function

What is the output of the ReLU function for the input array [-2, 0, 3]?

ML Python

import numpy as np
inputs = np.array([-2, 0, 3])
outputs = np.maximum(0, inputs)
print(outputs.tolist())

A [0, 0, 3]

B [-2, 0, 3]

C [0, 0, 0]

D [-2, 0, 0]

💡 Hint

ReLU outputs zero for negative inputs and the input itself if positive.

ReLU returns 0 for negative inputs and the input value for zero or positive inputs.

❓ Hyperparameter

advanced

Choosing activation function for hidden layers

Which activation function is generally preferred for hidden layers in deep neural networks to avoid vanishing gradients?

A ReLU

B Tanh

C Sigmoid

D Linear

💡 Hint

Consider which function keeps gradients strong during backpropagation.

ReLU helps avoid vanishing gradients by outputting zero or positive values, keeping gradients alive during training.

❓ Metrics

advanced

Effect of activation function on model accuracy

A model trained with sigmoid activation in hidden layers achieves 70% accuracy. After switching to ReLU, accuracy improves to 85%. What is the most likely reason?

A ReLU reduces the number of model parameters

B ReLU allows faster training and better gradient flow, improving accuracy

C Sigmoid activation always produces lower accuracy regardless of data

D Sigmoid causes the model to overfit the training data

💡 Hint

Think about how activation functions affect training dynamics and gradients.

ReLU improves gradient flow and speeds up training, often leading to better accuracy compared to sigmoid in hidden layers.

🔧 Debug

expert

Identifying error in activation function usage

What does this code output when applying the sigmoid function incorrectly?

import numpy as np
def sigmoid(x):
    return 1 / (1 + np.exp(x))
print(sigmoid(1))

A Outputs 0.0

B Outputs 1.0

C Outputs approximately 0.268

D Outputs 0.5

💡 Hint

Check the formula for sigmoid and the sign inside the exponent.

The sigmoid function is defined incorrectly: it should be 1 / (1 + np.exp(-x)). For x=1, the correct output is ~0.731, but this code outputs ~0.268.