Bird
Raised Fist0

Implement Logistic Regression - Binary Cross-Entropy, Sigmoid

CODINGbeginnerInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
⚖️
Logistic Regression with Binary Cross-Entropy vs Logistic Regression with Sigmoid

When implementing logistic regression, would you use binary cross-entropy or sigmoid for the loss function? Justify your choice.

📊
Comparison
DimensionLogistic Regression with Binary Cross-EntropyLogistic Regression with Sigmoid
AccuracyHighMedium
InterpretabilityMediumHigh
Computational EfficiencyMediumHigh
Numerical StabilityLowHigh
Ease of ImplementationMediumHigh
✅
Verdict
Default to binary cross-entropy for large datasets where accuracy is critical. Use sigmoid when interpretability and computational efficiency are more important, especially for smaller datasets.
💡
Key Insight
Choose binary cross-entropy for accuracy and sigmoid for interpretability and efficiency.
⚠️
Interview Trap
❌ Wrong: Always use binary cross-entropy because it's more accurate.
This ignores the importance of interpretability and computational constraints in some scenarios.
✅ Consider the dataset size, computational resources, and whether interpretability or accuracy is more critical for the task.
Logistic Regression with Binary Cross-Entropy
"Directly optimizes the classification objective by minimizing the log loss."
Weaknesses
  • Requires numerical stability handling for extreme probabilities.
  • Less interpretable in terms of probability outputs directly.
  • Can be computationally intensive for large datasets due to log calculations.
Ideal When
  • n > 10,000
  • When model accuracy is prioritized over interpretability
  • When you have sufficient computational resources to handle log calculations
Google uses binary cross-entropy in its ad click prediction models to optimize for accuracy.
Logistic Regression with Sigmoid
"Provides direct probability outputs for binary classification."
Weaknesses
  • Does not directly optimize the classification objective.
  • Can be less accurate than binary cross-entropy for imbalanced datasets.
  • Requires additional thresholding to convert probabilities to class labels.
Ideal When
  • n < 10,000
  • When interpretability of probability outputs is crucial
  • When computational resources are limited
Startups often use sigmoid in logistic regression for quick prototyping due to its simplicity and interpretability.
🐍
Logistic Regression with Binary Cross-Entropy - Example
▼ Show
Python
import numpy as np

# Binary Cross-Entropy Loss

def binary_cross_entropy(y_true, y_pred):
    y_pred = np.clip(y_pred, 1e-15, 1 - 1e-15)  # Avoid log(0)
    return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))

# Example usage
y_true = np.array([1, 0, 1])
y_pred = np.array([0.9, 0.1, 0.8])
loss = binary_cross_entropy(y_true, y_pred)
print('Loss:', loss)
🐍
Logistic Regression with Sigmoid - Example
▼ Show
Python
import numpy as np

# Sigmoid Function

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

# Example usage
z = np.array([0.5, -1.0, 2.0])
probabilities = sigmoid(z)
print('Probabilities:', probabilities)
✅
Use Logistic Regression with Binary Cross-Entropy When
  • •n > 10,000
  • •When model accuracy is the primary goal
  • •When you have sufficient computational resources
✅
Use Logistic Regression with Sigmoid When
  • •n < 10,000
  • •When interpretability of outputs is important
  • •When computational resources are limited
↳
Nuances
  • For small datasets, sigmoid can be sufficient and more interpretable.
  • Binary cross-entropy is preferred for large-scale applications where accuracy is critical.
  • In cases of imbalanced datasets, binary cross-entropy often provides better performance.

Practice

(1/5)
1. 
from sklearn.model_selection import KFold
import numpy as np

X = np.array([1, 2, 3, 4, 5])
kf = KFold(n_splits=2)

for train_index, test_index in kf.split(X):
    print("Train:", train_index, "Test:", test_index)
What will be the output of the above code?
easy
A. Train: [0 1 2 3] Test: [4] and Train: [4] Test: [0 1 2 3]
B. Train: [0 1 2] Test: [3 4] and Train: [3 4] Test: [0 1 2]
C. Train: [0 1] Test: [2 3 4] and Train: [2 3 4] Test: [0 1]
D. Train: [0 1 2 3] Test: [4] and Train: [0 1 2 4] Test: [3]

Solution

  1. Step 1: Understand KFold behavior

    KFold splits the data into n_splits parts, here 2, and iterates over them.

  2. Step 2: Eliminate wrong options

    Train: [0 1 2 3] Test: [4] and Train: [4] Test: [0 1 2 3] and D suggest incorrect split sizes. Train: [0 1] Test: [2 3 4] and Train: [2 3 4] Test: [0 1] suggests incorrect indexing.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    KFold splits into equal parts. [OK]
Hint: KFold splits data into equal parts. [OK]
Common Mistakes:
2. What is the time complexity of cropping an image of size n x n to a smaller size m x m using NumPy slicing?
easy
A. O(n^2)
B. O(1)
C. O(n*m)
D. O(m^2)

Solution

  1. Step 1: Understand NumPy slicing

    NumPy slicing creates a view, not a copy, so it operates in constant time.

  2. Step 2: Evaluate the complexity

    The operation does not depend on the size of the arrays but on the slicing indices.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    NumPy slicing is O(1) due to views. [OK]
Hint: NumPy slicing is O(1) due to views. [OK]
Common Mistakes:
3. What is the time complexity of computing the forward pass for a single layer in a neural network using NumPy?
easy
A. O(n^2)
B. O(n)
C. O(n log n)
D. O(1)

Solution

  1. Step 1: Identify operations

    Forward pass involves matrix-vector multiplication.

  2. Step 2: Determine complexity

    Matrix-vector multiplication is O(n) where n is the number of neurons.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Matrix-vector multiplication is O(n) [OK]
Hint: Matrix-vector multiplication is O(n) [OK]
Common Mistakes:
4. What is the time complexity of calculating the mean Average Precision (mAP) for a single image with N predicted bounding boxes and M ground truth boxes?
easy
A. O(N + M)
B. O(N^2)
C. O(N * M)
D. O(M^2)

Solution

  1. Step 1: Understand the Process

    For each predicted box, IoU is calculated with each ground truth box, leading to N * M comparisons.

  2. Step 2: Determine Complexity

    Each comparison is constant time, so the overall complexity is O(N * M).

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    IoU comparisons drive complexity [OK]
Hint: IoU comparisons drive complexity [OK]
Common Mistakes:
5. What is the time complexity of computing the scaled dot-product attention for a single query with n keys?
easy
A. O(n^2)
B. O(n log n)
C. O(n)
D. O(1)

Solution

  1. Step 1: Identify operations

    For a single query, the main operations are dot product and softmax, both O(n).

  2. Step 2: Eliminate incorrect complexities

    O(n^2) and O(n log n) are incorrect as they imply more complex operations than present.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Single query, n keys -> O(n) [OK]
Hint: Single query, n keys -> O(n) [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard