Bird
Raised Fist0

Implement Logistic Regression - Binary Cross-Entropy, Sigmoid

CODINGbeginnerInterview freq: very_high
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
🤖
Implement Logistic Regression - Binary Cross-Entropy, Sigmoid
Implement logistic regression from scratch using NumPy.
codingbeginnerFreq: very_highClassical ML Coding
Prerequisites:Logistic Regression - Decision BoundaryLog-OddsLinear Regression implementation
📝
Problem Statement

Implement logistic regression using binary cross-entropy loss and sigmoid activation from scratch.

I/O Specification
Input:  X: np.ndarray of shape (n_samples, n_features), y: np.ndarray of shape (n_samples,)
Output: weights: np.ndarray of shape (n_features,), bias: float
Example
{
  "input": "X = np.array([[0.5, 1.5], [1.0, 2.0], [1.5, 0.5]]), y = np.array([0, 1, 0])",
  "expected_output": "weights = np.array([...]), bias = ..."
}
🎯
Interview Angles
  • Implementing logistic regression from scratch to test understanding of gradient descent
  • Understanding vectorization to optimize performance
  • Handling edge cases like empty datasets or single samples
  • Explaining the role of the sigmoid function in logistic regression
🔵
Naive Approach

Use gradient descent to iteratively update weights and bias. Compute gradients manually for each sample.

O(n_samples * n_features * epochs) - n_samples iterations × n_features operations per sample × epochs
Why Insufficient
O(n_samples * n_features * epochs) because it updates weights for each sample individually - inefficient for large datasets
🐍
Naive Code
Python
import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def logistic_regression_naive(X, y, lr=0.01, epochs=1000):
    n_samples, n_features = X.shape
    weights = np.zeros(n_features)
    bias = 0
    for _ in range(epochs):
        for i in range(n_samples):
            linear_model = np.dot(X[i], weights) + bias
            y_pred = sigmoid(linear_model)
            dw = (y_pred - y[i]) * X[i]
            db = y_pred - y[i]
            weights -= lr * dw
            bias -= lr * db
    return weights, bias

# Driver example
X = np.array([[0.5, 1.5], [1.0, 2.0], [1.5, 0.5]])
y = np.array([0, 1, 0])
weights, bias = logistic_regression_naive(X, y)
print(weights, bias)
💡
Key Insight
Vectorize the gradient computation to update weights and bias for all samples in one step per epoch.
🐍
Optimized Code
Python
import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def logistic_regression_optimized(X, y, lr=0.01, epochs=1000):
    n_samples, n_features = X.shape
    weights = np.zeros(n_features)
    bias = 0
    for _ in range(epochs):
        linear_model = np.dot(X, weights) + bias
        y_pred = sigmoid(linear_model)
        dw = np.dot(X.T, (y_pred - y)) / n_samples
        db = np.sum(y_pred - y) / n_samples
        weights -= lr * dw
        bias -= lr * db
    return weights, bias

# Driver example
X = np.array([[0.5, 1.5], [1.0, 2.0], [1.5, 0.5]])
y = np.array([0, 1, 0])
weights, bias = logistic_regression_optimized(X, y)
print(weights, bias)
🔍
Line-by-Line
linear_model = np.dot(X, weights) + bias
Compute the linear combination of inputs and weights for all samples - vectorized operation
y_pred = sigmoid(linear_model)
Apply the sigmoid function to get probabilities for all samples - vectorized
dw = np.dot(X.T, (y_pred - y)) / n_samples
Compute the gradient of weights - vectorized over all samples
db = np.sum(y_pred - y) / n_samples
Compute the gradient of bias - sum over all samples
weights -= lr * dw
Update weights using the computed gradient - single step for all features
bias -= lr * db
Update bias using the computed gradient - single step
⚡
Complexity
O(n_samples * n_features * epochs) - n_samples × n_features operations per epoch, but vectorized
Output
weights = np.array([...]), bias = ...
🔲
Edge Cases
Empty dataset
Returns initial weights and bias without error
n_samples, n_features = X.shape
Single sample
Updates weights and bias based on the single sample
linear_model = np.dot(X, weights) + bias
⚠️
Common Mistakes
Forgetting to initialize weights and bias
Symptom: Code crashes due to undefined variables
Incorrectly computing the gradient
Symptom: Weights and bias do not converge
Not vectorizing operations
Symptom: Code runs very slowly on larger datasets
Using incorrect learning rate
Symptom: Model fails to converge or converges too slowly

Practice

(1/5)
1. What is the time complexity of the K-Means clustering algorithm implemented from scratch using Lloyd's algorithm?
easy
A. O(n * k * d * i)
B. O(k^2)
C. O(n * log n)
D. O(n^2)

Solution

  1. Step 1: Analyze the algorithm

    The time complexity is determined by the number of data points (n), clusters (k), dimensions (d), and iterations (i).

  2. Step 2: Eliminate wrong options

    O(n^2) and C are incorrect as they don't account for k, d, and i. O(k^2) is incorrect because it doesn't consider n, d, and i.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Consider n, k, d, and i for complexity. [OK]
Hint: Consider n, k, d, and i for complexity. [OK]
Common Mistakes:
2. What is the time complexity of calculating DCG for a list of n relevance scores?
easy
A. O(n)
B. O(n log n)
C. O(n^2)
D. O(log n)

Solution

  1. Step 1: Analyze the DCG Calculation

    DCG involves iterating over the list once and performing constant-time operations for each element.

  2. Step 2: Determine Complexity

    Since each element is processed once, the complexity is O(n).

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    DCG iterates once over n elements. [OK]
Hint: DCG iterates once over n elements. [OK]
Common Mistakes:
3. 
import numpy as np
from sklearn.model_selection import train_test_split

X = np.array([1, 2, 3, 4, 5, 6])
y = np.array([0, 0, 1, 1, 0, 1])

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, stratify=y)
print(y_train)
easy
A. [0, 0, 1]
B. [0, 1, 0]
C. [1, 0, 1]
D. [0, 1, 1]

Solution

  1. Step 1: Understand stratification

    Stratification ensures that the class distribution in the train and test sets is similar to the original dataset.

  2. Step 2: Calculate class distribution

    Original y has 3 zeros and 3 ones. With stratification, y_train should have 1 zero and 2 ones.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Stratify maintains class distribution. [OK]
Hint: Stratify maintains class distribution. [OK]
Common Mistakes:
4. In a text classification task using Multinomial Naive Bayes, which preprocessing step is crucial for improving model performance?
medium
A. Normalizing feature values
B. Scaling features to [0, 1]
C. Removing stop words
D. Applying PCA for dimensionality reduction

Solution

  1. Step 1: Understand text preprocessing needs

    In text classification, irrelevant words can add noise.

  2. Step 2: Evaluate preprocessing steps

    Removing stop words (Removing stop words) reduces noise and improves performance. Options A and C are less relevant as MultinomialNB doesn't require normalized or scaled features. Applying PCA for dimensionality reduction is unnecessary as PCA isn't typically used for text data.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Remove stop words to reduce noise in text data. [OK]
Hint: Remove stop words to reduce noise in text data. [OK]
Common Mistakes:
5. How can you optimize the memory usage of a Min-Max scaling implementation for a large dataset?
hard
A. Process data in chunks instead of loading all at once
B. Use float64 data type for higher precision
C. Use a single loop to calculate min and max
D. Store scaled data in a new array to avoid overwriting

Solution

  1. Step 1: Identify Memory Bottlenecks

    Loading large datasets entirely into memory can be inefficient.

  2. Step 2: Optimize Memory Usage

    Processing data in chunks reduces memory footprint, making it feasible to handle large datasets.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Chunk processing saves memory. [OK]
Hint: Chunk processing saves memory. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard