Bird
Raised Fist0

Implement Naive Bayes - Gaussian and Multinomial

CODINGintermediateInterview freq: medium
Choose your preparation mode6 modes available
LearnTraceDebugTrade-offChallenge
⚖️
Gaussian Naive Bayes vs Multinomial Naive Bayes

In a text classification task with a large vocabulary and sparse feature vectors, would you use Gaussian Naive Bayes or Multinomial Naive Bayes? Justify your choice.

📊
Comparison
DimensionGaussian Naive BayesMultinomial Naive Bayes
Accuracy on continuous dataHighLow
Accuracy on text dataLowHigh
Training speedFastFast
Inference latencyFastFast
InterpretabilityHighMedium
✅
Verdict
Default to Multinomial Naive Bayes for text classification tasks with discrete word counts; switch to Gaussian Naive Bayes when dealing with continuous features or when feature distribution is approximately normal.
💡
Key Insight
Choose the Naive Bayes variant that aligns with the nature of your feature distribution: Gaussian for continuous, Multinomial for discrete.
⚠️
Interview Trap
❌ Wrong: Always use Multinomial Naive Bayes for any text data.
Ignoring the nature of the text data; if the text features are not frequency counts, Multinomial Naive Bayes may not be suitable.
✅ Consider the nature of your features: use Multinomial Naive Bayes for discrete frequency counts and Gaussian Naive Bayes for continuous features.
Gaussian Naive Bayes
"Handles continuous data effectively by assuming a Gaussian distribution."
Weaknesses
  • Assumes features are normally distributed, which may not hold for all datasets.
  • Less effective with sparse data, common in text classification.
  • Sensitive to outliers due to the Gaussian assumption.
Ideal When
  • n > 10K and features are continuous
  • When feature distribution is approximately normal
  • When interpretability of continuous feature impact is needed
Gaussian Naive Bayes is used in medical diagnosis systems where features like age, blood pressure, and cholesterol levels are continuous and approximately normally distributed.
Multinomial Naive Bayes
"Excels with discrete data, especially in text classification with word counts."
Weaknesses
  • Assumes feature independence, which may not hold in all cases.
  • Not suitable for continuous data without discretization.
  • Performance degrades with highly imbalanced classes.
Ideal When
  • n > 10K and features are discrete
  • When feature vectors are sparse and high-dimensional
  • When text data with word frequency is involved
Multinomial Naive Bayes is widely used in spam detection systems where the input data is text and features are word counts.
🐍
Gaussian Naive Bayes - Example
▼ Show
Python
import numpy as np

class GaussianNaiveBayes:
    def fit(self, X, y):
        self.classes = np.unique(y)
        self.mean = np.array([X[y==c].mean(axis=0) for c in self.classes])
        self.var = np.array([X[y==c].var(axis=0) for c in self.classes])
        self.priors = np.array([np.mean(y==c) for c in self.classes])

    def predict(self, X):
        posteriors = []
        for x in X:
            posteriors.append([np.sum(-0.5 * np.log(2 * np.pi * self.var[i]) - ((x - self.mean[i]) ** 2) / (2 * self.var[i])) + np.log(self.priors[i]) for i in range(len(self.classes))])
        return self.classes[np.argmax(posteriors, axis=1)]
🐍
Multinomial Naive Bayes - Example
▼ Show
Python
import numpy as np

class MultinomialNaiveBayes:
    def fit(self, X, y):
        self.classes = np.unique(y)
        self.feature_count = np.array([X[y==c].sum(axis=0) for c in self.classes])
        self.class_count = np.array([X[y==c].shape[0] for c in self.classes])
        self.feature_log_prob = np.log((self.feature_count + 1) / (self.feature_count.sum(axis=1, keepdims=True) + X.shape[1]))
        self.class_log_prior = np.log(self.class_count / X.shape[0])

    def predict(self, X):
        log_prob = X @ self.feature_log_prob.T + self.class_log_prior
        return self.classes[np.argmax(log_prob, axis=1)]
✅
Use Gaussian Naive Bayes When
  • •n > 10K and features are continuous
  • •When feature distribution is approximately normal
  • •When interpretability of continuous feature impact is needed
✅
Use Multinomial Naive Bayes When
  • •n > 10K and features are discrete
  • •When feature vectors are sparse and high-dimensional
  • •When text data with word frequency is involved
↳
Nuances
  • Gaussian Naive Bayes can perform poorly on text data due to its assumption of normal distribution.
  • Multinomial Naive Bayes may struggle with continuous data unless discretized.
  • In high-dimensional sparse data, Multinomial Naive Bayes often outperforms Gaussian Naive Bayes.
  • Gaussian Naive Bayes is more interpretable when dealing with continuous features.

Practice

(1/5)
1. What is the time complexity of calculating DCG for a list of n relevance scores?
easy
A. O(n)
B. O(n log n)
C. O(n^2)
D. O(log n)

Solution

  1. Step 1: Analyze the DCG Calculation

    DCG involves iterating over the list once and performing constant-time operations for each element.

  2. Step 2: Determine Complexity

    Since each element is processed once, the complexity is O(n).

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    DCG iterates once over n elements. [OK]
Hint: DCG iterates once over n elements. [OK]
Common Mistakes:
2. 
import numpy as np

# Backpropagation for a single neuron
inputs = np.array([1.0, 2.0, 3.0])
weights = np.array([0.2, 0.8, -0.5])
bias = 2.0
learning_rate = 0.01

d_output = np.array([1.0])  # Gradient of loss w.r.t output

# Incorrect gradient calculation
weights_gradient = d_output * inputs
weights -= learning_rate * weights_gradient
print(weights)
medium
A. Weights are updated correctly.
B. Weights are updated incorrectly due to incorrect learning rate.
C. Weights are not updated; learning rate is too small.
D. Weights are updated incorrectly due to missing chain rule.

Solution

  1. Step 1: Identify error in gradient calculation

    The code incorrectly applies the gradient directly without considering the chain rule.

  2. Step 2: Correct approach

    Should multiply by the derivative of the activation function.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Always apply chain rule in backpropagation [OK]
Hint: Always apply chain rule in backpropagation [OK]
Common Mistakes:
3. How can you optimize the backward pass of batch normalization to improve computational efficiency?
hard
A. Fuse batch normalization with preceding linear layers.
B. Pre-compute gradients for all layers.
C. Use a smaller batch size to reduce computation.
D. Use a higher learning rate to speed up convergence.

Solution

  1. Step 1: Identify optimization techniques

    Fusing operations can reduce computational overhead.

  2. Step 2: Evaluate each option

    Fusing batch normalization with linear layers reduces redundant computations.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Fuse layers to optimize batch norm [OK]
Hint: Fuse layers to optimize batch norm [OK]
Common Mistakes:
4. How can you optimize the implementation of K-Fold cross-validation to reduce computational overhead?
hard
A. Use fewer folds
B. Increase the number of epochs
C. Use a smaller dataset
D. Parallelize the training process

Solution

  1. Step 1: Identify optimization strategies

    Parallelizing the training process can significantly reduce computation time.

  2. Step 2: Eliminate wrong options

    Use fewer folds reduces validation quality. Use a smaller dataset affects model performance. Increase the number of epochs increases computation.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Parallelize to optimize K-Fold. [OK]
Hint: Parallelize to optimize K-Fold. [OK]
Common Mistakes:
5. How can you optimize the K-Means implementation to improve its performance on large datasets?
hard
A. Use the Elkan algorithm to reduce distance calculations.
B. Increase the number of iterations.
C. Decrease the number of clusters.
D. Use a smaller subset of data for training.

Solution

  1. Step 1: Identify performance bottlenecks

    The Elkan algorithm optimizes K-Means by reducing the number of distance calculations needed.

  2. Step 2: Eliminate wrong options

    Increase the number of iterations. increases computation. Decrease the number of clusters. may not be suitable for the data's natural structure. Use a smaller subset of data for training. reduces accuracy.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Elkan reduces distance calculations. [OK]
Hint: Elkan reduces distance calculations. [OK]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard