Logistic regression in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Logistic regression

Which metric matters for Logistic Regression and WHY

Logistic regression is used for classification, often binary (yes/no). The key metrics are Accuracy, Precision, Recall, and F1-score. Accuracy shows overall correctness. Precision tells us how many predicted positives are truly positive. Recall shows how many actual positives were found. F1-score balances precision and recall. These metrics help us understand how well the model separates classes.

Confusion Matrix for Logistic Regression

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP)  | False Positive (FP) |
      | False Negative (FN) | True Negative (TN)  |

Example:

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | 50                 | 5                  |
      | 10                 | 35                 |

Total samples = TP + FP + TN + FN = 50 + 5 + 35 + 10 = 100
    

Precision vs Recall Tradeoff in Logistic Regression

Imagine a spam email detector using logistic regression:

• High Precision: Few good emails are marked as spam. This means when the model says "spam," it is usually right.
• High Recall: Most spam emails are caught. This means the model finds almost all spam, but might mark some good emails as spam.

Choosing which to prioritize depends on the problem. For spam, high precision avoids losing good emails. For medical tests, high recall avoids missing sick patients.

Good vs Bad Metric Values for Logistic Regression

Good metrics mean the model predicts well:

• Accuracy above 80% is often good, but depends on data balance.
• Precision and recall above 70% show balanced performance.
• F1-score above 0.7 means good balance between precision and recall.

Bad metrics look like:

• Accuracy near 50% on balanced data means guessing.
• Precision very low means many false alarms.
• Recall very low means many misses.
• F1-score near 0.5 or below means poor balance.

Common Pitfalls in Logistic Regression Metrics

• Accuracy Paradox: High accuracy can be misleading if classes are imbalanced. For example, 95% accuracy if 95% of data is one class but model ignores the other.
• Data Leakage: If test data leaks into training, metrics look unrealistically good.
• Overfitting: Very high training accuracy but low test accuracy means the model memorizes training data but fails on new data.
• Ignoring Class Imbalance: Not using precision, recall, or F1 when classes are uneven can hide poor performance on minority class.

Self Check: Is a Model with 98% Accuracy but 12% Recall on Fraud Good?

No, this model is not good for fraud detection. Although 98% accuracy sounds high, the recall of 12% means it only finds 12% of actual fraud cases. This is dangerous because most fraud goes undetected. For fraud, high recall is critical to catch as many frauds as possible, even if precision is lower.

Key Result

Logistic regression evaluation focuses on precision, recall, and F1-score to balance correct positive predictions and missed cases.

Practice

1. What is the main purpose of logistic regression in machine learning?

easy

A. To cluster data points into groups

B. To predict binary outcomes like yes/no or true/false

C. To reduce the number of features in data

D. To generate random data samples

Solution

1. Step 1: Understand logistic regression goal

   Logistic regression is used to predict categories with two possible outcomes, such as yes/no or true/false.

2. Step 2: Compare with other options

   Clustering groups data (not logistic regression), feature reduction simplifies data, and random sampling is unrelated.

3. Final Answer:

   To predict binary outcomes like yes/no or true/false -> Option B

4. Quick Check:

   Logistic regression = binary prediction [OK]

Hint: Logistic regression predicts yes/no answers fast [OK]

Common Mistakes:

• Confusing logistic regression with clustering
• Thinking it reduces features
• Assuming it generates data

2. Which of the following is the correct way to import logistic regression from scikit-learn in Python?

easy

A. import logistic_regression from sklearn.linear_model

B. import LogisticRegression from sklearn.linear_model

C. from sklearn import LogisticRegression

D. from sklearn.linear_model import LogisticRegression

Solution

1. Step 1: Recall correct import syntax

   In Python, the correct import uses 'from module import class' format, so 'from sklearn.linear_model import LogisticRegression' is correct.

2. Step 2: Identify incorrect syntax

   Options B, C, and D use wrong order or wrong names causing syntax errors.

3. Final Answer:

   from sklearn.linear_model import LogisticRegression -> Option D

4. Quick Check:

   Correct import = from sklearn.linear_model import LogisticRegression [OK]

Hint: Use 'from module import class' for sklearn imports [OK]

Common Mistakes:

• Using wrong import order
• Wrong class or module names
• Using lowercase instead of correct class name

3. What will be the output of this Python code snippet?

from sklearn.linear_model import LogisticRegression
import numpy as np

X = np.array([[0], [1], [2], [3]])
y = np.array([0, 0, 1, 1])
model = LogisticRegression()
model.fit(X, y)
pred = model.predict([[1.5]])
print(pred)

medium

A. [1]

B. [0]

C. [2]

D. Error

Solution

1. Step 1: Understand training data and labels

   X has values 0,1 labeled 0 and 2,3 labeled 1. The model learns to separate near 1.5 between classes 0 and 1.

2. Step 2: Predict for input 1.5

   Since 1.5 is closer to class 1 range, logistic regression predicts class 1.

3. Final Answer:

   [1] -> Option A

4. Quick Check:

   Input 1.5 predicts class 1 [OK]

Hint: Predict near boundary to guess class [OK]

Common Mistakes:

• Assuming prediction is 0 for 1.5
• Expecting error due to input shape
• Confusing output format

4. Identify the error in this logistic regression code snippet:

from sklearn.linear_model import LogisticRegression
X = [[0], [1], [2], [3]]
y = [0, 0, 1, 1]
model = LogisticRegression()
model.fit(X, y)
pred = model.predict([1.5])
print(pred)

medium

A. Input to predict should be 2D array, not 1D

B. Labels y should be strings, not integers

C. Model must be initialized with parameters

D. fit method requires numpy arrays, not lists

Solution

1. Step 1: Check input shape for prediction

   Model expects 2D array for predict input, but [1.5] is 1D. It should be [[1.5]].

2. Step 2: Verify other parts

   Labels as integers are fine, model can be default, and fit accepts lists converted internally.

3. Final Answer:

   Input to predict should be 2D array, not 1D -> Option A

4. Quick Check:

   Predict input shape = 2D array [OK]

Hint: Always use double brackets for single sample prediction [OK]

Common Mistakes:

• Passing 1D array to predict
• Thinking labels must be strings
• Assuming model needs parameters to initialize

5. You have a dataset with features that are not scaled and vary widely in range. What is the best practice before applying logistic regression to improve model performance?

hard

A. Use logistic regression without any preprocessing

B. Remove half of the features randomly

C. Scale or normalize features to similar ranges

D. Increase the number of training samples without scaling

Solution

1. Step 1: Understand feature scaling importance

   Logistic regression uses weights sensitive to feature scale; large differences can cause poor training.

2. Step 2: Choose best preprocessing

   Scaling or normalizing features to similar ranges helps the model learn better and converge faster.

3. Final Answer:

   Scale or normalize features to similar ranges -> Option C

4. Quick Check:

   Feature scaling improves logistic regression [OK]

Hint: Always scale features before logistic regression [OK]

Common Mistakes:

• Ignoring feature scaling
• Randomly removing features
• Assuming more data fixes scaling issues