Logistic regression in ML Python - Deep Dive

Overview - Logistic regression

What is it?

Logistic regression is a way to predict if something belongs to one group or another using numbers. It looks at input features and calculates the chance that the answer is yes or no. Instead of drawing a straight line like in regular regression, it draws an S-shaped curve to keep predictions between 0 and 1. This helps us make decisions like yes/no or true/false based on data.

Why it matters

Without logistic regression, many important decisions like detecting spam emails, deciding if a patient has a disease, or approving loans would be much harder. It turns complex data into simple yes/no answers with a clear probability. This makes machines smarter and helps people make better choices quickly and reliably.

Where it fits

Before learning logistic regression, you should understand basic algebra and simple linear regression, which predicts continuous numbers. After logistic regression, you can explore more complex classification methods like decision trees, support vector machines, and neural networks.

Mental Model

Core Idea

Logistic regression predicts the chance of a yes/no outcome by squeezing a straight line into an S-shaped curve that outputs probabilities between 0 and 1.

Think of it like...

Imagine a dimmer switch that controls the brightness of a light. Instead of just on or off, it smoothly changes brightness from dark to bright. Logistic regression smoothly changes predictions from 0 (no) to 1 (yes) instead of jumping suddenly.

Input features (x) ──▶ Linear combination (z = w·x + b) ──▶ Sigmoid function (S-shaped curve) ──▶ Probability output (0 to 1) ──▶ Decision threshold (e.g., 0.5) ──▶ Class label (yes/no)

Build-Up - 7 Steps

1

Foundation

Understanding binary classification basics

Concept: Learn what it means to classify data into two groups.

Binary classification means sorting things into two categories, like yes/no or spam/not spam. Each example has features (like age, income) and a label (0 or 1). The goal is to find a rule that guesses the label from the features.

Result

You understand the problem logistic regression solves: deciding between two classes.

Knowing the problem type helps you see why logistic regression outputs probabilities instead of continuous numbers.

2

Foundation

Reviewing linear regression basics

Concept: Recall how linear regression predicts continuous values using a line.

Linear regression finds a line that best fits data points by combining features with weights and adding a bias. The output can be any number, positive or negative, which works for predicting things like price or temperature.

Result

You see how linear regression forms the base for logistic regression but needs adjustment for classification.

Understanding linear regression helps you grasp why logistic regression needs a special function to limit outputs.

3

Intermediate

Introducing the sigmoid function

🤔Before reading on: do you think a straight line can directly predict probabilities between 0 and 1? Commit to yes or no.

Concept: Learn the sigmoid function that turns any number into a value between 0 and 1.

The sigmoid function is S-shaped and defined as 1 / (1 + e^(-z)), where z is the linear combination of inputs. It squashes any input number into a probability between 0 and 1, perfect for yes/no predictions.

Result

You can convert linear outputs into probabilities, making logistic regression possible.

Knowing the sigmoid function is key because it transforms raw scores into meaningful probabilities.

4

Intermediate

Training logistic regression with loss function

🤔Before reading on: do you think minimizing squared error is best for logistic regression? Commit to yes or no.

Concept: Understand how logistic regression learns by minimizing a special loss called cross-entropy.

Instead of squared error, logistic regression uses cross-entropy loss, which measures how close predicted probabilities are to actual labels. The model adjusts weights to reduce this loss using methods like gradient descent.

Result

You know how the model learns to make better predictions by comparing probabilities to true labels.

Using cross-entropy loss matches the probability output and classification goal, improving learning effectiveness.

5

Intermediate

Making predictions and setting thresholds

Concept: Learn how to turn probabilities into yes/no decisions using a cutoff.

After training, logistic regression outputs probabilities. To decide the class, pick a threshold (usually 0.5). If probability ≥ threshold, predict yes (1); otherwise, no (0). Adjusting threshold changes sensitivity and specificity.

Result

You can convert smooth probabilities into clear class labels for practical use.

Understanding thresholds helps balance false positives and false negatives depending on the problem.

6

Advanced

Interpreting model coefficients

🤔Before reading on: do you think logistic regression coefficients represent direct changes in probability? Commit to yes or no.

Concept: Learn how to interpret weights as influence on odds, not probabilities directly.

Coefficients show how each feature changes the log-odds of the outcome. Positive weight increases odds, negative decreases. Odds are related to probability but not the same. This helps explain feature importance.

Result

You can explain model decisions and feature effects clearly.

Knowing the difference between odds and probabilities prevents misinterpretation of model outputs.

7

Expert

Handling imbalanced data and regularization

🤔Before reading on: do you think logistic regression always performs well on imbalanced classes without adjustments? Commit to yes or no.

Concept: Explore techniques to improve logistic regression when classes are uneven or to prevent overfitting.

Imbalanced data can bias predictions toward the majority class. Solutions include adjusting class weights or thresholds. Regularization (L1 or L2) adds penalties to weights to avoid overfitting and improve generalization.

Result

You can build robust logistic regression models that work well in real-world messy data.

Understanding these techniques is crucial for applying logistic regression beyond textbook examples.

Under the Hood

Logistic regression calculates a weighted sum of input features plus a bias, producing a number called logit. This logit passes through the sigmoid function, converting it into a probability between 0 and 1. During training, the model adjusts weights to minimize cross-entropy loss, which measures the difference between predicted probabilities and actual labels. Gradient descent updates weights step-by-step to reduce this loss. Internally, the model works with log-odds, which relate linearly to features, making interpretation possible.

Why designed this way?

Logistic regression was designed to extend linear regression to classification problems by mapping outputs to probabilities. The sigmoid function was chosen because it smoothly bounds outputs between 0 and 1, matching probability requirements. Cross-entropy loss aligns with maximum likelihood estimation for Bernoulli-distributed labels, providing a statistically sound training objective. Alternatives like thresholding linear regression outputs or using other link functions exist but sigmoid with cross-entropy became standard due to simplicity and effectiveness.

┌───────────────┐
│ Input features│
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Weighted sum  │ z = w·x + b
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Sigmoid func  │ σ(z) = 1/(1+e^-z)
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Probability   │ Between 0 and 1
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Thresholding  │ e.g., 0.5 cutoff
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Class label   │ 0 or 1
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does logistic regression predict exact probabilities or just class labels? Commit to one.

Common Belief: Logistic regression directly predicts the final class label without probabilities.

Reality: It predicts probabilities between 0 and 1, which are then converted to class labels using a threshold.

Why it matters: Ignoring probabilities loses valuable information about prediction confidence, which is important for decision-making.

Quick: Is logistic regression only useful for linear relationships? Commit to yes or no.

Common Belief: Logistic regression can only model linear boundaries between classes.

Reality: While logistic regression models linear decision boundaries, using feature transformations or interactions can capture non-linear patterns.

Why it matters: Believing it only models linear data limits its use and prevents creative feature engineering.

Quick: Does a positive coefficient mean the probability increases by that amount? Commit to yes or no.

Common Belief: A positive coefficient means the probability increases by that coefficient value directly.

Reality: Coefficients affect the log-odds, not the probability directly; the relationship is nonlinear.

Why it matters: Misinterpreting coefficients can lead to wrong conclusions about feature effects.

Quick: Can logistic regression handle multi-class problems without changes? Commit to yes or no.

Common Belief: Logistic regression naturally handles more than two classes without modification.

Reality: Standard logistic regression is binary; multi-class requires extensions like one-vs-rest or softmax regression.

Why it matters: Using logistic regression directly on multi-class data without adjustments leads to poor performance.

Expert Zone

1

Regularization not only prevents overfitting but can also perform feature selection when using L1 penalty, shrinking some weights exactly to zero.

2

The choice of threshold affects precision and recall trade-offs, which is critical in domains like medical diagnosis or fraud detection.

3

Logistic regression coefficients can be unstable with highly correlated features, requiring techniques like feature decorrelation or dimensionality reduction.

When NOT to use

Avoid logistic regression when the relationship between features and outcome is highly nonlinear and complex; consider decision trees, random forests, or neural networks instead. Also, logistic regression struggles with very high-dimensional sparse data without proper regularization or feature engineering.

Production Patterns

In production, logistic regression is often used as a baseline model due to its simplicity and interpretability. It is combined with feature scaling, regularization, and threshold tuning. It also serves as a component in ensemble methods or as a final layer in some neural networks for binary classification.

Connections

Linear regression

Logistic regression builds on linear regression by adding a sigmoid function to output probabilities.

Understanding linear regression helps grasp logistic regression’s foundation and why it needs a special function to handle classification.

Maximum likelihood estimation

Logistic regression training uses maximum likelihood estimation to find the best parameters.

Knowing maximum likelihood explains why cross-entropy loss is the natural choice for logistic regression.

Epidemiology (Odds ratio)

Logistic regression coefficients relate to odds ratios used in epidemiology to measure risk factors.

Recognizing this connection helps interpret model coefficients as changes in odds, bridging statistics and machine learning.

Common Pitfalls

#1 Using linear regression to predict binary outcomes directly.

Wrong approach:

model = LinearRegression()
model.fit(X_train, y_train)
predictions = model.predict(X_test)
labels = (predictions > 0.5).astype(int)

Correct approach:

model = LogisticRegression()
model.fit(X_train, y_train)
predictions = model.predict(X_test)

Root cause:

Misunderstanding that linear regression outputs unbounded values unsuitable for probabilities.

#2 Ignoring feature scaling before training logistic regression.

Wrong approach:

model = LogisticRegression()
model.fit(X_train, y_train)

Correct approach:

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
model = LogisticRegression()
model.fit(X_train_scaled, y_train)

Root cause:

Not realizing that features with different scales can cause slow or unstable training.

#3 Using default threshold 0.5 without considering class imbalance.

Wrong approach:

pred_probs = model.predict_proba(X_test)[:,1]
pred_labels = (pred_probs > 0.5).astype(int)

Correct approach:

pred_probs = model.predict_proba(X_test)[:,1]
threshold = 0.3 # adjusted for imbalance
pred_labels = (pred_probs > threshold).astype(int)

Root cause:

Assuming 0.5 threshold is always optimal regardless of data distribution.

Key Takeaways

Logistic regression predicts the probability of a binary outcome by applying a sigmoid function to a linear combination of features.

It uses cross-entropy loss to train the model, which aligns with the goal of predicting probabilities accurately.

Coefficients represent changes in log-odds, not direct probability changes, which is important for correct interpretation.

Adjusting the decision threshold and using regularization are key to handling real-world challenges like imbalanced data and overfitting.

Logistic regression is simple, interpretable, and a foundational tool that connects statistics and machine learning.