Creating interaction features in ML Python - Mechanics & Internals

Overview - Creating interaction features

What is it?

Creating interaction features means combining two or more original features in your data to make new features that capture how they work together. These new features can help machine learning models find patterns that single features alone might miss. For example, multiplying two features can show their combined effect on the target. Interaction features are especially useful when the relationship between features affects the outcome.

Why it matters

Without interaction features, models might miss important combined effects between variables, leading to weaker predictions. For example, in predicting house prices, the effect of location and house size together might be more important than each alone. Creating interaction features helps models understand these combined effects, improving accuracy and insights. This can lead to better decisions in business, healthcare, and many fields.

Where it fits

Before learning about interaction features, you should understand basic features and how machine learning models use them. After this, you can learn about feature engineering techniques like polynomial features, feature selection, and model interpretation. Interaction features are part of the broader skill of making data more informative for models.

Mental Model

Core Idea

Interaction features capture how two or more original features combine to influence the outcome in ways single features cannot show alone.

Think of it like...

It's like mixing colors: red and blue alone are simple, but when mixed, they create purple, a new color that tells a different story.

Original Features
  ├─ Feature A
  ├─ Feature B
  └─ Feature C

Interaction Features
  ├─ A × B
  ├─ B × C
  └─ A × C

Model Input
  ├─ Feature A
  ├─ Feature B
  ├─ Feature C
  ├─ A × B
  ├─ B × C
  └─ A × C

Build-Up - 7 Steps

1

Foundation

Understanding basic features

Concept: Learn what features are and how they represent data points.

Features are individual measurable properties or characteristics of data. For example, in a dataset about cars, features could be 'engine size', 'weight', or 'color'. Each feature helps the model understand the data better.

Result

You can identify and describe features in any dataset.

Knowing what features are is essential because interaction features build on combining these basic building blocks.

2

Foundation

Why features matter in models

Concept: Understand how machine learning models use features to make predictions.

Models look at features to find patterns that relate to the target outcome. For example, a model predicting house prices might learn that bigger houses usually cost more. Each feature contributes some information to the model's decision.

Result

You see how features influence model predictions.

Recognizing the role of features helps you appreciate why combining them can reveal deeper patterns.

3

Intermediate

What are interaction features

🤔Before reading on: do you think combining features means just adding them or something more? Commit to your answer.

Concept: Interaction features are new features created by combining two or more original features to capture their joint effect.

Instead of using features alone, interaction features multiply or combine them to show how they work together. For example, if feature A is 'hours studied' and feature B is 'class attendance', their product A×B can show how studying more and attending class together affect grades.

Result

You can create new features that represent combined effects.

Understanding interaction features lets you capture relationships that single features miss, improving model insight.

4

Intermediate

Common methods to create interaction features

🤔Before reading on: do you think interaction features are only products, or can they be other combinations? Commit to your answer.

Concept: Learn different ways to combine features, like multiplication, addition, or concatenation.

The most common interaction is multiplication (e.g., A×B). Addition (A+B) or subtraction (A-B) can also be interactions but usually less informative. For categorical features, combining categories (like 'red' and 'large') into a new category is another interaction method.

Result

You know multiple ways to create interaction features depending on data type.

Knowing various methods helps you choose the best interaction type for your data and problem.

5

Intermediate

Using interaction features in models

🤔Before reading on: do you think adding interaction features always improves model accuracy? Commit to your answer.

Concept: Understand how interaction features affect model training and performance.

Adding interaction features can help models learn complex patterns but also risks overfitting if too many are added. It's important to select meaningful interactions and sometimes use regularization to avoid noise.

Result

You can balance adding interaction features with model complexity.

Knowing the tradeoff prevents blindly adding features that hurt model generalization.

6

Advanced

Automated interaction feature generation

🤔Before reading on: do you think interaction features must be created manually or can tools help? Commit to your answer.

Concept: Learn about tools and techniques that automatically create interaction features.

Libraries like scikit-learn have classes (e.g., PolynomialFeatures) that generate interaction features automatically. These tools create all combinations up to a certain degree, saving time but requiring careful selection to avoid too many features.

Result

You can use automation to efficiently create interaction features.

Understanding automation helps scale feature engineering but requires knowledge to manage feature explosion.

7

Expert

Interaction features and model interpretability

🤔Before reading on: do you think interaction features make models easier or harder to interpret? Commit to your answer.

Concept: Explore how interaction features affect understanding model decisions.

While interaction features can improve accuracy, they can also make models harder to interpret because combined features are less intuitive. Techniques like SHAP values or partial dependence plots help explain how interactions influence predictions.

Result

You can balance model accuracy with interpretability when using interaction features.

Knowing interpretability challenges guides better feature engineering and model explanation.

Under the Hood

Interaction features work by combining original feature values mathematically or categorically to create new dimensions in the data space. This allows models, especially linear ones, to capture nonlinear relationships by including terms that represent joint effects. Internally, these new features increase the input size, enabling the model to fit more complex patterns.

Why designed this way?

Interaction features were introduced to help simple models like linear regression capture complex relationships without switching to more complex models. Instead of relying on the model to guess interactions, explicitly creating them guides learning. Alternatives like kernel methods or deep learning can learn interactions implicitly but require more data and computation.

Original Features
  ┌─────────┐   ┌─────────┐   ┌─────────┐
  │Feature A│   │Feature B│   │Feature C│
  └────┬────┘   └────┬────┘   └────┬────┘
       │             │             │
       │             │             │
       │             │             │
       └─────┬───────┴─────┬───────┘
             │             │
      ┌──────▼─────┐ ┌─────▼─────┐
      │A × B       │ │B × C      │
      └────────────┘ └───────────┘
             │             │
             └─────┬───────┘
                   │
           ┌───────▼────────┐
           │Model Input     │
           │(Features +     │
           │Interaction Fs) │
           └────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do interaction features always improve model accuracy? Commit yes or no.

Common Belief: Adding interaction features always makes the model better.

Reality: Interaction features can improve or harm model performance depending on data and how many are added. Too many can cause overfitting.

Why it matters: Blindly adding interactions can make models worse and harder to maintain.

Quick: Are interaction features only useful for linear models? Commit yes or no.

Common Belief: Only linear models benefit from interaction features.

Reality: While linear models rely heavily on interaction features, nonlinear models like trees or neural networks can learn interactions implicitly without manual features.

Why it matters: Knowing this helps choose when to engineer interactions or rely on model capacity.

Quick: Do interaction features always have to be products of features? Commit yes or no.

Common Belief: Interaction features must be the product of two features.

Reality: Interactions can be created by other combinations like sums, differences, or categorical concatenations, depending on the problem.

Why it matters: Limiting to products can miss useful interactions and reduce model effectiveness.

Quick: Can interaction features make models easier to interpret? Commit yes or no.

Common Belief: Interaction features always make models easier to understand.

Reality: They often make models more complex and harder to interpret without special tools.

Why it matters: Misunderstanding this can lead to models that are accurate but opaque, reducing trust.

Expert Zone

1

Interaction features can introduce multicollinearity, making model coefficients unstable and harder to interpret.

2

Not all interactions are meaningful; domain knowledge helps select interactions that improve model quality and reduce noise.

3

Automated interaction generation can cause feature explosion, so dimensionality reduction or feature selection is often necessary.

When NOT to use

Avoid manual interaction features when using models like random forests or deep neural networks that learn interactions internally. Instead, focus on raw features and let the model discover interactions. Also, skip interaction features if data is very sparse or if interpretability is a top priority without explanation tools.

Production Patterns

In production, interaction features are often created selectively based on domain knowledge or automated feature selection pipelines. They are combined with regularization techniques like Lasso to prevent overfitting. Monitoring feature importance and model explanations helps maintain balance between accuracy and interpretability.

Connections

Polynomial regression

Interaction features are a subset of polynomial features that include powers and products of features.

Understanding interaction features helps grasp polynomial regression, which extends linear models to capture nonlinear patterns.

Feature crossing in recommender systems

Feature crossing is a form of interaction feature used to combine categorical variables for better recommendations.

Knowing interaction features clarifies how recommender systems capture complex user-item relationships.

Human decision making

Humans often consider combined factors (interactions) when making decisions, similar to interaction features in models.

Recognizing this connection helps appreciate why modeling interactions improves machine learning predictions.

Common Pitfalls

#1 Adding all possible interaction features without selection.

Wrong approach:

from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2, interaction_only=True, include_bias=False)
X_poly = poly.fit_transform(X)

Correct approach:

# Select meaningful features or use feature selection after generating interactions
from sklearn.preprocessing import PolynomialFeatures
from sklearn.feature_selection import SelectKBest, f_regression
poly = PolynomialFeatures(degree=2, interaction_only=True, include_bias=False)
X_poly = poly.fit_transform(X)
selector = SelectKBest(f_regression, k=10)
X_selected = selector.fit_transform(X_poly, y)

Root cause:

Not controlling feature explosion leads to too many features, causing overfitting and slow training.

#2 Creating interaction features for categorical variables by multiplying their codes.

Wrong approach:

df['interaction'] = df['cat_feature1'].astype(int) * df['cat_feature2'].astype(int)

Correct approach:

# Combine categories as strings to create meaningful interaction
df['interaction'] = df['cat_feature1'].astype(str) + '_' + df['cat_feature2'].astype(str)

Root cause:

Multiplying categorical codes treats categories as numbers, which misrepresents their meaning.

#3 Assuming interaction features always improve model interpretability.

Wrong approach:

model = LinearRegression()
model.fit(X_with_interactions, y)
print(model.coef_)

Correct approach:

# Use interpretation tools like SHAP to explain interactions
import shap
explainer = shap.Explainer(model, X_with_interactions)
shap_values = explainer(X_with_interactions)
shap.plots.beeswarm(shap_values)

Root cause:

Ignoring that combined features complicate coefficient meanings without explanation tools.

Key Takeaways

Interaction features combine original features to capture joint effects that single features miss.

They help simple models learn complex patterns but can increase model complexity and risk overfitting.

Creating interaction features requires careful selection and sometimes automation with controls to avoid too many features.

Not all models need manual interaction features; some learn interactions internally.

Understanding interaction features improves both model accuracy and the ability to explain predictions when used thoughtfully.