Train-test split for time series in ML Python - Model Metrics & Evaluation

Metrics & Evaluation - Train-test split for time series

Which metric matters for train-test split in time series and WHY

In time series, the order of data is important. We split data by time, not randomly. This means the test set is always later in time than the train set. Metrics like Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), or for classification Precision and Recall are used to measure how well the model predicts future data.

We focus on metrics that show how well the model predicts unseen future points because time series models must generalize forward in time.

Confusion matrix or equivalent visualization

For classification time series, a confusion matrix shows:

      | Predicted Positive | Predicted Negative |
      |--------------------|--------------------|
      | True Positive (TP)  | False Positive (FP) |
      | False Negative (FN) | True Negative (TN)  |
    

For regression time series, we use error metrics like:

      MAE = (1/n) * Σ |y_true - y_pred|
      RMSE = sqrt((1/n) * Σ (y_true - y_pred)^2)
    

These measure how far predictions are from actual future values.

Precision vs Recall tradeoff with concrete examples

In time series classification, precision and recall tradeoff matters depending on the problem:

• High precision: If you predict an event (like a machine failure), you want to be sure it really happens. False alarms (false positives) are costly.
• High recall: If missing an event is dangerous (like predicting floods), you want to catch all events, even if some false alarms happen.

Choosing which metric to prioritize depends on the cost of false positives vs false negatives in your time series problem.

What "good" vs "bad" metric values look like for this use case

For time series train-test split:

• Good: Test error (MAE, RMSE) is close to train error, showing the model predicts future data well without overfitting.
• Bad: Test error is much higher than train error, meaning the model fails to generalize forward in time.
• For classification, precision and recall above 0.8 are usually good, but it depends on the problem.

Metrics pitfalls

• Random split: Splitting time series data randomly breaks time order and leaks future info into training, giving overly optimistic metrics.
• Ignoring seasonality: Not accounting for time patterns can cause misleading metrics.
• Overfitting: Very low train error but high test error means the model memorizes past data but fails on future data.
• Accuracy paradox: High accuracy can be misleading if data is imbalanced (e.g., many no-event days).

Self-check question

Your time series model has 98% accuracy but only 12% recall on rare event detection in the test set. Is it good for production? Why or why not?

Answer: No, it is not good. The model misses 88% of actual events (low recall), which is dangerous if events are important. High accuracy is misleading because most data points are no-event. You should improve recall to catch more events.

Key Result

In time series train-test split, preserving time order is key; metrics like MAE, RMSE, precision, and recall must reflect true future prediction performance without data leakage.

Practice

1. Why is it important to keep the order of data when doing a train-test split for time series?

easy

A. Because time series data depends on the order of events and future data should not be used to predict past data.

B. Because random shuffling improves model accuracy in time series.

C. Because train and test sets must have the same number of samples.

D. Because test data should always come before train data.

Solution

1. Step 1: Understand time series data nature

   Time series data is sequential and depends on the order of events over time.

2. Step 2: Importance of order in train-test split

   Using future data to predict past data breaks the time flow and causes unrealistic model evaluation.

3. Final Answer:

   Because time series data depends on the order of events and future data should not be used to predict past data. -> Option A

4. Quick Check:

   Keep order to respect time flow = A [OK]

Hint: Always keep time order to avoid future data leakage [OK]

Common Mistakes:

• Randomly shuffling time series data
• Mixing future data into training
• Ignoring time dependency

2. Which of the following Python code snippets correctly splits a time series dataset data into 80% train and 20% test sets while preserving order?

easy

A. train = data[:int(len(data)*0.8)]
test = data[int(len(data)*0.8):]

B. train = data.sample(frac=0.8)
test = data.drop(train.index)

C. train = data[int(len(data)*0.2):]
test = data[:int(len(data)*0.2)]

D. train = data.shuffle().iloc[:80]
test = data.shuffle().iloc[80:]

Solution

1. Step 1: Understand slicing for time series split

   We use slicing to keep the order: first 80% for training, last 20% for testing.

2. Step 2: Check each code snippet

   train = data[:int(len(data)*0.8)] test = data[int(len(data)*0.8):] slices data correctly without shuffling. Options B and D shuffle data, breaking order. train = data[int(len(data)*0.2):] test = data[:int(len(data)*0.2)] reverses train and test.

3. Final Answer:

   train = data[:int(len(data)*0.8)] test = data[int(len(data)*0.8):] -> Option A

4. Quick Check:

   Slicing without shuffle = C [OK]

Hint: Use slicing, not shuffle, to keep time order [OK]

Common Mistakes:

• Using sample() which shuffles data
• Reversing train and test slices
• Shuffling data before splitting

3. Given the following code, what will be the length of test if data has 1000 samples?

split_index = int(len(data) * 0.75)
train = data[:split_index]
test = data[split_index:]

medium

A. 750

B. 250

C. 1000

D. 500

Solution

1. Step 1: Calculate split index

   split_index = int(1000 * 0.75) = 750

2. Step 2: Calculate test length

   test = data[750:] means test has samples from index 750 to 999, total 1000 - 750 = 250 samples.

3. Final Answer:

   250 -> Option B

4. Quick Check:

   Test length = total - train length = 250 [OK]

Hint: Test size = total samples minus train size [OK]

Common Mistakes:

• Confusing train size with test size
• Forgetting zero-based indexing
• Using float instead of int for index

4. You wrote this code to split a time series dataset data:

from sklearn.model_selection import train_test_split
train, test = train_test_split(data, test_size=0.2)

What is the main problem with this approach?

medium

A. test_size=0.2 is too small for time series

B. train and test sets will have overlapping samples

C. train_test_split cannot handle numeric data

D. train_test_split shuffles data by default, breaking time order

Solution

1. Step 1: Understand train_test_split default behavior

   By default, train_test_split shuffles data before splitting.

2. Step 2: Why shuffling is a problem for time series

   Shuffling breaks the time order, causing future data to leak into training, invalidating model evaluation.

3. Final Answer:

   train_test_split shuffles data by default, breaking time order -> Option D

4. Quick Check:

   Default shuffle breaks time order = B [OK]

Hint: train_test_split shuffles unless shuffle=False [OK]

Common Mistakes:

• Ignoring shuffle=True default
• Assuming test_size controls order
• Thinking train_test_split is time-series aware

5. You have daily sales data for 3 years and want to train a model to predict future sales. Which approach correctly splits the data to train on the first 2.5 years and test on the last 0.5 year, ensuring no data leakage?

hard

A. train = data[int(len(data)*0.5):]
test = data[:int(len(data)*0.5)]

B. train = data.sample(frac=0.83)
test = data.drop(train.index)

C. train = data[:int(len(data)*5/6)]
test = data[int(len(data)*5/6):]

D. train = data.shuffle().iloc[:900]
test = data.shuffle().iloc[900:]

Solution

1. Step 1: Calculate split fraction for 2.5 years out of 3 years

   2.5 years / 3 years = 5/6 ≈ 0.8333, so train is first 5/6 of data.

2. Step 2: Use slicing to split data preserving order

   train = data[:int(len(data)*5/6)] test = data[int(len(data)*5/6):] slices data correctly from start to 5/6 for train, and last 1/6 for test, preserving time order and avoiding leakage.

3. Final Answer:

   train = data[:int(len(data)*5/6)] test = data[int(len(data)*5/6):] -> Option C

4. Quick Check:

   Slice first 5/6 for train, last 1/6 for test = A [OK]

Hint: Split by slicing using fraction of total length [OK]

Common Mistakes:

• Using random sampling instead of slicing
• Reversing train and test sets
• Shuffling data before splitting