Autocorrelation analysis in ML Python - ML Experiment: Train & Evaluate

Experiment - Autocorrelation analysis

Problem:You have a time series dataset and want to understand if past values influence future values by measuring autocorrelation.

Current Metrics:Autocorrelation at lag 1: 0.85, lag 2: 0.65, lag 3: 0.40

Issue:High autocorrelation at early lags indicates strong dependence, but you want to confirm if this is statistically significant and identify the lag where autocorrelation becomes negligible.

Your Task

Calculate and plot the autocorrelation function (ACF) for the time series up to lag 20 and identify the lag where autocorrelation drops below the significance threshold.

Use Python and standard libraries like pandas, numpy, matplotlib, and statsmodels.

Do not use any pre-built functions that automatically interpret autocorrelation results; focus on calculation and visualization.

Hint 1

Hint 2

Hint 3

Solution

ML Python

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from statsmodels.graphics.tsaplots import plot_acf

# Generate example time series data with autocorrelation
np.random.seed(42)
size = 100
noise = np.random.normal(0, 1, size)
# Create an AR(1) process: x_t = 0.8 * x_{t-1} + noise
x = np.zeros(size)
for t in range(1, size):
    x[t] = 0.8 * x[t-1] + noise[t]

time_series = pd.Series(x)

# Plot autocorrelation function up to lag 20
plt.figure(figsize=(10, 5))
plot_acf(time_series, lags=20, alpha=0.05)
plt.title('Autocorrelation Function (ACF) up to lag 20')
plt.xlabel('Lag')
plt.ylabel('Autocorrelation')
plt.show()

# Calculate autocorrelation values manually for lags 1 to 20
def autocorr(series, lag):
    return series.autocorr(lag=lag)

for lag in range(1, 21):
    ac = autocorr(time_series, lag)
    print(f'Lag {lag}: Autocorrelation = {ac:.3f}')

Generated a synthetic AR(1) time series with known autocorrelation.

Used statsmodels plot_acf to visualize autocorrelation with confidence intervals.

Printed autocorrelation values for lags 1 to 20 to identify where it drops below significance.

Results Interpretation

Before: Only autocorrelation at lags 1, 2, and 3 was known (0.85, 0.65, 0.40).

After: Full autocorrelation profile up to lag 20 shows gradual decay, crossing confidence bounds near lag 10.

Autocorrelation analysis helps identify how far back in time past values influence the current value. The confidence intervals show which lags have statistically significant correlation, guiding model choices like lag order in time series forecasting.

Bonus Experiment

Try the same autocorrelation analysis on a time series with no autocorrelation (pure random noise) and compare the results.

💡 Hint

Generate a time series with random normal values only and plot its ACF to see mostly insignificant autocorrelations within confidence bounds.