Autocorrelation analysis in ML Python - ML Experiment: Train & Evaluate

Experiment - Autocorrelation analysis

Problem:You have a time series dataset and want to understand if past values influence future values by measuring autocorrelation.

Current Metrics:Autocorrelation at lag 1: 0.85, lag 2: 0.65, lag 3: 0.40

Issue:High autocorrelation at early lags indicates strong dependence, but you want to confirm if this is statistically significant and identify the lag where autocorrelation becomes negligible.

Your Task

Calculate and plot the autocorrelation function (ACF) for the time series up to lag 20 and identify the lag where autocorrelation drops below the significance threshold.

Use Python and standard libraries like pandas, numpy, matplotlib, and statsmodels.

Do not use any pre-built functions that automatically interpret autocorrelation results; focus on calculation and visualization.

Hint 1

Hint 2

Hint 3

Solution

ML Python

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from statsmodels.graphics.tsaplots import plot_acf

# Generate example time series data with autocorrelation
np.random.seed(42)
size = 100
noise = np.random.normal(0, 1, size)
# Create an AR(1) process: x_t = 0.8 * x_{t-1} + noise
x = np.zeros(size)
for t in range(1, size):
    x[t] = 0.8 * x[t-1] + noise[t]

time_series = pd.Series(x)

# Plot autocorrelation function up to lag 20
plt.figure(figsize=(10, 5))
plot_acf(time_series, lags=20, alpha=0.05)
plt.title('Autocorrelation Function (ACF) up to lag 20')
plt.xlabel('Lag')
plt.ylabel('Autocorrelation')
plt.show()

# Calculate autocorrelation values manually for lags 1 to 20
def autocorr(series, lag):
    return series.autocorr(lag=lag)

for lag in range(1, 21):
    ac = autocorr(time_series, lag)
    print(f'Lag {lag}: Autocorrelation = {ac:.3f}')

Generated a synthetic AR(1) time series with known autocorrelation.

Used statsmodels plot_acf to visualize autocorrelation with confidence intervals.

Printed autocorrelation values for lags 1 to 20 to identify where it drops below significance.

Results Interpretation

Before: Only autocorrelation at lags 1, 2, and 3 was known (0.85, 0.65, 0.40).

After: Full autocorrelation profile up to lag 20 shows gradual decay, crossing confidence bounds near lag 10.

Autocorrelation analysis helps identify how far back in time past values influence the current value. The confidence intervals show which lags have statistically significant correlation, guiding model choices like lag order in time series forecasting.

Bonus Experiment

Try the same autocorrelation analysis on a time series with no autocorrelation (pure random noise) and compare the results.

💡 Hint

Generate a time series with random normal values only and plot its ACF to see mostly insignificant autocorrelations within confidence bounds.

Practice

1. What does autocorrelation measure in a time series dataset?

easy

A. The difference between the highest and lowest values in the data

B. The total sum of all data points in the series

C. The average value of the dataset

D. The relationship between current data points and past data points at different time lags

Solution

1. Step 1: Understand autocorrelation concept

   Autocorrelation checks how current values relate to past values at various time gaps (lags).

2. Step 2: Compare options to definition

   Only The relationship between current data points and past data points at different time lags correctly describes this relationship; others describe unrelated statistics.

3. Final Answer:

   The relationship between current data points and past data points at different time lags -> Option D

4. Quick Check:

   Autocorrelation = relationship with past points [OK]

Hint: Autocorrelation links current data to past data points [OK]

Common Mistakes:

• Confusing autocorrelation with average or sum
• Thinking it measures difference between max and min
• Assuming it only looks at immediate previous point

2. Which of the following Python code snippets correctly computes the autocorrelation at lag 1 for a list data?

easy

A. import numpy as np
np.corrcoef(data[:-1], data[1:])[0,1]

B. np.corrcoef(data, data)[0,1]

C. np.mean(data) - np.mean(data[1:])

D. np.sum(data) / len(data)

Solution

1. Step 1: Understand autocorrelation calculation

   Autocorrelation at lag 1 compares data points with the next point, so we correlate data[:-1] with data[1:].

2. Step 2: Check code correctness

   import numpy as np np.corrcoef(data[:-1], data[1:])[0,1] uses np.corrcoef correctly on shifted slices; others do not compute correlation at lag 1.

3. Final Answer:

   import numpy as np\nnp.corrcoef(data[:-1], data[1:])[0,1] -> Option A

4. Quick Check:

   Shifted slices correlation = import numpy as np np.corrcoef(data[:-1], data[1:])[0,1] [OK]

Hint: Use shifted slices for lag correlation in numpy [OK]

Common Mistakes:

• Using correlation of data with itself (option B)
• Calculating mean difference instead of correlation
• Using sum or mean instead of correlation

3. Given the time series data = [2, 4, 6, 8, 10], what is the autocorrelation at lag 1 using numpy's correlation coefficient?

medium

A. 0.9

B. 1.0

C. 0.8

D. 0.0

Solution

1. Step 1: Prepare shifted data slices

   data[:-1] = [2,4,6,8], data[1:] = [4,6,8,10]

2. Step 2: Calculate correlation coefficient

   These slices are perfectly linearly increasing, so correlation is 1.0.

3. Final Answer:

   1.0 -> Option B

4. Quick Check:

   Perfect linear increase = autocorrelation 1.0 [OK]

Hint: Perfect linear sequences have autocorrelation 1.0 [OK]

Common Mistakes:

• Calculating correlation with full data instead of shifted slices
• Confusing correlation with difference or ratio
• Rounding errors leading to wrong decimals

4. The following code attempts to compute autocorrelation at lag 2 but gives an error. What is the error?

import numpy as np
data = [1, 3, 5, 7, 9]
result = np.corrcoef(data[:-2], data[2:])[0,2]

medium

A. IndexError because index 2 is out of bounds for the correlation matrix

B. TypeError because data is a list, not a numpy array

C. ValueError because data slices have different lengths

D. No error, code runs correctly

Solution

1. Step 1: Analyze np.corrcoef output shape

   np.corrcoef returns a 2x2 matrix for two input arrays, so valid indices are 0 or 1.

2. Step 2: Check indexing in code

   Accessing [0,2] is invalid and causes IndexError.

3. Final Answer:

   IndexError because index 2 is out of bounds for the correlation matrix -> Option A

4. Quick Check:

   Correlation matrix max index = 1, so index 2 causes error [OK]

Hint: Correlation matrix for two arrays is 2x2, max index 1 [OK]

Common Mistakes:

• Assuming list input causes TypeError
• Thinking slices have different lengths (they are equal)
• Believing code runs without error

5. You have daily sales data showing a weekly pattern. How can autocorrelation analysis help you detect this seasonality?

hard

A. By plotting sales against time without any lag analysis

B. By calculating the average sales over the entire dataset

C. By computing autocorrelation at lag 7 to check if sales on a day relate to sales 7 days before

D. By computing autocorrelation only at lag 1

Solution

1. Step 1: Understand weekly seasonality

   Weekly seasonality means patterns repeat every 7 days.

2. Step 2: Use autocorrelation at lag 7

   Computing autocorrelation at lag 7 checks if sales today relate to sales 7 days ago, revealing weekly patterns.

3. Final Answer:

   By computing autocorrelation at lag 7 to check if sales on a day relate to sales 7 days before -> Option C

4. Quick Check:

   Weekly pattern detected by lag 7 autocorrelation [OK]

Hint: Match lag to season length to find repeating patterns [OK]

Common Mistakes:

• Using lag 1 only misses weekly pattern
• Ignoring lag and just averaging data
• Plotting without lag analysis misses seasonality