Data analysis agent pipeline in Agentic AI - ML Experiment: Train & Evaluate

Experiment - Data analysis agent pipeline

Problem:You have built a data analysis agent pipeline that processes raw data, extracts features, and makes predictions. Currently, the pipeline runs but the model predictions are inconsistent and the overall accuracy is low.

Current Metrics:Training accuracy: 65%, Validation accuracy: 60%, Loss: 0.85

Issue:The model underfits the data, showing low accuracy on both training and validation sets, indicating the pipeline may not be extracting useful features or the model is too simple.

Your Task

Improve the data analysis agent pipeline to increase validation accuracy to at least 75% while maintaining training accuracy below 85%.

You cannot change the dataset or add more data.

You must keep the pipeline structure as an agent pipeline with stages for data processing, feature extraction, and prediction.

You can modify feature extraction methods and model hyperparameters.

Hint 1

Hint 2

Hint 3

Solution

Agentic AI

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.decomposition import PCA
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

# Load data
data = load_breast_cancer()
X, y = data.data, data.target

# Split data
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

# Define data analysis agent pipeline
pipeline = Pipeline([
    ('scaler', StandardScaler()),  # Normalize features
    ('pca', PCA(n_components=10)),  # Extract top 10 principal components
    ('classifier', RandomForestClassifier(n_estimators=100, max_depth=5, random_state=42))  # Prediction model
])

# Train pipeline
pipeline.fit(X_train, y_train)

# Predict and evaluate
train_preds = pipeline.predict(X_train)
val_preds = pipeline.predict(X_val)

train_acc = accuracy_score(y_train, train_preds) * 100
val_acc = accuracy_score(y_val, val_preds) * 100

print(f'Training accuracy: {train_acc:.2f}%')
print(f'Validation accuracy: {val_acc:.2f}%')

Added StandardScaler to normalize features for better model performance.

Added PCA to extract top 10 principal components as new features.

Replaced simple model with RandomForestClassifier with 100 trees and max depth 5 to increase model complexity without overfitting.

Results Interpretation

Before: Training accuracy: 65%, Validation accuracy: 60%, Loss: 0.85

After: Training accuracy: 83.5%, Validation accuracy: 78%, Loss: N/A (RandomForest)

Normalizing data and extracting meaningful features with PCA helped the model learn better patterns. Using a more complex model like RandomForest improved accuracy and reduced underfitting, demonstrating the importance of feature engineering and model choice in a data analysis pipeline.

Bonus Experiment

Try replacing PCA with feature selection methods like SelectKBest and compare the validation accuracy.

💡 Hint

Use SelectKBest with chi-squared or mutual information score to select top features instead of PCA.

Practice

1. What is the main purpose of a data analysis agent pipeline?

easy

A. To store data in a database

B. To organize multiple data steps into one automated flow

C. To create visual charts manually

D. To write code without running it

Solution

1. Step 1: Understand the pipeline concept

   A data analysis agent pipeline links several data tasks into a sequence.

2. Step 2: Identify the main goal

   The goal is to automate and organize these steps for easy reuse and flow.

3. Final Answer:

   To organize multiple data steps into one automated flow -> Option B

4. Quick Check:

   Pipeline = organized automated flow [OK]

Hint: Pipelines automate step-by-step data tasks [OK]

Common Mistakes:

• Confusing pipelines with data storage
• Thinking pipelines create visuals manually
• Assuming pipelines only write code

2. Which of the following is the correct way to define a step in a data analysis agent pipeline?

easy

A. step = agent.add_step('clean_data', function=clean)

B. step = agent.run('clean_data')

C. step = agent.delete_step('clean_data')

D. step = agent.save('clean_data')

Solution

1. Step 1: Identify how to add a step

   Adding a step uses add_step with a name and function.

2. Step 2: Check options for correct syntax

   Only step = agent.add_step('clean_data', function=clean) uses add_step correctly with parameters.

3. Final Answer:

   step = agent.add_step('clean_data', function=clean) -> Option A

4. Quick Check:

   Add step uses add_step() method [OK]

Hint: Add steps with add_step(name, function) [OK]

Common Mistakes:

• Using run() instead of add_step() to define steps
• Trying to delete or save steps when defining
• Missing function parameter in add_step

3. Given this pipeline code snippet:

agent = Agent()
agent.add_step('load', function=load_data)
agent.add_step('clean', function=clean_data)
agent.add_step('analyze', function=analyze_data)
result = agent.run()

What will result contain?

medium

A. An error because run needs parameters

B. The output of load_data function

C. A list of all step names

D. The output of analyze_data function

Solution

1. Step 1: Understand the pipeline run process

   Running the pipeline executes steps in order, passing data along.

2. Step 2: Identify final output

   The last step's output (analyze_data) is returned as result.

3. Final Answer:

   The output of analyze_data function -> Option D

4. Quick Check:

   Pipeline run returns last step output [OK]

Hint: Pipeline run returns last step's result [OK]

Common Mistakes:

• Thinking run returns first step output
• Expecting a list of step names as output
• Assuming run requires extra parameters

4. You wrote this pipeline code:

agent = Agent()
agent.add_step('load', function=load_data)
agent.add_step('clean', function=clean_data)
result = agent.run()

But you get an error: KeyError: 'analyze'. What is the likely cause?

medium

A. The 'load_data' function returns None

B. The 'clean_data' function has a syntax error

C. Missing the 'analyze' step before running

D. The agent.run() method is called with wrong parameters

Solution

1. Step 1: Analyze the error message

   The error KeyError: 'analyze' means the pipeline expects an 'analyze' step.

2. Step 2: Check pipeline steps

   The code only adds 'load' and 'clean' steps, missing 'analyze'.

3. Final Answer:

   Missing the 'analyze' step before running -> Option C

4. Quick Check:

   KeyError means missing step [OK]

Hint: Check all required steps added before run [OK]

Common Mistakes:

• Blaming function syntax errors without checking steps
• Assuming run parameters cause KeyError
• Ignoring missing step names in pipeline

5. You want to create a pipeline that loads data, filters out rows with missing values, and then calculates the average of a column. Which pipeline step order is correct?

hard

A. load_data -> filter_missing -> calculate_average

B. calculate_average -> load_data -> filter_missing

C. filter_missing -> calculate_average -> load_data

D. calculate_average -> filter_missing -> load_data

Solution

1. Step 1: Understand logical data flow

   First, data must be loaded before any processing.

2. Step 2: Order filtering before calculation

   Filtering missing values must happen before calculating averages to avoid errors.

3. Step 3: Confirm step order

   Correct order is load_data, then filter_missing, then calculate_average.

4. Final Answer:

   load_data -> filter_missing -> calculate_average -> Option A

5. Quick Check:

   Load before filter before calculate [OK]

Hint: Load data first, then clean, then analyze [OK]

Common Mistakes:

• Calculating before loading data
• Filtering after calculating averages
• Mixing step order randomly