SciPy with Pandas for data handling - Practice Problems & Coding Challenges

Challenge - 5 Problems

❓ Predict Output

intermediate

Output of scipy.stats.ttest_ind with pandas DataFrames

Given two pandas DataFrames representing two groups of data, what is the output of the following code snippet?

SciPy

import pandas as pd
from scipy import stats

# Create two groups of data
data1 = pd.DataFrame({'score': [88, 92, 85, 90, 87]})
data2 = pd.DataFrame({'score': [78, 81, 79, 83, 80]})

# Perform independent t-test
result = stats.ttest_ind(data1['score'], data2['score'])
print((round(result.statistic, 2), round(result.pvalue, 3)))

A (-6.32, 0.000)

B (6.32, 0.000)

C (-1.23, 0.234)

D (1.23, 0.234)

💡 Hint

Remember that the t-statistic sign depends on which group mean is larger.

The first group has a higher mean score than the second, so the t-statistic is positive. The p-value is very small, indicating a significant difference.

❓ data_output

intermediate

Result of scipy.stats.linregress on pandas Series

What is the output of the linear regression performed on the following pandas Series using scipy.stats.linregress?

SciPy

import pandas as pd
from scipy import stats

x = pd.Series([1, 2, 3, 4, 5])
y = pd.Series([2, 4, 5, 4, 5])

result = stats.linregress(x, y)
print((round(result.slope, 2), round(result.intercept, 2)))

A (0.6, 2.2)

B (0.7, 1.6)

C (0.8, 1.4)

D (0.5, 2.5)

💡 Hint

Slope is the change in y per unit change in x.

The slope is approximately 0.7 and intercept about 1.6, fitting the data points best in a linear model.

❓ visualization

advanced

Visualizing correlation matrix with SciPy and Pandas

You have a pandas DataFrame with multiple numeric columns. You calculate the Pearson correlation matrix using SciPy. Which code snippet correctly produces a heatmap visualization of this correlation matrix using matplotlib?

SciPy

import pandas as pd
import numpy as np
from scipy.stats import pearsonr
import matplotlib.pyplot as plt

# Sample data
np.random.seed(0)
data = pd.DataFrame(np.random.rand(5, 4), columns=list('ABCD'))

# Calculate correlation matrix
corr = data.corr()

# Visualization code here

A

plt.bar(corr.columns, corr.iloc[0])
plt.show()

B

plt.plot(corr)
plt.show()

C

plt.imshow(corr, cmap='coolwarm', interpolation='none')
plt.colorbar()
plt.xticks(range(len(corr)), corr.columns)
plt.yticks(range(len(corr)), corr.columns)
plt.show()

D

plt.scatter(corr.columns, corr.columns)
plt.show()

💡 Hint

A heatmap uses imshow with a color map to show matrix values.

Option C correctly uses imshow to display the correlation matrix as a heatmap with colorbar and axis labels.

🔧 Debug

advanced

Identify the error in applying scipy.stats.kstest on pandas Series

What error will the following code raise when running the Kolmogorov-Smirnov test on a pandas Series?

SciPy

import pandas as pd
from scipy import stats

sample = pd.Series([0.1, 0.4, 0.35, 0.8, 0.9])

result = stats.kstest(sample, 'uniform')
print(result)

A TypeError: 'Series' object is not iterable

B AttributeError: 'Series' object has no attribute 'shape'

C ValueError: Data must be 1-dimensional array-like

D No error, outputs KstestResult(statistic=..., pvalue=...)

💡 Hint

Check if scipy.stats.kstest accepts pandas Series directly.

scipy.stats.kstest accepts array-like inputs including pandas Series, so it runs without error and returns the test result.

🚀 Application

expert

Using SciPy and Pandas to find the most correlated feature

Given a pandas DataFrame with numeric columns, which code snippet correctly finds the column most positively correlated with column 'target' using SciPy's pearsonr function?

SciPy

import pandas as pd
from scipy.stats import pearsonr

data = pd.DataFrame({
    'target': [1, 2, 3, 4, 5],
    'feat1': [5, 4, 3, 2, 1],
    'feat2': [2, 3, 4, 5, 6],
    'feat3': [5, 5, 5, 5, 5]
})

# Find most correlated feature with 'target'

A

corrs = {col: pearsonr(data['target'], data[col])[0] for col in data.columns if col != 'target'}
most_corr = max(corrs, key=corrs.get)
print(most_corr)

B

corrs = data.corr()['target'].drop('target')
most_corr = corrs.idxmax()
print(most_corr)

C

most_corr = data.corrwith(data['target']).idxmax()
print(most_corr)

D

most_corr = max(data.columns, key=lambda col: pearsonr(data['target'], data[col])[1])
print(most_corr)

💡 Hint

Use pearsonr to get correlation coefficient, not p-value.

Option A correctly uses pearsonr to compute correlation coefficients and finds the column with the highest positive correlation.

Practice

1. What is the main reason to use SciPy together with Pandas in data analysis?

easy

A. SciPy provides advanced math and stats functions, while Pandas organizes data in tables.

B. Pandas is used only for visualization, SciPy handles all data storage.

C. SciPy replaces Pandas for data cleaning tasks.

D. Pandas is used to write code, SciPy runs the code faster.

Solution

1. Step 1: Understand roles of Pandas and SciPy

   Pandas organizes data into tables called DataFrames, making it easy to handle data.

2. Step 2: Identify SciPy's role

   SciPy offers math and statistics tools to analyze data prepared by Pandas.

3. Final Answer:

   SciPy provides advanced math and stats functions, while Pandas organizes data in tables. -> Option A

4. Quick Check:

   Data organization = Pandas, Analysis = SciPy [OK]

Hint: Remember: Pandas for tables, SciPy for math [OK]

Common Mistakes:

• Thinking Pandas does advanced stats alone
• Confusing SciPy as a data storage tool
• Believing SciPy replaces Pandas for cleaning

2. Which of the following is the correct way to import SciPy's stats module and Pandas in Python?

easy

A. from scipy import stats; import pandas as pd

B. import scipy.stats; import pandas as pandas

C. from scipy.stats import stats; import pandas as pd

D. import scipy.stats as sp; import pandas as pd

Solution

1. Step 1: Check common import styles

   Using 'from scipy import stats' imports the stats module directly, which is common and clear.

2. Step 2: Verify Pandas import

   Importing pandas as 'pd' is the standard alias used in data science.

3. Final Answer:

   from scipy import stats; import pandas as pd -> Option A

4. Quick Check:

   Standard imports = from scipy import stats, import pandas as pd [OK]

Hint: Use 'from scipy import stats' and 'import pandas as pd' [OK]

Common Mistakes:

• Using wrong alias for pandas
• Importing scipy.stats without alias or direct import
• Mixing import styles incorrectly

3. Given the code below, what will be the output?

import pandas as pd
from scipy import stats

data = {'score': [10, 20, 20, 30, 40]}
df = pd.DataFrame(data)
mode_result = stats.mode(df['score'])
print(mode_result.mode[0])

medium

A. 10

B. 30

C. 20

D. 40

Solution

1. Step 1: Understand the data

   The 'score' column has values [10, 20, 20, 30, 40]. The number 20 appears twice, others once.

2. Step 2: Apply stats.mode

   stats.mode finds the most frequent value, which is 20 here.

3. Final Answer:

   20 -> Option C

4. Quick Check:

   Most frequent value = 20 [OK]

Hint: Mode is the most frequent value in the list [OK]

Common Mistakes:

• Choosing the first value instead of mode
• Confusing mean or median with mode
• Not accessing .mode[0] correctly

4. Identify the error in the following code snippet:

import pandas as pd
from scipy import stats

data = {'values': [1, 2, 3, 4, 5]}
df = pd.DataFrame(data)
result = stats.mean(df['values'])
print(result)

medium

A. DataFrame creation syntax is incorrect.

B. stats.mean does not exist; use numpy.mean or pandas mean method instead.

C. The print statement is missing parentheses.

D. The import statement for pandas is wrong.

Solution

1. Step 1: Check function availability in SciPy

   SciPy's stats module does not have a 'mean' function; mean is in numpy or pandas.

2. Step 2: Identify correct function usage

   Use df['values'].mean() or numpy.mean(df['values']) instead.

3. Final Answer:

   stats.mean does not exist; use numpy.mean or pandas mean method instead. -> Option B

4. Quick Check:

   stats.mean missing, use pandas or numpy mean [OK]

Hint: Use pandas or numpy for mean, not stats.mean [OK]

Common Mistakes:

• Assuming all stats functions exist in SciPy
• Ignoring error messages about missing attributes
• Confusing pandas and SciPy function locations

5. You have a Pandas DataFrame with a column 'height' containing some missing values (NaN). You want to fill these missing values with the median height calculated using SciPy. Which code snippet correctly does this?

hard

A. from scipy import stats
median_height = stats.median(df['height'])
df['height'] = df['height'].fillna(median_height)

B. from scipy import stats
median_height = stats.median(df['height'].dropna())
df['height'] = df['height'].fillna(median_height)

C. from scipy import stats
median_height = stats.mode(df['height'].dropna()).mode[0]
df['height'] = df['height'].fillna(median_height)

D. from scipy import stats
median_height = stats.scoreatpercentile(df['height'].dropna(), 50)
df['height'] = df['height'].fillna(median_height)

Solution

1. Step 1: Identify correct SciPy function for median

   SciPy's stats module does not have 'median', but 'scoreatpercentile' can find the 50th percentile (median).

2. Step 2: Handle missing values correctly

   Drop NaN values before calculating median, then fill NaNs with this median.

3. Final Answer:

   from scipy import stats median_height = stats.scoreatpercentile(df['height'].dropna(), 50) df['height'] = df['height'].fillna(median_height) -> Option D

4. Quick Check:

   Median via scoreatpercentile, fillna with median [OK]

Hint: Use scoreatpercentile for median in SciPy [OK]

Common Mistakes:

• Using stats.median which does not exist
• Not dropping NaN before median calculation
• Using mode instead of median