Pandas and NumPy connection - Step-by-Step Execution

Concept Flow - Pandas and NumPy connection

Create NumPy array

↓

Use array to create Pandas DataFrame

↓

Perform DataFrame operations

↓

Access underlying NumPy array with .values or .to_numpy()

↓

Use NumPy functions on DataFrame data

↓

Result: Combined power of Pandas and NumPy

This flow shows how we start with NumPy arrays, create Pandas DataFrames, and then switch between them to use both libraries' features.

Execution Sample

Pandas

import numpy as np
import pandas as pd

arr = np.array([[1, 2], [3, 4]])
df = pd.DataFrame(arr, columns=['A', 'B'])
print(df.to_numpy())

Create a NumPy array, convert it to a Pandas DataFrame, then convert back to a NumPy array and print it.

Execution Table

Step	Action	Variable	Value/Result	Notes
1	Create NumPy array	arr	[[1 2] [3 4]]	2x2 integer array
2	Create DataFrame from arr	df	A B 0 1 2 1 3 4	Columns named 'A' and 'B'
3	Convert DataFrame to NumPy array	df.to_numpy()	[[1 2] [3 4]]	Same data as original arr
4	Print NumPy array from DataFrame	Output	[[1 2] [3 4]]	Printed to console
5	Exit			End of code execution

💡 Code ends after printing the NumPy array converted from the DataFrame.

Variable Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	Final
arr	undefined	[[1 2] [3 4]]	[[1 2] [3 4]]	[[1 2] [3 4]]	[[1 2] [3 4]]
df	undefined	undefined	A B 0 1 2 1 3 4	A B 0 1 2 1 3 4	A B 0 1 2 1 3 4

Key Moments - 2 Insights

Why does converting a DataFrame back to a NumPy array keep the same data?

Because Pandas DataFrames store data internally as NumPy arrays, so .to_numpy() just accesses that data without changing it (see execution_table step 3).

Can we use NumPy functions directly on a Pandas DataFrame?

Yes, but it's often better to convert the DataFrame to a NumPy array first using .to_numpy() to ensure compatibility (see concept_flow and execution_table steps 3 and 4).

Visual Quiz - 3 Questions

Look at the execution_table, what is the value of 'df' after step 2?

A An empty DataFrame

B A 2x2 DataFrame with columns 'A' and 'B' containing [[1, 2], [3, 4]]

C A NumPy array [[1, 2], [3, 4]]

D A list of lists [[1, 2], [3, 4]]

💡 Hint

Check the 'Value/Result' column for step 2 in the execution_table.

At which step does the code print the NumPy array converted from the DataFrame?

A Step 4

B Step 3

C Step 1

D Step 5

💡 Hint

Look for the step where 'Print NumPy array from DataFrame' happens in execution_table.

If we changed the original NumPy array to have shape (3,2), how would the DataFrame 'df' change after step 2?

A It would have 2 rows and 3 columns

B It would cause an error

C It would have 3 rows and 2 columns named 'A' and 'B'

D It would remain the same as before

💡 Hint

DataFrame shape matches the NumPy array shape used to create it (see concept_flow).

Concept Snapshot

Pandas and NumPy connection:
- Create NumPy arrays with np.array()
- Make DataFrames from arrays: pd.DataFrame(array)
- Access DataFrame data as NumPy array: df.to_numpy()
- Use NumPy functions on DataFrame data via .to_numpy()
- DataFrames store data internally as NumPy arrays

Full Transcript

This visual execution shows how to connect Pandas and NumPy. First, we create a NumPy array named 'arr' with numbers. Then, we use this array to make a Pandas DataFrame called 'df' with columns named 'A' and 'B'. Next, we convert the DataFrame back to a NumPy array using the .to_numpy() method. Finally, we print this NumPy array. The key idea is that Pandas DataFrames store data as NumPy arrays internally, so converting back and forth keeps the data the same. This lets us use the strengths of both libraries together easily.

Practice

1. What is the main reason Pandas uses NumPy arrays internally?

easy

A. To handle text data only

B. To make data visualization easier

C. To store data efficiently and allow fast computations

D. To create interactive dashboards

Solution

1. Step 1: Understand Pandas data storage

   Pandas stores data in tables called DataFrames, which need efficient storage for speed.

2. Step 2: Role of NumPy arrays

   NumPy arrays provide fast, memory-efficient storage and operations on numerical data.

3. Final Answer:

   To store data efficiently and allow fast computations -> Option C

4. Quick Check:

   Pandas uses NumPy arrays for efficiency [OK]

Hint: Pandas uses NumPy arrays for speed and memory [OK]

Common Mistakes:

• Thinking Pandas uses NumPy only for visualization
• Believing Pandas stores data as plain Python lists
• Confusing NumPy arrays with text handling

2. Which of the following is the recommended way to convert a Pandas DataFrame df to a NumPy array?

easy

A. np.array(df)

B. df.to_numpy()

C. df.numpy()

D. df.asarray()

Solution

1. Step 1: Recall Pandas method for conversion

   Pandas DataFrames have a method to_numpy() to convert to NumPy arrays.

2. Step 2: Check other options

   np.array(df) works but is not the dedicated Pandas method; df.numpy() and df.asarray() do not exist.

3. Final Answer:

   df.to_numpy() -> Option B

4. Quick Check:

   Use to_numpy() to convert DataFrame [OK]

Hint: Use df.to_numpy() to get NumPy array from DataFrame [OK]

Common Mistakes:

• Using non-existent methods like df.numpy()
• Confusing np.array(df) as the best method
• Trying to use df.asarray() which is invalid

3. Given the code:

import pandas as pd
import numpy as np
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
result = np.sum(df)

What is the value of result?

medium

A. array([6, 15])

B. [6 15]

C. 21

D. Error: cannot sum DataFrame with NumPy

Solution

1. Step 1: Understand np.sum on DataFrame

   Applying np.sum to a DataFrame sums each column and returns a NumPy array.

2. Step 2: Calculate sums of columns

   Column 'A' sums to 1+2+3=6, column 'B' sums to 4+5+6=15, so result is array([6, 15]).

3. Final Answer:

   array([6, 15]) -> Option A

4. Quick Check:

   NumPy sums columns into array [OK]

Hint: np.sum on DataFrame sums columns, returns NumPy array [OK]

Common Mistakes:

• Expecting a single number sum of all values
• Thinking np.sum causes error on DataFrame
• Confusing list output with NumPy array

4. What is wrong with this code snippet?

import pandas as pd
import numpy as np
df = pd.DataFrame({'X': [1, 2], 'Y': [3, 4]})
arr = df.values()
print(arr + 1)

medium

A. df.values() should be df.values without parentheses

B. You cannot add 1 to a NumPy array

C. DataFrame does not have a values attribute

D. The print statement syntax is incorrect

Solution

1. Step 1: Check usage of df.values

   df.values is an attribute, not a method, so parentheses cause an error.

2. Step 2: Understand addition to NumPy array

   Adding 1 to a NumPy array works element-wise, so no error there.

3. Final Answer:

   df.values() should be df.values without parentheses -> Option A

4. Quick Check:

   Use df.values without () to get NumPy array [OK]

Hint: df.values is attribute, not method; no parentheses [OK]

Common Mistakes:

• Calling df.values() as a method
• Thinking addition to array is invalid
• Assuming print syntax is wrong in Python 3

5. You have a Pandas DataFrame df with numeric columns. You want to apply the NumPy function np.log to all values and get a new DataFrame with the results preserving the original index and column names. Which code correctly does this?

hard

A. new_df = pd.DataFrame(np.log(df))

B. new_df = df.to_numpy().log()

C. new_df = np.log(df)

D. new_df = df.apply(np.log)

Solution

1. Step 1: Understand applying NumPy functions on DataFrame

   Using df.apply(np.log) applies np.log element-wise to each column and returns a DataFrame preserving index and columns.

2. Step 2: Check other options

   np.log(df) returns a NumPy array, not DataFrame; pd.DataFrame(np.log(df)) creates DataFrame but loses original index and column names; df.to_numpy().log() is invalid syntax.

3. Final Answer:

   new_df = df.apply(np.log) -> Option D

4. Quick Check:

   Use df.apply(np.log) for DataFrame output [OK]

Hint: Use df.apply(np.log) to keep DataFrame structure [OK]

Common Mistakes:

• Using np.log(df) which returns array, not DataFrame
• Trying df.to_numpy().log() which is invalid
• Wrapping np.log(df) inside pd.DataFrame unnecessarily