Basic DataFrame info (shape, dtypes, describe) in Data Analysis Python - Deep Dive

Overview - Basic DataFrame info (shape, dtypes, describe)

What is it?

A DataFrame is like a table with rows and columns used to store data. Basic DataFrame info means learning how to quickly see the size of this table, the types of data in each column, and a summary of the numbers inside. This helps you understand what your data looks like before you analyze it. It is the first step to working with data in Python.

Why it matters

Without knowing the shape, data types, or summary of your data, you might make mistakes like treating numbers as words or missing empty spots. This can lead to wrong answers or errors in your work. Basic info helps you catch problems early and plan your analysis better, saving time and effort.

Where it fits

Before this, you should know how to create or load a DataFrame in Python using libraries like pandas. After this, you will learn how to clean data, select parts of it, and perform calculations or visualizations.

Mental Model

Core Idea

Basic DataFrame info quickly tells you the size, data types, and summary statistics of your data table to understand its structure and contents.

Think of it like...

It's like checking the size of a suitcase, knowing what types of clothes are inside, and getting a quick idea of how many shirts, pants, or socks you packed before you start your trip.

┌───────────────┐
│   DataFrame   │
├───────────────┤
│ shape: (rows, columns) │
│ dtypes: column types   │
│ describe: summary stats│
└───────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding DataFrame shape

Concept: Learn how to find the number of rows and columns in a DataFrame.

In pandas, every DataFrame has a 'shape' attribute that shows its size as (rows, columns). For example, df.shape returns a tuple like (100, 5), meaning 100 rows and 5 columns.

Result

You get a tuple showing how many rows and columns your data has.

Knowing the shape helps you understand the scale of your data and whether it matches your expectations.

2

Foundation

Identifying column data types

Concept: Discover how to check what kind of data each column holds.

Each column in a DataFrame has a data type, like numbers (int, float) or text (object). Using df.dtypes shows the type of each column, helping you know how to handle the data.

Result

You see a list of columns with their data types, like int64 or object.

Knowing data types prevents errors, like trying to do math on text or ignoring missing values.

3

Intermediate

Using describe() for summary stats

🤔Before reading on: do you think describe() shows all columns or only numbers? Commit to your answer.

Concept: Learn how to get quick statistics like mean, min, max for numeric columns.

The describe() method gives a summary of numeric columns by default, showing count, mean, std (spread), min, max, and quartiles. For example, df.describe() returns a table with these stats for each numeric column.

Result

You get a table summarizing key statistics for numeric data.

Summary stats give a quick sense of data distribution and spot unusual values or errors.

4

Intermediate

Describe() with non-numeric data

🤔Before reading on: does describe() work only on numbers or also on text? Commit to your answer.

Concept: Understand how describe() behaves with text or categorical columns.

By default, describe() skips text columns, but you can include them by using df.describe(include='all'). This shows counts, unique values, top (most common) value, and frequency for text columns.

Result

You get a summary table including text columns with counts and common values.

Including all columns in describe() helps understand categorical data and spot data quality issues.

5

Intermediate

Combining shape, dtypes, and describe()

Concept: Learn how to use these three tools together to get a full picture of your data.

Start by checking df.shape to know size, then df.dtypes to know data types, and finally df.describe(include='all') to get summaries of all columns. This sequence gives a quick but deep understanding of your dataset.

Result

You have a clear overview of your data's size, types, and content summaries.

Using these methods together is a powerful first step in any data analysis workflow.

6

Advanced

Handling missing data in describe()

🤔Before reading on: does describe() count missing values or ignore them? Commit to your answer.

Concept: Understand how missing data affects summary statistics and how describe() handles it.

Describe() counts only non-missing values in its 'count' row. Missing values are ignored in calculations like mean or std. Knowing this helps you realize that missing data can hide problems if you only look at describe().

Result

You see counts less than total rows if missing data exists, alerting you to incomplete data.

Recognizing missing data impact prevents wrong conclusions from incomplete summaries.

7

Expert

Customizing describe() for deeper insights

🤔Before reading on: can you customize describe() to show percentiles or specific stats? Commit to your answer.

Concept: Learn how to adjust describe() parameters to get tailored summaries.

You can pass parameters like percentiles=[0.1, 0.9] to describe() to see custom percentile values. Also, you can select specific data types or exclude some. This flexibility helps experts focus on relevant statistics for their analysis.

Result

You get a customized summary table that fits your specific needs.

Customizing describe() unlocks deeper understanding and better data-driven decisions.

Under the Hood

When you call shape, pandas returns the stored tuple of row and column counts without scanning data. dtypes inspects each column's underlying data type stored in memory. Describe() computes statistics by iterating over columns, applying functions like mean or count only on non-missing values, and assembling results into a summary DataFrame.

Why designed this way?

These methods are designed for speed and convenience. Shape is a simple attribute for quick size checks. dtypes reflect how data is stored internally for efficient operations. Describe() balances detail and speed by defaulting to numeric summaries but allows customization for flexibility.

┌───────────────┐
│   DataFrame   │
├───────────────┤
│ shape ────────┤─> (rows, columns)
│ dtypes ───────┤─> column: data type
│ describe() ───┤─> summary stats table
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does df.describe() include text columns by default? Commit yes or no.

Common Belief: Describe() shows summary stats for all columns, including text.

Reality: By default, describe() only summarizes numeric columns unless you specify include='all'.

Why it matters: Assuming text columns are summarized can cause you to miss important info about categorical data.

Quick: Does df.shape count missing rows? Commit yes or no.

Common Belief: Shape excludes rows with missing data.

Reality: Shape counts all rows, including those with missing values.

Why it matters: Misunderstanding this leads to wrong assumptions about data completeness.

Quick: Does df.dtypes convert data types automatically? Commit yes or no.

Common Belief: dtypes changes or fixes data types to the correct ones.

Reality: dtypes only shows data types; it does not change them.

Why it matters: Expecting automatic fixes can cause unnoticed errors in data processing.

Quick: Does describe() count missing values in its statistics? Commit yes or no.

Common Belief: Describe includes missing values in counts and calculations.

Reality: Describe ignores missing values in calculations and counts only non-missing entries.

Why it matters: Ignoring missing data can hide data quality problems and bias analysis.

Expert Zone

1

Describe() uses optimized Cython code internally for fast computation even on large datasets.

2

Data types shown by dtypes can be more specific than Python types, like int64 vs int32, affecting memory and speed.

3

Shape is a quick attribute but does not reflect filtered or view-based DataFrames unless explicitly checked.

When NOT to use

For very large datasets that don't fit in memory, these methods can be slow or impossible. Instead, use chunked reading or specialized tools like Dask or databases for summary info.

Production Patterns

In real-world pipelines, shape, dtypes, and describe() are used in automated data validation steps to catch schema changes or data drift before analysis or model training.

Connections

Database schema inspection

Similar pattern of checking table size and column types before queries

Understanding DataFrame info helps grasp how databases expose schema info for safe querying.

Exploratory Data Analysis (EDA)

Builds on basic info to deeper data understanding and visualization

Mastering basic info is the foundation for effective EDA workflows.

Inventory management

Both involve knowing quantity (shape) and types (dtypes) of items before decisions

Seeing data like inventory helps appreciate why size and type info is critical for planning.

Common Pitfalls

#1 Ignoring data types and assuming all columns are numeric.

Wrong approach:

df['column'] + 10 # fails if column is text

Correct approach:

df['column'] = pd.to_numeric(df['column'], errors='coerce')
df['column'] + 10

Root cause:

Not checking dtypes leads to errors when performing numeric operations on text data.

#2 Using describe() without including all columns and missing important categorical info.

Wrong approach:

df.describe() # misses text columns

Correct approach:

df.describe(include='all') # includes all columns

Root cause:

Assuming describe() covers all data types by default hides important summaries.

#3 Assuming shape changes after filtering without re-checking.

Wrong approach:

filtered = df[df['age'] > 30]
print(df.shape) # still original shape

Correct approach:

filtered = df[df['age'] > 30]
print(filtered.shape) # correct filtered shape

Root cause:

Confusing original and filtered DataFrames causes wrong assumptions about data size.

Key Takeaways

Basic DataFrame info methods like shape, dtypes, and describe() give a quick but powerful overview of your data.

Shape tells you how big your data is, dtypes tell you what kind of data each column holds, and describe() summarizes the data's key statistics.

Always check data types before performing operations to avoid errors and misunderstandings.

Describe() by default summarizes numeric data but can be customized to include all columns for a fuller picture.

Understanding these basics is essential for safe, effective data analysis and prevents common mistakes.

Practice

1. What does df.shape tell you about a DataFrame?

easy

A. The data types of each column

B. The number of rows and columns in the DataFrame

C. Summary statistics like mean and median

D. The first few rows of the DataFrame

Solution

1. Step 1: Understand what df.shape returns

   df.shape gives a tuple with two numbers: rows and columns count.

2. Step 2: Match the description to the options

   The number of rows and columns in the DataFrame correctly describes this as the number of rows and columns.

3. Final Answer:

   The number of rows and columns in the DataFrame -> Option B

4. Quick Check:

   Shape = (rows, columns) [OK]

Hint: Remember shape shows size: rows and columns [OK]

Common Mistakes:

• Confusing shape with data types
• Thinking shape shows data summary
• Mixing shape with head() output

2. Which of the following is the correct way to check the data types of columns in a DataFrame named df?

easy

A. df.dtypes

B. df.columns.types

C. df.describe()

D. df.types()

Solution

1. Step 1: Recall the attribute for column data types

   The correct attribute to check data types is df.dtypes, not a method.

2. Step 2: Verify the options

   df.dtypes uses df.dtypes correctly. Others are invalid or do not exist.

3. Final Answer:

   df.dtypes -> Option A

4. Quick Check:

   dtypes attribute shows column types [OK]

Hint: Use df.dtypes without parentheses to check types [OK]

Common Mistakes:

• Adding parentheses like df.dtypes()
• Using non-existent methods like df.types()
• Confusing describe() with dtypes

3. Given the DataFrame df below:

import pandas as pd
data = {'A': [1, 2, 3], 'B': [4.0, 5.5, 6.1], 'C': ['x', 'y', 'z']}
df = pd.DataFrame(data)
print(df.describe())

What will be the output of df.describe()?

medium

A. Summary statistics for all columns including C

B. An error because column C is non-numeric

C. Summary statistics for columns A and B only, excluding C

D. Only the count of rows

Solution

1. Step 1: Understand what df.describe() does

   It provides summary stats (count, mean, std, min, max, quartiles) for numeric columns only by default.

2. Step 2: Check the DataFrame columns

   Columns A and B are numeric (int and float), C is string (non-numeric), so C is excluded.

3. Final Answer:

   Summary statistics for columns A and B only, excluding C -> Option C

4. Quick Check:

   describe() skips non-numeric columns by default [OK]

Hint: describe() shows stats only for numeric columns by default [OK]

Common Mistakes:

• Expecting describe() to include string columns
• Thinking describe() throws error on strings
• Assuming describe() shows only count

4. You run the code below and get an error:

import pandas as pd
data = {'X': [10, 20, 30], 'Y': [5, 15, 25]}
df = pd.DataFrame(data)
print(df.dtypes())

What is the cause of the error?

medium

A. dtypes is an attribute, not a method, so parentheses cause error

B. DataFrame columns have mixed types

C. DataFrame is empty

D. Missing import statement for pandas

Solution

1. Step 1: Identify the error source

   The code calls df.dtypes() with parentheses, but dtypes is an attribute, not a method.

2. Step 2: Understand correct usage

   Correct usage is df.dtypes without parentheses to access data types.

3. Final Answer:

   dtypes is an attribute, not a method, so parentheses cause error -> Option A

4. Quick Check:

   Use df.dtypes without () [OK]

Hint: Remember dtypes is attribute, no parentheses [OK]

Common Mistakes:

• Adding parentheses to dtypes
• Assuming error is from missing import
• Thinking DataFrame is empty

5. You have a DataFrame df with columns: Age (int), Salary (float), and Department (string). You want to quickly check how many rows and columns it has, the data types of each column, and summary statistics for numeric columns. Which sequence of commands gives you all this information?

hard

A. df.columns; df.dtypes; df.tail()

B. df.describe(); df.shape; df.columns

C. df.info(); df.head(); df.describe()

D. df.shape; df.dtypes; df.describe()

Solution

1. Step 1: Identify commands for each info type

   df.shape gives rows and columns count, df.dtypes shows data types, and df.describe() gives summary stats for numeric columns.

2. Step 2: Check options for correct sequence

   df.shape; df.dtypes; df.describe() lists these three commands in order, so it provides all required info.

3. Final Answer:

   df.shape; df.dtypes; df.describe() -> Option D

4. Quick Check:

   shape, dtypes, describe cover size, types, stats [OK]

Hint: Use shape, dtypes, then describe for full quick info [OK]

Common Mistakes:

• Using df.columns instead of df.dtypes for types
• Confusing df.info() with df.describe()
• Missing shape to check size