Basic DataFrame info (shape, dtypes, describe) in Data Analysis Python - Deep Dive

Overview - Basic DataFrame info (shape, dtypes, describe)

What is it?

A DataFrame is like a table with rows and columns used to store data. Basic DataFrame info means learning how to quickly see the size of this table, the types of data in each column, and a summary of the numbers inside. This helps you understand what your data looks like before you analyze it. It is the first step to working with data in Python.

Why it matters

Without knowing the shape, data types, or summary of your data, you might make mistakes like treating numbers as words or missing empty spots. This can lead to wrong answers or errors in your work. Basic info helps you catch problems early and plan your analysis better, saving time and effort.

Where it fits

Before this, you should know how to create or load a DataFrame in Python using libraries like pandas. After this, you will learn how to clean data, select parts of it, and perform calculations or visualizations.

Mental Model

Core Idea

Basic DataFrame info quickly tells you the size, data types, and summary statistics of your data table to understand its structure and contents.

Think of it like...

It's like checking the size of a suitcase, knowing what types of clothes are inside, and getting a quick idea of how many shirts, pants, or socks you packed before you start your trip.

┌───────────────┐
│   DataFrame   │
├───────────────┤
│ shape: (rows, columns) │
│ dtypes: column types   │
│ describe: summary stats│
└───────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding DataFrame shape

Concept: Learn how to find the number of rows and columns in a DataFrame.

In pandas, every DataFrame has a 'shape' attribute that shows its size as (rows, columns). For example, df.shape returns a tuple like (100, 5), meaning 100 rows and 5 columns.

Result

You get a tuple showing how many rows and columns your data has.

Knowing the shape helps you understand the scale of your data and whether it matches your expectations.

2

Foundation

Identifying column data types

Concept: Discover how to check what kind of data each column holds.

Each column in a DataFrame has a data type, like numbers (int, float) or text (object). Using df.dtypes shows the type of each column, helping you know how to handle the data.

Result

You see a list of columns with their data types, like int64 or object.

Knowing data types prevents errors, like trying to do math on text or ignoring missing values.

3

Intermediate

Using describe() for summary stats

🤔Before reading on: do you think describe() shows all columns or only numbers? Commit to your answer.

Concept: Learn how to get quick statistics like mean, min, max for numeric columns.

The describe() method gives a summary of numeric columns by default, showing count, mean, std (spread), min, max, and quartiles. For example, df.describe() returns a table with these stats for each numeric column.

Result

You get a table summarizing key statistics for numeric data.

Summary stats give a quick sense of data distribution and spot unusual values or errors.

4

Intermediate

Describe() with non-numeric data

🤔Before reading on: does describe() work only on numbers or also on text? Commit to your answer.

Concept: Understand how describe() behaves with text or categorical columns.

By default, describe() skips text columns, but you can include them by using df.describe(include='all'). This shows counts, unique values, top (most common) value, and frequency for text columns.

Result

You get a summary table including text columns with counts and common values.

Including all columns in describe() helps understand categorical data and spot data quality issues.

5

Intermediate

Combining shape, dtypes, and describe()

Concept: Learn how to use these three tools together to get a full picture of your data.

Start by checking df.shape to know size, then df.dtypes to know data types, and finally df.describe(include='all') to get summaries of all columns. This sequence gives a quick but deep understanding of your dataset.

Result

You have a clear overview of your data's size, types, and content summaries.

Using these methods together is a powerful first step in any data analysis workflow.

6

Advanced

Handling missing data in describe()

🤔Before reading on: does describe() count missing values or ignore them? Commit to your answer.

Concept: Understand how missing data affects summary statistics and how describe() handles it.

Describe() counts only non-missing values in its 'count' row. Missing values are ignored in calculations like mean or std. Knowing this helps you realize that missing data can hide problems if you only look at describe().

Result

You see counts less than total rows if missing data exists, alerting you to incomplete data.

Recognizing missing data impact prevents wrong conclusions from incomplete summaries.

7

Expert

Customizing describe() for deeper insights

🤔Before reading on: can you customize describe() to show percentiles or specific stats? Commit to your answer.

Concept: Learn how to adjust describe() parameters to get tailored summaries.

You can pass parameters like percentiles=[0.1, 0.9] to describe() to see custom percentile values. Also, you can select specific data types or exclude some. This flexibility helps experts focus on relevant statistics for their analysis.

Result

You get a customized summary table that fits your specific needs.

Customizing describe() unlocks deeper understanding and better data-driven decisions.

Under the Hood

When you call shape, pandas returns the stored tuple of row and column counts without scanning data. dtypes inspects each column's underlying data type stored in memory. Describe() computes statistics by iterating over columns, applying functions like mean or count only on non-missing values, and assembling results into a summary DataFrame.

Why designed this way?

These methods are designed for speed and convenience. Shape is a simple attribute for quick size checks. dtypes reflect how data is stored internally for efficient operations. Describe() balances detail and speed by defaulting to numeric summaries but allows customization for flexibility.

┌───────────────┐
│   DataFrame   │
├───────────────┤
│ shape ────────┤─> (rows, columns)
│ dtypes ───────┤─> column: data type
│ describe() ───┤─> summary stats table
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does df.describe() include text columns by default? Commit yes or no.

Common Belief: Describe() shows summary stats for all columns, including text.

Reality: By default, describe() only summarizes numeric columns unless you specify include='all'.

Why it matters: Assuming text columns are summarized can cause you to miss important info about categorical data.

Quick: Does df.shape count missing rows? Commit yes or no.

Common Belief: Shape excludes rows with missing data.

Reality: Shape counts all rows, including those with missing values.

Why it matters: Misunderstanding this leads to wrong assumptions about data completeness.

Quick: Does df.dtypes convert data types automatically? Commit yes or no.

Common Belief: dtypes changes or fixes data types to the correct ones.

Reality: dtypes only shows data types; it does not change them.

Why it matters: Expecting automatic fixes can cause unnoticed errors in data processing.

Quick: Does describe() count missing values in its statistics? Commit yes or no.

Common Belief: Describe includes missing values in counts and calculations.

Reality: Describe ignores missing values in calculations and counts only non-missing entries.

Why it matters: Ignoring missing data can hide data quality problems and bias analysis.

Expert Zone

1

Describe() uses optimized Cython code internally for fast computation even on large datasets.

2

Data types shown by dtypes can be more specific than Python types, like int64 vs int32, affecting memory and speed.

3

Shape is a quick attribute but does not reflect filtered or view-based DataFrames unless explicitly checked.

When NOT to use

For very large datasets that don't fit in memory, these methods can be slow or impossible. Instead, use chunked reading or specialized tools like Dask or databases for summary info.

Production Patterns

In real-world pipelines, shape, dtypes, and describe() are used in automated data validation steps to catch schema changes or data drift before analysis or model training.

Connections

Database schema inspection

Similar pattern of checking table size and column types before queries

Understanding DataFrame info helps grasp how databases expose schema info for safe querying.

Exploratory Data Analysis (EDA)

Builds on basic info to deeper data understanding and visualization

Mastering basic info is the foundation for effective EDA workflows.

Inventory management

Both involve knowing quantity (shape) and types (dtypes) of items before decisions

Seeing data like inventory helps appreciate why size and type info is critical for planning.

Common Pitfalls

#1 Ignoring data types and assuming all columns are numeric.

Wrong approach:

df['column'] + 10 # fails if column is text

Correct approach:

df['column'] = pd.to_numeric(df['column'], errors='coerce')
df['column'] + 10

Root cause:

Not checking dtypes leads to errors when performing numeric operations on text data.

#2 Using describe() without including all columns and missing important categorical info.

Wrong approach:

df.describe() # misses text columns

Correct approach:

df.describe(include='all') # includes all columns

Root cause:

Assuming describe() covers all data types by default hides important summaries.

#3 Assuming shape changes after filtering without re-checking.

Wrong approach:

filtered = df[df['age'] > 30]
print(df.shape) # still original shape

Correct approach:

filtered = df[df['age'] > 30]
print(filtered.shape) # correct filtered shape

Root cause:

Confusing original and filtered DataFrames causes wrong assumptions about data size.

Key Takeaways

Basic DataFrame info methods like shape, dtypes, and describe() give a quick but powerful overview of your data.

Shape tells you how big your data is, dtypes tell you what kind of data each column holds, and describe() summarizes the data's key statistics.

Always check data types before performing operations to avoid errors and misunderstandings.

Describe() by default summarizes numeric data but can be customized to include all columns for a fuller picture.

Understanding these basics is essential for safe, effective data analysis and prevents common mistakes.