Selecting rows (loc, iloc) in Data Analysis Python - Deep Dive

Overview - Selecting rows (loc, iloc)

What is it?

Selecting rows in data analysis means choosing specific rows from a table of data. In Python's pandas library, two common ways to do this are loc and iloc. loc selects rows by their labels or names, while iloc selects rows by their position or number. This helps you focus on the data you want to analyze or change.

Why it matters

Without ways to select rows easily, working with large tables would be slow and confusing. You might have to look at all data even if you need just a few rows. loc and iloc let you quickly pick exactly what you want, saving time and avoiding mistakes. This makes data analysis faster and more accurate.

Where it fits

Before learning loc and iloc, you should know what a DataFrame is and how data is organized in rows and columns. After mastering row selection, you can learn about selecting columns, filtering data with conditions, and modifying data. This is a key step in exploring and cleaning data.

Mental Model

Core Idea

loc selects rows by their names or labels, iloc selects rows by their position number.

Think of it like...

Imagine a book with pages numbered from 1 to 100. iloc is like picking pages by their page number, while loc is like picking pages by the chapter title or section name written on the page.

DataFrame rows:
┌─────────────┐
│ Index Label │  <- loc uses these labels
├─────────────┤
│     A       │
│     B       │
│     C       │
└─────────────┘

Positions:
┌─────────────┐
│ Position #  │  <- iloc uses these numbers
├─────────────┤
│     0       │
│     1       │
│     2       │
└─────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding DataFrame Rows and Index

Concept: Learn what rows and index labels are in a DataFrame.

A DataFrame is like a table with rows and columns. Each row has a label called an index. By default, these labels are numbers starting from 0, but they can be any names or numbers. The index helps us find rows easily.

Result

You can identify rows by their position (0,1,2...) or by their label (like 'A', 'B', 'C').

Knowing that rows have labels (index) and positions helps you understand why there are two ways to select rows.

2

Foundation

Basic Row Selection with iloc

Concept: Select rows by their position number using iloc.

iloc uses numbers to pick rows. For example, df.iloc[0] picks the first row, df.iloc[1:3] picks the second and third rows. This works like counting rows from the top, starting at zero.

Result

You get the rows at the positions you asked for.

Selecting by position is simple and always works the same, even if row labels are strange or missing.

3

Intermediate

Basic Row Selection with loc

Concept: Select rows by their index labels using loc.

loc uses the row labels to pick rows. For example, if your DataFrame has labels 'A', 'B', 'C', then df.loc['A'] picks the row labeled 'A'. You can also select multiple rows like df.loc['A':'C'], which includes all rows from 'A' to 'C' inclusive.

Result

You get rows matching the labels you specified.

Selecting by label is powerful when your data has meaningful row names, making your code easier to read.

4

Intermediate

Differences Between loc and iloc

🤔Before reading on: Do you think loc and iloc behave the same when slicing rows? Commit to your answer.

Concept: Understand how loc and iloc handle slicing differently.

iloc slices rows by position and excludes the last position, like normal Python slices. loc slices rows by label and includes the last label. For example, df.iloc[0:2] gets rows at positions 0 and 1, but df.loc['A':'C'] gets rows labeled 'A', 'B', and 'C'.

Result

You see that loc includes the end label, iloc excludes the end position.

Knowing this difference prevents off-by-one errors when selecting rows.

5

Intermediate

Selecting Rows with Lists and Conditions

🤔Before reading on: Can loc and iloc both select multiple rows using a list? Commit to your answer.

Concept: Learn how to select multiple rows using lists and conditions.

You can pass a list of labels to loc, like df.loc[['A', 'C']], to get rows 'A' and 'C'. iloc accepts a list of positions, like df.iloc[[0, 2]]. Also, loc can select rows based on conditions, e.g., df.loc[df['Age'] > 30] picks rows where Age is over 30.

Result

You get exactly the rows you want, either by specific labels, positions, or conditions.

Combining loc with conditions lets you filter data easily without loops.

6

Advanced

Handling Missing or Non-Unique Indexes

🤔Before reading on: What happens if your DataFrame has duplicate row labels and you use loc? Commit to your answer.

Concept: Understand how loc and iloc behave with missing or duplicate row labels.

If your DataFrame has duplicate labels, loc returns all rows matching that label. iloc always selects by position, so duplicates don't affect it. If labels are missing, loc will raise an error if you try to select a label that does not exist.

Result

You learn how to handle or avoid errors and unexpected results when indexes are not unique.

Knowing this helps you write safer code and clean your data index properly.

7

Expert

Performance and Internals of loc and iloc

🤔Before reading on: Do you think loc and iloc have the same speed and internal process? Commit to your answer.

Concept: Explore how loc and iloc work internally and their performance differences.

iloc uses integer positions directly, so it is very fast and simple. loc needs to look up labels in the index, which can be slower if the index is complex or large. pandas uses optimized data structures like hash tables or trees for label lookup. Understanding this helps when working with very large datasets.

Result

You realize when to prefer iloc for speed or loc for readability and label-based selection.

Knowing internal workings helps optimize code and avoid slowdowns in big data projects.

Under the Hood

iloc works by directly accessing rows by their integer position in memory, like an array index. loc first looks up the label in the DataFrame's index structure, which can be a hash map or tree, to find the matching row positions. Then it retrieves those rows. This lookup step adds overhead but allows flexible label-based access.

Why designed this way?

pandas was designed to handle both positional and label-based indexing because data often has meaningful labels but also needs fast numeric access. The dual system balances ease of use and performance. Alternatives like only numeric or only label indexing would limit flexibility or speed.

DataFrame
┌───────────────────────────────┐
│ Index (labels) ──────────────┐│
│ ┌───────────────┐            ││
│ │ 'A' 'B' 'C'   │            ││
│ └───────────────┘            ││
│                              ││
│ Positions (0,1,2) ──────────>││
│                              ││
│ loc: label lookup ──────────>││
│ iloc: direct position access ││
└───────────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does df.loc[0] always select the first row? Commit to yes or no.

Common Belief: df.loc[0] selects the first row like iloc[0].

Reality: loc selects by label, so df.loc[0] selects the row with label 0, which may not be the first row if the index is not default or numeric.

Why it matters: Assuming loc works like iloc can cause wrong data selection and bugs, especially with custom indexes.

Quick: Does slicing with loc exclude the last label like normal Python slices? Commit to yes or no.

Common Belief: Slicing with loc excludes the last label, just like normal Python slices.

Reality: loc slicing includes the last label, unlike normal Python slices which exclude the end.

Why it matters: This difference can cause off-by-one errors and unexpected rows in your results.

Quick: Can iloc accept boolean conditions like loc? Commit to yes or no.

Common Belief: iloc can select rows using boolean conditions just like loc.

Reality: iloc only accepts integer positions or lists of positions, not boolean arrays. Boolean indexing works only with loc or directly on the DataFrame.

Why it matters: Trying to use boolean conditions with iloc causes errors and confusion.

Quick: If a DataFrame has duplicate labels, does loc return only the first matching row? Commit to yes or no.

Common Belief: loc returns only the first row when labels are duplicated.

Reality: loc returns all rows matching the duplicated label.

Why it matters: Not knowing this can lead to unexpected multiple rows in your output, affecting analysis.

Expert Zone

1

When the index is a MultiIndex (multiple levels), loc can select rows using tuples of labels, but iloc still uses simple integer positions.

2

Using loc with slices on unsorted indexes can cause unpredictable results or errors; sorting the index first is recommended.

3

Boolean indexing with loc creates a copy of the data, which can affect memory usage and performance in large datasets.

When NOT to use

Avoid using loc when your index labels are not unique or not meaningful; instead, reset the index or use iloc for position-based selection. For very large datasets where speed is critical, prefer iloc or specialized libraries like Dask.

Production Patterns

In real-world data pipelines, loc is often used for filtering data by meaningful keys like dates or IDs, while iloc is used in loops or algorithms needing fast positional access. Combining loc with boolean conditions is common for data cleaning and feature engineering.

Connections

SQL WHERE clause

loc with boolean conditions works like SQL's WHERE clause to filter rows.

Understanding loc filtering helps grasp how databases select rows, bridging pandas and SQL skills.

Array indexing in NumPy

iloc is similar to NumPy array indexing by position.

Knowing NumPy indexing makes iloc intuitive, as both use zero-based integer positions.

File directory navigation

Selecting rows by label (loc) is like choosing folders by name, while selecting by position (iloc) is like picking folders by their order in a list.

This connection helps understand why both label and position selection are useful in organizing and accessing data.

Common Pitfalls

#1 Using loc with integer positions assuming default index.

Wrong approach:

df.loc[0:2]

Correct approach:

df.iloc[0:2]

Root cause:

Confusing loc (label-based) with iloc (position-based) leads to wrong rows being selected.

#2 Using iloc with boolean conditions.

Wrong approach:

df.iloc[df['Age'] > 30]

Correct approach:

df.loc[df['Age'] > 30]

Root cause:

iloc does not support boolean indexing; only loc or direct DataFrame filtering does.

#3 Assuming loc slice excludes the last label.

Wrong approach:

df.loc['A':'C'] expecting rows 'A' and 'B' only

Correct approach:

df.loc['A':'C'] includes rows 'A', 'B', and 'C'

Root cause:

Not knowing loc slice includes the end label causes off-by-one errors.

Key Takeaways

loc selects rows by their labels, iloc selects rows by their integer positions.

loc includes the last label in slices, while iloc excludes the last position like normal Python slicing.

Boolean conditions work with loc but not with iloc.

Duplicate or missing labels affect loc but not iloc, so choose based on your data's index.

Understanding these differences helps avoid common bugs and write clearer, faster data selection code.