How to Handle Missing Values in read_csv in pandas

You can handle missing values in pandas.read_csv by using the na_values parameter to specify additional strings to recognize as missing, and keep_default_na to control default missing value detection. This lets you cleanly load data with missing entries identified correctly.

Why This Happens

When you load a CSV file with pandas.read_csv, some missing values might not be recognized automatically if they use unusual markers like 'NA', 'missing', or empty strings. This causes those values to be read as normal strings instead of NaN, which can lead to errors or wrong analysis later.

python

import pandas as pd
from io import StringIO

data = '''id,name,age
1,Alice,25
2,Bob,missing
3,Charlie,30
4,David,NA
5,Eve,'''

# Load CSV without specifying missing values
df = pd.read_csv(StringIO(data))
print(df)

Output

   id     name      age
0   1    Alice       25
1   2      Bob  missing
2   3  Charlie       30
3   4    David       NA
4   5      Eve       

The Fix

Use the na_values parameter to tell pandas which strings should be treated as missing values. You can add custom markers like 'missing', 'NA', or empty strings. Also, keep_default_na=True ensures pandas still recognizes its default missing markers.

python

import pandas as pd
from io import StringIO

data = '''id,name,age
1,Alice,25
2,Bob,missing
3,Charlie,30
4,David,NA
5,Eve,'''

# Load CSV with custom missing values
missing_markers = ['missing', 'NA', '']
df = pd.read_csv(StringIO(data), na_values=missing_markers, keep_default_na=True)
print(df)

Output

   id     name   age
0   1    Alice  25.0
1   2      Bob   NaN
2   3  Charlie  30.0
3   4    David   NaN
4   5      Eve   NaN

Prevention

To avoid missing value issues, always check your data source for how missing data is marked. Use na_values in read_csv to cover all those cases. Also, inspect your DataFrame after loading with df.info() or df.isna().sum() to confirm missing values are detected correctly.

Keeping consistent missing value markers in your data files helps prevent confusion. When possible, clean or standardize missing data before loading.

Related Errors

Sometimes missing values are read as strings causing type errors when performing calculations. Another common issue is forgetting to set na_values for custom markers, leading to wrong data types or analysis mistakes.

Quick fixes include reloading data with proper na_values or converting columns manually using pd.to_numeric(errors='coerce') to force invalid parsing to NaN.

Key Takeaways

Use the na_values parameter in read_csv to specify all missing value markers.

Keep keep_default_na=True to retain pandas default missing value detection.

Check your data source for missing value formats before loading.

Verify missing values after loading with df.info() or df.isna().sum().

Convert columns manually if missing values were not detected initially.