np.genfromtxt() for handling missing data in NumPy

Introduction

We use np.genfromtxt() to read data from text files, especially when some data is missing. It helps us load data smoothly without errors.

When reading CSV or text files that have empty or missing values.

When you want to replace missing data with a default value while loading.

When you need to load data but want to avoid program crashes due to missing entries.

When you want to specify how missing data should be handled during import.

Syntax

NumPy

np.genfromtxt(fname, delimiter=None, dtype=float, missing_values=None, filling_values=None, skip_header=0, usecols=None)

fname is the file name or path to read from.

missing_values tells which values to treat as missing (like empty strings).

Examples

Load data from a CSV file with default settings. Missing values become nan.

NumPy

data = np.genfromtxt('data.csv', delimiter=',')

Replace missing values with 0 while loading the data.

NumPy

data = np.genfromtxt('data.csv', delimiter=',', filling_values=0)

Treat empty strings as missing and fill them with -1.

NumPy

data = np.genfromtxt('data.csv', delimiter=',', missing_values='', filling_values=-1)

Sample Program

This code reads a small CSV-like text with missing values. It replaces missing spots with -999 so we can see where data was missing.

NumPy

import numpy as np
from io import StringIO

# Simulate a CSV file with missing data
csv_data = StringIO('''
1,2,3
4,,6
7,8,
,10,11
''')

# Load data treating empty fields as missing and fill with -999
array = np.genfromtxt(csv_data, delimiter=',', missing_values='', filling_values=-999)
print(array)

Important Notes

Missing values are converted to nan by default if no filling value is given.

You can specify which values count as missing using missing_values.

Use filling_values to replace missing data with a number you choose.

Summary

np.genfromtxt() helps load data files with missing values safely.

You can tell it what counts as missing and what to fill in instead.

This makes data loading easier and avoids errors from missing data.