Saving and loading data (scipy.io) - Deep Dive

Overview - Saving and loading data (scipy.io)

What is it?

Saving and loading data with scipy.io means storing your data in files and reading it back later using the SciPy library. This helps you keep your work safe and share it with others. SciPy provides tools to save data in formats like MATLAB files or simple binary files. It makes working with scientific data easier and more organized.

Why it matters

Without saving and loading data, you would lose your work every time you close your program. It would be hard to share data or continue analysis later. SciPy's saving and loading tools solve this by letting you store complex data structures efficiently. This saves time, avoids mistakes, and helps collaborate on data science projects.

Where it fits

Before learning this, you should know how to create and manipulate arrays with NumPy. After this, you can learn about data formats like HDF5 or databases for bigger projects. This topic fits in the data handling and storage part of your data science journey.

Mental Model

Core Idea

Saving and loading data with scipy.io is like packing your scientific data into a box to keep it safe and unpacking it later exactly as it was.

Think of it like...

Imagine you have a toolbox with different compartments for screws, nails, and tools. Saving data is like putting each item carefully into the right compartment so you can find it later. Loading data is opening the toolbox and taking out exactly what you packed before.

┌───────────────┐      ┌───────────────┐
│  Data in RAM  │─────▶│ Save to File  │
└───────────────┘      └───────────────┘
                           │
                           ▼
                     ┌───────────────┐
                     │ Data on Disk  │
                           │
                           ▼
                     ┌───────────────┐
                     │ Load from File│
                           │
                           ▼
                     ┌───────────────┐
                     │ Data in RAM   │

Build-Up - 7 Steps

1

Foundation

Understanding SciPy and io module

Concept: Learn what SciPy and its io module are for saving and loading data.

SciPy is a Python library for scientific computing. The io module inside SciPy helps you save and load data in different formats. It supports MATLAB files (.mat), simple binary files, and more. This module makes it easy to store arrays and other data types.

Result

You know that scipy.io is the part of SciPy used to save and load data files.

Understanding the role of scipy.io helps you see how data moves between your program and storage.

2

Foundation

Basic saving with savemat function

Concept: Learn how to save data to a MATLAB .mat file using savemat.

You can save Python data like NumPy arrays to a .mat file using scipy.io.savemat. You provide a filename and a dictionary where keys are variable names and values are data. For example: import numpy as np from scipy.io import savemat arr = np.array([1, 2, 3]) savemat('data.mat', {'array': arr})

Result

A file named 'data.mat' is created containing the array data.

Knowing how to save data in a common format like .mat lets you share data with MATLAB users or save your work.

3

Intermediate

Loading data with loadmat function

🤔Before reading on: do you think loadmat returns data exactly as saved or in a different structure? Commit to your answer.

Concept: Learn how to load data from a .mat file back into Python using loadmat.

To read data saved in a .mat file, use scipy.io.loadmat. It returns a dictionary where keys are variable names and values are the data. For example: from scipy.io import loadmat data = loadmat('data.mat') print(data['array']) Note: loadmat adds some extra keys and the data may be wrapped in arrays.

Result

You get a Python dictionary with your saved variables accessible by their names.

Understanding that loadmat returns a dictionary helps you access your saved data correctly and avoid confusion.

4

Intermediate

Saving and loading multiple variables

🤔Before reading on: do you think you can save multiple variables in one .mat file or only one? Commit to your answer.

Concept: Learn how to save and load several variables at once using dictionaries.

You can save many variables by putting them all in a dictionary passed to savemat. For example: savemat('multi.mat', {'x': np.array([1,2]), 'y': np.array([3,4])}) When loading, you get all variables in a dictionary: data = loadmat('multi.mat') print(data['x'], data['y'])

Result

Multiple variables are saved and loaded together in one file.

Knowing how to handle multiple variables in one file makes your data management cleaner and more efficient.

5

Intermediate

Using other scipy.io formats: wavfile

Concept: Learn about saving and loading audio data using scipy.io.wavfile.

Scipy.io also supports other formats like WAV audio files. You can save audio data with write and load it with read: from scipy.io import wavfile import numpy as np rate = 44100 samples = np.array([0, 1000, -1000, 0], dtype=np.int16) wavfile.write('sound.wav', rate, samples) rate_loaded, samples_loaded = wavfile.read('sound.wav')

Result

You create and read WAV audio files with sample rate and data.

Knowing scipy.io supports formats beyond .mat files broadens your ability to work with different scientific data types.

6

Advanced

Handling MATLAB file versions and options

🤔Before reading on: do you think savemat saves all .mat files in the same format or can it vary? Commit to your answer.

Concept: Learn about different MATLAB file versions and how to control saving options.

MATLAB files have versions like 4, 5, and 7.3. By default, savemat saves version 5 files. You can specify the version with the 'do_compression' and 'format' parameters: savemat('compressed.mat', {'a': arr}, do_compression=True) Some versions support compression, others don't. Version 7.3 uses HDF5 format but is not supported by scipy.io.

Result

You can save files with compression to reduce size or choose format compatibility.

Understanding file versions and options helps you create files compatible with different MATLAB versions and save disk space.

7

Expert

Limitations and alternatives to scipy.io saving

🤔Before reading on: do you think scipy.io can save any Python object or only arrays and simple types? Commit to your answer.

Concept: Learn the limits of scipy.io saving and when to use other tools like HDF5 or pickle.

Scipy.io mainly saves arrays and simple data types. It cannot save complex Python objects or very large datasets efficiently. For those, formats like HDF5 (via h5py) or Python's pickle module are better. Also, scipy.io cannot read MATLAB 7.3 files (HDF5-based). Knowing these limits helps choose the right tool.

Result

You understand when scipy.io is not enough and what alternatives to use.

Knowing the boundaries of scipy.io prevents wasted effort and data loss in complex projects.

Under the Hood

Scipy.io saves data by converting Python objects like NumPy arrays into a binary format compatible with MATLAB .mat files or other supported formats. When saving, it serializes the data dictionary into a structured file format with headers and data blocks. Loading reverses this by parsing the file, reconstructing arrays and variables into Python objects. For WAV files, it writes raw audio samples with headers describing sample rate and format.

Why designed this way?

Scipy.io was designed to bridge Python and MATLAB users, enabling easy data exchange. MATLAB's .mat format is widely used in science, so supporting it helps collaboration. The design balances simplicity and compatibility, focusing on common data types like arrays. It avoids complex Python objects to keep files portable and readable by MATLAB and other tools.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Python Object │──────▶│ Serialization │──────▶│  .mat or WAV  │
│ (e.g., array) │       │ (binary format)│       │   File on Disk│
└───────────────┘       └───────────────┘       └───────────────┘
       ▲                                               │
       │                                               ▼
┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Python Object │◀──────│ Deserialization│◀─────│  .mat or WAV  │
│ (loaded data) │       │ (read binary)  │       │   File on Disk│
└───────────────┘       └───────────────┘       └───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does loadmat return exactly the same data structure you saved with savemat? Commit to yes or no.

Common Belief: loadmat returns the exact same data structure as saved by savemat.

Reality: loadmat returns a dictionary with extra metadata keys and sometimes wraps arrays in extra dimensions.

Why it matters: Assuming exact equality can cause bugs when accessing data or comparing results.

Quick: Can scipy.io save any Python object like lists of custom classes? Commit to yes or no.

Common Belief: scipy.io can save any Python object, including complex custom classes.

Reality: scipy.io mainly saves arrays and simple data types; it cannot save complex Python objects.

Why it matters: Trying to save unsupported objects leads to errors or data loss.

Quick: Does savemat always save compressed files by default? Commit to yes or no.

Common Belief: savemat compresses files automatically to save space.

Reality: savemat does not compress files by default; compression must be enabled explicitly.

Why it matters: Expecting small files without enabling compression can waste disk space.

Quick: Can scipy.io read MATLAB 7.3 files saved in HDF5 format? Commit to yes or no.

Common Belief: scipy.io can read all MATLAB .mat files including version 7.3.

Reality: scipy.io cannot read MATLAB 7.3 files because they use HDF5 format not supported by scipy.io.

Why it matters: Trying to load 7.3 files with scipy.io causes errors and confusion.

Expert Zone

1

savemat's compression option uses zlib which can slow saving but reduce file size significantly.

2

loadmat returns MATLAB structs as numpy structured arrays or nested dictionaries, which can be tricky to navigate.

3

WAV files saved with scipy.io.wavfile require data to be integer types; floating point data must be scaled and converted.

When NOT to use

Do not use scipy.io for very large datasets, complex Python objects, or MATLAB 7.3 files. Instead, use h5py for HDF5 files, pickle for Python objects, or specialized libraries for large data like Zarr or Parquet.

Production Patterns

In production, scipy.io is often used for quick data exchange with MATLAB or small experiments. For large-scale or complex data pipelines, teams use HDF5 or databases. Audio processing pipelines use scipy.io.wavfile for reading and writing WAV files in signal processing tasks.

Connections

HDF5 file format

builds-on

Understanding scipy.io's limitations with MATLAB 7.3 files leads to learning HDF5, a powerful format for large scientific data.

Python pickle module

alternative

Knowing when scipy.io cannot save complex objects helps you choose pickle for serializing Python-specific data.

Data serialization in computer science

same pattern

Saving and loading data with scipy.io is an example of serialization, a core concept in computer science for storing and transmitting data.

Common Pitfalls

#1 Trying to access saved variables directly without dictionary keys after loadmat.

Wrong approach:

data = loadmat('file.mat')
print(data.array) # wrong, 'array' is a key, not attribute

Correct approach:

data = loadmat('file.mat')
print(data['array']) # correct way to access

Root cause:

Misunderstanding that loadmat returns a dictionary, not an object with attributes.

#2 Saving data without enabling compression and expecting small file size.

Wrong approach:

savemat('file.mat', {'a': arr}) # no compression, large file

Correct approach:

savemat('file.mat', {'a': arr}, do_compression=True) # compressed file

Root cause:

Assuming compression is automatic when it must be explicitly enabled.

#3 Trying to save Python lists or custom objects directly with savemat.

Wrong approach:

savemat('file.mat', {'list': [1, 2, 3]}) # may cause error or unexpected results

Correct approach:

Convert lists to numpy arrays before saving:
savemat('file.mat', {'array': np.array([1, 2, 3])})

Root cause:

Not converting data to supported types like numpy arrays before saving.

Key Takeaways

Scipy.io provides simple tools to save and load scientific data in formats like MATLAB .mat and WAV files.

Saving data means packing it into files; loading means unpacking it back into Python objects.

Loadmat returns a dictionary with your saved variables, not the raw data directly.

Scipy.io works best with numpy arrays and simple data types, not complex Python objects.

Knowing its limits helps you choose better tools like HDF5 or pickle for advanced needs.