Reading text files (readtable, textscan) in MATLAB - Deep Dive

Overview - Reading text files (readtable, textscan)

What is it?

Reading text files means opening files that store data as plain text and bringing that data into MATLAB so you can work with it. Two common ways to do this are using readtable and textscan. readtable automatically reads structured data into a table format, while textscan reads data piece by piece based on a format you specify. Both help turn raw text into organized data you can analyze.

Why it matters

Without tools like readtable and textscan, you would have to manually open files and parse text, which is slow and error-prone. These functions save time and reduce mistakes by automatically handling different data formats. This makes it easier to analyze data from experiments, logs, or reports, helping you make decisions faster and more accurately.

Where it fits

Before learning this, you should know basic MATLAB commands and how to work with variables and arrays. After mastering reading text files, you can learn data cleaning, visualization, and advanced data analysis techniques. This topic is a key step in the data science workflow where raw data becomes usable.

Mental Model

Core Idea

Reading text files means converting raw text data into structured MATLAB variables using functions that understand the file's layout.

Think of it like...

It's like unpacking a suitcase: readtable is like opening a suitcase with labeled compartments so you know exactly where each item goes, while textscan is like unpacking items one by one based on a checklist you have.

┌───────────────┐
│ Text File     │
│ (raw text)    │
└──────┬────────┘
       │
       ▼
┌───────────────┐       ┌───────────────┐
│ readtable     │──────▶│ Table in      │
│ (auto parse)  │       │ MATLAB       │
└───────────────┘       └───────────────┘
       │
       │
       ▼
┌───────────────┐       ┌───────────────┐
│ textscan      │──────▶│ Cell arrays / │
│ (custom parse)│       │ arrays in     │
└───────────────┘       │ MATLAB       │
                        └───────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding Text Files Basics

Concept: Learn what a text file is and how data is stored inside it.

A text file stores data as readable characters, like letters and numbers, arranged in lines. Each line can have values separated by spaces, commas, or tabs. For example, a file might look like: Name,Age,Score Alice,30,85 Bob,25,90 This is plain text, not formatted like Excel, but you can read it into MATLAB.

Result

You understand that text files are simple and can be opened with any text editor, and that data is organized by lines and separators.

Knowing the structure of text files helps you decide how to read and parse them correctly in MATLAB.

2

Foundation

Basic File Reading in MATLAB

Concept: Learn how to open and read text files using simple MATLAB commands.

You can open a file using fopen, read lines with fgetl, and close it with fclose. For example: fid = fopen('data.txt'); line = fgetl(fid); fclose(fid); This reads one line as a string. But this method requires manual parsing to get numbers or separate values.

Result

You can open and read raw text lines from a file in MATLAB.

Understanding manual file reading shows why higher-level functions like readtable and textscan are helpful.

3

Intermediate

Using readtable for Structured Data

🤔Before reading on: do you think readtable can automatically detect column names and data types, or do you need to specify them all manually? Commit to your answer.

Concept: readtable reads structured text files into tables, automatically detecting headers and data types.

readtable('data.csv') reads a CSV file and returns a table where each column has a name and data type. It handles commas, tabs, and other delimiters. You can access data by column names, like T.Age. It also handles missing data and different formats easily.

Result

You get a MATLAB table with named columns and typed data ready for analysis.

Knowing readtable automates parsing saves time and reduces errors when working with common structured files.

4

Intermediate

Using textscan for Custom Parsing

🤔Before reading on: do you think textscan reads the whole file at once or line by line? Commit to your answer.

Concept: textscan reads text files using a format string, allowing custom parsing of complex or irregular data.

You specify a format like '%s %d %f' to read a string, integer, and float per line. For example: fid = fopen('data.txt'); data = textscan(fid, '%s %d %f', 'Delimiter', ','); fclose(fid); This returns a cell array with each column's data. You control how data is read, useful for mixed or unusual formats.

Result

You get data separated into cells or arrays according to your format, even for complex files.

Understanding textscan's flexibility lets you handle files that don't fit standard table formats.

5

Intermediate

Handling Delimiters and Missing Data

Concept: Learn how to specify delimiters and manage missing or extra data in files.

Both readtable and textscan allow you to set delimiters like commas, tabs, or spaces. For example, readtable('file.txt', 'Delimiter', '\t') reads tab-separated files. You can also specify how to treat missing data, like empty fields or special markers, using options like 'TreatAsEmpty'.

Result

You can correctly read files with different separators and handle missing values without errors.

Knowing how to adjust delimiters and missing data handling prevents common reading errors and data corruption.

6

Advanced

Optimizing Performance for Large Files

🤔Before reading on: do you think readtable or textscan is faster for very large files? Commit to your answer.

Concept: Learn which function is more efficient for large datasets and how to optimize reading speed.

readtable is convenient but can be slower for huge files because it does more processing. textscan can be faster if you specify formats precisely and read only needed columns. You can also read files in chunks or use options like 'ReadVariableNames', false to speed up reading.

Result

You can read large files faster by choosing the right function and options.

Understanding performance trade-offs helps you handle big data efficiently in MATLAB.

7

Expert

Advanced Parsing with textscan Format Specifiers

🤔Before reading on: do you think textscan can skip unwanted data or read variable-length lines? Commit to your answer.

Concept: textscan supports advanced format specifiers to skip data, read variable-length fields, and handle complex patterns.

You can use '%*s' to skip a string, '%[chars]' to read specific characters, and '%n' for numbers. textscan can also read lines with different lengths by using 'CollectOutput' or reading line by line. This allows parsing of messy or irregular files that standard functions can't handle.

Result

You can extract exactly the data you want from complex text files, ignoring noise or irrelevant parts.

Mastering textscan's format specifiers unlocks powerful custom parsing capabilities for real-world messy data.

Under the Hood

readtable internally uses textscan and other parsing functions to read the entire file, detect headers, delimiters, and data types, then organizes data into a table structure. textscan reads the file sequentially, applying the format string to parse each piece of data into MATLAB variables. Both functions handle file opening and closing, memory allocation, and type conversion automatically.

Why designed this way?

MATLAB needed flexible yet easy-to-use functions to read diverse text data formats common in science and engineering. readtable was designed for quick import of well-structured data, while textscan was created for fine control over parsing. This separation balances ease of use and flexibility, avoiding one-size-fits-all limitations.

┌───────────────┐
│ Text File     │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ readtable     │
│ - Detects     │
│   headers     │
│ - Chooses     │
│   delimiters  │
│ - Calls       │
│   textscan    │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ textscan      │
│ - Reads file  │
│   sequentially│
│ - Parses data │
│   by format   │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ MATLAB Data   │
│ Variables     │
│ (table, cells)│
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does readtable always correctly guess the data types without errors? Commit to yes or no.

Common Belief: readtable perfectly detects all data types and formats automatically without any user input.

Reality: readtable guesses data types but can misinterpret columns, especially with mixed data or missing values, requiring user options to fix.

Why it matters: Relying blindly on readtable's guesses can cause wrong data types, leading to errors or incorrect analysis results.

Quick: Can textscan read files without specifying a format string? Commit to yes or no.

Common Belief: textscan can automatically detect the format of any text file without needing a format string.

Reality: textscan requires a format string to know how to parse each piece of data; it cannot guess formats automatically.

Why it matters: Without a correct format string, textscan will fail or produce wrong data, so users must understand the file structure.

Quick: Is readtable always faster than textscan for large files? Commit to yes or no.

Common Belief: readtable is always faster and better than textscan for reading any text file.

Reality: For very large or complex files, textscan can be faster and more memory efficient if used properly.

Why it matters: Choosing the wrong function for large data can cause slow performance or memory issues.

Quick: Does textscan close the file automatically after reading? Commit to yes or no.

Common Belief: textscan automatically closes the file after reading is done.

Reality: textscan does not close the file; the user must call fclose to release the file handle.

Why it matters: Not closing files can lead to resource leaks and errors when opening many files.

Expert Zone

1

readtable can be customized with many options like 'Format', 'TextType', and 'DatetimeType' to handle tricky data types and improve import accuracy.

2

textscan's 'HeaderLines' and 'EndOfLine' options allow skipping unwanted parts of files and handling different line endings, which is crucial for cross-platform compatibility.

3

Combining textscan with low-level file functions like fseek enables partial file reading and random access, useful for very large files or streaming data.

When NOT to use

Avoid readtable when files have irregular formats, mixed data types in columns, or require custom parsing logic; use textscan or custom parsing instead. For very large files where memory is limited, consider reading in chunks or using datastore objects. For binary or non-text files, neither function applies.

Production Patterns

Professionals often use readtable for quick imports of CSV or TSV files in data cleaning pipelines. textscan is used in scripts that process log files or scientific data with complex formats. Combining these with automated scripts and error handling ensures robust data ingestion in production.

Connections

Data Cleaning

builds-on

Understanding how to read raw text files correctly is essential before cleaning data, as errors in reading propagate to cleaning steps.

Regular Expressions

complements

Using regular expressions with textscan or preprocessing files helps extract complex patterns, enhancing parsing flexibility.

Natural Language Processing (NLP)

builds-on

Reading and parsing text files accurately is the first step in NLP workflows, where raw text is transformed into analyzable data.

Common Pitfalls

#1 Forgetting to close the file after reading.

Wrong approach:

fid = fopen('data.txt');
data = textscan(fid, '%s %d');
% forgot fclose(fid);

Correct approach:

fid = fopen('data.txt');
data = textscan(fid, '%s %d');
fclose(fid);

Root cause:

Beginners often overlook resource management, assuming MATLAB handles file closing automatically.

#2 Using wrong delimiter causing parsing errors.

Wrong approach:

T = readtable('data.csv', 'Delimiter', '\t'); % file is comma-separated

Correct approach:

T = readtable('data.csv', 'Delimiter', ',');

Root cause:

Not matching the delimiter to the actual file format leads to incorrect column splitting.

#3 Assuming readtable reads all data as strings.

Wrong approach:

T = readtable('data.csv', 'TextType', 'char'); % expecting all strings

Correct approach:

T = readtable('data.csv'); % lets MATLAB detect types automatically

Root cause:

Misunderstanding that readtable can detect numeric and categorical types, forcing all strings loses this benefit.

Key Takeaways

Reading text files in MATLAB transforms raw text data into usable variables for analysis.

readtable is best for structured files with consistent columns and headers, automating much of the parsing.

textscan offers flexible, custom parsing for complex or irregular text files using format strings.

Choosing the right function and options depends on file size, format complexity, and analysis needs.

Proper file handling, including closing files and setting delimiters, prevents common errors and resource issues.

Practice

1. What is the main advantage of using readtable in MATLAB to read text files?

easy

A. It reads files without opening them first.

B. It requires manual specification of each data type.

C. It only reads numeric data, ignoring text.

D. It automatically loads data into a table with proper formatting.

Solution

1. Step 1: Understand readtable functionality

   readtable is designed to load text files directly into tables, automatically detecting and formatting columns.

2. Step 2: Compare options

   It automatically loads data into a table with proper formatting. correctly describes this automatic loading and formatting. Other options describe incorrect behaviors.

3. Final Answer:

   It automatically loads data into a table with proper formatting. -> Option D

4. Quick Check:

   readtable = automatic table loading [OK]

Hint: Remember: readtable = easy table import [OK]

Common Mistakes:

• Thinking readtable needs manual type input
• Assuming readtable only reads numbers
• Believing files don't need opening before reading

2. Which of the following is the correct syntax to open a text file named data.txt for reading in MATLAB?

easy

A. fileID = fopen('data.txt', 'r');

B. fileID = open('data.txt', 'r');

C. fileID = readfile('data.txt');

D. fileID = textscan('data.txt', 'r');

Solution

1. Step 1: Recall file opening function

   In MATLAB, fopen is used to open files with a mode like 'r' for reading.

2. Step 2: Check each option

   fileID = fopen('data.txt', 'r'); uses fopen correctly. fileID = open('data.txt', 'r'); uses a non-existent open function. fileID = readfile('data.txt'); uses readfile which is not a MATLAB function. fileID = textscan('data.txt', 'r'); misuses textscan which reads from an open file, not opens it.

3. Final Answer:

   fileID = fopen('data.txt', 'r'); -> Option A

4. Quick Check:

   Open file = fopen(filename, 'r') [OK]

Hint: Use fopen to open files before reading [OK]

Common Mistakes:

• Using open instead of fopen
• Trying to read file directly without fopen
• Confusing textscan with fopen

3. Given the following MATLAB code snippet, what will be the output stored in data?

fileID = fopen('numbers.txt', 'r');
data = textscan(fileID, '%d %f');
fclose(fileID);

Assuming numbers.txt contains:

10 3.14
20 2.71
30 1.62

medium

A. A 1x2 cell array with first cell [10;20;30] and second cell [3.14;2.71;1.62]

B. A 3x2 numeric matrix with all numbers

C. A single numeric array combining all values

D. An error because textscan cannot read mixed data

Solution

1. Step 1: Understand textscan output format

   textscan returns a cell array where each cell contains a column of data matching the format specifiers.

2. Step 2: Analyze the format and file content

   Format '%d %f' reads an integer and a float per line. The file has 3 lines, so the first cell will be [10;20;30] and the second cell [3.14;2.71;1.62].

3. Final Answer:

   A 1x2 cell array with first cell [10;20;30] and second cell [3.14;2.71;1.62] -> Option A

4. Quick Check:

   textscan returns cell array columns [OK]

Hint: textscan returns cells, each for one format column [OK]

Common Mistakes:

• Expecting a numeric matrix instead of cell array
• Thinking textscan merges all data into one array
• Assuming textscan cannot read mixed types

4. Consider this MATLAB code snippet:

fileID = fopen('data.txt', 'r');
data = textscan(fileID, '%s %d');

What is the main problem with this code?

medium

A. textscan cannot read strings and integers together.

B. The format specifier '%s %d' is invalid.

C. The file is not closed after reading.

D. fopen should be called with 'w' mode to read.

Solution

1. Step 1: Check file handling steps

   The code opens the file with fopen and reads with textscan, but does not close the file.

2. Step 2: Validate format and usage

   The format '%s %d' is valid for reading a string and an integer. textscan supports mixed types. fopen with 'r' is correct for reading.

3. Final Answer:

   The file is not closed after reading. -> Option C

4. Quick Check:

   Always fclose after fopen [OK]

Hint: Always fclose after reading files [OK]

Common Mistakes:

• Forgetting to close the file
• Thinking '%s %d' is invalid format
• Confusing fopen modes

5. You have a text file records.txt with mixed data: a name (string), age (integer), and height (float) on each line, separated by commas, like:

John,25,5.9
Anna,30,5.5
Mike,22,6.1

Which MATLAB approach correctly reads this file into a table with proper data types?

hard

A. Use readtable('records.txt', 'Delimiter', ',', 'Format', '%s%d%f')

B. Use readtable('records.txt') without specifying delimiter

C. Use textscan(fopen('records.txt', 'r'), '%s %d %f', 'Delimiter', ',') and fclose after

D. Use textscan(fopen('records.txt'), '%s %d %f') without closing file

Solution

1. Step 1: Identify correct reading method for mixed CSV data into table

   readtable automatically detects common delimiters like commas and loads data into a table with inferred types (string for names, double for numbers).

2. Step 2: Compare options

   Use readtable('records.txt') without specifying delimiter: readtable('records.txt') auto-detects comma delimiter. Use readtable('records.txt', 'Delimiter', ',', 'Format', '%s%d%f') uses invalid 'Format' parameter (readtable lacks it). Use textscan(fopen('records.txt', 'r'), '%s %d %f', 'Delimiter', ',') and fclose after: textscan syntax invalid ('Delimiter' not for fid), returns cell array not table. Use textscan(fopen('records.txt'), '%s %d %f') without closing file: textscan fails to parse commas (default whitespace), no fclose.

3. Final Answer:

   Use readtable('records.txt') without specifying delimiter -> Option B

4. Quick Check:

   readtable = auto delimiter + table types [OK]

Hint: readtable auto-detects CSV delimiters into tables [OK]

Common Mistakes:

• Using textscan expecting table output
• Adding invalid 'Format' to readtable
• Omitting delimiter handling in textscan