Regular expressions in MATLAB - Deep Dive

Overview - Regular expressions in MATLAB

What is it?

Regular expressions in MATLAB are patterns used to find, match, and manipulate text. They allow you to search for specific sequences of characters within strings. This helps automate tasks like data cleaning, validation, and extraction. MATLAB provides built-in functions to work with these patterns easily.

Why it matters

Without regular expressions, searching and processing text data would be slow and error-prone, especially with large datasets. They save time by automating complex text tasks, making data analysis more efficient and reliable. This is crucial in data science where text data is common and messy.

Where it fits

Before learning regular expressions, you should understand basic MATLAB string handling and indexing. After mastering regular expressions, you can explore advanced text analytics, natural language processing, and data cleaning techniques.

Mental Model

Core Idea

Regular expressions are like flexible search patterns that let you find and work with text by describing what you want, not exactly how it looks.

Think of it like...

Imagine using a metal detector on a beach to find coins. You don’t know exactly where each coin is, but the detector beeps when it senses metal nearby. Regular expressions beep when they find text matching your pattern.

Pattern matching flow:

Input String ──> [Regular Expression Pattern] ──> Matches Found

Example:
"cat", "cot", "cut"
Pattern: c.t
Matches: cat, cot, cut

Legend:
. = any single character
c = character 'c'
t = character 't'

Build-Up - 7 Steps

1

Foundation

Understanding Basic String Matching

Concept: Learn how to find exact text matches using simple patterns.

In MATLAB, you can use the function regexp to search for text. For example, regexp('apple pie', 'apple') returns the starting index of 'apple' in the string. This is the simplest form of pattern matching.

Result

Output: 1 (because 'apple' starts at the first character)

Understanding exact matching is the base for building more complex patterns.

2

Foundation

Using Special Characters in Patterns

Concept: Introduce special symbols that represent flexible matching rules.

Special characters like '.' match any single character, '*' means zero or more of the previous character, and '\d' matches any digit. For example, regexp('cat', 'c.t') matches 'cat' because '.' matches 'a'.

Result

Output: 1 (match found starting at first character)

Special characters let you create patterns that match many similar strings, not just exact text.

3

Intermediate

Extracting Matched Text Segments

🤔Before reading on: do you think regexp can return the matched text itself or only the position? Commit to your answer.

Concept: Learn how to get the actual text that matches the pattern, not just where it is.

Using the 'match' option in regexp, you can get the matched substrings. For example, regexp('I have 2 cats', '\d', 'match') returns {'2'}. This helps extract useful data from text.

Result

Output: {'2'}

Extracting matched text is key for data cleaning and analysis tasks.

4

Intermediate

Using Character Classes and Quantifiers

🤔Before reading on: do you think '[a-z]' matches uppercase letters? Commit to your answer.

Concept: Character classes let you specify sets of characters, and quantifiers control how many times they appear.

For example, '[a-z]' matches any lowercase letter. Quantifiers like '+' mean one or more times. regexp('hello123', '[a-z]+') matches 'hello'.

Result

Output: 1 (match starts at first character)

Character classes and quantifiers let you build precise and flexible patterns.

5

Intermediate

Using Anchors to Match Positions

🤔Before reading on: does '^' match the end of a string or the start? Commit to your answer.

Concept: Anchors specify where in the string the pattern should match, like start or end.

The '^' symbol matches the start of a string, and '$' matches the end. For example, regexp('cat', '^c') matches because 'c' is at the start.

Result

Output: 1 (match at start)

Anchors help ensure matches occur only at desired positions, improving accuracy.

6

Advanced

Using Tokens and Grouping for Complex Patterns

🤔Before reading on: do you think parentheses in patterns capture parts of the match? Commit to your answer.

Concept: Grouping parts of patterns lets you capture and reuse matched segments.

Parentheses '()' group parts of the pattern. For example, regexp('abc123', '(abc)(\d+)', 'tokens') returns {{'abc', '123'}}. This helps break down matches into parts.

Result

Output: {{'abc', '123'}}

Grouping and tokens enable detailed extraction and manipulation of text data.

7

Expert

Optimizing Patterns and Avoiding Common Pitfalls

🤔Before reading on: do you think greedy quantifiers always find the shortest match? Commit to your answer.

Concept: Learn how pattern matching behavior affects performance and results, including greedy vs lazy matching.

By default, quantifiers like '*' are greedy and match as much as possible. Using '.*?' makes them lazy, matching as little as possible. For example, regexp('aabb', 'a.*b') matches 'aabb' greedily, while 'a.*?b' matches 'aab' lazily.

Result

Output greedy: 1 (match 'aabb'), lazy: 1 (match 'aab')

Understanding greedy vs lazy matching prevents bugs and improves efficiency in complex text processing.

Under the Hood

MATLAB's regexp function compiles the pattern into a finite state machine that scans the input string character by character. It uses backtracking to try different paths when multiple matches are possible. The engine processes special symbols by translating them into matching rules that guide the search.

Why designed this way?

This design balances flexibility and speed. Finite state machines allow fast scanning, while backtracking supports complex patterns. Alternatives like simpler substring search would be faster but less powerful. The tradeoff favors expressive power for data science needs.

Input String ──> [Finite State Machine]
                    │
                    ├─> Matches Found
                    └─> Backtracking on failure

Pattern Symbols:
.  → any char
*  → repeat
\d → digit
() → group

Myth Busters - 4 Common Misconceptions

Quick: Does '.' match newline characters by default? Commit yes or no.

Common Belief: The '.' character matches any character including newlines.

Reality: In MATLAB, '.' matches any character except newline by default.

Why it matters: Assuming '.' matches newlines can cause missed matches or incorrect parsing of multi-line text.

Quick: Does '*' mean match exactly one or more times? Commit your answer.

Common Belief: '*' means match one or more times.

Reality: '*' means match zero or more times, so it can match nothing.

Why it matters: Misunderstanding '*' can lead to unexpected matches or empty results, causing bugs in data extraction.

Quick: Can regexp return the matched text without extra options? Commit yes or no.

Common Belief: regexp always returns the matched text by default.

Reality: By default, regexp returns the starting indices, not the matched text. You must specify 'match' or 'tokens' to get text.

Why it matters: Expecting matched text without options leads to confusion and incorrect code.

Quick: Does '^' match the end of a string? Commit yes or no.

Common Belief: '^' matches the end of a string.

Reality: '^' matches only the start of a string; '$' matches the end.

Why it matters: Using '^' incorrectly can cause patterns to fail or match wrong parts of text.

Expert Zone

1

MATLAB's regexp supports Unicode, but some character classes behave differently with Unicode characters, requiring careful pattern design.

2

Backtracking can cause performance issues with certain patterns; understanding how to write non-backtracking patterns improves speed.

3

Using named tokens is not supported in MATLAB regexp, so capturing groups must be managed by position, which can be tricky in complex patterns.

When NOT to use

Regular expressions are not ideal for parsing nested or recursive structures like HTML or JSON. Instead, use dedicated parsers or MATLAB's built-in JSON functions for such tasks.

Production Patterns

In production, MATLAB users often combine regexp with string functions like extractBetween and replace to clean and transform large text datasets. Patterns are optimized for speed and readability, and results are validated with test cases.

Connections

Finite State Machines

Regular expressions are implemented using finite state machines internally.

Understanding finite state machines helps grasp why some patterns are fast and others slow, improving pattern design.

Data Cleaning

Regular expressions are a core tool for cleaning messy text data.

Mastering regex empowers efficient removal of unwanted characters, extraction of key info, and standardization of text.

Linguistics

Regular expressions relate to pattern matching in linguistics for analyzing language structure.

Knowing regex deepens understanding of how language patterns can be formally described and processed.

Common Pitfalls

#1 Using greedy quantifiers when lazy is needed causes overmatching.

Wrong approach:

regexp('aabb', 'a.*b')

Correct approach:

regexp('aabb', 'a.*?b')

Root cause:

Not understanding that '*' is greedy by default and matches as much as possible.

#2 Expecting regexp to return matched text without specifying 'match' option.

Wrong approach:

indices = regexp('test123', '\d+')

Correct approach:

matches = regexp('test123', '\d+', 'match')

Root cause:

Confusing default output of regexp which is indices, not matched strings.

#3 Using '.' to match newline characters in multi-line text.

Wrong approach:

regexp('line1\nline2', 'line.*line')

Correct approach:

regexp('line1\nline2', 'line[\s\S]*line')

Root cause:

Assuming '.' matches newline characters, which it does not by default.

Key Takeaways

Regular expressions in MATLAB let you find and manipulate text by describing flexible search patterns.

Special characters and quantifiers expand simple matching into powerful text processing tools.

Using options like 'match' and 'tokens' helps extract useful parts of text for analysis.

Understanding greedy vs lazy matching and anchors improves accuracy and performance.

Regular expressions are powerful but have limits; use dedicated parsers for complex nested data.

Practice

1. What does the MATLAB function regexp do?

easy

A. Searches for text patterns in strings

B. Replaces text in strings

C. Sorts arrays numerically

D. Calculates the length of a string

Solution

1. Step 1: Understand the purpose of regexp

   The regexp function in MATLAB is used to find matches of a pattern within text.

2. Step 2: Compare with other functions

   regexprep replaces text, but regexp only searches for patterns.

3. Final Answer:

   Searches for text patterns in strings -> Option A

4. Quick Check:

   regexp = search function [OK]

Hint: Remember: regexp = search, regexprep = replace [OK]

Common Mistakes:

• Confusing regexp with regexprep
• Thinking regexp modifies strings
• Mixing regexp with numeric functions

2. Which of the following is the correct syntax to find all digits in the string 'abc123' using regexp?

easy

A. regexp('abc123', '\\d+')

B. regexp('abc123', '\\w+')

C. regexp('abc123', '\\s+')

D. regexp('abc123', '\\D+')

Solution

1. Step 1: Identify the pattern for digits

   In regular expressions, \d matches digits, and + means one or more.

2. Step 2: Check the options

   regexp('abc123', '\\d+') uses \\d+ which correctly matches one or more digits. Others match word characters, spaces, or non-digits.

3. Final Answer:

   regexp('abc123', '\\d+') -> Option A

4. Quick Check:

   Digits = \d+ pattern [OK]

Hint: Use \\d+ to match one or more digits [OK]

Common Mistakes:

• Using \w+ which matches letters and digits
• Using \s+ which matches spaces
• Using \D+ which matches non-digits

3. What is the output of the following MATLAB code?

str = 'cat bat rat';
matches = regexp(str, '\\b\w{3}\b', 'match');
disp(matches);

medium

A. ['cat', 'bat']

B. ['catbat', 'rat']

C. [] (empty cell array)

D. ['cat', 'bat', 'rat']

Solution

1. Step 1: Understand the pattern '\\b\\w{3}\\b'

   The pattern matches word boundaries \\b around exactly 3 word characters \\w{3}.

2. Step 2: Apply pattern to string

   The string has three 3-letter words: 'cat', 'bat', and 'rat'. All match the pattern.

3. Final Answer:

   ['cat', 'bat', 'rat'] -> Option D

4. Quick Check:

   Three 3-letter words matched [OK]

Hint: \b\w{3}\b matches exactly 3-letter words [OK]

Common Mistakes:

• Ignoring word boundaries and matching partial words
• Expecting concatenated matches
• Confusing \w with digits only

4. The following MATLAB code is intended to replace all digits in str with '#' but does not work as expected:

str = 'Room 101';
newStr = regexprep(str, '\d', '#');
disp(newStr);

What is the error and how to fix it?

medium

A. Pattern should be '\\d+' to replace all digits at once

B. No error; code works correctly

C. Pattern should be '\\D' to match digits

D. Use regexp instead of regexprep

Solution

1. Step 1: Analyze the regexprep usage

   The pattern '\\d' matches any single digit, and regexprep replaces all occurrences by default.

2. Step 2: Check the output

   Replacing each digit with '#' in 'Room 101' results in 'Room ###', which is expected.

3. Final Answer:

   No error; code works correctly -> Option B

4. Quick Check:

   regexprep replaces all matches by default [OK]

Hint: regexprep replaces all matches even with single-digit pattern [OK]

Common Mistakes:

• Thinking \d replaces only one digit total
• Using \D which matches non-digits
• Confusing regexp with regexprep

5. You have a cell array of strings: {'abc123', 'def456', 'ghi789'}. How do you extract only the numeric parts from each string using MATLAB regular expressions?

hard

A. Use regexp(cellArray, '\\d+', 'match')

B. Use regexp(cellArray, '\\w+', 'match')

C. Use regexprep(cellArray, '\\D', '')

D. Use regexprep(cellArray, '\\d+', '')

Solution

1. Step 1: Understand the goal

   We want to keep only digits, removing all non-digit characters.

2. Step 2: Choose the correct function and pattern

   regexprep replaces patterns. Using \\D matches non-digits, so replacing them with empty string keeps only digits.

3. Final Answer:

   Use regexprep(cellArray, '\\D', '') -> Option C

4. Quick Check:

   Remove non-digits with \D pattern [OK]

Hint: Replace non-digits (\D) with empty string to keep digits [OK]

Common Mistakes:

• Using regexp with 'match' returns cell arrays, not strings
• Replacing digits instead of non-digits
• Using \w+ which matches letters and digits