String prefix removal (${var#pattern}) in Bash Scripting - Deep Dive

Overview - String prefix removal (${var#pattern})

What is it?

String prefix removal using ${var#pattern} is a way in bash scripting to delete the shortest matching part of a string from the start. It looks at the value stored in a variable and removes a pattern from its beginning. This helps to quickly cut off unwanted parts without changing the original variable. It is a simple and fast method to manipulate text in scripts.

Why it matters

Without this feature, removing parts of strings would require more complex commands or external tools, making scripts slower and harder to read. It solves the problem of quickly cleaning or extracting parts of text data in automation tasks. This makes scripts more efficient and easier to maintain, especially when working with file names, paths, or user input.

Where it fits

Before learning this, you should understand basic bash variables and how to assign and use them. After mastering prefix removal, you can learn suffix removal (${var%pattern}), substring extraction, and pattern matching with wildcards. This fits into the broader topic of bash parameter expansion and string manipulation.

Mental Model

Core Idea

Removing the shortest matching pattern from the start of a string lets you quickly cut off unwanted beginnings in bash variables.

Think of it like...

It's like trimming the smallest piece of tape stuck at the start of a ribbon, leaving the rest untouched.

Variable value:  ┌────────────────────────┐
Pattern to remove:  ┌───┐
After removal:     └────────────────────────┘

Example:
${var#pattern}

Where 'pattern' matches the start part to cut off.

Build-Up - 7 Steps

1

Foundation

Understanding bash variables basics

Concept: Learn what bash variables are and how to store text in them.

In bash, you can store text in a variable like this: name="hello_world" you can then use it by prefixing with $: echo $name This prints: hello_world

Result

hello_world

Knowing how to store and access text in variables is the base for any string manipulation in bash.

2

Foundation

Basic string pattern matching in bash

Concept: Learn how bash uses wildcards like * and ? to match parts of strings.

The * wildcard matches any number of characters, and ? matches exactly one character. Example: pattern="hel*" This matches any string starting with 'hel', like 'hello' or 'help'.

Result

Pattern 'hel*' matches 'hello', 'help', 'helium', etc.

Understanding wildcards is key to using patterns in prefix removal.

3

Intermediate

Using ${var#pattern} for prefix removal

🤔Before reading on: do you think ${var#pattern} removes the longest or shortest match from the start? Commit to your answer.

Concept: Learn how ${var#pattern} removes the shortest matching prefix from a variable's value.

Given a variable: file="/home/user/docs/report.txt" Using: echo ${file#*/} removes the shortest match of '*/' from the start, which is '/'. So output is: home/user/docs/report.txt If pattern matches more, only the shortest part is removed.

Result

home/user/docs/report.txt

Knowing that # removes the shortest match helps you control exactly how much prefix is cut off.

4

Intermediate

Difference between # and ## in prefix removal

🤔Before reading on: does ## remove more or less than #? Commit to your answer.

Concept: Understand that ${var##pattern} removes the longest matching prefix, unlike ${var#pattern} which removes the shortest.

Using the same variable: file="/home/user/docs/report.txt" ${file##*/} removes the longest match of '*/' from the start, which is '/home/user/docs/'. Output: report.txt This is useful to get the file name only.

Result

report.txt

Recognizing the difference between shortest and longest match lets you pick the right tool for your string task.

5

Intermediate

Combining prefix removal with wildcards

Concept: Learn how to use wildcards inside the pattern to remove flexible prefixes.

You can use * inside the pattern to match any characters. Example: url="https://example.com/page" Remove protocol prefix: echo ${url#*://} Output: example.com/page This removes everything up to and including '://'.

Result

example.com/page

Using wildcards in patterns makes prefix removal powerful for varied inputs.

6

Advanced

Using prefix removal in scripts for path manipulation

🤔Before reading on: do you think prefix removal can replace external tools like cut or sed for paths? Commit to your answer.

Concept: Apply prefix removal to extract parts of file paths efficiently in scripts without external commands.

Example: path="/var/log/nginx/access.log" Get directory path: dir=${path%/*} Get file name: file=${path##*/} Print: echo "Dir: $dir" echo "File: $file" Output: Dir: /var/log/nginx File: access.log

Result

Dir: /var/log/nginx File: access.log

Using parameter expansion for path parts is faster and cleaner than calling external tools.

7

Expert

Performance and edge cases of prefix removal

🤔Before reading on: do you think prefix removal always returns a changed string? Commit to your answer.

Concept: Understand how prefix removal behaves when no pattern matches and its performance benefits over external commands.

If the pattern does not match the start, the original string is returned unchanged. Example: text="hello" echo ${text#z*} Output: hello Also, prefix removal is done inside the shell, so it is very fast compared to running external programs like sed or awk. Beware that patterns are glob-style, not regex, so some complex matches need different tools.

Result

hello

Knowing that no match returns original string prevents bugs, and understanding performance helps write efficient scripts.

Under the Hood

Bash performs parameter expansion internally by matching the pattern against the start of the variable's value using glob-style matching. It finds the shortest substring from the start that fits the pattern and removes it, returning the rest. This happens in the shell process without spawning new programs, making it very fast. The pattern matching uses shell wildcards, not regular expressions.

Why designed this way?

This feature was designed to provide a lightweight, fast way to manipulate strings directly in the shell. External tools like sed or awk were slower and more complex for simple tasks. Using glob patterns aligns with existing shell wildcard syntax, making it intuitive for users familiar with shell scripting. The choice of shortest match (#) and longest match (##) gives flexibility for different use cases.

┌───────────────┐
│ Variable value│
│ "abcde"      │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Pattern match │
│ "a*"         │
└──────┬────────┘
       │
       ▼
┌─────────────────────────────┐
│ Remove shortest prefix match │
│ Result: "bcde"              │
└─────────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does ${var#pattern} remove the longest matching prefix? Commit to yes or no.

Common Belief: Many think ${var#pattern} removes the longest matching prefix.

Reality: ${var#pattern} removes the shortest matching prefix; ${var##pattern} removes the longest.

Why it matters: Using # instead of ## when you want the longest match can cause incomplete removals and bugs in scripts.

Quick: Does prefix removal modify the original variable? Commit to yes or no.

Common Belief: Some believe ${var#pattern} changes the variable's stored value.

Reality: It only returns the modified string; the original variable remains unchanged unless reassigned.

Why it matters: Assuming the variable changes can lead to unexpected behavior if you reuse the variable without reassignment.

Quick: Is the pattern in ${var#pattern} a regular expression? Commit to yes or no.

Common Belief: Many think the pattern supports full regular expressions.

Reality: The pattern uses shell globbing wildcards (*, ?, []) not regex syntax.

Why it matters: Using regex syntax causes patterns to fail or behave unexpectedly, leading to incorrect string removals.

Quick: Does ${var#pattern} always remove something? Commit to yes or no.

Common Belief: Some assume it always removes a prefix.

Reality: If no match is found, the original string is returned unchanged.

Why it matters: Not checking for no-match cases can cause logic errors in scripts expecting a changed string.

Expert Zone

1

When stacking multiple prefix removals, remember each expansion is independent; chaining requires reassignment or nested expansions.

2

Patterns are matched against the entire string start, so partial matches inside the string won't be removed unless the pattern includes wildcards.

3

Using parameter expansion avoids spawning external processes, which is critical in performance-sensitive or large-scale scripts.

When NOT to use

Avoid using prefix removal when you need complex pattern matching beyond glob wildcards, such as full regular expressions or multi-line patterns. In those cases, use tools like sed, awk, or grep. Also, if you need to modify the variable permanently, remember to reassign the result.

Production Patterns

In production scripts, prefix removal is commonly used to extract file names from paths, remove URL schemes, or clean user input. It is often combined with suffix removal (${var%pattern}) for full path parsing. Experts use it to write concise, fast scripts that avoid external calls, improving reliability and speed.

Connections

Regular Expressions

Related but different pattern matching system

Understanding the difference between shell glob patterns and regex helps avoid confusion and choose the right tool for string matching.

Functional Programming - String Manipulation

Both involve transforming strings by removing or extracting parts

Seeing string prefix removal as a pure function that returns a new string without side effects aligns with functional programming principles.

Human Language Editing

Both involve removing prefixes to clarify meaning

Just like removing prefixes in words changes their meaning, removing string prefixes in scripts changes data meaning, showing a cross-domain pattern of simplification.

Common Pitfalls

#1 Expecting the variable to change without reassignment

Wrong approach:

file="/path/to/file.txt"
echo ${file#*/}
echo $file

Correct approach:

file="/path/to/file.txt"
file=${file#*/}
echo $file

Root cause:

Misunderstanding that parameter expansion returns a value but does not modify the variable automatically.

#2 Using regex syntax in the pattern

Wrong approach:

text="abc123"
echo ${text#^abc[0-9]+}

Correct approach:

text="abc123"
echo ${text#abc*}

Root cause:

Confusing shell glob patterns with regular expressions.

#3 Using # when ## is needed for longest match

Wrong approach:

path="/a/b/c/file"
echo ${path#*/}

Correct approach:

path="/a/b/c/file"
echo ${path##*/}

Root cause:

Not knowing the difference between shortest (#) and longest (##) prefix removal.

Key Takeaways

String prefix removal with ${var#pattern} cuts off the shortest matching start pattern from a variable's value without changing the variable itself.

Patterns use shell wildcards, not regular expressions, so understanding glob syntax is essential.

The difference between # and ## is crucial: # removes the shortest match, ## removes the longest.

This feature is fast and efficient because it runs inside the shell, avoiding external commands.

Always reassign the result if you want to change the variable's value permanently.