String prefix removal (${var#pattern}) in Bash Scripting - Deep Dive

Overview - String prefix removal (${var#pattern})

What is it?

String prefix removal using ${var#pattern} is a way in bash scripting to delete the shortest matching part of a string from the start. It looks at the value stored in a variable and removes a pattern from its beginning. This helps to quickly cut off unwanted parts without changing the original variable. It is a simple and fast method to manipulate text in scripts.

Why it matters

Without this feature, removing parts of strings would require more complex commands or external tools, making scripts slower and harder to read. It solves the problem of quickly cleaning or extracting parts of text data in automation tasks. This makes scripts more efficient and easier to maintain, especially when working with file names, paths, or user input.

Where it fits

Before learning this, you should understand basic bash variables and how to assign and use them. After mastering prefix removal, you can learn suffix removal (${var%pattern}), substring extraction, and pattern matching with wildcards. This fits into the broader topic of bash parameter expansion and string manipulation.

Mental Model

Core Idea

Removing the shortest matching pattern from the start of a string lets you quickly cut off unwanted beginnings in bash variables.

Think of it like...

It's like trimming the smallest piece of tape stuck at the start of a ribbon, leaving the rest untouched.

Variable value:  ┌────────────────────────┐
Pattern to remove:  ┌───┐
After removal:     └────────────────────────┘

Example:
${var#pattern}

Where 'pattern' matches the start part to cut off.

Build-Up - 7 Steps

1

Foundation

Understanding bash variables basics

Concept: Learn what bash variables are and how to store text in them.

In bash, you can store text in a variable like this: name="hello_world" you can then use it by prefixing with $: echo $name This prints: hello_world

Result

hello_world

Knowing how to store and access text in variables is the base for any string manipulation in bash.

2

Foundation

Basic string pattern matching in bash

Concept: Learn how bash uses wildcards like * and ? to match parts of strings.

The * wildcard matches any number of characters, and ? matches exactly one character. Example: pattern="hel*" This matches any string starting with 'hel', like 'hello' or 'help'.

Result

Pattern 'hel*' matches 'hello', 'help', 'helium', etc.

Understanding wildcards is key to using patterns in prefix removal.

3

Intermediate

Using ${var#pattern} for prefix removal

🤔Before reading on: do you think ${var#pattern} removes the longest or shortest match from the start? Commit to your answer.

Concept: Learn how ${var#pattern} removes the shortest matching prefix from a variable's value.

Given a variable: file="/home/user/docs/report.txt" Using: echo ${file#*/} removes the shortest match of '*/' from the start, which is '/'. So output is: home/user/docs/report.txt If pattern matches more, only the shortest part is removed.

Result

home/user/docs/report.txt

Knowing that # removes the shortest match helps you control exactly how much prefix is cut off.

4

Intermediate

Difference between # and ## in prefix removal

🤔Before reading on: does ## remove more or less than #? Commit to your answer.

Concept: Understand that ${var##pattern} removes the longest matching prefix, unlike ${var#pattern} which removes the shortest.

Using the same variable: file="/home/user/docs/report.txt" ${file##*/} removes the longest match of '*/' from the start, which is '/home/user/docs/'. Output: report.txt This is useful to get the file name only.

Result

report.txt

Recognizing the difference between shortest and longest match lets you pick the right tool for your string task.

5

Intermediate

Combining prefix removal with wildcards

Concept: Learn how to use wildcards inside the pattern to remove flexible prefixes.

You can use * inside the pattern to match any characters. Example: url="https://example.com/page" Remove protocol prefix: echo ${url#*://} Output: example.com/page This removes everything up to and including '://'.

Result

example.com/page

Using wildcards in patterns makes prefix removal powerful for varied inputs.

6

Advanced

Using prefix removal in scripts for path manipulation

🤔Before reading on: do you think prefix removal can replace external tools like cut or sed for paths? Commit to your answer.

Concept: Apply prefix removal to extract parts of file paths efficiently in scripts without external commands.

Example: path="/var/log/nginx/access.log" Get directory path: dir=${path%/*} Get file name: file=${path##*/} Print: echo "Dir: $dir" echo "File: $file" Output: Dir: /var/log/nginx File: access.log

Result

Dir: /var/log/nginx File: access.log

Using parameter expansion for path parts is faster and cleaner than calling external tools.

7

Expert

Performance and edge cases of prefix removal

🤔Before reading on: do you think prefix removal always returns a changed string? Commit to your answer.

Concept: Understand how prefix removal behaves when no pattern matches and its performance benefits over external commands.

If the pattern does not match the start, the original string is returned unchanged. Example: text="hello" echo ${text#z*} Output: hello Also, prefix removal is done inside the shell, so it is very fast compared to running external programs like sed or awk. Beware that patterns are glob-style, not regex, so some complex matches need different tools.

Result

hello

Knowing that no match returns original string prevents bugs, and understanding performance helps write efficient scripts.

Under the Hood

Bash performs parameter expansion internally by matching the pattern against the start of the variable's value using glob-style matching. It finds the shortest substring from the start that fits the pattern and removes it, returning the rest. This happens in the shell process without spawning new programs, making it very fast. The pattern matching uses shell wildcards, not regular expressions.

Why designed this way?

This feature was designed to provide a lightweight, fast way to manipulate strings directly in the shell. External tools like sed or awk were slower and more complex for simple tasks. Using glob patterns aligns with existing shell wildcard syntax, making it intuitive for users familiar with shell scripting. The choice of shortest match (#) and longest match (##) gives flexibility for different use cases.

┌───────────────┐
│ Variable value│
│ "abcde"      │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Pattern match │
│ "a*"         │
└──────┬────────┘
       │
       ▼
┌─────────────────────────────┐
│ Remove shortest prefix match │
│ Result: "bcde"              │
└─────────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does ${var#pattern} remove the longest matching prefix? Commit to yes or no.

Common Belief: Many think ${var#pattern} removes the longest matching prefix.

Reality: ${var#pattern} removes the shortest matching prefix; ${var##pattern} removes the longest.

Why it matters: Using # instead of ## when you want the longest match can cause incomplete removals and bugs in scripts.

Quick: Does prefix removal modify the original variable? Commit to yes or no.

Common Belief: Some believe ${var#pattern} changes the variable's stored value.

Reality: It only returns the modified string; the original variable remains unchanged unless reassigned.

Why it matters: Assuming the variable changes can lead to unexpected behavior if you reuse the variable without reassignment.

Quick: Is the pattern in ${var#pattern} a regular expression? Commit to yes or no.

Common Belief: Many think the pattern supports full regular expressions.

Reality: The pattern uses shell globbing wildcards (*, ?, []) not regex syntax.

Why it matters: Using regex syntax causes patterns to fail or behave unexpectedly, leading to incorrect string removals.

Quick: Does ${var#pattern} always remove something? Commit to yes or no.

Common Belief: Some assume it always removes a prefix.

Reality: If no match is found, the original string is returned unchanged.

Why it matters: Not checking for no-match cases can cause logic errors in scripts expecting a changed string.

Expert Zone

1

When stacking multiple prefix removals, remember each expansion is independent; chaining requires reassignment or nested expansions.

2

Patterns are matched against the entire string start, so partial matches inside the string won't be removed unless the pattern includes wildcards.

3

Using parameter expansion avoids spawning external processes, which is critical in performance-sensitive or large-scale scripts.

When NOT to use

Avoid using prefix removal when you need complex pattern matching beyond glob wildcards, such as full regular expressions or multi-line patterns. In those cases, use tools like sed, awk, or grep. Also, if you need to modify the variable permanently, remember to reassign the result.

Production Patterns

In production scripts, prefix removal is commonly used to extract file names from paths, remove URL schemes, or clean user input. It is often combined with suffix removal (${var%pattern}) for full path parsing. Experts use it to write concise, fast scripts that avoid external calls, improving reliability and speed.

Connections

Regular Expressions

Related but different pattern matching system

Understanding the difference between shell glob patterns and regex helps avoid confusion and choose the right tool for string matching.

Functional Programming - String Manipulation

Both involve transforming strings by removing or extracting parts

Seeing string prefix removal as a pure function that returns a new string without side effects aligns with functional programming principles.

Human Language Editing

Both involve removing prefixes to clarify meaning

Just like removing prefixes in words changes their meaning, removing string prefixes in scripts changes data meaning, showing a cross-domain pattern of simplification.

Common Pitfalls

#1 Expecting the variable to change without reassignment

Wrong approach:

file="/path/to/file.txt"
echo ${file#*/}
echo $file

Correct approach:

file="/path/to/file.txt"
file=${file#*/}
echo $file

Root cause:

Misunderstanding that parameter expansion returns a value but does not modify the variable automatically.

#2 Using regex syntax in the pattern

Wrong approach:

text="abc123"
echo ${text#^abc[0-9]+}

Correct approach:

text="abc123"
echo ${text#abc*}

Root cause:

Confusing shell glob patterns with regular expressions.

#3 Using # when ## is needed for longest match

Wrong approach:

path="/a/b/c/file"
echo ${path#*/}

Correct approach:

path="/a/b/c/file"
echo ${path##*/}

Root cause:

Not knowing the difference between shortest (#) and longest (##) prefix removal.

Key Takeaways

String prefix removal with ${var#pattern} cuts off the shortest matching start pattern from a variable's value without changing the variable itself.

Patterns use shell wildcards, not regular expressions, so understanding glob syntax is essential.

The difference between # and ## is crucial: # removes the shortest match, ## removes the longest.

This feature is fast and efficient because it runs inside the shell, avoiding external commands.

Always reassign the result if you want to change the variable's value permanently.

Practice

1. What does the Bash expression ${var#pattern} do?

easy

A. Removes the shortest matching prefix pattern from the variable's value

B. Removes the longest matching suffix pattern from the variable's value

C. Removes the shortest matching suffix pattern from the variable's value

D. Removes the longest matching prefix pattern from the variable's value

Solution

1. Step 1: Understand the syntax

   The syntax ${var#pattern} is used in Bash to remove a prefix pattern from the start of the variable's value.

2. Step 2: Identify shortest prefix removal

   The single # removes the shortest matching prefix, unlike ## which removes the longest.

3. Final Answer:

   Removes the shortest matching prefix pattern from the variable's value -> Option A

4. Quick Check:

   Prefix removal = shortest match = A [OK]

Hint: Single # removes shortest prefix, double ## removes longest [OK]

Common Mistakes:

• Confusing # with ## for prefix length
• Mixing prefix removal with suffix removal
• Thinking it removes suffix instead of prefix

2. Which of the following is the correct syntax to remove the shortest prefix matching abc from variable var in Bash?

easy

A. ${var#abc}

B. ${var##abc}

C. ${var%abc}

D. ${var%%abc}

Solution

1. Step 1: Recall prefix removal syntax

   Single # removes shortest prefix, double ## removes longest prefix. % and %% remove suffixes.

2. Step 2: Match the pattern

   To remove shortest prefix matching 'abc', use ${var#abc}.

3. Final Answer:

   ${var#abc} -> Option A

4. Quick Check:

   Shortest prefix removal = single # = B [OK]

Hint: Use single # for shortest prefix removal syntax [OK]

Common Mistakes:

• Using ## instead of # for shortest prefix
• Using % or %% which remove suffixes
• Confusing prefix and suffix removal syntax

3. Given var="folder/subfolder/file.txt", what is the output of echo ${var#*/}?

medium

A. folder/subfolder/file.txt

B. subfolder/file.txt

C. file.txt

D. folder

Solution

1. Step 1: Understand the pattern */

   The pattern */ matches the shortest prefix ending with a slash.

2. Step 2: Remove shortest prefix matching */

   From "folder/subfolder/file.txt", the shortest prefix matching */ is "folder/". Removing it leaves "subfolder/file.txt".

3. Final Answer:

   subfolder/file.txt -> Option B

4. Quick Check:

   Remove shortest prefix ending with / = subfolder/file.txt [OK]

Hint: Pattern */ removes up to first slash [OK]

Common Mistakes:

• Removing longest prefix with ## instead of #
• Expecting only filename instead of subfolder/file.txt
• Confusing prefix removal with suffix removal

4. Consider the script snippet:

var="abc123abc456"
echo ${var##abc*}

It outputs an empty line. Why?

medium

A. Because the pattern is invalid and causes an error

B. Because abc* matches the entire string as shortest prefix

C. Because abc* matches the longest prefix, so nothing remains

D. Because the variable is empty

Solution

1. Step 1: Analyze the pattern abc*

   The pattern abc* matches 'abc' followed by zero or more any characters.

2. Step 2: Understand longest prefix removal

   With ##, it removes the longest matching prefix. The * matches greedily as many characters as possible after 'abc', so the match covers the entire string.

3. Final Answer:

   Because abc* matches the longest prefix, so nothing remains -> Option C

4. Quick Check:

   Longest prefix with ## + greedy * = B [OK]

Hint: Wildcard * matches greedily with longest prefix ## [OK]

Common Mistakes:

• Confusing ## (longest) with # (shortest)
• Underestimating greedy nature of * in longest match
• Assuming pattern causes error or variable is empty

5. You have a variable path="folder/home/user/docs/report.txt". Which command removes only the first slash and everything before it, leaving home/user/docs/report.txt?

hard

A. echo ${path#/*}

B. echo ${path##/*}

C. echo ${path##*/}

D. echo ${path#*/}

Solution

1. Step 1: Understand the variable value

   The variable path is "folder/home/user/docs/report.txt".

2. Step 2: Analyze each option's pattern

   ${path#/*} removes shortest prefix matching slash followed by anything, but path doesn't start with /, so no match.
   ${path##/*} removes longest prefix matching slash-anything, no match.
   ${path##*/} removes longest prefix up to last slash, leaving only filename "report.txt".
   ${path#*/} removes shortest prefix up to first slash ("folder/"), leaving "home/user/docs/report.txt".

3. Step 3: Choose correct option

   To remove up to the first slash, use ${path#*/}.

4. Final Answer:

   echo ${path#*/} -> Option D

5. Quick Check:

   Remove shortest prefix up to first slash = D [OK]

Hint: Use #*/ to remove up to first slash only [OK]

Common Mistakes:

• Using ##*/ removes up to last slash (only filename)
• Using #/* or ##/* which don't match without leading slash
• Confusing prefix removal with suffix removal