String suffix removal (${var%pattern}) in Bash Scripting - Deep Dive

Overview - String suffix removal (${var%pattern})

What is it?

String suffix removal using ${var%pattern} is a way in bash scripting to delete the shortest matching part of a pattern from the end of a string stored in a variable. It helps you quickly cut off unwanted endings without changing the original string. This is done directly in the shell without needing extra commands.

Why it matters

Without this feature, removing parts from the end of strings would require more complex commands or external tools, making scripts slower and harder to read. It simplifies text processing tasks like trimming file extensions or paths, which are common in automation and scripting. This makes scripts cleaner, faster, and easier to maintain.

Where it fits

Before learning this, you should understand basic bash variables and string handling. After mastering suffix removal, you can learn prefix removal (${var#pattern}), substring extraction, and advanced pattern matching in bash scripting.

Mental Model

Core Idea

You tell bash to cut off the shortest matching ending pattern from a string variable using ${var%pattern}.

Think of it like...

Imagine you have a ribbon with a tag at the end, and you want to cut off just the tag without touching the rest. The pattern is like the shape of the tag you want to remove from the ribbon's end.

String:  [HelloWorld.txt]
Pattern:          [*.txt]
Result:  [HelloWorld]

Build-Up - 7 Steps

1

Foundation

Understanding Bash Variables

Concept: Learn what bash variables are and how to store strings in them.

In bash, you can store text in variables like this: filename="HelloWorld.txt" echo $filename This prints the string stored in filename.

Result

HelloWorld.txt

Knowing how to store and access strings in variables is the base for manipulating them with suffix removal.

2

Foundation

Basic String Patterns in Bash

Concept: Learn how bash uses patterns with wildcards like * to match parts of strings.

The * wildcard matches any number of characters. For example: pattern="*.txt" matches any string ending with .txt You can use this in commands like: ls *.txt which lists all files ending with .txt.

Result

Lists files ending with .txt in the current directory

Understanding patterns with * is essential because suffix removal uses these patterns to decide what to cut off.

3

Intermediate

Using ${var%pattern} for Suffix Removal

Concept: Learn how to remove the shortest matching suffix pattern from a variable's value.

Given a variable: filename="report.pdf" You can remove the shortest suffix matching .pdf like this: base=${filename%.pdf} echo $base This prints 'report' by cutting off '.pdf' from the end.

Result

report

This shows how ${var%pattern} directly trims endings without extra commands, making scripts simpler.

4

Intermediate

Difference Between % and %% Operators

🤔Before reading on: do you think ${var%pattern} and ${var%%pattern} remove the same part or different parts of the suffix? Commit to your answer.

Concept: Learn the difference between removing the shortest vs longest matching suffix.

Using one % removes the shortest match from the end: filename="archive.tar.gz" ${filename%.gz} # removes '.gz' only Using two %% removes the longest match: ${filename%%.gz} # tries to remove longest match ending with '.gz', but here same as % More useful example: filename="file.backup.tar.gz" ${filename%.tar.gz} # removes '.tar.gz' ${filename%%.gz} # removes '.backup.tar.gz' because it matches longest suffix ending with '.gz'

Result

Short % removes shortest suffix; double %% removes longest suffix

Knowing shortest vs longest removal helps avoid bugs when suffixes have multiple parts.

5

Intermediate

Combining Suffix Removal with Other String Operations

🤔Before reading on: do you think you can chain suffix removal with prefix removal in one expression? Commit to your answer.

Concept: Learn how to combine suffix removal with prefix removal for complex string trimming.

You can combine suffix and prefix removal like this: filepath="/home/user/document.txt" filename=${filepath##*/} # removes longest prefix ending with '/', result: 'document.txt' base=${filename%.txt} # removes shortest suffix '.txt', result: 'document' echo $base This extracts the filename without path and extension.

Result

document

Combining these operations lets you extract or clean parts of strings efficiently in scripts.

6

Advanced

Using Patterns with Wildcards in Suffix Removal

🤔Before reading on: do you think you can use wildcards like * inside the pattern for suffix removal? Commit to your answer.

Concept: Learn how to use wildcards in the pattern to remove variable suffixes.

You can use * to match variable parts: filename="backup_2023_06_01.tar.gz" base=${filename%_*.tar.gz} echo $base This removes the shortest suffix matching '_*.tar.gz', so it cuts off '_06_01.tar.gz', leaving 'backup_2023'.

Result

backup_2023

Using wildcards in patterns makes suffix removal flexible for many filename or string formats.

7

Expert

Performance and Limitations of Suffix Removal

🤔Before reading on: do you think suffix removal is slower than external commands like sed or awk? Commit to your answer.

Concept: Understand the efficiency and edge cases of suffix removal in bash.

Suffix removal is a shell builtin operation, so it is very fast compared to calling external tools. However, it only supports glob patterns, not full regular expressions. Complex patterns or multi-line strings cannot be handled. Also, if the pattern does not match, the original string is returned unchanged. Example: filename="data.csv" base=${filename%.txt} echo $base Output is 'data.csv' because '.txt' suffix not found.

Result

data.csv

Knowing these limits helps choose the right tool: suffix removal for simple patterns and speed, external tools for complex text processing.

Under the Hood

Bash performs suffix removal by matching the pattern against the end of the string stored in the variable using shell globbing rules. It scans from the right side to find the shortest (or longest with %% operator) substring that fits the pattern and removes it. This happens internally in the shell parser before the command runs, so no external process is spawned.

Why designed this way?

This feature was designed to provide fast, simple string manipulation directly in the shell without overhead. Using glob patterns instead of full regex keeps it lightweight and easy to implement. It fits the Unix philosophy of small, composable tools and avoids the cost of calling external programs for common tasks.

Variable value: [filename = "archive.tar.gz"]

Suffix removal process:

  +-----------------------------+
  | Bash shell receives ${filename%.gz} |
  +-------------+---------------+
                |
                v
  +-----------------------------+
  | Match shortest suffix '.gz' |
  +-------------+---------------+
                |
                v
  +-----------------------------+
  | Remove '.gz' from end        |
  +-------------+---------------+
                |
                v
  +-----------------------------+
  | Result: 'archive.tar'        |
  +-----------------------------+

Myth Busters - 4 Common Misconceptions

Quick: Does ${var%pattern} remove the longest matching suffix or the shortest? Commit to your answer.

Common Belief: Many think ${var%pattern} removes the longest matching suffix from the string.

Reality: It actually removes the shortest matching suffix. To remove the longest, you must use ${var%%pattern}.

Why it matters: Using the wrong operator can lead to unexpected string results and bugs in scripts, especially with complex suffixes.

Quick: If the pattern does not match the string's suffix, does ${var%pattern} change the string? Commit to your answer.

Common Belief: Some believe that if the pattern doesn't match, the string will be altered or truncated anyway.

Reality: If the pattern does not match, the original string is returned unchanged.

Why it matters: Assuming the string changes can cause logic errors or data loss in scripts.

Quick: Can ${var%pattern} use full regular expressions for pattern matching? Commit to your answer.

Common Belief: People often think the pattern supports full regular expressions like grep or sed.

Reality: The pattern uses shell globbing (wildcards like *, ?, []) only, not full regex.

Why it matters: Expecting regex can lead to incorrect patterns and failed string manipulations.

Quick: Does ${var%pattern} modify the original variable's value? Commit to your answer.

Common Belief: Some think that using ${var%pattern} changes the variable's stored value automatically.

Reality: It only returns the modified string; the original variable remains unchanged unless reassigned.

Why it matters: Not reassigning can cause confusion when the original string is still used later.

Expert Zone

1

Suffix removal patterns are matched using globbing rules, so character classes and ranges can be used for precise matching.

2

When chaining multiple string operations, the order matters because each step works on the result of the previous one.

3

In some shells, parameter expansion can be combined with default values and substring extraction for powerful one-liners.

When NOT to use

Avoid using suffix removal when you need full regular expression support or complex multi-line text processing. Instead, use tools like sed, awk, or Perl for those cases.

Production Patterns

In production scripts, suffix removal is commonly used to strip file extensions, extract base filenames, or clean up paths quickly. It is often combined with prefix removal and conditional checks to handle file processing pipelines efficiently.

Connections

Regular Expressions

Suffix removal uses simpler glob patterns, which are a subset of regular expressions.

Understanding glob patterns helps bridge to learning regular expressions for more advanced text matching.

Unix Philosophy

Suffix removal embodies the Unix philosophy of doing one simple task efficiently within the shell.

Recognizing this helps appreciate why shell builtins avoid complexity and external dependencies.

Human Language Editing

Suffix removal is like trimming suffixes from words in language to get root forms.

This connection shows how pattern-based trimming is a universal concept across computing and linguistics.

Common Pitfalls

#1 Expecting ${var%pattern} to remove the longest suffix instead of the shortest.

Wrong approach:

filename="file.backup.tar.gz"
base=${filename%.gz}
echo $base # outputs 'file.backup.tar' but user expects 'file'

Correct approach:

filename="file.backup.tar.gz"
base=${filename%%.gz}
echo $base # outputs 'file.backup.tar' but still not 'file', correct pattern needed

Root cause:

Confusing % (shortest) with %% (longest) and misunderstanding pattern matching leads to wrong results.

#2 Using a pattern that does not match the suffix, expecting the string to change.

Wrong approach:

filename="document.txt"
base=${filename%.pdf}
echo $base # outputs 'document.txt' unchanged

Correct approach:

filename="document.txt"
base=${filename%.txt}
echo $base # outputs 'document'

Root cause:

Not verifying the pattern matches the actual suffix causes no change and confusion.

#3 Assuming ${var%pattern} modifies the variable without reassignment.

Wrong approach:

filename="image.png"
${filename%.png}
echo $filename # still 'image.png'

Correct approach:

filename="image.png"
filename=${filename%.png}
echo $filename # now 'image'

Root cause:

Not understanding parameter expansion returns a value but does not change the variable automatically.

Key Takeaways

The ${var%pattern} syntax removes the shortest matching suffix pattern from a string variable in bash.

It uses shell globbing patterns, not full regular expressions, so patterns must be crafted accordingly.

This operation is fast and built into the shell, making it ideal for simple string trimming tasks in scripts.

Understanding the difference between % and %% operators prevents common bugs in suffix removal.

Suffix removal does not change the original variable unless you explicitly assign the result back.

Practice

1. What does the Bash expression ${filename%.txt} do to the variable filename?

easy

A. Removes the shortest prefix matching '.txt' from the start of filename

B. Removes the longest suffix matching '.txt' from the end of filename

C. Removes the shortest suffix matching '.txt' from the end of filename

D. Removes the longest prefix matching '.txt' from the start of filename

Solution

1. Step 1: Understand the % operator in Bash

   The single % operator removes the shortest matching suffix pattern from the variable's value.

2. Step 2: Apply to the pattern '.txt'

   It removes '.txt' only if it appears at the end of the string, trimming that suffix.

3. Final Answer:

   Removes the shortest suffix matching '.txt' from the end of filename -> Option C

4. Quick Check:

   ${var%pattern} removes shortest suffix [OK]

Hint: Single % removes shortest suffix, double %% removes longest [OK]

Common Mistakes:

• Confusing % with prefix removal operators
• Using %% instead of %
• Thinking it removes prefix instead of suffix

2. Which of the following is the correct syntax to remove the shortest suffix '.log' from the variable file in Bash?

easy

A. ${file%%.log}

B. ${file%.log}

C. ${file#log}

D. ${file%log}

Solution

1. Step 1: Identify suffix removal syntax

   To remove the shortest suffix matching a pattern, use single % with the pattern including the dot: %.log

2. Step 2: Check each option

   A uses %% which removes longest suffix, C uses # which removes prefix, D misses the dot before 'log'. Only B is correct syntax.

3. Final Answer:

   ${file%.log} -> Option B

4. Quick Check:

   Use single % with pattern for shortest suffix removal [OK]

Hint: Use single % with exact suffix pattern to remove shortest suffix [OK]

Common Mistakes:

• Using %% instead of %
• Confusing % with # for prefix removal
• Omitting the dot in the pattern

3. Given the Bash code:

filename="report.final.txt"
echo ${filename%.txt}

What is the output?

medium

A. report.final.tx

B. report

C. report.final.txt

D. report.final

Solution

1. Step 1: Understand the variable value

   The variable filename contains 'report.final.txt'.

2. Step 2: Apply ${filename%.txt}

   This removes the shortest suffix matching '.txt' from the end, leaving 'report.final'.

3. Final Answer:

   report.final -> Option D

4. Quick Check:

   Suffix '.txt' removed once from end [OK]

Hint: Remove suffix by matching pattern at string end with % [OK]

Common Mistakes:

• Removing more than '.txt' suffix
• Confusing prefix removal with suffix removal
• Expecting the full string unchanged

4. Consider this Bash snippet:

file="archive.tar.gz"
echo ${file%*.gz}

It outputs archive.tar. You want to remove the entire suffix .tar.gz. What is the fix?

medium

A. Use ${file%%*.gz} to remove the longest matching suffix

B. Use ${file%.gz} to remove only '.gz' suffix

C. Use ${file#*.tar.gz} to remove prefix instead

D. Use ${file%.tar} to remove only '.tar' suffix

Solution

1. Step 1: Understand why ${file%*.gz} removes only '.gz'

   Single % removes shortest suffix matching *.gz pattern, so it matches '.gz' (with * matching empty).

2. Step 2: Use %% to remove longest suffix

   Double %% removes longest matching suffix, so ${file%%*.gz} removes entire '.tar.gz'.

3. Final Answer:

   Use ${file%%*.gz} to remove the longest matching suffix -> Option A

4. Quick Check:

   %% removes longest suffix, % removes shortest [OK]

Hint: Use %% to remove longest suffix pattern [OK]

Common Mistakes:

• Using single % when longest suffix removal needed
• Confusing prefix removal (#) with suffix removal (%)
• Trying to remove partial suffix only

5. You have a list of filenames:

files=("data1.csv" "data2.csv" "notes.txt" "summary.csv")

You want to remove the '.csv' suffix only from files that end with '.csv' using Bash parameter expansion. Which command correctly prints the filenames with '.csv' removed where applicable?

hard

A. for f in "${files[@]}"; do echo ${f%.csv}; done

B. for f in "${files[@]}"; do echo ${f%%csv}; done

C. for f in "${files[@]}"; do echo ${f%csv}; done

D. for f in "${files[@]}"; do echo ${f#%.csv}; done

Solution

1. Step 1: Understand the goal

   Remove '.csv' suffix only if it exists, leave other filenames unchanged.

2. Step 2: Check each option's effect

   A uses single % with '.csv' pattern, removing suffix if present. B uses %%csv missing the dot, incorrect. C misses the dot, incorrect. D uses # which removes prefix, wrong direction.

3. Step 3: Choose best option

   for f in "${files[@]}"; do echo ${f%.csv}; done is correct and simplest for shortest suffix removal of '.csv'.

4. Final Answer:

   for f in "${files[@]}"; do echo ${f%.csv}; done -> Option A

5. Quick Check:

   Use ${var%.csv} to remove '.csv' suffix if present [OK]

Hint: Use ${var%.ext} in loop to trim suffix conditionally [OK]

Common Mistakes:

• Using prefix removal (#) instead of suffix removal (%)
• Omitting the dot in suffix pattern
• Using %% unnecessarily when % suffices