Looping over files and directories in Bash Scripting - Deep Dive

Overview - Looping over files and directories

What is it?

Looping over files and directories means running a set of commands repeatedly for each file or folder in a location. In bash scripting, this helps automate tasks like checking, moving, or modifying many files without doing each one by hand. It uses simple loops that go through each item one by one. This makes managing files faster and less error-prone.

Why it matters

Without looping over files and directories, you would have to manually handle each file, which is slow and boring. Automating this saves time and reduces mistakes, especially when dealing with many files. It allows scripts to adapt to changing file lists, making your work more flexible and powerful. This is essential for tasks like backups, organizing files, or batch processing.

Where it fits

Before learning this, you should understand basic bash commands and how to write simple scripts. After mastering loops over files, you can learn advanced file handling, conditional processing, and automation workflows. This topic is a foundation for automating system tasks and managing data efficiently.

Mental Model

Core Idea

Looping over files and directories means repeating actions for each item in a folder automatically, like checking every book on a shelf one by one.

Think of it like...

Imagine you have a basket of apples and you want to check each apple for bruises. Instead of looking at all apples at once, you pick one apple, check it, then move to the next until all are checked. Looping over files is the same but with files instead of apples.

Folder
  ├─ file1
  ├─ file2
  ├─ dir1
  │    ├─ file3
  │    └─ file4
  └─ file5

Loop:
  for each item in folder:
    do something with item
    move to next item
  end

Build-Up - 7 Steps

1

Foundation

Basic for loop syntax in bash

Concept: Learn the simplest way to write a loop that repeats commands a fixed number of times.

In bash, a basic for loop looks like this: for i in 1 2 3 do echo "Number $i" done This prints numbers 1, 2, and 3 one by one.

Result

Number 1 Number 2 Number 3

Understanding the basic for loop syntax is the first step to automating repeated tasks in bash.

2

Foundation

Using wildcards to list files

Concept: Use wildcards like * to select multiple files or directories in a folder.

The * wildcard matches all files and folders in the current directory. Example: ls * lists all items. You can use patterns like *.txt to match only text files.

Result

file1.txt file2.txt image.png docs script.sh

Wildcards let you select groups of files easily, which is essential for looping over them.

3

Intermediate

Looping over files with for and wildcards

🤔Before reading on: do you think 'for file in *' loops over files only or files and directories? Commit to your answer.

Concept: Combine for loops with wildcards to process each file or directory in a folder.

Example: for file in * do echo "Found: $file" done This loop prints the name of every file and directory in the current folder.

Result

Found: file1.txt Found: file2.txt Found: image.png Found: docs Found: script.sh

Knowing that * matches both files and directories helps you handle all items or filter them as needed.

4

Intermediate

Filtering only files or directories

🤔Before reading on: can you guess how to check if an item is a file or directory inside a loop? Commit to your answer.

Concept: Use test commands inside the loop to act only on files or only on directories.

Inside the loop, use -f to check for files and -d for directories. Example: for item in * do if [ -f "$item" ]; then echo "$item is a file" elif [ -d "$item" ]; then echo "$item is a directory" fi done

Result

file1.txt is a file file2.txt is a file image.png is a file docs is a directory script.sh is a file

Filtering lets you target your commands precisely, avoiding errors or unwanted actions.

5

Intermediate

Handling filenames with spaces safely

🤔Before reading on: do you think 'for file in *' handles filenames with spaces correctly? Commit to your answer.

Concept: Learn how to loop safely over files with spaces or special characters in their names.

Using for file in * splits on spaces, breaking filenames with spaces. Better approach: while IFS= read -r -d '' file; do echo "File: $file" done < <(find . -maxdepth 1 -print0) This uses find with -print0 and reads filenames safely.

Result

File: ./file1.txt File: ./my file with spaces.txt File: ./docs File: ./script.sh

Handling spaces correctly prevents bugs and data loss in scripts working with real-world files.

6

Advanced

Recursively looping through directories

🤔Before reading on: do you think a simple for loop with * goes into subdirectories automatically? Commit to your answer.

Concept: Use recursion or find command to process files inside subfolders as well.

Simple * only lists current folder. To loop recursively: find . -type f | while read -r file; do echo "File: $file" done This finds all files in current and subfolders and loops over them.

Result

File: ./file1.txt File: ./docs/readme.md File: ./docs/manual.pdf File: ./script.sh

Recursion lets scripts handle complex folder trees, essential for backups or large data processing.

7

Expert

Avoiding common pitfalls with globbing and quoting

🤔Before reading on: do you think quoting variables inside loops is optional or necessary? Commit to your answer.

Concept: Understand how shell expands wildcards (globbing) and why quoting variables prevents bugs.

Globbing expands * before the loop runs. If you forget quotes: for file in *; do echo $file done Files with spaces split incorrectly. Correct: for file in *; do echo "$file" done Always quote variables to preserve exact filenames.

Result

Correctly prints filenames even with spaces, avoiding broken output.

Quoting is a simple habit that prevents subtle bugs and data corruption in scripts.

Under the Hood

When you write 'for file in *', the shell first expands the * wildcard into a list of matching filenames. Then the loop runs once for each filename. The shell treats filenames as strings, so if they contain spaces and are not quoted, the shell splits them into separate words, causing errors. Commands like 'find' generate lists of files differently, allowing safer handling of complex names. The shell's parsing and expansion rules control how loops receive and process file names.

Why designed this way?

The shell uses globbing (wildcard expansion) to let users easily select files without typing each name. This design is simple and fast but assumes filenames are simple strings. Quoting and special commands like 'find' were added later to handle real-world complexities like spaces and recursion. This balance keeps the shell lightweight but requires careful scripting to avoid pitfalls.

User writes: for file in *
       ↓
Shell expands * → file1.txt file2.txt my file.txt
       ↓
Loop runs:
  Iteration 1: file = file1.txt
  Iteration 2: file = file2.txt
  Iteration 3: file = my
  Iteration 4: file = file.txt  (wrong split)

Correct with quotes:
  Iteration 3: file = "my file.txt" (one item)

Myth Busters - 4 Common Misconceptions

Quick: Does 'for file in *' loop only over files, or also directories? Commit to your answer.

Common Belief: The loop 'for file in *' only processes files, not directories.

Reality: The * wildcard matches both files and directories, so the loop processes all items in the folder.

Why it matters: Scripts may fail or behave unexpectedly if they assume only files are processed, causing errors when directories appear.

Quick: Can you safely loop over filenames with spaces using 'for file in *'? Commit to your answer.

Common Belief: The for loop with * handles filenames with spaces correctly without extra care.

Reality: Without proper quoting or special handling, filenames with spaces break the loop, splitting one filename into multiple parts.

Why it matters: This causes bugs, data loss, or incorrect processing in scripts working with real files.

Quick: Does a simple for loop with * go into subdirectories automatically? Commit to your answer.

Common Belief: Using 'for file in *' loops through all files in current and all subdirectories automatically.

Reality: The * wildcard only matches items in the current directory; it does not recurse into subfolders.

Why it matters: Scripts that expect recursive processing will miss files deeper in folders, causing incomplete results.

Quick: Is quoting variables inside loops optional? Commit to your answer.

Common Belief: Quoting variables like "$file" inside loops is optional and mostly stylistic.

Reality: Quoting variables is necessary to preserve filenames exactly, especially those with spaces or special characters.

Why it matters: Not quoting leads to broken scripts and hard-to-find bugs when filenames contain spaces or unusual characters.

Expert Zone

1

Globbing happens before the loop runs, so the list of files is fixed at loop start; changes during the loop won't affect iteration.

2

Using 'find' with -print0 and 'read -d '' ' handles all filenames safely, including those with newlines, which normal loops can't handle.

3

Looping over files with spaces requires careful use of IFS (Internal Field Separator) and quoting to avoid splitting filenames incorrectly.

When NOT to use

Simple for loops with * are not suitable for filenames with newlines or very complex names; use 'find' with null-separated output instead. Also, for very large directories, 'find' is more efficient and safer. Avoid loops when a single command can process multiple files at once (like 'xargs' or built-in commands).

Production Patterns

In real systems, scripts often combine 'find' with while-read loops for safe recursive processing. They also use functions to encapsulate file handling and trap errors for robustness. Batch renaming, backups, and log processing commonly use these patterns with careful quoting and error checking.

Connections

Recursion in Computer Science

Looping recursively through directories builds on the idea of recursion, where a function calls itself to handle nested structures.

Understanding recursion helps grasp how scripts explore folder trees deeply and systematically.

Regular Expressions

Wildcards in shell globbing are simpler cousins of regular expressions used for pattern matching.

Knowing regex helps create more precise file selection patterns beyond basic wildcards.

Assembly Line in Manufacturing

Looping over files is like an assembly line where each item is processed step-by-step automatically.

Seeing loops as automation pipelines clarifies how scripts save time and reduce human error.

Common Pitfalls

#1 Loop breaks on filenames with spaces

Wrong approach:

for file in *
do
echo $file
done

Correct approach:

for file in *
do
echo "$file"
done

Root cause:

Not quoting $file causes the shell to split filenames on spaces, breaking the loop.

#2 Assuming loop processes only files, not directories

Wrong approach:

for file in *
do
# process as file without checking
cat "$file"
done

Correct approach:

for item in *
do
if [ -f "$item" ]; then
cat "$item"
fi
done

Root cause:

Wildcard * matches both files and directories; ignoring this causes errors when directories are processed as files.

#3 Expecting recursive processing with simple for loop

Wrong approach:

for file in *
do
echo "$file"
done

Correct approach:

find . -type f | while read -r file; do
echo "$file"
done

Root cause:

The * wildcard only matches current directory; recursion requires commands like find.

Key Takeaways

Looping over files and directories automates repetitive tasks, saving time and reducing errors.

The shell expands wildcards before loops run, so understanding globbing is key to correct file selection.

Always quote variables holding filenames to handle spaces and special characters safely.

Simple loops do not recurse into subdirectories; use 'find' or recursion for deep folder processing.

Mastering these concepts builds a strong foundation for powerful and reliable bash scripting.

Practice

1. What does the following bash loop do?
for file in *; do echo "$file"; done

easy

A. It runs the echo command once without any output.

B. It deletes all files in the current folder.

C. It lists only directories in the current folder.

D. It prints the names of all files and directories in the current folder.

Solution

1. Step 1: Understand the loop variable and pattern

   The loop uses * which matches all files and directories in the current folder.

2. Step 2: Analyze the command inside the loop

   Each matched item is stored in file and printed with echo.

3. Final Answer:

   It prints the names of all files and directories in the current folder. -> Option D

4. Quick Check:

   Loop over * prints all names [OK]

Hint: Remember * matches all files and folders [OK]

Common Mistakes:

• Thinking * matches only files, not directories
• Assuming the loop deletes files
• Believing echo runs only once

2. Which of the following is the correct syntax to loop over all directories only in the current folder?

easy

A. for dir in *; do if [ -f "$dir" ]; then echo "$dir"; fi; done

B. for dir in *; do echo "$dir"; done

C. for dir in */; do echo "$dir"; done

D. for dir in /*; do echo "$dir"; done

Solution

1. Step 1: Understand the pattern for directories

   The pattern */ matches only directories in the current folder because directories end with a slash.

2. Step 2: Check the loop and echo command

   The loop iterates over directories and prints their names correctly.

3. Final Answer:

   for dir in */; do echo "$dir"; done -> Option C

4. Quick Check:

   */ matches directories only [OK]

Hint: Use */ pattern to match directories only [OK]

Common Mistakes:

• Using * matches files and directories, not only directories
• Using -f to check for directories instead of -d
• Using absolute path /* instead of current folder

3. What will be the output of this script if the current folder contains files: file1.txt, file2.txt and a directory docs/?

for item in *; do
  if [ -f "$item" ]; then
    echo "File: $item"
  else
    echo "Not a file: $item"
  fi
done

medium

A. File: file1.txt
File: file2.txt
Not a file: docs

B. File: file1.txt
File: file2.txt
File: docs

C. Not a file: file1.txt
Not a file: file2.txt
Not a file: docs

D. File: file1.txt
Not a file: file2.txt
Not a file: docs

Solution

1. Step 1: Identify file and directory checks

   The script uses [ -f "$item" ] to check if the item is a file.

2. Step 2: Apply checks to each item

   file1.txt and file2.txt are files, so print "File: ..."; docs is a directory, so print "Not a file: docs".

3. Final Answer:

   File: file1.txt File: file2.txt Not a file: docs -> Option A

4. Quick Check:

   -f true for files only [OK]

Hint: Use -f to test files, else it's not a file [OK]

Common Mistakes:

• Assuming directories pass -f test
• Confusing -f with -d
• Expecting directory names with trailing slash

4. Find the error in this script that tries to print only directories:

for item in *; do
  if [ -f "$item" ]; then
    echo "$item"
  fi
done

medium

A. The echo command is missing quotes around $item.

B. It uses -f to check files instead of -d for directories.

C. The loop pattern * does not match any files.

D. The script will print directories correctly.

Solution

1. Step 1: Understand the test used

   The script uses [ -f "$item" ] which checks if the item is a file, not a directory.

2. Step 2: Identify correct test for directories

   To check directories, [ -d "$item" ] should be used instead.

3. Final Answer:

   It uses -f to check files instead of -d for directories. -> Option B

4. Quick Check:

   Use -d to test directories [OK]

Hint: Use -d to check directories, not -f [OK]

Common Mistakes:

• Confusing -f and -d tests
• Assuming * matches only directories
• Ignoring quotes around variables

5. You want to count how many files (not directories) are in the current folder using a bash loop. Which script correctly does this?

hard

A. count=0
for item in *; do
if [ -f "$item" ]; then
count=$((count + 1))
fi
done
echo $count

B. count=0
for item in */; do
count=$((count + 1))
done
echo $count

C. count=0
for item in *; do
if [ -d "$item" ]; then
count=$((count + 1))
fi
done
echo $count

D. count=0
for item in *; do
count=$((count + 1))
done
echo $count

Solution

1. Step 1: Understand the goal

   We want to count files only, so we must check each item with [ -f "$item" ].

2. Step 2: Check each option

   count=0 for item in *; do if [ -f "$item" ]; then count=$((count + 1)) fi done echo $count correctly loops over all items, increments count only if item is a file, then prints count.

3. Final Answer:

   count=0 for item in *; do if [ -f "$item" ]; then count=$((count + 1)) fi done echo $count -> Option A

4. Quick Check:

   Use -f test and increment count [OK]

Hint: Increment count only if -f test passes [OK]

Common Mistakes:

• Counting directories instead of files
• Counting all items without checks
• Using */ pattern which matches directories only