Process substitution (<() and >()) in Bash Scripting - Deep Dive

Overview - Process substitution (<() and >())

What is it?

Process substitution is a feature in bash scripting that lets you use the output or input of a command as if it were a file. It uses special syntax like <() for input and >() for output. This allows commands that expect file names to work with dynamic data streams without creating temporary files. It makes scripts cleaner and more efficient by connecting commands directly.

Why it matters

Without process substitution, you would often need to create temporary files to hold command outputs before passing them to other commands. This adds clutter, slows down scripts, and risks leftover files. Process substitution solves this by letting commands communicate directly through invisible pipes, making automation faster, safer, and easier to manage.

Where it fits

Before learning process substitution, you should understand basic bash commands, pipes, and redirection. After mastering it, you can explore advanced bash scripting techniques like command substitution, coprocesses, and complex data processing pipelines.

Mental Model

Core Idea

Process substitution lets you treat the output or input of a command as a temporary file, enabling seamless data flow between commands that expect file names.

Think of it like...

It's like having a magic folder where you can drop a live video feed or a live microphone input, and any program can open that folder to watch or listen as if it were a normal file.

Command A producing data ──> <() ──> Command B reading as a file

Command B writing to >() ──> Command A consuming data

┌─────────────┐       ┌─────────────┐
│ Command A   │       │ Command B   │
│ (producer)  │       │ (consumer)  │
└─────┬───────┘       └─────┬───────┘
      │                     │
      │ <() or >()          │
      └─────> Temporary pipe or FIFO
            (looks like a file)

Build-Up - 7 Steps

1

Foundation

Basics of Command Output Redirection

Concept: Learn how to redirect command output to files and pipes.

In bash, you can send the output of a command to a file using >, like `ls > files.txt`. You can also send output to another command using a pipe `|`, like `ls | grep txt`. These are the basic ways to move data around in scripts.

Result

The output of the first command is saved in a file or passed to the next command.

Understanding basic redirection is essential because process substitution builds on the idea of redirecting command outputs and inputs.

2

Foundation

Understanding File Descriptors and Pipes

Concept: Learn what file descriptors and pipes are in bash.

Every command reads input and writes output through file descriptors: 0 for input, 1 for output, and 2 for errors. Pipes connect the output of one command to the input of another, creating a data stream between them.

Result

Commands can communicate by passing data streams instead of files.

Knowing how file descriptors and pipes work helps you grasp how process substitution creates temporary files that are actually pipes.

3

Intermediate

Using <() for Input Process Substitution

🤔Before reading on: do you think <() creates a real file or a temporary pipe? Commit to your answer.

Concept: Learn how <() lets a command read the output of another command as if it were a file.

Syntax: `command1 <(command2)` runs command2 and provides its output as a file-like input to command1. For example, `diff <(ls dir1) <(ls dir2)` compares directory listings without temporary files.

Result

diff compares the outputs of the two ls commands as if they were files.

Understanding that <() creates a temporary pipe disguised as a file lets you use commands that require file arguments with dynamic data.

4

Intermediate

Using >() for Output Process Substitution

🤔Before reading on: does >() send output to a real file or a pipe? Commit to your answer.

Concept: Learn how >() lets a command write output to another command's input as if writing to a file.

Syntax: `command1 >(command2)` runs command2 and sends command1's output to it. For example, `tee >(grep error > errors.log)` sends output to grep and also to the screen.

Result

Output is split: one copy goes to grep and errors.log, another to the terminal.

Knowing >() creates a pipe that looks like a file lets you redirect output to commands that expect file names.

5

Intermediate

Combining Multiple Process Substitutions

🤔Before reading on: can you use multiple <() or >() in one command? Commit to your answer.

Concept: Learn how to use several process substitutions together for complex data flows.

Example: `diff <(sort file1) <(sort file2)` compares sorted versions of two files without creating temporary files. You can chain multiple <() or >() in one command to handle multiple streams.

Result

diff shows differences between sorted files efficiently.

Combining process substitutions allows powerful one-liner scripts that handle multiple data streams cleanly.

6

Advanced

Limitations and Compatibility of Process Substitution

🤔Before reading on: do you think process substitution works on all Unix systems? Commit to your answer.

Concept: Understand where process substitution works and its limitations.

Process substitution requires support for named pipes or /dev/fd. It works on bash and zsh but not all shells. Some commands may not accept the special file names created. Also, >() is less common and can be tricky with some commands.

Result

Scripts using process substitution may fail on incompatible shells or systems.

Knowing these limits helps you write portable scripts and choose alternatives when needed.

7

Expert

Internal Mechanics of Process Substitution

🤔Before reading on: do you think process substitution creates real files on disk? Commit to your answer.

Concept: Learn how bash implements process substitution using pipes and file descriptors.

When you use <(command), bash creates a named pipe or uses /dev/fd to connect the command's output to a file descriptor. The command reading the file name actually reads from this pipe. For >(), bash creates a pipe that the writing command writes to, and the reading command reads from it.

Result

Process substitution creates invisible pipes that look like files, enabling seamless data flow.

Understanding this internal mechanism explains why process substitution is efficient and why it behaves like files but isn't one.

Under the Hood

Bash creates a temporary named pipe (FIFO) or uses /dev/fd entries to connect the output or input of a command to another command expecting a file. The shell runs the inner command asynchronously and provides the outer command a file name that points to the pipe. Data flows through this pipe in real time, without disk storage.

Why designed this way?

Process substitution was designed to overcome the limitation that many Unix commands require file names as arguments, not data streams. Using pipes alone can't always replace files because some commands open files multiple times or seek inside them. Named pipes or /dev/fd provide a file-like interface to streams, enabling compatibility and efficiency.

┌───────────────┐          ┌───────────────┐
│ Outer Command │          │ Inner Command │
│ (expects file)│          │ (produces or  │
│               │          │ consumes data)│
└──────┬────────┘          └──────┬────────┘
       │ File name (e.g. /dev/fd/63)   │
       │──────────────────────────────▶│
       │                               │
       │◀──────────────────────────────│
       │          Pipe/FIFO             │
       └──────────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does <(command) create a real file on disk? Commit to yes or no.

Common Belief: Process substitution creates temporary files on disk that you can find and edit.

Reality: Process substitution creates named pipes or file descriptors, which are special files that act like streams, not real disk files.

Why it matters: Expecting real files can lead to confusion when you try to open or edit these 'files' and find they behave differently or disappear.

Quick: Can you use >() to redirect output to any command? Commit to yes or no.

Common Belief: >() works exactly like > and can redirect output to any command seamlessly.

Reality: >() creates a pipe to a command, but not all commands handle this well, and some shells or commands may not support it properly.

Why it matters: Using >() blindly can cause scripts to fail or behave unexpectedly, especially on systems without full support.

Quick: Does process substitution work in all shells? Commit to yes or no.

Common Belief: Process substitution is a universal shell feature available everywhere.

Reality: Process substitution is mainly supported in bash and zsh; other shells like dash or sh may not support it.

Why it matters: Scripts using process substitution may break on systems with different default shells, reducing portability.

Quick: Does process substitution always improve script performance? Commit to yes or no.

Common Belief: Using process substitution always makes scripts faster and better.

Reality: While it avoids temporary files, process substitution can add overhead and complexity; sometimes simple pipes or temporary files are better.

Why it matters: Misusing process substitution can make scripts harder to read or debug without real performance gains.

Expert Zone

1

Process substitution uses file descriptors that are unique per shell instance, so scripts relying on fixed file descriptor numbers can break.

2

When stacking multiple process substitutions, the order of execution and resource cleanup can cause subtle bugs or race conditions.

3

Some commands open files multiple times or seek inside them, which process substitution pipes cannot support, requiring fallback to real temporary files.

When NOT to use

Avoid process substitution when writing highly portable scripts for minimal shells like dash or sh. Also, if the command requires random access to the file or multiple reads, use temporary files instead. For very large data, temporary files may be more efficient than pipes.

Production Patterns

In production, process substitution is used to compare outputs of commands (e.g., diff <(cmd1) <(cmd2)), to feed multiple inputs to commands expecting files, and to split output streams with tee >(cmd). It enables concise one-liners and complex pipelines without cluttering the filesystem.

Connections

Named Pipes (FIFOs)

Process substitution uses named pipes internally to simulate files for data streams.

Understanding named pipes helps grasp how process substitution creates file-like interfaces for commands.

Unix File Descriptors

Process substitution relies on file descriptors to connect commands through pipes.

Knowing file descriptors clarifies how bash manages input/output streams behind the scenes.

Data Streaming in Networking

Both process substitution and network streaming handle continuous data flow without storing it permanently.

Recognizing this similarity helps understand the efficiency and real-time nature of process substitution.

Common Pitfalls

#1 Trying to edit or open the file created by <() directly.

Wrong approach:

vim <(ls)

Correct approach:

ls > temp.txt; vim temp.txt

Root cause:

Misunderstanding that <() creates a pipe, not a real file, so editors expecting seekable files cannot open it.

#2 Using process substitution in a shell that does not support it.

Wrong approach:

diff <(ls dir1) <(ls dir2) # run in sh or dash

Correct approach:

diff <(ls dir1) <(ls dir2) # run in bash or zsh

Root cause:

Assuming all shells support process substitution leads to script failures on minimal shells.

#3 Using >() with commands that do not read from standard input properly.

Wrong approach:

echo 'data' >() somecommand

Correct approach:

echo 'data' | somecommand

Root cause:

Not all commands accept input from pipes created by >(), causing unexpected behavior.

Key Takeaways

Process substitution lets you use command outputs or inputs as if they were files, enabling flexible data flow in bash scripts.

It creates temporary pipes or file descriptors, not real files, which is why some commands behave differently with it.

Understanding process substitution helps write cleaner, faster scripts without cluttering the filesystem with temporary files.

It is supported mainly in bash and zsh, so portability requires caution.

Knowing its internal mechanism and limits prevents common bugs and helps choose the right tool for the task.