sort and uniq in pipelines in Bash Scripting - Step-by-Step Execution

Concept Flow - sort and uniq in pipelines

Input lines

↓

sort command

↓

uniq command

↓

Output unique sorted lines

The pipeline takes input lines, sorts them alphabetically, then filters out duplicates to output unique sorted lines.

Execution Sample

Bash Scripting

printf "apple\nbanana\napple\ncherry\nbanana\n" | sort | uniq

This command sorts the list of fruits and removes duplicates, showing unique sorted fruits.

Execution Table

Step	Command	Input	Output	Explanation
1	printf	none	apple banana apple cherry banana	Prints the list of fruits with duplicates
2	sort	apple banana apple cherry banana	apple apple banana banana cherry	Sorts lines alphabetically, duplicates stay
3	uniq	apple apple banana banana cherry	apple banana cherry	Removes adjacent duplicate lines
4	end	N/A	apple banana cherry	Pipeline ends with unique sorted output

💡 uniq stops after processing all sorted lines, outputting unique lines only

Variable Tracker

Variable	Start	After printf	After sort	After uniq	Final
lines	empty	apple banana apple cherry banana	apple apple banana banana cherry	apple banana cherry	apple banana cherry

Key Moments - 3 Insights

Why does uniq only remove duplicates after sort, not before?

uniq only removes duplicates that are next to each other. Sorting groups duplicates together so uniq can remove them, as shown in execution_table step 2 and 3.

What happens if we use uniq before sort?

uniq would only remove duplicates that are adjacent in the original input order, missing duplicates separated by other lines. This is why sorting first is important.

Why do we use a pipeline with | between commands?

The pipe | sends the output of one command as input to the next, allowing sort to receive the printed lines and uniq to receive the sorted lines, as shown in the concept_flow.

Visual Quiz - 3 Questions

Look at the execution table, what is the output after the sort command?

A apple banana apple cherry banana

B apple apple banana banana cherry

C apple banana cherry

D banana apple cherry

💡 Hint

Check the 'Output' column for step 2 in the execution_table.

At which step does the pipeline remove duplicate lines?

A Step 3 - uniq

B Step 1 - printf

C Step 2 - sort

D Step 4 - end

💡 Hint

Look at the 'Explanation' column in execution_table for step 3.

If we remove the sort command, how would the uniq output change?

A It would still remove all duplicates correctly.

B It would output the lines in sorted order.

C It would remove only adjacent duplicates, missing others.

D It would cause an error.

💡 Hint

Refer to key_moments about uniq behavior without sorting.

Concept Snapshot

Use 'sort | uniq' in a pipeline to get unique sorted lines.
'sort' arranges lines alphabetically.
'uniq' removes only adjacent duplicates.
Pipe output of sort into uniq for correct unique filtering.
Without sort, uniq misses non-adjacent duplicates.

Full Transcript

This visual execution shows how the bash pipeline 'sort | uniq' works. First, input lines are printed with duplicates. Then 'sort' arranges these lines alphabetically but keeps duplicates. Next, 'uniq' removes duplicates only if they are next to each other. Sorting first groups duplicates together so uniq can remove them. The final output is unique sorted lines. Key points include that uniq alone only removes adjacent duplicates, so sorting first is essential. The pipeline uses the pipe symbol to send output from one command to the next. This step-by-step trace helps beginners see how each command changes the data and why the order matters.

Practice

1. What is the main purpose of using sort | uniq together in a bash pipeline?

easy

A. To get a sorted list with duplicates removed

B. To count the number of lines in a file

C. To sort lines but keep duplicates

D. To find the longest line in a file

Solution

1. Step 1: Understand what sort does

   The sort command arranges lines in alphabetical or numerical order.

2. Step 2: Understand what uniq does after sorting

   The uniq command removes duplicate adjacent lines, so sorting first groups duplicates together for removal.

3. Final Answer:

   To get a sorted list with duplicates removed -> Option A

4. Quick Check:

   sort + uniq = sorted unique lines [OK]

Hint: Always sort before uniq to remove all duplicates [OK]

Common Mistakes:

• Using uniq before sort, which misses duplicates
• Thinking uniq counts lines instead of removing duplicates
• Assuming sort removes duplicates by itself

2. Which of the following is the correct syntax to remove duplicate lines from file.txt and sort the output?

easy

A. uniq file.txt | sort

B. sort file.txt | uniq

C. sort | uniq file.txt

D. uniq | sort file.txt

Solution

1. Step 1: Identify correct command order

   The file must be sorted first, so sort file.txt comes before uniq.

2. Step 2: Check syntax correctness

   sort file.txt | uniq correctly pipes sorted output to uniq for duplicate removal.

3. Final Answer:

   sort file.txt | uniq -> Option B

4. Quick Check:

   sort first, then uniq = correct syntax [OK]

Hint: Pipe sorted output into uniq: sort file | uniq [OK]

Common Mistakes:

• Placing uniq before sort, which fails to remove all duplicates
• Incorrect pipe placement causing syntax errors
• Trying to pass file as argument to uniq instead of sort

3. Given a file names.txt with contents:

bob
alice
bob
carol
alice
bob

What is the output of sort names.txt | uniq?

medium

A. alice
bob
bob
carol
alice
bob

B. bob
alice
carol

C. bob
carol
alice

D. alice
bob
carol

Solution

1. Step 1: Sort the file contents

   Sorting lines alphabetically gives: alice, alice, bob, bob, bob, carol.

2. Step 2: Apply uniq to remove duplicates

   Uniq removes repeated adjacent lines, resulting in: alice, bob, carol.

3. Final Answer:

   alice\nbob\ncarol -> Option D

4. Quick Check:

   sort + uniq = unique sorted lines [OK]

Hint: Sort first to group duplicates, then uniq removes repeats [OK]

Common Mistakes:

• Not sorting first, so uniq misses duplicates
• Expecting original order instead of sorted
• Confusing uniq output with counting duplicates

4. You run uniq file.txt | sort to remove duplicates and sort lines, but duplicates remain. What is the likely problem?

medium

A. uniq only removes adjacent duplicates, so sorting after uniq misses duplicates

B. sort command is not working without options

C. file.txt is empty, so no output

D. uniq requires -c option to remove duplicates

Solution

1. Step 1: Understand uniq behavior

   Uniq removes only adjacent duplicate lines, so if duplicates are scattered, it won't remove them.

2. Step 2: Analyze command order effect

   Sorting after uniq means duplicates are not grouped before uniq runs, so duplicates remain.

3. Final Answer:

   uniq only removes adjacent duplicates, so sorting after uniq misses duplicates -> Option A

4. Quick Check:

   uniq needs sorted input to remove all duplicates [OK]

Hint: Sort before uniq to group duplicates for removal [OK]

Common Mistakes:

• Thinking uniq removes all duplicates regardless of order
• Believing sort alone removes duplicates
• Using uniq -c instead of uniq for removal

5. You have a log file access.log with many repeated IP addresses scattered randomly. You want a sorted list of unique IPs. Which command pipeline correctly achieves this?

hard

A. cat access.log | uniq | sort

B. uniq access.log | sort

C. sort -u access.log

D. uniq access.log | sort -u

Solution

1. Step 1: Understand sort -u option

   sort -u sorts and removes duplicates in one step efficiently.

2. Step 2: Why others fail

   Options A, B, and D all apply uniq before sorting, missing non-adjacent duplicates since IPs are scattered randomly.

3. Final Answer:

   sort -u access.log -> Option C

4. Quick Check:

   sort -u = sorted unique lines in one command [OK]

Hint: Use sort -u for sorted unique lines in one step [OK]

Common Mistakes:

• Using uniq before sort, missing duplicates
• Not knowing sort -u combines sorting and uniqueness
• Using unnecessary cat command