sort and uniq in Linux CLI - Commands & Configuration

Introduction

Sometimes you have a list of items with duplicates and want to organize them neatly and remove repeated entries. The sort and uniq commands help you do this by arranging items in order and filtering out duplicates.

When you have a list of names and want to see them in alphabetical order without repeats.

When you want to count how many unique IP addresses accessed your server from a log file.

When you need to clean up a list of installed packages by removing duplicate entries.

When you want to prepare a sorted list of unique words from a text file for analysis.

When you want to quickly find unique error messages from a system log.

Commands

Create a file named fruits.txt with a list of fruit names, some repeated, to use for sorting and removing duplicates.

Terminal

echo -e "apple\nbanana\napple\ncherry\nbanana" > fruits.txt

Expected Output

No output (command runs silently)

Sort the contents of fruits.txt alphabetically. This arranges the list so duplicates are next to each other, which uniq needs to work properly.

Terminal

sort fruits.txt

Expected Output

apple

apple

banana

banana

cherry

Sort the list and then pass it to uniq to remove duplicate lines, showing only unique fruit names in order.

Terminal

sort fruits.txt | uniq

Expected Output

apple

banana

cherry

Sort the list and use uniq with the -c flag to count how many times each unique item appears.

Terminal

sort fruits.txt | uniq -c

Expected Output

      2 apple

      2 banana

      1 cherry

→

-c - Count occurrences of each unique line

Key Concept

If you remember nothing else from this pattern, remember: uniq only removes duplicates that are next to each other, so always sort your data first.

Common Mistakes

Running uniq without sorting the input first.

uniq only removes adjacent duplicate lines, so duplicates separated by other lines will not be removed.

Always sort the input before piping it to uniq to group duplicates together.

Using uniq without understanding it only filters adjacent duplicates.

This leads to unexpected results where duplicates remain if they are not next to each other.

Use sort before uniq or use uniq -u or other options carefully depending on the goal.

Summary

Use sort to arrange lines alphabetically or numerically so duplicates are next to each other.

Pipe sorted output to uniq to remove duplicate lines or count occurrences with -c.

Remember uniq only works on adjacent duplicates, so sorting first is essential.