sort and uniq in pipelines in Bash Scripting

Introduction

We use sort and uniq together in pipelines to organize data and remove repeated lines. This helps clean up lists or logs quickly.

You have a list of names and want to see them in order without duplicates.

You want to count unique error messages from a log file.

You need to clean up repeated entries in a text file before processing.

You want to prepare a sorted list of unique IP addresses from server logs.

Syntax

Bash Scripting

command | sort | uniq

The pipe symbol (|) sends output from one command to the next.

sort arranges lines alphabetically or numerically before uniq removes duplicates.

Examples

Reads file.txt, sorts lines, then removes duplicates.

Bash Scripting

cat file.txt | sort | uniq

Sorts and removes duplicate 'apple' from the list.

Bash Scripting

echo -e "apple\nbanana\napple" | sort | uniq

Lists running processes, sorts them, and removes duplicate lines.

Bash Scripting

ps aux | sort | uniq

Sample Program

This script prints a list of animals with duplicates, sorts them alphabetically, and removes repeated lines.

Bash Scripting

echo -e "dog\ncat\ndog\nbird\ncat" | sort | uniq

Important Notes

uniq only removes duplicates that are next to each other, so sorting first is important.

You can use 'uniq -c' to count how many times each unique line appears.

If you want to ignore case when sorting, use 'sort -f'.

Summary

Use 'sort | uniq' to get a sorted list without duplicates.

Always sort before uniq to remove all duplicates.

This combination is great for cleaning and organizing text data.