Processing CSV files in Bash Scripting - Step-by-Step Execution

Concept Flow - Processing CSV files

Start

↓

Open CSV file

↓

Read line by line

↓

Split line by commas

↓

Process fields

↓

Output or store result

↓

Repeat until end of file

↓

Close file

↓

End

This flow shows how a bash script reads a CSV file line by line, splits each line into fields by commas, processes those fields, and outputs or stores the results until the file ends.

Execution Sample

Bash Scripting

while IFS=',' read -r name age city; do
  echo "Name: $name, Age: $age, City: $city"
done < people.csv

Reads a CSV file named people.csv line by line, splits each line into name, age, and city, then prints them.

Execution Table

Step	Line Read	Variables Assigned	Action	Output
1	Alice,30,New York	name=Alice, age=30, city=New York	Print formatted info	Name: Alice, Age: 30, City: New York
2	Bob,25,Los Angeles	name=Bob, age=25, city=Los Angeles	Print formatted info	Name: Bob, Age: 25, City: Los Angeles
3	Charlie,35,Chicago	name=Charlie, age=35, city=Chicago	Print formatted info	Name: Charlie, Age: 35, City: Chicago
4			End of file reached

💡 No more lines to read; loop ends

Variable Tracker

Variable	Start	After 1	After 2	After 3	Final
name		Alice	Bob	Charlie
age		30	25	35
city		New York	Los Angeles	Chicago

Key Moments - 3 Insights

Why do we use IFS=',' before read?

IFS=',' tells the shell to split each line by commas, so each field goes into the right variable. Without it, the whole line would go into the first variable (see execution_table rows 1-3).

What happens if the CSV line has more or fewer fields?

If there are fewer fields, some variables stay empty. If more, extra fields go into the last variable. This can cause wrong assignments, so the script expects exactly three fields per line (see variable_tracker).

Why do we redirect the file into the loop with done < people.csv?

This feeds the file line by line into the loop. Without it, the loop would wait for keyboard input instead of reading the file (see concept_flow step 'Read line by line').

Visual Quiz - 3 Questions

Look at the execution table, what is the value of 'city' at step 2?

A Chicago

B New York

C Los Angeles

D Empty

💡 Hint

Check the 'Variables Assigned' column at step 2 in the execution_table.

At which step does the loop stop reading lines from the file?

A Step 4

B Step 3

C Step 2

D Step 1

💡 Hint

Look for the row where 'End of file reached' is noted in the Action column.

If the CSV file had a line 'Diana,40', what would happen to the 'city' variable at that step?

A It would contain 'Diana'

B It would be empty

C It would contain '40'

D The script would crash

💡 Hint

Refer to key_moments about fewer fields and variable assignments.

Concept Snapshot

Processing CSV files in bash:
Use 'while IFS=',' read -r var1 var2 ...; do ... done < file.csv'
IFS=',' splits lines by commas into variables
Process each line inside the loop
Loop ends when file has no more lines

Full Transcript

This example shows how to process CSV files using bash scripting. The script reads the CSV file line by line, splitting each line by commas into variables using IFS=','. Each variable holds one field from the CSV line. The script then processes or prints these variables. The loop continues until all lines are read. Key points include setting IFS to comma for correct splitting, redirecting the file into the loop, and handling lines with the expected number of fields.

Practice

1. What does the cut -d, -f2 file.csv command do when processing a CSV file?

easy

A. Deletes the second column from the CSV file

B. Extracts the second column from the CSV file using comma as delimiter

C. Counts the number of commas in the CSV file

D. Replaces commas with tabs in the CSV file

Solution

1. Step 1: Understand the cut command with delimiter

   The -d, option sets the delimiter to a comma, which is the separator in CSV files.

2. Step 2: Extract the specified field

   The -f2 option extracts the second field (column) from each line of the file.

3. Final Answer:

   Extracts the second column from the CSV file using comma as delimiter -> Option B

4. Quick Check:

   cut -d, -f2 extracts column 2 [OK]

Hint: Use -d for delimiter and -f for field number in cut [OK]

Common Mistakes:

• Confusing -f with line number
• Using wrong delimiter character
• Thinking cut deletes columns

2. Which of the following is the correct syntax to print the third column of a CSV file using awk?

easy

A. awk -F, '{print $3}' file.csv

B. awk -d',' '{print $3}' file.csv

C. awk -F',' 'print $3' file.csv

D. awk '{print $3}' -F',' file.csv

Solution

1. Step 1: Identify the field separator option in awk

   The correct option to set the field separator is -F,, which tells awk to split fields by commas.

2. Step 2: Use the print statement correctly

   The print statement must be inside single quotes and include curly braces: {print $3} to print the third column.

3. Final Answer:

   awk -F, '{print $3}' file.csv -> Option A

4. Quick Check:

   Use -F for delimiter and print $3 inside braces [OK]

Hint: Use -F, and print $column inside braces in awk [OK]

Common Mistakes:

• Using -d instead of -F for delimiter
• Omitting curly braces around print
• Placing -F option after the script

3. Given a CSV file data.csv with content:

name,age,city
Alice,30,New York
Bob,25,Los Angeles
Carol,27,Chicago

What is the output of the command awk -F, '$2 > 26 {print $1}' data.csv?

medium

A. Alice

B. Alice
Bob
Carol

C. Bob
Carol

D. Alice
Carol

Solution

1. Step 1: Understand the awk condition and field separator

   The command uses -F, to split fields by commas. The condition $2 > 26 filters rows where the second field (age) is greater than 26.

2. Step 2: Apply the condition to each row

   Rows:
   - Header: 'age' > 26? No (string)
   - Alice,30: 30 > 26? Yes, print $1 = Alice
   - Bob,25: 25 > 26? No
   - Carol,27: 27 > 26? Yes, print $1 = Carol

3. Final Answer:

   Alice Carol -> Option D

4. Quick Check:

   Filter ages > 26, print names [OK]

Hint: Filter with $column condition and print desired field [OK]

Common Mistakes:

• Including header row in output
• Confusing field numbers
• Not using correct field separator

4. You wrote this command to extract the first column from a CSV file:
cut -f1 -d',' file.csv
But it returns the whole line instead of just the first column. What is the likely problem?

medium

A. The -f1 option is invalid

B. The file is empty

C. The delimiter option -d',' is missing or incorrect

D. cut cannot process CSV files

Solution

1. Step 1: Check the delimiter option usage

   The command uses -d',', which is incorrect. The single quotes make the delimiter the string ',' (quote comma quote) instead of plain comma (,). cut searches for ',' which isn't in the CSV.

2. Step 2: Understand cut behavior without correct delimiter

   Without splitting on ,, cut falls back to default (TAB), treating each line as one field, so -f1 outputs the entire line.

3. Final Answer:

   The delimiter option -d',' is missing or incorrect -> Option C

4. Quick Check:

   cut needs -d, (no quotes around comma) to split fields [OK]

Hint: Always specify -d, for CSV files with cut [OK]

Common Mistakes:

• Forgetting -d option
• Using wrong delimiter character
• Assuming cut auto-detects delimiter

5. You want to create a new CSV file containing only rows where the city is 'Chicago' from data.csv. Which bash command correctly filters and saves these rows?

hard

A. awk -F, '$3 == "Chicago"' data.csv > chicago.csv

B. grep 'Chicago' data.csv > chicago.csv

C. cut -d',' -f3 data.csv | grep 'Chicago' > chicago.csv

D. awk -F, '{if ($3 = "Chicago") print}' data.csv > chicago.csv

Solution

1. Step 1: Use awk with correct field and condition

   Using -F, sets comma as delimiter. The condition $3 == "Chicago" selects rows where the third column equals 'Chicago'.

2. Step 2: Redirect output to new file

   The output is redirected to chicago.csv using >, saving only matching rows.

3. Final Answer:

   awk -F, '$3 == "Chicago"' data.csv > chicago.csv -> Option A

4. Quick Check:

   Use awk with == for exact match and redirect output [OK]

Hint: Use awk with == for exact match and redirect output [OK]

Common Mistakes:

• Using single = instead of == in awk condition
• Using grep which may match partial fields
• Using cut which extracts columns, not rows