COPY command for bulk data loading in PostgreSQL - Step-by-Step Execution

Concept Flow - COPY command for bulk data loading

Start COPY command

↓

Read source file

↓

Parse each line

↓

Convert data to table format

↓

Insert rows into table

↓

Finish and report rows loaded

The COPY command reads data from a file, parses it line by line, converts it into table rows, inserts them, and finishes by reporting how many rows were loaded.

Execution Sample

PostgreSQL

COPY employees FROM '/tmp/employees.csv' DELIMITER ',' CSV HEADER;

This command loads data from a CSV file into the employees table, using comma as separator and skipping the header line.

Execution Table

Step	Action	Input/State	Output/Result
1	Start COPY command	Command issued	Ready to read file
2	Open file '/tmp/employees.csv'	File opened	File stream ready
3	Read first line	Line: 'id,name,salary'	Header detected, skipped
4	Read second line	Line: '1,John Doe,50000'	Parsed row: (1, 'John Doe', 50000)
5	Insert row into employees	Row data	Row inserted
6	Read third line	Line: '2,Jane Smith,60000'	Parsed row: (2, 'Jane Smith', 60000)
7	Insert row into employees	Row data	Row inserted
8	End of file reached	No more lines	COPY finished, 2 rows loaded

💡 End of file reached, all rows processed

Variable Tracker

Variable	Start	After Step 4	After Step 6	Final
Current Line	None	'1,John Doe,50000'	'2,Jane Smith,60000'	EOF
Rows Inserted	0	1	2	2

Key Moments - 3 Insights

Why does COPY skip the first line in this example?

Because the command uses CSV HEADER option, the first line is treated as column names and skipped, as shown in execution_table row 3.

What happens if the file has more rows than shown?

COPY will continue reading and inserting rows until it reaches the end of the file, as indicated by the loop in execution_table rows 4-7.

Why is the delimiter important in COPY?

The delimiter tells COPY how to split each line into columns. Here, ',' is used to separate fields, matching the CSV format.

Visual Quiz - 3 Questions

Look at the execution table, what is the output after reading the first data line?

A Header detected, skipped

B Parsed row: (1, 'John Doe', 50000)

C Row inserted

D File stream ready

💡 Hint

Check row 4 in the execution_table where the first data line is parsed.

At which step does the COPY command finish loading data?

A Step 6

B Step 7

C Step 8

D Step 3

💡 Hint

Look at the exit note and the last row in execution_table.

If the CSV file did not have a header, what would change in the execution?

A The first line would be inserted as data

B COPY would fail immediately

C Rows inserted would be zero

D Delimiter would change automatically

💡 Hint

Refer to key_moments about the HEADER option and execution_table row 3.

Concept Snapshot

COPY command syntax:
COPY table_name FROM 'file_path' DELIMITER ',' CSV HEADER;

- Reads data from a file
- Parses lines using delimiter
- Skips header if CSV HEADER used
- Inserts rows into table
- Efficient for bulk loading

Full Transcript

The COPY command in PostgreSQL loads bulk data from a file into a table. It starts by opening the file, then reads each line. If the CSV HEADER option is used, the first line is skipped as it contains column names. Each subsequent line is parsed using the specified delimiter, converting text into table columns. Each parsed row is inserted into the table. This continues until the end of the file is reached, and the command reports how many rows were loaded. This process is efficient for loading large amounts of data quickly.

Practice

1. What is the main purpose of the COPY command in PostgreSQL?

easy

A. To delete data from a table

B. To create a new table in the database

C. To quickly load or export large amounts of data between a file and a table

D. To update existing rows in a table

Solution

1. Step 1: Understand the function of COPY

   The COPY command is designed to transfer data efficiently between a file and a PostgreSQL table.

2. Step 2: Compare with other commands

   Unlike CREATE, UPDATE, or DELETE, COPY focuses on bulk data loading or exporting, not modifying table structure or individual rows.

3. Final Answer:

   To quickly load or export large amounts of data between a file and a table -> Option C

4. Quick Check:

   COPY = bulk data transfer [OK]

Hint: COPY is for fast bulk data load/export, not table changes [OK]

Common Mistakes:

• Confusing COPY with CREATE TABLE
• Thinking COPY updates rows
• Assuming COPY deletes data

2. Which of the following is the correct syntax to copy data from a CSV file into a PostgreSQL table named employees?

easy

A. COPY employees TO '/path/to/file.csv' WITH CSV HEADER;

B. COPY employees FROM '/path/to/file.csv' WITH CSV HEADER;

C. COPY employees FROM '/path/to/file.csv' DELIMITER ';';

D. COPY employees FROM '/path/to/file.csv' FORMAT JSON;

Solution

1. Step 1: Identify direction of COPY

   To load data into a table, use COPY ... FROM. COPY employees FROM '/path/to/file.csv' WITH CSV HEADER; and C use FROM, B uses TO which exports data.

2. Step 2: Check format and options

   COPY employees FROM '/path/to/file.csv' WITH CSV HEADER; uses WITH CSV HEADER which is correct for CSV files with headers. COPY employees FROM '/path/to/file.csv' DELIMITER ';'; uses DELIMITER ';' but no CSV keyword, which is less clear. COPY employees FROM '/path/to/file.csv' FORMAT JSON; uses FORMAT JSON which is invalid for COPY.

3. Final Answer:

   COPY employees FROM '/path/to/file.csv' WITH CSV HEADER; -> Option B

4. Quick Check:

   Load CSV with header = COPY ... FROM ... WITH CSV HEADER [OK]

Hint: Use COPY ... FROM for loading, WITH CSV HEADER for CSV files [OK]

Common Mistakes:

• Using COPY ... TO instead of FROM to load data
• Missing WITH CSV HEADER for CSV files with headers
• Using wrong format like JSON in COPY

3. Given the table products(id INT, name TEXT, price NUMERIC), what will be the result of this command?

COPY products FROM '/tmp/products.csv' WITH CSV HEADER;

Assuming /tmp/products.csv contains:

id,name,price
1,Pen,1.20
2,Notebook,2.50
3,Eraser,0.80

medium

A. No rows inserted because HEADER skips all data

B. Error because price is numeric but CSV has strings

C. Only 2 rows inserted because of header misread

D. 3 rows inserted into products table

Solution

1. Step 1: Understand COPY with CSV HEADER

   The HEADER option tells COPY to skip the first line (column names) and load the rest as data rows.

2. Step 2: Check data types and rows

   The CSV data matches the table columns and types correctly: id as int, name as text, price as numeric. All 3 data rows will be inserted.

3. Final Answer:

   3 rows inserted into products table -> Option D

4. Quick Check:

   CSV HEADER skips first line, 3 data lines inserted [OK]

Hint: CSV HEADER skips first line; all data rows load if types match [OK]

Common Mistakes:

• Thinking HEADER skips all rows
• Assuming numeric columns reject string input
• Confusing COPY direction or syntax

4. You run this command:

COPY customers FROM '/data/customers.csv' WITH CSV;

But get an error: ERROR: could not open file "/data/customers.csv": Permission denied
What is the most likely cause?

medium

A. The PostgreSQL server process lacks permission to read the file

B. The CSV file format is incorrect

C. The COPY command syntax is wrong

D. The customers table does not exist

Solution

1. Step 1: Analyze the error message

   The error says "Permission denied" when opening the file, indicating a file access issue.

2. Step 2: Understand file permissions in COPY

   PostgreSQL server must have OS-level read permission on the file. If it lacks permission, this error occurs.

3. Final Answer:

   The PostgreSQL server process lacks permission to read the file -> Option A

4. Quick Check:

   Permission denied = file access issue [OK]

Hint: Permission denied means PostgreSQL can't read the file [OK]

Common Mistakes:

• Assuming syntax error causes permission denied
• Thinking CSV format causes file access error
• Ignoring file permissions on server

5. You want to load data from a CSV file into a table but skip rows where the age column is empty or NULL. Which approach using COPY and PostgreSQL features is best?

hard

A. Use COPY to load all data into a staging table, then insert filtered rows into the target table

B. Use COPY with WHERE clause to skip rows with empty age

C. Use COPY with an option to ignore NULL values in age column

D. Modify the CSV file manually to remove rows with empty age before COPY

Solution

1. Step 1: Understand COPY limitations

   COPY does not support filtering rows or WHERE clauses during import.

2. Step 2: Use staging table for filtering

   Load all data into a temporary or staging table, then run an INSERT INTO ... SELECT ... WHERE age IS NOT NULL to filter rows.

3. Step 3: Consider manual CSV editing

   While manual editing works, it is inefficient and error-prone for large files.

4. Final Answer:

   Use COPY to load all data into a staging table, then insert filtered rows into the target table -> Option A

5. Quick Check:

   COPY can't filter; use staging table + filter [OK]

Hint: COPY can't filter; load then filter with SQL [OK]

Common Mistakes:

• Expecting COPY to filter rows directly
• Looking for non-existent COPY options to ignore NULLs
• Relying on manual CSV edits for large data