COPY INTO command in Snowflake - Time & Space Complexity

Time Complexity: COPY INTO command

O(n)

Understanding Time Complexity

When loading data into Snowflake using the COPY INTO command, it is important to understand how the time taken grows as the data size increases.

We want to know how the number of operations changes when we load more data.

Scenario Under Consideration

Analyze the time complexity of the following operation sequence.

COPY INTO my_table
FROM @my_stage/data_files
FILE_FORMAT = (TYPE = 'CSV' FIELD_DELIMITER = ',' SKIP_HEADER = 1)
ON_ERROR = 'CONTINUE';

This command loads multiple CSV files from a stage into a table, skipping the header row and continuing on errors.

Identify Repeating Operations

Identify the API calls, resource provisioning, data transfers that repeat.

• Primary operation: Reading and parsing each file from the stage and inserting data into the table.
• How many times: Once per file, repeated for all files in the stage folder.

How Execution Grows With Input

As the number of files or total data size grows, the number of read and insert operations grows roughly in proportion.

Input Size (n)	Approx. API Calls/Operations
10 files	10 read and insert operations
100 files	100 read and insert operations
1000 files	1000 read and insert operations

Pattern observation: The operations increase linearly with the number of files or data size.

Final Time Complexity

Time Complexity: O(n)

This means the time to complete the COPY INTO command grows directly in proportion to the amount of data being loaded.

Common Mistake

[X] Wrong: "COPY INTO runs in constant time no matter how much data is loaded."

[OK] Correct: The command must read and process each file, so more data means more work and longer time.

Interview Connect

Understanding how data loading time grows helps you design efficient pipelines and explain performance in real projects.

Self-Check

"What if we changed the COPY INTO command to load compressed files instead? How would the time complexity change?"

Practice

1. What is the main purpose of the COPY INTO command in Snowflake?

easy

A. To load data files from cloud storage into Snowflake tables

B. To export data from Snowflake tables to local files

C. To create new tables in Snowflake

D. To delete data from Snowflake tables

Solution

1. Step 1: Understand the command purpose

   The COPY INTO command is designed to load data from external files into Snowflake tables.

2. Step 2: Compare options

   Options B, C, and D describe exporting, creating, or deleting data, which are not the function of COPY INTO.

3. Final Answer:

   To load data files from cloud storage into Snowflake tables -> Option A

4. Quick Check:

   Loading data = COPY INTO [OK]

Hint: COPY INTO means loading data into tables [OK]

Common Mistakes:

• Confusing COPY INTO with export commands
• Thinking COPY INTO creates tables
• Assuming COPY INTO deletes data

2. Which of the following is the correct basic syntax to load data using COPY INTO in Snowflake?

easy

A. COPY table_name INTO @stage_name FILE_FORMAT = (TYPE = 'CSV');

B. LOAD DATA INTO table_name FROM @stage_name FORMAT = CSV;

C. COPY INTO table_name FROM @stage_name FILE_FORMAT = (TYPE = 'CSV');

D. INSERT INTO table_name COPY FROM @stage_name FILE_FORMAT = CSV;

Solution

1. Step 1: Recall correct COPY INTO syntax

   The correct syntax is COPY INTO table_name FROM @stage_name FILE_FORMAT = (TYPE = 'CSV'); to load CSV files from a stage.

2. Step 2: Identify incorrect syntax

   Options A, C, and D use wrong keywords or order that Snowflake does not accept.

3. Final Answer:

   COPY INTO table_name FROM @stage_name FILE_FORMAT = (TYPE = 'CSV'); -> Option C

4. Quick Check:

   Correct COPY INTO syntax = COPY INTO table_name FROM @stage_name FILE_FORMAT = (TYPE = 'CSV'); [OK]

Hint: COPY INTO table FROM stage with FILE_FORMAT [OK]

Common Mistakes:

• Swapping INTO and FROM keywords
• Using LOAD DATA instead of COPY INTO
• Incorrect FILE_FORMAT syntax

3. Given the command:

COPY INTO my_table FROM @my_stage FILE_FORMAT = (TYPE = 'CSV' FIELD_DELIMITER = ',') ON_ERROR = 'skip_file';

What happens if one file in the stage has errors during loading?

medium

A. Snowflake retries loading the file indefinitely

B. The entire load fails and no data is loaded

C. Only the erroneous rows are skipped, file loads partially

D. The file with errors is skipped, other files load successfully

Solution

1. Step 1: Understand ON_ERROR = 'skip_file'

   This option tells Snowflake to skip any file that causes errors during loading instead of failing the entire load.

2. Step 2: Analyze the effect on other files

   Other files without errors will load normally, so only the bad file is skipped.

3. Final Answer:

   The file with errors is skipped, other files load successfully -> Option D

4. Quick Check:

   ON_ERROR = skip_file skips bad files [OK]

Hint: ON_ERROR = skip_file skips bad files, loads others [OK]

Common Mistakes:

• Assuming entire load fails on one bad file
• Thinking only bad rows are skipped with skip_file
• Believing Snowflake retries endlessly

4. You run this command:

COPY INTO sales FROM @data_stage FILE_FORMAT = (FORMAT_NAME = 'csv_format');

But get an error: SQL compilation error: file format not found. What is the likely cause?

medium

A. The stage @data_stage does not exist

B. The FILE_FORMAT clause references a named file format that does not exist

C. The sales table does not exist

D. The CSV files are empty

Solution

1. Step 1: Analyze the error message

   The error says file format not found, meaning Snowflake expects a named file format or a full definition.

2. Step 2: Check the FILE_FORMAT clause

   The clause FILE_FORMAT = (FORMAT_NAME = 'csv_format') references a named file format that does not exist.

3. Final Answer:

   The FILE_FORMAT clause references a named file format that does not exist -> Option B

4. Quick Check:

   File format error means missing or wrong FILE_FORMAT [OK]

Hint: File format errors mean missing or wrong FILE_FORMAT [OK]

Common Mistakes:

• Assuming stage or table missing causes file format error
• Ignoring need for file format definition
• Thinking empty files cause this error

5. You want to load JSON files from an external S3 bucket into a Snowflake table named events. Which COPY INTO command correctly handles JSON format and ignores files with errors?

hard

A. COPY INTO events FROM @s3_stage FILE_FORMAT = (TYPE = 'JSON') ON_ERROR = 'skip_file';

B. COPY INTO events FROM @s3_stage FILE_FORMAT = (TYPE = 'CSV') ON_ERROR = 'continue';

C. COPY INTO events FROM @s3_stage FILE_FORMAT = (TYPE = 'JSON') ON_ERROR = 'abort_statement';

D. COPY INTO events FROM @s3_stage FILE_FORMAT = (TYPE = 'XML') ON_ERROR = 'skip_file';

Solution

1. Step 1: Match file format to JSON files

   Since files are JSON, the FILE_FORMAT must specify TYPE = 'JSON'.

2. Step 2: Choose error handling to ignore bad files

   ON_ERROR = 'skip_file' skips files with errors, which matches the requirement.

3. Step 3: Eliminate incorrect options

   COPY INTO events FROM @s3_stage FILE_FORMAT = (TYPE = 'CSV') ON_ERROR = 'continue'; uses CSV format, wrong for JSON. COPY INTO events FROM @s3_stage FILE_FORMAT = (TYPE = 'JSON') ON_ERROR = 'abort_statement'; aborts on error, not ignoring. COPY INTO events FROM @s3_stage FILE_FORMAT = (TYPE = 'XML') ON_ERROR = 'skip_file'; uses XML format, incorrect.

4. Final Answer:

   COPY INTO events FROM @s3_stage FILE_FORMAT = (TYPE = 'JSON') ON_ERROR = 'skip_file'; -> Option A

5. Quick Check:

   JSON format + skip_file error handling = COPY INTO events FROM @s3_stage FILE_FORMAT = (TYPE = 'JSON') ON_ERROR = 'skip_file'; [OK]

Hint: Match FILE_FORMAT type to file type and use skip_file to ignore errors [OK]

Common Mistakes:

• Using wrong file format type
• Choosing abort instead of skip_file for errors
• Confusing JSON with CSV or XML