Why pipelines automate data freshness in Snowflake - Why It Works

Introduction

Data pipelines help keep your data up-to-date automatically. They run tasks in order to refresh data without manual work. This ensures reports and apps always use fresh information.

When you want your sales dashboard to show today's numbers without manual updates

When you need to combine data from multiple sources regularly without errors

When your marketing team relies on up-to-date customer data for campaigns

When you want to avoid manual steps that can cause delays or mistakes

When you want to schedule data updates during off-hours to save resources

Commands

This command creates a scheduled task in Snowflake that runs every hour. It calls a stored procedure to refresh sales data automatically.

Terminal

snowsql -q "CREATE OR REPLACE TASK refresh_sales_data WAREHOUSE = compute_wh SCHEDULE = 'USING CRON 0 * * * * UTC' AS CALL refresh_sales_procedure();"

Expected Output

Done.



Number of rows affected: 0

→

WAREHOUSE - Specifies the compute resource to run the task

→

SCHEDULE - Defines when the task runs automatically

This command starts the scheduled task so it begins running at the set times.

Terminal

snowsql -q "ALTER TASK refresh_sales_data RESUME;"

Expected Output

Done.



Number of rows affected: 0

This command checks the status of the task to confirm it is active and scheduled correctly.

Terminal

snowsql -q "SHOW TASKS LIKE 'refresh_sales_data';"

Expected Output

name               | database_name | schema_name | owner | state   | schedule           | last_scheduled_time       

refresh_sales_data  | MY_DB        | PUBLIC      | USER  | started | CRON 0 * * * * UTC | 2024-06-01 12:00:00.000 +0000

This command shows the last 5 runs of the task to verify it ran successfully and refreshed data.

Terminal

snowsql -q "SELECT SYSTEM$TASK_HISTORY('refresh_sales_data', 5);"

Expected Output

[{"name":"refresh_sales_data","scheduled_time":"2024-06-01 12:00:00","state":"SUCCEEDED"},{"name":"refresh_sales_data","scheduled_time":"2024-06-01 11:00:00","state":"SUCCEEDED"}]

Key Concept

If you remember nothing else from this pattern, remember: pipelines run tasks automatically on a schedule to keep data fresh without manual effort.

Common Mistakes

Creating a task but forgetting to resume it

The task stays paused and never runs, so data does not refresh.

Always run ALTER TASK task_name RESUME after creating the task.

Setting a schedule that runs too often or too rarely

Running too often wastes resources; too rarely causes stale data.

Choose a schedule that balances freshness with cost, like hourly or daily.

Not checking task history to confirm successful runs

You might miss failures and think data is fresh when it is not.

Regularly query SYSTEM$TASK_HISTORY to monitor task success.

Summary

Create a Snowflake task to run a data refresh procedure on a schedule.

Resume the task so it starts running automatically.

Check the task status and history to ensure it runs successfully and keeps data fresh.

Practice

1. What is the main reason to use tasks in Snowflake for data freshness?

easy

A. To automate data updates without manual steps

B. To store data permanently without changes

C. To manually refresh data when needed

D. To delete old data automatically

Solution

1. Step 1: Understand task purpose

   Tasks are designed to automate data updates, so they run without manual work.

2. Step 2: Identify automation benefit

   Automating data updates keeps data fresh and ready without needing manual refreshes.

3. Final Answer:

   To automate data updates without manual steps -> Option A

4. Quick Check:

   Automation means no manual steps [OK]

Hint: Tasks run automatically to keep data fresh [OK]

Common Mistakes:

• Thinking tasks only store data
• Assuming manual refresh is needed
• Confusing deletion with freshness

2. Which syntax correctly creates a Snowflake task that runs every hour?

easy

A. CREATE TASK my_task SCHEDULE = '1 HOUR' AS COPY INTO my_table FROM @my_stage;

B. CREATE TASK my_task SCHEDULE = 'USING CRON 0 * * * *' AS COPY INTO my_table FROM @my_stage;

C. CREATE TASK my_task SCHEDULE = '60 MINUTES' AS COPY INTO my_table FROM @my_stage;

D. CREATE TASK my_task SCHEDULE = 'EVERY HOUR' AS COPY INTO my_table FROM @my_stage;

Solution

1. Step 1: Check Snowflake task schedule syntax

   Snowflake uses CRON expressions for scheduling tasks, e.g., 'USING CRON 0 * * * *' means every hour at minute 0.

2. Step 2: Compare options

   Only CREATE TASK my_task SCHEDULE = 'USING CRON 0 * * * *' AS COPY INTO my_table FROM @my_stage; uses valid CRON syntax. Options A, B, and D use incorrect or unsupported schedule formats.

3. Final Answer:

   CREATE TASK my_task SCHEDULE = 'USING CRON 0 * * * *' AS COPY INTO my_table FROM @my_stage; -> Option B

4. Quick Check:

   Snowflake tasks use CRON for schedules [OK]

Hint: Snowflake task schedules use CRON syntax [OK]

Common Mistakes:

• Using plain time strings instead of CRON
• Writing 'EVERY HOUR' instead of CRON
• Confusing minutes and hours format

3. Given this task code:

CREATE TASK daily_update SCHEDULE = 'USING CRON 0 0 * * *' AS
COPY INTO sales FROM @sales_stage;

What happens when this task runs?

medium

A. It copies new data from sales_stage to sales table daily at midnight

B. It deletes all data from sales table daily at midnight

C. It copies data every minute from sales_stage to sales table

D. It runs only once when created

Solution

1. Step 1: Interpret CRON schedule

   'USING CRON 0 0 * * *' means the task runs daily at midnight (00:00 hours).

2. Step 2: Understand task action

   The task copies data from the stage 'sales_stage' into the 'sales' table each time it runs.

3. Final Answer:

   It copies new data from sales_stage to sales table daily at midnight -> Option A

4. Quick Check:

   CRON midnight + COPY INTO = daily data update [OK]

Hint: CRON 0 0 * * * means daily at midnight [OK]

Common Mistakes:

• Thinking it deletes data
• Assuming it runs every minute
• Believing it runs only once

4. You created a task but it never runs. Which fix will most likely solve this?

medium

A. Rename the task to a different name

B. Delete the task and recreate the table

C. Check if the task schedule uses valid CRON syntax

D. Run the COPY command manually every hour

Solution

1. Step 1: Identify common task issue

   If a task never runs, an invalid schedule is a common cause, especially if CRON syntax is wrong.

2. Step 2: Apply fix

   Correcting the schedule to valid CRON syntax enables the task to run automatically as expected.

3. Final Answer:

   Check if the task schedule uses valid CRON syntax -> Option C

4. Quick Check:

   Invalid schedule blocks task runs [OK]

Hint: Invalid CRON stops task from running [OK]

Common Mistakes:

• Deleting unrelated objects
• Running commands manually defeats automation
• Renaming does not fix schedule issues

5. You want to ensure your sales data is always fresh by running a task every 15 minutes. Which approach is best?

hard

A. Create a task without schedule and run it once daily

B. Manually run COPY commands every 15 minutes

C. Create a task with schedule 'USING CRON 15 * * * *' to run once per hour

D. Create a task with schedule 'USING CRON */15 * * * *' to automate data copy

Solution

1. Step 1: Understand CRON for 15-minute intervals

   'USING CRON */15 * * * *' means run every 15 minutes, which fits the requirement.

2. Step 2: Compare options for automation

   Create a task with schedule 'USING CRON */15 * * * *' to automate data copy automates the data copy every 15 minutes. Manually run COPY commands every 15 minutes is manual and not automated. Create a task with schedule 'USING CRON 15 * * * *' to run once per hour runs only once per hour, not every 15 minutes. Create a task without schedule and run it once daily runs once daily, too infrequent.

3. Final Answer:

   Create a task with schedule 'USING CRON */15 * * * *' to automate data copy -> Option D

4. Quick Check:

   CRON '*/15' means every 15 minutes [OK]

Hint: Use '*/15' in CRON for every 15 minutes [OK]

Common Mistakes:

• Choosing manual runs over automation
• Misunderstanding CRON syntax for intervals
• Setting schedule too infrequent