Why pipelines automate data freshness in Snowflake - Why It Works

Introduction

Data pipelines help keep your data up-to-date automatically. They run tasks in order to refresh data without manual work. This ensures reports and apps always use fresh information.

When you want your sales dashboard to show today's numbers without manual updates

When you need to combine data from multiple sources regularly without errors

When your marketing team relies on up-to-date customer data for campaigns

When you want to avoid manual steps that can cause delays or mistakes

When you want to schedule data updates during off-hours to save resources

Commands

This command creates a scheduled task in Snowflake that runs every hour. It calls a stored procedure to refresh sales data automatically.

Terminal

snowsql -q "CREATE OR REPLACE TASK refresh_sales_data WAREHOUSE = compute_wh SCHEDULE = 'USING CRON 0 * * * * UTC' AS CALL refresh_sales_procedure();"

Expected Output

Done.



Number of rows affected: 0

→

WAREHOUSE - Specifies the compute resource to run the task

→

SCHEDULE - Defines when the task runs automatically

This command starts the scheduled task so it begins running at the set times.

Terminal

snowsql -q "ALTER TASK refresh_sales_data RESUME;"

Expected Output

Done.



Number of rows affected: 0

This command checks the status of the task to confirm it is active and scheduled correctly.

Terminal

snowsql -q "SHOW TASKS LIKE 'refresh_sales_data';"

Expected Output

name               | database_name | schema_name | owner | state   | schedule           | last_scheduled_time       

refresh_sales_data  | MY_DB        | PUBLIC      | USER  | started | CRON 0 * * * * UTC | 2024-06-01 12:00:00.000 +0000

This command shows the last 5 runs of the task to verify it ran successfully and refreshed data.

Terminal

snowsql -q "SELECT SYSTEM$TASK_HISTORY('refresh_sales_data', 5);"

Expected Output

[{"name":"refresh_sales_data","scheduled_time":"2024-06-01 12:00:00","state":"SUCCEEDED"},{"name":"refresh_sales_data","scheduled_time":"2024-06-01 11:00:00","state":"SUCCEEDED"}]

Key Concept

If you remember nothing else from this pattern, remember: pipelines run tasks automatically on a schedule to keep data fresh without manual effort.

Common Mistakes

Creating a task but forgetting to resume it

The task stays paused and never runs, so data does not refresh.

Always run ALTER TASK task_name RESUME after creating the task.

Setting a schedule that runs too often or too rarely

Running too often wastes resources; too rarely causes stale data.

Choose a schedule that balances freshness with cost, like hourly or daily.

Not checking task history to confirm successful runs

You might miss failures and think data is fresh when it is not.

Regularly query SYSTEM$TASK_HISTORY to monitor task success.

Summary

Create a Snowflake task to run a data refresh procedure on a schedule.

Resume the task so it starts running automatically.

Check the task status and history to ensure it runs successfully and keeps data fresh.