Why Source freshness checks in dbt? - Purpose & Use Cases

The Big Idea

What if your data was outdated and you didn't even know it?

The Scenario

Imagine you run a small online store and update your sales data every day by copying files manually into your system.

Before making business decisions, you have to check if the data you copied is actually the latest.

You open files, check timestamps, and ask your team if the data is fresh.

The Problem

This manual checking is slow and easy to forget.

You might use old data by mistake, leading to wrong decisions like ordering too much or too little stock.

It's also hard to track when data updates fail or delay.

The Solution

Source freshness checks automatically verify if your data is up-to-date.

They run simple tests that tell you if the data arrived on time or if it's stale.

This saves time, reduces errors, and gives you confidence in your reports.

Before vs After

✗ Before

Check file date manually before loading data

✓ After

dbt source freshness --select my_source

What It Enables

It lets you trust your data pipelines and focus on insights, not on hunting for data problems.

Real Life Example

A marketing team uses source freshness checks to ensure daily campaign data is updated before running performance reports.

If data is late, they get alerts and avoid making decisions on incomplete information.

Key Takeaways

Manual data freshness checks are slow and risky.

Source freshness checks automate this process and catch issues early.

This leads to more reliable data and better business decisions.