Source freshness checks in dbt

Introduction

Source freshness checks help you know if your data is up-to-date. They tell you when the data was last updated so you can trust your reports.

You want to make sure your sales data updates every day before running reports.

You need to check if the data from an external system is arriving on time.

You want to alert your team if the data source is delayed or missing updates.

You want to monitor data freshness automatically in your data pipeline.

You want to avoid making decisions based on old or stale data.

Syntax

dbt

sources:
  - name: your_source_name
    tables:
      - name: your_table_name
        freshness:
          warn_after:
            count: 24
            period: hour
          error_after:
            count: 48
            period: hour

warn_after sets when dbt should warn you if data is too old.

error_after sets when dbt should stop the run and show an error if data is too old.

Examples

This checks if sales_data is older than 12 hours to warn, and older than 24 hours to error.

dbt

sources:
  - name: sales_data
    tables:
      - name: sales_data
        freshness:
          warn_after:
            count: 12
            period: hour
          error_after:
            count: 24
            period: hour

This checks if user_events data is older than 1 day to warn, and 2 days to error.

dbt

sources:
  - name: user_events
    tables:
      - name: user_events
        freshness:
          warn_after:
            count: 1
            period: day
          error_after:
            count: 2
            period: day

Sample Program

This dbt source config checks the freshness of the 'orders' table in 'my_source'. It warns if data is older than 6 hours and errors if older than 12 hours.

dbt

version: 2

sources:
  - name: my_source
    tables:
      - name: orders
        freshness:
          warn_after:
            count: 6
            period: hour
          error_after:
            count: 12
            period: hour

Important Notes

Freshness checks require your source tables to have a timestamp column that shows when data was last updated.

You can run freshness checks separately using dbt source freshness command.

Set warn_after and error_after based on how often your data updates in real life.

Summary

Source freshness checks help you track how recent your data is.

Use warn_after and error_after to set thresholds for alerts and errors.

Run freshness checks regularly to keep your data trustworthy.