Source freshness checks in dbt - Mini Project: Build & Apply

Source Freshness Checks with dbt

📖 Scenario: You work as a data analyst in a company that relies on daily data updates from various sources. To ensure your reports are accurate, you need to check how fresh the data is in your source tables.Using dbt, you will set up source freshness checks to monitor the last update time of your data sources.

🎯 Goal: Build a dbt project that defines sources and configures freshness checks to monitor the last update timestamps of source tables.

📋 What You'll Learn

Create a source definition for a table named orders in the raw schema

Add a freshness check configuration with warn_after and error_after thresholds

Run the freshness check and output the results

💡 Why This Matters

🌍 Real World

Data teams use source freshness checks to ensure their reports and models rely on up-to-date data, preventing decisions based on stale information.

💼 Career

Knowing how to configure and run source freshness checks is essential for data analysts and engineers to maintain data quality and trust in analytics pipelines.

1

Define the source for the orders table

Create a source named raw with a table called orders. The table has a freshness timestamp column named last_updated. Write this in a YAML file named sources.yml.

dbt

# Define the source 'raw' with table 'orders' and freshness column 'last_updated'
# Your code here

Need a hint?

Use the sources key and define name, tables, and freshness with warn_after and error_after thresholds.

2

Configure freshness thresholds

In the sources.yml file, set the freshness check to warn if data is older than 24 hours and error if older than 48 hours. Use warn_after and error_after with count and period keys.

dbt

version: 2
sources:
  - name: raw
    tables:
      - name: orders
        freshness:
          # Set warn_after threshold
          # Set error_after threshold
        loaded_at_field: last_updated
# Your code here

Need a hint?

Use warn_after and error_after with count and period to set thresholds.

3

Run the freshness check in dbt

Run the dbt command to check source freshness using dbt source freshness in your terminal.

dbt

# Run the freshness check command in your terminal
# Your code here

Need a hint?

Use the command dbt source freshness to run freshness checks.

4

Display the freshness check results

After running dbt source freshness, print the output showing the freshness status of the orders source table.

dbt

# Print the freshness check output below
# Your code here

Need a hint?

Look at the terminal output after running dbt source freshness to see the freshness status.