Source freshness checks in dbt - Cheat Sheet & Quick Revision

Recall & Review

beginner

What is a source freshness check in dbt?

A source freshness check in dbt is a way to monitor how up-to-date your source data is by checking the age of the newest data record against defined thresholds.

beginner

Which configuration key in dbt defines the freshness thresholds for a source?

The key freshness defines thresholds like warn_after and error_after to set limits on acceptable data age.

intermediate

What happens if the source data is older than the error_after threshold in a freshness check?

dbt will mark the freshness check as failed and raise an error, signaling that the source data is too old and may need attention.

intermediate

How do you define a freshness check for a source table in the sources.yml file?

You add a freshness block under the source with warn_after and error_after times, and specify the column to check for freshness.

beginner

Why are source freshness checks important in data pipelines?

They help ensure that data is updated on time, so downstream analysis and reports use fresh and reliable data, preventing decisions based on stale information.

What does the warn_after threshold in a freshness check do?

A Automatically refreshes the source data

B Stops the dbt run immediately

C Deletes old data from the source

D Triggers a warning if data is older than this time

The warn_after threshold triggers a warning when the source data is older than the specified time but does not fail the run.

Where do you define source freshness checks in dbt?

A In the <code>profiles.yml</code> file

B In the <code>sources.yml</code> file

C In the model SQL files

D In the <code>dbt_project.yml</code> file

Source freshness checks are defined in the sources.yml file where you configure your sources.

What column type is typically used for freshness checks?

A Integer column

B Text column

C Timestamp or date column

D Boolean column

Freshness checks use timestamp or date columns to measure how recent the data is.

If a freshness check fails with an error, what should you do?

A Investigate why the source data is stale and fix the data pipeline

B Ignore the error and continue

C Delete the source table

D Change the <code>warn_after</code> threshold to a higher value

A failed freshness check means the data is too old; you should investigate and fix the pipeline to ensure fresh data.

Which dbt command runs source freshness checks?

A <code>dbt source freshness</code>

B <code>dbt run</code>

C <code>dbt test</code>

D <code>dbt compile</code>

The command dbt source freshness runs the freshness checks on your sources.

Explain how to set up a source freshness check in dbt and why it is useful.

Think about configuration and the purpose of freshness checks.

Describe what happens when source data exceeds the error_after threshold in a freshness check.

Focus on the consequences of stale data detection.