Source freshness checks in dbt - Interactive Code Practice

Practice - 5 Tasks

1fill in blank

easy

Complete the code to define a source freshness check in dbt.

dbt

sources:
  - name: my_source
    freshness:
      [1]: 24 hours

Drag options to blanks, or click blank then click option'

Awarn_after

Berror_after

Cfreshness_check

Dmax_age

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'error_after' instead of 'warn_after' for warning threshold.

Confusing 'max_age' with freshness config keys.

The warn_after key defines the time after which dbt will warn if the source data is stale.

2fill in blank

medium

Complete the code to set an error threshold for source freshness in dbt.

dbt

sources:
  - name: my_source
    freshness:
      warn_after: 24 hours
      [1]: 48 hours

Drag options to blanks, or click blank then click option'

Aerror_after

Bfail_after

Cmax_age

Dalert_after

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'fail_after' which is not a valid key.

Confusing 'max_age' with error threshold.

The error_after key sets the time after which dbt will fail the run if the source data is too stale.

3fill in blank

hard

Fix the error in the freshness check configuration.

dbt

sources:
  - name: sales_data
    freshness:
      warn_after: 12 hours
      [1]: 6 hours

Drag options to blanks, or click blank then click option'

Afail_after

Bwarn_after

Cmax_age

Derror_after

Attempts:

3 left

💡 Hint

Common Mistakes

Setting 'error_after' to a smaller duration than 'warn_after'.

Using invalid keys like 'fail_after'.

The error_after value must be greater than warn_after. Here, 6 hours is less than 12 hours, so the error threshold should be set correctly.

4fill in blank

hard

Fill both blanks to create a source freshness check with a warning after 1 day and error after 2 days.

dbt

sources:
  - name: user_data
    freshness:
      [1]: 1 day
      [2]: 2 days

Drag options to blanks, or click blank then click option'

Awarn_after

Berror_after

Cmax_age

Dfreshness_check

Attempts:

3 left

💡 Hint

Common Mistakes

Swapping 'warn_after' and 'error_after' values.

Using invalid keys like 'max_age' or 'freshness_check'.

warn_after sets the warning threshold, and error_after sets the error threshold for freshness checks.

5fill in blank

hard

Fill all three blanks to define a source freshness check with a warning after 6 hours, error after 12 hours, and a filter condition.

dbt

sources:
  - name: orders
    freshness:
      [1]: 6 hours
      [2]: 12 hours
      filter: "[3]"

Drag options to blanks, or click blank then click option'

Awarn_after

Berror_after

Cstatus = 'active'

Dmax_age

Attempts:

3 left

💡 Hint

Common Mistakes

Using 'max_age' instead of 'warn_after' or 'error_after'.

Not quoting the filter condition properly.

The warn_after and error_after keys set freshness thresholds, and the filter applies a condition to the source data freshness check.

Practice

1. What is the main purpose of source freshness checks in dbt?

easy

A. To track how recent the data in your source tables is

B. To create new tables from raw data

C. To optimize SQL query performance

D. To schedule dbt runs automatically

Solution

1. Step 1: Understand the role of freshness checks

   Freshness checks monitor the age of data in source tables to ensure it is up-to-date.

2. Step 2: Compare options to the purpose

   Only To track how recent the data in your source tables is describes tracking data recency, which matches the purpose of freshness checks.

3. Final Answer:

   To track how recent the data in your source tables is -> Option A

4. Quick Check:

   Freshness checks = track data recency [OK]

Hint: Freshness checks measure data age, not table creation or scheduling [OK]

Common Mistakes:

• Confusing freshness checks with table creation
• Thinking freshness checks optimize queries
• Assuming freshness checks schedule runs

2. Which of the following is the correct way to set a freshness check with a warning threshold of 1 day and an error threshold of 2 days in dbt YAML?

easy

A. freshness:
warn_after: 1 day
error_after: 2 day

B. freshness:
warn_after: {count: 1, period: day}
error_after: {count: 2, period: day}

C. freshness:
warn_after: '1 day'
error_after: '2 days'

D. freshness:
warn_after: {count: 2, period: day}
error_after: {count: 1, period: day}

Solution

1. Step 1: Recall correct YAML syntax for freshness

   dbt expects warn_after and error_after as objects with count and period keys.

2. Step 2: Match options to syntax

   freshness: warn_after: {count: 1, period: day} error_after: {count: 2, period: day} correctly uses {count: X, period: day} format; others use incorrect formats or swap thresholds.

3. Final Answer:

   freshness: warn_after: {count: 1, period: day} error_after: {count: 2, period: day} -> Option B

4. Quick Check:

   Use count and period keys in YAML freshness [OK]

Hint: Use {count: X, period: day} format for freshness thresholds [OK]

Common Mistakes:

• Using strings instead of objects for thresholds
• Swapping warn_after and error_after values
• Missing count or period keys

3. Given this freshness check result output, what is the status if the last loaded timestamp is 3 days ago, warn_after is 1 day, and error_after is 2 days?

{"status": "", "max_loaded_at": "2024-04-20T00:00:00Z"}

medium

A. error

B. warn

C. pass

D. unknown

Solution

1. Step 1: Calculate data age from last loaded timestamp

   If today is 2024-04-23, data is 3 days old (2024-04-23 - 2024-04-20).

2. Step 2: Compare data age to thresholds

   3 days > error_after (2 days), so status is error.

3. Final Answer:

   error -> Option A

4. Quick Check:

   Data age > error_after = error status [OK]

Hint: If data age > error_after, status is error [OK]

Common Mistakes:

• Confusing warn_after and error_after thresholds
• Assuming status is warn for data older than error_after
• Ignoring current date when calculating age

4. You wrote this freshness check YAML but it fails to run:

sources:
  - name: my_source
    freshness:
      warn_after: {count: 1, period: day}
      error_after: {count: 2, period: days}

What is the likely cause of the error?

medium

A. The count values must be strings, not numbers

B. Missing quotes around the period values

C. warn_after and error_after keys are swapped

D. The period value 'days' should be singular 'day'

Solution

1. Step 1: Check period values in freshness YAML

   dbt expects period values as singular strings like 'day', not plural 'days'.

2. Step 2: Identify error cause

   Using 'days' causes a validation error; changing to 'day' fixes it.

3. Final Answer:

   The period value 'days' should be singular 'day' -> Option D

4. Quick Check:

   Period values must be singular like 'day' [OK]

Hint: Use singular period names like 'day', not 'days' [OK]

Common Mistakes:

• Using plural period names
• Swapping warn_after and error_after
• Adding unnecessary quotes around numbers

5. You want to set up a freshness check for a source table that updates hourly. You want to warn if data is older than 2 hours and error if older than 4 hours. Which YAML snippet correctly sets this up?

hard

A. freshness:
warn_after: {count: '2', period: hour}
error_after: {count: '4', period: hour}

B. freshness:
warn_after: {count: 2, period: hours}
error_after: {count: 4, period: hours}

C. freshness:
warn_after: {count: 2, period: hour}
error_after: {count: 4, period: hour}

D. freshness:
warn_after: {count: 4, period: hour}
error_after: {count: 2, period: hour}

Solution

1. Step 1: Identify correct period and count values

   Period should be singular 'hour', counts are numbers without quotes.

2. Step 2: Check warn_after and error_after order

   warn_after must be less than error_after; 2 < 4 is correct.

3. Step 3: Validate options

   freshness: warn_after: {count: 2, period: hour} error_after: {count: 4, period: hour} matches correct syntax and logic; A uses strings for counts, B uses plural 'hours', D swaps thresholds.

4. Final Answer:

   freshness: warn_after: {count: 2, period: hour} error_after: {count: 4, period: hour} -> Option C

5. Quick Check:

   Use singular period and correct threshold order [OK]

Hint: Use singular period and warn_after < error_after [OK]

Common Mistakes:

• Using plural period names like 'hours'
• Putting counts as strings instead of numbers
• Swapping warn_after and error_after values