dbt-expectations for data quality

Introduction

dbt-expectations helps you check if your data is correct and clean. It makes sure your data follows rules you set.

You want to check if a column has no missing values.

You need to make sure numbers in a column are within a certain range.

You want to confirm that values in a column are unique.

You want to test if text columns match a pattern like email format.

You want to automate data quality checks in your data pipeline.

Syntax

dbt

tests:
  - dbt_expectations.expect_column_values_to_not_be_null:
      column: your_column_name
  - dbt_expectations.expect_column_values_to_be_between:
      column: your_column_name
      min_value: 0
      max_value: 100
  - dbt_expectations.expect_column_values_to_be_unique:
      column: your_column_name

Each test is written under the tests section in your dbt model or schema file.

You specify the test name and the column it applies to, plus any extra parameters like min or max values.

Examples

This test checks that the user_id column has no missing values.

dbt

tests:
  - dbt_expectations.expect_column_values_to_not_be_null:
      column: user_id

This test ensures the age column values are between 0 and 120.

dbt

tests:
  - dbt_expectations.expect_column_values_to_be_between:
      column: age
      min_value: 0
      max_value: 120

This test checks that all emails in the email column are unique.

dbt

tests:
  - dbt_expectations.expect_column_values_to_be_unique:
      column: email

Sample Program

This example shows a dbt schema file defining tests for the customers model. It checks that customer_id is never null and unique, and age is between 0 and 120.

dbt

version: 2
models:
  - name: customers
    columns:
      - name: customer_id
        tests:
          - dbt_expectations.expect_column_values_to_not_be_null
          - dbt_expectations.expect_column_values_to_be_unique
      - name: age
        tests:
          - dbt_expectations.expect_column_values_to_be_between:
              min_value: 0
              max_value: 120

Important Notes

dbt-expectations is built on top of Great Expectations, so it uses similar test ideas but fits into dbt workflows.

You can add many types of tests like checking patterns, value sets, or statistical properties.

Running these tests regularly helps catch data problems early.

Summary

dbt-expectations lets you write simple tests to check your data quality.

You add tests in your dbt schema files under the tests section.

Common tests check for nulls, uniqueness, value ranges, and patterns.