Data mesh patterns with dbt - Mini Project: Build & Apply

Data Mesh Patterns with dbt

📖 Scenario: You work in a company that uses data mesh principles to organize data ownership by domains. Each domain manages its own data models using dbt. Your task is to create a simple dbt model that follows data mesh patterns by defining a domain-specific source, a configuration variable for freshness, and a core transformation model that applies this freshness threshold.

🎯 Goal: Build a dbt project with a source definition for a domain dataset, a freshness threshold configuration, a transformation model that filters data based on freshness, and finally display the filtered results.

📋 What You'll Learn

Create a source configuration for the 'sales' domain with a table named 'orders'.

Define a freshness threshold variable called freshness_days set to 30.

Write a dbt model SQL that selects orders from the source where the order date is within the freshness threshold.

Print the final filtered orders in the model output.

💡 Why This Matters

🌍 Real World

Data mesh organizes data ownership by domains. Each domain manages its own data models with dbt, enabling scalable and autonomous data pipelines.

💼 Career

Understanding data mesh patterns with dbt is valuable for data engineers and analysts working in modern data platforms that emphasize domain ownership and modular data transformations.

1

Define the source configuration for the sales domain

Create a source configuration in sources.yml with the source name sales and a table named orders. The source should point to the schema raw_sales and database analytics.

dbt

# Define the sales source with orders table in sources.yml
# Your code here

Hint

Use the sources key and define the source with name, database, schema, and tables.

2

Set a freshness threshold variable

In your dbt_project.yml, create a variable called freshness_days and set it to 30.

dbt

version: 2

sources:
  - name: sales
    database: analytics
    schema: raw_sales
    tables:
      - name: orders

# Add the freshness_days variable in dbt_project.yml
# Your code here

Hint

Use the vars section in dbt_project.yml to define variables.

3

Create a dbt model filtering fresh orders

Create a dbt model SQL file named fresh_orders.sql that selects all columns from the source sales.orders where the order_date is within the last freshness_days days. Use the variable freshness_days in your SQL with the syntax {{ var('freshness_days') }}.

dbt

version: 2

sources:
  - name: sales
    database: analytics
    schema: raw_sales
    tables:
      - name: orders

vars:
  freshness_days: 30

-- Create fresh_orders.sql model
-- Your code here

Hint

Use {{ source('sales', 'orders') }} to reference the source and filter by order_date using the variable.

4

Display the filtered fresh orders

Run the dbt model fresh_orders and print the resulting rows to display the fresh orders filtered by the freshness threshold.

dbt

-- fresh_orders.sql
select *
from {{ source('sales', 'orders') }}
where order_date >= current_date - interval '{{ var('freshness_days') }} day'

# Print the results of fresh_orders model
# Your code here

Hint

Use the command dbt run --models fresh_orders to execute and see the filtered data.

Practice

1. What is the main purpose of using data mesh patterns with dbt?

easy

A. To write complex Python scripts for data processing

B. To enable teams to share clean and trusted data easily

C. To create visual dashboards directly in dbt

D. To store raw data without transformation

Solution

1. Step 1: Understand data mesh goals

   Data mesh patterns focus on decentralizing data ownership and sharing clean, trusted data across teams.

2. Step 2: Role of dbt in data mesh

   dbt helps build and test data models, making data reliable and easy to share.

3. Final Answer:

   To enable teams to share clean and trusted data easily -> Option B

4. Quick Check:

   Data mesh = share clean data [OK]

Hint: Data mesh is about sharing clean data across teams [OK]

Common Mistakes:

• Confusing dbt with dashboard tools
• Thinking data mesh stores raw data only
• Assuming dbt runs Python scripts

2. Which of the following is the correct way to reference another model in a dbt SQL file?

easy

A. SELECT * FROM ref('model_name')

B. SELECT * FROM model_name()

C. SELECT * FROM {{ ref('model_name') }}

D. SELECT * FROM $ref.model_name

Solution

1. Step 1: Recall dbt ref syntax

   dbt uses the Jinja function ref() inside double curly braces to reference models.

2. Step 2: Match syntax with options

   SELECT * FROM {{ ref('model_name') }} uses the correct syntax: {{ ref('model_name') }} inside SQL.

3. Final Answer:

   SELECT * FROM {{ ref('model_name') }} -> Option C

4. Quick Check:

   dbt ref() = {{ ref('model') }} [OK]

Hint: Use double curly braces and ref() to reference models [OK]

Common Mistakes:

• Omitting curly braces around ref()
• Using parentheses after model name
• Using incorrect variable syntax

3. Given this dbt model SQL snippet, what will be the output columns?

SELECT user_id, COUNT(*) AS order_count
FROM {{ ref('orders') }}
GROUP BY user_id

medium

A. An error because GROUP BY is missing

B. Only user_id column without counts

C. All columns from orders table without aggregation

D. user_id and order_count columns with counts per user

Solution

1. Step 1: Analyze SELECT and GROUP BY

   The query selects user_id and counts rows grouped by user_id, so output has user_id and order_count.

2. Step 2: Confirm aggregation correctness

   GROUP BY user_id matches the aggregation COUNT(*), so query runs correctly and outputs two columns.

3. Final Answer:

   user_id and order_count columns with counts per user -> Option D

4. Quick Check:

   GROUP BY user_id + COUNT(*) = user_id, order_count [OK]

Hint: GROUP BY columns appear with aggregated counts [OK]

Common Mistakes:

• Thinking all orders columns appear
• Missing GROUP BY causes error
• Ignoring COUNT(*) aggregation

4. You wrote this dbt test to check for nulls:

SELECT * FROM {{ ref('customers') }} WHERE customer_id IS NULL

But it always fails even when nulls exist. What is the likely issue?

medium

A. The test should return zero rows to pass, but it returns rows with nulls

B. The ref function is used incorrectly without quotes

C. The test SQL syntax is invalid and causes failure

D. dbt tests cannot check for null values

Solution

1. Step 1: Understand dbt test logic

   dbt tests pass if the query returns zero rows (no errors found).

2. Step 2: Analyze test query behavior

   This query returns rows where customer_id is null, so if nulls exist, rows are returned and test fails.

3. Final Answer:

   The test should return zero rows to pass, but it returns rows with nulls -> Option A

4. Quick Check:

   dbt test passes if zero rows returned [OK]

Hint: dbt tests pass when query returns zero rows [OK]

Common Mistakes:

• Expecting test to pass when rows are returned
• Misusing ref without quotes
• Thinking dbt can't test nulls

5. In a data mesh using dbt, you want to create a shared model that combines sales data from multiple domains. Which approach best fits data mesh principles?

hard

A. Create a dbt model in a shared repository that references domain models via ref() and applies transformations

B. Copy all domain data into one big table manually outside dbt

C. Write a single SQL file that queries raw source tables directly without using refs

D. Use dbt to create separate models per domain but never combine them

Solution

1. Step 1: Understand data mesh domain ownership

   Each domain owns its data models; shared models combine them via references.

2. Step 2: Use dbt refs to combine domain models

   dbt's ref() function allows building shared models that depend on domain models cleanly.

3. Step 3: Avoid manual copying or bypassing refs

   Manual copying or querying raw tables breaks mesh principles and dbt best practices.

4. Final Answer:

   Create a dbt model in a shared repository that references domain models via ref() and applies transformations -> Option A

5. Quick Check:

   Shared model + ref() = data mesh best practice [OK]

Hint: Use shared dbt models referencing domain models with ref() [OK]

Common Mistakes:

• Copying data manually outside dbt
• Querying raw tables directly
• Not combining domain data at all