Data mesh patterns with dbt

Introduction

Data mesh helps teams share and manage data easily across an organization. Using dbt makes building and organizing data models simple and clear.

When different teams own different parts of data and want to share clean data.

When you want to build data models that are easy to understand and update.

When you want to automate data transformations and keep data quality high.

When you want to track changes and test data models automatically.

When you want to create a clear structure for data ownership and access.

Syntax

dbt

model_name.sql
-- SQL code for transforming data

models/
  └─ model_name.sql

dbt_project.yml
-- Configuration file for dbt project

schema.yml
-- File to define tests and documentation

dbt uses simple SQL files to build data models step-by-step.

Each model is a SQL file that creates a table or view in your data warehouse.

Examples

This model filters completed orders from raw data.

dbt

-- models/orders.sql
select * from raw.orders where status = 'completed'

This model selects basic customer info for sharing.

dbt

-- models/customers.sql
select id, name, email from raw.customers

This schema.yml file adds tests to check order IDs are unique and not empty.

dbt

version: 2
models:
  - name: orders
    description: 'Cleaned orders data'
    tests:
      - unique:
          column_name: id
      - not_null:
          column_name: id

Sample Program

This example creates a sales summary model using the orders model. It also adds tests to ensure customer IDs are valid and linked to the customers table.

dbt

-- models/sales_summary.sql
select
  customer_id,
  count(*) as total_orders,
  sum(amount) as total_amount
from {{ ref('orders') }}
where order_date >= '2024-01-01'
group by customer_id

-- schema.yml
version: 2
models:
  - name: sales_summary
    description: 'Summary of sales by customer'
    tests:
      - not_null:
          column_name: customer_id
      - relationships:
          column_name: customer_id
          to: ref('customers')
          field: id

Important Notes

Use {{ ref('model_name') }} to link models and build dependencies.

Tests help catch data problems early and keep data reliable.

Organize models by domain to match data mesh ownership.

Summary

Data mesh patterns help teams share clean, trusted data.

dbt makes building and testing data models easy with SQL files.

Use refs and tests to connect models and ensure data quality.

Practice

1. What is the main purpose of using data mesh patterns with dbt?

easy

A. To write complex Python scripts for data processing

B. To enable teams to share clean and trusted data easily

C. To create visual dashboards directly in dbt

D. To store raw data without transformation

Solution

1. Step 1: Understand data mesh goals

   Data mesh patterns focus on decentralizing data ownership and sharing clean, trusted data across teams.

2. Step 2: Role of dbt in data mesh

   dbt helps build and test data models, making data reliable and easy to share.

3. Final Answer:

   To enable teams to share clean and trusted data easily -> Option B

4. Quick Check:

   Data mesh = share clean data [OK]

Hint: Data mesh is about sharing clean data across teams [OK]

Common Mistakes:

• Confusing dbt with dashboard tools
• Thinking data mesh stores raw data only
• Assuming dbt runs Python scripts

2. Which of the following is the correct way to reference another model in a dbt SQL file?

easy

A. SELECT * FROM ref('model_name')

B. SELECT * FROM model_name()

C. SELECT * FROM {{ ref('model_name') }}

D. SELECT * FROM $ref.model_name

Solution

1. Step 1: Recall dbt ref syntax

   dbt uses the Jinja function ref() inside double curly braces to reference models.

2. Step 2: Match syntax with options

   SELECT * FROM {{ ref('model_name') }} uses the correct syntax: {{ ref('model_name') }} inside SQL.

3. Final Answer:

   SELECT * FROM {{ ref('model_name') }} -> Option C

4. Quick Check:

   dbt ref() = {{ ref('model') }} [OK]

Hint: Use double curly braces and ref() to reference models [OK]

Common Mistakes:

• Omitting curly braces around ref()
• Using parentheses after model name
• Using incorrect variable syntax

3. Given this dbt model SQL snippet, what will be the output columns?

SELECT user_id, COUNT(*) AS order_count
FROM {{ ref('orders') }}
GROUP BY user_id

medium

A. An error because GROUP BY is missing

B. Only user_id column without counts

C. All columns from orders table without aggregation

D. user_id and order_count columns with counts per user

Solution

1. Step 1: Analyze SELECT and GROUP BY

   The query selects user_id and counts rows grouped by user_id, so output has user_id and order_count.

2. Step 2: Confirm aggregation correctness

   GROUP BY user_id matches the aggregation COUNT(*), so query runs correctly and outputs two columns.

3. Final Answer:

   user_id and order_count columns with counts per user -> Option D

4. Quick Check:

   GROUP BY user_id + COUNT(*) = user_id, order_count [OK]

Hint: GROUP BY columns appear with aggregated counts [OK]

Common Mistakes:

• Thinking all orders columns appear
• Missing GROUP BY causes error
• Ignoring COUNT(*) aggregation

4. You wrote this dbt test to check for nulls:

SELECT * FROM {{ ref('customers') }} WHERE customer_id IS NULL

But it always fails even when nulls exist. What is the likely issue?

medium

A. The test should return zero rows to pass, but it returns rows with nulls

B. The ref function is used incorrectly without quotes

C. The test SQL syntax is invalid and causes failure

D. dbt tests cannot check for null values

Solution

1. Step 1: Understand dbt test logic

   dbt tests pass if the query returns zero rows (no errors found).

2. Step 2: Analyze test query behavior

   This query returns rows where customer_id is null, so if nulls exist, rows are returned and test fails.

3. Final Answer:

   The test should return zero rows to pass, but it returns rows with nulls -> Option A

4. Quick Check:

   dbt test passes if zero rows returned [OK]

Hint: dbt tests pass when query returns zero rows [OK]

Common Mistakes:

• Expecting test to pass when rows are returned
• Misusing ref without quotes
• Thinking dbt can't test nulls

5. In a data mesh using dbt, you want to create a shared model that combines sales data from multiple domains. Which approach best fits data mesh principles?

hard

A. Create a dbt model in a shared repository that references domain models via ref() and applies transformations

B. Copy all domain data into one big table manually outside dbt

C. Write a single SQL file that queries raw source tables directly without using refs

D. Use dbt to create separate models per domain but never combine them

Solution

1. Step 1: Understand data mesh domain ownership

   Each domain owns its data models; shared models combine them via references.

2. Step 2: Use dbt refs to combine domain models

   dbt's ref() function allows building shared models that depend on domain models cleanly.

3. Step 3: Avoid manual copying or bypassing refs

   Manual copying or querying raw tables breaks mesh principles and dbt best practices.

4. Final Answer:

   Create a dbt model in a shared repository that references domain models via ref() and applies transformations -> Option A

5. Quick Check:

   Shared model + ref() = data mesh best practice [OK]

Hint: Use shared dbt models referencing domain models with ref() [OK]

Common Mistakes:

• Copying data manually outside dbt
• Querying raw tables directly
• Not combining domain data at all