Model dependencies and parallelism in dbt

Introduction

Model dependencies help dbt know which data models rely on others. Parallelism lets dbt run independent models at the same time to save time.

When you have multiple data models where some need results from others before running.

When you want to speed up your data build by running models that don't depend on each other at the same time.

When you want to avoid errors by making sure models run in the right order.

When you want to organize your data pipeline clearly by showing how models connect.

When you want to optimize resource use by running many models in parallel safely.

Syntax

dbt

# model_a.sql
select * from source_table

# model_b.sql
select * from {{ ref('model_a') }} where condition

# model_c.sql
select * from {{ ref('model_a') }}

# model_d.sql
select * from {{ ref('model_b') }} join {{ ref('model_c') }} on ...

# Run command with parallelism:
dbt run --models model_a model_b model_c model_d --threads 4

Use {{ ref('model_name') }} to tell dbt that one model depends on another.

The --threads option controls how many models run at the same time.

Examples

Here, active_customers depends on base_customers. dbt runs base_customers first.

dbt

# Example 1: Simple dependency
# base_customers.sql
select * from raw.customers

# active_customers.sql
select * from {{ ref('base_customers') }} where active = true

Since sales_data and product_data don't depend on each other, dbt can run them in parallel.

dbt

# Example 2: Parallel models with no dependencies
# sales_data.sql
select * from raw.sales

# product_data.sql
select * from raw.products

If you run dbt run without specifying models, dbt runs all models respecting dependencies.

dbt

# Example 3: Edge case - empty model list
# Running dbt with no models specified runs all models in dependency order.

A single model with no dependencies runs immediately and alone.

dbt

# Example 4: Single model with no dependencies
# standalone_model.sql
select 1 as id

Sample Program

This example shows four models with dependencies. Model B and C depend on A. Model D depends on B and C. Running with 4 threads lets dbt run independent models at the same time.

dbt

# dbt_project.yml
name: 'my_project'
version: '1.0'

# models/my_project/model_a.sql
select 1 as id

# models/my_project/model_b.sql
select id from {{ ref('model_a') }} where id = 1

# models/my_project/model_c.sql
select id from {{ ref('model_a') }}

# models/my_project/model_d.sql
select b.id from {{ ref('model_b') }} b join {{ ref('model_c') }} c on b.id = c.id

# Command to run models with parallelism
# dbt run --threads 4

# Expected output in terminal:
# Running with dbt=1.x.x
# Found 4 models, 0 tests, 0 snapshots, 0 analyses, 0 macros
# 
# 1 of 4 START table model my_project.model_a................ [RUN]
# 1 of 4 OK created table model my_project.model_a........... [OK in 0.1s]
# 2 of 4 START table model my_project.model_b................ [RUN]
# 3 of 4 START table model my_project.model_c................ [RUN]
# 2 of 4 OK created table model my_project.model_b........... [OK in 0.1s]
# 3 of 4 OK created table model my_project.model_c........... [OK in 0.1s]
# 4 of 4 START table model my_project.model_d................ [RUN]
# 4 of 4 OK created table model my_project.model_d........... [OK in 0.1s]
# 
# Finished running 4 table models in 1.0s.

Important Notes

Time complexity depends on the number of models and their dependencies; dbt builds a graph to manage order.

Space complexity is mainly the storage used by the models' data.

Common mistake: forgetting to use {{ ref() }} causes dbt to run models in wrong order or fail.

Use parallelism to speed up builds when models are independent; use dependencies to ensure correct order.

Summary

Model dependencies tell dbt which models need others to run first.

Parallelism lets dbt run models without dependencies at the same time to save time.

Use {{ ref('model_name') }} to set dependencies and --threads to control parallel runs.

Practice

1. What does the {{ ref('model_name') }} function do in dbt models?

easy

A. It sets the number of threads for execution.

B. It creates a dependency on another model, ensuring it runs first.

C. It runs the model in parallel with others.

D. It deletes the referenced model from the project.

Solution

1. Step 1: Understand the purpose of {{ ref() }}

   The {{ ref('model_name') }} function tells dbt that the current model depends on another model named 'model_name'.

2. Step 2: Recognize dependency effect

   This means dbt will run the referenced model first before running the current one, ensuring correct order.

3. Final Answer:

   It creates a dependency on another model, ensuring it runs first. -> Option B

4. Quick Check:

   ref() sets dependencies = C [OK]

Hint: Remember: ref() links models to run in order [OK]

Common Mistakes:

• Thinking ref() controls parallelism
• Confusing ref() with thread settings
• Assuming ref() deletes models

2. Which command line option controls how many models dbt runs at the same time?

easy

A. --models

B. --exclude

C. --full-refresh

D. --threads

Solution

1. Step 1: Identify parallelism control option

   The --threads option tells dbt how many models it can run simultaneously.

2. Step 2: Differentiate from other options

   Other options like --models select which models to run, but do not control parallelism.

3. Final Answer:

   --threads -> Option D

4. Quick Check:

   Parallelism = --threads [OK]

Hint: Threads set parallel runs count in dbt [OK]

Common Mistakes:

• Confusing --models with parallelism control
• Using --full-refresh to control threads
• Thinking --exclude affects parallelism

3. Given these models and dependencies:

model_a.sql (no dependencies)
model_b.sql (depends on model_a)
model_c.sql (depends on model_a)
model_d.sql (depends on model_b and model_c)

If you run dbt run --threads 2, which models can run in parallel first?

medium

A. model_a only

B. model_b and model_c

C. model_a and model_b

D. model_c and model_d

Solution

1. Step 1: Identify models without dependencies

   Only model_a has no dependencies, so it must run first.

2. Step 2: Understand parallelism with threads=2

   Since only model_a is ready, it runs alone first. After it finishes, model_b and model_c can run in parallel.

3. Final Answer:

   model_a only -> Option A

4. Quick Check:

   First run models without dependencies = model_a [OK]

Hint: Run models without dependencies first, then parallelize [OK]

Common Mistakes:

• Running dependent models before their dependencies
• Assuming multiple models run before dependencies finish
• Ignoring dependency order

4. You wrote this dbt model:

select * from {{ ref('model_x') }}

But when running, dbt errors saying model_x does not exist. What is the likely problem?

medium

A. You used --threads with too many threads.

B. The ref function is misspelled.

C. You forgot to create or run model_x before this model.

D. You need to add --full-refresh to run.

Solution

1. Step 1: Understand the error meaning

   The error says model_x does not exist, meaning dbt cannot find it to build dependency.

2. Step 2: Check model creation and run order

   This usually happens if model_x was not created or run before this model, so dependency is missing.

3. Final Answer:

   You forgot to create or run model_x before this model. -> Option C

4. Quick Check:

   Missing dependency model causes error = B [OK]

Hint: Ensure all referenced models exist and run first [OK]

Common Mistakes:

• Blaming thread count for missing model error
• Assuming ref() spelling error without checking
• Thinking --full-refresh fixes missing model

5. You have 5 models with these dependencies:

model_1 (no dependencies)
model_2 (depends on model_1)
model_3 (depends on model_1)
model_4 (depends on model_2)
model_5 (depends on model_3)

If you run dbt run --threads 3, what is the minimum number of sequential steps dbt needs to complete all models?

hard

A. 3

B. 4

C. 5

D. 2

Solution

1. Step 1: Identify dependency layers

   Layer 1: model_1 (no dependencies)
   Layer 2: model_2 and model_3 (depend on model_1)
   Layer 3: model_4 (depends on model_2) and model_5 (depends on model_3)

2. Step 2: Calculate steps with 3 threads

   Step 1: Run model_1 alone
   Step 2: Run model_2 and model_3 in parallel (2 threads used)
   Step 3: Run model_4 and model_5 in parallel (2 threads used)

3. Final Answer:

   3 -> Option A

4. Quick Check:

   Dependency layers = 3 steps [OK]

Hint: Count dependency layers, parallelize within each layer [OK]

Common Mistakes:

• Counting each model as separate step ignoring parallelism
• Ignoring dependency order
• Assuming all models run in one step