Model dependencies and parallelism in dbt - Step-by-Step Execution

Concept Flow - Model dependencies and parallelism

Start: Define models

↓

Identify dependencies

↓

Build dependency graph

↓

Find models with no dependencies

↓

Run independent models in parallel

↓

Wait for dependent models

↓

Run dependent models

↓

Repeat until all models built

↓

End: All models built

This flow shows how dbt finds which models depend on others, then runs independent models at the same time, waiting to run dependent models after their dependencies finish.

Execution Sample

dbt

models:
  - name: customers
    depends_on: []
  - name: orders
    depends_on: [customers]
  - name: order_items
    depends_on: [orders]
  - name: products
    depends_on: []

Defines four models with dependencies: 'orders' depends on 'customers', 'order_items' depends on 'orders', 'products' has no dependencies.

Execution Table

Step	Action	Models Run	Dependencies Met?	Parallelism	Notes
1	Identify models with no dependencies	customers, products	Yes	Run both in parallel	Start with independent models
2	Wait for customers and products to finish	customers, products	Yes	Waiting	Dependent models wait
3	Run orders (depends on customers)	orders	customers done	Single run	Orders can run now
4	Wait for orders to finish	orders	Yes	Waiting	order_items waits
5	Run order_items (depends on orders)	order_items	orders done	Single run	Final dependent model
6	All models built	-	-	-	Process complete

💡 All models have been run respecting dependencies and maximizing parallelism.

Variable Tracker

Variable	Start	After Step 1	After Step 3	After Step 5	Final
customers_status	not run	running	done	done	done
products_status	not run	running	done	done	done
orders_status	not run	not run	running	done	done
order_items_status	not run	not run	not run	running	done

Key Moments - 3 Insights

Why can 'customers' and 'products' run at the same time?

Because they have no dependencies, as shown in execution_table step 1, they can run in parallel safely.

Why must 'orders' wait until 'customers' finishes?

Orders depend on customers, so orders can only run after customers are done, as shown in step 3 where dependencies are met.

What happens if a model has multiple dependencies?

It waits until all dependencies are done before running, ensuring data correctness, similar to how 'order_items' waits for 'orders' in step 5.

Visual Quiz - 3 Questions

Look at the execution table, which models run in parallel at step 1?

A customers and products

B orders and order_items

C orders only

D order_items only

💡 Hint

Check the 'Models Run' and 'Parallelism' columns in step 1 of the execution_table.

At which step does 'orders' start running?

A Step 5

B Step 1

C Step 3

D Step 2

💡 Hint

Look at the 'Action' and 'Models Run' columns in the execution_table to find when 'orders' runs.

If 'products' depended on 'customers', how would step 1 change?

A 'orders' would run at step 1

B Only 'customers' would run at step 1

C 'products' would run before 'customers'

D No models would run at step 1

💡 Hint

Consider dependencies and which models have none at step 1 in the execution_table.

Concept Snapshot

dbt runs models respecting dependencies.
Models with no dependencies run in parallel.
Dependent models wait until their dependencies finish.
This speeds up builds and keeps data correct.
Define dependencies clearly to maximize parallelism.

Full Transcript

In dbt, models can depend on other models. The system builds a graph of these dependencies. Models without dependencies run first and can run at the same time, or in parallel. Models that depend on others wait until those finish. This process repeats until all models are built. For example, 'customers' and 'products' run together because they have no dependencies. 'orders' waits for 'customers' to finish. 'order_items' waits for 'orders'. This approach speeds up the build while ensuring data is ready when needed.

Practice

1. What does the {{ ref('model_name') }} function do in dbt models?

easy

A. It sets the number of threads for execution.

B. It creates a dependency on another model, ensuring it runs first.

C. It runs the model in parallel with others.

D. It deletes the referenced model from the project.

Solution

1. Step 1: Understand the purpose of {{ ref() }}

   The {{ ref('model_name') }} function tells dbt that the current model depends on another model named 'model_name'.

2. Step 2: Recognize dependency effect

   This means dbt will run the referenced model first before running the current one, ensuring correct order.

3. Final Answer:

   It creates a dependency on another model, ensuring it runs first. -> Option B

4. Quick Check:

   ref() sets dependencies = C [OK]

Hint: Remember: ref() links models to run in order [OK]

Common Mistakes:

• Thinking ref() controls parallelism
• Confusing ref() with thread settings
• Assuming ref() deletes models

2. Which command line option controls how many models dbt runs at the same time?

easy

A. --models

B. --exclude

C. --full-refresh

D. --threads

Solution

1. Step 1: Identify parallelism control option

   The --threads option tells dbt how many models it can run simultaneously.

2. Step 2: Differentiate from other options

   Other options like --models select which models to run, but do not control parallelism.

3. Final Answer:

   --threads -> Option D

4. Quick Check:

   Parallelism = --threads [OK]

Hint: Threads set parallel runs count in dbt [OK]

Common Mistakes:

• Confusing --models with parallelism control
• Using --full-refresh to control threads
• Thinking --exclude affects parallelism

3. Given these models and dependencies:

model_a.sql (no dependencies)
model_b.sql (depends on model_a)
model_c.sql (depends on model_a)
model_d.sql (depends on model_b and model_c)

If you run dbt run --threads 2, which models can run in parallel first?

medium

A. model_a only

B. model_b and model_c

C. model_a and model_b

D. model_c and model_d

Solution

1. Step 1: Identify models without dependencies

   Only model_a has no dependencies, so it must run first.

2. Step 2: Understand parallelism with threads=2

   Since only model_a is ready, it runs alone first. After it finishes, model_b and model_c can run in parallel.

3. Final Answer:

   model_a only -> Option A

4. Quick Check:

   First run models without dependencies = model_a [OK]

Hint: Run models without dependencies first, then parallelize [OK]

Common Mistakes:

• Running dependent models before their dependencies
• Assuming multiple models run before dependencies finish
• Ignoring dependency order

4. You wrote this dbt model:

select * from {{ ref('model_x') }}

But when running, dbt errors saying model_x does not exist. What is the likely problem?

medium

A. You used --threads with too many threads.

B. The ref function is misspelled.

C. You forgot to create or run model_x before this model.

D. You need to add --full-refresh to run.

Solution

1. Step 1: Understand the error meaning

   The error says model_x does not exist, meaning dbt cannot find it to build dependency.

2. Step 2: Check model creation and run order

   This usually happens if model_x was not created or run before this model, so dependency is missing.

3. Final Answer:

   You forgot to create or run model_x before this model. -> Option C

4. Quick Check:

   Missing dependency model causes error = B [OK]

Hint: Ensure all referenced models exist and run first [OK]

Common Mistakes:

• Blaming thread count for missing model error
• Assuming ref() spelling error without checking
• Thinking --full-refresh fixes missing model

5. You have 5 models with these dependencies:

model_1 (no dependencies)
model_2 (depends on model_1)
model_3 (depends on model_1)
model_4 (depends on model_2)
model_5 (depends on model_3)

If you run dbt run --threads 3, what is the minimum number of sequential steps dbt needs to complete all models?

hard

A. 3

B. 4

C. 5

D. 2

Solution

1. Step 1: Identify dependency layers

   Layer 1: model_1 (no dependencies)
   Layer 2: model_2 and model_3 (depend on model_1)
   Layer 3: model_4 (depends on model_2) and model_5 (depends on model_3)

2. Step 2: Calculate steps with 3 threads

   Step 1: Run model_1 alone
   Step 2: Run model_2 and model_3 in parallel (2 threads used)
   Step 3: Run model_4 and model_5 in parallel (2 threads used)

3. Final Answer:

   3 -> Option A

4. Quick Check:

   Dependency layers = 3 steps [OK]

Hint: Count dependency layers, parallelize within each layer [OK]

Common Mistakes:

• Counting each model as separate step ignoring parallelism
• Ignoring dependency order
• Assuming all models run in one step