Creating your own dbt package - Performance & Efficiency

Time Complexity: Creating your own dbt package

O(n)

Understanding Time Complexity

When creating your own dbt package, it's important to understand how the time to run your models grows as you add more models or data.

We want to know how the execution time changes when the package size or data size increases.

Scenario Under Consideration

Analyze the time complexity of the following dbt package structure.

-- models/my_package/model_a.sql
select * from source_table;

-- models/my_package/model_b.sql
select * from {{ ref('model_a') }} where condition = 'value';

-- models/my_package/model_c.sql
select * from {{ ref('model_b') }} join another_table on key = key;

This package has three models where each model depends on the previous one, building on the data step by step.

Identify Repeating Operations

Look at what repeats when dbt runs this package.

• Primary operation: Running each model's SQL query once in order.
• How many times: Once per model, so three times here.

How Execution Grows With Input

As you add more models to your package, dbt runs each one in sequence.

Input Size (models)	Approx. Operations (queries run)
3	3
10	10
100	100

Pattern observation: The total work grows directly with the number of models you have.

Final Time Complexity

Time Complexity: O(n)

This means the time to run your package grows linearly with the number of models you include.

Common Mistake

[X] Wrong: "Adding more models won't affect run time much because dbt runs them fast."

[OK] Correct: Each model runs a query, so more models mean more queries and longer total run time.

Interview Connect

Understanding how your dbt package scales helps you design efficient data workflows and shows you can think about performance as your projects grow.

Self-Check

"What if your models had multiple dependencies and some ran in parallel? How would that affect the time complexity?"

Practice

1. What is the main purpose of creating your own dbt package?

easy

A. To reuse and share data models easily across projects

B. To write SQL queries faster

C. To create dashboards automatically

D. To store raw data files

Solution

1. Step 1: Understand the role of dbt packages

   dbt packages are designed to bundle data models and logic for reuse.

2. Step 2: Identify the main benefit

   Sharing and reusing models saves time and keeps projects consistent.

3. Final Answer:

   To reuse and share data models easily across projects -> Option A

4. Quick Check:

   dbt package purpose = reuse/share models [OK]

Hint: Think: Why share code? To reuse it easily [OK]

Common Mistakes:

• Confusing packages with dashboards
• Thinking packages store raw data
• Believing packages speed up SQL writing only

2. Which file do you edit to add a new dbt package to your main project?

easy

A. models.sql

B. dbt_project.yml

C. profiles.yml

D. packages.yml

Solution

1. Step 1: Recall where packages are declared

   dbt uses packages.yml to list external or local packages.

2. Step 2: Differentiate from other config files

   dbt_project.yml configures the project itself, not packages.

3. Final Answer:

   packages.yml -> Option D

4. Quick Check:

   Package list file = packages.yml [OK]

Hint: Add packages in packages.yml, not dbt_project.yml [OK]

Common Mistakes:

• Adding packages in dbt_project.yml
• Confusing profiles.yml with packages.yml
• Trying to add packages in SQL files

3. Given this packages.yml snippet:

packages:
  - local: ../my_dbt_package

What command installs this package into your project?

easy

A. dbt install

B. dbt build

C. dbt deps

D. dbt run

Solution

1. Step 1: Identify the command for package installation

   dbt uses dbt deps to download and install packages listed in packages.yml.

2. Step 2: Differentiate from other commands

   dbt run runs models, dbt install is not a valid command.

3. Final Answer:

   dbt deps -> Option C

4. Quick Check:

   Install packages with dbt deps [OK]

Hint: Use dbt deps to install packages listed in packages.yml [OK]

Common Mistakes:

• Using dbt install (not a valid command)
• Confusing dbt run with package install
• Trying dbt build instead of dbt deps

4. You created a package but when running dbt deps, you get an error: "Could not find package at path '../my_package'". What is the likely cause?

medium

A. The relative path in packages.yml is incorrect or folder missing

B. You forgot to run dbt run first

C. The package name is missing in dbt_project.yml

D. You need to delete the profiles.yml file

Solution

1. Step 1: Understand the error message

   "Could not find package at path" means dbt cannot locate the folder specified.

2. Step 2: Check the path in packages.yml

   Relative paths must be correct and folder must exist for local packages.

3. Final Answer:

   The relative path in packages.yml is incorrect or folder missing -> Option A

4. Quick Check:

   Path error = wrong folder path [OK]

Hint: Check folder path in packages.yml matches actual location [OK]

Common Mistakes:

• Running dbt run before dbt deps
• Editing dbt_project.yml instead of packages.yml
• Deleting unrelated config files

5. You want to use a model named customer_summary from your custom package my_package. Which is the correct way to reference it in your SQL model?

hard

A. select * from {{ ref('my_package', 'customer_summary') }}

B. select * from {{ ref('my_package.customer_summary') }}

C. select * from {{ ref('customer_summary') }}

D. select * from {{ ref('my_package__customer_summary') }}

Solution

1. Step 1: Recall how to reference models from packages

   Use ref('package_name.model_name') to refer to models in packages.

2. Step 2: Check each option's syntax

   select * from {{ ref('my_package.customer_summary') }} uses the correct single-argument ref syntax with dot notation for packages.

3. Final Answer:

   select * from {{ ref('my_package.customer_summary') }} -> Option B

4. Quick Check:

   Package model ref = ref('package.model') [OK]

Hint: Use ref('package.model') to access package models [OK]

Common Mistakes:

• Using ref('package', 'model') syntax (that's for sources)
• Forgetting the package name in ref
• Using double underscore instead of dot