Creating your own dbt package

Introduction

Creating your own dbt package helps you share and reuse data transformation code easily across projects.

You want to reuse common SQL models in multiple dbt projects.

You want to share your data transformation logic with your team.

You want to organize your dbt code into smaller, manageable parts.

You want to publish your dbt models as a package for others to use.

You want to maintain consistent data transformations across different environments.

Syntax

dbt

1. Create a new directory for your package.
2. Add a dbt_project.yml file with package details.
3. Add your models, macros, and tests inside the package folder.
4. In your main project, add the package to packages.yml.
5. Run `dbt deps` to install the package.
6. Use the package models and macros in your project.

The dbt_project.yml file defines your package name and version.

The packages.yml file in your main project lists all external packages to use.

Examples

This file sets the package name and default model materialization.

dbt

# dbt_project.yml in your package folder
name: my_dbt_package
version: 0.1.0
config-version: 2

models:
  my_dbt_package:
    +materialized: view

This tells your main project to use the local package you created.

dbt

# packages.yml in your main project
packages:
  - local: ../my_dbt_package

This shows how to reference a model from your package.

dbt

# Using a model from the package in your main project
select * from {{ ref('my_dbt_package.example_model') }}

Sample Program

This example shows how to create a simple dbt package with one model and use it in another project.

dbt

# Step 1: Create package folder 'my_dbt_package'
# Step 2: Add dbt_project.yml inside 'my_dbt_package'
# Content:
# name: my_dbt_package
# version: 0.1.0
# config-version: 2
# models:
#   my_dbt_package:
#     +materialized: view

# Step 3: Add a model file 'models/example_model.sql' inside 'my_dbt_package'
# Content:
# select 1 as id, 'hello' as greeting

# Step 4: In your main dbt project, add packages.yml:
# packages:
#   - local: ../my_dbt_package

# Step 5: Run dbt deps to install the package
# $ dbt deps

# Step 6: Use the package model in your main project model:
# select * from {{ ref('my_dbt_package.example_model') }}

# For demonstration, simulate the output of running the package model:
print('id | greeting')
print('1  | hello')

Important Notes

Always keep your package version updated in dbt_project.yml for clarity.

Use dbt deps to install or update packages after changing packages.yml.

You can publish your package to a git repository for sharing with others.

Summary

Creating a dbt package helps reuse and share data models easily.

Define your package with dbt_project.yml and add it to your main project via packages.yml.

Use dbt deps to install packages and ref() to use package models.

Practice

1. What is the main purpose of creating your own dbt package?

easy

A. To reuse and share data models easily across projects

B. To write SQL queries faster

C. To create dashboards automatically

D. To store raw data files

Solution

1. Step 1: Understand the role of dbt packages

   dbt packages are designed to bundle data models and logic for reuse.

2. Step 2: Identify the main benefit

   Sharing and reusing models saves time and keeps projects consistent.

3. Final Answer:

   To reuse and share data models easily across projects -> Option A

4. Quick Check:

   dbt package purpose = reuse/share models [OK]

Hint: Think: Why share code? To reuse it easily [OK]

Common Mistakes:

• Confusing packages with dashboards
• Thinking packages store raw data
• Believing packages speed up SQL writing only

2. Which file do you edit to add a new dbt package to your main project?

easy

A. models.sql

B. dbt_project.yml

C. profiles.yml

D. packages.yml

Solution

1. Step 1: Recall where packages are declared

   dbt uses packages.yml to list external or local packages.

2. Step 2: Differentiate from other config files

   dbt_project.yml configures the project itself, not packages.

3. Final Answer:

   packages.yml -> Option D

4. Quick Check:

   Package list file = packages.yml [OK]

Hint: Add packages in packages.yml, not dbt_project.yml [OK]

Common Mistakes:

• Adding packages in dbt_project.yml
• Confusing profiles.yml with packages.yml
• Trying to add packages in SQL files

3. Given this packages.yml snippet:

packages:
  - local: ../my_dbt_package

What command installs this package into your project?

easy

A. dbt install

B. dbt build

C. dbt deps

D. dbt run

Solution

1. Step 1: Identify the command for package installation

   dbt uses dbt deps to download and install packages listed in packages.yml.

2. Step 2: Differentiate from other commands

   dbt run runs models, dbt install is not a valid command.

3. Final Answer:

   dbt deps -> Option C

4. Quick Check:

   Install packages with dbt deps [OK]

Hint: Use dbt deps to install packages listed in packages.yml [OK]

Common Mistakes:

• Using dbt install (not a valid command)
• Confusing dbt run with package install
• Trying dbt build instead of dbt deps

4. You created a package but when running dbt deps, you get an error: "Could not find package at path '../my_package'". What is the likely cause?

medium

A. The relative path in packages.yml is incorrect or folder missing

B. You forgot to run dbt run first

C. The package name is missing in dbt_project.yml

D. You need to delete the profiles.yml file

Solution

1. Step 1: Understand the error message

   "Could not find package at path" means dbt cannot locate the folder specified.

2. Step 2: Check the path in packages.yml

   Relative paths must be correct and folder must exist for local packages.

3. Final Answer:

   The relative path in packages.yml is incorrect or folder missing -> Option A

4. Quick Check:

   Path error = wrong folder path [OK]

Hint: Check folder path in packages.yml matches actual location [OK]

Common Mistakes:

• Running dbt run before dbt deps
• Editing dbt_project.yml instead of packages.yml
• Deleting unrelated config files

5. You want to use a model named customer_summary from your custom package my_package. Which is the correct way to reference it in your SQL model?

hard

A. select * from {{ ref('my_package', 'customer_summary') }}

B. select * from {{ ref('my_package.customer_summary') }}

C. select * from {{ ref('customer_summary') }}

D. select * from {{ ref('my_package__customer_summary') }}

Solution

1. Step 1: Recall how to reference models from packages

   Use ref('package_name.model_name') to refer to models in packages.

2. Step 2: Check each option's syntax

   select * from {{ ref('my_package.customer_summary') }} uses the correct single-argument ref syntax with dot notation for packages.

3. Final Answer:

   select * from {{ ref('my_package.customer_summary') }} -> Option B

4. Quick Check:

   Package model ref = ref('package.model') [OK]

Hint: Use ref('package.model') to access package models [OK]

Common Mistakes:

• Using ref('package', 'model') syntax (that's for sources)
• Forgetting the package name in ref
• Using double underscore instead of dot