Creating your own dbt package - Mechanics & Internals

Overview - Creating your own dbt package

What is it?

Creating your own dbt package means building a reusable set of data transformation models, tests, and macros that you can share and use across multiple projects. A dbt package is like a mini project inside dbt that others can install easily. It helps organize your SQL code and logic in a clean, modular way. This makes managing and scaling data transformations simpler and more consistent.

Why it matters

Without dbt packages, teams often copy and paste SQL code between projects, leading to errors and inconsistent data logic. Creating your own package solves this by letting you write code once and reuse it everywhere. This saves time, reduces mistakes, and helps teams work together smoothly. It also makes updating logic easier because changes in the package apply to all projects using it.

Where it fits

Before creating your own dbt package, you should understand basic dbt concepts like models, macros, and how to run dbt projects. After learning to create packages, you can explore publishing them publicly, versioning, and advanced package dependency management.

Mental Model

Core Idea

A dbt package is a reusable, shareable container of data transformation logic that you can plug into any dbt project to keep your work consistent and DRY (Don't Repeat Yourself).

Think of it like...

Creating a dbt package is like making a recipe book that you can share with friends. Instead of telling each friend the recipe every time, you give them the book. They can use the recipes anytime, and if you improve a recipe, everyone benefits.

┌─────────────────────┐
│ Your dbt Package    │
│ ┌───────────────┐   │
│ │ Models (SQL)  │   │
│ │ Macros        │   │
│ │ Tests         │   │
│ └───────────────┘   │
└─────────┬───────────┘
          │
          ▼
┌─────────────────────┐
│ Other dbt Projects   │
│ ┌───────────────┐   │
│ │ Use Package   │◄──┘
│ └───────────────┘   │
└─────────────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding dbt Project Structure

Concept: Learn the basic parts of a dbt project to see where packages fit.

A dbt project has folders like models (where SQL lives), macros (reusable SQL snippets), and tests (checks on data). Each project has a dbt_project.yml file that tells dbt how to run. Knowing this helps you organize your package correctly.

Result

You can identify where to put models, macros, and tests when creating a package.

Understanding the project structure is key because a package is just a special kind of dbt project designed for reuse.

2

Foundation

What is a dbt Package?

Concept: A dbt package is a dbt project designed to be shared and reused by other projects.

Instead of writing SQL models directly in your main project, you can create a package with models, macros, and tests. Other projects can then install this package via the packages.yml file and use its contents as if they were their own.

Result

You know that a package is a modular, shareable dbt project.

Seeing a package as a reusable module helps you avoid repeating code and encourages collaboration.

3

Intermediate

Setting Up Your Package Structure

Concept: Learn how to organize files and folders to create a valid dbt package.

Create a new folder for your package. Inside, add a dbt_project.yml file with a unique name and version. Add folders like models/, macros/, and tests/ to hold your SQL and code. Make sure your models have unique names to avoid conflicts.

Result

You have a folder ready to be a dbt package with proper structure.

Proper structure ensures your package works smoothly when installed in other projects.

4

Intermediate

Writing Reusable Models and Macros

🤔Before reading on: Do you think macros can only be used inside the package or also by projects that install it? Commit to your answer.

Concept: Create SQL models and macros that other projects can use directly.

Write SQL files in models/ that define transformations. Write macros in macros/ using Jinja templating to create reusable SQL snippets or logic. When other projects install your package, they can call these macros and use your models as dependencies.

Result

Your package contains reusable code that other projects can call and extend.

Knowing macros are shareable functions unlocks powerful ways to standardize logic across teams.

5

Intermediate

Configuring Package Metadata

Concept: Use dbt_project.yml to define your package's name, version, and dependencies.

In dbt_project.yml, set the 'name' field to your package's unique identifier. Add a 'version' to track releases. Optionally, define dependencies if your package uses other packages. This metadata helps dbt manage your package correctly.

Result

Your package has clear identity and versioning for easy sharing.

Versioning your package prevents breaking changes and helps users upgrade safely.

6

Advanced

Publishing and Installing Your Package

🤔Before reading on: Do you think installing a package requires copying files manually or can it be automated? Commit to your answer.

Concept: Learn how to share your package by publishing it and how others install it via packages.yml.

You can publish your package to a git repository (like GitHub). Other projects add your package's git URL and version to their packages.yml file. Running 'dbt deps' downloads and installs your package automatically. This makes sharing easy and consistent.

Result

Your package is available for others to install and use with a simple command.

Automating package installation saves time and reduces errors compared to manual copying.

7

Expert

Managing Package Dependencies and Conflicts

🤔Before reading on: Can two packages with the same model name coexist without issues? Commit to your answer.

Concept: Understand how dbt handles multiple packages, name conflicts, and dependency trees.

When multiple packages are installed, dbt merges their models and macros. If two packages have models with the same name, dbt raises errors. You can use namespaces or carefully name models to avoid conflicts. Also, packages can depend on other packages, creating a dependency tree that dbt resolves automatically.

Result

You can create complex package setups without breaking projects.

Knowing how dbt resolves dependencies and conflicts helps you design packages that play well with others.

Under the Hood

dbt packages are essentially separate dbt projects with their own dbt_project.yml files. When you run 'dbt deps', dbt clones the package repositories into the 'dbt_modules' folder inside your project. During compilation, dbt merges the SQL models, macros, and tests from your project and all installed packages into one unified DAG (Directed Acyclic Graph). This allows seamless use of package code as if it were local. The package metadata guides version control and dependency resolution.

Why designed this way?

dbt packages were designed to promote code reuse and collaboration across teams and projects. Before packages, teams duplicated SQL code, causing maintenance headaches. Using git repositories for packages leverages existing version control tools and workflows. The merging approach allows flexible composition of multiple packages without changing core dbt behavior.

Your Project
  │
  ├─ dbt_modules/
  │    ├─ package_a/
  │    │    ├─ models/
  │    │    ├─ macros/
  │    │    └─ dbt_project.yml
  │    └─ package_b/
  │         ├─ models/
  │         ├─ macros/
  │         └─ dbt_project.yml
  │
  └─ models/

Compilation Process:
  ┌─────────────────────────────┐
  │ Your Project + Packages      │
  │ Combined DAG of models/macros│
  └─────────────────────────────┘

Myth Busters - 3 Common Misconceptions

Quick: Do you think you must publish your package publicly to use it in your own projects? Commit to yes or no.

Common Belief: You must publish your dbt package publicly (like on GitHub) to use it in any project.

Reality: You can use private or local packages by referencing local paths or private git repos. Publishing publicly is optional.

Why it matters: Believing this limits your ability to share packages securely within your organization or test packages locally.

Quick: Do you think macros inside a package are private and cannot be used by projects that install it? Commit to yes or no.

Common Belief: Macros inside a dbt package are only for internal use and cannot be called from outside projects.

Reality: Macros in packages are fully accessible to projects that install the package and can be called or overridden.

Why it matters: Misunderstanding this prevents teams from creating powerful reusable functions that standardize logic.

Quick: Do you think installing multiple packages with the same model names will work without errors? Commit to yes or no.

Common Belief: dbt allows multiple packages to have models with the same name without any conflict.

Reality: dbt raises errors if two packages have models with the same name because it cannot resolve which to use.

Why it matters: Ignoring this causes build failures and confusion in large projects with many dependencies.

Expert Zone

1

Packages can include hooks and operations that run before or after models, enabling complex orchestration beyond simple SQL transformations.

2

You can override package models or macros in your main project by defining models/macros with the same name, allowing customization without changing the package code.

3

Semantic versioning in packages is critical; minor version bumps can add features without breaking, but major bumps may require careful testing to avoid breaking dependent projects.

When NOT to use

Avoid creating a package if your code is very specific to one project and unlikely to be reused. Instead, keep it inside the project. Also, if your team does not use version control or cannot manage dependencies, packages add complexity without benefit.

Production Patterns

In production, teams create internal package registries or private git repos to share packages securely. They use CI/CD pipelines to test package changes and publish new versions. Packages often include standardized tests and macros to enforce data quality and consistency across projects.

Connections

Software Package Management

dbt packages work like software libraries or packages in programming languages (e.g., Python pip packages).

Understanding software package management helps grasp how dbt packages enable code reuse, versioning, and dependency resolution in data projects.

Modular Programming

Creating dbt packages applies the modular programming principle by breaking code into independent, reusable modules.

Knowing modular programming clarifies why separating logic into packages improves maintainability and collaboration.

Supply Chain Management

Package dependencies in dbt resemble supply chains where components depend on others to deliver a final product.

Seeing package dependencies as supply chains highlights the importance of version control and conflict management to avoid 'broken' data pipelines.

Common Pitfalls

#1 Naming models in your package with generic names that clash with other packages.

Wrong approach:

models/
customer.sql

-- generic model name 'customer' used without namespace

Correct approach:

models/
mypackage_customer.sql

-- prefixed model name to avoid conflicts

Root cause:

Not considering that other packages or projects may have models with the same name causes conflicts during compilation.

#2 Forgetting to add your package to the main project's packages.yml file.

Wrong approach:

# packages.yml is missing your package

# Running dbt deps does not install your package

Correct approach:

packages.yml:
- git: 'https://github.com/yourorg/yourpackage.git'
revision: 0.1.0

Root cause:

Not declaring the package dependency means dbt never downloads or uses your package.

#3 Editing package code directly inside dbt_modules folder in your project.

Wrong approach:

# Editing files inside dbt_modules/yourpackage/models/model.sql
# Changes lost on next dbt deps

Correct approach:

# Edit package source in its own repo/folder
# Then update version and run dbt deps

Root cause:

dbt_modules is a read-only cache of packages; changes here are overwritten and not tracked.

Key Takeaways

Creating your own dbt package lets you write reusable, shareable data transformation code that multiple projects can use.

A package is a special dbt project with its own structure, metadata, and versioning designed for reuse.

Proper naming and versioning prevent conflicts and make upgrading packages safe and predictable.

Publishing packages via git and installing them with packages.yml automates sharing and dependency management.

Understanding package internals and dependency resolution helps avoid common pitfalls and enables advanced use cases.