Calling macros across projects in dbt - Deep Dive

Overview - Calling macros across projects

What is it?

Calling macros across projects in dbt means using reusable pieces of SQL or logic defined in one project inside another project. Macros are like small functions that help you avoid repeating code. When you call a macro from another project, you can share common logic across different teams or environments easily. This helps keep your data transformations consistent and easier to maintain.

Why it matters

Without the ability to call macros across projects, teams would have to copy and paste the same code in multiple places. This leads to mistakes, inconsistent results, and more work when updating logic. Sharing macros across projects saves time, reduces errors, and makes collaboration smoother. It helps organizations keep their data pipelines clean and reliable.

Where it fits

Before learning this, you should understand basic dbt projects, how to write macros, and how dbt compiles SQL. After this, you can explore advanced dbt features like packages, hooks, and testing macros across projects.

Mental Model

Core Idea

Calling macros across projects lets you reuse and share SQL logic like functions between different dbt projects to keep your data code DRY and consistent.

Think of it like...

It's like borrowing a recipe from a friend’s cookbook instead of rewriting it yourself every time you want to cook the same dish.

┌───────────────┐       calls       ┌───────────────┐
│ Project A     │──────────────────▶│ Project B     │
│ (defines      │                   │ (calls macro) │
│  macros)      │                   │               │
└───────────────┘                   └───────────────┘

Build-Up - 8 Steps

1

Foundation

Understanding dbt Macros Basics

Concept: Learn what macros are and how they work inside a single dbt project.

Macros in dbt are reusable snippets of SQL or Jinja code. You define them in .sql files inside the macros folder of your project. You can call them in your models or other macros to avoid repeating code. For example, a macro can generate a common filter condition used in many models.

Result

You can write cleaner models by calling macros instead of repeating SQL code.

Understanding macros as reusable code blocks is the foundation for sharing logic across projects.

2

Foundation

How dbt Projects Are Structured

Concept: Know the basic layout of a dbt project and where macros live.

A dbt project has folders like models, macros, and tests. Macros live in the macros folder. Each project has a unique name in the dbt_project.yml file. This name is important when calling macros from other projects.

Result

You can locate and organize macros properly to prepare for cross-project usage.

Knowing project structure helps you understand how dbt finds and uses macros.

3

Intermediate

Calling Macros Within the Same Project

🤔Before reading on: do you think you can call a macro by just its name inside the same project? Commit to your answer.

Concept: Learn the syntax to call macros inside the same project.

Inside your models or other macros, you call a macro using the Jinja syntax: {{ macro_name(arguments) }}. For example, if you have a macro named 'filter_active', you call it as {{ filter_active() }}. This runs the macro and inserts its SQL code.

Result

You can reuse logic inside your project easily by calling macros.

Knowing how to call macros locally is essential before extending to cross-project calls.

4

Intermediate

Using dbt Packages to Share Macros

🤔Before reading on: do you think you can call macros from any project without setup? Commit to your answer.

Concept: Understand that to call macros from another project, you must add that project as a package dependency.

dbt allows projects to depend on other projects via packages. You add the other project to your packages.yml file with its git URL or local path. Then run 'dbt deps' to install it. This makes the other project's macros available to call.

Result

You can share macros by installing other projects as packages.

Recognizing packages as the bridge between projects unlocks cross-project macro calls.

5

Intermediate

Syntax for Calling Macros Across Projects

🤔Before reading on: do you think calling a macro from another project uses the same syntax as local macros? Commit to your answer.

Concept: Learn the special syntax to call macros from another project using the project name as a prefix.

To call a macro from another project, use the syntax: {{ project_name.macro_name(arguments) }}. For example, if the other project is named 'common_macros' and the macro is 'filter_active', call it as {{ common_macros.filter_active() }}. This tells dbt to look in that project's macros.

Result

You can reuse macros defined in other projects by prefixing with the project name.

Understanding the prefix syntax is key to accessing external macros correctly.

6

Advanced

Handling Macro Dependencies and Conflicts

🤔Before reading on: do you think two projects can have macros with the same name without issues? Commit to your answer.

Concept: Learn how dbt resolves macro names when multiple projects have macros with the same name.

If two projects have macros with the same name, you must always use the project prefix to avoid conflicts. Without the prefix, dbt uses the local project's macro by default. Also, macro resolution follows the order of packages in packages.yml. Being explicit avoids unexpected behavior.

Result

You avoid macro name conflicts and ensure the correct macro runs.

Knowing how dbt resolves macro names prevents bugs in complex multi-package setups.

7

Advanced

Testing and Debugging Cross-Project Macros

🤔Before reading on: do you think errors in cross-project macros are easy to trace? Commit to your answer.

Concept: Learn strategies to test and debug macros called across projects.

Use dbt's built-in debug commands and run models that call the macros to see errors. Add logging inside macros with 'log' statements. Also, keep macro logic simple and write unit tests in the macro's own project. This helps isolate issues when macros are shared.

Result

You can find and fix errors in cross-project macros efficiently.

Effective debugging practices save time and reduce frustration in shared macro development.

8

Expert

Advanced Macro Sharing with Versioning and Overrides

🤔Before reading on: do you think you can override a macro from a package in your local project? Commit to your answer.

Concept: Explore how to manage macro versions and override macros from packages for customization.

You can specify package versions in packages.yml to control which macro versions you use. Also, if you define a macro with the same name locally, it overrides the package macro. This allows customization without changing the original package. Use semantic versioning and clear documentation to manage this safely.

Result

You gain control over macro versions and can customize shared macros safely.

Understanding versioning and overrides empowers flexible and maintainable macro sharing.

Under the Hood

dbt compiles SQL by processing Jinja templates. When you call a macro, dbt looks up the macro definition in the current project or installed packages. It replaces the macro call with the macro's SQL code output. For cross-project calls, dbt uses the project prefix to find the macro in the package namespace. This happens during compilation before running SQL on the database.

Why designed this way?

This design keeps macro code modular and reusable while avoiding name clashes. Using packages and prefixes allows clear separation of code ownership. It also supports version control and dependency management, which are essential for large teams and complex projects.

┌───────────────┐
│ Model SQL     │
│ with macro    │
│ call          │
└──────┬────────┘
       │ dbt compiles
       ▼
┌───────────────┐
│ Macro lookup  │
│ - local       │
│ - packages    │
└──────┬────────┘
       │ macro code inserted
       ▼
┌───────────────┐
│ Final SQL     │
│ sent to DB    │
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Can you call any macro from any project without adding it as a package? Commit yes or no.

Common Belief: You can call macros from any project just by using the project name prefix.

Reality: You must add the other project as a package dependency in packages.yml and run 'dbt deps' before calling its macros.

Why it matters: Without adding the package, dbt won't find the macro and will throw an error, blocking your build.

Quick: If two projects have macros with the same name, will dbt automatically pick the right one? Commit yes or no.

Common Belief: dbt automatically resolves macro name conflicts without needing prefixes.

Reality: You must use the project prefix to specify which macro to call; otherwise, dbt uses the local macro by default.

Why it matters: Not using prefixes can cause unexpected macro execution and wrong SQL logic.

Quick: Can you override a macro from a package by defining it locally? Commit yes or no.

Common Belief: Macros from packages cannot be overridden by local macros.

Reality: Local macros with the same name override package macros, allowing customization.

Why it matters: Knowing this helps you customize shared macros safely without modifying package code.

Quick: Do cross-project macro calls affect runtime performance? Commit yes or no.

Common Belief: Calling macros across projects slows down query execution on the database.

Reality: Macro calls happen during dbt compilation, so they do not affect runtime database performance.

Why it matters: Misunderstanding this can lead to unnecessary optimization efforts or avoiding macro reuse.

Expert Zone

1

Macros can accept arguments and return complex SQL snippets, enabling highly dynamic and reusable logic across projects.

2

The order of packages in packages.yml affects macro resolution priority, which can cause subtle bugs if not managed carefully.

3

Macros can call other macros, even across projects, creating layered reusable logic but increasing complexity and debugging difficulty.

When NOT to use

Avoid calling macros across projects when the logic is very project-specific or changes frequently, as this can cause tight coupling. Instead, duplicate simple macros locally or use dbt hooks for project-specific logic.

Production Patterns

Teams create shared macro packages for common transformations like date handling or filtering. They version these packages and use semantic versioning to control upgrades. Overrides are used sparingly to customize behavior per project.

Connections

Software Package Management

Calling macros across projects in dbt is similar to importing libraries or packages in programming languages.

Understanding package dependencies and namespaces in software helps grasp how dbt manages macro sharing and avoids conflicts.

Modular Programming

Macros across projects embody modular programming principles by encapsulating reusable logic in separate modules.

Recognizing this connection highlights the importance of separation of concerns and code reuse in data engineering.

Supply Chain Management

Just like supply chains manage dependencies and versions of parts, dbt manages macro dependencies and versions across projects.

This cross-domain link shows how managing dependencies carefully is crucial in both software and physical goods to avoid failures.

Common Pitfalls

#1 Trying to call a macro from another project without adding it as a package.

Wrong approach:

{{ other_project.macro_name() }} -- without adding package in packages.yml

Correct approach:

Add the other project in packages.yml, run 'dbt deps', then call {{ other_project.macro_name() }}

Root cause:

Misunderstanding that dbt needs explicit package installation to access external macros.

#2 Calling a macro without the project prefix when multiple projects have macros with the same name.

Wrong approach:

{{ macro_name() }} -- ambiguous if macro_name exists in multiple projects

Correct approach:

{{ project_name.macro_name() }} -- explicitly specify which macro to call

Root cause:

Not realizing dbt defaults to local macros and requires prefixes to resolve conflicts.

#3 Overriding a macro unintentionally by defining a local macro with the same name as a package macro.

Wrong approach:

Defining a local macro named 'filter_active' without knowing it exists in a package.

Correct approach:

Rename local macro or explicitly call package macro with prefix to avoid accidental override.

Root cause:

Lack of awareness about macro override behavior in dbt.

Key Takeaways

Macros in dbt are reusable SQL snippets that help avoid repeating code.

To call macros from another project, you must add that project as a package dependency.

Use the syntax {{ project_name.macro_name() }} to call macros across projects and avoid name conflicts.

Local macros override package macros with the same name, enabling customization but requiring caution.

Macro calls happen during compilation, so they do not affect database runtime performance.