Environment management (dev, staging, prod) in dbt - Step-by-Step Execution

Concept Flow - Environment management (dev, staging, prod)

Start: Choose Environment

↓

Use dev settings

No↓

Use staging settings

No↓

Use prod settings

↓

Run dbt commands with chosen environment

↓

Deploy or test changes

↓

End

This flow shows how dbt selects settings based on environment (dev, staging, prod) and runs commands accordingly.

Execution Sample

dbt

profiles:
  my_project:
    target: dev
    outputs:
      dev:
        type: snowflake
        ...
      staging:
        type: snowflake
        ...
      prod:
        type: snowflake
        ...

This dbt profile configures three environments with different connection settings.

Execution Table

Step	Environment Variable	Target Selected	Action	Result
1	DBT_ENV=dev	dev	Load dev settings	Connect to dev database
2	Run dbt run	dev	Compile and run models	Models built in dev schema
3	DBT_ENV=staging	staging	Load staging settings	Connect to staging database
4	Run dbt test	staging	Run tests on staging	Tests executed on staging schema
5	DBT_ENV=prod	prod	Load prod settings	Connect to production database
6	Run dbt run	prod	Compile and run models	Models built in prod schema
7	DBT_ENV=unknown	prod (default)	Load prod settings	Connect to production database
8	Run dbt run	prod	Compile and run models	Models built in prod schema

💡 Execution stops after running commands in the selected environment.

Variable Tracker

Variable	Start	After Step 1	After Step 3	After Step 5	After Step 7
DBT_ENV	undefined	dev	staging	prod	unknown
target	undefined	dev	staging	prod	prod (default)
connection	undefined	dev connection	staging connection	prod connection	prod connection

Key Moments - 3 Insights

Why does dbt use the prod environment when DBT_ENV is unknown?

dbt defaults to the 'prod' target if the environment variable is not set or unknown, as shown in execution_table row 7.

How does dbt know which database to connect to?

dbt reads the target environment from DBT_ENV and loads the matching profile settings, as seen in steps 1, 3, and 5.

Can you run tests in the dev environment?

Yes, you can run tests in any environment by setting DBT_ENV accordingly and running 'dbt test', similar to step 4 for staging.

Visual Quiz - 3 Questions

Look at the execution_table at step 3. What environment is selected?

A staging

B prod

C dev

D unknown

💡 Hint

Check the 'Target Selected' column at step 3 in the execution_table.

At which step does dbt run models in the production environment?

A Step 2

B Step 6

C Step 4

D Step 8

💡 Hint

Look for 'prod' in 'Target Selected' and 'Run dbt run' in 'Action' column.

If DBT_ENV is set to 'dev', what will be the connection after step 1 according to variable_tracker?

A staging connection

B prod connection

C dev connection

D undefined

💡 Hint

Check 'connection' variable value after step 1 in variable_tracker.

Concept Snapshot

Environment management in dbt:
- Use DBT_ENV variable to select environment
- Profiles.yml defines dev, staging, prod targets
- dbt commands run with selected target
- Defaults to prod if unknown
- Enables safe testing and deployment

Full Transcript

This visual execution shows how dbt manages environments like dev, staging, and prod. It starts by reading the DBT_ENV variable to select the target environment. Based on this, dbt loads the corresponding connection settings from profiles.yml. Then, dbt runs commands like 'dbt run' or 'dbt test' in the chosen environment. If DBT_ENV is not set or unknown, dbt defaults to the production environment to avoid accidental use of wrong settings. Variables like DBT_ENV, target, and connection update step-by-step as commands execute. This helps keep development, testing, and production separate and safe.

Practice

1. What is the main purpose of having separate environments like dev, staging, and prod in dbt?

easy

A. To run multiple dbt projects at the same time

B. To safely develop and test changes before applying them to production

C. To store different versions of dbt itself

D. To backup data automatically

Solution

1. Step 1: Understand environment roles

   Development (dev) is for building and testing changes. Staging is for final testing. Production (prod) is the live environment.

2. Step 2: Purpose of separation

   This separation prevents untested changes from affecting live data and users.

3. Final Answer:

   To safely develop and test changes before applying them to production -> Option B

4. Quick Check:

   Environment separation = safe testing before production [OK]

Hint: Think: Why not test directly in production? [OK]

Common Mistakes:

• Confusing environment roles
• Thinking environments store dbt versions
• Assuming environments backup data automatically

2. Which command correctly runs dbt models in the staging environment?

easy

A. dbt run --profile staging

B. dbt run --env staging

C. dbt run --target staging

D. dbt run --environment staging

Solution

1. Step 1: Identify the correct flag for environment selection

   dbt uses the --target flag to specify which environment to run against.

2. Step 2: Match the flag with the environment name

   Using --target staging runs models in the staging environment.

3. Final Answer:

   dbt run --target staging -> Option C

4. Quick Check:

   Use --target to switch environments [OK]

Hint: Remember: --target selects environment in dbt commands [OK]

Common Mistakes:

• Using --env or --environment instead of --target
• Confusing --profile with environment selection
• Omitting the flag and running default environment

3. Given the following profiles.yml snippet, what database will dbt connect to when running with --target prod?

my_project:
  outputs:
    dev:
      type: postgres
      host: dev-db.example.com
      dbname: dev_db
    prod:
      type: postgres
      host: prod-db.example.com
      dbname: prod_db
  target: dev

medium

A. prod_db on prod-db.example.com

B. dev_db on dev-db.example.com

C. dev_db on prod-db.example.com

D. prod_db on dev-db.example.com

Solution

1. Step 1: Understand the --target flag effect

   Running with --target prod tells dbt to use the prod output configuration.

2. Step 2: Identify prod output details

   The prod output connects to prod-db.example.com and database prod_db.

3. Final Answer:

   prod_db on prod-db.example.com -> Option A

4. Quick Check:

   --target prod uses prod output config [OK]

Hint: Match --target value to profiles.yml output key [OK]

Common Mistakes:

• Ignoring --target and using default target
• Mixing host and dbname from different environments
• Assuming target in profiles.yml overrides command line

4. You run dbt run --target staging but get a connection error. The profiles.yml has a typo in the staging host key as hst instead of host. What is the best fix?

medium

A. Run dbt clean before running again

B. Change the command to dbt run --target hst

C. Remove the staging environment from profiles.yml

D. Correct the typo from hst to host in profiles.yml

Solution

1. Step 1: Identify cause of connection error

   A typo in the key hst means dbt cannot find the host to connect to.

2. Step 2: Fix the typo in profiles.yml

   Changing hst to host corrects the connection details.

3. Final Answer:

   Correct the typo from hst to host in profiles.yml -> Option D

4. Quick Check:

   Fix config typos to resolve connection errors [OK]

Hint: Check spelling of keys in profiles.yml for connection issues [OK]

Common Mistakes:

• Changing command flags instead of fixing config
• Deleting environment instead of fixing it
• Running unrelated dbt commands

5. You want to test a new model in your dev environment without affecting staging or prod. Which approach is best to manage this safely?

hard

A. Create a separate dev target in profiles.yml and run with --target dev

B. Run the model directly in prod and rollback if needed

C. Delete the staging environment temporarily

D. Run all models in dev and prod simultaneously

Solution

1. Step 1: Understand environment isolation

   Using a separate dev target isolates testing from staging and production.

2. Step 2: Use --target dev to run safely

   Running with --target dev ensures changes only affect the dev environment.

3. Final Answer:

   Create a separate dev target in profiles.yml and run with --target dev -> Option A

4. Quick Check:

   Use dev target for safe isolated testing [OK]

Hint: Always test in dev environment using --target dev [OK]

Common Mistakes:

• Testing directly in production
• Removing staging environment
• Running multiple environments at once