For loops for dynamic SQL in dbt - Step-by-Step Execution

Concept Flow - For loops for dynamic SQL

Start loop over list

↓

Get current item

↓

Insert item into SQL template

↓

Add to SQL string

↓

More items?

Yes→Next item

No↓

Execute full SQL

The loop goes through each item, inserts it into the SQL template, builds the full SQL string, then runs it.

Execution Sample

dbt

{% set columns = ['id', 'name', 'age'] %}
SELECT
{% for col in columns %}  {{ col }}{% if not loop.last %}, {% endif %}
{% endfor %}
FROM users

This code dynamically builds a SQL SELECT statement listing columns from a list.

Execution Table

Step	Current item (col)	Action	SQL string built so far
1	'id'	Add 'id' to SQL	SELECT id
2	'name'	Add ', name' to SQL	SELECT id, name
3	'age'	Add ', age' to SQL	SELECT id, name, age
4	No more items	Close SQL with FROM users	SELECT id, name, age FROM users

💡 All columns processed, loop ends, full SQL ready

Variable Tracker

Variable	Start	After 1	After 2	After 3	Final
col	None	'id'	'name'	'age'	None
SQL string	"SELECT\n"	"SELECT\n id"	"SELECT\n id, name"	"SELECT\n id, name, age"	"SELECT\n id, name, age\nFROM users"

Key Moments - 2 Insights

Why do we check if the current item is the last in the loop?

To avoid adding a comma after the last column, so the SQL syntax stays correct (see execution_table rows 2 and 3).

What happens if the list of columns is empty?

The loop body won't run, so the SQL will have no columns listed, which can cause an error when executed.

Visual Quiz - 3 Questions

Look at the execution table, what is the SQL string after step 2?

A SELECT id, name, age

B SELECT id

C SELECT id, name

D SELECT id, name, age FROM users

💡 Hint

Check the 'SQL string built so far' column at step 2 in the execution_table.

At which step does the loop add the last column without a trailing comma?

A Step 3

B Step 1

C Step 2

D Step 4

💡 Hint

Look at the 'Action' column in execution_table rows 2 and 3 to see where the comma is omitted.

If we add a new column 'email' to the list, how many loop iterations will the execution table show?

A 3

B 4

C 5

D 6

💡 Hint

The loop runs once per column; adding 'email' increases iterations by one (see variable_tracker for count).

Concept Snapshot

For loops in dbt let you build SQL dynamically by repeating code for each item in a list.
Use {% for item in list %} ... {% endfor %} to loop.
Inside, insert variables with {{ item }}.
Check loop.last to avoid trailing commas.
Build SQL piece by piece, then run it.

Full Transcript

This visual execution shows how a for loop in dbt builds a dynamic SQL SELECT statement. We start with a list of columns. The loop picks each column one by one, adds it to the SQL string, and adds commas except after the last column. After all columns are added, the SQL string is completed with the FROM clause. Variables like 'col' and the SQL string change step by step. Key points include avoiding a comma after the last column and what happens if the list is empty. The quiz checks understanding of the SQL string at different steps and loop behavior when adding columns.

Practice

1. What is the main purpose of using {% for %} loops in dbt SQL models?

easy

A. To create permanent tables in the database

B. To stop SQL queries from running

C. To write Python code inside SQL models

D. To repeat SQL code patterns dynamically and reduce manual repetition

Solution

1. Step 1: Understand the role of {% for %} in dbt

   The {% for %} loop is used to repeat code blocks dynamically in dbt SQL models.

2. Step 2: Identify the benefit of repeating code

   Repeating code patterns saves time and reduces errors by avoiding manual copy-pasting.

3. Final Answer:

   To repeat SQL code patterns dynamically and reduce manual repetition -> Option D

4. Quick Check:

   For loops = dynamic repetition [OK]

Hint: For loops repeat code blocks automatically in dbt [OK]

Common Mistakes:

• Thinking for loops create tables
• Confusing dbt loops with Python loops
• Believing loops stop queries

2. Which of the following is the correct syntax to start a for loop in a dbt SQL model?

easy

A. {% loop item in list %}

B. {{ for item in list }}

C. {% for item in list %}

D. <% for item in list %>

Solution

1. Step 1: Recall dbt Jinja syntax for loops

   dbt uses Jinja templating where loops start with {% for variable in sequence %}.

2. Step 2: Compare options to correct syntax

   Only {% for item in list %} matches the correct Jinja for loop syntax.

3. Final Answer:

   {% for item in list %} -> Option C

4. Quick Check:

   dbt for loop syntax = {% for %} [OK]

Hint: dbt loops start with {% for variable in sequence %} [OK]

Common Mistakes:

• Using {{ }} instead of {% %} for loops
• Using wrong keywords like 'loop'
• Using angle brackets instead of curly braces

3. Given the following dbt code snippet, what will be the output SQL?

{% for col in ['id', 'name', 'age'] %}SELECT {{ col }} FROM users{% if not loop.last %} UNION ALL {% endif %}{% endfor %}

medium

A. SELECT id FROM users UNION ALL SELECT name FROM users UNION ALL SELECT age FROM users

B. SELECT id, name, age FROM users

C. SELECT id FROM users SELECT name FROM users SELECT age FROM users

D. SyntaxError due to missing commas

Solution

1. Step 1: Understand the loop iteration

   The loop runs over ['id', 'name', 'age'], producing three SELECT statements.

2. Step 2: Check the UNION ALL insertion

   Between each SELECT except the last, 'UNION ALL' is added to combine results.

3. Final Answer:

   SELECT id FROM users UNION ALL SELECT name FROM users UNION ALL SELECT age FROM users -> Option A

4. Quick Check:

   Loop + UNION ALL = combined selects [OK]

Hint: Look for loop.last to add UNION ALL except last [OK]

Common Mistakes:

• Ignoring the UNION ALL between selects
• Assuming all columns selected in one query
• Thinking commas separate selects

4. Identify the error in this dbt for loop snippet:

{% for col in columns %}SELECT {{ col }} FROM users{% endfor %}

Assuming columns = ['id', 'name'].

medium

A. Wrong loop syntax with {{ }} instead of {% %}

B. Missing UNION ALL or separator between SELECT statements

C. Incorrect variable name inside the loop

D. Loop variable should be capitalized

Solution

1. Step 1: Analyze the loop output

   The loop outputs two SELECT statements back to back without any separator.

2. Step 2: Identify missing SQL syntax

   Without UNION ALL or a separator, the SQL is invalid and will cause errors.

3. Final Answer:

   Missing UNION ALL or separator between SELECT statements -> Option B

4. Quick Check:

   Multiple SELECTs need UNION ALL [OK]

Hint: Add UNION ALL between selects in loops [OK]

Common Mistakes:

• Forgetting to add UNION ALL
• Using wrong loop syntax
• Misnaming variables inside loop

5. You want to create a dynamic SQL query in dbt that selects multiple columns from a table, but only if the column name is not empty or None. Which for loop code snippet correctly achieves this?

hard

A. {% for col in columns if col %}SELECT {{ col }} FROM users{% if not loop.last %} UNION ALL {% endif %}{% endfor %}

B. {% for col in columns %}SELECT {{ col }} FROM users{% if col != '' or col != None %} UNION ALL {% endif %}{% endfor %}

C. {% for col in columns %}{% if col %}SELECT {{ col }} FROM users UNION ALL {% endif %}{% endfor %}

D. {% for col in columns %}SELECT {{ col }} FROM users UNION ALL {% endfor %}

Solution

1. Step 1: Filter columns inside the for loop

   {% for col in columns if col %}SELECT {{ col }} FROM users{% if not loop.last %} UNION ALL {% endif %}{% endfor %} uses for col in columns if col to skip empty or None values correctly.

2. Step 2: Add UNION ALL only between valid selects

   {% for col in columns if col %}SELECT {{ col }} FROM users{% if not loop.last %} UNION ALL {% endif %}{% endfor %} adds UNION ALL only if not the last valid column, avoiding trailing UNION ALL.

3. Final Answer:

   {% for col in columns if col %}SELECT {{ col }} FROM users{% if not loop.last %} UNION ALL {% endif %}{% endfor %} -> Option A

4. Quick Check:

   Filter in loop + conditional UNION ALL = correct [OK]

Hint: Use 'if col' in loop to skip empty values [OK]

Common Mistakes:

• Using incorrect if condition syntax
• Adding UNION ALL after last select
• Not filtering out empty or None columns