Why provisioners are a last resort in Terraform - Why It Works This Way

Overview - Why provisioners are a last resort

What is it?

Provisioners in Terraform are special instructions that run scripts or commands on a resource after it is created or before it is destroyed. They help perform setup tasks that Terraform cannot do by itself, like installing software or configuring settings inside a virtual machine. However, they are considered a last resort because they can make your infrastructure less predictable and harder to manage. Using provisioners means Terraform depends on external scripts and the state of the resource, which can cause problems.

Why it matters

Without understanding why provisioners should be avoided, you might rely on them too much, causing your infrastructure to break unexpectedly or become difficult to update. This can lead to downtime, wasted time fixing errors, and frustration. Knowing when and why to avoid provisioners helps you build stable, reliable infrastructure that is easier to maintain and scale.

Where it fits

Before learning about provisioners, you should understand basic Terraform concepts like resources, providers, and the Terraform lifecycle. After this, you can learn about better alternatives such as configuration management tools, cloud-init, or user data scripts that handle setup more cleanly and reliably.

Mental Model

Core Idea

Provisioners are like last-minute helpers that fix things after Terraform builds your infrastructure, but relying on them too much makes your setup fragile and unpredictable.

Think of it like...

Imagine building a house where the builder finishes the walls and roof, but then you hire a handyman to fix the plumbing and electricity after the house is done. If the handyman makes mistakes or is unavailable, the house won't work properly. It's better to have specialists do their jobs during the building process rather than fixing problems afterward.

Terraform Lifecycle
┌───────────────┐
│  Plan Phase   │
└──────┬────────┘
       │
┌──────▼────────┐
│ Apply Phase   │
│ ┌──────────┐  │
│ │ Resource │  │
│ │ Creation │  │
│ └────┬─────┘  │
│      │        │
│ ┌────▼─────┐  │
│ │Provision-│  │
│ │  ers     │  │
│ └──────────┘  │
└───────────────┘

Provisioners run after resource creation, outside Terraform's main control.

Build-Up - 6 Steps

1

Foundation

What are Terraform Provisioners

Concept: Provisioners run scripts or commands on resources during creation or destruction.

Terraform creates resources like virtual machines or databases. Sometimes, you want to run extra commands on these resources, like installing software or configuring settings. Provisioners let you do this by running scripts after the resource is ready or before it is removed.

Result

You can automate setup tasks that Terraform alone cannot do.

Understanding provisioners shows how Terraform can extend beyond just creating resources to configuring them.

2

Foundation

How Provisioners Fit in Terraform Lifecycle

Concept: Provisioners run after resource creation or before destruction, outside Terraform's normal state management.

Terraform first plans what to create or change. Then it applies those changes by creating or updating resources. After a resource is created, provisioners run their scripts on that resource. This means provisioners depend on the resource being reachable and ready.

Result

Provisioners add extra steps after resource creation, which can fail if the resource is not ready or accessible.

Knowing when provisioners run helps understand why they can cause unpredictable failures.

3

Intermediate

Why Provisioners Are Risky

🤔Before reading on: do you think provisioners always succeed if the resource is created? Commit to your answer.

Concept: Provisioners can fail due to network issues, timing, or script errors, causing Terraform apply to fail or resources to be in a bad state.

Provisioners rely on the resource being accessible (like SSH for a VM). If the network is slow or the resource is not ready, the provisioner script can fail. This failure stops Terraform from completing, leaving your infrastructure half-done. Also, if the script has bugs, it can cause unexpected problems.

Result

Provisioners introduce instability and can cause Terraform runs to fail unexpectedly.

Understanding the fragility of provisioners explains why they reduce infrastructure reliability.

4

Intermediate

Better Alternatives to Provisioners

🤔Before reading on: do you think provisioners are the only way to configure resources after creation? Commit to your answer.

Concept: Using cloud-init, user data, or configuration management tools is more reliable than provisioners for setup tasks.

Many cloud providers let you pass scripts or configuration data when creating resources (like user data for VMs). These run automatically inside the resource during startup, without Terraform needing to connect later. Tools like Ansible or Chef manage configuration separately and can be run independently. These methods are more stable and easier to maintain.

Result

Infrastructure setup becomes more predictable and less dependent on Terraform's apply process.

Knowing alternatives helps avoid common pitfalls and build more robust infrastructure.

5

Advanced

When Provisioners Are Acceptable

🤔Before reading on: do you think provisioners should be used for all resource configuration? Commit to your answer.

Concept: Provisioners are acceptable only for rare cases where no other option exists and must be used carefully with error handling.

Sometimes, no cloud-init or configuration management option exists for a resource. In these cases, provisioners can run small, simple scripts. You should use 'when = create' or 'when = destroy' carefully and add retries or timeouts. Always keep provisioner scripts idempotent (safe to run multiple times) to avoid problems.

Result

Provisioners can be a last-resort tool but require caution to avoid breaking infrastructure.

Understanding the limited, careful use of provisioners prevents overuse and instability.

6

Expert

How Provisioners Affect Terraform State and Idempotency

🤔Before reading on: do you think Terraform tracks provisioner success in its state file? Commit to your answer.

Concept: Terraform does not track provisioner script results in state, so repeated applies may rerun provisioners, causing side effects.

Terraform state tracks resource existence and attributes but not provisioner execution results. If you run 'terraform apply' again, provisioners may rerun if the resource changes or if you force it. This can cause scripts to run multiple times, leading to unexpected changes or failures. This breaks the principle of idempotency, where applying the same configuration repeatedly should not cause changes.

Result

Provisioners can cause unpredictable behavior and make infrastructure hard to maintain.

Knowing how provisioners interact with state explains why they break Terraform's declarative model and should be avoided.

Under the Hood

Provisioners execute external scripts or commands by connecting to the resource after Terraform creates it. For example, SSH provisioners connect to a VM using SSH to run shell commands. Terraform waits for the provisioner to finish before continuing. However, Terraform only knows if the provisioner succeeded or failed; it does not track what the script did inside the resource. This means Terraform cannot detect partial failures or changes made by the provisioner later.

Why designed this way?

Provisioners were introduced to fill gaps where Terraform's declarative model could not express imperative setup steps. They provide a way to run arbitrary commands on resources. However, this design trades off predictability and state management for flexibility. Alternatives like cloud-init or configuration management tools evolved to handle setup more cleanly, reducing the need for provisioners.

Terraform Apply Process
┌───────────────┐
│ Terraform     │
│ creates       │
│ resource      │
└──────┬────────┘
       │
┌──────▼────────┐
│ Resource is   │
│ ready &       │
│ accessible    │
└──────┬────────┘
       │
┌──────▼────────┐
│ Provisioner   │
│ connects via  │
│ SSH/WinRM     │
│ runs script   │
└──────┬────────┘
       │
┌──────▼────────┐
│ Terraform     │
│ checks exit   │
│ status        │
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: do you think provisioners guarantee your resource is fully configured after apply? Commit yes or no.

Common Belief: Provisioners always ensure the resource is correctly configured after Terraform apply.

Reality: Provisioners can fail silently or partially, leaving resources misconfigured without Terraform detecting it.

Why it matters: Believing this causes overconfidence and hidden errors that break applications or cause downtime.

Quick: do you think Terraform tracks provisioner script results in its state file? Commit yes or no.

Common Belief: Terraform records provisioner success or failure in its state and will not rerun them unnecessarily.

Reality: Terraform does not track provisioner script results in state, so provisioners may rerun on every apply if conditions change.

Why it matters: This can cause scripts to run multiple times, leading to unpredictable side effects and maintenance headaches.

Quick: do you think provisioners are the best way to configure resources in Terraform? Commit yes or no.

Common Belief: Provisioners are the recommended way to configure resources after creation in Terraform.

Reality: Provisioners are discouraged except as a last resort; better methods exist like cloud-init or configuration management tools.

Why it matters: Using provisioners as a first choice leads to fragile infrastructure and harder-to-debug failures.

Quick: do you think provisioners run before resource creation? Commit yes or no.

Common Belief: Provisioners run before the resource is created to prepare it.

Reality: Provisioners run only after resource creation or before destruction, never before creation.

Why it matters: Misunderstanding timing causes incorrect assumptions about when setup happens, leading to errors.

Expert Zone

1

Provisioners can cause race conditions if multiple resources depend on each other's provisioners running in a specific order, which Terraform does not guarantee.

2

Using 'local-exec' provisioners runs commands on the machine running Terraform, not on the resource, which can confuse beginners about where scripts execute.

3

Provisioners do not support rollback; if a provisioner fails, Terraform cannot undo partial changes made by the script, leading to inconsistent states.

When NOT to use

Avoid provisioners when you can use cloud-init, user data, or configuration management tools like Ansible, Chef, or Puppet. Use provisioners only when no other method exists to configure a resource, and keep scripts simple and idempotent.

Production Patterns

In production, teams use provisioners sparingly, often only for bootstrapping or one-time fixes. Most configuration is handled outside Terraform with dedicated tools. Provisioners are wrapped with retries and error handling to reduce failures. Infrastructure as code pipelines separate resource creation from configuration for better reliability.

Connections

Immutable Infrastructure

Provisioners conflict with immutable infrastructure principles by modifying resources after creation.

Understanding provisioners helps grasp why immutable infrastructure prefers replacing resources over changing them in place.

Configuration Management

Provisioners overlap with configuration management but are less reliable and flexible.

Knowing provisioners clarifies why dedicated configuration management tools are preferred for managing resource state.

Software Transactional Memory (STM)

Provisioners lack transactional guarantees similar to STM, leading to partial failures.

Comparing provisioners to STM highlights the importance of atomic, rollback-capable operations in reliable systems.

Common Pitfalls

#1 Using provisioners for all resource configuration without alternatives.

Wrong approach:

resource "aws_instance" "example" {
ami = "ami-123456"
instance_type = "t2.micro"

provisioner "remote-exec" {
inline = ["sudo apt-get update", "sudo apt-get install -y nginx"]
}
}

Correct approach:

resource "aws_instance" "example" {
ami = "ami-123456"
instance_type = "t2.micro"

user_data = <<-EOF
#!/bin/bash
apt-get update
apt-get install -y nginx
EOF
}

Root cause:

Misunderstanding that cloud-init user data can handle setup more reliably than provisioners.

#2 Not handling provisioner failures, causing Terraform apply to stop unexpectedly.

Wrong approach:

provisioner "remote-exec" {
inline = ["some-failing-command"]
}

Correct approach:

provisioner "remote-exec" {
inline = ["some-command || true"]
on_failure = "continue"
}

Root cause:

Ignoring that provisioner failures stop Terraform apply and not adding error handling.

#3 Using local-exec provisioner expecting commands to run on the resource.

Wrong approach:

provisioner "local-exec" {
command = "sudo apt-get install -y nginx"
}

Correct approach:

provisioner "remote-exec" {
inline = ["sudo apt-get install -y nginx"]
}

Root cause:

Confusing local-exec (runs on Terraform machine) with remote-exec (runs on resource).

Key Takeaways

Provisioners run scripts on resources after creation but are fragile and can cause unpredictable failures.

Terraform does not track provisioner script results in state, so provisioners may rerun and cause side effects.

Better alternatives like cloud-init, user data, or configuration management tools should be preferred over provisioners.

Provisioners should be used only as a last resort with careful error handling and idempotent scripts.

Understanding provisioners helps build more reliable, maintainable infrastructure by avoiding common pitfalls.