Provisioner failure behavior in Terraform - Deep Dive

Overview - Provisioner failure behavior

What is it?

Provisioner failure behavior in Terraform describes what happens when a provisioner, a tool that runs scripts or commands on a resource after it is created, does not complete successfully. Provisioners help configure resources beyond what Terraform can do natively. When a provisioner fails, Terraform must decide whether to stop, retry, or continue the deployment.

Why it matters

Without clear failure behavior, infrastructure deployments could become unpredictable or inconsistent. If a provisioner fails silently, resources might be left misconfigured, causing outages or security risks. Understanding failure behavior helps ensure reliable, repeatable infrastructure setups and faster troubleshooting.

Where it fits

Learners should first understand basic Terraform concepts like resources, state, and the apply lifecycle. After mastering provisioners and their failure behavior, learners can explore advanced Terraform features like modules, remote-exec, and lifecycle hooks.

Mental Model

Core Idea

Terraform provisioner failure behavior controls whether infrastructure deployment stops, retries, or continues when post-creation configuration commands fail.

Think of it like...

It's like baking a cake and then adding frosting: if the frosting spills or melts, you decide whether to stop baking, try again, or serve the cake as is.

┌───────────────┐
│ Terraform Run │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Create Resource│
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Run Provisioner│
└──────┬────────┘
       │
  ┌────┴─────┐
  │ Success  │
  └────┬─────┘
       │
       ▼
  Deployment Complete

  Or

  ┌────┴─────┐
  │ Failure  │
  └────┬─────┘
       │
  ┌────┴─────────────┐
  │ Stop / Retry / Continue │
  └─────────────────────┘

Build-Up - 7 Steps

1

Foundation

What is a Terraform Provisioner

Concept: Introduction to provisioners as tools to run scripts or commands on resources after creation.

Terraform creates infrastructure resources like servers or databases. Sometimes, you need to run extra commands on these resources, like installing software or configuring settings. Provisioners let you do this by running scripts or commands after the resource is ready.

Result

You can customize resources beyond basic creation using provisioners.

Understanding provisioners is key to extending Terraform's capabilities to real-world setups.

2

Foundation

How Provisioners Run in Terraform

Concept: Provisioners run after resource creation during the apply phase.

When you run 'terraform apply', Terraform first creates the resource. After creation, it runs the provisioner commands on that resource. If the provisioner succeeds, Terraform moves on. If it fails, Terraform must decide what to do next.

Result

Provisioners execute only after resources exist, affecting deployment flow.

Knowing when provisioners run helps understand their impact on deployment success.

3

Intermediate

Default Failure Behavior of Provisioners

🤔Before reading on: do you think Terraform stops the entire deployment if a provisioner fails, or does it continue? Commit to your answer.

Concept: By default, Terraform stops the deployment if a provisioner fails.

If a provisioner command fails (returns an error), Terraform stops the apply process and reports an error. This prevents incomplete or inconsistent infrastructure states. You must fix the problem and rerun apply.

Result

Terraform halts deployment on provisioner failure, requiring manual intervention.

Stopping on failure ensures infrastructure is not left in a half-configured state.

4

Intermediate

Using 'ignore_errors' to Continue on Failure

🤔Before reading on: do you think ignoring provisioner errors is safe for production? Commit to your answer.

Concept: Terraform allows ignoring provisioner errors to continue deployment despite failures.

You can set 'ignore_errors = true' in a provisioner block. This tells Terraform to continue applying even if the provisioner fails. This is useful for non-critical steps but can risk inconsistent setups.

Result

Deployment continues despite provisioner errors, possibly leaving resources partially configured.

Ignoring errors trades safety for deployment speed and flexibility.

5

Intermediate

Provisioner Retry Mechanism

🤔Before reading on: do you think Terraform retries provisioners automatically on failure? Commit to your answer.

Concept: Terraform supports retrying provisioners with configurable attempts and delays.

You can configure provisioners with 'retry_count' and 'retry_interval' to retry failed commands. Terraform will try again up to the count with pauses between attempts. This helps handle temporary issues like network glitches.

Result

Provisioners can automatically retry, improving robustness against transient failures.

Retries reduce manual fixes for temporary errors, improving automation reliability.

6

Advanced

Impact of Provisioner Failures on Terraform State

🤔Before reading on: does a provisioner failure roll back resource creation or leave it? Commit to your answer.

Concept: Provisioner failures do not roll back resource creation; the resource remains in state.

If a provisioner fails, Terraform stops but does not delete the created resource. The resource stays in the state file as created. You must fix the provisioner or manually handle the resource before reapplying.

Result

Resources remain created despite provisioner failures, requiring manual cleanup or fixes.

Knowing this prevents confusion about resource existence after failed applies.

7

Expert

Best Practices for Handling Provisioner Failures

🤔Before reading on: do you think provisioners are recommended for all resource configuration? Commit to your answer.

Concept: Experts recommend minimizing provisioner use and handling failures with external tools or configuration management.

Provisioners are a last resort when native Terraform support is missing. Use configuration management tools (like Ansible) or cloud-init for setup. If using provisioners, handle failures with retries, error ignoring carefully, and clear logging. Avoid complex logic in provisioners.

Result

More reliable, maintainable infrastructure with fewer failure surprises.

Understanding provisioner limits leads to better infrastructure design and fewer deployment issues.

Under the Hood

Terraform applies resources in order. After creating a resource, it runs provisioner commands via SSH or local execution. If the command returns a non-zero exit code, Terraform treats it as failure. Terraform then decides to stop or continue based on provisioner settings. The state file records resource creation but not provisioner success, so failures do not roll back resource creation automatically.

Why designed this way?

Terraform separates resource creation from configuration to keep infrastructure declarative and predictable. Provisioners are designed as a fallback for tasks Terraform cannot do natively. Stopping on failure prevents inconsistent states, while allowing retries and error ignoring provides flexibility. Rolling back resource creation on provisioner failure would complicate state management and risk data loss.

┌───────────────┐
│ Terraform CLI │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Create Resource│
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Run Provisioner│
└──────┬────────┘
       │
  ┌────┴─────┐
  │ Exit Code│
  └────┬─────┘
       │
  ┌────┴─────────────┐
  │ 0 (Success)      │
  │ Non-0 (Failure)  │
  └────┬─────────────┘
       │
┌──────┴─────────────┐
│ Stop / Retry / Ignore│
└─────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does a provisioner failure delete the created resource automatically? Commit yes or no.

Common Belief: If a provisioner fails, Terraform deletes the resource to keep things clean.

Reality: Terraform does NOT delete the resource on provisioner failure; the resource remains created and recorded in state.

Why it matters: Assuming deletion leads to orphaned resources and confusion during cleanup or reapply.

Quick: Can you safely ignore all provisioner errors in production? Commit yes or no.

Common Belief: Setting 'ignore_errors' means provisioner failures are harmless and can be ignored safely.

Reality: Ignoring errors can leave resources partially configured, causing hidden bugs or security issues.

Why it matters: Ignoring errors without understanding risks can cause unstable or insecure infrastructure.

Quick: Does Terraform retry provisioners automatically without configuration? Commit yes or no.

Common Belief: Terraform automatically retries provisioners on failure to fix temporary issues.

Reality: Terraform retries only if explicitly configured with retry_count and retry_interval.

Why it matters: Assuming automatic retries can cause unexpected failures and manual intervention.

Quick: Are provisioners the recommended way to configure all resources? Commit yes or no.

Common Belief: Provisioners are the best and only way to configure resources after creation.

Reality: Provisioners are a last resort; native Terraform features or external tools are preferred.

Why it matters: Overusing provisioners leads to fragile, hard-to-maintain infrastructure.

Expert Zone

1

Provisioner failures do not affect Terraform state consistency but can cause drift between declared and actual resource state.

2

Retries can mask underlying issues if overused, leading to longer deployment times without fixing root causes.

3

Provisioners run on resource creation and destroy phases differently; failure behavior varies accordingly.

When NOT to use

Avoid provisioners when native Terraform resource arguments or dedicated configuration management tools (like Ansible, Chef, or cloud-init) can achieve the same result more reliably and declaratively.

Production Patterns

In production, teams use provisioners sparingly, often only for bootstrapping. They combine Terraform with configuration management tools for complex setups and use retry and error handling settings carefully to balance reliability and speed.

Connections

Idempotency in Configuration Management

Provisioner failure handling relates to ensuring commands can run multiple times without causing errors.

Understanding idempotency helps design provisioner scripts that can safely retry or ignore errors without breaking infrastructure.

Transaction Rollback in Databases

Provisioner failure behavior contrasts with database transactions that rollback on failure.

Knowing this difference clarifies why Terraform does not rollback resource creation on provisioner failure, avoiding data loss risks.

Error Handling in Software Engineering

Provisioner failure behavior is a form of error handling strategy in automation workflows.

Grasping error handling patterns helps in choosing when to stop, retry, or ignore failures in infrastructure automation.

Common Pitfalls

#1 Assuming provisioner failures delete resources automatically.

Wrong approach:

terraform apply
# Provisioner fails but user expects resource gone
# User runs terraform apply again without cleanup

Correct approach:

terraform destroy
# Manually remove resource before reapplying
terraform apply

Root cause:

Misunderstanding that provisioner failure does not rollback resource creation.

#2 Setting ignore_errors = true on critical provisioners.

Wrong approach:

provisioner "remote-exec" {
inline = ["critical-setup.sh"]
ignore_errors = true
}

Correct approach:

provisioner "remote-exec" {
inline = ["critical-setup.sh"]
# Do not ignore errors for critical steps
}

Root cause:

Believing ignoring errors is safe for all provisioners.

#3 Not configuring retries for flaky network commands.

Wrong approach:

provisioner "remote-exec" {
inline = ["network-call.sh"]
# No retry settings
}

Correct approach:

provisioner "remote-exec" {
inline = ["network-call.sh"]
retry_count = 3
retry_interval = 10
}

Root cause:

Assuming provisioners retry automatically without configuration.

Key Takeaways

Provisioners run commands after resource creation to configure infrastructure beyond Terraform's native capabilities.

By default, Terraform stops deployment if a provisioner fails, preventing inconsistent infrastructure states.

Provisioner failures do not delete created resources; manual cleanup is needed to avoid orphaned resources.

You can configure retries and ignore errors, but these should be used carefully to balance reliability and deployment speed.

Experts recommend minimizing provisioner use and preferring native Terraform features or external configuration tools for robust infrastructure.