Remote-exec provisioner in Terraform - Deep Dive

Overview - Remote-exec provisioner

What is it?

The remote-exec provisioner in Terraform lets you run commands on a remote machine after it is created. It connects to the machine using SSH or WinRM and executes scripts or commands. This helps automate setup tasks like installing software or configuring settings. It works as part of the resource creation process.

Why it matters

Without remote-exec, you would have to manually log into each new machine to set it up, which is slow and error-prone. Remote-exec automates this, saving time and ensuring consistency. It makes infrastructure creation smoother and faster, especially when managing many machines.

Where it fits

Before learning remote-exec, you should understand basic Terraform resources and how to create infrastructure. After this, you can learn about other provisioners like local-exec or configuration management tools like Ansible. Remote-exec fits into the automation step after resource creation.

Mental Model

Core Idea

Remote-exec provisioner runs commands on a new machine by connecting remotely right after it is created.

Think of it like...

It's like hiring a cleaning service that arrives right after you move into a new house to set everything up for you automatically.

Terraform Resource Creation
┌─────────────────────────────┐
│ Create VM or Server          │
└─────────────┬───────────────┘
              │
              ▼
┌─────────────────────────────┐
│ Remote-exec connects via SSH │
│ or WinRM                    │
└─────────────┬───────────────┘
              │
              ▼
┌─────────────────────────────┐
│ Runs setup commands/scripts  │
└─────────────────────────────┘

Build-Up - 7 Steps

1

Foundation

What is a provisioner in Terraform

Concept: Provisioners let Terraform run scripts or commands on resources after creation.

Terraform creates resources like servers or databases. Sometimes, you want to run extra setup steps on these resources, like installing software. Provisioners are blocks inside resource definitions that let you do this by running commands locally or remotely.

Result

You can automate setup tasks during resource creation, reducing manual work.

Understanding provisioners is key to automating resource configuration beyond just creating infrastructure.

2

Foundation

Basics of remote-exec provisioner

Concept: Remote-exec runs commands on a remote machine using SSH or WinRM after it is created.

Inside a resource, you add a remote-exec block. You specify connection details like IP, username, and authentication method. Then you list commands or scripts to run. Terraform connects to the machine and runs these commands automatically.

Result

The remote machine is configured automatically right after creation.

Knowing how remote-exec connects and runs commands helps automate initial machine setup.

3

Intermediate

Configuring connection details properly

🤔Before reading on: do you think Terraform automatically knows how to connect to your new machine, or do you need to specify connection info? Commit to your answer.

Concept: You must provide accurate connection info for remote-exec to work, like IP, user, and keys.

Terraform needs to know how to reach the machine. You specify a connection block with details such as host (IP address), user (like 'ubuntu'), and private key or password. Without this, Terraform cannot connect and will fail the provisioner step.

Result

Terraform successfully connects to the remote machine and runs commands.

Understanding connection details prevents common errors where Terraform cannot reach the machine.

4

Intermediate

Running inline commands vs scripts

🤔Before reading on: do you think remote-exec can only run single commands, or can it run multiple commands or scripts? Commit to your answer.

Concept: Remote-exec can run single commands inline or execute entire scripts stored locally or remotely.

You can specify commands directly in the 'inline' array to run multiple commands in order. Alternatively, you can use 'script' to upload and run a local script file on the remote machine. This flexibility helps with simple or complex setups.

Result

Commands or scripts run on the remote machine as part of provisioning.

Knowing both methods lets you choose the simplest or most maintainable way to automate setup.

5

Intermediate

Handling errors and retries in remote-exec

🤔Before reading on: do you think remote-exec retries commands automatically if they fail, or does it stop immediately? Commit to your answer.

Concept: Remote-exec stops on errors by default but can be configured with retries and timeouts.

If a command fails, Terraform marks the provisioner as failed and stops. You can add 'connection' settings like 'timeout' and 'retries' to try connecting multiple times, useful if the machine takes time to be ready. However, command errors inside scripts usually stop the process.

Result

Provisioning is more reliable with retries, but command errors still cause failure.

Understanding error handling helps design robust provisioning that copes with delays or temporary issues.

6

Advanced

Limitations and best practices for remote-exec

🤔Before reading on: do you think remote-exec is the best tool for all configuration tasks, or are there better alternatives for complex setups? Commit to your answer.

Concept: Remote-exec is useful but has limits; for complex or repeatable setups, configuration management tools are better.

Remote-exec runs commands once during creation and is not idempotent (safe to run multiple times). For complex setups, tools like Ansible or Chef are preferred. Also, remote-exec requires open network access and credentials, which can be security risks. Use it for simple, initial tasks only.

Result

You avoid fragile or insecure setups by knowing when to use remote-exec appropriately.

Knowing remote-exec's limits prevents overusing it and encourages better automation practices.

7

Expert

How remote-exec fits in Terraform's lifecycle

🤔Before reading on: do you think remote-exec runs every time Terraform applies, or only on resource creation? Commit to your answer.

Concept: Remote-exec runs only during resource creation or replacement, not on every apply.

Terraform runs provisioners like remote-exec only when creating or replacing resources. If you run 'terraform apply' again without changes, remote-exec does not run. This means remote-exec is not for ongoing configuration but initial setup. Understanding this avoids confusion about when commands run.

Result

You correctly predict when remote-exec commands execute during Terraform runs.

Understanding provisioner lifecycle avoids surprises and helps design stable infrastructure automation.

Under the Hood

When Terraform creates a resource with a remote-exec provisioner, it waits until the resource is ready. Then it uses the connection details to open an SSH or WinRM session to the machine. It sends the commands or scripts over this session and waits for them to complete. Terraform monitors the command exit status to decide success or failure. If successful, it continues; if not, it stops and reports an error.

Why designed this way?

Terraform was designed to manage infrastructure declaratively but needed a way to handle imperative setup steps. Remote-exec was added to fill this gap simply, using existing remote access protocols. Alternatives like configuration management tools exist but require separate systems. Remote-exec keeps setup close to resource creation for convenience, trading off some flexibility and idempotency.

Terraform Apply
┌─────────────────────────────┐
│ Create Resource (e.g., VM)  │
└─────────────┬───────────────┘
              │
              ▼
┌─────────────────────────────┐
│ Wait for resource readiness  │
└─────────────┬───────────────┘
              │
              ▼
┌─────────────────────────────┐
│ Open SSH/WinRM connection    │
│ using connection details     │
└─────────────┬───────────────┘
              │
              ▼
┌─────────────────────────────┐
│ Run commands/scripts remotely│
│ Monitor exit status          │
└─────────────┬───────────────┘
              │
       Success│Failure
              ▼
┌─────────────┴───────────────┐
│ Continue or stop provisioning│
└─────────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does remote-exec run every time you run 'terraform apply'? Commit to yes or no.

Common Belief: Remote-exec runs every time you apply Terraform changes.

Reality: Remote-exec runs only when the resource is created or replaced, not on every apply.

Why it matters: Expecting remote-exec to run every time can cause confusion and failed assumptions about configuration updates.

Quick: Do you think remote-exec automatically retries failed commands inside scripts? Commit to yes or no.

Common Belief: Remote-exec retries commands inside scripts automatically if they fail.

Reality: Remote-exec stops immediately on command failure; retries apply only to connection attempts.

Why it matters: Assuming automatic retries can lead to unnoticed failures and unstable setups.

Quick: Can remote-exec be used as a full replacement for configuration management tools? Commit to yes or no.

Common Belief: Remote-exec can replace tools like Ansible or Chef for all configuration needs.

Reality: Remote-exec is best for simple, one-time setup; complex or repeatable configuration needs dedicated tools.

Why it matters: Using remote-exec for complex tasks leads to fragile, hard-to-maintain infrastructure.

Quick: Does Terraform automatically know how to connect to new machines without user input? Commit to yes or no.

Common Belief: Terraform automatically discovers connection details for remote-exec.

Reality: You must explicitly provide connection info; Terraform cannot guess IPs or credentials.

Why it matters: Missing connection details cause provisioning failures and wasted debugging time.

Expert Zone

1

Remote-exec provisioners run only during resource creation or replacement, so they do not handle ongoing configuration drift.

2

Connection blocks can inherit from resource attributes dynamically, allowing flexible and reusable provisioning code.

3

Using remote-exec with sensitive data requires careful handling to avoid exposing secrets in logs or state files.

When NOT to use

Avoid remote-exec for complex, repeatable, or idempotent configuration tasks. Instead, use dedicated configuration management tools like Ansible, Chef, or Puppet. Also, avoid remote-exec when network access or credentials are restricted or when security policies forbid remote command execution.

Production Patterns

In production, remote-exec is often used for simple bootstrapping tasks like installing an agent or setting environment variables. More complex setups delegate to configuration management or container orchestration. Teams use remote-exec sparingly to reduce fragility and improve security posture.

Connections

Configuration Management

Builds-on

Understanding remote-exec helps grasp why configuration management tools exist to handle complex, repeatable setups beyond initial provisioning.

SSH Protocol

Underlying technology

Knowing SSH basics clarifies how remote-exec securely connects and runs commands on remote machines.

Continuous Integration/Continuous Deployment (CI/CD)

Complementary automation

Remote-exec fits into CI/CD pipelines by automating environment setup, linking infrastructure provisioning with application deployment.

Common Pitfalls

#1 Forgetting to specify connection details causes remote-exec to fail.

Wrong approach:

resource "aws_instance" "example" {
ami = "ami-123456"
instance_type = "t2.micro"

provisioner "remote-exec" {
inline = ["echo Hello"]
}
}

Correct approach:

resource "aws_instance" "example" {
ami = "ami-123456"
instance_type = "t2.micro"

provisioner "remote-exec" {
connection {
type = "ssh"
user = "ubuntu"
private_key = file("~/.ssh/id_rsa")
host = self.public_ip
}
inline = ["echo Hello"]
}
}

Root cause:

Assuming Terraform knows how to connect without explicit connection info.

#2 Expecting remote-exec to run on every terraform apply.

Wrong approach:

Relying on remote-exec to update configuration on existing resources without recreating them.

Correct approach:

Use configuration management tools or terraform lifecycle hooks for ongoing configuration changes.

Root cause:

Misunderstanding provisioner lifecycle and when remote-exec runs.

#3 Running complex, multi-step configuration inside remote-exec scripts without error handling.

Wrong approach:

provisioner "remote-exec" {
inline = [
"apt-get update",
"apt-get install -y nginx",
"systemctl start nginx"
]
}

Correct approach:

Use a script file with proper error checks and idempotency, then call it via remote-exec script attribute.

Root cause:

Treating remote-exec commands as simple scripts without robustness.

Key Takeaways

Remote-exec provisioner automates running commands on new machines right after creation using SSH or WinRM.

You must provide accurate connection details for remote-exec to connect and work properly.

Remote-exec runs only during resource creation or replacement, not on every Terraform apply.

It is best suited for simple, one-time setup tasks, not complex or repeatable configuration.

Understanding remote-exec's lifecycle and limits helps avoid common errors and encourages better automation choices.

Practice

1. What is the main purpose of the remote-exec provisioner in Terraform?

easy

A. To run commands on a remote machine after it is created

B. To create a new virtual machine in the cloud

C. To store Terraform state files remotely

D. To manage user permissions on the local machine

Solution

1. Step 1: Understand the role of provisioners

   Provisioners in Terraform run scripts or commands on resources after creation.

2. Step 2: Identify what remote-exec does

   The remote-exec provisioner connects to the remote machine and runs commands there.

3. Final Answer:

   To run commands on a remote machine after it is created -> Option A

4. Quick Check:

   remote-exec runs remote commands = D [OK]

Hint: remote-exec runs commands remotely after resource creation [OK]

Common Mistakes:

• Confusing remote-exec with resource creation
• Thinking remote-exec manages local files
• Mixing remote-exec with state management

2. Which of the following is the correct syntax to define a remote-exec provisioner inside a Terraform resource?

easy

A. remote_exec { commands = ["echo Hello"] }

B. provisioner remote-exec { script = "hello.sh" }

C. provisioner "remote-exec" { inline = ["echo Hello"] }

D. exec_remote { inline = ["echo Hello"] }

Solution

1. Step 1: Recall Terraform provisioner syntax

   Provisioners are defined with the keyword provisioner followed by the provisioner type in quotes.

2. Step 2: Check the correct block structure for remote-exec

   The remote-exec provisioner uses inline to run commands directly.

3. Final Answer:

   provisioner "remote-exec" { inline = ["echo Hello"] } -> Option C

4. Quick Check:

   Correct provisioner block syntax = B [OK]

Hint: Use provisioner "remote-exec" with inline commands [OK]

Common Mistakes:

• Using underscores instead of hyphens in provisioner name
• Omitting quotes around provisioner type
• Using wrong block names like exec_remote

3. Given this Terraform snippet inside an AWS EC2 resource:

provisioner "remote-exec" {
  inline = ["sudo apt-get update", "sudo apt-get install -y nginx"]
  connection {
    type        = "ssh"
    user        = "ubuntu"
    private_key = file("~/.ssh/id_rsa")
    host        = self.public_ip
  }
}

What will happen when this runs successfully?

medium

A. The commands will run locally on the Terraform machine

B. The EC2 instance will update packages and install nginx via SSH

C. Terraform will fail because private_key cannot be a file

D. The EC2 instance will reboot automatically after installation

Solution

1. Step 1: Understand remote-exec with connection block

   The provisioner connects via SSH using the given user, key, and host to run commands remotely.

2. Step 2: Analyze the inline commands

   The commands update package lists and install nginx on the remote EC2 instance.

3. Final Answer:

   The EC2 instance will update packages and install nginx via SSH -> Option B

4. Quick Check:

   remote-exec runs commands remotely = C [OK]

Hint: remote-exec runs inline commands on remote host via SSH [OK]

Common Mistakes:

• Thinking commands run locally
• Assuming private_key cannot be a file
• Expecting automatic reboot after install

4. You have this Terraform snippet:

provisioner "remote-exec" {
  inline = ["sudo apt-get update", "sudo apt-get install -y nginx"]
  connection {
    type     = "ssh"
    user     = "ubuntu"
    password = "mypassword"
    host     = self.public_ip
  }
}

After running Terraform apply, you get a connection timeout error. What is the most likely cause?

medium

A. The EC2 instance's security group does not allow SSH (port 22) inbound

B. The password authentication is not supported by SSH

C. The inline commands have syntax errors

D. The host attribute should be a private IP, not public IP

Solution

1. Step 1: Understand connection timeout causes

   Timeout usually means network connection to SSH port 22 is blocked or unreachable.

2. Step 2: Check common network issues

   If the EC2 security group lacks inbound SSH rules, connection fails despite correct credentials.

3. Final Answer:

   The EC2 instance's security group does not allow SSH (port 22) inbound -> Option A

4. Quick Check:

   Timeout = network block on SSH port 22 [OK]

Hint: Timeout usually means SSH port 22 blocked in firewall/security group [OK]

Common Mistakes:

• Blaming inline command syntax for connection errors
• Assuming password auth is always unsupported
• Confusing public and private IP for SSH host

5. You want to use remote-exec to run a script on multiple newly created instances in a loop. Which approach is best to ensure commands run on each instance correctly?

hard

A. Use local-exec provisioner to SSH manually to each instance

B. Define one resource and run remote-exec with a list of all instance IPs in one provisioner

C. Run remote-exec provisioner only once after all instances are created using a null_resource

D. Use a for_each on the resource and include a remote-exec provisioner inside each resource block

Solution

1. Step 1: Understand resource loops with provisioners

   Using for_each creates multiple resource instances, each with its own provisioner execution.

2. Step 2: Why per-resource provisioner is best

   Provisioners inside each resource run commands on that specific instance, ensuring correct targeting.

3. Final Answer:

   Use a for_each on the resource and include a remote-exec provisioner inside each resource block -> Option D

4. Quick Check:

   Provisioner per resource instance = A [OK]

Hint: Use for_each with provisioner inside resource for multiple instances [OK]

Common Mistakes:

• Trying to run remote-exec on all IPs in one provisioner
• Using null_resource without proper triggers
• Using local-exec instead of remote-exec for remote commands