Cloud agent provisioning (EC2, Azure) in Jenkins - Commands & Configuration

Introduction

When you want Jenkins to run jobs on cloud servers, you need to create agents in the cloud automatically. This process is called cloud agent provisioning. It saves time by creating and removing servers only when needed.

When you want Jenkins to run builds on Amazon EC2 instances without manual setup.

When you need Jenkins agents to scale up and down automatically in Azure based on workload.

When you want to avoid keeping permanent build servers and save costs by using cloud resources on demand.

When you want to run different build environments for different projects using cloud agents.

When you want Jenkins to manage cloud servers lifecycle for running jobs.

Config File - Jenkinsfile

Jenkinsfile

pipeline {
  agent {
    label 'ec2-agent'
  }
  stages {
    stage('Build') {
      steps {
        echo 'Running build on EC2 agent'
      }
    }
  }
}

This Jenkinsfile defines a pipeline that runs on a cloud agent labeled 'ec2-agent' from the AWS EC2 cloud configuration. The agent block tells Jenkins to use an agent with the label 'ec2-agent', which should be configured to provision an EC2 instance automatically. The stages section runs the build steps on that cloud agent.

Commands

This command installs the Amazon EC2 plugin in Jenkins, which allows Jenkins to create and manage EC2 agents automatically.

Terminal

jenkins-cli install-plugin amazon-ec2

Expected Output

Installing plugin: amazon-ec2

Success

Restarts Jenkins safely to apply the new plugin without interrupting running jobs.

Terminal

jenkins-cli safe-restart

Expected Output

Jenkins is restarting...

Adds AWS credentials to Jenkins from an XML file so Jenkins can authenticate to AWS and provision EC2 instances.

Terminal

jenkins-cli create-credentials-by-xml system::system::jenkins _ < aws-credentials.xml

Expected Output

Credentials added successfully

Runs a Groovy script to configure the EC2 cloud in Jenkins with details like region, AMI, instance type, and credentials.

Terminal

jenkins-cli groovy = < configure-ec2-cloud.groovy

Expected Output

EC2 cloud configured successfully

Starts a Jenkins pipeline job that uses the EC2 cloud agent to run the build on a provisioned EC2 instance.

Terminal

jenkins-cli build example-pipeline

Expected Output

[Pipeline] Start of Pipeline

[Pipeline] node

Running build on EC2 agent

[Pipeline] End of Pipeline

Finished: SUCCESS

Key Concept

If you remember nothing else from this pattern, remember: Jenkins can create and destroy cloud servers automatically to run jobs, saving time and resources.

Common Mistakes

Not adding AWS credentials to Jenkins before configuring EC2 cloud.

Jenkins cannot authenticate to AWS and will fail to create EC2 agents.

Add valid AWS credentials in Jenkins credentials store before configuring the EC2 cloud.

Using incorrect AMI ID or instance type in EC2 cloud configuration.

Jenkins will fail to launch EC2 instances or launch wrong environments.

Use valid AMI IDs and instance types that exist in your AWS region.

Not restarting Jenkins after installing the EC2 plugin.

The plugin will not be active and Jenkins cannot provision EC2 agents.

Always restart Jenkins safely after installing new plugins.

Summary

Install the Amazon EC2 plugin in Jenkins to enable cloud agent provisioning.

Add AWS credentials to Jenkins so it can authenticate to AWS.

Configure the EC2 cloud in Jenkins with region, AMI, and instance details.

Use a Jenkins pipeline that specifies the cloud agent label to run jobs on EC2 instances.

Jenkins will create and remove EC2 agents automatically to run your builds.

Practice

1. What is the main purpose of cloud agent provisioning in Jenkins?

easy

A. To dynamically create and use cloud machines as Jenkins agents

B. To manually configure Jenkins agents on local machines

C. To backup Jenkins configuration files to the cloud

D. To monitor Jenkins job execution times

Solution

1. Step 1: Understand Jenkins agent provisioning

   Jenkins can create agents dynamically in the cloud to run jobs without manual setup.

2. Step 2: Identify the purpose of cloud provisioning

   Cloud agent provisioning automates creating cloud machines (like EC2 or Azure VMs) as Jenkins agents.

3. Final Answer:

   To dynamically create and use cloud machines as Jenkins agents -> Option A

4. Quick Check:

   Cloud agent provisioning = dynamic cloud agents [OK]

Hint: Cloud provisioning means automatic cloud agent creation [OK]

Common Mistakes:

• Confusing provisioning with manual setup
• Thinking it is for backup or monitoring
• Assuming it only works on local machines

2. Which of the following is the correct way to specify an EC2 AMI ID in Jenkins cloud agent configuration?

easy

A. ami-0abcdef1234567890

B. ec2-ami-123456

C. AMI:1234567890

D. image-ec2-abcdef

Solution

1. Step 1: Recognize EC2 AMI ID format

   EC2 AMI IDs start with 'ami-' followed by alphanumeric characters, e.g., 'ami-0abcdef1234567890'.

2. Step 2: Match the correct format

   ami-0abcdef1234567890 matches the standard EC2 AMI ID format exactly.

3. Final Answer:

   ami-0abcdef1234567890 -> Option A

4. Quick Check:

   EC2 AMI ID starts with 'ami-' [OK]

Hint: EC2 AMI IDs always start with 'ami-' [OK]

Common Mistakes:

• Using incorrect prefixes like 'ec2-ami-'
• Confusing AMI ID with image names
• Adding extra labels or colons

3. Given this Jenkins EC2 cloud agent snippet:

instanceCapStr: '3',
idleTerminationMinutes: 10,
ami: 'ami-0abc123def456',
remoteFS: '/home/jenkins'

What happens when 4 jobs are triggered simultaneously?

medium

A. Jenkins throws an error due to instance cap mismatch

B. 4 agents will be created ignoring the instance cap

C. All 4 jobs run on a single agent

D. Only 3 agents will be created; the 4th job waits in queue

Solution

1. Step 1: Understand instanceCapStr meaning

   The 'instanceCapStr' limits the max number of EC2 agents Jenkins can create to 3.

2. Step 2: Analyze job triggering behavior

   When 4 jobs start, Jenkins creates only 3 agents due to the cap; the 4th job waits in the queue.

3. Final Answer:

   Only 3 agents will be created; the 4th job waits in queue -> Option D

4. Quick Check:

   Instance cap limits agents = jobs queue if exceeded [OK]

Hint: Instance cap limits agents; extra jobs queue [OK]

Common Mistakes:

• Assuming Jenkins ignores instance cap
• Thinking all jobs run on one agent
• Expecting an error when cap exceeded

4. You configured an Azure cloud agent in Jenkins but agents never start. Which is the most likely cause?

medium

A. Instance cap set to zero

B. Jenkins master is offline

C. Incorrect Azure VM image name or ID

D. Idle timeout set too high

Solution

1. Step 1: Check Azure VM image configuration

   If the VM image name or ID is wrong, Jenkins cannot provision agents on Azure.

2. Step 2: Evaluate other options

   Jenkins master offline stops all jobs, but question focuses on agent provisioning. Instance cap zero means no agents, but usually agents would not start at all. Idle timeout affects termination, not start.

3. Final Answer:

   Incorrect Azure VM image name or ID -> Option C

4. Quick Check:

   Wrong VM image blocks agent start [OK]

Hint: Check VM image name first if agents don't start [OK]

Common Mistakes:

• Ignoring VM image correctness
• Blaming idle timeout for start issues
• Assuming Jenkins master offline without checking

5. You want to optimize Jenkins cloud agents on EC2 to reduce cost. Which combination of settings is best?

hard

A. Set instance cap high and idle termination to 60 minutes

B. Set instance cap low and idle termination to 5 minutes

C. Disable instance cap and idle termination

D. Set instance cap low and disable idle termination

Solution

1. Step 1: Understand instance cap effect

   A low instance cap limits max running agents, controlling cloud costs.

2. Step 2: Understand idle termination effect

   Setting idle termination to 5 minutes stops unused agents quickly, saving cost.

3. Step 3: Evaluate other options

   High cap or disabling termination increases cost. Disabling termination wastes resources.

4. Final Answer:

   Set instance cap low and idle termination to 5 minutes -> Option B

5. Quick Check:

   Low cap + quick termination = cost saving [OK]

Hint: Low cap + short idle timeout saves cloud cost [OK]

Common Mistakes:

• Setting high caps wastes money
• Disabling idle termination keeps idle agents running
• Ignoring cost impact of settings