Shell steps (sh, bat) in Jenkins - Deep Dive

Overview - Shell steps (sh, bat)

What is it?

Shell steps in Jenkins allow you to run command-line scripts during your automated build or deployment process. The 'sh' step runs shell commands on Unix-like systems, while the 'bat' step runs batch commands on Windows. These steps let Jenkins interact with the operating system to perform tasks like compiling code, running tests, or managing files.

Why it matters

Without shell steps, Jenkins would be limited to only built-in functions and plugins, making automation less flexible. Shell steps let you use any command-line tool or script, unlocking the full power of your system. This means faster, more customized automation that fits your exact needs, saving time and reducing errors.

Where it fits

Before learning shell steps, you should understand basic Jenkins pipelines and how automation works. After mastering shell steps, you can explore advanced pipeline scripting, error handling, and integrating other tools like Docker or Kubernetes.

Mental Model

Core Idea

Shell steps in Jenkins are like remote controls that let Jenkins operate your computer by typing commands for you.

Think of it like...

Imagine Jenkins as a robot assistant that follows your spoken instructions. The shell steps are like telling the robot exactly what to type on the computer keyboard to get things done.

┌───────────────┐
│   Jenkins     │
│  Pipeline     │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Shell Step    │
│ (sh or bat)   │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Operating     │
│ System Shell  │
│ (bash or cmd) │
└───────────────┘

Build-Up - 7 Steps

1

Foundation

What are Shell Steps in Jenkins

Concept: Introduce the basic idea of shell steps as commands Jenkins runs on the system.

In Jenkins pipelines, shell steps let you run commands just like you would in a terminal. The 'sh' step runs commands on Linux or macOS, while 'bat' runs commands on Windows. For example, sh('echo Hello') prints Hello in the console.

Result

Jenkins executes the command and shows the output in the build log.

Understanding that Jenkins can run system commands directly opens up endless automation possibilities.

2

Foundation

Basic Syntax of sh and bat Steps

Concept: Learn the simple syntax to write shell steps in Jenkins pipelines.

The 'sh' step uses a string with shell commands: sh 'ls -l'. The 'bat' step uses batch commands: bat 'dir'. You can write multiple commands by separating them with newlines inside triple quotes or using semicolons.

Result

Commands run in order, and their output appears in Jenkins logs.

Knowing the syntax lets you start writing your own commands to control the build process.

3

Intermediate

Using Multiline Scripts in Shell Steps

🤔Before reading on: do you think you can run multiple commands in one sh step by separating them with newlines or semicolons? Commit to your answer.

Concept: Learn how to write multiple commands in one shell step for complex tasks.

You can write multiline scripts inside triple quotes for 'sh' or 'bat'. For example: sh ''' echo Start ls -l echo End ''' This runs all commands in sequence.

Result

All commands execute in order, and their combined output shows in the build log.

Combining commands in one step reduces overhead and keeps related commands together for clarity.

4

Intermediate

Handling Command Failures in Shell Steps

🤔Before reading on: do you think Jenkins stops the build automatically if a command in sh or bat fails? Commit to your answer.

Concept: Understand how Jenkins reacts when shell commands fail and how to control this behavior.

By default, if a command returns a non-zero exit code, Jenkins marks the build as failed and stops. You can override this by adding 'returnStatus: true' to get the exit code without failing, or use 'try-catch' in scripted pipelines.

Result

You can detect failures and decide whether to stop or continue the build.

Knowing how Jenkins handles errors helps you write robust pipelines that handle problems gracefully.

5

Intermediate

Passing Environment Variables to Shell Steps

Concept: Learn how to use environment variables inside shell steps to make scripts dynamic.

You can access Jenkins environment variables inside shell steps. For example, in 'sh', use $VAR_NAME; in 'bat', use %VAR_NAME%. You can also define variables in the pipeline and pass them to shell commands.

Result

Scripts can adapt based on variables, making automation flexible.

Using environment variables connects Jenkins pipeline data with shell commands, enabling dynamic behavior.

6

Advanced

Using Shell Steps for Cross-Platform Pipelines

🤔Before reading on: do you think the same shell commands work on both Linux and Windows in Jenkins? Commit to your answer.

Concept: Explore how to write pipelines that run on different operating systems using sh and bat steps.

Since 'sh' works on Unix-like systems and 'bat' on Windows, you can use 'isUnix()' in Jenkins to detect OS and run the appropriate step. For example: if (isUnix()) { sh 'ls' } else { bat 'dir' } This makes pipelines portable.

Result

Your pipeline runs correctly on different OS agents without errors.

Handling OS differences ensures your automation works everywhere, increasing reliability.

7

Expert

Advanced Shell Step Features and Pitfalls

🤔Before reading on: do you think shell steps always run in the same directory as the Jenkins workspace? Commit to your answer.

Concept: Understand advanced behaviors like working directory, quoting issues, and security considerations in shell steps.

Shell steps run in the workspace directory by default, but you can change directories inside the script. Quoting is critical: use single quotes in pipeline to avoid Groovy interpolation, and double quotes inside shell for variables. Beware of shell injection risks when using user input. Also, multi-line scripts run in a single shell session, so variables persist between commands.

Result

You write safer, more predictable shell scripts in Jenkins pipelines.

Mastering these details prevents subtle bugs and security holes in your automation.

Under the Hood

When Jenkins runs a shell step, it starts a new shell process on the agent machine. For 'sh', it typically launches /bin/sh or bash; for 'bat', it runs cmd.exe. Jenkins sends the command string to this shell process, which executes it line by line. The shell process returns an exit code to Jenkins, which uses it to determine success or failure. Output streams (stdout and stderr) are captured and shown in Jenkins logs.

Why designed this way?

This design leverages the native shell environment of the agent machine, allowing Jenkins to run any command available there without needing special plugins. It keeps Jenkins lightweight and flexible. Alternatives like embedding commands inside Jenkins code would limit power and increase complexity.

┌───────────────┐
│ Jenkins Agent │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Shell Process │
│ (bash or cmd) │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Operating     │
│ System Kernel │
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does Jenkins automatically stop the build if any shell command fails? Commit yes or no.

Common Belief: Jenkins always stops the build immediately if any shell command returns an error.

Reality: By default, Jenkins stops on errors, but you can configure shell steps to continue by capturing exit codes or using try-catch blocks.

Why it matters: Assuming Jenkins always stops can lead to overly cautious pipelines or missed opportunities to handle errors gracefully.

Quick: Can you use the same shell commands in 'sh' and 'bat' steps? Commit yes or no.

Common Belief: The same shell commands work in both 'sh' and 'bat' steps regardless of OS.

Reality: 'sh' runs Unix shell commands; 'bat' runs Windows batch commands. They have different syntax and commands.

Why it matters: Using wrong commands causes build failures and confusion about pipeline behavior.

Quick: Do shell steps run in the Jenkins master machine by default? Commit yes or no.

Common Belief: Shell steps always run on the Jenkins master server.

Reality: Shell steps run on the agent (worker) node assigned to the job, not the master, unless no agents are configured.

Why it matters: Misunderstanding this can cause confusion about environment differences and where commands execute.

Quick: Does each command in a multiline shell step run in a separate shell session? Commit yes or no.

Common Belief: Each command in a multiline shell step runs in its own shell session, so variables don't persist.

Reality: All commands in a multiline shell step run in the same shell session, so variables and state persist between commands.

Why it matters: Incorrect assumptions here can cause bugs when scripts rely on variable persistence.

Expert Zone

1

Shell steps inherit the environment from the Jenkins agent but can be influenced by pipeline environment variables, which may differ from user shells.

2

Quoting and escaping rules differ between Groovy, shell, and batch scripts, requiring careful attention to avoid subtle bugs.

3

Security risks arise if shell commands include untrusted input, making parameter sanitization and use of credentials plugins critical.

When NOT to use

Avoid shell steps when complex logic or cross-platform consistency is needed; instead, use dedicated Jenkins plugins, pipeline steps, or containerized scripts for better control and security.

Production Patterns

In production, shell steps are often wrapped in functions or shared libraries for reuse, combined with error handling and logging. They are also used to invoke container tools like Docker or Kubernetes CLI for deployment automation.

Connections

Continuous Integration Pipelines

Shell steps are fundamental building blocks within CI pipelines to automate build and test commands.

Understanding shell steps helps grasp how CI pipelines execute real work on machines.

Operating System Shells

Shell steps directly invoke OS shells like bash or cmd, bridging Jenkins automation with system commands.

Knowing OS shell basics improves writing effective shell steps and debugging failures.

Remote Control Systems

Shell steps act like remote control commands sent to agents, similar to how remote devices receive instructions.

Seeing shell steps as remote commands clarifies their role in distributed automation.

Common Pitfalls

#1 Writing shell commands without proper quoting causes syntax errors or unexpected behavior.

Wrong approach:

sh "echo Hello $USER"

Correct approach:

sh 'echo Hello $USER'

Root cause:

Using double quotes in Groovy strings causes premature variable interpolation, breaking shell command syntax.

#2 Assuming shell steps run on Jenkins master leads to confusion about environment differences.

Wrong approach:

sh 'pwd' // expecting master directory

Correct approach:

sh 'pwd' // runs on agent workspace directory

Root cause:

Not understanding Jenkins distributed architecture and agent execution.

#3 Ignoring exit codes causes pipelines to continue despite failures.

Wrong approach:

sh 'some-failing-command || true'

Correct approach:

def status = sh(script: 'some-failing-command', returnStatus: true)
if (status != 0) { error('Command failed') }

Root cause:

Not handling shell command exit codes properly leads to silent failures.

Key Takeaways

Shell steps let Jenkins run system commands, unlocking powerful automation capabilities.

The 'sh' step runs Unix shell commands; 'bat' runs Windows batch commands, so use them according to the agent OS.

Multiline scripts and environment variables make shell steps flexible and dynamic.

Understanding error handling and execution context prevents common pipeline failures.

Mastering shell steps is essential for building robust, cross-platform Jenkins pipelines.