HTTP requests with curl in scripts in Bash Scripting

Introduction

curl lets you talk to websites or servers from your script. You can get or send data easily.

Check if a website is online by requesting its homepage.

Download a file or data from the internet automatically.

Send data to a web service, like submitting a form or API call.

Test if an API is working by making requests from your script.

Automate repetitive web tasks without opening a browser.

Syntax

Bash Scripting

curl [options] URL

Replace URL with the web address you want to access.

Options change what curl does, like -X to set request type or -d to send data.

Examples

Simple GET request to fetch the homepage of example.com.

Bash Scripting

curl https://example.com

Send a POST request with data name=John to a form URL.

Bash Scripting

curl -X POST -d "name=John" https://example.com/form

Fetch only the headers from the server, not the full page.

Bash Scripting

curl -I https://example.com

Download a file and save it locally as file.txt.

Bash Scripting

curl -o file.txt https://example.com/data.txt

Sample Program

This script uses curl to silently check the HTTP status code of a website. If the code is 200, it means the site is reachable.

Bash Scripting

#!/bin/bash

# Simple script to check if a website is reachable
URL="https://example.com"

response=$(curl -s -o /dev/null -w "%{http_code}" "$URL")

if [ "$response" -eq 200 ]; then
  echo "Website is reachable with status code 200."
else
  echo "Website returned status code $response."
fi

Important Notes

Use -s to make curl silent so it doesn't print progress.

The -w "%{http_code}" option tells curl to output the HTTP status code.

Redirect output to /dev/null with -o /dev/null if you only want the status code.

Summary

curl lets scripts communicate with websites easily.

You can send GET, POST, and other HTTP requests with options.

Use curl in scripts to automate web checks and data transfers.

Practice

1. What does the curl command do in a bash script?

easy

A. It creates a new directory on the system.

B. It sends HTTP requests to websites or APIs.

C. It compiles the bash script into an executable.

D. It displays the current date and time.

Solution

1. Step 1: Understand the purpose of curl

   The curl command is designed to transfer data from or to a server using various protocols, mainly HTTP.

2. Step 2: Match curl's function to the options

   Among the options, only sending HTTP requests matches curl's purpose.

3. Final Answer:

   It sends HTTP requests to websites or APIs. -> Option B

4. Quick Check:

   curl = HTTP requests [OK]

Hint: Remember curl talks to websites using HTTP [OK]

Common Mistakes:

• Confusing curl with system commands like mkdir or date
• Thinking curl compiles scripts
• Assuming curl only downloads files

2. Which of the following is the correct syntax to send a GET request with curl in a bash script?

easy

A. curl -send https://example.com

B. curl --post https://example.com

C. curl -X GET https://example.com

D. curl get https://example.com

Solution

1. Step 1: Identify the correct curl option for GET

   The -X GET option explicitly tells curl to use the GET method, which is the default.

2. Step 2: Check other options for correctness

   Options B, C, and D use invalid or incorrect flags for curl.

3. Final Answer:

   curl -X GET https://example.com -> Option C

4. Quick Check:

   GET request syntax = curl -X GET [OK]

Hint: Use -X GET to specify GET method explicitly [OK]

Common Mistakes:

• Using --post instead of -X POST for POST requests
• Typing invalid flags like -send or get
• Omitting the URL

3. What will the following script output?

response=$(curl -s -o /dev/null -w "%{http_code}" https://httpbin.org/status/200)
echo $response

medium

A. 200

B. OK

C. 404

D. curl: command not found

Solution

1. Step 1: Understand curl options used

   The -s option silences progress, -o /dev/null discards output, and -w "%{http_code}" prints the HTTP status code.

2. Step 2: Analyze the URL and expected status

   The URL https://httpbin.org/status/200 returns HTTP status 200, so the variable response will hold "200".

3. Final Answer:

   200 -> Option A

4. Quick Check:

   HTTP status code = 200 [OK]

Hint: Use -w "%{http_code}" to get HTTP status code [OK]

Common Mistakes:

• Expecting full page content instead of status code
• Missing quotes around %{http_code}
• Not using -s to silence output

4. Identify the error in this script snippet:

curl -X POST -d 'name=John' https://example.com/api
if [ $? -eq 0 ]; then
  echo "Success"
else
  echo "Failed"
fi

medium

A. No error; script works correctly

B. Missing quotes around URL

C. Single quotes prevent variable expansion in -d

D. Using -X POST with -d is invalid

Solution

1. Step 1: Check curl command syntax

   The command uses -X POST with -d 'name=John' correctly to send POST data.

2. Step 2: Verify script logic for success check

   The script checks the exit status $? properly and echoes success or failure accordingly.

3. Final Answer:

   No error; script works correctly -> Option A

4. Quick Check:

   Script syntax and logic = correct [OK]

Hint: Check exit status $? after curl for success [OK]

Common Mistakes:

• Thinking single quotes block variable expansion here (no variables used)
• Assuming -X POST and -d can't be combined
• Forgetting to check curl exit status

5. You want to write a bash script that sends a JSON POST request with curl to https://api.example.com/data and saves the response to response.json. Which script snippet correctly does this?

hard

A. curl -X POST -H 'Content-Type: application/json' -d '{key:value}' https://api.example.com/data >> response.json

B. curl -POST -H 'Content-Type: application/json' -d '{key:value}' https://api.example.com/data > response.json

C. curl -X POST -H Content-Type application/json -d '{"key":"value"}' https://api.example.com/data > response.json

D. curl -X POST -H 'Content-Type: application/json' -d '{"key":"value"}' https://api.example.com/data > response.json

Solution

1. Step 1: Check HTTP method and headers

   curl -X POST -H 'Content-Type: application/json' -d '{"key":"value"}' https://api.example.com/data > response.json correctly uses -X POST and sets the header -H 'Content-Type: application/json' with quotes.

2. Step 2: Validate JSON data and output redirection

   curl -X POST -H 'Content-Type: application/json' -d '{"key":"value"}' https://api.example.com/data > response.json uses valid JSON with escaped quotes and redirects output with > to overwrite response.json.

3. Step 3: Identify errors in other options

   curl -POST -H 'Content-Type: application/json' -d '{key:value}' https://api.example.com/data > response.json uses invalid -POST flag and unquoted JSON keys. curl -X POST -H Content-Type application/json -d '{"key":"value"}' https://api.example.com/data > response.json misses quotes around header value. curl -X POST -H 'Content-Type: application/json' -d '{key:value}' https://api.example.com/data >> response.json appends with >> and has invalid JSON keys.

4. Final Answer:

   curl -X POST -H 'Content-Type: application/json' -d '{"key":"value"}' https://api.example.com/data > response.json -> Option D

5. Quick Check:

   Correct POST JSON syntax and output redirection = curl -X POST -H 'Content-Type: application/json' -d '{"key":"value"}' https://api.example.com/data > response.json [OK]

Hint: Use -H for headers, -d with valid JSON, > to save output [OK]

Common Mistakes:

• Forgetting quotes around JSON keys and values
• Using invalid curl flags like -POST
• Appending instead of overwriting output file