Why HTTP requests with curl in scripts in Bash Scripting? - Purpose & Use Cases

The Big Idea

What if your computer could talk to websites and get data for you, all by itself?

The Scenario

Imagine you need to check the status of a website or download data from an online service every day. Doing this by opening a browser, clicking around, and copying information manually can take a lot of time and effort.

The Problem

Manually visiting websites or copying data is slow and easy to forget. It also leads to mistakes like copying wrong information or missing updates. Doing this repeatedly wastes your time and energy.

The Solution

Using curl in scripts lets you automatically send HTTP requests to websites or APIs. This means your computer can fetch data or check statuses for you, without any clicking or copying.

Before vs After

✗ Before

Open browser -> Type URL -> Copy data -> Paste into file

✓ After

curl https://example.com/api/data -o data.json

What It Enables

You can automate web data fetching and interaction, making repetitive online tasks fast, reliable, and hands-free.

Real Life Example

Automatically download daily weather reports from a public API and save them to a file every morning without lifting a finger.

Key Takeaways

Manual web data tasks are slow and error-prone.

curl automates HTTP requests in scripts.

This saves time and makes online tasks reliable.

Practice

1. What does the curl command do in a bash script?

easy

A. It creates a new directory on the system.

B. It sends HTTP requests to websites or APIs.

C. It compiles the bash script into an executable.

D. It displays the current date and time.

Solution

1. Step 1: Understand the purpose of curl

   The curl command is designed to transfer data from or to a server using various protocols, mainly HTTP.

2. Step 2: Match curl's function to the options

   Among the options, only sending HTTP requests matches curl's purpose.

3. Final Answer:

   It sends HTTP requests to websites or APIs. -> Option B

4. Quick Check:

   curl = HTTP requests [OK]

Hint: Remember curl talks to websites using HTTP [OK]

Common Mistakes:

• Confusing curl with system commands like mkdir or date
• Thinking curl compiles scripts
• Assuming curl only downloads files

2. Which of the following is the correct syntax to send a GET request with curl in a bash script?

easy

A. curl -send https://example.com

B. curl --post https://example.com

C. curl -X GET https://example.com

D. curl get https://example.com

Solution

1. Step 1: Identify the correct curl option for GET

   The -X GET option explicitly tells curl to use the GET method, which is the default.

2. Step 2: Check other options for correctness

   Options B, C, and D use invalid or incorrect flags for curl.

3. Final Answer:

   curl -X GET https://example.com -> Option C

4. Quick Check:

   GET request syntax = curl -X GET [OK]

Hint: Use -X GET to specify GET method explicitly [OK]

Common Mistakes:

• Using --post instead of -X POST for POST requests
• Typing invalid flags like -send or get
• Omitting the URL

3. What will the following script output?

response=$(curl -s -o /dev/null -w "%{http_code}" https://httpbin.org/status/200)
echo $response

medium

A. 200

B. OK

C. 404

D. curl: command not found

Solution

1. Step 1: Understand curl options used

   The -s option silences progress, -o /dev/null discards output, and -w "%{http_code}" prints the HTTP status code.

2. Step 2: Analyze the URL and expected status

   The URL https://httpbin.org/status/200 returns HTTP status 200, so the variable response will hold "200".

3. Final Answer:

   200 -> Option A

4. Quick Check:

   HTTP status code = 200 [OK]

Hint: Use -w "%{http_code}" to get HTTP status code [OK]

Common Mistakes:

• Expecting full page content instead of status code
• Missing quotes around %{http_code}
• Not using -s to silence output

4. Identify the error in this script snippet:

curl -X POST -d 'name=John' https://example.com/api
if [ $? -eq 0 ]; then
  echo "Success"
else
  echo "Failed"
fi

medium

A. No error; script works correctly

B. Missing quotes around URL

C. Single quotes prevent variable expansion in -d

D. Using -X POST with -d is invalid

Solution

1. Step 1: Check curl command syntax

   The command uses -X POST with -d 'name=John' correctly to send POST data.

2. Step 2: Verify script logic for success check

   The script checks the exit status $? properly and echoes success or failure accordingly.

3. Final Answer:

   No error; script works correctly -> Option A

4. Quick Check:

   Script syntax and logic = correct [OK]

Hint: Check exit status $? after curl for success [OK]

Common Mistakes:

• Thinking single quotes block variable expansion here (no variables used)
• Assuming -X POST and -d can't be combined
• Forgetting to check curl exit status

5. You want to write a bash script that sends a JSON POST request with curl to https://api.example.com/data and saves the response to response.json. Which script snippet correctly does this?

hard

A. curl -X POST -H 'Content-Type: application/json' -d '{key:value}' https://api.example.com/data >> response.json

B. curl -POST -H 'Content-Type: application/json' -d '{key:value}' https://api.example.com/data > response.json

C. curl -X POST -H Content-Type application/json -d '{"key":"value"}' https://api.example.com/data > response.json

D. curl -X POST -H 'Content-Type: application/json' -d '{"key":"value"}' https://api.example.com/data > response.json

Solution

1. Step 1: Check HTTP method and headers

   curl -X POST -H 'Content-Type: application/json' -d '{"key":"value"}' https://api.example.com/data > response.json correctly uses -X POST and sets the header -H 'Content-Type: application/json' with quotes.

2. Step 2: Validate JSON data and output redirection

   curl -X POST -H 'Content-Type: application/json' -d '{"key":"value"}' https://api.example.com/data > response.json uses valid JSON with escaped quotes and redirects output with > to overwrite response.json.

3. Step 3: Identify errors in other options

   curl -POST -H 'Content-Type: application/json' -d '{key:value}' https://api.example.com/data > response.json uses invalid -POST flag and unquoted JSON keys. curl -X POST -H Content-Type application/json -d '{"key":"value"}' https://api.example.com/data > response.json misses quotes around header value. curl -X POST -H 'Content-Type: application/json' -d '{key:value}' https://api.example.com/data >> response.json appends with >> and has invalid JSON keys.

4. Final Answer:

   curl -X POST -H 'Content-Type: application/json' -d '{"key":"value"}' https://api.example.com/data > response.json -> Option D

5. Quick Check:

   Correct POST JSON syntax and output redirection = curl -X POST -H 'Content-Type: application/json' -d '{"key":"value"}' https://api.example.com/data > response.json [OK]

Hint: Use -H for headers, -d with valid JSON, > to save output [OK]

Common Mistakes:

• Forgetting quotes around JSON keys and values
• Using invalid curl flags like -POST
• Appending instead of overwriting output file