HTTP requests with curl in scripts in Bash Scripting - Deep Dive

Overview - HTTP requests with curl in scripts

What is it?

HTTP requests with curl in scripts means using the curl command inside shell scripts to send and receive data over the internet. Curl is a tool that talks to websites or servers using the HTTP protocol, like when you open a webpage. By putting curl in scripts, you can automate tasks like downloading files, sending data, or checking website status without manual typing.

Why it matters

Without curl in scripts, you would have to manually open browsers or type commands every time you want to interact with web services. Automating HTTP requests saves time, reduces errors, and allows computers to handle repetitive internet tasks. This is crucial for monitoring websites, downloading updates, or integrating with online services automatically.

Where it fits

Before learning this, you should know basic shell scripting and understand what HTTP is at a simple level. After mastering curl in scripts, you can explore more advanced automation tools like wget, APIs with authentication, or scripting with programming languages like Python for web interactions.

Mental Model

Core Idea

Curl in scripts is like a remote control that sends commands over the internet to websites or servers and brings back their responses automatically.

Think of it like...

Imagine curl as a postal worker who delivers letters (requests) to a friend's house (server) and brings back their replies (responses) without you leaving your home (computer).

┌─────────────┐      ┌─────────────┐      ┌─────────────┐
│  Script     │─────▶│   curl      │─────▶│  Server     │
│ (your code) │      │ (messenger) │      │ (website)   │
└─────────────┘      └─────────────┘      └─────────────┘
       ▲                                         │
       │                                         ▼
       └─────────────────────────────────────────┘
                 Response returned to script

Build-Up - 7 Steps

1

Foundation

What is curl and HTTP basics

Concept: Introduce curl as a command-line tool and explain the basics of HTTP requests.

Curl is a program you run in the terminal to talk to websites. HTTP is the language websites use to send and receive information. Curl can send different types of HTTP requests like GET (ask for data) or POST (send data). For example, running 'curl https://example.com' asks the website for its homepage.

Result

The terminal shows the HTML content of the example.com homepage.

Understanding curl as a simple messenger for HTTP requests helps you see how scripts can automate web interactions.

2

Foundation

Running curl commands in scripts

Concept: How to put curl commands inside shell scripts to automate HTTP requests.

You can write a shell script file with curl commands. For example: #!/bin/bash curl https://example.com Save this as script.sh, make it executable with 'chmod +x script.sh', then run './script.sh'. The script runs curl and shows the website content automatically.

Result

Running the script prints the website's HTML content without typing curl manually.

Embedding curl in scripts turns manual web requests into repeatable automated tasks.

3

Intermediate

Using curl options for HTTP methods

🤔Before reading on: do you think curl can only send GET requests or can it send other types like POST? Commit to your answer.

Concept: Curl can send different HTTP methods using options like -X POST or -d for data.

By default, curl sends GET requests. To send data, use POST method: curl -X POST -d 'name=John' https://example.com/api This sends 'name=John' to the server. You can also add headers with -H, like 'Content-Type: application/json'.

Result

The server receives a POST request with the data 'name=John'.

Knowing how to change HTTP methods and add data lets you interact with APIs and web services beyond just fetching pages.

4

Intermediate

Saving and processing curl output

🤔Before reading on: do you think curl output can be saved directly to a file or only shown on screen? Commit to your answer.

Concept: Curl can save responses to files or pass them to other commands for processing.

Use '-o filename' to save output: curl https://example.com -o page.html Or use pipes to process output: curl https://example.com | grep 'Welcome' This finds the word 'Welcome' in the page content.

Result

The webpage is saved as page.html or filtered output is shown on screen.

Saving and processing output allows scripts to handle data automatically, enabling tasks like monitoring or data extraction.

5

Intermediate

Handling errors and HTTP status codes

🤔Before reading on: do you think curl automatically stops on errors or you must check status manually? Commit to your answer.

Concept: Curl can show HTTP status codes and handle errors, but scripts must check these to react properly.

Use '-w "%{http_code}"' to get status code: status=$(curl -s -o /dev/null -w "%{http_code}" https://example.com) if [ "$status" -eq 200 ]; then echo "Success" else echo "Failed with status $status" fi This script checks if the website responded OK (200).

Result

The script prints 'Success' if the site is reachable, otherwise an error message.

Checking HTTP status codes in scripts prevents silent failures and helps build reliable automation.

6

Advanced

Using curl with authentication and headers

🤔Before reading on: do you think curl can handle login or API keys automatically? Commit to your answer.

Concept: Curl supports sending authentication info and custom headers for secure or API access.

To send basic authentication: curl -u username:password https://example.com/secure To add API keys or tokens: curl -H "Authorization: Bearer TOKEN" https://api.example.com/data This lets scripts access protected resources.

Result

The server accepts the request if credentials or tokens are valid.

Handling authentication in curl scripts enables automation with secure services and APIs.

7

Expert

Optimizing curl scripts for production use

🤔Before reading on: do you think running many curl commands in scripts is always safe and efficient? Commit to your answer.

Concept: Advanced curl scripting includes retries, timeouts, parallel requests, and error logging for robust automation.

Example with retries and timeout: for i in {1..3}; do curl --max-time 10 https://example.com && break || sleep 5 done Use '--fail' to make curl exit on HTTP errors: curl --fail https://example.com Logging errors: curl https://example.com 2>error.log These techniques make scripts reliable in real environments.

Result

Scripts retry failed requests, avoid hanging, and log problems for troubleshooting.

Building resilience and observability into curl scripts is key for dependable automation in production.

Under the Hood

Curl works by opening a network connection to the server's address and sending a formatted HTTP request string. It waits for the server's response, reads the data stream, and outputs it to the terminal or file. Internally, curl uses system network libraries to handle protocols like TCP/IP and manages headers, cookies, and data encoding transparently.

Why designed this way?

Curl was designed as a lightweight, flexible tool to support many protocols and options from the command line. Its modular design allows easy extension and scripting. Alternatives existed but were often heavier or less script-friendly. Curl's design balances power and simplicity, making it ideal for automation.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│  User Script  │──────▶│    curl CLI   │──────▶│  Network Stack│
│ (bash script) │       │ (command line)│       │ (TCP/IP etc.) │
└───────────────┘       └───────────────┘       └───────────────┘
                                │                      │
                                ▼                      ▼
                        ┌───────────────┐       ┌───────────────┐
                        │ HTTP Request  │──────▶│   Server      │
                        └───────────────┘       └───────────────┘
                                ▲                      │
                                │                      ▼
                        ┌───────────────┐       ┌───────────────┐
                        │ HTTP Response │◀──────│   Server      │
                        └───────────────┘       └───────────────┘
                                │
                                ▼
                        ┌───────────────┐
                        │ Output to     │
                        │ Script/Screen │
                        └───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does curl automatically retry failed HTTP requests? Commit to yes or no.

Common Belief: Curl automatically retries requests if they fail due to network errors.

Reality: Curl does not retry failed requests unless explicitly told to do so with scripting or options.

Why it matters: Assuming automatic retries can cause scripts to miss failures or behave unpredictably in unreliable networks.

Quick: Does curl save cookies between separate script runs by default? Commit to yes or no.

Common Belief: Curl remembers cookies automatically across different script executions.

Reality: Curl does not save cookies between runs unless you use cookie jar options to store and load them.

Why it matters: Not managing cookies can cause authentication or session issues when automating web interactions.

Quick: Can curl send JSON data with just the -d option without headers? Commit to yes or no.

Common Belief: Using -d with JSON data is enough for the server to understand the content type.

Reality: You must also set the 'Content-Type: application/json' header explicitly for the server to parse JSON correctly.

Why it matters: Missing headers can cause servers to reject or misinterpret data, breaking automation.

Quick: Does curl output only the response body by default? Commit to yes or no.

Common Belief: Curl shows only the main content of the response, not headers or status codes.

Reality: By default, curl outputs the response body only; headers and status codes require extra options to view.

Why it matters: Not knowing this can confuse learners when debugging or trying to check server responses.

Expert Zone

1

Curl's --fail option causes it to exit with an error on HTTP 4xx or 5xx codes, but it still outputs the response body unless redirected.

2

Using curl's --next option allows chaining multiple requests in one command, useful for complex workflows.

3

Curl supports connection reuse and HTTP/2 multiplexing, which can improve performance in scripts making many requests.

When NOT to use

Curl is not ideal for very complex API interactions requiring JSON parsing or conditional logic; in such cases, use scripting languages like Python with HTTP libraries. Also, for large file downloads with resume support, wget might be better.

Production Patterns

In production, curl scripts often run as cron jobs for monitoring, use environment variables for secrets, log outputs for audits, and combine with tools like jq to parse JSON responses for decision-making.

Connections

REST APIs

Curl scripts are often used to interact with REST APIs by sending HTTP requests and processing responses.

Understanding curl helps you automate API testing and integration without needing complex tools.

Network Protocols

Curl operates over network protocols like TCP/IP and HTTP, making it a practical example of these protocols in action.

Knowing how curl works deepens your understanding of how data travels over the internet.

Postal Service Logistics

Like postal services route and deliver mail, curl routes requests and responses between client and server.

Seeing curl as a delivery system clarifies the flow of requests and responses in network communication.

Common Pitfalls

#1 Not checking HTTP status codes leads to assuming requests succeeded.

Wrong approach:

curl https://example.com
if [ $? -eq 0 ]; then echo 'Success'; else echo 'Fail'; fi

Correct approach:

status=$(curl -s -o /dev/null -w "%{http_code}" https://example.com)
if [ "$status" -eq 200 ]; then echo 'Success'; else echo "Fail with status $status"; fi

Root cause:

Confusing curl's exit code with HTTP success; curl exits 0 if it can connect, not if HTTP response is OK.

#2 Sending JSON data without setting Content-Type header causes server errors.

Wrong approach:

curl -X POST -d '{"name":"John"}' https://api.example.com

Correct approach:

curl -X POST -H 'Content-Type: application/json' -d '{"name":"John"}' https://api.example.com

Root cause:

Assuming data format is inferred automatically by the server without explicit headers.

#3 Assuming curl saves cookies between script runs causes login failures.

Wrong approach:

curl -c cookies.txt https://example.com/login
curl https://example.com/dashboard

Correct approach:

curl -c cookies.txt https://example.com/login
curl -b cookies.txt https://example.com/dashboard

Root cause:

Not using cookie jar options to persist session data across requests.

Key Takeaways

Curl is a command-line tool that sends HTTP requests and receives responses, perfect for automating web tasks in scripts.

Scripts using curl can perform various HTTP methods, handle data, and interact with APIs securely by adding headers and authentication.

Checking HTTP status codes and handling errors in curl scripts is essential for reliable automation.

Advanced curl scripting includes retries, timeouts, and logging to make scripts robust in real-world environments.

Understanding curl's internal network communication helps you debug and optimize your scripts effectively.