curl for HTTP requests in Linux CLI - Deep Dive

Overview - curl for HTTP requests

What is it?

curl is a command-line tool used to send and receive data over the internet using HTTP and other protocols. It lets you make requests to websites or APIs and see their responses directly in your terminal. You can use curl to download files, test web services, or automate web interactions without opening a browser.

Why it matters

Without curl or similar tools, testing and interacting with web services would require a browser or complex software. curl makes it easy to quickly check if a website or API is working, automate data fetching, and debug network issues. This saves time and helps developers and system administrators work efficiently.

Where it fits

Before learning curl, you should understand basic command-line usage and what HTTP is. After mastering curl, you can explore scripting automation with shell scripts, API testing tools, or more advanced HTTP clients like HTTPie or Postman.

Mental Model

Core Idea

curl is like a mail carrier that sends your message (request) to a website and brings back the reply (response) so you can read it in your terminal.

Think of it like...

Imagine you want to send a letter to a friend and get their reply. curl is the postman who takes your letter (request), delivers it, and returns with your friend's answer (response) without you leaving your desk.

┌───────────────┐       ┌───────────────┐
│ Your Terminal │──────▶│   Website/API │
│   (curl)      │◀──────│   (Server)    │
└───────────────┘       └───────────────┘

Request sent by curl →   Response received by curl ←

Build-Up - 7 Steps

1

Foundation

Basic HTTP GET Request

Concept: How to use curl to fetch a webpage or resource using a simple GET request.

Run the command: curl https://example.com This sends a GET request to example.com and prints the HTML content of the homepage in your terminal.

Result

\n... (HTML content of example.com homepage) ...

Understanding that curl by default sends a GET request helps you quickly retrieve web pages or data without extra options.

2

Foundation

Saving Output to a File

Concept: How to save the response from curl into a file instead of printing it on screen.

Use the -o option: curl -o homepage.html https://example.com This downloads the webpage and saves it as homepage.html in your current folder.

Result

A file named homepage.html is created containing the webpage's HTML.

Knowing how to save output lets you keep data for later use or processing, which is essential for automation.

3

Intermediate

Sending POST Requests with Data

🤔Before reading on: do you think curl sends POST data by default or do you need to specify it? Commit to your answer.

Concept: How to send data to a server using POST method with curl.

Use -X POST to specify POST and -d to send data: curl -X POST -d "name=John&age=30" https://httpbin.org/post This sends form data to the server and shows the response.

Result

{... "form": {"name": "John", "age": "30"} ...} (JSON response showing received data)

Knowing how to send POST requests with data is key for interacting with APIs that accept input, like login forms or data submission.

4

Intermediate

Adding Custom Headers

🤔Before reading on: do you think curl sends any headers by default or do you need to add all headers manually? Commit to your answer.

Concept: How to add or modify HTTP headers in curl requests.

Use -H option to add headers: curl -H "Authorization: Bearer TOKEN123" https://api.example.com/data This sends a request with a custom Authorization header.

Result

Server processes request with the Authorization header and returns data accordingly.

Understanding headers lets you control authentication, content type, and other important request details.

5

Intermediate

Handling JSON Data in Requests

Concept: How to send JSON data in a POST request with the correct headers.

Use -H to set Content-Type and -d to send JSON: curl -X POST -H "Content-Type: application/json" -d '{"user":"alice"}' https://api.example.com/users This sends JSON data to the server.

Result

Server receives JSON and responds with confirmation or data.

Knowing how to send JSON is essential because many modern APIs use JSON as their data format.

6

Advanced

Using curl for Authentication

🤔Before reading on: do you think curl can handle authentication automatically or do you need to manually add headers or options? Commit to your answer.

Concept: How to use curl to authenticate with servers using basic or token methods.

For basic auth: curl -u username:password https://secure.example.com For token auth: curl -H "Authorization: Bearer TOKEN" https://secure.example.com This lets you access protected resources.

Result

Server grants access and returns protected data if credentials are correct.

Understanding authentication options in curl is crucial for accessing secure APIs and services.

7

Expert

Advanced Options and Debugging

🤔Before reading on: do you think curl shows detailed request and response info by default or do you need special options? Commit to your answer.

Concept: How to use verbose mode, follow redirects, and handle cookies with curl.

Use -v for verbose output: curl -v https://example.com Use -L to follow redirects: curl -L http://short.url Use -c and -b to save and send cookies: curl -c cookies.txt -b cookies.txt https://example.com These help debug and automate complex interactions.

Result

Verbose output shows headers and connection info; redirects are followed; cookies persist between requests.

Knowing these options helps troubleshoot issues and automate sessions that require state, like login persistence.

Under the Hood

curl works by opening a network connection to the server using the specified protocol (usually HTTP). It constructs an HTTP request message with method, headers, and optional body, then sends it over the connection. The server processes the request and sends back an HTTP response message, which curl reads and outputs or saves. curl handles low-level details like TCP connections, SSL/TLS encryption, redirects, and cookies internally.

Why designed this way?

curl was designed to be a simple, flexible tool that works in scripts and terminals without a graphical interface. It supports many protocols and options to cover diverse use cases. The design favors composability and transparency, letting users see raw requests and responses for debugging. Alternatives like browsers hide these details, but curl exposes them for power users.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ curl Client   │──────▶│ TCP/IP Stack  │──────▶│   Server      │
│ (builds req)  │       │ (networking)  │       │ (process req) │
│               │◀──────│               │◀──────│               │
│ (shows resp)  │       │               │       │               │
└───────────────┘       └───────────────┘       └───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does curl send POST requests by default when you run 'curl https://example.com'? Commit to yes or no.

Common Belief: curl sends POST requests by default when you run it with a URL.

Reality: curl sends GET requests by default unless you specify otherwise.

Why it matters: Assuming POST by default can cause confusion when testing APIs or websites expecting GET, leading to unexpected results.

Quick: Do you think curl automatically follows redirects without options? Commit to yes or no.

Common Belief: curl follows HTTP redirects automatically without extra options.

Reality: curl does NOT follow redirects unless you add the -L option.

Why it matters: Not following redirects can cause curl to show only the redirect response, confusing users who expect the final page.

Quick: Does curl send cookies automatically between requests? Commit to yes or no.

Common Belief: curl manages cookies automatically across multiple requests by default.

Reality: curl does not save or send cookies between requests unless you explicitly use cookie options.

Why it matters: Without managing cookies, sessions requiring login or state won't work across multiple curl commands.

Quick: Is it true that curl can only handle HTTP and HTTPS protocols? Commit to yes or no.

Common Belief: curl only works with HTTP and HTTPS protocols.

Reality: curl supports many protocols including FTP, SMTP, IMAP, and more.

Why it matters: Limiting curl to HTTP misses its power as a versatile network tool for many protocols.

Expert Zone

1

curl's handling of HTTP/2 and HTTP/3 protocols can improve performance but requires recent versions and specific options.

2

The order of options matters in curl commands; some options override others if placed differently.

3

curl can be used as a library (libcurl) embedded in programs, which is how many applications perform network requests.

When NOT to use

curl is not ideal for complex API testing with assertions or interactive debugging; tools like Postman or HTTPie provide richer interfaces. For large file downloads with resume support, dedicated download managers may be better.

Production Patterns

In production, curl is often used in shell scripts for health checks, automated API calls, or deployment pipelines. It is combined with tools like jq to parse JSON responses and integrated into CI/CD workflows for testing endpoints.

Connections

HTTP Protocol

curl builds and sends HTTP requests and reads HTTP responses.

Understanding HTTP methods, headers, and status codes helps you use curl effectively to interact with web services.

Shell Scripting

curl is often used inside shell scripts to automate web requests and process responses.

Knowing shell scripting lets you combine curl commands into powerful automation workflows.

Postal Mail System

curl acts like a mail carrier delivering messages and bringing replies.

This analogy helps grasp the request-response cycle and the role of headers as message envelopes.

Common Pitfalls

#1 Expecting curl to follow redirects automatically.

Wrong approach:

curl http://short.url

Correct approach:

curl -L http://short.url

Root cause:

Not knowing that curl requires -L to follow HTTP redirects.

#2 Sending JSON data without setting Content-Type header.

Wrong approach:

curl -X POST -d '{"key":"value"}' https://api.example.com

Correct approach:

curl -X POST -H "Content-Type: application/json" -d '{"key":"value"}' https://api.example.com

Root cause:

Forgetting to tell the server the data format causes it to misinterpret the request.

#3 Assuming curl saves cookies between separate commands automatically.

Wrong approach:

curl -c cookies.txt https://example.com
curl https://example.com/profile

Correct approach:

curl -c cookies.txt https://example.com
curl -b cookies.txt https://example.com/profile

Root cause:

Not using both cookie save (-c) and send (-b) options to maintain session state.

Key Takeaways

curl is a simple but powerful tool to send HTTP requests and receive responses directly from the command line.

By default, curl sends GET requests and prints the response body to the terminal unless options change this behavior.

You can customize requests with headers, data, authentication, and control output to files or scripts.

Understanding curl's options for redirects, cookies, and verbose output is essential for effective debugging and automation.

curl supports many protocols and advanced features, making it a versatile tool beyond just basic web requests.