SimpleHttpOperator for API calls in Apache Airflow - Step-by-Step Execution

Process Flow - SimpleHttpOperator for API calls

Start DAG

↓

SimpleHttpOperator triggers

↓

Send HTTP Request

↓

Receive HTTP Response

↓

Process Response (optional)

↓

Task Success or Failure

↓

End DAG

The SimpleHttpOperator starts when the DAG runs, sends an HTTP request, receives the response, optionally processes it, then marks the task success or failure.

Execution Sample

Apache Airflow

from airflow import DAG
from airflow.providers.http.operators.http import SimpleHttpOperator
from datetime import datetime

default_args = {'start_date': datetime(2024, 1, 1)}

dag = DAG('api_call_dag', default_args=default_args, schedule_interval='@daily')

api_task = SimpleHttpOperator(
    task_id='call_api',
    method='GET',
    http_conn_id='my_api',
    endpoint='data',
    dag=dag
)

This code defines a DAG with a SimpleHttpOperator task that sends a GET request to the 'data' endpoint using a connection named 'my_api'.

Process Table

Step	Action	HTTP Method	Endpoint	Request Sent	Response Status	Task State
1	DAG starts, SimpleHttpOperator triggers	GET	data	No	N/A	Pending
2	Send HTTP request	GET	data	Yes	N/A	Running
3	Receive HTTP response	GET	data	Yes	200 OK	Running
4	Process response (optional)	GET	data	Yes	200 OK	Running
5	Mark task success	GET	data	Yes	200 OK	Success

💡 Task completes successfully after receiving 200 OK response.

Status Tracker

Variable	Start	After Step 2	After Step 3	After Step 5
task_state	Pending	Running	Running	Success
http_response_status	None	None	200 OK	200 OK
request_sent	False	True	True	True

Key Moments - 2 Insights

Why does the task state change from Pending to Running before the HTTP request is sent?

The task state changes to Running as soon as the operator starts executing (Step 2 in execution_table), even before the HTTP request is actually sent.

What happens if the HTTP response status is not 200 OK?

If the response status is not 200 OK, the task may fail or retry depending on configuration, but in this trace (Step 3), 200 OK leads to success.

Visual Quiz - 3 Questions

Look at the execution table, what is the task state at Step 3?

A Pending

B Success

C Running

D Failed

💡 Hint

Check the 'Task State' column at Step 3 in the execution_table.

At which step is the HTTP request actually sent?

A Step 2

B Step 4

C Step 1

D Step 5

💡 Hint

Look at the 'Request Sent' column in execution_table to find when it changes to Yes.

If the response status was 404, how would the task state likely change at Step 5?

A Running

B Failed

C Success

D Pending

💡 Hint

Consider what happens when the HTTP response status is not 200 OK as explained in key_moments.

Concept Snapshot

SimpleHttpOperator sends HTTP requests in Airflow DAGs.
Define method, endpoint, and connection ID.
Task state moves: Pending -> Running -> Success/Failure.
Response status 200 means success.
Used for calling APIs during workflows.

Full Transcript

This visual execution trace shows how the SimpleHttpOperator works in Airflow. When the DAG starts, the operator triggers and changes the task state from Pending to Running. It sends an HTTP GET request to the specified endpoint using the configured connection. After sending the request, it waits for the HTTP response. Upon receiving a 200 OK response, the task processes the response if needed and then marks the task as Success. Variables like task_state and http_response_status update step-by-step. Key moments clarify why the task state changes early and what happens on non-200 responses. The quiz tests understanding of task states and request timing. This helps beginners see the exact flow of an API call task in Airflow.

Practice

1. What is the main purpose of the SimpleHttpOperator in Airflow?

easy

A. To schedule Airflow DAGs

B. To manage database connections

C. To make HTTP API calls within Airflow tasks

D. To send emails from Airflow

Solution

1. Step 1: Understand the role of SimpleHttpOperator

   The SimpleHttpOperator is designed to perform HTTP requests like GET or POST inside Airflow tasks.

2. Step 2: Differentiate from other Airflow features

   Scheduling DAGs, managing databases, or sending emails are handled by other operators, not SimpleHttpOperator.

3. Final Answer:

   To make HTTP API calls within Airflow tasks -> Option C

4. Quick Check:

   SimpleHttpOperator = API calls [OK]

Hint: SimpleHttpOperator = HTTP requests in Airflow tasks [OK]

Common Mistakes:

• Confusing it with scheduling or email operators
• Thinking it manages database connections
• Assuming it triggers DAG runs

2. Which of the following is the correct way to create a SimpleHttpOperator task to perform a GET request to the endpoint '/status' using connection ID 'my_api'?

easy

A. SimpleHttpOperator(task_id='check_status', conn_id='my_api', endpoint='/status', method='POST')

B. SimpleHttpOperator(task_id='check_status', connection_id='my_api', url='/status', method='GET')

C. SimpleHttpOperator(task_id='check_status', http_conn_id='my_api', endpoint='/status', http_method='GET')

D. SimpleHttpOperator(task_id='check_status', http_conn_id='my_api', endpoint='/status', method='GET')

Solution

1. Step 1: Check parameter names for SimpleHttpOperator

   The correct parameters include task_id, http_conn_id, endpoint, and method.

2. Step 2: Validate method and connection ID usage

   SimpleHttpOperator(task_id='check_status', http_conn_id='my_api', endpoint='/status', method='GET') uses http_conn_id and method='GET', which are correct. Other options use wrong parameter names or HTTP methods.

3. Final Answer:

   SimpleHttpOperator(task_id='check_status', http_conn_id='my_api', endpoint='/status', method='GET') -> Option D

4. Quick Check:

   Correct params = SimpleHttpOperator(task_id='check_status', http_conn_id='my_api', endpoint='/status', method='GET') [OK]

Hint: Use http_conn_id and method='GET' for GET requests [OK]

Common Mistakes:

• Using wrong parameter names like connection_id or url
• Confusing method with http_method
• Setting wrong HTTP method for GET request

3. Given this Airflow task code snippet using SimpleHttpOperator:

task = SimpleHttpOperator(
    task_id='get_data',
    http_conn_id='api_conn',
    endpoint='data',
    method='GET'
)

What will be the HTTP method used when this task runs?

medium

A. POST

B. GET

C. PUT

D. DELETE

Solution

1. Step 1: Identify the method parameter in the code

   The code sets method='GET', which specifies the HTTP method to use.

2. Step 2: Confirm the HTTP method used

   Since method='GET', the task will perform a GET request.

3. Final Answer:

   GET -> Option B

4. Quick Check:

   method='GET' means GET request [OK]

Hint: Check the method parameter value for HTTP method [OK]

Common Mistakes:

• Assuming default method is POST
• Confusing method with endpoint
• Ignoring the method parameter

4. You wrote this SimpleHttpOperator task but it fails with an error about missing connection:

task = SimpleHttpOperator(
    task_id='call_api',
    http_conn_id='missing_conn',
    endpoint='info',
    method='GET'
)

What is the most likely cause and fix?

medium

A. The connection ID 'missing_conn' is not defined in Airflow connections; add it in UI

B. The endpoint 'info' is invalid; change it to '/'

C. The method should be lowercase 'get' instead of 'GET'

D. SimpleHttpOperator does not support GET method; use POST instead

Solution

1. Step 1: Understand the error about missing connection

   The error indicates Airflow cannot find the connection ID 'missing_conn' in its stored connections.

2. Step 2: Fix by adding the connection in Airflow UI or CLI

   To fix, define 'missing_conn' with the correct API base URL in Airflow connections before running the task.

3. Final Answer:

   The connection ID 'missing_conn' is not defined in Airflow connections; add it in UI -> Option A

4. Quick Check:

   Missing connection ID = add connection [OK]

Hint: Check Airflow connections for your http_conn_id [OK]

Common Mistakes:

• Assuming endpoint name causes connection error
• Changing HTTP method case unnecessarily
• Thinking GET is unsupported by SimpleHttpOperator

5. You want to use SimpleHttpOperator to POST JSON data {"name": "Alice"} to endpoint '/users' using connection 'user_api'. Which is the correct way to do this?

hard

A. SimpleHttpOperator(task_id='post_user', http_conn_id='user_api', endpoint='/users', method='POST', data='{"name": "Alice"}', headers={'Content-Type': 'application/json'})

B. SimpleHttpOperator(task_id='post_user', http_conn_id='user_api', endpoint='/users', method='POST', json={'name': 'Alice'})

C. SimpleHttpOperator(task_id='post_user', http_conn_id='user_api', endpoint='/users', method='POST', data={'name': 'Alice'})

D. SimpleHttpOperator(task_id='post_user', http_conn_id='user_api', endpoint='/users', method='POST', body='{"name": "Alice"}')

Solution

1. Step 1: Identify how to send JSON data with SimpleHttpOperator

   SimpleHttpOperator uses the data parameter for request body and headers to specify content type.

2. Step 2: Confirm correct JSON string and headers usage

   SimpleHttpOperator(task_id='post_user', http_conn_id='user_api', endpoint='/users', method='POST', data='{"name": "Alice"}', headers={'Content-Type': 'application/json'}) sends JSON as a string in data and sets Content-Type header to 'application/json', which is correct.

3. Final Answer:

   SimpleHttpOperator(task_id='post_user', http_conn_id='user_api', endpoint='/users', method='POST', data='{"name": "Alice"}', headers={'Content-Type': 'application/json'}) -> Option A

4. Quick Check:

   Use data string + JSON header for POST JSON [OK]

Hint: Send JSON as string in data with JSON content-type header [OK]

Common Mistakes:

• Using json parameter which SimpleHttpOperator does not support
• Passing data as dict without converting to string
• Using wrong parameter name like body