SimpleHttpOperator for API calls in Apache Airflow - Mini Project: Build & Apply

SimpleHttpOperator for API calls

📖 Scenario: You are working with Apache Airflow to automate API calls. You want to create a simple workflow that makes an HTTP GET request to a public API and processes the response.

🎯 Goal: Build an Airflow DAG that uses SimpleHttpOperator to call a public API endpoint and print the response.

📋 What You'll Learn

Create an Airflow DAG with the ID api_call_dag

Use SimpleHttpOperator to make a GET request to https://jsonplaceholder.typicode.com/todos/1

Set the task ID of the HTTP call to get_todo

Print the response content in the next Python task

💡 Why This Matters

🌍 Real World

Automating API calls is common in DevOps for monitoring, data collection, and triggering workflows based on external services.

💼 Career

Knowing how to use Airflow's SimpleHttpOperator helps you build reliable pipelines that integrate with APIs, a valuable skill for DevOps engineers and data engineers.

1

Create the Airflow DAG and import modules

Write code to import DAG from airflow, SimpleHttpOperator and PythonOperator from airflow.operators, and datetime from datetime. Then create a DAG called api_call_dag with start_date=datetime(2023, 1, 1) and schedule_interval='@daily'.

Apache Airflow

# Import DAG, SimpleHttpOperator, PythonOperator, and datetime
# Create DAG with id 'api_call_dag', start_date 2023-01-01, schedule_interval '@daily'
# Your code here

Hint

Use from airflow import DAG and from airflow.operators.http_operator import SimpleHttpOperator to import the needed classes. Then create the DAG with the exact ID and parameters.

2

Define the SimpleHttpOperator task

Add a SimpleHttpOperator task called get_todo inside the DAG. Set method='GET', http_conn_id='http_default', and endpoint='todos/1'.

Apache Airflow

from airflow import DAG
from airflow.operators.http_operator import SimpleHttpOperator
from airflow.operators.python_operator import PythonOperator
from datetime import datetime

dag = DAG(
    dag_id='api_call_dag',
    start_date=datetime(2023, 1, 1),
    schedule_interval='@daily'
)

# Define SimpleHttpOperator task with task_id 'get_todo', method 'GET', http_conn_id 'http_default', endpoint 'todos/1'
# Your code here

Hint

Use SimpleHttpOperator with the exact parameters inside the DAG.

3

Create a PythonOperator to print the API response

Define a Python function called print_response that takes ti as argument. Inside it, get the response from get_todo task using ti.xcom_pull(task_ids='get_todo') and print it. Then create a PythonOperator task called print_response_task that calls this function and belongs to the DAG.

Apache Airflow

from airflow import DAG
from airflow.operators.http_operator import SimpleHttpOperator
from airflow.operators.python_operator import PythonOperator
from datetime import datetime

dag = DAG(
    dag_id='api_call_dag',
    start_date=datetime(2023, 1, 1),
    schedule_interval='@daily'
)

get_todo = SimpleHttpOperator(
    task_id='get_todo',
    method='GET',
    http_conn_id='http_default',
    endpoint='todos/1',
    dag=dag
)

# Define function print_response(ti) to pull XCom from 'get_todo' and print it
# Create PythonOperator task 'print_response_task' that calls print_response
# Your code here

Hint

Define a function that uses ti.xcom_pull to get the API response and prints it. Then create a PythonOperator task that calls this function.

4

Set task dependencies and print the response

Set the task dependency so that get_todo runs before print_response_task. Then add a print statement outside the DAG to print 'DAG setup complete'.

Apache Airflow

from airflow import DAG
from airflow.operators.http_operator import SimpleHttpOperator
from airflow.operators.python_operator import PythonOperator
from datetime import datetime

dag = DAG(
    dag_id='api_call_dag',
    start_date=datetime(2023, 1, 1),
    schedule_interval='@daily'
)

get_todo = SimpleHttpOperator(
    task_id='get_todo',
    method='GET',
    http_conn_id='http_default',
    endpoint='todos/1',
    dag=dag
)

def print_response(ti):
    response = ti.xcom_pull(task_ids='get_todo')
    print(response)

print_response_task = PythonOperator(
    task_id='print_response_task',
    python_callable=print_response,
    dag=dag
)

# Set get_todo to run before print_response_task
# Print 'DAG setup complete'
# Your code here

Hint

Use the bitshift operator >> to set the task order. Then print the exact text 'DAG setup complete'.

Practice

1. What is the main purpose of the SimpleHttpOperator in Airflow?

easy

A. To schedule Airflow DAGs

B. To manage database connections

C. To make HTTP API calls within Airflow tasks

D. To send emails from Airflow

Solution

1. Step 1: Understand the role of SimpleHttpOperator

   The SimpleHttpOperator is designed to perform HTTP requests like GET or POST inside Airflow tasks.

2. Step 2: Differentiate from other Airflow features

   Scheduling DAGs, managing databases, or sending emails are handled by other operators, not SimpleHttpOperator.

3. Final Answer:

   To make HTTP API calls within Airflow tasks -> Option C

4. Quick Check:

   SimpleHttpOperator = API calls [OK]

Hint: SimpleHttpOperator = HTTP requests in Airflow tasks [OK]

Common Mistakes:

• Confusing it with scheduling or email operators
• Thinking it manages database connections
• Assuming it triggers DAG runs

2. Which of the following is the correct way to create a SimpleHttpOperator task to perform a GET request to the endpoint '/status' using connection ID 'my_api'?

easy

A. SimpleHttpOperator(task_id='check_status', conn_id='my_api', endpoint='/status', method='POST')

B. SimpleHttpOperator(task_id='check_status', connection_id='my_api', url='/status', method='GET')

C. SimpleHttpOperator(task_id='check_status', http_conn_id='my_api', endpoint='/status', http_method='GET')

D. SimpleHttpOperator(task_id='check_status', http_conn_id='my_api', endpoint='/status', method='GET')

Solution

1. Step 1: Check parameter names for SimpleHttpOperator

   The correct parameters include task_id, http_conn_id, endpoint, and method.

2. Step 2: Validate method and connection ID usage

   SimpleHttpOperator(task_id='check_status', http_conn_id='my_api', endpoint='/status', method='GET') uses http_conn_id and method='GET', which are correct. Other options use wrong parameter names or HTTP methods.

3. Final Answer:

   SimpleHttpOperator(task_id='check_status', http_conn_id='my_api', endpoint='/status', method='GET') -> Option D

4. Quick Check:

   Correct params = SimpleHttpOperator(task_id='check_status', http_conn_id='my_api', endpoint='/status', method='GET') [OK]

Hint: Use http_conn_id and method='GET' for GET requests [OK]

Common Mistakes:

• Using wrong parameter names like connection_id or url
• Confusing method with http_method
• Setting wrong HTTP method for GET request

3. Given this Airflow task code snippet using SimpleHttpOperator:

task = SimpleHttpOperator(
    task_id='get_data',
    http_conn_id='api_conn',
    endpoint='data',
    method='GET'
)

What will be the HTTP method used when this task runs?

medium

A. POST

B. GET

C. PUT

D. DELETE

Solution

1. Step 1: Identify the method parameter in the code

   The code sets method='GET', which specifies the HTTP method to use.

2. Step 2: Confirm the HTTP method used

   Since method='GET', the task will perform a GET request.

3. Final Answer:

   GET -> Option B

4. Quick Check:

   method='GET' means GET request [OK]

Hint: Check the method parameter value for HTTP method [OK]

Common Mistakes:

• Assuming default method is POST
• Confusing method with endpoint
• Ignoring the method parameter

4. You wrote this SimpleHttpOperator task but it fails with an error about missing connection:

task = SimpleHttpOperator(
    task_id='call_api',
    http_conn_id='missing_conn',
    endpoint='info',
    method='GET'
)

What is the most likely cause and fix?

medium

A. The connection ID 'missing_conn' is not defined in Airflow connections; add it in UI

B. The endpoint 'info' is invalid; change it to '/'

C. The method should be lowercase 'get' instead of 'GET'

D. SimpleHttpOperator does not support GET method; use POST instead

Solution

1. Step 1: Understand the error about missing connection

   The error indicates Airflow cannot find the connection ID 'missing_conn' in its stored connections.

2. Step 2: Fix by adding the connection in Airflow UI or CLI

   To fix, define 'missing_conn' with the correct API base URL in Airflow connections before running the task.

3. Final Answer:

   The connection ID 'missing_conn' is not defined in Airflow connections; add it in UI -> Option A

4. Quick Check:

   Missing connection ID = add connection [OK]

Hint: Check Airflow connections for your http_conn_id [OK]

Common Mistakes:

• Assuming endpoint name causes connection error
• Changing HTTP method case unnecessarily
• Thinking GET is unsupported by SimpleHttpOperator

5. You want to use SimpleHttpOperator to POST JSON data {"name": "Alice"} to endpoint '/users' using connection 'user_api'. Which is the correct way to do this?

hard

A. SimpleHttpOperator(task_id='post_user', http_conn_id='user_api', endpoint='/users', method='POST', data='{"name": "Alice"}', headers={'Content-Type': 'application/json'})

B. SimpleHttpOperator(task_id='post_user', http_conn_id='user_api', endpoint='/users', method='POST', json={'name': 'Alice'})

C. SimpleHttpOperator(task_id='post_user', http_conn_id='user_api', endpoint='/users', method='POST', data={'name': 'Alice'})

D. SimpleHttpOperator(task_id='post_user', http_conn_id='user_api', endpoint='/users', method='POST', body='{"name": "Alice"}')

Solution

1. Step 1: Identify how to send JSON data with SimpleHttpOperator

   SimpleHttpOperator uses the data parameter for request body and headers to specify content type.

2. Step 2: Confirm correct JSON string and headers usage

   SimpleHttpOperator(task_id='post_user', http_conn_id='user_api', endpoint='/users', method='POST', data='{"name": "Alice"}', headers={'Content-Type': 'application/json'}) sends JSON as a string in data and sets Content-Type header to 'application/json', which is correct.

3. Final Answer:

   SimpleHttpOperator(task_id='post_user', http_conn_id='user_api', endpoint='/users', method='POST', data='{"name": "Alice"}', headers={'Content-Type': 'application/json'}) -> Option A

4. Quick Check:

   Use data string + JSON header for POST JSON [OK]

Hint: Send JSON as string in data with JSON content-type header [OK]

Common Mistakes:

• Using json parameter which SimpleHttpOperator does not support
• Passing data as dict without converting to string
• Using wrong parameter name like body