SimpleHttpOperator for API calls in Apache Airflow - Commands & Configuration

Introduction

Sometimes you need to get data from or send data to a web service automatically. SimpleHttpOperator in Airflow helps you do this by making HTTP requests as part of your workflow.

When you want to fetch data from a public API every hour to update your database.

When you need to send a notification to a web service after a task completes.

When you want to check the status of a web service regularly and act on the response.

When you want to trigger an external system by calling its API as part of your workflow.

When you want to automate data collection from a REST API without writing complex code.

Config File - example_dag.py

example_dag.py

from airflow import DAG
from airflow.providers.http.operators.http import SimpleHttpOperator
from airflow.utils.dates import days_ago

with DAG(
    dag_id='http_api_call_example',
    schedule_interval='@hourly',
    start_date=days_ago(1),
    catchup=False
) as dag:

    call_api = SimpleHttpOperator(
        task_id='call_example_api',
        http_conn_id='example_api',
        endpoint='data',
        method='GET',
        response_check=lambda response: response.status_code == 200,
        log_response=True
    )

This DAG defines a workflow that runs every hour starting from yesterday.

SimpleHttpOperator is used to make a GET request to the 'data' endpoint of the HTTP connection named 'example_api'.

The response_check ensures the response status code is 200 (OK), and log_response will print the response content in the logs.

Commands

This command creates an Airflow connection named 'example_api' pointing to the base URL of the API you want to call. It lets SimpleHttpOperator know where to send requests.

Terminal

airflow connections add example_api --conn-type http --conn-host https://api.example.com

Expected Output

Connection `example_api` added successfully

→

--conn-type - Specifies the connection type, here 'http' for HTTP API

→

--conn-host - Sets the base URL for the API

This command lists all available DAGs to confirm your DAG file is recognized by Airflow.

Terminal

airflow dags list

Expected Output

dag_id          | owner | paused | last_run

http_api_call_example | airflow | False  | None

This command manually triggers the DAG to run immediately, so you can test the API call task.

Terminal

airflow dags trigger http_api_call_example

Expected Output

Created <DagRun http_api_call_example @ 2024-06-01T12:00:00+00:00: manual__2024-06-01T12:00:00+00:00, externally triggered: True>

This command shows the logs of the API call task to verify the request was sent and the response received.

Terminal

airflow tasks logs http_api_call_example call_example_api

Expected Output

[2024-06-01 12:00:01,000] {http.py:123} INFO - Making GET request to https://api.example.com/data

[2024-06-01 12:00:02,000] {http.py:145} INFO - Response status: 200

[2024-06-01 12:00:02,000] {http.py:146} INFO - Response content: {"key": "value"}

[2024-06-01 12:00:02,000] {taskinstance.py:123} INFO - Task succeeded

Key Concept

If you remember nothing else from this pattern, remember: SimpleHttpOperator lets you easily automate HTTP API calls inside Airflow workflows without extra coding.

Common Mistakes

Not creating the HTTP connection in Airflow before using SimpleHttpOperator

The operator needs a connection ID to know the API base URL; without it, the task will fail.

Use 'airflow connections add' to create the HTTP connection with the correct host before running the DAG.

Using the wrong HTTP method or endpoint in the operator

The API call will fail or return errors if the method or endpoint is incorrect.

Check the API documentation and set the 'method' and 'endpoint' parameters correctly in SimpleHttpOperator.

Not checking the response status or content

You might miss failed API calls or errors, causing silent failures in your workflow.

Use the 'response_check' parameter to verify the response status and handle errors properly.

Summary

Create an HTTP connection in Airflow with the API base URL using 'airflow connections add'.

Define a DAG with SimpleHttpOperator to make the API call using the connection and endpoint.

Trigger the DAG and check task logs to verify the API call and response.

Practice

1. What is the main purpose of the SimpleHttpOperator in Airflow?

easy

A. To schedule Airflow DAGs

B. To manage database connections

C. To make HTTP API calls within Airflow tasks

D. To send emails from Airflow

Solution

1. Step 1: Understand the role of SimpleHttpOperator

   The SimpleHttpOperator is designed to perform HTTP requests like GET or POST inside Airflow tasks.

2. Step 2: Differentiate from other Airflow features

   Scheduling DAGs, managing databases, or sending emails are handled by other operators, not SimpleHttpOperator.

3. Final Answer:

   To make HTTP API calls within Airflow tasks -> Option C

4. Quick Check:

   SimpleHttpOperator = API calls [OK]

Hint: SimpleHttpOperator = HTTP requests in Airflow tasks [OK]

Common Mistakes:

• Confusing it with scheduling or email operators
• Thinking it manages database connections
• Assuming it triggers DAG runs

2. Which of the following is the correct way to create a SimpleHttpOperator task to perform a GET request to the endpoint '/status' using connection ID 'my_api'?

easy

A. SimpleHttpOperator(task_id='check_status', conn_id='my_api', endpoint='/status', method='POST')

B. SimpleHttpOperator(task_id='check_status', connection_id='my_api', url='/status', method='GET')

C. SimpleHttpOperator(task_id='check_status', http_conn_id='my_api', endpoint='/status', http_method='GET')

D. SimpleHttpOperator(task_id='check_status', http_conn_id='my_api', endpoint='/status', method='GET')

Solution

1. Step 1: Check parameter names for SimpleHttpOperator

   The correct parameters include task_id, http_conn_id, endpoint, and method.

2. Step 2: Validate method and connection ID usage

   SimpleHttpOperator(task_id='check_status', http_conn_id='my_api', endpoint='/status', method='GET') uses http_conn_id and method='GET', which are correct. Other options use wrong parameter names or HTTP methods.

3. Final Answer:

   SimpleHttpOperator(task_id='check_status', http_conn_id='my_api', endpoint='/status', method='GET') -> Option D

4. Quick Check:

   Correct params = SimpleHttpOperator(task_id='check_status', http_conn_id='my_api', endpoint='/status', method='GET') [OK]

Hint: Use http_conn_id and method='GET' for GET requests [OK]

Common Mistakes:

• Using wrong parameter names like connection_id or url
• Confusing method with http_method
• Setting wrong HTTP method for GET request

3. Given this Airflow task code snippet using SimpleHttpOperator:

task = SimpleHttpOperator(
    task_id='get_data',
    http_conn_id='api_conn',
    endpoint='data',
    method='GET'
)

What will be the HTTP method used when this task runs?

medium

A. POST

B. GET

C. PUT

D. DELETE

Solution

1. Step 1: Identify the method parameter in the code

   The code sets method='GET', which specifies the HTTP method to use.

2. Step 2: Confirm the HTTP method used

   Since method='GET', the task will perform a GET request.

3. Final Answer:

   GET -> Option B

4. Quick Check:

   method='GET' means GET request [OK]

Hint: Check the method parameter value for HTTP method [OK]

Common Mistakes:

• Assuming default method is POST
• Confusing method with endpoint
• Ignoring the method parameter

4. You wrote this SimpleHttpOperator task but it fails with an error about missing connection:

task = SimpleHttpOperator(
    task_id='call_api',
    http_conn_id='missing_conn',
    endpoint='info',
    method='GET'
)

What is the most likely cause and fix?

medium

A. The connection ID 'missing_conn' is not defined in Airflow connections; add it in UI

B. The endpoint 'info' is invalid; change it to '/'

C. The method should be lowercase 'get' instead of 'GET'

D. SimpleHttpOperator does not support GET method; use POST instead

Solution

1. Step 1: Understand the error about missing connection

   The error indicates Airflow cannot find the connection ID 'missing_conn' in its stored connections.

2. Step 2: Fix by adding the connection in Airflow UI or CLI

   To fix, define 'missing_conn' with the correct API base URL in Airflow connections before running the task.

3. Final Answer:

   The connection ID 'missing_conn' is not defined in Airflow connections; add it in UI -> Option A

4. Quick Check:

   Missing connection ID = add connection [OK]

Hint: Check Airflow connections for your http_conn_id [OK]

Common Mistakes:

• Assuming endpoint name causes connection error
• Changing HTTP method case unnecessarily
• Thinking GET is unsupported by SimpleHttpOperator

5. You want to use SimpleHttpOperator to POST JSON data {"name": "Alice"} to endpoint '/users' using connection 'user_api'. Which is the correct way to do this?

hard

A. SimpleHttpOperator(task_id='post_user', http_conn_id='user_api', endpoint='/users', method='POST', data='{"name": "Alice"}', headers={'Content-Type': 'application/json'})

B. SimpleHttpOperator(task_id='post_user', http_conn_id='user_api', endpoint='/users', method='POST', json={'name': 'Alice'})

C. SimpleHttpOperator(task_id='post_user', http_conn_id='user_api', endpoint='/users', method='POST', data={'name': 'Alice'})

D. SimpleHttpOperator(task_id='post_user', http_conn_id='user_api', endpoint='/users', method='POST', body='{"name": "Alice"}')

Solution

1. Step 1: Identify how to send JSON data with SimpleHttpOperator

   SimpleHttpOperator uses the data parameter for request body and headers to specify content type.

2. Step 2: Confirm correct JSON string and headers usage

   SimpleHttpOperator(task_id='post_user', http_conn_id='user_api', endpoint='/users', method='POST', data='{"name": "Alice"}', headers={'Content-Type': 'application/json'}) sends JSON as a string in data and sets Content-Type header to 'application/json', which is correct.

3. Final Answer:

   SimpleHttpOperator(task_id='post_user', http_conn_id='user_api', endpoint='/users', method='POST', data='{"name": "Alice"}', headers={'Content-Type': 'application/json'}) -> Option A

4. Quick Check:

   Use data string + JSON header for POST JSON [OK]

Hint: Send JSON as string in data with JSON content-type header [OK]

Common Mistakes:

• Using json parameter which SimpleHttpOperator does not support
• Passing data as dict without converting to string
• Using wrong parameter name like body