SimpleHttpOperator for API calls in Apache Airflow - Mini Project: Build & Apply

SimpleHttpOperator for API calls

📖 Scenario: You are working with Apache Airflow to automate API calls. You want to create a simple workflow that makes an HTTP GET request to a public API and processes the response.

🎯 Goal: Build an Airflow DAG that uses SimpleHttpOperator to call a public API endpoint and print the response.

📋 What You'll Learn

Create an Airflow DAG with the ID api_call_dag

Use SimpleHttpOperator to make a GET request to https://jsonplaceholder.typicode.com/todos/1

Set the task ID of the HTTP call to get_todo

Print the response content in the next Python task

💡 Why This Matters

🌍 Real World

Automating API calls is common in DevOps for monitoring, data collection, and triggering workflows based on external services.

💼 Career

Knowing how to use Airflow's SimpleHttpOperator helps you build reliable pipelines that integrate with APIs, a valuable skill for DevOps engineers and data engineers.

1

Create the Airflow DAG and import modules

Write code to import DAG from airflow, SimpleHttpOperator and PythonOperator from airflow.operators, and datetime from datetime. Then create a DAG called api_call_dag with start_date=datetime(2023, 1, 1) and schedule_interval='@daily'.

Apache Airflow

# Import DAG, SimpleHttpOperator, PythonOperator, and datetime
# Create DAG with id 'api_call_dag', start_date 2023-01-01, schedule_interval '@daily'
# Your code here

Need a hint?

Use from airflow import DAG and from airflow.operators.http_operator import SimpleHttpOperator to import the needed classes. Then create the DAG with the exact ID and parameters.

2

Define the SimpleHttpOperator task

Add a SimpleHttpOperator task called get_todo inside the DAG. Set method='GET', http_conn_id='http_default', and endpoint='todos/1'.

Apache Airflow

from airflow import DAG
from airflow.operators.http_operator import SimpleHttpOperator
from airflow.operators.python_operator import PythonOperator
from datetime import datetime

dag = DAG(
    dag_id='api_call_dag',
    start_date=datetime(2023, 1, 1),
    schedule_interval='@daily'
)

# Define SimpleHttpOperator task with task_id 'get_todo', method 'GET', http_conn_id 'http_default', endpoint 'todos/1'
# Your code here

Need a hint?

Use SimpleHttpOperator with the exact parameters inside the DAG.

3

Create a PythonOperator to print the API response

Define a Python function called print_response that takes ti as argument. Inside it, get the response from get_todo task using ti.xcom_pull(task_ids='get_todo') and print it. Then create a PythonOperator task called print_response_task that calls this function and belongs to the DAG.

Apache Airflow

from airflow import DAG
from airflow.operators.http_operator import SimpleHttpOperator
from airflow.operators.python_operator import PythonOperator
from datetime import datetime

dag = DAG(
    dag_id='api_call_dag',
    start_date=datetime(2023, 1, 1),
    schedule_interval='@daily'
)

get_todo = SimpleHttpOperator(
    task_id='get_todo',
    method='GET',
    http_conn_id='http_default',
    endpoint='todos/1',
    dag=dag
)

# Define function print_response(ti) to pull XCom from 'get_todo' and print it
# Create PythonOperator task 'print_response_task' that calls print_response
# Your code here

Need a hint?

Define a function that uses ti.xcom_pull to get the API response and prints it. Then create a PythonOperator task that calls this function.

4

Set task dependencies and print the response

Set the task dependency so that get_todo runs before print_response_task. Then add a print statement outside the DAG to print 'DAG setup complete'.

Apache Airflow

from airflow import DAG
from airflow.operators.http_operator import SimpleHttpOperator
from airflow.operators.python_operator import PythonOperator
from datetime import datetime

dag = DAG(
    dag_id='api_call_dag',
    start_date=datetime(2023, 1, 1),
    schedule_interval='@daily'
)

get_todo = SimpleHttpOperator(
    task_id='get_todo',
    method='GET',
    http_conn_id='http_default',
    endpoint='todos/1',
    dag=dag
)

def print_response(ti):
    response = ti.xcom_pull(task_ids='get_todo')
    print(response)

print_response_task = PythonOperator(
    task_id='print_response_task',
    python_callable=print_response,
    dag=dag
)

# Set get_todo to run before print_response_task
# Print 'DAG setup complete'
# Your code here

Need a hint?

Use the bitshift operator >> to set the task order. Then print the exact text 'DAG setup complete'.