Cost optimization for cloud resources in Apache Airflow - Commands & Configuration

Introduction

Cloud resources can become expensive if left running without control. Cost optimization helps you use only what you need and stop or reduce resources when they are not required, saving money.

When you want to automatically stop cloud servers during nights or weekends to save costs

When you need to scale down resources during low traffic periods to reduce bills

When you want to track and report cloud spending regularly to avoid surprises

When you want to clean up unused or idle cloud resources automatically

When you want to schedule resource usage only during business hours

Config File - cost_optimization_dag.py

cost_optimization_dag.py

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime, timedelta

def stop_idle_resources():
    print("Stopping idle cloud resources to save cost...")
    # Here you would add code to call cloud APIs to stop resources

def report_costs():
    print("Generating cloud cost report...")
    # Here you would add code to fetch and log cost data

def cleanup_unused_resources():
    print("Cleaning up unused cloud resources...")
    # Here you would add code to delete unused resources

with DAG(
    'cost_optimization',
    default_args={
        'owner': 'airflow',
        'depends_on_past': False,
        'start_date': datetime(2024, 1, 1),
        'retries': 1,
        'retry_delay': timedelta(minutes=5),
    },
    schedule_interval='0 22 * * *',  # Every day at 10 PM
    catchup=False
) as dag:

    stop_resources = PythonOperator(
        task_id='stop_idle_resources',
        python_callable=stop_idle_resources
    )

    generate_report = PythonOperator(
        task_id='report_costs',
        python_callable=report_costs
    )

    cleanup_resources = PythonOperator(
        task_id='cleanup_unused_resources',
        python_callable=cleanup_unused_resources
    )

    stop_resources >> generate_report >> cleanup_resources

This Airflow DAG runs daily at 10 PM to perform cost optimization tasks.

stop_idle_resources: Stops cloud resources that are idle to save money.

report_costs: Generates a report of cloud spending for monitoring.

cleanup_unused_resources: Deletes unused resources to avoid unnecessary charges.

The tasks run in sequence to first stop resources, then report costs, and finally clean up.

Commands

Lists all available DAGs in Airflow to verify the cost optimization DAG is recognized.

Terminal

airflow dags list

Expected Output

dags

cost_optimization

example_bash_operator

example_python_operator

Manually triggers the cost optimization DAG to run the cost-saving tasks immediately.

Terminal

airflow dags trigger cost_optimization

Expected Output

Created <DagRun cost_optimization @ 2024-06-01T12:00:00+00:00: manual__2024-06-01T12:00:00+00:00, externally triggered: True>

Lists all tasks in the cost optimization DAG to see the steps involved.

Terminal

airflow tasks list cost_optimization

Expected Output

stop_idle_resources

report_costs

cleanup_unused_resources

Tests the stop_idle_resources task for the given date to check it runs correctly without executing the full DAG.

Terminal

airflow tasks test cost_optimization stop_idle_resources 2024-06-01

Expected Output

[2024-06-01 12:00:00,000] {taskinstance.py:876} INFO - Executing <Task(PythonOperator): stop_idle_resources> on 2024-06-01

Stopping idle cloud resources to save cost...

[2024-06-01 12:00:00,100] {taskinstance.py:1020} INFO - Marking task as SUCCESS. dag_id=cost_optimization, task_id=stop_idle_resources, execution_date=2024-06-01 00:00:00+00:00, start_date=2024-06-01 12:00:00, end_date=2024-06-01 12:00:00

Key Concept

If you remember nothing else from this pattern, remember: automate stopping and cleaning cloud resources during off-hours to save money without manual effort.

Common Mistakes

Not setting the DAG schedule or start date correctly

The DAG will not run automatically or may run at unexpected times, missing cost-saving windows.

Set a clear start_date and schedule_interval in the DAG to run during off-hours.

Running tasks without proper cloud API permissions

Tasks to stop or delete resources will fail, so no cost savings happen.

Ensure Airflow has credentials with permissions to manage cloud resources.

Not verifying task success after triggering the DAG

You might think resources are stopped but tasks could have failed silently.

Check Airflow logs and task status after runs to confirm success.

Summary

Create an Airflow DAG that runs cost optimization tasks like stopping idle resources and cleaning unused ones.

Use PythonOperator tasks to define each step and schedule the DAG to run during off-hours.

Trigger the DAG manually or let it run on schedule, then verify task success to ensure cost savings.

Practice

1. What is the main goal of cost optimization for cloud resources in Airflow?

easy

A. To increase the number of cloud resources used

B. To run all tasks continuously without stopping

C. To use cloud resources only when needed to save money

D. To avoid monitoring cloud resource usage

Solution

1. Step 1: Understand cost optimization meaning

   Cost optimization means using resources efficiently to reduce expenses.

2. Step 2: Relate to Airflow usage

   Airflow helps automate resource usage only when needed, saving money.

3. Final Answer:

   To use cloud resources only when needed to save money -> Option C

4. Quick Check:

   Cost optimization = Use resources only when needed [OK]

Hint: Cost optimization means saving money by using resources smartly [OK]

Common Mistakes:

• Thinking more resources always mean better performance
• Ignoring the need to stop unused resources
• Believing monitoring is unnecessary

2. Which Airflow feature helps automate starting and stopping cloud resources on a schedule?

easy

A. DAG scheduling

B. Manual triggering only

C. Static resource allocation

D. Ignoring resource states

Solution

1. Step 1: Identify Airflow automation features

   DAG scheduling allows tasks to run automatically at set times.

2. Step 2: Connect scheduling to resource control

   Scheduling can start or stop cloud resources based on time, saving costs.

3. Final Answer:

   DAG scheduling -> Option A

4. Quick Check:

   DAG scheduling = Automate resource start/stop [OK]

Hint: Airflow DAGs run tasks on schedule to control resources [OK]

Common Mistakes:

• Thinking manual triggers automate resource control
• Confusing static allocation with automation
• Ignoring the role of scheduling in cost saving

3. Given this Airflow DAG snippet, what happens to cloud resources?

with DAG('resource_control', schedule_interval='0 8 * * *') as dag:
    start_instance = BashOperator(
        task_id='start_instance',
        bash_command='cloud-cli start instance'
    )
    stop_instance = BashOperator(
        task_id='stop_instance',
        bash_command='cloud-cli stop instance',
        trigger_rule='all_done'
    )

start_instance >> stop_instance

medium

A. Instance stops before it starts

B. Instance starts at 8 AM and stops immediately after

C. Instance starts but never stops

D. Instance stops at 8 AM and starts later

Solution

1. Step 1: Analyze DAG schedule and tasks

   DAG runs daily at 8 AM, starting the instance first.

2. Step 2: Understand task order and trigger rules

   stop_instance runs after start_instance completes, stopping the instance.

3. Final Answer:

   Instance starts at 8 AM and stops immediately after -> Option B

4. Quick Check:

   Task order start >> stop means start then stop [OK]

Hint: Task order with >> shows start then stop sequence [OK]

Common Mistakes:

• Assuming stop runs before start
• Ignoring trigger_rule effects
• Thinking instance stays running all day

4. You wrote this Airflow task to stop a cloud instance:

stop_instance = BashOperator(
    task_id='stop_instance',
    bash_command='cloud-cli stop instance',
    trigger_rule='all_done'
)

Why does this code cause an error?

medium

A. task_id must be 'stop' not 'stop_instance'

B. trigger_rule is not a valid parameter

C. BashOperator cannot run cloud-cli commands

D. Missing comma between bash_command and trigger_rule

Solution

1. Step 1: Check syntax of BashOperator parameters

   Parameters must be separated by commas; here comma is missing after bash_command.

2. Step 2: Identify cause of syntax error

   Missing comma causes Python syntax error, preventing DAG parsing.

3. Final Answer:

   Missing comma between bash_command and trigger_rule -> Option D

4. Quick Check:

   Comma missing = Syntax error [OK]

Hint: Check commas between parameters in Python code [OK]

Common Mistakes:

• Thinking trigger_rule is invalid parameter
• Assuming BashOperator can't run shell commands
• Believing task_id naming causes error

5. You want to optimize cloud costs by running an Airflow DAG that starts instances only on weekdays at 9 AM and stops them at 6 PM. Which schedule_interval and task setup is best?

hard

A. schedule_interval='0 9 * * 1-5' to start, and a separate DAG with '0 18 * * 1-5' to stop

B. schedule_interval='@daily' for both start and stop tasks in one DAG

C. schedule_interval='0 9 * * *' to start and stop in same DAG immediately

D. No schedule_interval, run tasks manually when needed

Solution

1. Step 1: Choose schedule for weekdays only

   '0 9 * * 1-5' runs at 9 AM Monday to Friday, matching requirement.

2. Step 2: Separate start and stop times

   Stopping at 6 PM requires a different schedule '0 18 * * 1-5', best in separate DAG for clarity.

3. Final Answer:

   schedule_interval='0 9 * * 1-5' to start, and a separate DAG with '0 18 * * 1-5' to stop -> Option A

4. Quick Check:

   Separate schedules for start and stop on weekdays [OK]

Hint: Use cron for weekdays and separate DAGs for start/stop [OK]

Common Mistakes:

• Using '@daily' runs every day, not weekdays only
• Starting and stopping immediately in one DAG
• Relying on manual runs instead of automation