Cost optimization for cloud resources in Apache Airflow - Step-by-Step Execution

Process Flow - Cost optimization for cloud resources

Identify Cloud Resources

↓

Analyze Usage & Costs

↓

Find Idle/Underused Resources

↓

Apply Optimization Actions

↓

Monitor & Repeat

This flow shows how to optimize cloud costs by identifying resources, analyzing usage, finding waste, applying fixes, and monitoring continuously.

Execution Sample

Apache Airflow

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

def optimize_costs():
    print('Checking idle resources and shutting down...')

with DAG('cost_optimization', start_date=datetime(2024,1,1), schedule_interval='@daily') as dag:
    task = PythonOperator(task_id='optimize', python_callable=optimize_costs)

This Airflow DAG runs daily to check and optimize cloud resource costs by shutting down idle resources.

Process Table

Step	Action	Evaluation	Result
1	DAG starts at scheduled time	Current time matches schedule	DAG triggered
2	Run optimize_costs function	Check cloud resources usage	Idle resources found: 2 VMs
3	Shut down idle VMs	Send shutdown commands	2 VMs stopped
4	Log optimization result	Print confirmation	Output: 'Checking idle resources and shutting down...'
5	DAG run completes	All tasks done	DAG success

💡 All idle resources processed, DAG run ends successfully

Status Tracker

Variable	Start	After Step 2	After Step 3	Final
idle_resources	None	2 VMs identified	0 (after shutdown)	0

Key Moments - 2 Insights

Why do we check for idle resources before shutting down?

Because shutting down active resources would disrupt services. The execution_table step 2 shows we identify idle resources first to avoid this.

What happens if no idle resources are found?

The shutdown step would skip or do nothing. The DAG would still complete successfully as shown in the exit_note.

Visual Quiz - 3 Questions

Look at the execution_table, what is the output printed during the DAG run?

A 'Starting cloud resource analysis'

B 'No idle resources found'

C 'Checking idle resources and shutting down...'

D 'DAG failed due to error'

💡 Hint

Check execution_table row 4 for the printed output

At which step are idle resources identified?

A Step 2

B Step 1

C Step 3

D Step 5

💡 Hint

Look at execution_table row 2 where idle resources are found

If no idle VMs were found, what would be the value of 'idle_resources' after step 2 in variable_tracker?

A 2 VMs identified

B 0

C None

D Error

💡 Hint

Variable_tracker shows 'idle_resources' after step 2; if none found, it would be zero

Concept Snapshot

Cost optimization for cloud resources in Airflow:
- Schedule DAG to run regularly
- Identify idle or underused resources
- Shut down or resize these resources
- Log actions for audit
- Monitor continuously for savings

Full Transcript

This visual execution shows how an Airflow DAG can help optimize cloud costs. The DAG runs daily, checks for idle cloud resources like virtual machines, and shuts them down to save money. We track variables like idle_resources to see how many are found and stopped. Key moments include why we identify idle resources first to avoid stopping active ones, and what happens if no idle resources exist. The quiz tests understanding of the steps and variable changes. This approach helps keep cloud spending efficient by automating cleanup tasks.

Practice

1. What is the main goal of cost optimization for cloud resources in Airflow?

easy

A. To increase the number of cloud resources used

B. To run all tasks continuously without stopping

C. To use cloud resources only when needed to save money

D. To avoid monitoring cloud resource usage

Solution

1. Step 1: Understand cost optimization meaning

   Cost optimization means using resources efficiently to reduce expenses.

2. Step 2: Relate to Airflow usage

   Airflow helps automate resource usage only when needed, saving money.

3. Final Answer:

   To use cloud resources only when needed to save money -> Option C

4. Quick Check:

   Cost optimization = Use resources only when needed [OK]

Hint: Cost optimization means saving money by using resources smartly [OK]

Common Mistakes:

• Thinking more resources always mean better performance
• Ignoring the need to stop unused resources
• Believing monitoring is unnecessary

2. Which Airflow feature helps automate starting and stopping cloud resources on a schedule?

easy

A. DAG scheduling

B. Manual triggering only

C. Static resource allocation

D. Ignoring resource states

Solution

1. Step 1: Identify Airflow automation features

   DAG scheduling allows tasks to run automatically at set times.

2. Step 2: Connect scheduling to resource control

   Scheduling can start or stop cloud resources based on time, saving costs.

3. Final Answer:

   DAG scheduling -> Option A

4. Quick Check:

   DAG scheduling = Automate resource start/stop [OK]

Hint: Airflow DAGs run tasks on schedule to control resources [OK]

Common Mistakes:

• Thinking manual triggers automate resource control
• Confusing static allocation with automation
• Ignoring the role of scheduling in cost saving

3. Given this Airflow DAG snippet, what happens to cloud resources?

with DAG('resource_control', schedule_interval='0 8 * * *') as dag:
    start_instance = BashOperator(
        task_id='start_instance',
        bash_command='cloud-cli start instance'
    )
    stop_instance = BashOperator(
        task_id='stop_instance',
        bash_command='cloud-cli stop instance',
        trigger_rule='all_done'
    )

start_instance >> stop_instance

medium

A. Instance stops before it starts

B. Instance starts at 8 AM and stops immediately after

C. Instance starts but never stops

D. Instance stops at 8 AM and starts later

Solution

1. Step 1: Analyze DAG schedule and tasks

   DAG runs daily at 8 AM, starting the instance first.

2. Step 2: Understand task order and trigger rules

   stop_instance runs after start_instance completes, stopping the instance.

3. Final Answer:

   Instance starts at 8 AM and stops immediately after -> Option B

4. Quick Check:

   Task order start >> stop means start then stop [OK]

Hint: Task order with >> shows start then stop sequence [OK]

Common Mistakes:

• Assuming stop runs before start
• Ignoring trigger_rule effects
• Thinking instance stays running all day

4. You wrote this Airflow task to stop a cloud instance:

stop_instance = BashOperator(
    task_id='stop_instance',
    bash_command='cloud-cli stop instance',
    trigger_rule='all_done'
)

Why does this code cause an error?

medium

A. task_id must be 'stop' not 'stop_instance'

B. trigger_rule is not a valid parameter

C. BashOperator cannot run cloud-cli commands

D. Missing comma between bash_command and trigger_rule

Solution

1. Step 1: Check syntax of BashOperator parameters

   Parameters must be separated by commas; here comma is missing after bash_command.

2. Step 2: Identify cause of syntax error

   Missing comma causes Python syntax error, preventing DAG parsing.

3. Final Answer:

   Missing comma between bash_command and trigger_rule -> Option D

4. Quick Check:

   Comma missing = Syntax error [OK]

Hint: Check commas between parameters in Python code [OK]

Common Mistakes:

• Thinking trigger_rule is invalid parameter
• Assuming BashOperator can't run shell commands
• Believing task_id naming causes error

5. You want to optimize cloud costs by running an Airflow DAG that starts instances only on weekdays at 9 AM and stops them at 6 PM. Which schedule_interval and task setup is best?

hard

A. schedule_interval='0 9 * * 1-5' to start, and a separate DAG with '0 18 * * 1-5' to stop

B. schedule_interval='@daily' for both start and stop tasks in one DAG

C. schedule_interval='0 9 * * *' to start and stop in same DAG immediately

D. No schedule_interval, run tasks manually when needed

Solution

1. Step 1: Choose schedule for weekdays only

   '0 9 * * 1-5' runs at 9 AM Monday to Friday, matching requirement.

2. Step 2: Separate start and stop times

   Stopping at 6 PM requires a different schedule '0 18 * * 1-5', best in separate DAG for clarity.

3. Final Answer:

   schedule_interval='0 9 * * 1-5' to start, and a separate DAG with '0 18 * * 1-5' to stop -> Option A

4. Quick Check:

   Separate schedules for start and stop on weekdays [OK]

Hint: Use cron for weekdays and separate DAGs for start/stop [OK]

Common Mistakes:

• Using '@daily' runs every day, not weekdays only
• Starting and stopping immediately in one DAG
• Relying on manual runs instead of automation