Cost optimization for cloud resources in Apache Airflow - Practice Problems & Coding Challenges

Challenge - 5 Problems

🧠 Conceptual

intermediate

Understanding Cost Optimization Strategies in Airflow

Which of the following is the most effective way to reduce cloud costs when running Airflow workflows?

A Use Airflow's scheduling to run tasks only during off-peak hours to leverage cheaper cloud rates.

B Disable task retries to save compute resources even if tasks fail frequently.

C Always keep all Airflow workers running 24/7 to avoid startup delays.

D Increase the number of parallel tasks to finish workflows faster regardless of resource usage.

💡 Hint

Think about when cloud providers offer lower prices and how scheduling can help.

Scheduling tasks during off-peak hours can leverage cheaper cloud rates, reducing costs without sacrificing reliability. Increasing parallelism or keeping workers always on can increase costs. Disabling retries may cause failures and rework.

💻 Command Output

intermediate

Output of Airflow CLI Command for Worker Status

What is the output of the following command when no Airflow workers are currently running?

airflow celery status

A Error: command not found

B

celery@worker1: OK
celery@worker2: OK

C

celery@worker1: DOWN
celery@worker2: DOWN

D No nodes currently registered.

💡 Hint

Think about what the status command shows when no workers are connected.

The 'airflow celery status' command shows registered workers. If none are running, it outputs 'No nodes currently registered.'

❓ Configuration

advanced

Configuring Airflow to Scale Down Workers Automatically

Which Airflow configuration snippet correctly enables automatic scaling down of Celery workers to save cloud costs?

A

[celery]
worker_autoscale = 10,3

B

[scheduler]
worker_autoscale = 3,10

C

[celery]
worker_autoscale = 3,10

D

[core]
worker_autoscale = 10,3

💡 Hint

Check the correct section and the order of max and min workers in worker_autoscale.

The 'worker_autoscale' setting under the [celery] section uses the format 'min_workers,max_workers'. 'worker_autoscale = 3,10' means scale between 3 and 10 workers automatically.

❓ Troubleshoot

advanced

Troubleshooting Unexpected High Cloud Costs from Airflow

You notice your cloud bill increased sharply after deploying Airflow. Which of the following is the most likely cause?

A Airflow DAGs are scheduled too infrequently, causing idle resources.

B Airflow workers are not shutting down after task completion, running 24/7.

C Airflow is using the LocalExecutor instead of CeleryExecutor.

D Airflow logs are stored locally instead of cloud storage.

💡 Hint

Consider what causes continuous resource usage and costs.

If Airflow workers remain running all the time, they consume cloud resources continuously, increasing costs. Infrequent scheduling reduces costs. LocalExecutor runs tasks locally, not necessarily increasing cloud costs. Logs stored locally do not affect cloud costs.

🔀 Workflow

expert

Optimizing Airflow DAG for Cost Efficiency

You want to optimize an Airflow DAG to minimize cloud costs by reducing unnecessary task runs. Which workflow change achieves this best?

A Use task triggers to run downstream tasks only if upstream tasks succeed.

B Set all tasks to run daily regardless of upstream task success.

C Increase retries for all tasks to ensure completion even if costly.

D Disable task dependencies to allow parallel execution of all tasks.

💡 Hint

Think about avoiding running tasks that are not needed.

Using task triggers to run downstream tasks only if upstream tasks succeed prevents unnecessary task runs, saving cloud costs. Running all tasks daily or disabling dependencies can increase costs. Increasing retries may increase costs due to repeated runs.

Practice

1. What is the main goal of cost optimization for cloud resources in Airflow?

easy

A. To increase the number of cloud resources used

B. To run all tasks continuously without stopping

C. To use cloud resources only when needed to save money

D. To avoid monitoring cloud resource usage

Solution

1. Step 1: Understand cost optimization meaning

   Cost optimization means using resources efficiently to reduce expenses.

2. Step 2: Relate to Airflow usage

   Airflow helps automate resource usage only when needed, saving money.

3. Final Answer:

   To use cloud resources only when needed to save money -> Option C

4. Quick Check:

   Cost optimization = Use resources only when needed [OK]

Hint: Cost optimization means saving money by using resources smartly [OK]

Common Mistakes:

• Thinking more resources always mean better performance
• Ignoring the need to stop unused resources
• Believing monitoring is unnecessary

2. Which Airflow feature helps automate starting and stopping cloud resources on a schedule?

easy

A. DAG scheduling

B. Manual triggering only

C. Static resource allocation

D. Ignoring resource states

Solution

1. Step 1: Identify Airflow automation features

   DAG scheduling allows tasks to run automatically at set times.

2. Step 2: Connect scheduling to resource control

   Scheduling can start or stop cloud resources based on time, saving costs.

3. Final Answer:

   DAG scheduling -> Option A

4. Quick Check:

   DAG scheduling = Automate resource start/stop [OK]

Hint: Airflow DAGs run tasks on schedule to control resources [OK]

Common Mistakes:

• Thinking manual triggers automate resource control
• Confusing static allocation with automation
• Ignoring the role of scheduling in cost saving

3. Given this Airflow DAG snippet, what happens to cloud resources?

with DAG('resource_control', schedule_interval='0 8 * * *') as dag:
    start_instance = BashOperator(
        task_id='start_instance',
        bash_command='cloud-cli start instance'
    )
    stop_instance = BashOperator(
        task_id='stop_instance',
        bash_command='cloud-cli stop instance',
        trigger_rule='all_done'
    )

start_instance >> stop_instance

medium

A. Instance stops before it starts

B. Instance starts at 8 AM and stops immediately after

C. Instance starts but never stops

D. Instance stops at 8 AM and starts later

Solution

1. Step 1: Analyze DAG schedule and tasks

   DAG runs daily at 8 AM, starting the instance first.

2. Step 2: Understand task order and trigger rules

   stop_instance runs after start_instance completes, stopping the instance.

3. Final Answer:

   Instance starts at 8 AM and stops immediately after -> Option B

4. Quick Check:

   Task order start >> stop means start then stop [OK]

Hint: Task order with >> shows start then stop sequence [OK]

Common Mistakes:

• Assuming stop runs before start
• Ignoring trigger_rule effects
• Thinking instance stays running all day

4. You wrote this Airflow task to stop a cloud instance:

stop_instance = BashOperator(
    task_id='stop_instance',
    bash_command='cloud-cli stop instance',
    trigger_rule='all_done'
)

Why does this code cause an error?

medium

A. task_id must be 'stop' not 'stop_instance'

B. trigger_rule is not a valid parameter

C. BashOperator cannot run cloud-cli commands

D. Missing comma between bash_command and trigger_rule

Solution

1. Step 1: Check syntax of BashOperator parameters

   Parameters must be separated by commas; here comma is missing after bash_command.

2. Step 2: Identify cause of syntax error

   Missing comma causes Python syntax error, preventing DAG parsing.

3. Final Answer:

   Missing comma between bash_command and trigger_rule -> Option D

4. Quick Check:

   Comma missing = Syntax error [OK]

Hint: Check commas between parameters in Python code [OK]

Common Mistakes:

• Thinking trigger_rule is invalid parameter
• Assuming BashOperator can't run shell commands
• Believing task_id naming causes error

5. You want to optimize cloud costs by running an Airflow DAG that starts instances only on weekdays at 9 AM and stops them at 6 PM. Which schedule_interval and task setup is best?

hard

A. schedule_interval='0 9 * * 1-5' to start, and a separate DAG with '0 18 * * 1-5' to stop

B. schedule_interval='@daily' for both start and stop tasks in one DAG

C. schedule_interval='0 9 * * *' to start and stop in same DAG immediately

D. No schedule_interval, run tasks manually when needed

Solution

1. Step 1: Choose schedule for weekdays only

   '0 9 * * 1-5' runs at 9 AM Monday to Friday, matching requirement.

2. Step 2: Separate start and stop times

   Stopping at 6 PM requires a different schedule '0 18 * * 1-5', best in separate DAG for clarity.

3. Final Answer:

   schedule_interval='0 9 * * 1-5' to start, and a separate DAG with '0 18 * * 1-5' to stop -> Option A

4. Quick Check:

   Separate schedules for start and stop on weekdays [OK]

Hint: Use cron for weekdays and separate DAGs for start/stop [OK]

Common Mistakes:

• Using '@daily' runs every day, not weekdays only
• Starting and stopping immediately in one DAG
• Relying on manual runs instead of automation