Cloud Tasks for async processing in GCP - Commands & Configuration

Introduction

Sometimes your app needs to do work later or in the background so it can respond quickly. Cloud Tasks helps by letting you send jobs to a queue that runs them one by one without slowing your app down.

When you want to send emails after a user signs up without making them wait.

When your app needs to process images or videos but not immediately.

When you want to retry failed tasks automatically without losing data.

When you want to control how many tasks run at the same time to avoid overload.

When you want to separate slow work from fast user requests for better performance.

Config File - cloudtasks.yaml

cloudtasks.yaml

queue:
  name: my-queue
  rateLimits:
    maxDispatchesPerSecond: 5
  retryConfig:
    maxAttempts: 5
    maxRetryDuration: 3600s
    minBackoff: 10s
    maxBackoff: 300s
  appEngineRoutingOverride:
    service: default
    version: v1
    instance: ""

This file defines a Cloud Tasks queue named my-queue. The rateLimits section controls how fast tasks are sent to your app, here 5 tasks per second. The retryConfig sets how many times and how often failed tasks retry. The appEngineRoutingOverride tells Cloud Tasks which App Engine service and version to send tasks to.

Commands

This command creates a Cloud Tasks queue named 'my-queue' with limits on how fast tasks dispatch and how retries happen.

Terminal

gcloud tasks queues create my-queue --max-dispatches-per-second=5 --max-attempts=5 --min-backoff=10s --max-backoff=300s

Expected Output

Created queue [projects/your-project/locations/us-central1/queues/my-queue].

→

--max-dispatches-per-second - Limits how many tasks run each second to avoid overload.

→

--max-attempts - Sets how many times a failed task retries before giving up.

This command adds a task to 'my-queue' that calls the '/worker' URL on your app with a POST request and sends JSON data.

Terminal

gcloud tasks create-app-engine-task my-queue --url=/worker --method=POST --body-content="{\"task\":\"process\"}"

Expected Output

Created task [projects/your-project/locations/us-central1/queues/my-queue/tasks/1234567890].

→

--url - Specifies the app URL the task will call.

→

--method - Sets the HTTP method used by the task.

This command lists all tasks currently in the 'my-queue' queue so you can see what is waiting or running.

Terminal

gcloud tasks list --queue=my-queue

Expected Output

NAME                                    SCHEDULE_TIME          STATUS

projects/your-project/locations/us-central1/queues/my-queue/tasks/1234567890  2024-06-01T12:00:00Z  SCHEDULED

→

--queue - Specifies which queue's tasks to list.

This command deletes a specific task from the queue if you want to cancel it before it runs.

Terminal

gcloud tasks delete projects/your-project/locations/us-central1/queues/my-queue/tasks/1234567890

Expected Output

Deleted task [projects/your-project/locations/us-central1/queues/my-queue/tasks/1234567890].

Key Concept

If you remember nothing else from this pattern, remember: Cloud Tasks lets your app do slow work later by sending jobs to a queue that runs them safely and reliably.

Common Mistakes

Creating tasks without specifying the correct URL or method.

The task will fail because it cannot reach the right endpoint or uses the wrong HTTP method.

Always specify the exact URL and HTTP method your app expects when creating tasks.

Not setting retry limits, causing tasks to retry forever.

This can overload your app and waste resources if tasks keep failing without limit.

Set sensible retry limits and backoff times to control retries.

Ignoring task status and not checking if tasks are stuck or failed.

You might miss problems and your app won't process all work as expected.

Regularly list and monitor tasks to catch and fix issues early.

Summary

Create a Cloud Tasks queue with limits on dispatch rate and retries.

Add tasks to the queue specifying the app URL and HTTP method.

List tasks to monitor their status and delete tasks if needed.