Workflows for orchestration in GCP - Commands & Configuration

Introduction

When you have several tasks that need to run in order, it can be hard to keep track of them all. Workflows help by organizing these tasks so they run one after another automatically without mistakes.

When you want to automate a series of cloud tasks like calling APIs or moving data.

When you need to run tasks in a specific order without manual steps.

When you want to handle errors in a process and decide what to do next.

When you want to connect different cloud services smoothly.

When you want to save time by automating repeated sequences of actions.

Config File - workflow.yaml

workflow.yaml

main:
  params: [input]
  steps:
  - init:
      assign:
      - message: "Starting workflow"
  - call_api:
      call: http.get
      args:
        url: "https://jsonplaceholder.typicode.com/todos/1"
  - check_response:
      switch:
        - condition: ${call_api.status == 200}
          next: success
        - condition: ${true}
          next: failure
  - success:
      return: ${call_api.body}
  - failure:
      return: "API call failed"

This YAML file defines a simple workflow with these parts:

• main: The starting point with input parameters.
• steps: The list of actions to run in order.
• init: Sets a starting message.
• call_api: Calls an example API to get data.
• check_response: Checks if the API call was successful and decides next step.
• success/failure: Returns the API data or an error message.

Commands

This command uploads and deploys the workflow defined in workflow.yaml to Google Cloud in the us-central1 region.

Terminal

gcloud workflows deploy example-workflow --source=workflow.yaml --location=us-central1

Expected Output

Deploying workflow...done.

Workflow [example-workflow] deployed successfully.

→

--source - Specifies the workflow definition file to deploy.

→

--location - Sets the region where the workflow runs.

This command starts running the deployed workflow in the specified region.

Terminal

gcloud workflows run example-workflow --location=us-central1

Expected Output

Waiting for operation to finish...



+----------------------+----------------------+

|          FIELD       |        VALUE         |

+----------------------+----------------------+

| name                 | projects/project-id/locations/us-central1/workflows/example-workflow/executions/1234567890 |

| startTime            | 2024-06-01T12:00:00Z |

| state                | ACTIVE               |

+----------------------+----------------------+



Execution started.

→

--location - Specifies the region where the workflow is deployed.

This command checks the status and result of the workflow execution using its ID.

Terminal

gcloud workflows executions describe 1234567890 --workflow=example-workflow --location=us-central1

Expected Output

name: projects/project-id/locations/us-central1/workflows/example-workflow/executions/1234567890

startTime: '2024-06-01T12:00:00Z'

endTime: '2024-06-01T12:00:10Z'

state: SUCCEEDED

result: '{"userId":1,"id":1,"title":"delectus aut autem","completed":false}'

→

--workflow - Specifies which workflow's execution to describe.

→

--location - Specifies the region of the workflow.

Key Concept

If you remember nothing else from this pattern, remember: Workflows let you automate and control a series of cloud tasks step-by-step without manual effort.

Common Mistakes

Deploying the workflow without specifying the correct region using --location.

The workflow deploys to a default or wrong region, causing run commands to fail or not find the workflow.

Always use the same --location flag when deploying and running the workflow.

Not checking the execution status after starting the workflow.

You won't know if the workflow succeeded or failed and cannot see the output.

Use 'gcloud workflows executions describe' with the execution ID to check results.

Writing invalid YAML syntax in the workflow file.

The deployment fails because the workflow definition is not valid.

Use a YAML validator and follow the workflow syntax carefully before deploying.

Summary

Create a YAML file defining the workflow steps and logic.

Deploy the workflow to Google Cloud with gcloud workflows deploy.

Run the workflow using gcloud workflows run.

Check the workflow execution status and output with gcloud workflows executions describe.