Cloud Run jobs for batch work in GCP - Commands & Configuration

Introduction

Sometimes you need to run tasks that start, do some work, and then stop. Cloud Run jobs let you run these batch tasks easily without managing servers.

When you want to process a list of files one time and then stop.

When you need to run a script that cleans up old data once a day.

When you want to run a report that gathers information and then finishes.

When you want to run a task that sends emails to users in batches.

When you want to run a job that imports data from one place to another without keeping a server running.

Config File - job.yaml

job.yaml

apiVersion: run.googleapis.com/v1
kind: Job
metadata:
  name: example-batch-job
  namespace: default
spec:
  template:
    spec:
      containers:
      - image: gcr.io/cloudrun/hello
        command: ["/bin/sh", "-c", "echo Hello from batch job"]
      restartPolicy: Never
  parallelism: 1
  taskCount: 1

This YAML file defines a Cloud Run job named example-batch-job. It runs a container from the image gcr.io/cloudrun/hello that prints a message. The job runs one task at a time (parallelism: 1) and runs only one task total (taskCount: 1). The restartPolicy: Never means the task won't restart if it fails.

Commands

This command creates a Cloud Run job named example-batch-job using the specified container image and command. It sets the region to us-central1.

Terminal

gcloud run jobs create example-batch-job --image gcr.io/cloudrun/hello --command "/bin/sh" --args "-c","echo Hello from batch job" --region us-central1

Expected Output

Created job [example-batch-job] in region [us-central1].

→

--image - Specifies the container image to run.

→

--command - Overrides the container's default command.

→

--args - Arguments passed to the command.

This command starts running the example-batch-job job in the specified region.

Terminal

gcloud run jobs execute example-batch-job --region us-central1

Expected Output

Started execution [example-batch-job-00001] of job [example-batch-job].

→

--region - Specifies the region where the job runs.

This command shows details about the job execution, including status and logs.

Terminal

gcloud run jobs executions describe example-batch-job-00001 --region us-central1

Expected Output

name: projects/PROJECT_ID/locations/us-central1/jobs/example-batch-job/executions/example-batch-job-00001

status:

  conditions:

  - type: Completed

    status: "True"

  completionTime: '2024-06-01T12:00:00Z'

  startTime: '2024-06-01T11:59:50Z'

  succeededCount: 1

→

--region - Specifies the region of the job execution.

This command lists all Cloud Run jobs in the specified region to verify the job exists.

Terminal

gcloud run jobs list --region us-central1

Expected Output

NAME               REGION        LAST RUN

example-batch-job  us-central1   2024-06-01T12:00:00Z

→

--region - Specifies the region to list jobs from.

Key Concept

If you remember nothing else from this pattern, remember: Cloud Run jobs let you run one-time or batch tasks easily without managing servers.

Common Mistakes

Trying to run a Cloud Run job without creating it first.

The job must exist before you can execute it, or the command will fail.

Always create the job with 'gcloud run jobs create' before executing it.

Not specifying the region consistently in commands.

Cloud Run jobs are regional, so commands must use the same region or they won't find the job.

Use the --region flag with the same region for all job commands.

Setting restartPolicy to Always in a batch job.

Batch jobs should not restart tasks automatically; this can cause unexpected repeated runs.

Set restartPolicy to Never for batch jobs to run tasks only once.

Summary

Create a Cloud Run job with a container image and command using 'gcloud run jobs create'.

Run the job once with 'gcloud run jobs execute'.

Check the job execution status with 'gcloud run jobs executions describe'.

List all jobs in a region with 'gcloud run jobs list' to verify job presence.

Practice

1. What is the main purpose of Cloud Run jobs in Google Cloud?

easy

A. To manage user authentication

B. To host websites that run continuously

C. To store large amounts of data

D. To run containers that perform batch work and then stop

Solution

1. Step 1: Understand Cloud Run jobs purpose

   Cloud Run jobs are designed to run containers that start, perform batch tasks, and then stop.

2. Step 2: Compare with other options

   Options A, B, and C describe other services or functions not related to batch jobs.

3. Final Answer:

   To run containers that perform batch work and then stop -> Option D

4. Quick Check:

   Batch work = Cloud Run jobs [OK]

Hint: Batch tasks start and stop, not continuous [OK]

Common Mistakes:

• Thinking Cloud Run jobs run continuously
• Confusing jobs with storage or hosting
• Assuming jobs manage user authentication

2. Which command correctly creates a Cloud Run job named data-processor?

easy

A. gcloud beta run create job data-processor --image=gcr.io/my-project/my-image

B. gcloud run jobs create data-processor --image=gcr.io/my-project/my-image

C. gcloud beta run jobs create data-processor --image=gcr.io/my-project/my-image

D. gcloud run create job data-processor --image=gcr.io/my-project/my-image

Solution

1. Step 1: Identify correct command syntax

   The correct command uses 'gcloud beta run jobs create' followed by the job name and image.

2. Step 2: Check each option

   gcloud beta run jobs create data-processor --image=gcr.io/my-project/my-image matches the correct syntax. Options B, C, and D have incorrect order or missing 'beta' keyword.

3. Final Answer:

   gcloud beta run jobs create data-processor --image=gcr.io/my-project/my-image -> Option C

4. Quick Check:

   Correct command syntax = gcloud beta run jobs create data-processor --image=gcr.io/my-project/my-image [OK]

Hint: Use 'gcloud beta run jobs create' to create jobs [OK]

Common Mistakes:

• Omitting 'beta' keyword
• Swapping 'create' and 'jobs' keywords
• Using 'gcloud run create job' instead

3. Given this command sequence:

gcloud beta run jobs create batch-job --image=gcr.io/my-project/batch-image
gcloud beta run jobs execute batch-job

What happens when the execute command runs?

medium

A. The batch-job container starts, runs the batch task, then stops

B. The batch-job container runs continuously until manually stopped

C. The batch-job container fails because execute requires an extra flag

D. The batch-job container is deleted after execution

Solution

1. Step 1: Understand execute command behavior

   Executing a Cloud Run job starts the container, runs the batch task, and then stops automatically.

2. Step 2: Evaluate other options

   The batch-job container runs continuously until manually stopped is wrong because jobs do not run continuously. The batch-job container fails because execute requires an extra flag is incorrect; no extra flag is required. The batch-job container is deleted after execution is false; the job is not deleted after execution.

3. Final Answer:

   The batch-job container starts, runs the batch task, then stops -> Option A

4. Quick Check:

   Execute runs batch task once then stops [OK]

Hint: Execute runs job once, then container stops [OK]

Common Mistakes:

• Thinking execute runs container continuously
• Assuming extra flags are mandatory
• Believing job deletes after run

4. You try to run this command:

gcloud run jobs execute my-job

But get an error saying the command is not found. What is the likely cause?

medium

A. You forgot to use the 'beta' keyword before 'run jobs execute'

B. The job name 'my-job' does not exist

C. You need to add '--image' flag when executing

D. You must be logged out to run jobs

Solution

1. Step 1: Check command syntax for Cloud Run jobs

   The 'execute' command for Cloud Run jobs is currently in beta, so 'gcloud beta run jobs execute' is required.

2. Step 2: Analyze error cause

   Missing 'beta' keyword causes 'command not found' error. Other options do not cause this specific error.

3. Final Answer:

   You forgot to use the 'beta' keyword before 'run jobs execute' -> Option A

4. Quick Check:

   Missing 'beta' causes command error [OK]

Hint: Use 'gcloud beta run jobs execute' for job runs [OK]

Common Mistakes:

• Omitting 'beta' keyword
• Assuming job name error causes command not found
• Adding unnecessary flags on execute

5. You want to run a batch job every day at midnight using Cloud Run jobs. Which approach is best?

hard

A. Create a Cloud Run service and keep it running 24/7 to check the time and run batch tasks

B. Create the job with 'gcloud beta run jobs create' and schedule it using Cloud Scheduler to run 'gcloud beta run jobs execute' daily

C. Manually run 'gcloud beta run jobs execute' every day at midnight

D. Use Cloud Storage triggers to start the job automatically at midnight

Solution

1. Step 1: Understand scheduling batch jobs

   Cloud Run jobs are designed for batch tasks and can be triggered on a schedule using Cloud Scheduler.

2. Step 2: Evaluate options for automation

   Create the job with 'gcloud beta run jobs create' and schedule it using Cloud Scheduler to run 'gcloud beta run jobs execute' daily uses Cloud Scheduler to run the job daily, which is best practice. Create a Cloud Run service and keep it running 24/7 to check the time and run batch tasks wastes resources by running continuously. Manually run 'gcloud beta run jobs execute' every day at midnight is manual and not automated. Use Cloud Storage triggers to start the job automatically at midnight is incorrect because Cloud Storage triggers do not support time-based schedules.

3. Final Answer:

   Create the job with 'gcloud beta run jobs create' and schedule it using Cloud Scheduler to run 'gcloud beta run jobs execute' daily -> Option B

4. Quick Check:

   Use Cloud Scheduler to automate job runs [OK]

Hint: Use Cloud Scheduler to automate job execution [OK]

Common Mistakes:

• Running jobs manually instead of scheduling
• Keeping services running continuously for batch tasks
• Using unsupported triggers for time scheduling