Storage transfer service in GCP - Commands & Configuration

Introduction

Moving large amounts of data between storage locations can be slow and error-prone. Storage Transfer Service helps you copy data quickly and reliably between cloud storage buckets or from on-premises to cloud storage.

When you want to copy files from one Google Cloud Storage bucket to another without downloading them locally.

When you need to migrate data from an on-premises server to Google Cloud Storage efficiently.

When you want to schedule regular backups of your cloud storage data to another location.

When you want to transfer data from Amazon S3 to Google Cloud Storage.

When you want to move large datasets without writing custom scripts or manual copying.

Config File - transfer-job.json

transfer-job.json

{
  "description": "Daily transfer from source bucket to destination bucket",
  "status": "ENABLED",
  "projectId": "example-project-123",
  "transferSpec": {
    "gcsDataSource": {
      "bucketName": "source-bucket-example"
    },
    "gcsDataSink": {
      "bucketName": "destination-bucket-example"
    },
    "objectConditions": {
      "minTimeElapsedSinceLastModification": "86400s"
    },
    "transferOptions": {
      "overwriteObjectsAlreadyExistingInSink": true
    }
  },
  "schedule": {
    "scheduleStartDate": {
      "year": 2024,
      "month": 6,
      "day": 1
    },
    "startTimeOfDay": {
      "hours": 2,
      "minutes": 0,
      "seconds": 0
    }
  }
}

This JSON file defines a transfer job that copies data daily from one Google Cloud Storage bucket to another.

• description: A friendly name for the job.
• status: Enables the job to run.
• projectId: Your Google Cloud project ID.
• transferSpec: Defines source and destination buckets and transfer options.
• schedule: Sets the job to run daily at 2 AM starting June 1, 2024.

Commands

This command creates a new Storage Transfer Service job using the configuration in transfer-job.json.

Terminal

gcloud transfer jobs create transfer-job.json

Expected Output

Created transfer job: transferJobs/1234567890abcdef

Lists all transfer jobs in the specified project to verify the job was created.

Terminal

gcloud transfer jobs list --project=example-project-123

Expected Output

NAME                        DESCRIPTION                          STATUS

transferJobs/1234567890abcdef  Daily transfer from source bucket to destination bucket  ENABLED

→

--project - Specifies the Google Cloud project to list jobs from

Shows recent transfer operations to check the status of running or completed transfers.

Terminal

gcloud transfer operations list --project=example-project-123

Expected Output

NAME                                   STATUS

transferOperations/abcdef1234567890    SUCCESS

→

--project - Specifies the Google Cloud project to list operations from

Key Concept

If you remember nothing else from this pattern, remember: Storage Transfer Service automates and schedules reliable data copying between cloud storage locations without manual downloads.

Common Mistakes

Using incorrect bucket names in the transfer job JSON.

The transfer job will fail because it cannot find the source or destination buckets.

Double-check bucket names exist and are spelled exactly as in your Google Cloud Storage.

Not enabling the transfer job status (leaving it DISABLED).

The job will be created but never run automatically.

Set the status field to ENABLED in the JSON configuration to activate the job.

Forgetting to specify the project ID when listing jobs or operations.

Commands will show no results or errors because they don't know which project to query.

Always include --project=your-project-id when running gcloud transfer commands.

Summary

Create a transfer job JSON file defining source, destination, schedule, and options.

Use gcloud transfer jobs create to create the job from the JSON file.

Verify the job exists with gcloud transfer jobs list and check transfer status with gcloud transfer operations list.