Storage transfer service in GCP - Step-by-Step Execution

Process Flow - Storage transfer service

Define Transfer Job

↓

Set Source and Destination

↓

Configure Schedule and Options

↓

Start Transfer Job

↓

Service Copies Data

↓

Monitor Transfer Status

↓

Transfer Complete or Error

The flow shows how you create a transfer job, set source and destination, start it, and monitor until completion.

Execution Sample

GCP

gcloud transfer jobs create \
  --source-bucket=source-bucket-name \
  --sink-bucket=destination-bucket-name \
  --schedule-start-date=2024-06-01 \
  --schedule-end-date=2024-06-01 \
  --description="Daily backup"

This command creates a transfer job to copy data from one bucket to another on a specific date.

Process Table

Step	Action	Input/Config	Service Behavior	Result
1	Define Transfer Job	Job name, description	Registers job in service	Job created with ID
2	Set Source and Destination	Source: source-bucket-name Destination: destination-bucket-name	Validates buckets exist	Buckets linked to job
3	Configure Schedule	Start date: 2024-06-01 End date: 2024-06-01	Schedules job for that day	Job scheduled
4	Start Transfer Job	Trigger job start	Service begins copying data	Transfer in progress
5	Monitor Transfer Status	Check job status	Reports progress, errors	Status: Running or Completed
6	Transfer Complete	Job finishes copying	Marks job as done	Data copied successfully
7	Exit	No more scheduled runs	Job ends	Transfer job completed

💡 Transfer job completes after copying all data on scheduled date.

Status Tracker

Variable	Start	After Step 2	After Step 3	After Step 4	After Step 6	Final
Job ID	None	Job123	Job123	Job123	Job123	Job123
Source Bucket	None	source-bucket-name	source-bucket-name	source-bucket-name	source-bucket-name	source-bucket-name
Destination Bucket	None	destination-bucket-name	destination-bucket-name	destination-bucket-name	destination-bucket-name	destination-bucket-name
Schedule	None	None	2024-06-01 to 2024-06-01	2024-06-01 to 2024-06-01	2024-06-01 to 2024-06-01	2024-06-01 to 2024-06-01
Status	None	Pending	Pending	Running	Completed	Completed

Key Moments - 3 Insights

Why does the transfer job have a status of 'Pending' before it starts copying data?

Because after setting the schedule, the job waits for the scheduled time to start. This is shown in execution_table row 3 where the job is scheduled but not yet running.

What happens if the source or destination bucket does not exist when creating the job?

The service validates buckets at step 2 and will not create the job if buckets are invalid. This prevents errors during transfer.

How do you know when the transfer job is fully complete?

At step 6 in the execution_table, the status changes to 'Completed' indicating all data has been copied and the job is done.

Visual Quiz - 3 Questions

Look at the execution_table, what is the job status after step 4?

A Pending

B Completed

C Running

D Failed

💡 Hint

Check the 'Status' variable in variable_tracker after Step 4.

At which step does the service start copying data?

A Step 3

B Step 4

C Step 2

D Step 6

💡 Hint

Look at the 'Service Behavior' column in execution_table for when copying begins.

If the schedule end date is before the start date, what would happen?

A Job creation fails or no runs scheduled

B Job runs once on start date

C Job runs multiple times

D Job runs indefinitely

💡 Hint

Consider the scheduling logic in execution_table step 3 and how invalid dates affect scheduling.

Concept Snapshot

Storage Transfer Service lets you copy data between buckets or from other sources.
You create a transfer job, set source and destination buckets, and schedule when to run.
The service copies data on schedule and reports status.
You monitor progress and know when transfer completes.
It helps automate data movement safely and reliably.

Full Transcript

Storage Transfer Service in Google Cloud helps move data from one storage location to another automatically. First, you create a transfer job by defining its name and description. Then, you specify the source bucket where data currently lives and the destination bucket where data should go. Next, you set a schedule for when the transfer should happen, such as a specific date or recurring times. When the scheduled time arrives, the service starts copying data from source to destination. You can check the job status anytime to see if it is running, completed, or if there were errors. Once all data is copied, the job marks itself complete. This process helps keep data backed up or moved without manual copying. The execution table shows each step from job creation to completion, and the variable tracker shows how job status and settings change over time. Common confusions include understanding the 'Pending' status before start, bucket validation, and knowing when the job is done. The visual quiz tests understanding of these steps and statuses. Overall, Storage Transfer Service automates and simplifies data transfers in the cloud.

Practice

1. What is the primary purpose of the Google Cloud Storage Transfer Service?

easy

A. To move or copy data between different storage locations automatically

B. To create backups of virtual machines

C. To monitor network traffic between cloud services

D. To manage user permissions in Google Cloud

Solution

1. Step 1: Understand the service function

   Storage Transfer Service is designed to move or copy data between storage locations like on-premises, AWS S3, or Google Cloud Storage.

2. Step 2: Eliminate unrelated options

   Options B, C, and D describe different services unrelated to data transfer.

3. Final Answer:

   To move or copy data between different storage locations automatically -> Option A

4. Quick Check:

   Storage Transfer Service = Data movement [OK]

Hint: Remember: Transfer Service moves or copies data [OK]

Common Mistakes:

• Confusing transfer service with backup or monitoring tools
• Thinking it manages user permissions
• Assuming it only works within Google Cloud

2. Which of the following is the correct way to specify a source in a Storage Transfer Service job configuration?

easy

A. "source": {"network": {"subnet": "default"}}

B. "source": {"vmInstance": {"name": "instance-1"}}

C. "source": {"gcsDataSource": {"bucketName": "my-source-bucket"}}

D. "source": {"sqlDatabase": {"dbName": "mydb"}}

Solution

1. Step 1: Identify valid source types

   Storage Transfer Service supports sources like Google Cloud Storage buckets, AWS S3 buckets, or on-premises data.

2. Step 2: Match correct JSON syntax for GCS source

   The correct syntax uses "gcsDataSource" with a "bucketName" field, as shown in "source": {"gcsDataSource": {"bucketName": "my-source-bucket"}}.

3. Final Answer:

   "source": {"gcsDataSource": {"bucketName": "my-source-bucket"}} -> Option C

4. Quick Check:

   Source config for GCS = "source": {"gcsDataSource": {"bucketName": "my-source-bucket"}} [OK]

Hint: Look for "gcsDataSource" with bucketName for GCS source [OK]

Common Mistakes:

• Using unsupported source types like VM or SQL database
• Incorrect JSON structure for source
• Confusing source with destination fields

3. Given this Storage Transfer Service schedule configuration snippet:

{"schedule": {"scheduleStartDate": {"year": 2024, "month": 6, "day": 10}, "startTimeOfDay": {"hours": 3, "minutes": 0}}}

When will the transfer job start?

medium

A. At 3:00 AM on June 10, 2024

B. At 3:00 PM on June 10, 2024

C. At midnight on June 10, 2024

D. At 3:00 AM on June 9, 2024

Solution

1. Step 1: Read the scheduleStartDate and startTimeOfDay

   The date is June 10, 2024, and the time is 3 hours and 0 minutes, which means 3:00 AM.

2. Step 2: Confirm time format

   The time is in 24-hour format, so 3 means 3 AM, not PM.

3. Final Answer:

   At 3:00 AM on June 10, 2024 -> Option A

4. Quick Check:

   3 hours = 3 AM, date matches [OK]

Hint: Hours in 24-hour format; 3 means 3 AM [OK]

Common Mistakes:

• Mistaking 3 for 3 PM instead of 3 AM
• Ignoring the date and assuming current day
• Confusing startTimeOfDay with duration

4. You wrote this Storage Transfer Service job configuration but the transfer never starts:

{"transferJob": {"status": "ENABLED", "schedule": {"scheduleStartDate": {"year": 2024, "month": 7, "day": 20}, "startTimeOfDay": {"hours": 25, "minutes": 0}}}}

What is the problem?

medium

A. The transferJob status should be DISABLED to start

B. The scheduleStartDate is in the past

C. The minutes value must be 30 or 60

D. The startTimeOfDay hours value is invalid; it must be between 0 and 23

Solution

1. Step 1: Check startTimeOfDay values

   The hours field is set to 25, which is invalid because valid hours range from 0 to 23.

2. Step 2: Validate other fields

   The scheduleStartDate is a future date, status is ENABLED which is correct, and minutes is 0 which is valid.

3. Final Answer:

   The startTimeOfDay hours value is invalid; it must be between 0 and 23 -> Option D

4. Quick Check:

   Hours must be 0-23, 25 is invalid [OK]

Hint: Hours must be 0-23; 25 is invalid [OK]

Common Mistakes:

• Assuming status DISABLED starts the job
• Thinking minutes must be 30 or 60
• Ignoring invalid hour value

5. You want to transfer data daily from an AWS S3 bucket to a Google Cloud Storage bucket using Storage Transfer Service. Which combination of settings is required?

hard

A. Set GCS bucket as source, AWS S3 as destination, and schedule weekly

B. Set AWS S3 as source with access keys, GCS bucket as destination, and schedule daily

C. Use on-premises source with VPN, GCS bucket as destination, and schedule once

D. Set AWS S3 as source without credentials, GCS bucket as destination, and no schedule

Solution

1. Step 1: Identify source and destination

   The source is AWS S3 bucket, which requires access keys for authentication. The destination is a Google Cloud Storage bucket.

2. Step 2: Set schedule for daily transfers

   To transfer data daily, the schedule must be configured to run every day.

3. Final Answer:

   Set AWS S3 as source with access keys, GCS bucket as destination, and schedule daily -> Option B

4. Quick Check:

   AWS S3 source + credentials + daily schedule = Set AWS S3 as source with access keys, GCS bucket as destination, and schedule daily [OK]

Hint: AWS S3 needs keys; schedule daily for repeated transfers [OK]

Common Mistakes:

• Forgetting AWS credentials
• Reversing source and destination
• Not setting a schedule for repeated transfers