Storage transfer service in GCP - Mini Project: Build & Apply

Storage Transfer Service Setup on GCP

📖 Scenario: You work for a company that needs to move files from one Google Cloud Storage bucket to another regularly. You want to automate this process using Google Cloud's Storage Transfer Service.

🎯 Goal: Create a Storage Transfer Service job that copies data from a source bucket to a destination bucket on Google Cloud Storage.

📋 What You'll Learn

Create a dictionary called transfer_job with the source and destination bucket names

Add a configuration variable called schedule to specify the transfer start date

Write the core logic to define the transfer spec inside transfer_job

Complete the transfer job configuration by adding the status field set to ENABLED

💡 Why This Matters

🌍 Real World

Automating data movement between storage buckets is common in cloud projects for backup, migration, or data processing workflows.

💼 Career

Understanding how to configure and automate Storage Transfer Service jobs is valuable for cloud engineers and administrators managing data pipelines on Google Cloud.

1

Create the initial transfer job dictionary

Create a dictionary called transfer_job with the keys project_id set to "my-gcp-project", transfer_spec as an empty dictionary, and description set to "Daily bucket transfer".

GCP

# Create the transfer_job dictionary with project_id, transfer_spec, and description
# Your code here

Hint

Use curly braces to create a dictionary. Include the exact keys and values as specified.

2

Add the schedule configuration

Add a variable called schedule as a dictionary with the key start_date set to a dictionary containing year: 2024, month: 7, and day: 1.

GCP

transfer_job = {
    "project_id": "my-gcp-project",
    "transfer_spec": {},
    "description": "Daily bucket transfer"
}
# Add the schedule dictionary with start_date year 2024, month 7, day 1
# Your code here

Hint

Create a nested dictionary for the start_date inside schedule.

3

Define the transfer specification

Inside the transfer_job dictionary, set the transfer_spec key to a dictionary with gcs_data_source containing bucket_name set to "source-bucket", and gcs_data_sink containing bucket_name set to "destination-bucket".

GCP

transfer_job = {
    "project_id": "my-gcp-project",
    "transfer_spec": {},
    "description": "Daily bucket transfer"
}
schedule = {
    "start_date": {
        "year": 2024,
        "month": 7,
        "day": 1
    }
}
# Set transfer_spec with gcs_data_source and gcs_data_sink bucket names
# Your code here

Hint

Update the transfer_spec dictionary with source and sink bucket names exactly as shown.

4

Complete the transfer job with status

Add the key status with the value "ENABLED" to the transfer_job dictionary to activate the transfer job.

GCP

transfer_job = {
    "project_id": "my-gcp-project",
    "transfer_spec": {
        "gcs_data_source": {
            "bucket_name": "source-bucket"
        },
        "gcs_data_sink": {
            "bucket_name": "destination-bucket"
        }
    },
    "description": "Daily bucket transfer"
}
schedule = {
    "start_date": {
        "year": 2024,
        "month": 7,
        "day": 1
    }
}
# Add status key with value "ENABLED" to transfer_job
# Your code here

Hint

Set the status key in transfer_job to "ENABLED" to activate the job.

Practice

1. What is the primary purpose of the Google Cloud Storage Transfer Service?

easy

A. To move or copy data between different storage locations automatically

B. To create backups of virtual machines

C. To monitor network traffic between cloud services

D. To manage user permissions in Google Cloud

Solution

1. Step 1: Understand the service function

   Storage Transfer Service is designed to move or copy data between storage locations like on-premises, AWS S3, or Google Cloud Storage.

2. Step 2: Eliminate unrelated options

   Options B, C, and D describe different services unrelated to data transfer.

3. Final Answer:

   To move or copy data between different storage locations automatically -> Option A

4. Quick Check:

   Storage Transfer Service = Data movement [OK]

Hint: Remember: Transfer Service moves or copies data [OK]

Common Mistakes:

• Confusing transfer service with backup or monitoring tools
• Thinking it manages user permissions
• Assuming it only works within Google Cloud

2. Which of the following is the correct way to specify a source in a Storage Transfer Service job configuration?

easy

A. "source": {"network": {"subnet": "default"}}

B. "source": {"vmInstance": {"name": "instance-1"}}

C. "source": {"gcsDataSource": {"bucketName": "my-source-bucket"}}

D. "source": {"sqlDatabase": {"dbName": "mydb"}}

Solution

1. Step 1: Identify valid source types

   Storage Transfer Service supports sources like Google Cloud Storage buckets, AWS S3 buckets, or on-premises data.

2. Step 2: Match correct JSON syntax for GCS source

   The correct syntax uses "gcsDataSource" with a "bucketName" field, as shown in "source": {"gcsDataSource": {"bucketName": "my-source-bucket"}}.

3. Final Answer:

   "source": {"gcsDataSource": {"bucketName": "my-source-bucket"}} -> Option C

4. Quick Check:

   Source config for GCS = "source": {"gcsDataSource": {"bucketName": "my-source-bucket"}} [OK]

Hint: Look for "gcsDataSource" with bucketName for GCS source [OK]

Common Mistakes:

• Using unsupported source types like VM or SQL database
• Incorrect JSON structure for source
• Confusing source with destination fields

3. Given this Storage Transfer Service schedule configuration snippet:

{"schedule": {"scheduleStartDate": {"year": 2024, "month": 6, "day": 10}, "startTimeOfDay": {"hours": 3, "minutes": 0}}}

When will the transfer job start?

medium

A. At 3:00 AM on June 10, 2024

B. At 3:00 PM on June 10, 2024

C. At midnight on June 10, 2024

D. At 3:00 AM on June 9, 2024

Solution

1. Step 1: Read the scheduleStartDate and startTimeOfDay

   The date is June 10, 2024, and the time is 3 hours and 0 minutes, which means 3:00 AM.

2. Step 2: Confirm time format

   The time is in 24-hour format, so 3 means 3 AM, not PM.

3. Final Answer:

   At 3:00 AM on June 10, 2024 -> Option A

4. Quick Check:

   3 hours = 3 AM, date matches [OK]

Hint: Hours in 24-hour format; 3 means 3 AM [OK]

Common Mistakes:

• Mistaking 3 for 3 PM instead of 3 AM
• Ignoring the date and assuming current day
• Confusing startTimeOfDay with duration

4. You wrote this Storage Transfer Service job configuration but the transfer never starts:

{"transferJob": {"status": "ENABLED", "schedule": {"scheduleStartDate": {"year": 2024, "month": 7, "day": 20}, "startTimeOfDay": {"hours": 25, "minutes": 0}}}}

What is the problem?

medium

A. The transferJob status should be DISABLED to start

B. The scheduleStartDate is in the past

C. The minutes value must be 30 or 60

D. The startTimeOfDay hours value is invalid; it must be between 0 and 23

Solution

1. Step 1: Check startTimeOfDay values

   The hours field is set to 25, which is invalid because valid hours range from 0 to 23.

2. Step 2: Validate other fields

   The scheduleStartDate is a future date, status is ENABLED which is correct, and minutes is 0 which is valid.

3. Final Answer:

   The startTimeOfDay hours value is invalid; it must be between 0 and 23 -> Option D

4. Quick Check:

   Hours must be 0-23, 25 is invalid [OK]

Hint: Hours must be 0-23; 25 is invalid [OK]

Common Mistakes:

• Assuming status DISABLED starts the job
• Thinking minutes must be 30 or 60
• Ignoring invalid hour value

5. You want to transfer data daily from an AWS S3 bucket to a Google Cloud Storage bucket using Storage Transfer Service. Which combination of settings is required?

hard

A. Set GCS bucket as source, AWS S3 as destination, and schedule weekly

B. Set AWS S3 as source with access keys, GCS bucket as destination, and schedule daily

C. Use on-premises source with VPN, GCS bucket as destination, and schedule once

D. Set AWS S3 as source without credentials, GCS bucket as destination, and no schedule

Solution

1. Step 1: Identify source and destination

   The source is AWS S3 bucket, which requires access keys for authentication. The destination is a Google Cloud Storage bucket.

2. Step 2: Set schedule for daily transfers

   To transfer data daily, the schedule must be configured to run every day.

3. Final Answer:

   Set AWS S3 as source with access keys, GCS bucket as destination, and schedule daily -> Option B

4. Quick Check:

   AWS S3 source + credentials + daily schedule = Set AWS S3 as source with access keys, GCS bucket as destination, and schedule daily [OK]

Hint: AWS S3 needs keys; schedule daily for repeated transfers [OK]

Common Mistakes:

• Forgetting AWS credentials
• Reversing source and destination
• Not setting a schedule for repeated transfers