Storage transfer service in GCP - Mini Project: Build & Apply

Storage Transfer Service Setup on GCP

📖 Scenario: You work for a company that needs to move files from one Google Cloud Storage bucket to another regularly. You want to automate this process using Google Cloud's Storage Transfer Service.

🎯 Goal: Create a Storage Transfer Service job that copies data from a source bucket to a destination bucket on Google Cloud Storage.

📋 What You'll Learn

Create a dictionary called transfer_job with the source and destination bucket names

Add a configuration variable called schedule to specify the transfer start date

Write the core logic to define the transfer spec inside transfer_job

Complete the transfer job configuration by adding the status field set to ENABLED

💡 Why This Matters

🌍 Real World

Automating data movement between storage buckets is common in cloud projects for backup, migration, or data processing workflows.

💼 Career

Understanding how to configure and automate Storage Transfer Service jobs is valuable for cloud engineers and administrators managing data pipelines on Google Cloud.

1

Create the initial transfer job dictionary

Create a dictionary called transfer_job with the keys project_id set to "my-gcp-project", transfer_spec as an empty dictionary, and description set to "Daily bucket transfer".

GCP

# Create the transfer_job dictionary with project_id, transfer_spec, and description
# Your code here

Need a hint?

Use curly braces to create a dictionary. Include the exact keys and values as specified.

2

Add the schedule configuration

Add a variable called schedule as a dictionary with the key start_date set to a dictionary containing year: 2024, month: 7, and day: 1.

GCP

transfer_job = {
    "project_id": "my-gcp-project",
    "transfer_spec": {},
    "description": "Daily bucket transfer"
}
# Add the schedule dictionary with start_date year 2024, month 7, day 1
# Your code here

Need a hint?

Create a nested dictionary for the start_date inside schedule.

3

Define the transfer specification

Inside the transfer_job dictionary, set the transfer_spec key to a dictionary with gcs_data_source containing bucket_name set to "source-bucket", and gcs_data_sink containing bucket_name set to "destination-bucket".

GCP

transfer_job = {
    "project_id": "my-gcp-project",
    "transfer_spec": {},
    "description": "Daily bucket transfer"
}
schedule = {
    "start_date": {
        "year": 2024,
        "month": 7,
        "day": 1
    }
}
# Set transfer_spec with gcs_data_source and gcs_data_sink bucket names
# Your code here

Need a hint?

Update the transfer_spec dictionary with source and sink bucket names exactly as shown.

4

Complete the transfer job with status

Add the key status with the value "ENABLED" to the transfer_job dictionary to activate the transfer job.

GCP

transfer_job = {
    "project_id": "my-gcp-project",
    "transfer_spec": {
        "gcs_data_source": {
            "bucket_name": "source-bucket"
        },
        "gcs_data_sink": {
            "bucket_name": "destination-bucket"
        }
    },
    "description": "Daily bucket transfer"
}
schedule = {
    "start_date": {
        "year": 2024,
        "month": 7,
        "day": 1
    }
}
# Add status key with value "ENABLED" to transfer_job
# Your code here

Need a hint?

Set the status key in transfer_job to "ENABLED" to activate the job.