Data access logs in GCP - Mini Project: Build & Apply

Enable and Analyze Data Access Logs in GCP

📖 Scenario: You are a cloud administrator for a company using Google Cloud Platform (GCP). You want to enable data access logs for a specific Cloud Storage bucket to monitor who accesses the data. Then, you want to query these logs to find access patterns.

🎯 Goal: Enable data access logs for a Cloud Storage bucket and write a query to analyze the logs in BigQuery.

📋 What You'll Learn

Enable data access logs for a Cloud Storage bucket named my-data-bucket.

Create a sink to export logs to a BigQuery dataset named data_access_logs.

Write a BigQuery SQL query to count the number of accesses per user.

Use exact names: my-data-bucket, data_access_logs, access_logs_sink.

💡 Why This Matters

🌍 Real World

Monitoring data access is critical for security and compliance in cloud environments. This project shows how to enable and analyze data access logs in GCP.

💼 Career

Cloud administrators and security engineers often need to configure logging and analyze access patterns to protect data and meet audit requirements.

1

Enable Data Access Logs for Cloud Storage Bucket

Write a gcloud command to enable data access logs for the Cloud Storage bucket named my-data-bucket by updating its logging configuration to include DATA_READ and DATA_WRITE log types.

GCP

# Your gcloud command here to enable data access logs for my-data-bucket

Hint

Use gcloud logging buckets update with a filter for my-data-bucket and include storage.objects.get and storage.objects.create methods.

2

Create a Logging Sink to Export Logs to BigQuery

Write a gcloud command to create a logging sink named access_logs_sink that exports logs from the my-data-bucket data access logs to a BigQuery dataset named data_access_logs.

GCP

gcloud logging buckets update _Default --location=global --add-log-filter='resource.type="gcs_bucket" AND resource.labels.bucket_name="my-data-bucket" AND (protoPayload.methodName:"storage.objects.get" OR protoPayload.methodName:"storage.objects.create")'
# Your gcloud command here to create a sink named access_logs_sink exporting to BigQuery dataset data_access_logs

Hint

Use gcloud logging sinks create with the sink name access_logs_sink and target BigQuery dataset data_access_logs. Filter logs for my-data-bucket.

3

Write a BigQuery SQL Query to Count Accesses per User

Write a BigQuery SQL query that selects the protoPayload.authenticationInfo.principalEmail as user_email and counts the number of accesses as access_count from the data_access_logs dataset's cloudaudit_googleapis_com_data_access table, grouping by user_email and ordering by access_count descending.

GCP

gcloud logging buckets update _Default --location=global --add-log-filter='resource.type="gcs_bucket" AND resource.labels.bucket_name="my-data-bucket" AND (protoPayload.methodName:"storage.objects.get" OR protoPayload.methodName:"storage.objects.create")'
gcloud logging sinks create access_logs_sink bigquery.googleapis.com/projects/$(gcloud config get-value project)/datasets/data_access_logs --log-filter='resource.type="gcs_bucket" AND resource.labels.bucket_name="my-data-bucket"' --description="Sink for my-data-bucket data access logs"

# Your BigQuery SQL query here to count accesses per user

Hint

Use protoPayload.authenticationInfo.principalEmail as user_email and count rows grouped by user_email.

4

Complete the Logging Sink with Permissions

Write a gcloud command to grant the logging service account the BigQuery Data Editor role on the data_access_logs dataset to allow the sink access_logs_sink to write logs.

GCP

gcloud logging buckets update _Default --location=global --add-log-filter='resource.type="gcs_bucket" AND resource.labels.bucket_name="my-data-bucket" AND (protoPayload.methodName:"storage.objects.get" OR protoPayload.methodName:"storage.objects.create")'
gcloud logging sinks create access_logs_sink bigquery.googleapis.com/projects/$(gcloud config get-value project)/datasets/data_access_logs --log-filter='resource.type="gcs_bucket" AND resource.labels.bucket_name="my-data-bucket"' --description="Sink for my-data-bucket data access logs"

SELECT protoPayload.authenticationInfo.principalEmail AS user_email, COUNT(*) AS access_count
FROM `$(gcloud config get-value project).data_access_logs.cloudaudit_googleapis_com_data_access`
GROUP BY user_email
ORDER BY access_count DESC

# Your gcloud command here to grant BigQuery Data Editor role to the sink's service account

Hint

Use gcloud logging sinks describe access_logs_sink to get the sink's service account, then grant it roles/bigquery.dataEditor on the project.

Practice

1. What is the main purpose of data access logs in Google Cloud?

easy

A. To reduce storage costs

B. To record who accessed or changed your cloud data

C. To automatically backup your data

D. To increase the speed of data processing

Solution

1. Step 1: Understand what data access logs record

   Data access logs track user actions on cloud data, such as access or changes.

2. Step 2: Identify the main purpose

   The main goal is to monitor and audit who accessed or modified data for security and compliance.

3. Final Answer:

   To record who accessed or changed your cloud data -> Option B

4. Quick Check:

   Data access logs = record access and changes [OK]

Hint: Logs track data access and changes, not performance or cost [OK]

Common Mistakes:

• Confusing logs with backup or performance tools
• Thinking logs reduce costs automatically
• Assuming logs speed up data processing

2. Which of the following is the correct way to enable data access logs for a Google Cloud service?

easy

A. Install a logging agent on your local machine

B. Create a firewall rule to allow logging

C. Use Cloud Storage lifecycle rules

D. Enable audit logs in Cloud IAM & Admin for the service

Solution

1. Step 1: Identify how to enable data access logs

   Data access logs are enabled by configuring audit logs in Cloud IAM & Admin for the specific service.

2. Step 2: Eliminate unrelated options

   Firewall rules, local agents, and storage lifecycle rules do not enable audit logging.

3. Final Answer:

   Enable audit logs in Cloud IAM & Admin for the service -> Option D

4. Quick Check:

   Audit logs enabled via IAM & Admin [OK]

Hint: Audit logs are enabled in IAM & Admin, not by firewall or agents [OK]

Common Mistakes:

• Confusing firewall rules with logging setup
• Thinking local agents enable cloud audit logs
• Using storage lifecycle rules for logging

3. Given this filter for a logging sink:
resource.type="bigquery_resource" AND protoPayload.methodName="tableservice.insert"
What logs will this sink collect?

medium

A. All Cloud Storage bucket creations

B. All BigQuery query executions

C. All BigQuery table creation events

D. All Compute Engine VM starts

Solution

1. Step 1: Analyze the filter components

   The filter targets resource type 'bigquery_resource' and method 'tableservice.insert', which means table creation in BigQuery.

2. Step 2: Match filter to log events

   This filter collects logs about BigQuery tables being created, not queries or other services.

3. Final Answer:

   All BigQuery table creation events -> Option C

4. Quick Check:

   Filter matches BigQuery table inserts [OK]

Hint: Look for resource type and method name to identify event type [OK]

Common Mistakes:

• Confusing table creation with query execution
• Mixing BigQuery with Cloud Storage or Compute Engine
• Ignoring methodName in filter

4. You created a logging sink to export data access logs to BigQuery but see no data arriving. What is a likely cause?

medium

A. Data access logs are disabled in the project

B. BigQuery dataset is in a different region

C. The sink uses a Cloud Storage bucket instead of BigQuery

D. The sink filter excludes data access logs

Solution

1. Step 1: Check if data access logs are enabled

   If data access logs are disabled, no logs will be generated or exported.

2. Step 2: Consider other causes

   While filters and regions matter, the most common cause is logs not enabled at all.

3. Final Answer:

   Data access logs are disabled in the project -> Option A

4. Quick Check:

   Logs must be enabled to export data [OK]

Hint: Verify logs are enabled before checking sinks [OK]

Common Mistakes:

• Assuming filter or region is always the problem
• Confusing sink destination with log generation
• Not enabling data access logs in IAM & Admin

5. You want to audit all data access to Cloud Storage buckets in your project and store logs in BigQuery for analysis. Which steps should you take?

hard

A. Enable data access audit logs for Cloud Storage, create a sink with filter resource.type="gcs_bucket", export to BigQuery dataset

B. Enable system logs only, create a sink exporting all logs to Cloud Storage

C. Install logging agents on VMs, export logs to BigQuery without filters

D. Create a firewall rule to allow logging traffic, enable data access logs

Solution

1. Step 1: Enable data access audit logs for Cloud Storage

   This ensures logs capture who accessed or changed Cloud Storage buckets.

2. Step 2: Create a sink with filter resource.type="gcs_bucket" and export to BigQuery

   This filters logs to only Cloud Storage bucket access and stores them in BigQuery for analysis.

3. Final Answer:

   Enable data access audit logs for Cloud Storage, create a sink with filter resource.type="gcs_bucket", export to BigQuery dataset -> Option A

4. Quick Check:

   Enable logs + filter + export to BigQuery = audit setup [OK]

Hint: Enable logs, filter by resource, export to BigQuery [OK]

Common Mistakes:

• Using system logs instead of data access logs
• Exporting all logs without filtering
• Relying on firewall rules for logging