Data access logs in GCP - Commands & Configuration

Introduction

Data access logs help you see who looked at or used your data in Google Cloud. They show details about data reads and writes, helping you keep track of access and improve security.

When you want to know who accessed your Cloud Storage files and when.

When you need to audit BigQuery queries that read sensitive data.

When you want to monitor access to your Cloud SQL databases for compliance.

When you suspect unauthorized data access and want to investigate.

When you want to keep a record of data usage for billing or analysis.

Config File - audit_config.yaml

audit_config.yaml

auditConfigs:
- service: storage.googleapis.com
  auditLogConfigs:
  - logType: DATA_READ
  - logType: DATA_WRITE
- service: bigquery.googleapis.com
  auditLogConfigs:
  - logType: DATA_READ
  - logType: DATA_WRITE
- service: sqladmin.googleapis.com
  auditLogConfigs:
  - logType: DATA_READ
  - logType: DATA_WRITE

This file configures audit logging for data access on Cloud Storage, BigQuery, and Cloud SQL services.

auditConfigs: List of services to enable logging for.

service: The Google Cloud service name.

auditLogConfigs: Types of logs to capture, here both data reads and writes.

Commands

Create a logging sink to export data access logs from Cloud Storage to a storage bucket. This helps keep logs for review.

Terminal

gcloud logging sinks create data-access-sink storage.googleapis.com/my-data-access-logs-bucket --log-filter='resource.type="gcs_bucket" AND protoPayload.methodName:("storage.objects.get" OR "storage.objects.list")'

Expected Output

Created sink [data-access-sink].

→

--log-filter - Filters logs to only include data read operations on Cloud Storage buckets.

Check the storage bucket to see if data access logs are arriving correctly.

Terminal

gsutil ls gs://my-data-access-logs-bucket

Expected Output

logs_2024-06-01T00:00:00Z.json

logs_2024-06-01T01:00:00Z.json

Read the last 5 BigQuery job completion logs to see data access events like queries run.

Terminal

gcloud logging read 'resource.type="bigquery_resource" AND protoPayload.methodName="jobservice.jobcompleted"' --limit=5 --format='json'

Expected Output

[

  {

    "protoPayload": {

      "methodName": "jobservice.jobcompleted",

      "serviceName": "bigquery.googleapis.com"

    },

    "resource": {

      "type": "bigquery_resource"

    }

  },

  {

    "protoPayload": {

      "methodName": "jobservice.jobcompleted",

      "serviceName": "bigquery.googleapis.com"

    },

    "resource": {

      "type": "bigquery_resource"

    }

  }

]

→

--limit - Limits the number of log entries returned.

→

--format - Formats the output as JSON for easy reading.

Key Concept

If you remember nothing else from this pattern, remember: data access logs show who used your data and when, helping you keep your cloud data safe and compliant.

Common Mistakes

Not enabling data access audit logs in the IAM policy or audit config.

Without enabling, no data access logs are recorded, so you cannot track data usage.

Always configure auditConfigs for the services you want to monitor before expecting logs.

Using a log filter that excludes data access events.

Logs will not include the data read/write events you want to see, missing important info.

Use filters that include protoPayload.methodName for data read/write operations.

Checking logs in the wrong project or storage bucket.

Logs won't appear if you look in the wrong place, causing confusion.

Verify the project and bucket names match your sink configuration.

Summary

Configure audit logging to capture data read and write events for your cloud services.

Create a logging sink to export these logs to a storage bucket for long-term access.

Use gcloud logging read commands with filters to view recent data access events.

Practice

1. What is the main purpose of data access logs in Google Cloud?

easy

A. To reduce storage costs

B. To record who accessed or changed your cloud data

C. To automatically backup your data

D. To increase the speed of data processing

Solution

1. Step 1: Understand what data access logs record

   Data access logs track user actions on cloud data, such as access or changes.

2. Step 2: Identify the main purpose

   The main goal is to monitor and audit who accessed or modified data for security and compliance.

3. Final Answer:

   To record who accessed or changed your cloud data -> Option B

4. Quick Check:

   Data access logs = record access and changes [OK]

Hint: Logs track data access and changes, not performance or cost [OK]

Common Mistakes:

• Confusing logs with backup or performance tools
• Thinking logs reduce costs automatically
• Assuming logs speed up data processing

2. Which of the following is the correct way to enable data access logs for a Google Cloud service?

easy

A. Install a logging agent on your local machine

B. Create a firewall rule to allow logging

C. Use Cloud Storage lifecycle rules

D. Enable audit logs in Cloud IAM & Admin for the service

Solution

1. Step 1: Identify how to enable data access logs

   Data access logs are enabled by configuring audit logs in Cloud IAM & Admin for the specific service.

2. Step 2: Eliminate unrelated options

   Firewall rules, local agents, and storage lifecycle rules do not enable audit logging.

3. Final Answer:

   Enable audit logs in Cloud IAM & Admin for the service -> Option D

4. Quick Check:

   Audit logs enabled via IAM & Admin [OK]

Hint: Audit logs are enabled in IAM & Admin, not by firewall or agents [OK]

Common Mistakes:

• Confusing firewall rules with logging setup
• Thinking local agents enable cloud audit logs
• Using storage lifecycle rules for logging

3. Given this filter for a logging sink:
resource.type="bigquery_resource" AND protoPayload.methodName="tableservice.insert"
What logs will this sink collect?

medium

A. All Cloud Storage bucket creations

B. All BigQuery query executions

C. All BigQuery table creation events

D. All Compute Engine VM starts

Solution

1. Step 1: Analyze the filter components

   The filter targets resource type 'bigquery_resource' and method 'tableservice.insert', which means table creation in BigQuery.

2. Step 2: Match filter to log events

   This filter collects logs about BigQuery tables being created, not queries or other services.

3. Final Answer:

   All BigQuery table creation events -> Option C

4. Quick Check:

   Filter matches BigQuery table inserts [OK]

Hint: Look for resource type and method name to identify event type [OK]

Common Mistakes:

• Confusing table creation with query execution
• Mixing BigQuery with Cloud Storage or Compute Engine
• Ignoring methodName in filter

4. You created a logging sink to export data access logs to BigQuery but see no data arriving. What is a likely cause?

medium

A. Data access logs are disabled in the project

B. BigQuery dataset is in a different region

C. The sink uses a Cloud Storage bucket instead of BigQuery

D. The sink filter excludes data access logs

Solution

1. Step 1: Check if data access logs are enabled

   If data access logs are disabled, no logs will be generated or exported.

2. Step 2: Consider other causes

   While filters and regions matter, the most common cause is logs not enabled at all.

3. Final Answer:

   Data access logs are disabled in the project -> Option A

4. Quick Check:

   Logs must be enabled to export data [OK]

Hint: Verify logs are enabled before checking sinks [OK]

Common Mistakes:

• Assuming filter or region is always the problem
• Confusing sink destination with log generation
• Not enabling data access logs in IAM & Admin

5. You want to audit all data access to Cloud Storage buckets in your project and store logs in BigQuery for analysis. Which steps should you take?

hard

A. Enable data access audit logs for Cloud Storage, create a sink with filter resource.type="gcs_bucket", export to BigQuery dataset

B. Enable system logs only, create a sink exporting all logs to Cloud Storage

C. Install logging agents on VMs, export logs to BigQuery without filters

D. Create a firewall rule to allow logging traffic, enable data access logs

Solution

1. Step 1: Enable data access audit logs for Cloud Storage

   This ensures logs capture who accessed or changed Cloud Storage buckets.

2. Step 2: Create a sink with filter resource.type="gcs_bucket" and export to BigQuery

   This filters logs to only Cloud Storage bucket access and stores them in BigQuery for analysis.

3. Final Answer:

   Enable data access audit logs for Cloud Storage, create a sink with filter resource.type="gcs_bucket", export to BigQuery dataset -> Option A

4. Quick Check:

   Enable logs + filter + export to BigQuery = audit setup [OK]

Hint: Enable logs, filter by resource, export to BigQuery [OK]

Common Mistakes:

• Using system logs instead of data access logs
• Exporting all logs without filtering
• Relying on firewall rules for logging