BigQuery SQL and pricing model in GCP - Commands & Configuration

Introduction

BigQuery lets you run SQL queries on large amounts of data quickly. It charges you based on the amount of data your queries scan, helping you control costs by writing efficient queries.

When you want to analyze large datasets without managing servers.

When you need to run fast SQL queries on data stored in the cloud.

When you want to pay only for the data you actually query.

When you want to store data in tables and run standard SQL commands.

When you want to avoid upfront costs and pay per query usage.

Commands

This command runs a SQL query on BigQuery public data to find the top 5 most common names in Texas. It uses standard SQL syntax by disabling legacy SQL.

Terminal

bq query --use_legacy_sql=false 'SELECT name, COUNT(*) as count FROM `bigquery-public-data.usa_names.usa_1910_2013` WHERE state = "TX" GROUP BY name ORDER BY count DESC LIMIT 5;'

Expected Output

Waiting on bqjob_r1234567890abcdef_00000123456789_1 ... (0s) Current status: DONE

+--------+-------+

| name   | count |

+--------+-------+

| Mary   | 12345 |

| James  | 11234 |

| John   | 11000 |

| Robert | 10500 |

| Linda  | 10000 |

+--------+-------+

→

--use_legacy_sql=false - Use standard SQL syntax instead of legacy SQL.

This command shows detailed metadata about the public dataset table, including schema and size, helping understand the data before querying.

Terminal

bq show --format=prettyjson bigquery-public-data:usa_names.usa_1910_2013

Expected Output

{

  "kind": "bigquery#table",

  "etag": "etagvalue",

  "id": "bigquery-public-data:usa_names.usa_1910_2013",

  "selfLink": "https://www.googleapis.com/bigquery/v2/projects/bigquery-public-data/datasets/usa_names/tables/usa_1910_2013",

  "tableReference": {

    "projectId": "bigquery-public-data",

    "datasetId": "usa_names",

    "tableId": "usa_1910_2013"

  },

  "schema": {

    "fields": [

      {"name": "state", "type": "STRING"},

      {"name": "gender", "type": "STRING"},

      {"name": "year", "type": "INTEGER"},

      {"name": "name", "type": "STRING"},

      {"name": "number", "type": "INTEGER"}

    ]

  },

  "numRows": "1000000",

  "creationTime": "1356998400000",

  "lastModifiedTime": "1672531200000",

  "type": "TABLE"

}

→

--format=prettyjson - Show output in readable JSON format.

This command performs a dry run to estimate how much data the query will scan without running it, helping control costs.

Terminal

bq query --dry_run --use_legacy_sql=false 'SELECT * FROM `bigquery-public-data.usa_names.usa_1910_2013` WHERE state = "CA";'

Expected Output

Query successfully validated. Assuming the tables are not modified, running this query will process 123456789 bytes of data.

→

--dry_run - Estimate query cost without running the query.

→

--use_legacy_sql=false - Use standard SQL syntax.

Key Concept

If you remember nothing else, remember: BigQuery charges based on the amount of data your SQL queries scan, so writing efficient queries saves money.

Common Mistakes

Running queries without estimating data scanned first.

This can lead to unexpectedly high costs if the query scans large amounts of data.

Use the --dry_run flag to estimate data scanned before running expensive queries.

Using legacy SQL syntax without setting the flag.

Queries may fail or behave unexpectedly because legacy SQL differs from standard SQL.

Always use --use_legacy_sql=false to run standard SQL queries.

Selecting all columns (*) when only a few are needed.

This increases the amount of data scanned and raises costs unnecessarily.

Select only the columns you need in your query.

Summary

Use the bq command-line tool to run SQL queries on BigQuery datasets.

Estimate query costs with --dry_run to avoid unexpected charges.

Use standard SQL syntax by setting --use_legacy_sql=false for better compatibility.

Practice

1. What does BigQuery charge you for when you run a SQL query?

easy

A. The amount of data scanned by the query

B. The number of SQL commands used

C. The time it takes to write the query

D. The number of tables in your project

Solution

1. Step 1: Understand BigQuery pricing basics

   BigQuery charges based on how much data your query scans, not on commands or tables.

2. Step 2: Identify the correct pricing factor

   The cost depends on the volume of data scanned during query execution.

3. Final Answer:

   The amount of data scanned by the query -> Option A

4. Quick Check:

   Data scanned = cost [OK]

Hint: Remember: BigQuery bills by data scanned, not query length [OK]

Common Mistakes:

• Thinking cost depends on query complexity
• Assuming storage size affects query cost directly
• Confusing number of tables with cost

2. Which of the following is the correct way to write a simple BigQuery SQL SELECT statement?

easy

A. SELECT * FROM dataset.table;

B. GET * FROM dataset.table;

C. FETCH ALL FROM dataset.table;

D. READ * dataset.table;

Solution

1. Step 1: Recall BigQuery SQL syntax

   BigQuery uses standard SQL syntax where SELECT is used to retrieve data.

2. Step 2: Identify the correct syntax

   Only SELECT * FROM dataset.table; uses the correct SELECT statement format with FROM clause.

3. Final Answer:

   SELECT * FROM dataset.table; -> Option A

4. Quick Check:

   SELECT + FROM = correct query [OK]

Hint: SELECT is the keyword to get data in SQL [OK]

Common Mistakes:

• Using GET or FETCH instead of SELECT
• Omitting FROM keyword
• Incorrect command verbs

3. Consider this query:
SELECT COUNT(*) FROM `project.dataset.sales` WHERE sale_date >= '2023-01-01';
What will this query return?

medium

A. A list of all sales records from 2023-01-01 onwards

B. The sum of all sales amounts after 2023-01-01

C. The total number of sales records from 2023-01-01 onwards

D. An error because of incorrect date format

Solution

1. Step 1: Analyze the SELECT clause

   COUNT(*) counts the number of rows matching the condition.

2. Step 2: Understand the WHERE clause

   It filters records with sale_date on or after 2023-01-01.

3. Final Answer:

   The total number of sales records from 2023-01-01 onwards -> Option C

4. Quick Check:

   COUNT(*) + WHERE = count filtered rows [OK]

Hint: COUNT(*) counts rows, not sums values [OK]

Common Mistakes:

• Confusing COUNT with SUM
• Thinking it returns all records instead of count
• Assuming date format causes error

4. You wrote this query but it fails:
SELECT * FROM dataset.table WHERE date = 2023-01-01;
What is the error and how to fix it?

medium

A. Missing semicolon; add ; at end

B. Date value missing quotes; fix with WHERE date = '2023-01-01'

C. Table name incorrect; use backticks around table

D. SELECT * not allowed; specify columns explicitly

Solution

1. Step 1: Identify the error in WHERE clause

   Date literals must be in quotes in SQL; missing quotes cause syntax error.

2. Step 2: Correct the date format

   Enclose date in single quotes: '2023-01-01' to fix the query.

3. Final Answer:

   Date value missing quotes; fix with WHERE date = '2023-01-01' -> Option B

4. Quick Check:

   Date literals need quotes [OK]

Hint: Always quote date strings in SQL WHERE clauses [OK]

Common Mistakes:

• Forgetting quotes around dates
• Assuming semicolon fixes syntax
• Thinking SELECT * is invalid

5. You want to reduce BigQuery costs by limiting data scanned. Which approach is best?

hard

A. Use SELECT * to get all columns for flexibility

B. Run queries without filters to scan all data quickly

C. Store data in multiple small tables instead of one large table

D. Use partitioned tables and filter on partition column in queries

Solution

1. Step 1: Understand cost drivers in BigQuery

   Costs depend on data scanned; scanning less data reduces cost.

2. Step 2: Identify cost-saving technique

   Partitioning tables lets queries scan only relevant partitions, saving cost.

3. Final Answer:

   Use partitioned tables and filter on partition column in queries -> Option D

4. Quick Check:

   Partition + filter = less data scanned [OK]

Hint: Partition tables and filter to scan less data [OK]

Common Mistakes:

• Thinking scanning all data is cheaper
• Splitting tables without partitioning helps cost
• Using SELECT * increases scanned data