Bigtable for time-series data in GCP - Commands & Configuration

Introduction

Time-series data is a sequence of data points collected over time. Bigtable is a Google Cloud database designed to store large amounts of this data efficiently. It helps you save and quickly access time-stamped information like sensor readings or stock prices.

When you want to store temperature readings from thousands of sensors every minute.

When you need to track user activity logs over days or months for analysis.

When you want to record financial market data that updates every second.

When you need a database that can handle large volumes of time-stamped data with fast reads and writes.

When you want to analyze trends over time without delays.

Config File - bigtable-instance-config.yaml

bigtable-instance-config.yaml

apiVersion: bigtableadmin.googleapis.com/v2
kind: Instance
metadata:
  name: example-time-series-instance
spec:
  displayName: "Example Time Series Instance"
  type: PRODUCTION
  clusters:
  - clusterId: example-cluster
    zone: us-central1-b
    serveNodes: 3
    defaultStorageType: SSD
  labels:
    environment: development

This file creates a Bigtable instance named example-time-series-instance with one cluster in the us-central1-b zone. It uses 3 nodes for serving data and SSD storage for fast access. Labels help organize and identify the instance.

Commands

This command creates a Bigtable instance with one cluster. It sets the name, location, number of nodes, and storage type to handle time-series data efficiently.

Terminal

gcloud bigtable instances create example-time-series-instance --cluster=example-cluster --cluster-zone=us-central1-b --display-name="Example Time Series Instance" --cluster-num-nodes=3 --cluster-storage-type=ssd

Expected Output

Created Bigtable instance [example-time-series-instance].

→

--cluster-num-nodes - Sets the number of nodes to serve data, affecting performance.

→

--cluster-storage-type - Chooses SSD for faster data access.

This command lists all Bigtable instances in your project to verify the instance was created successfully.

Terminal

gcloud bigtable instances list

Expected Output

NAME                         DISPLAY_NAME                 TYPE        LABELS

example-time-series-instance  Example Time Series Instance PRODUCTION environment=development

This command creates a table named 'time_series_table' in the Bigtable instance to store time-series data.

Terminal

cbt createtable time_series_table

Expected Output

Created table time_series_table

This command creates a split point in the table to help Bigtable organize data by time, improving read and write speed.

Terminal

cbt createtablesplit time_series_table 20240101000000

Expected Output

Created split at 20240101000000

This command looks up a specific row in the time-series table to check if data is stored correctly.

Terminal

cbt lookup time_series_table row-key-20240101000000

Expected Output

Row key: row-key-20240101000000

Column family: metrics

  Column: temperature

    Timestamp: 2024-01-01T00:00:00.000Z

    Value: 22.5

Key Concept

If you remember nothing else from this pattern, remember: Bigtable organizes time-series data by row keys that include timestamps to enable fast reads and writes over large datasets.

Common Mistakes

Using simple incremental row keys without timestamps

This causes data hotspots and slow queries because Bigtable stores rows in sorted order by key.

Include timestamps in row keys in a way that distributes writes evenly, such as reversing timestamps or adding prefixes.

Creating too few nodes for the cluster

This limits performance and can cause slow data processing under load.

Start with at least 3 nodes for production workloads and scale as needed.

Not creating table splits for large time ranges

Without splits, Bigtable can become slow because all data is stored in one region.

Create splits at logical time boundaries to distribute data evenly.

Summary

Create a Bigtable instance with enough nodes and SSD storage for fast time-series data handling.

Create tables and use row keys with timestamps to organize data efficiently.

Use table splits to improve performance by distributing data across regions.