Firestore queries and indexes in GCP - Deep Dive

Overview - Firestore queries and indexes

What is it?

Firestore is a cloud database that stores data in documents and collections. Queries let you find specific documents by asking questions about their fields. Indexes help Firestore find answers quickly by organizing data in a way that matches your questions. Without indexes, queries would be slow or impossible.

Why it matters

Without queries and indexes, finding data in a large database would be like searching for a book in a huge library without a catalog. This would make apps slow and frustrating. Firestore queries and indexes make data retrieval fast and efficient, improving user experience and saving computing resources.

Where it fits

Before learning Firestore queries and indexes, you should understand basic database concepts like documents and collections. After this, you can learn about Firestore security rules and data modeling to build secure and scalable apps.

Mental Model

Core Idea

Firestore uses indexes to organize data so queries can quickly find matching documents without scanning everything.

Think of it like...

Imagine a library where books are arranged by author and topic. When you want a book on gardening by a certain author, you go straight to the right shelf instead of searching every book. Firestore indexes are like those shelves, and queries are like your search requests.

┌─────────────┐       ┌─────────────┐       ┌─────────────┐
│ Collection  │──────▶│ Index Table │──────▶│ Query Result│
└─────────────┘       └─────────────┘       └─────────────┘
       │                    ▲                      ▲
       │                    │                      │
       └───── Documents ────┘                      │
                                                  │
                                        Query uses index to
                                        find matching documents

Build-Up - 7 Steps

1

Foundation

Understanding Firestore Data Model

Concept: Learn what documents and collections are in Firestore.

Firestore stores data in documents, which are like records with fields. Documents are grouped into collections, which are like folders. Each document has a unique ID within its collection. Collections can contain many documents, and documents can contain subcollections.

Result

You can organize data in a tree-like structure with collections and documents.

Understanding the data model is essential because queries and indexes operate on documents within collections.

2

Foundation

Basics of Firestore Queries

Concept: Learn how to ask Firestore for documents matching certain conditions.

Queries let you find documents by specifying conditions on fields, like 'age > 20' or 'status == active'. You can combine conditions with AND logic. Firestore returns documents that match all conditions. Simple queries can filter by one field or order results.

Result

You can retrieve only the documents you need instead of all documents.

Queries let you focus on relevant data, which is crucial for app performance and user experience.

3

Intermediate

Role of Indexes in Firestore

Concept: Indexes speed up queries by pre-sorting data based on fields.

Firestore automatically creates single-field indexes for each field. These indexes let Firestore quickly find documents matching simple queries. For queries with multiple conditions or ordering on multiple fields, Firestore uses composite indexes. Without the right index, Firestore cannot run the query efficiently.

Result

Queries run fast because Firestore uses indexes instead of scanning all documents.

Knowing that indexes are behind query speed helps you understand why some queries need extra setup.

4

Intermediate

Composite Indexes for Complex Queries

🤔Before reading on: do you think Firestore can run any query without extra indexes? Commit to yes or no.

Concept: Composite indexes combine multiple fields to support complex queries.

When a query filters or sorts on multiple fields, Firestore needs a composite index that includes those fields in the right order. Firestore shows an error with a link to create the needed index if it doesn't exist. You can create composite indexes manually in the Firebase console or automatically via error links.

Result

Complex queries become possible and efficient with composite indexes.

Understanding composite indexes prevents confusion when queries fail and teaches how to fix them.

5

Intermediate

Limitations on Query Operators

🤔Before reading on: can Firestore queries combine 'OR' conditions on different fields? Commit to yes or no.

Concept: Firestore supports certain query operators and has rules on combining them.

Firestore supports operators like ==, <, <=, >, >=, array-contains, and in. However, it does not support OR queries across different fields directly. Queries must use AND logic on multiple conditions. To perform OR-like queries, you combine multiple queries in your app code.

Result

You know which queries Firestore can run natively and which need workarounds.

Knowing operator limits helps design queries that work and avoid unexpected errors.

6

Advanced

Indexing and Query Performance Tradeoffs

🤔Before reading on: do you think adding more indexes always improves performance? Commit to yes or no.

Concept: Indexes speed queries but add storage and write costs.

Each index consumes storage space and slows down writes because Firestore updates indexes on every document change. Over-indexing can increase costs and reduce write speed. It's best to create only needed indexes and remove unused ones. Monitoring index usage helps optimize performance and cost.

Result

You balance query speed with cost and write performance by managing indexes wisely.

Understanding tradeoffs prevents blindly adding indexes that hurt overall app performance.

7

Expert

Behind the Scenes: Firestore Query Execution

🤔Before reading on: do you think Firestore scans all documents for queries without indexes? Commit to yes or no.

Concept: Firestore uses indexes to avoid scanning all documents and merges index results for complex queries.

When you run a query, Firestore looks up matching entries in the relevant index. For composite indexes, it uses a sorted structure to quickly find matching documents. If no index exists, Firestore rejects the query instead of scanning all documents to keep performance predictable. Firestore merges index results when combining filters and orders.

Result

Queries run efficiently by using indexes as shortcuts to data.

Knowing Firestore never scans all documents explains why indexes are mandatory for many queries and why query failures happen.

Under the Hood

Firestore stores indexes as sorted lists of document references keyed by field values. When a query runs, Firestore uses these sorted indexes to quickly locate matching documents by performing range scans or lookups. Composite indexes combine multiple fields in a defined order to support complex queries. Firestore updates indexes asynchronously but ensures consistency. Queries without matching indexes are rejected to avoid slow full scans.

Why designed this way?

Firestore was designed for scalability and low latency. Using indexes avoids scanning large datasets, which would be slow and costly. Automatic single-field indexes cover most queries, while composite indexes let developers optimize complex queries. Rejecting queries without indexes enforces good performance practices. This design balances speed, cost, and developer control.

┌─────────────┐       ┌───────────────┐       ┌───────────────┐
│ Documents   │──────▶│ Index Storage │──────▶│ Query Engine  │
│ (Data Store)│       │ (Sorted Keys) │       │ (Uses Indexes)│
└─────────────┘       └───────────────┘       └───────────────┘
       ▲                      │                      │
       │                      │                      │
       └───── Updates ─────────┘                      │
                                                      │
                                            Reject query if
                                            no matching index

Myth Busters - 4 Common Misconceptions

Quick: can Firestore run any query without an index? Commit yes or no.

Common Belief: Firestore can run all queries even without indexes by scanning documents.

Reality: Firestore rejects queries that need indexes not yet created to avoid slow scans.

Why it matters: Assuming queries run without indexes leads to unexpected errors and confusion.

Quick: does adding more indexes always make queries faster? Commit yes or no.

Common Belief: More indexes always improve query speed.

Reality: Too many indexes slow down writes and increase storage costs, hurting overall performance.

Why it matters: Over-indexing can degrade app responsiveness and increase expenses.

Quick: can Firestore queries combine OR conditions on different fields? Commit yes or no.

Common Belief: Firestore supports OR queries across multiple fields natively.

Reality: Firestore does not support OR queries across fields; you must run multiple queries and merge results in code.

Why it matters: Expecting OR support causes query design errors and runtime failures.

Quick: are Firestore indexes only for speeding up queries? Commit yes or no.

Common Belief: Indexes only affect query speed and have no other impact.

Reality: Indexes also affect write speed and storage costs because they update on every document change.

Why it matters: Ignoring write cost impact leads to inefficient app design and unexpected billing.

Expert Zone

1

Composite indexes must list fields in the exact order queries use filters and sorting; wrong order causes query failures.

2

Firestore automatically creates single-field indexes but requires manual composite index creation, which can be automated via error links.

3

Firestore's index updates are eventually consistent but guarantee strong consistency for queries, balancing speed and correctness.

When NOT to use

Avoid complex composite indexes when your app can denormalize data or use simpler queries. For OR queries, use multiple queries and merge results in code. If write performance is critical, minimize indexes or consider other databases optimized for heavy writes.

Production Patterns

In production, developers monitor index usage and remove unused indexes to save cost. They design queries to use existing indexes and create composite indexes only when needed. Apps often handle OR logic by running parallel queries and merging results client-side.

Connections

Database Indexing

Firestore indexes are a cloud-managed form of database indexing.

Understanding traditional database indexes helps grasp how Firestore organizes data for fast queries.

Caching Systems

Both caching and Firestore indexes speed up data retrieval by avoiding full data scans.

Knowing caching principles clarifies why pre-organizing data (indexing) improves performance.

Library Catalog Systems

Firestore indexes function like library catalogs that organize books for quick lookup.

Recognizing this connection helps appreciate the importance of indexes in managing large data collections.

Common Pitfalls

#1 Query fails because required composite index is missing.

Wrong approach:

db.collection('users').where('age', '>', 20).where('status', '==', 'active').get() // No composite index created

Correct approach:

Create composite index for fields 'age' and 'status' in Firebase console or follow error link, then run query.

Root cause:

Not creating composite indexes for multi-field queries causes Firestore to reject them.

#2 Trying to run OR queries directly in Firestore.

Wrong approach:

db.collection('users').where('status', '==', 'active').orWhere('age', '<', 18).get() // Invalid syntax

Correct approach:

Run two separate queries and merge results in app code:
const activeUsers = await db.collection('users').where('status', '==', 'active').get();
const youngUsers = await db.collection('users').where('age', '<', 18).get();
Merge activeUsers and youngUsers results.

Root cause:

Firestore does not support OR queries across fields natively.

#3 Adding indexes for every possible field combination without need.

Wrong approach:

Creating composite indexes for all field pairs regardless of query usage.

Correct approach:

Create indexes only for queries your app actually runs, monitor usage, and remove unused indexes.

Root cause:

Misunderstanding that more indexes always improve performance leads to over-indexing.

Key Takeaways

Firestore queries find documents by filtering and sorting fields within collections.

Indexes organize data to make queries fast and efficient, avoiding full data scans.

Single-field indexes are automatic; composite indexes must be created for multi-field queries.

Firestore rejects queries without required indexes to maintain performance guarantees.

Balancing index creation is key to optimizing both query speed and write costs.