Global Secondary Index (GSI) concept in DynamoDB - Deep Dive

Overview - Global Secondary Index (GSI) concept

What is it?

A Global Secondary Index (GSI) in DynamoDB is a way to create an alternate view of your data with a different key structure. It lets you query the table using different attributes than the main primary key. This helps you find data quickly in ways the original table design might not support. GSIs are separate from the main table but stay updated automatically.

Why it matters

Without GSIs, you can only efficiently query data by the table's primary key, which limits how you access your data. GSIs solve this by letting you search using other attributes, making your app faster and more flexible. Without GSIs, you might have to scan the whole table, which is slow and costly.

Where it fits

Before learning GSIs, you should understand DynamoDB tables, primary keys, and basic querying. After GSIs, you can explore Local Secondary Indexes (LSIs), query optimization, and advanced data modeling in DynamoDB.

Mental Model

Core Idea

A Global Secondary Index is like a separate, automatically updated shortcut that lets you find data using different keys than the main table.

Think of it like...

Imagine a library where books are organized by author (main table). A GSI is like a separate catalog organized by genre, letting you find books by genre quickly without rearranging the whole library.

Main Table (Primary Key: UserID)
┌─────────────┐
│ UserID (PK) │
│ Name        │
│ Email       │
│ City        │
└─────────────┘

Global Secondary Index (GSI) (Partition Key: City)
┌───────────────┐
│ City (GSI PK) │
│ UserID        │
│ Name          │
└───────────────┘

Query main table by UserID or query GSI by City.

Build-Up - 7 Steps

1

Foundation

Understanding DynamoDB Primary Keys

Concept: Learn what a primary key is and how it uniquely identifies items in a DynamoDB table.

In DynamoDB, every item in a table must have a primary key. This key can be simple (one attribute) or composite (partition key + sort key). The primary key ensures each item is unique and helps DynamoDB find items quickly.

Result

You know how DynamoDB organizes data and why the primary key is essential for fast lookups.

Understanding primary keys is crucial because GSIs build on this concept by creating alternate keys for querying.

2

Foundation

Basics of Querying in DynamoDB

Concept: Learn how queries use primary keys to find data efficiently.

A query in DynamoDB uses the primary key to quickly find matching items. If you query by an attribute that is not part of the primary key, DynamoDB must scan the whole table, which is slow and expensive.

Result

You see why querying by primary key is fast and why querying by other attributes is slow without indexes.

Knowing this limitation sets the stage for why GSIs are needed to improve query flexibility.

3

Intermediate

Introducing Global Secondary Indexes

🤔Before reading on: do you think GSIs store copies of data or just pointers? Commit to your answer.

Concept: GSIs create a new index with a different key structure, storing copies of selected attributes to enable fast queries on those keys.

A GSI has its own partition key and optional sort key, different from the main table. It stores copies of attributes you choose, so you can query the table using these alternate keys without scanning the whole table.

Result

You can query data efficiently using the GSI keys, even if they are different from the main table's keys.

Understanding that GSIs store copies of data explains why they improve query speed but also affect storage and write costs.

4

Intermediate

How GSIs Stay Updated Automatically

🤔Before reading on: do you think updating a GSI requires manual syncing? Commit to your answer.

Concept: DynamoDB automatically updates GSIs when the main table changes, keeping the index in sync without extra work from you.

When you add, update, or delete items in the main table, DynamoDB updates the GSI behind the scenes. This means your queries on the GSI always see the latest data without manual intervention.

Result

Your GSI queries return fresh data automatically, simplifying application logic.

Knowing this automatic syncing helps you trust GSIs for real-time queries and reduces complexity in your code.

5

Intermediate

Choosing Attributes for a GSI

🤔Before reading on: do you think GSIs copy all table attributes by default? Commit to your answer.

Concept: You select which attributes the GSI copies to balance query needs and storage costs.

GSIs only copy the attributes you specify in the projection. You can choose to copy all attributes, only keys, or a subset. This choice affects storage size and query performance.

Result

You optimize GSIs to include only needed data, saving costs and improving speed.

Understanding attribute projection helps you design efficient GSIs tailored to your query patterns.

6

Advanced

Impact of GSIs on Write Capacity and Costs

🤔Before reading on: do you think writing to a table with GSIs costs the same as without? Commit to your answer.

Concept: Writes to a table with GSIs consume additional write capacity because DynamoDB must update the index too.

Every time you write to the main table, DynamoDB also writes to the GSI. This means your write capacity units (WCUs) are used for both the table and each GSI, increasing costs and requiring capacity planning.

Result

You understand that GSIs add overhead to writes and must be managed carefully.

Knowing this prevents unexpected cost spikes and helps you balance read and write needs.

7

Expert

Consistency and Latency Considerations with GSIs

🤔Before reading on: do you think GSI queries always return the latest data immediately? Commit to your answer.

Concept: GSI queries are eventually consistent, meaning there can be a slight delay before updates appear in the index.

Because GSIs update asynchronously, queries on a GSI might not reflect the most recent writes instantly. This eventual consistency is a tradeoff for performance and scalability.

Result

You know to design your application to handle slight delays in GSI query results.

Understanding eventual consistency helps avoid bugs and data confusion in real-world applications.

Under the Hood

When you write to a DynamoDB table, the service writes the item to the main storage and simultaneously updates any GSIs asynchronously. Each GSI maintains its own partition and sort keys and stores copies of projected attributes. The update to GSIs happens in the background, allowing the main write operation to complete quickly. Queries on GSIs use their own keys and storage structures, separate from the main table, enabling fast lookups on alternate keys.

Why designed this way?

GSIs were designed to provide flexible querying without redesigning the main table or duplicating data manually. The asynchronous update model balances write performance and index freshness, avoiding slow writes. Storing copies of attributes in GSIs allows queries without fetching the main table, improving speed. Alternatives like scanning the table were too slow and costly, so GSIs offer a scalable solution.

┌───────────────┐       ┌───────────────────────────────┐
│ Main Table    │       │ Global Secondary Index (GSI)   │
│ Partition Key │◄──────┤ Partition Key (different)      │
│ Sort Key      │       │ Sort Key (optional)            │
│ Attributes    │       │ Projected Attributes           │
└─────┬─────────┘       └───────────────┬───────────────┘
      │ Writes                    ▲ Updates asynchronously
      │                          │
      ▼                          │
┌─────────────────────────────────────────────┐
│ DynamoDB Storage and Indexing Engine         │
└─────────────────────────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do GSIs always return the latest data immediately after a write? Commit to yes or no.

Common Belief: GSIs are always up-to-date and return the latest data instantly.

Reality: GSIs update asynchronously, so queries may return slightly stale data for a short time.

Why it matters: Assuming immediate consistency can cause bugs where your app shows outdated information or misses recent changes.

Quick: Do GSIs reduce write costs because they are separate indexes? Commit to yes or no.

Common Belief: Using GSIs does not affect write costs since they are separate from the main table.

Reality: Writes to the main table also consume write capacity for each GSI, increasing costs.

Why it matters: Ignoring this leads to unexpected high bills and capacity throttling in production.

Quick: Do GSIs automatically include all attributes from the main table? Commit to yes or no.

Common Belief: GSIs copy all attributes from the main table by default.

Reality: You must specify which attributes to project; otherwise, only keys are copied.

Why it matters: Not projecting needed attributes can cause incomplete query results or require extra fetches.

Quick: Can you create a GSI with the same partition key as the main table? Commit to yes or no.

Common Belief: GSIs must have different partition keys than the main table.

Reality: GSIs can have the same partition key attribute but usually differ to enable new query patterns.

Why it matters: Misunderstanding this limits your design options and flexibility.

Expert Zone

1

GSIs consume additional write capacity units proportional to the size of projected attributes, so careful projection design can optimize costs.

2

The eventual consistency of GSIs means that in high-write scenarios, queries might miss recent updates, requiring application-level handling or fallback strategies.

3

GSIs have their own throughput limits separate from the main table, so heavy GSI queries can throttle independently, affecting performance.

When NOT to use

Avoid GSIs when you need strictly consistent reads or when write throughput is extremely high and cost-sensitive. Instead, consider Local Secondary Indexes (LSIs) for consistent queries on the same partition key or redesign your data model to fit primary key queries.

Production Patterns

In production, GSIs are used to support multiple query patterns without duplicating tables. For example, an e-commerce app might use the main table keyed by OrderID and a GSI keyed by CustomerID to quickly find all orders by a customer. Monitoring GSI usage and capacity is critical to avoid throttling and control costs.

Connections

Database Indexing

GSIs are a type of database index specialized for NoSQL key-value stores.

Understanding traditional database indexes helps grasp how GSIs speed up queries by creating alternate data access paths.

Eventual Consistency in Distributed Systems

GSIs exhibit eventual consistency similar to many distributed data stores.

Knowing eventual consistency concepts from distributed systems explains why GSI queries might lag behind writes.

Library Catalog Systems

GSIs function like alternate catalogs in libraries, enabling different ways to find books.

Recognizing this connection helps appreciate how GSIs provide flexible data access without reorganizing the main data.

Common Pitfalls

#1 Querying a GSI expecting strongly consistent results.

Wrong approach:

Query the GSI with ConsistentRead=true expecting immediate latest data.

Correct approach:

Query the GSI with ConsistentRead=false and design your app to handle eventual consistency.

Root cause:

Misunderstanding that GSIs only support eventually consistent reads.

#2 Creating a GSI without projecting needed attributes.

Wrong approach:

Create GSI with ProjectionType=KEYS_ONLY but query attributes not included in keys.

Correct approach:

Create GSI with ProjectionType=INCLUDE or ALL to include required attributes for queries.

Root cause:

Not realizing GSIs only copy attributes you specify, leading to incomplete query results.

#3 Ignoring increased write capacity when adding GSIs.

Wrong approach:

Add multiple GSIs but keep write capacity units unchanged, causing throttling.

Correct approach:

Increase write capacity units to account for GSI overhead or use on-demand mode.

Root cause:

Underestimating the write cost impact of GSIs.

Key Takeaways

Global Secondary Indexes let you query DynamoDB tables using alternate keys for flexible data access.

GSIs store copies of selected attributes and update asynchronously, enabling fast but eventually consistent queries.

Using GSIs increases write costs and capacity needs because updates must be written to both the main table and indexes.

Careful design of GSI keys and projected attributes balances query performance, cost, and storage.

Understanding GSIs' eventual consistency and capacity impact is essential for building reliable, scalable applications.

Practice

1. What is the main purpose of a Global Secondary Index (GSI) in DynamoDB?

easy

A. To allow queries using alternate keys different from the primary key

B. To store backup copies of the table data

C. To increase the size limit of the table

D. To encrypt data automatically

Solution

1. Step 1: Understand what a GSI does

   A GSI lets you query the table using different keys than the main primary key.

2. Step 2: Compare options to GSI purpose

   Options A, C, and D describe unrelated features like backup, size, or encryption, which are not GSI functions.

3. Final Answer:

   To allow queries using alternate keys different from the primary key -> Option A

4. Quick Check:

   GSI = alternate query keys [OK]

Hint: GSI lets you search by different keys, not backup or encryption [OK]

Common Mistakes:

• Confusing GSI with backups
• Thinking GSI increases table size
• Assuming GSI encrypts data

2. Which of the following is the correct way to define a Global Secondary Index (GSI) in a DynamoDB table creation JSON?

easy

A. {
"GlobalSecondaryIndexes": [
{
"IndexName": "GSI1",
"KeySchema": [
{"AttributeName": "GSIKey", "KeyType": "RANGE"}
],
"Projection": {"ProjectionType": "ALL"}
}
]
}

B. {
"SecondaryIndexes": [
{
"IndexName": "GSI1",
"KeySchema": [
{"AttributeName": "GSIKey", "KeyType": "RANGE"}
],
"Projection": {"ProjectionType": "KEYS_ONLY"}
}
]
}

C. {
"GlobalSecondaryIndexes": {
"IndexName": "GSI1",
"KeySchema": [
{"AttributeName": "GSIKey", "KeyType": "HASH"}
],
"Projection": {"ProjectionType": "ALL"}
}
}

D. {
"GlobalSecondaryIndexes": [
{
"IndexName": "GSI1",
"KeySchema": [
{"AttributeName": "GSIKey", "KeyType": "HASH"},
{"AttributeName": "SortKey", "KeyType": "RANGE"}
],
"Projection": {"ProjectionType": "INCLUDE", "NonKeyAttributes": ["Attr1"]}
}
]
}

Solution

1. Step 1: Check the correct JSON structure for GSI

   The GlobalSecondaryIndexes must be an array of objects, each with IndexName, KeySchema, and Projection.

2. Step 2: Validate key schema and projection

   { "GlobalSecondaryIndexes": [ { "IndexName": "GSI1", "KeySchema": [ {"AttributeName": "GSIKey", "KeyType": "HASH"}, {"AttributeName": "SortKey", "KeyType": "RANGE"} ], "Projection": {"ProjectionType": "INCLUDE", "NonKeyAttributes": ["Attr1"]} } ] } correctly uses an array, has both HASH and RANGE keys, and a valid projection with included attributes.

3. Final Answer:

   Correct GSI JSON with array, keys, and projection -> Option D

4. Quick Check:

   GSI JSON array with keys and projection = A [OK]

Hint: GSI must be an array with keys and projection defined [OK]

Common Mistakes:

• Using object instead of array for GlobalSecondaryIndexes
• Wrong key types or missing HASH key
• Incorrect property names like SecondaryIndexes

3. Given a DynamoDB table with a GSI on attribute Category (HASH key) and Price (RANGE key), what will the following query return?

Query({ IndexName: 'CategoryPriceIndex', KeyConditionExpression: 'Category = :cat and Price > :minPrice', ExpressionAttributeValues: { ':cat': 'Books', ':minPrice': 20 } })

medium

A. All items where Price is greater than 20 regardless of Category

B. All items where Category is 'Books' or Price is greater than 20

C. All items where Category is 'Books' and Price is greater than 20

D. Syntax error because Price cannot be used in KeyConditionExpression

Solution

1. Step 1: Understand KeyConditionExpression with GSI keys

   The query uses the GSI's HASH key 'Category' with equality and RANGE key 'Price' with a greater than condition, which is valid.

2. Step 2: Interpret the query filter

   The query returns items where Category equals 'Books' AND Price is greater than 20, matching the GSI keys.

3. Final Answer:

   All items where Category is 'Books' and Price is greater than 20 -> Option C

4. Quick Check:

   GSI query with HASH and RANGE keys = All items where Category is 'Books' and Price is greater than 20 [OK]

Hint: GSI query uses HASH = and RANGE with conditions correctly [OK]

Common Mistakes:

• Confusing AND with OR in KeyConditionExpression
• Thinking Price can't be in KeyConditionExpression
• Assuming query returns all items ignoring Category

4. You created a GSI with IndexName: 'UserEmailIndex' on attribute Email as HASH key. Your query on this GSI returns no results even though items with matching emails exist. What is the most likely cause?

medium

A. The GSI was created after the items were inserted and has not yet indexed them

B. You used the primary key instead of the GSI key in the query

C. The Email attribute is not projected to the GSI

D. DynamoDB does not support querying on GSIs

Solution

1. Step 1: Understand GSI indexing timing

   When a GSI is created after data insertion, it takes time to index existing items. Queries may return empty until indexing completes.

2. Step 2: Evaluate other options

   You used the primary key instead of the GSI key in the query is about query keys but question states query is on GSI. The Email attribute is not projected to the GSI is invalid because HASH key attributes must be projected automatically. DynamoDB does not support querying on GSIs is false; DynamoDB supports GSI queries.

3. Final Answer:

   The GSI was created after the items were inserted and has not yet indexed them -> Option A

4. Quick Check:

   New GSI indexing delay causes empty query [OK]

Hint: New GSI needs time to index existing data before queries work [OK]

Common Mistakes:

• Assuming GSI queries always return instantly after creation
• Confusing projection with key attributes
• Believing DynamoDB doesn't support GSI queries

5. You have a DynamoDB table with a primary key UserID (HASH) and a GSI named OrderDateIndex with OrderDate as HASH and Status as RANGE key. You want to efficiently find all orders placed on '2023-05-01' with status 'Shipped'. Which query approach is best?

hard

A. Use a Scan on the GSI with FilterExpression for OrderDate and Status

B. Query the OrderDateIndex with KeyConditionExpression 'OrderDate = :date and Status = :status'

C. Query the main table with KeyConditionExpression 'UserID = :user and Status = :status'

D. Scan the table filtering for OrderDate and Status attributes

Solution

1. Step 1: Identify the best index for the query

   The GSI OrderDateIndex has OrderDate as HASH and Status as RANGE key, matching the query conditions.

2. Step 2: Choose efficient query method

   Querying the GSI with KeyConditionExpression on HASH and RANGE keys is efficient and uses indexes. Scans are less efficient and should be avoided.

3. Final Answer:

   Query the OrderDateIndex with KeyConditionExpression 'OrderDate = :date and Status = :status' -> Option B

4. Quick Check:

   Use GSI query with HASH and RANGE keys for best performance [OK]

Hint: Query GSI with HASH and RANGE keys, avoid scans for efficiency [OK]

Common Mistakes:

• Using Scan instead of Query causing slow performance
• Querying main table with wrong keys
• Filtering after Scan instead of using KeyConditionExpression