Scan vs query performance in AWS - Trade-offs & Expert Analysis

Overview - Scan vs query performance

What is it?

Scan and query are two ways to read data from a database table. Scan reads every item in the table, checking each one to find matches. Query looks for items using specific keys, so it reads less data. Both are used to get information, but they work differently and affect speed.

Why it matters

Choosing between scan and query affects how fast your app gets data and how much it costs. If you use scan on big tables often, your app slows down and costs rise. Using query smartly makes your app faster and cheaper. Without understanding this, apps can be slow and expensive.

Where it fits

You should know basic database concepts like tables and keys before this. After this, you can learn about indexing, filtering, and optimizing database access for better performance.

Mental Model

Core Idea

Query finds data by looking directly where it should be, while scan checks every item one by one.

Think of it like...

Imagine finding a book in a library: query is like going straight to the shelf where the book belongs, scan is like checking every book in the entire library until you find it.

┌─────────────┐       ┌─────────────┐
│   Query     │       │    Scan     │
├─────────────┤       ├─────────────┤
│ Uses keys   │       │ Checks all  │
│ Direct hit  │       │ items       │
│ Fast        │       │ Slow        │
└─────┬───────┘       └─────┬───────┘
      │                     │       
      ▼                     ▼       
  Reads few items       Reads whole table

Build-Up - 7 Steps

1

Foundation

Understanding database tables and items

Concept: Learn what a database table and items are.

A database table is like a spreadsheet with rows and columns. Each row is an item with data in columns. For example, a table of users has items with name, email, and ID.

Result

You can picture data stored in rows and columns, ready to be read or searched.

Knowing the basic structure of data storage helps understand how reading methods work.

2

Foundation

What are keys in databases?

Concept: Keys uniquely identify items and help find them quickly.

A key is like a label or ID for each item. For example, a user ID uniquely identifies a user. Keys let the database find items without looking at everything.

Result

You understand that keys are shortcuts to find data fast.

Recognizing keys as shortcuts is essential to grasp why queries are faster than scans.

3

Intermediate

How query uses keys to find data

🤔Before reading on: do you think query reads the whole table or only parts? Commit to your answer.

Concept: Query uses keys to directly access matching items without scanning all data.

When you query, you tell the database the key value you want. The database goes straight to that item or group of items with that key. It skips everything else.

Result

Query returns only the items that match the key, quickly and efficiently.

Understanding query's direct access explains why it is faster and cheaper than scanning.

4

Intermediate

How scan reads all items in a table

🤔Before reading on: do you think scan checks some or all items? Commit to your answer.

Concept: Scan reads every item in the table to find matches, ignoring keys.

Scan looks at each item one by one, checking if it matches your filter. It does not use keys, so it reads the whole table even if only a few items match.

Result

Scan returns all matching items but takes longer and uses more resources.

Knowing scan reads everything helps understand why it is slower and costlier.

5

Intermediate

Comparing performance and cost of scan vs query

🤔Before reading on: which do you think costs more and why? Commit to your answer.

Concept: Query is faster and cheaper because it reads less data; scan is slower and more expensive because it reads everything.

Query reads only the needed items using keys, so it uses less processing and costs less. Scan reads the whole table, which takes more time and costs more, especially on big tables.

Result

You can predict which method to use based on table size and query needs.

Understanding cost and speed differences guides better database design and usage.

6

Advanced

When to use scan despite its cost

🤔Before reading on: do you think scan is ever the best choice? Commit to your answer.

Concept: Scan is useful when you need to read all data or when keys don't help find what you want.

If you want to process or analyze every item, scan is necessary. Also, if your query conditions don't use keys, scan is the only option. You can reduce scan cost by limiting data or using filters.

Result

You know when scan is appropriate and how to use it wisely.

Recognizing scan's role prevents misuse and helps balance performance and completeness.

7

Expert

Optimizing query and scan with indexes and filters

🤔Before reading on: do you think indexes affect scan performance? Commit to your answer.

Concept: Indexes speed up queries by providing alternate keys; filters reduce data returned in scans.

Secondary indexes let queries use keys other than the main one, making queries flexible and fast. Filters can be applied after scan or query to reduce returned data but do not reduce read cost for scans. Understanding how indexes and filters work helps optimize performance.

Result

You can design tables and queries that minimize cost and maximize speed.

Knowing how indexes and filters interact with scan and query unlocks advanced optimization strategies.

Under the Hood

Query uses the database's key-value store to jump directly to the data location using the key's hash or sorted order. Scan reads every data block sequentially, checking each item against filters. Query reads fewer data blocks and uses less I/O, while scan reads all blocks, causing more I/O and CPU usage.

Why designed this way?

Databases separate query and scan to balance speed and flexibility. Query is optimized for known keys to be fast. Scan exists to handle cases where keys don't help or full data processing is needed. This design avoids forcing all reads to be slow or all reads to require keys.

┌───────────────┐       ┌───────────────┐
│   Query       │       │    Scan       │
├───────────────┤       ├───────────────┤
│ Uses key index│       │ Reads all data│
│ Direct access │       │ Sequentially  │
│ Few I/O ops   │       │ Many I/O ops  │
└───────┬───────┘       └───────┬───────┘
        │                       │       
        ▼                       ▼       
  Fast response          Slower response

Myth Busters - 4 Common Misconceptions

Quick: Does query always return data faster than scan? Commit to yes or no.

Common Belief: Query is always faster than scan no matter what.

Reality: Query is faster only when you have the right key and indexes; if you query without keys or on unindexed attributes, it can be slower or impossible.

Why it matters: Assuming query is always faster can lead to wrong design and failed queries.

Quick: Does scan read less data if you use filters? Commit to yes or no.

Common Belief: Using filters with scan reduces the amount of data read from the database.

Reality: Filters only reduce the data returned, but scan still reads every item, so the read cost and time remain high.

Why it matters: Thinking filters reduce scan cost can cause unexpected high bills and slow performance.

Quick: Can scan be faster than query in some cases? Commit to yes or no.

Common Belief: Scan is always slower than query.

Reality: If the query uses inefficient keys or returns many items, and scan is limited or parallelized, scan can sometimes be faster.

Why it matters: Knowing this helps choose the right method for specific workloads.

Quick: Does adding more indexes always improve query speed? Commit to yes or no.

Common Belief: More indexes always make queries faster.

Reality: Too many indexes slow down writes and increase storage; indexes must be chosen carefully.

Why it matters: Mismanaging indexes can degrade overall database performance.

Expert Zone

1

Query performance depends heavily on how well the partition key distributes data; uneven keys cause hotspots.

2

Scan can be parallelized across partitions to improve speed but increases cost and complexity.

3

Filters applied after query or scan do not reduce read capacity units consumed; only key conditions do.

When NOT to use

Avoid scan on large tables when you can use query with proper keys or indexes. Use analytics tools or data pipelines for full data processing instead of scans. For real-time lookups, always prefer query.

Production Patterns

Use query for user-specific data retrieval with partition keys. Use scan for batch jobs like backups or analytics with parallel scans. Combine query with secondary indexes for flexible access patterns. Monitor read capacity and optimize keys to avoid throttling.

Connections

Indexing in Databases

Builds-on

Understanding scan vs query helps grasp why indexes exist and how they speed up data access.

Caching Systems

Similar pattern

Like query uses keys to find data fast, caches use keys to avoid scanning all data, improving speed.

Library Book Search

Opposite approach

Knowing how physical search in libraries works clarifies why direct access (query) is faster than checking every book (scan).

Common Pitfalls

#1 Using scan for frequent user lookups.

Wrong approach:

Performing scan operations to find user data by email every time a user logs in.

Correct approach:

Design the table with email as a key or use a secondary index and perform query operations for user lookups.

Root cause:

Not understanding that scan reads the whole table, causing slow and costly lookups.

#2 Relying on filters to reduce scan cost.

Wrong approach:

Using scan with filters expecting to lower read capacity usage and cost.

Correct approach:

Use query with key conditions to limit data read; filters only reduce returned data, not read cost.

Root cause:

Misunderstanding that filters apply after reading all data in scan.

#3 Adding too many indexes to speed queries.

Wrong approach:

Creating multiple secondary indexes on all attributes without planning.

Correct approach:

Create only necessary indexes based on query patterns to balance read speed and write cost.

Root cause:

Believing more indexes always improve performance without considering write overhead.

Key Takeaways

Query uses keys to find data directly, making it fast and cost-effective.

Scan reads every item in a table, which is slower and more expensive, suitable only for full data reads.

Filters reduce returned data but do not reduce scan read costs.

Indexes enable queries on different keys but must be used wisely to avoid overhead.

Choosing between scan and query impacts application speed, cost, and scalability.

Practice

1. Which AWS DynamoDB operation is faster when you know the exact primary key of the item you want to retrieve?

easy

A. Update

B. Scan

C. Query

D. Delete

Solution

1. Step 1: Understand Query operation

   Query retrieves items by specifying the primary key, making it fast and efficient.

2. Step 2: Compare with Scan operation

   Scan reads the entire table, which is slower and more costly when you know the key.

3. Final Answer:

   Query -> Option C

4. Quick Check:

   Known key = Query [OK]

Hint: Use Query for known keys, Scan reads all data [OK]

Common Mistakes:

• Thinking Scan is faster for known keys
• Confusing Update or Delete with data retrieval
• Assuming Query reads the whole table

2. Which of the following is the correct syntax to perform a Query operation in AWS DynamoDB using the AWS CLI?

easy

A. aws dynamodb query --table-name MyTable --key-condition-expression 'Id = :id' --expression-attribute-values '{":id":{"S":"123"}}'

B. aws dynamodb scan --table-name MyTable --key-condition-expression 'Id = :id' --expression-attribute-values '{":id":{"S":"123"}}'

C. aws dynamodb get-item --table-name MyTable --key '{"Id":{"S":"123"}}'

D. aws dynamodb update-item --table-name MyTable --key '{"Id":{"S":"123"}}'

Solution

1. Step 1: Identify Query command syntax

   The Query operation uses 'aws dynamodb query' with key-condition-expression and expression-attribute-values.

2. Step 2: Check other options

   Scan reads the entire table (and does not support --key-condition-expression), get-item retrieves a single item by exact key, update-item modifies items, so only the Query syntax is correct.

3. Final Answer:

   aws dynamodb query --table-name MyTable --key-condition-expression 'Id = :id' --expression-attribute-values '{":id":{"S":"123"}}' -> Option A

4. Quick Check:

   Query syntax uses 'query' command [OK]

Hint: Query uses 'aws dynamodb query' command [OK]

Common Mistakes:

• Using Scan command instead of Query
• Confusing GetItem with Query
• Incorrect expression-attribute-values format

3. Given a DynamoDB table with 1000 items, what will be the main difference in performance between these two operations?

1. Query with KeyConditionExpression on partition key
2. Scan without any filter

medium

A. Both read the same number of items and have similar speed

B. Scan reads fewer items and is faster; Query reads all items and is slower

C. Both read all items but Query is slower due to filtering

D. Query reads fewer items and is faster; Scan reads all items and is slower

Solution

1. Step 1: Analyze Query operation

   Query uses the partition key to directly find matching items, reading fewer items and faster.

2. Step 2: Analyze Scan operation

   Scan reads every item in the table, making it slower and more expensive.

3. Final Answer:

   Query reads fewer items and is faster; Scan reads all items and is slower -> Option D

4. Quick Check:

   Query = fewer reads, Scan = full table [OK]

Hint: Query targets keys, Scan reads whole table [OK]

Common Mistakes:

• Assuming Scan is faster than Query
• Thinking Query reads all items
• Ignoring cost differences

4. You wrote this DynamoDB Scan command but it is very slow:

aws dynamodb scan --table-name Users

What is the best way to improve performance if you only want users with UserId = '123'?

medium

A. Replace Scan with Query using KeyConditionExpression on UserId

B. Add a FilterExpression to the Scan command

C. Increase the Read Capacity Units of the table

D. Use UpdateItem to modify the UserId

Solution

1. Step 1: Identify problem with Scan

   Scan reads all items, causing slowness when filtering after reading.

2. Step 2: Use Query for known key

   Query with KeyConditionExpression on UserId reads only matching items, improving speed.

3. Final Answer:

   Replace Scan with Query using KeyConditionExpression on UserId -> Option A

4. Quick Check:

   Known key = Query, not Scan [OK]

Hint: Use Query with key condition, not Scan with filter [OK]

Common Mistakes:

• Adding filter to Scan instead of using Query
• Increasing capacity without optimizing query
• Using UpdateItem unrelated to read performance

5. You have a DynamoDB table with 1 million items. You need to retrieve all items where the attribute 'Status' equals 'Active'. The table's primary key is 'UserId'. Which approach gives the best performance and cost efficiency?

hard

A. Use Scan with FilterExpression 'Status = Active' to get all matching items

B. Create a Global Secondary Index (GSI) on 'Status' and use Query on the GSI

C. Use Query on the primary key 'UserId' with FilterExpression 'Status = Active'

D. Use Scan without any filter and filter results in your application code

Solution

1. Step 1: Understand Scan with filter

   Scan reads all 1 million items, filtering after reading, which is slow and costly.

2. Step 2: Use GSI for efficient querying

   Creating a GSI on 'Status' allows Query to directly find 'Active' items efficiently.

3. Step 3: Evaluate other options

   Query on primary key can't filter by 'Status' efficiently; scanning without filter wastes resources.

4. Final Answer:

   Create a Global Secondary Index (GSI) on 'Status' and use Query on the GSI -> Option B

5. Quick Check:

   GSI + Query = best performance for non-key attribute [OK]

Hint: Use GSI and Query for non-key attribute filtering [OK]

Common Mistakes:

• Using Scan with filter on large tables
• Querying primary key with filter instead of GSI
• Filtering in application after full Scan