Scan with filter expressions in DynamoDB - Deep Dive

Overview - Scan with filter expressions

What is it?

Scan with filter expressions is a way to read many items from a DynamoDB table but only keep those that match certain conditions. It reads all data but then filters out unwanted items based on rules you set. This helps you find specific data without needing to know exact keys. It is useful when you want to search broadly but narrow down results.

Why it matters

Without filter expressions, you would have to retrieve all data and then manually check each item, which wastes time and resources. Filter expressions let DynamoDB do the filtering for you, saving bandwidth and speeding up your app. This makes your database queries more efficient and your app more responsive, especially with large tables.

Where it fits

Before learning this, you should understand basic DynamoDB concepts like tables, items, and attributes, and how the Scan operation works. After this, you can learn about Query operations with key conditions, and advanced filtering techniques like using indexes or pagination.

Mental Model

Core Idea

Scan with filter expressions reads all items but only returns those that meet your conditions, filtering results after reading.

Think of it like...

Imagine you have a big box of mixed fruits and you want only the apples. You look through every fruit (scan), but only pick out the apples (filter expression) to keep.

┌─────────────┐
│   DynamoDB  │
│   Table     │
└─────┬───────┘
      │ Scan reads all items
      ▼
┌─────────────────────┐
│ Filter Expression   │
│ (e.g., color = red) │
└─────────┬───────────┘
          │ Filters items
          ▼
┌─────────────────────┐
│ Returned Items      │
│ (only matching ones)│
└─────────────────────┘

Build-Up - 7 Steps

1

Foundation

Understanding DynamoDB Scan Basics

Concept: Learn what a Scan operation does in DynamoDB and how it reads all items in a table.

A Scan operation reads every item in a DynamoDB table. It does not require any key information. This means it looks at all data, which can be slow for big tables. Scan returns all items unless you tell it to filter.

Result

You get all items from the table, which can be many rows.

Understanding that Scan reads everything helps you see why filtering afterward is important to avoid unnecessary data.

2

Foundation

What Are Filter Expressions?

Concept: Introduce filter expressions as conditions to select which items to keep after scanning.

Filter expressions are like rules you write to say which items you want. For example, you can say 'only items where age > 30'. DynamoDB applies these rules after reading all items but before returning results.

Result

Only items matching the filter rules are returned to you.

Knowing that filtering happens after reading all data explains why Scan can still be costly even with filters.

3

Intermediate

Writing Filter Expressions Syntax

🤔Before reading on: do you think filter expressions use the same syntax as DynamoDB key conditions? Commit to your answer.

Concept: Learn the syntax and operators used in filter expressions to build conditions.

Filter expressions use attribute names and values with operators like =, <, >, BETWEEN, and functions like begins_with. You write them as strings, for example: "#age > :val" with placeholders for attribute names and values to avoid conflicts.

Result

You can write flexible conditions to filter items based on attributes.

Understanding the syntax lets you create precise filters that match exactly the data you want.

4

Intermediate

Using Expression Attribute Names and Values

🤔Before reading on: do you think you can use attribute names directly in filter expressions without placeholders? Commit to your answer.

Concept: Learn why and how to use placeholders for attribute names and values in filter expressions.

DynamoDB requires placeholders to avoid conflicts with reserved words or special characters. You define ExpressionAttributeNames like {"#age": "age"} and ExpressionAttributeValues like {":val": 30}. Then use these in your filter expression string.

Result

Your filter expressions become safe and error-free when using placeholders.

Knowing this prevents common syntax errors and helps you write robust queries.

5

Intermediate

Filter Expressions vs Key Conditions

🤔Before reading on: do you think filter expressions reduce the amount of data read from the table? Commit to your answer.

Concept: Understand the difference between filtering after reading and querying by keys before reading.

Key conditions limit which items DynamoDB reads by using primary key attributes. Filter expressions only remove items after reading all. So filters reduce returned data but not read data, affecting performance differently.

Result

You learn when to use filters and when to use key conditions for efficiency.

Knowing this difference helps you design queries that balance speed and flexibility.

6

Advanced

Handling Large Tables and Pagination

🤔Before reading on: do you think Scan with filters returns all matching items in one go? Commit to your answer.

Concept: Learn how Scan handles large data sets with pagination and how filters affect this.

Scan returns results in pages. If the table is large, you get a subset and a LastEvaluatedKey to continue. Filters apply to each page's items. You must loop to get all filtered results. This affects how you write your code.

Result

You can retrieve all matching items safely without missing data.

Understanding pagination with filters prevents bugs where you miss some results.

7

Expert

Performance Implications and Best Practices

🤔Before reading on: do you think using filter expressions always improves query speed? Commit to your answer.

Concept: Explore how filters impact performance and how to optimize Scan usage in production.

Filters do not reduce read capacity units consumed because Scan reads all data first. Overusing Scan with filters on big tables can be costly and slow. Best practice is to use Query with key conditions when possible, or add indexes. Use filters only when necessary and paginate results.

Result

You write efficient, cost-effective DynamoDB queries in real apps.

Knowing the cost and limits of filters helps you avoid expensive queries and design scalable systems.

Under the Hood

When you run a Scan with a filter expression, DynamoDB reads every item in the table or index. It then applies the filter expression to each item in memory to decide if it should be included in the results. Items that do not match are discarded before sending data back to the client. This means the read capacity units are consumed for all scanned items, not just the filtered ones.

Why designed this way?

DynamoDB was designed for fast key-value access, so Query operations use keys to limit reads. Scan is a fallback for broad searches. Filtering after reading keeps Scan simple and flexible, allowing any attribute to be filtered without indexing. This design trades some efficiency for flexibility and simplicity.

┌───────────────┐
│   Scan Start  │
└──────┬────────┘
       │ Reads all items
       ▼
┌───────────────┐
│  All Items    │
└──────┬────────┘
       │ Apply filter expression
       ▼
┌───────────────┐
│ Filtered Items│
└──────┬────────┘
       │ Return to client
       ▼
┌───────────────┐
│   Results     │
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does a filter expression reduce the number of items DynamoDB reads from the table? Commit yes or no.

Common Belief: Filter expressions reduce the number of items DynamoDB reads, so they save read capacity units.

Reality: Filter expressions only remove items after DynamoDB reads them all. They do not reduce the read capacity units consumed.

Why it matters: Believing filters reduce reads can lead to unexpected high costs and slow queries when scanning large tables.

Quick: Can you use filter expressions to replace key conditions in a Query? Commit yes or no.

Common Belief: Filter expressions can be used instead of key conditions to efficiently find items by keys.

Reality: Filter expressions cannot replace key conditions. Key conditions limit which items are read, filters only remove items after reading.

Why it matters: Misusing filters instead of key conditions causes inefficient queries and poor performance.

Quick: Do filter expressions guarantee all matching items are returned in one Scan call? Commit yes or no.

Common Belief: A single Scan with filter expressions returns all matching items at once.

Reality: Scan results are paginated. You must handle pagination to get all matching items.

Why it matters:

Quick: Do filter expressions support complex logical operations like AND, OR, and NOT? Commit yes or no.

Common Belief: Filter expressions only support simple comparisons, not complex logical combinations.

Reality: Filter expressions support AND, OR, and NOT to combine multiple conditions.

Why it matters: Knowing this allows building powerful filters without extra code.

Expert Zone

1

Filter expressions do not reduce read capacity units but can reduce network bandwidth by returning fewer items.

2

Using ProjectionExpression with Scan and filters can reduce data size returned, improving performance.

3

Filter expressions are evaluated client-side after data is read internally, so they do not speed up Scan but help reduce client processing.

When NOT to use

Avoid Scan with filter expressions on large tables when you can use Query with key conditions or Global Secondary Indexes. For frequent queries, design your table keys and indexes to support efficient Query operations instead.

Production Patterns

In production, Scan with filters is often used for ad-hoc reports or admin tools where flexibility is needed. Developers combine filters with pagination and ProjectionExpressions to limit data. For high-scale apps, Query with indexes is preferred, and Scan is used sparingly.

Connections

Query operation in DynamoDB

Complementary operation that uses key conditions to limit reads before filtering

Understanding Scan filters highlights why Query with key conditions is more efficient and when to choose each.

SQL WHERE clause

Similar concept of filtering rows based on conditions after selecting data

Knowing SQL WHERE helps grasp how filter expressions work to narrow down results after reading data.

Data filtering in spreadsheet software

Both filter data sets to show only matching rows based on criteria

Seeing filter expressions like spreadsheet filters helps understand their purpose and effect on data views.

Common Pitfalls

#1 Expecting filter expressions to reduce read capacity units consumed.

Wrong approach:

Scan operation with filter expression expecting low cost:

Scan({
TableName: 'Users',
FilterExpression: '#age > :val',
ExpressionAttributeNames: {'#age': 'age'},
ExpressionAttributeValues: {':val': 30}
})

Correct approach:

Use Query with key condition if possible:

Query({
TableName: 'Users',
KeyConditionExpression: '#id = :id',
ExpressionAttributeNames: {'#id': 'userId'},
ExpressionAttributeValues: {':id': '123'}
})

Root cause:

Misunderstanding that filters reduce data read instead of only filtering returned results.

#2 Not handling pagination when scanning large tables with filters.

Wrong approach:

Scan({
TableName: 'Orders',
FilterExpression: '#status = :val',
ExpressionAttributeNames: {'#status': 'status'},
ExpressionAttributeValues: {':val': 'shipped'}
})
// Assume all results returned in one call

Correct approach:

Use loop to handle pagination:

let lastKey = null;
do {
const params = {
TableName: 'Orders',
FilterExpression: '#status = :val',
ExpressionAttributeNames: {'#status': 'status'},
ExpressionAttributeValues: {':val': 'shipped'},
ExclusiveStartKey: lastKey
};
const result = await dynamoDb.scan(params).promise();
process(result.Items);
lastKey = result.LastEvaluatedKey;
} while (lastKey);

Root cause:

Assuming Scan returns all data at once without pagination.

#3 Using attribute names directly in filter expressions without placeholders.

Wrong approach:

FilterExpression: 'age > :val',
ExpressionAttributeValues: {':val': 30}

Correct approach:

FilterExpression: '#age > :val',
ExpressionAttributeNames: {'#age': 'age'},
ExpressionAttributeValues: {':val': 30}

Root cause:

Not knowing DynamoDB reserved words and syntax require placeholders for attribute names.

Key Takeaways

Scan with filter expressions reads all items but only returns those matching your conditions, filtering after reading.

Filter expressions do not reduce the amount of data read or the cost, only the data returned to your app.

Use placeholders for attribute names and values in filter expressions to avoid syntax errors and reserved word conflicts.

Always handle pagination when scanning large tables to ensure you retrieve all matching items.

For better performance, prefer Query with key conditions and indexes over Scan with filters whenever possible.