Scan pagination in DynamoDB - Mini Project: Build & Apply

Scan Pagination in DynamoDB

📖 Scenario: You are working with a DynamoDB table that stores customer orders. The table has many items, and you want to retrieve all orders but only a few at a time to avoid overloading your application.

🎯 Goal: Build a DynamoDB scan operation with pagination to fetch all items in small batches.

📋 What You'll Learn

Create a dictionary called orders_table to simulate the DynamoDB table with 10 items.

Create a variable called page_size and set it to 3 to limit items per scan.

Write a function called scan_with_pagination that scans the orders_table in pages of page_size.

Add a variable called last_evaluated_key to keep track of the scan position and update it after each page.

💡 Why This Matters

🌍 Real World

In real applications, DynamoDB tables can have thousands or millions of items. Scanning all at once can be slow and costly. Pagination helps fetch data in smaller, manageable chunks.

💼 Career

Understanding scan pagination is essential for backend developers and database administrators working with DynamoDB to optimize data retrieval and application performance.

1

DATA SETUP: Create the orders table data

Create a dictionary called orders_table with 10 items. Each item should have keys OrderId (from 1 to 10) and CustomerName with values 'Customer1' to 'Customer10'.

DynamoDB

# Create the orders_table dictionary with 10 items
# Your code here

Hint

Use a dictionary with keys 1 to 10. Each value is another dictionary with keys 'OrderId' and 'CustomerName'.

2

CONFIGURATION: Set the page size for scan

Create a variable called page_size and set it to 3 to limit the number of items returned per scan page.

DynamoDB

orders_table = {
    1: {'OrderId': 1, 'CustomerName': 'Customer1'},
    2: {'OrderId': 2, 'CustomerName': 'Customer2'},
    3: {'OrderId': 3, 'CustomerName': 'Customer3'},
    4: {'OrderId': 4, 'CustomerName': 'Customer4'},
    5: {'OrderId': 5, 'CustomerName': 'Customer5'},
    6: {'OrderId': 6, 'CustomerName': 'Customer6'},
    7: {'OrderId': 7, 'CustomerName': 'Customer7'},
    8: {'OrderId': 8, 'CustomerName': 'Customer8'},
    9: {'OrderId': 9, 'CustomerName': 'Customer9'},
    10: {'OrderId': 10, 'CustomerName': 'Customer10'}
}

# Set the page_size variable to 3
# Your code here

Hint

Just create a variable named page_size and assign it the value 3.

3

CORE LOGIC: Write the scan_with_pagination function

Write a function called scan_with_pagination that takes orders_table and page_size as parameters. Inside, create a variable last_evaluated_key set to None. Use a while loop to scan the table in pages of page_size. In each loop, select the next page of items starting after last_evaluated_key. Update last_evaluated_key to the last item's key in the current page. Stop when all items are scanned. Return a list of all scanned items.

DynamoDB

def scan_with_pagination(orders_table, page_size):
    last_evaluated_key = None
    scanned_items = []
    keys = sorted(orders_table.keys())
    while True:
        if last_evaluated_key is None:
            start_index = 0
        else:
            start_index = keys.index(last_evaluated_key) + 1
        page_keys = keys[start_index:start_index + page_size]
        if not page_keys:
            break
        for key in page_keys:
            scanned_items.append(orders_table[key])
        last_evaluated_key = page_keys[-1]
    return scanned_items
# Create last_evaluated_key variable outside the function

Hint

Use a while True loop and update last_evaluated_key after each page. Use keys.index() to find the start position.

4

COMPLETION: Add the last_evaluated_key variable outside the function

Create a variable called last_evaluated_key and set it to None outside the function to simulate the initial scan state.

DynamoDB

last_evaluated_key = None

Hint

Just create the variable last_evaluated_key and set it to None outside the function.

Practice

1. What is the main purpose of using Scan pagination in DynamoDB?

easy

A. To speed up writes to the database

B. To delete items in batches

C. To create indexes automatically

D. To break a large scan into smaller parts for easier processing

Solution

1. Step 1: Understand Scan Pagination Concept

   Scan pagination is used to divide a large scan operation into smaller chunks to avoid timeouts and manage resource use.

2. Step 2: Identify the Purpose

   The main goal is to process big scans in parts, not to speed writes or create indexes.

3. Final Answer:

   To break a large scan into smaller parts for easier processing -> Option D

4. Quick Check:

   Scan pagination = break large scan into parts [OK]

Hint: Scan pagination splits big scans into smaller chunks [OK]

Common Mistakes:

• Thinking pagination speeds up writes
• Confusing scan with index creation
• Assuming pagination deletes items

2. Which of the following is the correct way to continue a paginated scan in DynamoDB?

easy

A. Use LastEvaluatedKey as the ExclusiveStartKey in the next scan

B. Use Limit as the ExclusiveStartKey

C. Use ScanIndexForward to continue scanning

D. Use StartKey to begin the scan

Solution

1. Step 1: Identify Pagination Parameters

   In DynamoDB, LastEvaluatedKey from the previous scan is used as ExclusiveStartKey to continue scanning.

2. Step 2: Eliminate Incorrect Options

   Limit sets chunk size, not start key. ScanIndexForward is for queries, not scans. StartKey is not a valid parameter.

3. Final Answer:

   Use LastEvaluatedKey as the ExclusiveStartKey in the next scan -> Option A

4. Quick Check:

   Continue scan = ExclusiveStartKey = LastEvaluatedKey [OK]

Hint: Use LastEvaluatedKey as ExclusiveStartKey to continue scan [OK]

Common Mistakes:

• Using Limit as ExclusiveStartKey
• Confusing scan with query parameters
• Using invalid parameter names

3. Given the following scan code snippet, what will be the output if the table has 15 items and Limit is set to 5?

response = table.scan(Limit=5)
items = response['Items']
last_key = response.get('LastEvaluatedKey')
print(len(items), last_key is not None)

medium

A. 5 False

B. 15 False

C. 5 True

D. 15 True

Solution

1. Step 1: Understand Limit Effect on Scan

   Setting Limit=5 returns up to 5 items in one scan call, not all 15.

2. Step 2: Check LastEvaluatedKey Presence

   Since there are more items after 5, LastEvaluatedKey will be present (not None), indicating more data.

3. Final Answer:

   5 True -> Option C

4. Quick Check:

   Limit=5 returns 5 items and LastEvaluatedKey exists [OK]

Hint: Limit controls items returned; LastEvaluatedKey shows more data [OK]

Common Mistakes:

• Assuming all items return ignoring Limit
• Expecting LastEvaluatedKey to be None always
• Confusing item count with total table size

4. You wrote this code to paginate a scan but it returns only the first page repeatedly:

response = table.scan(Limit=3)
items = response['Items']
while 'LastEvaluatedKey' in response:
    response = table.scan(Limit=3)
    items.extend(response['Items'])
print(len(items))

What is the error?

medium

A. Missing ExclusiveStartKey in the subsequent scan calls

B. Limit should be increased to get all items

C. Items list should be reset inside the loop

D. LastEvaluatedKey should be deleted after each scan

Solution

1. Step 1: Analyze Pagination Loop

   The loop calls scan repeatedly but does not pass ExclusiveStartKey, so it always scans from start.

2. Step 2: Identify Missing Parameter

   To continue scanning, ExclusiveStartKey must be set to previous LastEvaluatedKey in each call.

3. Final Answer:

   Missing ExclusiveStartKey in the subsequent scan calls -> Option A

4. Quick Check:

   Pagination needs ExclusiveStartKey to continue [OK]

Hint: Pass ExclusiveStartKey to continue scan pages [OK]

Common Mistakes:

• Not passing ExclusiveStartKey in loop
• Increasing Limit instead of fixing pagination
• Resetting items list inside loop

5. You want to scan a DynamoDB table with 1000 items but only want to process 100 items at a time to avoid timeouts. Which approach correctly implements scan pagination to achieve this?

hard

A. Set ExclusiveStartKey to null and scan with Limit=100 once

B. Use a loop with Limit=100 and pass ExclusiveStartKey from LastEvaluatedKey until no more keys

C. Use ScanIndexForward=true with Limit=100 to paginate

D. Scan once with Limit=1000 and split results in code

Solution

1. Step 1: Understand Pagination Requirements

   To process 100 items at a time, scan must be done in chunks with Limit=100 and continue using ExclusiveStartKey.

2. Step 2: Evaluate Options

   Use a loop with Limit=100 and pass ExclusiveStartKey from LastEvaluatedKey until no more keys correctly loops with Limit=100 and uses ExclusiveStartKey from LastEvaluatedKey. Scan once with Limit=1000 and split results in code fetches all at once, risking timeout. Use ScanIndexForward=true with Limit=100 to paginate uses wrong parameter for scan. Set ExclusiveStartKey to null and scan with Limit=100 once scans only once without continuation.

3. Final Answer:

   Use a loop with Limit=100 and pass ExclusiveStartKey from LastEvaluatedKey until no more keys -> Option B

4. Quick Check:

   Paginate with Limit and ExclusiveStartKey loop [OK]

Hint: Loop with Limit and ExclusiveStartKey until no LastEvaluatedKey [OK]

Common Mistakes:

• Fetching all items at once risking timeout
• Using query parameters for scan
• Not looping to continue scan