Batch limits and retries in DynamoDB

Introduction

Batch limits and retries help you send or receive many items efficiently without overloading the system or losing data.

When you want to read or write many items at once to save time.

When you need to handle limits set by DynamoDB on batch sizes.

When some requests fail and you want to try them again automatically.

When you want to avoid errors caused by sending too many items in one batch.

Syntax

DynamoDB

BatchWriteItemRequest {
  RequestItems: {
    TableName: [
      { PutRequest: { Item: {...} } },
      { DeleteRequest: { Key: {...} } }
    ]
  },
  ReturnConsumedCapacity: 'TOTAL' | 'NONE',
  ReturnItemCollectionMetrics: 'SIZE' | 'NONE'
}

BatchGetItemRequest {
  RequestItems: {
    TableName: {
      Keys: [ { KeyAttribute: value }, ... ],
      ProjectionExpression: 'attribute1, attribute2'
    }
  },
  ReturnConsumedCapacity: 'TOTAL' | 'NONE'
}

DynamoDB limits batch write requests to 25 items per batch.

Batch get requests can retrieve up to 100 items per batch.

Unprocessed items are returned and should be retried.

Examples

Batch write with 2 new books and 1 book deleted.

DynamoDB

BatchWriteItemRequest with 3 items:
{
  RequestItems: {
    'Books': [
      { PutRequest: { Item: { 'ISBN': '123', 'Title': 'Book A' } } },
      { PutRequest: { Item: { 'ISBN': '456', 'Title': 'Book B' } } },
      { DeleteRequest: { Key: { 'ISBN': '789' } } }
    ]
  }
}

Batch get to retrieve title and author of 2 books.

DynamoDB

BatchGetItemRequest with 2 keys:
{
  RequestItems: {
    'Books': {
      Keys: [ { 'ISBN': '123' }, { 'ISBN': '456' } ],
      ProjectionExpression: 'Title, Author'
    }
  }
}

Batch write with no items does nothing but is valid.

DynamoDB

Empty BatchWriteItemRequest:
{
  RequestItems: {
    'Books': []
  }
}

DynamoDB limits batch writes to 25 items per request.

DynamoDB

BatchWriteItemRequest with 26 items (exceeds limit):
// This will cause an error and needs splitting into two batches.

Sample Program

This program writes up to 25 items at a time to a DynamoDB table named 'Books'. If some items are not processed, it retries them until all are written.

DynamoDB

import boto3
from botocore.exceptions import ClientError

# Create DynamoDB client
client = boto3.client('dynamodb')

def batch_write_with_retries(table_name, items):
    max_batch_size = 25
    unprocessed_items = []

    # Split items into batches of 25
    for i in range(0, len(items), max_batch_size):
        batch = items[i:i+max_batch_size]
        request_items = {table_name: [{'PutRequest': {'Item': item}} for item in batch]}

        while True:
            response = client.batch_write_item(RequestItems=request_items)
            unprocessed = response.get('UnprocessedItems', {})

            if not unprocessed or not unprocessed.get(table_name):
                break  # All items processed

            # Retry unprocessed items
            request_items = unprocessed

    print(f"All items written to {table_name} with retries if needed.")

# Example items to write
items_to_write = [
    {'ISBN': {'S': '001'}, 'Title': {'S': 'Book One'}},
    {'ISBN': {'S': '002'}, 'Title': {'S': 'Book Two'}},
    {'ISBN': {'S': '003'}, 'Title': {'S': 'Book Three'}}
]

print("Before batch write")
batch_write_with_retries('Books', items_to_write)
print("After batch write")

Important Notes

Batch write operations have a limit of 25 items per request and batch get operations have a limit of 100 items.

Unprocessed items are common when throughput limits are exceeded; always check and retry them.

Retries should have a delay or backoff in real applications to avoid throttling.

Summary

Batch limits prevent sending too many items at once to DynamoDB.

Always check for unprocessed items and retry them to ensure data is saved.

Use batch operations to improve efficiency when handling many items.

Practice

1. What is the main reason DynamoDB enforces batch limits when using batch write operations?

easy

A. To allow unlimited retries automatically

B. To increase the size of each item stored

C. To prevent sending too many items at once and avoid throttling

D. To reduce the number of tables in the database

Solution

1. Step 1: Understand batch write limits

   DynamoDB limits batch write operations to a maximum of 25 items per request to avoid overloading the service.

2. Step 2: Reason about the purpose of limits

   These limits help prevent throttling and ensure stable performance by controlling request size.

3. Final Answer:

   To prevent sending too many items at once and avoid throttling -> Option C

4. Quick Check:

   Batch limits = prevent throttling [OK]

Hint: Batch limits protect DynamoDB from overloads [OK]

Common Mistakes:

• Thinking batch limits increase item size
• Assuming retries are automatic without checks
• Confusing batch limits with table count

2. Which of the following is the correct maximum number of items you can send in a single BatchWriteItem request in DynamoDB?

easy

A. 25 items

B. 50 items

C. 100 items

D. 10 items

Solution

1. Step 1: Recall DynamoDB batch write limits

   DynamoDB allows a maximum of 25 items per BatchWriteItem request.

2. Step 2: Compare options with the limit

   Only 25 items matches the correct limit of 25 items.

3. Final Answer:

   25 items -> Option A

4. Quick Check:

   BatchWriteItem max = 25 items [OK]

Hint: Remember 25 items max per batch write [OK]

Common Mistakes:

• Choosing 50 or 100 items which exceed the limit
• Confusing batch write with batch get limits
• Assuming 10 items is the max without checking docs

3. Consider this Python snippet using boto3 DynamoDB client:

response = client.batch_write_item(RequestItems={
  'MyTable': [
    {'PutRequest': {'Item': {'ID': {'S': '1'}}}},
    {'PutRequest': {'Item': {'ID': {'S': '2'}}}}
  ]
})
print(response.get('UnprocessedItems'))

If the response prints {}, what does it mean?

medium

A. The batch write failed completely

B. The request exceeded batch limits

C. Only the first item was processed

D. All items were processed successfully with no retries needed

Solution

1. Step 1: Understand UnprocessedItems meaning

   The UnprocessedItems key contains items that DynamoDB did not process and need retrying.

2. Step 2: Interpret empty dictionary response

   An empty dictionary {} means no unprocessed items, so all were processed successfully.

3. Final Answer:

   All items were processed successfully with no retries needed -> Option D

4. Quick Check:

   Empty UnprocessedItems = all processed [OK]

Hint: Empty UnprocessedItems means success [OK]

Common Mistakes:

• Assuming empty means failure
• Thinking only partial items processed
• Confusing batch limits with processing status

4. You wrote code to batch write 30 items to DynamoDB but got an error. Which fix will solve the problem?

medium

A. Split the 30 items into two batches of 25 and 5 items

B. Send all 30 items in one batch and increase timeout

C. Use batch get instead of batch write

D. Reduce each item size to half

Solution

1. Step 1: Identify batch write limit violation

   DynamoDB batch write limits max 25 items per request, so 30 items exceed this limit.

2. Step 2: Apply correct fix by splitting batches

   Splitting into two batches (25 and 5) respects the limit and avoids errors.

3. Final Answer:

   Split the 30 items into two batches of 25 and 5 items -> Option A

4. Quick Check:

   Split batches to respect 25 item limit [OK]

Hint: Split batches over 25 items to fix errors [OK]

Common Mistakes:

• Trying to send all items at once ignoring limits
• Using batch get which is for reads, not writes
• Reducing item size does not fix batch count limit

5. You have a batch write operation that returns some unprocessed items due to throttling. What is the best approach to ensure all items are eventually written?

hard

A. Retry the entire batch including processed items

B. Immediately retry only the unprocessed items with exponential backoff

C. Ignore unprocessed items and assume they are saved

D. Increase the batch size to include more items

Solution

1. Step 1: Understand unprocessed items cause

   Unprocessed items occur when DynamoDB throttles requests; these items need retrying.

2. Step 2: Apply best retry strategy

   Retry only unprocessed items with exponential backoff to avoid repeated throttling and ensure eventual success.

3. Final Answer:

   Immediately retry only the unprocessed items with exponential backoff -> Option B

4. Quick Check:

   Retry unprocessed items with backoff [OK]

Hint: Retry unprocessed items with backoff, not whole batch [OK]

Common Mistakes:

• Ignoring unprocessed items causing data loss
• Retrying entire batch wastes resources
• Increasing batch size worsens throttling