Parallel scan in DynamoDB

Introduction

Parallel scan helps you read a large table faster by splitting the work into parts that run at the same time.

You want to quickly read all items from a big DynamoDB table.

You need to process data faster by using multiple workers or threads.

You want to reduce the total time to scan a large dataset.

You have a batch job that reads the entire table for analysis.

You want to avoid one slow scan blocking your application.

Syntax

DynamoDB

Scan operation with parameters:
- TableName: name of the table
- Segment: the part number to scan (0 to TotalSegments-1)
- TotalSegments: total parts to split the scan into
- Other scan options like FilterExpression, ProjectionExpression

Example:
Scan({
  TableName: 'MyTable',
  Segment: 0,
  TotalSegments: 4
})

The Segment and TotalSegments parameters control how the scan is split.

Each segment is scanned independently and can run in parallel.

Examples

Scan the first part (segment 0) of the 'Products' table when split into 3 parts.

DynamoDB

Scan({ TableName: 'Products', Segment: 0, TotalSegments: 3 })

Scan the third part (segment 2) of the 'Orders' table with a filter to get only pending orders.

DynamoDB

Scan({ TableName: 'Orders', Segment: 2, TotalSegments: 5, FilterExpression: 'Status = :s', ExpressionAttributeValues: { ':s': 'Pending' } })

Sample Program

This scans the second half (segment 1) of the 'Employees' table when split into 2 parts.

DynamoDB

Scan({
  TableName: 'Employees',
  Segment: 1,
  TotalSegments: 2
})

Important Notes

Make sure to run all segments (from 0 to TotalSegments-1) to scan the entire table.

Parallel scans can increase read capacity usage, so watch your limits.

Use parallel scans only when you need faster full table reads, not for small queries.

Summary

Parallel scan splits a table scan into parts to run at the same time.

Use Segment and TotalSegments to control the parts.

Run all segments to get the full table data faster.