Lambda with DynamoDB Streams in AWS - Deep Dive

Overview - Lambda with DynamoDB Streams

What is it?

Lambda with DynamoDB Streams is a way to automatically run small pieces of code when data in a DynamoDB table changes. DynamoDB Streams capture these changes as events, and Lambda functions process these events to react in real time. This lets you build applications that respond instantly to database updates without managing servers.

Why it matters

Without this, developers would need to constantly check the database for changes or build complex polling systems, which wastes resources and slows down reactions. Using Lambda with DynamoDB Streams makes applications more efficient, scalable, and responsive, improving user experience and reducing operational work.

Where it fits

Before learning this, you should understand basic AWS Lambda functions and DynamoDB tables. After mastering this, you can explore event-driven architectures, serverless workflows, and integrating other AWS services like SNS or SQS for complex processing.

Mental Model

Core Idea

Lambda with DynamoDB Streams connects database changes to automatic code execution, creating instant reactions to data updates without manual checks.

Think of it like...

It's like having a smart mailbox that not only receives letters (data changes) but also immediately rings a bell and triggers a helper (Lambda) to sort or act on the mail as soon as it arrives.

┌───────────────┐      ┌───────────────┐      ┌───────────────┐
│ DynamoDB Table│─────▶│ DynamoDB Stream│─────▶│ Lambda Function│
└───────────────┘      └───────────────┘      └───────────────┘
       Data changes          Capture events          Process events

Build-Up - 7 Steps

1

Foundation

Understanding DynamoDB Tables

Concept: Learn what DynamoDB tables are and how they store data.

DynamoDB is a database service that stores data in tables made of items (rows) and attributes (columns). Each table has a primary key to uniquely identify items. You can add, update, or delete items in the table.

Result

You know how data is organized and changed in DynamoDB tables.

Understanding the structure of DynamoDB tables is essential because Streams track changes to these tables.

2

Foundation

Basics of AWS Lambda Functions

Concept: Learn what Lambda functions are and how they run code without servers.

AWS Lambda lets you write small programs that run automatically when triggered. You don't manage servers; AWS runs your code only when needed. You write the code, set triggers, and Lambda handles the rest.

Result

You can create and run simple Lambda functions triggered by events.

Knowing Lambda basics is key because Lambda will process the events from DynamoDB Streams.

3

Intermediate

What Are DynamoDB Streams?

Concept: DynamoDB Streams capture and record changes made to a table as a sequence of events.

When you enable Streams on a DynamoDB table, every change (insert, update, delete) is recorded in order. These records include the old and new data. Streams keep these events for 24 hours, allowing other services to react to them.

Result

You understand how DynamoDB Streams track table changes as events.

Knowing that Streams provide a time-ordered event log helps you see how Lambda can react to data changes reliably.

4

Intermediate

Connecting Lambda to DynamoDB Streams

🤔Before reading on: do you think Lambda polls DynamoDB Streams continuously or is triggered only when new events arrive? Commit to your answer.

Concept: Lambda functions can be set to trigger automatically when new events appear in DynamoDB Streams.

You configure Lambda to listen to a DynamoDB Stream. When new events arrive, Lambda runs your code with those events as input. This happens automatically and scales with the number of events.

Result

Lambda functions run instantly in response to table changes without manual intervention.

Understanding the automatic trigger mechanism shows how event-driven systems reduce latency and manual work.

5

Intermediate

Processing Stream Events in Lambda

🤔Before reading on: do you think Lambda receives one event at a time or batches multiple events from Streams? Commit to your answer.

Concept: Lambda receives batches of stream records and processes them in your function code.

When Lambda triggers, it gets a batch of records from the stream. Your code can loop through these records to handle inserts, updates, or deletes. You can filter or transform data, call other services, or update other databases.

Result

You can write Lambda code that reacts to multiple changes efficiently.

Knowing Lambda processes batches helps optimize performance and error handling.

6

Advanced

Handling Errors and Retries in Stream Processing

🤔Before reading on: do you think Lambda automatically retries failed stream events indefinitely or stops after some attempts? Commit to your answer.

Concept: Lambda retries failed batches and can send failed events to a dead-letter queue for later inspection.

If your Lambda function fails to process a batch, AWS retries it until success or until the data expires (24 hours). You can configure a dead-letter queue (DLQ) to capture failed events for debugging. Proper error handling in your code prevents data loss or duplicate processing.

Result

You understand how to build reliable stream processing with error recovery.

Knowing the retry and DLQ mechanisms prevents silent failures and data inconsistencies.

7

Expert

Scaling and Performance Considerations

🤔Before reading on: do you think Lambda scales automatically with stream volume or requires manual scaling? Commit to your answer.

Concept: Lambda scales automatically but has limits based on shard count and concurrency; understanding these helps optimize throughput and cost.

Each DynamoDB Stream is divided into shards. Lambda processes each shard in order, one batch at a time. If your table has many shards, Lambda can run many functions in parallel. However, there are concurrency limits and shard limits to consider. You can tune batch size, parallelization factor, and error handling to balance speed and cost.

Result

You can design stream processing that handles high loads efficiently and cost-effectively.

Understanding shard-based scaling and Lambda concurrency is crucial for building robust, scalable applications.

Under the Hood

DynamoDB Streams capture every data modification as a sequence of ordered events stored temporarily. Each event contains the before and after images of the item. Lambda polls these streams internally, fetching batches of events from shards. It then invokes your function with these batches. Lambda manages concurrency by processing shards independently but preserves order within each shard. Failed invocations trigger retries or send events to dead-letter queues if configured.

Why designed this way?

This design ensures reliable, ordered event processing without requiring developers to manage polling or state. Using shards allows parallel processing while preserving order per shard. Temporary event storage balances durability and cost. Lambda's serverless model removes infrastructure overhead, making event-driven architectures accessible and scalable.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ DynamoDB Table│──────▶│ DynamoDB Stream│──────▶│ Lambda Service│
└───────────────┘       └───────────────┘       └──────┬────────┘
                                   │                      │
                                   ▼                      ▼
                             ┌───────────┐         ┌────────────┐
                             │ Stream    │         │ Lambda     │
                             │ Shards    │────────▶│ Function   │
                             └───────────┘         └────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does Lambda process DynamoDB Stream events one by one or in batches? Commit to your answer.

Common Belief: Lambda processes each DynamoDB Stream event individually as it arrives.

Reality: Lambda receives and processes events in batches from the stream, not one at a time.

Why it matters: Assuming single-event processing can lead to inefficient code and missed opportunities for batch optimization.

Quick: Does DynamoDB Streams keep events forever? Commit to your answer.

Common Belief: DynamoDB Streams store all change events permanently for audit and replay.

Reality: DynamoDB Streams keep events only for 24 hours before they expire.

Why it matters: Expecting permanent storage can cause data loss if processing is delayed beyond 24 hours.

Quick: Does Lambda guarantee exactly-once processing of stream events? Commit to your answer.

Common Belief: Lambda guarantees each DynamoDB Stream event is processed exactly once.

Reality: Lambda provides at-least-once processing, so events may be processed more than once in rare cases.

Why it matters: Not handling duplicate events can cause data inconsistencies or repeated side effects.

Quick: Can Lambda process multiple shards in parallel without limits? Commit to your answer.

Common Belief: Lambda can process unlimited shards in parallel without any concurrency limits.

Reality: Lambda scales with shard count but is subject to account concurrency limits and shard parallelization factors.

Why it matters: Ignoring concurrency limits can cause throttling and delays in event processing.

Expert Zone

1

Lambda processes each shard sequentially to preserve event order, but shards run in parallel, so overall processing order across shards is not guaranteed.

2

Batch size and parallelization factor settings impact latency, cost, and error handling complexity; tuning these requires understanding workload patterns.

3

Dead-letter queues are essential for capturing failed events but require separate monitoring and cleanup to avoid silent data loss.

When NOT to use

Avoid using Lambda with DynamoDB Streams for workloads requiring guaranteed exactly-once processing or long-term event storage. Instead, consider using Kinesis Data Streams with enhanced processing guarantees or event buses with durable storage like EventBridge.

Production Patterns

In production, teams use Lambda with DynamoDB Streams for real-time analytics, cache invalidation, search indexing, and triggering workflows. They implement idempotent Lambda functions to handle retries safely and monitor stream lag and Lambda concurrency metrics to maintain performance.

Connections

Event-Driven Architecture

Lambda with DynamoDB Streams is a practical example of event-driven architecture where events trigger actions automatically.

Understanding this connection helps grasp how loosely coupled systems communicate and react in real time.

Message Queues

DynamoDB Streams act like a message queue that holds events for processing by Lambda.

Knowing this helps understand event buffering, ordering, and retry mechanisms common in distributed systems.

Human Reflexes

Like a reflex that automatically reacts to stimuli without conscious thought, Lambda functions respond instantly to database changes.

This cross-domain link shows how automation mimics natural quick responses to maintain system health and user experience.

Common Pitfalls

#1 Ignoring duplicate event processing can cause repeated side effects.

Wrong approach:

exports.handler = async (event) => {
event.Records.forEach(record => {
// Process record without checking if already processed
updateExternalSystem(record.dynamodb.NewImage);
});
};

Correct approach:

exports.handler = async (event) => {
event.Records.forEach(record => {
if (!hasProcessed(record.eventID)) {
updateExternalSystem(record.dynamodb.NewImage);
markProcessed(record.eventID);
}
});
};

Root cause:

Misunderstanding that Lambda processing is at-least-once, so duplicates can occur.

#2 Setting batch size too large causing Lambda timeouts or slow processing.

Wrong approach:

Configure event source mapping with batchSize: 1000 without testing workload.

Correct approach:

Start with batchSize: 100 and adjust based on processing time and Lambda limits.

Root cause:

Not tuning batch size to match function execution time and resource limits.

#3 Not enabling DynamoDB Streams on the table before connecting Lambda.

Wrong approach:

Create Lambda trigger without enabling Streams on DynamoDB table.

Correct approach:

Enable DynamoDB Streams with desired view type before attaching Lambda trigger.

Root cause:

Overlooking prerequisite configuration steps.

Key Takeaways

Lambda with DynamoDB Streams enables automatic, real-time reactions to database changes without managing servers.

DynamoDB Streams capture ordered change events for 24 hours, which Lambda processes in batches triggered automatically.

Understanding shard-based scaling and Lambda concurrency limits is essential for building efficient, reliable stream processing.

Proper error handling and idempotency in Lambda functions prevent data loss and duplicate processing.

This integration exemplifies event-driven architecture, improving application responsiveness and reducing operational overhead.

Practice

1. What is the main purpose of connecting AWS Lambda to DynamoDB Streams?

easy

A. To manually trigger Lambda functions from the AWS console

B. To automatically run code when data in the DynamoDB table changes

C. To backup DynamoDB data to S3 automatically

D. To increase the read capacity of the DynamoDB table

Solution

1. Step 1: Understand DynamoDB Streams role

   DynamoDB Streams capture data changes in the table as events.

2. Step 2: Understand Lambda integration

   Lambda can be triggered automatically by these stream events to run code.

3. Final Answer:

   To automatically run code when data in the DynamoDB table changes -> Option B

4. Quick Check:

   Lambda triggers on stream events = A [OK]

Hint: Streams trigger Lambda on data changes automatically [OK]

Common Mistakes:

• Thinking Lambda triggers manually from console
• Confusing streams with backups
• Assuming streams increase table capacity

2. Which of the following is the correct way to enable DynamoDB Streams on a table?

easy

A. Create a Lambda function and attach it to the table

B. Enable versioning on the DynamoDB table

C. Set the stream specification to NEW_AND_OLD_IMAGES in the table settings

D. Increase the write capacity units of the table

Solution

1. Step 1: Identify how to enable streams

   DynamoDB Streams are enabled by setting the stream specification on the table.

2. Step 2: Choose correct stream view type

   NEW_AND_OLD_IMAGES captures both before and after images of changes.

3. Final Answer:

   Set the stream specification to NEW_AND_OLD_IMAGES in the table settings -> Option C

4. Quick Check:

   Stream enabled via table settings = B [OK]

Hint: Enable streams in table settings, not in Lambda [OK]

Common Mistakes:

• Trying to enable streams by creating Lambda
• Confusing versioning with streams
• Changing capacity units instead of streams

3. Given this Lambda event handler code snippet triggered by DynamoDB Streams:

def lambda_handler(event, context):
    for record in event['Records']:
        if record['eventName'] == 'INSERT':
            print(record['dynamodb']['NewImage']['id']['S'])

What will this code print when a new item with id '123' is added to the DynamoDB table?

medium

A. '123' printed once for the new item

B. 'id' key printed instead of its value

C. An error because 'NewImage' does not exist

D. Nothing, because eventName is not 'INSERT'

Solution

1. Step 1: Understand event structure for INSERT

   For an INSERT event, 'NewImage' contains the new item data including 'id'.

2. Step 2: Analyze code behavior

   The code prints the string value of 'id' from 'NewImage' when eventName is 'INSERT'.

3. Final Answer:

   '123' printed once for the new item -> Option A

4. Quick Check:

   INSERT event prints new id value = D [OK]

Hint: INSERT events have NewImage with new item data [OK]

Common Mistakes:

• Assuming eventName is different
• Expecting OldImage on INSERT
• Confusing key name with value

4. You deployed a Lambda function triggered by DynamoDB Streams but it never runs. Which of the following is the most likely cause?

medium

A. The DynamoDB table does not have streams enabled

B. The Lambda function code has a syntax error

C. The DynamoDB table has no read capacity units

D. The Lambda function has no execution role

Solution

1. Step 1: Check stream enablement

   Lambda triggers on DynamoDB Streams events, so streams must be enabled on the table.

2. Step 2: Consider other causes

   While code errors or roles cause failures, the function would still be invoked if streams exist.

3. Final Answer:

   The DynamoDB table does not have streams enabled -> Option A

4. Quick Check:

   Streams must be enabled for Lambda trigger = C [OK]

Hint: Check if streams are enabled on the table first [OK]

Common Mistakes:

• Assuming code errors prevent invocation
• Ignoring missing streams
• Confusing read capacity with triggers

5. You want your Lambda function to process only updated attributes from DynamoDB Streams and ignore inserts and deletes. Which stream view type and event filter should you use?

hard

A. Enable NEW_AND_OLD_IMAGES stream view and filter events where eventName is 'INSERT'

B. Enable KEYS_ONLY stream view and filter events where eventName is 'INSERT'

C. Enable OLD_IMAGE stream view and filter events where eventName is 'REMOVE'

D. Enable NEW_IMAGE stream view and filter events where eventName is 'MODIFY'

Solution

1. Step 1: Choose stream view for updated data

   NEW_IMAGE provides the new state after update, suitable for processing changes.

2. Step 2: Filter events for updates only

   Filtering on eventName 'MODIFY' ensures only update events trigger Lambda.

3. Final Answer:

   Enable NEW_IMAGE stream view and filter events where eventName is 'MODIFY' -> Option D

4. Quick Check:

   Updates = NEW_IMAGE + 'MODIFY' filter [OK]

Hint: Use NEW_IMAGE and 'MODIFY' event filter for updates only [OK]

Common Mistakes:

• Using KEYS_ONLY which lacks attribute data
• Filtering on INSERT or REMOVE instead of MODIFY
• Choosing OLD_IMAGE which shows old data only