TTL with Streams for archival in DynamoDB - Step-by-Step Execution

Concept Flow - TTL with Streams for archival

Item inserted in DynamoDB table

↓

TTL attribute set with expiry timestamp

↓

Time passes until TTL expires

↓

DynamoDB deletes expired item

↓

Stream captures deletion event

↓

Lambda triggered by Stream event

↓

Lambda archives deleted item to S3 or another store

This flow shows how DynamoDB TTL deletes expired items, streams capture the deletion, and a Lambda archives the deleted data.

Execution Sample

DynamoDB

1. Insert item with TTL attribute (expiry timestamp)
2. Wait until TTL expires
3. DynamoDB deletes item
4. Stream records deletion event
5. Lambda triggered by stream
6. Lambda archives item

This sequence shows how an item with TTL is deleted and archived via DynamoDB Streams and Lambda.

Execution Table

Step	Action	TTL Attribute	Stream Event	Lambda Triggered	Archive Action
1	Insert item	TTL set to future timestamp	No	No	No
2	Wait for TTL expiry	TTL timestamp reached	No	No	No
3	DynamoDB deletes expired item	Expired	Deletion event generated	No	No
4	Stream sends deletion event	Expired	Deletion event present	No	No
5	Lambda triggered by stream	Expired	Deletion event present	Yes	Started archiving
6	Lambda archives item	Expired	Deletion event present	Yes	Item archived
7	Process complete	Expired	Deletion event processed	No	Archive done

💡 Process stops after Lambda archives the deleted item from the stream event.

Variable Tracker

Variable	Start	After Step 1	After Step 3	After Step 5	Final
Item TTL	None	Future timestamp set	Expired	Expired	Expired
Stream Event	None	None	Deletion event generated	Deletion event present	Processed
Lambda Triggered	No	No	No	Yes	No
Archive Status	Not started	Not started	Not started	Started	Done

Key Moments - 3 Insights

Why doesn't the Lambda trigger immediately after setting the TTL attribute?

Because TTL expiry happens asynchronously after the timestamp passes, so the stream deletion event only appears after DynamoDB deletes the expired item (see execution_table step 3 and 4).

What does the stream capture when an item expires?

The stream captures a deletion event for the expired item, which triggers the Lambda to archive it (see execution_table step 4).

Can the Lambda archive the item before the TTL expiry?

No, the Lambda only triggers after the deletion event from the stream, which happens after TTL expiry and item deletion (see execution_table steps 3 to 5).

Visual Quiz - 3 Questions

Look at the execution_table, at which step does the Lambda first get triggered?

A Step 3

B Step 4

C Step 5

D Step 6

💡 Hint

Check the 'Lambda Triggered' column in execution_table rows.

According to variable_tracker, what is the Archive Status after Step 5?

A Not started

B Started

C Done

D Failed

💡 Hint

Look at the 'Archive Status' row and the 'After Step 5' column.

If the TTL attribute is never set, what happens to the stream event?

A No deletion event is generated

B Lambda triggers without event

C Deletion event is generated anyway

D Item is archived immediately

💡 Hint

Refer to the flow where TTL expiry causes deletion and stream event.

Concept Snapshot

DynamoDB TTL deletes expired items automatically.
Deletion triggers a stream event.
Lambda listens to stream events.
Lambda archives deleted items.
This automates archival after TTL expiry.

Full Transcript

This visual execution shows how DynamoDB TTL works with Streams for archival. First, an item is inserted with a TTL attribute set to a future timestamp. Time passes until the TTL expires. DynamoDB then deletes the expired item. This deletion generates a stream event. The stream event triggers a Lambda function. The Lambda archives the deleted item to an external store like S3. Variables like TTL attribute, stream event presence, Lambda trigger, and archive status change step-by-step. Key moments clarify that Lambda triggers only after deletion, and the stream captures deletion events only after TTL expiry. The quiz tests understanding of when Lambda triggers, archive status progression, and what happens if TTL is not set.

Practice

1. What is the main purpose of enabling TTL (Time to Live) on a DynamoDB table?

easy

A. Increase read capacity units for faster queries

B. Encrypt data at rest for security

C. Create backups of the table automatically

D. Automatically delete expired items to keep the table clean

Solution

1. Step 1: Understand TTL function

   TTL marks items with an expiration timestamp so DynamoDB can delete them automatically after expiry.

2. Step 2: Identify TTL's main benefit

   This automatic deletion helps keep the table clean without manual intervention.

3. Final Answer:

   Automatically delete expired items to keep the table clean -> Option D

4. Quick Check:

   TTL = automatic deletion [OK]

Hint: TTL deletes expired data automatically [OK]

Common Mistakes:

• Confusing TTL with encryption
• Thinking TTL creates backups
• Assuming TTL increases capacity

2. Which of the following is the correct way to enable TTL on a DynamoDB table using AWS CLI?

easy

A. aws dynamodb enable-ttl --table MyTable --attribute expiryDate

B. aws dynamodb set-ttl --table-name MyTable --ttl-attribute expiryDate

C. aws dynamodb update-time-to-live --table-name MyTable --time-to-live-specification Enabled=true,AttributeName=expiryDate

D. aws dynamodb activate-ttl --table MyTable --attributeName expiryDate

Solution

1. Step 1: Recall AWS CLI TTL command

   The correct command is 'update-time-to-live' with parameters for table name and TTL specification.

2. Step 2: Match correct syntax

   aws dynamodb update-time-to-live --table-name MyTable --time-to-live-specification Enabled=true,AttributeName=expiryDate uses the exact syntax: --time-to-live-specification Enabled=true,AttributeName=expiryDate.

3. Final Answer:

   aws dynamodb update-time-to-live --table-name MyTable --time-to-live-specification Enabled=true,AttributeName=expiryDate -> Option C

4. Quick Check:

   Correct CLI command = aws dynamodb update-time-to-live --table-name MyTable --time-to-live-specification Enabled=true,AttributeName=expiryDate [OK]

Hint: Use 'update-time-to-live' with Enabled=true [OK]

Common Mistakes:

• Using non-existent commands like enable-ttl
• Wrong parameter names
• Missing 'time-to-live-specification' flag

3. Given a DynamoDB Stream enabled on a table with TTL, what type of event will appear in the stream when an item expires and is deleted?

medium

A. INSERT

B. REMOVE

C. MODIFY

D. UPDATE

Solution

1. Step 1: Understand TTL deletion effect on streams

   When TTL deletes an item, DynamoDB Streams capture this as a REMOVE event because the item is removed.

2. Step 2: Identify correct event type

   INSERT and MODIFY happen on new or updated items, not deletions. REMOVE is the correct event for deletions.

3. Final Answer:

   REMOVE -> Option B

4. Quick Check:

   TTL deletion = REMOVE event [OK]

Hint: TTL deletions trigger REMOVE events in streams [OK]

Common Mistakes:

• Choosing INSERT or MODIFY for deletions
• Confusing UPDATE with MODIFY
• Assuming TTL triggers no stream events

4. You wrote a Lambda function triggered by DynamoDB Streams to archive deleted items. However, the Lambda never receives any events for expired items. What is the most likely cause?

medium

A. The stream is not enabled or not set to capture REMOVE events

B. TTL is not enabled on the table

C. Lambda function has no permissions to read the table

D. The table has no primary key defined

Solution

1. Step 1: Check stream configuration

   If the Lambda never receives events, the stream might be disabled or not configured to capture REMOVE events, which TTL deletions generate.

2. Step 2: Verify other causes

   TTL must be enabled, but if streams are off or missing REMOVE events, Lambda won't get notifications. Permissions or missing primary key would cause other errors.

3. Final Answer:

   The stream is not enabled or not set to capture REMOVE events -> Option A

4. Quick Check:

   Missing REMOVE events in stream = The stream is not enabled or not set to capture REMOVE events [OK]

Hint: Ensure stream captures REMOVE events for TTL deletions [OK]

Common Mistakes:

• Ignoring stream settings
• Assuming permissions cause no events
• Overlooking TTL enablement

5. You want to archive expired DynamoDB items to S3 using a Lambda triggered by Streams. Which approach ensures no data loss and handles possible Lambda failures?

hard

A. Process REMOVE events in Lambda and write directly to S3, retrying on failure

B. Use TTL to delete items, ignore streams, and periodically scan table for expired items

C. Disable TTL and manually delete items after archiving

D. Process INSERT events in Lambda and archive items before expiration

Solution

1. Step 1: Understand archival with TTL and streams

   TTL deletes expired items, streams capture REMOVE events. Lambda triggered by these events can archive data safely.

2. Step 2: Ensure reliability and no data loss

   Processing REMOVE events and writing to S3 with retries ensures data is archived even if Lambda fails temporarily.

3. Step 3: Evaluate other options

   Use TTL to delete items, ignore streams, and periodically scan table for expired items ignores streams and risks missing data. Disable TTL and manually delete items after archiving disables TTL, losing automatic cleanup. Process INSERT events in Lambda and archive items before expiration archives before expiration, which is not TTL-based archival.

4. Final Answer:

   Process REMOVE events in Lambda and write directly to S3, retrying on failure -> Option A

5. Quick Check:

   Reliable archival = Process REMOVE events in Lambda and write directly to S3, retrying on failure [OK]

Hint: Archive REMOVE events with retries for safety [OK]

Common Mistakes:

• Ignoring stream REMOVE events
• Disabling TTL unnecessarily
• Archiving on INSERT instead of REMOVE