TTL with Streams for archival in DynamoDB - Deep Dive

Overview - TTL with Streams for archival

What is it?

TTL (Time to Live) is a feature in DynamoDB that automatically deletes items after a specified time. Streams capture changes in the table, including deletions caused by TTL. Combining TTL with Streams allows you to archive deleted data before it disappears permanently. This helps keep your database clean while preserving important historical records.

Why it matters

Without TTL, expired data would accumulate, making the database large and slow. Without Streams, once data is deleted by TTL, it is lost forever. Using TTL with Streams solves this by automatically cleaning old data and letting you archive it elsewhere. This keeps your system efficient and your data safe for future analysis or compliance.

Where it fits

Before learning this, you should understand basic DynamoDB tables and how TTL works. After this, you can explore advanced data lifecycle management, event-driven architectures using Lambda, and data warehousing for archived data.

Mental Model

Core Idea

TTL automatically removes expired data, and Streams capture those removals so you can archive the data before it disappears.

Think of it like...

It's like setting a timer on food in your fridge (TTL) that throws it away when expired, while a camera (Streams) records what was thrown out so you can keep a photo album of past meals.

┌─────────────┐       ┌───────────────┐       ┌───────────────┐
│ DynamoDB    │       │ TTL Feature   │       │ Streams       │
│ Table       │──────▶│ Deletes Items │──────▶│ Captures      │
│ (Data)      │       │ After Expiry  │       │ Deletion Event│
└─────────────┘       └───────────────┘       └───────────────┘
                                         │
                                         ▼
                               ┌───────────────────┐
                               │ Archival System   │
                               │ (e.g., S3, Redshift)│
                               └───────────────────┘

Build-Up - 6 Steps

1

Foundation

Understanding DynamoDB TTL Basics

Concept: Learn what TTL is and how it automatically deletes expired items.

TTL is a timestamp attribute you add to your DynamoDB items. When the current time passes this timestamp, DynamoDB marks the item for deletion. This process runs in the background and helps keep your table size manageable without manual cleanup.

Result

Expired items are automatically removed from the table after their TTL timestamp passes.

Understanding TTL helps you automate data cleanup, reducing manual work and storage costs.

2

Foundation

Introduction to DynamoDB Streams

Concept: Streams capture changes in your DynamoDB table as a sequence of events.

When items are added, updated, or deleted, Streams record these changes in order. You can enable Streams on your table and configure what information is captured, such as old and new images of the item.

Result

You get a real-time feed of all changes happening in your table.

Streams let you react to data changes, enabling workflows like archiving or triggering other processes.

3

Intermediate

How TTL Deletions Appear in Streams

🤔Before reading on: Do you think TTL deletions generate stream events like manual deletes? Commit to your answer.

Concept: TTL deletions do generate stream events, but only as delete records without old images by default.

When TTL deletes an item, DynamoDB Streams records a DELETE event. However, the old image (the deleted item data) is only available if you enable 'Old Image' in the stream settings. This is crucial for archiving because you need the data before it disappears.

Result

You receive delete events in Streams for TTL removals, which can be used to archive the deleted data.

Knowing TTL deletions appear in Streams allows you to capture and archive data that would otherwise be lost.

4

Intermediate

Setting Up Archival with Lambda and Streams

🤔Before reading on: Will a Lambda triggered by Streams see the full deleted item data by default? Commit to your answer.

Concept: Use a Lambda function triggered by Streams to process TTL delete events and archive the data to another storage.

Enable Streams with 'Old Image' to get the full deleted item. Configure a Lambda to trigger on delete events. The Lambda reads the old image and writes it to an archival system like S3 or Redshift. This preserves data before it's permanently gone.

Result

Deleted items are archived automatically as soon as TTL removes them.

Automating archival with Lambda and Streams ensures no data is lost and reduces manual intervention.

5

Advanced

Handling Stream Event Ordering and Duplication

🤔Before reading on: Do you think DynamoDB Streams guarantees exactly-once delivery and strict ordering? Commit to your answer.

Concept: Streams guarantee ordering per shard but can deliver events more than once, so your archival must handle duplicates and ordering carefully.

DynamoDB Streams deliver events in order within each shard but may retry events, causing duplicates. Your Lambda should be idempotent—able to process the same event multiple times without side effects. Use unique keys or checksums in your archival to avoid duplicate records.

Result

Your archival system remains consistent and accurate despite retries or duplicates.

Understanding Streams' delivery model prevents data corruption and ensures reliable archival.

6

Expert

Optimizing Cost and Performance in TTL Archival

🤔Before reading on: Is it cheaper to archive every deleted item immediately or batch archive later? Commit to your answer.

Concept: Balancing immediate archival with batching can optimize costs and performance in large-scale systems.

Archiving every TTL deletion immediately via Lambda can increase costs and throttling risk. Instead, buffer events in a queue or temporary store, then batch archive during off-peak times. Use DynamoDB Streams with Kinesis or SQS for buffering. This reduces Lambda invocations and storage API calls.

Result

You achieve cost-effective, scalable archival without losing data or performance.

Knowing how to balance immediacy and batching helps build efficient, production-ready archival pipelines.

Under the Hood

TTL works by DynamoDB scanning items with expired timestamps and marking them for deletion asynchronously. These deletions trigger stream events that record the change. Streams partition data into shards, maintaining order per shard and storing events for 24 hours. Lambda functions poll these shards to process events. The system ensures eventual consistency and retries on failure, requiring idempotent processing.

Why designed this way?

TTL was designed to automate data lifecycle management without burdening users with manual cleanup. Streams provide a near real-time change log to enable reactive architectures. The asynchronous, partitioned design balances scalability, performance, and cost. Alternatives like synchronous deletes or polling would not scale well or would increase latency and cost.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ DynamoDB Item │       │ TTL Expiry    │       │ Item Marked   │
│ with TTL      │──────▶│ Detected by   │──────▶│ for Deletion  │
└───────────────┘       │ Background   │       └───────────────┘
                        └───────────────┘               │
                                                        ▼
                                              ┌─────────────────┐
                                              │ DynamoDB Stream │
                                              │ DELETE Event    │
                                              └─────────────────┘
                                                        │
                                                        ▼
                                              ┌─────────────────┐
                                              │ Lambda Function │
                                              │ Processes Event │
                                              └─────────────────┘
                                                        │
                                                        ▼
                                              ┌─────────────────┐
                                              │ Archival Store  │
                                              │ (S3, Redshift)  │
                                              └─────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does TTL deletion immediately remove the item from the table? Commit yes or no.

Common Belief: TTL deletes items instantly at the expiration time.

Reality: TTL deletions happen asynchronously and can take up to 48 hours to remove expired items.

Why it matters: Assuming immediate deletion can cause stale data to appear in queries, leading to incorrect application behavior.

Quick: Do TTL deletions generate stream events by default? Commit yes or no.

Common Belief: TTL deletions do not generate stream events because they are automatic.

Reality: TTL deletions do generate DELETE events in Streams if Streams are enabled.

Why it matters: Missing this means you might not archive deleted data, losing important historical records.

Quick: Are DynamoDB Streams guaranteed to deliver each event exactly once? Commit yes or no.

Common Belief: Streams deliver each event exactly once and in perfect order.

Reality: Streams guarantee order per shard but can deliver events more than once (at-least-once delivery).

Why it matters: Not handling duplicates can cause archival systems to store repeated data, wasting space and causing confusion.

Quick: Can you rely on Streams to keep deleted item data forever? Commit yes or no.

Common Belief: Stream records are stored indefinitely, so you can archive anytime.

Reality: Stream records are kept only for 24 hours, so archival must happen promptly.

Why it matters: Delaying archival beyond 24 hours risks losing the deleted data forever.

Expert Zone

1

TTL deletions do not trigger DynamoDB Streams if Streams are disabled or configured without 'Old Image', so archival requires careful setup.

2

Lambda functions processing Streams must be idempotent and handle retries gracefully to avoid duplicate archival entries.

3

Partition keys affect stream shard distribution, influencing event ordering and parallelism in archival processing.

When NOT to use

Avoid using TTL with Streams for archival if your data requires immediate deletion without any retention. Instead, use manual deletion with backup snapshots. Also, for extremely high-throughput tables, consider dedicated change data capture tools or AWS Database Migration Service for archival.

Production Patterns

In production, teams use TTL with Streams combined with Lambda to move expired data to S3 in Parquet format for cost-effective long-term storage. They implement deduplication logic in Lambda and batch writes to reduce costs. Monitoring and alerting on stream lag and Lambda errors ensure reliable archival.

Connections

Event-driven Architecture

Builds-on

Understanding TTL with Streams helps grasp how event-driven systems react to data changes automatically, enabling scalable and decoupled workflows.

Data Lifecycle Management

Same pattern

TTL with Streams is a practical example of automating data lifecycle stages: creation, expiration, deletion, and archival.

Supply Chain Management

Analogous process

Just like expired products are removed from shelves but recorded for inventory and compliance, TTL with Streams removes data but archives it for future use.

Common Pitfalls

#1 Not enabling Streams or configuring them without 'Old Image' when using TTL for archival.

Wrong approach:

DynamoDB table with TTL enabled but Streams disabled or set to KEYS_ONLY.

Correct approach:

Enable DynamoDB Streams with 'OLD_IMAGE' to capture full deleted item data for archival.

Root cause:

Misunderstanding that TTL deletions automatically appear in Streams with full data.

#2 Assuming DynamoDB Streams deliver events exactly once and ignoring duplicate processing.

Wrong approach:

Lambda function archives data without checking for duplicates or idempotency.

Correct approach:

Implement idempotent Lambda logic using unique keys or checksums to avoid duplicate archival entries.

Root cause:

Not knowing Streams can deliver the same event multiple times.

#3 Delaying archival processing beyond Streams retention period.

Wrong approach:

Archival system polls Streams after 48 hours expecting to find delete events.

Correct approach:

Process Streams events promptly within 24 hours to ensure no data loss.

Root cause:

Unawareness of Streams' 24-hour retention limit.

Key Takeaways

TTL in DynamoDB automatically deletes expired items to keep tables clean and efficient.

DynamoDB Streams capture TTL deletions as events, enabling you to archive data before it is lost.

Enabling Streams with 'Old Image' is essential to get full deleted item data for archival.

Streams deliver events at least once and in order per shard, so your archival must handle duplicates and ordering.

Prompt processing of Streams events is critical because records expire after 24 hours.

Practice

1. What is the main purpose of enabling TTL (Time to Live) on a DynamoDB table?

easy

A. Increase read capacity units for faster queries

B. Encrypt data at rest for security

C. Create backups of the table automatically

D. Automatically delete expired items to keep the table clean

Solution

1. Step 1: Understand TTL function

   TTL marks items with an expiration timestamp so DynamoDB can delete them automatically after expiry.

2. Step 2: Identify TTL's main benefit

   This automatic deletion helps keep the table clean without manual intervention.

3. Final Answer:

   Automatically delete expired items to keep the table clean -> Option D

4. Quick Check:

   TTL = automatic deletion [OK]

Hint: TTL deletes expired data automatically [OK]

Common Mistakes:

• Confusing TTL with encryption
• Thinking TTL creates backups
• Assuming TTL increases capacity

2. Which of the following is the correct way to enable TTL on a DynamoDB table using AWS CLI?

easy

A. aws dynamodb enable-ttl --table MyTable --attribute expiryDate

B. aws dynamodb set-ttl --table-name MyTable --ttl-attribute expiryDate

C. aws dynamodb update-time-to-live --table-name MyTable --time-to-live-specification Enabled=true,AttributeName=expiryDate

D. aws dynamodb activate-ttl --table MyTable --attributeName expiryDate

Solution

1. Step 1: Recall AWS CLI TTL command

   The correct command is 'update-time-to-live' with parameters for table name and TTL specification.

2. Step 2: Match correct syntax

   aws dynamodb update-time-to-live --table-name MyTable --time-to-live-specification Enabled=true,AttributeName=expiryDate uses the exact syntax: --time-to-live-specification Enabled=true,AttributeName=expiryDate.

3. Final Answer:

   aws dynamodb update-time-to-live --table-name MyTable --time-to-live-specification Enabled=true,AttributeName=expiryDate -> Option C

4. Quick Check:

   Correct CLI command = aws dynamodb update-time-to-live --table-name MyTable --time-to-live-specification Enabled=true,AttributeName=expiryDate [OK]

Hint: Use 'update-time-to-live' with Enabled=true [OK]

Common Mistakes:

• Using non-existent commands like enable-ttl
• Wrong parameter names
• Missing 'time-to-live-specification' flag

3. Given a DynamoDB Stream enabled on a table with TTL, what type of event will appear in the stream when an item expires and is deleted?

medium

A. INSERT

B. REMOVE

C. MODIFY

D. UPDATE

Solution

1. Step 1: Understand TTL deletion effect on streams

   When TTL deletes an item, DynamoDB Streams capture this as a REMOVE event because the item is removed.

2. Step 2: Identify correct event type

   INSERT and MODIFY happen on new or updated items, not deletions. REMOVE is the correct event for deletions.

3. Final Answer:

   REMOVE -> Option B

4. Quick Check:

   TTL deletion = REMOVE event [OK]

Hint: TTL deletions trigger REMOVE events in streams [OK]

Common Mistakes:

• Choosing INSERT or MODIFY for deletions
• Confusing UPDATE with MODIFY
• Assuming TTL triggers no stream events

4. You wrote a Lambda function triggered by DynamoDB Streams to archive deleted items. However, the Lambda never receives any events for expired items. What is the most likely cause?

medium

A. The stream is not enabled or not set to capture REMOVE events

B. TTL is not enabled on the table

C. Lambda function has no permissions to read the table

D. The table has no primary key defined

Solution

1. Step 1: Check stream configuration

   If the Lambda never receives events, the stream might be disabled or not configured to capture REMOVE events, which TTL deletions generate.

2. Step 2: Verify other causes

   TTL must be enabled, but if streams are off or missing REMOVE events, Lambda won't get notifications. Permissions or missing primary key would cause other errors.

3. Final Answer:

   The stream is not enabled or not set to capture REMOVE events -> Option A

4. Quick Check:

   Missing REMOVE events in stream = The stream is not enabled or not set to capture REMOVE events [OK]

Hint: Ensure stream captures REMOVE events for TTL deletions [OK]

Common Mistakes:

• Ignoring stream settings
• Assuming permissions cause no events
• Overlooking TTL enablement

5. You want to archive expired DynamoDB items to S3 using a Lambda triggered by Streams. Which approach ensures no data loss and handles possible Lambda failures?

hard

A. Process REMOVE events in Lambda and write directly to S3, retrying on failure

B. Use TTL to delete items, ignore streams, and periodically scan table for expired items

C. Disable TTL and manually delete items after archiving

D. Process INSERT events in Lambda and archive items before expiration

Solution

1. Step 1: Understand archival with TTL and streams

   TTL deletes expired items, streams capture REMOVE events. Lambda triggered by these events can archive data safely.

2. Step 2: Ensure reliability and no data loss

   Processing REMOVE events and writing to S3 with retries ensures data is archived even if Lambda fails temporarily.

3. Step 3: Evaluate other options

   Use TTL to delete items, ignore streams, and periodically scan table for expired items ignores streams and risks missing data. Disable TTL and manually delete items after archiving disables TTL, losing automatic cleanup. Process INSERT events in Lambda and archive items before expiration archives before expiration, which is not TTL-based archival.

4. Final Answer:

   Process REMOVE events in Lambda and write directly to S3, retrying on failure -> Option A

5. Quick Check:

   Reliable archival = Process REMOVE events in Lambda and write directly to S3, retrying on failure [OK]

Hint: Archive REMOVE events with retries for safety [OK]

Common Mistakes:

• Ignoring stream REMOVE events
• Disabling TTL unnecessarily
• Archiving on INSERT instead of REMOVE