TTL with Streams for archival in DynamoDB

Introduction

TTL helps automatically remove old data. Streams let you capture changes to save that data somewhere else before it disappears.

You want to keep your database clean by deleting old records automatically.

You need to save deleted data for backup or audit before it is removed.

You want to archive expired data to a cheaper storage for future reference.

You want to track changes in your data over time for analysis.

You want to reduce storage costs by removing data but still keep a copy.

Syntax

DynamoDB

1. Enable TTL on a DynamoDB table by specifying an attribute (e.g., 'expireAt') that holds the expiration timestamp.
2. Enable DynamoDB Streams on the table to capture item removals.
3. Use a Lambda function triggered by the stream to process expired items and archive them (e.g., save to S3).

The TTL attribute must be a number representing epoch time in seconds.

Streams capture events like INSERT, MODIFY, and REMOVE. For TTL, focus on REMOVE events.

Examples

This attribute tells DynamoDB when to delete the item.

DynamoDB

TTL attribute: expireAt
Value example: 1686000000 (epoch seconds)

This event shows an item was deleted due to TTL expiration.

DynamoDB

Stream event example:
{
  "eventName": "REMOVE",
  "dynamodb": {
    "OldImage": {"id": {"S": "123"}, "expireAt": {"N": "1686000000"}}
  }
}

When an item is removed, the Lambda archives it.

DynamoDB

Lambda function pseudocode:
if eventName == 'REMOVE':
  archive(event.Records[0].dynamodb.OldImage)

Sample Program

This Lambda function listens to stream events and archives items removed by TTL.

DynamoDB

-- Step 1: Enable TTL on attribute 'expireAt' in DynamoDB console or CLI
-- Step 2: Enable Streams with NEW_AND_OLD_IMAGES
-- Step 3: Example Lambda function code snippet (Node.js)

const AWS = require('aws-sdk');

exports.handler = async (event) => {
  for (const record of event.Records) {
    if (record.eventName === 'REMOVE') {
      const oldItem = record.dynamodb.OldImage;
      // Convert DynamoDB JSON to normal JSON
      const archivedItem = AWS.DynamoDB.Converter.unmarshall(oldItem);
      // Save archivedItem to S3 or another storage
      console.log('Archiving item:', archivedItem);
      // (Archival code here)
    }
  }
};

Important Notes

TTL deletion is not immediate; it can take up to 48 hours.

Streams help you capture deleted items before they vanish.

Archiving lets you keep data for compliance or analysis.

Summary

TTL automatically deletes expired data to keep your table clean.

DynamoDB Streams capture these deletions as REMOVE events.

Use a Lambda triggered by streams to archive deleted data safely.