Stack drift detection in AWS - Deep Dive

Overview - Stack drift detection

What is it?

Stack drift detection is a way to find out if the actual resources in your cloud setup have changed from what you originally planned. It checks if anything was added, removed, or changed outside of your main setup instructions. This helps keep your cloud environment reliable and predictable. It is like a safety check to catch unexpected changes.

Why it matters

Without drift detection, your cloud setup can slowly change without you noticing. This can cause problems like security risks, broken applications, or unexpected costs. Drift detection helps you spot these changes early so you can fix them before they cause trouble. It keeps your cloud environment trustworthy and easier to manage.

Where it fits

Before learning drift detection, you should understand cloud infrastructure basics and how to define your setup using tools like AWS CloudFormation. After drift detection, you can learn about automated monitoring, compliance checks, and remediation strategies to keep your cloud environment healthy.

Mental Model

Core Idea

Stack drift detection compares your cloud setup plan with the real setup to find any differences caused by changes outside your main instructions.

Think of it like...

Imagine you have a recipe to bake a cake, but someone secretly changes the ingredients or steps while you’re not looking. Drift detection is like tasting the cake to see if it matches the original recipe or if something was changed.

┌───────────────────────────────┐
│       Cloud Setup Plan         │
│  (Your original instructions) │
└──────────────┬────────────────┘
               │
               │ Compare
               ▼
┌───────────────────────────────┐
│      Actual Cloud Setup        │
│ (What really exists in cloud)  │
└──────────────┬────────────────┘
               │
               │ Detect Differences
               ▼
┌───────────────────────────────┐
│        Drift Report            │
│ (Shows added, removed, changed│
│        resources)             │
└───────────────────────────────┘

Build-Up - 7 Steps

1

Foundation

What is a Cloud Stack

Concept: Introduce the idea of a cloud stack as a group of resources managed together.

A cloud stack is like a set of building blocks that make up an application or service in the cloud. For example, it can include servers, databases, and networks all defined in one place. Tools like AWS CloudFormation let you write down what you want in your stack so the cloud creates it for you.

Result

You understand that a stack is a planned collection of cloud resources managed as one unit.

Knowing what a stack is helps you see why tracking changes to it matters for keeping your cloud setup consistent.

2

Foundation

How Cloud Stacks Change Over Time

Concept: Explain that cloud stacks can change both through planned updates and unexpected manual changes.

After creating a stack, you might update it by changing the instructions and applying them. But sometimes, people or systems make changes directly in the cloud without updating the instructions. These unplanned changes cause the stack to drift from its original plan.

Result

You realize that cloud stacks can become different from their original design if changes happen outside the main instructions.

Understanding that changes can happen outside your control sets the stage for why drift detection is needed.

3

Intermediate

What is Stack Drift Detection

🤔Before reading on: do you think drift detection finds only missing resources or all types of changes? Commit to your answer.

Concept: Introduce drift detection as a process that finds all differences between the planned and actual stack state.

Stack drift detection compares each resource in your planned stack with what actually exists in the cloud. It checks if resources were added, removed, or modified. AWS CloudFormation provides a drift detection feature that scans your stack and reports any differences.

Result

You can identify any unexpected changes in your cloud stack, not just missing parts but also modifications and additions.

Knowing that drift detection covers all types of changes helps you trust it as a complete check for your cloud setup.

4

Intermediate

How to Run Drift Detection in AWS

🤔Before reading on: do you think drift detection runs automatically or must be started manually? Commit to your answer.

Concept: Teach the practical steps to start drift detection using AWS tools.

In AWS, you can start drift detection from the CloudFormation console, CLI, or API. When started, AWS compares the stack's template and resource properties with the actual resources. After the scan, it shows a report listing any drifted resources and details about the differences.

Result

You know how to trigger drift detection and read its results to find out if your stack has drifted.

Understanding how to run drift detection empowers you to keep your cloud environment consistent and secure.

5

Intermediate

Types of Drift Detected

🤔Before reading on: do you think drift detection can find changes in resource tags or only core settings? Commit to your answer.

Concept: Explain the kinds of changes drift detection can find, including properties and tags.

Drift detection checks resource properties like configuration settings and also resource metadata like tags. It can detect if a resource was deleted, replaced, or modified. However, some resource types or properties might not be supported for drift detection, so not all changes are always found.

Result

You understand the scope and limits of what drift detection can find in your stack.

Knowing what drift detection covers helps you interpret its reports correctly and plan additional checks if needed.

6

Advanced

Handling Drift in Production Environments

🤔Before reading on: do you think drift detection fixes drift automatically or requires manual action? Commit to your answer.

Concept: Discuss best practices for managing drift in real cloud environments.

Drift detection only reports differences; it does not fix them automatically. In production, teams use drift detection regularly to monitor stacks. When drift is found, they investigate and decide whether to update the stack template, revert manual changes, or redeploy resources. Automating remediation requires extra tools or scripts.

Result

You see how drift detection fits into a larger process of cloud management and governance.

Understanding that drift detection is a detection tool, not a fix, clarifies its role and helps you design proper response workflows.

7

Expert

Limitations and Surprises of Drift Detection

🤔Before reading on: do you think drift detection always finds all changes instantly? Commit to your answer.

Concept: Reveal subtle limitations and behaviors of drift detection that can surprise even experienced users.

Drift detection depends on resource type support and may not detect changes in some resources or properties. It also only detects drift at the time of the scan; changes after require new scans. Some changes, like external dependencies or out-of-band effects, are invisible to drift detection. Additionally, drift detection can take time on large stacks and may impact API limits.

Result

You gain a realistic understanding of drift detection’s boundaries and operational considerations.

Knowing these limits prevents overreliance on drift detection and encourages complementary monitoring strategies.

Under the Hood

Stack drift detection works by querying the current state of each resource in the cloud and comparing it to the expected state defined in the stack template and parameters. AWS CloudFormation uses APIs to fetch resource properties and metadata, then performs a property-by-property comparison. Differences are recorded as drift. This process involves mapping resource identifiers, handling resource types differently, and managing API rate limits.

Why designed this way?

Drift detection was designed to help users maintain infrastructure as code integrity without manual audits. It balances thoroughness with performance by focusing on supported resource types and key properties. Alternatives like continuous monitoring or manual checks were less scalable or error-prone. The design trades off full coverage for practical usability and integration with existing CloudFormation workflows.

┌───────────────────────────────┐
│     Stack Template & Params   │
└──────────────┬────────────────┘
               │
               │ Defines expected state
               ▼
┌───────────────────────────────┐
│   CloudFormation Drift Engine  │
│  ┌─────────────────────────┐  │
│  │ Query Actual Resources  │  │
│  └────────────┬────────────┘  │
│               │               │
│  ┌────────────▼────────────┐  │
│  │ Compare Expected vs     │  │
│  │ Actual Properties       │  │
│  └────────────┬────────────┘  │
│               │               │
│  ┌────────────▼────────────┐  │
│  │ Generate Drift Report   │  │
│  └─────────────────────────┘  │
└───────────────────────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does drift detection automatically fix detected drift? Commit to yes or no.

Common Belief: Drift detection automatically corrects any differences it finds in the stack.

Reality: Drift detection only reports differences; it does not change or fix any resources automatically.

Why it matters: Assuming automatic fixes can lead to ignoring drift reports and leaving your cloud environment inconsistent and vulnerable.

Quick: Can drift detection find changes in all AWS resource types? Commit to yes or no.

Common Belief: Drift detection works perfectly on every AWS resource type and property.

Reality: Some resource types or properties are not supported by drift detection, so some changes may go unnoticed.

Why it matters: Relying solely on drift detection can give a false sense of security if unsupported resources drift without detection.

Quick: Does drift detection monitor changes continuously in real time? Commit to yes or no.

Common Belief: Drift detection continuously watches your stack and alerts you instantly on any change.

Reality: Drift detection runs only when manually started or scheduled; it does not provide real-time monitoring.

Why it matters: Expecting real-time alerts can cause delays in detecting drift and responding to issues.

Quick: If a resource is changed outside the stack, will drift detection always find it immediately? Commit to yes or no.

Common Belief: Any out-of-stack change is instantly detected by drift detection.

Reality: Drift detection only finds changes present at the time of the scan; changes after require new scans.

Why it matters: Believing in instant detection can lead to missed drift if scans are infrequent.

Expert Zone

1

Drift detection results can be affected by eventual consistency in cloud APIs, causing temporary false positives or negatives.

2

Some resource properties are considered 'read-only' and changes to them do not cause drift, which can confuse users expecting full detection.

3

Stack drift detection does not detect changes in resources outside the stack even if they affect stack behavior, requiring complementary monitoring.

When NOT to use

Do not rely on drift detection as your only compliance or security tool. For real-time monitoring, use AWS Config or CloudTrail. For unsupported resources, consider custom scripts or third-party tools. Drift detection is also not suitable for very large stacks requiring continuous monitoring due to scan time and API limits.

Production Patterns

In production, teams schedule regular drift detection scans as part of their deployment pipelines or monitoring routines. They integrate drift reports with ticketing systems for manual review. Some automate remediation for common drift cases using infrastructure as code updates or rollback scripts. Drift detection is combined with AWS Config for comprehensive compliance.

Connections

Infrastructure as Code (IaC)

Builds-on

Understanding drift detection deepens your grasp of IaC by showing how to verify that your code matches reality, reinforcing the principle of treating infrastructure like software.

Version Control Systems

Similar pattern

Drift detection is like a version control diff tool but for cloud resources, comparing desired state with actual state to find changes, highlighting the importance of tracking changes in any system.

Quality Control in Manufacturing

Analogous process

Drift detection parallels quality control checks in factories, where products are inspected to ensure they match design specs, showing how verification is key in both physical and digital production.

Common Pitfalls

#1 Ignoring drift detection reports assuming no drift exists.

Wrong approach:

/* No drift detection run or ignoring drift report */

Correct approach:

aws cloudformation detect-stack-drift --stack-name MyStack
aws cloudformation describe-stack-drift-detection-status --stack-drift-detection-id

Root cause:

Misunderstanding that drift detection must be actively run and reviewed to be effective.

#2 Expecting drift detection to fix drift automatically.

Wrong approach:

/* Running drift detection and expecting AWS to fix resources */

Correct approach:

Review drift report and update CloudFormation template or manually fix resources as needed.

Root cause:

Confusing detection with remediation; drift detection only reports differences.

#3 Assuming drift detection covers all resource types and properties.

Wrong approach:

/* Trusting drift detection report as complete without verifying resource support */

Correct approach:

Check AWS documentation for supported resource types and supplement with other monitoring tools.

Root cause:

Lack of awareness about drift detection limitations.

Key Takeaways

Stack drift detection helps find differences between your planned cloud setup and what actually exists, catching unexpected changes.

It does not fix drift automatically; it only reports differences for you to review and act upon.

Drift detection supports many but not all resource types and properties, so it should be part of a broader monitoring strategy.

Running drift detection regularly helps maintain cloud environment consistency, security, and reliability.

Understanding drift detection’s limits and operation helps you use it effectively and avoid common mistakes.