SNS and SQS integration pattern (fan-out) in AWS - Deep Dive

Overview - SNS and SQS integration pattern (fan-out)

What is it?

SNS (Simple Notification Service) and SQS (Simple Queue Service) are AWS services used to send and receive messages. The fan-out pattern uses SNS to send a single message to multiple SQS queues at once. This allows many systems to get the same message independently and process it at their own pace.

Why it matters

Without this pattern, sending the same message to many receivers would require sending it multiple times, which is slow and error-prone. Fan-out makes it easy to broadcast messages reliably and scale processing by separating message delivery from message handling.

Where it fits

Learners should know basic AWS messaging services like SNS and SQS separately. After this, they can explore advanced messaging patterns, event-driven architectures, and serverless workflows that use fan-out for scaling.

Mental Model

Core Idea

Fan-out means sending one message to many receivers by connecting one SNS topic to multiple SQS queues.

Think of it like...

Imagine a town crier shouting a message in the town square (SNS), and many people with their own mailboxes (SQS queues) each get a copy to read and act on later.

SNS Topic
  │
  ├──> SQS Queue 1
  ├──> SQS Queue 2
  └──> SQS Queue 3

Each queue receives the same message independently.

Build-Up - 6 Steps

1

Foundation

Understanding SNS and SQS Basics

Concept: Learn what SNS and SQS are and how they work individually.

SNS is a service that sends messages to subscribers immediately. SQS is a service that stores messages in a queue until a receiver processes them. SNS pushes messages; SQS holds messages until pulled.

Result

You know SNS is for broadcasting messages instantly, and SQS is for storing messages safely until processed.

Understanding the difference between pushing (SNS) and pulling (SQS) messages is key to grasping how they work together.

2

Foundation

How SNS and SQS Connect

Concept: SNS can send messages to SQS queues by subscribing them to an SNS topic.

You create an SNS topic and multiple SQS queues. Then you subscribe each queue to the SNS topic. When a message is published to the topic, SNS copies it to all subscribed queues.

Result

One message sent to SNS is duplicated and stored in each SQS queue.

This connection allows decoupling message sending from processing, enabling multiple independent consumers.

3

Intermediate

Benefits of Fan-Out Pattern

🤔Before reading on: do you think fan-out improves message delivery speed or processing scalability? Commit to your answer.

Concept: Fan-out improves scalability by letting multiple systems process the same message independently and at their own pace.

With fan-out, each SQS queue acts as a separate buffer. Systems can process messages without blocking each other. This also increases fault tolerance because one system's failure doesn't affect others.

Result

Systems scale better and are more reliable when using fan-out.

Knowing fan-out separates message distribution from processing helps design systems that handle load spikes gracefully.

4

Intermediate

Configuring SNS to SQS Fan-Out

🤔Before reading on: do you think you need special permissions for SNS to send messages to SQS? Commit to your answer.

Concept: SNS needs permission to send messages to each SQS queue, configured via access policies.

Each SQS queue must allow the SNS topic to send messages by setting an access policy. Without this, messages won't be delivered. This ensures security by controlling who can send messages to queues.

Result

SNS successfully delivers messages to all subscribed SQS queues.

Understanding permissions prevents common delivery failures and secures message flow.

5

Advanced

Handling Message Duplication and Ordering

🤔Before reading on: do you think messages delivered via fan-out are guaranteed to arrive in order and only once? Commit to your answer.

Concept: Fan-out can cause message duplication and unordered delivery; systems must handle these cases.

SNS and SQS do not guarantee exactly-once delivery or strict ordering. Each queue may receive duplicates or messages out of order. Consumers should be designed to detect duplicates and handle unordered messages safely.

Result

Systems remain correct and reliable despite duplicates or ordering issues.

Knowing the limits of delivery guarantees helps design robust message processing.

6

Expert

Optimizing Fan-Out for Large Scale Systems

🤔Before reading on: do you think adding more queues always improves performance linearly? Commit to your answer.

Concept: Scaling fan-out requires balancing the number of queues, message volume, and processing capacity to avoid bottlenecks.

Too many queues increase management overhead and SNS delivery latency. Also, high message volume can cause throttling. Using batching, dead-letter queues, and monitoring helps optimize performance and reliability.

Result

Fan-out scales efficiently without overwhelming resources or losing messages.

Understanding trade-offs in scaling fan-out prevents costly mistakes in production.

Under the Hood

SNS acts as a publisher that receives messages and immediately pushes copies to all subscribed endpoints, including SQS queues. Each SQS queue stores messages durably until consumers poll and delete them. SNS uses HTTP APIs to deliver messages to SQS, which stores them redundantly across multiple servers for durability.

Why designed this way?

SNS and SQS were designed separately to solve different problems: SNS for real-time notifications and SQS for reliable message storage. Integrating them via subscriptions allows combining their strengths: fast broadcast and reliable queuing. This separation also allows independent scaling and failure isolation.

┌─────────────┐       ┌─────────────┐
│  Publisher  │──────▶│   SNS Topic │
└─────────────┘       └─────┬───────┘
                             │
          ┌──────────────────┼──────────────────┐
          │                  │                  │
  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐
  │  SQS Queue 1│    │  SQS Queue 2│    │  SQS Queue 3│
  └─────────────┘    └─────────────┘    └─────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does SNS guarantee messages arrive in order to all SQS queues? Commit to yes or no.

Common Belief: SNS guarantees that messages arrive in the exact order they were sent to all subscribers.

Reality: SNS does not guarantee message order; messages can arrive out of order or at different times in each queue.

Why it matters: Assuming order can cause bugs in systems that rely on sequence, leading to incorrect processing results.

Quick: Do you think SNS delivers messages exactly once to each SQS queue? Commit to yes or no.

Common Belief: SNS delivers each message exactly once to every subscribed SQS queue.

Reality: SNS and SQS provide at-least-once delivery, so duplicates can occur and consumers must handle them.

Why it matters: Ignoring duplicates can cause repeated processing, data corruption, or inconsistent states.

Quick: Can SNS send messages to SQS queues without any permissions set? Commit to yes or no.

Common Belief: SNS can send messages to any SQS queue without special permissions.

Reality: SQS queues require explicit access policies allowing SNS topics to send messages; otherwise, delivery fails.

Why it matters: Missing permissions cause silent message loss and debugging delays.

Quick: Does adding more SQS queues always improve fan-out performance linearly? Commit to yes or no.

Common Belief: More SQS queues always mean better performance and scalability in fan-out.

Reality: Too many queues can increase latency and management complexity, causing diminishing returns or bottlenecks.

Why it matters: Overprovisioning queues wastes resources and can degrade system performance.

Expert Zone

1

SNS message filtering lets queues receive only messages matching specific attributes, reducing unnecessary processing.

2

Dead-letter queues for SQS help isolate and analyze messages that fail processing, improving reliability.

3

Using FIFO (First-In-First-Out) queues with SNS requires special configuration and has different guarantees than standard queues.

When NOT to use

Avoid fan-out when message ordering and exactly-once processing are critical; consider using FIFO queues or event streaming platforms like Kafka instead. Also, if only one consumer needs the message, direct SQS or SNS usage without fan-out is simpler.

Production Patterns

In real systems, fan-out is used to trigger multiple microservices from one event, distribute workloads across teams, or replicate data changes. Monitoring and alerting on queue depth and delivery failures are standard practices.

Connections

Publish-Subscribe Messaging Pattern

SNS and SQS fan-out is a concrete AWS implementation of the publish-subscribe pattern.

Understanding this pattern helps grasp how messages flow from one sender to many independent receivers.

Event-Driven Architecture

Fan-out enables event-driven systems where multiple services react independently to the same event.

Knowing fan-out clarifies how loosely coupled services communicate asynchronously.

Broadcasting in Computer Networks

Fan-out is similar to network broadcasting where one message is sent to many devices simultaneously.

Recognizing this connection helps understand scalability and reliability challenges in message distribution.

Common Pitfalls

#1 Missing SQS queue permissions for SNS topic.

Wrong approach:

{
"Version": "2012-10-17",
"Statement": []
} // Empty policy, no permissions granted

Correct approach:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {"Service": "sns.amazonaws.com"},
"Action": "sqs:SendMessage",
"Resource": "arn:aws:sqs:region:account-id:queue-name",
"Condition": {
"ArnEquals": {"aws:SourceArn": "arn:aws:sns:region:account-id:topic-name"}
}
}
]
}

Root cause:

Learners often forget that SQS queues require explicit policies to accept messages from SNS, causing silent failures.

#2 Assuming message order is preserved across queues.

Wrong approach:

Designing consumers that process messages assuming strict order without checks.

Correct approach:

Implementing idempotency and sequence checks in consumers to handle unordered messages.

Root cause:

Misunderstanding SNS and SQS delivery guarantees leads to fragile processing logic.

#3 Ignoring duplicate messages in consumers.

Wrong approach:

Processing every received message as unique without deduplication.

Correct approach:

Using message IDs or deduplication logic to detect and ignore duplicates.

Root cause:

Believing SNS and SQS guarantee exactly-once delivery causes data inconsistencies.

Key Takeaways

SNS and SQS fan-out pattern lets one message reach many independent queues for scalable processing.

SNS pushes messages immediately, while SQS stores them until consumers pull and process.

Proper permissions are essential for SNS to deliver messages to SQS queues securely.

Fan-out does not guarantee message order or exactly-once delivery; consumers must handle duplicates and unordered messages.

Scaling fan-out requires balancing queue count, message volume, and processing capacity to avoid bottlenecks.

Practice

1. What is the main purpose of using SNS with multiple SQS queues in a fan-out pattern?

easy

A. To send the same message to multiple queues for parallel processing

B. To store messages permanently in one queue

C. To send messages only to one queue at a time

D. To delete messages immediately after sending

Solution

1. Step 1: Understand SNS role in fan-out

   SNS publishes messages to multiple subscribers simultaneously.

2. Step 2: Understand SQS role in fan-out

   Each SQS queue receives and holds messages independently for processing.

3. Final Answer:

   To send the same message to multiple queues for parallel processing -> Option A

4. Quick Check:

   SNS fan-out = multiple queues get same message [OK]

Hint: SNS fan-out means one message to many queues [OK]

Common Mistakes:

• Thinking SNS sends to only one queue
• Confusing message storage with deletion
• Assuming messages are deleted immediately

2. Which of the following is the correct way to subscribe an SQS queue to an SNS topic using AWS CLI?

easy

A. aws sns subscribe --topic-arn --protocol sqs --notification-endpoint

B. aws sqs subscribe --topic-arn --queue-url

C. aws sns create-subscription --topic --queue

D. aws sns add-queue --topic-arn --queue-arn

Solution

1. Step 1: Identify correct AWS CLI command

   The AWS CLI command to subscribe is 'aws sns subscribe'.

2. Step 2: Check parameters for SQS subscription

   Use '--protocol sqs' and '--notification-endpoint' with the queue ARN.

3. Final Answer:

   aws sns subscribe --topic-arn <topic-arn> --protocol sqs --notification-endpoint <queue-arn> -> Option A

4. Quick Check:

   Subscribe SQS to SNS uses 'aws sns subscribe' with protocol sqs [OK]

Hint: Subscribe SQS with 'aws sns subscribe' and protocol sqs [OK]

Common Mistakes:

• Using 'aws sqs subscribe' which does not exist
• Confusing queue URL with queue ARN
• Using wrong command like 'aws sns add-queue'

3. Given an SNS topic with two subscribed SQS queues, if a message is published to SNS, what will be the state of the message in the queues?

medium

A. The message is deleted from SNS and not delivered to any queue.

B. The message appears in only one queue randomly.

C. The message appears in both queues independently, waiting to be processed.

D. The message is duplicated in one queue only.

Solution

1. Step 1: Understand SNS fan-out delivery

   SNS sends a copy of the message to each subscribed queue.

2. Step 2: Understand SQS message handling

   Each queue holds its own copy until processed or deleted.

3. Final Answer:

   The message appears in both queues independently, waiting to be processed. -> Option C

4. Quick Check:

   Fan-out means each queue gets its own message copy [OK]

Hint: SNS fan-out sends message copy to each queue [OK]

Common Mistakes:

• Thinking message goes to only one queue
• Assuming message is deleted immediately
• Believing message duplicates in one queue only

4. You configured an SNS topic with two SQS queues subscribed, but only one queue receives messages. What is the most likely cause?

medium

A. SNS only sends messages to one queue by design

B. The second queue's subscription is not confirmed

C. The message size exceeds SNS limits

D. The first queue deleted the message too fast

Solution

1. Step 1: Check subscription confirmation

   SQS subscriptions to SNS require confirmation to activate delivery.

2. Step 2: Understand SNS delivery behavior

   Without confirmation, SNS does not send messages to that queue.

3. Final Answer:

   The second queue's subscription is not confirmed -> Option B

4. Quick Check:

   Unconfirmed subscription blocks message delivery [OK]

Hint: Confirm all subscriptions to receive messages [OK]

Common Mistakes:

• Assuming SNS sends to only one queue
• Blaming message size without checking subscription
• Thinking message deletion affects delivery

5. You want to build a system where an SNS topic sends messages to three SQS queues. One queue processes orders, another sends notifications, and the third logs events. How should you configure the system to ensure reliable processing and no message loss?

hard

A. Subscribe only the order queue to SNS and forward messages to others manually

B. Send messages directly to each SQS queue without SNS

C. Use one SQS queue for all processing and filter messages inside the consumer

D. Subscribe all three SQS queues to the SNS topic, enable dead-letter queues for each, and set appropriate message retention

Solution

1. Step 1: Use SNS fan-out to send messages to all queues

   Subscribing all three SQS queues to the SNS topic ensures each gets the message.

2. Step 2: Enable dead-letter queues and set retention

   Dead-letter queues catch failed messages; retention ensures messages stay until processed.

3. Final Answer:

   Subscribe all three SQS queues to the SNS topic, enable dead-letter queues for each, and set appropriate message retention -> Option D

4. Quick Check:

   Fan-out + DLQ + retention = reliable, no message loss [OK]

Hint: Use fan-out with DLQ and retention for reliability [OK]

Common Mistakes:

• Skipping dead-letter queues causing message loss
• Using one queue for all processing causing bottlenecks
• Not subscribing all queues to SNS topic