SNS and SQS integration pattern (fan-out) in AWS - Step-by-Step Execution

Process Flow - SNS and SQS integration pattern (fan-out)

Message published to SNS topic

↓

SNS topic receives message

↓

SNS fans out message to multiple SQS queues

↓

SQS Queue 1

↓

Consumers read messages independently

A message is sent to an SNS topic, which then copies and sends it to multiple SQS queues. Each queue can be processed separately by different consumers.

Execution Sample

AWS

Publish message to SNS topic
SNS fans out to SQS queues
Each SQS queue receives the message
Consumers poll SQS queues
Process messages independently

This shows how a single message sent to SNS is delivered to multiple SQS queues for parallel processing.

Process Table

Step	Action	SNS Topic State	SQS Queue 1 State	SQS Queue 2 State	SQS Queue 3 State	Consumer Processing
1	Message published to SNS topic	Message received	Empty	Empty	Empty	No processing yet
2	SNS fans out message to SQS queues	Messages delivered	Message received	Message received	Message received	No processing yet
3	Consumer 1 polls SQS Queue 1	Messages delivered	Message in flight	Message received	Message received	Consumer 1 processing message
4	Consumer 2 polls SQS Queue 2	Messages delivered	Message in flight	Message in flight	Message received	Consumer 1 and 2 processing messages
5	Consumer 3 polls SQS Queue 3	Messages delivered	Message in flight	Message in flight	Message in flight	All consumers processing messages
6	Consumers delete messages after processing	Messages delivered	Empty	Empty	Empty	All consumers done
7	No more messages	Empty	Empty	Empty	Empty	No processing
Exit	All messages processed and queues empty	No messages	No messages	No messages	No messages	Idle

💡 All messages processed and deleted from queues, SNS topic has no pending messages

Status Tracker

Variable	Start	After Step 1	After Step 2	After Step 3	After Step 4	After Step 5	After Step 6	Final
SNS Topic Messages	0	1	0	0	0	0	0	0
SQS Queue 1 Messages	0	0	1	0 (in flight)	0 (in flight)	0 (in flight)	0	0
SQS Queue 2 Messages	0	0	1	1 (in flight)	0 (in flight)	0 (in flight)	0	0
SQS Queue 3 Messages	0	0	1	1	1 (in flight)	0 (in flight)	0	0
Consumers Processing	None	None	None	Consumer 1	Consumer 1 & 2	All 3	None	None

Key Moments - 3 Insights

Why does each SQS queue get its own copy of the message?

Because SNS fans out the message to all subscribed queues independently, as shown in execution_table step 2 where all queues receive the message.

What happens if one consumer processes its message slower than others?

That queue's message stays 'in flight' until processed and deleted, while other queues can process their messages independently, as seen in steps 3 to 5.

Does deleting a message from one SQS queue remove it from others?

No, each SQS queue holds its own copy. Deleting from one queue does not affect messages in other queues, as shown in step 6.

Visual Quiz - 3 Questions

Look at the execution_table at step 2. What is the state of SQS Queue 2?

A Empty

B Message received

C Message in flight

D Message deleted

💡 Hint

Check the 'SQS Queue 2 State' column at step 2 in the execution_table.

At which step do all consumers start processing their messages simultaneously?

A Step 3

B Step 4

C Step 5

D Step 6

💡 Hint

Look at the 'Consumer Processing' column to see when all consumers are active.

If the message is deleted from SQS Queue 1 at step 6, what happens to the message in SQS Queue 3?

A It remains until processed

B It moves to SQS Queue 1

C It is also deleted

D It is duplicated

💡 Hint

Refer to the key moment about message deletion and independent queues.

Concept Snapshot

SNS and SQS fan-out pattern:
- Publish message to SNS topic
- SNS sends copies to all subscribed SQS queues
- Each SQS queue holds its own copy
- Consumers poll and process messages independently
- Deleting from one queue does not affect others

Full Transcript

This visual execution shows how a message published to an SNS topic is copied to multiple SQS queues. Each queue receives the message independently and holds it until a consumer processes and deletes it. Consumers can work in parallel without affecting each other. SNS delivers the message asynchronously to all subscribed queues. This pattern enables scalable, parallel processing of the same message by multiple consumers.

Practice

1. What is the main purpose of using SNS with multiple SQS queues in a fan-out pattern?

easy

A. To send the same message to multiple queues for parallel processing

B. To store messages permanently in one queue

C. To send messages only to one queue at a time

D. To delete messages immediately after sending

Solution

1. Step 1: Understand SNS role in fan-out

   SNS publishes messages to multiple subscribers simultaneously.

2. Step 2: Understand SQS role in fan-out

   Each SQS queue receives and holds messages independently for processing.

3. Final Answer:

   To send the same message to multiple queues for parallel processing -> Option A

4. Quick Check:

   SNS fan-out = multiple queues get same message [OK]

Hint: SNS fan-out means one message to many queues [OK]

Common Mistakes:

• Thinking SNS sends to only one queue
• Confusing message storage with deletion
• Assuming messages are deleted immediately

2. Which of the following is the correct way to subscribe an SQS queue to an SNS topic using AWS CLI?

easy

A. aws sns subscribe --topic-arn --protocol sqs --notification-endpoint

B. aws sqs subscribe --topic-arn --queue-url

C. aws sns create-subscription --topic --queue

D. aws sns add-queue --topic-arn --queue-arn

Solution

1. Step 1: Identify correct AWS CLI command

   The AWS CLI command to subscribe is 'aws sns subscribe'.

2. Step 2: Check parameters for SQS subscription

   Use '--protocol sqs' and '--notification-endpoint' with the queue ARN.

3. Final Answer:

   aws sns subscribe --topic-arn <topic-arn> --protocol sqs --notification-endpoint <queue-arn> -> Option A

4. Quick Check:

   Subscribe SQS to SNS uses 'aws sns subscribe' with protocol sqs [OK]

Hint: Subscribe SQS with 'aws sns subscribe' and protocol sqs [OK]

Common Mistakes:

• Using 'aws sqs subscribe' which does not exist
• Confusing queue URL with queue ARN
• Using wrong command like 'aws sns add-queue'

3. Given an SNS topic with two subscribed SQS queues, if a message is published to SNS, what will be the state of the message in the queues?

medium

A. The message is deleted from SNS and not delivered to any queue.

B. The message appears in only one queue randomly.

C. The message appears in both queues independently, waiting to be processed.

D. The message is duplicated in one queue only.

Solution

1. Step 1: Understand SNS fan-out delivery

   SNS sends a copy of the message to each subscribed queue.

2. Step 2: Understand SQS message handling

   Each queue holds its own copy until processed or deleted.

3. Final Answer:

   The message appears in both queues independently, waiting to be processed. -> Option C

4. Quick Check:

   Fan-out means each queue gets its own message copy [OK]

Hint: SNS fan-out sends message copy to each queue [OK]

Common Mistakes:

• Thinking message goes to only one queue
• Assuming message is deleted immediately
• Believing message duplicates in one queue only

4. You configured an SNS topic with two SQS queues subscribed, but only one queue receives messages. What is the most likely cause?

medium

A. SNS only sends messages to one queue by design

B. The second queue's subscription is not confirmed

C. The message size exceeds SNS limits

D. The first queue deleted the message too fast

Solution

1. Step 1: Check subscription confirmation

   SQS subscriptions to SNS require confirmation to activate delivery.

2. Step 2: Understand SNS delivery behavior

   Without confirmation, SNS does not send messages to that queue.

3. Final Answer:

   The second queue's subscription is not confirmed -> Option B

4. Quick Check:

   Unconfirmed subscription blocks message delivery [OK]

Hint: Confirm all subscriptions to receive messages [OK]

Common Mistakes:

• Assuming SNS sends to only one queue
• Blaming message size without checking subscription
• Thinking message deletion affects delivery

5. You want to build a system where an SNS topic sends messages to three SQS queues. One queue processes orders, another sends notifications, and the third logs events. How should you configure the system to ensure reliable processing and no message loss?

hard

A. Subscribe only the order queue to SNS and forward messages to others manually

B. Send messages directly to each SQS queue without SNS

C. Use one SQS queue for all processing and filter messages inside the consumer

D. Subscribe all three SQS queues to the SNS topic, enable dead-letter queues for each, and set appropriate message retention

Solution

1. Step 1: Use SNS fan-out to send messages to all queues

   Subscribing all three SQS queues to the SNS topic ensures each gets the message.

2. Step 2: Enable dead-letter queues and set retention

   Dead-letter queues catch failed messages; retention ensures messages stay until processed.

3. Final Answer:

   Subscribe all three SQS queues to the SNS topic, enable dead-letter queues for each, and set appropriate message retention -> Option D

4. Quick Check:

   Fan-out + DLQ + retention = reliable, no message loss [OK]

Hint: Use fan-out with DLQ and retention for reliability [OK]

Common Mistakes:

• Skipping dead-letter queues causing message loss
• Using one queue for all processing causing bottlenecks
• Not subscribing all queues to SNS topic