SNS and SQS integration pattern (fan-out) in AWS - Commands & Configuration

Introduction

Sometimes you want to send the same message to many places at once. SNS and SQS together help you do this easily. SNS sends the message, and multiple SQS queues receive it separately.

When you want to notify multiple services about the same event without mixing their messages.

When you need to process messages in parallel by different teams or systems.

When you want to keep messages safe in queues for later processing after sending a notification.

When you want to separate message sending from message processing for better reliability.

When you want to scale message processing independently for each receiver.

Config File - sns-sqs-fanout.yaml

sns-sqs-fanout.yaml

Resources:
  MyTopic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: my-fanout-topic

  QueueOne:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: my-queue-one

  QueueTwo:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: my-queue-two

  QueueOnePolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues:
        - !Ref QueueOne
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal: '*'
            Action: 'SQS:SendMessage'
            Resource: !GetAtt QueueOne.Arn
            Condition:
              ArnEquals:
                'aws:SourceArn': !Ref MyTopic

  QueueTwoPolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues:
        - !Ref QueueTwo
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal: '*'
            Action: 'SQS:SendMessage'
            Resource: !GetAtt QueueTwo.Arn
            Condition:
              ArnEquals:
                'aws:SourceArn': !Ref MyTopic

  SubscriptionOne:
    Type: AWS::SNS::Subscription
    Properties:
      TopicArn: !Ref MyTopic
      Protocol: sqs
      Endpoint: !GetAtt QueueOne.Arn

  SubscriptionTwo:
    Type: AWS::SNS::Subscription
    Properties:
      TopicArn: !Ref MyTopic
      Protocol: sqs
      Endpoint: !GetAtt QueueTwo.Arn

This file creates one SNS topic named my-fanout-topic.

It creates two SQS queues: my-queue-one and my-queue-two.

Each queue has a policy allowing the SNS topic to send messages to it.

Two subscriptions connect the SNS topic to each SQS queue, enabling the fan-out pattern.

Commands

This command creates the SNS topic, two SQS queues, their policies, and subscriptions using the CloudFormation template.

Terminal

aws cloudformation deploy --template-file sns-sqs-fanout.yaml --stack-name sns-sqs-fanout-stack

Expected Output

Waiting for stack create/update to complete...

Successfully created/updated stack - sns-sqs-fanout-stack

→

--template-file - Specifies the CloudFormation template file to use.

→

--stack-name - Names the CloudFormation stack.

This command sends a message to the SNS topic, which then forwards it to all subscribed SQS queues.

Terminal

aws sns publish --topic-arn arn:aws:sns:us-east-1:123456789012:my-fanout-topic --message "Hello, fan-out!"

Expected Output

{"MessageId":"a1b2c3d4-5678-90ab-cdef-EXAMPLE11111"}

→

--topic-arn - Specifies the SNS topic to send the message to.

→

--message - The message content to send.

This command retrieves the message from the first SQS queue to verify it received the SNS message.

Terminal

aws sqs receive-message --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue-one

Expected Output

{

    "Messages": [

        {

            "MessageId": "11111111-2222-3333-4444-555555555555",

            "ReceiptHandle": "AQEB123...",

            "Body": "{\"Type\":\"Notification\",\"MessageId\":\"a1b2c3d4-5678-90ab-cdef-EXAMPLE11111\",\"Message\":\"Hello, fan-out!\"}",

            "Attributes": {}

        }

    ]

}

→

--queue-url - Specifies which SQS queue to read from.

This command retrieves the message from the second SQS queue to verify it also received the SNS message.

Terminal

aws sqs receive-message --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue-two

Expected Output

{

    "Messages": [

        {

            "MessageId": "66666666-7777-8888-9999-000000000000",

            "ReceiptHandle": "AQEB456...",

            "Body": "{\"Type\":\"Notification\",\"MessageId\":\"a1b2c3d4-5678-90ab-cdef-EXAMPLE11111\",\"Message\":\"Hello, fan-out!\"}",

            "Attributes": {}

        }

    ]

}

→

--queue-url - Specifies which SQS queue to read from.

Key Concept

If you remember nothing else from this pattern, remember: SNS sends one message that multiple SQS queues receive independently, enabling parallel processing.

Common Mistakes

Not setting the SQS queue policy to allow SNS to send messages.

SNS cannot deliver messages to the queue without permission, so no messages arrive.

Add a queue policy that allows the SNS topic ARN to send messages to the SQS queue.

Using the SQS queue URL instead of ARN in the SNS subscription endpoint.

SNS subscriptions require the queue ARN for SQS protocol, so the subscription fails.

Use the SQS queue ARN as the subscription endpoint when subscribing to SNS.

Publishing messages to the wrong SNS topic ARN.

Messages go nowhere or to the wrong queues, so subscribers don't get the message.

Double-check and use the correct SNS topic ARN when publishing messages.

Summary

Create an SNS topic and multiple SQS queues with policies allowing SNS to send messages.

Subscribe each SQS queue to the SNS topic to enable message fan-out.

Publish a message to the SNS topic and verify each SQS queue receives it independently.

Practice

1. What is the main purpose of using SNS with multiple SQS queues in a fan-out pattern?

easy

A. To send the same message to multiple queues for parallel processing

B. To store messages permanently in one queue

C. To send messages only to one queue at a time

D. To delete messages immediately after sending

Solution

1. Step 1: Understand SNS role in fan-out

   SNS publishes messages to multiple subscribers simultaneously.

2. Step 2: Understand SQS role in fan-out

   Each SQS queue receives and holds messages independently for processing.

3. Final Answer:

   To send the same message to multiple queues for parallel processing -> Option A

4. Quick Check:

   SNS fan-out = multiple queues get same message [OK]

Hint: SNS fan-out means one message to many queues [OK]

Common Mistakes:

• Thinking SNS sends to only one queue
• Confusing message storage with deletion
• Assuming messages are deleted immediately

2. Which of the following is the correct way to subscribe an SQS queue to an SNS topic using AWS CLI?

easy

A. aws sns subscribe --topic-arn --protocol sqs --notification-endpoint

B. aws sqs subscribe --topic-arn --queue-url

C. aws sns create-subscription --topic --queue

D. aws sns add-queue --topic-arn --queue-arn

Solution

1. Step 1: Identify correct AWS CLI command

   The AWS CLI command to subscribe is 'aws sns subscribe'.

2. Step 2: Check parameters for SQS subscription

   Use '--protocol sqs' and '--notification-endpoint' with the queue ARN.

3. Final Answer:

   aws sns subscribe --topic-arn <topic-arn> --protocol sqs --notification-endpoint <queue-arn> -> Option A

4. Quick Check:

   Subscribe SQS to SNS uses 'aws sns subscribe' with protocol sqs [OK]

Hint: Subscribe SQS with 'aws sns subscribe' and protocol sqs [OK]

Common Mistakes:

• Using 'aws sqs subscribe' which does not exist
• Confusing queue URL with queue ARN
• Using wrong command like 'aws sns add-queue'

3. Given an SNS topic with two subscribed SQS queues, if a message is published to SNS, what will be the state of the message in the queues?

medium

A. The message is deleted from SNS and not delivered to any queue.

B. The message appears in only one queue randomly.

C. The message appears in both queues independently, waiting to be processed.

D. The message is duplicated in one queue only.

Solution

1. Step 1: Understand SNS fan-out delivery

   SNS sends a copy of the message to each subscribed queue.

2. Step 2: Understand SQS message handling

   Each queue holds its own copy until processed or deleted.

3. Final Answer:

   The message appears in both queues independently, waiting to be processed. -> Option C

4. Quick Check:

   Fan-out means each queue gets its own message copy [OK]

Hint: SNS fan-out sends message copy to each queue [OK]

Common Mistakes:

• Thinking message goes to only one queue
• Assuming message is deleted immediately
• Believing message duplicates in one queue only

4. You configured an SNS topic with two SQS queues subscribed, but only one queue receives messages. What is the most likely cause?

medium

A. SNS only sends messages to one queue by design

B. The second queue's subscription is not confirmed

C. The message size exceeds SNS limits

D. The first queue deleted the message too fast

Solution

1. Step 1: Check subscription confirmation

   SQS subscriptions to SNS require confirmation to activate delivery.

2. Step 2: Understand SNS delivery behavior

   Without confirmation, SNS does not send messages to that queue.

3. Final Answer:

   The second queue's subscription is not confirmed -> Option B

4. Quick Check:

   Unconfirmed subscription blocks message delivery [OK]

Hint: Confirm all subscriptions to receive messages [OK]

Common Mistakes:

• Assuming SNS sends to only one queue
• Blaming message size without checking subscription
• Thinking message deletion affects delivery

5. You want to build a system where an SNS topic sends messages to three SQS queues. One queue processes orders, another sends notifications, and the third logs events. How should you configure the system to ensure reliable processing and no message loss?

hard

A. Subscribe only the order queue to SNS and forward messages to others manually

B. Send messages directly to each SQS queue without SNS

C. Use one SQS queue for all processing and filter messages inside the consumer

D. Subscribe all three SQS queues to the SNS topic, enable dead-letter queues for each, and set appropriate message retention

Solution

1. Step 1: Use SNS fan-out to send messages to all queues

   Subscribing all three SQS queues to the SNS topic ensures each gets the message.

2. Step 2: Enable dead-letter queues and set retention

   Dead-letter queues catch failed messages; retention ensures messages stay until processed.

3. Final Answer:

   Subscribe all three SQS queues to the SNS topic, enable dead-letter queues for each, and set appropriate message retention -> Option D

4. Quick Check:

   Fan-out + DLQ + retention = reliable, no message loss [OK]

Hint: Use fan-out with DLQ and retention for reliability [OK]

Common Mistakes:

• Skipping dead-letter queues causing message loss
• Using one queue for all processing causing bottlenecks
• Not subscribing all queues to SNS topic