SNS and SQS integration pattern (fan-out) in AWS - Commands & Configuration

Introduction

Sometimes you want to send the same message to many places at once. SNS and SQS together help you do this easily. SNS sends the message, and multiple SQS queues receive it separately.

When you want to notify multiple services about the same event without mixing their messages.

When you need to process messages in parallel by different teams or systems.

When you want to keep messages safe in queues for later processing after sending a notification.

When you want to separate message sending from message processing for better reliability.

When you want to scale message processing independently for each receiver.

Config File - sns-sqs-fanout.yaml

sns-sqs-fanout.yaml

Resources:
  MyTopic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: my-fanout-topic

  QueueOne:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: my-queue-one

  QueueTwo:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: my-queue-two

  QueueOnePolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues:
        - !Ref QueueOne
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal: '*'
            Action: 'SQS:SendMessage'
            Resource: !GetAtt QueueOne.Arn
            Condition:
              ArnEquals:
                'aws:SourceArn': !Ref MyTopic

  QueueTwoPolicy:
    Type: AWS::SQS::QueuePolicy
    Properties:
      Queues:
        - !Ref QueueTwo
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal: '*'
            Action: 'SQS:SendMessage'
            Resource: !GetAtt QueueTwo.Arn
            Condition:
              ArnEquals:
                'aws:SourceArn': !Ref MyTopic

  SubscriptionOne:
    Type: AWS::SNS::Subscription
    Properties:
      TopicArn: !Ref MyTopic
      Protocol: sqs
      Endpoint: !GetAtt QueueOne.Arn

  SubscriptionTwo:
    Type: AWS::SNS::Subscription
    Properties:
      TopicArn: !Ref MyTopic
      Protocol: sqs
      Endpoint: !GetAtt QueueTwo.Arn

This file creates one SNS topic named my-fanout-topic.

It creates two SQS queues: my-queue-one and my-queue-two.

Each queue has a policy allowing the SNS topic to send messages to it.

Two subscriptions connect the SNS topic to each SQS queue, enabling the fan-out pattern.

Commands

This command creates the SNS topic, two SQS queues, their policies, and subscriptions using the CloudFormation template.

Terminal

aws cloudformation deploy --template-file sns-sqs-fanout.yaml --stack-name sns-sqs-fanout-stack

Expected Output

Waiting for stack create/update to complete...

Successfully created/updated stack - sns-sqs-fanout-stack

→

--template-file - Specifies the CloudFormation template file to use.

→

--stack-name - Names the CloudFormation stack.

This command sends a message to the SNS topic, which then forwards it to all subscribed SQS queues.

Terminal

aws sns publish --topic-arn arn:aws:sns:us-east-1:123456789012:my-fanout-topic --message "Hello, fan-out!"

Expected Output

{"MessageId":"a1b2c3d4-5678-90ab-cdef-EXAMPLE11111"}

→

--topic-arn - Specifies the SNS topic to send the message to.

→

--message - The message content to send.

This command retrieves the message from the first SQS queue to verify it received the SNS message.

Terminal

aws sqs receive-message --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue-one

Expected Output

{

    "Messages": [

        {

            "MessageId": "11111111-2222-3333-4444-555555555555",

            "ReceiptHandle": "AQEB123...",

            "Body": "{\"Type\":\"Notification\",\"MessageId\":\"a1b2c3d4-5678-90ab-cdef-EXAMPLE11111\",\"Message\":\"Hello, fan-out!\"}",

            "Attributes": {}

        }

    ]

}

→

--queue-url - Specifies which SQS queue to read from.

This command retrieves the message from the second SQS queue to verify it also received the SNS message.

Terminal

aws sqs receive-message --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue-two

Expected Output

{

    "Messages": [

        {

            "MessageId": "66666666-7777-8888-9999-000000000000",

            "ReceiptHandle": "AQEB456...",

            "Body": "{\"Type\":\"Notification\",\"MessageId\":\"a1b2c3d4-5678-90ab-cdef-EXAMPLE11111\",\"Message\":\"Hello, fan-out!\"}",

            "Attributes": {}

        }

    ]

}

→

--queue-url - Specifies which SQS queue to read from.

Key Concept

If you remember nothing else from this pattern, remember: SNS sends one message that multiple SQS queues receive independently, enabling parallel processing.

Common Mistakes

Not setting the SQS queue policy to allow SNS to send messages.

SNS cannot deliver messages to the queue without permission, so no messages arrive.

Add a queue policy that allows the SNS topic ARN to send messages to the SQS queue.

Using the SQS queue URL instead of ARN in the SNS subscription endpoint.

SNS subscriptions require the queue ARN for SQS protocol, so the subscription fails.

Use the SQS queue ARN as the subscription endpoint when subscribing to SNS.

Publishing messages to the wrong SNS topic ARN.

Messages go nowhere or to the wrong queues, so subscribers don't get the message.

Double-check and use the correct SNS topic ARN when publishing messages.

Summary

Create an SNS topic and multiple SQS queues with policies allowing SNS to send messages.

Subscribe each SQS queue to the SNS topic to enable message fan-out.

Publish a message to the SNS topic and verify each SQS queue receives it independently.