Fanout Pattern with SNS and SQS: How It Works and When to Use

The fanout pattern with AWS SNS and SQS is a way to send a single message to multiple queues at once. You publish a message to an SNS topic, which then forwards copies to multiple SQS queues, allowing parallel processing by different consumers.

How It Works

Imagine you want to send the same letter to several friends at once. Instead of writing and mailing each letter separately, you put one letter in a mailbox that automatically makes copies and sends them to each friend’s mailbox. This is how the fanout pattern works with AWS SNS and SQS.

You create an SNS topic, which acts like the central mailbox. Then, you create multiple SQS queues, each like a friend's mailbox. You subscribe each SQS queue to the SNS topic. When you publish a message to the SNS topic, it automatically copies and sends that message to all subscribed SQS queues. Each queue can then be processed independently by different applications or services.

This pattern helps distribute work evenly and allows different parts of your system to react to the same event without interfering with each other.

Example

This example shows how to create an SNS topic, two SQS queues, subscribe the queues to the topic, and publish a message that fans out to both queues.

python

import boto3
import json

sns = boto3.client('sns')
sqs = boto3.client('sqs')

# Create SNS topic
response_topic = sns.create_topic(Name='MyFanoutTopic')
topic_arn = response_topic['TopicArn']

# Create two SQS queues
response_queue1 = sqs.create_queue(QueueName='Queue1')
response_queue2 = sqs.create_queue(QueueName='Queue2')
queue_url1 = response_queue1['QueueUrl']
queue_url2 = response_queue2['QueueUrl']

# Get queue ARNs
attrs1 = sqs.get_queue_attributes(QueueUrl=queue_url1, AttributeNames=['QueueArn'])
attrs2 = sqs.get_queue_attributes(QueueUrl=queue_url2, AttributeNames=['QueueArn'])
queue_arn1 = attrs1['Attributes']['QueueArn']
queue_arn2 = attrs2['Attributes']['QueueArn']

# Subscribe queues to SNS topic
sns.subscribe(TopicArn=topic_arn, Protocol='sqs', Endpoint=queue_arn1)
sns.subscribe(TopicArn=topic_arn, Protocol='sqs', Endpoint=queue_arn2)

# Allow SNS to send messages to SQS queues
policy = {
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Principal": {"Service": "sns.amazonaws.com"},
        "Action": "sqs:SendMessage",
        "Resource": [queue_arn1, queue_arn2],
        "Condition": {
            "ArnEquals": {"aws:SourceArn": topic_arn}
        }
    }]
}

sqs.set_queue_attributes(
    QueueUrl=queue_url1,
    Attributes={"Policy": json.dumps(policy)}
)
sqs.set_queue_attributes(
    QueueUrl=queue_url2,
    Attributes={"Policy": json.dumps(policy)}
)

# Publish a message to SNS topic
sns.publish(TopicArn=topic_arn, Message='Hello to all queues!')

Output

Creates SNS topic and two SQS queues, subscribes queues to topic, sets permissions, and publishes message. Both queues receive the message.

When to Use

Use the fanout pattern when you want to send the same message to multiple independent systems or services. It is great for:

• Decoupling parts of your application so they can work independently.
• Parallel processing of the same event by different consumers.
• Sending notifications to multiple channels like email, SMS, and logging systems simultaneously.
• Scaling workloads by distributing messages to multiple queues for load balancing.

For example, an e-commerce site can publish an order event to SNS, and separate SQS queues can handle payment processing, inventory update, and shipping notifications independently.

Key Points

• SNS acts as a message broadcaster.
• SQS queues receive copies of messages for independent processing.
• Fanout pattern improves scalability and decoupling.
• Each queue processes messages at its own pace without blocking others.
• Proper permissions are needed to allow SNS to send messages to SQS.

Key Takeaways

The fanout pattern sends one message to multiple SQS queues via an SNS topic.

It helps decouple systems and allows parallel processing of the same event.

SNS broadcasts messages, and SQS queues independently receive and process them.

Use it when multiple services need to react to the same event without blocking each other.

Set correct permissions so SNS can deliver messages to SQS queues.