Sending and receiving messages in AWS - Deep Dive

Overview - Sending and receiving messages

What is it?

Sending and receiving messages in AWS means moving information between different parts of a system using special services. These messages are like notes passed between friends, allowing different programs or computers to talk without being connected all the time. AWS provides tools that help send, store, and get these messages safely and reliably. This helps systems work smoothly even if parts are busy or temporarily offline.

Why it matters

Without message sending and receiving, systems would have to wait for each other to be ready, causing delays and failures. Imagine trying to talk to a friend who is not always available; messages let you leave notes so they can reply later. This makes applications more reliable, scalable, and easier to maintain. It also helps handle sudden bursts of work without crashing.

Where it fits

Before learning this, you should understand basic cloud concepts like servers and storage. After this, you can explore advanced topics like event-driven architectures, microservices, and serverless computing where messaging is key.

Mental Model

Core Idea

Sending and receiving messages is like passing notes between friends so they can communicate asynchronously and reliably.

Think of it like...

Imagine a mailbox system where you drop a letter in your friend's mailbox, and they pick it up when they have time. You don't need to be there at the same moment, and the mailbox keeps the letter safe until they read it.

┌───────────────┐       ┌───────────────┐
│ Sender System │──────▶│ Message Queue │
└───────────────┘       └───────────────┘
                              │
                              ▼
                      ┌───────────────┐
                      │ Receiver System│
                      └───────────────┘

Build-Up - 7 Steps

1

Foundation

What is messaging in AWS

Concept: Introduction to the idea of messaging services in AWS.

Messaging means sending data from one part of a system to another using a service that holds the message until the receiver is ready. AWS offers services like Simple Queue Service (SQS) and Simple Notification Service (SNS) for this purpose.

Result

You understand that messaging allows parts of a system to communicate without needing to be active at the same time.

Understanding messaging as a way to decouple systems helps you design applications that are more flexible and fault-tolerant.

2

Foundation

Basic components of message flow

Concept: Learn the roles of sender, message queue, and receiver.

The sender creates and sends a message to a queue or topic. The message queue stores the message safely. The receiver retrieves and processes the message when ready. This flow ensures messages are not lost and can be retried if needed.

Result

You can identify the parts involved in sending and receiving messages and their responsibilities.

Knowing these roles clarifies how asynchronous communication works and why it improves system reliability.

3

Intermediate

Using AWS SQS for message queues

🤔Before reading on: do you think SQS guarantees message order or just delivery? Commit to your answer.

Concept: Introduction to AWS Simple Queue Service (SQS) and its features.

SQS is a fully managed message queue service that stores messages until a receiver processes them. It supports two types: Standard queues (high throughput, at-least-once delivery, best-effort ordering) and FIFO queues (exactly-once processing, strict order). You send messages via API calls and poll the queue to receive them.

Result

You can set up an SQS queue, send messages, and receive them reliably with control over order and duplication.

Understanding SQS types helps you choose the right queue for your application's needs, balancing speed and message order.

4

Intermediate

Using AWS SNS for message topics

🤔Before reading on: do you think SNS delivers messages to one or multiple receivers? Commit to your answer.

Concept: Introduction to AWS Simple Notification Service (SNS) and its publish-subscribe model.

SNS lets you send messages to multiple subscribers at once. You create a topic, and receivers subscribe to it via email, SMS, HTTP endpoints, or SQS queues. When you publish a message to the topic, all subscribers get a copy. This is useful for broadcasting events.

Result

You can broadcast messages to many receivers instantly and in different formats.

Knowing SNS's pub-sub model helps you design event-driven systems that notify many parts at once without tight coupling.

5

Intermediate

Combining SNS and SQS for robust messaging

🤔Before reading on: do you think combining SNS and SQS improves reliability or complicates it? Commit to your answer.

Concept: Learn how SNS and SQS work together to create scalable and reliable messaging patterns.

You can subscribe an SQS queue to an SNS topic. This means messages published to SNS are stored in SQS queues for each subscriber. Receivers then process messages from their queues at their own pace. This setup combines SNS's broadcast with SQS's reliable storage and processing.

Result

You get a system that can send messages to many receivers reliably, even if some are slow or offline.

Combining SNS and SQS leverages the strengths of both services to build fault-tolerant, scalable communication.

6

Advanced

Handling message failures and retries

🤔Before reading on: do you think failed messages are lost or retried automatically? Commit to your answer.

Concept: Explore how AWS handles messages that cannot be processed immediately or fail repeatedly.

SQS supports visibility timeouts that hide messages while being processed. If processing fails, the message becomes visible again for retry. Dead-letter queues (DLQs) store messages that fail too many times for later inspection. This prevents message loss and helps diagnose issues.

Result

You can build systems that gracefully handle errors without losing messages or blocking processing.

Understanding failure handling prevents data loss and helps maintain system health under errors.

7

Expert

Optimizing message throughput and cost

🤔Before reading on: do you think sending many small messages costs more or less than fewer large messages? Commit to your answer.

Concept: Learn best practices to balance performance and cost when sending and receiving messages in AWS.

Batching messages reduces API calls and cost. Long polling reduces empty responses and saves money. Choosing the right queue type (Standard vs FIFO) affects throughput and ordering guarantees. Monitoring metrics helps tune settings. Also, consider message size limits and encryption for security.

Result

You can design messaging systems that are fast, reliable, and cost-effective.

Knowing how to optimize messaging prevents unexpected costs and performance bottlenecks in production.

Under the Hood

AWS messaging services use distributed servers to store messages durably across multiple data centers. When a sender posts a message, it is replicated and stored until a receiver fetches it. The system manages message visibility and delivery attempts automatically. For SNS, messages are pushed to subscribers via protocols like HTTP or email. For SQS, receivers poll the queue to get messages. Internally, AWS ensures high availability and fault tolerance through replication and retry mechanisms.

Why designed this way?

Messaging was designed to decouple system components so they don't depend on each other's availability or speed. AWS built these services to be fully managed, scalable, and reliable, removing the need for users to manage servers or storage. The choice of queue types and pub-sub models reflects common messaging patterns in software design, balancing complexity and usability.

┌───────────────┐          ┌───────────────┐          ┌───────────────┐
│ Sender System │──Send──▶│ AWS Messaging │──Store──▶│ Receiver System│
│               │          │  (SQS/SNS)    │          │               │
└───────────────┘          └───────────────┘          └───────────────┘
         │                        │                           ▲
         │                        │                           │
         └─────────────Retry/Visibility Timeout─────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does SQS guarantee messages arrive in the exact order they were sent? Commit yes or no.

Common Belief: SQS always delivers messages in the order they were sent.

Reality: Only FIFO queues guarantee strict order; Standard queues do not guarantee order and may deliver duplicates.

Why it matters: Assuming order in Standard queues can cause bugs where data is processed out of sequence, leading to inconsistent results.

Quick: Does SNS send messages to only one subscriber or all subscribers? Commit your answer.

Common Belief: SNS sends messages to only one subscriber at a time.

Reality: SNS broadcasts messages to all subscribers simultaneously.

Why it matters: Misunderstanding this can lead to wrong system designs expecting single delivery, causing missed notifications.

Quick: Are messages lost if a receiver is offline? Commit yes or no.

Common Belief: If the receiver is offline, messages are lost.

Reality: Messages remain stored in queues until the receiver retrieves them or they expire.

Why it matters: Believing messages are lost leads to unnecessary retries or complex workarounds, increasing system complexity.

Quick: Does combining SNS and SQS complicate reliability? Commit yes or no.

Common Belief: Combining SNS and SQS makes the system more complex and less reliable.

Reality: Combining them improves reliability by adding durable storage and decoupling message delivery.

Why it matters: Avoiding this pattern can limit scalability and fault tolerance in distributed systems.

Expert Zone

1

SQS visibility timeout must be carefully tuned to avoid message duplication or premature message reprocessing.

2

SNS message delivery retries depend on the protocol; HTTP endpoints have different retry policies than email or SMS.

3

Message size limits (256 KB for SQS) require chunking or alternative storage for large payloads, affecting design.

When NOT to use

Avoid using SQS or SNS for real-time, low-latency communication where immediate response is critical; instead, use direct API calls or WebSocket connections. For complex workflows, consider AWS Step Functions or event buses like EventBridge.

Production Patterns

In production, teams use SNS to broadcast events to multiple microservices, each with its own SQS queue for reliable processing. Dead-letter queues monitor failures, and CloudWatch alarms track message backlogs. Batching and long polling optimize cost and performance.

Connections

Event-driven architecture

Messaging is the backbone that enables event-driven systems to react to changes asynchronously.

Understanding messaging clarifies how events trigger workflows without tight coupling, improving scalability.

Postal mail system

Messaging services mimic postal mail by storing and forwarding messages asynchronously.

Recognizing this helps grasp why messages can be delayed but still reliably delivered.

Human communication theory

Messaging in AWS parallels asynchronous communication where sender and receiver do not share time.

Knowing this connection helps appreciate the importance of message durability and retries in noisy channels.

Common Pitfalls

#1 Assuming messages are processed exactly once without duplicates.

Wrong approach:

Using SQS Standard queue without handling duplicate messages in the receiver logic.

Correct approach:

Use SQS FIFO queue for exactly-once processing or implement idempotent processing in the receiver.

Root cause:

Misunderstanding SQS Standard queue's at-least-once delivery model leads to duplicate message processing.

#2 Not setting visibility timeout properly causing message reprocessing.

Wrong approach:

Leaving default visibility timeout too short, causing messages to reappear before processing finishes.

Correct approach:

Adjust visibility timeout to exceed maximum processing time to prevent premature retries.

Root cause:

Ignoring processing time variability causes messages to be visible again too soon.

#3 Sending large messages exceeding size limits.

Wrong approach:

Sending 1 MB messages directly to SQS which only supports up to 256 KB.

Correct approach:

Store large payloads in S3 and send a reference link in the message.

Root cause:

Not knowing message size limits leads to rejected messages or failures.

Key Takeaways

Messaging in AWS enables asynchronous, reliable communication between system parts, improving flexibility and fault tolerance.

SQS provides durable queues with options for ordering and duplication control, while SNS broadcasts messages to multiple subscribers.

Combining SNS and SQS leverages the strengths of both for scalable, reliable message delivery.

Proper handling of message failures, retries, and visibility timeouts is essential to prevent data loss and duplication.

Optimizing message size, batching, and polling reduces cost and improves performance in production systems.