Retained messages in IOT Protocols - Deep Dive

Overview - Retained messages

What is it?

Retained messages are special messages in IoT communication protocols like MQTT that stay stored on the broker after being sent. When a new device subscribes to a topic, it immediately receives the last retained message without waiting for a new one. This helps devices get the latest state or information right away.

Why it matters

Without retained messages, new devices would have to wait for the next update to know the current state, causing delays or inconsistent data. Retained messages ensure that devices always start with the most recent information, improving reliability and user experience in IoT systems.

Where it fits

Learners should first understand basic MQTT messaging and topics before learning about retained messages. After this, they can explore advanced MQTT features like QoS levels, Last Will messages, and session persistence.

Mental Model

Core Idea

A retained message is like a sticky note on a bulletin board that always shows the latest important info for anyone who looks.

Think of it like...

Imagine a community bulletin board where the last important announcement is pinned and stays there until replaced. Anyone new who comes by immediately sees that announcement without waiting for someone to tell them again.

┌───────────────┐       ┌───────────────┐       ┌───────────────┐
│ Publisher     │──────▶│ Broker        │──────▶│ Subscriber    │
│ Sends message │       │ Stores message│       │ Receives last │
│ with retain   │       │ as retained   │       │ retained msg  │
└───────────────┘       └───────────────┘       └───────────────┘

Build-Up - 6 Steps

1

Foundation

Basic MQTT Message Flow

Concept: Understand how MQTT messages are sent from publishers to subscribers via a broker.

In MQTT, devices called publishers send messages to topics on a broker. Subscribers listen to these topics and receive messages when published. Normally, subscribers only get messages sent after they subscribe.

Result

Subscribers receive messages only after they subscribe; no past messages are delivered.

Knowing the basic flow clarifies why new subscribers might miss important past information.

2

Foundation

What is a Retained Message?

Concept: Learn that retained messages are stored by the broker to deliver the last message immediately to new subscribers.

When a publisher sends a message with the 'retain' flag, the broker saves this message as the last known good message for that topic. New subscribers get this message instantly upon subscribing.

Result

New subscribers receive the latest retained message immediately, even if it was sent before they subscribed.

Understanding retained messages solves the problem of new subscribers missing previous important data.

3

Intermediate

How Retained Messages Affect Device State

🤔Before reading on: do you think retained messages update automatically when a new message is sent without retain flag? Commit to your answer.

Concept: Retained messages represent the current state of a topic and update only when a new retained message arrives.

If a new message is sent with the retain flag, it replaces the old retained message. Messages without the retain flag do not change the stored retained message. This means the retained message always reflects the last retained update.

Result

The broker keeps only one retained message per topic, always the latest retained one.

Knowing this prevents confusion about why some messages appear immediately and others do not.

4

Intermediate

Clearing Retained Messages

🤔Before reading on: do you think sending an empty retained message deletes the retained message? Commit to your answer.

Concept: Learn how to remove a retained message by sending a zero-length message with the retain flag.

To clear a retained message, a publisher sends a message with the retain flag set but with an empty payload. The broker then deletes the stored retained message for that topic.

Result

Subscribers will no longer receive a retained message for that topic until a new one is published.

Understanding this helps manage stale or outdated retained messages in production.

5

Advanced

Retained Messages and Quality of Service

🤔Before reading on: do you think QoS affects how retained messages are delivered? Commit to your answer.

Concept: Explore how retained messages interact with MQTT Quality of Service (QoS) levels to guarantee delivery.

Retained messages are delivered to new subscribers with the QoS level of the subscription or the retained message, whichever is lower. This ensures reliable delivery according to MQTT QoS rules.

Result

Subscribers receive retained messages with appropriate delivery guarantees, balancing reliability and performance.

Knowing this helps design systems that balance message delivery guarantees with network efficiency.

6

Expert

Retained Messages in Large-Scale IoT Systems

🤔Before reading on: do you think retaining messages for all topics is always beneficial in large systems? Commit to your answer.

Concept: Understand the trade-offs and challenges of using retained messages at scale, including storage and stale data risks.

In large IoT deployments, retaining messages for many topics can consume broker storage and cause outdated information if not managed. Strategies include selective retention, expiration policies, and monitoring retained message health.

Result

Efficient use of retained messages improves system scalability and data accuracy.

Recognizing these trade-offs is key to building robust, scalable IoT solutions.

Under the Hood

The MQTT broker stores the retained message payload and topic string in persistent or memory storage. When a new subscription matches the topic, the broker immediately sends the stored retained message before any new messages. The retain flag in the MQTT packet header signals the broker to save or delete the retained message.

Why designed this way?

Retained messages were designed to solve the problem of late-joining subscribers missing important state information. Storing only the last message per topic keeps storage minimal and delivery simple, avoiding complex history management.

┌───────────────┐
│ Publisher     │
│ Sends message │
│ with retain   │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ MQTT Broker   │
│ Stores retained│
│ message       │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│ Subscriber    │
│ Receives last │
│ retained msg  │
└───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Does sending a message without retain flag remove the retained message? Commit yes or no.

Common Belief: Sending a normal message without the retain flag will overwrite or remove the retained message.

Reality: Only messages sent with the retain flag can create, update, or delete retained messages. Normal messages do not affect retained messages.

Why it matters: Misunderstanding this leads to unexpected stale data being delivered to new subscribers.

Quick: Do retained messages guarantee delivery to all subscribers regardless of QoS? Commit yes or no.

Common Belief: Retained messages are always delivered to subscribers no matter what QoS level is set.

Reality: Retained messages are delivered according to the QoS level negotiated between broker and subscriber, which may affect delivery guarantees.

Why it matters: Assuming guaranteed delivery can cause data loss or inconsistency in critical IoT applications.

Quick: Are retained messages stored forever by default? Commit yes or no.

Common Belief: Once a message is retained, it stays forever on the broker unless manually deleted.

Reality: Retention duration depends on broker configuration; some brokers support expiration or limits to avoid stale data buildup.

Why it matters: Ignoring retention policies can cause storage overload and outdated information delivery.

Quick: Can retained messages contain large payloads without issues? Commit yes or no.

Common Belief: Retained messages can be any size without impacting broker performance.

Reality: Large retained messages consume more broker memory and storage, potentially degrading performance and scalability.

Why it matters: Overloading retained messages can cause broker crashes or slowdowns in production.

Expert Zone

1

Retained messages only store the last message per topic, so publishing multiple retained messages to subtopics can create complex state hierarchies.

2

Some brokers allow retained messages to be combined with message expiry, balancing freshness and storage use.

3

Retained messages can cause unexpected behavior if clients do not handle duplicate messages on reconnect, requiring idempotent processing.

When NOT to use

Avoid using retained messages for high-frequency telemetry data where state changes rapidly; instead, use persistent sessions or database storage. Also, do not rely on retained messages for sensitive data that must not persist on the broker.

Production Patterns

In production, retained messages are used for device configuration, last known sensor states, or command acknowledgments. Systems often combine retained messages with monitoring tools to detect stale or missing retained data and automate cleanup.

Connections

Cache Invalidation

Retained messages act like a cache of the last message, similar to caching data in computing.

Understanding cache invalidation helps grasp why retained messages must be updated or cleared to avoid stale data.

Publish-Subscribe Messaging Pattern

Retained messages extend the basic pub-sub pattern by adding state persistence on the broker.

Knowing pub-sub fundamentals clarifies how retained messages improve message delivery for late subscribers.

Sticky Notes on a Fridge

Both keep important information visible for anyone who comes by later.

This cross-domain connection highlights the value of persistent reminders in communication systems.

Common Pitfalls

#1 Assuming all messages are retained by default.

Wrong approach:

publisher.publish(topic='home/temperature', payload='22C') # no retain flag

Correct approach:

publisher.publish(topic='home/temperature', payload='22C', retain=True)

Root cause:

Not setting the retain flag means the broker does not store the message for new subscribers.

#2 Sending a retained message with a non-empty payload to clear it.

Wrong approach:

publisher.publish(topic='home/light', payload='off', retain=False)

Correct approach:

publisher.publish(topic='home/light', payload='', retain=True)

Root cause:

Only an empty payload with retain flag clears the retained message; non-empty or no retain flag does not.

#3 Publishing large retained messages without considering broker limits.

Wrong approach:

publisher.publish(topic='camera/image', payload='very_large_binary_data', retain=True)

Correct approach:

Avoid retaining large payloads; use separate storage and publish only references or metadata.

Root cause:

Large retained messages consume excessive broker resources, risking performance.

Key Takeaways

Retained messages store the last message on a topic so new subscribers get immediate current state.

Only messages sent with the retain flag affect retained messages; normal messages do not update or clear them.

Clearing a retained message requires sending an empty payload with the retain flag set.

Retained messages interact with QoS levels to ensure appropriate delivery guarantees.

In large systems, careful management of retained messages is essential to avoid stale data and resource issues.