Topics and subscriptions in GCP - Deep Dive

Overview - Topics and subscriptions

What is it?

Topics and subscriptions are the main parts of a messaging system in the cloud. A topic is like a mailbox where messages are sent. A subscription is like a mailbox key that lets you receive messages from that topic. This system helps different parts of an application talk to each other without being directly connected.

Why it matters

Without topics and subscriptions, applications would need to be tightly linked, making them hard to change or scale. This messaging system allows apps to work independently and handle lots of messages smoothly. It makes systems more reliable and easier to grow as needs increase.

Where it fits

Before learning this, you should understand basic cloud concepts and what messaging means. After this, you can learn about message filtering, dead-letter topics, and how to secure and monitor messaging systems.

Mental Model

Core Idea

Topics send messages and subscriptions receive them, enabling loose and scalable communication between parts of a system.

Think of it like...

Imagine a bulletin board (topic) where anyone can post notes, and people with a special pass (subscription) can come and read those notes whenever they want.

┌───────────┐       ┌───────────────┐       ┌───────────────┐
│  Publisher│──────▶│    Topic      │──────▶│ Subscription  │
└───────────┘       └───────────────┘       └───────────────┘
                                             │
                                             ▼
                                       ┌───────────┐
                                       │ Subscriber│
                                       └───────────┘

Build-Up - 7 Steps

1

Foundation

Understanding messaging basics

Concept: Learn what messaging means in cloud systems and why it helps apps communicate.

Messaging is a way for different parts of a system to send information to each other without waiting. Instead of calling each other directly, they send messages to a shared place. This helps apps work independently and handle many messages at once.

Result

You understand why messaging is useful and how it helps apps stay flexible and scalable.

Knowing messaging basics sets the stage for understanding topics and subscriptions as tools to organize and deliver messages.

2

Foundation

What is a topic in messaging

Concept: A topic is a named channel where messages are sent for others to receive.

Think of a topic as a mailbox or a bulletin board. Publishers send messages to this topic. The topic holds messages until subscribers pick them up. Topics are central points for message distribution.

Result

You can identify a topic as the place where messages are published in a messaging system.

Understanding topics as message holders clarifies how messages flow from senders to receivers.

3

Intermediate

Role of subscriptions

🤔Before reading on: do you think subscriptions store messages or just point to topics? Commit to your answer.

Concept: Subscriptions connect to topics and let subscribers receive messages, often storing them temporarily.

A subscription is like a key to the topic's mailbox. It controls who can read messages and keeps track of which messages have been delivered. Subscriptions can hold messages until the subscriber is ready.

Result

You understand that subscriptions manage message delivery and tracking for subscribers.

Knowing subscriptions handle message delivery helps explain how systems ensure no message is lost or repeated unnecessarily.

4

Intermediate

Push vs Pull subscriptions

🤔Before reading on: do you think push or pull subscriptions require the subscriber to ask for messages? Commit to your answer.

Concept: Subscriptions can deliver messages by pushing them automatically or by letting subscribers pull them when ready.

Push subscriptions send messages to a subscriber's endpoint immediately. Pull subscriptions wait for the subscriber to request messages. Each method suits different use cases and system designs.

Result

You can choose between push and pull based on how your application wants to receive messages.

Understanding delivery methods helps design systems that balance immediacy and control over message processing.

5

Intermediate

Message acknowledgement and retries

🤔Before reading on: do you think messages disappear after delivery or require confirmation? Commit to your answer.

Concept: Subscribers must confirm they received messages; otherwise, messages are retried to ensure delivery.

When a subscriber gets a message, it sends an acknowledgement. If no acknowledgement arrives, the system assumes failure and retries sending the message. This ensures messages are not lost but can cause duplicates if not handled carefully.

Result

You understand how message delivery reliability is maintained through acknowledgements and retries.

Knowing this mechanism prevents surprises with duplicate messages and helps design idempotent processing.

6

Advanced

Scaling with multiple subscriptions

🤔Before reading on: do you think multiple subscriptions to one topic share messages or get copies? Commit to your answer.

Concept: Each subscription gets its own copy of messages, allowing different subscribers to process messages independently.

A topic can have many subscriptions. Each subscription receives all messages published to the topic. This allows different parts of a system to react to the same events in their own way without interfering with each other.

Result

You can design systems where multiple services listen to the same events without conflict.

Understanding message fan-out via subscriptions enables building complex, decoupled architectures.

7

Expert

Dead-letter topics and error handling

🤔Before reading on: do you think failed messages are lost or stored somewhere? Commit to your answer.

Concept: Messages that cannot be processed after retries are sent to a special dead-letter topic for later inspection.

If a subscriber repeatedly fails to acknowledge a message, it is moved to a dead-letter topic. This prevents blocking the system and allows developers to analyze and fix problematic messages separately.

Result

You can handle errors gracefully without losing messages or stopping the system.

Knowing dead-letter topics improves system reliability and helps maintain smooth message processing under failure.

Under the Hood

When a message is published to a topic, it is stored temporarily by the messaging system. Each subscription linked to that topic keeps track of which messages it has delivered and acknowledged. For push subscriptions, the system sends messages to subscriber endpoints automatically. For pull subscriptions, the subscriber requests messages. The system retries delivery if acknowledgements are not received, and moves undeliverable messages to dead-letter topics.

Why designed this way?

This design separates message sending from receiving, allowing systems to scale independently. It avoids tight coupling and supports reliable delivery with retries and error handling. Alternatives like direct calls would create dependencies and reduce flexibility.

┌───────────────┐
│   Publisher   │
└──────┬────────┘
       │
       ▼
┌───────────────┐
│     Topic     │
└──────┬────────┘
       │
┌──────┴───────┐      ┌───────────────┐
│ Subscription │      │ Subscription  │
│   (Push)    │      │    (Pull)     │
└──────┬───────┘      └──────┬────────┘
       │                     │
       ▼                     ▼
┌───────────────┐      ┌───────────────┐
│  Subscriber   │      │  Subscriber   │
│  Endpoint     │      │  Requests Msg │
└───────────────┘      └───────────────┘

Myth Busters - 4 Common Misconceptions

Quick: Do all subscriptions share the same message stream or get separate copies? Commit to your answer.

Common Belief: All subscriptions to a topic share the same message stream and split messages among themselves.

Reality: Each subscription receives its own full copy of every message published to the topic.

Why it matters: Believing messages are split can cause design errors where subscribers miss messages or assume exclusive processing.

Quick: Does a message disappear immediately after delivery or only after acknowledgement? Commit to your answer.

Common Belief: Messages are removed from the system as soon as they are delivered to a subscriber.

Reality: Messages remain until the subscriber acknowledges them, ensuring reliable delivery even if the subscriber fails.

Why it matters: Assuming immediate removal can lead to lost messages if subscribers crash before processing.

Quick: Are push subscriptions always better than pull? Commit to your answer.

Common Belief: Push subscriptions are always the best because they deliver messages instantly.

Reality: Push is good for real-time needs, but pull gives subscribers control over when and how many messages to receive, which can be better for batch processing or rate control.

Why it matters: Choosing push blindly can overwhelm subscribers or cause failures if they cannot handle the message rate.

Quick: Are dead-letter topics optional or mandatory for error handling? Commit to your answer.

Common Belief: Dead-letter topics are optional and not necessary for reliable messaging.

Reality: Dead-letter topics are essential for handling messages that repeatedly fail, preventing system blockage and data loss.

Why it matters: Ignoring dead-letter topics can cause message processing to stall and hide errors.

Expert Zone

1

Subscriptions maintain independent cursors, so message delivery order can differ per subscriber depending on acknowledgement timing.

2

Push subscriptions require subscriber endpoints to be highly available and secure, as messages are sent directly over the network.

3

Message retention policies affect how long unacknowledged messages stay available, impacting system behavior under load.

When NOT to use

Topics and subscriptions are not ideal for low-latency, synchronous communication where immediate response is needed. Alternatives like direct API calls or gRPC are better. Also, for very large message sizes or complex transactions, other systems like databases or event sourcing may be preferable.

Production Patterns

In production, topics often represent event streams like user actions or system logs. Multiple subscriptions allow different microservices to react independently, such as updating databases, triggering alerts, or generating reports. Dead-letter topics are monitored to catch and fix processing errors without stopping the whole system.

Connections

Event-driven architecture

Topics and subscriptions implement the messaging backbone for event-driven systems.

Understanding topics and subscriptions clarifies how events flow asynchronously between services in modern architectures.

Publish-subscribe pattern

Topics and subscriptions are a cloud implementation of the publish-subscribe messaging pattern.

Knowing this pattern helps grasp the decoupling and scalability benefits of topics and subscriptions.

Postal mail system

Topics and subscriptions function like mailboxes and mail keys in postal systems.

Recognizing this connection helps understand message delivery, storage, and retrieval in distributed systems.

Common Pitfalls

#1 Assuming messages are deleted immediately after delivery.

Wrong approach:

Publish message to topic; subscriber reads message; assume message is gone from system.

Correct approach:

Publish message to topic; subscriber reads and acknowledges message; system deletes message only after acknowledgement.

Root cause:

Misunderstanding the acknowledgement mechanism leads to lost messages if subscribers fail before confirming receipt.

#2 Using push subscriptions without ensuring subscriber endpoint reliability.

Wrong approach:

Configure push subscription to send messages to an unstable or unsecured endpoint.

Correct approach:

Ensure subscriber endpoint is highly available, secure, and can handle message load before using push subscriptions.

Root cause:

Ignoring endpoint readiness causes message loss or delivery failures.

#3 Not using dead-letter topics for failed messages.

Wrong approach:

Configure subscriptions without dead-letter topics; failed messages are retried indefinitely or lost.

Correct approach:

Set up dead-letter topics to capture messages that fail processing after retries for later analysis.

Root cause:

Overlooking error handling leads to blocked processing and hidden message failures.

Key Takeaways

Topics are channels where messages are published for others to receive.

Subscriptions connect to topics and manage message delivery and acknowledgement.

Push and pull are two delivery methods, each suited for different needs.

Reliable messaging depends on acknowledgements, retries, and dead-letter topics.

Multiple subscriptions allow independent processing of the same messages by different services.