Message retention and acknowledgment in GCP - Mini Project: Build & Apply

Message retention and acknowledgment

📖 Scenario: You are setting up a Google Cloud Pub/Sub topic and subscription for a simple messaging system. You want to control how long messages are kept and ensure messages are acknowledged properly to avoid duplicates.

🎯 Goal: Build a Pub/Sub topic and subscription with a message retention duration of 7 days and configure the subscription to require explicit message acknowledgment.

📋 What You'll Learn

Create a Pub/Sub topic named my-topic

Create a Pub/Sub subscription named my-subscription attached to my-topic

Set the message retention duration of the topic to 7 days

Configure the subscription to require explicit acknowledgment of messages

💡 Why This Matters

🌍 Real World

Message retention and acknowledgment settings are crucial for reliable message processing in distributed systems, ensuring no messages are lost or processed multiple times.

💼 Career

Understanding Pub/Sub configuration is important for cloud engineers and developers working with event-driven architectures and asynchronous communication.

1

Create the Pub/Sub topic

Create a Pub/Sub topic named my-topic using the Google Cloud SDK command gcloud pubsub topics create my-topic.

GCP

# Create the Pub/Sub topic named my-topic
# Your code here

Hint

Use the gcloud pubsub topics create command followed by the topic name.

2

Create the Pub/Sub subscription

Create a Pub/Sub subscription named my-subscription attached to the topic my-topic using the command gcloud pubsub subscriptions create my-subscription --topic=my-topic.

GCP

gcloud pubsub topics create my-topic
# Create the subscription named my-subscription attached to my-topic
# Your code here

Hint

Use the gcloud pubsub subscriptions create command with the --topic flag.

3

Set message retention duration

Update the topic my-topic to set the message retention duration to 7 days (604800 seconds) using the command gcloud pubsub topics update my-topic --message-retention-duration=604800s.

GCP

gcloud pubsub topics create my-topic
gcloud pubsub subscriptions create my-subscription --topic=my-topic
# Set message retention duration to 7 days for my-topic
# Your code here

Hint

Use the gcloud pubsub topics update command with the --message-retention-duration flag set to 604800s.

4

Configure acknowledgment requirement

Ensure the subscription my-subscription requires explicit acknowledgment by setting the --ack-deadline to 10 seconds using the command gcloud pubsub subscriptions update my-subscription --ack-deadline=10.

GCP

gcloud pubsub topics create my-topic
gcloud pubsub subscriptions create my-subscription --topic=my-topic
gcloud pubsub topics update my-topic --message-retention-duration=604800s
# Set acknowledgment deadline to 10 seconds for my-subscription
# Your code here

Hint

Use the gcloud pubsub subscriptions update command with the --ack-deadline flag set to 10 seconds.

Practice

1. What happens to a message in Google Cloud Pub/Sub if it is not acknowledged within the retention period?

easy

A. The message is automatically acknowledged.

B. The message is deleted and no longer available for delivery.

C. The message is moved to a dead-letter queue.

D. The message is duplicated and sent again.

Solution

1. Step 1: Understand message retention

   Message retention defines how long unacknowledged messages are kept before deletion.

2. Step 2: Effect of no acknowledgment within retention

   If a message is not acknowledged before retention expires, it is deleted and lost.

3. Final Answer:

   The message is deleted and no longer available for delivery. -> Option B

4. Quick Check:

   Unacknowledged message + retention expired = deleted [OK]

Hint: Unacknowledged messages expire and get deleted after retention [OK]

Common Mistakes:

• Thinking messages auto-acknowledge after retention
• Assuming messages move to dead-letter queue automatically
• Believing messages duplicate on retention expiry

2. Which of the following is the correct way to acknowledge a message in Google Cloud Pub/Sub client libraries?

easy

A. Call message.ack() after processing the message.

B. Call message.delete() to remove the message.

C. Call message.complete() to finish processing.

D. Call message.finish() to acknowledge.

Solution

1. Step 1: Identify acknowledgment method

   In Pub/Sub client libraries, acknowledging a message is done by calling ack() on the message object.

2. Step 2: Verify other options

   Methods like delete(), complete(), or finish() are not valid for acknowledgment.

3. Final Answer:

   Call message.ack() after processing the message. -> Option A

4. Quick Check:

   Use message.ack() to acknowledge [OK]

Hint: Use message.ack() to confirm message processed [OK]

Common Mistakes:

• Using incorrect method names like delete or finish
• Not acknowledging messages at all
• Confusing acknowledgment with message deletion

3. Consider this code snippet using Google Cloud Pub/Sub subscriber client:

def callback(message):
    print(f"Received: {message.data.decode('utf-8')}")
    # No acknowledgment called here

subscriber.subscribe(subscription_path, callback=callback)

What will happen to the messages if the callback never calls message.ack()?

medium

A. Messages will be acknowledged automatically after printing.

B. Messages will be lost immediately after printing.

C. Messages will be redelivered until acknowledged or retention expires.

D. Messages will be sent to a dead-letter topic instantly.

Solution

1. Step 1: Understand callback behavior without ack

   If message.ack() is not called, Pub/Sub considers the message unacknowledged.

2. Step 2: Effect on message delivery

   Unacknowledged messages are redelivered repeatedly until acknowledged or retention period expires.

3. Final Answer:

   Messages will be redelivered until acknowledged or retention expires. -> Option C

4. Quick Check:

   No ack means redelivery until retention ends [OK]

Hint: No ack means message redelivery until retention ends [OK]

Common Mistakes:

• Assuming messages auto-acknowledge after callback
• Thinking messages are lost immediately without ack
• Believing messages go to dead-letter topic without config

4. You have set a message retention duration of 7 days on your Pub/Sub subscription. However, messages are disappearing after 1 day. What is the most likely cause?

medium

A. Messages are being acknowledged by the subscriber before 1 day.

B. The retention duration setting is ignored by Pub/Sub.

C. The subscription is a push type and does not support retention.

D. Messages are automatically deleted after 24 hours regardless of settings.

Solution

1. Step 1: Check retention vs acknowledgment

   Retention controls how long unacknowledged messages stay. If messages are acknowledged, they are removed immediately.

2. Step 2: Analyze disappearance cause

   If messages disappear early, likely they were acknowledged by the subscriber before 1 day.

3. Final Answer:

   Messages are being acknowledged by the subscriber before 1 day. -> Option A

4. Quick Check:

   Acknowledged messages removed before retention ends [OK]

Hint: Acknowledged messages delete immediately, ignoring retention [OK]

Common Mistakes:

• Assuming retention is ignored by Pub/Sub
• Believing push subscriptions don't support retention
• Thinking messages auto-delete after 24 hours always

5. You want to ensure that messages in your Pub/Sub subscription are retained for 3 days even if they are not acknowledged, but also want to avoid message duplication after acknowledgment. Which configuration should you apply?

hard

A. Set message retention duration to 0 and rely on subscriber retries.

B. Set message retention duration to 3 days and disable acknowledgments.

C. Set message retention duration to 7 days and never acknowledge messages.

D. Set message retention duration to 3 days and ensure subscriber acknowledges messages promptly.

Solution

1. Step 1: Understand retention and acknowledgment roles

   Retention keeps unacknowledged messages for specified time. Acknowledgment removes messages to avoid duplicates.

2. Step 2: Choose configuration to meet goals

   Setting retention to 3 days keeps messages if unacknowledged. Prompt acknowledgment removes processed messages, preventing duplicates.

3. Final Answer:

   Set message retention duration to 3 days and ensure subscriber acknowledges messages promptly. -> Option D

4. Quick Check:

   Retention + prompt ack = no duplicates, messages kept [OK]

Hint: Retention keeps unacked; ack removes to prevent duplicates [OK]

Common Mistakes:

• Disabling acknowledgments causing duplicates
• Setting retention to 0 losing unacked messages quickly
• Never acknowledging causing message buildup and duplicates