Message retention and acknowledgment in GCP - Commands & Configuration

Introduction

When you send messages between services, you want to make sure they are not lost and are processed only once. Message retention keeps messages available for a time, and acknowledgment tells the system you received and handled a message.

When you want to keep messages available for a short time in case the receiver is slow or temporarily down

When you want to confirm that a message was processed so it is not sent again

When you want to control how long unacknowledged messages stay in the system before being redelivered

When you want to avoid losing messages during network or service interruptions

When you want to manage message flow to prevent overload on your processing service

Config File - subscription.yaml

subscription.yaml

apiVersion: pubsub.cnrm.cloud.google.com/v1beta1
kind: PubSubSubscription
metadata:
  name: example-subscription
spec:
  topicRef:
    name: example-topic
  ackDeadlineSeconds: 20
  retainAckedMessages: true
  messageRetentionDuration: 600s

This file creates a subscription to a Pub/Sub topic.

ackDeadlineSeconds sets how many seconds the subscriber has to acknowledge a message before it is redelivered.

retainAckedMessages keeps messages even after acknowledgment for the retention duration.

messageRetentionDuration sets how long messages are kept (600s = 10 minutes).

Commands

This command creates a subscription named 'example-subscription' to the topic 'example-topic'. It sets the acknowledgment deadline to 20 seconds, retains acknowledged messages, and keeps messages for 10 minutes.

Terminal

gcloud pubsub subscriptions create example-subscription --topic=example-topic --ack-deadline=20 --retain-acked-messages --message-retention-duration=600s

Expected Output

Created subscription [example-subscription].

→

--ack-deadline - Sets how long to wait for message acknowledgment before redelivery

→

--retain-acked-messages - Keeps messages after acknowledgment for the retention duration

→

--message-retention-duration - Sets how long messages are retained

This command pulls one message from the subscription and automatically acknowledges it, telling the system the message was received and processed.

Terminal

gcloud pubsub subscriptions pull example-subscription --auto-ack --limit=1

Expected Output

DATA: "Hello, world!"

ACK ID: ACK_ID_1234567890

→

--auto-ack - Automatically acknowledges the message after pulling

→

--limit - Limits the number of messages pulled

This command shows the current settings of the subscription, including acknowledgment deadline and message retention settings.

Terminal

gcloud pubsub subscriptions describe example-subscription

Expected Output

name: projects/my-project/subscriptions/example-subscription

ackDeadlineSeconds: 20

retainAckedMessages: true

messageRetentionDuration: 600s

Key Concept

If you remember nothing else from this pattern, remember: acknowledgment tells the system you processed a message, and retention keeps messages available for a set time even after acknowledgment.

Common Mistakes

Not acknowledging messages after pulling them

Messages will be redelivered repeatedly, causing duplicate processing

Always acknowledge messages after processing, either manually or with auto-ack flags

Setting acknowledgment deadline too short

Messages may be redelivered before processing finishes, causing duplicates

Set acknowledgment deadline long enough for your processing time

Not enabling message retention when needed

Messages are deleted immediately after acknowledgment, losing history or retry options

Enable retainAckedMessages and set messageRetentionDuration if you want to keep messages after acknowledgment

Summary

Create a subscription with acknowledgment deadline and message retention settings.

Pull messages and acknowledge them to prevent redelivery.

Check subscription settings to verify retention and acknowledgment configurations.

Practice

1. What happens to a message in Google Cloud Pub/Sub if it is not acknowledged within the retention period?

easy

A. The message is automatically acknowledged.

B. The message is deleted and no longer available for delivery.

C. The message is moved to a dead-letter queue.

D. The message is duplicated and sent again.

Solution

1. Step 1: Understand message retention

   Message retention defines how long unacknowledged messages are kept before deletion.

2. Step 2: Effect of no acknowledgment within retention

   If a message is not acknowledged before retention expires, it is deleted and lost.

3. Final Answer:

   The message is deleted and no longer available for delivery. -> Option B

4. Quick Check:

   Unacknowledged message + retention expired = deleted [OK]

Hint: Unacknowledged messages expire and get deleted after retention [OK]

Common Mistakes:

• Thinking messages auto-acknowledge after retention
• Assuming messages move to dead-letter queue automatically
• Believing messages duplicate on retention expiry

2. Which of the following is the correct way to acknowledge a message in Google Cloud Pub/Sub client libraries?

easy

A. Call message.ack() after processing the message.

B. Call message.delete() to remove the message.

C. Call message.complete() to finish processing.

D. Call message.finish() to acknowledge.

Solution

1. Step 1: Identify acknowledgment method

   In Pub/Sub client libraries, acknowledging a message is done by calling ack() on the message object.

2. Step 2: Verify other options

   Methods like delete(), complete(), or finish() are not valid for acknowledgment.

3. Final Answer:

   Call message.ack() after processing the message. -> Option A

4. Quick Check:

   Use message.ack() to acknowledge [OK]

Hint: Use message.ack() to confirm message processed [OK]

Common Mistakes:

• Using incorrect method names like delete or finish
• Not acknowledging messages at all
• Confusing acknowledgment with message deletion

3. Consider this code snippet using Google Cloud Pub/Sub subscriber client:

def callback(message):
    print(f"Received: {message.data.decode('utf-8')}")
    # No acknowledgment called here

subscriber.subscribe(subscription_path, callback=callback)

What will happen to the messages if the callback never calls message.ack()?

medium

A. Messages will be acknowledged automatically after printing.

B. Messages will be lost immediately after printing.

C. Messages will be redelivered until acknowledged or retention expires.

D. Messages will be sent to a dead-letter topic instantly.

Solution

1. Step 1: Understand callback behavior without ack

   If message.ack() is not called, Pub/Sub considers the message unacknowledged.

2. Step 2: Effect on message delivery

   Unacknowledged messages are redelivered repeatedly until acknowledged or retention period expires.

3. Final Answer:

   Messages will be redelivered until acknowledged or retention expires. -> Option C

4. Quick Check:

   No ack means redelivery until retention ends [OK]

Hint: No ack means message redelivery until retention ends [OK]

Common Mistakes:

• Assuming messages auto-acknowledge after callback
• Thinking messages are lost immediately without ack
• Believing messages go to dead-letter topic without config

4. You have set a message retention duration of 7 days on your Pub/Sub subscription. However, messages are disappearing after 1 day. What is the most likely cause?

medium

A. Messages are being acknowledged by the subscriber before 1 day.

B. The retention duration setting is ignored by Pub/Sub.

C. The subscription is a push type and does not support retention.

D. Messages are automatically deleted after 24 hours regardless of settings.

Solution

1. Step 1: Check retention vs acknowledgment

   Retention controls how long unacknowledged messages stay. If messages are acknowledged, they are removed immediately.

2. Step 2: Analyze disappearance cause

   If messages disappear early, likely they were acknowledged by the subscriber before 1 day.

3. Final Answer:

   Messages are being acknowledged by the subscriber before 1 day. -> Option A

4. Quick Check:

   Acknowledged messages removed before retention ends [OK]

Hint: Acknowledged messages delete immediately, ignoring retention [OK]

Common Mistakes:

• Assuming retention is ignored by Pub/Sub
• Believing push subscriptions don't support retention
• Thinking messages auto-delete after 24 hours always

5. You want to ensure that messages in your Pub/Sub subscription are retained for 3 days even if they are not acknowledged, but also want to avoid message duplication after acknowledgment. Which configuration should you apply?

hard

A. Set message retention duration to 0 and rely on subscriber retries.

B. Set message retention duration to 3 days and disable acknowledgments.

C. Set message retention duration to 7 days and never acknowledge messages.

D. Set message retention duration to 3 days and ensure subscriber acknowledges messages promptly.

Solution

1. Step 1: Understand retention and acknowledgment roles

   Retention keeps unacknowledged messages for specified time. Acknowledgment removes messages to avoid duplicates.

2. Step 2: Choose configuration to meet goals

   Setting retention to 3 days keeps messages if unacknowledged. Prompt acknowledgment removes processed messages, preventing duplicates.

3. Final Answer:

   Set message retention duration to 3 days and ensure subscriber acknowledges messages promptly. -> Option D

4. Quick Check:

   Retention + prompt ack = no duplicates, messages kept [OK]

Hint: Retention keeps unacked; ack removes to prevent duplicates [OK]

Common Mistakes:

• Disabling acknowledgments causing duplicates
• Setting retention to 0 losing unacked messages quickly
• Never acknowledging causing message buildup and duplicates