Dead letter topics in GCP - Commands & Configuration

Introduction

Sometimes messages sent to a topic in Google Cloud Pub/Sub cannot be processed correctly. Dead letter topics help catch these problematic messages so they are not lost and can be reviewed or retried later.

When your subscriber fails to process some messages and you want to keep those messages for later analysis.

When you want to avoid losing messages that cause errors in your application.

When you want to separate normal message flow from problematic messages to improve system reliability.

When you want to monitor and alert on message processing failures.

When you want to retry processing failed messages after fixing the issue.

Config File - subscription.yaml

subscription.yaml

apiVersion: pubsub.cnrm.cloud.google.com/v1beta1
kind: PubSubSubscription
metadata:
  name: example-subscription
spec:
  topicRef:
    name: example-topic
  ackDeadlineSeconds: 20
  deadLetterPolicy:
    deadLetterTopicRef:
      name: example-dead-letter-topic
    maxDeliveryAttempts: 5

This YAML file creates a Pub/Sub subscription named example-subscription that listens to example-topic.

The deadLetterPolicy section sets example-dead-letter-topic as the dead letter topic where messages that fail processing after 5 attempts are sent.

ackDeadlineSeconds sets the time the subscriber has to acknowledge a message before it is redelivered.

Commands

Create the main topic where messages will be published.

Terminal

gcloud pubsub topics create example-topic

Expected Output

Created topic [projects/your-project/topics/example-topic].

Create the dead letter topic to receive messages that fail processing multiple times.

Terminal

gcloud pubsub topics create example-dead-letter-topic

Expected Output

Created topic [projects/your-project/topics/example-dead-letter-topic].

Create the subscription with dead letter policy using the configuration file.

Terminal

kubectl apply -f subscription.yaml

Expected Output

pubsubsubscription.pubsub.cnrm.cloud.google.com/example-subscription created

Verify the subscription and check that the dead letter policy is set correctly.

Terminal

gcloud pubsub subscriptions describe example-subscription

Expected Output

name: projects/your-project/subscriptions/example-subscription

ackDeadlineSeconds: 20

deadLetterPolicy:

  deadLetterTopic: projects/your-project/topics/example-dead-letter-topic

  maxDeliveryAttempts: 5

pushConfig: {}

topic: projects/your-project/topics/example-topic

Key Concept

If you remember nothing else from this pattern, remember: dead letter topics catch messages that fail processing repeatedly so you can handle them separately without losing data.

Common Mistakes

Not creating the dead letter topic before assigning it in the subscription.

The subscription creation fails because the dead letter topic does not exist yet.

Always create the dead letter topic first before referencing it in the subscription.

Setting maxDeliveryAttempts too low or too high without testing.

Too low causes messages to go to dead letter topic too quickly; too high delays error handling.

Choose a reasonable maxDeliveryAttempts value like 5 and adjust based on your processing needs.

Not verifying the dead letter policy after subscription creation.

You might think the policy is set but it could be missing or misconfigured.

Always describe the subscription to confirm the dead letter policy is active.

Summary

Create the main topic and the dead letter topic first.

Create a subscription with a dead letter policy pointing to the dead letter topic.

Verify the subscription to ensure the dead letter policy is set correctly.

Dead letter topics help catch and isolate messages that fail processing repeatedly.

Practice

1. What is the main purpose of a dead letter topic in Google Cloud Pub/Sub?

easy

A. To store all successfully delivered messages

B. To speed up message delivery

C. To catch messages that fail delivery multiple times

D. To encrypt messages before sending

Solution

1. Step 1: Understand message delivery failures

   Messages that cannot be delivered after several attempts need special handling to avoid blocking the main flow.

2. Step 2: Role of dead letter topics

   Dead letter topics collect these failed messages separately so the main topic stays clean and reliable.

3. Final Answer:

   To catch messages that fail delivery multiple times -> Option C

4. Quick Check:

   Dead letter topic = catch failed messages [OK]

Hint: Dead letter topics catch failed messages only [OK]

Common Mistakes:

• Thinking dead letter topics speed delivery
• Confusing dead letter topics with encryption
• Assuming they store successful messages

2. Which block in a Pub/Sub subscription configuration is used to set a dead letter topic?

easy

A. message_storage_policy

B. retry_policy

C. ack_deadline_seconds

D. dead_letter_policy

Solution

1. Step 1: Identify configuration blocks for subscriptions

   Pub/Sub subscriptions have specific blocks for retry, dead letter, and storage policies.

2. Step 2: Locate dead letter topic setting

   The dead_letter_policy block is where you specify the dead letter topic and max delivery attempts.

3. Final Answer:

   dead_letter_policy -> Option D

4. Quick Check:

   Dead letter topic set in dead_letter_policy [OK]

Hint: Dead letter topic is in dead_letter_policy block [OK]

Common Mistakes:

• Confusing with retry_policy block
• Using message_storage_policy incorrectly
• Setting dead letter topic in ack_deadline_seconds

3. Given this subscription config snippet:

{
  "dead_letter_policy": {
    "dead_letter_topic": "projects/myproj/topics/dlt",
    "max_delivery_attempts": 5
  }
}

What happens when a message fails delivery 6 times?

medium

A. The message is retried indefinitely

B. The message is sent to the dead letter topic

C. The message is deleted immediately

D. The message is sent back to the main topic

Solution

1. Step 1: Understand max_delivery_attempts meaning

   Messages are retried up to the max_delivery_attempts count before moving to dead letter topic.

2. Step 2: Apply max_delivery_attempts = 5

   After 5 failed attempts, the 6th failure triggers sending the message to the dead letter topic.

3. Final Answer:

   The message is sent to the dead letter topic -> Option B

4. Quick Check:

   Failures > max_delivery_attempts = dead letter send [OK]

Hint: Messages move to dead letter after max attempts exceeded [OK]

Common Mistakes:

• Thinking messages retry forever
• Assuming immediate deletion on failure
• Believing message returns to main topic

4. You configured a dead letter topic but messages are not appearing there after failures. What is a likely cause?

medium

A. Dead letter topic does not exist or lacks permissions

B. max_delivery_attempts is set to 1

C. Subscription ack deadline is too long

D. Main topic has no messages

Solution

1. Step 1: Check dead letter topic existence and permissions

   If the dead letter topic is missing or the subscription lacks publish permission, messages cannot be sent there.

2. Step 2: Understand other options

   max_delivery_attempts=1 means one try before dead letter, ack deadline affects delivery timing, and main topic empty means no messages to fail.

3. Final Answer:

   Dead letter topic does not exist or lacks permissions -> Option A

4. Quick Check:

   Dead letter topic must exist and be accessible [OK]

Hint: Dead letter topic must exist and allow publishing [OK]

Common Mistakes:

• Ignoring permissions errors
• Confusing ack deadline with dead letter setup
• Assuming empty main topic causes dead letter issues

5. You want to ensure messages that fail more than 3 times go to a dead letter topic, but also want to limit the total retry time to 10 minutes. Which configuration approach is best?

hard

A. Set dead_letter_policy.max_delivery_attempts to 3 and configure retry_policy.minimum_backoff and maximum_backoff to limit retry duration

B. Set dead_letter_policy.max_delivery_attempts to 10 and rely on ack deadline for timing

C. Only set retry_policy.maximum_backoff to 10 minutes without dead letter policy

D. Set dead_letter_policy.max_delivery_attempts to 3 and disable retries

Solution

1. Step 1: Use max_delivery_attempts to limit retries

   Setting max_delivery_attempts to 3 ensures messages move to dead letter after 3 failed deliveries.

2. Step 2: Control retry timing with retry_policy

   Configuring minimum and maximum backoff controls how long retries take, helping keep total retry time within 10 minutes.

3. Step 3: Evaluate other options

   Set dead_letter_policy.max_delivery_attempts to 10 and rely on ack deadline for timing ignores retry timing, C lacks dead letter handling, and D disables retries which may lose messages prematurely.

4. Final Answer:

   Set dead_letter_policy.max_delivery_attempts to 3 and configure retry_policy minimum and maximum backoff -> Option A

5. Quick Check:

   Combine max_delivery_attempts and retry_policy for retries and dead letter [OK]

Hint: Use max_delivery_attempts plus retry_policy for timing control [OK]

Common Mistakes:

• Ignoring retry_policy for timing
• Disabling retries loses messages
• Setting max_delivery_attempts too high