Dead letter topics in GCP - Commands & Configuration

Introduction

Sometimes messages sent to a topic in Google Cloud Pub/Sub cannot be processed correctly. Dead letter topics help catch these problematic messages so they are not lost and can be reviewed or retried later.

When your subscriber fails to process some messages and you want to keep those messages for later analysis.

When you want to avoid losing messages that cause errors in your application.

When you want to separate normal message flow from problematic messages to improve system reliability.

When you want to monitor and alert on message processing failures.

When you want to retry processing failed messages after fixing the issue.

Config File - subscription.yaml

subscription.yaml

apiVersion: pubsub.cnrm.cloud.google.com/v1beta1
kind: PubSubSubscription
metadata:
  name: example-subscription
spec:
  topicRef:
    name: example-topic
  ackDeadlineSeconds: 20
  deadLetterPolicy:
    deadLetterTopicRef:
      name: example-dead-letter-topic
    maxDeliveryAttempts: 5

This YAML file creates a Pub/Sub subscription named example-subscription that listens to example-topic.

The deadLetterPolicy section sets example-dead-letter-topic as the dead letter topic where messages that fail processing after 5 attempts are sent.

ackDeadlineSeconds sets the time the subscriber has to acknowledge a message before it is redelivered.

Commands

Create the main topic where messages will be published.

Terminal

gcloud pubsub topics create example-topic

Expected Output

Created topic [projects/your-project/topics/example-topic].

Create the dead letter topic to receive messages that fail processing multiple times.

Terminal

gcloud pubsub topics create example-dead-letter-topic

Expected Output

Created topic [projects/your-project/topics/example-dead-letter-topic].

Create the subscription with dead letter policy using the configuration file.

Terminal

kubectl apply -f subscription.yaml

Expected Output

pubsubsubscription.pubsub.cnrm.cloud.google.com/example-subscription created

Verify the subscription and check that the dead letter policy is set correctly.

Terminal

gcloud pubsub subscriptions describe example-subscription

Expected Output

name: projects/your-project/subscriptions/example-subscription

ackDeadlineSeconds: 20

deadLetterPolicy:

  deadLetterTopic: projects/your-project/topics/example-dead-letter-topic

  maxDeliveryAttempts: 5

pushConfig: {}

topic: projects/your-project/topics/example-topic

Key Concept

If you remember nothing else from this pattern, remember: dead letter topics catch messages that fail processing repeatedly so you can handle them separately without losing data.

Common Mistakes

Not creating the dead letter topic before assigning it in the subscription.

The subscription creation fails because the dead letter topic does not exist yet.

Always create the dead letter topic first before referencing it in the subscription.

Setting maxDeliveryAttempts too low or too high without testing.

Too low causes messages to go to dead letter topic too quickly; too high delays error handling.

Choose a reasonable maxDeliveryAttempts value like 5 and adjust based on your processing needs.

Not verifying the dead letter policy after subscription creation.

You might think the policy is set but it could be missing or misconfigured.

Always describe the subscription to confirm the dead letter policy is active.

Summary

Create the main topic and the dead letter topic first.

Create a subscription with a dead letter policy pointing to the dead letter topic.

Verify the subscription to ensure the dead letter policy is set correctly.

Dead letter topics help catch and isolate messages that fail processing repeatedly.