Bird
Raised Fist0

How to Write a Post-Mortem - Blameless Culture, Root Cause Analysis, Action Items

HLDhardInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/6
Incident Detection

Detect incident and emit event

Incident Detection detects a system failure and emits an incident event to start post-mortem

Protocol:Kafka Event
BeforeNo active incident event exists for this failure
AfterIncident event published to Kafka for downstream processing
Post-Mortem Frontend

Receive incident event and open post-mortem UI

Frontend subscribes to incident events and opens UI for user to write post-mortem

Protocol:WebSocket
BeforeUser has no open post-mortem for this incident
AfterPost-mortem UI initialized with incident details for editing
Collaboration Service

Enable real-time collaborative editing

Collaboration Service manages WebSocket connections and Redis state for live editing

Protocol:WebSocket + Redis GET/SET
BeforeNo active collaboration session for this post-mortem
AfterMultiple users connected, edits synchronized in real-time via Redis
Document Store

Persist post-mortem document

Post-mortem content saved durably in Cassandra for long-term storage

Protocol:Cassandra Write
BeforePost-mortem content exists only in volatile memory
AfterPost-mortem document durably stored in Cassandra
Search & Analytics

Index post-mortem for search

New post-mortem document indexed in Elasticsearch for fast retrieval

Protocol:Elasticsearch Index
BeforePost-mortem not searchable in analytics system
AfterPost-mortem indexed and searchable in Elasticsearch
Action Item Tracker

Create and track action items

Action items extracted from post-mortem are tracked for follow-up

Protocol:Internal API
BeforeNo action items exist for this post-mortem
AfterAction items created and assigned for resolution
🏗️
Architecture Diagram
Architecture Diagram
┌───────────────────────┐
│   Incident Detection  │
│  (Detects incidents)  │
└────────────┬──────────┘
             │ Kafka Event
             ▼
┌───────────────────────┐
│ Post-Mortem Frontend  │
│ (User UI for reports) │
└───────┬───────────────┘
        │ WebSocket
        ▼
┌───────────────────────────────┐
│ Collaboration Service          │
│ (Redis + WebSocket for realtime│
│  editing & presence)           │
└─────────────┬─────────────────┘
              │ Redis GET/SET
              ▼
      ┌───────────────────┐
      │ Document Store    │
      │ (Cassandra DB)    │
      └─────────┬─────────┘
                │ Write Post-Mortem
                ▼
      ┌───────────────────┐
      │ Search & Analytics│
      │ (Elasticsearch)   │
      └─────────┬─────────┘
                │ Index Document
                ▼
      ┌───────────────────┐
      │ Action Item Tracker│
      │ (Tracks tasks)     │
      └───────────────────┘
Design Rationale

This architecture supports a blameless post-mortem workflow triggered by incident detection. The Incident Detection component emits an event that the Post-Mortem Frontend listens to, opening a UI for users to collaboratively write reports. The Collaboration Service uses Redis and WebSocket to synchronize edits in real-time among multiple users. Once finalized, the post-mortem document is durably stored in Cassandra, ensuring persistence. The document is then indexed in Elasticsearch to enable search and analytics on past incidents. Finally, the Action Item Tracker manages tasks derived from the post-mortem, ensuring follow-up and continuous improvement. This vertical flow ensures clear separation of concerns and real-time collaboration with durable storage and analytics.

Primary Focus

✓

Practice

(1/5)
1. For storing large blobs with frequent read access and occasional updates, which storage architecture is best suited to optimize cost and performance in this blob store design?
medium
A. Store entire blobs in a distributed key-value store with synchronous replication
B. Use a relational database with BLOB columns and transactional updates
C. Use chunked blob storage with immutable chunks and metadata pointers for versioning
D. Store blobs as single large files on a network file system with locking for updates

Solution

  1. Step 1: Analyze chunked storage with immutable chunks

    This approach enables efficient caching, deduplication, and cheap versioning by reusing unchanged chunks.

  2. Step 2: Eliminate other options

    Key-value stores struggle with large blobs and synchronous replication adds latency; network file systems have locking bottlenecks; relational DBs are inefficient for large binary data.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Chunked immutable storage optimizes cost and versioning [check]
Hint: Immutable chunking enables cheap versioning and caching [check]
Common Mistakes:
2. If the Kafka cluster responsible for connection event streaming goes down, what exactly breaks in the system's connection request processing?
medium
A. Connection requests are lost immediately with no retry or buffering
B. Connection requests are redirected to a backup database that duplicates all writes
C. Connection requests block the user synchronously until Kafka recovers, causing UI timeouts
D. Connection requests are accepted but downstream services do not receive updates, causing stale connection graphs

Solution

  1. Step 1: Understand Kafka's role in event streaming

    Kafka buffers and reliably delivers connection events to downstream services asynchronously.

  2. Step 2: Trace failure impact

    If Kafka is down, writes to Kafka fail but connection requests may still be accepted at the API layer; however, downstream consumers do not get updates, causing stale graph state.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Kafka down -> downstream stale, no immediate request loss [check]
Hint: Kafka failure stalls downstream, not immediate request loss [OK]
Common Mistakes:
3. You must choose between a monolithic architecture and microservices for a system with rapidly evolving features and a small team. Which choice best justifies your design decision to the interviewer?
medium
A. Choose microservices because they always scale better regardless of team size
B. Choose monolith to reduce operational complexity and speed up feature delivery for a small team
C. Choose microservices to isolate failures even if it increases deployment overhead
D. Choose monolith because it requires no testing or CI/CD pipelines

Solution

  1. Step 1: Analyze team size and feature velocity constraints

    Small teams benefit from simpler architectures to reduce overhead and speed delivery.

  2. Step 2: Evaluate operational complexity of microservices

    Microservices add deployment and operational complexity, which can slow small teams.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Monolith reduces complexity, speeds delivery for small teams [check]
Hint: Small teams prefer monoliths for simplicity and speed [check]
Common Mistakes:
4. At 10x scale, which component in a Write-Through caching system is most likely to become the bottleneck first, and why?
hard
A. The cache, because synchronous writes increase latency and reduce throughput
B. The application servers, because they block waiting for cache writes to complete
C. The database, because it must handle every write synchronously from both cache and application
D. The network, because cache-to-database communication saturates bandwidth

Solution

  1. Step 1: Identify Write-Through bottleneck

    Write-Through synchronously writes to DB on every write, increasing DB load linearly with scale.

  2. Step 2: Compare other components

    Cache handles fast reads; network bandwidth usually sufficient; app servers block but DB saturation causes overall failure first.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    DB synchronous writes bottleneck Write-Through at scale [check]
Hint: Write-Through sync DB writes bottleneck at scale [check]
Common Mistakes:
5. At 10x scale, which component in an idempotent payment API system is most likely to become the bottleneck first, and why?
hard
A. The payment processor, because it cannot handle increased transaction volume
B. The client SDK, as it struggles to generate unique idempotency keys at scale
C. The API gateway, because it cannot route 10x more requests efficiently
D. The idempotency key store, due to synchronous writes and read-after-write consistency requirements

Solution

  1. Step 1: Identify idempotency store workload at scale

    Idempotency store must synchronously write and read keys with strong consistency, creating a high throughput bottleneck.

  2. Step 2: Compare other components

    Payment processors can be scaled horizontally; API gateways are stateless and scalable; client SDKs generating keys is lightweight.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Synchronous idempotency store writes limit scaling first [check]
Hint: Idempotency store sync writes throttle scaling first [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard