Bird
Raised Fist0

How to Write a Post-Mortem - Blameless Culture, Root Cause Analysis, Action Items

HLDhardInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🏗️
How to Write a Post-Mortem - Blameless Culture, Root Cause Analysis, Action Items
Designing a scalable, blameless post-mortem system to analyze failures, identify root causes, and track action items for continuous reliability improvements
hardSystem Failures and Post-MortemsFreq: medium
Prerequisites:Distributed Systems BasicsObservabilityIncident ManagementData Analysis
⚙️
Functional Requirements
  • Capture detailed incident timeline and system state snapshots automatically
  • Enable collaborative editing of post-mortem documents with version control
  • Support structured root cause analysis with causal chains and evidence linking
  • Track and assign action items with deadlines and status updates
  • Provide blameless culture guidelines and templates to avoid finger-pointing
  • Generate reports and dashboards summarizing incident trends and remediation progress
📊
Non-Functional Requirements
  • Support up to 10,000 concurrent post-mortem documents being edited
  • Store and retrieve post-mortem data with 99.99% availability
  • Ensure post-mortem document load and save latency under 200ms P99
  • Handle up to 100,000 incidents per month with retention of 5 years
  • Secure data with role-based access control and audit logging
❓
Clarifying Questions to Ask
  • What is the expected volume of incidents and post-mortems per day/month?
  • Should the system integrate with existing observability tools like Prometheus or Datadog?
  • What level of automation is expected in root cause analysis versus manual input?
  • Are there specific compliance or security requirements for storing incident data?
  • How should action items be tracked and escalated if overdue?
  • Is multi-team collaboration and cross-organization visibility required?
Out of Scope
✗ Real-time incident detection and alerting systems
✗ Automated remediation or rollback mechanisms
✗ General-purpose document editing unrelated to post-mortems
✗ Integration with external HR or legal systems
📌
Assumptions
  • 2 billion daily active users generating incidents at 0.005% rate → 100,000 incidents/day
  • Each incident generates one post-mortem document averaging 50 KB
  • Each post-mortem document edited 5 times on average during creation
  • Retention period of 5 years for all post-mortem data
⚡
QPS Calculation
write qps100,000 incidents/day × 5 edits / 86400 ≈ 5.8 writes/sec
peak write qpsAssuming 10× peak traffic during major outages → 58 writes/sec
read qpsAssuming 10× reads per write for reviews and dashboards → 580 reads/sec
💾
Storage Estimate
daily storage100,000 incidents × 50 KB = 5 GB/day
retention storage5 GB/day × 365 days × 5 years = 9.125 TB
with replication9.125 TB × 3 replicas = 27.375 TB total storage
🖥️
Server Estimate
document storeUsing Cassandra nodes with 500 writes/sec capacity → 1 node sufficient for writes, 3 nodes for replication and failover
collaboration serviceUsing Elasticsearch for search and Redis for real-time collaborative editing state; 3 Redis nodes for sub-ms latency and 5 Elasticsearch nodes for indexing and querying
🏗️
Architecture Diagram
Architecture Diagram
          +----------------------+
          |  Incident Detection  |
          +----------+-----------+
                     |
                     v
          +----------------------+
          | Post-Mortem Frontend |<----------------+
          +----------+-----------+                 |
                     |                             |
                     v                             |
          +----------------------+                 |
          | Collaboration Service |                 |
          | (Redis + WebSocket)   |                 |
          +----------+-----------+                 |
                     |                             |
                     v                             |
          +----------------------+                 |
          | Document Store        |                 |
          | (Cassandra)           |                 |
          +----------+-----------+                 |
                     |                             |
                     v                             |
          +----------------------+                 |
          | Search & Analytics    |                 |
          | (Elasticsearch)       |                 |
          +----------+-----------+                 |
                     |                             |
                     v                             |
          +----------------------+                 |
          | Action Item Tracker   |-----------------+
          +----------------------+
🧩
Components
Incident Detection
Triggers creation of post-mortem documents upon incident detection
Why thisIntegrates with existing observability tools (e.g., Datadog) to automate incident capture
Why notCustom monitoring system rejected due to complexity and duplication of existing mature tools
Post-Mortem Frontend
User interface for writing, editing, and reviewing post-mortems
Why thisReact-based SPA for responsive collaborative editing and template enforcement
Why notTraditional server-rendered pages rejected due to poor real-time collaboration support
Collaboration Service (Redis + WebSocket)
Manages real-time collaborative editing state and presence
Why thisRedis supports sub-millisecond TTL and pub/sub for ephemeral session state; WebSocket enables low-latency updates
Why notPolling-based collaboration rejected due to high latency and poor UX
Document Store (Cassandra)
Stores post-mortem documents with version history and metadata
Why thisCassandra provides linear write scalability and high availability for 58 writes/sec peak with multi-year retention
Why notPostgreSQL rejected due to complex manual sharding at scale and slower write throughput
Search & Analytics (Elasticsearch)
Indexes post-mortem content and metadata for fast search and trend analysis
Why thisElasticsearch supports full-text search and aggregations for incident trend dashboards
Why notRelational DB full-text search rejected due to poor scalability and query flexibility
Action Item Tracker
Tracks remediation tasks with assignment, deadlines, and status
Why thisCustom microservice with PostgreSQL for relational consistency and complex queries on action items
Why notNoSQL rejected due to need for strong consistency and complex relational queries
🔄
Data Flow
1
Incident Detection → Post-Mortem Frontend
Trigger new post-mortem creation with incident metadata
Automates initial documen

Practice

(1/5)
1. A distributed cache system must evenly distribute keys across nodes and gracefully handle node failures without massive key remapping. Which component or technique best addresses this requirement?
easy
A. Implementing consistent hashing with virtual nodes
B. Storing all keys in a single master node for centralized control
C. Using a simple modulo hash function on node count to assign keys
D. Using random key assignment to nodes without hashing

Solution

  1. Step 1: Understand key distribution needs

    Simple modulo hashing (Using a simple modulo hash function on node count to assign keys) causes massive remapping when nodes change, failing graceful scaling.

  2. Step 2: Evaluate consistent hashing benefits

    Consistent hashing with virtual nodes (Implementing consistent hashing with virtual nodes) minimizes remapping and balances load even with node failures.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Consistent hashing with virtual nodes -> minimal key remapping on node changes [check]
Hint: Consistent hashing + virtual nodes minimize remap on node changes [OK]
Common Mistakes:
2. If the database system loses the statistics metadata used by the query optimizer for index selectivity, what is the most likely failure mode during query execution?
medium
A. The database will refuse to execute queries that use indexes
B. Queries will return incorrect results due to wrong index usage
C. The database will always use the composite index regardless of query filters
D. The optimizer may choose a full table scan instead of an index scan, causing slower queries

Solution

  1. Step 1: Role of statistics metadata

    Statistics guide the optimizer to estimate index selectivity and cost, influencing index usage decisions.

  2. Step 2: Effect of lost statistics

    Without statistics, the optimizer cannot accurately estimate costs and may default to safer full table scans, degrading performance but not correctness.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Missing stats cause poor plan choice, not query failure [check]
Hint: No stats -> optimizer picks full scan, slower but correct [check]
Common Mistakes:
3. If the Signaling Server crashes after forwarding Alice's SDP offer but before receiving Bob's answer, what is the most likely failure observed by Alice?
medium
A. Alice's client receives an immediate error indicating call failure
B. Alice's UI shows ringing indefinitely; the call never connects and no retry occurs
C. The media server starts relaying media streams without session establishment
D. Bob receives the offer twice causing duplicate call setup attempts

Solution

  1. Step 1: Trace failure impact of Signaling Server crash after forwarding offer

    If the server crashes before recording the offer state, it may resend the offer when restarted, causing duplicates.

  2. Step 2: Analyze other options

    No immediate error (A) is unlikely; indefinite ringing (B) happens if answer lost but not duplicated; media server (D) does not start without signaling.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Signaling crash causes duplicate offers, confusing Bob's client [check]
Hint: Signaling crash -> duplicate offers, not silent failure [check]
Common Mistakes:
4. If the client fails to validate the server's certificate chain during the TLS handshake, what is the most likely failure observed in the connection?
medium
A. The client proceeds with encrypted communication but logs a warning
B. The client aborts the handshake and the connection is terminated without data exchange
C. The server retries sending the certificate chain automatically
D. The client falls back to unencrypted HTTP automatically

Solution

  1. Step 1: Understand client certificate validation failure

    If the client detects an invalid certificate chain, it aborts the handshake and terminates the connection to prevent insecure communication.

  2. Step 2: Analyze options

    The client proceeds with encrypted communication but logs a warning is incorrect because clients do not proceed with encryption if validation fails. The server retries sending the certificate chain automatically is incorrect; servers do not retry sending certificates automatically. The client falls back to unencrypted HTTP automatically is incorrect because clients do not downgrade to HTTP automatically.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Client aborts handshake on certificate validation failure [check]
Hint: Certificate validation failure -> handshake abort, no fallback [check]
Common Mistakes:
5. For a distributed key-value store requiring minimal key remapping and simple node addition/removal, which approach is best and why?
medium
A. Use consistent hashing with virtual nodes to balance load evenly
B. Use random node assignment with periodic rebalancing
C. Use Rendezvous Hashing to minimize remapping without ring maintenance
D. Use modulo hashing with a centralized directory for simplicity

Solution

  1. Step 1: Compare remapping and node management

    Modulo hashing (Use modulo hashing with a centralized directory for simplicity) causes massive remapping on node changes. Random assignment (Use random node assignment with periodic rebalancing) causes uneven load and complex rebalancing. Consistent hashing with virtual nodes (Use consistent hashing with virtual nodes to balance load evenly) balances load but requires ring maintenance.

  2. Step 2: Identify Rendezvous Hashing advantages

    Rendezvous Hashing minimizes remapping and avoids ring maintenance, simplifying node addition/removal.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Minimal remapping + no ring maintenance -> Rendezvous Hashing [check]
Hint: Minimal remap, no ring -> Rendezvous Hashing [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard