Bird
Raised Fist0

Design a Payment Gateway like Razorpay or Stripe - Architecture, Webhook, Retry

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Design a Payment Gateway like Razorpay or Stripe - Architecture, Webhook, Retry
45 min · senior
senior

You are tasked with designing a payment gateway system that processes payments for millions of users daily. The system must handle 50 million daily active users (DAU) with peak transaction rates reaching 100K TPS. It must securely process payments, provide reliable webhook delivery to merchant systems, and implement robust retry mechanisms for failed notifications. The system should maintain P99 latency under 200ms for payment authorization and ensure end-to-end encryption of sensitive data.

Constraints
  • End-to-end encryption of payment data in transit and at rest is mandatory to comply with PCI-DSS standards.
  • System must guarantee at-least-once delivery semantics for webhooks with idempotency to avoid duplicate processing.
  • P99 latency for payment authorization API must be under 200ms to ensure good user experience.
  • Retry mechanism for webhook delivery must handle merchant downtime gracefully without overwhelming their endpoints.
  • The system must scale horizontally to handle 50M DAU and 100K TPS peak load.
  • Audit logs and transaction history must be immutable and queryable for compliance and dispute resolution.
📋
Guided Steps
1/5
Step 1
Design the core payment processing architecture
How would you architect the payment processing flow to handle 100K TPS with P99 latency under 200ms while ensuring data security?
💡 Hint
Consider asynchronous processing, encryption, and scalable data stores optimized for write-heavy workloads.
Key Decision
Use a combination of Apache Kafka for event ingestion with payment_id as partition key to ensure ordering, Apache Cassandra for storing transaction state due to its linear write scalability and tunable consistency, and TLS 1.3 plus field-level encryption (e.g., AWS KMS or HashiCorp Vault) for data security.
Interviewer Expectation
Mid: Names Kafka and Cassandra for scalability. Senior: Explains Kafka partitioning by payment_id to maintain order and Cassandra's LSM-tree design for high write throughput with tunable consistency for strong read-after-write guarantees.
Step 2
Design webhook delivery and retry mechanism
How would you design the webhook system to guarantee at-least-once delivery with idempotency and handle merchant endpoint failures gracefully?
💡 Hint
Think about durable queues, retry backoff strategies, and idempotency keys to avoid duplicate processing.
Key Decision
Use Kafka topics dedicated to webhook events with payment_id as partition key for ordering, combined with a Redis-based deduplication cache for idempotency, and implement exponential backoff retry with dead-letter queues for failed deliveries.
Interviewer Expectation
Mid: Mentions Kafka and retry queues. Senior: Details how Kafka partitioning ensures ordered retries per payment, Redis TTL-based deduplication prevents duplicate webhook processing, and exponential backoff avoids overwhelming merchant endpoints.
Step 3
Design API endpoints for payment initiation, webhook reception, and acknowledgment
What are the core API endpoints you would expose, and what are their request/response contracts?
💡 Hint
Focus on payment initiation with immediate response, webhook event payload structure, and acknowledgment to stop retries.
Key Decision
Expose POST /payments/initiate with encrypted payload returning payment_id and status; POST /webhooks/events for merchant webhook reception with event_id and signature verification; POST /webhooks/acknowledge with event_id to confirm receipt and stop retries.
Interviewer Expectation
Mid: Lists endpoints with basic request/response. Senior: Explains encryption on payment initiation, webhook signature verification for security, and idempotent acknowledgment to prevent duplicate retries.
Step 4
Design audit logging and compliance data storage
How would you store immutable audit logs and transaction history to support compliance and dispute resolution?
💡 Hint
Consider append-only storage with efficient query capabilities and tamper-evident properties.
Key Decision
Use Apache Kafka with log compaction enabled for immutable event logs, backed by Amazon S3 for long-term durable storage, and Apache Druid for low-latency analytical queries on transaction history.
Interviewer Expectation
Mid: Mentions Kafka for logs. Senior: Explains Kafka log compaction for immutable storage, S3 for durability and compliance, and Druid for real-time analytics and fast querying.
Step 5
Ensure end-to-end encryption and PCI-DSS compliance
What encryption and security mechanisms would you implement to protect sensitive payment data in transit and at rest?
💡 Hint
Think about TLS, field-level encryption, key management, and compliance standards.
Key Decision
Enforce TLS 1.3 for all client-server and inter-service communication, use field-level encryption with AWS KMS for sensitive fields like card data, implement hardware security modules (HSMs) for key storage, and conduct regular PCI-DSS audits.
Interviewer Expectation
Mid: Mentions TLS and encryption. Senior: Details field-level encryption, key rotation policies, HSM usage, and compliance audit integration.
🔌
API Design Challenge
Sketch 3 core API endpoints for the payment gateway system that cover payment initiation, webhook event reception, and webhook acknowledgment.
Hints
  • How does the client initiate a payment and receive immediate confirmation?
  • How does the system notify merchants asynchronously about payment status changes?
  • How does the merchant acknowledge receipt of webhook events to prevent retries?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use Kafka with payment_id partition key for ordered, scalable event processing.
  • Implement at-least-once webhook delivery with idempotency using Redis and retry queues.
  • Design API endpoints with encrypted payloads and signature verification.
Good to Have - 30%
    Bonus - 10%
      Anti-patterns to Avoid
      ✅
      Model Answer
      ▼ Reveal

      This design outlines a scalable, secure payment gateway capable of handling 100K TPS with P99 latency under 200ms. It uses Apache Kafka for ordered event ingestion partitioned by payment_id, Apache Cassandra for scalable transaction state storage, and Redis for webhook idempotency caching. Webhook delivery employs exponential backoff retries with dead-letter queues to handle merchant downtime gracefully. API endpoints are secured with encrypted payloads and signature verification. Immutable audit logs are stored using Kafka log compaction backed by Amazon S3, with Apache Druid enabling fast analytics. TLS 1.3 and AWS KMS with HSMs ensure end-to-end encryption and PCI-DSS compliance.

      Key Decisions
      • Partition Kafka topics by payment_id to maintain event ordering and scalability
      • Use Cassandra's LSM-tree storage for high write throughput and tunable consistency for strong read-after-write guarantees
      • Implement Redis with TTL-based deduplication cache to ensure webhook idempotency
      • Apply exponential backoff retry with dead-letter queues to prevent merchant endpoint overload and ensure eventual delivery
      • Design API endpoints with encrypted payloads and webhook signature verification to secure communication
      • Store audit logs immutably using Kafka log compaction backed by durable Amazon S3 storage
      • Use Apache Druid for real-time analytics on transaction history to support compliance and dispute resolution
      • Enforce TLS 1.3 for all communications and use AWS KMS with HSMs for field-level encryption and key management to comply with PCI-DSS
      • Kafka with payment_id as partition key ensures ordered, scalable event ingestion and processing.
      • Apache Cassandra chosen for transaction state storage due to linear write scalability and tunable consistency.
      • Redis used for webhook idempotency cache with TTL to prevent duplicate event processing.
      • Exponential backoff retry with dead-letter queues prevents merchant endpoint overload and ensures eventual delivery.
      • API endpoints designed with encrypted payloads and signature verification to secure communication.
      • Apache Kafka log compaction combined with Amazon S3 provides immutable, durable audit logs for compliance.
      • Apache Druid enables low-latency analytics on transaction history for dispute resolution.
      • TLS 1.3 enforced for all communication and AWS KMS with HSMs used for field-level encryption and key management.

      Practice

      (1/5)
      1. In the booking flow, at step 4 the Availability Service checks the calendar cache before confirming a booking. What happens if the cache entry for the requested dates is stale or missing at this step?
      easy
      A. The service immediately rejects the booking request without fallback
      B. The service queries the persistent calendar database synchronously to verify availability
      C. The service assumes availability and proceeds to reserve the booking optimistically
      D. The service delays the booking confirmation until a cache refresh completes asynchronously

      Solution

      1. Step 1: Trace Availability Service behavior at step 4

        The service uses cache for fast reads but must ensure correctness before booking confirmation.

      2. Step 2: Handle stale or missing cache entries

        Rejecting immediately (A) reduces UX; optimistic booking (C) risks double bookings; async delay (D) hurts latency. Synchronous DB query (B) ensures correctness.

      3. Final Answer:

        Option B -> Option B

      4. Quick Check:

        Fallback to DB ensures accurate availability check on cache miss [check]
      Hint: Cache miss triggers synchronous DB check for correctness [check]
      Common Mistakes:
      2. For a system requiring low-latency approximate frequency queries on a data stream with heavy skew, which approach is better and why?
      medium
      A. Use Count-Min Sketch with conservative update to reduce overestimation bias
      B. Use a single large hash table with exact counts to avoid approximation errors
      C. Use a Bloom filter combined with exact counters for heavy hitters
      D. Use a HyperLogLog sketch to estimate frequencies with lower memory

      Solution

      1. Step 1: Evaluate exact counts vs approximation

        Exact counts require large memory and high latency for heavy skew data streams.

      2. Step 2: Recognize Count-Min Sketch with conservative update

        Conservative update reduces overestimation bias, improving accuracy for skewed data.

      3. Step 3: Eliminate Bloom filter and HyperLogLog

        Bloom filters don't count frequencies; HyperLogLog estimates cardinality, not frequency.

      4. Final Answer:

        Option A -> Option A

      5. Quick Check:

        Conservative update Count-Min Sketch balances accuracy and memory for skewed streams [check]
      Hint: Conservative update reduces bias in Count-Min Sketch for skewed data [check]
      Common Mistakes:
      3. For candidate-job matching, which architecture better supports real-time personalized recommendations with low latency and frequent profile updates?
      medium
      A. Precomputed static candidate-job similarity scores stored in a relational database
      B. Batch processing with nightly offline model training and periodic reindexing
      C. Online feature store with streaming updates feeding a real-time scoring service
      D. Client-side filtering of job listings based on cached candidate preferences

      Solution

      1. Step 1: Analyze real-time personalization needs

        Low latency and frequent profile updates require online, streaming data processing.

      2. Step 2: Evaluate architectures

        Batch (A) is too stale; static precomputed scores (C) can't adapt quickly; client-side filtering (D) lacks server-side intelligence. Online feature store with streaming (B) supports fresh data and low latency scoring.

      3. Final Answer:

        Option C -> Option C

      4. Quick Check:

        Streaming updates + real-time scoring -> fresh personalized matches [check]
      Hint: Streaming + online features enable fresh real-time recommendations [check]
      Common Mistakes:
      4. If the distributed priority queue component fails and becomes unavailable, what is the immediate failure mode observed in the task scheduler?
      medium
      A. Worker nodes block waiting for tasks, causing execution delays and potential timeouts
      B. Tasks are silently dropped without retries, causing data loss
      C. Tasks continue executing but with incorrect priority ordering
      D. The cron scheduler automatically reschedules tasks on all nodes

      Solution

      1. Step 1: Understand role of distributed priority queue

        The queue is the source of truth for pending tasks; if unavailable, new tasks cannot be fetched.

      2. Step 2: Analyze failure impact

        Without the queue, tasks submitted may be lost if not persisted elsewhere; workers cannot fetch tasks, so blocking (B) is unlikely if queue is down; cron scheduler (D) does not reschedule tasks automatically.

      3. Step 3: Evaluate silent drops (C)

        Without queue durability, tasks may be lost silently if producers cannot enqueue or retries are not implemented.

      4. Final Answer:

        Option B -> Option B

      5. Quick Check:

        Queue failure causes silent task loss without retries [check]
      Hint: Queue failure causes silent task loss without retries [check]
      Common Mistakes:
      5. Without Kafka in the architecture, what exactly breaks in the message delivery flow when Alice sends a message to Bob?
      medium
      A. Delivery receipts are sent out of order causing UI inconsistencies
      B. Alice's message is encrypted incorrectly causing decryption failures on Bob's side
      C. Alice's send blocks until Bob's server responds; if Bob's server crashes, message is lost with no retry
      D. Messages are duplicated because of lack of idempotent writes

      Solution

      1. Step 1: Understand Kafka's role in decoupling send and delivery

        Kafka buffers messages asynchronously; without it, Chat Server must synchronously contact Bob's server.

      2. Step 2: Trace failure without Kafka

        If Bob's server is slow or crashes, Alice's send blocks or message is lost with no retry, breaking reliability.

      3. Final Answer:

        Option C -> Option C

      4. Quick Check:

        No Kafka -> synchronous send blocks, message loss on Bob's crash [check]
      Hint: No Kafka -> synchronous send blocks, no retry, message lost [check]
      Common Mistakes:
      Recommended

      Ready to prove it?

      Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

      10 questions~5 mineasy hard