Bird
Raised Fist0

Design a Digital Wallet System - Double-Entry Ledger, Consistency, Reconciliation

HLDhardInterview freq: high
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Design a Digital Wallet System - Double-Entry Ledger, Consistency, Reconciliation
45 min · senior
senior

You are designing a digital wallet system for a fintech startup with 50 million daily active users (DAU). The system must support real-time balance updates, strict double-entry accounting to prevent inconsistencies, and automated reconciliation to detect and fix discrepancies. The system must handle up to 10 million transactions per minute with P99 latency under 200ms. Data must be encrypted at rest and in transit. Offline transaction delivery and eventual consistency for some operations are allowed but core balance updates must be strongly consistent.

Constraints
  • End-to-end encryption of transaction data at rest and in transit
  • P99 latency for balance queries and transaction commits under 200ms
  • Support offline transaction delivery with eventual reconciliation
  • Double-entry ledger model to ensure accounting correctness
  • Automated reconciliation process to detect and fix inconsistencies daily
📋
Guided Steps
1/5
Step 1
Data Model and Ledger Design
How would you design the data model to implement a double-entry ledger ensuring atomicity and consistency at scale?
💡 Hint
Consider immutable transaction records with separate debit and credit entries and a way to enforce atomicity.
Key Decision
Use Apache Cassandra with a composite primary key (transaction_id, entry_type) to store immutable ledger entries, combined with lightweight transactions (LWT) for atomic double-entry writes.
Interviewer Expectation
Mid: Names Cassandra for scalability. Senior: Explains composite key for debit/credit entries and why relational DBs with ACID don't scale to 10M TPS.
⚠️Common Wrong Answer
Using a relational database with two separate tables for debit and credit entries leads to bottlenecks and poor write throughput at 10M TPS.
Step 2
Consistency and Transaction Processing
How will you ensure strong consistency and atomicity for wallet balance updates during transactions?
💡 Hint
Think about distributed consensus or lightweight transactions to prevent double spends and maintain ledger correctness.
Key Decision
Leverage Cassandra's lightweight transactions (LWT) with Paxos protocol to atomically write paired debit and credit entries, ensuring linearizable consistency per transaction.
Interviewer Expectation
Mid: Mentions LWT for atomic writes. Senior: Explains Paxos overhead trade-offs and why multi-region consensus is avoided for latency.
⚠️Common Wrong Answer
Relying on eventual consistency or asynchronous writes causes balance inconsistencies and double spends under concurrent transactions.
Step 3
Reconciliation and Discrepancy Detection
How would you design the reconciliation process to detect and fix ledger inconsistencies daily?
💡 Hint
Consider batch processing frameworks and idempotent correction transactions.
Key Decision
Use Apache Spark batch jobs on Cassandra snapshots to compute daily ledger balances and detect mismatches, triggering idempotent correction transactions via Kafka topics.
Interviewer Expectation
Mid: Names Spark for batch processing. Senior: Explains snapshot isolation, idempotent corrections, and Kafka for reliable correction event delivery.
⚠️Common Wrong Answer
Manual reconciliation or real-time scanning causes high operational overhead and misses subtle inconsistencies.
Step 4
API Design and Security
What are the core API endpoints you would expose, and how do you ensure secure and low-latency access?
💡 Hint
Think about transaction submission, balance queries with caching, and reconciliation status endpoints with authentication.
Key Decision
Expose RESTful endpoints secured with OAuth 2.0 and TLS: POST /transactions for atomic debit/credit, GET /balance with Redis caching for P99 < 200ms, and GET /reconciliation-status for admins.
Interviewer Expectation
Mid: Names REST and OAuth. Senior: Explains Redis caching with TTL invalidation on writes and encryption at transit and rest.
⚠️Common Wrong Answer
Using unauthenticated endpoints or synchronous DB queries for balance causes latency spikes and security risks.
Step 5
Offline Transaction Handling and Delivery
How would you support offline transaction delivery while maintaining ledger correctness?
💡 Hint
Consider message queues and idempotent processing to handle retries and out-of-order delivery.
Key Decision
Use Apache Kafka as a durable transaction queue with exactly-once semantics and idempotent consumers to process offline transactions reliably once connectivity is restored.
Interviewer Expectation
Mid: Mentions Kafka for buffering. Senior: Explains exactly-once semantics and idempotency to prevent duplicate ledger entries.
⚠️Common Wrong Answer
Using simple retry without idempotency leads to duplicate transactions and ledger corruption.
🔌
API Design Challenge
Sketch 3 core API endpoints for the digital wallet system that cover transaction initiation, balance query, and reconciliation status.
Hints
  • How does the client initiate a debit or credit transaction securely?
  • How does the client query the current wallet balance with strong consistency?
  • How does the system expose reconciliation results or discrepancies to admins?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Double-entry ledger implemented with immutable paired entries in Cassandra
  • Strong consistency via Cassandra Lightweight Transactions (LWT) for atomic writes
  • Automated reconciliation using batch processing (Apache Spark) with idempotent corrections
  • API design covering transaction initiation, balance query with caching, and reconciliation status
Good to Have - 30%
  • Offline transaction buffering with Apache Kafka exactly-once semantics
  • Redis caching for low-latency balance queries with TTL invalidation
  • End-to-end encryption at rest and in transit
  • OAuth 2.0 authentication securing APIs
Bonus - 10%
  • Detailed failure mode analysis of Paxos overhead and reconciliation edge cases
  • Design for multi-region deployment trade-offs balancing latency and consistency
Anti-patterns to Avoid
Using relational databases for ledger storage causing write bottlenecks at 10M TPS, leading to system slowdowns and inability to scale
Relying on eventual consistency for balance updates leading to double spends and user disputes due to stale data
Manual or real-time reconciliation causing high operational overhead and missing subtle inconsistencies that accumulate over time
Unauthenticated or unencrypted APIs exposing sensitive financial data to attackers, risking data breaches and fraud
Simple retry mechanisms without idempotency causing duplicate ledger entries and corrupting financial records
Polling-based offline transaction delivery causing high load and latency spikes, overwhelming system resources at scale
✅
Model Answer
▼ Reveal

The system uses Apache Cassandra with a composite primary key to store immutable double-entry ledger entries, ensuring atomicity and scalability. Strong consistency is achieved via Cassandra Lightweight Transactions (LWT) using the Paxos protocol to atomically write paired debit and credit entries. Daily reconciliation is performed using Apache Spark batch jobs on Cassandra snapshots to detect discrepancies, triggering idempotent correction transactions delivered reliably via Kafka. Offline transactions are buffered in Kafka with exactly-once semantics and processed idempotently to maintain ledger correctness. The API layer exposes RESTful endpoints secured with OAuth 2.0 and TLS, including POST /transactions for atomic transaction submission, GET /balance with Redis caching for low-latency queries, and GET /reconciliation-status for admin monitoring. End-to-end encryption is enforced with TLS in transit and AES-256 at rest.

Key Decisions
  • Choosing Cassandra for its scalability and support for lightweight transactions to handle 10M TPS
  • Implementing double-entry ledger with immutable paired entries to ensure accounting correctness
  • Using Paxos-based LWT to guarantee linearizable consistency and prevent double spends
  • Employing Apache Spark batch jobs on Cassandra snapshots for efficient and reliable reconciliation
  • Leveraging Kafka's exactly-once semantics and idempotent consumers for offline transaction delivery
  • Caching balance queries in Redis with TTL invalidation to meet P99 latency requirements
  • Securing APIs with OAuth 2.0 and TLS to protect sensitive financial data
  • Applying end-to-end encryption with AES-256 at rest and TLS in transit for data security
  • Apache Cassandra with composite primary key for immutable double-entry ledger storage
  • Cassandra Lightweight Transactions (LWT) using Paxos for atomic paired debit/credit writes
  • Apache Spark batch jobs on Cassandra snapshots for daily reconciliation and discrepancy detection
  • Apache Kafka with exactly-once semantics for offline transaction buffering and reliable delivery
  • Redis caching layer with TTL invalidation to serve balance queries under 200ms P99 latency
  • RESTful API secured with OAuth 2.0 and TLS for transaction submission, balance query, and reconciliation status
  • End-to-end encryption using TLS in transit and AES-256 encryption at rest for all sensitive data
  • Idempotent transaction processing to prevent duplicates during retries and reconciliation corrections

Practice

(1/5)
1. A social media platform needs to handle 1 million vote updates per second with low latency and durability. Which component is best suited to buffer and reliably process these vote events before updating the main datastore?
easy
A. Use a batch processing system that aggregates votes every hour
B. Use a relational database with immediate synchronous writes for each vote
C. Use an in-memory cache like Redis without persistence to handle votes
D. Use a distributed log system like Kafka to buffer and order vote events

Solution

  1. Step 1: Identify the need for durability and ordering

    Relational DB synchronous writes (Use a relational database with immediate synchronous writes for each vote) cause high latency and don't scale to 1M writes/sec. Redis without persistence (Use an in-memory cache like Redis without persistence to handle votes) risks data loss on failure. Batch processing (Use a batch processing system that aggregates votes every hour) introduces unacceptable latency.

  2. Step 2: Recognize Kafka's role

    Kafka provides durable, ordered, high-throughput buffering of vote events, decoupling ingestion from processing.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Kafka buffers high-write streams with durability and ordering [check]
Hint: Kafka buffers high-write streams with durability and ordering [check]
Common Mistakes:
2. A web application experiences frequent database read spikes causing high latency. Which component best reduces this latency by serving repeated reads efficiently?
easy
A. Implement an in-memory cache layer like Redis to serve repeated reads
B. Use a relational database with read replicas to distribute load
C. Add more application servers to handle increased traffic
D. Use a message queue to buffer incoming read requests

Solution

  1. Step 1: Identify the latency cause

    High latency is due to repeated database reads causing load spikes, which read replicas help but still hit the database.

  2. Step 2: Recognize caching benefits

    An in-memory cache like Redis serves repeated reads from memory, drastically reducing latency and database load.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Cache serves repeated reads faster than replicas [check]
Hint: Cache repeated reads to avoid DB spikes [check]
Common Mistakes:
3. If the Dead Letter Queue component is removed from the system, what is the most likely failure scenario when a message processing fails repeatedly?
medium
A. The main queue blocks processing as failed messages are retried indefinitely, causing backpressure
B. The system switches to synchronous processing, increasing latency but ensuring delivery
C. Messages are automatically routed to a backup queue with no delay
D. Messages are silently dropped without any retry or alerting

Solution

  1. Step 1: Understand DLQ role in failure isolation

    Without DLQ, failed messages remain in the main queue causing retries that block subsequent messages.

  2. Step 2: Evaluate failure outcomes

    Silent drops (A) lose data; automatic routing (C) requires DLQ logic; synchronous fallback (D) is not automatic.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    No DLQ -> retries block main queue causing backpressure [check]
Hint: No DLQ means retries block main queue, causing backpressure [OK]
Common Mistakes:
4. For the AI Code Assistant's latency budget of 100ms per request, which architecture best balances freshness of code context and response time?
medium
A. Use a synchronous call to the code repository for every request to get the latest code
B. Use a batch offline indexing updated once a day and serve completions from cached embeddings
C. Use a purely generative model without retrieval to avoid retrieval latency
D. Use a hybrid RAG architecture with pre-indexed code embeddings updated periodically and a fast vector search

Solution

  1. Step 1: Evaluate synchronous repository calls

    Use a synchronous call to the code repository for every request to get the latest code causes high latency due to network and disk IO, violating 100ms budget.

  2. Step 2: Compare batch offline indexing

    Use a batch offline indexing updated once a day and serve completions from cached embeddings sacrifices freshness, unsuitable for frequently changing codebases.

  3. Step 3: Assess purely generative model

    Use a purely generative model without retrieval to avoid retrieval latency lacks grounding in latest code, reducing accuracy.

  4. Step 4: Identify hybrid RAG architecture

    Use a hybrid RAG architecture with pre-indexed code embeddings updated periodically and a fast vector search balances freshness (periodic updates) and latency (fast vector search), meeting constraints.

  5. Final Answer:

    Option D -> Option D

  6. Quick Check:

    Hybrid RAG balances freshness and latency [check]
Hint: Hybrid RAG updates embeddings periodically for freshness and speed [check]
Common Mistakes:
5. Without the metadata service that tracks access frequency and last access time, what failure occurs in the storage tiering system?
medium
A. Clients cannot read data because metadata is required for decryption keys
B. Data migration workers cannot decide when to move data, causing stale data to remain in expensive hot storage
C. Data blocks are duplicated across all tiers, increasing storage cost
D. Data is immediately archived without validation, causing data loss

Solution

  1. Step 1: Role of metadata service

    Metadata tracks access patterns to trigger tier migration safely without data loss.

  2. Step 2: Consequence of missing metadata

    Without metadata, migration workers may archive data prematurely without validation, causing permanent data loss.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Missing metadata -> unsafe archiving -> data loss [check]
Hint: Metadata missing -> unsafe archiving -> data loss [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard