Bird
Raised Fist0

Design a Task Scheduler - Priority Queue, Cron, Distributed Execution

HLDhardInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
🎯
Design a Task Scheduler - Priority Queue, Cron, Distributed Execution
45 min · senior
senior

You are building a task scheduler for a 50M DAU SaaS platform that runs millions of scheduled and priority tasks daily. Tasks include user-triggered jobs, cron-based periodic jobs, and system maintenance tasks. The system must support distributed execution across multiple data centers, guarantee at-least-once execution semantics, and provide priority-based scheduling. Tasks can be delayed or retried on failure. The platform requires end-to-end encryption of task payloads at rest and in transit. The P99 latency for task dispatch must be under 200ms to meet SLAs.

Constraints
  • Support 10M scheduled tasks per day with peak QPS of 120K task dispatches/sec
  • End-to-end encryption of task payloads using AES-256 with key rotation
  • P99 dispatch latency under 200ms including queueing and execution trigger
  • Distributed execution across 3 geo-redundant data centers with failover
  • Support priority queues with at least 3 priority levels and fair scheduling
  • Cron expression parsing and scheduling with second-level granularity
📋
Guided Steps
1/5
Step 1
Choosing the Task Queueing System
Which technology will you use to implement the distributed priority queue for task scheduling and why?
💡 Hint
Consider a message broker that supports partitioning, ordering, and priority handling at scale.
Key Decision
Use Apache Kafka with topic partitioning keyed by task type and priority, leveraging Kafka's log ordering and consumer groups for distributed processing.
Interviewer Expectation
Mid: Names Kafka for queueing. Senior: Explains Kafka partition keys by priority and task type to ensure ordering and load balancing, and why RabbitMQ or Redis Streams fail at 120K QPS with multi-DC replication.
⚠️Common Wrong Answer
Using Redis Sorted Sets for priority queue - Redis single-threaded nature and lack of native multi-DC replication causes latency spikes and data loss at scale.
Step 2
Scheduling Cron and Delayed Tasks
How will you implement cron expression parsing and delayed task scheduling with second-level granularity?
💡 Hint
Think about a dedicated scheduler service that triggers tasks into the queue at the right time.
Key Decision
Use Quartz Scheduler for cron parsing and a distributed scheduler service that writes tasks into Kafka at scheduled times, ensuring idempotency and fault tolerance.
Interviewer Expectation
Mid: Mentions cron parsing library. Senior: Details Quartz's clustering mode for fault tolerance and how scheduler checkpoints state in Cassandra to avoid duplicate triggers.
⚠️Common Wrong Answer
Embedding cron logic in workers - leads to inconsistent scheduling and duplicate executions under failures.
Step 3
Ensuring Secure Task Payloads
How will you guarantee end-to-end encryption of task payloads at rest and in transit with key rotation?
💡 Hint
Consider envelope encryption and integration with a centralized key management system.
Key Decision
Encrypt task payloads client-side using AES-256-GCM with keys managed by HashiCorp Vault, rotating keys monthly and storing encrypted blobs in Kafka and Cassandra.
Interviewer Expectation
Mid: Mentions encryption at rest and TLS. Senior: Explains client-side encryption, envelope encryption pattern, Vault integration for key rotation, and how to handle key versioning during decryption.
⚠️Common Wrong Answer
Relying only on TLS - exposes payloads in Kafka logs and storage, violating compliance.
Step 4
Distributed Execution and Fault Tolerance
How will you design the distributed execution layer to guarantee at-least-once execution and handle worker failures?
💡 Hint
Think about consumer groups, task leasing, and checkpointing mechanisms.
Key Decision
Use Kafka consumer groups for task dispatch, with each worker leasing tasks via Zookeeper ephemeral nodes and checkpointing progress in Cassandra to enable retries on failure.
Interviewer Expectation
Mid: Describes Kafka consumer groups. Senior: Explains task leasing with ephemeral Zookeeper nodes to avoid duplicate execution, checkpointing offsets in Cassandra, and how to handle worker crashes gracefully.
⚠️Common Wrong Answer
Simple Kafka consumer offset commits without leasing - causes duplicate or lost executions during failures.
Step 5
API Design for Task Management
Design the APIs for task submission, status query, and cancellation. What parameters and response semantics will you include?
💡 Hint
Consider idempotency keys, pagination for status queries, and safe cancellation semantics.
Key Decision
REST APIs with JSON over HTTPS: POST /tasks with idempotency key, priority, schedule; GET /tasks/{id}/status with pagination for logs; DELETE /tasks/{id} supporting safe cancellation with state checks.
Interviewer Expectation
Mid: Lists basic endpoints. Senior: Details idempotency keys to avoid duplicate tasks, pagination for large status logs, and cancellation semantics that handle in-flight tasks and retries.
⚠️Common Wrong Answer
No idempotency or cancellation support - leads to duplicate executions and inconsistent states.
🔌
API Design Challenge
Sketch 3 core API endpoints for the task scheduler: task submission, task status query, and task cancellation.
Hints
  • How does the client submit tasks with priority and schedule info?
  • How can clients query task execution status and results efficiently?
  • What mechanism allows clients to cancel or reschedule tasks safely?
📊
Scoring Rubric
▼ Show
Must Have - 60%
  • Use of Kafka with partition keys for priority and task type to scale to 120K QPS
  • Quartz Scheduler or equivalent for cron parsing with fault tolerance
  • Client-side encryption with Vault-managed keys for payload security
  • Task leasing mechanism to avoid duplicate execution in distributed workers
  • APIs supporting idempotency, status querying, and cancellation
Good to Have - 30%
  • Checkpointing task state in Cassandra for recovery and retries
  • Multi-DC replication for geo-redundancy and failover
  • Monitoring with Prometheus/Grafana for SLA tracking
Bonus - 10%
  • Design considerations for key rotation and versioning in encryption
  • Handling of task retries with exponential backoff and dead-letter queues
Anti-patterns to Avoid
Using Redis Sorted Sets for priority queue - Redis single-threaded nature and lack of native multi-DC replication causes latency spikes and data loss at scale, making it unsuitable for 120K QPS with geo-redundancy requirements.
Relying only on TLS for encryption - exposes task payloads in Kafka logs and persistent storage, violating compliance and security requirements.
Embedding cron logic in workers - leads to inconsistent scheduling and duplicate executions under failures due to lack of centralized coordination.
Simple Kafka consumer offset commits without task leasing - causes duplicate or lost executions during worker failures as no mechanism prevents multiple workers from processing the same task.
No idempotency or cancellation support in APIs - leads to duplicate task submissions and inconsistent task states, complicating client-side error handling.
✅
Model Answer
▼ Reveal

This design uses Apache Kafka for distributed priority queueing with partition keys by task type and priority to ensure ordering and scalability. Quartz Scheduler in clustered mode handles cron parsing and scheduled task triggering with fault tolerance. Task payloads are encrypted client-side using AES-256-GCM with keys managed and rotated by HashiCorp Vault to meet security compliance. Kafka consumer groups combined with Zookeeper ephemeral nodes implement task leasing to guarantee at-least-once execution and avoid duplicates. Cassandra stores checkpointed task execution state and scheduler metadata enabling recovery and retries. RESTful APIs support idempotency keys for task submission, paginated status queries, and safe cancellation semantics. The system is deployed across multiple geo-redundant data centers with Kafka and Cassandra clusters replicated for high availability and disaster recovery. Monitoring and alerting are integrated using Prometheus and Grafana to track latency, failure rates, and SLA adherence.

Key Decisions
  • Use Apache Kafka with partition keys by task type and priority to achieve ordering and scale to 120K QPS
  • Implement cron parsing and scheduling using Quartz Scheduler in clustered mode with state checkpointing in Cassandra to avoid duplicate triggers
  • Encrypt task payloads client-side with AES-256-GCM using keys managed and rotated monthly by HashiCorp Vault, storing encrypted blobs in Kafka and Cassandra
  • Guarantee at-least-once execution by using Kafka consumer groups combined with Zookeeper ephemeral nodes for task leasing and checkpointing progress in Cassandra
  • Design REST APIs with idempotency keys for task submission to prevent duplicates, paginated status queries for large logs, and safe cancellation semantics that handle in-flight tasks and retries
  • Deploy Kafka and Cassandra clusters across 3 geo-redundant data centers with replication and failover to ensure high availability
  • Integrate monitoring and alerting with Prometheus and Grafana to track P99 latency, failure rates, and SLA adherence
  • Apache Kafka for distributed priority queueing with partition keys by task type and priority to ensure ordering and scalability
  • Quartz Scheduler in clustered mode for reliable cron expression parsing and scheduled task triggering with fault tolerance
  • Client-side AES-256-GCM encryption of task payloads with HashiCorp Vault for key management and rotation to meet security compliance
  • Kafka consumer groups combined with Zookeeper ephemeral nodes for task leasing to guarantee at-least-once execution and avoid duplicates
  • Cassandra as a durable store for checkpointing task execution state and scheduler metadata enabling recovery and retries
  • RESTful APIs with idempotency keys for task submission, paginated status queries, and safe cancellation semantics
  • Multi-DC deployment with geo-replication of Kafka and Cassandra clusters to ensure high availability and disaster recovery
  • Monitoring and alerting integrated with Prometheus and Grafana for latency, failure rates, and SLA adherence

Practice

(1/5)
1. In the failover incident trace, at Step 4 the Database Proxy receives a client query after the primary node crashed. What state does the proxy enter before forwarding the query?
easy
A. Marks the primary as unavailable and routes query to the new leader after election
B. Returns an error to the client without retrying
C. Queues the query indefinitely until the old primary recovers
D. Immediately forwards query to the old primary node, causing timeout

Solution

  1. Step 1: Trace Database Proxy behavior at Step 4

    Upon detecting primary failure, the proxy marks it unavailable and waits for leader election to complete before routing queries to the new leader.

  2. Step 2: Eliminate other options

    Forwarding immediately to old primary (A) causes timeouts; queuing indefinitely (C) stalls clients; returning errors without retry (B) reduces availability.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Proxy reroutes queries only after leader election [check]
Hint: Proxy reroutes only after leader election completes
Common Mistakes:
2. If the Write-Behind cache component crashes before flushing buffered writes to the database, what is the exact failure observed?
medium
A. The cache automatically retries writes on restart, ensuring no data loss
B. Writes buffered in cache are lost, causing data loss and inconsistency between cache and database
C. The cache returns stale data, but the database remains consistent with all writes
D. The database blocks writes until the cache recovers, causing system-wide latency spikes

Solution

  1. Step 1: Understand Write-Behind failure impact

    Buffered writes are not yet persisted; if cache crashes, writes are lost.

  2. Step 2: Analyze system behavior

    Database does not block writes; writes are lost, causing inconsistency. Automatic retries require explicit mechanisms.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Write-Behind crash causes lost buffered writes, no DB blocking [check]
Hint: Write-Behind crash loses buffered writes, no DB blocking [check]
Common Mistakes:
3. For a real-time multiplayer game backend requiring low latency and consistent player state, which architecture is more appropriate and why?
medium
A. Stateful servers storing player state only in a distributed cache without persistence
B. Stateless microservices with external database for player state
C. Stateless servers with client-side state management and server validation
D. Stateful game servers maintaining player state in memory with periodic persistence

Solution

  1. Step 1: Analyze latency and consistency needs

    Real-time games require fast access to player state with strong consistency.

  2. Step 2: Evaluate architecture trade-offs

    Stateful servers with in-memory state reduce latency and allow immediate state updates; periodic persistence balances durability.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Stateful in-memory state + persistence -> low latency, consistent [check]
Hint: In-memory stateful servers best for real-time consistency [check]
Common Mistakes:
4. At 10x transaction volume, which component of a relational database system is most likely to become the bottleneck first, and why?
hard
A. The storage engine, because disk I/O bandwidth saturates quickly
B. The query optimizer, because it cannot generate execution plans fast enough
C. The lock manager, due to increased contention causing transaction delays
D. The network layer, as increased client connections overwhelm bandwidth

Solution

  1. Step 1: Identify bottleneck under high concurrency

    Lock manager handles concurrency control; at 10x load, lock contention increases exponentially.

  2. Step 2: Compare other components

    Query optimizer scales well with caching; storage and network scale with hardware upgrades; lock manager contention is harder to mitigate.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Lock manager contention -> first bottleneck at high transaction scale [check]
Hint: Lock manager contention grows fastest at scale [check]
Common Mistakes:
5. At 10x scale, which component is most likely to fail first due to the exact lookup requirement, and why?
hard
A. The backend database, because it cannot handle the increased exact lookup queries without Bloom filter filtering
B. The network layer, because increased traffic from exact lookups saturates bandwidth
C. The Bloom filter component, because its false positive rate grows linearly with scale, causing excessive backend queries
D. The cache layer, because it cannot store all keys for exact lookups at 10x scale

Solution

  1. Step 1: Understand Bloom filter scaling behavior

    Bloom filter false positive rate increases with scale unless filter size grows proportionally, causing more backend queries.

  2. Step 2: Analyze component failure likelihood

    Backend (A) fails eventually but Bloom filter inefficiency causes overload first. Network (C) and cache (D) are bottlenecks but secondary to Bloom filter false positive explosion.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Bloom filter false positives grow with scale, triggering backend overload first [check]
Hint: Bloom filter false positives grow with scale, overload backend first [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard