Bird
Raised Fist0

Design a Task Scheduler - Priority Queue, Cron, Distributed Execution

HLDhardInterview freq: medium
Choose your preparation mode4 modes available
ArchitectureTraceChallenge
▶
Request Trace
Step 1/5
Cron Service

Trigger task creation

Cron Service fires a scheduled event for a high-priority task

Protocol:Cron Schedule
BeforeNo new task scheduled at this time
AfterNew task creation event generated with priority metadata
Scheduler Service

Receive and enqueue task

Scheduler Service receives task event, assigns priority, produces to Kafka

Protocol:Kafka Produce
BeforeTask event received but not yet enqueued
AfterTask message durably stored in Kafka with priority key
Kafka Cluster

Store and order task messages

Kafka queues tasks by priority partition key for ordered consumption

Protocol:Kafka Internal
BeforeTask message just produced, waiting in Kafka
AfterTask message ready for consumption by workers in priority order
Execution Workers

Consume and execute task

Worker consumes highest priority task from Kafka and executes it

Protocol:Kafka Consume
BeforeTask message available in Kafka partition
AfterTask execution started, worker holds task context
Metadata DB (Cassandra)

Update task status

Worker updates task execution status and logs result in Cassandra

Protocol:CQL (Cassandra Query Language)
BeforeTask status is pending or running
AfterTask status updated to completed or failed with timestamp
🏗️
Architecture Diagram
Architecture Diagram
┌───────────────────┐
│    Cron Service   │
│  (triggers tasks) │
└─────────┬─────────┘
          │ Cron Schedule
          ▼
┌───────────────────┐
│ Scheduler Service │
│ (priority enqueue)│
└─────────┬─────────┘
          │ Kafka Produce
          ▼
┌───────────────────┐
│  Kafka Cluster    │
│ (priority queue)  │
└─────────┬─────────┘
          │ Kafka Consume
          ▼
┌───────────────────┐
│ Execution Workers │
│ (execute tasks)   │
└─────────┬─────────┘
          │ Metadata Update
          ▼
┌────────────────────────┐
│ Metadata DB (Cassandra)│
│ (task state, history)  │
└────────────────────────┘
Design Rationale

This architecture shows a distributed task scheduler where the Cron Service triggers scheduled tasks, sending events to the Scheduler Service. The Scheduler Service assigns priority and enqueues tasks into Kafka, which acts as a distributed priority queue ensuring ordered delivery. Execution Workers consume tasks from Kafka partitions respecting priority and execute them. After execution, workers update task metadata and status in Apache Cassandra, which stores durable task state and history. This design decouples scheduling, queuing, and execution, enabling scalable, fault-tolerant distributed task processing with priority and cron-based triggers.

Primary Focus

✓

Practice

(1/5)
1. In a system using Long Polling for server-to-client updates, at step 3 the client receives a response and immediately sends a new request. What is the client's state at step 4 before the server responds?
easy
A. The client is disconnected and will retry after a timeout due to connection closure.
B. The client has an open HTTP request waiting for the server response, and a new request is queued but not sent.
C. The client is waiting for the server response on the previous request, so no new request is sent yet.
D. The client has closed the previous connection and sent a new HTTP request, waiting for the next server response.

Solution

  1. Step 1: Trace Long Polling request lifecycle

    In Long Polling, the client sends a request and waits for the server to respond. Upon receiving the response, the client immediately sends a new request.

  2. Step 2: Determine client state at step 4

    At step 4, the previous connection is closed, and the client has an open new HTTP request waiting for the server's next response.

  3. Final Answer:

    Option D -> Option D

  4. Quick Check:

    Long Polling client sends new request immediately after response [check]
Hint: Long Polling: response -> new request sent immediately [check]
Common Mistakes:
2. For storing aggregated logs with fast query response and cost efficiency, which storage architecture is best when logs are mostly write-once and rarely updated?
medium
A. Use a distributed relational database with strong consistency
B. Use a NoSQL key-value store with frequent compactions
C. Use a columnar data warehouse optimized for append-only writes
D. Use a distributed file system with manual indexing

Solution

  1. Step 1: Analyze log storage requirements

    Logs are write-once, large volume, and require fast analytical queries.

  2. Step 2: Evaluate columnar data warehouse

    Columnar stores optimize for append-only writes and fast scans, reducing cost and improving query speed.

  3. Final Answer:

    Option C -> Option C

  4. Quick Check:

    Columnar warehouse -> optimized for append-only and analytics [check]
Hint: Columnar storage excels at append-only analytics [check]
Common Mistakes:
3. Without the DAU Estimator component, what exactly breaks in the system's capacity planning process?
medium
A. Bandwidth estimation fails because DAU directly affects total data transfer calculations
B. The system cannot estimate peak QPS, causing under-provisioning of servers
C. Storage estimation becomes inaccurate because DAU is a key multiplier for daily data volume
D. User authentication fails because DAU tracks active sessions

Solution

  1. Step 1: Identify DAU Estimator role

    DAU Estimator provides daily active user count critical for bandwidth calculations as bandwidth depends on total active users.

  2. Step 2: Eliminate other options

    The system cannot estimate peak QPS, causing under-provisioning of servers is incorrect because peak QPS depends on instantaneous requests, not just DAU; Storage estimation becomes inaccurate because DAU is a key multiplier for daily data volume is partially true but storage depends more on total objects, not DAU alone; User authentication fails because DAU tracks active sessions is unrelated to authentication.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Bandwidth depends directly on DAU for total data transfer [check]
Hint: Bandwidth depends directly on DAU count [check]
Common Mistakes:
4. In an asynchronous messaging system, what exactly breaks if the message queue component is removed?
medium
A. Senders block until recipients acknowledge, causing high latency
B. Messages are duplicated due to lack of deduplication in the queue
C. Messages are delivered immediately but with no ordering guarantees
D. The system silently drops messages without notifying senders

Solution

  1. Step 1: Understand queue's role in message deduplication and buffering

    Queues often provide deduplication and buffering to ensure exactly-once or at-least-once delivery.

  2. Step 2: Consequence of removing the queue

    Without the queue, messages may be duplicated due to lack of deduplication and buffering mechanisms.

  3. Final Answer:

    Option B -> Option B

  4. Quick Check:

    Removing queue causes message duplication due to missing deduplication [check]
Hint: No queue means no deduplication, causing duplicates [check]
Common Mistakes:
5. Without the transaction log (WAL) in a relational database, what exactly breaks during a crash recovery scenario?
medium
A. Committed transactions may be lost or partially applied, causing data inconsistency
B. The database cannot enforce foreign key constraints during recovery
C. The database will deadlock on concurrent transactions after restart
D. The query optimizer will choose inefficient plans after recovery

Solution

  1. Step 1: Role of transaction log (WAL)

    The WAL records all changes before commit to enable redo/undo during crash recovery.

  2. Step 2: Consequence of missing WAL

    Without WAL, committed transactions might not be replayed, causing partial or lost updates and inconsistency.

  3. Final Answer:

    Option A -> Option A

  4. Quick Check:

    Missing WAL -> lost committed transactions, inconsistent state [check]
Hint: WAL ensures durability; without it, committed data can vanish [check]
Common Mistakes:
Recommended

Ready to prove it?

Pass the test and this pattern moves unknown strong while earning +1 Grip toward your certificate.

10 questions~5 mineasy hard