Design a Distributed Task Scheduler (Quartz / Temporal)

A Distributed Task Scheduler schedules, coordinates, and executes millions of delayed one-time jobs (e.g., send order reminder in 24 hours) and recurring cron jobs (e.g., generate monthly billing invoice at midnight).

Functional Requirements

1. Delayed Execution (One-Time Tasks): Schedule a task to execute at an exact future timestamp: scheduleTask(run_at, payload).
2. Recurring Schedules (Cron): Support standard cron expressions (e.g., 0 0 * * *).
3. Execution Guarantees: Support At-Least-Once execution with configurable retry policies (exponential backoff).
4. Task Cancellation & Rescheduling: Cancel or modify pending tasks before they execute.

Non-Functional Requirements

• High Scalability: Schedule and dispatch tens of millions of active delayed tasks per day (~10,000 tasks/sec).
• High Timing Precision: Dispatch tasks within < 1 second of their scheduled execution time.
• High Availability & Fault Tolerance: If a scheduler master or worker node crashes, no task can be permanently lost or starved.