Here is solutions for Design Job Scheduler - Follow-up questions.

Main topic:

• Job/Coordinate Service: assign worker for the job + Planner Service if needed.

• Watcher Service (Cronjob): push ready job to queue.

• Executor Service (Worker): consume signal from queue to do a job.

• Bucket sharding for job_id

• Windowing: watcher query [Now, Now + T]

• Checkpoint: recurence to continue the job.

• Idempotency: outbox table by job_id or task_id.

• State Machine: for job running.

1. Patterns for Design Job Scheduler

1. Split to 3 services: Coordinator, Watcher, Worker

2. Deduplicate job_id: kafka partition, database lock (status), distributed lock.

3. Job with N tasks: assign tasks per worker.

4. Checkpoint: how to check point of a task -> Recurring Job Handling

5. Handling time-zone: for the scheduler.

6. Alert jobs: wait too long to execute.

7. Windowing and Batching: Each watcher only queries its shards for due_at in [now, now+W], e.g., 1–2 minutes, sliding every few seconds.

8. Bucket Sharding: key (hourlyBucket, shardId) with attributes ownerId, leaseExpiry -> only query by hour is enough.

9. Scheduler-side idempotency: outbox table

10. Workder-side idempotency: If Execution status == ENQUEUED, update to RUNNING, set workerId, startedAt, attempt++ -> If the conditional fails (already RUNNING/DONE), drop the message.

11. Dynamic sweeper: for job in status SCHEDULING or ENQUEUED for too long

12. Job lag and alerts: count of due_at <= now but status < ENQUEUED per shard.

13. Time-zone in scheduler service: must be right with timestamp.

14. Planner Service: reads job definitions and pre-generates concrete future execution records for some time horizon, like the next 24 hours.