What it is
The Job Scheduler is a web application that runs recurring business-process jobs, compliance audits, notification digests, directory group monitoring, and gives staff a single place to see when each job last ran, what it found, when it will run next, and a button to trigger it manually. Every job can also be run as a dry run, producing the email it would have sent without taking any real action.
Why it mattered
Scheduled jobs had previously been scattered inside individual applications, each with its own scheduling mechanism, its own log format, and no shared way to answer the question staff actually asked: did that job run last night, and what did it do. Consolidating them into one application with a visible history was the point.
The reliability problem that defined the project
The most important thing I learned on this project came from a twenty-seven-day production outage. Scheduled jobs silently stopped running for nearly four weeks, and nothing about the hosting environment looked wrong while it was happening.
The mechanism turned out to be a chain of defaults, each individually reasonable. The web server recycles its worker process on a periodic timer by default, roughly every twenty-nine hours. The recycled worker failed to start. The server’s rapid-fail protection then stopped the application pool permanently rather than retrying. And crucially, the pool reported no crash loop and no obvious failure state, so the usual signals that something had broken were absent.
I fixed the root cause by disabling the periodic restart entirely, since an application whose purpose is to run continuously has no business being recycled on a timer. Days later I found a second, related gap: the setting that warms up the application on startup only applies to the pool’s first start, not to any later recycle. So a recycle triggered by a health check or a configuration change would leave the new process cold, and for an application with no regular user traffic, nothing would wake it up. That needed a separate setting enabled everywhere, and added to the setup script so future environments get it automatically.
The design rules that came out of it
Every job run leaves a record, even when it finds nothing. A notification job that found no results originally returned early without writing any history, so staff could not distinguish “ran and found nothing” from “did not run.” Now a zero-result run always writes a history entry, and a zero-result dry run always sends an all-clear email, since a test that produces no output defeats its own purpose.
Long-running infrastructure jobs do not belong here. Data pipeline jobs that run four or more hours stay on the database server’s own scheduling agent rather than inside this application, precisely because the outage above proved that a web-hosted scheduler is subject to the web server’s lifecycle. This boundary is documented explicitly so nobody adds a four-hour job to a scheduler that can be recycled underneath it.
When a third-party system will not let you write, notify instead of automating. One audit job was designed to deactivate accounts in an external training platform, and only during implementation did it become clear that the available credentials were read-only regardless of how the request was formed. Rather than escalating for write access, I redesigned the job to find the discrepancy, email the list to administrators, and let them act. Simpler, no external side effects, nothing to roll back.
The result
Five notification jobs run in production with consistent history, dry-run capability, and email delivery. Cron expressions are validated at startup rather than failing at first fire. Concurrent execution is disallowed across all jobs. Directory-backed staff lookup and per-group access control let different teams see only their own jobs.
Where it stands
The application runs in production across three environments. The remaining planned work is a documented but unbuilt proposal for automating runtime version patching across servers, deliberately written as a plan for review rather than implemented directly, since changing runtime versions on production servers carries real outage risk.