Job Scheduler

The Job Scheduler is a web application that runs recurring business-process jobs: compliance audits, notification digests, directory group monitoring. It gives staff a single place to see when each job last ran, what it found, and when it will run next, with a manual trigger and a dry-run mode that produces the email a job would have sent without taking any real action. Scheduled jobs had previously been scattered inside individual applications, each with its own scheduling mechanism and log format, and no shared way to answer the question staff asked: did that job run last night, and what did it do.

The project’s defining lesson came from production. Scheduled jobs silently stopped running for 27 days while nothing about the hosting environment looked wrong: the web server recycled its worker process on a default timer of roughly 29 hours, the recycled worker failed to start, rapid-fail protection stopped the application pool permanently, and no crash loop or error state ever appeared. Two more stoppages followed, one of 2.5 days and one of roughly 8 days after a site-wide power event, which proved the hosting layer’s startup warm-up never fires on these servers. The fix stopped trusting the hosting layer: a scheduled task pings the application every few minutes, a heartbeat writes from inside it, and a database-side watchdog alerts when the heartbeat goes stale or the active trigger count reaches zero, three independent detectors, each blind to the others’ failure modes.

Production also wrote the design rules, told in full on the detail page: every run leaves a record even when it finds nothing, because staff must be able to distinguish “ran and found nothing” from “did not run”; jobs that run four or more hours never live in a web-hosted scheduler; and when a third-party system grants only read access, the job notifies administrators instead of automating the change.

10 jobs run in production across three environments. The application has since been through a requirements-traceability review, 64 requirements across 10 areas, which produced 7 findings, each recorded with the file, the line, and a verification step, and 33 proposed tests now sit against 3 of them. The remaining planned work is a documented but unbuilt proposal for automating runtime version patching, deliberately written as a plan for review, since changing runtime versions on production servers carries real outage risk.

The technical detail

The Job Scheduler runs the organization’s recurring business-process jobs in one application, with visible history, dry runs, and per-group access, so nobody needs to wonder whether a job ran last night. It was hardened by three production outages that ordinary monitoring never saw.

The 27-day outage

The defining incident on this project is a failure mode ordinary monitoring does not catch.

Scheduled jobs silently stopped running for 27 days. Nothing about the hosting environment appeared wrong during that period. The application pool reported a started state. There was no crash loop, no restart cycle, no error page. Anyone checking would have concluded the application was healthy.

The mechanism was a chain of defaults, each individually defensible:

flowchart TD
    A[App pool running normally] --> B[Default periodic restart<br/>fires at ~29 hours]
    B --> C[Worker process recycles]
    C --> D{Recycled worker<br/>starts successfully?}
    D -->|No| E[Rapid-fail protection triggers]
    E --> F[Pool stopped permanently]
    F --> G[No automatic recovery<br/>No crash loop visible<br/>No alert]
    G --> H[27 days of jobs never run]
    D -->|Yes| A

The web server recycles worker processes on a periodic timer by default, roughly every 29 hours. The recycled worker failed to start. Rapid-fail protection then stopped the pool permanently instead of retrying indefinitely, which is correct behavior for a crashing application and wrong here, because it stopped without any signal that would draw attention.

The root fix was to disable periodic restart. An application whose entire purpose is running continuously has no business being recycled on a timer, and the default exists for web applications that serve requests and benefit from periodic memory cleanup, which is a different workload.

Four days later I found the second half of the problem. The setting that warms up an application on startup, which I had assumed covered this, applies only to the pool’s first start. Any later recycle, whether from a health check failure, a configuration change, or a manual restart, leaves the new process cold until a real user request arrives. For an application with no regular interactive traffic, nothing arrives to wake it. A separate setting is needed to reissue the warm-up request after every recycle, and it went into the setup script so future environments get it automatically, not depending on someone remembering.

This is the pattern I now watch for: a fix that addresses the incident you observed while leaving an adjacent path to the same outcome open.

The jobs, and the deliberate architecture choice

10 jobs now run here, with schedules driven from database rows, not hardcoded, so changing a schedule is a data change and not a deployment. The seven that defined the first phase:

The training audit later moved its data source to a table refreshed daily, which removed a class of false positives caused by timing mismatches between the two systems it compares.

An architectural note that runs against the pattern used elsewhere: this is a single-project application, not the five-project structure standard across the other internal applications. That was deliberate. The five-project split earns its ceremony when there is real domain logic to isolate. This application is a job runner and a portal; it has no domain of its own, and imposing a layered structure on it would have produced four projects that exist only to pass values through. The choice was later written back into the shared template as a documented second option, so the next job runner need not re-derive it.

Three databases are involved and only one uses an object mapper, since the other two are read against for reporting, never written to.

Design rules that came out of production experience

Every run leaves a record, even a run that finds nothing. A notification job that found no results originally returned early, before reaching the code that writes history. Staff therefore could not distinguish “ran, found nothing” from “did not run at all,” which is the distinction that matters after an outage. The rule is now enforced centrally in a shared helper: a real run with zero results writes a skipped entry and sends no email; a dry run with zero results writes a test entry and sends an all-clear email, because a test that produces no output defeats its own purpose. Every job that exits early must still call the completion path with a count of zero instead of returning bare.

Long-running infrastructure jobs are out of scope. Data pipeline work running four or more hours stays on the database server’s own scheduling agent, not in this application. This boundary exists because of the outage above: a web-hosted scheduler inherits the web server’s lifecycle, and a four-hour job plus an app pool recycle is a guaranteed failure. The scope boundary is documented so nobody adds one later with good intentions.

Universal behavior belongs in the shared helper, not copied per job. When a rule applies to every notification job, suppressing email when there is nothing to report, always writing history, adding a standard header, it goes into the shared completion helper once, not into each job class. The reasoning is drift: five copies of a rule become five slightly different rules, and a new job added later inherits none of them. Job classes stay thin and hold only what is job-specific. The temptation to handle it in the job “just for now” is how the five divergent copies happen.

One email per run, not one per recipient. Every notification sends a single message with all recipients in the address field, never looping and sending individually. Separate identical emails look unprofessional and create real confusion about whether colleagues received the same message. History still records one row per recipient, so the loop exists for logging only, never for sending.

Never restore the production database from another environment. Production holds live job triggers, active schedules, and run history that do not exist in test or development. Restoring over it would destroy live configuration. Schema changes reach production through idempotent scripts only, scripts safe to run twice, and this rule holds even when the environments appear to be in sync, because appearing to be in sync is the condition under which someone reaches for a restore.

Notify instead of automating when write access does not exist. One audit job was designed to deactivate stale accounts in an external training platform. During implementation, repeated attempts to perform the update returned an authorization failure regardless of how completely the request was formed. Investigation showed the available credentials were view-only, and that the predecessor system had never used the interface for writes at all, it had uploaded a file over a separate transfer channel.

Instead of pursuing write access, I redesigned the job to find the discrepancy, build an email table, send it to administrators, and let them act. This is now a documented pattern for third-party systems generally: simpler, no external side effects, nothing to roll back, and the grace period on any given account becomes an administrator’s judgment, not a hardcoded constant.

Bugs

Several of these are specific to the framework and would be difficult to predict from documentation alone.

• A comment between conditional branches silently broke the entire chain. In a help page template, section-separator comments were placed between the closing brace of one conditional block and the next branch condition. The template parser treats the comment as content emitted between two code statements, which terminates the chain. The result: the comment printed literally, the branch condition itself rendered as visible text on screen, and every subsequent branch rendered unconditionally, so all sections appeared at once. Valid syntax, no error, completely wrong output. The eventual fix replaced the chain with independent conditional blocks keyed on a single value.
• A shared service passed down the component tree arrived as null. The pattern was ported directly from an existing application where it worked. In this application, which uses a newer interactivity model with per-component boundaries, a non-serializable service does not cross that boundary. The parent layout could use it fine; pages received null and denied access to legitimate administrators. The fix was direct injection instead of cascading, which is now the documented rule for this application.
• Splitting a file into partial classes broke the build in a non-obvious way. Import statements do not transfer between partial files. Each new file needs its own imports for every type it references, including project namespaces and any file-scoped type aliases, which must be repeated per file, never inherited.
• A script run without specifying a database created its tables in the wrong one. A deployment script executed without an explicit database argument ran against the login’s default administrative database. The existence guard dutifully created the table there, and the next statement failed because the table it referenced does not exist in that database. Every script invocation now names its target database explicitly, never relying on the login default.
• Typographic punctuation in a SQL script corrupted seeded data. The script contained em dashes and multiplication signs in string literals. The command-line tool reads script files in an older character encoding than the file was saved in, so the punctuation was stored as garbled characters in seed rows on two servers and surfaced in the interface. The rule is plain characters only in SQL string literals; comments are harmless, data is not.
• Navigating between two routes on the same page left it stale. The scheduler page serves both a general route and a per-group route. The framework reuses the same component instance when moving between them, and the initialization method only runs on first load, so clicking a different group changed the parameter while the page continued showing the previous group’s jobs. The fix tracks the previously loaded value and reloads when it changes, with care taken not to double-load on the initial render.
• Headings do not control size in email. Heading tags render at the expected size in web mail clients and get overridden by one major desktop client, particularly in dark mode, so a heading appeared the same size as body text. Explicit inline font sizing on paragraph elements is the only reliable approach.
• Documentation drift shipped alongside working code. Several interface overhauls were committed and pushed promptly while the project’s own guidance document kept describing the previous version. The rule that came out of it: if a change alters behavior, the documentation update rides in the same commit, because that document is what the next session reads as truth.

The hardening pass

One session was spent on consolidation, not features, after a week of rapid development had left duplicated boilerplate across five notification jobs and two large page files.

The outcome: cron expressions are now validated before triggers register at startup instead of failing at first fire; concurrent execution is disallowed across all jobs through a base class attribute; database queries had change-tracking disabled where writes are not needed and were batched to remove repeated-query patterns; timezone conversion was centralized into a single helper; shared constants and a common completion helper were extracted for all five jobs; and two oversized page files were split into smaller focused ones. Roughly 220 lines came out of the job files with no build warnings.

The second and third outages

The 27-day outage had two successors: a second stoppage that lasted about 2.5 days, and a third that ran for roughly 8 days. The third is the one that changed the architecture, because its cause survived every fix made so far.

A site-wide power event shut the server down gracefully. The box rebooted, the application pool came up, and the scheduler returned as an empty shell: the process existed, the site answered requests, and zero triggers were active. My first diagnosis was wrong, and the corrected one is more uncomfortable. The hosting layer’s application-initialization feature, which is supposed to send a warm-up request on startup, never sends one on these servers at all. Every warm-up setting tuned so far had been tuning a feature that was not firing. The application had only ever started when a person happened to browse to it.

The fix stopped trusting the hosting layer. A scheduled task now pings the application every few minutes, so a cold process is warm within minutes of any restart. A heartbeat writes from inside the application, a watchdog on the database side raises an alert when that heartbeat goes stale or when active triggers drop to zero, and a narrower fallback can alert even if the main database is unreachable. Three detectors, none sharing a failure mode.

The watchdogs earned a correction of their own within days. A routine database server restart tripped a false error alert, because the underlying database library logs connection failures at error level before the application’s retry logic gets its chance to recover. The heartbeat writer was rewritten so a recoverable connection blip no longer reports as a failure.

An outbound-email kill switch shipped in the same period: suppression can be global, per job, or per feature. The two halves deliberately fail in opposite directions. Routine notification mail fails closed, so a suppressed job stays quiet. System alarms fail open, so no configuration state can silence the alert that says the scheduler itself is down.

The requirements review

Late in the project the application went through a requirements-traceability review covering 64 requirements in 10 areas, each traced to its implementing code, every finding pinned to a file and a line with a matching verification step. Seven findings came out, and the most consequential ones share a shape: a feature that promises a safety property the code does not deliver.

• A “Test Run” is not a dry run for 2 of the 10 jobs. Two audit jobs write and save their cache changes before the code checks the test flag, so a queued test run alters the audit baseline exactly as a real run would, while the interface promises no real actions will be taken.
• The grace-period deactivation for training-platform accounts never happens automatically. The only call to that platform anywhere in the code is read-only, so flagged accounts get re-emailed until a person deactivates them by hand. A one-off cleanup already sitting in the deployment scripts shows the drift this caused was real, not theoretical.
• The read-only role has no effect. The permission check that should enforce it is written correctly and called from nowhere that mutates data.
• The job detail page skips the visibility fence the main scheduler page enforces, so any authenticated user can read any job’s schedule and history directly by URL.
• A notification-type name that does not exactly match a lookup row makes the logging function return without writing anything and without an error, which leaves that job’s status chip and history permanently blank.
• Admin actions are audited unevenly. Configuration saves and suppression toggles are logged with the acting user’s name; user creation, role changes, and access grants are not logged at all.
• A job disabled for duplicating another application’s logic is still fully registered, trigger aside, so re-enabling it would recreate the exact double-processing incident that got it disabled.

A remediation backlog holds all 7, each carrying its file, its line, and a verification query. The review also drafted 33 tests covering 3 of the findings; the rest need a small refactor to become testable first, and the application had no automated tests in production before this. On the strength side, the review singled out the liveness architecture described above.

What the interface provides

The value of this application is almost entirely in what it makes visible, so the interface is the product rather than a wrapper around it.

• A tiles dashboard as the entry point, one tile per job group.
• A per-group scheduler page listing every job in that group with its last run outcome, next fire time, and a manual trigger button. Each job’s status shows as a colored dot, and a legend is pinned to the panel explaining the four states: sent successfully, failed, skipped because nothing was due, and never run. That fourth state matters more than it sounds, since distinguishing “ran and found nothing” from “never ran” is what was missing during the outage.
• A dry-run control on every job, producing the email a real run would send while changing nothing.
• Run history per job, which is where the always-write-a-record rule pays off.
• An email settings panel with a checkbox list of active users who have an email address, replacing an earlier free-text recipient field. That change moved override configuration out of code and into the database, so it can be managed without a redeploy.
• An admin area with directory-backed staff lookup, so adding a user means searching the corporate directory, not typing a login identifier and hoping.
• Per-group access control, so a team sees only its own jobs.

The in-app help was rebuilt around a section-per-URL structure so a help link from any page lands on the relevant section. That rebuild is where the conditional-chain bug described above surfaced, and the cleanup also stripped internal code identifiers out of user-facing text, which had leaked in from generating help content directly against the source.

Timeline and effort

Development spans roughly 12 weeks, with roughly 30 working sessions recorded. The most concentrated build period came early. The reliability work, the watchdogs, and the requirements review each landed later, after production had shown what they needed to cover.

Where it stands now

The application runs in production across three environments. 10 jobs execute with consistent history, dry-run capability, and email delivery. Directory-backed staff lookup and per-group access control let different teams see only their own jobs. The remediation backlog from the requirements review is open, with 33 drafted tests waiting on it. The project’s own task list is mirrored to a monday.com board through the sync built in the Shared AI Skills Library, with the repository’s backlog file staying authoritative and board status flipping automatically as work starts, publishes, and releases.

The remaining planned work is a documented proposal for automating runtime version patching across servers, covering detection, installation, validation, and promotion between environments. It is deliberately written as a plan for review rather than implemented: swapping runtime versions on a production server is real outage territory, and this project has already demonstrated what an unnoticed hosting change costs.