Job Scheduler

What it is

The Job Scheduler is a web application that runs recurring business-process jobs, compliance audits, notification digests, and directory group monitoring, and gives staff a single place to see when each job last ran, what it found, when it will run next, and a button to trigger it manually. Every job can also be run as a dry run, producing the email it would have sent without taking any real action.

Why it mattered

Scheduled jobs had previously lived inside individual applications, each with its own scheduling mechanism, log format, and no shared answer to the question staff actually asked: did that job run last night, and what did it find. Consolidating them into one application with visible history was the purpose.

The twenty-seven-day outage

The defining incident on this project is worth describing in full, because the failure mode is one that ordinary monitoring does not catch.

Scheduled jobs silently stopped running for twenty-seven days. Nothing about the hosting environment appeared wrong during that period. The application pool reported a started state. There was no crash loop, no restart cycle, no error page. Anyone checking would have concluded the application was healthy.

The mechanism was a chain of defaults, each individually defensible:

flowchart TD
    A[App pool running normally] --> B[Default periodic restart<br/>fires at ~29 hours]
    B --> C[Worker process recycles]
    C --> D{Recycled worker<br/>starts successfully?}
    D -->|No| E[Rapid-fail protection triggers]
    E --> F[Pool stopped permanently]
    F --> G[No automatic recovery<br/>No crash loop visible<br/>No alert]
    G --> H[27 days of jobs never run]
    D -->|Yes| A

The web server recycles worker processes on a periodic timer by default, roughly every twenty-nine hours. The recycled worker failed to start. Rapid-fail protection then stopped the pool permanently rather than retrying indefinitely, which is correct behavior for a crashing application and exactly wrong here, because it stopped without any signal that would draw attention.

The root fix was to disable periodic restart entirely. An application whose entire purpose is running continuously has no business being recycled on a timer, and the default exists for web applications that serve requests and benefit from periodic memory cleanup, which is a different workload.

Four days later I found the second half of the problem. The setting that warms up an application on startup, which I had assumed covered this, applies only to the pool’s first start. Any later recycle, whether from a health check failure, a configuration change, or a manual restart, leaves the new process cold until a real user request arrives. For an application with no regular interactive traffic, nothing arrives to wake it. A separate setting is needed to reissue the warm-up request after every recycle, and it went into the setup script so future environments get it automatically rather than depending on someone remembering.

This is the pattern I now watch for: a fix that addresses the incident you observed while leaving an adjacent path to the same outcome open.

The seven jobs, and the deliberate architecture choice

Seven jobs run here, with schedules driven from database rows rather than hardcoded, so changing a schedule is a data change rather than a deployment:

An architectural note that runs against the pattern used elsewhere: this is a single-project application, not the five-project structure standard across the other internal applications. That was deliberate. The five-project split earns its ceremony when there is real domain logic to isolate. This application is a job runner and a portal; it has no domain of its own, and imposing a layered structure on it would have produced four projects that exist only to pass values through. The choice was later written back into the shared template as an explicitly documented second option, so the next job runner does not have to re-derive it.

Three databases are involved and only one uses an object mapper, since the other two are read against for reporting rather than written to.

Design rules that came out of production experience

Every run leaves a record, even a run that finds nothing. A notification job that found no results originally returned early, before reaching the code that writes history. Staff therefore could not distinguish “ran, found nothing” from “did not run at all,” which is precisely the distinction that matters after an outage. The rule is now enforced centrally in a shared helper: a real run with zero results writes a skipped entry and sends no email; a dry run with zero results writes a test entry and sends an all-clear email, because a test that produces no output defeats its own purpose. Every job that exits early must still call the completion path with a count of zero rather than returning bare.

Long-running infrastructure jobs are explicitly out of scope. Data pipeline work running four or more hours stays on the database server’s own scheduling agent, not in this application. This boundary exists precisely because of the outage above: a web-hosted scheduler inherits the web server’s lifecycle, and a four-hour job plus an app pool recycle is a guaranteed failure. The scope boundary is documented so nobody adds one later with good intentions.

Universal behavior belongs in the shared helper, not copied per job. When a rule applies to every notification job, suppressing email when there is nothing to report, always writing history, adding a standard header, it goes into the shared completion helper once rather than into each job class. The reasoning is drift: five copies of a rule become five slightly different rules, and a new job added later inherits none of them. Job classes stay thin and hold only what is genuinely job-specific. The temptation to handle it in the job “just for now” is exactly how the five divergent copies happen.

One email per run, not one per recipient. Every notification sends a single message with all recipients in the address field, rather than looping and sending individually. Separate identical emails look unprofessional and create real confusion about whether colleagues received the same message. History still records one row per recipient, so the loop exists for logging only, never for sending.

Never restore the production database from another environment. Production holds live job triggers, active schedules, and run history that do not exist in test or development. Restoring over it would destroy live configuration. Schema changes reach production through idempotent scripts only, and this rule holds even when the environments appear to be in sync, because appearing to be in sync is exactly the condition under which someone reaches for a restore.

Notify rather than automate when write access does not exist. One audit job was designed to deactivate stale accounts in an external training platform. During implementation, repeated attempts to perform the update returned an authorization failure regardless of how completely the request was formed. Investigation showed the available credentials were view-only, and that the predecessor system had never used the interface for writes at all, it had uploaded a file over a separate transfer channel.

Rather than pursuing write access, I redesigned the job to find the discrepancy, build an email table, send it to administrators, and let them act. This is now a documented pattern for third-party systems generally: simpler, no external side effects, nothing to roll back, and the grace period on any given account becomes an administrator’s judgment rather than a hardcoded constant.

Bugs worth recording

Several of these are specific to the framework and would be difficult to predict from documentation alone.

• A comment between conditional branches silently broke the entire chain. In a help page template, section-separator comments were placed between the closing brace of one conditional block and the next branch condition. The template parser treats the comment as content emitted between two code statements, which terminates the chain. The result: the comment printed literally, the branch condition itself rendered as visible text on screen, and every subsequent branch rendered unconditionally, so all sections appeared at once. Valid syntax, no error, completely wrong output. The eventual fix replaced the chain with independent conditional blocks keyed on a single value.
• A shared service passed down the component tree arrived as null. The pattern was ported directly from an existing application where it worked. In this application, which uses a newer interactivity model with per-component boundaries, a non-serializable service does not cross that boundary. The parent layout could use it fine; pages received null and denied access to legitimate administrators. The fix was direct injection rather than cascading, which is now the documented rule for this application.
• Splitting a file into partial classes broke the build in a non-obvious way. Import statements do not transfer between partial files. Each new file needs its own imports for every type it references, including project namespaces and any file-scoped type aliases, which have to be repeated per file rather than inherited.
• A script run without specifying a database created its tables in the wrong one. A deployment script executed without an explicit database argument ran against the login’s default administrative database. The existence guard dutifully created the table there, and the next statement failed because the table it referenced does not exist in that database. Every script invocation now names its target database explicitly, never relying on the login default.
• Typographic punctuation in a SQL script corrupted seeded data. The script contained em dashes and multiplication signs in string literals. The command-line tool reads script files in an older character encoding than the file was saved in, so the punctuation was stored as garbled characters in seed rows on two servers and surfaced in the interface. The rule is plain characters only in SQL string literals; comments are harmless, data is not.
• Navigating between two routes on the same page left it stale. The scheduler page serves both a general route and a per-group route. The framework reuses the same component instance when moving between them, and the initialization method only runs on first load, so clicking a different group changed the parameter while the page continued showing the previous group’s jobs. The fix tracks the previously loaded value and reloads when it changes, with care taken not to double-load on the initial render.
• Headings do not control size in email. Heading tags render at the expected size in web mail clients and get overridden by one major desktop client, particularly in dark mode, so a heading appeared the same size as body text. Explicit inline font sizing on paragraph elements is the only reliable approach.
• Documentation drift shipped alongside working code. Several interface overhauls were committed and pushed promptly while the project’s own guidance document kept describing the previous version. The rule that came out of it: if a change alters behavior, the documentation update rides in the same commit, because that document is what the next session reads as truth.

The hardening pass

One session was spent entirely on consolidation rather than features, after a week of rapid development had left duplicated boilerplate across five notification jobs and two large page files.

The outcome: cron expressions are now validated before triggers register at startup, rather than failing at first fire; concurrent execution is disallowed across all jobs through a base class attribute; database queries had change-tracking disabled where writes are not needed and were batched to remove repeated-query patterns; timezone conversion was centralized into a single helper; shared constants and a common completion helper were extracted for all five jobs; and two oversized page files were split into smaller focused ones. Roughly two hundred twenty lines came out of the job files with no build warnings.

What the interface actually provides

The value of this application is almost entirely in what it makes visible, so the interface is the product rather than a wrapper around it.

• A tiles dashboard as the entry point, one tile per job group.
• A per-group scheduler page listing every job in that group with its last run outcome, next fire time, and a manual trigger button. Each job’s status shows as a coloured dot, and a legend is pinned to the panel explaining the four states: sent successfully, failed, skipped because nothing was due, and never run. That fourth state matters more than it sounds, since distinguishing “ran and found nothing” from “never ran” is exactly what was missing during the outage.
• A dry-run control on every job, producing the email the job would have sent without taking any real action.
• Run history per job, which is where the always-write-a-record rule pays off.
• An email settings panel with a checkbox list of active users who have an email address, replacing an earlier free-text recipient field. That change moved override configuration out of code and into the database, so it can be managed without a redeploy.
• An admin area with directory-backed staff lookup, so adding a user means searching the corporate directory rather than typing a login identifier and hoping.
• Per-group access control, so a team sees only its own jobs.

The in-app help was rebuilt around a section-per-URL structure so a help link from any page lands on the relevant section. That rebuild is where the conditional-chain bug described above surfaced, and the cleanup also stripped internal code identifiers out of user-facing text, which had leaked in from generating help content directly against the source.

Timeline and effort

Development ran across roughly 7 weeks, with 12 distinct working sessions recorded. The most concentrated build period came early, with the reliability work described above landing considerably later, after the outage had been discovered and diagnosed.

Where it stands now

The application runs in production across three environments. Five notification jobs execute with consistent history, dry-run capability, and email delivery. Directory-backed staff lookup and per-group access control let different teams see only their own jobs.

The remaining planned work is a documented proposal for automating runtime version patching across servers, covering detection, installation, validation, and promotion between environments. It is deliberately written as a plan for review rather than implemented, since changing runtime versions on production servers carries real outage risk and this project has already demonstrated what an unnoticed hosting change costs.