What it is
This project evaluated seven locally-hosted language models against real work from our own environment: transformation queries, reporting measures, spreadsheet formulas, and query-language conversions. The goal was to determine, by measurement rather than assumption, which models were good enough to trust for which categories of task, and what had to stay with a stronger external model.
Why it mattered
Working with protected health information means the default answer to “can we send this to an external service” is no, unless there is a specific agreement and a specific reason. A locally-hosted model changes that calculation completely, since nothing leaves the building. But only if it is actually capable enough to be useful, and the only way to know that is to test it against real work rather than benchmarks.
There was also a hardware constraint worth stating plainly: the machine running these models has no GPU. Any conclusion drawn here reflects CPU-only inference, which shaped the results significantly.
What the measurement found
Active parameter count predicted speed almost perfectly. Every mixture-of-experts model outran every dense model, regardless of total size. On a machine without a GPU, what matters is how many parameters are active per token, not how large the model is overall. This single finding eliminated several candidates immediately.
Specialization mattered more than architecture. Two models from the same family, same size, same architecture, one general-purpose and one code-specialized, were run against the same twelve coding tasks. The specialized model produced usable code in ten of twelve. The general model managed one. That gap is far larger than any architectural difference I measured.
Two capability ceilings were real and not fixable by prompting. Both surviving models fail consistently on slowly-changing-dimension loads, because they correctly split the operation into two statements and then reference a temporary expression from the first statement inside the second, where its scope has already ended. The query simply does not parse. Both also fail on filter-context reporting measures, where the distinction between three similar-looking functions determines whether a user’s own filter selection is respected or silently discarded. One model invents function names outright; the other substitutes a function that clears the user’s selection, which is worse than failing, because it returns a plausible number that is wrong.
Simpler time-based measures, prior-year, year-to-date, month-to-date, percent-of-parent, are reliable on both.
The methodology mistake worth publishing
Two of my documented conclusions turned out to be false, both from trusting a single run.
I recorded that one model could not produce a working spreadsheet formula for a specific problem, and built a routing recommendation on it. Retested properly, it returned the correct answer six times out of six, byte-identical. And separately, I credited a different model with solving a hard filter-context problem after it succeeded once. Multi-sampled, it scored zero of three.
The rule that came out of this is now applied to every claim: nothing that drives a routing, deletion, or purchasing decision may rest on a single sample. Single-sample results are labeled indicative. Corrections get written as explicit corrections in the document rather than as quiet edits to the tidier version, so the record shows what was believed and why it changed.
The result
Two models survived, one code-specialized general workhorse and one reasoning model, chosen by measurement from seven candidates. Five were deleted for failing correctness or being too slow. The privacy claim was verified rather than assumed: all outbound network traffic was blocked for the model runtime and inference confirmed to still work correctly.
The strategic conclusion was that local models complement a stronger external model rather than replacing it. The stronger model authors the first correct version of something genuinely difficult; local models extend a verified pattern across many similar cases. A proposal to buy GPU hardware to fix an accuracy gap was explicitly examined and rejected, since routing had already solved it for free.
Where it stands
The two-model roster is running and in use. The evaluation documents, including the corrections, are kept as the record of how those choices were made, so the reasoning is auditable rather than a matter of recollection.