On-Premises AI Evaluation

What it is

This project evaluated seven locally-hosted language models against real work from our own environment: transformation queries, reporting measures, spreadsheet formulas, and query-language conversions. The goal was to determine by measurement which models could be trusted for which categories of task, and what had to stay with a stronger external model.

Why it mattered

Working with protected health information means the default answer to whether data can go to an external service is no, absent a specific agreement and a specific reason. A locally-hosted model changes that calculation entirely, since nothing leaves the building. But only if it is capable enough to be useful, and the only way to establish that is testing against real work rather than published benchmarks.

The hardware constraint matters to every conclusion below: the machine running these models has no GPU. This is CPU-only inference on a general-purpose server, which is the realistic condition for a nonprofit that is not buying accelerator hardware speculatively.

Architecture findings that eliminated candidates immediately

Active parameter count predicts speed almost perfectly. Every mixture-of-experts model outran every dense model tested, regardless of total parameter count. A mixture-of-experts model activates only a fraction of its parameters per token, and on a machine without a GPU, that fraction is what determines throughput. A thirty-billion-parameter model with roughly three billion active parameters was faster than dense models a third its nominal size.

This finding alone ruled out several candidates before correctness testing began, and it inverts the intuition that a smaller model will be faster.

Specialization matters more than architecture. Two models from the same family, same nominal size, same architecture, one general-purpose and one code-specialized, were run against the same twelve coding tasks. The specialized model produced usable code in ten of twelve. The general model managed one of twelve.

That is not a marginal difference attributable to prompt phrasing. It is the single largest effect I measured, larger than any architectural difference, and it argues strongly against reaching for a general model on the assumption that code is just another kind of text.

The two capability ceilings, and why prompting cannot fix them

Both surviving models fail consistently on two categories, and both failures are structural rather than a matter of phrasing.

Slowly-changing-dimension loads. Both models correctly decompose the problem into expiring existing rows and then inserting new ones, which is the right approach. Both then reference a temporary named expression from the first statement inside the second statement, where its scope has already ended. The query does not parse. The models understand the pattern and produce syntactically invalid output implementing it, which is a particularly awkward failure mode, because the reasoning looks correct on inspection.

Filter-context reporting measures. Three similar-looking functions differ in whether they respect, clear, or partially clear a user’s own filter selections. Getting this wrong does not produce an error; it produces a plausible number that is silently wrong. One model scored zero of four, inventing function names outright. The other scored roughly one of four, substituting a function that clears the user’s slicer selection, which is arguably worse than failing outright, because the output looks correct.

Simpler time-based measures, prior-year, year-to-date, month-to-date, percent-of-parent, are reliable on both models. The line between reliable and unreliable is sharper than a general capability score would suggest, which is the argument for task-specific evaluation over benchmark scores.

The methodology mistake, published rather than buried

Two of my documented conclusions turned out to be false, and both came from trusting a single run of a probabilistic system.

I recorded that one model could not produce a working spreadsheet formula for a dynamic-array problem, wrote it into the comparison document, and built a routing recommendation on it. Retested properly, the same model returned the correct answer six times out of six, byte-identical. The original failure was an unlucky draw. The routing advice built on it was unfounded.

Separately, I credited a different model with solving a hard filter-context problem after it succeeded once. Multi-sampled properly, it scored zero of three. Routing away from the weaker model was still justified; claiming this one had solved the problem was not.

The rule that came out of it: nothing driving a routing, deletion, or purchasing decision may rest on a single sample. Anything that matters gets multi-sampled through a confirmation script first. Single-sample results are labeled indicative rather than established. And when a correction is needed, it gets written as an explicit correction in the document rather than a quiet edit to the tidier claim, so the record shows what was believed, what changed it, and when.

Worth recording how the first error surfaced. Someone pushed back on my conclusion citing an external source, and that pushback was partly wrong on its own terms. Investigating it anyway is what exposed the real methodology flaw, which mattered considerably more than the point being argued. The lesson I take is that a challenge does not have to be correct to be worth investigating properly.

Verifying the privacy claim rather than assuming it

The entire justification for local hosting is that data never leaves the infrastructure. That claim was tested rather than assumed: all outbound network traffic was blocked for the model runtime, and inference was confirmed to still work correctly with no external connectivity at all.

This is a small piece of work with disproportionate value, because it is the claim a security reviewer will ask about, and the honest answer needs to be that it was verified rather than that the vendor documentation says so.

The routing conclusion

The strategic finding was that local models complement a stronger external model rather than replacing it, and the division of labor is fairly clean.

A stronger external model authors the first correct version of something genuinely difficult, a slowly-changing-dimension load, a tricky filter-context measure. Local models handle the repetitive work of extending a verified pattern across many similar cases, which is a large share of the actual volume in warehouse work and requires reliability rather than insight.

A proposal to purchase GPU hardware to close the accuracy gap was explicitly examined and rejected. Routing had already solved the accuracy problem at no cost, and buying hardware to fix a problem already solved by sending the hard cases elsewhere would have been spending money to avoid a decision rather than to gain a capability.

The models: every candidate, what it scored, and the verdict

The evaluation ran a twelve-case harness against each candidate, with a written grading reference so scoring was against a stated expected answer rather than a judgment call. Hardware ceiling for every row below: ten virtual CPUs, sixty-four gigabytes of memory, no GPU.

Two kept, six deleted.

Three rows in that table are worth reading closely, because each one overturns an intuition:

The fastest model was deleted. deepseek-coder-v2:16b was the throughput winner at 8.04 tokens per second, roughly thirty-six percent faster than the model I kept as default, and scored a respectable 9 of 12. It was deleted for producing incorrect reporting measures. Speed does not buy correctness back, and a fast wrong answer in a warehouse context is worse than a slow right one because it is more likely to ship.

A correction I had to publish against myself. An earlier version of my own scoreboard claimed every dense model was both slower and less accurate. phi4:14b disproves the second half outright: it produced correct reporting measures, correctly quoting both table and column references, and was terse with no narration. I deleted it purely for speed at 2.15 tokens per second. Architecture predicts throughput on this hardware; it does not predict correctness. I had conflated the two and had to write the correction into the document rather than quietly softening the original claim.

The clearest controlled comparison in the whole evaluation. qwen3:30b-a3b and the kept default are the same vendor, same mixture-of-experts architecture, same roughly thirty billion total and three billion active parameters, same eighteen gigabytes on disk. The only difference is that one is code-specialized and one is general purpose. The general variant produced a usable code block in 1 of 12 cases, narrating an answer in prose on the other eleven. The coder variant produced 10 of 12. Same size, same architecture, same vendor, a nine-case swing from specialization alone.

Active parameter count predicts speed almost exactly. Reading the tok/s column against active parameters: ~2.4B active gives 8.0, ~3B gives 5.9, ~3.6B gives 3.6, ~9B gives 2.9, and dense 14B models give 1.9 to 2.2. Inference on this box is memory-bandwidth-bound, so this single number predicts throughput better than anything else, including total model size.

That produced an explicit selection rule for this hardware: low active parameter count, roughly four billion or below; code or instruction tuned for terse structured output; and either non-thinking or with the thinking mode genuinely suppressible.

The routing table, and how confident each row actually is

Rather than picking one model, the system routes per task type. The part I consider most important is that every row carries an explicit evidence strength, so a reader can tell which decisions rest on confirmed multi-sampled results and which are essentially a coin flip resolved by speed.

Two rules govern it. A confirmed failure routes away from that model regardless of speed. Weak or absent evidence routes to the faster model.

That second rule is deliberate and is a direct consequence of the methodology error described below: a single-sample failure documented earlier in this project later failed to reproduce six times out of six. Weak evidence does not justify accepting a sixty percent slowdown.

The most honest row in the table is the first one. Routing away from the default for reporting measures is strongly supported, because that model scored zero of four. Claiming the alternative fixes the problem is not supported to the same degree, and the table says so rather than presenting the routing as a solved problem.

Where both models fail, measured case by case

Twelve harder cases weighted toward warehouse work, with any failure that drives a routing decision multi-sampled rather than taken from one run:

The row that decided the division of labor is the incremental load case. The reasoning model returned an empty response, while the coder model produced a working answer in thirty-eight seconds. That is the clearest evidence that neither model dominates: each fails on cases the other handles, and a single-model deployment would have inherited whichever set of failures came with the choice.

What we actually built to run this

The evaluation is not a spreadsheet of impressions. It runs through a small purpose-built harness, and describing it matters because the harness is what makes the results reproducible rather than anecdotal.

• A twelve-case run harness executes every case against a named model and captures output verbatim, alongside a written grading reference stating the expected answer for each case, so scoring is against a stated standard rather than a recollection of what good looks like.
• A separate hard-case harness covers the twelve harder warehouse cases with its own grading reference.
• A confirmation runner re-executes a single case against a single model N times, which is the tool that exists specifically because of the single-sample error. Any claim intended to drive a decision goes through this first.
• A variance runner samples repeatedly to surface how stable a model’s output actually is on a given case, which is how the six-of-six byte-identical result was established.
• A routing module implements the table above, with a default of automatic selection. The response states which model ran and why, so a user of the system can see the routing decision rather than having it hidden.
• A local integration server exposes generation to the assistant used for development, which is what makes the local models usable in day-to-day work rather than only in evaluation.

There is no graphical interface, and that was a deliberate scoping decision. This project’s users are the routing module and one administrator. Building a management interface would have consumed effort that produced no additional measurement, and the outputs that matter, the scoreboard, the hard-case analysis, the routing rationale, are documents rather than screens.

Documentation as the actual deliverable

For a project whose output is a decision rather than an application, the documents are the product, and they are structured accordingly:

• An infrastructure document covering the host setup and the runtime.
• An integration server document describing the interface the assistant uses.
• A model scoreboard with the full verdict table, kept-and-deleted reasoning, and the methodology corrections written inline rather than as edits to the original claims.
• A hard-case analysis covering the two capability ceilings in detail.
• A routing document with the table, the evidence strength per row, and an explicit caveat section on the weakest row.
• A recommendations document with the strategic conclusion, including the rejected hardware purchase.

The convention I hold to across all of them is that a correction gets written as a visible correction, showing what was believed, what changed it, and when. A document that quietly presents only the corrected conclusion reads as more authoritative and is worth less, because a reader cannot tell which claims have been stress-tested.

Timeline and effort

Work ran across roughly a single week and 5 distinct working sessions, covering the initial infrastructure setup, the evaluation runs, the hard-case analysis, the multi-sample corrections, and the routing recommendations. The short span is a fair reflection of the work: this was a focused measurement exercise rather than a build.

Technical reference

Surviving roster. Two models, chosen by measurement from seven candidates. A thirty-billion-parameter code-specialized mixture-of-experts model with roughly three billion active parameters, at about 5.9 tokens per second, serves as the default for transformation queries, reporting transformations, spreadsheet formulas, and text analysis. A twenty-billion-parameter reasoning-oriented mixture-of-experts model at about 3.6 tokens per second handles reporting measures, explanation, and pattern extension.

Deleted candidates. Five were removed for failing correctness on reporting measures or for being too slow to be practical: two dense models in the fourteen-billion range, a general-purpose model of the same size and architecture as the surviving coder, a sixteen-billion mixture-of-experts coder, and a smaller dense model.

Evaluation artifacts. A scoreboard document records per-model, per-task results with sample counts. A separate hard-case analysis document covers the two capability ceilings in detail. Both include the corrections rather than the corrected conclusions alone.

Where it stands now

The two-model roster is running and in use on the internal server. The evaluation documents are kept as the record of how those choices were made, including what was measured, what was wrong, and what changed, so the reasoning is auditable rather than a matter of recollection.

The GPU decision is documented as deliberately deferred rather than closed, with the condition that would reopen it stated explicitly: a demonstrated need that routing cannot address.