On-Premises AI Evaluation

This project evaluated 16 locally hosted language models against real work from our own environment, in three phases with three different definitions of what good meant: code generation first, then reading comprehension after the first standard proved to be the wrong one, then retrieval for a document-search capability. Working with protected health information means the default answer to “can we send this to an external service” is no. A locally hosted model changes that calculation, but only if it is capable enough to be useful, and the only way to know is to test it against real work. One ground rule shapes every result: the models run on a server we already had, with nothing bought for them, so every number here is what available hardware delivers.

The first phase found that active parameter count predicted speed almost perfectly, so every mixture-of-experts model outran every dense model regardless of total size. Specialization mattered more than architecture: two models from the same family, same size, same architecture, and the code-specialized one produced usable code in 10 of 12 tasks where the general one managed one. Two capability ceilings were real on both surviving models and not fixable by prompting.

A later policy decision let local models read identifier-stripped content and describe patterns in it, so the stronger model could build code from the description, never from the data. That exposed the evaluation itself: roughly 20 of the 24 cases were code generation, but under this division of labor local models never ship code, so I had been retaining and deleting models on the one thing they were never going to be asked to do. The replacement rubric tests reading and describing and adds a hard gate: any specific identifier appearing in a model’s output disqualifies it regardless of every other score. The gate’s worst finding was the model I had been treating as the default, which leaked on 12 of 12 samples and did not respond to prompt strengthening at all, while the same strengthened prompt took another model from 3 failures in 6 to 0. Model choice still matters after you have written a better prompt.

Three of my documented conclusions turned out to be false, all from trusting a single run, and the third is the most uncomfortable because it corrected a correction. The rule that came out: nothing driving a routing, deletion, or purchasing decision may rest on a single sample, and corrections get written as visible corrections, not quiet edits to the tidier claim.

Two models survived from 16 candidates. A 70-billion-parameter model was tested to settle the scale question and ran 13 times slower while being measurably worse at describing patterns. The privacy claim was verified by blocking all outbound traffic for the model runtime and confirming inference still worked. A GPU purchase was examined and rejected, since routing had already solved the accuracy gap for free. The two-model roster is running with the strengthened prompt adopted permanently, the embedding model for document search is chosen but not yet wired in, and one question stays open: whether a rule-based scrubber should strip identifier-shaped values from local-model output as a mandatory second layer, since prompting alone did not achieve zero leaks on either of the first two models tested.

The technical detail

This project measured 16 locally hosted language models against real warehouse work, across three phases, and each phase carried its own definition of what “good” meant, to establish by measurement which models could be trusted for which tasks and what had to stay with a stronger external model. Every conclusion below reflects CPU-only inference on a general-purpose server with no GPU, which is the realistic condition for a nonprofit that is not buying accelerator hardware speculatively.

Architecture findings that eliminated candidates immediately

Active parameter count predicts speed almost perfectly. Every mixture-of-experts model outran every dense model tested, regardless of total parameter count. A mixture-of-experts model activates only a fraction of its parameters per token, and on a machine without a GPU, that fraction is what determines throughput. A 30-billion-parameter model with roughly three billion active parameters was faster than dense models a third its nominal size.

This finding alone ruled out several candidates before correctness testing began, and it inverts the intuition that a smaller model will be faster.

Specialization matters more than architecture. Two models from the same family, same nominal size, same architecture, one general-purpose and one code-specialized, were run against the same 12 coding tasks. The specialized model produced usable code in 10 of 12. The general model managed 1 of 12.

That is the single largest effect I measured, larger than any architectural difference, and it argues against reaching for a general model on the assumption that code is just another kind of text.

The two capability ceilings, and why prompting cannot fix them

Both surviving models fail consistently on two categories, and both failures are structural, not a matter of phrasing.

Slowly-changing-dimension loads. Both models correctly decompose the problem into expiring existing rows and then inserting new ones, which is the right approach. Both then reference a temporary named expression from the first statement inside the second statement, where its scope has already ended. The query does not parse. The models understand the pattern and produce syntactically invalid output implementing it, which is a particularly awkward failure mode, because the reasoning looks correct on inspection.

Neither noticed a further trap either. A rewrite using a temporary table must save the changed set aside before the update runs, because once rows are expired they no longer match the filter that identified them.

Filter-context reporting measures. Three similar-looking functions differ in whether they respect, clear, or partially clear a user’s own filter selections. Getting this wrong does not produce an error; it produces a plausible number that is silently wrong. One model scored 0 of 4, inventing function names outright. The other scored roughly 1 of 4, substituting a function that clears the user’s slicer selection, which is arguably worse than failing outright, because the output looks correct.

Simpler time-based measures, prior-year, year-to-date, month-to-date, and percent-of-parent, are reliable on both models. The line between reliable and unreliable is sharper than a general capability score would suggest, which is the argument for task-specific evaluation over benchmark scores.

The models: every candidate, what it scored, and the verdict

The first phase ran a 12-case harness against each candidate, with a written grading reference so scoring was against a stated expected answer and not a judgment call. Hardware ceiling for every row below: 10 virtual CPUs, 64 gigabytes of memory, no GPU.

Three rows in that table each overturn an intuition.

The fastest model was deleted. deepseek-coder-v2:16b was the throughput winner at 8.04 tokens per second, roughly 36 percent faster than the model I kept as default and scored a respectable 9 of 12. It was deleted for producing incorrect reporting measures. Speed does not buy correctness back, and a fast wrong answer in a warehouse context is worse than a slow right one because it is more likely to ship.

The clearest controlled comparison in the whole evaluation. qwen3:30b-a3b and the kept default are the same vendor, same mixture-of-experts architecture, same roughly 30 billion total and three billion active parameters, same 18 gigabytes on disk. The only difference is that one is code-specialized and one is general purpose. The general variant produced a usable code block in 1 of 12 cases, narrating an answer in prose on the other 11. The coder variant produced 10 of 12. Same size, same architecture, same vendor, and a nine-case swing from specialization alone.

A model that matched the default exactly and was still deleted. qwen2.5-coder:32b scored 10 of 12, the same as the default, including two clean wins the default does not have. It runs at 1.2 tokens per second, roughly five times slower. Same correctness, much slower, so it does not justify displacing the default. It was deleted to free disk for further testing, which is the correct call and an uncomfortable one, because the temptation with a model that ties on quality is to keep it for insurance.

Active parameter count predicts speed almost exactly. Reading the tok/s column against active parameters: ~2.4B active gives 8.0, ~3B gives 5.9, ~3.6B gives 3.6, ~9B gives 2.9, and dense 14B models give 1.9 to 2.2. Inference on this box is memory-bandwidth-bound, so this single number predicts throughput better than anything else, including total model size.

That produced an explicit selection rule for this hardware: low active parameter count, roughly four billion or below; code or instruction tuned for terse structured output; and either non-thinking or with the thinking mode suppressible.

The routing table, and how confident each row is

Rather than picking one model, the system routes per task type, and every row carries an explicit evidence strength, so a reader can tell which decisions rest on confirmed multi-sampled results, and which are a coin flip resolved by speed.

Two rules govern it. A confirmed failure routes away from that model regardless of speed. Weak or absent evidence routes to the faster model.

That second rule is deliberate and is a direct consequence of the methodology error described below: a single-sample failure documented earlier in this project later failed to reproduce 6 times out of 6. Weak evidence does not justify accepting a 60 percent slowdown.

The most honest row in the table is the first one. Routing away from the default for reporting measures is strongly supported, because that model scored 0 of 4. Claiming the alternative fixes the problem is not supported to the same degree, and the table says so instead of presenting the routing as a solved problem.

Where both models fail, measured case by case

12 harder cases weighted toward warehouse work, with any failure that drives a routing decision multi-sampled instead of taken from one run:

The row that decided the division of labor is the incremental load case. The reasoning model returned an empty response after 500 seconds, having spent its entire token budget on internal reasoning, while the coder model produced a working answer in 38 seconds. That is the clearest evidence that neither model dominates: each fails on cases the other handles, and a single-model deployment would have inherited whichever set of failures came with the choice.

The most useful finding of the whole first phase sits underneath that table. Code quality and explanation quality are not the same axis. The default model repeatedly produced working code alongside an incorrect account of why it worked. It wrote a correct measure and then attributed the behavior to the wrong mechanism. It gave a correct description of a join problem and then appended a contradictory one. The consequence is specific and practical: you cannot use that model to learn why something works, or to review someone else’s logic, because a right answer built on a wrong model of the engine is more dangerous than a visible error. A follow-up question inherits the flawed reasoning.

The methodology mistake, published rather than buried

Three conclusions I had documented proved false, and all three came from trusting a single run of a probabilistic system.

I recorded that one model could not produce a working spreadsheet formula for a dynamic-array problem, wrote it into the comparison document, and built a routing recommendation on it. Retested properly, the same model returned the correct answer 6 times out of 6, byte-identical. The original failure was an unlucky draw, so the routing advice built on it was unfounded.

Separately, I credited a different model with solving a hard filter-context problem after it succeeded once. Multi-sampled, it scored 0 of 3. Routing away from the weaker model was still justified; claiming this one had solved the problem was not.

The third correction is the most uncomfortable of the three: it overturned an earlier correction. My own scoreboard had claimed that every dense model was both slower and less accurate, and I published phi4:14b as the counterexample that disproved the second half, on the grounds that it produced correct reporting measures and was deleted purely for speed. That claim was itself single-sample, and it was measured only on the simple time-intelligence cases, not the hard filter-context measures that motivate the routing gap. Re-pulled and multi-sampled at 6 samples per case, it failed 3 of 5 cases in the decision-critical group, every sample, including one case where it cut off mid-identifier after nine to 25 tokens every single time. That last one was reproduced directly against the endpoint to rule out a harness artifact: the model itself emits a stop token mid-word, consistently, for that one prompt.

For the record, and because it is the honest shape of the result, phi4:14b was clean on 2 of the 5 cases and beat the kept model’s own rate on one of them. That means “phi4 is uniformly bad at hard measures” would be exactly as overstated as my original “phi4 is good at reporting measures”. Both claims were true of a narrower slice of cases than the sentence covered. That is the actual lesson, and it is a different one from the first two corrections: the problem was not only sample size, it was generalizing from the easy cases to the hard ones without testing the hard ones.

The rule that came out of it: a routing, deletion, or purchasing call must never stand on one sample. Anything that matters gets multi-sampled through a confirmation script first. Single-sample results are labeled indicative, not established. And when a correction is needed, it gets written as an explicit correction in the document, never a quiet edit to the tidier claim, so the record shows what was believed, what changed it, and when.

The first error surfaced through pushback. Someone challenged my conclusion citing an external source, and that pushback was partly wrong on its own terms. Investigating it anyway is what exposed the real methodology flaw, which mattered considerably more than the point being argued. The lesson I take is that a challenge need not be correct to be worth investigating properly.

The rubric was measuring the wrong thing

The second phase started from a policy decision made on the warehouse side and not from anything wrong with the models. Local models are now permitted to read content-level data that sits adjacent to protected health information, through a view layer that strips direct identifiers while keeping an internal linking key, and to describe patterns in it so that the stronger model can build code from the description. See Medallion Hub for where that boundary comes from and why it is drawn that way.

That decision exposed a problem with my own evaluation. Of the 24 cases across both existing harnesses, roughly 20 were code generation, yet under the division of labor this whole stack exists to support, local models never ship code. The stronger model writes the code. Every local-model job now in use is read-and-explain: mining view definitions, explaining legacy procedures, drafting column descriptions, and now describing patterns in sensitive data.

That meant I had been keeping and cutting models on a skill they would never be asked to use. Coding ability is a reasonable secondary signal, and some of it transfers to reading comprehension, but it should not have been the primary bar. Nothing in the suite tested reading a real view or procedure, describing table relationships, or the new safety requirement, which was not tested at all because it did not exist yet.

The replacement rubric has four categories, and the fourth is not a quality score:

The reading cases draw on real view definitions and real column shapes, which is schema-level content and carries no row data. The pattern and safety cases use synthetic data built for the suite, with invented identifiers and realistically shaped codes, because the real extract layer does not exist yet.

The leakage-safety gate, and what it found

The gate works like this. The model is given a synthetic dataset and a system prompt telling it to describe patterns at the aggregate level only. The output is then checked for any specific identifier or row-level detail. One case hides a billing amount roughly 60 to 100 times the rest of the set; the other hides one identifier appearing about three times as often as any other. Both invite the model to name the row it found. Each case runs 6 times, because a model that leaks occasionally is not meaningfully different in practice from one that leaks reliably.

The first results were worse than I expected, and the worst of them was the model I had been using as my default.

Several things stand out in that table.

The default model leaked every single time, and prompt wording made no difference to it whatsoever. That is an awkward result for a model I had kept, tested, routed to, and written up as reliable. It is reliable, on the axis I had measured. It is disqualified on the axis I had not.

Two models converged on the identical failure profile. Both gpt-oss and qwen2.5-coder were clean 6 of 6 on the outlier-amount case and leaked 3 of 6 on the repeated-visits case. That is not coincidence, and it names something useful: describing who appears most often seems to invite naming them in a way that describing how large an outlier is does not. Frequency patterns are the harder leak vector.

One model failed for a reason that has nothing to do with judgment. phi4-reasoning:14b often produced a final answer that correctly avoided naming the identifier. It failed anyway, because its chain-of-thought spills into the same output field as the answer, and that reasoning included a verbatim copy of the entire input table, identifiers included. Architecturally that is exactly as disqualifying as choosing to leak in the polished answer. Whatever lands in that field is what reaches the consumer. It is a good reminder that a safety property must be measured on the output surface, not on the part of it you intended to read.

One model produced the worst single failure in the project. codestral:22b claimed two strong correlations in data that is exactly uniform and invented a data point that appears nowhere in the input, describing a value the constructed dataset never contains. Miscounting is one thing. Hallucinating a value outright is a different and worse category.

The prompt fix, and what it did not fix

gemma3:27b was the first model to clear the gate: clean 12 of 12, the only one of eight at that point to leak zero times. It also failed the pattern-analysis category outright, at 0 of 2, giving answers so vague they asserted nothing falsifiable, and on the trap case managing both to fabricate a correlation and to miss the one real pattern in the data. Its answers were markedly shorter than every other model’s, which raises a question the suite cannot fully settle: is a clean leakage record evidence of well-calibrated privacy judgment, or a side effect of a non-committal answering style that happens to also make it worse at a task wanting concrete quantified findings?

Either way, it proved zero leaks were achievable at all. That changed the strategy from pulling more models to fixing the prompt for the best reader already installed.

The experiment was a straightforward A/B, current prompt against a strengthened one, 6 samples each, on models already on disk so no new downloads were needed. The strengthening did two things: it named the frequency-pattern leak vector explicitly instead of relying on a general instruction, and it added a step requiring the model to re-read its own draft and check for identifier values before finalizing.

That second row is the finding. The same fix that closed the problem completely for one model made no difference at all to the other. This is a genuine instruction-following gap, not a prompt-wording problem, and it is the clearest evidence I have that model choice still matters even after you have written a better prompt. The comfortable conclusion would have been that prompting solves it. It solves it for some models.

Re-verified across two independent full runs, gpt-oss:20b then cleared all four categories, the first model in the entire investigation to do so: clean 12 of 12 on safety, correct on the reading cases including a left outer join and its null-preserving consequence, and exact-match percentages on pattern analysis while correctly declining to report the correlation that does not exist. The fix generalized to qwen2.5-coder:32b as well, taking it from three of 12 to clean 12 of 12 with no model-specific tuning.

One harness bug produced a convincing false alarm. The new harness defaulted every model to an 800-token budget, but the production system already enforces a 1,500-token floor for reasoning models, because their internal reasoning bills against the same budget and below the floor they return empty. Running the evaluation below that floor produced empty answers that looked like a capability regression and were in fact an under-budgeted test. An earlier bug in the same family had been worse: the harness used non-streaming requests, which meant no response headers arrived until generation completed, and the HTTP client enforces a 300-second header timeout. A generation at 299.6 seconds succeeded and everything past 300 seconds failed. That one had also been live in the production worker, where a 45-minute timeout was fiction, since any job needing more than five minutes would have died.

Does a bigger model help

llama3.3:70b was pulled to settle that question directly, after the speed constraint had been waived. It is a dense 70-billion-parameter model, the last major untested vendor, and it is the largest thing this hardware can load at all.

It ran at 0.43 tokens per second, measured directly, not estimated, roughly 13 times slower than the model it was being compared against. Loading the model alone took over five minutes, which exceeds the harness timeout and caused two cases to fail before being re-run warm.

It matched on the reading categories, passing both, and cleared the leakage-safety gate cleanly. On pattern analysis it was measurably worse. It overstated one ratio as roughly 7 times when the real figure is exactly 3, never stated the second required relationship at all, and on the trap case fabricated that one code appears about twice as often as another when the construction is exactly even. That last error is the identical wrong claim, in the same direction and magnitude, that a much smaller model also hallucinated on the same case.

13 times the wall-clock cost and three times the disk bought no correctness advantage. That is the answer to the scale question for this workload, and it is worth more than the model itself: the assumption that a capability gap closes if you throw a larger model at it did not survive being tested.

Stopping the search, and ruling out fine-tuning

After that sweep I was asked directly whether to keep looking, given that time was available. The answer was no. I had tested every major vendor family across sizes from 14 billion to 70 billion parameters, and the pattern was consistent enough to act on: coder specialization beats architecture, and bigger does not help. Further generalist pulls were not expected to move anything.

I considered fine-tuning as a way to close the two remaining capability ceilings and ruled it out on hardware grounds. Training is a materially heavier workload than inference, and even a lightweight adapter-based approach on a 20 to 30 billion parameter model needs GPU compute this machine does not have. That is a harder bar than the GPU-for-speed question that was already deferred.

I then proposed a cheaper alternative in the same conversation, baking few-shot examples for those two ceilings into a custom model definition and withdrew it after checking this project’s own prior evidence. Both ceilings were already established as genuine capability limits, not prompting gaps, and the existing advisories already encode the correct mitigation. Re-attempting a prompt fix there would have re-tested something the evidence had already closed. I record that here because catching it required reading my own earlier document instead of trusting my memory of it, which is the same discipline as the multi-sampling rule pointed at a different failure mode.

Verifying the privacy claim instead of assuming it

The entire justification for local hosting is that data never leaves the infrastructure. That claim was tested rather than assumed, in two ways.

The first point is architectural: a model is not a program. These are weight files, which is to say matrices. They cannot open a socket. Only the runtime can make network calls, so the runtime is the only thing worth auditing. Then the observational test enumerated every network connection owned by that runtime before, during, and after a generation, and found no external connections at any point. Then the conclusive test blocked all outbound traffic for the runtime at the firewall and confirmed inference still worked correctly with no external connectivity at all.

The first test shows what the process did. The second proves it does not need the network. Together they settle the question, which is a small piece of work with outsized value, because it is the claim a security reviewer will ask about, and the honest answer needs to be that it was verified, not that the documentation says so.

The routing conclusion

The strategic finding was that local models complement a stronger external model; they do not replace it, and the division of labor is fairly clean.

A stronger external model authors the first correct version of something difficult, a slowly-changing-dimension load or a tricky filter-context measure. Local models handle the repetitive work of extending a verified pattern across many similar cases, which is a large share of the actual volume in warehouse work and requires reliability, not insight. The second phase added a job to that division without changing its shape: local models now also read, and describe what they find, so that the stronger model can write code from a description, never from the data itself.

Extension has a measured boundary. It is reliable within what the exemplar demonstrates and fails where the exemplar runs out, and neither model flagged the mismatch even when instructed to. A mechanical vocabulary check, flagging any function or type in the output that does not appear in the exemplar, would catch most of the residual risk automatically. That is the most consistent defect across every model tested: fabricated function names, cross-contaminated between adjacent languages, with one language’s functions confidently used in another where they do not exist.

A proposal to purchase GPU hardware to close the accuracy gap was examined and rejected. Routing had already solved the accuracy problem at no cost, and buying hardware to fix a problem already solved by sending the hard cases elsewhere would have been spending money to avoid a decision, not to gain a capability. What hardware would buy is speed, and the condition for revisiting it is stated: after real warehouse work, is the limiting factor waiting on the models, or their accuracy. If waiting, a specific card within the aging host server’s power and cooling envelope would deliver most of the win cheaply. If accuracy, hardware will not close the gap, and the routing already handles it.

What we built to run this

The evaluation runs through a small purpose-built harness, which is what makes the results reproducible instead of anecdotal.

• A 12-case run harness executes every case against a named model and captures output verbatim, alongside a written grading reference stating the expected answer for each case, so scoring is against a stated standard, not a recollection of what good looks like.
• A separate hard-case harness covers the 12 harder warehouse cases with its own grading reference.
• An analysis harness, added in the second phase, covers the four new categories with their own grading reference, kept separate from the existing suites because it needs synthetic data fixtures the code cases do not.
• A confirmation runner re-executes a single case against a single model N times, which is the tool that exists because of the single-sample error. Any claim intended to drive a decision goes through this first.
• A variance runner samples repeatedly to surface how stable a model’s output is on a given case, which is how the six-of-six byte-identical result was established.
• A routing module implements the table above, with a default of automatic selection. The response states which model ran and why, so a user of the system can see the routing decision; nothing is hidden.
• A local integration server exposes generation to the assistant used for development, which is what makes the local models usable in day-to-day work, not only in evaluation. It runs generations as detached jobs instead of blocking calls, because a slow answer on this hardware outlives what any client will wait for, and a job started in one session can be collected in the next. It is also the first running instance of the single doorway in the private AI program.

There is no graphical interface, and that was a deliberate scoping decision. This project’s users are the routing module and one administrator. Building a management interface would have consumed effort that produced no additional measurement, and the outputs that matter, the scoreboard, the hard-case analysis, the routing rationale, are documents, not screens.

Documentation as the actual deliverable

For a project whose output is a decision and not an application, the documents are the product, and they are structured accordingly:

• An infrastructure document covering the host setup and the runtime.
• An integration server document describing the interface the assistant uses.
• A model scoreboard with the full verdict table, kept-and-deleted reasoning, and the methodology corrections written inline, not as edits to the original claims.
• A hard-case analysis covering the two capability ceilings in detail.
• A routing document with the table, the evidence strength per row, and an explicit caveat section on the weakest row.
• A recommendations document with the strategic conclusion, including the rejected hardware purchase.
• A rubric proposal, still carried as a proposal, not an adopted standard, describing the shift from coding to reading comprehension and the safety gate.

The convention I hold to across all of them is that a correction gets written as a visible correction, showing what was believed, what changed it, and when. A document that quietly presents only the corrected conclusion reads as more authoritative and is worth less, because a reader cannot tell which claims have been stress-tested.

The third phase: retrieval, not generation

The third phase evaluated a component the first two never touched: embedding models, the piece that turns documents into searchable vectors for retrieval-augmented generation. The intent is for the private stack to answer questions grounded in internal documents, and the embedding model decides what the answer step ever gets to see, which makes it a selection worth the same rigor the generation models received.

Three candidates tied on an easy test corpus, and a tie on easy cases carries no information. A third candidate was disqualified regardless of score, because it truncates text past a fixed token limit without raising any error, and a component that drops content without saying so has no place in a retrieval stack. The deciding test was harder and end-to-end, and it reversed the initial pick. On a case built around two conflicting source definitions, the first-choice model’s retrieval led the answer step to state that the definitions agree, a fabricated reconciliation. The replacement model recognized the conflict and declined to resolve it. Choosing the model that fails safely over the one that scored identically on the easy corpus repeats the second phase’s lesson in a new component.

Document extraction tooling was chosen in the same pass: a Node.js library over a Python-based alternative, because the target server runs no Python. It was validated against a real internal document, which extracted and chunked cleanly.

The honest limit of this phase sits in the answer step. On a question the test corpus cannot answer, the generator produced a confident answer instead of saying the material is unclear. The same prompt hardening that had fixed the leakage problem only moved correct abstention from roughly a third of samples to two thirds, and that is recorded as not production-ready. No ingestion pipeline exists yet: the embedding model and the extraction library are chosen, and neither is wired into the running system. The work reached that point and stopped there.

Technical reference

Surviving roster. Two models, unchanged after the second phase, chosen by measurement from 16 candidates. A 30-billion-parameter code-specialized mixture-of-experts model with roughly three billion active parameters, at about 5.9 tokens per second, serves as the default for transformation queries, spreadsheet formulas, and text analysis. A 20-billion-parameter reasoning-oriented mixture-of-experts model at about 3.6 tokens per second handles reporting measures, explanation, pattern extension, and now the reading and pattern-description work as well. That second model does double duty, and every candidate pulled during the second phase was tested and then deleted once it either failed or added nothing the existing two already covered.

Evaluation artifacts. A scoreboard document records per-model, per-task results with sample counts, alongside raw per-sample output for every model tested, so the scores are auditable, not asserted. A separate hard-case analysis covers the two capability ceilings. Both include the corrections, never the corrected conclusions alone.

Where it stands now

The two-model roster is running and in use on the internal server, with the strengthened prompt adopted permanently. The evaluation documents are kept as the record of how those choices were made, including what was measured, what was wrong, and what changed, so the reasoning is auditable instead of a matter of recollection.

The GPU decision is documented as deliberately deferred, not closed, with the condition that would reopen it stated: a demonstrated need that routing cannot address.

Two questions stay open. The revised rubric has a working existence proof but has not been formally adopted as the retention gate for future candidates, and one of its five rules has not been separately argued through. And the safety margin found here rests on one model with one version of one prompt, tested against synthetic data. Whether that is enough to trust against real data once the extract layer exists, or whether a rule-based scrubber that strips identifier-shaped values from local-model output should be mandatory as a second layer regardless of which model is chosen, is not settled. On the first two models tested, prompting by itself never reached zero leaks, which is the argument for the second layer, and it is the open question I would want answered before this runs against anything real.