Private AI Program (RAG, MCP)

This program began with one practical question: what useful AI could run on a server we already had. Behind it sits the constraint that governs everything here. Working in behavioral health means patient data must stay inside our network, under HIPAA and 42 CFR Part 2, so a private AI matters more than any cloud service. The stack is open source, it runs on hardware we already had, and it required no new licenses.

The title’s two acronyms are the program in miniature. RAG, retrieval augmented generation, is the centerpiece now in build: the private AI first searches our own material, then answers grounded in what it found, citing sources. MCP, the Model Context Protocol, is the doorway underneath it: an emerging standard connecting AI systems to data and tools, used here so every AI request for data goes through one governed place that can control, limit, and log access.

The RAG build’s first customer is deliberately narrow, the data team building the warehouse, with four stated uses: confirming what one row of a table means from how staff describe counting things, discovering dimensions and facts from recurring terms, catching conflicting definitions before they ship in a report, and a data dictionary quoted from source material instead of reconstructed from memory. One boundary governs all of it: RAG surfaces evidence, the data team decides, and nothing changes the warehouse automatically. We could have had a managed version of this from Copilot Studio, whose licensing we had access to if needed but never bought. We chose to build instead, because its retrieval and models run in Microsoft’s cloud, and keeping that work inside our own walls is the reason this program exists. The full comparison is in the technical detail below.

The rest of the architecture makes the privacy promises physical instead of policy on paper. A privacy gate sits on the MCP doorway’s warehouse leg, stripping names, Social Security numbers, and birth dates before any AI reads warehouse data, so the AI sees patterns, never identities. Routing is by data classification, never by difficulty: anything touching clinical data stays on the locally hosted models, anything the router cannot classify also stays local, and protected health information never reaches a cloud model under any route.

Parts of this run today and parts are planned, and the program’s own documents refuse to blur that line. Built: the two models kept after a measured evaluation of 16, the first doorway instance connecting the coding assistant to the local models, and the routing table. In build now: the retrieval capability itself, with its AI model finalized. Planned: the transcription pipeline, which waits on a recording practice change that needs an executive decision, and a second doorway instance that refuses to serve any unregistered or unclassified data object, designed and not built.

The models are evaluated, measured, and ready; they are not yet serving staff-facing traffic at scale, and this page will say so until that changes. A GPU purchase is gated on measured waiting time becoming the constraint, not on accuracy, and it has not been triggered.

How all the pieces compose into one system, and why the whole machine was specified before the build, is told in One Machine.

The technical detail

This program keeps protected health information inside the organization’s network while AI still does useful work: our own RAG now in build, locally hosted models, one governed doorway between any AI and internal data, which is the MCP layer of the title, and a privacy gate on the warehouse leg. The line between what runs today and what is planned stays explicit throughout.

Why a private AI at all

The work opened with a single practical question: what useful AI could a server we already owned run. The answers became the reasons for the whole program, and they still hold. Patient data must stay inside our network, which follows directly from HIPAA, 42 CFR Part 2, and payer guidance, and that is the core reason this AI is private instead of a cloud service. The foundation could be built on hardware we already host, with free open-source tools, so it required no new licenses. Transcription of meetings was a real, concrete need from the start. And staff needed help with everyday document work, summarizing, drafting, and answering questions about internal material, without sending anything outside the organization.

The idea throughout is to use hardware already available rather than buy for the program, and the constraint that follows is stated wherever a result is stated: the server is roughly 10 years old, with 40 logical processors, 160 gigabytes of memory, and no GPU. The model evaluation was run on that machine, which is why its conclusions transfer to this program without adjustment.

The rules distill into a three-tier framework:

RAG, and its deliberately narrow first customer

Retrieval augmented generation lets the private AI answer questions using our own information instead of general knowledge: it first searches our material, then writes an answer grounded in what it found, citing where each fact came from. The build is five steps: capture the source material, convert speech to text with a private transcription tool, break the text into searchable pieces, index those pieces, and answer questions from the most relevant pieces.

The first customer is not a general-purpose chatbot rollout. It is the data team building the warehouse, because warehouse design depends on knowledge that lives in conversations and in old report code. Four uses matter most:

• Declaring grain. What one row of a table stands for is confirmed from how business users describe counting things, in their own words, before modeling.
• Discovering dimensions and facts. Terms that recur in conversation and documents, such as program, payer, and location, point at the dimensions that matter, and the numbers people keep asking about point at the facts.
• Catching conflicting definitions early. Two departments can mean different things by “active client”, and finding that before a report ships prevents the argument after it ships.
• A data dictionary that writes itself. Definitions staff state out loud become quoted source material for the warehouse documentation, instead of being reconstructed from institutional memory.

One boundary keeps this safe, and the program states it in every artifact: RAG surfaces evidence, the data team decides, and nothing changes the warehouse automatically. Every change still passes the human review gate.

The input sources carry their status explicitly, because conflating them would overstate what exists:

A first working version therefore needs nothing that does not already exist: the plan scopes it at roughly 20 to 50 policy and procedure documents, built by the current team alongside existing duties, and the plan’s own estimate is 6 to 10 weeks to a working version. The search index lives inside SQL Server, the same platform as reporting, and the embedding model underneath it was chosen through its own measured evaluation, which reversed its initial pick on a conflicting-definitions test case.

The storage decision had rejected alternatives, and they are recorded so they do not get re-litigated. I rejected a popular in-memory cache as a vector store because it is a key-value store with vector search bolted on. A cloud document database was rejected because it is cloud-only, which conflicts with the reason this program exists. Dedicated vector databases were considered and set aside because the more consistent answer was to avoid adding a new stateful technology when SQL Server, which already carries the backups, the security regime, and the team’s expertise, gains native vector search in its next version, with a lightweight stopgap until the upgrade.

The alternative not built: Copilot Studio

The fair version of the question first, since this is a Microsoft shop and the licensing was available to us if needed, though we never bought it. Copilot Studio implements the same retrieval-then-generation pattern this build uses: knowledge sources are connected, content is retrieved to ground the model’s answer, and citations point back to the source. As a pattern, it is genuine RAG, and it comes managed, with the retrieval, embedding, and model layers run in Microsoft’s cloud.

Where the components run is the whole decision. Copilot Studio sends document content out for embedding and processing, governed by the tenant agreement but outside the on-premises boundary, and for an organization handling protected health information that boundary is the constraint everything else on this page serves. The build keeps retrieval, storage, and the model on internal infrastructure: SQL Server as the vector store, the locally hosted models serving generation.

Two more reasons carried real weight. Cost shape: Copilot Studio bills per message or capacity and grows with usage, while the build runs on hardware and licensing already owned, with no per-query cost. And control: the orchestration layer owns chunking, retrieval logic, model choice, and routing, so an operational question like who to contact for a facilities issue can go to a structured contact table instead of document retrieval, which is not a shape a templated grounding pipeline offers. The same pipeline then extends to future use cases, the warehouse’s gold layer among them, with nothing new to license. The costs essay records the mirror image of this decision, a build dropped because buying was the better call; the two together are the same framework pointed at different facts.

One doorway, and why

Every AI request for data passes through a single connection layer, modeled on the Model Context Protocol, MCP, the emerging standard for connecting AI systems to data and tools. The working image in the program’s own documents is a wall socket: one standard plug between any AI and the organization’s data, so a local model and a cloud assistant connect through the same fitting instead of each arriving with its own wiring. One doorway means one place to control, limit, and log access. That is a deliberate architectural choice, not an accident of convenience: scattered integrations would each need their own controls, and the one that got them wrong would be the one nobody was watching.

The first instance of that doorway is built and running. It connects the coding assistant to the local models as asynchronous generation jobs, because CPU-only inference outlives what any client will wait for on a single call, and its mechanics are described on the evaluation page. Two of its conventions matter here. Job records contain full prompt text and are purged automatically after 7 days. And no real client data goes into prompts at all; test material uses synthetic, structurally similar values by standing convention.

The second instance is designed and not built, and its own documentation says so in plain words. It moves the central guarantee from a behavioral one to a structural one: instead of relying on the router classifying correctly every time, a separate gateway service refuses to serve any data object that is not registered and classified, before a query executes at all, regardless of what the router concluded. Sensitivity is declared per column, across five categories, because real tables mix identity columns with harmless codes. Its build order is deliberately reversed from the obvious one, audit logging before enforcement, chosen because the team doing this work is a single person and a log of what would have been blocked is the safest way to tune refusal rules before they can break anything. The warehouse page describes where this gateway sits in that project’s enforcement chain. In the program’s working order it comes after RAG: retrieval first, then this gateway, then the warehouse migration whose enforcement chain it joins.

The doorway is also the honest answer to a worry that reliably comes up when a local model is proposed: whether the model could act on the network by itself. It cannot; a model generates text, and any capability beyond that exists only where the surrounding layer deliberately grants it. That is why the design effort here goes into what the doorway permits and what gets indexed, and why the compute host sitting off the domain network is a separate, ordinary layer of defense rather than the thing standing between the model and the network.

The privacy gate and the routing rules

A privacy gate sits on the path between the doorway and the warehouse. Before any AI reads warehouse data, names, Social Security numbers, and birth dates are stripped, so the AI sees patterns, never identities. That is identifier removal, not full de-identification, and the program’s documents say so instead of glossing it; the stronger safeguard is that the identifier-bearing columns never reach a local model or its logs at all.

Routing between models is decided by data classification, never by difficulty or confidence:

flowchart TD
    Q["A task arrives, carrying its<br/>data-path classification"] --> O["Orchestrator<br/>a governed script today"]
    O --> R{"Touches clinical data?"}
    R -->|"Yes"| L["Local models<br/>own server, no egress"]
    R -->|"Cannot classify"| L
    R -->|"Clearly non-clinical"| C["Cloud model"]
    L --> G["The single doorway<br/>control, limit, log"]
    C --> G
    G --> P["Privacy gate<br/>strips names, SSNs, birth dates"]
    P --> W[("Data warehouse")]
    G --> F[("Documents and internal files")]

The rule the diagram cannot show is the failure direction. Anything the router cannot classify stays local. It fails closed, never toward the cloud, and protected health information never reaches the cloud model under any route. The orchestrator receiving requests is a simple, governed script with a real manual approval stop, and it becomes LangGraph only when the decision logic outgrows the script. That staging is stated in the program’s own documents, so nobody assumes the framework exists before it does.

Transcription, and the decision that is not the data team’s to make

Transcribing internal calls would create a searchable record that outlives attendance: decisions and commitments made in meetings are often remembered only by the people who were in the room, and a searchable archive keeps that knowledge when someone forgets, changes roles, or leaves. The transcription tool itself, Whisper, runs on our own server, so recordings would never leave the network.

Recording internal meetings is not current practice, and changing that is a decision for executive leadership, not a technical team. The proposal that exists carries its guardrails on its face: participants always know when a meeting is recorded, recordings stay on our own server with role-based access and defined retention, and they are never used to evaluate individuals. Until that decision is made, transcripts stay in the “comes with the build” column, and this page does not count them as a source.

Built and planned, without blur

The program’s companion guide opens with the sentence “Nothing here claims to be built when it is not,” and this page inherits the rule.

One sentence from the program’s own brief belongs here verbatim in spirit: do not assume the models are in production use for end users yet. They are evaluated, measured, and ready to deploy, and no staff-facing traffic runs on them at scale yet.

The open questions

Two are undecided, and both are recorded in the program’s documents with their blockers named.

Whether to pursue a governed external tier, a frontier model reached through a cloud platform under a business associate agreement, for non-clinical hard-reasoning work. Cost is not the blocker; the realistic volume prices in the tens to low hundreds of dollars a month. The blocker is whether 42 CFR Part 2 and payer rules permit it at all, and that needs its own one-page evaluation memo, not yet written.

And whether to buy a GPU for the server. The trigger is explicit: measured waiting time becoming the actual constraint, not accuracy, since the evaluation already established that routing solved the accuracy gap at no cost. The trigger has not fired.

Where it stands now

The first doorway instance runs today in the team’s daily development work. The evaluation that chose the models is complete and published, with its corrections. The data gateway is designed and not built. RAG has moved past design into an active build, and the AI model choice is final. The document extraction library and the SQL Server index design were settled during component choice. The program’s documents keep the built-versus-planned line explicit, and this page will keep inheriting it, because the site has corrected design published as achievement twice before and does not intend to do it a third time.