Our data platform grew the way most do, one system added at a time, each one reasonable for the moment it was built in. Some of those systems predate my time on the team. As the organization scaled and our core clinical database moved from on premises to the cloud, a few of those original designs started showing real strain: reports occasionally failed overnight for reasons that took until morning to track down, and a job that broke partway through meant starting over from scratch rather than resuming where it left off. None of that reflected bad engineering. It reflected systems built for one set of conditions, later asked to run under a different one.
Every project in this portfolio grew out of responding to that shift, and none of them happened in isolation. They also did not come out of nowhere: the longer story of how I got here, across several generations of tooling, sits behind all of them. Together they form one modernization effort for this organization’s data and reporting infrastructure, moving those aging systems toward a single coherent platform, with AI assistance woven into how the work itself gets done, not just what gets built.
The layers, and why they were built in this order
The data platform comes first, because everything else depends on it. The reason is best seen in what happens without it. A request for a quick figure resolves through hundreds of undocumented views, against schemas that change without announcing it, using business logic that exists only as stored procedures nobody can diff or test. So a twenty-minute ask costs days, every time, and the cost recurs on every request because nothing underneath it changes between them. That is a foundation problem rather than an estimation problem, which is why it gets fixed first rather than absorbed repeatedly. Medallion Hub is the data warehouse itself, built on the Medallion architecture, the industry pattern where data moves through three layers, raw, validated, and business-ready, with each layer reading only from the one below it so any number in a report stays traceable to its source. The migration onto it uses a blue/green switch: the old pipeline and the new warehouse both hold a complete copy of the same tables, and a layer of database aliases in the reporting database decides which one reports actually read from. Because the reporting logic references those aliases rather than physical tables, switching means repointing aliases rather than rewriting a single report. That warehouse needs something reliable feeding it. Sluice is what I built for that job, a parallel extraction platform replacing a job that had outgrown the infrastructure it was originally built for, with something fast and resilient enough that the rest of the platform can actually rely on fresh data landing every night without manual intervention. Neither piece is useful without the other, and building the warehouse without first fixing extraction would have meant building on the same foundation being replaced.
Underneath all of it sits work that predates every project here and never stops: keeping the databases themselves recoverable. Encrypted Backup Key Rotation is the smallest piece described on this site and the one where failure is least visible, since an encrypted backup that cannot be decrypted looks exactly like a working one until someone needs it.
Two choices inside that warehouse are worth naming, because they were decided against reasonable alternatives. Transformation logic runs on dbt Core rather than the integration tooling already in production, chosen because plain SQL and YAML files diff cleanly and can be generated, reviewed, and regenerated as text, which matters enormously when measurement showed most of the modelling work to be mechanical. And calculation logic lives in SQL views rather than inside the reporting tool’s own expression language, which keeps it version-controlled, reviewable by anyone who reads SQL, and portable if the reporting front end ever changes.
The shoemaker’s children
There is an analogy I keep returning to, and it explains a good deal of what follows.
The shoemaker’s children go barefoot. Not through neglect, but because every hour the shoemaker has goes into shoes someone is paying for, and their own family’s shoes never make it to the top of the list.
Technology teams do exactly this. Everything we build goes to the business, and it should, because that is where the visible need is and where the funding comes from. What never gets built is the small thing that would make our own work easier. A tool to read the logs properly instead of squinting at a table. A script that answers what is blocking what. A check that would have caught a misconfiguration before somebody noticed it in production. Every one of those is obviously worth having, none of them has a business sponsor, and so they sit permanently below the line.
I have always tried to build them anyway, and for most of my career that meant building them badly or not at all, because a two-day internal tool cannot be justified against two days of work somebody is waiting for.
This is the part of AI-assisted development I find most underrated. It did not only make business-facing work faster. It changed which work is worth doing at all. A diagnostic tool that would have taken three days, and therefore never got built, is now an afternoon, which puts it firmly inside the range of things you simply do without asking. The economics of small internal tooling changed, and a category of work that had been permanently deferred became affordable.
Several things on this site exist only because of that shift. A metadata snapshot tool that captures a drift baseline, so changes in a source system are detectable rather than surprising. Full-text search diagnostics kept in the repository, because index population state is worth querying directly rather than inferring. An evaluation harness with a confirmation runner, built because a single sample had already misled me once. A tool that audits every page in an application for layout problems and fixes what it finds. Another that regenerates an entire help system, annotated screenshots included, so documentation stops decaying the moment the interface changes.
None of those has a user outside the team. All of them make the team’s work better. None would have survived a prioritisation conversation.
What one of these actually looks like
The next one I intend to build is a wait statistics dashboard for SQL Server. The idea is not mine: it comes from a post on SQLAuthority describing exactly this kind of dashboard being assembled with AI assistance. I read it, recognised something I had wanted for years, and adapted the approach to what I actually need to see.
It is worth showing the specification rather than describing it, because the specification is the part that matters:
Categorise waits into CPU, I/O, memory and other. Show cumulative wait time since last restart, top wait types, average wait time per task, and active waits. Charts for wait distribution. Separate signal waits from resource waits. Generate recommendations based on the patterns found. Dark themed, auto-refreshing every few seconds. Highlight CPU pressure when signal wait exceeds twenty-five percent of total wait time.
Nothing in that is exotic. It reads from two standard diagnostic views the database engine already exposes. Every competent database administrator knows those views exist, and almost nobody has a decent interface onto them, because building one has never been worth two days against work somebody is waiting for.
The part of that specification I could not have delegated is the domain knowledge inside it. Separating signal waits from resource waits is the distinction that tells you whether a server is waiting on CPU scheduling or on something external. The twenty-five percent threshold is a real heuristic, not a round number, and knowing which patterns deserve a recommendation at all requires having spent years reading these numbers. That judgment is mine. The dashboard around it is an afternoon.
That division is this whole site in miniature: the reasoning about what should exist and whether the output is correct stays with me, while the implementation that used to make it unaffordable no longer does. It is also, I suspect, why the tools in this category are finally getting built after twenty years of being on a list.
The provenance is worth dwelling on for a moment, because it is the practice I am arguing for elsewhere on this site. I found the idea on a well-read community blog, the same way I have found solutions to problems for twenty-five years. I evaluated whether it applied, adapted the parts that did, and brought my own domain knowledge to the parts the source could not know, meaning which patterns matter in our environment and what should trigger a recommendation. Then I used an assistant to implement it.
Nobody would describe reading a blog post and adapting the idea as a shortcut, and that is precisely the inconsistency I keep running into when colleagues treat the assistant differently. The composite here is entirely ordinary professional practice: a public source, my own judgment, and a tool that types faster than I do.
Shared infrastructure comes next, so future applications are not each solving the same problems from scratch. The Job Scheduler exists so that different internal applications can run and monitor recurring jobs through one shared mechanism rather than each inventing its own, and its hosting configuration is now hardened by a twenty-seven-day outage that taught me more about production reliability than any amount of planning would have. The Application Template exists so that a new internal application starts from a working foundation, authentication, layout, established coding patterns, rather than being built from an empty folder every time. Almost nothing in it is theoretical: every documented pitfall traces back to a real incident in a real project. And the Shared AI Skills Library exists so that the team’s actual working habits with an AI assistant, the shared instructions, the memory conventions, the review discipline, live in one version-controlled place everyone pulls from rather than in each person’s head.
Concrete applications come after the foundation exists, and reuse it deliberately. The Residential Client Portal and the Legacy Ticket Archive were each built on top of the shared application template rather than as standalone efforts, which is why they could move quickly once the template existed. These are the shoemaker’s own shoes: nobody outside the team would notice if they did not exist, and everything built afterward was faster because they did. The Residential Client Portal is the largest of them, replacing spreadsheets and eleven separate lists with a single system of record. The Legacy Ticket Archive solves a narrower problem well: keeping a decade of retired support history searchable after the platform holding it was decommissioned, so the record survives without the licensing and maintenance cost of keeping the old system alive.
The reporting layer follows the same incremental logic. The BI Platform Upgrade plan covers moving an enterprise reporting platform across two major versions, including the decision to stop troubleshooting a failing authentication mechanism and rebuild on modern authentication instead. An interim web portal was built first so legacy report formats could be retired quietly, one report at a time, behind a consistent interface, without users noticing the migration happening underneath them.
AI capability is being built as its own layer, deliberately, rather than treated as a feature bolted onto individual projects. Claude Code is the primary way all of the above actually gets built, which is why the methodology behind that has its own separate document rather than being mentioned once and forgotten. It is also why what building this way actually costs has its own document: the failure modes are real, specific, and mostly invisible to the machinery that normally catches mistakes.
Alongside that, a genuinely internal AI capability was built and measured on existing hardware. The On-Premises AI Evaluation tested seven local language models against real warehouse work, established exactly what they are reliable for and where they fail, and verified rather than assumed that data never leaves the organization’s own infrastructure. It also produced two documented conclusions that turned out to be wrong, both from trusting a single sample, which changed how I evaluate anything probabilistic.
That capability is not a side experiment. It is the foundation of a three-tier approach to AI governance across the department: fully on-premises models for the most sensitive work, governed external tools for reasoning and authoring that benefits from a stronger model, and a clearly defined set of categories that should not touch AI assistance at all regardless of which tier is available.
The unfinished half of that is organisational rather than technical. Having a governed practice and having a team that actually uses it are different achievements, and the second one is where I have made the least progress.
Why this matters as a whole, not just project by project
Read individually, these projects look like a data warehouse, an extraction engine, a couple of internal web applications, and an experiment with local AI models. Read together, they describe a department moving deliberately from a collection of aging, individually-maintained systems toward one platform with a consistent foundation underneath it, and doing so with a documented, evidence-based approach to where AI assistance fits into that work, rather than adopting it informally and inconsistently across the team.
The order these were built in was not arbitrary. Fixing extraction before building on top of the warehouse it feeds, building shared infrastructure before the applications that would otherwise duplicate it, and evaluating AI tools with the same rigor applied to any other infrastructure decision, each reflects a department being built the same deliberate way twice, once in its data infrastructure and once in how it adopts new tools.