Sep28
Why AI governance breaks where healthcare organisations can least afford it, and how to build one that holds.
This September, EY asked senior AI leaders at some of the largest companies in America whether their organisation had formal AI governance policies. Almost all of them, 98%, said yes. Yet 47% said their organisation had previously not applied its governance process when a deployment was urgent.[1]
It’s worth pausing on who these people are. They work at billion-dollar companies with legal teams, compliance departments and advisers on call. They have the policies written down. And yet, when a deployment was urgent, nearly half of them hadn’t applied their own process. Even EY’s Americas Assurance chief technology officer acknowledged that governance needs to become “something more than just words.”[2]
If that’s what happens inside the best-resourced organisations in the world, it’s fair to ask what happens inside a community hospital or an independent clinic, where there’s no dedicated AI team and every decision competes with patient care for time and money.
Before going further, it helps to be clear about what AI governance is, because the word gets used loosely. At its simplest, it’s how an organisation decides which AI tools to use, how it checks that those tools actually work, and who’s accountable when they don’t.
A policy is only the written part of that. It describes what’s supposed to happen. Governance is what actually happens when someone has to make a real decision, often under pressure. Most of the risk lives in the space between the two.
The most reliable picture we have of how hospitals handle this comes from the US federal government, which draws on a national survey of hospitals weighted to reflect hospitals across the country.[3] It shows a clear divide.
Hospitals that belong to larger multi-hospital systems evaluated their predictive AI for accuracy 76% of the time, checked it for bias 62% of the time, and monitored it after launch 62% of the time. Independent hospitals did far less: 41% checked for accuracy, 31% for bias, and 38% kept monitoring after launch.[3] Size tells the same story. Small hospitals came in at 65%, 52% and 53%, compared with 74%, 61% and 65% for large ones.[3] There’s one more detail that turns out to matter a great deal: among small hospitals that use predictive AI, 85% get it from their EHR vendor rather than building or choosing it independently.[3]
The money follows the same pattern. One industry survey found that large health systems set aside a median 6.8% of their IT and quality budgets for AI governance, while small hospitals set aside just 2.3%.[4] When asked whether they could produce a complete record of how their AI was used within 30 days if a regulator asked, only 15% of small hospitals felt highly confident they could.[4] And when hospitals were asked what got in the way of auditing their AI, the answer they gave most often, at 41%, was that their AI vendors didn’t provide enough documentation.[4]
Even among health systems, the gap between what’s said and what’s done is striking. In a small study of 27 health system leaders, 23 of the 25 organisations using outside AI tools said those tools were tested and validated before going live. But only 11 of them actually had a proper environment in which to do that testing.[5]
Most governance advice assumes you can run an AI committee, keep a registry of every model and monitor each tool full-time. Smaller organisations usually can’t, so the policy they copy from a large hospital is the first thing to go when something becomes urgent. And because they mostly buy AI rather than build it, their governance really comes down to a buying decision that rests on documentation the vendor controls.
There has been real progress in 2026. In June, the Joint Commission launched a voluntary certification for the Responsible Use of AI in Healthcare.[6] That’s a genuine step forward, but it’s important to understand what it does and doesn’t cover. The certification doesn’t evaluate individual AI products or how they’re used. What it assesses is whether an organisation has sound governance practices in place.[7]
Across the Atlantic, NHS England maintains a registry of ambient voice technology suppliers, the AI tools that listen to consultations and draft clinical notes. The registry is self-certified, and it currently lists 19 suppliers.[8] At the same time, England’s independent patient safety investigator, HSSIB, has opened an investigation into how these tools are being used in hospitals. It found that adoption is accelerating while the safety implications still aren’t fully understood, and that the routes for reporting AI-related incidents aren’t yet mature.[9] Its findings are due in summer 2027.[9]
Taken together, these developments reveal something important. An organisation’s governance process can now be certified, and a supplier can vouch for its own product. But nobody is independently testing the specific decision a hospital makes: this particular tool, from this particular vendor, for these particular patients.
The answer isn’t a longer policy. What smaller organisations need are a few firm checkpoints at the moments that matter most.
The first is the moment of purchase. Before you sign anything, ask for evidence rather than assurances. Find out what the tool was tested on, and whether those patients look anything like yours. Ask what documentation you’ll receive, and make sure it’s written into the contract.
The second is the moment of urgency. Decide now, while there’s no pressure, what no deployment will ever skip, however urgent it seems. That 47% figure shows that governance fails precisely when the pressure is on, so your minimum standard has to be built for that moment.
The third comes after launch. Give every AI tool a single named owner, and make sure your staff know exactly how to report it when the AI gets something wrong.
Governance policies can be written internally and certified externally, but the individual decision still needs testing: whether this tool, from this vendor, holds up for these patients. That's the role of an independent validator, one with no stake in the vendor, the certification or the software, and no reason to prefer any particular answer.
Proceed, with conditions. Keep adopting AI, but stop judging your governance by how long your policy is. Judge it instead by whether it held the last time something was urgent.
If you’ve been following this topic, you’ve probably noticed that AI governance statistics rarely agree, ranging anywhere from 12% to 98% depending on who’s counting. What matters more than the number is who’s telling you, and what they happen to be selling. That’s exactly what we map at the Straven Institute.
→ Read: Who’s Telling You Your AI Governance Is Broken? on the Straven Institute
Data in this article is from US hospitals and companies unless stated otherwise. For general information only; not legal, clinical or regulatory advice.
Keywords: AI, AI Governance, Healthcare
Your Hospital Has an AI Policy. That Isn't the Same as AI Governance
No new technology investments needed to succeed with AI
Dreamforce signals where the market and our environments are headed
Dreamforce signals where the market and our environments are headed
A Smart City Needs More Than Data — It Needs Decision Architecture
