That is a practical case for governance before deployment. It is not evidence that hospitals generally buy AI before considering safety. Available survey data show both widespread adoption and substantial reported evaluation activity, with important gaps and limitations.
What hospital adoption data actually show
The US Office of the Assistant Secretary for Technology Policy's 2025 data brief reported that 71% of non-federal acute-care hospitals in its 2024 survey data used predictive AI integrated with or launched from their electronic health record. The analysis used responses from 2,080 hospitals and excluded missing responses from the relevant measures. It did not cover every hospital, every country or every form of AI. Hospital trends in predictive AI, 2023–2024.
Among hospitals using these systems, 82% reported evaluating accuracy for at least some models, 74% evaluating bias, and 79% conducting post-implementation monitoring. Coverage was lower when the question concerned all or most models. These are self-reported practices, not independent audits of their quality. The survey supports asking how complete and effective governance is; it does not establish that the institutions skipped it.
A useful governance article therefore needs to go beyond an adoption count. Which decision does a system influence? What happens if it is wrong? What evidence is sufficient for that use? Who acts when performance changes? Those questions apply whether a hospital has one tool or dozens.
Begin with the clinical task and the baseline
“Use AI” is not a sufficiently specific objective. A proposed deployment should identify a problem in the existing service and a measurable improvement: for example, reducing a defined reporting delay without increasing missed findings, or reducing documentation work while preserving the accuracy of the record. These are illustrative objectives, not claims that any particular product achieves them.
The comparison matters. A model may perform well against a retrospective label yet fail to improve the service compared with its existing process. A faster workflow may also create extra reviews downstream. Assessment should distinguish the model's predictive performance, the behavior of users, operational effects and patient outcomes.
The intended use should specify who uses the output, which patients or tasks are covered, which inputs are required and what use is excluded. A tool approved for one purpose should not quietly acquire another because it is easy to access inside the electronic record. The organization needs a process for reviewing changes of use as well as changes of software.
Maintain an inventory that leads to decisions
An inventory is useful because AI can arrive through a direct purchase, an existing vendor's update, a departmental pilot or a documentation service. Recording an application is only the beginning. The inventory should connect it to an owner, a clinical pathway and a review status.
A workable record includes the product and version, vendor, intended use, affected service, relevant inputs, risk assessment, supporting evidence, deployment status, monitoring owner, update arrangements and fallback. Patient-data flows and third-party access belong alongside those details. The content should be proportionate: a scheduling aid and a deterioration alert do not necessarily require identical evaluation.
The hospital also needs an accessible route for staff to report a new or changed tool. An inventory that cannot capture vendor-embedded features or locally initiated pilots will become incomplete even if its original list was accurate.
Check transfer to the local setting
Regulatory status, vendor validation and local suitability are related but distinct. Patient mix, recording practices, equipment, prevalence and the workflow around an output can affect performance or usefulness. The institution should understand how those features compare with the evidence supporting the product.
A concrete caution comes from a 2021 external evaluation of the Epic Sepsis Model at one US health system. The hospitalization-level area under the receiver-operating-characteristic curve was 0.63, below the developer's previously reported discrimination. This was a retrospective evaluation of a particular version and setting. It supports checking transferability, not a conclusion that all current sepsis AI fails or that this study alone proves clinical harm. External validation by Wong and colleagues.
Local evaluation need not mean that every hospital repeats every development study. Its scope should reflect the consequence of error, the strength of existing evidence, differences from the studied setting and uncertainty about the human workflow. A low-risk administrative use and a tool that changes urgent prioritization call for different questions and safeguards.
Relevant checks can include missing or delayed inputs, performance across clinically important groups, false-positive workload, missed cases and whether staff can interpret and act on the output. Small samples may make subgroup estimates uncertain; a report should disclose that uncertainty rather than declare equal performance from an underpowered comparison.
Assign work across the deployment lifecycle
The following is an illustrative governance structure. It is not a certification scheme or a complete statement of legal requirements.
- Before procurement. Evidence or decision to record: Problem, baseline, intended use, expected benefit and consequences of error Accountability to assign: Clinical sponsor and service owner
- Before a pilot. Evidence or decision to record: Regulatory and contractual fit, data flows, security review, evaluation plan and conditions of use Accountability to assign: Clinical, technical, information-governance and procurement leads
- Before routine deployment. Evidence or decision to record: Results against agreed criteria, staff preparation, escalation arrangements and workable fallback Accountability to assign: Named deployment decision-maker with patient-safety input
- During use. Evidence or decision to record: Performance, incidents, user feedback and material changes to inputs, workflow or software Accountability to assign: Monitoring owner and relevant clinical team
- When a concern arises. Evidence or decision to record: Investigation, temporary restrictions or suspension, continuity of care and criteria for restarting Accountability to assign: Named escalation lead with authority to act
Responsibility should include authority, resources and a substitute when the named person is unavailable. A committee can coordinate different expertise, but it should not leave urgent decisions dependent on a meeting that occurs weeks later.
The FUTURE-AI consensus guideline offers a lifecycle framework for trustworthy deployment, developed through an international expert process. It is a useful reference for organizing these questions. It is expert guidance, not evidence that a particular hospital's governance works or a substitute for applicable law. FUTURE-AI guideline, 2025.
Design oversight around the work people can actually do
A statement that a clinician “remains in the loop” does not tell us how an output is reviewed. The person needs relevant information, time, training and an escalation route. They also need a clear understanding of what the tool cannot establish. Confidence scores and explanations should be tested for usefulness; neither guarantees that a prediction is correct.
Evaluation should include the interface and pathway: whether a warning is visible, who receives it, what action is expected and how disagreement is handled. Excessively frequent alerts can impose a workload even when the software is operating as designed. The objective is not to maximize acceptance of recommendations, but to support appropriate responses.
Joint FDA, Health Canada and MHRA transparency principles emphasize information that helps users understand the device and the performance of the human–AI team. They provide design principles rather than a universal rule that a human click makes a system safe. Joint transparency principles, 2024.
Monitoring must lead to a response
A monitoring plan should identify metrics, review intervals, investigation triggers and the people who receive reports. The appropriate measures depend on the use: technical availability is different from calibration, diagnostic performance, workload or patient outcome. Feedback from clinicians can identify workflow problems that a model-performance dashboard misses.
A change in case mix or a data pipeline can justify investigation without proving that the model has deteriorated. Monitoring should distinguish a signal from an established problem and document what further assessment is needed. Vendor updates deserve an agreed notification and review process, particularly when they change inputs, output meaning or intended use.
Suspension also needs planning. Turning off a tool can disrupt a service that has come to depend on it. The hospital should know who can restrict use, how the clinical team is informed and what supported workflow maintains care. A fallback is a service arrangement, not merely an on/off switch.
Traceability, security and patients are part of the same service
Proportionate records can help reconstruct material AI influence: product version, configuration, outputs, delivery and relevant subsequent actions. Technical logs and clinical notes have different purposes. Access, retention and protection should follow applicable information-governance requirements; collecting every interaction forever is not a safety strategy.
Security review should examine data flows, permissions, remote support, update controls, incident reporting and loss of connectivity. Where a clinical pathway depends on a service, unavailability or corrupted inputs can become a patient-safety issue. Those risks should be considered in procurement and continuity planning.
Patients should receive clear information about a material AI role and a route for questions or concerns. What disclosure or consent is legally required depends on the context. Practical communication should explain the contribution and limits of the tool without portraying it as infallible or burying the patient in technical language.
Good governance produces decisions that can be examined: why a tool was introduced, what benefit was observed, what remains uncertain and what would change the decision to use it. That record is more informative than either the number of products purchased or the existence of an AI committee.
The related question of responsibility for a specific decision is examined in our clinical AI analysis.
