This is an analysis of clinical governance and published evidence. It does not determine liability in an individual case. Legal responsibility depends on the jurisdiction, the product, the professional roles, the applicable duties and the facts of what happened.
Influence matters before the final decision
Consider an illustrative deterioration alert. A system assigns a high score, the ward receives a notification, and a clinician examines the patient. Several different failures are possible: the input may be wrong, the model may be poorly calibrated for the population, the notification may arrive late, or the clinician may misunderstand what it predicts. An accurate score can also be ineffective if nobody is available to respond. Conversely, an absent alert must not become an assurance that the patient is safe unless the evidence supports that interpretation.
The example does not establish that any particular product causes these problems. It shows why evaluating a model in isolation is insufficient. The relevant unit is the system in use: software, people, data and the clinical pathway around them.
Nor is it accurate to say that medical AI never makes an automated clinical determination. The FDA's 2018 authorization of IDx-DR concerned a narrowly defined automated diabetic-retinopathy screening function. Its intended population, image requirements and output pathway matter. That authorization is evidence for a specific use, not permission for general autonomous diagnosis. FDA IDx-DR decision summary.
What studies can establish about reliance
A 2021 external validation of the Epic Sepsis Model examined 38,455 hospitalizations at one US health system. Its hospitalization-level discrimination was lower than previously reported by the developer: an area under the receiver-operating-characteristic curve of 0.63. This is a concrete reason to ask whether a model transfers to a new setting. It does not establish the performance of every sepsis model, every later version of this product, or the cause of harm in any individual patient's care. Wong and colleagues' external validation.
Another study asked whether access to a language model improved physicians' diagnostic reasoning on clinical vignettes. In that randomized study of 50 physicians, access did not significantly improve the measured reasoning score compared with conventional resources. The study concerned selected cases and a particular evaluation design, not outcomes in a hospital population. Its relevance is narrower: providing an apparently capable tool does not itself demonstrate that the human-and-tool combination performs better. Goh and colleagues, 2024.
These findings support a disciplined question rather than a verdict for or against AI: what evidence shows that this tool improves the intended task when used by these people in this workflow? Model accuracy, clinician performance, service efficiency and patient outcomes are different endpoints. A claim about one should not silently become a claim about all four.
Responsibility can be shared without becoming unassigned
It is tempting to answer every concern with “the clinician remains responsible.” That leaves out decisions clinicians may not control, including procurement, integration, model updates and staffing. The opposite claim, that a vendor or hospital is automatically liable whenever AI is involved, is equally unhelpful. Governance should assign work in advance; a court or other competent authority determines legal responsibility under the applicable rules.
The following is a proposed governance allocation, not a statement that the same legal duties apply everywhere.
- Vendor or developer. Practical accountability question: Are the intended use, supporting evidence, limitations, updates and technical failure modes described accurately enough for deployment and monitoring?
- Hospital or care organization. Practical accountability question: Who approves the use case, checks its fit, provides training and staffing, monitors performance, and maintains a safe fallback?
- Clinical team. Practical accountability question: How is output interpreted within the team's professional role, and how are discrepancies or unsafe behavior escalated?
- Safety, information and governance teams. Practical accountability question: Who investigates incidents, protects information, reviews material changes and confirms that corrective action happened?
- Regulator or professional authority. Practical accountability question: Which requirements apply to this product and use, and which reporting or oversight mechanisms are available?
A named owner needs authority and resources. Assigning a clinician responsibility for a system while withholding its limitations or preventing escalation creates a paper arrangement rather than a workable safeguard. A procurement contract also cannot replace the organization's assessment of what happens to patients when the product is used locally.
Human oversight needs a usable workflow
Automation bias describes inappropriate reliance on automated advice. It is a risk to examine, not an assumption that every clinician will defer or that every AI-assisted decision is unreliable. How an output is presented, how often it appears and how the organization expects staff to respond all deserve attention during evaluation.
Useful oversight means that the relevant person can understand the tool's purpose, recognize important limits, investigate an unexpected result and obtain help. An override button alone does not establish those conditions. Nor should a hospital make every disagreement burdensome enough that accepting the output becomes the easiest way to finish the task.
Explanations and confidence displays can assist interpretation, but they do not prove correctness. An explanation can be plausible while the output is wrong, and a confidence score is useful only if users understand what it represents and whether it is calibrated for the intended setting. Training should cover such limits, alongside the clinical pathway for responding to output.
The FDA, Health Canada and the UK's MHRA emphasize the performance of the human–AI team and information relevant to users in their joint transparency principles. These are useful design and evaluation principles; they are not a universal liability rule. Transparency for machine learning-enabled medical devices, 2024.
Traceability should support learning without overwhelming care
If an AI output materially influenced care, investigators may need to reconstruct what was available at the time. Relevant records can include the product and model version, its configuration, material outputs, delivery of alerts and subsequent action. A record of a generated output is different from evidence that a clinician saw, understood or relied on it.
This is an argument for proportionate traceability, not for recording every interaction indefinitely or adding an essay to every clinical note. Organizations need to distinguish the clinical record from technical logs and decide what information is necessary, who may access it, how long it is retained and how it is protected under applicable rules. Collecting more patient data than needed can create a separate risk.
Generated documentation introduces another responsibility: the organization should establish who checks a draft before it becomes an authoritative record, and how an omission or invented statement is corrected. A polished summary may conceal uncertainty in the underlying history. Making authorship, review and correction clear helps prevent that uncertainty from disappearing during handover.
Missing logs can make reconstruction substantially harder. They do not make accountability logically impossible: contemporaneous notes, other system records and testimony may still be relevant. The point of traceability is to improve the evidence available for safety learning and appropriate review.
Authorization is a starting condition, not a guarantee of local benefit
A product's regulatory status and its fit for a particular service answer different questions. The care organization still needs to understand the authorized or intended use, the population studied, input requirements and the consequences of errors in its own workflow. The depth of local evaluation should reflect the clinical risk and the evidence already available.
After deployment, a change in data, software or practice may justify reassessment. Monitoring should distinguish technical availability from clinical performance: a system can continue producing output even when its usefulness has changed. Someone must decide what signals require investigation, what changes require renewed evaluation and how care continues if use is suspended.
Patients belong in the accountability picture
Practical transparency means explaining a material AI role in language relevant to care: what the system contributes, what a clinician reviews and where the patient can ask questions or raise a concern. The exact disclosure and consent requirements depend on the setting and applicable law; they should not be invented from a general ethical preference.
The WHO's ethics guidance places autonomy, safety, transparency and accountability among the principles for health AI. Those principles support involving patients in governance and avoiding exaggerated claims that a system is objective or infallible. They do not settle whether a particular use is lawful or clinically appropriate. WHO ethics and governance of artificial intelligence for health.
Clinical AI can remove some work while creating new work in evaluation, monitoring and explanation. The strongest responsibility model makes that work visible before the tool becomes routine. It assigns decisions to people and organizations able to act on them, then preserves enough evidence to examine whether those arrangements worked.
For the institutional side of that responsibility, see our hospital AI procurement analysis.
