Browse the corpus

Regulators and developers favour human oversight models because placing the clinician between the algorithm and the patient seems to resolve both procedural and legal concerns. Human oversight becomes the safety backstop. In regulatory frameworks, such as those from the FDA and the EU, professional oversight is treated as a safeguard, resting on the assumption that qualified clinicians can independently appraise algorithmic recommendations and exercise sound judgment.1 2 This assumption may hold for well defined, explicit tools where the underlying logic is easily understandable. But extending it to opaque AI systems that continuously learn or produce high volumes of data and probabilistic alerts risks turning clinicians into guarantors of safety without giving them meaningful access to how models are built, validated, or are performing across settings.5 6 It also misreads clinical care. Clinicians care for multiple patients at once, making hundreds of decisions each day through poorly designed systems that often ignore basic human-computer interaction principles. The experience with electronic health record alerts is instructive: systems designed to enhance safety instead produced alert fatigue, as clinicians became desensitised to a flood of low value warnings.7 Expecting human vigilance to scale to a continuous stream of AI outputs risks repeating this failure.

on principles. The experience with electronic health record alerts is instructive: systems designed to enhance safety instead produced alert fatigue, as clinicians became desensitised to a flood of low value warnings.7 Expecting human vigilance to scale to a continuous stream of AI outputs risks repeating this failure. Clinicians routinely interpret automated outputs such as laboratory results or vital sign monitors without undue burden. But these tools are typically standardised, externally validated, and regulated within established device frameworks. By contrast, many predictive models enter practice through pathways that shift elements of contextual validation to local health systems, increasing the risk that bias is undetected. For example, in one study AI models that misinterpreted scanner artefacts reduced clinician accuracy at diagnosing pneumonia.18 AI outputs also differ from other automated outputs in a crucial way: whereas diagnostic tests provide measured data for interpretation, many AI systems present synthesised probabilistic judgments such as percentage risk of sepsis or high probability of malignancy. These can anchor decisions and make rejection less likely, fostering automation bias.

In clinical practice, the conditions required for effective human oversight— time, understanding, and agency—are rarely met. Clinicians often lack the time, and sometimes the motivation, to supervise tools introduced to serve institutional priorities rather than clinical needs. Generative AI reduces documentation time9 but produces fluent, plausible rationales that mask factual errors, thereby increasing the cognitive effort required to check and override its outputs.6 Reviewing probabilities demands a different kind of cognitive labour from assessing laboratory results—one that diverts presence and attention away from the patient. This “surveillance labour” can overwhelm clinicians, even when AI tools are designed to ease their work.10 Predictive AI systems tuned for high sensitivity further compound the problem, generating alert fatigue that can lead clinicians to accept outputs without review—a rational adaptation to excessive alarms.4 11 Even with deep clinical expertise, most clinicians lack the technical and statistical literacy to critically appraise AI outputs.4 Visual explanation tools, such as heatmaps on clinical images, can reinforce the illusion that algorithms reason as clinicians do. This false sense of understanding increases the likelihood of uncritical reliance.12 In one randomised study using an image based diagnostic model, participants performed worse after seeing an incorrect AI suggestion than with no AI at all, as the model’s “opinion” overrode their own informed judgment.8 Clinicians need not lose their reasoning skills to become dependent on AI outputs.13

cal reliance.12 In one randomised study using an image based diagnostic model, participants performed worse after seeing an incorrect AI suggestion than with no AI at all, as the model’s “opinion” overrode their own informed judgment.8 Clinicians need not lose their reasoning skills to become dependent on AI outputs.13 Making clinicians responsible for outcomes yet unable to meaningfully interrogate the system may create a new form of moral distress. Clinicians face liability both for overriding a correct AI verdict (cast as arrogance) and for following an incorrect one (cast as abdication). They become the “moral crumple zone” of the AI apparatus, absorbing the legal and reputational impact of system failure while shielding developers and implementers.4 10 This is not to argue that human oversight is always ineffective. AI can safely support care within well defined, verifiable tasks such as in radiation oncology contouring or automated insulin delivery.14 15 These systems operate within bounded loops, often featuring clear anatomical or physiological ground truths, where human oversight is feasible and meaningful. By contrast, applying oversight models to opaque, adaptive, or high volume predictive systems stretches human capacity beyond its limits. Different classes of AI therefore require different governance architectures: structured oversight may suffice for bounded tools, but more complex systems demand stronger upstream safeguards rather than reliance on the vigilance of clinicians or patients.8 11 12

The clinician’s primary role is to co-create pertinent, meaningful care with patients, not to be a technical validator of AI outputs. Poorly implemented AI systems risk accelerating the industrialisation of medicine, dehumanising both patients and clinicians. This is the predictable consequence of a technology first, human second model. Safe AI implementation requires abandoning the “loop” metaphor, strengthening instead the therapeutic alliance (fig 1, right).16 Because this relationship is the foundation of care, AI should operate in the background, supporting, not mediating it. Operationally, this means AI functions as an assistant during procedures or as an adviser during clinical encounters—that is, preserving human-first cognition with AI offering asynchronous second opinions, activated only after clinicians have formed an initial judgment (table 1). To mitigate the legal exposure and cognitive risks inherent in the loop model, a new foundation built on this principle, rests on three pillars (box 1). Contrasting AI governance models Shift primary legal responsibility from individual clinicians to developers and implementing organisations Promote overarching policies that give vendors accountability for algorithmic errors, enabling a safer procurement process for every institution Establish interdisciplinary AI governance committees empowered to evaluate, recalibrate, or withdraw clinical tools Expand patient agency by ensuring AI outputs are accessible and transparently discussed in clinical encounters, supporting patients and clinicians as partners in reasoning

Promote overarching policies that give vendors accountability for algorithmic errors, enabling a safer procurement process for every institution Establish interdisciplinary AI governance committees empowered to evaluate, recalibrate, or withdraw clinical tools Expand patient agency by ensuring AI outputs are accessible and transparently discussed in clinical encounters, supporting patients and clinicians as partners in reasoning Tailor oversight to clinical context, relying, for instance, on specialised system level safeguards rather than individual vigilance for integrated AI assistance in applications that are not patient facing Train clinicians to use and interpret AI with clear understanding of its limits, preserving clinical judgment and avoiding deskilling Implement continuous, automated auditing to detect performance drift and bias after deployment

Tailor oversight to clinical context, relying, for instance, on specialised system level safeguards rather than individual vigilance for integrated AI assistance in applications that are not patient facing Train clinicians to use and interpret AI with clear understanding of its limits, preserving clinical judgment and avoiding deskilling Implement continuous, automated auditing to detect performance drift and bias after deployment For AI to function as an assistant or adviser, clinicians and patients must be able to rely on system level oversight rather than individual vigilance.15 17 The key distinction is not whether AI is queried or pushes outputs, but where responsibility for safety assurance lies. In the oversight models, risk mitigation occurs at the bedside, with clinicians expected to detect and correct system failures in real time. In assistant or advisory models, safety is established upstream through rigorous validation, verification, and continuous organisational monitoring. Clinicians can then use these systems as they do other regulated tools, exercising professional judgment in patient care, while institutions and developers remain accountable for the system’s safety, validated performance, and ongoing oversight.

gorous validation, verification, and continuous organisational monitoring. Clinicians can then use these systems as they do other regulated tools, exercising professional judgment in patient care, while institutions and developers remain accountable for the system’s safety, validated performance, and ongoing oversight. This shift requires governance structures that make validation, procurement standards, and continuous monitoring explicit conditions of deployment. As smaller facilities may lack legal leverage, overarching policy should establish baseline vendor liability for defects so that healthcare organisations can use their purchasing power to select systems that do not depend on bedside vigilance for safe use. Since perfect upstream validation is impossible because the dataset used for training will differ from the applied population, automated monitoring of performance drift should be in real time, triggering review when safety thresholds are breached.18 Current broad regulatory clearance is often insufficient when a model is deployed in a different setting. Institutions must therefore test and certify systems for their local environment before deployment, rather than expecting clinicians to appraise opaque algorithms during clinical encounters. Clinician wellbeing should not be the hidden cost of innovation.

en insufficient when a model is deployed in a different setting. Institutions must therefore test and certify systems for their local environment before deployment, rather than expecting clinicians to appraise opaque algorithms during clinical encounters. Clinician wellbeing should not be the hidden cost of innovation. Governance must begin with clarity of purpose: what problem of care the AI system addresses and how. Developers should stress test prototypes in real clinical settings and establish continuous feedback loops so that end user experience informs recalibration. A good example of this is the use of “silent testing” in radiology. For instance, a new radiology triage algorithm can be deployed to process live scans in the background without altering workflows. This allows developers to use clinician feedback (such as reviewing the model’s false positive alerts) to recalibrate the tool before it goes live.

the use of “silent testing” in radiology. For instance, a new radiology triage algorithm can be deployed to process live scans in the background without altering workflows. This allows developers to use clinician feedback (such as reviewing the model’s false positive alerts) to recalibrate the tool before it goes live. Interdisciplinary AI governance committees, including patients and clinicians, should be empowered to approve, recalibrate, or withdraw inadequate tools. These committees would ideally build directly on existing hospital quality and safety structures, functioning much like a pharmacy and therapeutics or medical device committee and managing an “algorithmic formulary.” They would review local validation data to approve a tool, use continuous auditing to adjust its sensitivity thresholds (recalibrate), and hold the authority to disable the tool if ongoing monitoring detects harmful performance drift (withdraw). Governance falters when care is treated solely as a data problem.19

lary.” They would review local validation data to approve a tool, use continuous auditing to adjust its sensitivity thresholds (recalibrate), and hold the authority to disable the tool if ongoing monitoring detects harmful performance drift (withdraw). Governance falters when care is treated solely as a data problem.19 At the point of care, AI advisory systems should not render patients passive recipients of algorithmic decisions—a new paternalism. Instead, they should support explicit discussion of AI outputs, their validity, and their limits, enabling patients and clinicians to co-create care plans. Clinicians remain responsible for appropriate use in individual cases, which requires training in safe and effective AI use, akin to pharmacology, while preserving independent judgment. However, the presence of an AI adviser does not guarantee better deliberation. Studies show that trainees who defer prematurely to external advisers under time pressure are more likely to adopt a paternalistic style, conveying recommendations rather than co-creating care with patients.20 This caution applies equally to algorithmic advisers. The therapeutic alliance is strengthened when clinicians can take responsibility for the reasoning that informs care while collaborating with patients as partners in deliberation and decision making.

ying recommendations rather than co-creating care with patients.20 This caution applies equally to algorithmic advisers. The therapeutic alliance is strengthened when clinicians can take responsibility for the reasoning that informs care while collaborating with patients as partners in deliberation and decision making. Let’s return to our hypothetical thyroid ultrasound scenario to illustrate our proposal. Deployed as an advisory tool, the AI system offers an opinion only when the clinician seeks it, and the developer and the health system remain accountable for its safety, validated performance, and ongoing monitoring. Clinicians are trained to understand its probabilistic limits and potential for drift, and can review the AI’s discordant assessment with the patient (“I think this is a benign finding, but our system suggests it may be malignant”), reflect on it in light of their expertise (“on closer review, I can see why the AI raises concern”), and partner with the patient to arrive at a sensible plan (“let’s proceed with a biopsy”). The decision emerges from shared deliberation, grounded in clinical judgment, informed but not dictated by the algorithm.

ant”), reflect on it in light of their expertise (“on closer review, I can see why the AI raises concern”), and partner with the patient to arrive at a sensible plan (“let’s proceed with a biopsy”). The decision emerges from shared deliberation, grounded in clinical judgment, informed but not dictated by the algorithm. Policymakers should be cautious in treating clinician oversight models as the primary safeguard for AI safety. Expecting clinicians to compensate for inadequately governed systems overlooks the opaque and probabilistic nature of many models, the well documented effects of automation bias, and the realities of clinical work. When positioned as the central safety mechanism, loop approaches concentrate legal and moral exposure at the bedside while upstream technical responsibilities remain diffuse. Developers and implementing organisations should instead retain clear accountability for the performance of the systems they deploy, ensuring these tools genuinely support the relational work of care. This approach offers a more credible and sustainable path for integrating AI into everyday medicine. Regulators and health systems increasingly rely on human oversight at the point of use as a primary safeguard for AI Such systems place accountability and liability for AI systems on clinicians rather than on those who design and implement them Opaque models, limited training, and realities of clinical work make it unrealistic to expect clinicians to meaningfully oversee and override AI outputs and act as a failsafe

Regulators and health systems increasingly rely on human oversight at the point of use as a primary safeguard for AI Such systems place accountability and liability for AI systems on clinicians rather than on those who design and implement them Opaque models, limited training, and realities of clinical work make it unrealistic to expect clinicians to meaningfully oversee and override AI outputs and act as a failsafe AI governance should move towards developer accountability, supported by rigorous pre-market and post-market safety evaluation, and implementer liability Such governance would allow clinicians and patients to focus on care, using AI systems as an advisory tool rather than a risk to manage

Shift primary legal responsibility from individual clinicians to developers and implementing organisations Promote overarching policies that give vendors accountability for algorithmic errors, enabling a safer procurement process for every institution

Establish interdisciplinary AI governance committees empowered to evaluate, recalibrate, or withdraw clinical tools Expand patient agency by ensuring AI outputs are accessible and transparently discussed in clinical encounters, supporting patients and clinicians as partners in reasoning Tailor oversight to clinical context, relying, for instance, on specialised system level safeguards rather than individual vigilance for integrated AI assistance in applications that are not patient facing

Train clinicians to use and interpret AI with clear understanding of its limits, preserving clinical judgment and avoiding deskilling Implement continuous, automated auditing to detect performance drift and bias after deployment