The Deception of Certainty: how Non-Interpretable Machine Learning Outcomes Challenge the Epistemic Authority of Physicians. A deliberative-relational Approach

Clouds on the horizon: clinical decision support systems, the control problem, and physician-patient dialogue

Artificial intelligence-based clinical decision support systems have a potential to improve clinical practice, but they may have a negative impact on the physician-patient dialogue, because of the control problem. Physician-patient dialogue depends on human qualities such as compassion, trust, and empathy, which are shared by both parties. These qualities are necessary for the parties to reach a shared understanding -the merging of horizons- about clinical decisions. The patient attends the clinical encounter not only with a malfunctioning body, but also with an 'unhomelike' experience of illness that is related to a world of values and meanings, a life-world. Making wise individual decisions in accordance with the patient's life-world requires not only scientific analysis of causal relationships, but also listening with empathy to the patient's concerns. For a decision to be made, clinical information should be interpreted considering the patient's life-world. This side of clinical practice is not a job for computers, and they cannot be final decision-makers. On the other hand, in the control problem users blindly accept system output because of over-reliance, rather than evaluating it with their own judgement. This means over-reliant parties leave their place in the dialogue to the system. In this case, the dialogue may be disrupted and mutual trust may be lost. Therefore, it is necessary to design decision support systems to avoid the control problem and to limit their use when this is not possible, in order to protect the physician-patient dialogue.

Clinicians' roles and necessary levels of understanding in the use of artificial intelligence: A qualitative interview study with German medical students

BACKGROUND

Artificial intelligence-driven Clinical Decision Support Systems (AI-CDSS) are being increasingly introduced into various domains of health care for diagnostic, prognostic, therapeutic and other purposes. A significant part of the discourse on ethically appropriate conditions relate to the levels of understanding and explicability needed for ensuring responsible clinical decision-making when using AI-CDSS. Empirical evidence on stakeholders' viewpoints on these issues is scarce so far. The present study complements the empirical-ethical body of research by, on the one hand, investigating the requirements for understanding and explicability in depth with regard to the rationale behind them. On the other hand, it surveys medical students at the end of their studies as stakeholders, of whom little data is available so far, but for whom AI-CDSS will be an important part of their medical practice.

METHODS

Fifteen semi-structured qualitative interviews (each lasting an average of 56 min) were conducted with German medical students to investigate their perspectives and attitudes on the use of AI-CDSS. The problem-centred interviews draw on two hypothetical case vignettes of AI-CDSS employed in nephrology and surgery. Interviewees' perceptions and convictions of their own clinical role and responsibilities in dealing with AI-CDSS were elicited as well as viewpoints on explicability as well as the necessary level of understanding and competencies needed on the clinicians' side. The qualitative data were analysed according to key principles of qualitative content analysis (Kuckartz).

RESULTS

In response to the central question about the necessary understanding of AI-CDSS tools and the emergence of their outputs as well as the reasons for the requirements placed on them, two types of argumentation could be differentiated inductively from the interviewees' statements: the first type, the clinician as a systemic trustee (or "the one relying"), highlights that there needs to be empirical evidence and adequate approval processes that guarantee minimised harm and a clinical benefit from the employment of an AI-CDSS. Based on proof of these requirements, the use of an AI-CDSS would be appropriate, as according to "the one relying", clinicians should choose those measures that statistically cause the least harm. The second type, the clinician as an individual expert (or "the one controlling"), sets higher prerequisites that go beyond ensuring empirical evidence and adequate approval processes. These higher prerequisites relate to the clinician's necessary level of competence and understanding of how a specific AI-CDSS works and how to use it properly in order to evaluate its outputs and to mitigate potential risks for the individual patient. Both types are unified in their high esteem of evidence-based clinical practice and the need to communicate with the patient on the use of medical AI. However, the interviewees' different conceptions of the clinician's role and responsibilities cause them to have different requirements regarding the clinician's understanding and explicability of an AI-CDSS beyond the proof of benefit.

CONCLUSIONS

The study results highlight two different types among (future) clinicians regarding their view of the necessary levels of understanding and competence. These findings should inform the debate on appropriate training programmes and professional standards (e.g. clinical practice guidelines) that enable the safe and effective clinical employment of AI-CDSS in various clinical fields. While current approaches search for appropriate minimum requirements of the necessary understanding and competence, the differences between (future) clinicians in terms of their information and understanding needs described here can lead to more differentiated approaches to solutions.

The ethical requirement of explainability for AI-DSS in healthcare: a systematic review of reasons

BACKGROUND

Despite continuous performance improvements, especially in clinical contexts, a major challenge of Artificial Intelligence based Decision Support Systems (AI-DSS) remains their degree of epistemic opacity. The conditions of and the solutions for the justified use of the occasionally unexplainable technology in healthcare are an active field of research. In March 2024, the European Union agreed upon the Artificial Intelligence Act (AIA), requiring medical AI-DSS to be ad-hoc explainable or to use post-hoc explainability methods. The ethical debate does not seem to settle on this requirement yet. This systematic review aims to outline and categorize the positions and arguments in the ethical debate.

METHODS

We conducted a literature search on PubMed, BASE, and Scopus for English-speaking scientific peer-reviewed publications from 2016 to 2024. The inclusion criterion was to give explicit requirements of explainability for AI-DSS in healthcare and reason for it. Non-domain-specific documents, as well as surveys, reviews, and meta-analyses were excluded. The ethical requirements for explainability outlined in the documents were qualitatively analyzed with respect to arguments for the requirement of explainability and the required level of explainability.

RESULTS

The literature search resulted in 1662 documents; 44 documents were included in the review after eligibility screening of the remaining full texts. Our analysis showed that 17 records argue in favor of the requirement of explainable AI methods (xAI) or ad-hoc explainable models, providing 9 categories of arguments. The other 27 records argued against a general requirement, providing 11 categories of arguments. Also, we found that 14 works advocate the need for context-dependent levels of explainability, as opposed to 30 documents, arguing for context-independent, absolute standards.

CONCLUSIONS

The systematic review of reasons shows no clear agreement on the requirement of post-hoc explainability methods or ad-hoc explainable models for AI-DSS in healthcare. The arguments found in the debate were referenced and responded to from different perspectives, demonstrating an interactive discourse. Policymakers and researchers should watch the development of the debate closely. Conversely, ethicists should be well informed by empirical and technical research, given the frequency of advancements in the field.

When time is of the essence: ethical reconsideration of XAI in time-sensitive environments

The objective of explainable artificial intelligence systems designed for clinical decision support (XAI-CDSS) is to enhance physicians' diagnostic performance, confidence and trust through the implementation of interpretable methods, thus providing for a superior epistemic positioning, a robust foundation for critical reflection and trustworthiness in times of heightened technological dependence. However, recent studies have revealed shortcomings in achieving these goals, questioning the widespread endorsement of XAI by medical professionals, ethicists and policy-makers alike. Based on a surgical use case, this article challenges generalising calls for XAI-CDSS and emphasises the significance of time-sensitive clinical environments which frequently preclude adequate consideration of system explanations. Therefore, XAI-CDSS may not be able to meet expectations of augmenting clinical decision-making in specific circumstances where time is of the essence. This article, by employing a principled ethical balancing methodology, highlights several fallacies associated with XAI deployment in time-sensitive clinical situations and recommends XAI endorsement only where scientific evidence or stakeholder assessments do not contradict such deployment in specific target settings.

Impact of personalized risk scores on shared decision making in left ventricular assist device implantation: Findings from a qualitative study

OBJECTIVE

To assess stakeholders' perspectives on integrating personalized risk scores (PRS) into left ventricular assist device (LVAD) implantation decisions and how these perspectives might impact shared decision making (SDM).

METHODS

We conducted 40 in-depth interviews with physicians, nurse coordinators, patients, and caregivers about integrating PRS into LVAD implantation decisions. A codebook was developed to identify thematic patterns, and quotations were consolidated for analysis. We used Thematic Content Analysis in MAXQDA software to identify themes by abstracting relevant quotes.

RESULTS

Clinicians had varying preferences regarding PRS integration into LVAD decision making, while patients and caregivers preferred real-time discussions about PRS with their physicians. Physicians voiced concerns about time constraints and suggested delegating PRS discussions to advanced practice providers or nurse coordinators.

CONCLUSIONS

Integrating PRS information into LVAD decision aids presents both opportunities and challenges for SDM. Given variable preferences among clinicians and patients, clinicians should elicit patients' desired role in the decision-making process. Addressing time constraints and ensuring patient-centered care will be crucial for optimizing SDM. Practice implications Clinicians should elicit patient preferences for PRS information disclosure and address challenges, such as time constraints and delegation of PRS discussions to other team members.

Estimating biological age from retinal imaging: a scoping review

BACKGROUND/AIMS

The emerging concept of retinal age, a biomarker derived from retinal images, holds promise in estimating biological age. The retinal age gap (RAG) represents the difference between retinal age and chronological age, which serves as an indicator of deviations from normal ageing. This scoping review aims to collate studies on retinal age to determine its potential clinical utility and to identify knowledge gaps for future research.

METHODS

Using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses checklist, eligible non-review, human studies were identified, selected and appraised. PubMed, Scopus, SciELO, PsycINFO, Google Scholar, Cochrane, CINAHL, Africa Wide EBSCO, MedRxiv and BioRxiv databases were searched to identify literature pertaining to retinal age, the RAG and their associations. No restrictions were imposed on publication date.

RESULTS

Thirteen articles published between 2022 and 2023 were analysed, revealing four models capable of determining biological age from retinal images. Three models, 'Retinal Age', 'EyeAge' and a 'convolutional network-based model', achieved comparable mean absolute errors: 3.55, 3.30 and 3.97, respectively. A fourth model, 'RetiAGE', predicting the probability of being older than 65 years, also demonstrated strong predictive ability with respect to clinical outcomes. In the models identified, a higher predicted RAG demonstrated an association with negative occurrences, notably mortality and cardiovascular health outcomes.

CONCLUSION

This review highlights the potential clinical application of retinal age and RAG, emphasising the need for further research to establish their generalisability for clinical use, particularly in neuropsychiatry. The identified models showcase promising accuracy in estimating biological age, suggesting its viability for evaluating health status.

Impacts of Clinical Decision Support Systems on the Relationship, Communication, and Shared Decision-Making Between Health Care Professionals and Patients: Multistakeholder Interview Study

BACKGROUND

Clinical decision support systems (CDSSs) are increasingly being introduced into various domains of health care. Little is known so far about the impact of such systems on the health care professional-patient relationship, and there is a lack of agreement about whether and how patients should be informed about the use of CDSSs.

OBJECTIVE

This study aims to explore, in an empirically informed manner, the potential implications for the health care professional-patient relationship and to underline the importance of this relationship when using CDSSs for both patients and future professionals.

METHODS

Using a methodological triangulation, 15 medical students and 12 trainee nurses were interviewed in semistructured interviews and 18 patients were involved in focus groups between April 2021 and April 2022. All participants came from Germany. Three examples of CDSSs covering different areas of health care (ie, surgery, nephrology, and intensive home care) were used as stimuli in the study to identify similarities and differences regarding the use of CDSSs in different fields of application. The interview and focus group transcripts were analyzed using a structured qualitative content analysis.

RESULTS

From the interviews and focus groups analyzed, three topics were identified that interdependently address the interactions between patients and health care professionals: (1) CDSSs and their impact on the roles of and requirements for health care professionals, (2) CDSSs and their impact on the relationship between health care professionals and patients (including communication requirements for shared decision-making), and (3) stakeholders' expectations for patient education and information about CDSSs and their use.

CONCLUSIONS

The results indicate that using CDSSs could restructure established power and decision-making relationships between (future) health care professionals and patients. In addition, respondents expected that the use of CDSSs would involve more communication, so they anticipated an increased time commitment. The results shed new light on the existing discourse by demonstrating that the anticipated impact of CDSSs on the health care professional-patient relationship appears to stem less from the function of a CDSS and more from its integration in the relationship. Therefore, the anticipated effects on the relationship between health care professionals and patients could be specifically addressed in patient information about the use of CDSSs.

Challenges of artificial intelligence in medicine and dermatology

Artificial intelligence (AI) in medicine and dermatology brings additional challenges related to bias, transparency, ethics, security, and inequality. Bias in AI algorithms can arise from biased training data or decision-making processes, leading to disparities in health care outcomes. Addressing bias requires careful examination of the data used to train AI models and implementation of strategies to mitigate bias during algorithm development. Transparency is another critical challenge, as AI systems often operate as black boxes, making it difficult to understand how decisions are reached. Ensuring transparency in AI algorithms is vital to gaining trust from both patients and health care providers. Ethical considerations arise when using AI in health care, including issues such as informed consent, privacy, and the responsibility for the decisions made by AI systems. It is essential to establish clear guidelines and frameworks that govern the ethical use of AI, including maintaining patient autonomy and protecting sensitive health information. Security is a significant concern in AI systems, as they rely on vast amounts of sensitive patient data. Protecting these data from unauthorized access, breaches, or malicious attacks is paramount to maintaining patient privacy and trust in AI technologies. Lastly, the potential for inequality arises if AI technologies are not accessible to all populations, leading to a digital divide in health care. Efforts should be made to ensure that AI solutions are affordable, accessible, and tailored to the needs of diverse communities, mitigating the risk of exacerbating existing health care disparities. Addressing these challenges is crucial for AI's responsible and equitable integration in medicine and dermatology.

Physician's autonomy in the face of AI support: walking the ethical tightrope

The introduction of AI support tools raises questions about the normative orientation of medical practice and the need to rethink its basic concepts. One of these concepts that is central to the discussion is the physician’s autonomy and its appropriateness in the face of high-powered AI applications. In this essay, a differentiation of the physician’s autonomy is made on the basis of a conceptual analysis. It is argued that the physician’s decision-making autonomy is a purposeful autonomy. The physician’s decision-making autonomy is fundamentally anchored in the medical ethos for the purpose to promote the patient’s health and well-being and to prevent him or her from harm. It follows from this purposefulness that the physician’s autonomy is not to be protected for its own sake, but only insofar as it serves this end better than alternative means. We argue that today, given existing limitations of AI support tools, physicians still need physician’s decision-making autonomy. For the possibility of physicians to exercise decision-making autonomy in the face of AI support, we elaborate three conditions: (1) sufficient information about AI support and its statements, (2) sufficient competencies to integrate AI statements into clinical decision-making, and (3) a context of voluntariness that allows, in justified cases, deviations from AI support. If the physician should fulfill his or her moral obligation to promote the health and well-being of the patient, then the use of AI should be designed in such a way that it promotes or at least maintains the physician’s decision-making autonomy.

Responsibility and decision-making authority in using clinical decision support systems: an empirical-ethical exploration of German prospective professionals' preferences and concerns

Machine learning-driven clinical decision support systems (ML-CDSSs) seem impressively promising for future routine and emergency care. However, reflection on their clinical implementation reveals a wide array of ethical challenges. The preferences, concerns and expectations of professional stakeholders remain largely unexplored. Empirical research, however, may help to clarify the conceptual debate and its aspects in terms of their relevance for clinical practice. This study explores, from an ethical point of view, future healthcare professionals' attitudes to potential changes of responsibility and decision-making authority when using ML-CDSS. Twenty-seven semistructured interviews were conducted with German medical students and nursing trainees. The data were analysed based on qualitative content analysis according to Kuckartz. Interviewees' reflections are presented under three themes the interviewees describe as closely related: (self-)attribution of responsibility, decision-making authority and need of (professional) experience. The results illustrate the conceptual interconnectedness of professional responsibility and its structural and epistemic preconditions to be able to fulfil clinicians' responsibility in a meaningful manner. The study also sheds light on the four relata of responsibility understood as a relational concept. The article closes with concrete suggestions for the ethically sound clinical implementation of ML-CDSS.

[Factors influencing the implementation of AI-based decision support systems for antibiotic prescription in hospitals: a qualitative analysis from the perspective of health professionals]

BACKGROUND

Decision support systems based on artificial intelligence might optimize antibiotic prescribing in hospitals and prevent the development of antimicrobial resistance. The aim of this study was to identify impeding and facilitating factors for successful implementation from the perspective of health professionals.

METHODS

Problem-centered individual interviews were conducted with health professionals working in hospitals. Data evaluation was based on the structured qualitative content analysis according Kuckartz.

RESULTS

Attitudes of health professionals were presented along the Human-Organization -Technology-fit model. Technological and organizational themes were the most important factors for system implementation. Especially, compatibility with existing systems and user-friendliness were seen to play a major role in successful implementation. Additionally, the training of potential users and the technical equipment of the organization were considered essential. Finally, the importance of promoting technical skills of potential users in the long term and creating trust in the benefits of the system were highlighted.

CONCLUSION

The identified factors provide a basis for prioritizing and quantifying needs and attitudes in a next step. It becomes clear that, beside technological factors, attention to context-specific and user-related conditions are of fundamental importance to ensure successful implementation and system trust in the long term.