Automation bias and verification complexity: a systematic review

What Are Humans Doing in the Loop? Co-Reasoning and Practical Judgment When Using Machine Learning-Driven Decision Aids

Within the ethical debate on Machine Learning-driven decision support systems (ML_CDSS), notions such as "human in the loop" or "meaningful human control" are often cited as being necessary for ethical legitimacy. In addition, ethical principles usually serve as the major point of reference in ethical guidance documents, stating that conflicts between principles need to be weighed and balanced against each other. Starting from a neo-Kantian viewpoint inspired by Onora O'Neill, this article makes a concrete suggestion of how to interpret the role of the "human in the loop" and to overcome the perspective of rivaling ethical principles in the evaluation of AI in health care. We argue that patients should be perceived as "fellow workers" and epistemic partners in the interpretation of ML_CDSS outputs. We further highlight that a meaningful process of integrating (rather than weighing and balancing) ethical principles is most appropriate in the evaluation of medical AI.

Empowering nurses to champion Health equity & BE FAIR: Bias elimination for fair and responsible AI in healthcare

BACKGROUND

The concept of health equity by design encompasses a multifaceted approach that integrates actions aimed at eliminating biased, unjust, and correctable differences among groups of people as a fundamental element in the design of algorithms. As algorithmic tools are increasingly integrated into clinical practice at multiple levels, nurses are uniquely positioned to address challenges posed by the historical marginalization of minority groups and its intersections with the use of "big data" in healthcare settings; however, a coherent framework is needed to ensure that nurses receive appropriate training in these domains and are equipped to act effectively.

PURPOSE

We introduce the Bias Elimination for Fair AI in Healthcare (BE FAIR) framework, a comprehensive strategic approach that incorporates principles of health equity by design, for nurses to employ when seeking to mitigate bias and prevent discriminatory practices arising from the use of clinical algorithms in healthcare. By using examples from a "real-world" AI governance framework, we aim to initiate a wider discourse on equipping nurses with the skills needed to champion the BE FAIR initiative.

METHODS

Drawing on principles recently articulated by the Office of the National Coordinator for Health Information Technology, we conducted a critical examination of the concept of health equity by design. We also reviewed recent literature describing the risks of artificial intelligence (AI) technologies in healthcare as well as their potential for advancing health equity. Building on this context, we describe the BE FAIR framework, which has the potential to enable nurses to take a leadership role within health systems by implementing a governance structure to oversee the fairness and quality of clinical algorithms. We then examine leading frameworks for promoting health equity to inform the operationalization of BE FAIR within a local AI governance framework.

RESULTS

The application of the BE FAIR framework within the context of a working governance system for clinical AI technologies demonstrates how nurses can leverage their expertise to support the development and deployment of clinical algorithms, mitigating risks such as bias and promoting ethical, high-quality care powered by big data and AI technologies.

CONCLUSION AND RELEVANCE

As health systems learn how well-intentioned clinical algorithms can potentially perpetuate health disparities, we have an opportunity and an obligation to do better. New efforts empowering nurses to advocate for BE FAIR, involving them in AI governance, data collection methods, and the evaluation of tools intended to reduce bias, mark important steps in achieving equitable healthcare for all.

Evaluation and mitigation of the limitations of large language models in clinical decision-making

Clinical decision-making is one of the most impactful parts of a physician's responsibilities and stands to benefit greatly from artificial intelligence solutions and large language models (LLMs) in particular. However, while LLMs have achieved excellent performance on medical licensing exams, these tests fail to assess many skills necessary for deployment in a realistic clinical decision-making environment, including gathering information, adhering to guidelines, and integrating into clinical workflows. Here we have created a curated dataset based on the Medical Information Mart for Intensive Care database spanning 2,400 real patient cases and four common abdominal pathologies as well as a framework to simulate a realistic clinical setting. We show that current state-of-the-art LLMs do not accurately diagnose patients across all pathologies (performing significantly worse than physicians), follow neither diagnostic nor treatment guidelines, and cannot interpret laboratory results, thus posing a serious risk to the health of patients. Furthermore, we move beyond diagnostic accuracy and demonstrate that they cannot be easily integrated into existing workflows because they often fail to follow instructions and are sensitive to both the quantity and order of information. Overall, our analysis reveals that LLMs are currently not ready for autonomous clinical decision-making while providing a dataset and framework to guide future studies.

A review of incidents related to health information technology in Swedish healthcare to characterise system issues as a basis for improvement in clinical practice

This study examined health information technology-related incidents to characterise system issues as a basis for improvement in Swedish clinical practice. Incident reports were collected through interviews together with retrospectively collected incidents from voluntary incident databases, which were analysed using deductive and inductive approaches. Most themes pertained to system issues, such as functionality, design, and integration. Identified system issues were dominated by technical factors (74%), while human factors accounted for 26%. Over half of the incidents (55%) impacted on staff or the organisation, and the rest on patients - patient inconvenience (25%) and patient harm (20%). The findings indicate that it is vital to choose and commission suitable systems, design out "error-prone" features, ensure contingency plans are in place, implement clinical decision-support systems, and respond to incidents on time. Such strategies would improve the health information technology systems and Swedish clinical practice.

Machine Learning: A Potential Therapeutic Tool to Facilitate Neonatal Therapeutic Decision Making

Bacterial infection is one of the major causes of neonatal morbidity and mortality worldwide. Finding rapid and reliable methods for early recognition and diagnosis of bacterial infections and early individualization of antibacterial drug administration are essential to eradicate these infections and prevent serious complications. However, this is often difficult to perform due to non-specific clinical presentations, low accuracy of current diagnostic methods, and limited knowledge of neonatal pharmacokinetics. Although neonatal medicine has been relatively late to embrace the benefits of machine learning (ML), there have been some initial applications of ML for the early prediction of neonatal sepsis and individualization of antibiotics. This article provides a brief introduction to ML and discusses the current state of the art in diagnosing and treating neonatal bacterial infections, gaps, potential uses of ML, and future directions to address the limitations of current studies. Neonatal bacterial infections involve a combination of physiologic development, disease expression, and treatment response outcomes. To address this complex relationship, future models could consider appropriate ML algorithms to capture time series features while integrating influences from the host, microbes, and drugs to optimize antimicrobial drug use in neonates. All models require prospective clinical trials to validate their clinical utility before clinical use.

Large language models for science and medicine

Large language models (LLMs) are a type of machine learning model that learn statistical patterns over text, such as predicting the next words in a sequence of text. Both general purpose and task-specific LLMs have demonstrated potential across diverse applications. Science and medicine have many data types that are highly suitable for LLMs, such as scientific texts (publications, patents and textbooks), electronic medical records, large databases of DNA and protein sequences and chemical compounds. Carefully validated systems that can understand and reason across all these modalities may maximize benefits. Despite the inevitable limitations and caveats of any new technology and some uncertainties specific to LLMs, LLMs have the potential to be transformative in science and medicine.

Learning with an evolving medicine label: how artificial intelligence-based medication recommendation systems must adapt to changing medication labels

INTRODUCTION

Artificial intelligence or machine learning (AI/ML) based systems can help personalize prescribing decisions for individual patients. The recommendations of these clinical decision support systems must relate to the "label" of the medicines involved. The label of a medicine is an approved guide that indicates how to prescribe the drug in a safe and effective manner.

AREAS COVERED

The label for a medicine may evolve as new information on drug safety and effectiveness emerges, leading to the addition or removal of warnings, drug-drug interactions, or to permit new indications. However, the speed at which these updates are made to these AI/ML recommendation systems may be delayed and could influence the safety of prescribing decisions. This article explores the need to keep AI/ML tools 'in sync' with any label changes. Additionally, challenges relating to medicine availability and geographical suitability are discussed.

EXPERT OPINION

These considerations highlight the important role that pharmacoepidemiologists and drug safety professionals must play within the monitoring and use of these tools. Furthermore, these issues highlight the guiding role that regulators need to have in planning and oversight of these tools.

Artificial intelligence predictive analytics in heart failure: results of the pilot phase of a pragmatic randomized clinical trial

OBJECTIVES

We conducted an implementation planning process during the pilot phase of a pragmatic trial, which tests an intervention guided by artificial intelligence (AI) analytics sourced from noninvasive monitoring data in heart failure patients (LINK-HF2).

MATERIALS AND METHODS

A mixed-method analysis was conducted at 2 pilot sites. Interviews were conducted with 12 of 27 enrolled patients and with 13 participating clinicians. iPARIHS constructs were used for interview construction to identify workflow, communication patterns, and clinician's beliefs. Interviews were transcribed and analyzed using inductive coding protocols to identify key themes. Behavioral response data from the AI-generated notifications were collected.

RESULTS

Clinicians responded to notifications within 24 hours in 95% of instances, with 26.7% resulting in clinical action. Four implementation themes emerged: (1) High anticipatory expectations for reliable patient communications, reduced patient burden, and less proactive provider monitoring. (2) The AI notifications required a differential and tailored balance of trust and action advice related to role. (3) Clinic experience with other home-based programs influenced utilization. (4) Responding to notifications involved significant effort, including electronic health record (EHR) review, patient contact, and consultation with other clinicians.

DISCUSSION

Clinician's use of AI data is a function of beliefs regarding the trustworthiness and usefulness of the data, the degree of autonomy in professional roles, and the cognitive effort involved.

CONCLUSION

The implementation planning analysis guided development of strategies that addressed communication technology, patient education, and EHR integration to reduce clinician and patient burden in the subsequent main randomized phase of the trial. Our results provide important insights into the unique implications of implementing AI analytics into clinical workflow.

Diagnostic decisions of specialist optometrists exposed to ambiguous deep-learning outputs

Artificial intelligence (AI) has great potential in ophthalmology. We investigated how ambiguous outputs from an AI diagnostic support system (AI-DSS) affected diagnostic responses from optometrists when assessing cases of suspected retinal disease. Thirty optometrists (15 more experienced, 15 less) assessed 30 clinical cases. For ten, participants saw an optical coherence tomography (OCT) scan, basic clinical information and retinal photography ('no AI'). For another ten, they were also given AI-generated OCT-based probabilistic diagnoses ('AI diagnosis'); and for ten, both AI-diagnosis and AI-generated OCT segmentations ('AI diagnosis + segmentation') were provided. Cases were matched across the three types of presentation and were selected to include 40% ambiguous and 20% incorrect AI outputs. Optometrist diagnostic agreement with the predefined reference standard was lowest for 'AI diagnosis + segmentation' (204/300, 68%) compared to 'AI diagnosis' (224/300, 75% p = 0.010), and 'no Al' (242/300, 81%, p =  < 0.001). Agreement with AI diagnosis consistent with the reference standard decreased (174/210 vs 199/210, p = 0.003), but participants trusted the AI more (p = 0.029) with segmentations. Practitioner experience did not affect diagnostic responses (p = 0.24). More experienced participants were more confident (p = 0.012) and trusted the AI less (p = 0.038). Our findings also highlight issues around reference standard definition.

Human-artificial intelligence interaction in gastrointestinal endoscopy

The number and variety of applications of artificial intelligence (AI) in gastrointestinal (GI) endoscopy is growing rapidly. New technologies based on machine learning (ML) and convolutional neural networks (CNNs) are at various stages of development and deployment to assist patients and endoscopists in preparing for endoscopic procedures, in detection, diagnosis and classification of pathology during endoscopy and in confirmation of key performance indicators. Platforms based on ML and CNNs require regulatory approval as medical devices. Interactions between humans and the technologies we use are complex and are influenced by design, behavioural and psychological elements. Due to the substantial differences between AI and prior technologies, important differences may be expected in how we interact with advice from AI technologies. Human–AI interaction (HAII) may be optimised by developing AI algorithms to minimise false positives and designing platform interfaces to maximise usability. Human factors influencing HAII may include automation bias, alarm fatigue, algorithm aversion, learning effect and deskilling. Each of these areas merits further study in the specific setting of AI applications in GI endoscopy and professional societies should engage to ensure that sufficient emphasis is placed on human-centred design in development of new AI technologies.

The Unseen Hand: AI-Based Prescribing Decision Support Tools and the Evaluation of Drug Safety and Effectiveness

The use of artificial intelligence (AI)-based tools to guide prescribing decisions is full of promise and may enhance patient outcomes. These tools can perform actions such as choosing the 'safest' medication, choosing between competing medications, promoting de-prescribing or even predicting non-adherence. These tools can exist in a variety of formats; for example, they may be directly integrated into electronic medical records or they may exist in a stand-alone website accessible by a web browser. One potential impact of these tools is that they could manipulate our understanding of the benefit-risk of medicines in the real world. Currently, the benefit risk of approved medications is assessed according to carefully planned agreements covering spontaneous reporting systems and planned surveillance studies. But AI-based tools may limit or even block prescription to high-risk patients or prevent off-label use. The uptake and temporal availability of these tools may be uneven across healthcare systems and geographies, creating artefacts in data that are difficult to account for. It is also hard to estimate the 'true impact' that a tool had on a prescribing decision. International borders may also be highly porous to these tools, especially in cases where tools are available over the web. These tools already exist, and their use is likely to increase in the coming years. How they can be accounted for in benefit-risk decisions is yet to be seen.

The impact of AI errors in a human-in-the-loop process

Automated decision-making is becoming increasingly common in the public sector. As a result, political institutions recommend the presence of humans in these decision-making processes as a safeguard against potentially erroneous or biased algorithmic decisions. However, the scientific literature on human-in-the-loop performance is not conclusive about the benefits and risks of such human presence, nor does it clarify which aspects of this human-computer interaction may influence the final decision. In two experiments, we simulate an automated decision-making process in which participants judge multiple defendants in relation to various crimes, and we manipulate the time in which participants receive support from a supposed automated system with Artificial Intelligence (before or after they make their judgments). Our results show that human judgment is affected when participants receive incorrect algorithmic support, particularly when they receive it before providing their own judgment, resulting in reduced accuracy. The data and materials for these experiments are freely available at the Open Science Framework:  https://osf.io/b6p4z/ Experiment 2 was preregistered.

Challenges and Ethical Considerations to Successfully Implement Artificial Intelligence in Clinical Medicine and Neuroscience: a Narrative Review

This narrative review discusses how the safe and effective use of clinical artificial intelligence (AI) prediction tools requires recognition of the importance of human intelligence. Human intelligence, creativity, situational awareness, and professional knowledge, are required for successful implementation. The implementation of clinical AI prediction tools may change the workflow in medical practice resulting in new challenges and safety implications. Human understanding of how a clinical AI prediction tool performs in routine and exceptional situations is fundamental to successful implementation. Physicians must be involved in all aspects of the selection, implementation, and ongoing product monitoring of clinical AI prediction tools.

Norwegian radiologists' expectations of artificial intelligence in mammographic screening - A cross-sectional survey

PURPOSE

To explore Norwegian breast radiologists' expectations of adding artificial intelligence (AI) in the interpretation procedure of screening mammograms.

METHODS

All breast radiologists involved in interpretation of screening mammograms in BreastScreen Norway during 2021 and 2022 (n = 98) were invited to take part in this anonymous cross-sectional survey about use of AI in mammographic screening. The questionnaire included background information of the respondents, their expectations, considerations of biases, and ethical and social implications of implementing AI in screen reading. Data was collected digitally and analyzed using descriptive statistics.

RESULTS

The response rate was 61% (60/98), and 67% (40/60) of the respondents were women. Sixty percent (36/60) reported ≥10 years' experience in screen reading, while 82% (49/60) reported no or limited experience with AI in health care. Eighty-two percent of the respondents were positive to explore AI in the interpretation procedure in mammographic screening. When used as decision support, 68% (41/60) expected AI to increase the radiologists' sensitivity for cancer detection. As potential challenges, 55% (33/60) reported lack of trust in the AI system and 45% (27/60) reported discrepancy between radiologists and AI systems as possible challenges. The risk of automation bias was considered high among 47% (28/60). Reduced time spent reading mammograms was rated as a potential benefit by 70% (42/60).

CONCLUSION

The radiologists reported positive expectations of AI in the interpretation procedure of screening mammograms. Efforts to minimize the risk of automation bias and increase trust in the AI systems are important before and during future implementation of the tool.

Artificial intelligence suppression as a strategy to mitigate artificial intelligence automation bias

BACKGROUND

Incorporating artificial intelligence (AI) into clinics brings the risk of automation bias, which potentially misleads the clinician's decision-making. The purpose of this study was to propose a potential strategy to mitigate automation bias.

METHODS

This was a laboratory study with a randomized cross-over design. The diagnosis of anterior cruciate ligament (ACL) rupture, a common injury, on magnetic resonance imaging (MRI) was used as an example. Forty clinicians were invited to diagnose 200 ACLs with and without AI assistance. The AI's correcting and misleading (automation bias) effects on the clinicians' decision-making processes were analyzed. An ordinal logistic regression model was employed to predict the correcting and misleading probabilities of the AI. We further proposed an AI suppression strategy that retracted AI diagnoses with a higher misleading probability and provided AI diagnoses with a higher correcting probability.

RESULTS

The AI significantly increased clinicians' accuracy from 87.2%±13.1% to 96.4%±1.9% (P < .001). However, the clinicians' errors in the AI-assisted round were associated with automation bias, accounting for 45.5% of the total mistakes. The automation bias was found to affect clinicians of all levels of expertise. Using a logistic regression model, we identified an AI output zone with higher probability to generate misleading diagnoses. The proposed AI suppression strategy was estimated to decrease clinicians' automation bias by 41.7%.

CONCLUSION

Although AI improved clinicians' diagnostic performance, automation bias was a serious problem that should be addressed in clinical practice. The proposed AI suppression strategy is a practical method for decreasing automation bias.

Artificial intelligence in medicine: A comprehensive survey of medical doctor's perspectives in Portugal

Artificial Intelligence (AI) is increasingly influential across various sectors, including healthcare, with the potential to revolutionize clinical practice. However, risks associated with AI adoption in medicine have also been identified. Despite the general understanding that AI will impact healthcare, studies that assess the perceptions of medical doctors about AI use in medicine are still scarce. We set out to survey the medical doctors licensed to practice medicine in Portugal about the impact, advantages, and disadvantages of AI adoption in clinical practice. We designed an observational, descriptive, cross-sectional study with a quantitative approach and developed an online survey which addressed the following aspects: impact on healthcare quality of the extraction and processing of health data via AI; delegation of clinical procedures on AI tools; perception of the impact of AI in clinical practice; perceived advantages of using AI in clinical practice; perceived disadvantages of using AI in clinical practice and predisposition to adopt AI in professional activity. Our sample was also subject to demographic, professional and digital use and proficiency characterization. We obtained 1013 valid, fully answered questionnaires (sample representativeness of 99%, confidence level (p< 0.01), for the total universe of medical doctors licensed to practice in Portugal). Our results reveal that, in general terms, the medical community surveyed is optimistic about AI use in medicine and are predisposed to adopt it while still aware of some disadvantages and challenges to AI use in healthcare. Most medical doctors surveyed are also convinced that AI should be part of medical formation. These findings contribute to facilitating the professional integration of AI in medical practice in Portugal, aiding the seamless integration of AI into clinical workflows by leveraging its perceived strengths according to healthcare professionals. This study identifies challenges such as gaps in medical curricula, which hinder the adoption of AI applications due to inadequate digital health training. Due to high professional integration in the healthcare sector, particularly within the European Union, our results are also relevant for other jurisdictions and across diverse healthcare systems.

Artificial intelligence in clinical workflow processes in vascular surgery and beyond

In the past decade, artificial intelligence (AI)-based applications have exploded in health care. In cardiovascular disease, and vascular surgery specifically, AI tools such as machine learning, natural language processing, and deep neural networks have been applied to automatically detect underdiagnosed diseases, such as peripheral artery disease, abdominal aortic aneurysms, and atherosclerotic cardiovascular disease. In addition to disease detection and risk stratification, AI has been used to identify guideline-concordant statin therapy use and reasons for nonuse, which has important implications for population-based cardiovascular disease health. Although many studies highlight the potential applications of AI, few address true clinical workflow implementation of available AI-based tools. Specific examples, such as determination of optimal statin treatment based on individual patient risk factors and enhancement of intraoperative fluoroscopy and ultrasound imaging, demonstrate the potential promise of AI integration into clinical workflow. Many challenges to AI implementation in health care remain, including data interoperability, model bias and generalizability, prospective evaluation, privacy and security, and regulation. Multidisciplinary and multi-institutional collaboration, as well as adopting a framework for integration, will be critical for the successful implementation of AI tools into clinical practice.

Ethical Considerations of Using ChatGPT in Health Care

ChatGPT has promising applications in health care, but potential ethical issues need to be addressed proactively to prevent harm. ChatGPT presents potential ethical challenges from legal, humanistic, algorithmic, and informational perspectives. Legal ethics concerns arise from the unclear allocation of responsibility when patient harm occurs and from potential breaches of patient privacy due to data collection. Clear rules and legal boundaries are needed to properly allocate liability and protect users. Humanistic ethics concerns arise from the potential disruption of the physician-patient relationship, humanistic care, and issues of integrity. Overreliance on artificial intelligence (AI) can undermine compassion and erode trust. Transparency and disclosure of AI-generated content are critical to maintaining integrity. Algorithmic ethics raise concerns about algorithmic bias, responsibility, transparency and explainability, as well as validation and evaluation. Information ethics include data bias, validity, and effectiveness. Biased training data can lead to biased output, and overreliance on ChatGPT can reduce patient adherence and encourage self-diagnosis. Ensuring the accuracy, reliability, and validity of ChatGPT-generated content requires rigorous validation and ongoing updates based on clinical practice. To navigate the evolving ethical landscape of AI, AI in health care must adhere to the strictest ethical standards. Through comprehensive ethical guidelines, health care professionals can ensure the responsible use of ChatGPT, promote accurate and reliable information exchange, protect patient privacy, and empower patients to make informed decisions about their health care.

Automation in Contemporary Clinical Information Systems: a Survey of AI in Healthcare Settings

AIMS AND OBJECTIVES

To examine the nature and use of automation in contemporary clinical information systems by reviewing studies reporting the implementation and evaluation of artificial intelligence (AI) technologies in healthcare settings.

METHOD

PubMed/MEDLINE, Web of Science, EMBASE, the tables of contents of major informatics journals, and the bibliographies of articles were searched for studies reporting evaluation of AI in clinical settings from January 2021 to December 2022. We documented the clinical application areas and tasks supported, and the level of system autonomy. Reported effects on user experience, decision-making, care delivery and outcomes were summarised.

RESULTS

AI technologies are being applied in a wide variety of clinical areas. Most contemporary systems utilise deep learning, use routinely collected data, support diagnosis and triage, are assistive (requiring users to confirm or approve AI provided information or decisions), and are used by doctors in acute care settings in high-income nations. AI systems are integrated and used within existing clinical information systems including electronic medical records. There is limited support for One Health goals. Evaluation is largely based on quantitative methods measuring effects on decision-making.

CONCLUSION

AI systems are being implemented and evaluated in many clinical areas. There remain many opportunities to understand patterns of routine use and evaluate effects on decision-making, care delivery and patient outcomes using mixed-methods. Support for One Health including integrating data about environmental factors and social determinants needs further exploration.

Role of Artificial Intelligence in Global Surgery: A Review of Opportunities and Challenges

Global surgery broadly refers to a rapidly expanding multidisciplinary field concerned with providing better and equitable surgical care across international health systems. Global surgery initiatives primarily focus on capacity building, advocacy, education, research, and policy development in low- and middle-income countries (LMICs). The inadequate surgical, anesthetic, and obstetric care currently contributes to 18 million preventable deaths each year. Hence, there is a growing interest in the rapid growth of artificial intelligence (AI) that provides a distinctive opportunity to enhance surgical services in LMICs. AI modalities have been used for personalizing surgical education, automating administrative tasks, and developing realistic and cost-effective simulation-training programs with provisions for people with special needs. Furthermore, AI may assist with providing insights for governance, infrastructure development, and monitoring/predicting stock take or logistics failure that can help in strengthening global surgery pillars. Numerous AI-assisted telemedicine-based platforms have allowed healthcare professionals to virtually assist in complex surgeries that may help to improve surgical accessibility across LMICs. Challenges in implementing AI technology include the misrepresentation of minority populations in the datasets leading to discriminatory bias. Human hesitancy, employment uncertainty, automation bias, and role of confounding factors need to be further studied for equitable utilization of AI. With a focused and evidence-based approach, AI could help several LMICs overcome bureaucratic inefficiency and develop more efficient surgical systems.

A Review of Artificial Intelligence in the Rupture Risk Assessment of Intracranial Aneurysms: Applications and Challenges

Intracranial aneurysms (IAs) are highly prevalent in the population, and their rupture poses a significant risk of death or disability. However, the treatment of aneurysms, whether through interventional embolization or craniotomy clipping surgery, is not always safe and carries a certain proportion of morbidity and mortality. Therefore, early detection and prompt intervention of IAs with a high risk of rupture is of notable clinical significance. Moreover, accurately predicting aneurysms that are likely to remain stable can help avoid the risks and costs of over-intervention, which also has considerable social significance. Recent advances in artificial intelligence (AI) technology offer promising strategies to assist clinical trials. This review will discuss the state-of-the-art AI applications for assessing the rupture risk of IAs, with a focus on achievements, challenges, and potential opportunities.

The impact of AI suggestions on radiologists' decisions: a pilot study of explainability and attitudinal priming interventions in mammography examination

Various studies have shown that medical professionals are prone to follow the incorrect suggestions offered by algorithms, especially when they have limited inputs to interrogate and interpret such suggestions and when they have an attitude of relying on them. We examine the effect of correct and incorrect algorithmic suggestions on the diagnosis performance of radiologists when (1) they have no, partial, and extensive informational inputs for explaining the suggestions (study 1) and (2) they are primed to hold a positive, negative, ambivalent, or neutral attitude towards AI (study 2). Our analysis of 2760 decisions made by 92 radiologists conducting 15 mammography examinations shows that radiologists' diagnoses follow both incorrect and correct suggestions, despite variations in the explainability inputs and attitudinal priming interventions. We identify and explain various pathways through which radiologists navigate through the decision process and arrive at correct or incorrect decisions. Overall, the findings of both studies show the limited effect of using explainability inputs and attitudinal priming for overcoming the influence of (incorrect) algorithmic suggestions.

Impact of Different Artificial Intelligence User Interfaces on Lung Nodule and Mass Detection on Chest Radiographs

Purpose

To explore the impact of different user interfaces (UIs) for artificial intelligence (AI) outputs on radiologist performance and user preference in detecting lung nodules and masses on chest radiographs.

Materials and Methods

A retrospective paired-reader study with a 4-week washout period was used to evaluate three different AI UIs compared with no AI output. Ten radiologists (eight radiology attending physicians and two trainees) evaluated 140 chest radiographs (81 with histologically confirmed nodules and 59 confirmed as normal with CT), with either no AI or one of three UI outputs: (a) text-only, (b) combined AI confidence score and text, or (c) combined text, AI confidence score, and image overlay. Areas under the receiver operating characteristic curve were calculated to compare radiologist diagnostic performance with each UI with their diagnostic performance without AI. Radiologists reported their UI preference.

Results

The area under the receiver operating characteristic curve improved when radiologists used the text-only output compared with no AI (0.87 vs 0.82; P < .001). There was no difference in performance for the combined text and AI confidence score output compared with no AI (0.77 vs 0.82; P = .46) and for the combined text, AI confidence score, and image overlay output compared with no AI (0.80 vs 0.82; P = .66). Eight of the 10 radiologists (80%) preferred the combined text, AI confidence score, and image overlay output over the other two interfaces.

Conclusion

Text-only UI output significantly improved radiologist performance compared with no AI in the detection of lung nodules and masses on chest radiographs, but user preference did not correspond with user performance.Keywords: Artificial Intelligence, Chest Radiograph, Conventional Radiography, Lung Nodule, Mass Detection© RSNA, 2023.

The Evidence for Using Artificial Intelligence to Enhance Prostate Cancer MR Imaging

PURPOSE OF REVIEW

The purpose of this review is to summarize the current status of artificial intelligence applied to prostate cancer MR imaging.

RECENT FINDINGS

Artificial intelligence has been applied to prostate cancer MR imaging to improve its diagnostic accuracy and reproducibility of interpretation. Multiple models have been tested for gland segmentation and volume calculation, automated lesion detection, localization, and characterization, as well as prediction of tumor aggressiveness and tumor recurrence. Studies show, for example, that very robust automated gland segmentation and volume calculations can be achieved and that lesions can be detected and accurately characterized. Although results are promising, we should view these with caution. Most studies included a small sample of patients from a single institution and most models did not undergo proper external validation. More research is needed with larger and well-design studies for the development of reliable artificial intelligence tools.

Improving diagnostic decision support through deliberate reflection: a proposal

Digital decision support (DDS) is expected to play an important role in improving a physician's diagnostic performance and reducing the burden of diagnostic error. Studies with currently available DDS systems indicate that they lead to modest gains in diagnostic accuracy, and these systems are expected to evolve to become more effective and user-friendly in the future. In this position paper, we propose that a way towards this future is to rethink DDS systems based on deliberate reflection, a strategy by which physicians systematically review the clinical findings observed in a patient in the light of an initial diagnosis. Deliberate reflection has been demonstrated to improve diagnostic accuracy in several contexts. In this paper, we first describe the deliberate reflection strategy, including the crucial element that would make it useful in the interaction with a DDS system. We examine the nature of conventional DDS systems and their shortcomings. Finally, we propose what DDS based on deliberate reflection might look like, and consider why it would overcome downsides of conventional DDS.

A comprehensive survey of deep learning research on medical image analysis with focus on transfer learning

This survey aims to identify commonly used methods, datasets, future trends, knowledge gaps, constraints, and limitations in the field to provide an overview of current solutions used in medical image analysis in parallel with the rapid developments in transfer learning (TL). Unlike previous studies, this survey grouped the last five years of current studies for the period between January 2017 and February 2021 according to different anatomical regions and detailed the modality, medical task, TL method, source data, target data, and public or private datasets used in medical imaging. Also, it provides readers with detailed information on technical challenges, opportunities, and future research trends. In this way, an overview of recent developments is provided to help researchers to select the most effective and efficient methods and access widely used and publicly available medical datasets, research gaps, and limitations of the available literature.

Artificial Intelligence in Nuclear Medicine: Opportunities, Challenges, and Responsibilities Toward a Trustworthy Ecosystem

Trustworthiness is a core tenet of medicine. The patient-physician relationship is evolving from a dyad to a broader ecosystem of health care. With the emergence of artificial intelligence (AI) in medicine, the elements of trust must be revisited. We envision a road map for the establishment of trustworthy AI ecosystems in nuclear medicine. In this report, AI is contextualized in the history of technologic revolutions. Opportunities for AI applications in nuclear medicine related to diagnosis, therapy, and workflow efficiency, as well as emerging challenges and critical responsibilities, are discussed. Establishing and maintaining leadership in AI require a concerted effort to promote the rational and safe deployment of this innovative technology by engaging patients, nuclear medicine physicians, scientists, technologists, and referring providers, among other stakeholders, while protecting our patients and society. This strategic plan was prepared by the AI task force of the Society of Nuclear Medicine and Molecular Imaging.

Rams, hounds and white boxes: Investigating human-AI collaboration protocols in medical diagnosis

In this paper, we study human-AI collaboration protocols, a design-oriented construct aimed at establishing and evaluating how humans and AI can collaborate in cognitive tasks. We applied this construct in two user studies involving 12 specialist radiologists (the knee MRI study) and 44 ECG readers of varying expertise (the ECG study), who evaluated 240 and 20 cases, respectively, in different collaboration configurations. We confirm the utility of AI support but find that XAI can be associated with a "white-box paradox", producing a null or detrimental effect. We also find that the order of presentation matters: AI-first protocols are associated with higher diagnostic accuracy than human-first protocols, and with higher accuracy than both humans and AI alone. Our findings identify the best conditions for AI to augment human diagnostic skills, rather than trigger dysfunctional responses and cognitive biases that can undermine decision effectiveness.

Evidence synthesis, digital scribes, and translational challenges for artificial intelligence in healthcare

Healthcare has well-known challenges with safety, quality, and effectiveness, and many see artificial intelligence (AI) as essential to any solution. Emerging applications include the automated synthesis of best-practice research evidence including systematic reviews, which would ultimately see all clinical trial data published in a computational form for immediate synthesis. Digital scribes embed themselves in the process of care to detect, record, and summarize events and conversations for the electronic record. However, three persistent translational challenges must be addressed before AI is widely deployed. First, little effort is spent replicating AI trials, exposing patients to risks of methodological error and biases. Next, there is little reporting of patient harms from trials. Finally, AI built using machine learning may perform less effectively in different clinical settings.

Analysis of noise and bias errors in intelligence information systems

An intelligence information system (IIS) is a particular kind of information systems (IS) devoted to the analysis of intelligence relevant to national security. Professional and military intelligence analysts play a key role in this, but their judgments can be inconsistent, mainly due to noise and bias. The team-oriented aspects of the intelligence analysis process complicates the situation further. To enable analysts to achieve better judgments, the authors designed, implemented, and validated an innovative IIS for analyzing UK Military Signals Intelligence (SIGINT) data. The developed tool, the Team Information Decision Engine (TIDE), relies on an innovative preference learning method along with an aggregation procedure that permits combining scores by individual analysts into aggregated scores. This paper reports on a series of validation trials in which the performance of individual and team-oriented analysts was accessed with respect to their effectiveness and efficiency. Results show that the use of the developed tool enhanced the effectiveness and efficiency of intelligence analysis process at both individual and team levels.

How does the artificial intelligence-based image-assisted technique help physicians in diagnosis of pulmonary adenocarcinoma? A randomized controlled experiment of multicenter physicians in China

OBJECTIVE

Although artificial intelligence (AI) has achieved high levels of accuracy in the diagnosis of various diseases, its impact on physicians' decision-making performance in clinical practice is uncertain. This study aims to assess the impact of AI on the diagnostic performance of physicians with differing levels of self-efficacy under working conditions involving different time pressures.

MATERIALS AND METHODS

A 2 (independent diagnosis vs AI-assisted diagnosis) × 2 (no time pressure vs 2-minute time limit) randomized controlled experiment of multicenter physicians was conducted. Participants diagnosed 10 pulmonary adenocarcinoma cases and their diagnostic accuracy, sensitivity, and specificity were evaluated. Data analysis was performed using multilevel logistic regression.

RESULTS

One hundred and four radiologists from 102 hospitals completed the experiment. The results reveal (1) AI greatly increases physicians' diagnostic accuracy, either with or without time pressure; (2) when no time pressure, AI significantly improves physicians' diagnostic sensitivity but no significant change in specificity, while under time pressure, physicians' diagnostic sensitivity and specificity are both improved with the aid of AI; (3) when no time pressure, physicians with low self-efficacy benefit from AI assistance thus improving diagnostic accuracy but those with high self-efficacy do not, whereas physicians with low and high levels of self-efficacy both benefit from AI under time pressure.

DISCUSSION

This study is one of the first to provide real-world evidence regarding the impact of AI on physicians' decision-making performance, taking into account 2 boundary factors: clinical time pressure and physicians' self-efficacy.

CONCLUSION

AI-assisted diagnosis should be prioritized for physicians working under time pressure or with low self-efficacy.

Implementation of machine learning in the clinic: challenges and lessons in prospective deployment from the System for High Intensity EvaLuation During Radiation Therapy (SHIELD-RT) randomized controlled study

Background

Artificial intelligence (AI) and machine learning (ML) have resulted in significant enthusiasm for their promise in healthcare. Despite this, prospective randomized controlled trials and successful clinical implementation remain limited. One clinical application of ML is mitigation of the increased risk for acute care during outpatient cancer therapy. We previously reported the results of the System for High Intensity EvaLuation During Radiation Therapy (SHIELD-RT) study (NCT04277650), which was a prospective, randomized quality improvement study demonstrating that ML based on electronic health record (EHR) data can direct supplemental clinical evaluations and reduce the rate of acute care during cancer radiotherapy with and without chemotherapy. The objective of this study is to report the workflow and operational challenges encountered during ML implementation on the SHIELD-RT study.

Results

Data extraction and manual review steps in the workflow represented significant time commitments for implementation of clinical ML on a prospective, randomized study. Barriers include limited data availability through the standard clinical workflow and commercial products, the need to aggregate data from multiple sources, and logistical challenges from altering the standard clinical workflow to deliver adaptive care.

Conclusions

The SHIELD-RT study was an early randomized controlled study which enabled assessment of barriers to clinical ML implementation, specifically those which leverage the EHR. These challenges build on a growing body of literature and may provide lessons for future healthcare ML adoption.

Trial registration: NCT04277650. Registered 20 February 2020. Retrospectively registered quality improvement study.

Effect of risk, expectancy, and trust on clinicians' intent to use an artificial intelligence system -- Blood Utilization Calculator

A gap exists between the capabilities of artificial intelligence (AI) technologies in healthcare and the extent to which clinicians are willing to adopt these systems. Our study addressed this gap by leveraging 'expectancy-value theory' and 'modified extended unified theory of acceptance and use of technology' to understand why clinicians may be willing or unwilling to adopt AI systems. The study looked at the 'expectancy,' 'trust,' and 'perceptions' of clinicians related to their intention of using an AI-based decision support system known as the Blood Utilization Calculator (BUC). The study used purposive sampling to recruit BUC users and administered a validated online survey from a large hospital system in the Midwest in 2021. The findings captured the significant effect of 'perceived risk' (negatively) and 'expectancy' (positively) on clinicians' 'trust' in BUC. 'Trust' was also found to mediate the relationship of 'perceived risk' and 'expectancy' with the 'intent to use BUC.' The study's findings established pathways for future research and have implications on factors influencing BUC use.

Validation and algorithmic audit of a deep learning system for the detection of proximal femoral fractures in patients in the emergency department: a diagnostic accuracy study

BACKGROUND

Proximal femoral fractures are an important clinical and public health issue associated with substantial morbidity and early mortality. Artificial intelligence might offer improved diagnostic accuracy for these fractures, but typical approaches to testing of artificial intelligence models can underestimate the risks of artificial intelligence-based diagnostic systems.

METHODS

We present a preclinical evaluation of a deep learning model intended to detect proximal femoral fractures in frontal x-ray films in emergency department patients, trained on films from the Royal Adelaide Hospital (Adelaide, SA, Australia). This evaluation included a reader study comparing the performance of the model against five radiologists (three musculoskeletal specialists and two general radiologists) on a dataset of 200 fracture cases and 200 non-fractures (also from the Royal Adelaide Hospital), an external validation study using a dataset obtained from Stanford University Medical Center, CA, USA, and an algorithmic audit to detect any unusual or unexpected model behaviour.

FINDINGS

In the reader study, the area under the receiver operating characteristic curve (AUC) for the performance of the deep learning model was 0·994 (95% CI 0·988-0·999) compared with an AUC of 0·969 (0·960-0·978) for the five radiologists. This strong model performance was maintained on external validation, with an AUC of 0·980 (0·931-1·000). However, the preclinical evaluation identified barriers to safe deployment, including a substantial shift in the model operating point on external validation and an increased error rate on cases with abnormal bones (eg, Paget's disease).

INTERPRETATION

The model outperformed the radiologists tested and maintained performance on external validation, but showed several unexpected limitations during further testing. Thorough preclinical evaluation of artificial intelligence models, including algorithmic auditing, can reveal unexpected and potentially harmful behaviour even in high-performance artificial intelligence systems, which can inform future clinical testing and deployment decisions.

FUNDING

None.

The medical algorithmic audit

Artificial intelligence systems for health care, like any other medical device, have the potential to fail. However, specific qualities of artificial intelligence systems, such as the tendency to learn spurious correlates in training data, poor generalisability to new deployment settings, and a paucity of reliable explainability mechanisms, mean they can yield unpredictable errors that might be entirely missed without proactive investigation. We propose a medical algorithmic audit framework that guides the auditor through a process of considering potential algorithmic errors in the context of a clinical task, mapping the components that might contribute to the occurrence of errors, and anticipating their potential consequences. We suggest several approaches for testing algorithmic errors, including exploratory error analysis, subgroup testing, and adversarial testing, and provide examples from our own work and previous studies. The medical algorithmic audit is a tool that can be used to better understand the weaknesses of an artificial intelligence system and put in place mechanisms to mitigate their impact. We propose that safety monitoring and medical algorithmic auditing should be a joint responsibility between users and developers, and encourage the use of feedback mechanisms between these groups to promote learning and maintain safe deployment of artificial intelligence systems.

Artificial Intelligence in Emergency Medicine: Benefits, Risks, and Recommendations

BACKGROUND

Artificial intelligence (AI) can be described as the use of computers to perform tasks that formerly required human cognition. The American Medical Association prefers the term 'augmented intelligence' over 'artificial intelligence' to emphasize the assistive role of computers in enhancing physician skills as opposed to replacing them. The integration of AI into emergency medicine, and clinical practice at large, has increased in recent years, and that trend is likely to continue.

DISCUSSION

AI has demonstrated substantial potential benefit for physicians and patients. These benefits are transforming the therapeutic relationship from the traditional physician-patient dyad into a triadic doctor-patient-machine relationship. New AI technologies, however, require careful vetting, legal standards, patient safeguards, and provider education. Emergency physicians (EPs) should recognize the limits and risks of AI as well as its potential benefits.

CONCLUSIONS

EPs must learn to partner with, not capitulate to, AI. AI has proven to be superior to, or on a par with, certain physician skills, such as interpreting radiographs and making diagnoses based on visual cues, such as skin cancer. AI can provide cognitive assistance, but EPs must interpret AI results within the clinical context of individual patients. They must also advocate for patient confidentiality, professional liability coverage, and the essential role of specialty-trained EPs.

Tragic Choices and the Virtue of Techno-Responsibility Gaps

There is a concern that the widespread deployment of autonomous machines will open up a number of 'responsibility gaps' throughout society. Various articulations of such techno-responsibility gaps have been proposed over the years, along with several potential solutions. Most of these solutions focus on 'plugging' or 'dissolving' the gaps. This paper offers an alternative perspective. It argues that techno-responsibility gaps are, sometimes, to be welcomed and that one of the advantages of autonomous machines is that they enable us to embrace certain kinds of responsibility gap. The argument is based on the idea that human morality is often tragic. We frequently confront situations in which competing moral considerations pull in different directions and it is impossible to perfectly balance these considerations. This heightens the burden of responsibility associated with our choices. We cope with the tragedy of moral choice in different ways. Sometimes we delude ourselves into thinking the choices we make were not tragic (illusionism); sometimes we delegate the tragic choice to others (delegation); sometimes we make the choice ourselves and bear the psychological consequences (responsibilisation). Each of these strategies has its benefits and costs. One potential advantage of autonomous machines is that they enable a reduced cost form of delegation. However, we only gain the advantage of this reduced cost if we accept that some techno-responsibility gaps are virtuous.

Automated detection of pulmonary embolism from CT-angiograms using deep learning

Background

The aim of this study was to develop and evaluate a deep neural network model in the automated detection of pulmonary embolism (PE) from computed tomography pulmonary angiograms (CTPAs) using only weakly labelled training data.

Methods

We developed a deep neural network model consisting of two parts: a convolutional neural network architecture called InceptionResNet V2 and a long-short term memory network to process whole CTPA stacks as sequences of slices. Two versions of the model were created using either chest X-rays (Model A) or natural images (Model B) as pre-training data. We retrospectively collected 600 CTPAs to use in training and validation and 200 CTPAs to use in testing. CTPAs were annotated only with binary labels on both stack- and slice-based levels. Performance of the models was evaluated with ROC and precision–recall curves, specificity, sensitivity, accuracy, as well as positive and negative predictive values.

Results

Both models performed well on both stack- and slice-based levels. On the stack-based level, Model A reached specificity and sensitivity of 93.5% and 86.6%, respectively, outperforming Model B slightly (specificity 90.7% and sensitivity 83.5%). However, the difference between their ROC AUC scores was not statistically significant (0.94 vs 0.91, p = 0.07).

Conclusions

We show that a deep learning model trained with a relatively small, weakly annotated dataset can achieve excellent performance results in detecting PE from CTPAs.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12880-022-00763-z.

Artificial Intelligence Applications in Intracranial Aneurysm: Achievements, Challenges and Opportunities

Intracranial aneurysms present in about 3% of the general population and the number of detected aneurysms is continuously rising with the advances in imaging techniques. Intracranial aneurysm rupture carries a high risk of death or permanent disabilities; therefore assessment of the intracranial aneurysm along the entire course is of great clinical importance. Given the outstanding performance of artificial intelligence (AI) in image-based tasks, many AI-based applications have emerged in recent years for the assessment of intracranial aneurysms. In this review we will summarize the state-of-the-art of AI applications in intracranial aneurysms, emphasizing the achievements, and exploring the challenges. We will also discuss the future prospects and potential opportunities. This article provides an updated view of the AI applications in intracranial aneurysms and may act as a basis for guiding the related future works.

Advanced Machine Learning Tools to Monitor Biomarkers of Dysphagia: A Wearable Sensor Proof-of-Concept Study

Introduction

Difficulty swallowing (dysphagia) occurs frequently in patients with neurological disorders and can lead to aspiration, choking, and malnutrition. Dysphagia is typically diagnosed using costly, invasive imaging procedures or subjective, qualitative bedside examinations. Wearable sensors are a promising alternative to noninvasively and objectively measure physiological signals relevant to swallowing. An ongoing challenge with this approach is consolidating these complex signals into sensitive, clinically meaningful metrics of swallowing performance. To address this gap, we propose 2 novel, digital monitoring tools to evaluate swallows using wearable sensor data and machine learning.

Methods

Biometric swallowing and respiration signals from wearable, mechano-acoustic sensors were compared between patients with poststroke dysphagia and nondysphagic controls while swallowing foods and liquids of different consistencies, in accordance with the Mann Assessment of Swallowing Ability (MASA). Two machine learning approaches were developed to (1) classify the severity of impairment for each swallow, with model confidence ratings for transparent clinical decision support, and (2) compute a similarity measure of each swallow to nondysphagic performance. Task-specific models were trained using swallow kinematics and respiratory features from 505 swallows (321 from patients and 184 from controls).

Results

These models provide sensitive metrics to gauge impairment on a per-swallow basis. Both approaches demonstrate intrasubject swallow variability and patient-specific changes which were not captured by the MASA alone. Sensor measures encoding respiratory-swallow coordination were important features relating to dysphagia presence and severity. Puree swallows exhibited greater differences from controls than saliva swallows or liquid sips (p < 0.037).

Discussion

Developing interpretable tools is critical to optimize the clinical utility of novel, sensor-based measurement techniques. The proof-of-concept models proposed here provide concrete, communicable evidence to track dysphagia recovery over time. With refined training schemes and real-world validation, these tools can be deployed to automatically measure and monitor swallowing in the clinic and community for patients across the impairment spectrum.

The false hope of current approaches to explainable artificial intelligence in health care

The black-box nature of current artificial intelligence (AI) has caused some to question whether AI must be explainable to be used in high-stakes scenarios such as medicine. It has been argued that explainable AI will engender trust with the health-care workforce, provide transparency into the AI decision making process, and potentially mitigate various kinds of bias. In this Viewpoint, we argue that this argument represents a false hope for explainable AI and that current explainability methods are unlikely to achieve these goals for patient-level decision support. We provide an overview of current explainability techniques and highlight how various failure cases can cause problems for decision making for individual patients. In the absence of suitable explainability methods, we advocate for rigorous internal and external validation of AI models as a more direct means of achieving the goals often associated with explainability, and we caution against having explainability be a requirement for clinically deployed models.

Artificial intelligence sepsis prediction algorithm learns to say "I don't know"

Sepsis is a leading cause of morbidity and mortality worldwide. Early identification of sepsis is important as it allows timely administration of potentially life-saving resuscitation and antimicrobial therapy. We present COMPOSER (COnformal Multidimensional Prediction Of SEpsis Risk), a deep learning model for the early prediction of sepsis, specifically designed to reduce false alarms by detecting unfamiliar patients/situations arising from erroneous data, missingness, distributional shift and data drifts. COMPOSER flags these unfamiliar cases as indeterminate rather than making spurious predictions. Six patient cohorts (515,720 patients) curated from two healthcare systems in the United States across intensive care units (ICU) and emergency departments (ED) were used to train and externally and temporally validate this model. In a sequential prediction setting, COMPOSER achieved a consistently high area under the curve (AUC) (ICU: 0.925-0.953; ED: 0.938-0.945). Out of over 6 million prediction windows roughly 20% and 8% were identified as indeterminate amongst non-septic and septic patients, respectively. COMPOSER provided early warning within a clinically actionable timeframe (ICU: 12.2 [3.2 22.8] and ED: 2.1 [0.8 4.5] hours prior to first antibiotics order) across all six cohorts, thus allowing for identification and prioritization of patients at high risk for sepsis.