A framework for the oversight and local deployment of safe and high-quality prediction models

Expectations of healthcare AI and the role of trust: understanding patient views on how AI will impact cost, access, and patient-provider relationships

OBJECTIVES

Although efforts to effectively govern AI continue to develop, relatively little work has been done to systematically measure and include patient perspectives or expectations of AI in governance. This analysis is designed to understand patient expectations of healthcare AI.

MATERIALS AND METHODS

Cross-sectional nationally representative survey of US adults fielded from June to July of 2023. A total of 2039 participants completed the survey and cross-sectional population weights were applied to produce national estimates.

RESULTS

Among US adults, 19.55% expect AI to improve their relationship with their doctor, while 19.4% expect it to increase affordability and 30.28% expect it will improve their access to care. Trust in providers and the healthcare system are positively associated with expectations of AI when controlling for demographic factors, general attitudes toward technology, and other healthcare-related variables.

DISCUSSION

US adults generally have low expectations of benefit from AI in healthcare, but those with higher trust in their providers and health systems are more likely to expect to benefit from AI.

CONCLUSION

Trust and provider relationships should be key considerations for health systems as they create their AI governance processes and communicate with patients about AI tools. Evidence of patient benefit should be prioritized to preserve or promote trust.

Advancing laryngology through artificial intelligence: a comprehensive review of implementation frameworks and strategies

PURPOSE OF REVIEW

This review aims to explore the integration of artificial intelligence (AI) in laryngology, with specific focus on the barriers preventing translation from pilot studies into routine clinical practice and strategies for successful implementation.

RECENT FINDINGS

Laryngology has seen an increasing number of pilot and proof-of-concept studies demonstrating AI's ability to enhance diagnostics, treatment planning, and patient outcomes. Despite these advancements, few tools have been successfully adopted in clinical settings. Effective implementation requires the application of established implementation science frameworks early in the design phase. Additional factors required for the successful integration of AI applications include addressing specific clinical needs, fostering diverse and interdisciplinary teams, and ensuring scalability without compromising model performance. Governance, epistemic, and ethical considerations must also be continuously incorporated throughout the project lifecycle to ensure the safe, responsible, and equitable use of AI technologies.

SUMMARY

While AI hold significant promise for advancing laryngology, its implementation in clinical practice remains limited. Achieving meaningful integration will require a shift toward practical solutions that prioritize clinicians' and patients' needs, usability, sustainability, and alignment with clinical workflows.

The Role of Artificial Intelligence in Supporting the Core Mission of Nursing

Artificial intelligence (AI) has the potential to revolutionize nursing practice and care delivery by streamlining workflows, enhancing patient insights, and reducing cognitive burden. However, leaders must recognize that the adoption of AI introduces both promise and uncertainty. Nurse leaders must navigate the tension between driving innovation and addressing concerns about ethical implications, reliability, and the need to preserve high-quality, person-centered care. This article examines how leaders can thoughtfully integrate AI to support the core mission of nursing: to protect, promote, and optimize health for patients and communities.

Ambulatory Surgery Ensemble: Predicting Adult and Pediatric Same-Day Surgery Cases Across Specialties

Objective

To develop an ensemble model using case-posting data to predict which patients could be discharged on the day of surgery.

Background

Few models have predicted which surgeries are appropriate for day cases. Increasing the ratio of ambulatory surgeries can decrease costs and inpatient bed utilization while improving resource utilization.

Methods

Adult and pediatric patients undergoing elective surgery with any surgical specialty in a multisite academic health system from January 2021 to December 2023 were included in this retrospective study. We used surgical case data available at the time of case posting and created 3 gradient-boosting decision tree classification models to predict case length (CL) less than 6 hours, postoperative length of stay (LOS) less than 6 hours, and home discharge disposition (DD). The models were used to develop an ambulatory surgery ensemble (ASE) model to predict same-day surgery (SDS) cases.

Results

The ASE achieved an area under the receiver operating characteristic curve of 0.95 and an average precision of 0.96. In total, 139,593 cases were included, 48,464 of which were in 2023 and were used for model validation. These methods identified that up to 20% of inpatient cases could be moved to SDS and identified which specialties, procedures, and surgeons had the most opportunity to transition cases.

Conclusions

An ensemble model can predict CL, LOS, and DD for elective cases across multiple services and locations at the time of case posting. While limited in its inclusion of patient factors, this model can systematically facilitate clinical operations such as strategic planning, surgical block time, and case scheduling.

The urgency of centering safety-net organizations in AI governance

Although robust AI governance requires the engagement of diverse stakeholders across the artificial intelligence (AI) ecosystem, the US safety net has largely been excluded from this kind of collaboration. Without a reorientation of the AI governance agenda, marginalized patients will disproportionately bear the risks of AI in the US healthcare system. To prevent this replication of digital inequity and an organizational digital divide, we suggest specific next steps for diverse stakeholders to progress toward more equitable policy and practice.

Checklist Approach to Developing and Implementing AI in Clinical Settings: Instrument Development Study

Background

The integration of artificial intelligence (AI) in health care settings demands a nuanced approach that considers both technical performance and sociotechnical factors.

Objective

This study aimed to develop a checklist that addresses the sociotechnical aspects of AI deployment in health care and provides a structured, holistic guide for teams involved in the life cycle of AI systems.

Methods

A literature synthesis identified 20 relevant studies, forming the foundation for the Clinical AI Sociotechnical Framework checklist. A modified Delphi study was then conducted with 35 global health care professionals. Participants assessed the checklist's relevance across 4 stages: "Planning," "Design," "Development," and "Proposed Implementation." A consensus threshold of 80% was established for each item. IQRs and Cronbach α were calculated to assess agreement and reliability.

Results

The initial checklist had 45 questions. Following participant feedback, the checklist was refined to 34 items, and a final round saw 100% consensus on all items (mean score >0.8, IQR 0). Based on the outcome of the Delphi study, a final checklist was outlined, with 1 more question added to make 35 questions in total.

Conclusions

The Clinical AI Sociotechnical Framework checklist provides a comprehensive, structured approach to developing and implementing AI in clinical settings, addressing technical and social factors critical for adoption and success. This checklist is a practical tool that aligns AI development with real-world clinical needs, aiming to enhance patient outcomes and integrate smoothly into health care workflows.

Launching the Trustworthy and Responsible AI Network (TRAIN): A Consortium to Facilitate Safe and Effective AI Adoption

This Viewpoint lays out the genesis of Trustworthy and Responsible AI Network (TRAIN) and strategies for ensuring effective and safe use of AI throughout the US health care system.

The bias algorithm: how AI in healthcare exacerbates ethnic and racial disparities - a scoping review

This scoping review examined racial and ethnic bias in artificial intelligence health algorithms (AIHA), the role of stakeholders in oversight, and the consequences of AIHA for health equity. Using the PRISMA-ScR guidelines, databases were searched between 2020 and 2024 using the terms racial and ethnic bias in health algorithms resulting in a final sample of 23 sources. Suggestions for how to mitigate algorithmic bias were compiled and evaluated, roles played by stakeholders were identified, and governance and stewardship plans for AIHA were examined. While AIHA represent a significant breakthrough in predictive analytics and treatment optimization, regularly outperforming humans in diagnostic precision and accuracy, they also present serious challenges to patient privacy, data security, institutional transparency, and health equity. Evidence from extant sources including those in this review showed that AIHA carry the potential to perpetuate health inequities. While the current study considered AIHA in the US, the use of AIHA carries implications for global health equity.

In Silico Evaluation of Algorithm-Based Clinical Decision Support Systems: Protocol for a Scoping Review

BACKGROUND

Integrating algorithm-based clinical decision support (CDS) systems poses significant challenges in evaluating their actual clinical value. Such CDS systems are traditionally assessed via controlled but resource-intensive clinical trials.

OBJECTIVE

This paper presents a review protocol for preimplementation in silico evaluation methods to enable broadened impact analysis under simulated environments before clinical trials.

METHODS

We propose a scoping review protocol that follows an enhanced Arksey and O'Malley framework and PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) guidelines to investigate the scope and research gaps in the in silico evaluation of algorithm-based CDS models-specifically CDS decision-making end points and objectives, evaluation metrics used, and simulation paradigms used to assess potential impacts. The databases searched are PubMed, Embase, CINAHL, PsycINFO, Cochrane, IEEEXplore, Web of Science, and arXiv. A 2-stage screening process identified pertinent articles. The information extracted from articles was iteratively refined. The review will use thematic, trend, and descriptive analyses to meet scoping aims.

RESULTS

We conducted an automated search of the databases above in May 2023, with most title and abstract screenings completed by November 2023 and full-text screening extended from December 2023 to May 2024. Concurrent charting and full-text analysis were carried out, with the final analysis and manuscript preparation set for completion in July 2024. Publication of the review results is targeted from July 2024 to February 2025. As of April 2024, a total of 21 articles have been selected following a 2-stage screening process; these will proceed to data extraction and analysis.

CONCLUSIONS

We refined our data extraction strategy through a collaborative, multidisciplinary approach, planning to analyze results using thematic analyses to identify approaches to in silico evaluation. Anticipated findings aim to contribute to developing a unified in silico evaluation framework adaptable to various clinical workflows, detailing clinical decision-making characteristics, impact measures, and reusability of methods. The study's findings will be published and presented in forums combining artificial intelligence and machine learning, clinical decision-making, and health technology impact analysis. Ultimately, we aim to bridge the development-deployment gap through in silico evaluation-based potential impact assessments.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/63875.

The TRIPOD-LLM reporting guideline for studies using large language models

Large language models (LLMs) are rapidly being adopted in healthcare, necessitating standardized reporting guidelines. We present transparent reporting of a multivariable model for individual prognosis or diagnosis (TRIPOD)-LLM, an extension of the TRIPOD + artificial intelligence statement, addressing the unique challenges of LLMs in biomedical applications. TRIPOD-LLM provides a comprehensive checklist of 19 main items and 50 subitems, covering key aspects from title to discussion. The guidelines introduce a modular format accommodating various LLM research designs and tasks, with 14 main items and 32 subitems applicable across all categories. Developed through an expedited Delphi process and expert consensus, TRIPOD-LLM emphasizes transparency, human oversight and task-specific performance reporting. We also introduce an interactive website ( https://tripod-llm.vercel.app/ ) facilitating easy guideline completion and PDF generation for submission. As a living document, TRIPOD-LLM will evolve with the field, aiming to enhance the quality, reproducibility and clinical applicability of LLM research in healthcare through comprehensive reporting.

Empowering nurses to champion Health equity & BE FAIR: Bias elimination for fair and responsible AI in healthcare

BACKGROUND

The concept of health equity by design encompasses a multifaceted approach that integrates actions aimed at eliminating biased, unjust, and correctable differences among groups of people as a fundamental element in the design of algorithms. As algorithmic tools are increasingly integrated into clinical practice at multiple levels, nurses are uniquely positioned to address challenges posed by the historical marginalization of minority groups and its intersections with the use of "big data" in healthcare settings; however, a coherent framework is needed to ensure that nurses receive appropriate training in these domains and are equipped to act effectively.

PURPOSE

We introduce the Bias Elimination for Fair AI in Healthcare (BE FAIR) framework, a comprehensive strategic approach that incorporates principles of health equity by design, for nurses to employ when seeking to mitigate bias and prevent discriminatory practices arising from the use of clinical algorithms in healthcare. By using examples from a "real-world" AI governance framework, we aim to initiate a wider discourse on equipping nurses with the skills needed to champion the BE FAIR initiative.

METHODS

Drawing on principles recently articulated by the Office of the National Coordinator for Health Information Technology, we conducted a critical examination of the concept of health equity by design. We also reviewed recent literature describing the risks of artificial intelligence (AI) technologies in healthcare as well as their potential for advancing health equity. Building on this context, we describe the BE FAIR framework, which has the potential to enable nurses to take a leadership role within health systems by implementing a governance structure to oversee the fairness and quality of clinical algorithms. We then examine leading frameworks for promoting health equity to inform the operationalization of BE FAIR within a local AI governance framework.

RESULTS

The application of the BE FAIR framework within the context of a working governance system for clinical AI technologies demonstrates how nurses can leverage their expertise to support the development and deployment of clinical algorithms, mitigating risks such as bias and promoting ethical, high-quality care powered by big data and AI technologies.

CONCLUSION AND RELEVANCE

As health systems learn how well-intentioned clinical algorithms can potentially perpetuate health disparities, we have an opportunity and an obligation to do better. New efforts empowering nurses to advocate for BE FAIR, involving them in AI governance, data collection methods, and the evaluation of tools intended to reduce bias, mark important steps in achieving equitable healthcare for all.

Machine learning in the diagnosis, management, and care of patients with low back pain: a scoping review of the literature and future directions

BACKGROUND CONTEXT

Low back pain (LBP) remains the leading cause of disability globally. In recent years, machine learning (ML) has emerged as a potentially useful tool to aid the diagnosis, management, and prognostication of LBP.

PURPOSE

In this review, we assess the scope of ML applications in the LBP literature and outline gaps and opportunities.

STUDY DESIGN/SETTING

A scoping review was performed in accordance with the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guidelines.

METHODS

Articles were extracted from the Web of Science, Scopus, PubMed, and IEEE Xplore databases. Title/abstract and full-text screening was performed by two reviewers. Data on model type, model inputs, predicted outcomes, and ML methods were collected.

RESULTS

In total, 223 unique studies published between 1988 and 2023 were identified, with just over 50% focused on low-back-pain detection. Neural networks were used in 106 of these articles. Common inputs included patient history, demographics, and lab values (67% total). Articles published after 2010 were also likely to incorporate imaging data into their models (41.7% of articles). Of the 212 supervised learning articles identified, 168 (79.4%) mentioned use of a training or testing dataset, 116 (54.7%) utilized cross-validation, and 46 (21.7%) implemented hyperparameter optimization. Of all articles, only 8 included external validation and 9 had publicly available code.

CONCLUSIONS

Despite the rapid application of ML in LBP research, a majority of articles do not follow standard ML best practices. Furthermore, over 95% of articles cannot be reproduced or authenticated due to lack of code availability. Increased collaboration and code sharing are needed to support future growth and implementation of ML in the care of patients with LBP.

Enhancing Clinical Decision Support in Nephrology: Addressing Algorithmic Bias Through Artificial Intelligence Governance

There has been a steady rise in the use of clinical decision support (CDS) tools to guide nephrology as well as general clinical care. Through guidance set by federal agencies and concerns raised by clinical investigators, there has been an equal rise in understanding whether such tools exhibit algorithmic bias leading to unfairness. This has spurred the more fundamental question of whether sensitive variables such as race should be included in CDS tools. In order to properly answer this question, it is necessary to understand how algorithmic bias arises. We break down 3 sources of bias encountered when using electronic health record data to develop CDS tools: (1) use of proxy variables, (2) observability concerns and (3) underlying heterogeneity. We discuss how answering the question of whether to include sensitive variables like race often hinges more on qualitative considerations than on quantitative analysis, dependent on the function that the sensitive variable serves. Based on our experience with our own institution's CDS governance group, we show how health system-based governance committees play a central role in guiding these difficult and important considerations. Ultimately, our goal is to foster a community practice of model development and governance teams that emphasizes consciousness about sensitive variables and prioritizes equity.

Addressing ethical issues in healthcare artificial intelligence using a lifecycle-informed process

Objectives

Artificial intelligence (AI) proceeds through an iterative and evaluative process of development, use, and refinement which may be characterized as a lifecycle. Within this context, stakeholders can vary in their interests and perceptions of the ethical issues associated with this rapidly evolving technology in ways that can fail to identify and avert adverse outcomes. Identifying issues throughout the AI lifecycle in a systematic manner can facilitate better-informed ethical deliberation.

Materials and Methods

We analyzed existing lifecycles from within the current literature for ethical issues of AI in healthcare to identify themes, which we relied upon to create a lifecycle that consolidates these themes into a more comprehensive lifecycle. We then considered the potential benefits and harms of AI through this lifecycle to identify ethical questions that can arise at each step and to identify where conflicts and errors could arise in ethical analysis. We illustrated the approach in 3 case studies that highlight how different ethical dilemmas arise at different points in the lifecycle.

Results Discussion and Conclusion

Through case studies, we show how a systematic lifecycle-informed approach to the ethical analysis of AI enables mapping of the effects of AI onto different steps to guide deliberations on benefits and harms. The lifecycle-informed approach has broad applicability to different stakeholders and can facilitate communication on ethical issues for patients, healthcare professionals, research participants, and other stakeholders.

Translating AI for the Clinician

This Viewpoint explores how artificial intelligence technologies can adopt a clinical practice framework to identify use cases and outline the technology’s objectives and potential uses in modern health care.

The incremental design of a machine learning framework for medical records processing

OBJECTIVES

This work presents the development and evaluation of coordn8, a web-based application that streamlines fax processing in outpatient clinics using a "human-in-the-loop" machine learning framework. We demonstrate the effectiveness of the platform at reducing fax processing time and producing accurate machine learning inferences across the tasks of patient identification, document classification, spam classification, and duplicate document detection.

METHODS

We deployed coordn8 in 11 outpatient clinics and conducted a time savings analysis by observing users and measuring fax processing event logs. We used statistical methods to evaluate the machine learning components across different datasets to show generalizability. We conducted a time series analysis to show variations in model performance as new clinics were onboarded and to demonstrate our approach to mitigating model drift.

RESULTS

Our observation analysis showed a mean reduction in individual fax processing time by 147.5 s, while our event log analysis of over 7000 faxes reinforced this finding. Document classification produced an accuracy of 81.6%, patient identification produced an accuracy of 83.7%, spam classification produced an accuracy of 98.4%, and duplicate document detection produced a precision of 81.0%. Retraining document classification increased accuracy by 10.2%.

DISCUSSION

coordn8 significantly decreased fax-processing time and produced accurate machine learning inferences. Our human-in-the-loop framework facilitated the collection of high-quality data necessary for model training. Expanding to new clinics correlated with performance decline, which was mitigated through model retraining.

CONCLUSION

Our framework for automating clinical tasks with machine learning offers a template for health systems looking to implement similar technologies.

Clinical Research Informatics: a Decade-in-Review

BACKGROUND

Clinical Research Informatics (CRI) is a subspeciality of biomedical informatics that has substantially matured during the last decade. Advances in CRI have transformed the way clinical research is conducted. In recent years, there has been growing interest in CRI, as reflected by a vast and expanding scientific literature focused on the topic. The main objectives of this review are: 1) to provide an overview of the evolving definition and scope of this biomedical informatics subspecialty over the past 10 years; 2) to highlight major contributions to the field during the past decade; and 3) to provide insights about more recent CRI research trends and perspectives.

METHODS

We adopted a modified thematic review approach focused on understanding the evolution and current status of the CRI field based on literature sources identified through two complementary review processes (AMIA CRI year-in-review/IMIA Yearbook of Medical Informatics) conducted annually during the last decade.

RESULTS

More than 1,500 potentially relevant publications were considered, and 205 sources were included in the final review. The review identified key publications defining the scope of CRI and/or capturing its evolution over time as illustrated by impactful tools and methods in different categories of CRI focus. The review also revealed current topics of interest in CRI and prevailing research trends.

CONCLUSION

This scoping review provides an overview of a decade of research in CRI, highlighting major changes in the core CRI discoveries as well as increasingly impactful methods and tools that have bridged the principles-to-practice gap. Practical CRI solutions as well as examples of CRI-enabled large-scale, multi-organizational and/or multi-national research projects demonstrate the maturity of the field. Despite the progress demonstrated, some topics remain challenging, highlighting the need for ongoing CRI development and research, including the need of more rigorous evaluations of CRI solutions and further formalization and maturation of CRI services and capabilities across the research enterprise.

The TRIPOD-LLM Statement: A Targeted Guideline For Reporting Large Language Models Use

Large Language Models (LLMs) are rapidly being adopted in healthcare, necessitating standardized reporting guidelines. We present TRIPOD-LLM, an extension of the TRIPOD+AI statement, addressing the unique challenges of LLMs in biomedical applications. TRIPOD-LLM provides a comprehensive checklist of 19 main items and 50 subitems, covering key aspects from title to discussion. The guidelines introduce a modular format accommodating various LLM research designs and tasks, with 14 main items and 32 subitems applicable across all categories. Developed through an expedited Delphi process and expert consensus, TRIPOD-LLM emphasizes transparency, human oversight, and task-specific performance reporting. We also introduce an interactive website ( https://tripod-llm.vercel.app/ ) facilitating easy guideline completion and PDF generation for submission. As a living document, TRIPOD-LLM will evolve with the field, aiming to enhance the quality, reproducibility, and clinical applicability of LLM research in healthcare through comprehensive reporting.

COI

DSB: Editorial, unrelated to this work: Associate Editor of Radiation Oncology, HemOnc.org (no financial compensation); Research funding, unrelated to this work: American Association for Cancer Research; Advisory and consulting, unrelated to this work: MercurialAI. DDF: Editorial, unrelated to this work: Associate Editor of JAMIA, Editorial Board of Scientific Data, Nature; Funding, unrelated to this work: the intramural research program at the U.S. National Library of Medicine, National Institutes of Health. JWG: Editorial, unrelated to this work: Editorial Board of Radiology: Artificial Intelligence, British Journal of Radiology AI journal and NEJM AI. All other authors declare no conflicts of interest.

Designing an Implementable Clinical Prediction Model for Near-Term Mortality and Long-Term Survival in Patients on Maintenance Hemodialysis

RATIONALE & OBJECTIVE

The life expectancy of patients treated with maintenance hemodialysis (MHD) is heterogeneous. Knowledge of life-expectancy may focus care decisions on near-term versus long-term goals. The current tools are limited and focus on near-term mortality. Here, we develop and assess potential utility for predicting near-term mortality and long-term survival on MHD.

STUDY DESIGN

Predictive modeling study.

SETTING & PARTICIPANTS

42,351 patients contributing 997,381 patient months over 11 years, abstracted from the electronic health record (EHR) system of midsize, nonprofit dialysis providers.

NEW PREDICTORS & ESTABLISHED PREDICTORS

Demographics, laboratory results, vital signs, and service utilization data available within dialysis EHR.

OUTCOME

For each patient month, we ascertained death within the next 6 months (ie, near-term mortality) and survival over more than 5 years during receipt of MHD or after kidney transplantation (ie, long-term survival).

ANALYTICAL APPROACH

We used least absolute shrinkage and selection operator logistic regression and gradient-boosting machines to predict each outcome. We compared these to time-to-event models spanning both time horizons. We explored the performance of decision rules at different cut points.

RESULTS

All models achieved an area under the receiver operator characteristic curve of≥0.80 and optimal calibration metrics in the test set. The long-term survival models had significantly better performance than the near-term mortality models. The time-to-event models performed similarly to binary models. Applying different cut points spanning from the 1st to 90th percentile of the predictions, a positive predictive value (PPV) of 54% could be achieved for near-term mortality, but with poor sensitivity of 6%. A PPV of 71% could be achieved for long-term survival with a sensitivity of 67%.

LIMITATIONS

The retrospective models would need to be prospectively validated before they could be appropriately used as clinical decision aids.

CONCLUSIONS

A model built with readily available clinical variables to support easy implementation can predict clinically important life expectancy thresholds and shows promise as a clinical decision support tool for patients on MHD. Predicting long-term survival has better decision rule performance than predicting near-term mortality.

PLAIN-LANGUAGE SUMMARY

Clinical prediction models (CPMs) are not widely used for patients undergoing maintenance hemodialysis (MHD). Although a variety of CPMs have been reported in the literature, many of these were not well-designed to be easily implementable. We consider the performance of an implementable CPM for both near-term mortality and long-term survival for patients undergoing MHD. Both near-term and long-term models have similar predictive performance, but the long-term models have greater clinical utility. We further consider how the differential performance of predicting over different time horizons may be used to impact clinical decision making. Although predictive modeling is not regularly used for MHD patients, such tools may help promote individualized care planning and foster shared decision making.

Artificial Intelligence in Cardiovascular Care-Part 2: Applications: JACC Review Topic of the Week

Recent artificial intelligence (AI) advancements in cardiovascular care offer potential enhancements in effective diagnosis, treatment, and outcomes. More than 600 U.S. Food and Drug Administration-approved clinical AI algorithms now exist, with 10% focusing on cardiovascular applications, highlighting the growing opportunities for AI to augment care. This review discusses the latest advancements in the field of AI, with a particular focus on the utilization of multimodal inputs and the field of generative AI. Further discussions in this review involve an approach to understanding the larger context in which AI-augmented care may exist, and include a discussion of the need for rigorous evaluation, appropriate infrastructure for deployment, ethics and equity assessments, regulatory oversight, and viable business cases for deployment. Embracing this rapidly evolving technology while setting an appropriately high evaluation benchmark with careful and patient-centered implementation will be crucial for cardiology to leverage AI to enhance patient care and the provider experience.

Comments on Contemporary Uses of Machine Learning for Electronic Health Records

Various decisions concerning the management, display, and diagnostic use of electronic health records (EHR) data can be automated using machine learning (ML). We describe how ML is currently applied to EHR data and how it may be applied in the near future. Both benefits and shortcomings of ML are considered.

Transparent medical image AI via an image-text foundation model grounded in medical literature

Building trustworthy and transparent image-based medical artificial intelligence (AI) systems requires the ability to interrogate data and models at all stages of the development pipeline, from training models to post-deployment monitoring. Ideally, the data and associated AI systems could be described using terms already familiar to physicians, but this requires medical datasets densely annotated with semantically meaningful concepts. In the present study, we present a foundation model approach, named MONET (medical concept retriever), which learns how to connect medical images with text and densely scores images on concept presence to enable important tasks in medical AI development and deployment such as data auditing, model auditing and model interpretation. Dermatology provides a demanding use case for the versatility of MONET, due to the heterogeneity in diseases, skin tones and imaging modalities. We trained MONET based on 105,550 dermatological images paired with natural language descriptions from a large collection of medical literature. MONET can accurately annotate concepts across dermatology images as verified by board-certified dermatologists, competitively with supervised models built on previously concept-annotated dermatology datasets of clinical images. We demonstrate how MONET enables AI transparency across the entire AI system development pipeline, from building inherently interpretable models to dataset and model auditing, including a case study dissecting the results of an AI clinical trial.

Interpretable Predictive Models to Understand Risk Factors for Maternal and Fetal Outcomes

Although most pregnancies result in a good outcome, complications are not uncommon and can be associated with serious implications for mothers and babies. Predictive modeling has the potential to improve outcomes through a better understanding of risk factors, heightened surveillance for high-risk patients, and more timely and appropriate interventions, thereby helping obstetricians deliver better care. We identify and study the most important risk factors for four types of pregnancy complications: (i) severe maternal morbidity, (ii) shoulder dystocia, (iii) preterm preeclampsia, and (iv) antepartum stillbirth. We use an Explainable Boosting Machine (EBM), a high-accuracy glass-box learning method, for the prediction and identification of important risk factors. We undertake external validation and perform an extensive robustness analysis of the EBM models. EBMs match the accuracy of other black-box ML methods, such as deep neural networks and random forests, and outperform logistic regression, while being more interpretable. EBMs prove to be robust. The interpretability of the EBM models reveal surprising insights into the features contributing to risk (e.g., maternal height is the second most important feature for shoulder dystocia) and may have potential for clinical application in the prediction and prevention of serious complications in pregnancy.

Empirical data drift detection experiments on real-world medical imaging data

While it is common to monitor deployed clinical artificial intelligence (AI) models for performance degradation, it is less common for the input data to be monitored for data drift - systemic changes to input distributions. However, when real-time evaluation may not be practical (eg., labeling costs) or when gold-labels are automatically generated, we argue that tracking data drift becomes a vital addition for AI deployments. In this work, we perform empirical experiments on real-world medical imaging to evaluate three data drift detection methods' ability to detect data drift caused (a) naturally (emergence of COVID-19 in X-rays) and (b) synthetically. We find that monitoring performance alone is not a good proxy for detecting data drift and that drift-detection heavily depends on sample size and patient features. Our work discusses the need and utility of data drift detection in various scenarios and highlights gaps in knowledge for the practical application of existing methods.

Translating ethical and quality principles for the effective, safe and fair development, deployment and use of artificial intelligence technologies in healthcare

OBJECTIVE

The complexity and rapid pace of development of algorithmic technologies pose challenges for their regulation and oversight in healthcare settings. We sought to improve our institution's approach to evaluation and governance of algorithmic technologies used in clinical care and operations by creating an Implementation Guide that standardizes evaluation criteria so that local oversight is performed in an objective fashion.

MATERIALS AND METHODS

Building on a framework that applies key ethical and quality principles (clinical value and safety, fairness and equity, usability and adoption, transparency and accountability, and regulatory compliance), we created concrete guidelines for evaluating algorithmic technologies at our institution.

RESULTS

An Implementation Guide articulates evaluation criteria used during review of algorithmic technologies and details what evidence supports the implementation of ethical and quality principles for trustworthy health AI. Application of the processes described in the Implementation Guide can lead to algorithms that are safer as well as more effective, fair, and equitable upon implementation, as illustrated through 4 examples of technologies at different phases of the algorithmic lifecycle that underwent evaluation at our academic medical center.

DISCUSSION

By providing clear descriptions/definitions of evaluation criteria and embedding them within standardized processes, we streamlined oversight processes and educated communities using and developing algorithmic technologies within our institution.

CONCLUSIONS

We developed a scalable, adaptable framework for translating principles into evaluation criteria and specific requirements that support trustworthy implementation of algorithmic technologies in patient care and healthcare operations.

President Biden's Executive Order on Artificial Intelligence-Implications for Health Care Organizations

This Viewpoint discusses a recent executive order by US President Joe Biden about the development and implementation of AI, including the role of government vs the private sector and how the order may affect health care.

Dashboarding to Monitor Machine-Learning-Based Clinical Decision Support Interventions

BACKGROUND

 Existing monitoring of machine-learning-based clinical decision support (ML-CDS) is focused predominantly on the ML outputs and accuracy thereof. Improving patient care requires not only accurate algorithms but also systems of care that enable the output of these algorithms to drive specific actions by care teams, necessitating expanding their monitoring.

OBJECTIVES

 In this case report, we describe the creation of a dashboard that allows the intervention development team and operational stakeholders to govern and identify potential issues that may require corrective action by bridging the monitoring gap between model outputs and patient outcomes.

METHODS

 We used an iterative development process to build a dashboard to monitor the performance of our intervention in the broader context of the care system.

RESULTS

 Our investigation of best practices elsewhere, iterative design, and expert consultation led us to anchor our dashboard on alluvial charts and control charts. Both the development process and the dashboard itself illuminated areas to improve the broader intervention.

CONCLUSION

 We propose that monitoring ML-CDS algorithms with regular dashboards that allow both a context-level view of the system and a drilled down view of specific components is a critical part of implementing these algorithms to ensure that these tools function appropriately within the broader care system.

How patients distinguish between clinical and administrative predictive models in health care

OBJECTIVES

To understand patient perceptions of specific applications of predictive models in health care.

STUDY DESIGN

Original, cross-sectional national survey.

METHODS

We conducted a national online survey of US adults with the National Opinion Research Center from November to December 2021. Measures of internal consistency were used to identify how patients differentiate between clinical and administrative predictive models. Multivariable logistic regressions were used to identify relationships between comfort with various types of predictive models and patient demographics, perceptions of privacy protections, and experiences in the health care system.

RESULTS

A total of 1541 respondents completed the survey. After excluding observations with missing data for the variables of interest, the final analytic sample was 1488. We found that patients differentiate between clinical and administrative predictive models. Comfort with prediction of bill payment and missed appointments was especially low (21.6% and 36.6%, respectively). Comfort was higher with clinical predictive models, such as predicting stroke in an emergency (55.8%). Experiences of discrimination were significant negative predictors of comfort with administrative predictive models. Health system transparency around privacy policies was a significant positive predictor of comfort with both clinical and administrative predictive models.

CONCLUSIONS

Patients are more comfortable with clinical applications of predictive models than administrative ones. Privacy protections and transparency about how health care systems protect patient data may facilitate patient comfort with these technologies. However, larger inequities and negative experiences in health care remain important for how patients perceive administrative applications of prediction.

Practical Considerations and Applied Examples of Cross-Validation for Model Development and Evaluation in Health Care: Tutorial

Cross-validation remains a popular means of developing and validating artificial intelligence for health care. Numerous subtypes of cross-validation exist. Although tutorials on this validation strategy have been published and some with applied examples, we present here a practical tutorial comparing multiple forms of cross-validation using a widely accessible, real-world electronic health care data set: Medical Information Mart for Intensive Care-III (MIMIC-III). This tutorial explored methods such as K-fold cross-validation and nested cross-validation, highlighting their advantages and disadvantages across 2 common predictive modeling use cases: classification (mortality) and regression (length of stay). We aimed to provide readers with reproducible notebooks and best practices for modeling with electronic health care data. We also described sets of useful recommendations as we demonstrated that nested cross-validation reduces optimistic bias but comes with additional computational challenges. This tutorial might improve the community's understanding of these important methods while catalyzing the modeling community to apply these guides directly in their work using the published code.

Guiding Principles to Address the Impact of Algorithm Bias on Racial and Ethnic Disparities in Health and Health Care

Importance

Health care algorithms are used for diagnosis, treatment, prognosis, risk stratification, and allocation of resources. Bias in the development and use of algorithms can lead to worse outcomes for racial and ethnic minoritized groups and other historically marginalized populations such as individuals with lower income.

Objective

To provide a conceptual framework and guiding principles for mitigating and preventing bias in health care algorithms to promote health and health care equity.

Evidence Review

The Agency for Healthcare Research and Quality and the National Institute for Minority Health and Health Disparities convened a diverse panel of experts to review evidence, hear from stakeholders, and receive community feedback.

Findings

The panel developed a conceptual framework to apply guiding principles across an algorithm's life cycle, centering health and health care equity for patients and communities as the goal, within the wider context of structural racism and discrimination. Multiple stakeholders can mitigate and prevent bias at each phase of the algorithm life cycle, including problem formulation (phase 1); data selection, assessment, and management (phase 2); algorithm development, training, and validation (phase 3); deployment and integration of algorithms in intended settings (phase 4); and algorithm monitoring, maintenance, updating, or deimplementation (phase 5). Five principles should guide these efforts: (1) promote health and health care equity during all phases of the health care algorithm life cycle; (2) ensure health care algorithms and their use are transparent and explainable; (3) authentically engage patients and communities during all phases of the health care algorithm life cycle and earn trustworthiness; (4) explicitly identify health care algorithmic fairness issues and trade-offs; and (5) establish accountability for equity and fairness in outcomes from health care algorithms.

Conclusions and Relevance

Multiple stakeholders must partner to create systems, processes, regulations, incentives, standards, and policies to mitigate and prevent algorithmic bias. Reforms should implement guiding principles that support promotion of health and health care equity in all phases of the algorithm life cycle as well as transparency and explainability, authentic community engagement and ethical partnerships, explicit identification of fairness issues and trade-offs, and accountability for equity and fairness.

Development, Deployment, and Implementation of a Machine Learning Surgical Case Length Prediction Model and Prospective Evaluation

OBJECTIVE

To implement a machine learning model using only the restricted data available at case creation time to predict surgical case length for multiple services at different locations.

BACKGROUND

The operating room is one of the most expensive resources in a health system, estimated to cost $22 to $133 per minute and generate about 40% of hospital revenue. Accurate prediction of surgical case length is necessary for efficient scheduling and cost-effective utilization of the operating room and other resources.

METHODS

We introduced a similarity cascade to capture the complexity of cases and surgeon influence on the case length and incorporated that into a gradient-boosting machine learning model. The model loss function was customized to improve the balance between over- and under-prediction of the case length. A production pipeline was created to seamlessly deploy and implement the model across our institution.

RESULTS

The prospective analysis showed that the model output was gradually adopted by the schedulers and outperformed the scheduler-predicted case length from August to December 2022. In 33,815 surgical cases across outpatient and inpatient platforms, the operational implementation predicted 11.2% fewer underpredicted cases and 5.9% more cases within 20% of the actual case length compared with the schedulers and only overpredicted 5.3% more. The model assisted schedulers to predict 3.4% more cases within 20% of the actual case length and 4.3% fewer underpredicted cases.

CONCLUSIONS

We created a unique framework that is being leveraged every day to predict surgical case length more accurately at case posting time and could be potentially utilized to deploy future machine learning models.

Artificial Intelligence Governance and Otolaryngology-Head and Neck Surgery

This rapid communication highlights components of artificial intelligence governance in healthcare and suggests adopting key governance approaches in otolaryngology – head and neck surgery.

A translational perspective towards clinical AI fairness

Artificial intelligence (AI) has demonstrated the ability to extract insights from data, but the fairness of such data-driven insights remains a concern in high-stakes fields. Despite extensive developments, issues of AI fairness in clinical contexts have not been adequately addressed. A fair model is normally expected to perform equally across subgroups defined by sensitive variables (e.g., age, gender/sex, race/ethnicity, socio-economic status, etc.). Various fairness measurements have been developed to detect differences between subgroups as evidence of bias, and bias mitigation methods are designed to reduce the differences detected. This perspective of fairness, however, is misaligned with some key considerations in clinical contexts. The set of sensitive variables used in healthcare applications must be carefully examined for relevance and justified by clear clinical motivations. In addition, clinical AI fairness should closely investigate the ethical implications of fairness measurements (e.g., potential conflicts between group- and individual-level fairness) to select suitable and objective metrics. Generally defining AI fairness as "equality" is not necessarily reasonable in clinical settings, as differences may have clinical justifications and do not indicate biases. Instead, "equity" would be an appropriate objective of clinical AI fairness. Moreover, clinical feedback is essential to developing fair and well-performing AI models, and efforts should be made to actively involve clinicians in the process. The adaptation of AI fairness towards healthcare is not self-evident due to misalignments between technical developments and clinical considerations. Multidisciplinary collaboration between AI researchers, clinicians, and ethicists is necessary to bridge the gap and translate AI fairness into real-life benefits.

DEPLOYR: a technical framework for deploying custom real-time machine learning models into the electronic medical record

OBJECTIVE

Heatlhcare institutions are establishing frameworks to govern and promote the implementation of accurate, actionable, and reliable machine learning models that integrate with clinical workflow. Such governance frameworks require an accompanying technical framework to deploy models in a resource efficient, safe and high-quality manner. Here we present DEPLOYR, a technical framework for enabling real-time deployment and monitoring of researcher-created models into a widely used electronic medical record system.

MATERIALS AND METHODS

We discuss core functionality and design decisions, including mechanisms to trigger inference based on actions within electronic medical record software, modules that collect real-time data to make inferences, mechanisms that close-the-loop by displaying inferences back to end-users within their workflow, monitoring modules that track performance of deployed models over time, silent deployment capabilities, and mechanisms to prospectively evaluate a deployed model's impact.

RESULTS

We demonstrate the use of DEPLOYR by silently deploying and prospectively evaluating 12 machine learning models trained using electronic medical record data that predict laboratory diagnostic results, triggered by clinician button-clicks in Stanford Health Care's electronic medical record.

DISCUSSION

Our study highlights the need and feasibility for such silent deployment, because prospectively measured performance varies from retrospective estimates. When possible, we recommend using prospectively estimated performance measures during silent trials to make final go decisions for model deployment.

CONCLUSION

Machine learning applications in healthcare are extensively researched, but successful translations to the bedside are rare. By describing DEPLOYR, we aim to inform machine learning deployment best practices and help bridge the model implementation gap.

Implementation frameworks for end-to-end clinical AI: derivation of the SALIENT framework

OBJECTIVE

To derive a comprehensive implementation framework for clinical AI models within hospitals informed by existing AI frameworks and integrated with reporting standards for clinical AI research.

MATERIALS AND METHODS

(1) Derive a provisional implementation framework based on the taxonomy of Stead et al and integrated with current reporting standards for AI research: TRIPOD, DECIDE-AI, CONSORT-AI. (2) Undertake a scoping review of published clinical AI implementation frameworks and identify key themes and stages. (3) Perform a gap analysis and refine the framework by incorporating missing items.

RESULTS

The provisional AI implementation framework, called SALIENT, was mapped to 5 stages common to both the taxonomy and the reporting standards. A scoping review retrieved 20 studies and 247 themes, stages, and subelements were identified. A gap analysis identified 5 new cross-stage themes and 16 new tasks. The final framework comprised 5 stages, 7 elements, and 4 components, including the AI system, data pipeline, human-computer interface, and clinical workflow.

DISCUSSION

This pragmatic framework resolves gaps in existing stage- and theme-based clinical AI implementation guidance by comprehensively addressing the what (components), when (stages), and how (tasks) of AI implementation, as well as the who (organization) and why (policy domains). By integrating research reporting standards into SALIENT, the framework is grounded in rigorous evaluation methodologies. The framework requires validation as being applicable to real-world studies of deployed AI models.

CONCLUSIONS

A novel end-to-end framework has been developed for implementing AI within hospital clinical practice that builds on previous AI implementation frameworks and research reporting standards.

Effectiveness of an Emergency Department-Based Machine Learning Clinical Decision Support Tool to Prevent Outpatient Falls Among Older Adults: Protocol for a Quasi-Experimental Study

BACKGROUND

Emergency department (ED) providers are important collaborators in preventing falls for older adults because they are often the first health care providers to see a patient after a fall and because at-home falls are often preceded by previous ED visits. Previous work has shown that ED referrals to falls interventions can reduce the risk of an at-home fall by 38%. Screening patients at risk for a fall can be time-consuming and difficult to implement in the ED setting. Machine learning (ML) and clinical decision support (CDS) offer the potential of automating the screening process. However, it remains unclear whether automation of screening and referrals can reduce the risk of future falls among older patients.

OBJECTIVE

The goal of this paper is to describe a research protocol for evaluating the effectiveness of an automated screening and referral intervention. These findings will inform ongoing discussions about the use of ML and artificial intelligence to augment medical decision-making.

METHODS

To assess the effectiveness of our program for patients receiving the falls risk intervention, our primary analysis will be to obtain referral completion rates at 3 different EDs. We will use a quasi-experimental design known as a sharp regression discontinuity with regard to intent-to-treat, since the intervention is administered to patients whose risk score falls above a threshold. A conditional logistic regression model will be built to describe 6-month fall risk at each site as a function of the intervention, patient demographics, and risk score. The odds ratio of a return visit for a fall and the 95% CI will be estimated by comparing those identified as high risk by the ML-based CDS (ML-CDS) and those who were not but had a similar risk profile.

RESULTS

The ML-CDS tool under study has been implemented at 2 of the 3 EDs in our study. As of April 2023, a total of 1326 patient encounters have been flagged for providers, and 339 unique patients have been referred to the mobility and falls clinic. To date, 15% (45/339) of patients have scheduled an appointment with the clinic.

CONCLUSIONS

This study seeks to quantify the impact of an ML-CDS intervention on patient behavior and outcomes. Our end-to-end data set allows for a more meaningful analysis of patient outcomes than other studies focused on interim outcomes, and our multisite implementation plan will demonstrate applicability to a broad population and the possibility to adapt the intervention to other EDs and achieve similar results. Our statistical methodology, regression discontinuity design, allows for causal inference from observational data and a staggered implementation strategy allows for the identification of secular trends that could affect causal associations and allow mitigation as necessary.

TRIAL REGISTRATION

ClinicalTrials.gov NCT05810064; https://www.clinicaltrials.gov/study/NCT05810064.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/48128.

The Role of Human and Organizational Factors in the Pursuit of One Digital Health

OBJECTIVE

This paper surveys a subset of the 2022 human and organizational factor (HOF) literature to provide guidance on building a One Digital Health ecosystem.

METHODS

We searched a subset of journals in PubMed/Medline for studies with "human factors" or "organization" in the title or abstract. Papers published in 2022 were eligible for inclusion in the survey. Selected papers were categorized into structural and behavioural aspects to understand digital health enabled interactions across micro, meso, and macro systems.

RESULTS

Our survey of the 2022 HOF literature showed that while we continue to make meaningful progress at digital health enabled interactions across systems levels, there are still challenges that must be overcome. For example, we must continue to grow the breadth of HOF research beyond individual users and systems to assist with the scale up of digital health systems across and beyond organizations. We summarize the findings by providing five HOF considerations to help build a One Digital Health ecosystem.

CONCLUSION

One Digital Health challenges us to improve coordination, communication, and collaboration between the health, environmental and veterinary sectors. Doing so requires us to develop both the structural and behavioural capacity of digital health systems at the organizational level and beyond so that we can develop more robust and integrated systems across health, environmental and veterinary sectors. The HOF community has much to offer and must play a leading role in designing a One Digital Health ecosystem.

The Role of Artificial Intelligence Model Documentation in Translational Science: Scoping Review

BACKGROUND

Despite the touted potential of artificial intelligence (AI) and machine learning (ML) to revolutionize health care, clinical decision support tools, herein referred to as medical modeling software (MMS), have yet to realize the anticipated benefits. One proposed obstacle is the acknowledged gaps in AI translation. These gaps stem partly from the fragmentation of processes and resources to support MMS transparent documentation. Consequently, the absence of transparent reporting hinders the provision of evidence to support the implementation of MMS in clinical practice, thereby serving as a substantial barrier to the successful translation of software from research settings to clinical practice.

OBJECTIVE

This study aimed to scope the current landscape of AI- and ML-based MMS documentation practices and elucidate the function of documentation in facilitating the translation of ethical and explainable MMS into clinical workflows.

METHODS

A scoping review was conducted in accordance with PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines. PubMed was searched using Medical Subject Headings key concepts of AI, ML, ethical considerations, and explainability to identify publications detailing AI- and ML-based MMS documentation, in addition to snowball sampling of selected reference lists. To include the possibility of implicit documentation practices not explicitly labeled as such, we did not use documentation as a key concept but as an inclusion criterion. A 2-stage screening process (title and abstract screening and full-text review) was conducted by 1 author. A data extraction template was used to record publication-related information; barriers to developing ethical and explainable MMS; available standards, regulations, frameworks, or governance strategies related to documentation; and recommendations for documentation for papers that met the inclusion criteria.

RESULTS

Of the 115 papers retrieved, 21 (18.3%) papers met the requirements for inclusion. Ethics and explainability were investigated in the context of AI- and ML-based MMS documentation and translation. Data detailing the current state and challenges and recommendations for future studies were synthesized. Notable themes defining the current state and challenges that required thorough review included bias, accountability, governance, and explainability. Recommendations identified in the literature to address present barriers call for a proactive evaluation of MMS, multidisciplinary collaboration, adherence to investigation and validation protocols, transparency and traceability requirements, and guiding standards and frameworks that enhance documentation efforts and support the translation of AI- and ML-based MMS.

CONCLUSIONS

Resolving barriers to translation is critical for MMS to deliver on expectations, including those barriers identified in this scoping review related to bias, accountability, governance, and explainability. Our findings suggest that transparent strategic documentation, aligning translational science and regulatory science, will support the translation of MMS by coordinating communication and reporting and reducing translational barriers, thereby furthering the adoption of MMS.

APLUS: A Python library for usefulness simulations of machine learning models in healthcare

Despite the creation of thousands of machine learning (ML) models, the promise of improving patient care with ML remains largely unrealized. Adoption into clinical practice is lagging, in large part due to disconnects between how ML practitioners evaluate models and what is required for their successful integration into care delivery. Models are just one component of care delivery workflows whose constraints determine clinicians' abilities to act on models' outputs. However, methods to evaluate the usefulness of models in the context of their corresponding workflows are currently limited. To bridge this gap we developed APLUS, a reusable framework for quantitatively assessing via simulation the utility gained from integrating a model into a clinical workflow. We describe the APLUS simulation engine and workflow specification language, and apply it to evaluate a novel ML-based screening pathway for detecting peripheral artery disease at Stanford Health Care.