Explainable artificial intelligence models using real-world electronic health record data: a systematic scoping review

From explainable to interpretable deep learning for natural language processing in healthcare: How far from reality?

Deep learning (DL) has substantially enhanced natural language processing (NLP) in healthcare research. However, the increasing complexity of DL-based NLP necessitates transparent model interpretability, or at least explainability, for reliable decision-making. This work presents a thorough scoping review of explainable and interpretable DL in healthcare NLP. The term "eXplainable and Interpretable Artificial Intelligence" (XIAI) is introduced to distinguish XAI from IAI. Different models are further categorized based on their functionality (model-, input-, output-based) and scope (local, global). Our analysis shows that attention mechanisms are the most prevalent emerging IAI technique. The use of IAI is growing, distinguishing it from XAI. The major challenges identified are that most XIAI does not explore "global" modelling processes, the lack of best practices, and the lack of systematic evaluation and benchmarks. One important opportunity is to use attention mechanisms to enhance multi-modal XIAI for personalized medicine. Additionally, combining DL with causal logic holds promise. Our discussion encourages the integration of XIAI in Large Language Models (LLMs) and domain-specific smaller models. In conclusion, XIAI adoption in healthcare requires dedicated in-house expertise. Collaboration with domain experts, end-users, and policymakers can lead to ready-to-use XIAI methods across NLP and medical tasks. While challenges exist, XIAI techniques offer a valuable foundation for interpretable NLP algorithms in healthcare.

A fair individualized polysocial risk score for identifying increased social risk in type 2 diabetes

Racial and ethnic minorities bear a disproportionate burden of type 2 diabetes (T2D) and its complications, with social determinants of health (SDoH) recognized as key drivers of these disparities. Implementing efficient and effective social needs management strategies is crucial. We propose a machine learning analytic pipeline to calculate the individualized polysocial risk score (iPsRS), which can identify T2D patients at high social risk for hospitalization, incorporating explainable AI techniques and algorithmic fairness optimization. We use electronic health records (EHR) data from T2D patients in the University of Florida Health Integrated Data Repository, incorporating both contextual SDoH (e.g., neighborhood deprivation) and person-level SDoH (e.g., housing instability). After fairness optimization across racial and ethnic groups, the iPsRS achieved a C statistic of 0.71 in predicting 1-year hospitalization. Our iPsRS can fairly and accurately screen patients with T2D who are at increased social risk for hospitalization.

The application of eXplainable artificial intelligence in studying cognition: A scoping review

The rapid advancement of artificial intelligence (AI) has sparked renewed discussions on its trustworthiness and the concept of eXplainable AI (XAI). Recent research in neuroscience has emphasized the relevance of XAI in studying cognition. This scoping review aims to identify and analyze various XAI methods used to study the mechanisms and features of cognitive function and dysfunction. In this study, the collected evidence is qualitatively assessed to develop an effective framework for approaching XAI in cognitive neuroscience. Based on the Joanna Briggs Institute and preferred reporting items for systematic reviews and meta-analyses extension for scoping review guidelines, we searched for peer-reviewed articles on MEDLINE, Embase, Web of Science, Cochrane Central Register of Controlled Trials, and Google Scholar. Two reviewers performed data screening, extraction, and thematic analysis in parallel. Twelve eligible experimental studies published in the past decade were included. The results showed that the majority (75%) focused on normal cognitive functions such as perception, social cognition, language, executive function, and memory, while others (25%) examined impaired cognition. The predominant XAI methods employed were intrinsic XAI (58.3%), followed by attribution-based (41.7%) and example-based (8.3%) post hoc methods. Explainability was applied at a local (66.7%) or global (33.3%) scope. The findings, predominantly correlational, were anatomical (83.3%) or nonanatomical (16.7%). In conclusion, while these XAI techniques were lauded for their predictive power, robustness, testability, and plausibility, limitations included oversimplification, confounding factors, and inconsistencies. The reviewed studies showcased the potential of XAI models while acknowledging current challenges in causality and oversimplification, particularly emphasizing the need for reproducibility.

Trustworthy and ethical AI-enabled cardiovascular care: a rapid review

BACKGROUND

Artificial intelligence (AI) is increasingly used for prevention, diagnosis, monitoring, and treatment of cardiovascular diseases. Despite the potential for AI to improve care, ethical concerns and mistrust in AI-enabled healthcare exist among the public and medical community. Given the rapid and transformative recent growth of AI in cardiovascular care, to inform practice guidelines and regulatory policies that facilitate ethical and trustworthy use of AI in medicine, we conducted a literature review to identify key ethical and trust barriers and facilitators from patients' and healthcare providers' perspectives when using AI in cardiovascular care.

METHODS

In this rapid literature review, we searched six bibliographic databases to identify publications discussing transparency, trust, or ethical concerns (outcomes of interest) associated with AI-based medical devices (interventions of interest) in the context of cardiovascular care from patients', caregivers', or healthcare providers' perspectives. The search was completed on May 24, 2022 and was not limited by date or study design.

RESULTS

After reviewing 7,925 papers from six databases and 3,603 papers identified through citation chasing, 145 articles were included. Key ethical concerns included privacy, security, or confidentiality issues (n = 59, 40.7%); risk of healthcare inequity or disparity (n = 36, 24.8%); risk of patient harm (n = 24, 16.6%); accountability and responsibility concerns (n = 19, 13.1%); problematic informed consent and potential loss of patient autonomy (n = 17, 11.7%); and issues related to data ownership (n = 11, 7.6%). Major trust barriers included data privacy and security concerns, potential risk of patient harm, perceived lack of transparency about AI-enabled medical devices, concerns about AI replacing human aspects of care, concerns about prioritizing profits over patients' interests, and lack of robust evidence related to the accuracy and limitations of AI-based medical devices. Ethical and trust facilitators included ensuring data privacy and data validation, conducting clinical trials in diverse cohorts, providing appropriate training and resources to patients and healthcare providers and improving their engagement in different phases of AI implementation, and establishing further regulatory oversights.

CONCLUSION

This review revealed key ethical concerns and barriers and facilitators of trust in AI-enabled medical devices from patients' and healthcare providers' perspectives. Successful integration of AI into cardiovascular care necessitates implementation of mitigation strategies. These strategies should focus on enhanced regulatory oversight on the use of patient data and promoting transparency around the use of AI in patient care.

Leadership for AI Transformation in Health Care Organization: Scoping Review

BACKGROUND

The leaders of health care organizations are grappling with rising expenses and surging demands for health services. In response, they are increasingly embracing artificial intelligence (AI) technologies to improve patient care delivery, alleviate operational burdens, and efficiently improve health care safety and quality.

OBJECTIVE

In this paper, we map the current literature and synthesize insights on the role of leadership in driving AI transformation within health care organizations.

METHODS

We conducted a comprehensive search across several databases, including MEDLINE (via Ovid), PsycINFO (via Ovid), CINAHL (via EBSCO), Business Source Premier (via EBSCO), and Canadian Business & Current Affairs (via ProQuest), spanning articles published from 2015 to June 2023 discussing AI transformation within the health care sector. Specifically, we focused on empirical studies with a particular emphasis on leadership. We used an inductive, thematic analysis approach to qualitatively map the evidence. The findings were reported in accordance with the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analysis extension for Scoping Reviews) guidelines.

RESULTS

A comprehensive review of 2813 unique abstracts led to the retrieval of 97 full-text articles, with 22 included for detailed assessment. Our literature mapping reveals that successful AI integration within healthcare organizations requires leadership engagement across technological, strategic, operational, and organizational domains. Leaders must demonstrate a blend of technical expertise, adaptive strategies, and strong interpersonal skills to navigate the dynamic healthcare landscape shaped by complex regulatory, technological, and organizational factors.

CONCLUSIONS

In conclusion, leading AI transformation in healthcare requires a multidimensional approach, with leadership across technological, strategic, operational, and organizational domains. Organizations should implement a comprehensive leadership development strategy, including targeted training and cross-functional collaboration, to equip leaders with the skills needed for AI integration. Additionally, when upskilling or recruiting AI talent, priority should be given to individuals with a strong mix of technical expertise, adaptive capacity, and interpersonal acumen, enabling them to navigate the unique complexities of the healthcare environment.

Evolution of digital twins in precision health applications: a scoping review study

An increasing amount of research is incorporating the concept of Digital twin (DT) in biomedical and health care applications. This scoping review aims to summarize existing research and identify gaps in the development and use of DTs in the health care domain. The focus of this study lies on summarizing: the different types of DTs, the techniques employed in DT development, the DT applications in health care, and the data resources used for creating DTs. We identified fifty studies, which mainly focused on creating organ- (n=15) and patient-specific twins (n=30). The research predominantly centers on cardiology, endocrinology, orthopedics, and infectious diseases. Only a few studies used real-world datasets for developing their DTs. However, there remain unresolved questions and promising directions that require further exploration. This review provides valuable reference material and insights for researchers on DTs in health care and highlights gaps and unmet needs in this field.

Data Science Methods for Real-World Evidence Generation in Real-World Data

In the healthcare landscape, data science (DS) methods have emerged as indispensable tools to harness real-world data (RWD) from various data sources such as electronic health records, claim and registry data, and data gathered from digital health technologies. Real-world evidence (RWE) generated from RWD empowers researchers, clinicians, and policymakers with a more comprehensive understanding of real-world patient outcomes. Nevertheless, persistent challenges in RWD (e.g., messiness, voluminousness, heterogeneity, multimodality) and a growing awareness of the need for trustworthy and reliable RWE demand innovative, robust, and valid DS methods for analyzing RWD. In this article, I review some common current DS methods for extracting RWE and valuable insights from complex and diverse RWD. This article encompasses the entire RWE-generation pipeline, from study design with RWD to data preprocessing, exploratory analysis, methods for analyzing RWD, and trustworthiness and reliability guarantees, along with data ethics considerations and open-source tools. This review, tailored for an audience that may not be experts in DS, aspires to offer a systematic review of DS methods and assists readers in selecting suitable DS methods and enhancing the process of RWE generation for addressing their specific challenges.

Acute myocardial infarction prognosis prediction with reliable and interpretable artificial intelligence system

OBJECTIVE

Predicting mortality after acute myocardial infarction (AMI) is crucial for timely prescription and treatment of AMI patients, but there are no appropriate AI systems for clinicians. Our primary goal is to develop a reliable and interpretable AI system and provide some valuable insights regarding short, and long-term mortality.

MATERIALS AND METHODS

We propose the RIAS framework, an end-to-end framework that is designed with reliability and interpretability at its core and automatically optimizes the given model. Using RIAS, clinicians get accurate and reliable predictions which can be used as likelihood, with global and local explanations, and "what if" scenarios to achieve desired outcomes as well.

RESULTS

We apply RIAS to AMI prognosis prediction data which comes from the Korean Acute Myocardial Infarction Registry. We compared FT-Transformer with XGBoost and MLP and found that FT-Transformer has superiority in sensitivity and comparable performance in AUROC and F1 score to XGBoost. Furthermore, RIAS reveals the significance of statin-based medications, beta-blockers, and age on mortality regardless of time period. Lastly, we showcase reliable and interpretable results of RIAS with local explanations and counterfactual examples for several realistic scenarios.

DISCUSSION

RIAS addresses the "black-box" issue in AI by providing both global and local explanations based on SHAP values and reliable predictions, interpretable as actual likelihoods. The system's "what if" counterfactual explanations enable clinicians to simulate patient-specific scenarios under various conditions, enhancing its practical utility.

CONCLUSION

The proposed framework provides reliable and interpretable predictions along with counterfactual examples.

On relevant features for the recurrence prediction of urothelial carcinoma of the bladder

BACKGROUND

Urothelial bladder cancer (UBC) is characterized by a high recurrence rate, which is predicted by scoring systems. However, recent studies show the superiority of Machine Learning (ML) models. Nevertheless, these ML approaches are rarely used in medical practice because most of them are black-box models, that cannot adequately explain how a prediction is made.

OBJECTIVE

We investigate the global feature importance of different ML models. By providing information on the most relevant features, we can facilitate the use of ML in everyday medical practice.

DESIGN, SETTING, AND PARTICIPANTS

The data is provided by the cancer registry Rhineland-Palatinate gGmbH, Germany. It consists of numerical and categorical features of 1,944 patients with UBC. We retrospectively predict 2-year recurrence through ML models using Support Vector Machine, Gradient Boosting, and Artificial Neural Network. We then determine the global feature importance using performance-based Permutation Feature Importance (PFI) and variance-based Feature Importance Ranking Measure (FIRM).

RESULTS

We show reliable recurrence prediction of UBC with 82.02% to 83.89% F1-Score, 83.95% to 84.49% Precision, and an overall performance of 69.20% to 70.82% AUC on testing data, depending on the model. Gradient Boosting performs best among all black-box models with an average F1-Score (83.89%), AUC (70.82%), and Precision (83.95%). Furthermore, we show consistency across PFI and FIRM by identifying the same features as relevant across the different models. These features are exclusively therapeutic measures and are consistent with findings from both medical research and clinical trials.

CONCLUSIONS

We confirm the superiority of ML black-box models in predicting UBC recurrence compared to more traditional logistic regression. In addition, we present an approach that increases the explanatory power of black-box models by identifying the underlying influence of input features, thus facilitating the use of ML in clinical practice and therefore providing improved recurrence prediction through the application of black-box models.

Evaluating distributed-learning on real-world obstetrics data: comparing distributed, centralized and local models

This study focused on comparing distributed learning models with centralized and local models, assessing their efficacy in predicting specific delivery and patient-related outcomes in obstetrics using real-world data. The predictions focus on key moments in the obstetric care process, including discharge and various stages of hospitalization. Our analysis: using 6 different machine learning methods like Decision Trees, Bayesian methods, Stochastic Gradient Descent, K-nearest neighbors, AdaBoost, and Multi-layer Perceptron and 19 different variables with various distributions and types, revealed that distributed models were at least equal, and often superior, to centralized versions and local versions. We also describe thoroughly the preprocessing stage in order to help others implement this method in real-world scenarios. The preprocessing steps included cleaning and harmonizing missing values, handling missing data and encoding categorical variables with multisite logic. Even though the type of machine learning model and the distribution of the outcome variable can impact the result, we reached results of 66% being superior to the centralized and local counterpart and 77% being better than the centralized with AdaBoost. Our experiments also shed light in the preprocessing steps required to implement distributed models in a real-world scenario. Our results advocate for distributed learning as a promising tool for applying machine learning in clinical settings, particularly when privacy and data security are paramount, thus offering a robust solution for privacy-concerned clinical applications.

Genome Sequencing is Critical for Forecasting Outcomes following Congenital Cardiac Surgery

While genome sequencing has transformed medicine by elucidating the genetic underpinnings of both rare and common complex disorders, its utility to predict clinical outcomes remains understudied. Here, we used artificial intelligence (AI) technologies to explore the predictive value of genome sequencing in forecasting clinical outcomes following surgery for congenital heart defects (CHD). We report results for a cohort of 2,253 CHD patients from the Pediatric Cardiac Genomics Consortium with a broad range of complex heart defects, pre- and post-operative clinical variables and exome sequencing. Damaging genotypes in chromatin-modifying and cilia-related genes were associated with an elevated risk of adverse post-operative outcomes, including mortality, cardiac arrest and prolonged mechanical ventilation. The impact of damaging genotypes was further amplified in the context of specific CHD phenotypes, surgical complexity and extra-cardiac anomalies. The absence of a damaging genotype in chromatin-modifying and cilia-related genes was also informative, reducing the risk for adverse postoperative outcomes. Thus, genome sequencing enriches the ability to forecast outcomes following congenital cardiac surgery.

Achieving health equity through conversational AI: A roadmap for design and implementation of inclusive chatbots in healthcare

BACKGROUND

The rapid evolution of conversational and generative artificial intelligence (AI) has led to the increased deployment of AI tools in healthcare settings. While these conversational AI tools promise efficiency and expanded access to healthcare services, there are growing concerns ethically, practically and in terms of inclusivity. This study aimed to identify activities which reduce bias in conversational AI and make their designs and implementation more equitable.

METHODS

A qualitative research approach was employed to develop an analytical framework based on the content analysis of 17 guidelines about AI use in clinical settings. A stakeholder consultation was subsequently conducted with a total of 33 ethnically diverse community members, AI designers, industry experts and relevant health professionals to further develop a roadmap for equitable design and implementation of conversational AI in healthcare. Framework analysis was conducted on the interview data.

RESULTS

A 10-stage roadmap was developed to outline activities relevant to equitable conversational AI design and implementation phases: 1) Conception and planning, 2) Diversity and collaboration, 3) Preliminary research, 4) Co-production, 5) Safety measures, 6) Preliminary testing, 7) Healthcare integration, 8) Service evaluation and auditing, 9) Maintenance, and 10) Termination.

DISCUSSION

We have made specific recommendations to increase conversational AI's equity as part of healthcare services. These emphasise the importance of a collaborative approach and the involvement of patient groups in navigating the rapid evolution of conversational AI technologies. Further research must assess the impact of recommended activities on chatbots' fairness and their ability to reduce health inequalities.

Evaluating capabilities of large language models: Performance of GPT-4 on surgical knowledge assessments

BACKGROUND

Artificial intelligence has the potential to dramatically alter health care by enhancing how we diagnose and treat disease. One promising artificial intelligence model is ChatGPT, a general-purpose large language model trained by OpenAI. ChatGPT has shown human-level performance on several professional and academic benchmarks. We sought to evaluate its performance on surgical knowledge questions and assess the stability of this performance on repeat queries.

METHODS

We evaluated the performance of ChatGPT-4 on questions from the Surgical Council on Resident Education question bank and a second commonly used surgical knowledge assessment, referred to as Data-B. Questions were entered in 2 formats: open-ended and multiple-choice. ChatGPT outputs were assessed for accuracy and insights by surgeon evaluators. We categorized reasons for model errors and the stability of performance on repeat queries.

RESULTS

A total of 167 Surgical Council on Resident Education and 112 Data-B questions were presented to the ChatGPT interface. ChatGPT correctly answered 71.3% and 67.9% of multiple choice and 47.9% and 66.1% of open-ended questions for Surgical Council on Resident Education and Data-B, respectively. For both open-ended and multiple-choice questions, approximately two-thirds of ChatGPT responses contained nonobvious insights. Common reasons for incorrect responses included inaccurate information in a complex question (n = 16, 36.4%), inaccurate information in a fact-based question (n = 11, 25.0%), and accurate information with circumstantial discrepancy (n = 6, 13.6%). Upon repeat query, the answer selected by ChatGPT varied for 36.4% of questions answered incorrectly on the first query; the response accuracy changed for 6/16 (37.5%) questions.

CONCLUSION

Consistent with findings in other academic and professional domains, we demonstrate near or above human-level performance of ChatGPT on surgical knowledge questions from 2 widely used question banks. ChatGPT performed better on multiple-choice than open-ended questions, prompting questions regarding its potential for clinical application. Unique to this study, we demonstrate inconsistency in ChatGPT responses on repeat queries. This finding warrants future consideration including efforts at training large language models to provide the safe and consistent responses required for clinical application. Despite near or above human-level performance on question banks and given these observations, it is unclear whether large language models such as ChatGPT are able to safely assist clinicians in providing care.

A scoping review of fair machine learning techniques when using real-world data

OBJECTIVE

The integration of artificial intelligence (AI) and machine learning (ML) in health care to aid clinical decisions is widespread. However, as AI and ML take important roles in health care, there are concerns about AI and ML associated fairness and bias. That is, an AI tool may have a disparate impact, with its benefits and drawbacks unevenly distributed across societal strata and subpopulations, potentially exacerbating existing health inequities. Thus, the objectives of this scoping review were to summarize existing literature and identify gaps in the topic of tackling algorithmic bias and optimizing fairness in AI/ML models using real-world data (RWD) in health care domains.

METHODS

We conducted a thorough review of techniques for assessing and optimizing AI/ML model fairness in health care when using RWD in health care domains. The focus lies on appraising different quantification metrics for accessing fairness, publicly accessible datasets for ML fairness research, and bias mitigation approaches.

RESULTS

We identified 11 papers that are focused on optimizing model fairness in health care applications. The current research on mitigating bias issues in RWD is limited, both in terms of disease variety and health care applications, as well as the accessibility of public datasets for ML fairness research. Existing studies often indicate positive outcomes when using pre-processing techniques to address algorithmic bias. There remain unresolved questions within the field that require further research, which includes pinpointing the root causes of bias in ML models, broadening fairness research in AI/ML with the use of RWD and exploring its implications in healthcare settings, and evaluating and addressing bias in multi-modal data.

CONCLUSION

This paper provides useful reference material and insights to researchers regarding AI/ML fairness in real-world health care data and reveals the gaps in the field. Fair AI/ML in health care is a burgeoning field that requires a heightened research focus to cover diverse applications and different types of RWD.

Ethics of artificial intelligence in prenatal and pediatric genomic medicine

This paper examines the ethics of introducing emerging forms of artificial intelligence (AI) into prenatal and pediatric genomic medicine. Application of genomic AI to these early life settings has not received much attention in the ethics literature. We focus on three contexts: (1) prenatal genomic sequencing for possible fetal abnormalities, (2) rapid genomic sequencing for critically ill children, and (3) reanalysis of genomic data obtained from children for diagnostic purposes. The paper identifies and discusses various ethical issues in the possible application of genomic AI in these settings, especially as they relate to concepts of beneficence, nonmaleficence, respect for autonomy, justice, transparency, accountability, privacy, and trust. The examination will inform the ethically sound introduction of genomic AI in early human life.

Digital Transformation of Public Health for Noncommunicable Diseases: Narrative Viewpoint of Challenges and Opportunities

The recent SARS-CoV-2 pandemic underscored the effectiveness and rapid deployment of digital public health interventions, notably the digital proximity tracing apps, leveraging Bluetooth capabilities to trace and notify users about potential infection exposures. Backed by renowned organizations such as the World Health Organization and the European Union, digital proximity tracings showcased the promise of digital public health. As the world pivots from pandemic responses, it becomes imperative to address noncommunicable diseases (NCDs) that account for a vast majority of health care expenses and premature disability-adjusted life years lost. The narrative of digital transformation in the realm of NCD public health is distinct from infectious diseases. Public health, with its multifaceted approach from disciplines such as medicine, epidemiology, and psychology, focuses on promoting healthy living and choices through functions categorized as "Assessment," "Policy Development," "Resource Allocation," "Assurance," and "Access." The power of artificial intelligence (AI) in this digital transformation is noteworthy. AI can automate repetitive tasks, facilitating health care providers to prioritize personal interactions, especially those that cannot be digitalized like emotional support. Moreover, AI presents tools for individuals to be proactive in their health management. However, the human touch remains irreplaceable; AI serves as a companion guiding through the health care landscape. Digital evolution, while revolutionary, poses its own set of challenges. Issues of equity and access are at the forefront. Vulnerable populations, whether due to economic constraints, geographical barriers, or digital illiteracy, face the threat of being marginalized further. This transformation mandates an inclusive strategy, focusing on not amplifying existing health disparities but eliminating them. Population-level digital interventions in NCD prevention demand societal agreement. Policies, like smoking bans or sugar taxes, though effective, might affect those not directly benefiting. Hence, all involved parties, from policy makers to the public, should have a balanced perspective on the advantages, risks, and expenses of these digital shifts. For a successful digital shift in public health, especially concerning NCDs, AI's potential to enhance efficiency, effectiveness, user experience, and equity-the "quadruple aim"-is undeniable. However, it is vital that AI-driven initiatives in public health domains remain purposeful, offering improvements without compromising other objectives. The broader success of digital public health hinges on transparent benchmarks and criteria, ensuring maximum benefits without sidelining minorities or vulnerable groups. Especially in population-centric decisions, like resource allocation, AI's ability to avoid bias is paramount. Therefore, the continuous involvement of stakeholders, including patients and minority groups, remains pivotal in the progression of AI-integrated digital public health.

Explainable machine learning for predicting conversion to neurological disease: Results from 52,939 medical records

Objective

This study assesses the application of interpretable machine learning modeling using electronic medical record data for the prediction of conversion to neurological disease.

Methods

A retrospective dataset of Cleveland Clinic patients diagnosed with Alzheimer's disease, amyotrophic lateral sclerosis, multiple sclerosis, or Parkinson's disease, and matched controls based on age, sex, race, and ethnicity was compiled. Individualized risk prediction models were created using eXtreme Gradient Boosting for each neurological disease at four timepoints in patient history. The prediction models were assessed for transparency and fairness.

Results

At timepoints 0-months, 12-months, 24-months, and 60-months prior to diagnosis, Alzheimer's disease models achieved the area under the receiver operating characteristic curve on a holdout test dataset of 0.794, 0.742, 0.709, and 0.645; amyotrophic lateral sclerosis of 0.883, 0.710, 0.658, and 0.620; multiple sclerosis of 0.922, 0.877, 0.849, and 0.781; and Parkinson's disease of 0.809, 0.738, 0.700, and 0.651, respectively.

Conclusions

The results demonstrate that electronic medical records contain latent information that can be used for risk stratification for neurological disorders. In particular, patient-reported outcomes, sleep assessments, falls data, additional disease diagnoses, and longitudinal changes in patient health, such as weight change, are important predictors.

Predicting preterm birth using explainable machine learning in a prospective cohort of nulliparous and multiparous pregnant women

Preterm birth (PTB) presents a complex challenge in pregnancy, often leading to significant perinatal and long-term morbidities. "While machine learning (ML) algorithms have shown promise in PTB prediction, the lack of interpretability in existing models hinders their clinical utility. This study aimed to predict PTB in a pregnant population using ML models, identify the key risk factors associated with PTB through the SHapley Additive exPlanations (SHAP) algorithm, and provide comprehensive explanations for these predictions to assist clinicians in providing appropriate care. This study analyzed a dataset of 3509 pregnant women in the United Arab Emirates and selected 35 risk factors associated with PTB based on the existing medical and artificial intelligence literature. Six ML algorithms were tested, wherein the XGBoost model exhibited the best performance, with an area under the operator receiving curves of 0.735 and 0.723 for parous and nulliparous women, respectively. The SHAP feature attribution framework was employed to identify the most significant risk factors linked to PTB. Additionally, individual patient analysis was performed using the SHAP and the local interpretable model-agnostic explanation algorithms (LIME). The overall incidence of PTB was 11.23% (11 and 12.1% in parous and nulliparous women, respectively). The main risk factors associated with PTB in parous women are previous PTB, previous cesarean section, preeclampsia during pregnancy, and maternal age. In nulliparous women, body mass index at delivery, maternal age, and the presence of amniotic infection were the most relevant risk factors. The trained ML prediction model developed in this study holds promise as a valuable screening tool for predicting PTB within this specific population. Furthermore, SHAP and LIME analyses can assist clinicians in understanding the individualized impact of each risk factor on their patients and provide appropriate care to reduce morbidity and mortality related to PTB.

Synthetic Health Data Can Augment Community Research Efforts to Better Inform the Public During Emerging Pandemics

The COVID-19 pandemic had disproportionate effects on the Veteran population due to the increased prevalence of medical and environmental risk factors. Synthetic electronic health record (EHR) data can help meet the acute need for Veteran population-specific predictive modeling efforts by avoiding the strict barriers to access, currently present within Veteran Health Administration (VHA) datasets. The U.S. Food and Drug Administration (FDA) and the VHA launched the precisionFDA COVID-19 Risk Factor Modeling Challenge to develop COVID-19 diagnostic and prognostic models; identify Veteran population-specific risk factors; and test the usefulness of synthetic data as a substitute for real data. The use of synthetic data boosted challenge participation by providing a dataset that was accessible to all competitors. Models trained on synthetic data showed similar but systematically inflated model performance metrics to those trained on real data. The important risk factors identified in the synthetic data largely overlapped with those identified from the real data, and both sets of risk factors were validated in the literature. Tradeoffs exist between synthetic data generation approaches based on whether a real EHR dataset is required as input. Synthetic data generated directly from real EHR input will more closely align with the characteristics of the relevant cohort. This work shows that synthetic EHR data will have practical value to the Veterans' health research community for the foreseeable future.

Standardized image-based polysomnography database and deep learning algorithm for sleep-stage classification

STUDY OBJECTIVES

Polysomnography (PSG) scoring is labor-intensive, subjective, and often ambiguous. Recently several deep learning (DL) models for automated sleep scoring have been developed, they are tied to a fixed amount of input channels and resolution. In this study, we constructed a standardized image-based PSG dataset in order to overcome the heterogeneity of raw signal data obtained from various PSG devices and various sleep laboratory environments.

METHODS

All individually exported European data format files containing raw signals were converted into images with an annotation file, which contained the demographics, diagnoses, and sleep statistics. An image-based DL model for automatic sleep staging was developed, compared with a signal-based model, and validated in an external dataset.

RESULTS

We constructed 10253 image-based PSG datasets using a standardized format. Among these, 7745 diagnostic PSG data were used to develop our DL model. The DL model using the image dataset showed similar performance to the signal-based dataset for the same subject. The overall DL accuracy was greater than 80%, even with severe obstructive sleep apnea. Moreover, for the first time, we showed explainable DL in the field of sleep medicine as visualized key inference regions using Eigen-class activation maps. Furthermore, when a DL model for sleep scoring performs external validation, we achieved a relatively good performance.

CONCLUSIONS

Our main contribution demonstrates the availability of a standardized image-based dataset, and highlights that changing the data sampling rate or number of sensors may not require retraining, although performance decreases slightly as the number of sensors decreases.

A Fair Individualized Polysocial Risk Score for Identifying Increased Social Risk in Type 2 Diabetes

Background

Racial and ethnic minority groups and individuals facing social disadvantages, which often stem from their social determinants of health (SDoH), bear a disproportionate burden of type 2 diabetes (T2D) and its complications. It is crucial to implement effective social risk management strategies at the point of care.

Objective

To develop an electronic health records (EHR)-based machine learning (ML) analytical pipeline to address unmet social needs associated with hospitalization risk in patients with T2D.

Methods

We identified real-world patients with T2D from the EHR data from University of Florida (UF) Health Integrated Data Repository (IDR), incorporating both contextual SDoH (e.g., neighborhood deprivation) and individual-level SDoH (e.g., housing instability). The 2015-2020 data were used for training and validation and 2021-2022 data for independent testing. We developed a machine learning analytic pipeline, namely individualized polysocial risk score (iPsRS), to identify high social risk associated with hospitalizations in T2D patients, along with explainable AI (XAI) and fairness optimization.

Results

The study cohort included 10,192 real-world patients with T2D, with a mean age of 59 years and 58% female. Of the cohort, 50% were non-Hispanic White, 39% were non-Hispanic Black, 6% were Hispanic, and 5% were other races/ethnicities. Our iPsRS, including both contextual and individual-level SDoH as input factors, achieved a C statistic of 0.72 in predicting 1-year hospitalization after fairness optimization across racial and ethnic groups. The iPsRS showed excellent utility for capturing individuals at high hospitalization risk because of SDoH, that is, the actual 1-year hospitalization rate in the top 5% of iPsRS was 28.1%, ~13 times as high as the bottom decile (2.2% for 1-year hospitalization rate).

Conclusion

Our ML pipeline iPsRS can fairly and accurately screen for patients who have increased social risk leading to hospitalization in real word patients with T2D.

EAPR: explainable and augmented patient representation learning for disease prediction

Patient representation learning aims to encode meaningful information about the patient's Electronic Health Records (EHR) in the form of a mathematical representation. Recent advances in deep learning have empowered Patient representation learning methods with greater representational power, allowing the learned representations to significantly improve the performance of disease prediction models. However, the inherent shortcomings of deep learning models, such as the need for massive amounts of labeled data and inexplicability, limit the performance of deep learning-based Patient representation learning methods to further improvements. In particular, learning robust patient representations is challenging when patient data is missing or insufficient. Although data augmentation techniques can tackle this deficiency, the complex data processing further weakens the inexplicability of patient representation learning models. To address the above challenges, this paper proposes an Explainable and Augmented Patient Representation Learning for disease prediction (EAPR). EAPR utilizes data augmentation controlled by confidence interval to enhance patient representation in the presence of limited patient data. Moreover, EAPR proposes to use two-stage gradient backpropagation to address the problem of unexplainable patient representation learning models due to the complex data enhancement process. The experimental results on real clinical data validate the effectiveness and explainability of the proposed approach.

Patient-level explainable machine learning to predict major adverse cardiovascular events from SPECT MPI and CCTA imaging

BACKGROUND

Machine learning (ML) has shown promise in improving the risk prediction in non-invasive cardiovascular imaging, including SPECT MPI and coronary CT angiography. However, most algorithms used remain black boxes to clinicians in how they compute their predictions. Furthermore, objective consideration of the multitude of available clinical data, along with the visual and quantitative assessments from CCTA and SPECT, are critical for optimal patient risk stratification. We aim to provide an explainable ML approach to predict MACE using clinical, CCTA, and SPECT data.

METHODS

Consecutive patients who underwent clinically indicated CCTA and SPECT myocardial imaging for suspected CAD were included and followed up for MACEs. A MACE was defined as a composite outcome that included all-cause mortality, myocardial infarction, or late revascularization. We employed an Automated Machine Learning (AutoML) approach to predict MACE using clinical, CCTA, and SPECT data. Various mainstream models with different sets of hyperparameters have been explored, and critical predictors of risk are obtained using explainable techniques on the global and patient levels. Ten-fold cross-validation was used in training and evaluating the AutoML model.

RESULTS

A total of 956 patients were included (mean age 61.1 ±14.2 years, 54% men, 89% hypertension, 81% diabetes, 84% dyslipidemia). Obstructive CAD on CCTA and ischemia on SPECT were observed in 14% of patients, and 11% experienced MACE. ML prediction's sensitivity, specificity, and accuracy in predicting a MACE were 69.61%, 99.77%, and 96.54%, respectively. The top 10 global predictive features included 8 CCTA attributes (segment involvement score, number of vessels with severe plaque ≥70, ≥50% stenosis in the left marginal coronary artery, calcified plaque, ≥50% stenosis in the left circumflex coronary artery, plaque type in the left marginal coronary artery, stenosis degree in the second obtuse marginal of the left circumflex artery, and stenosis category in the marginals of the left circumflex artery) and 2 clinical features (past medical history of MI or left bundle branch block, being an ever smoker).

CONCLUSION

ML can accurately predict risk of developing a MACE in patients suspected of CAD undergoing SPECT MPI and CCTA. ML feature-ranking can also show, at a sample- as well as at a patient-level, which features are key in making such a prediction.

Integrating Medical Domain Knowledge for Early Diagnosis of Fever of Unknown Origin: An Interpretable Hierarchical Multimodal Neural Network Approach

Accurate and interpretable differential diagnostic technologies are crucial for supporting clinicians in decision-making and treatment-planning for patients with fever of unknown origin (FUO). Existing solutions commonly address the diagnosis of FUO by transforming it into a multi-classification task. However, after the emergence of COVID-19 pandemic, clinicians have recognized the heightened significance of early diagnosis in patients with FUO, particularly for practical needs such as early triage. This has resulted in increased demands for identifying a wider range of etiologies, shorter observation windows, and better model interpretability. In this article, we propose an interpretable hierarchical multimodal neural network framework (iHMNNF) to facilitate early diagnosis of FUO by incorporating medical domain knowledge and leveraging multimodal clinical data. The iHMNNF comprises a top-down hierarchical reasoning framework (Td-HRF) built on the class hierarchy of FUO etiologies, five local attention-based multimodal neural networks (La-MNNs) trained for each parent node of the class hierarchy, and an interpretable module based on layer-wise relevance propagation (LRP) and attention mechanism. Experimental datasets were collected from electronic health records (EHRs) at a large-scale tertiary grade-A hospital in China, comprising 34,051 hospital admissions of 30,794 FUO patients from January 2011 to October 2020. Our proposed La-MNNs achieved area under the receiver operating characteristic curve (AUROC) values ranging from 0.7809 to 0.9035 across all five decomposed tasks, surpassing competing machine learning (ML) and single-modality deep learning (DL) methods while also providing enhanced interpretability. Furthermore, we explored the feasibility of identifying FUO etiologies using only the first N-hour time series data obtained after admission.

A historical perspective of biomedical explainable AI research

The black-box nature of most artificial intelligence (AI) models encourages the development of explainability methods to engender trust into the AI decision-making process. Such methods can be broadly categorized into two main types: post hoc explanations and inherently interpretable algorithms. We aimed at analyzing the possible associations between COVID-19 and the push of explainable AI (XAI) to the forefront of biomedical research. We automatically extracted from the PubMed database biomedical XAI studies related to concepts of causality or explainability and manually labeled 1,603 papers with respect to XAI categories. To compare the trends pre- and post-COVID-19, we fit a change point detection model and evaluated significant changes in publication rates. We show that the advent of COVID-19 in the beginning of 2020 could be the driving factor behind an increased focus concerning XAI, playing a crucial role in accelerating an already evolving trend. Finally, we present a discussion with future societal use and impact of XAI technologies and potential future directions for those who pursue fostering clinical trust with interpretable machine learning models.

Perspective of Artificial Intelligence in Disease Diagnosis: A Review of Current and Future Endeavours in the Medical Field

Artificial intelligence (AI) has demonstrated significant promise for the present and future diagnosis of diseases. At the moment, AI-powered diagnostic technologies can help physicians decipher medical pictures like X-rays, magnetic resonance imaging, and computed tomography scans, resulting in quicker and more precise diagnoses. In order to make a prospective diagnosis, AI algorithms may also examine patient information, symptoms, and medical background. The application of AI in disease diagnosis is anticipated to grow as the field develops. In the future, AI may be used to find patterns in enormous volumes of medical data, aiding in disease prediction and prevention before symptoms appear. Additionally, by combining genetic data, lifestyle data, and environmental variables, AI may help in the diagnosis of complicated diseases. It is crucial to remember that while AI can be a powerful tool, it cannot take the place of qualified medical personnel. Instead, AI ought to support and improve diagnostic procedures, enhancing patient care and healthcare results. Future research and the use of AI for disease diagnosis must take ethical issues, data protection, and ongoing model validation into account.

Can Attention Be Used to Explain EHR-Based Mortality Prediction Tasks: A Case Study on Hemorrhagic Stroke

Stroke is a significant cause of mortality and morbidity, necessitating early predictive strategies to minimize risks. Traditional methods for evaluating patients, such as Acute Physiology and Chronic Health Evaluation (APACHE II, IV) and Simplified Acute Physiology Score III (SAPS III), have limited accuracy and interpretability. This paper proposes a novel approach: an interpretable, attention-based transformer model for early stroke mortality prediction. This model seeks to address the limitations of previous predictive models, providing both interpretability (providing clear, understandable explanations of the model) and fidelity (giving a truthful explanation of the model's dynamics from input to output). Furthermore, the study explores and compares fidelity and interpretability scores using Shapley values and attention-based scores to improve model explainability. The research objectives include designing an interpretable attention-based transformer model, evaluating its performance compared to existing models, and providing feature importance derived from the model.

Evaluating Capabilities of Large Language Models: Performance of GPT4 on Surgical Knowledge Assessments

Background

Artificial intelligence (AI) has the potential to dramatically alter healthcare by enhancing how we diagnosis and treat disease. One promising AI model is ChatGPT, a large general-purpose language model trained by OpenAI. The chat interface has shown robust, human-level performance on several professional and academic benchmarks. We sought to probe its performance and stability over time on surgical case questions.

Methods

We evaluated the performance of ChatGPT-4 on two surgical knowledge assessments: the Surgical Council on Resident Education (SCORE) and a second commonly used knowledge assessment, referred to as Data-B. Questions were entered in two formats: open-ended and multiple choice. ChatGPT output were assessed for accuracy and insights by surgeon evaluators. We categorized reasons for model errors and the stability of performance on repeat encounters.

Results

A total of 167 SCORE and 112 Data-B questions were presented to the ChatGPT interface. ChatGPT correctly answered 71% and 68% of multiple-choice SCORE and Data-B questions, respectively. For both open-ended and multiple-choice questions, approximately two-thirds of ChatGPT responses contained non-obvious insights. Common reasons for inaccurate responses included: inaccurate information in a complex question (n=16, 36.4%); inaccurate information in fact-based question (n=11, 25.0%); and accurate information with circumstantial discrepancy (n=6, 13.6%). Upon repeat query, the answer selected by ChatGPT varied for 36.4% of inaccurate questions; the response accuracy changed for 6/16 questions.

Conclusion

Consistent with prior findings, we demonstrate robust near or above human-level performance of ChatGPT within the surgical domain. Unique to this study, we demonstrate a substantial inconsistency in ChatGPT responses with repeat query. This finding warrants future consideration and presents an opportunity to further train these models to provide safe and consistent responses. Without mental and/or conceptual models, it is unclear whether language models such as ChatGPT would be able to safely assist clinicians in providing care.

Mutation-Attention (MuAt): deep representation learning of somatic mutations for tumour typing and subtyping

BACKGROUND

Cancer genome sequencing enables accurate classification of tumours and tumour subtypes. However, prediction performance is still limited using exome-only sequencing and for tumour types with low somatic mutation burden such as many paediatric tumours. Moreover, the ability to leverage deep representation learning in discovery of tumour entities remains unknown.

METHODS

We introduce here Mutation-Attention (MuAt), a deep neural network to learn representations of simple and complex somatic alterations for prediction of tumour types and subtypes. In contrast to many previous methods, MuAt utilizes the attention mechanism on individual mutations instead of aggregated mutation counts.

RESULTS

We trained MuAt models on 2587 whole cancer genomes (24 tumour types) from the Pan-Cancer Analysis of Whole Genomes (PCAWG) and 7352 cancer exomes (20 types) from the Cancer Genome Atlas (TCGA). MuAt achieved prediction accuracy of 89% for whole genomes and 64% for whole exomes, and a top-5 accuracy of 97% and 90%, respectively. MuAt models were found to be well-calibrated and perform well in three independent whole cancer genome cohorts with 10,361 tumours in total. We show MuAt to be able to learn clinically and biologically relevant tumour entities including acral melanoma, SHH-activated medulloblastoma, SPOP-associated prostate cancer, microsatellite instability, POLE proofreading deficiency, and MUTYH-associated pancreatic endocrine tumours without these tumour subtypes and subgroups being provided as training labels. Finally, scrunity of MuAt attention matrices revealed both ubiquitous and tumour-type specific patterns of simple and complex somatic mutations.

CONCLUSIONS

Integrated representations of somatic alterations learnt by MuAt were able to accurately identify histological tumour types and identify tumour entities, with potential to impact precision cancer medicine.

Learning Representations from Medical Text for Effective Diagnoses and Knowledge Discovery

Discovering knowledge and effectively predicting target events are two main goals of medical text mining. However, few models can achieve them simultaneously. In this study, we investigated the possibility of discovering knowledge and predicting diagnosis at once via raw medical text. We proposed the Enhanced Neural Topic Model (ENTM), a variant of the neural topic model, to learn interpretable representations. We introduced the auxiliary loss set to improve the effectiveness of learned representations. Then, we used learned representations to train a softmax regression model to predict target events. As each element in representations learned by the ENTM has an explicit semantic meaning, weights in softmax regression represent potential knowledge of whether an element is a significant factor in predicting diagnosis. We adopted two independent medical text datasets to evaluate our ENTM model. Results indicate that our model performed better than the latest pretrained neural language models. Meanwhile, analysis of model parameters indicates that our model has the potential discover knowledge from data.Clinical relevance- This work provides a model that can effectively predict patient diagnosis and has the potential to discover knowledge from medical text.

Prediction of Outcomes After Heart Transplantation in Pediatric Patients Using National Registry Data: Evaluation of Machine Learning Approaches

BACKGROUND

The prediction of posttransplant health outcomes for pediatric heart transplantation is critical for risk stratification and high-quality posttransplant care.

OBJECTIVE

The purpose of this study was to examine the use of machine learning (ML) models to predict rejection and mortality for pediatric heart transplant recipients.

METHODS

Various ML models were used to predict rejection and mortality at 1, 3, and 5 years after transplantation in pediatric heart transplant recipients using United Network for Organ Sharing data from 1987 to 2019. The variables used for predicting posttransplant outcomes included donor and recipient as well as medical and social factors. We evaluated 7 ML models-extreme gradient boosting (XGBoost), logistic regression, support vector machine, random forest (RF), stochastic gradient descent, multilayer perceptron, and adaptive boosting (AdaBoost)-as well as a deep learning model with 2 hidden layers with 100 neurons and a rectified linear unit (ReLU) activation function followed by batch normalization for each and a classification head with a softmax activation function. We used 10-fold cross-validation to evaluate model performance. Shapley additive explanations (SHAP) values were calculated to estimate the importance of each variable for prediction.

RESULTS

RF and AdaBoost models were the best-performing algorithms for different prediction windows across outcomes. RF outperformed other ML algorithms in predicting 5 of the 6 outcomes (area under the receiver operating characteristic curve [AUROC] 0.664 and 0.706 for 1-year and 3-year rejection, respectively, and AUROC 0.697, 0.758, and 0.763 for 1-year, 3-year, and 5-year mortality, respectively). AdaBoost achieved the best performance for prediction of 5-year rejection (AUROC 0.705).

CONCLUSIONS

This study demonstrates the comparative utility of ML approaches for modeling posttransplant health outcomes using registry data. ML approaches can identify unique risk factors and their complex relationship with outcomes, thereby identifying patients considered to be at risk and informing the transplant community about the potential of these innovative approaches to improve pediatric care after heart transplantation. Future studies are required to translate the information derived from prediction models to optimize counseling, clinical care, and decision-making within pediatric organ transplant centers.

Conceptual modelling for life sciences based on systemist foundations

BACKGROUND

All aspects of our society, including the life sciences, need a mechanism for people working within them to represent the concepts they employ to carry out their research. For the information systems being designed and developed to support researchers and scientists in conducting their work, conceptual models of the relevant domains are usually designed as both blueprints for a system being developed and as a means of communication between the designer and developer. Most conceptual modelling concepts are generic in the sense that they are applied with the same understanding across many applications. Problems in the life sciences, however, are especially complex and important, because they deal with humans, their well-being, and their interactions with the environment as well as other organisms.

RESULTS

This work proposes a "systemist" perspective for creating a conceptual model of a life scientist's problem. We introduce the notion of a system and then show how it can be applied to the development of an information system for handling genomic-related information. We extend our discussion to show how the proposed systemist perspective can support the modelling of precision medicine.

CONCLUSION

This research recognizes challenges in life sciences research of how to model problems to better represent the connections between physical and digital worlds. We propose a new notation that explicitly incorporates systemist thinking, as well as the components of systems based on recent ontological foundations. The new notation captures important semantics in the domain of life sciences. It may be used to facilitate understanding, communication and problem-solving more broadly. We also provide a precise, sound, ontologically supported characterization of the term "system," as a basic construct for conceptual modelling in life sciences.

XAI-reduct: accuracy preservation despite dimensionality reduction for heart disease classification using explainable AI

Machine learning (ML) has been used for classification of heart diseases for almost a decade, although understanding of the internal working of the black boxes, i.e., non-interpretable models, remain a demanding problem. Another major challenge in such ML models is the curse of dimensionality leading to resource intensive classification using the comprehensive set of feature vector (CFV). This study focuses on dimensionality reduction using explainable artificial intelligence, without negotiating on accuracy for heart disease classification. Four explainable ML models, using SHAP, were used for classification which reflected the feature contributions (FC) and feature weights (FW) for each feature in the CFV for generating the final results. FC and FW were taken into account in generating the reduced dimensional feature subset (FS). The findings of the study are as follows: (a) XGBoost classifies heart diseases best with explanations, with an increase in 2% in model accuracy over existing best proposals, (b) explainable classification using FS exhibits better accuracy than most of the literary proposals, and (c) with the increase in explainability, accuracy can be preserved using XGBoost classifier for classifying heart diseases, and (d) the top four features responsible for diagnosis of heart disease have been exhibited which have common occurrences in all the explanations reflected by the five explainable techniques used on XGBoost classifier based on feature contributions. To the best of our knowledge, this is first attempt to explain XGBoost classification for diagnosis of heart diseases using five explainable techniques.

Prediction of disease comorbidity using explainable artificial intelligence and machine learning techniques: A systematic review

OBJECTIVE

Disease comorbidity is a major challenge in healthcare affecting the patient's quality of life and costs. AI-based prediction of comorbidities can overcome this issue by improving precision medicine and providing holistic care. The objective of this systematic literature review was to identify and summarise existing machine learning (ML) methods for comorbidity prediction and evaluate the interpretability and explainability of the models.

MATERIALS AND METHODS

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) framework was used to identify articles in three databases: Ovid Medline, Web of Science and PubMed. The literature search covered a broad range of terms for the prediction of disease comorbidity and ML, including traditional predictive modelling.

RESULTS

Of 829 unique articles, 58 full-text papers were assessed for eligibility. A final set of 22 articles with 61 ML models was included in this review. Of the identified ML models, 33 models achieved relatively high accuracy (80-95%) and AUC (0.80-0.89). Overall, 72% of studies had high or unclear concerns regarding the risk of bias.

DISCUSSION

This systematic review is the first to examine the use of ML and explainable artificial intelligence (XAI) methods for comorbidity prediction. The chosen studies focused on a limited scope of comorbidities ranging from 1 to 34 (mean = 6), and no novel comorbidities were found due to limited phenotypic and genetic data. The lack of standard evaluation for XAI hinders fair comparisons.

CONCLUSION

A broad range of ML methods has been used to predict the comorbidities of various disorders. With further development of explainable ML capacity in the field of comorbidity prediction, there is a significant possibility of identifying unmet health needs by highlighting comorbidities in patient groups that were not previously recognised to be at risk for particular comorbidities.

Essential properties and explanation effectiveness of explainable artificial intelligence in healthcare: A systematic review

Background

Significant advancements in the field of information technology have influenced the creation of trustworthy explainable artificial intelligence (XAI) in healthcare. Despite improved performance of XAI, XAI techniques have not yet been integrated into real-time patient care.

Objective

The aim of this systematic review is to understand the trends and gaps in research on XAI through an assessment of the essential properties of XAI and an evaluation of explanation effectiveness in the healthcare field.

Methods

A search of PubMed and Embase databases for relevant peer-reviewed articles on development of an XAI model using clinical data and evaluating explanation effectiveness published between January 1, 2011, and April 30, 2022, was conducted. All retrieved papers were screened independently by the two authors. Relevant papers were also reviewed for identification of the essential properties of XAI (e.g., stakeholders and objectives of XAI, quality of personalized explanations) and the measures of explanation effectiveness (e.g., mental model, user satisfaction, trust assessment, task performance, and correctability).

Results

Six out of 882 articles met the criteria for eligibility. Artificial Intelligence (AI) users were the most frequently described stakeholders. XAI served various purposes, including evaluation, justification, improvement, and learning from AI. Evaluation of the quality of personalized explanations was based on fidelity, explanatory power, interpretability, and plausibility. User satisfaction was the most frequently used measure of explanation effectiveness, followed by trust assessment, correctability, and task performance. The methods of assessing these measures also varied.

Conclusion

XAI research should address the lack of a comprehensive and agreed-upon framework for explaining XAI and standardized approaches for evaluating the effectiveness of the explanation that XAI provides to diverse AI stakeholders.

Survey of explainable artificial intelligence techniques for biomedical imaging with deep neural networks

Artificial Intelligence (AI) techniques of deep learning have revolutionized the disease diagnosis with their outstanding image classification performance. In spite of the outstanding results, the widespread adoption of these techniques in clinical practice is still taking place at a moderate pace. One of the major hindrance is that a trained Deep Neural Networks (DNN) model provides a prediction, but questions about why and how that prediction was made remain unanswered. This linkage is of utmost importance for the regulated healthcare domain to increase the trust in the automated diagnosis system by the practitioners, patients and other stakeholders. The application of deep learning for medical imaging has to be interpreted with caution due to the health and safety concerns similar to blame attribution in the case of an accident involving autonomous cars. The consequences of both a false positive and false negative cases are far reaching for patients' welfare and cannot be ignored. This is exacerbated by the fact that the state-of-the-art deep learning algorithms comprise of complex interconnected structures, millions of parameters, and a 'black box' nature, offering little understanding of their inner working unlike the traditional machine learning algorithms. Explainable AI (XAI) techniques help to understand model predictions which help develop trust in the system, accelerate the disease diagnosis, and meet adherence to regulatory requirements. This survey provides a comprehensive review of the promising field of XAI for biomedical imaging diagnostics. We also provide a categorization of the XAI techniques, discuss the open challenges, and provide future directions for XAI which would be of interest to clinicians, regulators and model developers.

Explainable synthetic image generation to improve risk assessment of rare pediatric heart transplant rejection

Expert microscopic analysis of cells obtained from frequent heart biopsies is vital for early detection of pediatric heart transplant rejection to prevent heart failure. Detection of this rare condition is prone to low levels of expert agreement due to the difficulty of identifying subtle rejection signs within biopsy samples. The rarity of pediatric heart transplant rejection also means that very few gold-standard images are available for developing machine learning models. To solve this urgent clinical challenge, we developed a deep learning model to automatically quantify rejection risk within digital images of biopsied tissue using an explainable synthetic data augmentation approach. We developed this explainable AI framework to illustrate how our progressive and inspirational generative adversarial network models distinguish between normal tissue images and those containing cellular rejection signs. To quantify biopsy-level rejection risk, we first detect local rejection features using a binary image classifier trained with expert-annotated and synthetic examples. We converted these local predictions into a biopsy-wide rejection score via an interpretable histogram-based approach. Our model significantly improves upon prior works with the same dataset with an area under the receiver operating curve (AUROC) of 98.84% for the local rejection detection task and 95.56% for the biopsy-rejection prediction task. A biopsy-level sensitivity of 83.33% makes our approach suitable for early screening of biopsies to prioritize expert analysis. Our framework provides a solution to rare medical imaging challenges currently limited by small datasets.

Remembering Paul E. Meehl: Historical Contributions to Predictive Modeling in Human Behavior

ABSTRACT

This article briefly examines the life and work of the late clinical psychologist and philosopher of science Paul E. Meehl. His thesis in Clinical versus Statistical Prediction (1954) that the data combination performed by mechanical operations, as compared to clinicians, achieves higher accuracy in predicting human behavior is one of the earliest theoretical works that laid the groundwork for utilizing statistics and computational modeling in research in psychiatry and clinical psychology. For today's psychiatric researchers and clinicians grappling with the challenges of translating the ever-increasing data of the human mind into practice tools, Meehl's advocacy for both accurate modeling of the data and their clinically relevant use is timely.

Artificial Intelligence and Cancer Control: Toward Prioritizing Justice, Equity, Diversity, and Inclusion (JEDI) in Emerging Decision Support Technologies

PURPOSE FOR REVIEW

This perspective piece has two goals: first, to describe issues related to artificial intelligence-based applications for cancer control as they may impact health inequities or disparities; and second, to report on a review of systematic reviews and meta-analyses of artificial intelligence-based tools for cancer control to ascertain the extent to which discussions of justice, equity, diversity, inclusion, or health disparities manifest in syntheses of the field's best evidence.

RECENT FINDINGS

We found that, while a significant proportion of existing syntheses of research on AI-based tools in cancer control use formal bias assessment tools, the fairness or equitability of models is not yet systematically analyzable across studies. Issues related to real-world use of AI-based tools for cancer control, such as workflow considerations, measures of usability and acceptance, or tool architecture, are more visible in the literature, but still addressed only in a minority of reviews. Artificial intelligence is poised to bring significant benefits to a wide range of applications in cancer control, but more thorough and standardized evaluations and reporting of model fairness are required to build the evidence base for AI-based tool design for cancer and to ensure that these emerging technologies promote equitable healthcare.

Predict, diagnose, and treat chronic kidney disease with machine learning: a systematic literature review

OBJECTIVES

In this systematic review we aimed at assessing how artificial intelligence (AI), including machine learning (ML) techniques have been deployed to predict, diagnose, and treat chronic kidney disease (CKD). We systematically reviewed the available evidence on these innovative techniques to improve CKD diagnosis and patient management.

METHODS

We included English language studies retrieved from PubMed. The review is therefore to be classified as a "rapid review", since it includes one database only, and has language restrictions; the novelty and importance of the issue make missing relevant papers unlikely. We extracted 16 variables, including: main aim, studied population, data source, sample size, problem type (regression, classification), predictors used, and performance metrics. We followed the Preferred Reporting Items for Systematic Reviews (PRISMA) approach; all main steps were done in duplicate.

RESULTS

From a total of 648 studies initially retrieved, 68 articles met the inclusion criteria. Models, as reported by authors, performed well, but the reported metrics were not homogeneous across articles and therefore direct comparison was not feasible. The most common aim was prediction of prognosis, followed by diagnosis of CKD. Algorithm generalizability, and testing on diverse populations was rarely taken into account. Furthermore, the clinical evaluation and validation of the models/algorithms was perused; only a fraction of the included studies, 6 out of 68, were performed in a clinical context.

CONCLUSIONS

Machine learning is a promising tool for the prediction of risk, diagnosis, and therapy management for CKD patients. Nonetheless, future work is needed to address the interpretability, generalizability, and fairness of the models to ensure the safe application of such technologies in routine clinical practice.

Human Activity Recognition with an HMM-Based Generative Model

Human activity recognition (HAR) has become an interesting topic in healthcare. This application is important in various domains, such as health monitoring, supporting elders, and disease diagnosis. Considering the increasing improvements in smart devices, large amounts of data are generated in our daily lives. In this work, we propose unsupervised, scaled, Dirichlet-based hidden Markov models to analyze human activities. Our motivation is that human activities have sequential patterns and hidden Markov models (HMMs) are some of the strongest statistical models used for modeling data with continuous flow. In this paper, we assume that emission probabilities in HMM follow a bounded-scaled Dirichlet distribution, which is a proper choice in modeling proportional data. To learn our model, we applied the variational inference approach. We used a publicly available dataset to evaluate the performance of our proposed model.

Bibliometric analysis on the adoption of artificial intelligence applications in the e-health sector

Artificial Intelligent (AI) applications in e-health have evolved considerably in the last 25 years. To track the current research progress in this field, there is a need to analyze the most recent trend of adopting AI applications in e-health. This bibliometric analysis study covers AI applications in e-health. It differs from the existing literature review as the journal articles are obtained from the Scopus database from its beginning to late 2021 (25 years), which depicts the most recent trend of AI in e-health. The bibliometric analysis is employed to find the statistical and quantitative analysis of available literature of a specific field of study for a particular period. An extensive global literature review is performed to identify the significant research area, authors, or their relationship through published articles. It also provides the researchers with an overview of the work evolution of specific research fields. The study's main contribution highlights the essential authors, journals, institutes, keywords, and states in developing the AI field in e-health.

Artificial Intelligence in Medicine: Microbiome-Based Machine Learning for Phenotypic Classification

Advanced computational approaches in artificial intelligence, such as machine learning, have been increasingly applied in life sciences and healthcare to analyze large-scale complex biological data, such as microbiome data. In this chapter, we describe the experimental procedures for using microbiome-based machine learning models for phenotypic classification.

Why did AI get this one wrong? - Tree-based explanations of machine learning model predictions

Increasingly complex learning methods such as boosting, bagging and deep learning have made ML models more accurate, but harder to interpret and explain, culminating in black-box machine learning models. Model developers and users alike are often presented with a trade-off between performance and intelligibility, especially in high-stakes applications like medicine. In the present article we propose a novel methodological approach for generating explanations for the predictions of a generic machine learning model, given a specific instance for which the prediction has been made. The method, named AraucanaXAI, is based on surrogate, locally-fitted classification and regression trees that are used to provide post-hoc explanations of the prediction of a generic machine learning model. Advantages of the proposed XAI approach include superior fidelity to the original model, ability to deal with non-linear decision boundaries, and native support to both classification and regression problems. We provide a packaged, open-source implementation of the AraucanaXAI method and evaluate its behaviour in a number of different settings that are commonly encountered in medical applications of AI. These include potential disagreement between the model prediction and physician's expert opinion and low reliability of the prediction due to data scarcity.

Explaining predictive factors in patient pathways using autoencoders

This paper introduces an end-to-end methodology to predict a pathway-related outcome and identifying predictive factors using autoencoders. A formal description of autoencoders for explainable binary predictions is presented, along with two objective functions that allows for filtering and inverting negative examples during training. A methodology to model and transform complex medical event logs is also proposed, which keeps the pathway information in terms of events and time, as well as the hierarchy information carried in medical codes. A case study is presented, in which the short-term mortality after the implementation of an Implantable Cardioverter-Defibrillator is predicted. Proposed methodologies have been tested and compared to other predictive methods, both explainable and not explainable. Results show the competitiveness of the method in terms of performances, particularly the use of a Variational Auto Encoder with an inverse objective function. Finally, the explainability of the method has been demonstrated, allowing for the identification of interesting predictive factors validated using relative risks.

A manifesto on explainability for artificial intelligence in medicine

The rapid increase of interest in, and use of, artificial intelligence (AI) in computer applications has raised a parallel concern about its ability (or lack thereof) to provide understandable, or explainable, output to users. This concern is especially legitimate in biomedical contexts, where patient safety is of paramount importance. This position paper brings together seven researchers working in the field with different roles and perspectives, to explore in depth the concept of explainable AI, or XAI, offering a functional definition and conceptual framework or model that can be used when considering XAI. This is followed by a series of desiderata for attaining explainability in AI, each of which touches upon a key domain in biomedicine.