Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review

Privacy-preserving federated machine learning on FAIR health data: A real-world application

Objective

This paper introduces a privacy-preserving federated machine learning (ML) architecture built upon Findable, Accessible, Interoperable, and Reusable (FAIR) health data. It aims to devise an architecture for executing classification algorithms in a federated manner, enabling collaborative model-building among health data owners without sharing their datasets.

Materials and methods

Utilizing an agent-based architecture, a privacy-preserving federated ML algorithm was developed to create a global predictive model from various local models. This involved formally defining the algorithm in two steps: data preparation and federated model training on FAIR health data and constructing the architecture with multiple components facilitating algorithm execution. The solution was validated by five healthcare organizations using their specific health datasets.

Results

Five organizations transformed their datasets into Health Level 7 Fast Healthcare Interoperability Resources via a common FAIRification workflow and software set, thereby generating FAIR datasets. Each organization deployed a Federated ML Agent within its secure network, connected to a cloud-based Federated ML Manager. System testing was conducted on a use case aiming to predict 30-day readmission risk for chronic obstructive pulmonary disease patients and the federated model achieved an accuracy rate of 87%.

Discussion

The paper demonstrated a practical application of privacy-preserving federated ML among five distinct healthcare entities, highlighting the value of FAIR health data in machine learning when utilized in a federated manner that ensures privacy protection without sharing data.

Conclusion

This solution effectively leverages FAIR datasets from multiple healthcare organizations for federated ML while safeguarding sensitive health datasets, meeting legislative privacy and security requirements.

Artificial intelligence in colorectal multidisciplinary team meetings. What are the medicolegal implications?

AIM

To give an insight into areas for future development and suggestions in the complexities of incorporation of AI into human colorectal cancer (CRC) care while bringing into focus the importance of clinicians' roles in patient care.

METHODS

Existing literature around AI use in CRC care is reviewed and potential regulatory issues and medicolegal implications around its implementation in CRC multidisciplinary team meetings (MDTs) are identified.

RESULTS

Challenges with patient privacy and confidentiality, patient consent, inequity and bias, patient autonomy, as well as AI system transparency and the liability and accountability issues arising from complications that arise from AI-aided clinical decisions are important focusses associated with the use of AI in CRC MDTs.

CONCLUSION

Consideration of various medicolegal aspects of the use of AI in CRC MDTs is warranted to ensure its safe and smooth incorporation into CRC MDTs. AI function as a clinical decision support system and does not replace professional expertise.

Artificial Intelligence Needs Data: Challenges Accessing Italian Databases to Train AI

Population biobanks are an increasingly important infrastructure to support research and will be a much-needed resource in the delivery of personalised medicine. Artificial intelligence (AI) systems can process and cross-link very large amounts of data quickly and be used not only for improving research power but also for helping with complex diagnosis and prediction of diseases based on health profiles. AI, therefore, potentially has a critical role to play in personalised medicine, and biobanks can provide a lot of the necessary baseline data related to healthy populations that will enable the development of AI tools. To develop these tools, access to personal data, and in particular, sensitive data, is required. Such data could be accessed from biobanks. Biobanks are a valuable resource for research but accessing and using the data contained within such biobanks raise a host of legal, ethical, and social issues (ELSI). This includes the appropriate consent to manage the collection, storage, use, and sharing of samples and data, and appropriate governance models that provide oversight of secondary use of samples and data. Biobanks have developed new consent models and governance tools to enable access that address some of these ELSI-related issues. In this paper, we consider whether such governance frameworks can enable access to biobank data to develop AI. As Italy has one of the most restrictive regulatory frameworks on the use of genetic data in Europe, we examine the regulatory framework in Italy. We also look at the proposed changes under the European Health Data Space (EHDS). We conclude by arguing that currently, regulatory frameworks are misaligned and unless addressed, accessing data within Italian biobanks to train AI will be severely limited.

Precision public health in the era of genomics and big data

Precision public health (PPH) considers the interplay between genetics, lifestyle and the environment to improve disease prevention, diagnosis and treatment on a population level-thereby delivering the right interventions to the right populations at the right time. In this Review, we explore the concept of PPH as the next generation of public health. We discuss the historical context of using individual-level data in public health interventions and examine recent advancements in how data from human and pathogen genomics and social, behavioral and environmental research, as well as artificial intelligence, have transformed public health. Real-world examples of PPH are discussed, emphasizing how these approaches are becoming a mainstay in public health, as well as outstanding challenges in their development, implementation and sustainability. Data sciences, ethical, legal and social implications research, capacity building, equity research and implementation science will have a crucial role in realizing the potential for 'precision' to enhance traditional public health approaches.

Machine Learning for Predicting Biologic Agent Efficacy in Ulcerative Colitis: An Analysis for Generalizability and Combination with Computational Models

Machine learning (ML) has been applied to predict the efficacy of biologic agents in ulcerative colitis (UC). ML can offer precision, personalization, efficiency, and automation. Moreover, it can improve decision support in predicting clinical outcomes. However, it faces challenges related to data quality and quantity, overfitting, generalization, and interpretability. This paper comments on two recent ML models that predict the efficacy of vedolizumab and ustekinumab in UC. Models that consider multiple pathways, multiple ethnicities, and combinations of real-world and clinical trial data are required for optimal shared decision-making and precision medicine. This paper also highlights the potential of combining ML with computational models to enhance clinical outcomes and personalized healthcare. Key Insights: (1) ML offers precision, personalization, efficiency, and decision support for predicting the efficacy of biologic agents in UC. (2) Challenging aspects in ML prediction include data quality, overfitting, and interpretability. (3) Multiple pathways, multiple ethnicities, and combinations of real-world and clinical trial data should be considered in predictive models for optimal decision-making. (4) Combining ML with computational models may improve clinical outcomes and personalized healthcare.

Accelerated Design for Perovskite-Oxide-Based Photocatalysts Using Machine Learning Techniques

The rapid discovery of photocatalysts with desired performance among tens of thousands of potential perovskites represents a significant advancement. To expedite the design of perovskite-oxide-based photocatalysts, we developed a model of ABO3-type perovskites using machine learning methods based on atomic and experimental parameters. This model can be used to predict specific surface area (SSA), a key parameter closely associated with photocatalytic activity. The model construction involved several steps, including data collection, feature selection, model construction, web-service development, virtual screening and mechanism elucidation. Statistical analysis revealed that the support vector regression model achieved a correlation coefficient of 0.9462 for the training set and 0.8786 for the leave-one-out cross-validation. The potential perovskites with higher SSA than the highest SSA observed in the existing dataset were identified using the model and our computation platform. We also developed a webserver of the model, freely accessible to users. The methodologies outlined in this study not only facilitate the discovery of new perovskites but also enable exploration of the correlations between the perovskite properties and the physicochemical features. These findings provide valuable insights for further research and applications of perovskites using machine learning techniques.

Molecular genomic and epigenomic characteristics related to aspirin and clopidogrel resistance

BACKGROUND

Mediators, genomic and epigenomic characteristics involving in metabolism of arachidonic acid by cyclooxygenase (COX) and lipoxygenase (ALOX) and hepatic activation of clopidogrel have been individually suggested as factors associated with resistance against aspirin and clopidogrel. The present multi-center prospective cohort study evaluated whether the mediators, genomic and epigenomic characteristics participating in arachidonic acid metabolism and clopidogrel activation could be factors that improve the prediction of the aspirin and clopidogrel resistance in addition to cardiovascular risks.

METHODS

We enrolled 988 patients with transient ischemic attack and ischemic stroke who were evaluated for a recurrence of ischemic stroke to confirm clinical resistance, and measured aspirin (ARU) and P2Y12 reaction units (PRU) using VerifyNow to assess laboratory resistance 12 weeks after aspirin and clopidogrel administration. We investigated whether mediators, genotypes, and promoter methylation of genes involved in COX and ALOX metabolisms and clopidogrel activation could synergistically improve the prediction of ischemic stroke recurrence and the ARU and PRU levels by integrating to the established cardiovascular risk factors.

RESULTS

The logistic model to predict the recurrence used thromboxane A synthase 1 (TXAS1, rs41708) A/A genotype and ALOX12 promoter methylation as independent variables, and, improved sensitivity of recurrence prediction from 3.4% before to 13.8% after adding the mediators, genomic and epigenomic variables to the cardiovascular risks. The linear model we used to predict the ARU level included leukotriene B4, COX2 (rs20417) C/G and thromboxane A2 receptor (rs1131882) A/A genotypes with the addition of COX1 and ALOX15 promoter methylations as variables. The linear PRU prediction model included G/A and prostaglandin I receptor (rs4987262) G/A genotypes, COX2 and TXAS1 promoter methylation, as well as cytochrome P450 2C19*2 (rs4244285) A/A, G/A, and *3 (rs4986893) A/A genotypes as variables. The linear models for predicting ARU (r = 0.291, R2 = 0.033, p < 0.01) and PRU (r = 0.503, R2 = 0.210, p < 0.001) levels had improved prediction performance after adding the genomic and epigenomic variables to the cardiovascular risks.

CONCLUSIONS

This study demonstrates that different mediators, genomic and epigenomic characteristics of arachidonic acid metabolism and clopidogrel activation synergistically improved the prediction of the aspirin and clopidogrel resistance together with the cardiovascular risk factors.

TRIAL REGISTRATION

URL: https://www.

CLINICALTRIALS

gov ; Unique identifier: NCT03823274.

A Prediction Model for Identifying Seasonal Influenza Vaccination Uptake Among Children in Wuxi, China: Prospective Observational Study

BACKGROUND

Predicting vaccination behaviors accurately could provide insights for health care professionals to develop targeted interventions.

OBJECTIVE

The aim of this study was to develop predictive models for influenza vaccination behavior among children in China.

METHODS

We obtained data from a prospective observational study in Wuxi, eastern China. The predicted outcome was individual-level vaccine uptake and covariates included sociodemographics of the child and parent, parental vaccine hesitancy, perceptions of convenience to the clinic, satisfaction with clinic services, and willingness to vaccinate. Bayesian networks, logistic regression, least absolute shrinkage and selection operator (LASSO) regression, support vector machine (SVM), naive Bayes (NB), random forest (RF), and decision tree classifiers were used to construct prediction models. Various performance metrics, including area under the receiver operating characteristic curve (AUC), were used to evaluate the predictive performance of the different models. Receiver operating characteristic curves and calibration plots were used to assess model performance.

RESULTS

A total of 2383 participants were included in the study; 83.2% of these children (n=1982) were <5 years old and 6.6% (n=158) had previously received an influenza vaccine. More than half (1356/2383, 56.9%) the parents indicated a willingness to vaccinate their child against influenza. Among the 2383 children, 26.3% (n=627) received influenza vaccination during the 2020-2021 season. Within the training set, the RF model showed the best performance across all metrics. In the validation set, the logistic regression model and NB model had the highest AUC values; the SVM model had the highest precision; the NB model had the highest recall; and the logistic regression model had the highest accuracy, F1 score, and Cohen κ value. The LASSO and logistic regression models were well-calibrated.

CONCLUSIONS

The developed prediction model can be used to quantify the uptake of seasonal influenza vaccination for children in China. The stepwise logistic regression model may be better suited for prediction purposes.

The Use of Artificial Intelligence for Skin Disease Diagnosis in Primary Care Settings: A Systematic Review

The prevalence of dermatological conditions in primary care, coupled with challenges such as dermatologist shortages and rising consultation costs, highlights the need for innovative solutions. Artificial intelligence (AI) holds promise for improving the diagnostic analysis of skin lesion images, potentially enhancing patient care in primary settings. This systematic review following PRISMA guidelines examined primary studies (2012-2022) assessing AI algorithms' diagnostic accuracy for skin diseases in primary care. Studies were screened for eligibility based on their availability in the English language and exclusion criteria, with risk of bias evaluated using QUADAS-2. PubMed, Scopus, and Web of Science were searched. Fifteen studies (2019-2022), primarily from Europe and the USA, focusing on diagnostic accuracy were included. Sensitivity ranged from 58% to 96.1%, with accuracies varying from 0.41 to 0.93. AI applications encompassed triage and diagnostic support across diverse skin conditions in primary care settings, involving both patients and primary care professionals. While AI demonstrates potential for enhancing the accuracy of skin disease diagnostics in primary care, further research is imperative to address study heterogeneity and ensure algorithm reliability across diverse populations. Future investigations should prioritise robust dataset development and consider representative patient samples. Overall, AI may improve dermatological diagnosis in primary care, but careful consideration of algorithm limitations and implementation strategies is required.

Prediction models for amputation after diabetic foot: systematic review and critical appraisal

BACKGROUND

Numerous studies have developed or validated prediction models aimed at estimating the likelihood of amputation in diabetic foot (DF) patients. However, the quality and applicability of these models in clinical practice and future research remain uncertain. This study conducts a systematic review and assessment of the risk of bias and applicability of amputation prediction models among individuals with DF.

METHODS

A comprehensive search was conducted across multiple databases, including PubMed, Web of Science, EBSCO CINAHL Plus, Embase, Cochrane Library, China National Knowledge Infrastructure (CNKI), Wanfang, Chinese Biomedical Literature Database (CBM), and Weipu (VIP) from their inception to December 24, 2023. Two investigators independently screened the literature and extracted data using the checklist for critical appraisal and data extraction for systematic reviews of prediction modeling studies. The Prediction Model Risk of Bias Assessment Tool (PROBAST) checklist was employed to evaluate both the risk of bias and applicability.

RESULTS

A total of 20 studies were included in this analysis, comprising 17 development studies and three validation studies, encompassing 20 prediction models and 11 classification systems. The incidence of amputation in patients with DF ranged from 5.9 to 58.5%. Machine learning-based methods were employed in more than half of the studies. The reported area under the curve (AUC) varied from 0.560 to 0.939. Independent predictors consistently identified by multivariate models included age, gender, HbA1c, hemoglobin, white blood cell count, low-density lipoprotein cholesterol, diabetes duration, and Wagner's Classification. All studies were found to exhibit a high risk of bias, primarily attributed to inadequate handling of outcome events and missing data, lack of model performance assessment, and overfitting.

CONCLUSIONS

The assessment using PROBAST revealed a notable risk of bias in the existing prediction models for amputation in patients with DF. It is imperative for future studies to concentrate on enhancing the robustness of current prediction models or constructing new models with stringent methodologies.

Prognostic models for patients suffering a heart failure with a preserved ejection fraction: a systematic review

The purpose of this study was to systematically review the development, performance, and applicability of prognostic models developed for predicting poor events in patients with heart failure with preserved ejection fraction (HFpEF). Databases including Embase, PubMed, Web of Science Core Collection, the Cochrane Library, China National Knowledge Infrastructure, Wan Fang, Wei Pu, and China Biological Medicine were queried from their respective dates of inception to 1 June 2023, to examine multivariate models for prognostic prediction in HFpEF. Both forward and backward citations of all studies were included in our analysis. Two researchers individually used the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) checklist to extract data and assess the quality of the models using the Predictive Mode Bias Risk Assessment Tool (PROBAST). Among the 6897 studies screened, 16 studies derived and/or validated a total of 39 prognostic models. The sample size ranges for model development, internal validation, and external validation are 119 to 5988, 152 to 1000, and 30 to 5957, respectively. The most frequently employed modelling technique was Cox proportional hazards regression. Six studies (37.50%) conducted internal validation of models; bootstrap and k-fold cross-validation were the commonly used methods for internal validation of models. Ten of these models (25.64%) were validated externally, with reported the c-statistic in the external validation set ranging from 0.70 to 0.96, while the remaining models await external validation. The MEDIA echo score and I-PRESERVE-sudden cardiac death prediction mode have been externally validated using multiple cohorts, and the results consistently show good predictive performance. The most frequently used predictors identified among the models were age, n-terminal pro-brain natriuretic peptide, ejection fraction, albumin, and hospital stay in the last 5 months owing to heart failure. All study predictor domains and outcome domains were at low risk of bias, high or unclear risk of bias of all prognostic models due to underreporting in the area of analysis. All studies did not evaluate the clinical utility of the prognostic models. Predictive models for predicting prognostic outcomes in patients with HFpEF showed good discriminatory ability but their utility and generalization remain uncertain due to the risk of bias, differences in predictors between models, and the lack of clinical application studies. Future studies should improve the methodological quality of model development and conduct external validation of models.

SPIN-PM: a consensus framework to evaluate the presence of spin in studies on prediction models

OBJECTIVES

To develop a framework to identify and evaluate spin practices and its facilitators in studies on clinical prediction model regardless of the modeling technique.

STUDY DESIGN AND SETTING

We followed a three-phase consensus process: (1) premeeting literature review to generate items to be included; (2) a series of structured meetings to provide comments discussed and exchanged viewpoints on items to be included with a panel of experienced researchers; and (3) postmeeting review on final list of items and examples to be included. Through this iterative consensus process, a framework was derived after all panel's researchers agreed.

RESULTS

This consensus process involved a panel of eight researchers and resulted in SPIN-Prediction Models which consists of two categories of spin (misleading interpretation and misleading transportability), and within these categories, two forms of spin (spin practices and facilitators of spin). We provide criteria and examples.

CONCLUSION

We proposed this guidance aiming to facilitate not only the accurate reporting but also an accurate interpretation and extrapolation of clinical prediction models which will likely improve the reporting quality of subsequent research, as well as reduce research waste.

Prediction Models for Intravenous Immunoglobulin Non-Responders of Kawasaki Disease Using Machine Learning

BACKGROUND AND OBJECTIVE: Intravenous immunoglobulin (IVIG) is a prominent therapeutic agent for Kawasaki disease (KD) that significantly reduces the incidence of coronary artery anomalies. Various methodologies, including machine learning, have been employed to develop IVIG non-responder prediction models; however, their validation and reproducibility remain unverified. This study aimed to develop a predictive scoring system for identifying IVIG nonresponders and rigorously test the accuracy and reliability of this system. METHODS: The study included an exposure group of 228 IVIG non-responders and a control group of 997 IVIG responders. Subsequently, a predictive machine learning model was constructed. The Shizuoka score, including variables such as the "initial treatment date" (cutoff: < 4 days), sodium level (cutoff: < 133 mEq/L), total bilirubin level (cutoff: ≥ 0.5 mg/dL), and neutrophil-to-lymphocyte ratio (cutoff: ≥ 2.6), was established. Patients meeting two or more of these criteria were grouped as high-risk IVIG non-responders. Using the Shizuoka score to stratify IVIG responders, propensity score matching was used to analyze 85 patients each for IVIG and IVIG-added prednisolone treatment in the high-risk group. In the IVIG plus prednisolone group, the IVIG non-responder count significantly decreased (p < 0.001), with an odds ratio of 0.192 (95% confidence interval 0.078-0.441). CONCLUSIONS: Intravenous immunoglobulin non-responders were predicted using machine learning models and validated using propensity score matching. The initiation of initial IVIG-added prednisolone treatment in the high-risk group identified by the Shizuoka score, crafted using machine learning models, appears useful for predicting IVIG non-responders.

Applications and Performance of Machine Learning Algorithms in Emergency Medical Services: A Scoping Review

OBJECTIVE

The aim of this study was to summarize the literature on the applications of machine learning (ML) and their performance in Emergency Medical Services (EMS).

METHODS

Four relevant electronic databases were searched (from inception through January 2024) for all original studies that employed EMS-guided ML algorithms to enhance the clinical and operational performance of EMS. Two reviewers screened the retrieved studies and extracted relevant data from the included studies. The characteristics of included studies, employed ML algorithms, and their performance were quantitively described across primary domains and subdomains.

RESULTS

This review included a total of 164 studies published from 2005 through 2024. Of those, 125 were clinical domain focused and 39 were operational. The characteristics of ML algorithms such as sample size, number and type of input features, and performance varied between and within domains and subdomains of applications. Clinical applications of ML algorithms involved triage or diagnosis classification (n = 62), treatment prediction (n = 12), or clinical outcome prediction (n = 50), mainly for out-of-hospital cardiac arrest/OHCA (n = 62), cardiovascular diseases/CVDs (n = 19), and trauma (n = 24). The performance of these ML algorithms varied, with a median area under the receiver operating characteristic curve (AUC) of 85.6%, accuracy of 88.1%, sensitivity of 86.05%, and specificity of 86.5%. Within the operational studies, the operational task of most ML algorithms was ambulance allocation (n = 21), followed by ambulance detection (n = 5), ambulance deployment (n = 5), route optimization (n = 5), and quality assurance (n = 3). The performance of all operational ML algorithms varied and had a median AUC of 96.1%, accuracy of 90.0%, sensitivity of 94.4%, and specificity of 87.7%. Generally, neural network and ensemble algorithms, to some degree, out-performed other ML algorithms.

CONCLUSION

Triaging and managing different prehospital medical conditions and augmenting ambulance performance can be improved by ML algorithms. Future reports should focus on a specific clinical condition or operational task to improve the precision of the performance metrics of ML models.

Artificial intelligence in healthcare: why not apply the medico-legal method starting with the Collingridge dilemma?

Technology has greatly influenced and radically changed human life, from communication to creativity and from productivity to entertainment. The authors, starting from considerations concerning the implementation of new technologies with a strong impact on people's everyday lives, take up Collingridge's dilemma and relate it to the application of AI in healthcare. Collingridge's dilemma is an ethical and epistemological problem concerning the relationship between technology and society which involves two approaches. The proactive approach and socio-technological experimentation taken into account in the dilemma are discussed, the former taking health technology assessment (HTA) processes as a reference and the latter the AI studies conducted so far. As a possible prevention of the critical issues raised, the use of the medico-legal method is proposed, which classically lies between the prevention of possible adverse events and the reconstruction of how these occurred.The authors believe that this methodology, adopted as a European guideline in the medico-legal field for the assessment of medical liability, can be adapted to AI applied to the healthcare scenario and used for the assessment of liability issues. The topic deserves further investigation and will certainly be taken into consideration as a possible key to future scenarios.

Predictive value of machine learning for the severity of acute pancreatitis: A systematic review and meta-analysis

Background

Predicting the severity of acute pancreatitis (AP) early poses a challenge in clinical practice. While there are well-established clinical scoring tools, their actual predictive performance remains uncertain. Various studies have explored the application of machine-learning methods for early AP prediction. However, a more comprehensive evidence-based assessment is needed to determine their predictive accuracy. Hence, this systematic review and meta-analysis aimed to evaluate the predictive accuracy of machine learning in assessing the severity of AP.

Methods

PubMed, EMBASE, Cochrane Library, and Web of Science were systematically searched until December 5, 2023. The risk of bias in eligible studies was assessed using the Prediction Model Risk of Bias Assessment Tool (PROBAST). Subgroup analyses, based on different machine learning types, were performed. Additionally, the predictive accuracy of mainstream scoring tools was summarized.

Results

This systematic review ultimately included 33 original studies. The pooled c-index in both the training and validation sets was 0.87 (95 % CI: 0.84-0.89) and 0.88 (95 % CI: 0.86-0.90), respectively. The sensitivity in the training set was 0.81 (95 % CI: 0.77-0.84), and in the validation set, it was 0.79 (95 % CI: 0.71-0.85). The specificity in the training set was 0.84 (95 % CI: 0.78-0.89), and in the validation set, it was 0.90 (95 % CI: 0.86-0.93). The primary model incorporated was logistic regression; however, its predictive accuracy was found to be inferior to that of neural networks, random forests, and xgboost. The pooled c-index of the APACHE II, BISAP, and Ranson were 0.74 (95 % CI: 0.68-0.80), 0.77 (95 % CI: 0.70-0.85), and 0.74 (95 % CI: 0.68-0.79), respectively.

Conclusions

Machine learning demonstrates excellent accuracy in predicting the severity of AP, providing a reference for updating or developing a straightforward clinical prediction tool.

Investigating Rhythmicity in App Usage to Predict Depressive Symptoms: Protocol for Personalized Framework Development and Validation Through a Countrywide Study

BACKGROUND

Understanding a student's depressive symptoms could facilitate significantly more precise diagnosis and treatment. However, few studies have focused on depressive symptom prediction through unobtrusive systems, and these studies are limited by small sample sizes, low performance, and the requirement for higher resources. In addition, research has not explored whether statistically significant rhythms based on different app usage behavioral markers (eg, app usage sessions) exist that could be useful in finding subtle differences to predict with higher accuracy like the models based on rhythms of physiological data.

OBJECTIVE

The main objective of this study is to explore whether there exist statistically significant rhythms in resource-insensitive app usage behavioral markers and predict depressive symptoms through these marker-based rhythmic features. Another objective of this study is to understand whether there is a potential link between rhythmic features and depressive symptoms.

METHODS

Through a countrywide study, we collected 2952 students' raw app usage behavioral data and responses to the 9 depressive symptoms in the 9-item Patient Health Questionnaire (PHQ-9). The behavioral data were retrieved through our developed app, which was previously used in our pilot studies in Bangladesh on different research problems. To explore whether there is a rhythm based on app usage data, we will conduct a zero-amplitude test. In addition, we will develop a cosinor model for each participant to extract rhythmic parameters (eg, acrophase). In addition, to obtain a comprehensive picture of the rhythms, we will explore nonparametric rhythmic features (eg, interdaily stability). Furthermore, we will conduct regression analysis to understand the association of rhythmic features with depressive symptoms. Finally, we will develop a personalized multitask learning (MTL) framework to predict symptoms through rhythmic features.

RESULTS

After applying inclusion criteria (eg, having app usage data of at least 2 days to explore rhythmicity), we kept the data of 2902 (98.31%) students for analysis, with 24.48 million app usage events, and 7 days' app usage of 2849 (98.17%) students. The students are from all 8 divisions of Bangladesh, both public and private universities (19 different universities and 52 different departments). We are analyzing the data and will publish the findings in a peer-reviewed publication.

CONCLUSIONS

Having an in-depth understanding of app usage rhythms and their connection with depressive symptoms through a countrywide study can significantly help health care professionals and researchers better understand depressed students and may create possibilities for using app usage-based rhythms for intervention. In addition, the MTL framework based on app usage rhythmic features may more accurately predict depressive symptoms due to the rhythms' capability to find subtle differences.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/51540.

Use of Artificial Intelligence in Improving Outcomes in Heart Disease: A Scientific Statement From the American Heart Association

A major focus of academia, industry, and global governmental agencies is to develop and apply artificial intelligence and other advanced analytical tools to transform health care delivery. The American Heart Association supports the creation of tools and services that would further the science and practice of precision medicine by enabling more precise approaches to cardiovascular and stroke research, prevention, and care of individuals and populations. Nevertheless, several challenges exist, and few artificial intelligence tools have been shown to improve cardiovascular and stroke care sufficiently to be widely adopted. This scientific statement outlines the current state of the art on the use of artificial intelligence algorithms and data science in the diagnosis, classification, and treatment of cardiovascular disease. It also sets out to advance this mission, focusing on how digital tools and, in particular, artificial intelligence may provide clinical and mechanistic insights, address bias in clinical studies, and facilitate education and implementation science to improve cardiovascular and stroke outcomes. Last, a key objective of this scientific statement is to further the field by identifying best practices, gaps, and challenges for interested stakeholders.

Improving management of febrile neutropenia in oncology patients: the role of artificial intelligence and machine learning

INTRODUCTION

Artificial intelligence (AI) and machine learning (ML) have the potential to revolutionize the management of febrile neutropenia (FN) and drive progress toward personalized medicine.

AREAS COVERED

In this review, we detail how the collection of a large number of high-quality data can be used to conduct precise mathematical studies with ML and AI. We explain the foundations of these techniques, covering the fundamentals of supervised and unsupervised learning, as well as the most important challenges, e.g. data quality, 'black box' model interpretation and overfitting. To conclude, we provide detailed examples of how AI and ML have been used to enhance predictions of chemotherapy-induced FN, detection of bloodstream infections (BSIs) and multidrug-resistant (MDR) bacteria, and anticipation of severe complications and mortality.

EXPERT OPINION

There is promising potential of implementing accurate AI and ML models whilst managing FN. However, their integration as viable clinical tools poses challenges, including technical and implementation barriers. Improving global accessibility, fostering interdisciplinary collaboration, and addressing ethical and security considerations are essential. By overcoming these challenges, we could transform personalized care for patients with FN.

Measurement and Mitigation of Bias in Artificial Intelligence: A Narrative Literature Review for Regulatory Science

Artificial intelligence (AI) is increasingly being used in decision making across various industries, including the public health arena. Bias in any decision-making process can significantly skew outcomes, and AI systems have been shown to exhibit biases at times. The potential for AI systems to perpetuate and even amplify biases is a growing concern. Bias, as used in this paper, refers to the tendency toward a particular characteristic or behavior, and thus, a biased AI system is one that shows biased associations entities. In this literature review, we examine the current state of research on AI bias, including its sources, as well as the methods for measuring, benchmarking, and mitigating it. We also examine the biases and methods of mitigation specifically relevant to the healthcare field and offer a perspective on bias measurement and mitigation in regulatory science decision making.

Clinical performance of automated machine learning: A systematic review

Introduction

Automated machine learning (autoML) removes technical and technological barriers to building artificial intelligence models. We aimed to summarise the clinical applications of autoML, assess the capabilities of utilised platforms, evaluate the quality of the evidence trialling autoML, and gauge the performance of autoML platforms relative to conventionally developed models, as well as each other.

Method

This review adhered to a prospectively registered protocol (PROSPERO identifier CRD42022344427). The Cochrane Library, Embase, MEDLINE and Scopus were searched from inception to 11 July 2022. Two researchers screened abstracts and full texts, extracted data and conducted quality assessment. Disagreement was resolved through discussion and if required, arbitration by a third researcher.

Results

There were 26 distinct autoML platforms featured in 82 studies. Brain and lung disease were the most common fields of study of 22 specialties. AutoML exhibited variable performance: area under the receiver operator characteristic curve (AUCROC) 0.35-1.00, F1-score 0.16-0.99, area under the precision-recall curve (AUPRC) 0.51-1.00. AutoML exhibited the highest AUCROC in 75.6% trials; the highest F1-score in 42.3% trials; and the highest AUPRC in 83.3% trials. In autoML platform comparisons, AutoPrognosis and Amazon Rekognition performed strongest with unstructured and structured data, respectively. Quality of reporting was poor, with a median DECIDE-AI score of 14 of 27.

Conclusion

A myriad of autoML platforms have been applied in a variety of clinical contexts. The performance of autoML compares well to bespoke computational and clinical benchmarks. Further work is required to improve the quality of validation studies. AutoML may facilitate a transition to data-centric development, and integration with large language models may enable AI to build itself to fulfil user-defined goals.

The reporting completeness and transparency of systematic reviews of prognostic prediction models for COVID-19 was poor: a methodological overview of systematic reviews

OBJECTIVES

To conduct a methodological overview of reviews to evaluate the reporting completeness and transparency of systematic reviews (SRs) of prognostic prediction models (PPMs) for COVID-19.

STUDY DESIGN AND SETTING

MEDLINE, Scopus, Cochrane Database of Systematic Reviews, and Epistemonikos (epistemonikos.org) were searched for SRs of PPMs for COVID-19 until December 31, 2022. The risk of bias in systematic reviews tool was used to assess the risk of bias. The protocol for this overview was uploaded in the Open Science Framework (https://osf.io/7y94c).

RESULTS

Ten SRs were retrieved; none of them synthesized the results in a meta-analysis. For most of the studies, there was absence of a predefined protocol and missing information on study selection, data collection process, and reporting of primary studies and models included, while only one SR had its data publicly available. In addition, for the majority of the SRs, the overall risk of bias was judged as being high. The overall corrected covered area was 6.3% showing a small amount of overlapping among the SRs.

CONCLUSION

The reporting completeness and transparency of SRs of PPMs for COVID-19 was poor. Guidance is urgently required, with increased awareness and education of minimum reporting standards and quality criteria. Specific focus is needed in predefined protocol, information on study selection and data collection process, and in the reporting of findings to improve the quality of SRs of PPMs for COVID-19.

Prognostic Models in Nephrology: Where Do We Stand and Where Do We Go from Here? Mapping Out the Evidence in a Scoping Review

Prognostic models can strongly support individualized care provision and well-informed shared decision making. There has been an upsurge of prognostic research in the field of nephrology, but the uptake of prognostic models in clinical practice remains limited. Therefore, we map out the research field of prognostic models for kidney patients and provide directions on how to proceed from here. We performed a scoping review of studies developing, validating, or updating a prognostic model for patients with CKD. We searched all published models in PubMed and Embase and report predicted outcomes, methodological quality, and validation and/or updating efforts. We found 602 studies, of which 30.1% concerned CKD populations, 31.6% dialysis populations, and 38.4% kidney transplantation populations. The most frequently predicted outcomes were mortality ( n =129), kidney disease progression ( n =75), and kidney graft survival ( n =54). Most studies provided discrimination measures (80.4%), but much less showed calibration results (43.4%). Of the 415 development studies, 28.0% did not perform any validation and 57.6% performed only internal validation. Moreover, only 111 models (26.7%) were externally validated either in the development study itself or in an independent external validation study. Finally, in 45.8% of development studies no useable version of the model was reported. To conclude, many prognostic models have been developed for patients with CKD, mainly for outcomes related to kidney disease progression and patient/graft survival. To bridge the gap between prediction research and kidney patient care, patient-reported outcomes, methodological rigor, complete reporting of prognostic models, external validation, updating, and impact assessment urgently need more attention.

Risk prediction models for cardiovascular events in hemodialysis patients: A systematic review

OBJECTIVE

To perform a systematic review of risk prediction models for cardiovascular (CV) events in hemodialysis (HD) patients, and provide a reference for the application and optimization of related prediction models.

METHODS

PubMed, The Cochrane Library, Web of Science, and Embase databases were searched from inception to 1 February 2023. Two authors independently conducted the literature search, selection, and screening. The Prediction model Risk Of Bias Assessment Tool (PROBAST) was applied to evaluate the risk of bias and applicability of the included literature.

RESULTS

A total of nine studies containing 12 models were included, with performance measured by the area under the receiver operating characteristic curve (AUC) lying between 0.70 and 0.88. Age, diabetes mellitus (DM), C-reactive protein (CRP), and albumin (ALB) were the most commonly identified predictors of CV events in HD patients. While the included models demonstrated good applicability, there were still certain risks of bias, primarily related to inadequate handling of missing data and transformation of continuous variables, as well as a lack of model performance validation.

CONCLUSION

The included models showed good overall predictive performance and can assist healthcare professionals in the early identification of high-risk individuals for CV events in HD patients. In the future, the modeling methods should be improved, or the existing models should undergo external validation to provide better guidance for clinical practice.

A trustworthy AI reality-check: the lack of transparency of artificial intelligence products in healthcare

Trustworthy medical AI requires transparency about the development and testing of underlying algorithms to identify biases and communicate potential risks of harm. Abundant guidance exists on how to achieve transparency for medical AI products, but it is unclear whether publicly available information adequately informs about their risks. To assess this, we retrieved public documentation on the 14 available CE-certified AI-based radiology products of the II b risk category in the EU from vendor websites, scientific publications, and the European EUDAMED database. Using a self-designed survey, we reported on their development, validation, ethical considerations, and deployment caveats, according to trustworthy AI guidelines. We scored each question with either 0, 0.5, or 1, to rate if the required information was "unavailable", "partially available," or "fully available." The transparency of each product was calculated relative to all 55 questions. Transparency scores ranged from 6.4% to 60.9%, with a median of 29.1%. Major transparency gaps included missing documentation on training data, ethical considerations, and limitations for deployment. Ethical aspects like consent, safety monitoring, and GDPR-compliance were rarely documented. Furthermore, deployment caveats for different demographics and medical settings were scarce. In conclusion, public documentation of authorized medical AI products in Europe lacks sufficient public transparency to inform about safety and risks. We call on lawmakers and regulators to establish legally mandated requirements for public and substantive transparency to fulfill the promise of trustworthy AI for health.

ADNEX risk prediction model for diagnosis of ovarian cancer: systematic review and meta-analysis of external validation studies

Objectives

To conduct a systematic review of studies externally validating the ADNEX (Assessment of Different Neoplasias in the adnexa) model for diagnosis of ovarian cancer and to present a meta-analysis of its performance.

Design

Systematic review and meta-analysis of external validation studies.

Data sources

Medline, Embase, Web of Science, Scopus, and Europe PMC, from 15 October 2014 to 15 May 2023.

Eligibility criteria for selecting studies

All external validation studies of the performance of ADNEX, with any study design and any study population of patients with an adnexal mass. Two independent reviewers extracted the data. Disagreements were resolved by discussion. Reporting quality of the studies was scored with the TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) reporting guideline, and methodological conduct and risk of bias with PROBAST (Prediction model Risk Of Bias Assessment Tool). Random effects meta-analysis of the area under the receiver operating characteristic curve (AUC), sensitivity and specificity at the 10% risk of malignancy threshold, and net benefit and relative utility at the 10% risk of malignancy threshold were performed.

Results

47 studies (17 007 tumours) were included, with a median study sample size of 261 (range 24-4905). On average, 61% of TRIPOD items were reported. Handling of missing data, justification of sample size, and model calibration were rarely described. 91% of validations were at high risk of bias, mainly because of the unexplained exclusion of incomplete cases, small sample size, or no assessment of calibration. The summary AUC to distinguish benign from malignant tumours in patients who underwent surgery was 0.93 (95% confidence interval 0.92 to 0.94, 95% prediction interval 0.85 to 0.98) for ADNEX with the serum biomarker, cancer antigen 125 (CA125), as a predictor (9202 tumours, 43 centres, 18 countries, and 21 studies) and 0.93 (95% confidence interval 0.91 to 0.94, 95% prediction interval 0.85 to 0.98) for ADNEX without CA125 (6309 tumours, 31 centres, 13 countries, and 12 studies). The estimated probability that the model has use clinically in a new centre was 95% (with CA125) and 91% (without CA125). When restricting analysis to studies with a low risk of bias, summary AUC values were 0.93 (with CA125) and 0.91 (without CA125), and estimated probabilities that the model has use clinically were 89% (with CA125) and 87% (without CA125).

Conclusions

The results of the meta-analysis indicated that ADNEX performed well in distinguishing between benign and malignant tumours in populations from different countries and settings, regardless of whether the serum biomarker, CA125, was used as a predictor. A key limitation was that calibration was rarely assessed.

Systematic review registration

PROSPERO CRD42022373182.

Artificial intelligence in the risk prediction models of cardiovascular disease and development of an independent validation screening tool: a systematic review

BACKGROUND

A comprehensive overview of artificial intelligence (AI) for cardiovascular disease (CVD) prediction and a screening tool of AI models (AI-Ms) for independent external validation are lacking. This systematic review aims to identify, describe, and appraise AI-Ms of CVD prediction in the general and special populations and develop a new independent validation score (IVS) for AI-Ms replicability evaluation.

METHODS

PubMed, Web of Science, Embase, and IEEE library were searched up to July 2021. Data extraction and analysis were performed for the populations, distribution, predictors, algorithms, etc. The risk of bias was evaluated with the prediction risk of bias assessment tool (PROBAST). Subsequently, we designed IVS for model replicability evaluation with five steps in five items, including transparency of algorithms, performance of models, feasibility of reproduction, risk of reproduction, and clinical implication, respectively. The review is registered in PROSPERO (No. CRD42021271789).

RESULTS

In 20,887 screened references, 79 articles (82.5% in 2017-2021) were included, which contained 114 datasets (67 in Europe and North America, but 0 in Africa). We identified 486 AI-Ms, of which the majority were in development (n = 380), but none of them had undergone independent external validation. A total of 66 idiographic algorithms were found; however, 36.4% were used only once and only 39.4% over three times. A large number of different predictors (range 5-52,000, median 21) and large-span sample size (range 80-3,660,000, median 4466) were observed. All models were at high risk of bias according to PROBAST, primarily due to the incorrect use of statistical methods. IVS analysis confirmed only 10 models as "recommended"; however, 281 and 187 were "not recommended" and "warning," respectively.

CONCLUSION

AI has led the digital revolution in the field of CVD prediction, but is still in the early stage of development as the defects of research design, report, and evaluation systems. The IVS we developed may contribute to independent external validation and the development of this field.

Advancing Diagnostics and Patient Care: The Role of Biomarkers in Radiology

The integration of biomarkers into medical practice has revolutionized the field of radiology, allowing for enhanced diagnostic accuracy, personalized treatment strategies, and improved patient care outcomes. This review offers radiologists a comprehensive understanding of the diverse applications of biomarkers in medicine. By elucidating the fundamental concepts, challenges, and recent advancements in biomarker utilization, it will serve as a bridge between the disciplines of radiology and epidemiology. Through an exploration of various biomarker types, such as imaging biomarkers, molecular biomarkers, and genetic markers, I outline their roles in disease detection, prognosis prediction, and therapeutic monitoring. I also discuss the significance of robust study designs, blinding, power and sample size calculations, performance metrics, and statistical methodologies in biomarker research. By fostering collaboration between radiologists, statisticians, and epidemiologists, I hope to accelerate the translation of biomarker discoveries into clinical practice, ultimately leading to improved patient care.

Role of machine learning in the management of epilepsy: a systematic review protocol

INTRODUCTION

Machine learning is a rapidly expanding field and is already incorporated into many aspects of medicine including diagnostics, prognostication and clinical decision-support tools. Epilepsy is a common and disabling neurological disorder, however, management remains challenging in many cases, despite expanding therapeutic options. We present a systematic review protocol to explore the role of machine learning in the management of epilepsy.

METHODS AND ANALYSIS

This protocol has been drafted with reference to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) for Protocols. A literature search will be conducted in databases including MEDLINE, Embase, Scopus and Web of Science. A PRISMA flow chart will be constructed to summarise the study workflow. As the scope of this review is the clinical application of machine learning, the selection of papers will be focused on studies directly related to clinical decision-making in management of epilepsy, specifically the prediction of response to antiseizure medications, development of drug-resistant epilepsy, and epilepsy surgery and neuromodulation outcomes. Data will be extracted following the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies checklist. Prediction model Risk Of Bias ASsessment Tool will be used for the quality assessment of the included studies. Syntheses of quantitative data will be presented in narrative format.

ETHICS AND DISSEMINATION

As this study is a systematic review which does not involve patients or animals, ethics approval is not required. The results of the systematic review will be submitted to peer-review journals for publication and presented in academic conferences.

PROSPERO REGISTRATION NUMBER

CRD42023442156.

Machine Learning Approaches for the Image-Based Identification of Surgical Wound Infections: Scoping Review

BACKGROUND

Surgical site infections (SSIs) occur frequently and impact patients and health care systems. Remote surveillance of surgical wounds is currently limited by the need for manual assessment by clinicians. Machine learning (ML)-based methods have recently been used to address various aspects of the postoperative wound healing process and may be used to improve the scalability and cost-effectiveness of remote surgical wound assessment.

OBJECTIVE

The objective of this review was to provide an overview of the ML methods that have been used to identify surgical wound infections from images.

METHODS

We conducted a scoping review of ML approaches for visual detection of SSIs following the JBI (Joanna Briggs Institute) methodology. Reports of participants in any postoperative context focusing on identification of surgical wound infections were included. Studies that did not address SSI identification, surgical wounds, or did not use image or video data were excluded. We searched MEDLINE, Embase, CINAHL, CENTRAL, Web of Science Core Collection, IEEE Xplore, Compendex, and arXiv for relevant studies in November 2022. The records retrieved were double screened for eligibility. A data extraction tool was used to chart the relevant data, which was described narratively and presented using tables. Employment of TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) guidelines was evaluated and PROBAST (Prediction Model Risk of Bias Assessment Tool) was used to assess risk of bias (RoB).

RESULTS

In total, 10 of the 715 unique records screened met the eligibility criteria. In these studies, the clinical contexts and surgical procedures were diverse. All papers developed diagnostic models, though none performed external validation. Both traditional ML and deep learning methods were used to identify SSIs from mostly color images, and the volume of images used ranged from under 50 to thousands. Further, 10 TRIPOD items were reported in at least 4 studies, though 15 items were reported in fewer than 4 studies. PROBAST assessment led to 9 studies being identified as having an overall high RoB, with 1 study having overall unclear RoB.

CONCLUSIONS

Research on the image-based identification of surgical wound infections using ML remains novel, and there is a need for standardized reporting. Limitations related to variability in image capture, model building, and data sources should be addressed in the future.

A critical moment in machine learning in medicine: on reproducible and interpretable learning

Over the past two decades, advances in computational power and data availability combined with increased accessibility to pre-trained models have led to an exponential rise in machine learning (ML) publications. While ML may have the potential to transform healthcare, this sharp increase in ML research output without focus on methodological rigor and standard reporting guidelines has fueled a reproducibility crisis. In addition, the rapidly growing complexity of these models compromises their interpretability, which currently impedes their successful and widespread clinical adoption. In medicine, where failure of such models may have severe implications for patients' health, the high requirements for accuracy, robustness, and interpretability confront ML researchers with a unique set of challenges. In this review, we discuss the semantics of reproducibility and interpretability, as well as related issues and challenges, and outline possible solutions to counteracting the "black box". To foster reproducibility, standard reporting guidelines need to be further developed and data or code sharing encouraged. Editors and reviewers may equally play a critical role by establishing high methodological standards and thus preventing the dissemination of low-quality ML publications. To foster interpretable learning, the use of simpler models more suitable for medical data can inform the clinician how results are generated based on input data. Model-agnostic explanation tools, sensitivity analysis, and hidden layer representations constitute further promising approaches to increase interpretability. Balancing model performance and interpretability are important to ensure clinical applicability. We have now reached a critical moment for ML in medicine, where addressing these issues and implementing appropriate solutions will be vital for the future evolution of the field.

Artificial intelligence in fracture detection with different image modalities and data types: A systematic review and meta-analysis

Artificial Intelligence (AI), encompassing Machine Learning and Deep Learning, has increasingly been applied to fracture detection using diverse imaging modalities and data types. This systematic review and meta-analysis aimed to assess the efficacy of AI in detecting fractures through various imaging modalities and data types (image, tabular, or both) and to synthesize the existing evidence related to AI-based fracture detection. Peer-reviewed studies developing and validating AI for fracture detection were identified through searches in multiple electronic databases without time limitations. A hierarchical meta-analysis model was used to calculate pooled sensitivity and specificity. A diagnostic accuracy quality assessment was performed to evaluate bias and applicability. Of the 66 eligible studies, 54 identified fractures using imaging-related data, nine using tabular data, and three using both. Vertebral fractures were the most common outcome (n = 20), followed by hip fractures (n = 18). Hip fractures exhibited the highest pooled sensitivity (92%; 95% CI: 87-96, p< 0.01) and specificity (90%; 95% CI: 85-93, p< 0.01). Pooled sensitivity and specificity using image data (92%; 95% CI: 90-94, p< 0.01; and 91%; 95% CI: 88-93, p < 0.01) were higher than those using tabular data (81%; 95% CI: 77-85, p< 0.01; and 83%; 95% CI: 76-88, p < 0.01), respectively. Radiographs demonstrated the highest pooled sensitivity (94%; 95% CI: 90-96, p < 0.01) and specificity (92%; 95% CI: 89-94, p< 0.01). Patient selection and reference standards were major concerns in assessing diagnostic accuracy for bias and applicability. AI displays high diagnostic accuracy for various fracture outcomes, indicating potential utility in healthcare systems for fracture diagnosis. However, enhanced transparency in reporting and adherence to standardized guidelines are necessary to improve the clinical applicability of AI. Review Registration: PROSPERO (CRD42021240359).

BOO-ST and CBCEC: two novel hybrid machine learning methods aim to reduce the mortality of heart failure patients

Heart failure (HF) is a leading cause of mortality worldwide. Machine learning (ML) approaches have shown potential as an early detection tool for improving patient outcomes. Enhancing the effectiveness and clinical applicability of the ML model necessitates training an efficient classifier with a diverse set of high-quality datasets. Hence, we proposed two novel hybrid ML methods ((a) consisting of Boosting, SMOTE, and Tomek links (BOO-ST); (b) combining the best-performing conventional classifier with ensemble classifiers (CBCEC)) to serve as an efficient early warning system for HF mortality. The BOO-ST was introduced to tackle the challenge of class imbalance, while CBCEC was responsible for training the processed and selected features derived from the Feature Importance (FI) and Information Gain (IG) feature selection techniques. We also conducted an explicit and intuitive comprehension to explore the impact of potential characteristics correlating with the fatality cases of HF. The experimental results demonstrated the proposed classifier CBCEC showcases a significant accuracy of 93.67% in terms of providing the early forecasting of HF mortality. Therefore, we can reveal that our proposed aspects (BOO-ST and CBCEC) can be able to play a crucial role in preventing the death rate of HF and reducing stress in the healthcare sector.

Five critical quality criteria for artificial intelligence-based prediction models

To raise the quality of clinical artificial intelligence (AI) prediction modelling studies in the cardiovascular health domain and thereby improve their impact and relevancy, the editors for digital health, innovation, and quality standards of the European Heart Journal propose five minimal quality criteria for AI-based prediction model development and validation studies: complete reporting, carefully defined intended use of the model, rigorous validation, large enough sample size, and openness of code and software.

Stability of clinical prediction models developed using statistical or machine learning methods

Clinical prediction models estimate an individual's risk of a particular health outcome. A developed model is a consequence of the development dataset and model-building strategy, including the sample size, number of predictors, and analysis method (e.g., regression or machine learning). We raise the concern that many models are developed using small datasets that lead to instability in the model and its predictions (estimated risks). We define four levels of model stability in estimated risks moving from the overall mean to the individual level. Through simulation and case studies of statistical and machine learning approaches, we show instability in a model's estimated risks is often considerable, and ultimately manifests itself as miscalibration of predictions in new data. Therefore, we recommend researchers always examine instability at the model development stage and propose instability plots and measures to do so. This entails repeating the model-building steps (those used to develop the original prediction model) in each of multiple (e.g., 1000) bootstrap samples, to produce multiple bootstrap models, and deriving (i) a prediction instability plot of bootstrap model versus original model predictions; (ii) the mean absolute prediction error (mean absolute difference between individuals' original and bootstrap model predictions), and (iii) calibration, classification, and decision curve instability plots of bootstrap models applied in the original sample. A case study illustrates how these instability assessments help reassure (or not) whether model predictions are likely to be reliable (or not), while informing a model's critical appraisal (risk of bias rating), fairness, and further validation requirements.

The Lifelong Impact of Artificial Intelligence and Clinical Prediction Models on Patients With Tetralogy of Fallot

Medical advancements in the diagnosis, surgical techniques, perioperative care, and continued care throughout childhood have transformed the outlook for individuals with tetralogy of Fallot (TOF), improving survival and shifting the perspective towards lifelong care. However, with a growing population of survivors, longstanding challenges have been accentuated, and new challenges have surfaced, necessitating a re-evaluation of TOF care. Availability of prenatal diagnostics, insufficient information from traditional imaging techniques, previously unforeseen medical complications, and debates surrounding optimal timing and indications for reintervention are among the emerging issues. To address these challenges, the integration of artificial intelligence and machine learning holds great promise as they have the potential to revolutionize patient management and positively impact lifelong outcomes for individuals with TOF. Innovative applications of artificial intelligence and machine learning have spanned across multiple domains of TOF care, including screening and diagnosis, automated image processing and interpretation, clinical risk stratification, and planning and performing cardiac interventions. By embracing these advancements and incorporating them into routine clinical practice, personalized medicine could be delivered, leading to the best possible outcomes for patients. In this review, we provide an overview of these evolving applications and emphasize the challenges, limitations, and future potential for integrating them into clinical care.

Machine learning-based prediction models for pressure injury: A systematic review and meta-analysis

Despite the fact that machine learning (ML) algorithms to construct predictive models for pressure injury development are widely reported, the performance of the model remains unknown. The goal of the review was to systematically appraise the performance of ML models in predicting pressure injury. PubMed, Embase, Cochrane Library, Web of Science, CINAHL, Grey literature and other databases were systematically searched. Original journal papers were included which met the inclusion criteria. The methodological quality was assessed independently by two reviewers using the Prediction Model Risk of Bias Assessment Tool (PROBAST). Meta-analysis was performed with Metadisc software, with the area under the receiver operating characteristic curve, sensitivity and specificity as effect measures. Chi-squared and I2 tests were used to assess the heterogeneity. A total of 18 studies were included for the narrative review, and 14 of them were eligible for meta-analysis. The models achieved excellent pooled AUC of 0.94, sensitivity of 0.79 (95% CI [0.78-0.80]) and specificity of 0.87 (95% CI [0.88-0.87]). Meta-regressions did not provide evidence that model performance varied by data or model types. The present findings indicate that ML models show an outstanding performance in predicting pressure injury. However, good-quality studies should be conducted to verify our results and confirm the clinical value of ML in pressure injury development.

Performances of machine learning algorithms in discriminating sacroiliitis features on MRI: a systematic review

OBJECTIVES

Summarise the evidence of the performance of the machine learning algorithm in discriminating sacroiliitis features on MRI and compare it with the accuracy of human physicians.

METHODS

MEDLINE, EMBASE, CIHNAL, Web of Science, IEEE, American College of Rheumatology and European Alliance of Associations for Rheumatology abstract archives were searched for studies published between 2008 and 4 June 2023. Two authors independently screened and extracted the variables, and the results are presented using tables and forest plots.

RESULTS

Ten studies were selected from 2381. Over half of the studies used deep learning models, using Assessment of Spondyloarthritis International Society sacroiliitis criteria as the ground truth, and manually extracted the regions of interest. All studies reported the area under the curve as a performance index, ranging from 0.76 to 0.99. Sensitivity and specificity were the second-most commonly reported indices, with sensitivity ranging from 0.56 to 1.00 and specificity ranging from 0.67 to 1.00; these results are comparable to a radiologist's sensitivity of 0.67-1.00 and specificity of 0.78-1.00 in the same cohort. More than half of the studies showed a high risk of bias in the analysis domain of quality appraisal owing to the small sample size or overfitting issues.

CONCLUSION

The performance of machine learning algorithms in discriminating sacroiliitis features on MRI varied owing to the high heterogeneity between studies and the small sample sizes, overfitting, and under-reporting issues of individual studies. Further well-designed and transparent studies are required.

Prediction models using artificial intelligence and longitudinal data from electronic health records: a systematic methodological review

OBJECTIVE

To describe and appraise the use of artificial intelligence (AI) techniques that can cope with longitudinal data from electronic health records (EHRs) to predict health-related outcomes.

METHODS

This review included studies in any language that: EHR was at least one of the data sources, collected longitudinal data, used an AI technique capable of handling longitudinal data, and predicted any health-related outcomes. We searched MEDLINE, Scopus, Web of Science, and IEEE Xplorer from inception to January 3, 2022. Information on the dataset, prediction task, data preprocessing, feature selection, method, validation, performance, and implementation was extracted and summarized using descriptive statistics. Risk of bias and completeness of reporting were assessed using a short form of PROBAST and TRIPOD, respectively.

RESULTS

Eighty-one studies were included. Follow-up time and number of registers per patient varied greatly, and most predicted disease development or next event based on diagnoses and drug treatments. Architectures generally were based on Recurrent Neural Networks-like layers, though in recent years combining different layers or transformers has become more popular. About half of the included studies performed hyperparameter tuning and used attention mechanisms. Most performed a single train-test partition and could not correctly assess the variability of the model's performance. Reporting quality was poor, and a third of the studies were at high risk of bias.

CONCLUSIONS

AI models are increasingly using longitudinal data. However, the heterogeneity in reporting methodology and results, and the lack of public EHR datasets and code sharing, complicate the possibility of replication.

REGISTRATION

PROSPERO database (CRD42022331388).

Clinical Prediction Models for Hospital-Induced Delirium Using Structured and Unstructured Electronic Health Record Data: Protocol for a Development and Validation Study

BACKGROUND

Hospital-induced delirium is one of the most common and costly iatrogenic conditions, and its incidence is predicted to increase as the population of the United States ages. An academic and clinical interdisciplinary systems approach is needed to reduce the frequency and impact of hospital-induced delirium.

OBJECTIVE

The long-term goal of our research is to enhance the safety of hospitalized older adults by reducing iatrogenic conditions through an effective learning health system. In this study, we will develop models for predicting hospital-induced delirium. In order to accomplish this objective, we will create a computable phenotype for our outcome (hospital-induced delirium), design an expert-based traditional logistic regression model, leverage machine learning techniques to generate a model using structured data, and use machine learning and natural language processing to produce an integrated model with components from both structured data and text data.

METHODS

This study will explore text-based data, such as nursing notes, to improve the predictive capability of prognostic models for hospital-induced delirium. By using supervised and unsupervised text mining in addition to structured data, we will examine multiple types of information in electronic health record data to predict medical-surgical patient risk of developing delirium. Development and validation will be compliant to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement.

RESULTS

Work on this project will take place through March 2024. For this study, we will use data from approximately 332,230 encounters that occurred between January 2012 to May 2021. Findings from this project will be disseminated at scientific conferences and in peer-reviewed journals.

CONCLUSIONS

Success in this study will yield a durable, high-performing research-data infrastructure that will process, extract, and analyze clinical text data in near real time. This model has the potential to be integrated into the electronic health record and provide point-of-care decision support to prevent harm and improve quality of care.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/48521.

Prediction models in prehospital and emergency medicine research: How to derive and internally validate a clinical prediction model

Clinical prediction models are created to help clinicians with medical decision making, aid in risk stratification, and improve diagnosis and/or prognosis. With growing availability of both prehospital and in-hospital observational registries and electronic health records, there is an opportunity to develop, validate, and incorporate prediction models into clinical practice. However, many prediction models have high risk of bias due to poor methodology. Given that there are no methodological standards aimed at developing prediction models specifically in the prehospital setting, the objective of this paper is to describe the appropriate methodology for the derivation and validation of clinical prediction models in this setting. What follows can also be applied to the emergency medicine (EM) setting. There are eight steps that should be followed when developing and internally validating a prediction model: (1) problem definition, (2) coding of predictors, (3) addressing missing data, (4) ensuring adequate sample size, (5) variable selection, (6) evaluating model performance, (7) internal validation, and (8) model presentation. Subsequent steps include external validation, assessment of impact, and cost-effectiveness. By following these steps, researchers can develop a prediction model with the methodological rigor and quality required for prehospital and EM research.

Artificial intelligence-based risk stratification, accurate diagnosis and treatment prediction in gynecologic oncology

As data-driven science, artificial intelligence (AI) has paved a promising path toward an evolving health system teeming with thrilling opportunities for precision oncology. Notwithstanding the tremendous success of oncological AI in such fields as lung carcinoma, breast tumor and brain malignancy, less attention has been devoted to investigating the influence of AI on gynecologic oncology. Hereby, this review sheds light on the ever-increasing contribution of state-of-the-art AI techniques to the refined risk stratification and whole-course management of patients with gynecologic tumors, in particular, cervical, ovarian and endometrial cancer, centering on information and features extracted from clinical data (electronic health records), cancer imaging including radiological imaging, colposcopic images, cytological and histopathological digital images, and molecular profiling (genomics, transcriptomics, metabolomics and so forth). However, there are still noteworthy challenges beyond performance validation. Thus, this work further describes the limitations and challenges faced in the real-word implementation of AI models, as well as potential solutions to address these issues.