Reporting of artificial intelligence prediction models

Do machine learning methods improve prediction of ambient air pollutants with high spatial contrast? A systematic review

BACKGROUND & OBJECTIVE

The use of machine learning for air pollution modelling is rapidly increasing. We conducted a systematic review of studies comparing statistical and machine learning models predicting the spatiotemporal variation of ambient nitrogen dioxide (NO2), ultrafine particles (UFPs) and black carbon (BC) to determine whether and in which scenarios machine learning generates more accurate predictions.

METHODS

Web of Science and Scopus were searched up to June 13, 2024. All records were screened by two independent reviewers. Differences in the coefficient of determination (R2) and Root Mean Square Error (RMSE) between best statistical and machine learning methods were compared across categories of methodological elements.

RESULTS

A total of 38 studies with 46 model comparisons (30 for NO2, 8 for UFPs and 8 for BC) were included. Linear non-regularized methods and Random Forest were most frequently used. Machine learning outperformed statistical models in 34 comparisons. Mean differences (95% confidence intervals) in R2 and RMSE between best machine learning and statistical models were 0.12 (0.08, 0.17) and 20% (11%, 29%) respectively. Tree-based methods performed best in 12 of 17 multi-model comparisons. Nonlinear or regularization regression methods were used in only 12 comparisons and provided similar performance to machine learning methods.

CONCLUSION

This systematic review suggests that machine learning methods, especially tree-based methods, may be superior to linear non-regularized methods for predicting ambient concentrations of NO2, UFPs and BC. Additional comparison studies using nonlinear, regularized and a wider array of machine learning methods are needed to confirm their relative performance. Future air pollution studies would also benefit from more explicit and standardized reporting of methodologies and results.

Class imbalance on medical image classification: towards better evaluation practices for discrimination and calibration performance

PURPOSE

This work aims to assess standard evaluation practices used by the research community for evaluating medical imaging classifiers, with a specific focus on the implications of class imbalance. The analysis is performed on chest X-rays as a case study and encompasses a comprehensive model performance definition, considering both discriminative capabilities and model calibration.

MATERIALS AND METHODS

We conduct a concise literature review to examine prevailing scientific practices used when evaluating X-ray classifiers. Then, we perform a systematic experiment on two major chest X-ray datasets to showcase a didactic example of the behavior of several performance metrics under different class ratios and highlight how widely adopted metrics can conceal performance in the minority class.

RESULTS

Our literature study confirms that: (1) even when dealing with highly imbalanced datasets, the community tends to use metrics that are dominated by the majority class; and (2) it is still uncommon to include calibration studies for chest X-ray classifiers, albeit its importance in the context of healthcare. Moreover, our systematic experiments confirm that current evaluation practices may not reflect model performance in real clinical scenarios and suggest complementary metrics to better reflect the performance of the system in such scenarios.

CONCLUSION

Our analysis underscores the need for enhanced evaluation practices, particularly in the context of class-imbalanced chest X-ray classifiers. We recommend the inclusion of complementary metrics such as the area under the precision-recall curve (AUC-PR), adjusted AUC-PR, and balanced Brier score, to offer a more accurate depiction of system performance in real clinical scenarios, considering metrics that reflect both, discrimination and calibration performance.

CLINICAL RELEVANCE STATEMENT

This study underscores the critical need for refined evaluation metrics in medical imaging classifiers, emphasizing that prevalent metrics may mask poor performance in minority classes, potentially impacting clinical diagnoses and healthcare outcomes.

KEY POINTS

Common scientific practices in papers dealing with X-ray computer-assisted diagnosis (CAD) systems may be misleading. We highlight limitations in reporting of evaluation metrics for X-ray CAD systems in highly imbalanced scenarios. We propose adopting alternative metrics based on experimental evaluation on large-scale datasets.

Potential and challenge in accelerating high-value conversion of CO2 in microbial electrosynthesis system via data-driven approach

Microbial electrosynthesis for CO2 utilization (MESCU) producing valuable chemicals with high energy density has garnered attention due to its long-term stability and high coulombic efficiency. The data-driven approaches offer a promising avenue by leveraging existing data to uncover the underlying patterns. This comprehensive review firstly uncovered the potentials of utilizing data-driven approaches to enhance high-value conversion of CO2 via MESCU. Firstly, critical challenges of MESCU advancing have been identified, including reactor configuration, cathode design, and microbial analysis. Subsequently, the potential of data-driven approaches to tackle the corresponding challenges, encompassing the identification of pivotal parameters governing reactor setup and cathode design, alongside the decipheration of omics data derived from microbial communities, have been discussed. Correspondingly, the future direction of data-driven approaches in assisting the application of MESCU has been addressed. This review offers guidance and theoretical support for future data-driven applications to accelerate MESCU research and potential industrialization.

The emerging paradigm in pediatric rheumatology: harnessing the power of artificial intelligence

Artificial intelligence algorithms, with roots extending into the past but experiencing a resurgence and evolution in recent years due to their superiority over traditional methods and contributions to human capabilities, have begun to make their presence felt in the field of pediatric rheumatology. In the ever-evolving realm of pediatric rheumatology, there have been incremental advancements supported by artificial intelligence in understanding and stratifying diseases, developing biomarkers, refining visual analyses, and facilitating individualized treatment approaches. However, like in many other domains, these strides have yet to gain clinical applicability and validation, and ethical issues remain unresolved. Furthermore, mastering different and novel terminologies appears challenging for clinicians. This review aims to provide a comprehensive overview of the current literature, categorizing algorithms and their applications, thus offering a fresh perspective on the nascent relationship between pediatric rheumatology and artificial intelligence, highlighting both its advancements and constraints.

Machine Learning-Based Predictive Models for Patients with Venous Thromboembolism: A Systematic Review

BACKGROUND

 Venous thromboembolism (VTE) is a chronic disorder with a significant health and economic burden. Several VTE-specific clinical prediction models (CPMs) have been used to assist physicians in decision-making but have several limitations. This systematic review explores if machine learning (ML) can enhance CPMs by analyzing extensive patient data derived from electronic health records. We aimed to explore ML-CPMs' applications in VTE for risk stratification, outcome prediction, diagnosis, and treatment.

METHODS

 Three databases were searched: PubMed, Google Scholar, and IEEE electronic library. Inclusion criteria focused on studies using structured data, excluding non-English publications, studies on non-humans, and certain data types such as natural language processing and image processing. Studies involving pregnant women, cancer patients, and children were also excluded. After excluding irrelevant studies, a total of 77 studies were included.

RESULTS

 Most studies report that ML-CPMs outperformed traditional CPMs in terms of receiver operating area under the curve in the four clinical domains that were explored. However, the majority of the studies were retrospective, monocentric, and lacked detailed model architecture description and external validation, which are essential for quality audit. This review identified research gaps and highlighted challenges related to standardized reporting, reproducibility, and model comparison.

CONCLUSION

 ML-CPMs show promise in improving risk assessment and individualized treatment recommendations in VTE. Apparently, there is an urgent need for standardized reporting and methodology for ML models, external validation, prospective and real-world data studies, as well as interventional studies to evaluate the impact of artificial intelligence in VTE.

Aliado - A design concept of AI for decision support in oncological liver surgery

BACKGROUND

The interest in artificial intelligence (AI) is increasing. Systematic reviews suggest that there are many machine learning algorithms in surgery, however, only a minority of the studies integrate AI applications in clinical workflows. Our objective was to design and evaluate a concept to use different kinds of AI for decision support in oncological liver surgery along the treatment path.

METHODS

In an exploratory co-creation between design experts, surgeons, and data scientists, pain points along the treatment path were identified. Potential designs for AI-assisted solutions were developed and iteratively refined. Finally, an evaluation of the design concept was performed with n = 20 surgeons to get feedback on the different functionalities and evaluate the usability with the System Usability Scale (SUS). Participating surgeons had a mean of 14.0 ± 5.0 years of experience after graduation.

RESULTS

The design concept was named "Aliado". Five different scenarios were identified where AI could support surgeons. Mean score of SUS was 68.2 ( ± 13.6 SD). The highest valued functionalities were "individualized prediction of survival, short-term mortality and morbidity", and "individualized recommendation of surgical strategy".

CONCLUSION

Aliado is a design prototype that shows how AI could be integrated into the clinical workflow. Even without a fleshed out user interface, the SUS already yielded borderline good results. Expert surgeons rated the functionalities favorably, and most of them expressed their willingness to work with a similar application in the future. Thus, Aliado can serve as a surgical vision of how an ideal AI-based assistance could look like.

Machine learning methods in physical therapy: A scoping review of applications in clinical context

BACKGROUND

Machine learning (ML) efficiently processes large datasets, showing promise in enhancing clinical practice within physical therapy.

OBJECTIVE

The aim of this scoping review is to provide an overview of studies using ML approaches in clinical settings of physical therapy.

DATA SOURCES

A scoping review was performed in PubMed, EMBASE, PEDro, Cochrane, Web of Science, and Scopus.

SELECTION CRITERIA

We included studies utilizing ML methods. ML was defined as the utilization of computational systems to encode patterns and relationships, enabling predictions or classifications with minimal human interference.

DATA EXTRACTION AND DATA SYNTHESIS

Data were extracted regarding methods, data types, performance metrics, and model availability.

RESULTS

Forty-two studies were included. The majority were published after 2020 (n = 25). Fourteen studies (33.3%) were in the musculoskeletal physical therapy field, nine (21.4%) in neurological, and eight (19%) in sports physical therapy. We identified 44 different ML models, with random forest being the most used. Three studies reported on model availability. We identified several clinical applications for ML-based tools, including diagnosis (n = 14), prognosis (n = 7), treatment outcomes prediction (n = 7), clinical decision support (n = 5), movement analysis (n = 4), patient monitoring (n = 3), and personalized care plan (n = 2).

LIMITATION

Model performance metrics, costs, model interpretability, and explainability were not reported.

CONCLUSION

This scope review mapped the emerging landscape of machine learning applications in physical therapy. Despite the growing interest, the field still lacks high-quality studies on validation, model availability, and acceptability to advance from research to clinical practice.

Prediction Models for Perioperative Blood Transfusion in Patients Undergoing Gynecologic Surgery: A Systematic Review

This systematic review aimed to evaluate prediction models for perioperative blood transfusion in patients undergoing gynecologic surgery. Given the inherent risks associated with blood transfusion and the critical need for accurate prediction, this study identified and assessed models based on their development, validation, and predictive performance. The review included five studies encompassing various surgical procedures and approaches. Predicting factors commonly used across these models included preoperative hematocrit, race, surgical route, and uterine fibroid characteristics. However, the review highlighted significant variability in the definition of perioperative periods, a lack of standardization in transfusion criteria, and a high risk of bias in most models due to methodological issues, such as a low number of events per variable, inappropriate handling of continuous and categorical predictors, inappropriate handling of missing data, improper methods of predictor selection, inappropriate measurement methods for model performance, and inadequate evaluations of model overfitting and optimism in model performance. Despite some models demonstrating good discrimination and calibration, the overall quality and external validation of these models were limited. Consequently, there is a clear need for more robust and externally validated models to improve clinical decision-making and patient outcomes in gynecologic surgery. Future research should focus on refining these models, incorporating rigorous validation, and adhering to standardized reporting practices.

Linear endoscopic ultrasound: Current uses and future perspectives in mediastinal examination

This editorial elaborates on the current and future applications of linear endoscopic ultrasound (EUS), a substantial diagnostic and therapeutic modality for various anatomical regions. The scope of endosonographic assessment is broad and, among other factors, allows for the evaluation of the mediastinal anatomy and related pathologies, such as mediastinal lymphadenopathy and the staging of central malignant lung lesions. Moreover, EUS assessment has proven more accurate in detecting small lesions missed by standard imaging examinations, such as computed tomography or magnetic resonance imaging. We focus on its current uses in the mediastinum, including lung and esophageal cancer staging, as well as evaluating mediastinal lymphadenopathy and submucosal lesions. The editorial also explores future perspectives of EUS in mediastinal examination, including ultrasound-guided therapies, artificial intelligence integration, advancements in mediastinal modalities, and improved diagnostic approaches for various mediastinal lesions.

A new screening of preterm birth in gestation with short cervix after pessary plus progesterone

Objective

This study aims to create a new screening for preterm birth < 34 weeks after gestation with a cervical length (CL) ≤ 30 mm, based on clinical, demographic, and sonographic characteristics.

Methods

This is a post hoc analysis of a randomized clinical trial (RCT), which included pregnancies, in middle-gestation, screened with transvaginal ultrasound. After observing inclusion criteria, the patient was invited to compare pessary plus progesterone (PP) versus progesterone only (P) (1:1). The objective was to determine which variables were associated with severe preterm birth using logistic regression (LR). The area under the curve (AUC), sensitivity, specificity, and positive predictive value (PPV) and negative predictive value (NPV) were calculated for both groups after applying LR, with a false positive rate (FPR) set at 10%.

Results

The RCT included 936 patients, 475 in PP and 461 in P. The LR selected: ethnics white, absence of previous curettage, previous preterm birth, singleton gestation, precocious identification of short cervix, CL < 14.7 mm, CL in curve > 21.0 mm. The AUC (CI95%), sensitivity, specificity, PPV, and PNV, with 10% of FPR, were respectively 0.978 (0.961-0.995), 83.4%, 98.1%, 83.4% and 98.1% for PP < 34 weeks; and 0.765 (0.665-0.864), 38.7%, 92.1%, 26.1% and 95.4%, for P < 28 weeks.

Conclusion

Logistic regression can be effective to screen preterm birth < 34 weeks in patients in the PP Group and all pregnancies with CL ≤ 30 mm.

Future AI Will Most Likely Predict Antibody-Drug Conjugate Response in Oncology: A Review and Expert Opinion

The medical research field has been tremendously galvanized to improve the prediction of therapy efficacy by the revolution in artificial intelligence (AI). An earnest desire to find better ways to predict the effectiveness of therapy with the use of AI has propelled the evolution of new models in which it can become more applicable in clinical settings such as breast cancer detection. However, in some instances, the U.S. Food and Drug Administration was obliged to back some previously approved inaccurate models for AI-based prognostic models because they eventually produce inaccurate prognoses for specific patients who might be at risk of heart failure. In light of instances in which the medical research community has often evolved some unrealistic expectations regarding the advances in AI and its potential use for medical purposes, implementing standard procedures for AI-based cancer models is critical. Specifically, models would have to meet some general parameters for standardization, transparency of their logistic modules, and avoidance of algorithm biases. In this review, we summarize the current knowledge about AI-based prognostic methods and describe how they may be used in the future for predicting antibody-drug conjugate efficacy in cancer patients. We also summarize the findings of recent late-phase clinical trials using these conjugates for cancer therapy.

Availability of Evidence for Predictive Machine Learning Algorithms in Primary Care: A Systematic Review

Importance

The aging and multimorbid population and health personnel shortages pose a substantial burden on primary health care. While predictive machine learning (ML) algorithms have the potential to address these challenges, concerns include transparency and insufficient reporting of model validation and effectiveness of the implementation in the clinical workflow.

Objectives

To systematically identify predictive ML algorithms implemented in primary care from peer-reviewed literature and US Food and Drug Administration (FDA) and Conformité Européene (CE) registration databases and to ascertain the public availability of evidence, including peer-reviewed literature, gray literature, and technical reports across the artificial intelligence (AI) life cycle.

Evidence Review

PubMed, Embase, Web of Science, Cochrane Library, Emcare, Academic Search Premier, IEEE Xplore, ACM Digital Library, MathSciNet, AAAI.org (Association for the Advancement of Artificial Intelligence), arXiv, Epistemonikos, PsycINFO, and Google Scholar were searched for studies published between January 2000 and July 2023, with search terms that were related to AI, primary care, and implementation. The search extended to CE-marked or FDA-approved predictive ML algorithms obtained from relevant registration databases. Three reviewers gathered subsequent evidence involving strategies such as product searches, exploration of references, manufacturer website visits, and direct inquiries to authors and product owners. The extent to which the evidence for each predictive ML algorithm aligned with the Dutch AI predictive algorithm (AIPA) guideline requirements was assessed per AI life cycle phase, producing evidence availability scores.

Findings

The systematic search identified 43 predictive ML algorithms, of which 25 were commercially available and CE-marked or FDA-approved. The predictive ML algorithms spanned multiple clinical domains, but most (27 [63%]) focused on cardiovascular diseases and diabetes. Most (35 [81%]) were published within the past 5 years. The availability of evidence varied across different phases of the predictive ML algorithm life cycle, with evidence being reported the least for phase 1 (preparation) and phase 5 (impact assessment) (19% and 30%, respectively). Twelve (28%) predictive ML algorithms achieved approximately half of their maximum individual evidence availability score. Overall, predictive ML algorithms from peer-reviewed literature showed higher evidence availability compared with those from FDA-approved or CE-marked databases (45% vs 29%).

Conclusions and Relevance

The findings indicate an urgent need to improve the availability of evidence regarding the predictive ML algorithms' quality criteria. Adopting the Dutch AIPA guideline could facilitate transparent and consistent reporting of the quality criteria that could foster trust among end users and facilitating large-scale implementation.

A Scoping Review of Artificial Intelligence Detection of Voice Pathology: Challenges and Opportunities

OBJECTIVE

Survey the current literature on artificial intelligence (AI) applications for detecting and classifying vocal pathology using voice recordings, and identify challenges and opportunities for advancing the field forward.

DATA SOURCES

PubMed, EMBASE, CINAHL, and Scopus databases.

REVIEW METHODS

A comprehensive literature search was performed following the Preferred Reporting Items for Systematic Reviews and Meta-analyses Extension for Scoping Reviews guidelines. Peer-reviewed journal articles in the English language were included if they used an AI approach to detect or classify pathological voices using voice recordings from patients diagnosed with vocal pathologies.

RESULTS

Eighty-two studies were included in the review between the years 2000 and 2023, with an increase in publication rate from one study per year in 2012 to 10 per year in 2022. Seventy-two studies (88%) were aimed at detecting the presence of voice pathology, 24 (29%) at classifying the type of voice pathology present, and 4 (5%) at assessing pathological voice using the Grade, Roughness, Breathiness, Asthenia, and Strain scale. Thirty-six databases were used to collect and analyze speech samples. Fourteen articles (17%) did not provide information about their AI model validation methodology. Zero studies moved beyond the preclinical and offline AI model development stages. Zero studies specified following a reporting guideline for AI research.

CONCLUSION

There is rising interest in the potential of AI technology to aid the detection and classification of voice pathology. Three challenges-and areas of opportunities-for advancing this research are heterogeneity of databases, lack of clinical validation studies, and inconsistent reporting.

Identification of medication-related fall risk in adults and older adults admitted to hospital: A machine learning approach

The study aimed to develop and validate, through machine learning, a fall risk prediction model related to prescribed medications specific to adults and older adults admitted to hospital. A case-control study was carried out in a tertiary hospital, involving 9,037 adults and older adults admitted to hospital in 2016. The variables were analyzed using the algorithms: logistic regression, naive bayes, random forest and gradient boosting. The best model presented an area under the curve = 0.628 in the older adult subgroup, compared to an area under the curve (AUC) = 0.776 in the adult subgroup. A specific model was developed for this sample. The gradient boosting model presented the best performance in the sample of older adults (AUC = 0.71). Models developed to predict the risk of falls based on medications specifically aimed at older adults presented better performance in relation to models developed in the total population studied.

Machine learning in paediatric haematological malignancies: a systematic review of prognosis, toxicity and treatment response models

BACKGROUND

Machine Learning (ML) has demonstrated potential in enhancing care in adult oncology. However, its application in paediatric haematological malignancies is still emerging, necessitating a comprehensive review of its capabilities and limitations in this area.

METHODS

A literature search was conducted through Ovid. Studies included focused on ML models in paediatric patients with haematological malignancies. Studies were categorised into thematic groups for analysis.

RESULTS

Twenty studies, primarily on leukaemia, were included in this review. Studies were organised into thematic categories such as prognoses, treatment responses and toxicity predictions. Prognostic studies showed AUC scores between 0.685 and 0.929, indicating moderate-high predictive accuracy. Treatment response studies demonstrated AUC scores between 0.840 and 0.875, reflecting moderate accuracy. Toxicity prediction studies reported high accuracy with AUC scores from 0.870 to 0.927. Only five studies (25%) performed external validation. Significant heterogeneity was noted in ML tasks, reporting formats, and effect measures across studies, highlighting a lack of standardised reporting and challenges in data comparability.

CONCLUSION

The clinical applicability of these ML models remains limited by the lack of external validation and methodological heterogeneity. Addressing these challenges through standardised reporting and rigorous external validation is needed to translate ML from a promising research tool into a reliable clinical practice component.

IMPACT

Key message: Machine Learning (ML) significantly enhances predictive models in paediatric haematological cancers, offering new avenues for personalised treatment strategies. Future research should focus on developing ML models that can integrate with real-time clinical workflows. Addition to literature: Provides a comprehensive overview of current ML applications and trends. It identifies limitations to its applicability, including the limited diversity in datasets, which may affect the generalisability of ML models across different populations.

IMPACT

Encourages standardisation and external validation in ML studies, aiming to improve patient outcomes through precision medicine in paediatric haematological oncology.

Evaluating Artificial Intelligence in Clinical Settings-Let Us Not Reinvent the Wheel

Given the requirement to minimize the risks and maximize the benefits of technology applications in health care provision, there is an urgent need to incorporate theory-informed health IT (HIT) evaluation frameworks into existing and emerging guidelines for the evaluation of artificial intelligence (AI). Such frameworks can help developers, implementers, and strategic decision makers to build on experience and the existing empirical evidence base. We provide a pragmatic conceptual overview of selected concrete examples of how existing theory-informed HIT evaluation frameworks may be used to inform the safe development and implementation of AI in health care settings. The list is not exhaustive and is intended to illustrate applications in line with various stakeholder requirements. Existing HIT evaluation frameworks can help to inform AI-based development and implementation by supporting developers and strategic decision makers in considering relevant technology, user, and organizational dimensions. This can facilitate the design of technologies, their implementation in user and organizational settings, and the sustainability and scalability of technologies.

Artificial intelligence in liver cancer - new tools for research and patient management

Liver cancer has high incidence and mortality globally. Artificial intelligence (AI) has advanced rapidly, influencing cancer care. AI systems are already approved for clinical use in some tumour types (for example, colorectal cancer screening). Crucially, research demonstrates that AI can analyse histopathology, radiology and natural language in liver cancer, and can replace manual tasks and access hidden information in routinely available clinical data. However, for liver cancer, few of these applications have translated into large-scale clinical trials or clinically approved products. Here, we advocate for the incorporation of AI in all stages of liver cancer management. We present a taxonomy of AI approaches in liver cancer, highlighting areas with academic and commercial potential, and outline a policy for AI-based liver cancer management, including interdisciplinary training of researchers, clinicians and patients. The potential of AI in liver cancer is immense, but effort is required to ensure that AI can fulfil expectations.

A joint ESTRO and AAPM guideline for development, clinical validation and reporting of artificial intelligence models in radiation therapy

BACKGROUND AND PURPOSE

Artificial Intelligence (AI) models in radiation therapy are being developed with increasing pace. Despite this, the radiation therapy community has not widely adopted these models in clinical practice. A cohesive guideline on how to develop, report and clinically validate AI algorithms might help bridge this gap.

METHODS AND MATERIALS

A Delphi process with all co-authors was followed to determine which topics should be addressed in this comprehensive guideline. Separate sections of the guideline, including Statements, were written by subgroups of the authors and discussed with the whole group at several meetings. Statements were formulated and scored as highly recommended or recommended.

RESULTS

The following topics were found most relevant: Decision making, image analysis, volume segmentation, treatment planning, patient specific quality assurance of treatment delivery, adaptive treatment, outcome prediction, training, validation and testing of AI model parameters, model availability for others to verify, model quality assurance/updates and upgrades, ethics. Key references were given together with an outlook on current hurdles and possibilities to overcome these. 19 Statements were formulated.

CONCLUSION

A cohesive guideline has been written which addresses main topics regarding AI in radiation therapy. It will help to guide development, as well as transparent and consistent reporting and validation of new AI tools and facilitate adoption.

Predict nutrition-related adverse outcomes in head and neck cancer patients undergoing radiotherapy: A systematic review

BACKGROUND

Acute nutrition-related adverse outcomes are common in head and neck cancer patients undergoing radiotherapy. Predictive models can assist in identifying high-risk patients to enable targeted intervention. We aimed to systematically evaluate predictive models for predicting severe acute nutritional symptoms, insufficient intake, tube feeding, sarcopenia, and weight loss.

METHODS

We searched PubMed, Web of Science, EBSCO, Embase, WanFang, CNKI, and SinoMed. We selected studies developing predictive models for the aforementioned outcomes. Data were extracted using a predefined checklist. Risk of bias and applicability assessment were assessed using the Prediction model Risk of Bias Assessment Tool. A narrative synthesis was conducted to summarize the model characteristics, risk of bias, and performance.

RESULTS

A total of 2941 studies were retrieved and 19 were included. Study outcome measure were different symptoms (n = 11), weight loss (n = 5), tube feeding (n = 3), and symptom or tube feeding (n = 1). Predictive factors mainly encompassed sociodemographic data, disease-related data, and treatment-related data. Seventeen studies reported area under the curve or C-index values ranging from 0.610 to 0.96, indicating moderate to good predictive performance. However, candidate predictors were incomplete, outcome measures were diverse, and the risk of bias was high. Most of them used traditional model development methods, and only two used machine learning.

CONCLUSIONS

Most current models showed moderate to good predictive performance. However, predictors are incomplete, outcome are inconsistent, and the risk of bias is high. Clinicians could carefully select the models with better model performance from the available models according to their actual conditions. Future research should include comprehensive and modifiable indicators and prioritize well-designed and reported studies for model development.

Artificial intelligence and endo-histo-omics: new dimensions of precision endoscopy and histology in inflammatory bowel disease

Integrating artificial intelligence into inflammatory bowel disease (IBD) has the potential to revolutionise clinical practice and research. Artificial intelligence harnesses advanced algorithms to deliver accurate assessments of IBD endoscopy and histology, offering precise evaluations of disease activity, standardised scoring, and outcome prediction. Furthermore, artificial intelligence offers the potential for a holistic endo-histo-omics approach by interlacing and harmonising endoscopy, histology, and omics data towards precision medicine. The emerging applications of artificial intelligence could pave the way for personalised medicine in IBD, offering patient stratification for the most beneficial therapy with minimal risk. Although artificial intelligence holds promise, challenges remain, including data quality, standardisation, reproducibility, scarcity of randomised controlled trials, clinical implementation, ethical concerns, legal liability, and regulatory issues. The development of standardised guidelines and interdisciplinary collaboration, including policy makers and regulatory agencies, is crucial for addressing these challenges and advancing artificial intelligence in IBD clinical practice and trials.

Pitfalls in Developing Machine Learning Models for Predicting Cardiovascular Diseases: Challenge and Solutions

In recent years, there has been explosive development in artificial intelligence (AI), which has been widely applied in the health care field. As a typical AI technology, machine learning models have emerged with great potential in predicting cardiovascular diseases by leveraging large amounts of medical data for training and optimization, which are expected to play a crucial role in reducing the incidence and mortality rates of cardiovascular diseases. Although the field has become a research hot spot, there are still many pitfalls that researchers need to pay close attention to. These pitfalls may affect the predictive performance, credibility, reliability, and reproducibility of the studied models, ultimately reducing the value of the research and affecting the prospects for clinical application. Therefore, identifying and avoiding these pitfalls is a crucial task before implementing the research. However, there is currently a lack of a comprehensive summary on this topic. This viewpoint aims to analyze the existing problems in terms of data quality, data set characteristics, model design, and statistical methods, as well as clinical implications, and provide possible solutions to these problems, such as gathering objective data, improving training, repeating measurements, increasing sample size, preventing overfitting using statistical methods, using specific AI algorithms to address targeted issues, standardizing outcomes and evaluation criteria, and enhancing fairness and replicability, with the goal of offering reference and assistance to researchers, algorithm developers, policy makers, and clinical practitioners.

Ensemble learning for fetal ultrasound and maternal-fetal data to predict mode of delivery after labor induction

Providing adequate counseling on mode of delivery after induction of labor (IOL) is of utmost importance. Various AI algorithms have been developed for this purpose, but rely on maternal-fetal data, not including ultrasound (US) imaging. We used retrospectively collected clinical data from 808 subjects submitted to IOL, totaling 2024 US images, to train AI models to predict vaginal delivery (VD) and cesarean section (CS) outcomes after IOL. The best overall model used only clinical data (F1-score: 0.736; positive predictive value (PPV): 0.734). The imaging models employed fetal head, abdomen and femur US images, showing limited discriminative results. The best model used femur images (F1-score: 0.594; PPV: 0.580). Consequently, we constructed ensemble models to test whether US imaging could enhance the clinical data model. The best ensemble model included clinical data and US femur images (F1-score: 0.689; PPV: 0.693), presenting a false positive and false negative interesting trade-off. The model accurately predicted CS on 4 additional cases, despite misclassifying 20 additional VD, resulting in a 6.0% decrease in average accuracy compared to the clinical data model. Hence, integrating US imaging into the latter model can be a new development in assisting mode of delivery counseling.

A machine learning model for early and accurate prediction of overt disseminated intravascular coagulation before its progression to an overt stage

Background

Recent studies suggested an expected survival benefit associated with anticoagulant therapies for sepsis in patients with disseminated intravascular coagulation (DIC). However, anticoagulant therapies for overt DIC are no longer assumed to regulate pathologic progression as overt DIC is a late-phase coagulation disorder. Therefore, methods for early prediction of sepsis-induced DIC before its progression to an overt stage are strongly required.

Objectives

We aimed to develop a prediction model for overt DIC using machine learning.

Methods

This retrospective, observational study included adult septic patients without overt DIC. The objective variable was binary classification of whether patients developed overt DIC based on International Society on Thrombosis and Haemostasis (ISTH) overt DIC criteria. Explanatory variables were the baseline and time series data within 7 days from sepsis diagnosis. Light Gradient Boosted Machine method was used to construct the prediction model. For controls, we assessed sensitivity and specificity of Japanese Association for Acute Medicine DIC criteria and ISTH sepsis-induced coagulopathy criteria for subsequent onset of overt DIC.

Results

Among 912 patients with sepsis, 139 patients developed overt DIC within 7 days from diagnosis of sepsis. Sensitivity, specificity, and area under the receiver operating characteristic curve for predicting onset of overt DIC within 7 days were 84.4%, 87.5%, and 0.867 in the test cohort and 95.0%, 75.9%, and 0.851 in the validation cohort, respectively. Sensitivity and specificity by the diagnostic thresholds were 54.7% and 74.9% for Japanese Association for Acute Medicine DIC criteria and 63.3% and 71.9% for ISTH sepsis-induced coagulopathy criteria, respectively.

Conclusion

Compared with conventional DIC scoring systems, a machine learning model might exhibit higher prediction accuracy.

Diagnostic machine learning applications on clinical populations using functional near infrared spectroscopy: a review

Functional near-infrared spectroscopy (fNIRS) and its interaction with machine learning (ML) is a popular research topic for the diagnostic classification of clinical disorders due to the lack of robust and objective biomarkers. This review provides an overview of research on psychiatric diseases by using fNIRS and ML. Article search was carried out and 45 studies were evaluated by considering their sample sizes, used features, ML methodology, and reported accuracy. To our best knowledge, this is the first review that reports diagnostic ML applications using fNIRS. We found that there has been an increasing trend to perform ML applications on fNIRS-based biomarker research since 2010. The most studied populations are schizophrenia (n = 12), attention deficit and hyperactivity disorder (n = 7), and autism spectrum disorder (n = 6) are the most studied populations. There is a significant negative correlation between sample size (>21) and accuracy values. Support vector machine (SVM) and deep learning (DL) approaches were the most popular classifier approaches (SVM = 20) (DL = 10). Eight of these studies recruited a number of participants more than 100 for classification. Concentration changes in oxy-hemoglobin (ΔHbO) based features were used more than concentration changes in deoxy-hemoglobin (ΔHb) based ones and the most popular ΔHbO-based features were mean ΔHbO (n = 11) and ΔHbO-based functional connections (n = 11). Using ML on fNIRS data might be a promising approach to reveal specific biomarkers for diagnostic classification.

Advancing healthcare: the role and impact of AI and foundation models

BACKGROUND

The integration of artificial intelligence (AI) into the healthcare domain is a monumental shift with profound implications for diagnostics, medical interventions, and the overall structure of healthcare systems.

PURPOSE

This study explores the transformative journey of foundation AI models in healthcare, shedding light on the challenges, ethical considerations, and vast potential they hold for improving patient outcome and system efficiency. Notably, in this investigation we observe a relatively slow adoption of AI within the public sector of healthcare. The evolution of AI in healthcare is un-paralleled, especially its prowess in revolutionizing diagnostic processes.

RESULTS

This research showcases how these foundational models can unravel hidden patterns within complex medical datasets. The impact of AI reverberates through medical interventions, encompassing pathology, imaging, genomics, and personalized healthcare, positioning AI as a cornerstone in the quest for precision medicine. The paper delves into the applications of generative AI models in critical facets of healthcare, including decision support, medical imaging, and the prediction of protein structures. The study meticulously evaluates various AI models, such as transfer learning, RNN, autoencoders, and their roles in the healthcare landscape. A pioneering concept introduced in this exploration is that of General Medical AI (GMAI), advocating for the development of reusable and flexible AI models.

CONCLUSION

The review article discusses how AI can revolutionize healthcare by stressing the significance of transparency, fairness and accountability, in AI applications regarding patient data privacy and biases. By tackling these issues and suggesting a governance structure the article adds to the conversation about AI integration in healthcare environments.

ChatGPT achieves comparable accuracy to specialist physicians in predicting the efficacy of high-flow oxygen therapy

Background

The failure of high-flow nasal cannula (HFNC) oxygen therapy can necessitate endotracheal intubation in patients, making timely prediction of the intubation risk following HFNC therapy crucial for reducing mortality due to delays in intubation.

Objectives

To investigate the accuracy of ChatGPT in predicting the endotracheal intubation risk within 48 h following HFNC therapy and compare it with the predictive accuracy of specialist and non-specialist physicians.

Methods

We conducted a prospective multicenter cohort study based on the data of 71 adult patients who received HFNC therapy. For each patient, their baseline data and physiological parameters after 6-h HFNC therapy were recorded to create a 6-alternative-forced-choice questionnaire that asked participants to predict the 48-h endotracheal intubation risk using scale options ranging from 1 to 6, with higher scores indicating a greater risk. GPT-3.5, GPT-4.0, respiratory and critical care specialist physicians and non-specialist physicians completed the same questionnaires (N = 71) respectively. We then determined the optimal diagnostic cutoff point, using the Youden index, for each predictor and 6-h ROX index, and compared their predictive performance using receiver operating characteristic (ROC) analysis.

Results

The optimal diagnostic cutoff points were determined to be ≥ 4 for both GPT-4.0 and specialist physicians. GPT-4.0 demonstrated a precision of 76.1 %, with a specificity of 78.6 % (95%CI = 52.4-92.4 %) and sensitivity of 75.4 % (95%CI = 62.9-84.8 %). In comparison, the precision of specialist physicians was 80.3 %, with a specificity of 71.4 % (95%CI = 45.4-88.3 %) and sensitivity of 82.5 % (95%CI = 70.6-90.2 %). For GPT-3.5 and non-specialist physicians, the optimal diagnostic cutoff points were ≥5, with precisions of 73.2 % and 64.8 %, respectively. The area under the curve (AUC) in ROC analysis for GPT-4.0 was 0.821 (95%CI = 0.698-0.943), which was the highest among the predictors and significantly higher than that of non-specialist physicians [0.662 (95%CI = 0.518-0.805), P = 0.011].

Conclusion

GPT-4.0 achieves an accuracy level comparable to specialist physicians in predicting the 48-h endotracheal intubation risk following HFNC therapy, based on patient baseline data and physiological parameters after 6-h HFNC therapy.

Time-Dependent Deep Learning Prediction of Multiple Sclerosis Disability

The majority of deep learning models in medical image analysis concentrate on single snapshot timepoint circumstances, such as the identification of current pathology on a given image or volume. This is often in contrast to the diagnostic methodology in radiology where presumed pathologic findings are correlated to prior studies and subsequent changes over time. For multiple sclerosis (MS), the current body of literature describes various forms of lesion segmentation with few studies analyzing disability progression over time. For the purpose of longitudinal time-dependent analysis, we propose a combinatorial analysis of a video vision transformer (ViViT) benchmarked against traditional recurrent neural network of Convolutional Neural Network-Long Short-Term Memory (CNN-LSTM) architectures and a hybrid Vision Transformer-LSTM (ViT-LSTM) to predict long-term disability based upon the Extended Disability Severity Score (EDSS). The patient cohort was procured from a two-site institution with 703 patients' multisequence, contrast-enhanced MRIs of the cervical spine between the years 2002 and 2023. Following a competitive performance analysis, a VGG-16-based CNN-LSTM was compared to ViViT with an ablation analysis to determine time-dependency of the models. The VGG16-LSTM predicted trinary classification of EDSS score in 6 years with 0.74 AUC versus the ViViT with 0.84 AUC (p-value < 0.001 per 5 × 2 cross-validation F-test) on an 80:20 hold-out testing split. However, the VGG16-LSTM outperformed ViViT when patients with only 2 years of MRIs (n = 94) (0.75 AUC versus 0.72 AUC, respectively). Exact EDSS classification was investigated for both models using both classification and regression strategies but showed collectively worse performance. Our experimental results demonstrate the ability of time-dependent deep learning models to predict disability in MS using trinary stratification of disability, mimicking clinical practice. Further work includes external validation and subsequent observational clinical trials.

The use of artificial intelligence to optimize medication alerts generated by clinical decision support systems: a scoping review

OBJECTIVE

Current Clinical Decision Support Systems (CDSSs) generate medication alerts that are of limited clinical value, causing alert fatigue. Artificial Intelligence (AI)-based methods may help in optimizing medication alerts. Therefore, we conducted a scoping review on the current state of the use of AI to optimize medication alerts in a hospital setting. Specifically, we aimed to identify the applied AI methods used together with their performance measures and main outcome measures.

MATERIALS AND METHODS

We searched Medline, Embase, and Cochrane Library database on May 25, 2023 for studies of any quantitative design, in which the use of AI-based methods was investigated to optimize medication alerts generated by CDSSs in a hospital setting. The screening process was supported by ASReview software.

RESULTS

Out of 5625 citations screened for eligibility, 10 studies were included. Three studies (30%) reported on both statistical performance and clinical outcomes. The most often reported performance measure was positive predictive value ranging from 9% to 100%. Regarding main outcome measures, alerts optimized using AI-based methods resulted in a decreased alert burden, increased identification of inappropriate or atypical prescriptions, and enabled prediction of user responses. In only 2 studies the AI-based alerts were implemented in hospital practice, and none of the studies conducted external validation.

DISCUSSION AND CONCLUSION

AI-based methods can be used to optimize medication alerts in a hospital setting. However, reporting on models' development and validation should be improved, and external validation and implementation in hospital practice should be encouraged.

Quantifying the benefit-risk trade-off for individual patients in a clinical trial: principles and antithrombotic case study

BACKGROUND

A treatment's overall favorable benefit-risk profile does not imply that every individual patient will benefit from the treatment.

OBJECTIVES

To describe a statistical methodology for quantifying the benefit-risk trade-off in individual patients.

METHODS

The method requires a large randomized controlled trial containing a primary efficacy outcome and a primary safety outcome, for instance, the Thrombin Receptor Antagonist in Secondary Prevention of Atherothrombotic Ischemic Events-Thrombolysis in Myocardial Infarction 50 placebo-controlled trial of vorapaxar in 17 779 patients following myocardial infarction. Multivariate regression models predict each individual patient's risk of ischemic events (benefit) and major bleeding events (harm) based on their profile. Hence, each patient's predicted benefit from vorapaxar (reduction in ischemic events) and predicted risk (increase in bleeding events) were estimated. The relative importance of ischemic and bleeding events based on links to all-cause mortality was quantified, although the limitations of such weightings are noted.

RESULTS

Overall results demonstrated both clear benefit and harm from vorapaxar. Substantial interindividual variation in both benefit and risk facilitated distinguishing patients with a favorable benefit-risk trade-off from those who did not. Such findings were applied to recommend vorapaxar in as many as 98.3% of patients in which a favorable mortality-weighted benefit-risk trade-off was present, in 77.2% of patients with ischemic benefit 20% greater than bleeding risk, or in as few as 45.5% of patients if an annual decrease in ischemic risk of ≥0.5% was also required.

CONCLUSION

While overall randomized controlled trials of treatment benefit vs risk are valuable, models determining each individual patient's estimated absolute benefit and risk provide more useful insight regarding patient-specific benefit-risk trade-offs to better enable personalized therapeutic decision-making.

PRERISK: A Personalized, Artificial Intelligence-Based and Statistically-Based Stroke Recurrence Predictor for Recurrent Stroke

BACKGROUND

Predicting stroke recurrence for individual patients is difficult, but individualized prediction may improve stroke survivors' engagement in self-care. We developed PRERISK: a statistical and machine learning classifier to predict individual risk of stroke recurrence.

METHODS

We analyzed clinical and socioeconomic data from a prospectively collected public health care-based data set of 41 975 patients admitted with stroke diagnosis in 88 public health centers over 6 years (2014-2020) in Catalonia-Spain. A new stroke diagnosis at least 24 hours after the index event was considered as a recurrent stroke, which was considered as our outcome of interest. We trained several supervised machine learning models to provide individualized risk over time and compared them with a Cox regression model. Models were trained to predict early, late, and long-term recurrence risk, within 90, 91 to 365, and >365 days, respectively. C statistics and area under the receiver operating characteristic curve were used to assess the accuracy of the models.

RESULTS

Overall, 16.21% (5932 of 36 114) of patients had stroke recurrence during a median follow-up of 2.69 years. The most powerful predictors of stroke recurrence were time from previous stroke, Barthel Index, atrial fibrillation, dyslipidemia, age, diabetes, and sex, which were used to create a simplified model with similar performance, together with modifiable vascular risk factors (glycemia, body mass index, high blood pressure, cholesterol, tobacco dependence, and alcohol abuse). The areas under the receiver operating characteristic curve were 0.76 (95% CI, 0.74-0.77), 0.60 (95% CI, 0.58-0.61), and 0.71 (95% CI, 0.69-0.72) for early, late, and long-term recurrence risk, respectively. The areas under the receiver operating characteristic curve of the Cox risk class probability were 0.73 (95% CI, 0.72-0.75), 0.59 (95% CI, 0.57-0.61), and 0.67 (95% CI, 0.66-0.70); machine learning approaches (random forest and AdaBoost) showed statistically significant improvement (P<0.05) over the Cox model for the 3 recurrence time periods. Stroke recurrence curves can be simulated for each patient under different degrees of control of modifiable factors.

CONCLUSIONS

PRERISK is a novel approach that provides a personalized and fairly accurate risk prediction of stroke recurrence over time. The model has the potential to incorporate dynamic control of risk factors.

Artificial intelligence and Machine Learning approaches in sports: Concepts, applications, challenges, and future perspectives

BACKGROUND

The development and application of Artificial Intelligence (AI) and Machine Learning (ML) in healthcare have gained attention as a promising and powerful resource to change the landscape of healthcare. The potential of these technologies for injury prediction, performance analysis, personalized training, and treatment comes with challenges related to the complexity of sports dynamics and the multidimensional aspects of athletic performance.

OBJECTIVES

We aimed to present the current state of AI and ML applications in sports science, specifically in the areas of injury prediction, performance enhancement, and rehabilitation. We also examine the challenges of incorporating AI and ML into sports and suggest directions for future research.

METHOD

We conducted a comprehensive literature review, focusing on publications related to AI and ML applications in sports. This review encompassed studies on injury prediction, performance analysis, and personalized training, emphasizing the AI and ML models applied in sports.

RESULTS

The findings highlight significant advancements in injury prediction accuracy, performance analysis precision, and the customization of training programs through AI and ML. However, future studies need to address challenges such as ethical considerations, data quality, interpretability of ML models, and the integration of complex data.

CONCLUSION

AI and ML may be useful for the prevention, detection, diagnosis, and treatment of health conditions. In this Masterclass paper, we introduce AI and ML concepts, outline recent breakthroughs in AI technologies and their applications, identify the challenges for further progress of AI systems, and discuss ethical issues, clinical and research opportunities, and future perspectives.

Machine learning algorithms to predict outcomes in children and adolescents with COVID-19: A systematic review

BACKGROUND AND OBJECTIVES

We aimed to analyze the study designs, modeling approaches, and performance evaluation metrics in studies using machine learning techniques to develop clinical prediction models for children and adolescents with COVID-19.

METHODS

We searched four databases for articles published between 01/01/2020 and 10/25/2023, describing the development of multivariable prediction models using any machine learning technique for predicting several outcomes in children and adolescents who had COVID-19.

RESULTS

We included ten articles, six (60 % [95 % confidence interval (CI) 0.31 - 0.83]) were predictive diagnostic models and four (40% [95 % CI 0.170.69]) were prognostic models. All models were developed to predict a binary outcome (n= 10/10, 100 % [95 % CI 0.72-1]). The most frequently predicted outcome was disease detection (n=3/10, 30% [95 % CI 0.11-0.60]). The most commonly used machine learning models in the studies were tree-based (n=12/33, 36.3% [95 % CI 0.17-0.47]) and neural networks (n=9/27, 33.2% [95% CI 0.15-0.44]).

CONCLUSION

Our review revealed that attention is required to address problems including small sample sizes, inconsistent reporting practices on data preparation, biases in data sources, lack of reporting metrics such as calibration and discrimination, hyperparameters and other aspects that allow reproducibility by other researchers and might improve the methodology.

Validation of the Emergency Department-Paediatric Early Warning Score (ED-PEWS) for use in low- and middle-income countries: A multicentre observational study

Early recognition of children at risk of serious illness is essential in preventing morbidity and mortality, particularly in low- and middle-income countries (LMICs). This study aimed to validate the Emergency Department-Paediatric Early Warning Score (ED-PEWS) for use in acute care settings in LMICs. This observational study is based on previously collected clinical data from consecutive children attending four diverse settings in LMICs. Inclusion criteria and study periods (2010-2021) varied. We simulated the ED-PEWS, consisting of patient age, consciousness, work of breathing, respiratory rate, oxygen saturation, heart rate, and capillary refill time, based on the first available parameters. Discrimination was assessed by the area under the curve (AUC), sensitivity and specificity (previously defined cut-offs < 6 and ≥ 15). The outcome measure was for each setting a composite marker of high urgency. 41,917 visits from Gambia rural, 501 visits from Gambia urban, 2,608 visits from Suriname, and 1,682 visits from Tanzania were included. The proportion of high urgency was variable (range 4.6% to 24.9%). Performance ranged from AUC 0.80 (95%CI 0.70-0.89) in Gambia urban to 0.62 (95%CI 0.55-0.67) in Tanzania. The low-urgency cut-off showed a high sensitivity in all settings ranging from 0.83 (95%CI 0.81-0.84) to 1.00 (95%CI 0.97-1.00). The high-urgency cut-off showed a specificity ranging from 0.71 (95%CI 0.66-0.75) to 0.97 (95%CI 0.97-0.97). The ED-PEWS has a moderate to good performance for the recognition of high urgency children in these LMIC settings. The performance appears to have potential in improving the identification of high urgency children in LMICs.

External validation of a deep learning model for predicting bone mineral density on chest radiographs

We developed a new model for predicting bone mineral density on chest radiographs and externally validated it using images captured at facilities other than the development environment. The model performed well and showed potential for clinical use.

PURPOSE

In this study, we performed external validation (EV) of a developed deep learning model for predicting bone mineral density (BMD) of femoral neck on chest radiographs to verify the usefulness of this model in clinical practice.

METHODS

This study included patients who visited any of the collaborating facilities from 2010 to 2020 and underwent chest radiography and dual-energy X-ray absorptiometry (DXA) at the femoral neck in the year before and after their visit. A total of 50,114 chest radiographs were obtained, and BMD was measured using DXA. We developed the model with 47,150 images from 17 facilities and performed EV with 2914 images from three other facilities (EV dataset). We trained the deep learning model via ensemble learning based on chest radiographs, age, and sex to predict BMD using regression. The outcomes were the correlation of the predicted BMD and measured BMD with diagnoses of osteoporosis and osteopenia using the T-score estimated from the predicted BMD.

RESULTS

The mean BMD was 0.64±0.14 g/cm2 in the EV dataset. The BMD predicted by the model averaged 0.61±0.08 g/cm2, with a correlation coefficient of 0.68 (p<0.01) when compared with the BMD measured using DXA. The accuracy, sensitivity, and specificity of the model were 79.0%, 96.6%, and 34.1% for T-score < -1 and 79.7%, 77.1%, and 80.4% for T-score ≤ -2.5, respectively.

CONCLUSION

Our model, which was externally validated using data obtained at facilities other than the development environment, predicted BMD of femoral neck on chest radiographs. The model performed well and showed potential for clinical use.

Estimation of aortic stiffness by finger photoplethysmography using enhanced pulse wave analysis and machine learning

Introduction

Aortic stiffness plays a critical role in the evolution of cardiovascular diseases, but the assessment requires specialized equipment. Photoplethysmography (PPG) and single-lead electrocardiogram (ECG) are readily available in healthcare and wearable devices. We studied whether a brief PPG registration, alone or in combination with single-lead ECG, could be used to reliably estimate aortic stiffness.

Methods

A proof-of-concept study with simultaneous high-resolution index finger recordings of infrared PPG, single-lead ECG, and finger blood pressure (Finapres) was performed in 33 participants [median age 44 (range 21-66) years, 19 men] and repeated within 2 weeks. Carotid-femoral pulse wave velocity (cfPWV; two-site tonometry with SphygmoCor) was used as a reference. A brachial single-cuff oscillometric device assessed aortic pulse wave velocity (aoPWV; Arteriograph) for further comparisons. We extracted 136 established PPG waveform features and engineered 13 new with improved coupling to the finger blood pressure curve. Height-normalized pulse arrival time (NPAT) was derived using ECG. Machine learning methods were used to develop prediction models.

Results

The best PPG-based models predicted cfPWV and aoPWV well (root-mean-square errors of 0.70 and 0.52 m/s, respectively), with minor improvements by adding NPAT. Repeatability and agreement were on par with the reference equipment. A new PPG feature, an amplitude ratio from the early phase of the waveform, was most important in modelling, showing strong correlations with cfPWV and aoPWV (r = -0.81 and -0.75, respectively, both P < 0.001).

Conclusion

Using new features and machine learning methods, a brief finger PPG registration can estimate aortic stiffness without requiring additional information on age, anthropometry, or blood pressure. Repeatability and agreement were comparable to those obtained using non-invasive reference equipment. Provided further validation, this readily available simple method could improve cardiovascular risk evaluation, treatment, and prognosis.

Artificial intelligence image-based prediction models in IBD exhibit high risk of bias: A systematic review

BACKGROUND

There has been an increase in the development of both machine learning (ML) and deep learning (DL) prediction models in Inflammatory Bowel Disease. We aim in this systematic review to assess the methodological quality and risk of bias of ML and DL IBD image-based prediction studies.

METHODS

We searched three databases, PubMed, Scopus and Embase, to identify ML and DL diagnostic or prognostic predictive models using imaging data in IBD, to Dec 31, 2022. We restricted our search to include studies that primarily used conventional imaging data, were undertaken in human participants, and published in English. Two reviewers independently reviewed the abstracts. The methodological quality of the studies was determined, and risk of bias evaluated using the prediction risk of bias assessment tool (PROBAST).

RESULTS

Forty studies were included, thirty-nine developed diagnostic models. Seven studies utilized ML approaches, six were retrospective and none used multicenter data for model development. Thirty-three studies utilized DL approaches, ten were prospective, and twelve multicenter studies. Overall, all studies demonstrated high risk of bias. ML studies were evaluated in 4 domains all rated as high risk of bias: participants (6/7), predictors (1/7), outcome (3/7), and analysis (7/7), and DL studies evaluated in 3 domains: participants (24/33), outcome (10/33), and analysis (18/33). The majority of image-based studies used colonoscopy images.

CONCLUSION

The risk of bias was high in AI IBD image-based prediction models, owing to insufficient sample size, unreported missingness and lack of an external validation cohort. Models with a high risk of bias are unlikely to be generalizable and suitable for clinical implementation.

A literature review of quality assessment and applicability to HTA of risk prediction models of coronary heart disease in patients with diabetes

This literature review had two objectives: to identify models for predicting the risk of coronary heart diseases in patients with diabetes (DM); and to assess model quality in terms of risk of bias (RoB) and applicability for the purpose of health technology assessment (HTA). We undertook a targeted review of journal articles published in English, Dutch, Chinese, or Spanish in 5 databases from 1st January 2016 to 18th December 2022, and searched three systematic reviews for the models published after 2012. We used PROBAST (Prediction model Risk Of Bias Assessment Tool) to assess RoB, and used findings from Betts et al. 2019, which summarized recommendations and criticisms of HTA agencies on cardiovascular risk prediction models, to assess model applicability for the purpose of HTA. As a result, 71 % and 67 % models reporting C-index showed good discrimination abilities (C-index >= 0.7). Of the 26 model studies and 30 models identified, only one model study showed low RoB in all domains, and no model was fully applicable for HTA. Since the major cause of high RoB is inappropriate use of analysis method, we advise clinicians to carefully examine the model performance declared by model developers, and to trust a model if all PROBAST domains except analysis show low RoB and at least one validation study conducted in the same setting (e.g. country) is available. Moreover, since general model applicability is not informative for HTA, novel adapted tools may need to be developed.

Clinical data mining: challenges, opportunities, and recommendations for translational applications

Clinical data mining of predictive models offers significant advantages for re-evaluating and leveraging large amounts of complex clinical real-world data and experimental comparison data for tasks such as risk stratification, diagnosis, classification, and survival prediction. However, its translational application is still limited. One challenge is that the proposed clinical requirements and data mining are not synchronized. Additionally, the exotic predictions of data mining are difficult to apply directly in local medical institutions. Hence, it is necessary to incisively review the translational application of clinical data mining, providing an analytical workflow for developing and validating prediction models to ensure the scientific validity of analytic workflows in response to clinical questions. This review systematically revisits the purpose, process, and principles of clinical data mining and discusses the key causes contributing to the detachment from practice and the misuse of model verification in developing predictive models for research. Based on this, we propose a niche-targeting framework of four principles: Clinical Contextual, Subgroup-Oriented, Confounder- and False Positive-Controlled (CSCF), to provide guidance for clinical data mining prior to the model's development in clinical settings. Eventually, it is hoped that this review can help guide future research and develop personalized predictive models to achieve the goal of discovering subgroups with varied remedial benefits or risks and ensuring that precision medicine can deliver its full potential.

A trustworthy AI reality-check: the lack of transparency of artificial intelligence products in healthcare

Trustworthy medical AI requires transparency about the development and testing of underlying algorithms to identify biases and communicate potential risks of harm. Abundant guidance exists on how to achieve transparency for medical AI products, but it is unclear whether publicly available information adequately informs about their risks. To assess this, we retrieved public documentation on the 14 available CE-certified AI-based radiology products of the II b risk category in the EU from vendor websites, scientific publications, and the European EUDAMED database. Using a self-designed survey, we reported on their development, validation, ethical considerations, and deployment caveats, according to trustworthy AI guidelines. We scored each question with either 0, 0.5, or 1, to rate if the required information was "unavailable", "partially available," or "fully available." The transparency of each product was calculated relative to all 55 questions. Transparency scores ranged from 6.4% to 60.9%, with a median of 29.1%. Major transparency gaps included missing documentation on training data, ethical considerations, and limitations for deployment. Ethical aspects like consent, safety monitoring, and GDPR-compliance were rarely documented. Furthermore, deployment caveats for different demographics and medical settings were scarce. In conclusion, public documentation of authorized medical AI products in Europe lacks sufficient public transparency to inform about safety and risks. We call on lawmakers and regulators to establish legally mandated requirements for public and substantive transparency to fulfill the promise of trustworthy AI for health.

Generative artificial intelligence in drug discovery: basic framework, recent advances, challenges, and opportunities

There are two main ways to discover or design small drug molecules. The first involves fine-tuning existing molecules or commercially successful drugs through quantitative structure-activity relationships and virtual screening. The second approach involves generating new molecules through de novo drug design or inverse quantitative structure-activity relationship. Both methods aim to get a drug molecule with the best pharmacokinetic and pharmacodynamic profiles. However, bringing a new drug to market is an expensive and time-consuming endeavor, with the average cost being estimated at around $2.5 billion. One of the biggest challenges is screening the vast number of potential drug candidates to find one that is both safe and effective. The development of artificial intelligence in recent years has been phenomenal, ushering in a revolution in many fields. The field of pharmaceutical sciences has also significantly benefited from multiple applications of artificial intelligence, especially drug discovery projects. Artificial intelligence models are finding use in molecular property prediction, molecule generation, virtual screening, synthesis planning, repurposing, among others. Lately, generative artificial intelligence has gained popularity across domains for its ability to generate entirely new data, such as images, sentences, audios, videos, novel chemical molecules, etc. Generative artificial intelligence has also delivered promising results in drug discovery and development. This review article delves into the fundamentals and framework of various generative artificial intelligence models in the context of drug discovery via de novo drug design approach. Various basic and advanced models have been discussed, along with their recent applications. The review also explores recent examples and advances in the generative artificial intelligence approach, as well as the challenges and ongoing efforts to fully harness the potential of generative artificial intelligence in generating novel drug molecules in a faster and more affordable manner. Some clinical-level assets generated form generative artificial intelligence have also been discussed in this review to show the ever-increasing application of artificial intelligence in drug discovery through commercial partnerships.

Artificial intelligence in the risk prediction models of cardiovascular disease and development of an independent validation screening tool: a systematic review

BACKGROUND

A comprehensive overview of artificial intelligence (AI) for cardiovascular disease (CVD) prediction and a screening tool of AI models (AI-Ms) for independent external validation are lacking. This systematic review aims to identify, describe, and appraise AI-Ms of CVD prediction in the general and special populations and develop a new independent validation score (IVS) for AI-Ms replicability evaluation.

METHODS

PubMed, Web of Science, Embase, and IEEE library were searched up to July 2021. Data extraction and analysis were performed for the populations, distribution, predictors, algorithms, etc. The risk of bias was evaluated with the prediction risk of bias assessment tool (PROBAST). Subsequently, we designed IVS for model replicability evaluation with five steps in five items, including transparency of algorithms, performance of models, feasibility of reproduction, risk of reproduction, and clinical implication, respectively. The review is registered in PROSPERO (No. CRD42021271789).

RESULTS

In 20,887 screened references, 79 articles (82.5% in 2017-2021) were included, which contained 114 datasets (67 in Europe and North America, but 0 in Africa). We identified 486 AI-Ms, of which the majority were in development (n = 380), but none of them had undergone independent external validation. A total of 66 idiographic algorithms were found; however, 36.4% were used only once and only 39.4% over three times. A large number of different predictors (range 5-52,000, median 21) and large-span sample size (range 80-3,660,000, median 4466) were observed. All models were at high risk of bias according to PROBAST, primarily due to the incorrect use of statistical methods. IVS analysis confirmed only 10 models as "recommended"; however, 281 and 187 were "not recommended" and "warning," respectively.

CONCLUSION

AI has led the digital revolution in the field of CVD prediction, but is still in the early stage of development as the defects of research design, report, and evaluation systems. The IVS we developed may contribute to independent external validation and the development of this field.

Role of machine learning in the management of epilepsy: a systematic review protocol

INTRODUCTION

Machine learning is a rapidly expanding field and is already incorporated into many aspects of medicine including diagnostics, prognostication and clinical decision-support tools. Epilepsy is a common and disabling neurological disorder, however, management remains challenging in many cases, despite expanding therapeutic options. We present a systematic review protocol to explore the role of machine learning in the management of epilepsy.

METHODS AND ANALYSIS

This protocol has been drafted with reference to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) for Protocols. A literature search will be conducted in databases including MEDLINE, Embase, Scopus and Web of Science. A PRISMA flow chart will be constructed to summarise the study workflow. As the scope of this review is the clinical application of machine learning, the selection of papers will be focused on studies directly related to clinical decision-making in management of epilepsy, specifically the prediction of response to antiseizure medications, development of drug-resistant epilepsy, and epilepsy surgery and neuromodulation outcomes. Data will be extracted following the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies checklist. Prediction model Risk Of Bias ASsessment Tool will be used for the quality assessment of the included studies. Syntheses of quantitative data will be presented in narrative format.

ETHICS AND DISSEMINATION

As this study is a systematic review which does not involve patients or animals, ethics approval is not required. The results of the systematic review will be submitted to peer-review journals for publication and presented in academic conferences.

PROSPERO REGISTRATION NUMBER

CRD42023442156.

Deep learning and machine learning predictive models for neurological function after interventional embolization of intracranial aneurysms

Objective

The objective of this study is to develop a model to predicts the postoperative Hunt-Hess grade in patients with intracranial aneurysms by integrating radiomics and deep learning technologies, using preoperative CTA imaging data. Thereby assisting clinical decision-making and improving the assessment and prognosis of postoperative neurological function.

Methods

This retrospective study encompassed 101 patients who underwent aneurysm embolization surgery. 851 radiomic features were extracted from CTA images. 512 deep learning features are extracted from last layer of ResNet50 deep convolutional neural network model. The feature screening process pipeline encompassed intraclass correlation coefficient analysis, principal component analysis, U test, spearman correlation analysis, minimum redundancy maximum relevance algorithm and Lasso regression, to identify features most correlated with postoperative Hunt-Hess grading. In the model construction phase, three distinct models were constructed: radiomics feature-based model (RSM), deep learning feature-based model (DLM), and deep learning-radiomics feature fusion model (DLRSCM). The study also calculated the radiomics score and combined it with clinical data to construct a Nomogram for predictive modeling. DLM, RSM and DLRSCM model was constructed by 9 base algorithms and 1 ensemble learning algorithm - Stacking ensemble model. Model performance was evaluated based on the area under the Receiver Operating Characteristic (ROC) curve (AUC), Matthews Correlation Coefficient (MCC), calibration curves, and decision curves analysis.

Results

5 significant radiomic feature and 4 significant deep learning features were obtained through the feature selection process. These features were utilized for model construction. Bootstrap resampling method was used for internal validation of the models. In terms of model evaluation, the DLM model, the stacking ensemble algorithm results achieved an AUC of 0.959 and MCC of 0.815. In the RSM model, the stacking ensemble model AUC was 0.935 and MCC was 0.793. The stacking ensemble model in DLRSCM outperformed others, with an AUC of 0.968 and MCC of 0.820. Results indicated that the ANN performed optimally among all base models, while the stacked ensemble learning model exhibited the highest predictive performance.

Conclusion

This study demonstrates that the combination of radiomics and deep learning is an effective approach to predict the postoperative Hunt-Hess grade in patients with intracranial aneurysms. This holds significant value in the early identification of postoperative neurological complications and in enhancing clinical decision-making.

A critical moment in machine learning in medicine: on reproducible and interpretable learning

Over the past two decades, advances in computational power and data availability combined with increased accessibility to pre-trained models have led to an exponential rise in machine learning (ML) publications. While ML may have the potential to transform healthcare, this sharp increase in ML research output without focus on methodological rigor and standard reporting guidelines has fueled a reproducibility crisis. In addition, the rapidly growing complexity of these models compromises their interpretability, which currently impedes their successful and widespread clinical adoption. In medicine, where failure of such models may have severe implications for patients' health, the high requirements for accuracy, robustness, and interpretability confront ML researchers with a unique set of challenges. In this review, we discuss the semantics of reproducibility and interpretability, as well as related issues and challenges, and outline possible solutions to counteracting the "black box". To foster reproducibility, standard reporting guidelines need to be further developed and data or code sharing encouraged. Editors and reviewers may equally play a critical role by establishing high methodological standards and thus preventing the dissemination of low-quality ML publications. To foster interpretable learning, the use of simpler models more suitable for medical data can inform the clinician how results are generated based on input data. Model-agnostic explanation tools, sensitivity analysis, and hidden layer representations constitute further promising approaches to increase interpretability. Balancing model performance and interpretability are important to ensure clinical applicability. We have now reached a critical moment for ML in medicine, where addressing these issues and implementing appropriate solutions will be vital for the future evolution of the field.

Systematic review of machine-learning models in orthopaedic trauma

Aims

Machine-learning (ML) prediction models in orthopaedic trauma hold great promise in assisting clinicians in various tasks, such as personalized risk stratification. However, an overview of current applications and critical appraisal to peer-reviewed guidelines is lacking. The objectives of this study are to 1) provide an overview of current ML prediction models in orthopaedic trauma; 2) evaluate the completeness of reporting following the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement; and 3) assess the risk of bias following the Prediction model Risk Of Bias Assessment Tool (PROBAST) tool.

Methods

A systematic search screening 3,252 studies identified 45 ML-based prediction models in orthopaedic trauma up to January 2023. The TRIPOD statement assessed transparent reporting and the PROBAST tool the risk of bias.

Results

A total of 40 studies reported on training and internal validation; four studies performed both development and external validation, and one study performed only external validation. The most commonly reported outcomes were mortality (33%, 15/45) and length of hospital stay (9%, 4/45), and the majority of prediction models were developed in the hip fracture population (60%, 27/45). The overall median completeness for the TRIPOD statement was 62% (interquartile range 30 to 81%). The overall risk of bias in the PROBAST tool was low in 24% (11/45), high in 69% (31/45), and unclear in 7% (3/45) of the studies. High risk of bias was mainly due to analysis domain concerns including small datasets with low number of outcomes, complete-case analysis in case of missing data, and no reporting of performance measures.

Conclusion

The results of this study showed that despite a myriad of potential clinically useful applications, a substantial part of ML studies in orthopaedic trauma lack transparent reporting, and are at high risk of bias. These problems must be resolved by following established guidelines to instil confidence in ML models among patients and clinicians. Otherwise, there will remain a sizeable gap between the development of ML prediction models and their clinical application in our day-to-day orthopaedic trauma practice.

Use of artificial intelligence in the management of T1 colorectal cancer: a new tool in the arsenal or is deep learning out of its depth?

The field of artificial intelligence is rapidly evolving, and there has been an interest in its use to predict the risk of lymph node metastasis in T1 colorectal cancer. Accurately predicting lymph node invasion may result in fewer patients undergoing unnecessary surgeries; conversely, inadequate assessments will result in suboptimal oncological outcomes. This narrative review aims to summarize the current literature on deep learning for predicting the probability of lymph node metastasis in T1 colorectal cancer, highlighting areas of potential application and barriers that may limit its generalizability and clinical utility.

Open science practices need substantial improvement in prognostic model studies in oncology using machine learning

OBJECTIVE

To describe the frequency of open science practices in a contemporary sample of studies developing prognostic models using machine learning methods in the field of oncology.

STUDY DESIGN AND SETTING

We conducted a systematic review, searching the MEDLINE database between December 1, 2022, and December 31, 2022, for studies developing a multivariable prognostic model using machine learning methods (as defined by the authors) in oncology. Two authors independently screened records and extracted open science practices.

RESULTS

We identified 46 publications describing the development of a multivariable prognostic model. The adoption of open science principles was poor. Only one study reported availability of a study protocol, and only one study was registered. Funding statements and conflicts of interest statements were common. Thirty-five studies (76%) provided data sharing statements, with 21 (46%) indicating data were available on request to the authors and seven declaring data sharing was not applicable. Two studies (4%) shared data. Only 12 studies (26%) provided code sharing statements, including 2 (4%) that indicated the code was available on request to the authors. Only 11 studies (24%) provided sufficient information to allow their model to be used in practice. The use of reporting guidelines was rare: eight studies (18%) mentioning using a reporting guideline, with 4 (10%) using the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis Or Diagnosis statement, 1 (2%) using Minimum Information About Clinical Artificial Intelligence Modeling and Consolidated Standards Of Reporting Trials-Artificial Intelligence, 1 (2%) using Strengthening The Reporting Of Observational Studies In Epidemiology, 1 (2%) using Standards for Reporting Diagnostic Accuracy Studies, and 1 (2%) using Transparent Reporting of Evaluations with Nonrandomized Designs.

CONCLUSION

The adoption of open science principles in oncology studies developing prognostic models using machine learning methods is poor. Guidance and an increased awareness of benefits and best practices of open science are needed for prediction research in oncology.