Time to reality check the promises of machine learning-powered precision medicine

Using machine learning to develop preoperative model for lymph node metastasis in patients with bladder urothelial carcinoma

BACKGROUND

Lymph node metastasis (LNM) is associated with worse prognosis in bladder urothelial carcinoma (BUC) patients. This study aimed to develop and validate machine learning (ML) models to preoperatively predict LNM in BUC patients treated with radical cystectomy (RC).

METHODS

We retrospectively collected demographic, pathological, imaging, and laboratory information of BUC patients who underwent RC and bilateral lymphadenectomy in our institution. Patients were randomly categorized into training set and testing set. Five ML algorithms were utilized to establish prediction models. The performance of each model was assessed by the area under the receiver operating characteristic curve (AUC) and accuracy. Finally, we calculated the corresponding variable coefficients based on the optimal model to reveal the contribution of each variable to LNM.

RESULTS

A total of 524 and 131 BUC patients were finally enrolled into training set and testing set, respectively. We identified that the support vector machine (SVM) model had the best prediction ability with an AUC of 0.934 (95% confidence interval [CI]: 0.903-0.964) and accuracy of 0.916 in the training set, and an AUC of 0.855 (95%CI: 0.777-0.933) and accuracy of 0.809 in the testing set. The SVM model contained 14 predictors, and positive lymph node in imaging contributed the most to the prediction of LNM in BUC patients.

CONCLUSIONS

We developed and validated the ML models to preoperatively predict LNM in BUC patients treated with RC, and identified that the SVM model with 14 variables had the best performance and high levels of clinical applicability.

Prediction of suicide attempts among persons with depression: a population-based case cohort study

Studies have highlighted the potential importance of modeling interactions for suicide attempt prediction. This case-cohort study identified risk factors for suicide attempts among persons with depression in Denmark using statistical approaches that do (random forests) or do not (least absolute shrinkage and selection operator regression [LASSO]) model interactions. Cases made a nonfatal suicide attempt (n = 6032) between 1995 and 2015. The comparison subcohort was a 5% random sample of all persons in Denmark on January 1, 1995 (n = 11 963). We used random forests and LASSO for sex-stratified prediction of suicide attempts from demographic variables, psychiatric and somatic diagnoses, and treatments. Poisonings, psychiatric disorders, and medications were important predictors for both sexes. Area under the receiver-operating characteristic curve (AUC) values were higher in LASSO models (in men, 0.85, 95% CI, 0.84-0.86; in women, 0.89, 95% C, 0.88-0.90) than random forests (in men, 0.76, 95% CI, 0.74-0.78; in women, 0.79, 95% CI = 0.78-0.81). Automatic detection of interactions via random forests did not result in better model performance than LASSO models that did not model interactions. Due to the complex nature of psychiatric comorbidity and suicide, modeling interactions may not always be the optimal statistical approach to enhancing suicide attempt prediction in high-risk samples. This article is part of a Special Collection on Mental Health.

Predicting the risk of pancreatic cancer in adults with new-onset diabetes: development and internal-external validation of a clinical risk prediction model

BACKGROUND

The National Institute for Health and Care Excellence (NICE) recommends that people aged 60+ years with newly diagnosed diabetes and weight loss undergo abdominal imaging to assess for pancreatic cancer. More nuanced stratification could lead to enrichment of these referral pathways.

METHODS

Population-based cohort study of adults aged 30-85 years at type 2 diabetes diagnosis (2010-2021) using the QResearch primary care database in England linked to secondary care data, the national cancer registry and mortality registers. Clinical prediction models were developed to estimate risks of pancreatic cancer diagnosis within 2 years and evaluated using internal-external cross-validation.

RESULTS

Seven hundred and sixty-seven of 253,766 individuals were diagnosed with pancreatic cancer within 2 years. Models included age, sex, BMI, prior venous thromboembolism, digoxin prescription, HbA1c, ALT, creatinine, haemoglobin, platelet count; and the presence of abdominal pain, weight loss, jaundice, heartburn, indigestion or nausea (previous 6 months). The Cox model had the highest discrimination (Harrell's C-index 0.802 (95% CI: 0.797-0.817)), the highest clinical utility, and was well calibrated. The model's highest 1% of predicted risks captured 12.51% of pancreatic cancer cases. NICE guidance had 3.95% sensitivity.

DISCUSSION

A new prediction model could have clinical utility in identifying individuals with recent onset diabetes suitable for fast-track abdominal imaging.

Assessing the potential for precision medicine in body weight reduction with regard to type 2 diabetes mellitus therapies: A meta-regression analysis of 120 randomized controlled trials

AIMS

To assess the potential for precision medicine in type 2 diabetes by quantifying the variability of body weight as response to pharmacological treatment and to identify predictors which could explain this variability.

METHODS

We used randomized clinical trials (RCTs) comparing glucose-lowering drugs (including but not limited to sodium-glucose cotransporter-2 inhibitors, glucagon-like peptide-1 receptor agonists and thiazolidinediones) to placebo from four recent systematic reviews. RCTs reporting on body weight after treatment to allow for calculation of its logarithmic standard deviation (log[SD], i.e., treatment response heterogeneity) in verum (i.e., treatment) and placebo groups were included. Meta-regression analyses were performed with respect to variability of body weight after treatment and potential predictors.

RESULTS

A total of 120 RCTs with a total of 43 663 participants were analysed. A slightly larger treatment response heterogeneity was shown in the verum groups, with a median log(SD) of 2.83 compared to 2.79 from placebo. After full adjustment in the meta-regression model, the difference in body weight log(SD) was -0.026 (95% confidence interval -0.044; 0.008), with greater variability in the placebo groups. Scatterplots did not show any slope divergence (i.e., interaction) between clinical predictors and the respective treatment (verum or placebo).

CONCLUSIONS

We found no major treatment response heterogeneity in RCTs of glucose-lowering drugs for body weight reduction in type 2 diabetes. The precision medicine approach may thus be of limited value in this setting.

SPIN-PM: a consensus framework to evaluate the presence of spin in studies on prediction models

OBJECTIVES

To develop a framework to identify and evaluate spin practices and its facilitators in studies on clinical prediction model regardless of the modeling technique.

STUDY DESIGN AND SETTING

We followed a three-phase consensus process: (1) premeeting literature review to generate items to be included; (2) a series of structured meetings to provide comments discussed and exchanged viewpoints on items to be included with a panel of experienced researchers; and (3) postmeeting review on final list of items and examples to be included. Through this iterative consensus process, a framework was derived after all panel's researchers agreed.

RESULTS

This consensus process involved a panel of eight researchers and resulted in SPIN-Prediction Models which consists of two categories of spin (misleading interpretation and misleading transportability), and within these categories, two forms of spin (spin practices and facilitators of spin). We provide criteria and examples.

CONCLUSION

We proposed this guidance aiming to facilitate not only the accurate reporting but also an accurate interpretation and extrapolation of clinical prediction models which will likely improve the reporting quality of subsequent research, as well as reduce research waste.

Personalized medicine in old age psychiatry and Alzheimer's disease

Elderly patients show us unfolded lives with unique individual characteristics. An increasing life span is associated with increasing physical and mental disease burden. Alzheimer's disease (AD) is an increasing challenge in old age. AD cannot be cured but it can be treated. The complexity of old age and AD offer targets for personalized medicine (PM). Targets for stratification of patients, detection of patients at risk for AD or for future targeted therapy are plentiful and can be found in several omic-levels.

Current status of artificial intelligence technologies in pituitary adenoma surgery: a scoping review

PURPOSE

Pituitary adenoma surgery is a complex procedure due to critical adjacent neurovascular structures, variations in size and extensions of the lesions, and potential hormonal imbalances. The integration of artificial intelligence (AI) and machine learning (ML) has demonstrated considerable potential in assisting neurosurgeons in decision-making, optimizing surgical outcomes, and providing real-time feedback. This scoping review comprehensively summarizes the current status of AI/ML technologies in pituitary adenoma surgery, highlighting their strengths and limitations.

METHODS

PubMed, Embase, Web of Science, and Scopus were searched following the PRISMA-ScR guidelines. Studies discussing the use of AI/ML in pituitary adenoma surgery were included. Eligible studies were grouped to analyze the different outcomes of interest of current AI/ML technologies.

RESULTS

Among the 2438 identified articles, 44 studies met the inclusion criteria, with a total of seventeen different algorithms utilized across all studies. Studies were divided into two groups based on their input type: clinicopathological and imaging input. The four main outcome variables evaluated in the studies included: outcome (remission, recurrence or progression, gross-total resection, vision improvement, and hormonal recovery), complications (CSF leak, readmission, hyponatremia, and hypopituitarism), cost, and adenoma-related factors (aggressiveness, consistency, and Ki-67 labeling) prediction. Three studies focusing on workflow analysis and real-time navigation were discussed separately.

CONCLUSION

AI/ML modeling holds promise for improving pituitary adenoma surgery by enhancing preoperative planning and optimizing surgical strategies. However, addressing challenges such as algorithm selection, performance evaluation, data heterogeneity, and ethics is essential to establish robust and reliable ML models that can revolutionize neurosurgical practice and benefit patients.

Generative models improve fairness of medical classifiers under distribution shifts

Domain generalization is a ubiquitous challenge for machine learning in healthcare. Model performance in real-world conditions might be lower than expected because of discrepancies between the data encountered during deployment and development. Underrepresentation of some groups or conditions during model development is a common cause of this phenomenon. This challenge is often not readily addressed by targeted data acquisition and 'labeling' by expert clinicians, which can be prohibitively expensive or practically impossible because of the rarity of conditions or the available clinical expertise. We hypothesize that advances in generative artificial intelligence can help mitigate this unmet need in a steerable fashion, enriching our training dataset with synthetic examples that address shortfalls of underrepresented conditions or subgroups. We show that diffusion models can automatically learn realistic augmentations from data in a label-efficient manner. We demonstrate that learned augmentations make models more robust and statistically fair in-distribution and out of distribution. To evaluate the generality of our approach, we studied three distinct medical imaging contexts of varying difficulty: (1) histopathology, (2) chest X-ray and (3) dermatology images. Complementing real samples with synthetic ones improved the robustness of models in all three medical tasks and increased fairness by improving the accuracy of clinical diagnosis within underrepresented groups, especially out of distribution.

Assessing the properties of patient-specific treatment effect estimates from causal forest algorithms under essential heterogeneity

BACKGROUND

Treatment variation from observational data has been used to estimate patient-specific treatment effects. Causal Forest Algorithms (CFAs) developed for this task have unknown properties when treatment effect heterogeneity from unmeasured patient factors influences treatment choice - essential heterogeneity.

METHODS

We simulated eleven populations with identical treatment effect distributions based on patient factors. The populations varied in the extent that treatment effect heterogeneity influenced treatment choice. We used the generalized random forest application (CFA-GRF) to estimate patient-specific treatment effects for each population. Average differences between true and estimated effects for patient subsets were evaluated.

RESULTS

CFA-GRF performed well across the population when treatment effect heterogeneity did not influence treatment choice. Under essential heterogeneity, however, CFA-GRF yielded treatment effect estimates that reflected true treatment effects only for treated patients and were on average greater than true treatment effects for untreated patients.

CONCLUSIONS

Patient-specific estimates produced by CFAs are sensitive to why patients in real-world practice make different treatment choices. Researchers using CFAs should develop conceptual frameworks of treatment choice prior to estimation to guide estimate interpretation ex post.

Clinical Decision Support for Improved Neonatal Care: The Development of a Machine Learning Model for the Prediction of Late-onset Sepsis and Necrotizing Enterocolitis

OBJECTIVE

To develop an artificial intelligence-based software system for predicting late-onset sepsis (LOS) and necrotizing enterocolitis (NEC) in infants admitted to the neonatal intensive care unit (NICU).

STUDY DESIGN

Single-center, retrospective cohort study, conducted in the NICU of the Antwerp University Hospital. Continuous monitoring data of 865 preterm infants born at <32 weeks gestational age, admitted to the NICU in the first week of life, were used to train an XGBoost machine learning (ML) algorithm for LOS and NEC prediction in a cross-validated setup. Afterward, the model's performance was assessed on an independent test set of 148 patients (internal validation).

RESULTS

The ML model delivered hourly risk predictions with an overall sensitivity of 69% (142/206) for all LOS/NEC episodes and 81% (67/83) for severe LOS/NEC episodes. The model showed a median time gain of ≤10 hours (IQR, 3.1-21.0 hours), compared with historical clinical diagnosis. On the complete retrospective dataset, the ML model made 721 069 predictions, of which 9805 (1.3%) depicted a LOS/NEC probability of ≥0.15, resulting in a total alarm rate of <1 patient alarm-day per week. The model reached a similar performance on the internal validation set.

CONCLUSIONS

Artificial intelligence technology can assist clinicians in the early detection of LOS and NEC in the NICU, which potentially can result in clinical and socioeconomic benefits. Additional studies are required to quantify further the effect of combining artificial and human intelligence on patient outcomes in the NICU.

Integrating artificial intelligence into lung cancer screening: a randomised controlled trial protocol

INTRODUCTION

Lung cancer (LC) is the most common cause of cancer-related deaths worldwide. Its early detection can be achieved with a CT scan. Two large randomised trials proved the efficacy of low-dose CT (LDCT)-based lung cancer screening (LCS) in high-risk populations. The decrease in specific mortality is 20%-25%.Nonetheless, implementing LCS on a large scale faces obstacles due to the low number of thoracic radiologists and CT scans available for the eligible population and the high frequency of false-positive screening results and the long period of indeterminacy of nodules that can reach up to 24 months, which is a source of prolonged anxiety and multiple costly examinations with possible side effects.Deep learning, an artificial intelligence solution has shown promising results in retrospective trials detecting lung nodules and characterising them. However, until now no prospective studies have demonstrated their importance in a real-life setting.

METHODS AND ANALYSIS

This open-label randomised controlled study focuses on LCS for patients aged 50-80 years, who smoked more than 20 pack-years, whether active or quit smoking less than 15 years ago. Its objective is to determine whether assisting a multidisciplinary team (MDT) with a 3D convolutional network-based analysis of screening chest CT scans accelerates the definitive classification of nodules into malignant or benign. 2722 patients will be included with the aim to demonstrate a 3-month reduction in the delay between lung nodule detection and its definitive classification into benign or malignant.

ETHICS AND DISSEMINATION

The sponsor of this study is the University Hospital of Nice. The study was approved for France by the ethical committee CPP (Comités de Protection des Personnes) Sud-Ouest et outre-mer III (No. 2022-A01543-40) and the Agence Nationale du Medicament et des produits de Santé (Ministry of Health) in December 2023. The findings of the trial will be disseminated through peer-reviewed journals and national and international conference presentations.

TRIAL REGISTRATION NUMBER

NCT05704920.

Lesion detection in epilepsy surgery: Lessons from a prospective evaluation of a machine learning algorithm

AIM

To evaluate a lesion detection algorithm designed to detect focal cortical dysplasia (FCD) in children undergoing stereoelectroencephalography (SEEG) as part of their presurgical evaluation for drug-resistant epilepsy.

METHOD

This was a prospective, single-arm, interventional study (Idea, Development, Exploration, Assessment, and Long-Term Follow-Up phase 1/2a). After routine SEEG planning, structural magnetic resonance imaging sequences were run through an FCD lesion detection algorithm to identify putative clusters. If the top three clusters were not already sampled, up to three additional SEEG electrodes were added. The primary outcome measure was the proportion of patients who had additional electrode contacts in the SEEG-defined seizure-onset zone (SOZ).

RESULTS

Twenty patients (median age 12 years, range 4-18 years) were enrolled, one of whom did not undergo SEEG. Additional electrode contacts were part of the SOZ in 1 out of 19 patients while 3 out of 19 patients had clusters that were part of the SOZ but they were already implanted. A total of 16 additional electrodes were implanted in nine patients and there were no adverse events from the additional electrodes.

INTERPRETATION

We demonstrate early-stage prospective clinical validation of a machine learning lesion detection algorithm used to aid the identification of the SOZ in children undergoing SEEG. We share key lessons learnt from this evaluation and emphasize the importance of robust prospective evaluation before routine clinical adoption of such algorithms.

WHAT THIS PAPER ADDS

The focal cortical dysplasia detection algorithm collocated with the seizure-onset zone (SOZ) in 4 out of 19 patients. The algorithm changed the resection boundaries in 1 of 19 patients undergoing stereoelectroencephalography for drug-resistant epilepsy. The patient with an altered resection due to the algorithm was seizure-free 1 year after resective surgery. Overall, the algorithm did not increase the proportion of patients in whom SOZ was identified.

Opportunities for basic, clinical, and bioethics research at the intersection of machine learning and genomics

The data-intensive fields of genomics and machine learning (ML) are in an early stage of convergence. Genomics researchers increasingly seek to harness the power of ML methods to extract knowledge from their data; conversely, ML scientists recognize that genomics offers a wealth of large, complex, and well-annotated datasets that can be used as a substrate for developing biologically relevant algorithms and applications. The National Human Genome Research Institute (NHGRI) inquired with researchers working in these two fields to identify common challenges and receive recommendations to better support genomic research efforts using ML approaches. Those included increasing the amount and variety of training datasets by integrating genomic with multiomics, context-specific (e.g., by cell type), and social determinants of health datasets; reducing the inherent biases of training datasets; prioritizing transparency and interpretability of ML methods; and developing privacy-preserving technologies for research participants' data.

Machine learning-based clinical decision support system for treatment recommendation and overall survival prediction of hepatocellular carcinoma: a multi-center study

The treatment decisions for patients with hepatocellular carcinoma are determined by a wide range of factors, and there is a significant difference between the recommendations of widely used staging systems and the actual initial treatment choices. Herein, we propose a machine learning-based clinical decision support system suitable for use in multi-center settings. We collected data from nine institutions in South Korea for training and validation datasets. The internal and external datasets included 935 and 1750 patients, respectively. We developed a model with 20 clinical variables consisting of two stages: the first stage which recommends initial treatment using an ensemble voting machine, and the second stage, which predicts post-treatment survival using a random survival forest algorithm. We derived the first and second treatment options from the results with the highest and the second-highest probabilities given by the ensemble model and predicted their post-treatment survival. When only the first treatment option was accepted, the mean accuracy of treatment recommendation in the internal and external datasets was 67.27% and 55.34%, respectively. The accuracy increased to 87.27% and 86.06%, respectively, when the second option was included as the correct answer. Harrell's C index, integrated time-dependent AUC curve, and integrated Brier score of survival prediction in the internal and external datasets were 0.8381 and 0.7767, 91.89 and 86.48, 0.12, and 0.14, respectively. The proposed system can assist physicians by providing data-driven predictions for reference from other larger institutions or other physicians within the same institution when making treatment decisions.

Status quo and future directions of digitalization in gynecology and obstetrics in Germany: a survey of the commission Digital Medicine of the German Society for Gynecology and Obstetrics

PURPOSE

Digitalization plays a critical role and is beginning to impact every part of the patient journey, from drug discovery and data collection to treatment and patient-reported outcomes. We aimed to evaluate the status quo and future directions of digital medicine in the specialty of gynecology and obstetrics in Germany.

METHODS

An anonymous questionnaire was distributed via the German Society of Gynecology and Obstetrics newsletter in December 2022. The questionnaire covered the domains baseline demographic information, telemedicine, digital health applications (DIGAs), and future expectations.

RESULTS

In all, 91 participants completed the survey. Median age was 34 years; 67.4% (60 of 89) were female and 32.6% (29 of 89) were male. About 10% (9 of 88) have prescribed DIGAs to date and 14% (12 of 86) offer telemedical appointments. Among those who do not use digital medicine, very few plan to do so in the near future. Reasons include missing software interfaces, lack of time to try out new things, lack of knowledge, lack of monetary compensation (66.3%), and employee concerns. A majority agreed that digitalization will help to save time and improve patient care and that intelligent algorithms will aid clinicians in providing patient care to women.

CONCLUSIONS

The status quo and future directions of digital medicine in gynecology and obstetrics in Germany are characterized by contradicting expectations regarding the benefits of digital medicine and its actual implementation in clinical routine. This represents an important call to action to meet the requirements of modern patient care.

The Influence of Using Novel Predictive Technologies on Judgments of Stigma, Empathy, and Compassion among Healthcare Professionals

BACKGROUND

Our objective was to evaluate whether the description of a machine learning (ML) app or brain imaging technology to predict the onset of schizophrenia or alcohol use disorder (AUD) influences healthcare professionals' judgments of stigma, empathy, and compassion.

METHODS

We randomized healthcare professionals (N = 310) to one vignette about a person whose clinician seeks to predict schizophrenia or an AUD, using a ML app, brain imaging, or a psychosocial assessment. Participants used scales to measure their judgments of stigma, empathy, and compassion.

RESULTS

Participants randomized to the ML vignette endorsed less anger and more fear relative to the psychosocial vignette, and the brain imaging vignette elicited higher pity ratings. The brain imaging and ML vignettes evoked lower personal responsibility judgments compared to the psychosocial vignette. Physicians and nurses reported less empathy than clinical psychologists.

CONCLUSIONS

The use of predictive technologies may reinforce essentialist views about mental health and substance use that may increase specific aspects of stigma and reduce others.

Machine Learning and Artificial Intelligence Applications to Epilepsy: a Review for the Practicing Epileptologist

PURPOSE OF REVIEW

Machine Learning (ML) and Artificial Intelligence (AI) are data-driven techniques to translate raw data into applicable and interpretable insights that can assist in clinical decision making. Some of these tools have extremely promising initial results, earning both great excitement and creating hype. This non-technical article reviews recent developments in ML/AI in epilepsy to assist the current practicing epileptologist in understanding both the benefits and limitations of integrating ML/AI tools into their clinical practice.

RECENT FINDINGS

ML/AI tools have been developed to assist clinicians in almost every clinical decision including (1) predicting future epilepsy in people at risk, (2) detecting and monitoring for seizures, (3) differentiating epilepsy from mimics, (4) using data to improve neuroanatomic localization and lateralization, and (5) tracking and predicting response to medical and surgical treatments. We also discuss practical, ethical, and equity considerations in the development and application of ML/AI tools including chatbots based on Large Language Models (e.g., ChatGPT). ML/AI tools will change how clinical medicine is practiced, but, with rare exceptions, the transferability to other centers, effectiveness, and safety of these approaches have not yet been established rigorously. In the future, ML/AI will not replace epileptologists, but epileptologists with ML/AI will replace epileptologists without ML/AI.

Wavelet scattering networks in deep learning for discovering protein markers in a cohort of Swedish rectal cancer patients

BACKGROUND

Cancer biomarkers play a pivotal role in the diagnosis, prognosis, and treatment response prediction of the disease. In this study, we analyzed the expression levels of RhoB and DNp73 proteins in rectal cancer, as captured in immunohistochemical images, to predict the 5-year survival time of two patient groups: one with preoperative radiotherapy and one without.

METHODS

The utilization of deep convolutional neural networks in medical research, particularly in clinical cancer studies, has been gaining substantial attention. This success primarily stems from their ability to extract intricate image features that prove invaluable in machine learning. Another innovative method for extracting features at multiple levels is the wavelet-scattering network. Our study combines the strengths of these two convolution-based approaches to robustly extract image features related to protein expression.

RESULTS

The efficacy of our approach was evaluated across various tissue types, including tumor, biopsy, metastasis, and adjacent normal tissue. Statistical assessments demonstrated exceptional performance across a range of metrics, including prediction accuracy, classification accuracy, precision, and the area under the receiver operating characteristic curve.

CONCLUSION

These results underscore the potential of dual convolutional learning to assist clinical researchers in the timely validation and discovery of cancer biomarkers.

Multiclass risk models for ovarian malignancy: an illustration of prediction uncertainty due to the choice of algorithm

BACKGROUND

Assessing malignancy risk is important to choose appropriate management of ovarian tumors. We compared six algorithms to estimate the probabilities that an ovarian tumor is benign, borderline malignant, stage I primary invasive, stage II-IV primary invasive, or secondary metastatic.

METHODS

This retrospective cohort study used 5909 patients recruited from 1999 to 2012 for model development, and 3199 patients recruited from 2012 to 2015 for model validation. Patients were recruited at oncology referral or general centers and underwent an ultrasound examination and surgery ≤ 120 days later. We developed models using standard multinomial logistic regression (MLR), Ridge MLR, random forest (RF), XGBoost, neural networks (NN), and support vector machines (SVM). We used nine clinical and ultrasound predictors but developed models with or without CA125.

RESULTS

Most tumors were benign (3980 in development and 1688 in validation data), secondary metastatic tumors were least common (246 and 172). The c-statistic (AUROC) to discriminate benign from any type of malignant tumor ranged from 0.89 to 0.92 for models with CA125, from 0.89 to 0.91 for models without. The multiclass c-statistic ranged from 0.41 (SVM) to 0.55 (XGBoost) for models with CA125, and from 0.42 (SVM) to 0.51 (standard MLR) for models without. Multiclass calibration was best for RF and XGBoost. Estimated probabilities for a benign tumor in the same patient often differed by more than 0.2 (20% points) depending on the model. Net Benefit for diagnosing malignancy was similar for algorithms at the commonly used 10% risk threshold, but was slightly higher for RF at higher thresholds. Comparing models, between 3% (XGBoost vs. NN, with CA125) and 30% (NN vs. SVM, without CA125) of patients fell on opposite sides of the 10% threshold.

CONCLUSION

Although several models had similarly good performance, individual probability estimates varied substantially.

How amenable is type 2 diabetes treatment for precision diabetology? A meta-regression of glycaemic control data from 174 randomised trials

AIMS/HYPOTHESIS

There are two prerequisites for the precision medicine approach to be beneficial for treated individuals. First, there must be treatment heterogeneity; second, in the case of treatment heterogeneity, we need to detect clinical predictors to identify people who would benefit from one treatment more than from others. There is an established meta-regression approach to assess these two prerequisites that relies on measuring the variability of a clinical outcome after treatment in placebo-controlled randomised trials. Our aim was to apply this approach to the treatment of type 2 diabetes.

METHODS

We performed a meta-regression analysis using information from 174 placebo-controlled randomised trials with 178 placebo and 272 verum (i.e. active treatment) arms including 86,940 participants with respect to the variability of glycaemic control as assessed by HbA1c after treatment and its potential predictors.

RESULTS

The adjusted difference in log(SD) values between the verum and placebo arms was 0.037 (95% CI: 0.004, 0.069). That is, we found a small increase in the variability of HbA1c values after treatment in the verum arms. In addition, one potentially relevant predictor for explaining this increase, drug class, was observed, and GLP-1 receptor agonists yielded the largest differences in log(SD) values.

CONCLUSIONS/INTERPRETATION

The potential of the precision medicine approach in the treatment of type 2 diabetes is modest at best, at least with regard to an improvement in glycaemic control. Our finding of a larger variability after treatment with GLP-1 receptor agonists in individuals with poor glycaemic control should be replicated and/or validated with other clinical outcomes and with different study designs.

FUNDING

The research reported here received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

DATA AVAILABILITY

Two datasets (one for the log[SD] and one for the baseline-corrected log[SD]) to reproduce the analyses from this paper are available on https://zenodo.org/record/7956635 .

Artificial intelligence vs. evolving super-complex tumor intelligence: critical viewpoints

Recent developments in various domains have led to a growing interest in the potential of artificial intelligence to enhance our lives and environments. In particular, the application of artificial intelligence in the management of complex human diseases, such as cancer, has garnered significant attention. The evolution of artificial intelligence is thought to be influenced by multiple factors, including human intervention and environmental factors. Similarly, tumors, being heterogeneous and complex diseases, continue to evolve due to changes in the physical, chemical, and biological environment. Additionally, the concept of cellular intelligence within biological systems has been recognized as a potential attribute of biological entities. Therefore, it is plausible that the tumor intelligence present in cancer cells of affected individuals could undergo super-evolution due to changes in the pro-tumor environment. Thus, a comparative analysis of the evolution of artificial intelligence and super-complex tumor intelligence could yield valuable insights to develop better artificial intelligence-based tools for cancer management.

Development & Deployment of a Real-time Healthcare Predictive Analytics Platform

The deployment of predictive analytic algorithms that can safely and seamlessly integrate into existing healthcare workflows remains a significant challenge. Here, we present a scalable, cloud-based, fault-tolerant platform that is capable of extracting and processing electronic health record (EHR) data for any patient at any time following admission and transferring results back into the EHR. This platform has been successfully deployed within the UC San Diego Health system and utilizes interoperable data standards to enable portability.Clinical relevance- This platform is currently hosting a deep learning model for the early prediction of sepsis that is operational in two emergency departments.

Generating synthetic mixed-type longitudinal electronic health records for artificial intelligent applications

The recent availability of electronic health records (EHRs) have provided enormous opportunities to develop artificial intelligence (AI) algorithms. However, patient privacy has become a major concern that limits data sharing across hospital settings and subsequently hinders the advances in AI. Synthetic data, which benefits from the development and proliferation of generative models, has served as a promising substitute for real patient EHR data. However, the current generative models are limited as they only generate single type of clinical data for a synthetic patient, i.e., either continuous-valued or discrete-valued. To mimic the nature of clinical decision-making which encompasses various data types/sources, in this study, we propose a generative adversarial network (GAN) entitled EHR-M-GAN that simultaneously synthesizes mixed-type timeseries EHR data. EHR-M-GAN is capable of capturing the multidimensional, heterogeneous, and correlated temporal dynamics in patient trajectories. We have validated EHR-M-GAN on three publicly-available intensive care unit databases with records from a total of 141,488 unique patients, and performed privacy risk evaluation of the proposed model. EHR-M-GAN has demonstrated its superiority over state-of-the-art benchmarks for synthesizing clinical timeseries with high fidelity, while addressing the limitations regarding data types and dimensionality in the current generative models. Notably, prediction models for outcomes of intensive care performed significantly better when training data was augmented with the addition of EHR-M-GAN-generated timeseries. EHR-M-GAN may have use in developing AI algorithms in resource-limited settings, lowering the barrier for data acquisition while preserving patient privacy.

Development and internal-external validation of statistical and machine learning models for breast cancer prognostication: cohort study

OBJECTIVE

To develop a clinically useful model that estimates the 10 year risk of breast cancer related mortality in women (self-reported female sex) with breast cancer of any stage, comparing results from regression and machine learning approaches.

DESIGN

Population based cohort study.

SETTING

QResearch primary care database in England, with individual level linkage to the national cancer registry, Hospital Episodes Statistics, and national mortality registers.

PARTICIPANTS

141 765 women aged 20 years and older with a diagnosis of invasive breast cancer between 1 January 2000 and 31 December 2020.

MAIN OUTCOME MEASURES

Four model building strategies comprising two regression (Cox proportional hazards and competing risks regression) and two machine learning (XGBoost and an artificial neural network) approaches. Internal-external cross validation was used for model evaluation. Random effects meta-analysis that pooled estimates of discrimination and calibration metrics, calibration plots, and decision curve analysis were used to assess model performance, transportability, and clinical utility.

RESULTS

During a median 4.16 years (interquartile range 1.76-8.26) of follow-up, 21 688 breast cancer related deaths and 11 454 deaths from other causes occurred. Restricting to 10 years maximum follow-up from breast cancer diagnosis, 20 367 breast cancer related deaths occurred during a total of 688 564.81 person years. The crude breast cancer mortality rate was 295.79 per 10 000 person years (95% confidence interval 291.75 to 299.88). Predictors varied for each regression model, but both Cox and competing risks models included age at diagnosis, body mass index, smoking status, route to diagnosis, hormone receptor status, cancer stage, and grade of breast cancer. The Cox model's random effects meta-analysis pooled estimate for Harrell's C index was the highest of any model at 0.858 (95% confidence interval 0.853 to 0.864, and 95% prediction interval 0.843 to 0.873). It appeared acceptably calibrated on calibration plots. The competing risks regression model had good discrimination: pooled Harrell's C index 0.849 (0.839 to 0.859, and 0.821 to 0.876, and evidence of systematic miscalibration on summary metrics was lacking. The machine learning models had acceptable discrimination overall (Harrell's C index: XGBoost 0.821 (0.813 to 0.828, and 0.805 to 0.837); neural network 0.847 (0.835 to 0.858, and 0.816 to 0.878)), but had more complex patterns of miscalibration and more variable regional and stage specific performance. Decision curve analysis suggested that the Cox and competing risks regression models tested may have higher clinical utility than the two machine learning approaches.

CONCLUSION

In women with breast cancer of any stage, using the predictors available in this dataset, regression based methods had better and more consistent performance compared with machine learning approaches and may be worthy of further evaluation for potential clinical use, such as for stratified follow-up.

Artificial intelligence in medical device software and high-risk medical devices - a review of definitions, expert recommendations and regulatory initiatives

INTRODUCTION

Artificial intelligence (AI) encompasses a wide range of algorithms with risks when used to support decisions about diagnosis or treatment, so professional and regulatory bodies are recommending how they should be managed.

AREAS COVERED

AI systems may qualify as standalone medical device software (MDSW) or be embedded within a medical device. Within the European Union (EU) AI software must undergo a conformity assessment procedure to be approved as a medical device. The draft EU Regulation on AI proposes rules that will apply across industry sectors, while for devices the Medical Device Regulation also applies. In the CORE-MD project (Coordinating Research and Evidence for Medical Devices), we have surveyed definitions and summarize initiatives made by professional consensus groups, regulators, and standardization bodies.

EXPERT OPINION

The level of clinical evidence required should be determined according to each application and to legal and methodological factors that contribute to risk, including accountability, transparency, and interpretability. EU guidance for MDSW based on international recommendations does not yet describe the clinical evidence needed for medical AI software. Regulators, notified bodies, manufacturers, clinicians and patients would all benefit from common standards for the clinical evaluation of high-risk AI applications and transparency of their evidence and performance.

Machine learning does not outperform traditional statistical modelling for kidney allograft failure prediction

Machine learning (ML) models have recently shown potential for predicting kidney allograft outcomes. However, their ability to outperform traditional approaches remains poorly investigated. Therefore, using large cohorts of kidney transplant recipients from 14 centers worldwide, we developed ML-based prediction models for kidney allograft survival and compared their prediction performances to those achieved by a validated Cox-Based Prognostication System (CBPS). In a French derivation cohort of 4000 patients, candidate determinants of allograft failure including donor, recipient and transplant-related parameters were used as predictors to develop tree-based models (RSF, RSF-ERT, CIF), Support Vector Machine models (LK-SVM, AK-SVM) and a gradient boosting model (XGBoost). Models were externally validated with cohorts of 2214 patients from Europe, 1537 from North America, and 671 from South America. Among these 8422 kidney transplant recipients, 1081 (12.84%) lost their grafts after a median post-transplant follow-up time of 6.25 years (Inter Quartile Range 4.33-8.73). At seven years post-risk evaluation, the ML models achieved a C-index of 0.788 (95% bootstrap percentile confidence interval 0.736-0.833), 0.779 (0.724-0.825), 0.786 (0.735-0.832), 0.527 (0.456-0.602), 0.704 (0.648-0.759) and 0.767 (0.711-0.815) for RSF, RSF-ERT, CIF, LK-SVM, AK-SVM and XGBoost respectively, compared with 0.808 (0.792-0.829) for the CBPS. In validation cohorts, ML models' discrimination performances were in a similar range of those of the CBPS. Calibrations of the ML models were similar or less accurate than those of the CBPS. Thus, when using a transparent methodological pipeline in validated international cohorts, ML models, despite overall good performances, do not outperform a traditional CBPS in predicting kidney allograft failure. Hence, our current study supports the continued use of traditional statistical approaches for kidney graft prognostication.

Critical Appraisal of Artificial Intelligence-Enabled Imaging Tools Using the Levels of Evidence System

Clinical adoption of an artificial intelligence-enabled imaging tool requires critical appraisal of its life cycle from development to implementation by using a systematic, standardized, and objective approach that can verify both its technical and clinical efficacy. Toward this concerted effort, the ASFNR/ASNR Artificial Intelligence Workshop Technology Working Group is proposing a hierarchal evaluation system based on the quality, type, and amount of scientific evidence that the artificial intelligence-enabled tool can demonstrate for each component of its life cycle. The current proposal is modeled after the levels of evidence in medicine, with the uppermost level of the hierarchy showing the strongest evidence for potential impact on patient care and health care outcomes. The intended goal of establishing an evidence-based evaluation system is to encourage transparency, foster an understanding of the creation of artificial intelligence tools and the artificial intelligence decision-making process, and to report the relevant data on the efficacy of artificial intelligence tools that are developed. The proposed system is an essential step in working toward a more formalized, clinically validated, and regulated framework for the safe and effective deployment of artificial intelligence imaging applications that will be used in clinical practice.

Development & Deployment of a Real-time Healthcare Predictive Analytics Platform

The deployment of predictive analytic algorithms that can safely and seamlessly integrate into existing healthcare workflows remains a significant challenge. Here, we present a scalable, cloud-based, fault-tolerant platform that is capable of extracting and processing electronic health record (EHR) data for any patient at any time following admission and transferring results back into the EHR. This platform has been successfully deployed within the UC San Diego Health system and utilizes interoperable data standards to enable portability.

DeepGeni: deep generalized interpretable autoencoder elucidates gut microbiota for better cancer immunotherapy

Recent studies revealed that gut microbiota modulates the response to cancer immunotherapy and fecal microbiota transplantation has clinical benefits in melanoma patients during treatment. Understanding how microbiota affects individual responses is crucial for precision oncology. However, it is challenging to identify key microbial taxa with limited data as statistical and machine learning models often lose their generalizability. In this study, DeepGeni, a deep generalized interpretable autoencoder, is proposed to improve the generalizability and interpretability of microbiome profiles by augmenting data and by introducing interpretable links in the autoencoder. DeepGeni-based machine learning classifier outperforms state-of-the-art classifier in the microbiome-driven prediction of responsiveness of melanoma patients treated with immune checkpoint inhibitors. Moreover, the interpretable links of DeepGeni elucidate the most informative microbiota associated with cancer immunotherapy response. DeepGeni not only improves microbiome-driven prediction of immune checkpoint inhibitor responsiveness but also suggests potential microbial targets for fecal microbiota transplant or probiotics improving the outcome of cancer immunotherapy.

Machine Learning Models to Forecast Outcomes of Pituitary Surgery: A Systematic Review in Quality of Reporting and Current Evidence

Background: The complex nature and heterogeneity involving pituitary surgery results have increased interest in machine learning (ML) applications for prediction of outcomes over the last decade. This study aims to systematically review the characteristics of ML models involving pituitary surgery outcome prediction and assess their reporting quality. Methods: We searched the PubMed, Scopus, and Web of Knowledge databases for publications on the use of ML to predict pituitary surgery outcomes. We used the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) to assess report quality. Our search strategy was based on the terms “artificial intelligence”, “machine learning”, and “pituitary”. Results: 20 studies were included in this review. The principal models reported in each article were post-surgical endocrine outcomes (n = 10), tumor management (n = 3), and intra- and postoperative complications (n = 7). Overall, the included studies adhered to a median of 65% (IQR = 60–72%) of TRIPOD criteria, ranging from 43% to 83%. The median reported AUC was 0.84 (IQR = 0.80–0.91). The most popular algorithms were support vector machine (n = 5) and random forest (n = 5). Only two studies reported external validation and adherence to any reporting guideline. Calibration methods were not reported in 15 studies. No model achieved the phase of actual clinical applicability. Conclusion: Applications of ML in the prediction of pituitary outcomes are still nascent, as evidenced by the lack of any model validated for clinical practice. Although studies have demonstrated promising results, greater transparency in model development and reporting is needed to enable their use in clinical practice. Further adherence to reporting guidelines can help increase AI’s real-world utility and improve clinical practice.

The foundation and architecture of precision medicine in neurology and psychiatry

Neurological and psychiatric diseases have high degrees of genetic and pathophysiological heterogeneity, irrespective of clinical manifestations. Traditional medical paradigms have focused on late-stage syndromic aspects of these diseases, with little consideration of the underlying biology. Advances in disease modeling and methodological design have paved the way for the development of precision medicine (PM), an established concept in oncology with growing attention from other medical specialties. We propose a PM architecture for central nervous system diseases built on four converging pillars: multimodal biomarkers, systems medicine, digital health technologies, and data science. We discuss Alzheimer's disease (AD), an area of significant unmet medical need, as a case-in-point for the proposed framework. AD can be seen as one of the most advanced PM-oriented disease models and as a compelling catalyzer towards PM-oriented neuroscience drug development and advanced healthcare practice.

Toward Precision Medicine: Circadian Rhythm of Blood Pressure and Chronotherapy for Hypertension - 2021 NHLBI Workshop Report

Healthy individuals exhibit blood pressure variation over a 24-hour period with higher blood pressure during wakefulness and lower blood pressure during sleep. Loss or disruption of the blood pressure circadian rhythm has been linked to adverse health outcomes, for example, cardiovascular disease, dementia, and chronic kidney disease. However, the current diagnostic and therapeutic approaches lack sufficient attention to the circadian rhythmicity of blood pressure. Sleep patterns, hormone release, eating habits, digestion, body temperature, renal and cardiovascular function, and other important host functions as well as gut microbiota exhibit circadian rhythms, and influence circadian rhythms of blood pressure. Potential benefits of nonpharmacologic interventions such as meal timing, and pharmacologic chronotherapeutic interventions, such as the bedtime administration of antihypertensive medications, have recently been suggested in some studies. However, the mechanisms underlying circadian rhythm-mediated blood pressure regulation and the efficacy of chronotherapy in hypertension remain unclear. This review summarizes the results of the National Heart, Lung, and Blood Institute workshop convened on October 27 to 29, 2021 to assess knowledge gaps and research opportunities in the study of circadian rhythm of blood pressure and chronotherapy for hypertension.

Human-guided deep learning with ante-hoc explainability by convolutional network from non-image data for pregnancy prognostication

BACKGROUND AND OBJECTIVE

Deep learning is applied in medicine mostly due to its state-of-the-art performance for diagnostic imaging. Supervisory authorities also require the model to be explainable, but most explain the model after development (post hoc) instead of incorporating explanation into the design (ante hoc). This study aimed to demonstrate a human-guided deep learning with ante-hoc explainability by convolutional network from non-image data to develop, validate, and deploy a prognostic prediction model for PROM and an estimator of time of delivery using a nationwide health insurance database.

METHODS

To guide modeling, we constructed and verified association diagrams respectively from literatures and electronic health records. Non-image data were transformed into meaningful images utilizing predictor-to-predictor similarities, harnessing the power of convolutional neural network mostly used for diagnostic imaging. The network architecture was also inferred from the similarities.

RESULTS

This resulted the best model for prelabor rupture of membranes (n=883, 376) with the area under curves 0.73 (95% CI 0.72 to 0.75) and 0.70 (95% CI 0.69 to 0.71) respectively by internal and external validations, and outperformed previous models found by systematic review. It was explainable by knowledge-based diagrams and model representation.

CONCLUSIONS

This allows prognostication with actionable insights for preventive medicine.

Logistic regression technique is comparable to complex machine learning algorithms in predicting cognitive impairment related to post intensive care syndrome

To evaluate the performance of machine learning (ML) models and to compare it with logistic regression (LR) technique in predicting cognitive impairment related to post intensive care syndrome (PICS-CI). We conducted a prospective observational study of ICU patients at two tertiary hospitals. A cohort of 2079 patients was screened, and finally 481 patients were included. Seven different ML models were considered, decision tree (DT), random forest (RF), XGBoost, neural network (NN), naïve bayes (NB), and support vector machine (SVM), and compared with logistic regression (LR). Discriminative ability was evaluated by area under the receiver operating characteristic curve (AUC), calibration belt plots, and Hosmer-Lemeshow test was used to assess calibration. Decision curve analysis was performed to quantify clinical utility. Duration of delirium, poor Richards-Campbell sleep questionnaire (RCSQ) score, advanced age, and sepsis were the most frequent and important candidates risk factors for PICS-CI. All ML models showed good performance (AUC range: 0.822-0.906). NN model had the highest AUC (0.906 [95% CI 0.857-0.955]), which was slightly higher than, but not significantly different from that of LR (0.898 [95% CI 0.847-0.949]) (P > 0.05, Delong test). Given the overfitting and complexity of some ML models, the LR model was then used to develop a web-based risk calculator to aid decision-making ( https://model871010.shinyapps.io/dynnomapp/ ). In a low dimensional data, LR may yield as good performance as other complex ML models to predict cognitive impairment after ICU hospitalization.

Modern views of machine learning for precision psychiatry

In light of the National Institute of Mental Health (NIMH)'s Research Domain Criteria (RDoC), the advent of functional neuroimaging, novel technologies and methods provide new opportunities to develop precise and personalized prognosis and diagnosis of mental disorders. Machine learning (ML) and artificial intelligence (AI) technologies are playing an increasingly critical role in the new era of precision psychiatry. Combining ML/AI with neuromodulation technologies can potentially provide explainable solutions in clinical practice and effective therapeutic treatment. Advanced wearable and mobile technologies also call for the new role of ML/AI for digital phenotyping in mobile mental health. In this review, we provide a comprehensive review of ML methodologies and applications by combining neuroimaging, neuromodulation, and advanced mobile technologies in psychiatry practice. We further review the role of ML in molecular phenotyping and cross-species biomarker identification in precision psychiatry. We also discuss explainable AI (XAI) and neuromodulation in a closed human-in-the-loop manner and highlight the ML potential in multi-media information extraction and multi-modal data fusion. Finally, we discuss conceptual and practical challenges in precision psychiatry and highlight ML opportunities in future research.

Accuracy of Machine Learning Classification Models for the Prediction of Type 2 Diabetes Mellitus: A Systematic Survey and Meta-Analysis Approach

Soft-computing and statistical learning models have gained substantial momentum in predicting type 2 diabetes mellitus (T2DM) disease. This paper reviews recent soft-computing and statistical learning models in T2DM using a meta-analysis approach. We searched for papers using soft-computing and statistical learning models focused on T2DM published between 2010 and 2021 on three different search engines. Of 1215 studies identified, 34 with 136952 patients met our inclusion criteria. The pooled algorithm's performance was able to predict T2DM with an overall accuracy of 0.86 (95% confidence interval [CI] of [0.82, 0.89]). The classification of diabetes prediction was significantly greater in models with a screening and diagnosis (pooled proportion [95% CI] = 0.91 [0.74, 0.97]) when compared to models with nephropathy (pooled proportion = 0.48 [0.76, 0.89] to 0.88 [0.83, 0.91]). For the prediction of T2DM, the decision trees (DT) models had a pooled accuracy of 0.88 [95% CI: 0.82, 0.92], and the neural network (NN) models had a pooled accuracy of 0.85 [95% CI: 0.79, 0.89]. Meta-regression did not provide any statistically significant findings for the heterogeneous accuracy in studies with different diabetes predictions, sample sizes, and impact factors. Additionally, ML models showed high accuracy for the prediction of T2DM. The predictive accuracy of ML algorithms in T2DM is promising, mainly through DT and NN models. However, there is heterogeneity among ML models. We compared the results and models and concluded that this evidence might help clinicians interpret data and implement optimum models for their dataset for T2DM prediction.

Prediction of black carbon in marine engines and correlation analysis of model characteristics based on multiple machine learning algorithms

Ship black carbon emissions have caused great harm to ecological environment. In order to estimate the black carbon emissions, thereby reducing the cost of black carbon experiments, here, we introduced four machine learning algorithms which are lasso regression, support vector machine, extreme gradient boosting, and artificial neural network to predict ship black carbon emissions. The prediction models were established with using the datasets acquired from similar marine engines under various steady-state conditions. The results show that SVM, XGB, and ANN have higher prediction accuracy than lasso regression, and the adjusted R² of each model is 0.9810, 0.9850, 0.9885, and 0.6088. Although ANN shows the best prediction performance, it is inferior to SVM and XGB in terms of model stability and training cost. Then, in order to simplify the optimization process of hyperparameters and improve the prediction accuracy of the model at the same time, we use three different swarm intelligence algorithms to automatically optimize the hyperparameters of SVM and XGB. In addition, we applied mutual information to measure the correlation between the characteristics of the prediction models and black carbon concentration and found that the characteristics which related to in-cylinder combustion have a strong correlation with the black carbon concentration. The findings in this paper prove the feasibility of machine learning in ship black carbon emission prediction and could provide references for reducing ship black carbon emissions and the formulation of emission regulations.

Machine learning in medicine: a practical introduction to techniques for data pre-processing, hyperparameter tuning, and model comparison

BACKGROUND

There is growing enthusiasm for the application of machine learning (ML) and artificial intelligence (AI) techniques to clinical research and practice. However, instructions on how to develop robust high-quality ML and AI in medicine are scarce. In this paper, we provide a practical example of techniques that facilitate the development of high-quality ML systems including data pre-processing, hyperparameter tuning, and model comparison using open-source software and data.

METHODS

We used open-source software and a publicly available dataset to train and validate multiple ML models to classify breast masses into benign or malignant using mammography image features and patient age. We compared algorithm predictions to the ground truth of histopathologic evaluation. We provide step-by-step instructions with accompanying code lines.

FINDINGS

Performance of the five algorithms at classifying breast masses as benign or malignant based on mammography image features and patient age was statistically equivalent (P > 0.05). Area under the receiver operating characteristics curve (AUROC) for the logistic regression with elastic net penalty was 0.89 (95% CI 0.85 - 0.94), for the Extreme Gradient Boosting Tree 0.88 (95% CI 0.83 - 0.93), for the Multivariate Adaptive Regression Spline algorithm 0.88 (95% CI 0.83 - 0.93), for the Support Vector Machine 0.89 (95% CI 0.84 - 0.93), and for the neural network 0.89 (95% CI 0.84 - 0.93).

INTERPRETATION

Our paper allows clinicians and medical researchers who are interested in using ML algorithms to understand and recreate the elements of a comprehensive ML analysis. Following our instructions may help to improve model generalizability and reproducibility in medical ML studies.

Ethical considerations for precision psychiatry: A roadmap for research and clinical practice

Precision psychiatry is an emerging field with transformative opportunities for mental health. However, the use of clinical prediction models carries unprecedented ethical challenges, which must be addressed before accessing the potential benefits of precision psychiatry. This critical review covers multidisciplinary areas, including psychiatry, ethics, statistics and machine-learning, healthcare and academia, as well as input from people with lived experience of mental disorders, their family, and carers. We aimed to identify core ethical considerations for precision psychiatry and mitigate concerns by designing a roadmap for research and clinical practice. We identified priorities: learning from somatic medicine; identifying precision psychiatry use cases; enhancing transparency and generalizability; fostering implementation; promoting mental health literacy; communicating risk estimates; data protection and privacy; and fostering the equitable distribution of mental health care. We hope this blueprint will advance research and practice and enable people with mental health problems to benefit from precision psychiatry.

Moving towards vertically integrated artificial intelligence development

Substantial interest and investment in clinical artificial intelligence (AI) research has not resulted in widespread translation to deployed AI solutions. Current attention has focused on bias and explainability in AI algorithm development, external validity and model generalisability, and lack of equity and representation in existing data. While of great importance, these considerations also reflect a model-centric approach seen in published clinical AI research, which focuses on optimising architecture and performance of an AI model on best available datasets. However, even robustly built models using state-of-the-art algorithms may fail once tested in realistic environments due to unpredictability of real-world conditions, out-of-dataset scenarios, characteristics of deployment infrastructure, and lack of added value to clinical workflows relative to cost and potential clinical risks. In this perspective, we define a vertically integrated approach to AI development that incorporates early, cross-disciplinary, consideration of impact evaluation, data lifecycles, and AI production, and explore its implementation in two contrasting AI development pipelines: a scalable “AI factory” (Mayo Clinic, Rochester, United States), and an end-to-end cervical cancer screening platform for resource poor settings (Paps AI, Mbarara, Uganda). We provide practical recommendations for implementers, and discuss future challenges and novel approaches (including a decentralised federated architecture being developed in the NHS (AI4VBH, London, UK)). Growth in global clinical AI research continues unabated, and introduction of vertically integrated teams and development practices can increase the translational potential of future clinical AI projects.

Subpopulation-specific machine learning prognosis for underrepresented patients with double prioritized bias correction

Background

Many clinical datasets are intrinsically imbalanced, dominated by overwhelming majority groups. Off-the-shelf machine learning models that optimize the prognosis of majority patient types (e.g., healthy class) may cause substantial errors on the minority prediction class (e.g., disease class) and demographic subgroups (e.g., Black or young patients). In the typical one-machine-learning-model-fits-all paradigm, racial and age disparities are likely to exist, but unreported. In addition, some widely used whole-population metrics give misleading results.

Methods

We design a double prioritized (DP) bias correction technique to mitigate representational biases in machine learning-based prognosis. Our method trains customized machine learning models for specific ethnicity or age groups, a substantial departure from the one-model-predicts-all convention. We compare with other sampling and reweighting techniques in mortality and cancer survivability prediction tasks.

Results

We first provide empirical evidence showing various prediction deficiencies in a typical machine learning setting without bias correction. For example, missed death cases are 3.14 times higher than missed survival cases for mortality prediction. Then, we show DP consistently boosts the minority class recall for underrepresented groups, by up to 38.0%. DP also reduces relative disparities across race and age groups, e.g., up to 88.0% better than the 8 existing sampling solutions in terms of the relative disparity of minority class recall. Cross-race and cross-age-group evaluation also suggests the need for subpopulation-specific machine learning models.

Conclusions

Biases exist in the widely accepted one-machine-learning-model-fits-all-population approach. We invent a bias correction method that produces specialized machine learning prognostication models for underrepresented racial and age groups. This technique may reduce potentially life-threatening prediction mistakes for minority populations.

Randomized Clinical Trials of Machine Learning Interventions in Health Care: A Systematic Review

IMPORTANCE

Despite the potential of machine learning to improve multiple aspects of patient care, barriers to clinical adoption remain. Randomized clinical trials (RCTs) are often a prerequisite to large-scale clinical adoption of an intervention, and important questions remain regarding how machine learning interventions are being incorporated into clinical trials in health care.

OBJECTIVE

To systematically examine the design, reporting standards, risk of bias, and inclusivity of RCTs for medical machine learning interventions.

EVIDENCE REVIEW

In this systematic review, the Cochrane Library, Google Scholar, Ovid Embase, Ovid MEDLINE, PubMed, Scopus, and Web of Science Core Collection online databases were searched and citation chasing was done to find relevant articles published from the inception of each database to October 15, 2021. Search terms for machine learning, clinical decision-making, and RCTs were used. Exclusion criteria included implementation of a non-RCT design, absence of original data, and evaluation of nonclinical interventions. Data were extracted from published articles. Trial characteristics, including primary intervention, demographics, adherence to the CONSORT-AI reporting guideline, and Cochrane risk of bias were analyzed.

FINDINGS

Literature search yielded 19 737 articles, of which 41 RCTs involved a median of 294 participants (range, 17-2488 participants). A total of 16 RCTS (39%) were published in 2021, 21 (51%) were conducted at single sites, and 15 (37%) involved endoscopy. No trials adhered to all CONSORT-AI standards. Common reasons for nonadherence were not assessing poor-quality or unavailable input data (38 trials [93%]), not analyzing performance errors (38 [93%]), and not including a statement regarding code or algorithm availability (37 [90%]). Overall risk of bias was high in 7 trials (17%). Of 11 trials (27%) that reported race and ethnicity data, the median proportion of participants from underrepresented minority groups was 21% (range, 0%-51%).

CONCLUSIONS AND RELEVANCE

This systematic review found that despite the large number of medical machine learning-based algorithms in development, few RCTs for these technologies have been conducted. Among published RCTs, there was high variability in adherence to reporting standards and risk of bias and a lack of participants from underrepresented minority groups. These findings merit attention and should be considered in future RCT design and reporting.

Predictive Power for Thrombus Detection after Atrial Appendage Closure: Machine Learning vs. Classical Methods

Device-related thrombus (DRT) after left atrial appendage (LAA) closure is infrequent but correlates with an increased risk of thromboembolism. Therefore, the search for DRT predictors is a topic of interest. In the literature, multivariable methods have been used achieving non-consistent results, and to the best of our knowledge, machine learning techniques have not been used yet for thrombus detection after LAA occlusion. Our aim is to compare both methodologies with respect to predictive power and the search for predictors of DRT. To this end, a multicenter study including 1150 patients who underwent LAA closure was analyzed. Two lines of experiments were performed: with and without resampling. Multivariate and machine learning methodologies were applied to both lines. Predictive power and the extracted predictors for all experiments were gathered. ROC curves of 0.5446 and 0.7974 were obtained for multivariate analysis and machine learning without resampling, respectively. However, the resampling experiment showed no significant difference between them (0.52 vs. 0.53 ROC AUC). A difference between the predictors selected was observed, with the multivariable methodology being more stable. These results question the validity of predictors reported in previous studies and demonstrate their disparity. Furthermore, none of the techniques analyzed is superior to the other for these data.

Development and Evaluation of a Natural Language Processing Annotation Tool to Facilitate Phenotyping of Cognitive Status in Electronic Health Records: Diagnostic Study

BACKGROUND

Electronic health records (EHRs) with large sample sizes and rich information offer great potential for dementia research, but current methods of phenotyping cognitive status are not scalable.

OBJECTIVE

The aim of this study was to evaluate whether natural language processing (NLP)-powered semiautomated annotation can improve the speed and interrater reliability of chart reviews for phenotyping cognitive status.

METHODS

In this diagnostic study, we developed and evaluated a semiautomated NLP-powered annotation tool (NAT) to facilitate phenotyping of cognitive status. Clinical experts adjudicated the cognitive status of 627 patients at Mass General Brigham (MGB) health care, using NAT or traditional chart reviews. Patient charts contained EHR data from two data sets: (1) records from January 1, 2017, to December 31, 2018, for 100 Medicare beneficiaries from the MGB Accountable Care Organization and (2) records from 2 years prior to COVID-19 diagnosis to the date of COVID-19 diagnosis for 527 MGB patients. All EHR data from the relevant period were extracted; diagnosis codes, medications, and laboratory test values were processed and summarized; clinical notes were processed through an NLP pipeline; and a web tool was developed to present an integrated view of all data. Cognitive status was rated as cognitively normal, cognitively impaired, or undetermined. Assessment time and interrater agreement of NAT compared to manual chart reviews for cognitive status phenotyping was evaluated.

RESULTS

NAT adjudication provided higher interrater agreement (Cohen κ=0.89 vs κ=0.80) and significant speed up (time difference mean 1.4, SD 1.3 minutes; P<.001; ratio median 2.2, min-max 0.4-20) over manual chart reviews. There was moderate agreement with manual chart reviews (Cohen κ=0.67). In the cases that exhibited disagreement with manual chart reviews, NAT adjudication was able to produce assessments that had broader clinical consensus due to its integrated view of highlighted relevant information and semiautomated NLP features.

CONCLUSIONS

NAT adjudication improves the speed and interrater reliability for phenotyping cognitive status compared to manual chart reviews. This study underscores the potential of an NLP-based clinically adjudicated method to build large-scale dementia research cohorts from EHRs.

Application of Machine Learning Models to Predict Recurrence After Surgical Resection of Nonmetastatic Renal Cell Carcinoma

BACKGROUND

Predictive tools can be useful for adapting surveillance or including patients in adjuvant trials after surgical resection of nonmetastatic renal cell carcinoma (RCC). Current models have been built using traditional statistical modelling and prespecified variables, which limits their performance.

OBJECTIVE

To investigate the performance of machine learning (ML) framework to predict recurrence after RCC surgery and compare them with current validated models.

DESIGN, SETTING, AND PARTICIPANTS

In this observational study, we derived and tested several ML-based models (Random Survival Forests [RSF], Survival Support Vector Machines [S-SVM], and Extreme Gradient Boosting [XG boost]) to predict recurrence of patients who underwent radical or partial nephrectomy for a nonmetastatic RCC, between 2013 and 2020, at 21 French medical centres.

OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS

The primary end point was disease-free survival. Model discrimination was assessed using the concordance index (c-index), and calibration was assessed using the Brier score. ML models were compared with four conventional prognostic models, using decision curve analysis (DCA).

RESULTS AND LIMITATIONS

A total of 4067 patients were included in this study (3253 in the development cohort and 814 in the validation cohort). Most tumours (69%) were clear cell RCC, 40% were of high grade (nuclear International Society of Urological Pathology grade 3 or 4), and 24% had necrosis. Of the patients, 4% had nodal involvement. After a median follow-up of 57 mo (interquartile range 29-76), 523 (13%) patients recurred. ML models obtained higher c-index values than conventional models. The RSF yielded the highest c-index values (0.794), followed by S-SVM (c-index 0.784) and XG boost (c-index 0.782). In addition, all models showed good calibration with low integrated Brier scores (all integrated brier scores <0.1). However, we found calibration drift over time for all models, albeit with a smaller magnitude for ML models. Finally, DCA showed an incremental net benefit from all ML models compared with conventional models currently used in practice.

CONCLUSIONS

Applying ML approaches to predict recurrence following surgical resection of RCC resulted in better prediction than that of current validated models available in clinical practice. However, there is still room for improvement, which may come from the integration of novel biological and/or imaging biomarkers.

PATIENT SUMMARY

We found that artificial intelligence algorithms could better predict the risk of recurrence after surgery for a localised kidney cancer. These algorithms may help better select patients who will benefit from medical treatment after surgery.

Critical appraisal of artificial intelligence-based prediction models for cardiovascular disease

The medical field has seen a rapid increase in the development of artificial intelligence (AI)-based prediction models. With the introduction of such AI-based prediction model tools and software in cardiovascular patient care, the cardiovascular researcher and healthcare professional are challenged to understand the opportunities as well as the limitations of the AI-based predictions. In this article, we present 12 critical questions for cardiovascular health professionals to ask when confronted with an AI-based prediction model. We aim to support medical professionals to distinguish the AI-based prediction models that can add value to patient care from the AI that does not.