Prognostication and Risk Factors for Cystic Fibrosis via Automated Machine Learning

Clinical performance of automated machine learning: A systematic review

Introduction

Automated machine learning (autoML) removes technical and technological barriers to building artificial intelligence models. We aimed to summarise the clinical applications of autoML, assess the capabilities of utilised platforms, evaluate the quality of the evidence trialling autoML, and gauge the performance of autoML platforms relative to conventionally developed models, as well as each other.

Method

This review adhered to a prospectively registered protocol (PROSPERO identifier CRD42022344427). The Cochrane Library, Embase, MEDLINE and Scopus were searched from inception to 11 July 2022. Two researchers screened abstracts and full texts, extracted data and conducted quality assessment. Disagreement was resolved through discussion and if required, arbitration by a third researcher.

Results

There were 26 distinct autoML platforms featured in 82 studies. Brain and lung disease were the most common fields of study of 22 specialties. AutoML exhibited variable performance: area under the receiver operator characteristic curve (AUCROC) 0.35-1.00, F1-score 0.16-0.99, area under the precision-recall curve (AUPRC) 0.51-1.00. AutoML exhibited the highest AUCROC in 75.6% trials; the highest F1-score in 42.3% trials; and the highest AUPRC in 83.3% trials. In autoML platform comparisons, AutoPrognosis and Amazon Rekognition performed strongest with unstructured and structured data, respectively. Quality of reporting was poor, with a median DECIDE-AI score of 14 of 27.

Conclusion

A myriad of autoML platforms have been applied in a variety of clinical contexts. The performance of autoML compares well to bespoke computational and clinical benchmarks. Further work is required to improve the quality of validation studies. AutoML may facilitate a transition to data-centric development, and integration with large language models may enable AI to build itself to fulfil user-defined goals.

Predicting non-improvement of symptoms in daily mental healthcare practice using routinely collected patient-level data: a machine learning approach

Objectives

Anxiety and mood disorders greatly affect the quality of life for individuals worldwide. A substantial proportion of patients do not sufficiently improve during evidence-based treatments in mental healthcare. It remains challenging to predict which patients will or will not benefit. Moreover, the limited research available on predictors of treatment outcomes comes from efficacy RCTs with strict selection criteria which may limit generalizability to a real-world context. The current study evaluates the performance of different machine learning (ML) models in predicting non-improvement in an observational sample of patients treated in routine specialized mental healthcare.

Methods

In the current longitudinal exploratory prediction study diagnosis-related, sociodemographic, clinical and routinely collected patient-reported quantitative outcome measures were acquired during treatment as usual of 755 patients with a primary anxiety, depressive, obsessive compulsive or trauma-related disorder in a specialized outpatient mental healthcare center. ML algorithms were trained to predict non-response (< 0.5 standard deviation improvement) in symptomatic distress 6 months after baseline. Different models were trained, including models with and without early change scores in psychopathology and well-being and models with a trimmed set of predictor variables. Performance of trained models was evaluated in a hold-out sample (30%) as a proxy for unseen data.

Results

ML models without early change scores performed poorly in predicting six-month non-response in the hold-out sample with Area Under the Curves (AUCs) < 0.63. Including early change scores slightly improved the models' performance (AUC range: 0.68-0.73). Computationally-intensive ML models did not significantly outperform logistic regression (AUC: 0.69). Reduced prediction models performed similar to the full prediction models in both the models without (AUC: 0.58-0.62 vs. 0.58-0.63) and models with early change scores (AUC: 0.69-0.73 vs. 0.68-0.71). Across different ML algorithms, early change scores in psychopathology and well-being consistently emerged as important predictors for non-improvement.

Conclusion

Accurately predicting treatment outcomes in a mental healthcare context remains challenging. While advanced ML algorithms offer flexibility, they showed limited additional value compared to traditional logistic regression in this study. The current study confirmed the importance of taking early change scores in both psychopathology and well-being into account for predicting longer-term outcomes in symptomatic distress.

Enhancing Kidney Transplant Care through the Integration of Chatbot

Kidney transplantation is a critical treatment option for end-stage kidney disease patients, offering improved quality of life and increased survival rates. However, the complexities of kidney transplant care necessitate continuous advancements in decision making, patient communication, and operational efficiency. This article explores the potential integration of a sophisticated chatbot, an AI-powered conversational agent, to enhance kidney transplant practice and potentially improve patient outcomes. Chatbots and generative AI have shown promising applications in various domains, including healthcare, by simulating human-like interactions and generating contextually appropriate responses. Noteworthy AI models like ChatGPT by OpenAI, BingChat by Microsoft, and Bard AI by Google exhibit significant potential in supporting evidence-based research and healthcare decision making. The integration of chatbots in kidney transplant care may offer transformative possibilities. As a clinical decision support tool, it could provide healthcare professionals with real-time access to medical literature and guidelines, potentially enabling informed decision making and improved knowledge dissemination. Additionally, the chatbot has the potential to facilitate patient education by offering personalized and understandable information, addressing queries, and providing guidance on post-transplant care. Furthermore, under clinician or transplant pharmacist supervision, it has the potential to support post-transplant care and medication management by analyzing patient data, which may lead to tailored recommendations on dosages, monitoring schedules, and potential drug interactions. However, to fully ascertain its effectiveness and safety in these roles, further studies and validation are required. Its integration with existing clinical decision support systems may enhance risk stratification and treatment planning, contributing to more informed and efficient decision making in kidney transplant care. Given the importance of ethical considerations and bias mitigation in AI integration, future studies may evaluate long-term patient outcomes, cost-effectiveness, user experience, and the generalizability of chatbot recommendations. By addressing these factors and potentially leveraging AI capabilities, the integration of chatbots in kidney transplant care holds promise for potentially improving patient outcomes, enhancing decision making, and fostering the equitable and responsible use of AI in healthcare.

AutoPrognosis 2.0: Democratizing diagnostic and prognostic modeling in healthcare with automated machine learning

Diagnostic and prognostic models are increasingly important in medicine and inform many clinical decisions. Recently, machine learning approaches have shown improvement over conventional modeling techniques by better capturing complex interactions between patient covariates in a data-driven manner. However, the use of machine learning introduces technical and practical challenges that have thus far restricted widespread adoption of such techniques in clinical settings. To address these challenges and empower healthcare professionals, we present an open-source machine learning framework, AutoPrognosis 2.0, to facilitate the development of diagnostic and prognostic models. AutoPrognosis leverages state-of-the-art advances in automated machine learning to develop optimized machine learning pipelines, incorporates model explainability tools, and enables deployment of clinical demonstrators, without requiring significant technical expertise. To demonstrate AutoPrognosis 2.0, we provide an illustrative application where we construct a prognostic risk score for diabetes using the UK Biobank, a prospective study of 502,467 individuals. The models produced by our automated framework achieve greater discrimination for diabetes than expert clinical risk scores. We have implemented our risk score as a web-based decision support tool, which can be publicly accessed by patients and clinicians. By open-sourcing our framework as a tool for the community, we aim to provide clinicians and other medical practitioners with an accessible resource to develop new risk scores, personalized diagnostics, and prognostics using machine learning techniques. Software: https://github.com/vanderschaarlab/AutoPrognosis.

Virtual monitoring in CF - the importance of continuous monitoring in a multi-organ chronic condition

Cystic Fibrosis (CF) is a chronic life-limiting condition that affects multiple organs within the body. Patients must adhere to strict medication regimens, physiotherapy, diet, and attend regular clinic appointments to manage their condition effectively. This necessary but burdensome requirement has prompted investigations into how different digital health technologies can enhance current care by providing the opportunity to virtually monitor patients. This review explores how virtual monitoring has been harnessed for assessment or performance of physiotherapy/exercise, diet/nutrition, symptom monitoring, medication adherence, and wellbeing/mental-health in people with CF. This review will also briefly discuss the potential future of CF virtual monitoring and some common barriers to its current adoption and implementation within CF. Due to the multifaceted nature of CF, it is anticipated that this review will be relevant to not only the CF community, but also those investigating and developing digital health solutions for the management of other chronic diseases.

Single nucleotide variants in Pseudomonas aeruginosa populations from sputum correlate with baseline lung function and predict disease progression in individuals with cystic fibrosis

The severity and progression of lung disease are highly variable across individuals with cystic fibrosis (CF) and are imperfectly predicted by mutations in the human gene CFTR, lung microbiome variation or other clinical factors. The opportunistic pathogen Pseudomonas aeruginosa (Pa) dominates airway infections in most CF adults. Here we hypothesized that within-host genetic variation of Pa populations would be associated with lung disease severity. To quantify Pa genetic variation within CF sputum samples, we used deep amplicon sequencing (AmpliSeq) of 209 Pa genes previously associated with pathogenesis or adaptation to the CF lung. We trained machine learning models using Pa single nucleotide variants (SNVs), microbiome diversity data and clinical factors to classify lung disease severity at the time of sputum sampling, and to predict lung function decline after 5 years in a cohort of 54 adult CF patients with chronic Pa infection. Models using Pa SNVs alone classified lung disease severity with good sensitivity and specificity (area under the receiver operating characteristic curve: AUROC=0.87). Models were less predictive of lung function decline after 5 years (AUROC=0.74) but still significantly better than random. The addition of clinical data, but not sputum microbiome diversity data, yielded only modest improvements in classifying baseline lung function (AUROC=0.92) and predicting lung function decline (AUROC=0.79), suggesting that Pa AmpliSeq data account for most of the predictive value. Our work provides a proof of principle that Pa genetic variation in sputum tracks lung disease severity, moderately predicts lung function decline and could serve as a disease biomarker among CF patients with chronic Pa infections.

Methods of analysis for survival outcomes with time-updated mediators, with application to longitudinal disease registry data

Mediation analysis is a useful tool to illuminate the mechanisms through which an exposure affects an outcome but statistical challenges exist with time-to-event outcomes and longitudinal observational data. Natural direct and indirect effects cannot be identified when there are exposure-induced confounders of the mediator-outcome relationship. Previous measurements of a repeatedly-measured mediator may themselves confound the relationship between the mediator and the outcome. To overcome these obstacles, two recent methods have been proposed, one based on path-specific effects and one based on an additive hazards model and the concept of exposure splitting. We investigate these techniques, focusing on their application to observational datasets. We apply both methods to an analysis of the UK Cystic Fibrosis Registry dataset to identify how much of the relationship between onset of cystic fibrosis-related diabetes and subsequent survival acts through pulmonary function. Statistical properties of the methods are investigated using simulation. Both methods produce unbiased estimates of indirect and direct effects in scenarios consistent with their stated assumptions but, if the data are measured infrequently, estimates may be biased. Findings are used to highlight considerations in the interpretation of the observational data analysis.

Development and Internal Validation of a Prognostic Model of the Probability of Death or Lung Transplantation Within 2 Years for Patients With Cystic Fibrosis and FEV1 ≤ 50% Predicted

BACKGROUND

Improved methods are needed to risk-stratify patients with cystic fibrosis (CF) and reduced FEV1.

RESEARCH QUESTIONS

What are the predictors of death or lung transplantation (LTx) within 2 years among patients with CF whose FEV1 ≤ 50% predicted? Do these markers similarly predict outcomes among G551D patients taking ivacaftor since 2012?

STUDY DESIGN AND METHODS

Patients with CF, age ≥ 6 years with FEV1 ≤ 50% predicted as of December 31, 2014, were identified in a data set that merged Cystic Fibrosis Foundation and United Network for Organ Sharing (UNOS) registries. The least absolute shrinkage and selection operator (LASSO) method was applied to a randomly selected training set to select important prognostic variables. Accuracy and association of the model with death or LTx with 2 years (2-year death or LTx) were validated via logistic regression on an independent test set. Sensitivity analyses explored predictors for patients with UNOS data.

RESULTS

FEV1 percent predicted (OR, 1.51 for 5% decrease; 95% CI, 1.27-1.81), number of pulmonary exacerbations treated with IV antibiotics (OR, 1.35; 95% CI, 1.11-1.65), and continuous or nocturnal oxygen (OR, 3.71; 95% CI, 1.81-7.59) were significantly associated with 2-year death or LTx. Our model predicted outcomes with greater sensitivity (ratio of sensitivity, 1.26; 95% CI, 1.02-1.54), ratio of positive predictive value (1.25; 95% CI, 1.05-1.51), and ratio of negative predictive value (1.04; 95% CI, 1.01-1.07) than FEV1 < 30% predicted. Among those taking ivacaftor in 2014, only FEV1 remained associated with 2-year death or LTx. For patients with UNOS data, LASSO identified additional covariates of interest, including noninvasive ventilation use, low hemoglobin, pulmonary arterial systolic pressure, supplemental oxygen, mechanical ventilation, FEV1 percent predicted, and cardiac index.

INTERPRETATION

Among individuals with CF and FEV1 ≤ 50% predicted, FEV1 percent predicted, oxygen therapy, and number of pulmonary exacerbations predicted 2-year death or LTx. Although limited by small sample size, only FEV1 remained predictive in patients receiving highly effective modulator therapy. Additional physiologic variables could improve prognostication in CF.

Machine learning-driven identification of novel patient factors for prediction of major complications after posterior cervical spinal fusion

PURPOSE

Posterior cervical fusion is associated with increased rates of complications and readmission when compared to anterior fusion. Machine learning (ML) models for risk stratification of patients undergoing posterior cervical fusion remain limited. We aim to develop a novel ensemble ML algorithm for prediction of major perioperative complications and readmission after posterior cervical fusion and identify factors important to model performance.

METHODS

This is a retrospective cohort study of adults who underwent posterior cervical fusion at non-federal California hospitals between 2015 and 2017. The primary outcome was readmission or major complication. We developed an ensemble model predicting complication risk using an automated ML framework. We compared performance with standard ML models and logistic regression (LR), ranking contribution of included variables to model performance.

RESULTS

Of the included 6822 patients, 18.8% suffered a major complication or readmission. The ensemble model demonstrated slightly superior predictive performance compared to LR and standard ML models. The most important features to performance include sex, malignancy, pneumonia, stroke, and teaching hospital status. Seven of the ten most important features for the ensemble model were markedly less important for LR.

CONCLUSION

We report an ensemble ML model for prediction of major complications and readmission after posterior cervical fusion with a modest risk prediction advantage compared to LR and benchmark ML models. Notably, the features most important to the ensemble are markedly different from those for LR, suggesting that advanced ML methods may identify novel prognostic factors for adverse outcomes after posterior cervical fusion.

Automatically Explaining Machine Learning Predictions on Severe Chronic Obstructive Pulmonary Disease Exacerbations: Retrospective Cohort Study

Background

Chronic obstructive pulmonary disease (COPD) is a major cause of death and places a heavy burden on health care. To optimize the allocation of precious preventive care management resources and improve the outcomes for high-risk patients with COPD, we recently built the most accurate model to date to predict severe COPD exacerbations, which need inpatient stays or emergency department visits, in the following 12 months. Our model is a machine learning model. As is the case with most machine learning models, our model does not explain its predictions, forming a barrier for clinical use. Previously, we designed a method to automatically provide rule-type explanations for machine learning predictions and suggest tailored interventions with no loss of model performance. This method has been tested before for asthma outcome prediction but not for COPD outcome prediction.

Objective

This study aims to assess the generalizability of our automatic explanation method for predicting severe COPD exacerbations.

Methods

The patient cohort included all patients with COPD who visited the University of Washington Medicine facilities between 2011 and 2019. In a secondary analysis of 43,576 data instances, we used our formerly developed automatic explanation method to automatically explain our model’s predictions and suggest tailored interventions.

Results

Our method explained the predictions for 97.1% (100/103) of the patients with COPD whom our model correctly predicted to have severe COPD exacerbations in the following 12 months and the predictions for 73.6% (134/182) of the patients with COPD who had ≥1 severe COPD exacerbation in the following 12 months.

Conclusions

Our automatic explanation method worked well for predicting severe COPD exacerbations. After further improving our method, we hope to use it to facilitate future clinical use of our model.

International Registered Report Identifier (IRRID)

RR2-10.2196/13783

Comparative analysis of molecular fingerprints in prediction of drug combination effects

Application of machine and deep learning methods in drug discovery and cancer research has gained a considerable amount of attention in the past years. As the field grows, it becomes crucial to systematically evaluate the performance of novel computational solutions in relation to established techniques. To this end, we compare rule-based and data-driven molecular representations in prediction of drug combination sensitivity and drug synergy scores using standardized results of 14 high-throughput screening studies, comprising 64 200 unique combinations of 4153 molecules tested in 112 cancer cell lines. We evaluate the clustering performance of molecular representations and quantify their similarity by adapting the Centered Kernel Alignment metric. Our work demonstrates that to identify an optimal molecular representation type, it is necessary to supplement quantitative benchmark results with qualitative considerations, such as model interpretability and robustness, which may vary between and throughout preclinical drug development projects.

The promise of automated machine learning for the genetic analysis of complex traits

The genetic analysis of complex traits has been dominated by parametric statistical methods due to their theoretical properties, ease of use, computational efficiency, and intuitive interpretation. However, there are likely to be patterns arising from complex genetic architectures which are more easily detected and modeled using machine learning methods. Unfortunately, selecting the right machine learning algorithm and tuning its hyperparameters can be daunting for experts and non-experts alike. The goal of automated machine learning (AutoML) is to let a computer algorithm identify the right algorithms and hyperparameters thus taking the guesswork out of the optimization process. We review the promises and challenges of AutoML for the genetic analysis of complex traits and give an overview of several approaches and some example applications to omics data. It is our hope that this review will motivate studies to develop and evaluate novel AutoML methods and software in the genetics and genomics space. The promise of AutoML is to enable anyone, regardless of training or expertise, to apply machine learning as part of their genetic analysis strategy.

Recent applications of quantitative systems pharmacology and machine learning models across diseases

Quantitative systems pharmacology (QSP) is a quantitative and mechanistic platform describing the phenotypic interaction between drugs, biological networks, and disease conditions to predict optimal therapeutic response. In this meta-analysis study, we review the utility of the QSP platform in drug development and therapeutic strategies based on recent publications (2019-2021). We gathered recent original QSP models and described the diversity of their applications based on therapeutic areas, methodologies, software platforms, and functionalities. The collection and investigation of these publications can assist in providing a repository of recent QSP studies to facilitate the discovery and further reusability of QSP models. Our review shows that the largest number of QSP efforts in recent years is in Immuno-Oncology. We also addressed the benefits of integrative approaches in this field by presenting the applications of Machine Learning methods for drug discovery and QSP models. Based on this meta-analysis, we discuss the advantages and limitations of QSP models and propose fields where the QSP approach constitutes a valuable interface for more investigations to tackle complex diseases and improve drug development.

Towards Pharma 4.0 in clinical trials: A future-orientated perspective

Pharma 4.0, a technology ecosystem in drug development analogous to Industry 4.0 in healthcare, is transforming the traditional approach to drug discovery and development, aligning product quality with less time to market, and creating intelligent stakeholder networks through effective collaborations. The wide range of potential Pharma 4.0 networks have produced several conceptualizations, which have led to a lack of clarity and definition. The main emphasis of this paper is on the clinical trial stage of drug development in the Pharma 4.0 era. It highlights the merged computerized technologies that are currently used in clinical research, and proposes a framework for integrating Pharma 4.0 technologies. The impact of and barriers to employing the proposed framework are discussed, highlighting its potential and some future research applications.

Ranking Rule-Based Automatic Explanations for Machine Learning Predictions on Asthma Hospital Encounters in Patients With Asthma: Retrospective Cohort Study

Background

Asthma hospital encounters impose a heavy burden on the health care system. To improve preventive care and outcomes for patients with asthma, we recently developed a black-box machine learning model to predict whether a patient with asthma will have one or more asthma hospital encounters in the succeeding 12 months. Our model is more accurate than previous models. However, black-box machine learning models do not explain their predictions, which forms a barrier to widespread clinical adoption. To solve this issue, we previously developed a method to automatically provide rule-based explanations for the model’s predictions and to suggest tailored interventions without sacrificing model performance. For an average patient correctly predicted by our model to have future asthma hospital encounters, our explanation method generated over 5000 rule-based explanations, if any. However, the user of the automated explanation function, often a busy clinician, will want to quickly obtain the most useful information for a patient by viewing only the top few explanations. Therefore, a methodology is required to appropriately rank the explanations generated for a patient. However, this is currently an open problem.

Objective

The aim of this study is to develop a method to appropriately rank the rule-based explanations that our automated explanation method generates for a patient.

Methods

We developed a ranking method that struck a balance among multiple factors. Through a secondary analysis of 82,888 data instances of adults with asthma from the University of Washington Medicine between 2011 and 2018, we demonstrated our ranking method on the test case of predicting asthma hospital encounters in patients with asthma.

Results

For each patient predicted to have asthma hospital encounters in the succeeding 12 months, the top few explanations returned by our ranking method typically have high quality and low redundancy. Many top-ranked explanations provide useful insights on the various aspects of the patient’s situation, which cannot be easily obtained by viewing the patient’s data in the current electronic health record system.

Conclusions

The explanation ranking module is an essential component of the automated explanation function, and it addresses the interpretability issue that deters the widespread adoption of machine learning predictive models in clinical practice. In the next few years, we plan to test our explanation ranking method on predictive modeling problems addressing other diseases as well as on data from other health care systems.

International Registered Report Identifier (IRRID)

RR2-10.2196/5039

A Novel, Potentially Universal Machine Learning Algorithm to Predict Complications in Total Knee Arthroplasty

BACKGROUND

There remains a lack of accurate and validated outcome-prediction models in total knee arthroplasty (TKA). While machine learning (ML) is a powerful predictive tool, determining the proper algorithm to apply across diverse data sets is challenging. AutoPrognosis (AP) is a novel method that uses automated ML framework to incorporate the best performing stages of prognostic modeling into a single well-calibrated algorithm. We aimed to compare various ML methods to AP in predictive performance of complications after TKA.

METHODS

Thirty-eight preoperative patient demographics and clinical features from all primary TKAs performed at California-licensed hospitals between 2015 and 2017 were evaluated as predictors of major complications after TKA. Traditional logistic regression (LR), various other ML methods (XGBoost, Gradient Boosting, AdaBoost, and Random Forest), and AP were used for model building to determine discriminative power (area under receiver operating curve), calibration (Brier score), and feature importance.

RESULTS

Between 2015 and 2017, there were a total of 156,750 TKAs with 1109 (0.7%) total major complications. AP had the highest discriminative performance with area under receiver operating curve 0.679 compared with LR, XGBoost, Gradient Boosting, AdaBoost, and Random Forest (0.617, 0.601, 0.662, 0.657, and 0.545, respectively). AP (Brier score 0.007) had similar calibration as the other ML methods (0.006, 0.006, 0.022, 0.007, and 0.008, respectively). The variables that are most important for AP differ from those that are most important for LR.

CONCLUSION

Compared to conventional ML algorithms, AP has superior discriminative ability with similar calibration and suggests nonlinear relationships between variables in outcomes of TKA.

A Roadmap for Automating Lineage Tracing to Aid Automatically Explaining Machine Learning Predictions for Clinical Decision Support

Using machine learning predictive models for clinical decision support has great potential in improving patient outcomes and reducing health care costs. However, most machine learning models are black boxes that do not explain their predictions, thereby forming a barrier to clinical adoption. To overcome this barrier, an automated method was recently developed to provide rule-style explanations of any machine learning model’s predictions on tabular data and to suggest customized interventions. Each explanation delineates the association between a feature value pattern and an outcome value. Although the association and intervention information is useful, the user of the automated explaining function often requires more detailed information to better understand the patient’s situation and to aid in decision making. More specifically, consider a feature value in the explanation that is computed by an aggregation function on the raw data, such as the number of emergency department visits related to asthma that the patient had in the prior 12 months. The user often wants to rapidly drill through to see certain parts of the related raw data that produce the feature value. This task is frequently difficult and time-consuming because the few pieces of related raw data are submerged by many pieces of raw data of the patient that are unrelated to the feature value. To address this issue, this paper outlines an automated lineage tracing approach, which adds automated drill-through capability to the automated explaining function, and provides a roadmap for future research.

Development of a Novel, Potentially Universal Machine Learning Algorithm for Prediction of Complications After Total Hip Arthroplasty

BACKGROUND

As the prevalence of hip osteoarthritis increases, the number of total hip arthroplasty (THA) procedures performed is also projected to increase. Accurately risk-stratifying patients who undergo THA would be of great utility, given the significant cost and morbidity associated with developing perioperative complications. We aim to develop a novel machine learning (ML)-based ensemble algorithm for the prediction of major complications after THA, as well as compare its performance against standard benchmark ML methods.

METHODS

This is a retrospective cohort study of 89,986 adults who underwent primary THA at any California-licensed hospital between 2015 and 2017. The primary outcome was major complications (eg infection, venous thromboembolism, cardiac complication, pulmonary complication). We developed a model predicting complication risk using AutoPrognosis, an automated ML framework that configures the optimally performing ensemble of ML-based prognostic models. We compared our model with logistic regression and standard benchmark ML models, assessing discrimination and calibration.

RESULTS

There were 545 patients who had major complications (0.61%). Our novel algorithm was well-calibrated and improved risk prediction compared to logistic regression, as well as outperformed the other four standard benchmark ML algorithms. The variables most important for AutoPrognosis (eg malnutrition, dementia, cancer) differ from those that are most important for logistic regression (eg chronic atherosclerosis, renal failure, chronic obstructive pulmonary disease).

CONCLUSION

We report a novel ensemble ML algorithm for the prediction of major complications after THA. It demonstrates superior risk prediction compared to logistic regression and other standard ML benchmark algorithms. By providing accurate prognostic information, this algorithm may facilitate more informed preoperative shared decision-making.

Generalizability of an Automatic Explanation Method for Machine Learning Prediction Results on Asthma-Related Hospital Visits in Patients With Asthma: Quantitative Analysis

Background

Asthma exerts a substantial burden on patients and health care systems. To facilitate preventive care for asthma management and improve patient outcomes, we recently developed two machine learning models, one on Intermountain Healthcare data and the other on Kaiser Permanente Southern California (KPSC) data, to forecast asthma-related hospital visits, including emergency department visits and hospitalizations, in the succeeding 12 months among patients with asthma. As is typical for machine learning approaches, these two models do not explain their forecasting results. To address the interpretability issue of black-box models, we designed an automatic method to offer rule format explanations for the forecasting results of any machine learning model on imbalanced tabular data and to suggest customized interventions with no accuracy loss. Our method worked well for explaining the forecasting results of our Intermountain Healthcare model, but its generalizability to other health care systems remains unknown.

Objective

The objective of this study is to evaluate the generalizability of our automatic explanation method to KPSC for forecasting asthma-related hospital visits.

Methods

Through a secondary analysis of 987,506 data instances from 2012 to 2017 at KPSC, we used our method to explain the forecasting results of our KPSC model and to suggest customized interventions. The patient cohort covered a random sample of 70% of patients with asthma who had a KPSC health plan for any period between 2015 and 2018.

Results

Our method explained the forecasting results for 97.57% (2204/2259) of the patients with asthma who were correctly forecasted to undergo asthma-related hospital visits in the succeeding 12 months.

Conclusions

For forecasting asthma-related hospital visits, our automatic explanation method exhibited an acceptable generalizability to KPSC.

International Registered Report Identifier (IRRID)

RR2-10.2196/resprot.5039

Identification of variation in nutritional practice in neonatal units in England and association with clinical outcomes using agnostic machine learning

We used agnostic, unsupervised machine learning to cluster a large clinical database of information on infants admitted to neonatal units in England. Our aim was to obtain insights into nutritional practice, an area of central importance in newborn care, utilising the UK National Neonatal Research Database (NNRD). We performed clustering on time-series data of daily nutritional intakes for very preterm infants born at a gestational age less than 32 weeks (n = 45,679) over a six-year period. This revealed 46 nutritional clusters heterogeneous in size, showing common interpretable clinical practices alongside rarer approaches. Nutritional clusters with similar admission profiles revealed associations between nutritional practice, geographical location and outcomes. We show how nutritional subgroups may be regarded as distinct interventions and tested for associations with measurable outcomes. We illustrate the potential for identifying relationships between nutritional practice and outcomes with two examples, discharge weight and bronchopulmonary dysplasia (BPD). We identify the well-known effect of formula milk on greater discharge weight as well as support for the plausible, but insufficiently evidenced view that human milk is protective against BPD. Our framework highlights the potential of agnostic machine learning approaches to deliver clinical practice insights and generate hypotheses using routine data.

Assessing the validity and applicability of the French 3-year prognostic score in the UK cystic fibrosis population - a national cohort study

Models that predict outcomes, aid prognostication and inform the assessment of urgency for lung transplantation (LT) in CF are in demand. A prognostic score derived from the French adult CF registry to predict death or LT over 3-year follow-up was described in 2017 and validated using Canadian CF registry data. We assessed its performance in the UK CF population. The French prognostic score was applied to untransplanted adults with CF. The index year (2014) and outcomes (Death or LT) were evaluated to 2017. Receiver operator characteristics plots and area under curve (AUC) was computed. 4407 adults with CF met the inclusion criteria. After 3 years, 7.1% (P < 0.001) were dead or had received LT compared to the French (12.8%) and Canadian (9.4%) cohorts. The French score deemed 592 (26.2%) 'High-risk' - death/LT occurred in 189/592 (30.2%), less than previously reported in France and Canada (P < 0.0001). The discriminatory power of the French score was lower (AUC 0.830) than reported. Recalibration yielded only marginal improvement in model performance (AUC 0.833). The French prognostic score does not perform as well in the UK as reported elsewhere. Bespoke UK scores are needed to aid prognostication and inform LT decision-making.

Automatically Explaining Machine Learning Prediction Results on Asthma Hospital Visits in Patients With Asthma: Secondary Analysis

Background

Asthma is a major chronic disease that poses a heavy burden on health care. To facilitate the allocation of care management resources aimed at improving outcomes for high-risk patients with asthma, we recently built a machine learning model to predict asthma hospital visits in the subsequent year in patients with asthma. Our model is more accurate than previous models. However, like most machine learning models, it offers no explanation of its prediction results. This creates a barrier for use in care management, where interpretability is desired.

Objective

This study aims to develop a method to automatically explain the prediction results of the model and recommend tailored interventions without lowering the performance measures of the model.

Methods

Our data were imbalanced, with only a small portion of data instances linking to future asthma hospital visits. To handle imbalanced data, we extended our previous method of automatically offering rule-formed explanations for the prediction results of any machine learning model on tabular data without lowering the model’s performance measures. In a secondary analysis of the 334,564 data instances from Intermountain Healthcare between 2005 and 2018 used to form our model, we employed the extended method to automatically explain the prediction results of our model and recommend tailored interventions. The patient cohort consisted of all patients with asthma who received care at Intermountain Healthcare between 2005 and 2018, and resided in Utah or Idaho as recorded at the visit.

Results

Our method explained the prediction results for 89.7% (391/436) of the patients with asthma who, per our model’s correct prediction, were likely to incur asthma hospital visits in the subsequent year.

Conclusions

This study is the first to demonstrate the feasibility of automatically offering rule-formed explanations for the prediction results of any machine learning model on imbalanced tabular data without lowering the performance measures of the model. After further improvement, our asthma outcome prediction model coupled with the automatic explanation function could be used by clinicians to guide the allocation of limited asthma care management resources and the identification of appropriate interventions.

Gender and ethnic differences in publication of BMJ letters to the editor: an observational study using machine learning

OBJECTIVES

To analyse the relationship between first author's gender and ethnicity (estimated from first name and surname), and chance of publication of rapid responses in the British Medical Journal (BMJ). To analyse whether other features of the rapid response account for any gender or ethnic differences, including the presence of multiple authors, declaration of conflicts of interests, the presence of Twitter handle, word count, reading ease, spelling and grammatical mistakes, and the presence of references.

DESIGN

A retrospective observational study.

SETTING

Website of the BMJ (BMJ.com).

PARTICIPANTS

Publicly available rapid responses submitted to BMJ.com between 1998 and 2018.

MAIN OUTCOME MEASURES

Publication of a rapid response as a letter to the editor in the BMJ.

RESULTS

We analysed 113 265 rapid responses, of which 8415 were published as letters to the editor (7.4%). Statistically significant univariate correlations were found between odds of publication and first author estimated gender and ethnicity, multiple authors, declaration of conflicts of interest, the presence of Twitter handle, word count, reading ease, spelling and grammatical mistakes, and the presence of references. Multivariate analysis showed that first author estimated gender and ethnicity predicted publication after taking into account the other factors. Compared to white authors, black authors were 26% less likely to be published (OR: 0.74, CI: 0.57-0.96), Asian and Pacific Islander authors were 46% less likely to be published (OR: 0.54, CI: 0.49-0.59) and Hispanic authors were 49% less likely to be published (OR: 0.51, CI: 0.41-0.64). Female authors were 10% less likely to be published (OR: 0.90, CI: 0.85-0.96) than male authors.

CONCLUSION

Ethnic and gender differences in rapid response publication remained after accounting for a broad range of features, themselves all predictive of publication. This suggests that the reasons for the differences of these groups lies elsewhere.

CPAS: the UK's national machine learning-based hospital capacity planning system for COVID-19

The coronavirus disease 2019 (COVID-19) global pandemic poses the threat of overwhelming healthcare systems with unprecedented demands for intensive care resources. Managing these demands cannot be effectively conducted without a nationwide collective effort that relies on data to forecast hospital demands on the national, regional, hospital and individual levels. To this end, we developed the COVID-19 Capacity Planning and Analysis System (CPAS)—a machine learning-based system for hospital resource planning that we have successfully deployed at individual hospitals and across regions in the UK in coordination with NHS Digital. In this paper, we discuss the main challenges of deploying a machine learning-based decision support system at national scale, and explain how CPAS addresses these challenges by (1) defining the appropriate learning problem, (2) combining bottom-up and top-down analytical approaches, (3) using state-of-the-art machine learning algorithms, (4) integrating heterogeneous data sources, and (5) presenting the result with an interactive and transparent interface. CPAS is one of the first machine learning-based systems to be deployed in hospitals on a national scale to address the COVID-19 pandemic—we conclude the paper with a summary of the lessons learned from this experience.

From Real-World Patient Data to Individualized Treatment Effects Using Machine Learning: Current and Future Methods to Address Underlying Challenges

Clinical decision making needs to be supported by evidence that treatments are beneficial to individual patients. Although randomized control trials (RCTs) are the gold standard for testing and introducing new drugs, due to the focus on specific questions with respect to establishing efficacy and safety vs. standard treatment, they do not provide a full characterization of the heterogeneity in the final intended treatment population. Conversely, real-world observational data, such as electronic health records (EHRs), contain large amounts of clinical information about heterogeneous patients and their response to treatments. In this paper, we introduce the main opportunities and challenges in using observational data for training machine learning methods to estimate individualized treatment effects and make treatment recommendations. We describe the modeling choices of the state-of-the-art machine learning methods for causal inference, developed for estimating treatment effects both in the cross-section and longitudinal settings. Additionally, we highlight future research directions that could lead to achieving the full potential of leveraging EHRs and machine learning for making individualized treatment recommendations. We also discuss how experimental data from RCTs and Pharmacometric and Quantitative Systems Pharmacology approaches can be used to not only improve machine learning methods, but also provide ways for validating them. These future research directions will require us to collaborate across the scientific disciplines to incorporate models based on RCTs and known disease processes, physiology, and pharmacology into these machine learning models based on EHRs to fully optimize the opportunity these data present.

Development and external validation of 1- and 2-year mortality prediction models in cystic fibrosis

INTRODUCTION

We aimed to develop a clinical tool for predicting 1- and 2-year risk of death for patients with cystic fibrosis (CF). The model considers patients' overall health status as well as risk of intermittent shock events in calculating the risk of death.

METHODS

Canadian CF Registry data from 1982 to 2015 were used to develop a predictive risk model using threshold regression. A 2-year risk of death estimated conditional probability of surviving the second year given survival for the first year. UK CF Registry data from 2007 to 2013 were used to externally validate the model.

RESULTS

The combined effect of CF chronic health status and CF intermittent shock risk provided a simple clinical scoring tool for assessing 1-year and 2-year risk of death for an individual CF patient. At a threshold risk of death of ≥20%, the 1-year model had a sensitivity of 74% and specificity of 96%. The area under the receiver operating curve (AUC) for the 2-year mortality model was significantly greater than the AUC for a model that predicted survival based on forced expiratory volume in 1 s <30% predicted (AUC 0.95 versus 0.68 respectively, p<0.001). The Canadian-derived model validated well with the UK data and correctly identified 79% of deaths and 95% of survivors in a single year in the UK.

CONCLUSIONS

The prediction models provide an accurate risk of death over a 1- and 2-year time horizon. The models performed equally well when validated in an independent UK CF population.

Diagnostic value of spirometry vs impulse oscillometry: A comparative study in children with sickle cell disease

BACKGROUND

Spirometry is conventionally used to diagnose airway diseases in children with sickle cell disease (C-SCD). However, spirometry is difficult for younger children to perform, is effort dependent, and it provides limited information on respiratory mechanics. Impulse oscillometry (IOS) is an effort-independent pulmonary function test (PFT), which measures total airway resistance (R5Hz) and reactance (AX). IOS could be advantageous without certain limitations of spirometry.

AIM

To compare the accuracy of IOS vs spirometry in making the diagnosis of asthma and assessing age-related pulmonary changes in C-SCD.

STUDY DESIGN

Retrospective chart review.

SUBJECT SELECTION

Fifty-six C-SCD and thirty-six controls (asthmatics without SCD) followed at Penn State with PFTs obtained during the initial pulmonary evaluation.

METHODOLOGY

We grouped C-SCD into asthmatics and non-asthmatics based on pre-referral diagnosis and compared PFTs between two groups. Receiver operating characteristic (ROC) curve analyses and machine learning tools (XGBoost and artificial neural network) were used to rank the spirometry and IOS measures based on their ability to predict a diagnosis of asthma. Robust linear regression was used to analyze association among height/age with various PFT measures.

RESULTS

Both ROC and XGBoost indicated that FEF_25-75 %, forced expiratory volume in 1 second (FEV1)/forced vital capacity, and R5Hz(%) were the top three predictors for asthma diagnosis. R5Hz(%) and AX had superior bronchodilator response (BDR) than FEV1. IOS parameters had significant association with height/age in C-SCD (possibly due to the stiff lungs) but not in controls.

CONCLUSION

IOS had advantages over spirometry in C-SCD because it is feasible in early childhood, provides insights into the pulmonary mechanics, and is more sensitive to detect BDR.

Trial Refresh: A Case for an Adaptive Platform Trial for Pulmonary Exacerbations of Cystic Fibrosis

Cystic fibrosis is a genetic disease typically characterized by progressive lung damage and premature mortality. Pulmonary exacerbations, or flare-ups of the lung disease, often require hospitalization for intensive treatment. Approximately 25% of patients with cystic fibrosis do not recover their baseline lung function after pulmonary exacerbations. There is a relative paucity of evidence to inform treatment strategies for exacerbations. Compounding this lack of evidence, there are a large number of treatment options already as well as becoming available. This results in significant variability between medication regimens prescribed by different physicians, treatment centers and regions with potentially adverse impact to patients. The conventional strategy is to undertake essential randomized clinical trials to inform treatment decisions and improve outcomes for patients with exacerbations. However, over the past several decades, clinical trials have generally failed to provide information critical to improved treatment and management of exacerbations. Bayesian adaptive platform trials hold the promise of addressing clinical uncertainties and informing treatment. Using modeling and response adaptive randomization, they allow for the evaluation of multiple treatments across different management domains, and progressive improvement in patient outcomes throughout the course of the trial. Bayesian adaptive platform trials require substantial amounts of preparation. Basic preparation includes extensive stakeholder involvement including elicitation of consumer preferences and clinician understanding of the research topic, defining the research questions, determining the best outcome measures, delineating study sub-groups, in depth statistical modeling, designing end-to-end digital solutions seamlessly supporting clinicians, researchers and patients, constructing randomisation algorithms and importantly, defining pre-determined intra-study end-points. This review will discuss the motivation and necessary steps required to embark on a Bayesian adaptive platform trial to optimize medication regimens for the treatment of pulmonary exacerbations of cystic fibrosis.