Dealing with missing predictor values when applying clinical prediction models

Addressing the implementation challenge of risk prediction model due to missing risk factors: The submodel approximation approach

Clinical prediction models have been widely acknowledged as informative tools providing evidence-based support for clinical decision making. However, prediction models are often underused in clinical practice due to many reasons including missing information upon real-time risk calculation in electronic health records (EHR) system. Existing literature to address this challenge focuses on statistical comparison of various approaches while overlooking the feasibility of their implementation in EHR. In this article, we propose a novel and feasible submodel approach to address this challenge for prediction models developed using the model approximation (also termed "preconditioning") method. The proposed submodel coefficients are equivalent to the corresponding original prediction model coefficients plus a correction factor. Comprehensive simulations were conducted to assess the performance of the proposed method and compared with the existing "one-step-sweep" approach as well as the imputation approach. In general, the simulation results show the preconditioning-based submodel approach is robust to various heterogeneity scenarios and is comparable to the imputation-based approach, while the "one-step-sweep" approach is less robust under certain heterogeneity scenarios. The proposed method was applied to facilitate real-time implementation of a prediction model to identify emergency department patients with acute heart failure who can be safely discharged home.

Adaptive predictor-set linear model: An imputation-free method for linear regression prediction on data sets with missing values

Linear regression (LR) is vastly used in data analysis for continuous outcomes in biomedicine and epidemiology. Despite its popularity, LR is incompatible with missing data, which frequently occur in health sciences. For parameter estimation, this shortcoming is usually resolved by complete-case analysis or imputation. Both work-arounds, however, are inadequate for prediction, since they either fail to predict on incomplete records or ignore missingness-induced reduction in prediction accuracy and rely on (unrealistic) assumptions about the missing mechanism. Here, we derive adaptive predictor-set linear model (aps-lm), capable of making predictions for incomplete data without the need for imputation. It is derived by using a predictor-selection operation, the Moore-Penrose pseudoinverse, and the reduced QR decomposition. aps-lm is an LR generalization that inherently handles missing values. It is applied on a reference data set, where complete predictors and outcome are available, and yields a set of privacy-preserving parameters. In a second stage, these are shared for making predictions of the outcome on external data sets with missing entries for predictors without imputation. Moreover, aps-lm computes prediction errors that account for the pattern of missing values even under extreme missingness. We benchmark aps-lm in a simulation study. aps-lm showed greater prediction accuracy and reduced bias compared to popular imputation strategies under a wide range of scenarios including variation of sample size, goodness of fit, missing value type, and covariance structure. Finally, as a proof-of-principle, we apply aps-lm in the context of epigenetic aging clocks, linear models that predict a person's biological age from epigenetic data with promising clinical applications.

Prediction model of deep vein thrombosis risk after lower extremity orthopedic surgery

Purpose

This investigation was conceived to engineer and appraise a pioneering clinical nomogram, crafted to bridge the extant chasm in literature regarding the postoperative risk stratification for deep vein thrombosis (DVT) in the aftermath of lower extremity orthopedic procedures. This novel tool offers a sophisticated and discerning algorithm for risk prediction, heretofore unmet by existing methodologies.

Methods

In this retrospective observational study, clinical records of hospitalized patients who underwent lower extremity orthopedic surgery were collected at the Wuxi TCM Hospital Affiliated to the Nanjing University of Chinese Medicine between Jan 2017 and Oct 2019. The univariate and multivariate analysis with the backward stepwise method was applied to select features for the predictive nomogram. The performance of the nomogram was evaluated with respect to its discriminant capability, calibration ability, and clinical utility.

Result

A total of 5773 in-hospital patients were eligible for the study, with the incidence of deep vein thrombosis being approximately 1 % in this population. Among 31 variables included, 5 of them were identified to be the predictive features in the nomogram, including age, mean corpuscular hemoglobin concentration (MCHC), D-dimer, platelet distribution width (PDW), and thrombin time (TT). The area under the receiver operating characteristic (ROC) curve in the training and validation cohort was 85.9 % (95%CI: 79.96 %-90.04 %) and 85.7 % (95%CI: 78.96 %-90.69 %), respectively. Both the calibration curves and decision curve analysis demonstrated the overall satisfactory performance of the model.

Conclusion

Our groundbreaking nomogram is distinguished by its unparalleled accuracy in discriminative and calibrating functions, complemented by its tangible clinical applicability. This innovative instrument is set to empower clinicians with a robust framework for the accurate forecasting of postoperative DVT, thus facilitating the crafting of bespoke and prompt therapeutic strategies, aligning with the rigorous standards upheld by the most esteemed biomedical journals.

A real-time prognostic model for venous thromboembolic events among hospitalized adults

Background

Hospital-acquired venous thromboembolism (HA-VTE) is a leading cause of morbidity and mortality among hospitalized adults. Guidelines recommend use of a risk-prediction model to estimate HA-VTE risk for individual patients. Extant models do not perform well for broad patient populations and are not conducive to automation in clinical practice.

Objectives

To develop an automated, real-time prognostic model for venous thromboembolism during hospitalization among all adult inpatients using readily available data from the electronic health record.

Methods

The derivation cohort included inpatient hospitalizations ("encounters") for patients ≥16 years old at Vanderbilt University Medical Center between 2018 and 2020 (n = 132,330). HA-VTE events were identified using International Classification of Diseases, 10th Revision, codes. The prognostic model was developed using least absolute shrinkage and selection operator regression. Temporal external validation was performed in a validation cohort of encounters between 2021 and 2022 (n = 62,546). Prediction performance was assessed by discrimination accuracy (C statistic) and calibration (integrated calibration index).

Results

There were 1187 HA-VTEs in the derivation cohort (9.0 per 1000 encounters) and 864 in the validation cohort (13.8 per 1000 encounters). The prognostic model included 25 variables, with placement of a central line among the most important predictors. Prediction performance of the model was excellent (C statistic, 0.891; 95% CI, 0.882-0.900; integrated calibration index, 0.001). The model performed similarly well across subgroups of patients defined by age, sex, race, and type of admission.

Conclusion

This fully automated prognostic model uses readily available data from the electronic health record, exhibits superior prediction performance compared with existing models, and generates granular risk stratification in the form of a predicted probability of HA-VTE for each patient.

Prediction models for post-discharge mortality among under-five children with suspected sepsis in Uganda: A multicohort analysis

In many low-income countries, over five percent of hospitalized children die following hospital discharge. The lack of available tools to identify those at risk of post-discharge mortality has limited the ability to make progress towards improving outcomes. We aimed to develop algorithms designed to predict post-discharge mortality among children admitted with suspected sepsis. Four prospective cohort studies of children in two age groups (0-6 and 6-60 months) were conducted between 2012-2021 in six Ugandan hospitals. Prediction models were derived for six-months post-discharge mortality, based on candidate predictors collected at admission, each with a maximum of eight variables, and internally validated using 10-fold cross-validation. 8,810 children were enrolled: 470 (5.3%) died in hospital; 257 (7.7%) and 233 (4.8%) post-discharge deaths occurred in the 0-6-month and 6-60-month age groups, respectively. The primary models had an area under the receiver operating characteristic curve (AUROC) of 0.77 (95%CI 0.74-0.80) for 0-6-month-olds and 0.75 (95%CI 0.72-0.79) for 6-60-month-olds; mean AUROCs among the 10 cross-validation folds were 0.75 and 0.73, respectively. Calibration across risk strata was good: Brier scores were 0.07 and 0.04, respectively. The most important variables included anthropometry and oxygen saturation. Additional variables included: illness duration, jaundice-age interaction, and a bulging fontanelle among 0-6-month-olds; and prior admissions, coma score, temperature, age-respiratory rate interaction, and HIV status among 6-60-month-olds. Simple prediction models at admission with suspected sepsis can identify children at risk of post-discharge mortality. Further external validation is recommended for different contexts. Models can be digitally integrated into existing processes to improve peri-discharge care as children transition from the hospital to the community.

Balance diagnostics in propensity score analysis following multiple imputation: A new method

The combination of propensity score analysis and multiple imputation has been prominent in epidemiological research in recent years. However, studies on the evaluation of balance in this combination are limited. In this paper, we propose a new method for assessing balance in propensity score analysis following multiple imputation. A simulation study was conducted to evaluate the performance of balance assessment methods (Leyrat's, Leite's, and new method). Simulated scenarios varied regarding the presence of missing data in the control or treatment and control group, and the imputation model with/without outcome. Leyrat's method was more biased in all the studied scenarios. Leite's method and the combine method yielded balanced results with lower mean absolute difference, regardless of whether the outcome was included in the imputation model or not. Leyrat's method had a higher false positive ratio and Leite's and combine method had higher specificity and accuracy, especially when the outcome was not included in the imputation model. According to simulation results, most of time, Leyrat's method and Leite's method contradict with each other on appraising the balance. This discrepancy can be solved using new combine method.

Lifestyle predictors of colorectal cancer in European populations: a systematic review

Background

Colorectal cancer (CRC) is the second most prevalent cancer in Europe, with one-fifth of cases attributable to unhealthy lifestyles. Risk prediction models for quantifying CRC risk and identifying high-risk groups have been developed or validated across European populations, some considering lifestyle as a predictor.

Purpose

To identify lifestyle predictors considered in existing risk prediction models applicable for European populations and characterise their corresponding parameter values for an improved understanding of their relative contribution to prediction across different models.

Methods

A systematic review was conducted in PubMed and Web of Science from January 2000 to August 2021. Risk prediction models were included if (1) developed and/or validated in an adult asymptomatic European population, (2) based on non-invasively measured predictors and (3) reported mean estimates and uncertainty for predictors included. To facilitate comparison, model-specific lifestyle predictors were visualised using forest plots.

Results

A total of 21 risk prediction models for CRC (reported in 16 studies) were eligible, of which 11 were validated in a European adult population but developed elsewhere, mostly USA. All models but two reported at least one lifestyle factor as predictor. Of the lifestyle factors, the most common predictors were body mass index (BMI) and smoking (each present in 13 models), followed by alcohol (11), and physical activity (7), while diet-related factors were less considered with the most commonly present meat (9), vegetables (5) or dairy (2). The independent predictive contribution was generally greater when they were collected with greater detail, although a noticeable variation in effect size estimates for BMI, smoking and alcohol.

Conclusions

Early identification of high-risk groups based on lifestyle data offers the potential to encourage participation in lifestyle change and screening programmes, hence reduce CRC burden. We propose the commonly shared lifestyle predictors to be further used in public health prediction modelling for improved uptake of the model.

Validation of the ADFICE_IT Models for Predicting Falls and Recurrent Falls in Geriatric Outpatients

OBJECTIVES

Before being used in clinical practice, a prediction model should be tested in patients whose data were not used in model development. Previously, we developed the ADFICE_IT models for predicting any fall and recurrent falls, referred as Any_fall and Recur_fall. In this study, we externally validated the models and compared their clinical value to a practical screening strategy where patients are screened for falls history alone.

DESIGN

Retrospective, combined analysis of 2 prospective cohorts.

SETTING AND PARTICIPANTS

Data were included of 1125 patients (aged ≥65 years) who visited the geriatrics department or the emergency department.

METHODS

We evaluated the models' discrimination using the C-statistic. Models were updated using logistic regression if calibration intercept or slope values deviated significantly from their ideal values. Decision curve analysis was applied to compare the models' clinical value (ie, net benefit) against that of falls history for different decision thresholds.

RESULTS

During the 1-year follow-up, 428 participants (42.7%) endured 1 or more falls, and 224 participants (23.1%) endured a recurrent fall (≥2 falls). C-statistic values were 0.66 (95% CI 0.63-0.69) and 0.69 (95% CI 0.65-0.72) for the Any_fall and Recur_fall models, respectively. Any_fall overestimated the fall risk and we therefore updated only its intercept whereas Recur_fall showed good calibration and required no update. Compared with falls history, Any_fall and Recur_fall showed greater net benefit for decision thresholds of 35% to 60% and 15% to 45%, respectively.

CONCLUSIONS AND IMPLICATIONS

The models performed similarly in this data set of geriatric outpatients as in the development sample. This suggests that fall-risk assessment tools that were developed in community-dwelling older adults may perform well in geriatric outpatients. We found that in geriatric outpatients the models have greater clinical value across a wide range of decision thresholds compared with screening for falls history alone.

Estimation and evaluation of individualized treatment rules following multiple imputation

An individualized treatment rule (ITR) is a function that inputs patient-level information and outputs a recommended treatment. An important focus of precision medicine is to develop optimal ITRs that maximize a population-level distributional summary. However, guidance for estimating and evaluating optimal ITRs in the presence of missing data is limited. Our work is motivated by the Social Incentives to Encourage Physical Activity and Understand Predictors (STEP UP) study. Participants were randomized to a control or one of three interventions designed to increase physical activity and were given wearable devices to record daily steps as a measure of physical activity. Many participants were missing at least one daily step count during the study period. In the primary analysis of the STEP UP trial, multiple imputation (MI) was used to address missingness in daily step counts. Despite ubiquitous use of MI in practice, it has been given relatively little attention in the context of personalized medicine. We fill this gap by describing two frameworks for estimation and evaluation of an optimal ITR following MI and assessing their performance using simulated data. One framework relies on splitting the data into independent training and testing sets for estimation and evaluation, respectively. The other framework estimates an optimal ITR using the full data and constructs anm $$ m $$ -out-of-n $$ n $$ bootstrap confidence interval to evaluate its performance. Finally, we provide an illustrative analysis to estimate and evaluate an optimal ITR from the STEP UP data with a focus on practical considerations such as choosing the number of imputations.

Imputation and missing indicators for handling missing data in the development and deployment of clinical prediction models: A simulation study

Background: In clinical prediction modelling, missing data can occur at any stage of the model pipeline; development, validation or deployment. Multiple imputation is often recommended yet challenging to apply at deployment; for example, the outcome cannot be in the imputation model, as recommended under multiple imputation. Regression imputation uses a fitted model to impute the predicted value of missing predictors from observed data, and could offer a pragmatic alternative at deployment. Moreover, the use of missing indicators has been proposed to handle informative missingness, but it is currently unknown how well this method performs in the context of clinical prediction models. Methods: We simulated data under various missing data mechanisms to compare the predictive performance of clinical prediction models developed using both imputation methods. We consider deployment scenarios where missing data is permitted or prohibited, imputation models that use or omit the outcome, and clinical prediction models that include or omit missing indicators. We assume that the missingness mechanism remains constant across the model pipeline. We also apply the proposed strategies to critical care data. Results: With complete data available at deployment, our findings were in line with existing recommendations; that the outcome should be used to impute development data when using multiple imputation and omitted under regression imputation. When missingness is allowed at deployment, omitting the outcome from the imputation model at the development was preferred. Missing indicators improved model performance in many cases but can be harmful under outcome-dependent missingness. Conclusion: We provide evidence that commonly taught principles of handling missing data via multiple imputation may not apply to clinical prediction models, particularly when data can be missing at deployment. We observed comparable predictive performance under multiple imputation and regression imputation. The performance of the missing data handling method must be evaluated on a study-by-study basis, and the most appropriate strategy for handling missing data at development should consider whether missing data are allowed at deployment. Some guidance is provided.

Vital signs-based deterioration prediction model assumptions can lead to losses in prediction performance

OBJECTIVE

Vital signs-based models are complicated by repeated measures per patient and frequently missing data. This paper investigated the impacts of common vital signs modeling assumptions during clinical deterioration prediction model development.

STUDY DESIGN AND SETTING

Electronic medical record (EMR) data from five Australian hospitals (1 January 2019-31 December 2020) were used. Summary statistics for each observation's prior vital signs were created. Missing data patterns were investigated using boosted decision trees, then imputed with common methods. Two example models predicting in-hospital mortality were developed, as follows: logistic regression and eXtreme Gradient Boosting. Model discrimination and calibration were assessed using the C-statistic and nonparametric calibration plots.

RESULTS

The data contained 5,620,641 observations from 342,149 admissions. Missing vitals were associated with observation frequency, vital sign variability, and patient consciousness. Summary statistics improved discrimination slightly for logistic regression and markedly for eXtreme Gradient Boosting. Imputation method led to notable differences in model discrimination and calibration. Model calibration was generally poor.

CONCLUSION

Summary statistics and imputation methods can improve model discrimination and reduce bias during model development, but it is questionable whether these differences are clinically significant. Researchers should consider why data are missing during model development and how this may impact clinical utility.

Clinical Impact and Risk Factors of Mortality in Hospitalized Children and Adolescents With Hematologic Diseases and COVID-19: An Observational Retrospective Cohort Study

This study aimed to evaluate the risk factors for COVID-19-related death in a large cohort of hospitalized children with hematological disorders. We performed an analysis of all pediatric patients with COVID-19 registered in a Brazilian nationwide surveillance database between February 2020 and May 2021. The primary outcome was time to death, which was evaluated considering discharge as a competitive risk by using the cumulative incidence function. Among 21,591 hospitalized pediatric patients with COVID-19, 596 cases (2.8%) had hematological diseases. Sixty-one children (27.4%) with malignant hematological diseases had a fatal outcome as compared with 4.2% and 7.4% of nonmalignant hematological and nonhematological cohorts, respectively ( P <0.0001). Children with hematological diseases had a significant increased hazard of death compared with those without these conditions (hazard ratio [HR],=2.40, 95% confidence interval, 1.98 - 2.91). In multivariable analysis, the factors associated with death were the presence of malignant hematological disease (HR, 2.22, 95% CI 1.47 - 3.36), age >10 years (HR 2.19, 95% CI 1.46 - 3.19), male (HR 1.52, 95% CI 1.02 - 2.27), oxygen saturation <95% (HR 2.02, 95% CI 1.38 - 2.96), and abdominal pain at admission (HR 2.75, 95% CI 1.76 - 4.27). Children with malignant hematological diseases had a higher risk of death compared with those without these disorders.

Outcomes and risk factors for death among hospitalized children and adolescents with kidney diseases and COVID-19: an analysis of a nationwide database

BACKGROUND

Patients with kidney diseases (KD) appear to be at particularly high risk for severe COVID-19. This study aimed to characterize the clinical outcomes and risk factors for COVID-19-related death in a large cohort of hospitalized pediatric patients with KD.

METHODS

We performed an analysis of all pediatric patients with KD and COVID-19 registered in SIVEP-Gripe, a Brazilian nationwide surveillance database, between February 16, 2020, and May 29, 2021. The primary outcome was time to death, which was evaluated considering discharge as a competitive risk by using cumulative incidence function.

RESULTS

Among 21,591 hospitalized patients with COVID-19, 290 cases (1.3%) had KD. Of these, 59 (20.8%) had a fatal outcome compared with 7.5% of the non-KD cohort (P < 0.001). Pediatric patients with KD had an increased hazard of death compared with the non-KD cohort (Hazard ratio [HR] = 2.85, 95% CI 2.21-3.68, P < 0.0001). After adjustment, the factors associated with the death among KD patients were living in Northeast (HR 2.16, 95% CI 1.13-4.31) or North regions (HR 3.50, 95% CI 1.57-7.80), oxygen saturation < 95% at presentation (HR 2.31, 95% CI 1.30-4.10), and presence of two or more associated comorbidities (HR 2.10, 95% CI 1.08-4.04).

CONCLUSIONS

Children and adolescents with KD had a higher risk of death compared with the non-KD cohort. The higher risk was associated with low oxygen saturation at admission, living in socioeconomically disadvantaged regions, and presence of other pre-existing comorbidities. A higher resolution version of the Graphical abstract is available as Supplementary information.

External validation and updating of prediction models for estimating the 1-year risk of low health-related quality of life in colorectal cancer survivors

OBJECTIVES

Timely identification of colorectal cancer (CRC) survivors at risk of experiencing low health-related quality of life (HRQoL) in the near future is important for enabling appropriately tailored preventive actions. We previously developed and internally validated risk prediction models to estimate the 1-year risk of low HRQoL in long-term CRC survivors. In this article, we aim to externally validate and update these models in a population of short-term CRC survivors.

STUDY DESIGN AND SETTING

In a pooled cohort of 1,596 CRC survivors, seven HRQoL domains (global QoL, cognitive/emotional/physical/role/social functioning, and fatigue) were measured prospectively at approximately 5 months postdiagnosis (baseline for prediction) and approximately 1 year later by a validated patient-reported outcome measure (European Organization for Research and Treatment of Cancer Quality of life Questionnaire-Core 30). For each HRQoL domain, 1-year scores were dichotomized into low vs. normal/high HRQoL. Performance of the previously developed multivariable logistic prediction models was evaluated (calibration and discrimination). Models were updated to create a more parsimonious predictor set for all HRQoL domains.

RESULTS

Updated models showed good calibration and discrimination (AUC ≥0.75), containing a single set of 15 predictors, including nonmodifiable (age, sex, education, time since diagnosis, chemotherapy, radiotherapy, stoma, and comorbidities) and modifiable predictors (body mass index, physical activity, smoking, anxiety/depression, and baseline fatigue and HRQoL domain scores).

CONCLUSION

Externally validated and updated prediction models performed well for estimating the 1-year risk of low HRQoL in CRC survivors within 6 months postdiagnosis. The impact of implementing the models in oncology practice to improve HRQoL outcomes in CRC survivors needs to be evaluated.

Standardization of risk prediction model reporting in cancer-associated thrombosis: Communication from the ISTH SSC subcommittee on hemostasis and malignancy

Since the development of the Khorana score to predict risk of cancer-associated venous thromboembolism (VTE), many modified and de novo risk prediction models (RPMs) have been proposed. Comparison of the prognostic performance across models requires comprehensive reporting and standardized methods for model development, validation and evaluation. To improve the standardization of RPM reporting, the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) tool was published in 2015. To better understand the quality of reporting and development of RPMs for cancer-associated VTE, we performed a literature search of published RPMs and assessed each model using the TRIPOD checklist. Our results yielded 29 RPMs for which 30 items were evaluated. There was a non-significant (p = 0.15) improvement in reporting of the 30 items in the post-TRIPOD era (81%) versus the pre-TRIPOD era (75%). Of seven items (title, sample size, missing data handling, baseline demographics, methods and results for model performance, and supplemental resources) with the lowest reporting in the pre-TRIPOD era (<70%), there was an average improvement of 22% in the post-TRIPOD era. Only two of the 22 studies published in the post-TRIPOD era acknowledged compliance with TRIPOD. Informed by the results of this assessment, the Scientific and Standardization Committee (SSC) Subcommittee on Hemostasis & Malignancy of the International Society on Thrombosis and Hemostasis (ISTH) advocates for standardization of four key elements of RPMs for cancer-associated VTE: (1) inclusion of the TRIPOD checklist, (2) clear definition of the derivation population, with justification of sample size, (3) clear definition of predictors, and (4) external validation prior to implementation.

Development and validation of a clinical prediction model of acute kidney injury in intensive care unit patients at a rural tertiary teaching hospital in South Africa: a study protocol

INTRODUCTION

Acute kidney injury (AKI) is a decline in renal function lasting hours to days. The rising global incidence of AKI, and associated costs of renal replacement therapy, is a public health priority. With the only therapeutic option being supportive therapy, prevention and early diagnosis will facilitate timely interventions to prevent progression to chronic kidney disease. While many factors have been identified as predictive of AKI, none have shown adequate sensitivity or specificity on their own. Many tools have been developed in developed-country cohorts with higher rates of non-communicable disease, and few have been validated and practically implemented. The development and validation of a predictive tool incorporating clinical, biochemical and imaging parameters, as well as quantification of their impact on the development of AKI, should make timely and improved prediction of AKI possible. This study is positioned to develop and validate an AKI prediction tool in critically ill patients at a rural tertiary hospital in South Africa.

METHOD AND ANALYSIS

Critically ill patients will be followed from admission until discharge or death. Risk factors for AKI will be identified and their impact quantified using statistical modelling. Internal validation of the developed model will be done on separate patients admitted at a different time. Furthermore, patients developing AKI will be monitored for 3 months to assess renal recovery and quality of life. The study will also explore the utility of endothelial monitoring using the biomarker Syndecan-1 and capillary leak measurements in predicting persistent AKI.

ETHICS AND DISSEMINATION

The study has been approved by the Walter Sisulu University Faculty of Health Science Research Ethics and Biosafety Committee (WSU No. 005/2021), and the Eastern Cape Department of Health Research Ethics (approval number: EC 202103006). The findings will be shared with facility management, and presented at relevant conferences and seminars.

Introduction to Clinical Prediction Models

Clinical prediction models include a diagnostic prediction model to estimate the probability of an individual currently having a disease (e.g., pulmonary embolism) and a prognostic prediction model to estimate the probability of an individual developing a specific health outcome over a specific time period (e.g., myocardial infarction and stroke in 10 years). Clinical prediction models can be developed by applying traditional regression models (e.g., logistic and Cox regression models) or emerging machine learning models to real-world data, such as electronic health records and administrative claims data. For derivation, researchers select candidate variables based on a literature review and clinical knowledge, and predictor variables in the final model based on pre-defined criteria (e.g., thresholds for the size of relative risk and p-values) or strategies such as the stepwise regression and the least absolute shrinkage and selection operator (LASSO) regression. For validation, the clinical prediction model's performance is evaluated in terms of goodness of fit (e.g., R2), discrimination (e.g., area under the receiver operating characteristic curve or c-statistics), and calibration (e.g., calibration plot and Hosmer-Lemeshow test). Performance of a new variable added to an existing clinical prediction model is evaluated in terms of reclassification (e.g., net reclassification improvement and integrated discrimination improvement). The model should be validated using the original data to examine internal validity through methods such as resampling (e.g., cross-validation and bootstrapping) and using other participants' data to examine external validity. For successful implementation of a clinical prediction model in actual clinical practice, presentation methods such as paper-based (nomogram) or web-based calculator and an easy-to-use risk score should be considered.

Comparison of the First and Second Waves of the Coronavirus Disease 2019 Pandemic in Children and Adolescents in a Middle-Income Country: Clinical Impact Associated with Severe Acute Respiratory Syndrome Coronavirus 2 Gamma Lineage

OBJECTIVE

To evaluate the severity and clinical outcomes of the SARS-CoV-2 gamma variant in children and adolescents hospitalized with COVID-19 in Brazil.

STUDY DESIGN

In this observational retrospective cohort study, we performed an analysis of all 21 591 hospitalized patients aged <20 years with confirmed SARS-CoV-2 infection registered in a national database in Brazil. The cohort was divided into 2 groups according to the predominance of SARS-CoV-2 lineages (WAVE1, n = 11 574; WAVE2, n = 10 017). The characteristics of interest were age, sex, geographic region, ethnicity, clinical presentation, and comorbidities. The primary outcome was time to death, which was evaluated by competing-risks analysis, using cumulative incidence functions. A predictive Fine and Gray competing-risks model was developed based on the WAVE1 cohort with temporal validation in the WAVE2 cohort.

RESULTS

Compared with children and adolescents admitted during the first wave, those admitted during the second wave had significantly more hypoxemia (52.5% vs 41.1%; P < .0001) and intensive care unit admissions (28.3% vs 24.9%; P < .0001) and needed more noninvasive ventilatory support (37.3% vs 31.6%; P < .0001). In-hospital deaths and death rates were 896 (7.7%) in the first wave and 765 (7.6%) in the second wave (P = .07). The prediction model of death included age, ethnicity, region, respiratory symptoms, and comorbidities. In the validation set (WAVE2), the C statistic was 0.750 (95% CI, 0.741-0.758; P < .0001).

CONCLUSIONS

This large national study found a more severe spectrum of risk for pediatric patients with COVID-19 caused by the gamma variant. However, there was no difference regarding the probability of death between the waves.

Outcomes and risk factors of death among hospitalized children and adolescents with obesity and COVID-19 in Brazil: An analysis of a nationwide database

BACKGROUND

Obesity is a well-recognized risk factor for critical illness and death among adult patients with SARS-CoV-2 infection.

OBJECTIVE

This study aimed to characterize the clinical outcomes and risk factors of death related to obesity in a cohort of hospitalized paediatric patients with COVID-19.

METHODS

We performed an analysis of all paediatric patients with obesity and COVID-19 registered in SIVEP-Gripe, a Brazilian nationwide surveillance database, between February 2020 and May 2021. The primary outcome was time to death, which was evaluated by using cumulative incidence function.

RESULTS

Among 21 591 hospitalized paediatric patients with COVID-19, 477 cases (2.2%) had obesity. Of them, 71 (14.9%) had a fatal outcome as compared with 7.5% for patients without obesity (hazard ratio [HR] = 2.0, 95% confidence interval [CI] 1.59-2.53, p < 0.001). After adjustment, the factors associated with death among patients with obesity were female gender (HR = 2.8, 95% CI 1.70-4.61), oxygen saturation < 95% (HR = 2.58, 95% CI 1.38-4.79), presence of one (HR = 1.91, 95% CI 1.11-3.26), and two or more comorbidities (HR = 4.0, 95% CI 2.21-7.56).

CONCLUSIONS

Children and adolescents with obesity had higher risk of death compared with those without obesity. The higher risk of death was associated with female gender, low oxygen saturation at admission, and presence of other comorbidities.

Risk factors for COVID-19-related mortality in hospitalized children and adolescents with diabetes mellitus: An observational retrospective cohort study

BACKGROUND

Diabetes has been recognized as a major comorbidity for COVID-19 severity in adults. This study aimed to characterize the clinical outcomes and risk factors for COVID-19-related death in a large cohort of hospitalized pediatric patients with diabetes.

METHODS

We performed an analysis of all pediatric patients with diabetes and COVID-19 registered in SIVEP-Gripe, a Brazilian nationwide surveillance database, between February 2020 and May 2021. The primary outcome was time to death, which was evaluated considering discharge as a competitive risk by using cumulative incidence function.

RESULTS

Among 21,591 hospitalized pediatric patients with COVID-19, 379 (1.8%) had diabetes. Overall, children and adolescents with diabetes had a higher prevalence of ICU admission (46.6% vs. 26%), invasive ventilation (16.9% vs. 10.3%), and death (15% vs. 7.6%) (all P < 0.0001). Children with diabetes had twice the hazard of death compared with pediatric patients without diabetes (Hazard ratio [HR] = 2.0, 95% CI, 1.58-2.66). Among children with diabetes, four covariates were independently associated with the primary outcome, living in the poorest regions of the country (Northeast, HR, 2.17, 95% CI 1.18-4.01, and North, (HR 4.0, 95% CI 1.79-8.94), oxygen saturation < 95% at admission (HR 2.97, 95% CI 1.64-5.36), presence of kidney disorders (HR 3.39, 95% CI 1.42-8.09), and presence of obesity (HR 3.77, 95% CI 1.83-7.76).

CONCLUSION

Children and adolescents with diabetes had a higher risk of death compared with patients without diabetes. The higher risk of death was associated with clinical and socioeconomic factors.

Characteristics of pediatric emergency department frequent visitors and their risk of a return visit: A large observational study using electronic health record data

Background

Among pediatric emergency department (ED) visits, a subgroup of children repeatedly visits the ED, making them frequent visitors (FVs). The aim of this study is to get insight into the group of pediatric ED FVs and to determine risk factors associated with a revisit.

Methods and findings

Data of all children aged 0–18 years visiting the ED of a university hospital in the Netherlands between 2017 and 2020 were included in this observational study based on routine data extraction. Children with 4 or more ED visits within 365 days were classified as FVs. Descriptive analysis of the study cohort at patient- and visit-level were performed. Risk factors for a recurrent ED visit were determined using a Prentice Williams and Peterson gap time cox-based model. Our study population of 10,209 children with 16,397 ED visits contained 500 FVs (4.9%) accounting for 3,481 visits (21.2%). At patient-level, FVs were younger and more often suffered from chronic diseases (CDs). At visit-level, frequent visits were more often initiated by self-referral and were more often related to medical problems (compared to trauma’s). Overall, FVs presented at the ED more often because of an infection (41.3%) compared to non-FVs (27.4%), either associated or not with the body system affected by the CD. We identified the presence of a comorbidity (non-complex CD HR 1.66; 1.52–1.81 and complex CD HR 2.00; 1.84–2.16) as determinants with the highest hazard for a return visit.

Conclusion

Pediatric ED FVs are a small group of children but account for a large amount of the total ED visits. FVs are younger patients, suffering from (complex) comorbidities and present more often with infectious conditions compared to non-FVs. Healthcare pathways, including safety-netting strategies for acute manifestations from their comorbidity, or for infectious conditions in general may contribute to support parents and redirect some patients from the ED.

OUP accepted manuscript

Background

As the outcome of modern colorectal cancer (CRC) surgery has significantly improved over the years, however, renewed and adequate risk stratification for mortality is important to identify high-risk patients. This population-based study was conducted to analyse postoperative outcomes in patients with CRC and to create a risk model for 30-day mortality.

Methods

Data from the Dutch Colorectal Audit were used to assess differences in postoperative outcomes (30-day mortality, hospital stay, blood transfusion, postoperative complications) in patients with CRC treated from 2009 to 2017. Time trends were analysed. Clinical variables were retrieved (including stage, age, sex, BMI, ASA grade, tumour location, timing, surgical approach) and a prediction model with multivariable regression was computed for 30-day mortality using data from 2009 to 2014. The predictive performance of the model was tested among a validation cohort of patients treated between 2015 and 2017.

Results

The prediction model was obtained using data from 51 484 patients and the validation cohort consisted of 32 926 patients. Trends of decreased length of postoperative hospital stay and blood transfusions were found over the years. In stage I–III, postoperative complications declined from 34.3 per cent to 29.0 per cent (P < 0.001) over time, whereas in stage IV complications increased from 35.6 per cent to 39.5 per cent (P = 0.010). Mortality decreased in stage I–III from 3.0 per cent to 1.4 per cent (P < 0.001) and in stage IV from 7.6 per cent to 2.9 per cent (P < 0.001). Eight factors, including stage, age, sex, BMI, ASA grade, tumour location, timing, and surgical approach were included in a 30-day mortality prediction model. The results on the validation cohort documented a concordance C statistic of 0.82 (95 per cent c.i. 0.80 to 0.83) for the prediction model, indicating good discriminative ability.

Conclusion

Postoperative outcome improved in all stages of CRC surgery in the Netherlands. The developed model accurately predicts postoperative mortality risk and is clinically valuable for decision-making.

Missing data is poorly handled and reported in prediction model studies using machine learning: a literature review

OBJECTIVES

Missing data is a common problem during the development, evaluation, and implementation of prediction models. Although machine learning (ML) methods are often said to be capable of circumventing missing data, it is unclear how these methods are used in medical research. We aim to find out if and how well prediction model studies using machine learning report on their handling of missing data.

STUDY DESIGN AND SETTING

We systematically searched the literature on published papers between 2018 and 2019 about primary studies developing and/or validating clinical prediction models using any supervised ML methodology across medical fields. From the retrieved studies information about the amount and nature (e.g. missing completely at random, potential reasons for missingness) of missing data and the way they were handled were extracted.

RESULTS

We identified 152 machine learning-based clinical prediction model studies. A substantial amount of these 152 papers did not report anything on missing data (n = 56/152). A majority (n = 96/152) reported details on the handling of missing data (e.g., methods used), though many of these (n = 46/96) did not report the amount of the missingness in the data. In these 96 papers the authors only sometimes reported possible reasons for missingness (n = 7/96) and information about missing data mechanisms (n = 8/96). The most common approach for handling missing data was deletion (n = 65/96), mostly via complete-case analysis (CCA) (n = 43/96). Very few studies used multiple imputation (n = 8/96) or built-in mechanisms such as surrogate splits (n = 7/96) that directly address missing data during the development, validation, or implementation of the prediction model.

CONCLUSION

Though missing values are highly common in any type of medical research and certainly in the research based on routine healthcare data, a majority of the prediction model studies using machine learning does not report sufficient information on the presence and handling of missing data. Strategies in which patient data are simply omitted are unfortunately the most often used methods, even though it is generally advised against and well known that it likely causes bias and loss of analytical power in prediction model development and in the predictive accuracy estimates. Prediction model researchers should be much more aware of alternative methodologies to address missing data.

Clinical Prediction Models in Sports Medicine: A Guide for Clinicians and Researchers

SYNOPSIS

Participating in sport carries inherent risk of injury. Clinicians execute high-level clinical reasoning and decision making to support athletes to achieve the best outcomes. Accurately diagnosing a problem, estimating prognosis, or selecting the most suitable intervention for each athlete is challenging. Clinical prediction models are tools to assist clinicians in estimating the risk or probability of a health outcome for an individual by using data from multiple predictors. Although common in general medical literature, clinical prediction models are rare in sports medicine. The purpose of this article was to (1) describe the steps required to develop and validate (ie, evaluate) a clinical prediction model for clinical researchers, and (2) help sports medicine clinicians understand and interpret clinical prediction model studies. Using a case study to illustrate how to implement clinical prediction models in practice, we address the following issues in developing and validating a clinical prediction model: study design and data, sample size, missing data, selecting predictors, handling continuous predictors, model fitting, internal and external validation, performance measures, reporting, and model presentation. Our work builds on initiatives to improve diagnostic and prognostic clinical research, including the PROGnosis RESearch Strategy (PROGRESS) series of papers and textbook and the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement. J Orthop Sports Phys Ther 2021;51(10):517-525. doi:10.2519/jospt.2021.10697.

The development and validation of prognostic models for overall survival in the presence of missing data in the training dataset: a strategy with a detailed example

BACKGROUND

The United Kingdom Myeloma Research Alliance (UK-MRA) Myeloma Risk Profile is a prognostic model for overall survival. It was trained and tested on clinical trial data, aiming to improve the stratification of transplant ineligible (TNE) patients with newly diagnosed multiple myeloma. Missing data is a common problem which affects the development and validation of prognostic models, where decisions on how to address missingness have implications on the choice of methodology.

METHODS

Model building The training and test datasets were the TNE pathways from two large randomised multicentre, phase III clinical trials. Potential prognostic factors were identified by expert opinion. Missing data in the training dataset was imputed using multiple imputation by chained equations. Univariate analysis fitted Cox proportional hazards models in each imputed dataset with the estimates combined by Rubin's rules. Multivariable analysis applied penalised Cox regression models, with a fixed penalty term across the imputed datasets. The estimates from each imputed dataset and bootstrap standard errors were combined by Rubin's rules to define the prognostic model. Model assessment Calibration was assessed by visualising the observed and predicted probabilities across the imputed datasets. Discrimination was assessed by combining the prognostic separation D-statistic from each imputed dataset by Rubin's rules. Model validation The D-statistic was applied in a bootstrap internal validation process in the training dataset and an external validation process in the test dataset, where acceptable performance was pre-specified. Development of risk groups Risk groups were defined using the tertiles of the combined prognostic index, obtained by combining the prognostic index from each imputed dataset by Rubin's rules.

RESULTS

The training dataset included 1852 patients, 1268 (68.47%) with complete case data. Ten imputed datasets were generated. Five hundred twenty patients were included in the test dataset. The D-statistic for the prognostic model was 0.840 (95% CI 0.716-0.964) in the training dataset and 0.654 (95% CI 0.497-0.811) in the test dataset and the corrected D-Statistic was 0.801.

CONCLUSION

The decision to impute missing covariate data in the training dataset influenced the methods implemented to train and test the model. To extend current literature and aid future researchers, we have presented a detailed example of one approach. Whilst our example is not without limitations, a benefit is that all of the patient information available in the training dataset was utilised to develop the model.

TRIAL REGISTRATION

Both trials were registered; Myeloma IX- ISRCTN68454111 , registered 21 September 2000. Myeloma XI- ISRCTN49407852 , registered 24 June 2009.

External validation of prognostic models for recovery in patients with neck pain

Background

Neck pain is one of the leading causes of disability in most countries and it is likely to increase further. Numerous prognostic models for people with neck pain have been developed, few have been validated. In a recent systematic review, external validation of three promising models was advised before they can be used in clinical practice.

Objective

The purpose of this study was to externally validate three promising models that predict neck pain recovery in primary care.

Methods

This validation cohort consisted of 1311 patients with neck pain of any duration who were prospectively recruited and treated by 345 manual therapists in the Netherlands. Outcome measures were disability (Neck Disability Index) and recovery (Global Perceived Effect Scale) post-treatment and at 1-year follow-up. The assessed models were an Australian Whiplash-Associated Disorders (WAD) model (Amodel), a multicenter WAD model (Mmodel), and a Dutch non-specific neck pain model (Dmodel). Models’ discrimination and calibration were evaluated.

Results

The Dmodel and Amodel discriminative performance (AUC < 0.70) and calibration measures (slope largely different from 1) were poor. The Mmodel could not be evaluated since several variables nor their proxies were available.

Conclusions

External validation of promising prognostic models for neck pain recovery was not successful and their clinical use cannot be recommended. We advise clinicians to underpin their current clinical reasoning process with evidence-based individual prognostic factors for recovery. Further research on finding new prognostic factors and developing and validating models with up-to-date methodology is needed for recovery in patients with neck pain in primary care.

Propensity Score Analysis with Partially Observed Baseline Covariates: A Practical Comparison of Methods for Handling Missing Data

(1) Background: Propensity score methods gained popularity in non-interventional clinical studies. As it may often occur in observational datasets, some values in baseline covariates are missing for some patients. The present study aims to compare the performances of popular statistical methods to deal with missing data in propensity score analysis. (2) Methods: Methods that account for missing data during the estimation process and methods based on the imputation of missing values, such as multiple imputations, were considered. The methods were applied on the dataset of an ongoing prospective registry for the treatment of unprotected left main coronary artery disease. The performances were assessed in terms of the overall balance of baseline covariates. (3) Results: Methods that explicitly deal with missing data were superior to classical complete case analysis. The best balance was observed when propensity scores were estimated with a method that accounts for missing data using a stochastic approximation of the expectation-maximization algorithm. (4) Conclusions: If missing at random mechanism is plausible, methods that use missing data to estimate propensity score or impute them should be preferred. Sensitivity analyses are encouraged to evaluate the implications methods used to handle missing data and estimate propensity score.

A Deep Learning-Based Unsupervised Method to Impute Missing Values in Patient Records for Improved Management of Cardiovascular Patients

Physicians increasingly depend on electronic health records (EHRs) to manage their patients. However, many patient records have substantial missing values that pose a fundamental challenge to their clinical use. To address this prevailing challenge, we propose an unsupervised deep learning-based method that can facilitate physicians' use of EHRs to improve their management of cardiovascular patients. By building on the deep autoencoder framework, we develop a novel method to impute missing values in patient records. To demonstrate its clinical applicability and values, we use data from cardiovascular patients and evaluate the proposed method's imputation effectiveness and predictive efficacy, in comparison with six prevalent benchmark techniques. The proposed method can impute missing values and predict important patient outcomes more effectively than all the benchmark techniques. This study reinforces the importance of adequately addressing missing values in patient records. It further illustrates how effective imputations can enable greater predictive efficacy with regard to important patient outcomes, which are crucial to the use of EHRs and health analytics for improved patient management. Supported by the complete data imputed by the proposed method, physicians can make timely patient outcome estimations (predictions) and therapeutic treatment assessments.

Evaluation of predictive model performance of an existing model in the presence of missing data

In medical research, the Brier score (BS) and the area under the receiver operating characteristic (ROC) curves (AUC) are two common metrics used to evaluate prediction models of a binary outcome, such as using biomarkers to predict the risk of developing a disease in the future. The assessment of an existing prediction models using data with missing covariate values is challenging. In this article, we propose inverse probability weighted (IPW) and augmented inverse probability weighted (AIPW) estimates of AUC and BS to handle the missing data. An alternative approach uses multiple imputation (MI), which requires a model for the distribution of the missing variable. We evaluated the performance of IPW and AIPW in comparison with MI in simulation studies under missing completely at random, missing at random, and missing not at random scenarios. When there are missing observations in the data, MI and IPW can be used to obtain unbiased estimates of BS and AUC if the imputation model for the missing variable or the model for the missingness is correctly specified. MI is more efficient than IPW. Our simulation results suggest that AIPW can be more efficient than IPW, and also achieves double robustness from miss-specification of either the missingness model or the imputation model. The outcome variable should be included in the model for the missing variable under all scenarios, while it only needs to be included in missingness model if the missingness depends on the outcome. We illustrate these methods using an example from prostate cancer.

The Peripheral Blood Transcriptome Is Correlated With PET Measures of Lung Inflammation During Successful Tuberculosis Treatment

Pulmonary tuberculosis (PTB) is characterized by lung granulomas, inflammation and tissue destruction. Here we used within-subject peripheral blood gene expression over time to correlate with the within-subject lung metabolic activity, as measured by positron emission tomography (PET) to identify biological processes and pathways underlying overall resolution of lung inflammation. We used next-generation RNA sequencing and [¹⁸F]FDG PET-CT data, collected at diagnosis, week 4, and week 24, from 75 successfully cured PTB patients, with the [¹⁸F]FDG activity as a surrogate for lung inflammation. Our linear mixed-effects models required that for each individual the slope of the line of [¹⁸F]FDG data in the outcome and the slope of the peripheral blood transcript expression data correlate, i.e., the slopes of the outcome and explanatory variables had to be similar. Of 10,295 genes that changed as a function of time, we identified 639 genes whose expression profiles correlated with decreasing [¹⁸F]FDG uptake levels in the lungs. Gene enrichment over-representation analysis revealed that numerous biological processes were significantly enriched in the 639 genes, including several well known in TB transcriptomics such as platelet degranulation and response to interferon gamma, thus validating our novel approach. Others not previously associated with TB pathobiology included smooth muscle contraction, a set of pathways related to mitochondrial function and cell death, as well as a set of pathways connecting transcription, translation and vesicle formation. We observed up-regulation in genes associated with B cells, and down-regulation in genes associated with platelet activation. We found 254 transcription factor binding sites to be enriched among the 639 gene promoters. In conclusion, we demonstrated that of the 10,295 gene expression changes in peripheral blood, only a subset of 639 genes correlated with inflammation in the lungs, and the enriched pathways provide a description of the biology of resolution of lung inflammation as detectable in peripheral blood. Surprisingly, resolution of PTB inflammation is positively correlated with smooth muscle contraction and, extending our previous observation on mitochondrial genes, shows the presence of mitochondrial stress. We focused on pathway analysis which can enable therapeutic target discovery and potential modulation of the host response to TB.

Real-time imputation of missing predictor values improved the application of prediction models in daily practice

OBJECTIVES

In clinical practice, many prediction models cannot be used when predictor values are missing. We, therefore, propose and evaluate methods for real-time imputation.

STUDY DESIGN AND SETTING

We describe (i) mean imputation (where missing values are replaced by the sample mean), (ii) joint modeling imputation (JMI, where we use a multivariate normal approximation to generate patient-specific imputations), and (iii) conditional modeling imputation (CMI, where a multivariable imputation model is derived for each predictor from a population). We compared these methods in a case study evaluating the root mean squared error (RMSE) and coverage of the 95% confidence intervals (i.e., the proportion of confidence intervals that contain the true predictor value) of imputed predictor values.

RESULTS

-RMSE was lowest when adopting JMI or CMI, although imputation of individual predictors did not always lead to substantial improvements as compared to mean imputation. JMI and CMI appeared particularly useful when the values of multiple predictors of the model were missing. Coverage reached the nominal level (i.e., 95%) for both CMI and JMI.

CONCLUSION

Multiple imputations using either CMI or JMI is recommended when dealing with missing predictor values in real-time settings.

Informative presence and observation in routine health data: A review of methodology for clinical risk prediction

Objective

Informative presence (IP) is the phenomenon whereby the presence or absence of patient data is potentially informative with respect to their health condition, with informative observation (IO) being the longitudinal equivalent. These phenomena predominantly exist within routinely collected healthcare data, in which data collection is driven by the clinical requirements of patients and clinicians. The extent to which IP and IO are considered when using such data to develop clinical prediction models (CPMs) is unknown, as is the existing methodology aiming at handling these issues. This review aims to synthesize such existing methodology, thereby helping identify an agenda for future methodological work.

Materials and Methods

A systematic literature search was conducted by 2 independent reviewers using prespecified keywords.

Results

Thirty-six articles were included. We categorized the methods presented within as derived predictors (including some representation of the measurement process as a predictor in the model), modeling under IP, and latent structures. Including missing indicators or summary measures as predictors is the most commonly presented approach amongst the included studies (24 of 36 articles).

Discussion

This is the first review to collate the literature in this area under a prediction framework. A considerable body relevant of literature exists, and we present ways in which the described methods could be developed further. Guidance is required for specifying the conditions under which each method should be used to enable applied prediction modelers to use these methods.

Conclusions

A growing recognition of IP and IO exists within the literature, and methodology is increasingly becoming available to leverage these phenomena for prediction purposes. IP and IO should be approached differently in a prediction context than when the primary goal is explanation. The work included in this review has demonstrated theoretical and empirical benefits of incorporating IP and IO, and therefore we recommend that applied health researchers consider incorporating these methods in their work.

Missing data and prediction: the pattern submodel

Missing data are a common problem for both the construction and implementation of a prediction algorithm. Pattern submodels (PS)—a set of submodels for every missing data pattern that are fit using only data from that pattern—are a computationally efficient remedy for handling missing data at both stages. Here, we show that PS (i) retain their predictive accuracy even when the missing data mechanism is not missing at random (MAR) and (ii) yield an algorithm that is the most predictive among all standard missing data strategies. Specifically, we show that the expected loss of a forecasting algorithm is minimized when each pattern-specific loss is minimized. Simulations and a re-analysis of the SUPPORT study confirms that PS generally outperforms zero-imputation, mean-imputation, complete-case analysis, complete-case submodels, and even multiple imputation (MI). The degree of improvement is highly dependent on the missingness mechanism and the effect size of missing predictors. When the data are MAR, MI can yield comparable forecasting performance but generally requires a larger computational cost. We also show that predictions from the PS approach are equivalent to the limiting predictions for a MI procedure that is dependent on missingness indicators (the MIMI model). The focus of this article is on out-of-sample prediction; implications for model inference are only briefly explored.

Recovery expectations of neck pain patients do not predict treatments outcome in manual therapy

Patient recovery expectations can predict treatment outcome. Little is known about the association of patient recovery expectations on treatment outcome in patients with neck pain consulting a manual therapist. This study evaluates the predictive value of recovery expectations in neck pain patients consulting manual therapists in the Netherlands. The primary outcome measure 'recovery' is defined as 'reduction in pain and perceived improvement'. A prospective cohort study a total of 1195 neck pain patients. Patients completed the Patient Expectancies List (PEL) at baseline (3 item questionnaire, score range from 3 to 12), functional status (NDI), the Global Perceived Effect (GPE) for recovery (7-points Likert scale) post treatment and pain scores (NRS) at baseline and post treatment. The relationship between recovery expectancy and recovery (dichotomized GPE scores) was assessed by logistic regression analysis. Patients generally reported high recovery expectations on all three questions of the PEL (mean sumscores ranging from 11.3 to 11.6). When adjusted for covariates the PEL sum-score did not predict recovery (explained variance was 0.10 for the total PEL). Separately, the first question of the PEL showed predictive potential (OR 3.7; 95%CI 0.19-73.74) for recovery, but failed to reach statistical significance. In this study patient recovery expectations did not predict treatment outcome. Variables predicting recovery were recurrence and duration of pain. The precise relationship between patient recovery expectations and outcome is complex and still inconclusive. Research on patient expectancy would benefit from more consistent use of theoretical expectancy and outcome models.

Handling missing predictor values when validating and applying a prediction model to new patients

Missing data present challenges for development and real‐world application of clinical prediction models. While these challenges have received considerable attention in the development setting, there is only sparse research on the handling of missing data in applied settings. The main unique feature of handling missing data in these settings is that missing data methods have to be performed for a single new individual, precluding direct application of mainstay methods used during model development. Correspondingly, we propose that it is desirable to perform model validation using missing data methods that transfer to practice in single new patients. This article compares existing and new methods to account for missing data for a new individual in the context of prediction. These methods are based on (i) submodels based on observed data only, (ii) marginalization over the missing variables, or (iii) imputation based on fully conditional specification (also known as chained equations). They were compared in an internal validation setting to highlight the use of missing data methods that transfer to practice while validating a model. As a reference, they were compared to the use of multiple imputation by chained equations in a set of test patients, because this has been used in validation studies in the past. The methods were evaluated in a simulation study where performance was measured by means of optimism corrected C‐statistic and mean squared prediction error. Furthermore, they were applied in data from a large Dutch cohort of prophylactic implantable cardioverter defibrillator patients.

Systematic Review and External Validation of Prediction Models Based on Symptoms and Biomarkers for Identifying Endoscopic Activity in Crohn's Disease

BACKGROUND & AIMS

Endoscopic healing, an important target of treatment for Crohn's disease (CD), requires ileocolonoscopy, which is costly and burdensome. We investigated whether published noninvasive models (based on symptoms and biomarkers) to evaluate CD activity have sufficient accuracy to replace ileocolonoscopy.

METHODS

We performed a systematic review of published noninvasive diagnostic models to evaluate CD activity that used endoscopic features of activity (endoscopic activity) or healing as the reference standard. We externally validated these models for the outcome endoscopic activity (CD endoscopic index of severity scores, ≥3) using data from the a randomized controlled trial investigating tailored treatment with infliximab for active luminal Crohn's disease (TAILORIX) study (346 ileocolonoscopies in 155 patients) and the Utrecht Activity Index (UAI) study (93 ileocolonoscopies in 82 patients). We calculated the area under the receiver operating characteristic curves (AUROCs) for the models using data from these studies, and compared the performance of these models against measurements of fecal calprotectin (FC) and C-reactive protein (CRP).

RESULTS

We screened 5303 articles and identified 27 models (from 21 studies) for our analysis. Seven models could be validated externally; in the TAILORIX data set, these models identified patients with endoscopic activity with AUROC values ranging from 0.61 (95% CI, 0.51-0.70) to 0.81 (95% CI, 0.76-0.86). In this data set, the AUROC value for FC concentration was 0.79 (95% CI, 0.74-0.85) and the AUROC value for CRP level was 0.72 (95% CI, 0.66-0.77). The AUROC values for the validation in the UAI data set were similar. In the TAILORIX and/or UAI data set, 4 of the 7 models, as well as the FC and CRP assays, were able to identify patients with endoscopic activity with positive predictive values of 90% or more. Two of the 7 models (but not the FC or CRP values) identified patients without endoscopic activity with a negative predictive value (NPV) of 90% or more, leading to correct prediction of endoscopic healing in 3.2% to 11.3% of all patients. For example, applying the Herranz-Bachiller model (1 of 7 models) at a NPV of 92.1% and a positive predictive value of 91.9% correctly identified 35.7% of all patients in whom ileocolonoscopy could be avoided for expected endoscopic activity or healing but incorrectly identified 3.2% of all patients. Most ileocolonoscopies (66.5% in TAILORIX and 72.6% in the UAI of all ileocolonoscopies) could be avoided correctly based on concentrations of FC of 100 μg/g or less and 250 μg/g or higher. However, using this range of FC concentrations to identify patients who do not require ileocolonoscopy caused 18.7% of all patients in the TAILORIX cohort and 19.8% of all patients in the UAI cohort to be predicted incorrectly to have endoscopic activity or healing.

CONCLUSIONS

In a systematic review and external validation of noninvasive models to identify patients with endoscopic activity of CD, we found only 2 of 7 models evaluated to have NPVs of 90% or more, however, leading to correctly predicted EH in only a small proportion of patients. Ileocolonoscopy therefore remains the mainstay to evaluate CD mucosal disease activity and healing.

Missing data should be handled differently for prediction than for description or causal explanation

Missing data are much studied in epidemiology and statistics. Theoretical development and application of methods for handling missing data have mostly been conducted in the context of prospective research data and with a goal of description or causal explanation. However, it is now common to build predictive models using routinely collected data, where missing patterns may convey important information, and one might take a pragmatic approach to optimizing prediction. Therefore, different methods to handle missing data may be preferred. Furthermore, an underappreciated issue in prediction modeling is that the missing data method used in model development may not match the method used when a model is deployed. This may lead to overoptimistic assessments of model performance. For prediction, particularly with routinely collected data, methods for handling missing data that incorporate information within the missingness pattern should be explored and further developed. Where missing data methods differ between model development and model deployment, the implications of this must be explicitly evaluated. The trade-off between building a prediction model that is causally principled, and building a prediction model that maximizes the use of all available information, should be carefully considered and will depend on the intended use of the model.

Prediction of contralateral breast cancer: external validation of risk calculators in 20 international cohorts

BACKGROUND

Three tools are currently available to predict the risk of contralateral breast cancer (CBC). We aimed to compare the performance of the Manchester formula, CBCrisk, and PredictCBC in patients with invasive breast cancer (BC).

METHODS

We analyzed data of 132,756 patients (4682 CBC) from 20 international studies with a median follow-up of 8.8 years. Prediction performance included discrimination, quantified as a time-dependent Area-Under-the-Curve (AUC) at 5 and 10 years after diagnosis of primary BC, and calibration, quantified as the expected-observed (E/O) ratio at 5 and 10 years and the calibration slope.

RESULTS

The AUC at 10 years was: 0.58 (95% confidence intervals [CI] 0.57-0.59) for CBCrisk; 0.60 (95% CI 0.59-0.61) for the Manchester formula; 0.63 (95% CI 0.59-0.66) and 0.59 (95% CI 0.56-0.62) for PredictCBC-1A (for settings where BRCA1/2 mutation status is available) and PredictCBC-1B (for the general population), respectively. The E/O at 10 years: 0.82 (95% CI 0.51-1.32) for CBCrisk; 1.53 (95% CI 0.63-3.73) for the Manchester formula; 1.28 (95% CI 0.63-2.58) for PredictCBC-1A and 1.35 (95% CI 0.65-2.77) for PredictCBC-1B. The calibration slope was 1.26 (95% CI 1.01-1.50) for CBCrisk; 0.90 (95% CI 0.79-1.02) for PredictCBC-1A; 0.81 (95% CI 0.63-0.99) for PredictCBC-1B, and 0.39 (95% CI 0.34-0.43) for the Manchester formula.

CONCLUSIONS

Current CBC risk prediction tools provide only moderate discrimination and the Manchester formula was poorly calibrated. Better predictors and re-calibration are needed to improve CBC prediction and to identify low- and high-CBC risk patients for clinical decision-making.

Double trouble: Do symptom severity and duration interact to predicting treatment outcomes in adolescent depression?

Studies suggest that depression severity and duration interact to predict outcomes in depression treatment. To our knowledge, no study has explored this question in a sample with a placebo control, two therapies, and their combination nor with adolescents. We used data from the Treatment of Adolescent Depression Study (N=439), in which adolescent were randomized to placebo (PBO), cognitive-behavioral therapy (CBT), antidepressants medications (MEDs), or their combination (COMB). We explore the interaction between depression severity, chronicity, and treatments (vs. placebo) in predicting outcomes. There was interaction between severity and chronicity when comparing COMB and CBT with PBO, but not MEDs. In non-chronic depression, the effects of CBT were inversely related to severity to the point that CBT appeared iatrogenic with more severe depression. In chronic depression, the effects of CBT did not vary by severity, but the relative effects of COMB grew, being smallest in milder, more dysthymic-like depression, and largest in chronic-severe depression. These findings support calls to classify depression by severity and chronicity as well efforts to risk stratify patients to different intensity of care according to these variables.

The application of unsupervised deep learning in predictive models using electronic health records

Background

The main goal of this study is to explore the use of features representing patient-level electronic health record (EHR) data, generated by the unsupervised deep learning algorithm autoencoder, in predictive modeling. Since autoencoder features are unsupervised, this paper focuses on their general lower-dimensional representation of EHR information in a wide variety of predictive tasks.

Methods

We compare the model with autoencoder features to traditional models: logistic model with least absolute shrinkage and selection operator (LASSO) and Random Forest algorithm. In addition, we include a predictive model using a small subset of response-specific variables (Simple Reg) and a model combining these variables with features from autoencoder (Enhanced Reg). We performed the study first on simulated data that mimics real world EHR data and then on actual EHR data from eight Advocate hospitals.

Results

On simulated data with incorrect categories and missing data, the precision for autoencoder is 24.16% when fixing recall at 0.7, which is higher than Random Forest (23.61%) and lower than LASSO (25.32%). The precision is 20.92% in Simple Reg and improves to 24.89% in Enhanced Reg. When using real EHR data to predict the 30-day readmission rate, the precision of autoencoder is 19.04%, which again is higher than Random Forest (18.48%) and lower than LASSO (19.70%). The precisions for Simple Reg and Enhanced Reg are 18.70 and 19.69% respectively. That is, Enhanced Reg can have competitive prediction performance compared to LASSO. In addition, results show that Enhanced Reg usually relies on fewer features under the setting of simulations of this paper.

Conclusions

We conclude that autoencoder can create useful features representing the entire space of EHR data and which are applicable to a wide array of predictive tasks. Together with important response-specific predictors, we can derive efficient and robust predictive models with less labor in data extraction and model training.

Predicting optimal interventions for clinical depression: Moderators of outcomes in a positive psychological intervention vs. cognitive-behavioral therapy

Identifying differences in the clinical response to specific interventions is an important challenge in the field of Clinical Psychology. This is especially true in the treatment of depression where many treatments appear to have comparable outcomes. In a controlled trial, we compared a positive psychology group intervention, the Integrative Positive Psychological Intervention for Depression (IPPI-D; n = 62) to a cognitive-behavioral therapy group intervention (CBT; n = 66) for depression. No statistically or clinically-significant differences between the treatments were found, but a slight advantage was observed, on average, for IPPI-D. The aim of the present study was to identify and combine moderators of the differential efficacy of these two psychological interventions for clinical depression. For this purpose, a secondary analysis using the Personalized Advantage Index (PAI) was performed to identify the intervention predicted to produce the better outcome for each patient. Six of the 21 potential moderators were found to predict differential efficacy between the treatments. IPPI-D was predicted to be the optimal treatment for 73% of the sample. Baseline features that characterized these individuals were: mental and physical comorbidity, prior antidepressant medication, higher levels of negative thoughts, and higher personal growth. The 27% who were predicted to achieve better outcomes in CBT than in IPPI-D tended to have these baseline features: no comorbidities, no prior antidepressant medication, lower levels of negative thoughts, and lower personal growth.

Prognostic Models for Predicting Overall Survival in Patients with Primary Gastric Cancer: A Systematic Review

Background

This study was designed to review the methodology and reporting of gastric cancer prognostic models and identify potential problems in model development.

Methods

This systematic review was conducted following the CHARMS checklist. MEDLINE and EMBASE were searched. Information on patient characteristics, methodological details, and models' performance was extracted. Descriptive statistics was used to summarize the methodological and reporting quality.

Results

In total, 101 model developments and 32 external validations were included. The median (range) of training sample size, number of death, and number of final predictors were 360 (29 to 15320), 193 (14 to 9560), and 5 (2 to 53), respectively. Ninety-one models were developed from routine clinical data. Statistical assumptions were reported to be checked in only nine models. Most model developments (94/101) used complete-case analysis. Discrimination and calibration were not reported in 33 and 55 models, respectively. The majority of models (81/101) have never been externally validated. None of the models have been evaluated regarding clinical impact.

Conclusions

Many prognostic models have been developed, but their usefulness in clinical practice remains uncertain due to methodological shortcomings, insufficient reporting, and lack of external validation and impact studies.

Impact

Future research should improve methodological and reporting quality and emphasize more on external validation and impact assessment.

Avoiding pitfalls when combining multiple imputation and propensity scores

Overcoming bias due to confounding and missing data is challenging when analyzing observational data. Propensity scores are commonly used to account for the first problem and multiple imputation for the latter. Unfortunately, it is not known how best to proceed when both techniques are required. We investigate whether two different approaches to combining propensity scores and multiple imputation (Across and Within) lead to differences in the accuracy or precision of exposure effect estimates. Both approaches start by imputing missing values multiple times. Propensity scores are then estimated for each resulting dataset. Using the Across approach, the mean propensity score across imputations for each subject is used in a single subsequent analysis. Alternatively, the Within approach uses propensity scores individually to obtain exposure effect estimates in each imputation, which are combined to produce an overall estimate. These approaches were compared in a series of Monte Carlo simulations and applied to data from the British Society for Rheumatology Biologics Register. Results indicated that the Within approach produced unbiased estimates with appropriate confidence intervals, whereas the Across approach produced biased results and unrealistic confidence intervals. Researchers are encouraged to implement the Within approach when conducting propensity score analyses with incomplete data.

Methodological standards for the development and evaluation of clinical prediction rules: a review of the literature

Clinical prediction rules (CPRs) that predict the absolute risk of a clinical condition or future outcome for individual patients are abundant in the medical literature; however, systematic reviews have demonstrated shortcomings in the methodological quality and reporting of prediction studies. To maximise the potential and clinical usefulness of CPRs, they must be rigorously developed and validated, and their impact on clinical practice and patient outcomes must be evaluated. This review aims to present a comprehensive overview of the stages involved in the development, validation and evaluation of CPRs, and to describe in detail the methodological standards required at each stage, illustrated with examples where appropriate. Important features of the study design, statistical analysis, modelling strategy, data collection, performance assessment, CPR presentation and reporting are discussed, in addition to other, often overlooked aspects such as the acceptability, cost-effectiveness and longer-term implementation of CPRs, and their comparison with clinical judgement. Although the development and evaluation of a robust, clinically useful CPR is anything but straightforward, adherence to the plethora of methodological standards, recommendations and frameworks at each stage will assist in the development of a rigorous CPR that has the potential to contribute usefully to clinical practice and decision-making and have a positive impact on patient care.

Fundamentals of Clinical Prediction Modeling for the Neurosurgeon

Clinical prediction models in neurosurgery are increasingly reported. These models aim to provide an evidence-based approach to the estimation of the probability of a neurosurgical outcome by combining 2 or more prognostic variables. Model development and model reporting are often suboptimal. A basic understanding of the methodology of clinical prediction modeling is needed when interpreting these models. We address basic statistical background, 7 modeling steps, and requirements of these models such that they may fulfill their potential for major impact for our daily clinical practice and for future scientific work.

A computerised decision support system for cardiovascular risk management 'live' in the electronic health record environment: development, validation and implementation-the Utrecht Cardiovascular Cohort Initiative

Purpose

We set out to develop a real-time computerised decision support system (CDSS) embedded in the electronic health record (EHR) with information on risk factors, estimated risk, and guideline-based advice on treatment strategy in order to improve adherence to cardiovascular risk management (CVRM) guidelines with the ultimate aim of improving patient healthcare.

Methods

We defined a project plan including the scope and requirements, infrastructure and interface, data quality and study population, validation and evaluation of the CDSS.

Results

In collaboration with clinicians, data scientists, epidemiologists, ICT architects, and user experience and interface designers we developed a CDSS that provides ‘live’ information on CVRM within the environment of the EHR. The CDSS provides information on cardiovascular risk factors (age, sex, medical and family history, smoking, blood pressure, lipids, kidney function, and glucose intolerance measurements), estimated 10-year cardiovascular risk, guideline-compliant suggestions for both pharmacological and non-pharmacological treatment to optimise risk factors, and an estimate on the change in 10-year risk of cardiovascular disease if treatment goals are adhered to. Our pilot study identified a number of issues that needed to be addressed, such as missing data, rules and regulations, privacy, and patient participation.

Conclusion

Development of a CDSS is complex and requires a multidisciplinary approach. We identified opportunities and challenges in our project developing a CDSS aimed at improving adherence to CVRM guidelines. The regulatory environment, including guidance on scientific evaluation, legislation, and privacy issues needs to evolve within this emerging field of eHealth.