Impact of sample size on the stability of risk scores from clinical prediction models: a case study in cardiovascular disease

Lower HAGOS subscale scores associated with a longer duration of groin problems in football players in the subsequent season

Introduction

Groin injuries represent a considerable problem in football. Although the Adductor Strengthening Programme reduced groin injury risk, players can still experience groin symptoms throughout the season. This study aimed to determine whether preseason Copenhagen Hip and Groin Outcome Score (HAGOS) and a history of previous injury can identify individuals at risk of having a longer duration of groin problems the subsequent season, using an 'any physical complaint' definition of injury.

Methods

Preseason HAGOS score and weekly groin problems were registered with the Oslo Sports Trauma Research Center Overuse questionnaire during one full season in 632 male semiprofessional adult players.

Results

The prognostic model showed a decreased number of weeks with groin problems for each increase in HAGOS score for 'groin-related quality of life' (QOL) (IRR=0.99, p=0.003). A 10-point higher 'QOL' score predicted 10% fewer weeks of groin problems. Additionally, previous hip/groin injury was associated with a 74% increase in the number of weeks with symptoms (p<0.001).

Conclusion

The HAGOS questionnaire applied preseason can detect players at risk of getting more weeks with groin problems the following season. The 'QOL' subscale seems to be the superior subscale for estimating subsequent groin problem duration. While HAGOS appears promising in identifying players at risk, previous groin injury is the most robust indicator, showing a substantial 74% increase in weeks with symptoms.

Clinical prediction models and the multiverse of madness

BACKGROUND

Each year, thousands of clinical prediction models are developed to make predictions (e.g. estimated risk) to inform individual diagnosis and prognosis in healthcare. However, most are not reliable for use in clinical practice.

MAIN BODY

We discuss how the creation of a prediction model (e.g. using regression or machine learning methods) is dependent on the sample and size of data used to develop it-were a different sample of the same size used from the same overarching population, the developed model could be very different even when the same model development methods are used. In other words, for each model created, there exists a multiverse of other potential models for that sample size and, crucially, an individual's predicted value (e.g. estimated risk) may vary greatly across this multiverse. The more an individual's prediction varies across the multiverse, the greater the instability. We show how small development datasets lead to more different models in the multiverse, often with vastly unstable individual predictions, and explain how this can be exposed by using bootstrapping and presenting instability plots. We recommend healthcare researchers seek to use large model development datasets to reduce instability concerns. This is especially important to ensure reliability across subgroups and improve model fairness in practice.

CONCLUSIONS

Instability is concerning as an individual's predicted value is used to guide their counselling, resource prioritisation, and clinical decision making. If different samples lead to different models with very different predictions for the same individual, then this should cast doubt into using a particular model for that individual. Therefore, visualising, quantifying and reporting the instability in individual-level predictions is essential when proposing a new model.

Multiclass risk models for ovarian malignancy: an illustration of prediction uncertainty due to the choice of algorithm

BACKGROUND

Assessing malignancy risk is important to choose appropriate management of ovarian tumors. We compared six algorithms to estimate the probabilities that an ovarian tumor is benign, borderline malignant, stage I primary invasive, stage II-IV primary invasive, or secondary metastatic.

METHODS

This retrospective cohort study used 5909 patients recruited from 1999 to 2012 for model development, and 3199 patients recruited from 2012 to 2015 for model validation. Patients were recruited at oncology referral or general centers and underwent an ultrasound examination and surgery ≤ 120 days later. We developed models using standard multinomial logistic regression (MLR), Ridge MLR, random forest (RF), XGBoost, neural networks (NN), and support vector machines (SVM). We used nine clinical and ultrasound predictors but developed models with or without CA125.

RESULTS

Most tumors were benign (3980 in development and 1688 in validation data), secondary metastatic tumors were least common (246 and 172). The c-statistic (AUROC) to discriminate benign from any type of malignant tumor ranged from 0.89 to 0.92 for models with CA125, from 0.89 to 0.91 for models without. The multiclass c-statistic ranged from 0.41 (SVM) to 0.55 (XGBoost) for models with CA125, and from 0.42 (SVM) to 0.51 (standard MLR) for models without. Multiclass calibration was best for RF and XGBoost. Estimated probabilities for a benign tumor in the same patient often differed by more than 0.2 (20% points) depending on the model. Net Benefit for diagnosing malignancy was similar for algorithms at the commonly used 10% risk threshold, but was slightly higher for RF at higher thresholds. Comparing models, between 3% (XGBoost vs. NN, with CA125) and 30% (NN vs. SVM, without CA125) of patients fell on opposite sides of the 10% threshold.

CONCLUSION

Although several models had similarly good performance, individual probability estimates varied substantially.

Predicting outcomes following short-term ventricular assist device implant with the MELD-XI score

BACKGROUND

Short-term continuous flow (STCF) ventricular assist devices (VADs) are utilized in adults with cardiogenic shock; however, mortality remains high. Previous studies have found that high pre-operative MELD-XI scores in durable VAD patients are associated with mortality. The use of the MELD-XI score to predict outcomes in STCF-VAD patients has not been explored. We sought to determine the relationship between MELD-XI and outcomes in adults with STCF-VADs.

METHODS

This was a retrospective review of adults implanted with STCF-VADs between 2009 and 2019. Receiver operating characteristic (ROC) analysis was performed to predict outcomes and Kaplan-Meier analysis was done to assess survival.

RESULTS

Seventy-nine patients were included with a median MELD-XI score of 21.2 (IQR 13.5, 27.0). Patients with an unsuccessful wean from support (p < 0.001) or major post-operative bleeding (p = 0.03) had significantly higher pre-implant MELD-XI scores. The optimal MELD-XI cut-point for mortality was 24.9 with 27.8 for major bleeding. Survival was worse among patients in the high-risk MELD-XI group, however, not statistically significant (p = 0.09). Prior ECMO support, but not MELD-XI, was an independent predictor of unsuccessful wean (p = 0.03).

CONCLUSIONS

Pre-operative MELD-XI score was a moderate predictor of unsuccessful wean with limited utility in predicting bleeding in patients on STCF-VAD support. This scoring system may be useful in the clinical setting for pre-implant risk stratification and counseling among patients and outcomes.

Imputation and missing indicators for handling missing data in the development and deployment of clinical prediction models: A simulation study

Background: In clinical prediction modelling, missing data can occur at any stage of the model pipeline; development, validation or deployment. Multiple imputation is often recommended yet challenging to apply at deployment; for example, the outcome cannot be in the imputation model, as recommended under multiple imputation. Regression imputation uses a fitted model to impute the predicted value of missing predictors from observed data, and could offer a pragmatic alternative at deployment. Moreover, the use of missing indicators has been proposed to handle informative missingness, but it is currently unknown how well this method performs in the context of clinical prediction models. Methods: We simulated data under various missing data mechanisms to compare the predictive performance of clinical prediction models developed using both imputation methods. We consider deployment scenarios where missing data is permitted or prohibited, imputation models that use or omit the outcome, and clinical prediction models that include or omit missing indicators. We assume that the missingness mechanism remains constant across the model pipeline. We also apply the proposed strategies to critical care data. Results: With complete data available at deployment, our findings were in line with existing recommendations; that the outcome should be used to impute development data when using multiple imputation and omitted under regression imputation. When missingness is allowed at deployment, omitting the outcome from the imputation model at the development was preferred. Missing indicators improved model performance in many cases but can be harmful under outcome-dependent missingness. Conclusion: We provide evidence that commonly taught principles of handling missing data via multiple imputation may not apply to clinical prediction models, particularly when data can be missing at deployment. We observed comparable predictive performance under multiple imputation and regression imputation. The performance of the missing data handling method must be evaluated on a study-by-study basis, and the most appropriate strategy for handling missing data at development should consider whether missing data are allowed at deployment. Some guidance is provided.

Suicidal thoughts and behaviors in patients with chronic pain, with and without co-occurring opioid use disorder

BACKGROUND

Individuals with chronic pain and a co-occurring substance use disorder present higher risk of suicide, but the individual and joint impacts of chronic pain and substance use disorders on suicide risk are not well defined. The objective of this study was to exam the factors associated with suicidal thoughts and behaviors in a cohort of patients with chronic non-cancer pain (CNCP), with or without concomitant opioid use disorder (OUD).

DESIGN

Cross sectional cohort design.

SETTING

Primary care clinics, pain clinics, and substance abuse treatment facilities in Pennsylvania, Washington, and Utah.

SUBJECTS

In total, 609 adults with CNCP treated with long-term opioid therapy (>/= 6 months) who either developed an OUD (cases, n = 175) or displayed no evidence of OUD (controls, n = 434).

METHODS

The predicted outcome was elevated suicidal behavior in patients with CNCP as indicated by a Suicide Behavior Questionnaire-Revised (SBQ-R) score of 8 or above. The presence of CNCP and OUD were key predictors. Covariates included demographics, pain severity, psychiatric history, pain coping, social support, depression, pain catastrophizing and mental defeat.

RESULTS

Participants with CNCP and co-occurring OUD had an increased odds ratio of 3.44 in reporting elevated suicide scores as compared to participants with chronic pain only. Multivariable modeling revealed that mental defeat, pain catastrophizing, depression, and having chronic pain, and co-occurring OUD significantly increased the odds of elevated suicide scores.

CONCLUSIONS

Patients with CNCP and co-morbid OUD are associated with a 3-fold increase in risk of suicide.

Uncertainty and the Value of Information in Risk Prediction Modeling

Background

Because of the finite size of the development sample, predicted probabilities from a risk prediction model are inevitably uncertain. We apply value-of-information methodology to evaluate the decision-theoretic implications of prediction uncertainty.

Methods

Adopting a Bayesian perspective, we extend the definition of the expected value of perfect information (EVPI) from decision analysis to net benefit calculations in risk prediction. In the context of model development, EVPI is the expected gain in net benefit by using the correct predictions as opposed to predictions from a proposed model. We suggest bootstrap methods for sampling from the posterior distribution of predictions for EVPI calculation using Monte Carlo simulations. We used subsets of data of various sizes from a clinical trial for predicting mortality after myocardial infarction to show how EVPI changes with sample size.

Results

With a sample size of 1000 and at the prespecified threshold of 2% on predicted risks, the gains in net benefit using the proposed and the correct models were 0.0006 and 0.0011, respectively, resulting in an EVPI of 0.0005 and a relative EVPI of 87%. EVPI was zero only at unrealistically high thresholds (>85%). As expected, EVPI declined with larger samples. We summarize an algorithm for incorporating EVPI calculations into the commonly used bootstrap method for optimism correction.

Conclusion

The development EVPI can be used to decide whether a model can advance to validation, whether it should be abandoned, or whether a larger development sample is needed. Value-of-information methods can be applied to explore decision-theoretic consequences of uncertainty in risk prediction and can complement inferential methods in predictive analytics. R code for implementing this method is provided.

Developing clinical prediction models when adhering to minimum sample size recommendations: The importance of quantifying bootstrap variability in tuning parameters and predictive performance

Recent minimum sample size formula (Riley et al.) for developing clinical prediction models help ensure that development datasets are of sufficient size to minimise overfitting. While these criteria are known to avoid excessive overfitting on average, the extent of variability in overfitting at recommended sample sizes is unknown. We investigated this through a simulation study and empirical example to develop logistic regression clinical prediction models using unpenalised maximum likelihood estimation, and various post-estimation shrinkage or penalisation methods. While the mean calibration slope was close to the ideal value of one for all methods, penalisation further reduced the level of overfitting, on average, compared to unpenalised methods. This came at the cost of higher variability in predictive performance for penalisation methods in external data. We recommend that penalisation methods are used in data that meet, or surpass, minimum sample size requirements to further mitigate overfitting, and that the variability in predictive performance and any tuning parameters should always be examined as part of the model development process, since this provides additional information over average (optimism-adjusted) performance alone. Lower variability would give reassurance that the developed clinical prediction model will perform well in new individuals from the same population as was used for model development.

Clinical prediction rule for SARS-CoV-2 infection from 116 U.S. emergency departments 2-22-2021

Objectives

Accurate and reliable criteria to rapidly estimate the probability of infection with the novel coronavirus-2 that causes the severe acute respiratory syndrome (SARS-CoV-2) and associated disease (COVID-19) remain an urgent unmet need, especially in emergency care. The objective was to derive and validate a clinical prediction score for SARS-CoV-2 infection that uses simple criteria widely available at the point of care.

Methods

Data came from the registry data from the national REgistry of suspected COVID-19 in EmeRgency care (RECOVER network) comprising 116 hospitals from 25 states in the US. Clinical variables and 30-day outcomes were abstracted from medical records of 19,850 emergency department (ED) patients tested for SARS-CoV-2. The criterion standard for diagnosis of SARS-CoV-2 required a positive molecular test from a swabbed sample or positive antibody testing within 30 days. The prediction score was derived from a 50% random sample (n = 9,925) using unadjusted analysis of 107 candidate variables as a screening step, followed by stepwise forward logistic regression on 72 variables.

Results

Multivariable regression yielded a 13-variable score, which was simplified to a 13-point score: +1 point each for age>50 years, measured temperature>37.5°C, oxygen saturation<95%, Black race, Hispanic or Latino ethnicity, household contact with known or suspected COVID-19, patient reported history of dry cough, anosmia/dysgeusia, myalgias or fever; and -1 point each for White race, no direct contact with infected person, or smoking. In the validation sample (n = 9,975), the probability from logistic regression score produced an area under the receiver operating characteristic curve of 0.80 (95% CI: 0.79–0.81), and this level of accuracy was retained across patients enrolled from the early spring to summer of 2020. In the simplified score, a score of zero produced a sensitivity of 95.6% (94.8–96.3%), specificity of 20.0% (19.0–21.0%), negative likelihood ratio of 0.22 (0.19–0.26). Increasing points on the simplified score predicted higher probability of infection (e.g., >75% probability with +5 or more points).

Conclusion

Criteria that are available at the point of care can accurately predict the probability of SARS-CoV-2 infection. These criteria could assist with decisions about isolation and testing at high throughput checkpoints.