Impact of correlation of predictors on discrimination of risk models in development and external populations

External validation of six COVID-19 prognostic models for predicting mortality risk in older populations in a hospital, primary care, and nursing home setting

OBJECTIVES

To systematically evaluate the performance of COVID-19 prognostic models and scores for mortality risk in older populations across three health-care settings: hospitals, primary care, and nursing homes.

STUDY DESIGN AND SETTING

This retrospective external validation study included 14,092 older individuals of ≥70 years of age with a clinical or polymerase chain reaction-confirmed COVID-19 diagnosis from March 2020 to December 2020. The six validation cohorts include three hospital-based (CliniCo, COVID-OLD, COVID-PREDICT), two primary care-based (Julius General Practitioners Network/Academisch network huisartsgeneeskunde/Network of Academic general Practitioners, PHARMO), and one nursing home cohort (YSIS) in the Netherlands. Based on a living systematic review of COVID-19 prediction models using Prediction model Risk Of Bias ASsessment Tool for quality and risk of bias assessment and considering predictor availability in validation cohorts, we selected six prognostic models predicting mortality risk in adults with COVID-19 infection (GAL-COVID-19 mortality, 4C Mortality Score, National Early Warning Score 2-extended model, Xie model, Wang clinical model, and CURB65 score). All six prognostic models were validated in the hospital cohorts and the GAL-COVID-19 mortality model was validated in all three healthcare settings. The primary outcome was in-hospital mortality for hospitals and 28-day mortality for primary care and nursing home settings. Model performance was evaluated in each validation cohort separately in terms of discrimination, calibration, and decision curves. An intercept update was performed in models indicating miscalibration followed by predictive performance re-evaluation.

MAIN OUTCOME MEASURE

In-hospital mortality for hospitals and 28-day mortality for primary care and nursing home setting.

RESULTS

All six prognostic models performed poorly and showed miscalibration in the older population cohorts. In the hospital settings, model performance ranged from calibration-in-the-large -1.45 to 7.46, calibration slopes 0.24-0.81, and C-statistic 0.55-0.71 with 4C Mortality Score performing as the most discriminative and well-calibrated model. Performance across health-care settings was similar for the GAL-COVID-19 model, with a calibration-in-the-large in the range of -2.35 to -0.15 indicating overestimation, calibration slopes of 0.24-0.81 indicating signs of overfitting, and C-statistic of 0.55-0.71.

CONCLUSION

Our results show that most prognostic models for predicting mortality risk performed poorly in the older population with COVID-19, in each health-care setting: hospital, primary care, and nursing home settings. Insights into factors influencing predictive model performance in the older population are needed for pandemic preparedness and reliable prognostication of health-related outcomes in this demographic.

Development and Validation of Prognostic Models for Oral Squamous Cell Carcinoma: A Systematic Review and Appraisal of the Literature

(1) Background: An accurate prediction of cancer survival is very important for counseling, treatment planning, follow-up, and postoperative risk assessment in patients with Oral Squamous Cell Carcinoma (OSCC). There has been an increased interest in the development of clinical prognostic models and nomograms which are their graphic representation. The study aimed to revise the prognostic performance of clinical-pathological prognostic models with internal validation for OSCC. (2) Methods: This systematic review was performed according to the Cochrane Handbook for Diagnostic Test Accuracy Reviews chapter on searching, the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis) guidelines, and the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS). (3) Results: Six studies evaluating overall survival in patients with OSCC were identified. All studies performed internal validation, while only four models were externally validated. (4) Conclusions: Based on the results of this systematic review, it is possible to state that it is necessary to carry out internal validation and shrinkage to correct overfitting and provide an adequate performance for optimism. Moreover, calibration, discrimination and nonlinearity of continuous predictors should always be examined. To reduce the risk of bias the study design used should be prospective and imputation techniques should always be applied to handle missing data. In addition, the complete equation of the prognostic model must be reported to allow updating, external validation in a new context and the subsequent evaluation of the impact on health outcomes and on the cost-effectiveness of care.

Association, prediction, generalizability: Cross-center validity of predicting tooth loss in periodontitis patients

OBJECTIVES

To predict patients' tooth loss during supportive periodontal therapy across four German university centers.

METHODS

Tooth loss in 897 patients in four centers (Kiel (KI) n = 391; Greifswald (GW) n = 282; Heidelberg (HD) n = 175; Frankfurt/Main (F) n = 49) during supportive periodontal therapy (SPT) was assessed. Our outcome was annualized tooth loss per patient. Multivariable linear regression models were built on data of 75 % of patients from one center and used for predictions on the remaining 25 % of this center and 100 % of data from the other three centers. The prediction error was assessed as root-mean-squared-error (RMSE), i.e., the deviation of predicted from actually lost teeth per patient and year.

RESULTS

Annualized tooth loss/patient differed significantly between centers (between median 0.00 (interquartile interval: 0.00, 0.17) in GW and 0.09 (0.00, 0.19) in F, p = 0.001). Age, smoking status and number of teeth before SPT were significantly associated with tooth loss (p < 0.03). Prediction within centers showed RMSE of 0.14-0.30, and cross-center RMSE was 0.15-0.31. Predictions were more accurate in F and KI than in HD and GW, while the center on which the model was trained had a less consistent impact. No model showed useful predictive values.

CONCLUSION

While covariates were significantly associated with tooth loss in linear regression models, a clinically useful prediction was not possible with any of the models and generalizability was not given. Predictions were more accurate for certain centers.

CLINICAL RELEVANCE

Association should not be confused with predictive value: Despite significant associations of covariates with tooth loss, none of our models was useful for prediction. Usually, model accuracy was even lower when tested across centers, indicating low generalizability.

Early Tooth Loss after Periodontal Diagnosis: Development and Validation of a Clinical Decision Model

The aim of this study was to develop and validate a predictive early tooth loss multivariable model for periodontitis patients before periodontal treatment. A total of 544 patients seeking periodontal care at the university dental hospital were enrolled in the study. Teeth extracted after periodontal diagnosis and due to periodontal reasons were recorded. Clinical and sociodemographic variables were analyzed, considering the risk of short-term tooth loss. This study followed the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) guidelines for development and validation, with two cohorts considered as follows: 455 patients in the development phase and 99 in the validation phase. As a result, it was possible to compute a predictive model based on tooth type and clinical attachment loss. The model explained 25.3% of the total variability and correctly ranked 98.9% of the cases. The final reduced model area under the curve (AUC) was 0.809 (95% confidence interval (95% CI): 0.629–0.989) for the validation sample and 0.920 (95% CI: 0.891–0.950) for the development cohort. The established model presented adequate prediction potential of early tooth loss due to periodontitis. This model may have clinical and epidemiologic relevance towards the prediction of tooth loss burden.

A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes

It is widely recommended that any developed-diagnostic or prognostic-prediction model is externally validated in terms of its predictive performance measured by calibration and discrimination. When multiple validations have been performed, a systematic review followed by a formal meta-analysis helps to summarize overall performance across multiple settings, and reveals under which circumstances the model performs suboptimal (alternative poorer) and may need adjustment. We discuss how to undertake meta-analysis of the performance of prediction models with either a binary or a time-to-event outcome. We address how to deal with incomplete availability of study-specific results (performance estimates and their precision), and how to produce summary estimates of the c-statistic, the observed:expected ratio and the calibration slope. Furthermore, we discuss the implementation of frequentist and Bayesian meta-analysis methods, and propose novel empirically-based prior distributions to improve estimation of between-study heterogeneity in small samples. Finally, we illustrate all methods using two examples: meta-analysis of the predictive performance of EuroSCORE II and of the Framingham Risk Score. All examples and meta-analysis models have been implemented in our newly developed R package "metamisc".

Evaluating Modeling and Validation Strategies for Tooth Loss

Prediction models learn patterns from available data (training) and are then validated on new data (testing). Prediction modeling is increasingly common in dental research. We aimed to evaluate how different model development and validation steps affect the predictive performance of tooth loss prediction models of patients with periodontitis. Two independent cohorts (627 patients, 11,651 teeth) were followed over a mean ± SD 18.2 ± 5.6 y (Kiel cohort) and 6.6 ± 2.9 y (Greifswald cohort). Tooth loss and 10 patient- and tooth-level predictors were recorded. The impact of different model development and validation steps was evaluated: 1) model complexity (logistic regression, recursive partitioning, random forest, extreme gradient boosting), 2) sample size (full data set or 10%, 25%, or 75% of cases dropped at random), 3) prediction periods (maximum 10, 15, or 20 y or uncensored), and 4) validation schemes (internal or external by centers/time). Tooth loss was generally a rare event (880 teeth were lost). All models showed limited sensitivity but high specificity. Patients' age and tooth loss at baseline as well as probing pocket depths showed high variable importance. More complex models (random forest, extreme gradient boosting) had no consistent advantages over simpler ones (logistic regression, recursive partitioning). Internal validation (in sample) overestimated the predictive power (area under the curve up to 0.90), while external validation (out of sample) found lower areas under the curve (range 0.62 to 0.82). Reducing the sample size decreased the predictive power, particularly for more complex models. Censoring the prediction period had only limited impact. When the model was trained in one period and tested in another, model outcomes were similar to the base case, indicating temporal validation as a valid option. No model showed higher accuracy than the no-information rate. In conclusion, none of the developed models would be useful in a clinical setting, despite high accuracy. During modeling, rigorous development and external validation should be applied and reported accordingly.

Impact of predictor measurement heterogeneity across settings on the performance of prediction models: A measurement error perspective

It is widely acknowledged that the predictive performance of clinical prediction models should be studied in patients that were not part of the data in which the model was derived. Out‐of‐sample performance can be hampered when predictors are measured differently at derivation and external validation. This may occur, for instance, when predictors are measured using different measurement protocols or when tests are produced by different manufacturers. Although such heterogeneity in predictor measurement between derivation and validation data is common, the impact on the out‐of‐sample performance is not well studied. Using analytical and simulation approaches, we examined out‐of‐sample performance of prediction models under various scenarios of heterogeneous predictor measurement. These scenarios were defined and clarified using an established taxonomy of measurement error models. The results of our simulations indicate that predictor measurement heterogeneity can induce miscalibration of prediction and affects discrimination and overall predictive accuracy, to extents that the prediction model may no longer be considered clinically useful. The measurement error taxonomy was found to be helpful in identifying and predicting effects of heterogeneous predictor measurements between settings of prediction model derivation and validation. Our work indicates that homogeneity of measurement strategies across settings is of paramount importance in prediction research.

Development and Validation of Lifestyle-Based Models to Predict Incidence of the Most Common Potentially Preventable Cancers

BACKGROUND

Most risk models for cancer are either specific to individual cancers or include complex or predominantly non-modifiable risk factors.

METHODS

We developed lifestyle-based models for the five cancers for which the most cases are potentially preventable through lifestyle change in the UK (lung, colorectal, bladder, kidney, and esophageal for men and breast, lung, colorectal, endometrial, and kidney for women). We selected lifestyle risk factors from the European Code against Cancer and obtained estimates of relative risks from meta-analyses of observational studies. We used mean values for risk factors from nationally representative samples and mean 10-year estimated absolute risks from routinely available sources. We then assessed the performance of the models in 23,768 participants in the EPIC-Norfolk cohort who had no history of the five selected cancers at baseline.

RESULTS

In men, the combined risk model showed good discrimination [AUC, 0.71; 95% confidence interval (CI), 0.69-0.73] and calibration. Discrimination was lower in women (AUC, 0.59; 95% CI, 0.57-0.61), but calibration was good. In both sexes, the individual models for lung cancer had the highest AUCs (0.83; 95% CI, 0.80-0.85 for men and 0.82; 95% CI, 0.76-0.87 for women). The lowest AUCs were for breast cancer in women and kidney cancer in men.

CONCLUSIONS

The discrimination and calibration of the models are both reasonable, with the discrimination for individual cancers comparable or better than many other published risk models.

IMPACT

These models could be used to demonstrate the potential impact of lifestyle change on risk of cancer to promote behavior change.

Ten Essential Practices for Developing or Reforming a Biostatistics Core for a NCI Designated Cancer Center

There are 69 National Cancer Institute (NCI) designated Cancer Centers (CCs) in the United States. Biostatistical collaboration is pivotal in cancer research, and support for a cancer biostatistics shared resource facility (C-BSRF) is included in the award. Although the services and staff needed in a C-BSRF have been outlined in general terms and best practices for biostatistical consultations and collaboration in an academic health center have been agreed upon, implementing these practices in the demanding setting of cancer centers interested in pursuing or maintaining NCI designation remains challenging. We surveyed all C-BSRF websites to assess their organizational charts, governance, size, services provided, and financial models and have identified 10 essential practices for the development of a successful C-BSRF. Here, we share our success with, and barriers to, implementation of these practices. Showcasing development plans for these essential practices resulted in an NCI score of "Excellent to Outstanding" for our C-BSRF in 2015, and performance metrics in 2016-2017 demonstrated notable improvement since our original Cancer Center Support Grant (CCSG) application in 2014. We believe that the essential practices described here can be adapted and adjusted, as needed, for CCs of various sizes and with different types of cancer research programs.