Facilitating validation of prediction models: a comparison of manual and semi-automated validation using registry-based data of breast cancer patients in the Netherlands

Prediction of all-cause death and specific causes of death in patients with gastric cancer with liver metastasis: a Surveillance, Epidemiology, and End Results-based study

BACKGROUND

Gastric cancer (GC), considered the fifth most prevalent malignancy, is the fourth leading cause of cancer death worldwide. This cancer is heterogeneous and invasive and often metastasizes to the liver. The survival of patients with GC, especially cancer-specific survival (CSS), is a matter of concern to their families and medical workers in clinical practice. However, efficient tools for early risk prediction are lacking. Thus, this study aimed to develop a nomogram for forecasting the overall survival (OS) and CSS of patients with GC with liver metastasis (GCLM) based on the Surveillance, Epidemiology, and End Results (SEER) database.

METHODS

Information on individuals with GCLM was acquired from the SEER database from January 2000 to December 2015. Patients' data were randomized into the train cohort and the test cohort. The independent factors for CSS and OS were determined by univariate and multivariate competing risk analyses and Cox proportional hazards analysis, and the nomograms for predicting CSS and OS were constructed. The receiver operating characteristic curve and calibration curve were used to measure the accuracy and calibration of nomograms.

RESULTS

Our study included 4372 patients with GCLM, with 3060 patients in the train set and 1312 in the test set. The mean follow-up period was 12.31 months. The independent factors influencing the OS of patients with GCLM were age, bone metastasis, chemotherapy, grade, lung metastasis, stage, primary site, radiotherapy, surgical primary site, T stage, and tumor size. The concordance Index (C-index) of the constructed nomogram for OS were 0.718 (SE, 0.004) in the train set and 0.0.680 (SE, 0.006) in the test set. The independent factors affecting the CSS of patients with GCLM were age, chemotherapy, grade, lung metastasis, stage, radiotherapy, regional lymph node positive, surgical primary site, and total number of tumors. The C-index for the constructed nomogram for CSS were 0.696 (SE, 0.005) in the train set and 0.696 (SE, 0.008) in the test set.

CONCLUSION

The constructed nomograms showed satisfactory performance in predicting the OS and CSS of patients with GCLM, which can help clinicians formulate follow-up and rehabilitation strategies conducive to survival. At the same time, it can provide more family and social support for high-risk groups.

Development and validation of two nomograms for predicting overall survival and Cancer-specific survival in prostate cancer patients with bone metastases: a population-based study

BACKGROUND

Prostate cancer with bone metastasis has significant invasiveness and markedly poorer prognosis. The purpose of this study is to establish two nomograms for predicting the overall survival (OS) and cancer-specific survival (CSS) of prostate cancer patients with bone metastasis.

METHODS

From January 2000 to December 2018, a total of 2683 prostate adenocarcinoma with bone metastasis patients were identified from the Surveillance, Epidemiology, and End Results Program (SEER) database. These patients were then divided into a training cohort and a validation cohort, with OS and CSS as the study endpoints. Correlation analyses were employed to assess the relationship between variables. Univariate and multivariate Cox analyses were utilized to ascertain the independent prognostic factors. Calibration curves and the area under the time-dependent receiver operating characteristic curve (time-dependent AUC) were employed to evaluate discrimination and calibration of the nomogram. DCA was applied to examine accuracy and clinical benefits. The clinical utility of the nomogram and the AJCC Stage System was compared using net reclassification improvement (NRI) and integrated discrimination improvement (IDI). Lastly, the risk stratifications of the nomogram and the AJCC Stage System were compared.

RESULTS

There was no collinearity among the variables that were screened. The results of multivariate Cox regression analysis showed that seven variables (age, surgery, brain metastasis, liver metastasis, lung metastasis, Gleason score, marital status) and six variables (age, surgery, lung metastasis, liver metastasis, Gleason score, marital status) were identified to establish the nomogram for OS and CSS, respectively. The calibration curves, time-dependent AUC curves, and DCA revealed that both nomograms had pleasant predictive power. Furthermore, NRI and IDI confirmed that the nomogram outperformed the AJCC Stage System.

CONCLUSION

Both nomograms had satisfactory accuracy and were validated to assist clinicians in evaluating the prognosis of PABM patients.

Validation of Predictive Analyses for Interim Decisions in Clinical Trials

PURPOSE

Adaptive clinical trials use algorithms to predict, during the study, patient outcomes and final study results. These predictions trigger interim decisions, such as early discontinuation of the trial, and can change the course of the study. Poor selection of the Prediction Analyses and Interim Decisions (PAID) plan in an adaptive clinical trial can have negative consequences, including the risk of exposing patients to ineffective or toxic treatments.

METHODS

We present an approach that leverages data sets from completed trials to evaluate and compare candidate PAIDs using interpretable validation metrics. The goal is to determine whether and how to incorporate predictions into major interim decisions in a clinical trial. Candidate PAIDs can differ in several aspects, such as the prediction models used, timing of interim analyses, and potential use of external data sets. To illustrate our approach, we considered a randomized clinical trial in glioblastoma. The study design includes interim futility analyses on the basis of the predictive probability that the final analysis, at the completion of the study, will provide significant evidence of treatment effects. We examined various PAIDs with different levels of complexity to investigate if the use of biomarkers, external data, or novel algorithms improved interim decisions in the glioblastoma clinical trial.

RESULTS

Validation analyses on the basis of completed trials and electronic health records support the selection of algorithms, predictive models, and other aspects of PAIDs for use in adaptive clinical trials. By contrast, PAID evaluations on the basis of arbitrarily defined ad hoc simulation scenarios, which are not tailored to previous clinical data and experience, tend to overvalue complex prediction procedures and produce poor estimates of trial operating characteristics such as power and the number of enrolled patients.

CONCLUSION

Validation analyses on the basis of completed trials and real world data support the selection of predictive models, interim analysis rules, and other aspects of PAIDs in future clinical trials.

External Validation of Mortality Prediction Models for Critical Illness Reveals Preserved Discrimination but Poor Calibration

OBJECTIVES

In a recent scoping review, we identified 43 mortality prediction models for critically ill patients. We aimed to assess the performances of these models through external validation.

DESIGN

Multicenter study.

SETTING

External validation of models was performed in the Simple Intensive Care Studies-I (SICS-I) and the Finnish Acute Kidney Injury (FINNAKI) study.

PATIENTS

The SICS-I study consisted of 1,075 patients, and the FINNAKI study consisted of 2,901 critically ill patients.

MEASUREMENTS AND MAIN RESULTS

For each model, we assessed: 1) the original publications for the data needed for model reconstruction, 2) availability of the variables, 3) model performance in two independent cohorts, and 4) the effects of recalibration on model performance. The models were recalibrated using data of the SICS-I and subsequently validated using data of the FINNAKI study. We evaluated overall model performance using various indexes, including the (scaled) Brier score, discrimination (area under the curve of the receiver operating characteristics), calibration (intercepts and slopes), and decision curves. Eleven models (26%) could be externally validated. The Acute Physiology And Chronic Health Evaluation (APACHE) II, APACHE IV, Simplified Acute Physiology Score (SAPS)-Reduced (SAPS-R)' and Simplified Mortality Score for the ICU models showed the best scaled Brier scores of 0.11' 0.10' 0.10' and 0.06' respectively. SAPS II, APACHE II, and APACHE IV discriminated best; overall discrimination of models ranged from area under the curve of the receiver operating characteristics of 0.63 (0.61-0.66) to 0.83 (0.81-0.85). We observed poor calibration in most models, which improved to at least moderate after recalibration of intercepts and slopes. The decision curve showed a positive net benefit in the 0-60% threshold probability range for APACHE IV and SAPS-R.

CONCLUSIONS

In only 11 out of 43 available mortality prediction models, the performance could be studied using two cohorts of critically ill patients. External validation showed that the discriminative ability of APACHE II, APACHE IV, and SAPS II was acceptable to excellent, whereas calibration was poor.

External Validation of a Risk Model for Severe Complications following Pancreatoduodenectomy Based on Three Preoperative Variables

BACKGROUND

Pancreatoduodenectomy (PD) is the only cure for periampullary and pancreatic cancer. It has morbidity rates of 40-60%, with severe complications in 30%. Prediction models to predict complications are crucial. A risk model for severe complications was developed by Schroder et al. based on BMI, ASA classification and Hounsfield Units of the pancreatic body on the preoperative CT scan. These variables were independent predictors for severe complications upon internal validation. Our aim was to externally validate this model using an independent cohort of patients.

METHODS

A retrospective analysis was performed on 318 patients who underwent PD at our institution from 2013 to 2021. The outcome of interest was severe complications Clavien-Dindo ≥ IIIa. Model calibration, discrimination and performance were assessed.

RESULTS

A total of 308 patients were included. Patients with incomplete data were excluded. A total of 89 (28.9%) patients had severe complications. The externally validated model achieved: C-index = 0.67 (95% CI: 0.60-0.73), regression coefficient = 0.37, intercept = 0.13, Brier score = 0.25.

CONCLUSIONS

The performance ability, discriminative power, and calibration of this model were acceptable. Our risk calculator can help surgeons identify high-risk patients for post-operative complications to improve shared decision-making and tailor perioperative management.

Development and validation of two nomograms for predicting overall survival and cancer-specific survival in gastric cancer patients with liver metastases: A retrospective cohort study from SEER database

•

Nomograms constructed by overall survival and cancer-specific survival can predict more accurately than AJCC stage system for GCLM patients.

•

The study includes the prognostic factor as many as possible and evaluated all of them in the cohort.

•

In our cohort, surgery is a beneficial factor associated with survival.

Background

Gastric cancer is heterogeneous and aggressive, especially with liver metastasis. This study aims to develop two nomograms to predict the overall survival (OS) and cancer-specific survival (CSS) of gastric cancer with liver metastasis (GCLM) patients.

Methods

From January 2000 to December 2018, a total of 1936 GCLM patients were selected from the Surveillance, Epidemiology, and End Results Program (SEER) database. They were further divided into a training cohort and a validation cohort, with the OS and CSS serving as the study's endpoints. The correlation analyses were used to determine the relationship between the variables. The univariate and multivariate Cox analyses were used to confirm the independent prognostic factors. To discriminate and calibrate the nomogram, calibration curves and the area under the time-dependent receiver operating characteristic curve (time-dependent AUC) were used. DCA curves were used to examine the accuracy and clinical benefits. The clinical utility of the nomogram and the AJCC Stage System was compared using net reclassification improvement (NRI) and integrated differentiation improvement (IDI) (IDI). Finally, the nomogram and the AJCC Stage System risk stratifications were compared.

Results

There was no collinearity among the variables that were screened. The results of multivariate Cox regression analysis showed that six variables (bone metastasis, lung metastasis, surgery, chemotherapy, grade, age) and five variables (lung metastasis, surgery, chemotherapy, grade, N stage) were identified to establish the nomogram for OS and CSS, respectively. The calibration curves, time-dependent AUC curves, and DCA revealed that both nomograms had pleasant predictive power. Furthermore, NRI and IDI confirmed that the nomogram outperformed the AJCC Stage System.

Conclusion

Both nomograms had satisfactory accuracy and were validated to assist clinicians in evaluating the prognosis of GCLM patients.

Head-to-head comparison of 14 prediction models for postoperative delirium in elderly non-ICU patients: an external validation study

OBJECTIVES

Delirium is associated with increased morbidity, mortality, prolonged hospitalisation and increased healthcare costs. The number of clinical prediction models (CPM) to predict postoperative delirium has increased exponentially. Our goal is to perform a head-to-head comparison of CPMs predicting postoperative delirium in non-intensive care unit (non-ICU) elderly patients to identify the best performing models.

SETTING

Single-site university hospital.

DESIGN

Secondary analysis of prospective cohort study.

PARTICIPANTS AND INCLUSION

CPMs published within the timeframe of 1 January 1990 to 1 May 2020 were checked for eligibility (Preferred Reporting Items for Systematic Reviews and Meta-Analyses). For the time period of 1 January 1990 to 1 January 2017, included CPMs were identified in systematic reviews based on prespecified inclusion and exclusion criteria. An extended literature search for original studies was performed independently by two authors, including CPMs published between 1 January 2017 and 1 May 2020. External validation was performed using a surgical cohort consisting of 292 elderly non-ICU patients.

PRIMARY OUTCOME MEASURES

Discrimination, calibration and clinical usefulness.

RESULTS

14 CPMs were eligible for analysis out of 366 full texts reviewed. External validation was previously published for 8/14 (57%) CPMs. C-indices ranged from 0.52 to 0.74, intercepts from -0.02 to 0.34, slopes from -0.74 to 1.96 and scaled Brier from -1.29 to 0.088. Based on predefined criteria, the two best performing models were those of Dai et al (c-index: 0.739; (95% CI: 0.664 to 0.813); intercept: -0.018; slope: 1.96; scaled Brier: 0.049) and Litaker et al (c-index: 0.706 (95% CI: 0.590 to 0.823); intercept: -0.015; slope: 0.995; scaled Brier: 0.088). For the remaining CPMs, model discrimination was considered poor with corresponding c-indices <0.70.

CONCLUSION

Our head-to-head analysis identified 2 out of 14 CPMs as best-performing models with a fair discrimination and acceptable calibration. Based on our findings, these models might assist physicians in postoperative delirium risk estimation and patient selection for preventive measures.

A scoping review of interactive and personalized web-based clinical tools to support treatment decision making in breast cancer

The increasing attention on personalized breast cancer care has resulted in an explosion of new interactive, tailored, web-based clinical decision tools for guiding treatment decisions in clinical practice. The goal of this study was to review, compare, and discuss the clinical implications of current tools, and highlight future directions for tools aiming to improve personalized breast cancer care. We searched PubMed, Embase, PsychInfo, Cochrane Database of Systematic Reviews, Web of Science, and Scopus to identify web-based decision tools addressing breast cancer treatment decisions. There was a total of 17 articles associated with 21 unique tools supporting decisions related to surgery, radiation therapy, hormonal therapy, bisphosphonates, HER2-targeted therapy, and chemotherapy. The quality of the tools was assessed using the International Patient Decision Aid Standard instrument. Overall, the tools considered clinical (e.g., age) and tumor characteristics (e.g., grade) to provide personalized outcomes (e.g., survival) associated with various treatment options. Fewer tools provided the adverse effects of the selected treatment. Only one tool was field-tested with patients, and none were tested with healthcare providers. Future studies need to assess the feasibility, usability, acceptability, as well as the effects of personalized web-based decision tools on communication and decision making from the patient and clinician perspectives.

Nomogram Predicting Cancer-Specific Death in Parotid Carcinoma: a Competing Risk Analysis

Purpose

Multiple factors have been shown to be tied to the prognosis of individuals with parotid cancer (PC); however, there are limited numbers of reliable as well as straightforward tools available for clinical estimation of individualized mortality. Here, a competing risk nomogram was established to assess the risk of cancer-specific deaths (CSD) in individuals with PC.

Methods

Data of PC patients analyzed in this work were retrieved from the Surveillance, Epidemiology, and End Results (SEER) data repository and the First Affiliated Hospital of Nanchang University (China). Univariate Lasso regression coupled with multivariate Cox assessments were adopted to explore the predictive factors influencing CSD. The cumulative incidence function (CIF) coupled with the Fine-Gray proportional hazards model was employed to determine the risk indicators tied to CSD as per the univariate, as well as multivariate analyses conducted in the R software. Finally, we created and validated a nomogram to forecast the 3- and 5-year CSD likelihood.

Results

Overall, 1,467 PC patients were identified from the SEER data repository, with the 3- and 5-year CSD CIF after diagnosis being 21.4% and 24.1%, respectively. The univariate along with the Lasso regression data revealed that nine independent risk factors were tied to CSD in the test dataset (n = 1,035) retrieved from the SEER data repository. Additionally, multivariate data of Fine-Gray proportional subdistribution hazards model illustrated that N stage, Age, T stage, Histologic, M stage, grade, surgery, and radiation were independent risk factors influencing CSD in an individual with PC in the test dataset (p < 0.05). Based on optimization performed using the Bayesian information criterion (BIC), six variables were incorporated in the prognostic nomogram. In the internal SEER data repository verification dataset (n = 432) and the external medical center verification dataset (n = 473), our nomogram was well calibrated and exhibited considerable estimation efficiency.

Conclusion

The competing risk nomogram presented here can be used for assessing cancer-specific mortality in PC patients.

Validation and comparison of 28 risk prediction models for coronary artery disease

AIMS

Risk prediction models (RPMs) for coronary artery disease (CAD), using variables to calculate CAD risk, are potentially valuable tools in prevention strategies. However, their use in the clinical practice is limited by a lack of poor model description, external validation, and head-to-head comparisons.

METHODS AND RESULTS

CAD RPMs were identified through Tufts PACE CPM Registry and a systematic PubMed search. Every RPM was externally validated in the three cohorts (the UK Biobank, LifeLines, and PREVEND studies) for the primary endpoint myocardial infarction (MI) and secondary endpoint CAD, consisting of MI, percutaneous coronary intervention, and coronary artery bypass grafting. Model discrimination (C-index), calibration (intercept and regression slope), and accuracy (Brier score) were assessed and compared head-to-head between RPMs. Linear regression analysis was performed to evaluate predictive factors to estimate calibration ability of an RPM. Eleven articles containing 28 CAD RPMs were included. No single best-performing RPM could be identified across all cohorts and outcomes. Most RPMs yielded fair discrimination ability: mean C-index of RPMs was 0.706 ± 0.049, 0.778 ± 0.097, and 0.729 ± 0.074 (P < 0.01) for prediction of MI in UK Biobank, LifeLines, and PREVEND, respectively. Endpoint incidence in the original development cohorts was identified as a significant predictor for external validation performance.

CONCLUSION

Performance of CAD RPMs was comparable upon validation in three large cohorts, based on which no specific RPM can be recommended for predicting CAD risk.

Assessment of predictive performance of caries risk assessment models based on a systematic review and meta-analysis

OBJECTIVES

To assess the predictive performance of caries risk assessment (CRA) models for prediction of caries increment for individuals based on a systematic review and meta-analyses.

DATA/SOURCES

We included external validation studies assessing the predictive performance of CRA models for prediction of caries increment for individuals, using discrimination and calibration as the outcome parameters. PubMed, EMBASE, and CINAHL were searched electronically on 10th September 2020 to identify prediction modeling studies on external validation of CRA models. The risk of bias of the included studies was assessed using the Prediction model Risk Of Bias ASsessment Tool (PROBAST).

STUDY SELECTION

A total of 22 studies with seven different CRA models were included. As for full Cariogram, the pooled area under the receiver operating characteristic curve (AUC) was 0.78 (95 %CI: 0.68; 0.85) based on eight studies regardless of the risk of bias levels, and 0.82 (95 %CI: 0.58; 0.93) based on four studies with low risk of bias only. The pooled observed: expected ratio (O:E ratio) of full Cariogram was 0.91 (95 %CI: 0.72; 1.14) based on 12 studies regardless of the risk of bias levels, and 0.89 (95 %CI: 0.71; 1.12) based on five studies with low risk of bias only. As for reduced Cariogram, the pooled AUC was 0.72 (95 %CI: 0.67; 0.77) based on six studies regardless of the risk of bias levels, and 0.74 (95 %CI: 0.45; 0.91) based on two studies with low risk of bias only. The pooled O:E ratio of reduced Cariogram was 0.84 (95 %CI: 0.59; 1.18) based on six studies regardless of the risk of bias levels, and 1.05 (95 %CI: 0.43; 2.59) based on two studies with low risk of bias only. Based on an insufficient number of studies for the other CRA models, the pooled AUCs ranged from 0.50 to 0.88, while the pooled O:E ratio ranged from 0.38 to 1.00.

CONCLUSION

The average predictive performance of both full and reduced Cariogram seems to be acceptable. However, the evidence from research does not allow a firm conclusion on the performance of the other included CRA models, due to the insufficient number of high-quality studies.

CLINICAL SIGNIFICANCE

Both full and reduced Cariogram were found to be reliable CRA models for prediction of caries increment in clinical practices for dental patients and communities for general populations. The reduced Cariogram showed better predictive performance and less burden in terms of time and resources to individuals than the full Cariogram. Therefore, the reduced Cariogram could be more recommended than the full Cariogram.

Validation of a Prognostic Multivariable Prediction Model for Insufficient Clinical Response to Methotrexate in Early Rheumatoid Arthritis and Its Clinical Application in Evidencio

INTRODUCTION

Methotrexate (MTX) constitutes the first-line therapy in rheumatoid arthritis (RA), yet approximately 30% of the patients do not benefit from MTX. Recently, we reported a prognostic multivariable prediction model for insufficient clinical response to MTX at 3 months of treatment in the treatment in the Rotterdam Early Arthritis Cohort (tREACH), including baseline predictors: Disease activity score 28 (DAS28), Health Assessment Questionnaire (HAQ), erythrocyte folate, single-nucleotide polymorphisms (SNPs; ABCB1, ABCC3), smoking, and BMI. The purpose of the current study was (1) to externally validate the model and (2) to enhance the model's clinical applicability.

METHODS

Erythrocyte folate and SNPs were assessed in 91 early disease-modifying antirheumatic drug (DMARD)-naïve RA patients starting MTX in the external validation cohort (U-Act-Early). Insufficient response (DAS28 > 3.2) was determined after 3 months and non-response after 6 months of therapy. The previously developed prediction model was considered successfully validated in the U-Act-Early (validation cohort) if the area under the curve (AUC) of the receiver operating characteristic (ROC) was not significantly lower than in the tREACH (derivation cohort).

RESULTS

The AUCs in U-Act-Early at three and 6 months were 0.75 (95% CI 0.64-0.85) and 0.71 (95% CI 0.60-0.82) respectively, similar to the tREACH. Baseline DAS28 > 5.1 and HAQ > 0.6 were the strongest predictors. The model was simplified by excluding the SNPs, while still classifying 73% correctly. Furthermore, interaction terms between BMI and HAQ and BMI and erythrocyte folate significantly improved the model increasing correct classification to 75%. Results were successfully implemented in Evidencio online platform assisting clinicians in shared decision-making to intensify treatment when appropriate.

CONCLUSIONS

We successfully externally validated our recently reported prediction model for MTX non-response and enhanced its clinical application thus enabling its evaluation in a clinical trial.

TRIAL REGISTRATION

The U-Act-Early is registered at ClinicalTrials.gov. number: NCT01034137. tREACH is registered retrospectively at ISRCTN registry, number: ISRCTN26791028 at 23 August 2007.