Statistical methods to assess the prognostic value of risk prediction rules in clinical research

Survival Risk Score for Invasive Nonmetastatic Breast Cancer: A Real-World Analysis

PURPOSE

This study aimed to develop a multivariable, weighted overall survival (OS) risk score (SRS) for nonmetastatic (M0) invasive breast cancer (M0-BC, SRSM0-BC).

MATERIALS AND METHODS

This study included a training (1,890 patients) and a validation cohort (850 patients) from the Reggio Emilia Cancer Registry (RE-CR). Ten traditional prognostic variables were evaluated.

RESULTS

In the training set, all the variables but the human epidermal growth factor receptor were significantly associated with OS at univariable analysis. A multivariable model identified an increased death risk for estrogen receptor (hazard ratio [HR], 2.0 [95% CI, 1.1 to 3.1]; P = .021), tumor stages T2-T3 (HR, 2.4 [95% CI, 1.3 to 4.7]; P = .009) and T4 (HR, 5.1 [95% CI, 2.0 to 13.0]; P < .001), and age >74 years (HR, 5.7 [95% CI, 4.0 to 8.2]; P < .001). By assigning scores according to HRs, four risk categories were generated (P for trend <.001). The HRs of death in the high- (282 patients, 15.6%), intermediate-high (275 patients, 15.2%), and intermediate-risk (349 patients, 19.2%) categories patients were, respectively, 27.3, 12.9, and 3.5 times higher, compared with the low-risk (909 patients, 50%) group. Harrell'C index was 81.1%, and the explained variation in mortality was 66.6. Internal cross-validation performed on the accrual index dates yielded a Harrell'C index ranging from 79.5% to 82.3% and an explained variation in mortality ranging from 60.3% to 69.4%. In the validation set, the same risk categories (P for trend <.001) were devised. The Harrell'C index and the explained variation in mortality were 76.1% and 53.7%, respectively, in the whole cohort, maintaining an elevated percentage according to the two accrual index dates.

CONCLUSION

SRSM0-BC using the real-world RE-CR data set may represent a low-cost, accessible, globally applicable model in daily clinical practice, helping to prognostically stratify patients with invasive M0-BC.

Integrating Machine Learning into Statistical Methods in Disease Risk Prediction Modeling: A Systematic Review

Background: Disease prediction models often use statistical methods or machine learning, both with their own corresponding application scenarios, raising the risk of errors when used alone. Integrating machine learning into statistical methods may yield robust prediction models. This systematic review aims to comprehensively assess current development of global disease prediction integration models. Methods: PubMed, EMbase, Web of Science, CNKI, VIP, WanFang, and SinoMed databases were searched to collect studies on prediction models integrating machine learning into statistical methods from database inception to 2023 May 1. Information including basic characteristics of studies, integrating approaches, application scenarios, modeling details, and model performance was extracted. Results: A total of 20 eligible studies in English and 1 in Chinese were included. Five studies concentrated on diagnostic models, while 16 studies concentrated on predicting disease occurrence or prognosis. Integrating strategies of classification models included majority voting, weighted voting, stacking, and model selection (when statistical methods and machine learning disagreed). Regression models adopted strategies including simple statistics, weighted statistics, and stacking. AUROC of integration models surpassed 0.75 and performed better than statistical methods and machine learning in most studies. Stacking was used for situations with >100 predictors and needed relatively larger amount of training data. Conclusion: Research on integrating machine learning into statistical methods in prediction models remains limited, but some studies have exhibited great potential that integration models outperform single models. This study provides insights for the selection of integration methods for different scenarios. Future research could emphasize on the improvement and validation of integrating strategies.

Performance of the Dutch clinical prediction rule for the ambulation outcome after spinal cord injury in a middle-income country clinical setting: an external validation study in the Thai retrospective cohort

OBJECTIVE

To perform external geographic and domain validation of the clinical prediction rule (CPR) of the ambulation outcome of patients with traumatic spinal cord injury (SCI) originally developed by van Middendorp, et al. (2011) in Thais with traumatic and non-traumatic SCI.

STUDY DESIGN

Retrospective cohort study.

SETTING

A tertiary rehabilitation facility in Chiang Mai, Thailand.

METHODS

A validation data set, including predictive (age and four neurological variables) and outcome (ambulation status) parameters was retrospectively collected from medical records of patients with traumatic and non-traumatic SCI admitted between December 2007 and December 2019. The performance of the original model was evaluated in both discrimination and calibration aspects, using an area under the receiver-operating characteristic curve (auROC) and calibration curves, respectively.

RESULTS

Three hundred and thirty-three patients with SCI were included in the validation set. The prevalence of ambulators was 59% (197 of 333 participants). An auROC of 0.93 (95% CI 0.90-0.96) indicated excellent discrimination whereas the calibration curve demonstrated underestimation, especially in patients with AIS grade D. Performance of the CPR was decreased but acceptable in patients with non-traumatic SCI.

CONCLUSIONS

Our external validation study demonstrated excellent discrimination but slightly underestimated calibration of the CPR of ambulation outcome after SCI. Regardless of the geographic and etiologic background of the population, the Dutch CPR could be applied to predict the ambulation outcome in patients with SCI.

Lymphocyte Doubling Time As A Key Prognostic Factor To Predict Time To First Treatment In Early-Stage Chronic Lymphocytic Leukemia

The prognostic role of lymphocyte doubling time (LDT) in chronic lymphocytic leukemia (CLL) was recognized more than three decades ago when the neoplastic clone’s biology was almost unknown. LDT was defined as the time needed for the peripheral blood lymphocyte count to double the of the initial observed value. Herein, the LDT prognostic value for time to first treatment (TTFT) was explored in our prospective O-CLL cohort and validated in in two additional CLL cohorts. Specifically, newly diagnosed Binet stage A CLL patients from 40 Italian Institutions, representative of the whole country, were prospectively enrolled into the O-CLL1-GISL protocol (clinicaltrial.gov identifier: NCT00917540). Two independent cohorts of newly diagnosed CLL patients recruited respectively at the Division of Hematology in Novara, Italy, and at the Hospital Clinic in Barcelona, Spain, were utilized as validation cohorts. In the training cohort, TTFT of patients with LDT >12 months was significantly longer related to those with a shorter LDT. At Cox multivariate regression model, LDT ≤ 12 months maintained a significant independent relationship with shorter TTFT along with IGHV unmutated (IGHVunmut) status, 11q and 17p deletions, elevated β2M, Rai stage I-II, and NOTCH1 mutations. Based on these statistics, two regression models were constructed including the same prognostic factors with or without the LDT. The model with the LTD provided a significantly better data fitting (χ² = 8.25, P=0.0041). The risk prediction developed including LDT had better prognostic accuracy than those without LDT. Moreover, the Harrell’C index for the scores including LDT were higher than those without LDT, although the accepted 0.70 threshold exceeded in both cases. These findings were also confirmed when the same analysis was carried out according to TTFT’s explained variation. When data were further analyzed based on the combination between LDT and IGHV mutational status in the training and validation cohorts, IGHVunmut and LDT>12months group showed a predominant prognostic role over IGHVmut LTD ≤ 12 months (P=0.006) in the O-CLL validation cohort. However, this predominance was of borden-line significance (P=0.06) in the Barcelona group, while the significant prognostic impact was definitely lost in the Novara group. Overall, in this study, we demonstrated that LDT could be re-utilized together with the more sophisticated prognostic factors to manage the follow-up plans for Binet stage A CLL patients.