Validation of prediction models in the presence of competing risks: a guide through modern methods

A tool to predict the risk of lower extremity amputation in patients starting dialysis

BACKGROUND

Non-traumatic lower extremity amputation (LEA) is a severe complication during dialysis. To inform decision-making for physicians, we developed a multivariable prediction model for LEA after starting dialysis.

METHODS

Data from the Swedish Renal Registry (SNR) between 2010 and 2020 were geographically split into a development and validation cohort. Data from Netherlands Cooperative Study on the Adequacy of Dialysis (NECOSAD) between 1997 and 2009 were used for validation targeted at Dutch patients. Inclusion criteria were no previous LEA and kidney transplant and age ≥40 years at baseline. A Fine-Gray model was developed with LEA within 3 years after starting dialysis as the outcome of interest. Death and kidney transplant were treated as competing events. One coefficient, ordered by expected relevance, per 20 events was estimated. Performance was assessed with calibration and discrimination.

RESULTS

SNR was split into an urban development cohort with 4771 individuals experiencing 201 (4.8%) events and a rural validation cohort with 4.876 individuals experiencing 155 (3.2%) events. NECOSAD contained 1658 individuals experiencing 61 (3.7%) events. Ten predictors were included: female sex, age, diabetes mellitus, peripheral artery disease, cardiovascular disease, congestive heart failure, obesity, albumin, haemoglobin and diabetic retinopathy. In SNR, calibration intercept and slope were -0.003 and 0.912, respectively. The C-index was estimated as 0.813 (0.783-0.843). In NECOSAD, calibration intercept and slope were 0.001 and 1.142 respectively. The C-index was estimated as 0.760 (0.697-0.824). Calibration plots showed good calibration.

CONCLUSION

A newly developed model to predict LEA after starting dialysis showed good discriminatory performance and calibration. By identifying high-risk individuals this model could help select patients for preventive measures.

Validation of the BOADICEA model for epithelial tubo-ovarian cancer risk prediction in UK Biobank

BACKGROUND

The clinical validity of the multifactorial BOADICEA model for epithelial tubo-ovarian cancer (EOC) risk prediction has not been assessed in a large sample size or over a longer term.

METHODS

We evaluated the model discrimination and calibration in the UK Biobank cohort comprising 199,429 women (733 incident EOCs) of European ancestry without previous cancer history. We predicted 10-year EOC risk incorporating data on questionnaire-based risk factors (QRFs), family history, a 36-SNP polygenic risk score and pathogenic variants (PV) in six EOC susceptibility genes (BRCA1, BRCA2, RAD51C, RAD51D, BRIP1 and PALB2).

RESULTS

Discriminative ability was maximised under the multifactorial model that included all risk factors (AUC = 0.68, 95% CI: 0.66-0.70). This model was well calibrated in deciles of predicted risk with calibration slope=0.99 (95% CI: 0.98-1.01). Discriminative ability was similar in women younger or older than 60 years. The AUC was higher when analyses were restricted to PV carriers (0.76, 95% CI: 0.69-0.82). Using relative risk (RR) thresholds, the full model classified 97.7%, 1.7%, 0.4% and 0.2% women in the RR < 2.0, 2.0 ≤ RR < 2.9, 2.9 ≤ RR < 6.0 and RR ≥ 6.0 categories, respectively, identifying 9.1 of incident EOC among those with RR ≥ 2.0.

DISCUSSION

BOADICEA, implemented in CanRisk ( www.canrisk.org ), provides valid 10-year EOC risks and can facilitate clinical decision-making in EOC risk management.

Machine Learning Did Not Outperform Conventional Competing Risk Modeling to Predict Revision Arthroplasty

BACKGROUND

Estimating the risk of revision after arthroplasty could inform patient and surgeon decision-making. However, there is a lack of well-performing prediction models assisting in this task, which may be due to current conventional modeling approaches such as traditional survivorship estimators (such as Kaplan-Meier) or competing risk estimators. Recent advances in machine learning survival analysis might improve decision support tools in this setting. Therefore, this study aimed to assess the performance of machine learning compared with that of conventional modeling to predict revision after arthroplasty.

QUESTION/PURPOSE

Does machine learning perform better than traditional regression models for estimating the risk of revision for patients undergoing hip or knee arthroplasty?

METHODS

Eleven datasets from published studies from the Dutch Arthroplasty Register reporting on factors associated with revision or survival after partial or total knee and hip arthroplasty between 2018 and 2022 were included in our study. The 11 datasets were observational registry studies, with a sample size ranging from 3038 to 218,214 procedures. We developed a set of time-to-event models for each dataset, leading to 11 comparisons. A set of predictors (factors associated with revision surgery) was identified based on the variables that were selected in the included studies. We assessed the predictive performance of two state-of-the-art statistical time-to-event models for 1-, 2-, and 3-year follow-up: a Fine and Gray model (which models the cumulative incidence of revision) and a cause-specific Cox model (which models the hazard of revision). These were compared with a machine-learning approach (a random survival forest model, which is a decision tree-based machine-learning algorithm for time-to-event analysis). Performance was assessed according to discriminative ability (time-dependent area under the receiver operating curve), calibration (slope and intercept), and overall prediction error (scaled Brier score). Discrimination, known as the area under the receiver operating characteristic curve, measures the model's ability to distinguish patients who achieved the outcomes from those who did not and ranges from 0.5 to 1.0, with 1.0 indicating the highest discrimination score and 0.50 the lowest. Calibration plots the predicted versus the observed probabilities; a perfect plot has an intercept of 0 and a slope of 1. The Brier score calculates a composite of discrimination and calibration, with 0 indicating perfect prediction and 1 the poorest. A scaled version of the Brier score, 1 - (model Brier score/null model Brier score), can be interpreted as the amount of overall prediction error.

RESULTS

Using machine learning survivorship analysis, we found no differences between the competing risks estimator and traditional regression models for patients undergoing arthroplasty in terms of discriminative ability (patients who received a revision compared with those who did not). We found no consistent differences between the validated performance (time-dependent area under the receiver operating characteristic curve) of different modeling approaches because these values ranged between -0.04 and 0.03 across the 11 datasets (the time-dependent area under the receiver operating characteristic curve of the models across 11 datasets ranged between 0.52 to 0.68). In addition, the calibration metrics and scaled Brier scores produced comparable estimates, showing no advantage of machine learning over traditional regression models.

CONCLUSION

Machine learning did not outperform traditional regression models.

CLINICAL RELEVANCE

Neither machine learning modeling nor traditional regression methods were sufficiently accurate in order to offer prognostic information when predicting revision arthroplasty. The benefit of these modeling approaches may be limited in this context.

Impact of age, comorbidities and relevant changes on surveillance strategy of intraductal papillary mucinous neoplasms: a competing risk analysis

OBJECTIVE

Cost-effectiveness of surveillance for branch-duct intraductal papillary mucinous neoplasms (BD-IPMNs) is debated. We combined different categories of risks of IPMN progression and of IPMN-unrelated mortality to improve surveillance strategies.

DESIGN

Retrospective analysis of 926 presumed BD-IPMNs lacking worrisome features (WFs)/high-risk stigmata (HRS) under surveillance. Charlson Comorbidity Index (CACI) defined the severity of comorbidities. IPMN relevant changes included development of WF/HRS, pancreatectomy or death for IPMN or pancreatic cancer. Pancreatic malignancy-unrelated death was recorded. Cumulative incidence of IPMN relevant changes were estimated using the competing risk approach.

RESULTS

5-year cumulative incidence of relevant changes was 17.83% and 1.6% developed pancreatic malignancy. 5-year cumulative incidences for IPMN relevant changes were 13.73%, 19.93% and 25.04% in low-risk, intermediate-risk and high-risk groups, respectively. Age ≥75 (HR: 4.15) and CACI >3 (HR: 3.61) were independent predictors of pancreatic malignancy-unrelated death. 5-year cumulative incidence for death for other causes was 15.93% for age ≥75+CACI >3 group and 1.49% for age <75+CACI ≤3. 5-year cumulative incidence of IPMN relevant changes were 13.94% in patients with age <75+CACI ≤3 compared with 29.60% in those with age ≥75+CACI >3. In this group 5-year rate of malignancy-free patients was 95.56% with a 5-year survival of 79.51%.

CONCLUSION

Although it is not uncommon the occurrence of changes considered by current guidelines as relevant during surveillance of low risk BD-IPMNs, malignancy rate is low and survival is significantly affected by competing patients' age and comorbidities. IPMN surveillance strategy should be tailored based on these features and modulated over time.

Time-varying effect in older patients with early-stage breast cancer: a model considering the competing risks based on a time scale

Background

Patients with early-stage breast cancer may have a higher risk of dying from other diseases, making a competing risks model more appropriate. Considering subdistribution hazard ratio, which is used often, limited to model assumptions and clinical interpretation, we aimed to quantify the effects of prognostic factors by an absolute indicator, the difference in restricted mean time lost (RMTL), which is more intuitive. Additionally, prognostic factors of breast cancer may have dynamic effects (time-varying effects) in long-term follow-up. However, existing competing risks regression models only provide a static view of covariate effects, leading to a distorted assessment of the prognostic factor.

Methods

To address this issue, we proposed a dynamic effect RMTL regression that can explore the between-group cumulative difference in mean life lost over a period of time and obtain the real-time effect by the speed of accumulation, as well as personalized predictions on a time scale.

Results

A simulation validated the accuracy of the coefficient estimates in the proposed regression. Applying this model to an older early-stage breast cancer cohort, it was found that 1) the protective effects of positive estrogen receptor and chemotherapy decreased over time; 2) the protective effect of breast-conserving surgery increased over time; and 3) the deleterious effects of stage T2, stage N2, and histologic grade II cancer increased over time. Moreover, from the view of prediction, the mean C-index in external validation reached 0.78.

Conclusion

Dynamic effect RMTL regression can analyze both dynamic cumulative effects and real-time effects of covariates, providing a more comprehensive prognosis and better prediction when competing risks exist.

Hepatocellular Carcinoma in Metabolic Dysfunction-Associated Steatotic Liver Disease

Importance

In the US, hepatocellular carcinoma (HCC) has been the most rapidly increasing cancer since 1980, and metabolic dysfunction-associated steatotic liver disease (MASLD) is expected to soon become the leading cause of HCC.

Objective

To develop a prediction model for HCC incidence in a cohort of patients with MASLD.

Design, Setting, and Participants

This prognostic study was conducted among patients aged at least 18 years with MASLD, identified using diagnosis of MASLD using International Classification of Diseases, Ninth Revision (ICD-9) or International Statistical Classification of Diseases and Related Health Problems, Tenth Revision (ICD-10) diagnosis codes; natural language processing of radiology imaging report text, which identified patients who had imaging evidence of MASLD but had not been formally diagnosed; or the Dallas Steatosis Index, a risk equation that identifies individuals likely to have MASLD with good precision. Patients were enrolled from Kaiser Permanente Northern California, an integrated health delivery system with more than 4.6 million members, with study entry between January 2009 and December 2018, and follow-up until HCC development, death, or study termination on September 30, 2021. Statistical analysis was performed during February 2023 and January 2024.

Exposure

Data were extracted from the electronic health record and included 18 routinely measured factors associated with MASLD.

Main Outcome and Measures

The cohort was split (70:30) into derivation and internal validation sets; extreme gradient boosting was used to model HCC incidence. HCC risk was divided into 3 categories, with the cumulative estimated probability of HCC 0.05% or less classified as low risk; 0.05% to 0.09%, medium risk; and 0.1% or greater, high risk.

Results

A total of 1 811 461 patients (median age [IQR] at baseline, 52 [41-63] years; 982 300 [54.2%] female) participated in the study. During a median (range) follow-up of 9.3 (5.8-12.4) years, 946 patients developed HCC, for an incidence rate of 0.065 per 1000 person-years. The model achieved an area under the curve of 0.899 (95% CI, 0.882-0.916) in the validation set. At the medium-risk threshold, the model had a sensitivity of 87.5%, specificity of 81.4%, and a number needed to screen of 406. At the high-risk threshold, the model had a sensitivity of 78.4%, a specificity of 90.1%, and a number needed to screen of 241.

Conclusions and Relevance

This prognostic study of more than 1.8 million patients with MASLD used electronic health record data to develop a prediction model to discriminate between individuals with and without incident HCC with good precision. This model could serve as a starting point to identify patients with MASLD who may need intervention and/or HCC surveillance.

Prognostic Models for Chronic Low Back Pain Outcomes in Primary Care Are at High Risk of Bias and Lack Validation-High-Quality Studies Are Needed: A Systematic Review

OBJECTIVE: To provide an updated overview of available prognostic models for people with chronic low back pain (LBP) in primary care. DESIGN: Prognosis systematic review LITERATURE SEARCH: We searched for relevant studies on MEDLINE, Embase, Web of Science, and CINAHL databases (up to July 13, 2022), and performed citation tracking in Web of Science. STUDY SELECTION CRITERIA: We included observational (cohort or nested case-control) studies and randomized controlled trials that developed or validated prognostic models for adults with chronic LBP in primary care. The outcomes of interest were physical functioning, pain intensity, and health-related quality of life at any follow-up time-point. DATA SYNTHESIS: Data were extracted using the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS), and the Prediction model Risk of Bias Assessment Tool (PROBAST) tool was used to evaluate the risk of bias of the models. Due to the number of studies retrieved and the heterogeneity, we reported the results descriptively. RESULTS: Ten studies (out of 5593 hits screened) with 34 models met our inclusion criteria, of which six are development studies and four are external validation studies. Five studies reported the area under the curve of the models (ranging from 0.48 to 0.84), whereas no study reported calibration indices. The most promising model is the Örebro Musculoskeletal Pain Screening Questionnaire Short-Form. CONCLUSIONS: Given the high risk of bias and lack of external validation, we cannot recommend that clinicians use prognostic models for patients with chronic LBP in primary care settings. J Orthop Sports Phys Ther 2024;54(5):1-13. Epub 15 February 2024. doi:10.2519/jospt.2024.12081.

Development and validation of a new algorithm for improved cardiovascular risk prediction

QRISK algorithms use data from millions of people to help clinicians identify individuals at high risk of cardiovascular disease (CVD). Here, we derive and externally validate a new algorithm, which we have named QR4, that incorporates novel risk factors to estimate 10-year CVD risk separately for men and women. Health data from 9.98 million and 6.79 million adults from the United Kingdom were used for derivation and validation of the algorithm, respectively. Cause-specific Cox models were used to develop models to predict CVD risk, and the performance of QR4 was compared with version 3 of QRISK, Systematic Coronary Risk Evaluation 2 (SCORE2) and atherosclerotic cardiovascular disease (ASCVD) risk scores. We identified seven novel risk factors in models for both men and women (brain cancer, lung cancer, Down syndrome, blood cancer, chronic obstructive pulmonary disease, oral cancer and learning disability) and two additional novel risk factors in women (pre-eclampsia and postnatal depression). On external validation, QR4 had a higher C statistic than QRISK3 in both women (0.835 (95% confidence interval (CI), 0.833-0.837) and 0.831 (95% CI, 0.829-0.832) for QR4 and QRISK3, respectively) and men (0.814 (95% CI, 0.812-0.816) and 0.812 (95% CI, 0.810-0.814) for QR4 and QRISK3, respectively). QR4 was also more accurate than the ASCVD and SCORE2 risk scores in both men and women. The QR4 risk score identifies new risk groups and provides superior CVD risk prediction in the United Kingdom compared with other international scoring systems for CVD risk.

Performance of risk scores in predicting major bleeding in left ventricular assist device recipients: a comparative external validation

Background

Implantation of a left ventricular assist device (LVAD) is a crucial therapeutic option for selected end-stage heart failure patients. However, major bleeding (MB) complications postimplantation are a significant concern.

Objectives

We evaluated current risk scores' predictive accuracy for MB in LVAD recipients.

Methods

We conducted an observational, single-center study of LVAD recipients (HeartWare or HeartMate-3, November 2010-December 2022) in the Netherlands. The primary outcome was the first post-LVAD MB (according to the International Society on Thrombosis and Haemostasis [ISTH] and Interagency Registry for Mechanically Assisted Circulatory Support [INTERMACS], and INTERMACS combined with intracranial bleeding [INTERMACS+] criteria). Mortality prior to MB was considered a competing event. Discrimination (C-statistic) and calibration were evaluated for the Hypertension, Abnormal Renal/Liver Function, Stroke, Bleeding History or Predisposition, Labile INR, Elderly, Drugs/Alcohol Concomitantly score, Hepatic or Renal Disease, Ethanol Abuse, Malignancy, Older Age, Reduced Platelet Count or Function, Re-Bleeding, Hypertension, Anemia, Genetic Factors, Excessive Fall Risk and Stroke score, Anticoagulation and Risk Factors in Atrial Fibrillation score, Outpatient Bleeding Risk Index, venous thromboembolism score, atrial fibrillation score, and Utah Bleeding Risk Score (UBRS).

Results

One hundred four patients were included (median age, 64 years; female, 20.2%; HeartWare, 90.4%; HeartMate-3, 9.6%). The cumulative MB incidence was 75.7% (95% CI 65.5%-85.9%) by ISTH and INTERMACS+ criteria and 67.0% (95% CI 56.0%-78.0%) per INTERMACS criteria over a median event-free follow-up time of 1916 days (range, 59-4521). All scores had poor discriminative ability on their intended prediction timeframe. Cumulative area under the receiving operator characteristic curve ranged from 0.49 (95% CI 0.35-0.63, venous thromboembolism-BLEED) to 0.56 (95% CI 0.47-0.65, UBRS) according to ISTH and INTERMACS+ criteria and from 0.48 (95% CI 0.40-0.56, Anticoagulation and Risk Factors in Atrial Fibrillation) to 0.56 (95% CI 0.47-0.65, UBRS) per INTERMACS criteria. All models showed poor calibration, largely underestimating MB risk.

Conclusion

Current bleeding risk scores exhibit inadequate predictive accuracy for LVAD recipients. There is a need for an accurate risk score to identify LVAD patients at high risk of MB who may benefit from patient-tailored antithrombotic therapy.

Predicting the risks of kidney failure and death in adults with moderate to severe chronic kidney disease: multinational, longitudinal, population based, cohort study

OBJECTIVE

To train and test a super learner strategy for risk prediction of kidney failure and mortality in people with incident moderate to severe chronic kidney disease (stage G3b to G4).

DESIGN

Multinational, longitudinal, population based, cohort study.

SETTINGS

Linked population health data from Canada (training and temporal testing), and Denmark and Scotland (geographical testing).

PARTICIPANTS

People with newly recorded chronic kidney disease at stage G3b-G4, estimated glomerular filtration rate (eGFR) 15-44 mL/min/1.73 m2.

MODELLING

The super learner algorithm selected the best performing regression models or machine learning algorithms (learners) based on their ability to predict kidney failure and mortality with minimised cross-validated prediction error (Brier score, the lower the better). Prespecified learners included age, sex, eGFR, albuminuria, with or without diabetes, and cardiovascular disease. The index of prediction accuracy, a measure of calibration and discrimination calculated from the Brier score (the higher the better) was used to compare KDpredict with the benchmark, kidney failure risk equation, which does not account for the competing risk of death, and to evaluate the performance of KDpredict mortality models.

RESULTS

67 942 Canadians, 17 528 Danish, and 7740 Scottish residents with chronic kidney disease at stage G3b to G4 were included (median age 77-80 years; median eGFR 39 mL/min/1.73 m2). Median follow-up times were five to six years in all cohorts. Rates were 0.8-1.1 per 100 person years for kidney failure and 10-12 per 100 person years for death. KDpredict was more accurate than kidney failure risk equation in prediction of kidney failure risk: five year index of prediction accuracy 27.8% (95% confidence interval 25.2% to 30.6%) versus 18.1% (15.7% to 20.4%) in Denmark and 30.5% (27.8% to 33.5%) versus 14.2% (12.0% to 16.5%) in Scotland. Predictions from kidney failure risk equation and KDpredict differed substantially, potentially leading to diverging treatment decisions. An 80-year-old man with an eGFR of 30 mL/min/1.73 m2 and an albumin-to-creatinine ratio of 100 mg/g (11 mg/mmol) would receive a five year kidney failure risk prediction of 10% from kidney failure risk equation (above the current nephrology referral threshold of 5%). The same man would receive five year risk predictions of 2% for kidney failure and 57% for mortality from KDpredict. Individual risk predictions from KDpredict with four or six variables were accurate for both outcomes. The KDpredict models retrained using older data provided accurate predictions when tested in temporally distinct, more recent data.

CONCLUSIONS

KDpredict could be incorporated into electronic medical records or accessed online to accurately predict the risks of kidney failure and death in people with moderate to severe CKD. The KDpredict learning strategy is designed to be adapted to local needs and regularly revised over time to account for changes in the underlying health system and care processes.

Validation of the American College of Surgeons National Surgical Quality Improvement Program risk predictor in an Australian general surgical cohort

BACKGROUND

The National Surgical Quality Improvement Program (NSQIP) Surgical Risk Calculator from the American College of Surgeons is a widely available tool for peri-operative risk prediction. This study investigates its predictive performance in an Australian setting.

METHODS

A single-centre retrospective external validation study was conducted at a tertiary referral centre in New South Wales, Australia. Data from a general surgical cohort in a 2-year period from 2020 to 2021 was collected from the NSQIP database and entered into the NSQIP calculator. The predictive performance of the calculator was analysed across the standard 14 NSQIP postoperative outcome measures at 30 days.

RESULTS

There were 2121 patient records analysed using tests of accuracy and in the discrimination and calibration domains. The overall predictive performance of the NSQIP calculator was reasonable. There was greater accuracy at lower-risk predictions. At higher-risk predictions, Readmission, Death, and Discharge to Nursing or Rehab Facility, and Length of Stay were overestimated, whilst other outcomes were underestimated.

CONCLUSION

This study demonstrates reasonable overall performance of the NSQIP calculator in the context of this cohort and provides data to support the need for locally adapted and validated risk prediction tools for use by Australian perioperative physicians.

An external validation of the Kidney Donor Risk Index in the UK transplant population in the presence of semi-competing events

BACKGROUND

Transplantation represents the optimal treatment for many patients with end-stage kidney disease. When a donor kidney is available to a waitlisted patient, clinicians responsible for the care of the potential recipient must make the decision to accept or decline the offer based upon complex and variable information about the donor, the recipient and the transplant process. A clinical prediction model may be able to support clinicians in their decision-making. The Kidney Donor Risk Index (KDRI) was developed in the United States to predict graft failure following kidney transplantation. The survival process following transplantation consists of semi-competing events where death precludes graft failure, but not vice-versa.

METHODS

We externally validated the KDRI in the UK kidney transplant population and assessed whether validation under a semi-competing risks framework impacted predictive performance. Additionally, we explored whether the KDRI requires updating. We included 20,035 adult recipients of first, deceased donor, single, kidney-only transplants between January 1, 2004, and December 31, 2018, collected by the UK Transplant Registry and held by NHS Blood and Transplant. The outcomes of interest were 1- and 5-year graft failure following transplantation. In light of the semi-competing events, recipient death was handled in two ways: censoring patients at the time of death and modelling death as a competing event. Cox proportional hazard models were used to validate the KDRI when censoring graft failure by death, and cause-specific Cox models were used to account for death as a competing event.

RESULTS

The KDRI underestimated event probabilities for those at higher risk of graft failure. For 5-year graft failure, discrimination was poorer in the semi-competing risks model (0.625, 95% CI 0.611 to 0.640;0.611, 95% CI 0.597 to 0.625), but predictions were more accurate (Brier score 0.117, 95% CI 0.112 to 0.121; 0.114, 95% CI 0.109 to 0.118). Calibration plots were similar regardless of whether the death was modelled as a competing event or not. Updating the KDRI worsened calibration, but marginally improved discrimination.

CONCLUSIONS

Predictive performance for 1-year graft failure was similar between death-censored and competing event graft failure, but differences appeared when predicting 5-year graft failure. The updated index did not have superior performance and we conclude that updating the KDRI in the present form is not required.

Accounting for time dependency in meta-analyses of concordance probability estimates

Recent years have seen the development of many novel scoring tools for disease prognosis and prediction. To become accepted for use in clinical applications, these tools have to be validated on external data. In practice, validation is often hampered by logistical issues, resulting in multiple small-sized validation studies. It is therefore necessary to synthesize the results of these studies using techniques for meta-analysis. Here we consider strategies for meta analyzing the concordance probability for time-to-event data ("C-index"), which has become a popular tool to evaluate the discriminatory power of prediction models with a right-censored outcome. We show that standard meta-analysis of the C-index may lead to biased results, as the magnitude of the concordance probability depends on the length of the time interval used for evaluation (defined e.g., by the follow-up time, which might differ considerably between studies). To address this issue, we propose a set of methods for random-effects meta-regression that incorporate time directly as covariate in the model equation. In addition to analyzing nonlinear time trends via fractional polynomial, spline, and exponential decay models, we provide recommendations on suitable transformations of the C-index before meta-regression. Our results suggest that the C-index is best meta-analyzed using fractional polynomial meta-regression with logit-transformed C-index values. Classical random-effects meta-analysis (not considering time as covariate) is demonstrated to be a suitable alternative when follow-up times are small. Our findings have implications for the reporting of C-index values in future studies, which should include information on the length of the time interval underlying the calculations.

Using the kidney failure risk equation to predict end-stage kidney disease in CKD patients of South Asian ethnicity: an external validation study

BACKGROUND

The kidney failure risk equation (KFRE) predicts the 2- and 5-year risk of needing kidney replacement therapy (KRT) using four risk factors - age, sex, urine albumin-to-creatinine ratio (ACR) and creatinine-based estimated glomerular filtration rate (eGFR). Although the KFRE has been recalibrated in a UK cohort, this did not consider minority ethnic groups. Further validation of the KFRE in different ethnicities is a research priority. The KFRE also does not consider the competing risk of death, which may lead to overestimation of KRT risk. This study externally validates the KFRE for patients of South Asian ethnicity and compares methods for accounting for ethnicity and the competing event of death.

METHODS

Data were gathered from an established UK cohort containing 35,539 individuals diagnosed with chronic kidney disease. The KFRE was externally validated and updated in several ways taking into account ethnicity, using recognised methods for time-to-event data, including the competing risk of death. A clinical impact assessment compared the updated models through consideration of referrals made to secondary care.

RESULTS

The external validation showed the risk of KRT differed by ethnicity. Model validation performance improved when incorporating ethnicity and its interactions with ACR and eGFR as additional risk factors. Furthermore, accounting for the competing risk of death improved prediction. Using criteria of 5 years ≥ 5% predicted KRT risk, the competing risks model resulted in an extra 3 unnecessary referrals (0.59% increase) but identified an extra 1 KRT case (1.92% decrease) compared to the previous best model. Hybrid criteria of predicted risk using the competing risks model and ACR ≥ 70 mg/mmol should be used in referrals to secondary care.

CONCLUSIONS

The accuracy of KFRE prediction improves when updated to consider South Asian ethnicity and to account for the competing risk of death. This may reduce unnecessary referrals whilst identifying risks of KRT and could further individualise the KFRE and improve its clinical utility. Further research should consider other ethnicities.

Germline mechanisms of immunotherapy toxicities in the era of genome-wide association studies

Cancer immunotherapy has revolutionized the treatment of advanced cancers and is quickly becoming an option for early-stage disease. By reactivating the host immune system, immunotherapy harnesses patients' innate defenses to eradicate the tumor. By putatively similar mechanisms, immunotherapy can also substantially increase the risk of toxicities or immune-related adverse events (irAEs). Severe irAEs can lead to hospitalization, treatment discontinuation, lifelong immune complications, or even death. Many irAEs present with similar symptoms to heritable autoimmune diseases, suggesting that germline genetics may contribute to their onset. Recently, genome-wide association studies (GWAS) of irAEs have identified common germline associations and putative mechanisms, lending support to this hypothesis. A wide range of well-established GWAS methods can potentially be harnessed to understand the etiology of irAEs specifically and immunotherapy outcomes broadly. This review summarizes current findings regarding germline effects on immunotherapy outcomes and discusses opportunities and challenges for leveraging germline genetics to understand, predict, and treat irAEs.

Development and validation of a novel risk prediction algorithm to estimate 10-year risk of oesophageal cancer in primary care: prospective cohort study and evaluation of performance against two other risk prediction models

Background

Methods to identify patients at increased risk of oesophageal cancer are needed to better identify those for targeted screening. We aimed to derive and validate novel risk prediction algorithms (CanPredict) to estimate the 10-year risk of oesophageal cancer and evaluate performance against two other risk prediction models.

Methods

Prospective open cohort study using routinely collected data from 1804 QResearch® general practices. We used 1354 practices (12.9 M patients) to develop the algorithm. We validated the algorithm in 450 separate practices from QResearch (4.12 M patients) and 355 Clinical Practice Research Datalink (CPRD) practices (2.53 M patients). The primary outcome was an incident diagnosis of oesophageal cancer found in GP, mortality, hospital, or cancer registry data. Patients were aged 25-84 years and free of oesophageal cancer at baseline. Cox proportional hazards models were used with prediction selection to derive risk equations. Risk factors included age, ethnicity, Townsend deprivation score, body mass index (BMI), smoking, alcohol, family history, relevant co-morbidities and medications. Measures of calibration, discrimination, sensitivity, and specificity were calculated in the validation cohorts.

Finding

There were 16,384 incident cases of oesophageal cancer in the derivation cohort (0.13% of 12.9 M). The predictors in the final algorithms were: age, BMI, Townsend deprivation score, smoking, alcohol, ethnicity, Barrett's oesophagus, hiatus hernia, H. pylori infection, use of proton pump inhibitors, anaemia, lung and blood cancer (with breast cancer in women). In the QResearch validation cohort in women the explained variation (R2) was 57.1%; Royston's D statistic 2.36 (95% CI 2.26-2.46); C statistic 0.859 (95% CI 0.849-0.868) and calibration was good. Results were similar in men. For the 20% at highest predicted risk, the sensitivity was 76%, specificity was 80.1% and the observed risk at 10 years was 0.76%. The results from the CPRD validation were similar.

Interpretation

We have developed and validated a novel prediction algorithm to quantify the absolute risk of oesophageal cancer. The CanPredict algorithms could be used to identify high risk patients for targeted screening.

Funding

Innovate UK and CRUK (grant 105857).

Machine learning models of healthcare expenditures predicting mortality: A cohort study of spousal bereaved Danish individuals

BACKGROUND

The ability to accurately predict survival in older adults is crucial as it guides clinical decision making. The added value of using health care usage for predicting mortality remains unexplored. The aim of this study was to investigate if temporal patterns of healthcare expenditures, can improve the predictive performance for mortality, in spousal bereaved older adults, next to other widely used sociodemographic variables.

METHODS

This is a population-based cohort study of 48,944 Danish citizens 65 years of age and older suffering bereavement within 2013-2016. Individuals were followed from date of spousal loss until death from all causes or 31st of December 2016, whichever came first. Healthcare expenditures were available on weekly basis for each person during the follow-up and used as predictors for mortality risk in Extreme Gradient Boosting models. The extent to which medical spending trajectories improved mortality predictions compared to models with sociodemographics, was assessed with respect to discrimination (AUC), overall prediction error (Brier score), calibration, and clinical benefit (decision curve analysis).

RESULTS

The AUC of age and sex for mortality the year after spousal loss was 70.8% [95% CI 68.8, 72.8]. The addition of sociodemographic variables led to an increase of AUC ranging from 0.9% to 3.1% but did not significantly reduce the overall prediction error. The AUC of the model combining the variables above plus medical spending usage was 80.8% [79.3, 82.4] also exhibiting smaller Brier score and better calibration. Overall, patterns of healthcare expenditures improved mortality predictions the most, also exhibiting the highest clinical benefit among the rest of the models.

CONCLUSION

Temporal patterns of medical spending have the potential to significantly improve our assessment on who is at high risk of dying after suffering spousal loss. The proposed methodology can assist in a more efficient risk profiling and prognosis of bereaved individuals.

External validation and updating of prediction models of bleeding risk in patients with cancer receiving anticoagulants

OBJECTIVE

Patients with cancer are at increased bleeding risk, and anticoagulants increase this risk even more. Yet, validated bleeding risk models for prediction of bleeding risk in patients with cancer are lacking. The aim of this study is to predict bleeding risk in anticoagulated patients with cancer.

METHODS

We performed a study using the routine healthcare database of the Julius General Practitioners' Network. Five bleeding risk models were selected for external validation. Patients with a new cancer episode during anticoagulant treatment or those initiating anticoagulation during active cancer were included. The outcome was the composite of major bleeding and clinically relevant non-major (CRNM) bleeding. Next, we internally validated an updated bleeding risk model accounting for the competing risk of death.

RESULTS

The validation cohort consisted of 1304 patients with cancer, mean age 74.0±10.9 years, 52.2% males. In total 215 (16.5%) patients developed a first major or CRNM bleeding during a mean follow-up of 1.5 years (incidence rate; 11.0 per 100 person-years (95% CI 9.6 to 12.5)). The c-statistics of all selected bleeding risk models were low, around 0.56. Internal validation of an updated model accounting for death as competing risk showed a slightly improved c-statistic of 0.61 (95% CI 0.54 to 0.70). On updating, only age and a history of bleeding appeared to contribute to the prediction of bleeding risk.

CONCLUSIONS

Existing bleeding risk models cannot accurately differentiate bleeding risk between patients. Future studies may use our updated model as a starting point for further development of bleeding risk models in patients with cancer.

Clinical prediction model for prognosis in kidney transplant recipients (KIDMO): study protocol

BACKGROUND

Many potential prognostic factors for predicting kidney transplantation outcomes have been identified. However, in Switzerland, no widely accepted prognostic model or risk score for transplantation outcomes is being routinely used in clinical practice yet. We aim to develop three prediction models for the prognosis of graft survival, quality of life, and graft function following transplantation in Switzerland.

METHODS

The clinical kidney prediction models (KIDMO) are developed with data from a national multi-center cohort study (Swiss Transplant Cohort Study; STCS) and the Swiss Organ Allocation System (SOAS). The primary outcome is the kidney graft survival (with death of recipient as competing risk); the secondary outcomes are the quality of life (patient-reported health status) at 12 months and estimated glomerular filtration rate (eGFR) slope. Organ donor, transplantation, and recipient-related clinical information will be used as predictors at the time of organ allocation. We will use a Fine & Gray subdistribution model and linear mixed-effects models for the primary and the two secondary outcomes, respectively. Model optimism, calibration, discrimination, and heterogeneity between transplant centres will be assessed using bootstrapping, internal-external cross-validation, and methods from meta-analysis.

DISCUSSION

Thorough evaluation of the existing risk scores for the kidney graft survival or patient-reported outcomes has been lacking in the Swiss transplant setting. In order to be useful in clinical practice, a prognostic score needs to be valid, reliable, clinically relevant, and preferably integrated into the decision-making process to improve long-term patient outcomes and support informed decisions for clinicians and their patients. The state-of-the-art methodology by taking into account competing risks and variable selection using expert knowledge is applied to data from a nationwide prospective multi-center cohort study. Ideally, healthcare providers together with patients can predetermine the risk they are willing to accept from a deceased-donor kidney, with graft survival, quality of life, and graft function estimates available for their consideration.

STUDY REGISTRATION

Open Science Framework ID: z6mvj.

Assessing Performance and Clinical Usefulness in Prediction Models With Survival Outcomes: Practical Guidance for Cox Proportional Hazards Models

Risk prediction models need thorough validation to assess their performance. Validation of models for survival outcomes poses challenges due to the censoring of observations and the varying time horizon at which predictions can be made. This article describes measures to evaluate predictions and the potential improvement in decision making from survival models based on Cox proportional hazards regression.

As a motivating case study, the authors consider the prediction of the composite outcome of recurrence or death (the "event") in patients with breast cancer after surgery. They developed a simple Cox regression model with 3 predictors, as in the Nottingham Prognostic Index, in 2982 women (1275 events over 5 years of follow-up) and externally validated this model in 686 women (285 events over 5 years). Improvement in performance was assessed after the addition of progesterone receptor as a prognostic biomarker.

The model predictions can be evaluated across the full range of observed follow-up times or for the event occurring by the end of a fixed time horizon of interest. The authors first discuss recommended statistical measures that evaluate model performance in terms of discrimination, calibration, or overall performance. Further, they evaluate the potential clinical utility of the model to support clinical decision making according to a net benefit measure. They provide SAS and R code to illustrate internal and external validation.

The authors recommend the proposed set of performance measures for transparent reporting of the validity of predictions from survival models.

Interpretable prognostic modeling of endometrial cancer

Endometrial carcinoma (EC) is one of the most common gynecological cancers in the world. In this work we apply Cox proportional hazards (CPH) and optimal survival tree (OST) algorithms to the retrospective prognostic modeling of disease-specific survival in 842 EC patients. We demonstrate that linear CPH models are preferred for the EC risk assessment based on clinical features alone, while interpretable, non-linear OST models are favored when patient profiles can be supplemented with additional biomarker data. We show how visually interpretable tree models can help generate and explore novel research hypotheses by studying the OST decision path structure, in which L1 cell adhesion molecule expression and estrogen receptor status are correctly indicated as important risk factors in the p53 abnormal EC subgroup. To aid further clinical adoption of advanced machine learning techniques, we stress the importance of quantifying model discrimination and calibration performance in the development of explainable clinical prediction models.