Improving prediction models with new markers: a comparison of updating strategies

Derivation and Internal Validation of a Mortality Prognostication Machine Learning Model in Ebola Virus Disease Based on Iterative Point-of-Care Biomarkers

Background

Although multiple prognostic models exist for Ebola virus disease mortality, few incorporate biomarkers, and none has used longitudinal point-of-care serum testing throughout Ebola treatment center care.

Methods

This retrospective study evaluated adult patients with Ebola virus disease during the 10th outbreak in the Democratic Republic of Congo. Ebola virus cycle threshold (Ct; based on reverse transcriptase polymerase chain reaction) and point-of-care serum biomarker values were collected throughout Ebola treatment center care. Four iterative machine learning models were created for prognosis of mortality. The base model used age and admission Ct as predictors. Ct and biomarkers from treatment days 1 and 2, days 3 and 4, and days 5 and 6 associated with mortality were iteratively added to the model to yield mortality risk estimates. Receiver operating characteristic curves for each iteration provided period-specific areas under curve with 95% CIs.

Results

Of 310 cases positive for Ebola virus disease, mortality occurred in 46.5%. Biomarkers predictive of mortality were elevated creatinine kinase, aspartate aminotransferase, blood urea nitrogen (BUN), alanine aminotransferase, and potassium; low albumin during days 1 and 2; elevated C-reactive protein, BUN, and potassium during days 3 and 4; and elevated C-reactive protein and BUN during days 5 and 6. The area under curve substantially improved with each iteration: base model, 0.74 (95% CI, .69-.80); days 1 and 2, 0.84 (95% CI, .73-.94); days 3 and 4, 0.94 (95% CI, .88-1.0); and days 5 and 6, 0.96 (95% CI, .90-1.0).

Conclusions

This is the first study to utilize iterative point-of-care biomarkers to derive dynamic prognostic mortality models. This novel approach demonstrates that utilizing biomarkers drastically improved prognostication up to 6 days into patient care.

Highlighting allelic variations at the interleukin-19 locus in term of preeclampsia predisposing factors and access to an accurate diagnostic/screening option

BACKGROUND

Preeclampsia is the main cause of preterm parturition and maternal-fetal complications. T helper 1 and T helper 2 cytokines balance is a requirement in normal pregnancy and aberrant in this immunologic balance, play an important role in the pathology of preeclampsia. In previous studies single nucleotide polymorphisms have been associated with the alteration of serum cytokine levels.

OBJECTIVE

This study was aimed to discover association between interleukin-13 (rs20541, and rs56035208) and interleukin-19 (rs1028181 (T/C) and rs2243191(T/C)) polymorphisms with susceptibility to preeclampsia.

METHODS

In this case-control study 300 women with and without preeclampsia (n = 150/each) who referred to Zeynabieh Hospital- Shiraz, Iran, from February 2021 to April 2022 were enrolled. For genotyping the interleukin-13 and interleukin-19 polymorphisms, the Allele-specific polymerase chain reaction and direct sequencing method was carried out.

RESULTS

Our statistical results revealed no significant differences in allele and genotype frequencies for interleukin-13 polymorphisms compared to controls. We found that the interleukin-13 polymorphisms are significantly associated with vulnerability to edema at rs20541 position and maternal drinking at rs56035208 position. But it was interesting to note that the differences of both the allele and genotype frequencies of interleukin-19 polymorphisms and their contribution to the risk of preeclampsia susceptibility were significant.

CONCLUSIONS

No risk of preeclampsia was found in all comparisons for interleukin-13 polymorphisms. However, the interleukin-19 polymorphisms were found to confer the risk of preeclampsia in our population.

Prognostic models for mortality risk in patients requiring ECMO

PURPOSE

To provide an overview and evaluate the performance of mortality prediction models for patients requiring extracorporeal membrane oxygenation (ECMO) support for refractory cardiocirculatory or respiratory failure.

METHODS

A systematic literature search was undertaken to identify studies developing and/or validating multivariable prediction models for all-cause mortality in adults requiring or receiving veno-arterial (V-A) or veno-venous (V-V) ECMO. Estimates of model performance (observed versus expected (O:E) ratio and c-statistic) were summarized using random effects models and sources of heterogeneity were explored by means of meta-regression. Risk of bias was assessed using the Prediction model Risk Of BiAS Tool (PROBAST).

RESULTS

Among 4905 articles screened, 96 studies described a total of 58 models and 225 external validations. Out of all 58 models which were specifically developed for ECMO patients, 14 (24%) were ever externally validated. Discriminatory ability of frequently validated models developed for ECMO patients (i.e., SAVE and RESP score) was moderate on average (pooled c-statistics between 0.66 and 0.70), and comparable to general intensive care population-based models (pooled c-statistics varying between 0.66 and 0.69 for the Simplified Acute Physiology Score II (SAPS II), Acute Physiology and Chronic Health Evaluation II (APACHE II) score and Sequential Organ Failure Assessment (SOFA) score). Nearly all models tended to underestimate mortality with a pooled O:E > 1. There was a wide variability in reported performance measures of external validations, reflecting a large between-study heterogeneity. Only 1 of the 58 models met the generally accepted Prediction model Risk Of BiAS Tool criteria of good quality. Importantly, all predicted outcomes were conditional on the fact that ECMO support had already been initiated, thereby reducing their applicability for patient selection in clinical practice.

CONCLUSIONS

A large number of mortality prediction models have been developed for ECMO patients, yet only a minority has been externally validated. Furthermore, we observed only moderate predictive performance, large heterogeneity between-study populations and model performance, and poor methodological quality overall. Most importantly, current models are unsuitable to provide decision support for selecting individuals in whom initiation of ECMO would be most beneficial, as all models were developed in ECMO patients only and the decision to start ECMO had, therefore, already been made.

Evolution of Making Clinical Predictions in Neurosurgery

Prediction of clinical outcomes is an essential task for every physician. Physicians may base their clinical prediction of an individual patient on their intuition and on scientific material such as studies presenting population risks and studies reporting on risk factors (prognostic factors). A relatively new and more informative approach for making clinical predictions relies on the use of statistical models that simultaneously consider multiple predictors that provide an estimate of the patient's absolute risk of an outcome. There is a growing body of literature in the neurosurgical field reporting on clinical prediction models. These tools have high potential in supporting (not replacing) neurosurgeons with their prediction of a patient's outcome. If used sensibly, these tools pave the way for more informed decision-making with or for individual patients. Patients and their significant others want to know their risk of the anticipated outcome, how it is derived, and the uncertainty associated with it. Learning from these prediction models and communicating the output to others has become an increasingly important skill neurosurgeons have to master. This article describes the evolution of making clinical predictions in neurosurgery, synopsizes key phases for the generation of a useful clinical prediction model, and addresses some considerations when deploying and communicating the results of a prediction model. The paper is illustrated with multiple examples from the neurosurgical literature, including predicting arachnoid cyst rupture, predicting rebleeding in patients suffering from aneurysmal subarachnoid hemorrhage, and predicting survival in glioblastoma patients.

Methodological guidance for the evaluation and updating of clinical prediction models: a systematic review

BACKGROUND

Clinical prediction models are often not evaluated properly in specific settings or updated, for instance, with information from new markers. These key steps are needed such that models are fit for purpose and remain relevant in the long-term. We aimed to present an overview of methodological guidance for the evaluation (i.e., validation and impact assessment) and updating of clinical prediction models.

METHODS

We systematically searched nine databases from January 2000 to January 2022 for articles in English with methodological recommendations for the post-derivation stages of interest. Qualitative analysis was used to summarize the 70 selected guidance papers.

RESULTS

Key aspects for validation are the assessment of statistical performance using measures for discrimination (e.g., C-statistic) and calibration (e.g., calibration-in-the-large and calibration slope). For assessing impact or usefulness in clinical decision-making, recent papers advise using decision-analytic measures (e.g., the Net Benefit) over simplistic classification measures that ignore clinical consequences (e.g., accuracy, overall Net Reclassification Index). Commonly recommended methods for model updating are recalibration (i.e., adjustment of intercept or baseline hazard and/or slope), revision (i.e., re-estimation of individual predictor effects), and extension (i.e., addition of new markers). Additional methodological guidance is needed for newer types of updating (e.g., meta-model and dynamic updating) and machine learning-based models.

CONCLUSION

Substantial guidance was found for model evaluation and more conventional updating of regression-based models. An important development in model evaluation is the introduction of a decision-analytic framework for assessing clinical usefulness. Consensus is emerging on methods for model updating.

PredictCBC-2.0: a contralateral breast cancer risk prediction model developed and validated in ~ 200,000 patients

BACKGROUND

Prediction of contralateral breast cancer (CBC) risk is challenging due to moderate performances of the known risk factors. We aimed to improve our previous risk prediction model (PredictCBC) by updated follow-up and including additional risk factors.

METHODS

We included data from 207,510 invasive breast cancer patients participating in 23 studies. In total, 8225 CBC events occurred over a median follow-up of 10.2 years. In addition to the previously included risk factors, PredictCBC-2.0 included CHEK2 c.1100delC, a 313 variant polygenic risk score (PRS-313), body mass index (BMI), and parity. Fine and Gray regression was used to fit the model. Calibration and a time-dependent area under the curve (AUC) at 5 and 10 years were assessed to determine the performance of the models. Decision curve analysis was performed to evaluate the net benefit of PredictCBC-2.0 and previous PredictCBC models.

RESULTS

The discrimination of PredictCBC-2.0 at 10 years was higher than PredictCBC with an AUC of 0.65 (95% prediction intervals (PI) 0.56-0.74) versus 0.63 (95%PI 0.54-0.71). PredictCBC-2.0 was well calibrated with an observed/expected ratio at 10 years of 0.92 (95%PI 0.34-2.54). Decision curve analysis for contralateral preventive mastectomy (CPM) showed the potential clinical utility of PredictCBC-2.0 between thresholds of 4 and 12% 10-year CBC risk for BRCA1/2 mutation carriers and non-carriers.

CONCLUSIONS

Additional genetic information beyond BRCA1/2 germline mutations improved CBC risk prediction and might help tailor clinical decision-making toward CPM or alternative preventive strategies. Identifying patients who benefit from CPM, especially in the general breast cancer population, remains challenging.

Constructing, validating, and updating machine learning models to predict survival in children with Ebola Virus Disease

Background

Ebola Virus Disease (EVD) causes high case fatality rates (CFRs) in young children, yet there are limited data focusing on predicting mortality in pediatric patients. Here we present machine learning-derived prognostic models to predict clinical outcomes in children infected with Ebola virus.

Methods

Using retrospective data from the Ebola Data Platform, we investigated children with EVD from the West African EVD outbreak in 2014–2016. Elastic net regularization was used to create a prognostic model for EVD mortality. In addition to external validation with data from the 2018–2020 EVD epidemic in the Democratic Republic of the Congo (DRC), we updated the model using selected serum biomarkers.

Findings

Pediatric EVD mortality was significantly associated with younger age, lower PCR cycle threshold (Ct) values, unexplained bleeding, respiratory distress, bone/muscle pain, anorexia, dysphagia, and diarrhea. These variables were combined to develop the newly described EVD Prognosis in Children (EPiC) predictive model. The area under the receiver operating characteristic curve (AUC) for EPiC was 0.77 (95% CI: 0.74–0.81) in the West Africa derivation dataset and 0.76 (95% CI: 0.64–0.88) in the DRC validation dataset. Updating the model with peak aspartate aminotransferase (AST) or creatinine kinase (CK) measured within the first 48 hours after admission increased the AUC to 0.90 (0.77–1.00) and 0.87 (0.74–1.00), respectively.

Conclusion

The novel EPiC prognostic model that incorporates clinical information and commonly used biochemical tests, such as AST and CK, can be used to predict mortality in children with EVD.

Although case fatality rates remain high, there are limited data on predicting mortality in children with Ebola Virus Disease (EVD). Furthermore, challenges in predicting EVD outcomes using clinical and laboratory data highlight the need for the development and validation of pediatric predictive models. The novel EVD Prognosis in Children (EPiC) model uses clinical and biochemical information, such as AST and CK, to predict mortality in infected children. While few prognostic models or scoring systems have been developed to predict clinical outcomes of EVD, the majority of them were limited in geographical and temporal scope having been derived using data from one location. As such, the EPiC model is the first externally validated model for the prognosis of pediatric EVD using diverse datasets from geographically and temporally separate outbreaks. This model can be easily applied by bedside clinicians to assess pediatric patients at risk for death and help to allocate resources accordingly.

Updating Clinical Prediction Models: An Illustrative Case Study

The performance of clinical prediction models tends to deteriorate over time. Researchers often develop a new prediction if an existing model performs poorly at external validation. Model updating is an efficient technique and promising alternative to the de novo development of clinical prediction models. Model updating has been recommended by the TRIPOD guidelines. To illustrate several model updating techniques, a case study is provided for the development and updating of a clinical prediction model assessing postoperative anxiety in data coming from two double-blinded placebo-controlled randomized controlled trials with a very similar methodological framework. Note that the developed model and updated model are for didactic purposes only. This paper discusses some common considerations and caveats for researchers to be aware of when planning or applying updating of a prediction model.

A prediction model based on clinical and histological features for predicting recurrence in patients with stage I-II endometrial cancer after surgical treatment

OBJECTIVE

The study aimed to develop a prediction model combining clinical and histological features to predict recurrence in patients with stage I-II endometrial cancer (EC) after surgical treatment.

METHODS

A total of 746 stage I-II EC patients who had received primary surgical treatment at Taizhou People's Hospital between 2014 and 2018 were included and randomly divided as a Training cohort (n = 520) and a Validation cohort (n = 226) at a 7:3 ratio. Clinical features including age, body mass index, comorbidities, lymphadenectomy, and adjuvant treatment, and histological features including histologic type, myometrial invasion, cervical stromal invasion, and expression levels of Ki67, estrogen receptor (ER), progesterone receptor (PR), whey acidic protein 4-disulphide core domain 2 (WFDC2), and p53 were used to develop a prediction model for EC recurrence in the Training cohort using a multivariable Cox regression model. Model discrimination and calibration were further evaluated in the Validation cohort.

RESULTS

EC recurrence was observed in 60 (11.54%) patients in the Training cohort with a median length of follow-up of 39 months. Age, adjuvant treatment, histologic type, cervical stromal invasion, and expression levels of Ki67, ER, PR, and WFDC2 were factors significantly associated with EC recurrence based on univariable Cox regression analysis. After a model selection by AIC in a stepwise algorithm, the final model incorporated the above predictors showed a C-index of 0.85 and fair calibration in the Training cohort. In the Validation cohort, the model still showed good discrimination power (C-index 0.80) but moderate calibration.

CONCLUSIONS

The developed prediction model combining clinical and histological features can help to predict the EC recurrence in patients with stage I-II EC after surgical treatment.

Screening for Lung Cancer in Individuals Who Never Smoked: An International Association for the Study of Lung Cancer Early Detection and Screening Committee Report

Screening with low-dose computed tomography of high-risk individuals with a smoking history reduces lung cancer mortality. Current screening guidelines and eligibility criteria can miss more than 50% of lung cancers, and in some geographic areas, such as East Asia, a large proportion of the missed lung cancers are in never-smokers. Although randomized trials revealed the benefits of screening for people who smoke, these trials generally excluded never-smokers. Thus, the feasibility and effectiveness of lung cancer screening of individuals who never smoked are uncertain. Several known and suspected risk factors for lung cancers in never-smokers such as exposure to secondhand smoke, occupational carcinogens, radon, air pollution, and pulmonary diseases, such as chronic obstructive pulmonary disease and interstitial lung diseases, and intrinsic factors, such as age, are well noted. In this regard, knowledge of risk factors may make possible quantification and prediction of lung cancer risk in never smokers. It is worth considering if and how never smokers could be included in population-based screening programs. As the implementation of these programs is challenging in many countries owing to multiple factors and the epidemiologic differences by global regions, these issues will need to be evaluated in each country taking into account various factors, including accuracy of risk assessment and cost-effectiveness of screening in never smokers. This report aims to outline current knowledge on risk factors for lung cancer in never smokers to propose research strategies for this topic and initiate a broader discussion on lung cancer screening of never smokers. Similar considerations can be made in current and ex-smokers, which do not fulfill the current screening inclusion criteria, but otherwise are at increased risk. Although screening of never smokers may in the future be effectively conducted, current evidence to support widespread implementation of this practice is lacking.

Application of clinical prediction modeling in pediatric neurosurgery: a case study

There has been an increasing interest in articles reporting on clinical prediction models in pediatric neurosurgery. Clinical prediction models are mathematical equations that combine patient-related risk factors for the estimation of an individual's risk of an outcome. If used sensibly, these evidence-based tools may help pediatric neurosurgeons in medical decision-making processes. Furthermore, they may help to communicate anticipated future events of diseases to children and their parents and facilitate shared decision-making accordingly. A basic understanding of this methodology is incumbent when developing or applying a prediction model. This paper addresses this methodology tailored to pediatric neurosurgery. For illustration, we use original pediatric data from our institution to illustrate this methodology with a case study. The developed model is however not externally validated, and clinical impact has not been assessed; therefore, the model cannot be recommended for clinical use in its current form.

Outcome Prediction after Moderate and Severe Traumatic Brain Injury: External Validation of Two Established Prognostic Models in 1742 European Patients

The International Mission on Prognosis and Analysis of Clinical Trials in Traumatic Brain Injury (IMPACT) and Corticoid Randomisation After Significant Head injury (CRASH) prognostic models predict functional outcome after moderate and severe traumatic brain injury (TBI). We aimed to assess their performance in a contemporary cohort of patients across Europe. The Collaborative European NeuroTrauma Effectiveness Research in Traumatic Brain Injury (CENTER-TBI) core study is a prospective, observational cohort study in patients presenting with TBI and an indication for brain computed tomography. The CENTER-TBI core cohort consists of 4509 TBI patients available for analyses from 59 centers in 18 countries across Europe and Israel. The IMPACT validation cohort included 1173 patients with GCS ≤12, age ≥14, and 6-month Glasgow Outcome Scale-Extended (GOSE) available. The CRASH validation cohort contained 1742 patients with GCS ≤14, age ≥16, and 14-day mortality or 6-month GOSE available. Performance of the three IMPACT and two CRASH model variants was assessed with discrimination (area under the receiver operating characteristic curve; AUC) and calibration (comparison of observed vs. predicted outcome rates). For IMPACT, model discrimination was good, with AUCs ranging between 0.77 and 0.85 in 1173 patients and between 0.80 and 0.88 in the broader CRASH selection (n = 1742). For CRASH, AUCs ranged between 0.82 and 0.88 in 1742 patients and between 0.66 and 0.80 in the stricter IMPACT selection (n = 1173). Calibration of the IMPACT and CRASH models was generally moderate, with calibration-in-the-large and calibration slopes ranging between -2.02 and 0.61 and between 0.48 and 1.39, respectively. The IMPACT and CRASH models adequately identify patients at high risk for mortality or unfavorable outcome, which supports their use in research settings and for benchmarking in the context of quality-of-care assessment.

Refining prediction of major bleeding on antiplatelet treatment after transient ischaemic attack or ischaemic stroke

Introduction

Bleeding is the main safety concern of treatment with antiplatelet drugs. We aimed to refine prediction of major bleeding on antiplatelet treatment after a transient ischaemic attack (TIA) or stroke by assessing the added value of new predictors to the existing S₂TOP-BLEED score.

Patients and methods

We used Cox regression analysis to study the association between candidate predictors and major bleeding among 2072 patients with a transient ischaemic attack or ischaemic stroke included in a population-based study (Oxford Vascular Study – OXVASC). An updated model was proposed and validated in 1094 patients with a myocardial infarction included in OXVASC. Models were compared with c-statistics, calibration plots, and net reclassification improvement.

Results

Independent predictors for major bleeding on top of S₂TOP-BLEED variables were peptic ulcer (hazard ratio (HR): 1.72; 1.04–2.86), cancer (HR: 2.40; 1.57–3.68), anaemia (HR: 1.55; 0.99–2.44) and renal failure (HR: 2.20; 1.57–4.28). Addition of those variables improved discrimination from 0.69 (0.64–0.73) to 0.73 (0.69–0.78) in the TIA/stroke cohort (p = 0.01). Performance improved particularly for upper gastro-intestinal bleeds (0.70; 0.64–0.75 to 0.77; 0.72–0.82). Net reclassification improved over the entire range of the score (net reclassification improvement: 0.56; 0.36–0.76). In the validation cohort, discriminatory performance improved from 0.68 (0.62–0.74) to 0.70 (0.64–0.76).

Discussion and Conclusion

Peptic ulcer, cancer, anaemia and renal failure improve predictive performance of the S₂TOP-BLEED score for major bleeding after stroke. Future external validation studies will be required to confirm the value of the STOP-BLEED+ score in transient ischaemic attack/stroke patients.

Polygenic Risk Score Contribution to Psychosis Prediction in a Target Population of Persons at Clinical High Risk

OBJECTIVE

The 2-year risk of psychosis in persons who meet research criteria for a high-risk syndrome is about 15%-25%; improvements in risk prediction accuracy would benefit the development and implementation of preventive interventions. The authors sought to assess polygenic risk score (PRS) prediction of subsequent psychosis in persons at high risk and to determine the impact of adding the PRS to a previously validated psychosis risk calculator.

METHODS

Persons meeting research criteria for psychosis high risk (N=764) and unaffected individuals (N=279) were followed for up to 2 years. The PRS was based on the latest schizophrenia and bipolar genome-wide association studies. Variables in the psychosis risk calculator included stressful life events, trauma, disordered thought content, verbal learning, information processing speed, and family history of psychosis.

RESULTS

For Europeans, the PRS varied significantly by group and was higher in the psychosis converter group compared with both the nonconverter and unaffected groups, but was similar for the nonconverter group compared with the unaffected group. For non-Europeans, the PRS varied significantly by group; the difference between the converters and nonconverters was not significant, but the PRS was significantly higher in converters than in unaffected individuals, and it did not differ between nonconverters and unaffected individuals. The R2liability (R2 adjusted for the rate of disease risk in the population being studied, here assuming a 2-year psychosis risk between 10% and 30%) for Europeans varied between 9.2% and 12.3% and for non-Europeans between 3.5% and 4.8%. The amount of risk prediction information contributed by the addition of the PRS to the risk calculator was less than severity of disordered thoughts and similar to or greater than for other variables. For Europeans, the PRS was correlated with risk calculator variables of information processing speed and verbal memory.

CONCLUSIONS

The PRS discriminates psychosis converters from nonconverters and modestly improves individualized psychosis risk prediction when added to a psychosis risk calculator. The schizophrenia PRS shows promise in enhancing risk prediction in persons at high risk for psychosis, although its potential utility is limited by poor performance in persons of non-European ancestry.

Prediction and clinical utility of a contralateral breast cancer risk model

BACKGROUND

Breast cancer survivors are at risk for contralateral breast cancer (CBC), with the consequent burden of further treatment and potentially less favorable prognosis. We aimed to develop and validate a CBC risk prediction model and evaluate its applicability for clinical decision-making.

METHODS

We included data of 132,756 invasive non-metastatic breast cancer patients from 20 studies with 4682 CBC events and a median follow-up of 8.8 years. We developed a multivariable Fine and Gray prediction model (PredictCBC-1A) including patient, primary tumor, and treatment characteristics and BRCA1/2 germline mutation status, accounting for the competing risks of death and distant metastasis. We also developed a model without BRCA1/2 mutation status (PredictCBC-1B) since this information was available for only 6% of patients and is routinely unavailable in the general breast cancer population. Prediction performance was evaluated using calibration and discrimination, calculated by a time-dependent area under the curve (AUC) at 5 and 10 years after diagnosis of primary breast cancer, and an internal-external cross-validation procedure. Decision curve analysis was performed to evaluate the net benefit of the model to quantify clinical utility.

RESULTS

In the multivariable model, BRCA1/2 germline mutation status, family history, and systemic adjuvant treatment showed the strongest associations with CBC risk. The AUC of PredictCBC-1A was 0.63 (95% prediction interval (PI) at 5 years, 0.52-0.74; at 10 years, 0.53-0.72). Calibration-in-the-large was -0.13 (95% PI: -1.62-1.37), and the calibration slope was 0.90 (95% PI: 0.73-1.08). The AUC of Predict-1B at 10 years was 0.59 (95% PI: 0.52-0.66); calibration was slightly lower. Decision curve analysis for preventive contralateral mastectomy showed potential clinical utility of PredictCBC-1A between thresholds of 4-10% 10-year CBC risk for BRCA1/2 mutation carriers and non-carriers.

CONCLUSIONS

We developed a reasonably calibrated model to predict the risk of CBC in women of European-descent; however, prediction accuracy was moderate. Our model shows potential for improved risk counseling, but decision-making regarding contralateral preventive mastectomy, especially in the general breast cancer population where limited information of the mutation status in BRCA1/2 is available, remains challenging.

Prediction models for delayed graft function: external validation on The Dutch Prospective Renal Transplantation Registry

Background

Delayed graft function (DGF) is a common complication after kidney transplantation in the era of accepting an equal number of brain- and circulatory-death donor kidneys in the Netherlands. To identify those cases with an increased risk of developing DGF, various multivariable algorithms have been proposed. The objective was to validate the reproducibility of four predictive algorithms by Irish et al. (A risk prediction model for delayed graft function in the current era of deceased donor renal transplantation. Am J Transplant 2010;10:2279-2286) (USA), Jeldres et al. (Prediction of delayed graft function after renal transplantation. Can Urol Assoc J 2009;3:377-382) (Canada), Chapal et al. (A useful scoring system for the prediction and management of delayed graft function following kidney transplantation from cadaveric donors. Kidney Int 2014;86:1130-1139) (France) and Zaza et al. (Predictive model for delayed graft function based on easily available pre-renal transplant variables. Intern Emerg Med 2015;10:135-141) (Italy) according to a novel framework for external validation.

Methods

We conducted a prospective observational study with data from the Dutch Organ Transplantation Registry (NOTR). Renal transplant recipients from all eight Dutch academic medical centers between 2002 and 2012 who received a deceased allograft were included (N = 3333). The four prediction algorithms were reconstructed from donor, recipient and transplantation data. Their predictive value for DGF was validated by c-statistics, calibration statistics and net benefit analysis. Case-mix (un)relatedness was investigated with a membership model and mean and standard deviation of the linear predictor.

Results

The prevalence of DGF was 37%. Despite a significantly different case-mix, the US algorithm by Irish was best reproducible, with a c-index of 0.761 (range 0.756 - 0.762), and well-calibrated over the complete range of predicted probabilities of having DGF. The US model had a net benefit of 0.242 at a threshold probability of 0.25, compared with 0.089 net benefit for the same threshold in the original study, equivalent to correctly identifying DGF in 24 cases per 100 patients (true positive results) without an increase in the number of false-positive results.

Conclusions

The US model by Irish et al. was generalizable and best transportable to Dutch recipients with a deceased donor kidney. The algorithm detects an increased risk of DGF after allocation and enables us to improve individual patient management.

Head-to-head comparison of prostate cancer risk calculators predicting biopsy outcome

Background

Multivariable risk calculators (RCs) predicting prostate cancer (PCa) aim to reduce unnecessary workup (e.g., MRI and biopsy) by selectively identifying those men at risk for PCa or clinically significant PCa (csPCa) (Gleason ≥7). The lack of an adequate comparison makes choosing between RCs difficult for patients, clinicians and guideline developers. We aim to perform a head-to-head comparison of seven well known RCs predicting biopsy outcome.

Methods

Our study comprised 7,119 men from ten independent contemporary cohorts in Europe and Australia, who underwent prostate biopsy between 2007 and 2015. We evaluated the performance of the ERSPC RPCRC, Finne, Chun, ProstataClass, Karakiewicz, Sunnybrook, and PCPT 2.0 (HG) RCs in predicting the presence of any PCa and csPCa. Performance was assessed by discrimination, calibration and net benefit analyses.

Results

A total of 3,458 (48%) PCa were detected; 1,784 (25%) men had csPCa. No particular RC stood out predicting any PCa: pooled area under the ROC-curve (AUC) ranged between 0.64 and 0.72. The ERSPC RPCRC had the highest pooled AUC 0.77 (95% CI: 0.73–0.80) when predicting csPCa. Decision curve analysis (DCA) showed limited net benefit in the detection of csPCa, but that can be improved by a simple calibration step. The main limitation is the retrospective design of the study.

Conclusions

No particular RC stands out when predicting biopsy outcome on the presence of any PCa. The ERSPC RPCRC is superior in identifying those men at risk for csPCa. Net benefit analyses show that a multivariate approach before further workup is advisable.