Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): the TRIPOD Statement

An appraisal of analytical tools used in predicting clinical outcomes following radiation therapy treatment of men with prostate cancer: a systematic review

Background

Prostate cancer can be treated with several different modalities, including radiation treatment. Various prognostic tools have been developed to aid decision making by providing estimates of the probability of different outcomes. Such tools have been demonstrated to have better prognostic accuracy than clinical judgment alone.

Methods

A systematic review was undertaken to identify papers relating to the prediction of clinical outcomes (biochemical failure, metastasis, survival) in patients with prostate cancer who received radiation treatment, with the particular aim of identifying whether published tools are adequately developed, validated, and provide accurate predictions. PubMed and EMBASE were searched from July 2007. Title and abstract screening, full text review, and critical appraisal were conducted by two reviewers. A review protocol was published in advance of commencing literature searches.

Results

The search strategy resulted in 165 potential articles, of which 72 were selected for full text review and 47 ultimately included. These papers described 66 models which were newly developed and 31 which were external validations of already published predictive tools. The included studies represented a total of 60,457 patients, recruited between 1984 and 2009. Sixty five percent of models were not externally validated, 57% did not report accuracy and 31% included variables which are not readily accessible in existing datasets. Most models (72, 74%) related to external beam radiation therapy with the remainder relating to brachytherapy (alone or in combination with external beam radiation therapy).

Conclusions

A large number of prognostic models (97) have been described in the recent literature, representing a rapid increase since previous reviews (17 papers, 1966–2007). Most models described were not validated and a third utilised variables which are not readily accessible in existing data collections. Where validation had occurred, it was often limited to data taken from single institutes in the US. While validated and accurate models are available to predict prostate cancer specific mortality following external beam radiation therapy, there is a scarcity of such tools relating to brachytherapy. This review provides an accessible catalogue of predictive tools for current use and which should be prioritised for future validation.

Electronic supplementary material

The online version of this article (doi:10.1186/s13014-017-0786-z) contains supplementary material, which is available to authorized users.

Risk prediction models for delirium in the intensive care unit after cardiac surgery: a systematic review and independent external validation

Numerous risk prediction models are available for predicting delirium after cardiac surgery, but few have been directly compared with one another or been validated in an independent data set. We conducted a systematic review to identify validated risk prediction models of delirium (using the Confusion Assessment Method-Intensive Care Unit tool) after cardiac surgery and assessed the transportability of the risk prediction models on a prospective cohort of 600 consecutive patients undergoing cardiac surgery at a university hospital in Hong Kong from July 2013 to July 2015. The discrimination (c-statistic), calibration (GiViTI calibration belt), and clinical usefulness (decision curve analysis) of the risk prediction models were examined in a stepwise manner. Three published high-quality intensive care unit delirium risk prediction models (n=5939) were identified: Katznelson, the original PRE-DELIRIC, and the international recalibrated PRE-DELIRIC model. Delirium occurred in 83 patients (13.8%, 95% CI: 11.2-16.9%). After updating the intercept and regression coefficients in the Katznelson model, there was fair discrimination (0.62, 95% CI: 0.58-0.66) and good calibration. As the original PRE-DELIRIC model was already validated externally and recalibrated in six countries, we performed a logistic calibration on the recalibrated model and found acceptable discrimination (0.75, 95% CI: 0.72-0.79) and good calibration. Decision curve analysis demonstrated that the recalibrated PRE-DELIRIC risk model was marginally more clinically useful than the Katznelson model. Current models predict delirium risk in the intensive care unit after cardiac surgery with only fair to moderate accuracy and are insufficient for routine clinical use.

Prognostic Abilities and Quality Assessment of Models for the Prediction of 90-Day Mortality in Liver Transplant Waiting List Patients

Background

Model of end-stage liver disease (MELD)-score and diverse variants are widely used for prognosis on liver transplant waiting-lists.

Methods

818 consecutive patients on the liver transplant waiting-list included to calculate the MELD, MESO Index, MELD-Na, UKELD, iMELD, refitMELD, refitMELD-Na, upMELD and PELD-scores. Prognostic abilities for 90-day mortality were investigated applying Receiver-operating-characteristic-curve analysis. Independent risk factors for 90-day mortality were identified with multivariable binary logistic regression modelling. Methodological quality of the underlying development studies was assessed with a systematic assessment tool.

Results

74 patients (9%) died on the liver transplant waiting list within 90 days after listing. All but one scores, refitMELD-Na, had acceptable prognostic performance with areas under the ROC-curves (AUROCs)>0.700. The iMELD performed best (AUROC = 0.798). In pediatric cases, the PELD-score just failed to reach the acceptable threshold with an AUROC = 0.699. All scores reached a mean quality score of 72.3%. Highest quality scores could be achieved by the UKELD and PELD-scores. Studies specifically lack statistical validity and model evaluation.

Conclusions

Inferior quality assessment of prognostic models does not necessarily imply inferior prognostic abilities. The iMELD might be a more reliable tool representing urgency of transplantation than the MELD-score. PELD-score is assumedly not accurate enough to allow graft allocation decision in pediatric liver transplantation.

A New Preprocedure Risk Score for Predicting Contrast-Induced Acute Kidney Injury

BACKGROUND

Most of the risk models for predicting contrast-induced acute kidney injury (CI-AKI) are available for only postcontrast exposure prediction; however, prediction before the procedure is more valuable in practice. This study aimed to develop a risk scoring system based on preprocedural characteristics for early prediction of CI-AKI in patients after coronary angiography or percutaneous coronary intervention (PCI).

METHODS

We prospectively recruited 1777 consecutive patients who were randomized in an approximate 3:2 ratio to create a development data set (n = 1076) and a validation data set (n = 701). A risk score model based on preprocedural risk factors was developed using stepwise logistic regression. Validation was performed by bootstrap and split-sample methods.

RESULTS

The occurrence of CI-AKI was 5.97% (106 of 1777), 5.95% (64 of 1076), and 5.99% (42 of 701) in the overall, developmental, and validation data sets, respectively. The risk score was developed with 5 prognostic factors (age, serum creatinine levels, N-terminal pro b-type natriuretic peptide levels, high-sensitivity C-reactive protein, and primary PCI), ranged from 0-36, and was well calibrated (Hosmer-Lemeshow χ² = 4.162; P = 0.842). Good discrimination was obtained both in the developmental and validation data sets (C-statistic, 0.809 and 0.798, respectively). The risk score was highly and positively associated with CI-AKI (P for trend < 0.001) in-hospital and long-term outcomes.

CONCLUSIONS

The novel risk score model we developed is a simple and accurate tool for early/preprocedural prediction of CI-AKI in patients undergoing coronary angiography or PCI. This tool allows assessment of the risk of CI-AKI before contrast exposure, allowing for timely initiation of appropriate preventive measures.

Development and validation of a prediction model for gestational hypertension in a Ghanaian cohort

OBJECTIVE

To develop and validate a prediction model for identifying women at increased risk of developing gestational hypertension (GH) in Ghana.

DESIGN

A prospective study. We used frequencies for descriptive analysis, χ² test for associations and logistic regression to derive the prediction model. Discrimination was estimated by the c-statistic. Calibration was assessed by calibration plot of actual versus predicted probability.

SETTING

Primary care antenatal clinics in Ghana.

PARTICIPANTS

2529 pregnant women in the development cohort and 647 pregnant women in the validation cohort. Inclusion criterion was women without chronic hypertension.

PRIMARY OUTCOME

Gestational hypertension.

RESULTS

Predictors of GH were diastolic blood pressure, family history of hypertension in parents, history of GH in a previous pregnancy, parity, height and weight. The c-statistic of the original model was 0.70 (95% CI 0.67-0.74) and 0.68 (0.60 to 0.77) in the validation cohort. Calibration was good in both cohorts. The negative predictive value of women in the development cohort at high risk of GH was 92.0% compared to 94.0% in the validation cohort.

CONCLUSIONS

The prediction model showed adequate performance after validation in an independent cohort and can be used to classify women into high, moderate or low risk of developing GH. It contributes to efforts to provide clinical decision-making support to improve maternal health and birth outcomes.

Methodological issues in current practice may lead to bias in the development of biomarker combinations for predicting acute kidney injury

Individual biomarkers of renal injury are only modestly predictive of acute kidney injury (AKI). Using multiple biomarkers has the potential to improve predictive capacity. In this systematic review, statistical methods of articles developing biomarker combinations to predict acute kidney injury were assessed. We identified and described three potential sources of bias (resubstitution bias, model selection bias and bias due to center differences) that may compromise the development of biomarker combinations. Fifteen studies reported developing kidney injury biomarker combinations for the prediction of AKI after cardiac surgery (8 articles), in the intensive care unit (4 articles) or other settings (3 articles). All studies were susceptible to at least one source of bias and did not account for or acknowledge the bias. Inadequate reporting often hindered our assessment of the articles. We then evaluated, when possible (7 articles), the performance of published biomarker combinations in the TRIBE-AKI cardiac surgery cohort. Predictive performance was markedly attenuated in six out of seven cases. Thus, deficiencies in analysis and reporting are avoidable and care should be taken to provide accurate estimates of risk prediction model performance. Hence, rigorous design, analysis and reporting of biomarker combination studies are essential to realizing the promise of biomarkers in clinical practice.

Ultrasound-based scores as predictors for nodular hyperplasia in patients with secondary hyperparathyroidism: a prospective validation study

PURPOSE

Former studies evaluated echostructural and vascular patterns in ultrasound of the parathyroid gland to identify nodular hyperplasia in patients with secondary hyperparathyroidism due to chronic kidney disease. This prospective study aims to externally validate suggested ultrasound classifications.

METHODS

Parathyroid glands of 27 patients with secondary hyperparathyroidism undergoing parathyroidectomy were prospectively analyzed. Ultrasound including Doppler imaging was performed 1 day prior to surgery. Ultrasound data were available for 70 parathyroid glands. Echostructural and vascular scores according to previous studies were applied calculating the area under the receiver operating characteristic curve (AUROC). Overall correctness, sensitivity, and specificity of the investigated scores were assessed with the Youden index method.

RESULTS

The Doppler score introduced by Vulpio and colleagues based on characteristic blood flow patterns in parathyroid glands showed an AUROC of 0.749 for the prediction of nodular hyperplasia with an overall correctness of 72.8%. Other ultrasound classifications based on blood flow patterns, as well as echostructure of the parathyroid gland displayed AUROCs of <0.700, thus, lacking sufficient capability as a prognostic model. Overall correctness of prediction varied from 53.8 to 55.9%.

CONCLUSIONS

The Vulpio Doppler score for the prediction of nodular hyperplasia in patients with secondary hyperparathyroidism was externally validated for the first time. Other ultrasound scores fail as prognostic models in this study population. Doppler sonography of the parathyroid gland has prognostic capability to identify nodular hyperplasia as surrogate marker for patients with secondary hyperparathyroidism indicating the need for ablative or surgical treatment when failing conservative therapy.

Hypermethylated DNA as a biomarker for colorectal cancer: a systematic review

AIM

Improved methods for early detection of colorectal cancer (CRC) are essential for increasing survival. Hypermethylated DNA in blood or stool has been proposed as a biomarker for CRC. Biochemical methods have improved in recent years, and several hypermethylated genes that are sensitive and specific for CRC have been proposed. Articles describing the use of hypermethylated promoter regions in blood or stool as biomarkers for CRC were systematically reviewed.

METHOD

A systematic literature search was performed using the Medline, Web of Science and Embase databases. Studies were included if they analysed hypermethylated genes from stool or blood samples in correlation with CRC. Studies in languages other than English and those based on animal models or cell lines were excluded.

RESULTS

The literature search yielded 74 articles, including 43 addressing blood samples and 31 addressing stool samples. In blood samples, hypermethylated ALX4, FBN2, HLTF, P16, TMEFF1 and VIM were associated with poor prognosis, hypermethylated APC, NEUROG1, RASSF1A, RASSF2A, SDC2, SEPT9, TAC1 and THBD were detected in early stage CRC and hypermethylated P16 and TFPI2 were associated with CRC recurrence. In stool samples, hypermethylated BMP3, PHACTR3, SFRP2, SPG20, TFPI2 and TMEFF2 were associated with early stage CRC.

CONCLUSION

Hypermethylation of the promoters of specific genes measured in blood or stool samples could be used as a CRC biomarker and provide prognostic information. The majority of studies, however, include only a few patients with poorly defined control groups. Further studies are therefore needed before hypermethylated DNA can be widely applied as a clinical biomarker for CRC detection and prognosis.

Albumin–bilirubin versus Child–Pugh score as a predictor of outcome after liver resection for hepatocellular carcinoma

Background

The Child–Pugh (CP) score is used widely to assess liver function and predict postoperative outcomes in patients with hepatocellular carcinoma (HCC). Recently, the albumin–bilirubin (ALBI) score has been validated as a predictor of overall survival in these patients. This study aimed to compare the ability of the ALBI and CP scores to predict outcomes in patients with HCC after liver resection with curative intent.

Methods

Consecutive patients who underwent liver resection with curative intent for HCC between January 2007 and July 2013 were included in this retrospective study. The performance of the ALBI score in predicting postoperative liver failure (PHLF) and long-term survival was compared with that of the CP score.

Results

A total of 1242 patients were enrolled. Of these, 166 (13·4 per cent) experienced PHLF. The area under the receiver operating characteristic (ROC) curve of the ALBI score for predicting PHLF was greater than that of the CP score (0·723 versus 0·607; P &lt; 0·001). Similar to findings for CP grade, the incidence and severity of PHLF increased with increasing ALBI grade. The ALBI grade stratified patients into at least two distinct overall survival cohorts (P &lt; 0·001), whereas the CP grade did not. The ALBI grade also classified patients with CP grade A disease into two distinct overall survival cohorts (P &lt; 0·001), and overall survival rates in the group with poorer survival were similar to those in the majority of patients with CP grade B disease. Both CP and ALBI scores had low power in predicting disease-free survival.

Conclusion

The ALBI grade predicted PHLF and overall survival in patients with HCC undergoing liver resection with curative intent more accurately than the CP grade.

Preoperative serum α-fetoprotein and prognosis after hepatectomy for hepatocellular carcinoma

Background

While the majority of studies report that a raised serum α-fetoprotein (AFP) level before operation is associated with a high risk of recurrence and death in patients who undergo hepatectomy for hepatocellular carcinoma (HCC), results are conflicting. The aim of this study was to assess the prognostic value of AFP.

Methods

Serum AFP levels were measured in patients with hepatitis-associated HCC who underwent hepatectomy between 1995 and 2012. Kaplan–Meier and multivariable analyses were performed to identify risk factors for overall and disease-free survival. Univariable and multivariable Cox proportional hazards regression was used to evaluate the predictive value of AFP. Receiver operating characteristic (ROC) curves were generated to identify the AFP level that had the highest accuracy in discriminating between survivors and non-survivors.

Results

Some 376 patients with hepatitis B virus (HBV)-associated HCC were included in the study. The overall survival rate was 58·8 per cent in patients with an AFP level of 400 ng/ml or less compared with 40·4 per cent for those with a level exceeding 400 ng/ml (P = 0·001). AFP concentration above 400 ng/ml was an independent risk factor for shorter disease-free and overall survival after surgery. ROC analysis indicated that the optimal cut-off values for AFP varied for different subtypes of HCC. The sensitivity and specificity were lower with areas under the ROC curve of less than 0·600. An AFP level greater than 400 ng/ml was not sensitive enough to predict the prognosis in patients with an HCC diameter smaller than 3 cm.

Conclusion

A serum AFP level above 400 ng/ml predicts poor overall and recurrence-free survival after hepatectomy in patients with HBV-associated HCC. AFP is not a strong prognostic marker given its poor discriminatory power, with low sensitivity and specificity.

Risk Prediction Tools for Hip and Knee Arthroplasty

The current healthcare environment in America is driven by the concepts of quality, cost containment, and value. In this environment, primary hip and knee arthroplasty procedures have been targeted for cost containment through quality improvement initiatives intended to reduce the incidence of costly complications and readmissions. Accordingly, risk prediction tools have been developed in an attempt to quantify the patient-specific assessment of risk. Risk prediction tools may be useful for the informed consent process, for enhancing risk mitigation efforts, and for risk-adjusting data used for reimbursement and the public reporting of outcomes. The evaluation of risk prediction tools involves statistical measures such as discrimination and calibration to assess accuracy and utility. Furthermore, prediction tools are tuned to the source dataset from which they are derived, require validation with external datasets, and should be recalibrated over time. However, a high-quality, externally validated risk prediction tool for adverse outcomes after primary total joint arthroplasty remains an elusive goal.

Standardized data collection to build prediction models in oncology: a prototype for rectal cancer

The advances in diagnostic and treatment technology are responsible for a remarkable transformation in the internal medicine concept with the establishment of a new idea of personalized medicine. Inter- and intra-patient tumor heterogeneity and the clinical outcome and/or treatment's toxicity's complexity, justify the effort to develop predictive models from decision support systems. However, the number of evaluated variables coming from multiple disciplines: oncology, computer science, bioinformatics, statistics, genomics, imaging, among others could be very large thus making traditional statistical analysis difficult to exploit. Automated data-mining processes and machine learning approaches can be a solution to organize the massive amount of data, trying to unravel important interaction. The purpose of this paper is to describe the strategy to collect and analyze data properly for decision support and introduce the concept of an 'umbrella protocol' within the framework of 'rapid learning healthcare'.

Comparison of National Operative Mortality in Gastroenterological Surgery Using Web-based Prospective Data Entry Systems

International collaboration is important in healthcare quality evaluation; however, few international comparisons of general surgery outcomes have been accomplished. Furthermore, predictive model application for risk stratification has not been internationally evaluated. The National Clinical Database (NCD) in Japan was developed in collaboration with the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP), with a goal of creating a standardized surgery database for quality improvement. The study aimed to compare the consistency and impact of risk factors of 3 major gastroenterological surgical procedures in Japan and the United States (US) using web-based prospective data entry systems: right hemicolectomy (RH), low anterior resection (LAR), and pancreaticoduodenectomy (PD).Data from NCD and ACS-NSQIP, collected over 2 years, were examined. Logistic regression models were used for predicting 30-day mortality for both countries. Models were exchanged and evaluated to determine whether the models built for one population were accurate for the other population.We obtained data for 113,980 patients; 50,501 (Japan: 34,638; US: 15,863), 42,770 (Japan: 35,445; US: 7325), and 20,709 (Japan: 15,527; US: 5182) underwent RH, LAR, and, PD, respectively. Thirty-day mortality rates for RH were 0.76% (Japan) and 1.88% (US); rates for LAR were 0.43% versus 1.08%; and rates for PD were 1.35% versus 2.57%. Patient background, comorbidities, and practice style were different between Japan and the US. In the models, the odds ratio for each variable was similar between NCD and ACS-NSQIP. Local risk models could predict mortality using local data, but could not accurately predict mortality using data from other countries.We demonstrated the feasibility and efficacy of the international collaborative research between Japan and the US, but found that local risk models remain essential for quality improvement.

Association of worker characteristics and early reimbursement for physical therapy, chiropractic and opioid prescriptions with workers' compensation claim duration, for cases of acute low back pain: an observational cohort study

OBJECTIVE

To assess the association between early reimbursement for physiotherapy, chiropractic and opioid prescriptions for acute low back pain (LBP) with disability claim duration.

DESIGN

Observational cohort study.

SETTING AND PARTICIPANTS

From a random sample of 6665 claims for acute, uncomplicated LBP approved by the Ontario Workplace Safety and Insurance Board (WSIB) in 2005, we analysed 1442 who remained on full benefits at 4 weeks after claim approval.

PRIMARY OUTCOME MEASURE

Our primary outcome was WSIB claim duration.

RESULTS

We had complete data for all but 3 variables, which had <15% missing data, and we included missing data as a category for these factors. Our time-to-event analysis was adjusted for demographic, workplace and treatment factors, but not injury severity, although we attempted to include a sample with very similar, less-severe injuries. Regarding significant factors and treatment variables in our adjusted analysis, older age (eg, HR for age ≥ 55 vs <25=0.52; 99% CI 0.36 to 0.74) and WSIB reimbursement for opioid prescription in the first 4 weeks of a claim (HR=0.68; 99% CI 0.53 to 0.88) were associated with longer claim duration. Higher predisability income was associated with longer claim duration, but only among persistent claims (eg, HR for active claims at 1 year with a predisability income >$920 vs ≤$480/week=0.34; 99% CI 0.17 to 0.68). Missing data for union membership (HR=1.27; 99% CI 1.01 to 1.59), and working for an employer with a return-to-work programme were associated with fewer days on claim (HR=1.78; 99% CI 1.45 to 2.18). Neither reimbursement for physiotherapy (HR=1.01; 99% CI 0.86 to 1.19) nor chiropractic care (HR for active claims at 60 days=1.15; 99% CI 0.94 to 1.41) within the first 4 weeks was associated with claim duration. Our meta-analysis of 3 studies (n=51,069 workers) confirmed a strong association between early opioid use and prolonged claim duration (HR=0.57, 95% CI 0.48 to 0.69; low certainty evidence).

CONCLUSIONS

Our analysis found that early WSIB reimbursement for physiotherapy or chiropractic care, in claimants fully off work for more than 4 weeks, was not associated with claim duration, and that early reimbursement for opioids predicted prolonged claim duration. Well-designed randomised controlled trials are needed to verify our findings and establish causality between these variables and claim duration.

Scoring system to distinguish uncomplicated from complicated acute appendicitis

Background

Non-operative management may be an alternative for uncomplicated appendicitis, but preoperative distinction between uncomplicated and complicated disease is challenging. This study aimed to develop a scoring system based on clinical and imaging features to distinguish uncomplicated from complicated appendicitis.

Methods

Patients with suspected acute appendicitis based on clinical evaluation and imaging were selected from two prospective multicentre diagnostic accuracy studies (OPTIMA and OPTIMAP). Features associated with complicated appendicitis were included in multivariable logistic regression analyses. Separate models were developed for CT and ultrasound imaging, internally validated and transformed into scoring systems.

Results

A total of 395 patients with suspected acute appendicitis based on clinical evaluation and imaging were identified, of whom 110 (27·8 per cent) had complicated appendicitis, 239 (60·5 per cent) had uncomplicated appendicitis and 46 (11·6 per cent) had an alternative disease. CT was positive for appendicitis in 284 patients, and ultrasound imaging in 312. Based on clinical and CT features, a model was created including age, body temperature, duration of symptoms, white blood cell count, C-reactive protein level, and presence of extraluminal free air, periappendiceal fluid and appendicolith. A scoring system was constructed, with a maximum possible score of 22 points. Of the 284 patients, 150 had a score of 6 points or less, of whom eight (5·3 per cent) had complicated appendicitis, giving a negative predictive value (NPV) of 94·7 per cent. The model based on ultrasound imaging included the same predictors except for extraluminal free air. The ultrasound score (maximum 19 points) was calculated for 312 patients; 105 had a score of 5 or less, of whom three (2·9 per cent) had complicated appendicitis, giving a NPV of 97·1 per cent.

Conclusion

With use of novel scoring systems combining clinical and imaging features, 95 per cent of the patients deemed to have uncomplicated appendicitis were correctly identified as such. The score can aid in selection for non-operative management in clinical trials.