Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration

Miscoding of Melanoma Thickness in SEER: Research and Clinical Implications

Melanoma-related deaths and metastases among patients with thin (≤1 mm) and ultrathin (≤0.25 mm) melanomas have been reported. These observations might reflect adverse biology and/or errors in administrative data. Cumulative melanoma-related death rates for thickness groups of patients with thin melanomas were compared among five cohorts including the Surveillance, Epidemiology, and End Results (SEER) registry. Thickness in one SEER region was reexamined in pathology reports. The 5-year cumulative melanoma-related death rate of patients with ultrathin melanomas was higher in SEER (2.8%) compared with other registries (0.6-0.9%). The rates across the 16 SEER regions were 0.25% to 8.4%. In SEER, 21% of thin melanomas were ultrathin; in other registries, they comprised 5.8-15%. A reexamination of thickness in one SEER site revealed that 114 of 447 ultrathin melanomas had errors; after correction, only 17 of the 114 remained ultrathin. The majority of errors were related to decimal point placement. The 86 thin melanomas reclassified to >1.00 mm included 96% of the original ultrathin-associated deaths and 100% of the original positive lymph nodes. Significant miscoding of thickness that is concentrated in ultrathin lesions is present in SEER and results in mischaracterization of patient outcomes. When using administrative data, validation of results can identify critical data issues.

Plasma N-terminal Prosomatostatin and Risk of Incident Cardiovascular Disease and All-Cause Mortality in a Prospective Observational Cohort: the PREVEND Study

BACKGROUND

Somatostatin is a component of the well-known insulin-like growth factor-1/growth hormone (GH) longevity axis. There is observational evidence that increased GH is associated with an increased risk of cardiovascular disease (CVD). We aimed to investigate the potential association of plasma N-terminal fragment prosomatostatin (NT-proSST) with incident CVD and all-cause mortality in apparently healthy adults.

METHODS

We studied 8134 participants without history of CVD (aged 28-75 years; women, 52.6%) from the Prevention of Renal and Vascular End-stage Disease (PREVEND) study in Groningen, the Netherlands. Plasma NT-proSST was measured in baseline samples. Outcomes were incidence of CVD and all-cause mortality.

RESULTS

In cross-sectional analyses, NT-proSST [mean (SD), 384.0 (169.3) pmol/L] was positively associated with male sex and age (both P < 0.001). During a median follow-up of 10.5 (Q1-Q3: 9.9-10.8) years, 708 (8.7%) participants developed CVD and 517 (6.4%) participants died. In univariable analyses, NT-proSST was associated with an increased risk of incident CVD and all-cause mortality (both P < 0.001). In multivariable analyses, these associations were independent of the Framingham risk factors, with hazard ratios (95% CI) per doubling of NT-proSST of 1.17 (1.03-1.34; P = 0.02) for incident CVD and of 1.28 (1.09-1.49; P = 0.002) for all-cause mortality. Addition of NT-proSST to the updated Framingham Risk Score improved reclassification (integrated discrimination improvement (P < 0.001); net reclassification improvement was 2.5% (P = 0.04)).

CONCLUSIONS

Plasma NT-proSST is positively associated with increased risk of future CVD and all-cause mortality, partly independent of traditional CVD risk factors. Further research is needed to address the nature of associations.

External validation of a multivariable claims-based rule for predicting in-hospital mortality and 30-day post-pulmonary embolism complications

BACKGROUND

Low-risk pulmonary embolism (PE) patients may be candidates for outpatient treatment or abbreviated hospital stay. There is a need for a claims-based prediction rule that payers/hospitals can use to risk stratify PE patients. We sought to validate the In-hospital Mortality for PulmonAry embolism using Claims daTa (IMPACT) prediction rule for in-hospital and 30-day outcomes.

METHODS

We used the Optum Research Database from 1/2008-3/2015 and included adults hospitalized for PE (415.1x in the primary position or secondary position when accompanied by a primary code for a PE complication) and having continuous medical and prescription coverage for ≥6-months prior and 3-months post-inclusion or until death. In-hospital and 30-day mortality and 30-day complications (recurrent venous thromboembolism, rehospitalization or death) were assessed and prognostic accuracies of IMPACT with 95 % confidence intervals (CIs) were calculated.

RESULTS

In total, 47,531 PE patients were included. In-hospital and 30-day mortality occurred in 7.9 and 9.4 % of patients and 20.8 % experienced any complication within 30-days. Of the 19.5 % of patients classified as low-risk by IMPACT, 2.0 % died in-hospital, resulting in a sensitivity and specificity of 95.2 % (95 % CI, 94.4-95.8) and 20.7 % (95 % CI, 20.4-21.1). Only 1 additional low-risk patient died within 30-days of admission and 12.2 % experienced a complication, translating into a sensitivity and specificity of 95.9 % (95 % CI, 95.3-96.5) and 21.1 % (95 % CI, 20.7-21.5) for mortality and 88.5 % (95 % CI, 87.9-89.2) and 21.6 % (95 % CI, 21.2-22.0) for any complication.

CONCLUSION

IMPACT had acceptable sensitivity for predicting in-hospital and 30-day mortality or complications and may be valuable for retrospective risk stratification of PE patients.

Quantifying the impact of different approaches for handling continuous predictors on the performance of a prognostic model

Continuous predictors are routinely encountered when developing a prognostic model. Investigators, who are often non-statisticians, must decide how to handle continuous predictors in their models. Categorising continuous measurements into two or more categories has been widely discredited, yet is still frequently done because of its simplicity, investigator ignorance of the potential impact and of suitable alternatives, or to facilitate model uptake. We examine three broad approaches for handling continuous predictors on the performance of a prognostic model, including various methods of categorising predictors, modelling a linear relationship between the predictor and outcome and modelling a nonlinear relationship using fractional polynomials or restricted cubic splines. We compare the performance (measured by the c-index, calibration and net benefit) of prognostic models built using each approach, evaluating them using separate data from that used to build them. We show that categorising continuous predictors produces models with poor predictive performance and poor clinical usefulness. Categorising continuous predictors is unnecessary, biologically implausible and inefficient and should not be used in prognostic model development. © 2016 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.

The long-term outcome of subthreshold depression in later life

BACKGROUND

Subthreshold depression (SUBD) in later life is common and important as prodromal state and prominent risk factor in the development of major depressive disorder (MDD). Indicated prevention can reduce the incidence of MDD among people with SUBD substantially, but needs to be targeted to those that are truly at risk of developing MDD.

METHOD

N = 341 eligible participants with SUBD were included from the first (1992/1993), second (1995/1996) and third (1998/1999) cycle from the Longitudinal Aging Study Amsterdam (LASA) by using a two-stage screening design. LASA is an ongoing prospective cohort study in The Netherlands among the older population (55-85 years). At baseline (1992/1993) N = 3107 participants were interviewed and follow-up cycles were conducted every 3 years until 2008/2009, resulting in maximal 17 years of observational period. The proportion of people that developed MDD, remained SUBD, or recovered from SUBD was measured and Cox proportional regression analyses were performed to investigate 29 putative predictors of MDD and recovery from SUBD.

RESULTS

N = 153 (44.9%) recovered from SUBD, N = 138 (40.5%) remained chronically SUBD, and N = 50 (14.7%) developed MDD (incidence rate 15.1/1000 person-years). Women, high neuroticism, more chronic diseases, high body mass index, smoking and less social support predicted conversion to MDD. Men, low neuroticism and absence of pain predicted recovery from SUBD.

CONCLUSIONS

Although older people with SUBD are clearly at risk of developing MDD, the majority did not, even after a long and thorough follow-up. Given the risk factors that were uncovered, targeting and prevention of MDD in those at very high risk is feasible.

Predicting the 10-Year Risks of Atherosclerotic Cardiovascular Disease in Chinese Population: The China-PAR Project (Prediction for ASCVD Risk in China)

BACKGROUND

The accurate assessment of individual risk can be of great value to guiding and facilitating the prevention of atherosclerotic cardiovascular disease (ASCVD). However, prediction models in common use were formulated primarily in white populations. The China-PAR project (Prediction for ASCVD Risk in China) is aimed at developing and validating 10-year risk prediction equations for ASCVD from 4 contemporary Chinese cohorts.

METHODS

Two prospective studies followed up together with a unified protocol were used as the derivation cohort to develop 10-year ASCVD risk equations in 21 320 Chinese participants. The external validation was evaluated in 2 independent Chinese cohorts with 14 123 and 70 838 participants. Furthermore, model performance was compared with the Pooled Cohort Equations reported in the American College of Cardiology/American Heart Association guideline.

RESULTS

Over 12 years of follow-up in the derivation cohort with 21 320 Chinese participants, 1048 subjects developed a first ASCVD event. Sex-specific equations had C statistics of 0.794 (95% confidence interval, 0.775-0.814) for men and 0.811 (95% confidence interval, 0.787-0.835) for women. The predicted rates were similar to the observed rates, as indicated by a calibration χ² of 13.1 for men (P=0.16) and 12.8 for women (P=0.17). Good internal and external validations of our equations were achieved in subsequent analyses. Compared with the Chinese equations, the Pooled Cohort Equations had lower C statistics and much higher calibration χ² values in men.

CONCLUSIONS

Our project developed effective tools with good performance for 10-year ASCVD risk prediction among a Chinese population that will help to improve the primary prevention and management of cardiovascular disease.

Refining Prognosis in Lung Cancer: A Report on the Quality and Relevance of Clinical Prognostic Tools

INTRODUCTION

Accurate, individualized prognostication for lung cancer patients requires the integration of standard patient and pathologic factors, biological, genetic, and other molecular characteristics of the tumor. Clinical prognostic tools aim to aggregate information on an individual patient to predict disease outcomes such as overall survival, but little is known about their clinical utility and accuracy in lung cancer.

METHODS

A systematic search of the scientific literature for clinical prognostic tools in lung cancer published from January 1, 1996 to January 27, 2015 was performed. In addition, web-based resources were searched. A priori criteria determined by the Molecular Modellers Working Group of the American Joint Committee on Cancer were used to investigate the quality and usefulness of tools. Criteria included clinical presentation, model development approaches, validation strategies, and performance metrics.

RESULTS

Thirty-two prognostic tools were identified. Patients with metastases were the most frequently considered population in non-small-cell lung cancer. All tools for small-cell lung cancer covered that entire patient population. Included prognostic factors varied considerably across tools. Internal validity was not formally evaluated for most tools and only 11 were evaluated for external validity. Two key considerations were highlighted for tool development: identification of an explicit purpose related to a relevant clinical population and clear decision points and prioritized inclusion of established prognostic factors over emerging factors.

CONCLUSIONS

Prognostic tools will contribute more meaningfully to the practice of personalized medicine if better study design and analysis approaches are used in their development and validation.

Does ignoring clustering in multicenter data influence the performance of prediction models? A simulation study

Clinical risk prediction models are increasingly being developed and validated on multicenter datasets. In this article, we present a comprehensive framework for the evaluation of the predictive performance of prediction models at the center level and the population level, considering population-averaged predictions, center-specific predictions, and predictions assuming an average random center effect. We demonstrated in a simulation study that calibration slopes do not only deviate from one because of over- or underfitting of patterns in the development dataset, but also as a result of the choice of the model (standard versus mixed effects logistic regression), the type of predictions (marginal versus conditional versus assuming an average random effect), and the level of model validation (center versus population). In particular, when data is heavily clustered (ICC 20%), center-specific predictions offer the best predictive performance at the population level and the center level. We recommend that models should reflect the data structure, while the level of model validation should reflect the research question.

American Joint Committee on Cancer acceptance criteria for inclusion of risk models for individualized prognosis in the practice of precision medicine

The American Joint Committee on Cancer (AJCC) has increasingly recognized the need for more personalized probabilistic predictions than those delivered by ordinal staging systems, particularly through the use of accurate risk models or calculators. However, judging the quality and acceptability of a risk model is complex. The AJCC Precision Medicine Core conducted a 2-day meeting to discuss characteristics necessary for a quality risk model in cancer patients. More specifically, the committee established inclusion and exclusion criteria necessary for a risk model to potentially be endorsed by the AJCC. This committee reviewed and discussed relevant literature before creating a checklist unique to this need of AJCC risk model endorsement. The committee identified 13 inclusion and 3 exclusion criteria for AJCC risk model endorsement in cancer. The emphasis centered on performance metrics, implementation clarity, and clinical relevance. The facilitation of personalized probabilistic predictions for cancer patients holds tremendous promise, and these criteria will hopefully greatly accelerate this process. Moreover, these criteria might be useful for a general audience when trying to judge the potential applicability of a published risk model in any clinical domain. CA Cancer J Clin 2016;66:370-374. © 2016 American Cancer Society.

Validation of Risk Assessment Models of Venous Thromboembolism in Hospitalized Medical Patients

BACKGROUND

Patients hospitalized for acute medical illness are at increased risk for venous thromboembolism. Although risk assessment is recommended and several at-admission risk assessment models have been developed, these have not been adequately derived or externally validated. Therefore, an optimal approach to evaluate venous thromboembolism risk in medical patients is not known.

METHODS

We conducted an external validation study of existing venous thromboembolism risk assessment models using data collected on 63,548 hospitalized medical patients as part of the Michigan Hospital Medicine Safety (HMS) Consortium. For each patient, cumulative venous thromboembolism risk scores and risk categories were calculated. Cox regression models were used to quantify the association between venous thromboembolism events and assigned risk categories. Model discrimination was assessed using Harrell's C-index.

RESULTS

Venous thromboembolism incidence in hospitalized medical patients is low (1%). Although existing risk assessment models demonstrate good calibration (hazard ratios for "at-risk" range 2.97-3.59), model discrimination is generally poor for all risk assessment models (C-index range 0.58-0.64).

CONCLUSIONS

The performance of several existing risk assessment models for predicting venous thromboembolism among acutely ill, hospitalized medical patients at admission is limited. Given the low venous thromboembolism incidence in this nonsurgical patient population, careful consideration of how best to utilize existing venous thromboembolism risk assessment models is necessary, and further development and validation of novel venous thromboembolism risk assessment models for this patient population may be warranted.

Performance and Validation of a Novel Biomarker-Based Stroke Risk Score for Atrial Fibrillation

BACKGROUND

Atrial fibrillation is associated with increased but variable risk of stroke. Our aim was to validate the recently developed biomarker-based ABC (age, biomarkers [high-sensitivity troponin and N-terminal fragment B-type natriuretic peptide], and clinical history of prior stroke/transient ischemic attack)-stroke risk score and compare its performance with the CHA₂DS₂VASc and ATRIA (Anticoagulation and Risk Factors in Atrial Fibrillation) risk scores.

METHODS

The ABC-stroke score includes age, biomarkers (N-terminal fragment B-type natriuretic peptide and high-sensitivity cardiac troponin), and clinical history (prior stroke). This validation was based on 8356 patients, 16 137 person-years of follow-up, and 219 adjudicated stroke or systemic embolic events in anticoagulated patients with atrial fibrillation in the RE-LY study (Randomized Evaluation of Long-Term Anticoagulation Therapy). Levels of N-terminal fragment B-type natriuretic peptide, high-sensitivity cardiac troponin T (hs-cTnT), and high-sensitivity cardiac troponin I (hs-cTnI) were determined in plasma samples obtained at study entry.

RESULTS

The ABC-stroke score was well calibrated with 0.76 stroke/systemic embolic events per 100 person-years in the predefined low (<1%/y) risk group, 1.48 in the medium (1%-2%/y) risk group, and 2.60 in the high (>2%/y) risk group for the ABC-stroke score with hs-cTnT. Hazard ratios for stroke/systemic embolic events were 1.95 for medium- versus low-risk groups, and 3.44 for high- versus low-risk groups. ABC-stroke score achieved C indices of 0.65 with both hs-cTnT and hs-cTnI, in comparison with 0.60 for CHA₂DS₂VASc (P=0.004 for hs-cTnT and P=0.022 hs-cTnI) and 0.61 for ATRIA scores (P=0.005 hs-cTnT and P=0.034 for hs-cTnI).

CONCLUSIONS

The biomarker-based ABC-stroke score was well calibrated and consistently performed better than both the CHA₂DS₂VASc and ATRIA stroke scores. The ABC score should be considered an improved decision support tool in the care of patients with atrial fibrillation.

CLINICAL TRIAL REGISTRATION

URL: http://www.clinicaltrials.gov. Unique identifiers: ARISTOTLE, NCT00412984; RE-LY, NCT00262600.

External validation of a prehospital risk score for critical illness

Background

Identification of critically ill patients during prehospital care could facilitate early treatment and aid in the regionalization of critical care. Tools to consistently identify those in the field with or at higher risk of developing critical illness do not exist. We sought to validate a prehospital critical illness risk score that uses objective clinical variables in a contemporary cohort of geographically and temporally distinct prehospital encounters.

Methods

We linked prehospital encounters at 21 emergency medical services (EMS) agencies to inpatient electronic health records at nine hospitals in southwestern Pennsylvania from 2010 to 2012. The primary outcome was critical illness during hospitalization, defined as an intensive care unit stay with delivery of organ support (mechanical ventilation or vasopressor use). We calculated the prehospital risk score using demographics and first vital signs from eligible EMS encounters, and we tested the association between score variables and critical illness using multivariable logistic regression. Discrimination was assessed using the AUROC curve, and calibration was determined by plotting observed versus expected events across score values. Operating characteristics were calculated at score thresholds.

Results

Among 42,550 nontrauma, non-cardiac arrest adult EMS patients, 1926 (4.5 %) developed critical illness during hospitalization. We observed moderate discrimination of the prehospital critical illness risk score (AUROC 0.73, 95 % CI 0.72–0.74) and adequate calibration based on observed versus expected plots. At a score threshold of 2, sensitivity was 0.63 (95 % CI 0.61–0.75), specificity was 0.73 (95 % CI 0.72–0.73), negative predictive value was 0.98 (95 % CI 0.98–0.98), and positive predictive value was 0.10 (95 % CI 0.09–0.10). The risk score performance was greater with alternative definitions of critical illness, including in-hospital mortality (AUROC 0.77, 95 % CI 0.7 –0.78).

Conclusions

In an external validation cohort, a prehospital risk score using objective clinical data had moderate discrimination for critical illness during hospitalization.

Electronic supplementary material

The online version of this article (doi:10.1186/s13054-016-1408-0) contains supplementary material, which is available to authorized users.

Risk scores for outcome in bacterial meningitis: Systematic review and external validation study

OBJECTIVES

To perform an external validation study of risk scores, identified through a systematic review, predicting outcome in community-acquired bacterial meningitis.

METHODS

MEDLINE and EMBASE were searched for articles published between January 1960 and August 2014. Performance was evaluated in 2108 episodes of adult community-acquired bacterial meningitis from two nationwide prospective cohort studies by the area under the receiver operating characteristic curve (AUC), the calibration curve, calibration slope or Hosmer-Lemeshow test, and the distribution of calculated risks.

FINDINGS

Nine risk scores were identified predicting death, neurological deficit or death, or unfavorable outcome at discharge in bacterial meningitis, pneumococcal meningitis and invasive meningococcal disease. Most studies had shortcomings in design, analyses, and reporting. Evaluation showed AUCs of 0.59 (0.57-0.61) and 0.74 (0.71-0.76) in bacterial meningitis, 0.67 (0.64-0.70) in pneumococcal meningitis, and 0.81 (0.73-0.90), 0.82 (0.74-0.91), 0.84 (0.75-0.93), 0.84 (0.76-0.93), 0.85 (0.75-0.95), and 0.90 (0.83-0.98) in meningococcal meningitis. Calibration curves showed adequate agreement between predicted and observed outcomes for four scores, but statistical tests indicated poor calibration of all risk scores.

INTERPRETATION

One score could be recommended for the interpretation and design of bacterial meningitis studies. None of the existing scores performed well enough to recommend routine use in individual patient management.

Development of a Risk Prediction Model to Individualize Risk Factors for Surgical Site Infection After Mastectomy

BACKGROUND

Little data are available regarding individual patients' risk of surgical site infection (SSI) following mastectomy with or without immediate reconstruction. Our objective was to develop a risk prediction model for mastectomy-related SSI.

METHODS

Using commercial claims data, we established a cohort of women <65 years of age who underwent a mastectomy from 1 January 2004-31 December 2011. International Classification of Diseases, Ninth Revision, Clinical Modification diagnosis codes were used to identify SSI within 180 days after surgery. SSI risk factors were determined with multivariable logistic regression using derivation data from 2004-2008 and validated with 2009-2011 data using discrimination and calibration measures.

RESULTS

In the derivation cohort, 595 SSIs were identified in 7607 (7.8 %) women, and 396 SSIs were coded in 4366 (9.1 %) women in the validation cohort. Independent risk factors for SSIs included rural residence, rheumatologic disease, depression, diabetes, hypertension, liver disease, obesity, pre-existing pneumonia or urinary tract infection, tobacco use disorder, smoking-related diseases, bilateral mastectomy, and immediate reconstruction. Receipt of home healthcare was associated with lower risk. The model performed equally in the validation cohort per discrimination (C-statistics 0.657 and 0.649) and calibration (Hosmer-Lemeshow p = 0.091 and 0.462 for derivation and validation, respectively). Three risk strata were created based on predicted SSI risk, which demonstrated good correlation with the proportion of observed infections in the strata.

CONCLUSIONS

We developed and internally validated an SSI risk prediction model that can be used to counsel women with regard to their individual risk of SSI post-mastectomy. Immediate reconstruction, diabetes, and smoking-related diseases were important risk factors for SSI in this non-elderly population of women undergoing mastectomy.

The IASLC Lung Cancer Staging Project: Methodology and Validation Used in the Development of Proposals for Revision of the Stage Classification of NSCLC in the Forthcoming (Eighth) Edition of the TNM Classification of Lung Cancer

INTRODUCTION

Stage classification provides a consistent language to describe the anatomic extent of disease and is therefore a critical tool in caring for patients. The Staging and Prognostic Factors Committee of the International Association for the Study of Lung Cancer developed proposals for revision of the classification of lung cancer for the eighth edition of the tumor, node, and metastasis (TNM) classification, which takes effect in 2017.

METHODS

An international database of 94,708 patients with lung cancer diagnosed in 1999-2010 was assembled. This article describes the process and statistical methods used to refine the lung cancer stage classification.

RESULTS

Extensive analysis allowed definition of tumor, node, and metastasis categories and stage groupings that demonstrated consistent discrimination overall and within multiple different patient cohorts (e.g., clinical or pathologic stage, R0 or R-any resection status, geographic region). Additional analyses provided evidence of applicability over time, across a spectrum of geographic regions, histologic types, evaluative approaches, and follow-up intervals.

CONCLUSIONS

An extensive analysis has produced stage classification proposals for lung cancer with a robust degree of discriminatory consistency and general applicability. Nevertheless, external validation is encouraged to identify areas of strength and weakness; a sound validation should have discriminatory ability and be based on an independent data set of adequate size and sufficient follow-up with enough patients for each subgroup.

Development and validation of a machine learning-based predictive model to improve the prediction of inguinal status of anal cancer patients: A preliminary report

Introduction

The role of prophylactic inguinal irradiation (PII) in the treatment of anal cancer patients is controversial. We developped an innovative algorithm based on the Machine Learning (ML) allowing the tailoring of the prescription of PII.

Results

Once verified on the independent testing set, J48 showed the better performances, with specificity, sensitivity, and accuracy rates in predicting relapsing patients of 86.4%, 50.0% and 83.1% respectively (vs 36.5%, 90.4% and 80.25%, respectively, for LR).

Methods

We classified 194 anal cancer patients with Logistic Regression (LR) and other 3 ML techniques based on decision trees (J48, Random Tree and Random Forest), using a large set of clinical and therapeutic variables. We tested obtained ML algorithms on an independent testing set of 65 anal cancer patients. TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis) methodology was used for the development, the Quality Assurance and the description of the experimental procedures.

Conclusion

In an internationally approved quality assurance framework, ML seems promising in predicting the outcome of patients that would benefit or not of the PII. Once confirmed in larger and/or multi-centric databases, ML could support the physician in tailoring the treatment and in deciding if deliver or not the PII.

An external validation of models to predict the onset of chronic kidney disease using population-based electronic health records from Salford, UK

BACKGROUND

Chronic kidney disease (CKD) is a major and increasing constituent of disease burdens worldwide. Early identification of patients at increased risk of developing CKD can guide interventions to slow disease progression, initiate timely referral to appropriate kidney care services, and support targeting of care resources. Risk prediction models can extend laboratory-based CKD screening to earlier stages of disease; however, to date, only a few of them have been externally validated or directly compared outside development populations. Our objective was to validate published CKD prediction models applicable in primary care.

METHODS

We synthesised two recent systematic reviews of CKD risk prediction models and externally validated selected models for a 5-year horizon of disease onset. We used linked, anonymised, structured (coded) primary and secondary care data from patients resident in Salford (population ~234 k), UK. All adult patients with at least one record in 2009 were followed-up until the end of 2014, death, or CKD onset (n = 178,399). CKD onset was defined as repeated impaired eGFR measures over a period of at least 3 months, or physician diagnosis of CKD Stage 3-5. For each model, we assessed discrimination, calibration, and decision curve analysis.

RESULTS

Seven relevant CKD risk prediction models were identified. Five models also had an associated simplified scoring system. All models discriminated well between patients developing CKD or not, with c-statistics around 0.90. Most of the models were poorly calibrated to our population, substantially over-predicting risk. The two models that did not require recalibration were also the ones that had the best performance in the decision curve analysis.

CONCLUSIONS

Included CKD prediction models showed good discriminative ability but over-predicted the actual 5-year CKD risk in English primary care patients. QKidney, the only UK-developed model, outperformed the others. Clinical prediction models should be (re)calibrated for their intended uses.

Colorectal cancer health services research study protocol: the CCR-CARESS observational prospective cohort project

BACKGROUND

Colorectal cancers are one of the most common forms of malignancy worldwide. But two significant areas of research less studied deserve attention: health services use and development of patient stratification risk tools for these patients.

METHODS

DESIGN

a prospective multicenter cohort study with a follow up period of up to 5 years after surgical intervention. Participant centers: 22 hospitals representing six autonomous communities of Spain. Participants/Study population: Patients diagnosed with colorectal cancer that have undergone surgical intervention and have consented to participate in the study between June 2010 and December 2012. Variables collected include pre-intervention background, sociodemographic parameters, hospital admission records, biological and clinical parameters, treatment information, and outcomes up to 5 years after surgical intervention. Patients completed the following questionnaires prior to surgery and in the follow up period: EuroQol-5D, EORTC QLQ-C30 (The European Organization for Research and Treatment of Cancer quality of life questionnaire) and QLQ-CR29 (module for colorectal cancer), the Duke Functional Social Support Questionnaire, the Hospital Anxiety and Depression Scale, and the Barthel Index. The main endpoints of the study are mortality, tumor recurrence, major complications, readmissions, and changes in health-related quality of life at 30 days and at 1, 2, 3 and 5 years after surgical intervention.

STATISTICAL ANALYSIS

In relation to the different endpoints, predictive models will be used by means of multivariate logistic models, Cox or linear mixed-effects regression models. Simulation models for the prediction of discrete events in the long term will also be used, and an economic evaluation of different treatment strategies will be performed through the use of generalized linear models.

DISCUSSION

The identification of potential risk factors for adverse events may help clinicians in the clinical decision making process. Also, the follow up by 5 years of this large cohort of patients may provide useful information to answer different health services research questions.

TRIAL REGISTRATION

ClinicalTrials.gov Identifier: NCT02488161 . Registration date: June 16, 2015.

Normal tissue complication probability (NTCP) modelling using spatial dose metrics and machine learning methods for severe acute oral mucositis resulting from head and neck radiotherapy

BACKGROUND AND PURPOSE

Severe acute mucositis commonly results from head and neck (chemo)radiotherapy. A predictive model of mucositis could guide clinical decision-making and inform treatment planning. We aimed to generate such a model using spatial dose metrics and machine learning.

MATERIALS AND METHODS

Predictive models of severe acute mucositis were generated using radiotherapy dose (dose-volume and spatial dose metrics) and clinical data. Penalised logistic regression, support vector classification and random forest classification (RFC) models were generated and compared. Internal validation was performed (with 100-iteration cross-validation), using multiple metrics, including area under the receiver operating characteristic curve (AUC) and calibration slope, to assess performance. Associations between covariates and severe mucositis were explored using the models.

RESULTS

The dose-volume-based models (standard) performed equally to those incorporating spatial information. Discrimination was similar between models, but the RFCstandard had the best calibration. The mean AUC and calibration slope for this model were 0.71 (s.d.=0.09) and 3.9 (s.d.=2.2), respectively. The volumes of oral cavity receiving intermediate and high doses were associated with severe mucositis.

CONCLUSIONS

The RFCstandard model performance is modest-to-good, but should be improved, and requires external validation. Reducing the volumes of oral cavity receiving intermediate and high doses may reduce mucositis incidence.

Long noncoding RNAs as novel predictors of survival in human cancer: a systematic review and meta-analysis

Background

Expression of various long noncoding RNAs (lncRNAs) may affect cancer prognosis. Here, we aim to gather and examine all evidence on the potential role of lncRNAs as novel predictors of survival in human cancer.

Methods

We systematically searched through PubMed, to identify all published studies reporting on the association between any individual lncRNA or group of lncRNAs with prognosis in human cancer (death or other clinical outcomes). Where appropriate, we then performed quantitative synthesis of those results using meta-analytic methods to identify the true effect size of lncRNAs on cancer prognosis. The reliability of those results was then examined using measures of heterogeneity and testing for selective reporting biases.

Results

Three hundred ninety-two studies were screened to eventually identify 111 eligible studies on 127 datasets. In total, these represented 16,754 independent participants pertaining to 53 individual and 6 grouped lncRNAs within a total of 19 cancer sites. Overall, 83 % of the studies we identified addressed overall survival and 32 % of the studies addressed recurrence-free survival. For overall survival, 96 % (88/92) of studies identified a statistically significant association of lncRNA expression to prognosis. Meta-analysis of 6 out of 7 lncRNAs for which three or more studies were available, identified statistically significant associations with overall survival. The lncRNA HOTAIR was by far the most broadly studied lncRNA (n = 29; of 111 studies) and featured a summary hazard ratio (HR) of 2.22 (95 % confidence interval (CI), 1.86–2.65) with modest heterogeneity (I² = 49 %; 95 % CI, 14–79 %). Prominent excess significance was demonstrated across all meta-analyses (p-value = 0.0003), raising the possibility of substantial selective reporting biases.

Conclusions

Multiple lncRNAs have been shown to be strongly associated with prognosis in diverse cancers, but substantial bias cannot be excluded in this field and larger studies are needed to understand whether these prognostic information may eventually be useful.

Electronic supplementary material

The online version of this article (doi:10.1186/s12943-016-0535-1) contains supplementary material, which is available to authorized users.

Radiomics Signature: A Potential Biomarker for the Prediction of Disease-Free Survival in Early-Stage (I or II) Non-Small Cell Lung Cancer

Purpose To develop a radiomics signature to estimate disease-free survival (DFS) in patients with early-stage (stage I-II) non-small cell lung cancer (NSCLC) and assess its incremental value to the traditional staging system and clinical-pathologic risk factors for individual DFS estimation. Materials and Methods Ethical approval by the institutional review board was obtained for this retrospective analysis, and the need to obtain informed consent was waived. This study consisted of 282 consecutive patients with stage IA-IIB NSCLC. A radiomics signature was generated by using the least absolute shrinkage and selection operator, or LASSO, Cox regression model. Association between the radiomics signature and DFS was explored. Further validation of the radiomics signature as an independent biomarker was performed by using multivariate Cox regression. A radiomics nomogram with the radiomics signature incorporated was constructed to demonstrate the incremental value of the radiomics signature to the traditional staging system and other clinical-pathologic risk factors for individualized DFS estimation, which was then assessed with respect to calibration, discrimination, reclassification, and clinical usefulness. Results The radiomics signature was significantly associated with DFS, independent of clinical-pathologic risk factors. Incorporating the radiomics signature into the radiomics-based nomogram resulted in better performance (P < .0001) for the estimation of DFS (C-index: 0.72; 95% confidence interval [CI]: 0.71, 0.73) than with the clinical-pathologic nomogram (C-index: 0.691; 95% CI: 0.68, 0.70), as well as a better calibration and improved accuracy of the classification of survival outcomes (net reclassification improvement: 0.182; 95% CI: 0.02, 0.31; P = .02). Decision curve analysis demonstrated that in terms of clinical usefulness, the radiomics nomogram outperformed the traditional staging system and the clinical-pathologic nomogram. Conclusion The radiomics signature is an independent biomarker for the estimation of DFS in patients with early-stage NSCLC. Combination of the radiomics signature, traditional staging system, and other clinical-pathologic risk factors performed better for individualized DFS estimation in patients with early-stage NSCLC, which might enable a step forward precise medicine. ^© RSNA, 2016 Online supplemental material is available for this article.

The novel biomarker-based ABC (age, biomarkers, clinical history)-bleeding risk score for patients with atrial fibrillation: a derivation and validation study

BACKGROUND

The benefit of oral anticoagulation in atrial fibrillation is based on a balance between reduction in ischaemic stroke and increase in major bleeding. We aimed to develop and validate a new biomarker-based risk score to improve the prognostication of major bleeding in patients with atrial fibrillation.

METHODS

We developed and internally validated a new biomarker-based risk score for major bleeding in 14,537 patients with atrial fibrillation randomised to apixaban versus warfarin in the ARISTOTLE trial and externally validated it in 8468 patients with atrial fibrillation randomised to dabigatran versus warfarin in the RE-LY trial. Plasma samples for determination of candidate biomarker concentrations were obtained at randomisation. Major bleeding events were centrally adjudicated. The predictive values of biomarkers and clinical variables were assessed with Cox regression models. The most important variables were included in the score with weights proportional to the model coefficients. The ARISTOTLE and RE-LY trials are registered with ClinicalTrials.gov, numbers NCT00412984 and NCT00262600, respectively.

FINDINGS

The most important predictors for major bleeding were the concentrations of the biomarkers growth differentiation factor-15 (GDF-15), high-sensitivity cardiac troponin T (cTnT-hs) and haemoglobin, age, and previous bleeding. The ABC-bleeding score (age, biomarkers [GDF-15, cTnT-hs, and haemoglobin], and clinical history [previous bleeding]) score yielded a higher c-index than the conventional HAS-BLED and the newer ORBIT scores for major bleeding in both the derivation cohort (0·68 [95% CI 0·66-0·70] vs 0·61 [0·59-0·63] vs 0·65 [0·62-0·67], respectively; ABC-bleeding vs HAS-BLED p<0·0001 and ABC-bleeding vs ORBIT p=0·0008). ABC-bleeding score also yielded a higher c-index score in the the external validation cohort (0·71 [95% CI 0·68-0·73] vs 0·62 [0·59-0·64] for HAS-BLED vs 0·68 [0·65-0·70] for ORBIT; ABC-bleeding vs HAS-BLED p<0·0001 and ABC-bleeding vs ORBIT p=0·0016). A modified ABC-bleeding score using alternative biomarkers (haematocrit, cTnI-hs, cystatin C, or creatinine clearance) also outperformed the HAS-BLED and ORBIT scores.

INTERPRETATION

The ABC-bleeding score, using age, history of bleeding, and three biomarkers (haemoglobin, cTn-hs, and GDF-15 or cystatin C/CKD-EPI) was internally and externally validated and calibrated in large cohorts of patients with atrial fibrillation receiving anticoagulation therapy. The ABC-bleeding score performed better than HAS-BLED and ORBIT scores and should be useful as decision support on anticoagulation treatment in patients with atrial fibrillation.

FUNDING

BMS, Pfizer, Boehringer Ingelheim, Roche Diagnostics.

Development and Validation of the Neonatal Risk Estimate Score for Children Using Extracorporeal Respiratory Support

OBJECTIVE

To develop and validate the Neonatal Risk Estimate Score for Children Using Extracorporeal Respiratory Support, which estimates the risk of in-hospital death for neonates prior to receiving respiratory extracorporeal membrane oxygenation (ECMO) support.

STUDY DESIGN

We used an international ECMO registry (2008-2013); neonates receiving ECMO for respiratory support were included. We divided the registry into a derivation sample and internal validation sample, by calendar date. We chose candidate variables a priori based on published evidence of association with mortality; variables independently associated with mortality in logistic regression were included in this parsimonious model of risk adjustment. We evaluated model discrimination with the area under the receiver operating characteristic curve (AUC), and we evaluated calibration with the Hosmer-Lemeshow goodness-of-fit test.

RESULTS

During 2008-2013, 4592 neonates received ECMO respiratory support with mortality of 31%. The development dataset contained 3139 patients treated in 2008-2011. The Neo-RESCUERS measure had an AUC of 0.78 (95% CI 0.76-0.79). The validation cohort had an AUC = 0.77 (0.75-0.80). Patients in the lowest risk decile had an observed mortality of 7.0% and a predicted mortality of 4.4%, and those in the highest risk decile had an observed mortality of 65.6% and a predicted mortality of 67.5%.

CONCLUSIONS

Neonatal Risk Estimate Score for Children Using Extracorporeal Respiratory Support offers severity-of-illness adjustment for neonatal patients with respiratory failure receiving ECMO. This score may be used to adjust patient survival to assess hospital-level performance in ECMO-based care.

External Validation of the Hestia Criteria for Identifying Acute Pulmonary Embolism Patients at Low Risk of Early Mortality

INTRODUCTION

There are limited studies evaluating the ability of the Hestia criteria to accurately identify patients with acute pulmonary embolism (PE) at low risk of early mortality. We sought to externally validate the Hestia criteria for predicting in-hospital and 30-day post-PE mortality.

METHODS

We retrospectively identified consecutive, adult, objectively confirmed PE patients presenting to the emergency department at our institution from November 21, 2010, to January 31, 2014. We ascertained the total number of Hestia criteria met for each patient, calculated the proportion of patients categorized as low risk (ie, no Hestia criteria met), and determined the accuracy of the Hestia criteria for predicting in-hospital and 30-day all-cause mortality. Mortality was determined through Social Security Death Index searches.

RESULTS

A total of 577 patients with PE were included, of which 19 (3.3%) and 35 (6.6%) died in hospital or within 30 days of presentation. Both in-hospital and 30-day case fatality rates rose as the number of Hestia criteria increased. One-hundred forty nine (25.8%) patients were classified as low risk for early mortality, and none of these patients died within 30 days (negative predictive values of 100%). The Hestia criteria had excellent sensitivity (100%, 95% confidence interval [CI] = 79.1%-100% and 100%, 95% CI = 87.7%-100%) for predicting in-hospital and 30-day mortality but low specificity (<27.5% for both). The c-statistics for in-hospital and 30-day mortality were 83.5%, 95% CI = 77.1%-89.9% and 78.5%, 95% CI = 71.9%-85.1%. The predictive accuracy of the Hestia criteria remained acceptable in patients >80 years of age, with active cancer or chronic cardiopulmonary disease.

CONCLUSION

The Hestia criteria have an acceptable predictive accuracy to identify patients with PE at low risk for in-hospital or 30-day mortality.

Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review

OBJECTIVE

Electronic health records (EHRs) are an increasingly common data source for clinical risk prediction, presenting both unique analytic opportunities and challenges. We sought to evaluate the current state of EHR based risk prediction modeling through a systematic review of clinical prediction studies using EHR data.

METHODS

We searched PubMed for articles that reported on the use of an EHR to develop a risk prediction model from 2009 to 2014. Articles were extracted by two reviewers, and we abstracted information on study design, use of EHR data, model building, and performance from each publication and supplementary documentation.

RESULTS

We identified 107 articles from 15 different countries. Studies were generally very large (median sample size = 26 100) and utilized a diverse array of predictors. Most used validation techniques (n = 94 of 107) and reported model coefficients for reproducibility (n = 83). However, studies did not fully leverage the breadth of EHR data, as they uncommonly used longitudinal information (n = 37) and employed relatively few predictor variables (median = 27 variables). Less than half of the studies were multicenter (n = 50) and only 26 performed validation across sites. Many studies did not fully address biases of EHR data such as missing data or loss to follow-up. Average c-statistics for different outcomes were: mortality (0.84), clinical prediction (0.83), hospitalization (0.71), and service utilization (0.71).

CONCLUSIONS

EHR data present both opportunities and challenges for clinical risk prediction. There is room for improvement in designing such studies.

Prediction models for cardiovascular disease risk in the general population: systematic review

OBJECTIVE

 To provide an overview of prediction models for risk of cardiovascular disease (CVD) in the general population.

DESIGN

 Systematic review.

DATA SOURCES

 Medline and Embase until June 2013.

ELIGIBILITY CRITERIA FOR STUDY SELECTION

 Studies describing the development or external validation of a multivariable model for predicting CVD risk in the general population.

RESULTS

 9965 references were screened, of which 212 articles were included in the review, describing the development of 363 prediction models and 473 external validations. Most models were developed in Europe (n=167, 46%), predicted risk of fatal or non-fatal coronary heart disease (n=118, 33%) over a 10 year period (n=209, 58%). The most common predictors were smoking (n=325, 90%) and age (n=321, 88%), and most models were sex specific (n=250, 69%). Substantial heterogeneity in predictor and outcome definitions was observed between models, and important clinical and methodological information were often missing. The prediction horizon was not specified for 49 models (13%), and for 92 (25%) crucial information was missing to enable the model to be used for individual risk prediction. Only 132 developed models (36%) were externally validated and only 70 (19%) by independent investigators. Model performance was heterogeneous and measures such as discrimination and calibration were reported for only 65% and 58% of the external validations, respectively.

CONCLUSIONS

 There is an excess of models predicting incident CVD in the general population. The usefulness of most of the models remains unclear owing to methodological shortcomings, incomplete presentation, and lack of external validation and model impact studies. Rather than developing yet another similar CVD risk prediction model, in this era of large datasets, future research should focus on externally validating and comparing head-to-head promising CVD risk models that already exist, on tailoring or even combining these models to local settings, and investigating whether these models can be extended by addition of new predictors.

Evaluation of the diagnostic value of 64 simultaneously measured autoantibodies for early detection of gastric cancer

Autoantibodies against tumor-associated antigens (TAAs) have been suggested as biomarkers for early detection of gastric cancer. However, studies that systematically assess the diagnostic performance of a large number of autoantibodies are rare. Here, we used bead-based multiplex serology to simultaneously measure autoantibody responses against 64 candidate TAAs in serum samples from 329 gastric cancer patients, 321 healthy controls and 124 participants with other diseases of the upper digestive tract. At 98% specificity, sensitivities for the 64 tested autoantibodies ranged from 0–12% in the training set and a combination of autoantibodies against five TAAs (MAGEA4 + CTAG1 + TP53 + ERBB2_C + SDCCAG8) was able to detect 32% of the gastric cancer patients at a specificity of 87% in the validation set. Sensitivities for early and late stage gastric cancers were similar, while chronic atrophic gastritis, a precursor lesion of gastric cancer, was not detectable. However, the 5-marker combination also detected 26% of the esophageal cancer patients. In conclusion, the tested autoantibodies and combinations alone did not reach sufficient sensitivity for gastric cancer screening. Nevertheless, some autoantibodies, such as anti-MAGEA4, anti-CTAG1 or anti-TP53 and their combinations could possibly contribute to the development of cancer early detection tests (not necessarily restricted to gastric cancer) when being combined with other markers.

Review of Research Reporting Guidelines for Radiology Researchers

Prior articles have reviewed reporting guidelines and study evaluation tools for clinical research. However, only some of the many available accepted reporting guidelines at the Enhancing the QUAlity and Transparency Of health Research Network have been discussed in previous reports. In this paper, we review the key Enhancing the QUAlity and Transparency Of health Research reporting guidelines that have not been previously discussed. The study types include diagnostic and prognostic studies, reliability and agreement studies, observational studies, analytical and descriptive, experimental studies, quality improvement studies, qualitative research, health informatics, systematic reviews and meta-analyses, economic evaluations, and mixed methods studies. There are also sections on study protocols, and statistical analyses and methods. In each section, there is a brief overview of the study type, and then the reporting guideline(s) that are most applicable to radiology researchers including radiologists involved in health services research are discussed.

How to Develop, Validate, and Compare Clinical Prediction Models Involving Radiological Parameters: Study Design and Statistical Methods

Clinical prediction models are developed to calculate estimates of the probability of the presence/occurrence or future course of a particular prognostic or diagnostic outcome from multiple clinical or non-clinical parameters. Radiologic imaging techniques are being developed for accurate detection and early diagnosis of disease, which will eventually affect patient outcomes. Hence, results obtained by radiological means, especially diagnostic imaging, are frequently incorporated into a clinical prediction model as important predictive parameters, and the performance of the prediction model may improve in both diagnostic and prognostic settings. This article explains in a conceptual manner the overall process of developing and validating a clinical prediction model involving radiological parameters in relation to the study design and statistical methods. Collection of a raw dataset; selection of an appropriate statistical model; predictor selection; evaluation of model performance using a calibration plot, Hosmer-Lemeshow test and c-index; internal and external validation; comparison of different models using c-index, net reclassification improvement, and integrated discrimination improvement; and a method to create an easy-to-use prediction score system will be addressed. This article may serve as a practical methodological reference for clinical researchers.

Candidate Predictors of Health-Related Quality of Life of Colorectal Cancer Survivors: A Systematic Review

The population of colorectal cancer (CRC) survivors is growing and many survivors experience deteriorated health-related quality of life (HRQoL) in both early and late post-treatment phases. Identification of CRC survivors at risk for HRQoL deterioration can be improved by using prediction models. However, such models are currently not available for oncology practice. As a starting point for developing prediction models of HRQoL for CRC survivors, a comprehensive overview of potential candidate HRQoL predictors is necessary. Therefore, a systematic literature review was conducted to identify candidate predictors of HRQoL of CRC survivors. Original research articles on associations of biopsychosocial factors with HRQoL of CRC survivors were searched in PubMed, Embase, and Google Scholar. Two independent reviewers assessed eligibility and selected articles for inclusion (N = 53). Strength of evidence for candidate HRQoL predictors was graded according to predefined methodological criteria. The World Health Organization's International Classification of Functioning, Disability and Health (ICF) was used to develop a biopsychosocial framework in which identified candidate HRQoL predictors were mapped across the main domains of the ICF: health condition, body structures and functions, activities, participation, and personal and environmental factors. The developed biopsychosocial ICF framework serves as a basis for selecting candidate HRQoL predictors, thereby providing conceptual guidance for developing comprehensive, evidence-based prediction models of HRQoL for CRC survivors. Such models are useful in clinical oncology practice to aid in identifying individual CRC survivors at risk for HRQoL deterioration and could also provide potential targets for a biopsychosocial intervention aimed at safeguarding the HRQoL of at-risk individuals.

IMPLICATIONS FOR PRACTICE

More and more people now survive a diagnosis of colorectal cancer. The quality of life of these cancer survivors is threatened by health problems persisting for years after diagnosis and treatment. Early identification of survivors at risk of experiencing low quality of life in the future is thus important for taking preventive measures. Clinical prediction models are tools that can help oncologists identify at-risk individuals. However, such models are currently not available for clinical oncology practice. This systematic review outlines candidate predictors of low quality of life of colorectal cancer survivors, providing a firm conceptual basis for developing prediction models.