Reporting performance of prognostic models in cancer: a review

Untapped potential of multicenter studies: a review of cardiovascular risk prediction models revealed inappropriate analyses and wide variation in reporting

BACKGROUND

Clinical prediction models are often constructed using multicenter databases. Such a data structure poses additional challenges for statistical analysis (clustered data) but offers opportunities for model generalizability to a broad range of centers. The purpose of this study was to describe properties, analysis, and reporting of multicenter studies in the Tufts PACE Clinical Prediction Model Registry and to illustrate consequences of common design and analyses choices.

METHODS

Fifty randomly selected studies that are included in the Tufts registry as multicenter and published after 2000 underwent full-text screening. Simulated examples illustrate some key concepts relevant to multicenter prediction research.

RESULTS

Multicenter studies differed widely in the number of participating centers (range 2 to 5473). Thirty-nine of 50 studies ignored the multicenter nature of data in the statistical analysis. In the others, clustering was resolved by developing the model on only one center, using mixed effects or stratified regression, or by using center-level characteristics as predictors. Twenty-three of 50 studies did not describe the clinical settings or type of centers from which data was obtained. Four of 50 studies discussed neither generalizability nor external validity of the developed model.

CONCLUSIONS

Regression methods and validation strategies tailored to multicenter studies are underutilized. Reporting on generalizability and potential external validity of the model lacks transparency. Hence, multicenter prediction research has untapped potential.

REGISTRATION

This review was not registered.

Persistent unexplained physical symptoms: a prospective longitudinal cohort study in UK primary care

BACKGROUND

Unexplained physical symptoms (UPS) are extremely common among primary care attenders, but little is known about their longer-term outcome.

AIM

To investigate the persistence of somatic symptoms at 6 months among a cohort with multiple UPS, and identify prognostic factors associated with worsening symptom scores.

DESIGN AND SETTING

Prospective longitudinal cohort study involving adults attending UK general practice in North and Central London between January and December 2013.

METHOD

Consecutive adults attending nine general practices were screened to identify those with at least three UPS. Eligible participants completed measures of symptom severity (measured using the Patient Health Questionnaire Somatic Symptom Module [PHQ-15]), physical and mental wellbeing, and past health and social history, and were followed up after 6 months. Multivariable linear regression analysis was conducted to identify prognostic factors associated with the primary outcome: somatic symptom severity.

RESULTS

Overall, 245/294 (83%) provided 6-month outcome data. Of these, 135/245 (55%) reported still having UPS, 103/245 (42%) had symptoms still under investigation, and only 26/245 (11%) reported complete symptom resolution. Being female, higher baseline somatic symptom severity, poorer physical functioning, experience of childhood physical abuse, and perception of poor financial wellbeing were significantly associated with higher somatic symptom severity scores at 6 months.

CONCLUSION

This study has shown that at 6 months few participants had complete resolution of unexplained somatic symptoms. GPs should be made aware of the likelihood of UPS persisting, and the factors that make this more likely, to inform decision making and care planning. There is a need to develop prognostic tools that can predict the risk of poor outcomes.

Prognostic models for predicting overall survival in metastatic castration-resistant prostate cancer: a systematic review

PURPOSE

Prognostic models are developed to estimate the probability of the occurrence of future outcomes incorporating multiple variables. We aimed to identify and summarize existing multivariable prognostic models developed for predicting overall survival in patients with metastatic castration-resistant prostate cancer (mCRPC).

METHODS

The protocol was prospectively registered (CRD42017064448). We systematically searched Medline and reference lists up to May 2018 and included experimental and observational studies, which developed and/or internally validated prognostic models for mCRPC patients and were further externally validated or updated. The outcome of interest was overall survival. Two authors independently performed literature screening and quality assessment.

RESULTS

We included 12 studies that developed models including 8750 patients aged 42-95 years. Models included 4-11 predictor variables, mostly hemoglobin, baseline PSA, alkaline phosphatase, performance status, and lactate dehydrogenase. Very few incorporated Gleason score. Two models included predictors related to docetaxel and mitoxantrone treatments. Model performance after internal validation showed similar discrimination power ranging from 0.62 to 0.73. Overall survival models were mainly constructed as nomograms or risk groups/score. Two models obtained an overall judgment of low risk of bias.

CONCLUSIONS

Most models were not suitable for clinical use due to methodological shortcomings and lack of external validation. Further external validation and/or model updating is required to increase prognostic accuracy and clinical applicability prior to their incorporation in clinical practice as a useful tool in patient management.

Butyrate-Induced In Vitro Colonocyte Differentiation Network Model Identifies ITGB1, SYK, CDKN2A, CHAF1A, and LRP1 as the Prognostic Markers for Colorectal Cancer Recurrence

Numerous mechanisms are believed to contribute to the role of dietary fiber-derived butyrate in the protection against the development of colorectal cancers (CRCs). To identify the most crucial butyrate-regulated genes, we exploited whole genome microarray of HT-29 cells differentiated in vitro by butyrate treatment. Butyrate differentiates HT-29 cells by relaxing the perturbation, caused by mutations of Adenomatous polyposis coli (APC) and TP53 genes, the most frequent mutations observed in CRC. We constructed protein-protein interaction network (PPIN) with the differentially expressed genes after butyrate treatment and extracted the hub genes from the PPIN, which also participated in the APC-TP53 network. The idea behind this approach was that the expression of these hub genes also regulated cell differentiation, and subsequently CRC prognosis by evading the APC-TP53 mutational effect. We used mRNA expression profile of these critical hub genes from seven large CRC cohorts. Logistic Regression showed strong evidence for association of these common hubs with CRC recurrence. In this study, we exploited PPIN to reduce the dimension of microarray biologically and identified five prognostic markers for the CRC recurrence, which were validated across different datasets. Moreover, these five biomarkers we identified increase the predictive value of the TNM staging for CRC recurrence.

Risk Stratification Tools and Prognostic Models in Non-muscle-invasive Bladder Cancer: A Critical Assessment from the European Association of Urology Non-muscle-invasive Bladder Cancer Guidelines Panel

CONTEXT

This review focuses on the most widely used risk stratification and prediction tools for non-muscle-invasive bladder cancer (NMIBC).

OBJECTIVE

To assess the clinical use and relevance of risk stratification and prediction tools to enhance clinical decision making and counselling of patients with NMIBC.

EVIDENCE ACQUISITION

The most frequent, currently used risk stratification tools and prognostic models for NMIBC patients were identified by the members of the European Association of Urology (EAU) Guidelines Panel on NMIBC.

EVIDENCE SYNTHESIS

The 2006 European Organization for Research and Treatment of Cancer (EORTC) risk tables are the most widely used and validated tools for risk stratification and prognosis prediction in NMIBC patients. The EAU risk categories constitute a simple alternative to the EORTC risk tables and can be used for comparable risk stratification. In the subgroup of NMIBC patients treated with a short maintenance schedule of bacillus Calmette-Guerin (BCG), the Club Urológico Español de Tratamiento Oncológico (CUETO) scoring model is more accurate than the EORTC risk tables. Both the EORTC risk tables and the CUETO scoring model overestimate the recurrence and progression risks in patients treated according to current guidelines. The new concept of conditional recurrence and progression estimates is very promising during follow-up but should be validated.

CONCLUSIONS

Risk stratification and prognostic models enable outcome comparisons and standardisation of treatment and follow-up. At present, none of the available risk stratification and prognostic models reflects current standards of treatment. The EORTC risk tables and CUETO scoring model should be updated with previously unavailable data and recalculated.

PATIENT SUMMARY

Non-muscle-invasive bladder cancer is a heterogeneous disease. A risk-based therapeutic approach is recommended. We present available risk stratification and prediction tools and the degree of their validation with the aim to increase their use in everyday clinical practice.

External validation of a prognostic model incorporating quantitative PET image features in oesophageal cancer

AIM

Enhanced prognostic models are required to improve risk stratification of patients with oesophageal cancer so treatment decisions can be optimised. The primary aim was to externally validate a published prognostic model incorporating PET image features. Transferability of the model was compared using only clinical variables.

METHODS

This was a Transparent Reporting of a multivariate prediction model for Individual Prognosis Or Diagnosis (TRIPOD) type 3 study. The model was validated against patients treated with neoadjuvant chemoradiotherapy according to the Neoadjuvant chemoradiotherapy plus surgery versus surgery alone for oesophageal or junctional cancer (CROSS) trial regimen using pre- and post-harmonised image features. The Kaplan-Meier method with log-rank significance tests assessed risk strata discrimination. A Cox proportional hazards model assessed model calibration. Primary outcome was overall survival (OS).

RESULTS

Between 2010 and 2015, 449 patients were included in the development (n = 302), internal validation (n = 101) and external validation (n = 46) cohorts. No statistically significant difference in OS between patient quartiles was demonstrated in prognostic models incorporating PET image features (X² = 1.42, df = 3, p = 0.70) or exclusively clinical variables (age, disease stage and treatment; X² = 1.19, df = 3, p = 0.75). The calibration slope β of both models was not significantly different from unity (p = 0.29 and 0.29, respectively). Risk groups defined using only clinical variables suggested differences in OS, although these were not statistically significant (X² = 0.71, df = 2, p = 0.70).

CONCLUSION

The prognostic model did not enable significant discrimination between the validation risk groups, but a second model with exclusively clinical variables suggested some transferable prognostic ability. PET harmonisation did not significantly change the results of model validation.

Additional value of EUS in oesophageal cancer patients staged N0 on PET/CT: validation of a prognostic model

Background

Lymph node metastases are a major prognostic indicator in oesophageal cancer. Radiological staging largely influences treatment decisions and is becoming more reliant on PET and CT. However, the sensitivity of these modalities is suboptimal and is known to under-stage disease. The primary aim of this study was to validate a published prognostic model in oesophageal cancer patients staged N0 with PET/CT, which showed that EUS nodal status was an independent predictor of survival. The secondary aim was to assess the prognostic significance of pathological lymph node metastases in this cohort.

Methods

An independent validation cohort included 139 consecutive patients from a regional upper gastrointestinal cancer network staged N0 with PET/CT between 1st January 2013 and 31st June 2015. Replicating the original study, two Cox regression models were produced: one included EUS T-stage and EUS N-stage, and one included EUS T-stage and EUS N0 versus N+. The primary outcome of the prognostic model was overall survival (OS). Kaplan–Meier analysis assessed differences in OS between pathological node-negative (pN0) and node-positive (pN+) groups. A p value of < 0.05 was considered statistically significant.

Results

The mean OS of the validation cohort was 29.8 months (95% CI 27.1–35.2). EUS T-stage was significantly and independently associated with OS in both models (p = 0.011 and p = 0.012, respectively). EUS N-stage and EUS N0 versus N+ were not significantly associated with OS (p = 0.553 and p = 0.359, respectively). There was a significant difference in OS between pN0 and pN+ groups (χ² 13.315, df 1, p < 0.001).

Conclusion

Lymph node metastases have a significant detrimental effect on OS. This validation study did not replicate the results of the developed prognostic model but the continued benefit of EUS in patients staged N0 with PET/CT was demonstrated. EUS remains a valuable component of a multi-modality approach to oesophageal cancer staging.

Prognostic Significance of Blood, Serum, and Ascites Parameters in Patients with Malignant Peritoneal Mesothelioma or Peritoneal Carcinomatosis

To determine effects of the biochemical and cytological properties of blood, serum, and ascites on survival of patients with malignant peritoneal effusion (MPeE), including malignant peritoneal mesothelioma (MPeM) and peritoneal carcinomatosis (PC), we conducted a retrospective study of patients with MPeE and healthy controls. Potential prognostic factors were identified as follows: age, sex, blood neutrophil-to-lymphocyte ratio (NLR), serum parameters, ascites parameters, serum-ascites albumin gradient, and the ascites-serum LDH ratio. Compared to those of the control group, serum albumin levels were significantly lower, and the NLR and serum LDH levels were significantly higher in the MPeE group. Overall survival (OS) was longer in patients with MPeM compared to that in patients with PC. Compared with patients in the MPeM, patients with PC had higher NLRs, ascites glucose levels, serum-ascites albumin gradients, and serum LDH levels. In contrast, their ascites albumin levels and ascites-serum LDH ratios were lower. Univariate analyses indicated that the NLR, serum LDH levels, ascites LDH levels, ascites coenocyte levels, and the ascites coenocyte-to-monocyte ratios affected the OS. Multivariate analyses identified only serum and ascites LDH levels as independent prognostic factors.

Large-scale external validation and comparison of prognostic models: an application to chronic obstructive pulmonary disease

BACKGROUND

External validations and comparisons of prognostic models or scores are a prerequisite for their use in routine clinical care but are lacking in most medical fields including chronic obstructive pulmonary disease (COPD). Our aim was to externally validate and concurrently compare prognostic scores for 3-year all-cause mortality in mostly multimorbid patients with COPD.

METHODS

We relied on 24 cohort studies of the COPD Cohorts Collaborative International Assessment consortium, corresponding to primary, secondary, and tertiary care in Europe, the Americas, and Japan. These studies include globally 15,762 patients with COPD (1871 deaths and 42,203 person years of follow-up). We used network meta-analysis adapted to multiple score comparison (MSC), following a frequentist two-stage approach; thus, we were able to compare all scores in a single analytical framework accounting for correlations among scores within cohorts. We assessed transitivity, heterogeneity, and inconsistency and provided a performance ranking of the prognostic scores.

RESULTS

Depending on data availability, between two and nine prognostic scores could be calculated for each cohort. The BODE score (body mass index, airflow obstruction, dyspnea, and exercise capacity) had a median area under the curve (AUC) of 0.679 [1st quartile-3rd quartile = 0.655-0.733] across cohorts. The ADO score (age, dyspnea, and airflow obstruction) showed the best performance for predicting mortality (difference AUCADO - AUCBODE = 0.015 [95% confidence interval (CI) = -0.002 to 0.032]; p = 0.08) followed by the updated BODE (AUCBODE updated - AUCBODE = 0.008 [95% CI = -0.005 to +0.022]; p = 0.23). The assumption of transitivity was not violated. Heterogeneity across direct comparisons was small, and we did not identify any local or global inconsistency.

CONCLUSIONS

Our analyses showed best discriminatory performance for the ADO and updated BODE scores in patients with COPD. A limitation to be addressed in future studies is the extension of MSC network meta-analysis to measures of calibration. MSC network meta-analysis can be applied to prognostic scores in any medical field to identify the best scores, possibly paving the way for stratified medicine, public health, and research.

Calibration drift in regression and machine learning models for acute kidney injury

Objective

Predictive analytics create opportunities to incorporate personalized risk estimates into clinical decision support. Models must be well calibrated to support decision-making, yet calibration deteriorates over time. This study explored the influence of modeling methods on performance drift and connected observed drift with data shifts in the patient population.

Materials and Methods

Using 2003 admissions to Department of Veterans Affairs hospitals nationwide, we developed 7 parallel models for hospital-acquired acute kidney injury using common regression and machine learning methods, validating each over 9 subsequent years.

Results

Discrimination was maintained for all models. Calibration declined as all models increasingly overpredicted risk. However, the random forest and neural network models maintained calibration across ranges of probability, capturing more admissions than did the regression models. The magnitude of overprediction increased over time for the regression models while remaining stable and small for the machine learning models. Changes in the rate of acute kidney injury were strongly linked to increasing overprediction, while changes in predictor-outcome associations corresponded with diverging patterns of calibration drift across methods.

Conclusions

Efficient and effective updating protocols will be essential for maintaining accuracy of, user confidence in, and safety of personalized risk predictions to support decision-making. Model updating protocols should be tailored to account for variations in calibration drift across methods and respond to periods of rapid performance drift rather than be limited to regularly scheduled annual or biannual intervals.

Early warning scores for detecting deterioration in adult hospital patients: a systematic review protocol

INTRODUCTION

Early warning scores (EWSs) are used extensively to identify patients at risk of deterioration in hospital. Previous systematic reviews suggest that studies which develop EWSs suffer methodological shortcomings and consequently may fail to perform well. The reviews have also identified that few validation studies exist to test whether the scores work in other settings. We will aim to systematically review papers describing the development or validation of EWSs, focusing on methodology, generalisability and reporting.

METHODS

We will identify studies that describe the development or validation of EWSs for adult hospital inpatients. Each study will be assessed for risk of bias using the Prediction model Risk of Bias ASsessment Tool (PROBAST). Two reviewers will independently extract information. A narrative synthesis and descriptive statistics will be used to answer the main aims of the study which are to assess and critically appraise the methodological quality of the EWS, to describe the predictors included in the EWSs and to describe the reported performance of EWSs in external validation.

ETHICS AND DISSEMINATION

This systematic review will only investigate published studies and therefore will not directly involve patient data. The review will help to establish whether EWSs are fit for purpose and make recommendations to improve the quality of future research in this area.

PROSPERO REGISTRATION NUMBER

CRD42017053324.

A Survey of the Prevalence and Impact of Reporting Guideline Endorsement in Pathology Journals

OBJECTIVES

To determine the prevalence of reporting guideline endorsement in pathology journals and to estimate the impact of guideline endorsement.

METHODS

We compared the quality of reporting in two sets of studies: (1) studies published in journals that explicitly mentioned a guideline vs studies published in journals that did not and (2) studies that cited a guideline vs studies that did not. The quality of reporting in prognostic biomarker studies was assessed using the REporting recommendations for tumor MARKer prognostic studies (REMARK) guideline.

RESULTS

We found that six (10%) of the 59 leading pathology journals explicitly mention reporting guidelines in the instructions to authors. Only one journal required authors to submit a checklist. There was significant variation in the rate at which various REMARK items were reported (P < .001). Journal endorsement was associated with more complete reporting (P = .04). Studies that cited REMARK had greater adherence to the REMARK reporting guidelines than studies that did not (P = .02).

CONCLUSIONS

The prevalence of guideline endorsement is relatively low in pathology journals, but guideline endorsement may improve the quality of reporting.

A Novel 18-Marker Panel Predicting Clinical Outcome in Breast Cancer

Background: Gene expression profiling has made considerable contributions to our understanding of cancer biology and clinical care. This study describes a novel gene expression signature for breast cancer-specific survival that was validated using external datasets.Methods: Gene expression signatures for invasive breast carcinomas (mainly luminal B subtype) corresponding to 136 patients were analyzed using Cox regression, and the effect of each gene on disease-specific survival (DSS) was estimated. Iterative Bayesian model averaging was applied on multivariable Cox regression models resulting in an 18-marker panel, which was validated using three external validation datasets. The 18 genes were analyzed for common pathways and functions using the Ingenuity Pathway Analysis software. This study complied with the REMARK criteria.Results: The 18-gene multivariable model showed a high predictive power for DSS in the training and validation cohort and a clear stratification between high- and low-risk patients. The differentially expressed genes were predominantly involved in biological processes such as cell cycle, DNA replication, recombination, and repair. Furthermore, the majority of the 18 genes were found to play a pivotal role in cancer.Conclusions: Our findings demonstrated that the 18 molecular markers were strong predictors of breast cancer-specific mortality. The stable time-dependent area under the ROC curve function (AUC(t)) and high C-indices in the training and validation cohorts were further improved by fitting a combined model consisting of the 18-marker panel and established clinical markers.Impact: Our work supports the applicability of this 18-marker panel to improve clinical outcome prediction for breast cancer patients. Cancer Epidemiol Biomarkers Prev; 26(11); 1619-28. ©2017 AACR.

Personalizing prognosis in colorectal cancer: A systematic review of the quality and nature of clinical prognostic tools for survival outcomes

Integrating diverse types of prognostic information into accurate, individualized estimates of outcome in colorectal cancer is challenging. Significant heterogeneity in colorectal cancer prognostication tool quality exists. Methodology is incompletely or inadequately reported. Evaluations of the internal or external validity of the prognostic model are rarely performed. Prognostication tools are important devices for patient management, but tool reliability is compromised by poor quality. Guidance for future development of prognostication tools in colorectal cancer is needed.

Machine learning and microsimulation techniques on the prognosis of dementia: A systematic literature review

Background

Dementia is a complex disorder characterized by poor outcomes for the patients and high costs of care. After decades of research little is known about its mechanisms. Having prognostic estimates about dementia can help researchers, patients and public entities in dealing with this disorder. Thus, health data, machine learning and microsimulation techniques could be employed in developing prognostic estimates for dementia.

Objective

The goal of this paper is to present evidence on the state of the art of studies investigating and the prognosis of dementia using machine learning and microsimulation techniques.

Method

To achieve our goal we carried out a systematic literature review, in which three large databases—Pubmed, Socups and Web of Science were searched to select studies that employed machine learning or microsimulation techniques for the prognosis of dementia. A single backward snowballing was done to identify further studies. A quality checklist was also employed to assess the quality of the evidence presented by the selected studies, and low quality studies were removed. Finally, data from the final set of studies were extracted in summary tables.

Results

In total 37 papers were included. The data summary results showed that the current research is focused on the investigation of the patients with mild cognitive impairment that will evolve to Alzheimer’s disease, using machine learning techniques. Microsimulation studies were concerned with cost estimation and had a populational focus. Neuroimaging was the most commonly used variable.

Conclusions

Prediction of conversion from MCI to AD is the dominant theme in the selected studies. Most studies used ML techniques on Neuroimaging data. Only a few data sources have been recruited by most studies and the ADNI database is the one most commonly used. Only two studies have investigated the prediction of epidemiological aspects of Dementia using either ML or MS techniques. Finally, care should be taken when interpreting the reported accuracy of ML techniques, given studies’ different contexts.

External validation of EORTC risk scores to predict recurrence after transurethral resection of brazilian patients with non -muscle invasive bladder cancer stages Ta and T1

Validate the EORTC risk tables in Brazilian patients with NMIBC.

METHODS

205 patients were analyzed. The 6 parameters analyzed were: histologic grading, pathologic stage, size and number of tumors, previous recurrence rate and concomitant CIS. The time for first recurrence (TFR), risk score and probability of re¬currence were calculated and compared to the probabilities obtained from EORTC risk tables. C-index was calculated and accuracy of EORTC tables was analyzed.

RESULTS

pTa was presented in 91 (44.4%) patients and pT1 in 114 (55.6%). Ninety-seven (47.3%) patients had solitary tumor, and 108 (52.7%) multiple tumors. One hundred and three (50.2%) patients had tumors smaller than 3 cm and 102 (40.8%) had bigger than 3 cm. Concomitant CIS was observed in 21 (10.2%) patients. Low grade was presented in 95 (46.3%) patients, and high grade in 110 (53.7%). Intravesical therapy was utilized in 105 (56.1%) patients. Recurrence was observed in 117 (57.1%) patients and the mean TFR was 14,2 ± 7,3 months. C-index was 0,72 for 1 year and 0,7 for 5 years. The re¬currence risk was 28,8% in 1 year and 57,1% in 5 years, independently of the scoring risk. In our population, the EORTC risk tables overestimated the risk of recurrence in 1 year and underestimated in 5 years.

CONCLUSION

The validation of the EORTC risk tables in Brazilian patients with NMIBC was satisfactory and should be stimulated to predict recurrence, although these may overestimated the risk of recurrence in 1 year and underestimated in 5 years.

Whole-Genome Sequence of the Metastatic PC3 and LNCaP Human Prostate Cancer Cell Lines

The bone metastasis-derived PC3 and the lymph node metastasis-derived LNCaP prostate cancer cell lines are widely studied, having been described in thousands of publications over the last four decades. Here, we report short-read whole-genome sequencing (WGS) and de novo assembly of PC3 (ATCC CRL-1435) and LNCaP (clone FGC; ATCC CRL-1740) at ∼70 × coverage. A known homozygous mutation in TP53 and homozygous loss of PTEN were robustly identified in the PC3 cell line, whereas the LNCaP cell line exhibited a larger number of putative inactivating somatic point and indel mutations (and in particular a loss of stop codon events). This study also provides preliminary evidence that loss of one or both copies of the tumor suppressor Capicua (CIC) contributes to primary tumor relapse and metastatic progression, potentially offering a treatment target for castration-resistant prostate cancer (CRPC). Our work provides a resource for genetic, genomic, and biological studies employing two commonly-used prostate cancer cell lines.

Which Models Can I Use to Predict Adult ICU Length of Stay? A Systematic Review

OBJECTIVE

We systematically reviewed models to predict adult ICU length of stay.

DATA SOURCES

We searched the Ovid EMBASE and MEDLINE databases for studies on the development or validation of ICU length of stay prediction models.

STUDY SELECTION

We identified 11 studies describing the development of 31 prediction models and three describing external validation of one of these models.

DATA EXTRACTION

Clinicians use ICU length of stay predictions for planning ICU capacity, identifying unexpectedly long ICU length of stay, and benchmarking ICUs. We required the model variables to have been published and for the models to be free of organizational characteristics and to produce accurate predictions, as assessed by R across patients for planning and identifying unexpectedly long ICU length of stay and across ICUs for benchmarking, with low calibration bias. We assessed the reporting quality using the Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies.

DATA SYNTHESIS

The number of admissions ranged from 253 to 178,503. Median ICU length of stay was between 2 and 6.9 days. Two studies had not published model variables and three included organizational characteristics. None of the models produced predictions with low bias. The R was 0.05-0.28 across patients and 0.01-0.64 across ICUs. The reporting scores ranged from 49 of 78 to 60 of 78 and the methodologic scores from 12 of 22 to 16 of 22.

CONCLUSION

No models completely satisfy our requirements for planning, identifying unexpectedly long ICU length of stay, or for benchmarking purposes. Physicians using these models to predict ICU length of stay should interpret them with reservation.

Characteristics and Validation Techniques for PCA-Based Gene-Expression Signatures

Background. Many gene-expression signatures exist for describing the biological state of profiled tumors. Principal Component Analysis (PCA) can be used to summarize a gene signature into a single score. Our hypothesis is that gene signatures can be validated when applied to new datasets, using inherent properties of PCA. Results. This validation is based on four key concepts. Coherence: elements of a gene signature should be correlated beyond chance. Uniqueness: the general direction of the data being examined can drive most of the observed signal. Robustness: if a gene signature is designed to measure a single biological effect, then this signal should be sufficiently strong and distinct compared to other signals within the signature. Transferability: the derived PCA gene signature score should describe the same biology in the target dataset as it does in the training dataset. Conclusions. The proposed validation procedure ensures that PCA-based gene signatures perform as expected when applied to datasets other than those that the signatures were trained upon. Complex signatures, describing multiple independent biological components, are also easily identified.

How to interpret the results of medical time series data analysis: Classical statistical approaches versus dynamic Bayesian network modeling

BACKGROUND

Classical statistics is a well-established approach in the analysis of medical data. While the medical community seems to be familiar with the concept of a statistical analysis and its interpretation, the Bayesian approach, argued by many of its proponents to be superior to the classical frequentist approach, is still not well-recognized in the analysis of medical data.

AIM

The goal of this study is to encourage data analysts to use the Bayesian approach, such as modeling with graphical probabilistic networks, as an insightful alternative to classical statistical analysis of medical data.

MATERIALS AND METHODS

This paper offers a comparison of two approaches to analysis of medical time series data: (1) classical statistical approach, such as the Kaplan-Meier estimator and the Cox proportional hazards regression model, and (2) dynamic Bayesian network modeling. Our comparison is based on time series cervical cancer screening data collected at Magee-Womens Hospital, University of Pittsburgh Medical Center over 10 years.

RESULTS

The main outcomes of our comparison are cervical cancer risk assessments produced by the three approaches. However, our analysis discusses also several aspects of the comparison, such as modeling assumptions, model building, dealing with incomplete data, individualized risk assessment, results interpretation, and model validation.

CONCLUSION

Our study shows that the Bayesian approach is (1) much more flexible in terms of modeling effort, and (2) it offers an individualized risk assessment, which is more cumbersome for classical statistical approaches.

Quantifying the impact of different approaches for handling continuous predictors on the performance of a prognostic model

Continuous predictors are routinely encountered when developing a prognostic model. Investigators, who are often non-statisticians, must decide how to handle continuous predictors in their models. Categorising continuous measurements into two or more categories has been widely discredited, yet is still frequently done because of its simplicity, investigator ignorance of the potential impact and of suitable alternatives, or to facilitate model uptake. We examine three broad approaches for handling continuous predictors on the performance of a prognostic model, including various methods of categorising predictors, modelling a linear relationship between the predictor and outcome and modelling a nonlinear relationship using fractional polynomials or restricted cubic splines. We compare the performance (measured by the c-index, calibration and net benefit) of prognostic models built using each approach, evaluating them using separate data from that used to build them. We show that categorising continuous predictors produces models with poor predictive performance and poor clinical usefulness. Categorising continuous predictors is unnecessary, biologically implausible and inefficient and should not be used in prognostic model development. © 2016 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.

Refining Prognosis in Lung Cancer: A Report on the Quality and Relevance of Clinical Prognostic Tools

INTRODUCTION

Accurate, individualized prognostication for lung cancer patients requires the integration of standard patient and pathologic factors, biological, genetic, and other molecular characteristics of the tumor. Clinical prognostic tools aim to aggregate information on an individual patient to predict disease outcomes such as overall survival, but little is known about their clinical utility and accuracy in lung cancer.

METHODS

A systematic search of the scientific literature for clinical prognostic tools in lung cancer published from January 1, 1996 to January 27, 2015 was performed. In addition, web-based resources were searched. A priori criteria determined by the Molecular Modellers Working Group of the American Joint Committee on Cancer were used to investigate the quality and usefulness of tools. Criteria included clinical presentation, model development approaches, validation strategies, and performance metrics.

RESULTS

Thirty-two prognostic tools were identified. Patients with metastases were the most frequently considered population in non-small-cell lung cancer. All tools for small-cell lung cancer covered that entire patient population. Included prognostic factors varied considerably across tools. Internal validity was not formally evaluated for most tools and only 11 were evaluated for external validity. Two key considerations were highlighted for tool development: identification of an explicit purpose related to a relevant clinical population and clear decision points and prioritized inclusion of established prognostic factors over emerging factors.

CONCLUSIONS

Prognostic tools will contribute more meaningfully to the practice of personalized medicine if better study design and analysis approaches are used in their development and validation.

Related factors to semi-recumbent position compliance and pressure ulcers in patients with invasive mechanical ventilation: An observational study (CAPCRI study)

BACKGROUND

Semi-recumbent position is recommended to prevent ventilator-associated pneumonia. Its implementation, however, is below optimal.

OBJECTIVES

We aimed to assess real semi-recumbent position compliance and the degree of head-of-bed elevation in Spanish intensive care units, along with factors determining compliance and head-of-bed elevation and their relationship with the development of pressure ulcers. Finally, we investigated the impact that might have the diagnosis of pressure ulcers in the attitude toward head-of-bed elevation.

METHODS

We performed a prospective, multicenter, observational study in 6 intensive care units. Inclusion criteria were patients ≥18 years old and expected to remain under mechanical ventilator for ≥48h. Exclusion criteria were patients with contraindications for semi-recumbent position from admission, mechanical ventilation during the previous 7 days and prehospital intubation. Head-of-bed elevation was measured 3 times/day for a maximum of 28 days using the BOSCH GLM80(®) device. The variables collected related to patient admission, risk of pressure ulcers and the measurements themselves. Bivariate and multivariate analyses were carried out using multiple binary logistic regression and linear regression as appropriate. Statistical significance was set at p<0.05. All analyses were performed with IBM SPSS for Windows Version 20.0.

RESULTS

276 patients were included (6894 measurements). 45.9% of the measurements were <30.0°. The mean head-of-bed elevation was 30.1 (SD 6.7)° and mean patient compliance was 53.6 (SD 26.1)%. The main reasons for non-compliance according to the staff nurses were those related to the patient's care followed by clinical reasons. The factors independently related to semi-recumbent position compliance were intensive care unit, ventilation mode, nurse belonging to the research team, intracranial pressure catheter, beds with head-of-bed elevation device, type of pathology, lateral position, renal replacement therapy, nursing shift, open abdomen, abdominal vacuum therapy and agitation. Twenty-five patients (9.1%) developed a total of 34 pressure ulcers. The diagnosis of pressure ulcers did not affect the head-of-bed elevation. In the multivariate analysis, head-of-bed elevation was not identified as an independent risk factor for pressure ulcers.

CONCLUSIONS

Semi-recumbent position compliance is below optimal despite the fact that it seems achievable most of the time. Factors that affect semi-recumbent position include the particular intensive care unit, abdominal conditions, renal replacement therapy, agitation and bed type. Head-of-bed elevation was not related to the risk of pressure ulcers. Efforts should be made to clarify semi-recumbent position contraindications and further analysis of its safety profile should be carried out.

Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review

OBJECTIVE

Electronic health records (EHRs) are an increasingly common data source for clinical risk prediction, presenting both unique analytic opportunities and challenges. We sought to evaluate the current state of EHR based risk prediction modeling through a systematic review of clinical prediction studies using EHR data.

METHODS

We searched PubMed for articles that reported on the use of an EHR to develop a risk prediction model from 2009 to 2014. Articles were extracted by two reviewers, and we abstracted information on study design, use of EHR data, model building, and performance from each publication and supplementary documentation.

RESULTS

We identified 107 articles from 15 different countries. Studies were generally very large (median sample size = 26 100) and utilized a diverse array of predictors. Most used validation techniques (n = 94 of 107) and reported model coefficients for reproducibility (n = 83). However, studies did not fully leverage the breadth of EHR data, as they uncommonly used longitudinal information (n = 37) and employed relatively few predictor variables (median = 27 variables). Less than half of the studies were multicenter (n = 50) and only 26 performed validation across sites. Many studies did not fully address biases of EHR data such as missing data or loss to follow-up. Average c-statistics for different outcomes were: mortality (0.84), clinical prediction (0.83), hospitalization (0.71), and service utilization (0.71).

CONCLUSIONS

EHR data present both opportunities and challenges for clinical risk prediction. There is room for improvement in designing such studies.

Predictors of time to initiation of symptomatic therapy in early Parkinson's disease

Objective

To determine clinical and biological variables that predict time to initiation of symptomatic therapy in de novo Parkinson's disease patients.

Methods

Parkinson's Progression Markers Initiative is a longitudinal case–control study of de novo, untreated Parkinson's disease participants at enrolment. Participants contribute a wide range of motor and non‐motor measures, including biofluids and imaging biomarkers. The machine learning method of random survival forests was used to examine the ability of baseline variables to predict time to initiation of symptomatic therapy since study enrollment (baseline).

Results

There were 423 PD participants enrolled in PPMI and 33 initial baseline variables. Cross‐validation results showed that the three‐predictor subset of disease duration (time from diagnosis to enrollment), the modified Schwab and England activities of daily living scale, and the Movement Disorder Society Unified Parkinson's Disease Rating Scale (MDS‐UPDRS) total score modestly predicted time to initiation of symptomatic therapy (C = 0.70, pseudo‐R² = 0.13). Prediction using the three variables was similar to using the entire set of 33. None of the biological variables increased accuracy of the prediction. A prognostic index for time to initiation of symptomatic therapy was created using the linear and nonlinear effects of the three top variables based on a post hoc Cox model.

Interpretation

Our findings using a novel machine learning method support previously reported clinical variables that predict time to initiation of symptomatic therapy. However, the inclusion of biological variables did not increase prediction accuracy. Our prognostic index constructed, based on the group‐level survival curve can provide an indication of the risk of initiation of ST for PD patients based on functions of the three top predictors.

Prognostic models in acute pulmonary embolism: a systematic review and meta-analysis

OBJECTIVE

To review the evidence for existing prognostic models in acute pulmonary embolism (PE) and determine how valid and useful they are for predicting patient outcomes.

DESIGN

Systematic review and meta-analysis.

DATA SOURCES

OVID MEDLINE and EMBASE, and The Cochrane Library from inception to July 2014, and sources of grey literature.

ELIGIBILITY CRITERIA

Studies aiming at constructing, validating, updating or studying the impact of prognostic models to predict all-cause death, PE-related death or venous thromboembolic events up to a 3-month follow-up in patients with an acute symptomatic PE.

DATA EXTRACTION

Study characteristics and study quality using prognostic criteria. Studies were selected and data extracted by 2 reviewers.

DATA ANALYSIS

Summary estimates (95% CI) for proportion of risk groups and event rates within risk groups, and accuracy.

RESULTS

We included 71 studies (44,298 patients). Among them, 17 were model construction studies specific to PE prognosis. The most validated models were the PE Severity Index (PESI) and its simplified version (sPESI). The overall 30-day mortality rate was 2.3% (1.7% to 2.9%) in the low-risk group and 11.4% (9.9% to 13.1%) in the high-risk group for PESI (9 studies), and 1.5% (0.9% to 2.5%) in the low-risk group and 10.7% (8.8% to12.9%) in the high-risk group for sPESI (11 studies). PESI has proved clinically useful in an impact study. Shifting the cut-off or using novel and updated models specifically developed for normotensive PE improves the ability for identifying patients at lower risk for early death or adverse outcome (0.5-1%) and those at higher risk (up to 20-29% of event rate).

CONCLUSIONS

We provide evidence-based information about the validity and utility of the existing prognostic models in acute PE that may be helpful for identifying patients at low risk. Novel models seem attractive for the high-risk normotensive PE but need to be externally validated then be assessed in impact studies.

Design Characteristics Influence Performance of Clinical Prediction Rules in Validation: A Meta-Epidemiological Study

BACKGROUND

Many new clinical prediction rules are derived and validated. But the design and reporting quality of clinical prediction research has been less than optimal. We aimed to assess whether design characteristics of validation studies were associated with the overestimation of clinical prediction rules' performance. We also aimed to evaluate whether validation studies clearly reported important methodological characteristics.

METHODS

Electronic databases were searched for systematic reviews of clinical prediction rule studies published between 2006 and 2010. Data were extracted from the eligible validation studies included in the systematic reviews. A meta-analytic meta-epidemiological approach was used to assess the influence of design characteristics on predictive performance. From each validation study, it was assessed whether 7 design and 7 reporting characteristics were properly described.

RESULTS

A total of 287 validation studies of clinical prediction rule were collected from 15 systematic reviews (31 meta-analyses). Validation studies using case-control design produced a summary diagnostic odds ratio (DOR) 2.2 times (95% CI: 1.2-4.3) larger than validation studies using cohort design and unclear design. When differential verification was used, the summary DOR was overestimated by twofold (95% CI: 1.2 -3.1) compared to complete, partial and unclear verification. The summary RDOR of validation studies with inadequate sample size was 1.9 (95% CI: 1.2 -3.1) compared to studies with adequate sample size. Study site, reliability, and clinical prediction rule was adequately described in 10.1%, 9.4%, and 7.0% of validation studies respectively.

CONCLUSION

Validation studies with design shortcomings may overestimate the performance of clinical prediction rules. The quality of reporting among studies validating clinical prediction rules needs to be improved.

A prognostic index model for predicting overall survival in patients with metastatic castration-resistant prostate cancer treated with abiraterone acetate after docetaxel

A prognostic index model was developed, composed of six readily available and assessable factors and categorizing patients with metastatic castration-resistant prostate cancer treated with abiraterone–prednisone into distinct prognostic risk groups. This model could be useful for determining patient prognosis for follow-up, monitoring and patient stratification for clinical trials.

Background

Few prognostic models for overall survival (OS) are available for patients with metastatic castration-resistant prostate cancer (mCRPC) treated with recently approved agents. We developed a prognostic index model using readily available clinical and laboratory factors from a phase III trial of abiraterone acetate (hereafter abiraterone) in combination with prednisone in post-docetaxel mCRPC.

Patients and methods

Baseline data were available from 762 patients treated with abiraterone–prednisone. Factors were assessed for association with OS through a univariate Cox model and used in a multivariate Cox model with a stepwise procedure to identify those of significance. Data were validated using an independent, external, population-based cohort.

Results

Six risk factors individually associated with poor prognosis were included in the final model: lactate dehydrogenase > upper limit of normal (ULN) [hazard ratio (HR) = 2.31], Eastern Cooperative Oncology Group performance status of 2 (HR = 2.19), presence of liver metastases (HR = 2.00), albumin ≤4 g/dl (HR = 1.54), alkaline phosphatase > ULN (HR = 1.38) and time from start of initial androgen-deprivation therapy to start of treatment ≤36 months (HR = 1.30). Patients were categorized into good (n = 369, 46%), intermediate (n = 321, 40%) and poor (n = 107, 13%) prognosis groups based on the number of risk factors and relative HRs. The C-index was 0.70 ± 0.014. The model was validated by the external dataset (n = 286).

Conclusion

This analysis identified six factors used to model survival in mCRPC and categorized patients into three distinct risk groups. Prognostic stratification with this model could assist clinical practice decisions for follow-up and monitoring, and may aid in clinical trial design.

Trial registration numbers

NCT00638690.

Socioeconomic status, human papillomavirus, and overall survival in head and neck squamous cell carcinomas in Toronto, Canada

BACKGROUND

Despite universal healthcare in some countries, lower socioeconomic status (SES) has been associated with worse cancer survival. The influence of SES on head and neck cancer (HNC) survival is of immense interest, since SES is associated with the risk and prognostic factors associated with this disease.

PATIENTS AND METHODS

Newly diagnosed HNC patients from 2003 to 2010 (n=2124) were identified at Toronto's Princess Margaret Cancer Centre. Principal component analysis was used to calculate a composite score using neighbourhood-level SES variables obtained from the 2006 Canada Census. Associations of SES with overall survival were evaluated in HNC subsets and by p16 status (surrogate for human papillomavirus).

RESULTS

SES score was higher for oral cavity (n=423) and p16-positive oropharyngeal cancer (OPC, n=404) patients compared with other disease sites. Lower SES was associated with worse survival [HR 1.14 (1.06-1.22), p=0.0002], larger tumor staging (p<0.001), current smoking (p<0.0001), heavier alcohol consumption (p<0.0001), and greater comorbidity (p<0.0002), but not with treatment regimen (p>0.20). After adjusting for age, sex, and stage, the lowest SES quintile was associated with the worst survival only for OPC patients [HR 1.66 (1.09-2.53), n=832], primarily in the p16-negative subset [HR 1.63 (0.96-2.79)]. The predictive ability of the prognostic models improved when smoking/alcohol was added to the model (c-index 0.71 vs. 0.69), but addition of SES did not (c-index 0.69).

CONCLUSION

SES was associated with survival, but this effect was lost after accounting for other factors (age, sex, TNM stage, smoking/alcohol). Lower SES was associated with greater smoking, alcohol consumption, comorbidity, and stage.

Poor quality in the reporting and use of statistical methods in public health - the case of unemployment and health

BACKGROUND

It has previously been reported that many research articles fail to fulfill important criteria for statistical analyses, but, to date, these reports have not focused on public health problems. The aim of this study was to investigate the quality of reporting and use of statistical methods in articles analyzing the effect of unemployment on health.

METHODS

Forty-one articles were identified and evaluated in terms of how they addressed 12 specified criteria.

RESULTS

For most of these criteria, the majority of articles were inadequate. These criteria were conformity with a linear gradient (100 % of the articles), validation of the statistical model (100 %), collinearity of independent variables (97 %), fitting procedure (93 %), goodness of fit test (78 %), selection of variables (68 % for the candidate model; 88 % for the final model), and interactions between independent variables (66 %). Fewer, but still alarmingly many articles, failed to fulfill the criteria coefficients presented in statistical models (48 %), coding of variables (34 %) and discussion of methodological concerns (24 %). There was a lack of explicit reporting of statistical significance/confidence intervals; 34 % of the articles only presented p-values as being above or below the significance level, and 42 % did not present confidence intervals. Events per variable was the only criterion met at an undoubtedly acceptable level (2.5 %).

CONCLUSIONS

There were critical methodological shortcomings in the reviewed studies. It is difficult to obtain unbiased estimates, but there clearly needs to be some improvement in the quality of documentation on the use and performance of statistical methods. A suggestion here is that journals not only demand that articles fulfill the criteria within the STROBE statement, but that they include additional criteria to decrease the risk of incorrect conclusions being drawn.

Development of a model to predict breast cancer survival using data from the National Cancer Data Base

BACKGROUND

With the large amounts of data on patient, tumor, and treatment factors available to clinicians, it has become critically important to harness this information to guide clinicians in discussing a patient's prognosis. However, no widely accepted survival calculator is available that uses national data and includes multiple prognostic factors. Our objective was to develop a model for predicting survival among patients diagnosed with breast cancer using the National Cancer Data Base (NCDB) to serve as a prototype for the Commission on Cancer's "Cancer Survival Prognostic Calculator."

PATIENTS AND METHODS

A retrospective cohort of patients diagnosed with breast cancer (2003-2006) in the NCDB was included. A multivariable Cox proportional hazards regression model to predict overall survival was developed. Model discrimination by 10-fold internal cross-validation and calibration was assessed.

RESULTS

There were 296,284 patients for model development and internal validation. The c-index for the 10-fold cross-validation ranged from 0.779 to 0.788 after inclusion of all available pertinent prognostic factors. A plot of the observed versus predicted 5 year overall survival showed minimal deviation from the reference line.

CONCLUSION

This breast cancer survival prognostic model to be used as a prototype for building the Commission on Cancer's "Cancer Survival Prognostic Calculator" will offer patients and clinicians an objective opportunity to estimate personalized long-term survival based on patient demographic characteristics, tumor factors, and treatment delivered.

Covariate-adjusted measures of discrimination for survival data

MOTIVATION

Discrimination statistics describe the ability of a survival model to assign higher risks to individuals who experience earlier events: examples are Harrell's C-index and Royston and Sauerbrei's D, which we call the D-index. Prognostic covariates whose distributions are controlled by the study design (e.g. age and sex) influence discrimination and can make it difficult to compare model discrimination between studies. Although covariate adjustment is a standard procedure for quantifying disease-risk factor associations, there are no covariate adjustment methods for discrimination statistics in censored survival data.

OBJECTIVE

To develop extensions of the C-index and D-index that describe the prognostic ability of a model adjusted for one or more covariate(s).

METHOD

We define a covariate-adjusted C-index and D-index for censored survival data, propose several estimators, and investigate their performance in simulation studies and in data from a large individual participant data meta-analysis, the Emerging Risk Factors Collaboration.

RESULTS

The proposed methods perform well in simulations. In the Emerging Risk Factors Collaboration data, the age-adjusted C-index and D-index were substantially smaller than unadjusted values. The study-specific standard deviation of baseline age was strongly associated with the unadjusted C-index and D-index but not significantly associated with the age-adjusted indices.

CONCLUSIONS

The proposed estimators improve meta-analysis comparisons, are easy to implement and give a more meaningful clinical interpretation.

Multivariate meta-analysis of individual participant data helped externally validate the performance and implementation of a prediction model

OBJECTIVES

Our aim was to improve meta-analysis methods for summarizing a prediction model's performance when individual participant data are available from multiple studies for external validation.

STUDY DESIGN AND SETTING

We suggest multivariate meta-analysis for jointly synthesizing calibration and discrimination performance, while accounting for their correlation. The approach estimates a prediction model's average performance, the heterogeneity in performance across populations, and the probability of "good" performance in new populations. This allows different implementation strategies (e.g., recalibration) to be compared. Application is made to a diagnostic model for deep vein thrombosis (DVT) and a prognostic model for breast cancer mortality.

RESULTS

In both examples, multivariate meta-analysis reveals that calibration performance is excellent on average but highly heterogeneous across populations unless the model's intercept (baseline hazard) is recalibrated. For the cancer model, the probability of "good" performance (defined by C statistic ≥0.7 and calibration slope between 0.9 and 1.1) in a new population was 0.67 with recalibration but 0.22 without recalibration. For the DVT model, even with recalibration, there was only a 0.03 probability of "good" performance.

CONCLUSION

Multivariate meta-analysis can be used to externally validate a prediction model's calibration and discrimination performance across multiple populations and to evaluate different implementation strategies.

A Survival Association Study of 102 Polymorphisms Previously Associated with Survival Outcomes in Colorectal Cancer

Several published studies identified associations of a number of polymorphisms with a variety of survival outcomes in colorectal cancer. In this study, we aimed to explore 102 previously reported common genetic polymorphisms and their associations with overall survival (OS) and disease-free survival (DFS) in a colorectal cancer patient cohort from Newfoundland (n = 505). Genotypes were obtained using a genomewide SNP genotyping platform. For each polymorphism, the best possible genetic model was estimated for both overall survival and disease-free survival using a previously published approach. These SNPs were then analyzed under their genetic models by Cox regression method. Correction for multiple comparisons was performed by the False Discovery Rate (FDR) method. Univariate analysis results showed that RRM1-rs12806698, IFNGR1-rs1327474, DDX20-rs197412, and PTGS2-rs5275 polymorphisms were nominally associated with OS or DFS (p < 0.01). In stage-adjusted analysis, the nominal associations of DDX20-rs197412, PTGS2-rs5275, and HSPA5-rs391957 with DFS were detected. However, after FDR correction none of these polymorphisms remained significantly associated with the survival outcomes. We conclude that polymorphisms investigated in this study are not associated with OS or DFS in our colorectal cancer patient cohort.

A strategy to build and validate a prognostic biomarker model based on RT-qPCR gene expression and clinical covariates

BACKGROUND

Construction and validation of a prognostic model for survival data in the clinical domain is still an active field of research. Nevertheless there is no consensus on how to develop routine prognostic tests based on a combination of RT-qPCR biomarkers and clinical or demographic variables. In particular, the estimation of the model performance requires to properly account for the RT-qPCR experimental design.

RESULTS

We present a strategy to build, select, and validate a prognostic model for survival data based on a combination of RT-qPCR biomarkers and clinical or demographic data and we provide an illustration on a real clinical dataset. First, we compare two cross-validation schemes: a classical outcome-stratified cross-validation scheme and an alternative one that accounts for the RT-qPCR plate design, especially when samples are processed by batches. The latter is intended to limit the performance discrepancies, also called the validation surprise, between the training and the test sets. Second, strategies for model building (covariate selection, functional relationship modeling, and statistical model) as well as performance indicators estimation are presented. Since in practice several prognostic models can exhibit similar performances, complementary criteria for model selection are discussed: the stability of the selected variables, the model optimism, and the impact of the omitted variables on the model performance.

CONCLUSION

On the training dataset, appropriate resampling methods are expected to prevent from any upward biases due to unaccounted technical and biological variability that may arise from the experimental and intrinsic design of the RT-qPCR assay. Moreover, the stability of the selected variables, the model optimism, and the impact of the omitted variables on the model performances are pivotal indicators to select the optimal model to be validated on the test dataset.

Important prognostic factors for survival in patients with malignant pleural effusion

BACKGROUND

The approach to palliative treatment of malignant pleural effusion (MPE) should be individualized because these patients generally have poor survival. Our study aimed to develop a model to identify prognostic factors or survival time in patients diagnosed with MPE.

METHODS

This is a retrospective, descriptive, observational study to identify prognostic factors related to MPE in patients with a confirmed cancer diagnosis. Cox regression analysis was used to determine significant potential prognostic factors with respect to survival time. Survival time was defined as the time from pathological diagnosis to death.

RESULTS

One hundred and sixty-five patients were included; 77 were men (47%) and 88 were women (53%). The median age was 60 years, and all of the patients were pathologically proven to have MPE. Non-small-cell lung cancer (36.0%), breast carcinoma (26%), and lymphoma (13.0%) were the most frequently diagnosed tumors. The median overall survival of patients from the initial diagnosis was 5 months (range: 1.0-96.0 months). Kaplan-Meier univariate analysis showed that survival was significantly related to the following prognostic factors: ECOG PS (hazard ratio [HR] 10.0, 95% confidence interval [95% CI] 5.96 to 18.50, p < 0.0001), primary cancer site (HR 1.99, 95% CI 1.23 to 3.22, p < 0.01), positive pleural cytology (HR 1.25, 95% CI 0.88 to 1.78, p = 0.04), and positive histology (HR 1.33, 95% CI 0.97 to 1.81, p = 0.04). Other potential independent diagnostic factors that were examined did not affect survival. Cox regression analysis showed that only the ECOG PS was highly predictive of survival (HR 73.58, 95% CI 23.44 to 230.95, p < 0.0001).

CONCLUSIONS

ECOG PS is an independent predictor of survival in patients with MPE at initial diagnosis. This prognostic factor can help physicians select patients for appropriate palliative treatment of this syndrome.

Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration

The TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) Statement includes a 22-item checklist, which aims to improve the reporting of studies developing, validating, or updating a prediction model, whether for diagnostic or prognostic purposes. The TRIPOD Statement aims to improve the transparency of the reporting of a prediction model study regardless of the study methods used. This explanation and elaboration document describes the rationale; clarifies the meaning of each item; and discusses why transparent reporting is important, with a view to assessing risk of bias and clinical usefulness of the prediction model. Each checklist item of the TRIPOD Statement is explained in detail and accompanied by published examples of good reporting. The document also provides a valuable reference of issues to consider when designing, conducting, and analyzing prediction model studies. To aid the editorial process and help peer reviewers and, ultimately, readers and systematic reviewers of prediction model studies, it is recommended that authors include a completed checklist in their submission. The TRIPOD checklist can also be downloaded from www.tripod-statement.org.

A predictive model of chronic kidney disease in patients with congenital anomalies of the kidney and urinary tract

BACKGROUND

The antenatal detection of congenital anomalies of the kidney and urinary tract (CAKUT) has permitted early management of these conditions. The aim of this study was to identify predictive factors associated with chronic kidney disease (CKD) in CAKUT. We also propose a risk score of CKD.

METHODS

In this cohort study, 822 patients with prenatally detected CAKUT were followed up for a median time of 43 months. The primary outcome was CKD stage III or higher. A predictive model was developed using the Cox proportional hazards model and evaluated by using c statistics.

RESULTS

Chronic kidney disease occurred in 49 of the 822 (6 %) children with prenatally detected CAKUT. The most accurate model included bilateral hydronephrosis, oligohydramnios, estimated glomerular filtration rate and postnatal diagnosis. The accuracy of the score was 0.95 [95 % confidence interval (CI) 0.89-0.99] and 0.92 (95 % CI 0.86-0.95) after a follow-up of 2 and 10 years, respectively. Based on survival curves, we estimated that at 10 years of age, the probability of survival without CKD stage III was approximately 98 and 58 % for the patients assigned to the low-risk and high-risk groups, respectively (p < 0.001).

CONCLUSIONS

Our predictive model of CKD may contribute to an early identification of a subgroup of patients at high risk for renal impairment. It should be pointed out, however, that this model requires external validation in a different cohort.

External validation of a prognostic model estimating the survival of patients with recurrent high-grade gliomas after reirradiation

PURPOSE

We aimed to validate a controversial prognostic model for the survival of relapsed malignant glioma patients after reirradiation with an independent, multicentric patient cohort.

METHODS AND MATERIALS

A total of 165 malignant glioma patients underwent reirradiation at 4 different institutions between 1994 and 2012. Twenty-two patients had a good (score 1), 44 had a moderate (score 2), and 99 had a poor prognosis (score 3 or 4). Four statistical methods were used to validate the prognostic model: First, we compared survival according to prognostic group in the construction and the validation cohort by visual comparison of the respective Kaplan-Meier plots. Second, discrimination was quantified by calculating hazard ratios for death for each prognostic group, with the worst prognostic group serving as the reference. Calibration was assessed by a calibration plot for the time point 12 months after reirradiation. Finally, we compared the predictive performance of the score and a hypothetical prognostic model ignoring all predictor variables over time by means of a prediction error curve.

RESULTS

On visual validation, the survival curves of the 3 patient groups with good, moderate, and poor prognoses nicely separated from each other. Median survival rates after reirradiation were 17.9, 9.0, and 7.7 months in the patient groups with good, moderate, and poor prognosis, respectively. Hazard ratios confirmed satisfactory discrimination. Calibration was satisfactory for all and most accurate for the worst prognostic group. The score improved the prognostic performance in comparison to the "zero-model."

CONCLUSIONS

We successfully validated a prognostic model for the survival of malignant glioma patients after reirradiation with a multicentric, independent dataset. Being reliable and easy to handle, the model can be useful in personalized patient counseling and clinical decision-making.