The Performance of Risk Prediction Models

Psychotropic drug use as indicator of mental health in adolescents affected by a plexus injury at birth: A large population-based study in Sweden

Chronic handicap in early life may have a long-term impact on children’s psychosocial well-being. Here, we investigated whether Brachialis Plexus Birth Injury (BPBI)—an unpredictable injury at birth—is associated with worse mental health later on, as indicated by prescription and use of psychotropic drugs in adolescence. We explored further whether this association is different depending on socioeconomic characteristics of the child’s family, as well as sex. Of the 641 151 children born to native parents in Sweden 1987–1993 (alive and still living in Sweden at the end of 2008), identified in the Swedish Medical Birth Registry, 1587 had suffered a BPBI. Logistic regression analysis was performed to assess the impact of socioeconomic characteristics and associations with later psychosocial health. Results show that beyond the known increased risks for females as compared to males, BPBI, but also lower family income, further increased the risk of burdened mental health requiring psychotropic drug use in adolescence. The effects were additive. Thus, compared to unaffected peers, teenagers who suffered a BPBI at birth are at higher risk of suffering poor mental health during adolescence, independently of surgical intervention and its outcome. Girls growing up in families with lower socioeconomic status have this risk added to their already increased risk of poor mental health during adolescence.

The index of prediction accuracy: an intuitive measure useful for evaluating risk prediction models

BACKGROUND

Many measures of prediction accuracy have been developed. However, the most popular ones in typical medical outcome prediction settings require additional investigation of calibration.

METHODS

We show how rescaling the Brier score produces a measure that combines discrimination and calibration in one value and improves interpretability by adjusting for a benchmark model. We have called this measure the index of prediction accuracy (IPA). The IPA permits a common interpretation across binary, time to event, and competing risk outcomes. We illustrate this measure using example datasets.

RESULTS

The IPA is simple to compute, and example code is provided. The values of the IPA appear very interpretable.

CONCLUSIONS

IPA should be a prominent measure reported in studies of medical prediction model performance. However, IPA is only a measure of average performance and, by default, does not measure the utility of a medical decision.

Comprehensive evaluation of the revised international staging system in multiple myeloma patients treated with novel agents as a primary therapy

The revised International Staging System (R-ISS) has recently been developed to improve the risk stratification of multiple myeloma (MM) patients over the ISS. We assessed the R-ISS in MM patients who were treated with novel agents as a primary therapy and evaluated its discriminative power and ability to reclassify patients from the ISS. A total of 514 newly diagnosed MM patients treated with novel agents including thalidomide, bortezomib, and lenalidomide as a primary therapy were included in this retrospective analysis. With a median follow-up duration of 42.3 months (range, 40.5-44.1), the median overall survival (OS) was 61.0 months. There was a significant difference in median OS (not reached, 60.9, and 50.1 months for stages 1, 2, and 3, respectively, P < 0.001) among the three stages of R-ISS. The C-statistic was significantly greater for R-ISS than for ISS (0.769 vs. 0.696, P < 0.001). The event NRI was -0.08 (95% confidence interval [CI], -0.18-0.01) and the non-event NRI was 0.05 (95% CI, -0.03-0.10), resulting in a total NRI of -0.03 (95% CI, -0.14-0.08, P = 0.602). The R-ISS performs well and has significantly better discriminative power than the ISS in MM patients treated with novel agents as a primary therapy. However, it does not better reclassify patients from the ISS, suggesting that there is still room to improve the staging system. Moreover, new statistical measures for assessing and quantifying the risk prediction of new prognostic models are necessary in future studies.

Augmented estimation for t-year survival with censored regression models

Reliable and accurate risk prediction is fundamental for successful management of clinical conditions. Estimating comprehensive risk prediction models precisely, however, is a difficult task, especially when the outcome of interest is time to a rare event and the number of candidate predictors, p, is not very small. Another challenge in developing accurate risk models arises from potential model misspecification. Time-specific generalized linear models estimated with inverse censoring probability weighting are robust to model misspecification, but may be inefficient in the rare event setting. To improve the efficiency of such robust estimation procedures, various augmentation methods have been proposed in the literature. These procedures can also leverage auxiliary variables such as intermediate outcomes that are predictive of event risk. However, most existing methods do not perform well in the rare event setting, especially when p is not small. In this article, we propose a two-step, imputation-based augmentation procedure that can improve estimation efficiency and that is robust to model misspecification. We also develop regularized augmentation procedures for settings where p is not small, along with procedures to improve the estimation of individualized treatment effect in risk reduction. Numerical studies suggest that our proposed methods substantially outperform existing methods in efficiency gains. The proposed methods are applied to an AIDS clinical trial for treating HIV-infected patients.

Routine blood tests are associated with short term mortality and can improve emergency department triage: a cohort study of >12,000 patients

BACKGROUND

Prioritization of acutely ill patients in the Emergency Department remains a challenge. We aimed to evaluate whether routine blood tests can predict mortality in unselected patients in an emergency department and to compare risk prediction with a formalized triage algorithm.

METHODS

A prospective observational cohort study of 12,661 consecutive admissions to the Emergency Department of Nordsjælland University Hospital during two separate periods in 2010 (primary cohort, n = 6279) and 2013 (validation cohort, n = 6383). Patients were triaged in five categories by a formalized triage algorithm. All patients with a full routine biochemical screening (albumin, creatinine, c-reactive protein, haemoglobin, lactate dehydrogenase, leukocyte count, potassium, and sodium) taken at triage were included. Information about vital status was collected from the Danish Central Office of Civil registration. Multiple logistic regressions were used to predict 30-day mortality. Validation was performed by applying the regression models on the 2013 validation cohort.

RESULTS

Thirty-day mortality was 5.3%. The routine blood tests had a significantly stronger discriminative value on 30-day mortality compared to the formalized triage (AUC 88.1 [85.7;90.5] vs. 63.4 [59.1;67.5], p < 0.01). Risk stratification by routine blood tests was able to identify a larger number of low risk patients (n = 2100, 30-day mortality 0.1% [95% CI 0.0;0.3%]) compared to formalized triage (n = 1591, 2.8% [95% CI 2.0;3.6%]), p < 0.01.

CONCLUSIONS

Routine blood tests were strongly associated with 30-day mortality in acutely ill patients and discriminatory ability was significantly higher than with a formalized triage algorithm. Thus routine blood tests allowed an improved risk stratification of patients presenting in an emergency department.

Robust Dynamic Risk Prediction with Longitudinal Studies

Providing accurate and dynamic age-specific risk prediction is a crucial step in precision medicine. In this manuscript, we introduce an approach for estimating the τ-year age-specific absolute risk directly via a flexible varying coefficient model. The approach facilitates the utilization of predictors varying over an individual's lifetime. By using a nonparametric inverse probability weighted kernel estimating equation, the age-specific effects of risk factors are estimated without requiring the specification of the functional form. The approach allows borrowing information across individuals of similar ages, and therefore provides a practical solution for situations where the longitudinal information is only measured sparsely. We evaluate the performance of the proposed estimation and inference procedures with numerical studies, and make comparisons with existing methods in the literature. We illustrate the performance of our proposed approach by developing a dynamic prediction model using data from the Framingham Study.

Evaluation of Probabilistic Disease Forecasts

The statistical evaluation of probabilistic disease forecasts often involves calculation of metrics defined conditionally on disease status, such as sensitivity and specificity. However, for the purpose of disease management decision making, metrics defined conditionally on the result of the forecast-predictive values-are also important, although less frequently reported. In this context, the application of scoring rules in the evaluation of probabilistic disease forecasts is discussed. An index of separation with application in the evaluation of probabilistic disease forecasts, described in the clinical literature, is also considered and its relation to scoring rules illustrated. Scoring rules provide a principled basis for the evaluation of probabilistic forecasts used in plant disease management. In particular, the decomposition of scoring rules into interpretable components is an advantageous feature of their application in the evaluation of disease forecasts.

Inclusion of Highest Glasgow Coma Scale Motor Component Score in Mortality Risk Adjustment for Benchmarking of Trauma Center Performance

BACKGROUND

The Glasgow Coma Scale (GCS) is the most widely used measure of traumatic brain injury (TBI) severity. Currently, the arrival GCS motor component (mGCS) score is used in risk-adjustment models for external benchmarking of mortality. However, there is evidence that the highest mGCS score in the first 24 hours after injury might be a better predictor of death. Our objective was to evaluate the impact of including the highest mGCS score on the performance of risk-adjustment models and subsequent external benchmarking results.

STUDY DESIGN

Data were derived from the Trauma Quality Improvement Program analytic dataset (January 2014 through March 2015) and were limited to the severe TBI cohort (16 years or older, isolated head injury, GCS ≤8). Risk-adjustment models were created that varied in the mGCS covariates only (initial score, highest score, or both initial and highest mGCS scores). Model performance and fit, as well as external benchmarking results, were compared.

RESULTS

There were 6,553 patients with severe TBI across 231 trauma centers included. Initial and highest mGCS scores were different in 47% of patients (n = 3,097). Model performance and fit improved when both initial and highest mGCS scores were included, as evidenced by improved C-statistic, Akaike Information Criterion, and adjusted R-squared values. Three-quarters of centers changed their adjusted odds ratio decile, 2.6% of centers changed outlier status, and 45% of centers exhibited a ≥0.5-SD change in the odds ratio of death after including highest mGCS score in the model.

CONCLUSIONS

This study supports the concept that additional clinical information has the potential to not only improve the performance of current risk-adjustment models, but can also have a meaningful impact on external benchmarking strategies. Highest mGCS score is a good potential candidate for inclusion in additional models.

Accuracy and utility of an epigenetic biomarker for smoking in populations with varying rates of false self-report

Better biomarkers to detect smoking are needed given the tremendous public health burden caused by smoking. Current biomarkers to detect smoking have significant limitations, notably a short half-life for detection and lack of sensitivity for light smokers. These limitations may be particularly problematic in populations with less accurate self-reporting. Prior epigenome-wide association studies indicate that methylation status at cg05575921, a CpG residue located in the aryl hydrocarbon receptor repressor (AHRR) gene, may be a robust indicator of smoking status in individuals with as little as half of a pack-year of smoking. In this study, we show that a novel droplet digital PCR assay for measuring methylation at cg05575921 can reliably detect smoking status, as confirmed by serum cotinine, in populations with different demographic characteristics, smoking histories, and rates of false-negative self-report of smoking behavior. Using logistic regression models, we show that obtaining maximum accuracy in predicting smoking status depends on appropriately weighting self-report and cg05575921 methylation according to the characteristics of the sample being tested. Furthermore, models using only cg05575921 methylation to predict smoking perform nearly as well as those also including self-report across populations. In conclusion, cg05575921 has significant potential as a clinical biomarker to detect smoking in populations with varying rates of accuracy in self-report of smoking behavior.

The tyranny of the averages and the indiscriminate use of risk factors in public health: The case of coronary heart disease

Modern medicine is overwhelmed by a plethora of both established risk factors and novel biomarkers for diseases. The majority of this information is expressed by probabilistic measures of association such as the odds ratio (OR) obtained by calculating differences in average “risk” between exposed and unexposed groups. However, recent research demonstrates that even ORs of considerable magnitude are insufficient for assessing the ability of risk factors or biomarkers to distinguish the individuals who will develop the disease from those who will not. In regards to coronary heart disease (CHD), we already know that novel biomarkers add very little to the discriminatory accuracy (DA) of traditional risk factors. However, the value added by traditional risk factors alongside simple demographic variables such as age and sex has been the subject of less discussion. Moreover, in public health, we use the OR to calculate the population attributable fraction (PAF), although this measure fails to consider the DA of the risk factor it represents. Therefore, focusing on CHD and applying measures of DA, we re-examine the role of individual demographic characteristics, risk factors, novel biomarkers and PAFs in public health and epidemiology. In so doing, we also raise a more general criticism of the traditional risk factors’ epidemiology. We investigated a cohort of 6103 men and women who participated in the baseline (1991–1996) of the Malmö Diet and Cancer study and were followed for 18 years. We found that neither traditional risk factors nor biomarkers substantially improved the DA obtained by models considering only age and sex. We concluded that the PAF measure provided insufficient information for the planning of preventive strategies in the population. We need a better understanding of the individual heterogeneity around the averages and, thereby, a fundamental change in the way we interpret risk factors in public health and epidemiology.

•

There is a plethora of differences in “average” risk between exposed and unexposed groups of individuals.

•

Individual heterogeneity around average values is seldom considered in Public Health.

•

Measures of discriminatory accuracy (DA) informs on the underlying individual heterogeneity.

•

Most know risk factors and other categorizations associated with diseases have low DA.

•

We need a fundamental change in the way we investigate risk factors and other categorizations in Public Health.

Predicting Triple-Negative Breast Cancer Subtype Using Multiple Single Nucleotide Polymorphisms for Breast Cancer Risk and Several Variable Selection Methods

INTRODUCTION

Studies of triple-negative breast cancer have recently been extending the inclusion criteria and incorporating additional molecular markers into the selection criteria, opening up scope for targeted therapies. The screening phases required for studies of this type are often prolonged, since the process of determining the molecular subtype and carrying out additional biomarker assessment is time-consuming. Parameters such as germline genotypes capable of predicting the molecular subtype before it becomes available from pathology might be helpful for treatment planning and optimizing the timing and cost of screening phases. This appears to be feasible, as rapid and low-cost genotyping methods are becoming increasingly available. The aim of this study was to identify single nucleotide polymorphisms (SNPs) for breast cancer risk capable of predicting triple negativity, in addition to clinical predictors, in breast cancer patients.

METHODS

This cross-sectional observational study included 1271 women with invasive breast cancer who were treated at a university hospital. A total of 76 validated breast cancer risk SNPs were successfully genotyped. Univariate associations between each SNP and triple negativity were explored using logistic regression analyses. Several variable selection and regression techniques were applied to identify a set of SNPs that together improve the prediction of triple negativity in addition to the clinical predictors of age at diagnosis and body mass index (BMI). The most accurate prediction method was determined by cross-validation.

RESULTS

The SNP rs10069690 (TERT, CLPTM1L) was the only significant SNP (corrected p = 0.02) after correction of p values for multiple testing in the univariate analyses. This SNP and three additional SNPs from the genes RAD51B, CCND1, and FGFR2 were selected for prediction of triple negativity. The addition of these SNPs to clinical predictors increased the cross-validated area under the curve (AUC) from 0.618 to 0.625. Age at diagnosis was the strongest predictor, stronger than any genetic characteristics.

CONCLUSION

Prediction of triple-negative breast cancer can be improved if SNPs associated with breast cancer risk are added to a prediction rule based on age at diagnosis and BMI. This finding could be used for prescreening purposes in complex molecular therapy studies for triple-negative breast cancer.

Standardized and weighted time-dependent receiver operating characteristic curves to evaluate the intrinsic prognostic capacities of a marker by taking into account confounding factors

Time-dependent receiver operating characteristic curves allow to evaluate the capacity of a marker to discriminate between subjects who experience the event up to a given prognostic time from those who are free of this event. In this article, we propose an inverse probability weighting estimator of a standardized and weighted time-dependent receiver operating characteristic curve. This estimator provides a measure of the prognostic capacities by taking into account potential confounding factors. We illustrate the robustness of the estimator by a simulation-based study and its usefulness by two applications in kidney transplantation.

Simulation of complex data structures for planning of studies with focus on biomarker comparison

BACKGROUND

There are a growing number of observational studies that do not only focus on single biomarkers for predicting an outcome event, but address questions in a multivariable setting. For example, when quantifying the added value of new biomarkers in addition to established risk factors, the aim might be to rank several new markers with respect to their prediction performance. This makes it important to consider the marker correlation structure for planning such a study. Because of the complexity, a simulation approach may be required to adequately assess sample size or other aspects, such as the choice of a performance measure.

METHODS

In a simulation study based on real data, we investigated how to generate covariates with realistic distributions and what generating model should be used for the outcome, aiming to determine the least amount of information and complexity needed to obtain realistic results. As a basis for the simulation a large epidemiological cohort study, the Gutenberg Health Study was used. The added value of markers was quantified and ranked in subsampling data sets of this population data, and simulation approaches were judged by the quality of the ranking. One of the evaluated approaches, the random forest, requires original data at the individual level. Therefore, also the effect of the size of a pilot study for random forest based simulation was investigated.

RESULTS

We found that simple logistic regression models failed to adequately generate realistic data, even with extensions such as interaction terms or non-linear effects. The random forest approach was seen to be more appropriate for simulation of complex data structures. Pilot studies starting at about 250 observations were seen to provide a reasonable level of information for this approach.

CONCLUSIONS

We advise to avoid oversimplified regression models for simulation, in particular when focusing on multivariable research questions. More generally, a simulation should be based on real data for adequately reflecting complex observational data structures, such as found in epidemiological cohort studies.

Statistical Methods for Cohort Studies of CKD: Prediction Modeling

Prediction models are often developed in and applied to CKD populations. These models can be used to inform patients and clinicians about the potential risks of disease development or progression. With increasing availability of large datasets from CKD cohorts, there is opportunity to develop better prediction models that will lead to more informed treatment decisions. It is important that prediction modeling be done using appropriate statistical methods to achieve the highest accuracy, while avoiding overfitting and poor calibration. In this paper, we review prediction modeling methods in general from model building to assessing model performance as well as the application to new patient populations. Throughout, the methods are illustrated using data from the Chronic Renal Insufficiency Cohort Study.

Predicting Disability after Ischemic Stroke Based on Comorbidity Index and Stroke Severity-From the Virtual International Stroke Trials Archive-Acute Collaboration

Background and aim

The availability and access of hospital administrative data [coding for Charlson comorbidity index (CCI)] in large data form has resulted in a surge of interest in using this information to predict mortality from stroke. The aims of this study were to determine the minimum clinical data set to be included in models for predicting disability after ischemic stroke adjusting for CCI and clinical variables and to evaluate the impact of CCI on prediction of outcome.

Method

We leverage anonymized clinical trial data in the Virtual International Stroke Trials Archive. This repository contains prospective data on stroke severity and outcome. The inclusion criteria were patients with available stroke severity score such as National Institutes of Health Stroke Scale (NIHSS), imaging data, and outcome disability score such as 90-day Rankin Scale. We calculate CCI based on comorbidity data in this data set. For logistic regression, we used these calibration statistics: Nagelkerke generalised R² and Brier score; and for discrimination we used: area under the receiver operating characteristics curve (AUC) and integrated discrimination improvement (IDI). The IDI was used to evaluate improvement in disability prediction above baseline model containing age, sex, and CCI.

Results

The clinical data among 5,206 patients (55% males) were as follows: mean age 69 ± 13 years, CCI 4.2 ± 0.8, and median NIHSS of 12 (IQR 8, 17) on admission and 9 (IQR 5, 15) at 24 h. In Model 2, adding admission NIHSS to the baseline model improved AUC from 0.67 (95% CI 0.65–0.68) to 0.79 (95% CI 0.78–0.81). In Model 3, adding 24-h NIHSS to the baseline model resulted in substantial improvement in AUC to 0.90 (95% CI 0.89–0.91) and increased IDI by 0.23 (95% CI 0.22–0.24). Adding the variable recombinant tissue plasminogen activator did not result in a further change in AUC or IDI to this regression model. In Model 3, the variable NIHSS at 24 h explains 87.3% of the variance of Model 3, follow by age (8.5%), comorbidity (3.7%), and male sex (0.5%).

Conclusion

Our results suggest that prediction of disability after ischemic stroke should at least include 24-h NIHSS and age. The variable CCI is less important for prediction of disability in this data set.

Life Expectancy Predictions for Older Diabetic Patients as Estimated by Physicians and a Prognostic Model

Background: Multiple medical organizations recommend using life expectancy (LE) to individualize diabetes care goals. We compare the performance of patient LE predictions made by physicians to LE predictions from a simulation model (the Chicago model) in a cohort of older diabetic patients. Design: Retrospective cohort study of a convenience sample (n = 447) of diabetes patients over 65 years and their physicians. Measurements: Physicians provided LE estimates for individual patients during a baseline survey (2000–2003). The prognostic model included a comprehensive geriatric type 2 diabetes simulation model (the Chicago model) and combinations of the physician estimate and the Chicago model (“And,” “Or,” and “Average” models). Observed survival was determined based on the National Death Index through 31 December 2010. The predictive accuracy of LE predictions was assessed using c-statistic for 5-year mortality; Harrell’s c-statistic, and Integrated Brier score for overall survival. Results: The patient cohort had a mean (SD) age of 73.4 (5.9) years. The majority were female (62.6%) and black (79.4%). At 5 years, 108 (24.2%) patients had died. The c-statistic for 5-year mortality was similar for physicians (0.69) and the Chicago model (0.68), while the average of estimates by physicians and Chicago model yielded the highest c-statistic of any method tested (0.73). The estimates of overall survival yielded a similar pattern of results. Limitations: Generalizability of patient cohort and lack of updated model parameters. Conclusions: Compared with individual methods, the average of LE estimates by physicians and the Chicago model had the best predictive performance. Prognostic models, such as the Chicago model, may complement and support physicians’ intuitions as they consider treatment decisions and goals for older patients with chronic conditions like diabetes.

Intersectionality and risk for ischemic heart disease in Sweden: Categorical and anti-categorical approaches

Intersectionality theory can contribute to epidemiology and public health by furthering understanding of power dynamics driving production of health disparities, and increasing knowledge about heterogeneities within, and overlap between, social categories. Drawing on McCall, we relate the first of these potential contributions to categorical intersectionality and the second to anti-categorical intersectionality. Both approaches are used in study of risk of ischemic heart disease (IHD), based on register data on 3.6 million adults residing in Sweden by 2010, followed for three years. Categorical intersectionality is here coupled with between-group differences in average risk calculation, as we use intersectional categorizations while estimating odds ratios through logistic regressions. The anti-categorical approach is operationalized through measurement of discriminatory accuracy (DA), i.e., capacity to accurately categorize individuals with or without a certain outcome, through computation of the area under the curve (AUC). Our results show substantial differences in average risk between intersectional groupings. The DA of social categorizations is found to be low, however, due to outcome variability within and overlap between categories. We argue that measures of DA should be used for proper interpretation of differences in average risk between social (or any other) categories. Tension between average between-group risk and the DA of categorizations, which can be related to categorical and anti-categorical intersectional analyses, should be made explicit and discussed to a larger degree in epidemiology and public health.

Evaluation of predictive capacities of biomarkers based on research synthesis

The objective of diagnostic studies or prognostic studies is to evaluate and compare predictive capacities of biomarkers. Suppose we are interested in evaluation and comparison of predictive capacities of continuous biomarkers for a binary outcome based on research synthesis. In analysis of each study, subjects are often classified into two groups of the high-expression and low-expression groups according to a cut-off value, and statistical analysis is based on a 2 × 2 table defined by the response and the high expression or low expression of the biomarker. Because the cut-off is study specific, it is difficult to interpret a combined summary measure such as an odds ratio based on the standard meta-analysis techniques. The summary receiver operating characteristic curve is a useful method for meta-analysis of diagnostic studies in the presence of heterogeneity of cut-off values to examine discriminative capacities of biomarkers. We develop a method to estimate positive or negative predictive curves, which are alternative to the receiver operating characteristic curve based on information reported in published papers of each study. These predictive curves provide a useful graphical presentation of pairs of positive and negative predictive values and allow us to compare predictive capacities of biomarkers of different scales in the presence of heterogeneity in cut-off values among studies. Copyright © 2016 John Wiley & Sons, Ltd.

Assessment of predictive performance in incomplete data by combining internal validation and multiple imputation

BACKGROUND

Missing values are a frequent issue in human studies. In many situations, multiple imputation (MI) is an appropriate missing data handling strategy, whereby missing values are imputed multiple times, the analysis is performed in every imputed data set, and the obtained estimates are pooled. If the aim is to estimate (added) predictive performance measures, such as (change in) the area under the receiver-operating characteristic curve (AUC), internal validation strategies become desirable in order to correct for optimism. It is not fully understood how internal validation should be combined with multiple imputation.

METHODS

In a comprehensive simulation study and in a real data set based on blood markers as predictors for mortality, we compare three combination strategies: Val-MI, internal validation followed by MI on the training and test parts separately, MI-Val, MI on the full data set followed by internal validation, and MI(-y)-Val, MI on the full data set omitting the outcome followed by internal validation. Different validation strategies, including bootstrap und cross-validation, different (added) performance measures, and various data characteristics are considered, and the strategies are evaluated with regard to bias and mean squared error of the obtained performance estimates. In addition, we elaborate on the number of resamples and imputations to be used, and adopt a strategy for confidence interval construction to incomplete data.

RESULTS

Internal validation is essential in order to avoid optimism, with the bootstrap 0.632+ estimate representing a reliable method to correct for optimism. While estimates obtained by MI-Val are optimistically biased, those obtained by MI(-y)-Val tend to be pessimistic in the presence of a true underlying effect. Val-MI provides largely unbiased estimates, with a slight pessimistic bias with increasing true effect size, number of covariates and decreasing sample size. In Val-MI, accuracy of the estimate is more strongly improved by increasing the number of bootstrap draws rather than the number of imputations. With a simple integrated approach, valid confidence intervals for performance estimates can be obtained.

CONCLUSIONS

When prognostic models are developed on incomplete data, Val-MI represents a valid strategy to obtain estimates of predictive performance measures.

A Roadmap for the Development of Applied Computational Psychiatry

BACKGROUND

Computational psychiatry is a burgeoning field that utilizes mathematical approaches to investigate psychiatric disorders, derive quantitative predictions, and integrate data across multiple levels of description. Computational psychiatry has already led to many new insights into the neurobehavioral mechanisms that underlie several psychiatric disorders, but its usefulness from a clinical standpoint is only now starting to be considered.

METHODS

Examples of computational psychiatry are highlighted, and a phase-based pipeline for the development of clinical computational-psychiatry applications is proposed, similar to the phase-based pipeline used in drug development. It is proposed that each phase has unique endpoints and deliverables, which will be important milestones to move tasks, procedures, computational models, and algorithms from the laboratory to clinical practice.

RESULTS

Application of computational approaches should be tested on healthy volunteers in Phase I, transitioned to target populations in Phase IB and Phase IIA, and thoroughly evaluated using randomized clinical trials in Phase IIB and Phase III. Successful completion of these phases should be the basis of determining whether computational models are useful tools for prognosis, diagnosis, or treatment of psychiatric patients.

CONCLUSIONS

A new type of infrastructure will be necessary to implement the proposed pipeline. This infrastructure should consist of groups of investigators with diverse backgrounds collaborating to make computational psychiatry relevant for the clinic.

Objective Bayesian model selection for Cox regression

There is now a large literature on objective Bayesian model selection in the linear model based on the g-prior. The methodology has been recently extended to generalized linear models using test-based Bayes factors. In this paper, we show that test-based Bayes factors can also be applied to the Cox proportional hazards model. If the goal is to select a single model, then both the maximum a posteriori and the median probability model can be calculated. For clinical prediction of survival, we shrink the model-specific log hazard ratio estimates with subsequent calculation of the Breslow estimate of the cumulative baseline hazard function. A Bayesian model average can also be employed. We illustrate the proposed methodology with the analysis of survival data on primary biliary cirrhosis patients and the development of a clinical prediction model for future cardiovascular events based on data from the Second Manifestations of ARTerial disease (SMART) cohort study. Cross-validation is applied to compare the predictive performance with alternative model selection approaches based on Harrell's c-Index, the calibration slope and the integrated Brier score. Finally, a novel application of Bayesian variable selection to optimal conditional prediction via landmarking is described. Copyright © 2016 John Wiley & Sons, Ltd.

Development of a simple tool to predict the risk of postpartum diabetes in women with gestational diabetes mellitus

AIMS

Women with gestational diabetes mellitus (GDM) have an increased risk of diabetes postpartum. We developed a score to predict the long-term risk of postpartum diabetes using clinical and anamnestic variables recorded during or shortly after delivery.

METHODS

Data from 257 GDM women who were prospectively followed for diabetes outcome over 20 years of follow-up were used to develop and validate the risk score. Participants were divided into training and test sets. The risk score was calculated using Lasso Cox regression and divided into four risk categories, and its prediction performance was assessed in the test set.

RESULTS

Postpartum diabetes developed in 110 women. The computed training set risk score of 5 × body mass index in early pregnancy (per kg/m(2)) + 132 if GDM was treated with insulin (otherwise 0) + 44 if the woman had a family history of diabetes (otherwise 0) - 35 if the woman lactated (otherwise 0) had R (2) values of 0.23, 0.25, and 0.33 at 5, 10, and 15 years postpartum, respectively, and a C-Index of 0.75. Application of the risk score in the test set resulted in observed risk of postpartum diabetes at 5 years of 11 % for low risk scores ≤140, 29 % for scores 141-220, 64 % for scores 221-300, and 80 % for scores >300.

CONCLUSIONS

The derived risk score is easy to calculate, allows accurate prediction of GDM-related postpartum diabetes, and may thus be a useful prediction tool for clinicians and general practitioners.

A Predictive Model for Root Caries Incidence

This study aimed to find the set of risk indicators best able to predict root caries (RC) incidence in caries-active adults utilizing data from the Xylitol for Adult Caries Trial (X-ACT). Five logistic regression models were compared with respect to their predictive performance for incident RC using data from placebo-control participants with exposed root surfaces at baseline and from two study centers with ancillary data collection (n = 155). Prediction performance was assessed from baseline variables and after including ancillary variables [smoking, diet, use of removable partial dentures (RPD), toothbrush use, income, education, and dental insurance]. A sensitivity analysis added treatment to the models for both the control and treatment participants (n = 301) to predict RC for the control participants. Forty-nine percent of the control participants had incident RC. The model including the number of follow-up years at risk, the number of root surfaces at risk, RC index, gender, race, age, and smoking resulted in the best prediction performance, having the highest AUC and lowest Brier score. The sensitivity analysis supported the primary analysis and gave slightly better performance summary measures. The set of risk indicators best able to predict RC incidence included an increased number of root surfaces at risk and increased RC index at baseline, followed by white race and nonsmoking, which were strong nonsignificant predictors. Gender, age, and increased number of follow-up years at risk, while included in the model, were also not statistically significant. The inclusion of health, diet, RPD use, toothbrush use, income, education, and dental insurance variables did not improve the prediction performance.

Estimating the Risk of Chronic Pain: Development and Validation of a Prognostic Model (PICKUP) for Patients with Acute Low Back Pain

BACKGROUND

Low back pain (LBP) is a major health problem. Globally it is responsible for the most years lived with disability. The most problematic type of LBP is chronic LBP (pain lasting longer than 3 mo); it has a poor prognosis and is costly, and interventions are only moderately effective. Targeting interventions according to risk profile is a promising approach to prevent the onset of chronic LBP. Developing accurate prognostic models is the first step. No validated prognostic models are available to accurately predict the onset of chronic LBP. The primary aim of this study was to develop and validate a prognostic model to estimate the risk of chronic LBP.

METHODS AND FINDINGS

We used the PROGRESS framework to specify a priori methods, which we published in a study protocol. Data from 2,758 patients with acute LBP attending primary care in Australia between 5 November 2003 and 15 July 2005 (development sample, n = 1,230) and between 10 November 2009 and 5 February 2013 (external validation sample, n = 1,528) were used to develop and externally validate the model. The primary outcome was chronic LBP (ongoing pain at 3 mo). In all, 30% of the development sample and 19% of the external validation sample developed chronic LBP. In the external validation sample, the primary model (PICKUP) discriminated between those who did and did not develop chronic LBP with acceptable performance (area under the receiver operating characteristic curve 0.66 [95% CI 0.63 to 0.69]). Although model calibration was also acceptable in the external validation sample (intercept = -0.55, slope = 0.89), some miscalibration was observed for high-risk groups. The decision curve analysis estimated that, if decisions to recommend further intervention were based on risk scores, screening could lead to a net reduction of 40 unnecessary interventions for every 100 patients presenting to primary care compared to a "treat all" approach. Limitations of the method include the model being restricted to using prognostic factors measured in existing studies and using stepwise methods to specify the model. Limitations of the model include modest discrimination performance. The model also requires recalibration for local settings.

CONCLUSIONS

Based on its performance in these cohorts, this five-item prognostic model for patients with acute LBP may be a useful tool for estimating risk of chronic LBP. Further validation is required to determine whether screening with this model leads to a net reduction in unnecessary interventions provided to low-risk patients.

A systematic review and individual patient data meta-analysis of prognostic factors for foot ulceration in people with diabetes: the international research collaboration for the prediction of diabetic foot ulcerations (PODUS)

BACKGROUND

Annual foot risk assessment of people with diabetes is recommended in national and international clinical guidelines. At present, these are consensus based and use only a proportion of the available evidence.

OBJECTIVES

We undertook a systematic review of individual patient data (IPD) to identify the most highly prognostic factors for foot ulceration (i.e. symptoms, signs, diagnostic tests) in people with diabetes.

DATA SOURCES

Studies were identified from searches of MEDLINE and EMBASE.

REVIEW METHODS

The electronic search strategies for MEDLINE and EMBASE databases created during an aggregate systematic review of predictive factors for foot ulceration in diabetes were updated and rerun to January 2013. One reviewer applied the IPD review eligibility criteria to the full-text articles of the studies identified in our literature search and also to all studies excluded from our aggregate systematic review to ensure that we did not miss eligible IPD. A second reviewer applied the eligibility criteria to a 10% random sample of the abstract search yield to check that no relevant material was missed. This review includes exposure variables (risk factors) only from individuals who were free of foot ulceration at the time of study entry and who had a diagnosis of diabetes mellitus (either type 1 or type 2). The outcome variable was incident ulceration.

RESULTS

Our search identified 16 cohort studies and we obtained anonymised IPD for 10. These data were collected from more than 16,000 people with diabetes worldwide and reanalysed by us. One data set was kept for independent validation. The data sets contributing IPD covered a range of temporal, geographical and clinical settings. We therefore selected random-effects meta-analysis, which assumes not that all the estimates from each study are estimates of the same underlying true value, but rather that the estimates belong to the same distribution. We selected candidate variables for meta-analysis using specific criteria. After univariate meta-analyses, the most clinically important predictors were identified by an international steering committee for inclusion in the primary, multivariable meta-analysis. Age, sex, duration of diabetes, monofilaments and pulses were considered most prognostically important. Meta-analyses based on data from the entire IPD population found that an inability to feel a 10-g monofilament [odds ratio (OR) 3.184, 95% confidence interval (CI) 2.654 to 3.82], at least one absent pedal pulse (OR 1.968, 95% CI 1.624 to 2.386), a longer duration of a diagnosis of diabetes (OR 1.024, 95% CI 1.011 to 1.036) and a previous history of ulceration (OR 6.589, 95% CI 2.488 to 17.45) were all predictive of risk. Female sex was protective (OR 0.743, 95% CI 0.598 to 0.922).

LIMITATIONS

It was not possible to perform a meta-analysis using a one-step approach because we were unable to procure copies of one of the data sets and instead accessed data via Safe Haven.

CONCLUSIONS

The findings from this review identify risk assessment procedures that can reliably inform national and international diabetes clinical guideline foot risk assessment procedures. The evidence from a large sample of patients in worldwide settings show that the use of a 10-g monofilament or one absent pedal pulse will identify those at moderate or intermediate risk of foot ulceration, and a history of foot ulcers or lower-extremity amputation is sufficient to identify those at high risk. We propose the development of a clinical prediction rule (CPR) from our existing model using the following predictor variables: insensitivity to a 10-g monofilament, absent pedal pulses and a history of ulceration or lower-extremities amputations. This CPR could replace the many tests, signs and symptoms that patients currently have measured using equipment that is either costly or difficult to use.

STUDY REGISTRATION

This study is registered as PROSPERO CRD42011001841.

FUNDING

The National Institute for Health Research Health Technology Assessment programme.

Prediction of coronary artery disease risk based on multiple longitudinal biomarkers

In the last decade, few topics in the area of cardiovascular disease (CVD) research have received as much attention as risk prediction. One of the well-documented risk factors for CVD is high blood pressure (BP). Traditional CVD risk prediction models consider BP levels measured at a single time and such models form the basis for current clinical guidelines for CVD prevention. However, in clinical practice, BP levels are often observed and recorded in a longitudinal fashion. Information on BP trajectories can be powerful predictors for CVD events. We consider joint modeling of time to coronary artery disease and individual longitudinal measures of systolic and diastolic BPs in a primary care cohort with up to 20 years of follow-up. We applied novel prediction metrics to assess the predictive performance of joint models. Predictive performances of proposed joint models and other models were assessed via simulations and illustrated using the primary care cohort.

An Application of Bayesian Approach in Modeling Risk of Death in an Intensive Care Unit

Background and Objectives

There are not many studies that attempt to model intensive care unit (ICU) risk of death in developing countries, especially in South East Asia. The aim of this study was to propose and describe application of a Bayesian approach in modeling in-ICU deaths in a Malaysian ICU.

Methods

This was a prospective study in a mixed medical-surgery ICU in a multidisciplinary tertiary referral hospital in Malaysia. Data collection included variables that were defined in Acute Physiology and Chronic Health Evaluation IV (APACHE IV) model. Bayesian Markov Chain Monte Carlo (MCMC) simulation approach was applied in the development of four multivariate logistic regression predictive models for the ICU, where the main outcome measure was in-ICU mortality risk. The performance of the models were assessed through overall model fit, discrimination and calibration measures. Results from the Bayesian models were also compared against results obtained using frequentist maximum likelihood method.

Results

The study involved 1,286 consecutive ICU admissions between January 1, 2009 and June 30, 2010, of which 1,111 met the inclusion criteria. Patients who were admitted to the ICU were generally younger, predominantly male, with low co-morbidity load and mostly under mechanical ventilation. The overall in-ICU mortality rate was 18.5% and the overall mean Acute Physiology Score (APS) was 68.5. All four models exhibited good discrimination, with area under receiver operating characteristic curve (AUC) values approximately 0.8. Calibration was acceptable (Hosmer-Lemeshow p-values > 0.05) for all models, except for model M3. Model M1 was identified as the model with the best overall performance in this study.

Conclusion

Four prediction models were proposed, where the best model was chosen based on its overall performance in this study. This study has also demonstrated the promising potential of the Bayesian MCMC approach as an alternative in the analysis and modeling of in-ICU mortality outcomes.

Evaluation of a Decision Support System for Obstructive Sleep Apnea with Nonlinear Analysis of Respiratory Signals

Introduction

Obstructive Sleep Apnea (OSA) is a common sleep disorder requiring the time/money consuming polysomnography for diagnosis. Alternative methods for initial evaluation are sought. Our aim was the prediction of Apnea-Hypopnea Index (AHI) in patients potentially suffering from OSA based on nonlinear analysis of respiratory biosignals during sleep, a method that is related to the pathophysiology of the disorder.

Materials and Methods

Patients referred to a Sleep Unit (135) underwent full polysomnography. Three nonlinear indices (Largest Lyapunov Exponent, Detrended Fluctuation Analysis and Approximate Entropy) extracted from two biosignals (airflow from a nasal cannula, thoracic movement) and one linear derived from Oxygen saturation provided input to a data mining application with contemporary classification algorithms for the creation of predictive models for AHI.

Results

A linear regression model presented a correlation coefficient of 0.77 in predicting AHI. With a cutoff value of AHI = 8, the sensitivity and specificity were 93% and 71.4% in discrimination between patients and normal subjects. The decision tree for the discrimination between patients and normal had sensitivity and specificity of 91% and 60%, respectively. Certain obtained nonlinear values correlated significantly with commonly accepted physiological parameters of people suffering from OSA.

Discussion

We developed a predictive model for the presence/severity of OSA using a simple linear equation and additional decision trees with nonlinear features extracted from 3 respiratory recordings. The accuracy of the methodology is high and the findings provide insight to the underlying pathophysiology of the syndrome.

Conclusions

Reliable predictions of OSA are possible using linear and nonlinear indices from only 3 respiratory signals during sleep. The proposed models could lead to a better study of the pathophysiology of OSA and facilitate initial evaluation/follow up of suspected patients OSA utilizing a practical low cost methodology.

Trial Registration

ClinicalTrials.gov NCT01161381

Swallowing sparing intensity modulated radiotherapy (SW-IMRT) in head and neck cancer: Clinical validation according to the model-based approach

PURPOSE

The aim of this study was to clinically validate a multivariable normal tissue complication probability (NTCP) model for grade 2-4 swallowing dysfunction at 6months after radiotherapy or chemoradiation (SWALM6) in head and neck cancer patients treated with swallowing sparing intensity modulated radiotherapy (SW-IMRT) and to test if SW-IMRT resulted in a reduction of the prevalence of SWALM6.

MATERIALS AND METHODS

The primary endpoint was SWALM6. For all 186 patients, a standard IMRT (parotid sparing) and a SW-IMRT plan (additional constraints for swallowing organs at risk) was created. The difference in NTCP for SWALM6 (ΔNTCPSWALM6=NTCPstandard-NTCPSW-IMRT) was calculated. Patients were treated with SW-IMRT. The external validation of the NTCP model was analyzed by comparing performance measures.

RESULTS

The mean ΔNTCPSWALM6 was 4.9% (range 0.01-17.3%), with a significant lower mean predicted NTCPSW-IMRT of 22.6% (95% CI 20.2-24.9%), compared to NTCPstandard of 27.5% (95% CI 24.9-29.9%) (p<0.001). There was a perfect correspondence of NTCPSW-IMRT with the observed prevalence of SWALM6 (22.6%). The overall model performance, discrimination and 'goodness of fit' were good.

CONCLUSION

We externally validated the multivariable NTCP model for SWALM6 in SW-IMRT treated patients, showing reduced swallowing dysfunction by reducing the dose parameters included in this NTCP model.

Variable selection in covariate dependent random partition models: an application to urinary tract infection

Lower urinary tract symptoms can indicate the presence of urinary tract infection (UTI), a condition that if it becomes chronic requires expensive and time consuming care as well as leading to reduced quality of life. Detecting the presence and gravity of an infection from the earliest symptoms is then highly valuable. Typically, white blood cell (WBC) count measured in a sample of urine is used to assess UTI. We consider clinical data from 1341 patients in their first visit in which UTI (i.e. WBC ≥ 1) is diagnosed. In addition, for each patient, a clinical profile of 34 symptoms was recorded. In this paper, we propose a Bayesian nonparametric regression model based on the Dirichlet process prior aimed at providing the clinicians with a meaningful clustering of the patients based on both the WBC (response variable) and possible patterns within the symptoms profiles (covariates). This is achieved by assuming a probability model for the symptoms as well as for the response variable. To identify the symptoms most associated to UTI, we specify a spike and slab base measure for the regression coefficients: this induces dependence of symptoms selection on cluster assignment. Posterior inference is performed through Markov Chain Monte Carlo methods.

A weighting approach for judging the effect of patient strata on high-dimensional risk prediction signatures

Background

High-dimensional molecular measurements, e.g. gene expression data, can be linked to clinical time-to-event endpoints by Cox regression models and regularized estimation approaches, such as componentwise boosting, and can incorporate a large number of covariates as well as provide variable selection. If there is heterogeneity due to known patient subgroups, a stratified Cox model allows for separate baseline hazards in each subgroup. Variable selection will still depend on the relative stratum sizes in the data, which might be a convenience sample and not representative for future applications. Such effects need to be systematically investigated and could even help to more reliably identify components of risk prediction signatures.

Results

Correspondingly, we propose a weighted regression approach based on componentwise likelihood-based boosting which is implemented in the R package CoxBoost (https://github.com/binderh/CoxBoost). This approach focuses on building a risk prediction signature for a specific stratum by down-weighting the observations from the other strata using a range of weights. Stability of selection for specific covariates as a function of the weights is investigated by resampling inclusion frequencies, and two types of corresponding visualizations are suggested. This is illustrated for two applications with methylation and gene expression measurements from cancer patients.

Conclusion

The proposed approach is meant to point out components of risk prediction signatures that are specific to the stratum of interest and components that are also important to other strata. Performance is mostly improved by incorporating down-weighted information from the other strata. This suggests more general usefulness for risk prediction signature development in data with heterogeneity due to known subgroups.

National Veterans Health Administration inpatient risk stratification models for hospital-acquired acute kidney injury

OBJECTIVE

Hospital-acquired acute kidney injury (HA-AKI) is a potentially preventable cause of morbidity and mortality. Identifying high-risk patients prior to the onset of kidney injury is a key step towards AKI prevention.

MATERIALS AND METHODS

A national retrospective cohort of 1,620,898 patient hospitalizations from 116 Veterans Affairs hospitals was assembled from electronic health record (EHR) data collected from 2003 to 2012. HA-AKI was defined at stage 1+, stage 2+, and dialysis. EHR-based predictors were identified through logistic regression, least absolute shrinkage and selection operator (lasso) regression, and random forests, and pair-wise comparisons between each were made. Calibration and discrimination metrics were calculated using 50 bootstrap iterations. In the final models, we report odds ratios, 95% confidence intervals, and importance rankings for predictor variables to evaluate their significance.

RESULTS

The area under the receiver operating characteristic curve (AUC) for the different model outcomes ranged from 0.746 to 0.758 in stage 1+, 0.714 to 0.720 in stage 2+, and 0.823 to 0.825 in dialysis. Logistic regression had the best AUC in stage 1+ and dialysis. Random forests had the best AUC in stage 2+ but the least favorable calibration plots. Multiple risk factors were significant in our models, including some nonsteroidal anti-inflammatory drugs, blood pressure medications, antibiotics, and intravenous fluids given during the first 48 h of admission.

CONCLUSIONS

This study demonstrated that, although all the models tested had good discrimination, performance characteristics varied between methods, and the random forests models did not calibrate as well as the lasso or logistic regression models. In addition, novel modifiable risk factors were explored and found to be significant.

Assessing incremental value of biomarkers with multi-phase nested case-control studies

Accurate risk prediction models are needed to identify different risk groups for individualized prevention and treatment strategies. In the Nurses' Health Study, to examine the effects of several biomarkers and genetic markers on the risk of rheumatoid arthritis (RA), a three-phase nested case-control (NCC) design was conducted, in which two sequential NCC subcohorts were formed with one nested within the other, and one set of new markers measured on each of the subcohorts. One objective of the study is to evaluate clinical values of novel biomarkers in improving upon existing risk models because of potential cost associated with assaying biomarkers. In this paper, we develop robust statistical procedures for constructing risk prediction models for RA and estimating the incremental value (IncV) of new markers based on three-phase NCC studies. Our method also takes into account possible time-varying effects of biomarkers in risk modeling, which allows us to more robustly assess the biomarker utility and address the question of whether a marker is better suited for short-term or long-term risk prediction. The proposed procedures are shown to perform well in finite samples via simulation studies.

Using prior knowledge from cellular pathways and molecular networks for diagnostic specimen classification

For many complex diseases, an earlier and more reliable diagnosis is considered a key prerequisite for developing more effective therapies to prevent or delay disease progression. Classical statistical learning approaches for specimen classification using omics data, however, often cannot provide diagnostic models with sufficient accuracy and robustness for heterogeneous diseases like cancers or neurodegenerative disorders. In recent years, new approaches for building multivariate biomarker models on omics data have been proposed, which exploit prior biological knowledge from molecular networks and cellular pathways to address these limitations. This survey provides an overview of these recent developments and compares pathway- and network-based specimen classification approaches in terms of their utility for improving model robustness, accuracy and biological interpretability. Different routes to translate omics-based multifactorial biomarker models into clinical diagnostic tests are discussed, and a previous study is presented as example.

Mortality risk prediction in burn injury: Comparison of logistic regression with machine learning approaches

INTRODUCTION

Predicting mortality from burn injury has traditionally employed logistic regression models. Alternative machine learning methods have been introduced in some areas of clinical prediction as the necessary software and computational facilities have become accessible. Here we compare logistic regression and machine learning predictions of mortality from burn.

METHODS

An established logistic mortality model was compared to machine learning methods (artificial neural network, support vector machine, random forests and naïve Bayes) using a population-based (England & Wales) case-cohort registry. Predictive evaluation used: area under the receiver operating characteristic curve; sensitivity; specificity; positive predictive value and Youden's index.

RESULTS

All methods had comparable discriminatory abilities, similar sensitivities, specificities and positive predictive values. Although some machine learning methods performed marginally better than logistic regression the differences were seldom statistically significant and clinically insubstantial. Random forests were marginally better for high positive predictive value and reasonable sensitivity. Neural networks yielded slightly better prediction overall. Logistic regression gives an optimal mix of performance and interpretability.

DISCUSSION

The established logistic regression model of burn mortality performs well against more complex alternatives. Clinical prediction with a small set of strong, stable, independent predictors is unlikely to gain much from machine learning outside specialist research contexts.

Translocation t(11;14) Is Associated With Adverse Outcome in Patients With Newly Diagnosed AL Amyloidosis When Treated With Bortezomib-Based Regimens

Purpose

Bortezomib has become a cornerstone in the treatment of AL amyloidosis. In this study, we addressed the prognostic impact of cytogenetic aberrations for bortezomib-treated patients.

Patients and Methods

We analyzed a consecutive series of 101 patients with AL amyloidosis treated with bortezomib-dexamethasone as first-line treatment by interphase fluorescence in situ hybridization (iFISH). Patients were ineligible for high-dose chemotherapy, which would put them at risk for cardiac or renal failure, and thus represented a poor-risk group.

Results

Presence of t(11;14), versus its absence, was associated with inferior hematologic event-free survival (median, 3.4 v 8.8 months, respectively; P = .002), overall survival (median, 8.7 v 40.7 months, respectively; P = .05), and remission rate (≥ very good partial remission; 23% v 47%, respectively; P = .02). In multivariable Cox regression models incorporating established hematologic and clinical risk factors, t(11;14) was an independent adverse prognostic marker for hematologic event-free survival (hazard ratio, 2.94; 95% CI, 1.37 to 6.25; P = .006) and overall survival (hazard ratio, 3.13; 95% CI, 1.16 to 8.33; P = .03), but not for remission (≥ very good partial remission). Markedly, the multiple myeloma high-risk iFISH aberrations t(4;14), t(14;16), del(17p), and gain of 1q21 conferred no adverse prognosis in this bortezomib-dexamethasone–treated group. After backward variable selection, the final multivariable model was validated in a consecutive series of 32 patients treated with bortezomib, dexamethasone, and cyclophosphamide.

Conclusion

iFISH results are important independent prognostic factors in AL amyloidosis. In contrast to our recently published results with melphalan and dexamethasone standard therapy, bortezomib is less beneficial to patients harboring t(11;14), whereas it effectively alleviates the poor prognosis inherent to high-risk aberrations. Given the discrepant response to different treatment modalities, iFISH may help to guide therapeutic choices in these poor-risk patients requiring rapid hematologic response.

Incident atrial fibrillation hazard in hypertensive population: a risk function from and for clinical practice

Determining the risk of atrial fibrillation within the hypertensive population without ischemic vascular disease would aid in decision making on preventive approaches. Accordingly, we aimed to estimate the risk of incident atrial fibrillation in this population. We conducted an historical cohort study between July 1, 2006, and December 31, 2011, using anonymized longitudinal patient information from primary care and hospital discharge records contained in the System for the Development of Research in Primary Care database. We included 255 440 hypertensive patients, aged ≥55 years at the time of study entry. Individuals with previous atrial fibrillation, ischemic heart disease, stroke, and peripheral artery disease were excluded. To build the incident atrial fibrillation risk function, a derivation and a validation cohort were defined, representing 60% and 40% of the entire database, respectively, and a Cox proportional hazards model was fitted. Atrial fibrillation incidence was 7.24 per 1000 person-years (95% confidence interval, 7.08-7.40). The final model included age, weight, total cholesterol, heart failure, valvular heart disease, and antihypertensive treatment. Its concordance index (standard error) was 0.769 (0.004) and 0.768 (0.005) in the derivation and validation datasets, respectively. This research provides a tool, built with variables from daily clinical practice, that can be readily used in the primary care setting to predict atrial fibrillation incidence in the hypertensive population without ischemic vascular disease. The tool may help tailor individualized diagnostic and preventive care decisions.

The performance of diabetes risk prediction models in new populations: the role of ethnicity of the development cohort

It is believed that diabetes risk scores need to be ethnic specific. However, this prerequisite has not been tested. We examined the performance of several risk models, developed in various populations, in a Europid and a South Asian population. The performance of 14 published risk prediction models were tested in two prospective studies: the Australian Diabetes, Obesity and Lifestyle (AusDiab) study and the Mauritius non-communicable diseases survey. Eight models were developed in Europid populations; the remainder in various non-Europid populations. Model performance was assessed using area under the receiver operating characteristic curves (discrimination), Hosmer-Lemeshow tests (goodness-of-fit) and Brier scores (accuracy). In both AusDiab and Mauritius, discrimination was highest for a model developed in a mixed population (non-Hispanic white and African American) and lowest for a model developed in a Europid population. Discrimination for all scores was higher in AusDiab than in Mauritius. For almost all models, goodness-of-fit was poor irrespective of the ethnicity of the development cohort, and accuracy was higher in AusDiab compared to Mauritius. Our results suggest that similarity of ethnicity or similarity of diabetes risk may not be the best way of identifying models that will perform well in another population. Differences in study methodology likely account for much of the difference in the performance. Thus, identifying models which use measurements that are clearly described and easily reproducible for both research and clinical settings may be more important.

Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration

The TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) Statement includes a 22-item checklist, which aims to improve the reporting of studies developing, validating, or updating a prediction model, whether for diagnostic or prognostic purposes. The TRIPOD Statement aims to improve the transparency of the reporting of a prediction model study regardless of the study methods used. This explanation and elaboration document describes the rationale; clarifies the meaning of each item; and discusses why transparent reporting is important, with a view to assessing risk of bias and clinical usefulness of the prediction model. Each checklist item of the TRIPOD Statement is explained in detail and accompanied by published examples of good reporting. The document also provides a valuable reference of issues to consider when designing, conducting, and analyzing prediction model studies. To aid the editorial process and help peer reviewers and, ultimately, readers and systematic reviewers of prediction model studies, it is recommended that authors include a completed checklist in their submission. The TRIPOD checklist can also be downloaded from www.tripod-statement.org.

Applying measures of discriminatory accuracy to revisit traditional risk factors for being small for gestational age in Sweden: a national cross-sectional study

OBJECTIVES

Small for gestational age (SGA) is considered as an indicator of intrauterine growth restriction, and multiple maternal and newborn characteristics have been identified as risk factors for SGA. This knowledge is mainly based on measures of average association (ie, OR) that quantify differences in average risk between exposed and unexposed groups. Nevertheless, average associations do not assess the discriminatory accuracy of the risk factors (ie, its ability to discriminate the babies who will develop SGA from those that will not). Therefore, applying measures of discriminatory accuracy rather than measures of association only, our study revisits known risk factors of SGA and discusses their role from a public health perspective.

DESIGN

Cross-sectional study. We measured maternal (ie, smoking, hypertension, age, marital status, education) and delivery (ie, sex, gestational age, birth order) characteristics and performed logistic regression models to estimate both ORs and measures of discriminatory accuracy, like the area under the receiver operating characteristic curve (AU-ROC) and the net reclassification improvement.

SETTING

Data were obtained from the Swedish Medical Birth Registry.

PARTICIPANTS

Our sample included 731 989 babies born during 1987-1993.

RESULTS

We replicated the expected associations. For instance, smoking (OR=2.57), having had a previous SGA baby (OR=5.48) and hypertension (OR=4.02) were strongly associated with SGA. However, they show a very small discriminatory accuracy (AU-ROC≈0.5). The discriminatory accuracy increased, but remained unsatisfactorily low (AU-ROC=0.6), when including all variables studied in the same model.

CONCLUSIONS

Traditional risk factors for SGA alone or in combination have a low accuracy for discriminating babies with SGA from those without SGA. A proper understanding of these findings is of fundamental relevance to address future research and to design policymaking recommendations in a more informed way.

Prediction of future labour market outcome in a cohort of long-term sick-listed Danes

BACKGROUND

Targeted interventions for the long-term sick-listed may prevent permanent exclusion from the labour force. We aimed to develop a prediction method for identifying high risk groups for continued or recurrent long-term sickness absence, unemployment, or disability among persons on long-term sick leave.

METHODS

We obtained individual characteristics and follow-up data from the Danish Register of Sickness Absence Compensation Benefits and Social Transfer Payments (RSS) during 2004 to 2010 for 189,279 Danes who experienced a period of long-term sickness absence (4+ weeks). In a learning data set, statistical prediction methods were built using logistic regression and a discrete event simulation approach for a one year prediction horizon. Personalized risk profiles were obtained for five outcomes: employment, unemployment, recurrent sickness absence, continuous long-term sickness absence, and early retirement from the labour market. Predictor variables included gender, age, socio-economic position, job type, chronic disease status, history of sickness absence, and prior history of unemployment. Separate models were built for times of economic growth (2005-2007) and times of recession (2008-2010). The accuracy of the prediction models was assessed with analyses of Receiver Operating Characteristic (ROC) curves and the Brier score in an independent validation data set.

RESULTS

In comparison with a null model which ignored the predictor variables, logistic regression achieved only moderate prediction accuracy for the five outcome states. Results obtained with discrete event simulation were comparable with logistic regression.

CONCLUSIONS

Only moderate prediction accuracy could be achieved using the selected information from the Danish register RSS. Other variables need to be included in order to establish a prediction method which provides more accurate risk profiles for long-term sick-listed persons.

Derivation and validation of multimarker prognostication for normotensive patients with acute symptomatic pulmonary embolism

RATIONALE

Not all patients with acute pulmonary embolism (PE) have a high risk of an adverse short-term outcome.

OBJECTIVES

This prospective cohort study aimed to develop a multimarker prognostic model that accurately classifies normotensive patients with PE into low and high categories of risk of adverse medical outcomes.

METHODS

The study enrolled 848 outpatients from the PROTECT (PROgnosTic valuE of Computed Tomography) study (derivation cohort) and 529 patients from the Prognostic Factors for Pulmonary Embolism (PREP) study (validation cohort). Investigators assessed study participants for a 30-day complicated course, defined as death from any cause, hemodynamic collapse, and/or adjudicated recurrent PE.

MEASUREMENTS AND MAIN RESULTS

A complicated course occurred in 63 (7.4%) of the 848 normotensive patients with acute symptomatic PE in the derivation cohort and in 24 patients (4.5%) in the validation cohort. The final model included the simplified Pulmonary Embolism Severity Index, cardiac troponin I, brain natriuretic peptide, and lower limb ultrasound testing. The model performed similarly in the derivation (c-index of 0.75) and validation (c-index of 0.85) cohorts. The combination of the simplified Pulmonary Embolism Severity Index and brain natriuretic peptide testing showed a negative predictive value for a complicated course of 99.1 and 100% in the derivation and validation cohorts, respectively. The combination of all modalities had a positive predictive value for the prediction of a complicated course of 25.8% in the derivation cohort and 21.2% in the validation cohort.

CONCLUSIONS

For normotensive patients who have acute PE, we derived and validated a multimarker model that predicts all-cause mortality, hemodynamic collapse, and/or recurrent PE within the following 30 days.