Construction and validation of a prognostic model across several studies, with an application in superficial bladder cancer

Meta-analysis using individual participant data: one-stage and two-stage approaches, and why they may differ

Meta‐analysis using individual participant data (IPD) obtains and synthesises the raw, participant‐level data from a set of relevant studies. The IPD approach is becoming an increasingly popular tool as an alternative to traditional aggregate data meta‐analysis, especially as it avoids reliance on published results and provides an opportunity to investigate individual‐level interactions, such as treatment‐effect modifiers. There are two statistical approaches for conducting an IPD meta‐analysis: one‐stage and two‐stage. The one‐stage approach analyses the IPD from all studies simultaneously, for example, in a hierarchical regression model with random effects. The two‐stage approach derives aggregate data (such as effect estimates) in each study separately and then combines these in a traditional meta‐analysis model. There have been numerous comparisons of the one‐stage and two‐stage approaches via theoretical consideration, simulation and empirical examples, yet there remains confusion regarding when each approach should be adopted, and indeed why they may differ.

In this tutorial paper, we outline the key statistical methods for one‐stage and two‐stage IPD meta‐analyses, and provide 10 key reasons why they may produce different summary results. We explain that most differences arise because of different modelling assumptions, rather than the choice of one‐stage or two‐stage itself. We illustrate the concepts with recently published IPD meta‐analyses, summarise key statistical software and provide recommendations for future IPD meta‐analyses. © 2016 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.

Developing a predictive risk model for first-line antiretroviral therapy failure in South Africa

Introduction

A substantial number of patients with HIV in South Africa have failed first-line antiretroviral therapy (ART). Although individual predictors of first-line ART failure have been identified, few studies in resource-limited settings have been large enough for predictive modelling. Understanding the absolute risk of first-line failure is useful for patient monitoring and for effectively targeting limited resources for second-line ART. We developed a predictive model to identify patients at the greatest risk of virologic failure on first-line ART, and to estimate the proportion of patients needing second-line ART over five years on treatment.

Methods

A cohort of patients aged ≥18 years from nine South African HIV clinics on first-line ART for at least six months were included. Viral load measurements and baseline predictors were obtained from medical records. We used stepwise selection of predictors in accelerated failure-time models to predict virologic failure on first-line ART (two consecutive viral load levels >1000 copies/mL). Multiple imputations were used to assign missing baseline variables. The final model was selected using internal-external cross-validation maximizing model calibration at five years on ART, and model discrimination, measured using Harrell's C-statistic. Model covariates were used to create a predictive score for risk group of ART failure.

Results

A total of 72,181 patients were included in the analysis, with an average of 21.5 months (IQR: 8.8–41.5) of follow-up time on first-line ART. The final predictive model had a Weibull distribution and the final predictors of virologic failure were men of all ages, young women, nevirapine use in first-line regimen, low baseline CD4 count, high mean corpuscular volume, low haemoglobin, history of TB and missed visits during the first six months on ART. About 24.4% of patients in the highest quintile and 9.4% of patients in the lowest quintile of risk were predicted to experience treatment failure over five years on ART.

Conclusions

Age, sex, CD4 count and having any missed visits during the first six months on ART were the strongest predictors of ART failure. The predictive model identified patients at high risk of failure, and the predicted failure rates over five years closely reflected actual rates of failure.

Geographic and temporal validity of prediction models: different approaches were useful to examine model performance

OBJECTIVE

Validation of clinical prediction models traditionally refers to the assessment of model performance in new patients. We studied different approaches to geographic and temporal validation in the setting of multicenter data from two time periods.

STUDY DESIGN AND SETTING

We illustrated different analytic methods for validation using a sample of 14,857 patients hospitalized with heart failure at 90 hospitals in two distinct time periods. Bootstrap resampling was used to assess internal validity. Meta-analytic methods were used to assess geographic transportability. Each hospital was used once as a validation sample, with the remaining hospitals used for model derivation. Hospital-specific estimates of discrimination (c-statistic) and calibration (calibration intercepts and slopes) were pooled using random-effects meta-analysis methods. I² statistics and prediction interval width quantified geographic transportability. Temporal transportability was assessed using patients from the earlier period for model derivation and patients from the later period for model validation.

RESULTS

Estimates of reproducibility, pooled hospital-specific performance, and temporal transportability were on average very similar, with c-statistics of 0.75. Between-hospital variation was moderate according to I² statistics and prediction intervals for c-statistics.

CONCLUSION

This study illustrates how performance of prediction models can be assessed in settings with multicenter data at different time periods.

Prediction models for cardiovascular disease risk in the general population: systematic review

OBJECTIVE

 To provide an overview of prediction models for risk of cardiovascular disease (CVD) in the general population.

DESIGN

 Systematic review.

DATA SOURCES

 Medline and Embase until June 2013.

ELIGIBILITY CRITERIA FOR STUDY SELECTION

 Studies describing the development or external validation of a multivariable model for predicting CVD risk in the general population.

RESULTS

 9965 references were screened, of which 212 articles were included in the review, describing the development of 363 prediction models and 473 external validations. Most models were developed in Europe (n=167, 46%), predicted risk of fatal or non-fatal coronary heart disease (n=118, 33%) over a 10 year period (n=209, 58%). The most common predictors were smoking (n=325, 90%) and age (n=321, 88%), and most models were sex specific (n=250, 69%). Substantial heterogeneity in predictor and outcome definitions was observed between models, and important clinical and methodological information were often missing. The prediction horizon was not specified for 49 models (13%), and for 92 (25%) crucial information was missing to enable the model to be used for individual risk prediction. Only 132 developed models (36%) were externally validated and only 70 (19%) by independent investigators. Model performance was heterogeneous and measures such as discrimination and calibration were reported for only 65% and 58% of the external validations, respectively.

CONCLUSIONS

 There is an excess of models predicting incident CVD in the general population. The usefulness of most of the models remains unclear owing to methodological shortcomings, incomplete presentation, and lack of external validation and model impact studies. Rather than developing yet another similar CVD risk prediction model, in this era of large datasets, future research should focus on externally validating and comparing head-to-head promising CVD risk models that already exist, on tailoring or even combining these models to local settings, and investigating whether these models can be extended by addition of new predictors.

Predicting microbiologically defined infection in febrile neutropenic episodes in children: global individual participant data multivariable meta-analysis

BACKGROUND

Risk-stratified management of fever with neutropenia (FN), allows intensive management of high-risk cases and early discharge of low-risk cases. No single, internationally validated, prediction model of the risk of adverse outcomes exists for children and young people. An individual patient data (IPD) meta-analysis was undertaken to devise one.

METHODS

The 'Predicting Infectious Complications in Children with Cancer' (PICNICC) collaboration was formed by parent representatives, international clinical and methodological experts. Univariable and multivariable analyses, using random effects logistic regression, were undertaken to derive and internally validate a risk-prediction model for outcomes of episodes of FN based on clinical and laboratory data at presentation.

RESULTS

Data came from 22 different study groups from 15 countries, of 5127 episodes of FN in 3504 patients. There were 1070 episodes in 616 patients from seven studies available for multivariable analysis. Univariable analyses showed associations with microbiologically defined infection (MDI) in many items, including higher temperature, lower white cell counts and acute myeloid leukaemia, but not age. Patients with osteosarcoma/Ewings sarcoma and those with more severe mucositis were associated with a decreased risk of MDI. The predictive model included: malignancy type, temperature, clinically 'severely unwell', haemoglobin, white cell count and absolute monocyte count. It showed moderate discrimination (AUROC 0.723, 95% confidence interval 0.711-0.759) and good calibration (calibration slope 0.95). The model was robust to bootstrap and cross-validation sensitivity analyses.

CONCLUSIONS

This new prediction model for risk of MDI appears accurate. It requires prospective studies assessing implementation to assist clinicians and parents/patients in individualised decision making.

Development of risk scoring system for stratifying population for hepatocellular carcinoma screening

The age and risk level that warrants hepatocellular carcinoma (HCC) screening remains to be defined. To develop risk scores for stratifying average-risk population for mass HCC screening, we conducted a pooled analysis using data from three cohorts involving 12,377 Taiwanese adults 20-80 years of age. During 191,240.3 person-years of follow-up, 387 HCCs occurred. We derived risk scores from Cox's model in two thirds of participants and used another one third for model validation. Besides assessing discrimination and calibration, we performed decision curve analysis to translate findings into public health policy. A risk score according to age, sex, alanine aminotransferase, previous chronic liver disease, family history of HCC, and cumulative smoking had good discriminatory accuracy in both model derivation and validation sets (c-statistics for 3-, 5-, and 10-year risk prediction: 0.76-0.83). It also performed well across cohorts and diverse subgroups. Decision curve analyses revealed that use of the score in selecting persons for screening improved benefit at threshold probabilities of >2% 10-year risk, compared with current guidelines and a strategy of screening all hepatitis B carriers. Using 10-year risk 2% as a threshold for initiating screening, the screening age ranged from 20 to ≥60 years, depending on the tertile of risk scores and status of hepatitis B/C virus infection. Combining risk-score tertile levels and hepatitis virus status to stratify participants was more sensitive than current guidelines for HCC detection within 10 years (89.4% vs. 76.8%), especially for young-onset HCCs <50 years (79.4% vs. 40.6%), under slightly lower specificity (67.8% vs. 71.8%).

CONCLUSION

A simple HCC prediction algorithm was developed using accessible variables combined with hepatitis virus status, which allows selection of asymptomatic persons for priority of HCC screening.

Multivariate meta-analysis of individual participant data helped externally validate the performance and implementation of a prediction model

OBJECTIVES

Our aim was to improve meta-analysis methods for summarizing a prediction model's performance when individual participant data are available from multiple studies for external validation.

STUDY DESIGN AND SETTING

We suggest multivariate meta-analysis for jointly synthesizing calibration and discrimination performance, while accounting for their correlation. The approach estimates a prediction model's average performance, the heterogeneity in performance across populations, and the probability of "good" performance in new populations. This allows different implementation strategies (e.g., recalibration) to be compared. Application is made to a diagnostic model for deep vein thrombosis (DVT) and a prognostic model for breast cancer mortality.

RESULTS

In both examples, multivariate meta-analysis reveals that calibration performance is excellent on average but highly heterogeneous across populations unless the model's intercept (baseline hazard) is recalibrated. For the cancer model, the probability of "good" performance (defined by C statistic ≥0.7 and calibration slope between 0.9 and 1.1) in a new population was 0.67 with recalibration but 0.22 without recalibration. For the DVT model, even with recalibration, there was only a 0.03 probability of "good" performance.

CONCLUSION

Multivariate meta-analysis can be used to externally validate a prediction model's calibration and discrimination performance across multiple populations and to evaluate different implementation strategies.

An updated prediction model of the global risk of cardiovascular disease in HIV-positive persons: The Data-collection on Adverse Effects of Anti-HIV Drugs (D:A:D) study

BACKGROUND

With the aging of the population living with HIV, the absolute risk of cardiovascular disease (CVD) is increasing. There is a need to further facilitate the identification of persons at elevated risk in routine practice.

METHODS AND RESULTS

Prospective information was collected on 32,663 HIV-positive persons from 20 countries in Europe and Australia, who were free of CVD at entry into the Data-collection on Adverse Effects of Anti-HIV Drugs (D:A:D) study. Cox regression models (full and reduced) were developed that predict the risk of a global CVD endpoint. The predictive performance of the D:A:D models were compared with a recent CVD prediction model from the Framingham study, which was assessed recalibrated to the D:A:D dataset. A total of 1010 CVD events occurred during 186,364.5 person-years. The full D:A:D CVD prediction model included age, gender, systolic blood pressure, smoking status, family history of CVD, diabetes, total cholesterol, high-density lipoprotein, CD4 lymphocyte count, cumulative exposure to protease- and nucleoside reverse transcriptase-inhibitors, and current use of abacavir. A reduced model omitted antiretroviral therapies. The D:A:D models statistically significantly predicted risk more accurately than the recalibrated Framingham model (Harrell's c-statistic of 0.791, 0.783 and 0.766 for the D:A:D full, D:A:D reduced, and Framingham models respectively; p < 0.001). The D:A:D models also more accurately predicted five-year CVD-risk for key prognostic subgroups.

CONCLUSIONS

An updated, easily recalibrated, global CVD-risk equation tailored to HIV-positive persons was developed using routinely collected CVD risk parameters and incorporating markers on immune function (CD4 lymphocyte count), and exposure to antiretroviral therapies. The estimated CVD risk can be used to quantify risk and to guide preventive care.

A strategy to build and validate a prognostic biomarker model based on RT-qPCR gene expression and clinical covariates

BACKGROUND

Construction and validation of a prognostic model for survival data in the clinical domain is still an active field of research. Nevertheless there is no consensus on how to develop routine prognostic tests based on a combination of RT-qPCR biomarkers and clinical or demographic variables. In particular, the estimation of the model performance requires to properly account for the RT-qPCR experimental design.

RESULTS

We present a strategy to build, select, and validate a prognostic model for survival data based on a combination of RT-qPCR biomarkers and clinical or demographic data and we provide an illustration on a real clinical dataset. First, we compare two cross-validation schemes: a classical outcome-stratified cross-validation scheme and an alternative one that accounts for the RT-qPCR plate design, especially when samples are processed by batches. The latter is intended to limit the performance discrepancies, also called the validation surprise, between the training and the test sets. Second, strategies for model building (covariate selection, functional relationship modeling, and statistical model) as well as performance indicators estimation are presented. Since in practice several prognostic models can exhibit similar performances, complementary criteria for model selection are discussed: the stability of the selected variables, the model optimism, and the impact of the omitted variables on the model performance.

CONCLUSION

On the training dataset, appropriate resampling methods are expected to prevent from any upward biases due to unaccounted technical and biological variability that may arise from the experimental and intrinsic design of the RT-qPCR assay. Moreover, the stability of the selected variables, the model optimism, and the impact of the omitted variables on the model performances are pivotal indicators to select the optimal model to be validated on the test dataset.

Summarising and validating test accuracy results across multiple studies for use in clinical practice

Following a meta-analysis of test accuracy studies, the translation of summary results into clinical practice is potentially problematic. The sensitivity, specificity and positive (PPV) and negative (NPV) predictive values of a test may differ substantially from the average meta-analysis findings, because of heterogeneity. Clinicians thus need more guidance: given the meta-analysis, is a test likely to be useful in new populations, and if so, how should test results inform the probability of existing disease (for a diagnostic test) or future adverse outcome (for a prognostic test)? We propose ways to address this. Firstly, following a meta-analysis, we suggest deriving prediction intervals and probability statements about the potential accuracy of a test in a new population. Secondly, we suggest strategies on how clinicians should derive post-test probabilities (PPV and NPV) in a new population based on existing meta-analysis results and propose a cross-validation approach for examining and comparing their calibration performance. Application is made to two clinical examples. In the first example, the joint probability that both sensitivity and specificity will be >80% in a new population is just 0.19, because of a low sensitivity. However, the summary PPV of 0.97 is high and calibrates well in new populations, with a probability of 0.78 that the true PPV will be at least 0.95. In the second example, post-test probabilities calibrate better when tailored to the prevalence in the new population, with cross-validation revealing a probability of 0.97 that the observed NPV will be within 10% of the predicted NPV.

Imputation of systematically missing predictors in an individual participant data meta-analysis: a generalized approach using MICE

Individual participant data meta-analyses (IPD-MA) are increasingly used for developing and validating multivariable (diagnostic or prognostic) risk prediction models. Unfortunately, some predictors or even outcomes may not have been measured in each study and are thus systematically missing in some individual studies of the IPD-MA. As a consequence, it is no longer possible to evaluate between-study heterogeneity and to estimate study-specific predictor effects, or to include all individual studies, which severely hampers the development and validation of prediction models. Here, we describe a novel approach for imputing systematically missing data and adopt a generalized linear mixed model to allow for between-study heterogeneity. This approach can be viewed as an extension of Resche-Rigon's method (Stat Med 2013), relaxing their assumptions regarding variance components and allowing imputation of linear and nonlinear predictors. We illustrate our approach using a case study with IPD-MA of 13 studies to develop and validate a diagnostic prediction model for the presence of deep venous thrombosis. We compare the results after applying four methods for dealing with systematically missing predictors in one or more individual studies: complete case analysis where studies with systematically missing predictors are removed, traditional multiple imputation ignoring heterogeneity across studies, stratified multiple imputation accounting for heterogeneity in predictor prevalence, and multilevel multiple imputation (MLMI) fully accounting for between-study heterogeneity. We conclude that MLMI may substantially improve the estimation of between-study heterogeneity parameters and allow for imputation of systematically missing predictors in IPD-MA aimed at the development and validation of prediction models.

Prognostic survival model for people diagnosed with invasive cutaneous melanoma

BACKGROUND

The ability of medical practitioners to communicate risk estimates effectively to patients diagnosed with melanoma relies on accurate information about prognostic factors and their impact on survival. This study reports the development of one of the few melanoma prognostic models, called the Melanoma Severity Index (MSI), based on population-based cancer registry data.

METHODS

Data from the Queensland Cancer Registry for people (20-89 years) diagnosed with a single invasive melanoma between 1995 and 2008 (n = 28,654; 1,700 melanoma deaths). Additional clinical information about metastasis, ulceration and positive lymph nodes was manually extracted from pathology forms. Flexible parametric survival models were combined with multivariable fractional polynomial for selecting variables and transformations of continuous variables. Multiple imputation was used for missing covariate values.

RESULTS

The MSI contained the variables thickness (transformed, explained 40.6% of variation in survival), body site (additional 1.9% in variation), metastasis (1.8%), positive nodes (0.7%), ulceration (1.3%), age (1.1%). Royston and Sauerbrei's D statistic (measure of discrimination) was 1.50 (95% CI = 1.44, 1.56) and the corresponding RD2 (measure of explained variation) was 0.47 (0.45, 0.49), demonstrating strong explanatory performance. The Harrell-C statistic was 0.88 (0.88, 0.89). Lacking an external validation dataset, we applied internal-external cross validation to demonstrate the consistency of the prognostic information across geographically-defined subsets of the cohort.

CONCLUSIONS

The MSI provides good ability to predict survival for melanoma patients. Beyond the immediate clinical use, the MSI may have important public health and research applications for evaluations of public health interventions aimed at reducing deaths from melanoma.

Prognosis of children with HIV-1 infection starting antiretroviral therapy in Southern Africa: a collaborative analysis of treatment programs

BACKGROUND

Prognostic models for children starting antiretroviral therapy (ART) in Africa are lacking. We developed models to estimate the probability of death during the first year receiving ART in Southern Africa.

METHODS

We analyzed data from children ≤10 years of age who started ART in Malawi, South Africa, Zambia or Zimbabwe from 2004 to 2010. Children lost to follow up or transferred were excluded. The primary outcome was all-cause mortality in the first year of ART. We used Weibull survival models to construct 2 prognostic models: 1 with CD4%, age, World Health Organization clinical stage, weight-for-age z-score (WAZ) and anemia and the other without CD4%, because it is not routinely measured in many programs. We used multiple imputation to account for missing data.

RESULTS

Among 12,655 children, 877 (6.9%) died in the first year of ART. We excluded 1780 children who were lost to follow up/transferred from main analyses; 10,875 children were therefore included. With the CD4% model probability of death at 1 year ranged from 1.8% [95% confidence interval (CI): 1.5-2.3] in children 5-10 years with CD4% ≥10%, World Health Organization stage I/II, WAZ ≥ -2 and without severe anemia to 46.3% (95% CI: 38.2-55.2) in children <1 year with CD4% < 5%, stage III/IV, WAZ< -3 and severe anemia. The corresponding range for the model without CD4% was 2.2% (95% CI: 1.8-2.7) to 33.4% (95% CI: 28.2-39.3). Agreement between predicted and observed mortality was good (C-statistics = 0.753 and 0.745 for models with and without CD4%, respectively).

CONCLUSIONS

These models may be useful to counsel children/caregivers, for program planning and to assess program outcomes after allowing for differences in patient disease severity characteristics.

Assessing risk prediction models using individual participant data from multiple studies

Individual participant time-to-event data from multiple prospective epidemiologic studies enable detailed investigation into the predictive ability of risk models. Here we address the challenges in appropriately combining such information across studies. Methods are exemplified by analyses of log C-reactive protein and conventional risk factors for coronary heart disease in the Emerging Risk Factors Collaboration, a collation of individual data from multiple prospective studies with an average follow-up duration of 9.8 years (dates varied). We derive risk prediction models using Cox proportional hazards regression analysis stratified by study and obtain estimates of risk discrimination, Harrell's concordance index, and Royston's discrimination measure within each study; we then combine the estimates across studies using a weighted meta-analysis. Various weighting approaches are compared and lead us to recommend using the number of events in each study. We also discuss the calculation of measures of reclassification for multiple studies. We further show that comparison of differences in predictive ability across subgroups should be based only on within-study information and that combining measures of risk discrimination from case-control studies and prospective studies is problematic. The concordance index and discrimination measure gave qualitatively similar results throughout. While the concordance index was very heterogeneous between studies, principally because of differing age ranges, the increments in the concordance index from adding log C-reactive protein to conventional risk factors were more homogeneous.

Developing and validating risk prediction models in an individual participant data meta-analysis

Background

Risk prediction models estimate the risk of developing future outcomes for individuals based on one or more underlying characteristics (predictors). We review how researchers develop and validate risk prediction models within an individual participant data (IPD) meta-analysis, in order to assess the feasibility and conduct of the approach.

Methods

A qualitative review of the aims, methodology, and reporting in 15 articles that developed a risk prediction model using IPD from multiple studies.

Results

The IPD approach offers many opportunities but methodological challenges exist, including: unavailability of requested IPD, missing patient data and predictors, and between-study heterogeneity in methods of measurement, outcome definitions and predictor effects. Most articles develop their model using IPD from all available studies and perform only an internal validation (on the same set of data). Ten of the 15 articles did not allow for any study differences in baseline risk (intercepts), potentially limiting their model’s applicability and performance in some populations. Only two articles used external validation (on different data), including a novel method which develops the model on all but one of the IPD studies, tests performance in the excluded study, and repeats by rotating the omitted study.

Conclusions

An IPD meta-analysis offers unique opportunities for risk prediction research. Researchers can make more of this by allowing separate model intercept terms for each study (population) to improve generalisability, and by using ‘internal-external cross-validation’ to simultaneously develop and validate their model. Methodological challenges can be reduced by prospectively planned collaborations that share IPD for risk prediction.

Widening clinical applications of the SYNTAX Score

The SYNTAX Score (http://www.syntaxscore.com) has established itself as an anatomical based tool for objectively determining the complexity of coronary artery disease and guiding decision-making between coronary artery bypass graft (CABG) surgery and percutaneous coronary intervention (PCI). Since the landmark SYNTAX (Synergy between PCI with Taxus and Cardiac Surgery) Trial comparing CABG with PCI in patients with complex coronary artery disease (unprotected left main or de novo three vessel disease), numerous validation studies have confirmed the clinical validity of the SYNTAX Score for identifying higher-risk subjects and aiding decision-making between CABG and PCI in a broad range of patient types. The SYNTAX Score is now advocated in both the European and US revascularisation guidelines for decision-making between CABG and PCI as part of a SYNTAX-pioneered heart team approach. Since establishment of the SYNTAX Score, widening clinical applications of this clinical tool have emerged. The purpose of this review is to systematically examine the widening applications of tools based on the SYNTAX Score: (1) by improving the diagnostic accuracy of the SYNTAX Score by adding a functional assessment of lesions; (2) through amalgamation of the anatomical SYNTAX Score with clinical variables to enhance decision-making between CABG and PCI, culminating in the development and validation of the SYNTAX Score II, in which objective and tailored decisions can be made for the individual patient; (3) through assessment of completeness of revascularisation using the residual and post-CABG SYNTAX Scores for PCI and CABG patients, respectively. Finally, the future direction of the SYNTAX Score is covered through discussion of the ongoing development of a non-invasive, functional SYNTAX Score and review of current and planned clinical trials.

Protocol for a systematic review of the diagnostic and prognostic utility of tests currently available for the detection of aspirin resistance in patients with established cardiovascular or cerebrovascular disease

BACKGROUND

The benefits of aspirin as an anti-platelet agent are well established; however, there has been much debate about the lack of uniformity in the efficacy of aspirin to inhibit platelet function. In some patients, aspirin fails to inhibit platelets even where compliance has been verified, a phenomenon which has been termed "aspirin resistance". These patients may in turn be at a higher risk of future vascular events. The proportion of "resistant" patients identified depends on the type of platelet function test. Therefore, the aim of this systematic review is to determine which, if any, platelet function test has utility in terms of identifying patients with a high risk of vascular events. The review has been registered with PROSPERO (CRD42012002151).

METHODS

Relevant studies will be sought from bibliographic databases. Trials registers will be searched for ongoing studies. Reference lists will be checked and subject experts contacted. There will be no date or language restrictions. Standard reviewing methodology to minimise bias will be employed. Any prospective studies in patients on aspirin therapy and assessing platelet function in relation to relevant clinical outcomes will be included, as will studies reporting prognostic models. Risk of bias assessment will be based on the Quality Assessment of Diagnostic Accuracy Studies guidelines, and suitable criteria for assessing quality of prognostic studies. Data on test accuracy measures, relative risks, odds or hazard ratios will be extracted and meta-analysed, where possible, using a random-effects model to account for between-study heterogeneity. Where appropriate, the causes of heterogeneity will be explored through meta-regression and sub-group or sensitivity analyses. If platelet function testing is demonstrated to have diagnostic/predictive utility in a specific population, the potential for a cost-effectiveness analysis will be considered and, if possible, an economic model constructed. This will be supported by a systematic review of existing economic evaluation studies.

DISCUSSION

The results of the review could indicate if platelet function test(s) could lead to a reliable prediction of the risk of clinically important events in a defined population, and thus support investigations into adjustments to therapy in order to compensate for a predicted poor response to standard aspirin.

A framework for developing, implementing, and evaluating clinical prediction models in an individual participant data meta-analysis

The use of individual participant data (IPD) from multiple studies is an increasingly popular approach when developing a multivariable risk prediction model. Corresponding datasets, however, typically differ in important aspects, such as baseline risk. This has driven the adoption of meta-analytical approaches for appropriately dealing with heterogeneity between study populations. Although these approaches provide an averaged prediction model across all studies, little guidance exists about how to apply or validate this model to new individuals or study populations outside the derivation data. We consider several approaches to develop a multivariable logistic regression model from an IPD meta-analysis (IPD-MA) with potential between-study heterogeneity. We also propose strategies for choosing a valid model intercept for when the model is to be validated or applied to new individuals or study populations. These strategies can be implemented by the IPD-MA developers or future model validators. Finally, we show how model generalizability can be evaluated when external validation data are lacking using internal-external cross-validation and extend our framework to count and time-to-event data. In an empirical evaluation, our results show how stratified estimation allows study-specific model intercepts, which can then inform the intercept to be used when applying the model in practice, even to a population not represented by included studies. In summary, our framework allows the development (through stratified estimation), implementation in new individuals (through focused intercept choice), and evaluation (through internal-external validation) of a single, integrated prediction model from an IPD-MA in order to achieve improved model performance and generalizability.

Predicting the short-term risk of diabetes in HIV-positive patients: the Data Collection on Adverse Events of Anti-HIV Drugs (D:A:D) study

Introduction

HIV-positive patients receiving combination antiretroviral therapy (cART) frequently experience metabolic complications such as dyslipidemia and insulin resistance, as well as lipodystrophy, increasing the risk of cardiovascular disease (CVD) and diabetes mellitus (DM). Rates of DM and other glucose-associated disorders among HIV-positive patients have been reported to range between 2 and 14%, and in an ageing HIV-positive population, the prevalence of DM is expected to continue to increase. This study aims to develop a model to predict the short-term (six-month) risk of DM in HIV-positive populations and to compare the existing models developed in the general population.

Methods

All patients recruited to the Data Collection on Adverse events of Anti-HIV Drugs (D:A:D) study with follow-up data, without prior DM, myocardial infarction or other CVD events and with a complete DM risk factor profile were included. Conventional risk factors identified in the general population as well as key HIV-related factors were assessed using Poisson-regression methods. Expected probabilities of DM events were also determined based on the Framingham Offspring Study DM equation. The D:A:D and Framingham equations were then assessed using an internal-external validation process; area under the receiver operating characteristic (AUROC) curve and predicted DM events were determined.

Results

Of 33,308 patients, 16,632 (50%) patients were included, with 376 cases of new onset DM during 89,469 person-years (PY). Factors predictive of DM included higher glucose, body mass index (BMI) and triglyceride levels, and older age. Among HIV-related factors, recent CD4 counts of<200 cells/µL and lipodystrophy were predictive of new onset DM. The mean performance of the D:A:D and Framingham equations yielded AUROC of 0.894 (95% CI: 0.849, 0.940) and 0.877 (95% CI: 0.823, 0.932), respectively. The Framingham equation over-predicted DM events compared to D:A:D for lower glucose and lower triglycerides, and for BMI levels below 25 kg/m².

Conclusions

The D:A:D equation performed well in predicting the short-term onset of DM in the validation dataset and for specific subgroups provided better estimates of DM risk than the Framingham.

Mobility has a non-linear association with falls risk among people in residential aged care: an observational study

QUESTIONS

What is the association between mobility and falls risk for people living in residential aged care? Can the Physical Mobility Scale discriminate between residents at risk of falling and those not at risk?

DESIGN

Prospective longitudinal observational study.

SETTING

Six residential aged care facilities in Australia.

PARTICIPANTS

Eighty-seven high- and low-level care permanent residents.

OUTCOME MEASURES

The primary outcome measure was the number of falls in the six months after the initial mobility assessment. Mobility of all participants was assessed using the Physical Mobility Scale, which includes nine mobility items assessed on a 0-5 scale yielding a total score out of 45.

RESULTS

During the six-month study period, 131 falls were reported. Residents with mild mobility impairment (Physical Mobility Scale total score 28-36) had the highest fall risk (hazard ratio = 1.98, 95% CI 1.30 to 3.03). Residents with fully dependent mobility (Physical Mobility Scale total score 0-9) had the lowest risk for falls (HR=0.05, 95% CI 0.01 to 0.32).

CONCLUSION

Aged care residents with mild mobility impairment are at increased risk of falls and are an appropriate target for falls prevention strategies. Although improving the mobility of residents with moderate to severe mobility impairment may enhance their independence and reduce their burden on staff, paradoxically this may also increase their risk of falls. When these residents improve enough to progress into a higher category of mobility, physiotherapists should be aware that this may increase the risk of falls and should consider instituting appropriate falls prevention strategies.

Predicting infectious complications in neutropenic children and young people with cancer (IPD protocol)

BACKGROUND

A common and potentially life-threatening complication of the treatment of childhood cancer is infection, which frequently presents as fever with neutropenia. The standard management of such episodes is the extensive use of intravenous antibiotics, and though it produces excellent survival rates of over 95%, it greatly inconveniences the three-fourths of patients who do not require such aggressive treatment. There have been a number of studies which have aimed to develop risk prediction models to stratify treatment. Individual participant data (IPD) meta-analysis in therapeutic studies has been developed to improve the precision and reliability of answers to questions of treatment effect and recently have been suggested to be used to answer questions regarding prognosis and diagnosis to gain greater power from the frequently small individual studies.

DESIGN

In the IPD protocol, we will collect and synthesise IPD from multiple studies and examine the outcomes of episodes of febrile neutropenia as a consequence of their treatment for malignant disease. We will develop and evaluate a risk stratification model using hierarchical regression models to stratify patients by their risk of experiencing adverse outcomes during an episode. We will also explore specific practical and methodological issues regarding adaptation of established techniques of IPD meta-analysis of interventions for use in synthesising evidence derived from IPD from multiple studies for use in predictive modelling contexts.

DISCUSSION

Our aim in using this model is to define a group of individuals at low risk for febrile neutropenia who might be treated with reduced intensity or duration of antibiotic therapy and so reduce the inconvenience and cost of these episodes, as well as to define a group of patients at very high risk of complications who could be subject to more intensive therapies. The project will also help develop methods of IPD predictive modelling for use in future studies of risk prediction.

Predicting the risk of cardiovascular disease in HIV-infected patients: the data collection on adverse effects of anti-HIV drugs study

AIMS

HIV-infected patients receiving combination antiretroviral therapy may experience metabolic complications, potentially increasing their risk of cardiovascular diseases (CVDs). Furthermore, exposures to some antiretroviral drugs seem to be independently associated with increased CVD risk. We aimed to develop cardiovascular risk-assessment models tailored to HIV-infected patients.

METHODS AND RESULTS

Prospective multinational cohort study. The data set included 22,625 HIV-infected patients from 20 countries in Europe and Australia who were free of CVD at entry into the Data collection on Adverse Effects of Anti-HIV Drugs Study. Using cross-validation methods, separate models were developed to predict the risk of myocardial infarction, coronary heart disease, and a composite CVD endpoint. Model performance was compared with the Framingham score. The models included age, sex, systolic blood pressure, smoking status, family history of CVD, diabetes, total cholesterol, HDL cholesterol and indinavir, lopinavir/r and abacavir exposure. The models performed well with area under the receiver operator curve statistics of 0.783 (range 0.642-0.820) for myocardial infarction, 0.776 (0.670-0.818) for coronary heart disease and 0.769 (0.695-0.824) for CVD. The models estimated more accurately the outcomes in the subgroups than the Framingham score.

CONCLUSION

Risk equations developed from a population of HIV-infected patients, incorporating routinely collected cardiovascular risk parameters and exposure to individual antiretroviral therapy drugs, might be more useful in estimating CVD risks in HIV-infected persons than conventional risk prediction models.

Prognosis of patients with HIV-1 infection starting antiretroviral therapy in sub-Saharan Africa: a collaborative analysis of scale-up programmes

BACKGROUND

Prognostic models have been developed for patients infected with HIV-1 who start combination antiretroviral therapy (ART) in high-income countries, but not for patients in sub-Saharan Africa. We developed two prognostic models to estimate the probability of death in patients starting ART in sub-Saharan Africa.

METHODS

We analysed data for adult patients who started ART in four scale-up programmes in Côte d'Ivoire, South Africa, and Malawi from 2004 to 2007. Patients lost to follow-up in the first year were excluded. We used Weibull survival models to construct two prognostic models: one with CD4 cell count, clinical stage, bodyweight, age, and sex (CD4 count model); and one that replaced CD4 cell count with total lymphocyte count and severity of anaemia (total lymphocyte and haemoglobin model), because CD4 cell count is not routinely measured in many African ART programmes. Death from all causes in the first year of ART was the primary outcome.

FINDINGS

912 (8.2%) of 11 153 patients died in the first year of ART. 822 patients were lost to follow-up and not included in the main analysis; 10 331 patients were analysed. Mortality was strongly associated with high baseline CD4 cell count (>/=200 cells per muL vs <25; adjusted hazard ratio 0.21, 95% CI 0.17-0.27), WHO clinical stage (stages III-IV vs I-II; 3.45, 2.43-4.90), bodyweight (>/=60 kg vs <45 kg; 0.23, 0.18-0.30), and anaemia status (none vs severe: 0.27, 0.20-0.36). Other independent risk factors for mortality were low total lymphocyte count, advanced age, and male sex. Probability of death at 1 year ranged from 0.9% (95% CI 0.6-1.4) to 52.5% (43.8-61.7) with the CD4 model, and from 0.9% (0.5-1.4) to 59.6% (48.2-71.4) with the total lymphocyte and haemoglobin model. Both models accurately predict early mortality in patients starting ART in sub-Saharan Africa compared with observed data.

INTERPRETATION

Prognostic models should be used to counsel patients, plan health services, and predict outcomes for patients with HIV-1 infection in sub-Saharan Africa.

FUNDING

US National Institute of Allergy And Infectious Diseases, Eunice Kennedy Shriver National Institute of Child Health and Human Development, and National Cancer Institute.

Robust prediction of t-year survival with data from multiple studies

Recently meta-analysis has been widely utilized to combine information across multiple studies to evaluate a common effect. Integrating data from similar studies is particularly useful in genomic studies where the individual study sample sizes are not large relative to the number of parameters of interest. In this article, we are interested in developing robust prognostic rules for the prediction of t-year survival based on multiple studies. We propose to construct a composite score for prediction by fitting a stratified semiparametric transformation model that allows the studies to have related but not identical outcomes. To evaluate the accuracy of the resulting score, we provide point and interval estimators for the commonly used accuracy measures including the time-specific receiver operating characteristic curves, and positive and negative predictive values. We apply the proposed procedures to develop prognostic rules for the 5-year survival of breast cancer patients based on five breast cancer genomic studies.

Validation of prognostic indices using the frailty model

A major issue when proposing a new prognostic index is its generalisibility to daily clinical practice. Validation is therefore required. Most validation techniques assess whether "on average" the results obtained by the prognostic index in classifying patients in a new sample of patients are similar to the results obtained in the construction set. We introduce a new important aspect of the generalisibility of a prognostic index: the heterogeneity of the prognostic index risk group hazard ratios over different centers. If substantial variability between centers exists, the prognostic index may have no discriminatory capability in some of the centers. To model such heterogeneity, we use a frailty model including a random center effect and a random prognostic index by center interaction. Statistical inference is based on a Bayesian approach using a Laplacian approximation for the marginal posterior distribution of the variances of the random effects. We investigate different ways to summarize the information available from this marginal posterior distribution. Our approach is applied to a real bladder cancer database for which we demonstrate how to investigate and interpret heterogeneity in prognostic index effect over centers.

Utility of the Seattle Heart Failure Model in patients with advanced heart failure

OBJECTIVES

The aim of this study was to validate the Seattle Heart Failure Model (SHFM) in patients with advanced heart failure (HF).

BACKGROUND

The SHFM was developed primarily from clinical trial databases and extrapolated the benefit of interventions from published data.

METHODS

We evaluated the discrimination and calibration of SHFM in 445 advanced HF patients (age 52 +/- 12 years, 68.5% male, 52.4% white, ejection fraction 18 +/- 8%) referred for cardiac transplantation. The primary end point was death (n = 92), urgent transplantation (n = 14), or left ventricular assist device (LVAD) implantation (n = 3); a secondary analysis was performed on mortality alone.

RESULTS

Patients were receiving optimal therapy (angiotensin-II modulation 92.8%, beta-blockers 91.5%, aldosterone antagonists 46.3%), and 71.0% had an implantable device (defibrillator 30.4%, biventricular pacemaker 3.4%, combined 37.3%). During a median follow-up of 21 months, 109 patients (24.5%) had an event. Although discrimination was adequate (c-statistic >0.7), the SHFM overall underestimated absolute risk (observed vs. predicted event rate: 11.0% vs. 9.2%, 21.0% vs. 16.6%, and 27.9% vs. 22.8% at 1, 2, and 3 years, respectively). Risk underprediction was more prominent in patients with an implantable device. The SHFM had different calibration properties in white versus black patients, leading to net underestimation of absolute risk in blacks. Race-specific recalibration improved the accuracy of predictions. When analysis was restricted to mortality, the SHFM exhibited better performance.

CONCLUSIONS

In patients with advanced HF, the SHFM offers adequate discrimination, but absolute risk is underestimated, especially in blacks and in patients with devices. This is more prominent when including transplantation and LVAD implantation as an end point.

Prediction of incident hip fracture risk by femur geometry variables measured by hip structural analysis in the study of osteoporotic fractures

The role of bone tissue's geometric distribution in hip fracture risk requires full evaluation in large population-based datasets. We tested whether section modulus, a geometric index of bending strength, predicted hip fracture better than BMD. Among 7474 women from the Study of Osteoporotic Fractures (SOF) with hip DXA scans at baseline, there were 635 incident hip fractures recorded over 13 yr. Hip structural analysis software was used to derive variables from the DXA scans at the narrow neck (NN), intertrochanter (IT), and shaft (S) regions. Associations of derived structural variables with hip fracture were assessed using Cox proportional hazard modeling. Hip fracture prediction was assessed using the C-index concordance statistic. Incident hip fracture cases had larger neck-shaft angles, larger subperiosteal and estimated endosteal diameters, greater distances from lateral cortical margin to center of mass (lateral distance), and higher estimated buckling ratios (p < 0.0001 for each). Areal BMD, cross-sectional area, cross-sectional moment of inertia, section modulus, estimated cortical thickness, and centroid position were all lower in hip fracture cases (p < 0.044). In hip fracture prediction using NN region parameters, estimated cortical thickness, areal BMD, and estimated buckling ratio were equivalent (C-index = 0.72; 95% CI, 0.70, 0.74), but section modulus performed less well (C-index = 0.61; 95% CI, 0.58, 0.63; p < 0.0001 for difference). In multivariable models combining hip structural analysis variables and age, effects of bone dimensions (i.e., lateral distance, subperiosteal diameter, and estimated endosteal width) were interchangeable, whereas age and neck-shaft angle were independent predictors. Several parsimonious multivariable models that were prognostically equivalent for the NN region were obtained combining a measure of width, a measure of mass, age, and neck-shaft angle (BMD is a ratio of mass to width in the NN region; C-index = 0.77; 95% CI, 0.75, 0.79). Trochanteric fractures were best predicted by analysis of the IT region. Because section modulus failed to predict hip fracture risk as well as areal BMD, the thinner cortices and wider bones among those who fractured may imply that simple failure in bending is not the usual event in fracture. Fracture might require initiation (e.g., by localized crushing or buckling of the lateral cortex).

Potential demographic and baselines variables for risk stratification of high-risk post-myocardial infarction patients in the era of implantable cardioverter-defibrillator--a prognostic indicator

BACKGROUND

Risk stratification after myocardial infarction (MI) remains expensive and disappointing. We designed a prognostic indicator using demographic information to select patients at risk of dying after MI.

METHOD AND RESULTS

We combined individual patient data from the placebo arms of EMIAT, CAMIAT, TRACE and DIAMOND-MI with LVEF <or=40% or ventricular arrhythmias (i.e. >10 ventricular premature beats/hour or a run of ventricular tachycardia). Risk factors for mortality beginning at day 45 post-MI up to 2 years were examined using Cox regression analysis. Risk scores were derived from the equation of a Cox regression model containing only significant variables. The prognostic index was the sum of the individual contribution from the risk factors. 2707 patients were pooled (age: 66 (23-92) years, 78.8% M) with 480 deaths at 2-years (44% arrhythmic and 35.6% non-arrhythmic cardiac deaths). Variables predicting mortality were age, sex, previous MI or angina, hypertension, diabetes, systolic blood pressure, heart rate, NYHA functional class and non-Q wave infarct on electrocardiogram. Distinct survival curves were obtained for 3 risk groups based on the median and inter-quartile range for the prognostic index. In the high-risk group, up to 40% of patients died (all-cause mortality), 19.1% died of arrhythmic and 18.2% died of non-arrhythmic cardiac causes at 2-years.

CONCLUSION

In post-MI patients with LVEF <or=40% or frequent ventricular premature beats, the additional use of a simple prognostic indicator based on demographic information was able to provide clinically meaningful risk stratification on patients that were at high risk of dying and may be used to identify patients for prophylactic implantable cardioverter-defibrillator therapy.

A consensus prognostic gene expression classifier for ER positive breast cancer

BACKGROUND

A consensus prognostic gene expression classifier is still elusive in heterogeneous diseases such as breast cancer.

RESULTS

Here we perform a combined analysis of three major breast cancer microarray data sets to hone in on a universally valid prognostic molecular classifier in estrogen receptor (ER) positive tumors. Using a recently developed robust measure of prognostic separation, we further validate the prognostic classifier in three external independent cohorts, confirming the validity of our molecular classifier in a total of 877 ER positive samples. Furthermore, we find that molecular classifiers may not outperform classical prognostic indices but that they can be used in hybrid molecular-pathological classification schemes to improve prognostic separation.

CONCLUSION

The prognostic molecular classifier presented here is the first to be valid in over 877 ER positive breast cancer samples and across three different microarray platforms. Larger multi-institutional studies will be needed to fully determine the added prognostic value of molecular classifiers when combined with standard prognostic factors.