Construction and validation of a prognostic model across several studies, with an application in superficial bladder cancer

Musculoskeletal Health and Work: Development and Internal-External Cross-Validation of a Model to Predict Risk of Work Absence and Presenteeism in People Seeking Primary Healthcare

PURPOSE

To develop and validate prediction models for the risk of future work absence and level of presenteeism, in adults seeking primary healthcare with musculoskeletal disorders (MSD).

METHODS

Six studies from the West-Midlands/Northwest regions of England, recruiting adults consulting primary care with MSD were included for model development and internal-external cross-validation (IECV). The primary outcome was any work absence within 6 months of their consultation. Secondary outcomes included 6-month presenteeism and 12-month work absence. Ten candidate predictors were included: age; sex; multisite pain; baseline pain score; pain duration; job type; anxiety/depression; comorbidities; absence in the previous 6 months; and baseline presenteeism.

RESULTS

For the 6-month absence model, 2179 participants (215 absences) were available across five studies. Calibration was promising, although varied across studies, with a pooled calibration slope of 0.93 (95% CI: 0.41-1.46) on IECV. On average, the model discriminated well between those with work absence within 6 months, and those without (IECV-pooled C-statistic 0.76, 95% CI: 0.66-0.86). The 6-month presenteeism model, while well calibrated on average, showed some individual-level variation in predictive accuracy, and the 12-month absence model was poorly calibrated due to the small available size for model development.

CONCLUSIONS

The developed models predict 6-month work absence and presenteeism with reasonable accuracy, on average, in adults consulting with MSD. The model to predict 12-month absence was poorly calibrated and is not yet ready for use in practice. This information may support shared decision-making and targeting occupational health interventions at those with a higher risk of absence or presenteeism in the 6 months following consultation. Further external validation is needed before the models' use can be recommended or their impact on patients can be fully assessed.

The potential benefit of statin prescription based on prediction of treatment responsiveness in older individuals: an application to the PROSPER randomized controlled trial

AIMS

Clinical guidelines often recommend treating individuals based on their cardiovascular risk. We revisit this paradigm and quantify the efficacy of three treatment strategies: (i) overall prescription, i.e. treatment to all individuals sharing the eligibility criteria of a trial; (ii) risk-stratified prescription, i.e. treatment only to those at an elevated outcome risk; and (iii) prescription based on predicted treatment responsiveness.

METHODS AND RESULTS

We reanalysed the PROSPER randomized controlled trial, which included individuals aged 70-82 years with a history of, or risk factors for, vascular diseases. We conducted the derivation and internal-external validation of a model predicting treatment responsiveness. We compared with placebo (n = 2913): (i) pravastatin (n = 2891); (ii) pravastatin in the presence of previous vascular diseases and placebo in the absence thereof (n = 2925); and (iii) pravastatin in the presence of a favourable prediction of treatment response and placebo in the absence thereof (n = 2890). We found an absolute difference in primary outcome events composed of coronary death, non-fatal myocardial infarction, and fatal or non-fatal stroke, per 10 000 person-years equal to: -78 events (95% CI, -144 to -12) when prescribing pravastatin to all participants; -66 events (95% CI, -114 to -18) when treating only individuals with an elevated vascular risk; and -103 events (95% CI, -162 to -44) when restricting pravastatin to individuals with a favourable prediction of treatment response.

CONCLUSION

Pravastatin prescription based on predicted responsiveness may have an encouraging potential for cardiovascular prevention. Further external validation of our results and clinical experiments are needed.

TRIAL REGISTRATION

ISRCTN40976937.

Estimating individualized treatment effects using an individual participant data meta-analysis

BACKGROUND

One key aspect of personalized medicine is to identify individuals who benefit from an intervention. Some approaches have been developed to estimate individualized treatment effects (ITE) with a single randomized control trial (RCT) or observational data, but they are often underpowered for the ITE estimation. Using individual participant data meta-analyses (IPD-MA) might solve this problem. Few studies have investigated how to develop risk prediction models with IPD-MA, and it remains unclear how to combine those methods with approaches used for ITE estimation. In this article, we compared different approaches using both simulated and real data with binary and time-to-event outcomes to estimate the individualized treatment effects from an IPD-MA in a one-stage approach.

METHODS

We compared five one-stage models: naive model (NA), random intercept (RI), stratified intercept (SI), rank-1 (R1), and fully stratified (FS), built with two different strategies, the S-learner and the T-learner constructed with a Monte Carlo simulation study in which we explored different scenarios with a binary or a time-to-event outcome. To evaluate the performance of the models, we used the c-statistic for benefit, the calibration of predictions, and the mean squared error. The different models were also used on the INDANA IPD-MA, comparing an anti-hypertensive treatment to no treatment or placebo ( N = 40 237 , 836 events).

RESULTS

Simulation results showed that using the S-learner led to better ITE estimation performances for both binary and time-to-event outcomes. None of the risk models stand out and had significantly better results. For the INDANA dataset with a binary outcome, the naive and the random intercept models had the best performances.

CONCLUSIONS

For the choice of the strategy, using interactions with treatment (the S-learner) is preferable. For the choice of the method, no approach is better than the other.

Characteristics of Medical Deserts and Approaches to Mitigate Their Health Workforce Issues: A Scoping Review of Empirical Studies in Western Countries

BACKGROUND

Medical deserts are considered a problematic issue for many Western countries which try to employ multitude of policies and initiatives to achieve a better distribution of their health workforce (HWF). The aim of this study was to systematically map research and provide an overview of definitions, characteristics, contributing factors and approaches to mitigate medical deserts within the European Union (EU)-funded project "ROUTE-HWF" (a Roadmap OUT of mEdical deserts into supportive Health WorkForce initiatives and policies).

METHODS

We performed a scoping review to identify knowledge clusters/research gaps in the field of medical deserts focusing on HWF issues. Six databases were searched till June 2021. Studies reporting primary research from Western countries on definitions, characteristics, contributing factors, and approaches were included. Two independent reviewers assessed studies for eligibility, extracted data and clustered studies according to the four defined outcomes.

RESULTS

Two-hundred and forty studies were included (n=116, 48% Australia/New Zealand; n=105, 44% North America; n=20, 8% Europe). All used observational designs except for five quasi-experimental studies. Studies provided definitions (n=171, 71%), characteristics (n=95, 40%), contributing factors (n=112, 47%), and approaches to mitigate medical deserts (n=87, 36%). Most medical deserts were defined by the density of the population in an area. Contributing factors to HWF issues in medical deserts consisted in work-related (n=55, 23%) and lifestyle-related factors (n=33, 14%) of the HWF as well as sociodemographic characteristics (n=79, 33%). Approaches to mitigate them focused on training adapted to the scope of rural practice (n=67, 28%), HWF distribution (n=3, 1%), support/infrastructure (n=8, 3%) and innovative models of care (n=7, 3%).

CONCLUSION

Our study provides the first scoping review that presents and categorizes definitions, characteristics, contributing factors, and approaches to mitigate HWF issues in medical deserts. We identified gaps such as the scarcity of longitudinal studies to investigate the impact of factors contributing to medical deserts, and interventional studies to evaluate the effectiveness of approaches to mitigate HWF issues.

Assessing Performance and Clinical Usefulness in Prediction Models With Survival Outcomes: Practical Guidance for Cox Proportional Hazards Models

Risk prediction models need thorough validation to assess their performance. Validation of models for survival outcomes poses challenges due to the censoring of observations and the varying time horizon at which predictions can be made. This article describes measures to evaluate predictions and the potential improvement in decision making from survival models based on Cox proportional hazards regression.

As a motivating case study, the authors consider the prediction of the composite outcome of recurrence or death (the "event") in patients with breast cancer after surgery. They developed a simple Cox regression model with 3 predictors, as in the Nottingham Prognostic Index, in 2982 women (1275 events over 5 years of follow-up) and externally validated this model in 686 women (285 events over 5 years). Improvement in performance was assessed after the addition of progesterone receptor as a prognostic biomarker.

The model predictions can be evaluated across the full range of observed follow-up times or for the event occurring by the end of a fixed time horizon of interest. The authors first discuss recommended statistical measures that evaluate model performance in terms of discrimination, calibration, or overall performance. Further, they evaluate the potential clinical utility of the model to support clinical decision making according to a net benefit measure. They provide SAS and R code to illustrate internal and external validation.

The authors recommend the proposed set of performance measures for transparent reporting of the validity of predictions from survival models.

ESRRG, ATP4A, and ATP4B as Diagnostic Biomarkers for Gastric Cancer: A Bioinformatic Analysis Based on Machine Learning

Based on multiple bioinformatics methods and machine learning techniques, this study was designed to explore potential hub genes of gastric cancer with a diagnostic value. The novel biomarkers were detected through multiple databases of gastric cancer–related genes. The NCBI Gene Expression Omnibus (GEO) database was used to obtain gene expression files. Three hub genes (ESRRG, ATP4A, and ATP4B) were detected through a combination of weighted gene co-expression network analysis (WGCNA), gene–gene interaction network analysis, and supervised feature selection method. GEPIA2 was used to verify the differences in the expression levels of the hub genes in normal and cancer tissues in the RNA-seq levels of Genotype-Tissue Expression (GTEx) and The Cancer Genome Atlas (TCGA) databases. The objectivity of potential hub genes was also verified by immunohistochemistry in the Human Protein Atlas (HPA) database and transcription factor–hub gene regulatory network. Machine learning (ML) methods including data pre-processing, model selection and cross-validation, and performance evaluation were examined on the hub-gene expression profiles in five Gene Expression Omnibus datasets and verified on a GEO external validation (EV) dataset. Six supervised learning models (support vector machine, random forest, k-nearest neighbors, neural network, decision tree, and eXtreme Gradient Boosting) and one semi-supervised learning model (label spreading) were established to evaluate the diagnostic value of biomarkers. Among the six supervised models, the support vector machine (SVM) algorithm was the most effective one according to calculated performance metrics, including 0.93 and 0.99 area under the curve (AUC) scores on the test and external validation datasets, respectively. Furthermore, the semi-supervised model could also successfully learn and predict sample types, achieving a 0.986 AUC score on the EV dataset, even when 10% samples in the five GEO datasets were labeled. In conclusion, three hub genes (ATP4A, ATP4B, and ESRRG) closely related to gastric cancer were mined, based on which the ML diagnostic model of gastric cancer was conducted.

Development of a Model to Predict 10-Year Risk of Ischemic and Hemorrhagic Stroke and Ischemic Heart Disease Using the China Kadoorie Biobank

BACKGROUND AND OBJECTIVES

Contemporary cardiovascular disease (CVD) risk prediction models are rarely applied in routine clinical practice in China due to substantial regional differences in absolute risks of major CVD types within China. Moreover, the inclusion of blood lipids in most risk prediction models also limits their use in the Chinese population. We developed 10-year CVD risk prediction models excluding blood lipids that may be applicable to diverse regions of China.

METHODS

We derived sex-specific models separately for ischemic heart disease (IHD), ischemic stroke (IS), and hemorrhagic stroke (HS) in addition to total CVD in the China Kadoorie Biobank. Participants were age 30-79 years without CVD at baseline. Predictors included age, systolic and diastolic blood pressure, use of blood pressure-lowering treatment, current daily smoking, diabetes, and waist circumference. Total CVD risks were combined in terms of conditional probability using the predicted risks of 3 submodels. Risk models were recalibrated in each region by 2 methods (practical and ideal) and risk prediction was estimated before and after recalibration.

RESULTS

Model derivation involved 489,596 individuals, including 45,947 IHD, 43,647 IS, and 11,168 HS cases during 11 years of follow-up. In women, the Harrell C was 0.732 (95% CI 0.706-0.758), 0.759 (0.738-0.779), and 0.803 (0.778-0.827) for IHD, IS, and HS, respectively. The Harrell C for total CVD was 0.734 (0.732-0.736), 0.754 (0.752-0.756), and 0.774 (0.772-0.776) for models before recalibration, after practical recalibration, and after ideal recalibration. The calibration performances improved after recalibration, with models after ideal recalibration showing the best model performances. The results for men were comparable to those for women.

DISCUSSION

Our CVD risk prediction models yielded good discrimination of IHD and stroke subtypes in addition to total CVD without including blood lipids. Flexible recalibration of our models for different regions could enable more widespread use using resident health records covering the overall Chinese population.

CLASSIFICATION OF EVIDENCE

This study provides Class I evidence that a prediction model incorporating accessible clinical variables predicts 10-year risk of IHD, IS, and HS in the Chinese population age 30-79 years.

Personalized Model to Predict Keratoconus Progression From Demographic, Topographic, and Genetic Data

PURPOSE

To generate a prognostic model to predict keratoconus progression to corneal crosslinking (CXL).

DESIGN

Retrospective cohort study.

METHODS

We recruited 5025 patients (9341 eyes) with early keratoconus between January 2011 and November 2020. Genetic data from 926 patients were available. We investigated both keratometry or CXL as end points for progression and used the Royston-Parmar method on the proportional hazards scale to generate a prognostic model. We calculated hazard ratios (HRs) for each significant covariate, with explained variation and discrimination, and performed internal-external cross validation by geographic regions.

RESULTS

After exclusions, model fitting comprised 8701 eyes, of which 3232 underwent CXL. For early keratoconus, CXL provided a more robust prognostic model than keratometric progression. The final model explained 33% of the variation in time to event: age HR (95% CI) 0.9 (0.90-0.91), maximum anterior keratometry 1.08 (1.07-1.09), and minimum corneal thickness 0.95 (0.93-0.96) as significant covariates. Single-nucleotide polymorphisms (SNPs) associated with keratoconus (n=28) did not significantly contribute to the model. The predicted time-to-event curves closely followed the observed curves during internal-external validation. Differences in discrimination between geographic regions was low, suggesting the model maintained its predictive ability.

CONCLUSIONS

A prognostic model to predict keratoconus progression could aid patient empowerment, triage, and service provision. Age at presentation is the most significant predictor of progression risk. Candidate SNPs associated with keratoconus do not contribute to progression risk.

Individual participant data meta-analysis with mixed-effects transformation models

One-stage meta-analysis of individual participant data (IPD) poses several statistical and computational challenges. For time-to-event outcomes, the approach requires the estimation of complicated nonlinear mixed-effects models that are flexible enough to realistically capture the most important characteristics of the IPD. We present a model class that incorporates general normally distributed random effects into linear transformation models. We discuss extensions to model between-study heterogeneity in baseline risks and covariate effects and also relax the assumption of proportional hazards. Within the proposed framework, data with arbitrary random censoring patterns can be handled. The accompanying \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} }{}$\textsf{R}$\end{document} package tramME utilizes the Laplace approximation and automatic differentiation to perform efficient maximum likelihood estimation and inference in mixed-effects transformation models. We compare several variants of our model to predict the survival of patients with chronic obstructive pulmonary disease using a large data set of prognostic studies. Finally, a simulation study is presented that verifies the correctness of the implementation and highlights its efficiency compared to an alternative approach.

Deep Learning for Automated Triaging of 4581 Breast MRI Examinations from the DENSE Trial

Background Supplemental screening with MRI has proved beneficial in women with extremely dense breasts. Most MRI examinations show normal anatomic and physiologic variation that may not require radiologic review. Thus, ways to triage these normal MRI examinations to reduce radiologist workload are needed. Purpose To determine the feasibility of an automated triaging method using deep learning (DL) to dismiss the highest number of MRI examinations without lesions while still identifying malignant disease. Materials and Methods This secondary analysis of data from the Dense Tissue and Early Breast Neoplasm Screening, or DENSE, trial evaluated breast MRI examinations from the first screening round performed in eight hospitals between December 2011 and January 2016. A DL model was developed to differentiate between breasts with lesions and breasts without lesions. The model was trained to dismiss breasts with normal phenotypical variation and to triage lesions (Breast Imaging Reporting and Data System [BI-RADS] categories 2-5) using eightfold internal-external validation. The model was trained on data from seven hospitals and tested on data from the eighth hospital, alternating such that each hospital was used once as an external test set. Performance was assessed using receiver operating characteristic analysis. At 100% sensitivity for malignant disease, the fraction of examinations dismissed from radiologic review was estimated. Results A total of 4581 MRI examinations of extremely dense breasts from 4581women (mean age, 54.3 years; interquartile range, 51.5-59.8 years) were included. Of the 9162 breasts, 838 had at least one lesion (BI-RADS category 2-5, of which 77 were malignant) and 8324 had no lesions. At 100% sensitivity for malignant lesions, the DL model considered 90.7% (95% CI: 86.7, 94.7) of the MRI examinations with lesions to be nonnormal and triaged them to radiologic review. The DL model dismissed 39.7% (95% CI: 30.0, 49.4) of the MRI examinations without lesions. The DL model had an average area under the receiver operating characteristic curve of 0.83 (95% CI: 0.80, 0.85) in the differentiation between normal breast MRI examinations and MRI examinations with lesions. Conclusion Automated analysis of breast MRI examinations in women with dense breasts dismissed nearly 40% of MRI scans without lesions while not missing any cancers. ClinicalTrials.gov: NCT01315015 © RSNA, 2021 Online supplemental material is available for this article. See also the editorial by Joe in this issue.

Validation and development of models using clinical, biochemical and ultrasound markers for predicting pre-eclampsia: an individual participant data meta-analysis

BACKGROUND

Pre-eclampsia is a leading cause of maternal and perinatal mortality and morbidity. Early identification of women at risk is needed to plan management.

OBJECTIVES

To assess the performance of existing pre-eclampsia prediction models and to develop and validate models for pre-eclampsia using individual participant data meta-analysis. We also estimated the prognostic value of individual markers.

DESIGN

This was an individual participant data meta-analysis of cohort studies.

SETTING

Source data from secondary and tertiary care.

PREDICTORS

We identified predictors from systematic reviews, and prioritised for importance in an international survey.

PRIMARY OUTCOMES

Early-onset (delivery at < 34 weeks' gestation), late-onset (delivery at ≥ 34 weeks' gestation) and any-onset pre-eclampsia.

ANALYSIS

We externally validated existing prediction models in UK cohorts and reported their performance in terms of discrimination and calibration. We developed and validated 12 new models based on clinical characteristics, clinical characteristics and biochemical markers, and clinical characteristics and ultrasound markers in the first and second trimesters. We summarised the data set-specific performance of each model using a random-effects meta-analysis. Discrimination was considered promising for C-statistics of ≥ 0.7, and calibration was considered good if the slope was near 1 and calibration-in-the-large was near 0. Heterogeneity was quantified using I ² and τ². A decision curve analysis was undertaken to determine the clinical utility (net benefit) of the models. We reported the unadjusted prognostic value of individual predictors for pre-eclampsia as odds ratios with 95% confidence and prediction intervals.

RESULTS

The International Prediction of Pregnancy Complications network comprised 78 studies (3,570,993 singleton pregnancies) identified from systematic reviews of tests to predict pre-eclampsia. Twenty-four of the 131 published prediction models could be validated in 11 UK cohorts. Summary C-statistics were between 0.6 and 0.7 for most models, and calibration was generally poor owing to large between-study heterogeneity, suggesting model overfitting. The clinical utility of the models varied between showing net harm to showing minimal or no net benefit. The average discrimination for IPPIC models ranged between 0.68 and 0.83. This was highest for the second-trimester clinical characteristics and biochemical markers model to predict early-onset pre-eclampsia, and lowest for the first-trimester clinical characteristics models to predict any pre-eclampsia. Calibration performance was heterogeneous across studies. Net benefit was observed for International Prediction of Pregnancy Complications first and second-trimester clinical characteristics and clinical characteristics and biochemical markers models predicting any pre-eclampsia, when validated in singleton nulliparous women managed in the UK NHS. History of hypertension, parity, smoking, mode of conception, placental growth factor and uterine artery pulsatility index had the strongest unadjusted associations with pre-eclampsia.

LIMITATIONS

Variations in study population characteristics, type of predictors reported, too few events in some validation cohorts and the type of measurements contributed to heterogeneity in performance of the International Prediction of Pregnancy Complications models. Some published models were not validated because model predictors were unavailable in the individual participant data.

CONCLUSION

For models that could be validated, predictive performance was generally poor across data sets. Although the International Prediction of Pregnancy Complications models show good predictive performance on average, and in the singleton nulliparous population, heterogeneity in calibration performance is likely across settings.

FUTURE WORK

Recalibration of model parameters within populations may improve calibration performance. Additional strong predictors need to be identified to improve model performance and consistency. Validation, including examination of calibration heterogeneity, is required for the models we could not validate.

STUDY REGISTRATION

This study is registered as PROSPERO CRD42015029349.

FUNDING

This project was funded by the National Institute for Health Research (NIHR) Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 24, No. 72. See the NIHR Journals Library website for further project information.

Predicting hospital admissions from individual patient data (IPD): an applied example to explore key elements driving external validity

OBJECTIVE

To explore factors that potentially impact external validation performance while developing and validating a prognostic model for hospital admissions (HAs) in complex older general practice patients.

STUDY DESIGN AND SETTING

Using individual participant data from four cluster-randomised trials conducted in the Netherlands and Germany, we used logistic regression to develop a prognostic model to predict all-cause HAs within a 6-month follow-up period. A stratified intercept was used to account for heterogeneity in baseline risk between the studies. The model was validated both internally and by using internal-external cross-validation (IECV).

RESULTS

Prior HAs, physical components of the health-related quality of life comorbidity index, and medication-related variables were used in the final model. While achieving moderate discriminatory performance, internal bootstrap validation revealed a pronounced risk of overfitting. The results of the IECV, in which calibration was highly variable even after accounting for between-study heterogeneity, agreed with this finding. Heterogeneity was equally reflected in differing baseline risk, predictor effects and absolute risk predictions.

CONCLUSIONS

Predictor effect heterogeneity and differing baseline risk can explain the limited external performance of HA prediction models. With such drivers known, model adjustments in external validation settings (eg, intercept recalibration, complete updating) can be applied more purposefully.

TRIAL REGISTRATION NUMBER

PROSPERO id: CRD42018088129.

A new prognostic model for predicting 30-day mortality in acute oncology patients

INTRODUCTION

Acute oncology services (AOS) provide rapid review and expedited pathways for referral to specialist care for cancer patients. Blood tests may support AOS in providing estimates of prognosis. We aimed to develop and validate a prognostic model of 30-day mortality based on routine blood markers to inform an AOS decision to actively treat or palliate patients.

METHODS AND MATERIALS

Using clinical data from 752 AOS referrals, multivariable logistic regression analysis was conducted to develop a 30-day mortality prognostic model. Internal validation and then internal-external cross-validation were used to examine overfitting and generalizability of the model's predictive performance.

RESULTS

Urea, alkaline phosphatase, albumin and neutrophils were the strongest predictors of outcome. The model separated patients into distinct prognostic groups from the cross-validation (C Statistic: 0.70; 95% CI: 0.64-0.76). Admission year was included as a predictor in the model to improve the model calibration.

CONCLUSION

The developed prediction model was able to classify patients into distinct prognostic risk groups, which is clinically useful for delivering an evidence-based AOS. Collation of data from other AOS centers would allow for the development of a more generalizable prognostic model.

The development and validation of a prognostic model to PREDICT Relapse of depression in adult patients in primary care: protocol for the PREDICTR study

BACKGROUND

Most patients who present with depression are treated in primary care by general practitioners (GPs). Relapse of depression is common (at least 50% of patients treated for depression will relapse after a single episode) and leads to considerable morbidity and decreased quality of life for patients. The majority of patients will relapse within 6 months, and those with a history of relapse are more likely to relapse in the future than those with no such history. GPs see a largely undifferentiated case-mix of patients, and once patients with depression reach remission, there is limited guidance to help GPs stratify patients according to risk of relapse. We aim to develop a prognostic model to predict an individual's risk of relapse within 6-8 months of entering remission. The long-term objective is to inform the clinical management of depression after the acute phase.

METHODS

We will develop a prognostic model using secondary analysis of individual participant data drawn from seven RCTs and one longitudinal cohort study in primary or community care settings. We will use logistic regression to predict the outcome of relapse of depression within 6-8 months. We plan to include the following established relapse predictors in the model: residual depressive symptoms, number of previous depressive episodes, co-morbid anxiety and severity of index episode. We will use a "full model" development approach, including all available predictors. Performance statistics (optimism-adjusted C-statistic, calibration-in-the-large, calibration slope) and calibration plots (with smoothed calibration curves) will be calculated. Generalisability of predictive performance will be assessed through internal-external cross-validation. Clinical utility will be explored through net benefit analysis.

DISCUSSION

We will derive a statistical model to predict relapse of depression in remitted depressed patients in primary care. Assuming the model has sufficient predictive performance, we outline the next steps including independent external validation and further assessment of clinical utility and impact.

STUDY REGISTRATION

ClinicalTrials.gov ID: NCT04666662.

Developing more generalizable prediction models from pooled studies and large clustered data sets

Prediction models often yield inaccurate predictions for new individuals. Large data sets from pooled studies or electronic healthcare records may alleviate this with an increased sample size and variability in sample characteristics. However, existing strategies for prediction model development generally do not account for heterogeneity in predictor‐outcome associations between different settings and populations. This limits the generalizability of developed models (even from large, combined, clustered data sets) and necessitates local revisions. We aim to develop methodology for producing prediction models that require less tailoring to different settings and populations. We adopt internal‐external cross‐validation to assess and reduce heterogeneity in models' predictive performance during the development. We propose a predictor selection algorithm that optimizes the (weighted) average performance while minimizing its variability across the hold‐out clusters (or studies). Predictors are added iteratively until the estimated generalizability is optimized. We illustrate this by developing a model for predicting the risk of atrial fibrillation and updating an existing one for diagnosing deep vein thrombosis, using individual participant data from 20 cohorts (N = 10 873) and 11 diagnostic studies (N = 10 014), respectively. Meta‐analysis of calibration and discrimination performance in each hold‐out cluster shows that trade‐offs between average and heterogeneity of performance occurred. Our methodology enables the assessment of heterogeneity of prediction model performance during model development in multiple or clustered data sets, thereby informing researchers on predictor selection to improve the generalizability to different settings and populations, and reduce the need for model tailoring. Our methodology has been implemented in the R package metamisc.

Development of imaging-based risk scores for prediction of intracranial haemorrhage and ischaemic stroke in patients taking antithrombotic therapy after ischaemic stroke or transient ischaemic attack: a pooled analysis of individual patient data from cohort studies

BACKGROUND

Balancing the risks of recurrent ischaemic stroke and intracranial haemorrhage is important for patients treated with antithrombotic therapy after ischaemic stroke or transient ischaemic attack. However, existing predictive models offer insufficient performance, particularly for assessing the risk of intracranial haemorrhage. We aimed to develop new risk scores incorporating clinical variables and cerebral microbleeds, an MRI biomarker of intracranial haemorrhage and ischaemic stroke risk.

METHODS

We did a pooled analysis of individual-patient data from the Microbleeds International Collaborative Network (MICON), which includes 38 hospital-based prospective cohort studies from 18 countries. All studies recruited participants with previous ischaemic stroke or transient ischaemic attack, acquired baseline MRI allowing quantification of cerebral microbleeds, and followed-up participants for ischaemic stroke and intracranial haemorrhage. Participants not taking antithrombotic drugs were excluded. We developed Cox regression models to predict the 5-year risks of intracranial haemorrhage and ischaemic stroke, selecting candidate predictors on biological relevance and simplifying models using backward elimination. We derived integer risk scores for clinical use. We assessed model performance in internal validation, adjusted for optimism using bootstrapping. The study is registered on PROSPERO, CRD42016036602.

FINDINGS

The included studies recruited participants between Aug 28, 2001, and Feb 4, 2018. 15 766 participants had follow-up for intracranial haemorrhage, and 15 784 for ischaemic stroke. Over a median follow-up of 2 years, 184 intracranial haemorrhages and 1048 ischaemic strokes were reported. The risk models we developed included cerebral microbleed burden and simple clinical variables. Optimism-adjusted c indices were 0·73 (95% CI 0·69-0·77) with a calibration slope of 0·94 (0·81-1·06) for the intracranial haemorrhage model and 0·63 (0·62-0·65) with a calibration slope of 0·97 (0·87-1·07) for the ischaemic stroke model. There was good agreement between predicted and observed risk for both models.

INTERPRETATION

The MICON risk scores, incorporating clinical variables and cerebral microbleeds, offer predictive value for the long-term risks of intracranial haemorrhage and ischaemic stroke in patients prescribed antithrombotic therapy for secondary stroke prevention; external validation is warranted.

FUNDING

British Heart Foundation and Stroke Association.

Individual participant data meta-analysis for external validation, recalibration, and updating of a flexible parametric prognostic model

Individual participant data (IPD) from multiple sources allows external validation of a prognostic model across multiple populations. Often this reveals poor calibration, potentially causing poor predictive performance in some populations. However, rather than discarding the model outright, it may be possible to modify the model to improve performance using recalibration techniques. We use IPD meta-analysis to identify the simplest method to achieve good model performance. We examine four options for recalibrating an existing time-to-event model across multiple populations: (i) shifting the baseline hazard by a constant, (ii) re-estimating the shape of the baseline hazard, (iii) adjusting the prognostic index as a whole, and (iv) adjusting individual predictor effects. For each strategy, IPD meta-analysis examines (heterogeneity in) model performance across populations. Additionally, the probability of achieving good performance in a new population can be calculated allowing ranking of recalibration methods. In an applied example, IPD meta-analysis reveals that the existing model had poor calibration in some populations, and large heterogeneity across populations. However, re-estimation of the intercept substantially improved the expected calibration in new populations, and reduced between-population heterogeneity. Comparing recalibration strategies showed that re-estimating both the magnitude and shape of the baseline hazard gave the highest predicted probability of good performance in a new population. In conclusion, IPD meta-analysis allows a prognostic model to be externally validated in multiple settings, and enables recalibration strategies to be compared and ranked to decide on the least aggressive recalibration strategy to achieve acceptable external model performance without discarding existing model information.

A heart failure phenotype stratified model for predicting 1-year mortality in patients admitted with acute heart failure: results from an individual participant data meta-analysis of four prospective European cohorts

BACKGROUND

Prognostic models developed in general cohorts with a mixture of heart failure (HF) phenotypes, though more widely applicable, are also likely to yield larger prediction errors in settings where the HF phenotypes have substantially different baseline mortality rates or different predictor-outcome associations. This study sought to use individual participant data meta-analysis to develop an HF phenotype stratified model for predicting 1-year mortality in patients admitted with acute HF.

METHODS

Four prospective European cohorts were used to develop an HF phenotype stratified model. Cox model with two rounds of backward elimination was used to derive the prognostic index. Weibull model was used to obtain the baseline hazard functions. The internal-external cross-validation (IECV) approach was used to evaluate the generalizability of the developed model in terms of discrimination and calibration.

RESULTS

3577 acute HF patients were included, of which 2368 were classified as having HF with reduced ejection fraction (EF) (HFrEF; EF < 40%), 588 as having HF with midrange EF (HFmrEF; EF 40-49%), and 621 as having HF with preserved EF (HFpEF; EF ≥ 50%). A total of 11 readily available variables built up the prognostic index. For four of these predictor variables, namely systolic blood pressure, serum creatinine, myocardial infarction, and diabetes, the effect differed across the three HF phenotypes. With a weighted IECV-adjusted AUC of 0.79 (0.74-0.83) for HFrEF, 0.74 (0.70-0.79) for HFmrEF, and 0.74 (0.71-0.77) for HFpEF, the model showed excellent discrimination. Moreover, there was a good agreement between the average observed and predicted 1-year mortality risks, especially after recalibration of the baseline mortality risks.

CONCLUSIONS

Our HF phenotype stratified model showed excellent generalizability across four European cohorts and may provide a useful tool in HF phenotype-specific clinical decision-making.

Association Between US Physician Malpractice Claims Rates and Hospital Admission Rates Among Patients With Lower-Risk Syncope

IMPORTANCE

The US Government Accountability Office has changed its estimate of the annual costs of defensive medicine, largely because it has been difficult to objectively measure its impact. Evaluating the association of malpractice claims rates with hospital admission rates and the costs of admitting patients with low-risk conditions would help to document the impact of defensive medicine. Although syncope is a concerning symptom, most patients with syncope have a low risk of adverse outcomes. However, many low-risk patients are still admitted to the hospital, with associated costs of more than $2.5 billion per year in the US.

OBJECTIVE

To assess whether hospital admission rates after emergency department visits among patients with lower-risk syncope are associated with state-level variations in malpractice claims rates.

DESIGN, SETTING, AND PARTICIPANTS

This cross-sectional study of emergency department visits among patients with lower-risk syncope used deidentified data from the Clinformatics Data Mart database (Optum). Lower-risk syncope visits were defined as those with a primary diagnosis of syncope and collapse based on International Classification of Diseases, Ninth Revision, Clinical Modification code 780.2 or International Classification of Diseases, Tenth Revision, Clinical Modification code R55 that did not include another major diagnostic code for a condition requiring hospital admission (such as heart disease, cancer, or medical shock) or an inpatient hospital stay of more than 3 days. These data were linked to publicly available data from the National Practitioner Data Bank pertaining to physician malpractice claims between January 1, 2008, and December 31, 2017. The 2 data sets were linked at the state-year level. Data were analyzed from October 2, 2019, to September 12, 2020.

MAIN OUTCOMES AND MEASURES

The association between the rate of hospital admission after emergency department visits among patients with lower-risk syncope and the rate of physician malpractice claims was assessed at the state-year level using a state-level fixed-effects model. Standardized costs obtained from the Clinformatics Data Mart database were adjusted for inflation and expressed in 2017 US dollars using the Consumer Price Index.

RESULTS

Among 40 482 813 emergency department visits between 2008 and 2017, 519 724 visits (1.3%) were associated with syncope. Of those, 234 750 visits (45.2%) met the criteria for lower-risk syncope. The mean (SD) age of patients in the lower-risk cohort was 71.8 (13.5) years; 141 050 patients (60.1%) were female, and 44 115 patients (18.8%) were admitted to the hospital, representing an extra cost of $6542 per admission. The mean rate of physician malpractice claims varied from 0.27 claims per 100 000 people to 8.63 claims per 100 000 people across states and across years within states. A state-level fixed-effects regression model indicated that, for every 1 in 100 000-person increase in the physician malpractice claims rate, there was an absolute increase of 6.70% (95% CI, 4.65%-8.75%) or a relative increase of 35.6% in the hospital admission rate, which represented an additional $102 million in costs associated with this lower-risk cohort.

CONCLUSIONS AND RELEVANCE

In this study, increases in physician malpractice claims rates were associated with increases in hospital admission rates and substantial health care costs for patients with lower-risk syncope, and these increases are likely associated with the practice of defensive medicine.

A prognostic model predicted deterioration in health-related quality of life in older patients with multimorbidity and polypharmacy

OBJECTIVES

To develop and validate a prognostic model to predict deterioration in health-related quality of life (dHRQoL) in older general practice patients with at least one chronic condition and one chronic prescription.

STUDY DESIGN AND SETTING

We used individual participant data from five cluster-randomized trials conducted in the Netherlands and Germany to predict dHRQoL, defined as a decrease in EQ-5D-3 L index score of ≥5% after 6-month follow-up in logistic regression models with stratified intercepts to account for between-study heterogeneity. The model was validated internally and by using internal-external cross-validation (IECV).

RESULTS

In 3,582 patients with complete data, of whom 1,046 (29.2%) showed deterioration in HRQoL, and 12/87 variables were selected that were related to single (chronic) conditions, inappropriate medication, medication underuse, functional status, well-being, and HRQoL. Bootstrap internal validation showed a C-statistic of 0.71 (0.69 to 0.72) and a calibration slope of 0.88 (0.78 to 0.98). In the IECV loop, the model provided a pooled C-statistic of 0.68 (0.65 to 0.70) and calibration-in-the-large of 0 (-0.13 to 0.13). HRQoL/functionality had the strongest prognostic value.

CONCLUSION

The model performed well in terms of discrimination, calibration, and generalizability and might help clinicians identify older patients at high risk of dHRQoL.

REGISTRATION

PROSPERO ID: CRD42018088129.

Can personalized treatment prediction improve the outcomes, compared with the group average approach, in a randomized trial? Developing and validating a multivariable prediction model in a pragmatic megatrial of acute treatment for major depression

BACKGROUND

Clinical trials have traditionally been analysed at the aggregate level, assuming that the group average would be applicable to all eligible and similar patients. We re-analyzed a mega-trial of antidepressant therapy for major depression to explore whether a multivariable prediction model may lead to different treatment recommendations for individual participants.

METHODS

The trial compared the second-line treatment strategies of continuing sertraline, combining it with mirtazapine or switching to mirtazapine after initial failure to remit on sertraline among 1,544 patients with major depression. The outcome was the Personal Health Questionnaire-9 (PHQ-9) at week 9: the original analyses showed that both combining and switching resulted in greater reduction in PHQ-9 by 1.0 point than continuing. We considered several models of penalized regression or machine learning.

RESULTS

Models using support vector machines (SVMs) provided the best performance. Using SVMs, continuing sertraline was predicted to be the best treatment for 123 patients, combining for 696 patients, and switching for 725 patients. In the last two subgroups, both combining and switching were equally superior to continuing by 1.2 to 1.4 points, resulting in the same treatment recommendations as with the original aggregate data level analyses; in the first subgroup, however, switching was substantively inferior to combining (-3.1, 95%CI: -5.4 to -0.5).

LIMITATIONS

Stronger predictors are needed to make more precise predictions.

CONCLUSIONS

The multivariable prediction models led to improved recommendations for a minority of participants than the group average approach in a megatrial.

External validation of a prognostic model for intensive care unit mortality: a retrospective study using the Ontario Critical Care Information System

PURPOSE

To externally validate an intensive care unit (ICU) mortality prediction model that was created using the Ontario Critical Care Information System (CCIS), which includes the Multiple Organ Dysfunction Score (MODS).

METHODS

We applied the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) recommendations to a prospective longitudinal cohort of patients discharged between 1 July 2015 and 31 December 31 2016 from 90 adult level-3 critical care units in Ontario. We used multivariable logistic regression with measures of discrimination, calibration-in-the-large, calibration slope, and flexible calibration plots to compare prediction model performance of the entire data set and for each ICU subtype.

RESULTS

Among 121,201 CCIS records with ICU mortality of 11.3%, the C-statistic for the validation data set was 0.805. The C-statistic ranged from 0.775 to 0.846 among the ICU subtypes. After intercept recalibration to adjust the baseline risk, the mean predicted risk of death matched actual ICU mortality. The calibration slope was close to 1 with all CCIS data and ICU subtypes of cardiovascular and community hospitals with low ventilation rates. Calibration slopes significantly less than 1 were found for ICUs in teaching hospitals and community hospitals with high ventilation rates whereas coronary care units had a calibration slope significantly higher than 1. Calibration plots revealed over-prediction in high risk groups to a varying degree across all cohorts.

CONCLUSIONS

A risk prediction model primarily based on the MODS shows reproducibility and transportability after intercept recalibration. Risk adjusting models that use existing and feasible data collection can support performance measurement at the individual ICU level.

Machine learning algorithms performed no better than regression models for prognostication in traumatic brain injury

OBJECTIVE

We aimed to explore the added value of common machine learning (ML) algorithms for prediction of outcome for moderate and severe traumatic brain injury.

STUDY DESIGN AND SETTING

We performed logistic regression (LR), lasso regression, and ridge regression with key baseline predictors in the IMPACT-II database (15 studies, n = 11,022). ML algorithms included support vector machines, random forests, gradient boosting machines, and artificial neural networks and were trained using the same predictors. To assess generalizability of predictions, we performed internal, internal-external, and external validation on the recent CENTER-TBI study (patients with Glasgow Coma Scale <13, n = 1,554). Both calibration (calibration slope/intercept) and discrimination (area under the curve) was quantified.

RESULTS

In the IMPACT-II database, 3,332/11,022 (30%) died and 5,233(48%) had unfavorable outcome (Glasgow Outcome Scale less than 4). In the CENTER-TBI study, 348/1,554(29%) died and 651(54%) had unfavorable outcome. Discrimination and calibration varied widely between the studies and less so between the studied algorithms. The mean area under the curve was 0.82 for mortality and 0.77 for unfavorable outcomes in the CENTER-TBI study.

CONCLUSION

ML algorithms may not outperform traditional regression approaches in a low-dimensional setting for outcome prediction after moderate or severe traumatic brain injury. Similar to regression-based prediction models, ML algorithms should be rigorously validated to ensure applicability to new populations.

Machine-learning-based patient-specific prediction models for knee osteoarthritis

Osteoarthritis (OA) is an extremely common musculoskeletal disease. However, current guidelines are not well suited for diagnosing patients in the early stages of disease and do not discriminate patients for whom the disease might progress rapidly. The most important hurdle in OA management is identifying and classifying patients who will benefit most from treatment. Further efforts are needed in patient subgrouping and developing prediction models. Conventional statistical modelling approaches exist; however, these models are limited in the amount of information they can adequately process. Comprehensive patient-specific prediction models need to be developed. Approaches such as data mining and machine learning should aid in the development of such models. Although a challenging task, technology is now available that should enable subgrouping of patients with OA and lead to improved clinical decision-making and precision medicine.

World Health Organization cardiovascular disease risk charts: revised models to estimate risk in 21 global regions

BACKGROUND

To help adapt cardiovascular disease risk prediction approaches to low-income and middle-income countries, WHO has convened an effort to develop, evaluate, and illustrate revised risk models. Here, we report the derivation, validation, and illustration of the revised WHO cardiovascular disease risk prediction charts that have been adapted to the circumstances of 21 global regions.

METHODS

In this model revision initiative, we derived 10-year risk prediction models for fatal and non-fatal cardiovascular disease (ie, myocardial infarction and stroke) using individual participant data from the Emerging Risk Factors Collaboration. Models included information on age, smoking status, systolic blood pressure, history of diabetes, and total cholesterol. For derivation, we included participants aged 40-80 years without a known baseline history of cardiovascular disease, who were followed up until the first myocardial infarction, fatal coronary heart disease, or stroke event. We recalibrated models using age-specific and sex-specific incidences and risk factor values available from 21 global regions. For external validation, we analysed individual participant data from studies distinct from those used in model derivation. We illustrated models by analysing data on a further 123 743 individuals from surveys in 79 countries collected with the WHO STEPwise Approach to Surveillance.

FINDINGS

Our risk model derivation involved 376 177 individuals from 85 cohorts, and 19 333 incident cardiovascular events recorded during 10 years of follow-up. The derived risk prediction models discriminated well in external validation cohorts (19 cohorts, 1 096 061 individuals, 25 950 cardiovascular disease events), with Harrell's C indices ranging from 0·685 (95% CI 0·629-0·741) to 0·833 (0·783-0·882). For a given risk factor profile, we found substantial variation across global regions in the estimated 10-year predicted risk. For example, estimated cardiovascular disease risk for a 60-year-old male smoker without diabetes and with systolic blood pressure of 140 mm Hg and total cholesterol of 5 mmol/L ranged from 11% in Andean Latin America to 30% in central Asia. When applied to data from 79 countries (mostly low-income and middle-income countries), the proportion of individuals aged 40-64 years estimated to be at greater than 20% risk ranged from less than 1% in Uganda to more than 16% in Egypt.

INTERPRETATION

We have derived, calibrated, and validated new WHO risk prediction models to estimate cardiovascular disease risk in 21 Global Burden of Disease regions. The widespread use of these models could enhance the accuracy, practicability, and sustainability of efforts to reduce the burden of cardiovascular disease worldwide.

FUNDING

World Health Organization, British Heart Foundation (BHF), BHF Cambridge Centre for Research Excellence, UK Medical Research Council, and National Institute for Health Research.

Assessment of heterogeneity in an individual participant data meta-analysis of prediction models: An overview and illustration

Clinical prediction models aim to provide estimates of absolute risk for a diagnostic or prognostic endpoint. Such models may be derived from data from various studies in the context of a meta‐analysis. We describe and propose approaches for assessing heterogeneity in predictor effects and predictions arising from models based on data from different sources. These methods are illustrated in a case study with patients suffering from traumatic brain injury, where we aim to predict 6‐month mortality based on individual patient data using meta‐analytic techniques (15 studies, n = 11 022 patients). The insights into various aspects of heterogeneity are important to develop better models and understand problems with the transportability of absolute risk predictions.

Development and validation of a prediction model for fat mass in children and adolescents: meta-analysis using individual participant data

OBJECTIVES

To develop and validate a prediction model for fat mass in children aged 4-15 years using routinely available risk factors of height, weight, and demographic information without the need for more complex forms of assessment.

DESIGN

Individual participant data meta-analysis.

SETTING

Four population based cross sectional studies and a fifth study for external validation, United Kingdom.

PARTICIPANTS

A pooled derivation dataset (four studies) of 2375 children and an external validation dataset of 176 children with complete data on anthropometric measurements and deuterium dilution assessments of fat mass.

MAIN OUTCOME MEASURE

Multivariable linear regression analysis, using backwards selection for inclusion of predictor variables and allowing non-linear relations, was used to develop a prediction model for fat-free mass (and subsequently fat mass by subtracting resulting estimates from weight) based on the four studies. Internal validation and then internal-external cross validation were used to examine overfitting and generalisability of the model's predictive performance within the four development studies; external validation followed using the fifth dataset.

RESULTS

Model derivation was based on a multi-ethnic population of 2375 children (47.8% boys, n=1136) aged 4-15 years. The final model containing predictor variables of height, weight, age, sex, and ethnicity had extremely high predictive ability (optimism adjusted R²: 94.8%, 95% confidence interval 94.4% to 95.2%) with excellent calibration of observed and predicted values. The internal validation showed minimal overfitting and good model generalisability, with excellent calibration and predictive performance. External validation in 176 children aged 11-12 years showed promising generalisability of the model (R²: 90.0%, 95% confidence interval 87.2% to 92.8%) with good calibration of observed and predicted fat mass (slope: 1.02, 95% confidence interval 0.97 to 1.07). The mean difference between observed and predicted fat mass was -1.29 kg (95% confidence interval -1.62 to -0.96 kg).

CONCLUSION

The developed model accurately predicted levels of fat mass in children aged 4-15 years. The prediction model is based on simple anthropometric measures without the need for more complex forms of assessment and could improve the accuracy of assessments for body fatness in children (compared with those provided by body mass index) for effective surveillance, prevention, and management of clinical and public health obesity.

Evidence synthesis in prognosis research

Over the past few years, evidence synthesis has become essential to investigate and improve the generalizability of medical research findings. This strategy often involves a meta-analysis to formally summarize quantities of interest, such as relative treatment effect estimates. The use of meta-analysis methods is, however, less straightforward in prognosis research because substantial variation exists in research objectives, analysis methods and the level of reported evidence. We present a gentle overview of statistical methods that can be used to summarize data of prognostic factor and prognostic model studies. We discuss how aggregate data, individual participant data, or a combination thereof can be combined through meta-analysis methods. Recent examples are provided throughout to illustrate the various methods.

Understanding the relation between Zika virus infection during pregnancy and adverse fetal, infant and child outcomes: a protocol for a systematic review and individual participant data meta-analysis of longitudinal studies of pregnant women and their infants and children

INTRODUCTION

Zika virus (ZIKV) infection during pregnancy is a known cause of microcephaly and other congenital and developmental anomalies. In the absence of a ZIKV vaccine or prophylactics, principal investigators (PIs) and international leaders in ZIKV research have formed the ZIKV Individual Participant Data (IPD) Consortium to identify, collect and synthesise IPD from longitudinal studies of pregnant women that measure ZIKV infection during pregnancy and fetal, infant or child outcomes.

METHODS AND ANALYSIS

We will identify eligible studies through the ZIKV IPD Consortium membership and a systematic review and invite study PIs to participate in the IPD meta-analysis (IPD-MA). We will use the combined dataset to estimate the relative and absolute risk of congenital Zika syndrome (CZS), including microcephaly and late symptomatic congenital infections; identify and explore sources of heterogeneity in those estimates and develop and validate a risk prediction model to identify the pregnancies at the highest risk of CZS or adverse developmental outcomes. The variable accuracy of diagnostic assays and differences in exposure and outcome definitions means that included studies will have a higher level of systematic variability, a component of measurement error, than an IPD-MA of studies of an established pathogen. We will use expert testimony, existing internal and external diagnostic accuracy validation studies and laboratory external quality assessments to inform the distribution of measurement error in our models. We will apply both Bayesian and frequentist methods to directly account for these and other sources of uncertainty.

ETHICS AND DISSEMINATION

The IPD-MA was deemed exempt from ethical review. We will convene a group of patient advocates to evaluate the ethical implications and utility of the risk stratification tool. Findings from these analyses will be shared via national and international conferences and through publication in open access, peer-reviewed journals.

TRIAL REGISTRATION NUMBER

PROSPERO International prospective register of systematic reviews (CRD42017068915).

Prognostic model for traumatic death due to bleeding: cross-sectional international study

OBJECTIVE

To develop and validate a prognostic model and a simple model to predict death due to bleeding in trauma patients.

DESIGN

Cross-sectional study with multivariable logistic regression using data from two large trauma cohorts.

SETTING

274 hospitals from 40 countries in the Clinical Randomisation of Anti-fibrinolytic in Significant Haemorrhage (CRASH-2) trial and 24 hospitals in the Northern French Alps Trauma registry.

PARTICIPANTS

13 485 trauma patients in the CRASH-2 trial and 9945 patients in the Northern French Alps Trauma registry who were admitted to hospital within 3 hours of injury.

MAIN OUTCOME MEASURE

In-hospital death due to bleeding within 28 days.

RESULTS

There were 815 (6%) deaths from bleeding in the CRASH-2 trial and 102 (1%) in the Northern French Alps Trauma registry. The full model included age, systolic blood pressure (SBP), Glasgow Coma Scale (GCS), heart rate, respiratory rate and type of injury (penetrating). The simple model included age, SBP and GCS. In a cross-validation procedure by country, discrimination and calibration were adequate (pooled C-statistic 0.85 (95% CI 0.81 to 0.88) for the full model and 0.84 (95% CI 0.80 to 0.88) for the simple model).

CONCLUSION

This prognostic model can identify trauma patients at risk of death due to bleeding in a wide range of settings and can support prehospital triage and trauma audit, including audit of tranexamic acid use.

External validation of a modified trauma and injury severity score model in major trauma injury

BACKGROUND

The establishment of an accurate prognostic model in major trauma patients is important mainly because this group of patients will benefit the most. Clinical prediction models must be validated internally and externally on a regular basis to ensure the prediction is accurate and current. This study aims to externally validate two prediction models, the Trauma and Injury Severity Score model developed using the Major Trauma Outcome Study in North America (MTOS-TRISS model), and the NTrD-TRISS model, which is a refined MTOS-TRISS model with coefficients derived from the Malaysian National Trauma Database (NTrD), by regarding mortality as the outcome measurement.

METHOD

This retrospective study included patients with major trauma injuries reported to a trauma centre of Hospital Sultanah Aminah over a 6-year period from 2011 and 2017. Model validation was examined using the measures of discrimination and calibration. Discrimination was assessed using the area under the receiver operating characteristic curve (AUC) and 95% confidence interval (CI). The Hosmer-Lemeshow (H-L) goodness-of-fit test was used to examine calibration capabilities. The predictive validity of both MTOS-TRISS and NTrD-TRISS models were further evaluated by incorporating parameters such as the New Injury Severity Scale and the Injury Severity Score.

RESULTS

Total patients of 3788 (3434 blunt and 354 penetrating injuries) with average age of 37 years (standard deviation of 16 years) were included in this study. All MTOS-TRISS and NTrD-TRISS models examined in this study showed adequate discriminative ability with AUCs ranged from 0.86 to 0.89 for patients with blunt trauma mechanism and 0.89 to 0.99 for patients with penetrating trauma mechanism. The H-L goodness-of-fit test indicated the NTrD-TRISS model calibrated as good as the MTOS-TRISS model for patients with blunt trauma mechanism.

CONCLUSION

For patients with blunt trauma mechanism, both the MTOS-TRISS and NTrD-TRISS models showed good discrimination and calibration performances. Discrimination performance for the NTrD-TRISS model was revealed to be as good as the MTOS-TRISS model specifically for patients with penetrating trauma mechanism. Overall, this validation study has ascertained the discrimination and calibration performances of the NTrD-TRISS model to be as good as the MTOS-TRISS model particularly for patients with blunt trauma mechanism.

Untapped potential of multicenter studies: a review of cardiovascular risk prediction models revealed inappropriate analyses and wide variation in reporting

BACKGROUND

Clinical prediction models are often constructed using multicenter databases. Such a data structure poses additional challenges for statistical analysis (clustered data) but offers opportunities for model generalizability to a broad range of centers. The purpose of this study was to describe properties, analysis, and reporting of multicenter studies in the Tufts PACE Clinical Prediction Model Registry and to illustrate consequences of common design and analyses choices.

METHODS

Fifty randomly selected studies that are included in the Tufts registry as multicenter and published after 2000 underwent full-text screening. Simulated examples illustrate some key concepts relevant to multicenter prediction research.

RESULTS

Multicenter studies differed widely in the number of participating centers (range 2 to 5473). Thirty-nine of 50 studies ignored the multicenter nature of data in the statistical analysis. In the others, clustering was resolved by developing the model on only one center, using mixed effects or stratified regression, or by using center-level characteristics as predictors. Twenty-three of 50 studies did not describe the clinical settings or type of centers from which data was obtained. Four of 50 studies discussed neither generalizability nor external validity of the developed model.

CONCLUSIONS

Regression methods and validation strategies tailored to multicenter studies are underutilized. Reporting on generalizability and potential external validity of the model lacks transparency. Hence, multicenter prediction research has untapped potential.

REGISTRATION

This review was not registered.

The use of prognostic scores for causal inference with general treatment regimes

In nonrandomised studies, inferring causal effects requires appropriate methods for addressing confounding bias. Although it is common to adopt propensity score analysis to this purpose, prognostic score analysis has recently been proposed as an alternative strategy. While both approaches were originally introduced to estimate causal effects for binary interventions, the theory of propensity score has since been extended to the case of general treatment regimes. Indeed, many treatments are not assigned in a binary fashion and require a certain extent of dosing. Hence, researchers may often be interested in estimating treatment effects across multiple exposures. To the best of our knowledge, the prognostic score analysis has not been yet generalised to this case. In this article, we describe the theory of prognostic scores for causal inference with general treatment regimes. Our methods can be applied to compare multiple treatments using nonrandomised data, a topic of great relevance in contemporary evaluations of clinical interventions. We propose estimators for the average treatment effects in different populations of interest, the validity of which is assessed through a series of simulations. Finally, we present an illustrative case in which we estimate the effect of the delay to Aspirin administration on a composite outcome of death or dependence at 6 months in stroke patients.

Development and validation of a prediction model to estimate the risk of liver cirrhosis in primary care patients with abnormal liver blood test results: protocol for an electronic health record study in Clinical Practice Research Datalink

BACKGROUND

Driven by alcohol consumption and obesity, the prevalence of non-viral liver disease in the UK is increasing. Due to its silent and slow nature, the progression of liver disease is currently unpredictable and challenging to monitor. The latest National Institute for Health Care Excellence cirrhosis guidelines call for a validated risk tool that would allow general practitioners to identify patients that are at high risk of developing cirrhosis.

METHODS

Using linked electronic health records from the Clinical Practice Research Datalink (a database of > 10 million patients in England), we aim to develop and validate a prediction model to estimate 2-, 5- and 10-year risk of cirrhosis. The model will provide individualised cirrhosis risk predictions for adult primary care patients, free from underlying liver disease or viral hepatitis infection, whose liver blood test results come back abnormal. We will externally validate the model in patients from 30 further Clinical Practice Research Datalink general practices in England.

DISCUSSION

The prediction model will provide estimates of cirrhosis risk in primary care patients with abnormal liver blood test results to guide referral to secondary care, to identify patients who are in serious need of preventative health interventions and to help reassure patients at low risk of cirrhosis in the long term.

Development and validation of prediction models to estimate risk of primary total hip and knee replacements using data from the UK: two prospective open cohorts using the UK Clinical Practice Research Datalink

Objectives

The ability to efficiently and accurately predict future risk of primary total hip and knee replacement (THR/TKR) in earlier stages of osteoarthritis (OA) has potentially important applications. We aimed to develop and validate two models to estimate an individual’s risk of primary THR and TKR in patients newly presenting to primary care.

Methods

We identified two cohorts of patients aged ≥40 years newly consulting hip pain/OA and knee pain/OA in the Clinical Practice Research Datalink. Candidate predictors were identified by systematic review, novel hypothesis-free ‘Record-Wide Association Study’ with replication, and panel consensus. Cox proportional hazards models accounting for competing risk of death were applied to derive risk algorithms for THR and TKR. Internal–external cross-validation (IECV) was then applied over geographical regions to validate two models.

Results

45 predictors for THR and 53 for TKR were identified, reviewed and selected by the panel. 301 052 and 416 030 patients newly consulting between 1992 and 2015 were identified in the hip and knee cohorts, respectively (median follow-up 6 years). The resultant model C-statistics is 0.73 (0.72, 0.73) and 0.79 (0.78, 0.79) for THR (with 20 predictors) and TKR model (with 24 predictors), respectively. The IECV C-statistics ranged between 0.70–0.74 (THR model) and 0.76–0.82 (TKR model); the IECV calibration slope ranged between 0.93–1.07 (THR model) and 0.92–1.12 (TKR model).

Conclusions

Two prediction models with good discrimination and calibration that estimate individuals’ risk of THR and TKR have been developed and validated in large-scale, nationally representative data, and are readily automated in electronic patient records.

Lifestyle index for mortality prediction using multiple ageing cohorts in the USA, UK and Europe

Current mortality prediction indexes are mainly based on functional morbidity and comorbidity, with limited information for risk prevention. This study aimed to develop and validate a modifiable lifestyle-based mortality predication index for older adults. Data from 51,688 participants (56% women) aged ≥50 years in 2002 Health and Retirement Study, 2002 English Longitudinal Study of Ageing and 2004 Survey of Health Ageing and Retirement in Europe were used to estimate coefficients of the index with cohort-stratified Cox regression. Models were validated across studies and compared to the Lee index (having comorbid and morbidity predictors). Over an average of 11-year follow-up, 10,240 participants died. The lifestyle index includes smoking, drinking, exercising, sleep quality, BMI, sex and age; showing adequate model performance in internal validation (C-statistic 0.79; D-statistic 1.94; calibration slope 1.13) and in all combinations of internal-external cross-validation. It outperformed Lee index (e.g. differences in C-statistic = 0.01, D-statistic = 0.17, P < 0.001) consistently across health status. The lifestyle index stratified participants into varying mortality risk groups, with those in the top quintile having 13.5% excess absolute mortality risk over 10 years than those in the bottom 50th centile. Our lifestyle index with easy-assessable behavioural factors and improved generalizability may maximize its usability for personalized risk management.

The current application of the Royston-Parmar model for prognostic modeling in health research: a scoping review

BACKGROUND

Prognostic models incorporating survival analysis predict the risk (i.e., probability) of experiencing a future event over a specific time period. In 2002, Royston and Parmar described a type of flexible parametric survival model called the Royston-Parmar model in Statistics in Medicine, a model which fits a restricted cubic spline to flexibly model the baseline log cumulative hazard on the proportional hazards scale. This feature permits absolute measures of effect (e.g., hazard rates) to be estimated at all time points, an important feature when using the model. The Royston-Parmar model can also incorporate time-dependent effects and be used on different scales (e.g., proportional odds, probit). These features make the Royston-Parmar model attractive for prediction, yet their current uptake for prognostic modeling is unknown. Thus, the objectives were to conduct a scoping review of how the Royston-Parmar model has been applied to prognostic models in health research, to raise awareness of the model, to identify gaps in current reporting, and to offer model building considerations and reporting suggestions for other researchers.

METHODS

Five electronic databases and gray literature indexed in web sources from 2001 to 2016 were searched to identify articles for inclusion in the scoping review. Two reviewers independently screened 1429 articles, and after applying exclusion criteria through a two-step screening process, data from 12 studies were abstracted.

RESULTS

Since 2001, only 12 studies were identified that used the Royston-Parmar model in some capacity for prognostic modeling, 10 of which used the model as the basis for their prognostic model. The restricted cubic spline varied across studies in the number of interior knots (range 1 to 6), and only three studies reported knot placement. Three studies provided details about the baseline function, with two studies using a figure and the third providing coefficients. However, no studies provided adequate information on their restricted cubic spline to permit others to validate or completely use the model.

CONCLUSIONS

Despite the advantages of the Royston-Parmar model for prognostic models, they are not widely used in health research. Better reporting of details about the restricted cubic spline is needed, so the prognostic model can be used and validated by others.

REGISTRATION

The protocol was registered with Open Science Framework (https://osf.io/r3232/).

A multiple-model generalisation of updating clinical prediction models

There is growing interest in developing clinical prediction models (CPMs) to aid local healthcare decision‐making. Frequently, these CPMs are developed in isolation across different populations, with repetitive de novo derivation a common modelling strategy. However, this fails to utilise all available information and does not respond to changes in health processes through time and space. Alternatively, model updating techniques have previously been proposed that adjust an existing CPM to suit the new population, but these techniques are restricted to a single model. Therefore, we aimed to develop a generalised method for updating and aggregating multiple CPMs. The proposed “hybrid method” re‐calibrates multiple CPMs using stacked regression while concurrently revising specific covariates using individual participant data (IPD) under a penalised likelihood. The performance of the hybrid method was compared with existing methods in a clinical example of mortality risk prediction after transcatheter aortic valve implantation, and in 2 simulation studies. The simulation studies explored the effect of sample size and between‐population‐heterogeneity on the method, with each representing a situation of having multiple distinct CPMs and 1 set of IPD. When the sample size of the IPD was small, stacked regression and the hybrid method had comparable but highest performance across modelling methods. Conversely, in large IPD samples, development of a new model and the hybrid method gave the highest performance. Hence, the proposed strategy can inform the choice between utilising existing CPMs or developing a model de novo, thereby incorporating IPD, existing research, and prior (clinical) knowledge into the modelling strategy.

Diagnostic and therapeutic medical devices for safer blood management in cardiac surgery: systematic reviews, observational studies and randomised controlled trials

Background

Anaemia, coagulopathic bleeding and transfusion are strongly associated with organ failure, sepsis and death following cardiac surgery.

Objective

To evaluate the clinical effectiveness and cost-effectiveness of medical devices used as diagnostic and therapeutic tools for the management of anaemia and bleeding in cardiac surgery.

Methods and results

Workstream 1 – in the COagulation and Platelet laboratory Testing in Cardiac surgery (COPTIC) study we demonstrated that risk assessment using baseline clinical factors predicted bleeding with a high degree of accuracy. The results from point-of-care (POC) platelet aggregometry or viscoelastometry tests or an expanded range of laboratory reference tests for coagulopathy did not improve predictive accuracy beyond that achieved with the clinical risk score alone. The routine use of POC tests was not cost-effective. A systematic review concluded that POC-based algorithms are not clinically effective. We developed two new clinical risk prediction scores for transfusion and bleeding that are available as e-calculators. Workstream 2 – in the PAtient-SPecific Oxygen monitoring to Reduce blood Transfusion during heart surgery (PASPORT) trial and a systematic review we demonstrated that personalised near-infrared spectroscopy-based algorithms for the optimisation of tissue oxygenation, or as indicators for red cell transfusion, were neither clinically effective nor cost-effective. Workstream 3 – in the REDWASH trial we failed to demonstrate a reduction in inflammation or organ injury in recipients of mechanically washed red cells compared with standard (unwashed) red cells.

Limitations

Existing studies evaluating the predictive accuracy or effectiveness of POC tests of coagulopathy or near-infrared spectroscopy were at high risk of bias. Interventions that alter red cell transfusion exposure, a common surrogate outcome in most trials, were not found to be clinically effective.

Conclusions

A systematic assessment of devices in clinical use as blood management adjuncts in cardiac surgery did not demonstrate clinical effectiveness or cost-effectiveness. The contribution of anaemia and coagulopathy to adverse clinical outcomes following cardiac surgery remains poorly understood. Further research to define the pathogenesis of these conditions may lead to more accurate diagnoses, more effective treatments and potentially improved clinical outcomes.

Study registration

Current Controlled Trials ISRCTN20778544 (COPTIC study) and PROSPERO CRD42016033831 (systematic review) (workstream 1); Current Controlled Trials ISRCTN23557269 (PASPORT trial) and PROSPERO CRD4201502769 (systematic review) (workstream 2); and Current Controlled Trials ISRCTN27076315 (REDWASH trial) (workstream 3).

Funding

This project was funded by the National Institute for Health Research (NIHR) Programme Grants for Applied Research programme and will be published in full inProgramme Grants for Applied Research; Vol. 5, No. 17. See the NIHR Journals Library website for further project information.

Predicting major bleeding in patients with noncardioembolic stroke on antiplatelets: S2TOP-BLEED

OBJECTIVE

To develop and externally validate a prediction model for major bleeding in patients with a TIA or ischemic stroke on antiplatelet agents.

METHODS

We combined individual patient data from 6 randomized clinical trials (CAPRIE, ESPS-2, MATCH, CHARISMA, ESPRIT, and PRoFESS) investigating antiplatelet therapy after TIA or ischemic stroke. Cox regression analyses stratified by trial were performed to study the association between predictors and major bleeding. A risk prediction model was derived and validated in the PERFORM trial. Performance was assessed with the c statistic and calibration plots.

RESULTS

Major bleeding occurred in 1,530 of the 43,112 patients during 94,833 person-years of follow-up. The observed 3-year risk of major bleeding was 4.6% (95% confidence interval [CI] 4.4%-4.9%). Predictors were male sex, smoking, type of antiplatelet agents (aspirin-clopidogrel), outcome on modified Rankin Scale ≥3, prior stroke, high blood pressure, lower body mass index, elderly, Asian ethnicity, and diabetes (S₂TOP-BLEED). The S₂TOP-BLEED score had a c statistic of 0.63 (95% CI 0.60-0.64) and showed good calibration in the development data. Major bleeding risk ranged from 2% in patients aged 45-54 years without additional risk factors to more than 10% in patients aged 75-84 years with multiple risk factors. In external validation, the model had a c statistic of 0.61 (95% CI 0.59-0.63) and slightly underestimated major bleeding risk.

CONCLUSIONS

The S₂TOP-BLEED score can be used to estimate 3-year major bleeding risk in patients with a TIA or ischemic stroke who use antiplatelet agents, based on readily available characteristics. The discriminatory performance may be improved by identifying stronger predictors of major bleeding.

Associations between interarm differences in blood pressure and cardiovascular disease outcomes: protocol for an individual patient data meta-analysis and development of a prognostic algorithm

INTRODUCTION

Individual cohort studies in various populations and study-level meta-analyses have shown interarm differences (IAD) in blood pressure to be associated with increased cardiovascular and all-cause mortality. However, key questions remain, such as follows: (1) What is the additional contribution of IAD to prognostic risk estimation for cardiovascular and all-cause mortality? (2) What is the minimum cut-off value for IAD that defines elevated risk? (3) Is there a prognostic value of IAD and do different methods of IAD measurement impact on the prognostic value of IAD? We aim to address these questions by conducting an individual patient data (IPD) meta-analysis.

METHODS AND ANALYSIS

This study will identify prospective cohort studies that measured blood pressure in both arms during recruitment, and invite authors to contribute IPD datasets to this collaboration. All patient data received will be combined into a single dataset. Using one-stage meta-analysis, we will undertake multivariable time-to-event regression modelling, with the aim of developing a new prognostic model for cardiovascular risk estimation that includes IAD. We will explore variations in risk contribution of IAD across predefined population subgroups (eg, hypertensives, diabetics), establish the lower limit of IAD that is associated with additional cardiovascular risk and assess the impact of different methods of IAD measurement on risk prediction.

ETHICS AND DISSEMINATION

This study will not include any patient identifiable data. Included datasets will already have ethical approval and consent from their sponsors. Findings will be presented to international conferences and published in peer reviewed journals, and we have a comprehensive dissemination strategy in place with integrated patient and public involvement.

PROSPERO REGISTRATION NUMBER

CRD42015031227.

Measuring the statistical validity of summary meta-analysis and meta-regression results for use in clinical practice

An important question for clinicians appraising a meta‐analysis is: are the findings likely to be valid in their own practice—does the reported effect accurately represent the effect that would occur in their own clinical population? To this end we advance the concept of statistical validity—where the parameter being estimated equals the corresponding parameter for a new independent study. Using a simple (‘leave‐one‐out’) cross‐validation technique, we demonstrate how we may test meta‐analysis estimates for statistical validity using a new validation statistic, Vn, and derive its distribution.

We compare this with the usual approach of investigating heterogeneity in meta‐analyses and demonstrate the link between statistical validity and homogeneity. Using a simulation study, the properties of Vn and the Q statistic are compared for univariate random effects meta‐analysis and a tailored meta‐regression model, where information from the setting (included as model covariates) is used to calibrate the summary estimate to the setting of application. Their properties are found to be similar when there are 50 studies or more, but for fewer studies Vn has greater power but a higher type 1 error rate than Q. The power and type 1 error rate of Vn are also shown to depend on the within‐study variance, between‐study variance, study sample size, and the number of studies in the meta‐analysis. Finally, we apply Vn to two published meta‐analyses and conclude that it usefully augments standard methods when deciding upon the likely validity of summary meta‐analysis estimates in clinical practice. © 2017 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.

Integrating Tenascin-C protein expression and 1q25 copy number status in pediatric intracranial ependymoma prognostication: A new model for risk stratification

PURPOSE

Despite multimodal therapy, prognosis of pediatric intracranial ependymomas remains poor with a 5-year survival rate below 70% and frequent late deaths.

EXPERIMENTAL DESIGN

This multicentric European study evaluated putative prognostic biomarkers. Tenascin-C (TNC) immunohistochemical expression and copy number status of 1q25 were retained for a pooled analysis of 5 independent cohorts. The prognostic value of TNC and 1q25 on the overall survival (OS) was assessed using a Cox model adjusted to age at diagnosis, tumor location, WHO grade, extent of resection, radiotherapy and stratified by cohort. Stratification on a predictor that did not satisfy the proportional hazards assumption was considered. Model performance was evaluated and an internal-external cross validation was performed.

RESULTS

Among complete cases with 5-year median follow-up (n = 470; 131 deaths), TNC and 1q25 gain were significantly associated with age at diagnosis and posterior fossa tumor location. 1q25 status added independent prognostic value for death beyond the classical variables with a hazard ratio (HR) = 2.19 95%CI = [1.29; 3.76] (p = 0.004), while TNC prognostic relation was tumor location-dependent with HR = 2.19 95%CI = [1.29; 3.76] (p = 0.004) in posterior fossa and HR = 0.64 [0.28; 1.48] (p = 0.295) in supratentorial (interaction p value = 0.015). The derived prognostic score identified 3 different robust risk groups. The omission of upfront RT was not associated with OS for good and intermediate prognostic groups while the absence of upfront RT was negatively associated with OS in the poor risk group.

CONCLUSION

Integrated TNC expression and 1q25 status are useful to better stratify patients and to eventually adapt treatment regimens in pediatric intracranial ependymoma.

External validation, update and development of prediction models for pre-eclampsia using an Individual Participant Data (IPD) meta-analysis: the International Prediction of Pregnancy Complication Network (IPPIC pre-eclampsia) protocol

BACKGROUND

Pre-eclampsia, a condition with raised blood pressure and proteinuria is associated with an increased risk of maternal and offspring mortality and morbidity. Early identification of mothers at risk is needed to target management.

METHODS/DESIGN

We aim to systematically review the existing literature to identify prediction models for pre-eclampsia. We have established the International Prediction of Pregnancy Complication Network (IPPIC), made up of 72 researchers from 21 countries who have carried out relevant primary studies or have access to existing registry databases, and collectively possess data from more than two million patients. We will use the individual participant data (IPD) from these studies to externally validate these existing prediction models and summarise model performance across studies using random-effects meta-analysis for any, late (after 34 weeks) and early (before 34 weeks) onset pre-eclampsia. If none of the models perform well, we will recalibrate (update), or develop and validate new prediction models using the IPD. We will assess the differential accuracy of the models in various settings and subgroups according to the risk status. We will also validate or develop prediction models based on clinical characteristics only; clinical and biochemical markers; clinical and ultrasound parameters; and clinical, biochemical and ultrasound tests.

DISCUSSION

Numerous systematic reviews with aggregate data meta-analysis have evaluated various risk factors separately or in combination for predicting pre-eclampsia, but these are affected by many limitations. Our large-scale collaborative IPD approach encourages consensus towards well developed, and validated prognostic models, rather than a number of competing non-validated ones. The large sample size from our IPD will also allow development and validation of multivariable prediction model for the relatively rare outcome of early onset pre-eclampsia.

TRIAL REGISTRATION

The project was registered on Prospero on the 27 November 2015 with ID: CRD42015029349.

Three general concepts to improve risk prediction: good data, wisdom of the crowd, recalibration

In today's information age, the necessary means exist for clinical risk prediction to capitalize on a multitude of data sources, increasing the potential for greater accuracy and improved patient care. Towards this objective, the Prostate Cancer DREAM Challenge posted comprehensive information from three clinical trials recording survival for patients with metastatic castration-resistant prostate cancer treated with first-line docetaxel. A subset of an independent clinical trial was used for interim evaluation of model submissions, providing critical feedback to participating teams for tailoring their models to the desired target. Final submitted models were evaluated and ranked on the independent clinical trial. Our team, called "A Bavarian Dream", utilized many of the common statistical methods for data dimension reduction and summarization during the trial. Three general modeling principles emerged that were deemed helpful for building accurate risk prediction tools and ending up among the winning teams of both sub-challenges. These principles included: first, good data, encompassing the collection of important variables and imputation of missing data; second, wisdom of the crowd, extending beyond the usual model ensemble notion to the inclusion of experts on specific risk ranges; and third, recalibration, entailing transfer learning to the target source. In this study, we illustrate the application and impact of these principles applied to data from the Prostate Cancer DREAM Challenge.

Prediction of risk of recurrence of venous thromboembolism following treatment for a first unprovoked venous thromboembolism: systematic review, prognostic model and clinical decision rule, and economic evaluation

BACKGROUND

Unprovoked first venous thromboembolism (VTE) is defined as VTE in the absence of a temporary provoking factor such as surgery, immobility and other temporary factors. Recurrent VTE in unprovoked patients is highly prevalent, but easily preventable with oral anticoagulant (OAC) therapy. The unprovoked population is highly heterogeneous in terms of risk of recurrent VTE.

OBJECTIVES

The first aim of the project is to review existing prognostic models which stratify individuals by their recurrence risk, therefore potentially allowing tailored treatment strategies. The second aim is to enhance the existing research in this field, by developing and externally validating a new prognostic model for individual risk prediction, using a pooled database containing individual patient data (IPD) from several studies. The final aim is to assess the economic cost-effectiveness of the proposed prognostic model if it is used as a decision rule for resuming OAC therapy, compared with current standard treatment strategies.

METHODS

Standard systematic review methodology was used to identify relevant prognostic model development, validation and cost-effectiveness studies. Bibliographic databases (including MEDLINE, EMBASE and The Cochrane Library) were searched using terms relating to the clinical area and prognosis. Reviewing was undertaken by two reviewers independently using pre-defined criteria. Included full-text articles were data extracted and quality assessed. Critical appraisal of included full texts was undertaken and comparisons made of model performance. A prognostic model was developed using IPD from the pooled database of seven trials. A novel internal-external cross-validation (IECV) approach was used to develop and validate a prognostic model, with external validation undertaken in each of the trials iteratively. Given good performance in the IECV approach, a final model was developed using all trials data. A Markov patient-level simulation was used to consider the economic cost-effectiveness of using a decision rule (based on the prognostic model) to decide on resumption of OAC therapy (or not).

RESULTS

Three full-text articles were identified by the systematic review. Critical appraisal identified methodological and applicability issues; in particular, all three existing models did not have external validation. To address this, new prognostic models were sought with external validation. Two potential models were considered: one for use at cessation of therapy (pre D-dimer), and one for use after cessation of therapy (post D-dimer). Model performance measured in the external validation trials showed strong calibration performance for both models. The post D-dimer model performed substantially better in terms of discrimination (c = 0.69), better separating high- and low-risk patients. The economic evaluation identified that a decision rule based on the final post D-dimer model may be cost-effective for patients with predicted risk of recurrence of over 8% annually; this suggests continued therapy for patients with predicted risks ≥ 8% and cessation of therapy otherwise.

CONCLUSIONS

The post D-dimer model performed strongly and could be useful to predict individuals' risk of recurrence at any time up to 2-3 years, thereby aiding patient counselling and treatment decisions. A decision rule using this model may be cost-effective for informing clinical judgement and patient opinion in treatment decisions. Further research may investigate new predictors to enhance model performance and aim to further externally validate to confirm performance in new, non-trial populations. Finally, it is essential that further research is conducted to develop a model predicting bleeding risk on therapy, to manage the balance between the risks of recurrence and bleeding.

STUDY REGISTRATION

This study is registered as PROSPERO CRD42013003494.

FUNDING

The National Institute for Health Research Health Technology Assessment programme.