Developing and validating risk prediction models in an individual participant data meta-analysis

Multiple imputation of systematically missing data on gait speed in the Swedish National Study on Aging and Care

BACKGROUND

There is insufficient investigation of multiple imputation for systematically missing discrete variables in individual participant data meta-analysis (IPDMA) with a small number of included studies. Therefore, this study aims to evaluate the performance of three multiple imputation strategies - fully conditional specification (FCS), multivariate normal (MVN), conditional quantile imputation (CQI) - on systematically missing data on gait speed in the Swedish National Study on Aging and Care (SNAC).

METHODS

In total, 1 000 IPDMA were simulated with four prospective cohort studies based on the characteristics of the SNAC. The three multiple imputation strategies were analysed with a two-stage common-effect multivariable logistic model targeting the effect of three levels of gait speed (100% missing in one study) on 5-years mortality with common odds ratios set to OR1 = 0.55 (0.8-1.2 vs ≤0.8 m/s), and OR2 = 0.29 (>1.2 vs ≤0.8 m/s).

RESULTS

The average combined estimate for the mortality odds ratio OR1 (relative bias %) were 0.58 (8.2%), 0.58 (7.5%), and 0.55 (0.7%) for the FCS, MVN, and CQI, respectively. The average combined estimate for the mortality odds ratio OR2 (relative bias %) were 0.30 (2.5%), 0.33 (10.0%), and 0.29 (0.9%) for the FCS, MVN, and CQI respectively.

CONCLUSIONS

In our simulations of an IPDMA based on the SNAC where gait speed data was systematically missing in one study, all three imputation methods performed relatively well. The smallest bias was found for the CQI approach.

Study of How Adiposity in Pregnancy has an Effect on outcomeS (SHAPES): protocol for a prospective cohort study

INTRODUCTION

Maternal obesity increases the risk of multiple maternal and infant pregnancy complications, such as gestational diabetes and pre-eclampsia. Current UK guidelines use body mass index (BMI) to identify which women require additional care due to increased risk of complications. However, BMI may not accurately predict which women will develop complications during pregnancy as it does not determine amount and distribution of adipose tissue. Some adiposity measures (eg, waist circumference, ultrasound measures of abdominal visceral fat) can better identify where body fat is stored, which may be useful in predicting those women who need additional care.

METHODS AND ANALYSIS

This prospective cohort study (SHAPES, Study of How Adiposity in Pregnancy has an Effect on outcomeS) aims to evaluate the prognostic performance of adiposity measures (either alone or in combination with other adiposity, sociodemographic or clinical measures) to estimate risk of adverse pregnancy outcomes. Pregnant women (n=1400) will be recruited at their first trimester ultrasound scan (11+2-14+1 weeks') at Newcastle upon Tyne National Health Service Foundation Trust, UK. Early pregnancy adiposity measures and clinical and sociodemographic data will be collected. Routine data on maternal and infant pregnancy outcomes will be collected from routine hospital records. Regression methods will be used to compare the different adiposity measures with BMI in terms of their ability to predict pregnancy complications. If no individual measure performs better than BMI, multivariable models will be developed and evaluated to identify the most parsimonious model. The apparent performance of the developed model will be summarised using calibration, discrimination and internal validation analyses.

ETHICS AND DISSEMINATION

Ethical favourable opinion has been obtained from the North East: Newcastle & North Tyneside 1 Research Ethics Committee (REC reference: 22/NE/0035). All participants provide informed consent to take part in SHAPES. Planned dissemination includes peer-reviewed publications and additional dissemination appropriate to target audiences, including policy briefs for policymakers, media/social-media coverage for public and conferences for research TRIAL REGISTRATION NUMBER: ISRCTN82185177.

Poor handling of continuous predictors in clinical prediction models using logistic regression: a systematic review

BACKGROUND AND OBJECTIVES

When developing a clinical prediction model, assuming a linear relationship between the continuous predictors and outcome is not recommended. Incorrect specification of the functional form of continuous predictors could reduce predictive accuracy. We examine how continuous predictors are handled in studies developing a clinical prediction model.

METHODS

We searched PubMed for clinical prediction model studies developing a logistic regression model for a binary outcome, published between July 01, 2020, and July 30, 2020.

RESULTS

In total, 118 studies were included in the review (18 studies (15%) assessed the linearity assumption or used methods to handle nonlinearity, and 100 studies (85%) did not). Transformation and splines were commonly used to handle nonlinearity, used in 7 (n = 7/18, 39%) and 6 (n = 6/18, 33%) studies, respectively. Categorization was most often used method to handle continuous predictors (n = 67/118, 56.8%) where most studies used dichotomization (n = 40/67, 60%). Only ten models included nonlinear terms in the final model (n = 10/18, 56%).

CONCLUSION

Though widely recommended not to categorize continuous predictors or assume a linear relationship between outcome and continuous predictors, most studies categorize continuous predictors, few studies assess the linearity assumption, and even fewer use methodology to account for nonlinearity. Methodological guidance is provided to guide researchers on how to handle continuous predictors when developing a clinical prediction model.

Clinical utility of WHO-recommended screening tools and development and validation of novel clinical prediction models for pulmonary tuberculosis screening among outpatients living with HIV: an individual participant data meta-analysis

BACKGROUND

The World Health Organization (WHO) recommends that outpatient people living with HIV (PLHIV) undergo tuberculosis screening with the WHO four-symptom screen (W4SS) or C-reactive protein (CRP) (5 mg·L-1 cut-off) followed by confirmatory testing if screen positive. We conducted an individual participant data meta-analysis to determine the performance of WHO-recommended screening tools and two newly developed clinical prediction models (CPMs).

METHODS

Following a systematic review, we identified studies that recruited adult outpatient PLHIV irrespective of tuberculosis signs and symptoms or with a positive W4SS, evaluated CRP and collected sputum for culture. We used logistic regression to develop an extended CPM (which included CRP and other predictors) and a CRP-only CPM. We used internal-external cross-validation to evaluate performance.

RESULTS

We pooled data from eight cohorts (n=4315 participants). The extended CPM had excellent discrimination (C-statistic 0.81); the CRP-only CPM had similar discrimination. The C-statistics for WHO-recommended tools were lower. Both CPMs had equivalent or higher net benefit compared with the WHO-recommended tools. Compared with both CPMs, CRP (5 mg·L-1 cut-off) had equivalent net benefit across a clinically useful range of threshold probabilities, while the W4SS had a lower net benefit. The W4SS would capture 91% of tuberculosis cases and require confirmatory testing for 78% of participants. CRP (5 mg·L-1 cut-off), the extended CPM (4.2% threshold) and the CRP-only CPM (3.6% threshold) would capture similar percentages of cases but reduce confirmatory tests required by 24, 27 and 36%, respectively.

CONCLUSIONS

CRP sets the standard for tuberculosis screening among outpatient PLHIV. The choice between using CRP at 5 mg·L-1 cut-off or in a CPM depends on available resources.

Cardiovascular disease and type 2 diabetes in older adults: a combined protocol for an individual participant data analysis for risk prediction and a network meta-analysis of novel anti-diabetic drugs

Introduction

Older and multimorbid adults with type 2 diabetes (T2D) are at high risk of cardiovascular disease (CVD) and chronic kidney disease (CKD). Estimating risk and preventing CVD is a challenge in this population notably because it is underrepresented in clinical trials. Our study aims to (1) assess if T2D and haemoglobin A1c (HbA1c) are associated with the risk of CVD events and mortality in older adults, (2) develop a risk score for CVD events and mortality for older adults with T2D, (3) evaluate the comparative efficacy and safety of novel antidiabetics.

Methods and analysis

For Aim 1, we will analyse individual participant data on individuals aged ≥65 years from five cohort studies: the Optimising Therapy to Prevent Avoidable Hospital Admissions in Multimorbid Older People study; the Cohorte Lausannoise study; the Health, Aging and Body Composition study; the Health and Retirement Study; and the Survey of Health, Ageing and Retirement in Europe. We will fit flexible parametric survival models (FPSM) to assess the association of T2D and HbA1c with CVD events and mortality. For Aim 2, we will use data on individuals aged ≥65 years with T2D from the same cohorts to develop risk prediction models for CVD events and mortality using FPSM. We will assess model performance, perform internal-external cross validation, and derive a point-based risk score. For Aim 3, we will systematically search randomized controlled trials of novel antidiabetics. Network meta-analysis will be used to determine comparative efficacy in terms of CVD, CKD, and retinopathy outcomes, and safety of these drugs. Confidence in results will be judged using the CINeMA tool.

Ethics and dissemination

Aims 1 and 2 were approved by the local ethics committee (Kantonale Ethikkommission Bern); no approval is required for Aim 3. Results will be published in peer-reviewed journals and presented in scientific conferences.

Area under the curve-optimized synthesis of prediction models from a meta-analytical perspective

The number of clinical prediction models sharing the same prediction task has increased in the medical literature. However, evidence synthesis methodologies that use the results of these prediction models have not been sufficiently studied, particularly in the context of meta-analysis settings where only summary statistics are available. In particular, we consider the following situation: we want to predict an outcome Y, that is not included in our current data, while the covariate data are fully available. In addition, the summary statistics from prior studies, which share the same prediction task (i.e., the prediction of Y), are available. This study introduces a new method for synthesizing the summary results of binary prediction models reported in the prior studies using a linear predictor under a distributional assumption between the current and prior studies. The method provides an integrated predictor combining all predictors reported in the prior studies with weights. The vector of the weights is designed to achieve the hypothetical improvement of area under the receiver operating characteristic curve (AUC) on the current available data under a practical situation where there are different sets of covariates in the prior studies. We observe a counterintuitive aspect in typical situations where a part of weight components in the proposed method becomes negative. It implies that flipping the sign of the prediction results reported in each individual study would improve the overall prediction performance. Finally, numerical and real-world data analysis were conducted and showed that our method outperformed conventional methods in terms of AUC.

How do autistic people fare in adult life and can we predict it from childhood?

This study describes social, mental health, and quality of life outcomes in early adulthood, and examines childhood predictors in the Special Needs and Autism Project (SNAP), a longitudinal population-based cohort. Young autistic adults face variable but often substantial challenges across many areas of life. Prediction of outcomes is important to set expectations and could lead to the development of targeted early intervention. Autistic children were enrolled at age 12 and parents reported outcomes 11 years later when their children were age 23 (n = 121). Thirty six percent of autistic adults were in competitive employment or education and 54% had frequent contact with friends. Only 5% of autistic adults were living independently, and 37% required overnight care. Moderate or severe anxiety and depression symptoms were found for 11% and 12% of young adults, respectively. Subjective quality of life was similar to UK averages except for social relationships. Using childhood IQ, autism traits and adaptive functioning meaningful predictions can be made of living situation, employment and education and physical health. Prediction was poor for friendships, mental health outcomes and other aspects of quality of life. Our results suggest that although young autistic adults face challenges across normative, social outcomes, they may be faring better in regard to mental health or quality of life. Childhood IQ, autism traits and adaptive functioning are most useful for predicting outcomes. After accounting for these factors, childhood measurements of behavioral and emotional problems and language offered little improvement in prediction of adult outcomes.

A comparison of machine learning algorithms and traditional regression-based statistical modeling for predicting hypertension incidence in a Canadian population

Risk prediction models are frequently used to identify individuals at risk of developing hypertension. This study evaluates different machine learning algorithms and compares their predictive performance with the conventional Cox proportional hazards (PH) model to predict hypertension incidence using survival data. This study analyzed 18,322 participants on 24 candidate features from the large Alberta's Tomorrow Project (ATP) to develop different prediction models. To select the top features, we applied five feature selection methods, including two filter-based: a univariate Cox p-value and C-index; two embedded-based: random survival forest and least absolute shrinkage and selection operator (Lasso); and one constraint-based: the statistically equivalent signature (SES). Five machine learning algorithms were developed to predict hypertension incidence: penalized regression Ridge, Lasso, Elastic Net (EN), random survival forest (RSF), and gradient boosting (GB), along with the conventional Cox PH model. The predictive performance of the models was assessed using C-index. The performance of machine learning algorithms was observed, similar to the conventional Cox PH model. Average C-indexes were 0.78, 0.78, 0.78, 0.76, 0.76, and 0.77 for Ridge, Lasso, EN, RSF, GB and Cox PH, respectively. Important features associated with each model were also presented. Our study findings demonstrate little predictive performance difference between machine learning algorithms and the conventional Cox PH regression model in predicting hypertension incidence. In a moderate dataset with a reasonable number of features, conventional regression-based models perform similar to machine learning algorithms with good predictive accuracy.

Opening the black box: interpretable machine learning for predictor finding of metabolic syndrome

OBJECTIVE

The internal workings ofmachine learning algorithms are complex and considered as low-interpretation "black box" models, making it difficult for domain experts to understand and trust these complex models. The study uses metabolic syndrome (MetS) as the entry point to analyze and evaluate the application value of model interpretability methods in dealing with difficult interpretation of predictive models.

METHODS

The study collects data from a chain of health examination institution in Urumqi from 2017 ~ 2019, and performs 39,134 remaining data after preprocessing such as deletion and filling. RFE is used for feature selection to reduce redundancy; MetS risk prediction models (logistic, random forest, XGBoost) are built based on a feature subset, and accuracy, sensitivity, specificity, Youden index, and AUROC value are used to evaluate the model classification performance; post-hoc model-agnostic interpretation methods (variable importance, LIME) are used to interpret the results of the predictive model.

RESULTS

Eighteen physical examination indicators are screened out by RFE, which can effectively solve the problem of physical examination data redundancy. Random forest and XGBoost models have higher accuracy, sensitivity, specificity, Youden index, and AUROC values compared with logistic regression. XGBoost models have higher sensitivity, Youden index, and AUROC values compared with random forest. The study uses variable importance, LIME and PDP for global and local interpretation of the optimal MetS risk prediction model (XGBoost), and different interpretation methods have different insights into the interpretation of model results, which are more flexible in model selection and can visualize the process and reasons for the model to make decisions. The interpretable risk prediction model in this study can help to identify risk factors associated with MetS, and the results showed that in addition to the traditional risk factors such as overweight and obesity, hyperglycemia, hypertension, and dyslipidemia, MetS was also associated with other factors, including age, creatinine, uric acid, and alkaline phosphatase.

CONCLUSION

The model interpretability methods are applied to the black box model, which can not only realize the flexibility of model application, but also make up for the uninterpretable defects of the model. Model interpretability methods can be used as a novel means of identifying variables that are more likely to be good predictors.

Development of hypertensive complications in oocyte donation pregnancy: protocol for a systematic review and individual participant data meta-analysis (DONOR IPD)

INTRODUCTION

The assisted reproductive technique of oocyte donation (OD) is comparable to in vitro fertilisation (IVF), with the distinction of using a donated oocyte and thus involving two women. Compared with IVF and naturally conceived (NC) pregnancies, OD pregnancies have a higher risk for pregnancy complications as pregnancy-induced hypertension (PIH) and pre-eclampsia (PE). Various covariates among women pregnant by OD, however, also contribute to an increased risk for developing hypertensive complications. Therefore, we will conduct the DONation of Oocytes in Reproduction individual participant data (DONOR IPD) meta-analysis to determine the risk for the development of hypertensive complications in OD pregnancy, in comparison to autologous oocyte pregnancy (non-donor IVF/intracytoplasmic sperm injection (ICSI) and NC pregnancy). The DONOR IPD meta-analysis will provide an opportunity to adjust for confounders and perform subgroup analyses. Furthermore, IPD will be used to externally validate a prediction model for the development of PE in OD pregnancy.

METHODS AND ANALYSIS

A systematic literature search will be performed to search for studies that included women pregnant by OD, and documented on hypertensive complications in OD pregnancy. The authors from each study will be asked to collaborate and share IPD. Using the pseudoanonymised combined IPD, we will perform statistical analyses with one-stage and two-stage approaches, subgroup analyses and possibly time-to-event analyses to investigate the risk of developing hypertensive complications in OD pregnancy. Furthermore, we will formally assess a prediction model on its performance in an external validation with the use of IPD.

ETHICS AND DISSEMINATION

Ethical approval and individual patient consent will not be required in most cases since this IPD meta-analysis will use existing pseudoanonymised data from cohort studies. Results will be disseminated through peer-reviewed journals and international conferences.

PROSPERO REGISTRATION NUMBER

CRD42021267908.

Machine Learning Model to Identify Sepsis Patients in the Emergency Department: Algorithm Development and Validation

Accurate stratification of sepsis can effectively guide the triage of patient care and shared decision making in the emergency department (ED). However, previous research on sepsis identification models focused mainly on ICU patients, and discrepancies in model performance between the development and external validation datasets are rarely evaluated. The aim of our study was to develop and externally validate a machine learning model to stratify sepsis patients in the ED. We retrospectively collected clinical data from two geographically separate institutes that provided a different level of care at different time periods. The Sepsis-3 criteria were used as the reference standard in both datasets for identifying true sepsis cases. An eXtreme Gradient Boosting (XGBoost) algorithm was developed to stratify sepsis patients and the performance of the model was compared with traditional clinical sepsis tools; quick Sequential Organ Failure Assessment (qSOFA) and Systemic Inflammatory Response Syndrome (SIRS). There were 8296 patients (1752 (21%) being septic) in the development and 1744 patients (506 (29%) being septic) in the external validation datasets. The mortality of septic patients in the development and validation datasets was 13.5% and 17%, respectively. In the internal validation, XGBoost achieved an area under the receiver operating characteristic curve (AUROC) of 0.86, exceeding SIRS (0.68) and qSOFA (0.56). The performance of XGBoost deteriorated in the external validation (the AUROC of XGBoost, SIRS and qSOFA was 0.75, 0.57 and 0.66, respectively). Heterogeneity in patient characteristics, such as sepsis prevalence, severity, age, comorbidity and infection focus, could reduce model performance. Our model showed good discriminative capabilities for the identification of sepsis patients and outperformed the existing sepsis identification tools. Implementation of the ML model in the ED can facilitate timely sepsis identification and treatment. However, dataset discrepancies should be carefully evaluated before implementing the ML approach in clinical practice. This finding reinforces the necessity for future studies to perform external validation to ensure the generalisability of any developed ML approaches.

Internal-external cross-validation helped to evaluate the generalizability of prediction models in large clustered datasets

OBJECTIVE

To illustrate how to evaluate the need of complex strategies for developing generalizable prediction models in large clustered datasets.

STUDY DESIGN AND SETTING

We developed eight Cox regression models to estimate the risk of heart failure using a large population-level dataset. These models differed in the number of predictors, the functional form of the predictor effects (non-linear effects and interaction) and the estimation method (maximum likelihood and penalization). Internal-external cross-validation was used to evaluate the models' generalizability across the included general practices.

RESULTS

Among 871,687 individuals from 225 general practices, 43,987 (5.5%) developed heart failure during a median follow-up time of 5.8 years. For discrimination, the simplest prediction model yielded a good concordance statistic, which was not much improved by adopting complex strategies. Between-practice heterogeneity in discrimination was similar in all models. For calibration, the simplest model performed satisfactorily. Although accounting for non-linear effects and interaction slightly improved the calibration slope, it also led to more heterogeneity in the observed/expected ratio. Similar results were found in a second case study involving patients with stroke.

CONCLUSION

In large clustered datasets, prediction model studies may adopt internal-external cross-validation to evaluate the generalizability of competing models, and to identify promising modelling strategies.

Progress and challenges of network meta-analysis

In the past years, network meta-analysis (NMA) has been widely used among clinicians, guideline makers, and health technology assessment agencies and has played an important role in clinical decision-making and guideline development. To inform further development of NMAs, we conducted a bibliometric analysis to assess the current status of published NMA methodological studies, summarized the methodological progress of seven types of NMAs, and discussed the current challenges of NMAs.

Assessment of predictive performance of caries risk assessment models based on a systematic review and meta-analysis

OBJECTIVES

To assess the predictive performance of caries risk assessment (CRA) models for prediction of caries increment for individuals based on a systematic review and meta-analyses.

DATA/SOURCES

We included external validation studies assessing the predictive performance of CRA models for prediction of caries increment for individuals, using discrimination and calibration as the outcome parameters. PubMed, EMBASE, and CINAHL were searched electronically on 10th September 2020 to identify prediction modeling studies on external validation of CRA models. The risk of bias of the included studies was assessed using the Prediction model Risk Of Bias ASsessment Tool (PROBAST).

STUDY SELECTION

A total of 22 studies with seven different CRA models were included. As for full Cariogram, the pooled area under the receiver operating characteristic curve (AUC) was 0.78 (95 %CI: 0.68; 0.85) based on eight studies regardless of the risk of bias levels, and 0.82 (95 %CI: 0.58; 0.93) based on four studies with low risk of bias only. The pooled observed: expected ratio (O:E ratio) of full Cariogram was 0.91 (95 %CI: 0.72; 1.14) based on 12 studies regardless of the risk of bias levels, and 0.89 (95 %CI: 0.71; 1.12) based on five studies with low risk of bias only. As for reduced Cariogram, the pooled AUC was 0.72 (95 %CI: 0.67; 0.77) based on six studies regardless of the risk of bias levels, and 0.74 (95 %CI: 0.45; 0.91) based on two studies with low risk of bias only. The pooled O:E ratio of reduced Cariogram was 0.84 (95 %CI: 0.59; 1.18) based on six studies regardless of the risk of bias levels, and 1.05 (95 %CI: 0.43; 2.59) based on two studies with low risk of bias only. Based on an insufficient number of studies for the other CRA models, the pooled AUCs ranged from 0.50 to 0.88, while the pooled O:E ratio ranged from 0.38 to 1.00.

CONCLUSION

The average predictive performance of both full and reduced Cariogram seems to be acceptable. However, the evidence from research does not allow a firm conclusion on the performance of the other included CRA models, due to the insufficient number of high-quality studies.

CLINICAL SIGNIFICANCE

Both full and reduced Cariogram were found to be reliable CRA models for prediction of caries increment in clinical practices for dental patients and communities for general populations. The reduced Cariogram showed better predictive performance and less burden in terms of time and resources to individuals than the full Cariogram. Therefore, the reduced Cariogram could be more recommended than the full Cariogram.

Developing more generalizable prediction models from pooled studies and large clustered data sets

Prediction models often yield inaccurate predictions for new individuals. Large data sets from pooled studies or electronic healthcare records may alleviate this with an increased sample size and variability in sample characteristics. However, existing strategies for prediction model development generally do not account for heterogeneity in predictor‐outcome associations between different settings and populations. This limits the generalizability of developed models (even from large, combined, clustered data sets) and necessitates local revisions. We aim to develop methodology for producing prediction models that require less tailoring to different settings and populations. We adopt internal‐external cross‐validation to assess and reduce heterogeneity in models' predictive performance during the development. We propose a predictor selection algorithm that optimizes the (weighted) average performance while minimizing its variability across the hold‐out clusters (or studies). Predictors are added iteratively until the estimated generalizability is optimized. We illustrate this by developing a model for predicting the risk of atrial fibrillation and updating an existing one for diagnosing deep vein thrombosis, using individual participant data from 20 cohorts (N = 10 873) and 11 diagnostic studies (N = 10 014), respectively. Meta‐analysis of calibration and discrimination performance in each hold‐out cluster shows that trade‐offs between average and heterogeneity of performance occurred. Our methodology enables the assessment of heterogeneity of prediction model performance during model development in multiple or clustered data sets, thereby informing researchers on predictor selection to improve the generalizability to different settings and populations, and reduce the need for model tailoring. Our methodology has been implemented in the R package metamisc.

Evaluation and improvement of the National Early Warning Score (NEWS2) for COVID-19: a multi-hospital study

BACKGROUND

The National Early Warning Score (NEWS2) is currently recommended in the UK for the risk stratification of COVID-19 patients, but little is known about its ability to detect severe cases. We aimed to evaluate NEWS2 for the prediction of severe COVID-19 outcome and identify and validate a set of blood and physiological parameters routinely collected at hospital admission to improve upon the use of NEWS2 alone for medium-term risk stratification.

METHODS

Training cohorts comprised 1276 patients admitted to King's College Hospital National Health Service (NHS) Foundation Trust with COVID-19 disease from 1 March to 30 April 2020. External validation cohorts included 6237 patients from five UK NHS Trusts (Guy's and St Thomas' Hospitals, University Hospitals Southampton, University Hospitals Bristol and Weston NHS Foundation Trust, University College London Hospitals, University Hospitals Birmingham), one hospital in Norway (Oslo University Hospital), and two hospitals in Wuhan, China (Wuhan Sixth Hospital and Taikang Tongji Hospital). The outcome was severe COVID-19 disease (transfer to intensive care unit (ICU) or death) at 14 days after hospital admission. Age, physiological measures, blood biomarkers, sex, ethnicity, and comorbidities (hypertension, diabetes, cardiovascular, respiratory and kidney diseases) measured at hospital admission were considered in the models.

RESULTS

A baseline model of 'NEWS2 + age' had poor-to-moderate discrimination for severe COVID-19 infection at 14 days (area under receiver operating characteristic curve (AUC) in training cohort = 0.700, 95% confidence interval (CI) 0.680, 0.722; Brier score = 0.192, 95% CI 0.186, 0.197). A supplemented model adding eight routinely collected blood and physiological parameters (supplemental oxygen flow rate, urea, age, oxygen saturation, C-reactive protein, estimated glomerular filtration rate, neutrophil count, neutrophil/lymphocyte ratio) improved discrimination (AUC = 0.735; 95% CI 0.715, 0.757), and these improvements were replicated across seven UK and non-UK sites. However, there was evidence of miscalibration with the model tending to underestimate risks in most sites.

CONCLUSIONS

NEWS2 score had poor-to-moderate discrimination for medium-term COVID-19 outcome which raises questions about its use as a screening tool at hospital admission. Risk stratification was improved by including readily available blood and physiological parameters measured at hospital admission, but there was evidence of miscalibration in external sites. This highlights the need for a better understanding of the use of early warning scores for COVID.

The diagnosis of tuberculous meningitis in adults and adolescents: protocol for a systematic review and individual patient data meta-analysis to inform a multivariable prediction model

Background: Tuberculous meningitis (TBM) is the most lethal and disabling form of tuberculosis. Delayed diagnosis and treatment, which is a risk factor for poor outcome, is caused in part by lack of availability of diagnostic tests that are both rapid and accurate. Several attempts have been made to develop clinical scoring systems to fill this gap, but none have performed sufficiently well to be broadly implemented. We aim to identify and validate a set of clinical predictors that accurately classify TBM using individual patient data (IPD) from published studies. Methods: We will perform a systematic review and obtain IPD from studies published from the year 1990 which undertook diagnostic testing for TBM in adolescents or adults using at least one of, microscopy for acid-fast bacilli, commercial nucleic acid amplification test for Mycobacterium tuberculosis or mycobacterial culture of cerebrospinal fluid.  Clinical data that have previously been shown to be associated with TBM, and can inform the final diagnosis, will be requested. The data-set will be divided into training and test/validation data-sets for model building. A predictive logistic model will be built using a training set with patients with definite TBM and no TBM. Should it be warranted, factor analysis may be employed, depending on evidence for multicollinearity or the case for including latent variables in the model. Discussion: We will systematically identify and extract key clinical parameters associated with TBM from published studies and use a 'big data' approach to develop and validate a clinical prediction model with enhanced generalisability. The final model will be made available through a smartphone application. Further work will be external validation of the model and test of efficacy in a randomised controlled trial.

The diagnosis of tuberculous meningitis in adults and adolescents: protocol for a systematic review and individual patient data meta-analysis to inform a multivariable prediction model

Background: Tuberculous meningitis (TBM) is the most lethal and disabling form of tuberculosis. Delayed diagnosis and treatment, which is a risk factor for poor outcome, is caused in part by lack of availability of diagnostic tests that are both rapid and accurate. Several attempts have been made to develop clinical scoring systems to fill this gap, but none have performed sufficiently well to be broadly implemented. We aim to identify and validate a set of clinical predictors that accurately classify TBM using individual patient data (IPD) from published studies. Methods: We will perform a systematic review and obtain IPD from studies published from the year 1990 which undertook diagnostic testing for TBM in adolescents or adults using at least one of, microscopy for acid-fast bacilli, commercial nucleic acid amplification test for Mycobacterium tuberculosis or mycobacterial culture of cerebrospinal fluid.  Clinical data that have previously been shown to be associated with TBM, and can inform the final diagnosis, will be requested. The data-set will be divided into training and test/validation data-sets for model building. A predictive logistic model will be built using a training set with patients with definite TBM and no TBM. Should it be warranted, factor analysis may be employed, depending on evidence for multicollinearity or the case for including latent variables in the model. Discussion: We will systematically identify and extract key clinical parameters associated with TBM from published studies and use a 'big data' approach to develop and validate a clinical prediction model with enhanced generalisability. The final model will be made available through a smartphone application. Further work will be external validation of the model and test of efficacy in a randomised controlled trial.

Protocol for the development and validation of a risk prediction model for stillbirths from 35 weeks gestation in Australia

BACKGROUND

Despite advances in the care of women and their babies in the past century, an estimated 1.7 million babies are born still each year throughout the world. A robust method to estimate a pregnant woman's individualized risk of late-pregnancy stillbirth is needed to inform decision-making around the timing of birth to reduce the risk of stillbirth from 35 weeks of gestation in Australia, a high-resource setting.

METHODS

This is a protocol for a cross-sectional study of all late-pregnancy births in Australia (2005-2015) from 35 weeks of gestation including 5188 stillbirths among 3.1 million births at an estimated rate of 1.7 stillbirths per 1000 births. A multivariable logistic regression model will be developed in line with current Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) guidelines to estimate the gestation-specific probability of stillbirth with prediction intervals. Candidate predictors were identified from systematic reviews and clinical consultation and will be described through univariable regression analysis. To generate a final model, elimination by backward stepwise multivariable logistic regression will be performed. The model will be internally validated using bootstrapping with 1000 repetitions and externally validated using a temporally unique dataset. Overall model performance will be assessed with R², calibration, and discrimination. Calibration will be reported using a calibration plot with 95% confidence intervals (α = 0.05). Discrimination will be measured by the C-statistic and area underneath the receiver-operator curves. Clinical usefulness will be reported as positive and negative predictive values, and a decision curve analysis will be considered.

DISCUSSION

A robust method to predict a pregnant woman's individualized risk of late-pregnancy stillbirth is needed to inform timely, appropriate care to reduce stillbirth. Among existing prediction models designed for obstetric use, few have been subject to internal and external validation and many fail to meet recommended reporting standards. In developing a risk prediction model for late-gestation stillbirth with both providers and pregnant women in mind, we endeavor to develop a validated model for clinical use in Australia that meets current reporting standards.

Predicting the treatment response of certolizumab for individual adult patients with rheumatoid arthritis: protocol for an individual participant data meta-analysis

BACKGROUND

A model that can predict treatment response for a patient with specific baseline characteristics would help decision-making in personalized medicine. The aim of the study is to develop such a model in the treatment of rheumatoid arthritis (RA) patients who receive certolizumab (CTZ) plus methotrexate (MTX) therapy, using individual participant data meta-analysis (IPD-MA).

METHODS

We will search Cochrane CENTRAL, PubMed, and Scopus as well as clinical trial registries, drug regulatory agency reports, and the pharmaceutical company websites from their inception onwards to obtain randomized controlled trials (RCTs) investigating CTZ plus MTX compared with MTX alone in treating RA. We will request the individual-level data of these trials from an independent platform (http://vivli.org). The primary outcome is efficacy defined as achieving either remission (based on ACR-EULAR Boolean or index-based remission definition) or low disease activity (based on either of the validated composite disease activity measures). The secondary outcomes include ACR50 (50% improvement based on ACR core set variables) and adverse events. We will use a two-stage approach to develop the prediction model. First, we will construct a risk model for the outcomes via logistic regression to estimate the baseline risk scores. We will include baseline demographic, clinical, and biochemical features as covariates for this model. Next, we will develop a meta-regression model for treatment effects, in which the stage 1 risk score will be used both as a prognostic factor and as an effect modifier. We will calculate the probability of having the outcome for a new patient based on the model, which will allow estimation of the absolute and relative treatment effect. We will use R for our analyses, except for the second stage which will be performed in a Bayesian setting using R2Jags.

DISCUSSION

This is a study protocol for developing a model to predict treatment response for RA patients receiving CTZ plus MTX in comparison with MTX alone, using a two-stage approach based on IPD-MA. The study will use a new modeling approach, which aims at retaining the statistical power. The model may help clinicians individualize treatment for particular patients.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO registration number pending (ID#157595).

Individual participant data (IPD) meta-analysis of psychological relapse prevention interventions versus control for patients in remission from depression: a protocol

INTRODUCTION

Psychological interventions and antidepressant medication can be effective interventions to prevent depressive relapse for patients currently in remission of depression. Less is known about overall factors that predict or moderate treatment response for patients receiving a psychological intervention for recurrent depression. This is a protocol for an individual participant data (IPD) meta-analysis which aims to assess predictors and moderators of relapse or recurrence for patients currently in remission from depression.

METHODS AND ANALYSIS

Searches of PubMed, PsycINFO, Embase and Cochrane Central Register of Controlled Trials were completed on 13 October 2019. Study extractions and risk of bias assessments have been completed. Study authors will be asked to contribute IPD. Standard aggregate meta-analysis and IPD analysis will be conducted, and the outcomes will be compared with assess whether results differ between studies supplying data and those that did not. IPD files of individual data will be merged and variables homogenised where possible for consistency. IPD will be analysed via Cox regression and one and two-stage analyses will be conducted.

ETHICS AND DISSEMINATION

The results will be published in peer review journals and shared in a policy briefing as well as accessible formats and shared with a range of stakeholders. The results will inform patients and clinicians and researchers about our current understanding of more personalised ways to prevent a depressive relapse. No local ethics approval was necessary following consultation with the legal department. Guidance on patient data storage and management will be adhered to.

PROSPERO REGISTRATION NUMBER

CRD42019127844.

Examining Bias and Reporting in Oral Health Prediction Modeling Studies

Recent efforts to improve the reliability and efficiency of scientific research have caught the attention of researchers conducting prediction modeling studies (PMSs). Use of prediction models in oral health has become more common over the past decades for predicting the risk of diseases and treatment outcomes. Risk of bias and insufficient reporting present challenges to the reproducibility and implementation of these models. A recent tool for bias assessment and a reporting guideline—PROBAST (Prediction Model Risk of Bias Assessment Tool) and TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis)—have been proposed to guide researchers in the development and reporting of PMSs, but their application has been limited. Following the standards proposed in these tools and a systematic review approach, a literature search was carried out in PubMed to identify oral health PMSs published in dental, epidemiologic, and biostatistical journals. Risk of bias and transparency of reporting were assessed with PROBAST and TRIPOD. Among 2,881 papers identified, 34 studies containing 58 models were included. The most investigated outcomes were periodontal diseases (42%) and oral cancers (30%). Seventy-five percent of the studies were susceptible to at least 4 of 20 sources of bias, including measurement error in predictors ( n = 12) and/or outcome ( n = 7), omitting samples with missing data ( n = 10), selecting variables based on univariate analyses ( n = 9), overfitting ( n = 13), and lack of model performance assessment ( n = 24). Based on TRIPOD, at least 5 of 31 items were inadequately reported in 95% of the studies. These items included sampling approaches ( n = 15), participant eligibility criteria ( n = 6), and model-building procedures ( n = 16). There was a general lack of transparent reporting and identification of bias across the studies. Application of the recommendations proposed in PROBAST and TRIPOD can benefit future research and improve the reproducibility and applicability of prediction models in oral health.

The Predictive Approaches to Treatment effect Heterogeneity (PATH) Statement: Explanation and Elaboration

The PATH (Predictive Approaches to Treatment effect Heterogeneity) Statement was developed to promote the conduct of, and provide guidance for, predictive analyses of heterogeneity of treatment effects (HTE) in clinical trials. The goal of predictive HTE analysis is to provide patient-centered estimates of outcome risk with versus without the intervention, taking into account all relevant patient attributes simultaneously, to support more personalized clinical decision making than can be made on the basis of only an overall average treatment effect. The authors distinguished 2 categories of predictive HTE approaches (a "risk-modeling" and an "effect-modeling" approach) and developed 4 sets of guidance statements: criteria to determine when risk-modeling approaches are likely to identify clinically meaningful HTE, methodological aspects of risk-modeling methods, considerations for translation to clinical practice, and considerations and caveats in the use of effect-modeling approaches. They discuss limitations of these methods and enumerate research priorities for advancing methods designed to generate more personalized evidence. This explanation and elaboration document describes the intent and rationale of each recommendation and discusses related analytic considerations, caveats, and reservations.

Evidence-Based Utilization of Prognostic Prediction Models in Cardiovascular Medicine

Prediction models are combinations of predictors to assess the risks of specific endpoints such as the presence or prognosis of a disease. Many novel predictors have been developed, modelling techniques have been evolving, and prediction models are currently abundant in the medical literature, especially in cardiovascular medicine, but evidence is still lacking regarding how to use them. Recent methodological advances in systematic reviews and meta-analysis have enabled systematic evaluation of prediction model studies and quantitative analysis to identify determinants of model performance. Knowing what is critical to model performance, under what circumstances model performance remains adequate, and when a model might require further adjustment and improvement will facilitate effective utilization of prediction models and will enhance diagnostic and prognostic accuracy in clinical practice. In this review article, we provide a current methodological overview of the attempts to implement evidence-based utilization of prognostic prediction models for all potential model users, including patients and their families, health-care providers, administrators, researchers, guideline developers and policy makers.

Multi-cohort modeling strategies for scalable globally accessible prostate cancer risk tools

BACKGROUND

Online clinical risk prediction tools built on data from multiple cohorts are increasingly being utilized for contemporary doctor-patient decision-making and validation. This report outlines a comprehensive data science strategy for building such tools with application to the Prostate Biopsy Collaborative Group prostate cancer risk prediction tool.

METHODS

We created models for high-grade prostate cancer risk using six established risk factors. The data comprised 8492 prostate biopsies collected from ten institutions, 2 in Europe and 8 across North America. We calculated area under the receiver operating characteristic curve (AUC) for discrimination, the Hosmer-Lemeshow test statistic (HLS) for calibration and the clinical net benefit at risk threshold 15%. We implemented several internal cross-validation schemes to assess the influence of modeling method and individual cohort on validation performance.

RESULTS

High-grade disease prevalence ranged from 18% in Zurich (1863 biopsies) to 39% in UT Health San Antonio (899 biopsies). Visualization revealed outliers in terms of risk factors, including San Juan VA (51% abnormal digital rectal exam), Durham VA (63% African American), and Zurich (2.8% family history). Exclusion of any cohort did not significantly affect the AUC or HLS, nor did the choice of prediction model (pooled, random-effects, meta-analysis). Excluding the lowest-prevalence Zurich cohort from training sets did not statistically significantly change the validation metrics for any of the individual cohorts, except for Sunnybrook, where the effect on the AUC was minimal. Therefore the final multivariable logistic model was built by pooling the data from all cohorts using logistic regression. Higher prostate-specific antigen and age, abnormal digital rectal exam, African ancestry and a family history of prostate cancer increased risk of high-grade prostate cancer, while a history of a prior negative prostate biopsy decreased risk (all p-values < 0.004).

CONCLUSIONS

We have outlined a multi-cohort model-building internal validation strategy for developing globally accessible and scalable risk prediction tools.

Assessment of heterogeneity in an individual participant data meta-analysis of prediction models: An overview and illustration

Clinical prediction models aim to provide estimates of absolute risk for a diagnostic or prognostic endpoint. Such models may be derived from data from various studies in the context of a meta‐analysis. We describe and propose approaches for assessing heterogeneity in predictor effects and predictions arising from models based on data from different sources. These methods are illustrated in a case study with patients suffering from traumatic brain injury, where we aim to predict 6‐month mortality based on individual patient data using meta‐analytic techniques (15 studies, n = 11 022 patients). The insights into various aspects of heterogeneity are important to develop better models and understand problems with the transportability of absolute risk predictions.

Evidence synthesis in prognosis research

Over the past few years, evidence synthesis has become essential to investigate and improve the generalizability of medical research findings. This strategy often involves a meta-analysis to formally summarize quantities of interest, such as relative treatment effect estimates. The use of meta-analysis methods is, however, less straightforward in prognosis research because substantial variation exists in research objectives, analysis methods and the level of reported evidence. We present a gentle overview of statistical methods that can be used to summarize data of prognostic factor and prognostic model studies. We discuss how aggregate data, individual participant data, or a combination thereof can be combined through meta-analysis methods. Recent examples are provided throughout to illustrate the various methods.

Understanding the relation between Zika virus infection during pregnancy and adverse fetal, infant and child outcomes: a protocol for a systematic review and individual participant data meta-analysis of longitudinal studies of pregnant women and their infants and children

INTRODUCTION

Zika virus (ZIKV) infection during pregnancy is a known cause of microcephaly and other congenital and developmental anomalies. In the absence of a ZIKV vaccine or prophylactics, principal investigators (PIs) and international leaders in ZIKV research have formed the ZIKV Individual Participant Data (IPD) Consortium to identify, collect and synthesise IPD from longitudinal studies of pregnant women that measure ZIKV infection during pregnancy and fetal, infant or child outcomes.

METHODS AND ANALYSIS

We will identify eligible studies through the ZIKV IPD Consortium membership and a systematic review and invite study PIs to participate in the IPD meta-analysis (IPD-MA). We will use the combined dataset to estimate the relative and absolute risk of congenital Zika syndrome (CZS), including microcephaly and late symptomatic congenital infections; identify and explore sources of heterogeneity in those estimates and develop and validate a risk prediction model to identify the pregnancies at the highest risk of CZS or adverse developmental outcomes. The variable accuracy of diagnostic assays and differences in exposure and outcome definitions means that included studies will have a higher level of systematic variability, a component of measurement error, than an IPD-MA of studies of an established pathogen. We will use expert testimony, existing internal and external diagnostic accuracy validation studies and laboratory external quality assessments to inform the distribution of measurement error in our models. We will apply both Bayesian and frequentist methods to directly account for these and other sources of uncertainty.

ETHICS AND DISSEMINATION

The IPD-MA was deemed exempt from ethical review. We will convene a group of patient advocates to evaluate the ethical implications and utility of the risk stratification tool. Findings from these analyses will be shared via national and international conferences and through publication in open access, peer-reviewed journals.

TRIAL REGISTRATION NUMBER

PROSPERO International prospective register of systematic reviews (CRD42017068915).

Individualized Prediction of Transition to Psychosis in 1,676 Individuals at Clinical High Risk: Development and Validation of a Multivariable Prediction Model Based on Individual Patient Data Meta-Analysis

Background: The Clinical High Risk state for Psychosis (CHR-P) has become the cornerstone of modern preventive psychiatry. The next stage of clinical advancements rests on the ability to formulate a more accurate prognostic estimate at the individual subject level. Individual Participant Data Meta-Analyses (IPD-MA) are robust evidence synthesis methods that can also offer powerful approaches to the development and validation of personalized prognostic models. The aim of the study was to develop and validate an individualized, clinically based prognostic model for forecasting transition to psychosis from a CHR-P stage. Methods: A literature search was performed between January 30, 2016, and February 6, 2016, consulting PubMed, Psychinfo, Picarta, Embase, and ISI Web of Science, using search terms ("ultra high risk" OR "clinical high risk" OR "at risk mental state") AND [(conver* OR transition* OR onset OR emerg* OR develop*) AND psychosis] for both longitudinal and intervention CHR-P studies. Clinical knowledge was used to a priori select predictors: age, gender, CHR-P subgroup, the severity of attenuated positive psychotic symptoms, the severity of attenuated negative psychotic symptoms, and level of functioning at baseline. The model, thus, developed was validated with an extended form of internal validation. Results: Fifteen of the 43 studies identified agreed to share IPD, for a total sample size of 1,676. There was a high level of heterogeneity between the CHR-P studies with regard to inclusion criteria, type of assessment instruments, transition criteria, preventive treatment offered. The internally validated prognostic performance of the model was higher than chance but only moderate [Harrell's C-statistic 0.655, 95% confidence interval (CIs), 0.627-0.682]. Conclusion: This is the first IPD-MA conducted in the largest samples of CHR-P ever collected to date. An individualized prognostic model based on clinical predictors available in clinical routine was developed and internally validated, reaching only moderate prognostic performance. Although personalized risk prediction is of great value in the clinical practice, future developments are essential, including the refinement of the prognostic model and its external validation. However, because of the current high diagnostic, prognostic, and therapeutic heterogeneity of CHR-P studies, IPD-MAs in this population may have an limited intrinsic power to deliver robust prognostic models.

Towards more Accessible Precision Medicine: Building a more Transferable Machine Learning Model to Support Prognostic Decisions for Micro- and Macrovascular Complications of Type 2 Diabetes Mellitus

Although machine learning models are increasingly being developed for clinical decision support for patients with type 2 diabetes, the adoption of these models into clinical practice remains limited. Currently, machine learning (ML) models are being constructed on local healthcare systems and are validated internally with no expectation that they would validate externally and thus, are rarely transferrable to a different healthcare system. In this work, we aim to demonstrate that (1) even a complex ML model built on a national cohort can be transferred to two local healthcare systems, (2) while a model constructed on a local healthcare system's cohort is difficult to transfer; (3) we examine the impact of training cohort size on the transferability; and (4) we discuss criteria for external validity. We built a model using our previously published Multi-Task Learning-based methodology on a national cohort extracted from OptumLabs® Data Warehouse and transferred the model to two local healthcare systems (i.e., University of Minnesota Medical Center and Mayo Clinic) for external evaluation. The model remained valid when applied to the local patient populations and performed as well as locally constructed models (concordance: .73-.92), demonstrating transferability. The performance of the locally constructed models reduced substantially when applied to each other's healthcare system (concordance: .62-.90). We believe that our modeling approach, in which a model is learned from a national cohort and is externally validated, produces a transferable model, allowing patients at smaller healthcare systems to benefit from precision medicine.

The diagnosis of tuberculous meningitis in adults and adolescents: protocol for a systematic review and individual patient data meta-analysis to inform a multivariable prediction model

Background: Tuberculous meningitis (TBM) is the most lethal and disabling form of tuberculosis. Delayed diagnosis and treatment, which is a risk factor for poor outcome, is caused in part by lack of availability of diagnostic tests that are both rapid and accurate. Several attempts have been made to develop clinical scoring systems to fill this gap, but none have performed sufficiently well to be broadly implemented. We aim to identify and validate a set of clinical predictors that accurately classify TBM using individual patient data (IPD) from published studies. Methods: We will perform a systematic review and obtain IPD from studies published from the year 1990 which undertook diagnostic testing for TBM in adolescents or adults using at least one of, microscopy for acid-fast bacilli, commercial nucleic acid amplification test for Mycobacterium tuberculosis or mycobacterial culture of cerebrospinal fluid.  Clinical data that have previously been shown to be associated with TBM, and can inform the final diagnosis, will be requested. The data-set will be divided into training and test/validation data-sets for model building. A predictive logistic model will be built using a training set with patients with definite TBM and no TBM. Should it be warranted, factor analysis may be employed, depending on evidence for multicollinearity or the case for including latent variables in the model. Discussion: We will systematically identify and extract key clinical parameters associated with TBM from published studies and use a ‘big data’ approach to develop and validate a clinical prediction model with enhanced generalisability. The final model will be made available through a smartphone application. Further work will be external validation of the model and test of efficacy in a randomised controlled trial.

PROBAST: A Tool to Assess Risk of Bias and Applicability of Prediction Model Studies: Explanation and Elaboration

Prediction models in health care use predictors to estimate for an individual the probability that a condition or disease is already present (diagnostic model) or will occur in the future (prognostic model). Publications on prediction models have become more common in recent years, and competing prediction models frequently exist for the same outcome or target population. Health care providers, guideline developers, and policymakers are often unsure which model to use or recommend, and in which persons or settings. Hence, systematic reviews of these studies are increasingly demanded, required, and performed. A key part of a systematic review of prediction models is examination of risk of bias and applicability to the intended population and setting. To help reviewers with this process, the authors developed PROBAST (Prediction model Risk Of Bias ASsessment Tool) for studies developing, validating, or updating (for example, extending) prediction models, both diagnostic and prognostic. PROBAST was developed through a consensus process involving a group of experts in the field. It includes 20 signaling questions across 4 domains (participants, predictors, outcome, and analysis). This explanation and elaboration document describes the rationale for including each domain and signaling question and guides researchers, reviewers, readers, and guideline developers in how to use them to assess risk of bias and applicability concerns. All concepts are illustrated with published examples across different topics. The latest version of the PROBAST checklist, accompanying documents, and filled-in examples can be downloaded from www.probast.org.

Predicting 7-year mortality for use with evidence-based guidelines for Prostate-Specific Antigen (PSA) testing: findings from a large prospective study of 123 697 Australian men

OBJECTIVES

To develop and validate a prediction model for short-term mortality in Australian men aged ≥45years, using age and self-reported health variables, for use when implementing the Australian Clinical Practice Guidelines for Prostate-Specific Antigen (PSA) Testing and Early Management of Test-Detected Prostate Cancer. Implementation of one of the Guideline recommendations requires an estimate of 7-year mortality.

DESIGN

Prospective cohort study using questionnaire data linked to mortality data.

SETTING

Men aged ≥45years randomly sampled from the general population of New South Wales, Australia, participating in the 45 and Up Study.

PARTICIPANTS

123 697 men who completed the baseline postal questionnaire (distributed from 1 January 2006 to 31 December 2008) and gave informed consent for follow-up through linkage of their data to population health databases.

PRIMARY OUTCOME MEASURES

The primary outcome was all-cause mortality.

RESULTS

12 160 died during follow-up (median=5.9 years). Following age-adjustment, self-reported health was the strongest predictor of all-cause mortality (C-index: 0.827; 95% CI 0.824 to 0.831). Three prediction models for all-cause mortality were validated, with predictors: Model-1: age group and self-rated health; Model-2: variables common to the 45 and Up Study and the Australian Health Survey and subselected using stepwise regression and Model-3: all variables selected using stepwise regression. Final predictions calibrated well with observed all-cause mortality rates. The 90th percentile for the 7-year mortality risks ranged from 1.92% to 83.94% for ages 45-85 years.

CONCLUSIONS

We developed prediction scores for short-term mortality using age and self-reported health measures and validated the scores against national mortality rates. Along with age, simple measures such as self-rated health, which can be easily obtained without physical examination, were strong predictors of all-cause mortality in the 45 and Up Study. Seven-year mortality risk estimates from Model-3 suggest that the impact of the mortality risk prediction tool on men's decision making would be small in the recommended age (50-69 years) for PSA testing, but it may discourage testing at older ages.

Efficacy and Acceptability of Interventions for Attenuated Positive Psychotic Symptoms in Individuals at Clinical High Risk of Psychosis: A Network Meta-Analysis

Background: Attenuated positive psychotic symptoms represent the defining features of the clinical high-risk for psychosis (CHR-P) criteria. The effectiveness of each available treatment for reducing attenuated positive psychotic symptoms remains undetermined. This network meta-analysis (NMA) investigates the consistency and magnitude of the effects of treatments on attenuated positive psychotic symptoms in CHR-P individuals, weighting the findings for acceptability. Methods: Web of Science (MEDLINE), PsycInfo, CENTRAL and unpublished/gray literature were searched up to July 18, 2017. Randomized controlled trials in CHR-P individuals, comparing at least two interventions and reporting on attenuated positive psychotic symptoms at follow-up were included, following PRISMA guidelines. The primary outcome (efficacy) was level of attenuated positive psychotic symptoms at 6 and 12 months; effect sizes reported as standardized mean difference (SMD) and 95% CIs in mean follow-up scores between two compared interventions. The secondary outcome was treatment acceptability [reported as odds ratio (OR)]. NMAs were conducted for both primary and secondary outcomes. Treatments were cluster-ranked by surface under the cumulative ranking curve values for efficacy and acceptability. Assessments of biases, assumptions, sensitivity analyses and complementary pairwise meta-analyses for the primary outcome were also conducted. Results: Overall, 1,707 patients from 14 studies (57% male, mean age = 20) were included, representing the largest evidence synthesis of the effect of preventive treatments on attenuated positive psychotic symptoms to date. In the NMA for efficacy, ziprasidone + Needs-Based Intervention (NBI) was found to be superior to NBI (SMD = -1.10, 95% CI -2.04 to -0.15), Cognitive Behavioral Therapy-French and Morrison protocol (CBT-F) + NBI (SMD = -1.03, 95% CI -2.05 to -0.01), and risperidone + CBT-F + NBI (SMD = -1.18, 95% CI -2.29 to -0.07) at 6 months. However, these findings did not survive sensitivity analyses. For acceptability, aripiprazole + NBI was significantly more acceptable than olanzapine + NBI (OR = 3.73; 95% CI 1.01 to 13.81) at 12 months only. No further significant NMA effects were observed at 6 or 12 months. The results were not affected by inconsistency or evident small-study effects, but only two studies had an overall low risk of bias. Conclusion: On the basis of the current literature, there is no robust evidence to favor any specific intervention for improving attenuated positive psychotic symptoms in CHR-P individuals.

Lifestyle index for mortality prediction using multiple ageing cohorts in the USA, UK and Europe

Current mortality prediction indexes are mainly based on functional morbidity and comorbidity, with limited information for risk prevention. This study aimed to develop and validate a modifiable lifestyle-based mortality predication index for older adults. Data from 51,688 participants (56% women) aged ≥50 years in 2002 Health and Retirement Study, 2002 English Longitudinal Study of Ageing and 2004 Survey of Health Ageing and Retirement in Europe were used to estimate coefficients of the index with cohort-stratified Cox regression. Models were validated across studies and compared to the Lee index (having comorbid and morbidity predictors). Over an average of 11-year follow-up, 10,240 participants died. The lifestyle index includes smoking, drinking, exercising, sleep quality, BMI, sex and age; showing adequate model performance in internal validation (C-statistic 0.79; D-statistic 1.94; calibration slope 1.13) and in all combinations of internal-external cross-validation. It outperformed Lee index (e.g. differences in C-statistic = 0.01, D-statistic = 0.17, P < 0.001) consistently across health status. The lifestyle index stratified participants into varying mortality risk groups, with those in the top quintile having 13.5% excess absolute mortality risk over 10 years than those in the bottom 50th centile. Our lifestyle index with easy-assessable behavioural factors and improved generalizability may maximize its usability for personalized risk management.

The Collaborative IPD of Sleep and Stillbirth (Cribss): is maternal going-to-sleep position a risk factor for late stillbirth and does maternal sleep position interact with fetal vulnerability? An individual participant data meta-analysis study protocol

INTRODUCTION

Accumulating evidence has shown an association between maternal supine going-to-sleep position and stillbirth in late pregnancy. Advising women not to go-to-sleep on their back can potentially reduce late stillbirth rate by 9%. However, the association between maternal right-sided going-to-sleep position and stillbirth is inconsistent across studies. Furthermore, individual studies are underpowered to investigate interactions between maternal going-to-sleep position and fetal vulnerability, which is potentially important for producing clear and tailored public health messages on safe going-to-sleep position. We will use individual participant data (IPD) from existing studies to assess whether right-side and supine going-to-sleep positions are independent risk factors for late stillbirth and to test the interaction between going-to-sleep position and fetal vulnerability.

METHODS AND ANALYSIS

An IPD meta-analysis approach will be used using the Cochrane Collaboration-endorsed methodology. We will identify case-control and prospective cohort studies and randomised trials which collected maternal going-to-sleep position data and pregnancy outcome data that included stillbirth. The primary outcome is stillbirth. A one stage procedure meta-analysis, stratified by study with adjustment of a priori confounders will be carried out.

ETHICS AND DISSEMINATION

The IPD meta-analysis has obtained central ethics approval from the New Zealand Health and Disability Ethics Committee, ref: NTX/06/05/054/AM06. Individual studies should also have ethical approval from relevant local ethics committees. Interpretation of the results will be discussed with consumer representatives. Results of the study will be published in peer-reviewed journals and presented at international conferences.

PROSPERO REGISTRATION NUMBER

CRD42017047703.

Ruling out pulmonary embolism across different subgroups of patients and healthcare settings: protocol for a systematic review and individual patient data meta-analysis (IPDMA)

BACKGROUND

Diagnosing pulmonary embolism in suspected patients is notoriously difficult as signs and symptoms are non-specific. Different diagnostic strategies have been developed, usually combining clinical probability assessment with D-dimer testing. However, their predictive performance differs across different healthcare settings, patient subgroups, and clinical presentation, which are currently not accounted for in the available diagnostic approaches.

METHODS

This is a protocol for a large diagnostic individual patient data meta-analysis (IPDMA) of currently available diagnostic studies in the field of pulmonary embolism. We searched MEDLINE (search date January 1, 1995, till August 25, 2016) to retrieve all primary diagnostic studies that had evaluated diagnostic strategies for pulmonary embolism. Two authors independently screened titles, abstracts, and subsequently full-text articles for eligibility from 3145 individual studies. A total of 40 studies were deemed eligible for inclusion into our IPDMA set, and principal investigators from these studies were invited to participate in a meeting at the 2017 conference from the International Society on Thrombosis and Haemostasis. All authors agreed on data sharing and participation into this project. The process of data collection of available datasets as well as potential identification of additional new datasets based upon personal contacts and an updated search will be finalized early 2018. The aim is to evaluate diagnostic strategies across three research domains: (i) the optimal diagnostic approach for different healthcare settings, (ii) influence of comorbidity on the predictive performance of each diagnostic strategy, and (iii) optimize and tailor the efficiency and safety of ruling out PE across a broad spectrum of patients with a new, patient-tailored clinical decision model that combines clinical items with quantitative D-dimer testing.

DISCUSSION

This pre-planned individual patient data meta-analysis aims to contribute in resolving remaining diagnostic challenges of time-efficient diagnosis of pulmonary embolism by tailoring available diagnostic strategies for different healthcare settings and comorbidity.

SYSTEMATIC REVIEW REGISTRATION

Prospero trial registration: ID 89366.

Multiple Score Comparison: a network meta-analysis approach to comparison and external validation of prognostic scores

BACKGROUND

Prediction models and prognostic scores have been increasingly popular in both clinical practice and clinical research settings, for example to aid in risk-based decision making or control for confounding. In many medical fields, a large number of prognostic scores are available, but practitioners may find it difficult to choose between them due to lack of external validation as well as lack of comparisons between them.

METHODS

Borrowing methodology from network meta-analysis, we describe an approach to Multiple Score Comparison meta-analysis (MSC) which permits concurrent external validation and comparisons of prognostic scores using individual patient data (IPD) arising from a large-scale international collaboration. We describe the challenges in adapting network meta-analysis to the MSC setting, for instance the need to explicitly include correlations between the scores on a cohort level, and how to deal with many multi-score studies. We propose first using IPD to make cohort-level aggregate discrimination or calibration scores, comparing all to a common comparator. Then, standard network meta-analysis techniques can be applied, taking care to consider correlation structures in cohorts with multiple scores. Transitivity, consistency and heterogeneity are also examined.

RESULTS

We provide a clinical application, comparing prognostic scores for 3-year mortality in patients with chronic obstructive pulmonary disease using data from a large-scale collaborative initiative. We focus on the discriminative properties of the prognostic scores. Our results show clear differences in performance, with ADO and eBODE showing higher discrimination with respect to mortality than other considered scores. The assumptions of transitivity and local and global consistency were not violated. Heterogeneity was small.

CONCLUSIONS

We applied a network meta-analytic methodology to externally validate and concurrently compare the prognostic properties of clinical scores. Our large-scale external validation indicates that the scores with the best discriminative properties to predict 3 year mortality in patients with COPD are ADO and eBODE.

A multiple-model generalisation of updating clinical prediction models

There is growing interest in developing clinical prediction models (CPMs) to aid local healthcare decision‐making. Frequently, these CPMs are developed in isolation across different populations, with repetitive de novo derivation a common modelling strategy. However, this fails to utilise all available information and does not respond to changes in health processes through time and space. Alternatively, model updating techniques have previously been proposed that adjust an existing CPM to suit the new population, but these techniques are restricted to a single model. Therefore, we aimed to develop a generalised method for updating and aggregating multiple CPMs. The proposed “hybrid method” re‐calibrates multiple CPMs using stacked regression while concurrently revising specific covariates using individual participant data (IPD) under a penalised likelihood. The performance of the hybrid method was compared with existing methods in a clinical example of mortality risk prediction after transcatheter aortic valve implantation, and in 2 simulation studies. The simulation studies explored the effect of sample size and between‐population‐heterogeneity on the method, with each representing a situation of having multiple distinct CPMs and 1 set of IPD. When the sample size of the IPD was small, stacked regression and the hybrid method had comparable but highest performance across modelling methods. Conversely, in large IPD samples, development of a new model and the hybrid method gave the highest performance. Hence, the proposed strategy can inform the choice between utilising existing CPMs or developing a model de novo, thereby incorporating IPD, existing research, and prior (clinical) knowledge into the modelling strategy.

Automatic identification of variables in epidemiological datasets using logic regression

Background

For an individual participant data (IPD) meta-analysis, multiple datasets must be transformed in a consistent format, e.g. using uniform variable names. When large numbers of datasets have to be processed, this can be a time-consuming and error-prone task. Automated or semi-automated identification of variables can help to reduce the workload and improve the data quality. For semi-automation high sensitivity in the recognition of matching variables is particularly important, because it allows creating software which for a target variable presents a choice of source variables, from which a user can choose the matching one, with only low risk of having missed a correct source variable.

Methods

For each variable in a set of target variables, a number of simple rules were manually created. With logic regression, an optimal Boolean combination of these rules was searched for every target variable, using a random subset of a large database of epidemiological and clinical cohort data (construction subset). In a second subset of this database (validation subset), this optimal combination rules were validated.

Results

In the construction sample, 41 target variables were allocated on average with a positive predictive value (PPV) of 34%, and a negative predictive value (NPV) of 95%. In the validation sample, PPV was 33%, whereas NPV remained at 94%. In the construction sample, PPV was 50% or less in 63% of all variables, in the validation sample in 71% of all variables.

Conclusions

We demonstrated that the application of logic regression in a complex data management task in large epidemiological IPD meta-analyses is feasible. However, the performance of the algorithm is poor, which may require backup strategies.

Electronic supplementary material

The online version of this article (doi:10.1186/s12911-017-0429-1) contains supplementary material, which is available to authorized users.

Hierarchical Bayesian Logistic Regression to forecast metabolic control in type 2 DM patients

In this work we present our efforts in building a model able to forecast patients' changes in clinical conditions when repeated measurements are available. In this case the available risk calculators are typically not applicable. We propose a Hierarchical Bayesian Logistic Regression model, which allows taking into account individual and population variability in model parameters estimate. The model is used to predict metabolic control and its variation in type 2 diabetes mellitus. In particular we have analyzed a population of more than 1000 Italian type 2 diabetic patients, collected within the European project Mosaic. The results obtained in terms of Matthews Correlation Coefficient are significantly better than the ones gathered with standard logistic regression model, based on data pooling.

External validation, update and development of prediction models for pre-eclampsia using an Individual Participant Data (IPD) meta-analysis: the International Prediction of Pregnancy Complication Network (IPPIC pre-eclampsia) protocol

BACKGROUND

Pre-eclampsia, a condition with raised blood pressure and proteinuria is associated with an increased risk of maternal and offspring mortality and morbidity. Early identification of mothers at risk is needed to target management.

METHODS/DESIGN

We aim to systematically review the existing literature to identify prediction models for pre-eclampsia. We have established the International Prediction of Pregnancy Complication Network (IPPIC), made up of 72 researchers from 21 countries who have carried out relevant primary studies or have access to existing registry databases, and collectively possess data from more than two million patients. We will use the individual participant data (IPD) from these studies to externally validate these existing prediction models and summarise model performance across studies using random-effects meta-analysis for any, late (after 34 weeks) and early (before 34 weeks) onset pre-eclampsia. If none of the models perform well, we will recalibrate (update), or develop and validate new prediction models using the IPD. We will assess the differential accuracy of the models in various settings and subgroups according to the risk status. We will also validate or develop prediction models based on clinical characteristics only; clinical and biochemical markers; clinical and ultrasound parameters; and clinical, biochemical and ultrasound tests.

DISCUSSION

Numerous systematic reviews with aggregate data meta-analysis have evaluated various risk factors separately or in combination for predicting pre-eclampsia, but these are affected by many limitations. Our large-scale collaborative IPD approach encourages consensus towards well developed, and validated prognostic models, rather than a number of competing non-validated ones. The large sample size from our IPD will also allow development and validation of multivariable prediction model for the relatively rare outcome of early onset pre-eclampsia.

TRIAL REGISTRATION

The project was registered on Prospero on the 27 November 2015 with ID: CRD42015029349.