Meta-analysis and aggregation of multiple published prediction models

Clinical prognostic factors for older people: A systematic review and meta-analysis

OBJECTIVE

To explore the accuracy and precision of prognostic tools used in older people in predicting mortality, hospitalization, and nursing home admission across different settings and timings.

DESIGN

Systematic review and meta-analysis of prospective and retrospective studies.

DATA SOURCES

A systematic search from database inception until 01st February 2023 was run in Medline, Embase, Cinhal, Cochrane Library.

ELIGIBILITY CRITERIA

Studies were eligible if they reported accuracy (area under the curve [AUC]) and/or precision (C-index) for the prognostic index in relation to any of the following outcomes: mortality, hospitalization, and nursing home admission.

DATA EXTRACTION AND SYNTHESIS

Two independent reviewers extracted data. Data were pooled using a random effects model. The risk of bias was assessed with the Quality in Prognosis Studies (QUIPS) tool. If more than three studies for the same setting and time were available, a meta-analysis was performed and evaluated using the GRADE tool; other data were reported descriptively.

RESULTS

Among 16,082 studies initially considered, 159 studies with a total of 2398856 older people (mean age: 78 years) were included. The majority of the studies was carried out in hospital or medical wards. In the community setting, only two tools (Health Assessment Tool and the Multidimensional Prognostic Index, MPI) had good precision for long-term mortality. In emergency department setting, Barthel Index had an excellent accuracy in predicting short-term mortality. In medical wards, the MPI had a moderate certainty of the evidence in predicting short-term mortality (13 studies; 11,787 patients; AUC=0.79 and 4 studies; 3915 patients; C-index=0.82). Similar findings were available for MPI when considering longer follow-up periods. When considering nursing home and surgical wards, the literature was limited. The risk of bias was generally acceptable; observed bias was mainly owing to attrition and confounding.

CONCLUSIONS

Several tools are used to predict poor prognosis in geriatric patients, but only those derived from a multidimensional evaluation have the characteristics of precision and accuracy.

Why do probabilistic clinical models fail to transport between sites

The rising popularity of artificial intelligence in healthcare is highlighting the problem that a computational model achieving super-human clinical performance at its training sites may perform substantially worse at new sites. In this perspective, we argue that we should typically expect this failure to transport, and we present common sources for it, divided into those under the control of the experimenter and those inherent to the clinical data-generating process. Of the inherent sources we look a little deeper into site-specific clinical practices that can affect the data distribution, and propose a potential solution intended to isolate the imprint of those practices on the data from the patterns of disease cause and effect that are the usual target of probabilistic clinical models.

Artificial intelligence in the risk prediction models of cardiovascular disease and development of an independent validation screening tool: a systematic review

BACKGROUND

A comprehensive overview of artificial intelligence (AI) for cardiovascular disease (CVD) prediction and a screening tool of AI models (AI-Ms) for independent external validation are lacking. This systematic review aims to identify, describe, and appraise AI-Ms of CVD prediction in the general and special populations and develop a new independent validation score (IVS) for AI-Ms replicability evaluation.

METHODS

PubMed, Web of Science, Embase, and IEEE library were searched up to July 2021. Data extraction and analysis were performed for the populations, distribution, predictors, algorithms, etc. The risk of bias was evaluated with the prediction risk of bias assessment tool (PROBAST). Subsequently, we designed IVS for model replicability evaluation with five steps in five items, including transparency of algorithms, performance of models, feasibility of reproduction, risk of reproduction, and clinical implication, respectively. The review is registered in PROSPERO (No. CRD42021271789).

RESULTS

In 20,887 screened references, 79 articles (82.5% in 2017-2021) were included, which contained 114 datasets (67 in Europe and North America, but 0 in Africa). We identified 486 AI-Ms, of which the majority were in development (n = 380), but none of them had undergone independent external validation. A total of 66 idiographic algorithms were found; however, 36.4% were used only once and only 39.4% over three times. A large number of different predictors (range 5-52,000, median 21) and large-span sample size (range 80-3,660,000, median 4466) were observed. All models were at high risk of bias according to PROBAST, primarily due to the incorrect use of statistical methods. IVS analysis confirmed only 10 models as "recommended"; however, 281 and 187 were "not recommended" and "warning," respectively.

CONCLUSION

AI has led the digital revolution in the field of CVD prediction, but is still in the early stage of development as the defects of research design, report, and evaluation systems. The IVS we developed may contribute to independent external validation and the development of this field.

Prognostic models for clinical outcomes in patients with venous leg ulcers: A systematic review

OBJECTIVE

The purpose of this review was to identify prognostic models for clinical application in patients with venous leg ulcers (VLUs).

METHODS

Literature searches were conducted in Embase, Medline, Cochrane, and CINAHL databases from inception to December 22, 2021. Eligible studies reported prognostic models aimed at developing, validating, and adjusting multivariable prognostic models that include multiple prognostic factors combined, and that predicted clinical outcomes. Methodological quality was assessed using the CHARMS checklist and PROBAST short form questionnaire.

RESULTS

Thirteen studies were identified, of which three were validation studies of previously published models, four reported derivation and validation of models, and the remainder reported derivation models only. There was substantial heterogeneity in the model characteristics, including 11 studies focused on wound healing outcomes reporting 91 different predictors. Three studies shared similar predicted outcomes, follow-up timepoint and used a Cox proportional hazards model. However, these models reported different predictor selection methods and different predictors and it was therefore not feasible to summarize performance, such as discriminative ability.

CONCLUSIONS

There are no standout risk prediction models in the literature with promising clinical application for patients with VLUs. Future research should focus on developing and validating high-performing models in wider VLU populations.

Prognostic prediction models for oropharyngeal squamous cell carcinoma (OPSCC): a protocol for systematic review, critical appraisal and meta-analysis

INTRODUCTION

Oropharyngeal squamous cell carcinoma (OPSCC) is increasingly prevalent and has significantly heterogeneous risks of survival for diagnosed individuals due to the inter-related risk factors. Precise prediction of the risk of survival for an individual patient with OPSCC presents a useful adjunct to therapeutic decision-making regarding the management of OPSCC. The aim of this systematic review, critical appraisal and meta-analysis is to assess prognostic prediction models for OPSCC and lay a foundation for future research programmes to develop and validate prognostic prediction models for OPSCC.

METHODS AND ANALYSIS

This protocol will follow the Preferred Reporting Items for Systematic Review and Meta-Analyses Protocol statement. Based on predefined criteria, electronic databases including MEDLINE, Embase, Web of Science, the Cochrane Library and China National Knowledge Infrastructure (CNKI) will be searched for relevant studies without language restrictions from inception of databases to present. This study will systematically review published prognostic prediction models for survival outcomes in patients with OPSCC, describe their characteristics, compare performance and assess risk of bias and real-world clinical utility. Selection of eligible studies, data extraction and critical appraisal will be conducted independently by two reviewers. A third reviewer will resolve any disagreements. Included studies will be systematically summarised using appropriate tools designed for prognostic prediction modelling studies. Risk of bias and quality of studies will be assessed using the Prediction Model Risk of Bias Assessment Tool and the Transparent Reporting of a multivariable prediction model for individual prognosis or diagnosis. Performance measures of these models will be pooled and analysed with meta-analyses if feasible.

ETHICS AND DISSEMINATION

This review will be conducted completely based on published data, so approval from an ethics committee or written consent is not required. The results will be disseminated through a peer-reviewed publication.

PROSPERO REGISTRATION NUMBER

CRD42023400272.

Predictive accuracy of risk prediction models for recurrence, metastasis and survival for early-stage cutaneous melanoma: a systematic review

OBJECTIVES

To identify prognostic models for melanoma survival, recurrence and metastasis among American Joint Committee on Cancer stage I and II patients postsurgery; and evaluate model performance, including overall survival (OS) prediction.

DESIGN

Systematic review and narrative synthesis.

DATA SOURCES

Searched MEDLINE, Embase, CINAHL, Cochrane Library, Science Citation Index and grey literature sources including cancer and guideline websites from 2000 to September 2021.

ELIGIBILITY CRITERIA

Included studies on risk prediction models for stage I and II melanoma in adults ≥18 years. Outcomes included OS, recurrence, metastases and model performance. No language or country of publication restrictions were applied.

DATA EXTRACTION AND SYNTHESIS

Two pairs of reviewers independently screened studies, extracted data and assessed the risk of bias using the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies checklist and the Prediction study Risk of Bias Assessment Tool. Heterogeneous predictors prevented statistical synthesis.

RESULTS

From 28 967 records, 15 studies reporting 20 models were included; 8 (stage I), 2 (stage II), 7 (stages I-II) and 7 (stages not reported), but were clearly applicable to early stages. Clinicopathological predictors per model ranged from 3-10. The most common were: ulceration, Breslow thickness/depth, sociodemographic status and site. Where reported, discriminatory values were ≥0.7. Calibration measures showed good matches between predicted and observed rates. None of the studies assessed clinical usefulness of the models. Risk of bias was high in eight models, unclear in nine and low in three. Seven models were internally and externally cross-validated, six models were externally validated and eight models were internally validated.

CONCLUSIONS

All models are effective in their predictive performance, however the low quality of the evidence raises concern as to whether current follow-up recommendations following surgical treatment is adequate. Future models should incorporate biomarkers for improved accuracy.

PROSPERO REGISTRATION NUMBER

CRD42018086784.

Hospital length of stay prediction tools for all hospital admissions and general medicine populations: systematic review and meta-analysis

Background

Unwarranted extended length of stay (LOS) increases the risk of hospital-acquired complications, morbidity, and all-cause mortality and needs to be recognized and addressed proactively.

Objective

This systematic review aimed to identify validated prediction variables and methods used in tools that predict the risk of prolonged LOS in all hospital admissions and specifically General Medicine (GenMed) admissions.

Method

LOS prediction tools published since 2010 were identified in five major research databases. The main outcomes were model performance metrics, prediction variables, and level of validation. Meta-analysis was completed for validated models. The risk of bias was assessed using the PROBAST checklist.

Results

Overall, 25 all admission studies and 14 GenMed studies were identified. Statistical and machine learning methods were used almost equally in both groups. Calibration metrics were reported infrequently, with only 2 of 39 studies performing external validation. Meta-analysis of all admissions validation studies revealed a 95% prediction interval for theta of 0.596 to 0.798 for the area under the curve. Important predictor categories were co-morbidity diagnoses and illness severity risk scores, demographics, and admission characteristics. Overall study quality was deemed low due to poor data processing and analysis reporting.

Conclusion

To the best of our knowledge, this is the first systematic review assessing the quality of risk prediction models for hospital LOS in GenMed and all admissions groups. Notably, both machine learning and statistical modeling demonstrated good predictive performance, but models were infrequently externally validated and had poor overall study quality. Moving forward, a focus on quality methods by the adoption of existing guidelines and external validation is needed before clinical application.

Systematic review registration

https://www.crd.york.ac.uk/PROSPERO/, identifier: CRD42021272198.

Multi-domain prognostic models used in middle-aged adults without known cognitive impairment for predicting subsequent dementia

BACKGROUND

Dementia, a global health priority, has no current cure. Around 50 million people worldwide currently live with dementia, and this number is expected to treble by 2050. Some health conditions and lifestyle behaviours can increase or decrease the risk of dementia and are known as 'predictors'. Prognostic models combine such predictors to measure the risk of future dementia. Models that can accurately predict future dementia would help clinicians select high-risk adults in middle age and implement targeted risk reduction.

OBJECTIVES

Our primary objective was to identify multi-domain prognostic models used in middle-aged adults (aged 45 to 65 years) for predicting dementia or cognitive impairment. Eligible multi-domain prognostic models involved two or more of the modifiable dementia predictors identified in a 2020 Lancet Commission report and a 2019 World Health Organization (WHO) report (less education, hearing loss, traumatic brain injury, hypertension, excessive alcohol intake, obesity, smoking, depression, social isolation, physical inactivity, diabetes mellitus, air pollution, poor diet, and cognitive inactivity). Our secondary objectives were to summarise the prognostic models, to appraise their predictive accuracy (discrimination and calibration) as reported in the development and validation studies, and to identify the implications of using dementia prognostic models for the management of people at a higher risk for future dementia.

SEARCH METHODS

We searched MEDLINE, Embase, PsycINFO, CINAHL, and ISI Web of Science Core Collection from inception until 6 June 2022. We performed forwards and backwards citation tracking of included studies using the Web of Science platform.  SELECTION CRITERIA: We included development and validation studies of multi-domain prognostic models. The minimum eligible follow-up was five years. Our primary outcome was an incident clinical diagnosis of dementia based on validated diagnostic criteria, and our secondary outcome was dementia or cognitive impairment determined by any other method.

DATA COLLECTION AND ANALYSIS

Two review authors independently screened the references, extracted data using a template based on the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS), and assessed risk of bias and applicability of included studies using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). We synthesised the C-statistics of models that had been externally validated in at least three comparable studies.  MAIN RESULTS: We identified 20 eligible studies; eight were development studies and 12 were validation studies. There were 14 unique prognostic models: seven models with validation studies and seven models with development-only studies. The models included a median of nine predictors (range 6 to 34); the median number of modifiable predictors was five (range 2 to 11). The most common modifiable predictors in externally validated models were diabetes, hypertension, smoking, physical activity, and obesity. In development-only models, the most common modifiable predictors were obesity, diabetes, hypertension, and smoking. No models included hearing loss or air pollution as predictors. Nineteen studies had a high risk of bias according to the PROBAST assessment, mainly because of inappropriate analysis methods, particularly lack of reported calibration measures. Applicability concerns were low for 12 studies, as their population, predictors, and outcomes were consistent with those of interest for this review. Applicability concerns were high for nine studies, as they lacked baseline cognitive screening or excluded an age group within the range of 45 to 65 years. Only one model, Cardiovascular Risk Factors, Ageing, and Dementia (CAIDE), had been externally validated in multiple studies, allowing for meta-analysis. The CAIDE model included eight predictors (four modifiable predictors): age, education, sex, systolic blood pressure, body mass index (BMI), total cholesterol, physical activity and APOEƐ4 status. Overall, our confidence in the prediction accuracy of CAIDE was very low; our main reasons for downgrading the certainty of the evidence were high risk of bias across all the studies, high concern of applicability, non-overlapping confidence intervals (CIs), and a high degree of heterogeneity. The summary C-statistic was 0.71 (95% CI 0.66 to 0.76; 3 studies; very low-certainty evidence) for the incident clinical diagnosis of dementia, and 0.67 (95% CI 0.61 to 0.73; 3 studies; very low-certainty evidence) for dementia or cognitive impairment based on cognitive scores. Meta-analysis of calibration measures was not possible, as few studies provided these data.

AUTHORS' CONCLUSIONS

We identified 14 unique multi-domain prognostic models used in middle-aged adults for predicting subsequent dementia. Diabetes, hypertension, obesity, and smoking were the most common modifiable risk factors used as predictors in the models. We performed meta-analyses of C-statistics for one model (CAIDE), but the summary values were unreliable. Owing to lack of data, we were unable to meta-analyse the calibration measures of CAIDE. This review highlights the need for further robust external validations of multi-domain prognostic models for predicting future risk of dementia in middle-aged adults.

Combining Breast Cancer Risk Prediction Models

Accurate risk stratification is key to reducing cancer morbidity through targeted screening and preventative interventions. Multiple breast cancer risk prediction models are used in clinical practice, and often provide a range of different predictions for the same patient. Integrating information from different models may improve the accuracy of predictions, which would be valuable for both clinicians and patients. BRCAPRO is a widely used model that predicts breast cancer risk based on detailed family history information. A major limitation of this model is that it does not consider non-genetic risk factors. To address this limitation, we expand BRCAPRO by combining it with another popular existing model, BCRAT (i.e., Gail), which uses a largely complementary set of risk factors, most of them non-genetic. We consider two approaches for combining BRCAPRO and BCRAT: (1) modifying the penetrance (age-specific probability of developing cancer given genotype) functions in BRCAPRO using relative hazard estimates from BCRAT, and (2) training an ensemble model that takes BRCAPRO and BCRAT predictions as input. Using both simulated data and data from Newton-Wellesley Hospital and the Cancer Genetics Network, we show that the combination models are able to achieve performance gains over both BRCAPRO and BCRAT. In the Cancer Genetics Network cohort, we show that the proposed BRCAPRO + BCRAT penetrance modification model performs comparably to IBIS, an existing model that combines detailed family history with non-genetic risk factors.

Residual Disease Threshold After Primary Surgical Treatment for Advanced Epithelial Ovarian Cancer, Part 1: A Systematic Review and Network Meta-Analysis

BACKGROUND

We present a systematic review and network meta-analysis (NMA) that is the precursor underpinning the Bayesian analyses that adjust for publication bias, presented in the same edition in AJT. The review assesses optimal cytoreduction for women undergoing primary advanced epithelial ovarian cancer (EOC) surgery.

AREAS OF UNCERTAINTY

To assess the impact of residual disease (RD) after primary debulking surgery in women with advanced EOC. This review explores the impact of leaving varying levels of primary debulking surgery.

DATA SOURCES

We conducted a systematic review and random-effects NMA for overall survival (OS) to incorporate direct and indirect estimates of RD thresholds, including concurrent comparative, retrospective studies of ≥100 adult women (18+ years) with surgically staged advanced EOC (FIGO stage III/IV) who had confirmed histological diagnoses of ovarian cancer. Pairwise meta-analyses of all directly compared RD thresholds was previously performed before conducting this NMA, and the statistical heterogeneity of studies within each comparison was evaluated using recommended methods.

THERAPEUTIC ADVANCES

Twenty-five studies (n = 20,927) were included. Analyses demonstrated the prognostic importance of complete cytoreduction to no macroscopic residual disease (NMRD), with a hazard ratio for OS of 2.0 (95% confidence interval, 1.8-2.2) for <1 cm RD threshold versus NMRD. NMRD was associated with prolonged survival across all RD thresholds. Leaving NMRD was predicted to provide longest survival (probability of being best = 99%). The results were robust to sensitivity analysis including only those studies that adjusted for extent of disease at primary surgery (hazard ratio 2.3, 95% confidence interval, 1.9-2.6). The overall certainty of evidence was moderate and statistical adjustment of effect estimates in included studies minimized bias.

CONCLUSIONS

The results confirm a strong association between complete cytoreduction to NMRD and improved OS. The NMA approach forms part of the methods guidance underpinning policy making in many jurisdictions. Our analyses present an extension to the previous work in this area.

Methodological guidance for the evaluation and updating of clinical prediction models: a systematic review

BACKGROUND

Clinical prediction models are often not evaluated properly in specific settings or updated, for instance, with information from new markers. These key steps are needed such that models are fit for purpose and remain relevant in the long-term. We aimed to present an overview of methodological guidance for the evaluation (i.e., validation and impact assessment) and updating of clinical prediction models.

METHODS

We systematically searched nine databases from January 2000 to January 2022 for articles in English with methodological recommendations for the post-derivation stages of interest. Qualitative analysis was used to summarize the 70 selected guidance papers.

RESULTS

Key aspects for validation are the assessment of statistical performance using measures for discrimination (e.g., C-statistic) and calibration (e.g., calibration-in-the-large and calibration slope). For assessing impact or usefulness in clinical decision-making, recent papers advise using decision-analytic measures (e.g., the Net Benefit) over simplistic classification measures that ignore clinical consequences (e.g., accuracy, overall Net Reclassification Index). Commonly recommended methods for model updating are recalibration (i.e., adjustment of intercept or baseline hazard and/or slope), revision (i.e., re-estimation of individual predictor effects), and extension (i.e., addition of new markers). Additional methodological guidance is needed for newer types of updating (e.g., meta-model and dynamic updating) and machine learning-based models.

CONCLUSION

Substantial guidance was found for model evaluation and more conventional updating of regression-based models. An important development in model evaluation is the introduction of a decision-analytic framework for assessing clinical usefulness. Consensus is emerging on methods for model updating.

Clinical prediction models assessing response to radiotherapy for rectal cancer: protocol for a systematic review

BACKGROUND

Rectal cancer has a high prevalence. The standard of care for management of localised disease involves major surgery and/or chemoradiotherapy, but these modalities are sometimes associated with mortality and morbidity. The notion of 'watch and wait' has therefore emerged and offers an organ-sparing approach to patients after administering a less invasive initial treatment, such as X-ray brachytherapy (Papillon technique). It is thus important to evaluate how likely patients are to respond to such therapies, to develop patient-tailored treatment pathways. We propose a systematic review to identify published clinical prediction models of the response of rectal cancer to treatment that includes radiotherapy and here present our protocol.

METHODS

Included studies will develop multivariable clinical prediction models which assess response to treatment and overall survival of adult patients who have been diagnosed with any stage of rectal cancer and have received radiotherapy treatment with curative intent. Cohort studies and randomised controlled trials will be included. The primary outcome will be the occurrence of salvage surgery at 1 year after treatment. Secondary outcomes include salavage surgery at at any reported time point, the predictive accuracy of models, the quality of the developed models and the feasibility of using the model in clinical practice. Ovid MEDLINE, PubMed, Cochrane Library, EMBASE and CINAHL will be searched from inception to 24 February 2022. Keywords and phrases related to rectal cancer, radiotherapy and prediction models will be used. Studies will be selected once the deduplication, title, abstract and full-text screening process have been completed by two independent reviewers. The PRISMA-P checklist will be followed. A third reviewer will resolve any disagreement. The data extraction form will be pilot-tested using a representative 5% sample of the studies reviewed. The CHARMS checklist will be implemented. Risk of bias in each study will be assessed using the PROBAST tool. A narrative synthesis will be performed and if sufficient data are identified, meta-analysis will be undertaken as described in Debray et al. DISCUSSION: This systematic review will identify factors that predict response to the treatment protocol. Any gaps for potential development of new clinical prediction models will be highlighted.

TRIAL REGISTRATION

CRD42022277704.

Distributed Quasi-Poisson regression algorithm for modeling multi-site count outcomes in distributed data networks

BACKGROUND

Observational studies incorporating real-world data from multiple institutions facilitate study of rare outcomes or exposures and improve generalizability of results. Due to privacy concerns surrounding patient-level data sharing across institutions, methods for performing regression analyses distributively are desirable. Meta-analysis of institution-specific estimates is commonly used, but has been shown to produce biased estimates in certain settings. While distributed regression methods are increasingly available, methods for analyzing count outcomes are currently limited. Count data in practice are commonly subject to overdispersion, exhibiting greater variability than expected under a given statistical model.

OBJECTIVE

We propose a novel computational method, a one-shot distributed algorithm for quasi-Poisson regression (ODAP), to distributively model count outcomes while accounting for overdispersion.

METHODS

ODAP incorporates a surrogate likelihood approach to perform distributed quasi-Poisson regression without requiring patient-level data sharing, only requiring sharing of aggregate data from each participating institution. ODAP requires at most three rounds of non-iterative communication among institutions to generate coefficient estimates and corresponding standard errors. In simulations, we evaluate ODAP under several data scenarios possible in multi-site analyses, comparing ODAP and meta-analysis estimates in terms of error relative to pooled regression estimates, considered the gold standard. In a proof-of-concept real-world data analysis, we similarly compare ODAP and meta-analysis in terms of relative error to pooled estimatation using data from the OneFlorida Clinical Research Consortium, modeling length of stay in COVID-19 patients as a function of various patient characteristics. In a second proof-of-concept analysis, using the same outcome and covariates, we incorporate data from the UnitedHealth Group Clinical Discovery Database together with the OneFlorida data in a distributed analysis to compare estimates produced by ODAP and meta-analysis.

RESULTS

In simulations, ODAP exhibited negligible error relative to pooled regression estimates across all settings explored. Meta-analysis estimates, while largely unbiased, were increasingly variable as heterogeneity in the outcome increased across institutions. When baseline expected count was 0.2, relative error for meta-analysis was above 5% in 25% of iterations (250/1000), while the largest relative error for ODAP in any iteration was 3.59%. In our proof-of-concept analysis using only OneFlorida data, ODAP estimates were closer to pooled regression estimates than those produced by meta-analysis for all 15 covariates. In our distributed analysis incorporating data from both OneFlorida and the UnitedHealth Group Clinical Discovery Database, ODAP and meta-analysis estimates were largely similar, while some differences in estimates (as large as 13.8%) could be indicative of bias in meta-analytic estimates.

CONCLUSIONS

ODAP performs privacy-preserving, communication-efficient distributed quasi-Poisson regression to analyze count outcomes using data stored within multiple institutions. Our method produces estimates nearly matching pooled regression estimates and sometimes more accurate than meta-analysis estimates, most notably in settings with relatively low counts and high outcome heterogeneity across institutions.

DLMM as a lossless one-shot algorithm for collaborative multi-site distributed linear mixed models

Linear mixed models are commonly used in healthcare-based association analyses for analyzing multi-site data with heterogeneous site-specific random effects. Due to regulations for protecting patients’ privacy, sensitive individual patient data (IPD) typically cannot be shared across sites. We propose an algorithm for fitting distributed linear mixed models (DLMMs) without sharing IPD across sites. This algorithm achieves results identical to those achieved using pooled IPD from multiple sites (i.e., the same effect size and standard error estimates), hence demonstrating the lossless property. The algorithm requires each site to contribute minimal aggregated data in only one round of communication. We demonstrate the lossless property of the proposed DLMM algorithm by investigating the associations between demographic and clinical characteristics and length of hospital stay in COVID-19 patients using administrative claims from the UnitedHealth Group Clinical Discovery Database. We extend this association study by incorporating 120,609 COVID-19 patients from 11 collaborative data sources worldwide.

A lossless, one-shot and privacy-preserving distributed algorithm was revealed for fitting linear mixed models on multi-site data. The algorithm was applied to a study of 120,609 COVID-19 patients using only minimal aggregated data from each of 14 sites.

Optimal surveillance strategies for patients with stage 1 cutaneous melanoma post primary tumour excision: three systematic reviews and an economic model

BACKGROUND

Malignant melanoma is the fifth most common cancer in the UK, with rates continuing to rise, resulting in considerable burden to patients and the NHS.

OBJECTIVES

The objectives were to evaluate the effectiveness and cost-effectiveness of current and alternative follow-up strategies for stage IA and IB melanoma.

REVIEW METHODS

Three systematic reviews were conducted. (1) The effectiveness of surveillance strategies. Outcomes were detection of new primaries, recurrences, metastases and survival. Risk of bias was assessed using the Cochrane Collaboration's Risk-of-Bias 2.0 tool. (2) Prediction models to stratify by risk of recurrence, metastases and survival. Model performance was assessed by study-reported measures of discrimination (e.g. D-statistic, Harrel's c-statistic), calibration (e.g. the Hosmer-Lemeshow 'goodness-of-fit' test) or overall performance (e.g. Brier score, R ²). Risk of bias was assessed using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). (3) Diagnostic test accuracy of fine-needle biopsy and ultrasonography. Outcomes were detection of new primaries, recurrences, metastases and overall survival. Risk of bias was assessed using the Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2) tool. Review data and data from elsewhere were used to model the cost-effectiveness of alternative surveillance strategies and the value of further research.

RESULTS

(1) The surveillance review included one randomised controlled trial. There was no evidence of a difference in new primary or recurrence detected (risk ratio 0.75, 95% confidence interval 0.43 to 1.31). Risk of bias was considered to be of some concern. Certainty of the evidence was low. (2) Eleven risk prediction models were identified. Discrimination measures were reported for six models, with the area under the operating curve ranging from 0.59 to 0.88. Three models reported calibration measures, with coefficients of ≥ 0.88. Overall performance was reported by two models. In one, the Brier score was slightly better than the American Joint Committee on Cancer scheme score. The other reported an R ² of 0.47 (95% confidence interval 0.45 to 0.49). All studies were judged to have a high risk of bias. (3) The diagnostic test accuracy review identified two studies. One study considered fine-needle biopsy and the other considered ultrasonography. The sensitivity and specificity for fine-needle biopsy were 0.94 (95% confidence interval 0.90 to 0.97) and 0.95 (95% confidence interval 0.90 to 0.97), respectively. For ultrasonography, sensitivity and specificity were 1.00 (95% confidence interval 0.03 to 1.00) and 0.99 (95% confidence interval 0.96 to 0.99), respectively. For the reference standards and flow and timing domains, the risk of bias was rated as being high for both studies. The cost-effectiveness results suggest that, over a lifetime, less intensive surveillance than recommended by the National Institute for Health and Care Excellence might be worthwhile. There was considerable uncertainty. Improving the diagnostic performance of cancer nurse specialists and introducing a risk prediction tool could be promising. Further research on transition probabilities between different stages of melanoma and on improving diagnostic accuracy would be of most value.

LIMITATIONS

Overall, few data of limited quality were available, and these related to earlier versions of the American Joint Committee on Cancer staging. Consequently, there was considerable uncertainty in the economic evaluation.

CONCLUSIONS

Despite adoption of rigorous methods, too few data are available to justify changes to the National Institute for Health and Care Excellence recommendations on surveillance. However, alternative strategies warrant further research, specifically on improving estimates of incidence, progression of recurrent disease; diagnostic accuracy and health-related quality of life; developing and evaluating risk stratification tools; and understanding patient preferences.

STUDY REGISTRATION

This study is registered as PROSPERO CRD42018086784.

FUNDING

This project was funded by the National Institute for Health Research Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol 25, No. 64. See the NIHR Journals Library website for further project information.

Prognostic models for mortality after cardiac surgery in patients with infective endocarditis: a systematic review and aggregation of prediction models

BACKGROUND

There are several prognostic models to estimate the risk of mortality after surgery for active infective endocarditis (IE). However, these models incorporate different predictors and their performance is uncertain.

OBJECTIVE

We systematically reviewed and critically appraised all available prediction models of postoperative mortality in patients undergoing surgery for IE, and aggregated them into a meta-model.

DATA SOURCES

We searched Medline and EMBASE databases from inception to June 2020.

STUDY ELIGIBILITY CRITERIA

We included studies that developed or updated a prognostic model of postoperative mortality in patient with IE.

METHODS

We assessed the risk of bias of the models using PROBAST (Prediction model Risk Of Bias ASsessment Tool) and we aggregated them into an aggregate meta-model based on stacked regressions and optimized it for a nationwide registry of IE patients. The meta-model performance was assessed using bootstrap validation methods and adjusted for optimism.

RESULTS

We identified 11 prognostic models for postoperative mortality. Eight models had a high risk of bias. The meta-model included weighted predictors from the remaining three models (EndoSCORE, specific ES-I and specific ES-II), which were not rated as high risk of bias and provided full model equations. Additionally, two variables (age and infectious agent) that had been modelled differently across studies, were estimated based on the nationwide registry. The performance of the meta-model was better than the original three models, with the corresponding performance measures: C-statistics 0.79 (95% CI 0.76-0.82), calibration slope 0.98 (95% CI 0.86-1.13) and calibration-in-the-large -0.05 (95% CI -0.20 to 0.11).

CONCLUSIONS

The meta-model outperformed published models and showed a robust predictive capacity for predicting the individualized risk of postoperative mortality in patients with IE.

PROTOCOL REGISTRATION

PROSPERO (registration number CRD42020192602).

Risk models for VTE and bleeding in medical inpatients: systematic identification and expert assessment

Risk assessment models (RAMs) for venous thromboembolism (VTE) and bleeding in hospitalized medical patients inform appropriate use of thromboprophylaxis. Our aim was to use a novel approach for selecting risk factors for VTE and bleeding to be included in RAMs. First, we used the results of a systematic review of all candidate factors. Second, we used the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach to assess the certainty of the evidence for the identified factors. Third, we using a structured approach to select factors to develop the RAMs, by building on clinical and methodological expertise. The expert panel made judgments on whether to include, potentially include, or exclude risk factors, according to domains of the GRADE approach and the Delphi method. The VTE RAM included age >60 years, previous VTE, acute infections, immobility, acute paresis, active malignancy, critical illness, and known thrombophilia. The bleeding RAM included age ≥65 years, renal failure, thrombocytopenia, active gastroduodenal ulcers, hepatic disease, recent bleeding, and critical illness. We identified acute infection as a factor that was not considered in widely used RAMs. Also, we identified factors that require further research to confirm or refute their importance in a VTE RAM (eg, D-dimer). We excluded autoimmune disease which is included in the IMPROVE (International Medical Prevention Registry on Venous Thromboembolism) bleeding RAM. Our results also suggest that sex, malignancy, and use of central venous catheters (factors in the IMPROVE bleeding RAM) require further research. In conclusion, our study presents a novel approach to systematically identifying and assessing risk factors to be included or further explored during RAM development.

Effects of low-dose hydrocortisone and hydrocortisone plus fludrocortisone in adults with septic shock: a protocol for a systematic review and meta-analysis of individual participant data

INTRODUCTION

The benefits and risks of low-dose hydrocortisone in patients with septic shock have been investigated in numerous randomised controlled trials and trial-level meta-analyses. Yet, the routine use of this treatment remains controversial. To overcome the limitations of previous meta-analyses inherent to the use of aggregate data, we will perform an individual patient data meta-analysis (IPDMA) on the effect of hydrocortisone with or without fludrocortisone compared with placebo or usual care on 90-day mortality and other outcomes in patients with septic shock.

METHODS AND ANALYSIS

To assess the benefits and risks of hydrocortisone, with or without fludrocortisone for adults with septic shock, we will search major electronic databases from inception to September 2020 (Cochrane Central Register of Controlled Trials, MEDLINE, EMBASE and Latin American Caribbean Health Sciences Literature), complimented by a search for unpublished trials. The primary analysis will compare hydrocortisone with or without fludrocortisone to placebo or no treatment in adult patients with septic shock. Secondary analyses will compare hydrocortisone to placebo (or usual care), hydrocortisone plus fludrocortisone to placebo (or usual care), and hydrocortisone versus hydrocortisone plus fludrocortisone. The primary outcome will be all cause mortality at 90 days. We will conduct both one-stage IPDMA using mixed-effect models and machine learning with targeted maximum likelihood analyses. We will assess the risk of bias related to unshared data and related to the quality of individual trial.

ETHICS AND DISSEMINATION

This IPDMA will use existing data from completed randomised clinical trials and will comply with the ethical and regulatory requirements regarding data sharing for each of the component trials. The findings of this study will be submitted for publication in a peer-review journal with straightforward policy for open access.

PROSPERO REGISTRATION NUMBER

CRD42017062198.

Clinical prediction models to predict the risk of multiple binary outcomes: a comparison of approaches

Clinical prediction models (CPMs) can predict clinically relevant outcomes or events. Typically, prognostic CPMs are derived to predict the risk of a single future outcome. However, there are many medical applications where two or more outcomes are of interest, meaning this should be more widely reflected in CPMs so they can accurately estimate the joint risk of multiple outcomes simultaneously. A potentially naïve approach to multi-outcome risk prediction is to derive a CPM for each outcome separately, then multiply the predicted risks. This approach is only valid if the outcomes are conditionally independent given the covariates, and it fails to exploit the potential relationships between the outcomes. This paper outlines several approaches that could be used to develop CPMs for multiple binary outcomes. We consider four methods, ranging in complexity and conditional independence assumptions: namely, probabilistic classifier chain, multinomial logistic regression, multivariate logistic regression, and a Bayesian probit model. These are compared with methods that rely on conditional independence: separate univariate CPMs and stacked regression. Employing a simulation study and real-world example, we illustrate that CPMs for joint risk prediction of multiple outcomes should only be derived using methods that model the residual correlation between outcomes. In such a situation, our results suggest that probabilistic classification chains, multinomial logistic regression or the Bayesian probit model are all appropriate choices. We call into question the development of CPMs for each outcome in isolation when multiple correlated or structurally related outcomes are of interest and recommend more multivariate approaches to risk prediction.

Prognostic models for chronic kidney disease: a systematic review and external validation

BACKGROUND

Accurate risk prediction is needed in order to provide personalized healthcare for chronic kidney disease (CKD) patients. An overload of prognosis studies is being published, ranging from individual biomarker studies to full prediction studies. We aim to systematically appraise published prognosis studies investigating multiple biomarkers and their role in risk predictions. Our primary objective was to investigate if the prognostic models that are reported in the literature were of sufficient quality and to externally validate them.

METHODS

We undertook a systematic review and appraised the quality of studies reporting multivariable prognosis models for end-stage renal disease (ESRD), cardiovascular (CV) events and mortality in CKD patients. We subsequently externally validated these models in a randomized trial that included patients from a broad CKD population.

RESULTS

We identified 91 papers describing 36 multivariable models for prognosis of ESRD, 50 for CV events, 46 for mortality and 17 for a composite outcome. Most studies were deemed of moderate quality. Moreover, they often adopted different definitions for the primary outcome and rarely reported full model equations (21% of the included studies). External validation was performed in the Multifactorial Approach and Superior Treatment Efficacy in Renal Patients with the Aid of Nurse Practitioners trial (n = 788, with 160 events for ESRD, 79 for CV and 102 for mortality). The 24 models that reported full model equations showed a great variability in their performance, although calibration remained fairly adequate for most models, except when predicting mortality (calibration slope >1.5).

CONCLUSIONS

This review shows that there is an abundance of multivariable prognosis models for the CKD population. Most studies were considered of moderate quality, and they were reported and analysed in such a manner that their results cannot directly be used in follow-up research or in clinical practice.

UMBRELLA protocol: systematic reviews of multivariable biomarker prognostic models developed to predict clinical outcomes in patients with heart failure

BACKGROUND

Heart failure (HF) is a chronic and common condition with a rising prevalence, especially in the elderly. Morbidity and mortality rates in people with HF are similar to those with common forms of cancer. Clinical guidelines highlight the need for more detailed prognostic information to optimise treatment and care planning for people with HF. Besides proven prognostic biomarkers and numerous newly developed prognostic models for HF clinical outcomes, no risk stratification models have been adequately established. Through a number of linked systematic reviews, we aim to assess the quality of the existing models with biomarkers in HF and summarise the evidence they present.

METHODS

We will search MEDLINE, EMBASE, Web of Science Core Collection, and the prognostic studies database maintained by the Cochrane Prognosis Methods Group combining sensitive published search filters, with no language restriction, from 1990 onwards. Independent pairs of reviewers will screen and extract data. Eligible studies will be those developing, validating, or updating any prognostic model with biomarkers for clinical outcomes in adults with any type of HF. Data will be extracted using a piloted form that combines published good practice guidelines for critical appraisal, data extraction, and risk of bias assessment of prediction modelling studies. Missing information on predictive performance measures will be sought by contacting authors or estimated from available information when possible. If sufficient high quality and homogeneous data are available, we will meta-analyse the predictive performance of identified models. Sources of between-study heterogeneity will be explored through meta-regression using pre-defined study-level covariates. Results will be reported narratively if study quality is deemed to be low or if the between-study heterogeneity is high. Sensitivity analyses for risk of bias impact will be performed.

DISCUSSION

This project aims to appraise and summarise the methodological conduct and predictive performance of existing clinically homogeneous HF prognostic models in separate systematic reviews.Registration: PROSPERO registration number CRD42019086990.

Prognostic models for newly-diagnosed chronic lymphocytic leukaemia in adults: a systematic review and meta-analysis

BACKGROUND

Chronic lymphocytic leukaemia (CLL) is the most common cancer of the lymphatic system in Western countries. Several clinical and biological factors for CLL have been identified. However, it remains unclear which of the available prognostic models combining those factors can be used in clinical practice to predict long-term outcome in people newly-diagnosed with CLL.

OBJECTIVES

To identify, describe and appraise all prognostic models developed to predict overall survival (OS), progression-free survival (PFS) or treatment-free survival (TFS) in newly-diagnosed (previously untreated) adults with CLL, and meta-analyse their predictive performances.

SEARCH METHODS

We searched MEDLINE (from January 1950 to June 2019 via Ovid), Embase (from 1974 to June 2019) and registries of ongoing trials (to 5 March 2020) for development and validation studies of prognostic models for untreated adults with CLL. In addition, we screened the reference lists and citation indices of included studies.

SELECTION CRITERIA

We included all prognostic models developed for CLL which predict OS, PFS, or TFS, provided they combined prognostic factors known before treatment initiation, and any studies that tested the performance of these models in individuals other than the ones included in model development (i.e. 'external model validation studies'). We included studies of adults with confirmed B-cell CLL who had not received treatment prior to the start of the study. We did not restrict the search based on study design.

DATA COLLECTION AND ANALYSIS

We developed a data extraction form to collect information based on the Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS). Independent pairs of review authors screened references, extracted data and assessed risk of bias according to the Prediction model Risk Of Bias ASsessment Tool (PROBAST). For models that were externally validated at least three times, we aimed to perform a quantitative meta-analysis of their predictive performance, notably their calibration (proportion of people predicted to experience the outcome who do so) and discrimination (ability to differentiate between people with and without the event) using a random-effects model. When a model categorised individuals into risk categories, we pooled outcome frequencies per risk group (low, intermediate, high and very high). We did not apply GRADE as guidance is not yet available for reviews of prognostic models.

MAIN RESULTS

From 52 eligible studies, we identified 12 externally validated models: six were developed for OS, one for PFS and five for TFS. In general, reporting of the studies was poor, especially predictive performance measures for calibration and discrimination; but also basic information, such as eligibility criteria and the recruitment period of participants was often missing. We rated almost all studies at high or unclear risk of bias according to PROBAST. Overall, the applicability of the models and their validation studies was low or unclear; the most common reasons were inappropriate handling of missing data and serious reporting deficiencies concerning eligibility criteria, recruitment period, observation time and prediction performance measures. We report the results for three models predicting OS, which had available data from more than three external validation studies: CLL International Prognostic Index (CLL-IPI) This score includes five prognostic factors: age, clinical stage, IgHV mutational status, B2-microglobulin and TP53 status. Calibration: for the low-, intermediate- and high-risk groups, the pooled five-year survival per risk group from validation studies corresponded to the frequencies observed in the model development study. In the very high-risk group, predicted survival from CLL-IPI was lower than observed from external validation studies. Discrimination: the pooled c-statistic of seven external validation studies (3307 participants, 917 events) was 0.72 (95% confidence interval (CI) 0.67 to 0.77). The 95% prediction interval (PI) of this model for the c-statistic, which describes the expected interval for the model's discriminative ability in a new external validation study, ranged from 0.59 to 0.83. Barcelona-Brno score Aimed at simplifying the CLL-IPI, this score includes three prognostic factors: IgHV mutational status, del(17p) and del(11q). Calibration: for the low- and intermediate-risk group, the pooled survival per risk group corresponded to the frequencies observed in the model development study, although the score seems to overestimate survival for the high-risk group. Discrimination: the pooled c-statistic of four external validation studies (1755 participants, 416 events) was 0.64 (95% CI 0.60 to 0.67); 95% PI 0.59 to 0.68. MDACC 2007 index score The authors presented two versions of this model including six prognostic factors to predict OS: age, B2-microglobulin, absolute lymphocyte count, gender, clinical stage and number of nodal groups. Only one validation study was available for the more comprehensive version of the model, a formula with a nomogram, while seven studies (5127 participants, 994 events) validated the simplified version of the model, the index score. Calibration: for the low- and intermediate-risk groups, the pooled survival per risk group corresponded to the frequencies observed in the model development study, although the score seems to overestimate survival for the high-risk group. Discrimination: the pooled c-statistic of the seven external validation studies for the index score was 0.65 (95% CI 0.60 to 0.70); 95% PI 0.51 to 0.77.

AUTHORS' CONCLUSIONS

Despite the large number of published studies of prognostic models for OS, PFS or TFS for newly-diagnosed, untreated adults with CLL, only a minority of these (N = 12) have been externally validated for their respective primary outcome. Three models have undergone sufficient external validation to enable meta-analysis of the model's ability to predict survival outcomes. Lack of reporting prevented us from summarising calibration as recommended. Of the three models, the CLL-IPI shows the best discrimination, despite overestimation. However, performance of the models may change for individuals with CLL who receive improved treatment options, as the models included in this review were tested mostly on retrospective cohorts receiving a traditional treatment regimen. In conclusion, this review shows a clear need to improve the conducting and reporting of both prognostic model development and external validation studies. For prognostic models to be used as tools in clinical practice, the development of the models (and their subsequent validation studies) should adapt to include the latest therapy options to accurately predict performance. Adaptations should be timely.

On the aggregation of published prognostic scores for causal inference in observational studies

As real world evidence on drug efficacy involves nonrandomized studies, statistical methods adjusting for confounding are needed. In this context, prognostic score (PGS) analysis has recently been proposed as a method for causal inference. It aims to restore balance across the different treatment groups by identifying subjects with a similar prognosis for a given reference exposure ("control"). This requires the development of a multivariable prognostic model in the control arm of the study sample, which is then extrapolated to the different treatment arms. Unfortunately, large cohorts for developing prognostic models are not always available. Prognostic models are therefore subject to a dilemma between overfitting and parsimony; the latter being prone to a violation of the assumption of no unmeasured confounders when important covariates are ignored. Although it is possible to limit overfitting by using penalization strategies, an alternative approach is to adopt evidence synthesis. Aggregating previously published prognostic models may improve the generalizability of PGS, while taking account of a large set of covariates-even when limited individual participant data are available. In this article, we extend a method for prediction model aggregation to PGS analysis in nonrandomized studies. We conduct extensive simulations to assess the validity of model aggregation, compared with other methods of PGS analysis for estimating marginal treatment effects. We show that aggregating existing PGS into a "meta-score" is robust to misspecification, even when elementary scores wrongfully omit confounders or focus on different outcomes. We illustrate our methods in a setting of treatments for asthma.

Ensemble Learning to Improve the Prediction of Fetal Macrosomia and Large-for-Gestational Age

Background: The objective of this study was to investigate the use of ensemble methods to improve the prediction of fetal macrosomia and large for gestational age from prenatal ultrasound imaging measurements. Methods: We evaluated and compared the prediction accuracies of nonlinear and quadratic mixed-effects models coupled with 26 different empirical formulas for estimating fetal weights in predicting large fetuses at birth. The data for the investigation were taken from the Successive Small-for-Gestational-Age-Births study. Ensemble methods, a class of machine learning techniques, were used to improve the prediction accuracies by combining the individual models and empirical formulas. Results: The prediction accuracy of individual statistical models and empirical formulas varied considerably in predicting macrosomia but varied less in predicting large for gestational age. Two ensemble methods, voting and stacking, with model selection, can combine the strengths of individual models and formulas and can improve the prediction accuracy. Conclusions: Ensemble learning can improve the prediction of fetal macrosomia and large for gestational age and have the potential to assist obstetricians in clinical decisions.

Prognostic prediction models for pregnancy complications in women with gestational diabetes: a protocol for systematic review, critical appraisal and meta-analysis

BACKGROUND

Gestational diabetes (GDM) is increasingly common and has significant implications during pregnancy and for the long-term health of the mother and offspring. However, it is a heterogeneous condition with inter-related factors including ethnicity, body mass index and gestational weight gain significantly modifying the absolute risk of complications at an individual level. Predicting the risk of pregnancy complications for an individual woman with GDM presents a useful adjunct to therapeutic decision-making and patient education. Diagnostic prediction models for GDM are prevalent. In contrast, prediction models for risk of complications in those with GDM are relatively novel. This study will systematically review published prognostic prediction models for pregnancy complications in women with GDM, describe their characteristics, compare performance and assess methodological quality and applicability.

METHODS

Studies will be identified by searching MEDLINE and Embase electronic databases. Title and abstract screening, full-text review and data extraction will be completed independently by two reviewers. The included studies will be systematically assessed for risk of bias and applicability using appropriate tools designed for prediction modelling studies. Extracted data will be tabulated to facilitate qualitative comparison of published prediction models. Quantitative data on predictive performance of these models will be synthesised with meta-analyses if appropriate.

DISCUSSION

This review will identify and summarise all published prognostic prediction models for pregnancy complications in women with GDM. We will compare model performance across different settings and populations with meta-analysis if appropriate. This work will guide subsequent phases in the prognosis research framework: further model development, external validation and model updating, and impact assessment. The ultimate model will estimate the absolute risk of pregnancy complications for women with GDM and will be implemented into routine care as an evidence-based GDM complication risk prediction model. It is anticipated to offer value to women and their clinicians with individualised risk assessment and may assist decision-making. Ultimately, this systematic review is an important step towards a personalised risk-stratified model-of-care for GDM to allow preventative and therapeutic interventions for the maximal benefit to women and their offspring, whilst sparing expense and harm for those at low risk.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO registration number CRD42019115223.

Machine Learning For Tuning, Selection, And Ensemble Of Multiple Risk Scores For Predicting Type 2 Diabetes

Background

This study proposes the use of machine learning algorithms to improve the accuracy of type 2 diabetes predictions using non-invasive risk score systems.

Methods

We evaluated and compared the prediction accuracies of existing non-invasive risk score systems using the data from the REACTION study (Risk Evaluation of Cancers in Chinese Diabetic Individuals: A Longitudinal Study). Two simple risk scores were established on the bases of logistic regression. Machine learning techniques (ensemble methods) were used to improve prediction accuracies by combining the individual score systems.

Results

Existing score systems from Western populations performed worse than the scores from Eastern populations in general. The two newly established score systems performed better than most existing scores systems but a little worse than the Chinese score system. Using ensemble methods with model selection algorithms yielded better prediction accuracy than all the simple score systems.

Conclusion

Our proposed machine learning methods can be used to improve the accuracy of screening the undiagnosed type 2 diabetes and identifying the high-risk patients.

A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes

It is widely recommended that any developed-diagnostic or prognostic-prediction model is externally validated in terms of its predictive performance measured by calibration and discrimination. When multiple validations have been performed, a systematic review followed by a formal meta-analysis helps to summarize overall performance across multiple settings, and reveals under which circumstances the model performs suboptimal (alternative poorer) and may need adjustment. We discuss how to undertake meta-analysis of the performance of prediction models with either a binary or a time-to-event outcome. We address how to deal with incomplete availability of study-specific results (performance estimates and their precision), and how to produce summary estimates of the c-statistic, the observed:expected ratio and the calibration slope. Furthermore, we discuss the implementation of frequentist and Bayesian meta-analysis methods, and propose novel empirically-based prior distributions to improve estimation of between-study heterogeneity in small samples. Finally, we illustrate all methods using two examples: meta-analysis of the predictive performance of EuroSCORE II and of the Framingham Risk Score. All examples and meta-analysis models have been implemented in our newly developed R package "metamisc".

Clinical prognostic models for severe dengue: a systematic review protocol

Background: Dengue is a common mosquito-borne, with high morbidity rates recorded in the annual. Dengue contributes to a major disease burden in many tropical countries. This demonstrates the urgent need in developing effective approaches to identify severe cases early. For this purpose, many multivariable prognostic models using multiple prognostic variables were developed to predict the risk of progression to severe outcomes. The aim of the planned systematic review is to identify and describe the existing clinical multivariable prognostic models for severe dengue as well as examine the possibility of combining them. These findings will suggest directions for further research of this field. Methods: This protocol has followed the guidelines of the Preferred Reporting Items for Systematic Reviews and Meta - Analyses Protocol (PRISMA-P). We will conduct a comprehensive search of Pubmed, Embase, and Web of Science. Eligibility criteria include being published in peer-review journals, focusing on human subjects and developing the multivariable prognostic model for severe dengue, without any restriction on language, location and period of publication, and study design. The reference list will be captured and removed from duplications. We will use the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) checklist to extract data and Prediction study risk of bias assessment tool (PROBAST) to assess the study quality. Discussion: This systematic review will describe the existing prediction models, summarize the current status of prognostic research on dengue, and report the possibility to combine the models to optimize the power of each paradigm. PROSPERO registration: CRD42018102907.

Assessment of heterogeneity in an individual participant data meta-analysis of prediction models: An overview and illustration

Clinical prediction models aim to provide estimates of absolute risk for a diagnostic or prognostic endpoint. Such models may be derived from data from various studies in the context of a meta‐analysis. We describe and propose approaches for assessing heterogeneity in predictor effects and predictions arising from models based on data from different sources. These methods are illustrated in a case study with patients suffering from traumatic brain injury, where we aim to predict 6‐month mortality based on individual patient data using meta‐analytic techniques (15 studies, n = 11 022 patients). The insights into various aspects of heterogeneity are important to develop better models and understand problems with the transportability of absolute risk predictions.

Evidence synthesis in prognosis research

Over the past few years, evidence synthesis has become essential to investigate and improve the generalizability of medical research findings. This strategy often involves a meta-analysis to formally summarize quantities of interest, such as relative treatment effect estimates. The use of meta-analysis methods is, however, less straightforward in prognosis research because substantial variation exists in research objectives, analysis methods and the level of reported evidence. We present a gentle overview of statistical methods that can be used to summarize data of prognostic factor and prognostic model studies. We discuss how aggregate data, individual participant data, or a combination thereof can be combined through meta-analysis methods. Recent examples are provided throughout to illustrate the various methods.

Clinical prognostic models for severe dengue: a systematic review protocol

Background: Dengue is a common mosquito-borne, with high morbidity rates recorded in the annually. Dengue contributes a major disease burden in many tropical countries. This demonstrates the urgent need in developing effective approaches to identify severe cases early. For this purpose, many multivariable prognostic models using multiple prognostic variables were developed to predict the risk of progression to severe outcomes. The aim of the planned systematic review is to identify and describe the existing clinical multivariable prognostic models for severe dengue as well as examine the possibility of combining them. These findings will suggest directions for further research of this field. Methods: This protocol has followed the guidelines of the Preferred Reporting Items for Systematic Reviews and Meta - Analyses Protocol (PRISMA-P). We will conduct a comprehensive search of Pubmed, Embase and Web of Science. Eligiblity criteria include being published in peer-review journals, focusing on human subjects and developing the multivariable prognostic model for severe dengue, without any restriction on language, location and period of publication, and study design. The reference list will be captured and removed from duplications. We will use the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) checklist to extract data and Prediction study risk of bias assessment tool (PROBAST) to assess the study quality. Discussion: This systematic review will describe the existing prediction models, summarize the current status of prognostic research on dengue, and report the possibility to combine the models to optimize the power of each paradigm. PROSPERO registration: CRD42018102907.

The use of prognostic scores for causal inference with general treatment regimes

In nonrandomised studies, inferring causal effects requires appropriate methods for addressing confounding bias. Although it is common to adopt propensity score analysis to this purpose, prognostic score analysis has recently been proposed as an alternative strategy. While both approaches were originally introduced to estimate causal effects for binary interventions, the theory of propensity score has since been extended to the case of general treatment regimes. Indeed, many treatments are not assigned in a binary fashion and require a certain extent of dosing. Hence, researchers may often be interested in estimating treatment effects across multiple exposures. To the best of our knowledge, the prognostic score analysis has not been yet generalised to this case. In this article, we describe the theory of prognostic scores for causal inference with general treatment regimes. Our methods can be applied to compare multiple treatments using nonrandomised data, a topic of great relevance in contemporary evaluations of clinical interventions. We propose estimators for the average treatment effects in different populations of interest, the validity of which is assessed through a series of simulations. Finally, we present an illustrative case in which we estimate the effect of the delay to Aspirin administration on a composite outcome of death or dependence at 6 months in stroke patients.

PROBAST: A Tool to Assess Risk of Bias and Applicability of Prediction Model Studies: Explanation and Elaboration

Prediction models in health care use predictors to estimate for an individual the probability that a condition or disease is already present (diagnostic model) or will occur in the future (prognostic model). Publications on prediction models have become more common in recent years, and competing prediction models frequently exist for the same outcome or target population. Health care providers, guideline developers, and policymakers are often unsure which model to use or recommend, and in which persons or settings. Hence, systematic reviews of these studies are increasingly demanded, required, and performed. A key part of a systematic review of prediction models is examination of risk of bias and applicability to the intended population and setting. To help reviewers with this process, the authors developed PROBAST (Prediction model Risk Of Bias ASsessment Tool) for studies developing, validating, or updating (for example, extending) prediction models, both diagnostic and prognostic. PROBAST was developed through a consensus process involving a group of experts in the field. It includes 20 signaling questions across 4 domains (participants, predictors, outcome, and analysis). This explanation and elaboration document describes the rationale for including each domain and signaling question and guides researchers, reviewers, readers, and guideline developers in how to use them to assess risk of bias and applicability concerns. All concepts are illustrated with published examples across different topics. The latest version of the PROBAST checklist, accompanying documents, and filled-in examples can be downloaded from www.probast.org.

Individualised prediction of psychosis in individuals meeting at-risk mental state (ARMS) criteria: protocol for a systematic review of clinical prediction models

BACKGROUND

Psychotic disorders affect about 3% of the population worldwide and are associated with high personal, social and economic costs. They tend to have their first onset in adolescence. Increasing emphasis has been placed on early intervention to detect illness and minimise disability. In the late 1990s, criteria were developed to identify individuals at high risk for psychotic disorder. These are known as the at-risk mental state (ARMS) criteria. While ARMS individuals have a risk of psychosis much greater than the general population, most individuals meeting the ARMS criteria will not develop psychosis. Despite this, the National Institute for Health and Care Excellence recommends cognitive behavioural therapy (CBT) for all ARMS people.Clinical prediction models that combine multiple patient characteristics to predict individual outcome risk may facilitate identification of patients who would benefit from CBT and conversely those that would benefit from less costly and less intensive regular mental state monitoring. The study will systematically review the evidence on clinical prediction models aimed at making individualised predictions for the transition to psychosis.

METHODS

Database searches will be conducted on PsycINFO, Medline, EMBASE and CINAHL. Reference lists and subject experts will be utilised. No language restrictions will be placed on publications, but searches will be restricted to 1994 onwards, the initial year of the first prospective study using ARMS criteria. Studies of any design will be included if they examined, in ARMS patients, whether more than one factor in combination is associated with the risk of transition to psychosis. Study quality will be assessed using the prediction model risk of bias assessment tool (PROBAST). Clinical prediction models will be summarised qualitatively, and if tested in multiple validation studies, their predictive performance will be summarised using a random-effects meta-analysis model.

DISCUSSION

The results of the review will identify prediction models for the risk of transition to psychosis. These will be informative for clinicians currently treating ARMS patients and considering potential preventive interventions. The conclusions of the review will also inform the possible update and external validation of prediction models and clinical prediction rules to identify those at high or low risk of transition to psychosis.

TRIAL REGISTRATION

The review has been registered with PROSPERO (CRD42018108488).

Pre-procedural risk models for patients undergoing transcatheter aortic valve implantation

Transcatheter aortic valve implantation (TAVI) has emerged as the standard treatment option for patients with symptomatic aortic stenosis who are considered intermediate to high surgical risk. Nonetheless, optimal clinical outcomes following the procedure require careful consideration of procedural risk by the Heart Team. While this decision-making could be supported through the development of TAVI-specific clinical prediction models (CPMs), current models remain suboptimal. In this review paper, we aimed to outline the performance of several recently derived TAVI CPMs that predict mortality and present some future research directions. We discuss how the existing risk models have achieved only moderate discrimination but highlight that some of the models are well calibrated across multiple populations, indicating the feasibility of using them to aid benchmarking analyses. Moreover, we suggest that future work should focus on the development of CPMs in cohorts of patients with aortic stenosis that include multiple treatment modalities. Supported by appropriate modelling of 'what if' scenarios, this would allow the Heart Teams to predict and compare outcomes across surgical aortic valve replacement, medical management and TAVI, thereby allowing one to personalise treatment decisions to the individual patient. Such a goal could be facilitated by considering novel risk factors, shifting the focus to endpoints other than mortality, and through collaborative efforts to combine the evidence base and existing models across wider populations.

POPCORN: A web service for individual PrognOsis prediction based on multi-center clinical data CollabORatioN without patient-level data sharing

BACKGROUND AND OBJECTIVE

Clinical prognosis prediction plays an important role in clinical research and practice. The construction of prediction models based on electronic health record data has recently become a research focus. Due to the lack of external validation, prediction models based on single-center, hospital-specific datasets may not perform well with datasets from other medical institutions. Therefore, research investigating prognosis prediction model construction based on a collaborative analysis of multi-center electronic health record data could increase the number and coverage of patients used for model training, enrich patient prognostic features and ultimately improve the accuracy and generalization of prognosis prediction.

MATERIALS AND METHODS

A web service for individual prognosis prediction based on multi-center clinical data collaboration without patient-level data sharing (POPCORN) was proposed. POPCORN focuses on solving key issues in multi-center collaborative research based on electronic health record systems; these issues include the standardization of clinical data expression, the preservation of patient privacy during model training and the effect of case mix variance on the prediction model construction and application. POPCORN is based on a multivariable meta-analysis and a Bayesian framework and can construct suitable prediction models for multiple clinical scenarios that can effectively adapt to complex clinical application environments.

RESULTS

POPCORN was validated using a joint, multi-center collaborative research network between China and the United States with patients diagnosed with colorectal cancer. The performance of the models based on POPCORN was comparable to that of the standard prognosis prediction model; however, POPCORN did not expose raw patient data. The prediction models had similar AUC, but the BMA model had the lowest ECI across all prediction models, indicating that this model had better calibration performance than the other models, especially for patients in Chinese hospitals.

CONCLUSIONS

The POPCORN system can build prediction models that perform well in complex clinical application scenarios and can provide effective decision support for individual patient prognostic predictions.

Systematic review of prediction models for delirium in the older adult inpatient

OBJECTIVE

To identify existing prognostic delirium prediction models and evaluate their validity and statistical methodology in the older adult (≥60 years) acute hospital population.

DESIGN

Systematic review.

DATA SOURCES AND METHODS

PubMed, CINAHL, PsychINFO, SocINFO, Cochrane, Web of Science and Embase were searched from 1 January 1990 to 31 December 2016. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses and CHARMS Statement guided protocol development.

INCLUSION CRITERIA

age >60 years, inpatient, developed/validated a prognostic delirium prediction model.

EXCLUSION CRITERIA

alcohol-related delirium, sample size ≤50. The primary performance measures were calibration and discrimination statistics. Two authors independently conducted search and extracted data. The synthesis of data was done by the first author. Disagreement was resolved by the mentoring author.

RESULTS

The initial search resulted in 7,502 studies. Following full-text review of 192 studies, 33 were excluded based on age criteria (<60 years) and 27 met the defined criteria. Twenty-three delirium prediction models were identified, 14 were externally validated and 3 were internally validated. The following populations were represented: 11 medical, 3 medical/surgical and 13 surgical. The assessment of delirium was often non-systematic, resulting in varied incidence. Fourteen models were externally validated with an area under the receiver operating curve range from 0.52 to 0.94. Limitations in design, data collection methods and model metric reporting statistics were identified.

CONCLUSIONS

Delirium prediction models for older adults show variable and typically inadequate predictive capabilities. Our review highlights the need for development of robust models to predict delirium in older inpatients. We provide recommendations for the development of such models.

A multiple-model generalisation of updating clinical prediction models

There is growing interest in developing clinical prediction models (CPMs) to aid local healthcare decision‐making. Frequently, these CPMs are developed in isolation across different populations, with repetitive de novo derivation a common modelling strategy. However, this fails to utilise all available information and does not respond to changes in health processes through time and space. Alternatively, model updating techniques have previously been proposed that adjust an existing CPM to suit the new population, but these techniques are restricted to a single model. Therefore, we aimed to develop a generalised method for updating and aggregating multiple CPMs. The proposed “hybrid method” re‐calibrates multiple CPMs using stacked regression while concurrently revising specific covariates using individual participant data (IPD) under a penalised likelihood. The performance of the hybrid method was compared with existing methods in a clinical example of mortality risk prediction after transcatheter aortic valve implantation, and in 2 simulation studies. The simulation studies explored the effect of sample size and between‐population‐heterogeneity on the method, with each representing a situation of having multiple distinct CPMs and 1 set of IPD. When the sample size of the IPD was small, stacked regression and the hybrid method had comparable but highest performance across modelling methods. Conversely, in large IPD samples, development of a new model and the hybrid method gave the highest performance. Hence, the proposed strategy can inform the choice between utilising existing CPMs or developing a model de novo, thereby incorporating IPD, existing research, and prior (clinical) knowledge into the modelling strategy.

Meta-analytical synthesis of regression coefficients under different categorization scheme of continuous covariates

Recently, the number of clinical prediction models sharing the same regression task has increased in the medical literature. However, evidence synthesis methodologies that use the results of these regression models have not been sufficiently studied, particularly in meta-analysis settings where only regression coefficients are available. One of the difficulties lies in the differences between the categorization schemes of continuous covariates across different studies. In general, categorization methods using cutoff values are study specific across available models, even if they focus on the same covariates of interest. Differences in the categorization of covariates could lead to serious bias in the estimated regression coefficients and thus in subsequent syntheses. To tackle this issue, we developed synthesis methods for linear regression models with different categorization schemes of covariates. A 2-step approach to aggregate the regression coefficient estimates is proposed. The first step is to estimate the joint distribution of covariates by introducing a latent sampling distribution, which uses one set of individual participant data to estimate the marginal distribution of covariates with categorization. The second step is to use a nonlinear mixed-effects model with correction terms for the bias due to categorization to estimate the overall regression coefficients. Especially in terms of precision, numerical simulations show that our approach outperforms conventional methods, which only use studies with common covariates or ignore the differences between categorization schemes. The method developed in this study is also applied to a series of WHO epidemiologic studies on white blood cell counts.

Risk prediction models for delirium in the intensive care unit after cardiac surgery: a systematic review and independent external validation

Numerous risk prediction models are available for predicting delirium after cardiac surgery, but few have been directly compared with one another or been validated in an independent data set. We conducted a systematic review to identify validated risk prediction models of delirium (using the Confusion Assessment Method-Intensive Care Unit tool) after cardiac surgery and assessed the transportability of the risk prediction models on a prospective cohort of 600 consecutive patients undergoing cardiac surgery at a university hospital in Hong Kong from July 2013 to July 2015. The discrimination (c-statistic), calibration (GiViTI calibration belt), and clinical usefulness (decision curve analysis) of the risk prediction models were examined in a stepwise manner. Three published high-quality intensive care unit delirium risk prediction models (n=5939) were identified: Katznelson, the original PRE-DELIRIC, and the international recalibrated PRE-DELIRIC model. Delirium occurred in 83 patients (13.8%, 95% CI: 11.2-16.9%). After updating the intercept and regression coefficients in the Katznelson model, there was fair discrimination (0.62, 95% CI: 0.58-0.66) and good calibration. As the original PRE-DELIRIC model was already validated externally and recalibrated in six countries, we performed a logistic calibration on the recalibrated model and found acceptable discrimination (0.75, 95% CI: 0.72-0.79) and good calibration. Decision curve analysis demonstrated that the recalibrated PRE-DELIRIC risk model was marginally more clinically useful than the Katznelson model. Current models predict delirium risk in the intensive care unit after cardiac surgery with only fair to moderate accuracy and are insufficient for routine clinical use.

Clinical prediction in defined populations: a simulation study investigating when and how to aggregate existing models

Background

Clinical prediction models (CPMs) are increasingly deployed to support healthcare decisions but they are derived inconsistently, in part due to limited data. An emerging alternative is to aggregate existing CPMs developed for similar settings and outcomes. This simulation study aimed to investigate the impact of between-population-heterogeneity and sample size on aggregating existing CPMs in a defined population, compared with developing a model de novo.

Methods

Simulations were designed to mimic a scenario in which multiple CPMs for a binary outcome had been derived in distinct, heterogeneous populations, with potentially different predictors available in each. We then generated a new ‘local’ population and compared the performance of CPMs developed for this population by aggregation, using stacked regression, principal component analysis or partial least squares, with redevelopment from scratch using backwards selection and penalised regression.

Results

While redevelopment approaches resulted in models that were miscalibrated for local datasets of less than 500 observations, model aggregation methods were well calibrated across all simulation scenarios. When the size of local data was less than 1000 observations and between-population-heterogeneity was small, aggregating existing CPMs gave better discrimination and had the lowest mean square error in the predicted risks compared with deriving a new model. Conversely, given greater than 1000 observations and significant between-population-heterogeneity, then redevelopment outperformed the aggregation approaches. In all other scenarios, both aggregation and de novo derivation resulted in similar predictive performance.

Conclusion

This study demonstrates a pragmatic approach to contextualising CPMs to defined populations. When aiming to develop models in defined populations, modellers should consider existing CPMs, with aggregation approaches being a suitable modelling strategy particularly with sparse data on the local population.

Electronic supplementary material

The online version of this article (doi:10.1186/s12874-016-0277-1) contains supplementary material, which is available to authorized users.

A review of statistical updating methods for clinical prediction models

A clinical prediction model is a tool for predicting healthcare outcomes, usually within a specific population and context. A common approach is to develop a new clinical prediction model for each population and context; however, this wastes potentially useful historical information. A better approach is to update or incorporate the existing clinical prediction models already developed for use in similar contexts or populations. In addition, clinical prediction models commonly become miscalibrated over time, and need replacing or updating. In this article, we review a range of approaches for re-using and updating clinical prediction models; these fall in into three main categories: simple coefficient updating, combining multiple previous clinical prediction models in a meta-model and dynamic updating of models. We evaluated the performance (discrimination and calibration) of the different strategies using data on mortality following cardiac surgery in the United Kingdom: We found that no single strategy performed sufficiently well to be used to the exclusion of the others. In conclusion, useful tools exist for updating existing clinical prediction models to a new population or context, and these should be implemented rather than developing a new clinical prediction model from scratch, using a breadth of complementary statistical methods.

Prediction models for cardiovascular disease risk in the general population: systematic review

OBJECTIVE

 To provide an overview of prediction models for risk of cardiovascular disease (CVD) in the general population.

DESIGN

 Systematic review.

DATA SOURCES

 Medline and Embase until June 2013.

ELIGIBILITY CRITERIA FOR STUDY SELECTION

 Studies describing the development or external validation of a multivariable model for predicting CVD risk in the general population.

RESULTS

 9965 references were screened, of which 212 articles were included in the review, describing the development of 363 prediction models and 473 external validations. Most models were developed in Europe (n=167, 46%), predicted risk of fatal or non-fatal coronary heart disease (n=118, 33%) over a 10 year period (n=209, 58%). The most common predictors were smoking (n=325, 90%) and age (n=321, 88%), and most models were sex specific (n=250, 69%). Substantial heterogeneity in predictor and outcome definitions was observed between models, and important clinical and methodological information were often missing. The prediction horizon was not specified for 49 models (13%), and for 92 (25%) crucial information was missing to enable the model to be used for individual risk prediction. Only 132 developed models (36%) were externally validated and only 70 (19%) by independent investigators. Model performance was heterogeneous and measures such as discrimination and calibration were reported for only 65% and 58% of the external validations, respectively.

CONCLUSIONS

 There is an excess of models predicting incident CVD in the general population. The usefulness of most of the models remains unclear owing to methodological shortcomings, incomplete presentation, and lack of external validation and model impact studies. Rather than developing yet another similar CVD risk prediction model, in this era of large datasets, future research should focus on externally validating and comparing head-to-head promising CVD risk models that already exist, on tailoring or even combining these models to local settings, and investigating whether these models can be extended by addition of new predictors.

Predicting surgical outcome in patients with International Federation of Gynecology and Obstetrics stage III or IV ovarian cancer using computed tomography: a systematic review of prediction models

OBJECTIVE

Maximal cytoreduction to no residual disease is an important predictor of prognosis in patients with advanced-stage epithelial ovarian cancer. Preoperative prediction of outcome of surgery should guide treatment decisions, for example, primary debulking or neoadjuvant chemotherapy followed by interval debulking surgery. The objective of this study was to systematically review studies evaluating computed tomography imaging based models predicting the amount of residual tumor after cytoreductive surgery for advanced-stage epithelial ovarian cancer.

METHODS

We systematically searched the literature for studies investigating multivariable models that predicted the amount of residual disease after cytoreductive surgery in advanced-stage epithelial ovarian cancer using computed tomography imaging. Detected studies were scored for quality and classified as model derivation or validation studies. We summarized their performance in terms of discrimination when possible.

RESULTS

We identified 11 studies that described 13 models. The 4 models that were externally validated all had a poor discriminative capacity (sensitivity, 15%-79%; specificity, 32%-64%). The only internal validated model had an area under the receiver operating characteristic curve of 0.67. Peritoneal thickening, mesenterial and diaphragm disease, and ascites were most often used as predictors in the final models. We did not find studies that assessed the impact of prediction model on outcomes.

CONCLUSIONS

Currently, there are no external validated studies with a good predictive performance for residual disease. Studies of better quality are needed, especially studies that focus on predicting any residual disease after surgery.