Assessment of heterogeneity in an individual participant data meta-analysis of prediction models: An overview and illustration

Pitfalls of single-study external validation illustrated with a model predicting functional outcome after aneurysmal subarachnoid hemorrhage

BACKGROUND

Prediction models are often externally validated with data from a single study or cohort. However, the interpretation of performance estimates obtained with single-study external validation is not as straightforward as assumed. We aimed to illustrate this by conducting a large number of external validations of a prediction model for functional outcome in subarachnoid hemorrhage (SAH) patients.

METHODS

We used data from the Subarachnoid Hemorrhage International Trialists (SAHIT) data repository (n = 11,931, 14 studies) to refit the SAHIT model for predicting a dichotomous functional outcome (favorable versus unfavorable), with the (extended) Glasgow Outcome Scale or modified Rankin Scale score, at a minimum of three months after discharge. We performed leave-one-cluster-out cross-validation to mimic the process of multiple single-study external validations. Each study represented one cluster. In each of these validations, we assessed discrimination with Harrell's c-statistic and calibration with calibration plots, the intercepts, and the slopes. We used random effects meta-analysis to obtain the (reference) mean performance estimates and between-study heterogeneity (I2-statistic). The influence of case-mix variation on discriminative performance was assessed with the model-based c-statistic and we fitted a "membership model" to obtain a gross estimate of transportability.

RESULTS

Across 14 single-study external validations, model performance was highly variable. The mean c-statistic was 0.74 (95%CI 0.70-0.78, range 0.52-0.84, I2 = 0.92), the mean intercept was -0.06 (95%CI -0.37-0.24, range -1.40-0.75, I2 = 0.97), and the mean slope was 0.96 (95%CI 0.78-1.13, range 0.53-1.31, I2 = 0.90). The decrease in discriminative performance was attributable to case-mix variation, between-study heterogeneity, or a combination of both. Incidentally, we observed poor generalizability or transportability of the model.

CONCLUSIONS

We demonstrate two potential pitfalls in the interpretation of model performance with single-study external validation. With single-study external validation. (1) model performance is highly variable and depends on the choice of validation data and (2) no insight is provided into generalizability or transportability of the model that is needed to guide local implementation. As such, a single single-study external validation can easily be misinterpreted and lead to a false appreciation of the clinical prediction model. Cross-validation is better equipped to address these pitfalls.

Development and Validation of a Prediction Model for 1-Year Mortality in Patients With a Hematologic Malignancy Admitted to the ICU

OBJECTIVES

To develop and validate a prediction model for 1-year mortality in patients with a hematologic malignancy acutely admitted to the ICU.

DESIGN

A retrospective cohort study.

SETTING

Five university hospitals in the Netherlands between 2002 and 2015.

PATIENTS

A total of 1097 consecutive patients with a hematologic malignancy were acutely admitted to the ICU for at least 24 h.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

We created a 13-variable model from 22 potential predictors. Key predictors included active disease, age, previous hematopoietic stem cell transplantation, mechanical ventilation, lowest platelet count, acute kidney injury, maximum heart rate, and type of malignancy. A bootstrap procedure reduced overfitting and improved the model's generalizability. This involved estimating the optimism in the initial model and shrinking the regression coefficients accordingly in the final model. We assessed performance using internal-external cross-validation by center and compared it with the Acute Physiology and Chronic Health Evaluation II model. Additionally, we evaluated clinical usefulness through decision curve analysis. The overall 1-year mortality rate observed in the study was 62% (95% CI, 59-65). Our 13-variable prediction model demonstrated acceptable calibration and discrimination at internal-external validation across centers (C-statistic 0.70; 95% CI, 0.63-0.77), outperforming the Acute Physiology and Chronic Health Evaluation II model (C-statistic 0.61; 95% CI, 0.57-0.65). Decision curve analysis indicated overall net benefit within a clinically relevant threshold probability range of 60-100% predicted 1-year mortality.

CONCLUSIONS

Our newly developed 13-variable prediction model predicts 1-year mortality in hematologic malignancy patients admitted to the ICU more accurately than the Acute Physiology and Chronic Health Evaluation II model. This model may aid in shared decision-making regarding the continuation of ICU care and end-of-life considerations.

Do machine learning methods lead to similar individualized treatment rules? A comparison study on real data

Identifying patients who benefit from a treatment is a key aspect of personalized medicine, which allows the development of individualized treatment rules (ITRs). Many machine learning methods have been proposed to create such rules. However, to what extent the methods lead to similar ITRs, that is, recommending the same treatment for the same individuals is unclear. In this work, we compared 22 of the most common approaches in two randomized control trials. Two classes of methods can be distinguished. The first class of methods relies on predicting individualized treatment effects from which an ITR is derived by recommending the treatment evaluated to the individuals with a predicted benefit. In the second class, methods directly estimate the ITR without estimating individualized treatment effects. For each trial, the performance of ITRs was assessed by various metrics, and the pairwise agreement between all ITRs was also calculated. Results showed that the ITRs obtained via the different methods generally had considerable disagreements regarding the patients to be treated. A better concordance was found among akin methods. Overall, when evaluating the performance of ITRs in a validation sample, all methods produced ITRs with limited performance, suggesting a high potential for optimism. For non-parametric methods, this optimism was likely due to overfitting. The different methods do not lead to similar ITRs and are therefore not interchangeable. The choice of the method strongly influences for which patients a certain treatment is recommended, drawing some concerns about their practical use.

Heterogeneity in meta-analysis: a comprehensive overview

In recent years, meta-analysis has evolved to a critically important field of Statistics, and has significant applications in Medicine and Health Sciences. In this work we briefly present existing methodologies to conduct meta-analysis along with any discussion and recent developments accompanying them. Undoubtedly, studies brought together in a systematic review will differ in one way or another. This yields a considerable amount of variability, any kind of which may be termed heterogeneity. To this end, reports of meta-analyses commonly present a statistical test of heterogeneity when attempting to establish whether the included studies are indeed similar in terms of the reported output or not. We intend to provide an overview of the topic, discuss the potential sources of heterogeneity commonly met in the literature and provide useful guidelines on how to address this issue and to detect heterogeneity. Moreover, we review the recent developments in the Bayesian approach along with the various graphical tools and statistical software that are currently available to the analyst. In addition, we discuss sensitivity analysis issues and other approaches of understanding the causes of heterogeneity. Finally, we explore heterogeneity in meta-analysis for time to event data in a nutshell, pointing out its unique characteristics.

Estimating individualized treatment effects using an individual participant data meta-analysis

BACKGROUND

One key aspect of personalized medicine is to identify individuals who benefit from an intervention. Some approaches have been developed to estimate individualized treatment effects (ITE) with a single randomized control trial (RCT) or observational data, but they are often underpowered for the ITE estimation. Using individual participant data meta-analyses (IPD-MA) might solve this problem. Few studies have investigated how to develop risk prediction models with IPD-MA, and it remains unclear how to combine those methods with approaches used for ITE estimation. In this article, we compared different approaches using both simulated and real data with binary and time-to-event outcomes to estimate the individualized treatment effects from an IPD-MA in a one-stage approach.

METHODS

We compared five one-stage models: naive model (NA), random intercept (RI), stratified intercept (SI), rank-1 (R1), and fully stratified (FS), built with two different strategies, the S-learner and the T-learner constructed with a Monte Carlo simulation study in which we explored different scenarios with a binary or a time-to-event outcome. To evaluate the performance of the models, we used the c-statistic for benefit, the calibration of predictions, and the mean squared error. The different models were also used on the INDANA IPD-MA, comparing an anti-hypertensive treatment to no treatment or placebo ( N = 40 237 , 836 events).

RESULTS

Simulation results showed that using the S-learner led to better ITE estimation performances for both binary and time-to-event outcomes. None of the risk models stand out and had significantly better results. For the INDANA dataset with a binary outcome, the naive and the random intercept models had the best performances.

CONCLUSIONS

For the choice of the strategy, using interactions with treatment (the S-learner) is preferable. For the choice of the method, no approach is better than the other.

Predictors of Intensive Care Unit Admissions in Patients Presenting with Coronavirus Disease 2019

Background  Increased mortality rates among coronavirus disease 2019 (COVID-19) positive patients admitted to intensive care units (ICUs) highlight a compelling need to establish predictive criteria for ICU admissions. The aim of our study was to identify criteria for recognizing patients with COVID-19 at elevated risk for ICU admission. Methods  We identified patients who tested positive for COVID-19 and were hospitalized between March and May 2020. Patients' data were manually abstracted through review of electronic medical records. An ICU admission prediction model was derived from a random sample of half the patients using multivariable logistic regression. The model was validated with the remaining half of the patients using c-statistic. Results  We identified 1,094 patients; 204 (18.6%) were admitted to the ICU. Correlates of ICU admission were age, body mass index (BMI), quick Sequential Organ Failure Assessment (qSOFA) score, arterial oxygen saturation to fraction of inspired oxygen ratio, platelet count, and white blood cell count. The c-statistic in the derivation subset (0.798, 95% confidence interval [CI]: 0.748, 0.848) and the validation subset (0.764, 95% CI: 0.706, 0.822) showed excellent comparability. At 22% predicted probability for ICU admission, the derivation subset estimated sensitivity was 0.721, (95% CI: 0.637, 0.804) and specificity was 0.763, (95% CI: 0.722, 0.804). Our pilot predictive model identified the combination of age, BMI, qSOFA score, and oxygenation status as significant predictors for ICU admission. Conclusion  ICU admission among patients with COVID-19 can be predicted by age, BMI, level of hypoxia, and severity of illness.

Discriminative performance of pancreatic stone protein in predicting ICU mortality and infection severity in adult patients with infection: a systematic review and individual patient level meta-analysis

BACKGROUND

Several studies suggested pancreatic stone protein (PSP) as a promising biomarker to predict mortality among patients with severe infection. The objective of the study was to evaluate the performance of PSP in predicting intensive care unit (ICU) mortality and infection severity among critically ill adults admitted to the hospital for infection.

METHODS

A systematic search across Cochrane Central Register of Controlled Trials and MEDLINE databases (1966 to February 2022) for studies on PSP published in English using 'pancreatic stone protein', 'PSP', 'regenerative protein', 'lithostatin' combined with 'infection' and 'sepsis' found 46 records. The search was restricted to the five trials that measured PSP using the enzyme-linked immunosorbent assay technique (ELISA). We used Bayesian hierarchical regression models for pooled estimates and to predict mortality or disease severity using PSP, C-Reactive Protein (CRP) and procalcitonin (PCT) as main predictor. We used statistical discriminative measures, such as the area under the receiver operating characteristic curve (AUC) and classification plots.

RESULTS

Among the 678 patients included, the pooled ICU mortality was 17.8% (95% prediction interval 4.1% to 54.6%) with a between-study heterogeneity (I-squared 87%). PSP was strongly associated with ICU mortality (OR = 2.7, 95% credible interval (CrI) [1.3-6.0] per one standard deviation increase; age, gender and sepsis severity adjusted OR = 1.5, 95% CrI [0.98-2.8]). The AUC was 0.69 for PSP 95% confidence interval (CI) [0.64-0.74], 0.61 [0.56-0.66] for PCT and 0.52 [0.47-0.57] for CRP. The sensitivity was 0.96, 0.52, 0.30 for risk thresholds 0.1, 0.2 and 0.3; respective false positive rate values were 0.84, 0.25, 0.10.

CONCLUSIONS

We found that PSP showed a very good discriminative ability for both investigated study endpoints ICU mortality and infection severity; better in comparison to CRP, similar to PCT. Combinations of biomarkers did not improve their predictive ability.

Prognosis and prediction of antibiotic benefit in adults with clinically diagnosed acute rhinosinusitis: an individual participant data meta-analysis

BACKGROUND

A previous individual participant data meta-analysis (IPD-MA) of antibiotics for adults with clinically diagnosed acute rhinosinusitis (ARS) showed a marginal overall effect of antibiotics, but was unable to identify patients that are most likely to benefit from antibiotics when applying conventional (i.e. univariable or one-variable-at-a-time) subgroup analysis. We updated the systematic review and investigated whether multivariable prediction of patient-level prognosis and antibiotic treatment effect may lead to more tailored treatment assignment in adults presenting to primary care with ARS.

METHODS

An IPD-MA of nine double-blind placebo-controlled trials of antibiotic treatment (n=2539) was conducted, with the probability of being cured at 8-15 days as the primary outcome. A logistic mixed effects model was developed to predict the probability of being cured based on demographic characteristics, signs and symptoms, and antibiotic treatment assignment. Predictive performance was quantified based on internal-external cross-validation in terms of calibration and discrimination performance, overall model fit, and the accuracy of individual predictions.

RESULTS

Results indicate that the prognosis with respect to risk of cure could not be reliably predicted (c-statistic 0.58 and Brier score 0.24). Similarly, patient-level treatment effect predictions did not reliably distinguish between those that did and did not benefit from antibiotics (c-for-benefit 0.50).

CONCLUSIONS

In conclusion, multivariable prediction based on patient demographics and common signs and symptoms did not reliably predict the patient-level probability of cure and antibiotic effect in this IPD-MA. Therefore, these characteristics cannot be expected to reliably distinguish those that do and do not benefit from antibiotics in adults presenting to primary care with ARS.

Development and External Validation of the International Early Warning Score for Improved Age- and Sex-Adjusted In-Hospital Mortality Prediction in the Emergency Department

OBJECTIVES

Early Warning Scores (EWSs) have a great potential to assist clinical decision-making in the emergency department (ED). However, many EWS contain methodological weaknesses in development and validation and have poor predictive performance in older patients. The aim of this study was to develop and externally validate an International Early Warning Score (IEWS) based on a recalibrated National Early warning Score (NEWS) model including age and sex and evaluate its performance independently at arrival to the ED in three age categories (18-65, 66-80, > 80 yr).

DESIGN

International multicenter cohort study.

SETTING

Data was used from three Dutch EDs. External validation was performed in two EDs in Denmark.

PATIENTS

All consecutive ED patients greater than or equal to 18 years in the Netherlands Emergency department Evaluation Database (NEED) with at least two registered vital signs were included, resulting in 95,553 patients. For external validation, 14,809 patients were included from a Danish Multicenter Cohort (DMC).

MEASUREMENTS AND MAIN RESULTS

Model performance to predict in-hospital mortality was evaluated by discrimination, calibration curves and summary statistics, reclassification, and clinical usefulness by decision curve analysis. In-hospital mortality rate was 2.4% ( n = 2,314) in the NEED and 2.5% ( n = 365) in the DMC. Overall, the IEWS performed significantly better than NEWS with an area under the receiving operating characteristic of 0.89 (95% CIs, 0.89-0.90) versus 0.82 (0.82-0.83) in the NEED and 0.87 (0.85-0.88) versus 0.82 (0.80-0.84) at external validation. Calibration for NEWS predictions underestimated risk in older patients and overestimated risk in the youngest, while calibration improved for IEWS with a substantial reclassification of patients from low to high risk and a standardized net benefit of 5-15% in the relevant risk range for all age categories.

CONCLUSIONS

The IEWS substantially improves in-hospital mortality prediction for all ED patients greater than or equal to18 years.

Embracing cohort heterogeneity in clinical machine learning development: a step toward generalizable models

This study is a simple illustration of the benefit of averaging over cohorts, rather than developing a prediction model from a single cohort. We show that models trained on data from multiple cohorts can perform significantly better in new settings than models based on the same amount of training data but from just a single cohort. Although this concept seems simple and obvious, no current prediction model development guidelines recommend such an approach.

Measuring the performance of prediction models to personalize treatment choice

When data are available from individual patients receiving either a treatment or a control intervention in a randomized trial, various statistical and machine learning methods can be used to develop models for predicting future outcomes under the two conditions, and thus to predict treatment effect at the patient level. These predictions can subsequently guide personalized treatment choices. Although several methods for validating prediction models are available, little attention has been given to measuring the performance of predictions of personalized treatment effect. In this article, we propose a range of measures that can be used to this end. We start by defining two dimensions of model accuracy for treatment effects, for a single outcome: discrimination for benefit and calibration for benefit. We then amalgamate these two dimensions into an additional concept, decision accuracy, which quantifies the model's ability to identify patients for whom the benefit from treatment exceeds a given threshold. Subsequently, we propose a series of performance measures related to these dimensions and discuss estimating procedures, focusing on randomized data. Our methods are applicable for continuous or binary outcomes, for any type of prediction model, as long as it uses baseline covariates to predict outcomes under treatment and control. We illustrate all methods using two simulated datasets and a real dataset from a trial in depression. We implement all methods in the R package predieval. Results suggest that the proposed measures can be useful in evaluating and comparing the performance of competing models in predicting individualized treatment effect.

There is no such thing as a validated prediction model

BACKGROUND

Clinical prediction models should be validated before implementation in clinical practice. But is favorable performance at internal validation or one external validation sufficient to claim that a prediction model works well in the intended clinical context?

MAIN BODY

We argue to the contrary because (1) patient populations vary, (2) measurement procedures vary, and (3) populations and measurements change over time. Hence, we have to expect heterogeneity in model performance between locations and settings, and across time. It follows that prediction models are never truly validated. This does not imply that validation is not important. Rather, the current focus on developing new models should shift to a focus on more extensive, well-conducted, and well-reported validation studies of promising models.

CONCLUSION

Principled validation strategies are needed to understand and quantify heterogeneity, monitor performance over time, and update prediction models when appropriate. Such strategies will help to ensure that prediction models stay up-to-date and safe to support clinical decision-making.

Assessing Performance and Clinical Usefulness in Prediction Models With Survival Outcomes: Practical Guidance for Cox Proportional Hazards Models

Risk prediction models need thorough validation to assess their performance. Validation of models for survival outcomes poses challenges due to the censoring of observations and the varying time horizon at which predictions can be made. This article describes measures to evaluate predictions and the potential improvement in decision making from survival models based on Cox proportional hazards regression.

As a motivating case study, the authors consider the prediction of the composite outcome of recurrence or death (the "event") in patients with breast cancer after surgery. They developed a simple Cox regression model with 3 predictors, as in the Nottingham Prognostic Index, in 2982 women (1275 events over 5 years of follow-up) and externally validated this model in 686 women (285 events over 5 years). Improvement in performance was assessed after the addition of progesterone receptor as a prognostic biomarker.

The model predictions can be evaluated across the full range of observed follow-up times or for the event occurring by the end of a fixed time horizon of interest. The authors first discuss recommended statistical measures that evaluate model performance in terms of discrimination, calibration, or overall performance. Further, they evaluate the potential clinical utility of the model to support clinical decision making according to a net benefit measure. They provide SAS and R code to illustrate internal and external validation.

The authors recommend the proposed set of performance measures for transparent reporting of the validity of predictions from survival models.

Accommodating population differences when validating risk prediction models

Validation of risk prediction models in independent data provides a more rigorous assessment of model performance than internal assessment, for example, done by cross-validation in the data used for model development. However, several differences between the populations that gave rise to the training and the validation data can lead to seemingly poor performance of a risk model. In this paper we formalize the notions of "similarity" or "relatedness" of the training and validation data, and define reproducibility and transportability. We address the impact of different distributions of model predictors and differences in verifying the disease status or outcome on measures of calibration, accuracy and discrimination of a model. When individual level information from both the training and validation data sets is available, we propose and study weighted versions of the validation metrics that adjust for differences in the risk factor distributions and in outcome verification between the training and validation data to provide a more comprehensive assessment of model performance. We provide conditions on the risk model and the populations that gave rise to the training and validation data that ensure a model's reproducibility or transportability, and show how to check these conditions using weighted and unweighted performance measures. We illustrate the method by developing and validating a model that predicts the risk of developing prostate cancer using data from two large prostate cancer screening trials.

Development and validation of clinical prediction models for breast cancer incidence and mortality: a protocol for a dual cohort study

INTRODUCTION

Breast cancer is the most common cancer and the leading cause of cancer-related death in women worldwide. Risk prediction models may be useful to guide risk-reducing interventions (such as pharmacological agents) in women at increased risk or inform screening strategies for early detection methods such as screening.

METHODS AND ANALYSIS

The study will use data for women aged 20-90 years between 2000 and 2020 from QResearch linked at the individual level to hospital episodes, cancer registry and death registry data. It will evaluate a set of modelling approaches to predict the risk of developing breast cancer within the next 10 years, the 'combined' risk of developing a breast cancer and then dying from it within 10 years, and the risk of breast cancer mortality within 10 years of diagnosis. Cox proportional hazards, competing risks, random survival forest, deep learning and XGBoost models will be explored. Models will be developed on the entire dataset, with 'apparent' performance reported, and internal-external cross-validation used to assess performance and geographical and temporal transportability (two 10-year time periods). Random effects meta-analysis will pool discrimination and calibration metric estimates from individual geographical units obtained from internal-external cross-validation. We will then externally validate the models in an independent dataset. Evaluation of performance heterogeneity will be conducted throughout, such as exploring performance across ethnic groups.

ETHICS AND DISSEMINATION

Ethics approval was granted by the QResearch scientific committee (reference number REC 18/EM/0400: OX129). The results will be written up for submission to peer-reviewed journals.

Individual participant data meta-analysis with mixed-effects transformation models

One-stage meta-analysis of individual participant data (IPD) poses several statistical and computational challenges. For time-to-event outcomes, the approach requires the estimation of complicated nonlinear mixed-effects models that are flexible enough to realistically capture the most important characteristics of the IPD. We present a model class that incorporates general normally distributed random effects into linear transformation models. We discuss extensions to model between-study heterogeneity in baseline risks and covariate effects and also relax the assumption of proportional hazards. Within the proposed framework, data with arbitrary random censoring patterns can be handled. The accompanying \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} }{}$\textsf{R}$\end{document} package tramME utilizes the Laplace approximation and automatic differentiation to perform efficient maximum likelihood estimation and inference in mixed-effects transformation models. We compare several variants of our model to predict the survival of patients with chronic obstructive pulmonary disease using a large data set of prognostic studies. Finally, a simulation study is presented that verifies the correctness of the implementation and highlights its efficiency compared to an alternative approach.

A meta-analysis of XRCC1 single nucleotide polymorphism and susceptibility to gynecological malignancies

BACKGROUND

Gynecological malignant tumor is a serious threat to women's health, cervical cancer, endometrial cancer and ovarian cancer are the most common. The eponymous protein encoded by the XRCC1 (X-ray repair cross complementation 1) gene is an important functional protein in the process of single-stranded DNA damage. Non-synonymous mutations of XRCC1 gene cause amino acid sequence changes that affect protein function and DNA repair ability, and may affect the interaction with other DNA repair proteins, leading to increased risk of tumor development. Many studies have assessed the association between XRCC1 gene polymorphism and the risk of cancer in the female reproductive system, but the results have been inconclusive. In this study, the relationship between XRCC1 Arg399Gln, Arg194Trp, Arg280His single nucleotide polymorphisms and susceptibility to gynecological malignancies was further explored by meta-analysis.

METHODS

English database: Pubmed, Medline, Excerpta Medica Database, Cochrance, etc; Chinese database: China national knowledge infrastructure, Wanfang Database, etc. STATA14 was used for statistical analysis, such as odd ratio (OR) value, subgroup analysis, heterogeneity test, sensitivity analysis, and publication bias.

RESULTS

In gynecologic cancers, the allele frequency difference of Arg399Gln case control group was statistically significant (GvsA: P = .007). There was no significant difference in allele frequency in the Arg194Trp and Arg280His case control groups (P = .065, 0.198). In different gene models, Arg399Gln was significantly correlated with gynecologic cancers susceptibility (GGvs AA: OR 0.91; 95% confidence interval [CI], 0.85 0.98); Arg194Trp was significantly correlated with gynecologic cancers susceptibility (CCvs TT: OR 0.94; 95% CI 0.88,1.00; CCvs CT: OR 0.97; 95% CI 0.90, 1.05); Arg280His was significantly correlated with gynecologic cancers susceptibility (GGvs AA: OR 0.98; 95% CI 0.94, 1.02; GGvs GA: OR 1.00;95% CI 0.97, 1.04). In the subgroup analysis, Arg399Gln and Arg194Trp were significantly correlated with gynecologic cancers susceptibility in the Asian race (P = .000, 0.049). In the analysis of different cancer subgroups, Arg399Gln and cervical cancer susceptibility were statistically significant (P = .039). Arg194Trp and endometrial cancer susceptibility were statistically significant (P = .033, 0.001).

CONCLUSIONS

XRCC1 Arg399Gln, Arg194Trp, Arg280His single nucleotide polymorphisms were associated with gynecologic cancer susceptibility. Arg399Gln genotype was statistically significant in relation to cervical cancer susceptibility. Arg194Trp genotype was statistically significant in relation to endometrial cancer susceptibility.

Meta-analysis for individual participant data with a continuous exposure: A case study

OBJECTIVE

Methods for meta-analysis of studies with individual participant data and continuous exposure variables are well described in the statistical literature but are not widely used in clinical and epidemiological research. The purpose of this case study is to make the methods more accessible.

STUDY DESIGN AND SETTING

A two-stage process is demonstrated. Response curves are estimated separately for each study using fractional polynomials. The study-specific curves are then averaged pointwise over all studies at each value of the exposure. The averaging can be implemented using fixed effects or random effects methods.

RESULTS

The methodology is illustrated using samples of real data with continuous outcome and exposure data and several covariates. The sample data set, segments of Stata and R code, and outputs are provided to enable replication of the results.

CONCLUSION

These methods and tools can be adapted to other situations, including for time-to-event or categorical outcomes, different ways of modelling exposure-outcome curves, and different strategies for covariate adjustment.

Systematic review and meta-analysis of mortality risk prediction models in adult cardiac surgery

OBJECTIVES

The most used mortality risk prediction models in cardiac surgery are the European System for Cardiac Operative Risk Evaluation (ES) and Society of Thoracic Surgeons (STS) score. There is no agreement on which score should be considered more accurate nor which score should be utilized in each population subgroup. We sought to provide a thorough quantitative assessment of these 2 models.

METHODS

We performed a systematic literature review and captured information on discrimination, as quantified by the area under the receiver operator curve (AUC), and calibration, as quantified by the ratio of observed-to-expected mortality (O:E). We performed random effects meta-analysis of the performance of the individual models as well as pairwise comparisons and subgroup analysis by procedure type, time and continent.

RESULTS

The ES2 {AUC 0.783 [95% confidence interval (CI) 0.765-0.800]; O:E 1.102 (95% CI 0.943-1.289)} and STS [AUC 0.757 (95% CI 0.727-0.785); O:E 1.111 (95% CI 0.853-1.447)] showed good overall discrimination and calibration. There was no significant difference in the discrimination of the 2 models (difference in AUC -0.016; 95% CI -0.034 to -0.002; P = 0.09). However, the calibration of ES2 showed significant geographical variations (P < 0.001) and a trend towards miscalibration with time (P=0.057). This was not seen with STS.

CONCLUSIONS

ES2 and STS are reliable predictors of short-term mortality following adult cardiac surgery in the populations from which they were derived. STS may have broader applications when comparing outcomes across continents as compared to ES2.

REGISTRATION

Prospero (https://www.crd.york.ac.uk/PROSPERO/) CRD42020220983.

A NICE combination for predicting hospitalisation at the Emergency Department: a European multicentre observational study of febrile children

Background

Prolonged Emergency Department (ED) stay causes crowding and negatively impacts quality of care. We developed and validated a prediction model for early identification of febrile children with a high risk of hospitalisation in order to improve ED flow.

Methods

The MOFICHE study prospectively collected data on febrile children (0-18 years) presenting to 12 European EDs. A prediction models was constructed using multivariable logistic regression and included patient characteristics available at triage. We determined the discriminative values of the model by calculating the area under the receiver operating curve (AUC).

Findings

Of 38,424 paediatric encounters, 9,735 children were admitted to the ward and 157 to the PICU. The prediction model, combining patient characteristics and NICE alarming, yielded an AUC of 0.84 (95%CI 0.83-0.84).The model performed well for a rule-in threshold of 75% (specificity 99.0% (95%CI 98.9-99.1%, positive likelihood ratio 15.1 (95%CI 13.4-17.1), positive predictive value 0.84 (95%CI 0.82-0.86)) and a rule-out threshold of 7.5% (sensitivity 95.4% (95%CI 95.0-95.8), negative likelihood ratio 0.15 (95%CI 0.14-0.16), negative predictive value 0..95 (95%CI 0.95-9.96)). Validation in a separate dataset showed an excellent AUC of 0.91 (95%CI 0.90- 0.93). The model performed well for identifying children needing PICU admission (AUC 0.95, 95%CI 0.93-0.97). A digital calculator was developed to facilitate clinical use.

Interpretation

Patient characteristics and NICE alarming signs available at triage can be used to identify febrile children at high risk for hospitalisation and can be used to improve ED flow.

Funding

European Union, NIHR, NHS.

Prognostic Research in Traumatic Brain Injury: Markers, Modeling, and Methodological Principles

Prognostic assessment in traumatic brain injury (TBI) is embedded deeply in clinical care. Considering the limitations of current prognostic indicators, there is increasing interest in understanding the role of new biomarkers, and in finding other prognostic indicators of long-term outcomes following TBI. New prognostic indicators may result in the development of more accurate prediction models that could be useful for both risk stratification and clinical decision making. We aimed to review methodological issues and provide tentative guidelines for prognostic research in TBI. Prognostic factor research focuses on the role of a specific patient or disease-related characteristic in relation to outcome. Typically, univariable relations of the prognostic factor are studied, followed by analyses adjusting for other variables related to the outcome. Following existing guidelines, we emphasize the importance of transparent reporting of patient and specimen characteristics, study design, clinical end-points, and statistical analysis. Prognostic model research considers combinations of predictors, with challenges for model specification, estimation, evaluation, validation, and presentation. We highlight modern approaches and opportunities related to missing values, exploration of non-linear effects, and assessing between-study heterogeneity. Prognostic research in TBI can be improved if key methodological principles are adhered to and when research is performed in collaboration among multiple centers to ensure generalizability.

Predicting hospital admissions from individual patient data (IPD): an applied example to explore key elements driving external validity

OBJECTIVE

To explore factors that potentially impact external validation performance while developing and validating a prognostic model for hospital admissions (HAs) in complex older general practice patients.

STUDY DESIGN AND SETTING

Using individual participant data from four cluster-randomised trials conducted in the Netherlands and Germany, we used logistic regression to develop a prognostic model to predict all-cause HAs within a 6-month follow-up period. A stratified intercept was used to account for heterogeneity in baseline risk between the studies. The model was validated both internally and by using internal-external cross-validation (IECV).

RESULTS

Prior HAs, physical components of the health-related quality of life comorbidity index, and medication-related variables were used in the final model. While achieving moderate discriminatory performance, internal bootstrap validation revealed a pronounced risk of overfitting. The results of the IECV, in which calibration was highly variable even after accounting for between-study heterogeneity, agreed with this finding. Heterogeneity was equally reflected in differing baseline risk, predictor effects and absolute risk predictions.

CONCLUSIONS

Predictor effect heterogeneity and differing baseline risk can explain the limited external performance of HA prediction models. With such drivers known, model adjustments in external validation settings (eg, intercept recalibration, complete updating) can be applied more purposefully.

TRIAL REGISTRATION NUMBER

PROSPERO id: CRD42018088129.

A clinical prediction model to identify children at risk for revisits with serious illness to the emergency department: A prospective multicentre observational study

BACKGROUND

To develop a clinical prediction model to identify children at risk for revisits with serious illness to the emergency department.

METHODS AND FINDINGS

A secondary analysis of a prospective multicentre observational study in five European EDs (the TRIAGE study), including consecutive children aged <16 years who were discharged following their initial ED visit ('index' visit), in 2012-2015. Standardised data on patient characteristics, Manchester Triage System urgency classification, vital signs, clinical interventions and procedures were collected. The outcome measure was serious illness defined as hospital admission or PICU admission or death in ED after an unplanned revisit within 7 days of the index visit. Prediction models were developed using multivariable logistic regression using characteristics of the index visit to predict the likelihood of a revisit with a serious illness. The clinical model included day and time of presentation, season, age, gender, presenting problem, triage urgency, and vital signs. An extended model added laboratory investigations, imaging, and intravenous medications. Cross validation between the five sites was performed, and discrimination and calibration were assessed using random effects models. A digital calculator was constructed for clinical implementation. 7,891 children out of 98,561 children had a revisit to the ED (8.0%), of whom 1,026 children (1.0%) returned to the ED with a serious illness. Rates of revisits with serious illness varied between the hospitals (range 0.7-2.2%). The clinical model had a summary Area under the operating curve (AUC) of 0.70 (95% CI 0.65-0.74) and summary calibration slope of 0.83 (95% CI 0.67-0.99). 4,433 children (5%) had a risk of > = 3%, which was useful for ruling in a revisit with serious illness, with positive likelihood ratio 4.41 (95% CI 3.87-5.01) and specificity 0.96 (95% CI 0.95-0.96). 37,546 (39%) had a risk <0.5%, which was useful for ruling out a revisit with serious illness (negative likelihood ratio 0.30 (95% CI 0.25-0.35), sensitivity 0.88 (95% CI 0.86-0.90)). The extended model had an improved summary AUC of 0.71 (95% CI 0.68-0.75) and summary calibration slope of 0.84 (95% CI 0.71-0.97). As study limitations, variables on ethnicity and social deprivation could not be included, and only return visits to the original hospital and not to those of surrounding hospitals were recorded.

CONCLUSION

We developed a prediction model and a digital calculator which can aid physicians identifying those children at highest and lowest risks for developing a serious illness after initial discharge from the ED, allowing for more targeted safety netting advice and follow-up.

Large-scale validation of the prediction model risk of bias assessment Tool (PROBAST) using a short form: high risk of bias models show poorer discrimination

OBJECTIVE

To assess whether the Prediction model Risk Of Bias ASsessment Tool (PROBAST) and a shorter version of this tool can identify clinical prediction models (CPMs) that perform poorly at external validation.

STUDY DESIGN AND SETTING

We evaluated risk of bias (ROB) on 102 CPMs from the Tufts CPM Registry, comparing PROBAST to a short form consisting of six PROBAST items anticipated to best identify high ROB. We then applied the short form to all CPMs in the Registry with at least 1 validation (n=556) and assessed the change in discrimination (dAUC) in external validation cohorts (n=1,147).

RESULTS

PROBAST classified 98/102 CPMS as high ROB. The short form identified 96 of these 98 as high ROB (98% sensitivity), with perfect specificity. In the full CPM registry, 527 of 556 CPMs (95%) were classified as high ROB, 20 (3.6%) low ROB, and 9 (1.6%) unclear ROB. Only one model with unclear ROB was reclassified to high ROB after full PROBAST assessment of all low and unclear ROB models. Median change in discrimination was significantly smaller in low ROB models (dAUC -0.9%, IQR -6.2-4.2%) compared to high ROB models (dAUC -11.7%, IQR -33.3-2.6%; P<0.001).

CONCLUSION

High ROB is pervasive among published CPMs. It is associated with poor discriminative performance at validation, supporting the application of PROBAST or a shorter version in CPM reviews.

Individual participant data systematic reviews with meta-analyses of psychotherapies for borderline personality disorder

INTRODUCTION

The heterogeneity in people with borderline personality disorder (BPD) and the range of specialised psychotherapies means that people with certain BPD characteristics might benefit more or less from different types of psychotherapy. Identifying moderating characteristics of individuals is a key to refine and tailor standard treatments so they match the specificities of the individual participant. The objective of this is to improve the quality of care and the individual outcomes. We will do so by performing three systematic reviews with meta-analyses of individual participant data (IPD). The aim of these reviews is to investigate potential predictors and moderating patient characteristics on treatment outcomes for patients with BPD.

METHODS AND ANALYSIS

We performed comprehensive searches in 22 databases and trial registries up to October 6th 2020. These will be updated with a top-up search up until June 2021. Our primary meta-analytic method will be the one-stage random-effects approach. To identify predictors, we will use the one-stage model that accounts for interaction between covariates and treatment allocation. Heterogeneity in case-mix will be assessed with a membership model based on a multinomial logistic regression where study membership is the outcome. A random-effects meta-analysis is chosen to account for expected levels of heterogeneity.

ETHICS AND DISSEMINATION

The statistical analyses will be conducted on anonymised data that have already been approved by the respective ethical committees that originally assessed the included trials. The three IPD reviews will be published in high-impact factor journals and their results will be presented at international conferences and national seminars.

PROSPERO REGISTRATION NUMBER

CRD42021210688.

Accuracy of pancreatic stone protein for the diagnosis of infection in hospitalized adults: a systematic review and individual patient level meta-analysis

Background

Accurate biomarkers to diagnose infection are lacking. Studies reported good performance of pancreatic stone protein (PSP) to detect infection. The objective of the study was to determine the performance of PSP in diagnosing infection across hospitalized patients and calculate a threshold value for that purpose.

Methods

A systematic search across Cochrane Central Register of Controlled Trials and MEDLINE databases (1966–March 2019) for studies on PSP published in English using ‘pancreatic stone protein’, ‘PSP’, ‘regenerative protein’, ‘lithostatin’ combined with ‘infection’ and ‘sepsis’ found 44 records. The search was restricted to the five trials that evaluated PSP for the initial detection of infection in hospitalized adults. Individual patient data were obtained from the investigators of all eligible trials. Data quality and validity was assessed according to PRISMA guidelines. We choose a fixed-effect model to calculate the PSP cut-off value that best discriminates infected from non-infected patients.

Results

Infection was confirmed in 371 of 631 patients. The median (IQR) PSP value of infected versus uninfected patients was 81.5 (30.0–237.5) versus 19.2 (12.6–33.57) ng/ml, compared to 150 (82.70–229.55) versus 58.25 (15.85–120) mg/l for C-reactive protein (CRP) and 0.9 (0.29–4.4) versus 0.15 (0.08–0.5) ng/ml for procalcitonin (PCT). Using a PSP cut-off of 44.18 ng/ml, the ROC AUC to detect infection was 0.81 (0.78–0.85) with a sensitivity of 0.66 (0.61–0.71), specificity of 0.83 (0.78–0.88), PPV of 0.85 (0.81–0.89) and NPV of 0.63 (0.58–0.68). When a model combining PSP and CRP was used, the ROC AUC improved to 0.90 (0.87–0.92) with higher sensitivity 0.81 (0.77–0.85) and specificity 0.84 (0.79–0.90) for discriminating infection from non-infection. Adding PCT did not improve the performance further.

Conclusions

PSP is a promising biomarker to diagnose infections in hospitalized patients. Using a cut-off value of 44.18 ng/ml, PSP performs better than CRP or PCT across the considered studies. The combination of PSP with CRP further enhances its accuracy.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13054-021-03609-2.

Machine learning for developing a prediction model of hospital admission of emergency department patients: Hype or hope?

OBJECTIVE

Early identification of emergency department (ED) patients who need hospitalization is essential for quality of care and patient safety. We aimed to compare machine learning (ML) models predicting the hospitalization of ED patients and conventional regression techniques at three points in time after ED registration.

METHODS

We analyzed consecutive ED patients of three hospitals using the Netherlands Emergency Department Evaluation Database (NEED). We developed prediction models for hospitalization using an increasing number of data available at triage, ∼30 min (including vital signs) and ∼2 h (including laboratory tests) after ED registration, using ML (random forest, gradient boosted decision trees, deep neural networks) and multivariable logistic regression analysis (including spline transformations for continuous predictors). Demographics, urgency, presenting complaints, disease severity and proxies for comorbidity, and complexity were used as covariates. We compared the performance using the area under the ROC curve in independent validation sets from each hospital.

RESULTS

We included 172,104 ED patients of whom 66,782 (39 %) were hospitalized. The AUC of the multivariable logistic regression model was 0.82 (0.78-0.86) at triage, 0.84 (0.81-0.86) at ∼30 min and 0.83 (0.75-0.92) after ∼2 h. The best performing ML model over time was the gradient boosted decision trees model with an AUC of 0.84 (0.77-0.88) at triage, 0.86 (0.82-0.89) at ∼30 min and 0.86 (0.74-0.93) after ∼2 h.

CONCLUSIONS

Our study showed that machine learning models had an excellent but similar predictive performance as the logistic regression model for predicting hospital admission. In comparison to the 30-min model, the 2-h model did not show a performance improvement. After further validation, these prediction models could support management decisions by real-time feedback to medical personal.

Developing more generalizable prediction models from pooled studies and large clustered data sets

Prediction models often yield inaccurate predictions for new individuals. Large data sets from pooled studies or electronic healthcare records may alleviate this with an increased sample size and variability in sample characteristics. However, existing strategies for prediction model development generally do not account for heterogeneity in predictor‐outcome associations between different settings and populations. This limits the generalizability of developed models (even from large, combined, clustered data sets) and necessitates local revisions. We aim to develop methodology for producing prediction models that require less tailoring to different settings and populations. We adopt internal‐external cross‐validation to assess and reduce heterogeneity in models' predictive performance during the development. We propose a predictor selection algorithm that optimizes the (weighted) average performance while minimizing its variability across the hold‐out clusters (or studies). Predictors are added iteratively until the estimated generalizability is optimized. We illustrate this by developing a model for predicting the risk of atrial fibrillation and updating an existing one for diagnosing deep vein thrombosis, using individual participant data from 20 cohorts (N = 10 873) and 11 diagnostic studies (N = 10 014), respectively. Meta‐analysis of calibration and discrimination performance in each hold‐out cluster shows that trade‐offs between average and heterogeneity of performance occurred. Our methodology enables the assessment of heterogeneity of prediction model performance during model development in multiple or clustered data sets, thereby informing researchers on predictor selection to improve the generalizability to different settings and populations, and reduce the need for model tailoring. Our methodology has been implemented in the R package metamisc.

A tutorial on calibration measurements and calibration models for clinical prediction models

Our primary objective is to provide the clinical informatics community with an introductory tutorial on calibration measurements and calibration models for predictive models using existing R packages and custom implemented code in R on real and simulated data. Clinical predictive model performance is commonly published based on discrimination measures, but use of models for individualized predictions requires adequate model calibration. This tutorial is intended for clinical researchers who want to evaluate predictive models in terms of their applicability to a particular population. It is also for informaticians and for software engineers who want to understand the role that calibration plays in the evaluation of a clinical predictive model, and to provide them with a solid starting point to consider incorporating calibration evaluation and calibration models in their work. Covered topics include (1) an introduction to the importance of calibration in the clinical setting, (2) an illustration of the distinct roles that discrimination and calibration play in the assessment of clinical predictive models, (3) a tutorial and demonstration of selected calibration measurements, (4) a tutorial and demonstration of selected calibration models, and (5) a brief discussion of limitations of these methods and practical suggestions on how to use them in practice.

Effect of mannose-binding lectin gene polymorphisms on the risk of rheumatoid arthritis: Evidence from a meta-analysis

BACKGROUND

The effect of mannose-binding lectin (MBL) gene polymorphisms on susceptibility of rheumatoid arthritis (RA) were evaluated in ethnically different populations, whereas the results were always inconsistent.

MATERIALS AND METHODS

Fourteen articles involving 36 datasets were recruited to evaluate the association between MBL gene polymorphisms and rheumatoid arthritis in a meta-analysis. The random or fixed effect models were used to evaluate the pooled odds ratios (ORs) and their corresponding 95% confidence intervals (CIs).

RESULTS

Stratified analysis by ethnicities was conducted and the result revealed that rs1800450 (T vs C, OR = 1.32, 95% CI: 1.04-1.67, P < .05) and MBL-A/O (T vs C, OR = 1.20, 95% CI: 1.08-1.34, P < .001) were strongly associated with RA in Brazilian populations. In addition, the significant relationship between rs11003125 (T vs C, OR = 1.16, 95% CI: 1.06-1.26, P < .05) with RA were also observed in East Asian populations. Meanwhile, the inverse associations between rs5030737 with RA in East Asians and rs1800450 with RA in Indians were acquired. However, no association between any MBL polymorphism with RA susceptibility was confirmed in Caucasians.

CONCLUSIONS

The structural polymorphisms in exon 1 of MBL gene may significantly contribute to susceptibility and development of RA in Brazilian and Indian populations, whereas the functional polymorphisms in the promoter region were more likely to associate with RA in East Asians.

Missing Data in Prediction Research: A Five-Step Approach for Multiple Imputation, Illustrated in the CENTER-TBI Study

In medical research, missing data is common. In acute diseases, such as traumatic brain injury (TBI), even well-conducted prospective studies may suffer from missing data in baseline characteristics and outcomes. Statistical models may simply drop patients with any missing values, potentially leaving a selected subset of the original cohort. Imputation is widely accepted by methodologists as an appropriate way to deal with missing data. We aim to provide practical guidance on handling missing data for prediction modeling. We hereto propose a five-step approach, centered around single and multiple imputation: 1) explore the missing data patterns; 2) choose a method of imputation; 3) perform imputation; 4) assess diagnostics of the imputation; and 5) analyze the imputed data sets. We illustrate these five steps with the estimation and validation of the IMPACT (International Mission on Prognosis and Analysis of Clinical Trials in Traumatic Brain Injury) prognostic model in 1375 patients from the CENTER-TBI database, included in 53 centers across 17 countries, with moderate or severe TBI in the prospective European CENTER-TBI study. Future prediction modeling studies in acute diseases may benefit from following the suggested five steps for optimal statistical analysis and interpretation, after maximal effort has been made to minimize missing data.

Higher Risk of Depression After Total Mastectomy Versus Breast Reconstruction Among Adult Women With Breast Cancer: A Systematic Review and Metaregression

This systematic review with a meta-regression was conducted to determine the risk of depression after mastectomy compared to breast reconstruction among women with breast cancer 1 year after surgery. A literature search was conducted according to PRISMA guidelines using 4 databases: Medline (Ovid), Embase, Cinahl, and the Cochrane Library for the period January 2000 to March 2019. Studies that measured the status of depression within 1 year and immediately after surgery were included. Outcomes related to depression were analyzed by using a pool of event rates and a risk ratio of 95% confidence interval (CI), P value, and a fitting model based on the results of a heterogeneity test of mastectomy and BR. The statistical analysis was conducted using Comprehensive Meta-analysis 3.0 software. Nine studies met the inclusion criteria. There were 865 cases of mastectomy only, with a 22.2% risk of depression (95% CI, 12.4-36.2). In 869 women who underwent BR, the risk of depression was 15.7% (95% CI, 8.8-26.2). The depression risk ratio for mastectomy compared to BR was 1.36 (95% CI, 1.11-1.65). Patients with delayed reconstruction exhibited lower levels of depression (risk ratio 0.96, 95% CI 0.57-1.01). The Beck Depression Inventory (BDI) scale showed high sensitivity, and the Hospital Anxiety Depression Scale (HADS) with a cutoff of > 7 could measure even low to moderate depressive symptoms. One in 4 women with breast cancer had symptoms of depression after mastectomy; both surgeries were associated with depression in women 1 year after surgery. Our results will permit the development of proactive treatment plans before and after surgery to mitigate risk and prevent depression through the use of sensitive depression scales like BDI.

Can personalized treatment prediction improve the outcomes, compared with the group average approach, in a randomized trial? Developing and validating a multivariable prediction model in a pragmatic megatrial of acute treatment for major depression

BACKGROUND

Clinical trials have traditionally been analysed at the aggregate level, assuming that the group average would be applicable to all eligible and similar patients. We re-analyzed a mega-trial of antidepressant therapy for major depression to explore whether a multivariable prediction model may lead to different treatment recommendations for individual participants.

METHODS

The trial compared the second-line treatment strategies of continuing sertraline, combining it with mirtazapine or switching to mirtazapine after initial failure to remit on sertraline among 1,544 patients with major depression. The outcome was the Personal Health Questionnaire-9 (PHQ-9) at week 9: the original analyses showed that both combining and switching resulted in greater reduction in PHQ-9 by 1.0 point than continuing. We considered several models of penalized regression or machine learning.

RESULTS

Models using support vector machines (SVMs) provided the best performance. Using SVMs, continuing sertraline was predicted to be the best treatment for 123 patients, combining for 696 patients, and switching for 725 patients. In the last two subgroups, both combining and switching were equally superior to continuing by 1.2 to 1.4 points, resulting in the same treatment recommendations as with the original aggregate data level analyses; in the first subgroup, however, switching was substantively inferior to combining (-3.1, 95%CI: -5.4 to -0.5).

LIMITATIONS

Stronger predictors are needed to make more precise predictions.

CONCLUSIONS

The multivariable prediction models led to improved recommendations for a minority of participants than the group average approach in a megatrial.

Value of soluble fms-like tyrosine 1 in early prediction of severity of acute pancreatitis: A systematic review and meta-analysis

BACKGROUND

Biomarkers for the early prediction of the severity of acute pancreatitis (AP) are urgently needed for clinical management of this disease. Soluble fms-like tyrosine 1 (sFlt-1), one of the vascular endothelial growth factor receptors, has been found to be associated with various diseases, including AP.

AIM

To summarize all the relevant literature to determine the overall clinical value of sFlt-1 in the early diagnosis of severity of AP by meta-analysis.

METHODS

CNKI, Wanfang Database, Chinese BioMedicine Database, WEIPU database, PubMed, Cochrane Library, and EMBASE database were searched systematically. The time range was from the inception of the database to 15 February 2020. Eligible cohort studies on the early predictive value of sFlt-1 for AP of different severities were collected. Quality assessment and data extraction were performed in those clinical trials in line with the inclusion criteria. Stata software was applied to carry out meta-analysis.

RESULTS

A total of 7 articles reporting 8 case-control studies were included in the analysis. The pooled sensitivity, specificity, positive likelihood ratio, negative likelihood ratio, and diagnostic odds ratio for early prediction of AP severity were 0.65 (95%CI: 0.57-0.72), 0.88 (95%CI: 0.81-0.92), 5.36 (95%CI: 3.30-8.69), 0.40 (95%CI: 0.32-0.51), and 13.35 (95%CI: 6.88-25.88), respectively. The area under the receiver operating characteristic curve was 0.81 (95%CI: 0.78-0.84). Subgroup analysis showed that geographical location (Asia, Europe, and America), time of onset (≥ 24 h or < 24 h), and severity assessment method (the Atlanta classification or others) were probably the sources of overall heterogeneity.

CONCLUSION

SFlt-1 has only moderate value for early prediction of the severity of AP, with a low sensitivity. Therefore, it needs to be combined with other relevant examinations and clinical indicators for early prediction of the severity of AP.

Assessment of heterogeneity in an individual participant data meta-analysis of prediction models: An overview and illustration

Clinical prediction models aim to provide estimates of absolute risk for a diagnostic or prognostic endpoint. Such models may be derived from data from various studies in the context of a meta‐analysis. We describe and propose approaches for assessing heterogeneity in predictor effects and predictions arising from models based on data from different sources. These methods are illustrated in a case study with patients suffering from traumatic brain injury, where we aim to predict 6‐month mortality based on individual patient data using meta‐analytic techniques (15 studies, n = 11 022 patients). The insights into various aspects of heterogeneity are important to develop better models and understand problems with the transportability of absolute risk predictions.

Evidence synthesis in prognosis research

Over the past few years, evidence synthesis has become essential to investigate and improve the generalizability of medical research findings. This strategy often involves a meta-analysis to formally summarize quantities of interest, such as relative treatment effect estimates. The use of meta-analysis methods is, however, less straightforward in prognosis research because substantial variation exists in research objectives, analysis methods and the level of reported evidence. We present a gentle overview of statistical methods that can be used to summarize data of prognostic factor and prognostic model studies. We discuss how aggregate data, individual participant data, or a combination thereof can be combined through meta-analysis methods. Recent examples are provided throughout to illustrate the various methods.