Individual participant data (IPD) meta-analyses of diagnostic and prognostic modeling studies: guidance on their use

The effectiveness of coronary computed tomography angiography and functional testing for the diagnosis of obstructive coronary artery disease: results from the individual patient data Collaborative Meta-Analysis of Cardiac CT (COME-CCT)

AIM

To determine the effectiveness of functional stress testing and computed tomography angiography (CTA) for diagnosis of obstructive coronary artery disease (CAD).

METHODS AND RESULTS

Two-thousand nine-hundred twenty symptomatic stable chest pain patients were included in the international Collaborative Meta-Analysis of Cardiac CT consortium to compare CTA with exercise electrocardiography (exercise-ECG) and single-photon emission computed tomography (SPECT) for diagnosis of CAD defined as ≥ 50% diameter stenosis by invasive coronary angiography (ICA) as reference standard. Generalised linear mixed models were used for calculating the diagnostic accuracy of each diagnostic test including non-diagnostic results as dependent variables in a logistic regression model with random intercepts and slopes. Covariates were the reference standard ICA, the type of diagnostic method, and their interactions. CTA showed significantly better diagnostic performance (p < 0.0001) with a sensitivity of 94.6% (95% CI 92.7-96) and a specificity of 76.3% (72.2-80) compared to exercise-ECG with 54.9% (47.9-61.7) and 60.9% (53.4-66.3), SPECT with 72.9% (65-79.6) and 44.9% (36.8-53.4), respectively. The positive predictive value of CTA was ≥ 50% in patients with a clinical pretest probability of 10% or more while this was the case for ECG and SPECT at pretest probabilities of ≥ 40 and 28%. CTA reliably excluded obstructive CAD with a post-test probability of below 15% in patients with a pretest probability of up to 74%.

CONCLUSION

In patients with stable chest pain, CTA is more effective than functional testing for the diagnosis as well as for reliable exclusion of obstructive CAD. CTA should become widely adopted in patients with intermediate pretest probability.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO Database for Systematic Reviews-CRD42012002780.

CRITICAL RELEVANCE STATEMENT

In symptomatic stable chest pain patients, coronary CTA is more effective than functional testing for diagnosis and reliable exclusion of obstructive CAD in intermediate pretest probability of CAD.

KEY POINTS

Coronary computed tomography angiography showed significantly better diagnostic performance (p < 0.0001) for diagnosis of coronary artery disease compared to exercise-ECG and SPECT. The positive predictive value of coronary computed tomography angiography was ≥ 50% in patients with a clinical pretest probability of at least 10%, for ECG ≥ 40%, and for SPECT 28%. Coronary computed tomography angiography reliably excluded obstructive coronary artery disease with a post-test probability of below 15% in patients with a pretest probability of up to 74%.

Development and validation of prediction models for fetal growth restriction and birthweight: an individual participant data meta-analysis

Background

Fetal growth restriction is associated with perinatal morbidity and mortality. Early identification of women having at-risk fetuses can reduce perinatal adverse outcomes.

Objectives

To assess the predictive performance of existing models predicting fetal growth restriction and birthweight, and if needed, to develop and validate new multivariable models using individual participant data.

Design

Individual participant data meta-analyses of cohorts in International Prediction of Pregnancy Complications network, decision curve analysis and health economics analysis.

Participants

Pregnant women at booking. External validation of existing models (9 cohorts, 441,415 pregnancies); International Prediction of Pregnancy Complications model development and validation (4 cohorts, 237,228 pregnancies).

Predictors

Maternal clinical characteristics, biochemical and ultrasound markers.

Primary outcomes

fetal growth restriction defined as birthweight <10th centile adjusted for gestational age and with stillbirth, neonatal death or delivery before 32 weeks' gestation birthweight.

Analysis

First, we externally validated existing models using individual participant data meta-analysis. If needed, we developed and validated new International Prediction of Pregnancy Complications models using random-intercept regression models with backward elimination for variable selection and undertook internal-external cross-validation. We estimated the study-specific performance (c-statistic, calibration slope, calibration-in-the-large) for each model and pooled using random-effects meta-analysis. Heterogeneity was quantified using τ2 and 95% prediction intervals. We assessed the clinical utility of the fetal growth restriction model using decision curve analysis, and health economics analysis based on National Institute for Health and Care Excellence 2008 model.

Results

Of the 119 published models, one birthweight model (Poon) could be validated. None reported fetal growth restriction using our definition. Across all cohorts, the Poon model had good summary calibration slope of 0.93 (95% confidence interval 0.90 to 0.96) with slight overfitting, and underpredicted birthweight by 90.4 g on average (95% confidence interval 37.9 g to 142.9 g). The newly developed International Prediction of Pregnancy Complications-fetal growth restriction model included maternal age, height, parity, smoking status, ethnicity, and any history of hypertension, pre-eclampsia, previous stillbirth or small for gestational age baby and gestational age at delivery. This allowed predictions conditional on a range of assumed gestational ages at delivery. The pooled apparent c-statistic and calibration were 0.96 (95% confidence interval 0.51 to 1.0), and 0.95 (95% confidence interval 0.67 to 1.23), respectively. The model showed positive net benefit for predicted probability thresholds between 1% and 90%. In addition to the predictors in the International Prediction of Pregnancy Complications-fetal growth restriction model, the International Prediction of Pregnancy Complications-birthweight model included maternal weight, history of diabetes and mode of conception. Average calibration slope across cohorts in the internal-external cross-validation was 1.00 (95% confidence interval 0.78 to 1.23) with no evidence of overfitting. Birthweight was underestimated by 9.7 g on average (95% confidence interval -154.3 g to 173.8 g).

Limitations

We could not externally validate most of the published models due to variations in the definitions of outcomes. Internal-external cross-validation of our International Prediction of Pregnancy Complications-fetal growth restriction model was limited by the paucity of events in the included cohorts. The economic evaluation using the published National Institute for Health and Care Excellence 2008 model may not reflect current practice, and full economic evaluation was not possible due to paucity of data.

Future work

International Prediction of Pregnancy Complications models' performance needs to be assessed in routine practice, and their impact on decision-making and clinical outcomes needs evaluation.

Conclusion

The International Prediction of Pregnancy Complications-fetal growth restriction and International Prediction of Pregnancy Complications-birthweight models accurately predict fetal growth restriction and birthweight for various assumed gestational ages at delivery. These can be used to stratify the risk status at booking, plan monitoring and management.

Study registration

This study is registered as PROSPERO CRD42019135045.

Funding

This award was funded by the National Institute for Health and Care Research (NIHR) Health Technology Assessment programme (NIHR award ref: 17/148/07) and is published in full in Health Technology Assessment; Vol. 28, No. 14. See the NIHR Funding and Awards website for further award information.

Comparison of Two Modern Survival Prediction Tools, SORG-MLA and METSSS, in Patients With Symptomatic Long-bone Metastases Who Underwent Local Treatment With Surgery Followed by Radiotherapy and With Radiotherapy Alone

BACKGROUND

Survival estimation for patients with symptomatic skeletal metastases ideally should be made before a type of local treatment has already been determined. Currently available survival prediction tools, however, were generated using data from patients treated either operatively or with local radiation alone, raising concerns about whether they would generalize well to all patients presenting for assessment. The Skeletal Oncology Research Group machine-learning algorithm (SORG-MLA), trained with institution-based data of surgically treated patients, and the Metastases location, Elderly, Tumor primary, Sex, Sickness/comorbidity, and Site of radiotherapy model (METSSS), trained with registry-based data of patients treated with radiotherapy alone, are two of the most recently developed survival prediction models, but they have not been tested on patients whose local treatment strategy is not yet decided.

QUESTIONS/PURPOSES

(1) Which of these two survival prediction models performed better in a mixed cohort made up both of patients who received local treatment with surgery followed by radiotherapy and who had radiation alone for symptomatic bone metastases? (2) Which model performed better among patients whose local treatment consisted of only palliative radiotherapy? (3) Are laboratory values used by SORG-MLA, which are not included in METSSS, independently associated with survival after controlling for predictions made by METSSS?

METHODS

Between 2010 and 2018, we provided local treatment for 2113 adult patients with skeletal metastases in the extremities at an urban tertiary referral academic medical center using one of two strategies: (1) surgery followed by postoperative radiotherapy or (2) palliative radiotherapy alone. Every patient's survivorship status was ascertained either by their medical records or the national death registry from the Taiwanese National Health Insurance Administration. After applying a priori designated exclusion criteria, 91% (1920) were analyzed here. Among them, 48% (920) of the patients were female, and the median (IQR) age was 62 years (53 to 70 years). Lung was the most common primary tumor site (41% [782]), and 59% (1128) of patients had other skeletal metastases in addition to the treated lesion(s). In general, the indications for surgery were the presence of a complete pathologic fracture or an impending pathologic fracture, defined as having a Mirels score of ≥ 9, in patients with an American Society of Anesthesiologists (ASA) classification of less than or equal to IV and who were considered fit for surgery. The indications for radiotherapy were relief of pain, local tumor control, prevention of skeletal-related events, and any combination of the above. In all, 84% (1610) of the patients received palliative radiotherapy alone as local treatment for the target lesion(s), and 16% (310) underwent surgery followed by postoperative radiotherapy. Neither METSSS nor SORG-MLA was used at the point of care to aid clinical decision-making during the treatment period. Survival was retrospectively estimated by these two models to test their potential for providing survival probabilities. We first compared SORG to METSSS in the entire population. Then, we repeated the comparison in patients who received local treatment with palliative radiation alone. We assessed model performance by area under the receiver operating characteristic curve (AUROC), calibration analysis, Brier score, and decision curve analysis (DCA). The AUROC measures discrimination, which is the ability to distinguish patients with the event of interest (such as death at a particular time point) from those without. AUROC typically ranges from 0.5 to 1.0, with 0.5 indicating random guessing and 1.0 a perfect prediction, and in general, an AUROC of ≥ 0.7 indicates adequate discrimination for clinical use. Calibration refers to the agreement between the predicted outcomes (in this case, survival probabilities) and the actual outcomes, with a perfect calibration curve having an intercept of 0 and a slope of 1. A positive intercept indicates that the actual survival is generally underestimated by the prediction model, and a negative intercept suggests the opposite (overestimation). When comparing models, an intercept closer to 0 typically indicates better calibration. Calibration can also be summarized as log(O:E), the logarithm scale of the ratio of observed (O) to expected (E) survivors. A log(O:E) > 0 signals an underestimation (the observed survival is greater than the predicted survival); and a log(O:E) < 0 indicates the opposite (the observed survival is lower than the predicted survival). A model with a log(O:E) closer to 0 is generally considered better calibrated. The Brier score is the mean squared difference between the model predictions and the observed outcomes, and it ranges from 0 (best prediction) to 1 (worst prediction). The Brier score captures both discrimination and calibration, and it is considered a measure of overall model performance. In Brier score analysis, the "null model" assigns a predicted probability equal to the prevalence of the outcome and represents a model that adds no new information. A prediction model should achieve a Brier score at least lower than the null-model Brier score to be considered as useful. The DCA was developed as a method to determine whether using a model to inform treatment decisions would do more good than harm. It plots the net benefit of making decisions based on the model's predictions across all possible risk thresholds (or cost-to-benefit ratios) in relation to the two default strategies of treating all or no patients. The care provider can decide on an acceptable risk threshold for the proposed treatment in an individual and assess the corresponding net benefit to determine whether consulting with the model is superior to adopting the default strategies. Finally, we examined whether laboratory data, which were not included in the METSSS model, would have been independently associated with survival after controlling for the METSSS model's predictions by using the multivariable logistic and Cox proportional hazards regression analyses.

RESULTS

Between the two models, only SORG-MLA achieved adequate discrimination (an AUROC of > 0.7) in the entire cohort (of patients treated operatively or with radiation alone) and in the subgroup of patients treated with palliative radiotherapy alone. SORG-MLA outperformed METSSS by a wide margin on discrimination, calibration, and Brier score analyses in not only the entire cohort but also the subgroup of patients whose local treatment consisted of radiotherapy alone. In both the entire cohort and the subgroup, DCA demonstrated that SORG-MLA provided more net benefit compared with the two default strategies (of treating all or no patients) and compared with METSSS when risk thresholds ranged from 0.2 to 0.9 at both 90 days and 1 year, indicating that using SORG-MLA as a decision-making aid was beneficial when a patient's individualized risk threshold for opting for treatment was 0.2 to 0.9. Higher albumin, lower alkaline phosphatase, lower calcium, higher hemoglobin, lower international normalized ratio, higher lymphocytes, lower neutrophils, lower neutrophil-to-lymphocyte ratio, lower platelet-to-lymphocyte ratio, higher sodium, and lower white blood cells were independently associated with better 1-year and overall survival after adjusting for the predictions made by METSSS.

CONCLUSION

Based on these discoveries, clinicians might choose to consult SORG-MLA instead of METSSS for survival estimation in patients with long-bone metastases presenting for evaluation of local treatment. Basing a treatment decision on the predictions of SORG-MLA could be beneficial when a patient's individualized risk threshold for opting to undergo a particular treatment strategy ranged from 0.2 to 0.9. Future studies might investigate relevant laboratory items when constructing or refining a survival estimation model because these data demonstrated prognostic value independent of the predictions of the METSSS model, and future studies might also seek to keep these models up to date using data from diverse, contemporary patients undergoing both modern operative and nonoperative treatments.

LEVEL OF EVIDENCE

Level III, diagnostic study.

Artificial intelligence in the risk prediction models of cardiovascular disease and development of an independent validation screening tool: a systematic review

BACKGROUND

A comprehensive overview of artificial intelligence (AI) for cardiovascular disease (CVD) prediction and a screening tool of AI models (AI-Ms) for independent external validation are lacking. This systematic review aims to identify, describe, and appraise AI-Ms of CVD prediction in the general and special populations and develop a new independent validation score (IVS) for AI-Ms replicability evaluation.

METHODS

PubMed, Web of Science, Embase, and IEEE library were searched up to July 2021. Data extraction and analysis were performed for the populations, distribution, predictors, algorithms, etc. The risk of bias was evaluated with the prediction risk of bias assessment tool (PROBAST). Subsequently, we designed IVS for model replicability evaluation with five steps in five items, including transparency of algorithms, performance of models, feasibility of reproduction, risk of reproduction, and clinical implication, respectively. The review is registered in PROSPERO (No. CRD42021271789).

RESULTS

In 20,887 screened references, 79 articles (82.5% in 2017-2021) were included, which contained 114 datasets (67 in Europe and North America, but 0 in Africa). We identified 486 AI-Ms, of which the majority were in development (n = 380), but none of them had undergone independent external validation. A total of 66 idiographic algorithms were found; however, 36.4% were used only once and only 39.4% over three times. A large number of different predictors (range 5-52,000, median 21) and large-span sample size (range 80-3,660,000, median 4466) were observed. All models were at high risk of bias according to PROBAST, primarily due to the incorrect use of statistical methods. IVS analysis confirmed only 10 models as "recommended"; however, 281 and 187 were "not recommended" and "warning," respectively.

CONCLUSION

AI has led the digital revolution in the field of CVD prediction, but is still in the early stage of development as the defects of research design, report, and evaluation systems. The IVS we developed may contribute to independent external validation and the development of this field.

Development and validation of a risk prediction model for pulmonary tuberculosis among presumptive tuberculosis cases in Ethiopia

BACKGROUND

Early diagnosis and treatment of tuberculosis (TB) is one of the key strategies to achieve the WHO End TB targets. This study aimed to develop and validate a simple, convenient risk score to diagnose pulmonary TB among presumptive TB cases.

METHODS

This prediction model used Ethiopian national TB prevalence survey data and included 5459 presumptive TB cases from all regions of Ethiopia. Logistic regression was used to determine which variables are predictive of pulmonary TB. A risk prediction model was developed, incorporating significant variables (p<0.05). The Youden Index method was used to choose the optimal cut-off point to separate the risk score of the patients as high and low. Model performance was assessed using discrimination power and calibration. Internal validation of the model was assessed using Efron's enhanced bootstrap method, and the clinical utility of the risk score was assessed using decision curve analysis.

RESULTS

Of total participants, 94 (1.7%) were confirmed to have TB. The final prediction model included three factors with different scores: (1) TB contact history, (2) chest X-ray (CXR) abnormality and (3) two or more symptoms of TB. The optimal cut-off point for the risk score was 6 and was found to have a good discrimination accuracy (c-statistic=0.70, 95% CI: 0.65 to 0.75). The risk score has sensitivity of 51.1%, specificity of 79.9%, positive predictive value of 4.3% and negative predictive value of 98.9%. After internal validation, the optimism coefficient was 0.003, which indicates the model is internally valid.

CONCLUSION

We developed a risk score that combines TB contact, number of TB symptoms and CXR abnormality to estimate individual risk of pulmonary TB among presumptive TB cases. Though the score is easy to calculate and internally validated, it needs external validation before widespread implementation in a new setting.

Aberrant Salience Inventory: A meta-analysis to investigate its psychometric properties and identify screening cutoff scores

INTRODUCTION

The Aberrant Salience Inventory (ASI) is a useful tool to measure salience abnormalities among the general population. There is strong clinical and scientific evidence that salience alteration is linked to psychosis. To the present day, no meta-analysis evaluating ASI's psychometric properties and screening potential has been published.

MATERIALS AND METHODS

PubMed, Google Scholar, Scopus, and Embase were searched using terms including "psychosis," "schizophrenia," and "Aberrant Salience Inventory." Observational and experimental studies employing ASI on populations of non-psychotic controls and patients with psychosis were included. ASI scores and other demographic measures (age, gender, ethnicity) were extracted as outcomes. Individual patients' data (IPD) were collected. Exploratory factor analysis (EFA) was performed on the IPD.

RESULTS

Eight articles were finally included in the meta-analysis. ASI scores differ significantly between psychotic and non-psychotic populations; a novel three-factor model is proposed regarding subscales structure. Theoretical positive predictive values (PPVs) and negative predictive values (NPVs) were calculated and presented together with different cutoff points depending on preselected specific populations of interest.

DISCUSSION

PPV and NPV values reached levels adequate for ASI to be considered a viable screening tool for psychosis. The factor analysis highlights the presence of a novel subscale that was named "Unveiling experiences." Implications regarding the meaning of the new factor structure are discussed, as well as ASI's potential as a screening tool.

Prognostic prediction models for clinical outcomes in patients diagnosed with visceral leishmaniasis: protocol for a systematic review

INTRODUCTION

Visceral leishmaniasis (VL) is a neglected tropical disease responsible for many thousands of preventable deaths each year. Symptomatic patients often struggle to access effective treatment, without which death is the norm. Risk prediction tools support clinical teams and policymakers in identifying high-risk patients who could benefit from more intensive management pathways. Investigators interested in using their clinical data for prognostic research should first identify currently available models that are candidates for validation and possible updating. Addressing these needs, we aim to identify, summarise and appraise the available models predicting clinical outcomes in VL patients.

METHODS AND ANALYSIS

We will include studies that have developed, validated or updated prognostic models predicting future clinical outcomes in patients diagnosed with VL. Systematic reviews and meta-analyses that include eligible studies are also considered for review. Conference abstracts and educational theses are excluded. Data extraction, appraisal and reporting will follow current methodological guidelines. Ovid Embase; Ovid MEDLINE; the Web of Science Core Collection, SciELO and LILACS are searched from database inception to 1 March 2023 using terms developed for the identification of prediction models, and with no language restriction. Screening, data extraction and risk of bias assessment will be performed in duplicate with discordance resolved by a third independent reviewer. Risk of bias will be assessed using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). Tables and figures will compare and contrast key model information, including source data, participants, model development and performance measures, and risk of bias. We will consider the strengths, limitations and clinical applicability of the identified models.

ETHICS AND DISSEMINATION

Ethics approval is not required for this review. The systematic review and all accompanying data will be submitted to an open-access journal. Findings will also be disseminated through the research group's website (www.iddo.org/research-themes/visceral-leishmaniasis) and social media channels.

PROSPERO REGISTRATION NUMBER

CRD42023417226.

Prognosis and prediction of antibiotic benefit in adults with clinically diagnosed acute rhinosinusitis: an individual participant data meta-analysis

BACKGROUND

A previous individual participant data meta-analysis (IPD-MA) of antibiotics for adults with clinically diagnosed acute rhinosinusitis (ARS) showed a marginal overall effect of antibiotics, but was unable to identify patients that are most likely to benefit from antibiotics when applying conventional (i.e. univariable or one-variable-at-a-time) subgroup analysis. We updated the systematic review and investigated whether multivariable prediction of patient-level prognosis and antibiotic treatment effect may lead to more tailored treatment assignment in adults presenting to primary care with ARS.

METHODS

An IPD-MA of nine double-blind placebo-controlled trials of antibiotic treatment (n=2539) was conducted, with the probability of being cured at 8-15 days as the primary outcome. A logistic mixed effects model was developed to predict the probability of being cured based on demographic characteristics, signs and symptoms, and antibiotic treatment assignment. Predictive performance was quantified based on internal-external cross-validation in terms of calibration and discrimination performance, overall model fit, and the accuracy of individual predictions.

RESULTS

Results indicate that the prognosis with respect to risk of cure could not be reliably predicted (c-statistic 0.58 and Brier score 0.24). Similarly, patient-level treatment effect predictions did not reliably distinguish between those that did and did not benefit from antibiotics (c-for-benefit 0.50).

CONCLUSIONS

In conclusion, multivariable prediction based on patient demographics and common signs and symptoms did not reliably predict the patient-level probability of cure and antibiotic effect in this IPD-MA. Therefore, these characteristics cannot be expected to reliably distinguish those that do and do not benefit from antibiotics in adults presenting to primary care with ARS.

Clinical Validity of a Machine Learning Decision Support System for Early Detection of Hepatitis B Virus: A Binational External Validation Study

HepB LiveTest is a machine learning decision support system developed for the early detection of hepatitis B virus (HBV). However, there is a lack of evidence on its generalisability. In this study, we aimed to externally assess the clinical validity and portability of HepB LiveTest in predicting HBV infection among independent patient cohorts from Nigeria and Australia. The performance of HepB LiveTest was evaluated by constructing receiver operating characteristic curves and estimating the area under the curve. Delong's method was used to estimate the 95% confidence interval (CI) of the area under the receiver-operating characteristic curve (AUROC). Compared to the Australian cohort, patients in the derivation cohort of HepB LiveTest and the hospital-based Nigerian cohort were younger (mean age, 45.5 years vs. 38.8 years vs. 40.8 years, respectively; p < 0.001) and had a higher incidence of HBV infection (1.9% vs. 69.4% vs. 57.3%). In the hospital-based Nigerian cohort, HepB LiveTest performed optimally with an AUROC of 0.94 (95% CI, 0.91-0.97). The model provided tailored predictions that ensured most cases of HBV infection did not go undetected. However, its discriminatory measure dropped to 0.60 (95% CI, 0.56-0.64) in the Australian cohort. These findings indicate that HepB LiveTest exhibits adequate cross-site transportability and clinical validity in the hospital-based Nigerian patient cohort but shows limited performance in the Australian cohort. Whilst HepB LiveTest holds promise for reducing HBV prevalence in underserved populations, caution is warranted when implementing the model in older populations, particularly in regions with low incidence of HBV infection.

Clinical utility of WHO-recommended screening tools and development and validation of novel clinical prediction models for pulmonary tuberculosis screening among outpatients living with HIV: an individual participant data meta-analysis

BACKGROUND

The World Health Organization (WHO) recommends that outpatient people living with HIV (PLHIV) undergo tuberculosis screening with the WHO four-symptom screen (W4SS) or C-reactive protein (CRP) (5 mg·L-1 cut-off) followed by confirmatory testing if screen positive. We conducted an individual participant data meta-analysis to determine the performance of WHO-recommended screening tools and two newly developed clinical prediction models (CPMs).

METHODS

Following a systematic review, we identified studies that recruited adult outpatient PLHIV irrespective of tuberculosis signs and symptoms or with a positive W4SS, evaluated CRP and collected sputum for culture. We used logistic regression to develop an extended CPM (which included CRP and other predictors) and a CRP-only CPM. We used internal-external cross-validation to evaluate performance.

RESULTS

We pooled data from eight cohorts (n=4315 participants). The extended CPM had excellent discrimination (C-statistic 0.81); the CRP-only CPM had similar discrimination. The C-statistics for WHO-recommended tools were lower. Both CPMs had equivalent or higher net benefit compared with the WHO-recommended tools. Compared with both CPMs, CRP (5 mg·L-1 cut-off) had equivalent net benefit across a clinically useful range of threshold probabilities, while the W4SS had a lower net benefit. The W4SS would capture 91% of tuberculosis cases and require confirmatory testing for 78% of participants. CRP (5 mg·L-1 cut-off), the extended CPM (4.2% threshold) and the CRP-only CPM (3.6% threshold) would capture similar percentages of cases but reduce confirmatory tests required by 24, 27 and 36%, respectively.

CONCLUSIONS

CRP sets the standard for tuberculosis screening among outpatient PLHIV. The choice between using CRP at 5 mg·L-1 cut-off or in a CPM depends on available resources.

Estimating Patient-Specific Relative Benefit of Adding Biologics to Conventional Rheumatoid Arthritis Treatment: An Individual Participant Data Meta-Analysis

Importance

Current evidence remains ambiguous regarding whether biologics should be added to conventional treatment of rheumatoid arthritis for specific patients, which may cause potential overuse or treatment delay.

Objectives

To estimate the benefit of adding biologics to conventional antirheumatic drugs for the treatment of rheumatoid arthritis given baseline characteristics.

Data Sources

Cochrane CENTRAL, Scopus, MEDLINE, and the World Health Organization International Clinical Trials Registry Platform were searched for articles published from database inception to March 2, 2022.

Study Selection

Randomized clinical trials comparing certolizumab plus conventional antirheumatic drugs with placebo plus conventional drugs were selected.

Data Extraction and Synthesis

Individual participant data of the prespecified outcomes and covariates were acquired from the Vivli database. A 2-stage model was fitted to estimate patient-specific relative outcomes of adding certolizumab vs conventional drugs only. Stage 1 was a penalized logistic regression model to estimate the baseline expected probability of the outcome regardless of treatment using baseline characteristics. Stage 2 was a bayesian individual participant data meta-regression model to estimate the relative outcomes for a particular baseline expected probability. Patient-specific results were displayed interactively on an application based on a 2-stage model.

Main Outcomes and Measures

The primary outcome was low disease activity or remission at 3 months, defined by 3 disease activity indexes (ie, Disease Activity Score based on the evaluation of 28 joints, Clinical Disease Activity Index, or Simplified Disease Activity Index).

Results

Individual participant data were obtained from 3790 patients (2996 female [79.1%] and 794 male [20.9%]; mean [SD] age, 52.7 [12.3] years) from 5 large randomized clinical trials for moderate to high activity rheumatoid arthritis with usable data for 22 prespecified baseline covariates. Overall, adding certolizumab was associated with a higher probability of reaching low disease activity. The odds ratio for patients with an average baseline expected probability of the outcome was 6.31 (95% credible interval, 2.22-15.25). However, the benefits differed in patients with different baseline characteristics. For example, the estimated risk difference was smaller than 10% for patients with either low or high baseline expected probability.

Conclusions and Relevance

In this individual participant data meta-analysis, adding certolizumab was associated with more effectiveness for rheumatoid arthritis in general. However, the benefit was uncertain for patients with low or high baseline expected probability, for whom other evaluations were necessary. The interactive application displaying individual estimates may help with treatment selection.

CA-125 Early Dynamics to Predict Overall Survival in Women with Newly Diagnosed Advanced Ovarian Cancer Based on Meta-Analysis Data

(1) Background: Cancer antigen 125 (CA-125) is a protein produced by ovarian cancer cells that is used for patients' monitoring. However, the best ways to analyze its decline and prognostic role are poorly quantified. (2) Methods: We leveraged individual patient data from the Gynecologic Cancer Intergroup (GCIG) meta-analysis (N = 5573) to compare different approaches summarizing the early trajectory of CA-125 before the prediction time (called the landmark time) at 3 or 6 months after treatment initiation in order to predict overall survival. These summaries included observed and estimated measures obtained by a linear mixed model (LMM). Their performances were evaluated by 10-fold cross-validation with the Brier score and the area under the ROC (AUC). (3) Results: The estimated value and the last observed value at 3 months were the best measures used to predict overall survival, with an AUC of 0.75 CI 95% [0.70; 0.80] at 24 and 36 months and 0.74 [0.69; 0.80] and 0.75 [0.69; 0.80] at 48 months, respectively, considering that CA-125 over 6 months did not improve the AUC, with 0.74 [0.68; 0.78] at 24 months and 0.71 [0.65; 0.76] at 36 and 48 months. (4) Conclusions: A 3-month surveillance provided reliable individual information on overall survival until 48 months for patients receiving first-line chemotherapy.

Cardiovascular disease and type 2 diabetes in older adults: a combined protocol for an individual participant data analysis for risk prediction and a network meta-analysis of novel anti-diabetic drugs

Introduction

Older and multimorbid adults with type 2 diabetes (T2D) are at high risk of cardiovascular disease (CVD) and chronic kidney disease (CKD). Estimating risk and preventing CVD is a challenge in this population notably because it is underrepresented in clinical trials. Our study aims to (1) assess if T2D and haemoglobin A1c (HbA1c) are associated with the risk of CVD events and mortality in older adults, (2) develop a risk score for CVD events and mortality for older adults with T2D, (3) evaluate the comparative efficacy and safety of novel antidiabetics.

Methods and analysis

For Aim 1, we will analyse individual participant data on individuals aged ≥65 years from five cohort studies: the Optimising Therapy to Prevent Avoidable Hospital Admissions in Multimorbid Older People study; the Cohorte Lausannoise study; the Health, Aging and Body Composition study; the Health and Retirement Study; and the Survey of Health, Ageing and Retirement in Europe. We will fit flexible parametric survival models (FPSM) to assess the association of T2D and HbA1c with CVD events and mortality. For Aim 2, we will use data on individuals aged ≥65 years with T2D from the same cohorts to develop risk prediction models for CVD events and mortality using FPSM. We will assess model performance, perform internal-external cross validation, and derive a point-based risk score. For Aim 3, we will systematically search randomized controlled trials of novel antidiabetics. Network meta-analysis will be used to determine comparative efficacy in terms of CVD, CKD, and retinopathy outcomes, and safety of these drugs. Confidence in results will be judged using the CINeMA tool.

Ethics and dissemination

Aims 1 and 2 were approved by the local ethics committee (Kantonale Ethikkommission Bern); no approval is required for Aim 3. Results will be published in peer-reviewed journals and presented in scientific conferences.

Systematic review and meta-analysis protocol for development and validation of a prediction model for gestational hypertension in Africa

Objective

Examining the development and validation of predictive models for gestational hypertension, evaluating the validity of the methodology, and investigating predictors typically employed in such models.

Design

Systematic review and meta-analysis protocol.

Methods

The Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocols (PRISMA-P) guideline will be used to carry out the study procedure. Using the key phrases "Gestational hypertension," "prediction, risk prediction," and "validation," a full systematic search will be conducted in PubMed/MEDLINE, Hinari, Cochrane Library, and Google Scholar. The methodological quality of the included studies will be evaluated using the prediction model risk of bias assessment tool. The CHARMS (checklist for critical evaluation and data extraction for systematic reviews of prediction modeling research) will be used to extract the data, and STATA 16 will be used to analyze it. The degree of study heterogeneity will be assessed using Cochrane I2 statistics.

Discussion

A subgroup analysis will be performed to reduce the variance between primary studies. To examine the impact of individual studies on the pooled estimates, a sensitivity analysis will be performed. The funnel plot test and Egger's statistical test will be used to assess the small study effect. The presence of a modest study effect is shown by Egger's test (p-value 0.05), which will be handled by nonparametric trim and fill analysis using the random-effects model. The protocol has been registered in the PROSPERO-International Prospective Register of systematic reviews, with the registration number CRD42022314601.

Methodological guidance for the evaluation and updating of clinical prediction models: a systematic review

BACKGROUND

Clinical prediction models are often not evaluated properly in specific settings or updated, for instance, with information from new markers. These key steps are needed such that models are fit for purpose and remain relevant in the long-term. We aimed to present an overview of methodological guidance for the evaluation (i.e., validation and impact assessment) and updating of clinical prediction models.

METHODS

We systematically searched nine databases from January 2000 to January 2022 for articles in English with methodological recommendations for the post-derivation stages of interest. Qualitative analysis was used to summarize the 70 selected guidance papers.

RESULTS

Key aspects for validation are the assessment of statistical performance using measures for discrimination (e.g., C-statistic) and calibration (e.g., calibration-in-the-large and calibration slope). For assessing impact or usefulness in clinical decision-making, recent papers advise using decision-analytic measures (e.g., the Net Benefit) over simplistic classification measures that ignore clinical consequences (e.g., accuracy, overall Net Reclassification Index). Commonly recommended methods for model updating are recalibration (i.e., adjustment of intercept or baseline hazard and/or slope), revision (i.e., re-estimation of individual predictor effects), and extension (i.e., addition of new markers). Additional methodological guidance is needed for newer types of updating (e.g., meta-model and dynamic updating) and machine learning-based models.

CONCLUSION

Substantial guidance was found for model evaluation and more conventional updating of regression-based models. An important development in model evaluation is the introduction of a decision-analytic framework for assessing clinical usefulness. Consensus is emerging on methods for model updating.

Predicting COVID-19 prognosis in the ICU remained challenging: external validation in a multinational regional cohort

OBJECTIVES

Many prediction models for coronavirus disease 2019 (COVID-19) have been developed. External validation is mandatory before implementation in the intensive care unit (ICU). We selected and validated prognostic models in the Euregio Intensive Care COVID (EICC) cohort.

STUDY DESIGN AND SETTING

In this multinational cohort study, routine data from COVID-19 patients admitted to ICUs within the Euregio Meuse-Rhine were collected from March to August 2020. COVID-19 models were selected based on model type, predictors, outcomes, and reporting. Furthermore, general ICU scores were assessed. Discrimination was assessed by area under the receiver operating characteristic curves (AUCs) and calibration by calibration-in-the-large and calibration plots. A random-effects meta-analysis was used to pool results.

RESULTS

551 patients were admitted. Mean age was 65.4 ± 11.2 years, 29% were female, and ICU mortality was 36%. Nine out of 238 published models were externally validated. Pooled AUCs were between 0.53 and 0.70 and calibration-in-the-large between -9% and 6%. Calibration plots showed generally poor but, for the 4C Mortality score and Spanish Society of Infectious Diseases and Clinical Microbiology (SEIMC) score, moderate calibration.

CONCLUSION

Of the nine prognostic models that were externally validated in the EICC cohort, only two showed reasonable discrimination and moderate calibration. For future pandemics, better models based on routine data are needed to support admission decision-making.

Predicting major complications in patients undergoing laparoscopic and open hysterectomy for benign indications

BACKGROUND

Hysterectomy, the most common gynecological operation, requires surgeons to counsel women about their operative risks. We aimed to develop and validate multivariable logistic regression models to predict major complications of laparoscopic or abdominal hysterectomy for benign conditions.

METHODS

We obtained routinely collected health administrative data from the English National Health Service (NHS) from 2011 to 2018. We defined major complications based on core outcomes for postoperative complications including ureteric, gastrointestinal and vascular injury, and wound complications. We specified 11 predictors a priori. We used internal-external cross-validation to evaluate discrimination and calibration across 7 NHS regions in the development cohort. We validated the final models using data from an additional NHS region.

RESULTS

We found that major complications occurred in 4.4% (3037/68 599) of laparoscopic and 4.9% (6201/125 971) of abdominal hysterectomies. Our models showed consistent discrimination in the development cohort (laparoscopic, C-statistic 0.61, 95% confidence interval [CI] 0.60 to 0.62; abdominal, C-statistic 0.67, 95% CI 0.64 to 0.70) and similar or better discrimination in the validation cohort (laparoscopic, C-statistic 0.67, 95% CI 0.65 to 0.69; abdominal, C-statistic 0.67, 95% CI 0.65 to 0.69). Adhesions were most predictive of complications in both models (laparoscopic, odds ratio [OR] 1.92, 95% CI 1.73 to 2.13; abdominal, OR 2.46, 95% CI 2.27 to 2.66). Other factors predictive of complications included adenomyosis in the laparoscopic model, and Asian ethnicity and diabetes in the abdominal model. Protective factors included age and diagnoses of menstrual disorders or benign adnexal mass in both models and diagnosis of fibroids in the abdominal model.

INTERPRETATION

Personalized risk estimates from these models, which showed moderate discrimination, can inform clinical decision-making for people with benign conditions who may require hysterectomy.

Development and Validation of a Model for Predicting Surgical Site Infection After Pelvic Organ Prolapse Surgery

IMPORTANCE

Surgical site infection (SSI) is a common and costly complication. Targeted interventions in high-risk patients may lead to a reduction in SSI; at present, there is no method to consistently identify patients at increased risk of SSI.

OBJECTIVE

The aim of this study was to develop and validate a model for predicting risk of SSI after pelvic organ prolapse surgery.

STUDY DESIGN

Women undergoing surgery between 2011 and 2017 were identified using Current Procedural Terminology codes from the Centers for Medicare and Medicaid Services 5% Limited Data Set. Surgical site infection ≤90 days of surgery was the primary outcome, with 41 candidate predictors identified, including demographics, comorbidities, and perioperative variables. Generalized linear regression was used to fit a full specified model, including all predictors and a reduced penalized model approximating the full model. Model performance was measured using the c-statistic, Brier score, and calibration curves. Accuracy measures were internally validated using bootstrapping to correct for bias and overfitting. Decision curves were used to determine the net benefit of using the model.

RESULTS

Of 12,334 women, 4.7% experienced SSI. The approximated model included 10 predictors. Model accuracy was acceptable (bias-corrected c-statistic [95% confidence interval], 0.603 [0.578-0.624]; Brier score, 0.045). The model was moderately calibrated when predicting up to 5-6 times the average risk of SSI between 0 and 25-30%. There was a net benefit for clinical use when risk thresholds for intervention were between 3% and 12%.

CONCLUSIONS

This model provides estimates of probability of SSI within 90 days after pelvic organ prolapse surgery and demonstrates net benefit when considering prevention strategies to reduce SSI.

Development of hypertensive complications in oocyte donation pregnancy: protocol for a systematic review and individual participant data meta-analysis (DONOR IPD)

INTRODUCTION

The assisted reproductive technique of oocyte donation (OD) is comparable to in vitro fertilisation (IVF), with the distinction of using a donated oocyte and thus involving two women. Compared with IVF and naturally conceived (NC) pregnancies, OD pregnancies have a higher risk for pregnancy complications as pregnancy-induced hypertension (PIH) and pre-eclampsia (PE). Various covariates among women pregnant by OD, however, also contribute to an increased risk for developing hypertensive complications. Therefore, we will conduct the DONation of Oocytes in Reproduction individual participant data (DONOR IPD) meta-analysis to determine the risk for the development of hypertensive complications in OD pregnancy, in comparison to autologous oocyte pregnancy (non-donor IVF/intracytoplasmic sperm injection (ICSI) and NC pregnancy). The DONOR IPD meta-analysis will provide an opportunity to adjust for confounders and perform subgroup analyses. Furthermore, IPD will be used to externally validate a prediction model for the development of PE in OD pregnancy.

METHODS AND ANALYSIS

A systematic literature search will be performed to search for studies that included women pregnant by OD, and documented on hypertensive complications in OD pregnancy. The authors from each study will be asked to collaborate and share IPD. Using the pseudoanonymised combined IPD, we will perform statistical analyses with one-stage and two-stage approaches, subgroup analyses and possibly time-to-event analyses to investigate the risk of developing hypertensive complications in OD pregnancy. Furthermore, we will formally assess a prediction model on its performance in an external validation with the use of IPD.

ETHICS AND DISSEMINATION

Ethical approval and individual patient consent will not be required in most cases since this IPD meta-analysis will use existing pseudoanonymised data from cohort studies. Results will be disseminated through peer-reviewed journals and international conferences.

PROSPERO REGISTRATION NUMBER

CRD42021267908.

Development and validation of a decision support tool for the diagnosis of acute heart failure: systematic review, meta-analysis, and modelling study

OBJECTIVES

To evaluate the diagnostic performance of N-terminal pro-B-type natriuretic peptide (NT-proBNP) thresholds for acute heart failure and to develop and validate a decision support tool that combines NT-proBNP concentrations with clinical characteristics.

DESIGN

Individual patient level data meta-analysis and modelling study.

SETTING

Fourteen studies from 13 countries, including randomised controlled trials and prospective observational studies.

PARTICIPANTS

Individual patient level data for 10 369 patients with suspected acute heart failure were pooled for the meta-analysis to evaluate NT-proBNP thresholds. A decision support tool (Collaboration for the Diagnosis and Evaluation of Heart Failure (CoDE-HF)) that combines NT-proBNP with clinical variables to report the probability of acute heart failure for an individual patient was developed and validated.

MAIN OUTCOME MEASURE

Adjudicated diagnosis of acute heart failure.

RESULTS

Overall, 43.9% (4549/10 369) of patients had an adjudicated diagnosis of acute heart failure (73.3% (2286/3119) and 29.0% (1802/6208) in those with and without previous heart failure, respectively). The negative predictive value of the guideline recommended rule-out threshold of 300 pg/mL was 94.6% (95% confidence interval 91.9% to 96.4%); despite use of age specific rule-in thresholds, the positive predictive value varied at 61.0% (55.3% to 66.4%), 73.5% (62.3% to 82.3%), and 80.2% (70.9% to 87.1%), in patients aged <50 years, 50-75 years, and >75 years, respectively. Performance varied in most subgroups, particularly patients with obesity, renal impairment, or previous heart failure. CoDE-HF was well calibrated, with excellent discrimination in patients with and without previous heart failure (area under the receiver operator curve 0.846 (0.830 to 0.862) and 0.925 (0.919 to 0.932) and Brier scores of 0.130 and 0.099, respectively). In patients without previous heart failure, the diagnostic performance was consistent across all subgroups, with 40.3% (2502/6208) identified at low probability (negative predictive value of 98.6%, 97.8% to 99.1%) and 28.0% (1737/6208) at high probability (positive predictive value of 75.0%, 65.7% to 82.5%) of having acute heart failure.

CONCLUSIONS

In an international, collaborative evaluation of the diagnostic performance of NT-proBNP, guideline recommended thresholds to diagnose acute heart failure varied substantially in important patient subgroups. The CoDE-HF decision support tool incorporating NT-proBNP as a continuous measure and other clinical variables provides a more consistent, accurate, and individualised approach.

STUDY REGISTRATION

PROSPERO CRD42019159407.

Prospective validation of the 4C prognostic models for adults hospitalised with COVID-19 using the ISARIC WHO Clinical Characterisation Protocol

PURPOSE

To prospectively validate two risk scores to predict mortality (4C Mortality) and in-hospital deterioration (4C Deterioration) among adults hospitalised with COVID-19.

METHODS

Prospective observational cohort study of adults (age ≥18 years) with confirmed or highly suspected COVID-19 recruited into the International Severe Acute Respiratory and emerging Infections Consortium (ISARIC) WHO Clinical Characterisation Protocol UK (CCP-UK) study in 306 hospitals across England, Scotland and Wales. Patients were recruited between 27 August 2020 and 17 February 2021, with at least 4 weeks follow-up before final data extraction. The main outcome measures were discrimination and calibration of models for in-hospital deterioration (defined as any requirement of ventilatory support or critical care, or death) and mortality, incorporating predefined subgroups.

RESULTS

76 588 participants were included, of whom 27 352 (37.4%) deteriorated and 12 581 (17.4%) died. Both the 4C Mortality (0.78 (0.77 to 0.78)) and 4C Deterioration scores (pooled C-statistic 0.76 (95% CI 0.75 to 0.77)) demonstrated consistent discrimination across all nine National Health Service regions, with similar performance metrics to the original validation cohorts. Calibration remained stable (4C Mortality: pooled slope 1.09, pooled calibration-in-the-large 0.12; 4C Deterioration: 1.00, -0.04), with no need for temporal recalibration during the second UK pandemic wave of hospital admissions.

CONCLUSION

Both 4C risk stratification models demonstrate consistent performance to predict clinical deterioration and mortality in a large prospective second wave validation cohort of UK patients. Despite recent advances in the treatment and management of adults hospitalised with COVID-19, both scores can continue to inform clinical decision making.

TRIAL REGISTRATION NUMBER

ISRCTN66726260.

Development of a clinical prediction model for perinatal deaths in low resource settings

BACKGROUND

Most pregnancy-related deaths in low and middle income countries occur around the time of birth and are avoidable with timely care. This study aimed to develop a prognostic model to identify women at risk of intrapartum-related perinatal deaths in low-resourced settings, by (1) external validation of an existing prediction model, and subsequently (2) development of a novel model.

METHODS

A prospective cohort study was conducted among pregnant women who presented consecutively for delivery at the maternity unit of Zanzibar's tertiary hospital, Mnazi Mmoja Hospital, the Republic of Tanzania between October 2017 and May 2018. Candidate predictors of perinatal deaths included maternal and foetal characteristics obtained from routine history and physical examination at the time of admission to the labour ward. The outcomes were intrapartum stillbirths and neonatal death before hospital discharge. An existing stillbirth prediction model with six predictors from Nigeria was applied to the Zanzibar cohort to assess its discrimination and calibration performance. Subsequently, a new prediction model was developed using multivariable logistic regression. Model performance was evaluated through internal validation and corrected for overfitting using bootstrapping methods.

FINDINGS

5747 mother-baby pairs were analysed. The existing model showed poor discrimination performance (c-statistic 0·57). The new model included 15 clinical predictors and showed promising discriminative and calibration performance after internal validation (optimism adjusted c-statistic of 0·78, optimism adjusted calibration slope =0·94).

INTERPRETATION

The new model consisted of predictors easily obtained through history-taking and physical examination at the time of admission to the labour ward. It had good performance in predicting risk of perinatal death in women admitted in labour wards. Therefore, it has the potential to assist skilled birth attendance to triage women for appropriate management during labour. Before routine implementation, external validation and usefulness should be determined in future studies.

FUNDING

The study received funding from Laerdal Foundation, Otto Kranendonk Fund and UMC Global Health Fellowship. TD acknowledges financial support from the Netherlands Organisation for Health Research and Development (grant 91617050).

External validation of prognostic models to predict stillbirth using International Prediction of Pregnancy Complications (IPPIC) Network database: individual participant data meta-analysis

OBJECTIVE

Stillbirth is a potentially preventable complication of pregnancy. Identifying women at high risk of stillbirth can guide decisions on the need for closer surveillance and timing of delivery in order to prevent fetal death. Prognostic models have been developed to predict the risk of stillbirth, but none has yet been validated externally. In this study, we externally validated published prediction models for stillbirth using individual participant data (IPD) meta-analysis to assess their predictive performance.

METHODS

MEDLINE, EMBASE, DH-DATA and AMED databases were searched from inception to December 2020 to identify studies reporting stillbirth prediction models. Studies that developed or updated prediction models for stillbirth for use at any time during pregnancy were included. IPD from cohorts within the International Prediction of Pregnancy Complications (IPPIC) Network were used to validate externally the identified prediction models whose individual variables were available in the IPD. The risk of bias of the models and cohorts was assessed using the Prediction study Risk Of Bias ASsessment Tool (PROBAST). The discriminative performance of the models was evaluated using the C-statistic, and calibration was assessed using calibration plots, calibration slope and calibration-in-the-large. Performance measures were estimated separately in each cohort, as well as summarized across cohorts using random-effects meta-analysis. Clinical utility was assessed using net benefit.

RESULTS

Seventeen studies reporting the development of 40 prognostic models for stillbirth were identified. None of the models had been previously validated externally, and the full model equation was reported for only one-fifth (20%, 8/40) of the models. External validation was possible for three of these models, using IPD from 19 cohorts (491 201 pregnant women) within the IPPIC Network database. Based on evaluation of the model development studies, all three models had an overall high risk of bias, according to PROBAST. In the IPD meta-analysis, the models had summary C-statistics ranging from 0.53 to 0.65 and summary calibration slopes ranging from 0.40 to 0.88, with risk predictions that were generally too extreme compared with the observed risks. The models had little to no clinical utility, as assessed by net benefit. However, there remained uncertainty in the performance of some models due to small available sample sizes.

CONCLUSIONS

The three validated stillbirth prediction models showed generally poor and uncertain predictive performance in new data, with limited evidence to support their clinical application. The findings suggest methodological shortcomings in their development, including overfitting. Further research is needed to further validate these and other models, identify stronger prognostic factors and develop more robust prediction models. © 2021 The Authors. Ultrasound in Obstetrics & Gynecology published by John Wiley & Sons Ltd on behalf of International Society of Ultrasound in Obstetrics and Gynecology.

Computational Models for Clinical Applications in Personalized Medicine—Guidelines and Recommendations for Data Integration and Model Validation

The future development of personalized medicine depends on a vast exchange of data from different sources, as well as harmonized integrative analysis of large-scale clinical health and sample data. Computational-modelling approaches play a key role in the analysis of the underlying molecular processes and pathways that characterize human biology, but they also lead to a more profound understanding of the mechanisms and factors that drive diseases; hence, they allow personalized treatment strategies that are guided by central clinical questions. However, despite the growing popularity of computational-modelling approaches in different stakeholder communities, there are still many hurdles to overcome for their clinical routine implementation in the future. Especially the integration of heterogeneous data from multiple sources and types are challenging tasks that require clear guidelines that also have to comply with high ethical and legal standards. Here, we discuss the most relevant computational models for personalized medicine in detail that can be considered as best-practice guidelines for application in clinical care. We define specific challenges and provide applicable guidelines and recommendations for study design, data acquisition, and operation as well as for model validation and clinical translation and other research areas.

Ruling out pulmonary embolism across different healthcare settings: A systematic review and individual patient data meta-analysis

BACKGROUND

The challenging clinical dilemma of detecting pulmonary embolism (PE) in suspected patients is encountered in a variety of healthcare settings. We hypothesized that the optimal diagnostic approach to detect these patients in terms of safety and efficiency depends on underlying PE prevalence, case mix, and physician experience, overall reflected by the type of setting where patients are initially assessed. The objective of this study was to assess the capability of ruling out PE by available diagnostic strategies across all possible settings.

METHODS AND FINDINGS

We performed a literature search (MEDLINE) followed by an individual patient data (IPD) meta-analysis (MA; 23 studies), including patients from self-referral emergency care (n = 12,612), primary healthcare clinics (n = 3,174), referred secondary care (n = 17,052), and hospitalized or nursing home patients (n = 2,410). Multilevel logistic regression was performed to evaluate diagnostic performance of the Wells and revised Geneva rules, both using fixed and adapted D-dimer thresholds to age or pretest probability (PTP), for the YEARS algorithm and for the Pulmonary Embolism Rule-out Criteria (PERC). All strategies were tested separately in each healthcare setting. Following studies done in this field, the primary diagnostic metrices estimated from the models were the "failure rate" of each strategy-i.e., the proportion of missed PE among patients categorized as "PE excluded" and "efficiency"-defined as the proportion of patients categorized as "PE excluded" among all patients. In self-referral emergency care, the PERC algorithm excludes PE in 21% of suspected patients at a failure rate of 1.12% (95% confidence interval [CI] 0.74 to 1.70), whereas this increases to 6.01% (4.09 to 8.75) in referred patients to secondary care at an efficiency of 10%. In patients from primary healthcare and those referred to secondary care, strategies adjusting D-dimer to PTP are the most efficient (range: 43% to 62%) at a failure rate ranging between 0.25% and 3.06%, with higher failure rates observed in patients referred to secondary care. For this latter setting, strategies adjusting D-dimer to age are associated with a lower failure rate ranging between 0.65% and 0.81%, yet are also less efficient (range: 33% and 35%). For all strategies, failure rates are highest in hospitalized or nursing home patients, ranging between 1.68% and 5.13%, at an efficiency ranging between 15% and 30%. The main limitation of the primary analyses was that the diagnostic performance of each strategy was compared in different sets of studies since the availability of items used in each diagnostic strategy differed across included studies; however, sensitivity analyses suggested that the findings were robust.

CONCLUSIONS

The capability of safely and efficiently ruling out PE of available diagnostic strategies differs for different healthcare settings. The findings of this IPD MA help in determining the optimum diagnostic strategies for ruling out PE per healthcare setting, balancing the trade-off between failure rate and efficiency of each strategy.

Individual participant data meta-analysis with mixed-effects transformation models

One-stage meta-analysis of individual participant data (IPD) poses several statistical and computational challenges. For time-to-event outcomes, the approach requires the estimation of complicated nonlinear mixed-effects models that are flexible enough to realistically capture the most important characteristics of the IPD. We present a model class that incorporates general normally distributed random effects into linear transformation models. We discuss extensions to model between-study heterogeneity in baseline risks and covariate effects and also relax the assumption of proportional hazards. Within the proposed framework, data with arbitrary random censoring patterns can be handled. The accompanying \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} }{}$\textsf{R}$\end{document} package tramME utilizes the Laplace approximation and automatic differentiation to perform efficient maximum likelihood estimation and inference in mixed-effects transformation models. We compare several variants of our model to predict the survival of patients with chronic obstructive pulmonary disease using a large data set of prognostic studies. Finally, a simulation study is presented that verifies the correctness of the implementation and highlights its efficiency compared to an alternative approach.

Prognostic models for mortality after cardiac surgery in patients with infective endocarditis: a systematic review and aggregation of prediction models

BACKGROUND

There are several prognostic models to estimate the risk of mortality after surgery for active infective endocarditis (IE). However, these models incorporate different predictors and their performance is uncertain.

OBJECTIVE

We systematically reviewed and critically appraised all available prediction models of postoperative mortality in patients undergoing surgery for IE, and aggregated them into a meta-model.

DATA SOURCES

We searched Medline and EMBASE databases from inception to June 2020.

STUDY ELIGIBILITY CRITERIA

We included studies that developed or updated a prognostic model of postoperative mortality in patient with IE.

METHODS

We assessed the risk of bias of the models using PROBAST (Prediction model Risk Of Bias ASsessment Tool) and we aggregated them into an aggregate meta-model based on stacked regressions and optimized it for a nationwide registry of IE patients. The meta-model performance was assessed using bootstrap validation methods and adjusted for optimism.

RESULTS

We identified 11 prognostic models for postoperative mortality. Eight models had a high risk of bias. The meta-model included weighted predictors from the remaining three models (EndoSCORE, specific ES-I and specific ES-II), which were not rated as high risk of bias and provided full model equations. Additionally, two variables (age and infectious agent) that had been modelled differently across studies, were estimated based on the nationwide registry. The performance of the meta-model was better than the original three models, with the corresponding performance measures: C-statistics 0.79 (95% CI 0.76-0.82), calibration slope 0.98 (95% CI 0.86-1.13) and calibration-in-the-large -0.05 (95% CI -0.20 to 0.11).

CONCLUSIONS

The meta-model outperformed published models and showed a robust predictive capacity for predicting the individualized risk of postoperative mortality in patients with IE.

PROTOCOL REGISTRATION

PROSPERO (registration number CRD42020192602).

Validation and development of models using clinical, biochemical and ultrasound markers for predicting pre-eclampsia: an individual participant data meta-analysis

BACKGROUND

Pre-eclampsia is a leading cause of maternal and perinatal mortality and morbidity. Early identification of women at risk is needed to plan management.

OBJECTIVES

To assess the performance of existing pre-eclampsia prediction models and to develop and validate models for pre-eclampsia using individual participant data meta-analysis. We also estimated the prognostic value of individual markers.

DESIGN

This was an individual participant data meta-analysis of cohort studies.

SETTING

Source data from secondary and tertiary care.

PREDICTORS

We identified predictors from systematic reviews, and prioritised for importance in an international survey.

PRIMARY OUTCOMES

Early-onset (delivery at < 34 weeks' gestation), late-onset (delivery at ≥ 34 weeks' gestation) and any-onset pre-eclampsia.

ANALYSIS

We externally validated existing prediction models in UK cohorts and reported their performance in terms of discrimination and calibration. We developed and validated 12 new models based on clinical characteristics, clinical characteristics and biochemical markers, and clinical characteristics and ultrasound markers in the first and second trimesters. We summarised the data set-specific performance of each model using a random-effects meta-analysis. Discrimination was considered promising for C-statistics of ≥ 0.7, and calibration was considered good if the slope was near 1 and calibration-in-the-large was near 0. Heterogeneity was quantified using I ² and τ². A decision curve analysis was undertaken to determine the clinical utility (net benefit) of the models. We reported the unadjusted prognostic value of individual predictors for pre-eclampsia as odds ratios with 95% confidence and prediction intervals.

RESULTS

The International Prediction of Pregnancy Complications network comprised 78 studies (3,570,993 singleton pregnancies) identified from systematic reviews of tests to predict pre-eclampsia. Twenty-four of the 131 published prediction models could be validated in 11 UK cohorts. Summary C-statistics were between 0.6 and 0.7 for most models, and calibration was generally poor owing to large between-study heterogeneity, suggesting model overfitting. The clinical utility of the models varied between showing net harm to showing minimal or no net benefit. The average discrimination for IPPIC models ranged between 0.68 and 0.83. This was highest for the second-trimester clinical characteristics and biochemical markers model to predict early-onset pre-eclampsia, and lowest for the first-trimester clinical characteristics models to predict any pre-eclampsia. Calibration performance was heterogeneous across studies. Net benefit was observed for International Prediction of Pregnancy Complications first and second-trimester clinical characteristics and clinical characteristics and biochemical markers models predicting any pre-eclampsia, when validated in singleton nulliparous women managed in the UK NHS. History of hypertension, parity, smoking, mode of conception, placental growth factor and uterine artery pulsatility index had the strongest unadjusted associations with pre-eclampsia.

LIMITATIONS

Variations in study population characteristics, type of predictors reported, too few events in some validation cohorts and the type of measurements contributed to heterogeneity in performance of the International Prediction of Pregnancy Complications models. Some published models were not validated because model predictors were unavailable in the individual participant data.

CONCLUSION

For models that could be validated, predictive performance was generally poor across data sets. Although the International Prediction of Pregnancy Complications models show good predictive performance on average, and in the singleton nulliparous population, heterogeneity in calibration performance is likely across settings.

FUTURE WORK

Recalibration of model parameters within populations may improve calibration performance. Additional strong predictors need to be identified to improve model performance and consistency. Validation, including examination of calibration heterogeneity, is required for the models we could not validate.

STUDY REGISTRATION

This study is registered as PROSPERO CRD42015029349.

FUNDING

This project was funded by the National Institute for Health Research (NIHR) Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 24, No. 72. See the NIHR Journals Library website for further project information.

Practical Considerations and Challenges When Conducting an Individual Participant Data (IPD) Meta-Analysis

This chapter provides a broad overview of the use of individual participant (sometimes referred to as patient) data (IPD ) within meta-analyses, the associated advantages of using IPD in meta-analysis compared to aggregate data, and when IPD should be used in meta-analysis.This chapter also outlines the steps of conducting an IPD meta-analysis, with practical guidance relating to requesting and obtaining IPD for meta-analysis. Challenges that can be associated with conducting an IPD meta-analysis are also discussed, including consideration of availability bias, when a subset of the relevant IPD is not available for meta-analysis.

Progress and challenges of network meta-analysis

In the past years, network meta-analysis (NMA) has been widely used among clinicians, guideline makers, and health technology assessment agencies and has played an important role in clinical decision-making and guideline development. To inform further development of NMAs, we conducted a bibliometric analysis to assess the current status of published NMA methodological studies, summarized the methodological progress of seven types of NMAs, and discussed the current challenges of NMAs.

Internal-external cross-validation helped to evaluate the generalizability of prediction models in large clustered datasets

OBJECTIVE

To illustrate how to evaluate the need of complex strategies for developing generalizable prediction models in large clustered datasets.

STUDY DESIGN AND SETTING

We developed eight Cox regression models to estimate the risk of heart failure using a large population-level dataset. These models differed in the number of predictors, the functional form of the predictor effects (non-linear effects and interaction) and the estimation method (maximum likelihood and penalization). Internal-external cross-validation was used to evaluate the models' generalizability across the included general practices.

RESULTS

Among 871,687 individuals from 225 general practices, 43,987 (5.5%) developed heart failure during a median follow-up time of 5.8 years. For discrimination, the simplest prediction model yielded a good concordance statistic, which was not much improved by adopting complex strategies. Between-practice heterogeneity in discrimination was similar in all models. For calibration, the simplest model performed satisfactorily. Although accounting for non-linear effects and interaction slightly improved the calibration slope, it also led to more heterogeneity in the observed/expected ratio. Similar results were found in a second case study involving patients with stroke.

CONCLUSION

In large clustered datasets, prediction model studies may adopt internal-external cross-validation to evaluate the generalizability of competing models, and to identify promising modelling strategies.

Identifying adults with acute rhinosinusitis in primary care that benefit most from antibiotics: protocol of an individual patient data meta-analysis using multivariable risk prediction modelling

INTRODUCTION

Acute rhinosinusitis (ARS) is a prime reason for doctor visits and among the conditions with highest antibiotic overprescribing rates in adults. To reduce inappropriate prescribing, we aim to predict the absolute benefit of antibiotic treatment for individual adult patients with ARS by applying multivariable risk prediction methods to individual patient data (IPD) of multiple randomised placebo-controlled trials.

METHODS AND ANALYSIS

This is an update and re-analysis of a 2008 IPD meta-analysis on antibiotics for adults with clinically diagnosed ARS. First, the reference list of the 2018 Cochrane review on antibiotics for ARS will be reviewed for relevant studies published since 2008. Next, the systematic searches of CENTRAL, MEDLINE and Embase of the Cochrane review will be updated to 1 September 2020. Methodological quality of eligible studies will be assessed using the Cochrane Risk of Bias 2 tool. The primary outcome is cure at 8-15 days. Regression-based methods will be used to model the risk of being cured based on relevant predictors and treatment, while accounting for clustering. Such model allows for risk predictions as a function of treatment and individual patient characteristics and hence gives insight into individualised absolute benefit. Candidate predictors will be based on literature, clinical reasoning and availability. Calibration and discrimination will be evaluated to assess model performance. Resampling techniques will be used to assess internal validation. In addition, internal-external cross-validation procedures will be used to inform on between-study differences and estimate out-of-sample model performance. Secondarily, we will study possible heterogeneity of treatment effect as a function of outcome risk.

ETHICS AND DISSEMINATION

In this study, no identifiable patient data will be used. As such, the Medical Research Involving Humans Subject Act (WMO) does not apply and official ethical approval is not required. Results will be submitted for publication in international peer-reviewed journals.

PROSPERO REGISTRATION NUMBER

CRD42020220108.

Right ventricle assessment in patients with pulmonary embolism at low risk for death based on clinical models: an individual patient data meta-analysis

AIMS

Patients with acute pulmonary embolism (PE) at low risk for short-term death are candidates for home treatment or short-hospital stay. We aimed at determining whether the assessment of right ventricle dysfunction (RVD) or elevated troponin improves identification of low-risk patients over clinical models alone.

METHODS AND RESULTS

Individual patient data meta-analysis of studies assessing the relationship between RVD or elevated troponin and short-term mortality in patients with acute PE at low risk for death based on clinical models (Pulmonary Embolism Severity Index, simplified Pulmonary Embolism Severity Index or Hestia). The primary study outcome was short-term death defined as death occurring in hospital or within 30 days. Individual data of 5010 low-risk patients from 18 studies were pooled. Short-term mortality was 0.7% [95% confidence interval (CI) 0.4-1.3]. RVD at echocardiography, computed tomography or B-type natriuretic peptide (BNP)/N-terminal pro BNP (NT-proBNP) was associated with increased risk for short-term death (1.5 vs. 0.3%; OR 4.81, 95% CI 1.98-11.68), death within 3 months (1.6 vs. 0.4%; OR 4.03, 95% CI 2.01-8.08), and PE-related death (1.1 vs. 0.04%; OR 22.9, 95% CI 2.89-181). Elevated troponin was associated with short-term death (OR 2.78, 95% CI 1.06-7.26) and death within 3 months (OR 3.68, 95% CI 1.75-7.74).

CONCLUSION

RVD assessed by echocardiography, computed tomography, or elevated BNP/NT-proBNP levels and increased troponin are associated with short-term death in patients with acute PE at low risk based on clinical models. RVD assessment, mainly by BNP/NT-proBNP or echocardiography, should be considered to improve identification of low-risk patients that may be candidates for outpatient management or short hospital stay.

Developing more generalizable prediction models from pooled studies and large clustered data sets

Prediction models often yield inaccurate predictions for new individuals. Large data sets from pooled studies or electronic healthcare records may alleviate this with an increased sample size and variability in sample characteristics. However, existing strategies for prediction model development generally do not account for heterogeneity in predictor‐outcome associations between different settings and populations. This limits the generalizability of developed models (even from large, combined, clustered data sets) and necessitates local revisions. We aim to develop methodology for producing prediction models that require less tailoring to different settings and populations. We adopt internal‐external cross‐validation to assess and reduce heterogeneity in models' predictive performance during the development. We propose a predictor selection algorithm that optimizes the (weighted) average performance while minimizing its variability across the hold‐out clusters (or studies). Predictors are added iteratively until the estimated generalizability is optimized. We illustrate this by developing a model for predicting the risk of atrial fibrillation and updating an existing one for diagnosing deep vein thrombosis, using individual participant data from 20 cohorts (N = 10 873) and 11 diagnostic studies (N = 10 014), respectively. Meta‐analysis of calibration and discrimination performance in each hold‐out cluster shows that trade‐offs between average and heterogeneity of performance occurred. Our methodology enables the assessment of heterogeneity of prediction model performance during model development in multiple or clustered data sets, thereby informing researchers on predictor selection to improve the generalizability to different settings and populations, and reduce the need for model tailoring. Our methodology has been implemented in the R package metamisc.

Development and validation of the ISARIC 4C Deterioration model for adults hospitalised with COVID-19: a prospective cohort study

BACKGROUND

Prognostic models to predict the risk of clinical deterioration in acute COVID-19 cases are urgently required to inform clinical management decisions.

METHODS

We developed and validated a multivariable logistic regression model for in-hospital clinical deterioration (defined as any requirement of ventilatory support or critical care, or death) among consecutively hospitalised adults with highly suspected or confirmed COVID-19 who were prospectively recruited to the International Severe Acute Respiratory and Emerging Infections Consortium Coronavirus Clinical Characterisation Consortium (ISARIC4C) study across 260 hospitals in England, Scotland, and Wales. Candidate predictors that were specified a priori were considered for inclusion in the model on the basis of previous prognostic scores and emerging literature describing routinely measured biomarkers associated with COVID-19 prognosis. We used internal-external cross-validation to evaluate discrimination, calibration, and clinical utility across eight National Health Service (NHS) regions in the development cohort. We further validated the final model in held-out data from an additional NHS region (London).

FINDINGS

74 944 participants (recruited between Feb 6 and Aug 26, 2020) were included, of whom 31 924 (43·2%) of 73 948 with available outcomes met the composite clinical deterioration outcome. In internal-external cross-validation in the development cohort of 66 705 participants, the selected model (comprising 11 predictors routinely measured at the point of hospital admission) showed consistent discrimination, calibration, and clinical utility across all eight NHS regions. In held-out data from London (n=8239), the model showed a similarly consistent performance (C-statistic 0·77 [95% CI 0·76 to 0·78]; calibration-in-the-large 0·00 [-0·05 to 0·05]); calibration slope 0·96 [0·91 to 1·01]), and greater net benefit than any other reproducible prognostic model.

INTERPRETATION

The 4C Deterioration model has strong potential for clinical utility and generalisability to predict clinical deterioration and inform decision making among adults hospitalised with COVID-19.

FUNDING

National Institute for Health Research (NIHR), UK Medical Research Council, Wellcome Trust, Department for International Development, Bill & Melinda Gates Foundation, EU Platform for European Preparedness Against (Re-)emerging Epidemics, NIHR Health Protection Research Unit (HPRU) in Emerging and Zoonotic Infections at University of Liverpool, NIHR HPRU in Respiratory Infections at Imperial College London.

Development and Validation of a Prediction Model for 6-Month Societal Costs in Older Community Care-Recipients in Multiple Countries; the IBenC Study

This study aims to develop and validate a prediction model of societal costs during a period of 6-months in older community care-recipients across multiple European countries. Participants were older community care-recipients from 5 European countries. The outcome measure was mean 6-months total societal costs of resource utilisation (healthcare and informal care). Potential predictors included sociodemographic characteristics, functional limitations, clinical conditions, and diseases/disorders. The model was developed by performing Linear Mixed Models with a random intercept for the effect of country and validated by an internal-external validation procedure. Living alone, caregiver distress, (I)ADL impairment, required level of care support, health instability, presence of pain, behavioural problems, urinary incontinence and multimorbidity significantly predicted societal costs during 6 months. The model explained 32% of the variation within societal costs and showed good calibration in Iceland, Finland and Germany. Minor model adaptations improved model performance in The Netherland and Italy. The results can provide a valuable orientation for policymakers to better understand cost development among older community care-recipients. Despite substantial differences of countries’ care systems, a validated cross-national set of key predictors could be identified.

Continual updating and monitoring of clinical prediction models: time for dynamic prediction systems?

Clinical prediction models (CPMs) have become fundamental for risk stratification across healthcare. The CPM pipeline (development, validation, deployment, and impact assessment) is commonly viewed as a one-time activity, with model updating rarely considered and done in a somewhat ad hoc manner. This fails to address the fact that the performance of a CPM worsens over time as natural changes in populations and care pathways occur. CPMs need constant surveillance to maintain adequate predictive performance. Rather than reactively updating a developed CPM once evidence of deteriorated performance accumulates, it is possible to proactively adapt CPMs whenever new data becomes available. Approaches for validation then need to be changed accordingly, making validation a continuous rather than a discrete effort. As such, "living" (dynamic) CPMs represent a paradigm shift, where the analytical methods dynamically generate updated versions of a model through time; one then needs to validate the system rather than each subsequent model revision.

IPDmada: An R Shiny tool for analyzing and visualizing individual patient data meta-analyses of diagnostic test accuracy

BACKGROUND

Individual patient data meta-analyses (IPD-MA) are regarded as the gold standard for systematic reviews, which also applies to systematic reviews of diagnostic test accuracy (DTA) studies. An increasing number of DTA systematic reviews with IPD-MA have been published in recent years, but there is much variation in how these IPD-MA were performed. A number of existing methods were found, but there is no consensus as to which methods are preferred as the standard methods for statistical analysis in DTA IPD-MA.

OBJECTIVES

To create a web-based tool which integrates recommended statistical analyses for DTA IPD-MA, and allows researchers to analyse the data and visualize the results with interactive plots.

METHODS

A systematic methodological review was performed to identify statistical analyses and data visualization methods used in DTA IPD-MA. Methods were evaluated by the authors and recommended analyses were integrated into the IPDmada tool which is freely available online with the user interface developed with R Shiny package.

RESULTS

IPDmada allows users to upload their own data, perform the meta-analysis with both continuous and dichotomized tests, and incorporate individual level covariate-adjusted analysis. All tables and figures can be exported as .csv or .pdf files. A hypothetical dataset was used to illustrate the application of IPDmada.

CONCLUSIONS

IPDmada will be very helpful to researchers doing DTA IPD-MA, since it not only facilitates the statistical analysis but also provides a standard framework. The introduction of IPDmada will harmonize the methods used in DTA IPD-MA and ensure the quality of such analyses.

HIGHLIGHTS

IPDmada is a newly developed web-based tool for performing statistical analysis of individual patient data meta-analysis of diagnostic accuracy and visualizing the results. The tool is freely available to all the researchers, and requiring no installation of statistical software/packages. The tool has an user-friendly interface, and allows meta-analysis on both dichotomized and continuous test results. Researchers can easily use this tool to investigate the threshold effect and covariate effect on the summary accuracy. The introduction and implementation of IPDmada will serve as a useful tool for DTA IPD-MA and increase the quality of such studies.

Systematic evaluation and external validation of 22 prognostic models among hospitalised adults with COVID-19: an observational cohort study

The number of proposed prognostic models for coronavirus disease 2019 (COVID-19) is growing rapidly, but it is unknown whether any are suitable for widespread clinical implementation.We independently externally validated the performance of candidate prognostic models, identified through a living systematic review, among consecutive adults admitted to hospital with a final diagnosis of COVID-19. We reconstructed candidate models as per original descriptions and evaluated performance for their original intended outcomes using predictors measured at the time of admission. We assessed discrimination, calibration and net benefit, compared to the default strategies of treating all and no patients, and against the most discriminating predictors in univariable analyses.We tested 22 candidate prognostic models among 411 participants with COVID-19, of whom 180 (43.8%) and 115 (28.0%) met the endpoints of clinical deterioration and mortality, respectively. Highest areas under receiver operating characteristic (AUROC) curves were achieved by the NEWS2 score for prediction of deterioration over 24 h (0.78, 95% CI 0.73-0.83), and a novel model for prediction of deterioration <14 days from admission (0.78, 95% CI 0.74-0.82). The most discriminating univariable predictors were admission oxygen saturation on room air for in-hospital deterioration (AUROC 0.76, 95% CI 0.71-0.81), and age for in-hospital mortality (AUROC 0.76, 95% CI 0.71-0.81). No prognostic model demonstrated consistently higher net benefit than these univariable predictors, across a range of threshold probabilities.Admission oxygen saturation on room air and patient age are strong predictors of deterioration and mortality among hospitalised adults with COVID-19, respectively. None of the prognostic models evaluated here offered incremental value for patient stratification to these univariable predictors.

Discovery and validation of a personalized risk predictor for incident tuberculosis in low transmission settings

The risk of tuberculosis (TB) is variable among individuals with latent Mycobacterium tuberculosis infection (LTBI), but validated estimates of personalized risk are lacking. In pooled data from 18 systematically identified cohort studies from 20 countries, including 80,468 individuals tested for LTBI, 5-year cumulative incident TB risk among people with untreated LTBI was 15.6% (95% confidence interval (CI), 8.0-29.2%) among child contacts, 4.8% (95% CI, 3.0-7.7%) among adult contacts, 5.0% (95% CI, 1.6-14.5%) among migrants and 4.8% (95% CI, 1.5-14.3%) among immunocompromised groups. We confirmed highly variable estimates within risk groups, necessitating an individualized approach to risk stratification. Therefore, we developed a personalized risk predictor for incident TB (PERISKOPE-TB) that combines a quantitative measure of T cell sensitization and clinical covariates. Internal-external cross-validation of the model demonstrated a random effects meta-analysis C-statistic of 0.88 (95% CI, 0.82-0.93) for incident TB. In decision curve analysis, the model demonstrated clinical utility for targeting preventative treatment, compared to treating all, or no, people with LTBI. We challenge the current crude approach to TB risk estimation among people with LTBI in favor of our evidence-based and patient-centered method, in settings aiming for pre-elimination worldwide.

Influence of the month of birth on persistence of ADHD in prospective studies: protocol for an individual patient data meta-analysis

INTRODUCTION

Attention-deficit/hyperactivity disorder (ADHD) is a neurodevelopmental disorder with symptoms, especially the hyperactive ones, that tend to decrease in severity with age. Interestingly, children born just before the school-entry cut-off date (ie, the youngest pupils of a classroom) are at higher risk of being diagnosed with ADHD compared with children born just after the cut-off date. Noteworthy, this month-of-birth effect tends to disappear with increasing absolute age. Therefore, it is possible that young children erroneously diagnosed with ADHD due to their month of birth present a lower chance to have their diagnosis confirmed at a later age, artificially reinforcing the low persistence of ADHD across the lifespan. This protocol outlines an individual patient data (IPD) meta-analysis of prospective observational studies to explore the role of the month of birth in the low persistence of ADHD across the lifespan.

METHODS AND ANALYSIS

Five databases will be systematically searched in order to find prospective observational studies where the presence of ADHD is assessed both at baseline and at a follow-up of at least 4 years. We will use a two-stage IPD meta-analytic approach to estimate the role of the month of birth in the persistence of ADHD. Various sensitivity analyses will be performed to assess the robustness of the results.

ETHICS AND DISSEMINATION

No additional data will be collected and no de-identified raw data will be used. Ethics approval is thus not required for the present study. Results of this IPD meta-analysis will be submitted for publication in a peer-reviewed journal.

PROSPERO REGISTRATION NUMBER

CRD42020212650.

Accuracy of signs, symptoms and blood tests for diagnosing acute bacterial rhinosinusitis and CT-confirmed acute rhinosinusitis in adults: protocol of an individual patient data meta-analysis

INTRODUCTION

This protocol outlines a diagnostic individual patient data (IPD) meta-analysis aimed at developing simple prediction models based on readily available signs, symptoms and blood tests to accurately predict acute bacterial rhinosinusitis and CT-confirmed (fluid level or total opacification in any sinus) acute rhinosinusitis (ARS) in adults presenting to primary care with clinically diagnosed ARS, target conditions associated with antibiotic benefit.

METHODS AND ANALYSIS

The systematic searches of PubMed and Embase of a review on the accuracy of signs and symptoms for diagnosing ARS in ambulatory care will be updated to April 2020 to identify relevant studies. Authors of eligible studies will be contacted and invited to provide IPD. Methodological quality of the studies will be assessed using the Quality Assessment of Diagnostic Accuracy Studies-2 tool. Candidate predictor selection will be based on knowledge from existing literature, clinical reasoning and availability. Multivariable logistic regression analyses will be used to develop prediction models aimed at calculating absolute risk estimates. Large unexplained between-study heterogeneity in predictive accuracy of the models will be explored and may lead to either model adjustment or derivation of separate context-specific models. Calibration and discrimination will be evaluated to assess the models' performance. Bootstrap resampling techniques will be used to assess internal validation and to inform on possible adjustment for overfitting. In addition, we aim to perform internal-external cross-validation procedures.

ETHICS AND DISSEMINATION

In this IPD meta-analysis, no identifiable patient data will be used. As such, the Medical Research Involving Humans Subject Act does not apply, and official ethical approval is not required. Findings will be published in international peer-reviewed journals and presented at scientific conferences.

PROSPERO REGISTRATION NUMBER

PROSPERO CRD42020175659.

External validation of prognostic models predicting pre-eclampsia: individual participant data meta-analysis

BACKGROUND

Pre-eclampsia is a leading cause of maternal and perinatal mortality and morbidity. Early identification of women at risk during pregnancy is required to plan management. Although there are many published prediction models for pre-eclampsia, few have been validated in external data. Our objective was to externally validate published prediction models for pre-eclampsia using individual participant data (IPD) from UK studies, to evaluate whether any of the models can accurately predict the condition when used within the UK healthcare setting.

METHODS

IPD from 11 UK cohort studies (217,415 pregnant women) within the International Prediction of Pregnancy Complications (IPPIC) pre-eclampsia network contributed to external validation of published prediction models, identified by systematic review. Cohorts that measured all predictor variables in at least one of the identified models and reported pre-eclampsia as an outcome were included for validation. We reported the model predictive performance as discrimination (C-statistic), calibration (calibration plots, calibration slope, calibration-in-the-large), and net benefit. Performance measures were estimated separately in each available study and then, where possible, combined across studies in a random-effects meta-analysis.

RESULTS

Of 131 published models, 67 provided the full model equation and 24 could be validated in 11 UK cohorts. Most of the models showed modest discrimination with summary C-statistics between 0.6 and 0.7. The calibration of the predicted compared to observed risk was generally poor for most models with observed calibration slopes less than 1, indicating that predictions were generally too extreme, although confidence intervals were wide. There was large between-study heterogeneity in each model's calibration-in-the-large, suggesting poor calibration of the predicted overall risk across populations. In a subset of models, the net benefit of using the models to inform clinical decisions appeared small and limited to probability thresholds between 5 and 7%.

CONCLUSIONS

The evaluated models had modest predictive performance, with key limitations such as poor calibration (likely due to overfitting in the original development datasets), substantial heterogeneity, and small net benefit across settings. The evidence to support the use of these prediction models for pre-eclampsia in clinical decision-making is limited. Any models that we could not validate should be examined in terms of their predictive performance, net benefit, and heterogeneity across multiple UK settings before consideration for use in practice.

TRIAL REGISTRATION

PROSPERO ID: CRD42015029349 .

Prognostic models for chronic kidney disease: a systematic review and external validation

BACKGROUND

Accurate risk prediction is needed in order to provide personalized healthcare for chronic kidney disease (CKD) patients. An overload of prognosis studies is being published, ranging from individual biomarker studies to full prediction studies. We aim to systematically appraise published prognosis studies investigating multiple biomarkers and their role in risk predictions. Our primary objective was to investigate if the prognostic models that are reported in the literature were of sufficient quality and to externally validate them.

METHODS

We undertook a systematic review and appraised the quality of studies reporting multivariable prognosis models for end-stage renal disease (ESRD), cardiovascular (CV) events and mortality in CKD patients. We subsequently externally validated these models in a randomized trial that included patients from a broad CKD population.

RESULTS

We identified 91 papers describing 36 multivariable models for prognosis of ESRD, 50 for CV events, 46 for mortality and 17 for a composite outcome. Most studies were deemed of moderate quality. Moreover, they often adopted different definitions for the primary outcome and rarely reported full model equations (21% of the included studies). External validation was performed in the Multifactorial Approach and Superior Treatment Efficacy in Renal Patients with the Aid of Nurse Practitioners trial (n = 788, with 160 events for ESRD, 79 for CV and 102 for mortality). The 24 models that reported full model equations showed a great variability in their performance, although calibration remained fairly adequate for most models, except when predicting mortality (calibration slope >1.5).

CONCLUSIONS

This review shows that there is an abundance of multivariable prognosis models for the CKD population. Most studies were considered of moderate quality, and they were reported and analysed in such a manner that their results cannot directly be used in follow-up research or in clinical practice.

Can personalized treatment prediction improve the outcomes, compared with the group average approach, in a randomized trial? Developing and validating a multivariable prediction model in a pragmatic megatrial of acute treatment for major depression

BACKGROUND

Clinical trials have traditionally been analysed at the aggregate level, assuming that the group average would be applicable to all eligible and similar patients. We re-analyzed a mega-trial of antidepressant therapy for major depression to explore whether a multivariable prediction model may lead to different treatment recommendations for individual participants.

METHODS

The trial compared the second-line treatment strategies of continuing sertraline, combining it with mirtazapine or switching to mirtazapine after initial failure to remit on sertraline among 1,544 patients with major depression. The outcome was the Personal Health Questionnaire-9 (PHQ-9) at week 9: the original analyses showed that both combining and switching resulted in greater reduction in PHQ-9 by 1.0 point than continuing. We considered several models of penalized regression or machine learning.

RESULTS

Models using support vector machines (SVMs) provided the best performance. Using SVMs, continuing sertraline was predicted to be the best treatment for 123 patients, combining for 696 patients, and switching for 725 patients. In the last two subgroups, both combining and switching were equally superior to continuing by 1.2 to 1.4 points, resulting in the same treatment recommendations as with the original aggregate data level analyses; in the first subgroup, however, switching was substantively inferior to combining (-3.1, 95%CI: -5.4 to -0.5).

LIMITATIONS

Stronger predictors are needed to make more precise predictions.

CONCLUSIONS

The multivariable prediction models led to improved recommendations for a minority of participants than the group average approach in a megatrial.