Minimum sample size for developing a multivariable prediction model: PART II - binary and time-to-event outcomes

When designing a study to develop a new prediction model with binary or time-to-event outcomes, researchers should ensure their sample size is adequate in terms of the number of participants (n) and outcome events (E) relative to the number of predictor parameters (p) considered for inclusion. We propose that the minimum values of n and E (and subsequently the minimum number of events per predictor parameter, EPP) should be calculated to meet the following three criteria: (i) small optimism in predictor effect estimates as defined by a global shrinkage factor of ≥0.9, (ii) small absolute difference of ≤ 0.05 in the model's apparent and adjusted Nagelkerke's R2 , and (iii) precise estimation of the overall risk in the population. Criteria (i) and (ii) aim to reduce overfitting conditional on a chosen p, and require prespecification of the model's anticipated Cox-Snell R2 , which we show can be obtained from previous studies. The values of n and E that meet all three criteria provides the minimum sample size required for model development. Upon application of our approach, a new diagnostic model for Chagas disease requires an EPP of at least 4.8 and a new prognostic model for recurrent venous thromboembolism requires an EPP of at least 23. This reinforces why rules of thumb (eg, 10 EPP) should be avoided. Researchers might additionally ensure the sample size gives precise estimates of key predictor effects; this is especially important when key categorical predictors have few events in some categories, as this may substantially increase the numbers required.

Minimum sample size for developing a multivariable prediction model: Part I - Continuous outcomes

In the medical literature, hundreds of prediction models are being developed to predict health outcomes in individuals. For continuous outcomes, typically a linear regression model is developed to predict an individual's outcome value conditional on values of multiple predictors (covariates). To improve model development and reduce the potential for overfitting, a suitable sample size is required in terms of the number of subjects (n) relative to the number of predictor parameters (p) for potential inclusion. We propose that the minimum value of n should meet the following four key criteria: (i) small optimism in predictor effect estimates as defined by a global shrinkage factor of ≥0.9; (ii) small absolute difference of ≤ 0.05 in the apparent and adjusted R2 ; (iii) precise estimation (a margin of error ≤ 10% of the true value) of the model's residual standard deviation; and similarly, (iv) precise estimation of the mean predicted outcome value (model intercept). The criteria require prespecification of the user's chosen p and the model's anticipated R2 as informed by previous studies. The value of n that meets all four criteria provides the minimum sample size required for model development. In an applied example, a new model to predict lung function in African-American women using 25 predictor parameters requires at least 918 subjects to meet all criteria, corresponding to at least 36.7 subjects per predictor parameter. Even larger sample sizes may be needed to additionally ensure precise estimates of key predictor effects, especially when important categorical predictors have low prevalence in certain categories.

PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies

Clinical prediction models combine multiple predictors to estimate risk for the presence of a particular condition (diagnostic models) or the occurrence of a certain event in the future (prognostic models). PROBAST (Prediction model Risk Of Bias ASsessment Tool), a tool for assessing the risk of bias (ROB) and applicability of diagnostic and prognostic prediction model studies, was developed by a steering group that considered existing ROB tools and reporting guidelines. The tool was informed by a Delphi procedure involving 38 experts and was refined through piloting. PROBAST is organized into the following 4 domains: participants, predictors, outcome, and analysis. These domains contain a total of 20 signaling questions to facilitate structured judgment of ROB, which was defined to occur when shortcomings in study design, conduct, or analysis lead to systematically distorted estimates of model predictive performance. PROBAST enables a focused and transparent approach to assessing the ROB and applicability of studies that develop, validate, or update prediction models for individualized predictions. Although PROBAST was designed for systematic reviews, it can be used more generally in critical appraisal of prediction model studies. Potential users include organizations supporting decision making, researchers and clinicians who are interested in evidence-based medicine or involved in guideline development, journal editors, and manuscript reviewers.

Predictors of the effects of treatment for shoulder pain: protocol of an individual participant data meta-analysis

BACKGROUND

Shoulder pain is one of the most common presentations of musculoskeletal pain with a 1-month population prevalence of between 7 and 26%. The overall prognosis of shoulder pain is highly variable with 40% of patients reporting persistent pain 1 year after consulting their primary care clinician. Despite evidence for prognostic value of a range of patient and disease characteristics, it is not clear whether these factors also predict (moderate) the effect of specific treatments (such as corticosteroid injection, exercise, or surgery).

OBJECTIVES

This study aims to identify predictors of treatment effect (i.e. treatment moderators or effect modifiers) by investigating the association between a number of pre-defined individual-level factors and the effects of commonly used treatments on shoulder pain and disability outcomes.

METHODS

This will be a meta-analysis using individual participant data (IPD). Eligible trials investigating the effectiveness of advice and analgesics, corticosteroid injection, physiotherapy-led exercise, psychological interventions, and/or surgical treatment in patients with shoulder conditions will be identified from systematic reviews and an updated systematic search for trials, and risk of bias will be assessed. Authors of all eligible trials will be approached for data sharing. Outcomes measured will be shoulder pain and disability, and our previous work has identified candidate predictors. The main analysis will be conducted using hierarchical one-stage IPD meta-analysis models, examining the effect of treatment-predictor interaction on outcome for each of the candidate predictors and describing relevant subgroup effects where significant interaction effects are detected. Random effects will be used to account for clustering and heterogeneity. Sensitivity analyses will be based on (i) exclusion of trials at high risk of bias, (ii) use of restricted cubic splines to model potential non-linear associations for candidate predictors measured on a continuous scale, and (iii) the use of a two-stage IPD meta-analysis framework.

DISCUSSION

Our study will collate, appraise, and synthesise IPD from multiple studies to examine potential predictors of treatment effect in order to assess the potential for better and more efficient targeting of specific treatments for individuals with shoulder pain.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42018088298.

PROBAST: A Tool to Assess Risk of Bias and Applicability of Prediction Model Studies: Explanation and Elaboration

Prediction models in health care use predictors to estimate for an individual the probability that a condition or disease is already present (diagnostic model) or will occur in the future (prognostic model). Publications on prediction models have become more common in recent years, and competing prediction models frequently exist for the same outcome or target population. Health care providers, guideline developers, and policymakers are often unsure which model to use or recommend, and in which persons or settings. Hence, systematic reviews of these studies are increasingly demanded, required, and performed. A key part of a systematic review of prediction models is examination of risk of bias and applicability to the intended population and setting. To help reviewers with this process, the authors developed PROBAST (Prediction model Risk Of Bias ASsessment Tool) for studies developing, validating, or updating (for example, extending) prediction models, both diagnostic and prognostic. PROBAST was developed through a consensus process involving a group of experts in the field. It includes 20 signaling questions across 4 domains (participants, predictors, outcome, and analysis). This explanation and elaboration document describes the rationale for including each domain and signaling question and guides researchers, reviewers, readers, and guideline developers in how to use them to assess risk of bias and applicability concerns. All concepts are illustrated with published examples across different topics. The latest version of the PROBAST checklist, accompanying documents, and filled-in examples can be downloaded from www.probast.org.

PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies

Clinical prediction models combine multiple predictors to estimate risk for the presence of a particular condition (diagnostic models) or the occurrence of a certain event in the future (prognostic models). PROBAST (Prediction model Risk Of Bias ASsessment Tool), a tool for assessing the risk of bias (ROB) and applicability of diagnostic and prognostic prediction model studies, was developed by a steering group that considered existing ROB tools and reporting guidelines. The tool was informed by a Delphi procedure involving 38 experts and was refined through piloting. PROBAST is organized into the following 4 domains: participants, predictors, outcome, and analysis. These domains contain a total of 20 signaling questions to facilitate structured judgment of ROB, which was defined to occur when shortcomings in study design, conduct, or analysis lead to systematically distorted estimates of model predictive performance. PROBAST enables a focused and transparent approach to assessing the ROB and applicability of studies that develop, validate, or update prediction models for individualized predictions. Although PROBAST was designed for systematic reviews, it can be used more generally in critical appraisal of prediction model studies. Potential users include organizations supporting decision making, researchers and clinicians who are interested in evidence-based medicine or involved in guideline development, journal editors, and manuscript reviewers.

Meta-analysis of prediction model performance across multiple studies: Which scale helps ensure between-study normality for the C-statistic and calibration measures?

If individual participant data are available from multiple studies or clusters, then a prediction model can be externally validated multiple times. This allows the model's discrimination and calibration performance to be examined across different settings. Random-effects meta-analysis can then be used to quantify overall (average) performance and heterogeneity in performance. This typically assumes a normal distribution of 'true' performance across studies. We conducted a simulation study to examine this normality assumption for various performance measures relating to a logistic regression prediction model. We simulated data across multiple studies with varying degrees of variability in baseline risk or predictor effects and then evaluated the shape of the between-study distribution in the C-statistic, calibration slope, calibration-in-the-large, and E/O statistic, and possible transformations thereof. We found that a normal between-study distribution was usually reasonable for the calibration slope and calibration-in-the-large; however, the distributions of the C-statistic and E/O were often skewed across studies, particularly in settings with large variability in the predictor effects. Normality was vastly improved when using the logit transformation for the C-statistic and the log transformation for E/O, and therefore we recommend these scales to be used for meta-analysis. An illustrated example is given using a random-effects meta-analysis of the performance of QRISK2 across 25 general practices.

Development and validation of prediction models to estimate risk of primary total hip and knee replacements using data from the UK: two prospective open cohorts using the UK Clinical Practice Research Datalink

Objectives

The ability to efficiently and accurately predict future risk of primary total hip and knee replacement (THR/TKR) in earlier stages of osteoarthritis (OA) has potentially important applications. We aimed to develop and validate two models to estimate an individual’s risk of primary THR and TKR in patients newly presenting to primary care.

Methods

We identified two cohorts of patients aged ≥40 years newly consulting hip pain/OA and knee pain/OA in the Clinical Practice Research Datalink. Candidate predictors were identified by systematic review, novel hypothesis-free ‘Record-Wide Association Study’ with replication, and panel consensus. Cox proportional hazards models accounting for competing risk of death were applied to derive risk algorithms for THR and TKR. Internal–external cross-validation (IECV) was then applied over geographical regions to validate two models.

Results

45 predictors for THR and 53 for TKR were identified, reviewed and selected by the panel. 301 052 and 416 030 patients newly consulting between 1992 and 2015 were identified in the hip and knee cohorts, respectively (median follow-up 6 years). The resultant model C-statistics is 0.73 (0.72, 0.73) and 0.79 (0.78, 0.79) for THR (with 20 predictors) and TKR model (with 24 predictors), respectively. The IECV C-statistics ranged between 0.70–0.74 (THR model) and 0.76–0.82 (TKR model); the IECV calibration slope ranged between 0.93–1.07 (THR model) and 0.92–1.12 (TKR model).

Conclusions

Two prediction models with good discrimination and calibration that estimate individuals’ risk of THR and TKR have been developed and validated in large-scale, nationally representative data, and are readily automated in electronic patient records.

Factors affecting local regrowth after watch and wait for patients with a clinical complete response following chemoradiotherapy in rectal cancer (InterCoRe consortium): an individual participant data meta-analysis

BACKGROUND

In patients with rectal cancer who achieve clinical complete response after neoadjuvant chemoradiotherapy, watch and wait is a novel management strategy with potential to avoid major surgery. Study-level meta-analyses have reported wide variation in the proportion of patients with local regrowth. We did an individual participant data meta-analysis to investigate factors affecting occurrence of local regrowth.

METHODS

We updated search results of a recent systematic review by searching MEDLINE and Embase from Jan 1, 2016, to May 5, 2017, and used expert knowledge to identify published studies reporting on local regrowth in patients with rectal cancer managed by watch and wait after clinical complete response to neoadjuvant chemoradiotherapy. We restricted studies to those that defined clinical complete response using criteria equivalent to São Paulo benchmarks (ie, absence of residual ulceration, stenosis, or mass within the rectum on clinical and endoscopic examination). The primary outcome was 2-year cumulative incidence of local regrowth, estimated with a two-stage random-effects individual participant data meta-analysis. We assessed the effects of clinical and treatment factors using Cox frailty models, expressed as hazard ratios (HRs). From these models, we derived percentage differences in mean θ as an approximation of the effect of measured covariates on between-centre heterogeneity. This study is registered with PROSPERO, number CRD42017070934.

FINDINGS

We obtained individual participant data from 11 studies, including 602 patients enrolled between March 11, 1990, and Feb 13, 2017, with a median follow-up of 37·6 months (IQR 25·0-58·7). Ten of the 11 datasets were judged to be at low risk of bias. 2-year cumulative incidence of local regrowth was 21·4% (random-effects 95% CI 15·3-27·6), with high levels of between-study heterogeneity (I²=61%). We noted wide between-centre variation in patient, tumour, and treatment characteristics. We found some evidence that increasing cT stage was associated with increased risk of local regrowth (random-effects HR per cT stage 1·40, 95% CI 1·00-1·94; p_trend=0·048). In a subgroup of 459 patients managed after 2008 (when pretreatment staging by MRI became standard), 2-year cumulative incidence of local regrowth was 19% (95% CI 13-28) for stage cT1 and cT2 tumours, 31% (26-37) for cT3, and 37% (21-60) for cT4 (random-effects HR per cT stage 1·50, random-effects 95% CI 1·03-2·17; p_trend=0·0330). We estimated that measured factors contributed 4·8-45·3% of observed between-centre heterogeneity.

INTERPRETATION

In patients with rectal cancer and clinical complete response after chemoradiotherapy managed by watch and wait, we found some evidence that increasing cT stage predicts for local regrowth. These data will inform clinician-patient decision making in this setting. Research is needed to determine other predictors of a sustained clinical complete response.

FUNDING

None.

Measurement error and timing of predictor values for multivariable risk prediction models are poorly reported

OBJECTIVE

Measurement error in predictor variables may threaten the validity of clinical prediction models. We sought to evaluate the possible extent of the problem. A secondary objective was to examine whether predictors are measured at the intended moment of model use.

METHODS

A systematic search of Medline was used to identify a sample of articles reporting the development of a clinical prediction model published in 2015. After screening according to a predefined inclusion criteria, information on predictors, strategies to control for measurement error, and intended moment of model use were extracted. Susceptibility to measurement error for each predictor was classified into low and high risks.

RESULTS

Thirty-three studies were reviewed, including 151 different predictors in the final prediction models. Fifty-one (33.7%) predictors were categorized as high risk of error; however, this was not accounted for in the model development. Only 8 (24.2%) studies explicitly stated the intended moment of model use and when the predictors were measured.

CONCLUSION

Reporting of measurement error and intended moment of model use is poor in prediction model studies. There is a need to identify circumstances where ignoring measurement error in prediction models is consequential and whether accounting for the error will improve the predictions.

Persistent sex disparities in clinical outcomes with percutaneous coronary intervention: Insights from 6.6 million PCI procedures in the United States

BACKGROUND

Prior studies have reported inconsistencies in the baseline risk profile, comorbidity burden and their association with clinical outcomes in women compared to men. More importantly, there is limited data around the sex differences and how these have changed over time in contemporary percutaneous coronary intervention (PCI) practice.

METHODS AND RESULTS

We used the Nationwide Inpatient Sample to identify all PCI procedures based on ICD-9 procedure codes in the United States between 2004-2014 in adult patients. Descriptive statistics were used to describe sex-based differences in baseline characteristics and comorbidity burden of patients. Multivariable logistic regressions were used to investigate the association between these differences and in-hospital mortality, complications, length of stay and total hospital charges. Among 6,601,526 patients, 66% were men and 33% were women. Women were more likely to be admitted with diagnosis of NSTEMI (non-ST elevation acute myocardial infarction), were on average 5 years older (median age 68 compared to 63) and had higher burden of comorbidity defined by Charlson score ≥3. Women also had higher in-hospital crude mortality (2.0% vs 1.4%) and any complications compared to men (11.1% vs 7.0%). These trends persisted in our adjusted analyses where women had a significant increase in the odds of in-hospital mortality men (OR 1.20 (95% CI 1.16,1.23) and major bleeding (OR 1.81 (95% CI 1.77,1.86).

CONCLUSION

In this national unselected contemporary PCI cohort, there are significant sex-based differences in presentation, baseline characteristics and comorbidity burden. These differences do not fully account for the higher in-hospital mortality and procedural complications observed in women.

Protease activity as a prognostic factor for wound healing in venous leg ulcers

BACKGROUND

Venous leg ulcers (VLUs) are a common type of complex wound that have a negative impact on people's lives and incur high costs for health services and society. It has been suggested that prolonged high levels of protease activity in the later stages of the healing of chronic wounds may be associated with delayed healing. Protease modulating treatments have been developed which seek to modulate protease activity and thereby promote healing in chronic wounds.

OBJECTIVES

To determine whether protease activity is an independent prognostic factor for the healing of venous leg ulcers.

SEARCH METHODS

In February 2018, we searched the following databases: Cochrane Central Register of Controlled Trials (CENTRAL), Ovid MEDLINE, Ovid Embase and CINAHL.

SELECTION CRITERIA

We included prospective and retrospective longitudinal studies with any follow-up period that recruited people with VLUs and investigated whether protease activity in wound fluid was associated with future healing of VLUs. We included randomised controlled trials (RCTs) analysed as cohort studies, provided interventions were taken into account in the analysis, and case-control studies if there were no available cohort studies. We also included prediction model studies provided they reported separately associations of individual prognostic factors (protease activity) with healing. Studies of any type of protease or combination of proteases were eligible, including proteases from bacteria, and the prognostic factor could be examined as a continuous or categorical variable; any cut-off point was permitted. The primary outcomes were time to healing (survival analysis) and the proportion of people with ulcers completely healed; the secondary outcome was change in ulcer size/rate of wound closure. We extracted unadjusted (simple) and adjusted (multivariable) associations between the prognostic factor and healing.

DATA COLLECTION AND ANALYSIS

Two review authors independently assessed studies for inclusion at each stage, and undertook data extraction, assessment of risk of bias and GRADE assessment. We collected association statistics where available. No study reported adjusted analyses: instead we collected unadjusted results or calculated association measures from raw data. We calculated risk ratios when both outcome and prognostic factor were dichotomous variables. When the prognostic factor was reported as continuous data and healing outcomes were dichotomous, we either performed regression analysis or analysed the impact of healing on protease levels, analysing as the standardised mean difference. When both prognostic factor and outcome were continuous data, we reported correlation coefficients or calculated them from individual participant data.We displayed all results on forest plots to give an overall visual representation. We planned to conduct meta-analyses where this was appropriate, otherwise we summarised narratively.

MAIN RESULTS

We included 19 studies comprising 21 cohorts involving 646 participants. Only 11 studies (13 cohorts, 522 participants) had data available for analysis. Of these, five were prospective cohort studies, four were RCTs and two had a type of case-control design. Follow-up time ranged from four to 36 weeks. Studies covered 10 different matrix metalloproteases (MMPs) and two serine proteases (human neutrophil elastase and urokinase-type plasminogen activators). Two studies recorded complete healing as an outcome; other studies recorded partial healing measures. There was clinical and methodological heterogeneity across studies; for example, in the definition of healing, the type of protease and its measurement, the distribution of active and bound protease species, the types of treatment and the reporting of results. Therefore, meta-analysis was not performed. No study had conducted multivariable analyses and all included evidence was of very low certainty because of the lack of adjustment for confounders, the high risk of bias for all studies except one, imprecision around the measures of association and inconsistency in the direction of association. Collectively the research indicated complete uncertainty as to the association between protease activity and VLU healing.

AUTHORS' CONCLUSIONS

This review identified very low validity evidence regarding any association between protease activity and VLU healing and there is complete uncertainty regarding the relationship. The review offers information for both future research and systematic review methodology.

Individual participant data meta-analysis of continuous outcomes: A comparison of approaches for specifying and estimating one-stage models

One‐stage individual participant data meta‐analysis models should account for within‐trial clustering, but it is currently debated how to do this. For continuous outcomes modeled using a linear regression framework, two competing approaches are a stratified intercept or a random intercept. The stratified approach involves estimating a separate intercept term for each trial, whereas the random intercept approach assumes that trial intercepts are drawn from a normal distribution. Here, through an extensive simulation study for continuous outcomes, we evaluate the impact of using the stratified and random intercept approaches on statistical properties of the summary treatment effect estimate. Further aims are to compare (i) competing estimation options for the one‐stage models, including maximum likelihood and restricted maximum likelihood, and (ii) competing options for deriving confidence intervals (CI) for the summary treatment effect, including the standard normal‐based 95% CI, and more conservative approaches of Kenward‐Roger and Satterthwaite, which inflate CIs to account for uncertainty in variance estimates. The findings reveal that, for an individual participant data meta‐analysis of randomized trials with a 1:1 treatment:control allocation ratio and heterogeneity in the treatment effect, (i) bias and coverage of the summary treatment effect estimate are very similar when using stratified or random intercept models with restricted maximum likelihood, and thus either approach could be taken in practice, (ii) CIs are generally best derived using either a Kenward‐Roger or Satterthwaite correction, although occasionally overly conservative, and (iii) if maximum likelihood is required, a random intercept performs better than a stratified intercept model. An illustrative example is provided.

Effects of antenatal diet and physical activity on maternal and fetal outcomes: individual patient data meta-analysis and health economic evaluation

BACKGROUND

Diet- and physical activity-based interventions in pregnancy have the potential to alter maternal and child outcomes.

OBJECTIVES

To assess whether or not the effects of diet and lifestyle interventions vary in subgroups of women, based on maternal body mass index (BMI), age, parity, Caucasian ethnicity and underlying medical condition(s), by undertaking an individual patient data (IPD) meta-analysis. We also evaluated the association of gestational weight gain (GWG) with adverse pregnancy outcomes and assessed the cost-effectiveness of the interventions.

DATA SOURCES

MEDLINE, EMBASE, Cochrane Central Register of Controlled Trials, Database of Abstracts of Reviews of Effects and Health Technology Assessment database were searched from October 2013 to March 2015 (to update a previous search).

REVIEW METHODS

Researchers from the International Weight Management in Pregnancy Collaborative Network shared the primary data. For each intervention type and outcome, we performed a two-step IPD random-effects meta-analysis, for all women (except underweight) combined and for each subgroup of interest, to obtain summary estimates of effects and 95% confidence intervals (CIs), and synthesised the differences in effects between subgroups. In the first stage, we fitted a linear regression adjusted for baseline (for continuous outcomes) or a logistic regression model (for binary outcomes) in each study separately; estimates were combined across studies using random-effects meta-analysis models. We quantified the relationship between weight gain and complications, and undertook a decision-analytic model-based economic evaluation to assess the cost-effectiveness of the interventions.

RESULTS

Diet and lifestyle interventions reduced GWG by an average of 0.70 kg (95% CI -0.92 to -0.48 kg; 33 studies, 9320 women). The effects on composite maternal outcome [summary odds ratio (OR) 0.90, 95% CI 0.79 to 1.03; 24 studies, 8852 women] and composite fetal/neonatal outcome (summary OR 0.94, 95% CI 0.83 to 1.08; 18 studies, 7981 women) were not significant. The effect did not vary with baseline BMI, age, ethnicity, parity or underlying medical conditions for GWG, and composite maternal and fetal outcomes. Lifestyle interventions reduce Caesarean sections (OR 0.91, 95% CI 0.83 to 0.99), but not other individual maternal outcomes such as gestational diabetes mellitus (OR 0.89, 95% CI 0.72 to 1.10), pre-eclampsia or pregnancy-induced hypertension (OR 0.95, 95% CI 0.78 to 1.16) and preterm birth (OR 0.94, 95% CI 0.78 to 1.13). There was no significant effect on fetal outcomes. The interventions were not cost-effective. GWG, including adherence to the Institute of Medicine-recommended targets, was not associated with a reduction in complications. Predictors of GWG were maternal age (summary estimate -0.10 kg, 95% CI -0.14 to -0.06 kg) and multiparity (summary estimate -0.73 kg, 95% CI -1.24 to -0.23 kg).

LIMITATIONS

The findings were limited by the lack of standardisation in the components of intervention, residual heterogeneity in effects across studies for most analyses and the unavailability of IPD in some studies.

CONCLUSION

Diet and lifestyle interventions in pregnancy are clinically effective in reducing GWG irrespective of risk factors, with no effects on composite maternal and fetal outcomes.

FUTURE WORK

The differential effects of lifestyle interventions on individual pregnancy outcomes need evaluation.

STUDY REGISTRATION

This study is registered as PROSPERO CRD42013003804.

FUNDING

The National Institute for Health Research Health Technology Assessment programme.

Simulation-based power calculations for planning a two-stage individual participant data meta-analysis

Background

Researchers and funders should consider the statistical power of planned Individual Participant Data (IPD) meta-analysis projects, as they are often time-consuming and costly. We propose simulation-based power calculations utilising a two-stage framework, and illustrate the approach for a planned IPD meta-analysis of randomised trials with continuous outcomes where the aim is to identify treatment-covariate interactions.

Methods

The simulation approach has four steps: (i) specify an underlying (data generating) statistical model for trials in the IPD meta-analysis; (ii) use readily available information (e.g. from publications) and prior knowledge (e.g. number of studies promising IPD) to specify model parameter values (e.g. control group mean, intervention effect, treatment-covariate interaction); (iii) simulate an IPD meta-analysis dataset of a particular size from the model, and apply a two-stage IPD meta-analysis to obtain the summary estimate of interest (e.g. interaction effect) and its associated p-value; (iv) repeat the previous step (e.g. thousands of times), then estimate the power to detect a genuine effect by the proportion of summary estimates with a significant p-value.

Results

In a planned IPD meta-analysis of lifestyle interventions to reduce weight gain in pregnancy, 14 trials (1183 patients) promised their IPD to examine a treatment-BMI interaction (i.e. whether baseline BMI modifies intervention effect on weight gain). Using our simulation-based approach, a two-stage IPD meta-analysis has < 60% power to detect a reduction of 1 kg weight gain for a 10-unit increase in BMI. Additional IPD from ten other published trials (containing 1761 patients) would improve power to over 80%, but only if a fixed-effect meta-analysis was appropriate. Pre-specified adjustment for prognostic factors would increase power further. Incorrect dichotomisation of BMI would reduce power by over 20%, similar to immediately throwing away IPD from ten trials.

Conclusions

Simulation-based power calculations could inform the planning and funding of IPD projects, and should be used routinely.

Electronic supplementary material

The online version of this article (10.1186/s12874-018-0492-z) contains supplementary material, which is available to authorized users.

Study protocol: quantitative fibronectin to help decision-making in women with symptoms of preterm labour (QUIDS) part 2, UK Prospective Cohort Study

INTRODUCTION

The aim of the QUIDS study is to develop a decision support tool for the management of women with symptoms and signs of preterm labour, based on a validated prognostic model using quantitative fetal fibronectin (fFN) concentration, in combination with clinical risk factors.

METHODS AND ANALYSIS

The study will evaluate the Rapid fFN 10Q System (Hologic, Marlborough, Massachusetts, USA) which quantifies fFN in a vaginal swab. In QUIDS part 2, we will perform a prospective cohort study in at least eight UK consultant-led maternity units, in women with symptoms of preterm labour at 22⁺⁰ to 34⁺⁶ weeks gestation to externally validate a prognostic model developed in QUIDS part 1. The effects of quantitative fFN on anxiety will be assessed, and acceptability of the test and prognostic model will be evaluated in a subgroup of women and clinicians (n=30). The sample size is 1600 women (with estimated 96-192 events of preterm delivery within 7 days of testing). Clinicians will be informed of the qualitative fFN result (positive/negative) but be blinded to quantitative fFN result. Research midwives will collect outcome data from the maternal and neonatal clinical records. The final validated prognostic model will be presented as a mobile or web-based application.

ETHICS AND DISSEMINATION

The study is funded by the National Institute of Healthcare Research Health Technology Assessment (HTA 14/32/01). It has been approved by the West of Scotland Research Ethics Committee (16/WS/0068).

VERSION

Protocol V.2, Date 1 November 2016.

TRIAL REGISTRATION NUMBER

ISRCTN 41598423andCPMS: 31277.

Quantitative fibronectin to help decision-making in women with symptoms of preterm labour (QUIDS) part 1: Individual participant data meta-analysis and health economic analysis

INTRODUCTION

The aim of the QUIDS study is to develop a decision support tool for the management of women with symptoms and signs of preterm labour, based on a validated prognostic model using quantitative fetal fibronectin (qfFN) concentration, in combination with clinical risk factors.

METHODS AND ANALYSIS

The study will evaluate the Rapid fFN 10Q System (Hologic, Marlborough, Massachusetts) which quantifies fFN in a vaginal swab. In part 1 of the study, we will develop and internally validate a prognostic model using an individual participant data (IPD) meta-analysis of existing studies containing women with symptoms of preterm labour alongside fFN measurements and pregnancy outcome. An economic analysis will be undertaken to assess potential cost-effectiveness of the qfFN prognostic model. The primary endpoint will be the ability of the prognostic model to rule out spontaneous preterm birth within 7 days. Six eligible studies were identified by systematic review of the literature and five agreed to provide their IPD (n=5 studies, 1783 women and 139 events of preterm delivery within 7 days of testing).

ETHICS AND DISSEMINATION

The study is funded by the National Institute of Healthcare Research Health Technology Assessment (HTA 14/32/01). It has been approved by the West of Scotland Research Ethics Committee (16/WS/0068).

PROSPERO REGISTRATION NUMBER

CRD42015027590.

VERSION

Protocol version 2, date 1 November 2016.

Random-effects meta-analysis of the clinical utility of tests and prediction models

The use of data from multiple studies or centers for the validation of a clinical test or a multivariable prediction model allows researchers to investigate the test's/model's performance in multiple settings and populations. Recently, meta-analytic techniques have been proposed to summarize discrimination and calibration across study populations. Here, we rather consider performance in terms of net benefit, which is a measure of clinical utility that weighs the benefits of true positive classifications against the harms of false positives. We posit that it is important to examine clinical utility across multiple settings of interest. This requires a suitable meta-analysis method, and we propose a Bayesian trivariate random-effects meta-analysis of sensitivity, specificity, and prevalence. Across a range of chosen harm-to-benefit ratios, this provides a summary measure of net benefit, a prediction interval, and an estimate of the probability that the test/model is clinically useful in a new setting. In addition, the prediction interval and probability of usefulness can be calculated conditional on the known prevalence in a new setting. The proposed methods are illustrated by 2 case studies: one on the meta-analysis of published studies on ear thermometry to diagnose fever in children and one on the validation of a multivariable clinical risk prediction model for the diagnosis of ovarian cancer in a multicenter dataset. Crucially, in both case studies the clinical utility of the test/model was heterogeneous across settings, limiting its usefulness in practice. This emphasizes that heterogeneity in clinical utility should be assessed before a test/model is routinely implemented.

The role of secondary outcomes in multivariate meta-analysis

Univariate meta‐analysis concerns a single outcome of interest measured across a number of independent studies. However, many research studies will have also measured secondary outcomes. Multivariate meta‐analysis allows us to take these secondary outcomes into account and can also include studies where the primary outcome is missing. We define the efficiency E as the variance of the overall estimate from a multivariate meta‐analysis relative to the variance of the overall estimate from a univariate meta‐analysis. The extra information gained from a multivariate meta‐analysis of n studies is then similar to the extra information gained if a univariate meta‐analysis of the primary effect had a further n(1−E)/E studies. The variance contribution of a study's secondary outcomes (its borrowing of strength) can be thought of as a contrast between the variance matrix of the outcomes in that study and the set of variance matrices of all the studies in the meta‐analysis. In the bivariate case this is given a simple graphical interpretation as the borrowing‐of‐strength plot. We discuss how these findings can also be used in the context of random‐effects meta‐analysis. Our discussion is motivated by a published meta‐analysis of 10 antihypertension clinical trials.

Does vitamin D supplementation improve bone density in vitamin D-deficient children? Protocol for an individual patient data meta-analysis

INTRODUCTION

Our previous study-level (aggregate data) meta-analysis suggested that vitamin D supplements may be beneficial for bone density specifically in children with vitamin D deficiency. However, the misclassification of vitamin D status inherent in study-level data means that the results are not definitive and cannot provide an accurate assessment of the size of any effect. Therefore, we propose to undertake an individual patient data (IPD) meta-analysis to determine whether the effect of vitamin D supplementation on bone density in children differs according to baseline vitamin D status, and to specifically estimate the effect of vitamin D in children who are vitamin D deficient.

METHODS AND ANALYSIS

This study has been designed to adhere to the Preferred Reporting Items for Systematic Review and Meta-Analyses of IPD statement. We will include randomised placebo-controlled trials of vitamin D supplementation reporting bone density outcomes at least 6 months after the study commenced in children and adolescents (aged <20 years) without coexistent medical conditions or treatments causing osteoporosis. We will update the search of the original review to cover the period 2009-2017, using the same methods as the original review. Fully anonymised data on all randomised patients will be requested. Outcomes will be femoral neck, total hip, lumbar spine and proximal and distal forearm bone mineral density, and total body bone mineral content. A two-stage IPD meta-analysis will be used to examine the effect of baseline serum 25-hydroxyvitamin D (25(OH)D) on treatment effect for each bone density outcome. Restricted maximum likelihood will be used to estimate the random-effects meta-analysis models, with 95% CI for summary effects. Heterogeneity will be assessed by I² and potential publication bias (small-study effects) and availability bias by funnel plots, Egger's test and Peter's test.

ETHICS AND DISSEMINATION

Ethics approval will not be required as the data are to be used for the primary purpose for which they were collected and all original individual studies had ethics approval. Results of the IPD meta-analysis will be submitted for publication in a peer-reviewed journal.

PROSPERO REGISTRATION NUMBER

CRD42017068772.

Subgrouping and TargetEd Exercise pRogrammes for knee and hip OsteoArthritis (STEER OA): a systematic review update and individual participant data meta-analysis protocol

INTRODUCTION

Knee and hip osteoarthritis (OA) is a leading cause of disability worldwide. Therapeutic exercise is a recommended core treatment for people with knee and hip OA, however, the observed effect sizes for reducing pain and improving physical function are small to moderate. This may be due to insufficient targeting of exercise to subgroups of people who are most likely to respond and/or suboptimal content of exercise programmes. This study aims to identify: (1) subgroups of people with knee and hip OA that do/do not respond to therapeutic exercise and to different types of exercise and (2) mediators of the effect of therapeutic exercise for reducing pain and improving physical function. This will enable optimal targeting and refining the content of future exercise interventions.

METHODS AND ANALYSIS

Systematic review and individual participant data meta-analyses. A previous comprehensive systematic review will be updated to identify randomised controlled trials that compare the effects of therapeutic exercise for people with knee and hip OA on pain and physical function to a non-exercise control. Lead authors of eligible trials will be invited to share individual participant data. Trial-level and participant-level characteristics (for baseline variables and outcomes) of included studies will be summarised. Meta-analyses will use a two-stage approach, where effect estimates are obtained for each trial and then synthesised using a random effects model (to account for heterogeneity). All analyses will be on an intention-to-treat principle and all summary meta-analysis estimates will be reported as standardised mean differences with 95% CI.

ETHICS AND DISSEMINATION

Research ethical or governance approval is exempt as no new data are being collected and no identifiable participant information will be shared. Findings will be disseminated via national and international conferences, publication in peer-reviewed journals and summaries posted on websites accessed by the public and clinicians.

PROSPERO REGISTRATION NUMBER

CRD42017054049.