Inclusion of nonrandomized studies of interventions in systematic reviews of interventions: updated guidance from the Agency for Health Care Research and Quality Effective Health Care program

Quality Assessment of Cohort Studies in Complementary and Alternative Medicine: A Scoping Review Over Two Decades

Background

We aimed to investigate the subject matters and the quality of publications detailing the findings of cohort studies within the realm of complementary and alternative medicine (CAM).

Methods

A scoping review was conducted on cohort studies in the CAM field up to the conclusion of 2023. The evaluation of their quality was carried out utilizing the 'Strengthening the Reporting of Observational Studies in Epidemiology' (STROBE) checklist. Moreover, an analysis of their research settings and associated variables, including publication year, type of disease, intervention method, and study field, was conducted.

Results

Overall, 215 articles were identified. The majority of these cohorts, approximately 42.3%, originated from Taiwan, with stroke and cardiovascular diseases emerging as the most prevalent outcomes of interest. The mean STROBE score was 1.38 (SD=0.57) out of 2. The lowest scores were associated with the methods and funding sections. Methodologically, the principal weaknesses were linked to sample size, loss to follow-up, and bias control.

Conclusion

The frequency of cohort studies in CAM was limited, predominantly concentrated in a few countries. Chinese medicine and acupuncture were the main intervention methods, while other CAM interventions received less focus. Furthermore, the quality of these studies was deemed unsatisfactory in most cases.

Integration of non-randomized studies with randomized controlled trials in meta-analyses of clinical studies: a meta-epidemiological study on effect estimation of interventions

BACKGROUNDS

Syntheses of non-randomized studies of interventions (NRSIs) and randomized controlled trials (RCTs) are increasingly used in decision-making. This study aimed to summarize when NRSIs are included in evidence syntheses of RCTs, with a particular focus on the methodological issues associated with combining NRSIs and RCTs.

METHODS

We searched PubMed to identify clinical systematic reviews published between 9 December 2017 and 9 December 2022, randomly sampling reviews in a 1:1 ratio of Core and non-Core clinical journals. We included systematic reviews with RCTs and NRSIs for the same clinical question. Clinical scenarios for considering the inclusion of NRSIs in eligible studies were classified. We extracted the methodological characteristics of the included studies, assessed the concordance of estimates between RCTs and NRSIs, calculated the ratio of the relative effect estimate from NRSIs to that from RCTs, and evaluated the impact on the estimates of pooled estimates when NRSIs are included.

RESULTS

Two hundred twenty systematic reviews were included in the analysis. The clinical scenarios for including NRSIs were grouped into four main justifications: adverse outcomes (n = 140, 63.6%), long-term outcomes (n = 36, 16.4%), the applicability of RCT results to broader populations (n = 11, 5.0%), and other (n = 33, 15.0%). When conducting a meta-analysis, none of these reviews assessed the compatibility of the different types of evidence prior, 203 (92.3%) combined estimates from RCTs and NRSIs in the same meta-analysis. Of the 203 studies, 169 (76.8%) used crude estimates of NRSIs, and 28 (13.8%) combined RCTs and multiple types of NRSIs. Seventy-seven studies (35.5%) showed "qualitative disagree" between estimates from RCTs and NRSIs, and 101 studies (46.5%) found "important difference". The integration of NRSIs changed the qualitative direction of estimates from RCTs in 72 out of 200 studies (36.0%).

CONCLUSIONS

Systematic reviews typically include NRSIs in the context of assessing adverse or long-term outcomes. The inclusion of NRSIs in a meta-analysis of RCTs has a substantial impact on effect estimates, but discrepancies between RCTs and NRSIs are often ignored. Our proposed recommendations will help researchers to consider carefully when and how to synthesis evidence from RCTs and NRSIs.

Development of a Canadian Guidance for reporting real-world evidence for regulatory and health-technology assessment (HTA) decision-making

BACKGROUND AND OBJECTIVE

Real-world evidence (RWE) can complement and fill knowledge gaps from randomized controlled trials to assist in health-technology assessment (HTA) for regulatory decision-making. However, the generation of RWE is an intricate process with many sequential decision points, and different methods and approaches may impact the quality and reliability of evidence. Standardization and transparency in reporting these decisions is imperative to appraise RWE and incorporate it into HTA decision-making. A partnership between Canadian health system stakeholders, namely, Health Canada and Canada's Drug Agency (formerly the Canadian Agency for Drugs and Technologies in Health), was established to develop guidance for the standardization of reporting of RWE for regulatory and HTA decision-making in Canada.

STUDY DESIGN AND SETTING

A collaborative initiative to create structured guidance for RWE reporting in the context of regulatory and HTA decision-making.

RESULTS

The developed guidance aims to standardize and ensure transparent reporting of RWE to improve its reliability and usefulness in regulatory and HTA processes.

CONCLUSION

This guidance can be adapted for other jurisdictions and will have future extensions to incorporate emerging issues with RWE and HTA decision-making.

The reporting quality of meta-epidemiological studies needs substantial improvement: a research on research study

BACKGROUND

Meta-epidemiological research plays a vital role in providing empirical evidence needed to develop methodological manuals and tools, but the reporting quality has not been comprehensively assessed, and the influence of reporting guidelines remains unclear. The current study aims to evaluate the reporting quality of meta-epidemiological studies, assess the impact of reporting guidelines, and identify factors influencing reporting quality.

METHODS

We searched PubMed and Embase for meta-epidemiological studies. The reporting quality of these studies was assessed for adherence to established reporting guidelines. Two researchers independently screened the studies and assessed the quality of the included studies. Time-series segmented linear regression was used to evaluate changes in reporting quality over time, while beta-regression analysis was performed to identify factors significantly associated with reporting quality.

RESULTS

We initially identified 1720 articles, of which 125 meta-epidemiological studies met the inclusion criteria. Of these, 65 (52%) had low reporting quality, 60 (48%) had moderate quality, and none achieved high quality. Of the 24 items derived from established reporting guidelines, 4 had poor adherence, 13 had moderate adherence, and 7 had high adherences. High journal impact factor (≥ 10) (OR = 1.42, 95% CI: 1.13, 1.80; P = 0.003) and protocol registration (OR = 1.70, 95% CI: 1.30, 2.22; P < 0.001) were significantly associated with better reporting quality. The publication of the reporting guideline did not significantly increase the mean reporting quality score (- 0.53, 95% CI: - 3.37, 2.31; P = 0.67) or the trend (- 0.38, 95% CI: - 1.02, 0.26; P = 0.20).

CONCLUSIONS

Our analysis showed suboptimal reporting quality in meta-epidemiological studies, with no improvement post-2017 guidelines. This potential shortcoming could hinder stakeholders' ability to draw reliable conclusions from these studies. While preregistration could reduce reporting bias, its adoption remains low. Registration platforms could consider creating tailored types for meta-epidemiological research, and journals need to adopt more proactive measures to enforce reporting standards.

Integrating randomized controlled trials and non-randomized studies of interventions to assess the effect of rare events: a Bayesian re-analysis of two meta-analyses

BACKGROUND

There is a growing trend to include non-randomised studies of interventions (NRSIs) in rare events meta-analyses of randomised controlled trials (RCTs) to complement the evidence from the latter. An important consideration when combining RCTs and NRSIs is how to address potential bias and down-weighting of NRSIs in the pooled estimates. The aim of this study is to explore the use of a power prior approach in a Bayesian framework for integrating RCTs and NRSIs to assess the effect of rare events.

METHODS

We proposed a method of specifying the down-weighting factor based on judgments of the relative magnitude (no information, and low, moderate, serious and critical risk of bias) of the overall risk of bias for each NRSI using the ROBINS-I tool. The methods were illustrated using two meta-analyses, with particular interest in the risk of diabetic ketoacidosis (DKA) in patients using sodium/glucose cotransporter-2 (SGLT-2) inhibitors compared with active comparators, and the association between low-dose methotrexate exposure and melanoma.

RESULTS

No significant results were observed for these two analyses when the data from RCTs only were pooled (risk of DKA: OR = 0.82, 95% confidence interval (CI): 0.25-2.69; risk of melanoma: OR = 1.94, 95%CI: 0.72-5.27). When RCTs and NRSIs were directly combined without distinction in the same meta-analysis, both meta-analyses showed significant results (risk of DKA: OR = 1.50, 95%CI: 1.11-2.03; risk of melanoma: OR = 1.16, 95%CI: 1.08-1.24). Using Bayesian analysis to account for NRSI bias, there was a 90% probability of an increased risk of DKA in users receiving SGLT-2 inhibitors and an 91% probability of an increased risk of melanoma in patients using low-dose methotrexate.

CONCLUSIONS

Our study showed that including NRSIs in a meta-analysis of RCTs for rare events could increase the certainty and comprehensiveness of the evidence. The estimates obtained from NRSIs are generally considered to be biased, and the possible influence of NRSIs on the certainty of the combined evidence needs to be carefully investigated.

Non-pharmacological interventions targeting mobility among people with advanced cancer: a systematic review

PURPOSE

To synthesise evidence evaluating non-pharmacological interventions targeting mobility among people with advanced cancer, considering the type, efficacy and contextual factors that may influence outcome.

METHODS

Systematic review of studies of non-pharmacological interventions in adults (≥ 18 years) with advanced (stage III-IV) cancer, and assessing mobility using clinical or patient-reported outcome measures. Searches were conducted across three electronic databases (MEDLINE, EMBASE and CINAHL) up to June 2024. Methodological quality was assessed using Joanna Briggs Institute tools and contextual factors were evaluated through the Context and Implementation of Complex Interventions framework. A narrative synthesis was conducted due to clinical heterogeneity of included studies.

RESULTS

38 studies encompassing 2,464 participants were included. The most frequent mobility outcome measure was the 6-min walk test (26/38 studies). Exercise was the most common intervention, (33 studies: 27 aerobic and resistance, 5 aerobic, 1 resistance versus aerobic training) and improvements in mobility were found in 21/33 outcomes. Electrotherapy interventions led to significant improvements in mobility in 3/5 studies. Geographical factors (e.g. distance, transport, parking requirements) potentially limited participation in 18/38 studies. A lack of ethnic diversity among populations was evident and language proficiency was an inclusion criterion in 12 studies.

CONCLUSION

Exercise and neuromuscular electrical stimulation appear to improve mobility outcomes in advanced cancer. The evaluation of other non-pharmacological interventions targeting mobility should consider access and inclusivity, and be adaptable to the needs of this population.

Impact of Cognitive Behavior Therapy on Sexual Dysfunction of Women in Reproductive Age: A Systematic Review

Background

A large number of women suffer from sexual dysfunction. Cognitive behavior therapy (CBT) is a psycho-educational therapy the main purpose of which is to help the individual alter their dysfunctional beliefs and behaviors. We investigated the impact of CBT on reproductive-aged women's sexual function.

Methods

The present study is a systematic review and meta-analysis in which a number of databases (Web of Science, PubMed, Scopus, Cochrane Library, and Embase) were searched from inception to November 2023. Clinical trials examining the impact of CBT on female sexual dysfunction were included. This study evaluated female sexual function and its components, including satisfaction, discomfort, lubrication, arousal, climax, desire, and assertiveness. We used Review Manager version 5.3 for performing the meta-analysis. P values less than 0.05 were considered statistically significant.

Results

Seven articles including a total of 448 women with sexual dysfunction were reviewed. The participants' mean age in the CBT groups was 33.43 ± 6.04 years and that of the control group was 33.24 ± 5.97, which were not significantly different. Our results indicated that CBT is efficient in improving the total score of women's sexual function (SMD = 1.34, 95% CI = (1.02, 1.65), I2 = 86%), sexual satisfaction, lubrication, desire, orgasm, and assertiveness compared with the control group. However, CBT had no positive effect on reducing sexual pain.

Conclusion

Based on the result of the current study, CBT can be suggested as an effective treatment method along with other treatments for women with sexual problems.

Evaluating agreement between evidence from randomised controlled trials and cohort studies in nutrition: a meta-research replication study

This meta-research study aims to evaluate the agreement of effect estimates between bodies of evidence (BoE) from RCTs and cohort studies included in the same nutrition evidence synthesis, to identify factors associated with disagreement, and to replicate the findings of a previous study. We searched Medline, Epistemonikos and the Cochrane Database of Systematic Reviews for nutrition systematic reviews that included both RCTs and cohort studies for the same patient-relevant outcome or intermediate-disease marker. We rated similarity of PI/ECO (population, intervention/exposure, comparison, outcome) between BoE from RCTs and cohort studies. Agreement of effect estimates across BoE was analysed by pooling ratio of risk ratios (RRR) for binary outcomes and difference of standardised mean differences (DSMD) for continuous outcomes. We performed subgroup and sensitivity analyses to explore determinants associated with disagreements. We included 82 BoE-pairs from 51 systematic reviews. For binary outcomes, the RRR was 1.04 (95% confidence interval (CI) 0.99 to 1.10, I2 = 59%, τ2 = 0.02, prediction interval (PI) 0.77 to 1.41). For continuous outcomes, the pooled DSMD was  - 0.09 (95% CI  - 0.26 to 0.09, PI  - 0.55 to 0.38). Subgroup analyses yielded that differences in type of intake/exposure were drivers towards disagreement. We replicated the findings of a previous study, where on average RCTs and cohort studies had similar effect estimates. Disagreement and wide prediction intervals were mainly driven by PI/ECO-dissimilarities. More research is needed to explore other potentially influencing factors (e.g. risk of bias) on the disagreement between effect estimates of both BoE.Trial registration: CRD42021278908.

Comparison of statistical methods for integrating real-world evidence in a rare events meta-analysis of randomized controlled trials

Rare events meta-analyses of randomized controlled trials (RCTs) are often underpowered because the outcomes are infrequent. Real-world evidence (RWE) from non-randomized studies may provide valuable complementary evidence about the effects of rare events, and there is growing interest in including such evidence in the decision-making process. Several methods for combining RCTs and RWE studies have been proposed, but the comparative performance of these methods is not well understood. We describe a simulation study that aims to evaluate an array of alternative Bayesian methods for including RWE in rare events meta-analysis of RCTs: the naïve data synthesis, the design-adjusted synthesis, the use of RWE as prior information, the three-level hierarchical models, and the bias-corrected meta-analysis model. The percentage bias, root-mean-square-error, mean 95% credible interval width, coverage probability, and power are used to measure performance. The various methods are illustrated using a systematic review to evaluate the risk of diabetic ketoacidosis among patients using sodium/glucose co-transporter 2 inhibitors as compared with active-comparators. Our simulations show that the bias-corrected meta-analysis model is comparable to or better than the other methods in terms of all evaluated performance measures and simulation scenarios. Our results also demonstrate that data solely from RCTs may not be sufficiently reliable for assessing the effects of rare events. In summary, the inclusion of RWE could increase the certainty and comprehensiveness of the body of evidence of rare events from RCTs, and the bias-corrected meta-analysis model may be preferable.

The effect of postpartum lifestyle interventions on blood pressure: a systematic literature review

Postpartum lifestyle modification is recommended to hypertension risk. We conducted a systematic literature review to assess the evidence for postpartum lifestyle interventions to reduce blood pressure. We searched for relevant publications from 2010 through November 2022. Two authors independently conducted article screening and data extraction; a third resolved discrepancies. Ultimately, nine studies met inclusion criteria. Most were randomized controlled trials and had sample sizes <100. In all but one of the eight studies reporting race data, nearly all participants identified as White. None of the studies reported a significant intervention effect on blood pressure. However, most interventions were associated with improvements in other outcomes, such as physical activity. Overall, the evidence for postpartum lifestyle interventions to reduce blood pressure is limited to a handful of studies characterized by small sample sizes and a lack of racial diversity. Additional research with larger samples, more diverse populations, and intermediate outcomes is warranted.

Evaluating the impact of including non-randomised studies of interventions in meta-analysis of randomised controlled trials: a protocol for a meta-epidemiological study

INTRODUCTION

Although interest in including non-randomised studies of interventions (NRSIs) in meta-analysis of randomised controlled trials (RCTs) is growing, estimates of effectiveness obtained from NRSIs are vulnerable to greater bias than RCTs. The objectives of this study are to: (1) explore how NRSIs can be integrated into a meta-analysis of RCTs; (2) assess concordance of the evidence from non-randomised and randomised trials and explore factors associated with agreement; and (3) investigate the impact on estimates of pooled bodies of evidence when NRSIs are included.

METHODS AND ANALYSIS

We will conduct a systematic survey of 210 systematic reviews that include both RCTs and NRSIs, published from 2017 to 2022. We will randomly select reviews, stratified in a 1:1 ratio by Core vs non-Core clinical journals, as defined by the National Library of Medicine. Teams of paired reviewers will independently determine eligibility and abstract data using standardised, pilot-tested forms. The concordance of the evidence will be assessed by exploring agreement in the relative effect reported by NRSIs and RCT addressing the same clinical question, defined as similarity of the population, intervention/exposure, control and outcomes. We will conduct univariable and multivariable logistic regression analyses to examine the association of prespecified study characteristics with agreement in the estimates between NRSIs and RCTs. We will calculate the ratio of the relative effect estimate from NRSIs over that from RCTs, along with the corresponding 95% CI. We will use a bias-corrected meta-analysis model to investigate the influence on pooled estimates when NRSIs are included in the evidence synthesis.

ETHICS AND DISSEMINATION

Ethics approval is not required. The findings of this study will be disseminated through peer-reviewed publications, conference presentations and condensed summaries for clinicians, health policymakers and guideline developers regarding the design, conduct, analysis, and interpretation of meta-analysis that integrate RCTs and NRSIs.