Conducting separate reviews of benefits and harms could improve systematic reviews and meta-analyses

Guidance for systematic reviews of interventions recommends both benefits and harms be included. Systematic reviews may reach conclusions about harms (or lack of harms) that are not true when reviews include only some relevant studies, rely on incomplete data from eligible studies, use inappropriate methods for synthesizing data, and report results selectively. Separate reviews about harms could address some of these problems, and we argue that conducting separate reviews of harms is a feasible alternative to current standards and practices. Systematic reviews of potential benefits could be organized around the use of interventions for specific health problems. Systematic reviews of potential harms could be broader, including more diverse study designs and including all people at risk of harms (who might use the same intervention to treat different health problems). Multiple reviews about benefits could refer to a single review of harms. This approach could improve the reliability, completeness, and efficiency of systematic reviews.

Biases in reporting of adverse effects in clinical trials, and potential impact on safety assessments in systematic reviews and therapy guidelines

BACKGROUND

Clinical trials are an important source of adverse effects data, including analyses in systematic reviews and recommendations in therapy guidelines. Trial publication bias may have profound effects on safety perceptions. This MiniReview presents and discusses biases in reporting of safety data in clinical trials and the implications for systematic reviews and guidelines.

OBJECTIVES

The objectives of this work are to analyse risk of gastrointestinal bleeding in systemic corticosteroid trials and to assess adverse effects reporting in a fluoxetine trial in depression (Treatment for Adolescents With Depression Study [TADS]) and descriptions of adverse effects in adolescent depression therapy guidelines.

METHODS

We performed literature reviews and descriptive analyse of clinical trials with corticosteroids, and publications from the TADS trial. Risk of gastrointestinal bleeding from corticosteroids was analysed by meta-analysis.

FINDINGS

Gastrointestinal bleeding definitions varied considerably between trials. The incidence was significantly increased in hospitalized, but not in ambulant, patients compared to placebo. We identified several biases concerning TADS safety reporting, including severity thresholds and nonpublication of most adverse effects data beyond the initial 12 weeks. Therapy guidelines on adolescent depression mentioned suicidality risk, but many failed to mention other adverse effects.

CONCLUSIONS

We identified several pitfalls in adverse effects reporting in clinical trials. These include heterogeneous disease definitions, reporting thresholds, and incomplete reporting. Trial bias may have great impact on risk assessments in systematic reviews and meta-analyses.

Assessing the quality of meta-analyses in systematic reviews in pharmaceutical research in Iran by 2016: A systematic review

Background: Meta-analyses, like all other studies, may be poorly designed and implemented. This study was designed to determine the quality of meta-analyses in systematic reviews in the field of pharmaceutical research in Iran. Methods: Web of Science Core Collection, EMBASE, Ovid Medline, CINAHL, Scopus, and PubMed were systematically searched on June 4, 2017. The search was limited to the researches in the field of pharmaceutical studies. Based on inclusion criteria, 104 systematic reviews with meta-analysis (SRMA) were selected and assessed using quality assessment tools introduced by Higgins. Results: Participants, experimental interventions, and outcomes were reported in all the articles. Comparator intervention and study design were correctly reported in 103 (99.04%) and 101 (97.12%) articles, respectively. The comprehensive search strategy was available only in 48 articles (46.16%), and there was no evidence of a comprehensive search in 56 articles (53.84%). Risk of bias was investigated in 78 articles (75%). Also, funnel plots were the most commonly used method for reporting the bias in 64 articles (46.42%). Conclusion: In many of the meta-analyses, several items of the tool that represented a high-quality meta-analysis were absent. According to the findings, the comprehensive search and quality assessment were not at an appropriate level. Thus, the importance of reproducibility of information and quality assessment of included studies should be emphasized.

Mapping of reporting guidance for systematic reviews and meta-analyses generated a comprehensive item bank for future reporting guidelines

OBJECTIVES

The aim of the study was to generate a comprehensive bank of systematic review (SR) reporting items to inform an update of the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) 2009 statement.

METHODS

We searched the Enhancing the QUAlity and Transparency Of health Research Network library in May 2019 to identify all reporting guidelines for SRs that were published after 2009, regardless of the scope of the guideline. We also conducted a selective review of four guidance manuals for SRs, three tools for assessing the risk of bias in SRs, six meta-research studies evaluating the reporting quality of SRs using a tailored checklist, and five reporting guidelines for other study designs. One author screened and selected sources for inclusion, extracted reporting guidance from sources, and mapped guidance against the PRISMA 2009 checklist items.

RESULTS

We included 60 sources providing guidance on reporting of SRs and meta-analyses. From these, we collated a list of 221 unique reporting items. Items were categorized into title (four items), abstract (10 items), introduction (12 items), methods (111 items), results (61 items), discussion (12 items), funding and conflicts of interest (four items), administrative information (three items), and data availability (four items). This exercise generated 175 reporting items that could be added to the guidance in the PRISMA 2009 statement.

CONCLUSION

Generation of a comprehensive item bank through review and mapping of the literature facilitates identification of missing items and those needing modification, which may not otherwise be identified by the guideline development team or from other activities commonly used to develop reporting guidelines.

Seeking adverse effects in systematic reviews of orthodontic interventions: protocol for a cross-sectional study

BACKGROUND

Before implementing healthcare interventions, clinicians need to weigh the beneficial and adverse effects of interventions. However, a large body of evidence has demonstrated that seeking and reporting of adverse effects is suboptimal in clinical trials and in systematic reviews of interventions. This cross-sectional study will investigate the status of this problem in orthodontics. This study will assess whether adverse effects were sought and whether findings related to adverse effects were reported in systematic reviews of orthodontic interventions in the five leading orthodontic journals and in the Cochrane Database of Systematic Reviews.

METHODS

Systematic reviews of clinical orthodontic interventions published between 01 August 2009 and 31 July 2019 in the five leading orthodontic journals and in the Cochrane Database will be included. Empty reviews will be excluded. The reporting of outcomes on adverse effects will not determine eligibility, i.e., reviews will not be excluded, because they did not report usable data. Study selection and data extraction will be conducted independently by two authors. Our primary outcome will be the prevalence of systematic reviews of orthodontic interventions that sought any findings related to adverse effects in the included studies. Additional prevalence statistics will be calculated on a series of items related to seeking of adverse effects in the eligible reviews. All statistics will be calculated for (1) all journals together, (2) the group of five orthodontic journals and the Cochrane Database of Systematic Reviews separately, and (3) each individual journal separately. Chi-square tests of independence will be used to compare these groups.

DISCUSSION

This study will assess whether adverse effects were sought in systematic reviews of orthodontic interventions. This knowledge is important, because reviews that present an incomplete picture on adverse effects can have unfavorable consequences for the end-users. Also not reporting that no adverse effects were assessed in eligible studies included in a systematic review can mislead pertinent stakeholders. Our findings could have policy implications for making judgments on accepting or rejecting an intervention systematic review for publication, for example, by directing editors and peer-reviewers to adopt the various items on adverse effects defined in the MECIR standards and in the PRISMA harm checklist.

Evaluations of the uptake and impact of the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) Statement and extensions: a scoping review

BACKGROUND

The PRISMA Statement is a reporting guideline designed to improve transparency of systematic reviews (SRs) and meta-analyses. Seven extensions to the PRISMA Statement have been published to address the reporting of different types or aspects of SRs, and another eight are in development. We performed a scoping review to map the research that has been conducted to evaluate the uptake and impact of the PRISMA Statement and extensions. We also synthesised studies evaluating how well SRs published after the PRISMA Statement was disseminated adhere to its recommendations.

METHODS

We searched for meta-research studies indexed in MEDLINE® from inception to 31 July 2017, which investigated some component of the PRISMA Statement or extensions (e.g. SR adherence to PRISMA, journal endorsement of PRISMA). One author screened all records and classified the types of evidence available in the studies. We pooled data on SR adherence to individual PRISMA items across all SRs in the included studies and across SRs published after 2009 (the year PRISMA was disseminated).

RESULTS

We included 100 meta-research studies. The most common type of evidence available was data on SR adherence to the PRISMA Statement, which has been evaluated in 57 studies that have assessed 6487 SRs. The pooled results of these studies suggest that reporting of many items in the PRISMA Statement is suboptimal, even in the 2382 SRs published after 2009 (where nine items were adhered to by fewer than 67% of SRs). Few meta-research studies have evaluated the adherence of SRs to the PRISMA extensions or strategies to increase adherence to the PRISMA Statement and extensions.

CONCLUSIONS

Many studies have evaluated how well SRs adhere to the PRISMA Statement, and the pooled result of these suggest that reporting of many items is suboptimal. An update of the PRISMA Statement, along with a toolkit of strategies to help journals endorse and implement the updated guideline, may improve the transparency of SRs.

Benefit-Risk of Therapies for Relapsing-Remitting Multiple Sclerosis: Testing the Number Needed to Treat to Benefit (NNTB), Number Needed to Treat to Harm (NNTH) and the Likelihood to be Helped or Harmed (LHH): A Systematic Review and Meta-Analysis

OBJECTIVE

This study aimed to test the number needed to treat to benefit (NNTB) and to harm (NNTH), and the likelihood to be helped or harmed (LHH) when assessing benefits, risks, and benefit-risk ratios of disease-modifying treatments (DMTs) approved for relapsing-remitting multiple sclerosis (RRMS).

METHODS

In May 2016, we conducted a systematic review using the PubMed and Cochrane Central Register of Controlled Trials databases to identify phase III, randomized controlled trials with a duration of ≥2 years that assessed first-line (dimethyl fumarate [DMF], glatiramer acetate [GA], β-interferons [IFN], and teriflunomide) or second-line (alemtuzumab, fingolimod, and natalizumab) DMTs in patients with RRMS. Meta-analyses were performed to estimate relative risks (RRs) on annualized relapse rate (ARR), proportion of relapse-free patients (PPR-F), disability progression (PP-F-CDPS3M), and safety outcomes. NNTB and NNTH values were calculated applying RRs to control event rates. LHH was calculated as NNTH/NNTB ratio.

RESULTS

The lowest NNTBs on ARR, PPR-F, and PP-F-CDPS3M were found with IFN-β-1a-SC (NNTB 3, 95 % CI 2-4; NNTB 7, 95 % CI 4-18; NNTB 4, 95 % CI 3-7, respectively) and natalizumab (NNTB 2, 95 % CI 2-3; NNTB 4, 95 % CI 3-6; NNTB 9, 95 % CI 6-19, respectively). The lowest NNTH on adverse events leading to treatment discontinuation was found with IFN-β-1b (NNTH 14, 95 % 2-426) versus placebo; a protective effect was noted with alemtuzumab versus IFN-β-1a-SC (NNTB 22, 95 % 17-41). LHHs >1 were more frequent with IFN-β-1a-SC and natalizumab.

CONCLUSIONS

These metrics may be valuable for benefit-risk assessments, as they reflect baseline risks and are easily interpreted. Before making treatment decisions, clinicians must acknowledge that a higher RR reduction with drug A as compared with drug B (versus a common comparator in trial A and trial B, respectively) does not necessarily mean that the number of patients needed to be treated for one patient to encounter one aditional outcome of interest over a defined period of time is lower with drug A than with drug B. Overall, IFN-β-1a-SC and natalizumab seem to have the most favorable benefit-risk ratios among first- and second-line DMTs, respectively.

An appraisal of meta-analysis guidelines: how do they relate to safety outcomes?

Although well developed to assess efficacy questions, meta-analyses and, more generally, systematic reviews, have received less attention in application to safety-related questions. As a result, many open questions remain on how best to apply meta-analyses in the safety setting. This appraisal attempts to: (i) summarize the current guidelines for assessing individual studies, systematic reviews, and network meta-analyses; (ii) describe several publications on safety meta-analytic approaches; and (iii) present some of the questions and issues that arise with safety data. A number of gaps in the current quality guidelines are identified along with issues to consider when performing a safety meta-analysis. While some work is ongoing to provide guidance to improve the quality of safety meta-analyses, this review emphasizes the critical need for better reporting and increased transparency regarding safety data in the systematic review guidelines. Copyright © 2016 John Wiley & Sons, Ltd.

Scientific hypotheses can be tested by comparing the effects of one treatment over many diseases in a systematic review

OBJECTIVES

To describe the use of systematic reviews or overviews (systematic reviews of systematic reviews) to synthesize quantitative evidence of intervention effects across multiple indications (multiple-indication reviews) and to highlight issues pertaining to such reviews.

STUDY DESIGN AND SETTING

MEDLINE was searched from 2003 to January 2014. We selected multiple-indication reviews of interventions of allopathic medicine that included evidence from randomized controlled trials. We categorized the subject areas evaluated by these reviews and examined their methodology. Utilities and caveats of multiple-indication reviews are illustrated with examples drawn from published literature.

RESULTS

We retrieved 52 multiple-indication reviews covering a wide range of interventions. The method has been used to detect unintended effects, improve precision by pooling results across indications, and examine scientific hypotheses across disease classes.

CONCLUSION

Systematic reviews of interventions are typically used to evaluate the effects of treatments, one indication at a time. Here, we argue that, with due attention to methodological caveats, much can be learned by comparing the effects of a given treatment across many related indications.

Quality of reporting in systematic reviews of adverse events: systematic review

OBJECTIVES

To examine the quality of reporting of harms in systematic reviews, and to determine the need for a reporting guideline specific for reviews of harms.

DESIGN

Systematic review.

DATA SOURCES

Cochrane Database of Systematic Reviews (CDSR) and Database of Abstracts of Reviews of Effects (DARE).

REVIEW METHODS

Databases were searched for systematic reviews having an adverse event as the main outcome, published from January 2008 to April 2011. Adverse events included an adverse reaction, harms, or complications associated with any healthcare intervention. Articles with a primary aim to investigate the complete safety profile of an intervention were also included. We developed a list of 37 items to measure the quality of reporting on harms in each review; data were collected as dichotomous outcomes ("yes" or "no" for each item).

RESULTS

Of 4644 reviews identified, 309 were systematic reviews or meta-analyses primarily assessing harms (13 from CDSR; 296 from DARE). Despite a short time interval, the comparison between the years of 2008 and 2010-11 showed no difference on the quality of reporting over time (P=0.079). Titles in fewer than half the reviews (proportion of reviews 0.46 (95% confidence interval 0.40 to 0.52)) did not mention any harm related terms. Almost one third of DARE reviews (0.26 (0.22 to 0.31)) did not clearly define the adverse events reviewed, nor did they specify the study designs selected for inclusion in their methods section. Almost half of reviews (n=170) did not consider patient risk factors or length of follow-up when reviewing harms of an intervention. Of 67 reviews of complications related to surgery or other procedures, only four (0.05 (0.01 to 0.14)) reported professional qualifications of the individuals involved. The overall, unweighted, proportion of reviews with good reporting was 0.56 (0.55 to 0.57); corresponding proportions were 0.55 (0.53 to 0.57) in 2008, 0.55 (0.54 to 0.57) in 2009, and 0.57 (0.55 to 0.58) in 2010-11.

CONCLUSION

Systematic reviews compound the poor reporting of harms data in primary studies by failing to report on harms or doing so inadequately. Improving reporting of adverse events in systematic reviews is an important step towards a balanced assessment of an intervention.