Some improvements are apparent in identifying adverse effects in systematic reviews from 1994 to 2011

Conducting separate reviews of benefits and harms could improve systematic reviews and meta-analyses

Guidance for systematic reviews of interventions recommends both benefits and harms be included. Systematic reviews may reach conclusions about harms (or lack of harms) that are not true when reviews include only some relevant studies, rely on incomplete data from eligible studies, use inappropriate methods for synthesizing data, and report results selectively. Separate reviews about harms could address some of these problems, and we argue that conducting separate reviews of harms is a feasible alternative to current standards and practices. Systematic reviews of potential benefits could be organized around the use of interventions for specific health problems. Systematic reviews of potential harms could be broader, including more diverse study designs and including all people at risk of harms (who might use the same intervention to treat different health problems). Multiple reviews about benefits could refer to a single review of harms. This approach could improve the reliability, completeness, and efficiency of systematic reviews.

Appraising the safety and reporting quality of thread-embedding acupuncture: a protocol for a systematic review and meta-analysis

INTRODUCTION

Thread-embedding acupuncture (TEA) is a special type of acupuncture treatment in which medical threads are inserted into subcutaneous tissues or muscles at therapeutic points. TEA is a medical practice that combines acupuncture and medical threads. As such, it is necessary to evaluate the safety of TEA. This systematic review and meta-analysis aimed to assess the safety of TEA and reporting quality of studies regarding TEA.

METHODS AND ANALYSIS

The systematic review will be conducted using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines and the Cochrane Handbook for Systematic Reviews of Interventions. Searching strategies will be systemically conducted using the following databases from their inception date to September 2022: MEDLINE, Embase, Cochrane Library, Web of Science, China National Knowledge Infrastructure (CNKI), CiNii, J-STAGE, Korean Medical Database, Korean Studies Information Service System (KISS), ScienceON and Oriental Medicine Advanced Searching Integrated System (OASIS). The search strategies will be adjusted for each database as appropriate. The risk of bias will be assessed using the McMaster tool to identify the quality of harm assessment and reporting in study reports (McHarm). A meta-analysis will be used to synthesise the frequency and incidence of adverse events.

ETHICS AND DISSEMINATION

No ethical approval and consent is required for this systematic review. The results of this systematic review will be disseminated through peer-reviewed publications and conference presentations.

PROSPERO REGISTRATION NUMBER

CRD42022297123.

A systematic survey showed important limitations in the methods for assessing drug safety among systematic reviews

OBJECTIVES

This study aimed to examine the design, conduct, and analysis of systematic reviews assessing drug safety through a cross-sectional survey.

STUDY DESIGN AND SETTING

We searched PubMed to identity systematic reviews published in the Cochrane Database of Systematic Reviews and Core Clinical Journals indexed in 2015 and randomly sampled systematic reviews assessing drug effects at a 1:1 ratio of Cochrane and non-Cochrane reviews. Teams of two investigators independently conducted study screening and collected data, using prespecified, standardized questionnaires. In addition to general information, we collected details about the planning and analyses of safety outcomes.

RESULTS

We included 120 systematic reviews, including 60 Cochrane and 60 non-Cochrane reviews. Most reviews searched PubMed/MEDLINE (n = 117, 97.5%), EMBASE (n = 105, 87.5%), and Cochrane CENTRAL (n = 110, 91.7%) and conducted independent and duplicate study selection (n = 98, 81.7%), risk of bias assessment (n = 105, 87.5%), and data collection (n = 105, 87.5%). Only nine (7.5%) reviews clearly defined safety outcomes, and seven (5.8%) defined a primary safety outcome; none stated whether the primary safety outcome was predefined. Among the 80 reviews that pooled the primary dichotomous safety data across studies, less than half (41%, n = 33) conducted subgroup analysis to explore for sources of heterogeneity or reported a GRADE assessment for the overall quality of evidence. Cochrane reviews were more likely to provide a study protocol (100% vs. 23.3%; P < 0.001), involve methodologists (53.3% vs. 20.0%; P < 0.001), and report a GRADE assessment for the primary safety outcome (70.6% vs. 19.6%; P < 0.001).

CONCLUSION

Our findings highlighted areas for improved planning and analysis in the assessment of drug safety among systematic reviews. Cochrane reviews were superior to non-Cochrane reviews; however, most reviews did not prespecify their safety outcomes or methods for analysis, explore sources of heterogeneity among pooled effects, or assess the overall quality of evidence with the GRADE approach.

Spin in the reporting, interpretation, and extrapolation of adverse effects of orthodontic interventions: protocol for a cross-sectional study of systematic reviews

Background

Titles and abstracts are the most read sections of biomedical papers. It is therefore important that abstracts transparently report both the beneficial and adverse effects of health care interventions and do not mislead the reader. Misleading reporting, interpretation, or extrapolation of study results is called “spin”. In this study, we will assess whether adverse effects of orthodontic interventions were reported or considered in the abstracts of both Cochrane and non-Cochrane reviews and whether spin was identified and what type of spin.

Methods

Eligibility criteria were defined for the type of study designs, participants, interventions, outcomes, and settings. We will include systematic reviews of clinical orthodontic interventions published in the five leading orthodontic journals and in the Cochrane Database. Empty reviews will be excluded. We will manually search eligible reviews published between 1 August 2009 and 31 July 2019. Data collection forms were developed a priori. All study selection and data extraction procedures will be conducted by two reviewers independently. Our main outcomes will be the prevalence of reported or considered adverse effects of orthodontic interventions in the abstract of systematic reviews and the prevalence of “spin” related to these adverse effects. We will also record the prevalence of three subtypes of spin, i.e., misleading reporting, misleading interpretation, and misleading extrapolation-related spin. All statistics will be calculated for the following groups: (1) all journals individually, (2) all journals together, and (3) the five leading orthodontic journals and the Cochrane Database of Systematic Reviews separately. Generalized linear models will be developed to compare the various groups.

Discussion

We expect that our results will raise the awareness of the importance of reporting and considering of adverse effects and the presence of the phenomenon of spin related to these effects in abstracts of systematic reviews of orthodontic interventions. This is important, because an incomplete and inadequate reporting, interpretation, or extrapolation of findings on adverse effects in abstracts of systematic reviews can mislead readers and could lead to inadequate clinical practice. Our findings could result in policy implications for making judgments about the acceptance for publication of systematic reviews of orthodontic interventions.

Searching for qualitative health research required several databases and alternative search strategies: a study of coverage in bibliographic databases

OBJECTIVE

Retrieving the qualitative literature can be challenging, but the number and specific choice of databases are key factors. The aim of the present study is to provide guidance for the choice of databases for retrieving qualitative health research.

STUDY DESIGN AND SETTING

Seventy-one qualitative systematic reviews, from the Cochrane Database of Systematic Reviews and JBI database of Systematic Reviews and Implementation Reports, including 927 qualitative studies, were used to analyze the coverage of the qualitative literature in nine bibliographic databases.

RESULTS

The results show that 94.4% of the qualitative studies are indexed in at least one database, with a lower coverage for publication types other than journal articles. Maximum recall with two databases is 89.1%, with three databases recall increases to 92% and maximum recall with four databases is 93.1%. The remaining 6.9% of the publications consists of 1.3% scattered across five databases and 5.6% that are not indexed in any of the nine databases used in this study.

CONCLUSION

Retrieval in one or a few-although well selected-databases does not provide all the relevant qualitative studies. The remaining studies needs to be located using several other databases and alternative search strategies.

Overview: comprehensive and carefully constructed strategies are required when conducting searches for adverse effects data

OBJECTIVES

Methodological research has been undertaken to investigate the many challenges in searching for adverse effects data. It is imperative that the search approach adopted in systematic reviews is based on the best available evidence. We provide a detailed summary of the results and implications of the current evidence base to assist future searches for adverse effects.

STUDY DESIGN AND SETTING

This article is a narrative review from the authors of the Cochrane Handbook chapter on adverse effects.

RESULTS

The specified search strategy must be based on the population, intervention, comparator, outcome(s) format for question formulation and appropriate study designs for adverse effects data. Search filters and suggested search terms are available for the adverse effects of drug, medical devices, and surgical interventions. The use of generic adverse effects terms (such as harms and complications) as text words and indexing terms and specific adverse effects terms (such as rash and wound infection) are warranted. Searching databases beyond MEDLINE has proven useful, as well as the use of nondatabase sources.

CONCLUSION

This article provides the most up-to-date evidence-based guidance in identifying adverse effects data in the literature. It will support searchers and researchers evaluating the potential for harm of medical interventions in systematic reviews.

Seeking adverse effects in systematic reviews of orthodontic interventions: protocol for a cross-sectional study

BACKGROUND

Before implementing healthcare interventions, clinicians need to weigh the beneficial and adverse effects of interventions. However, a large body of evidence has demonstrated that seeking and reporting of adverse effects is suboptimal in clinical trials and in systematic reviews of interventions. This cross-sectional study will investigate the status of this problem in orthodontics. This study will assess whether adverse effects were sought and whether findings related to adverse effects were reported in systematic reviews of orthodontic interventions in the five leading orthodontic journals and in the Cochrane Database of Systematic Reviews.

METHODS

Systematic reviews of clinical orthodontic interventions published between 01 August 2009 and 31 July 2019 in the five leading orthodontic journals and in the Cochrane Database will be included. Empty reviews will be excluded. The reporting of outcomes on adverse effects will not determine eligibility, i.e., reviews will not be excluded, because they did not report usable data. Study selection and data extraction will be conducted independently by two authors. Our primary outcome will be the prevalence of systematic reviews of orthodontic interventions that sought any findings related to adverse effects in the included studies. Additional prevalence statistics will be calculated on a series of items related to seeking of adverse effects in the eligible reviews. All statistics will be calculated for (1) all journals together, (2) the group of five orthodontic journals and the Cochrane Database of Systematic Reviews separately, and (3) each individual journal separately. Chi-square tests of independence will be used to compare these groups.

DISCUSSION

This study will assess whether adverse effects were sought in systematic reviews of orthodontic interventions. This knowledge is important, because reviews that present an incomplete picture on adverse effects can have unfavorable consequences for the end-users. Also not reporting that no adverse effects were assessed in eligible studies included in a systematic review can mislead pertinent stakeholders. Our findings could have policy implications for making judgments on accepting or rejecting an intervention systematic review for publication, for example, by directing editors and peer-reviewers to adopt the various items on adverse effects defined in the MECIR standards and in the PRISMA harm checklist.

The reporting of safety among drug systematic reviews was poor before the implementation of the PRISMA harms checklist

OBJECTIVES

To examine, through a cross-sectional survey, how well safety information was reported among drug systematic reviews predating PRISMA harms checklist and explore factors associated with better reporting.

STUDY DESIGN AND SETTING

We searched PubMed to identify all systematic reviews published in the Cochrane Database of Systematic Review or the core clinical journals in 2015, one year before the PRISMA harms checklist was published. We randomly selected, in a 1:1 ratio, Cochrane and non-Cochrane systematic reviews assessing drug effects (including both efficacy and safety). We used the PRISMA harms checklist published in 2016 to assess the quality of reporting of drug safety information. Multivariable linear regression analyses were used to explore the association of six prespecified variables with more complete reporting of PRISMA harms items.

RESULTS

We included 120 systematic reviews, including 60 Cochrane and 60 non-Cochrane reviews. Scores on the PRISMA harms checklist (23 items) were low (median 4, [first, third quartile: 2, 6]), with no difference between Cochrane and non-Cochrane reviews (4.5 [2, 7] vs. 4 [2.5, 5]; P = 0.29). Among all eligible reviews, only one item (i.e., state conclusions in coherence with the review findings) was reported adequately (proportion of adherence 81.6%); proportion of reporting for other items ranged from 1.7% to 68.3%. The four essential reporting items from PRISMA harms checklist were also poorly complied (proportion of adherence ranged from 1.7% to 9.2%). Multivariable linear regression analyses found no significant associations between any study characteristic and reporting on the PRISMA harms, likely because of limited variability in scores across studies.

CONCLUSIONS

The reporting of safety information was poor both for Cochrane and non-Cochrane drug systematic reviews predating PRISMA harms checklist. The findings suggested a strong need to use the PRISMA harms checklist for reporting safety among drug systematic reviews.

The development of search filters for adverse effects of surgical interventions in medline and Embase

Background

Search filter development for adverse effects has tended to focus on retrieving studies of drug interventions. However, a different approach is required for surgical interventions.

Objective

To develop and validate search filters for medline and Embase for the adverse effects of surgical interventions.

Methods

Systematic reviews of surgical interventions where the primary focus was to evaluate adverse effect(s) were sought. The included studies within these reviews were divided randomly into a development set, evaluation set and validation set. Using word frequency analysis we constructed a sensitivity maximising search strategy and this was tested in the evaluation and validation set.

Results

Three hundred and fifty eight papers were included from 19 surgical intervention reviews. Three hundred and fifty two papers were available on medline and 348 were available on Embase. Generic adverse effects search strategies in medline and Embase could achieve approximately 90% relative recall. Recall could be further improved with the addition of specific adverse effects terms to the search strategies.

Conclusion

We have derived and validated a novel search filter that has reasonable performance for identifying adverse effects of surgical interventions in medline and Embase. However, we appreciate the limitations of our methods, and recommend further research on larger sample sizes and prospective systematic reviews.

Side-effects associated with ketamine use in depression: a systematic review

This is the first systematic review of the safety of ketamine in the treatment of depression after single and repeated doses. We searched MEDLINE, PubMed, PsycINFO, and Cochrane Databases and identified 288 articles, 60 of which met the inclusion criteria. After acute dosing, psychiatric, psychotomimetic, cardiovascular, neurological, and other side-effects were more frequently reported after ketamine treatment than after placebo in patients with depresssion. Our findings suggest a selective reporting bias with limited assessment of long-term use and safety and after repeated dosing, despite these being reported in other patient groups exposed to ketamine (eg, those with chronic pain) and in recreational users. We recommend large-scale clinical trials that include multiple doses of ketamine and long-term follow up to assess the safety of long-term regular use.

Impact of librarians on reporting of the literature searching component of pediatric systematic reviews

OBJECTIVE

A critical element in conducting a systematic review is the identification of studies. To date, very little empirical evidence has been reported on whether the presence of a librarian or information professional can contribute to the quality of the final product. The goal of this study was to compare the reporting rigor of the literature searching component of systematic reviews with and without the help of a librarian.

METHOD

Systematic reviews published from 2002 to 2011 in the twenty highest impact factor pediatrics journals were collected from MEDLINE. Corresponding authors were contacted via an email survey to determine if a librarian was involved, the role that the librarian played, and functions that the librarian performed. The reviews were scored independently by two reviewers using a fifteen-item checklist.

RESULTS

There were 186 reviews that met the inclusion criteria, and 44% of the authors indicated the involvement of a librarian in conducting the systematic review. With the presence of a librarian as coauthor or team member, the mean checklist score was 8.40, compared to 6.61 (p<0.001) for reviews without a librarian.

CONCLUSIONS

Findings indicate that having a librarian as a coauthor or team member correlates with a higher score in the literature searching component of systematic reviews.

Reporting of Adverse Events in Published and Unpublished Studies of Health Care Interventions: A Systematic Review

BACKGROUND

We performed a systematic review to assess whether we can quantify the underreporting of adverse events (AEs) in the published medical literature documenting the results of clinical trials as compared with other nonpublished sources, and whether we can measure the impact this underreporting has on systematic reviews of adverse events.

METHODS AND FINDINGS

Studies were identified from 15 databases (including MEDLINE and Embase) and by handsearching, reference checking, internet searches, and contacting experts. The last database searches were conducted in July 2016. There were 28 methodological evaluations that met the inclusion criteria. Of these, 9 studies compared the proportion of trials reporting adverse events by publication status. The median percentage of published documents with adverse events information was 46% compared to 95% in the corresponding unpublished documents. There was a similar pattern with unmatched studies, for which 43% of published studies contained adverse events information compared to 83% of unpublished studies. A total of 11 studies compared the numbers of adverse events in matched published and unpublished documents. The percentage of adverse events that would have been missed had each analysis relied only on the published versions varied between 43% and 100%, with a median of 64%. Within these 11 studies, 24 comparisons of named adverse events such as death, suicide, or respiratory adverse events were undertaken. In 18 of the 24 comparisons, the number of named adverse events was higher in unpublished than published documents. Additionally, 2 other studies demonstrated that there are substantially more types of adverse events reported in matched unpublished than published documents. There were 20 meta-analyses that reported the odds ratios (ORs) and/or risk ratios (RRs) for adverse events with and without unpublished data. Inclusion of unpublished data increased the precision of the pooled estimates (narrower 95% confidence intervals) in 15 of the 20 pooled analyses, but did not markedly change the direction or statistical significance of the risk in most cases. The main limitations of this review are that the included case examples represent only a small number amongst thousands of meta-analyses of harms and that the included studies may suffer from publication bias, whereby substantial differences between published and unpublished data are more likely to be published.

CONCLUSIONS

There is strong evidence that much of the information on adverse events remains unpublished and that the number and range of adverse events is higher in unpublished than in published versions of the same study. The inclusion of unpublished data can also reduce the imprecision of pooled effect estimates during meta-analysis of adverse events.

Most systematic reviews of adverse effects did not include unpublished data

OBJECTIVES

We sought to identify the proportion of systematic reviews of adverse effects which search for unpublished data and the success rates of identifying unpublished data for inclusion in a systematic review.

STUDY DESIGN AND SETTING

Two reviewers independently screened all records published in 2014 in the Database of Abstracts of Reviews of Effects (DARE) for systematic reviews where the primary aim was to evaluate an adverse effect or effects. Data were extracted on the types of adverse effects and interventions evaluated, sources searched, how many unpublished studies were included, and source or type of unpublished data included.

RESULTS

From 9,129 DARE abstracts, 348 met our inclusion criteria. Most of these reviews evaluated a drug intervention (237/348, 68%) with specified adverse effects (250/348, 72%). Over a third (136/348, 39%) of all the reviews searched, a specific source for unpublished data, such as conference abstracts or trial registries, and nearly half of these reviews (65/136, 48%) included unpublished data. An additional 13 reviews included unpublished data despite not searching specific sources for unpublished studies. Overall, 22% (78/348) of reviews included unpublished data/studies.

CONCLUSION

Most reviews of adverse effects do not search specifically for unpublished data but, of those that do, nearly half are successful.

Use of recommended search strategies in systematic reviews and the impact of librarian involvement: a cross-sectional survey of recent authors

Background

Previous research looking at published systematic reviews has shown that their search strategies are often suboptimal and that librarian involvement, though recommended, is low. Confidence in the results, however, is limited due to poor reporting of search strategies the published articles.

Objectives

To more accurately measure the use of recommended search methods in systematic reviews, the levels of librarian involvement, and whether librarian involvement predicts the use of recommended methods.

Methods

A survey was sent to all authors of English-language systematic reviews indexed in the Database of Abstracts of Reviews of Effects (DARE) from January 2012 through January 2014. The survey asked about their use of search methods recommended by the Institute of Medicine, Cochrane Collaboration, and the Agency for Healthcare Research and Quality and if and how a librarian was involved in the systematic review. Rates of use of recommended methods and librarian involvement were summarized. The impact of librarian involvement on use of recommended methods was examined using a multivariate logistic regression.

Results

1560 authors completed the survey. Use of recommended search methods ranged widely from 98% for use of keywords to 9% for registration in PROSPERO and were generally higher than in previous studies. 51% of studies involved a librarian, but only 64% acknowledge their assistance. Librarian involvement was significantly associated with the use of 65% of recommended search methods after controlling for other potential predictors. Odds ratios ranged from 1.36 (95% CI 1.06 to 1.75) for including multiple languages to 3.07 (95% CI 2.06 to 4.58) for using controlled vocabulary.

Conclusions

Use of recommended search strategies is higher than previously reported, but many methods are still under-utilized. Librarian involvement predicts the use of most methods, but their involvement is under-reported within the published article.

Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews

OBJECTIVES

To determine whether librarian and information specialist authorship was associated with better reported systematic review (SR) search quality.

STUDY DESIGN AND SETTING

SRs from high-impact general internal medicine journals were reviewed for search quality characteristics and reporting quality by independent reviewers using three instruments, including a checklist of Institute of Medicine Recommended Standards for the Search Process and a scored modification of the Peer Review of Electronic Search Strategies instrument.

RESULTS

The level of librarian and information specialist participation was significantly associated with search reproducibility from reported search strategies (Χ(2) = 23.5; P < 0.0001). Librarian co-authored SRs had significantly higher odds of meeting 8 of 13 analyzed search standards than those with no librarian participation and six more than those with mentioned librarian participation. One-way ANOVA showed that differences in total search quality scores between all three groups were statistically significant (F2,267 = 10.1233; P < 0.0001).

CONCLUSION

Problems remain with SR search quality and reporting. SRs with librarian or information specialist co-authors are correlated with significantly higher quality reported search strategies. To minimize bias in SRs, authors and editors could encourage librarian engagement in SRs including authorship as a potential way to help improve documentation of the search strategy.

Bidirectional citation searching to completion: an exploration of literature searching methods

Literature reviews underpin the majority of research projects in the health sciences, and yet relatively little analysis has been published as to the most appropriate method to identify relevant literature, outside of specialist information journals. The method of applying keyword search queries to bibliographic databases using Boolean logic dominates literature reviews due to its easy application to the major online databases. However, it is recognised increasingly as being problematic where the research question cannot be clearly defined or requires an element of exploration, due to its reliance on author's use of titling and keywords and is unable to identify topics other than those defined in the search query. This paper discusses the relative merits of a systematic citation searching approach as both an alternative and a concurrent method to keyword searching. A method of citation searching, both forwards and backwards, which is iterated to form a closed loop solution, is discussed. An illustrative example is presented of both methods, applying them to the topic of the UK National Institute for Health and Care Excellence (NICE) cost-effectiveness threshold. The case study finds the citation searching approach dominates the traditional keyword searching approach, finding 76 papers of relevance, including all 15 found by the alternative approach. Conceptually, and in the example presented, it is demonstrated that the proposed method can represent a dominant strategy to the more traditional approach in some situations, highlighting that, wherever possible, it is preferential to employ multiple methods of searching. However, it is clear that a better understanding is required as to how we can most efficiently search the ever-growing sea of literature.

Selective reporting bias of harm outcomes within studies: findings from a cohort of systematic reviews

OBJECTIVE

To determine the extent and nature of selective non-reporting of harm outcomes in clinical studies that were eligible for inclusion in a cohort of systematic reviews.

DESIGN

Cohort study of systematic reviews from two databases.

SETTING

Outcome reporting bias in trials for harm outcomes (ORBIT II) in systematic reviews from the Cochrane Library and a separate cohort of systematic reviews of adverse events.

PARTICIPANTS

92 systematic reviews of randomised controlled trials and non-randomised studies published in the Cochrane Library between issue 9, 2012 and issue 2, 2013 (Cochrane cohort) and 230 systematic reviews published between 1 January 2007 and 31 December 2011 in other publications, synthesising data on harm outcomes (adverse event cohort).

METHODS

A 13 point classification system for missing outcome data on harm was developed and applied to the studies.

RESULTS

86% (79/92) of reviews in the Cochrane cohort did not include full data from the main harm outcome of interest of each review for all of the eligible studies included within that review; 76% (173/230) for the adverse event cohort. Overall, the single primary harm outcome was inadequately reported in 76% (705/931) of the studies included in the 92 reviews from the Cochrane cohort and not reported in 47% (4159/8837) of the 230 reviews in the adverse event cohort. In a sample of primary studies not reporting on the single primary harm outcome in the review, scrutiny of the study publication revealed that outcome reporting bias was suspected in nearly two thirds (63%, 248/393).

CONCLUSIONS

The number of reviews suspected of outcome reporting bias as a result of missing or partially reported harm related outcomes from at least one eligible study is high. The declaration of important harms and the quality of the reporting of harm outcomes must be improved in both primary studies and systematic reviews.

THE CONTRIBUTION OF DIFFERENT INFORMATION SOURCES TO IDENTIFY ADVERSE EFFECTS OF A MEDICAL DEVICE: A CASE STUDY USING A SYSTEMATIC REVIEW OF SPINAL FUSION

Background: The most effective sources to search to identify adverse effects data for medical devices are currently unknown.

Methods: The included studies from a systematic review of the safety of recombinant human bone morphogenetic protein-2 (rhBMP-2) for spinal fusion were used for analysis. For each source searched, a record was made for each relevant publication of whether it was retrieved by the search strategy used and whether it was available in the database but not retrieved. To account for multiple publications of the same study, a record was made of the relevant studies identified. The sensitivity, precision, and number needed to read were calculated as well as the minimum combination of sources to identify all the publications or studies.

Results: There were eighty-two publications (forty-nine studies) included in the systematic review. Only one article was available in a database searched but not retrieved by our search strategy. Science Citation Index (SCI) and EMBASE both achieved the highest sensitivity (62 percent), followed closely by MEDLINE/PubMED (56 percent). With the search strategies used, the minimum combination of sources needed to identify all the publications was SCI, EMBASE, CENTRAL, and either MEDLINE or PubMED, in addition to reference checking, contacting authors and an automated current awareness service. In relation to identifying all the relevant studies, the minimum combination of studies was similar with the exclusion of CENTRAL.

Conclusions: To identify all the relevant publications or studies included in this case study systematic review, several different sources needed to be searched.

Comparison of search strategies in systematic reviews of adverse effects to other systematic reviews

BACKGROUND

Research indicates that the methods used to identify data for systematic reviews of adverse effects may need to differ from other systematic reviews.

OBJECTIVES

To compare search methods in systematic reviews of adverse effects with other reviews.

METHODS

The search methodologies in 849 systematic reviews of adverse effects were compared with other reviews.

RESULTS

Poor reporting of search strategies is apparent in both systematic reviews of adverse effects and other types of systematic reviews. Systematic reviews of adverse effects are less likely to restrict their searches to MEDLINE or include only randomised controlled trials (RCTs). The use of other databases is largely dependent on the topic area and the year the review was conducted, with more databases searched in more recent reviews. Adverse effects search terms are used by 72% of reviews and despite recommendations only two reviews report using floating subheadings.

CONCLUSIONS

The poor reporting of search strategies in systematic reviews is universal, as is the dominance of searching MEDLINE. However, reviews of adverse effects are more likely to include a range of study designs (not just RCTs) and search beyond MEDLINE.

Quality of reporting in systematic reviews of adverse events: systematic review

OBJECTIVES

To examine the quality of reporting of harms in systematic reviews, and to determine the need for a reporting guideline specific for reviews of harms.

DESIGN

Systematic review.

DATA SOURCES

Cochrane Database of Systematic Reviews (CDSR) and Database of Abstracts of Reviews of Effects (DARE).

REVIEW METHODS

Databases were searched for systematic reviews having an adverse event as the main outcome, published from January 2008 to April 2011. Adverse events included an adverse reaction, harms, or complications associated with any healthcare intervention. Articles with a primary aim to investigate the complete safety profile of an intervention were also included. We developed a list of 37 items to measure the quality of reporting on harms in each review; data were collected as dichotomous outcomes ("yes" or "no" for each item).

RESULTS

Of 4644 reviews identified, 309 were systematic reviews or meta-analyses primarily assessing harms (13 from CDSR; 296 from DARE). Despite a short time interval, the comparison between the years of 2008 and 2010-11 showed no difference on the quality of reporting over time (P=0.079). Titles in fewer than half the reviews (proportion of reviews 0.46 (95% confidence interval 0.40 to 0.52)) did not mention any harm related terms. Almost one third of DARE reviews (0.26 (0.22 to 0.31)) did not clearly define the adverse events reviewed, nor did they specify the study designs selected for inclusion in their methods section. Almost half of reviews (n=170) did not consider patient risk factors or length of follow-up when reviewing harms of an intervention. Of 67 reviews of complications related to surgery or other procedures, only four (0.05 (0.01 to 0.14)) reported professional qualifications of the individuals involved. The overall, unweighted, proportion of reviews with good reporting was 0.56 (0.55 to 0.57); corresponding proportions were 0.55 (0.53 to 0.57) in 2008, 0.55 (0.54 to 0.57) in 2009, and 0.57 (0.55 to 0.58) in 2010-11.

CONCLUSION

Systematic reviews compound the poor reporting of harms data in primary studies by failing to report on harms or doing so inadequately. Improving reporting of adverse events in systematic reviews is an important step towards a balanced assessment of an intervention.

Optimising the retrieval of information on adverse drug effects

Pharmaceutical interventions have brought about many benefits to health, improving the population's well-being and life expectancy. However, these interventions are not without potential harmful side-effects and yet searching for the evidence on adverse effects is challenging. This article summarises a PhD whose main aim was to develop a better understanding of the implications of using different sources and approaches to identifying relevant data on adverse effects. The author is Su Golder, who has recently completed her PhD at the University of York and who has already published several articles on specific aspects of her research, including this journal. This article is the first in the Dissertations into Practice series to report on a PhD study, and it summarises her research in a way which emphasises the implications for practice.

Methodological developments in searching for studies for systematic reviews: past, present and future?

The Cochrane Collaboration was established in 1993, following the opening of the UK Cochrane Centre in 1992, at a time when searching for studies for inclusion in systematic reviews was not well-developed. Review authors largely conducted their own searches or depended on medical librarians, who often possessed limited awareness and experience of systematic reviews. Guidance on the conduct and reporting of searches was limited. When work began to identify reports of randomized controlled trials (RCTs) for inclusion in Cochrane Reviews in 1992, there were only approximately 20,000 reports indexed as RCTs in MEDLINE and none indexed as RCTs in Embase. No search filters had been developed with the aim of identifying all RCTs in MEDLINE or other major databases. This presented The Cochrane Collaboration with a considerable challenge in identifying relevant studies.Over time, the number of studies indexed as RCTs in the major databases has grown considerably and the Cochrane Central Register of Controlled Trials (CENTRAL) has become the best single source of published controlled trials, with approximately 700,000 records, including records identified by the Collaboration from Embase and MEDLINE. Search filters for various study types, including systematic reviews and the Cochrane Highly Sensitive Search Strategies for RCTs, have been developed. There have been considerable advances in the evidence base for methodological aspects of information retrieval. The Cochrane Handbook for Systematic Reviews of Interventions now provides detailed guidance on the conduct and reporting of searches. Initiatives across The Cochrane Collaboration to improve the quality inter alia of information retrieval include: the recently introduced Methodological Expectations for Cochrane Intervention Reviews (MECIR) programme, which stipulates 'mandatory' and 'highly desirable' standards for various aspects of review conduct and reporting including searching, the development of Standard Training Materials for Cochrane Reviews and work on peer review of electronic search strategies. Almost all Cochrane Review Groups and some Cochrane Centres and Fields now have a Trials Search Co-ordinator responsible for study identification and medical librarians and other information specialists are increasingly experienced in searching for studies for systematic reviews.Prospective registration of clinical trials is increasing and searching trials registers is now mandatory for Cochrane Reviews, where relevant. Portals such as the WHO International Clinical Trials Registry Platform (ICTRP) are likely to become increasingly attractive, given concerns about the number of trials which may not be registered and/or published. The importance of access to information from regulatory and reimbursement agencies is likely to increase. Cross-database searching, gateways or portals and improved access to full-text databases will impact on how searches are conducted and reported, as will services such as Google Scholar, Scopus and Web of Science. Technologies such as textual analysis, semantic analysis, text mining and data linkage will have a major impact on the search process but efficient and effective updating of reviews may remain a challenge.In twenty years' time, we envisage that the impact of universal social networking, as well as national and international legislation, will mean that all trials involving humans will be registered at inception and detailed trial results will be routinely available to all. Challenges will remain, however, to ensure the discoverability of relevant information in diverse and often complex sources and the availability of metadata to provide the most efficient access to information. We envisage an ongoing role for information professionals as experts in identifying new resources, researching efficient ways to link or mine them for relevant data and managing their content for the efficient production of systematic reviews.

Comparison of pooled risk estimates for adverse effects from different observational study designs: methodological overview

BACKGROUND

A diverse range of study designs (e.g. case-control or cohort) are used in the evaluation of adverse effects. We aimed to ascertain whether the risk estimates from meta-analyses of case-control studies differ from that of other study designs.

METHODS

Searches were carried out in 10 databases in addition to reference checking, contacting experts, and handsearching key journals and conference proceedings. Studies were included where a pooled relative measure of an adverse effect (odds ratio or risk ratio) from case-control studies could be directly compared with the pooled estimate for the same adverse effect arising from other types of observational studies.

RESULTS

We included 82 meta-analyses. Pooled estimates of harm from the different study designs had 95% confidence intervals that overlapped in 78/82 instances (95%). Of the 23 cases of discrepant findings (significant harm identified in meta-analysis of one type of study design, but not with the other study design), 16 (70%) stemmed from significantly elevated pooled estimates from case-control studies. There was associated evidence of funnel plot asymmetry consistent with higher risk estimates from case-control studies. On average, cohort or cross-sectional studies yielded pooled odds ratios 0.94 (95% CI 0.88-1.00) times lower than that from case-control studies.

INTERPRETATION

Empirical evidence from this overview indicates that meta-analysis of case-control studies tend to give slightly higher estimates of harm as compared to meta-analyses of other observational studies. However it is impossible to rule out potential confounding from differences in drug dose, duration and populations when comparing between study designs.