Methodology and reporting of meta-analyses in the neurosurgical literature

Systematic reviews and meta-analyses in neurosurgery Part II: a guide to designing the protocol

Despite clearly established guidelines, recent audits have found the conduct and reporting of systematic reviews and meta-analyses (SRMAs) within neurosurgery to be relatively lackluster in methodological rigor and compliance. Protocols of SRMAs allow for planning and documentation of review methods, guard against arbitrary decision-making during the review process, and enable readers to assess for the presence of selective reporting. To aid transparency, authors should provide sufficient detail in their protocol so that the readers could reproduce the study themselves. Development of our guideline drew heavily from the Preferred Reporting Items for Systematic Reviews and Meta-analyses Protocols (PRISMA-P) initiative. The objective of this article is not to enumerate every detail of this checklist, but to provide guidance to authors preparing their protocol, with examples, for a systematic review in neurosurgery. Particularly, we emphasize on the PICO framework - population (P), interventions (I), comparators (C), outcomes (O) - which is central to constructing a clinical question, defining the scope of the systematic review, defining and prioritizing the primary outcome, to specifying the eligibility criteria, designing the search strategy, and identifying potential sources of heterogeneity. We encourage our readers to make use of this guideline alongside the PRISMA-P 2015 statement, when drafting and appraising systematic review protocols.

Systematic reviews and meta-analyses in neurosurgery part I: interpreting and critically appraising as a guide for clinical practice

Neurosurgeons are inundated with the Herculean task to keep abreast with the rapid pace at which clinical research is proliferating. Systematic reviews and meta-analyses (SRMAs) have consequently surged in popularity because when executed properly, they constitute the highest level of evidence, and may save busy neurosurgeons many hours of combing the literature. Well-executed SRMAs may prove instructive for clinical practice, but poorly conducted reviews sow confusion and may potentially cause harm. Unfortunately, many SRMAs within neurosurgery are relatively lackluster in methodological rigor. When neurosurgeons apply the results of an SRMA to patient care, they should start by evaluating the extent to which the employed methods have likely protected against misleading results. The present article aims to educate the reader about how to interpret an SRMA, to assess the potential relevance of its results in the special context of the neurosurgical patient population.

Meningioma systematic reviews and meta-analyses: an assessment of reporting and methodological quality

INTRODUCTION

Systematic reviews (SR) and systematic reviews with meta-analysis (SRMA) can constitute the highest level of research evidence. Such evidence syntheses are relied upon heavily to inform the clinical knowledge base and to guide clinical practice for meningioma. This review evaluates the reporting and methodological quality of published meningioma evidence syntheses to date.

METHODS

Eight electronic databases/registries were searched to identify eligible meningioma SRs with and without meta-analysis published between January 1990 and December 2020. Articles concerning spinal meningioma were excluded. Reporting and methodological quality were assessed against the following tools: Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), A MeaSurement Tool to Assess systematic Reviews (AMSTAR 2), and Risk Of Bias in Systematic reviews (ROBIS).

RESULTS

116 SRs were identified, of which 57 were SRMAs (49.1%). The mean PRISMA score for SRMA was 20.9 out of 27 (SD 3.9, 77.0% PRISMA adherence) and for SR without meta-analysis was 13.8 out of 22 (SD 3.4, 63% PRISMA adherence). Thirty-eight studies (32.8%) achieved greater than 80% adherence to PRISMA. Methodological quality assessment against AMSTAR 2 revealed that 110 (94.8%) studies were of critically low quality. Only 21 studies (18.1%) were judged to have a low risk of bias against ROBIS.

CONCLUSION

The reporting and methodological quality of meningioma evidence syntheses was poor. Established guidelines and critical appraisal tools may be used as an adjunct to aid methodological conduct and reporting of such reviews, in order to improve the validity and transparency of research which may influence clinical practice.

Ventral Intermediate Nucleus of the Thalamus versus Posterior Subthalamic Area: Network Meta-Analysis of DBS Target Site Efficacy for Essential Tremor

BACKGROUND

Deep brain stimulation (DBS) targeting the ventral intermediate nucleus (Vim) of the thalamus or the posterior subthalamic area (PSA) are effective treatments for essential tremor (ET). However, their relative efficacy is unknown.

OBJECTIVE

Here, we present the first systematic review and network meta-analysis, examining the efficacy of Vim versus PSA DBS for treating medically refractory ET.

METHODS

We included all primary studies that reported validated Fahn-Tolosa-Marin Tremor Rating Scale (FTM-TRS) scores pre-/postimplantation or on-/off-stimulation postimplantation, for patients receiving either Vim or PSA DBS. The primary outcome was FTM-TRS score reduction; the secondary outcome was percent reduction in score. We categorized all outcomes as short-term (≤12 months) or long-term (>12 months).

RESULTS

For pre-/postimplantation comparisons, 19 and 11 studies met inclusion criteria for short- and long-term follow-ups, respectively. For on-/off-stimulation tremor score comparisons, 8 studies met inclusion criteria for short-term follow-up. Network meta-analysis of pre-/postimplantation tremor scores showed greater tremor reduction with PSA implantation short-term (absolute tremor reduction: PSA: -30.94 [95% confidence interval (CI): -34.93, -26.95]; Vim: -26.26 [95% CI: -33.39, -19.12]; relative tremor reduction: PSA: 63.3% [95% CI: 61.8%-64.8%]; Vim: 57.8% [95% CI: 56.5%-59.0%]). However, there was no difference in efficacy between PSA and Vim DBS when comparing tremor on-versus off-stimulation at short-term follow-up or pre- versus postimplantation tremor reduction long-term.

CONCLUSION

Our systematic review highlighted both heterogeneity in scoring systems used and lack of transparency in reporting total scores, limiting direct comparison across studies. We found a modestly superior efficacy with PSA stimulation in the short term, but no difference in tremor reduction long-term.

Use of Topical Vancomycin Powder to Reduce Surgical Site Infections after Deep Brain Stimulation Surgery: UCSF Experience and Meta-Analysis

OBJECTIVE

Surgical site infection (SSI) is the most common serious complication of deep brain stimulation (DBS) implantation surgery. Here, we report a single-surgeon experience on the efficacy of topical, intrawound vancomycin powder (VP) in reducing SSI for DBS surgery and present the first systematic review and meta-analysis examining the effect of topical vancomycin on SSI in patients after DBS surgery.

METHODS

For the retrospective review, all unique patients undergoing DBS surgery at UCSF for new hardware implantation or internal pulse generator (IPG) replacement by a single surgeon from September 2013 to March 2019, with at least 1 year of follow-up data, were included. For the meta-analysis, we included all primary studies that compared SSIs with and without application of topical vancomycin in DBS surgeries.

RESULTS

368 unique patients met inclusion criteria; 195 patients received topical VP (VP group) and 173 did not (control). 99/195 patients in the VP group underwent new DBS implantation and 96/195 had IPG replacement. 71/173 patients in the control group had new DBS implantation and 102/173 had IPG replacement. There were 10 total cases of SSI: 4 patients from the VP group (3 new implants and 1 IPG replacement) and 6 patients from the control group (3 new implants and 3 IPG replacements), resulting in SSI rates of 2.1 and 3.5%, respectively (p value = 0.337). Including our retrospective analysis, 6 studies met inclusion criteria for the systematic review and meta-analysis. In the 4 studies that examined primary DBS implants, 479 total patients received topical VP and 436 did not; mean odds ratio for SSI with topical vancomycin was 0.802 (95% confidence interval [CI] 0.175-3.678). Across the 5 studies that examined IPG implantations or replacements, 606 total patients received topical VP while 1,173 patients did not; mean odds ratio for SSI with topical vancomycin was 0.492 (95% CI 0.164-1.475). In either case, topical VP application did not significantly decrease risk of SSI.

CONCLUSION

Surgical infections after DBS surgery are uncommon events, with studies demonstrating mixed results on whether topical vancomycin reduces this risk. Our single-institution retrospective analysis and systematic review of prior studies both demonstrated no significant SSI rate reduction with topical VP. This is likely due to low baseline SSI rates, resulting in a small effect size for prevention. Given the cost-effectiveness, simplicity, and low risk, topical, intrawound VP remains a treatment option to further reduce risk of SSI, particularly in settings with higher baseline infection rates.

Evidence-based medicine in neurosurgery: why and how?

Since its inception, medical training and practice are based primarily on the clinical experience provided by the mentors of each of the medical schools. In the 1990s, the first steps were taken for the construction of what we now know as evidence-based medicine. Evidence-based medicine is the set of studies, methods, and principles for the prevention of medical diseases, management guidelines, and algorithms used in sources of evidence. Neurosurgery based on evidence has emerged thanks to advances in neuroscience and information technology that allows the globalization of current scientific information. The results of important reviews on the levels of evidence in neurosurgery are low in percentages of high-quality evidence in this field of medicine. Based on the above, the objective of this manuscript is to describe the application of evidence in neurosurgery, the current state of the art in evidence-based medicine, and the steps needed to create evidence of the best quality in neurosurgery.

Quality of Literature Searches Published in Leading Neurosurgical Journals: A Review of Reviews

BACKGROUND

There is mounting evidence that the search strategies upon which systematic reviews (SRs) are based frequently contain errors are incompletely reported or insensitive.

OBJECTIVE

To appraise the quality of search strategies in the 10 leading specialty neurosurgical journals and identify factors associated with superior searches.

METHODS

This research-on-research study systematically surveyed SRs published in the 10 leading neurosurgical journals between 01/10/2017 and 31/10/2019. All SRs were eligible for assessment using a predefined coding manual that was adapted from the preferred reporting items for systematic reviews and meta-analyses (PRISMA), a measurement tool to assess systematic reviews (AMSTAR), and Cochrane Collaboration guidelines. The PubMed interface was used to search the MEDLINE database, which was supplemented by individual journal searches. Descriptive statistics were utilized to identify factors associated with improved search strategies.

RESULTS

A total of 633 articles were included and contained a median of 19.00 (2.00-1654.00) studies. Less than half (45.97%) of included search strategies were considered to be reproducible. Aggregated reporting score was positively associated with in-text reference to reporting guideline adherence (τb = 0.156, P < .01). The number of articles retrieved by a search (τb = 0.11, P < .01) was also associated with the reporting of a reproducible search strategy.

CONCLUSION

This study demonstrates that the search strategies used in neurosurgical SRs require improvement. In addition to increasing awareness of reporting standards, we propose that this be achieved by the incorporation of PRISMA and other guidelines into article submission and peer-review processes. This may lead to the conduct of more informative SRs, which may result in improved clinician decision-making and patient outcomes.

Critically Low Confidence in the Results Produced by Spine Surgery Systematic Reviews: An AMSTAR-2 Evaluation From 4 Spine Journals

STUDY DESIGN

A systematic cross-sectional survey of systematic reviews (SRs).

OBJECTIVE

To evaluate the methodological quality of spine surgery SRs published in 2018 using the updated AMSTAR 2 critical appraisal instrument.

METHODS

We identified the PubMed indexed journals devoted to spine surgery research in 2018. All SRs of spine surgical interventions from those journals were critically appraised for quality independently by 2 reviewers using the AMSTAR 2 instrument. We calculated the percentage of SRs achieving a positive response for each AMSTAR 2 domain item and assessed the levels of confidence in the results of each SR.

RESULTS

We identified 28 SRs from 4 journals that met our criteria for inclusion. Only 49.5% of the AMSTAR 2 domain items satisfied the AMSTAR 2 criteria. Critical domain items were satisfied less often (39.1%) compared with noncritical domain items (57.3%). Domain items most poorly reported include accounting for individual study risk of bias when interpreting results (14%), list and justification of excluded articles (18%), and an a priori establishment of methods prior to the review or registered protocol (18%). The overall confidence in the results was rated "low" in 2 SRs and "critically low" in 26.

CONCLUSIONS

The credibility of a SR and its value to clinicians and policy makers are dependent on its methodological quality. This appraisal found significant methodological limitations in several critical domains, such that the confidence in the findings of these reviews is "critically low."

Pearls for Interpreting Neurosurgical Systematic Reviews and Meta-Analyses: Lessons From a Collaborative Effort

Systematic reviews and meta-analyses in the neurosurgical literature have surged in popularity over the last decade. It is our concern that, without a renewed effort to critically interpret and appraise these studies as high or low quality, we run the risk of the quality and value of evidence-based medicine in neurosurgery being misinterpreted. Correspondingly, we have outlined 4 major domains to target in interpreting neurosurgical systematic reviews and meta-analyses based on the lessons learned by a collaboration of clinicians and academics summarized as 4 pearls. The domains of (1) heterogeneity, (2) modeling, (3) certainty, and (4) bias in neurosurgical systematic reviews and meta-analyses were identified as aspects in which the authors’ approaches have changed over time to improve robustness and transparency. Examples of how and why these pearls were adapted were provided in areas of cranial neuralgia, spine, pediatric, and neuro-oncology to demonstrate how neurosurgical readers and writers may improve their interpretation of these domains. The incorporation of these pearls into practice will empower neurosurgical academics to effectively interpret systematic reviews and meta-analyses, enhancing the quality of our evidence-based medicine literature while maintaining a critical focus on the needs of the individual patients in neurosurgery.

Reporting trends of p values in the neurosurgical literature

OBJECTIVE

The objective of this study was to evaluate the trends in reporting of p values in the neurosurgical literature from 1990 through 2017.

METHODS

All abstracts from the Journal of Neurology, Neurosurgery, and Psychiatry (JNNP), Journal of Neurosurgery (JNS) collection (including Journal of Neurosurgery: Spine and Journal of Neurosurgery: Pediatrics), Neurosurgery (NS), and Journal of Neurotrauma (JNT) available on PubMed from 1990 through 2017 were retrieved. Automated text mining was performed to extract p values from relevant abstracts. Extracted p values were analyzed for temporal trends and characteristics.

RESULTS

The search yielded 47,889 relevant abstracts. A total of 34,324 p values were detected in 11,171 abstracts. Since 1990 there has been a steady, proportionate increase in the number of abstracts containing p values. There were average absolute year-on-year increases of 1.2% (95% CI 1.1%-1.3%; p < 0.001), 0.93% (95% CI 0.75%-1.1%; p < 0.001), 0.70% (95% CI 0.57%-0.83%; p < 0.001), and 0.35% (95% CI 0.095%-0.60%; p = 0.0091) of abstracts reporting p values in JNNP, JNS, NS, and JNT, respectively. There have also been average year-on-year increases of 0.045 (95% CI 0.031-0.059; p < 0.001), 0.052 (95% CI 0.037-0.066; p < 0.001), 0.042 (95% CI 0.030-0.054; p < 0.001), and 0.041 (95% CI 0.026-0.056; p < 0.001) p values reported per abstract for these respective journals. The distribution of p values showed a positive skew and strong clustering of values at rounded decimals (i.e., 0.01, 0.02, etc.). Between 83.2% and 89.8% of all reported p values were at or below the "significance" threshold of 0.05 (i.e., p ≤ 0.05).

CONCLUSIONS

Trends in reporting of p values and the distribution of p values suggest publication bias remains in the neurosurgical literature.

Methodological and Clinical Heterogeneity and Extraction Errors in Meta-Analyses of Catheter Ablation for Atrial Fibrillation in Heart Failure

Background

Meta‐analyses are expected to follow a standardized process, and thus, they have become highly formulaic, although there is little evidence that such regimentation yields high‐quality results.

Methods and Results

This article describes the results of a critical examination of 14 published meta‐analyses of catheter ablation for atrial fibrillation in heart failure that were based on a nearly identical core set of 4 to 6 primary trials. Methodological issues included (1) the neglect of primary data or the failure to report any primary data; (2) the inaccurate recording of the number of randomized patients; (3) the lack of attention to data missingness or baseline imbalances; (4) the failure to contact investigators of primary trials for additional data; (5) the incorrect extraction of data, the misidentification of events, and the assignment of events to the wrong treatment groups; (6) the calculation of summary estimates based on demonstrably heterogenous data, methods of differing reliability, or unrelated end points; and (7) the development of conclusions based on sparse numbers of events or overly reliant on the results of 1 dominant trial.

Conclusions

These findings reinforce existing concerns about the methodological validity of meta‐analyses and their current status in the hierarchy of medical evidence, and they raise new questions about the process by which meta‐analyses undergo peer review by medical journals.

Reporting Quality of Systematic Review Abstracts Published in Leading Neurosurgical Journals: A Research on Research Study

BACKGROUND

Systematic review (SR) abstracts are frequently relied upon to guide clinical decision-making. However, there is mounting evidence that the quality of abstract reporting in the medical literature is suboptimal.

OBJECTIVE

To appraise SR abstract reporting quality in neurosurgical journals and identify factors associated with improved reporting.

METHODS

This study systematically surveyed SR abstracts published in 8 leading neurosurgical journals between 8 April 2007 and 21 August 2017. Abstracts were identified through a search of the MEDLINE database and their reporting quality was determined in duplicate using a tool derived from the Preferred Reporting Items for Systematic Reviews and Meta-analyses for Abstracts (PRISMA-A) statement. All SR abstracts that provided comparison between treatment strategies were eligible for inclusion. Descriptive statistics were utilized to identify factors associated with improved reporting.

RESULTS

A total of 257 abstracts were included in the analysis, with a mean of 22.8 (±25.3) included studies. The overall quality of reporting in included abstracts was suboptimal, with a mean score of 53.05% (±11.18). Reporting scores were higher among abstracts published after the release of the PRISMA-A guidelines (M = 56.52; 21.74-73.91) compared with those published beforehand (M = 47.83; 8.70-69.57; U = 4346.00, z = -4.61, P < .001). Similarly, both word count (r = 0.338, P < .001) and journal impact factor (r = 0.199, P = .001) were associated with an improved reporting score.

CONCLUSION

This study demonstrates that the overall reporting quality of abstracts in leading neurosurgical journals requires improvement. Strengths include the large number abstracts assessed, and its weaknesses include the fact that only neurosurgery-specific journals were surveyed. We recommend that attention be turned toward strengthening abstract submission and peer-review processes.

Effectiveness of perioperative antiepileptic drug prophylaxis for early and late seizures following oncologic neurosurgery: a meta-analysis

OBJECTIVE

The purpose of this meta-analysis was to evaluate the impact of perioperative antiepileptic drug (AED) prophylaxis on short- and long-term seizure incidence among patients undergoing brain tumor surgery. It is the first meta-analysis to focus exclusively on perioperative AED prophylaxis among patients undergoing brain tumor surgery.

METHODS

The authors searched PubMed/MEDLINE, Embase, Cochrane Central Register of Controlled Trials, clinicaltrials.gov, and the System for Information on Gray Literature in Europe for records related to perioperative AED prophylaxis for patients with brain tumors. Risk of bias in the included studies was assessed using the Cochrane risk of bias tool. Incidence rates for early seizures (within the first postoperative week) and total seizures were estimated based on data from randomized controlled trials. A Mantel-Haenszel random-effects model was used to analyze pooled relative risk (RR) of early seizures (within the first postoperative week) and total seizures associated with perioperative AED prophylaxis versus control.

RESULTS

Four RCTs involving 352 patients met the criteria of inclusion. The results demonstrated that perioperative AED prophylaxis for patients undergoing brain tumor surgery provides a statistically significant reduction in risk of early postoperative seizures compared with control (RR = 0.352, 95% confidence interval 0.130-0.949, p = 0.039). AED prophylaxis had no statistically significant effect on the total (combined short- and long-term) incidence of seizures.

CONCLUSIONS

This meta-analysis demonstrates for the first time that perioperative AED prophylaxis for brain tumor surgery provides a statistically significant reduction in early postoperative seizure risk.

Evaluations of the uptake and impact of the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) Statement and extensions: a scoping review

BACKGROUND

The PRISMA Statement is a reporting guideline designed to improve transparency of systematic reviews (SRs) and meta-analyses. Seven extensions to the PRISMA Statement have been published to address the reporting of different types or aspects of SRs, and another eight are in development. We performed a scoping review to map the research that has been conducted to evaluate the uptake and impact of the PRISMA Statement and extensions. We also synthesised studies evaluating how well SRs published after the PRISMA Statement was disseminated adhere to its recommendations.

METHODS

We searched for meta-research studies indexed in MEDLINE® from inception to 31 July 2017, which investigated some component of the PRISMA Statement or extensions (e.g. SR adherence to PRISMA, journal endorsement of PRISMA). One author screened all records and classified the types of evidence available in the studies. We pooled data on SR adherence to individual PRISMA items across all SRs in the included studies and across SRs published after 2009 (the year PRISMA was disseminated).

RESULTS

We included 100 meta-research studies. The most common type of evidence available was data on SR adherence to the PRISMA Statement, which has been evaluated in 57 studies that have assessed 6487 SRs. The pooled results of these studies suggest that reporting of many items in the PRISMA Statement is suboptimal, even in the 2382 SRs published after 2009 (where nine items were adhered to by fewer than 67% of SRs). Few meta-research studies have evaluated the adherence of SRs to the PRISMA extensions or strategies to increase adherence to the PRISMA Statement and extensions.

CONCLUSIONS

Many studies have evaluated how well SRs adhere to the PRISMA Statement, and the pooled result of these suggest that reporting of many items is suboptimal. An update of the PRISMA Statement, along with a toolkit of strategies to help journals endorse and implement the updated guideline, may improve the transparency of SRs.

Reporting and methodologic evaluation of meta-analyses published in the anesthesia literature according to AMSTAR and PRISMA checklists: a preliminary study

BACKGROUND

There have been few recent reports on the methodological quality of meta-analysis, despite the enormous number of studies using meta-analytic techniques in the field of anesthesia. The purpose of this study was to evaluate the quality of meta-analyses and systematic reviews according to the Assessment of Multiple Systematic Reviews (AMSTAR) and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines in the anesthesia literature.

METHODS

A search was conducted to identify all meta-analyses ever been published in the British Journal of Anaesthesia (BJA), Anaesthesia, and Korean Journal of Anesthesiology (KJA) between Jan. 01, 2004 and Nov. 31, 2016. We aimed to apply the AMSTAR and PRISMA checklists to all published meta-analyses.

RESULTS

We identified 121 meta-analyses in the anesthesia literature from January 2004 through the end of November 2016 (BJA; 75, Anaesthesia; 43, KJA; 3). The number of studies published and percentage of 'Yes' responses for meta-analysis articles published after the year 2010 was significantly increased compared to that of studies published before the year 2009 (P = 0.014 for Anaesthesia). In the anesthesia literature as a whole, participation of statisticians as authors statistically improved average scores of PRISMA items (P = 0.004) especially in the BJA (P = 0.003).

CONCLUSIONS

Even though there is little variability in the reporting and methodology of meta-analysis in the anesthesia literature, significant quality improvement in the reporting was observed in the Anaesthesia by applying the PRISMA checklist. Participation of a statistician as an author improved the reporting quality of the meta-analysis.

Compliance of systematic reviews articles in brain arteriovenous malformation with PRISMA statement guidelines: Review of literature

BACKGROUND

The knowledge of reporting compliance of systematic reviews with PRISMA guidelines may assist in improving the quality of secondary research in brain AVM management and subsequently application to patient population and clinical practice. This may allow researchers and clinicians to be equipped to appraise existing literatures based on known deficit to look for or expect. The objective of this study was to assess the compliance of systematic reviews and meta-analysis in the management of brain AVM.

METHODS

Systematic reviews and meta-analyses articles published in medical journals between 1st of May 2011 and 30th April 2016 (five-year period) were examined. Exclusion criteria were articles that were not systematic reviews and not meta-analyses, narrative literature reviews, historical literature reviews, animal studies, unpublished articles, commentaries and letter to the editor. Electronic database search performed through Medline PubMed on 20th September 2016.

RESULTS

This systematic review examined seven systematic review articles on intracranial arteriovenous malformation compliance with PRISMA statement guidelines. The mean percentage of applicable PRISMA items across all studies was 74% (range 67-93%). Protocol registration and declaration, risk of bias and funding sources were the most poorly reported of the PRISMA items (14% each). A significant variance in the total percentages was evident between studies (67-93%).

CONCLUSION

Systematic review reporting in medical literature is excessively variable and overall poor. As these papers are being published with increasing frequency, need to fully adhere to PRISMA statement guide for systematic review to ensure high-quality publications. Complete reporting of PRISMA items within systematic reviews in cerebral arteriovenous malformation enhance quality assessment, robust critical appraisal, better judgement and ultimately sound application to practice thereby improving research standards and patients care.

Longitudinal analysis of reporting and quality of systematic reviews in high-impact surgical journals

BACKGROUND

The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) Statement aims to optimize the reporting of systematic reviews. The performance of the PRISMA Statement in improving the reporting and quality of surgical systematic reviews remains unclear.

METHODS

Systematic reviews published in five high-impact surgical journals between 2007 and 2015 were identified from online archives. Manuscripts blinded to journal, publication year and authorship were assessed according to 27 reporting criteria described by the PRISMA Statement and scored using a validated quality appraisal tool (AMSTAR, Assessing the Methodological Quality of Systematic Reviews). Comparisons were made between studies published before (2007-2009) and after (2011-2015) its introduction. The relationship between reporting and study quality was measured using Spearman's rank test.

RESULTS

Of 281 eligible manuscripts, 80 were published before the PRISMA Statement and 201 afterwards. Most manuscripts (208) included a meta-analysis, with the remainder comprising a systematic review only. There was no meaningful change in median compliance with the PRISMA Statement (19 (i.q.r. 16-21) of 27 items before versus 19 (17-22) of 27 after introduction of PRISMA) despite achieving statistical significance (P = 0·042). Better reporting compliance was associated with higher methodological quality (r_s  = 0·70, P < 0·001).

CONCLUSION

The PRISMA Statement has had minimal impact on the reporting of surgical systematic reviews. Better compliance was associated with higher-quality methodology.

Reproducibility of Search Strategies Is Poor in Systematic Reviews Published in High-Impact Pediatrics, Cardiology and Surgery Journals: A Cross-Sectional Study

Background

A high-quality search strategy is considered an essential component of systematic reviews but many do not contain reproducible search strategies. It is unclear if low reproducibility spans medical disciplines, is affected by librarian/search specialist involvement or has improved with increased awareness of reporting guidelines.

Objectives

To examine the reporting of search strategies in systematic reviews published in Pediatrics, Surgery or Cardiology journals in 2012 and determine rates and predictors of including a reproducible search strategy.

Methods

We identified all systematic reviews published in 2012 in the ten highest impact factor journals in Pediatrics, Surgery and Cardiology. Each search strategy was coded to indicate what elements were reported and whether the overall search was reproducible. Reporting and reproducibility rates were compared across disciplines and we measured the influence of librarian/search specialist involvement, discipline or endorsement of a reporting guideline on search reproducibility.

Results

272 articles from 25 journals were included. Reporting of search elements ranged widely from 91% of articles naming search terms to 33% providing a full search strategy and 22% indicating the date the search was executed. Only 22% of articles provided at least one reproducible search strategy and 13% provided a reproducible strategy for all databases searched in the article. Librarians or search specialists were reported as involved in 17% of articles. There were strong disciplinary differences on the reporting of search elements. In the multivariable analysis, only discipline (Pediatrics) was a significant predictor of the inclusion of a reproducible search strategy.

Conclusions

Despite recommendations to report full, reproducible search strategies, many articles still do not. In addition, authors often report a single strategy as covering all databases searched, further decreasing reproducibility. Further research is needed to determine how disciplinary culture may encourage reproducibility and the role that journal editors and peer reviewers could play.

Is the Best Evidence Good Enough: Quality Assessment and Factor Analysis of Meta-Analyses on Depression

Background

The quality of meta-analyses (MAs) on depression remains uninvestigated.

Objective

To assess the overall reporting and methodological qualities of MAs on depression and to explore potential factors influencing both qualities.

Methods

MAs investigating epidemiology and interventions for depression published in the most recent year (2014–2015) were selected from PubMed, EMBASE, PsycINFO and Cochrane Library. The characteristics of the included studies were collected and the total and per-item quality scores of the included studies were calculated based on the two checklists. Univariate and multivariate linear regression analyses were used to explore the potential factors influencing the quality of the articles.

Results

A total of 217 MAs from 74 peer-reviewed journals were included. The mean score of Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) was 23.0 of 27 and mean score of Assessment of Multiple Systematic Reviews (AMSTAR) was 8.3 of 11. Items assessing registration and protocol (14.2%, 37/217) in PRISMA and item requiring a full list of included and excluded studies (16.1%, 40/217) in AMSTAR had poorer adherences than other items. The MAs that included only RCTs, pre-registered, had five more authors or authors from Cochrane groups and the MAs found negative results had better reporting and methodological qualities.

Conclusions

The reporting and methodological qualities of MAs on depression remained to be improved. Design of included studies, characteristics of authors and pre-registration in PROSPERO database are important factors influencing quality of MAs in the field of depression.

Limitations of A Measurement Tool to Assess Systematic Reviews (AMSTAR) and suggestions for improvement

A Measurement Tool to Assess Systematic Reviews (AMSTAR) is a commonly used tool to assess the quality of systematic reviews; however, modifications are needed to improve its usability, reliability, and validity. In this commentary, we summarize our experience and the experiences of others who have used AMSTAR and provide suggestions for its improvement. We propose that AMSTAR should modify a number of individual items and their instructions and responses to make them more congruent with an assessment of the methodologic quality of systematic reviews. We recommend adding new items and modifying existing items to assess the quality of the body of evidence and to address subgroup and sensitivity analyses. More detailed instructions are needed for scoring individual items across multiple reviewers, and we recommend that a total score should not be calculated. These suggestions need to be empirically tested prior to implementation.

Controversies in epidemiology of intracranial aneurysms and SAH

Rupture of an intracranial aneurysm is the most common cause of subarachnoid haemorrhage (SAH), which is a life-threatening acute cerebrovascular event that typically affects working-age people. The exact prevalence of unruptured intracranial aneurysms (UIAs) is unknown, but at least one in 20 to 30 adults is likely to carry an asymptomatic UIA. Approximately one quarter of these UIAs rupture in a lifetime. Complex methodological challenges in conducting studies of epidemiology and risk factors for UIAs and SAH might have led to conclusions being drawn on the basis of epidemiological data of variable quality. We believe that, as a result, misconceptions about UIAs and SAH may have arisen. In this Perspectives article, we discuss three possible misconceptions about the epidemiology of UIAs and SAH, and suggest how the quality of future research could be improved.

Methodological quality assessment of paper-based systematic reviews published in oral health

OBJECTIVES

This study aimed to conduct a methodological assessment of paper-based systematic reviews (SR) published in oral health using a validated checklist. A secondary objective was to explore temporal trends on methodological quality.

MATERIAL AND METHODS

Two electronic databases (OVID Medline and OVID EMBASE) were searched for paper-based SR of interventions published in oral health from inception to October 2014. Manual searches of the reference lists of paper-based SR were also conducted. Methodological quality of included paper-based SR was assessed using an 11-item questionnaire, Assessment of Multiple Systematic Reviews (AMSTAR) checklist. Methodological quality was summarized using the median and inter-quartile range (IQR) of the AMSTAR score over different categories and time periods.

RESULTS

A total of 643 paper-based SR were included. The overall median AMSTAR score was 4 (IQR 2-6). The highest median score (5) was found in the pain dentistry and periodontology fields, while the lowest median score (3) was found in implant dentistry, restorative dentistry, oral medicine, and prosthodontics. The number of paper-based SR per year and the median AMSTAR score increased over time (median score in 1990s was 2 (IQR 2-3), 2000s was 4 (IQR 2-5), and 2010 onwards was 5 (IQR 3-6)).

CONCLUSION

Although the methodological quality of paper-based SR published in oral health has improved in the last few years, there is still scope for improving quality in most evaluated dental specialties.

CLINICAL RELEVANCE

Large-scale assessment of methodological quality of dental SR highlights areas of methodological strengths and weaknesses that can be targeted in future publications to encourage better quality review methodology.

Strategies to improve the credibility of meta-analyses in spine surgery: a systematic survey

BACKGROUND CONTEXT

Meta-analyses are powerful tools that can synthesize existing research, inform clinical practice, and support evidence-based care. These studies have become increasingly popular in the spine surgery literature, but the rigor with which they are being conducted has not yet been evaluated.

PURPOSE

Our primary objectives were to evaluate the methodological quality (credibility) of spine surgery meta-analyses and to propose strategies to improve future research. Our secondary objectives were to evaluate completeness of reporting and identify factors associated with higher credibility and completeness of reporting.

STUDY DESIGN

This study is based on a systematic survey of meta-analyses.

OUTCOME MEASURES

We evaluated credibility according to the Users' Guide to the Medical Literature and completeness of reporting according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses checklist.

METHODS

We systematically searched MEDLINE, EMBASE, and The Cochrane Library, and two reviewers independently assessed eligibility, credibility, and completeness of reporting. We used multivariable linear regression to evaluate potential associations. Interrater agreement was quantified using kappa and intraclass correlation (ICC) coefficients.

RESULTS

We identified 132 eligible meta-analyses of spine surgery interventions. The mean credibility score was 3 of 7 (standard deviation [SD], 1.4; ICC, 0.86), with agreement for each item ranging from 0.54 (moderate) to 0.83 (almost perfect). Clinical questions were judged as sensible in 125 (95%), searches were exhaustive in 102 (77%), and risk of bias assessments were undertaken in 91 (69%). Seven (5%) meta-analyses addressed possible explanations for heterogeneity using a priori subgroup hypotheses and 24 (18%) presented results that were immediately clinically applicable. Investigators undertook duplicate assessments of eligibility, risk of bias, and data extraction in 46 (35%) and rated overall confidence in the evidence in 24 (18%). Later publication year, increasing Journal Impact Factor, increasing number of databases, inclusion of Randomized Controlled Trials, and inclusion of non-English studies were significantly associated with higher credibility scores (p<.05). The mean score for reporting was 18 of 27 (SD, 4.4; ICC, 0.94).

CONCLUSIONS

The credibility of many current spine surgery meta-analyses is limited. Researchers can improve future meta-analyses by performing exhaustive literature searches, addressing possible explanations of heterogeneity, presenting results in a clinically useful manner, reproducibly selecting and assessing primary studies, addressing confidence in the pooled effect estimates, and adhering to guidelines for complete reporting.

Use of recommended search strategies in systematic reviews and the impact of librarian involvement: a cross-sectional survey of recent authors

Background

Previous research looking at published systematic reviews has shown that their search strategies are often suboptimal and that librarian involvement, though recommended, is low. Confidence in the results, however, is limited due to poor reporting of search strategies the published articles.

Objectives

To more accurately measure the use of recommended search methods in systematic reviews, the levels of librarian involvement, and whether librarian involvement predicts the use of recommended methods.

Methods

A survey was sent to all authors of English-language systematic reviews indexed in the Database of Abstracts of Reviews of Effects (DARE) from January 2012 through January 2014. The survey asked about their use of search methods recommended by the Institute of Medicine, Cochrane Collaboration, and the Agency for Healthcare Research and Quality and if and how a librarian was involved in the systematic review. Rates of use of recommended methods and librarian involvement were summarized. The impact of librarian involvement on use of recommended methods was examined using a multivariate logistic regression.

Results

1560 authors completed the survey. Use of recommended search methods ranged widely from 98% for use of keywords to 9% for registration in PROSPERO and were generally higher than in previous studies. 51% of studies involved a librarian, but only 64% acknowledge their assistance. Librarian involvement was significantly associated with the use of 65% of recommended search methods after controlling for other potential predictors. Odds ratios ranged from 1.36 (95% CI 1.06 to 1.75) for including multiple languages to 3.07 (95% CI 2.06 to 4.58) for using controlled vocabulary.

Conclusions

Use of recommended search strategies is higher than previously reported, but many methods are still under-utilized. Librarian involvement predicts the use of most methods, but their involvement is under-reported within the published article.

A comprehensive assessment of the risk of bone morphogenetic protein use in spinal fusion surgery and postoperative cancer diagnosis

The risk of postoperative cancer following the use of recombinant human bone morphogenetic protein (BMP)-2 in spinal fusion is one potential complication that has received significant interest. Until recently, there has been little clinical evidence to support the assertion of potential cancer induction after BMP use in spinal surgery. This report aims to summarize the findings from clinical data available to date from the Yale University Open Data Access (YODA) project as well as more recently published large database studies regarding the association of BMP use in spinal fusion and the risk of postoperative cancer. A detailed review was based on online databases, primary studies, FDA reports, and bibliographies of key articles for studies that assessed the efficacy and safety of BMP in spinal fusion. In an analysis of the YODA project, one meta-analysis detected a statistically significant increase in cancer occurrence at 24 months but not at 48 months, and the other meta-analysis did not detect a significant increase in postoperative cancer occurrence. Analysis of 3 large health care data sets (Medicare, MarketScan, and PearlDiver) revealed that none were able to detect a significant increase in risk of malignant cancers when BMP was used compared with controls. The potential risk of postoperative cancer formation following the use of BMP in spinal fusion must be interpreted on an individual basis for each patient by the surgeon. There is no conclusive evidence that application of the common formulations of BMP during spinal surgery results in the formation of cancer locally or at a distant site.

Bleeding and infection with external ventricular drainage: a systematic review in comparison with adjudicated adverse events in the ongoing Clot Lysis Evaluating Accelerated Resolution of Intraventricular Hemorrhage Phase III (CLEAR-III IHV) trial

BACKGROUND

Retrospective series report varied rates of bleeding and infection with external ventricular drainage (EVD). There have been no prospective studies of these risks with systematic surveillance, threshold definitions, or independent adjudication.

OBJECTIVE

To analyze the rate of complications in the ongoing Clot Lysis: Evaluating Accelerated Resolution of Intraventricular Hemorrhage Phase III (CLEAR III) trial, providing a comparison with a systematic review of complications of EVD in the literature.

METHODS

Patients were prospectively enrolled in the CLEAR III trial after placement of an EVD for obstructive intraventricular hemorrhage and randomized to receive recombinant tissue-type plasminogen activator or placebo. We counted any detected new hemorrhage (catheter tract hemorrhage or any other distant hemorrhage) on computed tomography scan within 30 days from the randomization. Meta-analysis of published series of EVD placement was compiled with STATA software.

RESULTS

Growing or unstable hemorrhage was reported as a cause of exclusion from the trial in 74 of 5707 cases (1.3%) screened for CLEAR III. The first 250 patients enrolled have completed adjudication of adverse events. Forty-two subjects (16.8%) experienced ≥1 new bleeds or expansions, and 6 of 250 subjects (2.4%) suffered symptomatic hemorrhages. Eleven cases (4.4%) had culture-proven bacterial meningitis or ventriculitis.

CONCLUSION

Risks of bleeding and infection in the ongoing CLEAR III trial are comparable to those previously reported in EVD case series. In the present study, rates of new bleeds and bacterial meningitis/ventriculitis are very low despite multiple daily injections, blood in the ventricles, the use of thrombolysis in half the cases, and generalization to >60 trial sites.

Librarian co-authors correlated with higher quality reported search strategies in general internal medicine systematic reviews

OBJECTIVES

To determine whether librarian and information specialist authorship was associated with better reported systematic review (SR) search quality.

STUDY DESIGN AND SETTING

SRs from high-impact general internal medicine journals were reviewed for search quality characteristics and reporting quality by independent reviewers using three instruments, including a checklist of Institute of Medicine Recommended Standards for the Search Process and a scored modification of the Peer Review of Electronic Search Strategies instrument.

RESULTS

The level of librarian and information specialist participation was significantly associated with search reproducibility from reported search strategies (Χ(2) = 23.5; P < 0.0001). Librarian co-authored SRs had significantly higher odds of meeting 8 of 13 analyzed search standards than those with no librarian participation and six more than those with mentioned librarian participation. One-way ANOVA showed that differences in total search quality scores between all three groups were statistically significant (F2,267 = 10.1233; P < 0.0001).

CONCLUSION

Problems remain with SR search quality and reporting. SRs with librarian or information specialist co-authors are correlated with significantly higher quality reported search strategies. To minimize bias in SRs, authors and editors could encourage librarian engagement in SRs including authorship as a potential way to help improve documentation of the search strategy.

Cost-effectiveness analysis in minimally invasive spine surgery

OBJECT

Medical care has been evolving with the increased influence of a value-based health care system. As a result, more emphasis is being placed on ensuring cost-effectiveness and utility in the services provided to patients. This study looks at this development in respect to minimally invasive spine surgery (MISS) costs.

METHODS

A literature review using PubMed, the Cost-Effectiveness Analysis (CEA) Registry, and the National Health Service Economic Evaluation Database (NHS EED) was performed. Papers were included in the study if they reported costs associated with minimally invasive spine surgery (MISS). If there was no mention of cost, CEA, cost-utility analysis (CUA), quality-adjusted life year (QALY), quality, or outcomes mentioned, then the article was excluded.

RESULTS

Fourteen studies reporting costs associated with MISS in 12,425 patients (3675 undergoing minimally invasive procedures and 8750 undergoing open procedures) were identified through PubMed, the CEA Registry, and NHS EED. The percent cost difference between minimally invasive and open approaches ranged from 2.54% to 33.68%-all indicating cost saving with a minimally invasive surgical approach. Average length of stay (LOS) for minimally invasive surgery ranged from 0.93 days to 5.1 days compared with 1.53 days to 12 days for an open approach. All studies reporting EBL reported lower volume loss in an MISS approach (range 10-392.5 ml) than in an open approach (range 55-535.5 ml).

CONCLUSIONS

There are currently an insufficient number of studies published reporting the costs of MISS. Of the studies published, none have followed a standardized method of reporting and analyzing cost data. Preliminary findings analyzing the 14 studies showed both cost saving and better outcomes in MISS compared with an open approach. However, more Level I CEA/CUA studies including cost/QALY evaluations with specifics of the techniques utilized need to be reported in a standardized manner to make more accurate conclusions on the cost effectiveness of minimally invasive spine surgery.

Case-control studies in neurosurgery

OBJECT

Observational studies, such as cohort and case-control studies, are valuable instruments in evidence-based medicine. Case-control studies, in particular, are becoming increasingly popular in the neurosurgical literature due to their low cost and relative ease of execution; however, no one has yet systematically assessed these types of studies for quality in methodology and reporting.

METHODS

The authors performed a literature search using PubMed/MEDLINE to identify all studies that explicitly identified themselves as "case-control" and were published in the JNS Publishing Group journals (Journal of Neurosurgery, Journal of Neurosurgery: Pediatrics, Journal of Neurosurgery: Spine, and Neurosurgical Focus) or Neurosurgery. Each paper was evaluated for 22 descriptive variables and then categorized as having either met or missed the basic definition of a case-control study. All studies that evaluated risk factors for a well-defined outcome were considered true case-control studies. The authors sought to identify key features or phrases that were or were not predictive of a true case-control study. Those papers that satisfied the definition were further evaluated using the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) checklist.

RESULTS

The search detected 67 papers that met the inclusion criteria, of which 32 (48%) represented true case-control studies. The frequency of true case-control studies has not changed with time. Use of odds ratios (ORs) and logistic regression (LR) analysis were strong positive predictors of true case-control studies (for odds ratios, OR 15.33 and 95% CI 4.52-51.97; for logistic regression analysis, OR 8.77 and 95% CI 2.69-28.56). Conversely, negative predictors included focus on a procedure/intervention (OR 0.35, 95% CI 0.13-0.998) and use of the word "outcome" in the Results section (OR 0.23, 95% CI 0.082-0.65). After exclusion of nested case-control studies, the negative correlation between focus on a procedure/intervention and true case-control studies was strengthened (OR 0.053, 95% CI 0.0064-0.44). There was a trend toward a negative association between the use of survival analysis or Kaplan-Meier curves and true case-control studies (OR 0.13, 95% CI 0.015-1.12). True case-control studies were no more likely than their counterparts to use a potential study design "expert" (OR 1.50, 95% CI 0.57-3.95). The overall average STROBE score was 72% (range 50-86%). Examples of reporting deficiencies were reporting of bias (28%), missing data (55%), and funding (44%).

CONCLUSIONS

The results of this analysis show that the majority of studies in the neurosurgical literature that identify themselves as "case-control" studies are, in fact, labeled incorrectly. Positive and negative predictors were identified. The authors provide several recommendations that may reverse the incorrect and inappropriate use of the term "case-control" and improve the quality of design and reporting of true case-control studies in neurosurgery.