Flaws in the application and interpretation of statistical analyses in systematic reviews of therapeutic interventions were common: a cross-sectional analysis

The PRISMA 2020 statement: An updated guideline for reporting systematic reviews

The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement, published in 2009, was designed to help systematic reviewers transparently report why the review was done, what the authors did, and what they found. Over the past decade, advances in systematic review methodology and terminology have necessitated an update to the guideline. The PRISMA 2020 statement replaces the 2009 statement and includes new reporting guidance that reflects advances in methods to identify, select, appraise, and synthesise studies. The structure and presentation of the items have been modified to facilitate implementation. In this article, we present the PRISMA 2020 27-item checklist, an expanded checklist that details reporting recommendations for each item, the PRISMA 2020 abstract checklist, and the revised flow diagrams for original and updated reviews.

Characteristics and quality of systematic reviews and meta-analyses of observational nutritional epidemiology: a cross-sectional study

BACKGROUND

Dietary recommendations and policies should be guided by rigorous systematic reviews. Reviews that are of poor methodological quality may be ineffective or misleading. Most of the evidence in nutrition comes from nonrandomized studies of nutritional exposures (usually referred to as nutritional epidemiology studies), but to date methodological evaluations of the quality of systematic reviews of such studies have been sparse and inconsistent.

OBJECTIVES

We aimed to investigate the quality of recently published systematic reviews and meta-analyses of nutritional epidemiology studies and to propose guidance addressing major limitations.

METHODS

We searched MEDLINE (January 2018-August 2019), EMBASE (January 2018-August 2019), and the Cochrane Database of Systematic Reviews (January 2018-February 2019) for systematic reviews of nutritional epidemiology studies. We included a random sample of 150 reviews.

RESULTS

Most reviews were published by authors from Asia (n = 49; 32.7%) or Europe (n = 43; 28.7%) and investigated foods or beverages (n = 60; 40.0%) and cancer morbidity and mortality (n = 54; 36%). Reviews often had important limitations: less than one-quarter (n = 30; 20.0%) reported preregistration of a protocol and almost one-third (n = 42; 28.0%) did not report a replicable search strategy. Suboptimal practices and errors in the synthesis of results were common: one-quarter of meta-analyses (n = 30; 26.1%) selected the meta-analytic model based on statistical indicators of heterogeneity and almost half of meta-analyses (n = 50; 43.5%) did not consider dose-response associations even when it was appropriate to do so. Only 16 (10.7%) reviews used an established system to evaluate the certainty of evidence.

CONCLUSIONS

Systematic reviews of nutritional epidemiology studies often have serious limitations. Authors can improve future reviews by involving statisticians, methodologists, and researchers with substantive knowledge in the specific area of nutrition being studied and using a rigorous and transparent system to evaluate the certainty of evidence.

The PRISMA 2020 statement: An updated guideline for reporting systematic reviews

Matthew Page and co-authors describe PRISMA 2020, an updated reporting guideline for systematic reviews and meta-analyses.

Updating guidance for reporting systematic reviews: development of the PRISMA 2020 statement

OBJECTIVES

To describe the processes used to update the PRISMA 2009 statement for reporting systematic reviews, present results of a survey conducted to inform the update, summarize decisions made at the PRISMA update meeting, and describe and justify changes made to the guideline.

METHODS

We reviewed 60 documents with reporting guidance for systematic reviews to generate suggested modifications to the PRISMA 2009 statement. We invited 220 systematic review methodologists and journal editors to complete a survey about the suggested modifications. The results of these projects were discussed at a 21-member in-person meeting. Following the meeting, we drafted the PRISMA 2020 statement and refined it based on feedback from co-authors and a convenience sample of 15 systematic reviewers.

RESULTS

The review of 60 documents revealed that all topics addressed by the PRISMA 2009 statement could be modified. Of the 110 survey respondents, more than 66% recommended keeping six of the original checklist items as they were and modifying 15 of them using wording suggested by us. Attendees at the in-person meeting supported the revised wording for several items but suggested rewording for most to enhance clarity, and further refinements were made over six drafts of the guideline.

CONCLUSIONS

The PRISMA 2020 statement consists of updated reporting guidance for systematic reviews. We hope that providing this detailed description of the development process will enhance the acceptance and uptake of the guideline and assist those developing and updating future reporting guidelines.

Doug Altman: Driving critical appraisal and improvements in the quality of methodological and medical research

Doug Altman was a visionary leader and one of the most influential medical statisticians of the last 40 years. Based on a presentation in the "Invited session in memory of Doug Altman" at the 40th Annual Conference of the International Society for Clinical Biostatistics (ISCB) in Leuven, Belgium and our long-standing collaborations with Doug, we discuss his contributions to regression modeling, reporting, prognosis research, as well as some more general issues while acknowledging that we cannot cover the whole spectrum of Doug's considerable methodological output. His statement "To maximize the benefit to society, you need to not just do research but do it well" should be a driver for all researchers. To improve current and future research, we aim to summarize Doug's messages for these three topics.

Gabapentin for phantom limb pain after amputation in pediatric oncology: a systematic review protocol

BACKGROUND

Phantom limb pain (PLP) is a prevalent problem for children after amputation because of the chemotherapy treatment. Gabapentin is a potential option to manage PLP after amputation in pediatric oncology. However, no systematic review specifically investigated this topic. Thus, this study aims to appraise the efficacy and safety of gabapentin for post-amputation PLP in pediatric oncology.

METHODS

Electronic databases (Cochrane Library, MEDLINE, EMBASE, Web of Science, CINAHL, PsychINFO, Scopus, WANGFANG, and Chinese Biomedical Literature Database) will be systematically searched from the beginning to the present without limitations to publication status and language. Primary outcome is pain intensity. Secondary outcomes are analgesic drug consumption, sleep quality, depression, anxiety, health-related quality of life, and adverse events. The treatment effect of all dichotomous outcome data will be estimated as risk ratio and 95% confidence intervals (CIs) and that of continuous outcome data will be calculated as mean difference or standardized mean difference and 95% CIs. Methodological quality of randomized controlled trials (RCTs) will be assessed using Cochrane risk of bias tool and that of case-controlled studies (CCSs) will be appraised using Newcastle-Ottawa Tool. Statistical analysis will be conducted using RevMan 5.3 software.

DISCUSSION

This study will summarize up-to-date high-quality RCTs and CCSs to assess the efficacy and safety of gabapentin for PLP after amputation in pediatric oncology. The findings of this study will help to determine whether or not gabapentin is effective and safe for children with PLP after amputation.

SYSTEMATIC REVIEW REGISTRATION

INPLASY202060090.

Association of working conditions including digital technology use and systemic inflammation among employees: study protocol for a systematic review

BACKGROUND

With the dynamic advancement of digitalization, working environments are changing and risk for employee stress may be increasing. Work stress has been associated with a dysregulation of inflammatory processes as a component of immune function. Systemic low-grade inflammation is discussed as a key player in the relation between stress exposure and chronic illness, such as cardiovascular diseases. The objective of this investigation will be to evaluate the association of working conditions including digital technology use and systemic inflammation among employees.

METHODS

We designed and registered a study protocol for a systematic review of randomized controlled trials and prospective non-randomized studies (e.g., cohort, interrupted time series, or before-after studies). We will include studies conducted among adult workers reporting associations of working conditions and inflammatory activity. The outcome will be biomarkers of systemic low-grade inflammation on cell, plasma molecule and intracellular level, such as C-reactive protein, or different types of leukocytes, cytokines, etc. Literature searches will be conducted in several electronic databases (from January 1982 onwards), including PubMed/MEDLINE, Embase, PsycINFO, Web of Science, and CENTRAL. Two reviewers will independently screen all retrieved records, full-text articles, and extract data. The study methodological quality (or bias) will be appraised using appropriate tools. Our results will be described qualitatively. Random effects meta-analysis will be conducted, if feasible and appropriate. Additional analyses will be performed to explore potential sources of heterogeneity.

DISCUSSION

This systematic review and meta-analysis will provide a synthesis of studies evaluating the association of working conditions and systemic inflammation. We anticipate our findings to identify knowledge gaps in the literature that future research should address. Moreover, results of our review may provide implications for corporate and public policy action for employee health promotion and prevention of occupational stress.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO ID: CRD42020166887.

Best-worst scaling identified adequate statistical methods and literature search as the most important items of AMSTAR2 (A measurement tool to assess systematic reviews)

OBJECTIVE

To assess the relative importance of A MeaSurement Tool to Assess systematic Reviews 2 (AMSTAR2) items.

STUDY DESIGN AND SETTING

A best-worst scaling object case was conducted among a sample of experts in the field of systematic reviews (SRs) and meta-analyses (MAs). Respondents were asked in a series of 15 choice tasks to choose the most and the least important item from a set of four items from the master list, which included the 16 AMSTAR2 items. Hierarchical Bayes analysis was used to generate the relative importance score for each item.

RESULTS

The most important items highlighted by our 242 experts to conduct overview of reviews and critically assess SRs/MAs were the appropriateness of statistical analyses and adequacy of the literature search, followed by items regarding the assessment of risk of bias, the research protocol, and the assessment of heterogeneity (relative importance score >6.5). Items related to funding sources and the assessment of study selection and data extraction in duplicate were rated as least important.

CONCLUSION

Although all AMSTAR2 items can be considered as important, our results highlighted the importance of keeping the two items (the appropriateness of statistical analyses and the adequacy of the literature search) among the critical items proposed by AMSTAR2 to critically appraise SRs/MAs.

Methodological quality of meta-analyses indexed in PsycINFO: leads for enhancements: a meta-epidemiological study

OBJECTIVES

Meta-analyses (MAs) are often used because they are lauded to provide robust evidence that synthesises information from multiple studies. However, the validity of MA conclusions relies on the procedural rigour applied by the authors. Therefore, this meta-research study aims to characterise the methodological quality and meta-analytic practices of MAs indexed in PsycINFO.

DESIGN

A meta-epidemiological study.

PARTICIPANTS

We evaluated a random sample of 206 MAs indexed in the PsycINFO database in 2016.

PRIMARY AND SECONDARY OUTCOMES

Two authors independently extracted the methodological characteristics of all MAs and checked their quality according to the 16 items of the A MeaSurement Tool to Assess systematic Reviews (AMSTAR2) tool for MA critical appraisal. Moreover, we investigated the effect of mentioning Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) on the methodological quality of MAs.

RESULTS

According to AMSTAR2 criteria, 95% of the 206 MAs were rated as critically low quality. Statistical methods were appropriate and publication bias was well evaluated in 87% and 70% of the MAs, respectively. However, much improvement is needed in data collection and analysis: only 11% of MAs published a research protocol, 44% had a comprehensive literature search strategy, 37% assessed and 29% interpreted the risk of bias in the individual included studies, and 11% presented a list of excluded studies. Interestingly, the explicit mentioning of PRISMA suggested a positive influence on the methodological quality of MAs.

CONCLUSION

The methodological quality of MAs in our sample was critically low according to the AMSTAR2 criteria. Some efforts to tremendously improve the methodological quality of MAs could increase their robustness and reliability.

Methodological quality of meta-analyses indexed in PsycINFO: leads for enhancements: a meta-epidemiological study

OBJECTIVES

Meta-analyses (MAs) are often used because they are lauded to provide robust evidence that synthesises information from multiple studies. However, the validity of MA conclusions relies on the procedural rigour applied by the authors. Therefore, this meta-research study aims to characterise the methodological quality and meta-analytic practices of MAs indexed in PsycINFO.

DESIGN

A meta-epidemiological study.

PARTICIPANTS

We evaluated a random sample of 206 MAs indexed in the PsycINFO database in 2016.

PRIMARY AND SECONDARY OUTCOMES

Two authors independently extracted the methodological characteristics of all MAs and checked their quality according to the 16 items of the A MeaSurement Tool to Assess systematic Reviews (AMSTAR2) tool for MA critical appraisal. Moreover, we investigated the effect of mentioning Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) on the methodological quality of MAs.

RESULTS

According to AMSTAR2 criteria, 95% of the 206 MAs were rated as critically low quality. Statistical methods were appropriate and publication bias was well evaluated in 87% and 70% of the MAs, respectively. However, much improvement is needed in data collection and analysis: only 11% of MAs published a research protocol, 44% had a comprehensive literature search strategy, 37% assessed and 29% interpreted the risk of bias in the individual included studies, and 11% presented a list of excluded studies. Interestingly, the explicit mentioning of PRISMA suggested a positive influence on the methodological quality of MAs.

CONCLUSION

The methodological quality of MAs in our sample was critically low according to the AMSTAR2 criteria. Some efforts to tremendously improve the methodological quality of MAs could increase their robustness and reliability.

Parenteral thiamine for prevention and treatment of delirium in critically ill adults: a systematic review protocol

BACKGROUND

Delirium is an acute confusional state, common in critical illness and associated with cognitive decline. There is no effective pharmacotherapy to prevent or treat delirium, although it is scientifically plausible that thiamine could be effective. Thiamine studies in dementia patients are inconclusive. Aside from small numbers, all used oral administration: bioavailability of thiamine is poor; parenteral thiamine bypasses this. In the UK, parenteral thiamine is administered as a compound vitamin B and C solution (Pabrinex®). The aim of this review is to evaluate the effectiveness of parenteral thiamine (alone or in a compound solution) in preventing or treating delirium in critical illness.

METHODS

We will search for studies in electronic databases (MEDLINE (Pro-Quest), EMBASE, CINAHL, LILACS, CNKI, AMED, and Cochrane CENTRAL), clinical trials registries (WHO International Clinical Trials Registry, ClinicalTrials.gov, and Controlled-trials.com), and grey literature (Google Scholar, conference proceedings, and Index to Theses). We will perform complementary searches of reference lists of included studies, relevant reviews, clinical practice guidelines, or other pertinent documents (e.g. official documents and government reports). We will consider quasi-randomised or randomised controlled trials in critically ill adults. We will include studies that evaluate parenteral thiamine versus standard of care, placebo, or any other non-pharmacological or pharmacological interventions. The primary outcomes will be the delirium core outcome set, including incidence and severity of delirium and cognition. Secondary outcomes are adapted from the ventilation core outcome set: duration of mechanical ventilation, length of stay, and adverse events incidence. Screening, data extraction, and risk of bias assessment will be undertaken independently by two reviewers. If data permits, we will conduct meta-analyses using a random effects model and, where appropriate, sensitivity and subgroup analyses to explore sources of heterogeneity.

DISCUSSION

This review will provide evidence for the effectiveness of parental thiamine in the prevention or treatment of delirium in critical care. Findings will contribute to establishing the need for a multicentre study of parenteral thiamine in the prevention and treatment of critical care delirium.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42019118808.

Low-level laser therapy for carpal tunnel syndrome: systematic review and network meta-analysis

BACKGROUND

Splinting is recommended by various organisations as a non-surgical first-line treatment for carpal tunnel syndrome (CTS), despite the limited evidence supporting its effectiveness. Previous studies on the effectiveness of low-level laser therapy (LLLT) have reported mixed results, and this systematic review aimed to resolve this controversy.

OBJECTIVE

To perform a network meta-analysis (NMA) for evaluating the effectiveness of LLLT compared with other conservative treatments for CTS.

METHODS

Eighteen electronic databases were searched for potential randomised controlled trials (RCTs). RCTs evaluating LLLT or other non-surgical treatments as an add-on to splinting were included. Included RCTs measured at least one of the following three outcomes with validated instruments: pain, symptom severity and functional status.

RESULTS

Six RCTs (418 patients) were included. NMA suggested that LLLT plus splinting has the highest probability (75%) of pain reduction, compared with sham laser plus splinting (61%), ultrasound plus splinting (57%) and splinting alone (8%). However, while LLLT plus splinting is significantly more effective than sham laser plus splinting for pain reduction, the magnitude is not clinically significant (Visual Analogue Scale mean difference -0.53cm, 95% confidence interval -1.01 to -0.05cm; P=0.03, I²=25%). The effect of LLLT plus splinting on symptom severity and functional status was not superior to splinting alone.

CONCLUSION

The use of LLLT in addition to splinting for the management of CTS is not recommended, as LLLT offers limited additional benefits over splining alone in terms of pain reduction, reduction of symptom severity or improved functional status. PROSPERO for systematic reviews and meta-analyses registration number CRD42017082650.

Mapping the nomenclature, methodology, and reporting of studies that review methods: a pilot methodological review

BACKGROUND

A relatively novel method of appraisal, methodological reviews (MRs) are used to synthesize information on the methods used in health research. There are currently no guidelines available to inform the reporting of MRs.

OBJECTIVES

This pilot review aimed to determine the feasibility of a full review and the need for reporting guidance for methodological reviews.

METHODS

Search strategy: We conducted a search of PubMed, restricted to 2017 to include the most recently published studies, using different search terms often used to describe methodological reviews: "literature survey" OR "meta-epidemiologic* review" OR "meta-epidemiologic* survey" OR "methodologic* review" OR "methodologic* survey" OR "systematic survey."Data extraction: Study characteristics including country, nomenclature, number of included studies, search strategy, a priori protocol use, and sampling methods were extracted in duplicate and summarized.Outcomes: Primary feasibility outcomes were the sensitivity and specificity of the search terms (criteria for success of feasibility set at sensitivity and specificity of ≥ 70%).Analysis: The estimates are reported as a point estimate (95% confidence interval).

RESULTS

Two hundred thirty-six articles were retrieved and 31 were included in the final analysis. The most accurate search term was "meta-epidemiological" (sensitivity [Sn] 48.39; 95% CI 31.97-65.16; specificity [Sp] 97.56; 94.42-98.95). The majority of studies were published by authors from Canada (n = 12, 38.7%), and Japan and USA (n = 4, 12.9% each). The median (interquartile range [IQR]) number of included studies in the MRs was 77 (13-1127). Reporting of a search strategy was done in most studies (n = 23, 74.2%). The use of a pre-published protocol (n = 7, 22.6%) or a justifiable sampling method (n = 5, 16.1%) occurred rarely.

CONCLUSIONS

Using the MR nomenclature identified, it is feasible to build a comprehensive search strategy and conduct a full review. Given the variation in reporting practices and nomenclature attributed to MRs, there is a need for guidance on standardized and transparent reporting of MRs. Future guideline development would likely include stakeholders from Canada, USA, and Japan.

Error rates of human reviewers during abstract screening in systematic reviews

Background

Automated approaches to improve the efficiency of systematic reviews are greatly needed. When testing any of these approaches, the criterion standard of comparison (gold standard) is usually human reviewers. Yet, human reviewers make errors in inclusion and exclusion of references.

Objectives

To determine citation false inclusion and false exclusion rates during abstract screening by pairs of independent reviewers. These rates can help in designing, testing and implementing automated approaches.

Methods

We identified all systematic reviews conducted between 2010 and 2017 by an evidence-based practice center in the United States. Eligible reviews had to follow standard systematic review procedures with dual independent screening of abstracts and full texts, in which citation inclusion by one reviewer prompted automatic inclusion through the next level of screening. Disagreements between reviewers during full text screening were reconciled via consensus or arbitration by a third reviewer. A false inclusion or exclusion was defined as a decision made by a single reviewer that was inconsistent with the final included list of studies.

Results

We analyzed a total of 139,467 citations that underwent 329,332 inclusion and exclusion decisions from 86 unique reviewers. The final systematic reviews included 5.48% of the potential references identified through bibliographic database search (95% confidence interval (CI): 2.38% to 8.58%). After abstract screening, the total error rate (false inclusion and false exclusion) was 10.76% (95% CI: 7.43% to 14.09%).

Conclusions

This study suggests important false inclusion and exclusion rates by human reviewers. When deciding the validity of a future automated study selection algorithm, it is important to keep in mind that the gold standard is not perfect and that achieving error rates similar to humans may be adequate and can save resources and time.

Examining overlap of included studies in meta-reviews: Guidance for using the corrected covered area index

Overlap in meta-reviews results from the use of multiple identical primary studies in similar reviews. It is an important area for research synthesists because overlap indicates the degree to which reviews address the same or different literatures of primary research. Current guidelines to address overlap suggest that assessing and documenting the degree of overlap in primary studies, calculated via the corrected covered area (CCA) is a promising method. Yet, the CCA is a simple percentage of overlap and current guidelines do not detail ways that reviewers can use the CCA as a diagnostic tool while also comprehensively incorporating these findings into their conclusions. Furthermore, we maintain that meta-review teams must address non-independence via overlap more thoroughly than by simply estimating and reporting the CCA. Instead, we recommend and elaborate five steps to take when examining overlap, illustrating these steps through the use of an empirical example of primary study overlap in a recently conducted meta-review. This work helps to show that overlap of primary studies included in a meta-review is not necessarily a bias but often can be a benefit. We also highlight further areas of caution in this task and potential for the development of new tools to address non-independence issues.

Mapping of reporting guidance for systematic reviews and meta-analyses generated a comprehensive item bank for future reporting guidelines

OBJECTIVES

The aim of the study was to generate a comprehensive bank of systematic review (SR) reporting items to inform an update of the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) 2009 statement.

METHODS

We searched the Enhancing the QUAlity and Transparency Of health Research Network library in May 2019 to identify all reporting guidelines for SRs that were published after 2009, regardless of the scope of the guideline. We also conducted a selective review of four guidance manuals for SRs, three tools for assessing the risk of bias in SRs, six meta-research studies evaluating the reporting quality of SRs using a tailored checklist, and five reporting guidelines for other study designs. One author screened and selected sources for inclusion, extracted reporting guidance from sources, and mapped guidance against the PRISMA 2009 checklist items.

RESULTS

We included 60 sources providing guidance on reporting of SRs and meta-analyses. From these, we collated a list of 221 unique reporting items. Items were categorized into title (four items), abstract (10 items), introduction (12 items), methods (111 items), results (61 items), discussion (12 items), funding and conflicts of interest (four items), administrative information (three items), and data availability (four items). This exercise generated 175 reporting items that could be added to the guidance in the PRISMA 2009 statement.

CONCLUSION

Generation of a comprehensive item bank through review and mapping of the literature facilitates identification of missing items and those needing modification, which may not otherwise be identified by the guideline development team or from other activities commonly used to develop reporting guidelines.

Comparison of Corneal Biomechanics Between Low and High Myopic Eyes-A Meta-analysis

PURPOSE

To compare the corneal biomechanical difference between the low myopic eyes and high myopic eyes.

DESIGN

Systematic review and meta-analysis.

METHODS

Data sources, including PubMed, Medline, EMBASE, Web of Science, and Chinese databases including Wanfang and China National Knowledge Infrastructure, were searched to find the relevant studies. Primary outcomes were corneal hysteresis (CH), corneal resistance factor (CRF), corneal-compensated intraocular pressure (IOPcc), Goldmann-correlated intraocular pressure (IOPg), and central corneal thickness (CCT) in high myopic eyes and low myopic eyes.

RESULTS

Eleven studies were enrolled in this study. CH and CRF were significantly higher in the low myopic eyes. The mean difference of CH was 0.73 mm Hg, 95% confidence interval (CI) [0.53 to 0.93], P < .001. The mean difference of CRF was 0.20 mm Hg, 95% CI [0.04 to 0.37], P = .02. The IOPcc and IOPg were significantly lower in the low myopic eyes. The mean difference of IOPcc was -2.53 mm Hg, 95% CI [-3.24, -1.83], P < .01. The mean difference of IOPg was -1.42 mm Hg, 95% CI [-2.26, -0.58], P = .0009. There was no significant difference between the 2 groups on CCT; the mean difference was -2.85 μm, 95% CI [-9.64.3.93], P = .41.

CONCLUSION

Corneal biomechanics are different in the high myopic eyes. Low CH and CRF and high IOPcc and IOPg are suggested to be associated factors for high myopia. Future studies are needed to investigate the underlying corneal structure difference that causes the low CH and CRF value in the high myopic eyes.

A meta-epidemiological study of subgroup analyses in cochrane systematic reviews of atrial fibrillation

BACKGROUND

Information on subgroup assessments in systematic reviews (SR) of atrial fibrillation (AF) is limited. This review aims to describe subgroup analyses in AF SRs to inform the design of SRs and randomized trials as well as clinical practice.

METHODS

We conducted a cross sectional meta-epidemiological study of Cochrane AF reviews by searching AF (including variants) in the title, abstract, or keyword field without date or language restrictions (Issue 9; September 2018). Two reviewers independently extracted study characteristics to summarize frequency of subgroups pre-specified and conducted and report credibility of subgroup effects claimed.

RESULTS

Of 39 Cochrane reviews identified, 17 met inclusion criteria (including 168 reports of 127 randomized trials) and the majority (16; 94.1%) conducted meta-analysis of outcomes. Most (13; 76.5%) planned pre-specified subgroup analyses; 7 of which (41.2%) conducted subgroups. In these 7 reviews, 56 subgroups were planned, 17 (30.4%) conducted and 6 (10.7%) yielded subgroup effects. Variables such as co-morbid disease, stroke risk factors, prior stroke/transient ischemic attack, age, race, and sex represented 44% (24 subgroups) of all planned subgroups (8 conducted; 14.3%); however, information on covariate selection was lacking. Overall, more subgroups were planned than conducted (mean difference (95% CI) 2.3 (1.2-3.5, p < 0.001)). Of all subgroups conducted, anticoagulant characteristics comprised a third of all subgroup effects (n = 5, 35.7%). The credibility of subgroups identified (n = 14) was assessed and less than half (43%) represented one of a small number of pre-specified hypothesis and rarely were effects seen within studies (7%). Of 5 reviews that reported subgroup effects, only 3 discussed subgroup effects as part of the overall conclusions; none discussed credibility of subgroup effects.

CONCLUSIONS

This meta-epidemiological review of a subset of Cochrane AF reviews suggests that planning and reporting of subgroup analyses in AF reviews can be improved to better inform clinical management. Most pre-specified subgroup analyses were not performed, important variables (such as stroke, bleeding risk, and other comorbidities) were rarely examined and credibility of subgroup effects claimed was low. Future reviews should aim to identify important subgroups in their protocols and use recommended approaches to test subgroup effects in order to better support clinical decision-making.

Protocol registration or development may benefit the design, conduct and reporting of dose-response meta-analysis: empirical evidence from a literature survey

Background

To investigate the prevalence of protocol registration (or development) among published dose-response meta-analyses (DRMAs), and whether DRMAs with a protocol are better than those not.

Methods

Three databases were searched for eligible DRMAs. The modified AMSTAR (14 items) and PRISMA checklists (26 items) were used to assess the methodological and reporting quality, with each item assigned 1 point if it met the requirement or 0 if not. We matched (1,2) DRMAs with registered or published protocol to those not, by region and publication years. The summarized quality score and compliance rate of each item were compared between the two groups. Multivariable regression was employed to see if protocol registration or development was associated with total quality score.

Results

We included 529 DRMAs, with 45 (8.51%) completed protocol registration or development. We observed a higher methodological score for DRMAs with protocol than the matched controls (9.47 versus 8.58, P <  0.01); this embodied in 4 out of 14 items of AMSTAR [e.g., Duplicate data extraction (rate difference, RD = 0.17, 95% CI: 0.04, 0.30; P = 0.01). A higher reporting score (cubic transformed) for DRMAs with protocol than the matched controls was also observed (11,875.00 versus 10,229.53, P <  0.01); which embodied in 6 out of 26 items of PRISMA [e.g. Describe methods for publication bias (RD = 0.08, 95% CI: 0.01, 0.14; P = 0.02)]. Regression analysis suggested positive association between protocol registration or development and total reporting score (P = 0.012) while not for methodological score (P = 0.87).

Conclusions

Only a small proportion of DRMAs completed protocol registration or development, and those with protocol were better reported than those not. Protocol registration or development is highly desirable.

Electronic supplementary material

The online version of this article (10.1186/s12874-019-0715-y) contains supplementary material, which is available to authorized users.

Reporting guidelines for health research: protocol for a cross-sectional analysis of the EQUATOR Network Library

INTRODUCTION

Transparency and completeness of health research is highly variable, with important deficiencies in the reporting of methods and results of studies. Reporting guidelines aim to improve transparency and quality of research reports, and are often developed by consortia of journal editors, peer reviewers, authors, consumers and other key stakeholders. The objective of this study will be to investigate the characteristics of scientific collaboration among developers and the citation metrics of reporting guidelines of health research.

METHODS AND ANALYSIS

This is the study protocol for a cross-sectional analysis of completed reporting guidelines indexed in the Enhancing the QUAlity and Transparency Of health Research Network Library. We will search PubMed/MEDLINE and the Web of Science. Screening, selection and data abstraction will be conducted by one researcher and verified by a second researcher. Potential discrepancies will be resolved via discussion. We will include published papers of reporting guidelines written in English. Published papers will have to meet the definition of a reporting guideline related to health research (eg, a checklist, flow diagram or explicit text), with no restrictions by study design, medical specialty, disease or condition. Raw data from each included paper (including title, publication year, journal, subject category, keywords, citations, and the authors' names, author's affiliated institution and country) will be exported from the Web of Science. Descriptive analyses will be conducted (including the number of papers, citations, authors, countries, journals, keywords and main collaboration metrics). We will identify the most prolific authors, institutions, countries, journals and the most cited papers. Network analyses will be carried out to study the structure of collaborations.

ETHICS AND DISSEMINATION

No ethical approval will be required. Findings from this study will be published in peer-reviewed journals. All data will be deposited in a cross-disciplinary public repository. It is anticipated the study findings could be relevant to a variety of audiences.

Meta-analysis is not an exact science: Call for guidance on quantitative synthesis decisions

Meta-analysis is becoming increasingly popular in the field of ecology and environmental management. It increases the effective power of analyses relative to single studies, and allows researchers to investigate effect modifiers and sources of heterogeneity that could not be easily examined within single studies. Many systematic reviewers will set out to conduct a meta-analysis as part of their synthesis, but meta-analysis requires a niche set of skills that are not widely held by the environmental research community. Each step in the process of carrying out a meta-analysis requires decisions that have both scientific and statistical implications. Reviewers are likely to be faced with a plethora of decisions over which effect size to choose, how to calculate variances, and how to build statistical models. Some of these decisions may be simple based on appropriateness of the options. At other times, reviewers must choose between equally valid approaches given the information available to them. This presents a significant problem when reviewers are attempting to conduct a reliable synthesis, such as a systematic review, where subjectivity is minimised and all decisions are documented and justified transparently. We propose three urgent, necessary developments within the evidence synthesis community. Firstly, we call on quantitative synthesis experts to improve guidance on how to prepare data for quantitative synthesis, providing explicit detail to support systematic reviewers. Secondly, we call on journal editors and evidence synthesis coordinating bodies (e.g. CEE) to ensure that quantitative synthesis methods are adequately reported in a transparent and repeatable manner in published systematic reviews. Finally, where faced with two or more broadly equally valid alternative methods or actions, reviewers should conduct multiple analyses, presenting all options, and discussing the implications of the different analytical approaches. We believe it is vital to tackle the possible subjectivity in quantitative synthesis described herein to ensure that the extensive efforts expended in producing systematic reviews and other evidence synthesis products is not wasted because of a lack of rigour or reliability in the final synthesis step.