The Quality of the Evidence According to GRADE Is Predominantly Low or Very Low in Oral Health Systematic Reviews

Cochrane systematic reviews in orthodontics: trends across updates

BACKGROUND

Systematic reviews (SR) are regularly updated to reflect new evidence. However, updates are time-consuming and costly, and therefore should ideally be informed by new high-quality research. The purpose of this study is to assess trends in the quantity, quality, and recency of evidence intervening updates of orthodontic SR.

METHODS

SR relevant to orthodontics with at least two versions were identified from the Cochrane Database. The number, risk of bias, and year of publication of included trials were recorded for each update. Multivariate regression was conducted to assess factors affecting the risk of bias in trials, and the proportions within SR.

RESULTS

Forty-five SR inclusive of updates were included. The median number of trials was three per review and this increased across subsequent versions. Seven reviews (15.6%) included no evidence, and 40.74% of updates included no new evidence. Most of the primary research was considered high risk of bias (57.3%), although this was reduced marginally across updates. The proportion of studies considered low risk did not improve significantly between updates. There was no impact of publication year of clinical trials on the risk of bias (P = 0.349). However, average age of trials included in a systematic review significantly affected the proportion of low risk-of-bias studies (P = 0.039).

CONCLUSIONS

SR are frequently updated without including new evidence. New evidence that is included is commonly deemed to be at high risk of bias. Targeted strategies to improve the efficient use of resources and improve research quality should be considered.

The impact of an online course on agreement rates of the certainty of evidence assessment using Grading of Recommendations, Assessment, Development, and Evaluation Approach: a before-and-after study

BACKGROUND AND OBJECTIVE

The Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) approach is a systematic method for assessing the certainty of evidence (CoE) and strength of recommendations in health care. We aimed to verify the effects of an online-based GRADE course on multirater consistency in the evaluation of the CoE in systematic reviews (SRs) analysis.

STUDY DESIGN AND SETTINGS

Sixty-five Brazilian methodologists and researchers participated in an online course over 8 weeks. Asynchronous lessons and weekly synchronous meetings addressed the GRADE system in the context of CoE assessment. We asked participants to evaluate the CoE of random SRs (two before and another two after the course). Analyzes focused on the multirater agreement with a standard response, in the interrater agreement, and before-after changes in the proportion of participants that rated down the domains.

RESULTS

48 individuals completed the course. Participants presented improvements in the raters' assessment of the CoE using the GRADE approach after the course. The multirater consistency of indirectness, imprecision, and the overall CoE increased after the course, as well as the agreement between raters and the standard response. Furthermore, interrater reliability increased for risk of bias, inconsistency, indirectness, publication bias, and overall CoE, indicating progress in between-raters consistency. After the course, approximately 78% of individuals rated down the overall CoE to a low/very low degree, and participants presented more explanations for the judgment of each domain.

CONCLUSION

An online GRADE course improved the consistency and agreement of the CoE assessment by Brazilian researchers. Online training courses have the potential to improve skills in guideline methodology development.

Pediatric dentistry systematic reviews using the GRADE approach: methodological study

BACKGROUND

To assess the reporting of the certainty of the evidence using the GRADE approach in systematic reviews of interventions in pediatric dentistry.

METHODS

The inclusion criteria were systematic reviews of randomized clinical trials (RCTs) and non-randomized studies of interventions (NRSIs) in pediatric dentistry that reported the certainty of the evidence through the GRADE approach. Paired independent reviewers screened the studies, extracted data, and appraised the methodological quality using the Assessing the Methodological Quality of Systematic Reviews (AMSTAR 2) tool. The certainty of the evidence was extracted for each outcome. A descriptive analysis was conducted.

RESULTS

Around 28% of pediatric dentistry reviews of interventions used the GRADE approach (n = 24). Twenty reviews reported 112 evidence outcomes from RCTs and 13 from NRSIs using GRADE evidence profile tables. The methodological quality was high (16.7%), moderate (12.5%), low (37.5%), and critically low (33.3%), fulfilling the majority of the AMSTAR 2 criteria. The certainty of the evidence for outcomes generated from RCTs and NRSIs was very low (40.2% and 84.6%), low (33.1% and 7.7%), moderate (17.8% and 7.7%), and high (9.8% and 0.0%). The main reasons to downgrade the certainty were due to (for RCTs and NRSIs, respectively): risk of bias (68.8% and 84.6%), imprecision (67.8% and 100.0%), inconsistency (18.8% and 23.1%), indirectness (17.8% and 0.0%), and publication bias (7.1% and 0.0%).

CONCLUSION

The proportion of systematic reviews assessing the certainty of the evidence using the GRADE approach was considered small, considering the total initial number of published pediatric dentistry reviews of intervention. The certainty of the evidence was mainly very low and low, and the main problems for downgrading the certainty of evidence were due to risk of bias and imprecision.

REGISTRATION

PROSPERO database #CRD42022365443.

Vaccination for the prevention of equine herpesvirus-1 disease in domesticated horses: A systematic review and meta-analysis

BACKGROUND

Equine herpes virus type 1 (EHV-1) infection in horses is associated with respiratory and neurologic disease, abortion, and neonatal death.

HYPOTHESIS

Vaccines decrease the occurrence of clinical disease in EHV-1-infected horses.

METHODS

A systematic review was performed searching multiple databases to identify relevant studies. Selection criteria were original peer-reviewed research reports that investigated the in vivo use of vaccines for the prevention of disease caused by EHV-1 in domesticated horses. Main outcomes of interest included pyrexia, abortion, neurologic disease, viremia, and nasal shedding. We evaluated risk of bias, conducted exploratory meta-analyses of incidence data for the main outcomes, and performed a GRADE evaluation of the quality of evidence for each vaccine subtype.

RESULTS

A total of 1018 unique studies were identified, of which 35 met the inclusion criteria. Experimental studies accounted for 31/35 studies, with the remainder being observational studies. Eight vaccine subclasses were identified including commercial (modified-live, inactivated, mixed) and experimental (modified-live, inactivated, deletion mutant, DNA, recombinant). Risk of bias was generally moderate, often because of underreporting of research methods, and sample sizes were small leading to imprecision in the estimate of the effect size. Several studies reported either no benefit or minimal vaccine efficacy for the primary outcomes of interest. Meta-analyses revealed significant heterogeneity was present, and our confidence in the quality of evidence for most outcomes was low to moderate.

CONCLUSIONS AND CLINICAL IMPORTANCE

Our review indicates that commercial and experimental vaccines minimally reduce the incidence of clinical disease associated with EHV-1 infection.

A case study evaluating the effect of clustering, publication bias, and heterogeneity on the meta-analysis estimates in implant dentistry

Meta-analyses may provide imprecise estimates when important meta-analysis parameters are not considered during the synthesis. The aim of this case study was to highlight the influence of meta-analysis parameters that can affect reported estimates using as an example pre-existing meta-analyses on the association between implant survival and sinus membrane perforation. PubMed was searched on 7 July 2021 for meta-analyses comparing implant failure in perforated and non-perforated sinus membranes. Primary studies identified in these meta-analyses were combined in a new random-effects model with odds ratios (ORs), confidence intervals (CIs), and prediction intervals reported. Using this new meta-analysis, further meta-analyses were then undertaken considering the clinical, methodological, and statistical heterogeneity of the primary studies, publication bias, and clustering effects. The meta-analyses with the greatest number and more homogeneous studies provided lower odds of implant failure in non-perforated sites (OR 0.49, 95 % CI = [0.26, 0.92]). However, when considering heterogeneity, publication bias, and clustering (number of implants), the confidence in these results was reduced. Interpretation of estimates reported in systematic reviews can vary depending on the assumptions made in the meta-analysis. Users of these analyses need to carefully consider the impact of heterogeneity, publication bias, and clustering, which can affect the size, direction, and interpretation of the reported estimates.

Efficiency based updating of evidence based surgical guidelines - Experiences from a multidisciplinary guideline upon Minimally Invasive Surgery

Background

Updating evidence-based clinical practice guidelines is an onerous process and there is a call for more efficient determination of key questions that need updating. Especially for surgical techniques it is unclear if new evidence will result in substantial changes after wide implementation and if continuous updating is always necessary.

Objectives

This study analyses the impact of updating a surgical guideline and proposes suggestions for optimising this process.

Materials and methods

The Dutch Minimally Invasive Surgery guideline was developed in 2011 and updated in 2021. For both versions a multidisciplinary guideline working group (GDG) was created, that determined key questions. Changes in conclusions and recommendations were analysed by the GDG and statements for expected change of recommendations in the future were made.

Results

15 key questions were formed, of which 12 were updates of the previous guideline. For only 27% of the updated key questions, the conclusions changed. In ten years, the body grew only marginally for most key questions and quality of the evidence did not improve substantially for almost all key questions. However, in this first update of the MIC guideline, many recommendations did change due to a more robust interpretation of the conclusions by the GDG. Based on analysis of this updating process, the GDG expects that only four out of 15 recommendations may change in the future.

Conclusion

We propose an additional step at the end of guideline development and updating, where the necessity for updating in the future is determined for each key question by the GDG, using their valuable knowledge gained from developing or updating the guideline. For surgical guidelines, the authors suggest updating key issues if it includes a relatively newly introduced surgical- or adapted technique or a new patient group. Low quality or small body of evidence should not be a reason in itself for updating, as this mostly does not lead to new evidence-based conclusions. This new step is expected to result in a more efficient prioritising of key questions that need updating.

What’s new?

By adding one additional step at the end of the updating process, the future updating process could become more efficient.

Standing on the shoulders of bias: lack of transparency and reporting of critical rigor characteristics in pain research

ABSTRACT

Rigorous experimental design with transparent reporting in biomedical science reduces risk of bias and allows for scientists to judge the quality of the research. Basic factors of rigor such as blinding, randomization, power analysis, and inclusion of both sexes impact the reproducibility by reducing experimental bias. We designed a systematic study to analyze basic factors of rigor, inclusion of sex, and whether data were analyzed or disaggregated by sex over the past 10 years in the journal PAIN . Studies that included humans reported randomization in 81%, blinding in 48%, and the use of a power analysis calculation in 27% over the past 10 years. Studies that included mice reported randomization in 35%, blinding in 70%, and the use of a power analysis in 9%. Studies that included rats reported randomization in 38%, blinding in 63%, and the use of power analysis in 12%. This study also found that human studies consistently included both sexes over the past decade, but less than 20% of data were disaggregated or analyzed for sex differences. Although mouse and rat studies predominately used males only, there has been a slight increase in inclusion of both sexes over the past few years. Justification for single-sex studies was below 50% in both human and rodent data. In both human and animal studies, transparency in reporting of experimental design and inclusion of both sexes should be considered standard practice and will result in improved quality and reproducibility of published research.

Evaluation of the use of GRADE in dentistry systematic reviews and its impact on conclusions: a protocol for a methodological study

BACKGROUND

There is a growing body of evidence of systematic reviews (SRs) with varying degrees of methodological quality. The Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) approach allows SR authors to assess the certainty of the evidence they found and transparently relay their conclusions. As there appears to be infrequent utilization of GRADE in the field of dentistry, to identify the impact of GRADE, the aim of this study is to evaluate the use of GRADE in the dental literature and determine whether SRs that use GRADE differ from those that do not with respect to their conclusions.

METHODS/DESIGN

We will search Ovid MEDLINE for SRs published from 2016 to the present. We will conduct both screening and data extraction independently and in duplicate and use pre-piloted, standardized forms for data extraction. We will determine the frequency of the use of GRADE and the varying levels of certainty in the current literature and evaluate whether GRADE is being used appropriately. We will also evaluate whether SRs not using GRADE differ from those that use GRADE with regard to methodological quality. We will also determine whether the conclusions of SRs that do not use GRADE would change had GRADE been utilized. Additionally, we will evaluate whether SRs using GRADE are more likely to formulate appropriate conclusions compared to SRs that do not use it.

DISCUSSION

This study will investigate the frequency of GRADE assessments in dentistry SRs and the impact of GRADE assessments on the conclusions of a SR. It has important implications for both SR authors and users of this type of literature.

One-Third of Systematic Reviews in Rehabilitation Applied the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) System to Evaluate Certainty of Evidence: A Meta-Research Study

OBJECTIVE

To determine how many systematic reviews (SRs) of the literature in rehabilitation assess the certainty of evidence (CoE) and how many apply the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system to do this.

DATA SOURCES

For this meta-research study, we searched PubMed and Cochrane Database of Systematic Reviews databases for SRs on rehabilitation published in 2020.

STUDY SELECTION AND DATA EXTRACTION

Two reviewers independently selected the SRs and extracted the data. Reporting characteristics and appropriate use of the GRADE system were assessed.

DATA SYNTHESIS

The search retrieved 827 records: 29% (239/827) SRs evaluated CoE, 68% (163/239) of which applied the GRADE system. GRADE was used by SRs of randomized controlled trials (RCTs, 88%; 144/163), non-randomized intervention studies (NRIS, 2%; 3/163), and both RCT and NRIS (10%; 16/163). In the latter case, a separate GRADE assessment according to the study design was not provided in 75% (12/16). The reasons for GRADE judgment were reported in 82% (134/163) of SRs.

CONCLUSIONS

One-third of SRs in rehabilitation assessed CoE with the GRADE system. GRADE assessment was presented transparently by most SRs. Journal editors and funders should encourage the uptake of the GRADE system when considering SRs in rehabilitation for publication. The authors should pre-define GRADE assessment in a registered and/or published protocol.

Faculty appointment and promotion in Taiwan's medical schools, a systematic analysis

BACKGROUND

A rigorous faculty appointment and promotion (FAP) system is vital for the success of any academic institution. However, studies examining the FAP system in Asian universities are lacking. We surveyed the FAP policies of Taiwan's medical schools and identified an overreliance on the CJA score (manuscript Category, Journal quality, and Author order). The potential shortcomings of this metric and recommendations for refinement were discussed.

METHODS

We obtained the FAP documents from all 12 medical schools in Taiwan, and analyzed their use of traditional versus non-traditional criteria for FAP according to a published methodology. The influence of the journal impact factor (JIF) on the FAP process was quantified by comparing its relative weight between papers with two extreme JIFs. To better understand the research impact and international standing of each school, we utilized the public bibliographic database to rank universities by the number of papers, and the proportions of papers within the top 10% or 50% citation.

RESULTS

Compared with other countries, Taiwan's medical schools focus more on the quantifiable quality of the research, mostly using a "CJA" score that integrates the category, JIF or ranking, and authorship of a paper, with the JIF being the most influential factor. The CJA score for an article with a JIF of 20 can be up to three times the threshold for promotion to Assistant Professor. The emphasis on JIF is based on a presumed correlation between JIF and citation counts. However, our analysis shows that Taiwan's medical schools have lower-than-average citation counts despite a competitive rank in the number of publications.

CONCLUSIONS

The JIF plays an unrivaled role in determining the outcome of FAP in Taiwan's medical schools, mostly via the CJA system. The questionable effectiveness of the current system in elevating the international standing of Taiwan's higher-education institutions calls for a re-examination of the FAP system. We recommend a reduction in the relative importance of CJA score in the FAP system, adopting more rigorous metrics such as the h-index for evaluating research quality, and supporting more research aimed at improving the FAP system.

Treatment-resistant depression: therapeutic options when first-line treatments fail

The Cochrane review by Davies et al aimed to address the lack of clarity on the risks and benefits of switching and augmentation strategies in the pharmacological treatment of treatment-resistant depression in adults who did not respond (or partially responded) to at least 4 weeks of antidepressant treatment at a recommended dose. This commentary assesses their review and their conclusion that augmenting the current antidepressant with mianserin or with an antipsychotic improves depressive symptoms over the short-term (8 to 12 weeks). Their results need to be treated with caution owing to the small body of evidence and individual comparisons supported by one, two or three studies, the limited evidence on long-term effects and the significant gaps in the literature (e.g. a lack of studies assessing dose increases).

The certainty of the evidence in oral health has not improved according to GRADE: a meta-epidemiological study

OBJECTIVE

The aim of this meta-epidemiological study was to provide an update of the certainty of the evidence in oral health by using the GRADE rating reported in oral health Cochrane systematic reviews (CSR).

STUDY DESIGN AND SETTING

All interventional oral health CSR published between 2003-2021 were sourced. Study characteristics were extracted at the level of the CSR and the outcome/meta-analysis. One-hundred-five CSR were eligible and analysed.

RESULTS

Almost a third of CSR (n=67) were excluded as a GRADE rating was not available. The most prevalent type of primary studies included in the CSR were randomized studies (93.4%) and the most used measure of effect was the risk ratio (67.3%). Overall, the certainty of the evidence according to the GRADE rating for all examined outcomes was very low/low (88%). The two most common reasons for downgrading the confidence in the evidence were study limitations (Risk of bias) and imprecision. The odds of moderate/high vs. low/very low-GRADE rating are higher for the primary compared to the secondary outcomes after adjusting for year and number of trials (OR 2.49; 95% CI: 1.09, 5.65; P=0.02). Per year (2010-2021 period) the odds of moderate/high vs. low/very low-GRADE rating decrease (OR 0.73; 95% CI: 0.60, 0.90; P=0.01), and as the number of trials per comparison increase the odds of moderate/high vs. low/very low GRADE rating increase (OR 1.13; 95% CI: 1.01, 1.25; P=0.001).

CONCLUSIONS

The certainty of the evidence in oral health when assessed with the GRADE rating remains predominantly low or very low.

Comparison Between Removable and Fixed Devices for Nonskeletal Anterior Crossbite Correction in Children and Adolescents: A Systematic Review

OBJECTIVE

To compare removable and fixed orthodontic devices in the correction of non-skeletal anterior crossbite in children and adolescents in the mixed dentition.

MATERIALS AND METHODS

Electronic searches were conducted in the following databases: PubMed, Web of Science, Scopus, Medline Ovid, Lilacs, US Clinical Trials, and Proquest. A hand search of the reference lists of the included articles and a Google Scholar search were also conducted. References were evaluated by 2 review authors. Articles that met the eligibility criteria were included. Data extraction, methodological quality assessment (Cochrane tool), and strength of the evidence evaluation (GRADE) were also carried out.

RESULTS

Seven articles were included. The results showed that removable and fixed devices were equally efficacious for overjet correction. Removable and fixed devices can also present inconveniences regarding pain and discomfort levels, the accomplishment of everyday activities (leisure and school), and the performance of functions, such as chewing and speech. However, treatment time and costs were significantly lower in orthodontic therapy with fixed appliances. Sequence generation, allocation concealment, and complete outcome data were not a concern. Blinding of participants or personnel was not reported in any article, and blinding of the assessor was a concern in 2 articles. Selective reporting was a concern in 2 articles. The certainty of the evidence for overjet correction was very low.

CONCLUSION

Removable and fixed orthodontic devices are efficacious for overjet correction in non-skeletal anterior crossbite. However, treatment time and costs are lower for cases treated with fixed devices.

Effect of medications on prevention of secondary osteoporotic vertebral compression fracture, non-vertebral fracture, and discontinuation due to adverse events: a meta-analysis of randomized controlled trials

BACKGROUND

Bone loss with aging and menopause increases the risk of fragile vertebral fracture, osteoporotic vertebral compression fracture (OVCF). The fracture causes severe pain, impedes respiratory function, lower the quality of life, and increases the risk of new fractures and deaths. Various medications have been prescribed to prevent a secondary fracture, but few study summarized their effects. Therefore, we investigated their effects on preventing subsequent OVCF via meta-analyses of randomized controlled trials.

METHODS

Electronic databases, including MEDLINE, EMBASE, CENTRAL, and Web of Science were searched for published randomized controlled trials from June 2015 to June 2019. The trials that recruited participants with at least one OVCF were included. We assessed the risk of bias of every study, estimated relative risk ratio of secondary OVCF, non-vertebral fracture, gastrointestinal complaints and discontinuation due to adverse events. Finally, we evaluated the quality of evidence.

RESULTS

Forty-one articles were included. Moderate to high quality evidence proved the effectiveness of zoledronate (Relative Risk, RR: 0.34; 95% CI, 0.17-0.69, p = 0.003), alendronate (RR: 0.54; 95% CI: 0.43-0.68; p < 0.0001), risedronate (RR: 0.61; 95% CI: 0.51-0.73; p < 0.0001), etidronate (RR, 0.50; 95% CI, 0.29-0.87, p < 0.01), ibandronate (RR: 0.52; 95% CI: 0.38-0.71; p < 0.0001), parathyroid hormone (RR: 0.31; 95% CI: 0.23-0.41; p < 0.0001), denosumab (RR, 0.41; 95% CI, 0.29-0.57; p < 0.0001) and selective estrogen receptor modulators (Raloxifene, RR: 0.58; 95% CI: 0.44-0.76; p < 0.0001; Bazedoxifene, RR: 0.66; 95% CI: 0.53-0.82; p = 0.0002) in preventing secondary fractures. Moderate quality evidence proved romosozumab had better effect than alendronate (Romosozumab vs. alendronate, RR: 0.64; 95% CI: 0.49-0.84; p = 0.001) and high quality evidence proved that teriparatide had better effect than risedronate (risedronate vs. teriparatide, RR: 1.98; 95% CI: 1.44-2.70; p < 0.0001).

CONCLUSION

Zoledronate, alendronate, risedronate, etidronate, ibandronate, parathyroid hormone, denosumab and selective estrogen receptor modulators had significant secondary prevention effects on OVCF. Moderate quality evidence proved romosozumab had better effect than alendronate. High quality evidence proved PTH had better effect than risedronate, but with higher risk of adverse events.

High quality of evidence is uncommon in Cochrane systematic reviews in Anaesthesia, Critical Care and Emergency Medicine

BACKGROUND

The association between the quality of evidence in systematic reviews and authors’ conclusions regarding the effectiveness of interventions relevant to anaesthesia has not been examined.

OBJECTIVE

The objectives of this study were: to determine the proportion of systematic reviews in which the authors made a conclusive statement about the effect of an intervention; to describe the quality of evidence derived from outcomes in reviews that used the Grades of Recommendation, Assessment, Development and Evaluation (GRADE) working group system for grading the quality of evidence; and to identify review characteristics associated with conclusiveness.

DESIGN

Cross-sectional analysis of Cochrane systematic reviews from the Anaesthesia, Critical Care and Emergency Review Group was undertaken.

DATA SOURCES

The Cochrane webpage was used to identify reviews for inclusion (http://.ace.cochrane.org/).

ELIGIBILITY CRITERIA

New and updated versions of systematic reviews published up to 17 September 2015 were eligible. Protocols for systematic reviews were excluded.

RESULTS

A total of 159 reviews were included. GRADE was used in 103 reviews (65%). Of these, high-level evidence for the primary outcome was identified in 11 reviews (10%). The main reasons that quality of evidence for the primary outcome was downgraded were risk of bias (n = 44; 43%) and imprecision (n = 36; 35%). Authors of 47% (n = 75) of the total number of reviews made conclusive statements about the effects of interventions. Independent predictors of conclusiveness in the subgroup of reviews with GRADE assessments were quality of evidence for the primary outcome (odds ratio 2.03; 95% confidence interval: [1.18 to 3.52] and an increasing number of studies included in reviews (OR 1.05; 95% CI: [1.01 to 1.09]).

CONCLUSION

It was common for conclusive statements to be made about the effects of interventions despite evidence for the primary outcome being rated less than high quality. Improving methodological quality of trials would have the greatest impact on improving the quality of evidence.

Assessing imprecision in Cochrane systematic reviews: a comparison of GRADE and Trial Sequential Analysis

BACKGROUND

The evaluation of imprecision is a key dimension of the grading of the confidence in the estimate. Grading of Recommendations Assessment, Development and Evaluation (GRADE) gives recommendations on how to downgrade evidence for imprecision, but authors vary in their use. Trial Sequential Analysis (TSA) has been advocated for a more reliable assessment of imprecision. We aimed to evaluate reporting of and adherence to GRADE and to compare the assessment of imprecision of intervention effects assessed by GRADE and TSA in Cochrane systematic reviews.

METHODS

In this cross-sectional study, we included 100 Cochrane reviews irrespective of type of intervention with a key dichotomous outcome meta-analyzed and assessed by GRADE. The methods and results sections of each review were assessed for adequacy of imprecision evaluation. We re-analyzed imprecision following the GRADE Handbook and the TSA Manual.

RESULTS

Overall, only 13.0% of reviews stated the criteria they applied to assess imprecision. The most common dimensions were the 95% width of the confidence intervals and the optimal information size. Review authors downgraded 48.0% of key outcomes due to imprecision. When imprecision was re-analyzed following the GRADE Handbook, 64% of outcomes were downgraded. Agreement between review authors' assessment and assessment by the authors of this study was moderate (kappa 0.43, 95% confidence interval [CI] 0.23 to 0.58). TSA downgraded 69.0% outcomes due to imprecision. Agreement between review authors' GRADE assessment and TSA, irrespective of downgrading levels, was moderate (kappa 0.43, 95% CI 0.21 to 0.57). Agreement between our GRADE assessment following the Handbook and TSA was substantial (kappa 0.66, 95% CI 0.49 to 0.79).

CONCLUSIONS

In a sample of Cochrane reviews, methods for assessing imprecision were rarely reported. GRADE according to Handbook guidelines and TSA led to more severe judgment of imprecision rather than GRADE adopted by reviews' authors. Cochrane initiatives to improve adherence to GRADE Handbook are warranted. TSA may transparently assist in such development.

Published intimate partner violence studies often differ from their trial registration records

INTRODUCTION

Registering study protocols in a trial registry is important for methodologic transparency and reducing selective reporting bias. The objective of this investigation was to determine whether published studies of intimate partner violence (IPV) that had been registered matched the registration record on key study design elements.

METHODS

We systematically searched three trial registries to identify registered IPV studies and the published literature for the associated publication. Two authors independently determined for each study whether key study elements in the registry matched those in the published paper.

RESULTS

We included 66 studies published between 2006 and 2017. Nearly half (29/66, 44%) were registered after study completion. Many (26/66, 39%) had discrepancies regarding the primary outcome, and nearly two-thirds (42/66, 64%) had discrepancies in secondary outcomes. Discrepancies in study design were less frequent (13/66, 20%). However, large changes in sample size (26/66, 39%) and discrepancies in funding source (28/66, 42%) were frequently observed.

CONCLUSIONS

Trial registries are important tools for research transparency and identifying and preventing outcome switching and selective outcome reporting bias. Published IPV studies often differ from their records in trial registries. Researchers should pay close attention to the accuracy of trial registry records.

Methodological and Reporting Quality of Systematic Reviews Published in the Highest Ranking Journals in the Field of Pain

BACKGROUND

Systematic reviews (SRs) are important for making clinical recommendations and guidelines. We analyzed methodological and reporting quality of pain-related SRs published in the top-ranking anesthesiology journals.

METHODS

This was a cross-sectional meta-epidemiological study. SRs published from 2005 to 2015 in the first quartile journals within the Journal Citation Reports category Anesthesiology were analyzed based on the Journal Citation Reports impact factor for year 2014. Each SR was assessed by 2 independent authors using Assessment of Multiple Systematic Reviews (AMSTAR) and Preferred Reporting Items of Systematic reviews and Meta-Analyses (PRISMA) tools. Total score (median and interquartile range, IQR) on checklists, temporal trends in total score, correlation in total scores between the 2 checklists, and variability of those results between journals were analyzed.

RESULTS

A total of 446 SRs were included. Median total score of AMSTAR was 6/11 (IQR: 4-7) and of PRISMA 18.5/27 (IQR: 15-22). High compliance (reported in over 90% SRs) was found in only 1 of 11 AMSTAR and 5 of 27 PRISMA items. Low compliance was found for the majority of AMSTAR and PRISMA individual items. Linear regression indicated that there was no improvement in the methodological and reporting quality of SRs before and after the publication of the 2 checklists (AMSTAR: F(1,8) = 0.22; P = .65, PRISMA: F(1,7) = 0.22; P = .47). Total scores of AMSTAR and PRISMA had positive association (R = 0.71; P < .0001).

CONCLUSIONS

Endorsement of PRISMA in instructions for authors was not a guarantee of compliance. Methodological and reporting quality of pain-related SRs should be improved using relevant checklists. This can be remedied by a joint effort of authors, editors, and peer reviewers.

Evaluation of risk of bias assessment of trials in systematic reviews of oral health interventions, 1991-2014: A methodology study

BACKGROUND

The authors aimed to describe how often and by what means investigators assessed the risk of bias of clinical trials in systematic reviews of oral health interventions and to identify factors associated with risk of bias assessments.

METHODS

The authors selected therapeutic oral health systematic reviews published from 1991 through 2014. They extracted data related to the tools used for risk of bias assessment of primary studies and data related to other review characteristics. They descriptively analyzed the data and used multivariate logistic regression.

RESULTS

The authors identified 1,114 oral health systematic reviews (130 Cochrane reviews and 984 non-Cochrane reviews). The investigators of the primary studies assessed risk of bias in 61.4% of the reviews, and the risk of bias assessments occurred more often in Cochrane reviews than in non-Cochrane reviews (100% versus 56.3%; P < .001) and in reviews published after the dissemination of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement (odds ratio [OR], 1.55; 95% confidence interval [CI], 1.17-2.06). Compared with the investigators of reviews of public oral health interventions, investigators of reviews of oral surgery were less likely to assess risk of bias (OR, 0.41; 95% CI, 0.25-0.67). Furthermore, the investigators of systematic reviews published in dental journals were less likely to assess risk of bias of individual trials (OR, 0.28; 95% CI, 0.19-0.41) compared with the investigators of reviews published in nondental journals.

CONCLUSIONS

The investigators of primary studies did not undertake risk of bias assessment in a considerable portion of non-Cochrane oral health systematic reviews. The investigators of reviews published in dental journals were less likely to assess risk of bias than the investigators of reviews published in nondental journals. The results of this study provide evidence of the need for improving the conduct and reporting of oral health systematic reviews with respect to risk of bias assessment.

PRACTICAL IMPLICATIONS

Clinicians should determine to what extent the findings of a systematic review are valid on the basis of whether the investigators assessed and considered risk of bias during the interpretation of findings.