Inter-review agreement of risk-of-bias judgments varied in Cochrane reviews

Reliability of the risk of bias assessment in randomized controlled trials for nursing: A cross-sectional study

AIM

To evaluate the percentage and reasons for disagreements in the risk of bias (RoB) assessments for randomized controlled trials (RCTs) included in more than one Cochrane review in the field of nursing.

BACKGROUND

Disagreement in RoB assessments reduces the credibility of the evidence summarized by systematic reviews (SRs). There is no study that evaluates the reliability of RoB assessments in nursing studies.

DESIGN

Secondary data analysis based on research reports.

METHODS

RCTs included in more than one review in the nursing have been included. The disagreement of the assessment was analysed, and the possible reasons for disagreements were investigated.

RESULTS

Twenty-three RCTs were included in more than one review. The agreement of assessment ranged from 36.84% for "selective reporting" to 91.30% for "random sequence generation". "Allocation concealment" showed the optimal agreement (84.21%). The items "blinding of participants and personnel", "blinding of outcome assessment" and "incomplete outcome data" showed poor agreement, with 50.00%, 58.82% and 66.67%, respectively. Most disagreements came from extracting incomplete or different RCTs' information.

CONCLUSIONS

The level of agreement of the assessment between reviews has varied greatly in the field of nursing. More complete and accurate information of RCTs needs to be collected when conducting a SR.

Guidance on how to efficiently find, choose, and use available systematic reviews was developed

OBJECTIVES

The aim of this paper is to provide clinicians and authors of clinical guidelines or patient information with practical guidance on searching and choosing systematic reviews(s) (SR[s]) and, where adequate, on making use of SR(s).

STUDY DESIGN AND SETTING

At the German conference of the Evidence-Based Medicine Network (EbM Network) a workshop on the topic was held to identify the most important areas where guidance for practice appears necessary. After the workshop, we established working groups. These included SR users with different backgrounds (eg, information specialists, epidemiologists) and working areas. Each working group developed and consented a draft guidance based on their expert knowledge and experiences. The results were presented to the entire group and finalized in an iterative process.

RESULTS

We developed a practical guidance that answers questions that usually arise when choosing and using SR(s). (1) How to efficiently find high-quality SRs? (2) How to choose the most appropriate SR? (3) What to do if no SR of sufficient quality could be identified? In addition, we developed an algorithm that links these steps and accounts for their interaction. The resulting guidance is primarily directed at clinicians and developers of clinical practice guidelines or patient information resources.

CONCLUSION

We suggest practical guidance for making the best use of SRs when answering a specific research question. The guidance may contribute to the efficient use of existing SRs. Potential benefits when using existing SRs should be always weighted against potential limitations.

How to conduct a meta-analysis in hand surgery. Part I: principles and procedures

Meta-analysis (MA) is a fundamental statistical tool for combining the results of different studies to obtain potentially high-level evidence that can be implemented in clinical practice. Although its use in clinical research is increasing, MAs are still relatively rare in hand surgery. Therefore, it should be important for every hand surgeon to not only know how to interpret a MA, but also how to perform one. The purpose of this first of a two-part article is to introduce the principles of MA and describe the main models and methods used to pool effect estimates.

Disagreements in risk of bias assessment for randomized controlled trials in hypertension-related Cochrane reviews

BACKGROUND

The inter-reviewer reliability of the risk of bias (RoB) assessment lacked agreement in previous studies. It is important to analyse these disagreements to improve the repeatability of RoB assessment. The objective of the study was to evaluate the frequency and reasons for disagreements in RoB assessments for randomised controlled trials (RCTs) that were included in multiple Cochrane reviews in the field of hypertension.

METHODS

A cross-sectional study was employed. We retrieved any RCTs that had been included in multiple Cochrane reviews in the field of hypertension from ARCHIE. The results of the RoB assessments were extracted, and the distributions of agreements and possible reasons for disagreement were analyzed.

RESULTS

Twenty-six Cochrane reviews were included in this study. A total of 78 RCTs appeared in more than one Cochrane review. The level of agreement ranged from domain to domain. "Blinding of outcome assessment" showed a reasonably high level of agreement (94.9%), while "incomplete outcome data", "selective outcome reporting" and "other sources of bias" showed moderate levels of agreement (74.6%, 79.2% and 75.6%, respectively). However, the domains of "allocation concealment", "random sequence generation" and "blinding of participants and personnel" showed low levels of agreement (24.4%, 23.5%, and 47.4%, respectively). In the domains of "allocation concealment" and "blinding of participants and personnel", the agreement group had higher proportion of publication year ≤ 1996 than the disagreement group (P = 0.008 and P < 0.001, respectively). In the "blinding of participants and personnel", the impact factor was higher in the agreement group (P < 0.001). By analyzing the support text, we found that the most likely reason for disagreement was extracting different information from the same RCT.

CONCLUSION

For Cochrane reviews in the field of hypertension using the 2011 version of the RoB tool, there was a large disagreement in the RoB assessment. It is suggested that the results of RoB assessments in systematic reviews that used the 2011 version of the RoB tool need to be interpreted with caution. More accurate information from RCTs needs to be collected when we synthesize clinical evidence.

Inter-Rater Agreement in Assessing Risk of Bias in Melanoma Prediction Studies Using the Prediction Model Risk of Bias Assessment Tool (PROBAST): Results from a Controlled Experiment on the Effect of Specific Rater Training

Assessing the risk of bias (ROB) of studies is an important part of the conduct of systematic reviews and meta-analyses in clinical medicine. Among the many existing ROB tools, the Prediction Model Risk of Bias Assessment Tool (PROBAST) is a rather new instrument specifically designed to assess the ROB of prediction studies. In our study we analyzed the inter-rater reliability (IRR) of PROBAST and the effect of specialized training on the IRR. Six raters independently assessed the risk of bias (ROB) of all melanoma risk prediction studies published until 2021 (n = 42) using the PROBAST instrument. The raters evaluated the ROB of the first 20 studies without any guidance other than the published PROBAST literature. The remaining 22 studies were assessed after receiving customized training and guidance. Gwet's AC₁ was used as the primary measure to quantify the pairwise and multi-rater IRR. Depending on the PROBAST domain, results before training showed a slight to moderate IRR (multi-rater AC₁ ranging from 0.071 to 0.535). After training, the multi-rater AC₁ ranged from 0.294 to 0.780 with a significant improvement for the overall ROB rating and two of the four domains. The largest net gain was achieved in the overall ROB rating (difference in multi-rater AC₁: 0.405, 95%-CI 0.149-0.630). In conclusion, without targeted guidance, the IRR of PROBAST is low, questioning its use as an appropriate ROB instrument for prediction studies. Intensive training and guidance manuals with context-specific decision rules are needed to correctly apply and interpret the PROBAST instrument and to ensure consistency of ROB ratings.

A Meta-Analysis of the Outcomes of Ligament Reconstruction Compared to No Reconstruction with or without Tendon Interposition Following Trapeziectomy for Thumb Carpometacarpal Joint Osteoarthritis

Background: The outcomes of trapeziectomy with ligament reconstruction and tendon interposition (LRTI) compared to the trapeziectomy (T) alone or in combination with tendon interposition (TIA) in the treatment of osteoarthritis (OA) of the carpometacarpal joint (CMCJ) of the thumb are still debated. The aim of this study is to conduct a meta-analysis to determine whether the outcomes of LRTI were better compared to T or TIA. Methods: A meta-analysis of randomised trials that included the outcomes of patients with thumb CMCJ OA that underwent LRTI, T or TIA with at least 1 year follow-up. The outcomes included number of patients with pain 1 year after surgery (NPP), Visual Analog Scale for Pain (VAS Pain), key pinch strength, patient reported outcome measurements (PROMs) and number of adverse effect (AEs). The effect was assessed using mean difference for quantitative variables (VAS pain and key pinch), standardised mean difference for PROMs and relative risk for binary variables (NPP and AEs). Results: Seven studies met the inclusion criteria (857 participants: 438 LRTI vs 419 T/TIA). We did not find significant differences between LRTI and T/TIA in NPP, VAS pain, key pinch at 1 year and 5 years or more, PROMs, and AEs at 1 year and at 5 years or more after surgery. Conclusions: The outcomes of LRTI are not superior to T or TIA in the treatment of OA of the thumb CMCJ. Level of Evidence: Level I (Therapeutic).

A new method for testing reproducibility in systematic reviews was developed, but needs more testing

BACKGROUND

To develop and test an approach to test reproducibility of SRs.

METHODS

Case study. We have developed an approach to test reproducibility retrospectively while focusing on the whole conduct of an SR instead of single steps of it. We replicated the literature searches and drew a 25% random sample followed by study selection, data extraction, and risk of bias (ROB) assessments performed by two reviewers independently. These results were compared narratively with the original review.

RESULTS

We were not able to fully reproduce the original search resulting in minor differences in the number of citations retrieved. The biggest disagreements were found in study selection. The most difficult section to be reproduced was the RoB assessment due to the lack of reporting clear criteria to support the judgement of RoB ratings, although agreement was still found to be satisfactory.

CONCLUSION

Our approach as well as other approaches needs to undergo testing and comparison in the future as the area of testing for reproducibility of SRs is still in its infancy.

Using existing systematic reviews for developing vaccination recommendations: Results of an international expert workshop

National immunization technical advisory groups (NITAGs) develop immunization-related recommendations. Systematic reviews are recommended to be used in this process, but conducting them requires significant resources, which many NITAGs lack. Using existing systematic reviews could help address this problem. The Robert Koch Institute and collaborators set up the SYSVAC2 project to facilitate the retrieval of existing systematic reviews and offer guidance on using them. This will include an online registry of systematic reviews relevant to immunization policy and an online course on how to use existing reviews. This report describes an international expert workshop held in December 2019 to develop consensus on methods for using existing reviews and other relevant factors for the registry and course. Members from NITAGs representing different regions of the world presented their experiences of using systematic reviews and reflected on challenges inhibiting use. Three methodologists considered different aspects of using systematic reviews. Interactive sessions followed, where implications for SYSVAC2 were discussed. Participants supported having critical appraisal ratings, plain language summaries, keyword search, and data visualization functions in the registry. They suggested tailoring course content to different audiences and including overviews of reviews as a topic and examples of how NITAGs have used or could use existing reviews. Participants agreed that whether a review is out-of-date should be decided by those using the review rather than registry staff. The registry could help by highlighting the date of literature search or included primary studies. Participants recommended a visualization function to highlight overlap across reviews and guidance on handling challenges to using reviews, ideally, involving a practical element. No consensus was reached on which critical appraisal tool to use for reviews in the registry, but a majority of participants wanted registry staff to perform appraisals. Formative research is planned before the registry and online course are launched in 2020.

The methodological quality of 176,620 randomized controlled trials published between 1966 and 2018 reveals a positive trend but also an urgent need for improvement

Many randomized controlled trials (RCTs) are biased and difficult to reproduce due to methodological flaws and poor reporting. There is increasing attention for responsible research practices and implementation of reporting guidelines, but whether these efforts have improved the methodological quality of RCTs (e.g., lower risk of bias) is unknown. We, therefore, mapped risk-of-bias trends over time in RCT publications in relation to journal and author characteristics. Meta-information of 176,620 RCTs published between 1966 and 2018 was extracted. The risk-of-bias probability (random sequence generation, allocation concealment, blinding of patients/personnel, and blinding of outcome assessment) was assessed using a risk-of-bias machine learning tool. This tool was simultaneously validated using 63,327 human risk-of-bias assessments obtained from 17,394 RCTs evaluated in the Cochrane Database of Systematic Reviews (CDSR). Moreover, RCT registration and CONSORT Statement reporting were assessed using automated searches. Publication characteristics included the number of authors, journal impact factor (JIF), and medical discipline. The annual number of published RCTs substantially increased over 4 decades, accompanied by increases in authors (5.2 to 7.8) and institutions (2.9 to 4.8). The risk of bias remained present in most RCTs but decreased over time for allocation concealment (63% to 51%), random sequence generation (57% to 36%), and blinding of outcome assessment (58% to 52%). Trial registration (37% to 47%) and the use of the CONSORT Statement (1% to 20%) also rapidly increased. In journals with a higher impact factor (>10), the risk of bias was consistently lower with higher levels of RCT registration and the use of the CONSORT Statement. Automated risk-of-bias predictions had accuracies above 70% for allocation concealment (70.7%), random sequence generation (72.1%), and blinding of patients/personnel (79.8%), but not for blinding of outcome assessment (62.7%). In conclusion, the likelihood of bias in RCTs has generally decreased over the last decades. This optimistic trend may be driven by increased knowledge augmented by mandatory trial registration and more stringent reporting guidelines and journal requirements. Nevertheless, relatively high probabilities of bias remain, particularly in journals with lower impact factors. This emphasizes that further improvement of RCT registration, conduct, and reporting is still urgently needed.

Many randomized controlled trials (RCTs) are biased and difficult to reproduce due to methodological flaws and poor reporting. Analysis of 176,620 RCTs published between 1966 and 2018 reveals that the risk of bias in RCTs generally decreased. Nevertheless, relatively high probabilities of bias remain, showing that further improvement of RCT registration, conduct, and reporting is still urgently needed.

Adequacy of risk of bias assessment in surgical vs non-surgical trials in Cochrane reviews: a methodological study

Background

Bias in randomized controlled trials (RCTs) can lead to underestimation or overestimation of the true effects of interventions. Surgical RCTs may suffer from the risk of bias (RoB) that is avoidable in trials of other interventions, and vice versa. We aimed to compare the adequacy of RoB assessments in surgical versus non-surgical RCTs included in Cochrane reviews and to assess the most common differences in those RoB assessments. Due to specificities of surgical trials, i.e. difficulties associated with blinding of surgical interventions, we hypothesized that assessments of surgical trials may be more adequate, compared to RCTs of non-surgical interventions.

Methods

This was a methodological study, analyzing methods of published Cochrane systematic reviews. Data were extracted from RoB tables in Cochrane reviews (judgments and accompanying explanatory comment) for the following four RoB domains used in the 2011 Cochrane RoB tool: randomization, allocation concealment, blinding of participants and personnel, and blinding of outcome assessors. We defined adequate assessments as those that were in line with instructions from the Cochrane Handbook for Systematic Reviews of Interventions. The prevalence of adequate assessments was compared in surgical versus non-surgical trials. The most common differences in both groups of reviews were presented.

Results

In 729 analyzed Cochrane reviews, there were 10,537 included trials. The prevalence of adequate RoB judgments made by Cochrane authors ranged from 87.9, 95%CI (87.3 to 88.6%) for randomization to 70.7, 95%CI (69.8 to 71.5%) for blinding of participants and personnel. For all analyzed RoB domains, the prevalence of adequate RoB domains was higher in surgical trials than in non-surgical trials. For two RoB domains assessing blinding, this difference between surgical and non-surgical trials was statistically significant (P < 0.001), while the difference was not significant for the RoB domain regarding randomization (P = 0.124) and allocation concealment (P = 0.039, β < 0.8).

Conclusions

RoB judgments were more in line with instructions from the Cochrane Handbook when Cochrane reviews assessed surgical trials, compared to those that analyzed non-surgical interventions. However, further steps are warranted to scrutinize RoB assessment in trials of both surgical and non-surgical interventions.

Agreement of Risk-of -Bias varied in systematic reviews on acupuncture and was associated with methodological quality

OBJECTIVES

The objective of the study was to evaluate the consistency of risk of bias assessments for overlapping randomized controlled trials (RCTs) included in systematic reviews (SRs) on acupuncture.

STUDY DESIGN AND SETTING

Databases were searched for acupuncture SRs. A weighted kappa (κ) statistic was calculated, and logistic regression was used to explore the factors of disagreements.

RESULTS

We included 241 RCTs from 109 SRs on acupuncture. The percentage disagreements ranged from 25% to 44%, with moderate agreement for random sequence generation (κ = 0.57), allocation concealment (κ = 0.50), and incomplete outcome data (κ = 0.50), besides fair agreement for blinding of participants and personnel (κ = 0.44), blinding of outcome assessment (κ = 0.31), and selective reporting (κ = 0.39). Only 19% RCTs were evaluated completely consistent. Methodological quality (random sequence generation, odds ratio (OR) = 3.46), international cooperation (allocation concealment, OR = 0.14; incomplete outcome data, OR = 0.14; selective reporting, OR = 0.05), and risk of bias reporting completeness score (selective reporting, OR = 0.53) significantly affected the relative odds of disagreements.

CONCLUSION

The level of agreement varied from fair to moderate agreement depending on the risk of bias domain. Methodological quality appears to be an overarching factor to account for disagreements.