1
|
Babić A, Barcot O, Visković T, Šarić F, Kirkovski A, Barun I, Križanac Z, Ananda RA, Fuentes Barreiro YV, Malih N, Dimcea DAM, Ordulj J, Weerasekara I, Spezia M, Žuljević MF, Šuto J, Tancredi L, Pijuk A, Sammali S, Iascone V, von Groote T, Poklepović Peričić T, Puljak L. Frequency of use and adequacy of Cochrane risk of bias tool 2 in non-Cochrane systematic reviews published in 2020: Meta-research study. Res Synth Methods 2024; 15:430-440. [PMID: 38262609 DOI: 10.1002/jrsm.1695] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2023] [Revised: 12/04/2023] [Accepted: 12/06/2023] [Indexed: 01/25/2024]
Abstract
Risk of bias (RoB) assessment is essential to the systematic review methodology. The new version of the Cochrane RoB tool for randomized trials (RoB 2) was published in 2019 to address limitations identified since the first version of the tool was published in 2008 and to increase the reliability of assessments. This study analyzed the frequency of usage of the RoB 2 and the adequacy of reporting the RoB 2 assessments in non-Cochrane reviews published in 2020. This meta-research study included non-Cochrane systematic reviews of interventions published in 2020. For the reviews that used the RoB 2 tool, we analyzed the reporting of the RoB 2 assessment. Among 3880 included reviews, the Cochrane RoB 1 tool was the most frequently used (N = 2228; 57.4%), followed by the Cochrane RoB 2 tool (N = 267; 6.9%). From 267 reviews that reported using the RoB 2 tool, 213 (79.8%) actually used it. In 26 (12.2%) reviews, erroneous statements were used to indicate the RoB 2 assessment. Only 20 (9.4%) reviews presented a complete RoB 2 assessment with a detailed table of answers to all signaling questions. The judgment of risk of bias by the RoB 2 tool was not justified by a comment in 158 (74.2%) reviews. Only in 33 (14.5%) of reviews the judgment in all domains was justified in the accompanying comment. In most reviews (81.7%), the RoB was inadequately assessed at the study level. In conclusion, the majority of non-Cochrane reviews published in 2020 still used the Cochrane RoB 1 tool. Many reviews used the RoB 2 tool inadequately. Further studies about the uptake and the use of the RoB 2 tool are needed.
Collapse
Affiliation(s)
- Andrija Babić
- Institute of Emergency Medicine in Split-Dalmatia County, Split, Croatia
| | - Ognjen Barcot
- Department of Surgery, University Hospital Split, Split, Croatia
| | - Tomislav Visković
- Institute of Emergency Medicine in Split-Dalmatia County, Split, Croatia
| | - Frano Šarić
- Department of Radiology, University Hospital Split, Split, Croatia
| | | | - Ivana Barun
- Department of Ophthalmology, University Hospital Split, Split, Croatia
| | | | - Roshan Arjun Ananda
- Department of General Medicine, Box Hill Hospital, Eastern Health, Box Hill, Australia
| | | | - Narges Malih
- Research Group on Global Health and Human Development, University of the Balearic Islands (UIB), Palma, Spain
| | - Daiana Anne-Marie Dimcea
- Department of Obstetrics and Gynaecology, Elias Emergency University Hospital, Bucharest, Romania
| | | | - Ishanka Weerasekara
- Department of Health and Functioning, Faculty of Health and Social Sciences, Western Norway University of Applied Sciences, Bergen, Norway
| | | | - Marija Franka Žuljević
- Centre for Evidence-Based Medicine, University of Split School of Medicine, Split, Croatia
| | - Jelena Šuto
- Department of Oncology and Radiotherapy, University Hospital of Split, Split, Croatia
| | - Luca Tancredi
- Geriatric Rehabilitation Clinic of the Hessing Foundation, Augsburg, Germany
- Medical School, Coburg, Germany
| | - Anđela Pijuk
- Division of Hematology, Department of Internal Medicine, University Hospital of Split, Split, Croatia
| | - Susanna Sammali
- University of Bologna, Bologna, Italy
- University of Florence, Florence, Italy
| | | | - Thilo von Groote
- Department of Anaesthesiology, Intensive Care and Pain Medicine, University Hospital Münster, Münster, Germany
| | | | - Livia Puljak
- Center for Evidence-Based Medicine and Health Care, Catholic University of Croatia, Zagreb, Croatia
| |
Collapse
|
2
|
Rehlicki D, Plenkovic M, Delac L, Pieper D, Marušić A, Puljak L. Author instructions in biomedical journals infrequently address systematic review reporting and methodology: a cross-sectional study. J Clin Epidemiol 2024; 166:111218. [PMID: 37993073 DOI: 10.1016/j.jclinepi.2023.11.008] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/26/2023] [Revised: 11/08/2023] [Accepted: 11/14/2023] [Indexed: 11/24/2023]
Abstract
OBJECTIVES We aimed to analyze how instructions for authors in journals indexed in MEDLINE address systematic review (SR) reporting and methodology. STUDY DESIGN AND SETTING We analyzed instructions for authors in 20% of MEDLINE-indexed journals listed in the online catalog of the National Library of Medicine on July 27, 2021. We extracted data only from the instructions published in English. We extracted data on the existence of instructions for reporting and methodology of SRs. RESULTS Instructions from 1,237 journals mentioned SRs in 45% (n = 560) of the cases. Systematic review (SR) registration was mentioned in 104/1,237 (8%) of instructions. Guidelines for reporting SR protocols were found in 155/1,237 (13%) of instructions. Guidelines for reporting SRs were explicitly mentioned in 461/1,237 (37%), whereas the EQUATOR (Enhancing the Quality and Transparency of Health Research) network was referred to in 474/1,237 (38%) of instructions. Less than 2% (n = 20) of instructions mentioned risk of bias and meta-analyses; less than 1% mentioned certainty of evidence assessment, methodological expectations, updating of SRs, overviews of SRs, or scoping reviews. CONCLUSION Journals indexed in MEDLINE rarely provide instructions for authors regarding SR reporting and methodology. Such instructions could potentially raise authors' awareness and improve how SRs are prepared and reported.
Collapse
Affiliation(s)
- Daniel Rehlicki
- Centre for Evidence-Based Medicine and Health Care, Catholic University of Croatia, Zagreb, Croatia
| | - Mia Plenkovic
- Department of Psychiatry, University of Split School of Medicine, Split, Croatia
| | - Ljerka Delac
- Division of Neurogeriatrics Karolinska Institutet, Department of Neurobiology, Care Sciences and Society, Solna, Sweden
| | - Dawid Pieper
- Faculty of Health Sciences Brandenburg, Brandenburg Medical School Theodor Fontane, Institute for Health Services and Health System Research, Rüdersdorf, Germany; Centre for Health Services Research, Brandenburg Medical School Theodor Fontane, Rüdersdorf, Germany
| | - Ana Marušić
- Department of Research in Biomedicine and Health, Centre for Evidence-based Medicine, University of Split School of Medicine, Split, Croatia
| | - Livia Puljak
- Centre for Evidence-Based Medicine and Health Care, Catholic University of Croatia, Zagreb, Croatia.
| |
Collapse
|
3
|
Minozzi S, Gonzalez-Lorenzo M, Cinquini M, Berardinelli D, Cagnazzo C, Ciardullo S, De Nardi P, Gammone M, Iovino P, Lando A, Rissone M, Simeone G, Stracuzzi M, Venezia G, Moja L, Costantino G. Adherence of systematic reviews to Cochrane RoB2 guidance was frequently poor: a meta epidemiological study. J Clin Epidemiol 2022; 152:47-55. [PMID: 36156301 DOI: 10.1016/j.jclinepi.2022.09.003] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2022] [Revised: 08/23/2022] [Accepted: 09/06/2022] [Indexed: 01/25/2023]
Abstract
OBJECTIVES To assess whether the use of the revised Cochrane risk of bias tool for randomized trials (RoB2) in systematic reviews (SRs) adheres to RoB2 guidance. METHODS We searched MEDLINE, Embase, Cochrane Library from 2019 to May 2021 to identify SRs using RoB2. We analyzed methods and results sections to see whether risk of bias was assessed at outcome measure level and applied to primary outcomes of the SR as per RoB2 guidance. The relation between SR characteristics and adequacy of RoB2 use was examined by logistic regression analysis. RESULTS Two hundred-eight SRs were included. We could assess adherence in 137 SRs as 12 declared using RoB2 but actually used RoB1 and 59 did not report the number of primary outcomes. The tool usage was adherent in 69.3% SRs. Considering SRs with multiple primary outcomes, adherence dropped to 28.8%. We found a positive association between RoB2 guidance adherence and the methodological quality of the reviews assessed by AMSTAR2 (p-for-trend 0.007). Multivariable regression analysis suggested journal impact factor [first quartile vs. other quartiles] was associated with RoB2 adherence (OR 0.34; 95% CI: 0.16-0.72). CONCLUSIONS Many SRs did not adhere to RoB2 guidance as they applied the tool at the study level rather than at the outcome measure level. Lack of adherence was more likely among low and very low quality reviews.
Collapse
Affiliation(s)
- Silvia Minozzi
- Department of Epidemiology, Lazio Regional Health Service, Rome, Italy
| | - Marien Gonzalez-Lorenzo
- Laboratorio di Metodologia delle revisioni sistematiche e produzione di Linee Guida, Istituto di Ricerche Farmacologiche Mario Negri IRCCS, Milan, Italy.
| | - Michela Cinquini
- Laboratorio di Metodologia delle revisioni sistematiche e produzione di Linee Guida, Istituto di Ricerche Farmacologiche Mario Negri IRCCS, Milan, Italy
| | - Daniela Berardinelli
- Department of Clinical and Biological Sciences, San Luigi Hospital, University of Turin, Turin, Italy
| | - Celeste Cagnazzo
- Department of Sciences of Public Health and Pediatrics, University of Turin, Turin, Italy; Division of Paediatric Onco-Haematology, Stem Cell Transplantation and Cellular Therapy, Città della Salute e della Scienza Hospital, Turin, Italy
| | - Stefano Ciardullo
- Department of Medicine and Rehabilitation, Policlinico di Monza, Monza, Italy; Department of Medicine and Surgery, University of Milano Bicocca, Milan, Italy
| | - Paola De Nardi
- Gastrointestinal Surgery, IRCCS San Raffaele Scientific Institute, Milan, Italy
| | | | - Paolo Iovino
- Department of Biomedicine and Prevention University of Rome Tor Vergata, Rome, Italy; School of Nursing, Midwifery and Paramedicine Faculty of Health Science, Australian Catholic University, Melbourne, Australia
| | - Alex Lando
- Laboratory of Rehabilitation Technologies, IRCCS San Camillo Hospital, Venice, Italy
| | - Marco Rissone
- Department of Public Health, Experimental and Forensic Medicine, University of Pavia, Pavia, Italy
| | - Giovanni Simeone
- Local Health Unit of Brindisi, Pediatric Department, Brindisi, Italy
| | - Marta Stracuzzi
- Pediatric Infectious Disease Unit, Department of Pediatrics, Luigi Sacco Hospital, University of Milan, Milan, Italy
| | | | - Lorenzo Moja
- Department of Biomedical Sciences for Health, University of Milan, Milan, Italy
| | - Giorgio Costantino
- Fondazione IRCCS Ca' Granda, Ospedale Maggiore Policlinico, University of Milan, Milan, Italy
| | | |
Collapse
|
4
|
Torensma B, Hisham M, Eldawlatly AA, Hany M. Differences Between the 2016 and 2022 Editions of the Enhanced Recovery After Bariatric Surgery (ERABS) Guidelines: Call to Action of FAIR Data and the Creation of a Global Consortium of Bariatric Care and Research. Obes Surg 2022; 32:2753-2763. [PMID: 35654929 PMCID: PMC9162377 DOI: 10.1007/s11695-022-06132-7] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/21/2022] [Revised: 05/25/2022] [Accepted: 05/25/2022] [Indexed: 12/19/2022]
Abstract
In 2016, the Enhanced Recovery After Bariatric Surgery guidelines (G16) was published, and in 2022, an update to it was released (G22). Grading of recommendations, assessment, development, and evaluations (GRADE), emphasizing the level of evidence (LoE) of both the guidelines, was performed. An overview of methodology was also performed, considering the following questions: how can research be improved, what can be done in the future using data, and how to collaborate more? Both guidelines did not explain how the LoE conclusions were derived regarding the risk of bias. There is also potential for forming a global consortium that deals with bariatric research, which can serve as a repository for all relevant data. Ensuring that this data is FAIR (findability, accessibility, interoperability, reusability) compliant and using this data to formulate future guidelines will benefit clinicians and patients alike.
Collapse
Affiliation(s)
- Bart Torensma
- Clinical Epidemiologist, Leiden University Medical Center (LUMC), Albinusdreef 2, 2333 ZA, Leiden, The Netherlands.
| | - Mohamed Hisham
- Medical Research Institute, Alexandria University, Alexandria, Egypt
| | - Abdelazeem A Eldawlatly
- Head of the Department of Anaesthesiology, King Saudi University Hospital, Riyadh, Saudi Arabia
| | - Mohamed Hany
- Medical Research Institute, Alexandria University, Alexandria, Egypt.,Head of the Department of Surgery, Medical Research Institute, Alexandria University, Alexandria, Egypt
| |
Collapse
|
5
|
Assessing the risk of performance and detection bias in Cochrane reviews as a joint domain is less accurate compared to two separate domains. BMC Med Res Methodol 2021; 21:149. [PMID: 34275437 PMCID: PMC8286598 DOI: 10.1186/s12874-021-01339-1] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/13/2021] [Accepted: 06/17/2021] [Indexed: 11/16/2022] Open
Abstract
Background Initially, the Cochrane risk of bias (RoB) tool had a domain for “blinding of participants, personnel and outcome assessors”. In the 2011 tool, the assessment of blinding was split into two domains: blinding of participants and personnel (performance bias) and blinding of outcome assessors (detection bias). The aims of this study were twofold; first, to analyze the frequency of usage of the joint blinding domain (a single domain for performance and detection bias), and second, to assess the proportion of adequate assessments made in the joint versus single RoB domains for blinding by comparing whether authors’ RoB judgments were supported by explanatory comments in line with the Cochrane Handbook recommendations. Methods We extracted information about the assessment of blinding from RoB tables (judgment, comment, and whether it was specified which outcome type; e.g., objective, subjective) of 729 Cochrane reviews published in 2015-2016. In the Cochrane RoB tool, judgment (low, unclear or high risk) needs to be accompanied by a transparent comment, in which authors provide a summary justifying RoB judgment, to ensure transparency in how these judgments were reached. We reassessed RoB based on the supporting comments reported in Cochrane RoB tables, in line with instructions from the Cochrane Handbook. Then, we compared our new assessments to judgments made by Cochrane authors. We compared the frequency of adequate judgments in reviews with two separate domains for blinding versus those with a joint domain for blinding. Results The total number of assessments for performance bias was 6918, with 8656 for detection bias and 3169 for the joint domain. The frequency of adequate assessments was 74% for performance bias, 78% for detection bias, and 59% for the joint domain. The lowest frequency of adequate assessments was found when Cochrane authors judged low risk – 47% in performance bias, 62% in detection bias, and 31% in the joint domain. The joint domain and detection bias domain had a similar proportion of specified outcome types (17% and 18%, respectively). Conclusions Splitting joint RoB assessment about blinding into two domains was justified because the frequency of adequate judgments was higher in separate domains. Specification of outcome types in RoB domains should be further scrutinized. Supplementary Information The online version contains supplementary material available at 10.1186/s12874-021-01339-1.
Collapse
|
6
|
Barcot O, Ivanda M, Buljan I, Pieper D, Puljak L. Enhanced access to recommendations from the Cochrane Handbook for improving authors' judgments about risk of bias: A randomized controlled trial. Res Synth Methods 2021; 12:618-629. [PMID: 34050603 DOI: 10.1002/jrsm.1499] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/17/2020] [Revised: 04/24/2021] [Accepted: 05/24/2021] [Indexed: 12/24/2022]
Abstract
This randomized controlled trial (RCT) aimed to test the efficacy of enhanced access to Cochrane Handbook (Handbook) recommendations for judging the 2011 Cochrane risk of bias (RoB) domains for improving the adequacy of RoB judgments. Parallel-group RCT with a 1:1 allocation ratio (N = 2271 per group) was conducted. Eligible participants were corresponding authors of all published Cochrane reviews and protocols. After allocation by a random number generator, participants received 20 scenarios for assessing RoB. The intervention group was shown tables from the Handbook with instructions for assessing 2011 RoB tool together with scenarios they were supposed to assess-enhanced access to the Handbook. The control group was shown only a general link to the Handbook. The primary outcome was the proportion of participants that made an adequate judgment of RoB scenarios for analyzed domains. There were 240 responses out of 2020 delivered e-mail invitations in the intervention and 197/2254 in the control group. Only five participants from the intervention group judged RoB adequately in all the 20 scenarios and no one in the control group. The proportion of participants who adequately assessed all the scenarios within a domain was significantly higher in the intervention than in the control group. The frequency of adequate RoB judgments was 7.1% (95% CI: 5.0-9.3%, p < 0.001) higher in the intervention group (76.2%) than in the control group (69.0%). The enhanced access yields more adequate RoB assessments and could be incorporated in software supporting the RoB tool.
Collapse
Affiliation(s)
- Ognjen Barcot
- Department of Abdominal Surgery, University Hospital Split, Split, Croatia
| | - Matej Ivanda
- Department of Abdominal Surgery, University Hospital Split, Split, Croatia
| | - Ivan Buljan
- Department of Research in Biomedicine and Health, University of Split School of Medicine, Split, Croatia
| | - Dawid Pieper
- Institute for Research in Operative Medicine, Faculty of Health, School of Medicine, Witten/Herdecke University, Cologne, Germany
| | - Livia Puljak
- Center for Evidence-Based Medicine and Health Care, Catholic University of Croatia, Zagreb, Croatia
| |
Collapse
|
7
|
Adequacy of risk of bias assessment in surgical vs non-surgical trials in Cochrane reviews: a methodological study. BMC Med Res Methodol 2020; 20:240. [PMID: 32993499 PMCID: PMC7526117 DOI: 10.1186/s12874-020-01123-7] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/04/2020] [Accepted: 09/16/2020] [Indexed: 01/09/2023] Open
Abstract
Background Bias in randomized controlled trials (RCTs) can lead to underestimation or overestimation of the true effects of interventions. Surgical RCTs may suffer from the risk of bias (RoB) that is avoidable in trials of other interventions, and vice versa. We aimed to compare the adequacy of RoB assessments in surgical versus non-surgical RCTs included in Cochrane reviews and to assess the most common differences in those RoB assessments. Due to specificities of surgical trials, i.e. difficulties associated with blinding of surgical interventions, we hypothesized that assessments of surgical trials may be more adequate, compared to RCTs of non-surgical interventions. Methods This was a methodological study, analyzing methods of published Cochrane systematic reviews. Data were extracted from RoB tables in Cochrane reviews (judgments and accompanying explanatory comment) for the following four RoB domains used in the 2011 Cochrane RoB tool: randomization, allocation concealment, blinding of participants and personnel, and blinding of outcome assessors. We defined adequate assessments as those that were in line with instructions from the Cochrane Handbook for Systematic Reviews of Interventions. The prevalence of adequate assessments was compared in surgical versus non-surgical trials. The most common differences in both groups of reviews were presented. Results In 729 analyzed Cochrane reviews, there were 10,537 included trials. The prevalence of adequate RoB judgments made by Cochrane authors ranged from 87.9, 95%CI (87.3 to 88.6%) for randomization to 70.7, 95%CI (69.8 to 71.5%) for blinding of participants and personnel. For all analyzed RoB domains, the prevalence of adequate RoB domains was higher in surgical trials than in non-surgical trials. For two RoB domains assessing blinding, this difference between surgical and non-surgical trials was statistically significant (P < 0.001), while the difference was not significant for the RoB domain regarding randomization (P = 0.124) and allocation concealment (P = 0.039, β < 0.8). Conclusions RoB judgments were more in line with instructions from the Cochrane Handbook when Cochrane reviews assessed surgical trials, compared to those that analyzed non-surgical interventions. However, further steps are warranted to scrutinize RoB assessment in trials of both surgical and non-surgical interventions.
Collapse
|