1
|
Núñez-Núñez M, Maes-Carballo M, Mignini LE, Chien PFW, Khalaf Y, Fawzy M, Zamora J, Khan KS, Bueno-Cavanillas A. Research integrity in randomized clinical trials: A scoping umbrella review. Int J Gynaecol Obstet 2023; 162:860-876. [PMID: 37062861 DOI: 10.1002/ijgo.14762] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2023] [Revised: 03/12/2023] [Accepted: 03/16/2023] [Indexed: 04/18/2023]
Abstract
BACKGROUND Randomized clinical trials (RCTs) are experiencing a crisis of confidence in their trustworthiness. Although a comprehensive literature search yielded several reviews on RCT integrity, an overarching overview is lacking. OBJECTIVES The authors undertook a scoping umbrella review of the research integrity literature concerning RCTs. SEARCH STRATEGY AND SELECTION CRITERIA Following prospective registration (https://osf.io/3ursn), two reviewers independently searched PubMed, Scopus, The Cochrane Library, and Google Scholar, without language or time restrictions, until November 2021. The authors included systematic reviews covering any aspect of research integrity throughout the RCT lifecycle. DATA COLLECTION AND ANALYSIS The authors assessed methodological quality using a modified AMSTAR 2 (A Measurement Tool to Assess Systematic Reviews) tool and collated the main findings. MAIN RESULTS A total of 55 relevant reviews, summarizing 6001 studies (median per review, 63; range, 8-1106) from 1964 to 2021, had an overall critically low quality of 96% (53 reviews). Topics covered included general aspects (15%), design and approval (22%), conduct and monitoring (11%), reporting (38%), postpublication concerns (2%), and future research (13%). The most common integrity issues covered were ethics (18%) and transparency (18%). CONCLUSIONS Low-quality reviews identified various integrity issues across the RCT lifecycle, emphasizing the importance of high ethical standards and professionalism while highlighting gaps in the integrity landscape. Multistakeholder consensus is needed to develop specific RCT integrity standards.
Collapse
Affiliation(s)
- María Núñez-Núñez
- Pharmacy Department, University Hospital Clínico San Cecilio, Granada, Spain
- Biomedical research institute of Granada (IBS-Granada), Granada, Spain
- Consortium for Biomedical Research in Epidemiology and Public Health (CIBERESP-Spain), Madrid, Spain
| | - Marta Maes-Carballo
- General Surgery Department. Breast Cancer Unit, Complexo Hospitalario de Ourense, Ourense, Spain
- General Surgery Department, Hospital Público Verín, Ourense, Spain
| | | | | | - Yacoub Khalaf
- Guy's & St Thomas' Hospital Foundation Trust, London, UK
| | - Mohamed Fawzy
- IbnSina (Sohag), Banon (Assiut), Qena (Qena), Amshag (Sohag) IVF Facilities, Cairo, Egypt
| | - Javier Zamora
- Consortium for Biomedical Research in Epidemiology and Public Health (CIBERESP-Spain), Madrid, Spain
- Clinical Biostatistics Unit, Hospital Ramón y Cajal (IRYCIS), Madrid, Spain
| | - Khalid S Khan
- Consortium for Biomedical Research in Epidemiology and Public Health (CIBERESP-Spain), Madrid, Spain
- Preventive Medicine and Public Health, University of Granada Faculty of Medicine, Granada, Spain
| | - Aurora Bueno-Cavanillas
- Consortium for Biomedical Research in Epidemiology and Public Health (CIBERESP-Spain), Madrid, Spain
- Preventive Medicine and Public Health, University of Granada Faculty of Medicine, Granada, Spain
| |
Collapse
|
2
|
Statistical methods for handling compliance in randomized controlled trials of device interventions: a systematic review. J Clin Epidemiol 2022; 152:226-237. [PMID: 36183902 DOI: 10.1016/j.jclinepi.2022.09.015] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/01/2022] [Revised: 09/20/2022] [Accepted: 09/27/2022] [Indexed: 01/25/2023]
Abstract
OBJECTIVES We aimed to review the extent to which analysis of randomized controlled trials (RCTs) of device interventions includes methods to handle compliance to the study intervention as described in the protocol. STUDY DESIGN AND SETTING We conducted a systematic review of the statistical methods used to handle compliance to a device intervention when estimating the effect of the device compared to another intervention in RCTs. We searched Embase, MEDLINE, PsychInfo, and the Cochrane Central Register of Controlled Trials. We sought to evaluate what methods were used and how using these methods impacted the estimate of the effect size. RESULTS One hundred fifty eight RCTs were identified for inclusion, of which only 21 (13%) described using a method to account for compliance to the device intervention, consisting of alternative analysis populations such as per-protocol, modified intention-to-treat, or as-treated, alongside a primary intention-to-treat analysis. No causal inference methods were used. Fourteen (9%) studies included compliance as a factor in the analysis and investigated its effect on outcomes. CONCLUSION Although some studies consider methods to handle compliance, causal inference methods have not been well adopted in the analysis of device trials. An increased awareness of the applications of statistical methods to adjust for compliance is needed.
Collapse
|
3
|
Gooden MJ, Norato G, Martin SB, Nath A, Reoma L. Reducing Events of Noncompliance in Neurology Human Subjects Research: the Effect of Human Subjects Research Protection Training and Site Initiation Visits. Neurotherapeutics 2021; 18:859-865. [PMID: 33475954 PMCID: PMC8423976 DOI: 10.1007/s13311-020-01003-4] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 12/28/2020] [Indexed: 11/29/2022] Open
Abstract
In an effort to minimize protocol noncompliance in neurological research studies that can potentially compromise patient safety, delay completion of the study, and result in premature termination and added costs, we determined the effect of investigator trainings and site initiation visits (SIVs) on the occurrence of noncompliance events. Results of protocol audits conducted at the National Institute of Neurological Disorders and Stroke from 2003 to 2019 on 97 research protocols were retrospectively analyzed. Based on the depth of auditing and provision of investigator research training, audit data were separated into four arms: 1) Early Period, 2003 to 2012; 2) Middle Period, 2013 to 2016; and Late Period, 2017 to 2019, further divided into 3) Late Period without SIVs; and 4) Late Period with SIVs. Events of noncompliance were classified by the type of protocol deviation, the category, and the cause. In total, 952 events occurred across 1080 participants. Protocols audited during the Middle Period, compared to the Early Period, showed a decrease in the percentage of protocols with at least 1 noncompliance event. Protocols with SIVs had a further decrease in major, minor, procedural, eligibility, and policy events. Additionally, protocols audited during the Early Period had on average 0.46 major deviations per participant, compared to 0.26 events in protocols audited during the Middle Period, and 0.08 events in protocols audited during the Late Period with SIVs. Protocol deviations and noncompliance events in neurological clinical trials can be reduced by targeted investigator trainings and SIVs. These measures have major impacts on the integrity, safety, and effectiveness of human subjects research in neurology.
Collapse
Affiliation(s)
- Matthew J Gooden
- Clinical Trials Unit, National Institute of Neurological Disorders and Stroke, National Institutes of Health, 10 Center Drive, Building 10, Room 2A23, Bethesda, MD, 20814, USA
| | - Gina Norato
- Clinical Trials Unit, National Institute of Neurological Disorders and Stroke, National Institutes of Health, 10 Center Drive, Building 10, Room 2A23, Bethesda, MD, 20814, USA
| | - Sandra B Martin
- Clinical Trials Unit, National Institute of Neurological Disorders and Stroke, National Institutes of Health, 10 Center Drive, Building 10, Room 2A23, Bethesda, MD, 20814, USA
| | - Avindra Nath
- Section of Infections of the Nervous System, National Institute of Neurological Disorders and Stroke, National Institutes of Health, 10 Center Drive, Building 10, Room 7C103, Bethesda, MD, 20814, USA
| | - Lauren Reoma
- Clinical Trials Unit, National Institute of Neurological Disorders and Stroke, National Institutes of Health, 10 Center Drive, Building 10, Room 2A23, Bethesda, MD, 20814, USA.
- Section of Infections of the Nervous System, National Institute of Neurological Disorders and Stroke, National Institutes of Health, 10 Center Drive, Building 10, Room 7C103, Bethesda, MD, 20814, USA.
| |
Collapse
|
4
|
Kahale LA, Khamis AM, Diab B, Chang Y, Lopes LC, Agarwal A, Li L, Mustafa RA, Koujanian S, Waziry R, Busse JW, Dakik A, Hooft L, Guyatt GH, Scholten RJPM, Akl EA. Meta-Analyses Proved Inconsistent in How Missing Data Were Handled Across Their Included Primary Trials: A Methodological Survey. Clin Epidemiol 2020; 12:527-535. [PMID: 32547244 PMCID: PMC7266325 DOI: 10.2147/clep.s242080] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/05/2023] Open
Abstract
Background How systematic review authors address missing data among eligible primary studies remains uncertain. Objective To assess whether systematic review authors are consistent in the way they handle missing data, both across trials included in the same meta-analysis, and with their reported methods. Methods We first identified 100 eligible systematic reviews that included a statistically significant meta-analysis of a patient-important dichotomous efficacy outcome. Then, we successfully retrieved 638 of the 653 trials included in these systematic reviews’ meta-analyses. From each trial report, we extracted statistical data used in the analysis of the outcome of interest to compare with the data used in the meta-analysis. First, we used these comparisons to classify the “analytical method actually used” for handling missing data by the systematic review authors for each included trial. Second, we assessed whether systematic reviews explicitly reported their analytical method of handling missing data. Third, we calculated the proportion of systematic reviews that were consistent in their “analytical method actually used” across trials included in the same meta-analysis. Fourth, among systematic reviews that were consistent in the “analytical method actually used” across trials and explicitly reported on a method for handling missing data, we assessed whether the “analytical method actually used” and the reported methods were consistent. Results We were unable to determine the “analytical method reviews actually used” for handling missing outcome data among 397 trials. Among the remaining 241, systematic review authors most commonly conducted “complete case analysis” (n=128, 53%) or assumed “none of the participants with missing data had the event of interest” (n=58, 24%). Only eight of 100 systematic reviews were consistent in their approach to handling missing data across included trials, but none of these reported methods for handling missing data. Among seven reviews that did explicitly report their analytical method of handling missing data, only one was consistent in their approach across included trials (using complete case analysis), and their approach was inconsistent with their reported methods (assumed all participants with missing data had the event). Conclusion The majority of systematic review authors were inconsistent in their approach towards reporting and handling missing outcome data across eligible primary trials, and most did not explicitly report their methods to handle missing data. Systematic review authors should clearly identify missing outcome data among their eligible trials, specify an approach for handling missing data in their analyses, and apply their approach consistently across all primary trials.
Collapse
Affiliation(s)
- Lara A Kahale
- Clinical Research Institute, American University of Beirut, Beirut, Lebanon
| | - Assem M Khamis
- Wolfson Palliative Care Research Centre, Hull York Medical School, University of Hull, Hull, UK
| | - Batoul Diab
- Clinical Research Institute, American University of Beirut, Beirut, Lebanon
| | - Yaping Chang
- Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Canada
| | - Luciane Cruz Lopes
- Pharmaceutical Sciences Post Graduate Course, University of Sorocaba, UNISO, Sorocaba, Sao Paulo, Brazil
| | - Arnav Agarwal
- Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Canada.,Department of Medicine, University of Toronto, Toronto, Ontario, Canada
| | - Ling Li
- Chinese Evidence-Based Medicine Center, West China Hospital, Sichuan University, Chengdu, People's Republic of China
| | - Reem A Mustafa
- Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Canada.,Departments of Medicine and Biomedical & Health Informatics, University of Missouri-Kansas City, Kansas City, MO, USA
| | - Serge Koujanian
- Department of Evaluative Clinical Sciences, Sunnybrook Health Sciences Centre, Toronto, Ontario, Canada
| | - Reem Waziry
- Department of Epidemiology, Harvard T.H. Chan School of Public Health, Boston, MA, USA
| | - Jason W Busse
- Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Canada.,Department of Anesthesia, McMaster University, Hamilton, Canada.,The Michael G. DeGroote National Pain Centre, McMaster University, Hamilton, Canada.,Chronic Pain Centre of Excellence for Canadian Veterans, Hamilton, Canada
| | - Abeer Dakik
- Clinical Research Institute, American University of Beirut, Beirut, Lebanon
| | - Lotty Hooft
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands
| | - Gordon H Guyatt
- Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Canada.,Department of Medicine, McMaster University, Hamilton, Canada
| | - Rob J P M Scholten
- Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, the Netherlands
| | - Elie A Akl
- Clinical Research Institute, American University of Beirut, Beirut, Lebanon.,Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Canada
| |
Collapse
|
5
|
Mostazir M, Taylor RS, Henley W, Watkins E. An overview of statistical methods for handling nonadherence to intervention protocol in randomized control trials: a methodological review. J Clin Epidemiol 2019; 108:121-131. [DOI: 10.1016/j.jclinepi.2018.12.002] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2018] [Revised: 10/29/2018] [Accepted: 12/04/2018] [Indexed: 11/24/2022]
|
6
|
Kahale LA, Diab B, Khamis AM, Chang Y, Lopes LC, Agarwal A, Li L, Mustafa RA, Koujanian S, Waziry R, Busse JW, Dakik A, Guyatt G, Akl EA. Potentially missing data are considerably more frequent than definitely missing data: a methodological survey of 638 randomized controlled trials. J Clin Epidemiol 2019; 106:18-31. [DOI: 10.1016/j.jclinepi.2018.10.001] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/14/2018] [Revised: 09/21/2018] [Accepted: 10/01/2018] [Indexed: 12/11/2022]
|
7
|
Cook JA, MacLennan GS, Palmer T, Lois N, Emsley R. Instrumental variable methods for a binary outcome were used to informatively address noncompliance in a randomized trial in surgery. J Clin Epidemiol 2017; 96:126-132. [PMID: 29157924 PMCID: PMC5862096 DOI: 10.1016/j.jclinepi.2017.11.011] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/15/2017] [Revised: 10/24/2017] [Accepted: 11/13/2017] [Indexed: 12/01/2022]
Abstract
Objectives Randomization can be used as an instrumental variable (IV) to account for unmeasured confounding when seeking to assess the impact of noncompliance with treatment allocation in a randomized trial. We present and compare different methods to calculate the treatment effect on a binary outcome as a rate ratio in a randomized surgical trial. Study Design and Setting The effectiveness of peeling versus not peeling the internal limiting membrane of the retina as part of the surgery for a full thickness macular hole. We compared the IV-based estimates (nonparametric causal bound and two-stage residual inclusion approach [2SRI]) with standard treatment effect measures (intention to treat, per protocol and treatment received [TR]). Compliance was defined in two ways (initial and up to the time point of interest). Poisson regression was used for the model-based approaches with robust standard errors to calculate the risk ratio (RR) with 95% confidence intervals. Results Results were similar for 1-month macular hole status across methods. For 3- and 6-month macular hole status, nonparametric causal bounds provided a narrower range of uncertainty than other methods, though still had substantial imprecision. For 3-month macular hole status, the TR estimate was substantially different from the other point estimates. Conclusion Nonparametric causal bound approaches are a useful addition to an IV estimation approach, which tend to have large levels of uncertainty. Methods which allow RRs to be calculated when addressing noncompliance in randomized trials exist and may be superior to standard estimates. Further research is needed to explore the properties of different IV methods in a broad range of randomized controlled trial scenarios.
Collapse
Affiliation(s)
- Jonathan A Cook
- Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Botnar Research Centre, Nuffield Orthopaedic Centre, Windmill Road, Oxford, OX3 7LD, UK.
| | - Graeme S MacLennan
- The Centre for Healthcare Randomised Trials (CHaRT), Health Sciences Building, University of Aberdeen, Foresterhill, Aberdeen, AB25 2ZD, UK
| | - Tom Palmer
- Department of Mathematics and Statistics, Fylde College, Lancaster University, Lancaster, LA1 4YF, UK
| | - Noemi Lois
- Wellcome-Wolfson Institute of Experimental Medicine, School of Medicine, Dentistry and Biomedical Sciences, Queens University, 97 Lisburn Road, Belfast, BT9 7BL, UK
| | - Richard Emsley
- Centre for Biostatistics, School of Health Sciences, Manchester Academic Health Science Centre, The University of Manchester, Jean McFarlane Building, Oxford Road, Manchester, M139PL, UK
| |
Collapse
|
8
|
Bashir R, Bourgeois FT, Dunn AG. A systematic review of the processes used to link clinical trial registrations to their published results. Syst Rev 2017; 6:123. [PMID: 28669351 PMCID: PMC5494826 DOI: 10.1186/s13643-017-0518-3] [Citation(s) in RCA: 33] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/28/2016] [Accepted: 06/09/2017] [Indexed: 02/06/2023] Open
Abstract
BACKGROUND Studies measuring the completeness and consistency of trial registration and reporting rely on linking registries with bibliographic databases. In this systematic review, we quantified the processes used to identify these links. METHODS PubMed and Embase databases were searched from inception to May 2016 for studies linking trial registries with bibliographic databases. The processes used to establish these links were categorised as automatic when the registration identifier was available in the bibliographic database or publication, or manual when linkage required inference or contacting of trial investigators. The number of links identified by each process was extracted where available. Linear regression was used to determine whether the proportions of links available via automatic processes had increased over time. RESULTS In 43 studies that examined cohorts of registry entries, 24 used automatic and manual processes to find articles; 3 only automatic; and 11 only manual (5 did not specify). Twelve studies reported results for both manual and automatic processes and showed that a median of 23% (range from 13 to 42%) included automatic links to articles, while 17% (range from 5 to 42%) of registry entries required manual processes to find articles. There was no evidence that the proportion of registry entries with automatic links had increased (R 2 = 0.02, p = 0.36). In 39 studies that examined cohorts of articles, 21 used automatic and manual processes; 9 only automatic; and 2 only manual (7 did not specify). Sixteen studies reported numbers for automatic and manual processes and indicated that a median of 49% (range from 8 to 97%) of articles had automatic links to registry entries, and 10% (range from 0 to 28%) required manual processes to find registry entries. There was no evidence that the proportion of articles with automatic links to registry entries had increased (R 2 = 0.01, p = 0.73). CONCLUSIONS The linkage of trial registries to their corresponding publications continues to require extensive manual processes. We did not find that the use of automatic linkage has increased over time. Further investigation is needed to inform approaches that will ensure publications are properly linked to trial registrations, thus enabling efficient monitoring of trial reporting.
Collapse
Affiliation(s)
- Rabia Bashir
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, NSW, 2109, Australia.
| | - Florence T Bourgeois
- Computational Health Informatics Program, Boston Children's Hospital, Boston, MA, USA.,Departments of Pediatrics and Emergency Medicine, Harvard Medical School, Boston, MA, USA
| | - Adam G Dunn
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, NSW, 2109, Australia
| |
Collapse
|
9
|
Heddle NM, Ness PM. CONSORT and clinical trial reporting: room for improvement. Transfusion 2016; 56:781-3. [PMID: 27079308 DOI: 10.1111/trf.13539] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/20/2016] [Accepted: 01/20/2016] [Indexed: 01/20/2023]
Affiliation(s)
- Nancy M Heddle
- Department of Medicine, McMaster University, Hamilton, ON, Canada
| | - Paul M Ness
- Department of Pathology, Medicine and Oncology, Johns Hopkins University School of Medicine, Baltimore, MD
| |
Collapse
|