1
|
Holst M, Carlisle BG. Trials that turn from retrospectively registered to prospectively registered: a cohort study of "retroactively prospective" clinical trial registration using history data. Trials 2024; 25:189. [PMID: 38486299 PMCID: PMC10938677 DOI: 10.1186/s13063-024-08029-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/17/2023] [Accepted: 03/01/2024] [Indexed: 03/18/2024] Open
Abstract
BACKGROUND Prospective registration of clinical trials is mandated by various regulations. However, clinical trial registries like ClinicalTrials.gov allow registry entries to be updated at any time, and key study elements, including the start date, may change before the first patient is enrolled. If a trial changes its start date after recruiting began, however, it may indicate a reason for concern. This study aimed to measure the rate of "retroactively prospective" trials. This refers to trials that are originally registered retrospectively, with the start date before the registration date, but that retroactively change their start date to be after the registration date, making them appear as if they were prospectively registered. METHODS We retrieved clinical trial history data for all clinical trials registered on ClinicalTrials.gov with a first registration date in the year 2015 (N = 11,908). Using automated analyses, we determined the timepoints of registration in relation to the start date of the trial over time. For retroactively prospective trials and a set of control trials, we manually checked the accompanying publications to determine which start date they report and whether they report changes to the start date. RESULTS We found 235 clinical trials to be retroactively prospective, comprising 2.0% of all clinical trials in our sample of 11,908 trials. Among the 113 retroactively prospective clinical trials with an accompanying publication, 12 (10.6%) explicitly stated in the publication that they had been prospectively registered. CONCLUSIONS Retroactively prospective trial registration happens in one in 50 trials. While these changes to the start date could be mistakes or legitimate edits based on the most up-to-date information, they could also indicate a retrospectively registered trial that has been made to appear as a prospectively registered trial, which would lead to biases unapparent to reviewers. Our results point to the need for more transparent reporting of changes to a trial's details and have implications for the review and conduct of clinical trials, with our fully automated and freely available tools allowing reviewers or editors to detect these changes. TRIAL REGISTRATION The preregistered protocol of our study is available via https://osf.io/rvq53 . The most recent version of the protocol lists all deviations from the original study plan, including the rationale behind the changes, and additional analyses that were conducted.
Collapse
Affiliation(s)
- Martin Holst
- Berlin Institute of Health at Charité - Universitätsmedizin Berlin, QUEST Center for Responsible Research, Anna-Louisa-Karsch-Str. 2, 10178, Berlin, Germany.
- Medizinische Hochschule Hannover, Institute for Ethics, History and Philosophy of Medicine, Carl-Neuberg-Str. 1, 30625, Hannover, Germany.
| | - Benjamin Gregory Carlisle
- Berlin Institute of Health at Charité - Universitätsmedizin Berlin, QUEST Center for Responsible Research, Anna-Louisa-Karsch-Str. 2, 10178, Berlin, Germany
- Present Address: Department of Equity, Ethics and Policy, McGill University, 2001 McGill College Avenue, Suite 1200, Montreal, QC, H3A 1G1, Canada
| |
Collapse
|
2
|
Showell M, Farquhar CM, Greenwood G, Jordan VMB. Is our public research money well spent? Publication of research outputs from Health Research Council of New Zealand-funded studies: a cross-sectional study. BMJ Open 2023; 13:e072446. [PMID: 37258081 DOI: 10.1136/bmjopen-2023-072446] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 06/02/2023] Open
Abstract
OBJECTIVE To evaluate the reporting of results from the projects and programmes funded by the Health Research Council (HRC) New Zealand. DESIGN A cross-sectional analysis. SETTING Research projects and programmes funded by the HRC New Zealand from 2006 to 2014. PARTICIPANTS Publicly available data provided by the HRC. MAIN OUTCOME MEASURES The number and proportion with evidence of publication and dissemination of a research output from HRC grants and the time taken to disseminate the results. RESULTS Of the 374 HRC grants from 2006 to 2014, there was no evidence of publication or reporting of any research output for 48 studies (13%). Of the 326 (87%) grants with research outputs, there was a mean dissemination time of 4.73 years (SD 2.37). The total funding provided by the HRC was NZ$471 663 336, while the 48 grants with no evidence of dissemination represented NZ$47 095 727 (10%). CONCLUSIONS Thirteen per cent of the HRC projects and programmes from 2006 to 2014 have not contributed to the healthcare evidence as their results remain unknown.
Collapse
Affiliation(s)
- Marian Showell
- Obstetrics and Gynaecology, Faculty of Medical and Health Sciences, The University of Auckland, Auckland, Auckland, New Zealand
| | - Cynthia M Farquhar
- Obstetrics and Gynaecology, Faculty of Medical and Health Sciences, The University of Auckland, Auckland, Auckland, New Zealand
| | - Grace Greenwood
- Obstetrics and Gynaecology, Faculty of Medical and Health Sciences, The University of Auckland, Auckland, Auckland, New Zealand
| | - Vanessa M B Jordan
- Obstetrics and Gynaecology, Faculty of Medical and Health Sciences, The University of Auckland, Auckland, Auckland, New Zealand
| |
Collapse
|
3
|
Eysenbach G, Šuster S, Baldwin T, Verspoor K. Predicting Publication of Clinical Trials Using Structured and Unstructured Data: Model Development and Validation Study. J Med Internet Res 2022; 24:e38859. [PMID: 36563029 PMCID: PMC9823568 DOI: 10.2196/38859] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2022] [Revised: 10/14/2022] [Accepted: 11/16/2022] [Indexed: 12/24/2022] Open
Abstract
BACKGROUND Publication of registered clinical trials is a critical step in the timely dissemination of trial findings. However, a significant proportion of completed clinical trials are never published, motivating the need to analyze the factors behind success or failure to publish. This could inform study design, help regulatory decision-making, and improve resource allocation. It could also enhance our understanding of bias in the publication of trials and publication trends based on the research direction or strength of the findings. Although the publication of clinical trials has been addressed in several descriptive studies at an aggregate level, there is a lack of research on the predictive analysis of a trial's publishability given an individual (planned) clinical trial description. OBJECTIVE We aimed to conduct a study that combined structured and unstructured features relevant to publication status in a single predictive approach. Established natural language processing techniques as well as recent pretrained language models enabled us to incorporate information from the textual descriptions of clinical trials into a machine learning approach. We were particularly interested in whether and which textual features could improve the classification accuracy for publication outcomes. METHODS In this study, we used metadata from ClinicalTrials.gov (a registry of clinical trials) and MEDLINE (a database of academic journal articles) to build a data set of clinical trials (N=76,950) that contained the description of a registered trial and its publication outcome (27,702/76,950, 36% published and 49,248/76,950, 64% unpublished). This is the largest data set of its kind, which we released as part of this work. The publication outcome in the data set was identified from MEDLINE based on clinical trial identifiers. We carried out a descriptive analysis and predicted the publication outcome using 2 approaches: a neural network with a large domain-specific language model and a random forest classifier using a weighted bag-of-words representation of text. RESULTS First, our analysis of the newly created data set corroborates several findings from the existing literature regarding attributes associated with a higher publication rate. Second, a crucial observation from our predictive modeling was that the addition of textual features (eg, eligibility criteria) offers consistent improvements over using only structured data (F1-score=0.62-0.64 vs F1-score=0.61 without textual features). Both pretrained language models and more basic word-based representations provide high-utility text representations, with no significant empirical difference between the two. CONCLUSIONS Different factors affect the publication of a registered clinical trial. Our approach to predictive modeling combines heterogeneous features, both structured and unstructured. We show that methods from natural language processing can provide effective textual features to enable more accurate prediction of publication success, which has not been explored for this task previously.
Collapse
Affiliation(s)
| | - Simon Šuster
- School of Computing and Information Systems, University of Melbourne, Melbourne, Australia
| | - Timothy Baldwin
- School of Computing and Information Systems, University of Melbourne, Melbourne, Australia.,Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates
| | - Karin Verspoor
- School of Computing Technologies, RMIT University, Melbourne, Australia
| |
Collapse
|
4
|
Blümle A, Wollmann K, Bischoff K, Kapp P, Lohner S, Nury E, Nitschke K, Zähringer J, Rücker G, Schumacher M. Investigator initiated trials versus industry sponsored trials - translation of randomized controlled trials into clinical practice (IMPACT). BMC Med Res Methodol 2021; 21:182. [PMID: 34465296 PMCID: PMC8406615 DOI: 10.1186/s12874-021-01359-x] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/09/2020] [Accepted: 06/24/2021] [Indexed: 12/14/2022] Open
Abstract
BACKGROUND Healthcare decisions are ideally based on clinical trial results, published in study registries, as journal articles or summarized in secondary research articles. In this research project, we investigated the impact of academically and commercially sponsored clinical trials on medical practice by measuring the proportion of trials published and cited by systematic reviews and clinical guidelines. METHODS We examined 691 multicenter, randomized controlled trials that started in 2005 or later and were completed by the end of 2016. To determine whether sponsorship/funding and place of conduct influence a trial's impact, we created four sub-cohorts of investigator initiated trials (IITs) and industry sponsored trials (ISTs): 120 IITs and 171 ISTs with German contribution compared to 200 IITs and 200 ISTs without German contribution. We balanced the groups for study phase and place of conduct. German IITs were funded by the German Research Foundation (DFG), the Federal Ministry of Education and Research (BMBF), or by another non-commercial research organization. All other trials were drawn from the German Clinical Trials Register or ClinicalTrials.gov. We investigated, to what extent study characteristics were associated with publication and impact using multivariable logistic regressions. RESULTS For 80% of the 691 trials, results were published as result articles in a medical journal and/or study registry, 52% were cited by a systematic review, and 26% reached impact in a clinical guideline. Drug trials and larger trials were associated with a higher probability to be published and to have an impact than non-drug trials and smaller trials. Results of IITs were more often published as a journal article while results of ISTs were more often published in study registries. International ISTs less often gained impact by inclusion in systematic reviews or guidelines than IITs. CONCLUSION An encouraging high proportion of the clinical trials were published, and a considerable proportion gained impact on clinical practice. However, there is still room for improvement. For publishing study results, study registries have become an alternative or complement to journal articles, especially for ISTs. IITs funded by governmental bodies in Germany reached an impact that is comparable to international IITs and ISTs.
Collapse
Affiliation(s)
- Anette Blümle
- Institute for Evidence in Medicine (for Cochrane Germany Foundation), Faculty of Medicine and Medical Center, University of Freiburg, Breisacher Str. 86, 79110 Freiburg, Germany
- Clinical Trials Unit, Faculty of Medicine and Medical Center, University of Freiburg, Elsässer Straße 2, 79110 Freiburg, Germany
| | - Katharina Wollmann
- Institute for Evidence in Medicine (for Cochrane Germany Foundation), Faculty of Medicine and Medical Center, University of Freiburg, Breisacher Str. 86, 79110 Freiburg, Germany
| | - Karin Bischoff
- Institute for Evidence in Medicine (for Cochrane Germany Foundation), Faculty of Medicine and Medical Center, University of Freiburg, Breisacher Str. 86, 79110 Freiburg, Germany
| | - Philipp Kapp
- Institute for Evidence in Medicine (for Cochrane Germany Foundation), Faculty of Medicine and Medical Center, University of Freiburg, Breisacher Str. 86, 79110 Freiburg, Germany
| | - Szimonetta Lohner
- Cochrane Hungary, Clinical Centre of the University of Pécs, Medical School, University of Pécs, Pécs, Hungary
| | - Edris Nury
- Institute for Evidence in Medicine (for Cochrane Germany Foundation), Faculty of Medicine and Medical Center, University of Freiburg, Breisacher Str. 86, 79110 Freiburg, Germany
| | - Kai Nitschke
- Institute for Evidence in Medicine (for Cochrane Germany Foundation), Faculty of Medicine and Medical Center, University of Freiburg, Breisacher Str. 86, 79110 Freiburg, Germany
| | - Jasmin Zähringer
- Institute for Evidence in Medicine (for Cochrane Germany Foundation), Faculty of Medicine and Medical Center, University of Freiburg, Breisacher Str. 86, 79110 Freiburg, Germany
| | - Gerta Rücker
- Institute of Medical Biometry and Statistics, Faculty of Medicine and Medical Center, University of Freiburg, Stefan-Meier-Str. 26, 79104 Freiburg, Germany
| | - Martin Schumacher
- Institute of Medical Biometry and Statistics, Faculty of Medicine and Medical Center, University of Freiburg, Stefan-Meier-Str. 26, 79104 Freiburg, Germany
| |
Collapse
|
5
|
Huiskens J, Kool BR, Bakker J, Bruns ER, de Jonge SW, Olthof PB, van Rosmalen BV, van Gulik TM, Hooft L, Punt CJ. From registration to publication: A study on Dutch academic randomized controlled trials. Res Synth Methods 2020; 11:218-226. [PMID: 31614063 PMCID: PMC7078864 DOI: 10.1002/jrsm.1379] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/07/2019] [Revised: 07/28/2019] [Accepted: 09/04/2019] [Indexed: 11/25/2022]
Abstract
INTRODUCTION Registration of clinical trials has been initiated in order to assess adherence of the reported results to the original trial protocol. This study aimed to investigate the publication rates, timely dissemination of results, and the prevalence of consistency in hypothesis, sample size, and primary endpoint of Dutch investigator-initiated randomized controlled clinical trials (RCTs). METHODS All Dutch investigator-initiated RCTs with a completion date between December 31, 2010, and January 1, 2012, and registered in the Trial Register of The Netherlands database were included. PubMed was searched for the publication of these RCT results until September 2016, and the time to the publication date was calculated. Consistency in hypothesis, sample size, and primary endpoint compared with the registry data were assessed. RESULTS The search resulted in a total of 168 Dutch investigator-initiated RCTs. In September 2016, the results of 129 (77%) trials had been published, of which 50 (39%) within 2 years after completion of accrual. Consistency in hypothesis with the original protocol was observed in 108 (84%) RCTs; in 71 trials (55%), the planned sample size was reached; and 103 trials (80%) presented the original primary endpoint. Consistency in all three parameters was observed in 50 studies (39%). CONCLUSION This study shows that approximately one out of four Dutch investigator-initiated RCTs remains unpublished 5 years after initiation. The observed low overall consistency with the initial study outline is a matter of concern and warrants improvements in trial design and assessment of trial feasibility.
Collapse
Affiliation(s)
- Joost Huiskens
- Department of Surgery, Amsterdam UMC, Location AMCUniversity of AmsterdamAmsterdamThe Netherlands
- Department of Medical Oncology, Amsterdam UMC, Location AMCUniversity of AmsterdamAmsterdamThe Netherlands
| | - Boudewijn R.J. Kool
- Department of Surgery, Amsterdam UMC, Location AMCUniversity of AmsterdamAmsterdamThe Netherlands
| | - Jean‐Michel Bakker
- Department of Medical Oncology, Amsterdam UMC, Location AMCUniversity of AmsterdamAmsterdamThe Netherlands
| | - Emma R.J. Bruns
- Department of Surgery, Amsterdam UMC, Location AMCUniversity of AmsterdamAmsterdamThe Netherlands
| | - Stijn W. de Jonge
- Department of Surgery, Amsterdam UMC, Location AMCUniversity of AmsterdamAmsterdamThe Netherlands
- Department of EpidemiologyHarvard T.H. Chan School of Public HealthBostonMassachusetts
| | - Pim B. Olthof
- Department of Surgery, Amsterdam UMC, Location AMCUniversity of AmsterdamAmsterdamThe Netherlands
- Department of SurgeryErasmus Medical CenterRotterdamThe Netherlands
| | - Belle V. van Rosmalen
- Department of Surgery, Amsterdam UMC, Location AMCUniversity of AmsterdamAmsterdamThe Netherlands
| | - Thomas M. van Gulik
- Department of Surgery, Amsterdam UMC, Location AMCUniversity of AmsterdamAmsterdamThe Netherlands
| | - Lotty Hooft
- Cochrane Netherlands, Julius Center for Health Sciences and Primary CareUniversity Medical Center UtrechtUtrechtThe Netherlands
| | - Cornelis J.A. Punt
- Department of Medical Oncology, Amsterdam UMC, Location AMCUniversity of AmsterdamAmsterdamThe Netherlands
| |
Collapse
|