Primary outcome switching among drug trials with and without principal investigator financial ties to industry: a cross-sectional study

Estimating the prevalence of discrepancies between study registrations and publications: a systematic review and meta-analyses

OBJECTIVES

Prospectively registering study plans in a permanent time-stamped and publicly accessible document is becoming more common across disciplines and aims to reduce risk of bias and make risk of bias transparent. Selective reporting persists, however, when researchers deviate from their registered plans without disclosure. This systematic review aimed to estimate the prevalence of undisclosed discrepancies between prospectively registered study plans and their associated publication. We further aimed to identify the research disciplines where these discrepancies have been observed, whether interventions to reduce discrepancies have been conducted, and gaps in the literature.

DESIGN

Systematic review and meta-analyses.

DATA SOURCES

Scopus and Web of Knowledge, published up to 15 December 2019.

ELIGIBILITY CRITERIA

Articles that included quantitative data about discrepancies between registrations or study protocols and their associated publications.

DATA EXTRACTION AND SYNTHESIS

Each included article was independently coded by two reviewers using a coding form designed for this review (osf.io/728ys). We used random-effects meta-analyses to synthesise the results.

RESULTS

We reviewed k=89 articles, which included k=70 that reported on primary outcome discrepancies from n=6314 studies and, k=22 that reported on secondary outcome discrepancies from n=1436 studies. Meta-analyses indicated that between 29% and 37% (95% CI) of studies contained at least one primary outcome discrepancy and between 50% and 75% (95% CI) contained at least one secondary outcome discrepancy. Almost all articles assessed clinical literature, and there was considerable heterogeneity. We identified only one article that attempted to correct discrepancies.

CONCLUSIONS

Many articles did not include information on whether discrepancies were disclosed, which version of a registration they compared publications to and whether the registration was prospective. Thus, our estimates represent discrepancies broadly, rather than our target of undisclosed discrepancies between prospectively registered study plans and their associated publications. Discrepancies are common and reduce the trustworthiness of medical research. Interventions to reduce discrepancies could prove valuable.

REGISTRATION

osf.io/ktmdg. Protocol amendments are listed in online supplemental material A.

Frequency of multiple changes to prespecified primary outcomes of clinical trials completed between 2009 and 2017 in German university medical centers: A meta-research study

BACKGROUND

Clinical trial registries allow assessment of deviations of published trials from their protocol, which may indicate a considerable risk of bias. However, since entries in many registries can be updated at any time, deviations may go unnoticed. We aimed to assess the frequency of changes to primary outcomes in different historical versions of registry entries, and how often they would go unnoticed if only deviations between published trial reports and the most recent registry entry are assessed.

METHODS AND FINDINGS

We analyzed the complete history of changes of registry entries in all 1746 randomized controlled trials completed at German university medical centers between 2009 and 2017, with published results up to 2022, that were registered in ClinicalTrials.gov or the German WHO primary registry (German Clinical Trials Register; DRKS). Data were retrieved on 24 January 2022. We assessed deviations between registry entries and publications in a random subsample of 292 trials. We determined changes of primary outcomes (1) between different versions of registry entries at key trial milestones, (2) between the latest registry entry version and the results publication, and (3) changes that occurred after trial start with no change between latest registry entry version and publication (so that assessing the full history of changes is required for detection of changes). We categorized changes as major if primary outcomes were added, dropped, changed to secondary outcomes, or secondary outcomes were turned into primary outcomes. We also assessed (4) the proportion of publications transparently reporting changes and (5) characteristics associated with changes. Of all 1746 trials, 23% (n = 393) had a primary outcome change between trial start and latest registry entry version, with 8% (n = 142) being major changes, that is, primary outcomes were added, dropped, changed to secondary outcomes, or secondary outcomes were turned into primary outcomes. Primary outcomes in publications were different from the latest registry entry version in 41% of trials (120 of the 292 sampled trials; 95% confidence interval (CI) [35%, 47%]), with major changes in 18% (54 of 292; 95% CI [14%, 23%]). Overall, 55% of trials (161 of 292; 95% CI [49%, 61%]) had primary outcome changes at any timepoint over the course of a trial, with 23% of trials (67 of 292; 95% CI [18%, 28%]) having major changes. Changes only within registry records, with no apparent discrepancy between latest registry entry version and publication, were observed in 14% of trials (41 of 292; 95% CI [10%, 19%]), with 4% (13 of 292; 95% CI [2%, 7%]) being major changes. One percent of trials with a change reported this in their publication (2 of 161 trials; 95% CI [0%, 4%]). An exploratory logistic regression analysis indicated that trials were less likely to have a discrepant registry entry if they were registered more recently (odds ratio (OR) 0.74; 95% CI [0.69, 0.80]; p<0.001), were not registered on ClinicalTrials.gov (OR 0.41; 95% CI [0.23, 0.70]; p = 0.002), or were not industry-sponsored (OR 0.29; 95% CI [0.21, 0.41]; p<0.001). Key limitations include some degree of subjectivity in the categorization of outcome changes and inclusion of a single geographic region.

CONCLUSIONS

In this study, we observed that changes to primary outcomes occur in 55% of trials, with 23% trials having major changes. They are rarely transparently reported in the results publication and often not visible in the latest registry entry version. More transparency is needed, supported by deeper analysis of registry entries to make these changes more easily recognizable. Protocol registration: Open Science Framework (https://osf.io/t3qva; amendment in https://osf.io/qtd2b).

Completeness and consistency of primary outcome reporting in COVID-19 publications in the early pandemic phase: a descriptive study

BACKGROUND

The COVID-19 pandemic saw a steep increase in the number of rapidly published scientific studies, especially early in the pandemic. Some have suggested COVID-19 trial reporting is of lower quality than typical reports, but there is limited evidence for this in terms of primary outcome reporting. The objective of this study was to assess the prevalence of completely defined primary outcomes reported in registry entries, preprints, and journal articles, and to assess consistent primary outcome reporting between these sources.

METHODS

This is a descriptive study of a cohort of registered interventional clinical trials for the treatment and prevention of COVID-19, drawn from the DIssemination of REgistered COVID-19 Clinical Trials (DIRECCT) study dataset. The main outcomes are: 1) Prevalence of complete primary outcome reporting; 2) Prevalence of consistent primary outcome reporting between registry entry and preprint as well as registry entry and journal article pairs.

RESULTS

We analyzed 87 trials with 116 corresponding publications (87 registry entries, 53 preprints and 63 journal articles). All primary outcomes were completely defined in 47/87 (54%) registry entries, 31/53 (58%) preprints and 44/63 (70%) journal articles. All primary outcomes were consistently reported in 13/53 (25%) registry-preprint pairs and 27/63 (43%) registry-journal article pairs. No primary outcome was specified in 13/53 (25%) preprints and 8/63 (13%) journal articles. In this sample, complete primary outcome reporting occurred more frequently in trials with vs. without involvement of pharmaceutical companies (76% vs. 45%), and in RCTs vs. other study designs (68% vs. 49%). The same pattern was observed for consistent primary outcome reporting (with vs. without pharma: 56% vs. 12%, RCT vs. other: 43% vs. 22%).

CONCLUSIONS

In COVID-19 trials in the early phase of the pandemic, all primary outcomes were completely defined in 54%, 58%, and 70% of registry entries, preprints and journal articles, respectively. Only 25% of preprints and 43% of journal articles reported primary outcomes consistent with registry entries.

Publication of neurology clinical trials registered with clinical trial registry of India: A cross-sectional study

OBJECTIVES

Unbiased and full disclosure of trial results is vital to evidence-based medicine. Non-publication and selective publication leads to publication bias and unrealistic risk-benefit ratio. In the present study, we aim to determine the publication rate of clinical trials related to neurology registered with the Clinical Trial Registry of India (CTRI), compare the characteristics of published and unpublished trials, and evaluate the adherence of investigators to ethics-approved criteria and outcomes.

MATERIALS AND METHODS

A cross-sectional search using the keyword "neurology" was carried out in CTRI registry. Two independent investigators searched Pubmed, Medline, Scopus, and Google Scholar for published manuscripts. The final literature search occurred in November 2021.

RESULTS

Out of 325 trials, 102 trials were published (31.4%). Ninety-one trials were beyond 3 years of expected time of trial completion and were still unpublished. Randomized trials had a slightly higher publication rate than non-randomized ones (56% vs. 46%, p = .223); however the difference was not statistically significant. Majority of trials sponsored by pharmaceutical companies were not published, while majority of those sponsored by non-pharmaceutical institutions were published (34.5% vs. 69.3%, p < .001). Feedback to CTRI about trial status was particularly poor (31.5% - informed vs. 68.5% - not informed, p < .001). 52 (50.9%) and 65 (63.7%) of the 102 published trials had changed the registered inclusion and exclusion criteria, respectively, in the CTRI registry compared to those in the published manuscript. In 29 (28.3%) of the 102 trials, the primary outcome did not match with that registered in the CTRI and in 73 (57.8%) trials, the secondary outcomes did not match.

CONCLUSION

A large proportion of neurology registered trials are still unpublished, with a majority of pharmaceutical company-sponsored trials not being published. There is scope for improving the provisions in CTRI for enlisting trial results, that may prevent publication bias and also ensure the investigators adhere to the pre-specified ethics approved trial procedures and outcomes.

Broad Medical Uncertainty and the ethical obligation for openness

This paper argues that there exists a collective epistemic state of 'Broad Medical Uncertainty' (BMU) regarding the effectiveness of many medical interventions. We outline the features of BMU, and describe some of the main contributing factors. These include flaws in medical research methodologies, bias in publication practices, financial and other conflicts of interest, and features of how evidence is translated into practice. These result in a significant degree of uncertainty regarding the effectiveness of many medical treatments and unduly optimistic beliefs about the benefit/harm profiles of such treatments. We argue for an ethical presumption in favour of openness regarding BMU as part of a 'Corrective Response'. We then consider some objections to this position (the 'Anti-Corrective Response'), including concerns that public honesty about flaws in medical research could undermine trust in healthcare institutions. We suggest that, as it stands, the Anti-Corrective Response is unconvincing.

The Unknown Prevalence of Postrandomization Bias in 15 Physical Therapy Journals: A Methods Review

OBJECTIVES

To determine the prevalence of prospective clinical trial registration and postrandomization bias in published musculoskeletal physical therapy randomized clinical trials (RCTs).

DESIGN

A methods review.

LITERATURE SEARCH

Articles indexed in MEDLINE and published between January 2016 and July 2020 were included.

STUDY SELECTION CRITERIA

Two independent blinded reviewers identified the RCTs using Covidence. We included RCTs related to musculoskeletal interventions that were published in International Society of Physiotherapy Journal Editors member journals.

DATA SYNTHESIS

Data were extracted independently for the variables of interest from the identified RCTs by 2 blinded reviewers. The data were presented descriptively or in frequency tables.

RESULTS

One hundred thirty-eight RCTs were identified. One third of RCTs were consistent with their prospectively registered intent (49/138); consistency with prospectively registered intent could not be determined for two thirds (89/138) of the RCTs. Four RCTs (8%)reported inconsistent results with the primary aims and 7 (14%) with the outcomes from the prospective clinical trial registry, despite high methodological quality (Physiotherapy Evidence Database [PEDro] scale score). Differences between prospectively registered and non-prospectively registered RCTs for PEDro scale scores had a medium effect size (r = 0.30). Two of 15 journals followed their clinical trial registration policy 100% of the time; in 1 journal, the published RCTs were consistent with the clinical trial registration.

CONCLUSION

Postrandomization bias in musculoskeletal physical therapy RCTs could not be ruled out, due to the lack of prospective clinical trial registration and detailed data analysis plans. J Orthop Sports Phys Ther 2021;51(11):542-550. Epub 21 Sep 2021. doi:10.2519/jospt.2021.10491.

Association between switching of primary outcomes and reported trial findings among randomized drug trials from China

BACKGROUND AND OBJECTIVE

To evaluate the association between the nature of findings and the switching of registered primary outcomes among randomized controlled trials (RCTs) from mainland China.

METHODS

This is a retrospective cohort study. We retrieved RCTs from trial registries and identified the corresponding journal articles from bibliographic databases until August 2019. Trial registries and journal articles were compared to evaluate whether registered primary outcomes with negative findings were more likely to be switched to secondary outcomes in the subsequent journal articles than those with positive findings.

RESULTS

Switching of registered primary outcomes occurred in 131 (45%) of 294 RCTs. A total of 450 registered primary outcomes were matched to 522 (37%) primary outcomes and 871 (63%) secondary outcomes in the journal articles. Among RCTs registered before they started, the odds of switching primary outcomes with negative findings were 2.64 (95% CI: 1.16-6.02) times the odds of switching those with positive findings. Among RCTs registered when they were ongoing, the odds of switching primary outcomes with negative findings were 8.84 (95% CI: 3.62-25.93) times the odds of switching those with positive findings.

CONCLUSION

The nature of findings may play a role in how likely a prespecified primary outcome is switched subsequently.

Changes to aspects of ongoing randomised controlled trials with fixed designs

Background

Despite careful planning, changes to some aspects of an ongoing randomised clinical trial (RCT), with a fixed design, may be warranted. We sought to elucidate the distinction between legitimate versus illegitimate changes to serve as a guide for less experienced clinical trialists and other stakeholders.

Methods

Using data from a large trial of statin therapy for secondary prevention, we generated a set of simulated trial datasets under the null hypothesis (H0) and a set under an alternative hypothesis (H1). Through analysis of these simulated trials, we assessed the performance of the strategy of changing aspects of the design/analysis with knowledge of treatment allocation (illegitimate) versus the strategy of making changes without knowledge of treatment allocation (legitimate). Performance was assessed using the type 1 error, as well as measures of absolute and relative bias in the treatment effect.

Results

Illegitimate changes led to a relative bias of 61% under H1, and a type 1 error rate under H0 of 23%—well in excess of the 5% significance level targeted. Legitimate changes produced unbiased estimates under H1 and did not inflate the type 1 error rate under H0.

Conclusions

Changes to pre-specified aspects of the design and analysis of an ongoing RCT may be a necessary response to unforeseen circumstances. Such changes risk introducing a bias if undertaken with knowledge of treatment allocation. Legitimate changes need to be adequately documented to provide assurance to all stakeholders of their validity.

Conflicts of interest in randomized controlled trials reported in neurosurgical journals

OBJECTIVE

Randomized controlled trials (RCTs) form the basis of today's evidence-based approach to medicine, and play a critical role in guidelines and the drug and device approval process. Conflicts of interest (COIs) are commonplace in medical research, but little is known about their influence. The authors aimed to evaluate the extent and influence of COIs in recent RCTs published in core neurosurgical journals using a cross-sectional analysis.

METHODS

Through review of 6 general neurosurgical journals, all interventional RCTs published from January 2009 to January 2019 were identified. Because it is difficult to objectively assess study outcome, the authors opted for a strict rating approach based on the statistical significance of unambiguously reported primary endpoints, and the reported statistical protocol.

RESULTS

A total of 129 RCTs met the inclusion criteria. During the study period, the Journal of Neurosurgery published the largest number of RCTs (n = 40, 31%). Any potential COI was disclosed by 57%, and a mean of 12% of authors had a personal COI. Nonfinancial industry involvement was reported in 10%, while 31% and 20% received external support and sponsoring, respectively. Study registration was reported by 56%, while 51% of studies were blinded. Registration showed an increasing trend from 17% to 76% (p < 0.001). The median randomized sample size was 92 (interquartile range 50-153), and 8% were designed to investigate noninferiority or equality. Sixty-three RCTs (49%) unambiguously reported a primary endpoint, of which 13% were composite primary endpoints. In 43%, study outcome was positive, which was associated with a noninferiority design (31% vs 3%, p = 0.007) and a composite primary endpoint (46% vs 9%, p = 0.002). Potential COIs were not significantly associated with study positivity (69% vs 59%, p = 0.433). In the multivariate analysis, only a composite primary endpoint remained predictive of a positive study outcome (odds ratio 6.34, 95% confidence interval 1.51-33.61, p = 0.017).

CONCLUSIONS

This analysis provides an overview of COIs and their potential influence on recent trials published in core neurosurgical journals. Reporting of primary endpoints, study registration, and uniform disclosure of COIs are crucial to ensure the quality of future neurosurgical randomized trials. COIs do not appear to significantly influence the outcome of randomized neurosurgical trials.

Comparison of Clinical Trial Changes in Primary Outcome and Reported Intervention Effect Size Between Trial Registration and Publication

IMPORTANCE

Primary outcome change could threaten the validity of a clinical trial; however, evidence about the consequences on the reported intervention effect size is unclear.

OBJECTIVES

To examine the status of randomized clinical trials whose primary outcome changed between trial registration and publication and to quantify the association of this change with the reported intervention effect size.

DESIGN, SETTING, AND PARTICIPANTS

In this cross-sectional study on the primary report of randomized clinical trials with clear prospectively registered primary outcomes, PubMed and Embase were searched for articles published between January 1, 2011, and December 31, 2015. The search was conducted in January 2016, identifying randomized clinical trials and the combination of keywords and text words related to registry.

MAIN OUTCOMES AND MEASURES

Based on the developed approach, trials were classified as having primary outcome change when there was a major discrepancy between the registered and published primary outcomes. Intervention effect was estimated or recalculated using the odds ratio (OR) for each comparison. Each component OR is structured so that an OR is less than 1 if the intervention group has a more favorable result than the control group. The ratio of ORs (ROR), which is the summary OR for trials with primary outcome change divided by those without, and its 95% CI were calculated, with a value less than 1 indicating a larger reported intervention effect size in trials with primary outcome change than those without.

RESULTS

Among 29 749 searched articles (28 810 MEDLINE and 939 Embase), 1488 articles were randomly selected for review. Of 389 trials with clear primary outcomes prospectively described in the registry (416 outcomes reported), 33.4% (130 of 389) of trials had at least 1 primary outcome change. Most (66 of 130) of the changes were either not reporting or omitting the primary outcome. In total, 338 trials (365 outcomes and 487 comparisons) were available for quantitative analysis on the reported intervention effect size bias assessment. Compared with those without primary outcome change, trials with primary outcome change showed a 16% (pooled ROR, 0.84; 95% CI, 0.73-0.96) larger reported intervention effect size. The result persisted after adjustment for potential confounders (ROR, 0.81; 95% CI, 0.71-0.93) and other sensitivity and subgroup analyses.

CONCLUSIONS AND RELEVANCE

Results of this study suggest that inconsistencies between registered and published primary outcomes of clinical trials are common, and trials with primary outcome change are likely to have a larger intervention effect than those without.

Peer reviewed evaluation of registered end-points of randomised trials (the PRE-REPORT study): protocol for a stepped-wedge, cluster-randomised trial

INTRODUCTION

Clinical trials are critical to the advancement of medical knowledge. However, the reliability of trial conclusions depends in part on consistency between pre-planned and reported study outcomes. Unfortunately, selective outcome reporting, in which outcomes reported in published manuscripts differ from pre-specified study outcomes, is common. Trial registries such as ClinicalTrials.gov have the potential to help identify and stop selective outcome reporting during peer review by allowing peer reviewers to compare outcomes between registry entries and submitted manuscripts. However, the persistently high rate of selective outcome reporting among published clinical trials indicates that the current peer review process at most journals does not effectively address the problem of selective outcome reporting.

METHODS AND ANALYSIS

PRE-REPORT is a stepped-wedge cluster-randomised trial that will test whether providing peer reviewers with a summary of registered, pre-specified primary trial outcomes decreases inconsistencies between prospectively registered and published primary outcomes. Peer reviewed manuscripts describing clinical trial results will be included. Eligible manuscripts submitted to each participating journal during the study period will comprise each cluster. After an initial control phase, journals will transition to the intervention phase in random order, after which peer reviewers will be emailed registry information consisting of the date of registration and any prospectively defined primary outcomes. Blinded outcome assessors will compare registered and published primary outcomes for all included trials. The primary PRE-REPORT outcome is the presence of a published primary outcome that is consistent with a prospectively defined primary outcome in the study's trial registry. The primary outcome will be analysed using a mixed effect logistical regression model to compare results between the intervention and control phases.

ETHICS AND DISSEMINATION

The Cooper Health System Institutional Review Board determined that this study does not meet criteria for human subject research. Findings will be published in peer-reviewed journals.

TRIAL REGISTRATION NUMBER

ISRCTN41225307; Pre-results.