Research protocols: waiving confidentiality for the greater good

Reporting completeness of nutrition and diet-related randomised controlled trials protocols

BACKGROUND AND AIMS

There is a need to consolidate reporting guidance for nutrition randomised controlled trial (RCT) protocols. The reporting completeness in nutrition RCT protocols and study characteristics associated with adherence to SPIRIT and TIDieR reporting guidelines are unknown. We, therefore, assessed reporting completeness and its potential predictors in a random sample of published nutrition and diet-related RCT protocols.

METHODS

We conducted a meta-research study of 200 nutrition and diet-related RCT protocols published in 2019 and 2021 (aiming to consider periods before and after the start of the COVID pandemic). Data extraction included bibliometric information, general study characteristics, compliance with 122 questions corresponding to items and subitems in the SPIRIT and TIDieR checklists combined, and mention to these reporting guidelines in the publications. We calculated the proportion of protocols reporting each item and the frequency of items reported for each protocol. We investigated associations between selected publication aspects and reporting completeness using linear regression analysis.

RESULTS

The majority of protocols included adults and elderly as their study population (n = 73; 36.5%), supplementation as intervention (n = 96; 48.0%), placebo as comparator (n = 89; 44.5%), and evaluated clinical status as the outcome (n = 80; 40.0%). Most protocols described a parallel RCT (n = 188; 94.0%) with a superiority framework (n = 141; 70.5%). Overall reporting completeness was 52.0% (SD = 10.8%). Adherence to SPIRIT items ranged from 0% (n = 0) (data collection methods) to 98.5% (n = 197) (eligibility criteria). Adherence to TIDieR items ranged from 5.5% (n = 11) (materials used in the intervention) to 98.5% (n = 197) (description of the intervention). The multivariable regression analysis suggests that a higher number of authors [β = 0.53 (95%CI: 0.28-0.78)], most recent published protocols [β = 3.19 (95%CI: 0.24-6.14)], request of reporting guideline checklist during the submission process by the journal [β = 6.50 (95%CI: 2.56-10.43)] and mention of SPIRIT by the authors [β = 5.15 (95%CI: 2.44-7.86)] are related to higher reporting completeness scores.

CONCLUSIONS

Reporting completeness in a random sample of 200 diet or nutrition-related RCT protocols was low. Number of authors, year of publication, self-reported adherence to SPIRIT, and journals' endorsement of reporting guidelines seem to be positively associated with reporting completeness in nutrition and diet-related RCT protocols.

Not enough SPIRIT shown in the registration and reporting of orthodontic trial protocols

BACKGROUND

To reduce bias associated with selective reporting, the registration and publication of clinical trial protocols before or at the time of patient enrolment has been advocated. The aim of this investigation was to assess the frequency of registration and reporting adherence of orthodontic trial protocols pre- and post-introduction of the Standard Protocol Items Recommendations for Interventional Trials (SPIRIT) Statement.

MATERIALS AND METHOD

Trial protocols registered in four online registries were sourced at two time periods: (1 January 2010-1 January 2013) and (1 January 2017-1 January 2021). Protocols were screened and data extracted, in duplicate and independently. The reporting adherence of each protocol was assessed in relation to the thirty-three item SPIRIT statement. Fisher's exact test was used to determine associations between time periods and trial protocol characteristics. Median regression was implemented to assess potential associations between the percent score per protocol and protocol characteristics.

RESULTS

A total of 100 protocols were analysed. Thirty-three and sixty-seven protocols were registered in the first and second time periods, respectively. An association between period and the timing of registration (prospectively or retrospectively) (P < 0.001) and funding source (University or Company) (P < 0.001) was evident. Overall, 25 of the 33 (75.5%) SPIRIT statement items were not reported in either timeframe. The median percent reporting quality score was 26.9 (IQR 6.9). The type of registry was associated with percent scores and published studies received better percent scores compared to unpublished studies and academic or private protocol submissions.

CONCLUSIONS

There is a general lack of awareness of the importance and relevance of the SPIRIT statement. Registration of orthodontic trial protocols has apparently improved; however, 75.5% SPIRIT statement items were not reported in either study time period. The registration and reporting of orthodontic trial protocols should be advocated to circumvent issues relating to selective reporting and outcome reporting bias.

Protocol for a meta-research study of protocols for diet or nutrition-related trials published in indexed journals: general aspects of study design, rationale and reporting limitations

INTRODUCTION

The Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) reporting guideline establishes a minimum set of items to be reported in any randomised controlled trial (RCT) protocol. The Template for Intervention Description and Replication (TIDieR) reporting guideline was developed to improve the reporting of interventions in RCT protocols and results papers. Reporting completeness in protocols of diet or nutrition-related RCTs has not been systematically investigated. We aim to identify published protocols of diet or nutrition-related RCTs, assess their reporting completeness and identify the main reporting limitations remaining in this field.

METHODS AND ANALYSIS

We will conduct a meta-research study of RCT protocols published in journals indexed in at least one of six selected databases between 2012 and 2022. We have run a search in PubMed, Embase, CINAHL, Web of Science, PsycINFO and Global Health using a search strategy designed to identify protocols of diet or nutrition-related RCTs. Two reviewers will independently screen the titles and abstracts of records yielded by the search in Rayyan. The full texts will then be read to confirm protocol eligibility. We will collect general study features (publication information, types of participants, interventions, comparators, outcomes and study design) of all eligible published protocols in this contemporary sample. We will assess reporting completeness in a randomly selected sample of them and identify their main reporting limitations. We will compare this subsample with the items in the SPIRIT and TIDieR statements. For all data collection, we will use data extraction forms in REDCap. This protocol is registered on the Open Science Framework (DOI: 10.17605/OSF.IO/YWEVS).

ETHICS AND DISSEMINATION

This study will undertake a secondary analysis of published data and does not require ethical approval. The results will be disseminated through journals and conferences targeting stakeholders involved in nutrition research.

Systematic review and narrative review lead experts to different cancer trial predictions: a randomized trial

OBJECTIVES

The objective of the study was to assess the impact of narrative review (NR) vs. systematic review (SR) on expert assessments of a clinical trial.

STUDY DESIGN AND SETTING

Experts in colon and rectal surgery were randomized to read an NR or SR for an ongoing clinical trial involving surgery for colorectal cancer. Experts from the United States and Canada completed online or paper surveys between December 2018 and June 2019. After reading the NR or SR, experts predicted the trial's outcome and evaluated the trial under a hypothetical ethical review.

RESULTS

Experts who read the NR (n = 55) compared with those who read the SR (n = 56) were more likely to predict a higher absolute risk reduction, 58% vs. 33%, P = 0.018, mean predictions 10.6% vs. 6.6%, mean difference 4.0% [95% confidence interval: 0.3%, 7.8%]. Experts who read the NR were more likely to evaluate the trial more favorably under a hypothetical ethical review, 48% vs. 26%, P = 0.039, 20.0% vs. 8.9% "strongly in favor" of trial being pursued.

CONCLUSION

An NR and an SR for the same trial led to different judgments of likely outcomes and ethical appropriateness. These differences point to a potential source of unaddressed bias in ethical review.

Investigator brochures for phase I/II trials lack information on the robustness of preclinical safety studies

AIM

Meaningful and ethical phase I/II trials can only be conducted with supportive prospective risk-benefit assessment. This relies largely on preclinical animal studies addressing the safety and efficacy of treatments. These studies are reported in an Investigator's Brochure (IB) to inform ethics review boards and regulatory authorities. Our study investigated the extent, reporting quality and accessibility of preclinical safety studies (PCSSs) compiled in IBs.

METHODS

We analysed a sample of 46 IBs for phase I/II trials approved at a leading German university medical centre from 2010 to 2016. We extracted all PCSSs presented in the 46 IBs and assessed them for reporting on methodological measures to reduce validity threats.

RESULTS

The 46 IBs included 777 PCSSs. Blinded outcome assessment, randomization and sample size calculation were reported for fewer than 1% of studies. Only 5% of the PCSSs provided a reference to published data. Compliance with Good Laboratory Practice (GLP) guidance was reported for 52% of PCSSs, but the GLP document itself does not include any relevant methodological requirements for the reduction of validity threats.

CONCLUSION

Scarce reporting in IBs and the very limited publicly available data on PCSSs make it almost impossible for investigators to critically evaluate the robustness of preclinical evidence of drug safety. Combined with recent findings on the presentation of preclinical efficacy studies in IBs, we conclude that the current reporting patterns in IBs strongly limit the independent review of evidential support for early human trials. Regulatory authorities and IRBs should require better reporting in IBs.

Has the reporting quality of published randomised controlled trial protocols improved since the SPIRIT statement? A methodological study

OBJECTIVES

To determine the reporting quality of published randomised controlled trial (RCT) protocols before and after the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) statement (2013), and any association with author, trial or journal factors.

DESIGN

Methodological study.

DATA SOURCES

MEDLINE, Embase and CENTRAL were electronically searched using optimised search strategies.

ELIGIBILITY CRITERIA

Protocols written for an RCT of living humans, published in full text in a peer-reviewed journal and published in the English language.

MAIN OUTCOME

Primary outcome was the overall proportion of checklist items which were adequately reported in RCT protocols published before and after the SPIRIT statement.

RESULTS

300 RCT protocols were retrieved; 150 from the period immediately before the SPIRIT statement (9 July 2012 to 28 December 2012) and 150 from a recent period after the SPIRIT statement (25 January 2019 to 20 March 2019). 47.9% (95% CI, 46.5% to 49.3%) of checklist items were adequately reported in RCT protocols before the SPIRIT statement and 56.7% (95% CI, 54.9% to 58.5%) after the SPIRIT statement. This represents an 8.8% (95% CI, 6.6% to 11.1%; p<0.0001) mean improvement in the overall proportion of checklist items adequately reported since the SPIRIT statement. While 40% of individual checklist items had a significant improvement in adequate reporting after the SPIRIT statement, 11.3% had a significant deterioration and there were no RCT protocols in which all individual checklist items were complete. The factors associated with higher reporting quality of RCT protocols in multiple regression analysis were author expertise or experience in epidemiology or statistics, multicentre trials, longer protocol word length and publicly reported journal policy of compliance with the SPIRIT statement.

CONCLUSION

The overall reporting quality of RCT protocols has significantly improved since the SPIRIT statement, although a substantial proportion of individual checklist items remain poorly reported. Continued and concerted efforts are required by journals, editors, reviewers and investigators to improve the completeness and transparency of RCT protocols.

The clinical trial transparency in oncology significantly increased over the recent years

OBJECTIVE

Persistent efforts have been made to promote clinical trial transparency, which included encouraging trial registration and prospective registration, as well as protocol disclosure. This study aims to analyze the extent of registration, prospective registration and protocol disclosure in oncology clinical trials and their changing trends.

STUDY DESIGN AND SETTING

All phase II and phase III oncology clinical trials published in 5 major journals, the Annals of Oncology, the Journal of Clinical Oncology, JAMA Oncology, The Lancet Oncology, and The New England Journal of Medicine, between January 2013 and December 2017, were included. Data on trial characteristics as well as registration status and availability of protocol and its location were collected.

RESULTS

In total, 625 articles were included, 92% were registered, of which 76% were prospectively registered. Overall, 27% provided protocols. Increasing trends were observed in registration, prospective registration, and protocol disclosure (all P < 0.001). Studies with enrollment number larger than median number were more likely to be registered (adjusted odds ratio (aOR), 3.14 [95% confidence interval (CI), 1.21-8.15]) and to provide protocols (aOR, 3.84 [95% CI, 2.24-6.57]) than those with smaller enrollment number. Studies with nonindustry funding was less likely to be prospectively registered (aOR, 0.37 [95% CI, 0.25-0.55]) but more likely to provide protocols (aOR, 1.69 [95% CI, 1.13-2.52]) than those with industry funding only.

CONCLUSION

Although the rates of registration, prospective registration, and protocol disclosure of oncology trials have significantly increased over the years, there is still room for improvement.

Comparison between protocols and publications for prognostic and predictive cancer biomarker studies

BACKGROUND

Method prespecification in study protocols is important for controlling bias in reports. The primary goal of this study was to assess potential for discordance between study protocols and publications reporting predictive or prognostic cancer biomarker research. Secondary objectives included comparing characteristics of publications with accessible protocols compared to those without.

METHODS

Publications reporting predictive or prognostic cancer biomarker research were identified from 15 major journals, 2012-2015. Protocols were sought online or through repeated queries of corresponding authors. The following four items were extracted: (1) biomarkers, (2) biospecimen/assays, (3) sample size, (4) endpoints. We defined "explicit discordance" as the presence of major inconsistencies on these items.

RESULTS

Of 149 eligible publications, we obtained 19 eligible protocols online (13%). Out of a random sample of 103 publications where protocols were not available online, 12 protocols (12%) were furnished by corresponding authors; 8 (8% of authors) explicitly stated the absence of a protocol. Among 24 retrospective cohort studies, no protocol could be accessed. We found explicit discordance between publications and protocols for 18 studies (58%), in particular choice of biomarkers (36%), biospecimen/assays (6%), or endpoints (29%).

CONCLUSION

Protocols are generally not accessible or not used for cancer biomarker studies. Publications were often explicitly discordant with protocols, particularly regarding biomarkers and endpoints. Our findings point to common unaddressed risk of bias in publications of major journals reporting the relationship between cancer biomarkers and clinical endpoints.

Cross-sectional analysis of UK research studies in 2015: results from a scoping project with the UK Health Research Authority

OBJECTIVES

To determine whether data on research studies held by the UK Health Research Authority (HRA) could be summarised automatically with minimal manual intervention. There are numerous initiatives to reduce research waste by improving the design, conduct, analysis and reporting of clinical studies. However, quantitative data on the characteristics of clinical studies and the impact of the various initiatives are limited.

DESIGN

Feasibility study, using 1 year of data.

SETTING

We worked with the HRA on a pilot study using research applications submitted for UK-wide ethical review. We extracted into a single dataset, information held in anonymised XML files by the Integrated Research Application System (IRAS) and the HRA Assessment Review Portal (HARP). Research applications from 2014 to 2016 were provided. We used standard text extraction methods to assess information held in free-text fields. We use simple, descriptive methods to summarise the research activities that we extracted.

PARTICIPANTS

Not applicable-records-based study INTERVENTIONS: Not applicable.

PRIMARY AND SECONDARY OUTCOME MEASURES

Feasibility of extraction and processing.

RESULTS

We successfully imported 1775 non-duplicate research applications from the XML files into a single database. Of these, 963 were randomised controlled trials and 812 were other studies. Most studies received a favourable opinion. There was limited patient and public involvement in the studies. Most, but not all, studies were planned for publication of results. Novel study designs (eg, adaptive and Bayesian designs) were infrequently reported.

CONCLUSIONS

We have demonstrated that the data submitted from IRAS to the HRA and its HARP system are accessible and can be queried for information. We strongly encourage the development of fully resourced collaborative projects to further this work. This would aid understanding of how study characteristics change over time and across therapeutic areas, as well as the progress of initiatives to improve the quality and relevance of research studies.

Preclinical efficacy studies in investigator brochures: Do they enable risk-benefit assessment?

Human protection policies require favorable risk–benefit judgments prior to launch of clinical trials. For phase I and II trials, evidence for such judgment often stems from preclinical efficacy studies (PCESs). We undertook a systematic investigation of application materials (investigator brochures [IBs]) presented for ethics review for phase I and II trials to assess the content and properties of PCESs contained in them. Using a sample of 109 IBs most recently approved at 3 institutional review boards based at German Medical Faculties between the years 2010–2016, we identified 708 unique PCESs. We then rated all identified PCESs for their reporting on study elements that help to address validity threats, whether they referenced published reports, and the direction of their results. Altogether, the 109 IBs reported on 708 PCESs. Less than 5% of all PCESs described elements essential for reducing validity threats such as randomization, sample size calculation, and blinded outcome assessment. For most PCESs (89%), no reference to a published report was provided. Only 6% of all PCESs reported an outcome demonstrating no effect. For the majority of IBs (82%), all PCESs were described as reporting positive findings. Our results show that most IBs for phase I/II studies did not allow evaluators to systematically appraise the strength of the supporting preclinical findings. The very rare reporting of PCESs that demonstrated no effect raises concerns about potential design or reporting biases. Poor PCES design and reporting thwart risk–benefit evaluation during ethical review of phase I/II studies.

To make a clinical trial ethical, regulatory agencies and institutional review boards have to judge whether the trial-related benefits (the knowledge gain) outweigh the trial-inherent risks. For early-phase human research, these risk–benefit assessments are often based on evidence from preclinical animal studies reported in so-called “investigator brochures.” However, our analysis shows that the vast majority of such investigator brochures lack sufficient information to systematically appraise the strength of the supporting preclinical findings. Furthermore, the very rare reporting of preclinical efficacy studies that demonstrated no effect raises concerns about potential design and/or reporting biases. The poor preclinical study design and reporting thwarts risk–benefit evaluation during ethical review of early human research. Regulators should develop standards for the design and reporting of preclinical efficacy studies in order to support the conduct of ethical clinical trials.

Promoting public access to clinical trial protocols: challenges and recommendations

BACKGROUND

Recognizing the value of promoting public access to clinical trial protocols, Trials pioneered the way for their publication over a decade ago. However, despite major advances in the public accessibility of information about trial methods and results, protocol sharing remains relatively rare.

MAIN BODY

Protocol sharing facilitates the critical appraisal of clinical trials and helps to identify and deter the selective reporting of outcomes and analyses. Challenges to the routine availability of high quality trial protocols include the gaps in incentives and adherence mechanisms, limited venues for sharing the original and final protocol versions, and the need for mechanisms to ensure transparent and complete protocol content.

CONCLUSIONS

We propose recommendations for addressing key challenges to protocol sharing in order to promote routine public access to protocols for the benefit of patients and other users of evidence from clinical trials.

Trends in the number and the quality of trial protocols involving children submitted to a French Institutional Review Board

Background

There is a great need for high quality clinical research for children. The European Pediatric Regulation aimed to improve the quality of clinical trials in order to increase the availability of treatments for children. The main purpose of this study was to assess the evolution of both the number and the quality of pediatric trial protocols that were submitted to a French Institutional Review Board (IRB00009118) before and after the initiation of the EU Pediatric Regulation.

Methods

All protocols submitted to the IRB00009118 between 2003 and 2014 and conducting research on subjects under eighteen years of age were eligible. The quality of randomized clinical trials was assessed according to the guidelines developed by the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network and ranked using the Jadad score.

Results

Out of 622 protocols submitted to the Institutional Review Board (IRB), 21% (133/622) included children. Among these 133 pediatric protocols, the number of submitted pediatric protocols doubled between the two studied periods. From 2003 to 2008, 47 protocols including 21 institutionally sponsored were submitted to the IRB and from 2009 until 2014, 86 protocols including 48 institutionally sponsored were submitted. No significant trend was observed on the quality of RCTs. The overall median score of RCTs on the Jadad scale was high (3.5), 70.0% of protocols had a Jadad score ≥ 3, and 30.0% had a score < 3.

Conclusion

Following the EU Pediatric Regulation, the number of pediatric protocols submitted to the IRB00009118 tends to increase, but no change was noticed regarding their quality.

Discontinuation and non-publication of randomised clinical trials supported by the main public funding body in Switzerland: a retrospective cohort study

OBJECTIVE

The Swiss National Science Foundation (SNSF) promotes academic excellence through competitive selection of study proposals and rigorous evaluation of feasibility, but completion status and publication history of SNSF-supported randomised clinical trials (RCTs) remain unclear. The main objectives were to review all healthcare RCTs supported by the SNSF for trial discontinuation and non-publication, to investigate potential risk factors for trial discontinuation due to poor recruitment and non-publication, and to compare findings to other Swiss RCTs not supported by the SNSF.

DESIGN

We established a retrospective cohort of all SNSF-supported RCTs for which recruitment and funding had ended in 2015 or earlier. For each RCT, two investigators independently searched corresponding publications in electronic databases. In addition, we approached all principal investigators to ask for additional publications and information about trial discontinuation. Teams of two investigators independently extracted details about study design, recruitment of participants, outcomes, analysis and sample size from the original proposal and, if available, from trial registries and publications. We used multivariable regression analysis to explore potential risk factors associated with discontinuation due to poor recruitment and with non-publication, and to compare our results with data from a previous cohort of Swiss RCTs not supported by the SNSF.

RESULTS

We included 101 RCTs supported by the SNSF between 1986 and 2015. Eighty-seven (86%) principal investigators responded to our survey. Overall, 69 (68%) RCTs were completed, 26 (26%) RCTs were prematurely discontinued (all due to slow recruitment) and the completion status remained unclear for 6 (6%) RCTs. For analysing publication status, we excluded 4 RCTs for which follow-up was still ongoing and 9 for which manuscripts were still in preparation. Of the remaining 88 RCTs, 53 (60%) were published as full articles in peer-reviewed journals. Multivariable regression models suggested that discontinued trials were at higher risk for non-publication than completed trials (adjusted OR 7.61; 95% CI 2.44 to 27.09). Compared with other Swiss RCTs, the risk of discontinuation for SNSF-supported RCTs was higher than in industry-initiated RCTs (adjusted OR 3.84; 95% CI 1.68 to 8.74), but not significantly different from investigator-initiated RCTs not supported by the SNSF (adjusted OR 1.05; 95% CI 0.51 to 2.11). We found no evidence that the proportion of discontinued or unpublished RCTs decreased over the last 20 years.

CONCLUSIONS

One out of four SNSF-supported RCTs were prematurely discontinued due to slow recruitment, 40% of all included RCTs and 70% of all discontinued RCTs were not published in peer-reviewed journals. There is a case to reconsider how public funding bodies such as the SNSF could improve their feasibility assessment and promote publication of RCTs irrespective of completion status.

Authorization of Animal Experiments Is Based on Confidence Rather than Evidence of Scientific Rigor

Accumulating evidence indicates high risk of bias in preclinical animal research, questioning the scientific validity and reproducibility of published research findings. Systematic reviews found low rates of reporting of measures against risks of bias in the published literature (e.g., randomization, blinding, sample size calculation) and a correlation between low reporting rates and inflated treatment effects. That most animal research undergoes peer review or ethical review would offer the possibility to detect risks of bias at an earlier stage, before the research has been conducted. For example, in Switzerland, animal experiments are licensed based on a detailed description of the study protocol and a harm-benefit analysis. We therefore screened applications for animal experiments submitted to Swiss authorities (n = 1,277) for the rates at which the use of seven basic measures against bias (allocation concealment, blinding, randomization, sample size calculation, inclusion/exclusion criteria, primary outcome variable, and statistical analysis plan) were described and compared them with the reporting rates of the same measures in a representative sub-sample of publications (n = 50) resulting from studies described in these applications. Measures against bias were described at very low rates, ranging on average from 2.4% for statistical analysis plan to 19% for primary outcome variable in applications for animal experiments, and from 0.0% for sample size calculation to 34% for statistical analysis plan in publications from these experiments. Calculating an internal validity score (IVS) based on the proportion of the seven measures against bias, we found a weak positive correlation between the IVS of applications and that of publications (Spearman's rho = 0.34, p = 0.014), indicating that the rates of description of these measures in applications partly predict their rates of reporting in publications. These results indicate that the authorities licensing animal experiments are lacking important information about experimental conduct that determines the scientific validity of the findings, which may be critical for the weight attributed to the benefit of the research in the harm-benefit analysis. Similar to manuscripts getting accepted for publication despite poor reporting of measures against bias, applications for animal experiments may often be approved based on implicit confidence rather than explicit evidence of scientific rigor. Our findings shed serious doubt on the current authorization procedure for animal experiments, as well as the peer-review process for scientific publications, which in the long run may undermine the credibility of research. Developing existing authorization procedures that are already in place in many countries towards a preregistration system for animal research is one promising way to reform the system. This would not only benefit the scientific validity of findings from animal experiments but also help to avoid unnecessary harm to animals for inconclusive research.

Agreements between Industry and Academia on Publication Rights: A Retrospective Study of Protocols and Publications of Randomized Clinical Trials

BACKGROUND

Little is known about publication agreements between industry and academic investigators in trial protocols and the consistency of these agreements with corresponding statements in publications. We aimed to investigate (i) the existence and types of publication agreements in trial protocols, (ii) the completeness and consistency of the reporting of these agreements in subsequent publications, and (iii) the frequency of co-authorship by industry employees.

METHODS AND FINDINGS

We used a retrospective cohort of randomized clinical trials (RCTs) based on archived protocols approved by six research ethics committees between 13 January 2000 and 25 November 2003. Only RCTs with industry involvement were eligible. We investigated the documentation of publication agreements in RCT protocols and statements in corresponding journal publications. Of 647 eligible RCT protocols, 456 (70.5%) mentioned an agreement regarding publication of results. Of these 456, 393 (86.2%) documented an industry partner's right to disapprove or at least review proposed manuscripts; 39 (8.6%) agreements were without constraints of publication. The remaining 24 (5.3%) protocols referred to separate agreement documents not accessible to us. Of those 432 protocols with an accessible publication agreement, 268 (62.0%) trials were published. Most agreements documented in the protocol were not reported in the subsequent publication (197/268 [73.5%]). Of 71 agreements reported in publications, 52 (73.2%) were concordant with those documented in the protocol. In 14 of 37 (37.8%) publications in which statements suggested unrestricted publication rights, at least one co-author was an industry employee. In 25 protocol-publication pairs, author statements in publications suggested no constraints, but 18 corresponding protocols documented restricting agreements.

CONCLUSIONS

Publication agreements constraining academic authors' independence are common. Journal articles seldom report on publication agreements, and, if they do, statements can be discrepant with the trial protocol.

From protocol to published report: a study of consistency in the reporting of academic drug trials

BACKGROUND

Unacknowledged inconsistencies in the reporting of clinical trials undermine the validity of the results of the trials. Little is known about inconsistency in the reporting of academic clinical drug trials. Therefore, we investigated the prevalence of consistency between protocols and published reports of academic clinical drug trials.

METHODS

A comparison was made between study protocols and their corresponding published reports. We assessed the overall consistency, which was defined as the absence of discrepancy regarding study type (categorized as either exploratory or confirmatory), primary objective, primary endpoint, and--for confirmatory trials only--hypothesis and sample size calculation. We used logistic regression, χ(2), and Fisher's exact test.

RESULTS

A total of 282 applications of academic clinical drug trials were submitted to the Danish Health and Medicines Authority in 1999, 2001, and 2003, 95 of which fulfilled the eligibility criteria and had at least one corresponding published report reporting data on trial subjects. Overall consistency was observed in 39% of the trials (95% CI: 29 to 49%). Randomized controlled trials (RCTs) constituted 72% (95% CI: 63 to 81%) of the sample, and 87% (95% CI: 80 to 94%) of the trials were hospital based.

CONCLUSIONS

Overall consistency between protocols and their corresponding published reports was low. Motivators for the inconsistencies are unknown but do not seem restricted to economic incentives.

Does Publication Bias Inflate the Apparent Efficacy of Psychological Treatment for Major Depressive Disorder? A Systematic Review and Meta-Analysis of US National Institutes of Health-Funded Trials

BACKGROUND

The efficacy of antidepressant medication has been shown empirically to be overestimated due to publication bias, but this has only been inferred statistically with regard to psychological treatment for depression. We assessed directly the extent of study publication bias in trials examining the efficacy of psychological treatment for depression.

METHODS AND FINDINGS

We identified US National Institutes of Health grants awarded to fund randomized clinical trials comparing psychological treatment to control conditions or other treatments in patients diagnosed with major depressive disorder for the period 1972-2008, and we determined whether those grants led to publications. For studies that were not published, data were requested from investigators and included in the meta-analyses. Thirteen (23.6%) of the 55 funded grants that began trials did not result in publications, and two others never started. Among comparisons to control conditions, adding unpublished studies (Hedges' g = 0.20; CI95% -0.11~0.51; k = 6) to published studies (g = 0.52; 0.37~0.68; k = 20) reduced the psychotherapy effect size point estimate (g = 0.39; 0.08~0.70) by 25%. Moreover, these findings may overestimate the "true" effect of psychological treatment for depression as outcome reporting bias could not be examined quantitatively.

CONCLUSION

The efficacy of psychological interventions for depression has been overestimated in the published literature, just as it has been for pharmacotherapy. Both are efficacious but not to the extent that the published literature would suggest. Funding agencies and journals should archive both original protocols and raw data from treatment trials to allow the detection and correction of outcome reporting bias. Clinicians, guidelines developers, and decision makers should be aware that the published literature overestimates the effects of the predominant treatments for depression.

Journal editors impasse with outcome reporting bias

Several requirements and regulations have been issued to promote clinical trial transparency through prospective registration of trials, disclosure of results, access to trial reports submitted to regulatory agencies and access to anonymized patient-level data. Clinical trial results are disseminated through articles. Yet, many present outcome reporting bias. Open access to trial data will help to deter outcome reporting bias. However, this is not enough to clinicians. Access to trial protocols by journal staff has proven rather inefficacious in preventing outcome reporting bias. Two proposals have been suggested or implemented to tackle outcome reporting bias that are discussed in this article. Editors should implement quality-control processes aiming at preventing outcome reporting bias. Readers should be informed as to the efficiency of the implemented process.

Completion and Publication Rates of Randomized Controlled Trials in Surgery: An Empirical Study

OBJECTIVE

To investigate the prevalence of discontinuation and nonpublication of surgical versus medical randomized controlled trials (RCTs) and to explore risk factors for discontinuation and nonpublication of surgical RCTs.

BACKGROUND

Trial discontinuation has significant scientific, ethical, and economic implications. To date, the prevalence of discontinuation of surgical RCTs is unknown.

METHODS

All RCT protocols approved between 2000 and 2003 by 6 ethics committees in Canada, Germany, and Switzerland were screened. Baseline characteristics were collected and, if published, full reports retrieved. Risk factors for early discontinuation for slow recruitment and nonpublication were explored using multivariable logistic regression analyses.

RESULTS

In total, 863 RCT protocols involving adult patients were identified, 127 in surgery (15%) and 736 in medicine (85%). Surgical trials were discontinued for any reason more often than medical trials [43% vs 27%, risk difference 16% (95% confidence interval [CI]: 5%-26%); P = 0.001] and more often discontinued for slow recruitment [18% vs 11%, risk difference 8% (95% CI: 0.1%-16%); P = 0.020]. The percentage of trials not published as full journal article was similar in surgical and medical trials (44% vs 40%, risk difference 4% (95% CI: -5% to 14%); P = 0.373). Discontinuation of surgical trials was a strong risk factor for nonpublication (odds ratio = 4.18, 95% CI: 1.45-12.06; P = 0.008).

CONCLUSIONS

Discontinuation and nonpublication rates were substantial in surgical RCTs and trial discontinuation was strongly associated with nonpublication. These findings need to be taken into account when interpreting surgical literature. Surgical trialists should consider feasibility studies before embarking on full-scale trials.

Subgroup analyses in randomised controlled trials: cohort study on trial protocols and journal publications

OBJECTIVE

To investigate the planning of subgroup analyses in protocols of randomised controlled trials and the agreement with corresponding full journal publications.

DESIGN

Cohort of protocols of randomised controlled trial and subsequent full journal publications.

SETTING

Six research ethics committees in Switzerland, Germany, and Canada.

DATA SOURCES

894 protocols of randomised controlled trial involving patients approved by participating research ethics committees between 2000 and 2003 and 515 subsequent full journal publications.

RESULTS

Of 894 protocols of randomised controlled trials, 252 (28.2%) included one or more planned subgroup analyses. Of those, 17 (6.7%) provided a clear hypothesis for at least one subgroup analysis, 10 (4.0%) anticipated the direction of a subgroup effect, and 87 (34.5%) planned a statistical test for interaction. Industry sponsored trials more often planned subgroup analyses compared with investigator sponsored trials (195/551 (35.4%) v 57/343 (16.6%), P<0.001). Of 515 identified journal publications, 246 (47.8%) reported at least one subgroup analysis. In 81 (32.9%) of the 246 publications reporting subgroup analyses, authors stated that subgroup analyses were prespecified, but this was not supported by 28 (34.6%) corresponding protocols. In 86 publications, authors claimed a subgroup effect, but only 36 (41.9%) corresponding protocols reported a planned subgroup analysis.

CONCLUSIONS

Subgroup analyses are insufficiently described in the protocols of randomised controlled trials submitted to research ethics committees, and investigators rarely specify the anticipated direction of subgroup effects. More than one third of statements in publications of randomised controlled trials about subgroup prespecification had no documentation in the corresponding protocols. Definitive judgments regarding credibility of claimed subgroup effects are not possible without access to protocols and analysis plans of randomised controlled trials.

Increasing value and reducing waste: addressing inaccessible research

The methods and results of health research are documented in study protocols, full study reports (detailing all analyses), journal reports, and participant-level datasets. However, protocols, full study reports, and participant-level datasets are rarely available, and journal reports are available for only half of all studies and are plagued by selective reporting of methods and results. Furthermore, information provided in study protocols and reports varies in quality and is often incomplete. When full information about studies is inaccessible, billions of dollars in investment are wasted, bias is introduced, and research and care of patients are detrimentally affected. To help to improve this situation at a systemic level, three main actions are warranted. First, academic institutions and funders should reward investigators who fully disseminate their research protocols, reports, and participant-level datasets. Second, standards for the content of protocols and full study reports and for data sharing practices should be rigorously developed and adopted for all types of health research. Finally, journals, funders, sponsors, research ethics committees, regulators, and legislators should endorse and enforce policies supporting study registration and wide availability of journal reports, full study reports, and participant-level datasets.

Reporting of analyses from randomized controlled trials with multiple arms: a systematic review

BACKGROUND

Multiple-arm randomized trials can be more complex in their design, data analysis, and result reporting than two-arm trials. We conducted a systematic review to assess the reporting of analyses in reports of randomized controlled trials (RCTs) with multiple arms.

METHODS

The literature in the MEDLINE database was searched for reports of RCTs with multiple arms published in 2009 in the core clinical journals. Two reviewers extracted data using a standardized extraction form.

RESULTS

In total, 298 reports were identified. Descriptions of the baseline characteristics and outcomes per group were missing in 45 reports (15.1%) and 48 reports (16.1%), respectively. More than half of the articles (n = 171, 57.4%) reported that a planned global test comparison was used (that is, assessment of the global differences between all groups), but 67 (39.2%) of these 171 articles did not report details of the planned analysis. Of the 116 articles reporting a global comparison test, 12 (10.3%) did not report the analysis as planned. In all, 60% of publications (n = 180) described planned pairwise test comparisons (that is, assessment of the difference between two groups), but 20 of these 180 articles (11.1%) did not report the pairwise test comparisons. Of the 204 articles reporting pairwise test comparisons, the comparisons were not planned for 44 (21.6%) of them. Less than half the reports (n = 137; 46%) provided baseline and outcome data per arm and reported the analysis as planned.

CONCLUSIONS

Our findings highlight discrepancies between the planning and reporting of analyses in reports of multiple-arm trials.

Sample size determinations in original research protocols for randomised clinical trials submitted to UK research ethics committees: review

OBJECTIVES

To assess the completeness of reporting of sample size determinations in unpublished research protocols and to develop guidance for research ethics committees and for statisticians advising these committees.

DESIGN

Review of original research protocols.

STUDY SELECTION

Unpublished research protocols for phase IIb, III, and IV randomised clinical trials of investigational medicinal products submitted to research ethics committees in the United Kingdom during 1 January to 31 December 2009.

MAIN OUTCOME MEASURES

Completeness of reporting of the sample size determination, including the justification of design assumptions, and disagreement between reported and recalculated sample size.

RESULTS

446 study protocols were reviewed. Of these, 190 (43%) justified the treatment effect and 213 (48%) justified the population variability or survival experience. Only 55 (12%) discussed the clinical importance of the treatment effect sought. Few protocols provided a reasoned explanation as to why the design assumptions were plausible for the planned study. Sensitivity analyses investigating how the sample size changed under different design assumptions were lacking; six (1%) protocols included a re-estimation of the sample size in the study design. Overall, 188 (42%) protocols reported all of the information to accurately recalculate the sample size; the assumed withdrawal or dropout rate was not given in 177 (40%) studies. Only 134 of the 446 (30%) sample size calculations could be accurately reproduced. Study size tended to be over-estimated rather than under-estimated. Studies with non-commercial sponsors justified the design assumptions used in the calculation more often than studies with commercial sponsors but less often reported all the components needed to reproduce the sample size calculation. Sample sizes for studies with non-commercial sponsors were less often reproduced.

CONCLUSIONS

Most research protocols did not contain sufficient information to allow the sample size to be reproduced or the plausibility of the design assumptions to be assessed. Greater transparency in the reporting of the determination of the sample size and more focus on study design during the ethical review process would allow deficiencies to be resolved early, before the trial begins. Guidance for research ethics committees and statisticians advising these committees is needed.

Learning from failure--rationale and design for a study about discontinuation of randomized trials (DISCO study)

BACKGROUND

Randomized controlled trials (RCTs) may be discontinued because of apparent harm, benefit, or futility. Other RCTs are discontinued early because of insufficient recruitment. Trial discontinuation has ethical implications, because participants consent on the premise of contributing to new medical knowledge, Research Ethics Committees (RECs) spend considerable effort reviewing study protocols, and limited resources for conducting research are wasted. Currently, little is known regarding the frequency and characteristics of discontinued RCTs.

METHODS/DESIGN

Our aims are, first, to determine the prevalence of RCT discontinuation for specific reasons; second, to determine whether the risk of RCT discontinuation for specific reasons differs between investigator- and industry-initiated RCTs; third, to identify risk factors for RCT discontinuation due to insufficient recruitment; fourth, to determine at what stage RCTs are discontinued; and fifth, to examine the publication history of discontinued RCTs.We are currently assembling a multicenter cohort of RCTs based on protocols approved between 2000 and 2002/3 by 6 RECs in Switzerland, Germany, and Canada. We are extracting data on RCT characteristics and planned recruitment for all included protocols. Completion and publication status is determined using information from correspondence between investigators and RECs, publications identified through literature searches, or by contacting the investigators. We will use multivariable regression models to identify risk factors for trial discontinuation due to insufficient recruitment. We aim to include over 1000 RCTs of which an anticipated 150 will have been discontinued due to insufficient recruitment.

DISCUSSION

Our study will provide insights into the prevalence and characteristics of RCTs that were discontinued. Effective recruitment strategies and the anticipation of problems are key issues in the planning and evaluation of trials by investigators, Clinical Trial Units, RECs and funding agencies. Identification and modification of barriers to successful study completion at an early stage could help to reduce the risk of trial discontinuation, save limited resources, and enable RCTs to better meet their ethical requirements.

Optimal designs for prediction studies of whiplash

STUDY DESIGN

Commentary.

OBJECTIVE

To provide guidance for the design and interpretation of predictive studies of whiplash associated disorders (WAD).

SUMMARY OF BACKGROUND DATA

Numerous studies have sought to define and explain the clinical course and response to treatment of people with WAD. Design of these studies is often suboptimal, which can lead to biased findings and issues with interpreting the results.

METHODS

Literature review and commentary.

RESULTS

Predictive studies can be grouped into four broad categories; studies of symptomatic course, studies that aim to identify factors that predict outcome, studies that aim to isolate variables that are causally responsible for outcome, and studies that aim to identify patients who respond best to particular treatments. Although the specific research question will determine the optimal methods, there are a number of generic features that should be incorporated into design of such studies. The aim of these features is to minimize bias, generate adequately precise prognostic estimates, and ensure generalizability of the findings.

CONCLUSION

This paper provides a summary of important considerations in the design, conduct, and reporting of prediction studies in the field of whiplash.

Frequency and reasons for outcome reporting bias in clinical trials: interviews with trialists

OBJECTIVES

To provide information on the frequency and reasons for outcome reporting bias in clinical trials.

DESIGN

Trial protocols were compared with subsequent publication(s) to identify any discrepancies in the outcomes reported, and telephone interviews were conducted with the respective trialists to investigate more extensively the reporting of the research and the issue of unreported outcomes.

PARTICIPANTS

Chief investigators, or lead or coauthors of trials, were identified from two sources: trials published since 2002 covered in Cochrane systematic reviews where at least one trial analysed was suspected of being at risk of outcome reporting bias (issue 4, 2006; issue 1, 2007, and issue 2, 2007 of the Cochrane library); and a random sample of trial reports indexed on PubMed between August 2007 and July 2008.

SETTING

Australia, Canada, Germany, the Netherlands, New Zealand, the United Kingdom, and the United States.

MAIN OUTCOME MEASURES

Frequency of incomplete outcome reporting-signified by outcomes that were specified in a trial's protocol but not fully reported in subsequent publications-and trialists' reasons for incomplete reporting of outcomes.

RESULTS

268 trials were identified for inclusion (183 from the cohort of Cochrane systematic reviews and 85 from PubMed). Initially, 161 respective investigators responded to our requests for interview, 130 (81%) of whom agreed to be interviewed. However, failure to achieve subsequent contact, obtain a copy of the study protocol, or both meant that final interviews were conducted with 59 (37%) of the 161 trialists. Sixteen trial investigators failed to report analysed outcomes at the time of the primary publication, 17 trialists collected outcome data that were subsequently not analysed, and five trialists did not measure a prespecified outcome over the course of the trial. In almost all trials in which prespecified outcomes had been analysed but not reported (15/16, 94%), this under-reporting resulted in bias. In nearly a quarter of trials in which prespecified outcomes had been measured but not analysed (4/17, 24%), the "direction" of the main findings influenced the investigators' decision not to analyse the remaining data collected. In 14 (67%) of the 21 randomly selected PubMed trials, there was at least one unreported efficacy or harm outcome. More than a quarter (6/21, 29%) of these trials were found to have displayed outcome reporting bias.

CONCLUSION

The prevalence of incomplete outcome reporting is high. Trialists seemed generally unaware of the implications for the evidence base of not reporting all outcomes and protocol changes. A general lack of consensus regarding the choice of outcomes in particular clinical settings was evident and affects trial design, conduct, analysis, and reporting.

Obstacles to researching the researchers: a case study of the ethical challenges of undertaking methodological research investigating the reporting of randomised controlled trials

BACKGROUND

Recent cohort studies of randomised controlled trials have provided evidence of within-study selective reporting bias; where statistically significant outcomes are more likely to be more completely reported compared to non-significant outcomes. Bias resulting from selective reporting can impact on meta-analyses, influencing the conclusions of systematic reviews, and in turn, evidence based clinical practice guidelines.In 2006 we received funding to investigate if there was evidence of within-study selective reporting in a cohort of RCTs submitted to New Zealand Regional Ethics Committees in 1998/99. This research involved accessing ethics applications, their amendments and annual reports, and comparing these with corresponding publications. We did not plan to obtain informed consent from trialists to view their ethics applications for practical and scientific reasons. In November 2006 we sought ethical approval to undertake the research from our institutional ethics committee. The Committee declined our application on the grounds that we were not obtaining informed consent from the trialists to view their ethics application. This initiated a seventeen month process to obtain ethical approval. This publication outlines what we planned to do, the issues we encountered, discusses the legal and ethical issues, and presents some potential solutions.

DISCUSSION AND CONCLUSION

Methodological research such as this has the potential for public benefit and there is little or no harm for the participants (trialists) in undertaking it. Further, in New Zealand, there is freedom of information legislation, which in this circumstance, unambiguously provided rights of access and use of the information in the ethics applications. The decision of our institutional ethics committee defeated this right and did not recognise the nature of this observational research. Methodological research, such as this, can be used to develop processes to improve quality in research reporting. Recognition of the potential benefit of this research in the broader research community, and those who sit on ethics committees, is perhaps needed. In addition, changes to the ethical review process which involve separation between those who review proposals to undertake methodological research using ethics applications, and those with responsibility for reviewing ethics applications for trials, should be considered. Finally, we contend that the research community could benefit from quality improvement approaches used in allied sectors.

Discrepancies in sample size calculations and data analyses reported in randomised trials: comparison of publications with protocols

OBJECTIVE

To evaluate how often sample size calculations and methods of statistical analysis are pre-specified or changed in randomised trials.

DESIGN

Retrospective cohort study. Data source Protocols and journal publications of published randomised parallel group trials initially approved in 1994-5 by the scientific-ethics committees for Copenhagen and Frederiksberg, Denmark (n=70).

MAIN OUTCOME MEASURE

Proportion of protocols and publications that did not provide key information about sample size calculations and statistical methods; proportion of trials with discrepancies between information presented in the protocol and the publication.

RESULTS

Only 11/62 trials described existing sample size calculations fully and consistently in both the protocol and the publication. The method of handling protocol deviations was described in 37 protocols and 43 publications. The method of handling missing data was described in 16 protocols and 49 publications. 39/49 protocols and 42/43 publications reported the statistical test used to analyse primary outcome measures. Unacknowledged discrepancies between protocols and publications were found for sample size calculations (18/34 trials), methods of handling protocol deviations (19/43) and missing data (39/49), primary outcome analyses (25/42), subgroup analyses (25/25), and adjusted analyses (23/28). Interim analyses were described in 13 protocols but mentioned in only five corresponding publications.

CONCLUSION

When reported in publications, sample size calculations and statistical methods were often explicitly discrepant with the protocol or not pre-specified. Such amendments were rarely acknowledged in the trial publication. The reliability of trial reports cannot be assessed without having access to the full protocols.

Bias, spin, and misreporting: time for full access to trial protocols and results

An-Wen Chan discusses the implications of a new study inPLoS Medicine that suggests that the randomized trial literature is skewed towards reporting favorable results.