Prospective registration and outcome-reporting bias in randomized controlled trials of eczema treatments: a systematic review

Estimating the prevalence of discrepancies between study registrations and publications: a systematic review and meta-analyses

OBJECTIVES

Prospectively registering study plans in a permanent time-stamped and publicly accessible document is becoming more common across disciplines and aims to reduce risk of bias and make risk of bias transparent. Selective reporting persists, however, when researchers deviate from their registered plans without disclosure. This systematic review aimed to estimate the prevalence of undisclosed discrepancies between prospectively registered study plans and their associated publication. We further aimed to identify the research disciplines where these discrepancies have been observed, whether interventions to reduce discrepancies have been conducted, and gaps in the literature.

DESIGN

Systematic review and meta-analyses.

DATA SOURCES

Scopus and Web of Knowledge, published up to 15 December 2019.

ELIGIBILITY CRITERIA

Articles that included quantitative data about discrepancies between registrations or study protocols and their associated publications.

DATA EXTRACTION AND SYNTHESIS

Each included article was independently coded by two reviewers using a coding form designed for this review (osf.io/728ys). We used random-effects meta-analyses to synthesise the results.

RESULTS

We reviewed k=89 articles, which included k=70 that reported on primary outcome discrepancies from n=6314 studies and, k=22 that reported on secondary outcome discrepancies from n=1436 studies. Meta-analyses indicated that between 29% and 37% (95% CI) of studies contained at least one primary outcome discrepancy and between 50% and 75% (95% CI) contained at least one secondary outcome discrepancy. Almost all articles assessed clinical literature, and there was considerable heterogeneity. We identified only one article that attempted to correct discrepancies.

CONCLUSIONS

Many articles did not include information on whether discrepancies were disclosed, which version of a registration they compared publications to and whether the registration was prospective. Thus, our estimates represent discrepancies broadly, rather than our target of undisclosed discrepancies between prospectively registered study plans and their associated publications. Discrepancies are common and reduce the trustworthiness of medical research. Interventions to reduce discrepancies could prove valuable.

REGISTRATION

osf.io/ktmdg. Protocol amendments are listed in online supplemental material A.

Frequency of multiple changes to prespecified primary outcomes of clinical trials completed between 2009 and 2017 in German university medical centers: A meta-research study

BACKGROUND

Clinical trial registries allow assessment of deviations of published trials from their protocol, which may indicate a considerable risk of bias. However, since entries in many registries can be updated at any time, deviations may go unnoticed. We aimed to assess the frequency of changes to primary outcomes in different historical versions of registry entries, and how often they would go unnoticed if only deviations between published trial reports and the most recent registry entry are assessed.

METHODS AND FINDINGS

We analyzed the complete history of changes of registry entries in all 1746 randomized controlled trials completed at German university medical centers between 2009 and 2017, with published results up to 2022, that were registered in ClinicalTrials.gov or the German WHO primary registry (German Clinical Trials Register; DRKS). Data were retrieved on 24 January 2022. We assessed deviations between registry entries and publications in a random subsample of 292 trials. We determined changes of primary outcomes (1) between different versions of registry entries at key trial milestones, (2) between the latest registry entry version and the results publication, and (3) changes that occurred after trial start with no change between latest registry entry version and publication (so that assessing the full history of changes is required for detection of changes). We categorized changes as major if primary outcomes were added, dropped, changed to secondary outcomes, or secondary outcomes were turned into primary outcomes. We also assessed (4) the proportion of publications transparently reporting changes and (5) characteristics associated with changes. Of all 1746 trials, 23% (n = 393) had a primary outcome change between trial start and latest registry entry version, with 8% (n = 142) being major changes, that is, primary outcomes were added, dropped, changed to secondary outcomes, or secondary outcomes were turned into primary outcomes. Primary outcomes in publications were different from the latest registry entry version in 41% of trials (120 of the 292 sampled trials; 95% confidence interval (CI) [35%, 47%]), with major changes in 18% (54 of 292; 95% CI [14%, 23%]). Overall, 55% of trials (161 of 292; 95% CI [49%, 61%]) had primary outcome changes at any timepoint over the course of a trial, with 23% of trials (67 of 292; 95% CI [18%, 28%]) having major changes. Changes only within registry records, with no apparent discrepancy between latest registry entry version and publication, were observed in 14% of trials (41 of 292; 95% CI [10%, 19%]), with 4% (13 of 292; 95% CI [2%, 7%]) being major changes. One percent of trials with a change reported this in their publication (2 of 161 trials; 95% CI [0%, 4%]). An exploratory logistic regression analysis indicated that trials were less likely to have a discrepant registry entry if they were registered more recently (odds ratio (OR) 0.74; 95% CI [0.69, 0.80]; p<0.001), were not registered on ClinicalTrials.gov (OR 0.41; 95% CI [0.23, 0.70]; p = 0.002), or were not industry-sponsored (OR 0.29; 95% CI [0.21, 0.41]; p<0.001). Key limitations include some degree of subjectivity in the categorization of outcome changes and inclusion of a single geographic region.

CONCLUSIONS

In this study, we observed that changes to primary outcomes occur in 55% of trials, with 23% trials having major changes. They are rarely transparently reported in the results publication and often not visible in the latest registry entry version. More transparency is needed, supported by deeper analysis of registry entries to make these changes more easily recognizable. Protocol registration: Open Science Framework (https://osf.io/t3qva; amendment in https://osf.io/qtd2b).

Methods Reporting Quality and its Association with Methods Section Length: A Cross-sectional Assessment of STROBE and CONSORT Reporting Adherence in Top-cited Dermatology Journals

BACKGROUND

Adequate methods reporting in observational and trial literature is critical to interpretation and implementation.

OBJECTIVE

Evaluate methodology reporting adherence in the dermatology literature and compare this to internal medicine (IM) literature.

METHODS

We performed a cross-sectional review of randomly-selected dermatology and IM manuscripts published between 2014-2018. Observational and trial articles were retrieved from PubMed. The primary outcome was percent adherence to STROBE or CONSORT methods-related checklist items (methods reporting score, MRS). Secondary outcomes included the relationship between methods section length (MSL) and MRS. We additionally compared these with IM literature. MRS and MSL were compared by overall article length, checklist type, field, journal, study topic, and funding source. Comparisons were assessed using univariable and multivariable linear regression.

RESULTS

We identified 389 articles (172 dermatology and 217 IM). Within dermatology, we identified 83 clinical trials and 89 observational studies. Mean MRS was 61.4 percent. A one word increase in MSL corresponded to a 0.02 percent increase MRS (β=0.02, 95% CI 0.01-0.03). Mean MRS was 12.8 percent lower in the dermatology literature compared with IM (β=-12.8%, -15.6-[-9.91]). Mean dermatology MSL was 345 words shorter (β=-345, -413-[-277]). Studies from JAMA Dermatology, Journal of Investigative Dermatology, and British Journal of Dermatology, with government funding, and having supplemental methods had higher mean MRS's.

CONCLUSION

Methods reporting quality was low in dermatology. A weak relationship between MRS and MSL was observed. These data support enhancing researcher emphasis on methods reporting, editorial staff, and peer reviewers that more strictly enforce checklist reporting.

Inconsistent Outcome Reporting in Heart Failure Randomized Controlled Trials

BACKGROUND

Randomized controlled trials (RCTs) may report outcomes different from those prespecified on trial-registration websites, protocols and statistical analysis plans (SAPs). This study sought to investigate the prevalence and characteristics of heart failure (HF) RCTs that report outcomes different from those prespecified.

METHODS AND RESULTS

MEDLINE via PubMed was searched to include phase II-IV HF RCTs in 9 high-impact journals from 2010 to 2020. Outcomes reported in trial publications were compared with prespecified outcomes in protocols, registration websites and SAPs. We used the χ² or Fisher exact test to analyze correlations between trial characteristics and inconsistencies. Among 216 trials, 32 inconsistencies were observed in 28 trials (13.0%). Among 32 inconsistencies, 2 (6.3%) pertained to omission of prespecified primary outcomes, 4 (12.5%) to omission of prespecified secondary outcomes, 2 (6.3%) to changing prespecified primary outcomes to secondary outcomes, and 2 (6.3%) to changing prespecified secondary outcomes to primary outcomes. Of the inconsistencies, 3 (9.4%) pertained to addition of new primary outcomes, 17 (53.1%) to addition of new secondary outcomes, and 2 (6.3%,) to changes in the timing of assessment of primary outcomes. The majority of the inconsistencies favored statistically significant findings; 78 (36.1%) were registered retrospectively. Single-center recruitment was associated with outcome inconsistencies (β = -0.14; 95% CI, -0.22 - -0.01; P = 0.035).

CONCLUSIONS

More than 1 in 10 trials reported outcomes inconsistent with those specified in trial registration websites, SAPs and protocols. An action plan is warranted to minimize selective reporting and improve transparency.

Evaluating publication bias for clinical trials supporting new dermatologic drug approvals from 2003 to 2018

The degree of publication bias and impact of the Food and Drug Administration Amendments Act (FDAAA) of 2007, which aimed to improve clinical trial transparency, has yet to be examined for recent dermatologic drugs. The objective of our study was to estimate the degree of publication bias for clinical trials supporting FDA approval of new dermatologic drugs. This retrospective cohort study examined all phase II and III efficacy trials supporting approval of new dermatologic drugs from 2003 to 2018. FDA drug approval documents were reviewed for supportive clinical trial information, and publications were matched using PubMed and Google Scholar searches. Ratios of relative risks (RRR) comparing positive versus non-positive trials before and after FDAAA enactment served to estimate publication bias. We found that the likelihood of publishing positive versus non-positive drug trials in dermatology was unchanged before and after FDAAA enactment (RRR 0.87, 95% CI 0.37-2.08), as was the likelihood of publishing without misleading interpretation (RRR 1.51, 95% CI 0.22-10.50). There was no measurable publication bias for efficacy trials supporting new drug approvals in dermatology over the past 15 years. Fewer pre-FDAAA trials (n = 21) compared to post-FDAAA trials (n = 106) met inclusion criteria. Though not analyzed in this study, safety and secondary efficacy results are other potential sources for publication bias.

Assessing the compliance of systematic review articles published in leading dermatology journals with the PRISMA statement guidelines: A systematic review

Background

Reporting quality of systematic reviews and meta-analyses is of critical importance in dermatology because of their key role in informing health care decisions.

Objective

To assess the compliance of systematic reviews and meta-analyses in leading dermatology journals with the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement guidelines.

Methods

This review was carried out in accordance with PRISMA guidelines. Included studies were reviews published across 6 years in the top 4 highest-impact-factor dermatology journals of 2017. Records and full texts were screened independently. Data analysis was conducted with univariate multivariable linear regression. The primary outcome was to assess the compliance of systematic reviews and meta-analyses in leading dermatology journals with the PRISMA statement.

Results

A total of 166 studies were included and mean PRISMA compliance across all articles was 73%. Compliance significantly improved over time (β = .016; P = <.001). The worst reported checklist item was item 5 (reporting on protocol existence), with a compliance of 15% of articles.

Conclusion

PRISMA compliance within leading dermatology journals could be improved; however, it is steadily improving.

Selective outcome reporting in root coverage randomized clinical trials

BACKGROUND

Outcome discrepancies between protocols and respective publications represent a concerning bias. The purpose of this study was to assess the prevalence of selective outcome reporting (SOR) in root coverage randomized clinical trials (RCTs).

METHODS

Published root coverage RCTs (July 2005 to March 2020) were included if a corresponding protocol could be identified in a public registry. Discrepancies between protocol and its correspondent publication(s) were compared regarding primary and secondary outcomes and other study characteristics. Associations between trial characteristics and SOR were evaluated.

RESULTS

Forty four studies (54 publications) were included. The majority of studies (77.3%) were retrospectively registered. SOR was frequent (40.9% of trials) and consisted of primary outcome downgrade (22.7%); secondary outcome upgrade (11.4%); new primary outcome introduced in publication (25%); protocol primary outcome omitted from publication (13.6%) and discrepancy in primary outcome timing (18.2%). SOR was unclear in 20.5% of studies and favoured statistical significance in 12 studies (27.3%). SOR was significantly associated with study significance (p < 0.001) and unclear outcome definition in the publication (p < 0.001). Only a third (32.8%) of primary outcomes were completely defined.

CONCLUSIONS

The present study identified high prevalence of SOR in root coverage RCTs.

Reporting Quality of Randomized Controlled Trials for the Treatment of Eczema with Chinese Patent Medicine Based on the CONSORT-CHM Formulas 2017

OBJECTIVE

Chinese patent medicine (CPM) has been widely used to treat eczema in mainland China for decades. This study aims to investigate circulating CPM for eczema in mainland China and to evaluate the reporting quality of randomized controlled trials (RCTs) of them by using the CONSORT-CHM formulas 2017 (Consolidated Standards of Reporting Trials for Chinese herbal medicine formulas 2017).

METHODS

Circulating CPM with the indication for eczema was selected by searching three drug databases and confirmed by contacting the manufacturers. RCTs for the treatment of eczema with CPM were selected in four Chinese literature databases and four English literature databases from their inception to August 31, 2019. The reporting quality of included RCTs was assessed based on the CONSORT-CHM formulas 2017. A univariate analysis was conducted to identify the factors associated with the reporting quality.

RESULTS

A total of 70 circulating CPMs had the indication for eczema. Among them, 21 CPMs with 144 RCTs reached the eligible criteria. The mean overall quality score (OQS) of 144 RCTs was 19.85 ± 2.73, which was much less than the maximum score of 38. Of the 38 items, 12 items were reported in over 70% of the trials, 6 items were reported in 50%-70% of the trials, and 16 items were reported in less than 50% of the trials. Publication after 2015 (P < 0.001) and the first author from a university hospital (P=0.010) were associated with the better reporting quality.

CONCLUSION

There are a lot of circulating CPMs with the indication for eczema in mainland China, but both the quantity and the reporting quality of RCTs regarding those CPMs are suboptimal. It is necessary that authors and journal editors learn and adhere to the CONSORT-CHM formulas 2017 to enhance the reporting quality of RCTs for the treatment of eczema with CPM.

Complete and transparent reporting of primary end points of randomized trials in the dermatology literature: A comparison of registered and published primary end points

BACKGROUND

Appropriate primary end points in randomized controlled trials (RCTs) improve the quality of the measurement and enable comparison of the findings with those of other trials.

OBJECTIVE

To assess the quality of reporting primary end points in RCTs recently published in dermatology journals.

METHODS

We identified 134 primary reports of RCTs among original articles in 4 dermatology journals published from January 2016 to December 2018. Details were extracted from articles, supplements, and trial registries. A multivariable logistic regression analysis was conducted to identify factors associated with adequate primary end point reporting.

RESULTS

Adequate primary end point reporting was conducted in 76 of 134 RCTs (56.7%). Nine missed the definition of primary end points, and 13 did not define the timing of primary end points in the publications. Among 113 RCTs reporting primary end points explicitly in the articles, 16 showed discrepancies between registration and publication, and 21 were not able to valuate prespecification of primary end points. Multicenter studies and sponsor-initiated trials were significantly associated with adequate reporting quality after adjusting for covariates.

LIMITATIONS

Prespecification was evaluated based on a comparison of the article and registry.

CONCLUSIONS

The quality of primary end point reporting, particularly in prespecification, has remained unsatisfactory in the recent dermatology literature.

Transparent, Reproducible, and Open Science Practices of Published Literature in Dermatology Journals: Cross-Sectional Analysis

Background

Reproducible research is a foundational component for scientific advancements, yet little is known regarding the extent of reproducible research within the dermatology literature.

Objective

This study aimed to determine the quality and transparency of the literature in dermatology journals by evaluating for the presence of 8 indicators of reproducible and transparent research practices.

Methods

By implementing a cross-sectional study design, we conducted an advanced search of publications in dermatology journals from the National Library of Medicine catalog. Our search included articles published between January 1, 2014, and December 31, 2018. After generating a list of eligible dermatology publications, we then searched for full text PDF versions by using Open Access Button, Google Scholar, and PubMed. Publications were analyzed for 8 indicators of reproducibility and transparency—availability of materials, data, analysis scripts, protocol, preregistration, conflict of interest statement, funding statement, and open access—using a pilot-tested Google Form.

Results

After exclusion, 127 studies with empirical data were included in our analysis. Certain indicators were more poorly reported than others. We found that most publications (113, 88.9%) did not provide unmodified, raw data used to make computations, 124 (97.6%) failed to make the complete protocol available, and 126 (99.2%) did not include step-by-step analysis scripts.

Conclusions

Our sample of studies published in dermatology journals do not appear to include sufficient detail to be accurately and successfully reproduced in their entirety. Solutions to increase the quality, reproducibility, and transparency of dermatology research are warranted. More robust reporting of key methodological details, open data sharing, and stricter standards journals impose on authors regarding disclosure of study materials might help to better the climate of reproducible research in dermatology.

Trial registration as a safeguard against outcome reporting bias and spin? A case study of randomized controlled trials of acupuncture

BACKGROUND AND OBJECTIVE

Trial registration is widely endorsed as it is considered not only to enhance transparency and quality of reporting but also to help safeguard against outcome reporting bias and probably spin, known as specific reporting that could distort the interpretation of results thus mislead readers. We planned to investigate the current registration status of recently published randomized controlled trials (RCTs) of acupuncture, outcome reporting bias in the prospectively registered trials, and the association between trial registration and presence of spin and methodological factors in acupuncture RCTs.

METHODS

Acupuncture RCTs published in English in recent 5 years (January 2013 to December 2017) were searched in PubMed, Cochrane Central Register of Controlled Trials, and EMBASE. Trial registration records identified in the publications and trial registries were classified into prospectively registered, retrospectively registered, or unregistered. Primary outcomes were identified and the direction of the results was judged as statistically significant (positive) or statistically nonsignificant (negative). We compared registered and published primary outcomes to assess outcome reporting bias and assessed whether discrepancies favored statistically significant outcomes. Frequency and strategies of spin in published reports with statistically nonsignificant results for primary outcomes were then identified. We also analyzed whether the trial registration status was associated with spin and quality of methodological factors.

RESULTS

Of the 322 included RCTs, 41.9% (n = 135) were prospectively registered. Among 64 studies that were prospectively registered and specified primary outcomes, 25 trials had the discrepancies between the registered and published primary outcomes and 60% of them (15 trials) favored the statistically significant findings. Among 169 studies that specified primary outcomes, trial registration status was not associated with the direction of results, i.e., statistically significant or not. Spin was identified in 56.4% out of 78 studies with statistically nonsignificant primary outcomes and claiming efficacy with no consideration of statistically nonsignificant primary outcomes was the most common strategy for spin. Trial registration status was not statistically different between studies with and without spin.

CONCLUSION

While trial registration seemed to have improved over time, primary outcomes in registered records and publications were often inconsistent, tending to favor statistically significant findings and spin was common in studies with statistically nonsignificant primary outcomes. Journal editors and researchers in this field should be alerted to still prevalent reporting bias and spin.

Comparison of Clinical Trial Changes in Primary Outcome and Reported Intervention Effect Size Between Trial Registration and Publication

IMPORTANCE

Primary outcome change could threaten the validity of a clinical trial; however, evidence about the consequences on the reported intervention effect size is unclear.

OBJECTIVES

To examine the status of randomized clinical trials whose primary outcome changed between trial registration and publication and to quantify the association of this change with the reported intervention effect size.

DESIGN, SETTING, AND PARTICIPANTS

In this cross-sectional study on the primary report of randomized clinical trials with clear prospectively registered primary outcomes, PubMed and Embase were searched for articles published between January 1, 2011, and December 31, 2015. The search was conducted in January 2016, identifying randomized clinical trials and the combination of keywords and text words related to registry.

MAIN OUTCOMES AND MEASURES

Based on the developed approach, trials were classified as having primary outcome change when there was a major discrepancy between the registered and published primary outcomes. Intervention effect was estimated or recalculated using the odds ratio (OR) for each comparison. Each component OR is structured so that an OR is less than 1 if the intervention group has a more favorable result than the control group. The ratio of ORs (ROR), which is the summary OR for trials with primary outcome change divided by those without, and its 95% CI were calculated, with a value less than 1 indicating a larger reported intervention effect size in trials with primary outcome change than those without.

RESULTS

Among 29 749 searched articles (28 810 MEDLINE and 939 Embase), 1488 articles were randomly selected for review. Of 389 trials with clear primary outcomes prospectively described in the registry (416 outcomes reported), 33.4% (130 of 389) of trials had at least 1 primary outcome change. Most (66 of 130) of the changes were either not reporting or omitting the primary outcome. In total, 338 trials (365 outcomes and 487 comparisons) were available for quantitative analysis on the reported intervention effect size bias assessment. Compared with those without primary outcome change, trials with primary outcome change showed a 16% (pooled ROR, 0.84; 95% CI, 0.73-0.96) larger reported intervention effect size. The result persisted after adjustment for potential confounders (ROR, 0.81; 95% CI, 0.71-0.93) and other sensitivity and subgroup analyses.

CONCLUSIONS AND RELEVANCE

Results of this study suggest that inconsistencies between registered and published primary outcomes of clinical trials are common, and trials with primary outcome change are likely to have a larger intervention effect than those without.

Registration of published randomized trials: a systematic review and meta-analysis

BACKGROUND

Prospective trial registration is a powerful tool to prevent reporting bias. We aimed to determine the extent to which published randomized controlled trials (RCTs) were registered and registered prospectively.

METHODS

We searched MEDLINE and EMBASE from January 2005 to October 2017; we also screened all articles cited by or citing included and excluded studies, and the reference lists of related reviews. We included studies that examined published RCTs and evaluated their registration status, regardless of medical specialty or language. We excluded studies that assessed RCT registration status only through mention of registration in the published RCT, without searching registries or contacting the trial investigators. Two independent reviewers blinded to the other's work performed the selection. Following PRISMA guidelines, two investigators independently extracted data, with discrepancies resolved by consensus. We calculated pooled proportions and 95% confidence intervals using random-effects models.

RESULTS

We analyzed 40 studies examining 8773 RCTs across a wide range of clinical specialties. The pooled proportion of registered RCTs was 53% (95% confidence interval 44% to 58%), with considerable between-study heterogeneity. A subset of 24 studies reported data on prospective registration across 5529 RCTs. The pooled proportion of prospectively registered RCTs was 20% (95% confidence interval 15% to 25%). Subgroup analyses showed that registration was higher for industry-supported and larger RCTs. A meta-regression analysis across 19 studies (5144 RCTs) showed that the proportion of registered trials significantly increased over time, with a mean proportion increase of 27%, from 25 to 52%, between 2005 and 2015.

CONCLUSIONS

The prevalence of trial registration has increased over time, but only one in five published RCTs is prospectively registered, undermining the validity and integrity of biomedical research.

Registration and Outcome-Reporting Bias in Randomized Controlled Trials of Distal Radial Fracture Treatment

BACKGROUND

The purpose of the present study was to systematically evaluate the completeness of trial registration and the extent of outcome-reporting bias in modern randomized controlled trials (RCTs) relating to the treatment of distal radial fracture.

METHODS

With use of 4 databases (PubMed, Cochrane CENTRAL, Embase, and PEDro), this systematic review identified all RCTs of distal radial fracture treatment published from January 1, 2010, to December 31, 2015. We independently determined the registration status of these trials in a public trial registry and compared the characteristics of registered and non-registered trials. We assessed the quality and consistency of primary outcome measure (POM) reporting between the registration data and the final published studies.

RESULTS

Ninety studies met the inclusion criteria. Of those, only 28 (31%) were registered, and only 3 (3%) were "appropriately registered" (i.e., prospectively registered and identifying and fully describing the POM). Registered trials had larger sample sizes and were more likely to be multicenter, to report funding sources, and to be published in higher-impact-factor journals. Sixteen (18%) of the 90 registered RCTs named a POM in the registry; 7 (44%) of those 16 registered RCTs stated a different POM, an additional POM, or no POM at all in the final publication than was stated in the registry data. Additionally, 13 (81%) of those 16 registered RCTs had discrepancies in the time point reported for the POM.

CONCLUSIONS

In an attempt to address publication and outcome-reporting bias, prospective trial registration in a public registry has been deemed a condition for publication by the International Committee of Medical Journal Editors (ICMJE) since 2005. This study shows poor registration rates as well as inconsistencies in the reporting of POMs of recent trials relating to the treatment of distal radial fracture, one of the most common and most investigated injuries in orthopaedic practice.

CLINICAL RELEVANCE

The problems of registration and outcome-reporting bias in RCTs are important to highlight and address, and to find a solution will require the cooperation of researchers, reviewers, and journal editors. Increasing the transparency and consistency of reporting will help to increase the quality of research, which can impact patient care through evidence-based guidelines.

An Evaluation of Reporting Guidelines and Clinical Trial Registry Requirements Among Orthopaedic Surgery Journals

BACKGROUND

The responsibility for ensuring that studies are adequately reported is primarily that of those conducting the study; however, journal policies may influence how thoroughly authors choose to report their research. The use of reporting guidelines and prospective trial registration are promising avenues for ensuring that published studies adhere to the highest methodological standards. The purpose of this study is to evaluate orthopaedic surgery journal policies regarding reporting guidelines and trial registration, and to evaluate the effects that these policies have on adherence to reporting.

METHODS

We conducted a cross-sectional survey of journal policies and "Instructions for Authors" to determine the journals' policies and guidance regarding use of reporting guidelines and study registration. We also examined whether trials published in journals referencing CONSORT (Consolidated Standards of Reporting Trials) had higher rates of compliance with publishing a CONSORT flow diagram and whether journals with trial registration policies were more likely to contain registered trials than journals without these requirements.

RESULTS

Of the 21 orthopaedic surgery journals, 6 (29%) did not mention a single guideline, and clinical trial registration was required by 11 (52%) orthopaedic surgery journals and recommended by 2 (10%). Of the 21 general medical journals, 3 (14%) did not mention a single guideline, and trial registration was required by 13 (62%) general medical journals and recommended by 5 (24%) others. Furthermore, journals that referenced CONSORT were more likely to publish trials with a CONSORT flow diagram. Journals with trial registration policies were more likely to publish registered trials.

CONCLUSIONS

Reporting guidelines and trial registration are suboptimally required or recommended by orthopaedic surgery journals. These 2 mechanisms may improve methodology and quality, and should be considered for adoption by journal editors in orthopaedic surgery.

CLINICAL RELEVANCE

Because orthopaedic surgeons rely on high-quality research to direct patient care, measures must be taken to ensure that published research is of the highest quality. The use of reporting guidelines and prospective clinical trial registration may improve the quality of orthopaedic research, thereby improving patient care.

A systematic review of comparisons between protocols or registrations and full reports in primary biomedical research

BACKGROUND

Prospective study protocols and registrations can play a significant role in reducing incomplete or selective reporting of primary biomedical research, because they are pre-specified blueprints which are available for the evaluation of, and comparison with, full reports. However, inconsistencies between protocols or registrations and full reports have been frequently documented. In this systematic review, which forms part of our series on the state of reporting of primary biomedical, we aimed to survey the existing evidence of inconsistencies between protocols or registrations (i.e., what was planned to be done and/or what was actually done) and full reports (i.e., what was reported in the literature); this was based on findings from systematic reviews and surveys in the literature.

METHODS

Electronic databases, including CINAHL, MEDLINE, Web of Science, and EMBASE, were searched to identify eligible surveys and systematic reviews. Our primary outcome was the level of inconsistency (expressed as a percentage, with higher percentages indicating greater inconsistency) between protocols or registration and full reports. We summarized the findings from the included systematic reviews and surveys qualitatively.

RESULTS

There were 37 studies (33 surveys and 4 systematic reviews) included in our analyses. Most studies (n = 36) compared protocols or registrations with full reports in clinical trials, while a single survey focused on primary studies of clinical trials and observational research. High inconsistency levels were found in outcome reporting (ranging from 14% to 100%), subgroup reporting (from 12% to 100%), statistical analyses (from 9% to 47%), and other measure comparisons. Some factors, such as outcomes with significant results, sponsorship, type of outcome and disease speciality were reported to be significantly related to inconsistent reporting.

CONCLUSIONS

We found that inconsistent reporting between protocols or registrations and full reports of primary biomedical research is frequent, prevalent and suboptimal. We also identified methodological issues such as the need for consensus on measuring inconsistency across sources for trial reports, and more studies evaluating transparency and reproducibility in reporting all aspects of study design and analysis. A joint effort involving authors, journals, sponsors, regulators and research ethics committees is required to solve this problem.

Agreement in reporting between trial publications and current clinical trial registry in high impact journals: A methodological review

OBJECTIVES

The primary objective of this systematic survey was to examine the percentage of studies in which there was agreement in the reporting of the primary outcome between the currently updated version of the clinical trial registry and the published paper. We also investigated the factors associated with agreement in reporting of the primary outcome.

METHODS

We searched PubMed for all randomized control trials (RCT)s published in 2012-2015 in the top five general medicine journals (based on the 2014 impact factor). Two hundred abstracts (50 from each year) were randomly selected for data extraction. Agreement in reporting of 11 key study conduct items (e.g., sample size) and study characteristics (e.g., funding, number of sites) were extracted by two independent reviewers.

ANALYSIS

Descriptive analyses were conducted to determine the proportion of studies on which there was agreement in reporting of key study conduct items. Generalized estimating equations were used to explore factors associated with agreement in reporting of the primary outcome.

RESULTS

Of the 200 included studies, 87% had agreement in reporting of the primary outcome. After adjusting for other covariates, having greater than 50 sites was associated with an increased likelihood of agreement in reporting of the primary outcome (odds ratio=7.1, 95% confidence interval=1.39, 36.27, p=0.018).

CONCLUSIONS

We identified substantive disagreement in reporting between publications and current clinical trial registry, which were associated with several study characteristics. Further measures are needed to improve reporting given the potential threats to the quality and integrity of scientific research.

Severity assessments used for inclusion criteria and baseline severity evaluation in atopic dermatitis clinical trials: a systematic review

BACKGROUND

Numerous inclusion criteria and baseline severity assessments are used in clinical trials of atopic dermatitis (AD), which may limit comparison of results.

OBJECTIVE

We sought to characterize the inclusion criteria and baseline severity assessments used in randomized controlled trials (RCT) of AD internationally.

METHODS

We performed a systematic review of RCT with a pharmacological intervention from 2007 to 2016. Cochrane Library, EMBASE, GREAT, LILACS, MEDLINE and Scopus were searched. Two authors independently performed study selection and data extraction.

RESULTS

Overall, 212 RCT met inclusion/exclusion criteria. Target population and inclusion criteria based on AD severity were not documented in 78 (36.8%) and 25 (18.7%) studies, respectively. Thirty and 58 severity assessments were used for inclusion criteria and baseline severity, respectively, with only 60.3% concordance between their uses. Global assessments were most frequently used for both inclusion criteria and baseline severity assessment in North America (39.5% and 32.1%), while SCORing AD (SCORAD) or objective-SCORAD index was most frequently used in Europe (23.5% and 23.0%) and Asia (34.2% and 43.5%). Minimum and maximum thresholds of severity assessments were inconsistently used between studies for inclusion criteria, even within similar target populations. SCORAD, global assessments and body surface area were most frequently used for both inclusion criteria and baseline severity assessment. IGA was particularly used in trials of topical agents.

CONCLUSIONS

There were considerable variability and poor documentation of inclusion criteria and baseline severity assessments in RCT for AD. These differences may limit interpretation of a study and comparison of results between studies.

Selective outcome reporting in obesity clinical trials: a cross-sectional review

Selective outcome reporting is a form of bias resulting from discrepancies between outcomes presented in a trial's registration and the published report. We investigate this selective bias in obesity clinical trials. A PubMed search was conducted to identify randomized controlled trials (RCTs) published in four obesity journals from 2013 to 2015. Primary, secondary and tertiary outcomes were recorded for each trial and compared to pre-specified outcomes in each trial's registration. Of the 392 identified articles, 142 were included in the final analysis; 22 (15%) RCTs demonstrated major outcome discrepancies between registration and publication: No primary outcomes were demoted to a secondary or tertiary outcome; 14 (36.84%) primary outcomes were omitted; 14 (36.84%) primary outcomes were added: 5 (13.16%) secondary outcomes were upgraded to primary outcomes; and timing of assessment for a primary outcome changed 5 (13.16%) times. Out of the 63 prospectively registered studies, 53 had no discrepancies. A total of 76 of the studies (29.80%) were unregistered or did not have an associated registration number. Our results suggest that selective outcome reporting may be a concern in obesity clinical trials. As selective outcome reporting may distort clinical findings and limit outcomes in systematic reviews, we encourage trialists and journal editors to work towards solutions to mitigate this issue.

Systematic review: Outcome reporting bias is a problem in high impact factor neurology journals

BACKGROUND

Selective outcome reporting is a significant methodological concern. Comparisons between the outcomes reported in clinical trial registrations and those later published allow investigators to understand the extent of selection bias among trialists. We examined the possibility of selective outcome reporting in randomized controlled trials (RCTs) published in neurology journals.

METHODS

We searched PubMed for randomized controlled trials from Jan 1, 2010 -Dec 31, 2015 published in the top 3 impact factor neurology journals. These articles were screened according to specific inclusion criteria. Each author individually extracted data from trials following a standardized protocol. A second author verified each extracted element and discrepancies were resolved. Consistency between registered and published outcomes was evaluated and correlations between discrepancies and funding, journal, and temporal trends were examined.

RESULTS

180 trials were included for analysis. 10 (6%) primary outcomes were demoted, 38 (21%) primary outcomes were omitted from the publication, and 61 (34%) unregistered primary outcomes were added to the published report. There were 18 (10%) cases of secondary outcomes being upgraded to primary outcomes in the publication, and there were 53 (29%) changes in timing of assessment. Of 82 (46%) major discrepancies with reported p-values, 54 (66.0%) favored publication of statistically significant results.

CONCLUSION

Across trials, we found 180 major discrepancies. 66% of major discrepancies with a reported p-value (n = 82) favored statistically significant results. These results suggest a need within neurology to provide more consistent and timely registration of outcomes.

A systematic review of the processes used to link clinical trial registrations to their published results

BACKGROUND

Studies measuring the completeness and consistency of trial registration and reporting rely on linking registries with bibliographic databases. In this systematic review, we quantified the processes used to identify these links.

METHODS

PubMed and Embase databases were searched from inception to May 2016 for studies linking trial registries with bibliographic databases. The processes used to establish these links were categorised as automatic when the registration identifier was available in the bibliographic database or publication, or manual when linkage required inference or contacting of trial investigators. The number of links identified by each process was extracted where available. Linear regression was used to determine whether the proportions of links available via automatic processes had increased over time.

RESULTS

In 43 studies that examined cohorts of registry entries, 24 used automatic and manual processes to find articles; 3 only automatic; and 11 only manual (5 did not specify). Twelve studies reported results for both manual and automatic processes and showed that a median of 23% (range from 13 to 42%) included automatic links to articles, while 17% (range from 5 to 42%) of registry entries required manual processes to find articles. There was no evidence that the proportion of registry entries with automatic links had increased (R ² = 0.02, p = 0.36). In 39 studies that examined cohorts of articles, 21 used automatic and manual processes; 9 only automatic; and 2 only manual (7 did not specify). Sixteen studies reported numbers for automatic and manual processes and indicated that a median of 49% (range from 8 to 97%) of articles had automatic links to registry entries, and 10% (range from 0 to 28%) required manual processes to find registry entries. There was no evidence that the proportion of articles with automatic links to registry entries had increased (R ² = 0.01, p = 0.73).

CONCLUSIONS

The linkage of trial registries to their corresponding publications continues to require extensive manual processes. We did not find that the use of automatic linkage has increased over time. Further investigation is needed to inform approaches that will ensure publications are properly linked to trial registrations, thus enabling efficient monitoring of trial reporting.

A programme of research to set priorities and reduce uncertainties for the prevention and treatment of skin disease

Background

Skin diseases are very common and can have a large impact on the quality of life of patients and caregivers. This programme addressed four diseases: (1) eczema, (2) vitiligo, (3) squamous cell skin cancer (SCC) and (4) pyoderma gangrenosum (PG).

Objective

To set priorities and reduce uncertainties for the treatment and prevention of skin disease in our four chosen diseases.

Design

Mixed methods including eight systematic reviews, three prioritisation exercises, two pilot randomised controlled trials (RCTs), three feasibility studies, two core outcome initiatives, four funding proposals for national RCTs and one completed national RCT.

Setting

Secondary care, primary care and the general population.

Participants

Patients (and their caregivers) with eczema, vitiligo, SCC and PG, plus health-care professionals with an interest in skin disease.

Interventions

Our three intervention studies included (1) barrier enhancement using emollients from birth to prevent eczema (pilot RCT); (2) handheld narrowband ultraviolet light B therapy for treating vitiligo (pilot RCT); and (3) oral ciclosporin (Neoral®, Novartis Pharmaceuticals) compared with oral prednisolone for managing PG (pragmatic national RCT).

Results

Systematic reviews included two overarching systematic reviews of RCTs of treatments for eczema and vitiligo, an umbrella review of systematic reviews of interventions for the prevention of eczema, two reviews of treatments for SCC (one included RCTs and the second included observational studies), and three reviews of outcome measures and outcome reporting. Three prioritisation partnership exercises identified 26 priority areas for future research in eczema, vitiligo and SCC. Two international consensus initiatives identified four core domains for future eczema trials and seven core domains for vitiligo trials. Two pilot RCTs and three feasibility studies critically informed development of four trial proposals for external funding, three of which are now funded and one is pending consideration by funders. Our pragmatic RCT tested the two commonly used systemic treatments for PG (prednisolone vs. ciclosporin) and found no difference in their clinical effectiveness or cost-effectiveness. Both drugs showed limited benefit. Only half of the participants’ ulcers had healed by 6 months. For those with healed ulcers, recurrence was common (30%). Different side effect profiles were noted for each drug, which can inform clinical decisions on an individual patient basis. Three researchers were trained to PhD level and a dermatology patient panel was established to ensure patient involvement in all aspects of the programme.

Conclusions

Findings from this programme of work have already informed clinical guidelines and patient information resources. Feasibility studies have ensured that large national pragmatic trials will now be conducted on important areas of treatment uncertainty that address the needs of patients and the NHS. There is scope for considerable improvement in terms of trial design, conduct and reporting for RCTs of skin disease, which can be improved through wider collaboration, registration of trial protocols and complete reporting and international consensus over core outcome sets. Three national trials have now been funded as a result of this work. Two international initiatives to establish how best to measure the core outcome domains for eczema and vitiligo are ongoing.

Trial registration

Current Controlled Trials Barrier Enhancement for Eczema Prevention (BEEP) (ISRCTN84854178 and NCT01142999), Study of Treatments fOr Pyoderma GAngrenosum Patients (STOP GAP) (ISRCTN35898459) and Hand Held NB-UVB for Early or Focal Vitiligo at Home (HI-Light Pilot Trial) (NCT01478945).

Funding

This project was funded by the National Institute for Health Research (NIHR) Programme Grants for Applied Research programme and will be published in full inProgramme Grants for Applied Research; Vol. 4, No. 18. See the NIHR Journals Library website for further project information.

Outcome Reporting Bias in Government-Sponsored Policy Evaluations: A Qualitative Content Analysis of 13 Studies

The reporting of evaluation outcomes can be a point of contention between evaluators and policy-makers when a given reform fails to fulfil its promises. Whereas evaluators are required to report outcomes in full, policy-makers have a vested interest in framing these outcomes in a positive light–especially when they previously expressed a commitment to the reform. The current evidence base is limited to a survey of policy evaluators, a study on reporting bias in education research and several studies investigating the influence of industry sponsorship on the reporting of clinical trials. The objective of this study was twofold. Firstly, it aimed to assess the risk of outcome reporting bias (ORB or ‘spin’) in pilot evaluation reports, using seven indicators developed by clinicians. Secondly, it sought to examine how the government’s commitment to a given reform may affect the level of ORB found in the corresponding evaluation report. To answer these questions, 13 evaluation reports were content-analysed, all of which found a non-significant effect of the intervention on its stated primary outcome. These reports were systematically selected from a dataset of 233 pilot and experimental evaluations spanning three policy areas and 13 years of government-commissioned research in the UK. The results show that the risk of ORB is real. Indeed, all studies reviewed here resorted to at least one of the presentational strategies associated with a risk of spin. This study also found a small, negative association between the seniority of the reform’s champion and the risk of ORB in the evaluation of that reform. The publication of protocols and the use of reporting guidelines are recommended.

Publication proportions for registered breast cancer trials: before and following the introduction of the ClinicalTrials.gov results database

Background

To limit selective and incomplete publication of the results of clinical trials, registries including ClinicalTrials.gov were introduced. The ClinicalTrials.gov registry added a results database in 2008 to enable researchers to post the results of their trials as stipulated by the Food and Drug Administration Amendment Act of 2007. This study aimed to determine the direction and magnitude of any change in publication proportions of registered breast cancer trials that occurred since the inception of the ClinicalTrials.gov results database.

Methods

A cross-sectional study design was employed using ClinicalTrials.gov, a publicly available registry/results database as the primary data source. Registry contents under the subcategories ‘Breast Neoplasms’ and ‘Breast Neoplasms, Male’ were downloaded on 1 August 2015. A literature search for included trials was afterwards conducted using MEDLINE and DISCOVER databases to determine publication status of the registered breast cancer trials.

Results

Nearly half (168/340) of the listed trials had been published, with a median time to publication of 24 months (Q1 = 14 months, Q3 = 42 months). Only 86 trials were published within 24 months of completion. There was no significant increase in publication proportions of trials that were completed before the introduction of the results database compared to those completed after (OR = 1.00, 95 % CI = .61 to 1.63; adjusted OR = 0.84, 95 % CI = .51 to 1.39). Characteristics associated with publication included trial type (observational versus interventional adjusted OR = .28, 95 % CI = .10 to .74) and completion/termination status (terminated versus completed adjusted OR = .22, 95 % CI = .09 to .51).

Conclusions

Less than a half of breast cancer trials registered in ClinicalTrials.gov are published in peer-reviewed journals.

Electronic supplementary material

The online version of this article (doi:10.1186/s41073-016-0017-4) contains supplementary material, which is available to authorized users.

Protocol registration and selective outcome reporting in recent psychiatry trials: new antidepressants and cognitive behavioural therapies

OBJECTIVE

The selective reporting of favorable outcomes has a serious influence on our evidence base. However, this problem has not yet been systematically investigated in the field of psychiatry. Our study aimed to evaluate registration and outcome reporting in randomized controlled trials (RCTs) of standard treatments for depression: cognitive behavioural therapy (CBT) or new-generation antidepressants (ADs).

METHOD

We searched for reports of RCTs examining the efficacy of CBT or AD for depression that were published between 2011 and 2013. We then compared their primary outcomes in the trial registries and those in publications.

RESULTS

We identified 170 trials. Among them, 92 trials (54.1%) were registered, 43 trials (25.3%) were properly registered, and only 32 (18.8%) trials were both properly registered and reported (the primary outcomes as recorded in the registries were reported in publications). There was no statistically significant difference in the proportions of properly registered and reported trials for CBT or AD (relative risk: 0.51, 95% CI: 0.25-1.03). High impact factor journals, commercial funding, publication of protocol, and relatively large sample size were significant predictors of proper registration and reporting.

CONCLUSION

The prevalence of proper registration and reporting is still very low in depression trials.

Comparison of registered and published outcomes in randomized controlled trials: a systematic review

BACKGROUND

Clinical trial registries can improve the validity of trial results by facilitating comparisons between prospectively planned and reported outcomes. Previous reports on the frequency of planned and reported outcome inconsistencies have reported widely discrepant results. It is unknown whether these discrepancies are due to differences between the included trials, or to methodological differences between studies. We aimed to systematically review the prevalence and nature of discrepancies between registered and published outcomes among clinical trials.

METHODS

We searched MEDLINE via PubMed, EMBASE, and CINAHL, and checked references of included publications to identify studies that compared trial outcomes as documented in a publicly accessible clinical trials registry with published trial outcomes. Two authors independently selected eligible studies and performed data extraction. We present summary data rather than pooled analyses owing to methodological heterogeneity among the included studies.

RESULTS

Twenty-seven studies were eligible for inclusion. The overall risk of bias among included studies was moderate to high. These studies assessed outcome agreement for a median of 65 individual trials (interquartile range [IQR] 25-110). The median proportion of trials with an identified discrepancy between the registered and published primary outcome was 31%; substantial variability in the prevalence of these primary outcome discrepancies was observed among the included studies (range 0% (0/66) to 100% (1/1), IQR 17-45%). We found less variability within the subset of studies that assessed the agreement between prospectively registered outcomes and published outcomes, among which the median observed discrepancy rate was 41% (range 30% (13/43) to 100% (1/1), IQR 33-48%). The nature of observed primary outcome discrepancies also varied substantially between included studies. Among the studies providing detailed descriptions of these outcome discrepancies, a median of 13 % of trials introduced a new, unregistered outcome in the published manuscript (IQR 5-16%).

CONCLUSIONS

Discrepancies between registered and published outcomes of clinical trials are common regardless of funding mechanism or the journals in which they are published. Consistent reporting of prospectively defined outcomes and consistent utilization of registry data during the peer review process may improve the validity of clinical trial publications.

How Often Do Comparative Randomised Controlled Trials in the Field of Eczema Fail to Directly Compare the Treatments Being Tested?

The objective of the study was to identify all parallel design randomised controlled trials (RCTs) comparing treatments for eczema in recent dermatology literature that have failed to report a between-group analysis. The GREAT database (www.greatdatabase.org.uk) was searched to identify parallel group RCTs comparing two or more interventions published in the English language in the last decade, 2004 to 2013. The primary outcome was the number of studies that had not reported a between-group analysis for any of the outcomes. Where possible we re-analysed the data to determine whether a between-group analysis would have given a different conclusion to that reported. Out of a total of 304 RCTs in the study period, 173 (56.9%) met the inclusion criteria. Of the 173 eligible studies, 12 (6.9%) had not conducted a between-group analysis for any of the reported outcomes. There was no clear improvement over time. Five of the eight studies that were re-analysed yielded non-significant between-group differences yet reported significant within-group comparisons. All but one of the 12 studies implied that the experimental intervention was successful despite not undertaking any between-group comparisons. Although the proportion of all RCTs that fail to report an appropriate between-group analysis is small, the fact that any scientist who purports to compare one treatment against another then chooses to omit the key comparison statistic is worrying.

Declaration of transparency for each research article

An antidote to inadequate reporting of research

Use of trial register information during the peer review process

INTRODUCTION

Evidence in the medical literature suggests that trial registration may not be preventing selective reporting of results. We wondered about the place of such information in the peer-review process.

METHOD

We asked 1,503 corresponding authors of clinical trials and 1,733 reviewers to complete an online survey soliciting their views on the use of trial registry information during the peer-review process.

RESULTS

1,136 authors (n = 713) and reviewers (n = 423) responded (37.5%); 676 (59.5%) had reviewed an article reporting a clinical trial in the past 2 years. Among these, 232 (34.3%) examined information registered on a trial registry. If one or more items (primary outcome, eligibility criteria, etc.) differed between the registry record and the manuscript, 206 (88.8%) mentioned the discrepancy in their review comments, 46 (19.8%) advised editors not to accept the manuscript, and 8 did nothing. The reviewers' reasons for not using the trial registry information included a lack of registration number in the manuscript (n = 132; 34.2%), lack of time (n = 128; 33.2%), lack of usefulness of registered information for peer review (n = 100; 25.9%), lack of awareness about registries (n = 54; 14%), and excessive complexity of the process (n = 39; 10.1%).

CONCLUSION

This survey revealed that only one-third of the peer reviewers surveyed examined registered trial information and reported any discrepancies to journal editors.