Evidence of selective reporting bias in hematology journals: A systematic review

Evidence-b(i)ased practice: Selective and inadequate reporting in early childhood autism intervention research

LAY ABSTRACT

When researchers fail to report their findings or only report some of their findings, it can make it difficult for clinicians to provide effective intervention recommendations. However, no one has examined whether this is a problem in studies of early childhood autism interventions. We studied how researchers that study early childhood autism interventions report their findings. We found that most researchers did not register their studies when they were supposed to (before the start of the study), and that many researchers did not provide all of the needed information in the registration. We also found that researchers frequently did not publish their findings when their studies were complete. When we looked at published reports, we found that many of the studies did not report enough information, and that many studies were reported differently from their registrations, suggesting that researchers were selectively reporting positive outcomes and ignoring or misrepresenting less positive outcomes. Because we found so much evidence that researchers are failing to report their findings quickly and correctly, we suggested some practical changes to make it better.

Selective outcome reporting in randomized clinical trials using the third molar surgery model

Selective outcome reporting (SOR) can threaten the validity of results found in clinical trials. Some studies in the literature have analyzed SOR in dentistry, but there is no study that has observed SOR in clinical trials in oral and maxillofacial surgery. Impacted third molar surgery is one of the most used models in clinical trials to study mainly analgesic and anti-inflammatory drug interventions. Our study aimed to evaluate the prevalence of SOR in publications employing the third molar extraction clinical trial model, and to verify whether there was an association between the statistical significance of outcomes and other characteristics that could lead to SOR. A systematic search was performed on the ClinicialTrials.gov platform for randomized clinical trial protocols, using the condition of third molar extraction. The corresponding published articles were sourced in PubMed, Scopus, and Embase databases, and compared with the registered protocols regarding the methodological data, in terms of: sample calculation, primary outcome identification, end-point periods, insertion of new outcomes in the publication, and results of outcomes. 358 protocol records were retrieved; 87 presented their corresponding articles. SOR was identified in 28.74% of the publications, and had a significant relationship with changes in the protocol, insertions of new outcomes, and discrepancies in the types of study. General risk of bias was found to be low. There were associations between SOR and the discrepancies in terms of the type of study, the choice of new outcome, and changes in the history of protocol records. The prevalence of SOR in clinical research using the third molar extraction surgery model is moderate. The quality of the scientific reporting of the results and, consequently, the certainty of evidence relating to the intervention tested can be overstated, increasing the chances of misinterpretation by health professionals.

Is Childhood Immune Thrombocytopenia Associated With Acquired Toxoplasmosis? An Unusual Case of Infection That Led to Acute ITP in a Greek Male Toddler and Implications for Guidelines

INTRODUCTION

Childhood immune thrombocytopenic purpura (ITP) is a heterogeneous immune-mediated process triggered by infections, vaccines, allergies and parasites. Currently, there is little evidence in the literature beyond case reports of an association with Toxoplasma gondii (T. gondii).

METHODS

The authors describe the unusual case of an earlier healthy 2.5-year-old Greek boy who developed acute ITP with a life-threatening platelet count a few days after a T. gondii infection. Evidence for the infection onset was found incidentally 3 months after the initial admission to the hospital and only after any other plausible cause of thrombocytopenia was excluded, according to diagnosis guidelines.

RESULTS

The boy underwent 3 intravenous immunoglobulin treatments within a trimester, a period during which his alarming platelets count levels led to housebound activities. A quite slow recovery was only ignited after the third treatment, which was administered in conjunction with a mild antibiotic medication for the T. gondii infection. Full recovery was obtained 9 months after the initial admission, although the boy's potential scored high in clinical prediction models for developing transient ITP.

CONCLUSION

There is a need for more research on ITPs with no obvious cause to investigate a causal association with toxoplasmosis. Currently, testing for diseases of greater rarity and of higher diagnostic cost than T. gondii is included in the ITP guidelines. Hence, routinely testing for toxoplasmosis when considering potential childhood ITP triggers and infection treatment complementary to treating the ITP might be the key to accelerating the healing process and improving the quality of life of otherwise confined children.

Reasons for missing evidence in rehabilitation meta-analyses: a cross-sectional meta-research study

BACKGROUND

Systematic reviews of randomized controlled trials are the best evidence for informing on intervention effectiveness. Their results, however, can be biased due to omitted evidence in the quantitative analyses. We aimed to assess the proportion of randomized controlled trials omitted from meta-analyses in the rehabilitation field and explore related reasons.

METHODS

This is a cross-sectional meta-research study. For each systematic review included in a published selected sample in the rehabilitation field, we identified an index meta-analysis on the primary outcome and the main comparison. We then looked at all the studies considered eligible for the chosen comparison in the systematic review and identified those trials that have been omitted (i.e., not included) from each index meta-analysis. Reasons for omission were collected based on an eight-reason classification. We used descriptive statistics to describe the proportion of omitted trials overall and according to each reason.

RESULTS

Starting from a cohort of 827 systematic reviews, 131 index meta-analyses comprising a total of 1761 eligible trials were selected. Only 16 index meta-analyses included all eligible studies while 15 omitted studies without providing references. From the remaining 100 index meta-analyses, 717 trials (40,7%) were omitted overall. Specific reasons for omission were: "unable to distinguish between selective reporting and inadequate planning" (39,3%, N = 282), "inadequate planning" (17%, N = 122), "justified to be not included" (15,1%, N = 108), "incomplete reporting" (8,4%, N = 60), "selective reporting" (3,3%, N = 24) and other situations (e.g., outcome present but no motivation for omission) (5,2%, N = 37). The 11,7% (N = 84) of omitted trials were not assessed due to non-English language or full text not available.

CONCLUSIONS

Almost half of the eligible trials were omitted from their index meta-analyses. Better reporting, protocol registration, definition and adoption of core outcome sets are needed to prevent omission of evidence in systematic reviews.

Estimating the prevalence of discrepancies between study registrations and publications: a systematic review and meta-analyses

OBJECTIVES

Prospectively registering study plans in a permanent time-stamped and publicly accessible document is becoming more common across disciplines and aims to reduce risk of bias and make risk of bias transparent. Selective reporting persists, however, when researchers deviate from their registered plans without disclosure. This systematic review aimed to estimate the prevalence of undisclosed discrepancies between prospectively registered study plans and their associated publication. We further aimed to identify the research disciplines where these discrepancies have been observed, whether interventions to reduce discrepancies have been conducted, and gaps in the literature.

DESIGN

Systematic review and meta-analyses.

DATA SOURCES

Scopus and Web of Knowledge, published up to 15 December 2019.

ELIGIBILITY CRITERIA

Articles that included quantitative data about discrepancies between registrations or study protocols and their associated publications.

DATA EXTRACTION AND SYNTHESIS

Each included article was independently coded by two reviewers using a coding form designed for this review (osf.io/728ys). We used random-effects meta-analyses to synthesise the results.

RESULTS

We reviewed k=89 articles, which included k=70 that reported on primary outcome discrepancies from n=6314 studies and, k=22 that reported on secondary outcome discrepancies from n=1436 studies. Meta-analyses indicated that between 29% and 37% (95% CI) of studies contained at least one primary outcome discrepancy and between 50% and 75% (95% CI) contained at least one secondary outcome discrepancy. Almost all articles assessed clinical literature, and there was considerable heterogeneity. We identified only one article that attempted to correct discrepancies.

CONCLUSIONS

Many articles did not include information on whether discrepancies were disclosed, which version of a registration they compared publications to and whether the registration was prospective. Thus, our estimates represent discrepancies broadly, rather than our target of undisclosed discrepancies between prospectively registered study plans and their associated publications. Discrepancies are common and reduce the trustworthiness of medical research. Interventions to reduce discrepancies could prove valuable.

REGISTRATION

osf.io/ktmdg. Protocol amendments are listed in online supplemental material A.

Selective outcome reporting bias is highly prevalent in randomized clinical trials of nonsurgical periodontal therapy

Selective outcome reporting (SOR) is a type of bias that can compromise the validity of results and affect evidence-based practice. SOR can overestimate the effect of an intervention and lead to conclusions that a treatment is effective when it is not. This study aimed to investigate the prevalence of SOR in publications of RCTs on nonsurgical periodontal therapy (NSPT) and to verify associated factors. The protocols were searched and selected on the www.clinicaltrials.gov platform up to January 16, 2022. Corresponding publications were identified, and data extraction and discrepancy analysis were performed. The risk of bias was assessed according to the RoB2 tool. One hundred forty-five studies (174 publications) were included. The prevalence of SOR was 49.7% and was unclear in nearly one third of studies (27.6%). Only 31.7% of the primary outcomes were completely described in the publications. The overall risk of bias was high in 60% of the included studies. SOR was associated with statistical significance (p < .001), and multiple publications of the same study (p = .005). Our study demonstrated the high prevalence of SOR, highlighting the need to improve the quality of reporting of RCTs on NSPT studies.

Clinical trial data-sharing policies among journals, funding agencies, foundations, and other professional organizations: a scoping review

BACKGROUND AND OBJECTIVES

To identify the similarities and differences in data-sharing policies for clinical trial data that are endorsed by biomedical journals, funding agencies, and other professional organizations. Additionally, to determine the beliefs, and opinions regarding data-sharing policies for clinical trials discussed in articles published in biomedical journals.

METHODS

Two searches were conducted, a bibliographic search for published articles that present beliefs, opinions, similarities, and differences regarding policies governing the sharing of clinical trial data. The second search analyzed the gray literature (non-peer-reviewed publications) to identify important data-sharing policies in selected biomedical journals, foundations, funding agencies, and other professional organizations.

RESULTS

A total of 471 articles were included after database search and screening, with 45 from the bibliographic search and 426 from the gray literature search. A total of 424 data-sharing policies were included. Fourteen of the 45 published articles from the bibliographic search (31.1%) discussed only advantages specific to data-sharing policies, 27 (27/45; 60%) discussed both advantages and disadvantages, and 4 (4/45; 8.9%) discussed only disadvantages specific. A total of 216 journals (of 270; 80%) specified a data-sharing policy provided by the journal itself. One hundred industry data-sharing policies were included, and 32 (32%) referenced a data-sharing policy on their website. One hundred and thirty-six (42%) organizations (of 327) specified a data-sharing policy.

CONCLUSION

We found many similarities listed as advantages to data-sharing and fewer disadvantages were discussed within the literature. Additionally, we found a wide variety of commonalities and differences-such as the lack of standardization between policies, and inadequately addressed details regarding the accessibility of research data-that exist in data-sharing policies endorsed by biomedical journals, funding agencies, and other professional organizations. Our study may not include information on all data sharing policies and our data is limited to the entities' descriptions of each policy.

Selective outcome reporting in paediatric dentistry restorative treatment randomised clinical trials-A meta-research

BACKGROUND

Selective outcome reporting (SOR) is a bias that occurs when the primary outcome of a randomised clinical trial (RCT) is omitted or changed.

AIM

To evaluate the prevalence of SOR in RCTs on restorative treatment in primary teeth.

DESIGN

We conducted an electronic search on ClinicalTrials.gov and the World Health Organization platform (International Clinical Trials Registry Platform) on 1^st of April 2021, with no registry time or language restrictions. We included RCT protocols that evaluated restorative treatments in primary teeth and excluded trials that did not have a complete publication in a scientific journal. The chi-squared test was used to identify the association between SOR and variables as a discrepancy in the follow-up period, the timing of registration, the type of sponsorship and the type of study design (α = 5%).

RESULTS

Of the 294 identified protocols, 30 were included in the study. 83.3% of trials were registered retrospectively. SOR was observed in 53.3% (n = 16) of the published trials and was significantly associated with a discrepancy in the follow-up period (p = .017).

CONCLUSIONS

The high prevalence of SOR in RCTs on restorative treatment proves that this is a prominent threat. A proper preregistered protocol, declaration of any protocol deviation and allowance of stakeholders to compare the protocol with that of the submitted papers will achieve transparency.

Digital training for building resilience: Systematic review, meta-analysis, and meta-regression

Resilience is learnable and broadly described as an individual's adaptive coping ability, its potential value for stress reduction must be explored. With a global coronavirus pandemic, innovative ways to deliver resilience training amidst heightened mental health concerns must be urgently examined. This systematic review aimed to (1) evaluate the effectiveness of digital training for building resilience and reducing anxiety, depressive and stress symptoms and (2) to identify essential features for designing future digital training. A three-step search was conducted in eight electronic databases, trial registries and grey literature to locate eligible studies. Randomised controlled trials examining the effects of digital training aimed at enhancing resilience were included. Data analysis was conducted using the Stata version 17. Twenty-two randomised controlled trials involving 2876 participants were included. Meta-analysis revealed that digital training significantly enhanced the participants' resilience with moderate to large effect (g = 0.54-1.09) at post-intervention and follow-up. Subgroup analyses suggested that training delivered via the Internet with a flexible programme schedule was more effective than its counterparts. This review supports the use of digital training in improving resilience. Further high-quality randomised controlled trials with large sample size are needed.

Evaluating Reporting Completeness of Patient-Reported Outcomes in Esophageal Motility Disorders: A Cross-Sectional Analysis of Randomized Controlled Trials

Esophageal motility disorders (EMD) can have significant effects on quality of life. Patient-reported outcomes (PROs) provide valuable insight into the patient's perspective on their treatment and are becoming increasingly used in randomized controlled trials (RCTs). Thus, our investigation aims to evaluate the completeness of reporting of PROs in RCTs pertaining to EMDs. We searched MEDLINE, Embase, and Cochrane Central Register of Controlled Trials for published RCTs focused on EMDs. Included RCTs were published between 2006 and 2020, reported a primary outcome related to an EMDs, and listed at least one PRO measure as a primary or secondary outcome. Investigators screened and extracted data in a masked, duplicate fashion. Data extraction was carried out using both the CONSORT-PRO adaptation and Cochrane Collaboration Risk of Bias 2.0 tool. We assessed overall mean percent completion of the CONSORT-PRO adaptation and a bivariate regression analysis was used to assess relationships between trial characteristics and completeness of reporting. The overall mean percent completion of the CONSORT-PRO checklist adaptation was 43.86% (SD = 17.03). RCTs with a primary PRO had a mean completeness of 47.73% (SD = 17.32) and RCTs with a secondary PRO was 35.36% (SD = 13.52). RCTs with a conflict of interest statement were 18.15% (SE = 6.5) more complete (t = 2.79, P = .009) than trials lacking a statement. No additional significant associations between trial characteristics and completeness of reporting were found. PRO reporting completeness in RCTs focused on EMDs was inadequate. We urge EMD researchers to prioritize complete PRO reporting to foster patient-centered research for future RCTs on EMDs.

Registry versus publication: discrepancy of primary outcomes and possible outcome reporting bias in child and adolescent mental health

Outcome reporting bias is one of the fundamental forms of publication bias. It implies publishing only outcomes that have positive results. The aim of this observational study was to explore primary outcome discrepancies between registry of clinical trials and their corresponding publications, since these can indicate outcome reporting bias in child mental health. Data were extracted from completed interventional clinical trials from ClinicalTrial.gov registry and its Archive site. Trials were registered under "Behaviours and Mental Disorders" category, and conducted on underage participants (0-17 years). Their primary outcomes were compared to those published in publication which had a corresponding NCT number stated in the text. Sixteen percent of trials did not have the minimum information on primary outcome stated in the registry-neither the measure used nor the measurement time points; 38.9% of trials had the minimum information stated to describe primary outcome, while only 3.3% of trials had all the necessary elements stated in the registry. Most of the publication in our sample had positive results (66.4%). Half of the trials registered before completion had non-matching primary outcomes in the registry and publication; 85.4% of trials with non-matching outcomes indicated possible outcome reporting bias for some of the primary outcome. Middle-sized trials and industry-funded trials were related with higher quality of primary outcome registration. Industry funding was related with positive findings in publication. Non-industry funding proved to be the only significant predictor of discrepancy between registered and published primary outcomes, and possible outcome reporting bias. Journal impact factor was not related with any of the outcome measures. The main limitation of the study is that it primarily offers an insight into discrepancy of registered and published outcomes. The methodology does not imply an access to results of unpublished outcomes - therefore, it was not possible to determine the presence of the bias with sufficient certainty in large number of trials. Further research should be done with improved methodology and additional data.

Usefulness of Cochrane Reviews in Clinical Guideline Development-A Survey of 585 Recommendations

The Danish Health Authority develops clinical practice guidelines to support clinical decision-making based on the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) system and prioritizes using Cochrane reviews. The objective of this study was to explore the usefulness of Cochrane reviews as a source of evidence in the development of clinical recommendations. Evidence-based recommendations in guidelines published by the Danish Health Authority between 2014 and 2021 were reviewed. For each recommendation, it was noted if and how Cochrane reviews were utilized. In total, 374 evidence-based recommendations and 211 expert consensus recommendations were published between 2014 and 2021. Of the 374 evidence-based recommendations, 106 included evidence from Cochrane reviews. In 28 recommendations, all critical and important outcomes included evidence from Cochrane reviews. In 36 recommendations, a minimum of all critical outcomes included evidence from Cochrane reviews, but not all important outcomes. In 33 recommendations, some but not all critical outcomes included evidence from Cochrane reviews. Finally, in nine recommendations, some of the important outcomes included evidence from Cochrane reviews. In almost one-third of the evidence-based recommendations, Cochrane reviews were used to inform clinical recommendations. This evaluation should inform future evaluations of Cochrane review uptake in clinical practice guidelines concerning outcomes important for clinical decision-making.

Evaluation of Risks of Bias in Addiction Medicine Randomized Controlled Trials

AIMS

Perhaps the most important step when designing and conducting randomized controlled trials (RCTs) in addiction is to put methodological safeguards in place to minimize the likelihood for bias to affect trial outcomes. In this study, we applied the revised Cochrane risk of bias tool (ROB 2) to RCTs of drug, alcohol or tobacco interventions.

METHODS

We searched for trials published in 15 addiction medicine journals over a 7-year period. Our primary endpoint is the risk of bias of included studies. We conducted a sensitivity analysis of publicly funded trials.

RESULTS

Overall, included RCTs were most often at high risk of bias per our judgments (244/487, 50.1%). However, significant proportions of included RCTs were at low risk of bias (123/487, 25.3%) or some concerns for bias (120/497, 24.6%). RCTs with behavioral modification interventions (19/44, 43.2%) and alcohol interventions (80/150, 53.3%) had the highest proportion of high-risk judgments. In a sensitivity analysis of publicly funded RCTs), 195/386 (50.5%) were at high risk of bias.

CONCLUSIONS

Approximately half of included drug, alcohol or tobacco RCTs in our sample were judged to be at high risk of bias with the most common reason being a lack of proper blinding or proper description of blinding. Key action items to reduce bias in future addiction RCTs include adequate randomization, blinding and inclusion of a trial registry number and protocol.

Selective outcome reporting in root coverage randomized clinical trials

BACKGROUND

Outcome discrepancies between protocols and respective publications represent a concerning bias. The purpose of this study was to assess the prevalence of selective outcome reporting (SOR) in root coverage randomized clinical trials (RCTs).

METHODS

Published root coverage RCTs (July 2005 to March 2020) were included if a corresponding protocol could be identified in a public registry. Discrepancies between protocol and its correspondent publication(s) were compared regarding primary and secondary outcomes and other study characteristics. Associations between trial characteristics and SOR were evaluated.

RESULTS

Forty four studies (54 publications) were included. The majority of studies (77.3%) were retrospectively registered. SOR was frequent (40.9% of trials) and consisted of primary outcome downgrade (22.7%); secondary outcome upgrade (11.4%); new primary outcome introduced in publication (25%); protocol primary outcome omitted from publication (13.6%) and discrepancy in primary outcome timing (18.2%). SOR was unclear in 20.5% of studies and favoured statistical significance in 12 studies (27.3%). SOR was significantly associated with study significance (p < 0.001) and unclear outcome definition in the publication (p < 0.001). Only a third (32.8%) of primary outcomes were completely defined.

CONCLUSIONS

The present study identified high prevalence of SOR in root coverage RCTs.

Ensuring the quality and specificity of preregistrations

Researchers face many, often seemingly arbitrary, choices in formulating hypotheses, designing protocols, collecting data, analyzing data, and reporting results. Opportunistic use of "researcher degrees of freedom" aimed at obtaining statistical significance increases the likelihood of obtaining and publishing false-positive results and overestimated effect sizes. Preregistration is a mechanism for reducing such degrees of freedom by specifying designs and analysis plans before observing the research outcomes. The effectiveness of preregistration may depend, in part, on whether the process facilitates sufficiently specific articulation of such plans. In this preregistered study, we compared 2 formats of preregistration available on the OSF: Standard Pre-Data Collection Registration and Prereg Challenge Registration (now called "OSF Preregistration," http://osf.io/prereg/). The Prereg Challenge format was a "structured" workflow with detailed instructions and an independent review to confirm completeness; the "Standard" format was "unstructured" with minimal direct guidance to give researchers flexibility for what to prespecify. Results of comparing random samples of 53 preregistrations from each format indicate that the "structured" format restricted the opportunistic use of researcher degrees of freedom better (Cliff's Delta = 0.49) than the "unstructured" format, but neither eliminated all researcher degrees of freedom. We also observed very low concordance among coders about the number of hypotheses (14%), indicating that they are often not clearly stated. We conclude that effective preregistration is challenging, and registration formats that provide effective guidance may improve the quality of research.

Evaluation of Selective Outcome Reporting Bias in Efficacy Endpoints in Print and Television Advertisements for Oncology Drugs

IMPORTANCE

Selective outcome reporting bias in oncology drug advertisements may encourage misconceptions about a drug's efficacy profile.

OBJECTIVE

We sought to determine the rates of selective outcome reporting in published cancer clinical trials and in television and print advertisements for anticancer medications. We also quantified the number of advertisements that did not include or cite any studies with mature overall survival (OS) data (i.e., data with all required patient events for final analysis).

DESIGN/SETTING/PARTICIPANTS

We conducted a cross-sectional investigation of advertisements uploaded to the AdPharm Database (repository of pharmaceutical advertisements); the clinical trials supporting the ads; and the trial registrations associated with the trials. Data were extracted by two investigators who were blinded to each other's data.

MAIN OUTCOME MEASURES

The first co-primary objective was to investigate selective outcome reporting between trial registrations and published trials. The second co-primary objective was to investigate selective outcome reporting between the same published trials and drug advertisements.

RESULTS

We included 74 advertisements and 48 clinical trials. Print ads were the most common (n = 66), and most print advertisements were targeted to health care providers (n = 55, 83.3%). Overall, 41/48 (85.4%) trials were registered prior to study enrollment, and 41/48 (85.4%) did not deviate from the registered primary endpoints. Across all advertisements (n = 74), statistically significant endpoints were more often reported (unadjusted risk ratio [uRR] 1.26; 95% confidence interval [CI] (1.14-1.40)) and 22/55 (40.0%) advertisements cited trials with immature overall survival data (i.e., data without the required number of events for final analysis).

CONCLUSIONS

In our sample, statistically significant endpoints were more commonly reported than nonsignificant endpoints. Immature endpoints (those analyzed before the required number of accrued patient events) were often reported. By reporting only significant endpoints and those that are immature, advertisers may encourage misconceptions about a drug's efficacy profile.

Risk of Bias and Quality of Reporting in Colon and Rectal Cancer Systematic Reviews Cited by National Comprehensive Cancer Network Guidelines

INTRODUCTION

Given the changing landscape of colorectal cancer, systematic reviews are likely to play a key role in advancing the understanding of prevention, diagnosis, and treatment.

METHODS

We conducted a cross-sectional investigation of the risk of bias and reporting quality of systematic reviews referenced by colon and rectal cancer National Comprehensive Cancer Network (NCCN) guidelines. We used two widely accepted tools: Risk of Bias in Systematic reviews (ROBIS) and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA).

RESULTS

Using ROBIS, only 3 (4.8%) systematic reviews were judged with low risk of bias, 35 (55.6%) systematic reviews were judged with unclear risk of bias, and 25 (39.7%) systematic reviews were judged with high risk of bias. Across all systematic reviews, the individual bias domains at the highest risk of bias were domains 1 (protocol and eligibility criteria) and 2 (methods to identify and select studies). Across all studies, the median adherence to PRISMA was 74.1% (IQR 69.2-80.0%), corresponding to approximately 20 of 27 items.

CONCLUSIONS

Systematic reviews cited in NCCN guidelines for colon and rectal cancer are frequently at unclear or high risk of bias and do not report key systematic review items that are important for the critical appraisal of results.

Evaluation of selective outcome reporting and trial registration practices among addiction clinical trials

BACKGROUND AND AIMS

Selective outcome reporting occurs when trialists pre-specify primary and secondary outcomes during trial planning but alter the definitions in the published report. Here, we investigate selective outcome reporting in published addiction randomized controlled trials (RCTs) and evaluate whether particular funding sources are associated with an increased likelihood of selective outcome reporting.

DESIGN

We conducted a cross-sectional study of published addiction clinical trials. A PubMed search was performed to identify RCTs in addiction journals from 2013 to 2017. Included studies used a randomized design to address one of the following topics: (1) drug, alcohol and tobacco addiction prevention, (2) stabilization following excessive use of a substance, (3) relapse prevention or (4) recovery maintenance.

SETTING

Single-center, medical research institution.

PARTICIPANTS

Our sample included 162 RCTs that were prospectively registered with a clearly defined primary outcome.

MEASUREMENT

We extracted the following items from addiction RCTs: journal, funding source, trial registry number (if included), sample size, dates of subject enrollment, whether primary and secondary outcomes were denoted, all published outcomes, P-value for all outcomes and whether authors mentioned any deviations from the trial protocol as it related to RCT outcomes.

FINDINGS

In total, 47 of 162 RCTs (29.0%) had at least one major discrepancy between the trial registry and published RCT. Overall, these 47 RCTs included 54 major discrepancies. The most common major discrepancy was demotion of a primary registered outcome (19/54, 35.2%). The majority of RCTs (132/162, 81.5%) were funded from public sources. Additionally, 166 RCTs were excluded from our sample because registration could not be confirmed.

CONCLUSIONS

There is evidence suggestive of selective outcome reporting in addiction randomized controlled trials (RCTs). The most common major discrepancies pertained to the primary outcome.

Development of a core outcome set for myelodysplastic syndromes - a Delphi study from the EUMDS Registry Group

Treatment options for myelodysplastic syndromes (MDS) vary widely, depending on the natural disease course and patient‐related factors. Comparison of treatment effectiveness is challenging as different endpoints have been included in clinical trials and outcome reporting. Our goal was to develop the first MDS core outcome set (MDS‐COS) defining a minimum set of outcomes that should be reported in future clinical studies. We performed a comprehensive systematic literature review among MDS studies to extract patient‐ and/or clinically relevant outcomes. Clinical experts from the European LeukemiaNet MDS (EUMDS) identified 26 potential MDS core outcomes and participated in a three‐round Delphi survey. After the first survey (56 experts), 15 outcomes met the inclusion criteria and one additional outcome was included. The second round (38 experts) resulted in six included outcomes. In the third round, a final check on plausibility and practicality of the six included outcomes and their definitions was performed. The final MDS‐COS includes: health‐related quality of life, treatment‐related mortality, overall survival, performance status, safety, and haematological improvement. This newly developed MDS‐COS represents the first minimum set of outcomes aiming to enhance comparability across future MDS studies and facilitate a better understanding of treatment effectiveness.

Data sharing practices in randomized trials of addiction interventions

INTRODUCTION

Transparent, open scientific research practices aim to improve the validity and reproducibility of research findings. A key component of open science is the public sharing of data and metadata that constitute the basis for research findings.

METHODS

We conducted a 6 year cross-sectional investigation of the rates and methods of data sharing in 15 high-impact addiction journals that publish clinical trials. We extracted trial characteristics and whether the trial data were shared publicly in any form. We conducted a sensitivity analysis of only trials with public funding sources.

RESULTS

In the included journals, zero (0/394, 0.0%) RCTs shared their data publicly. The large majority (315/394, 79.9%) of included trials received funding from public sources. Eight journals had data sharing policies and published 299 of the included trials (75.9%).

CONCLUSION

Our finding has significant implications for the addiction research community. These implications are broad, ranging from possibly slowed scientific advancement to noncompliance with obligations to the public whose tax dollars funded a large majority of the included RCTs. To improve the rates of data sharing, we recommend studying incentive systems, while simultaneously working to cultivate a data sharing system that emphasizes scientific, rather than author, accuracy.

Assessment of transparent and reproducible research practices in the psychiatry literature

BACKGROUND

Reproducibility is a cornerstone of scientific advancement; however, many published works may lack the core components needed for study reproducibility.

AIMS

In this study, we evaluate the state of transparency and reproducibility in the field of psychiatry using specific indicators as proxies for these practices.

METHODS

An increasing number of publications have investigated indicators of reproducibility, including research by Harwicke et al, from which we based the methodology for our observational, cross-sectional study. From a random 5-year sample of 300 publications in PubMed-indexed psychiatry journals, two researchers extracted data in a duplicate, blinded fashion using a piloted Google form. The publications were examined for indicators of reproducibility and transparency, which included availability of: materials, data, protocol, analysis script, open-access, conflict of interest, funding and online preregistration.

RESULTS

This study ultimately evaluated 296 randomly-selected publications with a 3.20 median impact factor. Only 107 were available online. Most primary authors originated from USA, UK and the Netherlands. The top three publication types were cohort studies, surveys and clinical trials. Regarding indicators of reproducibility, 17 publications gave access to necessary materials, four provided in-depth protocol and one contained raw data required to reproduce the outcomes. One publication offered its analysis script on request; four provided a protocol availability statement. Only 107 publications were publicly available: 13 were registered in online repositories and four, ten and eight publications included their hypothesis, methods and analysis, respectively. Conflict of interest was addressed by 177 and reported by 31 publications. Of 185 publications with a funding statement, 153 publications were funded and 32 were unfunded.

CONCLUSIONS

Currently, Psychiatry research has significant potential to improve adherence to reproducibility and transparency practices. Thus, this study presents a reference point for the state of reproducibility and transparency in Psychiatry literature. Future assessments are recommended to evaluate and encourage progress.

Do oncology researchers adhere to reproducible and transparent principles? A cross-sectional survey of published oncology literature

OBJECTIVES

As much as 50%-90% of research is estimated to be irreproducible, costing upwards of $28 billion in USA alone. Reproducible research practices are essential to improving the reproducibility and transparency of biomedical research, such as including preregistering studies, publishing a protocol, making research data and metadata publicly available, and publishing in open access journals. Here we report an investigation of key reproducible or transparent research practices in the published oncology literature.

DESIGN

We performed a cross-sectional analysis of a random sample of 300 oncology publications published from 2014 to 2018. We extracted key reproducibility and transparency characteristics in a duplicative fashion by blinded investigators using a pilot tested Google Form.

PRIMARY OUTCOME MEASURES

The primary outcome of this investigation is the frequency of key reproducible or transparent research practices followed in published biomedical and clinical oncology literature.

RESULTS

Of the 300 publications randomly sampled, 296 were analysed for reproducibility characteristics. Of these 296 publications, 194 contained empirical data that could be analysed for reproducible and transparent research practices. Raw data were available for nine studies (4.6%). Five publications (2.6%) provided a protocol. Despite our sample including 15 clinical trials and 7 systematic reviews/meta-analyses, only 7 included a preregistration statement. Less than 25% (65/194) of publications provided an author conflict of interest statement.

CONCLUSION

We found that key reproducibility and transparency characteristics were absent from a random sample of published oncology publications. We recommend required preregistration for all eligible trials and systematic reviews, published protocols for all manuscripts, and deposition of raw data and metadata in public repositories.

Using the CONSORT statement to evaluate the completeness of reporting of addiction randomised trials: a cross-sectional review

OBJECTIVES

Evaluate the completeness of reporting of addiction randomised controlled trials (RCTs) using the Consolidated Standards of Reporting Trials (CONSORT) statement.

SETTING

Not applicable.

PARTICIPANTS

RCTs identified using a PubMed search of 15 addiction journals and a 5-year cross-section.

OUTCOME MEASURES

Completeness of reporting.

RESULTS

Our analysis of 394 addiction RCTs found that the mean number of CONSORT items reported was 19.2 (SD 5.2), out of a possible 31. Twelve items were reported in <50% of RCTs; similarly, 12 items were reported in >75% of RCTs. Journal endorsement of CONSORT was found to improve the number of CONSORT items reported.

CONCLUSIONS

Poor reporting quality may prohibit readers from critically appraising the methodological quality of addiction trials. We recommend journal endorsement of CONSORT since our study and those previous have shown that CONSORT endorsement improves the quality of reporting.

Comparisons between protocols and publications of case-control studies: analysis of potential causes of non-reproducibility and recommendations for enhancing the quality of personalization in healthcare

Background

Selective reporting of results in published case-control studies has been widely suspected, but little comprehensive information on selective reporting is available with regard to case-control studies. We aimed to evaluate the concordance of findings between publications and the protocols of case-control studies and to assess the level of selective reporting of results in case-control studies.

Methods

The databases of Embase, Medline, Scopus, and Web of Science were searched to identify case-control study protocols published between January 1, 1990 and December 31, 2017. The numbers and characteristics of predefined exposures (or factors) were extracted from the protocols. The reported and unreported factors were both collected from the published studies and protocols. The frequency of selective reporting of results were estimated by identifying the discrepancies of factors between the protocols and the published studies. Study sample size and the extent of selective reporting of factors were measured by a Spearman correlation analysis.

Results

Fourteen protocols with 24 published studies and 159 factors were identified, of which eight protocols (57.1%) had discrepancies between the publications and protocols. The prevalence of incomplete reporting in published case-control studies was 42.9% (6/14), with participant characteristics, anthropometric and laboratory measurement variables more likely to be unreported. A total of 16,835 cases and 56,049 controls were recruited in the 14 protocols of case-control studies (sample size ranges from 428 to 52,596 per study). Sample size had no statistical significance with selective reporting of results (P > 0.05).

Conclusion

The study protocols should be publicly available prior to the completion of case-control studies so that the potential bias can be assessed by the readers. Our findings highlight the need for investigators, peer reviewers, and readers to exercise increased awareness and scrutiny due to the undesirable practice of selective reporting of results in medical sciences causing the loss of potentially important information, thus impacting quality of personalized attitude in healthcare in the context of the predictive, preventive, and personalized medicine.

Selective outcome reporting in randomized clinical trials of dental implants

AIM

Selective outcome reporting (SOR) is a type of bias that occurs when the primary outcome of a trial protocol is changed or omitted in the paper. The purpose of this study was to evaluate the prevalence of SOR in publications of randomized clinical trials (RCT) concerning dental implants.

MATERIALS AND METHODS

Two reviewers independently screened protocols registered at ClinicalTrials.gov until February/2019. If the protocol met the eligibility criteria, the reviewers tried to identify the corresponding publication. Data extraction was carried out by the same reviewers. Primary and secondary outcomes were recorded for each trial and compared to outcomes previously described in protocols.

RESULTS

A total of 49 protocols were included. SOR was identified in 27 (55.1%) trials. The major discrepancies were as follows: protocol-defined primary outcome omitted in the publication (n = 6, 12.2%), new primary outcome introduced (n = 8, 16.3%), discrepancy in the primary outcome time frame (n = 17, 34.7%) and new secondary outcome introduced (n = 31, 63.3%). SOR was significantly associated with industry funding (p = 0.04) and timing of registration (p = 0.04).

CONCLUSIONS

Our results indicate a high rate of SOR in dental implants clinical trials. Use of registry data during the peer-review process may help decreasing SOR.

Peer reviewed evaluation of registered end-points of randomised trials (the PRE-REPORT study): protocol for a stepped-wedge, cluster-randomised trial

INTRODUCTION

Clinical trials are critical to the advancement of medical knowledge. However, the reliability of trial conclusions depends in part on consistency between pre-planned and reported study outcomes. Unfortunately, selective outcome reporting, in which outcomes reported in published manuscripts differ from pre-specified study outcomes, is common. Trial registries such as ClinicalTrials.gov have the potential to help identify and stop selective outcome reporting during peer review by allowing peer reviewers to compare outcomes between registry entries and submitted manuscripts. However, the persistently high rate of selective outcome reporting among published clinical trials indicates that the current peer review process at most journals does not effectively address the problem of selective outcome reporting.

METHODS AND ANALYSIS

PRE-REPORT is a stepped-wedge cluster-randomised trial that will test whether providing peer reviewers with a summary of registered, pre-specified primary trial outcomes decreases inconsistencies between prospectively registered and published primary outcomes. Peer reviewed manuscripts describing clinical trial results will be included. Eligible manuscripts submitted to each participating journal during the study period will comprise each cluster. After an initial control phase, journals will transition to the intervention phase in random order, after which peer reviewers will be emailed registry information consisting of the date of registration and any prospectively defined primary outcomes. Blinded outcome assessors will compare registered and published primary outcomes for all included trials. The primary PRE-REPORT outcome is the presence of a published primary outcome that is consistent with a prospectively defined primary outcome in the study's trial registry. The primary outcome will be analysed using a mixed effect logistical regression model to compare results between the intervention and control phases.

ETHICS AND DISSEMINATION

The Cooper Health System Institutional Review Board determined that this study does not meet criteria for human subject research. Findings will be published in peer-reviewed journals.

TRIAL REGISTRATION NUMBER

ISRCTN41225307; Pre-results.

Registration Practices of Randomized Clinical Trials in Rhinosinusitis: A Cross-sectional Review

Importance

Randomized clinical trials (RCTs) play an important role in the development of clinical practice guidelines and in clinical decision making. Little is known about the registration practices of RCTs in the diagnosis and treatment of rhinosinusitis.

Objectives

The primary outcome was the frequency of reported RCT registry numbers by authors of rhinosinusitis RCTs. A secondary outcome was the rates of selective reporting bias in RCTs that were prospectively registered. A tertiary outcome end point was the frequency of publication of RCTs registered on ClinicalTrials.gov.

Evidence Review

Our sample was derived from a PubMed (MEDLINE) search performed on October 31, 2017, using the keywords "sinusitis OR rhinosinusitis OR rhinitis," and filtering by date (January 1, 2015, through October 31, 2017) and study type (RCT). Studies that were considered an RCT were included for the primary outcome analysis. All RCTs that were registered prior to or during patient enrollment were included for secondary outcome analysis. For the tertiary outcome, a search was performed on ClinicalTrials.gov on September 4, 2018, using the keywords "sinusitis OR rhinosinusitis," and filtering by date (January 1, 2013, through October 31, 2015). All analysis took place between October 29, 2018, and October 31, 2018.

Findings

A total of 179 RCTs were analyzed for our primary outcome: 94 (52.5%) included a registration number in their publication, and 70 (39.1%) were included for secondary outcome analysis of rates of selective reporting bias. Of these 70 RCTs, 22 (31%) were found to have at least 1 major discrepancy between trial registration and publication. For the tertiary outcome, 52 completed clinical trials were identified on ClinicalTrials.gov, of which 21 (40%) had listed publication in the registry.

Conclusions and Relevance

We found that published rhinosinusitis RCTs frequently do not include a trial registration number and that registered RCTs frequently are not published. Furthermore, RCTs that are registered often display selective reporting of their outcomes and frequently favor positive results. We recommend strict adherence to RCT registration policies and the enforcement of accurate reporting to help strengthen the evidence behind clinical decision making.

Recent Status and Methodological Quality of Return-to-Work Rates of Cancer Patients Reported in Japan: A Systematic Review

Cancer patients’ return-to-work rates in Japan and their methodological quality have been little reported. We conducted a systematic review to explore the recent return-to-work rates and to assess the methodological quality of the existing literature. We selected 13 papers (2 in English and 11 in Japanese) published between 2005 and 2017. The return-to-work rates ranged from 53.8% to 95.2%. Of the selected papers, 12 papers employed a cross-sectional design, possessing high risk of selection bias due to participant selection. A total of 8 papers did not fully report the subjects’ sex, age, employment status at cancer diagnosis, cancer site, stage, and treatment, suggesting high risk of selection bias due to confounding variables. High or unclear risk of attrition bias due to incomplete outcome data was detected in 12 papers in which data on return to work were not collected from all participants. High risk of reporting bias due to selective outcome reporting was pointed out in 6 studies in which the subjects’ employment status at return to work or the duration between cancer diagnosis and assessment of return to work was unclear. Future studies must reduce the risk of selection, attrition, and reporting biases for specifying accurate return-to-work rates.

Adherence to reporting guidelines and clinical trial registration policies in oncology journals: a cross-sectional review

Reporting guidelines (RG) aim to improve research transparency and ensure high-quality study reporting. Similarly, clinical trial registration policies aim to reduce bias in results reporting by ensuring prospective registration of all trial outcomes. Failure to adhere to quality standards documented in RGs may result in low-quality and irreproducible research. Herein, we investigate the adherence to common RGs and trial registration policies in 21 oncology journals. We surveyed the Instructions for Authors page for each of the included oncology journals for adherence to common reporting guidelines and trial registration policies. We corresponded with editors to determine accepted study types and cross-referenced this information with a journal's RGs and trial registration policies to calculate the per cent of journals that adhere to a specific guideline or policy. 76.2% (16/21) of oncology journals surveyed adhere to Consolidated Standards of Reporting Trials guidelines for clinical trials while only 33.3% (7/21) adhere to Strengthening the Reporting of Observational Studies in Epidemiology for observational studies. Similarly, 76.2% (16/21) of oncology journals adhere to clinical trial registration policies. We further demonstrate that journal adherence to RGs positively affects author reporting, despite adherence to trial registration policies showing no such benefit. Our results show that oncology journals adhere to RGs and trial registration policies at a higher rate than other specialties, but nonetheless show room for improvement. We conclude that oncology journal adherence to RGs and trial registration policies is encouraging, but nonetheless suboptimal. We recommend the adoption of RGs and trial registration policies by all oncology journals.

Does access to clinical study reports from the European Medicines Agency reduce reporting biases? A systematic review and meta-analysis of randomized controlled trials on the effect of erythropoiesis-stimulating agents in cancer patients

Since 2010, the European Medicines Agency (EMA) has provided access to clinical study reports (CSRs). We requested CSRs for randomized controlled trials (RCTs) of erythropoiesis-stimulating agents (ESAs) in cancer patients from EMA and identified RCT publications with literature searches. We assessed CSR availability and completeness, the impact of unreported and unpublished data obtained from CSRs on the effects of ESAs on quality of life (QoL) of cancer patients, and discrepancies between data reported in the public domain and in CSRs. We used random-effects meta-analyses to evaluate the effect of ESAs on QoL measured with Functional Assessment of Cancer Therapy-Anemia (FACT-An), FACT-Fatigue (FACT-F) and FACT-Anemia Total (FACT-An Total) stratified by data source and the impact of discrepancies on QoL, mortality, adverse events, and clinical effectiveness outcomes. We identified 94 eligible RCTs; CSRs or other study documentation were available for 17 (18%) RCTs at EMA. Median report length was 1,825 pages (range 72-14,569). Of 180 outcomes of interest reported in the EMA documentation, 127 (71%) were publicly available. For 80 of those (63%) we noted discrepancies, but these had little impact on the pooled effect estimates. Of 27 QoL outcomes reported in the CSRs, 17 (63%) were unpublished. Including six unpublished comparisons (pooled mean difference [MD] 0.20; 95% confidence interval [CI] -1.93, 2.33) reduced the pooled effect of ESAs for FACT-An from MD 5.51 (95% CI 4.20, 6.82) in published data to MD 3.21 (95% CI 1.38, 5.03), which is below a clinically important difference (defined as MD ≥4). Effects were similar for FACT-F and FACT-An Total. Access to CSRs from EMA reduced reporting biases for QoL outcomes. However, EMA received documentation for a fraction of all RCTs on effects of ESAs in cancer patients. Additional efforts by other agencies and institutions are needed to make CSRs universally available for all RCTs.

The Weak Spots in Contemporary Science (and How to Fix Them)

In this review, the author discusses several of the weak spots in contemporary science, including scientific misconduct, the problems of post hoc hypothesizing (HARKing), outcome switching, theoretical bloopers in formulating research questions and hypotheses, selective reading of the literature, selective citing of previous results, improper blinding and other design failures, p-hacking or researchers' tendency to analyze data in many different ways to find positive (typically significant) results, errors and biases in the reporting of results, and publication bias. The author presents some empirical results highlighting problems that lower the trustworthiness of reported results in scientific literatures, including that of animal welfare studies. Some of the underlying causes of these biases are discussed based on the notion that researchers are only human and hence are not immune to confirmation bias, hindsight bias, and minor ethical transgressions. The author discusses solutions in the form of enhanced transparency, sharing of data and materials, (post-publication) peer review, pre-registration, registered reports, improved training, reporting guidelines, replication, dealing with publication bias, alternative inferential techniques, power, and other statistical tools.

Inconsistent selection of outcomes and measurement devices found in shoulder arthroplasty research: An analysis of studies on ClinicalTrials.gov

Introduction

Recent evidence suggests a lack of standardization of shoulder arthroplasty outcomes. This issue is a limiting factor in systematic reviews. Core outcome set (COS) methodology could address this problem by delineating a minimum set of outcomes for measurement in all shoulder arthroplasty trials.

Methods

A ClinicalTrials.gov search yielded 114 results. Eligible trials were coded on the following characteristics: study status, study type, arthroplasty type, sample size, measured outcomes, outcome measurement device, specific metric of measurement, method of aggregation, outcome classification, and adverse events.

Results

Sixty-six trials underwent data abstraction and data synthesis. Following abstraction, 383 shoulder arthroplasty outcomes were organized into 11 outcome domains. The most commonly reported outcomes were shoulder outcome score (n = 58), pain (n = 33), and quality of life (n = 15). The most common measurement devices were the Constant-Murley Shoulder Outcome Score (n = 38) and American Shoulder and Elbow Surgeons Shoulder Score (n = 33). Temporal patterns of outcome use was also found.

Conclusion

Our study suggests the need for greater standardization of outcomes and instruments. The lack of consistency across trials indicates that developing a core outcome set for shoulder arthroplasty trials would be worthwhile. Such standardization would allow for more effective comparison across studies in systematic reviews, while at the same time consider important outcomes that may be underrepresented otherwise. This review of outcomes provides an evidence-based foundation for the development of a COS for shoulder arthroplasty.