Randomized trials published in higher vs. lower impact journals differ in design, conduct, and analysis

Quality of reporting inflammatory bowel disease randomised controlled trials: a systematic review

OBJECTIVE

Our objective was to perform a systemic evaluation of the risk of bias in randomised controlled trial (RCT) reports published on inflammatory bowel disease (IBD).

DESIGN

We assessed the risk of bias using the Cochrane tool, as indicators of poor methodology or subsequently poor reporting. We systematically selected, with dual independent judgements, all studies published on IBD with no time limits and assessed the methodological quality of included studies again using independent dual ratings.

RESULTS

563 full texts were included after selection and review. No abstract publications were free of any source of bias. Full-text publications still fared badly, as only 103 full-text papers exhibited a low risk of bias in all reporting domains when excluding blinding. RCTs published in journals with higher impact factor (IF) were associated with an overall reduced rate of being at high risk. However, only 6% of full RCT publications in journals with an IF greater than 10, published in the past 5 years, were free of bias.The trend over time is towards improved reporting in all areas. Trials published by larger author teams, in full-text form and by industry and public sponsorship were positively correlated with a lower risk of bias. Only allocation concealment showed a statistically significant improvement with time (p=0.037).

CONCLUSION

These findings are consistent with those of other specialties in the literature. While this unclear risk of bias may represent poor reporting of methods instead of poor methodological quality, it leaves readers and future secondary researchers with significant questions regarding such key issues.

Risk of bias in exercise science: A systematic review of 340 studies

Risk of bias can contribute to irreproducible science and mislead decision making. Analyses of smaller subsections of the exercise science literature suggest many exercise science studies have unclear or high risk of bias. The current review (osf.io/jznv8) assesses whether this unclear or high risk of bias is more widespread in the exercise science literature and whether this bias has decreased since the publication of the 1996 Consolidated Standards of Reporting Trials (CONSORT) guidelines. We report significant reductions in selection, performance, detection, and reporting biases in 2020 compared with 1995 in the 340 of 5,451 studies assessed using the Cochrane Risk of Bias tool. Despite these improvements, most 2020 studies still had unclear or high risks of bias. These results underscore the need for methodological vigilance, adherence to reporting standards, and education on experimental bias. Factors contributing to these improvements, such advancements in education and journal requirements, remain uncertain.

Participant retention in paediatric randomised controlled trials published in six major journals 2015-2019: systematic review and meta-analysis

BACKGROUND

The factors which influence participant retention in paediatric randomised controlled trials are under-researched. Retention may be more challenging due to child developmental stages, involving additional participants, and proxy-reporting of outcomes. This systematic review and meta-analysis explores the factors which may influence retention in paediatric trials.

METHODS

Using the MEDLINE database, paediatric randomised controlled trials published between 2015 and 2019 were identified from six general and specialist high-impact factor medical journals. The review outcome was participant retention for each reviewed trial's primary outcome. Context (e.g. population, disease) and design (e.g. length of trial) factors were extracted. Retention was examined for each context and design factor in turn, with evidence for an association being determined by a univariate random-effects meta-regression analysis.

RESULTS

Ninety-four trials were included, and the median total retention was 0.92 (inter-quartile range 0.83 to 0.98). Higher estimates of retention were seen for trials with five or more follow-up assessments before the primary outcome, those less than 6 months between randomisation and primary outcome, and those that used an inactive data collection method. Trials involving children aged 11 and over had the higher estimated retention compared with those involving younger children. Those trials which did not involve other participants also had higher retention, than those where they were involved. There was also evidence that a trial which used an active or placebo control treatment had higher estimated retention, than treatment-as-usual. Retention increased if at least one engagement method was used. Unlike reviews of trials including all ages of participants, we did not find any association between retention and the number of treatment groups, size of trial, or type of treatment.

CONCLUSIONS

Published paediatric RCTs rarely report the use of specific modifiable factors that improve retention. Including multiple, regular follow-ups with participants before the primary outcome may reduce attrition. Retention may be highest when the primary outcome is collected up to 6 months after a participant is recruited. Our findings suggest that qualitative research into improving retention when trials involve multiple participants such as young people, and their caregivers or teachers would be worthwhile. Those designing paediatric trials also need to consider the use of appropriate engagement methods. RESEARCH ON RESEARCH (ROR) REGISTRY: https://ror-hub.org/study/2561.

Study results from journals with a higher impact factor are closer to "truth": a meta-epidemiological study

BACKGROUND

Scientists, physicians, and the general public legitimately expect scholarly publications to give true answers to study questions raised. We investigated whether findings from studies published in journals with higher Journal Impact Factors (JIFs) are closer to truth than findings from studies in less-cited journals via a meta-epidemiological approach.

METHODS

We screened intervention reviews from the Cochrane Database of Systematic Reviews (CDSR) and sought well-appraised meta-analyses. We used the individual RCT study estimates' relative deviation from the pooled effect estimate as a proxy for the deviation of the study results from the truth. The effect of the JIF on the relative deviation was estimated with linear regression and with local polynomial regression, both with adjustment for the relative size of studies. Several sensitivity analyses for various sub-group analyses and for alternative impact metrics were conducted.

RESULTS

In 2459 results from 446 meta-analyses, results with a higher JIF were on average closer to "truth" than the results with a lower JIF. The relative deviation decreased on average by -0.023 per JIF (95% CI -0.32 to -0.21). A decrease was consistently found in all sensitivity analyses.

CONCLUSIONS

Our results indicate that study results published in higher-impact journals are on average closer to truth. However, the JIF is only one weak and impractical indicator among many that determine a studies' accuracy.

How many US obstetrical trials reach publication? A cross-sectional analysis of ClinicalTrials.gov and PubMed from 2007 to 2019

BACKGROUND

Obstetrical clinical trials are the foundation of evidence-based medicine during pregnancy. As more obstetrical trials are conducted, understanding the publication characteristics of these trials is of utmost importance to advance obstetrical health.

OBJECTIVE

This study aimed to characterize the frequency of publication and trial characteristics associated with publication among obstetrical clinical trials in the United States. We additionally sought to examine time from trial completion to publication.

STUDY DESIGN

This was a cross-sectional analysis of completed obstetrical trials with an intervention design and at least 1 site in the United States registered to ClinicalTrials.gov from 2007 to 2019. Trial characteristics were cross-referenced with PubMed to determine publication status up to 2021 using the National Clinical Trial identification number. Bivariable analyses were conducted to determine trial characteristics associated with publication. Multivariable logistic regression models controlling for prespecified covariates were generated to estimate the relationship between funding, primary purpose, and therapeutic foci with publication. Additional exploratory analyses of other trial characteristics were conducted. Time to publication was analyzed using Kaplan-Meier curves and Cox regression models.

RESULTS

Of the 1879 obstetrical trials with registered completion, a total of 575 (30.6%) had at least 1 site in the United States, were completed before October 1, 2019, and were included in this analysis. Between October 2007 and October 2019, fewer than two-thirds (N=348, 60.5%) of trials reached publication. Annual rates of publication ranged from 46.4% in 2018 to 70.0% in 2007. No difference was observed in publication by funding, primary purpose, or therapeutic foci (all P>.05). Trials with characteristics indicating high trial quality-including randomized allocation scheme, ≥50 participants enrolled, ≥2 sites, and presence of a data safety monitoring committee-had increased odds of publication compared with those without such characteristics (all P<.05). For example, studies with randomized allocation of intervention had 2-fold greater odds of publication than nonrandomized studies (adjusted odds ratio, 2.09; 95% confidence interval, 1.30-3.37). Studies with ≥150 participants had nearly 8-fold odds of publication (adjusted odds ratio, 7.90; 95% confidence interval, 3.78-17.49) relative to studies with <50 participants. Temporal analysis demonstrated variability in time to publication among obstetrical trials, with a median time of 20.1 months after trial completion, and with most trials that reached publication having been published by 40 months. No difference was observed in time to publication by funding, primary purpose, or therapeutic foci (all P>.05).

CONCLUSION

Publication of obstetrical trials remains suboptimal, with significant differences observed between trials with indicators of high quality and those without. Most trials that reach publication are published within 2 years of registered completion on ClinicalTrials.gov.

Specification of interventions and selection of controls in randomized controlled trials of acupuncture: a cross-sectional survey

OBJECTIVE

Specification of interventions and selection of controls are two methodological determinants for a successful acupuncture trial. However, current practice with respect to these two determinants is not fully understood. Thus, we conducted a cross-sectional survey to examine the specification of interventions and selection of controls among published randomized controlled trials (RCTs) of acupuncture.

STUDY DESIGN AND SETTING

We searched PubMed for acupuncture RCTs published in core clinical journals and complementary and alternative medicine (CAM) journals from January 2010 to December 2019 (10 years) and included RCTs that assessed treatment effects of acupuncture versus any type of control. We used network meta-analyses to explore whether there were differential treatment effects in patients with chronic pain when using sham acupuncture as a control versus using waiting list or no treatment.

RESULTS

Most of the 319 eligible RCTs specified well the style of acupuncture (86.8%), traditional acupuncture point locations (96.2%), type of needle stimulation (90.3%) and needle retention time (85.6%). However, other acupuncture details were less-frequently specified, including response sought (65.5%), needle manipulation (50.5%), number of needle insertions (21.9%), angle and direction of insertion (31.3%), patient posture (32.3%) and co-interventions (22.9%). Sham acupuncture (41.4%) was the most frequently used control, followed by waiting list or no treatment (32.9%). There was no differential treatment effect when using sham acupuncture versus waiting list/no treatment as a control (standardized mean difference = -0.15, 95% confidence interval: -0.91 to 0.62).

CONCLUSION

Over a decade of research practice, important gaps have remained in the specification of acupuncture interventions, including response sought, needle manipulation, and co-interventions. While sham acupuncture has been widely used, waiting list or no treatment may also be considered as an appropriate control.

Impact of time-varying exposure on estimated effects in observational studies using routinely collected data: protocol for a cross-sectional study

INTRODUCTION

Time-varying exposure is an important issue that should be addressed in longitudinal observational studies using routinely collected data (RCD) for drug treatment effects. How well investigators designed, analysed and reported time-varying exposure, and to what extent the divergence that can be observed between different methods used for handling time-varying exposure in these studies remains uncertain. We will conduct a cross-sectional study to comprehensively address this question.

METHODS AND ANALYSIS

We have developed a comprehensive search strategy to identify all studies exploring drug treatment effects including both effectiveness and safety that used RCD and were published in core journals between 2018 and 2020. We will collect information regarding general study characteristics, data source profile, methods for handling time-varying exposure, results and the interpretation of findings from each eligibility. Paired reviewers will screen and extract data, resolving disagreements through discussion. We will describe the characteristics of included studies, and summarise the method used for handling time-varying exposure in primary analysis and sensitivity analysis. We will also compare the divergence between different approaches for handling time-varying exposure using ratio of risk ratios.

ETHICS AND DISSEMINATION

No ethical approval is required because the data we will use do not include individual patient data. Findings will be disseminated through peer-reviewed publications.

Improving the Science in Plastic Surgery

SUMMARY

In 1906, George Bernard Shaw criticized the medical profession for its lack of science and compassion. Since then, advances in both medical and surgical subspecialties have improved quality of patient care. Unfortunately, the reporting of these advances is variable and is frequently biased. Such limitations lead to false claims, wasted research dollars, and inability to synthesize and apply evidence to practice. It was hoped that the introduction of evidence-based medicine would improve the quality of health care and decrease health dollar waste. For this to occur, however, credible "best evidence"-one of the components of evidence-based medicine-is required. This article provides a framework for credible research evidence in plastic surgery, as follows: (1) stating the clinical research question, (2) selecting the proper study design, (3) measuring critical (important) outcomes, (4) using the correct scale(s) to measure the outcomes, (5) including economic evaluations with clinical (effectiveness) studies, and (6) reporting a study's results using the Enhancing the Quality and Transparency of Health Research, or EQUATOR, guidelines. Surgeon investigators are encouraged to continue improving the science in plastic surgery by applying the framework outlined in this article. Improving surgical clinical research should decrease resource waste and provide patients with improved evidence-based care.

Assessment of Nonfatal Myocardial Infarction as a Surrogate for All-Cause and Cardiovascular Mortality in Treatment or Prevention of Coronary Artery Disease: A Meta-analysis of Randomized Clinical Trials

IMPORTANCE

Although nonfatal myocardial infarction (MI) is associated with an increased risk of mortality, evidence validating nonfatal MI as a surrogate end point for all-cause or cardiovascular (CV) mortality is lacking.

OBJECTIVE

To examine whether nonfatal MI may be a surrogate for all-cause or CV mortality in patients with or at risk for coronary artery disease.

DATA SOURCES

In this meta-analysis, PubMed was searched from inception until December 31, 2020, for randomized clinical trials of interventions to treat or prevent coronary artery disease reporting mortality and nonfatal MI published in 3 leading journals.

STUDY SELECTION

Randomized clinical trials including at least 1000 patients with 24 months of follow-up.

DATA EXTRACTION AND SYNTHESIS

Trial-level correlations between nonfatal MI and all-cause or CV mortality were assessed for surrogacy using the coefficient of determination (R2). The criterion for surrogacy was set at 0.8. Subgroup analyses based on study subject (primary prevention, secondary prevention, mixed primary and secondary prevention, and revascularization), era of trial (before 2000, 2000-2009, and 2010 and after), and follow-up duration (2.0-3.9, 4.0-5.9, and ≥6.0 years) were performed.

MAIN OUTCOMES AND MEASURES

All-cause or CV mortality and nonfatal MI.

RESULTS

A total of 144 articles randomizing 1 211 897 patients met the criteria for inclusion. Nonfatal MI did not meet the threshold for surrogacy for all-cause (R2 = 0.02; 95% CI, 0.00-0.08) or CV (R2 = 0.11; 95% CI, 0.02-0.27) mortality. Nonfatal MI was not a surrogate for all-cause mortality in primary (R2 = 0.01; 95% CI, 0.001-0.26), secondary (R2 = 0.03; 95% CI, 0.00-0.20), mixed primary and secondary prevention (R2 = 0.001; 95% CI, 0.00-0.08), or revascularization trials (R2 = 0.21; 95% CI, 0.002-0.50). For trials enrolling patients before 2000 (R2 = 0.22; 95% CI, 0.08-0.36), between 2000 and 2009 (R2 = 0.02; 95% CI, 0.00-0.17), and from 2010 and after (R2 = 0.01; 95% CI, 0.00-0.09), nonfatal MI was not a surrogate for all-cause mortality. Nonfatal MI was not a surrogate for all-cause mortality in randomized clinical trials with 2.0 to 3.9 (R2 = 0.004; 95% CI, 0.00-0.08), 4.0 to 5.9 (R2 = 0.06; 95% CI, 0.001-0.16), or 6.0 or more years of follow-up (R2 = 0.30; 95% CI, 0.01-0.55).

CONCLUSIONS AND RELEVANCE

The findings of this meta-analysis do not appear to establish nonfatal MI as a surrogate for all-cause or CV mortality in randomized clinical trials of interventions to treat or prevent coronary artery disease.

Number and quality of randomized controlled trials in obstetrics published in the top general medical and obstetrics and gynecology journals

BACKGROUND

There has been an increasing number of randomized controlled trials published in obstetrics and maternal-fetal medicine to reduce biases of treatment effect and to provide insights on the cause-effect of the relationship between treatment and outcomes.

OBJECTIVE

This study aimed to identify obstetrical randomized controlled trials published in top weekly general medical journals and monthly obstetrics and gynecology journals, to assess their quality in reporting and identify factors associated with publication in different journals.

STUDY DESIGN

The 4 weekly medical journals with the highest 2019 impact factor (New England Journal of Medicine, The Lancet, The Journal of the American Medical Association, and British Medical Journal), the top 4 monthly obstetrics and gynecology journals with obstetrics-related research (American Journal of Obstetrics & Gynecology, Ultrasound in Obstetrics & Gynecology, Obstetrics & Gynecology, and the British Journal of Obstetrics and Gynaecology), and the American Journal of Obstetrics & Gynecology Maternal-Fetal Medicine were searched for obstetrical randomized controlled trials in the years 2018 to 2020. The primary outcome was the number of obstetrical randomized controlled trials published in the obstetrics and gynecology journals vs the weekly medical journals and the percentage of trials published, overall and per journal. The secondary outcomes included the proportion of positive vs negative trials overall and per journal and the assessment of the study characteristics of published trials, including quality assessment criteria.

RESULTS

Of the 4024 original research articles published in the 9 journals during the 3-year study period, 1221 (30.3%) were randomized controlled trials, with 137 (11.2%) randomized controlled trials being in obstetrics (46 in 2018, 47 in 2019, and 44 studies in 2020). Furthermore, 33 (24.1%) were published in weekly medical journals, and 104 (75.9%) were published in obstetrics and gynecology journals. The percentage of obstetrical randomized controlled trials published ranged from 1.5% to 9.6% per journal. Overall, 34.3% of obstetrical trials were statistically significant or "positive" for the primary outcome. Notably, 24.8% of the trials were retrospectively registered after the enrollment of the first study patient. Trials published in the 4 weekly medical journals enrolled significantly more patients (1801 vs 180; P<.001), received more often funding from the federal government (78.8% vs 35.6%; P<.001), and were more likely to be multicenter (90.9% vs 42.3%; P<.001), non-United States based (69.7% vs 49.0%; P=.03), and double blinded (45.5% vs 18.3%; P=.003) than trials published in the obstetrics and gynecology journals. There was no difference in study type (noninferiority vs superiority) and trial quality characteristics, including pretrial registration, ethics approval statement, informed consent statement, and adherence to the Consolidated Standards of Reporting Trials guidelines statement between studies published in weekly medical journals and studies published in obstetrics and gynecology journals.

CONCLUSION

Approximately 45 trials in obstetrics are being published every year in the highest impact journals, with one-fourth being in the weekly medical journals and the remainder in the obstetrics and gynecology journals. Only about a third of published obstetrical trials are positive. Trials published in weekly medical journals are larger, more likely to be funded by the government, multicenter, international, and double blinded. Quality metrics are similar between weekly medical journals and obstetrics and gynecology journals.

Eligibility Criteria and Representativeness of Randomized Clinical Trials That Include Infants Born Extremely Premature: A Systematic Review

OBJECTIVE

To assess the eligibility criteria and trial characteristics among contemporary (2010-2019) randomized clinical trials (RCTs) that included infants born extremely preterm (<28 weeks of gestation) and to evaluate whether eligibility criteria result in underrepresentation of high-risk subgroups (eg, infants born at <24 weeks of gestation).

STUDY DESIGN

PubMed and Scopus were searched January 1, 2010, to December 31, 2019, with no language restrictions. RCTs with mean or median gestational ages at birth of <28 weeks of gestation were included. The study followed the PRISMA guidelines; outcomes were registered prospectively. Data extraction was performed independently by multiple observers. Study quality was evaluated using a modified Jadad scale.

RESULTS

Among RCTs (n = 201), 32 552 infants were included. Study participant characteristics, interventions, and outcomes were highly variable. A total of 1603 eligibility criteria were identified; rationales were provided for 18.8% (n = 301) of criteria. Fifty-five RCTs (27.4%) included infants <24 weeks of gestation; 454 (1.4%) infants were identified as <24 weeks of gestation.

CONCLUSIONS

The present study identifies sources of variability across RCTs that included infants born extremely preterm and reinforces the critical need for consistent and transparent policies governing eligibility criteria.

Blinding in randomised clinical trials of psychological interventions: a retrospective study of published trial reports

OBJECTIVES

To study the extent of blinding in randomised clinical trials of psychological interventions and the interpretative considerations if randomised clinical trials are not blinded.

DESIGN

Retrospective study of trial reports published in six high impact factor journals within the field of psychiatry in 2017 and 2018.

SETTING

Trial reports published in World Psychiatry, JAMA Psychiatry, Lancet Psychiatry, American Journal of Psychiatry, British Journal of Psychiatry, or Psychotherapy and Psychosomatics.

MAIN OUTCOME MEASURES

Blinding status of participants, treatment providers, outcome assessors, data managers, the data safety and monitoring committee, statisticians and conclusion makers, if trialists rejected the null hypothesis on the primary outcome measure, and if trialists discussed the potential bias risk from lack of blinding in the published trial report.

RESULTS

63 randomised clinical trials of psychological interventions were identified. None (0%; 95% CI 0% to 5.75%) of the trials reported blinding of all possible key persons. 37 (58.7%; 95% CI 46.42% to 70.04%) trials reported blinding of outcome assessors. Two (3.2%; 95% CI 0.87% to 10.86%) trials reported blinding of participants. Two (3.2%; 95% CI 0.87% to 10.86%) trials reported blinding of data managers. Three (4.8%; 95% CI 1.63% to 13.09%) trials reported blinding of statisticians. None of the trials reported blinding of treatment providers, the data safety and monitoring committee, and conclusion makers. 45 (71.4%; 95% CI 59.30% to 81.10%) trials rejected the null hypothesis on the primary outcome(s). 13 (20.7%; 95% CI 12.48% to 32.17%) trials discussed the potential bias risk from lack of blinding in the published trial report.

CONCLUSIONS

Blinding of key persons involved in randomised clinical trials of psychological interventions is rarely sufficiently documented. The possible interpretative limitations are only rarely considered. There is a need of randomised clinical trials of psychological interventions with documented blinding attempts of all possible key persons.

Methodological review showed that time-to-event outcomes are often inadequately handled in cluster randomized trials

OBJECTIVES

To estimate the prevalence of time-to-event (TTE) outcomes in cluster randomized trials (CRTs) and to examine their statistical management.

STUDY DESIGN AND SETTING

We searched PubMed to identify primary reports of CRTs published in six major general medical journals (2013-2018). Nature of outcomes and, for TTE outcomes, statistical methods for sample size, analysis, and measures of intracluster correlation were extracted.

RESULTS

A TTE analysis was used in 17% of the CRTs (32/184) either as a primary or secondary outcome analysis, or in a sensitivity analysis. Among the five CRTs with a TTE primary outcome, two accounted for both intracluster correlation and the TTE nature of the outcome in sample size calculation; one reported a measure of intracluster correlation in the analysis. Among the 32 CRTs with a least one TTE analysis, 44% (14/32) accounted for clustering in all TTE analyses. We identified 12 additional CRTs in which there was at least one outcome not analyzed as TTE for which a TTE analysis might have been preferred.

CONCLUSION

TTE outcomes are not uncommon in CRTs but appropriate statistical methods are infrequently used. Our results suggest that further methodological development and explicit recommendations for TTE outcomes in CRTs are needed.

A tutorial on methodological studies: the what, when, how and why

BACKGROUND

Methodological studies - studies that evaluate the design, analysis or reporting of other research-related reports - play an important role in health research. They help to highlight issues in the conduct of research with the aim of improving health research methodology, and ultimately reducing research waste.

MAIN BODY

We provide an overview of some of the key aspects of methodological studies such as what they are, and when, how and why they are done. We adopt a "frequently asked questions" format to facilitate reading this paper and provide multiple examples to help guide researchers interested in conducting methodological studies. Some of the topics addressed include: is it necessary to publish a study protocol? How to select relevant research reports and databases for a methodological study? What approaches to data extraction and statistical analysis should be considered when conducting a methodological study? What are potential threats to validity and is there a way to appraise the quality of methodological studies?

CONCLUSION

Appropriate reflection and application of basic principles of epidemiology and biostatistics are required in the design and analysis of methodological studies. This paper provides an introduction for further discussion about the conduct of methodological studies.

Characteristics of Drug Intervention Clinical Trials and Scientific Impact of the Trial Outcome: A Bibliometric Analysis Using the Relative Citation Ratio in Non-small Cell Lung Cancer from 2007 to 2016

BACKGROUND

Although a large number of clinical trials have been conducted, the types of clinical trials that are scientifically influential, frequently utilized by society, and contribute to the progress of evidence-based medicine (EBM) have not been studied. Thus, we aimed to investigate the relationship between the characteristics of clinical trials and the scientific impact of the outcome in non-small cell lung cancer (NSCLC) by performing a bibliometric analysis using relative citation ratio (RCR), a newly developed bibliometric index by the National Institutes of Health (NIH).

METHODS

Primary publications of drug intervention clinical trials for NSCLC between 2007 and 2016 were included in the study. The characteristics of clinical trials were compared among four RCR categories with 50 trials in each [LOW50, 50 NIH percentile (50NIH%ile), 95 NIH percentile (95NIH%ile), and TOP50], totaling to 200 trials.

RESULTS

Median RCRs of LOW50, 50NIH%ile, 95NIH%ile, and TOP50 were 0.03, 1.00, 5.76, and 26.89, respectively. Publications of Phase 3, randomized, blinded, for-profit-company supported/sponsored, multi-center trials, and trials with a larger number of subjects were shown to have a higher scientific impact. Publications of clinical trials of newly developed molecular target drugs, including epidermal growth factor receptor-tyrosine kinase inhibitors, anaplastic lymphoma kinase inhibitors, and immune checkpoint inhibitors demonstrated a higher scientific impact than those of traditional chemotherapies.

CONCLUSION

Clinical trials designed to have a high evidence level would improve the scientific impact of the outcome, and novel interventions would be another factor to improve the clinical trials' influence.

A systematic survey showed important limitations in the methods for assessing drug safety among systematic reviews

OBJECTIVES

This study aimed to examine the design, conduct, and analysis of systematic reviews assessing drug safety through a cross-sectional survey.

STUDY DESIGN AND SETTING

We searched PubMed to identity systematic reviews published in the Cochrane Database of Systematic Reviews and Core Clinical Journals indexed in 2015 and randomly sampled systematic reviews assessing drug effects at a 1:1 ratio of Cochrane and non-Cochrane reviews. Teams of two investigators independently conducted study screening and collected data, using prespecified, standardized questionnaires. In addition to general information, we collected details about the planning and analyses of safety outcomes.

RESULTS

We included 120 systematic reviews, including 60 Cochrane and 60 non-Cochrane reviews. Most reviews searched PubMed/MEDLINE (n = 117, 97.5%), EMBASE (n = 105, 87.5%), and Cochrane CENTRAL (n = 110, 91.7%) and conducted independent and duplicate study selection (n = 98, 81.7%), risk of bias assessment (n = 105, 87.5%), and data collection (n = 105, 87.5%). Only nine (7.5%) reviews clearly defined safety outcomes, and seven (5.8%) defined a primary safety outcome; none stated whether the primary safety outcome was predefined. Among the 80 reviews that pooled the primary dichotomous safety data across studies, less than half (41%, n = 33) conducted subgroup analysis to explore for sources of heterogeneity or reported a GRADE assessment for the overall quality of evidence. Cochrane reviews were more likely to provide a study protocol (100% vs. 23.3%; P < 0.001), involve methodologists (53.3% vs. 20.0%; P < 0.001), and report a GRADE assessment for the primary safety outcome (70.6% vs. 19.6%; P < 0.001).

CONCLUSION

Our findings highlighted areas for improved planning and analysis in the assessment of drug safety among systematic reviews. Cochrane reviews were superior to non-Cochrane reviews; however, most reviews did not prespecify their safety outcomes or methods for analysis, explore sources of heterogeneity among pooled effects, or assess the overall quality of evidence with the GRADE approach.

Design, Conduct, and Analysis of Surgical Randomized Controlled Trials: A Cross-sectional Survey

BACKGROUND

Randomized controlled trial (RCT) testing surgical intervention faced challenges due to complexities of surgery and made it more difficult for surgeons and methodologists than pharmaceutical providers to build a well-design, conduct RCT.

OBJECTIVE

We conducted a cross-sectional survey to address the methodological challenges of RCTs on surgical intervention and offer potential solutions.

METHODS

We searched PubMed in order to summarize 2-arm parallel randomized trials for surgical interventions published in 2013. The information regarding general characteristics, general methodological and special surgical characteristics related to surgical trials comparing alternative procedures was gathered.

RESULTS

Some 200 surgical trials were identified. The extent to which these trials in design, conduct and analysis differed substantially across items. The general information about sample size calculation (77.0%), lost to follow-up (71.5%), trial registration (55.5%), protocols of trials (56.0%), implementation of randomization (59.5%), concealment of randomization (56.0%); reporting of primary outcome as P value (67.0%). Surgery special information revealed that only 21.0% of trials considered surgeons' preference, approximately 12% to 50% of them controlled the quality of surgical interventions and none evaluated the effect of the learning curve.

CONCLUSION

There is much room for improvement concerning the reported designs, conduct, and analysis of surgical RCTs. Considering the difficulty of surgical RCTs, some other approaches, such as surgeons' eligibility, performance of pilot studies, or implementation of pragmatic RCTs/expertise-based trials, should be feasibly implemented to overcome the presented challenges.

Journal impact factor, trial effect size, and methodological quality appear scantly related: a systematic review and meta-analysis

BACKGROUND

As systematic reviews' limited coverage of the medical literature necessitates decision-making based on unsystematic review, we investigated a possible advantage of systematic review (aside from dataset size and systematic analysis): does systematic review avoid potential bias in sampling primary studies from high impact factor journals? If randomized controlled trials (RCTs) reported in higher-impact journals present different treatment benefits than RCTs reported in lower-impact journals, readers who focus on higher-impact journals for their rapid literature reviews may introduce bias which could be mitigated by complete, systematic sampling.

METHODS

We randomly sampled Cochrane Library (20 July 2005) treatment reviews that measured mortality as a binary outcome, published in English or French, with at least five RCTs with one or more deaths. Our domain-based assessment of risk of bias included funding source, randomness of allocation sequence, blinding, and allocation concealment. The primary analysis employed logistic regression by a generalized linear model with a generalized estimating equation to estimate the association between various factors and publication in a journal with a high journal impact factor (JIF).

RESULTS

From the 29 included systematic reviews, 189 RCTs contributed data. However, in the primary analyses comparing RCT results within meta-analyses, there was no statistically significant association: unadjusted odds of greater than 50% mortality protection in high-JIF (> 5) journals were 1.4 (95% CI 0.42, 4.4) and adjusted, 2.5 (95% CI 0.6, 10). Elements of study quality were weakly, inconsistently, and not statistically significantly correlated with journal impact factor.

CONCLUSIONS

Journal impact factor may have little to no association with study results, or methodological quality, but the evidence is very uncertain.

Quality of reporting of systematic reviews and meta-analyses of surgical randomized clinical trials

Background

Well designed and conducted systematic reviews are essential to clinical practice. Surgical intervention is more complex than medical intervention when considering special items related to procedures. There has been no cross‐sectional study of the reporting quality of systematic reviews of surgical randomized trials focused on special items relating to surgical interventions.

Methods

A cross‐sectional survey of systematic reviews of surgical randomized trials published in 2007 and 2017 was undertaken via a PubMed search. Quality of reporting was assessed by the PRISMA checklist, with intervention details containing 27 items. Univariable and multivariable linear regression was used to explore factors in the checklist as indicators of reporting quality.

Results

A total of 204 systematic reviews were identified. The median score for the PRISMA checklist was 22 (i.q.r. 20–24), and systematic reviews published in 2017 had a significantly higher median score than those from 2007 (22 (i.q.r. 21–24) versus 20 (17–22); P < 0·001). Among the 27 items, 15 were reported adequately and three were reported poorly (in less than 50 per cent of reports). The proportion of other items reported ranged from 54·4 to 77·9 per cent. In multivariable analysis, systematic reviews published in 2017 (coefficient 0·59, 95 per cent c.i. 0·50 to 0·69) and Cochrane reviews (coefficient 0·67, 0·55 to 0·81) were associated with better reporting.

Conclusion

The quality of reporting of systematic reviews of surgical randomized trials has improved in the past 10 years. Some information relating to specific surgical interventions is, however, still reported poorly.

Discrepancies between Registered and Published Primary and Secondary Outcomes in Randomized Controlled Trials within the Plastic Surgery Literature: A Systematic Review

BACKGROUND

Recent studies have identified a high incidence of discrepancy between registered and published outcomes in registered medical and surgical randomized controlled trials. This has not yet been studied in the plastic surgery literature.

METHODS

The authors systematically assessed plastic surgery randomized controlled trials published between 2012 and 2016 in seven high-impact plastic surgery journals. Data were collected from the registration website and published articles using a standardized data extraction form.

RESULTS

A total of 145 randomized controlled trials were identified, with a 39 percent trial registration rate (n = 57). Forty-nine trials were included in the final analysis. Forty-three (88 percent) had a discrepancy between registered and published outcomes: 26 (53 percent) for primary outcome(s), and 39 (80 percent) for secondary outcome(s). The number of discrepancies in an individual trial ranged from one to seven for primary outcomes and one to 12 for secondary outcomes. Aesthetic surgery had the largest number of trials with outcome discrepancies (n = 15). The prevalence of unreported registered outcomes was 13 percent for primary outcomes and 38 percent for secondary outcomes. Registered nonsignificant primary outcomes were published as nonsignificant secondary outcomes in 30 percent of trials. Publishing new nonregistered secondary outcomes (65 percent) and changing the assessment timing of published primary outcomes (61 percent) were the most common types of discrepancies. Discrepancies favored a statistically significant positive outcome in 19 (44 percent) of the 43 trials with an outcome discrepancy. Discrepancies that resulted in published outcomes with improved patient relevance were found in eight trials (16 percent) for primary outcome discrepancies and 14 trials (29 percent) for secondary outcome discrepancies.

CONCLUSIONS

The plastic surgery literature has high rates of discrepancies between registered and published trial outcomes. Outcome reporting discrepancy is even more problematic for secondary outcomes, an area of analysis that has previously been poorly studied. The high rate of discrepancy change favoring a statistically significant outcome and more patient-relevant outcomes may indicate the pressure to demonstrate significant results to be accepted for publication in high-impact journals.

Trends and predictors of biomedical research quality, 1990-2015: a meta-research study

OBJECTIVE

To measure the frequency of adequate methods, inadequate methods and poor reporting in published randomised controlled trials (RCTs) and test potential factors associated with adequacy of methods and reporting.

DESIGN

Retrospective analysis of RCTs included in Cochrane reviews. Time series describes the proportion of RCTs using adequate methods, inadequate methods and poor reporting. A multinomial logit model tests potential factors associated with methods and reporting, including funding source, first author affiliation, clinical trial registration status, study novelty, team characteristics, technology and geography.

DATA

Risk of bias assessments for random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data and selective reporting, for each RCT, were mapped to bibliometric and funding data.

OUTCOMES

Risk of bias on six methodological dimensions and RCT-level overall assessment of adequate methods, inadequate methods or poor reporting.

RESULTS

This study analysed 20 571 RCTs. 5.7% of RCTs used adequate methods (N=1173). 59.3% used inadequate methods (N=12 190) and 35.0% were poorly reported (N=7208). The proportion of poorly reported RCTs decreased from 42.5% in 1990 to 30.2% in 2015. The proportion of RCTs using adequate methods increased from 2.6% in 1990 to 10.3% in 2015. The proportion of RCTs using inadequate methods increased from 54.9% in 1990 to 59.5% in 2015. Industry funding, top pharmaceutical company affiliation, trial registration, larger authorship teams, international teams and drug trials were associated with a greater likelihood of using adequate methods. National Institutes of Health funding and university prestige were not.

CONCLUSION

Even though reporting has improved since 1990, the proportion of RCTs using inadequate methods is high (59.3%) and increasing, potentially slowing progress and contributing to the reproducibility crisis. Stronger incentives for the use of adequate methods are needed.

Noninferiority Designed Cardiovascular Trials in Highest-Impact Journals

BACKGROUND

Noninferiority trials are increasingly being performed. However, little is known about their methodological quality. We sought to characterize noninferiority cardiovascular trials published in the highest-impact journals, features that may bias results toward noninferiority, features related to reporting of noninferiority trials, and the time trends.

METHODS

We identified cardiovascular noninferiority trials published in JAMA, Lancet, or New England Journal of Medicine from 1990 to 2016. Two independent reviewers extracted the data. Data elements included the noninferiority margin and the success of studies in achieving noninferiority. The proportion of trials showing major or minor features that may have affected the noninferiority inference was determined. Major factors included the lack of presenting the results in both intention-to-treat and per-protocol/as-treated cohorts, α>0.05, the new intervention not being compared with the best alternative, not justifying the noninferiority margin, and exclusion or loss of ≥10% of the cohort. Minor factors included suboptimal blinding, allocation concealment, and others.

RESULTS

From 2544 screened studies, we identified 111 noninferiority cardiovascular trials. Noninferiority margins varied widely: risk differences of 0.4% to 25%, hazard ratios of 1.05 to 2.85, odds ratios of 1.1 to 2.0, and relative risks of 1.1 to 1.8. Eighty-six trials claimed noninferiority, of which 20 showed superiority, whereas 23 (21.1%) did not show noninferiority, of which 8 also demonstrated inferiority. Only 7 (6.3%) trials were considered low risk for all the major and minor biasing factors. Among common major factors for bias, 41 (37%) did not confirm the findings in both intention-to-treat and per-protocol/as-treated cohorts and 4 (3.6%) reported discrepant results between intention-to-treat and per-protocol analyses. Forty-three (38.7%) did not justify the noninferiority margin. Overall, 27 (24.3%) underenrolled or had >10% exclusions. Sixty trials (54.0%) were open label. Allocation concealment was not maintained or unclear in 11 (9.9%). Publication of noninferiority trials increased over time (P<0.001). Fifty-two (46.8%) were published after 2010 and had a lower risk of methodological or reporting limitations for major (P=0.03) and minor factors (P=0.002).

CONCLUSIONS

Noninferiority trials in highest-impact journals commonly conclude noninferiority of the tested intervention, but vary markedly in the selected noninferiority margin, and frequently have limitations that may impact the inference related to noninferiority.

The fragility of statistically significant results in otolaryngology randomized trials

OBJECTIVES

The American Academy of Otolaryngology-Head and Neck Surgery regards randomized controlled trials as class A evidence. A novel method to determine the robustness of outcomes in trials is the fragility index. This index represents the number of patients whose status would have to change from a non-event to an event to make a statistically significant result non-significant.

METHODS

Investigators included otolaryngology journals listed in the top 10 of one or both of Google Scholar Metrics and Clarivate Analytics' Journal rankings. For inclusion, a randomized controlled trial needed to report a one-to-one random assignment of participants to condition, contain two parallel arms or have used a two-by-two factorial design, and report at least one statistically significant dichotomous outcome.

RESULTS

Sixty-nine trials met inclusion criteria. The median fragility index was three events (interquartile range 1-7.5). Median sample size was 72 (interquartile range 50-102.5). Modest correlations were observed between fragility index and total sample size (r = 0.27) and fragility index and event rate (r = 0.46). Investigators found no correlation between fragility index and impact factor or Science Citation Index. In 39% (27/69) of trials, the number lost to follow-up was equal to or greater than the fragility index.

CONCLUSION

A median fragility index of 3 indicates that three people, on average, are needed to alter the outcomes in otolaryngology trials. This indicates that the results of two-group randomized controlled trials reporting binary endpoints published in otolaryngology journals may frequently be fragile.

Quality of reporting in randomized controlled trials of therapeutic cardiovascular medical devices

BACKGROUND

Therapeutic medical devices play an important role in the treatment of cardiovascular diseases. The reliability of the randomized controlled trial, which is the best design for assessing treatment effects, largely depends on the information found in published reports. Limited information regarding the quality of reporting about therapeutic medical devices in trials was provided.

METHOD

A cross-sectional study was conducted to assess the reporting quality of randomized controlled trials that tested the effects of therapeutic cardiovascular medical devices. The quality of reporting was assessed against a modified Consolidated Standards of Reporting Trials checklist, including 47 items from the Consolidated Standards of Reporting Trials statement and Consolidated Standards of Reporting Trials extension. We also examined the specific items regarding medical devices. Univariable and multivariable linear regressions were undertaken to explore potential factors associated with Consolidated Standards of Reporting Trials scores.

RESULT

Some 115 randomized controlled trials were identified. The mean (standard deviation) Consolidated Standards of Reporting Trials score was 20.5 (5.0). The extent of compliance with the Consolidated Standards of Reporting Trials reporting guideline differed substantially across items: 5 of the 47 items were reported adequately across trials (more than 90%), and 10 were reported adequately in less than 5% of trials. Less than 50% of the trials reported additional items related to the medical device. Multivariable regression analysis showed that trials published in general journals (coefficient 7.44, 95% confidence interval [CI]: 5.50-9.38), with larger sample sizes (coefficient 2.30, 95% CI: 0.76-3.83), and multiple-center studies (coefficient 3.14, 95% CI: 1.27-5.01) were associated with a higher quality of reporting.

CONCLUSION

The overall reporting quality in randomized controlled trials of therapeutic medical device trials is suboptimal, particularly in terms of items regarding surgeons and hospitals. We suggest that the existing Consolidated Standards of Reporting Trials and extension should be modified to be more applicable to therapeutic medical devices.

The reporting of safety among drug systematic reviews was poor before the implementation of the PRISMA harms checklist

OBJECTIVES

To examine, through a cross-sectional survey, how well safety information was reported among drug systematic reviews predating PRISMA harms checklist and explore factors associated with better reporting.

STUDY DESIGN AND SETTING

We searched PubMed to identify all systematic reviews published in the Cochrane Database of Systematic Review or the core clinical journals in 2015, one year before the PRISMA harms checklist was published. We randomly selected, in a 1:1 ratio, Cochrane and non-Cochrane systematic reviews assessing drug effects (including both efficacy and safety). We used the PRISMA harms checklist published in 2016 to assess the quality of reporting of drug safety information. Multivariable linear regression analyses were used to explore the association of six prespecified variables with more complete reporting of PRISMA harms items.

RESULTS

We included 120 systematic reviews, including 60 Cochrane and 60 non-Cochrane reviews. Scores on the PRISMA harms checklist (23 items) were low (median 4, [first, third quartile: 2, 6]), with no difference between Cochrane and non-Cochrane reviews (4.5 [2, 7] vs. 4 [2.5, 5]; P = 0.29). Among all eligible reviews, only one item (i.e., state conclusions in coherence with the review findings) was reported adequately (proportion of adherence 81.6%); proportion of reporting for other items ranged from 1.7% to 68.3%. The four essential reporting items from PRISMA harms checklist were also poorly complied (proportion of adherence ranged from 1.7% to 9.2%). Multivariable linear regression analyses found no significant associations between any study characteristic and reporting on the PRISMA harms, likely because of limited variability in scores across studies.

CONCLUSIONS

The reporting of safety information was poor both for Cochrane and non-Cochrane drug systematic reviews predating PRISMA harms checklist. The findings suggested a strong need to use the PRISMA harms checklist for reporting safety among drug systematic reviews.

Examining How Treatment Fidelity Is Supported, Measured, and Reported in K-3 Reading Intervention Research

Treatment fidelity data (descriptive and statistical) are critical to interpreting and generalizing outcomes of intervention research. Despite recommendations for treatment fidelity reporting from funding agencies and researchers, past syntheses have found treatment fidelity is frequently unreported (e.g., Swanson, The Journal of Special Education, 47, 3-13, 2011) in educational interventions and fidelity data are seldom used to analyze its relation to student outcomes (O'Donnell, Review of Educational Research, 78(1), 33-84, 2008). The purpose of this synthesis was to examine how treatment fidelity is supported, measured, and reported in reading intervention studies conducted with students at risk or with reading difficulties in grades K-3 from 1995 through 2015. All studies (k = 175) were coded to extract and classify information related to (a) the characteristics of the intervention study (e.g., publication year, research design); (b) treatment implementer training and support; (c) treatment fidelity data collection procedures, dimensions (i.e., adherence, quality, receipt, dosage, and differentiation), and levels of treatment fidelity data; and (d) the use of fidelity scores in the analysis of treatment effects. Results indicated that less than half (47%) of the reading intervention studies synthesized reported treatment fidelity data (numeric or narrative). Exploratory analyses showed that several study features were associated with the prevalence of fidelity reporting. Studies reporting treatment fidelity largely measured treatment adherence, and scores were, on average, high. Other dimensions of treatment fidelity (e.g., treatment differentiation), and analyses relating fidelity data to outcomes, were consistently absent from the corpus of reading intervention studies reviewed. Recommendations for enhancing how treatment fidelity data in intervention studies are collected and reported are presented.

Clinical trial design and dissemination: comprehensive analysis of clinicaltrials.gov and PubMed data since 2005

OBJECTIVE

To investigate the distribution, design characteristics, and dissemination of clinical trials by funding organisation and medical specialty.

DESIGN

Cross sectional descriptive analysis.

DATA SOURCES

Trial protocol information from clinicaltrials.gov, metadata of journal articles in which trial results were published (PubMed), and quality metrics of associated journals from SCImago Journal and Country Rank database.

SELECTION CRITERIA

All 45 620 clinical trials evaluating small molecule therapeutics, biological drugs, adjuvants, and vaccines, completed after January 2006 and before July 2015, including randomised controlled trials and non-randomised studies across all clinical phases.

RESULTS

Industry was more likely than non-profit funders to fund large international randomised controlled trials, although methodological differences have been decreasing with time. Among 27 835 completed efficacy trials (phase II-IV), 15 084 (54.2%) had disclosed their findings publicly. Industry was more likely than non-profit trial funders to disseminate trial results (59.3% (10 444/17 627) v 45.3% (4555/10 066)), and large drug companies had higher disclosure rates than small ones (66.7% (7681/11 508) v 45.2% (2763/6119)). Trials funded by the National Institutes of Health (NIH) were disseminated more often than those of other non-profit institutions (60.0% (1451/2417) v 40.6% (3104/7649)). Results of studies funded by large drug companies and NIH were more likely to appear on clinicaltrials.gov than were those from non-profit funders, which were published mainly as journal articles. Trials reporting the use of randomisation were more likely than non-randomised studies to be published in a journal article (6895/19 711 (34.9%) v 1408/7748 (18.2%)), and journal publication rates varied across disease areas, ranging from 42% for autoimmune diseases to 20% for oncology.

CONCLUSIONS

Trial design and dissemination of results vary substantially depending on the type and size of funding institution as well as the disease area under study.

The quality of reporting in randomized controlled trials of acupuncture for knee osteoarthritis: A cross-sectional survey

OBJECTIVE

To assess the reporting quality of acupuncture trials for knee osteoarthritis (KOA), and explore the factors associated with the reporting.

METHOD

Three English and four Chinese databases were searched from inception to December 2016 for randomized control trials testing effects of acupuncture for knee osteoarthritis. We used the standard CONSORT (2010 version), CONSORT Extension for Non-Pharmacological Treatments, and STRICTA for measuring the quality of reporting. Using pre-specified study characteristics, we undertook regression analyses to examine factors associated with the reporting quality.

RESULTS

A total of 318 RCT reports were included. For the standard CONSORT, ten items were substantially under-reported (reported in less than 5% of RCTs), including specification of important changes to methods after trial commencement (0.6%), description of any changes to trial outcomes (0.0%), implementation of interim analyses and stopping guidelines (0.6%), statement about why the trial ended or was stopped (1.6%), statement about the registration status (4.4%), accessibility of full trial protocol (4.7%), implementation of randomization (4.7%), description of the similarity of interventions (3.5%), conduct of ancillary analyses (3.8%) and presentation of methods for additional analyses (4.4%). Four of the STRICTA items were under-reported (reported in less than 10% of RCTs), including description of acupuncture style (8.5%), presentation of extent to which treatment varied (1.3%), statement of practitioner background (7.2%) and rationale for the control (9.1%). For CONSORT Extension, the reporting was poor across all items (reported in less than 10% of trials). Trials including authors with expertise in epidemiology or statistics, published in English, or enrolling patients from multiple centers were more likely to have better reporting.

CONCLUSIONS

The reporting in RCTs of acupuncture for KOA was generally poor. To improve the reporting quality, journals should encourage strict adherence to the reporting guidelines.

Quality of missing data reporting and handling in palliative care trials demonstrates that further development of the CONSORT statement is required: a systematic review

Objectives

Assess (i) the quality of reporting and handling of missing data (MD) in palliative care trials, (ii) whether there are differences in the reporting of criteria specified by the Consolidated Standards of Reporting Trials (CONSORT) 2010 statement compared with those not specified, and (iii) the association of the reporting of MD with journal impact factor and CONSORT endorsement status.

Study Design and Setting

Systematic review of palliative care randomized controlled trials. CENTRAL, MEDLINE, and EMBASE (2009–2014) were searched.

Results

One hundred and eight trials (15,560 participants) were included. MD was incompletely reported and not handled in accordance with current guidance. Reporting criteria specified by the CONSORT statement were better reported than those not specified (participant flow, 69%; number of participants not included in the primary outcome analysis, 94%; and the reason for MD, 71%). However, MD in items contributing to scale summaries (10%) and secondary outcomes (9%) were poorly reported, so the proportion of MD stated is likely to be an underestimate. The reason for MD provided was unclear for 54% of participants and only 16% of trials with MD reported a MD sensitivity analysis. The odds of reporting most of the MD and other risk of bias reporting criteria were increased as the journal impact factor increased and in journals that endorsed the CONSORT statement.

Conclusion

Further development of the CONSORT MD reporting guidance is likely to improve the quality of reporting. Reporting recommendations are provided.

Quality of randomized controlled trials in eating disorder prevention

OBJECTIVE

To investigate the quality of randomized controlled trials (RCTs) of eating disorder prevention.

METHOD

A systematic literature search was conducted in Medline, PsycINFO, Embase, Scopus, and the Cochrane Collaboration Library to January 2016. Studies were included if they were RCTs that tested an eating disorder prevention program. We identified 96 studies with a total 15,350 participants (91% female, M age = 17 years) and rated quality with the Quality Rating Scale (QRS; Moncrieff et al., 2001).

RESULTS

The mean QRS score was 62% (SD = 13%). Several standards of quality were not frequently fulfilled (i.e., failed to achieve an optimal rating), for example, power calculation (85%), intent-to-treat analysis (54%), blinding of assessor (75%), representative sample (78%), adequate sample size (75%), and appropriate duration of trial including follow-up (67%). QRS was positively and significantly associated with publication year, number of authors, and PubMed-indexation.

DISCUSSION

Given the majority of eating disorder prevention studies had problems with trial quality, it is recommended that future RCTs follow quality checklists and CONSORT guidelines, that RCTs are registered, and protocols published in advance. In addition, funding bodies are called on to deliver the support needed to ensure that preventions for eating disorders are efficiently and cost-effectively achieved.

Is bilingualism losing its advantage? A bibliometric approach

This study uses several bibliometric indices to explore the temporal course of publication trends regarding the bilingual advantage in executive control over a ten-year window. These indices include the number of published papers, numbers of citations, and the journal impact factor. According to the information available in their abstracts, studies were classified into one of four categories: supporting, ambiguous towards, not mentioning, or challenging the bilingual advantage. Results show that the number of papers challenging the bilingual advantage increased notably in 2014 and 2015. Both the average impact factor and the accumulated citations as of June 2016 were equivalent between categories. However, of the studies published in 2014, those that challenge the bilingual advantage accumulated more citations in June 2016 than those supporting it. Our findings offer evidence-based bibliometric information about the current state of the literature and suggest a change in publication trends regarding the literature on the bilingual advantage.

Two Decades of Cardiovascular Trials With Primary Surrogate Endpoints: 1990-2011

Background

Surrogate endpoint trials test strategies more efficiently but are accompanied by uncertainty about the relationship between changes in surrogate markers and clinical outcomes.

Methods and Results

We identified cardiovascular trials with primary surrogate endpoints published in the New England Journal of Medicine, Lancet, and JAMA: Journal of the American Medical Association from 1990 to 2011 and determined the trends in publication of surrogate endpoint trials and the success of the trials in meeting their primary endpoints. We tracked for publication of clinical outcome trials on the interventions tested in surrogate trials. We screened 3016 articles and identified 220 surrogate endpoint trials. From the total of 220 surrogate trials, 157 (71.4%) were positive for their primary endpoint. Only 59 (26.8%) surrogate trials had a subsequent clinical outcomes trial. Among these 59 trials, 24 outcomes trial results validated the positive surrogates, whereas 20 subsequent outcome trials were negative following positive results on a surrogate. We identified only 3 examples in which the surrogate trial was negative but a subsequent outcomes trial was conducted and showed benefit. Findings were consistent in a sample cohort of 383 screened articles inclusive of 37 surrogate endpoint trials from 6 other high‐impact journals.

Conclusions

Although cardiovascular surrogate outcomes trials frequently show superiority of the tested intervention, they are infrequently followed by a prominent outcomes trial. When there was a high‐profile clinical outcomes study, nearly half of the positive surrogate trials were not validated. Cardiovascular surrogate outcome trials may be more appropriate for excluding benefit from the patient perspective than for identifying it.

Industry sponsorship and research outcome

BACKGROUND

Clinical research affecting how doctors practice medicine is increasingly sponsored by companies that make drugs and medical devices. Previous systematic reviews have found that pharmaceutical-industry sponsored studies are more often favorable to the sponsor's product compared with studies with other sources of sponsorship. A similar association between sponsorship and outcomes have been found for device studies, but the body of evidence is not as strong as for sponsorship of drug studies. This review is an update of a previous Cochrane review and includes empirical studies on the association between sponsorship and research outcome.

OBJECTIVES

To investigate whether industry sponsored drug and device studies have more favorable outcomes and differ in risk of bias, compared with studies having other sources of sponsorship.

SEARCH METHODS

In this update we searched MEDLINE (2010 to February 2015), Embase (2010 to February 2015), the Cochrane Methodology Register (2015, Issue 2) and Web of Science (June 2015). In addition, we searched reference lists of included papers, previous systematic reviews and author files.

SELECTION CRITERIA

Cross-sectional studies, cohort studies, systematic reviews and meta-analyses that quantitatively compared primary research studies of drugs or medical devices sponsored by industry with studies with other sources of sponsorship. We had no language restrictions.

DATA COLLECTION AND ANALYSIS

Two assessors screened abstracts and identified and included relevant papers. Two assessors extracted data, and we contacted authors of included papers for additional unpublished data. Outcomes included favorable results, favorable conclusions, effect size, risk of bias and whether the conclusions agreed with the study results. Two assessors assessed risk of bias of included papers. We calculated pooled risk ratios (RR) for dichotomous data (with 95% confidence intervals (CIs)).

MAIN RESULTS

Twenty-seven new papers were included in this update and in total the review contains 75 included papers. Industry sponsored studies more often had favorable efficacy results, RR: 1.27 (95% CI: 1.17 to 1.37) (25 papers) (moderate quality evidence), similar harms results RR: 1.37 (95% CI: 0.64 to 2.93) (four papers) (very low quality evidence) and more often favorable conclusions RR: 1.34 (95% CI: 1.19 to 1.51) (29 papers) (low quality evidence) compared with non-industry sponsored studies. Nineteen papers reported on sponsorship and efficacy effect size, but could not be pooled due to differences in their reporting of data and the results were heterogeneous. We did not find a difference between drug and device studies in the association between sponsorship and conclusions (test for interaction, P = 0.98) (four papers). Comparing industry and non-industry sponsored studies, we did not find a difference in risk of bias from sequence generation, allocation concealment, follow-up and selective outcome reporting. However, industry sponsored studies more often had low risk of bias from blinding, RR: 1.25 (95% CI: 1.05 to 1.50) (13 papers), compared with non-industry sponsored studies. In industry sponsored studies, there was less agreement between the results and the conclusions than in non-industry sponsored studies, RR: 0.83 (95% CI: 0.70 to 0.98) (six papers).

AUTHORS' CONCLUSIONS

Sponsorship of drug and device studies by the manufacturing company leads to more favorable efficacy results and conclusions than sponsorship by other sources. Our analyses suggest the existence of an industry bias that cannot be explained by standard 'Risk of bias' assessments.

Reporting quality of randomized controlled trial abstracts: survey of leading general dental journals

BACKGROUND

The authors conducted a study to assess the reporting quality of randomized controlled trial (RCT) abstracts published in leading general dental journals, investigate any improvement after the release of the Consolidated Standards of Reporting Trials (CONSORT) for Abstracts guidelines, and identify factors associated with better reporting quality.

METHODS

The authors searched PubMed for RCTs published in 10 leading general dental journals during the periods from 2005 to 2007 (pre-CONSORT period) and 2010 to 2012 (post-CONSORT period). The authors evaluated and scored the reporting quality of included abstracts by using the original 16-item CONSORT for Abstracts checklist. The authors used risk ratios and the t test to compare the adequate reporting rate of each item and the overall quality in the 2 periods. The authors used univariate and multivariate regressions to identify predictors of better reporting quality.

RESULTS

The authors included and evaluated 276 RCT abstracts. Investigators reported significantly more checklist items during the post-CONSORT period (mean [standard deviation {SD}], 4.53 [1.69]) than during the pre-CONSORT period (mean [SD], 3.87 [1.10]; mean difference, -0.66 [95% confidence interval, -0.99 to -0.33]; P < .001). Investigators reported 3 items-interventions, objective, and conclusions-adequately in most of the abstracts (> 80%). In contrast, the authors saw sufficient reporting of randomization, recruitment, outcome in the results section, and funding in none of the pre-CONSORT abstracts and less than 2% of the post-CONSORT abstracts. On the basis of the multivariate analysis, a higher impact factor (P < .001) and a publication date in the post-CONSORT period (P = .003) were associated significantly with higher reporting quality.

CONCLUSIONS

The reporting quality of RCT abstracts from leading general dental journals has improved significantly, but there is still room for improvement.

PRACTICAL IMPLICATIONS

Joint efforts by authors, reviewers, journal editors, and other stakeholders to improve the reporting of dental RCT abstracts are needed.

A systematic review of trial registration and selective outcome reporting in psychotherapy randomized controlled trials

OBJECTIVE

Selective outcome reporting poses serious implications on our evidence base for best practice. The extent to which selective outcome reporting and trial registration occurs in the wider psychotherapy literature needs to be investigated.

METHOD

Randomized controlled psychotherapy trials published between 2010 and 2014 were selected from the five highest impact factor journals in clinical psychology that publish clinical trials. Data on primary and secondary outcomes, funding, and participant numbers were extracted from the article and registry and compared.

RESULTS

From 112 trials, 67 (59.8%) were registered, 27 (24.1%) were prospectively registered, and only 13 (11.6%) were correctly registered and reported. Seven of these 13 trials showed evidence of selective outcome reporting, of which four had discrepancies favoring significant outcomes. One of the remaining six trials changed their primary outcomes during participant enrollment. Overall, only five (4.5%) trials were free from selective outcome reporting. Three of these five trials had more than a 10% change between planned and achieved sample size. Funding was not associated with correct registration or reporting.

CONCLUSIONS

The proportion of psychotherapy randomized controlled trials correctly registered and transparently reported is poor. Psychologists should consider the impact these results have on public confidence in reported outcomes.

Tools to assess risk of bias in systematic reviews of nursing intervention in China: Global implications of the findings

BACKGROUND

The number of systematic reviews (SRs) of nursing interventions is increasing in China. Authors of such studies are encouraged to use a risk of bias tool for assessing individual studies. Therefore, it is important to know how these tools have been applied in SRs in Chinese nursing.

PURPOSE

The purpose of the study is to examine risk of bias tools used by Chinese nursing researchers to assess the quality of individual studies included in SRs of nursing interventions.

METHODS

We searched the Chinese Biomedical Literature Database, Chinese Journal Full-Text Database, Chinese Academic Journal Full-Text Database, and Wanfang Database. Each relevant review found was subjected to quality assessment, data synthesis, and comprehensively described.

DISCUSSION

Two hundred eight SRs were identified, most (94.7%) of which used the words "systematic review" or "meta-analysis" in their titles. Most used quality assessment rather than risk of bias as their major method to gauge quality; 7.6% reported on six domains of the Cochrane risk of bias tool. Only four provided a risk of bias graph/summary figure.

CONCLUSIONS

Many SRs of nursing interventions are published in Chinese journals, and their assessment of risk of bias is usually either lacking or incomplete, potentially producing misleading results. This may also be the case in other countries. Assessment of risk of bias in SRs is a requisite. Authors who systematically review nursing literature should follow the latest Cochrane Collaboration Handbook.

Design and Validity of Randomized Controlled Dental Restorative Trials

Background: The evidence stemming from trials on restorative materials is shaped not only by trial findings, but also trial design and validity. We aimed to evaluate both aspects in randomized controlled dental restorative trials published from 2005-2015. Methods: Using systematic review methodology, we retrieved trials comparing restorative or adhesive dental materials. Two authors independently assessed design, risk of bias, registration status, and findings of trials. Descriptive and regression analyses were performed. Results: 114 studies on 15,321 restorations placed mainly in permanent teeth of 5232 patients were included. Per trial, the median number of patients was 37 (25th/75th percentiles: 30/51). Follow-up was 24 (20/48) months. Seventeen percent of trials reported on sample size calculations, 2% had been registered. Most trials (90%) used US Public Health Service (USPHS) criteria, and had a high risk of bias. More recent trials were more likely to have been registered, to have reported on sample size calculations, to be of low risk of bias, and to use other than USPHS-criteria. Twenty-three percent of trials yielded significant differences between groups. The likelihood of such differences was significantly increased in older studies, studies with potential reporting bias, published in journals with high impact factor (>2), longer follow-up periods, and not using USPHS-criteria. Conclusions: The majority of dental restorative trials published from 2005-2015 had limited validity. Risk of bias decreased in more recent trials. Future trials should aim for high validity, be registered, and use defined and appropriate sample sizes, follow-up periods, and outcome measures.

Associated Factors and Consequences of Risk of Bias in Randomized Controlled Trials of Yoga: A Systematic Review

BACKGROUND

Bias in randomized controlled trials (RCTs) of complementary therapy interventions seems to be associated with specific factors and to potentially distort the studies' conclusions. This systematic review assessed associated factors of risk of bias and consequences for the studies' conclusions in RCTs of yoga as one of the most commonly used complementary therapies.

METHODS

Medline/PubMed, Scopus, IndMED and the Cochrane Library were searched through February 2014 for yoga RCTs. Risk of selection bias was assessed using the Cochrane tool and regressed to a) publication year; b) country of origin; c) journal type; and d) impact factor using multiple logistic regression analysis. Likewise, the authors' conclusions were regressed to risk of bias.

RESULTS

A total of 312 RCTs were included. Impact factor ranged from 0.0 to 39.2 (median = 1.3); 60 RCT (19.2%) had a low risk of selection bias, and 252 (80.8%) had a high or unclear risk of selection bias. Only publication year and impact factor significantly predicted low risk of bias; RCTs published after 2001 (adjusted odds ratio (OR) = 12.6; 95% confidence interval (CI) = 1.7, 94.0; p<0.001) and those published in journals with impact factor (adjusted OR = 2.6; 95%CI = 1.4, 4.9; p = 0.004) were more likely to have low risk of bias. The authors' conclusions were not associated with risk of bias.

CONCLUSIONS

Risk of selection bias was generally high in RCTs of yoga; although the situation has improved since the publication of the revised CONSORT statement 2001. Pre-CONSORT RCTs and those published in journals without impact factor should be handled with increased care; although risk of bias is unlikely to distort the RCTs' conclusions.

Use and reporting of restricted randomization: a review

RATIONALE, AIMS AND OBJECTIVES

Restricted randomization, such as blocking or minimization, allows for the creation of balanced groups and even distribution of covariates, but it increases the risk of selection bias and technical error. Various methods are available to reduce these risks but there is limited evidence about their current usage, and there are also indications that reporting of these methods may not be adequate. This review aims to identify how frequently different methods of restriction are being used and to assess the reporting of these methods against established reporting standards.

METHODS

82 reports of randomized controlled trial were reviewed. For each trial, the reported method of randomization was recorded and the reporting of randomization was assessed. Where the method of randomization was not clear from the main paper, protocols and other published materials were also reviewed, and authors were contacted for further information.

RESULTS

For 11% of trials the method of randomization was not reported in either the paper or a published protocol, and in a further 39% of cases the report omitted key details so that the predictability of the method could not be evaluated. In total, 88% of trials appear to have used some form of restricted randomization, and all of those that report the exact methods used either blocking or minimization. 15% of trials reported using blocks of six or less and 4% used minimization with no random element reported, both of which are highly predictable.

CONCLUSION

Our results indicate that the majority of trials use some form of restriction, with many using relatively predictable methods that put them at greater risk of selection bias and technical error. Reporting of randomization methods often falls short of the minimum requirements set out by the CONSORT statement, leaving the reader unable to make an informed judgement about the risk of bias.

The fragility of statistically significant findings from randomized trials in spine surgery: a systematic survey

BACKGROUND CONTEXT

Randomized controlled trials (RCTs) are the most trustworthy source for evaluating treatment effects, but RCTs of spine surgery interventions often produce discordant results. The Fragility Index is a novel metric to inform about the robustness of statistically significant results.

PURPOSE

The aim was to determine the robustness of statistically significant results from RCTs of spine surgery interventions.

STUDY DESIGN/SETTING

This was a systematic survey.

PATIENT SAMPLE

The sample included RCTs of spine surgery interventions.

OUTCOME MEASURES

The Fragility Index is the minimum number of patients in a trial whose status would have to change from a nonevent to an event to change a statistically significant result to a nonsignificant result. Events refer to the occurrence of any dichotomous outcome, such as successful fusion, incident fracture, adjacent segment degeneration, or achievement of a certain functional score. A small Fragility Index indicates that the statistical significance of a result hinges on only a few events, and a large Fragility Index increases one's confidence in the observed treatment effects.

METHODS

We systematically reviewed a database for evidence-based orthopedics and identified all the RCTs that reported at least one positive outcome (ie, p<.05). Two reviewers independently assessed eligibility and extracted data. We used the Fisher exact test to compute Fragility Index values and multivariable linear regression to evaluate potential associated factors.

RESULTS

We identified 40 eligible RCTs with a median sample size of 132 patients (interquartile range [IQR] 79-208) and a median total number of outcome events for the chosen outcome of 31 (IQR 13-63). The median Fragility Index was two (IQR 1-3), which means that adding two events to one of the trial's treatment arms eliminated its statistical significance. The Fragility Index was less than or equal to three events in 75% of the trials, and was less than or equal to the number of patients lost to follow-up in 65% of the trials. Fragility Index values correlated positively with total sample size (r=0.35; p<.05). When adjusted for losses to follow-up and risk of bias, increasing Fragility Index values were associated only with increasingly significant reported p values (p<.01).

CONCLUSIONS

Statistically significant results in spine surgery RCTs are frequently fragile. The addition of only a small number of outcome events can completely eliminate significance. Surgeons, researchers, and other evidence users should exercise caution when interpreting the findings from RCTs with low Fragility Index values and applying these results to patient care.