Quality of reporting of randomized controlled trials published in Intensive Care Medicine from 2001 to 2010

Delta Inflation, Optimism Bias, and Uncertainty in Clinical Trials

The phenomenon of delta inflation, in which design treatment effects tend to exceed observed treatment effects, has been documented in several therapeutic areas. Delta inflation has often been attributed to investigators' optimism bias, or an unwarranted belief in the efficacy of new treatments. In contrast, we argue that delta inflation may be a natural consequence of clinical equipoise, that is, genuine uncertainty about the relative benefits of treatments before a trial is initiated. We review alternative methodologies that can offer more direct evidence about investigators' beliefs, including Bayesian priors and forecasting analysis. The available evidence for optimism bias appears to be mixed, and can be assessed only where uncertainty is expressed explicitly at the trial design stage.

Research Letter: Characterizing spin in psychiatric clinical research literature using large language models

Importance

Spin is a common form of biased reporting that misrepresents study results in publications as more positive than an objective assessment would indicate, but its prevalence in psychiatric journals is unknown.

Objective

To apply a large language model to characterize the extent to which original reports of pharmacologic and non-pharmacologic interventions in psychiatric journals reflect spin.

Design

We identified abstracts from studies published between 2013 and 2023 in 3 high-impact psychiatric journals describing randomized trials or meta-analyses of interventions.

Main Outcome and Measure

Presence or absence of spin estimated by a large language model (GPT4-turbo, turbo-2024-04-09), validated using gold standard abstracts with and without spin.

Results

Among a total of 663 abstracts, 296 (44.6%) exhibited possible or probable spin - 230/529 (43.5%) randomized trials, 66/134 (49.3%) meta-analyses; 148/310 (47.7%) for medication, 107/238 (45.0%) for psychotherapy, and 41/115 (35.7%) for other interventions. In a multivariable logistic regression model, reports of randomized trials, and non-pharmacologic/non-psychotherapy interventions, were less likely to exhibit spin, as were more recent publications.

Conclusions and Relevance

A substantial subset of psychiatric intervention abstracts in high-impact journals may contain results presented in a potentially misleading way, with the potential to impact clinical practice. The success in automating spin detection via large language models may facilitate identification and revision to minimize spin in future publications.

Quality reporting of randomized controlled trials on SGLT2 inhibitors for heart failure: a comprehensive assessment

Randomised controlled trials (RCTs) provide clinicians with the best evidence of the effectiveness of an intervention, and complete and transparent trial reports help to critically assess and use trial results. The objective of our study was to assess the quality of reporting in RCTs of sodium-glucose co-transporter protein 2 (SGLT2) inhibitors for heart failure (HF) and identify factors associated with improved reporting quality. Two researchers conducted a comprehensive search in four databases (PubMed, Web of Science, EMBASE, and Cochrane). The quality of each report was assessed using a 25-point Overall Quality Score (OQS) based on the guidelines provided in the 2010 Consolidated Standards for Reporting of Trials (CONSORT) statement. We included a total of 58 relevant RCTs. The median OQS in the 2010 CONSORT statement was 15 (range 7.5-24). The missing items were primarily found in the 'Methods' and 'Results' sections of the 2010 CONSORT statement. Multivariate regression modeling revealed that a more recent publication year, high impact factor, and large sample size were significant predictors of OQS improvement. The findings suggest that the overall quality of reported RCTs of SGLT2 inhibitors in HF is unsatisfactory, which reduces their potential usefulness.

Linguistic spin in randomized controlled trials about age-related macular degeneration

Objective

To evaluate the prevalence, type and extent of linguistic spin in randomized controlled trials (RCTs) exploring interventions in patients with age-related macular degeneration (AMD), as well as to investigate whether study variables were correlated with linguistic spin.

Study design and setting

PubMed was searched from 2011 to 2020 to identify RCTs including patients with AMD. Two authors independently assessed a total of 96 RCTs. Linear regression analyses were performed to investigate whether linguistic spin was correlated with predefined study variables.

Results

Linguistic spin was found in 61 of 96 abstracts (63.5%) and in 90 of 96 main texts (93.8%). Use of words pointing out the beneficial effect of a treatment and the use of '(statistically) significant/significance' without reporting a P-value or a 95% confidence interval (CI) were the most frequently identified categories of linguistic spin. Sample size was significantly correlated with the total linguistic spin score (95% CI 0.38-5.23, P = 0.02).

Conclusion

A high prevalence and extent of linguistic spin in RCTs about AMD was found. We highlighted the importance of objective reporting and awareness of linguistic spin among ophthalmologists and other readers.

A multivariate model for successful publication of intensive care medicine randomized controlled trials in the highest impact factor journals: the SCOTI score

Background

Critical care randomized controlled trials (RCTs) are often published in high-impact journals, whether general journals [the New England Journal of Medicine (NEJM), The Lancet, the Journal of the American Medical Association (JAMA)] or critical care journals [Intensive Care Medicine (ICM), the American Journal of Respiratory and Critical Care Medicine (AJRCCM), Critical Care Medicine (CCM)]. As rejection occurs in up to 97% of cases, it might be appropriate to assess pre-submission probability of being published. The objective of this study was to develop and internally validate a simplified score predicting whether an ongoing trial stands a chance of being published in high-impact general journals.

Methods

A cohort of critical care RCTs published between 1999 and 2018 in the three highest impact medical journals (NEJM, The Lancet, JAMA) or the three highest impact critical care journals (ICM, AJRCCM, CCM) was split into two samples (derivation cohort, validation cohort) to develop and internally validate the simplified score. Primary outcome was journal of publication assessed as high-impact general journal (NEJM, The Lancet, JAMA) or critical care journal (ICM, AJRCCM, CCM).

Results

A total of 968 critical care RCTs were included in the predictive cohort and split into a derivation cohort (n = 510) and a validation cohort (n = 458). In the derivation cohort, the sample size (P value < 0.001), the number of centers involved (P value = 0.01), mortality as primary outcome (P value = 0.002) or a composite item including mortality as primary outcome (P value = 0.004), and topic [ventilation (P value < 0.001) or miscellaneous (P value < 0.001)] were independent factors predictive of publication in high-impact general journals, compared to high-impact critical care journals. The SCOTI score (Sample size, Centers, Outcome, Topic, and International score) was developed with an area under the ROC curve of 0.84 (95% Confidence Interval, 0.80–0.88) in validation by split sample.

Conclusions

The SCOTI score, developed and validated by split sample, accurately predicts the chances of a critical care RCT being published in high-impact general journals, compared to high-impact critical care journals.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13613-021-00954-x.

Spin in Published Reports of Tinnitus Randomized Controlled Trials: Evidence of Overinterpretation of Results

Background: Spin refers to reporting practices that could distort the interpretation and mislead readers by being more optimistic than the results justify, thereby possibly changing the perception of clinicians and influence their decisions. Because of the clinical importance of accurate interpretation of results and the evidence of spin in other research fields, we aim to identify the nature and frequency of spin in published reports of tinnitus randomized controlled trials (RCTs) and to assess possible determinants and effects of spin. Methods: We searched PubMed systematically for RCTs with tinnitus-related outcomes published from 2015 to 2019. All eligible articles were assessed on actual and potential spin using prespecified criteria. Results: Our search identified 628 studies, of which 87 were eligible for evaluation. A total of 95% of the studies contained actual or potential spin. Actual spin was found mostly in the conclusion of articles, which reflected something else than the reported point estimate (or CI) of the outcome (n = 34, 39%) or which was selectively focused (n = 49, 56%). Linguistic spin ("trend," "marginally significant," or "tendency toward an effect") was found in 17% of the studies. We were not able to assess the association between study characteristics and the occurrence of spin due to the low number of trials for some categories of the study characteristics. We found no effect of spin on type of journal [odds ratio (OR) -0.13, 95% CI -0.56-0.31], journal impact factor (OR 0.17, 95% CI -0.18-0.51), or number of citations (OR 1.95, CI -2.74-6.65). Conclusion: There is a large amount of spin in tinnitus RCTs. Our findings show that there is room for improvement in reporting and interpretation of results. Awareness of different forms of spin must be raised to improve research quality and reduce research waste.

Abstracts for reports of randomised trials of COVID-19 interventions had low quality and high spin

OBJECTIVES

To assess the reporting quality of abstracts for published randomized controlled trials (RCTs) of interventions for coronavirus disease 2019 (COVID-19), including the use of spin strategies and the level of spin for RCTs with statistically non-significant primary outcomes, and to explore potential predictors for reporting quality and the severity of spin.

STUDY DESIGN AND SETTING

PubMed was searched to find RCTs that tested interventions for COVID-19, and the reporting quality and spin in the abstracts were assessed. Linear regression analyses were used to identify potential predictors.

RESULTS

Forty RCT abstracts were included in our assessment of reporting quality, and a higher word count in the abstract was significantly correlated with higher reporting scores (95% CI 0.044 to 0.658, P=0.026). Multiple spin strategies were identified. Our multivariate analyses showed that geographical origin was associated with severity of spin, with research from non-Asian regions containing fewer spin strategies (95% CI -0.760 to -0.099, P=0.013).

CONCLUSIONS

The reporting quality of abstracts of RCTs of interventions for COVID-19 is far from satisfactory. A relatively high proportion of the abstracts contained spin, and the findings reported in the results and conclusion sections of these abstracts need to be interpreted with caution.

Evaluation of Spin in the Abstracts of Systematic Reviews and Meta-Analyses of Atopic Dermatitis Treatments and Interventions

BACKGROUND

Spin - the misrepresentation of a study's results - has been identified in abstracts of studies focused on a variety of disorders from multiple fields of medicine.

OBJECTIVES

This study's primary objective was to evaluate the abstracts of systematic reviews and meta-analyses focused on the treatment of atopic dermatitis for the nine most severe forms of spin.

METHODS

We systematically searched Embase and MEDLINE for systematic reviews of atopic dermatitis therapies. Screening and data extraction occurred in a masked, duplicate fashion. Each included study was evaluated for the nine most severe types of spin and other study characteristics.

RESULTS

Our searches retrieved 2,456 studies, of which 113 were included for data extraction. Spin was found in 74.3% of our included studies (84/113). Spin type 6 occurred most frequently (68/113, 60.2%). Spin types 1, 2, and 9 were not identified. All industry-funded systematic reviews contained spin in their abstract. The presence of spin was not associated with any specific study characteristics, including the methodological quality of the study.

CONCLUSIONS

Severe forms of spin were found in the majority of abstracts for systematic reviews of atopic dermatitis treatments. Steps should be taken to prevent spin to improve the quality of reporting in abstracts.

Evaluation of misrepresented findings in the abstracts of acute respiratory distress syndrome randomized trials with nonsignificant primary endpoints

OBJECTIVE

We investigated the randomized controlled trials (RCTs) related to acute respiratory distress syndrome (ARDS) to assess the presentation and frequency of misrepresented research findings, also known as spin.

METHODS

We searched PubMed (MEDLINE) for studies published from January 1, 2011 to December 31, 2019. We included randomized controlled trials with an ARDS intervention and a nonsignificant primary endpoint. Trial screening and data extraction was performed on all studies independently and in duplicate. The primary endpoint was to investigate the frequency and manifestation of spin in RCT abstracts. Our secondary endpoint was to investigate associations between funding source and spin.

RESULTS

Our PubMed search returned 766 articles with 37 meeting inclusion criteria. Spin was present in 14 (14/37, 37.8%; 95% CI 22.5%-55.2%) abstracts. The most common manifestations of spin were claiming benefit based on a significant secondary endpoint (6/14, 42.9%), followed by the use of 'trend' statements, such as 'trend toward significance' (2/14, 14.3%; 95% CI 1.8%-42.8%). The most common spin in abstract conclusions was in the form of claiming benefit due to a significant secondary endpoint (3/4, 75%; 95% CI 19.4%-99.4%). Our secondary endpoint did not identify a significant difference in the prevalence of spin in publicly funded (5/19, 26.3%; 95% CI 9.1%-51.2%) compared to privately funded (4/12, 33.3%; 95% CI 9.9%-65.1%) studies (p>.05).

CONCLUSIONS

RCTs of ARDS interventions with nonsignificant primary endpoints often included spin in the abstract. Spin in the abstract may influence clinician appraisal and interpretation of diagnostic or treatment modalities.

Typology of Published Randomized Controlled Trials Investigating Initial Ventilation Strategy in Critically Ill Patients With Acute Respiratory Failure: A Methodologic Review

BACKGROUND

Randomized controlled trials (RCTs) in patients with hypoxemic acute respiratory failure (ARF) often failed to show survival benefits and resulted in varying clinical end points.

RESEARCH QUESTION

This methodologic review was conducted of published RCTs on ARF, with a careful attention to whether the study results were positive or negative.

STUDY DESIGN AND METHODS

MEDLINE, Cochrane Central Register of Controlled Trials, and Web of Science were searched for published RCTs in adult patients with ARF between January 1995 and December 2019. The objective was to investigate sources of heterogeneity and factors associated with a positive RCT (ie, with a significant difference on the primary end point). To determine the importance of the primary end point choice, end points were classified as follows: (1) respiratory event end points (corresponding to modification of the ventilation support); (2) physiologic/clinical end points (corresponding to oxygenation or clinical parameters); and (3) mortality. The Cochrane risk-of-bias tool was used to assess study quality.

RESULTS

Seventy-four RCTs were included (57% were single-center RCTs) comparing mainly oxygenation/ventilation strategies (95%) in patients with various ARF causes (62%); studies were stopped prematurely in 20% of the trials. A standardized management of ARF was observed in 24 (32%) trials. Twenty-two distinct primary end points have been used, the first of which were those based on respiratory events (44 RCTs [49%]), namely intubation in 76% of the cases. Physiologic/clinical end points have been used in 21 trials (29%) and mortality in nine (12%). Overall, 42 (57%) RCTs were positive, 52% in studies with respiratory event end points, 76% in studies with physiologic/clinical end points, and 33% in studies with mortality end points. Adjusted for study quality (Cochrane risk-of-bias tool), factors associated with a positive RCT included clinically based primary end points (OR, 8.40; 95% CI, 1.35-65.79), the use of standardized ARF management (OR, 4.55; 95% CI, 1.02-22.88), and single-center trials (OR, 3.85; 95% CI, 1.25-13.11).

INTERPRETATION

The typology of published RCTs in patients with ARF could be used to frame future trial designs in this field and guide clinicians and researchers toward optimal research transfer to the bedside.

Spin in the reporting, interpretation, and extrapolation of adverse effects of orthodontic interventions: protocol for a cross-sectional study of systematic reviews

Background

Titles and abstracts are the most read sections of biomedical papers. It is therefore important that abstracts transparently report both the beneficial and adverse effects of health care interventions and do not mislead the reader. Misleading reporting, interpretation, or extrapolation of study results is called “spin”. In this study, we will assess whether adverse effects of orthodontic interventions were reported or considered in the abstracts of both Cochrane and non-Cochrane reviews and whether spin was identified and what type of spin.

Methods

Eligibility criteria were defined for the type of study designs, participants, interventions, outcomes, and settings. We will include systematic reviews of clinical orthodontic interventions published in the five leading orthodontic journals and in the Cochrane Database. Empty reviews will be excluded. We will manually search eligible reviews published between 1 August 2009 and 31 July 2019. Data collection forms were developed a priori. All study selection and data extraction procedures will be conducted by two reviewers independently. Our main outcomes will be the prevalence of reported or considered adverse effects of orthodontic interventions in the abstract of systematic reviews and the prevalence of “spin” related to these adverse effects. We will also record the prevalence of three subtypes of spin, i.e., misleading reporting, misleading interpretation, and misleading extrapolation-related spin. All statistics will be calculated for the following groups: (1) all journals individually, (2) all journals together, and (3) the five leading orthodontic journals and the Cochrane Database of Systematic Reviews separately. Generalized linear models will be developed to compare the various groups.

Discussion

We expect that our results will raise the awareness of the importance of reporting and considering of adverse effects and the presence of the phenomenon of spin related to these effects in abstracts of systematic reviews of orthodontic interventions. This is important, because an incomplete and inadequate reporting, interpretation, or extrapolation of findings on adverse effects in abstracts of systematic reviews can mislead readers and could lead to inadequate clinical practice. Our findings could result in policy implications for making judgments about the acceptance for publication of systematic reviews of orthodontic interventions.

Measuring and Analyzing Length of Stay in Critical Care Trials

BACKGROUND

In randomized clinical trials among critically ill patients, it is uncertain how choices regarding the measurement and analysis of nonmortal outcomes measured in terms of duration, such as intensive care unit (ICU) length of stay (LOS), affect studies' conclusions.

OBJECTIVES

Assess the definitions and analytic methods used for ICU LOS analyses in published randomized clinical trials.

RESEARCH DESIGN

This is a systematic review and statistical simulation study.

RESULTS

Among the 80 of 150 trials providing sufficient information regarding the chosen definition of ICU LOS, 3 different start times (ICU admission, trial enrollment/randomization, receipt of intervention) and 2 end times (discharge readiness, actual discharge) were used. In roughly three quarters of these studies, ICU LOS was compared using approaches that did not explicitly account for death, either by ignoring it entirely or stratifying the analyses by survival status. The remaining studies used time-to-event (discharge) models censoring at death or applied a fixed LOS value to patients who died. In statistical simulations, we showed that each analytic approach tested a different question regarding ICU LOS, and that approaches that do not explicitly account for death often produce misleading or ambiguous conclusions when treatments produce small effects on mortality, even if those are not detected as significant in the trial.

CONCLUSIONS

There is considerable variability in how ICU LOS is measured and analyzed which impairs the ability to compare results across trials and can produce spurious conclusions. Analyses of duration-based outcomes such as LOS should jointly assess the impact of the intervention on mortality to yield correct interpretations.

Requirements for trial registration and adherence to reporting guidelines in critical care journals: a meta-epidemiological study of journals' instructions for authors

BACKGROUND

The purpose of this study was to investigate the policies of critical care journals with regard to guideline adoption and clinical trial registration to understand the extent to which journals use these mechanisms to improve reporting practices.

METHODS

The current study's sample comprised 37 critical care journals cataloged in the Expanded Science Citation Index of the 2015 Journal Citation Reports and Google Scholar Metrics h5-index critical care subcategory. A web-based data abstraction was performed to identify which journals required, recommended, or made no mention of 17 different reporting guidelines. We also extracted whether journals required or recommended trial registration. Authors were blinded to one another's ratings until completion of the data validation. Cross tabulations and descriptive statistics were calculated by using STATA 13.

RESULTS

Of the 37 critical care journals, 15 (15/37, 40.5%) did not mention a single guideline within their instructions for authors, whereas the remaining 22 (22/37, 59.5%) mentioned one or more guidelines. The Quality of Reporting of Meta-analyses statement and Standards for Reporting Qualitative Research were not mentioned by any journals, whereas the International Committee of Medical Journal Editors Uniform Requirements for Manuscripts (26/37, 70.3%) and Consolidated Standards of Reporting Trials statement (17/37, 45.9%) were mentioned most often. Of the 37 critical care journals, 21 (21/37, 56.8%) did not mention trial or review registration, but the remaining 16 (16/37, 43.2%) mentioned at least one of the two. Trial registration through ClinicalTrials.gov was mentioned by six (6/37, 16.2%) journals, whereas the WHO registry was mentioned by five (5/37, 13.5%). Sixteen (16/37, 43.2%) journals mentioned trial registration through a registry platform.

CONCLUSION

Nearly half of the journals in our sample did not mention a reporting guideline, and only a small percentage of journals required the registration of clinical trials as a condition for publication. Implementing these two mechanisms may limit bias, and their adoption should be considered by journal editors in critical care.

TRIAL REGISTRATION

UMIN000024081.

Comparison between a nurse-led weaning protocol and weaning based on physician's clinical judgment in tracheostomized critically ill patients: a pilot randomized controlled clinical trial

Background

Weaning protocols expedite extubation in mechanically ventilated patients, yet the literature investigating the application in tracheostomized patients remains scarce. The primary objective of this parallel randomized controlled pilot trial (RCT) was to assess the feasibility and safety of a nurse-led weaning protocol (protocol) compared to weaning based on physician’s clinical judgment (control) in tracheostomized critically ill patients.

Results

We enrolled 65 patients, 27 were in the protocol group and 38 in the control group. Of 27 patients in the protocol group, 1 (3.7%) died in the ICU, 24 (88.9%) were successfully weaned from tracheostomy, and 2 (7.4%) were transferred still on the ventilator. Of 38 patients in the control group, 2 (5.3%) died in the ICU, 22 (57.9%) were successfully weaned from tracheostomy, and 14 were transferred still on the ventilator (36.8%). Risk of being discharged from the ICU on the ventilator was higher in the control group (relative risk: 1.5, IC 95% 1.14–2.01). Concerning safety and feasibility, no patients were excluded after randomization. There was no crossover between the two study arms nor missing data, and no severe adverse event related to the study protocol application was recorded by the staff. Weaning time and rate of successful weaning were not different in the protocol group compared to the control group (long-rank test, p = 0.31 for MV duration, p = 0.45 for weaning time). Based on our results and assuming a 30% reduction of the weaning time for the protocol group, 280 patients would be needed for a RCT to establish efficacy.

Conclusions

In this pilot RCT we demonstrated that a nurse-led weaning protocol from tracheostomy was feasible and safe. A larger RCT is justified to assess efficacy.

Electronic supplementary material

The online version of this article (10.1186/s13613-018-0354-1) contains supplementary material, which is available to authorized users.

Interpretation of health news items reported with or without spin: protocol for a prospective meta-analysis of 16 randomised controlled trials

INTRODUCTION

We aim to compare the interpretation of health news items reported with or without spin. 'Spin' is defined as a misrepresentation of study results, regardless of motive (intentionally or unintentionally) that overemphasises the beneficial effects of the intervention and overstates safety compared with that shown by the results.

METHODS AND ANALYSIS

We have planned a series of 16 randomised controlled trials (RCTs) to perform a prospective meta-analysis. We will select a sample of health news items reporting the results of four types of study designs, evaluating the effect of pharmacological treatment and containing the highest amount of spin in the headline and text. News items reporting four types of studies will be included: (1) preclinical studies; (2) phase I/II (non-randomised) trials; (3) RCTs and (4) observational studies. We will rewrite the selected news items and remove the spin. The original news and rewritten news will be appraised by four types of populations: (1) French-speaking patients; (2) French-speaking general public; (3) English-speaking patients and (4) English-speaking general public. Each RCT will explore the interpretation of news items reporting one of the four study designs by each type of population and will include a sample size of 300 participants. The primary outcome will be participants' interpretation of the benefit of treatment after reading the news items: (What do you think is the probability that treatment X would be beneficial to patients? (scale, 0 (very unlikely) to 10 (very likely)).This study will evaluate the impact of spin on the interpretation of health news reporting results of studies by patients and the general public.

ETHICS AND DISSEMINATION

This study has obtained ethics approval from the Institutional Review Board of the Institut national de la santé et de la recherche médicale (INSERM) (registration no: IRB00003888). The description of all the steps and the results of this prospective meta-analysis will be available online and will be disseminated as a published article. On the completion of this study, the results will be sent to all participants.

PROSPERO REGISTRATION NUMBER

CRD42017058941.

'Spin' in published biomedical literature: A methodological systematic review

In the scientific literature, spin refers to reporting practices that distort the interpretation of results and mislead readers so that results are viewed in a more favourable light. The presence of spin in biomedical research can negatively impact the development of further studies, clinical practice, and health policies. This systematic review aims to explore the nature and prevalence of spin in the biomedical literature. We searched MEDLINE, PreMEDLINE, Embase, Scopus, and hand searched reference lists for all reports that included the measurement of spin in the biomedical literature for at least 1 outcome. Two independent coders extracted data on the characteristics of reports and their included studies and all spin-related outcomes. Results were grouped inductively into themes by spin-related outcome and are presented as a narrative synthesis. We used meta-analyses to analyse the association of spin with industry sponsorship of research. We included 35 reports, which investigated spin in clinical trials, observational studies, diagnostic accuracy studies, systematic reviews, and meta-analyses. The nature of spin varied according to study design. The highest (but also greatest) variability in the prevalence of spin was present in trials. Some of the common practices used to spin results included detracting from statistically nonsignificant results and inappropriately using causal language. Source of funding was hypothesised by a few authors to be a factor associated with spin; however, results were inconclusive, possibly due to the heterogeneity of the included papers. Further research is needed to assess the impact of spin on readers’ decision-making. Editors and peer reviewers should be familiar with the prevalence and manifestations of spin in their area of research in order to ensure accurate interpretation and dissemination of research.

In the scientific literature, spin refers to reporting practices that distort the interpretation of results and mislead readers so that results are viewed in a more favourable light. The presence of spin in biomedical research can negatively impact the development of further studies, clinical practice, and health policies. We conducted a systematic review to explore the nature and prevalence of spin in the biomedical literature. We included 35 reports, which investigated spin in clinical trials, observational studies, diagnostic accuracy studies, systematic reviews, and meta-analyses. The nature of spin varied according to study design. The highest (but also greatest) variability in the prevalence of spin was present in trials. Some of the common practices used to spin results included detracting from statistically nonsignificant results and inappropriately using causal language. Source of funding was hypothesised by a few authors to be a factor associated with spin; however, results were inconclusive, possibly due to the heterogeneity of the included papers. Further research is needed to assess the impact of spin on readers’ decision-making. Editors and peer reviewers should be familiar with the prevalence and manifestations of spin in their area of research in order to ensure accurate interpretation and dissemination of research.

Reporting of Randomized Controlled Trials With Statistically Nonsignificant Primary Outcomes Published in High-impact Surgical Journals

OBJECTIVE

To determine the nature and frequency of distorted presentation or "spin" (ie, specific reporting strategies which highlight that the experimental treatment is beneficial, despite a statistically nonsignificant difference for the primary outcome, or distract the reader from statistically nonsignificant results) in published reports of randomized controlled trials (RCTs) with statistically nonsignificant results for primary outcomes in surgical journals.

BACKGROUND

Multiple reports have suggested that interpretation of RCT results in medical journals can be distorted by authors of published reports.

METHODS

Using a defined search strategy, RCTs with clearly nonsignificant results for the primary outcome (P > 0.05) form 10 high-impact factor surgical journals (Annals of Surgery, Journal of Neurology, Neurosurgery and Psychiatry, Journal of Heart and Lung Transplantation, American Journal of Transplantation, British Journal of Surgery, Journal of Bone and Joint Surgery, Journal of the American College of Surgeons, Endoscopy, Archives of Surgery, and Liver transplantation), published between July 2013 to July 2015, were identified. Two reviewers independently appraised each selected article using a validated, standardized data abstraction form.

RESULTS

In all, 110 eligible RCTs with nonsignificant primary outcomes were appraised. The title was reported with spin in 8 (7%) articles. Forty-four (40%) included abstracts and 39 (35%) main texts were classified as having spin in at least 1 section. The level of spin was high in 16 (14%) abstract and 19 (19%) main-text "Conclusions" sections. Twenty-five articles (23%) recommended the intervention of interest despite a nonsignificant primary outcome. There was no relationship between trial funding source, use of statistician and article section, and the presence of spin.

CONCLUSIONS

In RCTs with statistically nonsignificant primary outcomes published in surgical journals, the reporting and interpretation of findings was frequently inconsistent with the results.

Overinterpretation of Research Findings: Evidence of "Spin" in Systematic Reviews of Diagnostic Accuracy Studies

BACKGROUND

We wished to assess the frequency of overinterpretation in systematic reviews of diagnostic accuracy studies.

METHODS

MEDLINE was searched through PubMed from December 2015 to January 2016. Systematic reviews of diagnostic accuracy studies in English were included if they reported one or more metaanalyses of accuracy estimates. We built and piloted a list of 10 items that represent actual overinterpretation in the abstract and/or full-text conclusion, and a list of 9 items that represent potential overinterpretation. Two investigators independently used the items to score each included systematic review, with disagreements resolved by consensus.

RESULTS

We included 112 systematic reviews. The majority had a positive conclusion regarding the accuracy or clinical usefulness of the investigated test in the abstract (n = 83; 74%) and full-text (n = 83; 74%). Of the 112 reviews, 81 (72%) contained at least 1 actual form of overinterpretation in the abstract, and 77 (69%) in the full-text. This was most often a "positive conclusion, not reflecting the reported summary accuracy estimates," in 55 (49%) abstracts and 56 (50%) full-texts and a "positive conclusion, not taking high risk of bias and/or applicability concerns into account," in 47 abstracts (42%) and 26 full-texts (23%). Of these 112 reviews, 107 (96%) contained a form of potential overinterpretation, most frequently "nonrecommended statistical methods for metaanalysis performed" (n = 57; 51%).

CONCLUSIONS

Most recent systematic reviews of diagnostic accuracy studies present positive conclusions and a majority contain a form of overinterpretation. This may lead to unjustified optimism about test performance and erroneous clinical decisions and recommendations.

Clinical trials in palliative care: a systematic review of their methodological characteristics and of the quality of their reporting

Background

Over the past decades there has been a significant increase in the number of published clinical trials in palliative care. However, empirical evidence suggests that there are methodological problems in the design and conduct of studies, which raises questions about the validity and generalisability of the results and of the strength of the available evidence. We sought to evaluate the methodological characteristics and assess the quality of reporting of clinical trials in palliative care.

Methods

We performed a systematic review of published clinical trials assessing therapeutic interventions in palliative care. Trials were identified using MEDLINE (from its inception to February 2015). We assessed methodological characteristics and describe the quality of reporting using the Cochrane Risk of Bias tool.

Results

We retrieved 107 studies. The most common medical field studied was oncology, and 43.9% of trials evaluated pharmacological interventions. Symptom control and physical dimensions (e.g. intervention on pain, breathlessness, nausea) were the palliative care-specific issues most studied. We found under-reporting of key information in particular on random sequence generation, allocation concealment, and blinding.

Conclusions

While the number of clinical trials in palliative care has increased over time, methodological quality remains suboptimal. This compromises the quality of studies. Therefore, a greater effort is needed to enable the appropriate performance of future studies and increase the robustness of evidence-based medicine in this important field.

Electronic supplementary material

The online version of this article (doi:10.1186/s12904-016-0181-9) contains supplementary material, which is available to authorized users.

Outcome Reporting Bias in Government-Sponsored Policy Evaluations: A Qualitative Content Analysis of 13 Studies

The reporting of evaluation outcomes can be a point of contention between evaluators and policy-makers when a given reform fails to fulfil its promises. Whereas evaluators are required to report outcomes in full, policy-makers have a vested interest in framing these outcomes in a positive light–especially when they previously expressed a commitment to the reform. The current evidence base is limited to a survey of policy evaluators, a study on reporting bias in education research and several studies investigating the influence of industry sponsorship on the reporting of clinical trials. The objective of this study was twofold. Firstly, it aimed to assess the risk of outcome reporting bias (ORB or ‘spin’) in pilot evaluation reports, using seven indicators developed by clinicians. Secondly, it sought to examine how the government’s commitment to a given reform may affect the level of ORB found in the corresponding evaluation report. To answer these questions, 13 evaluation reports were content-analysed, all of which found a non-significant effect of the intervention on its stated primary outcome. These reports were systematically selected from a dataset of 233 pilot and experimental evaluations spanning three policy areas and 13 years of government-commissioned research in the UK. The results show that the risk of ORB is real. Indeed, all studies reviewed here resorted to at least one of the presentational strategies associated with a risk of spin. This study also found a small, negative association between the seniority of the reform’s champion and the risk of ORB in the evaluation of that reform. The publication of protocols and the use of reporting guidelines are recommended.

Event-rate and delta inflation when evaluating mortality as a primary outcome from randomized controlled trials of nutritional interventions during critical illness: a systematic review

BACKGROUND

There is a lack of high-quality evidence that proves that nutritional interventions during critical illness reduce mortality.

OBJECTIVES

We evaluated whether power calculations for randomized controlled trials (RCTs) of nutritional interventions that used mortality as the primary outcome were realistic, and whether overestimation was systematic in the studies identified to determine whether this was due to overestimates of event rate or delta.

DESIGN

A systematic review of the literature between 2005 and 2015 was performed to identify RCTs of nutritional interventions administered to critically ill adults that had mortality as the primary outcome. Predicted event rate (predicted mortality during the control), predicted mortality during intervention, predicted delta (predicted difference between mortality during the control and intervention), actual event rate (observed mortality during control), observed mortality during intervention, and actual delta (difference between observed mortality during the control and intervention) were recorded. The event-rate gap (predicted event rate minus observed event rate), the delta gap (predicted delta minus observed delta), and the predicted number needed to treat were calculated. Data are shown as median (range).

RESULTS

Fourteen articles were extracted, with power calculations provided for 10 studies. The predicted event rate was 29.9% (20.0–52.4%), and the predicted delta was 7.9% (3.0–20.0%). If the study hypothesis was proven correct then, on the basis of the power calculations, the number needed to treat would have been 12.7 (5.0–33.3) patients. The actual event rate was 25.3% (6.1–50.0%), the observed mortality during the intervention was 24.4% (6.3–39.7%), and the actual delta was 0.5% (−10.2–10.3%), such that the event-rate gap was 2.6% (−3.9–23.7%) and delta gap was 7.5% (3.2–25.2%).

CONCLUSIONS

Overestimates of delta occur frequently in RCTs of nutritional interventions in the critically ill that are powered to determine a mortality benefit. Delta inflation may explain the number of "negative" studies in this field of research.

Interpretation of Results of Studies Evaluating an Intervention Highlighted in Google Health News: A Cross-Sectional Study of News

BACKGROUND

Mass media through the Internet is a powerful means of disseminating medical research. We aimed to determine whether and how the interpretation of research results is misrepresented by the use of "spin" in the health section of Google News. Spin was defined as specific way of reporting, from whatever motive (intentional or unintentional), to emphasize that the beneficial effect of the intervention is greater than that shown by the results.

METHODS

We conducted a cross-sectional study of news highlighted in the health section of US, UK and Canada editions of Google News between July 2013 and January 2014. We searched for news items for 3 days a week (i.e., Monday, Wednesday, and Friday) during 6 months and selected a sample of 130 news items reporting a scientific article evaluating the effect of an intervention on human health.

RESULTS

In total, 78% of the news did not provide a full reference or electronic link to the scientific article. We found at least one spin in 114 (88%) news items and 18 different types of spin in news. These spin were mainly related to misleading reporting (59%) such as not reporting adverse events that were reported in the scientific article (25%), misleading interpretation (69%) such as claiming a causal effect despite non-randomized study design (49%) and overgeneralization/misleading extrapolation (41%) of the results such as extrapolating a beneficial effect from an animal study to humans (21%). We also identified some new types of spin such as highlighting a single patient experience for the success of a new treatment instead of focusing on the group results.

CONCLUSIONS

Interpretation of research results was frequently misrepresented in the health section of Google News. However, we do not know whether these spin were from the scientific articles themselves or added in the news.

Findings from a novel approach to publication guideline revision: user road testing of a draft version of SQUIRE 2.0

Background

The Standards for Quality Improvement Reporting Excellence (SQUIRE) Guideline was published in 2008 (SQUIRE 1.0) and was the first publication guideline specifically designed to advance the science of healthcare improvement. Advances in the discipline of improvement prompted us to revise it. We adopted a novel approach to the revision by asking end-users to ‘road test’ a draft version of SQUIRE 2.0. The aim was to determine whether they understood and implemented the guidelines as intended by the developers.

Methods

Forty-four participants were assigned a manuscript section (ie, introduction, methods, results, discussion) and asked to use the draft Guidelines to guide their writing process. They indicated the text that corresponded to each SQUIRE item used and submitted it along with a confidential survey. The survey examined usability of the Guidelines using Likert-scaled questions and participants’ interpretation of key concepts in SQUIRE using open-ended questions. On the submitted text, we evaluated concordance between participants’ item usage/interpretation and the developers’ intended application. For the survey, the Likert-scaled responses were summarised using descriptive statistics and the open-ended questions were analysed by content analysis.

Results

Consistent with the SQUIRE Guidelines’ recommendation that not every item be included, less than one-third (n=14) of participants applied every item in their section in full. Of the 85 instances when an item was partially used or was omitted, only 7 (8.2%) of these instances were due to participants not understanding the item. Usage of Guideline items was highest for items most similar to standard scientific reporting (ie, ‘Specific aim of the improvement’ (introduction), ‘Description of the improvement’ (methods) and ‘Implications for further studies’ (discussion)) and lowest (<20% of the time) for those unique to healthcare improvement (ie, ‘Assessment methods for context factors that contributed to success or failure’ and ‘Costs and strategic trade-offs’). Items unique to healthcare improvement, specifically ‘Evolution of the improvement’, ‘Context elements that influenced the improvement’, ‘The logic on which the improvement was based’, ‘Process and outcome measures’, demonstrated poor concordance between participants’ interpretation and developers’ intended application.

Conclusions

User testing of a draft version of SQUIRE 2.0 revealed which items have poor concordance between developer intent and author usage, which will inform final editing of the Guideline and development of supporting supplementary materials. It also identified the items that require special attention when teaching about scholarly writing in healthcare improvement.

CONSORT item reporting quality in the top ten ranked journals of critical care medicine in 2011: a retrospective analysis

INTRODUCTION

Reporting randomised controlled trials is a key element in order to disseminate research findings. The CONSORT statement was introduced to improve the reporting quality. We assessed the adherence to the CONSORT statement of randomised controlled trials published 2011 in the top ten ranked journals of critical care medicine (ISI Web of Knowledge 2011, Thomson Reuters, London UK).

METHODS

Design. We performed a retrospective cross sectional data analysis. Setting. This study was executed at the University Hospital of RWTH, Aachen. Participants. We selected the following top ten listed journals according to ISI Web of Knowledge (Thomson Reuters, London, UK) critical care medicine ranking in the year 2011: American Journal of Respiratory and Critical Care Medicine, Critical Care Medicine, Intensive Care Medicine, CHEST, Critical Care, Journal of Neurotrauma, Resuscitation, Pediatric Critical Care Medicine, Shock and Minerva Anestesiologica. Main outcome measures. We screened the online table of contents of each included journal, to identify the randomised controlled trials. The adherence to the items of the CONSORT Checklist in each trial was evaluated. Additionally we correlated the citation frequency of the articles and the impact factor of the respective journal with the amount of reported items per trial.

RESULTS

We analysed 119 randomised controlled trials and found, 15 years after the implementation of the CONSORT statement, that a median of 61,1% of the checklist-items were reported. Only 55.5% of the articles were identified as randomised trials in their titles. The citation frequency of the trials correlated significantly (rs = 0,433; p<0,001 and r = 0,331; p<0,001) to the CONSORT statement adherence. The impact factor showed also a significant correlation to the CONSORT adherence (r = 0,386; p<0,001).

CONCLUSION

The reporting quality of randomised controlled trials in the field of critical care medicine remains poor and needs considerable improvement.

A systematic review and analysis of factors associated with methodological quality in laparoscopic randomized controlled trials

Several methods for assessment of methodological quality in randomized controlled trials (RCTs) have been developed during the past few years. Factors associated with quality in laparoscopic surgery have not been defined till date. The aim of this study was to investigate the relationship between bibliometric and the methodological quality of laparoscopic RCTs. The PubMed search engine was queried to identify RCTs on minimally invasive surgery published in 2012 in the 10 highest impact factor surgery journals and the 5 highest impact factor laparoscopic journals. Eligible studies were blindly assessed by two independent investigators using the Scottish Intercollegiate Guidelines Network (SIGN) tool for RCTs. Univariate and multivariate analyses were performed to identify potential associations with methodological quality. A total of 114 relevant RCTs were identified. More than half of the trials were of high or acceptable quality. Half of the reports provided information on comparative demo graphic data and only 21% performed intention-to-treat analysis. RCTs with sample size of at least 60 patients presented higher methodological quality (p = 0.025). Upon multiple regression, reporting on preoperative care and the experience level of surgeons were independent factors of quality.

Impact of spin in the abstracts of articles reporting results of randomized controlled trials in the field of cancer: the SPIIN randomized controlled trial

PURPOSE

We aimed to assess the impact of spin (ie, reporting to convince readers that the beneficial effect of the experimental treatment is greater than shown by the results) on the interpretation of results of abstracts of randomized controlled trials (RCTs) in the field of cancer.

METHODS

We performed a two-arm, parallel-group RCT. We selected a sample of published RCTs with statistically nonsignificant primary outcome and with spin in the abstract conclusion. Two versions of these abstracts were used-the original with spin and a rewritten version without spin. Participants were clinician corresponding authors of articles reporting RCTs, investigators of trials, and reviewers of French national grants. The primary outcome was clinicians' interpretation of the beneficial effect of the experimental treatment (0 to 10 scale). Participants were blinded to study hypothesis.

RESULTS

Three hundred clinicians were randomly assigned using a Web-based system; 150 clinicians assessed an abstract with spin and 150 assessed an abstract without spin. For abstracts with spin, the experimental treatment was rated as being more beneficial (mean difference, 0.71; 95% CI, 0.07 to 1.35; P = .030), the trial was rated as being less rigorous (mean difference, -0.59; 95% CI, -1.13 to 0.05; P = .034), and clinicians were more interested in reading the full-text article (mean difference, 0.77; 95% CI, 0.08 to 1.47; P = .029). There was no statistically significant difference in the clinicians' rating of the importance of the study or the need to run another trial.

CONCLUSION

Spin in abstracts can have an impact on clinicians' interpretation of the trial results.

Data interpretation in analgesic clinical trials with statistically nonsignificant primary analyses: an ACTTION systematic review

Peer-reviewed publications of randomized clinical trials (RCTs) are the primary means of disseminating research findings. "Spin" in RCT publications is misrepresentation of statistically nonsignificant research findings to suggest treatment benefit. Spin can influence the way readers interpret clinical trials and use the information to make decisions about treatments and medical policies. The objective of this study was to determine the frequency with which 4 types of spin were used in publications of analgesic RCTs with nonsignificant primary analyses in 6 major pain journals. In the 76 articles included in our sample, 28% of the abstracts and 29% of the main texts emphasized secondary analyses with P values <.05; 22% of abstracts and 29% of texts emphasized treatment benefit based on nonsignificant primary results; 14% of abstracts and 18% of texts emphasized within-group improvements over time, rather than primary between-group comparisons; and 13% of abstracts and 10% of texts interpreted a nonsignificant difference between groups in a superiority study as comparable effectiveness. When considering the article conclusion sections, 21% did not mention the nonsignificant primary result, 22% were presented with no uncertainty or qualification, 30% did not acknowledge that future research was required, and 8% recommended the intervention for clinical use.

PERSPECTIVE

This article identifies relatively frequent "spin" in analgesic RCTs. These findings highlight a need for authors, reviewers, and editors to be more cognizant of how analgesic RCT results are presented and attempt to minimize spin in future clinical trial publications.

Outcomes and statistical power in adult critical care randomized trials

RATIONALE

Intensive care unit (ICU)-based randomized clinical trials (RCTs) among adult critically ill patients commonly fail to detect treatment benefits.

OBJECTIVES

Appraise the rates of success, outcomes used, statistical power, and design characteristics of published trials.

METHODS

One hundred forty-six ICU-based RCTs of diagnostic, therapeutic, or process/systems interventions published from January 2007 to May 2013 in 16 high-impact general or critical care journals were studied.

MEASUREMENT AND MAIN RESULTS

Of 146 RCTs, 54 (37%) were positive (i.e., the a priori hypothesis was found to be statistically significant). The most common primary outcomes were mortality (n = 40 trials), infection-related outcomes (n = 33), and ventilation-related outcomes (n = 30), with positive results found in 10, 58, and 43%, respectively. Statistical power was discussed in 135 RCTs (92%); 92 cited a rationale for their power parameters. Twenty trials failed to achieve at least 95% of their reported target sample size, including 11 that were stopped early due to insufficient accrual/logistical issues. Of 34 superiority RCTs comparing mortality between treatment arms, 13 (38%) accrued a sample size large enough to find an absolute mortality reduction of 10% or less. In 22 of these trials the observed control-arm mortality rate differed from the predicted rate by at least 7.5%.

CONCLUSIONS

ICU-based RCTs are commonly negative and powered to identify what appear to be unrealistic treatment effects, particularly when using mortality as the primary outcome. Additional concerns include a lack of standardized methods for assessing common outcomes, unclear justifications for statistical power calculations, insufficient patient accrual, and incorrect predictions of baseline event rates.