Nonrandomized quality improvement intervention trials might overstate the strength of causal inference of their findings

High prevalence of spin was found in pharmacovigilance studies using disproportionality analyses to detect safety signals: a meta-epidemiological study

OBJECTIVE

To systematically review and appraise misinterpretation of pharmacovigilance disproportionality analysis results in published studies.

STUDY DESIGN AND SETTING

We randomly selected 100 studies that performed disproportionality analyses and indexed in Medline identified during a systematic literature search. Titles, abstracts and main texts (results, discussion and conclusion) were evaluated for spin independently by two reviewers. Spin in pharmacovigilance studies was classified according to three main categories: inappropriate interpretation, inappropriate extrapolations and misleading reporting.

RESULTS

Of the 100 studies evaluated, we found that 63%, 56% and 51% had at least one type of spin in their abstract, main text or conclusion respectively, and 40% used causal language to interpret their results in the abstract or conclusion. Spin in titles and results were exclusively represented by inappropriate interpretations of findings (12% and 21% respectively), with terms such as "risk of" or "risks associated with" or results erroneously presented as regular Odds Ratios. Spin in discussion sections mostly concerned inappropriate interpretations (38%)and misleading reporting (12%). Misleading reporting, notably failing to acknowledge the limitations of disproportionality analyses, was the most frequent type of spin in abstracts (55%) and conclusion sections (37%).

CONCLUSION

We found that spin is frequent in publications of pharmacovigilance disproportionality analyses, notably in abstracts. This consisted notably in an over-interpretation of the results suggesting a proven causative link between a drug use and the risk of an event.

The SSSPIN study-spin in studies of spin: meta-research analysis

OBJECTIVES

To identify and calculate the prevalence of spin in studies of spin.

DESIGN

Meta-research analysis (research on research).

SETTING

35 studies of spin in the scientific literature.

MAIN OUTCOME MEASURES

Spin, categorised as: reporting practices that distort the presentation and interpretation of results, creating misleading conclusions; discordance between results and their interpretation, with presentation of favourable conclusions that are not supported by the data or results; attribution of causality when study design does not support it; and over-interpretation or inappropriate extrapolation of results.

RESULTS

Five (14%) of 35 spin studies contained spin categorised as reporting practices that distort the presentation and interpretation of results (n=2) or categorised as over-interpretation or inappropriate extrapolation of results (n=3).

CONCLUSION

Spin occurs in research on spin. Although researchers on this topic should be sensitive to spinning their findings, our study does not undermine the need for rigorous interventions to reduce spin across various research fields.

CONCLUSION WITH SPIN

Our hypothesis that spin will be less prevalent in spin studies than in studies on other topics has been proven. Spin scholars are less likely to spin their conclusions than other researchers, and they should receive substantial resources to launch and test interventions to reduce spin and research waste in reporting.

E-learning for health professionals

BACKGROUND

The use of e-learning, defined as any educational intervention mediated electronically via the Internet, has steadily increased among health professionals worldwide. Several studies have attempted to measure the effects of e-learning in medical practice, which has often been associated with large positive effects when compared to no intervention and with small positive effects when compared with traditional learning (without access to e-learning). However, results are not conclusive.

OBJECTIVES

To assess the effects of e-learning programmes versus traditional learning in licensed health professionals for improving patient outcomes or health professionals' behaviours, skills and knowledge.

SEARCH METHODS

We searched CENTRAL, MEDLINE, Embase, five other databases and three trial registers up to July 2016, without any restrictions based on language or status of publication. We examined the reference lists of the included studies and other relevant reviews. If necessary, we contacted the study authors to collect additional information on studies.

SELECTION CRITERIA

Randomised trials assessing the effectiveness of e-learning versus traditional learning for health professionals. We excluded non-randomised trials and trials involving undergraduate health professionals.

DATA COLLECTION AND ANALYSIS

Two authors independently selected studies, extracted data and assessed risk of bias. We graded the certainty of evidence for each outcome using the GRADE approach and standardised the outcome effects using relative risks (risk ratio (RR) or odds ratio (OR)) or standardised mean difference (SMD) when possible.

MAIN RESULTS

We included 16 randomised trials involving 5679 licensed health professionals (4759 mixed health professionals, 587 nurses, 300 doctors and 33 childcare health consultants).When compared with traditional learning at 12-month follow-up, low-certainty evidence suggests that e-learning may make little or no difference for the following patient outcomes: the proportion of patients with low-density lipoprotein (LDL) cholesterol of less than 100 mg/dL (adjusted difference 4.0%, 95% confidence interval (CI) -0.3 to 7.9, N = 6399 patients, 1 study) and the proportion with glycated haemoglobin level of less than 8% (adjusted difference 4.6%, 95% CI -1.5 to 9.8, 3114 patients, 1 study). At 3- to 12-month follow-up, low-certainty evidence indicates that e-learning may make little or no difference on the following behaviours in health professionals: screening for dyslipidaemia (OR 0.90, 95% CI 0.77 to 1.06, 6027 patients, 2 studies) and treatment for dyslipidaemia (OR 1.15, 95% CI 0.89 to 1.48, 5491 patients, 2 studies). It is uncertain whether e-learning improves or reduces health professionals' skills (2912 health professionals; 6 studies; very low-certainty evidence), and it may make little or no difference in health professionals' knowledge (3236 participants; 11 studies; low-certainty evidence).Due to the paucity of studies and data, we were unable to explore differences in effects across different subgroups. Owing to poor reporting, we were unable to collect sufficient information to complete a meaningful 'Risk of bias' assessment for most of the quality criteria. We evaluated the risk of bias as unclear for most studies, but we classified the largest trial as being at low risk of bias. Missing data represented a potential source of bias in several studies.

AUTHORS' CONCLUSIONS

When compared to traditional learning, e-learning may make little or no difference in patient outcomes or health professionals' behaviours, skills or knowledge. Even if e-learning could be more successful than traditional learning in particular medical education settings, general claims of it as inherently more effective than traditional learning may be misleading.

'Spin' in published biomedical literature: A methodological systematic review

In the scientific literature, spin refers to reporting practices that distort the interpretation of results and mislead readers so that results are viewed in a more favourable light. The presence of spin in biomedical research can negatively impact the development of further studies, clinical practice, and health policies. This systematic review aims to explore the nature and prevalence of spin in the biomedical literature. We searched MEDLINE, PreMEDLINE, Embase, Scopus, and hand searched reference lists for all reports that included the measurement of spin in the biomedical literature for at least 1 outcome. Two independent coders extracted data on the characteristics of reports and their included studies and all spin-related outcomes. Results were grouped inductively into themes by spin-related outcome and are presented as a narrative synthesis. We used meta-analyses to analyse the association of spin with industry sponsorship of research. We included 35 reports, which investigated spin in clinical trials, observational studies, diagnostic accuracy studies, systematic reviews, and meta-analyses. The nature of spin varied according to study design. The highest (but also greatest) variability in the prevalence of spin was present in trials. Some of the common practices used to spin results included detracting from statistically nonsignificant results and inappropriately using causal language. Source of funding was hypothesised by a few authors to be a factor associated with spin; however, results were inconclusive, possibly due to the heterogeneity of the included papers. Further research is needed to assess the impact of spin on readers’ decision-making. Editors and peer reviewers should be familiar with the prevalence and manifestations of spin in their area of research in order to ensure accurate interpretation and dissemination of research.

In the scientific literature, spin refers to reporting practices that distort the interpretation of results and mislead readers so that results are viewed in a more favourable light. The presence of spin in biomedical research can negatively impact the development of further studies, clinical practice, and health policies. We conducted a systematic review to explore the nature and prevalence of spin in the biomedical literature. We included 35 reports, which investigated spin in clinical trials, observational studies, diagnostic accuracy studies, systematic reviews, and meta-analyses. The nature of spin varied according to study design. The highest (but also greatest) variability in the prevalence of spin was present in trials. Some of the common practices used to spin results included detracting from statistically nonsignificant results and inappropriately using causal language. Source of funding was hypothesised by a few authors to be a factor associated with spin; however, results were inconclusive, possibly due to the heterogeneity of the included papers. Further research is needed to assess the impact of spin on readers’ decision-making. Editors and peer reviewers should be familiar with the prevalence and manifestations of spin in their area of research in order to ensure accurate interpretation and dissemination of research.

Improving Adolescent Preventive Care in an Urban Pediatric Clinic: Capturing Missed Opportunities

PURPOSE

To increase the proportion of adolescents with Medicaid who receive preventive care services in an urban pediatric clinic.

METHODS

A quality improvement intervention was implemented at an urban pediatric primary care clinic between November 2013 and October 2014. The intervention systematically "flipped" acute visits into well-care visits for patients ages 12-21 years, when overdue. The primary process measure was the percentage of acute visits expanded to include well-care components out of total eligible opportunities. The primary outcome measure was adolescent well-care (AWC) completion in 2014 versus 2013 and 2012.

RESULTS

Among 857 adolescents with Medicaid, 124 additional AWC visits were completed by October 2014 compared to 2013 and 71 additional visits compared to 2012. The gap to achieving Healthcare Effectiveness Data and Information Set neutral zone targets for AWC was reduced by 59% compared to 2013 and by 54% compared to 2012. The mean proportion of eligible acute opportunities "flipped" monthly increased from 17% (range: 10%-21%) during the initial 3 months of implementation to 30% (range: 5%-50%) in the last 3 months.

CONCLUSIONS

Systematically flipping acute visits into well visits resulted in reaching Healthcare Effectiveness Data and Information Set quality targets for AWC, which had not previously been accomplished by this clinic. Incorporating staff and provider feedback strengthened intervention fidelity and buy-in despite time constraints in a busy, urban setting.

The Development of a Checklist to Enhance Methodological Quality in Intervention Programs

The methodological quality of primary studies is an important issue when performing meta-analyses or systematic reviews. Nevertheless, there are no clear criteria for how methodological quality should be analyzed. Controversies emerge when considering the various theoretical and empirical definitions, especially in relation to three interrelated problems: the lack of representativeness, utility, and feasibility. In this article, we (a) systematize and summarize the available literature about methodological quality in primary studies; (b) propose a specific, parsimonious, 12-items checklist to empirically define the methodological quality of primary studies based on a content validity study; and (c) present an inter-coder reliability study for the resulting 12-items. This paper provides a precise and rigorous description of the development of this checklist, highlighting the clearly specified criteria for the inclusion of items and a substantial inter-coder agreement in the different items. Rather than simply proposing another checklist, however, it then argues that the list constitutes an assessment tool with respect to the representativeness, utility, and feasibility of the most frequent methodological quality items in the literature, one that provides practitioners and researchers with clear criteria for choosing items that may be adequate to their needs. We propose individual methodological features as indicators of quality, arguing that these need to be taken into account when designing, implementing, or evaluating an intervention program. This enhances methodological quality of intervention programs and fosters the cumulative knowledge based on meta-analyses of these interventions. Future development of the checklist is discussed.

Classification and prevalence of spin in abstracts of non-randomized studies evaluating an intervention

Background

Spin represents specific reporting strategies, either intentional or unintentional, to convince the reader that the beneficial effect of the experimental intervention in terms of efficacy and safety is greater than that shown by the results. The objectives of this study were to 1) develop a classification of spin specific to non-randomized studies assessing an intervention and 2) estimate the prevalence of spin in abstracts of reports of such studies.

Methods

In a first step, we developed a specific classification of spin for non-randomized studies by a literature review and pilot study. In a second step, 2 researchers trained in the field of methodology evaluated the prevalence of spin in the abstract of all non-randomized studies assessing an intervention published in the BioMed Central Medical Series journals between January 1, 2011 and December 31, 2013. All disagreements were resolved by consensus. We also determined whether the level of spin in abstract conclusions was high (spin reported without uncertainty or recommendations for further trials), moderate (spin reported with some uncertainty or recommendations for further trials) or low (spin reported with uncertainty and recommendations for further trials).

Results

Among the 128 assessed articles assessed, 107 (84 %) had at least one example of spin in their abstract. The most prevalent strategy of spin was the use of causal language, identified in 68 (53 %) abstracts. Other frequent strategies were linguistic spin, inadequate implications for clinical practice, and lack of focus on harm, identified in 33 (26 %), 25 (20 %), and 34 (27 %) abstracts respectively. Abstract conclusions of 61 (48 %) articles featured a high level of spin.

Conclusion

Abstract of reports of non-randomized studies assessing an intervention frequently includes spin. Efforts to reduce the prevalence of spin in abstract for such studies are needed.

Electronic supplementary material

The online version of this article (doi:10.1186/s12874-015-0079-x) contains supplementary material, which is available to authorized users.

Multiple systematic reviews: methods for assessing discordances of results

BACKGROUND

The process of systematically reviewing research evidence is useful for collecting, assessing and summarizing results from multiple studies planned to answer the same clinical question. The term "systematic" implies that the process, besides being organized and complete, is transparent and fully reported to allow other independent researchers to replicate the results, and therefore come to the same conclusions. Hundreds of new systematic reviews are indexed every year. The growing number increases the likelihood of finding multiple and discordant results.

OBJECTIVES

To clarify the impact of multiple and discordant systematic reviews, we designed a program aimed at finding out: (a) how often different systematic reviews are done on the same subject; (b) how often different systematic reviews on the same topic give different results or conclusions; (c) which methods or interpretation characteristics can explain the differences in results or conclusions.

METHODS

This paper outlines the method used to explore the frequency and the causes of discordance among multiple systematic reviews on the same topic. These methods were then applied to a few medical fields as case studies.

CONCLUSION

This aim is particularly relevant for both clinicians and policy makers. Judgments about evidence and recommendation in health care are complex, and often rely on discordant results, especially when there are no empirical results to help serve as a guideline.

The uncontrolled clinical trial: scientific, ethical, and practical reasons for being

According to principles of clinical trial design, the demonstration of efficacy of a new treatment is based on comparing the response in the treated group with that of a control group receiving placebo or another active treatment. The need for a control group is also recommended by the major international institutions that govern the ethics and the practice of clinical research. Despite these principles and recommendations, inspection of a purposive sample of ongoing clinical trials listed in the NIH registry ( http://ClinicalTrials.gov ) reveals that as many as one-third of trials are uncontrolled. Since these trials were approved through a formal evaluation by ethics committees, the lack of adequate control was not perceived as a major deficiency in the study design. Most uncontrolled trials belong to the oncology/hematology area. If two extreme disease conditions for nature and progression are analyzed, such as acute myeloid leukemia (AML) and chronic heart failure (CHF), the difference in the prevalence of uncontrolled trials is very striking. The number of uncontrolled trials is only 13% in CHF, whereas it reaches 66% in the AML group. I believe that the underlying disease condition plays a primary role in orienting the design of the study: oncology and hematology may be fields in which uncontrolled studies are common, whereas in other fields, e.g., cardiology, this phenomenon can be reduced. Within the limitations of the selection process of the examined trials, the current analysis indicates that the clinical trial reality does not strictly follow experimental design theory and official recommendations.