1
|
Moore A, Karadag P, Fisher E, Crombez G, Straube S, Eccleston C. Narrative bias ("spin") is common in randomised trials and systematic reviews of cannabinoids for pain. Pain 2024; 165:1380-1390. [PMID: 38227560 DOI: 10.1097/j.pain.0000000000003140] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2023] [Accepted: 08/21/2023] [Indexed: 01/18/2024]
Abstract
ABSTRACT We define narrative bias as a tendency to interpret information as part of a larger story or pattern, regardless of whether the facts support the full narrative. Narrative bias in title and abstract means that results reported in the title and abstract of an article are done so in a way that could distort their interpretation and mislead readers who had not read the whole article. Narrative bias is often referred to as "spin." It is prevalent in abstracts of scientific papers and is impactful because abstracts are often the only part of an article read. We found no extant narrative bias instrument suitable for exploring both efficacy and safety statements in randomized trials and systematic reviews of pain. We constructed a 6-point instrument with clear instructions and tested it on randomised trials and systematic reviews of cannabinoids and cannabis-based medicines for pain, with updated searches to April 2021. The instrument detected moderate or severe narrative bias in the title and abstract of 24% (8 of 34) of randomised controlled trials and 17% (11 of 64) of systematic reviews; narrative bias for efficacy and safety occurred equally. There was no significant or meaningful association between narrative bias and study characteristics in correlation or cluster analyses. Bias was always in favour of the experimental cannabinoid or cannabis-based medicine. Put simply, reading title and abstract only could give an incorrect impression of efficacy or safety in about 1 in 5 papers reporting on these products.
Collapse
Affiliation(s)
- Andrew Moore
- Court Road, Newton Ferrers, Plymouth, United Kingdom
| | - Paige Karadag
- Department of Psychology at the University of Bath, Claverton Down, Bath, United Kingdom
- School of Health, Science and Wellbeing, Staffordshire University, College Road, University Quarter, Stoke on Trent, Staffordshire, United Kingdom
| | - Emma Fisher
- Centre for Pain Research, University of Bath, Claverton Down, Bath, United Kingdom
| | - Geert Crombez
- Department of Experimental-Clinical and Health Psychology, Faculty of Psychology and Educational Sciences, Ghent University, Ghent, Belgium
| | - Sebastian Straube
- Division of Preventive Medicine, Department of Medicine, University of Alberta, Edmonton, AB, Canada
| | - Christopher Eccleston
- Centre for Pain Research, University of Bath, Claverton Down, Bath, United Kingdom
- Department of Experimental-Clinical and Health Psychology, Ghent University, Ghent, Belgium
- Department of Psychology, The University of Helsinki, Helsinki, Finland
| |
Collapse
|
2
|
Halvorson RT, Wong LH, Feeley BT. Increasing Use of Promotional Language in Orthopaedic Surgery Abstracts-An Analysis of 112,916 Abstracts 1985 to 2020. J Am Acad Orthop Surg Glob Res Rev 2024; 8:01979360-202405000-00017. [PMID: 38775596 PMCID: PMC11111393 DOI: 10.5435/jaaosglobal-d-24-00109] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2024] [Accepted: 03/23/2024] [Indexed: 05/25/2024]
Abstract
BACKGROUND Increasing use of "hype" language (eg, language overstating research impact) has been documented in the scientific community. Evaluating language in abstracts is important because readers may use abstracts to extrapolate findings to entire publications. Our purpose was to assess the frequency of hype language within orthopaedic surgery. METHODS One hundred thirty-nine hype adjectives were previously identified using a linguistics approach. All publicly available abstracts from 18 orthopaedic surgery journals between 1985 and 2020 were obtained, and hype adjectives were tabulated. Change in frequency of these adjectives was calculated. RESULTS A total of 112,916 abstracts were identified. 67.0% (948/1414) of abstracts in 1985 contained hype adjectives, compared with 92.5% (5287/5714) in 2020. The average number of hype adjectives per abstract increased by 136% (1.1 to 2.6). Of the 139 adjectives, 87 (62.5%) increased in frequency and 40 (28.7%) decreased in frequency while 12 (9%) were not used. The hype adjectives with the largest absolute increases in frequency were quality (+324wpm), significant (+320wpm), systematic (+246wpm), top (+239wpm), and international (+201wpm). The five hype adjectives with the largest relative increases in frequency were novel (+10500%), international (+2850%), urgent (+2600%), robust (+2300%), and emerging (+1400%). CONCLUSION Promotional language is increasing in orthopaedic surgery abstracts. Authors, editors, and reviewers should seek to minimize the usage of nonobjective language.
Collapse
Affiliation(s)
- Ryan T Halvorson
- From the Department of Orthopaedic Surgery, University of California San Francisco, San Francisco, CA
| | | | | |
Collapse
|
3
|
Guo F, Zhao T, Zhai Q, Fang X, Yue H, Hua F, He H. 'Spin' among abstracts of randomised controlled trials in sleep medicine: A research-on-research study. Sleep 2023; 46:zsad041. [PMID: 36861330 DOI: 10.1093/sleep/zsad041] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/28/2022] [Indexed: 03/03/2023] Open
Abstract
STUDY OBJECTIVES 'Spin', using reporting strategies to distort study results, can mislead readers of medical research. This study aimed to evaluate the prevalence and characteristics of 'spin' among randomised controlled trial (RCT) abstracts published in sleep medicine journals, and to identify factors associated with its presence and severity. METHODS The search for RCTs published between 2010 and 2020 were conducted in seven reputable journals of sleep medicine. Abstracts of RCTs with statistically nonsignificant primary outcomes were included and analysed for 'spin', according to pre-determined 'spin' strategies. Chi-square tests or logistic regression analyses were performed to detect the association between characteristics of included abstracts and the presence and severity of 'spin'. RESULTS A total of 114 RCT abstracts were included in this study, of which 89 (78.1%) were identified as having at least one type of 'spin' strategy. Sixty-six abstracts (57.9%) had 'spin' in the Results section, 82 (71.9%) abstracts presented with 'spin' in the Conclusions section. The presence of 'spin' varied significantly among RCTs based on the different categories of research area (P=0.047) and the statistician involvement (P=0.045). Furthermore, research area (P=0.019) and funding status (P=0.033) were significant factors associated with the severity of 'spin'. CONCLUSIONS The prevalence of 'spin' is high among RCT abstracts in sleep medicine. This raise the need for researchers, editors and other stakeholders to be aware of the issue of 'spin' and make joint efforts to eliminate it in future publications.
Collapse
Affiliation(s)
- Feiyang Guo
- Hubei-MOST KLOS and KLOBM, School and Hospital of Stomatology, Wuhan University, Wuhan, China
- Department of Orthodontics, School and Hospital of Stomatology, Wuhan University, Wuhan, China
| | - Tingting Zhao
- Hubei-MOST KLOS and KLOBM, School and Hospital of Stomatology, Wuhan University, Wuhan, China
- Department of Orthodontics, School and Hospital of Stomatology, Wuhan University, Wuhan, China
- Center for Dentofacial Development and Sleep Medicine, School and Hospital of Stomatology, Wuhan University, Wuhan, China
| | - Qianglan Zhai
- Hubei-MOST KLOS and KLOBM, School and Hospital of Stomatology, Wuhan University, Wuhan, China
| | - Xiaolin Fang
- Hubei-MOST KLOS and KLOBM, School and Hospital of Stomatology, Wuhan University, Wuhan, China
| | - Haoze Yue
- Department of Epidemiology and Public Health, University College London, London, UK
| | - Fang Hua
- Center for Dentofacial Development and Sleep Medicine, School and Hospital of Stomatology, Wuhan University, Wuhan, China
- Center for Evidence-Based Stomatology, School and Hospital of Stomatology, Wuhan University, Wuhan, China
- Center for Orthodontics and Pediatric Dentistry, Optics Valley Branch, School and Hospital of Stomatology, Wuhan University, Wuhan, China
- Division of Dentistry, School of Medical Sciences, Faculty of Biology, Medicine and Health, University of Manchester, Manchester, UK
| | - Hong He
- Hubei-MOST KLOS and KLOBM, School and Hospital of Stomatology, Wuhan University, Wuhan, China
- Department of Orthodontics, School and Hospital of Stomatology, Wuhan University, Wuhan, China
- Center for Dentofacial Development and Sleep Medicine, School and Hospital of Stomatology, Wuhan University, Wuhan, China
| |
Collapse
|
4
|
Xie S, Mi C. Promotion and caution in research article abstracts: The use of positive, negative and hedge words across disciplines and rankings. LEARNED PUBLISHING 2023. [DOI: 10.1002/leap.1515] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/16/2023]
Affiliation(s)
- Shaoliang Xie
- Foreign Language and Literature Institute, Xi'an International Studies University Xi'an China
| | - Chenggang Mi
- Foreign Language and Literature Institute, Xi'an International Studies University Xi'an China
| |
Collapse
|
5
|
Bowcut J, Levi L, Livnah O, Ross JS, Knable M, Davidson M, Davis JM, Weiser M. Misreporting of Results of Research in Psychiatry. Schizophr Bull 2021; 47:1254-1260. [PMID: 33860793 PMCID: PMC8379531 DOI: 10.1093/schbul/sbab040] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/13/2022]
Abstract
Few studies address publication and outcome reporting biases of randomized controlled trials (RCTs) in psychiatry. The objective of this study was to determine publication and outcome reporting bias in RCTs funded by the Stanley Medical Research Institute (SMRI), a U.S. based, non-profit organization funding RCTs in schizophrenia and bipolar disorder. We identified all RCTs (n = 280) funded by SMRI between 2000 and 2011, and using non-public, final study reports and published manuscripts, we classified the results as positive or negative in terms of the drug compared to placebo. Design, outcome measures and statistical methods specified in the original protocol were compared to the published manuscript. Of 280 RCTs funded by SMRI between 2000 and 2011, at the time of this writing, three RCTs were ongoing and 39 were not performed. Among the 238 completed RCTs, 86 (36.1%) reported positive and 152 (63.9%) reported negative results: 86% (74/86) of those with positive findings were published in contrast to 53% (80/152) of those with negative findings (P < .001). In 70% of the manuscripts published, there were major discrepancies between the published manuscript and the original RCT protocol (change in the primary outcome measure or statistics, change in a number of patient groups, 25% or more reduction in sample size). We conclude that publication bias and outcome reporting bias is common in papers reporting RCTs in schizophrenia and bipolar disorder. These data have major implications regarding the validity of the reports of clinical trials published in the literature.
Collapse
Affiliation(s)
- Jana Bowcut
- Stanley Medical Research Institute, Kensington, MD, USA
| | - Linda Levi
- Department of Psychiatry, Sheba Medical Center, Tel Hashomer,Israel
| | - Ortal Livnah
- Department of Psychiatry, Sheba Medical Center, Tel Hashomer,Israel
| | - Joseph S Ross
- Department of Internal Medicine, Yale University School of Medicine; Department of Health Policy and Management, Yale University School of Public Health; and the Center for Outcomes Research and Evaluation, Yale-New Haven Hospital, New Haven, CT,USA
| | | | | | - John M Davis
- Department of Psychiatry, University of Illinois, Chicago, IL,USA
| | - Mark Weiser
- Stanley Medical Research Institute, Kensington, MD, USA
- Department of Psychiatry, Sheba Medical Center, Tel Hashomer,Israel
- Sackler School of Medicine, Tel Aviv University, Tel Aviv,Israel
| |
Collapse
|
6
|
Bordignon F, Ermakova L, Noel M. Over-promotion and caution in abstracts of preprints during the COVID-19 crisis. LEARNED PUBLISHING 2021; 34:622-636. [PMID: 34539096 PMCID: PMC8441756 DOI: 10.1002/leap.1411] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2021] [Revised: 05/16/2021] [Accepted: 05/29/2021] [Indexed: 01/18/2023]
Abstract
The abstract is known to be a promotional genre where researchers tend to exaggerate the benefit of their research and use a promotional discourse to catch the reader's attention. The COVID-19 pandemic has prompted intensive research and has changed traditional publishing with the massive adoption of preprints by researchers. Our aim is to investigate whether the crisis and the ensuing scientific and economic competition have changed the lexical content of abstracts. We propose a comparative study of abstracts associated with preprints issued in response to the pandemic relative to abstracts produced during the closest pre-pandemic period. We show that with the increase (on average and in percentage) of positive words (especially effective) and the slight decrease of negative words, there is a strong increase in hedge words (the most frequent of which are the modal verbs can and may). Hedge words counterbalance the excessive use of positive words and thus invite the readers, who go probably beyond the 'usual' audience, to be cautious with the obtained results. The abstracts of preprints urgently produced in response to the COVID-19 crisis stand between uncertainty and over-promotion, illustrating the balance that authors have to achieve between promoting their results and appealing for caution.
Collapse
Affiliation(s)
| | | | - Marianne Noel
- LISIS, CNRS, INRAEUniversité Gustave EiffelMarne‐La‐ValléeFrance
| |
Collapse
|
7
|
Lawson DO, Leenus A, Mbuagbaw L. Mapping the nomenclature, methodology, and reporting of studies that review methods: a pilot methodological review. Pilot Feasibility Stud 2020; 6:13. [PMID: 32699641 PMCID: PMC7003412 DOI: 10.1186/s40814-019-0544-0] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/24/2019] [Accepted: 12/20/2019] [Indexed: 02/06/2023] Open
Abstract
BACKGROUND A relatively novel method of appraisal, methodological reviews (MRs) are used to synthesize information on the methods used in health research. There are currently no guidelines available to inform the reporting of MRs. OBJECTIVES This pilot review aimed to determine the feasibility of a full review and the need for reporting guidance for methodological reviews. METHODS Search strategy: We conducted a search of PubMed, restricted to 2017 to include the most recently published studies, using different search terms often used to describe methodological reviews: "literature survey" OR "meta-epidemiologic* review" OR "meta-epidemiologic* survey" OR "methodologic* review" OR "methodologic* survey" OR "systematic survey."Data extraction: Study characteristics including country, nomenclature, number of included studies, search strategy, a priori protocol use, and sampling methods were extracted in duplicate and summarized.Outcomes: Primary feasibility outcomes were the sensitivity and specificity of the search terms (criteria for success of feasibility set at sensitivity and specificity of ≥ 70%).Analysis: The estimates are reported as a point estimate (95% confidence interval). RESULTS Two hundred thirty-six articles were retrieved and 31 were included in the final analysis. The most accurate search term was "meta-epidemiological" (sensitivity [Sn] 48.39; 95% CI 31.97-65.16; specificity [Sp] 97.56; 94.42-98.95). The majority of studies were published by authors from Canada (n = 12, 38.7%), and Japan and USA (n = 4, 12.9% each). The median (interquartile range [IQR]) number of included studies in the MRs was 77 (13-1127). Reporting of a search strategy was done in most studies (n = 23, 74.2%). The use of a pre-published protocol (n = 7, 22.6%) or a justifiable sampling method (n = 5, 16.1%) occurred rarely. CONCLUSIONS Using the MR nomenclature identified, it is feasible to build a comprehensive search strategy and conduct a full review. Given the variation in reporting practices and nomenclature attributed to MRs, there is a need for guidance on standardized and transparent reporting of MRs. Future guideline development would likely include stakeholders from Canada, USA, and Japan.
Collapse
Affiliation(s)
- Daeria O. Lawson
- Department of Health Research Methods, Evidence, and Impact, McMaster University, 1280 Main Street West, Hamilton, ON L8S 4K1 Canada
| | - Alvin Leenus
- Faculty of Health Sciences, McMaster University, 1280 Main Street West, Hamilton, ON L8S 4K1 Canada
| | - Lawrence Mbuagbaw
- Department of Health Research Methods, Evidence, and Impact, McMaster University, 1280 Main Street West, Hamilton, ON L8S 4K1 Canada
- Biostatistics Unit, Father Sean O’Sullivan Research Centre, St. Joseph’s Healthcare Hamilton, Hamilton, ON L8N 4A6 Canada
| |
Collapse
|
8
|
Bossema FG, Burger P, Bratton L, Challenger A, Adams RC, Sumner P, Schat J, Numans ME, Smeets I. Expert quotes and exaggeration in health news: a retrospective quantitative content analysis. Wellcome Open Res 2019; 4:56. [PMID: 31346551 PMCID: PMC6619368 DOI: 10.12688/wellcomeopenres.15147.2] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 07/02/2019] [Indexed: 12/25/2022] Open
Abstract
Background This research is an investigation into the role of expert quotes in health news, specifically whether news articles containing a quote from an independent expert are less often exaggerated than articles without such a quote. Methods Retrospective quantitative content analysis of journal articles, press releases, and associated news articles was performed. The investigated sample are press releases on peer-reviewed health research and the associated research articles and news stories. Our sample consisted of 462 press releases and 668 news articles from the UK (2011) and 129 press releases and 185 news articles from The Netherlands (2015). We hand-coded all journal articles, press releases and news articles for correlational claims, using a well-tested codebook. The main outcome measures are types of sources that were quoted and exaggeration of correlational claims. We used counts, 2x2 tables and odds ratios to assess the relationship between presence of quotes and exaggeration of the causal claim. Results Overall, 99.1% of the UK press releases and 84.5% of the Dutch press releases contain at least one quote. For the associated news articles these percentages are: 88.6% in the UK and 69.7% in the Netherlands. Authors of the study are most often quoted and only 7.5% of UK and 7.0% of Dutch news articles contained a new quote by an expert source, i.e. one not provided by the press release. The relative odds that an article without an external expert quote contains an exaggeration of causality is 2.6. Conclusions The number of articles containing a quote from an independent expert is low, but articles that cite an external expert do contain less exaggeration.
Collapse
Affiliation(s)
- Francien G. Bossema
- Department of Science Communication and Society, Leiden University, Leiden, The Netherlands
- Computational Imaging, Centrum Wiskunde & Informatica, Amsterdam, The Netherlands
| | - Peter Burger
- Media Studies, Leiden University, Leiden, The Netherlands
| | - Luke Bratton
- School of Psychology, Cardiff University, Cardiff, UK
| | | | | | - Petroc Sumner
- School of Psychology, Cardiff University, Cardiff, UK
| | - Joop Schat
- Department Public Health and Primary Care, Leiden University Medical Center, Leiden, The Netherlands
| | - Mattijs E. Numans
- Department Public Health and Primary Care, Leiden University Medical Center, Leiden, The Netherlands
| | - Ionica Smeets
- Department of Science Communication and Society, Leiden University, Leiden, The Netherlands
| |
Collapse
|
9
|
Bossema FG, Burger P, Bratton L, Challenger A, Adams RC, Sumner P, Schat J, Numans ME, Smeets I. Expert quotes and exaggeration in health news: a retrospective quantitative content analysis. Wellcome Open Res 2019; 4:56. [PMID: 31346551 PMCID: PMC6619368 DOI: 10.12688/wellcomeopenres.15147.1] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 03/20/2019] [Indexed: 10/13/2023] Open
Abstract
Background This research is an investigation into the role of expert quotes in health news, specifically whether news articles containing a quote from an independent expert are less often exaggerated than articles without such a quote. Methods Retrospective quantitative content analysis of journal articles, press releases, and associated news articles was performed. The investigated sample are press releases on peer-reviewed health research and the associated research articles and news stories. Our sample consisted of 462 press releases and 668 news articles from the UK (2011) and 129 press releases and 185 news articles from The Netherlands (2015). We hand-coded all journal articles, press releases and news articles for correlational claims, using a well-tested codebook. The main outcome measures are types of sources that were quoted and exaggeration of correlational claims. We used counts, 2x2 tables and odds ratios to assess the relationship between presence of quotes and exaggeration of the causal claim. Results Overall, 99.1% of the UK press releases and 84.5% of the Dutch press releases contain at least one quote. For the associated news articles these percentages are: 88.6% in the UK and 69.7% in the Netherlands. Authors of the study are most often quoted and only 7.5% of UK and 7.0% of Dutch news articles contained a new quote by an expert source, i.e. one not provided by the press release. The relative odds that an article without an external expert quote contains an exaggeration of causality is 2.6. Conclusions The number of articles containing a quote from an independent expert is low, but articles that cite an external expert do contain less exaggeration.
Collapse
Affiliation(s)
- Francien G. Bossema
- Department of Science Communication and Society, Leiden University, Leiden, The Netherlands
- Computational Imaging, Centrum Wiskunde & Informatica, Amsterdam, The Netherlands
| | - Peter Burger
- Media Studies, Leiden University, Leiden, The Netherlands
| | - Luke Bratton
- School of Psychology, Cardiff University, Cardiff, UK
| | | | | | - Petroc Sumner
- School of Psychology, Cardiff University, Cardiff, UK
| | - Joop Schat
- Department Public Health and Primary Care, Leiden University Medical Center, Leiden, The Netherlands
| | - Mattijs E. Numans
- Department Public Health and Primary Care, Leiden University Medical Center, Leiden, The Netherlands
| | - Ionica Smeets
- Department of Science Communication and Society, Leiden University, Leiden, The Netherlands
| |
Collapse
|
10
|
Tsujimoto Y, Aoki T, Shinohara K, So R, Suganuma AM, Kimachi M, Yamamoto Y, Furukawa TA. Physician characteristics associated with proper assessment of overstated conclusions in research abstracts: A secondary analysis of a randomized controlled trial. PLoS One 2019; 14:e0211206. [PMID: 30682128 PMCID: PMC6347200 DOI: 10.1371/journal.pone.0211206] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/13/2018] [Accepted: 01/09/2019] [Indexed: 11/18/2022] Open
Abstract
OBJECTIVES Little is known about the physician characteristics associated with appraisal skills of research evidence, especially the assessment of the validity of study methodology. This study aims to explore physician characteristics associated with proper assessment of overstated conclusions in research abstracts. DESIGN A secondary analysis of a randomized controlled trial. SETTING AND PARTICIPANTS We recruited 567 volunteers from the Japan Primary Care Association. METHODS Participants were randomly assigned to read the abstract of a research paper, with or without an overstatement, and to rate its validity. Our primary outcome was proper assessment of the validity of its conclusions. We investigated the association of physician characteristics and proper assessment using logistic regression models and evaluated the interaction between the associated characteristics and overstatement. RESULTS We found significant associations between proper assessment and post-graduate year (odds ratio [OR] = 0.67, 95% confidence interval [CI] 0.49 to 0.91, for every 10-year increase) and research experience as a primary investigator (PI; OR = 2.97, 95% CI 1.65 to 5.34). Post-graduate year and PI had significant interaction with overstatement (P = 0.015 and < 0.001, respectively). Among participants who read abstracts without an overstatement, post-graduate year was not associated with proper assessment (OR = 1.04, 95% CI 0.82 to 1.33), and PI experience was associated with lower scores of the validity (OR = 0.58, 95% CI 0.35 to 0.96). CONCLUSION Physicians who have been in practice longer should be trained in distinguishing overstatements in abstract conclusions. Physicians with research experience might be informed that they tend to rate the validity of research lower regardless of the presence or absence of overstatements. TRIAL REGISTRATION UMIN000026269.
Collapse
Affiliation(s)
- Yasushi Tsujimoto
- Department of Healthcare Epidemiology, School of Public Health in the Graduate School of Medicine, Kyoto University, Kyoto, Japan
- Department of Nephrology and Dialysis, Kyoritsu Hospital, Kawanishi, Hyogo, Japan
| | - Takuya Aoki
- Department of Healthcare Epidemiology, School of Public Health in the Graduate School of Medicine, Kyoto University, Kyoto, Japan
| | - Kiyomi Shinohara
- Department of Health Promotion and Human Behavior Kyoto University Graduate School of Medicine / School of Public Health, Kyoto, Japan
| | - Ryuhei So
- Department of Health Promotion and Human Behavior Kyoto University Graduate School of Medicine / School of Public Health, Kyoto, Japan
| | - Aya M. Suganuma
- Department of Health Promotion and Human Behavior Kyoto University Graduate School of Medicine / School of Public Health, Kyoto, Japan
| | - Miho Kimachi
- Department of Healthcare Epidemiology, School of Public Health in the Graduate School of Medicine, Kyoto University, Kyoto, Japan
| | - Yosuke Yamamoto
- Department of Healthcare Epidemiology, School of Public Health in the Graduate School of Medicine, Kyoto University, Kyoto, Japan
- * E-mail:
| | - Toshi A. Furukawa
- Department of Health Promotion and Human Behavior Kyoto University Graduate School of Medicine / School of Public Health, Kyoto, Japan
| |
Collapse
|
11
|
Shinohara K, Aoki T, So R, Tsujimoto Y, Suganuma AM, Kise M, Furukawa TA. Influence of overstated abstract conclusions on clinicians: a web-based randomised controlled trial. BMJ Open 2017; 7:e018355. [PMID: 29247102 PMCID: PMC5736039 DOI: 10.1136/bmjopen-2017-018355] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/04/2022] Open
Abstract
OBJECTIVES To investigate whether overstatements in abstract conclusions influence primary care physicians' evaluations when they read reports of randomised controlled trials (RCTs) DESIGN: RCT setting: This study was a parallel-group randomised controlled survey, conducted online while masking the study hypothesis. PARTICIPANTS Volunteers were recruited from members of the Japan Primary Care Association in January 2017. We sent email invitations to 7040 primary care physicians. Among the 787 individuals who accessed the website, 622 were eligible and automatically randomised into 'without overstatement' (n=307) and 'with overstatement' (n=315) groups. INTERVENTIONS We selected five abstracts from published RCTs with at least one non-significant primary outcome and overstatement in the abstract conclusion. To construct a version without overstatement, we rewrote the conclusion sections. The methods and results sections were standardised to provide the necessary information of primary outcome information when it was missing in the original abstract. Participants were randomly assigned to read an abstract either with or without overstatements and asked to evaluate the benefit of the intervention. OUTCOME MEASURES The primary outcome was the participants' evaluation of the benefit of the intervention discussed in the abstract, on a scale from 0 to 10. A secondary outcome was the validity of the conclusion. RESULTS There was no significant difference between the groups with respect to their evaluation of the benefit of the intervention (mean difference: 0.07, 95% CI -0.28 to 0.42, p=0.69). Participants in the 'without' group considered the study conclusion to be more valid than those in the 'with' group (mean difference: 0.97, 95% CI 0.59 to 1.36, P<0.001). CONCLUSION The overstatements in abstract conclusions did not significantly influence the primary care physicians' evaluations of the intervention effect when necessary information about the primary outcomes was distinctly reported. TRIAL REGISTRATION NUMBER UMIN000025317; Pre-results.
Collapse
Affiliation(s)
- Kiyomi Shinohara
- Department of Health Promotion and Human Behavior, Kyoto University Graduate School of Medicine/School of Public Health, Kyoto, Japan
| | - Takuya Aoki
- Department of Health care Epidemiology, Kyoto University Graduate School of Medicine/School of Public Health, Kyoto, Japan
| | - Ryuhei So
- Department of Health Promotion and Human Behavior, Kyoto University Graduate School of Medicine/School of Public Health, Kyoto, Japan
- Okayama Psychiatric Medical Center, Okayama, Okayama, Japan
| | - Yasushi Tsujimoto
- Department of Health care Epidemiology, Kyoto University Graduate School of Medicine/School of Public Health, Kyoto, Japan
- Department of Nephrology and Dialysis, Kyoritsu Hospital, Kawanishi, Hyogo, Japan
| | - Aya M Suganuma
- Department of Health Promotion and Human Behavior, Kyoto University Graduate School of Medicine/School of Public Health, Kyoto, Japan
| | - Morito Kise
- Centre for Family Medicine Development, Japanese Health and Welfare Co-operative Federation, Tokyo, Japan
| | - Toshi A Furukawa
- Department of Health Promotion and Human Behavior, Kyoto University Graduate School of Medicine/School of Public Health, Kyoto, Japan
| |
Collapse
|