Physician characteristics associated with proper assessment of overstated conclusions in research abstracts: A secondary analysis of a randomized controlled trial

Increasing Use of Promotional Language in Orthopaedic Surgery Abstracts-An Analysis of 112,916 Abstracts 1985 to 2020

BACKGROUND

Increasing use of "hype" language (eg, language overstating research impact) has been documented in the scientific community. Evaluating language in abstracts is important because readers may use abstracts to extrapolate findings to entire publications. Our purpose was to assess the frequency of hype language within orthopaedic surgery.

METHODS

One hundred thirty-nine hype adjectives were previously identified using a linguistics approach. All publicly available abstracts from 18 orthopaedic surgery journals between 1985 and 2020 were obtained, and hype adjectives were tabulated. Change in frequency of these adjectives was calculated.

RESULTS

A total of 112,916 abstracts were identified. 67.0% (948/1414) of abstracts in 1985 contained hype adjectives, compared with 92.5% (5287/5714) in 2020. The average number of hype adjectives per abstract increased by 136% (1.1 to 2.6). Of the 139 adjectives, 87 (62.5%) increased in frequency and 40 (28.7%) decreased in frequency while 12 (9%) were not used. The hype adjectives with the largest absolute increases in frequency were quality (+324wpm), significant (+320wpm), systematic (+246wpm), top (+239wpm), and international (+201wpm). The five hype adjectives with the largest relative increases in frequency were novel (+10500%), international (+2850%), urgent (+2600%), robust (+2300%), and emerging (+1400%).

CONCLUSION

Promotional language is increasing in orthopaedic surgery abstracts. Authors, editors, and reviewers should seek to minimize the usage of nonobjective language.

Promotional Language (Hype) in Abstracts of Publications of National Institutes of Health-Funded Research, 1985-2020

Importance

Investigators applying for National Institutes of Health (NIH) funding increasingly use promotional language (or hype) that has the potential to undermine objective evaluation. Whether or not the same investigators use hype in subsequent research reports has yet to be investigated.

Objective

To assess changes in the use of hype in journal abstracts reporting research funded by the NIH and to compare those trends with previously reported trends in the associated NIH funding applications.

Design, Setting, and Participants

This cross-sectional study assessed trends (from 1985 to 2020) in the use of promotional adjectives in abstracts of journal articles reporting NIH-funded research, and then compared those trends with previously reported trends for the associated NIH funding applications. Articles included in analyses had abstracts available in PubMed.

Main Outcomes and Measures

Absolute change for the 139 adjective forms that have previously been identified as representing hype in NIH funding applications was measured as the difference in frequency between 1985 and 2020. Relative change was measured as the percentage change in frequency in 2020 relative to 1985, or the first year of occurrence. Consistency of change was measured by the rank order correlation (Kendall τ). Concordance between longitudinal trends in the journal abstracts and NIH funding applications was measured by the rank-order cross-correlation.

Results

In a total of 2 394 480 journal abstracts, all 139 adjective forms were identified in 2 793 592 total occurrences. Among these adjectives, 133 increased in absolute frequency by 5335 words per million (wpm), with a mean (SD) relative increase of 1404% (2371%). The largest absolute increases were for novel (524 wpm), important (414 wpm), and key (378 wpm). The largest relative increases were for scalable (22 wpm [19 964%]), unmet (23 wpm [12 126%]), and tailored (40 wpm [8169%]). The mean (SD) correlation for all adjectives was 0.70 (0.30) with 95 adjectives showing a strong positive correlation (τ > 0.7; P < .001), 24 a moderate positive correlation (0.5 < τ < 0.7; P < .001), and 3 a moderate negative correlation (-0.5 < τ < -0.7; P < .001). The mean (SD) cross-correlation was 0.64 (0.19) with 61 of the 139 adjectives showing a strong positive cross-correlations (τ > 0.7; P < .001), 53 a moderate positive cross-correlations (0.5 < τ < 0.7; P < .001), and 3 a moderate negative cross-correlation (-0.7 < τ < -0.5; P < .001).

Conclusions and Relevance

In this analysis of journal abstracts reporting NIH-funded research from 1985 to 2020, levels of promotional language were found to be increasing and trends were closely associated with previously reported trends in the related NIH funding applications. This suggests that increasing levels of salesmanship may in part be a downstream effect of salesmanship infused during earlier stages of the research cascade.

Trends in the Use of Promotional Language (Hype) in Abstracts of Successful National Institutes of Health Grant Applications, 1985-2020

IMPORTANCE

The integrity of the grant application process is important to the success of the entire research enterprise. However, little information is available concerning the prevalence and evolution of subjective or promotional language ("hype") that has the potential to undermine objectivity in the writing and evaluation of grant applications.

OBJECTIVE

To assess changes over time in the use of hype in abstracts of National Institutes of Health (NIH) grant applications.

DESIGN, SETTING, AND PARTICIPANTS

This cross-sectional study assessed the prevalence of promotional adjectives in abstracts in the NIH archive from 1985 to 2020.

MAIN OUTCOMES AND MEASURES

From all abstracts in the NIH RePORTER (Research Portfolio Online Reporting Tools: Expenditures and Results) archive, adjectives were automatically extracted, and their frequencies in the most recent year (2020) were assessed relative to the start year (1985). Adjectives that shifted significantly in frequency and that carried a promotional sense (ie, hype) were retained, and patterns of change were assessed by plotting yearly frequencies (1985-2020). By grouping the adjectives based on shared semantic properties, broad meanings commonly expressed by hype were identified. Absolute change was measured as the difference in normalized frequency between 1985 and 2020. Relative change was measured as the percentage change in normalized frequency in 2020 relative to 1985, or the first year of occurrence.

RESULTS

In total, 901 717 abstracts were analyzed and 139 adjective forms were identified as hype. Among these 139 adjective forms, 130 hype adjectives increased in frequency by 7690 words per million (wpm) (mean [SD] relative increase, 1378% [3132%]), while 9 hype adjectives decreased in frequency by 686 wpm (mean [SD] relative decrease, 44% [18%]). The largest absolute increases were for the terms novel (1054 wpm), critical (555 wpm), and key (461 wpm), while the largest relative increases were for the terms sustainable (25 157%), actionable (16 114%), and scalable (13 029%). Hype most often serves to promote the significance, novelty, scale, and rigor of a project; the utility of the expected outcomes; the qualities of the investigators and research environment; and the gravity of the problem; as well as conveying the personal attitudes of the applicants.

CONCLUSIONS AND RELEVANCE

Levels of hype in successful NIH grant applications have increased over time from 1985 to 2020. The findings in this study should serve to sensitize applicants, reviewers, and funding agencies to the increasing prevalence of subjective, promotional language in funding applications.