Cultural distance, gender and praise in peer review

Background: Understanding review comments holds significant importance within the realm of scientific discourse. This study aims to conduct an empirical analysis of factors associated with praise in peer review.Methods: The study involved manual labeling of "praise" in 952 review comments drawn from 301 articles published in the British Medical Journal, followed by regression analysis.Results: The study reveals that authors tend to receive longer praise when they share a cultural proximity with the reviewers. Additionally, it is observed that female reviewers are more inclined to provide praiseConclusions: In summary, these discoveries contribute valuable insights for the development of a constructive peer review process and the establishment of a more inclusive research culture.

Are numerical scores important for grant assessment? A cross-sectional study

Background: In the evaluation of research proposals, reviewers are often required to provide their opinions using various forms of quantitative and qualitative criteria. In 2020, the European Commission removed, for the Marie Skłodowska-Curie Actions (MSCA) Innovative Training Networks (ITN) funding scheme, the numerical scores from the individual evaluations but retained them in the consensus report. This study aimed to assess whether there were any differences in reviewer comments' linguistic characteristics after the numerical scoring was removed, compared to comments from 2019 when numerical scoring was still present. Methods: This was an observational study and the data were collected for the Marie Skłodowska-Curie Actions (MSCA) Innovative Training Networks (ITN) evaluation reports from the calls of 2019 and 2020, for both individual and consensus comments and numerical scores about the quality of the research proposal on three evaluation criteria: Excellence, Impact and Implementation. All comments were analyzed using the Linguistic Inquiry and Word Count (LIWC) program. Results: For both years, the comments for proposal's strengths were written in a style that reflects objectivity, clout, and positive affect, while in weaknesses cold and objective style dominated, and that pattern remained stable across proposal status and research domains. Linguistic variables explained a very small proportion of the variance of the differences between 2019 and 2020 (McFadden R 2=0.03). Conclusions: Removing the numerical scores was not associated with the differences in linguistic characteristics of the reviewer comments. Future studies should adopt a qualitative approach to assess whether there are conceptual changes in the content of the comments.

Promotional language and the adoption of innovative ideas in science

How are the merits of innovative ideas communicated in science? Here, we conduct semantic analyses of grant application success with a focus on scientific promotional language, which may help to convey an innovative idea's originality and significance. Our analysis attempts to surmount the limitations of prior grant studies by examining the full text of tens of thousands of both funded and unfunded grants from three leading public and private funding agencies: the NIH, the NSF, and the Novo Nordisk Foundation, one of the world's largest private science funding foundations. We find a robust association between promotional language and the support and adoption of innovative ideas by funders and other scientists. First, a grant proposal's percentage of promotional language is associated with up to a doubling of the grant's probability of being funded. Second, a grant's promotional language reflects its intrinsic innovativeness. Third, the percentage of promotional language is predictive of the expected citation and productivity impact of publications that are supported by funded grants. Finally, a computer-assisted experiment that manipulates the promotional language in our data demonstrates how promotional language can communicate the merit of ideas through cognitive activation. With the incidence of promotional language in science steeply rising, and the pivotal role of grants in converting promising and aspirational ideas into solutions, our analysis provides empirical evidence that promotional language is associated with effectively communicating the merits of innovative scientific ideas.

ChatGPT identifies gender disparities in scientific peer review

The peer review process is a critical step in ensuring the quality of scientific research. However, its subjectivity has raised concerns. To investigate this issue, I examined over 500 publicly available peer review reports from 200 published neuroscience papers in 2022-2023. OpenAI's generative artificial intelligence ChatGPT was used to analyze language use in these reports, which demonstrated superior performance compared to traditional lexicon- and rule-based language models. As expected, most reviews for these published papers were seen as favorable by ChatGPT (89.8% of reviews), and language use was mostly polite (99.8% of reviews). However, this analysis also demonstrated high levels of variability in how each reviewer scored the same paper, indicating the presence of subjectivity in the peer review process. The results further revealed that female first authors received less polite reviews than their male peers, indicating a gender bias in reviewing. In addition, published papers with a female senior author received more favorable reviews than papers with a male senior author, for which I discuss potential causes. Together, this study highlights the potential of generative artificial intelligence in performing natural language processing of specialized scientific texts. As a proof of concept, I show that ChatGPT can identify areas of concern in scientific peer review, underscoring the importance of transparent peer review in studying equitability in scientific publishing.

Relationship between journal impact factor and the thoroughness and helpfulness of peer reviews

The Journal Impact Factor is often used as a proxy measure for journal quality, but the empirical evidence is scarce. In particular, it is unclear how peer review characteristics for a journal relate to its impact factor. We analysed 10,000 peer review reports submitted to 1,644 biomedical journals with impact factors ranging from 0.21 to 74.7. Two researchers hand-coded sentences using categories of content related to the thoroughness of the review (Materials and Methods, Presentation and Reporting, Results and Discussion, Importance and Relevance) and helpfulness (Suggestion and Solution, Examples, Praise, Criticism). We fine-tuned and validated transformer machine learning language models to classify sentences. We then examined the association between the number and percentage of sentences addressing different content categories and 10 groups defined by the Journal Impact Factor. The median length of reviews increased with higher impact factor, from 185 words (group 1) to 387 words (group 10). The percentage of sentences addressing Materials and Methods was greater in the highest Journal Impact Factor journals than in the lowest Journal Impact Factor group. The results for Presentation and Reporting went in the opposite direction, with the highest Journal Impact Factor journals giving less emphasis to such content. For helpfulness, reviews for higher impact factor journals devoted relatively less attention to Suggestion and Solution than lower impact factor journals. In conclusion, peer review in journals with higher impact factors tends to be more thorough, particularly in addressing study methods while giving relatively less emphasis to presentation or suggesting solutions. Differences were modest and variability high, indicating that the Journal Impact Factor is a bad predictor of the quality of peer review of an individual manuscript.

“This article is interesting, however”: exploring the language use in the peer review comment of articles published in the BMJ

Purpose

The purpose of this paper is to reveal a symbol – “however” that authors are very interested in, but few research studies pay attention to the existing literature. The authors aim to further insight its function.

Design/methodology/approach

In this research, the authors selected 3,329 valid comments on articles published in the British Medical Journal (BMJ) from 2015 to 2020 as the research objects. The authors showed the length distribution of reviewers' comments. In what follows, the authors analyzed the general distribution of words in comments and reviewer comments’ position to understand reviewers' comments qualitatively in word dimension. Specially, the authors analyzed functions of “however” and “but”, words that authors are most concerned with. In addition, the authors also discussed some factors, which may be related to “however,” that reflect reviewers' praise through regression analysis.

Findings

The authors found that there are marked differences in the length of reviewers' comments under different review rounds. By mapping the reviewers' comments to different sections, the authors found that reviewers are deeply concerned to methods section. Adjectives and adverbs in comments written in different sections of the manuscripts also have different characteristics. The authors tried to interpret the turning function of “however” in scientific communication. Its frequency of use is related to reviewers' identities, specifically academic status. More precisely, junior researchers use “however” in praise more frequently than senior researchers do.

Research limitations/implications

The linguistic feature and function of “however” and “but” in the reviewers' comments of the rejected manuscripts may be different from accepted papers and also worth exploring. Regrettably, the authors cannot obtain the peer review comments of rejected manuscripts. This point may limit the conclusion of the investigation of this article.

Originality/value

Overall, the survey results revealed some language features of reviewers' comments, which could provide a basis of future endeavors for many reviewers in open peer review (OPR) field. Specially, the authors also put forward an interesting symbol to examine the review comments, “however”, for the first time.