The peer review game: an agent-based model of scientists facing resource constraints and institutional pressures

Measuring the developmental function of peer review: a multi-dimensional, cross-disciplinary analysis of peer review reports from 740 academic journals

Reviewers do not only help editors to screen manuscripts for publication in academic journals; they also serve to increase the rigor and value of manuscripts by constructive feedback. However, measuring this developmental function of peer review is difficult as it requires fine-grained data on reports and journals without any optimal benchmark. To fill this gap, we adapted a recently proposed quality assessment tool and tested it on a sample of 1.3 million reports submitted to 740 Elsevier journals in 2018-2020. Results showed that the developmental standards of peer review are shared across areas of research, yet with remarkable differences. Reports submitted to social science and economics journals show the highest developmental standards. Reports from junior reviewers, women and reviewers from Western Europe are generally more developmental than those from senior, men and reviewers working in academic institutions outside Western regions. Our findings suggest that increasing the standards of peer review at journals requires effort to assess interventions and measure practices with context-specific and multi-dimensional frameworks.

Incentives for Research Effort: An Evolutionary Model of Publication Markets with Double-Blind and Open Review

Contemporary debates about scientific institutions and practice feature many proposed reforms. Most of these require increased efforts from scientists. But how do scientists' incentives for effort interact? How can scientific institutions encourage scientists to invest effort in research? We explore these questions using a game-theoretic model of publication markets. We employ a base game between authors and reviewers, before assessing some of its tendencies by means of analysis and simulations. We compare how the effort expenditures of these groups interact in our model under a variety of settings, such as double-blind and open review systems. We make a number of findings, including that open review can increase the effort of authors in a range of circumstances and that these effects can manifest in a policy-relevant period of time. However, we find that open review's impact on authors' efforts is sensitive to the strength of several other influences.

Exploring Data-Driven Components of Socially Intelligent AI through Cooperative Game Paradigms

The development of new approaches for creating more “life-like” artificial intelligence (AI) capable of natural social interaction is of interest to a number of scientific fields, from virtual reality to human–robot interaction to natural language speech systems. Yet how such “Social AI” agents might be manifested remains an open question. Previous research has shown that both behavioral factors related to the artificial agent itself as well as contextual factors beyond the agent (i.e., interaction context) play a critical role in how people perceive interactions with interactive technology. As such, there is a need for customizable agents and customizable environments that allow us to explore both sides in a simultaneous manner. To that end, we describe here the development of a cooperative game environment and Social AI using a data-driven approach, which allows us to simultaneously manipulate different components of the social interaction (both behavioral and contextual). We conducted multiple human–human and human–AI interaction experiments to better understand the components necessary for creation of a Social AI virtual avatar capable of autonomously speaking and interacting with humans in multiple languages during cooperative gameplay (in this case, a social survival video game) in context-relevant ways.

Peer-reviewing in Surgical Journals: Revolutionize or Perish?

The gold standard of safe-guarding the quality of published science is peer review. However, this long-standing system has not evolved in today's digital world, where there has been an explosion in the number of publications and surgical journals. A journal's quality depends not only on the quality of papers submitted but is reflected upon the quality of its peer review process. Over the past decade journals are experiencing a rapidly escalating "peer review crisis" with editors struggling in recruiting reliable reviewers who will provide their skilled work for free with ever-diminishing incentives within today's restricted time-constraints. The problem is complex and difficult to solve, but more urgent than ever. Time is valuable and academicians, researchers and clinicians are overburdened and already extremely busy publishing their own research along with their ever growing clinical and administrative duties. Fewer and fewer individuals volunteer to provide their skilled work for free which is expected. The current incentives to review do not have a big impact on one's career and therefore are not realistic effective countermeasures. As the limits of the system are constantly stretched, there will inevitably come a "point of no return" and Surgical Journals will be the ones to first take the hit as there is an overwhelming evidence of burnout in the surgical specialties and the Surgical community is almost 50% smaller than its Medical counterpart. This review identifies the potential causes of the peer-review crisis, outlines the incentives and drawbacks of being a reviewer, summarizes the currently established common practices of rewarding reviewers and the existing and potential solutions to the problem. The magnitude of the problem and unsustainability that will make it perish are discussed along with its current flaws. Finally, recommendations are made to address many of the weaknesses of the system with the hope to revive it.

Rigor and reproducibility for data analysis and design in the behavioral sciences

The rigor and reproducibility of science methods depends heavily on the appropriate use of statistical methods to answer research questions and make meaningful and accurate inferences based on data. The increasing analytic complexity and valuation of novel statistical and methodological approaches to data place greater emphasis on statistical review. We will outline the controversies within statistical sciences that threaten rigor and reproducibility of research published in the behavioral sciences and discuss ongoing approaches to generate reliable and valid inferences from data. We outline nine major areas to consider for generally evaluating the rigor and reproducibility of published articles and apply this framework to the 116 Behaviour Research and Therapy (BRAT) articles published in 2018. The results of our analysis highlight a pattern of missing rigor and reproducibility elements, especially pre-registration of study hypotheses, links to statistical code/output, and explicit archiving or sharing data used in analyses. We recommend reviewers consider these elements in their peer review and that journals consider publishing results of these rigor and reproducibility ratings with manuscripts to incentivize authors to publish these elements with their manuscript.

A scoping review of simulation models of peer review

Peer review is a process used in the selection of manuscripts for journal publication and proposals for research grant funding. Though widely used, peer review is not without flaws and critics. Performing large-scale experiments to evaluate and test correctives and alternatives is difficult, if not impossible. Thus, many researchers have turned to simulation studies to overcome these difficulties. In the last 10 years this field of research has grown significantly but with only limited attempts to integrate disparate models or build on previous work. Thus, the resulting body of literature consists of a large variety of models, hinging on incompatible assumptions, which have not been compared, and whose predictions have rarely been empirically tested. This scoping review is an attempt to understand the current state of simulation studies of peer review. Based on 46 articles identified through literature searching, we develop a proposed taxonomy of model features that include model type (e.g. formal models vs. ABMs or other) and the type of modeled peer review system (e.g. peer review in grants vs. in journals or other). We classify the models by their features (including some core assumptions) to help distinguish between the modeling approaches. Finally, we summarize the models' findings around six general themes: decision-making, matching submissions/reviewers, editorial strategies; reviewer behaviors, comparisons of alternative peer review systems, and the identification and addressing of biases. We conclude with some open challenges and promising avenues for future modeling work.

Are non-monetary rewards effective in attracting peer reviewers? A natural experiment

Editors of scientific journals meet increasing challenges to find peer reviewers. Rewarding reviewers has been proposed as a solution to incentives peer review, and journals have already started to offer different kinds of rewards, particularly non-monetary ones. However, research so far has mainly explored the efficacy of monetary rewards, while research on non-monetary rewards is barely absent. The goal of this article is to fill this gap by exploring whether and under what conditions a rather common non-monetary reward employed by journals, i.e., to recognize reviewers work by publishing their names on a yearly issue, is effective in increasing the willingness of scientists to become peer reviewers. We test the efficacy of three different reward settings identified in the literature: (1) engagement contingent, (2) task-completion contingent, and (3) performance contingent, through a natural experiment involving 1865 scientists in faculties of business and economics of Romanian universities. We explore whether reward efficacy varies across scientists depending on their gender, academic rank, research productivity, and type of institution to which they are affiliated. The results show that the performance contingency strongly reduces the number of respondents willing to become reviewers (− 60 % compared to a no-reward setting), particularly males and research productive scientists. Scientists affiliated with private universities are strongly discouraged by the reward. In sum, the results suggest that non-monetary rewards are not necessarily effective, as in some cases they may actually discourage the most intrinsically motivated and competent reviewers.

The miracle of peer review and development in science: an agent-based model

It is not easy to rationalize how peer review, as the current grassroots of science, can work based on voluntary contributions of reviewers. There is no rationale to write impartial and thorough evaluations. If reviewers are unmotivated to carefully select high quality contributions, there is no risk in submitting low-quality work by authors. As a result, scientists face a social dilemma: if everyone acts according to his or her own self-interest, the outcome is low scientific quality. We examine how the increased relevance of public good benefits (journal impact factor), the editorial policy of handling incoming reviews, and the acceptance decisions that take into account reputational information, can help the evolution of high-quality contributions from authors. High effort from the side of reviewers is problematic even if authors cooperate: reviewers are still best off by producing low-quality reviews, which does not hinder scientific development, just adds random noise and unnecessary costs to it. We show with agent-based simulations why certain self-emerged current practices, such as the increased reliance on journal metrics and the reputation bias in acceptance, work efficiently for scientific development. Our results find no proper guidelines, however, how the system of voluntary peer review with impartial and thorough evaluations could be sustainable jointly with rapid scientific development.