A scoping review of simulation models of peer review

How many submissions are needed to discover friendly suggested reviewers?

It is common in scientific publishing to request from authors reviewer suggestions for their own manuscripts. The question then arises: How many submissions are needed to discover friendly suggested reviewers? To answer this question, as the data we would need is anonymized, we present an agent-based simulation of (single-blinded) peer review to generate synthetic data. We then use a Bayesian framework to classify suggested reviewers. To set a lower bound on the number of submissions possible, we create an optimistically simple model that should allow us to more readily deduce the degree of friendliness of the reviewer. Despite this model's optimistic conditions, we find that one would need hundreds of submissions to classify even a small reviewer subset. Thus, it is virtually unfeasible under realistic conditions. This ensures that the peer review system is sufficiently robust to allow authors to suggest their own reviewers.

Agent-based model: A method worthy of promotion in Library and Information Science

Agent-based model (ABM) is a branch of artificial intelligence. Its specialty is to construct a complex macro-system model by describing the perception, decision, learning and action of micro-agents. This method is widely used in many fields from natural science to social science. We discuss ABM by collecting relevant academic papers which apply to the field of Library and Information Science (LIS). This article systematically reviews how ABM is applied to the LIS field and argues that ABM can provide an exploratory tool with quantifiability, repeatability, interpretability, contingency, adaptability and other types of advantages. Finally, it is pointed out that this method is a research tool worthy of careful exploration.

Peer-reviewing in Surgical Journals: Revolutionize or Perish?

The gold standard of safe-guarding the quality of published science is peer review. However, this long-standing system has not evolved in today's digital world, where there has been an explosion in the number of publications and surgical journals. A journal's quality depends not only on the quality of papers submitted but is reflected upon the quality of its peer review process. Over the past decade journals are experiencing a rapidly escalating "peer review crisis" with editors struggling in recruiting reliable reviewers who will provide their skilled work for free with ever-diminishing incentives within today's restricted time-constraints. The problem is complex and difficult to solve, but more urgent than ever. Time is valuable and academicians, researchers and clinicians are overburdened and already extremely busy publishing their own research along with their ever growing clinical and administrative duties. Fewer and fewer individuals volunteer to provide their skilled work for free which is expected. The current incentives to review do not have a big impact on one's career and therefore are not realistic effective countermeasures. As the limits of the system are constantly stretched, there will inevitably come a "point of no return" and Surgical Journals will be the ones to first take the hit as there is an overwhelming evidence of burnout in the surgical specialties and the Surgical community is almost 50% smaller than its Medical counterpart. This review identifies the potential causes of the peer-review crisis, outlines the incentives and drawbacks of being a reviewer, summarizes the currently established common practices of rewarding reviewers and the existing and potential solutions to the problem. The magnitude of the problem and unsustainability that will make it perish are discussed along with its current flaws. Finally, recommendations are made to address many of the weaknesses of the system with the hope to revive it.

Analyzing sentiments in peer review reports: Evidence from two science funding agencies

Using a novel combination of methods and data sets from two national funding agency contexts, this study explores whether review sentiment can be used as a reliable proxy for understanding peer reviewer opinions. We measure reviewer opinions via their review sentiments on both specific review subjects and proposals’ overall funding worthiness with three different methods: manual content analysis and two dictionary-based sentiment analysis algorithms (TextBlob and VADER). The reliability of review sentiment to detect reviewer opinions is addressed by its correlation with review scores and proposals’ rankings and funding decisions. We find in our samples that review sentiments correlate with review scores or rankings positively, and the correlation is stronger for manually coded than for algorithmic results; manual and algorithmic results are overall correlated across different funding programs, review sections, languages, and agencies, but the correlations are not strong; and manually coded review sentiments can quite accurately predict whether proposals are funded, whereas the two algorithms predict funding success with moderate accuracy. The results suggest that manual analysis of review sentiments can provide a reliable proxy of grant reviewer opinions, whereas the two SA algorithms can be useful only in some specific situations.