The cases for and against double-blind reviews

To date, the majority of authors on scientific publications have been men. While much of this gender bias can be explained by historic sexism and discrimination, there is concern that women may still be disadvantaged by the peer review process if reviewers' biases lead them to reject publications with female authors more often. One potential solution to this perceived gender bias in the reviewing process is for journals to adopt double-blind reviews whereby neither the authors nor the reviewers are aware of each other's identity and gender. To test the efficacy of double-blind reviews in one behavioral ecology journal (Behavioral Ecology, BE), we assigned gender to every authorship of every paper published for 2010-2018 in that journal compared to four other journals with single-blind reviews but similar subject matter and impact factors. While female authorships comprised only 35% of the total in all journals, the double-blind journal (BE) did not have more female authorships than its single-blind counterparts. Interestingly, the incidence of female authorship is higher at behavioral ecology journals (BE and Behavioral Ecology and Sociobiology) than in the ornithology journals (Auk, Condor, Ibis) for papers on all topics as well as those on birds. These analyses suggest that double-blind review does not currently increase the incidence of female authorship in the journals studied here. We conclude, at least for these journals, that double-blind review no longer benefits female authors and we discuss the pros and cons of the double-blind reviewing process based on our findings.

Efficacy of Double-Blind Peer Review in an Imaging Subspecialty Journal

BACKGROUND AND PURPOSE

Many scientific journals use double-blind peer review to minimize potential reviewer bias concerning publication recommendations. However, because neuroradiology is a relatively small subspecialty, this process may be limited by prior knowledge of the authors' work or associated institutions. We sought to investigate the efficacy of reviewer blinding and determine the impact that unblinding may have on manuscript acceptance.

MATERIALS AND METHODS

For manuscripts submitted to the American Journal of Neuroradiology (AJNR) from January through June 2015, reviewers completed a brief anonymous questionnaire after submitting their evaluations, assessing whether they were familiar with the research or had knowledge of the authors or institutions from which the work originated.

RESULTS

The response rate for 1079 questionnaires was 98.8%; 12.9% of reviewers knew or suspected that they knew authors, and 15.3% knew or suspected that they knew the associated institutions. Reviewers correctly identified the authors in 90.3% of cases and correctly stated the institutions in 86.8% of cases. Unblinding resulted from self-citation in 34.1% for both authorship and institutions. The acceptance rate when reviewers knew or suspected that they knew the authors was 57/137 (41.6%) and 262/929 (28.2%) when reviewers did not. The acceptance rate when reviewers knew or suspected that they knew the institutions was 60/163 (36.8%) and 259/903 (28.7%) when they did not. The Fisher exact test showed that author (P < .038) and institution (P < .039) familiarity was associated with greater manuscript acceptance.

CONCLUSIONS

While the AJNR process of double-blind peer review minimizes reviewer bias, perceived knowledge of the author and institution is associated with a higher rate of manuscript acceptance.

Evaluating the Pros and Cons of Different Peer Review Policies via Simulation

In the academic world, peer review is one of the major processes in evaluating a scholars contribution. In this study, we are interested in quantifying the merits of different policies in a peer review process, such as single-blind review, double-blind review, and obtaining authors feedback. Currently, insufficient work has been undertaken to evaluate the benefits of different peer review policies. One of the major reasons for this situation is the inability to conduct any empirical study because data are presently unavailable. In this case, a computer simulation is one of the best ways to conduct a study. We perform a series of simulations to study the effects of different policies on a peer review process. In this study, we focus on the peer review process of a typical computer science conference. Our results point to the crucial role of program chairs in determining the quality and diversity of the articles to be accepted for publication. We demonstrate the importance of discussion among reviewers, suggest circumstances in which the double-blind review policy should be adopted, and question the credibility of the authors feedback mechanism. Finally, we stress that randomness plays an important role in the peer review process, and this role cannot be eliminated. Although our model may not capture every component of a peer review process, it covers some of the most essential elements. Thus, even the simulation results clearly cannot be taken as literal descriptions of an actual peer review process. However, we can at least still use them to identify alternative directions for future study.

Journal Reviewer Ratings: Issues of Particularistic Bias, Agreement, and Predictive Validity Within the Manuscript Review Process

Reviewer evaluations and recommendations for 853 manuscript submissions, over a span of 4 years, are analyzed for evidence of particularistic bias, reviewer agreement, and predictive validity for forecasting a published manuscript's citation impact. Attributes of the submitters, their affiliated institutions, and the reviewers have little consistent association with reviewers' recommendations or editorial decision outcomes. Furthermore, reviewers' recommendations demonstrate a reasonable degree of agreement. However, neither reviewers' evaluative ratings across five dimensions nor publication recommendations can predict the number of citations that a published article subsequently receives. Strengths and limitations of various features of the manuscript review process, as well as the importance of monitoring the process for particularistic biases and evidence of predictive validity, are discussed.

The Review Process at PSPB: Correlates of Interreviewer Agreement and Manuscript Acceptance

Reviewer agreement and the predictors of publication judgments were investigated for first-submission manuscripts to the Personality and Social Psychology Bulletin during a 3 1/2—year period (i.e., one editor’s tenure). Among the findings were the following: Reviewers’ judgments of manuscripts were multirather than unidimensional; reviewer agreement about methodology and overall recommendation was greater among high-prestige than mixed-prestige reviewers; authors with high prestige and authors with low professional experience submitted longer manuscripts than their counterparts; author prestige and text length were positively related to publication judgments of reviewers and editors; and author gender was related to editor’s decisions with female authors receiving less favorable decisions than males. The possible mediation of these findings and their implications for understanding the peer-review process in personality and social psychology are discussed.

The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation

The reliability of peer review of scientific documents and the evaluative criteria scientists use to judge the work of their peers are critically reexamined with special attention to the consistently low levels of reliability that have been reported. Referees of grant proposals agree much more about what is unworthy of support than about what does have scientific value. In the case of manuscript submissions this seems to depend on whether a discipline (or subfield) is general and diffuse (e.g., cross-disciplinary physics, general fields of medicine, cultural anthropology, social psychology) or specific and focused (e.g., nuclear physics, medical specialty areas, physical anthropology, and behavioral neuroscience). In the former there is also much more agreement on rejection than acceptance, but in the latter both the wide differential in manuscript rejection rates and the high correlation between referee recommendations and editorial decisions suggests that reviewers and editors agree more on acceptance than on rejection. Several suggestions are made for improving the reliability and quality of peer review. Further research is needed, especially in the physical sciences.

Masked Reviews Are Not Fairer Reviews

Many people believe that reviewers are less likely to produce unfair and biased reviews when they do not know the identity of the authors. However, there is surprisingly little evidence of such bias, there is little evidence that masked review is effective in addressing bias, and masked review has potential costs and drawbacks to the research community. More empirical work on the existence and nature of bias is needed.

Characteristics of peer reviewed clinical medicine journals

Many variations exist in editorial peer review practices in clinical medicine journals. These practices will become more crucial as medical journals make more of their contents available via the World Wide Web. This paper explores five fundamental dimensions of editorial peer review variations: (1) the extent to which manuscripts are subjected to peer review; (2) the sequence of decision points in the peer review process; (3) blinding practices; (4) acceptance rates for submitted manuscripts; and (5) guidelines stating editors' expectations of reviewers. Variations in editorial peer review practices make it difficult to define a "peer reviewed journal" in clinical medicine. Research in this relatively new area of inquiry has not established the relative strengths of each variation in relation to a journal's quality.