Peer review of health research funding proposals: A systematic map and systematic review of innovations for effectiveness and efficiency

'Science by consensus' impedes scientific creativity and progress: A simple alternative to funding biomedical research

The very low success rates of grant applications to the National Institutes of Health (NIH) and the National Science Foundation (NSF) are highly detrimental to the progress of science and the careers of scientists. The peer review process that evaluates proposals has been claimed arbitrarily to be the best there is. This consensus system, however, has never been evaluated scientifically against an alternative. Here we delineate the 15 major problems with the peer review process. We challenge the Science Advisor to the President, and the leadership of NIH, NSF, the U.S. National Academy of Sciences and other funding agencies throughout the world to refute each of these criticisms. We call for the implementation of more equitable alternatives that will not constrain the progress of science. We propose a system that will fund at least 80,000 principal investigators, including young scientists, with about half the current NIH budget, seven-times as many as the current number of NIH "research project grants," and that will forego the cumbersome, expensive, and counterproductive "peer" review stage. Further, we propose that the success of the two systems over 5-10 years be compared scientifically.

FENS-Kavli Network of Excellence: Postponed, non-competitive peer review for research funding

Receiving research grants is among the highlights of an academic career, affirming previous accomplishments and enabling new research endeavours. Much of the process of acquiring research funding, however, belongs to the less favourite duties of many researchers: It is time consuming, often stressful and, in the majority of cases, unsuccessful. This resentment towards funding acquisition is backed up by empirical research: The current system to distribute research funding, via competitive calls for extensive research applications that undergo peer review, has repeatedly been shown to fail in its task to reliably rank proposals according to their merit, while at the same time being highly inefficient. The simplest, fairest and broadly supported alternative would be to distribute funding more equally across researchers, for example, by an increase of universities' base funding, thereby saving considerable time that can be spent on research instead. Here, I propose how to combine such a 'funding flat rate' model-or other efficient distribution strategies-with quality control through postponed, non-competitive peer review using open science practices.

Can cash transfer interventions increase contraceptive use and reduce adolescent birth and pregnancy in low and middle income countries? A systematic review and meta-analysis

Becoming pregnant and giving birth under the age of 20 is associated with a range of adverse social, socioeconomic and health outcomes for adolescent girls and their children in Low and middle income countries. Cash transfers are an example of a structural intervention that can change the local social and economic environment, and have been linked with positive health and social outcomes across several domains. As part of a wider review of structural adolescent contraception interventions, we conducted a systematic review on the impact of cash transfers on adolescent contraception and fertility. Fifteen studies were included in the review with eleven studies providing evidence for meta-analyses on contraception use, pregnancy and childbearing. The evidence suggests that cash transfer interventions are generally ineffective in raising levels of contraceptive use. However, cash transfer interventions did reduce levels of early pregnancy (OR 0.90, 95% CI 0.81 to 1.00). There was suggestive evidence that conditional, but not unconditional, cash transfers reduce levels of early childbearing. Given that much of the evidence is drawn from interventions providing cash transfers conditional on school attendance, supporting school attendance may enable adolescent girls and young women to make life choices that do not involve early pregnancy.

Blinding Models for Scientific Peer-Review of Biomedical Research Proposals: A Systematic Review

Objective: The aim of this systematic review is to estimate: (i) the overall effect of blinding models on bias; (ii) the effect of each blinding model; and (iii) the effect of un-blinding on reviewer's accountability in biomedical research proposals. Methods: Systematic review of prospective or retrospective comparative studies that evaluated two or more peer review blinding models for biomedical research proposals/funding applications and reported outcomes related to peer review efficiency. Results: Three studies that met the inclusion criteria were included in this review and assessed using the QualSyst tool by two authors. Conclusion: Our systematic review is the first to assess peer review blinding models in the context of funding. While only three studies were included, this highlighted the dire need for further RCTs that generate validated evidence. We also discussed multiple aspects of peer review, such as peer review in manuscripts vs proposals and peer review in other fields.

What works for peer review and decision-making in research funding: a realist synthesis

Introduction

Allocation of research funds relies on peer review to support funding decisions, and these processes can be susceptible to biases and inefficiencies. The aim of this work was to determine which past interventions to peer review and decision-making have worked to improve research funding practices, how they worked, and for whom.

Methods

Realist synthesis of peer-review publications and grey literature reporting interventions in peer review for research funding.

Results

We analysed 96 publications and 36 website sources. Sixty publications enabled us to extract stakeholder-specific context-mechanism-outcomes configurations (CMOCs) for 50 interventions, which formed the basis of our synthesis. Shorter applications, reviewer and applicant training, virtual funding panels, enhanced decision models, institutional submission quotas, applicant training in peer review and grant-writing reduced interrater variability, increased relevance of funded research, reduced time taken to write and review applications, promoted increased investment into innovation, and lowered cost of panels.

Conclusions

Reports of 50 interventions in different areas of peer review provide useful guidance on ways of solving common issues with the peer review process. Evidence of the broader impact of these interventions on the research ecosystem is still needed, and future research should aim to identify processes that consistently work to improve peer review across funders and research contexts.

Supplementary Information

The online version contains supplementary material available at 10.1186/s41073-022-00120-2.

A billion-dollar donation: estimating the cost of researchers' time spent on peer review

BACKGROUND

The amount and value of researchers' peer review work is critical for academia and journal publishing. However, this labor is under-recognized, its magnitude is unknown, and alternative ways of organizing peer review labor are rarely considered.

METHODS

Using publicly available data, we provide an estimate of researchers' time and the salary-based contribution to the journal peer review system.

RESULTS

We found that the total time reviewers globally worked on peer reviews was over 100 million hours in 2020, equivalent to over 15 thousand years. The estimated monetary value of the time US-based reviewers spent on reviews was over 1.5 billion USD in 2020. For China-based reviewers, the estimate is over 600 million USD, and for UK-based, close to 400 million USD.

CONCLUSIONS

By design, our results are very likely to be under-estimates as they reflect only a portion of the total number of journals worldwide. The numbers highlight the enormous amount of work and time that researchers provide to the publication system, and the importance of considering alternative ways of structuring, and paying for, peer review. We foster this process by discussing some alternative models that aim to boost the benefits of peer review, thus improving its cost-benefit ratio.

The modified lottery: Formalizing the intrinsic randomness of research funding

Competition for research funds has, in the recent decade, become hypercompetitive. Commonly, to determine which proposals receive funding, a system of peer review is used, which is broadly accepted, easily understood, and broadly trusted among researchers. It is often considered the best system in use, but it suffers from important shortcomings, and adaptations to overcome these shortcomings have small and often short-lived effects. Hence, the preference for peer review does not mean it necessarily outperforms all other systems. In fact, it is time for an open discussion about alternative allocation mechanisms. Random allocation of research funding may be a viable alternative to the current peer review system. In particular the "organized randomness" of a modified lottery is interesting, combining the benefits of randomization with some of the most valuable aspects of peer review. Still, many questions remain and this is certainly not a plea to allocate all research funds using lotteries without further research. But we need to be prepared to consider alternatives, even though they are not perfect, and modified lotteries should be part of the solution.

A retrospective analysis of the peer review of more than 75,000 Marie Curie proposals between 2007 and 2018

Most funding agencies rely on peer review to evaluate grant applications and proposals, but research into the use of this process by funding agencies has been limited. Here we explore if two changes to the organization of peer review for proposals submitted to various funding actions by the European Union has an influence on the outcome of the peer review process. Based on an analysis of more than 75,000 applications to three actions of the Marie Curie programme over a period of 12 years, we find that the changes – a reduction in the number of evaluation criteria used by reviewers and a move from in-person to virtual meetings – had little impact on the outcome of the peer review process. Our results indicate that other factors, such as the type of grant or area of research, have a larger impact on the outcome.

Decision-making approaches used by UK and international health funding organisations for allocating research funds: A survey of current practice

Innovations in decision-making practice for allocation of funds in health research are emerging; however, it is not clear to what extent these are used. This study aims to better understand current decision-making practices for the allocation of research funding from the perspective of UK and international health funders. An online survey (active March-April 2019) was distributed by email to UK and international health and health-related funding organisations (e.g., biomedical and social), and was publicised on social media. The survey collected information about decision-making approaches for research funding allocation, and covered assessment criteria, current and past practices, and considerations for improvements or future practice. A mixed methods analysis provided descriptive statistics (frequencies and percentages of responses) and an inductive thematic framework of key experiences. Thirty-one responses were analysed, representing government-funded organisations and charities in the health sector from the UK, Europe and Australia. Four themes were extracted and provided a narrative framework. 1. The most reported decision-making approaches were external peer review, triage, and face-to-face committee meetings; 2. Key values underpinned decision-making processes. These included transparency and gaining perspectives from reviewers with different expertise (e.g., scientific, patient and public); 3. Cross-cutting challenges of the decision-making processes faced by funders included bias, burden and external limitations; 4. Evidence of variations and innovations from the most reported decision-making approaches, including proportionate peer review, number of decision-points, virtual committee meetings and sandpits (interactive workshop). Broadly similar decision-making processes were used by all funders in this survey. Findings indicated a preference for funders to adapt current decision-making processes rather than using more innovative approaches: however, there is a need for more flexibility in decision-making and support to applicants. Funders indicated the need for information and empirical evidence on innovations which would help to inform decision-making in research fund allocation.

Peer review at the Ministry of Silly Walks

BACKGROUND

Fifty years ago, the groundbreaking British sketch series Monty Python's Flying Circus premiered on BBC One and forever changed the world of comedy. The humour transcended mere absurdity by poking a subversive finger in the eye of buttoned-up British society. Here, we commemorate this cultural milestone and simultaneously call attention to an emerging concept in the health sciences, termed simplified peer review. The union of these disparate subjects motivates a formal gait analysis based on one of the troupe's most iconic sketches, "The Ministry of Silly Walks", a satire of bureaucratic inefficiency.

RESEARCH QUESTION

The sketch portrays peer review as exceedingly efficient, lasting all of 20 s. But was it fair? The answer depends on how one measures silliness. If silly walking can be defined as deviations from typical walking, then it can be quantified using video-based gait analysis.

METHODS

To assess the quality of peer review at the Ministry of Silly Walks, we measured knee flexion in the sagittal plane of motion and calculated the Gait Variable Score (GVS) for three gait cycles, those of the Minister (n = 2) and Mr. Pudey (n = 1), an applicant for a Research Fellowship.

RESULTS

For the Minister, we found large deviations from typical walking across two gait cycles (GVS_knee(1) = 33.6, GVS_knee(2) = 23.3), whereas the gait of Mr Pudey produced an intermediate score (GVS_knee = 16.3). By this measure, Mr Pudney's walk is 3.3 times more variable than typical walking, whereas an exemplary silly walk is 6.7 and 4.7 times more variable, respectively, than typical walking.

SIGNIFICANCE

Our analysis corroborates the Minister's assessment: Mr Pudey is a promising applicant and deserving of a Research Fellowship to advance his silly walk. We suggest that the sketch holds special resonance and uncanny prescience for researchers in the health sciences today.

Digital tools for the recruitment and retention of participants in randomised controlled trials: a systematic map

BACKGROUND

Recruiting and retaining participants in randomised controlled trials (RCTs) is challenging. Digital tools, such as social media, data mining, email or text-messaging, could improve recruitment or retention, but an overview of this research area is lacking. We aimed to systematically map the characteristics of digital recruitment and retention tools for RCTs, and the features of the comparative studies that have evaluated the effectiveness of these tools during the past 10 years.

METHODS

We searched Medline, Embase, other databases, the Internet, and relevant web sites in July 2018 to identify comparative studies of digital tools for recruiting and/or retaining participants in health RCTs. Two reviewers independently screened references against protocol-specified eligibility criteria. Included studies were coded by one reviewer with 20% checked by a second reviewer, using pre-defined keywords to describe characteristics of the studies, populations and digital tools evaluated.

RESULTS

We identified 9163 potentially relevant references, of which 104 articles reporting 105 comparative studies were included in the systematic map. The number of published studies on digital tools has doubled in the past decade, but most studies evaluated digital tools for recruitment rather than retention. The key health areas investigated were health promotion, cancers, circulatory system diseases and mental health. Few studies focussed on minority or under-served populations, and most studies were observational. The most frequently-studied digital tools were social media, Internet sites, email and tv/radio for recruitment; and email and text-messaging for retention. One quarter of the studies measured efficiency (cost per recruited or retained participant) but few studies have evaluated people's attitudes towards the use of digital tools.

CONCLUSIONS

This systematic map highlights a number of evidence gaps and may help stakeholders to identify and prioritise further research needs. In particular, there is a need for rigorous research on the efficiency of the digital tools and their impact on RCT participants and investigators, perhaps as studies-within-a-trial (SWAT) research. There is also a need for research into how digital tools may improve participant retention in RCTs which is currently underrepresented relative to recruitment research.

REGISTRATION

Not registered; based on a pre-specified protocol, peer-reviewed by the project's Advisory Board.

What makes an effective grants peer reviewer? An exploratory study of the necessary skills

This exploratory mixed methods study describes skills required to be an effective peer reviewer as a member of review panels conducted for federal agencies that fund research, and examines how reviewer experience and the use of technology within such panels impacts reviewer skill development. Two specific review panel formats are considered: in-person face-to-face and virtual video conference. Data were collected through interviews with seven program officers and five expert peer review panelists, and surveys from 51 respondents. Results include the skills reviewers’ consider necessary for effective review panel participation, their assessment of the relative importance of these skills, how they are learned, and how review format affects skill development and improvement. Results are discussed relative to the peer review literature and with consideration of the importance of professional skills needed by successful scientists and peer reviewers.

Assessing health research grant applications: A retrospective comparative review of a one-stage versus a two-stage application assessment process

Background

Research funders use a wide variety of application assessment processes yet there is little evidence on their relative advantages and disadvantages. A broad distinction can be made between processes with a single stage assessment of full proposals and those that first invite an outline, with full proposals invited at a second stage only for those which are shortlisted. This paper examines the effects of changing from a one-stage to a two-stage process within the UK’s National Institute for Health Research’s (NIHR) Research for Patient Benefit (RfPB) Programme which made this change in 2015.

Methods

A retrospective comparative design was used to compare eight one-stage funding competitions (912 applications) with eight two-stage funding competitions (1090 applications). Comparisons were made between the number of applications submitted, number of peer and lay reviews required, the duration of the funding round, average external peer review scores, and the total costs involved.

Results

There was a mean number of 114 applications per funding round for the one-stage process and 136 for the two-stage process. The one-stage process took a mean of 274 days and the two-stage process 348 days to complete, although those who were not funded (i.e. the majority) were informed at a mean of 195 days (mean 79 days earlier) under the two-stage process. The mean peer review score for full applications using the one-stage process was 6.46 and for the two-stage process 6.82 (5.6% difference using a 1–10 scale (with 10 being the highest), but there was no significant difference between the lay reviewer scores. The one-stage process required a mean of 423 peer reviews and 102 lay reviewers and the two-stage process required a mean of 208 peer reviews and 50 lay reviews (mean difference of 215 peer reviews and 52 lay reviews) per funding round. Overall cost per funding round changed from £148,908 for the one-stage process to £105,342 for the two-stage process saving approximately £43,566 per round.

Conclusion

We conclude that a two-stage application process increases the number of applications submitted to a funding round, is less burdensome and more efficient for all those involved with the process, is cost effective and has a small increase in peer reviewer scores. For the addition of fewer than 11 weeks to the process substantial efficiencies are gained which benefit funders, applicants and science. Funding agencies should consider adopting a two-stage application assessment process.

The CTSA External Reviewer Exchange Consortium (CEREC): Engagement and efficacy

Introduction

Many institutions evaluate applications for local seed funding by recruiting peer reviewers from their own institutional community. Smaller institutions, however, often face difficulty locating qualified local reviewers who are not in conflict with the proposal. As a larger pool of reviewers may be accessed through a cross-institutional collaborative process, nine Clinical and Translational Science Award (CTSA) hubs formed a consortium in 2016 to facilitate reviewer exchanges. Data were collected to evaluate the feasibility and preliminary efficacy of the consortium.

Methods

The CTSA External Reviewer Exchange Consortium (CEREC) has been supported by a custom-built web-based application that facilitates the process and tracks the efficiency and productivity of the exchange.

Results

All nine of the original CEREC members remain actively engaged in the exchange. Between January 2017 and May 2019, CEREC supported the review process for 23 individual calls for proposals. Out of the 412 reviews requested, 368 were received, for a fulfillment ratio of 89.3%. The yield on reviewer invitations has remained consistently high, with approximately one-third of invitations being accepted, and of the reviewers who agreed to provide a review, 88.3% submitted a complete review. Surveys of reviewers and pilot program administrators indicate high satisfaction with the process.

Conclusions

These data indicate that a reviewer exchange consortium is feasible, adds value to participating partners, and is sustainable over time.

Flipping the grant application review process

The return on research investment resulting from new breakthrough scientific discoveries may be decreasing over time due to the law of diminishing returns, the relative decrease of research funding in terms of purchasing power parity, and various activities gaming the system. By altering the grant-review process, the scientific community may directly address the third problem. There is evidence that peer reviews of research proposals may lack reliability and may produce invalid or inconsistent ratings. In addition, extreme focus on grantsmanship threatens to uproot a cornerstone principle that scientific-value should be the key driver in funding decision-making. This opinion provides (1) a justification of the need to consider alternative strategies to boost the impact of public investment in innovative scientific discovery, (2) proposes a framework for flipping the traditional front-loaded peer-review approach to allocation of research funding, into a new back-loaded assessment of scholarly return on investment, and (3) provokes the scientific community to accelerate the debate on alternative funding mechanisms, as the stakes of inaction may be very high.

Do funding applications where peer reviewers disagree have higher citations? A cross-sectional study

Background: Decisions about which applications to fund are generally based on the mean scores of a panel of peer reviewers. As well as the mean, a large disagreement between peer reviewers may also be worth considering, as it may indicate a high-risk application with a high return.

Methods: We examined the peer reviewers' scores for 227 funded applications submitted to the American Institute of Biological Sciences between 1999 and 2006. We examined the mean score and two measures of reviewer disagreement: the standard deviation and range. The outcome variable was the relative citation ratio, which is the number of citations from all publications associated with the application, standardised by field and publication year.

Results: There was a clear increase in relative citations for applications with a better mean. There was no association between relative citations and either of the two measures of disagreement.

Conclusions: We found no evidence that reviewer disagreement was able to identify applications with a higher than average return. However, this is the first study to empirically examine this association, and it would be useful to examine whether reviewer disagreement is associated with research impact in other funding schemes and in larger sample sizes.