What works for peer review and decision-making in research funding: a realist synthesis

Are numerical scores important for grant assessment? A cross-sectional study

Background: In the evaluation of research proposals, reviewers are often required to provide their opinions using various forms of quantitative and qualitative criteria. In 2020, the European Commission removed, for the Marie Skłodowska-Curie Actions (MSCA) Innovative Training Networks (ITN) funding scheme, the numerical scores from the individual evaluations but retained them in the consensus report. This study aimed to assess whether there were any differences in reviewer comments' linguistic characteristics after the numerical scoring was removed, compared to comments from 2019 when numerical scoring was still present. Methods: This was an observational study and the data were collected for the Marie Skłodowska-Curie Actions (MSCA) Innovative Training Networks (ITN) evaluation reports from the calls of 2019 and 2020, for both individual and consensus comments and numerical scores about the quality of the research proposal on three evaluation criteria: Excellence, Impact and Implementation. All comments were analyzed using the Linguistic Inquiry and Word Count (LIWC) program. Results: For both years, the comments for proposal's strengths were written in a style that reflects objectivity, clout, and positive affect, while in weaknesses cold and objective style dominated, and that pattern remained stable across proposal status and research domains. Linguistic variables explained a very small proportion of the variance of the differences between 2019 and 2020 (McFadden R 2=0.03). Conclusions: Removing the numerical scores was not associated with the differences in linguistic characteristics of the reviewer comments. Future studies should adopt a qualitative approach to assess whether there are conceptual changes in the content of the comments.

Promoting equality, diversity and inclusion in research and funding: reflections from a digital manufacturing research network

BACKGROUND

Equal, diverse, and inclusive teams lead to higher productivity, creativity, and greater problem-solving ability resulting in more impactful research. However, there is a gap between equality, diversity, and inclusion (EDI) research and practices to create an inclusive research culture. Research networks are vital to the research ecosystem, creating valuable opportunities for researchers to develop their partnerships with both academics and industrialists, progress their careers, and enable new areas of scientific discovery. A feature of a network is the provision of funding to support feasibility studies - an opportunity to develop new concepts or ideas, as well as to 'fail fast' in a supportive environment. The work of networks can address inequalities through equitable allocation of funding and proactive consideration of inclusion in all of their activities.

METHODS

This study proposes a strategy to embed EDI within research network activities and funding review processes. This paper evaluates 21 planned mitigations introduced to address known inequalities within research events and how funding is awarded. EDI data were collected from researchers engaging in a digital manufacturing network activities and funding calls to measure the impact of the proposed method.

RESULTS

Quantitative analysis indicates that the network's approach was successful in creating a more ethnically diverse network, engaging with early career researchers, and supporting researchers with care responsibilities. However, more work is required to create a gender balance across the network activities and ensure the representation of academics who declare a disability. Preliminary findings suggest the network's anonymous funding review process has helped address inequalities in funding award rates for women and those with care responsibilities, more data are required to validate these observations and understand the impact of different interventions individually and in combination.

CONCLUSIONS

In summary, this study offers compelling evidence regarding the efficacy of a research network's approach in advancing EDI within research and funding. The network hopes that these findings will inform broader efforts to promote EDI in research and funding and that researchers, funders, and other stakeholders will be encouraged to adopt evidence-based strategies for advancing this important goal.

A maturity model for the scientific review of clinical trial designs and their informativeness

BACKGROUND

Informativeness, in the context of clinical trials, defines whether a study's results definitively answer its research questions with meaningful next steps. Many clinical trials end uninformatively. Clinical trial protocols are required to go through reviews in regulatory and ethical domains: areas that focus on specifics outside of trial design, biostatistics, and research methods. Private foundations and government funders rarely require focused scientific design reviews for these areas. There are no documented standards and processes, or even best practices, toward a capability for funders to perform scientific design reviews after their peer review process prior to a funding commitment.

MAIN BODY

Considering the investment in and standardization of ethical and regulatory reviews, and the prevalence of studies never finishing or failing to provide definitive results, it may be that scientific reviews of trial designs with a focus on informativeness offer the best chance for improved outcomes and return-on-investment in clinical trials. A maturity model is a helpful tool for knowledge transfer to help grow capabilities in a new area or for those looking to perform a self-assessment in an existing area. Such a model is offered for scientific design reviews of clinical trial protocols. This maturity model includes 11 process areas and 5 maturity levels. Each of the 55 process area levels is populated with descriptions on a continuum toward an optimal state to improve trial protocols in the areas of risk of failure or uninformativeness.

CONCLUSION

This tool allows for prescriptive guidance on next investments to improve attributes of post-funding reviews of trials, with a focus on informativeness. Traditional pre-funding peer review has limited capacity for trial design review, especially for detailed biostatistical and methodological review. Select non-industry funders have begun to explore or invest in post-funding review programs of grantee protocols, based on exemplars of such programs. Funders with a desire to meet fiduciary responsibilities and mission goals can use the described model to enhance efforts supporting trial participant commitment and faster cures.

Online survey exploring researcher experiences of research funding processes in the UK: the effort and burden of applying for funding and fulfilling reporting requirements

OBJECTIVE

To explore researchers' experiences of funding processes, the effort and burden involved in applying for funding, obtaining funding and/or fulfilling reporting requirements with a UK health and social care research funder.

DESIGN/SETTING

A cross-sectional online survey study with open (free-text) and closed questions (August to November 2021).

PARTICIPANTS

Researchers with experience of applying for/obtaining funding and/or experience of fulfilling reporting requirements for UK health and social care research funded between January 2018 and June 2021.

RESULTS

The survey was completed by 182 researchers, of which 176 had experience with applying for/obtaining funding, and 143 had experience with fulfilling reporting requirements during the timeframe. The majority of the 176 respondents (58%) completed between 7 and 13 key processes in order to submit an application and 69% felt that it was critically important to undertake these key processes. Respondents (n=143) reported submitting an average of 17 reports as part of research monitoring to a range of organisations (eg, funders, Higher Education Institutions). However, only 33% of respondents felt it was critically important to provide the requested reporting information to the different organisations. Thematic analysis of free-text questions on application and reporting identified themes relating to process inefficiencies including streamlining and alignment of systems, lack of understanding of processes including a need for improved communication and feedback from organisations with clear explanations about what information is needed, when and why, the support required by respondents and the time, effort and impact on workload and well-being.

CONCLUSIONS

Through this study, we were able to identify funding processes that are considered by some to be effortful, but necessary, as well as those that were perceived as unnecessary, complex and repetitive, and may waste some researchers time and effort and impact on well-being. Possible solutions to increase efficiency and enhance value in these processes were identified.

How Competition for Funding Impacts Scientific Practice: Building Pre-fab Houses but no Cathedrals

In the research integrity literature, funding plays two different roles: it is thought to elevate questionable research practices (QRPs) due to perverse incentives, and it is a potential actor to incentivize research integrity standards. Recent studies, asking funders, have emphasized the importance of the latter. However, the perspective of active researchers on the impact of competitive research funding on science has not been explored yet. Here, I address this issue by conducting a series of group sessions with researchers in two different countries with different degrees of competition for funding, from three scientific fields (medical sciences, natural sciences, humanities), and in two different career stages (permanent versus temporary employment). Researchers across all groups experienced that competition for funding shapes science, with many unintended negative consequences. Intriguingly, these consequences had little to do with the type of QRPs typically being presented in the research integrity literature. Instead, the researchers pointed out that funding could result in predictable, fashionable, short-sighted, and overpromising science. This was seen as highly problematic: scientists experienced that the 'projectification' of science makes it more and more difficult to do any science of real importance: plunging into the unknown or addressing big issues that need a long-term horizon to mature. They also problematized unintended negative effects from collaboration and strategizing. I suggest it may be time to move away from a focus on QRPs in connection with funding, and rather address the real problems. Such a shift may then call for entirely different types of policy actions.

FENS-Kavli Network of Excellence: Postponed, non-competitive peer review for research funding

Receiving research grants is among the highlights of an academic career, affirming previous accomplishments and enabling new research endeavours. Much of the process of acquiring research funding, however, belongs to the less favourite duties of many researchers: It is time consuming, often stressful and, in the majority of cases, unsuccessful. This resentment towards funding acquisition is backed up by empirical research: The current system to distribute research funding, via competitive calls for extensive research applications that undergo peer review, has repeatedly been shown to fail in its task to reliably rank proposals according to their merit, while at the same time being highly inefficient. The simplest, fairest and broadly supported alternative would be to distribute funding more equally across researchers, for example, by an increase of universities' base funding, thereby saving considerable time that can be spent on research instead. Here, I propose how to combine such a 'funding flat rate' model-or other efficient distribution strategies-with quality control through postponed, non-competitive peer review using open science practices.

Reviewer training for improving grant and journal peer review

BACKGROUND

Funders and scientific journals use peer review to decide which projects to fund or articles to publish. Reviewer training is an intervention to improve the quality of peer review. However, studies on the effects of such training yield inconsistent results, and there are no up-to-date systematic reviews addressing this question.

OBJECTIVES

To evaluate the effect of peer reviewer training on the quality of grant and journal peer review.

SEARCH METHODS

We used standard, extensive Cochrane search methods. The latest search date was 27 April 2022.

SELECTION CRITERIA

We included randomized controlled trials (RCTs; including cluster-RCTs) that evaluated peer review with training interventions versus usual processes, no training interventions, or other interventions to improve the quality of peer review.

DATA COLLECTION AND ANALYSIS

We used standard Cochrane methods. Our primary outcomes were 1. completeness of reporting and 2. peer review detection of errors. Our secondary outcomes were 1. bibliometric scores, 2. stakeholders' assessment of peer review quality, 3. inter-reviewer agreement, 4. process-centred outcomes, 5. peer reviewer satisfaction, and 6. completion rate and speed of funded projects. We used the first version of the Cochrane risk of bias tool to assess the risk of bias, and we used GRADE to assess the certainty of evidence.

MAIN RESULTS

We included 10 RCTs with a total of 1213 units of analysis. The unit of analysis was the individual reviewer in seven studies (722 reviewers in total), and the reviewed manuscript in three studies (491 manuscripts in total). In eight RCTs, participants were journal peer reviewers. In two studies, the participants were grant peer reviewers. The training interventions can be broadly divided into dialogue-based interventions (interactive workshop, face-to-face training, mentoring) and one-way communication (written information, video course, checklist, written feedback). Most studies were small. We found moderate-certainty evidence that emails reminding peer reviewers to check items of reporting checklists, compared with standard journal practice, have little or no effect on the completeness of reporting, measured as the proportion of items (from 0.00 to 1.00) that were adequately reported (mean difference (MD) 0.02, 95% confidence interval (CI) -0.02 to 0.06; 2 RCTs, 421 manuscripts). There was low-certainty evidence that reviewer training, compared with standard journal practice, slightly improves peer reviewer ability to detect errors (MD 0.55, 95% CI 0.20 to 0.90; 1 RCT, 418 reviewers). We found low-certainty evidence that reviewer training, compared with standard journal practice, has little or no effect on stakeholders' assessment of review quality in journal peer review (standardized mean difference (SMD) 0.13 standard deviations (SDs), 95% CI -0.07 to 0.33; 1 RCT, 418 reviewers), or change in stakeholders' assessment of review quality in journal peer review (SMD -0.15 SDs, 95% CI -0.39 to 0.10; 5 RCTs, 258 reviewers). We found very low-certainty evidence that a video course, compared with no video course, has little or no effect on inter-reviewer agreement in grant peer review (MD 0.14 points, 95% CI -0.07 to 0.35; 1 RCT, 75 reviewers). There was low-certainty evidence that structured individual feedback on scoring, compared with general information on scoring, has little or no effect on the change in inter-reviewer agreement in grant peer review (MD 0.18 points, 95% CI -0.14 to 0.50; 1 RCT, 41 reviewers, low-certainty evidence).

AUTHORS' CONCLUSIONS

Evidence from 10 RCTs suggests that training peer reviewers may lead to little or no improvement in the quality of peer review. There is a need for studies with more participants and a broader spectrum of valid and reliable outcome measures. Studies evaluating stakeholders' assessments of the quality of peer review should ensure that these instruments have sufficient levels of validity and reliability.

Blinding Models for Scientific Peer-Review of Biomedical Research Proposals: A Systematic Review

Objective: The aim of this systematic review is to estimate: (i) the overall effect of blinding models on bias; (ii) the effect of each blinding model; and (iii) the effect of un-blinding on reviewer's accountability in biomedical research proposals. Methods: Systematic review of prospective or retrospective comparative studies that evaluated two or more peer review blinding models for biomedical research proposals/funding applications and reported outcomes related to peer review efficiency. Results: Three studies that met the inclusion criteria were included in this review and assessed using the QualSyst tool by two authors. Conclusion: Our systematic review is the first to assess peer review blinding models in the context of funding. While only three studies were included, this highlighted the dire need for further RCTs that generate validated evidence. We also discussed multiple aspects of peer review, such as peer review in manuscripts vs proposals and peer review in other fields.

The post-award effort of managing and reporting on funded research: a scoping review

INTRODUCTION

Reporting is a mechanism for funding organisations to monitor and manage the progress, outputs, outcomes, and impacts of the research they fund. Inconsistent approaches to reporting and post-award management, and a growing demand for research information, can lead to perception of unnecessary administrative effort that impacts on decision-making and research activity. Identifying this effort, and what stakeholders see as unmet need for improvement, is crucial if funders and Higher Education Institutions (HEIs) are to streamline their practices and provide better support with reporting activities. In this review, we summarise the processes in post-award management, compare current practices, and explore the purpose of collecting information on funded research. We also identify areas where unnecessary effort is perceived and improvement is needed, using previously reported solutions to inform recommendations for funders and HEIs.

METHODS

We conducted a scoping review of the relevant research and grey literature. Electronic searches of databases, and manual searches of journals and funder websites, resulted in inclusion of 52 records and 11 websites. Information on HEI and funder post-award management processes was extracted, catalogued, and summarised to inform discussion.

RESULTS

Post-award management is a complex process that serves many purposes but requires considerable effort, particularly in the set up and reporting of research. Perceptions of unnecessary effort stem from inefficiencies in compliance, data management and reporting approaches, and there is evidence of needed improvement in mechanisms of administrative support, research impact assessment, monitoring, and evaluation. Solutions should focus on integrating digital systems to reduce duplication, streamlining reporting methods, and improving administrative resources in HEIs.

CONCLUSIONS

Funders and HEIs should work together to support a more efficient post-award management process. The value of research information, and how it is collected and used, can be improved by aligning practices and addressing the specific issues highlighted in this review.

An in-depth exploration of researcher experiences of time and effort involved in health and social care research funding in the UK: The need for changes

The need to reform the way in which research is undertaken is clear, with reducing research bureaucracy and waste at the forefront of this issue for the UK government, funding organisations, higher education institutions and wider research community. The aim of this study was to describe researchers' experiences of the time, effort and burden involved in funding processes-namely applying for research funding and fulfilling reporting requirements. This was an in-depth qualitative study using semi-structured interviews with researchers who had experience applying for funding and/or completing reporting requirements for a UK health and social care research funder between January 2018 and June 2021. Following thematic analysis, five key themes were identified describing researcher experiences of key issues around time, efforts and burden associated with funding processes. These themes encompassed (1) issues with the current funding model for health and social care research, (2) time and effort involved in funding processes, (3) the need for a streamlined end-to-end process, (4) implications for work-life balance, and (5) addressing the need for better support and communication. The findings from this study describe researcher experiences of tasks in the research pathway that currently take considerable time and effort. It was clear that whilst some of this time and effort is considered necessary, some is exacerbated by inefficient and ineffective processes, such as perceived under-funding of research or lack of clarity with regards to funder expectations. This in turn contributes to unnecessary researcher burden, research waste and negative research culture. Better investment in health and social care research and in the researchers themselves who design and deliver the research, alongside improvements in transparency, streamlining and research support could ensure a more positive research culture, and improve the quality of funded research.

Scientists' perception of pilot study quality was influenced by statistical significance and study design

OBJECTIVES

Preliminary studies play a key role in developing large-scale interventions but may be held to higher or lower scientific standards during the peer review process because of their preliminary study status.

STUDY DESIGN AND SETTING

Abstracts from 5 published obesity prevention preliminary studies were systematically modified to generate 16 variations of each abstract. Variations differed by 4 factors: sample size (n = 20 vs. n = 150), statistical significance (P < 0.05 vs. P > 0.05), study design (single group vs. randomized 2 groups), and preliminary study status (presence/absence of pilot language). Using an online survey, behavioral scientists were provided with a randomly selected variation of each of the 5 abstracts and blinded to the existence of other variations. Respondents rated each abstract on aspects of study quality.

RESULTS

Behavioral scientists (n = 271, 79.7% female, median age 34 years) completed 1,355 abstract ratings. Preliminary study status was not associated with perceived study quality. Statistically significant effects were rated as more scientifically significant, rigorous, innovative, clearly written, warranted further testing, and had more meaningful results. Randomized designs were rated as more rigorous, innovative, and meaningful.

CONCLUSION

Findings suggest reviewers place a greater value on statistically significant findings and randomized control design and may overlook other important study characteristics.

Canada's First Experience With a Patient-Adjudicated Cancer Research Grant Competition: A Case Study

Although patient partners have been part of grant review panels, they are rarely given decision-making authority and never given sole responsibility for determining what research to fund. In patient-oriented research, however, patient partners may be in the best position to determine what is needed and whether the proposed research will engage patients in meaningful ways. The objective of this case study was to demonstrate that patient partners can adjudicate patient-oriented research proposals without the inclusion of researcher/expert reviewers.