Bias in Research Grant Evaluation Has Dire Consequences for Small Universities

Engineering Features From Advanced Medical Technology Initiative Submissions to Enable Predictive Modeling for Proposal Success

INTRODUCTION

The U.S. Army Telemedicine and Advanced Technology Research Center Advanced Medical Technology Initiative (AMTI) demonstrate key emerging technologies related to military medicine. AMTI invites researchers to submit proposals for short-term funding opportunities that support this goal. AMTI proposal selection is guided by a time-intensive peer review process, where proposals are rated on innovation, military relevance, metrics for success, and return on investment. Utilizing machine learning (ML) could assist in proposal evaluations by learning relationships between proposal performance and proposal features. This research explores the viability of artificial intelligence/ML for predicting proposal ratings given content-based proposal features. Although not meant to replace experts, a model-based approach to evaluating proposal quality could work alongside experts to provide a fast, minimally biased estimate of proposal performance. This article presents initial stages of a project aiming to use ML to prioritize research proposals.

MATERIALS AND METHODS

The initial steps included a literature review to identify potential features. Then, these features were extracted from a dataset consisting of past proposals submissions. The dataset includes 824 proposals submitted to the AMTI program from 2010 to 2022. The analysis will inform a discussion of anticipated next steps toward developing a ML model. The following features were created for future modeling: requested funds; word count by section; readability by section; citations and partners identified; and term frequency-inverse document frequency word vectors.

RESULTS

This initial process identified the top ranked words (data, health, injury, device, treatment, technology, etc.) among the abstract, problem to be solved, military relevance, and metrics/outcomes text proposal fields. The analysis also evaluated the text fields for readability using the Flesch readability scale. Most proposals text fields were categorized as "college graduate," indicating a challenging readability level. Finally, citations and partners were reviewed as an indicator of proposal successfulness.

CONCLUSIONS

This research was the first stage of a larger project to explore the use of ML to predict proposal ratings for the purpose of providing automated support to proposal reviewers and to reveal the preferences and values of AMTI proposal reviewers and other decision-makers. The result of this work will provide practical insights regarding the review process for the AMTI program. This will facilitate reduction in bias for AMTI innovators and a streamlined and subjective process for AMTI administrators, which benefits the military health system overall.

Mapping philanthropic support of science

While philanthropic support for science has increased in the past decade, there is limited quantitative knowledge about the patterns that characterize it and the mechanisms that drive its distribution. Here, we map philanthropic funding to universities and research institutions based on IRS tax forms from 685,397 non-profit organizations. We identify nearly one million grants supporting institutions involved in science and higher education, finding that in volume and scope, philanthropy is a significant source of funds, reaching an amount that rivals some of the key federal agencies like the NSF and NIH. Our analysis also reveals that philanthropic funders tend to focus locally, indicating that criteria beyond research excellence play an important role in funding decisions, and that funding relationships are stable, i.e. once a grant-giving relationship begins, it tends to continue in time. Finally, we show that the bipartite funder-recipient network displays a highly overrepresented motif indicating that funders who share one recipient also share other recipients and we show that this motif contains predictive power for future funding relationships. We discuss the policy implications of our findings on inequality in science, scientific progress, and the role of quantitative approaches to philanthropy.

Blinding reduces institutional prestige bias during initial review of applications for a young investigator award

Organizations that fund research are keen to ensure that their grant selection processes are fair and equitable for all applicants. In 2020, the Arnold and Mabel Beckman Foundation introduced blinding to the first stage of the process used to review applications for Beckman Young Investigator (BYI) awards: applicants were instructed to blind the technical proposal in their initial Letter of Intent by omitting their name, gender, gender-identifying pronouns, and institutional information. Here we examine the impact of this change by comparing the data on gender and institutional prestige of the applicants in the first four years of the new policy (BYI award years 2021-2024) with data on the last four years of the old policy (2017-2020). We find that under the new policy, the distribution of applicants invited to submit a full application shifted from those affiliated with institutions regarded as more prestigious to those outside of this group, and that this trend continued through to the final program awards. We did not find evidence of a shift in the distribution of applicants with respect to gender.

Targeted, actionable and fair: Reviewer reports as feedback and its effect on ECR career choices

Previous studies of the use of peer review for the allocation of competitive funding agencies have concentrated on questions of efficiency and how to make the 'best' decision, by ensuring that successful applicants are also the more productive or visible in the long term. This paper examines the components of feedback received from an unsuccessful grant application, is associated with motivating applicants career decisions to persist (reapply for funding at T1), or to switch (not to reapply, or else leave academia). This study combined data from interviews with unsuccessful ECR applicants (n = 19) to The Wellcome Trust 2009-19, and manual coding of reviewer comments received by applicants (n = 81). All applicants received feedback on their application at T0 with a large proportion of unsuccessful applicants reapplying for funding at T1. Here, peer-review-comments-as-feedback sends signals to applicants to encourage them to persist (continue) or switch (not continue) even when the initial application has failed. Feedback associated by unsuccessful applicants as motivating their decision to resubmit had three characteristics: actionable; targeted; and fair. The results lead to identification of standards of feedback for funding agencies and peer-reviewers to promote when providing reviewer feedback to applicants as part of their peer review process. The provision of quality reviewer-reports-as-feedback to applicants, ensures that peer review acts as a participatory research governance tool focused on supporting the development of individuals and their future research plans.

Commentary on Highly Successful Female Educational Psychologists: Equity and Intersectionality in Success Definitions

Despite the international focus on validation and success indicators of academic work across disciplines, examples of accomplished educational psychologists and their personal stories have been limited in the literature. Prinz et al. (2021) interviewed Five Highly Successful Female Educational Psychologists and offered a content analysis of their success stories, including the academics' strategies and trademark characteristics. In this Commentary, I expand on their findings in light of equity and intersectionality perspectives on academic success. I problematise academic success indicators (publication records and impact) with reference to literature on gender disparities in publication metrics and lack of inclusivity in generic measures of academic success. I outline how individual success narratives intersect with our collective responsibility for higher wellbeing and professional recognition of all colleagues. I argue that the equity and intersectionality perspectives are fundamental to inclusive narratives of success and highlight the power structures that have historically impeded access of diverse and minority scholars to top academic positions. I conclude with four recommendations for addressing the persistent structures of inequities in academic career opportunities.

Faculty Time Allocation at Historically Black Universities and Its Relationship to Institutional Expectations

University faculty divide their time into their main academic responsibilities, typically identified as teaching, research, service, and, at institutions with strong ties to their surrounding community, outreach. Most studies of time allocation have focused on faculty at Primarily White Institutions. The present study investigated how faculty at five Historically Black Universities (HBUs) allocate their time to their academic responsibilities. Data were analyzed based on their tenure status, gender, and representation in science, technology, engineering, and mathematics. Faculty estimated the percentage of time they currently allocate (current), the time they would ideally allocate (ideal), and the time they estimate their institution expects them to allocate (expected) to each academic responsibility. Across all demographics, there were discrepancies between current and ideal time allocation to research and teaching and, in some demographics, outreach. The greatest discrepancy between current and expected time allocation was observed in time allocated to research, with women and untenured faculty also showing a discrepancy in time allocated to teaching, and underrepresented faculty showing no discrepancies between current and expected time allocation. Women, untenured, and underrepresented faculty reported that their time allocation patterns were guided by external factors rather than personal preferences. The surveyed faculty also stated that the patterns of effort distribution expected to obtain tenure were not necessarily guided by the faculty handbooks at their institution. Although this study is limited by its relatively small sample size, it provides an insight into how faculty at HBUs divide their time and the reasons for them to do so.

An experimental test of the effects of redacting grant applicant identifiers on peer review outcomes

Background:

Blinding reviewers to applicant identity has been proposed to reduce bias in peer review.

Methods:

This experimental test used 1200 NIH grant applications, 400 from Black investigators, 400 matched applications from White investigators, and 400 randomly selected applications from White investigators. Applications were reviewed by mail in standard and redacted formats.

Results:

Redaction reduced, but did not eliminate, reviewers’ ability to correctly guess features of identity. The primary, preregistered analysis hypothesized a differential effect of redaction according to investigator race in the matched applications. A set of secondary analyses (not preregistered) used the randomly selected applications from White scientists and tested the same interaction. Both analyses revealed similar effects: Standard format applications from White investigators scored better than those from Black investigators. Redaction cut the size of the difference by about half (e.g. from a Cohen’s d of 0.20–0.10 in matched applications); redaction caused applications from White scientists to score worse but had no effect on scores for Black applications.

Conclusions:

Grant-writing considerations and halo effects are discussed as competing explanations for this pattern. The findings support further evaluation of peer review models that diminish the influence of applicant identity.

Funding:

Funding was provided by the NIH.

The unintended consequences of the pandemic on non-pandemic research activities

Research about the Covid-19 pandemic has taken center stage in shaping the work of many scholars, inter alia highlighting the importance of research in addressing the grand challenges humanity faces. However, the pandemic has also ushered in increased administrative, teaching and out of work commitments for many researchers, leading to concerns that academics will become less willing to invest time in obtaining resources to undertake non-Covid-related projects. Using a large-scale survey of business, economics and management researchers, coupled with their publication histories and additional institutional data, we examine how far individuals experienced the focus on the pandemic as ‘crowding out’ interest in, and undermining their confidence in applying for grants for work not focused on the pandemic. We found 40% of the sample agree that the pandemic has impaired their confidence in applying for non-pandemic-related grants and ‘crowded out’ other projects. Researchers with current and prior grants, particularly those with the most experience of holding grants, scholars whose work ‘impacted’ beyond academia, and early career researchers, disproportionately considered themselves to be most affected. We also found that researchers’ perceptions differed based on institutional characteristics. We discuss the implications of these findings for grant providers and national research agencies as well as for individual academic researchers and the institutions in which they work.

Implicit bias in healthcare: clinical practice, research and decision making

Bias is the evaluation of something or someone that can be positive or negative, and implicit or unconscious bias is when the person is unaware of their evaluation. This is particularly relevant to policymaking during the coronavirus pandemic and racial inequality highlighted during the support for the Black Lives Matter movement. A literature review was performed to define bias, identify the impact of bias on clinical practice and research as well as clinical decision making (cognitive bias). Bias training could bridge the gap from the lack of awareness of bias to the ability to recognise bias in others and within ourselves. However, there are no effective debiasing strategies. Awareness of implicit bias must not deflect from wider socio-economic, political and structural barriers as well ignore explicit bias such as prejudice.

Generating Evidence From Contextual Clinical Research in Low- to Middle Income Countries: A Roadmap Based on Theory of Change

Along with inadequate access to high-quality care, competing health priorities, fragile health systems, and conflicts, there is an associated delay in evidence generation and research from LMICs. Lack of basic epidemiologic understanding of the disease burden in these regions poses a significant knowledge gap as solutions can only be developed and sustained if the scope of the problem is accurately defined. Congenital heart disease (CHD), for example, is the most common birth defect in children. The prevalence of CHD from 1990 to 2017 has progressively increased by 18.7% and more than 90% of children with CHD are born in Low and Middle-Income Countries (LMICs). If diagnosed and managed in a timely manner, as in high-income countries (HICs), most children lead a healthy life and achieve adulthood. However, children with CHD in LMICs have limited care available with subsequent impact on survival. The large disparity in global health research focus on this complex disease makes it a solid paradigm to shape the debate. Despite many challenges, an essential aspect of improving research in LMICs is the realization and ownership of the problem around paucity of local evidence by patients, health care providers, academic centers, and governments in these countries. We have created a theory of change model to address these challenges at a micro- (individual patient or physician or institutions delivering health care) and a macro- (government and health ministries) level, presenting suggested solutions for these complex problems. All stakeholders in the society, from government bodies, health ministries, and systems, to frontline healthcare workers and patients, need to be invested in addressing the local health problems and significantly increase data to define and improve the gaps in care in LMICs. Moreover, interventions can be designed for a more collaborative and effective HIC-LMIC and LMIC-LMIC partnership to increase resources, capacity building, and representation for long-term productivity.

Are we paying-to-play? A quantitative assessment of Canadian open access research in ecology and evolution

Open access (OA) allows for peer-reviewed research to be freely accessed and there has been a collective shift from both researchers and publishers towards more OA publishing. OA typically occurs either through article-processing charges (the gold road) or via self-archiving (the green road); the former can be expensive, while the latter has seen minimal uptake. The gold road of OA has led to predatory publishers and, to some, questionable publications. Here, I used publicly available grant information in Canada and combined this with individual publishing statistics to test a variety of factors and their influence on OA publishing. I showed that an individual’s award amount, H-index, and gender did not influence the proportion of OA articles they published, but an individual’s H-index scaled with the number of OA publications. Institute size influenced OA publishing patterns, with researchers at large universities (i.e., >20 000 full-time students) publishing proportionately more OA articles than medium and small institutes. I discuss the potential for this pattern to build on pre-existing systemic biases when it comes to funding and publishing.

The leaky pipeline in research grant peer review and funding decisions: challenges and future directions

The growing literature on gender inequality in academia attests to the challenge that awaits female researchers during their academic careers. However, research has not yet conclusively resolved whether these biases persist during the peer review process of research grant funding and whether they impact respective funding decisions. Whereas many have argued for the existence of gender inequality in grant peer reviews and outcomes, others have demonstrated that gender equality is upheld during these processes. In the present paper, we illustrate how these opinions have come to such opposing conclusions and consider methodological and contextual factors that render these findings inconclusive. More specifically, we argue that a more comprehensive approach is needed to further the debate, encompassing individual and systemic biases as well as more global social barriers. We also argue that examining gender biases during the peer review process of research grant funding poses critical methodological challenges that deserve special attention. We conclude by providing directions for possible future research and more general considerations that may improve grant funding opportunities and career paths for female researchers.

From bench to bedside: trends in National Institutes of Health funding for neurosurgeons from 1991 to 2015

OBJECTIVE

Neurosurgeons play an important role in advancing medicine through research, the funding of which is historically linked to the National Institutes of Health (NIH). The authors defined variables associated with neurosurgical NIH funding, prevalence of funded topics by neurosurgical subspecialty, and temporal trends in NIH neurosurgical funding.

METHODS

The authors conducted a retrospective review of NIH-funded American Association of Neurological Surgeons members using NIH RePORTER (http://report.nih.gov/) for the years 1991-2015.

RESULTS

The authors followed 6515 neurosurgeons from 1991 to 2015, including 6107 (94%) non-MD-PhD physicians and 408 (6%) MD-PhDs. NIH grants were awarded to 393 (6%) neurosurgeons, with 23.2% of all first-time grants awarded to the top 5 funded institutions. The average total funded grant-years per funded neurosurgeon was 12.5 (range 1-85 grant-years). A higher percentage of MD-PhDs were NIH funded than MDs (22% [n = 91] vs 5% [n = 297], p < 0.0001). The most common grants awarded were R01 (128, 33%), K08 (69, 18%), F32 (60, 15%), M01 (50, 13%), and R21 (39, 10%). F32 and K08 recipients were 9-fold (18% vs 2%, p < 0.001) and 19-fold (38% vs 2%, p < 0.001) more likely to procure an R01 and procured R01 funding earlier in their careers (F32: 7 vs 12 years after residency, p = 0.03; K08: 9 vs 12 years, p = 0.01). Each year, the number of neurosurgeons with active grants linearly increased by 2.2 (R2 = 0.81, p < 0.001), whereas the number of total active grants run by neurosurgeons increased at nearly twice the rate (4.0 grants/year) (R2 = 0.91, p < 0.001). Of NIH-funded neurosurgical grants, 33 (9%) transitioned to funded clinical trial(s). Funded neurosurgical subspecialties included neuro-oncology (33%), functional/epilepsy (32%), cerebrovascular (17%), trauma (10%), and spine (6%). Finally, the authors modeled trends in the number of active training grants and found a linear increase in active R01s (R2 = 0.95, p < 0.001); however, both F32 (R2 = 0.36, p = 0.01) and K08 (R2 = 0.67, p < 0.001) funding had a significant parabolic rise and fall centered around 2003.

CONCLUSIONS

The authors observed an upward trend in R01s awarded to neurosurgeons during the last quarter century. However, their findings of decreased K08 and F32 training grant funding to neurosurgeons and the impact of these training grants on the ultimate success and time to success for neurosurgeons seeking R01 funding suggests that this upward trend in R01 funding for neurosurgeons will be difficult to maintain. The authors' work underscores the importance of continued selection and mentorship of neurosurgeons capable of impacting patient care through research, including the MD-PhDs, who are noted to be more represented among NIH-funded neurosurgeons.

Public communication by research institutes compared across countries and sciences: Building capacity for engagement or competing for visibility?

Leading academic institutions, governments, and funders of research across the world have spent the last few decades fretting publicly about the need for scientists and research organisations to engage more widely with the public and be open about their research. While a global literature asserts that public communication has changed from a virtue to a duty for scientists in many countries and disciplines, our knowledge about what research institutions are doing and what factors drive their ‘going public’ is very limited. Here we present the first cross-national study of N = 2,030 research institutes within universities and large scientific organisations in Brazil, Germany, Italy, Japan, the Netherlands, Portugal, the United Kingdom, and the United States of America. We find that institutes embrace communication with non-peers and do so through a variety of public events and traditional news media–less so through new media channels–and we find variation across countries and sciences, yet these are less evident than we expected. Country and disciplinary cultures contribute to the level of this communication, as do the resources that institutes make available for the effort; institutes with professionalised staff show higher activity online. Future research should examine whether a real change in the organisational culture is happening or whether this activity and resource allocation is merely a means to increase institutional visibility.

Building the Case for Localized Approaches to HIV: Structural Conditions and Health System Capacity to Address the HIV/AIDS Epidemic in Six US Cities

Since the discovery of the secondary preventive benefits of antiretroviral therapy, national and international governing bodies have called for countries to reach 90% diagnosis, ART engagement and viral suppression among people living with HIV/AIDS. The US HIV epidemic is dispersed primarily across large urban centers, each with different underlying epidemiological and structural features. We selected six US cities, including Atlanta, Baltimore, Los Angeles, Miami, New York, and Seattle, with the objective of demonstrating the breadth of epidemiological and structural differences affecting the HIV/AIDS response across the US. We synthesized current and publicly-available surveillance, legal statutes, entitlement and discretionary funding, and service location data for each city. The vast differences we observed in each domain reinforce disparities in access to HIV treatment and prevention, and necessitate targeted, localized strategies to optimize the limited resources available for each city's HIV/AIDS response.

The NIH must reduce disparities in funding to maximize its return on investments from taxpayers

New data from the NIH reveal that the scientific return on its sponsored research reaches a maximum at around $400,000 of annual support per principal investigator. We discuss the implications of this 'sweet spot' for funding policy, and propose that the NIH should limit both the minimum and maximum amount of funding per researcher.