Do common metrics add value? Perspectives from NIH Clinical and Translational Science Awards (CTSA) Consortium hubs

What funders are doing to assess the impact of their investments in health and biomedical research

As pressures to maximize research funding grow, biomedical research funders are increasingly tasked with demonstrating the long-term and real-world impacts of their funded research investments. Over the past three decades, research impact assessments (RIA) have emerged as an important tool for analysing the impacts of research by incorporating logic models, frameworks and indicators to track measures of knowledge production, capacity-building, development of research products, adoption of research into clinical guidelines and policies, and the realization of health, economic and social benefits. While there are currently several models for RIA within the literature, less attention has been paid to how funders can practically select and implement a RIA model to demonstrate the impacts of their own research portfolios. In this paper, a literature review was performed to understand (1) which research funders have performed RIAs of their research portfolios to date; (2) how funders have designed their assessments, including the models and tools they have used; (3) what challenges to and facilitators of success have funders found when adopting the RIA model to their own portfolio; and (4) who participates in the assessments. Forty-four papers from both published and grey literature were found to meet the review criteria and were examined in detail. There is a growing culture of RIA among funders, and included papers spanned a diverse set of funders from 10 countries or regions. Over half of funders (59.1%) used a framework to conduct their assessment, and a variety of methods for collecting impact data were reported. Issues of methodological rigour were observed across studies in the review, and this was related to numerous challenges funders faced in designing timely RIAs with quality impact data. Over a third of articles (36.4%) included input from stakeholders, yet only one article reported surveying patients and members of the public as part of the assessment. To advance RIA among funders, we offer several recommendations for increasing the methodological rigour of RIAs and suggestions for future research, and call for a careful reflection of the voices needed in an impact assessment to ensure that RIAs are having a meaningful impact on patients and the public.

The 2010-2020 National Natural Science Foundation of China: Radiation Therapy

OBJECTIVE

This study explains the application number and funding rate of oncology projects undertaken by the National Natural Science Foundation of China (NSFC), with focus on tumor radiotherapy-related research over the past 11 years.

METHODS

A stratified analysis was carried out on the application and funding status of tumor radiotherapy studies in different NSFC project categories, different research areas, and different tumor types. Research areas that required specific focus, such as immunology-related radiotherapy, multimodality imaging and radiomics, and post-radiotherapy organ injury, were separately analyzed.

RESULTS

The status and development trends of various related research fields were studied, and the research results were presented with the support of the NSFC, in order to provide reference for future applications and funding allocations.

CONCLUSION

The number of applications for funding increases every year. Although the total number of funded projects has also increased every year, the funding rate has decreased year by year. Projects on radiotherapy and immunization have been at the forefront in recent years, and the funding rate for these projects increases yearly.

Do common metrics add value? Perspectives from NIH Clinical and Translational Science Awards (CTSA) Consortium hubs

INTRODUCTION

The Clinical and Translational Science Awards (CTSA) Consortium, a network of academic health care institutions with CTSA hubs, is charged with improving the national clinical and translational research enterprise. The CTSA Consortium and the NIH National Center for Advancing Translational Sciences implemented the Common Metrics Initiative comprised of standardized metrics and a shared performance improvement framework. This article summarizes hubs' perspectives on its value during the initial implementation.

METHODS

The value was assessed across 58 hubs. Survey items assessed change in perceived ability to manage performance and advance clinical and translational science. Semi-structured interviews elicited hubs' perspectives on meaningfulness and value-added of the Common Metrics Initiative and hubs' recommendations.

RESULTS

Hubs considered their abilities to manage performance to have improved, but there was no change in perceived ability to advance clinical and translational science. The initiative added value by providing a formal structured process, enabling strategic conversations, facilitating improvements in processes, providing an external impetus for improvement, and providing justification for funds invested. Hubs were concerned about the usefulness of the metrics chosen and whether the value-added was sufficient relative to the effort required. Hubs recommended useful benchmarking, disseminating best practices and promoting peer-to-peer learning, and expanding the use of data to inform the initiative.

CONCLUSIONS

Implementing Common Metrics and a performance improvement framework yielded concrete short-term benefits, but concerns about usefulness remained, particularly considering the effort required. The Common Metrics Initiative should focus on facilitating cross-hub collaboration around metrics that address high-priority impact areas for individual hubs and the Consortium.