Developing the Translational Research Workforce: A Pilot Study of Common Metrics for Evaluating the Clinical and Translational Award KL2 Program

The Anesthesiology Physician-Scientist Pipeline: Current Status and Recommendations for Future Growth-An Initiative of the Anesthesia Research Council

The limited number and diversity of resident physicians pursuing careers as physician-scientists in medicine has been a concern for many decades. The Anesthesia Research Council aimed to address the status of the anesthesiology physician-scientist pipeline, benchmarked against other medical specialties, and to develop strategic recommendations to sustain and expand the number and diversity of anesthesiology physician-scientists. The working group analyzed data from the Association of American Medical Colleges and the National Resident Matching Program to characterize the diversity and number of research-oriented residents from US and international allopathic medical schools entering 11 medical specialties from 2009 to 2019. Two surveys were developed to assess the research culture of anesthesiology departments. National Institutes of Health (NIH) funding information awarded to anesthesiology physician-scientists and departments was collected from NIH RePORTER and the Blue Ridge Medical Institute. Anesthesiology ranked eighth to tenth place of 11 medical specialties in the percent of "research-oriented" entering residents, defined as those with advanced degrees (Master's or PhDs) in addition to the MD degree or having published at least 3 research publications before residency. Anesthesiology ranked eighth of 11 specialties in the percent of entering residents who were women but ranked fourth of 11 specialties in the percent of entering residents who self-identified as belonging to an underrepresented group in medicine. There has been a 72% increase in both the total NIH funding awarded to anesthesiology departments and the number of NIH K-series mentored training grants (eg, K08 and K23) awarded to anesthesiology physician-scientists between 2015 and 2020. Recommendations for expanding the size and diversity of the anesthesiology physician-scientist pipeline included (1) developing strategies to increase the number of research intensive anesthesiology departments; (2) unifying the diverse programs among academic anesthesiology foundations and societies that seek to grow research in the specialty; (3) adjusting American Society of Anesthesiologists metrics of success to include the number of anesthesiology physician-scientists with extramural research support; (4) increasing the number of mentored awards from Foundation of Anesthesia Education and Research (FAER) and International Anesthesia Research Society (IARS); (5) supporting an organized and concerted effort to inform research-oriented medical students of the diverse research opportunities within anesthesiology should include the specialty being represented at the annual meetings of Medical Scientist Training Program (MSTP) students and the American Physician Scientist Association, as well as in institutional MSTP programs. The medical specialty of anesthesiology is defined by new discoveries and contributions to perioperative medicine which will only be sustained by a robust pipeline of anesthesiology physician-scientists.

Funding change: An environmental scan of research funders’ knowledge translation strategic plans and initiatives across 10 high-income countries/regions

Knowledge translation (KT) is the science and practice of dissemination and implementation of evidence. We describe how research funders operationalize and evaluate KT initiatives, identify challenges and opportunities, and suggest strategic considerations for KT support. We conducted an environmental scan, which included a systematic search of published and grey literature and a focus group with Canadian funders. A total of 130 published articles and 2415 grey literature sources were screened; 212 unique data sources were included. Published literature commonly described KT initiatives related to “KT practice and science funding.” These initiatives commonly provided funds for infrastructure development (e.g., clinical technologies, database subscriptions) to facilitate translational or applied research to address regional health priorities. Of the articles, 44% outlined an evaluation plan; few provided validated KT metrics. In the grey literature, 364 initiatives were described; the most commonly described initiatives related to “exchange and integrated KT.” Focus group participants hoped to see increased resources to support KT, exchange opportunities with policy/decision-makers, and evaluate KT initiatives. Funders completed various KT initiatives, which tended to engage stakeholders to set research priorities, collaborate with a range of stakeholders, build KT capacity, and mandate KT requirements. We provide six considerations for funders to support KT.

Building an institutional K awardee program at UC Davis through utilization of CTSA resources

NIH offers multiple mentored career development award mechanisms. By building on the UC Davis Clinical and Translational Science Center (CTSC) from its initial NIH funding in 2006, we created an institution-wide K scholar resource. We investigated subsequent NIH funding for K scholars and to what extent CTSC research resources were used. Using NIH RePORTER, we created a database of UC Davis investigators who obtained K01, K08, K23, K25, or K99, as well as institutional KL2 or K12 awards and tracked CTSC research resource use and subsequent funding success. Overall, 94 scholars completed K training between 2007 and 2020, of which 70 participated in one of four institutional, NIH-funded K programs. An additional 103 scholars completed a mentored clinical research training program. Of 94 K awardees, 61 (65%) later achieved NIH funding, with the majority receiving a subsequent individual K award. A higher proportion (73%) of funded scholars used CTSC resources compared to unfunded (48%). Biostatistics and Biomedical Informatics were most commonly used and 55% of scholars used one or more CTSC resource. We conclude that institutional commitment to create a K scholar platform and use of CTSC research resources is associated with high NIH funding rates for early career investigators.

Scholarly Productivity Evaluation of KL2 Scholars Using Bibliometrics and Federal Follow-on Funding: Cross-Institution Study

Background

Evaluating outcomes of the clinical and translational research (CTR) training of a Clinical and Translational Science Award (CTSA) hub (eg, the KL2 program) requires the selection of reliable, accessible, and standardized measures. As measures of scholarly success usually focus on publication output and extramural funding, CTSA hubs have started to use bibliometrics to evaluate the impact of their supported scholarly activities. However, the evaluation of KL2 programs across CTSAs is limited, and the use of bibliometrics and follow-on funding is minimal.

Objective

This study seeks to evaluate scholarly productivity, impact, and collaboration using bibliometrics and federal follow-on funding of KL2 scholars from 3 CTSA hubs and to define and assess CTR training success indicators.

Methods

The sample included KL2 scholars from 3 CTSA institutions (A-C). Bibliometric data for each scholar in the sample were collected from both SciVal and iCite, including scholarly productivity, citation impact, and research collaboration. Three federal follow-on funding measures (at the 5-year, 8-year, and overall time points) were collected internally and confirmed by examining a federal funding database. Both descriptive and inferential statistical analyses were computed using SPSS to assess the bibliometric and federal follow-on funding results.

Results

A total of 143 KL2 scholars were included in the sample with relatively equal groups across the 3 CTSA institutions. The included KL2 scholars produced more publications and citation counts per year on average at the 8-year time point (3.75 publications and 26.44 citation counts) than the 5-year time point (3.4 publications vs 26.16 citation counts). Overall, the KL2 publications from all 3 institutions were cited twice as much as others in their fields based on the relative citation ratio. KL2 scholars published work with researchers from other US institutions over 2 times (5-year time point) or 3.5 times (8-year time point) more than others in their research fields. Within 5 years and 8 years postmatriculation, 44.1% (63/143) and 51.7% (74/143) of KL2 scholars achieved federal funding, respectively. The KL2-scholars of Institution C had a significantly higher citation rate per publication than the other institutions (P<.001). Institution A had a significantly lower rate of nationally field-weighted collaboration than did the other institutions (P<.001). Institution B scholars were more likely to have received federal funding than scholars at Institution A or C (P<.001).

Conclusions

Multi-institutional data showed a high level of scholarly productivity, impact, collaboration, and federal follow-on funding achieved by KL2 scholars. This study provides insights on the use of bibliometric and federal follow-on funding data to evaluate CTR training success across institutions. CTSA KL2 programs and other CTR career training programs can benefit from these findings in terms of understanding metrics of career success and using that knowledge to develop highly targeted strategies to support early-stage career development of CTR investigators.

KL2 mentored career development programs at clinical and translational science award hubs: Practices and outcomes

INTRODUCTION

NIH Clinical and Translational Science Awards (CTSAs) include KL2 mentored career development awards for faculty commencing clinical and translational research. A survey of KL2 leaders revealed program practices, curricular elements and compelling data about scholar characteristics and outcomes.

METHODS

We conducted a literature review, framed the survey construct, and obtained input from across the CTSA consortium. A REDCap survey was emailed in fall 2016 to 61 active programs.

RESULTS

Fifty-five programs (90.2%) responded. Respondents had been funded from 3 to 11 years, including 22 "mature" hubs funded for ≥8 years. Program cohort sizes were 56% "small", 22% "medium", and 22% "large." Hubs offer extensive competency-aligned training opportunities relevant to clinical and translational research, including graduate degrees, mentorship, and grant-writing. Seventy-two percent of hubs report parallel "KL2-equivalent" career development programs. All hubs share their training and facilitate intermingling with other early stage investigators. A total of 1,517 KL2 scholars were funded. KL2 awardees are diverse in their disciplines, research projects, and representation; 54% are female and 12% self-identified as underrepresented in biomedical research. Eighty-seven percent of scholars have 2-3 mentors and are currently supported for 2-3 years. Seventy-eight percent of alumni remain at CTSA institutions in translational science. The most common form of NIH support following scholars' KL2 award is an individual career development award.

CONCLUSIONS

The KL2 is a unique career development award, shaped by competency-aligned training opportunities and interdisciplinary mentorship that inform translational research pathways. Tracking both traditional and novel outcomes of KL2 scholars is essential to capture their career trajectories and impact on health.

Building a Professional Identity and an Academic Career Track in Translational Medicine

Biomedical scientists aim to contribute to further understanding of disease pathogenesis and to develop new diagnostic and therapeutic tools that relieve disease burden. Yet the majority of biomedical scientists do not develop their academic career or professional identity as “translational scientists,” and are not actively involved in the continuum from scientific concept to development of new strategies that change medical practice. The collaborative nature of translational medicine and the lengthy process of bringing innovative findings from bench to bedside conflict with established pathways of building a career in academia. This collaborative approach also poses a problem for evaluating individual contributions and progress. The traditional evaluation of scientific success measured by the impact and number of publications and grants scientists achieve is inadequate when the product is a team effort that may take decades to complete. Further, where scientists are trained to be independent thinkers and to establish unique scientific niches, translational medicine depends on combining individual insights and strengths for the greater good. Training programs that are specifically geared to prepare scientists for a career in translational medicine are not widespread. In addition, the legal, regulatory, scientific and clinical infrastructure and support required for translational research is often underdeveloped in academic institutions and funding organizations, further discouraging the development and success of translational scientists in the academic setting. In this perspective we discuss challenges and potential solutions that could allow for physicians, physician scientists and basic scientists to develop a professional identity and a fruitful career in translational medicine.

Using publication data to evaluate a Clinical and Translational Science Award (CTSA) career development program: Early outcomes from KL2 scholars

Introduction

This study uses KL2 scholars' publications to evaluate the types of research the KL2 program supports and to assess the initial productivity and impact of its scholars.

Methods

We illustrate the feasibility of 3 different approaches to bibliometrics, one viable method for determining the types of research a program or hub supports, and demonstrate how these data can be further combined with internal data records.

Results

Gender differences were observed in the types of research scholars undertake. Overall KL2 scholars are performing well, with their publications being cited more than the norm for National Institutes of Health publications. Favorable results were also observed in scholars' continued engagement in research.

Conclusion

This study illustrates that linking bibliometric data and data categorizing publications along the translational spectrum with a Clinical and Translational Science Award hub's internal data records is feasible and offers a number of innovative possibilities for the evaluation of a Clinical and Translational Science Award hub's programs and investigators.

Evaluation of a mid-career investigator career development award: Assessing the ability of OppNet K18 awardees to obtain NIH follow-on research funding

The National Institutes of Health (NIH) K18 award mechanism provides funded opportunities for established investigators to gain knowledge in fields outside of their primary disciplines, but outcomes associated with these awards have not been evaluated to date. NIH's Basic Behavioral and Social Sciences Opportunity Network (OppNet) is one of the few initiatives that has used this award mechanism. We explored how the unique features of K18 awards affect the ability of recipients to obtain follow-on NIH research funding. We compared outcomes (ability to obtain follow-on funding and interval between receipt of the primary award and receipt of the first follow-on award) associated with OppNet K18 awards to findings from evaluations of other NIH career development (K) awards, which usually target early-career investigators. We hypothesized that K18 award recipients might be (1) more successful than are other K award recipients in obtaining follow-on NIH research funding due to their career experience or (2) less successful due to the competing demands of other projects. By analyzing follow-on NIH research awards and interview data, we found that OppNet K18 award recipients were at least as successful as were other K award recipients in obtaining follow-on funding and may have been more successful by certain measures. K18 awards produce their outcomes with a lower investment per investigator than do other K awards, suggesting continued or enhanced use of the mechanism.

Scholarly productivity and professional advancement of junior researchers receiving KL2, K23, or K08 awards at a large public research institution

Background

How the productivity and careers of KL2 scholars compare with scholars receiving individual K-awards is unknown.

Methods

The productivity of KL2 scholars (n=21) at our institution was compared with that of K08 (n=34) and K23 (n=26) scholars.

Results

KL2 and K23 scholars had greater productivity than K08 scholars (p=0.01). Professional advancement was similar among groups.

Conclusion

At our institution, scholarly productivity and professional advancement did not differ by type of K-award.

Emergency Medicine Resources Within the Clinical Translational Science Institutes: A Cross-sectional Study

BACKGROUND

The Clinical and Translational Science Award (CTSA) program aims to strengthen and support translational research by accelerating the process of translating laboratory discoveries into treatments for patients, training a new generation of clinical and translational researchers, and engaging communities in clinical research efforts. Yet, little is known about how emergency care researchers have interacted with and utilized the resources of academic institutions with CTSAs.

OBJECTIVES

The purpose of this survey was to describe how emergency care researchers use local CTSA resources, to ascertain what proportion of CTSA consortium members have active emergency care research (ECR) programs, and to solicit participation in a national CTSA-associated emergency care translational research network.

METHODS

This study was a survey of all emergency departments affiliated with a CTSA.

RESULTS

Of the 65 CTSA consortium members, three had no ECR program and we obtained responses from 46 of the remaining 62 (74% response rate). The interactions with and resources used by emergency care researchers varied widely. Methodology and biostatistics support was most frequently accessed (77%), followed closely by education and training programs (60%). Several ECR programs (76%) had submitted for funding through CTSAs, with 71% receiving awards. Most CTSA consortium members had an active ECR infrastructure: 21 (46%) had 24/7 availability to recruit and screen for research, and 21 (46%) had less than 24/7 research recruitment. A number of emergency care research programs participated in National Institutes of Health research networks with the Neurological Emergencies Treatment Trials network most highly represented with 23 (59%) sites. Most ECR programs (96%) were interested in participating in a CTSA-based emergency care translational research network.

CONCLUSIONS

Despite little initial involvement in development of the CTSA program, there has been moderate interaction between CTSAs and emergency care. There is considerable interest in participating in a CTSA consortium-based emergency care translational research network.