Using publication data to evaluate a Clinical and Translational Science Award (CTSA) career development program: Early outcomes from KL2 scholars

Translation in action: Influence, collaboration, and evolution of COVID-19 research with Clinical and Translational Science Awards consortium support

The National Institutes of Health (NIH)'s Clinical and Translational Science Awards (CTSA) consortium aims to accelerate translational processes that move discoveries from bench to bedside. The coronavirus disease 2019 (COVID-19) pandemic presented unmatched challenges and applications for CTSA hubs nationwide. Our study used bibliometrics to assess features of COVID-19 publications supported by the national CTSA program to characterize the consortium's response to the pandemic. Our goal was to understand relative scientific influence, collaboration across hubs, and trends in research emphasis over time. We identified publications from NIH's curated iSearch COVID-19 Publication Portfolio from February 2020 to February 2023; 3234 peer-reviewed articles relevant to COVID-19 cited a CTSA grant. All 66 CTSA hubs were represented, with large-size and longstanding hubs contributing more publications. Most publications cited UL1 grants, 457 cited KL2/TL1 training grants, and 164 cited multiple hub grants. Compared to a random sample of non-CTSA-supported COVID-19 publications, the CTSA portfolio exhibited greater clinical relevance, more human research, and higher altmetric and citation influence. Results were similar for multi-hub publications involving networked initiatives like multi-site clinical trials or the National COVID-19 Cohort Collaborative. Shifts from molecular/cellular-oriented research toward human-oriented research over time were evident, demonstrating translation in action. Results illuminate how the CTSA consortium confronted the pandemic through high-quality projects oriented toward human research, working across hubs on high-value collaborations, advancing along the translational spectrum over time. Findings validate CTSA hubs as critical support structures during health emergencies.

Evaluating the impact of a CTSA program from 2008 to 2021 through bibliometrics, social network analysis, and altmetrics

Introduction

We evaluate a CTSA program hub by applying bibliometrics, social network analysis (SNA), and altmetrics and examine the changes in research productivity, citation impact, research collaboration, and CTSA-supported research topics since our pilot study in 2017.

Methods

The sampled data included North Carolina Translational and Clinical Science Institute (NC TraCS)-supported publications produced between September 2008 and March 2021. We applied measures and metrics from bibliometrics, SNA, and altmetrics to the dataset. In addition, we analyzed research topics and correlations between different metrics.

Results

1154 NC TraCS-supported publications generated over 53,560 citation counts by April 2021. The average cites per year and the relative citation ratio (RCR) mean of these publications improved from 33 and 2.26 in 2017 to 48 and 2.58 in 2021. The number of involved UNC units in the most published authors' collaboration network increased from 7 (2017) to 10 (2021). NC TraCS-supported co-authorship involved 61 NC organizations. PlumX metrics identified articles with the highest altmetrics scores. About 96% NC TraCS-supported publications have above the average SciVal Topic Prominence Percentile; the average approximate potential to translate of the included publication was 54.2%; and 177 publications addressed health disparity issues. Bibliometric measures (e.g., citation counts, RCR) and PlumX metrics (i.e., Citations, Captures, and Social-Media) are positively correlated (p < .05).

Conclusion

Bibliometrics, SNA, and altmetrics offer distinctive but related perspectives to examine CTSA research performance and longitudinal growth, especially at the individual program hub level. These perspectives can help CTSAs build program foci.

Growing the next generation of trauma surgeon-scientists: Reflections on 20 years of research investment

INTRODUCTION

The Eastern Association for the Surgery of Trauma mission includes fostering research and providing career development opportunities. Eastern Association for the Surgery of Trauma has awarded for 20 years a research scholarship to a promising young investigator. The research mentorship efforts were expanded 5 years ago with the INVEST-C Hack-a-thon. INVEST-C provides an intensive, short-term engagement to propel junior faculty toward establishing research independence. This study investigates the impact of these programs on academic productivity.

METHODS

Pubmed records, National Institutes of Health (NIH) Reporter data, and SCOPUS h-index were acquired for all scholarship (SCH) awardees from 2002 to 2021 (n = 20) and all INVEST-C (INV) participants (2017-2020, n = 19). Current type of practice, total number of funding awards, and timing of first award were ascertained. INVEST-C participants were also surveyed on an annual basis to track their progress. Medians (interquartile range [IQR]) are reported and compared (analysis of variance).

RESULTS

Median publications (PUBs) of SCH awardees were 56 (IQR, 33-88), h-index was 16 (IQR, 12-21), and 25% of awardees have ≥1 NIH grant since their SCH. Among the last 10 awardees with a minimum of 2 years from SCH, 40% have received an NIH award compared with a mean NIH funding rate of 18.5% over the same period. For those remaining in academics (90% SCH), PUBs were higher for those >5 years (66 [IQR, 51-115]) versus <5 years from their SCH (33 [22-59]; p = 0.05), but there was no difference in h-index (16 [IQR, 14-25] vs. 15 [9-19], p = NS). Comparing the most recent 5 years of SCH to INV group, there was no difference in academic productivity as measured by total PUBs (SCH, 33 [IQR, 22-59] vs. INV, 34 [IQR, 18-44]; p = 0.7) or h-index (INV, 9 [IQR, 5-14]; p = 0.1). However, no attendee held research funding before INV, but 31.6% (6 of 19 attendees) have subsequently acquired ≥1 funding award (11 non-NIH, 1 NIH) in the short interval since participation.

CONCLUSION

Investments in research activities have translated to significant extramural funding. Those in the last 5 years have been particularly fruitful with INV participants already achieving equal median academic productivity to SCH recipients.

LEVEL OF EVIDENCE

Prognostic and Epidemiologic; Level III.

Scholarly Productivity Evaluation of KL2 Scholars Using Bibliometrics and Federal Follow-on Funding: Cross-Institution Study

Background

Evaluating outcomes of the clinical and translational research (CTR) training of a Clinical and Translational Science Award (CTSA) hub (eg, the KL2 program) requires the selection of reliable, accessible, and standardized measures. As measures of scholarly success usually focus on publication output and extramural funding, CTSA hubs have started to use bibliometrics to evaluate the impact of their supported scholarly activities. However, the evaluation of KL2 programs across CTSAs is limited, and the use of bibliometrics and follow-on funding is minimal.

Objective

This study seeks to evaluate scholarly productivity, impact, and collaboration using bibliometrics and federal follow-on funding of KL2 scholars from 3 CTSA hubs and to define and assess CTR training success indicators.

Methods

The sample included KL2 scholars from 3 CTSA institutions (A-C). Bibliometric data for each scholar in the sample were collected from both SciVal and iCite, including scholarly productivity, citation impact, and research collaboration. Three federal follow-on funding measures (at the 5-year, 8-year, and overall time points) were collected internally and confirmed by examining a federal funding database. Both descriptive and inferential statistical analyses were computed using SPSS to assess the bibliometric and federal follow-on funding results.

Results

A total of 143 KL2 scholars were included in the sample with relatively equal groups across the 3 CTSA institutions. The included KL2 scholars produced more publications and citation counts per year on average at the 8-year time point (3.75 publications and 26.44 citation counts) than the 5-year time point (3.4 publications vs 26.16 citation counts). Overall, the KL2 publications from all 3 institutions were cited twice as much as others in their fields based on the relative citation ratio. KL2 scholars published work with researchers from other US institutions over 2 times (5-year time point) or 3.5 times (8-year time point) more than others in their research fields. Within 5 years and 8 years postmatriculation, 44.1% (63/143) and 51.7% (74/143) of KL2 scholars achieved federal funding, respectively. The KL2-scholars of Institution C had a significantly higher citation rate per publication than the other institutions (P<.001). Institution A had a significantly lower rate of nationally field-weighted collaboration than did the other institutions (P<.001). Institution B scholars were more likely to have received federal funding than scholars at Institution A or C (P<.001).

Conclusions

Multi-institutional data showed a high level of scholarly productivity, impact, collaboration, and federal follow-on funding achieved by KL2 scholars. This study provides insights on the use of bibliometric and federal follow-on funding data to evaluate CTR training success across institutions. CTSA KL2 programs and other CTR career training programs can benefit from these findings in terms of understanding metrics of career success and using that knowledge to develop highly targeted strategies to support early-stage career development of CTR investigators.

Scope, Influence, and Interdisciplinary Collaboration: The Publication Portfolio of the NIH Clinical and Translational Science Awards (CTSA) Program From 2006 Through 2017

The Clinical and Translational Science Awards (CTSA) program sponsors an array of innovative, collaborative research. This study uses complementary bibliometric approaches to assess the scope, influence, and interdisciplinary collaboration of publications supported by single CTSA hubs and those supported by multiple hubs. Authors identified articles acknowledging CTSA support and assessed the disciplinary scope of research areas represented in that publication portfolio, their citation influence, interdisciplinary overlap among research categories, and characteristics of publications supported by multihub collaborations. Since 2006, CTSA hubs supported 69,436 articles published in 4,927 journals and 189 research areas. The portfolio is well distributed across diverse research areas with above-average citation influence. Most supported publications involved clinical/health sciences, for example, neurology and pediatrics; life sciences, for example, neuroscience and immunology; or a combination of the two. Publications supported by multihub collaborations had distinct content emphasis, stronger citation influence, and greater interdisciplinary overlap. This study characterizes the CTSA consortium's contributions to clinical and translational science, identifies content areas of strength, and provides evidence for the success of multihub collaborations. These methods lay the foundation for future investigation of the best policies and priorities for fostering translational science and allow hubs to understand their progress benchmarked against the larger consortium.

Using bibliometrics to evaluate translational science training: evidence for early career success of KL2 scholars

Introduction:

Evaluating clinical and translational research (CTR) mentored training programs is challenging because no two programs are alike. Careful selection of appropriate metrics is required to make valid comparisons between individuals and between programs. The KL2 program provides mentored-training for early-stage CTR investigators. Clinical and Translational Awards across the country have unique KL2 programs. The evaluation of KL2 programs has begun to incorporate bibliometrics to measure KL2 scholar and program impact.

Methods:

This study investigated demographic differences in bibliometric performance and post-K award funding of KL2 scholars and compared the bibliometric performance and post-K award federal funding of KL2 scholars and other mentored-K awardees at the same institution. Data for this study included SciVal and iCite bibliometrics and National Institutions of Health RePORTER grant information for mentored-K awardees (K08, K23, and KL2) at Case Western Reserve University between 2005 and 2013.

Results:

Results showed no demographics differences within the KL2 program scholars. Bibliometric differences between KL2 and other mentored-K awardee indicated an initial KL2 advantage for the number of publications at 5 years’ post-matriculation (i.e., the start of the K award). Regression analyses indicated the number of initial publications was a significant predictor of federal grant funding at the same time point. Analysis beyond the 5-year post-matriculation point did not result in a sustained, significant KL2 advantage.

Conclusions:

Factors that contributed to the grant funding advantage need to be determined. Additionally, differences between translational and clinical bibliometrics must be interpreted with caution, and appropriate metrics for translational science must be established.

Bibliometrics approach to evaluating the research impact of CTSAs: A pilot study

INTRODUCTION

To enhance the performance evaluation of Clinical and Translational Science Award (CTSA) hubs, we examined the utility of advanced bibliometric measures that go beyond simple publication counts to demonstrate the impact of translational research output.

METHODS

The sampled data included North Carolina Translational and Clinical Science Institute (NC TraCS)-supported publications produced between September 2008 and March 2017. We adopted advanced bibliometric measures and a state-of-the-art bibliometric network analysis tool to assess research productivity, citation impact, the scope of research collaboration, and the clusters of research topics.

RESULTS

Totally, 754 NC TraCS-supported publications generated over 24,000 citation counts by April 2017 with an average of 33 cites per article. NC TraCS-supported research papers received more than twice as many cites per year as the average National Institute of Health-funded research publications from the same field and time. We identified the top productive researchers and their networks within the CTSA hub. Findings demonstrated the impact of NC TraCS in facilitating interdisciplinary collaborations within the CTSA hub and across the CTSA consortium and connecting researchers with right peers and organizations.

CONCLUSION

Both improved bibliometrics measures and bibliometric network analysis can bring new perspectives to CTSA evaluation via citation influence and the scope of research collaborations.

Impact of master's degree attainment upon academic career placement in neurosurgery

OBJECTIVE

Previous authors have investigated many factors that predict an academic neurosurgical career over private practice, including attainment of a Doctor of Philosophy (PhD) and number of publications. Research has yet to demonstrate whether a master's degree predicts an academic neurosurgical career. This study quantifies the association between obtaining a Master of Science (MS), Master of Public Health (MPH), or Master of Business Administration (MBA) degree and pursuing a career in academic neurosurgery.

METHODS

Public data on neurosurgeons who had graduated from Accreditation Council for Graduate Medical Education (ACGME)-accredited residency programs in the period from 1949 to 2019 were collected from residency and professional websites. Residency graduates with a PhD were excluded to isolate the effect of only having a master's degree. A position was considered "academic" if it was affiliated with a hospital that had a neurosurgery residency program; other positions were considered nonacademic. Bivariate analyses were performed with Fisher's exact test. Multivariate analysis was performed using a logistic regression model.

RESULTS

Within our database of neurosurgery residency alumni, there were 47 (4.1%) who held an MS degree, 31 (2.7%) who held an MPH, and 10 (0.9%) who held an MBA. In bivariate analyses, neurosurgeons with MS degrees were significantly more likely to pursue academic careers (OR 2.65, p = 0.0014, 95% CI 1.40-5.20), whereas neurosurgeons with an MPH (OR 1.41, p = 0.36, 95% CI 0.64-3.08) or an MBA (OR 1.00, p = 1.00, 95% CI 0.21-4.26) were not. In the multivariate analysis, an MS degree was independently associated with an academic career (OR 2.48, p = 0.0079, 95% CI 1.28-4.93). Moreover, postresidency h indices of 1 (OR 1.44, p = 0.048, 95% CI 1.00-2.07), 2-3 (OR 2.76, p = 2.01 × 10-8, 95% CI 1.94-3.94), and ≥ 4 (OR 4.88, p < 2.00 × 10-16, 95% CI 3.43-6.99) were all significantly associated with increased odds of pursuing an academic career. Notably, having between 1 and 11 months of protected research time was significantly associated with decreased odds of pursuing academic neurosurgery (OR 0.46, p = 0.049, 95% CI 0.21-0.98).

CONCLUSIONS

Neurosurgery residency graduates with MS degrees are more likely to pursue academic neurosurgical careers relative to their non-MS counterparts. Such findings may be used to help predict residency graduates' future potential in academic neurosurgery.