Publication track records as a metric of clinical research training effectiveness

Assessments of Research Competencies for Clinical Investigators: A Systematic Review

Although there is extensive research literature on clinical skill competencies and the use of competency-based frameworks for clinical research, the appropriate methods to assess these competencies are not as well understood. Our goal in this systematic literature review is to identify, compare, and critique assessments of clinical research competencies. Articles were included in this review if they examined clinical investigators or clinical investigators in training, focused on research-based skills, and included some form of assessment of research-based competencies. A total of 76 articles were identified as part of the initial search; 16 met the criteria for inclusion. Two types of assessments of clinical research competence were identified: subjective self-assessments (n = 13) and objective tests (n = 6). These assessments covered a wide range of competencies, but there were no competency domains common to all. Most assessments had limited validation. Training was consistently associated with self-assessed competence but had little relationship to objective measures of competence. In contrast, experience was consistently associated with objectively assessed competence but not with self-assessed competence. These findings have important implications for those interested in assessing medical education programs. We describe a recommended standard for validity for assessments used for the purposes of summative program assessment.

A multimodal international collaborative clinical research training program in China

Background: There is a strong need to include training of research methods in training programs for physicians. International clinical research training programs (CRTP) that comprehensively introduce the methodology of clinical research and combined with practice should be a priority. However, few studies have reported a multimodal international CRTP that provides clinicians with an introduction to the quantitative and methodological principles of clinical research. Objective: This manuscript is intended to comprehensively describe the development process and the structure of this multimodal training program. Methods: The CRTP was comprised of three distinct, sequential learning components: part 1 - a six-week online eLearning self-study; part 2 - a series of three weekly interactive synchronous webinars conducted between Durham, North Carolina, USA and Beijing, China; and part 3 - a five-day in-person workshop held at Beijing Friendship Hospital, Capital Medical University (BFH-CMU). Self-assessment quiz scores and participation rates were used to evaluate effectiveness of the training program. Participants' demographic characteristics, research experience, satisfaction and feedback on the program were collected using questionnaires. Results: A total of 50 participants joined the CRTP. Forty-four participants (88%) completed the program satisfaction questionnaires. The average quiz score of the six eLearning units varied from 31% to 73%. Among the three components of the program, the online eLearning self-study was felt to be the most challenging. Thirty-nine (89%) of the surveyed respondents were satisfied with all components of the training program. Among the respondents, 43 (98%) felt the training was helpful in preparing them for future clinical research projects and expressed willingness to recommend the program to other colleagues. Conclusions: We established a multimodal international collaborative training program. The program demonstrated acceptable participation rates and high satisfaction among Chinese clinicians. It provides a model that may be used by others developing similar international clinical research training programs for physicians.

A matched cohort examination of publication rates among clinical subspecialty fellows enrolled in a translational science training program

Purpose

This study examined the effectiveness of a formal postdoctoral education program designed to teach skills in clinical and translational science, using scholar publication rates as a measure of research productivity.

Method

Participants included 70 clinical fellows who were admitted to a master’s or certificate training program in clinical and translational science from 1999 to 2015 and 70 matched control peers. The primary outcomes were the number of publications 5 years post-fellowship matriculation and time to publishing 15 peer-reviewed manuscripts post-matriculation.

Results

Clinical and translational science program graduates published significantly more peer-reviewed manuscripts at 5 years post-matriculation (median 8 vs 5, p=0.041) and had a faster time to publication of 15 peer-reviewed manuscripts (matched hazard ratio = 2.91, p=0.002). Additionally, program graduates’ publications yielded a significantly higher average H-index (11 vs. 7, p=0.013).

Conclusion

These findings support the effectiveness of formal training programs in clinical and translational science by increasing academic productivity.

Prediction of junior faculty success in biomedical research: comparison of metrics and effects of mentoring programs

Measuring and predicting the success of junior faculty is of considerable interest to faculty, academic institutions, funding agencies and faculty development and mentoring programs. Various metrics have been proposed to evaluate and predict research success and impact, such as the h-index, and modifications of this index, but they have not been evaluated and validated side-by-side in a rigorous empirical study. Our study provides a retrospective analysis of how well bibliographic metrics and formulas (numbers of total, first- and co-authored papers in the PubMed database, numbers of papers in high-impact journals) would have predicted the success of biomedical investigators (n = 40) affiliated with the University of Nevada, Reno, prior to, and after completion of significant mentoring and research support (through funded Centers of Biomedical Research Excellence, COBREs), or lack thereof (unfunded COBREs), in 2000–2014. The h-index and similar indices had little prognostic value. Publishing as mid- or even first author in only one high-impact journal was poorly correlated with future success. Remarkably, junior investigators with >6 first-author papers within 10 years were significantly (p < 0.0001) more likely (93%) to succeed than those with ≤6 first-author papers (4%), regardless of the journal’s impact factor. The benefit of COBRE-support increased the success rate of junior faculty approximately 3-fold, from 15% to 47%. Our work defines a previously neglected set of metrics that predicted the success of junior faculty with high fidelity—thus defining the pool of faculty that will benefit the most from faculty development programs such as COBREs.

Teaching Translational Research to Medical Students: The New York University School of Medicine's Master's of Science in Clinical Investigation Dual-Degree Program

To develop the next generation of translational investigators, New York University School of Medicine (NYUSOM) and the NYU-NYC Health and Hospitals Corporation Clinical and Translational Science Institute (NYU-HHC CTSI) developed the Master's of Science in Clinical Investigation dual-degree (MD/MSCI) program. This 5-year program dedicates 1 year to coursework and biomedical research, followed by a medical school/research overlap year, to prepare students for academic research careers. This paper details the MD/MSCI program's curriculum and approach to mentorship, describes the research/professional interests of students, and reports student productivity. In the first 4 years of the program (2010-2014) 20 students were matriculated; 7 (35%) were women, and 12 (60%) research projects were in surgical specialties. To date, 14 students have applied to residency, and half pursued surgical residency programs. Our students have produced 68 accepted abstracts, 15 abstracts in submission, 38 accepted papers, and 24 papers in submission. Despite the time-limited nature of this program, additional training in research design and implementation has promoted a high level of productivity. We conclude that dual-degree training in medicine and translational research is feasible for medical students and allows for meaningful participation in valuable projects. Follow-up is warranted to evaluate the academic trajectory of these students.

NIH grant awards as a metric of clinical and translational research training effectiveness

The number of clinical research training programs has increased over the past 5-10 years, but few studies have quantitatively evaluated the effectiveness of these programs. The goal of this study was to evaluate the clinical and translational research training program at the University of Cincinnati by comparing the number of National Institutes of Health grants awarded to pediatric fellows who graduated from the MS degree program between 1995 and 2013 versus fellows who did not pursue an MS degree. Among 394 pediatric fellows, 16 of 81 (20%) MS alumni were awarded at least one NIH grant, as compared with 28 of 313 (9%) fellows who did not obtain an MS degree (p < 0.02). In multivariable analysis, MS alumni were more than three times as likely to have received at least one grant than were non-MS fellows (OR = 3.5, 95% CI [1.7-7.2]; C-statistic = 0.71) and MS alumni were more likely to obtain at least one K-series (OR = 4.1, 95% CI [1.6-10.2]; C-statistic = 0.74), M-series (OR = 11.8, 95% CI [3.4-41.4]; C-statistic = 0.81), or R-series (OR = 10.1, 95% CI [2.4-42.8]; C-statistic = 0.74) grant than were non-MS fellows. These findings suggest that graduate training in clinical and translational research prepares graduates for the highly competitive field of clinical and translational research.