Development of a learning health system science competency assessment to guide training and proficiency assessment

Enrichment of core competencies to maximize health system impact: An analysis of an embedded research training program

Introduction

The Health System Impact (HSI) Fellowship is an embedded research training program that aims to prepare doctoral trainees and postdoctoral fellows for stronger career readiness and greater impact as emerging leaders within and beyond the academy, including in learning health systems (LHS). The program supports fellows to develop 10 leadership and research competencies that comprise the Enriched Core Competency Framework in Health Services and Policy Research through a combination of experiential learning, mentorship, and professional development training. This study tracks competency development of HSI fellows over time and examines fellows' perspectives on which program design elements support their competency development.

Methods

A competency assessment tool developed for the program was independently completed by 95 postdoctoral and 36 doctoral fellows (self-assessments) and their respective 203 dyad (academic and health system) supervisors in the 2017 to 2019 program cohorts, who independently rated the strength of fellows' 10 competencies at baseline and several points thereafter. Competency strength ratings were analyzed to understand change over time and differences in ratings across groups (between fellows' sex, supervisor type, and supervisor vs. fellow). Program design element ratings were examined to understand perspectives on their contribution toward fellows' competency development.

Results

Fellows' competency strength significantly improved in all 10 domains over time, based on independent assessments by the fellows and their dyad supervisors. Supervisors tended to rate the fellows' competency strength higher than the fellows did. Differences in competency ratings between male and female fellows (self-assessments) and between academic and health system supervisors were either negligble or not significant. Fellows identified all nine program design elements as enriching their competency development.

Conclusion

The HSI Fellowship provides an opportunity for fellows to develop the full suite of enriched core competencies and to prepare a cadre of emerging leaders with the skills and experience to contribute to the advancement of LHS.

Assessment of learning health system science competency in the equity and justice domain

Seven knowledge domains were originally defined for the learning health system (LHS) scientist. To assess proficiency in each of these domains, we developed and published an assessment tool for use by emerging LHS scientists and training programs. (LHS, October 2022). In mid-2022, the AHRQ adopted an eighth LHS knowledge domain, Equity and Justice. The addition of this eighth domain emphasizes the importance and centrality of equity in the LHS and improvement science. To extend our prior LHS competency assessment, we developed a proficiency assessment for the new equity and justice domain. Content experts and trainees iteratively defined, reviewed, and edited the assessment criteria. The items were developed by trainees and experts at one LHS training center with experience conducting research focused on healthcare inequities among marginalized populations. The proficiency assessment criteria for the Equity domain apply the same four levels of mastery: "no exposure," "foundational awareness," "emerging," and "proficient" as were used for original competencies. LHS training programs can use these proficiency criteria to monitor skills among emerging scientists across the eight domains, with particular attention to equity and justice.

Development and use of a novel tool for assessing and improving researcher embeddedness in learning health systems and applied system improvements

This paper outlines the development, deployment and use, and testing of a tool for measuring and improving healthcare researcher embeddedness - i.e., being connected to and engaged with key leverage points and stakeholders in a health system. Despite the widely acknowledged importance of embeddedness for learning health systems and late-stage translational research, we were not aware of useful tools for addressing and improving embeddedness in scholar training programs. We developed the MN-LHS Embeddedness Tool covering connections to committees, working groups, leadership, and other points of contact across four domains: patients and caregivers; local practice (e.g., operations and workflows); local institutional research (e.g., research committees and agenda- or initiative-setting groups); and national (strategic connections within professional groups, conferences, etc.). We used qualitative patterns and narrative findings from 11 learning health system training program scholars to explore variation in scholar trajectories and the embeddedness tool's usefulness in scholar professional development. Tool characteristics showed moderate evidence of construct validity; secondarily, we found significant differences in embeddedness, as a score, from baseline through program completion. The tool has demonstrated simple, practical utility in making embeddedness an explicit (rather than hidden) part of applied and learning health system researcher training, alongside emerging evidence for validity.