Medical education practice-based research networks: Facilitating collaborative research

Level of Supervision for the Entrustable Professional Activities Common to General Pediatrics and the Subspecialties Decreases from Residency to Fellowship

OBJECTIVES

To compare level of supervision (LOS) ratings of graduating pediatric residents with their assessments as fellows for the five Entrustable Professional Activities (EPAs) common to general pediatrics and the subspecialties and to determine if the difference between ratings from residency to fellowship is less for the QI and Practice Management EPAs, since the skills needed to perform these may be less context-dependent.

METHODS

We compared ratings of graduating residents with their assessments as fellows using LOS data from two sequential EPA studies.

RESULTS

There were 65 ratings from 41 residents at the first fellow assessment. At graduation, most residents needed little to no supervision for all EPAs with 94% (61/65) of ratings level four or five. In contrast, only 5/65 (8%) of the first fellow assessments were level four or five. The ratings difference for the QI and Practice Management EPAs was similar to the others.

CONCLUSIONS

LOS ratings for the EPAs common to generalists and subspecialists reset as residents become fellows. There was no evidence that the QI and Practice Management EPAs are less context-dependent. This study provides additional validity evidence for using these LOS scales to assess trainees in pediatric residency and fellowship.

Tips and tricks for successfully conducting a multicenter study

Multicenter studies play a crucial role in medical research and advancement, facilitating the application of new knowledge to clinical practice. These studies are associated with multiple benefits but are more complex than those involving a single center. With the philosophy that most of the qualities required to lead a multicenter study depend on attitude and can be learned, developed, and improved, in this manuscript, we share with the reader a series of recommendations that we consider important for successfully conducting such studies. The tips and tricks that will be discussed in detail are as follows: effectively leading the project; clearly defining viable and relevant objectives; designing a clear and detailed protocol; carefully selecting centers and collaborating investigators; meticulously designing the case report form; centrally managing the project efficiently; maintaining fluent communication with investigators; and, finally, designing a clear authorship policy and ensuring the appropriate publication of the study results. We hope that these suggestions encourage potential researchers to conduct multicenter studies, thereby collectively enhancing the quality of research and its application to clinical practice.

The rapid growth and expansion of Master of health professions education (MHPE) programs: A mixed-methods study of international curricular trends and guidelines for programs

PURPOSE

This study explores international trends and standards of Master's degree programs through a comprehensive environmental scan and focus group interviews to understand curricular structure, content, program director expectations, educational context, and future directions.

METHOD

Authors conducted a two-phase mixed-methods sequential explanatory design to conduct the environmental scan (phase 1), and subsequently conducting focus groups (phase 2) with program directors. A population list of Master's programs was used to generate a sampling frame, considering the geographic region (continent) and institution type (university, organization, public institution). Qualitative data were coded to analyze the breadth and depth of courses. Three one-hour virtual focus group interviews were conducted with ten program directors.

RESULTS

The population list of 159 Masters programs worldwide was used to create a sample for analysis in the environmental scan (n = 46 Masters programs), representing programs from North America, Europe, Australia, and South Africa. Most programs (39%) delivered their courses online, with 20% exclusively offering an in-person program. Focus group participants indicated expectations of graduates, context in which they learn, as well as future directions for improving health professions education graduate programs.

CONCLUSION

Program directors should consider programmatic aims, localized needs, and quality/standard of the program in designing Masters programs, with individualized growth opportunities for learners.

Improving Graduate Medical Education by Aggregating Data Across the Medical Education Continuum

ABSTRACT

Meaningful improvements to graduate medical education (GME) have been achieved in recent decades, yet many GME improvement pilots have been small trials without rigorous outcome measures and with limited generalizability. Thus, lack of access to large-scale data is a key barrier to generating empiric evidence to improve GME. In this article, the authors examine the potential of a national GME data infrastructure to improve GME, review the output of 2 national workshops on this topic, and propose a path toward achieving this goal.The authors envision a future where medical education is shaped by evidence from rigorous research powered by comprehensive, multi-institutional data. To achieve this goal, premedical education, undergraduate medical education, GME, and practicing physician data must be collected using a common data dictionary and standards and longitudinally linked using unique individual identifiers. The envisioned data infrastructure could provide a foundation for evidence-based decisions across all aspects of GME and help optimize the education of individual residents.Two workshops hosted by the National Academies of Sciences, Engineering, and Medicine Board on Health Care Services explored the prospect of better using GME data to improve education and its outcomes. There was broad consensus about the potential value of a longitudinal data infrastructure to improve GME. Significant obstacles were also noted.Suggested next steps outlined by the authors include producing a more complete inventory of data already being collected and managed by key medical education leadership organizations, pursuing a grass-roots data sharing pilot among GME-sponsoring institutions, and formulating the technical and governance frameworks needed to aggregate data across organizations.The power and potential of big data is evident across many disciplines, and the authors believe that harnessing the power of big data in GME is the best next step toward advancing evidence-based physician education.

Developing Evidence for Equitable Assessment Characteristics Based on Clinical Learner Preferences Using Discrete Choice Experiments

PURPOSE

Medical education is only beginning to explore the factors that contribute to equitable assessment in clinical settings. Increasing knowledge about equitable assessment ensures a quality medical education experience that produces an excellent, diverse physician workforce equipped to address the health care disparities facing patients and communities. Through the lens of the Anti-Deficit Achievement framework, the authors aimed to obtain evidence for a model for equitable assessment in clinical training.

METHOD

A discrete choice experiment approach was used which included an instrument with 6 attributes each at 2 levels to reveal learner preferences for the inclusion of each attribute in equitable assessment. Self-identified underrepresented in medicine (UIM) and not underrepresented in medicine (non-UIM) (N = 306) fourth-year medical students and senior residents in medicine, pediatrics, and surgery at 9 institutions across the United States completed the instrument. A mixed-effects logit model was used to determine attributes learners valued most.

RESULTS

Participants valued the inclusion of all assessment attributes provided except for peer comparison. The most valued attribute of an equitable assessment was how learner identity, background, and trajectory were appreciated by clinical supervisors. The next most valued attributes were assessment of growth, supervisor bias training, narrative assessments, and assessment of learner's patient care, with participants willing to trade off any of the attributes to get several others. There were no significant differences in value placed on assessment attributes between UIM and non-UIM learners. Residents valued clinical supervisors valuing learner identity, background, and trajectory and clinical supervisor bias training more so than medical students.

CONCLUSIONS

This study offers support for the components of an antideficit-focused model for equity in assessment and informs efforts to promote UIM learner success and guide equity, diversity, and inclusion initiatives in medical education.

Relationship between epa level of supervision with their associated subcompetency milestone levels in pediatric fellow assessment

BACKGROUND

Entrustable Professional Activities (EPA) and competencies represent components of a competency-based education framework. EPAs are assessed based on the level of supervision (LOS) necessary to perform the activity safely and effectively. The broad competencies, broken down into narrower subcompetencies, are assessed using milestones, observable behaviors of one's abilities along a developmental spectrum. Integration of the two methods, accomplished by mapping the most relevant subcompetencies to each EPA, may provide a cross check between the two forms of assessment and uncover those subcompetencies that have the greatest influence on the EPA assessment.

OBJECTIVES

We hypothesized that 1) there would be a strong correlation between EPA LOS ratings with the milestone levels for the subcompetencies mapped to the EPA; 2) some subcompetencies would be more critical in determining entrustment decisions than others, and 3) the correlation would be weaker if the analysis included only milestones reported to the Accreditation Council for Graduate Medical Education (ACGME).

METHODS

In fall 2014 and spring 2015, the Subspecialty Pediatrics Investigator Network asked Clinical Competency Committees to assign milestone levels to each trainee enrolled in a pediatric fellowship for all subcompetencies mapped to 6 Common Pediatric Subspecialty EPAs as well as provide a rating for each EPA based upon a 5-point LOS scale.

RESULTS

One-thousand forty fellows were assessed in fall and 1048 in spring, representing about 27% of all fellows. For each EPA and in both periods, the average milestone level was highly correlated with LOS (rho range 0.59-0.74; p < 0.001). Correlations were similar when using a weighted versus unweighted milestone score or using only the ACGME reported milestones (p > 0.05).

CONCLUSIONS

We found a strong relationship between milestone level and EPA LOS rating but no difference if the subcompetencies were weighted, or if only milestones reported to the ACGME were used. Our results suggest that representative behaviors needed to effectively perform the EPA, such as key subcompetencies and milestones, allow for future language adaptations while still supporting the current model of assessment. In addition, these data provide additional validity evidence for using these complementary tools in building a program of assessment.

Well-being factors associated with confidence in providing calm, compassionate care in pediatric residents

OBJECTIVE

Engagement in calm, compassionate care (CCC) is important in fostering patient-centered care. We aimed to study factors which predict confidence in providing CCC in pediatric residents.

METHODS

We performed a retrospective, multi-center, cohort study utilizing data from the Pediatric Resident Burnout and Resilience Study (PRB-RSC) from 2016 to 2018. The Calm Compassionate Care Scale (CCCS) was used to assess confidence in delivering CCC. We examined cross-sectional associations between CCC, demographic measures, programmatic features, and psychological scales.

RESULTS

The following showed significant positive associations with confidence in providing CCC: Cognitive and Affective Mindfulness Scale, Neff's Self Compassion, Patient Reported Outcomes Measures-mental health, and Interpersonal Reactivity Index-empathetic concern. For the Maslach Burnout Index subscales, decreased personal accomplishment, increased emotional exhaustion, and increased depersonalization showed significant negative associations.

CONCLUSIONS

We found that several well-being measures and an increased sense of personal accomplishment are associated with increased confidence in providing CCC. These findings underscore the interrelatedness of these measures and highlight the importance of personal accomplishment as a positive factor in trainee development.

PRACTICE IMPLICATIONS

Program level interventions that decrease trainee burnout and enhance resiliency as well as support trainees' development of empathy and compassion may help trainees develop skills that promote patient-centered, compassionate care.

Measuring the Learning Orientation Fostered by Pediatric Residency Programs With the Graduate Medical Education Learning Environment Inventory Instrument

INTRODUCTION

Mastery learning orientation, conceptualized as a growth mindset, can be beneficial to learners in medical education and is supported by a program...s learning environment. Currently, there are no instruments which reliably assess the learning orientation of a graduate medical education program...s learning environment.

OBJECTIVE

To explore the reliability and validity of the Graduate Medical Education Learning Environment Inventory (GME-LEI).

METHODS

Leaders of the Education in Pediatrics Across the Continuum (EPAC) project revised Krupat...s Educational Climate Inventory to create the GME-LEI. We investigated the GME-LEI...s reliability and validity through confirmatory factor and parallel factor analyses and calculated Cronbach...s alpha for each subscale. We compared mean subscale scores between residents in traditional programs and the EPAC project. As EPAC is known to foster a mastery-focused learning orientation, we hypothesized differences detected between resident groups would strengthen the instrument...s validity.

RESULTS

One hundred and twenty-seven pediatric residents completed the GME-LEI. The final 3-factor model was an acceptable fit to the data, and Cronbach...s alpha for each subscale was acceptable (Centrality: 0.87; Stress: 0.73; Support: 0.77). Mean scores on each subscale varied by program type (EPAC vs traditional) with EPAC residents reporting statistically significant higher scores in the Centrality of Learning subscale (2.03, SD 0.30, vs 1.79, SD 0.42; P.ß=.ß.023; scale of 1...4).

CONCLUSIONS

The GME-LEI reliably measures 3 distinct aspects of the GME learning environment with respect to learning orientation. The GME-LEI may be used to help programs better monitor the learning environment and make changes to support mastery-oriented learning.

Build It and They Shall Come: Medical Education Communities of Practice

Background

Producing scholarship in education is essential to the career development of a clinician-educator. Challenges to scholarly production include a lack of resources, time, expertise, and collaborators.

Objective

To develop communities of practice for education scholarship through an international society to increase community and academic productivity.

Methods

We developed multi-institutional scholarship pods within the American Thoracic Society through the creation of a working group (2017-2019). Pods met virtually, and meetings were goal focused to advance education scholarship within their area of interest. To understand the impact of these scholarship pods, we surveyed pod leaders and members in 2021 and analyzed the academic productivity of each pod via a survey of pod leaders and a review of the PubMed index.

Results

Nine pods were created, each with an assigned educational topic. The survey had a response rate of 76.6%. The perceived benefits were the opportunity to meet colleagues with similar interests at other institutions, production of scholarly work, and engagement in new experiences. The main challenges were difficulty finding times to meet because of competing clinical demands and aligning times among pod members. Regarding academic productivity, eight publications, four conference presentations, and one webinar/podcast were produced by six of the nine pods.

Conclusion

The development of communities of practice resulted in increased multi-site collaboration, with boosted academic productivity as well as an enhanced sense of belonging. Multiple challenges remain but can likely be overcome with accountability, early discussion of roles and expectations, and clear delegation of tasks and authorship.

The APPD Longitudinal Educational Assessment Research Network's First Decade

ABSTRACT

In 2009, the Association of Pediatric Program Directors (APPD) Longitudinal Educational Assessment Research Network (LEARN), a national educational research network, was formed. We report on evaluation of the network after 10 years of operation by reviewing program context, input, processes, and products to measure its progress in performing educational research that advances training of future pediatricians. Historical changes in medical education shaped the initial development of the network. APPD LEARN now includes 74% (148 of 201) of US Pediatric residency programs and has recently incorporated a network of Pediatric subspecialty fellowship programs. At the time of this evaluation, APPD LEARN had approved 19 member-initiated studies and 14 interorganizational studies, resulting in 23 peer-reviewed publications, numerous presentations, and 7 archived sharable data sets. Most publications focused on how and when interventions work rather than whether they work, had high scores for reporting rigor, and included organizational and objective performance outcomes. Member program representatives had positive perceptions of APPD LEARN's success, with most highly valuing participation in research that impacts training, access to expertise, and the ability to make authorship contributions for presentations and publication. Areas for development and improvement identified in the evaluation include adopting a formal research prioritization process, infrastructure changes to support educational research that includes patient data, and expanding educational outreach within and outside the network. APPD LEARN and similar networks contribute to high-rigor research in pediatric education that can lead to improvements in training and thereby the health care of children.

Prediction of the information processing speed performance in multiple sclerosis using a machine learning approach in a large multicenter magnetic resonance imaging data set

Many patients with multiple sclerosis (MS) experience information processing speed (IPS) deficits, and the Symbol Digit Modalities Test (SDMT) has been recommended as a valid screening test. Magnetic resonance imaging (MRI) has markedly improved the understanding of the mechanisms associated with cognitive deficits in MS. However, which structural MRI markers are the most closely related to cognitive performance is still unclear. We used the multicenter 3T-MRI data set of the Italian Neuroimaging Network Initiative to extract multimodal data (i.e., demographic, clinical, neuropsychological, and structural MRIs) of 540 MS patients. We aimed to assess, through machine learning techniques, the contribution of brain MRI structural volumes in the prediction of IPS deficits when combined with demographic and clinical features. We trained and tested the eXtreme Gradient Boosting (XGBoost) model following a rigorous validation scheme to obtain reliable generalization performance. We carried out a classification and a regression task based on SDMT scores feeding each model with different combinations of features. For the classification task, the model trained with thalamus, cortical gray matter, hippocampus, and lesions volumes achieved an area under the receiver operating characteristic curve of 0.74. For the regression task, the model trained with cortical gray matter and thalamus volumes, EDSS, nucleus accumbens, lesions, and putamen volumes, and age reached a mean absolute error of 0.95. In conclusion, our results confirmed that damage to cortical gray matter and relevant deep and archaic gray matter structures, such as the thalamus and hippocampus, is among the most relevant predictors of cognitive performance in MS.

Fellow Entrustment for the Common Pediatric Subspecialty Entrustable Professional Activities Across Subspecialties

OBJECTIVE

To determine the relationship between level of supervision (LOS) ratings for the Common Pediatric Subspecialty Entrustable Professional Activities (EPAs) with their associated subcompetency milestones across subspecialties and by fellowship training year.

METHODS

Clinical Competency Committees (CCCs) in 14 pediatric subspecialties submitted LOS ratings for 6 Common Subspecialty EPAs and subcompetency milestone levels mapped to these EPAs. We examined associations between these subcompetency milestone levels and LOS ratings across subspecialty training year by fitting per-EPA linear mixed effects models, regressing LOS rating on milestone level and on training year.

RESULTS

CCCs from 211 pediatric fellowship programs provided data for 369 first, 336 second, and 331 third year fellows. Mean subcompetency milestone levels increased similarly among subspecialties for most EPAs compared with the reference, Adolescent Medicine. Mean subcompetency milestones mapped to each EPA and mean EPA LOS ratings generally increased by training year across all subspecialties.

CONCLUSIONS

Subcompetency milestones levels mapped to each Common Subspecialty EPA and the EPA LOS ratings increase similarly across subspecialties and by training year, providing validity evidence for using EPA LOS to assess pediatric subspecialty trainee performance. This study supports the development of tools to facilitated the CCC evaluation process across all pediatric subspecialties.

Exploring how national educational organizations can promote educational research amongst members: a survey-based study

BACKGROUND

Engagement of academic faculty in research remains low. While barriers to research have been explored, there are no data on how national organizations can help overcome these barriers. Our study explored faculty satisfaction and motivational drivers for engagement with research opportunities offered by the Council on Medical Student Education in Pediatrics (COMSEP), an organization of pediatric medical educators, and characterize strategies perceived by faculty to promote the use of these opportunities.

METHODS

In 2021, 5 survey questions were administered to faculty members of COMSEP to explore satisfaction with COMSEP's research offerings, the perceived value of educational research, and the facilitators, barriers and potential opportunities for COMSEP to promote research. Clark's Commitment and Necessary Effort model on motivation served as the theoretical framework for our study, which explores motivation, self-efficacy and contextual factors influencing an individual's pursuit of goals. Chi-square analysis and Wilcoxon Signed Ranks Test were used to compare categorical and scaled variables among groups who did and did not avail of COMSEP's research offerings.

RESULTS

90 (25%) of 360 recipients responded. 61% expressed satisfaction with COMSEP's research offerings. 68% indicated research was an expectation of their academic appointment, that education was their primary research focus (74%) and that they did not have other research opportunities that met their needs (58%). Of respondents, 75.7% of females had submitted a proposal compared to 60% of non-responders who were females. The comparison by gender was not statistically significant. Exploration by academic rank revealed that 35% of instructor/assistant professors had submitted a proposal compared to 65% of associate professors/professors (p =.05). Barriers leading to non-submission to any of the offerings included having too much other work, lack of enjoyment in writing and inability to find mentors. Respondents endorsed the importance of several strategies to promote engagement in research-skill building opportunities, personalized consultations and increased funding.

CONCLUSIONS

Faculty educators value the importance of educational research and recognize that research opportunities offered by COMSEP address an unmet need, but express ambivalence in the enjoyment of writing (reflecting their mood), and endorse structural barriers, that are amenable to change, affecting their personal agency.

Social network analysis of publication collaboration of accelerating change in MedEd consortium

INTRODUCTION

The American Medical Association formed the Accelerating Change in Medical Education Consortium through grants to effect change in medical education. The dissemination of educational innovations through scholarship was a priority. The objective of this study was to explore the patterns of collaboration of educational innovation through the consortium's publications.

METHOD

Publications were identified from grantee schools' semi-annual reports. Each publication was coded for the number of citations, Altmetric score, domain of scholarship, and collaboration with other institutions. Social network analysis explored relationships at the midpoint and end of the grant.

RESULTS

Over five years, the 32 Consortium institutions produced 168 publications, ranging from 38 papers from one institution to no manuscripts from another. The two most common domains focused on health system science (92 papers) and competency-based medical education (30 papers). Articles were published in 54 different journals. Forty percent of publications involved more than one institution. Social network analysis demonstrated rich publishing relationships within the Consortium members as well as beyond the Consortium schools. In addition, there was growth of the network connections and density over time.

CONCLUSION

The Consortium fostered a scholarship network disseminating a broad range of educational innovations through publications of individual school projects and collaborations.

Parenthood and Parental Leave Decisions in Pediatric Residency

OBJECTIVES

The demands of residency training may impact trainees' decision to have children. We examined characteristics of pediatric residents' decisions regarding childbearing, determinants of resident parental leave, and associations with well-being.

METHODS

A survey of 845 pediatric residents at 13 programs was conducted between October 2019 and May 2020. Survey items included demographics, desire for future children, and logistics of parental leave. Outcomes included parental leave length, burnout and depression screening results, satisfaction with duration of breastfeeding, and satisfaction with parental leave and parenthood decisions.

RESULTS

Seventy-six percent (639 of 845) of residents responded to the survey. Fifty-two percent (330) of respondents reported delaying having children during residency, and 29% (97) of those were dissatisfied with their decision to do so. Busy work schedule (89.7%), finances (50.9%), and a desire not to extend residency (41.2%) were the most common reasons for delay. Of respondents, 16% were parents and 4% were pregnant or had pregnant partners. Sixty-one parental leaves were reported, and 67% of parents reported dissatisfaction with leave length. The most frequently self-reported determinant of leave duration was the desire not to extend residency training (74%). Program mean leave length was negatively associated with burnout, measured as a dichotomous outcome (odds ratio = 0.81 [95% confidence interval 0.68-0.98]; P = .02).

CONCLUSIONS

Many pediatric trainees delay parenthood during residency and are not satisfied with their decision to do so. Pediatric resident parental leave remains short and variable in duration, despite the positive association between longer leaves and overall well-being.

Learner Levels of Supervision Across the Continuum of Pediatrics Training

PURPOSE

To describe trajectories in level of supervision ratings for linked entrustable professional activities (EPAs) among pediatric learners in medical school, residency, fellowship.

METHOD

The authors performed secondary analyses of 3 linked datasets of level of supervision ratings for the Core EPAs for Entering Residency, the General Pediatrics EPAs, and the Subspecialty Pediatrics EPAs. After identifying 9 activities in common across training stages and aligning the level of entrustment-supervision scales across the datasets, piecewise ordinal and linear mixed effects models were fitted to characterize trajectories of supervision ratings.

RESULTS

Within each training period, learners were rated as needing less supervision over time in each activity. When transitioning from medical school to residency or during the first year of residency, learners were rated as needing greater supervision in activities related to patient management, teamwork, emergent care, and public health/QI than in earlier periods. When transitioning from residency to fellowship, learners were always rated as needing greater supervision than they had been accorded at the end of residency and sometimes even more than they had been accorded at the start of residency.

CONCLUSIONS

Although development over training is often imagined as continuous and monotonically increasing competence, this study provides empirical evidence supporting the idea that entrustment is a set of discrete decisions. The relaxation of supervision in training is not a linear process. Even with a seamless curriculum, supervision is tightly bound to the training setting. Several explanations for these findings are discussed.

Burnout and Association With Resident Performance as Assessed by Pediatric Milestones: An Exploratory Study

INTRODUCTION

Prior work demonstrating that burnout is associated with decreased performance in medical trainees has relied on self-report and/or single-site studies. We explored the relationship between burnout status and Milestones-based scores in pediatric residents nationally.

METHODS

In April to June 2016, we confidentially surveyed residents using the Maslach Burnout Inventory. Separately, programs submitted resident Milestones scores in June 2016. We examined the relationship between burnout and performance as assessed by Milestones scores for each domain of competence. We performed multivariate analysis to determine which components of burnout (depersonalization [DP], emotional exhaustion, and lack of personal accomplishment [PA]) were most impactful.

RESULTS

About 1494 of 2368 (63%) residents at 32 programs completed the Maslach Burnout Inventory and had Milestones scores submitted. Residents who scored positive for burnout scored lower in all Milestones domains. Subgroup analysis demonstrated that this association was only significant (P < .05) in the post-graduate year 1 (PGY1) categorical pediatric cohort. In the PGY1 residents (n = 442), those positive for burnout had lower Milestones scores in patient care (PC) (2.78 vs 2.98), systems-based practice (2.69 vs 2.87), practice-based learning and improvement (2.77 vs 2.93), professionalism (3.09 vs 3.24), and interpersonal and communication skills (2.95 vs 3.12), but not medical knowledge. Multivariate analysis demonstrated that, in PGY1 residents, lower PC score was associated with lower PA and higher DP.

CONCLUSIONS

Burnout is associated with decreased Milestones performance for pediatric PGY1 residents. DP and low PA were associated with lower PC scores in PGY1 residents. Future research should address whether strategies to mitigate burnout improve PGY1 performance.

Faculty and Peer Support During Pediatric Residency: Association With Performance Outcomes, Race, and Gender

PURPOSE

To examine the association of resident perception of colleague and faculty support with performance, as measured by milestones-based competency scores, exploring associations between race and gender and perception of support and milestone scoring.

METHODS

Resident satisfaction was measured using an annual survey of residents at 49 pediatric residency programs in 2016, 2017, and 2018. Satisfaction with colleague and faculty support was measured using Likert scale survey questions. Pediatric Milestone Competency scores were obtained from the Association of Pediatric Program Directors' Longitudinal Educational Assessment Research Network. Analysis included linear fixed-effects models to examine the relationship between support satisfaction, race, gender, and spring milestone scores.

RESULTS

Over 60% of eligible residents responded to the survey. The majority of residents were satisfied with colleague and faculty support, with those identifying as Asian or underrepresented in medicine (URM) reporting lower rates of satisfaction than White peers. Residents satisfied with colleague support had higher milestone scores compared to those with a neutral degree of satisfaction. Residents reporting dissatisfaction with colleague and faculty support had lower milestone scores in most competency domains. Residents identifying as URM had lower milestone scores than White residents, which was partially mediated by lower rates of support satisfaction.

CONCLUSIONS

Resident satisfaction with colleague and faculty support correlates with milestone performance. In particular, dissatisfied residents have lower scores than those who are neutral or satisfied. Racial inequities in resident milestone scores may be partially driven by lower rates of support satisfaction among underrepresented residents.

Diversity in health professional education scholarship: a document analysis of international author representation in leading journals

OBJECTIVES

The global distribution of health professionals and associated training programmes is wide but prior study has demonstrated reported scholarship of teaching and learning arises from predominantly Western perspectives.

DESIGN

We conducted a document analysis to examine authorship of recent publications to explore current international representation.

DATA SOURCES

The table of contents of seven high-impact English-language health professional education journals between 2008 and 2018 was extracted from Embase.

ELIGIBILITY CRITERIA

The journals were selected according to highest aggregate ranking across specific scientific impact indices and stating health professional education in scope; only original research and review articles from these publications were included for analysis.

DATA EXTRACTION AND SYNTHESIS

The table of contents was extracted and eligible publications screened by independent reviewers who further characterised the geographic affiliations of the publishing research teams and study settings (if applicable).

RESULTS

A total 12 018 titles were screened and 7793 (64.8%) articles included. Most were collaborations (7048, 90.4%) conducted by authors from single geographic regions (5851, 86%). Single-region teams were most often formed from countries in North America (56%), Northern Europe (14%) or Western Europe (10%). Overall lead authorship from Asian, African or South American regions was less than 15%, 5% and 1%, respectively. Geographic representation varied somewhat by journal, but not across time.

CONCLUSIONS

Diversity in health professional education scholarship, as marked by nation of authors' professional affiliations, remains low. Under-representation of published research outside Global North regions limits dissemination of novel ideas resulting in unidirectional flow of experiences and a concentrated worldview of teaching and learning.

The Hiker's Guide to the RIME Supplement: Choosing Directions in Research

In this Commentary, the authors seek to build on prior RIME commentaries by considering how researchers transition from worldviews, focal lengths, and research goals to research directions and methodological choices. The authors use the analogy of a hiker to illustrate how different researchers studying a similar phenomenon can choose among different research directions, which lead down different paths and offer different perspectives on a problem. Following the hiker analogy, the authors use the "Research Compass" to categorize the 15 research papers included in the 2020 Research in Medical Education supplement according to their research aim and corresponding methodological approach. The authors then discuss implications of the relative balance of these study types within this supplement and within health professions education research at large, emphasizing the critical importance of studying a topic from multiple vantage points to construct a richer and more nuanced understanding of health professions education challenges. The authors conclude by recognizing the challenges we face in the current era of COVID-19 and by calling health professions education researchers and practitioners to continue our collective efforts to improve learner education and patient care, as we together navigate the unfamiliar terrain of the present day.

The Emergence and Spread of Practice-Based Medical Education Research Networks

Educational research networks leverage shared goals and common infrastructure to overcome traditional barriers to medical education research, including small sample sizes, lack of generalizability, need for expertise in statistical analysis, and limitations on data sharing. The diversity of extant network models today is exciting and provides a set of common options and challenges that newly emerging networks can expect. These include decisions about network focus, organization of data, sampling strategies, funding, and governance. Common challenges include managing authorship, human subjects protection rules, data use agreements, and statistical disclosure control. Medical education research networks both advance the field and develop the researchers who participate in them. The authors repeat the call that they and others have made for the development of networks to promulgate best practices and coordinate multinetwork (multinational, multispecialty, and cross-curriculum) studies.

Narrative Performance Level Assignments at Initial Entrustment and Graduation: Integrating EPAs and Milestones to Improve Learner Assessment

PURPOSE

To determine which narrative performance level for each general pediatrics entrustable professional activity (EPA) reflects the minimum level clinical competency committees (CCCs) felt should be associated with graduation as well as initial entrustment and compare expected narrative performance levels (ENPLs) for each EPA with actual narrative performance levels (ANPLs) assigned to residents at initial entrustment.

METHOD

A series of 5 narratives, corresponding to the 5 milestone performance levels, were developed for each of the 17 general pediatrics EPAs. In academic year (AY) 2015-2016, the CCCs at 22 Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network member sites reported ENPLs for initial entrustment and at time of graduation. From AYs 2015-2016 to 2017-2018, programs reported ANPLs for initial entrustment decisions. ENPLs and ANPLs were compared using a logistic mixed effects model.

RESULTS

ENPLs for graduation and entrustment were most often level 3 (competent) followed by level 4 (proficient). For 8 EPAs, the ENPLs for graduation and entrustment were the same. For the remaining 9, some programs would entrust residents before graduation or graduate them before entrusting them. There were 4,266 supervision level reports for initial entrustment for which an ANPL was provided. ANPLs that were lower than the ENPLs were significantly more likely to be assigned to the medical home-well child (OR = 0.39; 95% CI: 0.26-0.57), transition to adult care (OR = 0.43; 95% CI: 0.19-0.95), behavioral or mental health (OR = 0.36; 95% CI: 0.18-0.71), make referrals (OR = 0.31; 95% CI: 0.17-0.55), lead a team (OR = 0.34; 95% CI: 0.22-0.52), and handovers (OR = 0.18; 95% CI: 0.09-0.36) EPAs.

CONCLUSIONS

CCCs reported lower ENPLs for graduation than for entrustment for 5 EPAs, possibly indicating curricular gaps that milestones and EPAs could help identify.

Scholarly Activity Training During Pediatric Residency: How Well Aligned Are Program Directors and Residents?

OBJECTIVE

Training residents to be scholars is endorsed by leading medical education organizations. Our previous research suggests that the scholarly activity (SA) training experience is highly variable across pediatric residency programs, and residents and program directors (PDs) are generally dissatisfied. Understanding how PD and resident views align can help programs better guide resource allocation and improvement efforts.

METHODS

We conducted cross-sectional surveys of second and third year pediatric residents and PDs at 22 diverse US categorical programs. We compared resident and PD responses to SA training beliefs, barriers, and satisfaction by 2-proportion z tests. We used descriptive statistics to describe resident responses in relation to same-institution PD responses.

RESULTS

About 464 of 771 residents (60.2%) and 22 PDs (100%) responded. Across programs, PDs more strongly agreed that all residents should participate in SA (59% of PDs [n = 13] versus 27% of residents [n = 127], P = .002). Residents more strongly believed all residents should have protected SA time (91% of residents [n = 424] versus 68% of PDs [n = 15], P = .001). PDs more strongly perceived gaps in other resources as barriers, including lack of funding to conduct or present SA (P < .001, P = .02), lack of statistical support (P = .03), and lack of faculty mentorship (P < .001). Within program concordance was low.

CONCLUSIONS

Discordance exists between PDs and residents with respect to SA participation and necessary resources, particularly, protected time. Programs must help residents identify when SA can be accomplished. Clearer national guidelines around SA training may also be necessary to reduce discordance and improve perceptions.

Longitudinal Assessment of Resident Performance Using Entrustable Professional Activities

IMPORTANCE

Entrustable professional activities (EPAs) are an emerging workplace-based, patient-oriented assessment approach with limited empirical evidence.

OBJECTIVE

To measure the development of pediatric trainees' clinical skills over time using EPA-based assessment data.

DESIGN, SETTING, AND PARTICIPANTS

Prospective cohort study of categorical pediatric residents over 3 academic years (2015-2016, 2016-2017, and 2017-2018) assessed on 17 American Board of Pediatrics EPAs. Residents in training at 23 pediatric residency programs in the Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network were included. Assessment was conducted by clinical competency committee members, who made summative assessment decisions regarding levels of supervision required for each resident and each EPA. Data were collected from May 2016 to November 2018 and analyzed from November to December 2018.

INTERVENTIONS

Longitudinal, prospective assessment using EPAs.

MAIN OUTCOMES AND MEASURES

Trajectories of supervision levels by EPA during residency training and how often graduating residents were deemed ready for unsupervised practice in each EPA.

RESULTS

Across the 5 data collection cycles, 1987 residents from all 3 postgraduate years in 23 residency programs were assigned 25 503 supervision level reports for the 17 general pediatrics EPAs. The 4 EPAs that required the most supervision across training were EPA 14 (quality improvement) on the 5-level scale (estimated mean level at graduation, 3.7; 95% CI, 3.6-3.7) and EPAs 8 (transition to adult care; mean, 7.0; 95% CI, 7.0-7.1), 9 (behavioral and mental health; mean, 6.6; 95% CI, 6.5-6.6), and 10 (resuscitate and stabilize; mean, 6.9; 95% CI, 6.8-7.0) on the expanded 5-level scale. At the time of graduation (36 months), the percentage of trainees who were rated at a supervision level corresponding to "unsupervised practice" varied by EPA from 53% to 98%. If performance standards were set to align with 90% of trainees achieving the level of unsupervised practice, this standard would be met for only 8 of the 17 EPAs (although 89% met this standard for EPA 17, performing the common procedures of the general pediatrician).

CONCLUSIONS AND RELEVANCE

This study presents initial evidence for empirically derived practice readiness and sets the stage for identifying curricular gaps that contribute to discrepancy between observed practice readiness and standards needed to produce physicians able to meet the health needs of the patient populations they serve. Future work should compare these findings with postgraduation outcomes data as a means of seeking validity evidence.

Agreement of Program Directors With Clinical Competency Committees for Fellow Entrustment

OBJECTIVES

Fellowship program directors (FPD) and Clinical Competency Committees (CCCs) both assess fellow performance. We examined the association of entrustment levels determined by the FPD with those of the CCC for 6 common pediatric subspecialty entrustable professional activities (EPAs), hypothesizing there would be strong correlation and minimal bias between these raters.

METHODS

The FPDs and CCCs separately assigned a level of supervision to each of their fellows for 6 common pediatric subspecialty EPAs. For each EPA, we determined the correlation between FPD and CCC assessments and calculated bias as CCC minus FPD values for when the FPD was or was not a member of the CCC. In addition, we examined the effect of program size, FPD understanding of EPAs, and subspecialty on the correlations. Data were obtained in fall 2014 and spring 2015.

RESULTS

A total of 1040 fellows were assessed in the fall and 1048 in the spring. In both periods and for each EPA, there was a strong correlation between FPD and CCC supervision levels (P < .001). The correlation was somewhat lower when the FPD was not a CCC member (P < .001). Overall bias in both periods was small.

CONCLUSIONS

The correlation between FPD and CCC assignment of EPA supervision levels is strong. Although slightly weaker when the FPD is not a CCC member, bias is small, so this is likely unimportant in determining fellow entrustment level. The similar performance ratings of FPDs and CCCs support the validity argument for EPAs as competency-based assessment tools.

Collaborative mentoring for effective medical research groups

This article was migrated. The article was marked as recommended. Research benefits professors and students, mentors and mentees, however, many Universities lack formal programs, especially in basic sciences such as anatomy. Faculty, many times, lack the time, resources, and a well-structured program. Mentoring and collaborative work, have played an important role in creating an effective environment that inspires its members into scientific production, enhances research skills, while gaining experience. The authors presented a student anatomy research group (Grupo de Investigación en Anatomía [GIA]) model that integrates faculty from basic and clinical fields through a collaborative mentoring and tightly organized structure that increases training, experience, and scientific output. The transformation stages described shows the progress, reporting the fundamental elements for integration of the model, with results presented at 16 years of experience of the group in the Anatomy department. Results evidence a steady increase in student/professor involvement, scientific publishing, presentations in meetings (national/international), and cites.

E-learning as valuable caregivers' support for people with dementia - A systematic review

Background

Present demographic trends show a considerable rise in elderly populations with aging disorders, such as dementia. The current article focused on the exploitation of e-learning as an informal support for caregivers of people with dementia and considered its benefits and limitations to provide proper and relevant care for this target group of people as well as maintain the quality of life of their caregivers.

Methods

The methodology of this study is based on a literature review of accessible peer-review articles from three recognized databases: Web of Science, Scopus, and PubMed. The findings of the selected studies were compared and evaluated.

Results

The findings showed that e-learning educational programs/courses helped caregivers feel more confident about dementia care, reduced their perceived stress and enhanced their feelings of empathy, understanding and concern.

Conclusions

The findings of this study reveal that the exploitation of e-learning as a support tool, especially for informal caregivers, in the management of dementia may be a promising method, but its implementation requires professional training of informal caregivers in the use of this technology. More evidence-based studies are needed on this topic.

Variability of Burnout and Stress Measures in Pediatric Residents: An Exploratory Single-Center Study From the Pediatric Resident Burnout-Resilience Study Consortium

Residency is a high-risk period for physician burnout. We aimed to determine the short-term stability of factors associated with burnout, application of these data to previous conceptual models, and the relationship of these factors over 3 months. Physician wellness questionnaire results were analyzed at 2 time points 3 months apart. Associations among variables within and across time points were analyzed. Logistic regression was used to predict burnout and compassionate care. A total of 74% of residents completed surveys. Over 3 months, burnout ( P = .005) and empathy ( P = .04) worsened. The most significant cross-sectional relationship was between stress and emotional exhaustion (time 1 r = 0.61, time 2 r = 0.68). Resilience was predictive of increased compassionate care and decreased burnout ( P < .05). Mindfulness was predictive of decreased burnout ( P < .05). Mitigating stress and fostering mindfulness and resilience longitudinally may be key areas of focus for improved wellness in pediatric residents. Larger studies are needed to better develop targeted wellness interventions.

Building a Community of Practice for Researchers: The International Network for Simulation-Based Pediatric Innovation, Research and Education

STATEMENT

The scope and breadth of simulation-based research is growing rapidly; however, few mechanisms exist for conducting multicenter, collaborative research. Failure to foster collaborative research efforts is a critical gap that lies in the path of advancing healthcare simulation. The 2017 Research Summit hosted by the Society for Simulation in Healthcare highlighted how simulation-based research networks can produce studies that positively impact the delivery of healthcare. In 2011, the International Network for Simulation-based Pediatric Innovation, Research and Education (INSPIRE) was formed to facilitate multicenter, collaborative simulation-based research with the aim of developing a community of practice for simulation researchers. Since its formation, the network has successfully completed and published numerous collaborative research projects. In this article, we describe INSPIRE's history, structure, and internal processes with the goal of highlighting the community of practice model for other groups seeking to form a simulation-based research network.

Does Incorporating a Measure of Clinical Workload Improve Workplace-Based Assessment Scores? Insights for Measurement Precision and Longitudinal Score Growth From Ten Pediatrics Residency Programs

PURPOSE

This study investigates the impact of incorporating observer-reported workload into workplace-based assessment (WBA) scores on (1) psychometric characteristics of WBA scores and (2) measuring changes in performance over time using workload-unadjusted versus workload-adjusted scores.

METHOD

Structured clinical observations and multisource feedback instruments were used to collect WBA data from first-year pediatrics residents at 10 residency programs between July 2016 and June 2017. Observers completed items in 8 subcompetencies associated with Pediatrics Milestones. Faculty and resident observers assessed workload using a sliding scale ranging from low to high; all item scores were rescaled to a 1-5 scale to facilitate analysis and interpretation. Workload-adjusted WBA scores were calculated at the item level using three different approaches, and aggregated for analysis at the competency level. Mixed-effects regression models were used to estimate variance components. Longitudinal growth curve analyses examined patterns of developmental score change over time.

RESULTS

On average, participating residents (n = 252) were assessed 5.32 times (standard deviation = 3.79) by different raters during the data collection period. Adjusting for workload yielded better discrimination of learner performance, and higher reliability, reducing measurement error by 28%. Projections in reliability indicated needing up to twice the number of raters when workload-unadjusted scores were used. Longitudinal analysis showed an increase in scores over time, with significant interaction between workload and time; workload also increased significantly over time.

CONCLUSIONS

Incorporating a measure of observer-reported workload could improve the measurement properties and the ability to interpret WBA scores.

Scholarly Activity Training During Residency: Are We Hitting the Mark? A National Assessment of Pediatric Residents

OBJECTIVE

Participation in scholarly activity (SA) is an Accreditation Council for Graduate Medical Education requirement. Our previous research with program directors (PDs) suggests that pediatric SA training is variable and suboptimal. To help programs better meet requirements, our objective was to understand the resident perspective regarding SA training, including factors associated with satisfaction and productivity.

METHODS

We conducted cross-sectional surveys of second- and third-year pediatric residents and PDs at 22 diverse programs in 2016. Surveys assessed resident demographics, career intentions, program characteristics, beliefs, barriers, satisfaction, and productivity, defined as SA accepted at a regional or national meeting, for publication, or grant funding. Data were analyzed using descriptive statistics and multivariable logistic regression.

RESULTS

A total of 464 (60.2%) of 771 residents and 22 PDs (100%) responded. Most residents believed that residents should participate in SA (n = 380, 81.9%). However, only 37.9% (n = 175) were extremely or very satisfied with their training. Residents who reported that training to conduct research (adjusted odds ratio [AOR]  = 1.9, 95% confidence interval [CI] 1.1-3.5), availability of a research curriculum (AOR = 1.9, 95% CI 1.2-3.1), and adequate faculty mentorship (AOR = 2.5, 95% CI 1.6-4.1) were not barriers were more satisfied. Protected time was associated with satisfaction (AOR = 1.7, 95% CI 1.1-2.7). A total of 43.8% of residents (n = 203) were productive. Productivity was associated with future plans to conduct research (AOR = 3.3, 95% CI 2.1-5.1).

CONCLUSIONS

Residents believe SA training is important. Dedicated program infrastructure, protected time, and adequate mentorship appear to be crucial to improving quality perceptions.

Conducting Quantitative Medical Education Research: From Design to Dissemination

Rigorous medical education research is critical to effectively develop and evaluate the training we provide our learners. Yet many clinical medical educators lack the training and skills needed to conduct high-quality medical education research. We offer guidance on conducting sound quantitative medical education research. Our aim is to equip readers with the key skills and strategies necessary to conduct successful research projects, highlighting new concepts and controversies in the field. We utilize Glassick's criteria for scholarship as a framework to discuss strategies to ensure that the research question of interest is worthy of further study and how to use existing literature and conceptual frameworks to strengthen a research study. Through discussions of the strengths and limitations of commonly used study designs, we expose the reader to particular nuances of these decisions in medical education research and discuss outcomes generally focused on, as well as strategies for determining the significance of consequent findings. We conclude with information on critiquing research findings and preparing results for dissemination to a broad audience. Practical planning worksheets and comprehensive tables illustrating key concepts are provided in order to guide researchers through each step of the process. Medical education research provides wonderful opportunities to improve how we teach our learners, to satisfy our own intellectual curiosity, and ultimately to enhance the care provided to patients.

Validity of Level of Supervision Scales for Assessing Pediatric Fellows on the Common Pediatric Subspecialty Entrustable Professional Activities

PURPOSE

Entrustable professional activities (EPAs) represent the routine and essential activities that physicians perform in practice. Although some level of supervision scales have been proposed, they have not been validated. In this study, the investigators created level of supervision scales for EPAs common to the pediatric subspecialties and then examined their validity in a study conducted by the Subspecialty Pediatrics Investigator Network (SPIN).

METHOD

SPIN Steering Committee members used a modified Delphi process to develop unique scales for six of the seven common EPAs. The investigators sought validity evidence in a multisubspecialty study in which pediatric fellowship program directors and Clinical Competency Committees used the scales to evaluate fellows in fall 2014 and spring 2015.

RESULTS

Separate scales for the six EPAs, each with five levels of progressive entrustment, were created. In both fall and spring, more than 300 fellows in each year of training from over 200 programs were assessed. In both periods and for each EPA, there was a progressive increase in entrustment levels, with second-year fellows rated higher than first-year fellows (P < .001) and third-year fellows rated higher than second-year fellows (P < .001). For each EPA, spring ratings were higher (P < .001) than those in the fall. Interrater reliability was high (Janson and Olsson's iota = 0.73).

CONCLUSIONS

The supervision scales developed for these six common pediatric subspecialty EPAs demonstrated strong validity evidence for use in EPA-based assessment of pediatric fellows. They may also inform the development of scales in other specialties.

Positive Change in Feedback Perceptions and Behavior: A 10-Year Follow-up Study

BACKGROUND

Providing and learning from feedback are essential components of medical education, and typically described as resistant to change. But given a decade of change in the clinical context in which feedback occurs, the authors asked if, and how, perceptions of feedback and feedback behaviors might have changed in response to contextual affordances.

METHODS

In 2017, the authors conducted a follow-up, ethnographic study on 2 general pediatric floors at the same children's hospital where another ethnographic study on a general pediatric floor was conducted in 2007. Data sources included (1) 21 and 34 hours of observation in 2007 and 2017, respectively, (2) 35 and 25 interviews with general pediatric attending physicians and residents in 2007 and 2017, respectively, and (3) a review of 120 program documents spanning 2007 to 2017. Data were coded and organized around 3 recommendations for feedback that were derived from 2007 data and served as standards for assessing change in 2017.

RESULTS

Data revealed progress in achieving each recommendation. Compared with 2007, participants in 2017 more clearly distinguished between feedback and evaluation; residents were more aware of in-the-moment feedback, and they had shifted their orientation from evaluation and grades to feedback and learning. Explanations for progress in achieving recommendations, which were derived from the data, pointed to institutional and national influences, namely, the pediatric milestones.

CONCLUSIONS

On the basis of follow-up, ethnographic data, changes in the clinical context of pediatric education may afford positive change in perceptions of feedback and feedback behavior and point to influences within and beyond the institution.

Integrating Education and Service in Pediatric Residency Training: Results of a National Survey

OBJECTIVE

The definition and proper role of service, as it relates to education, in the residency training experience has been long debated. In this study we aimed to develop definitions for service and education, delineate how each is perceived to contribute value to training, and to measure respondents' ratings of service and education using case vignettes.

METHODS

We conducted a multisite cohort survey study of pediatric residents (n = 797) and program directors (PDs; n = 37) using a region-stratified sample of 2 to 3 participating pediatric residency programs per region.

RESULTS

Surveys were completed by 34 PDs (92%) and 359 trainees (45%). PDs and residents agree that service can, in the absence of formal teaching, be considered educational. When asked how often rotations provide an appropriate balance between education and service, 94% of PDs responded 'extremely/very often' whereas only 68% of residents agreed (P = .005). Residents were significantly more likely than PDs to endorse definitions for service that included volunteer work (82% vs 59%; P = .002), going above and beyond for a patient (91% vs 78%; P = .017), and routine patient care activities (91% vs 72%; P < .001). For 6 of 12 case vignettes, trainees gave median service ratings that were significantly higher than PDs (P = .03).

CONCLUSIONS

Medical educators and pediatric residents hold mismatched impressions of their training programs' balance of service obligations with clinical education. Specifically, residents more frequently report an overabundance of service. Both groups acknowledge that service activities can be educationally valuable although the groups' definitions of service are not fully aligned.

Validity Evidence and Scoring Guidelines for Standardized Patient Encounters and Patient Notes From a Multisite Study of Clinical Performance Examinations in Seven Medical Schools

PURPOSE

To examine validity evidence of local graduation competency examination scores from seven medical schools using shared cases and to provide rater training protocols and guidelines for scoring patient notes (PNs).

METHOD

Between May and August 2016, clinical cases were developed, shared, and administered across seven medical schools (990 students participated). Raters were calibrated using training protocols, and guidelines were developed collaboratively across sites to standardize scoring. Data included scores from standardized patient encounters for history taking, physical examination, and PNs. Descriptive statistics were used to examine scores from the different assessment components. Generalizability studies (G-studies) using variance components were conducted to estimate reliability for composite scores.

RESULTS

Validity evidence was collected for response process (rater perception), internal structure (variance components, reliability), relations to other variables (interassessment correlations), and consequences (composite score). Student performance varied by case and task. In the PNs, justification of differential diagnosis was the most discriminating task. G-studies showed that schools accounted for less than 1% of total variance; however, for the PNs, there were differences in scores for varying cases and tasks across schools, indicating a school effect. Composite score reliability was maximized when the PN was weighted between 30% and 40%. Raters preferred using case-specific scoring guidelines with clear point-scoring systems.

CONCLUSIONS

This multisite study presents validity evidence for PN scores based on scoring rubric and case-specific scoring guidelines that offer rigor and feedback for learners. Variability in PN scores across participating sites may signal different approaches to teaching clinical reasoning among medical schools.

Characteristics of multi-institutional health sciences education research: a systematic review

OBJECTIVES

Multi-institutional research increases the generalizability of research findings. However, little is known about characteristics of collaborations across institutions in health sciences education research. Using a systematic review process, the authors describe characteristics of published, peer-reviewed multi-institutional health sciences education research to inform educators who are considering such projects.

METHODS

Two medical librarians searched MEDLINE, the Education Resources Information Center (ERIC), EMBASE, and CINAHL databases for English-language studies published between 2004 and 2013 using keyword terms related to multi-institutional systems and health sciences education. Teams of two authors reviewed each study and resolved coding discrepancies through consensus. Collected data points included funding, research network involvement, author characteristics, learner characteristics, and research methods. Data were analyzed using descriptive statistics.

RESULTS

One hundred eighteen of 310 articles met inclusion criteria. Sixty-three (53%) studies received external and/or internal financial support (87% listed external funding, 37% listed internal funding). Forty-five funded studies involved graduate medical education programs. Twenty (17%) studies involved a research or education network. Eighty-five (89%) publications listed an author with a master's degree or doctoral degree. Ninety-two (78%) studies were descriptive, whereas 26 studies (22%) were experimental. The reported study outcomes were changes in student attitude (38%; n=44), knowledge (26%; n=31), or skill assessment (23%; n=27), as well as patient outcomes (9%; n=11).

CONCLUSIONS

Multi-institutional descriptive studies reporting knowledge or attitude outcomes are highly published. Our findings indicate that funding resources are not essential to successfully undertake multi-institutional projects. Funded studies were more likely to originate from graduate medical or nursing programs.

Conducting multicenter research in healthcare simulation: Lessons learned from the INSPIRE network

Simulation-based research has grown substantially over the past two decades; however, relatively few published simulation studies are multicenter in nature. Multicenter research confers many distinct advantages over single-center studies, including larger sample sizes for more generalizable findings, sharing resources amongst collaborative sites, and promoting networking. Well-executed multicenter studies are more likely to improve provider performance and/or have a positive impact on patient outcomes. In this manuscript, we offer a step-by-step guide to conducting multicenter, simulation-based research based upon our collective experience with the International Network for Simulation-based Pediatric Innovation, Research and Education (INSPIRE). Like multicenter clinical research, simulation-based multicenter research can be divided into four distinct phases. Each phase has specific differences when applied to simulation research: (1) Planning phase, to define the research question, systematically review the literature, identify outcome measures, and conduct pilot studies to ensure feasibility and estimate power; (2) Project Development phase, when the primary investigator identifies collaborators, develops the protocol and research operations manual, prepares grant applications, obtains ethical approval and executes subsite contracts, registers the study in a clinical trial registry, forms a manuscript oversight committee, and conducts feasibility testing and data validation at each site; (3) Study Execution phase, involving recruitment and enrollment of subjects, clear communication and decision-making, quality assurance measures and data abstraction, validation, and analysis; and (4) Dissemination phase, where the research team shares results via conference presentations, publications, traditional media, social media, and implements strategies for translating results to practice. With this manuscript, we provide a guide to conducting quantitative multicenter research with a focus on simulation-specific issues.

A multi-source feedback tool for measuring a subset of Pediatrics Milestones

BACKGROUND

The Pediatrics Milestones Assessment Pilot employed a new multisource feedback (MSF) instrument to assess nine Pediatrics Milestones among interns and subinterns in the inpatient context.

OBJECTIVE

To report validity evidence for the MSF tool for informing milestone classification decisions.

METHODS

We obtained MSF instruments by different raters per learner per rotation. We present evidence for validity based on the unified validity framework.

RESULTS

One hundred and ninety two interns and 41 subinterns at 18 Pediatrics residency programs received a total of 1084 MSF forms from faculty (40%), senior residents (34%), nurses (22%), and other staff (4%). Variance in ratings was associated primarily with rater (32%) and learner (22%). The milestone factor structure fit data better than simpler structures. In domains except professionalism, ratings by nurses were significantly lower than those by faculty and ratings by other staff were significantly higher. Ratings were higher when the rater observed the learner for longer periods and had a positive global opinion of the learner. Ratings of interns and subinterns did not differ, except for ratings by senior residents. MSF-based scales correlated with summative milestone scores.

CONCLUSION

We obtain moderately reliable MSF ratings of interns and subinterns in the inpatient context to inform some milestone assignments.

Qualitative Study Exploring Implementation of a Point-of-Care Competency-Based Lumbar Puncture Program Across Institutions

OBJECTIVE

To explore the factors that facilitated or hindered successful implementation of a multi-centered infant lumbar puncture (LP) competency-based education program that required interns to demonstrate skills readiness on a task trainer before performing their first clinical LP.

METHODS

In 2013, investigators conducted a qualitative study utilizing semistructured interviews and focus groups of site directors (SDs) from the International Network for Simulation-Based Pediatric Innovation, Research, and Education (INSPIRE) who were responsible for implementing the LP competency-based education program. Transcripts were analyzed using grounded theory to identify and verify emergent themes and subthemes.

RESULTS

Thematic saturation was attained after interviewing 19 SDs in 12 interviews and 3 focus groups. The most significant strategies and barriers were organized into 4 main themes: 1) alignment of different visions to obtain buy-in, 2) balance between providing education versus patient care, 3) acceptance of novel teaching paradigms, and 4) communication logistics. The ability to overcome barriers was influenced by institutional culture on trainee education, patient safety and research; the level of relational coordination between different groups of stakeholders; and the ability of SDs to identify and diversify entrepreneurial strategies.

CONCLUSIONS

INSPIRE SDs reveal the challenges of implementing a network-wide competency-based educational initiative that determines interns' readiness to perform LPs in clinical settings. Strategizing to align the common goals of graduate medical training, patient care and research instructs clinician educators and leaders on how to successfully change educational culture in academic medicine.

Applying the institutional review board data repository approach to manage ethical considerations in evaluating and studying medical education

ISSUE

Medical educators and educational researchers continue to improve their processes for managing medical student and program evaluation data using sound ethical principles. This is becoming even more important as curricular innovations are occurring across undergraduate and graduate medical education. Dissemination of findings from this work is critical, and peer-reviewed journals often require an institutional review board (IRB) determination.

APPROACH

IRB data repositories, originally designed for the longitudinal study of biological specimens, can be applied to medical education research. The benefits of such an approach include obtaining expedited review for multiple related studies within a single IRB application and allowing for more flexibility when conducting complex longitudinal studies involving large datasets from multiple data sources and/or institutions. In this paper, we inform educators and educational researchers on our analysis of the use of the IRB data repository approach to manage ethical considerations as part of best practices for amassing, pooling, and sharing data for educational research, evaluation, and improvement purposes.

IMPLICATIONS

Fostering multi-institutional studies while following sound ethical principles in the study of medical education is needed, and the IRB data repository approach has many benefits, especially for longitudinal assessment of complex multi-site data.