Unprofessionalism in anesthesiology: A qualitative study on classifying unprofessional behavior in anesthesiology residency education

STUDY OBJECTIVE

This study aims to identify the domains that constitute behaviors perceived to be unprofessional in anesthesiology residency training programs.

DESIGN

Qualitative study.

SETTING

Anesthesiology residency training programs.

PATIENTS

Not applicable. The participants involved residents, fellows, and faculty members purposefully sampled in four US-based anesthesiology residency programs.

INTERVENTIONS

Participants were asked to submit examples of unprofessional behavior they witnessed in anesthesiology residents, fellows, or faculty members via a Qualtrics link.

MEASUREMENTS

Not applicable. The behavior examples were independently reviewed and categorized into themes using content analysis.

MAIN RESULTS

A total of 116 vignettes were collected, resulting in a final list of 111 vignettes after excluding those that did not describe behavior exhibited by anesthesiology faculty or trainees. Fifty-eight vignettes pertained to unprofessional behaviors observed in faculty members and 53 were observed in trainees (residents and fellows). Nine unprofessionalism themes emerged in the analysis. The most common themes were VERBAL, SUPERVISION, QUALITY, ENGAGEMENT, and TIME. As to the distribution of role group (faculty versus trainee) by theme, unprofessional behaviors falling into the categories of BIAS, GOSSIP, LEWD, and VERBAL were observed more in faculty; whereas themes with unprofessional behavior primarily attributed to trainees included ENGAGEMENT, QUALITY, TIME, and SUPERVISION.

CONCLUSION

By reviewing reported professionalism-related vignettes within residency training programs, we identified classification descriptors for defining unprofessional behavior specific to anesthesiology residency education. Findings from this study enrich the definition of professionalism as a multi-dimensional competency pertaining to anesthesiology graduate medical education. This framework may facilitate preventative intervention and timely remediation plans for unprofessional behavior in residents and faculty.

How to Use and Report on p-values

The use of the p-value in quantitative research, particularly its threshold of "P < 0.05" for determining "statistical significance," has long been a cornerstone of statistical analysis in research. However, this standard has been increasingly scrutinized for its potential to mislead findings, especially when the practical significance, the number of comparisons, or the suitability of statistical tests are not properly considered. In response to controversy around use of p-values, the American Statistical Association published a statement in 2016 that challenged the research community to abandon the term "statistically significant". This stance has been echoed by leading scientific journals to urge a significant reduction or complete elimination in the reliance on p-values when reporting results. To provide guidance to researchers in health professions education, this paper provides a succinct overview of the ongoing debate regarding the use of p-values and the definition of p-values. It reflects on the controversy by highlighting the common pitfalls associated with p-value interpretation and usage, such as misinterpretation, overemphasis, and false dichotomization between "significant" and "non-significant" results. This paper also outlines specific recommendations for the effective use of p-values in statistical reporting including the importance of reporting effect sizes, confidence intervals, the null hypothesis, and conducting sensitivity analyses for appropriate interpretation. These considerations aim to guide researchers toward a more nuanced and informative use of p-values.

Looking Across the Drape: A Novel Quality Improvement Approach to Understanding Surgeon and Anesthesiologist Burnout

OBJECTIVE

To identify well-being threats for surgeons and anesthesiologists and develop interventions using the Quality of Life Improvement (QOLI) approach.

BACKGROUND

Developing feasible perioperative well-being interventions requires identifying shared and specialty-specific well-being needs. The QOLI framework integrates human-centered design, implementation science, and quality improvement to address well-being needs.

METHODS

Anesthesia and surgery faculty in eight perioperative departments at an academic medical center completed cross-sectional surveys containing validated measures of well-being and workplace satisfaction, and open-ended questions about professional motivations, pain points, strategies for improvement, and well-being priorities. Using template analysis, we analyzed open-ended survey data and presented resulting themes at a joint-specialty town hall for live-voting to identify well-being priorities.

RESULTS

104 perioperative faculty completed the survey. Across specialties, higher MHC-SF scores (representative of individual global well-being) were associated with higher satisfaction with workplace control, values, decision latitude, and social support. Anesthesiologists reported lower satisfaction and control than surgeons across multiple domains. Template analysis yielded five areas for intervention: (1) Work culture, (2) Work environment/resources, (3) Sources of fulfillment, (4) Work/life harmony, (5) Financial compensation. Surgeons and anesthesiologists both prioritized high-quality patient care but differed in their other top priorities. The most frequently cited well-being threats for surgeons were OR inefficiencies/delays and excessive workload, while anesthesiologists cited understaffing and unpredictable work hours.

CONCLUSIONS

Surgeons and anesthesiologists share many needs and priorities, with pain points that are often negatively synergistic. Applying the QOLI approach across specialties allows for well-being interventions that honor complexity and promote the development of feasible solutions.

Engaging GME Learners in Health System-Aligned Improvement Work in the Clinical Learning Environment

Alignment between graduate medical education (GME) and health system priorities is foundational to meaningful engagement of residents and fellows in systems improvement work within the clinical learning environment. The Residents and Fellows Leading Interprofessional Continuous Improvement Teams program at the University of California San Francisco was designed over a decade ago to address barriers to trainee participation in health system-based improvement work. The program provides structure and support for health system-aligned trainee-led improvement projects in the clinic learning environment. Project champions (residents/fellows) from GME programs attend workshops where they learn improvement methodologies and develop proposals for health system-based improvement projects for their training programs. Proposals are supported by local faculty mentors and are reviewed and approved by GME and health systems' leaders. During the academic year, teams share their progress using visual management boards and interactive leader rounds. The health system provides a modest financial incentive for successful projects. Since the program's inception, thousands of trainees from 58 residency and fellowship programs have participated either as champions or participants in the program at least once, and in total over 300 projects have been implemented. Approximately three-quarters of the specific improvement goals were met, all projects meaningfully engaged residents and fellows, and many projects continued after the learners graduated. This active partnership between GME and a health system created a symbiotic relationship; trainees received education and support to complete improvement projects, while the health system reaped additional benefits from the alignment and impact of the projects. This partnership continues to grow with steady increases in participating programs, spread to partner health systems, and scholarship for trainees and faculty.

The Impact of an Interactive Unconscious Bias Training on Perioperative Learners

Background

Providers' unconscious biases reinforce health disparities through negative direct patient care and interactions with colleagues.

Objective

We created a workshop grounded in Critical Race Theory and the importance of different intersectionalities to improve medical trainees' self-assessment of their implicit biases in curated facilitated spaces.

Methods

A total of 44 UCSF first-year clinical anesthesiology residents (CA-1) (95% response rate) and 23 surgery residents in their research year (77% response rate) participated in this workshop over 4 separate sessions in September 2020 and 2021. Quantitative data from a pre-/post-workshop survey was analyzed via a paired t test to evaluate our workshop's effectiveness. Feedback on efficacy was obtained by coding themes from our survey's open-ended questions.

Results

The workshop was evaluated positively by a total of 65 of 67 participants in the post-workshop survey. On a 5-point Likert scale, participants self-reported they agreed that their unconscious biases affect their clinical interactions from a pre-workshop mean of 3.3 (SD ± 1.32) to a post-workshop mean of 3.9 (SD ± 0.87, P = .008).

Conclusion

Our findings suggest that this workshop was effective for perioperative residents and can be extrapolated to all residents by tailoring the workshop to their respective work environments.

ChatGPT and Generative Artificial Intelligence for Medical Education: Potential Impact and Opportunity

ABSTRACT

ChatGPT has ushered in a new era of artificial intelligence (AI) that already has significant consequences for many industries, including health care and education. Generative AI tools, such as ChatGPT, refer to AI that is designed to create or generate new content, such as text, images, or music, from their trained parameters. With free access online and an easy-to-use conversational interface, ChatGPT quickly accumulated more than 100 million users within the first few months of its launch. Recent headlines in the popular press have ignited concerns relevant to medical education over the possible implications of cheating and plagiarism in assessments as well as excitement over new opportunities for learning, assessment, and research. In this Scholarly Perspective, the authors offer insights and recommendations about generative AI for medical educators based on literature review, including the AI literacy framework. The authors provide a definition of generative AI, introduce an AI literacy framework and competencies, and offer considerations for potential impacts and opportunities to optimize integration of generative AI for admissions, learning, assessment, and medical education research to help medical educators navigate and start planning for this new environment. As generative AI tools continue to expand, educators need to increase their AI literacy through education and vigilance around new advances in the technology and serve as stewards of AI literacy to foster social responsibility and ethical awareness around the use of AI.

Drivers of Well-Being and Burnout in Anesthesiology Residents

Background

With more than 50% of anesthesiology residents reporting burnout, many residency programs have begun creating wellness programs to address burnout and promote well-being. However, to date, many wellness initiatives have focused on individual strategies rather than systems approaches to improve the learning environment. Individual-focused interventions in the absence of systematic efforts can lead to resentment, resistance, and worsening burnout and precipitate a loss of trust in leadership and the organization. Here, we describe a process to engage anesthesiology residents, who are key stakeholders, by exploring their perspectives on burnout and well-being to better inform systematic interventions to improve the clinical work and learning environments.

Methods

We conducted semistructured interviews with second- and third-year clinical anesthesia residents at the University of California, San Francisco, using the areas of worklife model as sensitizing concepts. We conducted a thematic analysis on transcribed interviews grounded in constructivist orientation.

Results

We identified the following 3 major categories of themes based on interviews with 10 residents: (1) definition of well-being, (2) challenges to well-being, and (3) strategies for coping with challenges and burnout. Challenges described by anesthesiology residents align with the areas of the worklife model, with the coronavirus disease 2019 pandemic precipitating additional threats in the domains of workload and community.

Conclusions

Anesthesiology residents' definition of well-being includes both individual (resilience) and systemic (meaning in work, job autonomy, and control) factors, reaffirming that positive work and learning environments are critical to professional well-being.

The efficacy of an anatomy and ultrasonography workshop on improving residents' confidence and knowledge in regional anesthesia

BACKGROUND

Ultrasound Guided Regional Anesthesia (UGRA) has become the standard for regional anesthesia practice, but there is not a standardized educational approach for training residents. The objective of this study was to evaluate the efficacy of an UGRA workshop utilizing the theoretical framework of embodied cognition for anesthesiology residents.

METHODS

A workshop was developed consisting of didactics, scanning training on standardized patients (SPs) and anatomy reviews on prosected cadavers that focused on the most common UGRA procedures for the upper and lower extremity. At the beginning of the workshop and at the end of the workshop residents completed pre-test and pre-confidence surveys, as well as post-test and post-confidence surveys, respectively to assess the impact of the workshop.

RESULTS

39 residents (100% of the possible residents) participated in the workshop in 2019. Residents' confidence in identifying relevant anatomy for the most common UGRA procedures significantly increased in 13 of the 14 measurements. Residents' knowledge gain was also statistically significant from the pre-test to post-test (20.13 ± 3.61 and 26.13 ± 2.34; p < .0001). The residents found the course overall to be very useful (4.90 ± 0.38) and in particular the cadaveric component was highly rated (4.74 ± 0.55).

CONCLUSIONS

In this study, we developed a workshop guided by the embodied cognition framework to aid in shortening the overall learning curve of UGRA for anesthesiology residents. Based on our results this workshop should be replicated by institutions that are hoping to decrease the learning curve associated with UGRA and increase residents' confidence in identifying the relevant anatomy in UGRA nerve blocks.

The fundamentals of Artificial Intelligence in medical education research: AMEE Guide No. 156

The use of Artificial Intelligence (AI) in medical education has the potential to facilitate complicated tasks and improve efficiency. For example, AI could help automate assessment of written responses, or provide feedback on medical image interpretations with excellent reliability. While applications of AI in learning, instruction, and assessment are growing, further exploration is still required. There exist few conceptual or methodological guides for medical educators wishing to evaluate or engage in AI research. In this guide, we aim to: 1) describe practical considerations involved in reading and conducting studies in medical education using AI, 2) define basic terminology and 3) identify which medical education problems and data are ideally-suited for using AI.

The Consequences of Step 2 Clinical Skills Examination Discontinuation for Medical Schools and Sustainability Plans for Clinical Skills Assessment

PURPOSE

Comprehensive clinical skills examinations using standardized patients are widely used to assess multiple physician competencies. However, these exams are resource intensive. With the discontinuation of the Step 2 Clinical Skills (CS) exam in 2021, how medical schools will change their approaches to comprehensive clinical skills exams is unknown. This study explores school responses to this change and future directions of comprehensive clinical skills exams using the program sustainability framework.

METHOD

This cross-sectional, descriptive study surveyed medical school curriculum deans at 150 Liaison Committee on Medical Education-accredited U.S. medical schools from September to October 2021. The 30-question survey included questions about medical school and participant role, current comprehensive clinical skills exams, sustainability dimensions, and challenges and future directions. Descriptive statistics were used to characterize responses, and content analysis was used to identify themes in the open-ended responses.

RESULTS

Educators at 75 of 150 institutions (50%) responded. Sixty-three respondents (84%) reported conducting a comprehensive clinical skills exam. The comprehensive clinical skills exam assessed readiness for graduation (51 [81%]), provided feedback for students (49 [78%]), evaluated curricula (38 [60%]), provided information for medical student performance evaluation or communication with residency (10 [16%]), and assessed other factors (6 [10%]), including preparation for Step 2 CS in the past and readiness for advancement to fourth year of medical school (multiple responses were allowed). Factors facilitating sustainability included sufficient funding to continue the exam (55 [87%]) and the belief that clinical skills assessment in medical school is now more important after discontinuation of the Step 2 CS exam (55 [87%]). Challenges to sustainability included organizational capacity and limited interinstitutional collaboration.

CONCLUSIONS

Educators remain committed to the purpose of comprehensive clinical skills exams. Adapting to changed licensing requirements while sustaining clinical skills exams enables innovation and improvement in assessment of clinical competence.

Students' Perspectives on Basic and Clinical Science Integration When Step 1 is Administered After the Core Clerkships

Phenomenon: According to adult learning theories, effective cognitive integration of basic and clinical science may promote the transfer of knowledge to patient care. The placement of the U.S. Medical Licensing Examination (USMLE) Step 1 after the core clerkships is one strategy intended to facilitate cognitive integration, though learner experiences with this model are unexplored. The purpose of this study is to understand students' perspectives on basic and clinical science integration in a post-clerkship Step 1 curriculum. Approach: Focus groups were conducted between August and September 2020 with senior medical students from the University of California, San Francisco School of Medicine and University of Michigan Medical School. Data were analyzed using a constructivist approach to thematic analysis. Findings: Thirty-three students participated in six focus groups. Participants described multiple barriers to cognitive integration in the clerkship learning environment, though they also identified examples of teaching and learning that facilitated integration. Early in their clerkships, students struggled to integrate because of their tenuous basic science foundation, cognitive overload, and difficulty perceiving the relevance of basic science to patient care. They felt that educators primarily focused on clinical science, and many basic science teaching sessions during clerkships felt irrelevant to patient care. However, students also described experiences that made the connection between basic and clinical science more explicit, including modeling by educators and clerkship learning activities that more overtly encouraged the application of basic science to clinical care. In addition, the return to basic science studying during the post-clerkship dedicated Step 1 study period offered powerful integration opportunities. These facilitators of cognitive integration helped students recognize the value of integration for enduring learning. Insights: There are myriad barriers to cognitive integration of basic and clinical science during clerkships in a post-clerkship Step 1 curriculum. The relevance of basic science to patient care needs to be made more explicit to students through modeling by clinician educators to augment the potential benefits of curricular change. The post-clerkship Step 1 study period appears to offer a unique opportunity for cognitive integration later in a learner's trajectory that may be related to curricular design. When learners recognize the applicability of basic science to patient care, they may more intentionally transfer basic science knowledge to clinical practice.

Engagement in Free Open Access Medical Education by US Nephrology Fellows

BACKGROUND

As free open access medical education (FOAMed) use increases, it is important to characterize how and why learners are using this educational material in nephrology. We describe the frequency, purpose, and type of FOAMed usage across US nephrology fellows.

METHODS

In this cross-sectional survey, items were emailed to all US adult and pediatric nephrology fellows via the American Society of Nephrology (ASN) Fellow Survey in May 2022. The eight-item survey, developed to measure FOAMed engagement, had previously undergone instrument validation. The results were analyzed by descriptive statistics.

RESULTS

In total, 43% (359/842) adult nephrology fellows and 51% (45/88) pediatric nephrology fellows completed the survey. Seventy-four percent (300/404) of fellows reported using FOAMed, and 72% (215/300) started using FOAMed within the past 2 years. Of FOAMed users, 41% (122/300) reported viewing FOAMed and 33% (99/300) reported applying knowledge gained from these resources daily or weekly. Common purposes for FOAMed engagement included searching Twitter to learn about others' opinions in the field (43%; 130/300), reading blogs to answer clinical questions (35%; 105/300), and listening to podcasts for the most up-to-date information (39%; 116/300). Compared with traditional educational resources, fellows preferred using FOAMed for staying up to date on nephrology topics (75%) and answering clinical questions (37%). Among all fellows, the greatest barriers to FOAMed use were unfamiliarity with FOAMed (27%; 111/404), validity concerns (22%; 90/404), and a lack of a local community of FOAMed users (22%; 87/404).

CONCLUSIONS

Seventy-four percent of nephrology fellows used FOAMed resources in a variety of ways, and of them, 33% of fellows clinically applied knowledge gained from these resources. Reasons for engaging with FOAMed varied across resources.

A Call to Action From the California Consortium for the Assessment of Clinical Competence: Making the Case for Regional Collaboration

The discontinuation of the United States Medical Licensing Examination Step 2 Clinical Skills Examination emphasizes the need for other reliable standardized assessments of medical student clinical skills. For 30 years, the California Consortium for the Assessment of Clinical Competence (CCACC) has collaborated in the development of clinical skills assessments and has become a valuable resource for clinicians, standardized patient educators, psychometricians, and medical educators. There are many merits to strong multi-institutional partnerships, including the integration of data across multiple schools to provide feedback to both students and curricula, pooled test performance statistics for analysis and quality assurance, shared best practices and resources, individual professional development, and opportunities for research and scholarship. The infrastructure of the CCACC allows member schools to adapt to a changing medical landscape, from emerging trends in clinical medicine to the limitations imposed by a global pandemic. In the absence of a national examination, there is now a greater need for all medical schools to develop a comprehensive, dynamic, and psychometrically sound assessment that accurately evaluates clinical skills. Medical schools working together in regional consortia have the opportunity to create and implement innovative and robust assessments that evaluate a wide range of clinical skills, ensure that medical students have met an expected level of clinical competency before graduation, and provide a framework that contributes to ongoing efforts for the development of new national clinical skills standards.

Profiles for Success: Examining the Relationship between Student Profiles and Clerkship Performance Using Latent Profile Analysis

PHENOMENON

Learning is a complex phenomenon in which learners can vary in their learning orientation, learning approaches, and perception of the learning environment. Learners may be characterized as mastery oriented or performance oriented, and this learning orientation can influence their learning approaches, such as whether to seek feedback or ask for help. The learning environment includes institutional assessment policies and informal interactions that emphasize outcomes and differentiation among students. When learners perceive the learning environment as performance oriented, they feel competition from other learners and fear negative feedback. This complex interplay of learning orientation, its influence on feedback-seeking behavior, and the environmental influence and reinforcement of both can be captured through learner profiles.

APPROACH

In this multi-institutional cross-sectional survey study of students from six U.S. medical schools, we investigated learner profiles that characterize how these domains manifest together across individual learners. We then determined how these profiles are associated with students' clerkship grades. Measures included the Patterns of Adaptive Learning for learning orientation, self-reports for learning approaches, and perceptions of the learning environment. We used latent profile analysis to cluster students who share common characteristics around orientation, behavior, and environment. The relationship between these profiles and the percentage of honors earned was examined using multiple regression analysis.

FINDINGS

We found four distinct learner profiles within 666 of 974 (68%) analyzable responses: (1) performance oriented with negative perceptions of environment, (2) mastery oriented with desirable learning approaches, (3) "average" group, and (4) mastery orientation only. Profile 1 (M = 39%) and Profile 4 (M = 38%) students received fewer clerkship honors compared with other profiles. Profile 2 students earned a significantly higher percentage of honors grades (M = 54%) compared with other students. Profile 3 students (average group) earned the second highest percentage of honors (M = 46%). Effect sizes comparing the percentage of honors varied from 0.32 to 0.65, indicating medium to large differences.

INSIGHTS

The highest performers, as measured by clerkship honors grades, were mastery oriented, perceived a more positive learning environment, and were comfortable asking questions and seeking feedback for their learning. The lowest performing students were performance oriented and had negative perceptions of their learning environments. Contrary to previous findings, we found that the relationship between mastery orientation and learning approaches is not unidirectional but rather seems to be mediated by perceptions of the learning environment.

Statistical points and pitfalls: growth modeling

None

Development and validation of the safe transfusion assessment tool

BACKGROUND

Given the prevalence and risks of blood transfusion, it is essential that trainees and practicing clinicians have a thorough understanding of relevant transfusion medicine competencies. The aim of this research was to develop and gather validity evidence for an instrument to assess knowledge of core transfusion-related competencies.

METHODS

We developed the safe transfusion assessment tool (STAT) using a multistep process. Initially, 20 core competencies in transfusion medicine were identified through a consensus-driven Delphi process. Learning objectives and assessment items pertinent to each competency were created. Next, a 13-item assessment tool was piloted with multidisciplinary experts and trainees. Multiple iterative revisions were made based on feedback. Finally, the 12-item STAT was administered to 100 participants of varying training level and specialty to establish validity, difficulty and item discrimination indices, and perceived utility.

RESULTS

Analysis of instrument item difficulty and item discrimination indices demonstrated the ability of the STAT to assess essential knowledge in transfusion medicine relevant to trainees and clinicians in multiple programs and practice settings. Eight of twelve items discriminated between learners with varying degrees of expertise. Hundred percent of students and trainees rated the STAT as Extremely Helpful or Somewhat Helpful and the majority planned to utilize the answer guide as a study aid.

CONCLUSION

The STAT is a concise, valid, and reliable knowledge assessment tool that may be used by researchers and educators to augment transfusion medicine curricula (www.safetransfusion.ucsf.edu). Scores can help inform departments on areas in which trainees require additional support and areas of potential educational interventions.

Validity evidence for flourishing as a measure of global wellbeing: a national multicenter study of academic general surgery residents

Purpose

Physician wellbeing is critical to high-quality sustainable healthcare and optimal patient experience. Few objective measures exist to assay wellbeing (as opposed to just pathology) in surgery, or to evaluate the efficacy of wellbeing interventions. Flourishing (as measured by the Mental Health Continuum, MHC) has been suggested as a concise measure of global wellbeing in surgeons. We aimed to establish validity evidence for flourishing in a large national sample of surgical trainees, explore differences by gender and race, and confirm support for the underlying constructs.

Methods

This cross-sectional study of all General Surgery residents at 16 ACGME-accredited academic programs included an online survey of published measures distributed in February 2021. The Mental Health Continuum (MHC), a three-factor model, assesses emotional, social, and psychological wellbeing and is an established metric of psychosocial thriving in non-physicians. A global score cut-off exists for flourishing which represents high wellbeing. Correlation between flourishing and established measures of risk and resilience in surgery were assessed for validity evidence. Differences by gender and race were explored. A confirmatory factor analysis (CFA) was performed to confirm the three-factor structure in surgical trainees.

Results

300 residents (60% non-male, 41% non-white) responded to the survey. For the overall group, flourishing was significantly positively correlated with all wellbeing resilience factors and negatively correlated with all risk factors. This held true for race and gender subgroups based on interaction analyses. CFA and sensitivity analysis results supported the three-factor structure.

Conclusions

Our findings offer validity evidence for flourishing as a measure of global wellbeing and confirm the three-factor structure of emotional, social, and psychological wellbeing in surgical trainees. Thus, the MHC may be a concise tool for assaying wellbeing, within and across subgroups, and for assessing wellbeing intervention effectiveness within the surgery.

Residents' Challenges in Transitioning to Residency and Recommended Strategies for Improvement

BACKGROUND

The transition from internship to residency can be a particularly stressful time for learners, adversely affecting residents' experience of training. Despite awareness of residents' stress during transitions, there is limited information available regarding how residents perceive these transitions or how they could be improved. We explored residents' accounts of the experience of transitioning from internship to residency to develop a better understanding of their challenges and recommended strategies for interventions.

METHODS

We conducted semistructured interviews with first-year anesthesia residents at the University of California, San Francisco. We conducted a thematic analysis through a general inductive approach on transcribed interviews.

RESULTS

Ten residents, evenly split among categorical and noncategorical residents, participated in the interviews. We identified seven challenges faced by residents during the transition, including cognitive load management, self-assessment and eliciting effective feedback, learning resource utilization, preoperative care planning and discussion, forming relationships with peers and faculty, and professional identity formation. Residents also recommended strategies to address these challenges, including early low-stake exposure to complex cases, standardized feedback structure, resource utilization guides, normalization of discussing errors with peers, and protected time for networking events.

CONCLUSION

Residents face multiple challenges at the personal, social, and structural levels during the transition. Their recommended strategies are actionable, including scaffolded learning opportunities with increasing difficulty, more standardized and structured communications around expectations and effective feedback, enhanced orientation through bootcamp, and integration of more formal and informal social networking opportunities to increase peer and faculty interaction.

Analysis of the Interprofessional Clinical Learning Environment for Quality Improvement and Patient Safety From Perspectives of Interprofessional Teams

BACKGROUND

In 2018 the Clinical Learning Environment Review (CLER) Program reported that quality improvement and patient safety (QIPS) programs in graduate medical education (GME) were largely unsuccessful in their efforts to transfer QI knowledge and substantive interprofessional QIPS experiences to residents, and CLER 2.0 called for improvement. However, little is known about how to improve the interprofessional clinical learning environment (IP-CLE) for QIPS in GME.

OBJECTIVE

To determine the current state of the IP-CLE for QIPS at our institution with a focus on factors affecting the IP-CLE and resident integration into interprofessional QIPS teams.

METHODS

We interviewed an interprofessional group of residents, faculty, and staff of key units engaged in IP QIPS activities. We performed thematic analysis through general inductive approach using template analysis methods on transcripts.

RESULTS

Twenty individuals from 6 units participated. Participants defined learning on interprofessional QIPS teams as learning from and about each other's roles through collaboration for improvement, which occurs naturally when patients are the focus, or experiential teamwork within QIPS projects. Resident integration into these teams had various benefits (learning about other professions, effective project dissemination), barriers (difficult rotations or program structure, inappropriate assumptions), and facilitators (institutional support structures, promotion of QIPS culture, patient adverse events). There were various benefits (strengthened relationships, lowered bar for further collaboration), barriers (limited time, poor communication), and facilitators (structured meetings, educational culture) to a positive IP-CLE for QIPS.

CONCLUSIONS

Cultural factors prominently affected the IP-CLE and patient unforeseen events were valuable triggers for IP QIPS learning opportunities.

Conflict Management Education for Anesthesiology Residents: Bridging the Gap in Interpersonal and Communication Skills Competency

Effective communication and conflict management are important skills for anesthesiologists and are designated by the Accreditation Council for Graduate Medical Education (ACGME) as elements of the "interpersonal and communication skill" competency (ACGME Anesthesiology Milestone Project 2020). However, structured conflict management education for anesthesiology residents remains limited. To address this gap, we developed and implemented a conflict management session incorporating didactics and application exercises using role-play and high-fidelity simulation (SIM) for anesthesiology residents (postgraduate years 3 and 4) at a tertiary academic medical institution. These sessions were well-received, and both role-play and SIM appear to help residents learn conflict management skills.

Increasing the use of multimodal analgesia during adult surgery in a tertiary academic anaesthesia department

Objective

Multimodal analgesia pathways have been shown to reduce opioid use and side effects in surgical patients. A quality improvement initiative was implemented to increase the use of multimodal analgesia in adult patients presenting for general anaesthesia at an academic tertiary care centre. The aim of this study was to increase adoption of a perioperative multimodal analgesia protocol across a broad population of surgical patients. The use of multimodal analgesia was tracked as a process metric. Our primary outcome was opioid use normalised to oral morphine equivalents (OME) intraoperatively, in the postanaesthesia care unit (PACU), and 48 hours postoperatively. Pain scores and use of antiemetics were measured as balancing metrics.

Methods

We conducted a quality improvement study of a multimodal analgesia protocol implemented for adult (≥18 and≤70) non-transplant patients undergoing general anaesthesia (≥180 min). Components of multimodal analgesia were defined as (1) preoperative analgesic medication (acetaminophen, celecoxib, diclofenac, gabapentin), (2) regional anaesthesia (peripheral nerve block or catheter, epidural catheter or spinal) or (3) intraoperative analgesic medication (ketamine, ketorolac, lidocaine infusion, magnesium, acetaminophen, dexamethasone ≥8 mg, dexmedetomidine). We compared opioid use, pain scores and antiemetic use for patients 1 year before (baseline group—1 July 2018 to 30 June 2019) and 1 year after (implementation group—1 July 2019 to 30 June 2020) project implementation.

Results

Use of multimodal analgesia improved from 53.9% in the baseline group to 67.5% in the implementation group (p<0.001). There was no significant difference in intraoperative OME use before and after implementation (β₀=44.0, β₂=0.52, p=0.875). OME decreased after the project implementation in the PACU (β₀=34.4, β₂=−3.88, p<0.001) and 48 hours postoperatively (β₀=184.9, β₂=−22.59, p<0.001), while pain scores during those time points were similar.

Conclusion

A perioperative pragmatic multimodal analgesic intervention was associated with reduced OME use in the PACU and 48 hours postoperatively.

Design and Implementation of an Innovative, Longitudinal Wellness Curriculum in an Anesthesiology Residency Program

Burnout is a serious problem that anesthesiologists face during training and in practice. To mitigate it, experts advocate for strategies focused on enhancing individual resilience in addition to organizational change in health care systems. To catalyze this change, wellness-focused education must incorporate foundational knowledge about the science of well-being and impart skills to empower trainees to lead change in the future. We developed and implemented a longitudinal, developmental 3-year curriculum in a large anesthesiology residency program that included strategies to strengthen community-building, enhance meaning from a career in medicine, and incorporated topics focused on career and leadership development.

The role of data science and machine learning in Health Professions Education: practical applications, theoretical contributions, and epistemic beliefs

Data science is an inter-disciplinary field that uses computer-based algorithms and methods to gain insights from large and often complex datasets. Data science, which includes Artificial Intelligence techniques such as Machine Learning (ML), has been credited with the promise to transform Health Professions Education (HPE) by offering approaches to handle big (and often messy) data. To examine this promise, we conducted a critical review to explore: (1) published applications of data science and ML in HPE literature and (2) the potential role of data science and ML in shifting theoretical and epistemological perspectives in HPE research and practice. Existing data science studies in HPE are often not informed by theory, but rather oriented towards developing applications for specific problems, uses, and contexts. The most common areas currently being studied are procedural (e.g., computer-based tutoring or adaptive systems and assessment of technical skills). We found that epistemic beliefs informing the use of data science and ML in HPE poses a challenge for existing views on what constitutes objective knowledge and the role of human subjectivity for instruction and assessment. As a result, criticisms have emerged that the integration of data science in the field of HPE is in danger of becoming technically driven and narrowly focused in its approach to teaching, learning and assessment. Our findings suggest that researchers tend to formalize around the epistemological stance driven largely by traditions of a research paradigm. Future data science studies in HPE need to involve both education scientists and data scientists to ensure mutual advancements in the development of educational theory and practical applications. This may be one of the most important tasks in the integration of data science and ML in HPE research in the years to come.

Predicting Performance on Clerkship Examinations and USMLE Step 1: What Is the Value of Open-Ended Question Examination?

PURPOSE

Despite advances in learning sciences that highlight the efficacy of elaborative interrogation, in which students explain and elaborate on concepts in their own words, assessment techniques in medical education have commonly employed multiple-choice questions (MCQs). Educators' reluctance to consider alternatives such as open-ended questions (OEQs) stems from practical advantages of MCQs and the lack of empirical data on the predictability of OEQs for performance on other high-stakes assessments. In this study, the authors compared the predictive value of preclerkship assessments using OEQs for the outcomes of clerkship examinations and United States Medical Licensing Examination (USMLE) Step 1.

METHOD

The authors compared outcomes of 2 assessment formats using multiyear performance data (2015 and 2016 cohorts) on preclerkship MCQ versus OEQ examinations for predicting students' subsequent performance on 6 clerkship examinations and USMLE Step 1. The authors conducted a regression analysis to compare the predictability of MCQs and OEQs by using clerkship exam scores and Step 1 scores as dependent variables and performance on MCQs and OEQs as predictors in the models.

RESULTS

Regression models with OEQs were consistently higher for predicting clerkship exam (NBME shelf-exam) scores, except for one clerkship, compared with models using MCQs. For Step 1, R-square using MCQs was higher with 59% of the variance explained compared with 46% with OEQs, but the OEQ cohort scored significantly higher on Step 1.

CONCLUSIONS

OEQ examinations predict performance on subsequent high-stakes MCQ examinations. Given the predictive value and closer alignment with scientific principles of effective learning, OEQ examinations are an examination format worthy of consideration in preclerkship medical education programs.

Pause, Persist, Pivot: Key Decisions Health Professions Education Researchers Must Make About Conducting Studies During Extreme Events

When extreme events occur, some research becomes a clear priority, but what becomes of all other research? Does it stop indefinitely, or can it be paused with plans to resume, persist with modifications, or pivot to address new priorities? Facing this dilemma and witnessing it among their fellow health professions education researchers, the authors recognized a need for guidance. This Invited Commentary presents a framework, organized as key questions related to the research stage and process, to assist health professions education researchers in making decisions about how to proceed with research that was planned or in progress when an extreme event occurred. Although at the time of this writing, the COVID-19 pandemic was the extreme event at hand, the authors intentionally created questions and discussed considerations that can be helpful for thinking through decisions in a variety of disruptions in health professions education research-many of which require similar difficult decisions and creative solutions to carry important research forward and maintain high quality.

Learning Challenges, Teaching Strategies, and Cognitive Load: Insights From the Experience of Seasoned Endoscopy Teachers

PURPOSE

Learners of medical procedures must develop, refine, and apply schemas for both cognitive and psychomotor constructs, which may strain working memory capacity. Procedures with limitations in visual and tactile information may add risk of cognitive overload. The authors sought to elucidate how experienced procedural teachers perceived learners' challenges and their own teaching strategies in the exemplar setting of gastrointestinal endoscopy.

METHOD

The authors interviewed 22 experienced endoscopy teachers in the United States, Canada, and the Netherlands between May 2016 and March 2019 and performed thematic analysis using template analysis method. Interviews addressed learner challenges and teaching strategies from the teacher participants' perspectives. Cognitive load theory informed data interpretation and analysis.

RESULTS

Participants described taking steps to "diagnose" trainee ability and identify struggling trainees. They described learning challenges related to trainees (performance over mastery goal orientation, low self-efficacy, lack of awareness), tasks (psychomotor challenges, mental model development, tactile understanding), teachers (teacher-trainee relationship, inadequate teaching, teaching variability), and settings (internal/external distractions, systems issues). Participants described employing strategies that could match intrinsic load to learners' levels (teaching along developmental continuum, motor instruction, technical assistance/takeover), minimize extraneous load (optimize environment, systems solutions, emotional support, define expectations), and optimize germane load (promote mastery, teach schemas, stop and focus).

CONCLUSIONS

Participants provided insight into possible challenges while learning complex medical procedures with limitations in sensory channels, as well as teaching strategies that may address these challenges at individual and systems levels. Using cognitive load theory, the authors provide recommendations for procedural teachers.

Trainee perception of cognitive load during observed faculty staff teaching of procedural skills

OBJECTIVES

Although teachers impact learners' cognitive load, how specific teaching activities affect intrinsic, germane and extraneous load during procedural skills training is unknown. We sought to characterise teaching activities used in the exemplar procedural setting of colonoscopy, and to explore how they were enacted and how learners perceived them as affecting intrinsic, germane and extraneous cognitive load.

METHODS

We observed 10 colonoscopies performed by eight different gastroenterology fellows and supervised by 10 different attending physicians at two hospitals, and recorded the teaching activities observed, as well as details of when they were used and how they were enacted. After the colonoscopy, each fellow completed the Cognitive Load Inventory for Colonoscopy to quantify intrinsic, germane and extraneous load. We then interviewed each fellow to determine how he or she perceived teaching as affecting cognitive load. Qualitative data were subjected to content analysis. Instances of germane load-promoting activities were correlated with measured germane load.

RESULTS

We observed 515 instances of teaching activities. The intensity of teaching varied substantially, ranging from 0.7 to 3.3 activities per minute, as did the pattern of teaching activities used by different attending physicians. Little teaching occurred immediately before or after a procedure. Fellows usually perceived teaching as affecting cognitive load in ways that promoted learning, particularly by reducing intrinsic load and increasing germane load. Fellows strongly perceived that the provision of autonomy promoted germane load. Conversely, fellows perceived that excessive teaching increased extraneous load. Instances of germane load-promoting teaching activities correlated moderately with measured germane load.

CONCLUSIONS

Teaching in the exemplar procedural setting of colonoscopy affected learners' cognitive load in mostly beneficial ways, yet even 'good' teaching activities had detrimental effects when used excessively. Teachers of procedures should consider learner experience, task complexity and environmental factors to modulate the modality, content and intensity of teaching to promote balanced cognitive load and learning. Teaching more reservedly during the procedure and taking advantage of pre- and post-procedure opportunities may help.

Learner, Patient, and Supervisor Features Are Associated With Different Types of Cognitive Load During Procedural Skills Training: Implications for Teaching and Instructional Design

PURPOSE

Cognitive load theory, focusing on limits of the working memory, is relevant to medical education; however, factors associated with cognitive load during procedural skills training are not well characterized. The authors sought to determine how features of learners, patients/tasks, settings, and supervisors were associated with three types of cognitive load among learners performing a specific procedure, colonoscopy, to identify implications for procedural teaching.

METHOD

Data were collected through an electronically administered survey sent to 1,061 U.S. gastroenterology fellows during the 2014-2015 academic year; 477 (45.0%) participated. Participants completed the survey immediately following a colonoscopy. Using multivariable linear regression analyses, the authors identified sets of features associated with intrinsic, extraneous, and germane loads.

RESULTS

Features associated with intrinsic load included learners (prior experience and year in training negatively associated, fatigue positively associated) and patient/tasks (procedural complexity positively associated, better patient tolerance negatively associated). Features associated with extraneous load included learners (fatigue positively associated), setting (queue order positively associated), and supervisors (supervisor engagement and confidence negatively associated). Only one feature, supervisor engagement, was (positively) associated with germane load.

CONCLUSIONS

These data support practical recommendations for teaching procedural skills through the lens of cognitive load theory. To optimize intrinsic load, level of experience and competence of learners should be balanced with procedural complexity; part-task approaches and scaffolding may be beneficial. To reduce extraneous load, teachers should remain engaged, and factors within the procedural setting that may interfere with learning should be minimized. To optimize germane load, teachers should remain engaged.

Can You See Me Now? Video Supplementation for Pediatric Teledermatology Cases

BACKGROUND/OBJECTIVES

Digital video is widely available and is used sporadically in clinical settings to evaluate patients, but whether it helps improve clinical management has not been determined. The aim of this study was to assess whether recorded video in addition to still images can improve residents' diagnostic and management accuracy and confidence with pediatric teledermatology cases.

METHODS

Dermatology residents from three programs were assigned alternately to an online survey with 15 pediatric teledermatology cases presented with still images only (still) or still images plus recorded video (mixed). Participants provided free-text diagnoses and management recommendations and rated their confidence and image quality. Responses were scored using a modified script concordance grading key based on reference panelists' responses.

RESULTS

Thirty-one residents participated (response rate 57%). Participants in the mixed group scored significantly higher on management accuracy (87.6 ± 12.9 vs 71.7 ± 14.2; p = 0.003). Both groups performed better on more common conditions than less common conditions. The mixed group outperformed the still group on less common conditions with respect to management recommendations.

CONCLUSION

This novel study suggests that supplemental recorded video may improve the management accuracy of pediatric teledermatology consultations, particularly for complex cases. Residents may benefit from training in recording and interpreting video.

Association Between Surgeon Scorecard Use and Operating Room Costs

Importance

Despite the significant contribution of surgical spending to health care costs, most surgeons are unaware of their operating room costs.

Objective

To examine the association between providing surgeons with individualized cost feedback and surgical supply costs in the operating room.

Design, Setting, and Participants

The OR Surgical Cost Reduction (OR SCORE) project was a single-health system, multihospital, multidepartmental prospective controlled study in an urban academic setting. Intervention participants were attending surgeons in orthopedic surgery, otolaryngology-head and neck surgery, and neurological surgery (n = 63). Control participants were attending surgeons in cardiothoracic surgery, general surgery, vascular surgery, pediatric surgery, obstetrics/gynecology, ophthalmology, and urology (n = 186).

Interventions

From January 1 to December 31, 2015, each surgeon in the intervention group received standardized monthly scorecards showing the median surgical supply direct cost for each procedure type performed in the prior month compared with the surgeon's baseline (July 1, 2012, to November 30, 2014) and compared with all surgeons at the institution performing the same procedure at baseline. All surgical departments were eligible for a financial incentive if they met a 5% cost reduction goal.

Main Outcomes and Measures

The primary outcome was each group's median surgical supply cost per case. Secondary outcome measures included total departmental surgical supply costs, case mix index-adjusted median surgical supply costs, patient outcomes (30-day readmission, 30-day mortality, and discharge status), and surgeon responses to a postintervention study-specific health care value survey.

Results

The median surgical supply direct costs per case decreased 6.54% in the intervention group, from $1398 (interquartile range [IQR], $316-$5181) (10 637 cases) in 2014 to $1307 (IQR, $319-$5037) (11 820 cases) in 2015. In contrast, the median surgical supply direct cost increased 7.42% in the control group, from $712 (IQR, $202-$1602) (16 441 cases) in 2014 to $765 (IQR, $233-$1719) (17 227 cases) in 2015. This decrease represents a total savings of $836 147 in the intervention group during the 1-year study. After controlling for surgeon, department, patient demographics, and clinical indicators in a mixed-effects model, there was a 9.95% (95% CI, 3.55%-15.93%; P = .003) surgical supply cost decrease in the intervention group over 1 year. Patient outcomes were equivalent or improved after the intervention, and surgeons who received scorecards reported higher levels of cost awareness on the health care value survey compared with controls.

Conclusions and Relevance

Cost feedback to surgeons, combined with a small departmental financial incentive, was associated with significantly reduced surgical supply costs, without negatively affecting patient outcomes.

Performance of a cognitive load inventory during simulated handoffs: Evidence for validity

BACKGROUND

Advancing patient safety during handoffs remains a public health priority. The application of cognitive load theory offers promise, but is currently limited by the inability to measure cognitive load types.

OBJECTIVE

To develop and collect validity evidence for a revised self-report inventory that measures cognitive load types during a handoff.

METHODS

Based on prior published work, input from experts in cognitive load theory and handoffs, and a think-aloud exercise with residents, a revised Cognitive Load Inventory for Handoffs was developed. The Cognitive Load Inventory for Handoffs has items for intrinsic, extraneous, and germane load. Students who were second- and sixth-year students recruited from a Dutch medical school participated in four simulated handoffs (two simple and two complex cases). At the end of each handoff, study participants completed the Cognitive Load Inventory for Handoffs, Paas' Cognitive Load Scale, and one global rating item for intrinsic load, extraneous load, and germane load, respectively. Factor and correlational analyses were performed to collect evidence for validity.

RESULTS

Confirmatory factor analysis yielded a single factor that combined intrinsic and germane loads. The extraneous load items performed poorly and were removed from the model. The score from the combined intrinsic and germane load items associated, as predicted by cognitive load theory, with a commonly used measure of overall cognitive load (Pearson's r = 0.83, p < 0.001), case complexity (beta = 0.74, p < 0.001), level of experience (beta = -0.96, p < 0.001), and handoff accuracy (r = -0.34, p < 0.001).

CONCLUSION

These results offer encouragement that intrinsic load during handoffs may be measured via a self-report measure. Additional work is required to develop an adequate measure of extraneous load.

Taking Rater Exposure to Trainees Into Account When Explaining Rater Variability

BACKGROUND

Rater-based judgments are widely used in graduate medical education to provide more meaningful assessments, despite concerns about rater reliability.

OBJECTIVE

We introduced a statistical modeling technique that corresponds to the new rater reliability framework, and present a case example to provide an illustration of the utility of this new approach to assessing rater reliability.

METHODS

We used mixed-effects models to simultaneously incorporate random effects for raters and systematic effects of rater role as fixed effects. Study data are clinical performance ratings collected from medical school graduates who were evaluated for their readiness for supervised clinical practice in authentic simulation settings at 2 medical schools in the Netherlands and Germany.

RESULTS

The medical schools recruited a maximum of 30 graduates out of 60 (50%) and 180 (17%) eligible candidates, respectively. Clinician raters (n = 25) for the study were selected based on their level of expertise and experience. Graduates were assessed on 7 facets of competence (FOCs) that are considered important in supervisors' entrustment decisions across the 5 cases used. Rater role was significantly associated with 2 FOCs: (1) teamwork and collegiality, and (2) verbal communication with colleagues/supervisors. For another 2 FOCs, rater variability was only partially explained by the role of the rater (a proxy for the amount of direct interaction with the trainee).

CONCLUSIONS

Consideration of raters as meaningfully idiosyncratic provides a new framework to explore their influence on assessment scores, which goes beyond considering them as random sources of variability.

Measuring cognitive load during procedural skills training with colonoscopy as an exemplar

OBJECTIVES

Few studies have investigated cognitive factors affecting learning of procedural skills in medical education. Cognitive load theory, which focuses on working memory, is highly relevant, but methods for measuring cognitive load during procedural training are not well understood. Using colonoscopy as an exemplar, we used cognitive load theory to develop a self-report instrument to measure three types of cognitive load (intrinsic, extraneous and germane load) and to provide evidence for instrument validity.

METHODS

We developed the instrument (the Cognitive Load Inventory for Colonoscopy [CLIC]) using a multi-step process. It included 19 items measuring three types of cognitive load, three global rating items and demographics. We then conducted a cross-sectional survey that was administered electronically to 1061 gastroenterology trainees in the USA. Participants completed the CLIC following a colonoscopy. The two study phases (exploratory and confirmatory) each lasted for 10 weeks during the 2014-2015 academic year. Exploratory factor analysis determined the most parsimonious factor structure; confirmatory factor analysis assessed model fit. Composite measures of intrinsic, extraneous and germane load were compared across years of training and with global rating items.

RESULTS

A total of 477 (45.0%) invitees participated (116 in the exploratory study and 361 in the confirmatory study) in 154 (95.1%) training programmes. Demographics were similar to national data from the USA. The most parsimonious factor structure included three factors reflecting the three types of cognitive load. Confirmatory factor analysis verified that a three-factor model was the best fit. Intrinsic, extraneous and germane load items had high internal consistency (Cronbach's alpha 0.90, 0.87 and 0.96, respectively) and correlated as expected with year in training and global assessment of cognitive load.

CONCLUSIONS

The CLIC measures three types of cognitive load during colonoscopy training. Evidence of validity is provided. Although CLIC items relate to colonoscopy, the development process we detail can be used to adapt the instrument for use in other learning settings in medical education.

Ensuring Resident Competence: A Narrative Review of the Literature on Group Decision Making to Inform the Work of Clinical Competency Committees

Background The expectation for graduate medical education programs to ensure that trainees are progressing toward competence for unsupervised practice prompted requirements for a committee to make decisions regarding residents' progress, termed a clinical competency committee (CCC). The literature on the composition of these committees and how they share information and render decisions can inform the work of CCCs by highlighting vulnerabilities and best practices. Objective We conducted a narrative review of the literature on group decision making that can help characterize the work of CCCs, including how they are populated and how they use information. Methods English language studies of group decision making in medical education, psychology, and organizational behavior were used. Results The results highlighted 2 major themes. Group member composition showcased the value placed on the complementarity of members' experience and lessons they had learned about performance review through their teaching and committee work. Group processes revealed strengths and limitations in groups' understanding of their work, leader role, and information-sharing procedures. Time pressure was a threat to the quality of group work. Conclusions Implications of the findings include the risks for committees that arise with homogeneous membership, limitations to available resident performance information, and processes that arise through experience rather than deriving from a well-articulated purpose of their work. Recommendations are presented to maximize the effectiveness of CCC processes, including their membership and access to, and interpretation of, information to yield evidence-based, well-reasoned judgments.

Optimizing the pre-referral workup for gastroenterology and hepatology specialty care: consensus using the Delphi method

RATIONALE, AIMS AND OBJECTIVES

Specialty care referrals have doubled in the last decade. Optimization of the pre-referral workup by a primary care doctor can lead to a more efficient first specialty visit with the patient. Guidance regarding pre-referral laboratory testing is a first step towards improving the specialty referral process. Our aim was to establish consensus regarding appropriate pre-referral workup for common gastrointestinal and liver conditions.

METHODS

The Delphi method was used to establish local consensus for recommending certain laboratory tests prior to specialty referral for 13 clinical conditions. Seven conditions from The University of Michigan outpatient referral guidelines were used as a baseline. An expert panel of three PCPs and nine gastroenterologists from three academic hospitals participated in three iterative rounds of electronic surveys. Each panellist ranked each test using a 5-point Likert scale (strongly disagree to strongly agree). Local panellists could recommend additional tests for the initial diagnoses, and also recommended additional diagnoses needing guidelines: iron deficiency anaemia, abdominal pain, irritable bowel syndrome, fatty liver disease, liver mass and cirrhosis. Consensus was defined as ≥70% of experts scoring ≥4 (agree or strongly agree).

RESULTS

Applying Delphi methodology to extrapolate externally developed referral guidelines for local implementation resulted in considerable modifications. For some conditions, many tests from the external group were eliminated by the local group (abdominal bloating; iron deficiency anaemia; irritable bowel syndrome). In contrast, for chronic diarrhoea, abnormal liver enzymes and viral hepatitis, all/most original tests were retained with additional tests added. For liver mass, fatty liver disease and cirrhosis, there was high concordance among the panel with few tests added or eliminated.

CONCLUSIONS

Consideration of externally developed referral guidelines using a consensus-building process leads to significant local tailoring and adaption. Our next steps include implementation and dissemination of these guidelines and evaluating their impact on care efficiency in clinical practice.

Predicting cost of care using self-reported health status data

Background

Methods

Results

Discussion

Conclusions

Reviewing residents' competence: a qualitative study of the role of clinical competency committees in performance assessment

PURPOSE

Clinical competency committees (CCCs) are now required in graduate medical education. This study examined how residency programs understand and operationalize this mandate for resident performance review.

METHOD

In 2013, the investigators conducted semistructured interviews with 34 residency program directors at five public institutions in California, asking about each institution's CCCs and resident performance review processes. They used conventional content analysis to identify major themes from the verbatim interview transcripts.

RESULTS

The purpose of resident performance review at all institutions was oriented toward one of two paradigms: a problem identification model, which predominated; or a developmental model. The problem identification model, which focused on identifying and addressing performance concerns, used performance data such as red-flag alerts and informal information shared with program directors to identify struggling residents.In the developmental model, the timely acquisition and synthesis of data to inform each resident's developmental trajectory was challenging. Participants highly valued CCC members' expertise as educators to corroborate the identification of struggling residents and to enhance credibility of the committee's outcomes. Training in applying the milestones to the CCC's work was minimal.Participants were highly committed to performance review and perceived the current process as adequate for struggling residents but potentially not for others.

CONCLUSIONS

Institutions orient resident performance review toward problem identification; a developmental approach is uncommon. Clarifying the purpose of resident performance review and employing efficient information systems that synthesize performance data and engage residents and faculty in purposeful feedback discussions could enable the meaningful implementation of milestones-based assessment.

Educational and individual factors associated with positive change in and reaffirmation of medical students' intention to practice in underserved areas

PURPOSE

The projected U.S. physician shortage will disproportionately affect underserved areas. This study examined the impact of medical school educational experiences on positive changes in and reaffirmation of students' intention to practice in underserved areas (practice intention).

METHOD

Medical students (n = 7,361) from 113 U.S. MD-granting medical schools who graduated in 2009-2010 and responded to both the Association of American Medical Colleges' 2006 Matriculating Student Questionnaire and 2010 Graduation Questionnaire were included. Multilevel logistic regression analyses were conducted to determine factors associated with change in and reaffirmation of practice intention.

RESULTS

After controlling for individual characteristics, community health field experience (adjusted odds ratio [OR]: 1.36; 95% CI: 1.18, 1.57), learning another language (OR: 1.41; 95% CI: 1.22, 1.63), cultural competence/awareness experience (OR: 1.38; 95% CI: 1.21, 1.58), becoming more aware of perspectives of individuals from different backgrounds (OR: 1.24; 95% CI: 1.04, 1.48), and attending schools with higher social mission scores (OR: 1.66; 95% CI: 1.28, 2.16) were all significantly associated with positive changes in practice intention from matriculation to graduation. Field experience in community health (OR: 1.24; 95% CI: 0.99, 1.53), learning another language (OR: 1.29; 95% CI: 1.01, 1.65), and attending schools with higher social mission scores (OR: 1.62; 95% CI: 1.09, 2.43) were all significantly associated with reaffirmation of practice intention at graduation.

CONCLUSIONS

Multifaceted factors are associated with practice intention. This study suggests medical schools can play active roles in alleviating the physician shortage in underserved areas through targeted curricular interventions and recruitment.

Assessing 3rd year medical students' interprofessional collaborative practice behaviors during a standardized patient encounter: A multi-institutional, cross-sectional study

BACKGROUND

To understand how third-year medical student interprofessional collaborative practice (IPCP) is affected by self-efficacy and interprofessional experiences (extracurricular experiences and formal curricula).

METHODS

The authors measured learner IPCP using an objective structured clinical examination (OSCE) with a standardized nurse (SN) and standardized patient (SP) during a statewide clinical performance examination. At four California medical schools from April to August 2012, SPs and SNs rated learner IPCP (10 items, range 0-100) and patient-centered communication (10 items, range 0-100). Post-OSCE, students reported their interprofessional self-efficacy (16 items, 2 factors, range 1-10) and prior extracurricular interprofessional experiences (3 items). School representatives shared their interprofessional curricula during guided interviews.

RESULTS

Four hundred sixty-four of 530 eligible medical students (88%) participated. Mean IPCP performance was 79.6 ± 14.1 and mean self-efficacy scores were 7.9 (interprofessional teamwork) and 7.1 (interprofessional feedback and evaluation). Seventy percent of students reported prior extracurricular interprofessional experiences; all schools offered formal interprofessional curricula. IPCP was associated with self-efficacy for interprofessional teamwork (β = 1.6, 95% CI [0.1, 3.1], p = 0.04) and patient-centered communication (β = 12.5, 95% CI [2.7, 22.3], p = 0.01).

CONCLUSIONS

Medical student IPCP performance was associated with self-efficacy for interprofessional teamwork and patient-centered communication. Increasing interprofessional opportunities that influence medical students' self-efficacy may increase engagement in IPCP.

Career-long learning: relationship between cognitive and metacognitive skills

OBJECTIVES

 Physicians need metacognitive skills including reflection and goal generation for effective lifelong learning (LLL). These skills are not readily assessed and may not correlate with cognitive skills. We examined early-career physicians' metacognition and relationships between metacognitive skills, cognitive skills, and orientation toward LLL.

METHODS

 Pediatric fellows at UCSF document career progress in annual Individual Development Plans (IDPs). To assess metacognitive skills, we scored narratives in IDPs with a Reflective Ability Rubric (RAR) and goal setting with a SMART Goal Rubric (SMART-GR: consists of global IDP score and four IDP domain subscores). To assess cognitive skills, we collected American Board of Pediatrics scores (ABP), and to measure orientation toward LLL, fellows completed the Jefferson Scale (JeffSPLL). We used Spearman's correlation to examine relationships between scores.

RESULTS

 About 57/66 (86%) fellows participated. Mean scores were: RAR 2.4 ± 1.3 (scale 0-6); SMART-GR global IDP 2.8 ± 1.0, (1-5); JeffSPLL 46.3 ± 3.9 (14-56); and ABP 559.4 ± 75.7. RAR scores correlated significantly with SMART-GR scores but metacognitive measures did not correlate with ABP scores.

CONCLUSIONS

 Our study suggests early-career physicians may have limited metacognitive skills; cognitive and metacognitive skills do not correlate; and orientation toward LLL does not predict metacognitive skills. Thus, we need improved methods to teach and assess metacognition.

How to develop a competency-based examination blueprint for longitudinal standardized patient clinical skills assessments

BACKGROUND

Objective Structured Clinical Exams (OSCEs) with standardized patients (SPs) are commonly used in medical education to assess learners' clinical skills. However, assessments are often discrete rather than intentionally developmentally sequenced.

AIMS

We developed an examination blueprint to optimize assessment and feedback to learners with purposeful sequence as a series of longitudinally integrated assessments based on performance milestones. Integrated and progressive clinical skills assessments offer several benefits: assessment of skill development over time, systematic identification of learning needs, data for individualized feedback and learning plans, and baseline reference points for reassessment.

METHODS

Using a competency-based medical education (CBME) framework, we translated pre-determined competency milestones for medical students' patient encounters into a four-year SP-based OSCE examination blueprint.

RESULTS

Initial evaluation of cases using the blueprint revealed opportunities to target less frequently assessed competencies and to align assessments with milestones for each year.

CONCLUSIONS

The examination blueprint can guide ongoing SP-based OSCE case design. Future iterations of examination blueprints can incorporate lessons learnt from evaluation data and student feedback.

Does student confidence on multiple-choice question assessments provide useful information?

CONTEXT

Feedback from multiple-choice question (MCQ) assessments is typically limited to a percentage correct score, from which estimates of student competence are inferred. The students' confidence in their answers and the potential impact of incorrect answers on clinical care are seldom recorded. Our purpose was to evaluate student confidence in incorrect responses and to establish how confidence was influenced by the potential clinical impact of answers, question type and gender.

METHODS

This was an exploratory, cross-sectional study conducted using a convenience sample of 104 Year 3 dental students completing 20 MCQs on implant dentistry. Students were asked to select the most correct response and to indicate their confidence in it for each question. Identifying both correctness and confidence allowed the designation of uninformed (incorrect and not confident) or misinformed (incorrect but confident) responses. In addition to recording correct/incorrect responses and student confidence, faculty staff designated incorrect responses as benign, inappropriate or potentially harmful if applied to clinical care. Question type was identified as factual or complex. Logistic regression was used to evaluate relationships between student confidence, and question type and gender.

RESULTS

Students were misinformed more often than uninformed (22% versus 8%), and misinformed responses were more common with complex than factual questions (p < 0.05). Students were significantly more likely to be confident of correct than incorrect benign, incorrect inappropriate or incorrect harmful answers (p < 0.001), but, contrary to expectations, confidence did not decrease as answers became more harmful.

CONCLUSIONS

Recording student confidence was helpful in identifying uninformed versus misinformed responses, which may allow for targeted remediation strategies. Making errors of calibration (confidence and accuracy) more visible may be relevant in feedback for professional development.

Online learning in a dermatology clerkship: piloting the new American Academy of Dermatology Medical Student Core Curriculum

BACKGROUND

Multiple studies have shown that both current and future primary care providers have insufficient education and training in dermatology. To address the limitations and wide variability in medical student dermatology instruction, the American Academy of Dermatology (AAD) created a standardized, online curriculum for both dermatology learners and educators.

OBJECTIVE

We sought to determine the impact of the integration of the AAD online curriculum into a 2-week introductory dermatology clerkship for fourth-year medical students.

METHODS

In addition to their clinical duties, we assigned 18 online modules at a rate of 1 to 3 per day. We evaluated knowledge acquisition using a 50-item, multiple-choice pretest and posttest. Postmodule and end-of-course questionnaires contained both closed and open-ended items soliciting students' perceptions about usability and satisfaction.

RESULTS

All 51 participants significantly improved in their dermatology knowledge (P < .001). The majority of students found the modules easy to navigate (95%) and worth their time (93%). All respondents supported the continuation of the modules as part of the dermatology clerkship.

LIMITATIONS

Without a control group who did not experience the online curriculum, we are unable to isolate the specific impact of the online modules on students' learning.

CONCLUSION

This study demonstrates the successful integration of this educational resource into a 2-week, university-based dermatology clerkship. Students' perceptions regarding usability and satisfaction were overwhelmingly positive, suggesting that the online curriculum is highly acceptable to learners. Widespread use of this curriculum may be a significant advancement in standardized dermatology learning for medical students.

The impact of demographic characteristics on nonresponse in an ambulatory patient satisfaction survey

BACKGROUND

Despite the increasing use of patient satisfaction data for compensation and comparison of performance, low response rate and potential bias of the respondents pose a significant threat to validity of the data. The demographic profiles of respondents and nonrespondents to a patient satisfaction survey in the ambulatory care setting were compared to explore the impact ofnonresponse bias.

METHODS

Patient satisfaction survey data were collected from October through December 2010 for outpatient facilities at a large academic medical center. The association between respondent characteristics and satisfaction ratings on three dimensions of the clinical care process--(1) interpersonal communication (clarity of language), (2) service delivery (overall care during visit), and (3) likelihood of recommending practice to others-were assessed with bivariate and multivariate linear regression. Weighted analyses were performed to examine the impact of nonresponse.

RESULTS

Surveys were mailed to 15,549 patients, of which 4,952 (32%) were returned. Respondents had greater proportions of elderly, female, and English speakers. Bivariate analyses showed significant difference in satisfaction ratings by age, language, and insurance type. Multivariate regression analysis showed significant confounding across variables. On the basis of the calculated weighted means, mean satisfaction ratings were discrepant for language and age; however, the overall satisfaction ratings for each dimension were minimally affected.

CONCLUSION

Nonresponse rates and satisfaction ratings differed by age, language, and insurance type. However, if it is assumed that nonrespondents within these demographic groups have similar satisfaction ratings as respondents, then nonresponse levels appear to have minimal effects on overall satisfaction ratings.

A comparison of two standard-setting approaches in high-stakes clinical performance assessment using generalizability theory

PURPOSE

Scoring clinical assessments in a reliable and valid manner using criterion-referenced standards remains an important issue and directly affects decisions made regarding examinee proficiency. This generalizability study of students' clinical performance examination (CPX) scores examines the reliability of those scores and of their interpretation, particularly according to a newly introduced, "critical actions" criterion-referenced standard and scoring approach.

METHOD

The authors applied a generalizability framework to the performance scores of 477 third-year students attending three different medical schools in 2008. The norm-referenced standard included all station checklist items. The criterion-referenced standard included only those items deemed critical to patient care by a faculty panel. The authors calculated and compared variance components and generalizability coefficients for each standard across six common stations.

RESULTS

Norm-referenced scores had moderate generalizability (ρ = 0.51), whereas criterion-referenced scores showed low dependability (φ = 0.20). The estimated 63% of measurement error associated with the person-by-station interaction suggests case specificity. Increasing the number of stations on the CPX from 6 to 24, an impractical solution both for cost and time, would still yield only moderate dependability (φ = 0.50).

CONCLUSIONS

Though the performance assessment of complex skills, like clinical competence, seems intrinsically valid, careful consideration of the scoring standard and approach is needed to avoid misinterpretation of proficiency. Further study is needed to determine how best to improve the reliability of criterion-referenced scores, by implementing changes to the examination structure, the process of standard-setting, or both.

Profiling students for remediation using latent class analysis

While clinical exams using SPs are used extensively across the medical schools for summative purposes and high-stakes decisions, the method of identifying students for remediation varies widely and there is a lack of consensus on the best methodological approach. The purpose of this study is to provide an alternative approach to identification of students for remediation using the latent class analysis (LCA) technique. 147 third year medical students participating in the Clinical Performance Examination (CPX) are included in the study. We used LCA to identify students who potentially need remediation based on their performance on CPX. Three distinct clusters of students with different performance profiles were identified. The identification of two rather than one low performing group has significant implications for identifying cut-points as well as for remediation programs. The two low performing groups in our study had low scores on contrasting sets of cases. LCA presents an alternative approach to identification of borderline or low performing groups. This method provides advantages over traditional statistical techniques such as cluster analysis used for grouping students. Based on the flexibility of the model specification, within the LCA framework, we were able to identify more than one group that may need remediation or instruction support.