Faculty and the observation of trainees' clinical skills: problems and opportunities

Performance of Cardiovascular Physical Exam Skills by Internal Medicine Residents

BACKGROUND

Few studies have assessed the ability of internal medicine residents to perform a cardiovascular physical examination using real patients.

METHODS

First year internal medicine interns from 2 large academic medical centers in Maryland examined the same patient with aortic insufficiency as part of the Assessment of Physical Examination and Communication Skills (APECS). Interns were assessed on 5 clinical domains: physical exam technique, identifying physical signs, generating a differential diagnosis, clinical judgment, and maintaining patient welfare. Spearman's correlation test was used to describe associations between clinical domains. Preceptor comments were examined to identify common errors in physical exam technique and identifying physical signs.

RESULTS

One hundred nine interns examined the same patient with aortic insufficiency across 14 APECS sessions. Only 58 interns (53.2%) correctly identified the presence of a diastolic murmur, and only 52 interns (47.7%) included aortic insufficiency on their differential diagnosis. There was a significant and positive correlation between physical exam technique and identification of the correct physical findings (r = 0.42, P < .001). Both technique (r = 0.34, P = .003) and identifying findings (r = 0.42, P < .001) were significantly associated with generating an appropriate differential diagnosis. Common errors in technique included auscultating over the gown, timing the cardiac cycle with the radial pulse, and failing to palpate for the apical impulse.

CONCLUSIONS

Internal medicine interns had variable skills in performing and interpreting the cardiovascular physical exam. Improving cardiovascular exam skills would likely lead to increased identification of relevant cardiovascular findings, inform clinical decision making and improve overall patient care.

Mobile Application to Improve Just-in-Time 2-Way Formative Feedback in Graduate Medical Education

Background An easy-to-use application to facilitate direct observation and allow for 2-way feedback between residents and faculty is needed. Objective To develop a mobile-based application (app) with the goals of (1) providing just-in-time feedback to residents; (2) improving timeliness of feedback by faculty; and (3) allowing residents to comment on the value of faculty feedback. Methods Fifty-one of 69 (74%) internal medicine (IM) residents and 20 of 25 (80%) IM core faculty participated in the study from July 1, 2020, to December 31, 2021. An iOS app was designed by authors with expertise in medical education and application development to capture entrustable professional activities (EPAs)-based feedback (eg, informed consent) based on direct observation of residents' skills in the workplace. App utilization and narrative feedback characteristics of faculty comments were examined by exporting the data from the database server. The end user satisfaction was examined using a survey instrument. Results Eighty-seven percent of assessments (117 of 134) initiated were fully completed by residents and faculty. Faculty narrative comments were noted in 97% (114 of 117) of completed assessments and 64% (75 of 117) of residents' feedback to the faculty contained narrative comments. Eighty-three percent (97 of 117) of comments were behaviorally specific and 71% (83 of 117) contained an actionable item. Eighty-six percent (18 of 21) of residents and 90% (9 of 10) of core faculty stated that this application promoted an educational interaction between them. Conclusions This app facilitates the efficient completion of EPA-based formative assessments and captures bidirectional feedback in the workplace setting.

Croix R, van Melle E. Competence By Design: a transformational national model of time-variable competency-based postgraduate medical education

Postgraduate medical education is an essential societal enterprise that prepares highly skilled physicians for the health workforce. In recent years, PGME systems have been criticized worldwide for problems with variable graduate abilities, concerns about patient safety, and issues with teaching and assessment methods. In response, competency based medical education approaches, with an emphasis on graduate outcomes, have been proposed as the direction for 21st century health profession education. However, there are few published models of large-scale implementation of these approaches. We describe the rationale and design for a national, time-variable competency-based multi-specialty system for postgraduate medical education called Competence by Design. Fourteen innovations were bundled to create this new system, using the Van Melle Core Components of competency based medical education as the basis for the transformation. The successful execution of this transformational training system shows competency based medical education can be implemented at scale. The lessons learned in the early implementation of Competence by Design can inform competency based medical education innovation efforts across professions worldwide.

Progress testing of an objective structured clinical examination during undergraduate clinical clerkship: a mixed-methods pilot study

BACKGROUND

Progress testing is an assessment method in which an examination reflecting competencies at graduation is regularly administered to students over multiple years, thereby facilitating self-directed learning. However, the significance of the objective structured clinical examination as a progress test in undergraduate education, needs to be determined. This study provides evidence of the role of the objective structured clinical examination for progress testing and optimal scoring methods for assessing students in different academic years.

METHODS

We conducted a sequential explanatory mixed-methods pilot study. Participants were assessed using the Item Rating Scale, the year-adjusted Global Rating Scale, and the Training Level Rating Scale. The characteristics of each scale were compared quantitatively. In addition, the influence of the objective structured clinical examination as a progress test on learning attitudes was examined. Qualitative data from a post-examination questionnaire were analyzed, using content analysis to explore influences on self-directed learning.

RESULTS

Sixth and fifth year clinical students (n = 235) took the objective structured clinical examination progress test. The total Item Rating Scales were recorded (%) as 59.03 ± 5.27 and 52.64 ± 5.08 (p < 0.01); Training Level Rating Scale was 3.94 ± 0.39 vs 3.22 ± 0.42 (p < 0.01); and the year-adjusted Global Rating Scale was 4.25 ± 0.44 vs 4.32 ± 0.52 (no significant difference), for the sixth and fifth year students, respectively. The correlations across stations and the reliability of each station were satisfactory. Four categories were identified in the qualitative analysis: "motivation to learn during the clinical clerkship was promoted," "dissatisfied with being asked about things they had not experienced," "confusion about being unable to use conventional test-taking strategies," and "insufficient understanding of competencies at graduation." The scores indicated significant differences in performance according to training year.

CONCLUSIONS

This study provides evidence that the objective structured clinical examination can be used as a progress testing tool for undergraduate clinical clerkships. Further enhancement of training opportunities and dissemination of performance competency goals in clerkship curricula are required if we intend to promote self-directed learning through progress testing.

"Rater training" re-imagined for work-based assessment in medical education

In this perspective, the authors critically examine "rater training" as it has been conceptualized and used in medical education. By "rater training," they mean the educational events intended to improve rater performance and contributions during assessment events. Historically, rater training programs have focused on modifying faculty behaviours to achieve psychometric ideals (e.g., reliability, inter-rater reliability, accuracy). The authors argue these ideals may now be poorly aligned with contemporary research informing work-based assessment, introducing a compatibility threat, with no clear direction on how to proceed. To address this issue, the authors provide a brief historical review of "rater training" and provide an analysis of the literature examining the effectiveness of rater training programs. They focus mainly on what has served to define effectiveness or improvements. They then draw on philosophical and conceptual shifts in assessment to demonstrate why the function, effectiveness aims, and structure of rater training requires reimagining. These include shifting competencies for assessors, viewing assessment as a complex cognitive task enacted in a social context, evolving views on biases, and reprioritizing which validity evidence should be most sought in medical education. The authors aim to advance the discussion on rater training by challenging implicit incompatibility issues and stimulating ways to overcome them. They propose that "rater training" (a moniker they suggest be reserved for strong psychometric aims) be augmented with "assessor readiness" programs that link to contemporary assessment science and enact the principle of compatibility between that science and ways of engaging with advances in real-world faculty-learner contexts.

Inclusion of the workshop model in the standardized training of emergency medicine residents

Background

Standardized training of resident physicians (STRP) includes clinical practice, professional required courses, and public required courses, among others. Of them, clinical practice is the most important as it allows residents to implement what they have learned in theoretical education to practice. Clinical practice includes different teaching methods, such as traditional lectures, bedside teaching, and workshops, and each method has its advantages and disadvantages in different situations of interest. Emergency medicine (EM) focuses on the diagnosis and treatment of urgent medical conditions and entails several emergency procedures. In this study, we aimed to compare the effects of workshop-based STRP and traditional STRP on emergency physicians.

Methods

Overall, 125 residents who received STRP in EM between January and December 2021 were selected and randomly divided into two groups: the control group (n = 60; received traditional teaching) and the intervention group (n = 65; received workshop-based training). The theoretical performance, operative performance, and satisfaction of both groups were compared and analyzed.

Results

Regarding theoretical assessment, the scores of airway management, cardiopulmonary resuscitation, and trauma management in the intervention group were 4.81 (t = 5.82, p < 0.001), 6.90 (t = 7.72, p < 0.001), and 5.25 (t = 6.14, p < 0.001), respectively. Regarding skill assessment, the scores for the same items in the intervention group were 4.43 (t = 5.30, p < 0.001), 4.55 (t = 5.61, p < 0.001), and 5.62 (t = 6.65, P < 0.001), respectively. Regarding satisfaction evaluation, the scores in the intervention group were 1.99 (t = 6.03, p < 0.001), 1.98 (t = 6.41, p < 0.001), and 1.96 (t = 6.14, p < 0.001), respectively. Overall, the scores were higher in the intervention group than in the control group.

Conclusion

The workshop training model effectively improves the theoretical knowledge and practical skills of EM residents undergoing standardized training. The residents found the training and its outcomes satisfactory, ultimately improving their emergency response and first-responder skills.

Entrustable Professional Activities and Learning: The Postgraduate Trainee Perspective

OBJECTIVE

Entrustable professional activities (EPAs) are used as clinical activities in postgraduate psychiatry training in Australasia. This study aimed to explore psychiatry trainees' perceptions of the impact of EPAs on their motivation and learning.

METHODS

A constructivist grounded theory approach was used to conceptualize the impact of EPAs on trainees' motivation and learning. A purposive sample of trainees was recruited from across New Zealand. Semi-structured individual interviews were used for data collection and continued until theoretical saturation was reached.

RESULTS

The impact of EPAs on learning was mediated by the trainee's appraisals of subjective control, value, and the costs of engaging with EPAs. When appraisals were positive, EPAs encouraged a focus on particular learning needs and structured learning with the supervisor. However, when appraisals were negative, EPAs encouraged a superficial approach to learning. Trainee appraisals and their subsequent impact on motivation and learning were most affected by EPA granularity, alignment of EPAs with clinical practice, and the supervisor's conscientiousness in their approach to EPAs.

CONCLUSIONS

To stimulate learning, EPAs must be valued by both trainees and supervisors as constituting a coherent work-based curriculum that encompasses the key fellowship competencies. If EPAs are to be effective as clinical tasks for learning, ongoing faculty development must be the leading priority.

Master clinical teachers and personalised learning

Clinical history taking and physical examination are two of the most important competencies of physicians. In addition to informing diagnoses, these activities build rapport and establish relationships between caregivers and patients. Despite this, emphasis on the assessment of bedside clinical skills is declining. To prepare our students for clinical work, we began a clinical competency, personalised teaching programme in which students perform a history and physical examination in front of a master clinical teacher (MCT) approximately every 2 weeks throughout their core clerkship year. The MCT works with the student in a clinical encounter, providing personalised bedside instruction on all features of being a clinician including bedside manner, history-taking skills, physical examination skills, and clinical reasoning. The MCT then provides an assessment of student's competency development and gives feedback to the student about what they do well and where they have opportunities for growth. Assessment data are collected and tracked longitudinally across the clerkship phase to ensure that each student is progressing developmentally. With over 6000 observations of student performance, we are able to discern competency development and growth over time. We can identify if a student is not improving as expected during their clerkship phase and intervene by providing extra practice and training. This core clerkship teaching programme has been well received by both students and instructors and has led us to pilot this approach during the post-clerkship phase of our medical training.

Utopia versus dystopia: Professional perspectives on the impact of healthcare artificial intelligence on clinical roles and skills

BACKGROUND

Alongside the promise of improving clinical work, advances in healthcare artificial intelligence (AI) raise concerns about the risk of deskilling clinicians. This purpose of this study is to examine the issue of deskilling from the perspective of diverse group of professional stakeholders with knowledge and/or experiences in the development, deployment and regulation of healthcare AI.

METHODS

We conducted qualitative, semi-structured interviews with 72 professionals with AI expertise and/or professional or clinical expertise who were involved in development, deployment and/or regulation of healthcare AI. Data analysis using combined constructivist grounded theory and framework approach was performed concurrently with data collection.

FINDINGS

Our analysis showed participants had diverse views on three contentious issues regarding AI and deskilling. The first involved competing views about the proper extent of AI-enabled automation in healthcare work, and which clinical tasks should or should not be automated. We identified a cluster of characteristics of tasks that were considered more suitable for automation. The second involved expectations about the impact of AI on clinical skills, and whether AI-enabled automation would lead to worse or better quality of healthcare. The third tension implicitly contrasted two models of healthcare work: a human-centric model and a technology-centric model. These models assumed different values and priorities for healthcare work and its relationship to AI-enabled automation.

CONCLUSION

Our study shows that a diverse group of professional stakeholders involved in healthcare AI development, acquisition, deployment and regulation are attentive to the potential impact of healthcare AI on clinical skills, but have different views about the nature and valence (positive or negative) of this impact. Detailed engagement with different types of professional stakeholders allowed us to identify relevant concepts and values that could guide decisions about AI algorithm development and deployment.

Initial Development of an Automated Platform for Assessing Trainee Performance on Case Presentations

Background

Oral case presentation is a crucial skill of physicians and a key component of team-based care. However, consistent and objective assessment and feedback on presentations during training are infrequent.

Objective

To determine the potential value of applying natural language processing, computer software that extracts meaning from text, to transcripts of oral case presentations as a strategy to assess their quality automatically and objectively.

Methods

We transcribed a collection of simulated oral case presentations. The presentations were from eight critical care fellows and one critical care attending. They were instructed to review the medical charts of 11 real intensive care unit patient cases and to audio record themselves, presenting each case as if they were doing so on morning rounds. We then used natural language processing to convert the transcripts from human-readable text into machine-readable numbers. These numbers represent details of the presentation style and content. The distance between the numeric representation of two different transcripts negatively correlates with the similarity of those two transcripts. We ranked fellows on the basis of how similar their presentations were to the attending's presentations.

Results

The 99 presentations included 260 minutes of audio (mean length: 2.6 ± 1.24 min per case). On average, 23.88 ± 2.65 sentences were spoken, and each sentence had 14.10 ± 0.67 words, 3.62 ± 0.15 medical concepts, and 0.75 ± 0.09 medical adjectives. When ranking fellows on the basis of how similar their presentations were to the attending's presentation, we found a gap between the five fellows with the most similar presentations and the three fellows with the least similar presentations (average group similarity scores of 0.62 ± 0.01 and 0.53 ± 0.01, respectively). Rankings were sensitive to whether presentation style or content information were weighted more heavily when calculating transcript similarity.

Conclusion

Natural language processing enabled the ranking of case presentations on the basis of how similar they were to a reference presentation. Although additional work is needed to convert these rankings, and underlying similarity scores, into actionable feedback for trainees, these methods may support new tools for improving medical education.

Feasibility and Effectiveness of Mini-Clinical Evaluation Exercise (Mini-CEX) in an Undergraduate Medical Program: A Study From Pakistan

Background

In clinical settings, direct observation (DO) with feedback is an effective method to assess and improve learner performance. One tool used for DO is the mini-clinical evaluation exercise (Mini-CEX). We conducted a study to assess the effectiveness and feasibility of Mini-CEX for medical students at Aga Khan University, Karachi.

Methods

Utilizing a purposive sampling technique, a total of 199 students in six core clerkships of Years 3 and 4 were selected for this study. Participating faculty underwent training workshops for use of Mini-CEX and feedback strategies. Each student was assessed twice by one faculty, using a modified version of the Mini-CEX, which assessed four domains of clinical skills: Data Gathering, Communication, Diagnosis/Differential, and Management Plan and Organization. Feedback was given after each encounter. Faculty and students also provided detailed feedback regarding the process of DO.

Data were analyzed using Statistical Package for Social Sciences (SPSS) version 26 (IBM Corp., Armonk, NY, USA), with categorical variables arranged as frequencies and percentages. The Chi-squared test was used for further statistical analyses, and a P-value of < 0.05 was considered statistically significant. Effectiveness was assessed via a change in student performance between the first and second Mini-CEX, and feasibility was assessed via qualitative feedback.

Results

We obtained three sets of results: Mini-CEX forms (523), from which we included a total 350 evaluations for analysis, 216 from Year 3 and 134 from Year 4, and feedback on DO: student (70) and faculty (18). Year 3 students performed significantly better in all foci of the Mini-CEX between the first and second assessment (P ≤ 0.001), whereas in Year 4, significant improvement was limited to only two domains of the Mini-CEX [Communication of History/Physical Examination (P = 0.040) and Diagnosis/Differential and Management Plan (P < 0.001)]. Students (65.7%) and faculty (94.4%) felt this exercise improved their interaction. 83.3% faculty recommended its formal implementation compared to 27.1% of students, who reported challenges in implementation of the Mini-CEX such as time constraints, logistics, the subjectivity of assessment, and varying interest by faculty.

Conclusion

Direct observation using Mini-CEX is effective in improving the clinical and diagnostic skills of medical students and strengthens student-faculty interaction. While challenges exist in its implementation, the strategic placement of Mini-CEX may enhance its utility in measuring student competency.

At the limits of digital education. The importance of practical education for clinical competencies learning in the field of emergency medicine: A controlled non-randomized interventional study

Introduction

A high-quality education of future physicians is essential. Modern approaches interlock the acquisition of theoretical knowledge and practical skills in a spiral curriculum, leading to a mutual learning benefit for knowledge and application. This model was challenged by the elimination of hands-on trainings during the pandemic, which were often replaced by purely digital teaching models. Given the holistic nature of the spiral curriculum, we assumed that a purely digital model would have an impact on knowledge acquisition due to missing hands-on learning opportunities. The aim of the study was to investigate, using an emergency seminar as an example, whether purely digital training leads to a difference in theoretical knowledge compared to the traditional model.

Materials and methods

Study design: We used a two-groups design comparing a sample of medical students taught in 2020 with a purely digital teaching format (DF; n = 152) with a historical control group taught with a traditional format (TF; n = 1060). Subject of investigation was a seminar on emergency medicine, taking place in the 4th year. Outcome parameters: The primary outcome parameter was the students' acquired knowledge, measured by the score achieved in the final exams. Students' evaluation of the seminar was used as a secondary outcome parameter.

Results

Students in the DF group scored significantly lower than students in the TF group in the final exams. Students in the DF group rated the course significantly worse than students in the TF group.

Discussion

The study results illustrate that purely digital education leads to inferior knowledge acquisition compared to the traditional spiral curriculum. A possible explanation may lie in a deeper processing of the information (e.g., understanding the information by experience and analysis) and accordingly a better memory recall. Moreover, the students' critical appraisal of the DF may have had an unfavorable effect on learning performance. Moderating factors may be lower learning motivation or the “zoom fatigue” effect.

Conclusion

These study results clearly illustrate the importance of hands-on teaching for knowledge acquisition. The interlocking of theoretical knowledge and practical skills, as ensured by the spiral curriculum, is essential.

Cancel culture: exploring the unintended consequences of cancelling the Canadian national licensing clinical examination

Assessment drives learning. However, when it comes to high-stakes examinations (e.g., for licensure or certification), these assessments of learning may be seen as unnecessary hurdles by some. Licensing clinical skills assessment in particular have come under fire over the years. Recently, assessments such as the Medical Council of Canada Qualifying Examination Part II, a clinical skills objective structured clinical examination, have been permanently cancelled. The authors explore potential consequences of this cancellation including those that are inadvertent and undesirable. Future next steps for clinical skills assessment are explored.

Does direct observation influence the quality of workplace-based assessment documentation?

BACKGROUND

A key component of competency-based medical education (CBME) is direct observation of trainees. Direct observation has been emphasized as integral to workplace-based assessment (WBA) yet previously identified challenges may limit its successful implementation. Given these challenges, it is imperative to fully understand the value of direct observation within a CBME program of assessment. Specifically, it is not known whether the quality of WBA documentation is influenced by observation type (direct or indirect).

METHODS

The objective of this study was to determine the influence of observation type (direct or indirect) on quality of entrustable professional activity (EPA) assessment documentation within a CBME program. EPA assessments were scored by four raters using the Quality of Assessment for Learning (QuAL) instrument, a previously published three-item quantitative measure of the quality of written comments associated with a single clinical performance score. An analysis of variance was performed to compare mean QuAL scores among the direct and indirect observation groups. The reliability of the QuAL instrument for EPA assessments was calculated using a generalizability analysis.

RESULTS

A total of 244 EPA assessments (122 direct observation, 122 indirect observation) were rated for quality using the QuAL instrument. No difference in mean QuAL score was identified between the direct and indirect observation groups (p = 0.17). The reliability of the QuAL instrument for EPA assessments was 0.84.

CONCLUSIONS

Observation type (direct or indirect) did not influence the quality of EPA assessment documentation. This finding raises the question of how direct and indirect observation truly differ and the implications for meta-raters such as competence committees responsible for making judgments related to trainee promotion.

Interprofessional bedside rounding improves quality of feedback to resident physicians

PURPOSE

Obtaining high quality feedback in residency education is challenging, in part due to limited opportunities for faculty observation of authentic clinical work. This study reviewed the impact of interprofessional bedside rounds ('iPACE™') on the length and quality of faculty narrative evaluations of residents as compared to usual inpatient teaching rounds.

METHODS

Narrative comments from faculty evaluations of Internal Medicine (IM) residents both on usual teaching service as well as the iPACE™ service (spanning 2017-2020) were reviewed and coded using a deductive content analysis approach.

RESULTS

Six hundred ninety-two narrative evaluations by 63 attendings of 103 residents were included. Evaluations of iPACE™ residents were significantly longer than those of residents on usual teams (109 vs. 69 words, p < 0.001). iPACE™ evaluations contained a higher average occurrence of direct observations of patient/family interactions (0.72 vs. 0.32, p < 0.001), references to interprofessionalism (0.17 vs. 0.05, p < 0.001), as well as specific (3.21 vs. 2.26, p < 0.001), actionable (1.01 vs. 0.69, p < 0.001), and corrective feedback (1.2 vs. 0.88, p = 0.001) per evaluation.

CONCLUSIONS

This study suggests that the iPACE™ model, which prioritizes interprofessional bedside rounds, had a positive impact on the quantity and quality of feedback, as measured via narrative comments on weekly evaluations.

Novice Doctors in the Emergency Department: A Scoping Review

In many emergency departments (EDs), young, inexperienced doctors treat patients who are critically ill. At the start of their career, these novice doctors are not sufficiently qualified to take care of these potentially critically ill patients in the highly demanding environment of an ED. This not only poses a threat to the well-being of the doctor, who feels inadequately prepared and experiences a lot of stress, but also to that of the patients, who may not receive optimal care. Lastly, young doctors may influence the efficiency of the organization, with longer throughput times, more orders of ancillary investigations, and more admissions.

Training novice doctors with regard to simple or complex skills using simulation techniques is part of the solution. However, the transfer of newly learned skills to clinical practice remains unexplored, and not everything can be trained before the actual skill is required. Therefore, it is important to train young doctors in their learning abilities, for instance, teach them how to be adaptive and how to use their skills in new situations. Lastly, the way care is organized is essential. Good supervision, leaving room for the learning processes of young doctors, developing a team with more experienced professionals (paramedics, nurses, and doctors), and well-organized processes, aiming to reduce the complexity of the work, are ways to improve the quality of care, independent of the experience level of the novice doctor.

Relationship of Physical Examination Technique to Associated Clinical Skills: Results from a Direct Observation Assessment

BACKGROUND

The purpose of this research was to use direct observation of the physical examination to elucidate the role physical examination technique plays in diagnostic accuracy. Physical examination is important for quality clinical care and requires multiple interrelated skills. The relationship of physical examination technique to related skills is poorly understood. Current methods of teaching and assessing physical examination skills provide few opportunities to evaluate physical examination technique and accuracy.

METHODS

The authors developed a clinical examination assessment using volunteer patients and direct observation. Trained faculty preceptors rated resident performance in 7 domains: 1) physical examination technique, 2) identification of physical signs, 3) clinical communication, 4) differential diagnosis, 5) clinical judgment, 6) managing patient concerns, and 7) maintaining patient welfare. The Pearson correlation coefficient was used to determine relationships between performance in each of these domains. Data on residents' self-assessed competency in the physical examination and perceptions of feedback received during the assessment were collected.

RESULTS

From December 2018 to February 2020, 113 interns from 2 internal medicine residency programs participated in the assessment. Physical examination technique was significantly correlated with accurate identification of physical signs, differential diagnosis and clinical judgment. Time spent in graduate medical education was negatively correlated with performance. Interns more highly rated the feedback received from this assessment than traditional clinical skills feedback.

CONCLUSIONS

Our findings emphasize the necessity of multi-dimensional physical examination assessment. Observed deterioration of physical examination skill during internship may reflect contemporary practice patterns, which deprioritize the physical examination. Future research on physical examination education should focus on the interface between physical examination technique and related clinical skills.

Supervisory dyads' communication and alignment regarding the use of workplace-based observations: a qualitative study in general practice residency

BACKGROUND

In medical residency, performance observations are considered an important strategy to monitor competence development, provide feedback and warrant patient safety. The aim of this study was to gain insight into whether and how supervisor-resident dyads build a working repertoire regarding the use of observations, and how they discuss and align goals and approaches to observation in particular.

METHODS

We used a qualitative, social constructivist approach to explore if and how supervisory dyads work towards alignment of goals and preferred approaches to performance observations. We conducted semi-structured interviews with supervisor-resident dyads, performing a template analysis of the data thus obtained.

RESULTS

The supervisory dyads did not frequently communicate about the use of observations, except at the start of training and unless they were triggered by internal or external factors. Their working repertoire regarding the use of observations seemed to be primarily driven by patient safety goals and institutional assessment requirements rather than by providing developmental feedback. Although intended as formative, the institutional test was perceived as summative by supervisors and residents, and led to teaching to the test rather than educating for purposes of competence development.

CONCLUSIONS

To unlock the full educational potential of performance observations, and to foster the development of an educational alliance, it is essential that supervisory dyads and the training institute communicate clearly about these observations and the role of assessment practices of- and for learning, in order to align their goals and respective approaches.

Virtual auscultation course via video chat in times of COVID-19 improves cardiac auscultation skills compared to literature self-study in third-year medical students: a prospective randomized controlled cross-over study

BACKGROUND

Cardiac auscultation is a core clinical skill taught in medical school. Due to contact restrictions during the SARS-CoV-2 pandemic, interaction with patients was very limited. Therefore, a peer-to-peer virtual case-based auscultation course via video conference was established.

METHODS

A randomized controlled cross-over study was conducted to evaluate whether participation in a virtual auscultation course could improve heart auscultation skills in 3^rd-year medical students. A total of sixty medical students were randomly assigned to either the experimental or control group after informed consent was obtained. Due to no-shows, 55 students participated. Depending on allocation, students attended three ninety-minute courses in intervals of one week in a different order: a virtual case-based auscultation course held via video chat, literature self-study, and an on-site course using a high-fidelity auscultation simulator (SAM II). The study's primary endpoint was the performance of the two groups at the simulator after participating in the virtual auscultation course or literature self-study. To evaluate their auscultation skills, students participated in five assessments using the same six pathologies: stenosis and regurgitation of the aortic and mitral valve, ventricular septal defect, and patent ductus arteriosus. Moreover, participants rated their satisfaction with each course and provided a self-assessment of competence.

RESULTS

Compared to literature self-study, participation in the virtual auscultation course led to a significantly improved description of heart murmurs at the auscultation simulator with regard to the presence in systole and diastole, low- and high-pitched sounds, and volume dynamics. There was no significant difference between the groups in diagnostic accuracy and identification of the point of maximal intensity. After the virtual course, students showed higher satisfaction rates and a higher increase in self-assessed competence compared to participants who engaged in literature self-study.

CONCLUSIONS

For the first time, this study demonstrates that a case-based virtual auscultation course can improve aspects of cardiac auscultation skills on a simulator. This may facilitate the further acquisition of an essential clinical skill, even when contact restrictions will be lifted.

A Question of Scale? Generalizability of the Ottawa and Chen Scales to Render Entrustment Decisions for the Core EPAs in the Workplace

PURPOSE

Assessments of the Core Entrustable Professional Activities (Core EPAs) are based on observations of supervisors throughout a medical student's progression toward entrustment. The purpose of this study was to compare generalizability of scores from 2 entrustment scales: the Ottawa Surgical Competency Operating Room Evaluation (Ottawa) scale and an undergraduate medical education supervisory scale proposed by Chen and colleagues (Chen). A secondary aim was to determine the impact of frequent assessors on generalizability of the data.

METHOD

For academic year 2019-2020, the Virginia Commonwealth University School of Medicine modified a previously described workplace-based assessment (WBA) system developed to provide feedback for the Core EPAs across clerkships. The WBA scored students' performance using both Ottawa and Chen scales. Generalizability (G) and decision (D) studies were performed using an unbalanced random-effects model to determine the reliability of each scale. Secondary G- and D-studies explored whether faculty who rated more than 5 students demonstrated better reliability. The Phi-coefficient was used to estimate reliability; a cutoff of at least 0.70 was used to conduct D-studies.

RESULTS

Using the Ottawa scale, variability attributable to the student ranged from 0.8% to 6.5%. For the Chen scale, student variability ranged from 1.8% to 7.1%. This indicates the majority of variation was due to the rater (42.8%-61.3%) and other unexplained factors. Between 28 and 127 assessments were required to obtain a Phi-coefficient of 0.70. For 2 EPAs, using faculty who frequently assessed the EPA improved generalizability, requiring only 5 and 13 assessments for the Chen scale.

CONCLUSIONS

Both scales performed poorly in terms of learner-attributed variance, with some improvement in 2 EPAs when considering only frequent assessors using the Chen scale. Based on these findings in conjunction with prior evidence, the authors provide a root cause analysis highlighting challenges with WBAs for Core EPAs.

A Pilot Study of the Generalizability of Preclinical Entrustment Assessments in Undergraduate Medical Education

PURPOSE

The reproducibility and consistency of assessments of entrustable professional activities (EPAs) in undergraduate medical education (UME) have been identified as potential areas of concern. EPAs were designed to facilitate workplace-based assessments by faculty with a shared mental model of a task who could observe a trainee complete the task multiple times. In UME, trainees are frequently assessed outside the workplace by faculty who only observe a task once.

METHOD

In November 2019, the authors conducted a generalizability study (G-study) to examine the impact of student, faculty, case, and faculty familiarity with the student on the reliability of 162 entrustment assessments completed in a preclerkship environment. Three faculty were recruited to evaluate 18 students completing 3 standardized patient (SP) cases. Faculty familiarity with each student was determined. Decision studies were also completed. Secondary analysis of the relationship between student performance and entrustment (scoring inference) compared average SP checklist scores and entrustment scores.

RESULTS

G-study analysis revealed that entrustment assessments struggled to achieve moderate reliability. The student accounted for 30.1% of the variance in entrustment scores with minimal influence from faculty and case, while the relationship between student and faculty accounted for 26.1% of the variance. G-study analysis also revealed a difference in generalizability between assessments by unfamiliar (φ = 0.75) and familiar (φ = 0.27) faculty. Subanalyses showed that entrustment assessments by familiar faculty were moderately correlated to average SP checklist scores (r = 0.44, P < .001), while those by unfamiliar faculty were weakly correlated (r = 0.16, P = .13).

CONCLUSIONS

While faculty and case had a limited impact on the generalizability of entrustment assessments made outside the workplace in UME, faculty who were familiar with a student's ability had a notable impact on generalizability and potentially on the scoring validity of entrustment assessments, which warrants further study.

Factors and Interactions Influencing Direct Observation: A Literature Review Guided by Activity Theory

PhenomenonEnsuring that future physicians are competent to practice medicine is necessary for high quality patient care and safety. The shift toward competency-based education has placed renewed emphasis on direct observation via workplace-based assessments in authentic patient care contexts. Despite this interest and multiple studies focused on improving direct observation, challenges regarding the objectivity of this assessment approach remain underexplored and unresolved. Approach: We conducted a literature review of direct observation in authentic patient contexts by systematically searching databases PubMed, Embase, Web of Science, and ERIC. Included studies comprised original research conducted in the patient care context with authentic patients, either as a live encounter or a video recording of an actual encounter, which focused on factors affecting the direct observation of undergraduate medical education (UME) or graduate medical education (GME) trainees. Because the patient care context adds factors that contribute to the cognitive load of the learner and of the clinician-observer we focused our question on such contexts, which are most useful in judgments about advancement to the next level of training or practice. We excluded articles or published abstracts not conducted in the patient care context (e.g., OSCEs) or those involving simulation, allied health professionals, or non-UME/GME trainees. We also excluded studies focused on end-of-rotation evaluations and in-training evaluation reports. We extracted key data from the studies and used Activity Theory as a lens to identify factors affecting these observations and the interactions between them. Activity Theory provides a framework to understand and analyze complex human activities, the systems in which people work, and the interactions or tensions between multiple associated factors. Findings: Nineteen articles were included in the analysis; 13 involved GME learners and 6 UME learners. Of the 19, six studies were set in the operating room and four in the Emergency department. Using Activity Theory, we discovered that while numerous studies focus on rater and tool influences, very few study the impact of social elements. These are the rules that govern how the activity happens, the environment and members of the community involved in the activity and how completion of the activity is divided up among the members of the community. Insights: Viewing direct observation via workplace-based assessment through the lens of Activity Theory may enable educators to implement curricular changes to improve direct observation of assessment. Activity Theory may allow researchers to design studies to focus on the identified underexplored interactions and influences in relation to direct observation.

Using mobile technology in assessment of entrustable professional activities in undergraduate medical education

BACKGROUND

The adoption of competency-based medical education requires objective assessments of a learner's capability to carry out clinical tasks within workplace-based learning settings. This study involved an evaluation of the use of mobile technology to record entrustable professional activity assessments in an undergraduate clerkship curriculum.

APPROACH

A paper-based form was adapted to a mobile platform called eClinic Card. Students documented workplace-based assessments throughout core clerkship and preceptors confirmed accuracy via mobile phones. Assessment scores for the 2017-2018 academic year were collated and analyzed for all core rotations, and preceptors and students were surveyed regarding the mobile assessment experience.

EVALUATION

The mobile system enabled 80 students and 624 preceptors to document 6850 assessment submissions across 47 clinical sites over a 48-week core clerkship curriculum. Students' scores demonstrated progressive improvement across all entrustable professional activities with stage-appropriate levels of independence reported by end of core clerkship. Preceptors and students were satisfied with ease of use and dependability of the mobile assessment platform; however, students felt quality of formative coaching feedback could be improved.

REFLECTION

Our preliminary evaluation suggests the use of mobile technology to assess entrustable professional activity achievement across a core clerkship curriculum is a feasible and acceptable modality for workplace-based assessment. The use of mobile technology supported a programmatic assessment approach. However, meaningful coaching feedback, as well as faculty development and support, emerged as key factors influencing successful adoption and usage of entrustable professional activities within an undergraduate medical curriculum.

The contribution of simulation in training for funduscopic examination

INTRODUCTION

Simulation is emerging as an essential part of health sciences training programs as it provides safer patient care by reducing the risk of error. In the healthcare environment, simulation continues to spread in emergency specialties, but it is still underdeveloped in ophthalmology and there is a shortage of publications on this subject in Tunisia.

OBJECTIVE

To evaluate the effectiveness of procedural simulation as a teaching tool for funduscopic examination training. Methods and population studied: This was a prospective study including students who underwent procedural simulation training sessions during their ophthalmology internship. The included students were assessed at the initiation and end of each session by a pre-test and post-test. The procedure for performing the funduscopic examination was evaluated by a specific performance score.  Student satisfaction was assessed at the end of each session.

RESULTS

During the study period, four groups of 12 students were included, for a total of 48 participants spread over 4 simulation sessions. Simulation training improved post-test.  assessment scores with an overall median delta of +4.00. It also provided specific skills for performing the funduscopic examination, with an overall median specific performance score of 5.5/8 (5/8 to 7/8). The majority of students were satisfied upon completion of the simulation session.

CONCLUSION

The training of fundus examination using an ophthalmoscopic simulator can improve the skills and knowledge of ophthalmic learners. This type of training can be an innovative addition to traditional learning.

Development of a Hand Motion-based Assessment System for Endotracheal Intubation Training

Endotracheal intubation (ETI) is a procedure to manage and secure an unconscious patient's airway. It is one of the most critical skills in emergency or intensive care. Regular training and practice are required for medical providers to maintain proficiency. Currently, ETI training is assessed by human supervisors who may make inconsistent assessments. This study aims at developing an automated assessment system that analyzes ETI skills and classifies a trainee into an experienced or a novice immediately after training. To make the system more available and affordable, we investigate the feasibility of utilizing only hand motion features as determining factors of ETI proficiency. To this end, we extract 18 features from hand motion in time and frequency domains, and also 12 force features for comparison. Subsequently, feature selection algorithms are applied to identify an ideal feature set for developing classification models. Experimental results show that an artificial neural network (ANN) classifier with five hand motion features selected by a correlation-based algorithm achieves the highest accuracy of 91.17% while an ANN with five force features has only 80.06%. This study corroborates that a simple assessment system based on a small number of hand motion features can be effective in assisting ETI training.

Online Problem-Based Learning in Clinical Dental Education: Students' Self-Perception and Motivation

The physical closure of higher education institutions due to coronavirus disease 2019 (COVID-19) shed a brighter light on the need to analyze, explore, and implement strategies that allow the development of clinical skills in a distance learning situation. This cross-sectional study aims to assess dental students’ self-perception, motivation, organization, acquired clinical skills, and knowledge using the online problem-based learning method, through the application of a 41-item questionnaire to 118 senior students. Answers were subjected to descriptive and inferential statistics analysis. Further, a principal component analysis was performed, in order to examine the factor structure of the questionnaire. Results show that online problem-based learning can be considered a relevant learning tool when utilized within the specific context of clinical dental education, displaying benefits over the traditional learning strategy. Overall, dental students prefer a hybrid system over the conventional one, in a distance learning context, and assume self-responsibility for their own learning, while knowledge thoroughness is perceived as inferior. This online active learning method is successful in improving information and clinical ability (visual/spatial and auditory) advancement in the scope of dental education, with similar results to presential settings. Further studies are required to assess clinical skill development through active learning methods, in a distance learning context.

Does direct observation happen early in a new competency-based residency program?

BACKGROUND

A key component of competency-based medical education is workplace-based assessment, which includes observation (direct or indirect) of residents. Direct observation has been emphasized as an ideal form of assessment yet challenges have been identified that may limit its adoption. At present, it remains unclear how often direct and indirect observation are being used within the clinical setting. The objective of this study was to describe patterns of observation in an emergency medicine competency-based program 2 years postimplementation.

METHODS

Emergency medicine residents (n = 19) recorded the type of observation they received (direct or indirect) following workplace-based entrustable professional activity (EPA) assessments from December 15, 2019, to April 30, 2020. Assessment forms were reviewed and analyzed to describe patters of observation.

RESULTS

Assessments were collected on all 19 eligible residents (100% participation). A total of 1,070 EPA assessments were completed during the study period, of which 798 (74.6%) had the type of observation recorded. Of these recorded observations, 546 (68.4%) were directly observed and 252 (31.6%) were indirectly observed. The length of written comments contained within assessments following direct and indirect observation did not differ significantly. There was no significant association between resident gender and observation type or resident stage of training and observation type. Certain EPA assessments showed a clear preference toward either direct or indirect observation.

CONCLUSIONS

To the best of our knowledge, this study is the first to report patterns of observation in a competency-based residency program. The results suggest that direct observation can be quickly adopted as the primary means of workplace-based assessment. Indirect observation comprised a sizeable minority of observations and may be an underrecognized contributor to workplace-based assessment. The preference toward either direct or indirect observation for certain EPA assessments suggests that the entrustable professional activity itself may influence the type of observation.

Development of Neurological Emergency Simulations for Assessment: Content Evidence and Response Process

OBJECTIVE

To document two sources of validity evidence for simulation-based assessment in neurological emergencies.

BACKGROUND

A critical aspect of education is development of evaluation techniques that assess learner's performance in settings that reflect actual clinical practice. Simulation-based evaluation affords the opportunity to standardize evaluations but requires validation.

METHODS

We identified topics from the Neurocritical Care Society's Emergency Neurological Life Support (ENLS) training, cross-referenced with the American Academy of Neurology's core clerkship curriculum. We used a modified Delphi method to develop simulations for assessment in neurocritical care. We constructed checklists of action items and communication skills, merging ENLS checklists with relevant clinical guidelines. We also utilized global rating scales, rated one (novice) through five (expert) for each case. Participants included neurology sub-interns, neurology residents, neurosurgery interns, non-neurology critical care fellows, neurocritical care fellows, and neurology attending physicians.

RESULTS

Ten evaluative simulation cases were developed. To date, 64 participants have taken part in 274 evaluative simulation scenarios. The participants were very satisfied with the cases (Likert scale 1-7, not at all satisfied-very satisfied, median 7, interquartile range (IQR) 7-7), found them to be very realistic (Likert scale 1-7, not at all realistic-very realistic, median 6, IQR 6-7), and appropriately difficult (Likert scale 1-7, much too easy-much too difficult, median 4, IQR 4-5). Interrater reliability was acceptable for both checklist action items (kappa = 0.64) and global rating scales (Pearson correlation r = .70).

CONCLUSIONS

We demonstrated two sources of validity in ten simulation cases for assessment in neurological emergencies.

Role of Communication, Professionalism, and Clinical Care Skills of Postgraduate Students on Patients Recall Visits in Dental School-An Observational Study

Preservation of periodontal health after periodontal therapy is paramount for the complete elimination of periodontal diseases. In most of the Dental Schools, recall appointments are considerably low, and in particular, to the Department of Periodontics, the compliance to recall visits by the patients diagnosed with periodontal disease is still inadequate. Faculty from the Department of Periodontics framed new criteria to follow in the comprehensive clinics by the postgraduate students. The criteria include communication, intraoral examination skills, and professionalism toward dental patients. Faculty in each comprehensive dental clinic observed the clinical encounters of postgraduate students with patients and provided the feedback. The study conducted from May 2018 to April 2019, and the patients attended were 1164 of 1544. Unattended patients were 380 of 1544. Effective evaluation of the feedback provided by faculty and communication with both the postgraduate students and unattended patients resulted in further improvement in recall, that is, 151 of 380 patients. Reframed criteria, direct observation of the postgraduate students by faculty during patients interaction, assessment of feedback forms, and immediately modifying the student’s way of communication allowed maximum recall visits to the Department of Periodontics.

Significant differences in written assessments as a result of a blended learning approach used in a clinical examination course in internal medicine: a randomized controlled pilot study

Background: Taking a medical history and performing a physical examination represent basic medical skills. However, numerous national and international studies show that medical students and physicians-to-be demonstrate substantial deficiencies in the proper examination of individual organ systems. Aim: The objective of this study was to conduct a randomized controlled pilot study to see if, in the context of a bedside clinical examination course in internal medicine, an additional app-based blended-learning strategy resulted in (a) higher satisfaction, better self-assessments by students when rating their history-taking skills (b1) and their ability to perform physical examinations (b2), as well as (c) higher multiple-choice test scores at the end of the course, when compared to a traditional teaching strategy. Methods: Within the scope of a bedside course teaching the techniques of clinical examination, 26 students out of a total of 335 students enrolled in the 2012 summer semester and 2012/2013 winter semester were randomly assigned to two groups of the same size. Thirteen students were in an intervention group (IG) with pre- and post-material for studying via an app-based blended-learning tool, and another 13 students were in a control group (CG) with the usual pre- and post-material (handouts). The IG was given an app specifically created for the history-taking and physical exam course, an application program for smartphones enabling them to view course material directly on the smartphone. The CG received the same information in the form of paper-based notes. Prior to course begin, all of the students filled out a questionnaire on sociodemographic data and took a multiple-choice pretest with questions on anamnesis and physical examination. After completing the course, the students again took a multiple-choice test with questions on anamnesis and physical examination. Results: When compared to the CG, the IG showed significantly more improvement on the multiple-choice tests after taking the clinical examination course (p=0.022). This improvement on the MC tests in the IG significantly correlated with the amount of time spent using the app (Spearman's rho=0.741, p=0.004). Conclusion: When compared to conventional teaching, an app-based blended-learning approach leads to improvement in test scores, possibly as a result of more intensive preparation for and review of the clinical examination course material.

Virtual auscultation course for medical students via video chat in times of COVID-19

Introduction: Auscultation skills are among the basic techniques to be learned in medical school. Such skills are achieved through supervised examination of patients often supported by simulator-based learning. The emergence of COVID-19 has disrupted and continues to hinder hands-on on-site medical training on a global scale. Project description: An effective virtual auscultation course was established in times of contact restrictions due to COVID-19 at the Medical Faculty of the Heinrich Heine University Düsseldorf. The interactive case-based webinar was designed to improve listening techniques, description and interpretation of auscultation findings in an off-site context. Clinical cases with pre-recorded auscultation sounds and additional case-based diagnostics were presented. The course focused on common heart murmurs including aortic and mitral valve stenosis and regurgitation as well as congenital heart defects (ventricular septal defect and patent ductus arteriosus). Results: The course was well received by the students and assessed as being useful and instructive. Assessment of learning effects, such as detection of pathological findings before and after training, is ongoing as part of a subsequent trial. Conclusion: Virtual interactive learning using a sound simulation lesson with clinical case presentations via video chat can well be used as a supplement to practical auscultation training. This learning format could also play a useful role in the curriculum of medical studies once contact restrictions are revoked.

[The Maastricht Education System in Aachen - a Few Miles Away or Worlds Apart?]

INTRODUCTION

Education for residents in surgery varies not only throughout the world, but also throughout Europe. Our clinic is well connected to Maastricht University Medical in Centre in the Netherlands (European Surgical Centre Aachen Maastricht). On the other hand, there are clear differences in resident programs. In the Netherlands, a structured feedback according to the OSATS concept (Objective Structured Assessment of Technical Skills) is mandatory after every operation performed by residents. The aim of the present study was to transfer the OSATS concept from Maastricht to Aachen and to evaluate the feasibility and benefits of this concept for surgical education.

MATERIAL AND METHODS

The OSATS concept was implemented for 3 months in our clinic within a prospective clinical trial. Seven out of 10 residents that were working in our clinic at that time participated in the study (70%). Half of these were assigned to structured written feedback after every autonomously performed operation. Additionally, all participants performed structured written proper feedback according to the OSATS concept. The primary endpoint was the feasibility of the OSATS concept in our clinic; secondary endpoints were the benefits for the residents and the differences between external and self-evaluation.

RESULTS

The OSATS-concept was easily implemented in our clinic and met wide acceptance. Evaluation was performed after a mean of 70% of operations. External evaluation was regarded as more beneficial for residents than self-evaluation. Structured written evaluation according to the OSATS concept was not time-consuming (< 3 minutes) and most residents (86%) supported permanent implementation of the OSATS concept in our clinic.

CONCLUSION

The OSATS concept is a suitable approach to provide structured feedback to residents in continuous education. It can easily be implemented in resident education in Germany. Structured, written feedback by senior physicians is perceived as beneficial by residents.

Overcoming the Challenges of Direct Observation and Feedback Programs: A Qualitative Exploration of Resident and Faculty Experiences

Problem: Prior studies have reported significant negative attitudes amongst both faculty and residents toward direct observation and feedback. Numerous contributing factors have been identified, including insufficient time for direct observation and feedback, poorly understood purpose, inadequate training, disbelief in the formative intent, inauthentic resident-patient clinical interactions, undermining of resident autonomy, lack of trust between the faculty-resident dyad, and low-quality feedback information that lacks credibility. Strategies are urgently needed to overcome these challenges and more effectively engage faculty and residents in direct observation and feedback. Otherwise, the primary goals of supporting both formative and summative assessment will not be realized and the viability of competency-based medical education will be threatened. Intervention: Toward this end, recent studies have recommended numerous strategies to overcome these barriers: protected time for direct observation and feedback; ongoing faculty and resident training on goals and bidirectional, co-constructed feedback; repeated direct observations and feedback within a longitudinal resident-supervisor relationship; utilization of assessment tools with evidence for validity; and monitoring for engagement. Given the complexity of the problem, it is likely that bundling multiple strategies together will be necessary to overcome the challenges. The Direct Observation Structured Feedback Program (DOSFP) incorporated many of the recommended features, including protected time for direct observation and feedback within longitudinal faculty-resident relationships. Using a qualitative thematic approach the authors conducted semi-structured interviews, during February and March, 2019, with 10 supervisors and ten residents. Participants were asked to reflect on their experiences. Interview guide questions explored key themes from the literature on direct observation and feedback. Transcripts were anonymized. Two authors independently and iteratively coded the transcripts. Coding was theory-driven and differences were discussed until consensus was reached. The authors then explored the relationships between the codes and used a semantic approach to construct themes. Context: The DOSFP was implemented in a psychiatry continuity clinic for second and third year residents. Impact: Faculty and residents were aligned around the goals. They both perceived the DOSFP as focused on growth rather than judgment even though residents understood that the feedback had both formative and summative purposes. The DOSFP facilitated educational alliances characterized by trust and respect. With repeated practice within a longitudinal relationship, trainees dropped the performance orientation and described their interactions with patients as authentic. Residents generally perceived the feedback as credible, described feedback quality as high, and valued the two-way conversation. However, when receiving feedback with which they did not agree, residents demurred or, at most, would ask a clarifying question, but then internally discounted the feedback. Lessons Learned: Direct observation and structured feedback programs that bundle recent recommendations may overcome many of the challenges identified by previous research. Yet, residents discounted disagreeable feedback, illustrating a significant limitation and the need for other strategies that help residents reconcile conflict between external data and one's self-appraisal.

The Modern Physical Exam - A Transatlantic Perspective from the Resident Level

Issue: The physical examination has been in decline for many years and poorer skills contribute to medical errors and adverse events. Diagnostic error is also increasing with the complexity of medicine. Comparing the physical examination in Ireland and the United States with a focus on education, assessment, culture, and health systems may provide insight into the decline of the physical exam in the United States, uncover possible strategies to improve clinical skills, and limit diagnostic error. Evidence: The physical exam is a core component of both undergraduate and postgraduate medical education in Ireland. This is reflected by the time and effort invested by medical schools and medical societies in Ireland in teaching and assessing skills. This high standard of skills results in the physical exam being a key component of the diagnostic process and a gatekeeper to expensive investigations essential in a resource-limited health system such as Ireland. Use of the physical exam in the United States is hindered by the high-tech transformation of healthcare and a more litigious society. Known strategies to highlight the role of the physical exam in clinical practice include creating an evidence base to show that better physical exam skills improve outcomes, identifying accurate physical exam maneuvers, stressing the therapeutic alliance the physical exam brings to the patient encounter, and the incorporation of technology into the bedside exam. Implications: Contrasting the education and clinical use of the physical examination in the United States with Ireland allowed us to identify a number of strategies which could be used to promote the physical exam among learners in both countries. Highlighting simple and pragmatic physical exam maneuvers combined with evidence-based physical exam diagnostic data may renew confidence in the physical exam as a core diagnostic tool. Use of the hypothesis-driven approach may streamline a clinician's physical exam during a patient encounter, focusing on the key examination components and avoiding unnecessary and low yield maneuvers. The absence of assessment of physical exam skills using real patients in United States licensing exams communicates to learners that these skills are not important. However, steps to introduce a culture of assessment to drive learning are being introduced. One area Ireland could learn from the United States is incorporating more technology into the bedside exam. Enhanced physical examination skills in both countries could reduce reliance on expensive investigations and improve diagnostic accuracy.

Are we generating more assessments without added value? Surgical trainees' perceptions of and receptiveness to cross-specialty assessment

INTRODUCTION

Competency-based medical education (CBME) hinges on robust assessment. However, integrating regular workplace-based assessment within demanding and sometimes chaotic clinical environments remains challenging. Many faculty lack assessment expertise, and some programs lack the infrastructure and faculty numbers to fulfill CBME's mandate. Recognizing this, we designed and implemented an assessment innovation that trains and deploys a cadre of faculty to assess in specialties outside their own. Specifically, we explored trainees' perceptions of and receptiveness to this novel assessment approach.

METHODS

Within Western University's Surgical Foundations program, 27 PGY‑1 trainees were formatively assessed by trained non-surgeons on a basic laparoscopic surgical skill. These assessments did not impact trainees' progression. Four focus groups were conducted to gauge residents' sentiments about the experience of cross-specialty assessment. Data were then analyzed using a thematic analysis approach.

RESULTS

While a few trainees found the experience motivating, more often trainees questioned the feedback they received and the practicality of this assessment approach to advance their procedural skill acquisition. What trainees wanted were strategies for improvement, not merely an assessment of performance.

DISCUSSION

Trainees' trepidation at the idea of using outside assessors to meet increased assessment demands appeared grounded in their expectations for assessment. What trainees appeared to desire was a coach-someone who could break their performance into its critical individual components-as opposed to an assessor whose role was limited to scoring their performance. Understanding trainees' receptivity to new assessment approaches is crucial; otherwise training programs run the risk of generating more assessments without added value.

Perceptions and barriers to competency-based education in Canadian postgraduate medical education

RATIONALE

Competency-based education (CBE) is currently being implemented across Canadian postgraduate medical education programmes through Competence by Design (CBD).¹ Queen's University received permission to initiate CBE in all programmes simultaneously starting in 2017; an institutional initiative termed Competency-based medical education (CBME).² We describe our initial experiences to highlight perceptions and barriers and facilitate implementation at other centers.

METHODS

Anonymous online surveys were administered to faculty and residents transitioning to CBE (138 respondents) including (a) Queen's programme leaders (Programme Directors and CBME Leads) [n = 27], (b) Queen's residents [n = 102], and (c) Canadian neurology programme directors [n = 9] and were analysed using descriptive and inferential statistical techniques.

RESULTS

Perceptions were favourable (x = 3.55/5, SD = 0.71) and 81.6% perceived CBE enhanced training; however, perceptions were more favourable among faculty. Queen's programme leaders indicated that CBE did not improve their ability to provide negative feedback. Queen's residents did not perceive improved quality of feedback. National Canadian neurology programme directors did not perceive that their institutions had adequately prepared them. There was variability in barriers perceived across groups. Queen's programme leaders were concerned about resident initiative. Queen's residents felt that assessment selection and faculty responsiveness to feedback were barriers. Canadian neurology programme directors were concerned about access to information technology.

RECOMMENDATIONS

Our results indicate that faculty were concerned about the reluctance of residents to actively participate in CBE, while residents were hesitant to assume such a role because of lack of familiarity and perceived benefit. This discrepancy indicates attention should be devoted to (a) institutional administrative/educational supports, (b) faculty development around feedback/assessment, and (c) resident development to foster ownership of their learning and familiarity with CBE.

Beyond summative decision making: Illuminating the broader roles of competence committees

CONTEXT

Although competence committees (CCs) are most often conceptualised as group decision-making bodies, policy documents forward a more ambitious vision and outline several additional roles for CCs that support the continuous quality improvement of education programmes and promote formative feedback. However, whether these functions are encompassed in the everyday work of CCs is currently unknown.

METHODS

This constructivist grounded theory study elaborates the range of roles taken up by CCs and illuminates the processes through which these roles are actualised. Two investigators observed 27 CC meetings (>80 hours) across seven diverse postgraduate programmes at a single Canadian institution. Following each observation, a semi-structured interview was conducted with one CC member. Data collection and analysis unfolded iteratively.

RESULTS

In this study, CCs did much more than make summative decisions about progression and entrustment; they identified a variety of problems that arose at the levels of both the individual learner and the programme, and leveraged their knowledge of assessment data to develop solutions. The problem-solving work of CCs was enabled by the in-depth data review, synthesis and analysis work that occurred before scheduled meetings, outside protected academic or administrative time. Although this work often appeared invisible to those outside the committee, the insights gleaned from data review provided committee members with essential information about how their programme of assessment was unfolding in practice.

CONCLUSIONS

Competence committees may be an untapped resource that can support assessment for learning, local evaluation of competency-based medical education (CBME) implementation and continuous quality improvement for programmes of assessment. However, the ability of CCs to engage in this work is fragile and is currently sustained by the willingness of faculty members to devote their time and energy to it. The resourcing of CCs may have profound implications for translation of the theory of programmatic assessment and CBME into practice.

Clinical Skills Evaluation in Neurology, Child Neurology and Psychiatry: Program Directors' Perspective and Future Opportunities

BACKGROUND

When the American Board of Psychiatry and Neurology (ABPN) eliminated the oral segment of the board-certification examination, it began requiring in-training assessments termed Clinical Skill Evaluations (CSEs).

OBJECTIVE

This study describes the experience of residency program directors (PDs) with CSEs and identifies opportunities for improvement.

METHODS

A 23-question survey was administered electronically to neurology, child neurology, and psychiatry PDs assessing their CSE testing procedures in April 2019. Data from the ABPN preCERT® Credentialing System CSE was analyzed to corroborate the survey results.

RESULTS

A total of 439 PDs were surveyed. The overall response rate was approximately 40% with a similar response across the 3 specialties. Overall, there was a strong enthusiasm for CSEs as they captured the essence of the physician-patient relationship. Most PDs encouraged trainees to attempt CSEs early in their training though the completion time frame varied by specialty. Approximately 57% of psychiatry residencies offered formal, in-person faculty training while less than one-fourth of neurology and child neurology programs offered such a program. Most PDs are interested in a faculty development course to ensure a standardized CSE testing process at their institution.

CONCLUSIONS

This survey confirmed earlier findings that CSEs are usually implemented early in the course of residency training and that most PDs think it captures the essence of the physician-patient relationship. While few residencies offer a CSE training course, there is widespread support for a formal approach to faculty development and this offers a specific opportunity for CSE improvement in the future.

Mini-Clinical Evaluation Exercise in the Era of Milestones and Entrustable Professional Activities in Obstetrics and Gynaecology: Resume or Reform?

OBJECTIVE

The Accreditation Council for Graduate Medical Education (ACGME) milestones and the core Entrustable Professional Activities (EPAs) provide guiding frameworks and requirements for assessing residents' progress. The Mini-Clinical Evaluation Exercise (Mini-CEX) is a formative assessment tool used to provide direct observation after an ambulatory or clinical encounter. This study aimed to investigate the feasibility and reliability of the Mini-CEX in the authors' obstetrics and gynaecology (OB/GYN) residency program and its ability to measure residents' progress and competencies in the frameworks of ACGME milestones and EPAs.

METHODS

OB/GYN residents' 5-academic-year Mini-CEX performance was analyzed retrospectively to measure reliability and feasibility. Additionally, realistic evaluation was conducted to assess the usefulness of Mini-CEX in the frameworks of ACGME milestones and EPAs.

RESULTS

A total of 395 Mini-CEX evaluations for 49 OB/GYN residents were analyzed. Mini-CEX evaluation data significantly discriminated among residents' training levels (P < 0.003). Residents had an average of 8.1 evaluations per resident completed; 10% of second-year residents and 28% of third-year residents were evaluated 10 or more times per year, whereas no postgraduate year 1 or postgraduate year 4 residents achieved this number. Mini-CEX data could contribute to all 6 primary measurement domains of OB/GYN milestones and 8 of 10 EPAs required for first-year residents.

CONCLUSION

The Mini-CEX demonstrated potential for measuring residents' clinical competencies in their ACGME milestones. Faculty time commitment was the main challenge. Reform is necessary for the current feedback structure in Mini-CEX, faculty development, and operational guidelines that help residency programs match residents' clinical competency ratings with ACGME milestones and EPAs.

The paediatric developmental toolkit: Facilitating learning of child development

Background

Recent paediatric and family medicine graduates report feeling unprepared to identify and address children and youth with developmental disorders. Developmental history taking and physical examination alone limit engagement with children and youth in an interactive manner to assess development. The paediatric developmental toolkit (PDT) was developed to provide trainees with the opportunity to interact with a child in a play-based manner.

Objectives

The primary objective of this study was to determine the feasibility of PDT within clinical settings, and qualitatively explore how the PDT can be used by teachers and trainees.

Methods

Trainees and their clinical teachers participated in a qualitative study. Trainees used the PDT in clinical settings and were interviewed following their clinical encounters. Interactions between clinical teachers and trainees following the use of the PDT were also recorded. Teachers were interviewed following the trainees' case presentations and closures of clinic visits. Trainee interviews, teacher and trainee interactions, and teacher interviews were audiotaped, transcribed, and analyzed thematically.

Results

Nine trainees (six paediatric residents, two family medicine residents, and one clinical clerk medical student) and four developmental paediatricians participated in the study. Each trainee used the PDT twice in two different clinical encounters. All residents agreed the PDT enabled them to observe a child's developmental skills in a short period of time. Clinical teachers all felt the toolkit allowed trainees to more holistically consider a child's development and diagnosis.

Conclusions

As medical education shifts to a competency-based education curriculum, the PDT is an innovative tool that can be used to enhance paediatric and family medicine residents' learning of child development by enabling opportunities for interaction with children.

How students and specialists appreciate the mini-clinical evaluation exercise (mini-CEX) in Indonesian clerkships

BACKGROUND

Cultural differences might challenge the acceptance of the implementation of assessment formats that are developed in other countries. Acceptance of assessment formats is essential for its effectiveness; therefore, we explored the views of students and specialists on the practicality and impact on learning of these formats. This study was conducted to explore Indonesian students' and specialists' appreciation of the implementation of the Mini-Clinical Evaluation Exercise (Mini-CEX) in Indonesian clerkships.

METHODS

This study was conducted at the Universitas Gadjah Mada, Indonesia. Participants were 52 students and 21 specialists in neurology and 78 students and 50 specialists in internal medicine. They were asked to complete a 19-item questionnaire that covered the characteristics of the mini-CEX such as its practicality, and the impact on learning and professional development. We used a Mann-Whitney U test to analyse the data.

RESULTS

In total, 124 students (46 from neurology and 78 from internal medicine) and 38 specialists (13 from neurology and 25 from internal medicine) participated in this study. Students and specialists were positive about the practicality of the mini-CEX and the impact of this assessment format on learning and on professional development. The Mann-Whitney U test showed that there were no significant differences between students' and specialists' opinions on the mini-CEX, except for 2 items: specialists' appreciation of direct observation (mean rank = 93.16) was statistically significantly higher than students' appreciation of it (mean rank = 77.93; z = 2.065; p < 0.05), but students' appreciation of the item that students' past mini-CEX results affected their recent mini-CEX outcomes (mean rank = 85.29) was significantly higher than specialists' appreciation of it (mean rank = 69.12; z = 2140; p < 0.05).

CONCLUSION

Students and specialists were positive about the mini-CEX in Indonesian clerkships, although it was developed and validated in another culture. We found only small differences between their appreciations, which could be explained by the patterns of specialist-student interaction in Indonesian culture as large power distance and low individualism country.

Development of e-portfolio in undergraduate clinical dentistry: How trainees select and reflect on evidence

INTRODUCTION

Self-reflection is a fundamental part of health professional development, especially in the unstructured clinical stage of dentistry, medical and health profession education, since it plays important role in experiential learning. A portfolio-both conventional and digital-records various aspects of knowledge, skills and attitudes over a long period of time to help students develop their professionalism by conducting self-reflection. This study aims to explore the process of selecting and reflecting on professionalism evidence recorded in e-portfolios during undergraduate clinical dentistry training.

METHOD

This pilot study is a qualitative study with a phenomenological design. The selection of respondents was conducted using a maximum variety sampling method. Following a 6-week pilot programme, a total of six in-depth interviews and five focus group discussions were conducted with undergraduate students representing different clinical rotations to explore the process conducted by undergraduate clinical dentistry students in e-portfolio development. The study of documents was also conducted on the respondents' reflective writing from the e-portfolio back-end data to explore their self-reflection skills.

RESULT

The results of the present study highlighted two different approaches used amongst trainees in developing a reflective e-portfolio: selective and non-selective. The observed reflective e-portfolio utilisation frequency and trainees' self-reflection skills were low, with several affecting factors identified. These identified factors consisted of external factors, such as the undergraduate clinical dentistry programme curriculum, the hidden curriculum, the availability of feedback, the availability of role models, and features of the e-portfolio and internal factors, such as understanding the self-reflection concept, the availability of time and mood, cultural factors, and understanding the content reflected.

CONCLUSION

Through the process of independently selecting and recording learning activities in e-portfolios as well as repeatedly conducting self-reflection, trainees can improve their overall reflective abilities. However, producing a systematic approach to forming a reflective learning environment is necessary to aid the implementation of reflective e-portfolios, especially at the early stage of e-portfolio implementation.

Supervisors' intention to observe clinical task performance: an exploratory study using the theory of planned behaviour during postgraduate medical training

BACKGROUND

Direct observation of clinical task performance plays a pivotal role in competency-based medical education. Although formal guidelines require supervisors to engage in direct observations, research demonstrates that trainees are infrequently observed. Supervisors may not only experience practical and socio-cultural barriers to direct observations in healthcare settings, they may also question usefulness or have low perceived self-efficacy in performing direct observations. A better understanding of how these multiple factors interact to influence supervisors' intention to perform direct observations may help us to more effectively implement the aforementioned guidelines and increase the frequency of direct observations.

METHODS

We conducted an exploratory quantitative study, using the Theory of Planned Behaviour (TPB) as our theoretical framework. In applying the TPB, we transfer a psychological theory to medical education to get insight in the influence of cognitive and emotional processes on intentions to use direct observations in workplace based learning and assessment. We developed an instrument to investigate supervisors intention to perform direct observations. The relationships between the TPB measures of our questionnaire were explored by computing bivariate correlations using Pearson's R tests. Hierarchical regression analysis was performed in order to assess the impact of the respective TPB measures as predictors on the intention to perform direct observations.

RESULTS

In our study 82 GP supervisors completed our TPB questionnaire. We found that supervisors had a positive attitude towards direct observations. Our TPB model explained 45% of the variance in supervisors' intentions to perform them. Normative beliefs and past behaviour were significant determinants of this intention.

CONCLUSION

Our study suggests that supervisors use their past experiences to form intentions to perform direct observations in a careful, thoughtful manner and, in doing so, also take the preferences of the learner and other stakeholders potentially engaged in direct observations into consideration. These findings have potential implications for research into work-based assessments and the development of training interventions to foster a shared mental model on the use of direct observations.

An Event-based Approach to Measurement: Facilitating Observational Measurement in Highly Variable Clinical Settings

BACKGROUND

Translational research in medical education requires the ability to rigorously measure learner performance in actual clinical settings; however, current measurement systems cannot accommodate the variability inherent in many patient care  environments. This is especially problematic in emergency medicine, where patients represent a wide spectrum of severity for a single clinical presentation. Our objective is to describe and implement EBAM, an event-based approach to measurement that can be applied to actual emergency medicine clinical events.

METHODS

We used a four-step event-based approach to create an emergency department trauma resuscitation patient care measure. We applied the measure to a database of 360 actual trauma resuscitations recorded in a Level I trauma center using trained raters. A subset (n = 50) of videos was independently rated in duplicate to determine inter-rater reliability. Descriptive analyses were performed to describe characteristics of resuscitation events and Cohen's kappa was used to calculate reliability.

RESULTS

The methodology created a metric containing both universal items that are applied to all trauma resuscitation events and conditional items that only apply in certain situations. For clinical trauma events, injury severity scores ranged from 1 to 75 with a mean (±SD) of 21 (±15) and included both blunt (254/360; 74%) and penetrating (86/360; 25%) traumatic injuries, demonstrating the diverse nature of the clinical encounters. The mean (±SD) Cohen's kappa for patient care items was 0.7 (±0.3).

CONCLUSION

We present an event-based approach to performance assessment that may address a major gap in translational education research. Our work centered on assessment of patient care behaviors during trauma resuscitation. More work is needed to evaluate this approach across a diverse array of clinical events.

Assessing physical examination skills using direct observation and volunteer patients

Background

Feedback based on direct observation of the physical examination (PE) is associated with enhanced educational outcomes, yet attending physicians do not frequently observe graduate trainees performing the PE.

Methods

We recruited volunteer patients (VPs), each with an abnormality of the cardiovascular, respiratory, or neurological system. Interns examined each VP, then presented a differential diagnosis and management plan to two clinician educators, who, themselves, had independently examined the VPs. The clinician educators assessed interns along five domains and provided post-examination feedback and teaching. We collected data on intern performance, faculty inter-rater reliability, correlation with a simulation-based measure of clinical skill, and resident and VP perceptions of the assessment.

Results

A total of 72 PGY-1 interns from a large academic training program participated. Performance on the cardiovascular and respiratory system was superior to performance on the neurologic exam. There was no correlation between results of an online test and directly observed cardiovascular skill. Interns preferred feedback from the direct observation sessions. VPs and faculty also rated the experience highly. Inter-rater reliability was good for the respiratory exam, but poor for the cardiovascular and neurologic exams.

Conclusions

Direct observation of trainees provides evidence about PE skill that cannot be obtained via simulation. Clinician educators’ ability to provide reliable PE assessment may depend on the portion of the PE being assessed. Our experience highlights the need for ongoing training of clinician educators in direct observation, standard setting, and assessment protocols. This assessment can inform summative or formative assessments of physical exam skill in graduate medical education.

The genealogy of teaching clinical reasoning and diagnostic skill: the GEL Study

The genealogy of graduate medical education in America begins at the bedside. However, today's graduate medical trainees work in a training environment that is vastly different from medical training a century ago. The goal of the Graduate Medical Education Laboratory (GEL) Study, supported by the American Medical Association's (AMA) "Reimagining Residency" initiative, is to determine the factors in the training environment that most contribute to resident well-being and developing diagnostic skills. We believe that increasing time at the bedside will improve clinical skill, increase professional fulfillment, and reduce workplace burnout. Our graduate medical education laboratory will test these ideas to understand which interventions can be shared among all training programs. Through the GEL Study, we aim to ensure resident readiness for practice as we understand, then optimize, the learning environment for trainees and staff.

Perceptions of scheduled vs. unscheduled directly observed visits in an internal medicine residency outpatient clinic

INTRODUCTION

Learners may subconsciously change their behavior once they know they are being observed, and this Hawthorne effect should be considered when designing assessments of learner behavior. While there is a growing body of literature to suggest direct observation is the ideal standard for formative assessment, the best method to directly observe learners is unknown. We explored scheduled and unscheduled methods of direct observation among internal medicine residents in the outpatient continuity clinic to advance the understanding of both observation methods.

METHODS

We conducted a thematic analysis of faculty and internal medicine residents in an outpatient clinic setting. A semi-structured interview guide for focus group sessions was created. Focus groups were used to explore the internal medicine resident and core teaching faculty perceptions of the scheduled and unscheduled direct observation methods in the outpatient clinc. An experienced qualitative research interviewer external to the internal medicine residency was moderating the sessions. Eight peer focus groups were held. Abstraction of themes from focus group transcripts identified resident and faculty perceptions of the different observation methods.

RESULTS

Focus groups had 14 resident participants and 14 faculty participants. Unscheduled observations were felt to be more authentic than scheduled observations since residents perceived their behavior to be unmodified. Unscheduled observations allowed for increased numbers of observations per resident, which permitted more frequent formative assessments. Residents and faculty preferred remote video observation compared to in-room observation. Participants found direct observation a useful learning tool for high-yield, specific feedback.

CONCLUSIONS

Unscheduled remote direct observation captures authentic clinical encounters while minimizing learner behavior modification. An unscheduled observation approach results in more frequent formative assessment and therefore in more instances of valuable feedback compared to scheduled observations. These findings can help guide the best practice approaches to direct clinical observation in order to enhance residents learning and experience.