Exploring the divergence between self-assessment and self-monitoring

Self-Regulated Learning and Learning Outcomes in Undergraduate and Graduate Medical Education: A Meta-Analysis

Self-regulated learning (SRL) plays a pivotal role in medical education. There is a pressing need for a meta-analytical review to comprehensively evaluate the effect sizes of SRL strategies across diverse learning outcomes and levels of medical trainees. A meta-analysis was executed by searching five databases and resulted in 61 studies that met our inclusion criteria. A three-level meta-analysis was performed to examine the association between SRL strategies and various levels of learning outcomes (i.e., affective, cognitive, and behavioral learning outcomes). Moderator analyses were conducted using meta-regression, considering factors such as types of learning outcomes, training levels, SRL subscales, and quality of the studies. The analysis yielded an average effect size of .212, using Pearson's correlation coefficient, demonstrating a positive and significant association between SRL strategies and overall learning outcomes. Although our moderator analyses indicated that SRL subscales and study quality did not significantly influence the relationship between SRL strategies and learning outcomes, SRL strategies had a more pronounced effect on affective outcomes than on test scores, behavioral outcomes, and mental health outcomes. In addition, the association between SRL strategies and learning outcomes were significantly higher among the clerkship phase of undergraduate medical education than the pre-clerkship phase.

The prospect of a mentor-protégé model in dental hygiene as a supplementary approach to traditional clinical education

PURPOSE

This study compared the person-centered approach alone and incorporated the mentor-protégé model to assess differences in self-efficacy, clinical skill acquisition, and anxiety/stress levels.

METHODS

This quasi-experimental study recruited a convenience sample of dental hygiene students at New York University during the 2022-2023 academic year. The dental hygiene students who were taught through traditional methods only (control group) and those who experienced the mentor-protégé approach (intervention group) were compared. Data was collected from participants through anonymous surveys at the beginning and conclusion of the academic year. Additional information was gathered from the comparison group through a focus group.

RESULTS

Participants from both groups responded favorably in measurements of self-efficacy. However, the comparison group had a significant increase in their self-rated ability to work independently (p = 0.008989). Changes in perceived clinical skill acquisition were not statistically significant, except for improvement in patient communication for the comparison group (p = 0.0197). The control group's anxiety/stress decreased by 12.5% (p = 0.0755) and the comparison group's anxiety/stress levels reduced by 28.3% (p = 0.0646) in the general clinics. The clinical course failing rate for the control group in fall 2022 semester was 9%, and the intervention group at 0%; no clinical course failures occurred in either group for Spring 2023 semester. There was a 100% passing rate achieved on the clinical board exam by both groups.

CONCLUSIONS

Implementing the mentor-protégé model in dental hygiene education enhances students' clinical learning experiences and aids in practice readiness.

Health literacy of patients using continuous glucose monitoring

BACKGROUND

Low health literacy and numeracy are associated with poor health outcomes and lower self-efficacy. Continuous glucose monitors (CGMs) can improve diabetes management, but their benefits may be limited by health literacy levels.

OBJECTIVES

Our objective was to characterize health literacy levels of ambulatory care patients using CGMs to manage their diabetes in 1 urban health system. Secondary aims were to identify specific knowledge deficits related to CGM education and determine predictors of self-rated comfort with and understanding of CGM use.

METHODS

Participants with type 1 or type 2 diabetes using CGMs were identified using electronic medical records. Participants completed a telephone survey, including the Health Literacy Scale/Subjective Numeracy Scale (HLS/SNS) and an investigator-developed survey assessing CGM comfort and understanding. Descriptive statistics were reported for demographic information. The associations between patient characteristics and survey responses were evaluated using the chi-square test, Fisher's exact test, or Wilcoxon rank-sum test.

RESULTS

Eighty-two participants completed the surveys. The median HLS/SNS score for study participants was 80 (Interquartile range 71-89). Associations were found between HLS/SNS scores and education level, reported income, and private insurance coverage. Participants with higher HLS/SNS scores reported higher levels of CGM understanding and comfort. Fifty-one percent of participants (n = 42) reported no or inadequate training prior to CGM initiation. Better A1C results (<8%) were associated with higher self-rated responses in the investigator-developed survey.

CONCLUSION

CGMs should not be withheld from individuals with low health literacy. Incorporating baseline health literacy assessment and offering literacy sensitive training will help optimize the benefits derived from this technology.

How does training format and clinical education model impact fidelity and confidence in a speech-language pathology rotation?

Clinical education rotations typically involve an initial training phase followed by supervised clinical practice. However, little research has explored the separate contributions of each component to the development of student confidence and treatment fidelity. The dual purpose of this study was to compare the impact of clinical training format (synchronous vs. asynchronous) and education model (traditional vs. collaborative) on student confidence and treatment fidelity. Thirty-six speech-language pathology graduate students completed this two-phase study during a one-term clinical rotation. Phase 1 investigated the impact of training condition (synchronous, asynchronous guided, asynchronous unguided) on student confidence and treatment fidelity. Phase 2 explored the impact of education model (traditional vs. collaborative) on student confidence and treatment fidelity. Treatment fidelity was measured at the conclusion of Phases 1 and 2. Students rated their confidence at six-time points throughout the study. Our results indicate that training condition did not differentially impact student confidence or treatment fidelity; however, education model did: students in the collaborative education model reported increased confidence compared to students in the traditional education model. Students in the collaborative education model also trended towards having higher treatment fidelity than students in the traditional education model. These results demonstrate that pre-clinical trainings can be effective in several different formats provided they cover the discrete skills needed for the clinical rotation. While preliminary, our results further suggest that students may benefit from working with peers during their clinical rotations.

The effects of gaze-display feedback on medical students' self-monitoring and learning in radiology

Self-monitoring is essential for effectively regulating learning, but difficult in visual diagnostic tasks such as radiograph interpretation. Eye-tracking technology can visualize viewing behavior in gaze displays, thereby providing information about visual search and decision-making. We hypothesized that individually adaptive gaze-display feedback improves posttest performance and self-monitoring of medical students who learn to detect nodules in radiographs. We investigated the effects of: (1) Search displays, showing which part of the image was searched by the participant; and (2) Decision displays, showing which parts of the image received prolonged attention in 78 medical students. After a pretest and instruction, participants practiced identifying nodules in 16 cases under search-display, decision-display, or no feedback conditions (n = 26 per condition). A 10-case posttest, without feedback, was administered to assess learning outcomes. After each case, participants provided self-monitoring and confidence judgments. Afterward, participants reported on self-efficacy, perceived competence, feedback use, and perceived usefulness of the feedback. Bayesian analyses showed no benefits of gaze displays for post-test performance, monitoring accuracy (absolute difference between participants' estimated and their actual test performance), completeness of viewing behavior, self-efficacy, and perceived competence. Participants receiving search-displays reported greater feedback utilization than participants receiving decision-displays, and also found the feedback more useful when the gaze data displayed was precise and accurate. As the completeness of search was not related to posttest performance, search displays might not have been sufficiently informative to improve self-monitoring. Information from decision displays was rarely used to inform self-monitoring. Further research should address if and when gaze displays can support learning.

Developing a scale to explore self-regulatory approaches to assessment and feedback with academics in higher education

Introduction

Students need to acquire high level self-regulatory skills if they are to be successful within higher education, and academics need support in facilitating this. In this article we explore how the current research gap between knowledge of self-regulatory assessment and feedback (SRAF) practices, and academics' professional training in it can be bridged.

Methods

SRAF tools were used with academics to explore their understandings of and training needs in SRAF; central to this work was the development of a SRAF scale. We consider the value of such tools in supporting academics' professional development needs in SRAF. The reliability and validity of the SRAF scale was tested using exploratory factor analyses (EFA).

Results

Iterative EFA resulted in a 17 item support required SRAF scale (SR). Two underpinning factors: Creating the Conditions for SRAF, and Supporting Students' SRAF Skills Development were identified. The reliability of the instrument supported its primary use as a tool to facilitate academics' professional development in fostering students' self-regulatory skills.

Discussion

Our findings highlight the importance of supporting academics in developing strategies to maximize students' metacognitive skills and motivation in assessment and feedback, contingent on effective assessment design. Such professional development needs to be mindful of individual and contextual factors impacting academics' access to, and confidence and competence in, using SRAF in practice. This research is important in highlighting potential disconnects between where academics' focus their attention in assessment, and what is known to have most impact on student learning success. The SRAF tools have considerable potential in supporting translation of theory into practice as part of sustained professional development for academics in higher education.

Impact of performance and information feedback on medical interns' confidence-accuracy calibration

Diagnostic errors are a major, largely preventable, patient safety concern. Error interventions cannot feasibly be implemented for every patient that is seen. To identify cases at high risk of error, clinicians should have a good calibration between their perceived and actual accuracy. This experiment studied the impact of feedback on medical interns' calibration and diagnostic process. In a two-phase experiment, 125 medical interns from Dutch University Medical Centers were randomized to receive no feedback (control), feedback on their accuracy (performance feedback), or feedback with additional information on why a certain diagnosis was correct (information feedback) on 20 chest X-rays they diagnosed in a feedback phase. A test phase immediately followed this phase and had all interns diagnose an additional 10 X-rays without feedback. Outcome measures were confidence-accuracy calibration, diagnostic accuracy, confidence, and time to diagnose. Both feedback types improved overall confidence-accuracy calibration (R2No Feedback = 0.05, R2Performance Feedback = 0.12, R2Information Feedback = 0.19), in line with the individual improvements in diagnostic accuracy and confidence. We also report secondary analyses to examine how case difficulty affected calibration. Time to diagnose did not differ between conditions. Feedback improved interns' calibration. However, it is unclear whether this improvement reflects better confidence estimates or an improvement in accuracy. Future research should examine more experienced participants and non-visual specialties. Our results suggest that feedback is an effective intervention that could be beneficial as a tool to improve calibration, especially in cases that are not too difficult for learners.

Understanding the interplay of self-regulated learning strategies in medical education: A cross-sectional structural equation modeling study

Background and Aims

Although previous studies have investigated self-regulated learning strategies, a holistic study has not been conducted on note-taking, environmental structuring, self-evaluation, and self-consequence strategies among medical students. The current study focused on the relationships between these four self-regulated learning strategies in a medical context.

Methods

A conceptual model of the four strategies was developed, supported by the relevant literature. This cross-sectional study used an electronic structured questionnaire. The sample consisted of 557 medical undergraduates. The data were analysed using confirmatory factor analysis and structural equation modeling to investigate the hypothetical model.

Results

The conceptual model fits the data well. All relationships between the strategies were significantly positive, except for a regression between environmental structuring and self-evaluation, which was nonsignificant. Self-evaluation strategies represented the highest mean, whereas self-consequence strategies represented the lowest.

Conclusion

The findings of this study have implications for medical students, their teachers, and their universities regarding ways to enhance learners' regulated learning strategies. Future research should be conducted to develop additional statistical and comparison models for use in experimental studies and longitudinal investigations.

Cognitive perspectives on maintaining physicians' medical expertise: III. Strengths and weaknesses of self-assessment

Is self-assessment enough to keep physicians' cognitive skills-such as diagnosis, treatment, basic biological knowledge, and communicative skills-current? We review the cognitive strengths and weaknesses of self-assessment in the context of maintaining medical expertise. Cognitive science supports the importance of accurately self-assessing one's own skills and abilities, and we review several ways such accuracy can be quantified. However, our review also indicates a broad challenge in self-assessment is that individuals do not have direct access to the strength or quality of their knowledge and instead must infer this from heuristic strategies. These heuristics are reasonably accurate in many circumstances, but they also suffer from systematic biases. For example, information that feels easy to process in the moment can lead individuals to overconfidence in their ability to remember it in the future. Another notable phenomenon is the Dunning-Kruger effect: the poorest performers in a domain are also the least accurate in self-assessment. Further, explicit instruction is not always sufficient to remove these biases. We discuss what these findings imply about when physicians' self-assessment can be useful and when it may be valuable to supplement with outside sources.

Cognitive perspectives on maintaining physicians' medical expertise: I. Reimagining Maintenance of Certification to promote lifelong learning

Until recently, physicians in the USA who were board-certified in a specialty needed to take a summative test every 6-10 years. However, the 24 Member Boards of the American Board of Medical Specialties are in the process of switching toward much more frequent assessments, which we refer to as longitudinal assessment. The goal of longitudinal assessments is to provide formative feedback to physicians to help them learn content they do not know as well as serve an evaluation for board certification. We present five articles collectively covering the science behind this change, the likely outcomes, and some open questions. This initial article introduces the context behind this change. This article also discusses various forms of lifelong learning opportunities that can help physicians stay current, including longitudinal assessment, and the pros and cons of each.

When theory beats practice: the implementation of competency-based education at healthcare workplaces : Focus group interviews with students, mentors, and educators of six healthcare disciplines

BACKGROUND

Work-integrated learning constitutes a large part of current healthcare education. During the last decades, a competency-based educational (CBE) approach has been introduced to reduce the theory-practice gap and to promote continuous competency development. Different frameworks and models have been developed to support CBE implementation in practice. Although CBE is now well-established, implementation at healthcare workplaces remains complex and controversial. This study aims to explore how students, mentors, and educators from different healthcare disciplines perceive the implementation of CBE at the workplace. The six-step model of Embo et al. (2015) was used as a base: (1) competency selection, (2) formulating learning goals, (3) self-monitoring performance, (4) self-assessing competency development, (5) summative assessment of individual competencies, and (6) summative assessment of global professional competence.

METHODS

Three semi-structured focus group interviews were conducted with (1) five students, (2) five mentors, and (3) five educators. We recruited participants from six different educational programs: audiology, midwifery, nursing (associate degree and bachelor), occupational therapy, or speech therapy. We used thematic analysis combining an inductive and deductive approach.

RESULTS

An overview of the predefined competencies was hard to find which complicated CBE implementation and resulted in a lack of consistency between the steps; e.g., the link between the selection of relevant competencies (step 1) and the formulation of learning goals based on these selected competencies (step 2) was absent. Furthermore, the analysis of the data helped identifying seven barriers for CBE implementation: (1) a gap between the educational program and the workplace, (2) a lacking overview of predefined competencies, (3) a major focus on technical competencies at the expense of generic competencies, (4) weak formulation of the learning goals, (5) obstacles related to reflection, (6) low feedback quality, and (7) perceived subjectivity of the assessment approach.

CONCLUSION

The present barriers to CBE implementation lead to a fragmentation of current work-integrated learning. In this way, theory beats practice when it comes to CBE implementation as the theory of CBE is not effectively implemented. However, the identification of these barriers might help to find solutions to optimize CBE implementation. Future research seems critical to optimize CBE so that theory can meet practice and the opportunities of CBE optimize healthcare education.

The development and validation of the assessment engagement scale

Introduction

The quality of student engagement in assessment within higher education affects learning outcomes. However, variations in conceptions of what quality in engagement looks like impacts assessment design and the way that students and lecturers engage with each other in the assessment process. Given that assessment is an important driver of student engagement in higher education, it is surprising that no specific measures to support understanding of this measure exist. To address this significant gap, we outline the evolution of an assessment engagement scale derived from a research-informed conceptual framework utilizing best practice in assessment and feedback.

Methods

We consider the validity and utility of the assessment engagement scale in supporting students' understanding of assessment and their role within it using exploratory and confirmatory factor analyses.

Results

The resultant nine-item assessment engagement scale's underpinning two factors included: (i) Understanding of the Assessment Context (UAC) including one's role within it, and confidence in navigating assessment requirement, and (ii) Realising Engagement Opportunities (REO) (i.e., willingness to engage and ability to utilise the assessment context effectively to support one's understanding). Construct, criterion, and convergent validity of the scale were established.

Discussion

The AES is a powerful tool in promoting dialogue between lecturers and students about what high quality engagement in assessment looks like, and the respective roles of all parties in realising this. Implications for assessment practices are discussed along with the potential of the scale as a predictive and developmental tool to support enhancements in assessment design and student learning outcomes in higher education.

Differences in Physician Performance and Self-rated Confidence on High- and Low-Stakes Knowledge Assessments in Board Certification

INTRODUCTION

Evidence links assessment to optimal learning, affirming that physicians are more likely to study, learn, and practice skills when some form of consequence ("stakes") may result from an assessment. We lack evidence, however, on how physicians' confidence in their knowledge relates to performance on assessments, and whether this varies based on the stakes of the assessment.

METHODS

Our retrospective repeated-measures design compared differences in patterns of physician answer accuracy and answer confidence among physicians participating in both a high-stakes and a low-stakes longitudinal assessment of the American Board of Family Medicine.

RESULTS

After 1 and 2 years, participants were more often correct but less confident in their accuracy on a higher-stakes longitudinal knowledge assessment compared with a lower-stakes assessment. There were no differences in question difficulty between the two platforms. Variation existed between platforms in time spent answering questions, use of resources to answer questions, and perceived question relevance to practice.

DISCUSSION

This novel study of physician certification suggests that the accuracy of physician performance increases with higher stakes, even as self-reported confidence in their knowledge declines. It suggests that physicians may be more engaged in higher-stakes compared with lower-stakes assessments. With medical knowledge growing exponentially, these analyses provide an example of the complementary roles of higher- and lower-stakes knowledge assessment in supporting physician learning during continuing specialty board certification.

Development of eConsult reflective learning tools for healthcare providers: a pragmatic mixed methods approach

BACKGROUND

Electronic consultation (eConsult) programs are crucial components of modern healthcare that facilitate communication between primary care providers (PCPs) and specialists. eConsults between PCPs and specialists. They also provide a unique opportunity to use real-world patient scenarios for reflective learning as part of professional development. However, tools that guide and document learning from eConsults are limited. The purpose of this study was to develop and pilot two eConsult reflective learning tools (RLTs), one for PCPs and one for specialists, for those participating in eConsults.

METHODS

We performed a four-phase pragmatic mixed methods study recruiting PCPs and specialists from two public health systems located in two countries: eConsult BASE in Canada and San Francisco Health Network eConsult in the United States. In phase 1, subject matter experts developed preliminary RLTs for PCPs and specialists. During phase 2, a Delphi survey among 20 PCPs and 16 specialists led to consensus on items for each RLT. In phase 3, we conducted cognitive interviews with three PCPs and five specialists as they applied the RLTs on previously completed consults. In phase 4, we piloted the RLTs with eConsult users.

RESULTS

The RLTs were perceived to elicit critical reflection among participants regarding their knowledge and practice habits and could be used for quality improvement and continuing professional development.

CONCLUSION

PCPs and specialists alike perceived that eConsult systems provided opportunities for self-directed learning wherein they were motivated to investigate topics further through the course of eConsult exchanges. We recommend the RLTs be subject to further evaluation through implementation studies at other sites.

Analysis of Portuguese Physiotherapists' Self-Knowledge on Temporomandibular Disorders

BACKGROUND

Physiotherapy is one of the most referenced and effective conservative strategies for treating patients with temporomandibular disorders (TMD). This study aimed to characterize and analyze the self-knowledge of TMD of Portuguese physiotherapists.

METHODS

an online questionnaire was carried out, and the data collected were descriptively analyzed.

RESULTS

A total of 338 physiotherapists participated, of which only 142 treated patients with TMD. Seventy-six percent of the physiotherapists reported that they had not received training in the TMD area during the physiotherapy degree course. Only 11% of the physiotherapists reported that treating patients with TMD adequately identified all symptoms of TMD.

CONCLUSIONS

the present study showed that it is necessary to integrate TMD-related content into the basic training of physiotherapists and promote an increase in evidence-based training.

Informed self-assessment during the transition to medical school: a longitudinal narrative study

OBJECTIVES

To explore how medical students' narratives of informed self-assessment (ISA) change during their first 18 months of study.

DESIGN

This longitudinal study used student narratives drawn from qualitative interviews and written reflections during the transition to medical school, to examine changes in ISA. Our analysis was informed by Situated Cognition Theory which recognises the impact and interplay of personal and environmental factors in cognition.

SETTING

To study medicine, first year students need to adapt their self-regulated learning in the context of a new peer group, study demands and educational culture. During this adaptation, students need to seek and interpret available cues to inform their self-assessment.

PARTICIPANTS

Longitudinal data were collected at five time points over 18 months from a diverse sample of seven first year medical students in an undergraduate medical programme, including 13.5 hours of interviews and 12 written reflections.

RESULTS

Before and after starting medical school, the participants' self-assessments were informed by environmental influences (exam results and comparison with peers), and personal influences (fear of failure and anxiety about not belonging). Early uncertainty meant self-assessments were overestimated and underestimated.By the end of first year, an enhanced sense of belonging coincided with less fear of failure, less emphasis on exam performance and reduced competition with peers. Self-assessments became increasingly informed by evidence of clinical skills and knowledge gained related to future professional competence.

CONCLUSION

Influences on medical students' ISAs change during the transition to studying medicine. A greater sense of belonging, and evidence of progress towards clinical competence became more important to self-assessment than comparison with peers and exam performance. Our findings reinforce the importance of formative assessments, opportunities to study and socialise with peers and early clinical experiences during first year. These experiences enhance ISA skills during the transition to medical school.

Perceived regulatory compliance in micro-firms: The case of English accommodation providers

Business regulation is only successful when firms comply. Compliance is determined by several factors, including knowledge-of/attitude-towards regulation, which we suggest consists of both burden and value. Existing literature generally ignores regulatory value and focuses on larger firms in “high-risk” industries. We explore Perceived Own-Firm and Perceived Competitor Compliance in English accommodation micro-firms. Perceived Own-Firm Compliance exceeds Perceived-Knowledge of regulation, and has the strongest (positive) association with Perceived-Value. Perceived Competitor Compliance is thought to be very low and is (negatively) associated with Perceived-Value and Perceived-Burden. Furthermore, the factors associated with greater Perceived Own-Firm Compliance are associated with lower Perceived Competitor Compliance.

Self-assessment: With all its limitations, why are we still measuring and teaching it? Lessons from a scoping review

INTRODUCTION

Self-Assessment (SA) is often assumed to be essential for learning, however, this assumption has been extensively rebutted. Research shows SA has significant limitations, including its lack of correlation with competence.

METHOD

We undertook a scoping review of SA in medical education (2011-2021) and surprisingly discovered substantial research where SA was erroneously assumed to be a valid measure of successful learning, or a skill needing to be taught. Although the initial intent of our scoping review was to explore where SA is being effectively used to advance lifelong learning, we paused to explore the extent of the problem of its misuse and resulting in wasted research, by examining excluded studies.

RESULTS

From 1151 articles, we identified 207 which ignored the documented limitations of SA. Thirty-nine studies explored SA of learning. This research has limited utility: increasing the accuracy of SA does not improve performance or lifelong learning. One hundred and sixty-eight used SA as an outcome measure to assess a program or intervention, including 63 where self-assessed improvement in knowledge/skills was the sole measure. SA of self-confidence was measured in 62 studies. When confidence was compared with an objective measure of performance, both invariably increased, but confidence did not always align with competence when measured.

DISCUSSION

Many researchers mistakenly assume the ability to accurately self-assess is essential for learning, so focus on teaching self-calibration rather than evidence-based methods of advancing learning. Other researchers incorrectly suppose that self-reported improvements in knowledge/skills provide evidence of the efficacy of a program/intervention. This is particularly troubling with regards to novices, who may believe that because they self-assess to have improved/gained confidence, they are now competent.

CONCLUSION

Our findings highlight the significant volume of research being done where SA is misunderstood and/or misused as a measurement. We posit reasons that such research continues to take place and suggest solutions moving forward.

On embedding assessments of self-regulated learning into licensure activities in the health professions: a call to action

How well have healthcare professionals and trainees been prepared for the inevitable demands for new learning that will arise in their future? Given the rapidity with which 'core healthcare knowledge' changes, medical educators have a responsibility to audit whether trainees have developed the capacity to effectively self-regulate their learning. Trainees who engage in effective self-regulated learning (SRL) skillfully monitor and control their cognition, motivation, behaviour, and environment to adaptively meet demands for new learning. However, medical curricula rarely assess trainees' capacity to engage in these strategic processes. In this position paper, we argue for a paradigm shift toward assessing SRL more deliberately in undergraduate and postgraduate programs, as well as in associated licensing activities. Specifically, we explore evidence supporting an innovative blend of principles from the science on SRL, and on preparation for future learning (PFL) assessments. We propose recommendations for how program designers, curriculum developers, and assessment leads in undergraduate and postgraduate training programs, and in licensing bodies can work together to develop integrated assessments that measure how and how well trainees engage in SRL. Claims about lifelong learning in health professions education have gone unmatched by responsive curricular changes for far too long. Further neglecting these important competencies represents a disservice to medical trainees and a potential risk to the future patients they will care for.

Discrepancies Between Preceptor and Resident Performance Assessment: Using an Electronic Formative Assessment Tool to Improve Residents' Self-Assessment Skills

PROBLEM

Accurate self-assessment is a critical skill for residents to develop to become safe, adaptive clinicians upon graduation. Physicians need to be able to identify and fill in knowledge and skill gaps to deal with the rapid expansion of medical knowledge and unpredicted novel emerging medical issues. Residency training to date has not consistently focused on building these overarching skills, nor have the burgeoning assessment data that competency-based medical education (CBME) affords been used beyond their initial intent to inform summative assessment decisions. Both are important missed opportunities.

APPROACH

The Queen's University Family Medicine Program adopted CBME in 2010. In 2011, it added the capacity for residents to electronically self-assess their daily performance, with preceptors reviewing and modifying as needed before submitting. In 2018, it designed software to report discordance between residents' self-assessment and preceptors' assessment of performance.

OUTCOMES

From 2011-2019, 56,585 field notes were submitted, 11,429 by residents, with 28% of those (3,200/11,429) showing discordance between residents' and preceptors' performance assessments. When discordant, residents assessed their performance as less competent (undercalled) than their preceptor did 73% of the time (2,336/3,200 field notes). For the 864 field notes (27% of 3,200 discordant notes) where residents rated their performance higher than their preceptor did (overcalled, for 162/1,120 [14%] residents), 6 residents overcalled performance to a dangerous extent (2 or 3 levels of supervision higher than what their supervisors assessed them at) and 26 repeatedly (greater than 5 times) overcalled their level of performance by 1 supervisory level.

NEXT STEPS

Inaccurate self-assessment (both overcalling and undercalling performance) has negative consequences. Awareness is a first step in addressing this. Discrepancy reports will be used during regular academic reviews with residents to discuss the nature, degree, and frequency of discrepancies, with the intent of fostering improved self-assessment of performance.

Bridging the gap between self-assessment and faculty assessment of clinical performance in restorative dentistry: A prospective pilot study

Purpose/Objectives

The current study was undertaken to investigate whether self‐assessment of clinical skills of undergraduate dental students could be bridged with faculty assessment by deliberate training over an extended period. A secondary aim was to explore students' perception of self‐assessment and its effect on their learning and motivation.

Material and Methods

A prospective pilot study was conducted at the Department of Restorative Dentistry at Damascus University. Sixteen students participated in the study, ranging in age between 22 and 23 years. A modified Direct Observation of Procedural Skills form with a grading rubric was used to register and guide students' self‐assessment; both were pretested on four students before the study. In total, four clinical encounters were completed by each student. Students were trained on how to conduct proper self‐assessment before and after each clinical encounter. A postcourse questionnaire was used to investigate students' perception of self‐assessment.

Results

Bias in self‐assessment decreased consistently after each encounter, and the difference in bias between the first (bias = 0.77) and the last encounter (bias = 0.21) was significant with a medium effect size (p = .022, d = 0.64). The percentage of disconfirming performance dimensions decreased from 39.7% to 26.9%. Students' ability to exactly pinpoint strengths improved consistently and significantly. However, their ability to pinpoint areas of improvement was volatile and showed no significant difference. Bland–Altman graph plots showed higher levels of agreement between self‐assessment and faculty assessment. Moreover, students' perception of self‐assessment was very positive overall.

Conclusions

These findings suggest the possibility that the gap between self‐assessment and faculty assessment could be bridged through deliberate training. Future longitudinal research using a larger sample size is still required to further explore whether self‐assessment can be actively bridged with faculty assessment by deliberate training.

Beyond competence: Towards a more holistic perspective in medical education

The authors draw connections between four State of the Science articles that encourage diversity in our field while worrying about heterogeneity of terminology impeding theory integration and practical impact.

Confidence-competence alignment and the role of self-confidence in medical education: A conceptual review

CONTEXT

There have been significant advances in competency-based medical education (CBME) within health professions education. While most of the efforts have focused on competency, less attention has been paid to the role of confidence as a factor in preparing for practice. This paper seeks to address this deficit by exploring the role of confidence and the calibration of confidence with regard to competence.

METHODS

This paper presents a conceptual review of confidence and the calibration of confidence in different medical education contexts. Building from an initial literature review, the authors engaged in iterative discussions exploring divergent and convergent perspectives, which were then supplemented with targeted literature reviews. Finally, a stakeholder consultation was conducted to situate and validate the provisional findings.

RESULTS

A series of axioms were developed to guide perceptions and responses to different states of confidence in health professionals: (a) confidence can shape how we act and is optimised when it closely corresponds to reality; (b) self-confidence is task-specific, but also inextricably influenced by the individual self-conceptualisation, the surrounding system and society; (c) confidence is shaped by many external factors and the context of the situation; (d) confidence must be considered in conjunction with competence and (e) the confidence-competence ratio (CCR) changes over time. It is important to track learners' CCRs and work with them to maintain balance.

CONCLUSION

Confidence is expressed in different ways and is shaped by a variety of modifiers. While CBME primarily focuses on competency, proportional confidence is an integral component in ensuring safe and professional practice. As such, it is important to consider both confidence and competence, as well as their relationship in CBME. The CCR can serve as a key construct in developing mindful and capable health professionals. Future research should evaluate strategies for assessing CCR, identify best practices for teaching confidence and guiding self-calibration of CCR and explore the role of CCR in continuing professional development for individuals and teams.

Do different response formats affect how test takers approach a clinical reasoning task? An experimental study on antecedents of diagnostic accuracy using a constructed response and a selected response format

The use of response formats in assessments of medical knowledge and clinical reasoning continues to be the focus of both research and debate. In this article, we report on an experimental study in which we address the question of how much list-type selected response formats and short-essay type constructed response formats are related to differences in how test takers approach clinical reasoning tasks. The design of this study was informed by a framework developed within cognitive psychology which stresses the importance of the interplay between two components of reasoning-self-monitoring and response inhibition-while solving a task or case. The results presented support the argument that different response formats are related to different processing behavior. Importantly, the pattern of how different factors are related to a correct response in both situations seem to be well in line with contemporary accounts of reasoning. Consequently, we argue that when designing assessments of clinical reasoning, it is crucial to tap into the different facets of this complex and important medical process.

Differential diagnosis checklists reduce diagnostic error differentially: A randomised experiment

INTRODUCTION

Wrong and missed diagnoses contribute substantially to medical error. Can a prompt to generate alternative diagnoses (prompt) or a differential diagnosis checklist (DDXC) increase diagnostic accuracy? How do these interventions affect the diagnostic process and self-monitoring?

METHODS

Advanced medical students (N = 90) were randomly assigned to one of four conditions to complete six computer-based patient cases: group 1 (prompt) was instructed to write down all diagnoses they considered while acquiring diagnostic test results and to finally rank them. Groups 2 and 3 received the same instruction plus a list of 17 differential diagnoses for the chief complaint of the patient. For half of the cases, the DDXC contained the correct diagnosis (DDXC+), and for the other half, it did not (DDXC-; counterbalanced). Group 4 (control) was only instructed to indicate their final diagnosis. Mixed-effects models were used to analyse results.

RESULTS

Students using a DDXC that contained the correct diagnosis had better diagnostic accuracy, mean (standard deviation), 0.75 (0.44), compared to controls without a checklist, 0.49 (0.50), P < 0.001, but those using a DDXC that did not contain the correct diagnosis did slightly worse, 0.43 (0.50), P = 0.602. The number and relevance of diagnostic tests acquired were not affected by condition, nor was self-monitoring. However, participants spent more time on a case in the DDXC-, 4:20 min (2:36), P ≤ 0.001, and DDXC+ condition, 3:52 min (2:09), than in the control condition, 2:59 min (1:44), P ≤ 0.001.

DISCUSSION

Being provided a list of possible diagnoses improves diagnostic accuracy compared with a prompt to create a differential diagnosis list, if the provided list contains the correct diagnosis. However, being provided a diagnosis list without the correct diagnosis did not improve and might have slightly reduced diagnostic accuracy. Interventions neither affected information gathering nor self-monitoring.

Randomised controlled trial of students access to resources in an examination

INTRODUCTION

Assessment of healthcare professionals should be authentic to clinical practice. As clinicians regularly use resources in practice, similar resources should be available to those sitting assessment. There is limited information on the impacts of open-book (resource) assessments on standard setting for use in high-stakes assessments. This research aims to explore the effects on standard setting and student perceptions when open vs closed resources are available in high-stakes assessment of medical students.

METHODS

Students sat multiple-choice question (MCQ) examinations under both closed- and open-resource conditions in a randomised crossover design. A standard setting panel set pass-marks for both closed- and open-resource conditions of delivery, and we compared these pass-marks with each other and with actual performance. Students responded to a survey on perceptions of open-resource assessments.

RESULTS

The pass-mark was set higher when panellists considered open-resource conditions compared to closed conditions (59% vs 47%), but actual student performance showed no difference in scores between the two conditions. The net effect was that the pass rate was higher for closed than open-resource conditions (71% vs 34%). Open-resource conditions increased the time to complete the questions. The students perceived that open resource was more authentic but was more time-consuming and would require different preparation. Regarding the acceptability of including open resources in high-stakes assessment, the responses of students were mixed.

DISCUSSION

Pass standards based solely on judgements by panellists experienced in closed-resource conditions might not be applicable under open conditions. Questions vary in how much time accessing resources may take and the degree of assistance in selecting the correct answer. A programme of assessment could be constructed to include both closed- and open-resource condition assessments depending on the questions' content and format. Open-resource conditions may promote assessment preparation that focuses more on seeking and evaluating resources rather than learning facts.

Supporting Self-Directed Learning: A National Needs Analysis

INTRODUCTION

It is broadly agreed that physicians must be self-directed learners, and many studies measuring medical student readiness and capacity for self-directed learning (SDL) have been reported. However, less attention has been paid to faculty tasked with guiding students to develop SDL skills. In an effort to determine whether faculty feel equipped to foster SDL, a needs assessment was developed.

METHODS

The faculty needs assessment for fostering SDL (FNA-fSDL) was constructed and distributed nationally to medical educators. Faculty perceptions of the value of SDL, administration support, and faculty development were queried. Items also measured faculty recognition of cognitive, affective, and metacognitive strategies shown to cultivate SDL. Descriptive statistical analysis and frequency comparison tests to identify potential group differences related to work responsibility, classroom/clinical setting, and level of learner served were performed.

RESULTS

Among the 359 respondents representing diverse areas of medical education, there was overwhelming agreement that physicians in training should be self-directed learners and that faculty require training in educational approaches promoting SDL. Faculty were more likely to recognize strategies supporting development of metacognitive skills and least likely to identify those related to cognitive aspects of learning. Fewer than half of faculty had received training in fostering SDL at their home institutions or at conferences.

DISCUSSION

While there is strong support for the development of SDL among learners, these results suggest most faculty do not feel sufficiently trained to accomplish this goal. The FNA-fSDL can be used by institutions to assess their own faculty.

Skeptical self-regulation: Resident experiences of uncertainty about uncertainty

OBJECTIVES

Managing uncertainty is central to expert practice, yet how novice trainees navigate these moments is likely different than what has been described by experienced clinicians. Exploring trainees' experiences with uncertainty could therefore help explicate the unique cues that they attend to, how they appraise their comfort in these moments and how they enact responses within the affordances of their training environment.

METHODS

Informed by constructivist grounded theory, we explored how novice emergency medicine trainees experienced and managed clinical uncertainty in practice. We used a critical incident technique to prompt participants to reflect on experiences with uncertainty immediately following a clinical shift, exploring the cues they attended to and the approaches they used to navigate these moments. Two investigators coded line-by-line using constant comparison, organising the data into focused codes. The research team discussed the relationships between these codes and developed a set of themes that supported our efforts to theorise about the phenomenon.

RESULTS

We enrolled 13 trainees in their first two years of postgraduate training across two institutions. They expressed uncertainty about the root causes of the patient problems they were facing and the potential management steps to take, but also expressed a pervasive sense of uncertainty about their own abilities and their appraisals of the situation. This, in turn, led to challenges with selecting, interpreting and using the cues in their environment effectively. Participants invoked several approaches to combat this sense of uncertainty about themselves, rehearsing steps before a clinical encounter, checking their interpretations with others and implicitly calibrating their appraisals to those of more experienced team members.

CONCLUSIONS

Trainees' struggles with the legitimacy of their interpretations impact their experiences with uncertainty. Recognising these ongoing struggles may enable supervisors and other team members to provide more effective scaffolding, validation and calibration of clinical judgments and patient management.

Pre-registration nursing and occupational therapy students' experience of interprofessional simulation training designed to develop communication and team-work skills: A mixed methods study

To bridge the gap between university and the clinical environment, the university where this study was conducted incorporated interprofessional simulation sessions into the curricula for pre-registration nursing and occupational therapy students. The purpose of the study was to evaluate the effectiveness of simulation training to support the development of students' interprofessional communication and teamwork skills. Study participants were first-year students. A mixed methods study design was used that included: (1) a cross-sectional survey, (using the Interprofessional Education Collaborative Competency Self-Assessment tool), before and after the simulation session; and (2) two focus groups with students and one with academic staff. A total of 91 students and 5 staff participated. All students increased their self-perceived skills in interprofessional interactions. Two major themes were identified: 'Communication and Teamwork', with four sub-themes, (1) 'Significance of Communication in teamwork'; (2) 'Learning about, from and with each other'; (3) 'Professional role identification and collaborative practice'; and (4) 'Clinical leadership facilitated collaboration' and 'Lessons learnt from the simulation session', with three subthemes: (1) 'enhanced preparation for clinical placement'; (2) 'the experience of interacting with a simulated patient'; and (3) 'holistic patient care'. The results from the study demonstrated that interprofessional simulation sessions are an effective approach to introducing and developing collaborative clinical practice.

Teaching Medical Research to Medical Students: a Systematic Review

PHENOMENON

Research literacy remains important for equipping clinicians with the analytical skills to tackle an ever-evolving medical landscape and maintain an evidence-based approach when treating patients. While the role of research in medical education has been justified and established, the nuances involving modes of instruction and relevant outcomes for students have yet to be analyzed. Institutions acknowledge an increasing need to dedicate time and resources towards educating medical undergraduates on research but have individually implemented different pedagogies over differing lengths of time.

APPROACH

While individual studies have evaluated the efficacy of these curricula, the evaluations of educational methods and curriculum design have not been reviewed systematically. This study thereby aims to perform a systematic review of studies incorporating research into the undergraduate medical curriculum, to provide insights on various pedagogies utilized to educate medical students on research.

FINDINGS

Studies predominantly described two major components of research curricula-(1) imparting basic research skills and the (2) longitudinal application of research skills. Studies were assessed according to the 4-level Kirkpatrick model for evaluation. Programs that spanned minimally an academic year had the greatest proportion of level 3 outcomes (50%). One study observed a level 4 outcome by assessing the post-intervention effects on participants. Studies primarily highlighted a shortage of time (53%), resulting in inadequate coverage of content.

INSIGHTS

This study highlighted the value in long-term programs that support students in acquiring research skills, by providing appropriate mentors, resources, and guidance to facilitate their learning. The Dreyfus model of skill acquisition underscored the importance of tailoring educational interventions to allow students with varying experience to develop their skills. There is still room for further investigation of multiple factors such as duration of intervention, student voluntariness, and participants' prior research experience. Nevertheless, it stands that mentoring is a crucial aspect of curricula that has allowed studies to achieve level 3 Kirkpatrick outcomes and engender enduring changes in students.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s40670-020-01183-w.

The myth of ivory tower versus practice-oriented research: A systematic review of randomised studies in medical education

CONTEXT

A long-standing myth in medical education research is a divide between two different poles: research aiming to advance theory with little focus on practical applications ('ivory tower' research) and practically oriented research aiming to serve educators and decision-makers with little focus on advancing theory ('in-the-trenches' practice). We explored this myth in a sample of randomised medical education studies using Stokes' four-quadrant framework for the classification of research perspective.

METHODS

We searched MEDLINE, Embase, CINAHL, PsychINFO, ERIC, Web of Science and Scopus for studies in medical education using a randomised design that were published between 1 January 2018 and 31 December 2018. We used Stokes' four-quadrant framework to categorise the studies according to their use of theory, concepts and their justification for practical use. We compared medical education research published in medical education journals and clinical journals.

RESULTS

A total of 150 randomised studies were included in the analysis. The largest segment of studies (46.7%) was categorised as use-inspired basic research (Pasteur's Quadrant), closely followed by pure applied research (40.7%, Edison's Quadrant). Only a few studies were categorised as aiming to advance knowledge with no thought for practical educational application (2.0%, Bohr's Quadrant). The proportion of studies that included educational concepts and theory differed according to publication in clinical journals or medical education journals: 40.5% vs 71.8%, respectively, P < .001. There were no differences between journals with regard to the proportion of studies that included a practical educational or clinical rationale (P = .99).

CONCLUSION

In a large sample of studies using randomised designs, we found no evidence to support the myth that medical education research divides between two singular poles represented by 'ivory tower research' and 'in-the-trenches practice'. We did confirm prevailing assumptions regarding an emphasis on non-theoretical medical education research in clinical journals.

Effect of gamified perceptual learning on visual detection and discrimination skills in equine gait assessment

BACKGROUND

Visual assessment of equine lameness is an everyday veterinary task suffering from poor diagnostic accuracy. The aim of this study was to quantify the impact of the perceptual learning game 'LamenessTrainer' on skill development.

METHODS

Thirty-six undergraduate veterinary students engaged in four game modules teaching the assessment of fore- and hindlimb lameness. Computer animations of horses in this game displayed 0% (sound) to 70% (moderately lame) vertical movement asymmetry of head and pelvis. Performance, learning effects, diagnostic accuracy, detection thresholds and survey responses were analysed.

RESULTS

Following staircase learning, more than 80% of students reliably classified horses with ≥20% asymmetry for forelimb lameness, ≥40% asymmetry for simplified hindlimb lameness and ≥50% asymmetry for realistic hindlimb lameness. During random presentation, on average 82% of sound and 65% of lame horses were assessed correctly during forelimb lameness evaluation, dropping to 39% of sound and 56% of lame horses for hindlimb lameness.

CONCLUSION

In less than two hours, systematic perceptual learning through deliberate practice can develop visual assessment skills to an accuracy level comparable to expert assessors scoring the same animations. Skills should be developed further to improve misclassifications of sound and mildly lame horses, especially for hindlimb lameness evaluation.

Anesthesiologists' airway management expertise: Identifying subjective and objective knowledge gaps

BACKGROUND

Failure in airway management continues to cause preventable patient harm, and the recommended continuing education is challenged by anesthesiologists' unknown knowledge gaps. This study aimed to identify anesthesiologists' subjective and objective knowledge gaps as well as areas where anesthesiologists are incorrect and unaware.

METHODS

An adaptive E-learning program with 103 questions on adult airway management was used for subjective and objective assessment of anesthesiologists' knowledge. All anesthesiologists in the Capital Region of Denmark were invited to participate.

RESULTS

The response rate was 67% (191/285). For preoperative planning, participants stated low confidence (subjective assessment) regarding predictors of difficult airway management in particular (69.1%-79.1%). Test scores (objective assessment) were lowest for obstructive sleep apnea as a predictor of difficult airway management (28.8% correct), with participants being incorrect and unaware in 33.5% of the answers. For optimization of basic techniques, the lowest confidence ratings related to patient positioning and prediction of difficulties (57.4%-83.2%), which agreed with the lowest test scores. Concerning advanced techniques, videolaryngoscopy prompted the lowest confidence (72.4%-85.9%), while emergency cricothyrotomy resulted in the lowest test scores (47.4%-67.8%). Subjective and objective assessments correlated and lower confidence was associated with lower test scores: preoperative planning [r = -.58, P < .001], optimization of basic techniques [r = -.58, P = .002], and advanced techniques [r = -.71, P < .001].

CONCLUSION

We identified knowledge gaps in important areas of adult airway management with differing findings from the subjective and objective assessments. This underlines the importance of objective assessment to guide continuing education.

Exploring why we learn from productive failure: insights from the cognitive and learning sciences

Advances in Health Sciences Education (AHSE) has been at the forefront of the cognitive wave in health professions education for the past 25 years. One example is research on productive failure, a teaching strategy that asks learners to attempt to generate solutions to difficult problems before receiving instruction. This study compared the effectiveness of productive failure with indirect failure to further characterize the underpinning cognitive mechanisms of productive failure. Year one pharmacy students (N = 42) were randomly assigned to a productive failure or an indirect failure learning condition. The problem of estimating renal function based on serum creatinine was described to participants in the productive failure learning condition, who were then asked to generate a solution. Participants in the indirect failure condition learned about the same problem and were given incorrect solutions that other students had created, as well as the Cockcroft-Gault formula, and asked to compare and contrast the equations. Immediately thereafter all participants completed a series of tests designed to assess acquisition, application, and preparation for future learning (PFL). The tests were repeated after a 1-week delay. Participants in the productive failure condition outperformed those in the indirect failure condition, both on the immediate PFL assessment, and after a 1-week delay. These results emphasize the crucial role of generation in learning. When preparing novice students to learn new knowledge in the future, generating solutions to problems prior to instruction may be more effective than simply learning about someone else's mistakes. Struggle and failure are most productive when experienced personally by a learner because it requires the learner to engage in generation, which deepens conceptual understanding.

Defining and tracking medical student self-monitoring using multiple-choice question item certainty

BACKGROUND

Self-monitoring is an important component of clinical practice. It is underpinned by the framework of self-efficacy which is concerned with judgments of how well one believes one can achieve or perform a task. This research aimed to develop criteria for adequate self-monitoring, then to measure patterns of self-monitoring, and to explore how these patterns relate to a student's year in a medical course and to patterns of knowledge.

METHODS

Analysis of individuals' levels of correctness in answering assessment items and their certainty in correctness may be used to inform assessments of ability to self-monitor. Two criteria were proposed and applied to define adequate self-monitoring. Firstly, increasing proportions correct with increasing levels of certainty. Secondly, having a proportion correct for high certainty responses that was not lower than cohort levels. Student responses in progress tests comprising multiple-choice questions (MCQs) and associated certainty were analysed. Criteria for the presence of adequate self-monitoring and for adequate knowledge were applied to the results of each of four tests conducted over 2 years, and used to categorise patterns of self-monitoring and knowledge.

RESULTS

Data from 3 year group cohorts totalling 737 students were analysed. The majority (58%) of students demonstrated adequate knowledge and met both criteria for adequate self-monitoring across all four tests. The most advanced year group cohort had the highest rates of adequate knowledge and the highest rates of meeting both self-monitoring criteria. The patterns of self-monitoring were the same as the patterns of knowledge across the four tests for 454 students, but for the remaining 283 the patterns of self-monitoring and knowledge differed.

CONCLUSION

Analysis of responses to item level certainty has informed development of a definition of adequate self-monitoring that may be applied to individual student's responses from a single test, and to track the adequacy of a student's self-monitoring over time. Patterns of self-monitoring tend to match patterns of knowledge, but not in all cases, suggesting the self-monitoring measure could provide additional information about student ability.

Self-monitoring accuracy does not increase throughout undergraduate medical education

CONTEXT

Accurate self-assessment of one's performace on a moment-by-moment basis (ie, accurate self-monitoring) is vital for the self-regulation of practising physicians and indeed for the effective regulation of self-directed learning during medical education. However, little is currently known about the functioning of self-monitoring and its co-development with medical knowledge across medical education. This study is the first to simultaneously investigate a number of relevant aspects and measures that have so far been studied separately: different measures of self-monitoring for a broad area of medical knowledge across 10 different performance levels.

METHODS

This study assessed the self-monitoring accuracy of medical students (n = 3145) across 10 semesters. Data collected during the administration of the formative Berlin Progress Test Medicine (PTM) were analysed. The PTM comprises 200 multiple-choice questions covering all major medical disciplines and organ systems. A self-report indicator (ie, confidence) and two behavioural indicators of self-monitoring accuracy (ie, response time and the likelihood of changing an initial answer to a correct rather than an incorrect item) were examined for their development over semesters.

RESULTS

Analyses of more than 390 000 observations (of approximately 250 students per semester) showed that confidence was higher for correctly than for incorrectly answered items and that 86% of items answered with high confidence were indeed correct. Response time and the likelihood of the initial answer being changed were higher when the initial answer was incorrect than when it was correct. Contrary to expectations, no differences in self-monitoring accuracy were observed across semesters.

CONCLUSIONS

Convergent evidence from different measures of self-monitoring suggests that medical students self-monitor their knowledge on a question-by-question basis well, although not perfectly, and to the same degree as has been found in studies outside medicine. Despite large differences in performance, no variations in self-monitoring across semesters (with the exception of the first semester) were observed.

Maintenance of certification for practicing physicians: a review of current challenges and considerations

Maintenance of certification (MOC) has become increasingly important in medicine to ensure maintenance of competence throughout a physician's career. This paper reviews current issues and challenges associated with MOC in medicine, including how to define medical competencies for practicing physicians, assessment, and how best to support physicians' lifelong learning in a continuous and self-motivated way. We explore how the combination of self-monitoring, regular feedback, and peer support could improve self-assessment. Effective MOC programs are learner-driven, focused on every day practice, and incorporate educational principles. We discuss the importance of MOC to the physicians' actual practice to improve acceptability. We review the benefits of tailored programs as well as decentralization of MOC programs to better characterize the physician's practice. Lastly, we discuss the value of simulation-based medical education in MOC programs. Simulation-based education could be used to practice uncommon complications, life-threatening scenarios, non-technical skills improvement, and become proficient with new technology. As learners find simulation experiences educationally valuable, clinically relevant, and positive, simulation could be a way of increasing physicians' participation in MOC programs.

Longitudinal evaluation of attitudes to interprofessional collaboration: time for a change?

Longitudinal evaluation of interprofessional programming is necessary to understand the impact of interprofessional education (IPE) for collaborative practice. Attitude is one measure of intended behavior. The Interprofessional Attitude Scale (IPAS) was developed as an improved measure to examine the readiness for interprofessional education and collaborative practice. As a means to pilot a protocol for a longitudinal comprehensive evaluation of an interprofessional program with 13 health professional programs in 6 health science faculties, the IPAS was administered over a 3-year period to three cohorts of students. The results of the IPAS indicated no significant differences detected across the 3 years. Although there was a significant difference on the Patient Centredness subscale for Cohorts 2 and 3 the effect size was small. Previous experience in an interprofessional course was a predictor of more positive patient centeredness attitudes, but other IPE experience and years of education were not predictors of any interprofessional attitudes. Students from the Medicine & Dentistry (M&D) faculty ranked significantly lower on the teamwork roles and responsibilities subscale compared to the other faculties, except Rehabilitation Medicine (RM). However, once the outliers in RM were removed, there was a significant difference on this subscale between RM and M&D. This paper outlines the challenges with using the IPAS as part of a comprehensive program evaluation and identifies issues with longitudinal data collection for evaluation with large student cohorts.

Meaningful feedback through a sociocultural lens

This AMEE guide provides a framework and practical strategies for teachers, learners and institutions to promote meaningful feedback conversations that emphasise performance improvement and professional growth. Recommended strategies are based on recent feedback research and literature, which emphasise the sociocultural nature of these complex interactions. We use key concepts from three theories as the underpinnings of the recommended strategies: sociocultural, politeness and self-determination theories. We view the content and impact of feedback conversations through the perspective of learners, teachers and institutions, always focussing on learner growth. The guide emphasises the role of teachers in forming educational alliances with their learners, setting a safe learning climate, fostering self-awareness about their performance, engaging with learners in informed self-assessment and reflection, and co-creating the learning environment and learning opportunities with their learners. We highlight the role of institutions in enhancing the feedback culture by encouraging a growth mind-set and a learning goal-orientation. Practical advice is provided on techniques and strategies that can be used and applied by learners, teachers and institutions to effectively foster all these elements. Finally, we highlight throughout the critical importance of congruence between the three levels of culture: unwritten values, espoused values and day to day behaviours.

First-year medical students' calibration bias and accuracy across clinical reasoning activities

To be safe and effective practitioners and learners, medical professionals must be able to accurately assess their own performance to know when they need additional help. This study explored the metacognitive judgments of 157 first-year medical students; in particular, the study examined students' self-assessments or calibration as they engaged in a virtual-patient simulation targeting clinical reasoning practices. Examining two key subtasks of a patient encounter, history (Hx) and physical exam (PE), the authors assessed the level of variation in students' behavioral performance (i.e., effectiveness and efficiency) and judgments of performance (i.e., calibration bias and accuracy) across the two subtasks. Paired t tests revealed that the Hx subtask was deemed to be more challenging than the PE subtask when viewed in terms of both actual and perceived performance. In addition to students performing worse on the Hx subtask than PE, they also perceived that they performed less well for Hx. Interestingly, across both subtasks, the majority of participants overestimated their performance (98% of participants for Hx and 95% for PE). Correlation analyses revealed that the participants' overall level of accuracy in metacognitive judgments was moderately stable across the Hx and PE subtasks. Taken together, findings underscore the importance of assessing medical students' metacognitive judgments at different points during a clinical encounter.

Assigning Medical Students Learning Goals: Do They Do It, and What Happens When They Don't?

Theory: Medical curricula now include more time for trainees to manage their studying independently, yet evidence suggests that time is not well spent without guidance. Social-cognitivist models of self-regulated learning suggest value when guiding learners to set goals related to their performance processes (actions producing outcomes) versus their performance outcomes (products of performance). Hypotheses: We expected participants oriented to set process goals would demonstrate better suturing skill retention compared with participants oriented to set outcome goals. Method: We randomly assigned 41 medical students to two groups: outcome oriented or process oriented. They self-scored their performance using a visual analog scale on every third trial during 25 training trials, and during 10 retention trials 2 weeks later. Two raters assessed participants' suturing performances (process) and final products (outcome). After finding weak support for our hypothesis, we calculated a "self-monitoring calibration coefficient" as the Pearson's correlation between the raters' average score and each participant's self-scores. We used a mixed-effects analysis of variance to compare participants' performance scores as well as t tests and an analysis of variance to compare their self-monitoring calibration coefficients. Results: Analysis of skill retention data revealed a significant Group × Trial interaction, suggesting a benefit for the process group only for the 10th retention trial (p = .03). During training, the process group had significantly better (p = .02) self-monitoring calibration (r = .71 ± .29) than the outcome group (r = .38 ± .55). In retention, participants in both groups were significantly better calibrated (p = .04) with rater's scores of performance processes (r = .39 ± .60) versus performance outcomes (r = .11 ± .63). Conclusions: Our findings provide limited evidence for our original hypothesis. Perhaps more important, however, our self-monitoring calibration data highlighted inconsistencies between our interventions and our participants' apparent preferences. Not all participants adopted their assigned goal setting orientation, showing that researchers and educators must consider the extent to which trainees adopt imposed instructions in any educational intervention.

Situational cues surrounding family physicians seeking external resources while self-monitoring in practice

Many models of safe and effective clinical decision making in medical practice emphasize the importance of recognizing moments of uncertainty and seeking help accordingly. This is not always done effectively, but we know little about what cues prompt health professionals to call on resources beyond their own knowledge or skill set. Such information would offer guidance regarding how systems might be designed to offer better individual support. In this study, the authors explored the situational factors that are present during moments of uncertainty that lead primary care physicians to access external resources. To do so, a generic qualitative exploratory analysis was conducted on 72 narratives collected through audio recorder-based, self-observational, journaling completed by 12 purposively selected family physicians. Participants were asked to provide a detailed descriptive account of the circumstances surrounding their consultation of external resources immediately after 6 sequential patient encounters in which they felt compelled to seek such support. Thematic analysis of the transcripts was performed to better understand participants' experiences of the social, contextual, and personal features surrounding decisions to seek support. When doing so we observed that specific features of patient encounters were routinely present when physicians decided to access external sources for help. These included medical aspects of the case (e.g., complex presentations), social aspects (e.g., the presence of another individual), and personal factors (e.g., feeling a need for reassurance). External resources were seen as an opportunity for verification, a mechanism to increase patient satisfaction, and a means through which to defend decision-making. Accessing such resources appeared to influence the physician-patient relationship for various reasons. Recognition and further study of the cues that prompt use of external information will further our understanding of physicians' behavioural responses to challenging/uncertain situations, highlight mechanisms through which a culture of self-directed assessment seeking might be encouraged, and offer guidance regarding ways in which physicians can be encouraged to practice mindfully. Our results make it clear that reasons for which primary care physicians seek the support of external resources may be multifactorial and, therefore, one should be cautious when inferring reasons for the pursuit of such support.

Using an Adaptive, Self-Directed Web-Based Learning Module to Enhance Residents' Medical Knowledge Prior to a New Clinical Rotation

BACKGROUND

The effect of self-directed adaptive learning on internal medicine residents' knowledge prior to a new clinical rotation is not known.

METHODS

We developed an adaptive, online, self-directed spaced repetition module and determined the effect on medical knowledge acquisition. We randomized postgraduate year 1 internal medicine residents into two groups. The intervention group (n = 27) received an electronic version of the clinical rotation curriculum as portable document format (PDF) files and participated in the online module, delivered via Moodle, a free, open-source learning management system. The non-intervention group (n = 27) only received the PDF files. All residents participated in a medical knowledge test at baseline and 3 months later.

RESULTS

Both groups were similar at study baseline in terms of age, trainee type, years since graduation, results at United States Medical Licensing Examination (USMLE) Step 1, 2, In-Training Examination (ITE), and pre-intervention evaluation. There was a statistically significant improvement in scores on the post-intervention medical knowledge assessment for the intervention group when compared with the non-intervention group (24.2 ± 15.4% vs. 8.6 ± 9.9%, p < 0.001).

CONCLUSION

An online, self-directed, adaptive spaced repetition-learning module can offer a simple and effective method to increase the medical knowledge present at the start of residents' clinical rotations.

Debiasing versus knowledge retrieval checklists to reduce diagnostic error in ECG interpretation

There is an ongoing debate regarding the cause of diagnostic errors. One view is that errors result from unconscious application of cognitive heuristics; the alternative is that errors are a consequence of knowledge deficits. The objective of this study was to compare the effectiveness of checklists that (a) identify and address cognitive biases or (b) promote knowledge retrieval, as a means to reduce errors in ECG interpretation. Novice postgraduate year (PGY) 1 emergency medicine and internal medicine residents (n = 40) and experienced cardiology fellows (PGY 4-6) (n = 21) were randomly allocated to three conditions: a debiasing checklist, a content (knowledge) checklist, or control (no checklist) to be used while interpreting 20 ECGs. Half of the ECGs were deliberately engineered to predispose to bias. Diagnostic performance under either checklist intervention was not significantly better than the control. As expected, more errors occurred when cases were designed to induce bias (F = 96.9, p < 0.0001). There was no significant interaction between the instructional condition and level of learner. Checklists attempting to help learners identify cognitive bias or mobilize domain-specific knowledge did not have an overall effect in reducing diagnostic errors in ECG interpretation, although they may help novices. Even when cognitive biases are deliberately inserted in cases, cognitive debiasing checklists did not improve participants' performance.

Accuracy of self-monitoring: does experience, ability or case difficulty matter?

CONTEXT

The ability to self-monitor one's performance in clinical settings is a critical determinant of safe and effective practice. Various studies have shown this form of self-regulation to be more trustworthy than aggregate judgements (i.e. self-assessments) of one's capacity in a given domain. However, little is known regarding what cues inform learners' self-monitoring, which limits an informed exploration of interventions that might facilitate improvements in self-monitoring capacity. The purpose of this study is to understand the influence of characteristics of the individual (e.g. ability) and characteristics of the problem (e.g. case difficulty) on the accuracy of self-monitoring by medical students.

METHODS

In a cross-sectional study, 283 medical students from 5 years of study completed a computer-based clinical reasoning exercise. Confidence ratings were collected after completing each of six cases and the accuracy of self-monitoring was considered to be a function of confidence when the eventual answer was correct relative to when the eventual answer was incorrect. The magnitude of that difference was then explored as a function of year of seniority, gender, case difficulty and overall aptitude.

RESULTS

Students demonstrated accurate self-monitoring by virtue of giving higher confidence ratings (57.3%) and taking a shorter time to work through cases (25.6 seconds) when their answers were correct relative to when they were wrong (41.8% and 52.0 seconds, respectively; p< 0.001 and d > 0.5 in both instances). Self-monitoring indices were related to student seniority and case difficulty, but not to overall ability or student gender.

CONCLUSIONS

This study suggests that the accuracy of self-monitoring is context specific, being heavily influenced by the struggles students experience with a particular case rather than reflecting a generic ability to know when one is right or wrong. That said, the apparent capacity to self-monitor increases developmentally because increasing experience provides a greater likelihood of success with presented problems.

Measuring Coaching in Undergraduate Medical Education: the Development and Psychometric Validation of New Instruments

BACKGROUND

Coaching is emerging as a novel approach to guide medical students toward becoming competent, reflective physicians and master adaptive learners. However, no instruments currently exist to measure academic coaching at the undergraduate medical education level.

OBJECTIVE

To describe the development and psychometric assessments of two instruments designed to assess academic coaching of medical students toward creating a robust measurement model of this educational paradigm.

DESIGN

Observational psychometric.

PARTICIPANTS

All medical students in the 2014 and 2015 cohorts and all their coaches were invited to complete the instruments being tested, which led to 662 medical student responses from 292 medical students and 468 coaching responses from 22 coaches being included in analyses. Medical student response rates were 75.7% from 2014 and 75.5% from 2015. Overall, the coaches' response rate was 71%.

MAIN MEASURES

Two 31-item instruments were initially developed, one for medical students to assess their coach and one for faculty coaches to assess their students, both of which evaluated coaching based on definitions we formulated using existing literature. Each was administered to two cohorts of medical students and coaches in 2015 and 2016. An exploratory factor analysis using principal component analysis as the extraction method and Varimax with Kaiser normalization as the rotation method was conducted.

KEY RESULTS

Eighteen items reflecting four domains were retained with eigenvalues higher than 1.0 for medical students' assessment of coaching, which measured promoting self-monitoring, relationship building, promoting reflective behavior, and establishing foundational ground rules. Sixteen items reflecting two domains were retained for the faculty assessment of coaching with eigenvalues higher than 1.0: the Practice of Coaching and Relationship Formation.

CONCLUSIONS

We successfully developed and psychometrically validated surveys designed to measure key aspects of the coaching relationship, coaching processes, and reflective outcomes. The new validated instruments offer a robust measurement model for academic coaching.