"I'll never play professional football" and other fallacies of self-assessment

Assessment in the context of licensure and certification

Over the past 25 years, three major forces have had a significant influence on licensure and certification: the shift in focus from educational process to educational outcomes, the increasing recognition of the need for learning and assessment throughout a physician's career, and the changes in technology and psychometrics that have opened new vistas for assessment. These forces have led to significant changes in assessment for licensure and certification. To respond to these forces, licensure and certification programs have improved the ways in which their examinations are constructed, scored, and delivered. In particular, we note the introduction of adaptive testing; automated item creation, scoring, and test assembly; assessment engineering; and data forensics. Licensure and certification programs have also expanded their repertoire of assessments with the rapid development and adoption of simulation and workplace-based assessment. Finally, they have invested in research intended to validate their programs in four ways: (a) the acceptability of the program to stakeholders, (b) the extent to which stakeholders are encouraged to learn and improve, (c) the extent to which there is a relationship between performance in the programs and external measures, and (d) the extent to which there is a relationship between performance as measured by the assessment and performance in practice. Over the past 25 years, changes in licensure and certification have been driven by the educational outcomes movement, the need for lifelong learning, and advances in technology and psychometrics. Over the next 25 years, we expect these forces to continue to exert pressure for change which will lead to additional improvement and expansion in examination processes, methods of assessment, and validation research.

American Board of Medical Specialties Maintenance of Certification: theory and evidence regarding the current framework

The American Board of Medical Specialties Maintenance of Certification Program (ABMS MOC) is designed to provide a comprehensive approach to physician lifelong learning, self-assessment, and quality improvement (QI) through its 4-part framework and coverage of the 6 competencies previously adopted by the ABMS and the Accreditation Council for Graduate Medical Education (ACGME). In this article, the theoretical rationale and exemplary empiric data regarding the MOC program and its individual parts are reviewed. The value of each part is considered in relation to 4 criteria about the relationship of the competencies addressed within that part to (1) patient outcomes, (2) physician performance, (3) validity of the assessment or educational methods utilized, and (4) learning or improvement potential. Overall, a sound theoretical rationale and a respectable evidence base exists to support the current structure and elements of the MOC program. However, it is incumbent on the ABMS and ABMS member boards to continue to examine their programs moving forward to assure the public and the profession that they are meeting expectations, are clinically relevant, and provide value to patients and participating physicians, and to refine and improve them as ongoing research indicates.

Self-other agreement in multisource feedback: the influence of doctor and rater group characteristics

INTRODUCTION

Multisource feedback (MSF) ratings provided by patients and colleagues are often poorly correlated with doctors' self-assessments. Doctors' reactions to feedback depend on its agreement with their own perceptions, but factors influencing self-other agreement in doctors' MSF ratings have received little attention. We aimed to identify the characteristics of doctors and their rater groups that affect self-other agreement in MSF ratings.

METHODS

We invited 2454 doctors to obtain patient and colleague feedback using the UK General Medical Council's MSF questionnaires and to self-assess on core items from both patient (PQ) and colleague (CQ) questionnaires. Correlations and differences between doctor, patient and colleague mean feedback scores were examined. Regression analyses identified the characteristics of doctors and their rater groups that influenced self-other score agreement.

RESULTS

1065 (43%) doctors returned at least one questionnaire, of whom 773 (73%) provided self and patient PQ scores and 1026 (96%) provided self and colleague CQ scores. Most doctors rated themselves less favourably than they were rated by either their patients or their colleagues. This tendency to underrate performance in comparison to external feedback was influenced by the doctor's place of training, clinical specialty, ethnicity and the profile of his/her patient and colleague rater samples but, in contrast to studies undertaken in nonmedical settings, was unaffected by age or gender.

DISCUSSION

Self-other agreement in MSF ratings is influenced by characteristics of both raters and ratees. Managers, appraisers, and others responsible for interpreting and reviewing feedback results with the doctor need to be aware of these influences.

Assessing interpersonal and communication skills in emergency medicine

Interpersonal and communication skills (ICS) are a key component of several competency-based schemata and key competency in the set of six Accreditation Council for Graduate Medical Education (ACGME) core competencies. With the shift toward a competency-based educational framework, the importance of robust learner assessment becomes paramount. The journal Academic Emergency Medicine (AEM) hosted a consensus conference to discuss education research in emergency medicine (EM). This article summarizes the initial preparatory research that was conducted to brief consensus conference attendees and reports the results of the consensus conference breakout session as it pertains to ICS assessment of learners. The goals of this consensus conference session were to twofold: 1) to determine the state of assessment of observable learner performance and 2) to determine a research agenda within the ICS field for medical educators. The working group identified six key recommendations for medical educators and researchers.

Self versus external assessment for technical tasks in surgery: a narrative review

BACKGROUND

Self-assessment is an intricate component of continuing professional development and lifelong learning for health professionals. The agreement between self and external assessment for cognitive tasks in health professionals is reported to be poor; however, this topic has not been reviewed for technical tasks in surgery.

OBJECTIVE

To compare self and external assessment for technical tasks in surgery.

METHODS

MEDLINE, ERIC, and Google Scholar databases were searched for data from January 1960 to November 2011. Inclusion criteria were restricted to articles published in English in peer-reviewed journals, which reported on a comparison between self and external assessment for a technical task in a surgical specialty and involved medical students, surgical residents, surgical fellows, or practicing surgeons. Abstracts of identified articles were reviewed and pertinent full-text versions were retrieved. Manual searching of bibliographies for additional studies was performed. Data were extracted in a systematic manner.

RESULTS

From a total of 49 citations, 17 studies (35%) were selected for review. Eight of the 17 studies (47%) reported no agreement, whereas 9 studies (53%) reported an agreement between self and external assessment for technical tasks in surgery. Four studies (24%) reported higher self versus external assessment scores, whereas 3 studies (18%) reported lower self versus external assessment scores. Sixteen studies (94%) focused on retrospective self-assessment and 1 study (6%) focused on predictive self-assessment. Agreement improved with higher levels of participant training; with high-quality, timely, and relevant feedback; and with postprocedure video review.

CONCLUSIONS

This review demonstrated mixed results regarding an agreement between self and external assessment scores for technical tasks in surgery. Future investigations should attempt to improve the study design by accounting for differences between men and women, conducting paired and independent mean comparisons of self and external assessments, and ensuring that external assessments are valid and reliable.

How do postgraduate GP trainees regulate their learning and what helps and hinders them? A qualitative study

BACKGROUND

Self-regulation is essential for professional development. It involves monitoring of performance, identifying domains for improvement, undertaking learning activities, applying newly learned knowledge and skills and self-assessing performance. Since self-assessment alone is ineffective in identifying weaknesses, learners should seek external feedback too. Externally regulated educational interventions, like reflection, learning portfolios, assessments and progress meetings, are increasingly used to scaffold self-regulation.The aim of this study is to explore how postgraduate trainees regulate their learning in the workplace, how external regulation promotes self-regulation and which elements facilitate or impede self-regulation and learning.

METHODS

In a qualitative study with a phenomenologic approach we interviewed first- and third-year GP trainees from two universities in the Netherlands. Twenty-one verbatim transcripts were coded. Through iterative discussion the researchers agreed on the interpretation of the data and saturation was reached.

RESULTS

Trainees used a short and a long self-regulation loop. The short loop took one week at most and was focused on problems that were easy to resolve and needed minor learning activities. The long loop was focused on complex or recurring problems needing multiple and planned longitudinal learning activities. External assessments and formal training affected the long but not the short loop. The supervisor had a facilitating role in both loops. Self-confidence was used to gauge competence.Elements influencing self-regulation were classified into three dimensions: personal (strong motivation to become a good doctor), interpersonal (stimulation from others) and contextual (organizational and educational features).

CONCLUSIONS

Trainees did purposefully self-regulate their learning. Learning in the short loop may not be visible to others. Trainees should be encouraged to actively seek and use external feedback in both loops. An important question for further research is which educational interventions might be used to scaffold learning in the short loop. Investing in supervisor quality remains important, since they are close to trainee learning in both loops.

Self-monitoring and its relationship to medical knowledge

In the domain of self-assessment, researchers have begun to draw distinctions between summative self-assessment activities (i.e., making an overall judgment of one's ability in a particular domain) and self-monitoring processes (i.e., an "in the moment" awareness of whether one has the necessary knowledge or skills to address a specific problem with which one is faced). Indeed, previous research has shown that, when responding to both short answer and multiple choice questions, individuals are able to assess the likelihood of answering questions correctly on a moment-by-moment basis, even though they are not able to generate an accurate self-assessment of overall performance on the test. These studies, however, were conducted in the context of low-stakes tests of general "trivia". The purpose of the present study was to further this line of research by investigating the relationship between self-monitoring and performance in the context of a high stakes test assessing medical knowledge. Using a recent administration of the Medical Council of Canada Qualifying Examination Part I, we examined three measures intended to capture self-monitoring: (1) the time taken to respond to each question, (2) the number of questions a candidate flagged as needing to be considered further, and (3) the likelihood of changing one's initial answer. Differences in these measures as a function of the accuracy of the candidate's response were treated as indices of each candidate's ability to judge his or her likelihood of responding correctly. The three self-monitoring indices were compared for candidates at three different levels of overall performance on the exam. Relative to correct responses, when examinees initially responded incorrectly, they spent more time answering the question, were more likely to flag the question for future consideration, and were more likely to change their answer before committing to a final answer. These measures of self-monitoring were modulated by candidate performance in that high performing examinees showed greater differences on these indices relative to poor performing examinees. Furthermore, reliability analyses suggest that these difference measures hold promise for reliably differentiating self-monitoring at the level of individuals, at least within a given content area. The results suggest that examinees were self-monitoring their knowledge and skills on a question by question basis and altering their behavior appropriately in the moment. High performing individuals showed stronger evidence of accurate self-monitoring than did low performing individuals and the reliability of these measures suggests that they have the potential to differentiate between individuals. How these findings relate to performance in actual clinical settings remains to be seen.

Reliable and valid tools for measuring surgeons' teaching performance: residents' vs. self evaluation

BACKGROUND

In surgical education, there is a need for educational performance evaluation tools that yield reliable and valid data. This paper describes the development and validation of robust evaluation tools that provide surgeons with insight into their clinical teaching performance. We investigated (1) the reliability and validity of 2 tools for evaluating the teaching performance of attending surgeons in residency training programs, and (2) whether surgeons' self evaluation correlated with the residents' evaluation of those surgeons.

MATERIALS AND METHODS

We surveyed 343 surgeons and 320 residents as part of a multicenter prospective cohort study of faculty teaching performance in residency training programs. The reliability and validity of the SETQ (System for Evaluation Teaching Qualities) tools were studied using standard psychometric techniques. We then estimated the correlations between residents' and surgeons' evaluations.

RESULTS

The response rate was 87% among surgeons and 84% among residents, yielding 2625 residents' evaluations and 302 self evaluations. The SETQ tools yielded reliable and valid data on 5 domains of surgical teaching performance, namely, learning climate, professional attitude towards residents, communication of goals, evaluation of residents, and feedback. The correlations between surgeons' self and residents' evaluations were low, with coefficients ranging from 0.03 for evaluation of residents to 0.18 for communication of goals.

CONCLUSIONS

The SETQ tools for the evaluation of surgeons' teaching performance appear to yield reliable and valid data. The lack of strong correlations between surgeons' self and residents' evaluations suggest the need for using external feedback sources in informed self evaluation of surgeons.

Giving feedback on clinical skills: are we starving our young?

BACKGROUND

Feedback to clinical learners about their performance is crucial to their development into competent clinicians. Most feedback is provided by clinicians who have little or no formal training for this aspect of their teaching role.

METHOD

Narrative review.

FINDINGS

Feedback is most effective if provided by clinical mentors based on observations of behavior, with emphasis on correctable deficiencies. The difference between feedback, which is formative, and evaluation, which is summative, needs to be understood by both the giver and receiver. The ability to self-assess is an important related concept, with studies showing that self-assessment skills are lowest in individuals found to be the least competent in external assessments and in individuals with a high level of confidence.

CONCLUSIONS

Feedback is an important component in learners' development, and clinical faculty should be educated about the importance of providing feedback, and the means to do so effectively must be provided to them. Despite several decades of accumulated knowledge in this area, the evidence from learners is that we continue to starve them of this critical nutrient for their growth.

Factors influencing responsiveness to feedback: on the interplay between fear, confidence, and reasoning processes

Self-appraisal has repeatedly been shown to be inadequate as a mechanism for performance improvement. This has placed greater emphasis on understanding the processes through which self-perception and external feedback interact to influence professional development. As feedback is inevitably interpreted through the lens of one's self-perceptions it is important to understand how learners interpret, accept, and use feedback (or not) and the factors that influence those interpretations. 134 participants from 8 health professional training/continuing competence programs were recruited to participate in focus groups. Analyses were designed to (a) elicit understandings of the processes used by learners and physicians to interpret, accept and use (or not) data to inform their perceptions of their clinical performance, and (b) further understand the factors (internal and external) believed to influence interpretation of feedback. Multiple influences appear to impact upon the interpretation and uptake of feedback. These include confidence, experience, and fear of not appearing knowledgeable. Importantly, however, each could have a paradoxical effect of both increasing and decreasing receptivity. Less prevalent but nonetheless important themes suggested mechanisms through which cognitive reasoning processes might impede growth from formative feedback. Many studies have examined the effectiveness of feedback through variable interventions focused on feedback delivery. This study suggests that it is equally important to consider feedback from the perspective of how it is received. The interplay observed between fear, confidence, and reasoning processes reinforces the notion that there is no simple recipe for the delivery of effective feedback. These factors should be taken into account when trying to understand (a) why self-appraisal can be flawed, (b) why appropriate external feedback is vital (yet can be ineffective), and (c) why we may need to disentangle the goals of performance improvement from the goals of improving self-assessment.

Factors influencing responsiveness to feedback: on the interplay between fear, confidence, and reasoning processes

Self-appraisal has repeatedly been shown to be inadequate as a mechanism for performance improvement. This has placed greater emphasis on understanding the processes through which self-perception and external feedback interact to influence professional development. As feedback is inevitably interpreted through the lens of one's self-perceptions it is important to understand how learners interpret, accept, and use feedback (or not) and the factors that influence those interpretations. 134 participants from 8 health professional training/continuing competence programs were recruited to participate in focus groups. Analyses were designed to (a) elicit understandings of the processes used by learners and physicians to interpret, accept and use (or not) data to inform their perceptions of their clinical performance, and (b) further understand the factors (internal and external) believed to influence interpretation of feedback. Multiple influences appear to impact upon the interpretation and uptake of feedback. These include confidence, experience, and fear of not appearing knowledgeable. Importantly, however, each could have a paradoxical effect of both increasing and decreasing receptivity. Less prevalent but nonetheless important themes suggested mechanisms through which cognitive reasoning processes might impede growth from formative feedback. Many studies have examined the effectiveness of feedback through variable interventions focused on feedback delivery. This study suggests that it is equally important to consider feedback from the perspective of how it is received. The interplay observed between fear, confidence, and reasoning processes reinforces the notion that there is no simple recipe for the delivery of effective feedback. These factors should be taken into account when trying to understand (a) why self-appraisal can be flawed, (b) why appropriate external feedback is vital (yet can be ineffective), and (c) why we may need to disentangle the goals of performance improvement from the goals of improving self-assessment.

Self-assessment and students' study strategies in a community of clinical practice: a qualitative study

BACKGROUND

Self-assessment is recognized as a necessary skill for lifelong learning. It is widely reported to offer numerous advantages to the learner. The research evaluated the impact of students' and supervisors' self-assessment and feedback training on students' perceptions and practices of self-assessment. Moreover, it evaluated the effect of self-assessment process on students' study strategies within a community of clinical practice.

METHODS

We conducted a qualitative phenomenological study from May 2008 to December 2009. We held 37 semi-structured individual interviews with three different cohorts of undergraduate medical students until we reached data saturation. The cohorts were exposed to different contexts while experiencing their clinical years' assessment program. In the interviews, students' perceptions and interpretations of 'self-assessment practice' and 'supervisor-provided feedback' within different contexts and the resulting study strategies were explored.

RESULTS

The analysis of interview data with the three cohorts of students yielded three major themes: strategic practice of self-assessment, self-assessment and study strategies, and feedback and study strategies. It appears that self-assessment is not appropriate within a summative context, and its implementation requires cultural preparation. Despite education and orientation on the two major components of the self-assessment process, feedback was more effective in enhancing deeper study strategies.

CONCLUSION

This research suggests that the theoretical advantages linked to the self-assessment process are a result of its feedback component rather than the practice of self-assessment isolated from feedback. Further research exploring the effects of different contextual and personal factors on students' self-assessment is needed.

Use of individualized learning plans among fourth-year sub-interns in pediatrics and internal medicine

BACKGROUND

Individualized Learning Plans (ILPs) are an effective tool for promoting self-directed learning among residents. However, no literature details ILP use among medical students.

METHODS

Fifty fourth-year sub-interns in pediatrics and internal medicine created ILPs, including a self-assessment of strengths and weaknesses based on ACGME core competencies and the setting of learning objectives. During weekly follow-up meetings with faculty mentors and peers, students discussed challenges and revised goals. Upon completion of the rotation, students completed a survey of Likert-scale questions addressing satisfaction with and perceived utility of ILP components.

RESULTS

Students most often self-identified strengths in the areas of Professionalism and Interpersonal and Communication Skills and weaknesses in Patient Care and Systems-Based Practice. Eighty-two percent set at least one learning objective in an identified area of weakness. Students expressed high confidence in their abilities to create achievable learning objectives and to generate strategies to meet those objectives. Students agreed that discussions during group meetings were meaningful, and they identified the setting learning objectives and weekly meetings as the most important elements of the exercise.

CONCLUSIONS

Fourth-year sub-interns reported that ILPs helped them to accomplish rotation goals, with the setting of learning objectives and weekly discussions being the most useful elements.

Critically reflective work behavior of health care professionals

INTRODUCTION

Better understanding of critically reflective work behavior (CRWB), an approach for work-related informal learning, is important in order to gain more profound insight in the continuing development of health care professionals.

METHODS

A survey, developed to measure CRWB and its predictors, was distributed to veterinary professionals. The authors specified a model relating CRWB to a Perceived Need for Lifelong Learning, Perceived Workload, and Opportunities for Feedback. Furthermore, research utilization was added to the concept of CRWB. The model was tested against the data, using structural equation modeling (SEM).

RESULTS

The model was well represented by the data. Four factors that reflect aspects of CRWB were distinguished: (1) individual CRWB; (2) being critical in interactions with others; (3) cross-checking of information; and (4) openness to new findings. The latter 2 originated from the factor research utilization in CRWB. The Perceived Need for Lifelong Learning predicts CRWB. Neither Perceived Workload nor Opportunities for Feedback of other practitioners was related to CRWB.

DISCUSSION

The results suggest that research utilization, such as cross-checking information and openness to new findings, is essential for CRWB. Furthermore, perceptions of the need for lifelong learning are more relevant for CRWB of health care professionals than qualities of the workplace.

Physicians' response to computerised alerts for psychotropic drugs in older persons: a multilevel analysis of the associated alert, patient and physician characteristics

OBJECTIVE

Computerised drug alerts are expected to reduce patients' risk of adverse drug events. However, physicians over-ride most drug alerts, because they believe that the benefit exceeds the risk. The purpose of this study was to determine the drug alert, patient and physician characteristics associated with the: (1) occurrence of psychotropic drug alerts for elderly patients and the (2) response to these alerts by their primary care physicians.

SETTING

Primary care, Quebec, Canada.

DESIGN

Prospective cohort study.

PARTICIPANTS

Sixty-one physicians using an electronic prescribing and drug alert decision-support system in their practice, and 3413 elderly patients using psychotropic drugs. PRIMARY AND SECONDARY MEASURES: Psychotropic drug class, alert severity, patient risk for fall injuries and physician experience, practice volume and computer use were evaluated in relationship to the likelihood of having: (1) a psychotropic drug alert, (2) the prescription revised in response to an alert. Cluster-adjusted alternating logistic regression was used to assess multilevel predictors of alert occurrence and response.

RESULTS

In total 13 080 psychotropic drug alerts were generated in 8931 visits. Alerts were more likely to be generated for male patients at higher risk of fall-related injury and for physicians who established the highest alert threshold. In 9.9% of alerts seen, the prescription was revised. The highest revision rate was for antipsychotic alerts (22.6%). Physicians were more likely to revise prescriptions for severe alerts (OR 2.03; 95%CI 1.39 to 2.98), if patients had cognitive impairment (OR 1.95; 95%CI 1.13 to 3.36), and if they made more visits to their physician (OR 1.05 per 5 visits; 95%CI 1 to 1.09).

CONCLUSIONS

Physicians view and respond to a small proportion of alerts, mainly for higher-risk patients. To reduce the risk of psychotropic drug-related fall injuries, a new generation of evidence-based drug alerts should be developed.

Review article: simulation in anesthesia: state of the science and looking forward

PURPOSE

Within the field of anesthesia, simulation has been used as a tool for training and assessment for over 30 years. The purpose of this review is to evaluate the state of the science in terms of its effectiveness as an approach to both training and assessment in anesthesia. Articles in the area of simulation and anesthesia published up to and including 2011 were reviewed for inclusion in this narrative review.

PRINCIPAL FINDINGS

Simulation-based training is generally well received by participants, it can lead to improved performance in subsequent simulation events, and some transfer of learning to the clinical setting is evident. There is also some early evidence that well-designed performance assessments could have the required reliability and validity to support high-stakes examinations. However, further work is needed in order to set standards and establish the predictive validity to support such assessments.

CONCLUSION

For simulation to realize its potential impact, further research is needed to understand how to optimize this modality of learning more effectively, how to transfer knowledge of research findings to practice, and also how to broaden the simulation modalities used in anesthesia. In future, the optimal use of simulation will depend on a clear understanding of what can and cannot be accomplished with simulation and its various modalities.

Steps to pluripotent learning: provocative teaching

Education involves interactions between students and teachers in a societal framework. Teachers can best serve their students and society by making students flexible enough to thrive under uncertain conditions. They should, in a sense, nourish, nurture, provoke, and stimulate pluripotent "educatoblasts."

An active-learning strategies primer for achieving ability-based educational outcomes

Active learning is an important component of pharmacy education. By engaging students in the learning process, they are better able to apply the knowledge they gain. This paper describes evidence supporting the use of active-learning strategies in pharmacy education and also offers strategies for implementing active learning in pharmacy curricula in the classroom and during pharmacy practice experiences.

Feedback in the emergency medicine clerkship

OBJECTIVE

Feedback is a technique used in medical education to help develop and improve clinical skills. A comprehensive review article specifically intended for the emergency medicine (EM) educator is lacking, and it is the intent of this article to provide the reader with an in-depth, up-to-date, and evidence-based review of feedback in the context of the EM clerkship.

METHODS

The review article is organized in a progressive manner, beginning with the definition of feedback, the importance of feedback in medical education, the obstacles limiting the effective delivery of feedback, and the techniques to overcome these obstacles then follows. The article concludes with practical recommendations to implement feedback in the EM clerkship. To advance the literature on feedback, the concept of receiving feedback is introduced.

RESULTS

The published literature regarding feedback is limited but generally supportive of its importance and effectiveness. Obstacles in the way of feedback include time constraints, lack of direct observation, and fear of negative emotional responses from students. Feedback should be timely, expected, focused, based on first-hand data, and limited to behaviors that are remediable. Faculty development and course structure can improve feedback in the EM clerkship. Teaching students to receive feedback is a novel educational technique that can improve the feedback process.

CONCLUSION

Feedback is an important educational technique necessary to improve clinical skills. Feedback can be delivered effectively in the EM clerkship.

Role-modelling in the operating room: medical student observations of exemplary behaviour

CONTEXT

Training future doctors to develop an appropriate professional persona is an important goal of medical student education and residency training. Most medical education research paradigms on professionalism have focused largely on lapses (e.g. yelling as an example of communication failure) and tend to emphasise behaviour that should be avoided. The assumption is that, if left unchecked, students will see these negative behaviours exhibited by their role models and possibly emulate them, allowing the potential reinforcement of the inappropriate behaviours.

OBJECTIVES

Identifying and characterising exemplary, or positive, behaviours can be similarly valuable to both medical students and residents as tangible examples of behaviours to strive towards. The goal of the present research was to determine and thematically define the exemplary professional actions that medical students observe in the intense and patient-focused environment of the operating room (OR).

METHODS

Using qualitative methodology of content analysis and theme identification, we systematically documented the type of exemplary professional behaviours reported by medical students (n=263) when observing health care teams on an anaesthesia rotation in the OR.

RESULTS

The analysis generated a taxonomy of exemplary OR behaviour that included six overarching themes (e.g. teamwork), 15 sub-themes (e.g. collegial) and numerous exemplars (e.g. showed mutual respect). These themes and sub-themes were then conceptually 'matched'--through the use of antonyms --to complement an existing framework focused on medical student reports of professional lapses witnessed during medical school.

CONCLUSIONS

Year 3 medical students in the USA reported observing very positive, exemplary health care provider interactions that were diverse in focus. Themes were identified regarding the OR team members' interactions with patients (calm, communication, comforting), with one another (teamwork, respect) and with the medical students (teaching). This classification of exemplary OR behaviour contributes to our understanding of how professional behaviour is viewed and potentially emulated by medical students on surgical rotations.

Tensions in informed self-assessment: how the desire for feedback and reticence to collect and use it can conflict

PURPOSE

Informed self-assessment describes the set of processes through which individuals use external and internal data to generate an appraisal of their own abilities. The purpose of this project was to explore the tensions described by learners and professionals when informing their self-assessments of clinical performance.

METHOD

This 2008 qualitative study was guided by principles of grounded theory. Eight programs in five countries across undergraduate, postgraduate, and continuing medical education were purposively sampled. Seventeen focus groups were held (134 participants). Detailed analyses were conducted iteratively to understand themes and relationships.

RESULTS

Participants experienced multiple tensions in informed self-assessment. Three categories of tensions emerged: within people (e.g., wanting feedback, yet fearing disconfirming feedback), between people (e.g., providing genuine feedback yet wanting to preserve relationships), and in the learning/practice environment (e.g., engaging in authentic self-assessment activities versus "playing the evaluation game"). Tensions were ongoing, contextual, and dynamic; they prevailed across participant groups, infusing all components of informed self-assessment. They also were present in varied contexts and at all levels of learners and practicing physicians.

CONCLUSIONS

Multiple tensions, requiring ongoing negotiation and renegotiation, are inherent in informed self-assessment. Tensions are both intraindividual and interindividual and they are culturally situated, reflecting both professional and institutional influences. Social learning theories (social cognitive theory) and sociocultural theories of learning (situated learning and communities of practice) may inform our understanding and interpretation of the study findings. The findings suggest that educational interventions should be directed at individual, collective, and institutional cultural levels. Implications for practice are presented.

Simulation-based emergency medicine resident self-assessment

BACKGROUND

Videotaped recordings of simulation-based performance may allow learners the opportunity to review, evaluate, and reflect upon their own performance.

OBJECTIVES

To determine the accuracy of resident performance self-assessment after a simulation-based encounter; compare low- and high-scoring residents' abilities to evaluate their performance; and determine if video-assisted performance review improves self-assessment accuracy.

METHODS

Emergency Medicine residents participated in a videotaped simulation-based assessment. Residents evaluated their performance immediately after completing simulated cases, and after reviewing the session's video. Self-ratings were compared to the faculty observers, and scores were divided based on the median.

RESULTS

Seventeen residents participated, providing 270 self-ratings before, and 269 after, video review. Before video review, residents accurately graded their performance in 73.7% of the items. High- and low-scoring residents accurately self-assessed 83.9% and 62.2% of items, respectively. The odds of a high scorer accurately rating their own performance were 3.2 times that of a low scorer (95% confidence interval [CI] 1.9-5.2, p<0.001). After video review, resident self-assessments were accurate for 73.6% of the items. High scorers were accurate in their post-video self-assessment in 83.3% of the items, vs. 62.4% for low scorers. After video review, the odds of a high scorer accurately self-rating their performance were 3.0 times that of a low scorer (95% CI 2.1-4.1, p<0.001).

CONCLUSIONS

Residents' abilities to self-assess vary, and performance quality may influence self-assessment. Video review did not significantly increase self-assessment accuracy. Improving self-assessment skills may assist residents in identifying practice gaps, thereby allowing them to focus their energy toward filling that gap.

Measuring the development of insight by dental health professionals in training using workplace-based assessment

INTRODUCTION

For health professionals, the development of insight into their performance is vital for safe practice, professional development and self-regulation. This study investigates whether the development of dental trainees' insight, when provided with external feedback on performance, can be assessed using a single criterion on a simple global ratings form such as the Longitudinal Evaluation of Performance or Mini Clinical Evaluation Exercise.

METHODS

Postgraduate dental trainees (N = 139) were assessed using this tool on a weekly basis for 6 months. Regression analysis of the data was carried out using SPSS, and a short trainer questionnaire was implemented to investigate feasibility.

RESULTS

Ratings for insight were shown to increase with time in a similar manner to the growth observed in other essential skills. The gradient of the slope for growth of insight was slightly less than that of the other observed skills. Trainers were mostly positive about the new criterion assessing trainees' insight, although the importance of training for trainers in this process was highlighted.

DISCUSSION

Our data suggest that practitioners' insight into their performance can be developed with experience and regular feedback. However, this is most likely a complex skill dependent on a number of intrinsic and external factors.

CONCLUSION

The development of trainees' insight into their performance can be assessed using a single criterion on a simple global ratings form. The process involves no additional burden on evaluators in terms of their time or cost, and promotes best practice in the provision of feedback for trainees.

Features of assessment learners use to make informed self-assessments of clinical performance

CONTEXT

Conceptualisations of self-assessment are changing as its role in professional development comes to be viewed more broadly as needing to be both externally and internally informed through activities that enable access to and the interpretation and integration of data from external sources. Education programmes use various activities to promote learners' reflection and self-direction, yet we know little about how effective these activities are in 'informing' learners' self-assessments.

OBJECTIVES

This study aimed to increase understanding of the specific ways in which undergraduate and postgraduate learners used learning and assessment activities to inform self-assessments of their clinical performance.

METHODS

We conducted an international qualitative study using focus groups and drawing on principles of grounded theory. We recruited volunteer participants from three undergraduate and two postgraduate programmes using structured self-assessment activities (e.g. portfolios). We asked learners to describe their perceptions of and experiences with formal and informal activities intended to inform self-assessment. We conducted analysis as a team using a constant comparative process.

RESULTS

Eighty-five learners (53 undergraduate, 32 postgraduate) participated in 10 focus groups. Two main findings emerged. Firstly, the perceived effectiveness of formal and informal assessment activities in informing self-assessment appeared to be both person- and context-specific. No curricular activities were considered to be generally effective or ineffective. However, the availability of high-quality performance data and standards was thought to increase the effectiveness of an activity in informing self-assessment. Secondly, the fostering and informing of self-assessment was believed to require credible and engaged supervisors.

CONCLUSIONS

Several contextual and personal conditions consistently influenced learners' perceptions of the extent to which assessment activities were useful in informing self-assessments of performance. Although learners are not guaranteed to be accurate in their perceptions of which factors influence their efforts to improve performance, their perceptions must be taken into account; assessment strategies that are perceived as providing untrustworthy information can be anticipated to have negligible impact.

Lessons from medical students' perceptions of learning reflective skills: a multi-institutional study

BACKGROUND

A core competency during undergraduate medical training is the development of reflective learning. The current literature is limited to demonstrating how reflective learning has been implemented or the approaches to its development. There is a lack of insight into students' perceptions of reflection and the factors that support development of reflective practice. Bridging this gap may provide insight into how reflective learning within the curriculum can be better developed to increase engagement from learners.

METHODS

Eight focus group interviews with second year students from four UK medical schools were held. Results were thematically analysed.

KEY FINDINGS

Students have a high level of understanding of the purpose of reflection in practice but they perceive that there is a tension between public and private reflections. Assessment of the reflective process was perceived to be useful for developing reflective skills but grading of their reflective writing was not considered to be useful. Staff who champion the development of reflective skills and mentor students were perceived to play key roles in aiding the development of reflective skills. Appropriate experiences were seen to be a key part of developing reflective skills.

CONCLUSION

These findings highlight potential ways to revise and improve engagement with the reflective learning components of undergraduate courses.

Medical and psychology students' self-assessed communication skills: A pilot study

OBJECTIVE

The purpose of this study was to describe how psychology and medical students assess their own competency and skills before and after training, in which role-play was used to teach interpersonal and communication skills.

METHOD

Interpersonal and communication skills were assessed with a semi-structured questionnaire before and after the training.

RESULTS

The students of both medicine and psychology estimated their skill levels to be higher after the course. The psychology students estimated their skills for communication, motivating interviewing, empathy and reflection, and change orientation to be better at the end of the course. Medical students estimated their communication skills, motivating interviewing skills, and change orientation skills to be better at the end of the course.

CONCLUSION

Even a short period of training in interpersonal and communication skills can positively affect the self-assessed skills of the medical students.

PRACTICE IMPLICATIONS

In the future, it would be worthwhile to pay attention to reflective teaching practices in the training of both medical and psychology students. The cognitive and emotional components of these practices help students to develop their own communication skills.

Self-assessment or self deception? A lack of association between nursing students' self-assessment and performance

AIM

The aim of this study was to examine senior year nursing students' ability to self-assess their performance when responding to simulated emergency situations.

BACKGROUND

Self-assessment is viewed as a critical skill in nursing and other health professional programmes. However, while students may spend considerable time completing self-assessments, there is little evidence that they actually acquire the skills to do so effectively. By contrast, a number of studies in medicine and elsewhere have cast doubt on the validity of self-assessment.

METHOD

In 2007, a one-group pre-test, post-test design was used to answer the question, 'How accurate are senior year nursing students in assessing their ability to respond to emergency situations in a simulated medical/surgical environment compared to observer assessment of their performance?' A total of 27 fourth year nursing students from a university in Ontario were asked to complete a questionnaire before and after an objective structured clinical examination which assessed their ability to respond to emergency situations. Self-assessments were compared with observed performance.

FINDINGS

The experience of dealing with the simulated crisis situations significantly increased perceived confidence and perceived competence in dealing with emergency situations, although it did not affect self-perceived ability to communicate or collaborate. All but 1 of the 16 correlations between self-assessment and the objective structured clinical examination total scores were negative. Their self-assessment was also unrelated to several indices of experience in critical care settings.

CONCLUSION

Self-assessment in nursing education to evaluate clinical competence and confidence requires serious reconsideration as our well-intentioned emphasis on this commonly used practice may be less than effective.

Faculty development in assessment: the missing link in competency-based medical education

As the medical education community celebrates the 100th anniversary of the seminal Flexner Report, medical education is once again experiencing significant pressure to transform. Multiple reports from many of medicine's specialties and external stakeholders highlight the inadequacies of current training models to prepare a physician workforce to meet the needs of an increasingly diverse and aging population. This transformation, driven by competency-based medical education (CBME) principles that emphasize the outcomes, will require more effective evaluation and feedback by faculty.Substantial evidence suggests, however, that current faculty are insufficiently prepared for this task across both the traditional competencies of medical knowledge, clinical skills, and professionalism and the newer competencies of evidence-based practice, quality improvement, interdisciplinary teamwork, and systems. The implication of these observations is that the medical education enterprise urgently needs an international initiative of faculty development around CBME and assessment. In this article, the authors outline the current challenges and provide suggestions on where faculty development efforts should be focused and how such an initiative might be accomplished. The public, patients, and trainees need the medical education enterprise to improve training and outcomes now.

Exploring deliberate practice in medicine: how do physicians learn in the workplace?

Medical professionals need to keep on learning as part of their everyday work to deliver high-quality health care. Although the importance of physicians' learning is widely recognized, few studies have investigated how they learn in the workplace. Based on insights from deliberate practice research, this study examined the activities physicians engage in during their work that might further their professional development. As deliberate practice requires a focused effort to improve performance, the study also examined the goals underlying this behaviour. Semi-structured interviews were conducted with 50 internal medicine physicians: 19 residents, 18 internists working at a university hospital, and 13 working at a non-university hospital. The results showed that learning in medical practice was very much embedded in clinical work. Most relevant learning activities were directly related to patient care rather than motivated by competence improvement goals. Advice and feedback were sought when necessary to provide this care. Performance standards were tied to patients' conditions. The patients encountered and the discussions with colleagues about patients were valued most for professional development, while teaching and updating activities were also valued in this respect. In conclusion, physicians' learning is largely guided by practical experience rather than deliberately sought. When professionals interact in diagnosing and treating patients to achieve high-quality care, their experiences contribute to expertise development. However, much could be gained from managing learning opportunities more explicitly. We offer suggestions for increasing the focus on learning in medical practice and further research.

Pediatric intensive care simulation course: a new paradigm in teaching

OBJECTIVE

True pediatric emergencies are rare. Because resident work hours are restricted and national attention turns toward patient safety, teaching methods to improve physician performance and patient care are vital. We hypothesize that a critical-care simulation course will improve resident confidence and performance in critical-care situations.

INTERVENTIONS

We developed a monthly pediatric intensive care unit simulation course for second-year pediatric residents that consisted of weekly 1-hour sessions during both of the residents' month-long pediatric intensive care unit rotations. All scenarios used high-fidelity pediatric simulators and immediate videotape-assisted debriefing sessions. In addition, simulated intraosseous line insertion and endotracheal intubations were also performed.

RESULTS

All residents improved their comfort level and confidence in performing individual key resuscitation tasks. The largest improvements were seen with their perceived ability to intubate children and place intraosseous lines. Both of these skills improved from baseline and compared to third-year-resident controls who had pediatric intensive care unit rotations but no simulations (P = .05 and P = .07, respectively). Videotape reviews showed only 54% ± 12% of skills from a scenario checklist performed correctly.

CONCLUSIONS

Our simulation-based pediatric intensive care unit training course improves second-year pediatric residents' comfort level but not performance during codes, as well as their perceived intubation and intraosseous ability. Videotape reviews show discordance between objective performance and self-assessment. Further work is necessary to elucidate the reasons for this difference as well as the appropriate role for simulation in the new graduate medical education climate, and to create new teaching modalities to improve resident performance.

Perspective: Competencies, outcomes, and controversy--linking professional activities to competencies to improve resident education and practice

Regulatory organizations have recently emphasized the importance of structuring graduate medical education around mastery of core competencies. The difficulty is that core competencies attempt to distill a range of professional behaviors into arguable abstractions. As such, competencies can be difficult to grasp for trainees and faculty, who see them as unrelated to the intricacies of daily patient care. In this article, the authors describe how two initiatives are converging in a way that should make competencies tangible and relevant. One initiative is based on the idea that competencies will be more meaningful if trainees understand specifically how they relate to important professional activities in their own specialty. The authors suggest that there is a dyadic relationship between competencies and major professional activities in pediatric medicine. They also suggest that these relationships should be discussed as part of the process by which trainees are entrusted to perform clinical activities without direct supervision. The other initiative proposes to construct narrative milestones that provide a picture of what progression toward mastery of core competencies might look like. Together, the authors argue, these two initiatives should illuminate the core competencies by providing relevant clinical context and valuable educational substance.

The rotational approach to medical education: time to confront our assumptions?

CONTEXT

Trainees in undergraduate and postgraduate medical education engage in multiple transitions as part of the educational process, including many transitions that occur on both periodic and daily bases within medical education programmes. The clinical rotation, based on either a medical discipline or clinical care setting and occurring over a predetermined, short period of time, is a deeply entrenched educational approach with its roots in Abraham Flexner's seminal report. Many assumptions about the presumed benefits of clinical rotations have become pervasive despite a lack of empirical evidence on their optimal timing and structure, and on how transitions between clinical rotations should occur.

METHODS

In this paper, we examine the issue of rotational transitions from the three perspectives of sociology, learning theory, and the improvement of quality and safety.

RESULTS

Discussion from the sociological perspective addresses the need for much greater attention to interprofessional relationships and professional development, whereas that from the learning theory perspective examines the gap between what is known from pedagogical and cognitive science and what is currently practised (learning theory). Discussion from the perspective of improving quality and safety refers to the critical need to embed trainees in functional clinical microsystems as meaningful participants.

CONCLUSIONS

Research is urgently needed on the effects of transitions on trainees, faculty staff, non-doctor health care providers and patients in order to optimise future competency-based training models and confirm or refute current assumptions.

Feedback data sources that inform physician self-assessment

BACKGROUND

Self-assessment is a process of interpreting data about one's performance and comparing it to explicit or implicit standards.

AIM

To examine the external data sources physicians used to monitor themselves.

METHODS

Focus groups were conducted with physicians who participated in three practice improvement activities: a multisource feedback program; a program providing patient and chart audit data; and practice-based learning groups. We used grounded theory strategies to understand the external sources that stimulated self-assessment and how they worked.

RESULTS

Data from seven focus groups (49 physicians) were analyzed. Physicians used information from structured programs, other educational activities, professional colleagues, and patients. Data were of varying quality, often from non-formal sources with implicit (not explicit) standards. Mandatory programs elicited variable responses, whereas data and activities the physicians selected themselves were more likely to be accepted. Physicians used the information to create a reference point against which they could weigh their performance using it variably depending on their personal interpretation of its accuracy, application, and utility.

CONCLUSIONS

Physicians use and interpret data and standards of varying quality to inform self-assessment. Physicians may benefit from regular and routine feedback and guidance on how to seek out data for self-assessment.

Twelve tips for teaching expertise in clinical reasoning

BACKGROUND

Clinical reasoning is one of the most critical skills to teach to medical learners, yet clinician educators rarely receive adequate training on how to teach this topic.

AIMS

To enhance clinician educators' ability to teach clinical reasoning.

METHODS

I conducted a review of cognitive, medical decision making, and expertise theory literature to develop practical tips that could be applied to typical teaching encounters.

RESULTS

Through the literature review, twelve tips were designed to provide a blueprint for teaching clinical reasoning on the wards or in the clinics.

CONCLUSIONS

Teaching clinical reasoning is important and feasible. Teachers who explicitly teach problem solving and decision making may help learners to improve their diagnostic accuracy and treatment choices.

How to become a better clinical teacher: a collaborative peer observation process

BACKGROUND

Peer observation of teaching (PoT) is most commonly done as a way of evaluating educators in lecture or small group teaching. Teaching in the clinical environment is a complex and hectic endeavor that requires nimble and innovative teaching on a daily basis. Most junior faculty start their careers with little formal training in education and with limited opportunity to be observed or to observe more experienced faculty.

AIM

Formal PoT would potentially ameliorate these challenges.

METHODS

This article describes a collaborative peer observation process that a group of 11 clinician educators is using as a longitudinal faculty development program.

RESULTS

The process described in this article provides detailed and specific teaching feedback for the observed teaching attending while prompting the observing faculty to reflect on their own teaching style and to borrow effective teaching techniques from the observation.

CONCLUSION

This article provides detailed examples from written feedback obtained during collaborative peer observation to emphasize the richness of this combined experience.

Twelve tips for implementing tools for direct observation of medical trainees' clinical skills during patient encounters

BACKGROUND

Direct observation of medical trainees by their supervisors with actual patients is essential for trainees to develop clinical skills competence. Despite the many available tools for direct observation of trainees by supervisors, it is unclear how educators should identify an appropriate tool for a particular clinical setting and implement the tool to maximize educational benefits for trainees in a manner that is feasible for faculty.

AIMS AND METHODS

Based on our previous systematic review of the literature, we provide 12 tips for selecting and incorporating a tool for direct observation into a medical training program. We focus specifically on direct observation that occurs in clinical settings with actual patients.

RESULTS

Educators should focus on the existing tools for direct observation that have evidence of validity. Tool implementation must be a component of an educational program that includes faculty development about rating performance, providing meaningful feedback, and developing action plans collaboratively with learners.

CONCLUSIONS

Educators can enhance clinical skills education with strategic incorporation of tools for direct observation into medical training programs. Identification of a psychometrically sound instrument and attention to faculty development and the feedback process are critical to the success of a program of direct observation.

Making a difference in curriculum reform and decision-making processes

CONTEXT

Although firmly grounded in Flexner's legacy of ideas, today's medical curriculum, as both an entity and a process, has become more and more complex. The curriculum as an entity is portrayed according to five key elements: the expected competencies and roles; the learners at the centre of the enterprise; assessment linking competencies and learners; the conditions and resources for learning; and a multifaceted socio-politico-cultural context in which the learning occurs. Significant developments have also occurred in the disciplines of curriculum studies, cognitive psychology and organisational change over the past century, as well as in institutional best practices, that help us to better understand and plan curricular innovations.

DISCUSSION

Practical advice is offered to help curriculum developers in designing or reforming the medical curriculum. The key points of this are: (i) while focusing reform and innovation on specific elements of the curriculum, consider how those elements affect other elements and vice versa, in positive and negative ways; (ii) while grounding the reform or innovation in sound conceptual frameworks, seize any opportunities to formulate a research agenda that can build upon and advance our understanding of curricular innovations, and, (iii) moving beyond considering the curriculum as an entity, use deliberative and leadership processes that can lead to enduring curriculum reform.

What are the perceived learning needs of Australian general practice registrars for quality prescribing?

BACKGROUND

Little is known about the perceived learning needs of Australian general practice (GP) registrars in relation to the quality use of medicines (QUM) or the difficulties experienced when learning to prescribe. This study aimed to address this gap.

METHODS

GP registrars' perceived learning needs were investigated through an online national survey, interviews and focus groups. Medical educators' perceptions were canvassed in semi-structured interviews in order to gain a broader perspective of the registrars' needs. Qualitative data analysis was informed by a systematic framework method involving a number of stages. Survey data were analysed descriptively.

RESULTS

The two most commonly attended QUM educational activities took place in the workplace and through regional training providers. Outside of these structured educational activities, registrars learned to prescribe mainly through social and situated means. Difficulties encountered by GP registrars included the transition from hospital prescribing to prescribing in the GP context, judging how well they were prescribing and identifying appropriate and efficient sources of information at the point of care.

CONCLUSIONS

GP registrars learn to prescribe primarily and opportunistically in the workplace. Despite many resources being expended on the provision of guidelines, decision-support systems and training, GP registrars expressed difficulties related to QUM. Ways of easing the transition into GP and of managing the information 'overload' related to medicines (and prescribing) in an evidence-guided, efficient and timely manner are needed. GP registrars should be provided with explicit feedback about the process and outcomes of prescribing decisions, including the use of audits, in order to improve their ability to judge their own prescribing.

Which factors, personal or external, most influence students' generation of learning goals?

BACKGROUND

While concern has been expressed about the validity of self-assessments, external feedback is likely filtered through self-assessment. This paper explores the relationship between self-assessments and feedback uptake.

METHOD

During an objective structured clinical examination, students were asked to evaluate their performance and rate the quality of feedback provided by observers. Afterward, they were asked to list learning goals they generated, to indicate what activities they would undertake to fulfill those goals, and to identify which station(s) led them to generate each response. Regression analyses were used to determine which variables predicted the generation of goals/activities.

RESULTS

Students' perceptions of their own performance were more likely to result in the generation of goals/strategies than was observer feedback or student perceptions of observer feedback quality. Later stations were more likely to result in goal/strategy generation than earlier stations.

CONCLUSIONS

While self-assessments may not validly indicate ability, it is still critical to determine how students perceive their ability because their opinions drive their learning goals.

Twelve tips for teaching reflection at all levels of medical education

BACKGROUND

Review of studies published in medical education journals over the last decade reveals a diversity of pedagogical approaches and educational goals related to teaching reflection.

AIM

The following tips outline an approach to the design, implementation, and evaluation of reflection in medical education.

METHOD

The method is based on the available literature and the author's experience. They are organized in the sequence that an educator might use in developing a reflective activity.

RESULTS

The 12 tips provide guidance from conceptualization and structure of the reflective exercise to implementation and feedback and assessment. The final tip relates to the development of the faculty member's own reflective ability.

CONCLUSION

With a better understanding of the conceptual frameworks underlying critical reflection and greater advance planning, medical educators will be able to create exercises and longitudinal curricula that not only enable greater learning from the experience being reflected upon but also develop reflective skills for life-long learning.