The assessment of professional competence: building blocks for theory development

Medical education research: a vibrant community of research and education practice

OBJECTIVES

Medical education research is thriving. In recent decades, numbers of journals and publications have increased enormously, as have the number and size of medical education meetings around the world. The aim of this paper is to shed some light on the origins of this success. My central argument is that dialogue between education practice (and its teachers) and education research (and its researchers) is indispensable.

REFLECTIONS

To illustrate how I have come to this perspective, I discuss two crucial developments of personal import to myself. The first is the development of assessment theory informed by both research findings and insights emerging from implementations conducted in collaboration with teachers and learners. The second is the establishment of a department of education that includes many members from the medical domain.

CONCLUSIONS

Medical education is thriving because it is shaped and nourished within a community of practice of collaborating teachers, practitioners and researchers. This obviates the threat of a fissure between education research and education practice. The values of this community of practice - inclusiveness, openness, supportiveness, nurture and mentorship - are key elements for its sustainability. In pacing the development of our research in a manner that maintains this synergy, we should be mindful of the zone of proximal development of our community of practice.

Are recent graduates enough prepared to perform obstetric skills in their rural and compulsory year? A study from Ecuador

OBJECTIVES

The aim of this study was to assess the possible mismatch of obstetrical skills between the training offered in Ecuadorian medical schools and the tasks required for compulsory rural service.

SETTING

Primary care, rural health centres in Southern Ecuador.

PARTICIPANTS

A total of 92 recent graduated medical doctors during their compulsory rural year.

PRIMARY AND SECONDARY OUTCOMES MEASURES

A web-based survey was developed with 21 obstetrical skills. The questionnaire was sent to all rural doctors who work in Loja province, Southern Ecuador, at the Ministry of Health (n=92).

WE MEASURED TWO CATEGORIES

'importance of skills in rural practice' with a five-point Likert-type scale (1= strongly disagree; 5= strongly agree); and 'clerkship experience' using a nominal scale divided in five levels: level 1 (not seen, not performed) to level 5 (performed 10 times or more). Spearman's rank correlation coefficient (r) was used to observe associations.

RESULTS

A negative correlation was found in the skills: 'episiotomy and repair', 'umbilical vein catheterisation', 'speculum examination', 'evaluation of cervical dilation during active labour', 'neonatal resuscitation' and 'vacuum-assisted vaginal delivery'. For instance 'Episiotomy and repair' is important (strongly agree and agree) to 100% of respondents, but in practice, only 38.9% of rural doctors performed the task three times and 8.3% only once during the internship, similar pattern is seen in the others.

CONCLUSIONS

In this study we have noted the gap between the medical needs of populations in rural areas and training provided during the clerkship experiences of physicians during their rural service year. It is imperative to ensure that rural doctors are appropriately trained and skilled in the performance of routine obstetrical duties. This will help to decrease perinatal morbidity and mortality in rural Ecuador.

A Generalizability Theory Study of Athletic Taping Using the Technical Skill Assessment Instrument

Context:

Athletic taping skills are highly valued clinical competencies in the athletic therapy and training profession. The Technical Skill Assessment Instrument (TSAI) has been content validated and tested for intrarater reliability.

Objective:

To test the reliability of the TSAI using a more robust measure of reliability, generalizability theory, and to hypothetically and mathematically project the optimal number of raters and scenarios to reliably measure athletic taping skills in the future.

Setting:

Mount Royal University.

Design:

Observational study.

Patients or Other Participants:

A total of 29 university students (8 men, 21 women; age = 20.79 ± 1.59 years) from the Athletic Therapy Program at Mount Royal University.

Intervention(s):

Participants were allowed 10 minutes per scenario to complete prophylactic taping for a standardized patient presenting with (1) a 4-week-old second-degree ankle sprain and (2) a thumb that had been hyperextended. Two raters judged student performance using the TSAI.

Main Outcome Measure(s):

Generalizability coefficients were calculated using variance scores for raters, participants, and scenarios. A decision study was calculated to project the optimal number of raters and scenarios to achieve acceptable levels of reliability. Generalizability coefficients were interpreted the same as other reliability coefficients, with 0 indicating no reliability and 1.0 indicating perfect reliability.

Results:

The result of our study design (2 raters, 1 standardized patient, 2 scenarios) was a generalizability coefficient of 0.67. Decision study projects indicated that 4 scenarios were necessary to reliably measure athletic taping skills.

Conclusions:

We found moderate reliability coefficients. Researchers should include more scenarios to reliably measure athletic taping skills. They should also focus on the development of evidence-based practice guidelines and standards of athletic taping and should test those standards using a psychometrically sound instrument, such as the TSAI.

What would happen to education if we take education evidence seriously?

Educational practice and educational research are not aligned with each other. Current educational practice heavily relies on information transmission or content delivery to learners. Yet evidence shows that delivery is only a minor part of learning. To illustrate the directions we might take to find better educational strategies, six areas of educational evidence are briefly reviewed. The flipped classroom idea is proposed to shift our expenditure and focus in education. All information delivery could be web distributed, thus creating more time for other more expensive educational strategies to support the learner. In research our focus should shift from comparing one curriculum to the other, to research that explains why things work in education and under which conditions. This may generate ideas for creative designers to develop new educational strategies. These best practices should be shared and further researched. At the same time attention should be paid to implementation and the realization that teachers learn in a way very similar to the people they teach. If we take the evidence seriously, our educational practice will look quite different to the way it does now.

Factors influencing the effectiveness of multisource feedback in improving the professional practice of medical doctors: a systematic review

BACKGROUND

Multisource feedback (MSF) is currently being introduced in the UK as part of a cycle of performance review for doctors. However, although it is suggested that the provision of feedback can lead to a positive change in performance and learning for medical professionals, the evidence supporting these assumptions is unclear. The aim of this review, therefore, was to identify the key factors that influence the effectiveness of multisource feedback in improving the professional practice of medical doctors.

METHOD

Relevant electronic bibliographic databases were searched for studies that aimed to assess the impact of MSF on professional practice. Two reviewers independently selected and quality assessed the studies and abstracted data regarding study design, setting, MSF instrument, behaviour changes identified and influencing factors using a standard data extraction form.

RESULTS

A total of 16 studies met the inclusion criteria and quality assessment criteria. While seven studies reported only a general change in professional practice, a further seven studies identified specific changes in behaviour. The main professional behaviours that were found to be influenced by the feedback were communication, both with colleagues and patients and an improvement in clinical competence/skills. The main factors found to influence the acceptance and use of MSF were the format of the feedback, specifically in terms of whether it was facilitated, or if narrative comments were included in the review, and if the feedback was from sources that the physician believed to be knowledgeable and credible.

CONCLUSIONS

While there is limited evidence suggesting that MSF can influence professional performance, the quality of this evidence is variable. Further research is necessary to establish how this type of feedback actually influences behaviours and what factors have greatest influence.

Residents' perceived barriers to communication skills learning: comparing two medical working contexts in postgraduate training

OBJECTIVE

Contextual factors are known to influence the acquisition and application of communication skills in clinical settings. Little is known about residents' perceptions of these factors. This article aims to explore residents' perceptions of contextual factors affecting the acquisition and application of communication skills in the medical workplace.

METHOD

We conducted an exploratory study comprising seven focus groups with residents in two different specialities: general practice (n=23) and surgery (n=18).

RESULTS

Residents perceive the use of summative assessment checklists that reduce communication skills to behavioural components as impeding the learning of their communication skills. Residents perceive encouragement to deliberately practise in an environment in which the value of communication skills is recognised and support is institutionalised with appropriate feedback from role models as the most important enhancing factors in communication skills learning.

CONCLUSION

To gradually realise a clinical working environment in which the above results are incorporated, we propose to use transformative learning theory to guide further studies.

PRACTICAL IMPLICATIONS

Provided it is used continuously, an approach that combines self-directed learning with observation and discussion of resident-patient consultations seems an effective method for transformative learning of communication skills.

Assessing communication quality of consultations in primary care: initial reliability of the Global Consultation Rating Scale, based on the Calgary-Cambridge Guide to the Medical Interview

OBJECTIVES

To investigate initial reliability of the Global Consultation Rating Scale (GCRS: an instrument to assess the effectiveness of communication across an entire doctor-patient consultation, based on the Calgary-Cambridge guide to the medical interview), in simulated patient consultations.

DESIGN

Multiple ratings of simulated general practitioner (GP)-patient consultations by trained GP evaluators.

SETTING

UK primary care.

PARTICIPANTS

21 GPs and six trained GP evaluators.

OUTCOME MEASURES

GCRS score.

METHODS

6 GP raters used GCRS to rate randomly assigned video recordings of GP consultations with simulated patients. Each of the 42 consultations was rated separately by four raters. We considered whether a fixed difference between scores had the same meaning at all levels of performance. We then examined the reliability of GCRS using mixed linear regression models. We augmented our regression model to also examine whether there were systematic biases between the scores given by different raters and to look for possible order effects.

RESULTS

Assessing the communication quality of individual consultations, GCRS achieved a reliability of 0.73 (95% CI 0.44 to 0.79) for two raters, 0.80 (0.54 to 0.85) for three and 0.85 (0.61 to 0.88) for four. We found an average difference of 1.65 (on a 0-10 scale) in the scores given by the least and most generous raters: adjusting for this evaluator bias increased reliability to 0.78 (0.53 to 0.83) for two raters; 0.85 (0.63 to 0.88) for three and 0.88 (0.69 to 0.91) for four. There were considerable order effects, with later consultations (after 15-20 ratings) receiving, on average, scores more than one point higher on a 0-10 scale.

CONCLUSIONS

GCRS shows good reliability with three raters assessing each consultation. We are currently developing the scale further by assessing a large sample of real-world consultations.

Impact of placement type on the development of clinical competency in speech-language pathology students

BACKGROUND

Speech-language pathology students gain experience and clinical competency through clinical education placements. However, currently little empirical information exists regarding how competency develops. Existing research about the effectiveness of placement types and models in developing competency is generally descriptive and based on opinions and perceptions. The changing nature of education of speech-language pathology students, diverse student cohorts, and the crisis in finding sufficient clinical education placements mean that establishing the most effective and efficient methods for developing clinical competency in students is needed.

AIMS

To gather empirical information regarding the development of competence in speech-language pathology students; and to determine if growth of competency differs in groups of students completing placements that differ in terms of caseload, intensity and setting.

METHODS & PROCEDURES

Participants were students in the third year of a four-year undergraduate speech-language pathology degree who completed three clinical placements across the year and were assessed with the COMPASS® competency assessment tool. Competency development for the whole group across the three placements is described. Growth of competency in groups of students completing different placement types is compared. Interval-level data generated from the students' COMPASS® results were subjected to parametric statistical analyses.

OUTCOMES & RESULTS

The whole group of students increased significantly in competency from placement to placement across different placement settings, intensities and client age groups. Groups completing child placements achieved significantly higher growth in competency when compared with the competency growth of students completing adult placements. Growth of competency was not significantly different for students experiencing different intensity of placements, or different placement settings.

CONCLUSIONS & IMPLICATIONS

These results confirm that the competency of speech-language pathology students develops across three clinical placements over a one-year period regardless of placement type or context, indicating that there may be a transfer of learning between placements types. Further research investigating patterns of competency development in speech-language pathology students is warranted to ensure that assumptions used to design clinical learning opportunities are based on valid evidence.

Validity in work-based assessment: expanding our horizons

CONTEXT

Although work-based assessments (WBA) may come closest to assessing habitual performance, their use for summative purposes is not undisputed. Most criticism of WBA stems from approaches to validity consistent with the quantitative psychometric framework. However, there is increasing research evidence that indicates that the assumptions underlying the predictive, deterministic framework of psychometrics may no longer hold. In this discussion paper we argue that meaningfulness and appropriateness of current validity evidence can be called into question and that we need alternative strategies to assessment and validity inquiry that build on current theories of learning and performance in complex and dynamic workplace settings.

METHODS

Drawing from research in various professional fields we outline key issues within the mechanisms of learning, competence and performance in the context of complex social environments and illustrate their relevance to WBA. In reviewing recent socio-cultural learning theory and research on performance and performance interpretations in work settings, we demonstrate that learning, competence (as inferred from performance) as well as performance interpretations are to be seen as inherently contextualised, and can only be under-stood 'in situ'. Assessment in the context of work settings may, therefore, be more usefully viewed as a socially situated interpretive act.

DISCUSSION

We propose constructivist-interpretivist approaches towards WBA in order to capture and understand contextualised learning and performance in work settings. Theoretical assumptions underlying interpretivist assessment approaches call for a validity theory that provides the theoretical framework and conceptual tools to guide the validation process in the qualitative assessment inquiry. Basic principles of rigour specific to qualitative research have been established, and they can and should be used to determine validity in interpretivist assessment approaches. If used properly, these strategies generate trustworthy evidence that is needed to develop the validity argument in WBA, allowing for in-depth and meaningful information about professional competence.

Composite reliability of a workplace-based assessment toolbox for postgraduate medical education

In recent years, postgraduate assessment programmes around the world have embraced workplace-based assessment (WBA) and its related tools. Despite their widespread use, results of studies on the validity and reliability of these tools have been variable. Although in many countries decisions about residents' continuation of training and certification as a specialist are based on the composite results of different WBAs collected in a portfolio, to our knowledge, the reliability of such a WBA toolbox has never been investigated. Using generalisability theory, we analysed the separate and composite reliability of three WBA tools [mini-Clinical Evaluation Exercise (mini-CEX), direct observation of procedural skills (DOPS), and multisource feedback (MSF)] included in a resident portfolio. G-studies and D-studies of 12,779 WBAs from a total of 953 residents showed that a reliability coefficient of 0.80 was obtained for eight mini-CEXs, nine DOPS, and nine MSF rounds, whilst the same reliability was found for seven mini-CEXs, eight DOPS, and one MSF when combined in a portfolio. At the end of the first year of residency a portfolio with five mini-CEXs, six DOPS, and one MSF afforded reliable judgement. The results support the conclusion that several WBA tools combined in a portfolio can be a feasible and reliable method for high-stakes judgements.

Frequency and determinants of residents' narrative feedback on the teaching performance of faculty: narratives in numbers

PURPOSE

Physicians involved in residency training often receive feedback from residents on their teaching. Research shows that learners value narrative feedback, but knowledge of the frequency and determinants of narrative feedback in teaching performance evaluation is lacking. This study aims to identifythe frequency with which residentsgave positive comments and suggestions for improvement to faculty, and the factors influencing that frequency.

METHOD

From September 2008 through May 2010, the authors collected data, using a validated formative feedback system (System for Evaluation of Teaching Qualities). The authors used univariate and multivariable analysis to investigate the associations between participants' characteristics, including faculty members' teaching performance, and the frequency of the two types of narrative comments.

RESULTS

In total, 659 residents (79% of 839) completed 6,216 evaluations on 917 faculty (95% of 964), resulting in 11,574 positive comments and 4,870 suggestions for improvement. On average, faculty members received 13 positive comments and 5 suggestions for improvement. Multivariable analysis showed that higher teaching performance was associated with higher numbers of positive comments (regression coefficient 0.538; 95% confidence interval: 0.464 to 0.613) and with lower numbers of suggestions for improvement (-0.802; -0.911 to -0.692), both P < .0001. Nonacademic hospitals, participation in teacher training, and female residents' evaluation were statistically significant determinants of receiving more narrative feedback.

CONCLUSIONS

Residents provided narrative feedback that paralleled and elaborated on quantitative evaluations they provided; therefore, faculty would be wise to attend to narrative feedback. Analysis of the quality of narrative feedback is needed to understand its effectiveness.

Reflection as a component of formative assessment appears to be instrumental in promoting the use of feedback; an observational study

BACKGROUND

Although the literature suggests that reflection has a positive impact on learning, there is a paucity of evidence to support this notion.

AIM

We investigated feedback and reflection in relation to the likelihood that feedback will be used to inform action plans. We hypothesised that feedback and reflection present a cumulative sequence (i.e. trainers only pay attention to trainees' reflections when they provided specific feedback) and we hypothesised a supplementary effect of reflection.

METHOD

We analysed copies of assessment forms containing trainees' reflections and trainers' feedback on observed clinical performance. We determined whether the response patterns revealed cumulative sequences in line with the Guttman scale. We further examined the relationship between reflection, feedback and the mean number of specific comments related to an action plan (ANOVA) and we calculated two effect sizes.

RESULTS

Both hypotheses were confirmed by the results. The response pattern found showed an almost perfect fit with the Guttman scale (0.99) and reflection seems to have supplementary effect on the variable action plan.

CONCLUSIONS

Reflection only occurs when a trainer has provided specific feedback; trainees who reflect on their performance are more likely to make use of feedback. These results confirm findings and suggestions reported in the literature.

Workplace-based assessment: raters' performance theories and constructs

Weaknesses in the nature of rater judgments are generally considered to compromise the utility of workplace-based assessment (WBA). In order to gain insight into the underpinnings of rater behaviours, we investigated how raters form impressions of and make judgments on trainee performance. Using theoretical frameworks of social cognition and person perception, we explored raters' implicit performance theories, use of task-specific performance schemas and the formation of person schemas during WBA. We used think-aloud procedures and verbal protocol analysis to investigate schema-based processing by experienced (N = 18) and inexperienced (N = 16) raters (supervisor-raters in general practice residency training). Qualitative data analysis was used to explore schema content and usage. We quantitatively assessed rater idiosyncrasy in the use of performance schemas and we investigated effects of rater expertise on the use of (task-specific) performance schemas. Raters used different schemas in judging trainee performance. We developed a normative performance theory comprising seventeen inter-related performance dimensions. Levels of rater idiosyncrasy were substantial and unrelated to rater expertise. Experienced raters made significantly more use of task-specific performance schemas compared to inexperienced raters, suggesting more differentiated performance schemas in experienced raters. Most raters started to develop person schemas the moment they began to observe trainee performance. The findings further our understanding of processes underpinning judgment and decision making in WBA. Raters make and justify judgments based on personal theories and performance constructs. Raters' information processing seems to be affected by differences in rater expertise. The results of this study can help to improve rater training, the design of assessment instruments and decision making in WBA.

What is wrong with assessment in postgraduate training? Lessons from clinical practice and educational research

Workplace-based assessment is more commonly given a lukewarm than a warm welcome by its prospective users. In this article, we summarise the workplace-based assessment literature as well as our own experiences with workplace-based assessment to derive lessons that can facilitate acceptance of workplace-based assessment in postgraduate specialty training. We propose to shift the emphasis in workplace-based assessment from assessment of trainee performance to the learning of trainees. Workplace-based assessment should focus on supporting supervisors in taking entrustment decisions by complementing their "gut feeling" with information from assessments and focus less on assessment and testability. One of the most stubborn problems with workplace-based assessment is the absence of observation of trainees and the lack of feedback based on observations. Non-standardised observations are used to organise feedback. To make these assessments meaningful for learning, it is essential that they are not perceived as summative by their users, that they provide narrative feedback for the learner and that there is a form of facilitation that helps to integrate the feedback in trainees' self-assessments.

Devising the optimal preclinical oncology curriculum for undergraduate medical students in the United States

A third of women and a near majority of men in the United States will be diagnosed with cancer in their lifetimes. To prepare future physicians for this reality, we have developed a preclinical oncology curriculum that introduces second-year medical students to essential concepts and practices in oncology to improve their abilities to appropriately care for these patients. We surveyed the oncology and education literature and compiled subjects important to students' education including basic science and clinical aspects of oncology and addressing patients' psychosocial needs. Along with the proposed curriculum content, scheduling, independent learning exercises, and case studies, we discuss practical considerations for curriculum implementation based on experience at our institution. Given the changing oncology healthcare landscape, all (new) physicians must competently address their cancer patients' needs, regardless of chosen specialty. A thorough and logically organized cancer curriculum for preclinical medical students should help achieve these aims. This new model curriculum, with accompanying strategies to evaluate its efforts, is essential to update how medical students are educated about cancer.

Assessment of clinical competencies using clinical images and videos "CIVA"

BACKGROUND

This paper describes an assessment approach of clinical competencies which widens the number of problems and tasks evaluated using videos and images.

METHOD

Clinical Image and Video Assessment (CIVA) was used to assess clinical reasoning and decision making of final year medical students. Forty to fifty clinical videos and images supported by rich text vignette and reviewed by subject matter experts were selected based on examination blueprints for analysis. CIVA scores were correlated with OSCE, Direct Observation Clinical Encounter Exam (DOCEE) and written exam scores, using the 2-sided Pearson correlation analysis, and their reliability was analyzed using Cronbach's Alpha Coefficient. Furthermore, students personally evaluated the CIVA using a 5- point Likert scale.

RESULTS

CIVA and OSCE scores showed a high correlation (r = 0.83) in contrast with the correlation scores of the written examination (r = .36) and the DOCEE (r = 0.35). Cronbach's Alpha for the OSCE and CIVA for the first batch was 0.71 and 0.78. As for the second batch it was 0.91 and 0.91 respectively. Eighty-two percent of students were very satisfied or satisfied with the CIVA process, contents and quality.

CONCLUSIONS

A well constructed CIVA type assessment with a rich authentic vignette and good quality videos and images could be used to assess clinical reasoning and decision making of final year medical students. CIVA is an assessment tool which correlates well with OSCE, compliments the written and DOCEE and is easier to conduct at a possibly reduced cost.

A laboratory study on the reliability estimations of the mini-CEX

Reliability estimations of workplace-based assessments with the mini-CEX are typically based on real-life data. Estimations are based on the assumption of local independence: the object of the measurement should not be influenced by the measurement itself and samples should be completely independent. This is difficult to achieve. Furthermore, the variance caused by the case/patient or by assessor is completely confounded. We have no idea how much each of these factors contribute to the noise in the measurement. The aim of this study was to use a controlled setup that overcomes these difficulties and to estimate the reproducibility of the mini-CEX. Three encounters were videotaped from 21 residents. The patients were the same for all residents. Each encounter was assessed by 3 assessors who assessed all encounters for all residents. This delivered a fully crossed (all random) two-facet generalizability design. A quarter of the total variance was associated with universe score variance (28%). The largest source of variance was the general error term (34%) followed by the main effect of assessors (18%). Generalizability coefficients indicated that an approximate sample of 9 encounters was needed assuming a single different assessor per encounter and assuming different cases per encounter (the usual situation in real practice), 4 encounters when 2 raters were used and 3 encounters when 3 raters are used. Unexplained general error and the leniency/stringency of assessors are the major causes for unreliability in mini-CEX. To optimize reliability rater training might have an effect.

An assessment program using standardized clients to determine student readiness for clinical practice

Objective. To develop, implement, and review a competence-assessment program to identify students at risk of underperforming at advanced pharmacy practice experience (APPE) sites and to facilitate remediation before they assume responsibility for patient care.Design. As part of the standardized client program, pharmacy students were examined in realistic live client-encounter simulations. Authentic scenarios were developed, and actors were recruited and trained to portray clients so students could be examined solving multiple pharmacy problems. Evaluations of students were conducted in the broad areas of knowledge and live performance.Assessment. Measurements included student-experience survey instruments used to evaluate case realism and challenge; videos used to determine the fidelity of standardized clients, and clerkship performance predictions used to identify students who required individual attention and improvement prior to clerkship courses.Conclusions. The assessment program showed promise as a means of discriminating between students who are prepared for APPEs and those at risk for underperforming.

Realizing the promise and importance of performance-based assessment

Work-based assessment (WBA) is the assessment of trainees and physicians across the educational continuum of day-to-day competencies and practices in authentic, clinical environments. What distinguishes WBA from other assessment modalities is that it enables the evaluation of performance in context. In this perspective, we describe the growing importance, relevance, and evolution of WBA as it relates to competency-based medical education, supervision, and entrustment. Although a systematic review is beyond the purview of this perspective, we highlight specific methods and needed shifts to WBA that (a) consider patient outcomes, (b) use nonphysician assessors, and (c) assess the care provided to populations of patients. We briefly describe strategies for the effective implementation of WBA and identify outstanding research questions related to its use.

From west to east; experience with adapting a curriculum in evidence-based medicine

Clinical epidemiology (CE) and evidence-based medicine (EBM) have become an important part of medical school curricula. This report describes the implementation and some preliminary outcomes of an integrated CE and EBM module in the Faculty of Medicine Universitas Indonesia (UI), Jakarta and in the University of Malaya (UM) in Kuala Lumpur. A CE and EBM module, originally developed at the University Medical Center Utrecht (UMCU), was adapted for implementation in Jakarta and Kuala Lumpur. Before the start of the module, UI and UM staff followed a training of teachers (TOT). Student competencies were assessed through pre and post multiple-choice knowledge tests, an oral and written structured evidence summary (evidence-based case report, EBCR) as well as a written exam. All students also filled in a module evaluation questionnaire. The TOT was well received by staff in Jakarta and Kuala Lumpur and after adaptation the CE and EBM modules were integrated in both medical schools. The pre-test results of UI and UM were significantly lower than those of UMCU students (p < 0.001). The post test results of UMCU students were comparable (p = 0.48) with UI, but significantly different (p < 0.001) from UM. Common problems for the modules in both UI and UM were limited access to literature and variability of the tutors' skills. Adoption and integration of an existing Western CE-EBM teaching module into Asian medical curricula is feasible while learning outcomes obtained are quite similar.

Objective Structured Assessment of Technical Skill in assessing technical competence to carry out caesarean section with increasing seniority

Since the incorporation of workplace-based assessment within the specialty training programme in obstetrics and gynaecology, the assessment of technical competence to carry out caesarean section has been undertaken by the Objective Structured Assessment of Technical Skill tool. This requirement has been formalised in the Matrix of Educational Progression, ensuring that the tool must assess trainees' technical competence in caesarean section procedures of varying levels of complexity throughout training. Trainee feedback suggests that the effectiveness of the tool diminishes as the seniority of the trainee increases, with technical competence assessed less effectively in more complex procedures. This seems to be a result of the generic design of the tool and insufficient training on the part of assessors. Both of these are due to be addressed within a division of the Objective Structured Assessment of Technical Skill tool into explicitly formative and summative assessments of technical skill, following a General Medical Council-led consultation on the future of workplace-based assessment.

The midwifery miniCEX--a valuable clinical assessment tool for midwifery education

BACKGROUND

Midwifery students, clinicians and educators in Australia identified the need for improved feedback for midwifery students whilst they are on clinical placement; in particular formative assessment. The miniCEX or mini-clinical evaluation exercise is one approach to assessment that has been proven valid and reliable in medical education. The aim of this research was to develop, implement and evaluate a miniCEX tool for midwifery education.

METHODS

Using an action research approach, this project engaged midwifery clinicians and midwifery students to adapt and implement the miniCEX in a postnatal ward environment. Focus groups were held to establish the clinical expectations and develop performance guidelines of students across the domains of midwifery practice, as well as evaluate their use in practice.

FINDINGS

Evaluation of the midwifery miniCEX, including its applicability from the perspective of staff and students was positive. The miniCEX was found to be easy to use, time efficient and valuable for learning.

DISCUSSION

The miniCEX is an innovative approach to assessment and feedback in midwifery education, and there is currently no identified evidence of its use in midwifery education despite broad use globally in medical education.

CONCLUSION

The implementation of the midwifery miniCEX offers broad benefit to both midwifery students and midwifery clinicians and educators globally.

Assessment of CanMEDS roles in postgraduate training: the validation of the Compass

OBJECTIVE

In medical education the focus has shifted from gaining knowledge to developing competencies. To effectively monitor performance in practice throughout the entire training, a new approach of assessment is needed. This study aimed to evaluate an instrument that monitors the development of competencies during postgraduate training in the setting of training of general practice: the Competency Assessment List (Compass).

METHODS

The distribution of scores, reliability, validity, responsiveness and feasibility of the Compass were evaluated.

RESULTS

Scores of the Compass ranged from 1 to 9 on a 10-point scale, showing excellent internal consistency ranging from .89 to .94. Most trainees showed improving ratings during training. Medium to large effect sizes (.31-1.41) were demonstrated when we compared mean scores of three consecutive periods. Content validity of the Compass was supported by the results of a qualitative study using the RAND modified Delphi Method. The feasibility of the Compass was demonstrated.

CONCLUSION

The Compass is a competency based instrument that shows in a reliable and valid way trainees' progress towards the standard of performance.

PRACTICE IMPLICATIONS

The programmatic approach of the Compass could be applied in other specialties provided that the instrument is tailored to specific needs of that specialism.

Uvedba 360-Stopinjskega ocenjevanja odnosa in obnašanja specializantov

Uvod:Učenje in ocenjevanje profesionalnega odnosa in obnašanja sta enako pomembna kot ocenjevanje teoretičnega znanja in veščin, vendar se malokrat izvaja.

Metode: Specializant razdeli ocenjevalne obrazce, ki jih ocenjevalci izpolnjene in podpisane pošljejo nacionalni koordinatorici; ko dobi deset ocen in samooceno, povabi specializanta na pogovor s presojo.

Rezultati: V 6 ciklusih 360-stopinjskega ocenjevanja je bilo opravljenih 118 pogovorov. Večina je bila ocenjena dobro. V manj kot 10 % je bila ocena »sem zaskrbljen« in »sem hudo zaskrbljen«.

Razprava: Večina specializantov je bila prijetno presenečena nad pohvalami, ki so jih napisali sodelavci. Osem specializantov so opozorili na manjše pomanjkljivosti v odnosu in obnašanju, ki se jih specializanti niso zavedali. Do naslednjega ocenjevanja so pomanjkljivosti večinoma odpravili. Specializanti, ocenjeni s »sem hudo zaskrbljen«, so bili redki, vendar so se pri njih take ocene ponavljale; niso bili samokritični in pogovora z odzivom niso sprejemali kot dobronamernega.

Zaključki: V specializacijo porodništva in ginekologije so uvedli ocenjevanje odnosa in obnašanja specializantov. Hkrati z ocenjevanjem so uvedli tudi druga orodja, ki vodijo k zavedanju profesionalizma: učiteljsko tutorstvo na Medicinski fakulteti v Ljubljani z delavnicami samorefleksije in delavnice treniranja trenerjev. Poudarek na profesionalizmu naj bo vidnejši v izbirnem postopku za specializacijo, uvodnem intervjuju s specializantom in v pogovorih s presojo z glavnim mentorjem v rednih presledkih, v večji komunikaciji med glavnimi in neposrednimi mentorji ter v ocenjevanju mentorjev. Na uvedbo še čaka pogovor o profesionalizmu ob sprejemu na medicinsko fakulteto ter predmet profesionalizem na medicinskih fakultetah in med specializacijo.

How do postgraduate GP trainees regulate their learning and what helps and hinders them? A qualitative study

BACKGROUND

Self-regulation is essential for professional development. It involves monitoring of performance, identifying domains for improvement, undertaking learning activities, applying newly learned knowledge and skills and self-assessing performance. Since self-assessment alone is ineffective in identifying weaknesses, learners should seek external feedback too. Externally regulated educational interventions, like reflection, learning portfolios, assessments and progress meetings, are increasingly used to scaffold self-regulation.The aim of this study is to explore how postgraduate trainees regulate their learning in the workplace, how external regulation promotes self-regulation and which elements facilitate or impede self-regulation and learning.

METHODS

In a qualitative study with a phenomenologic approach we interviewed first- and third-year GP trainees from two universities in the Netherlands. Twenty-one verbatim transcripts were coded. Through iterative discussion the researchers agreed on the interpretation of the data and saturation was reached.

RESULTS

Trainees used a short and a long self-regulation loop. The short loop took one week at most and was focused on problems that were easy to resolve and needed minor learning activities. The long loop was focused on complex or recurring problems needing multiple and planned longitudinal learning activities. External assessments and formal training affected the long but not the short loop. The supervisor had a facilitating role in both loops. Self-confidence was used to gauge competence.Elements influencing self-regulation were classified into three dimensions: personal (strong motivation to become a good doctor), interpersonal (stimulation from others) and contextual (organizational and educational features).

CONCLUSIONS

Trainees did purposefully self-regulate their learning. Learning in the short loop may not be visible to others. Trainees should be encouraged to actively seek and use external feedback in both loops. An important question for further research is which educational interventions might be used to scaffold learning in the short loop. Investing in supervisor quality remains important, since they are close to trainee learning in both loops.

The use of programmatic assessment in the clinical workplace: a Maastricht case report

The differences of learning experiences in the workplace put challenges on the assessment: the assessment programme should be aligned with the general competency framework of the curriculum and also fit to the differences in learning contexts of the workplace. We used van der Vleuten's programmatic assessment model to develop a workplace-based assessment programme for final year clerkships. We aimed to design a programme that stimulates learning, supports assessment decision, is feasible and non-bureaucratic. The first experiences with the programme show that students think that the programme has high learning value and the assessment is sufficiently robust. Many of the commonly reported weaknesses of work-based assessment (not a good fit with the educational context, too complex, too bureaucratic and too much work) were not mentioned by the students.

A model for programmatic assessment fit for purpose

We propose a model for programmatic assessment in action, which simultaneously optimises assessment for learning and assessment for decision making about learner progress. This model is based on a set of assessment principles that are interpreted from empirical research. It specifies cycles of training, assessment and learner support activities that are complemented by intermediate and final moments of evaluation on aggregated assessment data points. A key principle is that individual data points are maximised for learning and feedback value, whereas high-stake decisions are based on the aggregation of many data points. Expert judgement plays an important role in the programme. Fundamental is the notion of sampling and bias reduction to deal with the inevitable subjectivity of this type of judgement. Bias reduction is further sought in procedural assessment strategies derived from criteria for qualitative research. We discuss a number of challenges and opportunities around the proposed model. One of its prime virtues is that it enables assessment to move, beyond the dominant psychometric discourse with its focus on individual instruments, towards a systems approach to assessment design underpinned by empirically grounded theory.

Programmatic assessment and Kane's validity perspective

CONTEXT

Programmatic assessment is a notion that implies that the strength of the assessment process results from a careful combination of various assessment instruments. Accordingly, no single instrument is superior to another, but each has its own strengths, weaknesses and purpose in a programme. Yet, in terms of psychometric methods, a one-size-fits-all approach is often used. Kane's views on validity as represented by a series of arguments provide a useful framework from which to highlight the value of different widely used approaches to improve the quality and validity of assessment procedures.

METHODS

In this paper we discuss four inferences which form part of Kane's validity theory: from observations to scores; from scores to universe scores; from universe scores to target domain, and from target domain to construct. For each of these inferences, we provide examples and descriptions of approaches and arguments that may help to support the validity inference.

CONCLUSIONS

As well as standard psychometric methods, a programme of assessment makes use of various other arguments, such as: item review and quality control, structuring and examiner training; probabilistic methods, saturation approaches and judgement processes, and epidemiological methods, collation, triangulation and member-checking procedures. In an assessment programme each of these can be used.

Flower power: the armoured expert in the CanMEDS competency framework?

Competency frameworks based on roles definitions are currently being used extensively in health professions education internationally. One of the most successful and widely used models is the CanMEDS Roles Framework. The medical literature has raised questions about both the theoretical underpinnings and the practical application of outcomes-based frameworks, however little empirical research has yet been done examining specific roles frameworks. This study examines the historical development of an important early roles framework, the Educating Future Physicians of Ontario (EFPO) roles, which were instrumental in the development of the CanMEDS roles. Prominent discourses related to roles development are examined using critical discourse analysis methodology. Exploration of discourses that emerged in the development of this particular set of roles definitions highlights the contextual and negotiated nature of roles construction. The discourses of threat and protection prevalent in the EFPO roles development offer insight into the visual construction of a centre of medical expertise surrounded by supporting roles (such as collaborator and manager). Non-medical expert roles may perhaps play the part of 'armour' for the authority of medical expertise under threat. This research suggests that it may not be accurate to consider roles as objective ideals. Effective training models may require explicit acknowledgement of the socially negotiated and contextual nature of roles definitions.

Self-evaluation of assessment programs: a cross-case analysis

The goal of this article is to contribute to the validation of a self-evaluation method, which can be used by schools to evaluate the quality of their Competence Assessment Program (CAP). The outcomes of the self-evaluations of two schools are systematically compared: a novice school with little experience in competence-based education and assessment, and an innovative school with extensive experience. The self-evaluation was based on 12 quality criteria for CAPs, including both validity and reliability, and criteria stressing the importance of the formative function of assessment, such as meaningfulness and educational consequences. In each school, teachers, management and examination board participated. Results show that the two schools use different approaches to assure assessment quality. The innovative school seems to be more aware of its own strengths and weaknesses, to have a more positive attitude towards teachers, students, and educational innovations, and to explicitly involve stakeholders (i.e., teachers, students, and the work field) in their assessments. This school also had a more explicit vision of the goal of competence-based education and could design its assessments in accordance with these goals.

How do physicians assess their family physician colleagues' performance?: creating a rubric to inform assessment and feedback

INTRODUCTION

The Colleges of Physicians and Surgeons of Alberta and Nova Scotia (CPSNS) use a standardized multisource feedback program, the Physician Achievement Review (PAR/NSPAR), to provide physicians with performance assessment data via questionnaires from medical colleagues, coworkers, and patients on 5 practice domains: consultation communication, patient interaction, professional self-management, clinical competence, and psychosocial management of patients. Physicians receive a confidential report; the intent is practice improvement. However, research indicates that feedback from medical colleagues appears to be less understood than that from coworkers or patients, due to a lack of specificity and concerns regarding feedback credibility. The purpose of this study was to determine how physicians make decisions about performance ratings for family physician (FP) colleagues in the 5 practice domains.

METHODS

This was an exploratory qualitative study using focus groups-one with 11 family physicians and one with 12 specialists-who had served as NSPAR "medical colleague'' reviewers. We analyzed focus group transcripts using content analysis.

RESULTS

Family and specialist physicians provided examples of behaviors indicative of both high- and low-scoring performance for items within the 5 practice domains. From these, an assessment rubric was created to inform both external reviewers and the physicians being reviewed of performance expectations. Reviewers reported using varied sources of information to make assessments, including shared patients, medical records, referral letters, feedback from others, and self-reference.

DISCUSSION

The CPSNS has used the assessment rubric to create an online resource to inform medical colleague assessment and enhance the usefulness of their NSPAR scores. Further research will be required to determine its impact.