Current methods in use for assessing clinical competencies: what works?

Facing the "Fear of Failure": Veterinary Students in Clinical Rotations

Failing a student is difficult for both educator and student, but administering a failing grade is critical for protecting and ensuring adequate learning for an unsafe student. The failure to fail clinical students has been commonly reported and explored among educators in the human health professions but has not been formally addressed in veterinary education. Forty-three participants attending the Veterinary Educators Collaborative symposium were surveyed concerning their attitudes and experiences failing clinical veterinary students. Results indicated that the failure to fail phenomenon exists among veterinary educators, as the majority of veterinary educators often felt reluctant and unprepared to fail a student on clinical rotations. The most common barriers to failing students were institutional culture and unsatisfactory assessor development or evaluation tools. Veterinary educators must face this fear of failure and explore strategies to overcome existing barriers that can ultimately transform student failure into success.

Effect of Instructional Format on Veterinary Students' Task Performance and Emotional State during a Simulation-Based Canine Endotracheal Intubation Laboratory: Handout versus Video

Video- versus handout-based instructions may influence student outcomes during simulation training and competency-based assessments. Forty-five third-year veterinary students voluntarily participated in a simulation module on canine endotracheal intubation. A prospective, randomized, double-blinded study investigated the impact of video (n = 23) versus handout (n = 22) instructions on student confidence, anxiety, and task performance. Students self-scored their confidence and anxiety before and after the simulation. During the simulation laboratory, three raters independently evaluated student performance using a 20-item formal assessment tool with a 5-point global rating scale. No significant between- or within-group differences (p > .05) were found for both confidence and anxiety scores. Video-based instructions were associated with significantly higher (p < .05) total formal assessment scores compared with handout-based instructions. The video group had significantly higher scores than the handout group on 3 of the 20 individual skills (items) assessed: placement of tie to the adaptor-endotracheal tube complex (p < .05), using the anesthetic machine (p < .01), and pop-off valve management (p < .001). Inter-rater reliability as assessed by Cronbach's α (.92), and Kendall's W (.89) was excellent and almost perfect, respectively. A two-faceted crossed-design generalizability analysis yielded G coefficients for both the handout (E_p² = .68) and the video (E_p² = .72) groups. Video instructions may be associated with higher performance scores than handout instructions during endotracheal intubation simulation training. Further research into skill retention and learning styles is warranted.

A Competency-Guided Veterinary Curriculum Review Process

Competencies can guide outcomes assessment in veterinary medical education by providing a core set of specific abilities expected of new veterinary graduates. A competency-guided evaluation of Colorado State University's (CSU) equine veterinary curriculum was undertaken via an alumni survey. Published competencies for equine veterinary graduates were used to develop the survey, which was distributed to large animal alumni from CSU's Doctor of Veterinary Medicine program. The results of the survey indicated areas for improvement, specifically in equine business, surgery, dentistry, and radiology. The desire for more hands-on experiences in their training was repeatedly mentioned by alumni, with the largest discrepancies between didactic knowledge and hands-on skills in the areas of business and equine surgery. Alumni surveys allow graduates to voice their perceived levels of preparation by the veterinary program and should be used to inform curriculum revisions. It is proposed that the definition and utilization of competencies in each phase of a curricular review process (outcomes assessment, curriculum mapping, and curricular modifications), in addition to faculty experience and internal review, is warranted.

Identifying entrustable professional activities for surgical skills training in companion animal health

BACKGROUND

Veterinary medical education is increasingly moving towards outcome-based training based on competency frameworks. A source of concern is the translation of competencies into the practice of clinical teaching, for example, surgical skills training. It is suggested that the use of entrustable professional activities (EPAs) might bridge this gap. The purpose of this study, therefore, was to identify EPAs related to surgical skills for companion animal health to enhance competency-based education.

METHODS

Draft versions of EPAs related to surgical skills were established by an iterative consensus-based approach through 45-min interview sessions. These draft versions were used to explore the opinion of companion animal veterinarians, both veterinarians (specialists, residents and interns) involved in undergraduate teaching and veterinarians working in private practice involved in extramural clinical teaching, on the relevance and level of entrustment of the EPAs through a modified Delphi procedure. Mean (relevance) and median (level of entrustment) scores were calculated and textual comments were analysed to create a final framework of EPAs related to surgical skills.

RESULTS AND CONCLUSION

The Delphi panel reached consensus in three rounds. Thirty-four per cent of those invited to participate in the study completed the final survey. Finally, a list of 13 EPAs related to companion animal surgical skills a student should be entrusted to perform at time of graduation was established.

Assessing veterinary students using in-training evaluation scores: what is being assessed?

In-training evaluations are commonly used for assessing veterinary students during clinical training, but are criticised for being unable to discriminate dimensions of performance. This study investigated scores on an in-training evaluation in use at one veterinary school to determine the dimensions being assessed and the influence of the dimensions on the overall score awarded. Common factor analysis and ordinal logistic regression were conducted on a retrospective sample of 3466 evaluations of 197 final year veterinary students. The findings suggested a higher-order dimensional structure, with one overarching factor and two to four subfactors, consistent with the complex construct of competency that thSAS Institute e assessment was intended to assess. In the four -factor model, all dimensions were significantly related to overall grade, with the effects of the professional attitude factor and the knowledge factor dependent on the placement. The professional attitude factor had the strongest effect on overall grade (β=2.71, P=0.0004). There was a significant effect of placement on overall grade (P=0.021). Neither academic status of the supervisor nor grade point average had significant effects on the overall grade (P>0.49), and a student's overall grade did not significantly differ over time (P=1). The results suggest that the complexity of supervisor judgement is effectively represented in evaluation scores.

Evaluation of learning curves for ovariohysterectomy of dogs and cats and castration of dogs

OBJECTIVE To define learning curves for fourth-year veterinary students performing ovariohysterectomy procedures in dogs and cats and castration in dogs. DESIGN Retrospective study. SAMPLE 3,196 ovariohysterectomies or castrations performed in dogs and cats by 88 veterinary students during a spay-neuter surgery and animal shelter rotation (n = 3,056) or by 1 experienced general practitioner (n = 140). PROCEDURES Data collected from medical records included patient signalment, type and duration of procedure, and sequence (by date and time) of the procedure within a list of procedures of the same type generated for each student. For each procedure type, geometric mean surgery time and 95% confidence intervals were determined for each number of surgeries completed by ≥ 10 students. Median surgery times for the same procedure types were determined for the experienced practitioner. The learning curve for each procedure was modeled with nonlinear (3-factor exponential equation with a nonzero asymptote) and linear regression. For each procedure, the asymptote (optimal surgery time) for students was compared with the experienced practitioner's median surgery time. RESULTS 2,945 surgeries (mean, 33/student) performed by ≥ 10 students were analyzed. Surgery time decreased in a nonlinear manner as student experience increased for castration of adult or pediatric dogs and ovariohysterectomy of pediatric dogs and adult or pediatric cats. Surgery time decreased in a linear manner as experience increased for ovariohysterectomy of adult dogs. CONCLUSIONS AND CLINICAL RELEVANCE To the authors' knowledge, this was the first study to map surgery times for common surgical procedures consecutively performed by veterinary students. Results clearly indicated the value of repetition to improve surgical skills (as measured by surgery time) during a 3-week period.

Teaching Veterinary Anesthesia: A Survey-Based Evaluation of Two High-Fidelity Models and Live-Animal Experience for Undergraduate Veterinary Students

In veterinary medical education, reduction, replacement, and refinement (the three Rs) must be considered. Three clinical skills in anesthesia were identified as challenging to students: endotracheal intubation, intravenous catheterization, and drug dose calculations. The aims of this project were to evaluate students' perception of their level of confidence in performing these three clinical skills in veterinary anesthesia, to document the extent of students' previous experience in performing these three tasks, and to describe students' emotional states during this training. Veterinary students completed a series of four surveys over the period of their pre-clinical training to evaluate the usefulness of high-fidelity models for skill acquisition in endotracheal intubation and intravenous catheterization. In addition, practice and ongoing assessment in drug dose calculations were performed. The curriculum during this period of training progressed from lectures and non-animal training, to anesthesia of pigs undergoing surgery from which they did not recover, and finally to anesthesia of dogs and cats in a neutering clinic. The level of confidence for each of the three clinical skills increased over the study period. For each skill, the number of students with no confidence decreased to zero and the proportion of students with higher levels of confidence increased. The high-fidelity models for endotracheal intubation and intravenous catheterization used to complement the live-animal teaching were considered a useful adjunct to the teaching of clinical skills in veterinary anesthesia. With practice, students became more confident performing drug dose calculations.

Correlation of pre-veterinary admissions criteria, intra-professional curriculum measures, AVMA-COE professional competency scores, and the NAVLE

Data consisting of preadmission criteria scores, annual and final cumulative grade point averages (GPAs), grades from individual professional courses, American Veterinary Medical Association Council on Education (AVMA-COE) Competency scores, annual class rank, and North American Veterinary Licensing Exam (NAVLE) scores were collected on all graduating DVM students at Kansas State University in 2009 and 2010. Associations among the collected data were compared by Pearson correlation. Pre-veterinary admissions criteria infrequently correlated with annual GPAs of Years 1-3, rarely correlated with the AVMA-COE Competencies, and never correlated with the annual GPA of Year 4. Low positive correlations occurred between the NAVLE and the Verbal Graduate Record Examination (GRE) (r=.214), Total GRE (r=.171), and the mean GPA of pre-professional science courses (SGPA) (r=.236). Annual GPAs strongly correlated with didactic course scores. Annual GPAs and final class rank strongly correlated (mean r=-.849), and both strongly correlated with the NAVLE score (NAVLE: GPAs mean r=.628, NAVLE: final class rank r=-.714). Annual GPAs at the end of Years 1-4 weakly correlated or did not correlate with the AVMA-COE Competencies. The AVMA-COE Competencies weakly correlated with scores earned in didactic courses of Years 1-3. AVMA-COE Competencies were internally consistent (mean r=.796) but only moderately correlated with performance on the NAVLE (mean r=.319). Low correlations between admissions criteria and outcomes indicate a need to reevaluate admission criteria as predictors of school success. If the NAVLE remains the primary discriminator for veterinary licensure (and the gateway to professional activity), then the AVMA-COE Competencies should be refined to better improve and reflect the NAVLE, or the NAVLE examination should change to reflect AVMA-COE Competencies.

Parasitological procedures, skills, and areas of knowledge used by small-animal practitioners in North America

We designed a study to assess veterinarians' competency regarding parasitological procedures, skills, and areas of knowledge currently used in small-animal practice. The outcome will help us refine our curriculum on the basis of the parasitological working knowledge and skill sets that small-animal practitioners are using today. A questionnaire was developed and sent to small-animal practitioners. Their responses provided general information on practice characteristics, parasitological procedures used, and client education. Parasitological procedures included those to diagnose helminths, protozoa, and ectoparasites. We focused on three questions: "Do you perform or request this procedure?" "Where is this procedure performed?" and "What is your frequency?" The respondents were 478 small-animal practitioners. We performed descriptive analyses of practice characteristics along with bivariate and multivariate analyses. These analyses revealed the clinical competence of parasitological diagnoses performed or requested by small-animal practitioners. The results showed that more involved or time-consuming methods such as fecal flotation using centrifugation and the Baermann test are more often sent to a diagnostic laboratory and are requested more often by veterinarians in larger practices (i.e., those that employ more veterinarians). The outcomes also suggest that the main diagnostician may not fully understand the tests available at the diagnostic laboratory, which has an impact on decision making for management, treatment, and prevention of parasites and ultimately client education. In addition, small-animal practitioners who have been in practice longer and practices that employ five veterinarians or fewer (i.e., smaller practices) offer more client education.

Beyond NAVMEC: competency-based veterinary education and assessment of the professional competencies

The implementation of competency-based curricula within the health sciences has been an important paradigm shift over the past 30 years. As a result, one of the five strategic goals recommended by the North American Veterinary Medical Education Consortium (NAVMEC) report was to graduate career-ready veterinarians who are proficient in, and have the confidence to use, an agreed-upon set of core competencies. Of the nine competencies identified as essential for veterinary graduates, seven could be classified as professional or non-technical competencies: communication; collaboration; management (self, team, system); lifelong learning, scholarship, value of research; leadership; diversity and multicultural awareness; and adaptation to changing environments. Traditionally, the professional competencies have received less attention in veterinary curricula and their assessment is often sporadic or inconsistent. In contrast, the same or similar competencies are being increasingly recognized in other health professions as essential skills and abilities, and their assessment is being undertaken with enhanced scrutiny and critical appraisal. Several challenges have been associated with the assessment of professional competencies, including agreement as to their definition and therefore their evaluation, the fact that they are frequently complex and require multiple integrative assessments, and the ability and/or desire of faculty to teach and assess these competencies. To provide an improved context for assessment of the seven professional competencies identified in the NAVMEC report, this article describes a broad framework for their evaluation as well as specific examples of how these or similar competencies are currently being measured in medical and veterinary curricula.

Impact of outcome-based assessment on student learning and faculty instructional practices

Increased accountability has been a catalyst for the reformation of curriculum and assessment practices in postsecondary schools throughout North America, including veterinary schools. There is a call for a shift in assessment practices in clinical rotations, from a focus on content to a focus on assessing student performance. Learning is subsequently articulated in terms of observable outcomes and indicators that describe what the learner can do after engaging in a learning experience. The purpose of this study was to examine the ways in which a competency-based program in an early phase of implementation impacted student learning and faculty instructional practices. Findings revealed that negative student perceptions of the assessment instrument's reliability had a detrimental effect on the face validity of the instrument and, subsequently, on students' engagement with competency-based assessment and promotion of student-centered learning. While the examination of faculty practices echoed findings from other studies that cited the need for faculty development to improve rater reliability and for a better data management system, our study found that faculty members' instructional practices improved through the alignment of instruction and curriculum. This snap-shot of the early stages of implementing a competency-based program has been instrumental in refining and advancing the program.

Effects of a curricular revision on learner outcomes in veterinary clinical pathology

A mixed-methods evaluation was conducted to study learner attitudes and knowledge about clinical pathology across a curricular change that instituted a stand-alone clinical pathology course in place of content within a previously integrated pathology course structure. Groups of pre- and post-change students were assessed three times across the two semesters leading up to graduation. At each time, rank-ordered and open-ended response items probed attitudes, and multiple-choice items assessed knowledge. Data about student clinical pathology performance were also collected from clinical pathology instructors and supervising clinicians. Student rank-ordered items were evaluated by factor analysis; resulting factor-scale scores, multiple-choice scores, and rank responses from study cohorts were statistically assessed between groups and within each group over time. Intraclass correlations were calculated for the coding of student open-ended responses, and all coded responses were compared among groups. Analysis revealed that students in the revised curriculum had greater satisfaction with their training and greater confidence in data interpretation compared to students without exposure to an independent clinical pathology course. Although differences in knowledge of clinical pathology were not detected, it was also apparent that the independent clinical pathology course filled a student-perceived curricular need without raising criticisms related to diminished integration with anatomic pathology. Secondary study outcomes included formative feedback for course improvement, evidence of clerkship efficacy, and baseline data for further studies.

Developments in undergraduate teaching of small-animal soft-tissue surgical skills at the University of Sydney

This article discusses recent developments in soft-tissue surgery teaching at the University of Sydney, Faculty of Veterinary Science. An integrated teaching program was developed for Bachelor of Veterinary Science (BVSc) students with the aim of providing them with optimal learning opportunities to meet "Day One" small-animal soft-tissue surgical competencies. Didactic lectures and tutorials were introduced earlier into the curriculum to prepare students for live-animal surgery practical. In addition to existing clinics, additional spay/neuter clinics were established in collaboration with animal welfare organizations to increase student exposure to live-animal surgery. A silicon-based, life-like canine ovariohysterectomy model was developed with the assistance of a model-making and special effects company. The model features elastic ovarian pedicles and suspensory ligaments, which can be stretched and broken like those of an actual dog. To monitor the volume and type of student surgical experience, an E-portfolio resource was established. This resource allows for the tracking of numbers of live, student-performed desexing surgeries and incorporates competency-based assessments and reflective tasks to be completed by students. Student feedback on the integrated surgical soft-tissue teaching program was assessed. Respondents were assessed in the fourth year of the degree and will have further opportunities to develop Day One small-animal soft-tissue surgical competencies in the fifth year. Ninety-four percent of respondents agreed or strongly agreed that they were motivated to participate in all aspects of the program, while 78% agreed or strongly agreed that they received an adequate opportunity to develop their skills and confidence in ovariohysterectomy or castration procedures through the fourth-year curriculum.

Web-based documentation of clinical skills to assess the competency of veterinary students

Kansas State University implemented a Web-based program to assess students' competency to perform technical skills during clinical rotations throughout the fourth year of the veterinary curriculum. The classes of 2009 and 2010 recorded a minimum number of procedures (104 and 103, respectively) from a menu of more than 220 recommended procedures. Procedures were categorized by species (small animal, equine, food animal) and disciplines (imaging, anesthesia, diagnostic medicine/necropsy). Ophthalmology was added as a fourth discipline for the class of 2010. Students recorded procedures into the Web-based system, including information about the patient, procedure performed, supervisor, and a self-assessment of performance. Faculty, staff, and house officers evaluated the procedures electronically by confirming that they witnessed the procedure and providing qualitative and written feedback. The class of 2009 recorded 18,492 procedures (M=171/student) and the class of 2010 recorded 16,935 procedures (M=158/student). Two students from each class (2009 and 2010) did not complete the minimum required skills during clinical rotations and returned to perform procedures immediately before (n=3) or immediately after (n=1) graduation to receive their diploma. The Web-based system captured a large number of assessments of technical competency performed in the clinical setting. The system provided students with formative feedback throughout the clinical year, ensured equitable distribution of procedural opportunities across the student body, and required minimal additional resources.

Assessment of first-year veterinary students' clinical skills using objective structured clinical examinations

The DVM program at the University of Calgary offers a Clinical Skills course each year for the first three years. The course is designed to teach students the procedural skills required for entry-level general veterinary practice. Objective Structured Clinical Examinations (OSCEs) were used to assess students' performance on these procedural skills. A series of three OSCEs were developed for the first year. Content was determined by an exam blueprint, exam scoring sheets were created, rater training was provided, a mock OSCE was performed with faculty and staff, and the criterion-referencing Ebel method was used to set cut scores for each station using two content experts. Each station and the overall exam were graded as pass or fail. Thirty first-year DVM students were assessed. Content validity was ensured by the exam blueprint and expert review. Reliability (coefficient α) of the stations from the three OSCE exams ranged from 0.0 to 0.71. The three exam reliabilities (Generalizability Theory) were, for OSCE 1, G=0.56; OSCE 2, G=0.37; and OSCE 3, G=0.32. Preliminary analysis has suggested that the OSCEs demonstrate face and content validity, and certain stations demonstrated adequate reliability. Overall exam reliability was low, which reflects issues with first-time exam delivery. Because this year was the first that this course was taught and this exam format was used, work continues in the program on the teaching of the procedural skills and the development and revision of OSCE stations and scoring checklists.

Evaluation of radiographic interpretation competence of veterinary students in Finland

In the evaluation of the clinical competence of veterinary students, many different definitions and methods are approved. Due to the increasing discussion of the quality of outcomes produced by newly graduated veterinarians, methods for the evaluation of clinical competencies should also be evaluated. In this study, this was done by comparing two qualitative evaluation schemes: the well-known structure of observed learning outcome (SOLO) taxonomy and a modification of this taxonomy. A case-based final radiologic examination was selected and the investigation was performed by classifying students' outcomes. These classes were finally put next to original (quantitative) scores and the statistical calculations were initiated. Significant correlations between taxonomies (0.53) and the modified taxonomy and original scores (0.66) were found and some qualitative similarities between evaluation methods were observed. In addition, some supplements were recommended for the structure of evaluation schemes, especially for the structure of the modified SOLO taxonomy.