Validity, reliability, and defensibility of assessments in veterinary education

Development of the Mandarin Digit-in-Noise Test and Examination of the Effect of the Number of Digits Used in the Test

OBJECTIVES

The study aimed to develop and validate the Mandarin digit-in-noise (DIN) test using four digit (i.e., two-, three-, four-, and five-digit) sequences. Test-retest reliability and criterion validity were evaluated. How the number of digits affected the results was examined. The research might lead to more informed choice of DIN tests for populations with specific cognitive needs such as memory impairment.

DESIGN

The International Collegium of Rehabilitative Audiology guideline for developing the DIN was adapted to create test materials. The test-retest reliability and psychometric function of each digit sequence were determined among young normal-hearing adults. The criterion validity of each digit sequence was determined by comparing the measured performance of older adult hearing aid users with that obtained from two other well-established sentence-in-noise tests: the Mandarin hearing-in-noise test and the Mandarin Chinese matrix test. The relation between the speech reception thresholds (SRTs) of each digit sequence of the DIN test and working memory capacity measured using the digit span test and the reading span test were explored among older adult hearing aid users. Together, the study sample consisted of 54 young normal-hearing adults and 56 older adult hearing aid users.

RESULTS

The slopes associated with the two-, three-, four-, and five-digit DIN test were 16.58, 18.79, 20.42, and 21.09 %/dB, respectively, and the mean SRTs were -11.11, -10.99, -10.56, and -10.02 dB SNR, respectively. Test-retest SRTs did not differ by more than 0.74 dB across all digit sequences, suggesting good test-retest reliability. Spearman rank-order correlation coefficients between SRTs obtained using the DIN across the four digit (i.e., two-, three-, four-, and five-digit) sequences and the two sentence-in-noise tests were uniformly high ( rs = 0.9) across all participants, when data from all participants were considered. Results from the digit span test and reading span test correlated significantly with the results of the five-digit sequences ( rs = -0.37 and -0.42, respectively) but not with the results of the two-, three-, and four-digit sequences among older hearing aid users.

CONCLUSIONS

While the three-digit sequence was found to be appropriate for clinical use for assessment of auditory perception, the two-digit sequence could be used for hearing screening. The five-digit sequence could be difficult for older hearing aid users, and with its SRT related to working memory capacity, its use in the evaluation of speech perception should be investigated further. The Mandarin DIN test was found to be reliable, and the findings are in line with SRTs obtained using standardized sentence tests, suggesting good criterion validity.

Validation of a Rubric Used for Skills-Based Assessment of Veterinary Students Performing Simulated Ovariohysterectomy on a Model

Surgical skills are an important competency for new graduates. Simulators offer a means to train and assess veterinary students prior to their first surgical performance. A simulated ovariohysterectomy (OVH) rubric's validity was evaluated using a framework of content evidence, internal structure evidence, and evidence of relationship with other variables, specifically subsequent live surgical performance. Clinically experienced veterinarians (n = 13) evaluated the utility of each rubric item to collect evidence; each item's content validity index was calculated to determine its inclusion in the final rubric. After skills training, veterinary students (n = 57) were assessed using the OVH model rubric in March and August. Internal structure evidence was collected by video-recording 14 students' mock surgeries, each assessed by all five raters to calculate inter-rater reliability. Relationship with other variables evidence was collected by assessing 22 students performing their first live canine OVH in November. Experienced veterinarians included 22 items in the final rubric. The rubric generated scores with good to excellent internal consistency; inter-rater reliability was fair. Students' performance on the March model assessment was moderately correlated with their live surgical performance (ρ = 0.43) and moderately negatively correlated with their live surgical time (ρ = -0.42). Students' performance on the August model assessment, after a summer without surgical skills practice, was weakly correlated with their live surgical performance (ρ = 0.17). These data support validation of the simulated OVH rubric. The continued development of validated assessment instruments is critical as veterinary medicine seeks to become competency based.

If you build it, they will learn: A review of models in veterinary surgical education

Surgical skills are learned through deliberate practice, and veterinary educators are increasingly turning to models for teaching and assessing surgical skills. This review article sought to compile and review the literature specific to veterinary surgical skills models, and to discuss the themes of fidelity, educational outcomes, and validity evidence. Several literature searches using broad terms such as "veterinary surgery model," "veterinary surgical model," and "veterinary surgical simulator" were performed using PubMed, CAB abstracts, and Google scholar. All articles describing a model created and utilized for veterinary surgical training were included. Other review articles were used as a source for additional models. Commercially available models were found using review articles, internet browser searches, and communication with veterinary clinical skills educators. There has been an explosion of growth in the variety of small animal surgical task trainers published in the last several decades. These models teach orthopedic surgery, ligation and suturing, open celiotomy and abdominal surgery, sterilization surgeries, and minimally invasive surgeries. Some models were published with accompanying rubrics for learner assessment; these rubrics have been noted where present. Research in veterinary surgical models is expanding and becoming an area of focus for academic institutions. However, there is room for growth in the collection of validity evidence and in development of models for teaching large animal surgery, training surgical residents, and providing continuing education to practitioners. This review can assist with evaluation of current surgical models and trends, and provide a platform for additional studies and development of best practices.

Evaluating the contribution of a wildlife health capacity building program on orangutan conservation

One Health is increasingly being used as a tool in ecosystem protection. The Orangutan Veterinary Advisory Group (OVAG) is working to address One Health concerns in Pongo spp. (orangutan) welfare and conservation. Orangutans are vital contributors to the ecosystem health of their range areas. Strengthening national capacity is crucial to make a lasting difference in the currently bleak outlook for orangutan species survival. OVAG is a capacity strengthening and expertise network that brings together all those working with orangutans, in- and ex-situ, to share knowledge, skills, and to collectively learn. Using the One Health paradigm embedded to enhance professional development, the OVAG network is successfully supporting conservation outcomes and impact. As part of our adaptive management approach, and to assess individual and organizational change attributable to the capacity strengthening work of OVAG, we evaluated technical skill test data, program satisfaction data, and asked participants to complete a self-reflective survey. This pilot study of our work demonstrates statistically significant improvements in conservation medicine (t = 5.481, p < 0.0001) and wildlife clinical skills knowledge (t = 3.923, p < 0.001) for those in the OVAG program. Most consider OVAG participation to be either critical or very useful in their conservation medicine decision-making process, with a perceived positive impact on their skills at handling multiple situations. Additionally, participant feedback shows a sense of being able to drive positive change locally and nationally (within their own countries) as a consequence of OVAG participation. The authors hope the OVAG model including its associated capacity support mechanisms and pedagogical approaches can be used as a template for other One Health efforts.

Moving towards a contemporary chiropractic professional identity

Since the inception of the chiropractic profession, debate has continued on differing practice objectives and philosophical approaches to patient care. While the political and academic leaders of the profession continue to dominate the discourse, little is known on the perspectives of the everyday practising chiropractor on their professional identity. In this paper, professional identity within the profession of chiropractic was evaluated using a systematised search strategy of the literature from the year 2000 through to May 2019. Initially 562 articles were sourced, of which 24 met the criteria for review. The review confirmed three previously stated professional identity subgroups; two polarised approaches and a centrist or mixed view. The musculoskeletal biomedical approach is in contrast to the vertebral subluxation vitalistic practice approach. Whilst these three main chiropractic identity subtypes exist, within the literature the terminology used to describe them differs. Research aimed at categorising the chiropractic profession identity into exclusive subtypes found that at least 20% of chiropractors have an exclusive vertebral subluxation focus. However, deeper exploration of the literature shows that vertebral subluxation is an important practice consideration for up to 70% of chiropractors. Patient care with a musculoskeletal spine focus is dominant in clinical practice. This review found that practising chiropractors consider themselves to be primary care or primary contact practitioners with a broad scope of practice across a number of patient groups not limited to musculoskeletal management. Across the research, there is a marked difference in the categories of practice objectives evaluated, and future research could examine the relatedness of these. Additionally, future research could explore the professional identity construct over time and within different practice contexts to help facilitate the progression of the profession.

Student selection to a rural veterinary school. 1: applicant demographics and predictors of success within the application process

OBJECTIVE

To determine demographic details of applicants to the veterinary degree program at and relate this information to student selection processes.

DESIGN

Retrospective evaluation of applicant data.

METHODS

Database records of 4042 applicants to the veterinary program (2006-2016) were retrieved from university records. Summary statistics were used to determine basic demographic data, which were related to pre-entry academic achievement and results of selection processes using univariable and multivariable analyses.

RESULTS

The number of applications, interviews and acceptance of offers fluctuated but did not vary substantively for the duration of the study. Gender, rurality (as assessed by residential and school postcodes), socioeconomic background, academic background and pre-entry academic achievement all significantly influenced written application and interview scores in the multivariate models. Rural background, written application score, number of applications, pre-entry academic history and achievement significantly affected the chances of receiving an interview. Chances of receiving an offer and subsequent program entry were significantly influenced by academic background, interview score and pre-entry academic achievement. Male applicants were more likely to receive an offer, but gender had no effect on program entry. Rural residential and school background was associated with significantly reduced ATAR in comparison with city-based applicants.

CONCLUSION

Selection processes effectively identified applicants with demographic backgrounds and life experiences consistent with the School's commitment to the veterinary needs of rural and regional Australia. Findings support the provision of an academic loading for the ATAR results of rural students. Effects of these selection outcomes on student performance should be evaluated.

Comparison of 2 canine celiotomy closure models for training novice veterinary students

OBJECTIVE

To compare a low-fidelity foam and fabric (FF) model to a high fidelity silicone (SI) model for teaching canine celiotomy closure.

STUDY DESIGN

Prospective blinded comparison of learning outcomes.

SAMPLE POPULATION

Second-year veterinary students who had never performed surgery as a primary surgeon (n = 46) and veterinarians experienced in performing canine celiotomy (n = 10).

METHODS

Veterinary students performed a digitally recorded celiotomy closure on a canine cadaver before and after participation in 4 facilitated laboratory training sessions on their randomly assigned model. Recordings were scored by masked, trained educators with an 8-item task-specific rubric. Students completed surveys evaluating the models. Experienced veterinarians tested the models and provided feedback on their features.

RESULTS

Completed pretest and posttest recordings were available for 38 of 46 students. Students' performance improved regardless of the model used to practice (P = .04). The magnitude of improvement did not differ between the 2 groups (P = .10). All students (n = 46) described their models favorably. Ninety percent of veterinarians thought both models were helpful for training students and gave similar ratings on all measures except for realism, which was rated higher for the SI model's skin (median, agree) compared with the FF model (median, neutral, P = .02).

CONCLUSION

Model-based training was effective at improving students' surgical skills. Less experienced learners achieved similar skill gains after practicing with FF or SI models.

CLINICAL SIGNIFICANCE

The acquisition of surgical skills required to perform celiotomy closure in companion animals occurs similarly well on models made of foam and fabric or of silicone, providing flexibility in model selection.

Comparison of Two Veterinary Emergency and Critical Care Clerkship Grading Protocols

Universal guidelines for evaluating veterinary students' clinical clerkship performance are unavailable. At our institution, each service determines its own grading protocol. In this study, researchers compared clinician, staff, and student perceptions of a traditional and newly devised grading practice on the Emergency and Critical Care (ECC) clerkship. ECC clinicians and technicians were asked to assess the existing grading protocol for the clerkship. The system was then revised to better align with clerkship objectives. The revised protocol evaluated students on 12 items encompassing knowledge, clinical, and communication skills. Following the assignment of values for each category, letter grades were calculated automatically. Clinicians and staff were invited to rate the revised grading system. Throughout the study period, a corresponding survey was sent to students shortly after they had received their clerkship grades. Students' open-ended comments were analyzed qualitatively to identify common themes. Clinicians and technicians reported that the revised protocol was more inclusive and better able to provide fair and accurate assessments of students' performances. Students were generally satisfied with both grading protocols, however, in the open-ended comments students' frequently expressed desire for more directed and timely feedback on their performance. The results of this study indicate that the graders' believed that the revised evaluation protocol provides opportunities to provide fair and accurate assessments of student performance. Overall, students were satisfied with the new protocol and have a desire for tailored feedback provided in a timely fashion.

A Survey of Established Veterinary Clinical Skills Laboratories from Europe and North America: Present Practices and Recent Developments

Developing competence in clinical skills is important if graduates are to provide entry-level care, but it is dependent on having had sufficient hands-on practice. Clinical skills laboratories provide opportunities for students to learn on simulators and models in a safe environment and to supplement training with animals. Interest in facilities for developing veterinary clinical skills has increased in recent years as many veterinary colleges face challenges in training their students with traditional methods alone. For the present study, we designed a survey to gather information from established veterinary clinical skills laboratories with the aim of assisting others considering opening or expanding their own facility. Data were collated from 16 veterinary colleges in North America and Europe about the uses of their laboratory, the building and associated facilities, and the staffing, budgets, equipment, and supporting learning resources. The findings indicated that having a dedicated veterinary clinical skills laboratory is a relatively new initiative and that colleges have adopted a range of approaches to implementing and running the laboratory, teaching, and assessments. Major strengths were the motivation and positive characteristics of the staff involved, providing open access and supporting self-directed learning. However, respondents widely recognized the increasing demands placed on the facility to provide more space, equipment, and staff. There is no doubt that veterinary clinical skills laboratories are on the increase and provide opportunities to enhance student learning, complement traditional training, and benefit animal welfare.

Does a Rater's Professional Background Influence Communication Skills Assessment?

There is increasing pressure in veterinary education to teach and assess communication skills, with the Objective Structured Clinical Examination (OSCE) being the most common assessment method. Previous research reveals that raters are a large source of variance in OSCEs. This study focused on examining the effect of raters' professional background as a source of variance when assessing students' communication skills. Twenty-three raters were categorized according to their professional background: clinical sciences (n=11), basic sciences (n=4), clinical communication (n=5), or hospital administrator/clinical skills technicians (n=3). Raters from each professional background were assigned to the same station and assessed the same students during two four-station OSCEs. Students were in year 2 of their pre-clinical program. Repeated-measures ANOVA results showed that OSCE scores awarded by the rater groups differed significantly: (F(matched_station_1) [2,91]=6.97, p=.002), (F(matched_station_2) [3,90]=13.95, p=.001), (F(matched_station_3) [3,90]=8.76, p=.001), and ((Fmatched_station_4) [2,91]=30.60, p=.001). A significant time effect between the two OSCEs was calculated for matched stations 1, 2, and 4, indicating improved student performances. Raters with a clinical communication skills background assigned scores that were significantly lower compared to the other rater groups. Analysis of written feedback provided by the clinical sciences raters showed that they were influenced by the students' clinical knowledge of the case and that they did not rely solely on the communication checklist items. This study shows that it is important to consider rater background both in recruitment and training programs for communication skills' assessment.

Effects of a curricular revision on learner outcomes in veterinary clinical pathology

A mixed-methods evaluation was conducted to study learner attitudes and knowledge about clinical pathology across a curricular change that instituted a stand-alone clinical pathology course in place of content within a previously integrated pathology course structure. Groups of pre- and post-change students were assessed three times across the two semesters leading up to graduation. At each time, rank-ordered and open-ended response items probed attitudes, and multiple-choice items assessed knowledge. Data about student clinical pathology performance were also collected from clinical pathology instructors and supervising clinicians. Student rank-ordered items were evaluated by factor analysis; resulting factor-scale scores, multiple-choice scores, and rank responses from study cohorts were statistically assessed between groups and within each group over time. Intraclass correlations were calculated for the coding of student open-ended responses, and all coded responses were compared among groups. Analysis revealed that students in the revised curriculum had greater satisfaction with their training and greater confidence in data interpretation compared to students without exposure to an independent clinical pathology course. Although differences in knowledge of clinical pathology were not detected, it was also apparent that the independent clinical pathology course filled a student-perceived curricular need without raising criticisms related to diminished integration with anatomic pathology. Secondary study outcomes included formative feedback for course improvement, evidence of clerkship efficacy, and baseline data for further studies.

Assessment of first-year veterinary students' communication skills using an objective structured clinical examination: the importance of context

Communication skills are considered to be a core clinical skill in veterinary medicine and essential for practice success, including outcomes of care for patients and clients. While veterinary schools include communication skills training in their programs, there is minimal knowledge on how best to assess communication competence throughout the undergraduate program. The purpose of this study was to further our understanding of the reliability, utility, and suitability of a communication skills Objective Structured Clinical Examination (OSCE). Specifically we wanted to (1) identify the greatest source of variability (student, rater, station, and track) within a first-year, four station OSCE using exam scores and scores from videotape review by two trained raters, and (2) determine the effect of different stations on students' communication skills performance. Reliability of the scores from both the exam data and the two expert raters was 0.50 and 0.46 respectively, with the greatest amount of variance attributable to student by station. The percentage of variance due to raters in the exam data was 16.35%, whereas the percentage of variance for the two expert raters was 0%. These results have three important implications. First, the results reinforce the need for communication educators to emphasize that use of communication skills is moderated by the context of the clinical interaction. Second, by increasing rater training the amount of error in the scores due to raters can be reduced and inter-rater reliability increases. Third, the communication assessment method (in this case the OSCE checklist) should be built purposefully, taking into consideration the context of the case.

A review of student evaluation of teaching: applications to veterinary medical education

Student evaluation of teaching is ubiquitous to teaching in colleges and universities around the world. Since the implementation of student evaluations in the 1970s in the US, considerable research has been devoted to their appropriate use as a means of judging the effectiveness of teaching. The present article aims to (1) examine the evidence for the reliability, validity, and utility of student ratings; (2) provide seven guidelines for ways to identify effective instruction, given that the purpose of student evaluation is to assess effective teaching; and (3) conclude with recommendations for the integration of student ratings into the continuous evaluation of veterinary medical education.

Workplace-based assessment in a primary-care setting

Workplace-based assessment (WBA) is the process of directly observing students' work within a clinical setting, assessing their performance, and providing specific, goal oriented feedback. Assessment methods used for workplace-based assessment include tools developed for clinical interaction (e.g., the mini clinical evaluation exercise [mini-CEX]), for procedural or technical skills (e.g., the direct observation of procedural skills [DOPS]), and multi-source feedback tools to assess interpersonal and technical skills. While several of these assessment methods are being adopted by veterinary schools to evaluate students' progress through their clinical rotations, there is little reported at this time about their utility and effectiveness within the veterinary context. This article provides an introduction to the use of these tools and offers guidance in selecting appropriate methods for assessment in the primary health care setting.

The role of assessment in the student learning process

Assessment is a powerful learning tool that can enhance learning and education. The process of student assessment should align with curricular goals and educational objectives. Identifying the assessment strategies necessary for the proper evaluation of students' progress within individual programs is as important as establishing curricular content and delivery methods. The purpose of this paper is to discuss elements to be considered in assessment design and implementation as well as common challenges encountered during this process. Elements to be considered during assessment design include purpose of assessment, domains to be tested, and characteristics of the assessment tools to be employed. Assessment tools are evaluated according to four main characteristics: relevance, feasibility, validity, and reliability. Based on the evidence presented in the literature, the use of a variety of assessment tools is recommended to match diverse domains and learning styles. The assessment cycle concludes with the evaluation of the results and, based on these, the institution, program, or course can make changes to improve the quality of education. If assessment design aligns with educational outcomes and instructional methods, it improves the quality of education and supports student learning.