1
|
Saberzadeh-Ardestani B, Sima AR, Khosravi B, Young M, Mortaz Hejri S. The impact of prior performance information on subsequent assessment: is there evidence of retaliation in an anonymous multisource assessment system? ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2024; 29:531-550. [PMID: 37488326 DOI: 10.1007/s10459-023-10267-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/05/2023] [Accepted: 07/16/2023] [Indexed: 07/26/2023]
Abstract
Few studies have engaged in data-driven investigations of the presence, or frequency, of what could be considered retaliatory assessor behaviour in Multi-source Feedback (MSF) systems. In this study, authors explored how assessors scored others if, before assessing others, they received their own assessment score. The authors examined assessments from an established MSF system in which all clinical team members - medical students, interns, residents, fellows, and supervisors - anonymously assessed each other. The authors identified assessments in which an assessor (i.e., any team member providing a score to another) gave an aberrant score to another individual. An aberrant score was defined as one that was more than two standard deviations from the assessment receiver's average score. Assessors who gave aberrant scores were categorized according to whether their behaviour was preceded by: (1) receiving a score or not from another individual in the MSF system (2) whether the score they received was aberrant or not. The authors used a multivariable logistic regression model to investigate the association between the type of score received and the type of score given by that same individual. In total, 367 unique assessors provided 6091 scores on the performance of 484 unique individuals. Aberrant scores were identified in 250 forms (4.1%). The chances of giving an aberrant score were 2.3 times higher for those who had received a score, compared to those who had not (odds ratio 2.30, 95% CI:1.54-3.44, P < 0.001). Individuals who had received an aberrant score were also 2.17 times more likely to give an aberrant score to others compared to those who had received a non-aberrant score (2.17, 95% CI:1.39-3.39, P < 0.005) after adjusting for all other variables. This study documents an association between receiving scores within an anonymous multi-source feedback (MSF) system and providing aberrant scores to team members. These findings suggest care must be given to designing MSF systems to protect against potential downstream consequences of providing and receiving anonymous feedback.
Collapse
Affiliation(s)
- Bahar Saberzadeh-Ardestani
- Digestive Disease Research Center, Digestive Disease Research Institute, Tehran University of Medical Sciences, Tehran, Iran
| | - Ali Reza Sima
- Digestive Disease Research Center, Digestive Disease Research Institute, Tehran University of Medical Sciences, Tehran, Iran
| | - Bardia Khosravi
- Digestive Disease Research Center, Digestive Disease Research Institute, Tehran University of Medical Sciences, Tehran, Iran
| | - Meredith Young
- Institute of Health Sciences Education, McGill University, Montreal, QC, Canada
| | - Sara Mortaz Hejri
- Department of Medical Education, School of Medicine, Tehran University of Medical Sciences, Tehran, Iran.
| |
Collapse
|
2
|
Neuwirt H, Eder IE, Gauckler P, Horvath L, Koeck S, Noflatscher M, Schaefer B, Simeon A, Petzer V, Prodinger WM, Berendonk C. Impact of familiarity with the format of the exam on performance in the OSCE of undergraduate medical students - an interventional study. BMC MEDICAL EDUCATION 2024; 24:179. [PMID: 38395807 PMCID: PMC10893607 DOI: 10.1186/s12909-024-05091-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/23/2023] [Accepted: 01/24/2024] [Indexed: 02/25/2024]
Abstract
BACKGROUND Assessments, such as summative structured examinations, aim to verify whether students have acquired the necessary competencies. It is important to familiarize students with the examination format prior to the assessment to ensure that true competency is measured. However, it is unclear whether students can demonstrate their true potential or possibly perform less effectively due to the unfamiliar examination format. Hence, we questioned whether a 10-min active familiarization in the form of simulation improved medical students´ OSCE performance. Next, we wanted to elucidate whether the effect depends on whether the familiarization procedure is active or passive. METHODS We implemented an intervention consisting of a 10-min active simulation to prepare the students for the OSCE setting. We compared the impact of this intervention on performance to no intervention in 5th-year medical students (n = 1284) from 2018 until 2022. Recently, a passive lecture, in which the OSCE setting is explained without active participation of the students, was introduced as a comparator group. Students who participated in neither the intervention nor the passive lecture group formed the control group. The OSCE performance between the groups and the impact of gender was assessed using X2, nonparametric tests and regression analysis (total n = 362). RESULTS We found that active familiarization of students (n = 188) yields significantly better performance compared to the passive comparator (Cohen´s d = 0.857, p < 0.001, n = 52) and control group (Cohen´s d = 0.473, p < 0.001, n = 122). In multivariate regression analysis, active intervention remained the only significant variable with a 2.945-fold increase in the probability of passing the exam (p = 0.018). CONCLUSIONS A short 10-min active intervention to familiarize students with the OSCE setting significantly improved student performance. We suggest that curricula should include simulations on the exam setting in addition to courses that increase knowledge or skills to mitigate the negative effect of nonfamiliarity with the OSCE exam setting on the students.
Collapse
Affiliation(s)
- Hannes Neuwirt
- Department of Internal Medicine IV - Nephrology and Hypertension, Medical University of Innsbruck, Anichstrasse 35, 6020, Innsbruck, Austria.
| | - Iris E Eder
- Department of Urology, Medical University of Innsbruck, Anichstrasse 35, 6020, Innsbruck, Austria
| | - Philipp Gauckler
- Department of Internal Medicine IV - Nephrology and Hypertension, Medical University of Innsbruck, Anichstrasse 35, 6020, Innsbruck, Austria
| | - Lena Horvath
- Department of Internal Medicine V - Hematology and Oncology, Medical University of Innsbruck, Anichstrasse 35, 6020, Innsbruck, Austria
| | - Stefan Koeck
- Department of Internal Medicine V - Hematology and Oncology, Medical University of Innsbruck, Anichstrasse 35, 6020, Innsbruck, Austria
| | - Maria Noflatscher
- Department of Internal Medicine III - Cardiology and Angiology, Medical University of Innsbruck, Anichstrasse 35, 6020, Innsbruck, Austria
| | - Benedikt Schaefer
- Department of Internal Medicine I - Gastroenterology, Hepatology and Endocrinology, Medical University of Innsbruck, Anichstrasse 35, 6020, Innsbruck, Austria
| | - Anja Simeon
- Department of Internal Medicine II, Medical University of Innsbruck, Anichstrasse 35, 6020, Innsbruck, Austria
| | - Verena Petzer
- Department of Internal Medicine V - Hematology and Oncology, Medical University of Innsbruck, Anichstrasse 35, 6020, Innsbruck, Austria
| | - Wolfgang M Prodinger
- Vice Rectorate for Teaching and Study Affairs, Medical University of Innsbruck, Fritz-Pregl-Strasse 3, 6020, Innsbruck, Austria
| | - Christoph Berendonk
- Institute for Medical Education (IML) - Assessment and Evaluation Unit (AAE), University of Bern, Mittelstrasse 43, CH-3012, Bern, Switzerland
| |
Collapse
|
3
|
Yuan EJS, Huang SS, Hsu CA, Lirng JF, Li TH, Huang CC, Yang YY, Li CP, Chen CH. Negative effects on medical students' scores for clinical performance during the COVID-19 pandemic in Taiwan: a comparative study. JOURNAL OF EDUCATIONAL EVALUATION FOR HEALTH PROFESSIONS 2023; 20:37. [PMID: 38148494 PMCID: PMC10810719 DOI: 10.3352/jeehp.2023.20.37] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/19/2023] [Accepted: 12/12/2023] [Indexed: 12/28/2023]
Abstract
PURPOSE Coronavirus disease 2019 (COVID-19) has heavily impacted medical clinical education in Taiwan. Medical curricula have been altered to minimize exposure and limit transmission. This study investigated the effect of COVID-19 on Taiwanese medical students' clinical performance using online standardized evaluation systems and explored the factors influencing medical education during the pandemic. METHODS Medical students were scored from 0 to 100 based on their clinical performance from 1/1/2018 to 6/31/2021. The students were placed into pre-COVID-19 (before 2/1/2020) and midst-COVID-19 (on and after 2/1/2020) groups. Each group was further categorized into COVID-19-affected specialties (pulmonary, infectious, and emergency medicine) and other specialties. Generalized estimating equations (GEEs) were used to compare and examine the effects of relevant variables on student performance. RESULTS In total, 16,944 clinical scores were obtained for COVID-19-affected specialties and other specialties. For the COVID-19-affected specialties, the midst-COVID-19 score (88.513.52) was significantly lower than the pre-COVID-19 score (90.143.55) (P<0.0001). For the other specialties, the midst-COVID-19 score (88.323.68) was also significantly lower than the pre-COVID-19 score (90.063.58) (P<0.0001). There were 1,322 students (837 males and 485 females). Male students had significantly lower scores than female students (89.333.68 vs. 89.993.66, P=0.0017). GEE analysis revealed that the COVID-19 pandemic (unstandardized beta coefficient=-1.99, standard error [SE]=0.13, P<0.0001), COVID-19-affected specialties (B=0.26, SE=0.11, P=0.0184), female students (B=1.10, SE=0.20, P<0.0001), and female attending physicians (B=-0.19, SE=0.08, P=0.0145) were independently associated with students' scores. CONCLUSION COVID-19 negatively impacted medical students' clinical performance, regardless of their specialty. Female students outperformed male students, irrespective of the pandemic.
Collapse
Affiliation(s)
- Eunice Jia-Shiow Yuan
- Department of Medical Education, Taipei Veterans General Hospital, Taipei, Taiwan
- Department of Family Medicine, Taipei Veterans General Hospital, Taipei, Taiwan
- School of Medicine, College of Medicine, National Yang Ming Chiao Tung University, Taipei, Taiwan
| | - Shiau-Shian Huang
- Department of Medical Education, Taipei Veterans General Hospital, Taipei, Taiwan
- School of Medicine, College of Medicine, National Yang Ming Chiao Tung University, Taipei, Taiwan
- School of Public Health, National Defense Medical Center, Taipei, Taiwan
| | - Chia-An Hsu
- Department of Medical Education, Taipei Veterans General Hospital, Taipei, Taiwan
- School of Medicine, College of Medicine, National Yang Ming Chiao Tung University, Taipei, Taiwan
- Department of Ophthalmology, Taipei Veterans General Hospital, Taipei, Taiwan
| | - Jiing-Feng Lirng
- School of Medicine, College of Medicine, National Yang Ming Chiao Tung University, Taipei, Taiwan
- Department of Radiology, Taipei Veterans General Hospital, Taipei, Taiwan
| | - Tzu-Hao Li
- Department of Medical Education, Taipei Veterans General Hospital, Taipei, Taiwan
- School of Medicine, College of Medicine, National Yang Ming Chiao Tung University, Taipei, Taiwan
- Division of Allergy, Immunology, and Rheumatology, Department of Internal Medicine, Shin Kong Wu Ho-Su Memorial Hospital, Taipei, Taiwan
| | - Chia-Chang Huang
- Department of Medical Education, Taipei Veterans General Hospital, Taipei, Taiwan
- School of Medicine, College of Medicine, National Yang Ming Chiao Tung University, Taipei, Taiwan
- Division of Clinical Skills Training, Department of Medical Education, Taipei Veterans General Hospital, Taipei, Taiwan
- Clinical Innovation Center, Department of Medical Education, Taipei Veterans General Hospital, Taipei, Taiwan
| | - Ying-Ying Yang
- Department of Medical Education, Taipei Veterans General Hospital, Taipei, Taiwan
- School of Medicine, College of Medicine, National Yang Ming Chiao Tung University, Taipei, Taiwan
- Clinical Innovation Center, Department of Medical Education, Taipei Veterans General Hospital, Taipei, Taiwan
| | - Chung-Pin Li
- School of Medicine, College of Medicine, National Yang Ming Chiao Tung University, Taipei, Taiwan
- Division of Clinical Skills Training, Department of Medical Education, Taipei Veterans General Hospital, Taipei, Taiwan
- Division of Gastroenterology and Hepatology, Department of Medicine, Taipei Veterans General Hospital, Taipei, Taiwan
- Therapeutic and Research Center of Pancreatic Cancer, Taipei Veterans General Hospital, Taipei, Taiwan
| | - Chen-Huan Chen
- Department of Medical Education, Taipei Veterans General Hospital, Taipei, Taiwan
- School of Medicine, College of Medicine, National Yang Ming Chiao Tung University, Taipei, Taiwan
| |
Collapse
|
4
|
Zavodnick J, Doroshow J, Rosenberg S, Banks J, Leiby BE, Mingioni N. Hawks and Doves: Perceptions and Reality of Faculty Evaluations. JOURNAL OF MEDICAL EDUCATION AND CURRICULAR DEVELOPMENT 2023; 10:23821205231197079. [PMID: 37692558 PMCID: PMC10492463 DOI: 10.1177/23821205231197079] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/27/2023] [Accepted: 08/08/2023] [Indexed: 09/12/2023]
Abstract
OBJECTIVES Internal medicine clerkship grades are important for residency selection, but inconsistencies between evaluator ratings threaten their ability to accurately represent student performance and perceived fairness. Clerkship grading committees are recommended as best practice, but the mechanisms by which they promote accuracy and fairness are not certain. The ability of a committee to reliably assess and account for grading stringency of individual evaluators has not been previously studied. METHODS This is a retrospective analysis of evaluations completed by faculty considered to be stringent, lenient, or neutral graders by members of a grading committee of a single medical college. Faculty evaluations were assessed for differences in ratings on individual skills and recommendations for final grade between perceived stringency categories. Logistic regression was used to determine if actual assigned ratings varied based on perceived faculty's grading stringency category. RESULTS "Easy graders" consistently had the highest probability of awarding an above-average rating, and "hard graders" consistently had the lowest probability of awarding an above-average rating, though this finding only reached statistical significance only for 2 of 8 questions on the evaluation form (P = .033 and P = .001). Odds ratios of assigning a higher final suggested grade followed the expected pattern (higher for "easy" and "neutral" compared to "hard," higher for "easy" compared to "neutral") but did not reach statistical significance. CONCLUSIONS Perceived differences in faculty grading stringency have basis in reality for clerkship evaluation elements. However, final grades recommended by faculty perceived as "stringent" or "lenient" did not differ. Perceptions of "hawks" and "doves" are not just lore but may not have implications for students' final grades. Continued research to describe the "hawk and dove effect" will be crucial to enable assessment of local grading variation and empower local educational leadership to correct, but not overcorrect, for this effect to maintain fairness in student evaluations.
Collapse
Affiliation(s)
- Jillian Zavodnick
- Sidney Kimmel Medical College, Thomas Jefferson University, Philadelphia, USA
| | | | - Sarah Rosenberg
- Sidney Kimmel Medical College, Thomas Jefferson University, Philadelphia, USA
| | - Joshua Banks
- Department of Pharmacology and Experimental Therapeutics, Division of Biostatistics, Thomas Jefferson University, Philadelphia, USA
| | - Benjamin E Leiby
- Department of Pharmacology and Experimental Therapeutics, Division of Biostatistics, Thomas Jefferson University, Philadelphia, USA
| | - Nina Mingioni
- Sidney Kimmel Medical College, Thomas Jefferson University, Philadelphia, USA
| |
Collapse
|