1
|
Ollier E, Pelissier C, Boissier C, Barjat T, Berthelot P, Boutet C, Gocko X, Le Hello C, Perinel S. [Systematic analysis of evaluations of Objective Structured Clinical Exam (OSCE) multiple circuits: Explanatory variables and inter-rater correlations]. Rev Med Interne 2024; 45:327-334. [PMID: 38643040 DOI: 10.1016/j.revmed.2024.03.013] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/20/2023] [Revised: 01/26/2024] [Accepted: 03/31/2024] [Indexed: 04/22/2024]
Abstract
INTRODUCTION Objective Structured Clinical Examinations (OSCEs) assess professional performance in a simulated environment. Following their integration into the reform of the 2nd cycle of medical studies (R2C), this pedagogical modality was implemented in France. This study investigates the variability of students' OSCE scores, as well as their inter-rater reproducibility. METHODS This single-center retrospective study covered several sessions of evaluative OSCE circuits conducted between January 2022 and June 2023. Variables collected were: baseline situation family, competency domain, presence of a standardized participant for stations; gender and professional status for evaluators; scores (global, clinical and communication skills), number of previously completed OSCE circuits and faculty scores for students. RESULTS The variability of the overall score was explained mainly (79.7%, CI95% [77.4; 82.0]) by the station factor. The student factor and the circuit factor explained 7.5% [12.9; 20.2] and<0.01% [2.10-13; 2.10-9] respectively. The inter-rater intra-class correlation coefficient was 87.2% [86.4; 87.9] for the global score. Station characteristics (starting situation, domain) and evaluator characteristics (gender, status) were significantly associated with score variations. CONCLUSION This first study on the variability of OSCE circuit scores in France shows good reproducibility with influence of station characteristics. In order to standardize circuits, variability linked to the domain competency should be considered as well.
Collapse
Affiliation(s)
- E Ollier
- Groupe ECOS de la faculté de médecine Jacques-Lisfranc, université Jean-Monnet, Saint-Étienne, France
| | - C Pelissier
- Groupe ECOS de la faculté de médecine Jacques-Lisfranc, université Jean-Monnet, Saint-Étienne, France
| | - C Boissier
- Groupe ECOS de la faculté de médecine Jacques-Lisfranc, université Jean-Monnet, Saint-Étienne, France
| | - T Barjat
- Groupe ECOS de la faculté de médecine Jacques-Lisfranc, université Jean-Monnet, Saint-Étienne, France
| | - P Berthelot
- Groupe ECOS de la faculté de médecine Jacques-Lisfranc, université Jean-Monnet, Saint-Étienne, France
| | - C Boutet
- Groupe ECOS de la faculté de médecine Jacques-Lisfranc, université Jean-Monnet, Saint-Étienne, France
| | - X Gocko
- Groupe ECOS de la faculté de médecine Jacques-Lisfranc, université Jean-Monnet, Saint-Étienne, France
| | - C Le Hello
- Groupe ECOS de la faculté de médecine Jacques-Lisfranc, université Jean-Monnet, Saint-Étienne, France
| | - S Perinel
- Groupe ECOS de la faculté de médecine Jacques-Lisfranc, université Jean-Monnet, Saint-Étienne, France; Faculté de médecine Jacques-Lisfranc, campus santé innovations, 10, rue de la Marandière, 42270 Saint-Priest-en-Jarez, France.
| |
Collapse
|
2
|
Brown C, Khavandi S, Sebastian A, Badger K, Westacott R, Reed MWR, Gurnell M, Sam AH. The influence of candidates' race on examiners' ratings in standardised assessments of clinical practice. MEDICAL TEACHER 2024:1-6. [PMID: 38771961 DOI: 10.1080/0142159x.2024.2345266] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/03/2023] [Accepted: 04/16/2024] [Indexed: 05/23/2024]
Abstract
PURPOSE Delivering fair and reliable summative assessments in medical education assumes examiner decision making is devoid of bias. We investigated whether candidate racial appearances influenced examiner ratings in undergraduate clinical exams. METHODS We used an internet-based design. Examiners watched a randomised set of six videos of three different white candidates and three different non-white (Asian, black and Chinese) candidates taking a clinical history at either fail, borderline or pass grades. We compared the median and interquartile range (IQR) of the paired difference between scores for the white and non-white candidates at each performance grade and tested for statistical significance. RESULTS 160 Examiners participated. At the fail grade, the black and Chinese candidates scored lower than the white candidate, with median paired differences of -2.5 and -1 respectively (both p < 0.001). At the borderline grade, the black and Chinese candidates scored higher than the white candidate, with median paired differences of +2 and +3, respectively (both p < 0.001). At the passing grade, the Asian candidate scored lower than the white candidate (median paired difference -1, p < 0.001). CONCLUSION The racial appearance of candidates appeared to influence the scores awarded by examiners, but not in a uniform manner.
Collapse
Affiliation(s)
- Celia Brown
- Division of Health Sciences, Warwick Medical School, Coventry, UK
| | - Sarah Khavandi
- Imperial College School of Medicine, Imperial College London, London, UK
| | - Ann Sebastian
- Imperial College School of Medicine, Imperial College London, London, UK
| | - Kerry Badger
- Imperial College School of Medicine, Imperial College London, London, UK
| | - Rachel Westacott
- Birmingham Medical School, University of Birmingham, Birmingham, UK
| | - Malcolm W R Reed
- Brighton and Sussex Medical School, University of Sussex, Brighton, UK
| | - Mark Gurnell
- Wellcome-MRC Institute of Metabolic Science, University of Cambridge and NIHR Cambridge Biomedical Research Centre, Cambridge University Hospitals, Cambridge, UK
| | - Amir H Sam
- Imperial College School of Medicine, Imperial College London, London, UK
| |
Collapse
|
3
|
Mechtouff L, Balanca B, Jung J, Bourgeois-Vionnet J, Dumot C, Guery D, Picart T, Bapteste L, Demarquay G, Bani-Sadr A, Rascle L, Berthezène Y, Jacquesson T, Amaz C, Macabrey J, Ramos I, Viprey M, Rode G, Cortet M. Interrater reliability in neurology objective structured clinical examination across specialties. MEDICAL TEACHER 2024; 46:239-244. [PMID: 37605843 DOI: 10.1080/0142159x.2023.2244146] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/23/2023]
Abstract
PURPOSE To assess interrater reliability and examiners' characteristics, especially specialty, associated with scoring of neurology objective structured clinical examination (OSCE). MATERIAL AND METHODS During a neurology mock OSCE, five randomly chosen students volunteers were filmed while performing 1 of the 5 stations. Video recordings were scored by physicians from the Lyon and Clermont-Ferrand university teaching hospitals to assess students performance using both a checklist scoring and a global rating scale. Interrater reliability between examiners were assessed using intraclass coefficient correlation. Multivariable linear regression models including video recording as random effect dependent variable were performed to detect factors associated with scoring. RESULTS Thirty examiners including 15 (50%) neurologists participated. The intraclass correlation coefficient of checklist scores and global ratings between examiners were 0.71 (CI95% [0.45-0.95]) and 0.54 (CI95% [0.28-0.91]), respectively. In multivariable analyses, no factor was associated with checklist scores, while male gender of examiner was associated with lower global rating (β coefficient = -0.37; CI 95% [-0.62-0.11]). CONCLUSIONS Our study demonstrated through a video-based scoring method that agreement among examiners was good using checklist scoring while moderate using global rating scale in neurology OSCE. Examiner's specialty did not affect scoring whereas gender was associated with global rating scale.
Collapse
Affiliation(s)
| | - Baptiste Balanca
- Anesthesiology and Intensive Care Medicine, Hospices Civils de Lyon, Lyon, France
| | - Julien Jung
- Department of Functional Neurology and Epileptology, Hospices Civils de Lyon, Lyon, France
| | | | - Chloé Dumot
- Department of Vascular and Tumoral Neurosurgery, Hospices Civils de Lyon, Lyon, France
| | - Déborah Guery
- Department of Functional Neurology and Epileptology, Hospices Civils de Lyon, Lyon, France
| | - Thiébaud Picart
- Department of Neurosurgery, Hospices Civils de Lyon, Lyon, France
| | - Lionel Bapteste
- Anesthesiology and Intensive Care Medicine, Hospices Civils de Lyon, Lyon, France
| | - Geneviève Demarquay
- Department of Functional Neurology and Epileptology, Hospices Civils de Lyon, Lyon, France
| | | | - Lucie Rascle
- Stroke Department, Hospices Civils de Lyon, Lyon, France
| | - Yves Berthezène
- Neuroradiology Department, Hospices Civils de Lyon, Lyon, France
| | | | - Camille Amaz
- Clinical Investigation Center, Hospices Civils de Lyon, Lyon, France
| | - Juliette Macabrey
- RESearch on HealthcAre Performance RESHAPE, INSERM U1290, Université Claude Bernard, Lyon, France
| | - Inès Ramos
- RESearch on HealthcAre Performance RESHAPE, INSERM U1290, Université Claude Bernard, Lyon, France
| | - Marie Viprey
- RESearch on HealthcAre Performance RESHAPE, INSERM U1290, Université Claude Bernard, Lyon, France
| | - Gilles Rode
- Physical Medicine and Rehabilitation Department, Henry Gabrielle Hospital, Hospices Civils de Lyon, Saint-Genis-Laval, France
| | - Marion Cortet
- Gynecology and Obstetrics Department, Croix Rousse Hospital, Hospices Civils de Lyon, Lyon, France
| |
Collapse
|
4
|
Sheppard M, Johnson S, Quiroz V, Ward J. Interactions between the sex of the clinician grader and the sex of the chiropractic student intern on spinal manipulation assessment grade. THE JOURNAL OF CHIROPRACTIC EDUCATION 2023; 37:157-161. [PMID: 37655808 PMCID: PMC11095654 DOI: 10.7899/jce-22-12] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/03/2022] [Revised: 01/08/2023] [Accepted: 07/03/2023] [Indexed: 09/02/2023]
Abstract
OBJECTIVE The purpose of this project was to determine if there was any relationship between the sex of the clinician grader and the sex of the chiropractic student intern on student spinal manipulation assessment grades. METHODS Twelve thousand six hundred and thirty-one supervised patient adjustments by student interns were analyzed over a 3-year data collection window. Student interns were assessed by multiple male and female clinicians in a teaching clinic using a modified Dreyfus model scoring system on a 1-4 scale (1 = novice, 4 = proficient). A Mann-Whitney U test was used to compare the relationship between grader sex and student grade as well as student sex and student grade. RESULTS Sex of the grader had a statistically significant effect on spinal manipulation assessment grade, p < .001, with male clinician graders assigning average scores of 2.81 ± 0.39 (mean ± SD) and female clinician graders scores of 3.01 ± 0.52, r = .18. Sex of the student had a statistically significant but negligible (r = .08) effect on spinal manipulation assessment grade, p < .001, with male students averaging slightly higher scores (2.93 ± 0.47) than females (2.86 ± 0.44) on the modified Dreyfus scale. CONCLUSION Male clinicians tended to assign lower grades on spinal manipulation assessments than female clinicians. Male students on average received slightly higher scores than female students on spinal manipulation assessments.
Collapse
|
5
|
Duran M, Hendrix JM. The Role of Videotaped Clinical Skills Aggregated Peer Evaluations in the Enhancement of Evaluation Skills of Individual Medical School Faculty Members. Cureus 2023; 15:e37871. [PMID: 37091487 PMCID: PMC10116999 DOI: 10.7759/cureus.37871] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 04/18/2023] [Indexed: 04/25/2023] Open
Abstract
The conventional methodology for appraising medical students has some limitations like inherent subjectivity, unstructured nature, and bias. Implementing the Objective Structured Clinical Examination (OSCE) can mitigate these shortcomings. However, the OSCE presents challenges, including substantial financial costs and time-intensive processes, particularly when assessing a large cohort of students. Consequently, an alternative assessment was needed to keep the advantages of OSCE and mask its limitations, especially in resource-constrained settings. In addition, many scholars have expressed concern over medical students' and interns' inadequate interviewing and physical examination competencies in recent years. Due to easy availability, videotaping is a convenient method for objectively observing students for aggregate review by faculty in order to ascertain what exactly faculty are assessing during medical student evaluations. This technique allows for aggregate faculty group improvement in the ability of educators to assess students' technical proficiency, data collection capabilities, standardized patient interaction demeanor, and strategies for fostering standardized patient comfort. Nonetheless, aggregate evaluation of videotape recordings for faculty assessment development or reliance on verbal feedback from medical students about faculty's ability to assess student skill is a matter of debate. Due to increased subspecialization, the subspecialist or specialist examiners face difficulties in assessing students' skills in specific and/or generalized domains. Despite these challenges, assessing the ability of faculty members' observation and subsequent evaluation of medical trainees remains a vital aspect of assessment throughout various specialties. This paper presents the concept of faculty members individually observing and rating premade recordings of standardized students performing clinical skills for evaluation so that when the individual faculty members' ratings are aggregated and summarized collectively for evaluation by the faculty members as a group, both the group and the individual faculty members will gain a greater understanding of what are appropriate ratings versus outlier ratings.
Collapse
Affiliation(s)
- Munevver Duran
- Medical School, University of Texas Southwestern Medical School, Dallas, USA
| | - Joseph M Hendrix
- Anesthesiology, University of Texas Southwestern Medical School, Dallas, USA
| |
Collapse
|
6
|
Milan FB, Grochowalski JH. A resource efficient and reliable standard setting method for OSCEs: Borderline regression method using standardized patients as sole raters in clinical case encounters with medical students. MEDICAL TEACHER 2022; 44:878-885. [PMID: 35234562 DOI: 10.1080/0142159x.2022.2041586] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Finding a reliable, practical and low-cost criterion-referenced standard setting method for performance-based assessments has proved challenging. The borderline regression method of standard setting for OSCEs has been shown to estimate reliable scores in studies using faculty as raters. Standardized patients (SPs) have been shown to be reliable OSCE raters but have not been evaluated as raters using this standard setting method. Our study sought to find whether SPs could be reliably used as sole raters in an OSCE of clinical encounters using the borderline regression standard setting method.SPs were trained for on a five-point global rating scale. In an OSCE for medical students, SPs completed skills checklists and the global rating scale. The borderline regression method was used to create case passing scores. We estimated the dependability of the final pass or fail decisions and the absolute dependability coefficients for global ratings, checklist scores, and case pass-score decisions using generalizability theory.The overall dependability estimate is 0.92 for pass or fail decisions for the complete OSCE. Dependability coefficients (0.70-0.86) of individual case passing scores range demonstrated high dependability.Based on our findings, the borderline regression method of standard setting can be used with SPs as sole raters in a medical student OSCE to produce a dependable passing score. For those already using SPs as raters, this can provide a practical criterion-referenced standard setting method for no additional cost or faculty time.
Collapse
Affiliation(s)
- Felise B Milan
- Clinical Skills Center, Albert Einstein College of Medicine, Bronx, NY, USA
| | | |
Collapse
|
7
|
Katz SJ, Wang D. Comparing Entrustable Professional Activity Scores Given by Faculty Physicians and Senior Trainees to First-Year Residents. Cureus 2022; 14:e25798. [PMID: 35836450 PMCID: PMC9273189 DOI: 10.7759/cureus.25798] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 06/09/2022] [Indexed: 11/05/2022] Open
|
8
|
Roman P, Ruiz-Gonzalez C, Rodriguez-Arrastia M, Granero-Molina J, Fernández-Sola C, Hernández-Padilla JM. A serious game for online-based objective structured clinical examination in nursing: A qualitative study. NURSE EDUCATION TODAY 2022; 109:105246. [PMID: 34920398 DOI: 10.1016/j.nedt.2021.105246] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/03/2021] [Revised: 11/22/2021] [Accepted: 12/09/2021] [Indexed: 06/14/2023]
Abstract
BACKGROUND The current pandemic has led to unprecedented scenarios worldwide. In this context, educational institutions had to move abruptly from in-person to online classes, having to be flexible and innovative, one of the main concerns being the semester-end examinations in practical modules. OBJECTIVE This study aimed to explore nursing students' perceptions of the use of a serious game-like model in their final online objective structured clinical examination (OSCE). DESIGN An exploratory phenomenological study was conducted during the COVID-19 pandemic, in particular during June 2020. SETTINGS This study took place at the University of Almeria with nursing students enrolled in a clinical placement module. PARTICIPANTS Eighty-nine fourth-year nursing students took part in this study. METHODS Following the educational principles of the OSCE, a virtual OSCE was designed, using the Genially platform, an online platform for creating interactive content, to set up an assessment as a story-telling game-like model. A total of 5 focus groups (FGs) were conducted afterwards via the GoogleMeet platform. In addition, 30 semi-structured interviews were performed via the same platform. The data was analysed using a content analysis approach and supported by the ATLAS.ti 8.4 software. RESULTS Our finding revealed 2 main themes and 4 sub-themes. The two main themes were (i) generating emotions and feelings in times of virtuality; including emotions and feelings experienced by students during their online assessment process, and (ii) online assessment: a potential alternative to educational barriers; describing the variability of traditional modality, its implications for learning and the acquisition of competences. CONCLUSIONS Serious game-like models, such as the story-telling game proposed, as part of their online OSCE assessment appear to be an appropriate alternative assessment method for face-to-face approaches. This paper adds new evidence on the use of innovative and state-of-art resources as part of nursing OSCE assessments in a new reality for most students and teachers. Serious game-like models in online OSCE may empower students and help them to remove perceived barriers in face-to-face assessments.
Collapse
Affiliation(s)
- Pablo Roman
- Faculty of Health Sciences, Department of Nursing Science, Physiotherapy and Medicine, University of Almeria, 04120 Almeria, Spain; Research Group CTS-451 Health Sciences, University of Almeria, Almeria, Spain; Health Research Centre, University of Almeria, Almeria, Spain.
| | - Cristofer Ruiz-Gonzalez
- Faculty of Health Sciences, Department of Nursing Science, Physiotherapy and Medicine, University of Almeria, 04120 Almeria, Spain; Torrecardenas University Hospital, 04009 Almeria, Spain
| | - Miguel Rodriguez-Arrastia
- Faculty of Health Sciences, Pre-Department of Nursing, Jaume I University, 12071 Castello de la Plana, Spain; Research Group CYS, Faculty of Health Sciences, Jaume I University, Castello de la Plana, Spain.
| | - José Granero-Molina
- Faculty of Health Sciences, Department of Nursing Science, Physiotherapy and Medicine, University of Almeria, 04120 Almeria, Spain; Research Group CTS-451 Health Sciences, University of Almeria, Almeria, Spain; Health Research Centre, University of Almeria, Almeria, Spain; Faculty of Health Sciences, Universidad Autónoma de Chile, Temuco, Chile.
| | - Cayetano Fernández-Sola
- Faculty of Health Sciences, Department of Nursing Science, Physiotherapy and Medicine, University of Almeria, 04120 Almeria, Spain; Research Group CTS-451 Health Sciences, University of Almeria, Almeria, Spain; Health Research Centre, University of Almeria, Almeria, Spain; Faculty of Health Sciences, Universidad Autónoma de Chile, Temuco, Chile.
| | - José Manuel Hernández-Padilla
- Faculty of Health Sciences, Department of Nursing Science, Physiotherapy and Medicine, University of Almeria, 04120 Almeria, Spain; Research Group CTS-451 Health Sciences, University of Almeria, Almeria, Spain; Health Research Centre, University of Almeria, Almeria, Spain; Adult, Child and Midwifery Department, School of Health and Education, Middlesex University, The Burroughs, London, United Kingdom.
| |
Collapse
|
9
|
Wadi M, Yusoff MSB, Abdul Rahim AF, Lah NAZN. Factors affecting test anxiety: a qualitative analysis of medical students' views. BMC Psychol 2022; 10:8. [PMID: 34991718 PMCID: PMC8739979 DOI: 10.1186/s40359-021-00715-2] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/16/2021] [Accepted: 12/29/2021] [Indexed: 12/03/2022] Open
Abstract
BACKGROUND Medical students are vulnerable to test anxiety (TA), which impacts their professional lives and jeopardizes the optimal health care of their patients. The qualitative exploration of TA among medical students is crucial to understanding the problem. Hence, this study examined medical students' insights into TA and their suggestions on how to reduce it. METHODS We conducted a phenomenological study on medical students at a public university. We utilized focus group discussions (FGDs) to investigate their experiences of TA. The FGDs were transcribed verbatim, and these transcripts were analyzed using Atlas.ti software. The thematic analysis followed the recommended guidelines. RESULTS Seven FGD sessions were conducted with 45 students. Three major themes emerged: the students, their academic resources, and the examiner. Each theme comprised mutually exclusive subthemes. The "students" theme was divided into negative vs. positive thoughts and self-negligence vs. self-care, "academic recources" into heavy curriculum vs. facilitative curricular aids, and "examiner" into criticism vs. feedback and strict vs. kind approaches. CONCLUSION This study provides a solid foundation for policymakers and decision makers in medical education to improve current assessment practices and student well-being. Medical students will be able to significantly alter and reduce TA if they are provided with additional psychological support and their examiners are trained on how to deal with examinees.
Collapse
Affiliation(s)
- Majed Wadi
- Medical Education Department, College of Medicine, Qassim University, Buraidah, Saudi Arabia
- Medical Education Department, School of Medical Sciences, Universiti Sains Malaysia, Kota Bharu, Kelantan Malaysia
| | - Muhamad Saiful Bahri Yusoff
- Medical Education Department, School of Medical Sciences, Universiti Sains Malaysia, Kota Bharu, Kelantan Malaysia
| | - Ahmad Fuad Abdul Rahim
- Medical Education Department, School of Medical Sciences, Universiti Sains Malaysia, Kota Bharu, Kelantan Malaysia
| | - Nik Ahmad Zuky Nik Lah
- Obstetrics and Gynecology Department, School of Medical Sciences, Universiti Sains Malaysia, Kota Bharu, Kelantan Malaysia
| |
Collapse
|