1
|
Dziadzko M, Varvinskiy A, Di Loreto R, Scipioni H, Ateleanu B, Klimek M, Berger-Estilita J. Examiner workload comparison: three structured oral examination formats for the European diploma in anaesthesiology and intensive care. MEDICAL EDUCATION ONLINE 2024; 29:2364990. [PMID: 38848480 PMCID: PMC11164053 DOI: 10.1080/10872981.2024.2364990] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/13/2024] [Accepted: 06/03/2024] [Indexed: 06/09/2024]
Abstract
The COVID-19 pandemic triggered transformations in academic medicine, rapidly adopting remote teaching and online assessments. Whilst virtual environments show promise in evaluating medical knowledge, their impact on examiner workload is unclear. This study explores examiner's workload during different European Diploma in Anaesthesiology and Intensive Care Part 2 Structured Oral Examinations formats. We hypothesise that online exams result in lower examiner's workload than traditional face-to-face methods. We also investigate workload structure and its correlation with examiner characteristics and marking performance. In 2023, examiner's workload for three examination formats (face-to-face, hybrid, online) using the NASA TLX instrument was prospectively evaluated. The impact of examiner demographics, candidate scoring agreement, and examination scores on workload was analysed. The overall NASA TLX score from 215 workload measurements in 142 examiners was high at 59.61 ± 14.13. The online examination had a statistically higher workload (61.65 ± 12.84) than hybrid but not face-to-face. Primary contributors to workload were mental and temporal demands, and effort. Online exams were associated with elevated frustration. Male examiners and those spending more time on exam preparation experienced a higher workload. Multiple diploma specialties and familiarity with European Diploma in Anaesthesiology and Intensive Care exams were protective against high workload. Perceived workload did not impact marking agreement or examination scores across all formats. Examiners experience high workload. Online exams are not systematically associated with decreased workload, likely due to frustration. Despite workload differences, no impact on examiner's performance or examination scores was found. The hybrid examination mode, combining face-to-face and online, was associated with a minor but statistically significant workload reduction. This hybrid approach may offer a more balanced and efficient examination process while maintaining integrity, cost savings, and increased accessibility for candidates.
Collapse
Affiliation(s)
- Mikhail Dziadzko
- Department of Anesthesia, Intensive Care and Pain Management, Hospices Civils de Lyon, Hôpital de la Croix Rousse, Lyon, France
- Research on Healthcare Performance (RESHAPE) U1290-INSERM, Université Claude Bernard Lyon 1, Lyon, France
| | - Andrey Varvinskiy
- South Devon Healthcare NHS Foundation Trust, Department of Anesthesia and Intensive Care, Torquay, UK
| | - Rodolphe Di Loreto
- European Society of Anaesthesiology and Intensive Care, Examinations Office, Brussels, Belgium
| | - Hugues Scipioni
- European Society of Anaesthesiology and Intensive Care, Examinations Office, Brussels, Belgium
| | - Bazil Ateleanu
- European Society of Anesthesiology and Intensive Care, Examinations Committee, Brussels, Belgium
- Department of Anaesthesia, University Hospital of Wales, Cardiff, UK
| | - Markus Klimek
- European Society of Anesthesiology and Intensive Care, Examinations Committee, Brussels, Belgium
- Department of Anaesthesiology, Erasmus University Medical Centre, Rotterdam, Netherlands
| | - Joana Berger-Estilita
- European Society of Anesthesiology and Intensive Care, Examinations Committee, Brussels, Belgium
- Institute for Medical Education, University of Bern, Bern, Switzerland
- Hirslanden Hospital Group, Institute of Anaesthesiology and Intensive Care, Salem Spital, Bern, Switzerland
- CINTESIS - Centre for Health Technology and Services Research, Faculty of Medicine, Porto, Portugal
| |
Collapse
|
2
|
Gonzalez PR, Paravattil B, Wilby KJ. Mental effort in the assessment of critical reflection: Implications for assessment quality and scoring. CURRENTS IN PHARMACY TEACHING & LEARNING 2022; 14:830-834. [PMID: 35914842 DOI: 10.1016/j.cptl.2022.06.016] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/05/2021] [Revised: 05/16/2022] [Accepted: 06/16/2022] [Indexed: 06/15/2023]
Abstract
INTRODUCTION Critical reflection is a mainstay in the training of health professionals, yet assessment of reflection is commonly described as difficult, taxing, and resulting in inconsistent scoring across assessors. At the same time, there is evidence from experiential and simulation settings that assessors' mental effort may explain assessor variability, which could be a target for simplifications in assessment design. Assessors' mental effort for assessment of reflection is currently unknown. This study aimed to determine reliability of rubric scoring of critical reflection, variation in pass-fail rates, and the relationship between reflection scores and assessors' perceived mental effort. METHODS Eleven assessors were recruited to assess six reflection assignments using a published rubric. Mental effort was measured using the Paas scale for each assignment assessed and was correlated with rubric scores for each assignment. RESULTS Findings showed inconsistency in scoring between assessors, resulting in varying pass rates for each assignment (55-100%). All assignments demonstrated negative correlations between rubric scores and perceived mental effort (r = -0.115 to -0.649). CONCLUSIONS Findings support the notion that more work should be done to optimize assessment of critical reflection. Future studies should focus on disentangling the influence on mental effort of scoring tools, assignment structures, and writing quality.
Collapse
Affiliation(s)
| | | | - Kyle John Wilby
- College of Pharmacy, Faculty of Health, Dalhousie University, 5968 College Street, Halifax, Nova Scotia, Canada.
| |
Collapse
|
3
|
Malau-Aduli BS, Hays RB, D'Souza K, Jones K, Saad S, Celenza A, Turner R, Smith J, Ward H, Schlipalius M, Murphy R, Garg N. “Could You Work in My Team?”: Exploring How Professional Clinical Role Expectations Influence Decision-Making of Assessors During Exit-Level Medical School OSCEs. Front Med (Lausanne) 2022; 9:844899. [PMID: 35602481 PMCID: PMC9120654 DOI: 10.3389/fmed.2022.844899] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/29/2021] [Accepted: 04/08/2022] [Indexed: 11/13/2022] Open
Abstract
Decision-making in clinical assessment, such as exit-level medical school Objective Structured Clinical Examinations (OSCEs), is complex. This study utilized an empirical phenomenological qualitative approach with thematic analysis to explore OSCE assessors' perceptions of the concept of a “prototypical intern” expressed during focus group discussions. Topics discussed included the concept of a prototypical intern, qualities to be assessed, and approaches to clinical assessment decision-making. The thematic analysis was then applied to a theoretical framework (Cultural Historical Activity Theory—CHAT) that explored the complexity of making assessment decisions amidst potentially contradicting pressures from academic and clinical perspectives. Ten Australasian medical schools were involved with 15 experienced and five less experienced assessors participating. Thematic analysis of the data revealed four major themes in relation to how the prototypical intern concept influences clinical assessors' judgements: (a) Suitability of marking rubric based on assessor characteristics and expectations; (b) Competence as final year student vs. performance as a prototypical intern; (c) Safety, trustworthiness and reliability as constructs requiring assessment and (d) Contradictions in decision making process due to assessor differences. These themes mapped well within the interaction between two proposed activity systems in the CHAT model: academic and clinical. More clinically engaged and more experienced assessors tend to fall back on a heuristic, mental construct of a “prototypical intern,” to calibrate judgements, particularly, in difficult situations. Further research is needed to explore whether consensus on desirable intern qualities and their inclusion into OSCE marksheets decreases the cognitive load and increases the validity of assessor decision making.
Collapse
Affiliation(s)
- Bunmi S. Malau-Aduli
- College of Medicine and Dentistry, James Cook University, Townsville, QLD, Australia
- *Correspondence: Bunmi S. Malau-Aduli
| | - Richard B. Hays
- College of Medicine and Dentistry, James Cook University, Townsville, QLD, Australia
| | - Karen D'Souza
- School of Medicine, Deakin University, Geelong, VIC, Australia
| | - Karina Jones
- College of Medicine and Dentistry, James Cook University, Townsville, QLD, Australia
| | - Shannon Saad
- School of Medicine, Notre Dame University, Chippendale, NSW, Australia
| | - Antonio Celenza
- School of Medicine, University of Western Australia, Perth, WA, Australia
| | - Richard Turner
- School of Medicine, University of Tasmania, Hobart, TAS, Australia
| | - Jane Smith
- Medical Program, Bond University, Gold Coast, QLD, Australia
| | - Helena Ward
- Adelaide Medical School, University of Adelaide, Adelaide, SA, Australia
| | - Michelle Schlipalius
- School of Medicine and Health Sciences, Monash University, Melbourne, VIC, Australia
| | - Rinki Murphy
- Medical Program, University of Auckland, Auckland, New Zealand
| | - Nidhi Garg
- School of Medicine, University of Sydney, Sydney, NSW, Australia
| |
Collapse
|
4
|
Malau-Aduli BS, Hays RB, D'Souza K, Smith AM, Jones K, Turner R, Shires L, Smith J, Saad S, Richmond C, Celenza A, Sen Gupta T. Examiners' decision-making processes in observation-based clinical examinations. MEDICAL EDUCATION 2021; 55:344-353. [PMID: 32810334 DOI: 10.1111/medu.14357] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/19/2020] [Revised: 08/08/2020] [Accepted: 08/14/2020] [Indexed: 06/11/2023]
Abstract
BACKGROUND Objective structured clinical examinations (OSCEs) are commonly used to assess the clinical skills of health professional students. Examiner judgement is one acknowledged source of variation in candidate marks. This paper reports an exploration of examiner decision making to better characterise the cognitive processes and workload associated with making judgements of clinical performance in exit-level OSCEs. METHODS Fifty-five examiners for exit-level OSCEs at five Australian medical schools completed a NASA Task Load Index (TLX) measure of cognitive load and participated in focus group interviews immediately after the OSCE session. Discussions focused on how decisions were made for borderline and clear pass candidates. Interviews were transcribed, coded and thematically analysed. NASA TLX results were quantitatively analysed. RESULTS Examiners self-reported higher cognitive workload levels when assessing a borderline candidate in comparison with a clear pass candidate. Further analysis revealed five major themes considered by examiners when marking candidate performance in an OSCE: (a) use of marking criteria as a source of reassurance; (b) difficulty adhering to the marking sheet under certain conditions; (c) demeanour of candidates; (d) patient safety, and (e) calibration using a mental construct of the 'mythical [prototypical] intern'. Examiners demonstrated particularly higher mental demand when assessing borderline compared to clear pass candidates. CONCLUSIONS Examiners demonstrate that judging candidate performance is a complex, cognitively difficult task, particularly when performance is of borderline or lower standard. At programme exit level, examiners intuitively want to rate candidates against a construct of a prototypical graduate when marking criteria appear not to describe both what and how a passing candidate should demonstrate when completing clinical tasks. This construct should be shared, agreed upon and aligned with marking criteria to best guide examiner training and calibration. Achieving this integration may improve the accuracy and consistency of examiner judgements and reduce cognitive workload.
Collapse
Affiliation(s)
- Bunmi S Malau-Aduli
- College of Medicine and Dentistry, James Cook University, Townsville, QLD, Australia
| | - Richard B Hays
- College of Medicine and Dentistry, James Cook University, Townsville, QLD, Australia
| | - Karen D'Souza
- School of Medicine, Deakin University, Geelong, VIC, Australia
| | - Amy M Smith
- College of Medicine and Dentistry, James Cook University, Townsville, QLD, Australia
| | - Karina Jones
- College of Medicine and Dentistry, James Cook University, Townsville, QLD, Australia
| | - Richard Turner
- School of Medicine, University of Tasmania, Hobart, TAS, Australia
| | - Lizzi Shires
- School of Medicine, University of Tasmania, Hobart, TAS, Australia
| | - Jane Smith
- Medical Program, Bond University, Gold Coast, QLD, Australia
| | - Shannon Saad
- School of Medicine, Notre Dame University, Sydney, NSW, Australia
| | | | - Antonio Celenza
- School of Medicine, University of Western Australia, Perth, WA, Australia
| | - Tarun Sen Gupta
- College of Medicine and Dentistry, James Cook University, Townsville, QLD, Australia
| |
Collapse
|
5
|
Wilby KJ, Paravattil B. Cognitive load theory: Implications for assessment in pharmacy education. Res Social Adm Pharm 2020; 17:1645-1649. [PMID: 33358136 DOI: 10.1016/j.sapharm.2020.12.009] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/15/2019] [Revised: 11/09/2020] [Accepted: 12/15/2020] [Indexed: 11/28/2022]
Abstract
The concept of mental workload is well studied from a learner's perspective but has yet to be better understood from the perspective of an assessor. Mental workload is largely associated with cognitive load theory, which describes three different types of load. Intrinsic load deals with the complexity of the task, extraneous load describes distractors to the task at hand, and germane load focuses on the development of schemas in working memory for future recall. Studies from medical education show that all three types of load are relevant when considering rater -based assessment (e.g. Objective Structured Clinical Examinations (OSCEs), or experiential training). Assessments with high intrinsic and extraneous load may interfere with assessors' attention and working memory and result in poorer quality assessment. Reducing these loads within assessment tasks should therefore be a priority for pharmacy educators. This commentary aims to provide a theoretical overview of mental workload in assessment, outline research findings from the medical education context, and propose strategies to be considered for reducing mental workload in rater-based assessments relevant to pharmacy education. Suggestions for future research are also addressed.
Collapse
Affiliation(s)
- Kyle John Wilby
- School of Pharmacy, University of Otago, PO Box 56, Dunedin, 9054, New Zealand.
| | | |
Collapse
|
6
|
Ten Cate O, Dahdal S, Lambert T, Neubauer F, Pless A, Pohlmann PF, van Rijen H, Gurtner C. Ten caveats of learning analytics in health professions education: A consumer's perspective. MEDICAL TEACHER 2020; 42:673-678. [PMID: 32150499 DOI: 10.1080/0142159x.2020.1733505] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
A group of 22 medical educators from different European countries, gathered in a meeting in Utrecht in July 2019, discussed the topic of learning analytics (LA) in an open conversation and addressed its definition, its purposes and potential risks for learners and teachers. LA was seen as a significant advance with important potential to improve education, but the group felt that potential drawbacks of using LA may yet be under-exposed in the literature. After transcription and interpretation of the discussion's conclusions, a document was drafted and fed back to the group in two rounds to arrive at a series of 10 caveats educators should be aware of when developing and using LA, including too much standardized learning, with undue consequences of over-efficiency and pressure on learners and teachers, and a decrease of the variety of 'valid' learning resources. Learning analytics may misalign with eventual clinical performance and can run the risk of privacy breaches and inescapability of documented failures. These consequences may not happen, but the authors, on behalf of the full group of educators, felt it worth to signal these caveats from a consumers' perspective.
Collapse
Affiliation(s)
- Olle Ten Cate
- Center for Research and Development of Education, University Medical Center Utrecht, Utrecht, The Netherlands
| | | | - Thomas Lambert
- Kepler University Hospital Linz, Johannes Kepler University Linz, Linz, Austria
| | - Florian Neubauer
- Institute for Medical Education, University of Bern, Bern, Switzerland
| | - Anina Pless
- Institute of Primary Health Care (BIHAM), University of Bern, Bern, Switzerland
| | | | - Harold van Rijen
- Center for Research and Development of Education, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Corinne Gurtner
- Institute of Animal Pathology, Vetsuisse Faculty Bern, University of Bern, Bern, Switzerland
| |
Collapse
|
7
|
Gingerich A, Yeates P. The mental workload of conducting research in assessor cognition. PERSPECTIVES ON MEDICAL EDUCATION 2019; 8:315-316. [PMID: 31745709 PMCID: PMC6904388 DOI: 10.1007/s40037-019-00549-0] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Affiliation(s)
- Andrea Gingerich
- Northern Medical Program, University of Northern British Columbia, Prince George, British Columbia, Canada.
| | - Peter Yeates
- Medical School Education Research Group (MERG), Keele University School of Medicine, Keele University, Keele, UK
- Department of Acute Medicine, Fairfield General Hospital, Pennine Acute Hospitals NHS Trust, Bury, Lancashire, UK
| |
Collapse
|