1
|
Lingard L, Watling C. The writer's voice repertoire: Exploring how health researchers accomplish a distinctive 'voice' in their writing. MEDICAL EDUCATION 2024; 58:523-534. [PMID: 38233970 DOI: 10.1111/medu.15298] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/15/2023] [Revised: 11/20/2023] [Accepted: 11/24/2023] [Indexed: 01/19/2024]
Abstract
INTRODUCTION Much published research writing is dull and dry at best, impenetrable and off-putting at worst. This state of affairs both frustrates readers and impedes research uptake. Scientific conventions of objectivity and neutrality contribute to the problem, implying that 'good' research writing should have no discernible authorial 'voice'. Yet some research writers have a distinctive voice in their work that may contribute to their scholarly influence. In this study, we explore this notion of voice, examining what strong research writers aim for with their voice and what strategies they use. METHODS Using a combination of purposive, snowball and theoretical sampling, we recruited 21 scholars working in health professions education or adjacent health research fields, representing varied career stages, research paradigms and geographical locations. We interviewed participants about their approaches to writing and asked each to provide one to three illustrative publications. Iterative data collection and analysis followed constructivist grounded theory principles. We analysed interview transcripts thematically and examined publications for evidence of the writers' described approaches. RESULTS Participants shared goals of a voice that was clear and logical, and that engaged readers and held their attention. They accomplished these goals using approaches both conventional and unconventional. Conventional approaches included attention to coherence through signposting, symmetry and metacommentary. Unconventional approaches included using language that was evocative (metaphor, imagery), provocative (pointed critique), plainspoken ('non-academic' phrasing), playful (including humour) and lyrical (attending to cadence and sound). Unconventional elements were more prominent in non-standard genres (e.g. commentaries), but also appeared in empiric papers. DISCUSSION What readers interpret as 'voice' reflects strategic use of a repertoire of writing techniques. Conventional techniques, used expertly, can make for compelling reading, but strong writers also draw on unconventional strategies. A broadened writing repertoire might assist health professions education research writers in effectively communicating their work.
Collapse
Affiliation(s)
- Lorelei Lingard
- Department of Medicine and Centre for Education Research & Innovation, Schulich School of Medicine and Dentistry, Western University, London, Ontario, Canada
| | - Chris Watling
- Department of Oncology and Centre for Education Research & Innovation, Schulich School of Medicine and Dentistry, Western University, London, Ontario, Canada
| |
Collapse
|
2
|
Schauber SK, Olsen AO, Werner EL, Magelssen M. Inconsistencies in rater-based assessments mainly affect borderline candidates: but using simple heuristics might improve pass-fail decisions. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2024:10.1007/s10459-024-10328-0. [PMID: 38649529 DOI: 10.1007/s10459-024-10328-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/20/2023] [Accepted: 03/24/2024] [Indexed: 04/25/2024]
Abstract
INTRODUCTION Research in various areas indicates that expert judgment can be highly inconsistent. However, expert judgment is indispensable in many contexts. In medical education, experts often function as examiners in rater-based assessments. Here, disagreement between examiners can have far-reaching consequences. The literature suggests that inconsistencies in ratings depend on the level of performance a to-be-evaluated candidate shows. This possibility has not been addressed deliberately and with appropriate statistical methods. By adopting the theoretical lens of ecological rationality, we evaluate if easily implementable strategies can enhance decision making in real-world assessment contexts. METHODS We address two objectives. First, we investigate the dependence of rater-consistency on performance levels. We recorded videos of mock-exams and had examiners (N=10) evaluate four students' performances and compare inconsistencies in performance ratings between examiner-pairs using a bootstrapping procedure. Our second objective is to provide an approach that aids decision making by implementing simple heuristics. RESULTS We found that discrepancies were largely a function of the level of performance the candidates showed. Lower performances were rated more inconsistently than excellent performances. Furthermore, our analyses indicated that the use of simple heuristics might improve decisions in examiner pairs. DISCUSSION Inconsistencies in performance judgments continue to be a matter of concern, and we provide empirical evidence for them to be related to candidate performance. We discuss implications for research and the advantages of adopting the perspective of ecological rationality. We point to directions both for further research and for development of assessment practices.
Collapse
Affiliation(s)
- Stefan K Schauber
- Centre for Health Sciences Education, Faculty of Medicine, University of Oslo, Oslo, Norway.
- Centre for Educational Measurement (CEMO), Faculty of Educational Sciences, University of Oslo, Oslo, Norway.
| | - Anne O Olsen
- Department of Community Medicine and Global Health, Institute of Health and Society, University of Oslo, Oslo, Norway
| | - Erik L Werner
- Department of General Practice, Institute of Health and Society, University of Oslo, Oslo, Norway
| | - Morten Magelssen
- Centre for Medical Ethics, Institute of Health and Society, University of Oslo, Oslo, Norway
| |
Collapse
|
3
|
Yeates P, Maluf A, Kinston R, Cope N, Cullen K, Cole A, O'Neill V, Chung CW, Goodfellow R, Vallender R, Ensaff S, Goddard-Fuller R, McKinley R, Wong G. A realist evaluation of how, why and when objective structured clinical exams (OSCEs) are experienced as an authentic assessment of clinical preparedness. MEDICAL TEACHER 2024:1-9. [PMID: 38635469 DOI: 10.1080/0142159x.2024.2339413] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/28/2023] [Accepted: 04/02/2024] [Indexed: 04/20/2024]
Abstract
INTRODUCTION Whilst rarely researched, the authenticity with which Objective Structured Clinical Exams (OSCEs) simulate practice is arguably critical to making valid judgements about candidates' preparedness to progress in their training. We studied how and why an OSCE gave rise to different experiences of authenticity for different participants under different circumstances. METHODS We used Realist evaluation, collecting data through interviews/focus groups from participants across four UK medical schools who participated in an OSCE which aimed to enhance authenticity. RESULTS Several features of OSCE stations (realistic, complex, complete cases, sufficient time, autonomy, props, guidelines, limited examiner interaction etc) combined to enable students to project into their future roles, judge and integrate information, consider their actions and act naturally. When this occurred, their performances felt like an authentic representation of their clinical practice. This didn't work all the time: focusing on unavoidable differences with practice, incongruous features, anxiety and preoccupation with examiners' expectations sometimes disrupted immersion, producing inauthenticity. CONCLUSIONS The perception of authenticity in OSCEs appears to originate from an interaction of station design with individual preferences and contextual expectations. Whilst tentatively suggesting ways to promote authenticity, more understanding is needed of candidates' interaction with simulation and scenario immersion in summative assessment.
Collapse
Affiliation(s)
- Peter Yeates
- School of Medicine, Keele University, Keele, England
| | - Adriano Maluf
- Faculty of Health and Life Sciences, De Montford University, Leicester, England
| | - Ruth Kinston
- School of Medicine, Keele University, Keele, England
| | - Natalie Cope
- School of Medicine, Keele University, Keele, England
| | - Kathy Cullen
- School of Medicine, Dentistry and Biomedical Sciences, Queens University Belfast, Belfast, Northern Ireland
| | - Aidan Cole
- School of Medicine, Dentistry and Biomedical Sciences, Queens University Belfast, Belfast, Northern Ireland
| | - Vikki O'Neill
- School of Medicine, Dentistry and Biomedical Sciences, Queens University Belfast, Belfast, Northern Ireland
| | - Ching-Wa Chung
- School of Medicine, Medical Sciences and Nutrition, University of Aberdeen, Aberdeen, Scotland
| | | | | | - Sue Ensaff
- School of Medicine, Cardiff University, Cardiff, Wales
| | - Rikki Goddard-Fuller
- Christie Education, Christie Hospitals NHS Foundation Trust, Manchester, England
| | | | - Geoff Wong
- Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, England
| |
Collapse
|
4
|
Chan SCC, Rashid MA. The art of reinvention: The remarkable longevity of the OSCE. MEDICAL EDUCATION 2024; 58:177-179. [PMID: 37973158 DOI: 10.1111/medu.15266] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/10/2023] [Revised: 10/15/2023] [Accepted: 10/17/2023] [Indexed: 11/19/2023]
Abstract
This commentary delves into the theatre and performances of OSCEs. Join us in exploring their 50‐year journey of reinvention and how they continue to hold the stage in medical assessments.
Collapse
Affiliation(s)
- See Chai Carol Chan
- Department of Primary Care and Public Health, Imperial College London, London, UK
| | | |
Collapse
|
5
|
Tavares W, Kinnear B, Schumacher DJ, Forte M. "Rater training" re-imagined for work-based assessment in medical education. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2023; 28:1697-1709. [PMID: 37140661 DOI: 10.1007/s10459-023-10237-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/13/2023] [Accepted: 04/30/2023] [Indexed: 05/05/2023]
Abstract
In this perspective, the authors critically examine "rater training" as it has been conceptualized and used in medical education. By "rater training," they mean the educational events intended to improve rater performance and contributions during assessment events. Historically, rater training programs have focused on modifying faculty behaviours to achieve psychometric ideals (e.g., reliability, inter-rater reliability, accuracy). The authors argue these ideals may now be poorly aligned with contemporary research informing work-based assessment, introducing a compatibility threat, with no clear direction on how to proceed. To address this issue, the authors provide a brief historical review of "rater training" and provide an analysis of the literature examining the effectiveness of rater training programs. They focus mainly on what has served to define effectiveness or improvements. They then draw on philosophical and conceptual shifts in assessment to demonstrate why the function, effectiveness aims, and structure of rater training requires reimagining. These include shifting competencies for assessors, viewing assessment as a complex cognitive task enacted in a social context, evolving views on biases, and reprioritizing which validity evidence should be most sought in medical education. The authors aim to advance the discussion on rater training by challenging implicit incompatibility issues and stimulating ways to overcome them. They propose that "rater training" (a moniker they suggest be reserved for strong psychometric aims) be augmented with "assessor readiness" programs that link to contemporary assessment science and enact the principle of compatibility between that science and ways of engaging with advances in real-world faculty-learner contexts.
Collapse
Affiliation(s)
- Walter Tavares
- Department of Health and Society, Wilson Centre, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada.
| | - Benjamin Kinnear
- Department of Pediatrics, Cincinnati Children's Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, OH, USA
| | - Daniel J Schumacher
- Department of Pediatrics, Cincinnati Children's Hospital Medical Center, University of Cincinnati College of Medicine, Cincinnati, OH, USA
| | - Milena Forte
- Department of Family and Community Medicine, Temerty Faculty of Medicine, Mount Sinai Hospital, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
6
|
Eva KW. To a fault. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2023; 28:537-540. [PMID: 36449112 DOI: 10.1007/s10459-022-10181-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/04/2022] [Accepted: 10/31/2022] [Indexed: 05/11/2023]
Abstract
Are first impressions misleading? This commentary explores that question by drawing on the more general cognitive psychology literature aimed at understanding when, why, and how any non-analytic reasoning process can help or hurt decision-making.
Collapse
Affiliation(s)
- Kevin W Eva
- Centre for Health Education Scholarship, University of British Columbia, Vancouver, Canada.
| |
Collapse
|
7
|
Yeates P, Maluf A, Kinston R, Cope N, McCray G, Cullen K, O'Neill V, Cole A, Goodfellow R, Vallender R, Chung CW, McKinley RK, Fuller R, Wong G. Enhancing authenticity, diagnosticity and equivalence (AD-Equiv) in multicentre OSCE exams in health professionals education: protocol for a complex intervention study. BMJ Open 2022; 12:e064387. [PMID: 36600366 PMCID: PMC9730346 DOI: 10.1136/bmjopen-2022-064387] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/04/2022] [Accepted: 10/12/2022] [Indexed: 12/12/2022] Open
Abstract
INTRODUCTION Objective structured clinical exams (OSCEs) are a cornerstone of assessing the competence of trainee healthcare professionals, but have been criticised for (1) lacking authenticity, (2) variability in examiners' judgements which can challenge assessment equivalence and (3) for limited diagnosticity of trainees' focal strengths and weaknesses. In response, this study aims to investigate whether (1) sharing integrated-task OSCE stations across institutions can increase perceived authenticity, while (2) enhancing assessment equivalence by enabling comparison of the standard of examiners' judgements between institutions using a novel methodology (video-based score comparison and adjustment (VESCA)) and (3) exploring the potential to develop more diagnostic signals from data on students' performances. METHODS AND ANALYSIS The study will use a complex intervention design, developing, implementing and sharing an integrated-task (research) OSCE across four UK medical schools. It will use VESCA to compare examiner scoring differences between groups of examiners and different sites, while studying how, why and for whom the shared OSCE and VESCA operate across participating schools. Quantitative analysis will use Many Facet Rasch Modelling to compare the influence of different examiners groups and sites on students' scores, while the operation of the two interventions (shared integrated task OSCEs; VESCA) will be studied through the theory-driven method of Realist evaluation. Further exploratory analyses will examine diagnostic performance signals within data. ETHICS AND DISSEMINATION The study will be extra to usual course requirements and all participation will be voluntary. We will uphold principles of informed consent, the right to withdraw, confidentiality with pseudonymity and strict data security. The study has received ethical approval from Keele University Research Ethics Committee. Findings will be academically published and will contribute to good practice guidance on (1) the use of VESCA and (2) sharing and use of integrated-task OSCE stations.
Collapse
Affiliation(s)
- Peter Yeates
- School of Medicine, Keele University, Keele, Staffordshire, UK
| | - Adriano Maluf
- School of Medicine, Keele University, Keele, Staffordshire, UK
| | - Ruth Kinston
- School of Medicine, Keele University, Keele, Staffordshire, UK
| | - Natalie Cope
- School of Medicine, Keele University, Keele, Staffordshire, UK
| | - Gareth McCray
- School of Medicine, Keele University, Keele, Staffordshire, UK
| | - Kathy Cullen
- School of Medicine, Dentistry and Biomedical Sciences, Queen's University Belfast, Belfast, UK
| | - Vikki O'Neill
- School of Medicine, Dentistry and Biomedical Sciences, Queen's University Belfast, Belfast, UK
| | - Aidan Cole
- School of Medicine, Dentistry and Biomedical Sciences, Queen's University Belfast, Belfast, UK
| | | | | | - Ching-Wa Chung
- School of Medicine, Medical Sciences and Nutrition, University of Aberdeen, Aberdeen, Scotland, UK
| | | | - Richard Fuller
- School of Medicine, University of Liverpool Faculty of Health and Life Sciences, Liverpool, UK
| | - Geoff Wong
- Nuffield Department of Primary Care Health Sciences, University of Oxford Division of Public Health and Primary Health Care, Oxford, Oxfordshire, UK
| |
Collapse
|
8
|
Luu K, Sidhu R, Chadha NK, Eva KW. An exploration of "real time" assessments as a means to better understand preceptors' judgments of student performance. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2022:10.1007/s10459-022-10189-5. [PMID: 36441287 DOI: 10.1007/s10459-022-10189-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/28/2022] [Accepted: 11/19/2022] [Indexed: 06/16/2023]
Abstract
Clinical supervisors are known to assess trainee performance idiosyncratically, causing concern about the validity of their ratings. The literature on this issue relies heavily on retrospective collection of decisions, resulting in the risk of inaccurate information regarding what actually drives raters' perceptions. Capturing in-the-moment information about supervisors' impressions could yield better insight into how to intervene. The purpose of this study, therefore, was to gather "real-time" judgments to explore what drives preceptors' judgments of student performance. We performed a prospective study in which physicians were asked to adjust a rating scale in real-time while watching two video-recordings of trainee clinical performances. Scores were captured in 1-s increments, examined for frequency, direction, and magnitude of adjustments, and compared to assessors' final entrustability judgment as measured by the modified Ottawa Clinic Assessment Tool. The standard deviation in raters' judgment was examined as a function of time to determine how long it takes impressions to begin to vary. 20 participants viewed 2 clinical vignettes. Considerable variability in ratings was observed with different behaviours triggering scale adjustments for different raters. That idiosyncrasy occurred very quickly, with the standard deviation in raters' judgments rapidly increasing within 30 s of case onset. Particular moments appeared to generally be influential, but their degree of influence still varied. Correlations between the final assessment and (a) score assigned upon first adjustment of the scale, (b) upon last adjustment, and (c) the mean score, were r = 0.13, 0.32, and 0.57 for one video and r = 0.30, 0.50, and 0.52 for the other, indicating the degree to which overall impressions reflected accumulation of raters' idiosyncratic moment-by-moment observations. Our results demonstrated that variability in raters' impressions begins very early in a case presentation and is associated with different behaviours having different influence on different raters. More generally, this study outlines a novel methodology that offers a new path for gaining insight into factors influencing assessor judgments.
Collapse
Affiliation(s)
- Kimberly Luu
- Department of Otolaryngology, University of California San Francisco, San Francisco, CA, USA
| | - Ravi Sidhu
- Department of Surgery, University of British Columbia, Vancouver, BC, Canada
| | - Neil K Chadha
- Division of Otolaryngology, University of British Columbia, Vancouver, BC, Canada
| | - Kevin W Eva
- Department of Medicine, The University of British Columbia, 429K - 2194 Health Sciences Mall, Vancouver, BC, V6T 1Z3, Canada.
| |
Collapse
|
9
|
Gonzalez PR, Paravattil B, Wilby KJ. Mental effort in the assessment of critical reflection: Implications for assessment quality and scoring. CURRENTS IN PHARMACY TEACHING & LEARNING 2022; 14:830-834. [PMID: 35914842 DOI: 10.1016/j.cptl.2022.06.016] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/05/2021] [Revised: 05/16/2022] [Accepted: 06/16/2022] [Indexed: 06/15/2023]
Abstract
INTRODUCTION Critical reflection is a mainstay in the training of health professionals, yet assessment of reflection is commonly described as difficult, taxing, and resulting in inconsistent scoring across assessors. At the same time, there is evidence from experiential and simulation settings that assessors' mental effort may explain assessor variability, which could be a target for simplifications in assessment design. Assessors' mental effort for assessment of reflection is currently unknown. This study aimed to determine reliability of rubric scoring of critical reflection, variation in pass-fail rates, and the relationship between reflection scores and assessors' perceived mental effort. METHODS Eleven assessors were recruited to assess six reflection assignments using a published rubric. Mental effort was measured using the Paas scale for each assignment assessed and was correlated with rubric scores for each assignment. RESULTS Findings showed inconsistency in scoring between assessors, resulting in varying pass rates for each assignment (55-100%). All assignments demonstrated negative correlations between rubric scores and perceived mental effort (r = -0.115 to -0.649). CONCLUSIONS Findings support the notion that more work should be done to optimize assessment of critical reflection. Future studies should focus on disentangling the influence on mental effort of scoring tools, assignment structures, and writing quality.
Collapse
Affiliation(s)
| | | | - Kyle John Wilby
- College of Pharmacy, Faculty of Health, Dalhousie University, 5968 College Street, Halifax, Nova Scotia, Canada.
| |
Collapse
|
10
|
Ezegbogu MO, Omede PIO. The admissibility of fingerprint evidence: An African perspective. CANADIAN SOCIETY OF FORENSIC SCIENCE JOURNAL 2022. [DOI: 10.1080/00085030.2022.2068404] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/18/2022]
Affiliation(s)
- Mark O. Ezegbogu
- School of Physical Sciences, Division of Natural Sciences, University of Kent, Canterbury, UK
| | | |
Collapse
|
11
|
Yeates P, McCray G, Moult A, Cope N, Fuller R, McKinley R. Determining the influence of different linking patterns on the stability of students' score adjustments produced using Video-based Examiner Score Comparison and Adjustment (VESCA). BMC MEDICAL EDUCATION 2022; 22:41. [PMID: 35039023 PMCID: PMC8764767 DOI: 10.1186/s12909-022-03115-1] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 01/22/2021] [Accepted: 01/05/2022] [Indexed: 06/14/2023]
Abstract
BACKGROUND Ensuring equivalence of examiners' judgements across different groups of examiners is a priority for large scale performance assessments in clinical education, both to enhance fairness and reassure the public. This study extends insight into an innovation called Video-based Examiner Score Comparison and Adjustment (VESCA) which uses video scoring to link otherwise unlinked groups of examiners. This linkage enables comparison of the influence of different examiner-groups within a common frame of reference and provision of adjusted "fair" scores to students. Whilst this innovation promises substantial benefit to quality assurance of distributed Objective Structured Clinical Exams (OSCEs), questions remain about how the resulting score adjustments might be influenced by the specific parameters used to operationalise VESCA. Research questions, How similar are estimates of students' score adjustments when the model is run with either: fewer comparison videos per participating examiner?; reduced numbers of participating examiners? METHODS Using secondary analysis of recent research which used VESCA to compare scoring tendencies of different examiner groups, we made numerous copies of the original data then selectively deleted video scores to reduce the number of 1/ linking videos per examiner (4 versus several permutations of 3,2,or 1 videos) or 2/examiner participation rates (all participating examiners (76%) versus several permutations of 70%, 60% or 50% participation). After analysing all resulting datasets with Many Facet Rasch Modelling (MFRM) we calculated students' score adjustments for each dataset and compared these with score adjustments in the original data using Spearman's correlations. RESULTS Students' score adjustments derived form 3 videos per examiner correlated highly with score adjustments derived from 4 linking videos (median Rho = 0.93,IQR0.90-0.95,p < 0.001), with 2 (median Rho 0.85,IQR0.81-0.87,p < 0.001) and 1 linking videos (median Rho = 0.52(IQR0.46-0.64,p < 0.001) producing progressively smaller correlations. Score adjustments were similar for 76% participating examiners and 70% (median Rho = 0.97,IQR0.95-0.98,p < 0.001), and 60% (median Rho = 0.95,IQR0.94-0.98,p < 0.001) participation, but were lower and more variable for 50% examiner participation (median Rho = 0.78,IQR0.65-0.83, some ns). CONCLUSIONS Whilst VESCA showed some sensitivity to the examined parameters, modest reductions in examiner participation rates or video numbers produced highly similar results. Employing VESCA in distributed or national exams could enhance quality assurance or exam fairness.
Collapse
Affiliation(s)
- Peter Yeates
- School of Medicine, David Weatherall Building, Keele University, Keele, Staffordshire, ST5 5BG, UK.
- Fairfield General Hospital, Northern Care Alliance NHS Foundation Trust, Rochdale Old Road, Bury, BL9 7TD, Lancashire, UK.
| | - Gareth McCray
- School of Medicine, David Weatherall Building, Keele University, Keele, Staffordshire, ST5 5BG, UK
| | - Alice Moult
- School of Medicine, David Weatherall Building, Keele University, Keele, Staffordshire, ST5 5BG, UK
| | - Natalie Cope
- School of Medicine, David Weatherall Building, Keele University, Keele, Staffordshire, ST5 5BG, UK
| | - Richard Fuller
- Christie Education, Christie Hospitals NHS Foundation Trust, Wilmslow Rd, Manchester, M20 4BX, UK
| | - Robert McKinley
- School of Medicine, David Weatherall Building, Keele University, Keele, Staffordshire, ST5 5BG, UK
| |
Collapse
|
12
|
Hyde S, Fessey C, Boursicot K, MacKenzie R, McGrath D. OSCE rater cognition - an international multi-centre qualitative study. BMC MEDICAL EDUCATION 2022; 22:6. [PMID: 34980099 PMCID: PMC8721185 DOI: 10.1186/s12909-021-03077-w] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/11/2020] [Accepted: 12/06/2021] [Indexed: 05/09/2023]
Abstract
INTRODUCTION This study aimed to explore the decision-making processes of raters during objective structured clinical examinations (OSCEs), in particular to explore the tacit assumptions and beliefs of raters as well as rater idiosyncrasies. METHODS Thinking aloud protocol interviews were used to gather data on the thoughts of examiners during their decision-making, while watching trigger OSCE videos and rating candidates. A purposeful recruiting strategy was taken, with a view to interviewing both examiners with many years of experience (greater than six years) and those with less experience examining at final medical examination level. RESULTS Thirty-one interviews were conducted in three centres in three different countries. Three themes were identified during data analysis, entitled 'OSCEs are inauthentic', 'looking for glimpses of truth' and 'evolution with experience'. CONCLUSION Raters perceive that the shortcomings of OSCEs can have unwanted effects on student behaviour. Some examiners, more likely the more experienced group, may deviate from an organisations directions due to perceived shortcomings of the assessment. No method of assessment is without flaw, and it is important to be aware of the limitations and shortcomings of assessment methods on student performance and examiner perception. Further study of assessor and student perception of OSCE performance would be helpful.
Collapse
Affiliation(s)
- Sarah Hyde
- School of Medicine at the University of Limerick, Health Research Institute, Limerick, Ireland.
| | | | | | | | - Deirdre McGrath
- School of Medicine at the University of Limerick, Health Research Institute, Limerick, Ireland
| |
Collapse
|
13
|
Humphrey-Murto S, Shaw T, Touchie C, Pugh D, Cowley L, Wood TJ. Are raters influenced by prior information about a learner? A review of assimilation and contrast effects in assessment. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2021; 26:1133-1156. [PMID: 33566199 DOI: 10.1007/s10459-021-10032-3] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/17/2020] [Accepted: 01/25/2021] [Indexed: 06/12/2023]
Abstract
Understanding which factors can impact rater judgments in assessments is important to ensure quality ratings. One such factor is whether prior performance information (PPI) about learners influences subsequent decision making. The information can be acquired directly, when the rater sees the same learner, or different learners over multiple performances, or indirectly, when the rater is provided with external information about the same learner prior to rating a performance (i.e., learner handover). The purpose of this narrative review was to summarize and highlight key concepts from multiple disciplines regarding the influence of PPI on subsequent ratings, discuss implications for assessment and provide a common conceptualization to inform research. Key findings include (a) assimilation (rater judgments are biased towards the PPI) occurs with indirect PPI and contrast (rater judgments are biased away from the PPI) with direct PPI; (b) negative PPI appears to have a greater effect than positive PPI; (c) when viewing multiple performances, context effects of indirect PPI appear to diminish over time; and (d) context effects may occur with any level of target performance. Furthermore, some raters are not susceptible to context effects, but it is unclear what factors are predictive. Rater expertise and training do not consistently reduce effects. Making raters more accountable, providing specific standards and reducing rater cognitive load may reduce context effects. Theoretical explanations for these findings will be discussed.
Collapse
Affiliation(s)
- Susan Humphrey-Murto
- Department of Medicine, Faculty of Medicine, The Ottawa Hospital-Riverside Campus, University of Ottawa, 1967 Riverside Drive, Box 67, Ottawa, ON, Canada.
- Department of Innovation in Medical Education, Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada.
| | - Tammy Shaw
- Department of Medicine, Faculty of Medicine, The Ottawa Hospital-General Campus, Ottawa, ON, Canada
| | - Claire Touchie
- Department of Medicine, Faculty of Medicine, The Ottawa Hospital-Riverside Campus, University of Ottawa, 1967 Riverside Drive, Box 67, Ottawa, ON, Canada
- Medical Council of Canada, Ottawa, ON, Canada
| | - Debra Pugh
- Department of Medicine, Faculty of Medicine, The Ottawa Hospital-Riverside Campus, University of Ottawa, 1967 Riverside Drive, Box 67, Ottawa, ON, Canada
- Medical Council of Canada, Ottawa, ON, Canada
| | - Lindsay Cowley
- Department of Innovation in Medical Education, Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada
| | - Timothy J Wood
- Department of Innovation in Medical Education, Faculty of Medicine, University of Ottawa, Ottawa, ON, Canada
| |
Collapse
|
14
|
Yeates P, Moult A, Cope N, McCray G, Xilas E, Lovelock T, Vaughan N, Daw D, Fuller R, McKinley RK(B. Measuring the Effect of Examiner Variability in a Multiple-Circuit Objective Structured Clinical Examination (OSCE). ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:1189-1196. [PMID: 33656012 PMCID: PMC8300845 DOI: 10.1097/acm.0000000000004028] [Citation(s) in RCA: 15] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
PURPOSE Ensuring that examiners in different parallel circuits of objective structured clinical examinations (OSCEs) judge to the same standard is critical to the chain of validity. Recent work suggests examiner-cohort (i.e., the particular group of examiners) could significantly alter outcomes for some candidates. Despite this, examiner-cohort effects are rarely examined since fully nested data (i.e., no crossover between the students judged by different examiner groups) limit comparisons. In this study, the authors aim to replicate and further develop a novel method called Video-based Examiner Score Comparison and Adjustment (VESCA), so it can be used to enhance quality assurance of distributed or national OSCEs. METHOD In 2019, 6 volunteer students were filmed on 12 stations in a summative OSCE. In addition to examining live student performances, examiners from 8 separate examiner-cohorts scored the pool of video performances. Examiners scored videos specific to their station. Video scores linked otherwise fully nested data, enabling comparisons by Many Facet Rasch Modeling. Authors compared and adjusted for examiner-cohort effects. They also compared examiners' scores when videos were embedded (interspersed between live students during the OSCE) or judged later via the Internet. RESULTS Having accounted for differences in students' ability, different examiner-cohort scores for the same ability of student ranged from 18.57 of 27 (68.8%) to 20.49 (75.9%), Cohen's d = 1.3. Score adjustment changed the pass/fail classification for up to 16% of students depending on the modeled cut score. Internet and embedded video scoring showed no difference in mean scores or variability. Examiners' accuracy did not deteriorate over the 3-week Internet scoring period. CONCLUSIONS Examiner-cohorts produced a replicable, significant influence on OSCE scores that was unaccounted for by typical assessment psychometrics. VESCA offers a promising means to enhance validity and fairness in distributed OSCEs or national exams. Internet-based scoring may enhance VESCA's feasibility.
Collapse
Affiliation(s)
- Peter Yeates
- P. Yeates is a senior lecturer in medical education research, School of Medicine, Keele University, Keele, Staffordshire, and a consultant in acute and respiratory medicine, Fairfield General Hospital, Pennine Acute Hospitals, NHS Trust, Bury, Lancashire, United Kingdom; ORCID: https://orcid.org/0000-0001-6316-4051
| | - Alice Moult
- A. Moult is a research assistant in medical education, School of Medicine, Keele University, Keele, Staffordshire, United Kingdom; ORCID: https://orcid.org/0000-0002-9424-5660
| | - Natalie Cope
- N. Cope is a lecturer in clinical education (psychometrics), School of Medicine, Keele University, Keele, Staffordshire, United Kingdom
| | - Gareth McCray
- G. McCray is a researcher, School of Primary, Community and Social Care, Keele University, Keele, Staffordshire, United Kingdom
| | - Eleftheria Xilas
- E. Xilas is a foundation year 1 doctor and recent graduate, School of Medicine, Keele University, Keele, Staffordshire, United Kingdom
| | - Tom Lovelock
- T. Lovelock is an information technology services manager, Faculty of Medicine & Health Sciences, Keele University, Keele, Staffordshire, United Kingdom
| | - Nicholas Vaughan
- N. Vaughan is a senior application developer, directorate of digital strategy and information technology services, Keele University, Keele, Staffordshire, United Kingdom
| | - Dan Daw
- D. Daw is an information technology systems development engineer, School of Medicine, Keele University, Keele, Staffordshire, United Kingdom
| | - Richard Fuller
- R. Fuller is deputy dean, School of Medicine, University of Liverpool, Liverpool, United Kingdom; ORCID: https://orcid.org/0000-0001-7965-4864
| | - Robert K. (Bob) McKinley
- R.K. McKinley is an emeritus professor of education in general practice, School of Medicine, Keele University, Keele, Staffordshire, United Kingdom; ORCID: https://orcid.org/0000-0002-3684-3435
| |
Collapse
|
15
|
Baugh RF, Baugh AD. Cultural influences and the Objective Structured Clinical Examination. INTERNATIONAL JOURNAL OF MEDICAL EDUCATION 2021; 12:22-24. [PMID: 33507878 PMCID: PMC7883802 DOI: 10.5116/ijme.5ff9.b817] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/15/2020] [Accepted: 01/09/2021] [Indexed: 06/12/2023]
Affiliation(s)
- Reginald F. Baugh
- Department of Surgery, University of Toledo College of Medicine and Life Sciences, Toledo, OH, USA
| | - Aaron D. Baugh
- Pulmonary, Critical Care, Allergy, Sleep Medicine, Department of Internal Medicine University of California San Francis-co Medical School, University of California San Francisco Medical Center, San Francisco, CA, USA
| |
Collapse
|
16
|
Wilby KJ, Paravattil B. Cognitive load theory: Implications for assessment in pharmacy education. Res Social Adm Pharm 2020; 17:1645-1649. [PMID: 33358136 DOI: 10.1016/j.sapharm.2020.12.009] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/15/2019] [Revised: 11/09/2020] [Accepted: 12/15/2020] [Indexed: 11/28/2022]
Abstract
The concept of mental workload is well studied from a learner's perspective but has yet to be better understood from the perspective of an assessor. Mental workload is largely associated with cognitive load theory, which describes three different types of load. Intrinsic load deals with the complexity of the task, extraneous load describes distractors to the task at hand, and germane load focuses on the development of schemas in working memory for future recall. Studies from medical education show that all three types of load are relevant when considering rater -based assessment (e.g. Objective Structured Clinical Examinations (OSCEs), or experiential training). Assessments with high intrinsic and extraneous load may interfere with assessors' attention and working memory and result in poorer quality assessment. Reducing these loads within assessment tasks should therefore be a priority for pharmacy educators. This commentary aims to provide a theoretical overview of mental workload in assessment, outline research findings from the medical education context, and propose strategies to be considered for reducing mental workload in rater-based assessments relevant to pharmacy education. Suggestions for future research are also addressed.
Collapse
Affiliation(s)
- Kyle John Wilby
- School of Pharmacy, University of Otago, PO Box 56, Dunedin, 9054, New Zealand.
| | | |
Collapse
|
17
|
Steenhof N, Woods NN, Mylopoulos M. Exploring why we learn from productive failure: insights from the cognitive and learning sciences. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2020; 25:1099-1106. [PMID: 33180211 DOI: 10.1007/s10459-020-10013-y] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/16/2020] [Accepted: 10/30/2020] [Indexed: 06/11/2023]
Abstract
Advances in Health Sciences Education (AHSE) has been at the forefront of the cognitive wave in health professions education for the past 25 years. One example is research on productive failure, a teaching strategy that asks learners to attempt to generate solutions to difficult problems before receiving instruction. This study compared the effectiveness of productive failure with indirect failure to further characterize the underpinning cognitive mechanisms of productive failure. Year one pharmacy students (N = 42) were randomly assigned to a productive failure or an indirect failure learning condition. The problem of estimating renal function based on serum creatinine was described to participants in the productive failure learning condition, who were then asked to generate a solution. Participants in the indirect failure condition learned about the same problem and were given incorrect solutions that other students had created, as well as the Cockcroft-Gault formula, and asked to compare and contrast the equations. Immediately thereafter all participants completed a series of tests designed to assess acquisition, application, and preparation for future learning (PFL). The tests were repeated after a 1-week delay. Participants in the productive failure condition outperformed those in the indirect failure condition, both on the immediate PFL assessment, and after a 1-week delay. These results emphasize the crucial role of generation in learning. When preparing novice students to learn new knowledge in the future, generating solutions to problems prior to instruction may be more effective than simply learning about someone else's mistakes. Struggle and failure are most productive when experienced personally by a learner because it requires the learner to engage in generation, which deepens conceptual understanding.
Collapse
Affiliation(s)
- Naomi Steenhof
- Leslie Dan Faculty of Pharmacy, University of Toronto, 144 College Street, Toronto, ON, M5S 3M2, Canada.
- The Wilson Centre, University of Toronto and University Health Network, Toronto, Canada.
- The Institute for Education Research, University Health Network, Toronto, Canada.
| | - Nicole N Woods
- The Wilson Centre, University of Toronto and University Health Network, Toronto, Canada
- Faculty of Medicine, University of Toronto, Toronto, Canada
- The Institute for Education Research, University Health Network, Toronto, Canada
| | - Maria Mylopoulos
- The Wilson Centre, University of Toronto and University Health Network, Toronto, Canada
- Faculty of Medicine, University of Toronto, Toronto, Canada
- The Institute for Education Research, University Health Network, Toronto, Canada
| |
Collapse
|
18
|
Yeates P, Moult A, Lefroy J, Walsh-House J, Clews L, McKinley R, Fuller R. Understanding and developing procedures for video-based assessment in medical education. MEDICAL TEACHER 2020; 42:1250-1260. [PMID: 32749915 DOI: 10.1080/0142159x.2020.1801997] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
INTRODUCTION Novel uses of video aim to enhance assessment in health-professionals education. Whilst these uses presume equivalence between video and live scoring, some research suggests that poorly understood variations could challenge validity. We aimed to understand examiners' and students' interaction with video whilst developing procedures to promote its optimal use. METHODS Using design-based research we developed theory and procedures for video use in assessment, iteratively adapting conditions across simulated OSCE stations. We explored examiners' and students' perceptions using think-aloud, interviews and focus group. Data were analysed using constructivist grounded-theory methods. RESULTS Video-based assessment produced detachment and reduced volitional control for examiners. Examiners ability to make valid video-based judgements was mediated by the interaction of station content and specifically selected filming parameters. Examiners displayed several judgemental tendencies which helped them manage videos' limitations but could also bias judgements in some circumstances. Students rarely found carefully-placed cameras intrusive and considered filming acceptable if adequately justified. DISCUSSION Successful use of video-based assessment relies on balancing the need to ensure station-specific information adequacy; avoiding disruptive intrusion; and the degree of justification provided by video's educational purpose. Video has the potential to enhance assessment validity and students' learning when an appropriate balance is achieved.
Collapse
Affiliation(s)
- Peter Yeates
- School of Medicine, Keele University, Keele, UK
- Department of Acute Medicine, Fairfield General Hospital, Pennine Acute Hospital NHS Trust, Bury, UK
| | - Alice Moult
- School of Medicine, Keele University, Keele, UK
| | | | | | | | | | - Richard Fuller
- School of Medicine, University of Liverpool, Liverpool, UK
| |
Collapse
|
19
|
Hyde C, Yardley S, Lefroy J, Gay S, McKinley RK. Clinical assessors' working conceptualisations of undergraduate consultation skills: a framework analysis of how assessors make expert judgements in practice. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2020; 25:845-875. [PMID: 31997115 PMCID: PMC7471149 DOI: 10.1007/s10459-020-09960-3] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/19/2019] [Accepted: 01/18/2020] [Indexed: 06/10/2023]
Abstract
Undergraduate clinical assessors make expert, multifaceted judgements of consultation skills in concert with medical school OSCE grading rubrics. Assessors are not cognitive machines: their judgements are made in the light of prior experience and social interactions with students. It is important to understand assessors' working conceptualisations of consultation skills and whether they could be used to develop assessment tools for undergraduate assessment. To identify any working conceptualisations that assessors use while assessing undergraduate medical students' consultation skills and develop assessment tools based on assessors' working conceptualisations and natural language for undergraduate consultation skills. In semi-structured interviews, 12 experienced assessors from a UK medical school populated a blank assessment scale with personally meaningful descriptors while describing how they made judgements of students' consultation skills (at exit standard). A two-step iterative thematic framework analysis was performed drawing on constructionism and interactionism. Five domains were found within working conceptualisations of consultation skills: Application of knowledge; Manner with patients; Getting it done; Safety; and Overall impression. Three mechanisms of judgement about student behaviour were identified: observations, inferences and feelings. Assessment tools drawing on participants' conceptualisations and natural language were generated, including 'grade descriptors' for common conceptualisations in each domain by mechanism of judgement and matched to grading rubrics of Fail, Borderline, Pass, Very good. Utilising working conceptualisations to develop assessment tools is feasible and potentially useful. Work is needed to test impact on assessment quality.
Collapse
Affiliation(s)
- Catherine Hyde
- School of Medicine, Keele University, Keele, Staffordshire, ST5 5BG, UK
| | - Sarah Yardley
- School of Medicine, Keele University, Keele, Staffordshire, ST5 5BG, UK.
- Palliative Care Service, Central and North West London NHS Foundation Trust, St Pancras Hospital, 5th Floor South Wing, 4 St. Pancras Way, London, NW1 0PE, UK.
| | - Janet Lefroy
- School of Medicine, Keele University, Keele, Staffordshire, ST5 5BG, UK
| | - Simon Gay
- University of Leicester School of Medicine, Leicester, UK
| | - Robert K McKinley
- School of Medicine, Keele University, Keele, Staffordshire, ST5 5BG, UK
| |
Collapse
|
20
|
Paravattil B, Wilby KJ. Optimizing assessors' mental workload in rater-based assessment: a critical narrative review. PERSPECTIVES ON MEDICAL EDUCATION 2019; 8:339-345. [PMID: 31728841 PMCID: PMC6904389 DOI: 10.1007/s40037-019-00535-6] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
INTRODUCTION Rater-based assessment has resulted in high cognitive demands for assessors within the education of health professionals. Rating quality may be influenced by the mental workload required of assessors to complete rating tasks. The objective of this review was to explore interventions or strategies aimed at measuring and reducing mental workload for improvement in assessment outcomes in health professions education. METHODS A critical narrative review was conducted for English-language articles using the databases PubMed, EMBASE, and Google Scholar from conception until November 2018. To be included, articles were eligible if they reported results of interventions aimed at measuring or reducing mental workload in rater-based assessment. RESULTS A total of six articles were included in the review. All studies were conducted in simulation settings (OSCEs or videotaped interactions). Of the four studies that measured mental workload, none found any reduction in mental workload as demonstrated by objective secondary task performance after interventions of assessor training or reductions in competency dimension assessment. Reductions in competency dimensions, however, did result in improvements in assessment quality across three studies. DISCUSSION The concept of mental workload in assessment in medical education needs further exploration, including investigation into valid measures of assessors' mental workload. It appears that adjusting raters' focus may be a valid strategy to improve assessment outcomes. Future research should be designed to inform how to best reduce load in assessments to improve quality, while balancing the type and quantity of data needed for judgments.
Collapse
Affiliation(s)
| | - Kyle John Wilby
- School of Pharmacy, University of Otago, Dunedin, New Zealand.
| |
Collapse
|
21
|
Wilby KJ, Dolmans DHJM, Austin Z, Govaerts MJB. Assessors' interpretations of narrative data on communication skills in a summative OSCE. MEDICAL EDUCATION 2019; 53:1003-1012. [PMID: 31304615 DOI: 10.1111/medu.13924] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/30/2019] [Revised: 03/08/2019] [Accepted: 05/29/2019] [Indexed: 06/10/2023]
Abstract
OBJECTIVES Increasingly, narrative assessment data are used to substantiate and enhance the robustness of assessor judgements. However, the interpretation of written assessment comments is inherently complex and relies on human (expert) judgements. The purpose of this study was to explore how expert assessors process and construe or bring meaning to narrative data when interpreting narrative assessment comments written by others in the setting of standardised performance assessment. METHODS Narrative assessment comments on student communication skills and communication scores across six objective structured clinical examination stations were obtained for 24 final-year pharmacy students. Aggregated narrative data across all stations were sampled for nine students (three good, three average and three poor performers, based on communication scores). A total of 10 expert assessors reviewed the aggregated set of narrative comments for each student. Cognitive (information) processing was captured through think-aloud procedures and verbal protocol analysis. RESULTS Expert assessors primarily made use of two strategies to interpret the narratives, namely comparing and contrasting, and forming mental images of student performance. Assessors appeared to use three different perspectives when interpreting narrative comments, including those of: (i) the student (placing him- or herself in the shoes of the student); (ii) the examiner (adopting the role of examiner and reinterpreting comments according to his or her own standards or beliefs), and (iii) the professional (acting as the profession's gatekeeper by considering the assessment to be a representation of real-life practice). CONCLUSIONS The present findings add to current understandings of assessors' interpretations of narrative performance data by identifying the strategies and different perspectives used by expert assessors to frame and bring meaning to written comments. Assessors' perspectives affect assessors' interpretations of assessment comments and are likely to be influenced by their beliefs, interpretations of the assessment setting and personal performance theories. These results call for the use of multiple assessors to account for variations in assessor perspectives in the interpretation of narrative assessment data.
Collapse
Affiliation(s)
- Kyle John Wilby
- School of Pharmacy, University of Otago, Dunedin, New Zealand
| | - Diana H J M Dolmans
- School of Health Professions Education (SHE), Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands
| | - Zubin Austin
- Leslie Dan Faculty of Pharmacy, University of Toronto, Toronto, Ontario, Canada
| | - Marjan J B Govaerts
- School of Health Professions Education (SHE), Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands
| |
Collapse
|
22
|
Yeates P, Cope N, Luksaite E, Hassell A, Dikomitis L. Exploring differences in individual and group judgements in standard setting. MEDICAL EDUCATION 2019; 53:941-952. [PMID: 31264741 DOI: 10.1111/medu.13915] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/22/2018] [Revised: 03/08/2019] [Accepted: 04/25/2019] [Indexed: 06/09/2023]
Abstract
CONTEXT Standard setting is critically important to assessment decisions in medical education. Recent research has demonstrated variations between medical schools in the standards set for shared items. Despite the centrality of judgement to criterion-referenced standard setting methods, little is known about the individual or group processes that underpin them. This study aimed to explore the operation and interaction of these processes in order to illuminate potential sources of variability. METHODS Using qualitative research, we purposively sampled across UK medical schools that set a low, medium or high standard on nationally shared items, collecting data by observation of graduation-level standard-setting meetings and semi-structured interviews with standard-setting judges. Data were analysed using thematic analysis based on the principles of grounded theory. RESULTS Standard setting occurred through the complex interaction of institutional context, judges' individual perspectives and group interactions. Schools' procedures, panel members and atmosphere produced unique contexts. Individual judges formed varied understandings of the clinical and technical features of each question, relating these to their differing (sometimes contradictory) conceptions of minimally competent students, by balancing information and making suppositions. Conceptions of minimal competence variously comprised: limited attendance; limited knowledge; poor knowledge application; emotional responses to questions; 'test-savviness', or a strategic focus on safety. Judges experienced tensions trying to situate these abstract conceptions in reality, revealing uncertainty. Groups constructively revised scores through debate, sharing information and often constructing detailed clinical representations of cases. Groups frequently displayed conformity, illustrating a belief that outlying judges were likely to be incorrect. Less frequently, judges resisted change, using emphatic language, bargaining or, rarely, 'polarisation' to influence colleagues. CONCLUSIONS Despite careful conduct through well-established procedures, standard setting is judgementally complex and involves uncertainty. Understanding whether or how these varied processes produce the previously observed variations in outcomes may offer routes to enhance equivalence of criterion-referenced standards.
Collapse
Affiliation(s)
- Peter Yeates
- Medical School Education Research Group (MERG), Keele University School of Medicine, Keele, UK
- Department of Acute Medicine, Fairfield General Hospital, Pennine Acute Hospitals NHS Trust, Bury, UK
| | - Natalie Cope
- Medical School Education Research Group (MERG), Keele University School of Medicine, Keele, UK
| | - Eva Luksaite
- Medical School Education Research Group (MERG), Keele University School of Medicine, Keele, UK
| | - Andrew Hassell
- Medical School Education Research Group (MERG), Keele University School of Medicine, Keele, UK
- The Haywood Hospital, Midlands Partnership Foundation NHS Trust, Stafford, UK
| | - Lisa Dikomitis
- Medical School Education Research Group (MERG), Keele University School of Medicine, Keele, UK
- Research Institute for Primary Care and Health Sciences, Keele University, Keele, UK
| |
Collapse
|
23
|
Wilby KJ, Govaerts MJB, Dolmans DHJM, Austin Z, van der Vleuten C. Reliability of narrative assessment data on communication skills in a summative OSCE. PATIENT EDUCATION AND COUNSELING 2019; 102:1164-1169. [PMID: 30711383 DOI: 10.1016/j.pec.2019.01.018] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/16/2018] [Revised: 12/20/2018] [Accepted: 01/24/2019] [Indexed: 06/09/2023]
Abstract
OBJECTIVE To quantitatively estimate the reliability of narrative assessment data regarding student communication skills obtained from a summative OSCE and to compare reliability to that of communication scores obtained from direct observation. METHODS Narrative comments and communication scores (scale 1-5) were obtained for 14 graduating pharmacy students across 6 summative OSCE stations with 2 assessors per station who directly observed student performance. Two assessors who had not observed the OSCE reviewed narratives and independently scored communication skills according to the same 5-point scale. Generalizability theory was used to estimate reliability. Correlation was used to evaluate the relationship between scores from each assessment method. RESULTS A total of 168 narratives and communication scores were obtained. The G-coefficients were 0.571 for scores provided by assessors present during the OSCE and 0.612 for scores from assessors who provided scores based on narratives only. Correlation between the two sets of scores was 0.5. CONCLUSION Reliability of communication scores is not dependent on whether assessors directly observe student performance or assess written narratives, yet both conditions appear to measure communication skills somewhat differently. PRACTICE IMPLICATIONS Narratives may be useful for summative decision-making and help overcome the current limitations of using solely quantitative scores.
Collapse
Affiliation(s)
- Kyle John Wilby
- College of Pharmacy, Qatar University, PO Box 2713, Doha, Qatar.
| | - Marjan J B Govaerts
- School of Health Professions Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, Universiteitssingel 60, 6229 ER Maastricht, Netherlands
| | - Diana H J M Dolmans
- School of Health Professions Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, Universiteitssingel 60, 6229 ER Maastricht, Netherlands
| | - Zubin Austin
- Leslie Dan Faculty of Pharmacy, University of Toronto, 144 College St., Toronto ON, M5S 3M2, Canada
| | - Cees van der Vleuten
- School of Health Professions Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, Universiteitssingel 60, 6229 ER Maastricht, Netherlands
| |
Collapse
|
24
|
Wilby KJ. Rethinking standardization: A call for new approaches to competency-based assessment. CURRENTS IN PHARMACY TEACHING & LEARNING 2019; 11:428-429. [PMID: 31040019 DOI: 10.1016/j.cptl.2019.01.004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/17/2018] [Revised: 10/17/2018] [Accepted: 01/05/2019] [Indexed: 06/09/2023]
Affiliation(s)
- Kyle J Wilby
- School of Pharmacy, University of Otago, PO Box 56, Dunedin 9054, New Zealand.
| |
Collapse
|
25
|
Lee V, Brain K, Martin J. From opening the 'black box' to looking behind the curtain: cognition and context in assessor-based judgements. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2019; 24:85-102. [PMID: 30302670 DOI: 10.1007/s10459-018-9851-0] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/25/2018] [Accepted: 09/06/2018] [Indexed: 06/08/2023]
Abstract
The increasing use of direct observation tools to assess routine performance has resulted in the growing reliance on assessor-based judgements in the workplace. However, we have a limited understanding of how assessors make judgements and formulate ratings in real world contexts. The current research on assessor cognition has largely focused on the cognitive domain but the contextual factors are equally important, and both are closely interconnected. This study aimed to explore the perceived cognitive and contextual factors influencing Mini-CEX assessor judgements in the Emergency Department setting. We used a conceptual framework of assessor-based judgement to develop a sequential mixed methods study. We analysed and integrated survey and focus group results to illustrate self-reported cognitive and contextual factors influencing assessor judgements. We used situated cognition theory as a sensitizing lens to explore the interactions between people and their environment. The major factors highlighted through our mixed methods study were: clarity of the assessment, reliance on and variable approach to overall impression (gestalt), role tension especially when giving constructive feedback, prior knowledge of the trainee and case complexity. We identified prevailing tensions between participants (assessors and trainees), interactions (assessment and feedback) and setting. The two practical implications of our research are the need to broaden assessor training to incorporate both cognitive and contextual domains, and the need to develop a more holistic understanding of assessor-based judgements in real world contexts to better inform future research and development in workplace-based assessments.
Collapse
Affiliation(s)
- Victor Lee
- Department of Emergency Medicine, Austin Health, P.O. Box 5555, Heidelberg, VIC, 3084, Australia.
| | | | - Jenepher Martin
- Eastern Health Clinical School, Monash University and Deakin University, Box Hill, VIC, Australia
| |
Collapse
|
26
|
Butler AC. An introduction to the forum: Cognitive perspectives on the assessment of professional competence. JOURNAL OF APPLIED RESEARCH IN MEMORY AND COGNITION 2018. [DOI: 10.1016/j.jarmac.2018.04.002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
|
27
|
Mayer RE. What professionals know: Contributions of cognitive science to the assessment of professional competence. JOURNAL OF APPLIED RESEARCH IN MEMORY AND COGNITION 2018. [DOI: 10.1016/j.jarmac.2018.03.004] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
|