1
|
Rienits H. The other side of the mark sheet: lessons learnt when medical students assess peers in formative clinical examinations. Front Med (Lausanne) 2024; 11:1395466. [PMID: 38903805 PMCID: PMC11187237 DOI: 10.3389/fmed.2024.1395466] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2024] [Accepted: 05/27/2024] [Indexed: 06/22/2024] Open
Abstract
This study aimed to investigate the experience of medical students assessing their cohort peers in formative clinical assessment. The exercise was designed to provide students with a formative experience prior to their summative assessment, and to determine what students could learn by being on the "other side of the mark sheet." Students were grateful for the experience learning both from the assessment practice, and from the individual written feedback provided immediately afterwards. They also described how much they learnt from seeing the assessment from the assessor's viewpoint, with many students commenting that they learnt more from being the "assessor" than from being the "student" in the process. Students were asked how they felt about being assessed by their peers, with some describing the experience as being more intimidating and stressful than when compared to assessment by clinicians. An interesting aspect of this study is that it also demonstrates some findings which suggest that the students' current learning context appears to have an effect on their attitudes to their peers as assessors. It is possible the competitive cultural milieu of the teaching hospital environment may have a negative effect on medical student collegiality and peer support.
Collapse
Affiliation(s)
- Helen Rienits
- Graduate School of Medicine, Faculty of Science, Medicine and Health, University of Wollongong, Wollongong, NSW, Australia
| |
Collapse
|
2
|
Sims DA, Lucio-Ramirez CA, Cilliers FJ. Factors influencing clinician-educators' assessment practice in varied Southern contexts: a health behaviour theory perspective. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2024:10.1007/s10459-024-10341-3. [PMID: 38811446 DOI: 10.1007/s10459-024-10341-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/03/2023] [Accepted: 05/12/2024] [Indexed: 05/31/2024]
Abstract
In many contexts, responsibility for exit-level assessment design and implementation in undergraduate medical programmes lies with individuals who convene clinical clerkships. Their assessment practice has significant consequences for students' learning and the patients and communities that graduates will serve. Interventions to enhance assessment must involve these assessors, yet little is known about factors influencing their assessment practice. The purpose of this study was to explore factors that influence assessment practice of clerkship convenors in three varied low-and-middle income contexts in the global South. Taking assessment practice as a behaviour, Health Behaviour Theory (HBT) was deployed as a theoretical framework to explore, describe and explain assessor behaviour. Thirty-one clinician-educators responsible for designing and implementing high-stakes clerkship assessment were interviewed in South Africa and Mexico. Interacting personal and contextual factors influencing clinician-educator assessment intention and action were identified. These included attitude, influenced by impact and response appraisal, and perceived self-efficacy; along with interpersonal, physical and organisational, and distal contextual factors. Personal competencies and conducive environments supported intention to action transition. While previous research has typically explored factors in isolation, the HBT framing enabled a systematic and coherent account of assessor behaviour. These findings add a particular contextual perspective to understanding assessment practice, yet also resonate with and extend existing work that predominantly emanates from high-income contexts in the global North. These findings provide a foundation for the planning of assessment change initiatives, such as targeted, multi-factorial faculty development.
Collapse
Affiliation(s)
- Danica Anne Sims
- University of Oxford, Oxford, UK.
- University of Johannesburg, Johannesburg, South Africa.
| | | | | |
Collapse
|
3
|
Torre D, Daniel M, Ratcliffe T, Durning SJ, Holmboe E, Schuwirth L. Programmatic Assessment of Clinical Reasoning: New Opportunities to Meet an Ongoing Challenge. TEACHING AND LEARNING IN MEDICINE 2024:1-9. [PMID: 38794865 DOI: 10.1080/10401334.2024.2333921] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/13/2023] [Accepted: 02/29/2024] [Indexed: 05/26/2024]
Abstract
Issue: Clinical reasoning is essential to physicians' competence, yet assessment of clinical reasoning remains a significant challenge. Clinical reasoning is a complex, evolving, non-linear, context-driven, and content-specific construct which arguably cannot be assessed at one point in time or with a single method. This has posed challenges for educators for many decades, despite significant development of individual assessment methods. Evidence: Programmatic assessment is a systematic assessment approach that is gaining momentum across health professions education. Programmatic assessment, and in particular assessment for learning, is well-suited to address the challenges with clinical reasoning assessment. Several key principles of programmatic assessment are particularly well-aligned with developing a system to assess clinical reasoning: longitudinality, triangulation, use of a mix of assessment methods, proportionality, implementation of intermediate evaluations/reviews with faculty coaches, use of assessment for feedback, and increase in learners' agency. Repeated exposure and measurement are critical to develop a clinical reasoning assessment narrative, thus the assessment approach should optimally be longitudinal, providing multiple opportunities for growth and development. Triangulation provides a lens to assess the multidimensionality and contextuality of clinical reasoning and that of its different, yet related components, using a mix of different assessment methods. Proportionality ensures the richness of information on which to draw conclusions is commensurate with the stakes of the decision. Coaching facilitates the development of a feedback culture and allows to assess growth over time, while enhancing learners' agency. Implications: A programmatic assessment model of clinical reasoning that is developmentally oriented, optimizes learning though feedback and coaching, uses multiple assessment methods, and provides opportunity for meaningful triangulation of data can help address some of the challenges of clinical reasoning assessment.
Collapse
Affiliation(s)
- Dario Torre
- Department of Medical Education, University of Central Florida, Orlando, FL, USA
| | - Michelle Daniel
- Department of Emergency Medicine, University of California, San Diego, CA, USA
| | - Temple Ratcliffe
- Department of Medicine, The Joe R and Teresa Lozano Long School of Medicine at University of Texas Health, Texas, USA
| | - Steven J Durning
- Center for Heath Profession Education, Uniformed Services University Center for Neuroscience and Regenerative Medicine, Bethesda, Maryland, USA
| | - Eric Holmboe
- Milestones Development and Evaluation, Accreditation Council for Graduate Medical Education, Chicago, IL, USA
| | | |
Collapse
|
4
|
Tavares W, Pearce J. Attending to Variable Interpretations of Assessment Science and Practice. TEACHING AND LEARNING IN MEDICINE 2024; 36:244-252. [PMID: 37431929 DOI: 10.1080/10401334.2023.2231923] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/03/2022] [Accepted: 05/31/2023] [Indexed: 07/12/2023]
Abstract
Issue: The way educators think about the nature of competence, the approaches one selects for the assessment of competence, what generated data implies, and what counts as good assessment now involve broader and more diverse interpretive processes. Broadening philosophical positions in assessment has educators applying different interpretations to similar assessment concepts. As a result, what is claimed through assessment, including what counts as quality, can be different for each of us despite using similar activities and language. This is leading to some uncertainty on how to proceed or worse, provides opportunities for questioning the legitimacy of any assessment activity or outcome. While some debate in assessment is inevitable, most have been within philosophical positions (e.g., how best to minimize error), whereas newer debates are happening across philosophical positions (e.g., whether error is a useful concept). As new ways of approaching assessment have emerged, the interpretive nature of underlying philosophical positions has not been sufficiently attended to. Evidence: We illustrate interpretive processes of assessment in action by: (a) summarizing the current health professions assessment context from a philosophical perspective as a way of describing its evolution; (b) demonstrating implications in practice using two examples (i.e., analysis of assessment work and validity claims); and (c) examining pragmatism to demonstrate how even within specific philosophical positions opportunities for variable interpretations still exist. Implications: Our concern is not that assessment designers and users have different assumptions, but that practically, educators may unknowingly (or insidiously) apply different assumptions, and methodological and interpretive norms, and subsequently settle on different views on what serves as quality assessment even for the same assessment program or event. With the state of assessment in health professions in flux, we conclude by calling for a philosophically explicit approach to assessment, and underscore assessment as, fundamentally, an interpretive process - one which demands the careful elucidation of philosophical assumptions to promote understanding and ultimately defensibility of assessment processes and outcomes.
Collapse
Affiliation(s)
- Walter Tavares
- The Wilson Centre for Health Professions Education Research, and Post-Graduate Medical Education, Toronto, Canada
- Temerty Faculty of Medicine, University Health Network and University of Toronto, Toronto, Canada
- Department of Health and Society, University of Toronto, Toronto, Canada
- York Region Paramedic Services, Community Health Services, Regional Municipality of York, Newmarket, Canada
| | - Jacob Pearce
- Tertiary Education, Australian Council for Educational Research, Camberwell, Australia
| |
Collapse
|
5
|
Dickie J, Sherriff A, McEwan M, Bell A, Naudi K. Longitudinal assessment of undergraduate dental students: Building evidence for validity. EUROPEAN JOURNAL OF DENTAL EDUCATION : OFFICIAL JOURNAL OF THE ASSOCIATION FOR DENTAL EDUCATION IN EUROPE 2023; 27:1136-1150. [PMID: 37141495 DOI: 10.1111/eje.12908] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/21/2021] [Revised: 10/27/2022] [Accepted: 03/31/2023] [Indexed: 05/06/2023]
Abstract
PURPOSE To investigate the content and criterion validity, and reliability of longitudinal clinical assessment of undergraduate dental student clinical competence by determining patterns of clinical performance and comparing them with validated standalone undergraduate examinations. METHODS Group-based trajectory models tracking students' clinical performance over time were produced from LIFTUPP© data for three dental student cohorts (2017-19; n = 235) using threshold models based on the Bayesian information criterion. Content validity was investigated using LIFTUPP© performance indicator 4 as the threshold for competence. Criterion validity was investigated using performance indicator 5 to create distinct trajectories of performance before linking and cross-tabulating trajectory group memberships with a 'top 20%' performance in the final Bachelor of Dental Surgery (BDS) examinations. Reliability was calculated using Cronbach's alpha. RESULTS Threshold 4 models showed all students followed a single upward trajectory in all three cohorts, showing clear progression in competence over three clinical BDS years. A threshold 5 model produced two distinct trajectories, and in each cohort a 'better performing' trajectory was identified. Students allocated to the 'better performing' trajectories scored higher on average in the final examinations for cohort 2 (29% vs 18% (BDS4); 33% vs. 15% (BDS5)) and cohort 3 (19% vs. 16% (BDS4); 21% vs. 16% (BDS5)). Reliability for the undergraduate examinations was high for all three cohorts (≥0.8815) and did not change appreciably when longitudinal assessment was included. CONCLUSIONS There is some evidence to support that longitudinal data have a degree of content and criterion validity for assessing the development of clinical competence in undergraduate dental students, which should increase confidence in decisions based on these data. The findings also provide a good foundation for subsequent research.
Collapse
Affiliation(s)
- Jamie Dickie
- University of Glasgow School of Medicine, Dentistry & Nursing, College of Medical, Veterinary & Life Sciences, Glasgow, UK
| | - Andrea Sherriff
- University of Glasgow School of Medicine, Dentistry & Nursing, College of Medical, Veterinary & Life Sciences, Glasgow, UK
| | - Michael McEwan
- University of Glasgow, Learning Enhancement and Academic Development Service, Glasgow, UK
| | - Aileen Bell
- University of Glasgow School of Medicine, Dentistry & Nursing, College of Medical, Veterinary & Life Sciences, Glasgow, UK
| | - Kurt Naudi
- University of Glasgow School of Medicine, Dentistry & Nursing, College of Medical, Veterinary & Life Sciences, Glasgow, UK
| |
Collapse
|
6
|
Sims DA, Cilliers FJ. Clinician educators' conceptions of assessment in medical education. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2023; 28:1053-1077. [PMID: 36662334 PMCID: PMC10624725 DOI: 10.1007/s10459-022-10197-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/26/2022] [Accepted: 12/11/2022] [Indexed: 06/17/2023]
Abstract
In pursuing assessment excellence, clinician-educators who design and implement assessment are pivotal. The influence of their assessment practice in university-run licensure exams on student learning has direct implications for future patient care. While teaching practice has been shown to parallel conceptions of teaching, we know too little about conceptions of assessment in medical education to know if this is the case for assessment practice and conceptions of assessment. To explore clinician-educators' conceptions of assessment, a phenomenographic study was undertaken. Phenomenography explores conceptions, the qualitatively different ways of understanding a phenomenon. Data analysis identifies a range of hierarchically inclusive categories of understanding, from simple to more complex, and the dimensions that distinguish each category or conception. Thirty-one clerkship convenors in three diverse Southern settings were interviewed in three cycles of iterative data collection and analysis. Four conceptions of assessment were identified: passive operator, awakening enquirer, active owner and scholarly assessor. Six dimensions were elucidated to describe and distinguish each conception: purpose of assessment; temporal perspective; role and responsibility; accountability; reflexivity and emotional valence. Additionally, three characteristics that appeared to track the progressive nature of the conceptions were identified: professional identity, assessment literacy and self-efficacy. These conceptions encompass and extend previously described conceptions across different educational levels, disciplines and contexts, suggesting applicability to other settings. There is some evidence of a relationship between conceptions and practice, suggesting, together with the hierarchical nature of these conceptions, that targeting conceptions during faculty development may be an effective approach to enhance assessment practice.
Collapse
Affiliation(s)
- D A Sims
- University of the Western Cape, 14 Blanckenberg Street, Bellville, South Africa.
| | - F J Cilliers
- Faculty of Health Sciences, University of Cape Town, Cape Town, South Africa
| |
Collapse
|
7
|
Edwards C, Perry R, Chester D, Childs J. Entrustable professional activities of graduate accredited General Medical Sonographers in Australia - Industry perceptions. J Med Radiat Sci 2023; 70:229-238. [PMID: 37029950 PMCID: PMC10500106 DOI: 10.1002/jmrs.676] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/11/2023] [Accepted: 03/20/2023] [Indexed: 04/09/2023] Open
Abstract
INTRODUCTION Linking individual competencies to entrustable professional tasks provides a holistic view of Sonography graduate work readiness. The Australian Sonographers Accreditation Registry (ASAR) publishes a set of entrustable professional activities (EPAs) as part of its Standards for Accreditation of Sonography Courses. EPAs are distinct ultrasound examinations grouped within six critical practice units. This study reports on industry perspectives of current EPAs and their classification for graduates completing general sonography courses in Australia. The article also examines the value of EPAs and links their function to the assessment of graduate competency. METHODS An online survey tool elicited stakeholder feedback on graduate EPAs across six critical practice units and the potential for including a new Paediatric unit. From an original sample size of 655, 309 responded to questions about general sonography courses. RESULTS A majority (55.3%) recommended no changes to the existing EPA list, and 44.7% recommended amending the list. From respondents that recommended changes (138/309), all current EPAs received >80% agreement to be retained; in addition, nine new examinations received >70% agreement for inclusion at the graduate level. Whilst 42.7% (132/309) supported the current ASAR model requiring competency in five out of six critical practice units, 45.6% (141/309) recommended increasing it to all six. There was limited support, 11.7% (36/309), to reduce this number. Responding to the potential to add a new Paediatric specific critical practice unit, 61.8% (181/293) recommended its inclusion. CONCLUSIONS The findings demonstrate that the current list of EPAs aligns with industry expectations. In contrast, there are divergent views on the modelling and grouping of critical practice units. The article's critical analysis of the results and implications provides stakeholders with a practical approach to clinical teaching and EPA assessment, and helps to inform any review of accreditation standards.
Collapse
Affiliation(s)
- Christopher Edwards
- School of Clinical Sciences, Faculty of HealthQueensland University of TechnologyBrisbaneQueenslandAustralia
| | - Rebecca Perry
- Allied Health and Human PerformanceUniversity of South AustraliaAdelaideSouth AustraliaAustralia
| | - Deanne Chester
- School of Health, Medical and Applied SciencesCentral Queensland UniversityBrisbaneQueenslandAustralia
| | - Jessie Childs
- Allied Health and Human PerformanceUniversity of South AustraliaAdelaideSouth AustraliaAustralia
| |
Collapse
|
8
|
Wong WYA, Thistlethwaite J, Moni K, Roberts C. Using cultural historical activity theory to reflect on the sociocultural complexities in OSCE examiners' judgements. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2023; 28:27-46. [PMID: 35943605 PMCID: PMC9992227 DOI: 10.1007/s10459-022-10139-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 03/03/2021] [Accepted: 06/28/2022] [Indexed: 06/15/2023]
Abstract
Examiners' judgements play a critical role in competency-based assessments such as objective structured clinical examinations (OSCEs). The standardised nature of OSCEs and their alignment with regulatory accountability assure their wide use as high-stakes assessment in medical education. Research into examiner behaviours has predominantly explored the desirable psychometric characteristics of OSCEs, or investigated examiners' judgements from a cognitive rather than a sociocultural perspective. This study applies cultural historical activity theory (CHAT) to address this gap in exploring examiners' judgements in a high-stakes OSCE. Based on the idea that OSCE examiners' judgements are socially constructed and mediated by their clinical roles, the objective was to explore the sociocultural factors that influenced examiners' judgements of student competence and use the findings to inform examiner training to enhance assessment practice. Seventeen semi-structured interviews were conducted with examiners who assessed medical student competence in progressing to the next stage of training in a large-scale OSCE at one Australian university. The initial thematic analysis provided a basis for applying CHAT iteratively to explore the sociocultural factors and, specifically, the contradictions created by interactions between different elements such as examiners and rules, thus highlighting the factors influencing examiners' judgements. The findings indicated four key factors that influenced examiners' judgements: examiners' contrasting beliefs about the purpose of the OSCE; their varying perceptions of the marking criteria; divergent expectations of student competence; and idiosyncratic judgement practices. These factors were interrelated with the activity systems of the medical school's assessment practices and the examiners' clinical work contexts. Contradictions were identified through the guiding principles of multi-voicedness and historicity. The exploration of the sociocultural factors that may influence the consistency of examiners' judgements was facilitated by applying CHAT as an analytical framework. Reflecting upon these factors at organisational and system levels generated insights for creating fit-for-purpose examiner training to enhance assessment practice.
Collapse
Affiliation(s)
- Wai Yee Amy Wong
- School of Education and Faculty of Medicine, The University of Queensland, Brisbane, QLD, 4072, Australia.
- School of Nursing and Midwifery, Queen's University Belfast, Belfast, BT9 7BL, UK.
| | - Jill Thistlethwaite
- Faculty of Health, The University of Technology Sydney, Sydney, NSW, 2007, Australia
| | - Karen Moni
- School of Education, The University of Queensland, Brisbane, QLD, 4072, Australia
| | - Chris Roberts
- Sydney Medical School, Faculty of Medicine and Health, The University of Sydney, Sydney, NSW, 2006, Australia
| |
Collapse
|
9
|
Gonzalez PR, Paravattil B, Wilby KJ. Mental effort in the assessment of critical reflection: Implications for assessment quality and scoring. CURRENTS IN PHARMACY TEACHING & LEARNING 2022; 14:830-834. [PMID: 35914842 DOI: 10.1016/j.cptl.2022.06.016] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/05/2021] [Revised: 05/16/2022] [Accepted: 06/16/2022] [Indexed: 06/15/2023]
Abstract
INTRODUCTION Critical reflection is a mainstay in the training of health professionals, yet assessment of reflection is commonly described as difficult, taxing, and resulting in inconsistent scoring across assessors. At the same time, there is evidence from experiential and simulation settings that assessors' mental effort may explain assessor variability, which could be a target for simplifications in assessment design. Assessors' mental effort for assessment of reflection is currently unknown. This study aimed to determine reliability of rubric scoring of critical reflection, variation in pass-fail rates, and the relationship between reflection scores and assessors' perceived mental effort. METHODS Eleven assessors were recruited to assess six reflection assignments using a published rubric. Mental effort was measured using the Paas scale for each assignment assessed and was correlated with rubric scores for each assignment. RESULTS Findings showed inconsistency in scoring between assessors, resulting in varying pass rates for each assignment (55-100%). All assignments demonstrated negative correlations between rubric scores and perceived mental effort (r = -0.115 to -0.649). CONCLUSIONS Findings support the notion that more work should be done to optimize assessment of critical reflection. Future studies should focus on disentangling the influence on mental effort of scoring tools, assignment structures, and writing quality.
Collapse
Affiliation(s)
| | | | - Kyle John Wilby
- College of Pharmacy, Faculty of Health, Dalhousie University, 5968 College Street, Halifax, Nova Scotia, Canada.
| |
Collapse
|
10
|
Jeyalingam T, Walsh CM, Tavares W, Mylopoulos M, Hodwitz K, Liu LWC, Heitman SJ, Brydges R. Variable or Fixed? Exploring Entrustment Decision Making in Workplace- and Simulation-Based Assessments. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2022; 97:1057-1064. [PMID: 35263307 DOI: 10.1097/acm.0000000000004661] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
PURPOSE Many models of competency-based medical education (CBME) emphasize assessing entrustable professional activities (EPAs). Despite the centrality of EPAs, researchers have not compared rater entrustment decisions for the same EPA across workplace- and simulation-based assessments. This study aimed to explore rater entrustment decision making across these 2 assessment settings. METHOD An interview-based study using a constructivist grounded theory approach was conducted. Gastroenterology faculty at the University of Toronto and the University of Calgary completed EPA assessments of trainees' endoscopic polypectomy performance in both workplace and simulation settings between November 2019 and January 2021. After each assessment, raters were interviewed to explore how and why they made entrustment decisions within and across settings. Transcribed interview data were coded iteratively using constant comparison to generate themes. RESULTS Analysis of 20 interviews with 10 raters found that participants (1) held multiple meanings of entrustment and expressed variability in how they justified their entrustment decisions and scoring, (2) held personal caveats for making entrustment decisions "comfortably" (i.e., authenticity, task-related variability, opportunity to assess trainee responses to adverse events, and the opportunity to observe multiple performances over time), (3) experienced cognitive tensions between formative and summative purposes when assessing EPAs, and (4) experienced relative freedom when using simulation to formatively assess EPAs but constraint when using only simulation-based assessments for entrustment decision making. CONCLUSIONS Participants spoke about and defined entrustment variably, which appeared to produce variability in how they judged entrustment across participants and within and across assessment settings. These rater idiosyncrasies suggest that programs implementing CBME must consider how such variability affects the aggregation of EPA assessments, especially those collected in different settings. Program leaders might also consider how to fulfill raters' criteria for comfortably making entrustment decisions by ensuring clear definitions and purposes when designing and integrating workplace- and simulation-based assessments.
Collapse
Affiliation(s)
- Thurarshen Jeyalingam
- T. Jeyalingam is an advanced fellow in luminal therapeutic endoscopy, University of Calgary, Calgary, Alberta, Canada; ORCID: http://orcid.org/0000-0002-7254-9639
| | - Catharine M Walsh
- C.M. Walsh is a staff gastroenterologist, Division of Gastroenterology, Hepatology and Nutrition, educational researcher, SickKids Learning Institute, scientist, Child Health Evaluative Sciences, SickKids Research Institute, Hospital for Sick Children, scientist, Wilson Centre, and associate professor of paediatrics, University of Toronto, Toronto, Ontario, Canada; ORCID: http://orcid.org/0000-0003-3928-703X
| | - Walter Tavares
- W. Tavares is assistant professor and scientist, Wilson Centre and Temerty Faculty of Medicine, University Health Network and University of Toronto, Toronto, Ontario, Canada; ORCID: http://orcid.org/0000-0001-8267-9448
| | - Maria Mylopoulos
- M. Mylopoulos is associate professor, Department of Paediatrics, and scientist and associate director, Wilson Centre, University of Toronto, Temerty Faculty of Medicine, Toronto, Ontario, Canada; ORCID: http://orcid.org/0000-0003-0012-5375
| | - Kathryn Hodwitz
- K. Hodwitz is a clinical research specialist, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, Toronto, Ontario, Canada; ORCID: http://orcid.org/0000-0003-3099-1709
| | - Louis W C Liu
- L.W.C. Liu is associate professor, Department of Medicine, University of Toronto, and head, Division of Gastroenterology and Hepatology, University Health Network and Sinai Health, Toronto, Ontario, Canada; ORCID: http://orcid.org/0000-0001-6899-7941
| | - Steven J Heitman
- S.J. Heitman is associate professor, Departments of Medicine and Community Health Sciences, Cumming School of Medicine, holds the N.B. Hershfield Chair in Therapeutic Endoscopy, University of Calgary, is medical director, Forzani & MacPhail Colon Cancer Screening Centre, and scientific director, Digestive Health Strategic Clinical Network, Alberta Health Services, Calgary, Alberta, Canada; ORCID: http://orcid.org/0000-0002-4952-779X
| | - Ryan Brydges
- R. Brydges is a scientist and holds the Professorship in Technology-Enabled Education, St. Michael's Hospital, Unity Health Toronto, and is associate professor, Department of Medicine, University of Toronto, Toronto, Ontario, Canada; ORCID: https://orcid.org/0000-0001-5203-7049
| |
Collapse
|
11
|
Marceau M, St-Onge C, Gallagher F, Young M. Validity as a social imperative: users' and leaders' perceptions. CANADIAN MEDICAL EDUCATION JOURNAL 2022; 13:22-36. [PMID: 35875440 PMCID: PMC9297243 DOI: 10.36834/cmej.73518] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
INTRODUCTION Recently, validity as a social imperative was proposed as an emerging conceptualization of validity in the assessment literature in health professions education (HPE). To further develop our understanding, we explored the perceived acceptability and anticipated feasibility of validity as a social imperative with users and leaders engaged with assessment in HPE in Canada. METHODS We conducted a qualitative interpretive description study. Purposeful and snowball sampling were used to recruit participants for semi-structured individual interviews and focus groups. Each transcript was analyzed by two team members and discussed with the team until consensus was reached. RESULTS We conducted five focus group and eleven interviews with two different stakeholder groups (users and leaders). Our findings suggest that the participants perceived the concept of validity as a social imperative as acceptable. Regardless of group, participants shared similar considerations regarding: the limits of traditional validity models, the concept's timeliness and relevance, the need to clarify some terms used to characterize the concept, the similarities with modern theories of validity, and the anticipated challenges in applying the concept in practice. In addition, participants discussed some limits with current approaches to validity in the context of workplace-based and programmatic assessment. CONCLUSION Validity as a social imperative can be interwoven throughout existing theories of validity and may represent how HPE is adapting traditional models of validity in order to respond to the complexity of assessment in HPE; however, challenges likely remain in operationalizing the concept prior to its implementation.
Collapse
Affiliation(s)
- Mélanie Marceau
- School of Nursing, Faculty of Medicine and Health Sciences, Université de Sherbrooke, Quebec, Canada
| | - Christina St-Onge
- Department of Medicine, Faculty of Medicine and Health Sciences, Université de Sherbrooke, Quebec, Canada
| | - Frances Gallagher
- School of Nursing, Faculty of Medicine and Health Sciences, Université de Sherbrooke, Quebec, Canada
| | - Meredith Young
- Institute of Health Sciences Education, Faculty of Medicine and Health Sciences, McGill University, Québec, Canada
| |
Collapse
|
12
|
Katz SJ, Wang D. Comparing Entrustable Professional Activity Scores Given by Faculty Physicians and Senior Trainees to First-Year Residents. Cureus 2022; 14:e25798. [PMID: 35836450 PMCID: PMC9273189 DOI: 10.7759/cureus.25798] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 06/09/2022] [Indexed: 11/05/2022] Open
|
13
|
Barman L, McGrath C, Josephsson S, Silén C, Bolander Laksov K. Safeguarding fairness in assessments-How teachers develop joint practices. MEDICAL EDUCATION 2022; 56:651-659. [PMID: 35263464 PMCID: PMC9310582 DOI: 10.1111/medu.14789] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Received: 09/01/2021] [Revised: 02/03/2022] [Accepted: 02/26/2022] [Indexed: 06/14/2023]
Abstract
INTRODUCTION In light of reforms demanding increased transparency of student performance assessments, this study offers an in-depth perspective of how teachers develop their assessment practice. Much is known about factors that influence assessments, and different solutions claim to improve the validity and reliability of assessments of students' clinical competency. However, little is known about how teachers go about improving their assessment practices. This study aims to contribute empirical findings about how teachers' assessment practice may change when shared criteria for assessing students' clinical competency are developed and implemented. METHODS Using a narrative-in-action research approach grounded in narrative theory about human sense-making, one group including nine health professions teachers was studied over a period of 1 year. Drawing upon data from observations, interviews, formal documents and written reflections from these teachers, we performed a narrative analysis to reveal how these teachers made sense of experiences associated with the development and implementation of joint grading criteria for assessing students' clinical performances. RESULTS The findings present a narrative showing how a shared assessment practice took years to develop and was based on the teachers changed approach to scrutiny. The teachers became highly motivated to use grading criteria to ensure fairness in assessments, but more importantly, to fulfil their moral obligation towards patients. The narrative also demonstrates how these teachers reasoned about dilemmas that arose when they applied standardised assessment criteria. DISCUSSION The narrative analysis shows clearly how teachers' development and application of assessment standards are embedded in local practices. Our findings highlight the importance of teachers' joint discussions on how to interpret criteria applied in formative and summative assessments of students' performances. In particular, teachers' different approaches to assessing 'pieces of skills' versus making holistic judgements on students' performances, regardless of whether the grading criteria are clear and well-articulated on paper, should be acknowledged. Understanding the journey that these teachers made gives new perspectives as to how faculty can be supported when assessments of professionalism and clinical competency are developed.
Collapse
Affiliation(s)
- Linda Barman
- Department of Learning in Engineering SciencesKTH Royal Institute of TechnologyStockholmSweden
| | - Cormac McGrath
- Department of EducationStockholm UniversityStockholmSweden
| | - Staffan Josephsson
- Department of Neurobiology, Care Sciences and SocietyKarolinska InstitutetStockholmSweden
| | - Charlotte Silén
- Department of Learning, Informatics, Management and EthicsKarolinska InstitutetStockholmSweden
| | - Klara Bolander Laksov
- Department of EducationStockholm UniversityStockholmSweden
- Department of Learning, Informatics, Management and EthicsKarolinska InstitutetStockholmSweden
| |
Collapse
|
14
|
Ryan MS, Khamishon R, Richards A, Perera R, Garber A, Santen SA. A Question of Scale? Generalizability of the Ottawa and Chen Scales to Render Entrustment Decisions for the Core EPAs in the Workplace. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2022; 97:552-561. [PMID: 34074896 DOI: 10.1097/acm.0000000000004189] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
PURPOSE Assessments of the Core Entrustable Professional Activities (Core EPAs) are based on observations of supervisors throughout a medical student's progression toward entrustment. The purpose of this study was to compare generalizability of scores from 2 entrustment scales: the Ottawa Surgical Competency Operating Room Evaluation (Ottawa) scale and an undergraduate medical education supervisory scale proposed by Chen and colleagues (Chen). A secondary aim was to determine the impact of frequent assessors on generalizability of the data. METHOD For academic year 2019-2020, the Virginia Commonwealth University School of Medicine modified a previously described workplace-based assessment (WBA) system developed to provide feedback for the Core EPAs across clerkships. The WBA scored students' performance using both Ottawa and Chen scales. Generalizability (G) and decision (D) studies were performed using an unbalanced random-effects model to determine the reliability of each scale. Secondary G- and D-studies explored whether faculty who rated more than 5 students demonstrated better reliability. The Phi-coefficient was used to estimate reliability; a cutoff of at least 0.70 was used to conduct D-studies. RESULTS Using the Ottawa scale, variability attributable to the student ranged from 0.8% to 6.5%. For the Chen scale, student variability ranged from 1.8% to 7.1%. This indicates the majority of variation was due to the rater (42.8%-61.3%) and other unexplained factors. Between 28 and 127 assessments were required to obtain a Phi-coefficient of 0.70. For 2 EPAs, using faculty who frequently assessed the EPA improved generalizability, requiring only 5 and 13 assessments for the Chen scale. CONCLUSIONS Both scales performed poorly in terms of learner-attributed variance, with some improvement in 2 EPAs when considering only frequent assessors using the Chen scale. Based on these findings in conjunction with prior evidence, the authors provide a root cause analysis highlighting challenges with WBAs for Core EPAs.
Collapse
Affiliation(s)
- Michael S Ryan
- M.S. Ryan is associate professor and assistant dean for clinical medical education, Department of Pediatrics, Virginia Commonwealth University, Richmond, Virginia; ORCID: https://orcid.org/0000-0003-3266-9289
| | - Rebecca Khamishon
- R. Khamishon is a fourth-year medical student, Virginia Commonwealth University, Richmond, Virginia
| | - Alicia Richards
- A. Richards is a graduate student, Department of Biostatistics, Virginia Commonwealth University, Richmond, Virginia
| | - Robert Perera
- R. Perera is associate professor, Department of Biostatistics, Virginia Commonwealth University, Richmond, Virginia
| | - Adam Garber
- A. Garber is associate professor, Department of Internal Medicine, Virginia Commonwealth University, Richmond, Virginia; ORCID: https://orcid.org/0000-0002-7296-2896
| | - Sally A Santen
- S.A. Santen is professor and senior associate dean of assessment, evaluation, and scholarship, Department of Emergency Medicine, Virginia Commonwealth University, Richmond, Virginia; ORCID: https://orcid.org/0000-0002-8327-8002
| |
Collapse
|
15
|
Bramley AL, McKenna L. Entrustable professional activities in entry-level health professional education: A scoping review. MEDICAL EDUCATION 2021; 55:1011-1032. [PMID: 33884655 DOI: 10.1111/medu.14539] [Citation(s) in RCA: 28] [Impact Index Per Article: 9.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/23/2021] [Revised: 03/25/2021] [Accepted: 04/01/2021] [Indexed: 06/12/2023]
Abstract
Entrustable professional activities (EPAs) are a recent enhancement to competency-based health professional education that describe the observable work done by a competent health professional. Through defining education outcomes in a work-based context, EPAs offer potential to identify skill gaps in individual or student cohorts and focus improvements. Entrustable professional activities have been pioneered and gained rapid acceptance in postgraduate medical education; however, less is known about their application and use in undergraduate or entry-level health professional education. The Joanna Briggs Institute scoping review methodology was used to explore how and in what context EPAs are being used in entry-level health professional education. Databases searched include CINAHL, EMBASE, MEDLINE, Web of Science and PsycINFO. A total of 748 abstracts were returned after duplicates removed, and 127 full-text articles were screened with 30 included for data extraction. Publications in this area have recently accelerated with disciplines of professions of medicine, pharmacy, dietetics and physician assistants reporting on EPA development, implementation and evaluation. EPA use has been reported in the United States, Canada, Europe Australia and Central America. Major motivation reported for EPA use is to improve patient safety by aligning performance and expectations and to improve student assessment. Several studies report on the use of EPAs to evaluate different curriculum models or identify curriculum gaps representing potential application in education research.
Collapse
Affiliation(s)
- Andrea Louise Bramley
- Department of Nutrition and Dietetics, School of Allied Health, Human Services and Sport, College of Science, Health and Engineering, La Trobe University, Melbourne, VIC, Australia
| | - Lisa McKenna
- School of Nursing and Midwifery, College of Science, Health and Engineering, La Trobe University, Melbourne, VIC, Australia
| |
Collapse
|
16
|
Ryan MS, Richards A, Perera R, Park YS, Stringer JK, Waterhouse E, Dubinsky B, Khamishon R, Santen SA. Generalizability of the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) Scale to Assess Medical Student Performance on Core EPAs in the Workplace: Findings From One Institution. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2021; 96:1197-1204. [PMID: 33464735 DOI: 10.1097/acm.0000000000003921] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
PURPOSE Assessment of the Core Entrustable Professional Activities for Entering Residency (Core EPAs) requires direct observation of learners in the workplace to support entrustment decisions. The purpose of this study was to examine the internal structure validity evidence of the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) scale when used to assess medical student performance in the Core EPAs across clinical clerkships. METHOD During the 2018-2019 academic year, the Virginia Commonwealth University School of Medicine implemented a mobile-friendly, student-initiated workplace-based assessment (WBA) system to provide formative feedback for the Core EPAs across all clinical clerkships. Students were required to request a specified number of Core EPA assessments in each clerkship. A modified O-SCORE scale (1 = "I had to do" to 4 = "I needed to be in room just in case") was used to rate learner performance. Generalizability theory was applied to assess the generalizability (or reliability) of the assessments. Decision studies were then conducted to determine the number of assessments needed to achieve a reasonable reliability. RESULTS A total of 10,680 WBAs were completed on 220 medical students. The majority of ratings were completed on EPA 1 (history and physical) (n = 3,129; 29%) and EPA 6 (oral presentation) (n = 2,830; 26%). Mean scores were similar (3.5-3.6 out of 4) across EPAs. Variance due to the student ranged from 3.5% to 8%, with the majority of the variation due to the rater (29.6%-50.3%) and other unexplained factors. A range of 25 to 63 assessments were required to achieve reasonable reliability (Phi > 0.70). CONCLUSIONS The O-SCORE demonstrated modest reliability when used across clerkships. These findings highlight specific challenges for implementing WBAs for the Core EPAs including the process for requesting WBAs, rater training, and application of the O-SCORE scale in medical student assessment.
Collapse
Affiliation(s)
- Michael S Ryan
- M.S. Ryan is associate professor and assistant dean for clinical medical education, Department of Pediatrics, Virginia Commonwealth University, Richmond, Virginia; ORCID: https://orcid.org/0000-0003-3266-9289
| | - Alicia Richards
- A. Richards is a graduate student, Department of Biostatistics, Virginia Commonwealth University, Richmond, Virginia
| | - Robert Perera
- R. Perera is associate professor, Department of Biostatistics, Virginia Commonwealth University, Richmond, Virginia
| | - Yoon Soo Park
- Y.S. Park is associate professor and associate head, Department of Medical Education, University of Illinois College of Medicine, Chicago, Illinois
| | - J K Stringer
- J.K. Stringer is assessment manager, Office of Integrated Medical Education, Rush Medical College, Chicago, Illinois
| | - Elizabeth Waterhouse
- E. Waterhouse is professor, Department of Neurology, Virginia Commonwealth University, Richmond, Virginia
| | - Brieanne Dubinsky
- B. Dubinsky is business analyst, Office of Academic Information Systems, Virginia Commonwealth University, Richmond, Virginia
| | - Rebecca Khamishon
- R. Khamishon is a third-year medical student, Virginia Commonwealth University, Richmond, Virginia
| | - Sally A Santen
- S.A. Santen is professor and senior associate dean of assessment, evaluation, and scholarship, Department of Emergency Medicine, Virginia Commonwealth University, Richmond, Virginia; ORCID: https://orcid.org/0000-0002-8327-8002
| |
Collapse
|
17
|
Considering Objective and Subjective Measures for Police Use of Force Evaluation. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2021; 18:ijerph18105351. [PMID: 34069786 PMCID: PMC8157287 DOI: 10.3390/ijerph18105351] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 04/19/2021] [Revised: 05/14/2021] [Accepted: 05/16/2021] [Indexed: 11/28/2022]
Abstract
In spite of significant interest in the application of police use of force (UOF) from organisations, researchers, and the general public, there remains no industry standard for how police UOF is trained, and by extension, evaluated. While certain UOF behaviours can be objectively measured (e.g., correct shoot/no shoot decision making (DM), shot accuracy), the subjective evaluation of many UOF skills (e.g., situation awareness, SA) falls to the discretion of individual instructors. The aim of the current brief communication is to consider the operationalisation of essential UOF behaviours as objective and subjective measures, respectively. Using longitudinal data from a sample of Canadian police officers (n = 57) evaluated during UOF training scenarios, we discuss how objective and subjective measures reflect changes in officer performance over time. Objective lethal force DM was measured as a binary ‘correct–incorrect’ outcome and subjective SA was measured on a 5-point Likert scale ranging from ‘unacceptable’ to ‘exceptional’. Subjective evaluation of SA demonstrated significant changes over time, while DM remained relatively high and stable. Given the practical and professional implications of UOF, we recommend that a combination of objective and subjective measures is systematically implemented at all stages of police UOF training and evaluation (i.e., basic, advanced, in-service).
Collapse
|
18
|
Valentine N, Durning S, Shanahan EM, Schuwirth L. Fairness in human judgement in assessment: a hermeneutic literature review and conceptual framework. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2021; 26:713-738. [PMID: 33123837 DOI: 10.1007/s10459-020-10002-1] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/13/2020] [Accepted: 10/19/2020] [Indexed: 06/11/2023]
Abstract
Human judgement is widely used in workplace-based assessment despite criticism that it does not meet standards of objectivity. There is an ongoing push within the literature to better embrace subjective human judgement in assessment not as a 'problem' to be corrected psychometrically but as legitimate perceptions of performance. Taking a step back and changing perspectives to focus on the fundamental underlying value of fairness in assessment may help re-set the traditional objective approach and provide a more relevant way to determine the appropriateness of subjective human judgements. Changing focus to look at what is 'fair' human judgement in assessment, rather than what is 'objective' human judgement in assessment allows for the embracing of many different perspectives, and the legitimising of human judgement in assessment. However, this requires addressing the question: what makes human judgements fair in health professions assessment? This is not a straightforward question with a single unambiguously 'correct' answer. In this hermeneutic literature review we aimed to produce a scholarly knowledge synthesis and understanding of the factors, definitions and key questions associated with fairness in human judgement in assessment and a resulting conceptual framework, with a view to informing ongoing further research. The complex construct of fair human judgement could be conceptualised through values (credibility, fitness for purpose, transparency and defensibility) which are upheld at an individual level by characteristics of fair human judgement (narrative, boundaries, expertise, agility and evidence) and at a systems level by procedures (procedural fairness, documentation, multiple opportunities, multiple assessors, validity evidence) which help translate fairness in human judgement from concepts into practical components.
Collapse
Affiliation(s)
- Nyoli Valentine
- Prideaux Health Professions Education, Flinders University, Bedford Park 5042, SA, Australia.
| | - Steven Durning
- Center for Health Professions Education, Uniformed Services University of the Health Sciences, Bethesda, MD, USA
| | - Ernst Michael Shanahan
- Prideaux Health Professions Education, Flinders University, Bedford Park 5042, SA, Australia
| | - Lambert Schuwirth
- Prideaux Health Professions Education, Flinders University, Bedford Park 5042, SA, Australia
| |
Collapse
|
19
|
Upadhyaya S, Rashid M, Davila-Cervantes A, Oswald A. Exploring resident perceptions of initial competency based medical education implementation. CANADIAN MEDICAL EDUCATION JOURNAL 2021; 12:e42-e56. [PMID: 33995719 PMCID: PMC8105577 DOI: 10.36834/cmej.70943] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
BACKGROUND Competence by design (CBD) is a nationally developed hybrid competency based medical education (CBME) curricular model that focuses on residents' abilities to promote successful practice and better meet societal needs. CBD is based on a commonly used framework of five core components of CBME: outcome competencies, sequenced progression, tailored learning experiences, competency-focused instruction and programmatic assessment. There is limited literature concerning residents' perceptions of implementation of CBME. OBJECTIVE We explored resident perceptions of this transformation and their views as they relate to the intended framework. METHODS We recruited residents enrolled in current CBME implementation between August 2018 and January 2019. We interviewed residents representing eight disciplines from the initial two CBME implementation cohorts. Inductive thematic analysis was used to analyse the data through iterative consensus building until saturation. RESULTS We identified five themes: 1) Value of feedback for residents; 2) Resident strategies for successful Entrustable Professional Activity observation completion; 3) Residents experience challenges; 4) Resident concerns regarding CBME; and 5) Resident recommendations to improve existing challenges. We found that while there was clear alignment with residents' perceptions of the programmatic assessment core CBME component, alignment was not as clear for other components. CONCLUSIONS Residents perceived aspects of this transformation as helpful but overall had mixed perceptions and variable understanding of the intended underlying framework. Understanding and disseminating successes and challenges from the resident lens may assist programs at different stages of CBME implementation.
Collapse
Affiliation(s)
- Shivani Upadhyaya
- Core Internal Medicine Resident, Department of Medicine, PGY3, University of Alberta, Alberta, Canada
| | - Marghalara Rashid
- Department of Pediatrics, Faculty of Medicine & Dentistry, University of Alberta, Alberta, Canada
| | - Andrea Davila-Cervantes
- Office of Lifelong Learning, Faculty of Medicine & Dentistry, University of Alberta, Alberta, Canada
| | - Anna Oswald
- Division of Rheumatology, Department of Medicine, Faculty of Medicine & Dentistry, University of Alberta, Alberta, Canada
| |
Collapse
|
20
|
Malau-Aduli BS, Hays RB, D'Souza K, Smith AM, Jones K, Turner R, Shires L, Smith J, Saad S, Richmond C, Celenza A, Sen Gupta T. Examiners' decision-making processes in observation-based clinical examinations. MEDICAL EDUCATION 2021; 55:344-353. [PMID: 32810334 DOI: 10.1111/medu.14357] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/19/2020] [Revised: 08/08/2020] [Accepted: 08/14/2020] [Indexed: 06/11/2023]
Abstract
BACKGROUND Objective structured clinical examinations (OSCEs) are commonly used to assess the clinical skills of health professional students. Examiner judgement is one acknowledged source of variation in candidate marks. This paper reports an exploration of examiner decision making to better characterise the cognitive processes and workload associated with making judgements of clinical performance in exit-level OSCEs. METHODS Fifty-five examiners for exit-level OSCEs at five Australian medical schools completed a NASA Task Load Index (TLX) measure of cognitive load and participated in focus group interviews immediately after the OSCE session. Discussions focused on how decisions were made for borderline and clear pass candidates. Interviews were transcribed, coded and thematically analysed. NASA TLX results were quantitatively analysed. RESULTS Examiners self-reported higher cognitive workload levels when assessing a borderline candidate in comparison with a clear pass candidate. Further analysis revealed five major themes considered by examiners when marking candidate performance in an OSCE: (a) use of marking criteria as a source of reassurance; (b) difficulty adhering to the marking sheet under certain conditions; (c) demeanour of candidates; (d) patient safety, and (e) calibration using a mental construct of the 'mythical [prototypical] intern'. Examiners demonstrated particularly higher mental demand when assessing borderline compared to clear pass candidates. CONCLUSIONS Examiners demonstrate that judging candidate performance is a complex, cognitively difficult task, particularly when performance is of borderline or lower standard. At programme exit level, examiners intuitively want to rate candidates against a construct of a prototypical graduate when marking criteria appear not to describe both what and how a passing candidate should demonstrate when completing clinical tasks. This construct should be shared, agreed upon and aligned with marking criteria to best guide examiner training and calibration. Achieving this integration may improve the accuracy and consistency of examiner judgements and reduce cognitive workload.
Collapse
Affiliation(s)
- Bunmi S Malau-Aduli
- College of Medicine and Dentistry, James Cook University, Townsville, QLD, Australia
| | - Richard B Hays
- College of Medicine and Dentistry, James Cook University, Townsville, QLD, Australia
| | - Karen D'Souza
- School of Medicine, Deakin University, Geelong, VIC, Australia
| | - Amy M Smith
- College of Medicine and Dentistry, James Cook University, Townsville, QLD, Australia
| | - Karina Jones
- College of Medicine and Dentistry, James Cook University, Townsville, QLD, Australia
| | - Richard Turner
- School of Medicine, University of Tasmania, Hobart, TAS, Australia
| | - Lizzi Shires
- School of Medicine, University of Tasmania, Hobart, TAS, Australia
| | - Jane Smith
- Medical Program, Bond University, Gold Coast, QLD, Australia
| | - Shannon Saad
- School of Medicine, Notre Dame University, Sydney, NSW, Australia
| | | | - Antonio Celenza
- School of Medicine, University of Western Australia, Perth, WA, Australia
| | - Tarun Sen Gupta
- College of Medicine and Dentistry, James Cook University, Townsville, QLD, Australia
| |
Collapse
|
21
|
van der Want AC, Bloemendaal PM, van der Hage JA. Examiners' Perceptions in Surgical Education: The Blind Spot in the Assessment of OSCEs. JOURNAL OF SURGICAL EDUCATION 2021; 78:590-596. [PMID: 32819870 DOI: 10.1016/j.jsurg.2020.07.024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/10/2020] [Revised: 06/22/2020] [Accepted: 07/15/2020] [Indexed: 06/11/2023]
Abstract
PURPOSE OF THE ARTICLE Interpreting objective structured clinical examination (OSCE) scores remains a challenging task.1 It remains unclear how examiners make a decision to score an OSCE the way they do. The examiners' thoughts and decision-making processes remain a hidden part of the OSCE assessment. One cannot assume that all examiners have a shared belief about what is important in an OSCE or what is the aim of an OSCE assessment. These differences are problematic, since they lead to students not getting the 'correct' score for their OSCE. MATERIALS AND METHODS Using a qualitative case study design,2 this study aims to gain in-depth insight into examiners' perceptions of OSCEs and their role as an examiner. For this purpose, semi-structured interviews with OSCE observers (n = 6) were conducted. RESULTS AND CONCLUSIONS The results show that the perception of examiners on the aim of OSCES and their motivation and task as examiner are crucial in understanding the differences among OSCE scores of examiners. Training and discussion among the examiners are needed to decrease the differences among the OSCE scores of examiners.
Collapse
|
22
|
Koedijk M, Renden PG, Oudejans RRD, Kleygrewe L, Hutter RIV. Observational Behavior Assessment for Psychological Competencies in Police Officers: A Proposed Methodology for Instrument Development. Front Psychol 2021; 12:589258. [PMID: 33732178 PMCID: PMC7959728 DOI: 10.3389/fpsyg.2021.589258] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/30/2020] [Accepted: 02/08/2021] [Indexed: 12/14/2022] Open
Abstract
This paper proposes and showcases a methodology to develop an observational behavior assessment instrument to assess psychological competencies of police officers. We outline a step-by-step methodology for police organizations to measure and evaluate behavior in a meaningful way to assess these competencies. We illustrate the proposed methodology with a practical example. We posit that direct behavioral observation can be key in measuring the expression of psychological competence in practice, and that psychological competence in practice is what police organizations should care about. We hope this paper offers police organizations a methodology to perform scientifically informed observational behavior assessment of their police officers’ psychological competencies and inspires additional research efforts into this important area.
Collapse
Affiliation(s)
- Matthijs Koedijk
- Department of Human Movement Sciences, Faculty of Behavioural and Movement Sciences, Vrije Universiteit Amsterdam, Amsterdam, Netherlands.,Institute of Brain and Behavior Amsterdam, Amsterdam, Netherlands
| | - Peter G Renden
- Department of Human Movement Sciences, Faculty of Behavioural and Movement Sciences, Vrije Universiteit Amsterdam, Amsterdam, Netherlands.,Faculty of Health, Nutrition and Sport, The Hague University of Applied Sciences, The Hague, Netherlands
| | - Raôul R D Oudejans
- Department of Human Movement Sciences, Faculty of Behavioural and Movement Sciences, Vrije Universiteit Amsterdam, Amsterdam, Netherlands.,Institute of Brain and Behavior Amsterdam, Amsterdam, Netherlands.,Faculty of Sports and Nutrition, Amsterdam University of Applied Sciences, Amsterdam, Netherlands
| | - Lisanne Kleygrewe
- Department of Human Movement Sciences, Faculty of Behavioural and Movement Sciences, Vrije Universiteit Amsterdam, Amsterdam, Netherlands.,Institute of Brain and Behavior Amsterdam, Amsterdam, Netherlands
| | - R I Vana Hutter
- Department of Human Movement Sciences, Faculty of Behavioural and Movement Sciences, Vrije Universiteit Amsterdam, Amsterdam, Netherlands.,Institute of Brain and Behavior Amsterdam, Amsterdam, Netherlands
| |
Collapse
|
23
|
Baugh RF, Baugh AD. Cultural influences and the Objective Structured Clinical Examination. INTERNATIONAL JOURNAL OF MEDICAL EDUCATION 2021; 12:22-24. [PMID: 33507878 PMCID: PMC7883802 DOI: 10.5116/ijme.5ff9.b817] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/15/2020] [Accepted: 01/09/2021] [Indexed: 06/12/2023]
Affiliation(s)
- Reginald F. Baugh
- Department of Surgery, University of Toledo College of Medicine and Life Sciences, Toledo, OH, USA
| | - Aaron D. Baugh
- Pulmonary, Critical Care, Allergy, Sleep Medicine, Department of Internal Medicine University of California San Francis-co Medical School, University of California San Francisco Medical Center, San Francisco, CA, USA
| |
Collapse
|
24
|
Wilby KJ, Paravattil B. Cognitive load theory: Implications for assessment in pharmacy education. Res Social Adm Pharm 2020; 17:1645-1649. [PMID: 33358136 DOI: 10.1016/j.sapharm.2020.12.009] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/15/2019] [Revised: 11/09/2020] [Accepted: 12/15/2020] [Indexed: 11/28/2022]
Abstract
The concept of mental workload is well studied from a learner's perspective but has yet to be better understood from the perspective of an assessor. Mental workload is largely associated with cognitive load theory, which describes three different types of load. Intrinsic load deals with the complexity of the task, extraneous load describes distractors to the task at hand, and germane load focuses on the development of schemas in working memory for future recall. Studies from medical education show that all three types of load are relevant when considering rater -based assessment (e.g. Objective Structured Clinical Examinations (OSCEs), or experiential training). Assessments with high intrinsic and extraneous load may interfere with assessors' attention and working memory and result in poorer quality assessment. Reducing these loads within assessment tasks should therefore be a priority for pharmacy educators. This commentary aims to provide a theoretical overview of mental workload in assessment, outline research findings from the medical education context, and propose strategies to be considered for reducing mental workload in rater-based assessments relevant to pharmacy education. Suggestions for future research are also addressed.
Collapse
Affiliation(s)
- Kyle John Wilby
- School of Pharmacy, University of Otago, PO Box 56, Dunedin, 9054, New Zealand.
| | | |
Collapse
|
25
|
Schuwirth LWT, van der Vleuten CPM. A history of assessment in medical education. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2020; 25:1045-1056. [PMID: 33113056 DOI: 10.1007/s10459-020-10003-0] [Citation(s) in RCA: 35] [Impact Index Per Article: 8.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/28/2020] [Accepted: 10/19/2020] [Indexed: 06/11/2023]
Abstract
The way quality of assessment has been perceived and assured has changed considerably in the recent 5 decades. Originally, assessment was mainly seen as a measurement problem with the aim to tell people apart, the competent from the not competent. Logically, reproducibility or reliability and construct validity were seen as necessary and sufficient for assessment quality and the role of human judgement was minimised. Later, assessment moved back into the authentic workplace with various workplace-based assessment (WBA) methods. Although originally approached from the same measurement framework, WBA and other assessments gradually became assessment processes that included or embraced human judgement but based on good support and assessment expertise. Currently, assessment is treated as a whole system problem in which competence is evaluated from an integrated rather than a reductionist perspective. Current research therefore focuses on how to support and improve human judgement, how to triangulate assessment information meaningfully and how to construct fairness, credibility and defensibility from a systems perspective. But, given the rapid changes in society, education and healthcare, yet another evolution in our thinking about good assessment is likely to lurk around the corner.
Collapse
Affiliation(s)
- Lambert W T Schuwirth
- FHMRI: Prideaux Research in Health Professions Education, College of Medicine and Public Health, Flinders University, Sturt Road, Bedford Park, South Australia, 5042, GPO Box 2100, Adelaide, SA, 5001, Australia.
- Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands.
| | - Cees P M van der Vleuten
- FHMRI: Prideaux Research in Health Professions Education, College of Medicine and Public Health, Flinders University, Sturt Road, Bedford Park, South Australia, 5042, GPO Box 2100, Adelaide, SA, 5001, Australia
- Department of Educational Development and Research, Maastricht University, Maastricht, The Netherlands
| |
Collapse
|
26
|
McEllistrem B, Barrett A, Hanley K. Performance in practice; exploring trainer and trainee experiences of user-designed formative assessment tools. EDUCATION FOR PRIMARY CARE 2020; 32:27-33. [PMID: 33094687 DOI: 10.1080/14739879.2020.1815085] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/23/2022]
Abstract
INTRODUCTION General Practice training in Ireland currently has various methods of formative assessment and feedback delivered to trainees. In 2018 the Irish College of General Practitioners commissioned the generation of two new user-designed formative feedback tools that would allow trainee feedback to drive learning. These tools became known as the Performance in Practice (PiP) tools. AIMS To explore the experiences of General Practice (GP) trainers and trainees having completed a pilot of using the PiP tools for 4 months. METHODS An explorative phenomenological approach was taken to understand the experiences of trainers and trainees. One to one interviews were conducted, and the transcripts analysed for themes and sub-theme via Template analysis. RESULTS User experiences focused on two main areas; educational value and acceptability. In relation to educational value, the PiP tools were seen as an improvement over established forms of formative feedback, as they were centred around the curriculum and therefore reflected the unique multifaceted requirements of an independently practising GP. Acceptability primarily focused around data governance and structures, as well as practical issues such as ease of software use. CONCLUSIONS Overall, the experience of using the PiP tools was positive for both trainers and trainees. Future plans to further explore implementation of the PiP tools have been significantly informed by this research.
Collapse
Affiliation(s)
- B McEllistrem
- General Practice Training Unit, Irish College of General Practitioners, Dublin, Ireland
| | - A Barrett
- General Practice Training Unit, Irish College of General Practitioners, Dublin, Ireland
| | - K Hanley
- General Practice Training Unit, Irish College of General Practitioners, Dublin, Ireland
| |
Collapse
|
27
|
St-Onge C, Vachon Lachiver É, Langevin S, Boileau E, Bernier F, Thomas A. Lessons from the implementation of developmental progress assessment: A scoping review. MEDICAL EDUCATION 2020; 54:878-887. [PMID: 32083743 DOI: 10.1111/medu.14136] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/28/2019] [Revised: 01/21/2020] [Accepted: 02/06/2020] [Indexed: 06/10/2023]
Abstract
OBJECTIVES Educators and researchers recently implemented developmental progress assessment (DPA) in the context of competency-based education. To reap its anticipated benefits, much still remains to be understood about its implementation. In this study, we aimed to determine the nature and extent of the current evidence on DPA, in an effort to broaden our understanding of the major goals and intended outcomes of DPA as well as the lessons learned from how it has been executed in, or applied across, educational contexts. METHODS We conducted a scoping study based on the methodology of Arksey and O'Malley. Our search strategy yielded 2494 articles. These articles were screened for inclusion and exclusion (90% agreement), and numerical and qualitative data were extracted from 56 articles based on a pre-defined set of charting categories. The thematic analysis of the qualitative data was completed with iterative consultations and discussions until consensus was achieved for the interpretation of the results. RESULTS Tools used to document DPA include scales, milestones and portfolios. Performances were observed in clinical or standardised contexts. We identified seven major themes in our qualitative thematic analysis: (a) underlying aims of DPA; (b) sources of information; (c) barriers; (d) contextual factors that can act as barriers or facilitators to the implementation of DPA; (e) facilitators; (f) observed outcomes, and (g) documented validity evidences. CONCLUSIONS Developmental progress assessment seems to fill a need in the training of future competent health professionals. However, moving forward with a widespread implementation of DPA, factors such as lack of access to user-friendly technology and time to observe performance may render its operationalisation burdensome in the context of competency-based medical education.
Collapse
Affiliation(s)
- Christina St-Onge
- Department of Medicine, Faculty of Medicine and Health Sciences, University of Sherbrooke, Sherbrooke, Québec, Canada
| | - Élise Vachon Lachiver
- Faculty of Medicine and Health Sciences, University of Sherbrooke, Sherbrooke, Québec, Canada
| | - Serge Langevin
- Department of Medicine, Faculty of Medicine and Health Sciences, University of Sherbrooke, Sherbrooke, Québec, Canada
| | - Elisabeth Boileau
- Department of Family and Emergency Medicine, Faculty of Medicine and Health Sciences, University of Sherbrooke, Sherbrooke, Québec, Canada
| | - Frédéric Bernier
- Department of Medicine, Faculty of Medicine and Health Sciences, University of Sherbrooke, Sherbrooke, Québec, Canada
- Faculty of Medicine and Health Sciences, University of Sherbrooke, Sherbrooke, Québec, Canada
- Research Center - Sherbrooke University Hospital Center (CHUS), Integrated Health and Social Service Centers (CISSS) and Integrated University Health and Social Service Centres (CIUSSS), Sherbrooke, Québec, Canada
| | - Aliki Thomas
- School of Physical and Occupational Therapy, McGill University, Montreal, Québec, Canada
| |
Collapse
|
28
|
Bramley AL, Thomas CJ, Mc Kenna L, Itsiopoulos C. E-portfolios and Entrustable Professional Activities to support competency-based education in dietetics. Nurs Health Sci 2020; 23:148-156. [PMID: 32896047 DOI: 10.1111/nhs.12774] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/13/2020] [Revised: 08/31/2020] [Accepted: 09/02/2020] [Indexed: 11/27/2022]
Abstract
The concept of Entrustable Professional Activities, recently pioneered in medical education, has emerged to support the implementation of competency-based education. Although competency-based frameworks are widely used in healthcare professional education to develop outcomes-based curricula, assessment of student competency in professional placement settings remains challenging. The novel concept of Entrustable Professional Activities together with established methods of competency assessment, namely e-portfolios and self-assessment, was implemented in the "[La Trobe University Dietetic program in 2015-2016. This study aimed to appraise the e-portfolio and evaluate the use of Entrustable Professional Activities to assess competence. A mixed-methods evaluation, using qualitative and quantitative surveys with follow-up structured consultations, was conducted with final year dietetics students and their supervisors. Dietetics students were comfortable with Entrustable Professional Activities and competency-based assessment, whereas supervisors preferred Entrustable Professional Activity based assessment. All stakeholders valued student self-assessment and the ongoing use of structured e-portfolios to develop and document competency. The use of structured e-portfolios, student self-assessment, and the emerging concept of Entrustable Professional Activities are useful tools to support dietetics student education in professional placement settings.
Collapse
Affiliation(s)
- Andrea Louise Bramley
- Senior Lecturer in the Department of Dietetics and Human Nutrition, School of Allied Health and Human Services, College of Science, Health and Engineering, La Trobe University, Melbourne, Victoria, Australia
| | - Colleen J Thomas
- Associate Professor (Physiology) in the Department of Physiology, Anatomy and Microbiology, School of Life Sciences, College of Science, Health and Engineering, La Trobe University, Melbourne, Victoria, Australia
| | - Lisa Mc Kenna
- Head of School of Nursing and Midwifery, College of Science, Health and Engineering, La Trobe University, Melbourne, Victoria, Australia
| | - Catherine Itsiopoulos
- Pro Vice Chancellor and Executive Dean College of Science, Health Engineering and Education Murdoch University and an Adjunct Professor of Dietetics, La Trobe University, Melbourne, Victoria, Australia
| |
Collapse
|
29
|
Schuwirth LWT, Durning SJ, King SM. Assessment of clinical reasoning: three evolutions of thought. Diagnosis (Berl) 2020; 7:191-196. [PMID: 32182208 DOI: 10.1515/dx-2019-0096] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/01/2019] [Accepted: 02/12/2020] [Indexed: 02/17/2024]
Abstract
Although assessing clinical reasoning is almost universally considered central to medical education it is not a straightforward issue. In the past decades, our insights into clinical reasoning as a phenomenon, and consequently the best ways to assess it, have undergone significant changes. In this article, we describe how the interplay between fundamental research, practical applications, and evaluative research has pushed the evolution of our thinking and our practices in assessing clinical reasoning.
Collapse
Affiliation(s)
- Lambert W T Schuwirth
- Prideaux Centre for Research in Health Professions Education, Flinders University, Adelaide, South Australia, Australia
| | | | - Svetlana M King
- Prideaux Centre for Research in Health Professions Education, Flinders University, Adelaide, South Australia, Australia
| |
Collapse
|
30
|
Wong WYA, Roberts C, Thistlethwaite J. Impact of Structured Feedback on Examiner Judgements in Objective Structured Clinical Examinations (OSCEs) Using Generalisability Theory. HEALTH PROFESSIONS EDUCATION 2020. [DOI: 10.1016/j.hpe.2020.02.005] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022] Open
|
31
|
Chimea TL, Kanji Z, Schmitz S. Assessment of clinical competence in competency-based education. CANADIAN JOURNAL OF DENTAL HYGIENE : CJDH = JOURNAL CANADIEN DE L'HYGIENE DENTAIRE : JCHD 2020; 54:83-91. [PMID: 33240368 PMCID: PMC7668267] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Subscribe] [Scholar Register] [Received: 12/13/2019] [Revised: 05/04/2020] [Accepted: 05/12/2020] [Indexed: 06/11/2023]
Abstract
Objective The purpose of this review is to explore the literature on continuous assessment in the evaluation of clinical competence, to examine the variables influencing the assessment of clinical competence, and to consider the impact of high-stakes summative assessment practices on student experiences, learning, and achievement. Methods A literature search of CINAHL, PubMed, ERIC (EBSCO), Education Source, and Google Scholar was conducted using key terms. Articles reviewed were limited to full-text, peer-reviewed articles published in English from 2000 to 2019. Selected articles for this review include a meta-analysis, systematic reviews, and studies using qualitative and quantitative designs. Results Findings reveal that current assessment practices such as one-time high-stakes assessments in the evaluation of clinical competence are influenced by several variables: interexaminer differences in evaluation, variability with non-standardized client use in assessment, the failure to fail, and the impact of stress on performance outcomes. This literature review also highlights a programmatic assessment approach in which student competence is determined by a multitude of low-stakes assessments over time. Conclusion A review of the literature has highlighted current methods of clinical assessment relying on traditional, summative forms of evaluation, with reliability and validity of the assessment influenced by several variables. Emotions and student experiences related to one-time high-stakes summative assessments may negatively affect student learning and achievement outcomes. The design, implementation, and use of assessment practices within a competency-based education framework warrants further consideration so that optimal assessment for learning practices may be emphasized to enhance student learning and achievement.
Collapse
Affiliation(s)
- Teresa La Chimea
- Dental Hygiene Degree Program, Faculty of Dentistry, University of British Columbia, Vancouver, BC, Canada
| | - Zul Kanji
- Dental Hygiene Degree Program, Faculty of Dentistry, University of British Columbia, Vancouver, BC, Canada
| | - Susan Schmitz
- Dental Hygiene Degree Program, Faculty of Dentistry, University of British Columbia, Vancouver, BC, Canada
| |
Collapse
|
32
|
Barry ES, Dong T, Durning SJ, Schreiber-Gregory D, Torre D, Grunberg NE. Faculty Assessments in a Military Medical Field Practicum: Rater Experience and Gender Do Not Appear to Influence Scoring. Mil Med 2020; 185:e358-e363. [PMID: 31925445 DOI: 10.1093/milmed/usz364] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/13/2022] Open
Abstract
INTRODUCTION Any implicit and explicit biases that exist may alter our interpretation of people and events. Within the context of assessment, it is important to determine if biases exist and to decrease any existing biases, especially when rating student performance to provide meaningful, fair, and useful input. The purpose of this study was to determine if the experience and gender of faculty members contribute to their ratings of students in a military medical field practicum. This information is important for fair ratings of students. Three research questions were addressed: Were there differences between new versus experienced faculty raters? Were there differences in assessments provided by female and male faculty members? Did gender of faculty raters impact ratings of female and male students?. MATERIALS AND METHODS This study examined trained faculty evaluators' ratings of three cohorts of medical students during 2015-2017 during a medical field practicum. Female (n = 80) and male (n = 161) faculty and female (n = 158) and male (n = 311) students were included. Within this dataset, there were 469 students and 241 faculty resulting in 5,599 ratings for each of six outcome variables that relate to overall leader performance, leader competence, and leader communication. Descriptive statistics were computed for all variables for the first four observations of each student. Descriptive analyses were performed for evaluator experience status and gender differences by each of the six variables. A multivariate analyses of variance was performed to examine whether there were differences between gender of faculty and gender of students. RESULTS Descriptive analyses of the experience status of faculty revealed no significant differences between means on any of the rating elements. Descriptive analyses of faculty gender revealed no significant differences between female and male faculty ratings of the students. The overall MANOVA analyses found no statistically significant difference between female and male students on the combined dependent variables of leader performance for any of the four observations. CONCLUSIONS The study revealed that there were no differences in ratings of student leader performance based on faculty experience. In addition, there were no differences in ratings of student leader performance based on faculty gender.
Collapse
Affiliation(s)
- Erin S Barry
- Department of Military & Emergency Medicine, F. Edward Hébert School of Medicine, Uniformed Services University, 4301 Jones Bridge Rd, Bethesda, MD 20814
| | - Ting Dong
- Department of Medicine, F. Edward Hébert School of Medicine, Uniformed Services University, 4301 Jones Bridge Rd, Bethesda, MD 20814
| | - Steven J Durning
- Department of Medicine, F. Edward Hébert School of Medicine, Uniformed Services University, 4301 Jones Bridge Rd, Bethesda, MD 20814
| | - Deanna Schreiber-Gregory
- Department of Medicine, F. Edward Hébert School of Medicine, Uniformed Services University, 4301 Jones Bridge Rd, Bethesda, MD 20814
| | - Dario Torre
- Department of Medicine, F. Edward Hébert School of Medicine, Uniformed Services University, 4301 Jones Bridge Rd, Bethesda, MD 20814
| | - Neil E Grunberg
- Department of Military & Emergency Medicine, F. Edward Hébert School of Medicine, Uniformed Services University, 4301 Jones Bridge Rd, Bethesda, MD 20814
| |
Collapse
|
33
|
Mitchell C, Teti SL. Ensuring Certified Healthcare Ethics Consultants Are Competent to Practice. THE AMERICAN JOURNAL OF BIOETHICS : AJOB 2020; 20:24-27. [PMID: 32116169 DOI: 10.1080/15265161.2020.1714818] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
|
34
|
Paravattil B, Wilby KJ. Optimizing assessors' mental workload in rater-based assessment: a critical narrative review. PERSPECTIVES ON MEDICAL EDUCATION 2019; 8:339-345. [PMID: 31728841 PMCID: PMC6904389 DOI: 10.1007/s40037-019-00535-6] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
INTRODUCTION Rater-based assessment has resulted in high cognitive demands for assessors within the education of health professionals. Rating quality may be influenced by the mental workload required of assessors to complete rating tasks. The objective of this review was to explore interventions or strategies aimed at measuring and reducing mental workload for improvement in assessment outcomes in health professions education. METHODS A critical narrative review was conducted for English-language articles using the databases PubMed, EMBASE, and Google Scholar from conception until November 2018. To be included, articles were eligible if they reported results of interventions aimed at measuring or reducing mental workload in rater-based assessment. RESULTS A total of six articles were included in the review. All studies were conducted in simulation settings (OSCEs or videotaped interactions). Of the four studies that measured mental workload, none found any reduction in mental workload as demonstrated by objective secondary task performance after interventions of assessor training or reductions in competency dimension assessment. Reductions in competency dimensions, however, did result in improvements in assessment quality across three studies. DISCUSSION The concept of mental workload in assessment in medical education needs further exploration, including investigation into valid measures of assessors' mental workload. It appears that adjusting raters' focus may be a valid strategy to improve assessment outcomes. Future research should be designed to inform how to best reduce load in assessments to improve quality, while balancing the type and quantity of data needed for judgments.
Collapse
Affiliation(s)
| | - Kyle John Wilby
- School of Pharmacy, University of Otago, Dunedin, New Zealand.
| |
Collapse
|
35
|
Hodwitz K, Kuper A, Brydges R. Realizing One's Own Subjectivity: Assessors' Perceptions of the Influence of Training on Their Conduct of Workplace-Based Assessments. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2019; 94:1970-1979. [PMID: 31397710 DOI: 10.1097/acm.0000000000002943] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
PURPOSE Assessor training is essential for defensible assessments of physician performance, yet research on the effectiveness of training programs for promoting assessor consistency has produced mixed results. This study explored assessors' perceptions of the influence of training and assessment tools on their conduct of workplace-based assessments of physicians. METHOD In 2017, the authors used a constructivist grounded theory approach to interview 13 physician assessors about their perceptions of the effects of training and tool development on their conduct of assessments. RESULTS Participants reported that training led them to realize that there is a potential for variability in assessors' judgments, prompting them to change their scoring and feedback behaviors to enhance consistency. However, many participants noted they had not substantially changed their numerical scoring. Nonetheless, most thought training would lead to increased standardization and consistency among assessors, highlighting a "standardization paradox" in which participants perceived a programmatic shift toward standardization but minimal changes in their own ratings. An "engagement effect" was also found in which participants involved in both tool development and training cited more substantial learnings than participants involved only in training. CONCLUSIONS Findings suggest that training may help assessors recognize their own subjectivity when judging performance, which may prompt behaviors that support rigorous and consistent scoring but may not lead to perceptible changes in assessors' numeric ratings. Results also suggest that participating in tool development may help assessors align their judgments with the scoring criteria. Overall, results support the continued study of assessor training programs as a means of enhancing assessor consistency.
Collapse
Affiliation(s)
- Kathryn Hodwitz
- K. Hodwitz is research associate, College of Physicians and Surgeons of Ontario, Toronto, Ontario, Canada. A. Kuper is associate professor and faculty co-lead, Person-Centred Care Education, Department of Medicine, scientist and associate director, Wilson Centre for Research in Education, University Health Network, University of Toronto, and staff physician, Division of General Internal Medicine, Sunnybrook Health Sciences Centre, Toronto, Ontario, Canada. R. Brydges is research director and scientist and holds the professorship in Technology Enabled Education at the Allan Waters Family Simulation Centre, St. Michael's Hospital, and is associate professor, Department of Medicine and Wilson Centre for Research in Education, University of Toronto, Toronto, Ontario, Canada
| | | | | |
Collapse
|
36
|
Wilby KJ, Dolmans DHJM, Austin Z, Govaerts MJB. Assessors' interpretations of narrative data on communication skills in a summative OSCE. MEDICAL EDUCATION 2019; 53:1003-1012. [PMID: 31304615 DOI: 10.1111/medu.13924] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/30/2019] [Revised: 03/08/2019] [Accepted: 05/29/2019] [Indexed: 06/10/2023]
Abstract
OBJECTIVES Increasingly, narrative assessment data are used to substantiate and enhance the robustness of assessor judgements. However, the interpretation of written assessment comments is inherently complex and relies on human (expert) judgements. The purpose of this study was to explore how expert assessors process and construe or bring meaning to narrative data when interpreting narrative assessment comments written by others in the setting of standardised performance assessment. METHODS Narrative assessment comments on student communication skills and communication scores across six objective structured clinical examination stations were obtained for 24 final-year pharmacy students. Aggregated narrative data across all stations were sampled for nine students (three good, three average and three poor performers, based on communication scores). A total of 10 expert assessors reviewed the aggregated set of narrative comments for each student. Cognitive (information) processing was captured through think-aloud procedures and verbal protocol analysis. RESULTS Expert assessors primarily made use of two strategies to interpret the narratives, namely comparing and contrasting, and forming mental images of student performance. Assessors appeared to use three different perspectives when interpreting narrative comments, including those of: (i) the student (placing him- or herself in the shoes of the student); (ii) the examiner (adopting the role of examiner and reinterpreting comments according to his or her own standards or beliefs), and (iii) the professional (acting as the profession's gatekeeper by considering the assessment to be a representation of real-life practice). CONCLUSIONS The present findings add to current understandings of assessors' interpretations of narrative performance data by identifying the strategies and different perspectives used by expert assessors to frame and bring meaning to written comments. Assessors' perspectives affect assessors' interpretations of assessment comments and are likely to be influenced by their beliefs, interpretations of the assessment setting and personal performance theories. These results call for the use of multiple assessors to account for variations in assessor perspectives in the interpretation of narrative assessment data.
Collapse
Affiliation(s)
- Kyle John Wilby
- School of Pharmacy, University of Otago, Dunedin, New Zealand
| | - Diana H J M Dolmans
- School of Health Professions Education (SHE), Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands
| | - Zubin Austin
- Leslie Dan Faculty of Pharmacy, University of Toronto, Toronto, Ontario, Canada
| | - Marjan J B Govaerts
- School of Health Professions Education (SHE), Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands
| |
Collapse
|
37
|
Colbert-Getz JM, Lappe K, Northrup M, Roussel D. To What Degree Are the 13 Entrustable Professional Activities Already Incorporated Into Physicians' Performance Schemas for Medical Students? TEACHING AND LEARNING IN MEDICINE 2019; 31:361-369. [PMID: 30873878 DOI: 10.1080/10401334.2019.1573146] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/09/2023]
Abstract
Phenomenon: There is an abundance of literature on Entrustable Professional Activities (EPAs) in theory, but there are few studies on the EPAs in practice for undergraduate clinical education. In addition, little is known about the degree to which the EPAs are or are not aligned with physician assessors' performance schemas of the clerkship student. Investigating the degree to which physician assessors' performance schemas are already aligned with the activities described by the EPAs is critical for effective workplace assessment design. Approach: We sampled 1,032 areas of strength (strength) and areas for improvement (improvement) written evaluation comments by 423 physician assessors for clerkship students' performance in academic years 2014-15 and 2015-16 at the University of Utah School of Medicine. Two researchers independently categorized each comment by EPA and/or coded by non-EPA topic. The proportion of comment types was compared between strength comments and improvement comments with the Wilcoxon Signed-Rank Test. Findings: The most frequently mentioned EPAs in comments were about history gathering/physical exam, differential diagnosis, documentation, presentation, and interprofessional collaboration; few mentioned diagnostic tests, patient handovers, recognition of urgent patient care, and patient safety, and none mentioned orders/prescriptions and informed consent. The most frequent non-EPA topics were about medical knowledge, need to read more, learning attitude, work ethic, professionalism/maturity, and receptiveness to feedback. The proportion of comments aligned with an EPA only, a non-EPA topic only, or both an EPA and non-EPA topic was significantly different for clerkship students' strength compared to improvement. Insights: Physician assessors' performance schemas for clerkship students were aligned with EPAs to varying degrees depending on the specific EPA and whether describing strength or improvement. Of interest, the frequently mentioned non-EPA comments represented some of the competencies that contribute to effectively performing particular EPAs and are Accreditation Council for Graduate Medical Education (ACGME) core competencies (e.g., medical knowledge, professionalism), used in residency programs. Because physician assessors for undergraduate medical education often also participate in graduate medical education, the frequency of non-EPA topics aligned to ACGME competencies may suggest influence of graduate medical education evaluative frameworks on performance schemas for clerkship students; this could be important when considering implementation of EPAs in undergraduate medical education.
Collapse
Affiliation(s)
- Jorie M Colbert-Getz
- a Department of Internal Medicine, University of Utah School of Medicine , Salt Lake City , Utah , USA
| | - Katie Lappe
- b Department of Internal Medicine, University of Utah School of Medicine & George E. Whalen VA Hospital , Salt Lake City , Utah , USA
| | | | - Danielle Roussel
- d Department of Anesthesiology, University of Utah School of Medicine , Salt Lake City , Utah , USA
| |
Collapse
|
38
|
HangFu L, Nauhria S. The Need for Standardization, Reliability and Validity in Fundamental Roots for a Successful Problem-Based Learning Program. MEDEDPUBLISH 2019; 8:93. [PMID: 38089309 PMCID: PMC10712582 DOI: 10.15694/mep.2019.000093.1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/06/2024] Open
Abstract
This article was migrated. The article was marked as recommended. McMaster University has introduced the Problem-Based Learning (PBL) in medical education over half a century ago. Since then, hundreds of reviews and study reports have identified many critical issues affecting the success of the PBL unit. Nonetheless, we are still debating the efficacy and success of the PBL program. Over half of all medical schools globally have introduced various versions of PBL pedagogy in their medical education program achieving assorted modifications of outcomes. In this paper, I have reviewed from many scholars and through their publications; I have identified eight important Fundamental Roots for a successful PBL program. The success of any PBL program must be evaluated as a whole from the perspective of Standardization, Reliability and Validity working synchronously between these fundamental roots and not the individual PBL unit. The importance of Standardization must consider all the critical issues identified in many reviews and study reports. These issues are one of many factors when incorporated into the fundamental framework of a PBL program will regulate all PBL units in a unified outcome. The educational objective principles will guide the reliability of a PBL program by meeting the institutional mission and students' career success goals. The "Assessment as Learning" should incorporate the "Holistic and Divergent Approach" and the longitudinal "Progress Testing". These are the principle methods of evaluation to achieve a reliable and valid outcome assessment in a successful PBL program.
Collapse
|
39
|
Valentine N, Schuwirth L. Identifying the narrative used by educators in articulating judgement of performance. PERSPECTIVES ON MEDICAL EDUCATION 2019; 8:83-89. [PMID: 30915715 PMCID: PMC6468036 DOI: 10.1007/s40037-019-0500-y] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/15/2023]
Abstract
INTRODUCTION Modern assessment in medical education is increasingly reliant on human judgement, as it is clear that quantitative scales have limitations in fully assessing registrars' development of competence and providing them with meaningful feedback to assist learning. For this, possession of an expert vocabulary is essential. AIM This study aims to explore how medical education experts voice their subjective judgements about learners and to what extent they are using clear, information-rich terminology (high-level semantic qualifiers); and to gain a better understanding of the experts' language used in these subjective judgements. METHODS Six experienced medical educators from urban and rural environments were purposefully selected. Each educator reviewed a registrar clinical case analysis in a think out loud manner. The transcribed data were analyzed, codes were identified and ordered into themes. Analysis continued until saturation was reached. RESULTS Five themes with subthemes emerged. The main themes were: (1) Demonstration of expertise; (2) Personal credibility; (3) Professional credibility; (4) Using a predefined structure and (5) Relevance. DISCUSSION Analogous to what experienced clinicians do in clinical reasoning, experienced medical educators verbalize their judgements using high-level semantic qualifiers. In this study, we were able to unpack these. Although there may be individual variability in the exact words used, clear themes emerged. These findings can be used to develop a helpful shared narrative for educators in observation-based assessment. The provision of a rich, detailed narrative will also assist in providing clarity to registrar feedback with areas of weakness clearly articulated to improve learning and remediation.
Collapse
|
40
|
Transition to practice: Evaluating the need for formal training in supervision and assessment among senior emergency medicine residents and new to practice emergency physicians. CAN J EMERG MED 2019; 21:418-426. [PMID: 30841941 DOI: 10.1017/cem.2019.8] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
OBJECTIVES Emergency medicine residents may be transitioning to practice with minimal training on how to supervise and assess trainees. Our study sought to examine: 1) physician comfort with supervision and assessment, 2) what the current training gaps are within these competencies, and 3) what barriers or enablers might exist in implementing curricular improvements. METHODS Qualitative data were collected in two phases through individual interviews from September 2016 to November 2017, at the University of Toronto and McMaster University after receiving ethics approval from both sites. Eligible participants were final year emergency medicine residents, residents pursuing an enhanced skills program in emergency medicine, and attendings within their first 3 years of practice. A semi-structured interview guide was developed and refined after phase one, to reflect content identified in the first set of interviews. All interviews were recorded, transcribed, coded, and collapsed into themes. Data analysis was guided by constructivist grounded theory. RESULTS A thematic analysis revealed five themes: 1) Supervision and assessment skills were acquired passively through modelling, 2) the training available in these areas is variably used, creating a diversity of comfort levels, 3) competing priorities in the emergency department represent significant barriers to improving supervision and assessment; 4) providing negative feedback is difficult and often avoided; and 5) competence by design will act as an impetus for formal curriculum development in these areas. CONCLUSIONS As programs transition to competence by design, there will be a need for formal training in supervision and assessment, with a focus on negative feedback, to achieve a standardized level of competence among emergency physicians.
Collapse
|
41
|
Lee V, Brain K, Martin J. From opening the 'black box' to looking behind the curtain: cognition and context in assessor-based judgements. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2019; 24:85-102. [PMID: 30302670 DOI: 10.1007/s10459-018-9851-0] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/25/2018] [Accepted: 09/06/2018] [Indexed: 06/08/2023]
Abstract
The increasing use of direct observation tools to assess routine performance has resulted in the growing reliance on assessor-based judgements in the workplace. However, we have a limited understanding of how assessors make judgements and formulate ratings in real world contexts. The current research on assessor cognition has largely focused on the cognitive domain but the contextual factors are equally important, and both are closely interconnected. This study aimed to explore the perceived cognitive and contextual factors influencing Mini-CEX assessor judgements in the Emergency Department setting. We used a conceptual framework of assessor-based judgement to develop a sequential mixed methods study. We analysed and integrated survey and focus group results to illustrate self-reported cognitive and contextual factors influencing assessor judgements. We used situated cognition theory as a sensitizing lens to explore the interactions between people and their environment. The major factors highlighted through our mixed methods study were: clarity of the assessment, reliance on and variable approach to overall impression (gestalt), role tension especially when giving constructive feedback, prior knowledge of the trainee and case complexity. We identified prevailing tensions between participants (assessors and trainees), interactions (assessment and feedback) and setting. The two practical implications of our research are the need to broaden assessor training to incorporate both cognitive and contextual domains, and the need to develop a more holistic understanding of assessor-based judgements in real world contexts to better inform future research and development in workplace-based assessments.
Collapse
Affiliation(s)
- Victor Lee
- Department of Emergency Medicine, Austin Health, P.O. Box 5555, Heidelberg, VIC, 3084, Australia.
| | | | - Jenepher Martin
- Eastern Health Clinical School, Monash University and Deakin University, Box Hill, VIC, Australia
| |
Collapse
|
42
|
Exploring Faculty Approaches to Feedback in the Simulated Setting: Are They Evidence Informed? Simul Healthc 2018; 13:195-200. [PMID: 29381589 DOI: 10.1097/sih.0000000000000289] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/27/2022]
Abstract
INTRODUCTION Feedback in clinical education and after simulated experiences facilitates learning. Although evidence-based guidelines for feedback exist, faculty experience challenges in applying the guidelines. We set out to explore how faculty approach feedback and how these approaches align with current recommendations. METHODS There is strong evidence for the following four components of feedback: feedback as a social interaction, tailoring content, providing specific descriptions of performance, and identifying actionable items. Faculty preceptors participated in feedback simulations followed by debriefing. The simulations were video recorded, transcribed, and analyzed qualitatively using template analysis to examine faculty approaches to feedback relative to evidence-informed recommendations. RESULTS Recorded encounters involving 18 faculty and 11 facilitators yielded 111 videos. There was variability in the extent to which feedback approaches aligned with recommended practices. Faculty behaviors aligned with recommendations included a conversational approach, flexibly adapting feedback techniques to resident context, offering rich descriptions of observations with specific examples and concrete suggestions, achieving a shared understanding of strengths and gaps early on to allow sufficient time for problem-solving, and establishing a plan for ongoing development. Behaviors misaligned with guidelines included prioritizing the task of feedback over the relationship, lack of flexibility in techniques applied, using generic questions that did not explore residents' experiences, and ending with a vague plan for improvement. CONCLUSIONS Faculty demonstrate variability in feedback skills in relation to recommended practices. Simulated feedback experiences may offer a safe environment for faculty to further develop the skills needed to help residents progress within competency-based medical education.
Collapse
|
43
|
Porter J, Beck E, Gallegos D, Palermo C, Walton K, Yaxley A, Volders E, Wray A, Hannan-Jones M. Moderation of a foodservice assessment artefact in nutrition and dietetics programs. Nutr Diet 2018; 76:233-239. [PMID: 30338907 DOI: 10.1111/1747-0080.12484] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/19/2018] [Revised: 09/07/2018] [Accepted: 09/12/2018] [Indexed: 11/27/2022]
Abstract
AIM Foodservice is a key component of dietetics education and practice internationally yet benchmarks for competency are limited. This study sought to review and moderate an assessment artefact of foodservice work integrated learning (WIL) to develop a shared understanding of one tool which may be used in a suite of evidence to demonstrate competence. METHODS The foodservice curricula and assessment artefacts were described for the foodservice program at each of four participating universities. An assessment artefact from WIL, the report, was identified as an indicator of foodservice competence common to each program. Each university provided four purposively sampled WIL reports, assessed in duplicate by two academics from other participating universities using the corresponding university assessment rubric. Collated assessment results, along with the original assessment, were presented back to assessors. A semi-structured group discussion explored variations in assessment results, factors influencing decisions, and potential changes needed for assessment documentation. RESULTS There was variation in assessment outcomes between independent assessors. In some instances assessors did not consistently deliver the same assessment outcome, nor rank students in sequential order of performance. This variation was less where an absolute ranking of satisfactory/unsatisfactory was applied. The assessor discussion revealed three key concepts: importance of understanding the project scope; challenges which influence assessment decision making; importance of understanding the broader program of assessment. CONCLUSIONS Assessment inconsistencies emphasise the importance of multiple assessors and assessment artefacts across a programmatic assessment model, and the need for a clear understanding of competence in nutrition and dietetics.
Collapse
Affiliation(s)
- Judi Porter
- Department of Nutrition, Dietetics and Food, Monash University, Victoria, Australia.,Allied Health Clinical Research Office, Eastern Health, Melbourne, Victoria, Australia
| | - Eleanor Beck
- School of Medicine, University of Wollongong, Wollongong, New South Wales, Australia
| | - Danielle Gallegos
- School of Exercise and Nutrition Sciences, Queensland University of Technology, Brisbane, Queensland, Australia
| | - Claire Palermo
- Monash Centre for Scholarship in Health Education, Monash University, Melbourne, Victoria, Australia
| | - Karen Walton
- School of Medicine, University of Wollongong, Wollongong, New South Wales, Australia
| | - Alison Yaxley
- College of Nursing and Health Science, Flinders University, Adelaide, South Australia, Australia
| | - Evelyn Volders
- Department of Nutrition, Dietetics and Food, Monash University, Victoria, Australia
| | - Amanda Wray
- College of Nursing and Health Science, Flinders University, Adelaide, South Australia, Australia
| | - Mary Hannan-Jones
- School of Exercise and Nutrition Sciences, Queensland University of Technology, Brisbane, Queensland, Australia
| |
Collapse
|
44
|
Marceau M, Gallagher F, Young M, St-Onge C. Validity as a social imperative for assessment in health professions education: a concept analysis. MEDICAL EDUCATION 2018; 52:641-653. [PMID: 29878449 DOI: 10.1111/medu.13574] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/15/2017] [Revised: 10/02/2017] [Accepted: 01/30/2018] [Indexed: 06/08/2023]
Abstract
CONTEXT Assessment can have far-reaching consequences for future health care professionals and for society. Thus, it is essential to establish the quality of assessment. Few modern approaches to validity are well situated to ensure the quality of complex assessment approaches, such as authentic and programmatic assessments. Here, we explore and delineate the concept of validity as a social imperative in the context of assessment in health professions education (HPE) as a potential framework for examining the quality of complex and programmatic assessment approaches. METHODS We conducted a concept analysis using Rodgers' evolutionary method to describe the concept of validity as a social imperative in the context of assessment in HPE. Supported by an academic librarian, we developed and executed a search strategy across several databases for literature published between 1995 and 2016. From a total of 321 citations, we identified 67 articles that met our inclusion criteria. Two team members analysed the texts using a specified approach to qualitative data analysis. Consensus was achieved through full team discussions. RESULTS Attributes that characterise the concept were: (i) demonstration of the use of evidence considered credible by society to document the quality of assessment; (ii) validation embedded through the assessment process and score interpretation; (iii) documented validity evidence supporting the interpretation of the combination of assessment findings, and (iv) demonstration of a justified use of a variety of evidence (quantitative and qualitative) to document the quality of assessment strategies. CONCLUSIONS The emerging concept of validity as a social imperative highlights some areas of focus in traditional validation frameworks, whereas some characteristics appear unique to HPE and move beyond traditional frameworks. The study reflects the importance of embedding consideration for society and societal concerns throughout the assessment and validation process, and may represent a potential lens through which to examine the quality of complex and programmatic assessment approaches.
Collapse
Affiliation(s)
- Mélanie Marceau
- Department of Nursing, Faculty of Medicine and Health Sciences, Université de Sherbrooke, Sherbrooke, Quebec, Canada
| | - Frances Gallagher
- Department of Nursing, Faculty of Medicine and Health Sciences, Université de Sherbrooke, Sherbrooke, Quebec, Canada
| | - Meredith Young
- Department of Medicine and Center for Medical Education, Faculty of Medicine, McGill University, Montreal, Quebec, Canada
| | - Christina St-Onge
- Department of Medicine, Faculty of Medicine and Health Sciences, Université de Sherbrooke, Sherbrooke, Quebec, Canada
| |
Collapse
|
45
|
Duijn CCMA, Welink LS, Bok HGJ, Ten Cate OTJ. When to trust our learners? Clinical teachers' perceptions of decision variables in the entrustment process. PERSPECTIVES ON MEDICAL EDUCATION 2018; 7:192-199. [PMID: 29713908 PMCID: PMC6002285 DOI: 10.1007/s40037-018-0430-0] [Citation(s) in RCA: 32] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/07/2023]
Abstract
INTRODUCTION Clinical training programs increasingly use entrustable professional activities (EPAs) as focus of assessment. However, questions remain about which information should ground decisions to trust learners. This qualitative study aimed to identify decision variables in the workplace that clinical teachers find relevant in the elaboration of the entrustment decision processes. The findings can substantiate entrustment decision-making in the clinical workplace. METHODS Focus groups were conducted with medical and veterinary clinical teachers, using the structured consensus method of the Nominal Group Technique to generate decision variables. A ranking was made based on a relevance score assigned by the clinical teachers to the different decision variables. Field notes, audio recordings and flip chart lists were analyzed and subsequently translated and, as a form of axial coding, merged into one list, combining the decision variables that were similar in their meaning. RESULTS A list of 11 and 17 decision variables were acknowledged as relevant by the medical and veterinary teacher groups, respectively. The focus groups yielded 21 unique decision variables that were considered relevant to inform readiness to perform a clinical task on a designated level of supervision. The decision variables consisted of skills, generic qualities, characteristics, previous performance or other information. We were able to group the decision variables into five categories: ability, humility, integrity, reliability and adequate exposure. DISCUSSION To entrust a learner to perform a task at a specific level of supervision, a supervisor needs information to support such a judgement. This trust cannot be credited on a single case at a single moment of assessment, but requires different variables and multiple sources of information. This study provides an overview of decision variables giving evidence to justify the multifactorial process of making an entrustment decision.
Collapse
Affiliation(s)
- Chantal C M A Duijn
- Center for Quality Improvement in Veterinary Education, Faculty of Veterinary Medicine, Utrecht University, Utrecht, The Netherlands.
| | - Lisanne S Welink
- Center for Research and Development of Education, University Medical Center Utrecht, Utrecht, The Netherlands
| | - Harold G J Bok
- Center for Quality Improvement in Veterinary Education, Faculty of Veterinary Medicine, Utrecht University, Utrecht, The Netherlands
| | - Olle T J Ten Cate
- Center for Research and Development of Education, University Medical Center Utrecht, Utrecht, The Netherlands
| |
Collapse
|
46
|
Aubin AS, St-Onge C, Renaud JS. Detecting rater bias using a person-fit statistic: a Monte Carlo simulation study. PERSPECTIVES ON MEDICAL EDUCATION 2018; 7:83-92. [PMID: 29294255 PMCID: PMC5889374 DOI: 10.1007/s40037-017-0391-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/04/2023]
Abstract
INTRODUCTION With the Standards voicing concern for the appropriateness of response processes, we need to explore strategies that would allow us to identify inappropriate rater response processes. Although certain statistics can be used to help detect rater bias, their use is complicated by either a lack of data about their actual power to detect rater bias or the difficulty related to their application in the context of health professions education. This exploratory study aimed to establish the worthiness of pursuing the use of l z to detect rater bias. METHODS We conducted a Monte Carlo simulation study to investigate the power of a specific detection statistic, that is: the standardized likelihood l z person-fit statistics (PFS). Our primary outcome was the detection rate of biased raters, namely: raters whom we manipulated into being either stringent (giving lower scores) or lenient (giving higher scores), using the l z statistic while controlling for the number of biased raters in a sample (6 levels) and the rate of bias per rater (6 levels). RESULTS Overall, stringent raters (M = 0.84, SD = 0.23) were easier to detect than lenient raters (M = 0.31, SD = 0.28). More biased raters were easier to detect then less biased raters (60% bias: 62, SD = 0.37; 10% bias: 43, SD = 0.36). DISCUSSION The PFS l z seems to offer an interesting potential to identify biased raters. We observed detection rates as high as 90% for stringent raters, for whom we manipulated more than half their checklist. Although we observed very interesting results, we cannot generalize these results to the use of PFS with estimated item/station parameters or real data. Such studies should be conducted to assess the feasibility of using PFS to identify rater bias.
Collapse
|
47
|
de Jonge LPJWM, Timmerman AA, Govaerts MJB, Muris JWM, Muijtjens AMM, Kramer AWM, van der Vleuten CPM. Stakeholder perspectives on workplace-based performance assessment: towards a better understanding of assessor behaviour. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2017; 22:1213-1243. [PMID: 28155004 PMCID: PMC5663793 DOI: 10.1007/s10459-017-9760-7] [Citation(s) in RCA: 12] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/07/2016] [Accepted: 01/24/2017] [Indexed: 05/13/2023]
Abstract
Workplace-Based Assessment (WBA) plays a pivotal role in present-day competency-based medical curricula. Validity in WBA mainly depends on how stakeholders (e.g. clinical supervisors and learners) use the assessments-rather than on the intrinsic qualities of instruments and methods. Current research on assessment in clinical contexts seems to imply that variable behaviours during performance assessment of both assessors and learners may well reflect their respective beliefs and perspectives towards WBA. We therefore performed a Q methodological study to explore perspectives underlying stakeholders' behaviours in WBA in a postgraduate medical training program. Five different perspectives on performance assessment were extracted: Agency, Mutuality, Objectivity, Adaptivity and Accountability. These perspectives reflect both differences and similarities in stakeholder perceptions and preferences regarding the utility of WBA. In comparing and contrasting the various perspectives, we identified two key areas of disagreement, specifically 'the locus of regulation of learning' (i.e., self-regulated versus externally regulated learning) and 'the extent to which assessment should be standardised' (i.e., tailored versus standardised assessment). Differing perspectives may variously affect stakeholders' acceptance, use-and, consequently, the effectiveness-of assessment programmes. Continuous interaction between all stakeholders is essential to monitor, adapt and improve assessment practices and to stimulate the development of a shared mental model. Better understanding of underlying stakeholder perspectives could be an important step in bridging the gap between psychometric and socio-constructivist approaches in WBA.
Collapse
Affiliation(s)
- Laury P J W M de Jonge
- Department of Family Medicine, FHML, Maastricht University, P.O. Box 616, 6200 MD, Maastricht, The Netherlands.
| | - Angelique A Timmerman
- Department of Family Medicine, FHML, Maastricht University, P.O. Box 616, 6200 MD, Maastricht, The Netherlands
| | - Marjan J B Govaerts
- Department of Educational Research and Development, FHML, Maastricht University, Maastricht, The Netherlands
| | - Jean W M Muris
- Department of Family Medicine, FHML, Maastricht University, P.O. Box 616, 6200 MD, Maastricht, The Netherlands
| | - Arno M M Muijtjens
- Department of Educational Research and Development, FHML, Maastricht University, Maastricht, The Netherlands
| | - Anneke W M Kramer
- Department of Family Medicine, Leiden University, Leiden, The Netherlands
| | - Cees P M van der Vleuten
- Department of Educational Research and Development, FHML, Maastricht University, Maastricht, The Netherlands
| |
Collapse
|
48
|
Hauer KE, Nishimura H, Dubon D, Teherani A, Boscardin C. Competency assessment form to improve feedback. CLINICAL TEACHER 2017; 15:472-477. [PMID: 29045060 DOI: 10.1111/tct.12726] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
Abstract
BACKGROUND In-training evaluation reports are a commonly used assessment method for clinical learners that can characterise the development of competence in essential domains of practice. Strategies to increase the usefulness and specificity of written narrative comments about learner performance in these reports are needed to guide their learning. Soliciting narrative comments by competency domain from supervising doctors on in-training evaluation reports could improve the quality of written feedback to students. METHODS This is a pre-post study examining narrative comments derived from assessments of core clerkship students by faculty members and resident supervisors in seven clerkships using two assessment forms in academic years 2013/14 (pre; two comments fields - summative, constructive) and 2014/15 (post; seven comments fields - six competency domains, constructive comments). Using a purposive sample of 60 students based on overall clerkship performance, we conducted content analysis of written comments to compare comment quality based on word count, competencies addressed and reinforcing or constructive content. Differences between the two forms across these three components of quality were compared using Student's t-tests. RESULTS The revised form elicited more narrative comments in all seven clerkships, with more competencies addressed. The revised form led to a decrease in the proportion of constructive comments about the students' performances. In-training evaluation reports are a commonly used assessment method for clinical learners DISCUSSION: Structural changes to a medical student assessment form to elicit narrative comments by competency improved some measures of the quality of narrative comments provided by faculty members and residents. Additional study is needed to determine how learners use this information to improve their clinical practice.
Collapse
Affiliation(s)
- Karen E Hauer
- University of California at San Francisco, San Francisco, California, USA
| | - Holly Nishimura
- University of California at San Francisco, San Francisco, California, USA
| | - Diego Dubon
- University of California at Berkeley, Berkeley, California, USA
| | - Arianne Teherani
- University of California at San Francisco, San Francisco, California, USA
| | - Christy Boscardin
- University of California at San Francisco, San Francisco, California, USA
| |
Collapse
|
49
|
Kogan JR, Hatala R, Hauer KE, Holmboe E. Guidelines: The do's, don'ts and don't knows of direct observation of clinical skills in medical education. PERSPECTIVES ON MEDICAL EDUCATION 2017; 6:286-305. [PMID: 28956293 PMCID: PMC5630537 DOI: 10.1007/s40037-017-0376-7] [Citation(s) in RCA: 28] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/12/2023]
Abstract
INTRODUCTION Direct observation of clinical skills is a key assessment strategy in competency-based medical education. The guidelines presented in this paper synthesize the literature on direct observation of clinical skills. The goal is to provide a practical list of Do's, Don'ts and Don't Knows about direct observation for supervisors who teach learners in the clinical setting and for educational leaders who are responsible for clinical training programs. METHODS We built consensus through an iterative approach in which each author, based on their medical education and research knowledge and expertise, independently developed a list of Do's, Don'ts, and Don't Knows about direct observation of clinical skills. Lists were compiled, discussed and revised. We then sought and compiled evidence to support each guideline and determine the strength of each guideline. RESULTS A final set of 33 Do's, Don'ts and Don't Knows is presented along with a summary of evidence for each guideline. Guidelines focus on two groups: individual supervisors and the educational leaders responsible for clinical training programs. Guidelines address recommendations for how to focus direct observation, select an assessment tool, promote high quality assessments, conduct rater training, and create a learning culture conducive to direct observation. CONCLUSIONS High frequency, high quality direct observation of clinical skills can be challenging. These guidelines offer important evidence-based Do's and Don'ts that can help improve the frequency and quality of direct observation. Improving direct observation requires focus not just on individual supervisors and their learners, but also on the organizations and cultures in which they work and train. Additional research to address the Don't Knows can help educators realize the full potential of direct observation in competency-based education.
Collapse
Affiliation(s)
- Jennifer R Kogan
- Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA, USA.
| | - Rose Hatala
- University of British Columbia, Vancouver, British Columbia, Canada
| | - Karen E Hauer
- University of California San Francisco, San Francisco, CA, USA
| | - Eric Holmboe
- Accreditation Council of Graduate Medical Education, Chicago, IL, USA
| |
Collapse
|
50
|
St-Onge C, Young M, Eva KW, Hodges B. Validity: one word with a plurality of meanings. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2017; 22:853-867. [PMID: 27696103 DOI: 10.1007/s10459-016-9716-3] [Citation(s) in RCA: 40] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/13/2016] [Accepted: 09/26/2016] [Indexed: 06/06/2023]
Abstract
Validity is one of the most debated constructs in our field; debates abound about what is legitimate and what is not, and the word continues to be used in ways that are explicitly disavowed by current practice guidelines. The resultant tensions have not been well characterized, yet their existence suggests that different uses may maintain some value for the user that needs to be better understood. We conducted an empirical form of Discourse Analysis to document the multiple ways in which validity is described, understood, and used in the health professions education field. We created and analyzed an archive of texts identified from multiple sources, including formal databases such as PubMED, ERIC and PsycINFO as well as the authors' personal assessment libraries. An iterative analytic process was used to identify, discuss, and characterize emerging discourses about validity. Three discourses of validity were identified. Validity as a test characteristic is underpinned by the notion that validity is an intrinsic property of a tool and could, therefore, be seen as content and context independent. Validity as an argument-based evidentiary-chain emphasizes the importance of supporting the interpretation of assessment results with ongoing analysis such that validity does not belong to the tool/instrument itself. The emphasis is on process-based validation (emphasizing the journey instead of the goal). Validity as a social imperative foregrounds the consequences of assessment at the individual and societal levels, be they positive or negative. The existence of different discourses may explain-in part-results observed in recent systematic reviews that highlighted discrepancies and tensions between recommendations for practice and the validation practices that are actually adopted and reported. Some of these practices, despite contravening accepted validation 'guidelines', may nevertheless respond to different and somewhat unarticulated needs within health professional education.
Collapse
Affiliation(s)
| | | | - Kevin W Eva
- University of British Columbia, Vancouver, Canada
| | | |
Collapse
|