1
|
Miller DT, Michael S, Bell C, Brevik CH, Kaplan B, Svoboda E, Kendall J. Physical and biophysical markers of assessment in medical training: A scoping review of the literature. MEDICAL TEACHER 2024:1-9. [PMID: 38688520 DOI: 10.1080/0142159x.2024.2345269] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/16/2024] [Accepted: 04/16/2024] [Indexed: 05/02/2024]
Abstract
PURPOSE Assessment in medical education has changed over time to measure the evolving skills required of current medical practice. Physical and biophysical markers of assessment attempt to use technology to gain insight into medical trainees' knowledge, skills, and attitudes. The authors conducted a scoping review to map the literature on the use of physical and biophysical markers of assessment in medical training. MATERIALS AND METHODS The authors searched seven databases on 1 August 2022, for publications that utilized physical or biophysical markers in the assessment of medical trainees (medical students, residents, fellows, and synonymous terms used in other countries). Physical or biophysical markers included: heart rate and heart rate variability, visual tracking and attention, pupillometry, hand motion analysis, skin conductivity, salivary cortisol, functional magnetic resonance imaging (fMRI), and functional near-infrared spectroscopy (fNIRS). The authors mapped the relevant literature using Bloom's taxonomy of knowledge, skills, and attitudes and extracted additional data including study design, study environment, and novice vs. expert differentiation from February to June 2023. RESULTS Of 6,069 unique articles, 443 met inclusion criteria. The majority of studies assessed trainees using heart rate variability (n = 160, 36%) followed by visual attention (n = 143, 32%), hand motion analysis (n = 67, 15%), salivary cortisol (n = 67, 15%), fMRI (n = 29, 7%), skin conductivity (n = 26, 6%), fNIRs (n = 19, 4%), and pupillometry (n = 16, 4%). The majority of studies (n = 167, 38%) analyzed non-technical skills, followed by studies that analyzed technical skills (n = 155, 35%), knowledge (n = 114, 26%), and attitudinal skills (n = 61, 14%). 169 studies (38%) attempted to use physical or biophysical markers to differentiate between novice and expert. CONCLUSION This review provides a comprehensive description of the current use of physical and biophysical markers in medical education training, including the current technology and skills assessed. Additionally, while physical and biophysical markers have the potential to augment current assessment in medical education, there remains significant gaps in research surrounding reliability, validity, cost, practicality, and educational impact of implementing these markers of assessment.
Collapse
Affiliation(s)
- Danielle T Miller
- Department of Emergency Medicine, University of Colorado School of Medicine, Aurora, CO, USA
| | - Sarah Michael
- Department of Emergency Medicine, University of Colorado School of Medicine, Aurora, CO, USA
| | - Colin Bell
- Department of Emergency Medicine, University of Calgary, Calgary, Canada
| | - Cody H Brevik
- Department of Emergency Medicine, University of Colorado School of Medicine, Aurora, CO, USA
| | - Bonnie Kaplan
- Department of Emergency Medicine, University of Colorado School of Medicine, Aurora, CO, USA
| | - Ellie Svoboda
- Education Informationist, Strauss Health Sciences Library, University of Colorado Anschutz Medical Campus, Aurora, CO, USA
| | - John Kendall
- Department of Emergency Medicine, Stanford School of Medicine, Palo Alto, CA, USA
| |
Collapse
|
2
|
Boursicot K, Kemp S, Norcini J, Nadarajah VD, Humphrey-Murto S, Archer E, Williams J, Pyörälä E, Möller R. Synthesis and perspectives from the Ottawa 2022 conference on the assessment of competence. MEDICAL TEACHER 2023; 45:978-983. [PMID: 36786837 DOI: 10.1080/0142159x.2023.2174420] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/18/2023]
Abstract
INTRODUCTION The Ottawa Conference on the Assessment of Competence in Medicine and the Healthcare Professions was first convened in 1985 in Ottawa. Since then, what has become known as the Ottawa conference has been held in various locations around the world every 2 years. It has become an important conference for the community of assessment - including researchers, educators, administrators and leaders - to share contemporary knowledge and develop international standards for assessment in medical and health professions education. METHODS The Ottawa 2022 conference was held in Lyon, France, in conjunction with the AMEE 2022 conference. A diverse group of international assessment experts were invited to present a symposium at the AMEE conference to summarise key concepts from the Ottawa conference. This paper was developed from that symposium. RESULTS AND DISCUSSION This paper summarises key themes and issues that emerged from the Ottawa 2022 conference. It highlights the importance of the consensus statements and discusses challenges for assessment such as issues of equity, diversity, and inclusion, shifts in emphasis to systems of assessment, implications of 'big data' and analytics, and challenges to ensure published research and practice are based on contemporary theories and concepts.
Collapse
Affiliation(s)
| | - Sandra Kemp
- Graduate School of Medicine, University of Wollongong, Wollongong, Australia
| | - John Norcini
- Department of Psychiatry, Upstate Medical University, Syracuse, NY, USA
| | | | | | - Elize Archer
- Faculty of Medicine and Health Sciences, Stellenbosch University, Cape Town, South Africa
| | - Jen Williams
- Faculty Dean of Medicine and Health, University of New England, Armidale, Australia
| | - Eeva Pyörälä
- Center for University Teaching and Learning, University of Helsinki, Helsinki, Finland
| | - Riitta Möller
- Department of Medical Epidemiology and Biostatistics, Karolinska Institutet, Stockholm, Sweden
| |
Collapse
|
3
|
Buléon C, Mattatia L, Minehart RD, Rudolph JW, Lois FJ, Guillouet E, Philippon AL, Brissaud O, Lefevre-Scelles A, Benhamou D, Lecomte F, group TSAWS, Bellot A, Crublé I, Philippot G, Vanderlinden T, Batrancourt S, Boithias-Guerot C, Bréaud J, de Vries P, Sibert L, Sécheresse T, Boulant V, Delamarre L, Grillet L, Jund M, Mathurin C, Berthod J, Debien B, Gacia O, Der Sahakian G, Boet S, Oriot D, Chabot JM. Simulation-based summative assessment in healthcare: an overview of key principles for practice. ADVANCES IN SIMULATION (LONDON, ENGLAND) 2022; 7:42. [PMID: 36578052 PMCID: PMC9795938 DOI: 10.1186/s41077-022-00238-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 03/02/2022] [Accepted: 11/30/2022] [Indexed: 12/29/2022]
Abstract
BACKGROUND Healthcare curricula need summative assessments relevant to and representative of clinical situations to best select and train learners. Simulation provides multiple benefits with a growing literature base proving its utility for training in a formative context. Advancing to the next step, "the use of simulation for summative assessment" requires rigorous and evidence-based development because any summative assessment is high stakes for participants, trainers, and programs. The first step of this process is to identify the baseline from which we can start. METHODS First, using a modified nominal group technique, a task force of 34 panelists defined topics to clarify the why, how, what, when, and who for using simulation-based summative assessment (SBSA). Second, each topic was explored by a group of panelists based on state-of-the-art literature reviews technique with a snowball method to identify further references. Our goal was to identify current knowledge and potential recommendations for future directions. Results were cross-checked among groups and reviewed by an independent expert committee. RESULTS Seven topics were selected by the task force: "What can be assessed in simulation?", "Assessment tools for SBSA", "Consequences of undergoing the SBSA process", "Scenarios for SBSA", "Debriefing, video, and research for SBSA", "Trainers for SBSA", and "Implementation of SBSA in healthcare". Together, these seven explorations provide an overview of what is known and can be done with relative certainty, and what is unknown and probably needs further investigation. Based on this work, we highlighted the trustworthiness of different summative assessment-related conclusions, the remaining important problems and questions, and their consequences for participants and institutions of how SBSA is conducted. CONCLUSION Our results identified among the seven topics one area with robust evidence in the literature ("What can be assessed in simulation?"), three areas with evidence that require guidance by expert opinion ("Assessment tools for SBSA", "Scenarios for SBSA", "Implementation of SBSA in healthcare"), and three areas with weak or emerging evidence ("Consequences of undergoing the SBSA process", "Debriefing for SBSA", "Trainers for SBSA"). Using SBSA holds much promise, with increasing demand for this application. Due to the important stakes involved, it must be rigorously conducted and supervised. Guidelines for good practice should be formalized to help with conduct and implementation. We believe this baseline can direct future investigation and the development of guidelines.
Collapse
Affiliation(s)
- Clément Buléon
- grid.460771.30000 0004 1785 9671Department of Anesthesiology, Intensive Care and Perioperative Medicine, Caen Normandy University Hospital, 6th Floor, Caen, France ,grid.412043.00000 0001 2186 4076Medical School, University of Caen Normandy, Caen, France ,grid.419998.40000 0004 0452 5971Center for Medical Simulation, Boston, MA USA
| | - Laurent Mattatia
- grid.411165.60000 0004 0593 8241Department of Anesthesiology, Intensive Care and Perioperative Medicine, Nîmes University Hospital, Nîmes, France
| | - Rebecca D. Minehart
- grid.419998.40000 0004 0452 5971Center for Medical Simulation, Boston, MA USA ,grid.32224.350000 0004 0386 9924Department of Anesthesia, Critical Care and Pain Medicine, Massachusetts General Hospital, Boston, MA USA ,grid.38142.3c000000041936754XHarvard Medical School, Boston, MA USA
| | - Jenny W. Rudolph
- grid.419998.40000 0004 0452 5971Center for Medical Simulation, Boston, MA USA ,grid.32224.350000 0004 0386 9924Department of Anesthesia, Critical Care and Pain Medicine, Massachusetts General Hospital, Boston, MA USA ,grid.38142.3c000000041936754XHarvard Medical School, Boston, MA USA
| | - Fernande J. Lois
- grid.4861.b0000 0001 0805 7253Department of Anesthesiology, Intensive Care and Perioperative Medicine, Liège University Hospital, Liège, Belgique
| | - Erwan Guillouet
- grid.460771.30000 0004 1785 9671Department of Anesthesiology, Intensive Care and Perioperative Medicine, Caen Normandy University Hospital, 6th Floor, Caen, France ,grid.412043.00000 0001 2186 4076Medical School, University of Caen Normandy, Caen, France
| | - Anne-Laure Philippon
- grid.411439.a0000 0001 2150 9058Department of Emergency Medicine, Pitié Salpêtrière University Hospital, APHP, Paris, France
| | - Olivier Brissaud
- grid.42399.350000 0004 0593 7118Department of Pediatric Intensive Care, Pellegrin University Hospital, Bordeaux, France
| | - Antoine Lefevre-Scelles
- grid.41724.340000 0001 2296 5231Department of Emergency Medicine, Rouen University Hospital, Rouen, France
| | - Dan Benhamou
- grid.413784.d0000 0001 2181 7253Department of Anesthesiology, Intensive Care and Perioperative Medicine, Kremlin Bicêtre University Hospital, APHP, Paris, France
| | - François Lecomte
- grid.411784.f0000 0001 0274 3893Department of Emergency Medicine, Cochin University Hospital, APHP, Paris, France
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |
Collapse
|
4
|
Fuller R, Goddard VCT, Nadarajah VD, Treasure-Jones T, Yeates P, Scott K, Webb A, Valter K, Pyorala E. Technology enhanced assessment: Ottawa consensus statement and recommendations. MEDICAL TEACHER 2022; 44:836-850. [PMID: 35771684 DOI: 10.1080/0142159x.2022.2083489] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
INTRODUCTION In 2011, a consensus report was produced on technology-enhanced assessment (TEA), its good practices, and future perspectives. Since then, technological advances have enabled innovative practices and tools that have revolutionised how learners are assessed. In this updated consensus, we bring together the potential of technology and the ultimate goals of assessment on learner attainment, faculty development, and improved healthcare practices. METHODS As a material for the report, we used the scholarly publications on TEA in both HPE and general higher education, feedback from 2020 Ottawa Conference workshops, and scholarly publications on assessment technology practices during the Covid-19 pandemic. RESULTS AND CONCLUSION The group identified areas of consensus that remained to be resolved and issues that arose in the evolution of TEA. We adopted a three-stage approach (readiness to adopt technology, application of assessment technology, and evaluation/dissemination). The application stage adopted an assessment 'lifecycle' approach and targeted five key foci: (1) Advancing authenticity of assessment, (2) Engaging learners with assessment, (3) Enhancing design and scheduling, (4) Optimising assessment delivery and recording learner achievement, and (5) Tracking learner progress and faculty activity and thereby supporting longitudinal learning and continuous assessment.
Collapse
Affiliation(s)
- Richard Fuller
- Christie Education, The Christie NHS Foundation Trust, Manchester, UK
| | | | | | | | - Peter Yeates
- School of Medicine, University of Keele, Keele, UK
| | - Karen Scott
- Faculty of Medicine and Health, University of Sydney, Sydney, Australia
| | - Alexandra Webb
- College of Health and Medicine, Australian National University, Canberra, Australia
| | - Krisztina Valter
- John Curtin School of Medical Research, Australian National University, Canberra, Australia
| | - Eeva Pyorala
- Center for University Teaching and Learning, University of Helsinki, Helsinki, Finland
| |
Collapse
|
5
|
Egarter S, Mutschler A, Brass K. Impact of COVID-19 on digital medical education: compatibility of digital teaching and examinations with integrity and ethical principles. INTERNATIONAL JOURNAL FOR EDUCATIONAL INTEGRITY 2021; 17:18. [PMCID: PMC8421088 DOI: 10.1007/s40979-021-00084-8] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/01/2020] [Accepted: 07/05/2021] [Indexed: 06/01/2023]
Abstract
The COVID-19 (coronavirus disease 2019) pandemic in 2020 has had a lasting impact on all areas of personal life. However, the political, economic, legal and healthcare system, as well as the education system have also experienced the effects. Universities had to face new challenges and requirements in teaching and examinations as quickly as possible in order to be able to guarantee high-quality education for their students. This study aims to examine how the German-speaking medical faculties of the Umbrella Consortium of Assessment Network (UCAN) have dealt with the challenges but also the opportunities that the outbreak of the COVID-19 pandemic created in medical education and whether digitalisation has been driven forward as a result. In an initial online-survey we focused our questions on the current teaching situation with regard to digitised teaching content, the support or establishment of adequate framework conditions by the medical faculties and IT facilities and also the execution of examinations during the summer semester 2020. Between August and September 2020, a total of 88 examiners, educators, dean of study and/or technical admins from 32 partner faculties took part in the survey. Students were not included in our survey. Most respondents stated that a switch to a digital semester had worked, the use of e-learning increased compared to previous semesters and that most courses could be converted, with the exception of practical courses, which were largely cancelled. The respondents also indicated that most examinations could still be taken, with the exception of practical examination formats, like Objective Structured Clinical Examinations (OSCEs). However, in the case of face-to-face examinations, strict distance and hygiene conditions had to be obeyed or there had to be a switch to distance-online examinations, which raised many open issues such as equal opportunities of students (technical equipment, internet access, premises) and attempts at deception (third-party help with the exam, mutual exchange between students, web search). In conclusion, we identified several issues regarding the rapid transition to a digital semester due to COVID-19 which were categorised into the following topics: Face-to-face teaching could not take place, (2) know-how of educators, (3) integrity aspects, (4) technical aspects, (5) additional personnel required, (6) additional time and effort required for implementation of digital teaching. Our study shows that a switch to digital teaching and distance online examinations is feasible, but many problems were encountered concerning academic integrity and basic ethical principles still need to be solved. In order to investigate whether above mentioned issued could be solved one year after the transition to a digital semester, we conducted a second survey in which the 32 initially surveyed institutions were questioned again.
Collapse
Affiliation(s)
- Saskia Egarter
- Institute for Communication and Assessment Research, Wieblinger Weg 92A, 69123 Heidelberg, Germany
| | - Anna Mutschler
- Institute for Communication and Assessment Research, Wieblinger Weg 92A, 69123 Heidelberg, Germany
| | - Konstantin Brass
- Institute for Communication and Assessment Research, Wieblinger Weg 92A, 69123 Heidelberg, Germany
| |
Collapse
|
6
|
Yeates P, Moult A, Lefroy J, Walsh-House J, Clews L, McKinley R, Fuller R. Understanding and developing procedures for video-based assessment in medical education. MEDICAL TEACHER 2020; 42:1250-1260. [PMID: 32749915 DOI: 10.1080/0142159x.2020.1801997] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
INTRODUCTION Novel uses of video aim to enhance assessment in health-professionals education. Whilst these uses presume equivalence between video and live scoring, some research suggests that poorly understood variations could challenge validity. We aimed to understand examiners' and students' interaction with video whilst developing procedures to promote its optimal use. METHODS Using design-based research we developed theory and procedures for video use in assessment, iteratively adapting conditions across simulated OSCE stations. We explored examiners' and students' perceptions using think-aloud, interviews and focus group. Data were analysed using constructivist grounded-theory methods. RESULTS Video-based assessment produced detachment and reduced volitional control for examiners. Examiners ability to make valid video-based judgements was mediated by the interaction of station content and specifically selected filming parameters. Examiners displayed several judgemental tendencies which helped them manage videos' limitations but could also bias judgements in some circumstances. Students rarely found carefully-placed cameras intrusive and considered filming acceptable if adequately justified. DISCUSSION Successful use of video-based assessment relies on balancing the need to ensure station-specific information adequacy; avoiding disruptive intrusion; and the degree of justification provided by video's educational purpose. Video has the potential to enhance assessment validity and students' learning when an appropriate balance is achieved.
Collapse
Affiliation(s)
- Peter Yeates
- School of Medicine, Keele University, Keele, UK
- Department of Acute Medicine, Fairfield General Hospital, Pennine Acute Hospital NHS Trust, Bury, UK
| | - Alice Moult
- School of Medicine, Keele University, Keele, UK
| | | | | | | | | | - Richard Fuller
- School of Medicine, University of Liverpool, Liverpool, UK
| |
Collapse
|
7
|
Cleland J, Chu J, Lim S, Low J, Low-Beer N, Kwek TK. COVID 19: Designing and conducting an online mini-multiple interview (MMI) in a dynamic landscape. MEDICAL TEACHER 2020; 42:776-780. [PMID: 32412815 DOI: 10.1080/0142159x.2020.1762851] [Citation(s) in RCA: 18] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
Introduction: The COVID-19 pandemic presented numerous, significant challenges for medical schools, including how to select the best candidates from a pool of applicants when social distancing and other measures prevented "business as usual" admissions processes. However, selection into medical school is the gateway to medicine in many countries, and it is critical to use processes which are evidence-based, valid and reliable even under challenging circumstances. Our challenge was to plan and conduct a multiple-mini interview (MMI) in a dynamic and stringent safe distancing context.Methods: This paper reports a case study of how to plan, re-plan and conduct MMIs in an environment where substantially tighter safe distancing measures were introduced just before the MMI was due to be delivered.Results: We report on how to design and implement a fully remote, online MMI which ensured the safety of candidates and assessors.Discussion: We discuss the challenges of this approach and also reflect on broader issues associated with selection into medical school during a pandemic. The aim of the paper is to provide broadly generalizable guidance to other medical schools faced with the challenge of selecting future students under difficult conditions.
Collapse
Affiliation(s)
- Jennifer Cleland
- Lee Kong Chian School of Medicine (LKC Medicine), Nanyang Technological University, Singapore
| | - Jowe Chu
- Lee Kong Chian School of Medicine (LKC Medicine), Nanyang Technological University, Singapore
| | - Samuel Lim
- Lee Kong Chian School of Medicine (LKC Medicine), Nanyang Technological University, Singapore
| | - Jamie Low
- Lee Kong Chian School of Medicine (LKC Medicine), Nanyang Technological University, Singapore
| | - Naomi Low-Beer
- Lee Kong Chian School of Medicine (LKC Medicine), Nanyang Technological University, Singapore
| | - Tong Kiat Kwek
- Lee Kong Chian School of Medicine (LKC Medicine), Nanyang Technological University, Singapore
| |
Collapse
|
8
|
|
9
|
Egarter S, Mutschler A, Tekian A, Norcini J, Brass K. Medical assessment in the age of digitalisation. BMC MEDICAL EDUCATION 2020; 20:101. [PMID: 32234051 PMCID: PMC7110637 DOI: 10.1186/s12909-020-02014-7] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/08/2019] [Accepted: 03/20/2020] [Indexed: 06/02/2023]
Abstract
BACKGROUND Digital assessment is becoming more and more popular within medical education. To analyse the dimensions of this digital trend, we investigated how exam questions (items) are created and designed for use in digital medical assessments in Germany. Thus, we want to explore whether different types of media are used for item creation and if a digital trend in medical assessment can be observed. METHODS In a cross-sectional descriptive study, we examined data of 30 German medical faculties stored within a common assessment platform. More precise, 23,008 exams which contained 847,137 items were analysed concerning the exam type (paper-, computer- or tablet-based) and their respective media content (picture, video and/or audio). Out of these, 5252 electronic exams with 12,214 questions were evaluated. The media types per individual question were quantified. RESULTS The amount of computer- and tablet-based exams were rapidly increasing from 2012 until 2018. Computer- and tablet-based written exams showed with 45 and 66% a higher percentage of exams containing media in comparison to paper-based exams (33%). Analysis on the level of individual questions showed that 90.8% of questions had one single picture. The remaining questions contained either more than one picture (2.9%), video (2.7%), audio (0.2%) or 3.3% of questions had picture as well as video added. The main question types used for items with one picture are TypeA (54%) and Long_Menu (31%). In contrast, questions with video content contain only 11% TypeA questions, whereas Long_Menu is represented by 66%. Nearly all questions containing both picture and video are Long_Menu questions. CONCLUSIONS It can be stated that digital assessment formats are indeed on the raise. Moreover, our data indicates that electronic assessments formats have easier options to embed media items and thus show a higher frequency of media addition. We even identified the usage of different media types in the same question and this innovative item design could be a useful feature for the creation of medical assessments. Moreover, the choice of media type seems to depend on the respective question type.
Collapse
Affiliation(s)
- Saskia Egarter
- Institute for Communication and Assessment Research, Wieblinger Weg 92A, Heidelberg, Germany.
| | - Anna Mutschler
- Institute for Communication and Assessment Research, Wieblinger Weg 92A, Heidelberg, Germany
- Center of Excellence Assessment in Medicine, University of Heidelberg, Heidelberg, Germany
| | - Ara Tekian
- Department of Medical Education, University of Illinois, Chicago, USA
| | - John Norcini
- Foundation for the Advancement of International Medical Education Research, Philadelphia, PA, USA
| | - Konstantin Brass
- Institute for Communication and Assessment Research, Wieblinger Weg 92A, Heidelberg, Germany
| |
Collapse
|
10
|
Egarter S, Mutschler A, Tekian A, Norcini J, Brass K. Medical assessment in the age of digitalisation. BMC MEDICAL EDUCATION 2020; 20:101. [PMID: 32234051 DOI: 10.1186/s12909-020-02014-7.pmid:32234051;pmcid:pmc7110637] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Subscribe] [Scholar Register] [Received: 11/08/2019] [Accepted: 03/20/2020] [Indexed: 05/22/2023]
Abstract
BACKGROUND Digital assessment is becoming more and more popular within medical education. To analyse the dimensions of this digital trend, we investigated how exam questions (items) are created and designed for use in digital medical assessments in Germany. Thus, we want to explore whether different types of media are used for item creation and if a digital trend in medical assessment can be observed. METHODS In a cross-sectional descriptive study, we examined data of 30 German medical faculties stored within a common assessment platform. More precise, 23,008 exams which contained 847,137 items were analysed concerning the exam type (paper-, computer- or tablet-based) and their respective media content (picture, video and/or audio). Out of these, 5252 electronic exams with 12,214 questions were evaluated. The media types per individual question were quantified. RESULTS The amount of computer- and tablet-based exams were rapidly increasing from 2012 until 2018. Computer- and tablet-based written exams showed with 45 and 66% a higher percentage of exams containing media in comparison to paper-based exams (33%). Analysis on the level of individual questions showed that 90.8% of questions had one single picture. The remaining questions contained either more than one picture (2.9%), video (2.7%), audio (0.2%) or 3.3% of questions had picture as well as video added. The main question types used for items with one picture are TypeA (54%) and Long_Menu (31%). In contrast, questions with video content contain only 11% TypeA questions, whereas Long_Menu is represented by 66%. Nearly all questions containing both picture and video are Long_Menu questions. CONCLUSIONS It can be stated that digital assessment formats are indeed on the raise. Moreover, our data indicates that electronic assessments formats have easier options to embed media items and thus show a higher frequency of media addition. We even identified the usage of different media types in the same question and this innovative item design could be a useful feature for the creation of medical assessments. Moreover, the choice of media type seems to depend on the respective question type.
Collapse
Affiliation(s)
- Saskia Egarter
- Institute for Communication and Assessment Research, Wieblinger Weg 92A, Heidelberg, Germany.
| | - Anna Mutschler
- Institute for Communication and Assessment Research, Wieblinger Weg 92A, Heidelberg, Germany
- Center of Excellence Assessment in Medicine, University of Heidelberg, Heidelberg, Germany
| | - Ara Tekian
- Department of Medical Education, University of Illinois, Chicago, USA
| | - John Norcini
- Foundation for the Advancement of International Medical Education Research, Philadelphia, PA, USA
| | - Konstantin Brass
- Institute for Communication and Assessment Research, Wieblinger Weg 92A, Heidelberg, Germany
| |
Collapse
|
11
|
Roche C, Flood M, Lynch M, Sahm LJ. Balancing Assessment with "In-Service Practical Training": A Case Report on Collaborative Curriculum Design for Delivery in the Practice Setting. PHARMACY 2019; 7:pharmacy7030093. [PMID: 31315215 PMCID: PMC6789868 DOI: 10.3390/pharmacy7030093] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/10/2019] [Revised: 07/08/2019] [Accepted: 07/11/2019] [Indexed: 11/27/2022] Open
Abstract
Three Higher Education Institutions (HEIs) in Ireland are accredited to provide education and training, successful completion of which, entitles one to register as a pharmacist with the Pharmaceutical Society of Ireland (PSI). Legislation (2014) mandated that these HEIs replace their existing structure (four-year degree followed by a one-year internship), with a five-year ‘integrated Master’s programme’. Integration includes ‘in-service practical training’ (placement) at the beginning of Year 4 (four months), and the end of Year 5 (eight months). Year 4 placements do not have to be ‘patient-facing’. Students receive a Bachelor’s degree at the end of Year 4. The Affiliation for Pharmacy Practice Experiential Learning (APPEL), established by the HEIs, manages student placements, training establishments, preceptor training, the preceptors’ competency assessment process, and the virtual learning environment (VLE) that enables delivery of co-developed online modules aligned with placements in Years 4 and 5. This case report aims to describe the process by which this integration has taken place across and within these HEIs and the challenges faced by educators, students, preceptors, and other stakeholders along the way.
Collapse
Affiliation(s)
- Cicely Roche
- School of Pharmacy & Pharmaceutical Sciences, Trinity College, Panoz Building, College Green, Dublin 2, Ireland.
| | - Michelle Flood
- School of Pharmacy, Royal College of Surgeons, Dublin 2, Ireland
| | - Matthew Lynch
- School of Pharmacy, Royal College of Surgeons, Dublin 2, Ireland
| | - Laura J Sahm
- Pharmaceutical Care Research Group, School of Pharmacy, University College Cork, Cork T12 YN60, Ireland
- Department of Pharmacy, Mercy University Hospital, Grenville Place, Cork T12 WE28, Ireland
| |
Collapse
|
12
|
Laks M, Guerra CM, Miraglia JL, Medeiros EA. Distance learning in antimicrobial stewardship: innovation in medical education. BMC MEDICAL EDUCATION 2019; 19:191. [PMID: 31174524 PMCID: PMC6555969 DOI: 10.1186/s12909-019-1623-x] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/16/2018] [Accepted: 05/22/2019] [Indexed: 06/01/2023]
Abstract
BACKGROUND This study aimed to analyse the impact that web-based distance learning has on knowledge gain in medical students, as well as student perceptions of the methodology. METHODS This was an educational intervention study conducted at a tertiary teaching hospital in the city of São Paulo, Brazil. From 2008 to 2014, we offered a free web-based distance learning course, covering antimicrobial use and microbial resistance, to fifth-year medical students. The course encompassed 100 h of activities, with five theoretical modules, exercises and simulations, within a virtual learning environment. The students were tutored in their online activities, and some classes were conducted in real time for live discussions. In addition, students underwent face-to-face assessments of their knowledge of the topic before and after the course. Statistical analysis was performed and the means of the overall scores were obtained, as were the respective 95% confidence intervals (CIs). The means were compared by two-tailed paired t-tests and by the paired Wilcoxon test. RESULTS Of 814 eligible medical students, 606 (74.45%) completed the entire course during the study period. The mean score for knowledge of the topic was significantly higher on the final assessment than on the initial assessment (p < 0.001). We found that dedication (in hours) was directly proportional to the level of participation, as reflected in the mean final score (p = 0.009) and in the proportion of students who passed (p = 0.028). All of the participants considered their knowledge adequate or insufficient before the course, stating that it is quite important or important to address the topic during medical education. Although dedication levels were low, 70.5% stated that they had learned "quite a lot" or "more than expected" about the topic and would dedicate more time to it if they could. CONCLUSIONS The use of a virtual learning environment can promote teaching and learning in the infectious diseases field, specifically for antimicrobial stewardship, increasing knowledge significantly, and should be considered for inclusion in the final stages of medical education.
Collapse
Affiliation(s)
- Michel Laks
- Infectious Diseases Division, Department of Internal Medicine, Escola Paulista de Medicina, Universidade Federal de São Paulo, Napoleão de Barros, 690, second floor, São Paulo, Brazil
| | - Carla Morales Guerra
- Infectious Diseases Division, Department of Internal Medicine, Escola Paulista de Medicina, Universidade Federal de São Paulo, Napoleão de Barros, 690, second floor, São Paulo, Brazil
| | - João Luiz Miraglia
- Infectious Diseases Division, Department of Internal Medicine, Escola Paulista de Medicina, Universidade Federal de São Paulo, Napoleão de Barros, 690, second floor, São Paulo, Brazil
| | - Eduardo Alexandrino Medeiros
- Infectious Diseases Division, Department of Internal Medicine, Escola Paulista de Medicina, Universidade Federal de São Paulo, Napoleão de Barros, 690, second floor, São Paulo, Brazil
| |
Collapse
|
13
|
Weersink K, Hall AK, Rich J, Szulewski A, Dagnone JD. Simulation versus real-world performance: a direct comparison of emergency medicine resident resuscitation entrustment scoring. Adv Simul (Lond) 2019; 4:9. [PMID: 31061721 PMCID: PMC6492388 DOI: 10.1186/s41077-019-0099-4] [Citation(s) in RCA: 34] [Impact Index Per Article: 6.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/21/2018] [Accepted: 04/15/2019] [Indexed: 11/10/2022] Open
Abstract
Background Simulation is increasingly being used in postgraduate medical education as an opportunity for competency assessment. However, there is limited direct evidence that supports performance in the simulation lab as a surrogate of workplace-based clinical performance for non-procedural tasks such as resuscitation in the emergency department (ED). We sought to directly compare entrustment scoring of resident performance in the simulation environment to clinical performance in the ED. Methods The resuscitation assessment tool (RAT) was derived from the previously implemented and studied Queen's simulation assessment tool (QSAT) via a modified expert review process. The RAT uses an anchored global assessment scale to generate an entrustment score and narrative comments. Emergency medicine (EM) residents were assessed using the RAT on cases in simulation-based examinations and in the ED during resuscitation cases from July 2016 to June 2017. Resident mean entrustment scores were compared using Pearson's correlation coefficient to determine the relationship between entrustment in simulation cases and in the ED. Inductive thematic analysis of written commentary was conducted to compare workplace-based with simulation-based feedback. Results There was a moderate, positive correlation found between mean entrustment scores in the simulated and workplace-based settings, which was statistically significant (r = 0.630, n = 17, p < 0.01). Further, qualitative analysis demonstrated overall management and leadership themes were more common narratives in the workplace, while more specific task-based feedback predominated in the simulation-based assessment. Both workplace-based and simulation-based narratives frequently commented on communication skills. Conclusions In this single-center study with a limited sample size, assessment of residents using entrustment scoring in simulation settings was demonstrated to have a moderate positive correlation with assessment of resuscitation competence in the workplace. This study suggests that resuscitation performance in simulation settings may be an indicator of competence in the clinical setting. However, multiple factors contribute to this complicated and imperfect relationship. It is imperative to consider narrative comments in supporting the rationale for numerical entrustment scores in both settings and to include both simulation and workplace-based assessment in high-stakes decisions of progression.
Collapse
Affiliation(s)
- Kristen Weersink
- 1Department of Emergency Medicine, Queen's University, Kingston Health Sciences Center c/o 76 Stuart St, Kingston, ON K7L2V7 Canada
| | - Andrew K Hall
- 1Department of Emergency Medicine, Queen's University, Kingston Health Sciences Center c/o 76 Stuart St, Kingston, ON K7L2V7 Canada
| | - Jessica Rich
- 2Faculty of Education, Queen's University, Kingston, ON Canada
| | - Adam Szulewski
- 1Department of Emergency Medicine, Queen's University, Kingston Health Sciences Center c/o 76 Stuart St, Kingston, ON K7L2V7 Canada
| | - J Damon Dagnone
- 1Department of Emergency Medicine, Queen's University, Kingston Health Sciences Center c/o 76 Stuart St, Kingston, ON K7L2V7 Canada
| |
Collapse
|
14
|
Affiliation(s)
| | - Pat Lilley
- a AMEE and Medical Teacher , Dundee , UK
| |
Collapse
|
15
|
Vallangeon BD, Hawley JS, Sloane R, Bean SM. An Assessment of Pathology Resident Access to and Use of Technology: A Nationwide Survey. Arch Pathol Lab Med 2017; 141:431-436. [DOI: 10.5858/arpa.2016-0228-oa] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
Context.—
Current technologies including digital slide scanners and handheld devices can revolutionize clinical practice and pathology graduate medical education (GME). The extent to which these technologies are used in pathology GME is unknown.
Objectives.—
To determine the types of technologies used, usage amount, and how they are integrated into pathology residency/fellowship programs nationwide.
Design.—
A 40-question online survey for residents/fellows was developed and administered via the Research Electronic Data Capture System after institutional review board approval.
Results.—
Fifty-two program directors (37%) gave permission for participation. One-hundred seventy-one responses were received (18% response rate). Most respondents have access to personal technology (laptop = 78% [134 of 171]), smartphone = 81% [139 of 171], tablet = 49% [84 of 171]), and Web-based digital slide collections (82%, 141 of 171). Few residents are provided electronic devices by their programs (laptop = 22% [38 of 171], smartphone = 0.5% [1 of 171], and tablet = 12% [21 of 171]). Fifty-nine percent have access to digital slide scanners, 33% have access to a program-created database of digitized slides, and 52% use telepathology. Fifteen percent have access to asynchronous learning. Of those with access to video-recorded conferences, 89% review them. Program size was significantly positively correlated with resident access to program-provided laptops (P = .02) and tablets (P < .001), digital slide scanners (P = .01), and telepathology (P = .001). Of all devices, program-provided laptops are used most for professional work (60.5% use this device for more than 5 hours per day).
Conclusions.—
Most residents report access to multiple types of innovative technology, but incorporation of these tools within pathology training programs is highly variable. Opportunities for incorporating innovative technologies exist and could be further explored.
Collapse
Affiliation(s)
| | | | | | - Sarah M. Bean
- From the Department of Pathology (Dr Vallangeon) and Duke Office of Clinical Research (Messrs Hawley and Sloane), Duke University Hospital, Durham, North Carolina; and the Department of Pathology, Duke University Medical Center, Durham, North Carolina (Dr Bean). Dr Vallangeon is now with the Department of Pathology, East Carolina University, Greenville, North Carolina
| |
Collapse
|
16
|
Cook DA, Hatala R. Validation of educational assessments: a primer for simulation and beyond. Adv Simul (Lond) 2016; 1:31. [PMID: 29450000 PMCID: PMC5806296 DOI: 10.1186/s41077-016-0033-y] [Citation(s) in RCA: 166] [Impact Index Per Article: 20.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/20/2016] [Accepted: 11/16/2016] [Indexed: 12/22/2022] Open
Abstract
BACKGROUND Simulation plays a vital role in health professions assessment. This review provides a primer on assessment validation for educators and education researchers. We focus on simulation-based assessment of health professionals, but the principles apply broadly to other assessment approaches and topics. KEY PRINCIPLES Validation refers to the process of collecting validity evidence to evaluate the appropriateness of the interpretations, uses, and decisions based on assessment results. Contemporary frameworks view validity as a hypothesis, and validity evidence is collected to support or refute the validity hypothesis (i.e., that the proposed interpretations and decisions are defensible). In validation, the educator or researcher defines the proposed interpretations and decisions, identifies and prioritizes the most questionable assumptions in making these interpretations and decisions (the "interpretation-use argument"), empirically tests those assumptions using existing or newly-collected evidence, and then summarizes the evidence as a coherent "validity argument." A framework proposed by Messick identifies potential evidence sources: content, response process, internal structure, relationships with other variables, and consequences. Another framework proposed by Kane identifies key inferences in generating useful interpretations: scoring, generalization, extrapolation, and implications/decision. We propose an eight-step approach to validation that applies to either framework: Define the construct and proposed interpretation, make explicit the intended decision(s), define the interpretation-use argument and prioritize needed validity evidence, identify candidate instruments and/or create/adapt a new instrument, appraise existing evidence and collect new evidence as needed, keep track of practical issues, formulate the validity argument, and make a judgment: does the evidence support the intended use? CONCLUSIONS Rigorous validation first prioritizes and then empirically evaluates key assumptions in the interpretation and use of assessment scores. Validation science would be improved by more explicit articulation and prioritization of the interpretation-use argument, greater use of formal validation frameworks, and more evidence informing the consequences and implications of assessment.
Collapse
Affiliation(s)
- David A. Cook
- Mayo Clinic Online Learning, Mayo Clinic College of Medicine, Rochester, MN USA
- Office of Applied Scholarship and Education Science, Mayo Clinic College of Medicine, Rochester, MN USA
- Division of General Internal Medicine, Mayo Clinic College of Medicine, Mayo 17-W, 200 First Street SW, Rochester, MN 55905 USA
| | - Rose Hatala
- Department of Medicine, University of British Columbia, Vancouver, British Columbia Canada
| |
Collapse
|
17
|
Burr S, Chatterjee A, Gibson S, Coombes L, Wilkinson S. Key Points to Facilitate the Adoption of Computer-Based Assessments. JOURNAL OF MEDICAL EDUCATION AND CURRICULAR DEVELOPMENT 2016; 3:JMECD.S20379. [PMID: 29349322 PMCID: PMC5736289 DOI: 10.4137/jmecd.s20379] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 01/12/2016] [Revised: 04/18/2016] [Accepted: 04/20/2016] [Indexed: 06/07/2023]
Abstract
There are strong pedagogical arguments in favor of adopting computer-based assessment. The risks of technical failure can be managed and are offset by improvements in cost-effectiveness and quality assurance capability. Academic, administrative, and technical leads at an appropriately senior level within an institution need to be identified, so that they can act as effective advocates. All stakeholder groups need to be represented in undertaking a detailed appraisal of requirements and shortlisting software based on core functionality, summative assessment life cycle needs, external compatibility, security, and usability. Any software that is a candidate for adoption should be trialed under simulated summative conditions, with all stakeholders having a voice in agreeing the optimum solution. Transfer to a new system should be carefully planned and communicated, with a programme of training established to maximize the success of adoption.
Collapse
Affiliation(s)
- S.A. Burr
- Collaboration for the Advancement of Medical Education Research and Assessment (CAMERA), Plymouth University Peninsula Schools of Medicine and Dentistry, Plymouth, UK
| | - A. Chatterjee
- Collaboration for the Advancement of Medical Education Research and Assessment (CAMERA), Plymouth University Peninsula Schools of Medicine and Dentistry, Plymouth, UK
| | - S. Gibson
- Norwich Medical School, University of East Anglia, Norwich, Norfolk, UK
| | - L. Coombes
- Cardiff University School of Medicine, Heath Park, Cardiff, UK
| | - S. Wilkinson
- University of Nottingham Medical School, Nottingham, UK
| |
Collapse
|
18
|
Hochlehnert A, Schultz JH, Möltner A, Tımbıl S, Brass K, Jünger J. Electronic acquisition of OSCE performance using tablets. GMS ZEITSCHRIFT FUR MEDIZINISCHE AUSBILDUNG 2015; 32:Doc41. [PMID: 26483854 PMCID: PMC4606489 DOI: 10.3205/zma000983] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 12/05/2013] [Revised: 10/16/2014] [Accepted: 03/02/2015] [Indexed: 11/30/2022]
Abstract
BACKGROUND Objective Structured Clinical Examinations (OSCEs) often involve a considerable amount of resources in terms of materials and organization since the scores are often recorded on paper. Computer-assisted administration is an alternative with which the need for material resources can be reduced. In particular, the use of tablets seems sensible because these are easy to transport and flexible to use. AIM User acceptance concerning the use of tablets during OSCEs has not yet been extensively investigated. The aim of this study was to evaluate tablet-based OSCEs from the perspective of the user (examiner) and the student examinee. METHOD For two OSCEs in Internal Medicine at the University of Heidelberg, user acceptance was analyzed regarding tablet-based administration (satisfaction with functionality) and the subjective amount of effort as perceived by the examiners. Standardized questionnaires and semi-standardized interviews were conducted (complete survey of all participating examiners). In addition, for one OSCE, the subjective evaluation of this mode of assessment was gathered from a random sample of participating students in semi-standardized interviews. RESULTS Overall, the examiners were very satisfied with using tablets during the assessment. The subjective amount of effort to use the tablet was found on average to be "hardly difficult". The examiners identified the advantages of this mode of administration as being in particular the ease of use and low rate of error. During the interviews of the examinees, acceptance for the use of tablets during the assessment was also detected. DISCUSSION Overall, it was found that the use of tablets during OSCEs was well accepted by both examiners and examinees. We expect that this mode of assessment also offers advantages regarding assessment documentation, use of resources, and rate of error in comparison with paper-based assessments; all of these aspects should be followed up on in further studies.
Collapse
Affiliation(s)
- Achim Hochlehnert
- Universität Heidelberg, Kompetenzzentrum für Prüfungen in der Medizin/Baden-Württemberg, Heidelberg, Deutschland
| | - Jobst-Hendrik Schultz
- Universität Heidelberg, Kompetenzzentrum für Prüfungen in der Medizin/Baden-Württemberg, Heidelberg, Deutschland
| | - Andreas Möltner
- Universität Heidelberg, Kompetenzzentrum für Prüfungen in der Medizin/Baden-Württemberg, Heidelberg, Deutschland
| | - Sevgi Tımbıl
- Universität Heidelberg, Kompetenzzentrum für Prüfungen in der Medizin/Baden-Württemberg, Heidelberg, Deutschland
| | - Konstantin Brass
- Universität Heidelberg, Kompetenzzentrum für Prüfungen in der Medizin/Baden-Württemberg, Heidelberg, Deutschland
| | - Jana Jünger
- Universität Heidelberg, Kompetenzzentrum für Prüfungen in der Medizin/Baden-Württemberg, Heidelberg, Deutschland
| |
Collapse
|
19
|
Queen's University Emergency Medicine Simulation OSCE: an Advance in Competency-Based Assessment. CAN J EMERG MED 2015; 18:230-3. [PMID: 25990201 DOI: 10.1017/cem.2015.34] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
|
20
|
McMaster F, Resnicow K. Validation of the one pass measure for motivational interviewing competence. PATIENT EDUCATION AND COUNSELING 2015; 98:499-505. [PMID: 25624118 DOI: 10.1016/j.pec.2014.12.014] [Citation(s) in RCA: 29] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/29/2014] [Revised: 12/22/2014] [Accepted: 12/28/2014] [Indexed: 05/16/2023]
Abstract
OBJECTIVE This paper examines the psychometric properties of the OnePass coding system: a new, user-friendly tool for evaluating practitioner competence in motivational interviewing (MI). We provide data on reliability and validity with the current gold-standard: Motivational Interviewing Treatment Integrity tool (MITI). METHODS We compared scores from 27 videotaped MI sessions performed by student counselors trained in MI and simulated patients using both OnePass and MITI, with three different raters for each tool. Reliability was estimated using intra-class coefficients (ICCs), and validity was assessed using Pearson's r. RESULTS OnePass had high levels of inter-rater reliability with 19/23 items found from substantial to almost perfect agreement. Taking the pair of scores with the highest inter-rater reliability on the MITI, the concurrent validity between the two measures ranged from moderate to high. Validity was highest for evocation, autonomy, direction and empathy. CONCLUSION OnePass appears to have good inter-rater reliability while capturing similar dimensions of MI as the MITI. Despite the moderate concurrent validity with the MITI, the OnePass shows promise in evaluating both traditional and novel interpretations of MI. PRACTICE IMPLICATIONS OnePass may be a useful tool for developing and improving practitioner competence in MI where access to MITI coders is limited.
Collapse
Affiliation(s)
- Fiona McMaster
- Faculty of Medical Science Anglia Ruskin University, Cambridge, UK.
| | - Ken Resnicow
- Department of Health Behavior & Health Education, School of Public Health, University of Michigan, Ann Arbor, USA
| |
Collapse
|
21
|
|
22
|
Harden RM. Have we got assessment wrong? Thoughts from the Ottawa Conference and the need for a Pumpkin Plan. MEDICAL TEACHER 2015; 37:209-210. [PMID: 25651987 DOI: 10.3109/0142159x.2015.1010497] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/04/2023]
|
23
|
Cook DA. The value of online learning and MRI: finding a niche for expensive technologies. MEDICAL TEACHER 2014; 36:965-72. [PMID: 25072533 DOI: 10.3109/0142159x.2014.917284] [Citation(s) in RCA: 15] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/13/2023]
Abstract
BACKGROUND The benefits of online learning come at a price. How can we optimize the overall value? AIMS Critically appraise the value of online learning. METHODS Narrative review. RESULTS Several prevalent myths overinflate the value of online learning. These include that online learning is cheap and easy (it is usually more expensive), that it is more efficient (efficiency depends on the instructional design, not the modality), that it will transform education (fundamental learning principles have not changed), and that the Net Generation expects it (there is no evidence of pent-up demand). However, online learning does add real value by enhancing flexibility, control and analytics. Costs may also go down if disruptive innovations (e.g. low-cost, low-tech, but instructionally sound "good enough" online learning) supplant technically superior but more expensive online learning products. Cost-lowering strategies include focusing on core principles of learning rather than technologies, using easy-to-learn authoring tools, repurposing content (organizing and sequencing existing resources rather than creating new content) and using course templates. CONCLUSIONS Online learning represents just one tool in an educator's toolbox, as does the MRI for clinicians. We need to use the right tool(s) for the right learner at the right dose, time and route.
Collapse
|
24
|
|
25
|
|
26
|
Chretien KC, Yarris LM, Lin M. Technology in graduate medical education: shifting the paradigm and advancing the field. J Grad Med Educ 2014; 6:195-6. [PMID: 24949118 PMCID: PMC4054712 DOI: 10.4300/jgme-d-14-00157.1] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/06/2022] Open
|
27
|
“Addiction Pharmacy” and the Professionalization Process: Technology-Enhanced Assessment of Reflective Practice and Teamwork. PHARMACY 2014. [DOI: 10.3390/pharmacy2020175] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022] Open
|
28
|
Barnato AE, Mohan D, Lane RK, Huang YM, Angus DC, Farris C, Arnold RM. Advance care planning norms may contribute to hospital variation in end-of-life ICU use: a simulation study. Med Decis Making 2014; 34:473-84. [PMID: 24615275 DOI: 10.1177/0272989x14522099] [Citation(s) in RCA: 40] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
BACKGROUND There is wide variation in end-of-life (EOL) intensive care unit (ICU) use among academic medical centers (AMCs). Our objective was to develop hypotheses regarding medical decision-making factors underlying this variation. METHODS This was a high-fidelity simulation experiment involving a critically and terminally ill elder, followed by a survey and debriefing cognitive interview and evaluated using triangulated quantitative-qualitative comparative analysis. The study was conducted in 2 AMCs in the same state and health care system with disparate EOL ICU use. Subjects were hospital-based physicians responsible for ICU admission decisions. Measurements included treatment plan, prognosis, diagnosis, qualitative case perceptions, and clinical reasoning. RESULTS Sixty-seven of 111 (60%) eligible physicians agreed to participate; 48 (72%) could be scheduled. There were no significant between-AMC differences in 3-month prognosis or treatment plan, but there were systematic differences in perceptions of the case. Case perceptions at the low-intensity AMC seemed to be influenced by the absence of a do-not-resuscitate order in the context of norms of universal code status discussion and documentation upon admission, whereas case perceptions at the high-intensity AMC seemed to be influenced by the patient's known metastatic gastric cancer in the context of norms of oncologists' avoiding code status discussions. CONCLUSIONS In this simulation study of 2 AMCs, hospital-based physicians had different perceptions of an identical case. We hypothesize that different advance care planning norms may have influenced their decision-making heuristics.
Collapse
Affiliation(s)
- Amber E Barnato
- Center for Research on Health Care, University of Pittsburgh, Pittsburgh, PA (AEB, RMA).,Division of General Internal Medicine, Department of Medicine, University of Pittsburgh, Pittsburgh, PA (AEB, RMA),Department of Health Policy Management, Graduate School of Public Health, University of Pittsburgh, Pittsburgh,
PA (AEB),The CRISMA (Clinical Research, Investigation, and Systems Modeling of Acute Illness) Laboratory, Department of Critical Care Medicine, University of Pittsburgh, Pittsburgh, PA (AEB, DM, DCA)
| | - Deepika Mohan
- The CRISMA (Clinical Research, Investigation, and Systems Modeling of Acute Illness) Laboratory, Department of Critical Care Medicine, University of Pittsburgh, Pittsburgh, PA (AEB, DM, DCA)
| | - Rondall K Lane
- Department of Anesthesia and Perioperative Care, University of California–San Francisco Medical Center, San Francisco, CA (RKL)
| | - Yue Ming Huang
- Department of Anesthesiology, David Geffen School of Medicine at University of California, Los Angeles, CA (YMH, DCA)
| | - Derek C Angus
- Department of Anesthesiology, David Geffen School of Medicine at University of California, Los Angeles, CA (YMH, DCA)
| | | | - Robert M Arnold
- Center for Research on Health Care, University of Pittsburgh, Pittsburgh, PA (AEB, RMA).,Division of General Internal Medicine, Department of Medicine, University of Pittsburgh, Pittsburgh, PA (AEB, RMA),Section of Palliative Care and Medical Ethics, Division of General Internal Medicine, Department of Medicine, University of Pittsburgh, Pittsburgh, PA (RMA)
| |
Collapse
|
29
|
Hamstra SJ, Brydges R, Hatala R, Zendejas B, Cook DA. Reconsidering fidelity in simulation-based training. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2014; 89:387-92. [PMID: 24448038 DOI: 10.1097/acm.0000000000000130] [Citation(s) in RCA: 279] [Impact Index Per Article: 27.9] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/07/2023]
Abstract
In simulation-based health professions education, the concept of simulator fidelity is usually understood as the degree to which a simulator looks, feels, and acts like a human patient. Although this can be a useful guide in designing simulators, this definition emphasizes technological advances and physical resemblance over principles of educational effectiveness. In fact, several empirical studies have shown that the degree of fidelity appears to be independent of educational effectiveness. The authors confronted these issues while conducting a recent systematic review of simulation-based health professions education, and in this Perspective they use their experience in conducting that review to examine key concepts and assumptions surrounding the topic of fidelity in simulation.Several concepts typically associated with fidelity are more useful in explaining educational effectiveness, such as transfer of learning, learner engagement, and suspension of disbelief. Given that these concepts more directly influence properties of the learning experience, the authors make the following recommendations: (1) abandon the term fidelity in simulation-based health professions education and replace it with terms reflecting the underlying primary concepts of physical resemblance and functional task alignment; (2) make a shift away from the current emphasis on physical resemblance to a focus on functional correspondence between the simulator and the applied context; and (3) focus on methods to enhance educational effectiveness using principles of transfer of learning, learner engagement, and suspension of disbelief. These recommendations clarify underlying concepts for researchers in simulation-based health professions education and will help advance this burgeoning field.
Collapse
Affiliation(s)
- Stanley J Hamstra
- Dr. Hamstra is professor, Departments of Medicine, Anesthesia and Surgery; research director, University of Ottawa Skills and Simulation Centre; and director, Academy for Innovation in Medical Education, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada. Dr. Brydges is assistant professor, Department of Medicine, University of Toronto, Toronto, Ontario, Canada. Dr. Hatala is associate professor, Department of Medicine, University of British Columbia, Vancouver, British Columbia, Canada. Dr. Zendejas is resident, Department of Surgery, Mayo Clinic College of Medicine, Rochester, Minnesota. Dr. Cook is professor of medicine and medical education, Department of Medicine, Mayo Clinic College of Medicine, and director, Office of Education Research, Mayo Medical School, Rochester, Minnesota
| | | | | | | | | |
Collapse
|
30
|
Fullerton JT, Thompson JB, Johnson P. Competency-based education: The essential basis of pre-service education for the professional midwifery workforce. Midwifery 2013; 29:1129-36. [DOI: 10.1016/j.midw.2013.07.006] [Citation(s) in RCA: 58] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2013] [Revised: 06/29/2013] [Accepted: 07/01/2013] [Indexed: 11/29/2022]
|
31
|
Hochlehnert A, Brass K, Möltner A, Schultz JH, Norcini J, Tekian A, Jünger J. Good exams made easy: the item management system for multiple examination formats. BMC MEDICAL EDUCATION 2012; 12:63. [PMID: 22857655 PMCID: PMC3441576 DOI: 10.1186/1472-6920-12-63] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/24/2012] [Accepted: 07/26/2012] [Indexed: 05/30/2023]
Abstract
BACKGROUND The development, implementation and evaluation of assessments require considerable resources and often cannot be carried out by a single faculty/institution. Therefore some medical faculties have founded cooperation projects which mainly focus on the exchange of multiple choice questions (MCQs). METHODS Since these cooperation projects do not entirely support all relevant processes in terms of preparation, implementation and evaluation of assessment, in 2006 the Medical Assessment Alliance (MAA) was founded for mutual support. In addition to MCQs the MAA started to develop innovative assessment formats and facilitate content through a coordinated exchange of experiences. To support cooperation within this network, the web-based Item Management System (IMS) was developed which supports all processes of the assessment workflow as an all-in-one working platform. RESULTS At present, the Alliance has 28 partner faculties in Europe. More than 2.800 users in 750 working groups are collaborating. Currently 90.000 questions have been stored in the IMS. Since 2007, nearly 4.600 examinations have been successfully conducted. CONCLUSION This article describes in detail the unique features of the IMS and contrasts it with the item management systems of other associations.
Collapse
Affiliation(s)
- Achim Hochlehnert
- Department of Psychosomatic and General Internal Medicine, University of Heidelberg, Medical Clinic, Heidelberg, Germany
| | - Konstantin Brass
- Centre of Excellence for Medical Assessment, University of Heidelberg, Medical School, Heidelberg, Germany
| | - Andreas Möltner
- Centre of Excellence for Medical Assessment, University of Heidelberg, Medical School, Heidelberg, Germany
| | - Jobst-Hendrik Schultz
- Centre of Excellence for Medical Assessment, University of Heidelberg, Medical School, Heidelberg, Germany
| | - John Norcini
- Foundation for Advancement of International Medical Education and Research (FAIMER), Philadelphia, PA, 19104, USA
| | - Ara Tekian
- College of Medicine at Chicago, University of Illinois, Chicago, IL, 60612, USA
| | - Jana Jünger
- Department of Psychosomatic and General Internal Medicine, University of Heidelberg, Medical Clinic, Heidelberg, Germany
- Centre of Excellence for Medical Assessment, University of Heidelberg, Medical School, Heidelberg, Germany
| |
Collapse
|
32
|
Feldman M, Lazzara EH, Vanderbilt AA, DiazGranados D. Rater training to support high-stakes simulation-based assessments. THE JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS 2012; 32:279-86. [PMID: 23280532 PMCID: PMC3646087 DOI: 10.1002/chp.21156] [Citation(s) in RCA: 89] [Impact Index Per Article: 7.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/21/2023]
Abstract
Competency-based assessment and an emphasis on obtaining higher-level outcomes that reflect physicians' ability to demonstrate their skills has created a need for more advanced assessment practices. Simulation-based assessments provide medical education planners with tools to better evaluate the 6 Accreditation Council for Graduate Medical Education (ACGME) and American Board of Medical Specialties (ABMS) core competencies by affording physicians opportunities to demonstrate their skills within a standardized and replicable testing environment, thus filling a gap in the current state of assessment for regulating the practice of medicine. Observational performance assessments derived from simulated clinical tasks and scenarios enable stronger inferences about the skill level a physician may possess, but also introduce the potential of rater errors into the assessment process. This article reviews the use of simulation-based assessments for certification, credentialing, initial licensure, and relicensing decisions and describes rater training strategies that may be used to reduce rater errors, increase rating accuracy, and enhance the validity of simulation-based observational performance assessments.
Collapse
Affiliation(s)
- Moshe Feldman
- School of Medicine, Virginia Commonwealth University, Richmond, VA, 23298, USA.
| | | | | | | |
Collapse
|
33
|
Review article: Assessment in anesthesiology education. Can J Anaesth 2011; 59:182-92. [DOI: 10.1007/s12630-011-9637-9] [Citation(s) in RCA: 45] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/11/2011] [Accepted: 11/16/2011] [Indexed: 11/27/2022] Open
|