1
|
Ross L, Semaan E, Gosling CM, Fisk B, Shannon B. Clinical reasoning in undergraduate paramedicine: utilisation of a script concordance test. BMC MEDICAL EDUCATION 2023; 23:39. [PMID: 36658560 PMCID: PMC9849838 DOI: 10.1186/s12909-023-04020-x] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/30/2022] [Accepted: 01/11/2023] [Indexed: 06/17/2023]
Abstract
INTRODUCTION Clinical reasoning is a complex cognitive and metacognitive process paramount to patient care in paramedic practice. While universally recognised as an essential component of practice, clinical reasoning has been historically difficult to assess in health care professions. Is the Script Concordance Test (SCT) an achievable and reliable option to test clinical reasoning in undergraduate paramedic students? METHODS This was a single institution observational cohort study designed to use the SCT to measure clinical reasoning in paramedic students. Clinical vignettes were constructed across a range of concepts with varying shades of clinical ambiguity. A reference panel mean scores of the test were compared to that of students. Test responses were graded with the aggregate scoring method with scores awarded for both partially and fully correct responses. RESULTS Eighty-three student paramedic participants (mean age: 21.8 (3.5) years, 54 (65%) female, 27 (33%) male and 2 (2%) non-binary) completed the SCT. The difference between the reference group mean score of 80 (5) and student mean of score of 65.6 (8.4) was statistically significant (p < 0.001). DISCUSSION Clinical reasoning skills are not easily acquired as they are a culmination of education, experience and the ability to apply this in the context to a specific patient. The SCT has shown to be reliable and effective in measuring clinical reasoning in undergraduate paramedics as it has in other health professions such as nursing and medicine. More investigation is required to establish effective pedogeological techniques to optimise clinical reasoning in student and novice paramedics who are devoid of experience.
Collapse
Affiliation(s)
- Linda Ross
- Department of Paramedicine, School of Primary and Allied Health Care, Faculty of Medicine, Nursing and Health Science, Monash University, PO Box 527, Peninsula Campus, McMahons Road, Frankston, Melbourne, Victoria, 3199, Australia.
| | - Eli Semaan
- Ambulance Victoria, Melbourne, Australia
| | - Cameron M Gosling
- Department of Paramedicine, School of Primary and Allied Health Care, Faculty of Medicine, Nursing and Health Science, Monash University, PO Box 527, Peninsula Campus, McMahons Road, Frankston, Melbourne, Victoria, 3199, Australia
| | - Benjamin Fisk
- Department of Paramedicine, School of Primary and Allied Health Care, Faculty of Medicine, Nursing and Health Science, Monash University, PO Box 527, Peninsula Campus, McMahons Road, Frankston, Melbourne, Victoria, 3199, Australia
- Ambulance Victoria, Melbourne, Australia
| | - Brendan Shannon
- Department of Paramedicine, School of Primary and Allied Health Care, Faculty of Medicine, Nursing and Health Science, Monash University, PO Box 527, Peninsula Campus, McMahons Road, Frankston, Melbourne, Victoria, 3199, Australia
- Ambulance Victoria, Melbourne, Australia
| |
Collapse
|
2
|
Iglesias Gómez C, González Sequeros O, Salmerón Martínez D. Evaluación mediante script concordance test del razonamiento clínico de residentes en Atención Primaria. An Pediatr (Barc) 2022. [DOI: 10.1016/j.anpedi.2021.09.009] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022] Open
|
3
|
Iglesias Gómez C, González Sequeros O, Salmerón Martínez D. Clinical reasoning evaluation using script concordance test in primary care residents. ANALES DE PEDIATRÍA (ENGLISH EDITION) 2022; 97:87-94. [DOI: 10.1016/j.anpede.2022.06.005] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/01/2021] [Accepted: 09/30/2021] [Indexed: 11/28/2022] Open
|
4
|
|
5
|
Mutter MK, Martindale JR, Shah N, Gusic ME, Wolf SJ. Case-Based Teaching: Does the Addition of High-Fidelity Simulation Make a Difference in Medical Students' Clinical Reasoning Skills? MEDICAL SCIENCE EDUCATOR 2020; 30:307-313. [PMID: 34457672 PMCID: PMC8368304 DOI: 10.1007/s40670-019-00904-0] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/02/2023]
Abstract
CONTEXT Situativity theory posits that learning and the development of clinical reasoning skills are grounded in context. In case-based teaching, this context comes from recreating the clinical environment, through emulation, as with manikins, or description. In this study, we sought to understand the difference in student clinical reasoning abilities after facilitated patient case scenarios with or without a manikin. METHODS Fourth-year medical students in an internship readiness course were randomized into patient case scenarios without manikin (control group) and with manikin (intervention group) for a chest pain session. The control and intervention groups had identical student-led case progression and faculty debriefing objectives. Clinical reasoning skills were assessed after the session using a 64-question script concordance test (SCT). The test was developed and piloted prior to administration. Hospitalist and emergency medicine faculty responses on the test items served as the expert standard for scoring. RESULTS Ninety-six students were randomized to case-based sessions with (n = 48) or without (n = 48) manikin. Ninety students completed the SCT (with manikin n = 45, without manikin n = 45). A statistically significant mean difference on test performance between the two groups was found (t = 3.059, df = 88, p = .003), with the manikin group achieving higher SCT scores. CONCLUSION Use of a manikin in simulated patient case discussion significantly improves students' clinical reasoning skills, as measured by SCT. These results suggest that using a manikin to simulate a patient scenario situates learning, thereby enhancing skill development.
Collapse
Affiliation(s)
- Mary Kathryn Mutter
- Department of Emergency Medicine, University of Virginia School of Medicine, PO Box 800699, Charlottesville, VA 22908-0699 USA
| | - James R. Martindale
- University of Virginia School of Medicine, Box 800866 Suite 2008 Pinn Hall, Charlottesville, VA 22908-0866 USA
| | - Neeral Shah
- Division of Gastroenterology and Hepatology, University of Virginia School of Medicine, Box 800708, GI West Complex - 2nd Floor, Charlottesville, VA 22908 USA
| | - Maryellen E. Gusic
- University of Virginia School of Medicine, Box 800866 Suite 2008 Pinn Hall, Charlottesville, VA 22908-0866 USA
| | - Stephen J. Wolf
- Department of Emergency Medicine, Denver Health Medical Center, University of Colorado School of Medicine, Denver Health Medical Center, 777 Bannock St., Pavilion A, Denver, CO 80204 USA
| |
Collapse
|
6
|
Bordage G, Page G. The key-features approach to assess clinical decisions: validity evidence to date. ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2018; 23:1005-1036. [PMID: 29777464 DOI: 10.1007/s10459-018-9830-5] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/04/2017] [Accepted: 05/07/2018] [Indexed: 06/08/2023]
Abstract
The key-features (KFs) approach to assessment was initially proposed during the First Cambridge Conference on Medical Education in 1984 as a more efficient and effective means of assessing clinical decision-making skills. Over three decades later, we conducted a comprehensive, systematic review of the validity evidence gathered since then. The evidence was compiled according to the Standards for Educational and Psychological Testing's five sources of validity evidence, namely, Content, Response process, Internal structure, Relations to other variables, and Consequences, to which we added two other types related to Cost-feasibility and Acceptability. Of the 457 publications that referred to the KFs approach between 1984 and October 2017, 164 are cited here; the remaining 293 were either redundant or the authors simply mentioned the KFs concept in relation to their work. While one set of articles reported meeting the validity standards, another set examined KFs test development choices and score interpretation. The accumulated validity evidence for the KFs approach since its inception supports the decision-making construct measured and its use to assess clinical decision-making skills at all levels of training and practice and with various types of exam formats. Recognizing that gathering validity evidence is an ongoing process, areas with limited evidence, such as item factor analyses or consequences of testing, are identified as well as new topics needing further clarification, such as the use of the KFs approach for formative assessment and its place within a program of assessment.
Collapse
Affiliation(s)
- G Bordage
- Department of Medical Education, College of Medicine, University of Illinois at Chicago, Chicago, USA.
| | - G Page
- Department of Medicine, Faculty of Medicine, University of British Columbia, Vancouver, Canada
| |
Collapse
|
7
|
Bourgeois-Law G, Teunissen PW, Regehr G. Remediation in Practicing Physicians: Current and Alternative Conceptualizations. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2018; 93:1638-1644. [PMID: 29697432 DOI: 10.1097/acm.0000000000002266] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/08/2023]
Abstract
Suboptimal performance in practicing physicians is a decades-old problem. The lack of a universally accepted definition of remediation, the paucity of research on best remediation practices, and the ongoing controversy regarding the institutional responsibility for enacting and overseeing the remediation of physicians suggest that it is not merely a difficult problem to solve but a problem that the community does not grapple with meaningfully. Undoubtedly, logistical and political considerations contribute to this state of affairs; however, other underlying conceptual issues may also play a role in the medical profession's difficulties in engaging with the challenges around remediation.Through a review of the medical education and other literatures, the authors examined current conceptualizations of both reme-diation itself and the individual being remediated, as well as how the culture of medicine influences these conceptions. The authors explored how conceptualizations of remediation and the surrounding culture might affect not only the medical community's ability to support but also its willingness to engage with physicians in need of remediation.Viewing remediation as a means of supporting practice change-rather than as a means of redressing gaps in knowledge and skill-might be a useful alternative conceptualization, providing a good place to start exploring new avenues of research. However, moving forward will require more than simply a reconceptualization of remediation; it will also necessitate a change in how the community views its struggling members and a change in the medical culture that currently positions professional autonomy as the foundational premise for individual practice improvement.
Collapse
Affiliation(s)
- Gisèle Bourgeois-Law
- G. Bourgeois-Law is clinical associate professor, Department of Obstetrics and Gynecology, Faculty of Medicine, University of British Columbia, Vancouver, British Columbia, Canada, and PhD candidate, School of Health Professions Education, Maastricht University, Maastricht, the Netherlands; ORCID: https://orcid.org/0000-0002-4726-3663. P.W. Teunissen is professor of medical education, School of Health Professions Education, Faculty of Health Medicine and Life Sciences, Maastricht University, Maastricht, the Netherlands, and gynecologist, Department of Obstetrics and Gynecology, Vrije Universiteit Medical Center, Amsterdam, the Netherlands; ORCID: http://orcid.org/0000-0002-0930-0048. G. Regehr is professor, Department of Surgery, and associate director for research, Centre for Health Education Scholarship, Faculty of Medicine, University of British Columbia, Vancouver, British Columbia, Canada; ORCID: https://orcid.org/0000-0002-3144-331X
| | | | | |
Collapse
|
8
|
Le test de concordance de script : un outil pédagogique multimodal. Rev Med Interne 2018; 39:566-573. [DOI: 10.1016/j.revmed.2017.12.011] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/13/2017] [Revised: 12/17/2017] [Accepted: 12/28/2017] [Indexed: 11/18/2022]
|
9
|
Abstract
OBJECTIVES Script concordance testing (SCT) is used to assess clinical decision-making. We explore the use of SCT to (1) quantify practice variations in infant lumbar puncture (LP) and (2) analyze physician's characteristics affecting LP decision making. METHODS Using standard SCT processes, a panel of pediatric subspecialty physicians constructed 15 infant LP case vignettes, each with 2 to 4 SCT questions (a total of 47). The vignettes were distributed to pediatric attending physicians and fellows at 10 hospitals within the INSPIRE Network. We determined both raw scores (tendency to perform LP) and SCT scores (agreement with the reference panel) as well as the variation with participant factors. RESULTS Two hundred twenty-six respondents completed all 47 SCT questions. Pediatric emergency medicine physicians tended to select LP more frequently than did general pediatricians, with pediatric emergency medicine physicians showing significantly higher raw scores (20.2 ± 10.2) than general pediatricians (13 ± 15; 95% confidence interval for difference, 1, 13). Concordance with the reference panel varied among subspecialties and by the frequency with which practitioners perform LPs in their practices. CONCLUSION Script concordance testing questions can be used as a tool to detect subspecialty practice variation. We are able to detect significant practice variation in the self-report of use of LP for infants among different pediatric subspecialties.
Collapse
|
10
|
Griffits S, Hines S, Moloney C, Ralph N. Characteristics and processes of clinical reasoning in nurses and factors related to its use: a scoping review protocol. ACTA ACUST UNITED AC 2017; 15:2832-2836. [DOI: 10.11124/jbisrir-2016-003273] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/31/2022]
|
11
|
Rose KA, Babajanian J. The interrater reliability of an objective structured practical examination in measuring the clinical reasoning ability of chiropractic students. THE JOURNAL OF CHIROPRACTIC EDUCATION 2016; 30:99-103. [PMID: 27115474 PMCID: PMC5067129 DOI: 10.7899/jce-15-16] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/05/2023]
Abstract
OBJECTIVE The objective structured practical examination (OSPE) is a case-based assessment that can be used to assess the clinical reasoning ability of students. The reliability of using an OSPE for this purpose has not been reported in the literature. The objective of this study was to determine the interrater reliability of the OSPE in measuring the clinical reasoning ability of chiropractic students. METHODS Two examiners tested each student simultaneously when enough were available as a check for interrater reliability. The scores for students over 4 exam administrations were compiled, and we calculated an intraclass correlation coefficient (ICC) using 1-way random single measures. RESULTS Paired scores were available for 133 students. The ICC was .685, showing a fair-to-good level of agreement for faculty in assessing the clinical reasoning ability of chiropractic students using an OSPE. CONCLUSION The OSPE can be a valuable tool for testing clinical reasoning abilities because it can simulate the decision-making process that needs to be implemented in clinical practice. Faculty members at our chiropractic college were able to achieve an acceptable level of reliability in measuring the clinical reasoning abilities of students using an OSPE. Other health professional programs may consider using this tool for assessing the clinical reasoning skills of their students.
Collapse
|
12
|
Sibert L. Test de concordance de script et développement professionnel continu (DPC) : un mariage qui ne peut être que réussi ! Rev Mal Respir 2016; 33:329-31. [DOI: 10.1016/j.rmr.2016.05.005] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022]
|
13
|
Faucher C, Dufour-Guindon MP, Lapointe G, Gagnon R, Charlin B. Assessing clinical reasoning in optometry using the script concordance test. Clin Exp Optom 2016; 99:280-6. [PMID: 27087346 DOI: 10.1111/cxo.12354] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/24/2015] [Revised: 06/18/2015] [Accepted: 08/04/2015] [Indexed: 11/29/2022] Open
Abstract
BACKGROUND Clinical reasoning is central to any health profession but its development among learners is difficult to assess. Over the last few decades, the script concordance test (SCT) has been developed to solve this dilemma and has been used in many health professions; however, no study has been published on the use of the script concordance test in optometry. The purpose of this study was to develop and validate a script concordance test for the field of optometry. METHODS A 101-question script concordance test (27 short clinical scenarios) was developed and administered online to a convenience sample of 23 second-year and 19 fourth-year students of optometry. It was also administered to a reference panel of 12 experienced optometrists to develop the scoring key. An item-total correlation was calculated for each question. Cronbach's alpha coefficient was used to evaluate the script concordance test reliability and a t-test compared the two groups. RESULTS A final 77-question script concordance test was created by eliminating questions with low item-total correlation. Cronbach's alpha for this optimised 77-question script concordance test was 0.80. A group comparison revealed that the second-year students' scores (n = 23; mean score = 66.4 ± 7.87 per cent) were statistically lower (t = -4.141; p < 0.001) than those of the fourth-year students (n = 19; mean score = 75.5 ± 5.97 per cent). CONCLUSION The online script concordance test developed for this study was found to be both reliable and capable of discriminating between second- and fourth-year optometric students. These results demonstrate that the script concordance test may be considered as a new tool in the optometric educators' assessment arsenal. Further studies will be needed to cover additional levels of professional development.
Collapse
Affiliation(s)
| | | | | | - Robert Gagnon
- Centre de pédagogie appliquée aux sciences de la santé, Faculté de médecine, Université de Montréal, Montréal, Canada
| | - Bernard Charlin
- Centre de pédagogie appliquée aux sciences de la santé, Faculté de médecine, Université de Montréal, Montréal, Canada
| |
Collapse
|
14
|
Cooke S, Lemay JF, Beran T, Sandhu A, Amin H. Development of a Method to Measure Clinical Reasoning in Pediatric Residents: The Pediatric Script Concordance Test. ACTA ACUST UNITED AC 2016. [DOI: 10.4236/ce.2016.76084] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022]
|
15
|
Durning SJ, Lubarsky S, Torre D, Dory V, Holmboe E. Considering "Nonlinearity" Across the Continuum in Medical Education Assessment: Supporting Theory, Practice, and Future Research Directions. THE JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS 2015; 35:232-243. [PMID: 26378429 DOI: 10.1002/chp.21298] [Citation(s) in RCA: 18] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/05/2023]
Abstract
The purpose of this article is to propose new approaches to assessment that are grounded in educational theory and the concept of "nonlinearity." The new approaches take into account related phenomena such as "uncertainty," "ambiguity," and "chaos." To illustrate these approaches, we will use the example of assessment of clinical reasoning, although the principles we outline may apply equally well to assessment of other constructs in medical education. Theoretical perspectives include a discussion of script theory, assimilation theory, self-regulated learning theory, and situated cognition. Assessment examples to include script concordance testing, concept maps, self-regulated learning microanalytic technique, and work-based assessment, which parallel the above-stated theories, respectively, are also highlighted. We conclude with some practical suggestions for approaching nonlinearity.
Collapse
|
16
|
Hornos EH, Pleguezuelos EM, Brailovsky CA, Harillo LD, Dory V, Charlin B. The practicum script concordance test: an online continuing professional development format to foster reflection on clinical practice. THE JOURNAL OF CONTINUING EDUCATION IN THE HEALTH PROFESSIONS 2013; 33:59-66. [PMID: 23512561 DOI: 10.1002/chp.21166] [Citation(s) in RCA: 15] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/01/2023]
Abstract
INTRODUCTION Judgment in the face of uncertainty is an important dimension of expertise and clinical competence. However, it is challenging to conceive continuing professional development (CPD) initiatives aimed at helping physicians enhance their clinical judgment skills in ill-defined situations. We present an online script concordance-based CPD program (the Practicum Script Concordance Test, copyright ©2006 by Practicum Foundation), a tool that can be used to support health professionals in the development of their reflective clinical reasoning ability. We describe the rationale and principles and report on the implementation of 2 online programs based on this new CPD initiative. METHOD The Practicum Script Concordance Test program consists of daily testing and feedback over the course of a year using SCT items. Feedback is both global (eg, health professionals are told their cumulative mean score) and specific (eg, they can view the expert panel's responses together with their justifications for their answers). Participants have the option of contacting a personal tutor, to whom they can send questions. Data regarding feasibility, participation, and acceptability were collected. RESULTS Initial implementation took place in Mexico where 1901 physicians (1349 paediatricians, 552 cardiologists) were enrolled in Practicum programs. Around 70% of those enrolled pursued the program and were very satisfied with its format and content. The online format was an important factor in the development and maintenance of the programs. Dropouts had issues with the SCT concept and the time required to participate. DISCUSSION The on-line Practicum Script Concordance Test program was designed to foster expertise development based on practice, reflection and feedback. Although further research is needed to examine its impact physicians' practice and ultimately on patient outcomes, it is an original and promising development in CPD.
Collapse
Affiliation(s)
- Eduardo H Hornos
- Practicum Institute of Applied Research in Health Sciences Education, Canada
| | | | | | | | | | | |
Collapse
|
17
|
Lubarsky S, Dory V, Duggan P, Gagnon R, Charlin B. Script concordance testing: from theory to practice: AMEE guide no. 75. MEDICAL TEACHER 2013; 35:184-93. [PMID: 23360487 DOI: 10.3109/0142159x.2013.760036] [Citation(s) in RCA: 94] [Impact Index Per Article: 8.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/18/2023]
Abstract
The script concordance test (SCT) is used in health professions education to assess a specific facet of clinical reasoning competence: the ability to interpret medical information under conditions of uncertainty. Grounded in established theoretical models of knowledge organization and clinical reasoning, the SCT has three key design features: (1) respondents are faced with ill-defined clinical situations and must choose between several realistic options; (2) the response format reflects the way information is processed in challenging problem-solving situations; and (3) scoring takes into account the variability of responses of experts to clinical situations. SCT scores are meant to reflect how closely respondents' ability to interpret clinical data compares with that of experienced clinicians in a given knowledge domain. A substantial body of research supports the SCT's construct validity, reliability, and feasibility across a variety of health science disciplines, and across the spectrum of health professions education from pre-clinical training to continuing professional development. In practice, its performance as an assessment tool depends on careful item development and diligent panel selection. This guide, intended as a primer for the uninitiated in SCT, will cover the basic tenets, theoretical underpinnings, and construction principles governing script concordance testing.
Collapse
Affiliation(s)
- Stuart Lubarsky
- McGill Centre for Medical Education, McGill University, Canada.
| | | | | | | | | |
Collapse
|
18
|
Ilgen JS, Humbert AJ, Kuhn G, Hansen ML, Norman GR, Eva KW, Charlin B, Sherbino J. Assessing diagnostic reasoning: a consensus statement summarizing theory, practice, and future needs. Acad Emerg Med 2012; 19:1454-61. [PMID: 23279251 DOI: 10.1111/acem.12034] [Citation(s) in RCA: 49] [Impact Index Per Article: 4.1] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/26/2012] [Accepted: 06/27/2012] [Indexed: 01/22/2023]
Abstract
Assessment of an emergency physician (EP)'s diagnostic reasoning skills is essential for effective training and patient safety. This article summarizes the findings of the diagnostic reasoning assessment track of the 2012 Academic Emergency Medicine consensus conference "Education Research in Emergency Medicine: Opportunities, Challenges, and Strategies for Success." Existing theories of diagnostic reasoning, as they relate to emergency medicine (EM), are outlined. Existing strategies for the assessment of diagnostic reasoning are described. Based on a review of the literature, expert thematic analysis, and iterative consensus agreement during the conference, this article summarizes current assessment gaps and prioritizes future research questions concerning the assessment of diagnostic reasoning in EM.
Collapse
Affiliation(s)
- Jonathan S. Ilgen
- Division of Emergency Medicine; University of Washington School of Medicine; Seattle; WA
| | - Aloysius J. Humbert
- Department of Emergency Medicine; Indiana University School of Medicine; Indianapolis; IN
| | - Gloria Kuhn
- Department of Emergency Medicine; Wayne State University; Detroit; MI
| | - Matthew L. Hansen
- Department of Emergency Medicine; Oregon Health & Science University; Portland; OR
| | - Geoffrey R. Norman
- Department of Clinical Epidemiology and Biostatistics; McMaster University; Hamilton; Ontario; Canada
| | - Kevin W. Eva
- Centre for Health Education Scholarship; University of British Columbia; Vancouver; British Columbia; Canada
| | - Bernard Charlin
- Center of Pedagogy Applied to Health Sciences; University of Montreal; Montreal; Quebec; Canada
| | - Jonathan Sherbino
- Division of Emergency Medicine; Department of Medicine; McMaster University; Hamilton; Ontario; Canada
| |
Collapse
|
19
|
Goyal N, Aldeen A, Leone K, Ilgen JS, Branzetti J, Kessler C. Assessing medical knowledge of emergency medicine residents. Acad Emerg Med 2012; 19:1360-5. [PMID: 23252401 DOI: 10.1111/acem.12033] [Citation(s) in RCA: 8] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/26/2012] [Accepted: 06/28/2012] [Indexed: 11/30/2022]
Abstract
The Accreditation Council for Graduate Medical Education (ACGME) requires that emergency medicine (EM) residency graduates are competent in the medical knowledge (MK) core competency. EM educators use a number of tools to measure a resident's progress toward this goal; it is not always clear whether these tools provide a valid assessment. A workshop was convened during the 2012 Academic Emergency Medicine consensus conference "Education Research in Emergency Medicine: Opportunities, Challenges, and Strategies for Success" where assessment for each core competency was discussed in detail. This article provides a description of the validity evidence behind current MK assessment tools used in EM and other specialties. Tools in widespread use are discussed, as well as emerging methods that may form valid assessments in the future. Finally, an agenda for future research is proposed to help address gaps in the current understanding of MK assessment.
Collapse
Affiliation(s)
- Nikhil Goyal
- Department of Emergency Medicine; Henry Ford Hospital; Detroit; MI
| | - Amer Aldeen
- Department of Emergency Medicine; Northwestern University Feinberg School of Medicine; Chicago; IL
| | - Katrina Leone
- Department of Emergency Medicine; Oregon Health & Science University; Portland; OR
| | - Jonathan S. Ilgen
- Department of Emergency Medicine; University of Washington; Seattle; WA
| | - Jeremy Branzetti
- Department of Emergency Medicine; University of Washington; Seattle; WA
| | - Chad Kessler
- Department of Emergency Medicine; University of Illinois-Chicago; Chicago; IL
| |
Collapse
|
20
|
van Bruggen L, Manrique-van Woudenbergh M, Spierenburg E, Vos J. Preferred question types for computer-based assessment of clinical reasoning: a literature study. PERSPECTIVES ON MEDICAL EDUCATION 2012; 1:162-171. [PMID: 23205341 PMCID: PMC3508269 DOI: 10.1007/s40037-012-0024-1] [Citation(s) in RCA: 10] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/11/2023]
Abstract
Clinical reasoning is a core competence of doctors. Therefore, the assessment of clinical reasoning of undergraduate students is an important part of medical education. Three medical universities in the Netherlands wish to develop a shared question database in order to assess clinical reasoning of undergraduate students in Computer-Based Assessments (CBA). To determine suitable question types for this purpose a literature study was carried out. Search of ERIC and PubMed and subsequent cross referencing yielded 30 articles which met the inclusion criteria of a focus on question types suitable to assess clinical reasoning of medical students and providing recommendations for their use. Script Concordance Tests, Extended Matching Questions, Comprehensive Integrative Puzzles, Modified Essay Questions/Short Answer Questions, Long Menu Questions, Multiple Choice Questions, Multiple True/False Questions and Virtual Patients meet the above-mentioned criteria, but for different reasons not all types can be used easily in CBA. A combination of Comprehensive Integrative Puzzles and Extended Matching Questions seems to assess most aspects of clinical reasoning and these question types can be adapted for use in CBA. Regardless of the question type chosen, patient vignettes should be used as a standard stimulus format to assess clinical reasoning. Further research is necessary to ensure that the combination of these question types produces valid assessments and reliable test results.
Collapse
Affiliation(s)
- Lisette van Bruggen
- Center for Research and Development of Education, University Medical Center Utrecht, P.O. Box 85500, 3508 GA, Utrecht, the Netherlands.
| | - Margreet Manrique-van Woudenbergh
- Center for Research and Development of Education, University Medical Center Utrecht, P.O. Box 85500, 3508 GA, Utrecht, the Netherlands
| | | | | |
Collapse
|
21
|
van den Broek WES, van Asperen MV, Custers E, Valk GD, ten Cate OTJ. Effects of two different instructional formats on scores and reliability of a script concordance test. PERSPECTIVES ON MEDICAL EDUCATION 2012; 1:119-128. [PMID: 23316468 PMCID: PMC3540344 DOI: 10.1007/s40037-012-0017-0] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/01/2023]
Abstract
The script concordance test (SCT) is designed to assess clinical reasoning by adapting the likelihood of a case diagnosis, based on provided new information. In the standard instructions students are asked to exclude alternative diagnoses they have in mind when answering the questions, but it might be more authentic to include these. Fifty-nine final-year medical students completed an SCT. Twenty-nine were asked to take their differential diagnosis into account (adapted instructions). Thirty students were asked not to consider other diagnoses (standard instructions). All participants were asked to indicate for each question whether they were confused answering it with the given instructions ('confusion indication'). Mean score of the test with the adapted instructions was 81.5 (SD 3.8) and of the test with the standard instructions 82.9 (SD 5.0) (p = 0.220). Cronbach's alpha was 0.39 for the adapted instructions and 0.66 for the standard instructions. The mean number of confusion indications was 4.2 (SD 4.4) per student for the adapted instructions and 16.7 (SD 28.5) for the standard instructions (p = 0.139). Our attempt to improve SCTs reliability by modifying the instructions did not lead to a higher alpha; therefore we do not recommend this change in the instructional format.
Collapse
Affiliation(s)
| | - Marianne V. van Asperen
- Center for Research and Development of Education, University Medical Center Utrecht, Utrecht, the Netherlands
- Mentrum Institute for Mental Health, Amsterdam, the Netherlands
| | - Eugène Custers
- Center for Research and Development of Education, University Medical Center Utrecht, Room HB 4.05, PO Box 85500, 3508 GA Utrecht, the Netherlands
| | - Gerlof D. Valk
- Department of Internal Medicine, University Medical Center Utrecht, Utrecht, the Netherlands
| | - Olle Th. J. ten Cate
- Center for Research and Development of Education, University Medical Center Utrecht, Utrecht, the Netherlands
| |
Collapse
|
22
|
Dory V, Gagnon R, Vanpee D, Charlin B. How to construct and implement script concordance tests: insights from a systematic review. MEDICAL EDUCATION 2012; 46:552-563. [PMID: 22626047 DOI: 10.1111/j.1365-2923.2011.04211.x] [Citation(s) in RCA: 35] [Impact Index Per Article: 2.9] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/28/2023]
Abstract
CONTEXT Programmes of assessment should measure the various components of clinical competence. Clinical reasoning has been traditionally assessed using written tests and performance-based tests. The script concordance test (SCT) was developed to assess clinical data interpretation skills. A recent review of the literature examined the validity argument concerning the SCT. Our aim was to provide potential users with evidence-based recommendations on how to construct and implement an SCT. METHODS A systematic review of relevant databases (MEDLINE, ERIC [Education Resources Information Centre], PsycINFO, the Research and Development Resource Base [RDRB, University of Toronto]) and Google Scholar, medical education journals and conference proceedings was conducted for references in English or French. It was supplemented by ancestry searching and by additional references provided by experts. RESULTS The search yielded 848 references, of which 80 were analysed. Studies suggest that tests with around 100 items (25-30 cases), of which 25% are discarded after item analysis, should provide reliable scores. Panels with 10-20 members are needed to reach adequate precision in terms of estimated reliability. Panellists' responses can be analysed by checking for moderate variability among responses. Studies of alternative scoring methods are inconclusive, but the traditional scoring method is satisfactory. There is little evidence on how best to determine a pass/fail threshold for high-stakes examinations. CONCLUSIONS Our literature search was broad and included references from medical education journals not indexed in the usual databases, conference abstracts and dissertations. There is good evidence on how to construct and implement an SCT for formative purposes or medium-stakes course evaluations. Further avenues for research include examining the impact of various aspects of SCT construction and implementation on issues such as educational impact, correlations with other assessments, and validity of pass/fail decisions, particularly for high-stakes examinations.
Collapse
|
23
|
Kelly W, Durning S, Denton G. Comparing a script concordance examination to a multiple-choice examination on a core internal medicine clerkship. TEACHING AND LEARNING IN MEDICINE 2012; 24:187-193. [PMID: 22775780 DOI: 10.1080/10401334.2012.692239] [Citation(s) in RCA: 10] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/01/2023]
Abstract
BACKGROUND Script concordance (SC) questions, in which a learner is given a brief clinical scenario then asked if additional information makes one hypothesis more or less likely, with answers compared to a panel of experts, are designed to reflect a learner's clinical reasoning. PURPOSE The purpose is to compare reliability, validity, and learner satisfaction between a three-option modified SC examination to a multiple-choice question (MCQ) examination among medical students during a 3rd-year internal medicine clerkship, to compare reliability and learner satisfaction of SC between medical students and a convenience sample of house staff, and to compare learner satisfaction with SC between 1st- and 4th-quarter medical students. METHODS Using a prospective cohort design, we compared the reliability of 20-item SC and MCQ examinations, sequentially administered on the same day. To measure validity, scores were compared to scores on the National Board of Medical Examiners (NBME) subject examination in medicine and to a clinical performance measure. SC and MCQ were also administered to a convenience sample of internal medicine house staff. Medical student and house staff were anonymously surveyed regarding satisfaction with the examinations. RESULTS There were 163 students who completed the examinations. With students, the initial reliability of the SC was half that of MCQ (KR20 0.19 vs. 0.41), but with house staff (n = 15), reliability was the same (KR20 = 0.52 for both examinations). SC performance correlated with student clinical performance, whereas MCQ did not (r = .22, p = .005 vs. .11, p = .159). Students reported that SC questions were no more difficult and were answered more quickly than MCQ questions. Both exams were considered easier than NBME, and all 3 were considered equally fair. More students preferred MCQ over SC (55.8% vs. 18.0%), whereas house staff preferred SC (46% vs. 23%; p = .03). CONCLUSIONS This SC examination was feasible and was more valid than the MCQ examination because of better correlation with clinical performance, despite being initially less reliable and less preferred by students. SC was more reliable and preferred when administered to house staff.
Collapse
Affiliation(s)
- William Kelly
- Department of Medicine, Uniformed Services University of Health Sciences, Bethesda, Maryland, USA.
| | | | | |
Collapse
|
24
|
Varjavand N, Greco M, Novack DH, Schindler BA. Assessment of an innovative instructional program to return non-practicing physicians to the workforce. MEDICAL TEACHER 2012; 34:285-291. [PMID: 22455697 DOI: 10.3109/0142159x.2012.660215] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/27/2023]
Abstract
INTRODUCTION Few formal educational programs are available in the United States to assist physicians wishing to return to medical practice after clinical inactivity. Little published data on physicians who complete these programs exist. We describe the Drexel Medicine Physician Reentry/Refresher course and present our findings on participant demographics, performance, and goal attainment following course completion. METHODS Physician self-assessment, future career goals, recommendations of referring organizations, and a quantitative assessment of knowledge and skills were used to create individualized learning objectives and physician's curriculum. Initial assessment included demonstration of clinical skills using standardized patients and medical knowledge using the National Board of Medical Examiners Comprehensive Clinical Medicine Self-Assessment Examination. Progress in knowledge and clinical skills was measured by repeat assessment at course completion. We questioned physicians 3 months after course completion to determine if initial goals were attained. RESULTS Thirty-six physicians completed the program from November 2006 through November 2010. Most physicians demonstrated significant improvement in core clinical skills and knowledge at the end of the course. All physicians who sought employment, hospital privileges, and refreshing skills as initial goals were successful. CONCLUSION The Drexel Medicine Physician Reentry/Refresher course provides a unique model for successfully returning inactive physicians to clinical practice.
Collapse
MESH Headings
- Adult
- Clinical Competence/standards
- Education, Distance/methods
- Education, Professional, Retraining/methods
- Education, Professional, Retraining/organization & administration
- Education, Professional, Retraining/standards
- Female
- Humans
- Internet
- Licensure, Medical
- Male
- Middle Aged
- Philadelphia
- Physicians/standards
- Preceptorship
- Schools, Medical/standards
- Schools, Medical/trends
- Self-Assessment
- United States
Collapse
|
25
|
Dufour S, Latour S, Chicoine Y, Fecteau G, Forget S, Moreau J, Trépanier A. Use of the script concordance approach to evaluate clinical reasoning in food-ruminant practitioners. JOURNAL OF VETERINARY MEDICAL EDUCATION 2012; 39:267-275. [PMID: 22951462 DOI: 10.3138/jvme.0112-13r] [Citation(s) in RCA: 8] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/01/2023]
Abstract
A script concordance test (SCT) was developed measuring clinical reasoning of food-ruminant practitioners for whom potential clinical competence difficulties were identified by their provincial professional organization. The SCT was designed to be used as part of a broader evaluation procedure. A scoring key was developed based on answers from a reference panel of 12 experts and using the modified aggregate method commonly used for SCTs. A convenient sample of 29 food-ruminant practitioners was constituted to assess the reliability and precision of the SCT and to determine a fair threshold value for success. Cronbach's α coefficients were computed to evaluate internal reliability. To evaluate SCT precision, a test-retest methodology was used and measures of agreement beyond chance were computed at question and test levels. After optimization, the 36-question SCT yielded acceptable internal reliability (Cronbach's α=0.70). Precision of the SCT at question level was excellent with 33 questions (92%) yielding moderate to almost perfect agreement between administrations. At test level, fair agreement (concordance correlation coefficient=0.32) was observed between administrations. A slight SCT score improvement (M=+2.8 points) on the second administration was in part responsible for some of the disagreement and was potentially a result of an adaptation to the SCT format. Scores distribution was used to determine a fair threshold value for success, while considering the underlying objectives of the examination. The data suggest that the developed SCT can be used as a reliable and precise measurement of clinical reasoning of food-ruminant practitioners.
Collapse
Affiliation(s)
- Simon Dufour
- Department of Pathology and Microbiology, University of Montreal, St-Hyacinthe, Canada.
| | | | | | | | | | | | | |
Collapse
|