1
|
Mistry NP, Saeed H, Rafique S, Le T, Obaid H, Adams SJ. Large Language Models as Tools to Generate Radiology Board-Style Multiple-Choice Questions. Acad Radiol 2024:S1076-6332(24)00432-X. [PMID: 39013736 DOI: 10.1016/j.acra.2024.06.046] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2024] [Revised: 06/24/2024] [Accepted: 06/27/2024] [Indexed: 07/18/2024]
Abstract
RATIONALE AND OBJECTIVES To determine the potential of large language models (LLMs) to be used as tools by radiology educators to create radiology board-style multiple choice questions (MCQs), answers, and rationales. METHODS Two LLMs (Llama 2 and GPT-4) were used to develop 104 MCQs based on the American Board of Radiology exam blueprint. Two board-certified radiologists assessed each MCQ using a 10-point Likert scale across five criteria-clarity, relevance, suitability for a board exam based on level of difficulty, quality of distractors, and adequacy of rationale. For comparison, MCQs from prior American College of Radiology (ACR) Diagnostic Radiology In-Training (DXIT) exams were also assessed using these criteria, with radiologists blinded to the question source. RESULTS Mean scores (±standard deviation) for clarity, relevance, suitability, quality of distractors, and adequacy of rationale were 8.7 (±1.4), 9.2 (±1.3), 9.0 (±1.2), 8.4 (±1.9), and 7.2 (±2.2), respectively, for Llama 2; 9.9 (±0.4), 9.9 (±0.5), 9.9 (±0.4), 9.8 (±0.5), and 9.9 (±0.3), respectively, for GPT-4; and 9.9 (±0.3), 9.9 (±0.2), 9.9 (±0.2), 9.9 (±0.4), and 9.8 (±0.6), respectively, for ACR DXIT items (p < 0.001 for Llama 2 vs. ACR DXIT across all criteria; no statistically significant difference for GPT-4 vs. ACR DXIT). The accuracy of model-generated answers was 69% for Llama 2 and 100% for GPT-4. CONCLUSION A state-of-the art LLM such as GPT-4 may be used to develop radiology board-style MCQs and rationales to enhance exam preparation materials and expand exam banks, and may allow radiology educators to further use MCQs as teaching and learning tools.
Collapse
Affiliation(s)
- Neel P Mistry
- College of Medicine, University of Saskatchewan, Saskatoon, Saskatchewan, Canada (N.P.M., H.S., H.O., S.J.A.); Department of Medical Imaging, Royal University Hospital, Saskatoon, Saskatchewan, Canada (N.P.M., H.O., S.J.A.)
| | - Huzaifa Saeed
- College of Medicine, University of Saskatchewan, Saskatoon, Saskatchewan, Canada (N.P.M., H.S., H.O., S.J.A.)
| | - Sidra Rafique
- Department of Computer Science, University of Saskatchewan, Saskatoon, Saskatchewan, Canada (S.R.)
| | - Thuy Le
- Department of Community Health and Epidemiology, University of Saskatchewan, Saskatoon, Saskatchewan, Canada (T.L.)
| | - Haron Obaid
- College of Medicine, University of Saskatchewan, Saskatoon, Saskatchewan, Canada (N.P.M., H.S., H.O., S.J.A.); Department of Medical Imaging, Royal University Hospital, Saskatoon, Saskatchewan, Canada (N.P.M., H.O., S.J.A.)
| | - Scott J Adams
- College of Medicine, University of Saskatchewan, Saskatoon, Saskatchewan, Canada (N.P.M., H.S., H.O., S.J.A.); Department of Medical Imaging, Royal University Hospital, Saskatoon, Saskatchewan, Canada (N.P.M., H.O., S.J.A.).
| |
Collapse
|
2
|
Lee HY, Yune SJ, Lee SY, Im S, Kam BS. The impact of repeated item development training on the prediction of medical faculty members' item difficulty index. BMC MEDICAL EDUCATION 2024; 24:599. [PMID: 38816855 PMCID: PMC11140961 DOI: 10.1186/s12909-024-05577-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/26/2024] [Accepted: 05/20/2024] [Indexed: 06/01/2024]
Abstract
BACKGROUND Item difficulty plays a crucial role in assessing students' understanding of the concept being tested. The difficulty of each item needs to be carefully adjusted to ensure the achievement of the evaluation's objectives. Therefore, this study aimed to investigate whether repeated item development training for medical school faculty improves the accuracy of predicting item difficulty in multiple-choice questions. METHODS A faculty development program was implemented to enhance the prediction of each item's difficulty index, ensure the absence of item defects, and maintain the general principles of item development. The interrater reliability between the predicted, actual, and corrected item difficulty was assessed before and after the training, using either the kappa index or the correlation coefficient, depending on the characteristics of the data. A total of 62 faculty members participated in the training. Their predictions of item difficulty were compared with the analysis results of 260 items taken by 119 fourth-year medical students in 2016 and 316 items taken by 125 fourth-year medical students in 2018. RESULTS Before the training, significant agreement between the predicted and actual item difficulty indices was observed for only one medical subject, Cardiology (K = 0.106, P = 0.021). However, after the training, significant agreement was noted for four subjects: Internal Medicine (K = 0.092, P = 0.015), Cardiology (K = 0.318, P = 0.021), Neurology (K = 0.400, P = 0.043), and Preventive Medicine (r = 0.577, P = 0.039). Furthermore, a significant agreement was observed between the predicted and actual difficulty indices across all subjects when analyzing the average difficulty of all items (r = 0.144, P = 0.043). Regarding the actual difficulty index by subject, neurology exceeded the desired difficulty range of 0.45-0.75 in 2016. By 2018, however, all subjects fell within this range. CONCLUSION Repeated item development training, which includes predicting each item's difficulty index, can enhance faculty members' ability to predict and adjust item difficulty accurately. To ensure that the difficulty of the examination aligns with its intended purpose, item development training can be beneficial. Further studies on faculty development are necessary to explore these benefits more comprehensively.
Collapse
Affiliation(s)
- Hye Yoon Lee
- Division of Humanities and Social Medicine, Pusan National University School of Korean Medicine, Yangsan, Republic of Korea
| | - So Jung Yune
- Department of Medical Education, Pusan National University School of Medicine, Yangsan, Republic of Korea
| | - Sang Yeoup Lee
- Department of Medical Education, Pusan National University School of Medicine, Yangsan, Republic of Korea.
- Family Medicine Clinic and Biomedical Research Institute, Pusan National University Yangsan Hospital, Yangsan, 50612, Republic of Korea.
| | - Sunju Im
- Department of Medical Education, Pusan National University School of Medicine, Yangsan, Republic of Korea
| | - Bee Sung Kam
- Department of Medical Education, Pusan National University School of Medicine, Yangsan, Republic of Korea
| |
Collapse
|
3
|
Great Question! The Art and Science of Crafting High-Quality Multiple-Choice Questions. J Am Coll Radiol 2022; 19:687-692. [DOI: 10.1016/j.jacr.2022.01.016] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/19/2021] [Revised: 01/03/2022] [Accepted: 01/04/2022] [Indexed: 11/17/2022]
|
4
|
Smith EB, Gellatly M, Schwartz CJ, Jordan S. Training Radiology Residents, Bloom Style. Acad Radiol 2021; 28:1626-1630. [PMID: 32921568 DOI: 10.1016/j.acra.2020.08.013] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/24/2020] [Revised: 08/10/2020] [Accepted: 08/11/2020] [Indexed: 01/16/2023]
Abstract
Bloom's Taxonomy, an integral component of learning theory since its inception, describes cognitive skill levels in increasing complexity (Remember, Understand, Apply, Analyze, Evaluate, and Create). Considering Bloom's Taxonomy when writing learning objectives and lecture material, teaching residents at the workstation and creating multiple choice questions can increase an educator's effectiveness. The incorporation of higher Bloom levels aids in cultivating critical thinking skills vital to image interpretation and patient care, and becomes increasingly important as the radiologist's role evolves with the continued development of artificial intelligence. Following established tenets of multiple choice question writing, involving trainees in the question writing process, and incorporating audience response systems into lectures are all strategies in which higher Bloom level skills can be accomplished.
Collapse
Affiliation(s)
- Elana B Smith
- R. Adams Cowley Shock Trauma Center, University of Maryland Medical Center, Department of Radiology, 22 S. Greene St., Baltimore, MD 21201.
| | - Matthew Gellatly
- University of North Carolina School of Medicine, Chapel Hill, North Carolina
| | - Cody J Schwartz
- University of North Carolina School of Medicine, Department of Radiology, Chapel Hill, North Carolina
| | - Sheryl Jordan
- University of North Carolina School of Medicine, Department of Radiology, Chapel Hill, North Carolina
| |
Collapse
|
5
|
Abstract
Multiple-choice tests are the most used method of assessment in medical education. However, there is limited literature in medical education and psychiatry to inform the best practices in writing good-quality multiple-choice questions. Moreover, few physicians and psychiatrists have received training and have experience in writing them. This article highlights the strategies in writing high-quality multiple-choice items and discusses some common flaws that can impact validity and reliability of the assessment examinations.
Collapse
Affiliation(s)
- Vikas Gupta
- South Carolina Department of Mental Health, 2715 Colonial Drive, Suite 200-A, Colonial Drive, Columbia, SC 29201, USA
| | - Eric R Williams
- University of South Carolina School of Medicine, 6311 Garners Ferry Road, Suite 126, Columbia, SC 29209, USA.
| | - Roopma Wadhwa
- South Carolina Department of Mental Health, 2715 Colonial Drive, Suite 200-A, Colonial Drive, Columbia, SC 29201, USA
| |
Collapse
|
6
|
Scott KR, King AM, Estes MK, Conlon LW, Jones JS, Phillips AW. Evaluation of an Intervention to Improve Quality of Single-best Answer Multiple-choice Questions. West J Emerg Med 2019; 20:11-14. [PMID: 30643595 PMCID: PMC6324722 DOI: 10.5811/westjem.2018.11.39805] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2018] [Revised: 11/04/2018] [Accepted: 11/08/2018] [Indexed: 11/29/2022] Open
Abstract
Introduction Despite the ubiquity of single-best answer multiple-choice questions (MCQ) in assessments throughout medical education, question writers often receive little to no formal training, potentially decreasing the validity of assessments. While lengthy training opportunities in item writing exist, the availability of brief interventions is limited. Methods We developed and performed an initial validation of an item-quality assessment tool and measured the impact of a brief educational intervention on the quality of single-best answer MCQs. Results The item-quality assessment tool demonstrated moderate internal structure evidence when applied to the 20 practice questions (κ=.671, p<.001) and excellent internal structure when applied to the true dataset (κ=0.904, p<.001). Quality scale scores for pre-intervention questions ranged from 2–6 with a mean ± standard deviation (SD) of 3.79 ± 1.23, while post-intervention scores ranged from 4–6 with a mean ± SD of 5.42 ± 0.69. The post-intervention scores were significantly higher than the pre-intervention scores, x2(1) =38, p <0.001. Conclusion Our study demonstrated short-term improvement in single-best answer MCQ writing quality after a brief, open-access lecture, as measured by a simple, novel, grading rubric with reasonable validity evidence.
Collapse
Affiliation(s)
- Kevin R Scott
- Perelman School of Medicine at the University of Pennsylvania, Department of Emergency Medicine, Philadelphia, Pennsylvania
| | - Andrew M King
- The Ohio State University Wexner Medical Center, Department of Emergency Medicine, Columbus, Ohio
| | - Molly K Estes
- Loma Linda University Medical Center, Department of Emergency Medicine, Loma Linda, California
| | - Lauren W Conlon
- Perelman School of Medicine at the University of Pennsylvania, Department of Emergency Medicine, Philadelphia, Pennsylvania
| | - Jonathan S Jones
- Merit Health Central, Department of Emergency Medicine, Jackson, Mississippi
| | - Andrew W Phillips
- University of North Carolina, Department of Emergency Medicine, Chapel Hill, North Carolina
| |
Collapse
|
7
|
Zaidi NB, Hwang C, Scott S, Stallard S, Purkiss J, Hortsch M. Climbing Bloom's taxonomy pyramid: Lessons from a graduate histology course. ANATOMICAL SCIENCES EDUCATION 2017; 10:456-464. [PMID: 28231408 DOI: 10.1002/ase.1685] [Citation(s) in RCA: 29] [Impact Index Per Article: 4.1] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/18/2016] [Revised: 01/09/2017] [Accepted: 01/23/2017] [Indexed: 06/06/2023]
Abstract
Bloom's taxonomy was adopted to create a subject-specific scoring tool for histology multiple-choice questions (MCQs). This Bloom's Taxonomy Histology Tool (BTHT) was used to analyze teacher- and student-generated quiz and examination questions from a graduate level histology course. Multiple-choice questions using histological images were generally assigned a higher BTHT level than simple text questions. The type of microscopy technique (light or electron microscopy) used for these image-based questions did not result in any significant differences in their Bloom's taxonomy scores. The BTHT levels for teacher-generated MCQs correlated positively with higher discrimination indices and inversely with the percent of students answering these questions correctly (difficulty index), suggesting that higher-level Bloom's taxonomy questions differentiate well between higher- and lower-performing students. When examining BTHT scores for MCQs that were written by students in a Multiple-Choice Item Development Assignment (MCIDA) there was no significant correlation between these scores and the students' ability to answer teacher-generated MCQs. This suggests that the ability to answer histology MCQs relies on a different skill set than the aptitude to construct higher-level Bloom's taxonomy questions. However, students significantly improved their average BTHT scores from the midterm to the final MCIDA task, which indicates that practice, experience and feedback increased their MCQ writing proficiency. Anat Sci Educ 10: 456-464. © 2017 American Association of Anatomists.
Collapse
Affiliation(s)
- Nikki B Zaidi
- Office of Medical Student Education, University of Michigan Medical School, Ann Arbor, Michigan
| | - Charles Hwang
- Universty of Michigan Medical School, Ann Arbor, Michigan
| | - Sara Scott
- Universty of Michigan Medical School, Ann Arbor, Michigan
| | | | - Joel Purkiss
- Office of Medical Student Education, University of Michigan Medical School, Ann Arbor, Michigan
- Department of Learning Health Sciences, University of Michigan Medical School, Ann Arbor, Michigan
- Office of the Curriculum, School of Medicine, Baylor College for Medicine, Houston, Texas
| | - Michael Hortsch
- Department of Learning Health Sciences, University of Michigan Medical School, Ann Arbor, Michigan
- Department of Cell and Developmental Biology, University of Michigan Medical School, Ann Arbor, Michigan
| |
Collapse
|
8
|
Dell KA, Wantuch GA. How-to-guide for writing multiple choice questions for the pharmacy instructor. CURRENTS IN PHARMACY TEACHING & LEARNING 2017; 9:137-144. [PMID: 29180146 DOI: 10.1016/j.cptl.2016.08.036] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/17/2015] [Revised: 07/07/2016] [Accepted: 08/23/2016] [Indexed: 06/07/2023]
Abstract
BACKGROUND Writing multiple choice questions (MCQ) takes a lot of practice. Often, pharmacy practitioners lack the training to write effective MCQ. Sources for instruction in effective MCQ writing can be overwhelming with numerous suggestions of what should and should not be done. PURPOSE The following guide is prepared to serve as a succinct reference for creation and revision of MCQ by both novice and seasoned pharmacy faculty practitioners. METHODS The literature is summarized into 12 best practices for writing effective MCQ. Pharmacy specific examples that demonstrate violations of best practices and how they can be corrected are provided. IMPLICATIONS The guide can serve as a primer to write new MCQ, as a reference to revise previously created questions, or as a guide to peer review of MCQ.
Collapse
Affiliation(s)
- Kamila A Dell
- College of Pharmacy, University of South Florida, Tampa, FL; College of Medicine, University of South Florida, Tampa, FL.
| | - Gwendolyn A Wantuch
- College of Pharmacy, University of South Florida, Tampa, FL; College of Medicine, University of South Florida, Tampa, FL
| |
Collapse
|
9
|
Tests, Quizzes, and Self-Assessments: How to Construct a High-Quality Examination. AJR Am J Roentgenol 2016; 207:339-43. [DOI: 10.2214/ajr.15.15944] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
|
10
|
Jensen JT, Savran MM, Møller AM, Vilmann P, Hornslet P, Konge L. Development and validation of a theoretical test in non-anaesthesiologist-administered propofol sedation for gastrointestinal endoscopy. Scand J Gastroenterol 2016; 51:872-9. [PMID: 26872690 DOI: 10.3109/00365521.2016.1141433] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 02/04/2023]
Abstract
OBJECTIVE Safety with non-anaesthesiologist-administered propofol sedation (NAAP) during gastrointestinal (GI) endoscopy is related to theoretical knowledge. A summative testing of knowledge before attempting supervised nurse-administered propofol sedation (NAPS) in the clinic is advised. The aims of this study were to develop a theoretical test about propofol sedation, to gather validity evidence for the test and to measure the effect of a NAPS-specific training course. MATERIAL AND METHODS A three-phased psychometric study on multiple choice questionnaire (MCQ) test development, gathering of validity evidence and evaluation of the effect of a specific NAAP course on the test result. A MCQ containing 86 questions was developed and administered 113 times to 91 participants representing novices, intermediates and experienced. RESULTS Question difficulty analyses revealed 50 level I and II questions. The 50 MCQs showed mean (SD) intergroup differences (p = 0.001) between novices = 28.6 (4.82), intermediates = 36.8 (5.43) and experienced = 41.8 (4.65) and provided a pass score of 35.2. The course with pre-course test had significant effect on the knowledge of nurses (18% increase) and physicians (19% increase; p = 0.001 and 0.001, respectively). CONCLUSIONS Data supported the validity of the developed MCQ test. The NAPS-specific course with pre-course testing adds theoretical knowledge to already well-prepared participants.
Collapse
Affiliation(s)
- Jeppe Thue Jensen
- a Department of Surgical Gastroenterology , Herlev University Hospital , Herlev , Capital Region of Denmark
| | - Mona Meral Savran
- b Centre for HR , Copenhagen Academy for Medical Education and Simulation , Copenhagen , Capital Region of Denmark
| | - Ann Merete Møller
- c Department of Anaesthesiology , Herlev University Hospital , Herlev , Capital Region of Denmark
| | - Peter Vilmann
- a Department of Surgical Gastroenterology , Herlev University Hospital , Herlev , Capital Region of Denmark
| | - Pernille Hornslet
- a Department of Surgical Gastroenterology , Herlev University Hospital , Herlev , Capital Region of Denmark
| | - Lars Konge
- b Centre for HR , Copenhagen Academy for Medical Education and Simulation , Copenhagen , Capital Region of Denmark
| |
Collapse
|
11
|
Kelly AM, Slanetz PJ. Radiology Education: Creating Tomorrow's Radiologists with Yesterday's Budgets! Acad Radiol 2015; 22:1211-3. [PMID: 26314497 DOI: 10.1016/j.acra.2015.08.001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2015] [Revised: 08/01/2015] [Accepted: 08/02/2015] [Indexed: 10/23/2022]
Affiliation(s)
- Aine Marie Kelly
- Division of Cardiothoracic Radiology, Department of Radiology, University of Michigan Medical Center, B1 132K Taubman Center/5302, 1500 East Medical Center Drive, Ann Arbor, MI 48109.
| | - Priscilla J Slanetz
- Division of Breast Imaging, Department of Radiology, Beth Israel Deaconess Medical Center, Boston, Massachusetts; Harvard Medical School, Boston, Massachusetts
| |
Collapse
|