1
|
Goh E, Gallo R, Strong E, Weng Y, Kerman H, Freed J, Cool JA, Kanjee Z, Lane KP, Parsons AS, Ahuja N, Horvitz E, Yang D, Milstein A, Olson APJ, Hom J, Chen JH, Rodman A. Large Language Model Influence on Management Reasoning: A Randomized Controlled Trial. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2024:2024.08.05.24311485. [PMID: 39148822 PMCID: PMC11326321 DOI: 10.1101/2024.08.05.24311485] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Indexed: 08/17/2024]
Abstract
Importance Large language model (LLM) artificial intelligence (AI) systems have shown promise in diagnostic reasoning, but their utility in management reasoning with no clear right answers is unknown. Objective To determine whether LLM assistance improves physician performance on open-ended management reasoning tasks compared to conventional resources. Design Prospective, randomized controlled trial conducted from 30 November 2023 to 21 April 2024. Setting Multi-institutional study from Stanford University, Beth Israel Deaconess Medical Center, and the University of Virginia involving physicians from across the United States. Participants 92 practicing attending physicians and residents with training in internal medicine, family medicine, or emergency medicine. Intervention Five expert-developed clinical case vignettes were presented with multiple open-ended management questions and scoring rubrics created through a Delphi process. Physicians were randomized to use either GPT-4 via ChatGPT Plus in addition to conventional resources (e.g., UpToDate, Google), or conventional resources alone. Main Outcomes and Measures The primary outcome was difference in total score between groups on expert-developed scoring rubrics. Secondary outcomes included domain-specific scores and time spent per case. Results Physicians using the LLM scored higher compared to those using conventional resources (mean difference 6.5 %, 95% CI 2.7-10.2, p<0.001). Significant improvements were seen in management decisions (6.1%, 95% CI 2.5-9.7, p=0.001), diagnostic decisions (12.1%, 95% CI 3.1-21.0, p=0.009), and case-specific (6.2%, 95% CI 2.4-9.9, p=0.002) domains. GPT-4 users spent more time per case (mean difference 119.3 seconds, 95% CI 17.4-221.2, p=0.02). There was no significant difference between GPT-4-augmented physicians and GPT-4 alone (-0.9%, 95% CI -9.0 to 7.2, p=0.8). Conclusions and Relevance LLM assistance improved physician management reasoning compared to conventional resources, with particular gains in contextual and patient-specific decision-making. These findings indicate that LLMs can augment management decision-making in complex cases. Trial registration ClinicalTrials.gov Identifier: NCT06208423; https://classic.clinicaltrials.gov/ct2/show/NCT06208423.
Collapse
Affiliation(s)
- Ethan Goh
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, CA
- Stanford Clinical Excellence Research Center, Stanford University, Stanford, CA
| | - Robert Gallo
- Center for Innovation to Implementation, VA Palo Alto Health Care System, PA, CA
| | - Eric Strong
- Stanford University School of Medicine, Stanford, CA
| | - Yingjie Weng
- Quantitative Sciences Unit, Stanford University School of Medicine, Stanford, CA
| | - Hannah Kerman
- Beth Israel Deaconess Medical Center, Boston, MA
- Harvard Medical School, Boston, MA
| | - Jason Freed
- Beth Israel Deaconess Medical Center, Boston, MA
| | - Joséphine A Cool
- Beth Israel Deaconess Medical Center, Boston, MA
- Harvard Medical School, Boston, MA
| | - Zahir Kanjee
- Beth Israel Deaconess Medical Center, Boston, MA
- Harvard Medical School, Boston, MA
| | | | | | - Neera Ahuja
- Stanford University School of Medicine, Stanford, CA
| | - Eric Horvitz
- Microsoft, Redmond, WA
- Stanford Institute for Human-Centered Artificial Intelligence, Stanford, CA
| | | | - Arnold Milstein
- Stanford Clinical Excellence Research Center, Stanford University, Stanford, CA
| | | | - Jason Hom
- Stanford University School of Medicine, Stanford, CA
| | - Jonathan H Chen
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, CA
- Stanford Clinical Excellence Research Center, Stanford University, Stanford, CA
- Division of Hospital Medicine, Stanford University, Stanford, CA
| | - Adam Rodman
- Beth Israel Deaconess Medical Center, Boston, MA
- Harvard Medical School, Boston, MA
| |
Collapse
|
2
|
Krimmel-Morrison JD, Watsjold BK, Berger GN, Bowen JL, Ilgen JS. 'Walking together': How relationships shape physicians' clinical reasoning. MEDICAL EDUCATION 2024; 58:961-969. [PMID: 38525645 DOI: 10.1111/medu.15377] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/02/2023] [Revised: 02/16/2024] [Accepted: 02/21/2024] [Indexed: 03/26/2024]
Abstract
INTRODUCTION The clinical reasoning literature has increasingly considered context as an important influence on physicians' thinking. Physicians' relationships with patients, and their ongoing efforts to maintain these relationships, are important influences on how clinical reasoning is contextualised. The authors sought to understand how physicians' relationships with patients shaped their clinical reasoning. METHODS Drawing from constructivist grounded theory, the authors conducted semi-structured interviews with primary care physicians. Participants were asked to reflect on recent challenging clinical experiences, and probing questions were used to explore how participants attended to or leveraged relationships in conjunction with their clinical reasoning. Using constant comparison, three investigators coded transcripts, organising the data into codes and conceptual categories. The research team drew from these codes and categories to develop theory about the phenomenon of interest. RESULTS The authors interviewed 15 primary care physicians with a range of experience in practice and identified patient agency as a central influence on participants' clinical reasoning. Participants drew from and managed relationships with patients while attending to patients' agency in three ways. First, participants described how contextualised illness constructions enabled them to individualise their approaches to diagnosis and management. Second, participants managed tensions between enacting their typical approaches to clinical problems and adapting their approaches to foster ongoing relationships with patients. Finally, participants attended to relationships with patients' caregivers, seeing these individuals' contributions as important influences on how their clinical reasoning could be enacted within patients' unique social contexts. CONCLUSION Clinical reasoning is influenced in important ways by physicians' efforts to both draw from, and maintain, their relationships with patients and patients' caregivers. Such efforts create tensions between their professional standards of care and their orientations toward patient-centredness. These influences of relationships on physicians' clinical reasoning have important implications for training and clinical practice.
Collapse
Affiliation(s)
| | - Bjorn K Watsjold
- Department of Emergency Medicine, University of Washington School of Medicine, Seattle, Washington, USA
| | - Gabrielle N Berger
- Department of Medicine, University of Washington School of Medicine, Seattle, Washington, USA
| | - Judith L Bowen
- Department of Medical Education and Clinical Sciences, Washington State University Elson S. Floyd School of Medicine, Spokane, Washington, USA
| | - Jonathan S Ilgen
- Department of Emergency Medicine, University of Washington School of Medicine, Seattle, Washington, USA
| |
Collapse
|
3
|
Minter DJ, Parsons AS, Abdoler E. Reasoning Report: Engineering Case Conferences to Maximize Clinical Reasoning Education for All Learners. J Gen Intern Med 2024:10.1007/s11606-024-08778-8. [PMID: 38980464 DOI: 10.1007/s11606-024-08778-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/16/2024] [Accepted: 04/18/2024] [Indexed: 07/10/2024]
Abstract
Case conferences, specifically those in which an unknown case is presented and discussed, are widely utilized in the delivery of medical education. However, the format of case conferences is not always optimized to engage and challenge audience members' clinical reasoning (CR). Based on the current conception of CR and our experience, we provide recommendations on how to better engineer case conferences to maximize CR education for learners at all levels through case selection, conference format, and intentional case construction.
Collapse
Affiliation(s)
- Daniel J Minter
- Division of Infectious Diseases, Department of Medicine, University of California San Francisco, San Francisco, CA, 94143, USA.
| | - Andrew S Parsons
- Division of General, Geriatric, Palliative, and Hospital Medicine, Department of Medicine, University of Virginia, Charlottesville, VA, USA
| | - Emily Abdoler
- Division of Infectious Diseases, Department of Medicine, University of Michigan, Ann Arbor, MI, USA
| |
Collapse
|
4
|
Cook DA, Hargraves IG, Stephenson CR, Durning SJ. Management reasoning and patient-clinician interactions: Insights from shared decision-making and simulated outpatient encounters. MEDICAL TEACHER 2023; 45:1025-1037. [PMID: 36763491 DOI: 10.1080/0142159x.2023.2170776] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/18/2023]
Abstract
PURPOSE To expand understanding of patient-clinician interactions in management reasoning. METHODS We reviewed 10 videos of simulated patient-clinician encounters to identify instances of problematic and successful communication, then reviewed the videos again through the lens of two models of shared decision-making (SDM): an 'involvement-focused' model and a 'problem-focused' model. Using constant comparative qualitative analysis we explored the connections between these patient-clinician interactions and management reasoning. RESULTS Problems in patient-clinician interactions included failures to: encourage patient autonomy; invite the patient's involvement in decision-making; convey the health impact of the problem; explore and address concerns and questions; explore the context of decision-making (including patient preferences); meet the patient where they are; integrate situational preferences and priorities; offer >1 viable option; work with the patient to solve a problem of mutual concern; explicitly agree to a final care plan; and build the patient-clinician relationship. Clinicians' 'management scripts' varied along a continuum of prioritizing clinician vs patient needs. Patients also have their own cognitive scripts that guide their interactions with clinicians. The involvement-focused and problem-focused SDM models illuminated distinct, complementary issues. CONCLUSIONS Management reasoning is a deliberative interaction occurring in the space between individuals. Juxtaposing management reasoning alongside SDM generated numerous insights.
Collapse
Affiliation(s)
- David A Cook
- Office of Applied Scholarship and Education Science, Mayo Clinic College of Medicine and Science; and Division of General Internal Medicine, Mayo Clinic, Rochester, MN, USA
| | - Ian G Hargraves
- Mayo Clinic National Shared Decision Making Resource Center, Mayo Clinic, Rochester, MN, USA
| | | | - Steven J Durning
- Center for Health Professions Education, Uniformed Services University of the Health Sciences, Bethesda, MD, USA
| |
Collapse
|
5
|
Fong JMN, Hoe RHM, Huang DH, Wong JC, Kee XLJ, Teng KLA, Hong R, Saffari SE, Tan K, Tan NCK. Script concordance test to assess diagnostic and management reasoning in acute medicine. ANNALS OF THE ACADEMY OF MEDICINE, SINGAPORE 2023; 52:383-385. [PMID: 38904506 DOI: 10.47102/annals-acadmedsg.202327] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/22/2024]
Abstract
Clinical reasoning, an essential skill for patient care, can be difficult to assess. We created and validated a script concordance test (SCT) to assess clinical reasoning in acute medicine. This tool was used to provide feedback and targeted remediation for Postgraduate-Year-1 (PGY1) doctors, guide teaching and learning, and facilitate programme evaluation.
Collapse
Affiliation(s)
| | | | | | | | | | | | - Rilong Hong
- Department of Cardiology, National Heart Centre Singapore
| | - Seyed Ehsan Saffari
- Department of Neurology, National Neuroscience Institute, Singapore
- Health Services & Systems Research, Duke-NUS Medical School, National University of Singapore, Singapore
| | - Kevin Tan
- Department of Neurology, National Neuroscience Institute, Singapore
| | | |
Collapse
|
6
|
Cook DA, Stephenson CR, Gruppen LD, Durning SJ. Management reasoning scripts: Qualitative exploration using simulated physician-patient encounters. PERSPECTIVES ON MEDICAL EDUCATION 2022; 11:196-206. [PMID: 35653028 PMCID: PMC9391545 DOI: 10.1007/s40037-022-00714-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/17/2022] [Revised: 04/23/2022] [Accepted: 04/26/2022] [Indexed: 06/15/2023]
Abstract
INTRODUCTION Management reasoning is distinct from diagnostic reasoning and remains incompletely understood. The authors sought to empirically investigate the concept of management scripts. METHODS In November 2020, 4 investigators each reviewed 10 video clips of simulated outpatient physician-patient encounters, and used a coding form to document observations about management reasoning. The team used constant comparative analysis to integrate empirically-grounded insights with theories related to cognitive scripts and Type 1/Type 2 thinking. RESULTS Management scripts are precompiled conceptual knowledge structures that represent and connect management options and clinician tasks in a temporal or logical sequence. Management scripts appear to differ substantially from illness scripts. Management scripts varied in quality (in content, sequence, flexibility, and fluency) and generality. The authors empirically identified six key features (components) of management scripts: the problem (diagnosis); management options; preferences, values, and constraints; education needs; interactions; and encounter flow. The authors propose a heuristic framework describing script activation, selection, instantiation with case-specific details, and application to guide development of the management plan. They further propose that management reasoning reflects iterative, back-and-forth involvement of both Type 1 (non-analytic, effortless) and Type 2 (analytic, effortful) thinking. Type 1 thinking likely influences initial script activation, selection, and initial instantiation. Type 2 increasingly influences subsequent script revisions, as activation, selection, and instantiation become more deliberate (effortful) and more hypothetical (involving mental simulation). DISCUSSION Management scripts constitute a key feature of management reasoning, and could represent a new target for training in clinical reasoning (distinct from illness scripts).
Collapse
Affiliation(s)
- David A Cook
- Office of Applied Scholarship and Education Science, Mayo Clinic College of Medicine and Science, Rochester, MN, USA.
- Division of General Internal Medicine, Mayo Clinic, Rochester, MN, USA.
| | | | - Larry D Gruppen
- Department of Learning Health Sciences, University of Michigan, Ann Arbor, MI, USA
| | - Steven J Durning
- Center for Health Professions Education, Uniformed Services University of the Health Sciences, Bethesda, MD, USA
| |
Collapse
|