1
|
Gleason KT, Dukhanin V, Peterson SK, Gonzalez N, Austin JM, McDonald KM. Development and Psychometric Analysis of a Patient-Reported Measure of Diagnostic Excellence for Emergency and Urgent Care Settings. J Patient Saf 2024:01209203-990000000-00255. [PMID: 39194332 DOI: 10.1097/pts.0000000000001271] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 08/29/2024]
Abstract
BACKGROUND Emergency and urgent care settings face challenges with routinely obtaining performance feedback related to diagnostic care. Patients and their care partners provide an important perspective on the diagnostic process and outcome of care in these settings. We sought to develop and test psychometric properties of Patient-Report to IMprove Diagnostic Excellence in Emergency Department settings (PRIME-ED), a measure of patient-reported diagnostic excellence in these care settings. METHODS We developed PRIME-ED based on literature review, expert feedback, and cognitive testing. To assess psychometric properties, we surveyed AmeriSpeak, a probability-based panel that provides sample coverage of approximately 97% of the U.S. household population, in February 2022 to adult patients, or their care partners, who had presented to an emergency department or urgent care facility within the last 30 days. Respondents rated their agreement on a 5-point Likert scale with each of 17 statements across multiple domains of patient-reported diagnostic excellence. Demographics, visit characteristics, and a subset of the Emergency Department Consumer Assessment of Healthcare Providers & Systems were also collected. We conducted psychometric testing for reliability and validity. RESULTS Over a thousand (n = 1116) national panelists completed the PRIME-ED survey, of which 58.7% were patients and 40.9% were care partners; 49.6% received care at an emergency department and 49.9% at an urgent care facility. Responses had high internal consistency within 3 patient-reported diagnostic excellence domain groupings: diagnostic process (Cronbach's alpha 0.94), accuracy of diagnosis (0.93), and communication of diagnosis (0.94). Domain groupings were significantly correlated with concurrent Emergency Department Consumer Assessment of Healthcare Providers & Systems items. Factor analyses substantiated 3 domain groupings. CONCLUSIONS PRIME-ED has potential as a tool for capturing patient-reported diagnostic excellence in emergency and urgent care.
Collapse
Affiliation(s)
- Kelly T Gleason
- From the Johns Hopkins University School of Nursing, Baltimore, Maryland
| | - Vadim Dukhanin
- Department of Health Policy and Management, Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland
| | - Susan K Peterson
- Department of Emergency Medicine, Johns Hopkins University School of Medicine, Baltimore, Maryland
| | | | - J M Austin
- Armstrong Institute for Patient Safety and Quality and Department of Anesthesiology and Critical Care Medicine, Johns Hopkins University School of Medicine, Baltimore, Maryland
| | | |
Collapse
|
2
|
Shikino K, Shimizu T, Otsuka Y, Tago M, Takahashi H, Watari T, Sasaki Y, Iizuka G, Tamura H, Nakashima K, Kunitomo K, Suzuki M, Aoyama S, Kosaka S, Kawahigashi T, Matsumoto T, Orihara F, Morikawa T, Nishizawa T, Hoshina Y, Yamamoto Y, Matsuo Y, Unoki Y, Kimura H, Tokushima M, Watanuki S, Saito T, Otsuka F, Tokuda Y. Evaluation of ChatGPT-Generated Differential Diagnosis for Common Diseases With Atypical Presentation: Descriptive Research. JMIR MEDICAL EDUCATION 2024; 10:e58758. [PMID: 38915174 PMCID: PMC11199925 DOI: 10.2196/58758] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/27/2024] [Revised: 05/03/2024] [Accepted: 05/19/2024] [Indexed: 06/26/2024]
Abstract
Background The persistence of diagnostic errors, despite advances in medical knowledge and diagnostics, highlights the importance of understanding atypical disease presentations and their contribution to mortality and morbidity. Artificial intelligence (AI), particularly generative pre-trained transformers like GPT-4, holds promise for improving diagnostic accuracy, but requires further exploration in handling atypical presentations. Objective This study aimed to assess the diagnostic accuracy of ChatGPT in generating differential diagnoses for atypical presentations of common diseases, with a focus on the model's reliance on patient history during the diagnostic process. Methods We used 25 clinical vignettes from the Journal of Generalist Medicine characterizing atypical manifestations of common diseases. Two general medicine physicians categorized the cases based on atypicality. ChatGPT was then used to generate differential diagnoses based on the clinical information provided. The concordance between AI-generated and final diagnoses was measured, with a focus on the top-ranked disease (top 1) and the top 5 differential diagnoses (top 5). Results ChatGPT's diagnostic accuracy decreased with an increase in atypical presentation. For category 1 (C1) cases, the concordance rates were 17% (n=1) for the top 1 and 67% (n=4) for the top 5. Categories 3 (C3) and 4 (C4) showed a 0% concordance for top 1 and markedly lower rates for the top 5, indicating difficulties in handling highly atypical cases. The χ2 test revealed no significant difference in the top 1 differential diagnosis accuracy between less atypical (C1+C2) and more atypical (C3+C4) groups (χ²1=2.07; n=25; P=.13). However, a significant difference was found in the top 5 analyses, with less atypical cases showing higher accuracy (χ²1=4.01; n=25; P=.048). Conclusions ChatGPT-4 demonstrates potential as an auxiliary tool for diagnosing typical and mildly atypical presentations of common diseases. However, its performance declines with greater atypicality. The study findings underscore the need for AI systems to encompass a broader range of linguistic capabilities, cultural understanding, and diverse clinical scenarios to improve diagnostic utility in real-world settings.
Collapse
Affiliation(s)
- Kiyoshi Shikino
- Department of General Medicine, Chiba University Hospital, Chiba, Japan
- Department of Community-Oriented Medical Education, Chiba University Graduate School of Medicine, Chiba, Japan
| | - Taro Shimizu
- Department of Diagnostic and Generalist Medicine, Dokkyo Medical University, Tochigi, Japan
| | - Yuki Otsuka
- Department of General Medicine, Dentistry and Pharmaceutical Sciences, Okayama University Graduate School of Medicine, Okayama, Japan
| | - Masaki Tago
- Department of General Medicine, Saga University Hospital, Saga, Japan
| | - Hiromizu Takahashi
- Department of General Medicine, Juntendo University Hospital Faculty of Medicine, Tokyo, Japan
| | - Takashi Watari
- Integrated Clinical Education Center Hospital Integrated Clinical Education, Kyoto University Hospital, Kyoto, Japan
| | - Yosuke Sasaki
- Department of General Medicine and Emergency Care, Toho University School of Medicine, Tokyo, Japan
| | - Gemmei Iizuka
- Center for Preventive Medical Sciences, Chiba University, Chiba, Japan
- Tama Family Clinic, Kanagawa, Japan
| | - Hiroki Tamura
- Department of General Medicine, Chiba University Hospital, Chiba, Japan
| | - Koichi Nakashima
- Department of General Medicine, Awa Regional Medical Center, Chiba, Japan
| | - Kotaro Kunitomo
- Department of General Medicine, National Hospital Organization Kumamoto Medical Center, Kumamoto, Japan
| | - Morika Suzuki
- Department of General Medicine, National Hospital Organization Kumamoto Medical Center, Kumamoto, Japan
- Department of Neurology, University of Utah, Salt Lake City, UT, United States
| | - Sayaka Aoyama
- Department of Internal Medicine, Mito Kyodo General Hospital, Ibaraki, Japan
| | | | - Teiko Kawahigashi
- Department of Molecular and Human Genetics, Baylor College of Medicine, Houston, TX, United States
| | | | - Fumina Orihara
- Division of General Medicine, Nerima Hikarigaoka Hospital, Tokyo, Japan
| | - Toru Morikawa
- Department of General Medicine, Nara City Hospital, Nara, Japan
| | - Toshinori Nishizawa
- Department of General Internal Medicine, St. Luke’s International Hospital, Tokyo, Japan
| | - Yoji Hoshina
- Department of Neurology, University of Utah, Salt Lake City, UT, United States
| | - Yu Yamamoto
- Division of General Medicine, Center for Community Medicine, Jichi Medical University, Tochigi, Japan
| | - Yuichiro Matsuo
- Department of Clinical Epidemiology and Health Economics, The Graduate School of Medicine, The University of Tokyo, Tokyo, Japan
| | - Yuto Unoki
- Department of General Internal Medicine, Iizuka Hospital, Fukuoka, Japan
| | - Hirofumi Kimura
- Department of General Internal Medicine, Iizuka Hospital, Fukuoka, Japan
| | - Midori Tokushima
- Saga Medical Career Support Center, Saga University Hospital, Saga, Japan
| | - Satoshi Watanuki
- Department of Emergency and General Medicine, Tokyo Metropolitan Tama Medical Center, Tokyo, Japan
| | - Takuma Saito
- Department of Emergency and General Medicine, Tokyo Metropolitan Tama Medical Center, Tokyo, Japan
| | - Fumio Otsuka
- Department of General Medicine, Dentistry and Pharmaceutical Sciences, Okayama University Graduate School of Medicine, Okayama, Japan
| | - Yasuharu Tokuda
- Muribushi Okinawa Center for Teaching Hospitals, Okinawa, Japan
- Tokyo Foundation for Policy Research, Tokyo, Japan
| |
Collapse
|
3
|
Kotwal S, Howell M, Zwaan L, Wright SM. Exploring Clinical Lessons Learned by Experienced Hospitalists from Diagnostic Errors and Successes. J Gen Intern Med 2024; 39:1386-1392. [PMID: 38277023 PMCID: PMC11169201 DOI: 10.1007/s11606-024-08625-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/23/2023] [Accepted: 01/09/2024] [Indexed: 01/27/2024]
Abstract
BACKGROUND Diagnostic errors cause significant patient harm. The clinician's ultimate goal is to achieve diagnostic excellence in order to serve patients safely. This can be accomplished by learning from both errors and successes in patient care. However, the extent to which clinicians grow and navigate diagnostic errors and successes in patient care is poorly understood. Clinically experienced hospitalists, who have cared for numerous acutely ill patients, should have great insights from their successes and mistakes to inform others striving for excellence in patient care. OBJECTIVE To identify and characterize clinical lessons learned by experienced hospitalists from diagnostic errors and successes. DESIGN A semi-structured interview guide was used to collect qualitative data from hospitalists at five independently administered hospitals in the Mid-Atlantic area from February to June 2022. PARTICIPANTS 12 academic and 12 community-based hospitalists with ≥ 5 years of clinical experience. APPROACH A constructivist qualitative approach was used and "reflexive thematic analysis" of interview transcripts was conducted to identify themes and patterns of meaning across the dataset. RESULTS Five themes were generated from the data based on clinical lessons learned by hospitalists from diagnostic errors and successes. The ideas included appreciating excellence in clinical reasoning as a core skill, connecting with patients and other members of the health care team to be able to tap into their insights, reflecting on the diagnostic process, committing to growth, and prioritizing self-care. CONCLUSIONS The study identifies key lessons learned from the errors and successes encountered in patient care by clinically experienced hospitalists. These findings may prove helpful for individuals and groups that are authentically committed to moving along the continuum from diagnostic competence towards excellence.
Collapse
Affiliation(s)
- Susrutha Kotwal
- Department of Medicine, Division of Hospital Medicine, Johns Hopkins Bayview Medical Center, Johns Hopkins University School of Medicine, Baltimore, MD, USA.
| | - Mason Howell
- Department of Biosciences, Rice University, Houston, TX, USA
| | - Laura Zwaan
- Erasmus Medical Center, Institute of Medical Education Research Rotterdam, Rotterdam, The Netherlands
| | - Scott M Wright
- Department of Medicine, Division of Hospital Medicine, Johns Hopkins Bayview Medical Center, Johns Hopkins University School of Medicine, Baltimore, MD, USA
- Department of Medicine, Division of General Internal Medicine, Johns Hopkins Bayview Medical Center, Johns Hopkins University School of Medicine, Baltimore, MD, USA
| |
Collapse
|
4
|
Fukuzawa F, Yanagita Y, Yokokawa D, Uchida S, Yamashita S, Li Y, Shikino K, Tsukamoto T, Noda K, Uehara T, Ikusaka M. Importance of Patient History in Artificial Intelligence-Assisted Medical Diagnosis: Comparison Study. JMIR MEDICAL EDUCATION 2024; 10:e52674. [PMID: 38602313 PMCID: PMC11024399 DOI: 10.2196/52674] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/12/2023] [Revised: 01/31/2024] [Accepted: 02/15/2024] [Indexed: 04/12/2024]
Abstract
Background Medical history contributes approximately 80% to a diagnosis, although physical examinations and laboratory investigations increase a physician's confidence in the medical diagnosis. The concept of artificial intelligence (AI) was first proposed more than 70 years ago. Recently, its role in various fields of medicine has grown remarkably. However, no studies have evaluated the importance of patient history in AI-assisted medical diagnosis. Objective This study explored the contribution of patient history to AI-assisted medical diagnoses and assessed the accuracy of ChatGPT in reaching a clinical diagnosis based on the medical history provided. Methods Using clinical vignettes of 30 cases identified in The BMJ, we evaluated the accuracy of diagnoses generated by ChatGPT. We compared the diagnoses made by ChatGPT based solely on medical history with the correct diagnoses. We also compared the diagnoses made by ChatGPT after incorporating additional physical examination findings and laboratory data alongside history with the correct diagnoses. Results ChatGPT accurately diagnosed 76.6% (23/30) of the cases with only the medical history, consistent with previous research targeting physicians. We also found that this rate was 93.3% (28/30) when additional information was included. Conclusions Although adding additional information improves diagnostic accuracy, patient history remains a significant factor in AI-assisted medical diagnosis. Thus, when using AI in medical diagnosis, it is crucial to include pertinent and correct patient histories for an accurate diagnosis. Our findings emphasize the continued significance of patient history in clinical diagnoses in this age and highlight the need for its integration into AI-assisted medical diagnosis systems.
Collapse
Affiliation(s)
- Fumitoshi Fukuzawa
- Department of General Medicine, Chiba University Hospital, Chiba-shi, Japan
| | - Yasutaka Yanagita
- Department of General Medicine, Chiba University Hospital, Chiba-shi, Japan
| | - Daiki Yokokawa
- Department of General Medicine, Chiba University Hospital, Chiba-shi, Japan
| | - Shun Uchida
- Uchida Internal Medicine Clinic, Saitama-shi, Japan
| | - Shiho Yamashita
- Department of General Medicine, Chiba University Hospital, Chiba-shi, Japan
| | - Yu Li
- Department of General Medicine, Chiba University Hospital, Chiba-shi, Japan
| | - Kiyoshi Shikino
- Department of General Medicine, Chiba University Hospital, Chiba-shi, Japan
| | - Tomoko Tsukamoto
- Department of General Medicine, Chiba University Hospital, Chiba-shi, Japan
| | - Kazutaka Noda
- Department of General Medicine, Chiba University Hospital, Chiba-shi, Japan
| | - Takanori Uehara
- Department of General Medicine, Chiba University Hospital, Chiba-shi, Japan
| | - Masatomi Ikusaka
- Department of General Medicine, Chiba University Hospital, Chiba-shi, Japan
| |
Collapse
|
5
|
DeGennaro AP, Gonzalez N, Peterson S, Gleason KT. How do patients and care partners describe diagnostic uncertainty in an emergency department or urgent care setting? Diagnosis (Berl) 2024; 11:97-101. [PMID: 37747801 DOI: 10.1515/dx-2023-0085] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/11/2023] [Accepted: 09/04/2023] [Indexed: 09/27/2023]
Abstract
OBJECTIVES Little is known about how patients perceive diagnostic uncertainty. We sought to understand how patients and care partners perceive uncertainty in an emergency or urgent care setting, where making a final diagnosis is often not possible. METHODS We administered a survey to a nationally representative panel on patient-reported diagnostic excellence in an emergency department or urgent care setting. The survey included items specific to perceived diagnostic excellence, visit characteristics, and demographics. We analyzed responses to two open-ended questions among those who reported uncertainty in the explanation they were given. Themes were identified using an inductive approach, and compared by whether respondents agreed or disagreed the explanation they were given was true. RESULTS Of the 1,116 respondents, 106 (10 %) reported that the care team was not certain in the explanation of their health problem. Five themes were identified in the open-ended responses: poor communication (73 %), uncertainty made transparent (10 %), incorrect information provided (9 %), inadequate testing equipment (4 %), and unable to determine (4 %). Of the respondents who reported uncertainty, 21 % (n=22/106) reported the explanation of their problem given was not true. CONCLUSIONS The findings of this analysis suggest that the majority of patients and their care partners do not equate uncertainty with a wrong explanation of their health problem, and that poor communication was the most commonly cited reason for perceived uncertainty.
Collapse
Affiliation(s)
| | | | - Susan Peterson
- Johns Hopkins Department of Emergency Medicine, Baltimore, MD, USA
| | - Kelly T Gleason
- Johns Hopkins University School of Nursing, Baltimore, MD, USA
| |
Collapse
|
6
|
Dukhanin V, McDonald KM, Gonzalez N, Gleason KT. Patient Reasoning: Patients' and Care Partners' Perceptions of Diagnostic Accuracy in Emergency Care. Med Decis Making 2024; 44:102-111. [PMID: 37965762 PMCID: PMC10712203 DOI: 10.1177/0272989x231207829] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/27/2023] [Accepted: 09/24/2023] [Indexed: 11/16/2023]
Abstract
OBJECTIVES In the context of validating a measure of patient report specific to diagnostic accuracy in emergency department or urgent care, this study investigates patients' and care partners' perceptions of diagnoses as accurate and explores variations in how they reason while they assess accuracy. METHODS In February 2022, we surveyed a national panel of adults who had an emergency department or urgent care visit in the past month to test a patient-reported measure. As part of the survey validation, we asked for free-text responses about why the respondents indicated their (dis)agreement with 2 statements comprising patient-reported diagnostic accuracy: 1) the explanation they received of the health problem was true and 2) the explanation described what to expect of the health problem. Those paired free-text responses were qualitatively analyzed according to themes created inductively. RESULTS A total of 1,116 patients and care partners provided 982 responses coded into 10 themes, which were further grouped into 3 reasoning types. Almost one-third (32%) of respondents used only corroborative reasoning in assessing the accuracy of the health problem explanation (alignment of the explanation with either test results, patients' subsequent health trajectory, their medical knowledge, symptoms, or another doctor's opinion), 26% used only perception-based reasoning (perceptions of diagnostic process, uncertainty around the explanation received, or clinical team's attitudes), and 27% used both types of reasoning. The remaining 15% used general beliefs or nonexplicated logic (used only about accurate diagnoses) and combinations of general reasoning with perception-based and corroborative. CONCLUSIONS Patients and care partners used multifaceted reasoning in their assessment of diagnostic accuracy. IMPLICATIONS As health care shifts toward meaningful diagnostic co-production and shared decision making, in-depth understanding of variations in patient reasoning and mental models informs use in clinical practice. HIGHLIGHTS An analysis of 982 responses examined how patients and care partners reason about the accuracy of diagnoses they received in emergency or urgent care.In reasoning, people used their perception of the process and whether the diagnosis matched other factual information they have.We introduce "patient reasoning" in the diagnostic measurement context as an area of further research to inform diagnostic shared decision making and co-production of health.
Collapse
Affiliation(s)
- Vadim Dukhanin
- Department of Health Policy and Management, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA
| | - Kathryn M. McDonald
- Johns Hopkins University School of Nursing, Baltimore, MD, USA
- Division of General Internal Medicine, Department of Medicine, Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | | | | |
Collapse
|
7
|
Bhandari R, Bhandari A, Hsieh YH, Edlow J, Omron R. Prevalence of Horizontal Canal Variant in 3,975 Patients With Benign Paroxysmal Positional Vertigo: A Cross-sectional Study. Neurol Clin Pract 2023; 13:e200191. [PMID: 37664131 PMCID: PMC10473829 DOI: 10.1212/cpj.0000000000200191] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/17/2023] [Accepted: 07/07/2023] [Indexed: 09/05/2023]
Abstract
Background and Objectives Current understanding based on older studies is that pc-BPPV is far more common than hc-BPPV. Such studies are limited by small sample sizes, and often the supine roll test for hc-BPPV is not performed. To better estimate the prevalence of hc-BPPV, we studied a large cross-section of patients with VOG-diagnosed BPPV. Methods Using a cross-sectional study of patients with BPPV, we investigated patients referred to NeuroEquilibrium specialty clinics throughout India between January 1, 2021, and December 31, 2021. All patients were evaluated with video oculography (VOG) during positional tests, and all diagnoses were confirmed by a neurotologist and neurologist. Results Of 3,975 patients with VOG-confirmed and specialist-diagnosed BPPV (median age, 51 years; 56.6% women), pc-BPPV accounted for 1,901 (47.8%), hc-BPPV was seen in 1,842 (46.3%), and anterior canal BPPV was found in 28 (0.7%) patients. Discussion This study found that hc-BPPV is far more common than previously reported. It is important to perform the supine roll test in addition to the Dix-Hallpike in all patients suspected with BPPV. Better training to diagnose patients with BPPV and to accurately recognize the nystagmus pattern of hc-BPPV should be a priority.
Collapse
Affiliation(s)
- Rajneesh Bhandari
- NeuroEquilibrium Diagnostic Systems Pvt Ltd. (RB); Vertigo and Ear Clinic (AB), Jaipur, India; Johns Hopkins School of Medicine (Y-HH, RO), Baltimore, MD; and Harvard Medical School (JE), Boston, MA
| | - Anita Bhandari
- NeuroEquilibrium Diagnostic Systems Pvt Ltd. (RB); Vertigo and Ear Clinic (AB), Jaipur, India; Johns Hopkins School of Medicine (Y-HH, RO), Baltimore, MD; and Harvard Medical School (JE), Boston, MA
| | - Yu-Hsiang Hsieh
- NeuroEquilibrium Diagnostic Systems Pvt Ltd. (RB); Vertigo and Ear Clinic (AB), Jaipur, India; Johns Hopkins School of Medicine (Y-HH, RO), Baltimore, MD; and Harvard Medical School (JE), Boston, MA
| | - Jonathan Edlow
- NeuroEquilibrium Diagnostic Systems Pvt Ltd. (RB); Vertigo and Ear Clinic (AB), Jaipur, India; Johns Hopkins School of Medicine (Y-HH, RO), Baltimore, MD; and Harvard Medical School (JE), Boston, MA
| | - Rodney Omron
- NeuroEquilibrium Diagnostic Systems Pvt Ltd. (RB); Vertigo and Ear Clinic (AB), Jaipur, India; Johns Hopkins School of Medicine (Y-HH, RO), Baltimore, MD; and Harvard Medical School (JE), Boston, MA
| |
Collapse
|
8
|
Fraundorf SH, Caddick ZA, Nokes-Malach TJ, Rottman BM. Cognitive perspectives on maintaining physicians' medical expertise: III. Strengths and weaknesses of self-assessment. Cogn Res Princ Implic 2023; 8:58. [PMID: 37646932 PMCID: PMC10469193 DOI: 10.1186/s41235-023-00511-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2022] [Accepted: 08/09/2023] [Indexed: 09/01/2023] Open
Abstract
Is self-assessment enough to keep physicians' cognitive skills-such as diagnosis, treatment, basic biological knowledge, and communicative skills-current? We review the cognitive strengths and weaknesses of self-assessment in the context of maintaining medical expertise. Cognitive science supports the importance of accurately self-assessing one's own skills and abilities, and we review several ways such accuracy can be quantified. However, our review also indicates a broad challenge in self-assessment is that individuals do not have direct access to the strength or quality of their knowledge and instead must infer this from heuristic strategies. These heuristics are reasonably accurate in many circumstances, but they also suffer from systematic biases. For example, information that feels easy to process in the moment can lead individuals to overconfidence in their ability to remember it in the future. Another notable phenomenon is the Dunning-Kruger effect: the poorest performers in a domain are also the least accurate in self-assessment. Further, explicit instruction is not always sufficient to remove these biases. We discuss what these findings imply about when physicians' self-assessment can be useful and when it may be valuable to supplement with outside sources.
Collapse
Affiliation(s)
- Scott H Fraundorf
- Learning Research and Development Center, University of Pittsburgh, 3420 Forbes Ave., Pittsburgh, PA, 15260, USA.
- Department of Psychology, University of Pittsburgh, 3420 Forbes Ave., Pittsburgh, PA, 15260, USA.
| | - Zachary A Caddick
- Learning Research and Development Center, University of Pittsburgh, 3420 Forbes Ave., Pittsburgh, PA, 15260, USA
- Department of Psychology, University of Pittsburgh, 3420 Forbes Ave., Pittsburgh, PA, 15260, USA
| | - Timothy J Nokes-Malach
- Learning Research and Development Center, University of Pittsburgh, 3420 Forbes Ave., Pittsburgh, PA, 15260, USA
- Department of Psychology, University of Pittsburgh, 3420 Forbes Ave., Pittsburgh, PA, 15260, USA
| | - Benjamin M Rottman
- Learning Research and Development Center, University of Pittsburgh, 3420 Forbes Ave., Pittsburgh, PA, 15260, USA
- Department of Psychology, University of Pittsburgh, 3420 Forbes Ave., Pittsburgh, PA, 15260, USA
| |
Collapse
|
9
|
van Sassen C, Mamede S, Bos M, van den Broek W, Bindels P, Zwaan L. Do malpractice claim clinical case vignettes enhance diagnostic accuracy and acceptance in clinical reasoning education during GP training? BMC MEDICAL EDUCATION 2023; 23:474. [PMID: 37365590 DOI: 10.1186/s12909-023-04448-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/17/2023] [Accepted: 06/14/2023] [Indexed: 06/28/2023]
Abstract
BACKGROUND Using malpractice claims cases as vignettes is a promising approach for improving clinical reasoning education (CRE), as malpractice claims can provide a variety of content- and context-rich examples. However, the effect on learning of adding information about a malpractice claim, which may evoke a deeper emotional response, is not yet clear. This study examined whether knowing that a diagnostic error resulted in a malpractice claim affects diagnostic accuracy and self-reported confidence in the diagnosis of future cases. Moreover, suitability of using erroneous cases with and without a malpractice claim for CRE, as judged by participants, was evaluated. METHODS In the first session of this two-phased, within-subjects experiment, 81 first-year residents of general practice (GP) were exposed to both erroneous cases with (M) and erroneous cases without (NM) malpractice claim information, derived from a malpractice claims database. Participants rated suitability of the cases for CRE on a five-point Likert scale. In the second session, one week later, participants solved four different cases with the same diagnoses. Diagnostic accuracy was measured with three questions, scored on a 0-1 scale: (1) What is your next step? (2) What is your differential diagnosis? (3) What is your most probable diagnosis and what is your level of certainty on this? Both subjective suitability and diagnostic accuracy scores were compared between the versions (M and NM) using repeated measures ANOVA. RESULTS There were no differences in diagnostic accuracy parameters (M vs. NM next step: 0.79 vs. 0.77, p = 0.505; differential diagnosis 0.68 vs. 0.75, p = 0.072; most probable diagnosis 0.52 vs. 0.57, p = 0.216) and self-reported confidence (53.7% vs. 55.8% p = 0.390) of diagnoses previously seen with or without malpractice claim information. Subjective suitability- and complexity scores for the two versions were similar (suitability: 3.68 vs. 3.84, p = 0.568; complexity 3.71 vs. 3.88, p = 0.218) and significantly increased for higher education levels for both versions. CONCLUSION The similar diagnostic accuracy rates between cases studied with or without malpractice claim information suggests both versions are equally effective for CRE in GP training. Residents judged both case versions to be similarly suitable for CRE; both were considered more suitable for advanced than for novice learners.
Collapse
Affiliation(s)
- Charlotte van Sassen
- Department of General Practice, Erasmus Medical Center, Rotterdam, The Netherlands.
- Institute of Medical Education Research Rotterdam (iMERR), Erasmus Medical Center, Rotterdam, The Netherlands.
| | - Silvia Mamede
- Institute of Medical Education Research Rotterdam (iMERR), Erasmus Medical Center, Rotterdam, The Netherlands
- Department of Psychology, Education and Child Studies, Erasmus School of Social and Behavioral Sciences, Rotterdam, The Netherlands
| | - Michiel Bos
- Department of General Practice, Erasmus Medical Center, Rotterdam, The Netherlands
| | - Walter van den Broek
- Institute of Medical Education Research Rotterdam (iMERR), Erasmus Medical Center, Rotterdam, The Netherlands
| | - Patrick Bindels
- Department of General Practice, Erasmus Medical Center, Rotterdam, The Netherlands
| | - Laura Zwaan
- Institute of Medical Education Research Rotterdam (iMERR), Erasmus Medical Center, Rotterdam, The Netherlands
| |
Collapse
|
10
|
Redmond S, Barwise A, Zornes S, Dong Y, Herasevich S, Pinevich Y, Soleimani J, LeMahieu A, Leppin A, Pickering B. Contributors to Diagnostic Error or Delay in the Acute Care Setting: A Survey of Clinical Stakeholders. Health Serv Insights 2022; 15:11786329221123540. [PMID: 36119635 PMCID: PMC9476244 DOI: 10.1177/11786329221123540] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/01/2022] [Accepted: 08/03/2022] [Indexed: 11/16/2022] Open
Abstract
Diagnostic error or delay (DEOD) is common in the acute care setting and results in poor patient outcomes. Many factors contribute to DEOD, but little is known about how contributors may differ across acute care areas and professional roles. As part of a sequential exploratory mixed methods research study, we surveyed acute care clinical stakeholders about the frequency with which different factors contribute to DEOD. Survey respondents could also propose solutions in open text fields. N = 220 clinical stakeholders completed the survey. Care Team Interactions, Systems and Process, Patient, Provider, and Cognitive factors were perceived to contribute to DEOD with similar frequency. Organization and Infrastructure factors were perceived to contribute to DEOD significantly less often. Responses did not vary across acute care setting. Physicians perceived Cognitive factors to contribute to DEOD more frequently compared to those in other roles. Commonly proposed solutions included: technological solutions, organization level fixes, ensuring staff know and are encouraged to work to the full scope of their role, and cultivating a culture of collaboration and respect. Multiple factors contribute to DEOD with similar frequency across acute care areas, suggesting the need for a multi-pronged approach that can be applied across acute care areas.
Collapse
Affiliation(s)
- Sarah Redmond
- Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, MN, USA
| | - Amelia Barwise
- Division of Pulmonary and Critical Care Medicine, Mayo Clinic, Rochester, MN, USA
| | - Sarah Zornes
- Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, MN, USA
| | - Yue Dong
- Department of Anesthesiology and Perioperative Medicine, Mayo Clinic, Rochester, MN, USA
| | - Svetlana Herasevich
- Department of Anesthesiology and Perioperative Medicine, Mayo Clinic, Rochester, MN, USA
| | - Yuliya Pinevich
- Department of Anesthesiology and Perioperative Medicine, Mayo Clinic, Rochester, MN, USA
| | - Jalal Soleimani
- Department of Anesthesiology and Perioperative Medicine, Mayo Clinic, Rochester, MN, USA
| | - Allison LeMahieu
- Division of Clinical Trials and Biostatistics, Department of Quantitative Health Sciences, Mayo Clinic Rochester, Rochester, MN, USA
| | - Aaron Leppin
- Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, MN, USA.,Knowledge and Evaluation Research Unit (KER), Mayo Clinic, Rochester, MN, USA
| | - Brian Pickering
- Department of Anesthesiology and Perioperative Medicine, Mayo Clinic, Rochester, MN, USA
| |
Collapse
|
11
|
Black GB, van Os S, Renzi C, Walter FM, Hamilton W, Whitaker KL. How does safety netting for lung cancer symptoms help patients to reconsult appropriately? A qualitative study. BMC PRIMARY CARE 2022; 23:179. [PMID: 35858826 PMCID: PMC9298706 DOI: 10.1186/s12875-022-01791-y] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Received: 09/15/2021] [Accepted: 07/08/2022] [Indexed: 01/07/2023]
Abstract
Abstract
Background
Safety netting in primary care is considered an important intervention for managing diagnostic uncertainty. This is the first study to examine how patients understand and interpret safety netting advice around low-risk potential lung cancer symptoms, and how this affects reconsultation behaviours.
Methods
Qualitative interview study in UK primary care. Pre-covid-19, five patients were interviewed in person within 2–3 weeks of a primary care consultation for potential lung cancer symptom(s), and again 2–5 months later. The general practitioner (GP) they last saw was interviewed face-to-face once. During the covid-19 pandemic, an additional 15 patients were interviewed only once via telephone, and their GPs were not interviewed or contacted in any way. Audio-recorded interviews were transcribed verbatim and analysed using inductive thematic analysis.
Results
The findings from our thematic analysis suggest that patients prefer active safety netting, as part of thorough and logical diagnostic uncertainty management. Passive or ambiguous safety netting may be perceived as dismissive and cause delayed reconsultation. GP safety netting strategies are not always understood, potentially causing patient worry and dissatisfaction. Telephone consultations and the diagnostic overshadowing of COVID-19 on respiratory symptoms impacted GPs’ safety netting strategies and patients’ appetite for active follow up measures.
Conclusions
Safety netting guidelines do not yet offer solutions that have been proven to promote symptom vigilance and timely reconsultation for low-risk lung cancer symptoms. This may have been affected by primary care practices during the COVID-19 pandemic. Patients prefer active or pre-planned safety netting coupled with thorough consultation techniques and a comprehensible diagnostic strategy, and may respond adversely to passive safety netting advice.
Collapse
|
12
|
Villalba C, Burke RC, Gurley K, Dhaliwal G, Grossman S. Electronic health record-based patient tracking by emergency medicine physicians. AEM EDUCATION AND TRAINING 2022; 6:e10732. [PMID: 35368507 PMCID: PMC8923647 DOI: 10.1002/aet2.10732] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/05/2021] [Revised: 02/06/2022] [Accepted: 02/09/2022] [Indexed: 06/14/2023]
Abstract
Objectives Emergency medicine (EM) physicians commonly track the progress of former patients to learn about their clinical outcome. While some studies have described the behavior, little is known about the specific information sought during tracking. The objective of this study was to determine how often EM physicians track patients and the motivations, strategies, and barriers to tracking. Methods In June 2019 we surveyed EM physicians practicing at six hospitals. We defined patient tracking as viewing the chart of a patient who was no longer under the physician's care or contacting the patient or the patient's subsequent providers to learn about the patient's progress. The survey asked respondents how often they track patients, by what mechanisms, and for what reasons. The survey also asked what information physicians sought when tracking and what barriers to tracking exist. Results Of the 156 EM physicians invited to respond, 111 completed the survey (72% response rate). Of those, 109 (98%) reported tracking their patients, and residents reported tracking a higher percentage of patients than attendings. Reasons for tracking included an unusual or complex case (98%), uncertain diagnosis (89%), and concern about a potential error (48%). Most respondents (86%) said that knowledge gained from patient tracking changed their subsequent practice patterns, and almost all respondents (98%) strongly agreed or agreed that tracking helps physicians avoid future mistakes. The most commonly sought information types during tracking were the hospital discharge summary or emergency department note from the index visit, test results since the index visit, and new diagnoses added since the index visit. Physicians cited time limitations and difficulty accessing information as the leading barriers to tracking. Conclusions Patient tracking is nearly ubiquitous among surveyed EM physicians, who find it valuable for learning and patient safety.
Collapse
Affiliation(s)
| | - Ryan C. Burke
- Department of Emergency MedicineBeth Israel Deaconess Medical CenterBostonMassachusettsUSA
| | - Kiersten Gurley
- Department of Emergency MedicineBeth Israel Deaconess Medical CenterBostonMassachusettsUSA
- Anna Jaques HospitalNewburyportMassachusettsUSA
| | - Gurpreet Dhaliwal
- Department of MedicineUniversity of California, San FranciscoSan FranciscoCaliforniaUSA
- Medical ServiceSan Francisco VA Medical CenterSan FranciscoCaliforniaUSA
| | - Shamai Grossman
- Department of Emergency MedicineBeth Israel Deaconess Medical CenterBostonMassachusettsUSA
| |
Collapse
|
13
|
Muro-Fuentes EA, Stunkel L. Diagnostic Error in Neuro-ophthalmology: Avenues to Improve. Curr Neurol Neurosci Rep 2022; 22:243-256. [PMID: 35320466 PMCID: PMC8940596 DOI: 10.1007/s11910-022-01189-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 02/14/2022] [Indexed: 11/06/2022]
Abstract
Purpose of Review To highlight potential avenues to reduce preventable diagnostic error of neuro-ophthalmic conditions and avoid patient harm. Recent Findings Recent prospective studies and studies of patient harm have advanced our understanding. Additionally, recent studies of fundus photography, telemedicine, and artificial intelligence highlight potential avenues for diagnostic improvement. Summary Diagnostic error of neuro-ophthalmic conditions can often be traced to failure to gather an adequate history, perform a complete physical exam, obtain adequate/appropriate neuroimaging, and generate a complete, appropriate differential diagnosis. Improving triage and identification of neuro-ophthalmic conditions by other providers and increasing access to subspecialty neuro-ophthalmology evaluation are essential avenues to reduce diagnostic error. Further research should evaluate the relationship between misdiagnosis and patient harm, and help identify the most impactful potential targets for improvement.
Collapse
Affiliation(s)
| | - Leanne Stunkel
- John F. Hardesty, MD Department of Ophthalmology and Visual Sciences and Department of Neurology, Washington University in St. Louis, 660 S. Euclid Ave, Campus Box 8096, St. Louis, MO, 63110, USA
| |
Collapse
|
14
|
Kuhn J, van den Berg P, Mamede S, Zwaan L, Bindels P, van Gog T. Improving medical residents' self-assessment of their diagnostic accuracy: does feedback help? ADVANCES IN HEALTH SCIENCES EDUCATION : THEORY AND PRACTICE 2022; 27:189-200. [PMID: 34739632 PMCID: PMC8938348 DOI: 10.1007/s10459-021-10080-9] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/15/2021] [Accepted: 10/10/2021] [Indexed: 06/13/2023]
Abstract
When physicians do not estimate their diagnostic accuracy correctly, i.e. show inaccurate diagnostic calibration, diagnostic errors or overtesting can occur. A previous study showed that physicians' diagnostic calibration for easy cases improved, after they received feedback on their previous diagnoses. We investigated whether diagnostic calibration would also improve from this feedback when cases were more difficult. Sixty-nine general-practice residents were randomly assigned to one of two conditions. In the feedback condition, they diagnosed a case, rated their confidence in their diagnosis, their invested mental effort, and case complexity, and then were shown the correct diagnosis (feedback). This was repeated for 12 cases. Participants in the control condition did the same without receiving feedback. We analysed calibration in terms of (1) absolute accuracy (absolute difference between diagnostic accuracy and confidence), and (2) bias (confidence minus diagnostic calibration). There was no difference between the conditions in the measurements of calibration (absolute accuracy, p = .204; bias, p = .176). Post-hoc analyses showed that on correctly diagnosed cases (on which participants are either accurate or underconfident), calibration in the feedback condition was less accurate than in the control condition, p = .013. This study shows that feedback on diagnostic performance did not improve physicians' calibration for more difficult cases. One explanation could be that participants were confronted with their mistakes and thereafter lowered their confidence ratings even if cases were diagnosed correctly. This shows how difficult it is to improve diagnostic calibration, which is important to prevent diagnostic errors or maltreatment.
Collapse
Affiliation(s)
- Josepha Kuhn
- Department of General Practice, Erasmus Medical Centre, P.O. Box 2040, 3000 CA, Rotterdam, The Netherlands.
- Institute of Medical Education Research Rotterdam, Erasmus Medical Centre, Rotterdam, The Netherlands.
| | - Pieter van den Berg
- Department of General Practice, Erasmus Medical Centre, P.O. Box 2040, 3000 CA, Rotterdam, The Netherlands
| | - Silvia Mamede
- Institute of Medical Education Research Rotterdam, Erasmus Medical Centre, Rotterdam, The Netherlands
- Department of Psychology, Education and Child Studies, Erasmus University Rotterdam, Rotterdam, The Netherlands
| | - Laura Zwaan
- Institute of Medical Education Research Rotterdam, Erasmus Medical Centre, Rotterdam, The Netherlands
| | - Patrick Bindels
- Department of General Practice, Erasmus Medical Centre, P.O. Box 2040, 3000 CA, Rotterdam, The Netherlands
| | - Tamara van Gog
- Department of Education, Utrecht University, Utrecht, The Netherlands
| |
Collapse
|
15
|
Liberman AL, Cheng NT, Friedman BW, Gerstein MT, Moncrieffe K, Labovitz DL, Lipton RB. Emergency medicine physicians' perspectives on diagnostic accuracy in neurology: a qualitative study. Diagnosis (Berl) 2021; 9:225-235. [PMID: 34855312 DOI: 10.1515/dx-2021-0125] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/05/2021] [Accepted: 10/29/2021] [Indexed: 11/15/2022]
Abstract
OBJECTIVES We sought to understand the knowledge, attitudes, and beliefs of emergency medicine (EM) physicians towards non-specific neurological conditions and the use of clinical decision support (CDS) to improve diagnostic accuracy. METHODS We conducted semi-structured interviews of EM physicians at four emergency departments (EDs) affiliated with a single US healthcare system. Interviews were conducted until thematic saturation was achieved. Conventional content analysis was used to identify themes related to EM physicians' perspectives on acute diagnostic neurology; directed content analysis was used to explore views regarding CDS. Each interview transcript was independently coded by two researchers using an iteratively refined codebook with consensus-based resolution of coding differences. RESULTS We identified two domains regarding diagnostic safety: (1) challenges unique to neurological complaints and (2) challenges in EM more broadly. Themes relevant to neurology included: (1) knowledge gaps and uncertainty, (2) skepticism about neurology, (3) comfort with basic as opposed to detailed neurological examination, and (4) comfort with non-neurological diseases. Themes relevant to diagnostic decision making in the ED included: (1) cognitive biases, (2) ED system/environmental issues, (3) patient barriers, (4) comfort with diagnostic uncertainty, and (5) concerns regarding diagnostic error identification and measurement. Most participating EM physicians were enthusiastic about the potential for well-designed CDS to improve diagnostic accuracy for non-specific neurological complaints. CONCLUSIONS Physicians identified diagnostic challenges unique to neurological diseases as well as issues related more generally to diagnostic accuracy in EM. These physician-reported issues should be accounted for when designing interventions to improve ED diagnostic accuracy.
Collapse
Affiliation(s)
- Ava L Liberman
- Department of Neurology, Weill Cornell Medicine, New York, NY, USA
| | - Natalie T Cheng
- Department of Neurology, Montefiore Medical Center, Albert Einstein College of Medicine, Bronx, NY, USA
| | - Benjamin W Friedman
- Department of Emergency Medicine, Montefiore Medical Center, Albert Einstein College of Medicine, Bronx, NY, USA
| | | | - Khadean Moncrieffe
- Department of Neurology, Montefiore Medical Center, Albert Einstein College of Medicine, Bronx, NY, USA
| | - Daniel L Labovitz
- Department of Neurology, Montefiore Medical Center, Albert Einstein College of Medicine, Bronx, NY, USA
| | - Richard B Lipton
- Department of Neurology, Montefiore Medical Center, Albert Einstein College of Medicine, Bronx, NY, USA
| |
Collapse
|
16
|
Khoong EC, Nouri SS, Tuot DS, Nundy S, Fontil V, Sarkar U. Comparison of Diagnostic Recommendations from Individual Physicians versus the Collective Intelligence of Multiple Physicians in Ambulatory Cases Referred for Specialist Consultation. Med Decis Making 2021; 42:293-302. [PMID: 34378444 DOI: 10.1177/0272989x211031209] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
BACKGROUND Studies report higher diagnostic accuracy using the collective intelligence (CI) of multiple clinicians compared with individual clinicians. However, the diagnostic process is iterative, and unexplored is the value of CI in improving clinical recommendations leading to a final diagnosis. METHODS To compare the appropriateness of diagnostic recommendations advised by individual physicians versus the CI of physicians, we entered actual consultation requests sent by primary care physicians to specialists onto a web-based CI platform capable of collecting diagnostic recommendations (next steps for care) from multiple physicians. We solicited responses to 35 cases (12 endocrinology, 13 gynecology, 10 neurology) from ≥3 physicians of any specialty through the CI platform, which aggregated responses into a CI output. The primary outcome was the appropriateness of individual physician recommendations versus the CI output recommendations, using recommendations agreed upon by 2 specialists in the same specialty as a gold standard. The secondary outcome was the recommendations' potential for harm. RESULTS A total of 177 physicians responded. Cases had a median of 7 respondents (interquartile range: 5-10). Diagnostic recommendations in the CI output achieved higher levels of appropriateness (69%) than recommendations from individual physicians (45%; χ2 = 5.95, P = 0.015). Of the CI recommendations, 54% were potentially harmful, as compared with 41% of individuals' recommendations (χ2 = 2.49, P = 0.11). LIMITATIONS Cases were from a single institution. CI was solicited using a single algorithm/platform. CONCLUSIONS When seeking specialist guidance, diagnostic recommendations from the CI of multiple physicians are more appropriate than recommendations from most individual physicians, measured against specialist recommendations. Although CI provides useful recommendations, some have potential for harm. Future research should explore how to use CI to improve diagnosis while limiting harm from inappropriate tests/therapies.
Collapse
Affiliation(s)
- Elaine C Khoong
- Division of General Internal Medicine at Zuckerberg San Francisco General Hospital, Department of Medicine, UCSF, San Francisco, CA, USA.,Center for Vulnerable Populations at Zuckerberg San Francisco General Hospital, UCSF, San Francisco, CA,USA
| | - Sarah S Nouri
- Division of General Internal Medicine, Department of Medicine, UCSF, San Francisco, CA, USA
| | - Delphine S Tuot
- Center for Vulnerable Populations at Zuckerberg San Francisco General Hospital, UCSF, San Francisco, CA,USA.,Division of Nephrology, Department of Medicine, UCSF, San Francisco, CA, USA.,Center for Innovation in Access and Quality at Zuckerberg San Francisco General Hospital, UCSF, San Francisco, CA, USA
| | - Shantanu Nundy
- George Washington University Milken Institute School of Public Health, Washington, DC, USA.,Accolade, Inc, Plymouth Meeting, PA
| | - Valy Fontil
- Division of General Internal Medicine at Zuckerberg San Francisco General Hospital, Department of Medicine, UCSF, San Francisco, CA, USA.,Center for Vulnerable Populations at Zuckerberg San Francisco General Hospital, UCSF, San Francisco, CA,USA
| | - Urmimala Sarkar
- Division of General Internal Medicine at Zuckerberg San Francisco General Hospital, Department of Medicine, UCSF, San Francisco, CA, USA.,Center for Vulnerable Populations at Zuckerberg San Francisco General Hospital, UCSF, San Francisco, CA,USA
| |
Collapse
|
17
|
Chang TP, Bery AK, Wang Z, Sebestyen K, Ko YH, Liberman AL, Newman-Toker DE. Stroke hospitalization after misdiagnosis of "benign dizziness" is lower in specialty care than general practice: a population-based cohort analysis of missed stroke using SPADE methods. ACTA ACUST UNITED AC 2021; 9:96-106. [PMID: 34147048 DOI: 10.1515/dx-2020-0124] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/16/2020] [Accepted: 04/22/2021] [Indexed: 12/14/2022]
Abstract
OBJECTIVES Isolated dizziness is a challenging stroke presentation in the emergency department, but little is known about this problem in other clinical settings. We sought to compare stroke hospitalizations after treat-and-release clinic visits for purportedly "benign dizziness" between general and specialty care settings. METHODS This was a population-based retrospective cohort study from a national database. We included clinic patients with a first incident treat-and-release visit diagnosis of non-specific dizziness/vertigo or a peripheral vestibular disorder (ICD-9-CM 780.4 or 386.x [not 386.2]). We compared general care (internal medicine, family medicine) vs. specialty care (neurology, otolaryngology) providers. We used propensity scores to control for baseline stroke risk differences unrelated to dizziness diagnosis. We measured excess (observed>expected) stroke hospitalizations in the first 30 d (i.e., missed strokes associated with an adverse event). RESULTS We analyzed 144,355 patients discharged with "benign dizziness" (n=117,117 diagnosed in general care; n=27,238 in specialty care). After propensity score matching, patients in both groups were at higher risk of stroke in the first 30 d (rate difference per 10,000 treat-and-release visits for "benign dizziness" 24.9 [95% CI 18.6-31.2] in general care and 10.6 [95% CI 6.3-14.9] in specialty care). Short-term stroke risk was higher in general care than specialty care (relative risk, RR 2.2, 95% CI 1.5-3.2) while the long-term risk was not significantly different (RR 1.3, 95% CI 0.9-1.9), indicating higher misdiagnosis-related harms among dizzy patients who initially presented to generalists after adequate propensity matching. CONCLUSIONS Missed stroke-related harms in general care were roughly twice that in specialty care. Solutions are needed to address this care gap.
Collapse
Affiliation(s)
- Tzu-Pu Chang
- Department of Neurology/Neuro-Medical Scientific Center, Taichung Tzu Chi Hospital, Buddhist Tzu Chi Medical Foundation, Taichung, Taiwan
- Department of Neurology, School of Medicine, Tzu Chi University, Hualien, Taiwan
| | - Anand K Bery
- Division of Neurology, Department of Medicine, University of Ottawa, Ottawa, Canada
| | - Zheyu Wang
- Division of Biostatistics and Bioinformatics, Sidney Kimmel Comprehensive Cancer Center, Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - Krisztian Sebestyen
- Department of Surgery, Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - Yu-Hung Ko
- Department of Research, Taichung Tzu Chi Hospital, Buddhist Tzu Chi Medical Foundation, Taichung, Taiwan
| | - Ava L Liberman
- Department of Neurology, Albert Einstein College of Medicine, Bronx, NY, USA
| | - David E Newman-Toker
- Department of Neurology, Johns Hopkins Hospital, Pathology Building 2-221, 600 North Wolfe Street, Baltimore, MD 21287-6921, USA
- Armstrong Institute Center for Diagnostic Excellence, Johns Hopkins University School of Medicine, Baltimore, MD, USA
| |
Collapse
|
18
|
Kotwal S, Fanai M, Fu W, Wang Z, Bery AK, Omron R, Tevzadze N, Gold D, Garibaldi BT, Wright SM, Newman-Toker DE. Real-world virtual patient simulation to improve diagnostic performance through deliberate practice: a prospective quasi-experimental study. ACTA ACUST UNITED AC 2021; 8:489-496. [PMID: 33675203 DOI: 10.1515/dx-2020-0127] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/21/2020] [Accepted: 02/05/2021] [Indexed: 11/15/2022]
Abstract
OBJECTIVES Diagnostic errors are pervasive in medicine and most often caused by clinical reasoning failures. Clinical presentations characterized by nonspecific symptoms with broad differential diagnoses (e.g., dizziness) are especially prone to such errors. METHODS We hypothesized that novice clinicians could achieve proficiency diagnosing dizziness by training with virtual patients (VPs). This was a prospective, quasi-experimental, pretest-posttest study (2019) at a single academic medical center. Internal medicine interns (intervention group) were compared to second/third year residents (control group). A case library of VPs with dizziness was developed from a clinical trial (AVERT-NCT02483429). The approach (VIPER - Virtual Interactive Practice to build Expertise using Real cases) consisted of brief lectures combined with 9 h of supervised deliberate practice. Residents were provided dizziness-related reading and teaching modules. Both groups completed pretests and posttests. RESULTS For interns (n=22) vs. residents (n=18), pretest median diagnostic accuracy did not differ (33% [IQR 18-46] vs. 31% [IQR 13-50], p=0.61) between groups, while posttest accuracy did (50% [IQR 42-67] vs. 20% [IQR 17-33], p=0.001). Pretest median appropriate imaging did not differ (33% [IQR 17-38] vs. 31% [IQR 13-38], p=0.89) between groups, while posttest appropriateness did (65% [IQR 52-74] vs. 25% [IQR 17-36], p<0.001). CONCLUSIONS Just 9 h of deliberate practice increased diagnostic skills (both accuracy and testing appropriateness) of medicine interns evaluating real-world dizziness 'in silico' more than ∼1.7 years of residency training. Applying condensed educational experiences such as VIPER across a broad range of common presentations could significantly enhance diagnostic education and translate to improved patient care.
Collapse
Affiliation(s)
- Susrutha Kotwal
- Department of Medicine, Division of Hospital Medicine, Johns Hopkins Bayview Medical Center, Johns Hopkins University School of Medicine, Baltimore, MD, USA.,Center for Diagnostic Excellence, Armstrong Institute for Patient Safety and Quality, Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - Mehdi Fanai
- Center for Diagnostic Excellence, Armstrong Institute for Patient Safety and Quality, Johns Hopkins University School of Medicine, Baltimore, MD, USA.,Department of Neurology, Division of Neuro-Visual & Vestibular Disorders, Johns Hopkins Hospital, Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - Wei Fu
- Department of Oncology, Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - Zheyu Wang
- Center for Diagnostic Excellence, Armstrong Institute for Patient Safety and Quality, Johns Hopkins University School of Medicine, Baltimore, MD, USA.,Department of Oncology, Johns Hopkins University School of Medicine, Baltimore, MD, USA.,Department of Biostatistics, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA
| | - Anand K Bery
- Department of Medicine, Division of Neurology, The Ottawa Hospital, Ottawa, Canada
| | - Rodney Omron
- Center for Diagnostic Excellence, Armstrong Institute for Patient Safety and Quality, Johns Hopkins University School of Medicine, Baltimore, MD, USA.,Department of Emergency Medicine, Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - Nana Tevzadze
- Department of Neurology, Division of Neuro-Visual & Vestibular Disorders, Johns Hopkins Hospital, Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - Daniel Gold
- Department of Neurology, Division of Neuro-Visual & Vestibular Disorders, Johns Hopkins Hospital, Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - Brian T Garibaldi
- Center for Diagnostic Excellence, Armstrong Institute for Patient Safety and Quality, Johns Hopkins University School of Medicine, Baltimore, MD, USA.,Division of Pulmonary and Critical Care Medicine, Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - Scott M Wright
- Department of Medicine, Division of Hospital Medicine, Johns Hopkins Bayview Medical Center, Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - David E Newman-Toker
- Center for Diagnostic Excellence, Armstrong Institute for Patient Safety and Quality, Johns Hopkins University School of Medicine, Baltimore, MD, USA.,Department of Neurology, Division of Neuro-Visual & Vestibular Disorders, Johns Hopkins Hospital, Johns Hopkins University School of Medicine, Baltimore, MD, USA.,Department of Emergency Medicine, Johns Hopkins University School of Medicine, Baltimore, MD, USA
| |
Collapse
|
19
|
Meyer AND, Upadhyay DK, Collins CA, Fitzpatrick MH, Kobylinski M, Bansal AB, Torretti D, Singh H. A Program to Provide Clinicians with Feedback on Their Diagnostic Performance in a Learning Health System. Jt Comm J Qual Patient Saf 2020; 47:120-126. [PMID: 32980255 DOI: 10.1016/j.jcjq.2020.08.014] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/06/2020] [Revised: 08/25/2020] [Accepted: 08/26/2020] [Indexed: 02/07/2023]
Abstract
PROBLEM Reducing diagnostic errors requires improving both systems and individual clinical reasoning. One strategy to achieve diagnostic excellence is learning from feedback. However, clinicians remain uncomfortable receiving feedback on their diagnostic performance. Thus, a team of researchers and clinical leaders aimed to develop and implement a diagnostic performance feedback program for learning that mitigates potential clinician discomfort. APPROACH The program was developed as part of a larger project to create a learning health system around diagnostic safety at Geisinger, a large, integrated health care system in rural Pennsylvania. Steps included identifying potential missed opportunities in diagnosis (MODs) from various sources (for example, risk management, clinician reports, patient complaints); confirming MODs through chart review; and having trained facilitators provide feedback to clinicians about MODs as learning opportunities. The team developed a guide for facilitators to conduct effective diagnostic feedback sessions and surveyed facilitators and recipients about their experiences and perceptions of the feedback sessions. OUTCOMES 28 feedback sessions occurred from January 2019 to June 2020, involving MODs from emergency medicine, primary care, and hospital medicine. Most facilitators (90.6% [29/32]) reported that recipients were receptive to learning and discussing MODs. Most recipients reported that conversations were constructive and nonpunitive (83.3% [25/30]) and allowed them to take concrete steps toward improving diagnosis (76.7% [23/30]). Both groups believed discussions would improve future diagnostic safety (93.8% [30/32] and 70.0% [21/30], respectively). KEY INSIGHTS AND NEXT STEPS An institutional program was developed and implemented to deliver diagnostic performance feedback. Such a program may facilitate learning and improvement to reduce MODs. Future efforts should assess long-term effects on diagnostic performance and patient outcomes.
Collapse
|
20
|
Newman-Toker DE, Wang Z, Zhu Y, Nassery N, Saber Tehrani AS, Schaffer AC, Yu-Moe CW, Clemens GD, Fanai M, Siegal D. Rate of diagnostic errors and serious misdiagnosis-related harms for major vascular events, infections, and cancers: toward a national incidence estimate using the “Big Three”. Diagnosis (Berl) 2020; 8:67-84. [DOI: 10.1515/dx-2019-0104] [Citation(s) in RCA: 23] [Impact Index Per Article: 5.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/08/2019] [Accepted: 02/12/2020] [Indexed: 02/06/2023]
Abstract
Abstract
Background
Missed vascular events, infections, and cancers account for ~75% of serious harms from diagnostic errors. Just 15 diseases from these “Big Three” categories account for nearly half of all serious misdiagnosis-related harms in malpractice claims. As part of a larger project estimating total US burden of serious misdiagnosis-related harms, we performed a focused literature review to measure diagnostic error and harm rates for these 15 conditions.
Methods
We searched PubMed, Google, and cited references. For errors, we selected high-quality, modern, US-based studies, if available, and best available evidence otherwise. For harms, we used literature-based estimates of the generic (disease-agnostic) rate of serious harms (morbidity/mortality) per diagnostic error and applied claims-based severity weights to construct disease-specific rates. Results were validated via expert review and comparison to prior literature that used different methods. We used Monte Carlo analysis to construct probabilistic plausible ranges (PPRs) around estimates.
Results
Rates for the 15 diseases were drawn from 28 published studies representing 91,755 patients. Diagnostic error (false negative) rates ranged from 2.2% (myocardial infarction) to 62.1% (spinal abscess), with a median of 13.6% [interquartile range (IQR) 9.2–24.7] and an aggregate mean of 9.7% (PPR 8.2–12.3). Serious misdiagnosis-related harm rates per incident disease case ranged from 1.2% (myocardial infarction) to 35.6% (spinal abscess), with a median of 5.5% (IQR 4.6–13.6) and an aggregate mean of 5.2% (PPR 4.5–6.7). Rates were considered face valid by domain experts and consistent with prior literature reports.
Conclusions
Diagnostic improvement initiatives should focus on dangerous conditions with higher diagnostic error and misdiagnosis-related harm rates.
Collapse
Affiliation(s)
- David E. Newman-Toker
- Department of Neurology , The Johns Hopkins University School of Medicine , Baltimore, MD , USA
- Director, Armstrong Institute Center for Diagnostic Excellence , The Johns Hopkins University School of Medicine , Baltimore, MD , USA
- Professor, Department of Epidemiology , The Johns Hopkins Bloomberg School of Public Health , Baltimore, MD , USA
| | - Zheyu Wang
- Department of Oncology , The Johns Hopkins University School of Medicine , Baltimore, MD , USA
- Department of Biostatistics, The Johns Hopkins Bloomberg School of Public Health , Baltimore, MD , USA
| | - Yuxin Zhu
- Department of Oncology , The Johns Hopkins University School of Medicine , Baltimore, MD , USA
- Department of Biostatistics, The Johns Hopkins Bloomberg School of Public Health , Baltimore, MD , USA
| | - Najlla Nassery
- Department of Medicine , The Johns Hopkins University School of Medicine , Baltimore, MD , USA
| | - Ali S. Saber Tehrani
- Department of Neurology , The Johns Hopkins University School of Medicine , Baltimore, MD , USA
| | - Adam C. Schaffer
- Department of Patient Safety, CRICO , Boston, MA , USA
- Department of Medicine, Brigham and Women’s Hospital and Harvard Medical School , Boston, MA , USA
| | | | - Gwendolyn D. Clemens
- Department of Biostatistics, The Johns Hopkins Bloomberg School of Public Health , Baltimore, MD , USA
| | - Mehdi Fanai
- Department of Neurology , The Johns Hopkins University School of Medicine , Baltimore, MD , USA
| | - Dana Siegal
- Director of Patient Safety, CRICO Strategies , Boston, MA , USA
| |
Collapse
|
21
|
Cifra CL, Tigges CR, Miller SL, Herwaldt LA, Singh H. Updates to referring clinicians regarding critically ill children admitted to the pediatric intensive care unit: a state-wide survey. ACTA ACUST UNITED AC 2019; 7:123-128. [PMID: 31652117 DOI: 10.1515/dx-2019-0048] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/24/2019] [Accepted: 10/06/2019] [Indexed: 11/15/2022]
Abstract
Background Front-line clinicians are expected to make accurate and timely diagnostic decisions before transferring patients to the pediatric intensive care unit (PICU) but may not always learn their patients' outcomes. We evaluated the characteristics of post-transfer updates received by referring clinicians regarding PICU patients and determined preferences regarding content, delivery, and timing of such updates. Methods We administered an electronic cross-sectional survey to Iowa clinicians who billed for ≥5 pediatric patients or referred ≥1 patient to the University of Iowa (UI) PICU in the year before survey administration. Results One hundred and one clinicians (51 non-UI, 50 UI-affiliated) responded. Clinicians estimated that, on average, 8% of pediatric patients they saw over 1 year required PICU admission; clinicians received updates on 40% of patients. Seventy percent of UI clinicians obtained updates via self-initiated electronic record review, while 37% of non-UI clinicians relied on PICU communication (p = 0.013). Clinicians indicated that updates regarding diagnoses/outcomes will be most relevant to their practice. Among clinicians who received updates, 13% received unexpected information; 40% changed their practice as a result. Conclusions Clinicians received updates on less than half of the patients they referred to a PICU, although such updates could potentially influence clinical practice. Study findings will inform the development of a formal feedback system from the PICU to referring clinicians.
Collapse
Affiliation(s)
- Christina L Cifra
- Division of Pediatric Critical Care, Department of Pediatrics, University of Iowa Carver College of Medicine, Iowa City, IA, USA
| | - Cody R Tigges
- Division of Pediatric Critical Care, Department of Pediatrics, University of Iowa Carver College of Medicine, Iowa City, IA, USA
| | - Sarah L Miller
- Department of Emergency Medicine, University of Iowa Carver College of Medicine, Iowa City, IA, USA
| | - Loreen A Herwaldt
- Department of Internal Medicine, University of Iowa Carver College of Medicine, Iowa City, IA, USA.,Department of Epidemiology, University of Iowa College of Public Health, Iowa City, IA, USA
| | - Hardeep Singh
- Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey Veterans Affairs Medical Center and Baylor College of Medicine, Houston, TX, USA
| |
Collapse
|
22
|
Newman-Toker DE, Schaffer AC, Yu-Moe CW, Nassery N, Saber Tehrani AS, Clemens GD, Wang Z, Zhu Y, Fanai M, Siegal D. Serious misdiagnosis-related harms in malpractice claims: The “Big Three” – vascular events, infections, and cancers. Diagnosis (Berl) 2019; 6:227-240. [DOI: 10.1515/dx-2019-0019] [Citation(s) in RCA: 64] [Impact Index Per Article: 12.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/06/2019] [Accepted: 04/28/2019] [Indexed: 12/30/2022]
Abstract
Abstract
Background
Diagnostic errors cause substantial preventable harm, but national estimates vary widely from 40,000 to 4 million annually. This cross-sectional analysis of a large medical malpractice claims database was the first phase of a three-phase project to estimate the US burden of serious misdiagnosis-related harms.
Methods
We sought to identify diseases accounting for the majority of serious misdiagnosis-related harms (morbidity/mortality). Diagnostic error cases were identified from Controlled Risk Insurance Company (CRICO)’s Comparative Benchmarking System (CBS) database (2006–2015), representing 28.7% of all US malpractice claims. Diseases were grouped according to the Agency for Healthcare Research and Quality (AHRQ) Clinical Classifications Software (CCS) that aggregates the International Classification of Diseases diagnostic codes into clinically sensible groupings. We analyzed vascular events, infections, and cancers (the “Big Three”), including frequency, severity, and settings. High-severity (serious) harms were defined by scores of 6–9 (serious, permanent disability, or death) on the National Association of Insurance Commissioners (NAIC) Severity of Injury Scale.
Results
From 55,377 closed claims, we analyzed 11,592 diagnostic error cases [median age 49, interquartile range (IQR) 36–60; 51.7% female]. These included 7379 with high-severity harms (53.0% death). The Big Three diseases accounted for 74.1% of high-severity cases (vascular events 22.8%, infections 13.5%, and cancers 37.8%). In aggregate, the top five from each category (n = 15 diseases) accounted for 47.1% of high-severity cases. The most frequent disease in each category, respectively, was stroke, sepsis, and lung cancer. Causes were disproportionately clinical judgment factors (85.7%) across categories (range 82.0–88.8%).
Conclusions
The Big Three diseases account for about three-fourths of serious misdiagnosis-related harms. Initial efforts to improve diagnosis should focus on vascular events, infections, and cancers.
Collapse
|
23
|
Zwaan L, Hautz WE. Bridging the gap between uncertainty, confidence and diagnostic accuracy: calibration is key. BMJ Qual Saf 2019; 28:352-355. [DOI: 10.1136/bmjqs-2018-009078] [Citation(s) in RCA: 20] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 01/07/2019] [Indexed: 11/03/2022]
|