The Diagnostic Performance Feedback "Calibration Gap": Why Clinical Experience Alone Is Not Enough to Prevent Serious Diagnostic Errors

Development and Psychometric Analysis of a Patient-Reported Measure of Diagnostic Excellence for Emergency and Urgent Care Settings

BACKGROUND

Emergency and urgent care settings face challenges with routinely obtaining performance feedback related to diagnostic care. Patients and their care partners provide an important perspective on the diagnostic process and outcome of care in these settings. We sought to develop and test psychometric properties of Patient-Report to IMprove Diagnostic Excellence in Emergency Department settings (PRIME-ED), a measure of patient-reported diagnostic excellence in these care settings.

METHODS

We developed PRIME-ED based on literature review, expert feedback, and cognitive testing. To assess psychometric properties, we surveyed AmeriSpeak, a probability-based panel that provides sample coverage of approximately 97% of the U.S. household population, in February 2022 to adult patients, or their care partners, who had presented to an emergency department or urgent care facility within the last 30 days. Respondents rated their agreement on a 5-point Likert scale with each of 17 statements across multiple domains of patient-reported diagnostic excellence. Demographics, visit characteristics, and a subset of the Emergency Department Consumer Assessment of Healthcare Providers & Systems were also collected. We conducted psychometric testing for reliability and validity.

RESULTS

Over a thousand (n = 1116) national panelists completed the PRIME-ED survey, of which 58.7% were patients and 40.9% were care partners; 49.6% received care at an emergency department and 49.9% at an urgent care facility. Responses had high internal consistency within 3 patient-reported diagnostic excellence domain groupings: diagnostic process (Cronbach's alpha 0.94), accuracy of diagnosis (0.93), and communication of diagnosis (0.94). Domain groupings were significantly correlated with concurrent Emergency Department Consumer Assessment of Healthcare Providers & Systems items. Factor analyses substantiated 3 domain groupings.

CONCLUSIONS

PRIME-ED has potential as a tool for capturing patient-reported diagnostic excellence in emergency and urgent care.

Evaluation of ChatGPT-Generated Differential Diagnosis for Common Diseases With Atypical Presentation: Descriptive Research

Background

The persistence of diagnostic errors, despite advances in medical knowledge and diagnostics, highlights the importance of understanding atypical disease presentations and their contribution to mortality and morbidity. Artificial intelligence (AI), particularly generative pre-trained transformers like GPT-4, holds promise for improving diagnostic accuracy, but requires further exploration in handling atypical presentations.

Objective

This study aimed to assess the diagnostic accuracy of ChatGPT in generating differential diagnoses for atypical presentations of common diseases, with a focus on the model's reliance on patient history during the diagnostic process.

Methods

We used 25 clinical vignettes from the Journal of Generalist Medicine characterizing atypical manifestations of common diseases. Two general medicine physicians categorized the cases based on atypicality. ChatGPT was then used to generate differential diagnoses based on the clinical information provided. The concordance between AI-generated and final diagnoses was measured, with a focus on the top-ranked disease (top 1) and the top 5 differential diagnoses (top 5).

Results

ChatGPT's diagnostic accuracy decreased with an increase in atypical presentation. For category 1 (C1) cases, the concordance rates were 17% (n=1) for the top 1 and 67% (n=4) for the top 5. Categories 3 (C3) and 4 (C4) showed a 0% concordance for top 1 and markedly lower rates for the top 5, indicating difficulties in handling highly atypical cases. The χ2 test revealed no significant difference in the top 1 differential diagnosis accuracy between less atypical (C1+C2) and more atypical (C3+C4) groups (χ²1=2.07; n=25; P=.13). However, a significant difference was found in the top 5 analyses, with less atypical cases showing higher accuracy (χ²1=4.01; n=25; P=.048).

Conclusions

ChatGPT-4 demonstrates potential as an auxiliary tool for diagnosing typical and mildly atypical presentations of common diseases. However, its performance declines with greater atypicality. The study findings underscore the need for AI systems to encompass a broader range of linguistic capabilities, cultural understanding, and diverse clinical scenarios to improve diagnostic utility in real-world settings.

Exploring Clinical Lessons Learned by Experienced Hospitalists from Diagnostic Errors and Successes

BACKGROUND

Diagnostic errors cause significant patient harm. The clinician's ultimate goal is to achieve diagnostic excellence in order to serve patients safely. This can be accomplished by learning from both errors and successes in patient care. However, the extent to which clinicians grow and navigate diagnostic errors and successes in patient care is poorly understood. Clinically experienced hospitalists, who have cared for numerous acutely ill patients, should have great insights from their successes and mistakes to inform others striving for excellence in patient care.

OBJECTIVE

To identify and characterize clinical lessons learned by experienced hospitalists from diagnostic errors and successes.

DESIGN

A semi-structured interview guide was used to collect qualitative data from hospitalists at five independently administered hospitals in the Mid-Atlantic area from February to June 2022.

PARTICIPANTS

12 academic and 12 community-based hospitalists with ≥ 5 years of clinical experience.

APPROACH

A constructivist qualitative approach was used and "reflexive thematic analysis" of interview transcripts was conducted to identify themes and patterns of meaning across the dataset.

RESULTS

Five themes were generated from the data based on clinical lessons learned by hospitalists from diagnostic errors and successes. The ideas included appreciating excellence in clinical reasoning as a core skill, connecting with patients and other members of the health care team to be able to tap into their insights, reflecting on the diagnostic process, committing to growth, and prioritizing self-care.

CONCLUSIONS

The study identifies key lessons learned from the errors and successes encountered in patient care by clinically experienced hospitalists. These findings may prove helpful for individuals and groups that are authentically committed to moving along the continuum from diagnostic competence towards excellence.

Importance of Patient History in Artificial Intelligence-Assisted Medical Diagnosis: Comparison Study

Background

Medical history contributes approximately 80% to a diagnosis, although physical examinations and laboratory investigations increase a physician's confidence in the medical diagnosis. The concept of artificial intelligence (AI) was first proposed more than 70 years ago. Recently, its role in various fields of medicine has grown remarkably. However, no studies have evaluated the importance of patient history in AI-assisted medical diagnosis.

Objective

This study explored the contribution of patient history to AI-assisted medical diagnoses and assessed the accuracy of ChatGPT in reaching a clinical diagnosis based on the medical history provided.

Methods

Using clinical vignettes of 30 cases identified in The BMJ, we evaluated the accuracy of diagnoses generated by ChatGPT. We compared the diagnoses made by ChatGPT based solely on medical history with the correct diagnoses. We also compared the diagnoses made by ChatGPT after incorporating additional physical examination findings and laboratory data alongside history with the correct diagnoses.

Results

ChatGPT accurately diagnosed 76.6% (23/30) of the cases with only the medical history, consistent with previous research targeting physicians. We also found that this rate was 93.3% (28/30) when additional information was included.

Conclusions

Although adding additional information improves diagnostic accuracy, patient history remains a significant factor in AI-assisted medical diagnosis. Thus, when using AI in medical diagnosis, it is crucial to include pertinent and correct patient histories for an accurate diagnosis. Our findings emphasize the continued significance of patient history in clinical diagnoses in this age and highlight the need for its integration into AI-assisted medical diagnosis systems.

How do patients and care partners describe diagnostic uncertainty in an emergency department or urgent care setting?

OBJECTIVES

Little is known about how patients perceive diagnostic uncertainty. We sought to understand how patients and care partners perceive uncertainty in an emergency or urgent care setting, where making a final diagnosis is often not possible.

METHODS

We administered a survey to a nationally representative panel on patient-reported diagnostic excellence in an emergency department or urgent care setting. The survey included items specific to perceived diagnostic excellence, visit characteristics, and demographics. We analyzed responses to two open-ended questions among those who reported uncertainty in the explanation they were given. Themes were identified using an inductive approach, and compared by whether respondents agreed or disagreed the explanation they were given was true.

RESULTS

Of the 1,116 respondents, 106 (10 %) reported that the care team was not certain in the explanation of their health problem. Five themes were identified in the open-ended responses: poor communication (73 %), uncertainty made transparent (10 %), incorrect information provided (9 %), inadequate testing equipment (4 %), and unable to determine (4 %). Of the respondents who reported uncertainty, 21 % (n=22/106) reported the explanation of their problem given was not true.

CONCLUSIONS

The findings of this analysis suggest that the majority of patients and their care partners do not equate uncertainty with a wrong explanation of their health problem, and that poor communication was the most commonly cited reason for perceived uncertainty.

Patient Reasoning: Patients' and Care Partners' Perceptions of Diagnostic Accuracy in Emergency Care

OBJECTIVES

In the context of validating a measure of patient report specific to diagnostic accuracy in emergency department or urgent care, this study investigates patients' and care partners' perceptions of diagnoses as accurate and explores variations in how they reason while they assess accuracy.

METHODS

In February 2022, we surveyed a national panel of adults who had an emergency department or urgent care visit in the past month to test a patient-reported measure. As part of the survey validation, we asked for free-text responses about why the respondents indicated their (dis)agreement with 2 statements comprising patient-reported diagnostic accuracy: 1) the explanation they received of the health problem was true and 2) the explanation described what to expect of the health problem. Those paired free-text responses were qualitatively analyzed according to themes created inductively.

RESULTS

A total of 1,116 patients and care partners provided 982 responses coded into 10 themes, which were further grouped into 3 reasoning types. Almost one-third (32%) of respondents used only corroborative reasoning in assessing the accuracy of the health problem explanation (alignment of the explanation with either test results, patients' subsequent health trajectory, their medical knowledge, symptoms, or another doctor's opinion), 26% used only perception-based reasoning (perceptions of diagnostic process, uncertainty around the explanation received, or clinical team's attitudes), and 27% used both types of reasoning. The remaining 15% used general beliefs or nonexplicated logic (used only about accurate diagnoses) and combinations of general reasoning with perception-based and corroborative.

CONCLUSIONS

Patients and care partners used multifaceted reasoning in their assessment of diagnostic accuracy.

IMPLICATIONS

As health care shifts toward meaningful diagnostic co-production and shared decision making, in-depth understanding of variations in patient reasoning and mental models informs use in clinical practice.

HIGHLIGHTS

An analysis of 982 responses examined how patients and care partners reason about the accuracy of diagnoses they received in emergency or urgent care.In reasoning, people used their perception of the process and whether the diagnosis matched other factual information they have.We introduce "patient reasoning" in the diagnostic measurement context as an area of further research to inform diagnostic shared decision making and co-production of health.

Prevalence of Horizontal Canal Variant in 3,975 Patients With Benign Paroxysmal Positional Vertigo: A Cross-sectional Study

Background and Objectives

Current understanding based on older studies is that pc-BPPV is far more common than hc-BPPV. Such studies are limited by small sample sizes, and often the supine roll test for hc-BPPV is not performed. To better estimate the prevalence of hc-BPPV, we studied a large cross-section of patients with VOG-diagnosed BPPV.

Methods

Using a cross-sectional study of patients with BPPV, we investigated patients referred to NeuroEquilibrium specialty clinics throughout India between January 1, 2021, and December 31, 2021. All patients were evaluated with video oculography (VOG) during positional tests, and all diagnoses were confirmed by a neurotologist and neurologist.

Results

Of 3,975 patients with VOG-confirmed and specialist-diagnosed BPPV (median age, 51 years; 56.6% women), pc-BPPV accounted for 1,901 (47.8%), hc-BPPV was seen in 1,842 (46.3%), and anterior canal BPPV was found in 28 (0.7%) patients.

Discussion

This study found that hc-BPPV is far more common than previously reported. It is important to perform the supine roll test in addition to the Dix-Hallpike in all patients suspected with BPPV. Better training to diagnose patients with BPPV and to accurately recognize the nystagmus pattern of hc-BPPV should be a priority.

Cognitive perspectives on maintaining physicians' medical expertise: III. Strengths and weaknesses of self-assessment

Is self-assessment enough to keep physicians' cognitive skills-such as diagnosis, treatment, basic biological knowledge, and communicative skills-current? We review the cognitive strengths and weaknesses of self-assessment in the context of maintaining medical expertise. Cognitive science supports the importance of accurately self-assessing one's own skills and abilities, and we review several ways such accuracy can be quantified. However, our review also indicates a broad challenge in self-assessment is that individuals do not have direct access to the strength or quality of their knowledge and instead must infer this from heuristic strategies. These heuristics are reasonably accurate in many circumstances, but they also suffer from systematic biases. For example, information that feels easy to process in the moment can lead individuals to overconfidence in their ability to remember it in the future. Another notable phenomenon is the Dunning-Kruger effect: the poorest performers in a domain are also the least accurate in self-assessment. Further, explicit instruction is not always sufficient to remove these biases. We discuss what these findings imply about when physicians' self-assessment can be useful and when it may be valuable to supplement with outside sources.

Do malpractice claim clinical case vignettes enhance diagnostic accuracy and acceptance in clinical reasoning education during GP training?

BACKGROUND

Using malpractice claims cases as vignettes is a promising approach for improving clinical reasoning education (CRE), as malpractice claims can provide a variety of content- and context-rich examples. However, the effect on learning of adding information about a malpractice claim, which may evoke a deeper emotional response, is not yet clear. This study examined whether knowing that a diagnostic error resulted in a malpractice claim affects diagnostic accuracy and self-reported confidence in the diagnosis of future cases. Moreover, suitability of using erroneous cases with and without a malpractice claim for CRE, as judged by participants, was evaluated.

METHODS

In the first session of this two-phased, within-subjects experiment, 81 first-year residents of general practice (GP) were exposed to both erroneous cases with (M) and erroneous cases without (NM) malpractice claim information, derived from a malpractice claims database. Participants rated suitability of the cases for CRE on a five-point Likert scale. In the second session, one week later, participants solved four different cases with the same diagnoses. Diagnostic accuracy was measured with three questions, scored on a 0-1 scale: (1) What is your next step? (2) What is your differential diagnosis? (3) What is your most probable diagnosis and what is your level of certainty on this? Both subjective suitability and diagnostic accuracy scores were compared between the versions (M and NM) using repeated measures ANOVA.

RESULTS

There were no differences in diagnostic accuracy parameters (M vs. NM next step: 0.79 vs. 0.77, p = 0.505; differential diagnosis 0.68 vs. 0.75, p = 0.072; most probable diagnosis 0.52 vs. 0.57, p = 0.216) and self-reported confidence (53.7% vs. 55.8% p = 0.390) of diagnoses previously seen with or without malpractice claim information. Subjective suitability- and complexity scores for the two versions were similar (suitability: 3.68 vs. 3.84, p = 0.568; complexity 3.71 vs. 3.88, p = 0.218) and significantly increased for higher education levels for both versions.

CONCLUSION

The similar diagnostic accuracy rates between cases studied with or without malpractice claim information suggests both versions are equally effective for CRE in GP training. Residents judged both case versions to be similarly suitable for CRE; both were considered more suitable for advanced than for novice learners.

Contributors to Diagnostic Error or Delay in the Acute Care Setting: A Survey of Clinical Stakeholders

Diagnostic error or delay (DEOD) is common in the acute care setting and results in poor patient outcomes. Many factors contribute to DEOD, but little is known about how contributors may differ across acute care areas and professional roles. As part of a sequential exploratory mixed methods research study, we surveyed acute care clinical stakeholders about the frequency with which different factors contribute to DEOD. Survey respondents could also propose solutions in open text fields. N = 220 clinical stakeholders completed the survey. Care Team Interactions, Systems and Process, Patient, Provider, and Cognitive factors were perceived to contribute to DEOD with similar frequency. Organization and Infrastructure factors were perceived to contribute to DEOD significantly less often. Responses did not vary across acute care setting. Physicians perceived Cognitive factors to contribute to DEOD more frequently compared to those in other roles. Commonly proposed solutions included: technological solutions, organization level fixes, ensuring staff know and are encouraged to work to the full scope of their role, and cultivating a culture of collaboration and respect. Multiple factors contribute to DEOD with similar frequency across acute care areas, suggesting the need for a multi-pronged approach that can be applied across acute care areas.

How does safety netting for lung cancer symptoms help patients to reconsult appropriately? A qualitative study

Background

Safety netting in primary care is considered an important intervention for managing diagnostic uncertainty. This is the first study to examine how patients understand and interpret safety netting advice around low-risk potential lung cancer symptoms, and how this affects reconsultation behaviours.

Methods

Qualitative interview study in UK primary care. Pre-covid-19, five patients were interviewed in person within 2–3 weeks of a primary care consultation for potential lung cancer symptom(s), and again 2–5 months later. The general practitioner (GP) they last saw was interviewed face-to-face once. During the covid-19 pandemic, an additional 15 patients were interviewed only once via telephone, and their GPs were not interviewed or contacted in any way. Audio-recorded interviews were transcribed verbatim and analysed using inductive thematic analysis.

Results

The findings from our thematic analysis suggest that patients prefer active safety netting, as part of thorough and logical diagnostic uncertainty management. Passive or ambiguous safety netting may be perceived as dismissive and cause delayed reconsultation. GP safety netting strategies are not always understood, potentially causing patient worry and dissatisfaction. Telephone consultations and the diagnostic overshadowing of COVID-19 on respiratory symptoms impacted GPs’ safety netting strategies and patients’ appetite for active follow up measures.

Conclusions

Safety netting guidelines do not yet offer solutions that have been proven to promote symptom vigilance and timely reconsultation for low-risk lung cancer symptoms. This may have been affected by primary care practices during the COVID-19 pandemic. Patients prefer active or pre-planned safety netting coupled with thorough consultation techniques and a comprehensible diagnostic strategy, and may respond adversely to passive safety netting advice.

Electronic health record-based patient tracking by emergency medicine physicians

Objectives

Emergency medicine (EM) physicians commonly track the progress of former patients to learn about their clinical outcome. While some studies have described the behavior, little is known about the specific information sought during tracking. The objective of this study was to determine how often EM physicians track patients and the motivations, strategies, and barriers to tracking.

Methods

In June 2019 we surveyed EM physicians practicing at six hospitals. We defined patient tracking as viewing the chart of a patient who was no longer under the physician's care or contacting the patient or the patient's subsequent providers to learn about the patient's progress. The survey asked respondents how often they track patients, by what mechanisms, and for what reasons. The survey also asked what information physicians sought when tracking and what barriers to tracking exist.

Results

Of the 156 EM physicians invited to respond, 111 completed the survey (72% response rate). Of those, 109 (98%) reported tracking their patients, and residents reported tracking a higher percentage of patients than attendings. Reasons for tracking included an unusual or complex case (98%), uncertain diagnosis (89%), and concern about a potential error (48%). Most respondents (86%) said that knowledge gained from patient tracking changed their subsequent practice patterns, and almost all respondents (98%) strongly agreed or agreed that tracking helps physicians avoid future mistakes. The most commonly sought information types during tracking were the hospital discharge summary or emergency department note from the index visit, test results since the index visit, and new diagnoses added since the index visit. Physicians cited time limitations and difficulty accessing information as the leading barriers to tracking.

Conclusions

Patient tracking is nearly ubiquitous among surveyed EM physicians, who find it valuable for learning and patient safety.

Diagnostic Error in Neuro-ophthalmology: Avenues to Improve

Purpose of Review

To highlight potential avenues to reduce preventable diagnostic error of neuro-ophthalmic conditions and avoid patient harm.

Recent Findings

Recent prospective studies and studies of patient harm have advanced our understanding. Additionally, recent studies of fundus photography, telemedicine, and artificial intelligence highlight potential avenues for diagnostic improvement.

Summary

Diagnostic error of neuro-ophthalmic conditions can often be traced to failure to gather an adequate history, perform a complete physical exam, obtain adequate/appropriate neuroimaging, and generate a complete, appropriate differential diagnosis. Improving triage and identification of neuro-ophthalmic conditions by other providers and increasing access to subspecialty neuro-ophthalmology evaluation are essential avenues to reduce diagnostic error. Further research should evaluate the relationship between misdiagnosis and patient harm, and help identify the most impactful potential targets for improvement.

Improving medical residents' self-assessment of their diagnostic accuracy: does feedback help?

When physicians do not estimate their diagnostic accuracy correctly, i.e. show inaccurate diagnostic calibration, diagnostic errors or overtesting can occur. A previous study showed that physicians' diagnostic calibration for easy cases improved, after they received feedback on their previous diagnoses. We investigated whether diagnostic calibration would also improve from this feedback when cases were more difficult. Sixty-nine general-practice residents were randomly assigned to one of two conditions. In the feedback condition, they diagnosed a case, rated their confidence in their diagnosis, their invested mental effort, and case complexity, and then were shown the correct diagnosis (feedback). This was repeated for 12 cases. Participants in the control condition did the same without receiving feedback. We analysed calibration in terms of (1) absolute accuracy (absolute difference between diagnostic accuracy and confidence), and (2) bias (confidence minus diagnostic calibration). There was no difference between the conditions in the measurements of calibration (absolute accuracy, p = .204; bias, p = .176). Post-hoc analyses showed that on correctly diagnosed cases (on which participants are either accurate or underconfident), calibration in the feedback condition was less accurate than in the control condition, p = .013. This study shows that feedback on diagnostic performance did not improve physicians' calibration for more difficult cases. One explanation could be that participants were confronted with their mistakes and thereafter lowered their confidence ratings even if cases were diagnosed correctly. This shows how difficult it is to improve diagnostic calibration, which is important to prevent diagnostic errors or maltreatment.

Emergency medicine physicians' perspectives on diagnostic accuracy in neurology: a qualitative study

OBJECTIVES

We sought to understand the knowledge, attitudes, and beliefs of emergency medicine (EM) physicians towards non-specific neurological conditions and the use of clinical decision support (CDS) to improve diagnostic accuracy.

METHODS

We conducted semi-structured interviews of EM physicians at four emergency departments (EDs) affiliated with a single US healthcare system. Interviews were conducted until thematic saturation was achieved. Conventional content analysis was used to identify themes related to EM physicians' perspectives on acute diagnostic neurology; directed content analysis was used to explore views regarding CDS. Each interview transcript was independently coded by two researchers using an iteratively refined codebook with consensus-based resolution of coding differences.

RESULTS

We identified two domains regarding diagnostic safety: (1) challenges unique to neurological complaints and (2) challenges in EM more broadly. Themes relevant to neurology included: (1) knowledge gaps and uncertainty, (2) skepticism about neurology, (3) comfort with basic as opposed to detailed neurological examination, and (4) comfort with non-neurological diseases. Themes relevant to diagnostic decision making in the ED included: (1) cognitive biases, (2) ED system/environmental issues, (3) patient barriers, (4) comfort with diagnostic uncertainty, and (5) concerns regarding diagnostic error identification and measurement. Most participating EM physicians were enthusiastic about the potential for well-designed CDS to improve diagnostic accuracy for non-specific neurological complaints.

CONCLUSIONS

Physicians identified diagnostic challenges unique to neurological diseases as well as issues related more generally to diagnostic accuracy in EM. These physician-reported issues should be accounted for when designing interventions to improve ED diagnostic accuracy.

Comparison of Diagnostic Recommendations from Individual Physicians versus the Collective Intelligence of Multiple Physicians in Ambulatory Cases Referred for Specialist Consultation

BACKGROUND

Studies report higher diagnostic accuracy using the collective intelligence (CI) of multiple clinicians compared with individual clinicians. However, the diagnostic process is iterative, and unexplored is the value of CI in improving clinical recommendations leading to a final diagnosis.

METHODS

To compare the appropriateness of diagnostic recommendations advised by individual physicians versus the CI of physicians, we entered actual consultation requests sent by primary care physicians to specialists onto a web-based CI platform capable of collecting diagnostic recommendations (next steps for care) from multiple physicians. We solicited responses to 35 cases (12 endocrinology, 13 gynecology, 10 neurology) from ≥3 physicians of any specialty through the CI platform, which aggregated responses into a CI output. The primary outcome was the appropriateness of individual physician recommendations versus the CI output recommendations, using recommendations agreed upon by 2 specialists in the same specialty as a gold standard. The secondary outcome was the recommendations' potential for harm.

RESULTS

A total of 177 physicians responded. Cases had a median of 7 respondents (interquartile range: 5-10). Diagnostic recommendations in the CI output achieved higher levels of appropriateness (69%) than recommendations from individual physicians (45%; χ² = 5.95, P = 0.015). Of the CI recommendations, 54% were potentially harmful, as compared with 41% of individuals' recommendations (χ² = 2.49, P = 0.11).

LIMITATIONS

Cases were from a single institution. CI was solicited using a single algorithm/platform.

CONCLUSIONS

When seeking specialist guidance, diagnostic recommendations from the CI of multiple physicians are more appropriate than recommendations from most individual physicians, measured against specialist recommendations. Although CI provides useful recommendations, some have potential for harm. Future research should explore how to use CI to improve diagnosis while limiting harm from inappropriate tests/therapies.

Stroke hospitalization after misdiagnosis of "benign dizziness" is lower in specialty care than general practice: a population-based cohort analysis of missed stroke using SPADE methods

OBJECTIVES

Isolated dizziness is a challenging stroke presentation in the emergency department, but little is known about this problem in other clinical settings. We sought to compare stroke hospitalizations after treat-and-release clinic visits for purportedly "benign dizziness" between general and specialty care settings.

METHODS

This was a population-based retrospective cohort study from a national database. We included clinic patients with a first incident treat-and-release visit diagnosis of non-specific dizziness/vertigo or a peripheral vestibular disorder (ICD-9-CM 780.4 or 386.x [not 386.2]). We compared general care (internal medicine, family medicine) vs. specialty care (neurology, otolaryngology) providers. We used propensity scores to control for baseline stroke risk differences unrelated to dizziness diagnosis. We measured excess (observed>expected) stroke hospitalizations in the first 30 d (i.e., missed strokes associated with an adverse event).

RESULTS

We analyzed 144,355 patients discharged with "benign dizziness" (n=117,117 diagnosed in general care; n=27,238 in specialty care). After propensity score matching, patients in both groups were at higher risk of stroke in the first 30 d (rate difference per 10,000 treat-and-release visits for "benign dizziness" 24.9 [95% CI 18.6-31.2] in general care and 10.6 [95% CI 6.3-14.9] in specialty care). Short-term stroke risk was higher in general care than specialty care (relative risk, RR 2.2, 95% CI 1.5-3.2) while the long-term risk was not significantly different (RR 1.3, 95% CI 0.9-1.9), indicating higher misdiagnosis-related harms among dizzy patients who initially presented to generalists after adequate propensity matching.

CONCLUSIONS

Missed stroke-related harms in general care were roughly twice that in specialty care. Solutions are needed to address this care gap.

Real-world virtual patient simulation to improve diagnostic performance through deliberate practice: a prospective quasi-experimental study

OBJECTIVES

Diagnostic errors are pervasive in medicine and most often caused by clinical reasoning failures. Clinical presentations characterized by nonspecific symptoms with broad differential diagnoses (e.g., dizziness) are especially prone to such errors.

METHODS

We hypothesized that novice clinicians could achieve proficiency diagnosing dizziness by training with virtual patients (VPs). This was a prospective, quasi-experimental, pretest-posttest study (2019) at a single academic medical center. Internal medicine interns (intervention group) were compared to second/third year residents (control group). A case library of VPs with dizziness was developed from a clinical trial (AVERT-NCT02483429). The approach (VIPER - Virtual Interactive Practice to build Expertise using Real cases) consisted of brief lectures combined with 9 h of supervised deliberate practice. Residents were provided dizziness-related reading and teaching modules. Both groups completed pretests and posttests.

RESULTS

For interns (n=22) vs. residents (n=18), pretest median diagnostic accuracy did not differ (33% [IQR 18-46] vs. 31% [IQR 13-50], p=0.61) between groups, while posttest accuracy did (50% [IQR 42-67] vs. 20% [IQR 17-33], p=0.001). Pretest median appropriate imaging did not differ (33% [IQR 17-38] vs. 31% [IQR 13-38], p=0.89) between groups, while posttest appropriateness did (65% [IQR 52-74] vs. 25% [IQR 17-36], p<0.001).

CONCLUSIONS

Just 9 h of deliberate practice increased diagnostic skills (both accuracy and testing appropriateness) of medicine interns evaluating real-world dizziness 'in silico' more than ∼1.7 years of residency training. Applying condensed educational experiences such as VIPER across a broad range of common presentations could significantly enhance diagnostic education and translate to improved patient care.

A Program to Provide Clinicians with Feedback on Their Diagnostic Performance in a Learning Health System

PROBLEM

Reducing diagnostic errors requires improving both systems and individual clinical reasoning. One strategy to achieve diagnostic excellence is learning from feedback. However, clinicians remain uncomfortable receiving feedback on their diagnostic performance. Thus, a team of researchers and clinical leaders aimed to develop and implement a diagnostic performance feedback program for learning that mitigates potential clinician discomfort.

APPROACH

The program was developed as part of a larger project to create a learning health system around diagnostic safety at Geisinger, a large, integrated health care system in rural Pennsylvania. Steps included identifying potential missed opportunities in diagnosis (MODs) from various sources (for example, risk management, clinician reports, patient complaints); confirming MODs through chart review; and having trained facilitators provide feedback to clinicians about MODs as learning opportunities. The team developed a guide for facilitators to conduct effective diagnostic feedback sessions and surveyed facilitators and recipients about their experiences and perceptions of the feedback sessions.

OUTCOMES

28 feedback sessions occurred from January 2019 to June 2020, involving MODs from emergency medicine, primary care, and hospital medicine. Most facilitators (90.6% [29/32]) reported that recipients were receptive to learning and discussing MODs. Most recipients reported that conversations were constructive and nonpunitive (83.3% [25/30]) and allowed them to take concrete steps toward improving diagnosis (76.7% [23/30]). Both groups believed discussions would improve future diagnostic safety (93.8% [30/32] and 70.0% [21/30], respectively).

KEY INSIGHTS AND NEXT STEPS

An institutional program was developed and implemented to deliver diagnostic performance feedback. Such a program may facilitate learning and improvement to reduce MODs. Future efforts should assess long-term effects on diagnostic performance and patient outcomes.

Rate of diagnostic errors and serious misdiagnosis-related harms for major vascular events, infections, and cancers: toward a national incidence estimate using the “Big Three”

Background

Missed vascular events, infections, and cancers account for ~75% of serious harms from diagnostic errors. Just 15 diseases from these “Big Three” categories account for nearly half of all serious misdiagnosis-related harms in malpractice claims. As part of a larger project estimating total US burden of serious misdiagnosis-related harms, we performed a focused literature review to measure diagnostic error and harm rates for these 15 conditions.

Methods

We searched PubMed, Google, and cited references. For errors, we selected high-quality, modern, US-based studies, if available, and best available evidence otherwise. For harms, we used literature-based estimates of the generic (disease-agnostic) rate of serious harms (morbidity/mortality) per diagnostic error and applied claims-based severity weights to construct disease-specific rates. Results were validated via expert review and comparison to prior literature that used different methods. We used Monte Carlo analysis to construct probabilistic plausible ranges (PPRs) around estimates.

Results

Rates for the 15 diseases were drawn from 28 published studies representing 91,755 patients. Diagnostic error (false negative) rates ranged from 2.2% (myocardial infarction) to 62.1% (spinal abscess), with a median of 13.6% [interquartile range (IQR) 9.2–24.7] and an aggregate mean of 9.7% (PPR 8.2–12.3). Serious misdiagnosis-related harm rates per incident disease case ranged from 1.2% (myocardial infarction) to 35.6% (spinal abscess), with a median of 5.5% (IQR 4.6–13.6) and an aggregate mean of 5.2% (PPR 4.5–6.7). Rates were considered face valid by domain experts and consistent with prior literature reports.

Conclusions

Diagnostic improvement initiatives should focus on dangerous conditions with higher diagnostic error and misdiagnosis-related harm rates.

Updates to referring clinicians regarding critically ill children admitted to the pediatric intensive care unit: a state-wide survey

Background Front-line clinicians are expected to make accurate and timely diagnostic decisions before transferring patients to the pediatric intensive care unit (PICU) but may not always learn their patients' outcomes. We evaluated the characteristics of post-transfer updates received by referring clinicians regarding PICU patients and determined preferences regarding content, delivery, and timing of such updates. Methods We administered an electronic cross-sectional survey to Iowa clinicians who billed for ≥5 pediatric patients or referred ≥1 patient to the University of Iowa (UI) PICU in the year before survey administration. Results One hundred and one clinicians (51 non-UI, 50 UI-affiliated) responded. Clinicians estimated that, on average, 8% of pediatric patients they saw over 1 year required PICU admission; clinicians received updates on 40% of patients. Seventy percent of UI clinicians obtained updates via self-initiated electronic record review, while 37% of non-UI clinicians relied on PICU communication (p = 0.013). Clinicians indicated that updates regarding diagnoses/outcomes will be most relevant to their practice. Among clinicians who received updates, 13% received unexpected information; 40% changed their practice as a result. Conclusions Clinicians received updates on less than half of the patients they referred to a PICU, although such updates could potentially influence clinical practice. Study findings will inform the development of a formal feedback system from the PICU to referring clinicians.

Serious misdiagnosis-related harms in malpractice claims: The “Big Three” – vascular events, infections, and cancers

Background

Diagnostic errors cause substantial preventable harm, but national estimates vary widely from 40,000 to 4 million annually. This cross-sectional analysis of a large medical malpractice claims database was the first phase of a three-phase project to estimate the US burden of serious misdiagnosis-related harms.

Methods

We sought to identify diseases accounting for the majority of serious misdiagnosis-related harms (morbidity/mortality). Diagnostic error cases were identified from Controlled Risk Insurance Company (CRICO)’s Comparative Benchmarking System (CBS) database (2006–2015), representing 28.7% of all US malpractice claims. Diseases were grouped according to the Agency for Healthcare Research and Quality (AHRQ) Clinical Classifications Software (CCS) that aggregates the International Classification of Diseases diagnostic codes into clinically sensible groupings. We analyzed vascular events, infections, and cancers (the “Big Three”), including frequency, severity, and settings. High-severity (serious) harms were defined by scores of 6–9 (serious, permanent disability, or death) on the National Association of Insurance Commissioners (NAIC) Severity of Injury Scale.

Results

From 55,377 closed claims, we analyzed 11,592 diagnostic error cases [median age 49, interquartile range (IQR) 36–60; 51.7% female]. These included 7379 with high-severity harms (53.0% death). The Big Three diseases accounted for 74.1% of high-severity cases (vascular events 22.8%, infections 13.5%, and cancers 37.8%). In aggregate, the top five from each category (n = 15 diseases) accounted for 47.1% of high-severity cases. The most frequent disease in each category, respectively, was stroke, sepsis, and lung cancer. Causes were disproportionately clinical judgment factors (85.7%) across categories (range 82.0–88.8%).

Conclusions

The Big Three diseases account for about three-fourths of serious misdiagnosis-related harms. Initial efforts to improve diagnosis should focus on vascular events, infections, and cancers.