Diagnostic performance dashboards: tracking diagnostic errors using big data

Misdiagnosis-related harm quantification through mixture models and harm measures

Investigating and monitoring misdiagnosis-related harm is crucial for improving health care. However, this effort has traditionally focused on the chart review process, which is labor intensive, potentially unstable, and does not scale well. To monitor medical institutes' diagnostic performance and identify areas for improvement in a timely fashion, researchers proposed to leverage the relationship between symptoms and diseases based on electronic health records or claim data. Specifically, the elevated disease risk following a false-negative diagnosis can be used to signal potential harm. However, off-the-shelf statistical methods do not fully accommodate the data structure of a well-hypothesized risk pattern and thus fail to address the unique challenges adequately. To fill these gaps, we proposed a mixture regression model and its associated goodness-of-fit testing. We further proposed harm measures and profiling analysis procedures to quantify, evaluate, and compare misdiagnosis-related harm across institutes with potentially different patient population compositions. We studied the performance of the proposed methods through simulation studies. We then illustrated the methods through data analyses on stroke occurrence data from the Taiwan Longitudinal Health Insurance Database. From the analyses, we quantitatively evaluated risk factors for being harmed due to misdiagnosis, which unveiled some insights for health care quality research. We also compared general and special care hospitals in Taiwan and observed better diagnostic performance in special care hospitals using various new evaluation measures.

Contributors to Diagnostic Error or Delay in the Acute Care Setting: A Survey of Clinical Stakeholders

Diagnostic error or delay (DEOD) is common in the acute care setting and results in poor patient outcomes. Many factors contribute to DEOD, but little is known about how contributors may differ across acute care areas and professional roles. As part of a sequential exploratory mixed methods research study, we surveyed acute care clinical stakeholders about the frequency with which different factors contribute to DEOD. Survey respondents could also propose solutions in open text fields. N = 220 clinical stakeholders completed the survey. Care Team Interactions, Systems and Process, Patient, Provider, and Cognitive factors were perceived to contribute to DEOD with similar frequency. Organization and Infrastructure factors were perceived to contribute to DEOD significantly less often. Responses did not vary across acute care setting. Physicians perceived Cognitive factors to contribute to DEOD more frequently compared to those in other roles. Commonly proposed solutions included: technological solutions, organization level fixes, ensuring staff know and are encouraged to work to the full scope of their role, and cultivating a culture of collaboration and respect. Multiple factors contribute to DEOD with similar frequency across acute care areas, suggesting the need for a multi-pronged approach that can be applied across acute care areas.

Big Data Health Care Innovations: Performance Dashboarding as a Process of Collective Sensemaking

Big data is poised to revolutionize health care, and performance dashboards can be an important tool to manage big data innovations. Dashboards show the progress being made and provide critical management information about effectiveness and efficiency. However, performance dashboards are more than just a clear and straightforward representation of performance in the health care context. Instead, the development and maintenance of informative dashboards can be more productively viewed as an interactive and iterative process involving all stakeholders. We refer to this process as dashboarding and reflect on our learnings within a large European Union–funded project. Within this project, multiple big data applications in health care are being developed, piloted, and scaled up. In this paper, we discuss the ways in which we cope with the inherent sensitivities and tensions surrounding dashboarding in such a dynamic environment.

Dashboards for visual display of patient safety data: a systematic review

Background

Methods to visualise patient safety data can support effective monitoring of safety events and discovery of trends. While quality dashboards are common, use and impact of dashboards to visualise patient safety event data remains poorly understood.

Objectives

To understand development, use and direct or indirect impacts of patient safety dashboards.

Methods

We conducted a systematic review in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. We searched PubMed, EMBASE and CINAHL for publications between 1 January 1950 and 30 August 2018 involving use of dashboards to display data related to safety targets defined by the Agency for Healthcare Research and Quality’s Patient Safety Net. Two reviewers independently reviewed search results for inclusion in analysis and resolved disagreements by consensus. We collected data on development, use and impact via standardised data collection forms and analysed data using descriptive statistics.

Results

Literature search identified 4624 results which were narrowed to 33 publications after applying inclusion and exclusion criteria and consensus across reviewers. Publications included only time series and case study designs and were inpatient focused and emergency department focused. Information on direct impact of dashboards was limited, and only four studies included informatics or human factors principles in development or postimplementation evaluation.

Discussion

Use of patient-safety dashboards has grown over the past 15 years, but impact remains poorly understood. Dashboard design processes rarely use informatics or human factors principles to ensure that the available content and navigation assists task completion, communication or decision making.

Conclusion

Design and usability evaluation of patient safety dashboards should incorporate informatics and human factors principles. Future assessments should also rigorously explore their potential to support patient safety monitoring including direct or indirect impact on patient safety.

Statistical insights for crude-rate-based operational measures of misdiagnosis-related harms

In longitudinal event data, a crude rate is a simple quantification of the event rate, defined as the number of events during an evaluation window, divided by the at-risk population size at the beginning or mid-time point of that window. The crude rate recently received revitalizing interest from medical researchers who aimed to improve measurement of misdiagnosis-related harms using administrative or billing data by tracking unexpected adverse events following a "benign" diagnosis. The simplicity of these measures makes them attractive for implementation and routine operational monitoring at hospital or health system level. However, relevant statistical inference procedures have not been systematically summarized. Moreover, it is unclear to what extent the temporal changes of the at-risk population size would bias analyses and affect important conclusions concerning misdiagnosis-related harms. In this article, we present statistical inference tools for using crude-rate based harm measures, as well as formulas and simulation results that quantify the deviation of such measures from those based on the more sophisticated Nelson-Aalen estimator. Moreover, we present results for a generalized multibin version of the crude rate, for which the usual crude rate is a single-bin special case. The generalized multibin crude rate is more straightforward to compute than the Nelson-Aalen estimator and can reduce potential biases of the single-bin crude rate. For studies that seek to use multibin measures, we provide simulations to guide the choice regarding number of bins. We further bolster these results using a worked example of stroke after "benign" dizziness from a large data set.

Stroke hospitalization after misdiagnosis of "benign dizziness" is lower in specialty care than general practice: a population-based cohort analysis of missed stroke using SPADE methods

OBJECTIVES

Isolated dizziness is a challenging stroke presentation in the emergency department, but little is known about this problem in other clinical settings. We sought to compare stroke hospitalizations after treat-and-release clinic visits for purportedly "benign dizziness" between general and specialty care settings.

METHODS

This was a population-based retrospective cohort study from a national database. We included clinic patients with a first incident treat-and-release visit diagnosis of non-specific dizziness/vertigo or a peripheral vestibular disorder (ICD-9-CM 780.4 or 386.x [not 386.2]). We compared general care (internal medicine, family medicine) vs. specialty care (neurology, otolaryngology) providers. We used propensity scores to control for baseline stroke risk differences unrelated to dizziness diagnosis. We measured excess (observed>expected) stroke hospitalizations in the first 30 d (i.e., missed strokes associated with an adverse event).

RESULTS

We analyzed 144,355 patients discharged with "benign dizziness" (n=117,117 diagnosed in general care; n=27,238 in specialty care). After propensity score matching, patients in both groups were at higher risk of stroke in the first 30 d (rate difference per 10,000 treat-and-release visits for "benign dizziness" 24.9 [95% CI 18.6-31.2] in general care and 10.6 [95% CI 6.3-14.9] in specialty care). Short-term stroke risk was higher in general care than specialty care (relative risk, RR 2.2, 95% CI 1.5-3.2) while the long-term risk was not significantly different (RR 1.3, 95% CI 0.9-1.9), indicating higher misdiagnosis-related harms among dizzy patients who initially presented to generalists after adequate propensity matching.

CONCLUSIONS

Missed stroke-related harms in general care were roughly twice that in specialty care. Solutions are needed to address this care gap.

Missed acute myocardial infarction in the emergency department-standardizing measurement of misdiagnosis-related harms using the SPADE method

OBJECTIVES

Diagnostic error is a serious public health problem. Measuring diagnostic performance remains elusive. We sought to measure misdiagnosis-related harms following missed acute myocardial infarctions (AMI) in the emergency department (ED) using the symptom-disease pair analysis of diagnostic error (SPADE) method.

METHODS

Retrospective administrative data analysis (2009-2017) from a single, integrated health system using International Classification of Diseases (ICD) coded discharge diagnoses. We looked back 30 days from AMI hospitalizations for antecedent ED treat-and-release visits to identify symptoms linked to probable missed AMI (observed > expected). We then looked forward from these ED discharge diagnoses to identify symptom-disease pair misdiagnosis-related harms (AMI hospitalizations within 30-days, representing diagnostic adverse events).

RESULTS

A total of 44,473 AMI hospitalizations were associated with 2,874 treat-and-release ED visits in the prior 30 days. The top plausibly-related ED discharge diagnoses were "chest pain" and "dyspnea" with excess treat-and-release visit rates of 9.8% (95% CI 8.5-11.2%) and 3.4% (95% CI 2.7-4.2%), respectively. These represented 574 probable missed AMIs resulting in hospitalization (adverse event rate per AMI 1.3%, 95% CI 1.2-1.4%). Looking forward, 325,088 chest pain or dyspnea ED discharges were followed by 508 AMI hospitalizations (adverse event rate per symptom discharge 0.2%, 95% CI 0.1-0.2%).

CONCLUSIONS

The SPADE method precisely quantifies misdiagnosis-related harms from missed AMIs using administrative data. This approach could facilitate future assessment of diagnostic performance across health systems. These results correspond to ∼10,000 potentially-preventable harms annually in the US. However, relatively low error and adverse event rates may pose challenges to reducing harms for this ED symptom-disease pair.

Diagnostic Error of Neuro-ophthalmologic Conditions: State of the Science

BACKGROUND

Diagnostic error is prevalent and costly, occurring in up to 15% of US medical encounters and affecting up to 5% of the US population. One-third of malpractice payments are related to diagnostic error. A complex and specialized diagnostic process makes neuro-ophthalmologic conditions particularly vulnerable to diagnostic error.

EVIDENCE ACQUISITION

English-language literature on diagnostic errors in neuro-ophthalmology and neurology was identified through electronic search of PubMed and Google Scholar and hand search.

RESULTS

Studies investigating diagnostic error of neuro-ophthalmologic conditions have revealed misdiagnosis rates as high as 60%-70% before evaluation by a neuro-ophthalmology specialist, resulting in unnecessary tests and treatments. Correct performance and interpretation of the physical examination, appropriate ordering and interpretation of neuroimaging tests, and generation of a differential diagnosis were identified as pitfalls in the diagnostic process. Most studies did not directly assess patient harms or financial costs of diagnostic error.

CONCLUSIONS

As an emerging field, diagnostic error in neuro-ophthalmology offers rich opportunities for further research and improvement of quality of care.

Antecedent treat-and-release diagnoses prior to sepsis hospitalization among adult emergency department patients: a look-back analysis employing insurance claims data using Symptom-Disease Pair Analysis of Diagnostic Error (SPADE) methodology

OBJECTIVES

The aim of this study was to identify delays in early pre-sepsis diagnosis in emergency departments (ED) using the Symptom-Disease Pair Analysis of Diagnostic Error (SPADE) approach.

METHODS

SPADE methodology was employed using electronic health record and claims data from Kaiser Permanente Mid-Atlantic States (KPMAS). Study cohort included KPMAS members ≥18 years with ≥1 sepsis hospitalization 1/1/2013-12/31/2018. A look-back analysis identified treat-and-release ED visits in the month prior to sepsis hospitalizations. Top 20 diagnoses associated with these ED visits were identified; two diagnosis categories were distinguished as being linked to downstream sepsis hospitalizations. Observed-to-expected (O:E) and temporal analyses were performed to validate the symptom selection; results were contrasted to a comparison group. Demographics of patients that did and did not experience sepsis misdiagnosis were compared.

RESULTS

There were 3,468 sepsis hospitalizations during the study period and 766 treat-and-release ED visits in the month prior to hospitalization. Patients discharged from the ED with fluid and electrolyte disorders (FED) and altered mental status (AMS) were most likely to have downstream sepsis hospitalizations (O:E ratios of 2.66 and 2.82, respectively). Temporal analyses revealed that these symptoms were overrepresented and temporally clustered close to the hospitalization date. Approximately 2% of sepsis hospitalizations were associated with prior FED or AMS ED visits.

CONCLUSIONS

Treat-and-release ED encounters for FED and AMS may represent harbingers for downstream sepsis hospitalizations. The SPADE approach can be used to develop performance measures that identify pre-sepsis.

Time-ordered comorbidity correlations identify patients at risk of mis- and overdiagnosis

Diagnostic errors are common and can lead to harmful treatments. We present a data-driven, generic approach for identifying patients at risk of being mis- or overdiagnosed, here exemplified by chronic obstructive pulmonary disease (COPD). It has been estimated that 5-60% of all COPD cases are misdiagnosed. High-throughput methods are therefore needed in this domain. We have used a national patient registry, which contains hospital diagnoses for 6.9 million patients across the entire Danish population for 21 years and identified statistically significant disease trajectories for COPD patients. Using 284,154 patients diagnosed with COPD, we identified frequent disease trajectories comprising time-ordered comorbidities. Interestingly, as many as 42,459 patients did not present with these time-ordered, common comorbidities. Comparison of the individual disease history for each non-follower to the COPD trajectories, demonstrated that 9597 patients were unusual. Survival analysis showed that this group died significantly earlier than COPD patients following a trajectory. Out of the 9597 patients, we identified one subgroup comprising 2185 patients at risk of misdiagnosed COPD without the typical events of COPD patients. In all, 10% of these patients were diagnosed with lung cancer, and it seems likely that they are underdiagnosed for lung cancer as their laboratory test values and survival pattern are similar to such patients. Furthermore, only 4% had a lung function test to confirm the COPD diagnosis. Another subgroup with 2368 patients were found to be at risk of "classically" overdiagnosed COPD that survive >5.5 years after the COPD diagnosis, but without the typical complications of COPD.

Real-Time Monitoring System to Manage Mental Healthcare Emergency Unit

Objectives

Real-time relevant information helps guide the healthcare decision-making process in daily clinical practice as well as the management and optimization of healthcare processes. However, proprietary business intelligence suite solutions supporting the production of decision-making information requires investment that is out of reach of small and medium-sized healthcare facilities or those with limited resources, particularly in developing countries. This paper describes our experience in designing and implementing a real-time healthcare monitoring system solution to manage healthcare emergency units.

Methods

Through the use of free Business Intelligence tools and Python data science language we designed a real-time monitoring system, which was implemented to explore the Electronic Medical Records system of a university mental health emergency unit and render an electronic dashboard to support health professional daily practice.

Results

Three main dashboards were created to monitor patient waiting time, to access the clinical notes summary for the next waiting patient, and to obtain insights into activity during the last 24 hours.

Conclusions

The designed system could serve as a monitoring support model using free and user-friendly data science tools, which are good alternatives to proprietary business intelligence solutions and drastically reduce cost. Still, the key to success in decision-making systems is based on investment in human resources, business intelligence skills training, the organizational aspect of the decision-making process, and data production quality insurance.

Missed diagnosis and delayed treatment of acromegaly in a patient with severe diabetes: A case report

The early stages of acromegaly are characterized by slow and progressive acral overgrowth without major systemic complications. Failure to diagnose acromegaly at an early stage may have devastating consequences on patient care. The case in the present report was a 44-year-old Japanese man, referred to Kuwana City Medical Center due to severe hyperglycemia detected in a general checkup. The patient had no acromegaly-related complaints. Laboratory data revealed high blood levels of hemoglobin A1c and glucose. Careful physical examination revealed enlargement of extremities and soft tissues. Laboratory investigation indicated a high blood concentration of growth hormone, and magnetic resonance imaging disclosed an enhanced pituitary tumor. The diagnosis was pituitary tumor-associated acromegaly with severe diabetic complications. The pituitary tumor became large and unresectable following 10 years of misdiagnosis. The patient was treated with somatostatin receptor ligands (lanreotide and pasireotide), as well as bromocriptine in Mie University Hospital. The tumor size was reduced following treatment, though it was still unresectable at the time of this report. The case highlights the importance of hyperglycemia and abnormal manifestations of the feet in patients with acromegaly. In addition, these findings highlight the need for a thorough examination of the feet in diabetic patients, and the critical importance of the early diagnosis of acromegaly for preventing the consequences of inappropriate patient care.

Serious misdiagnosis-related harms in malpractice claims: The “Big Three” – vascular events, infections, and cancers

Background

Diagnostic errors cause substantial preventable harm, but national estimates vary widely from 40,000 to 4 million annually. This cross-sectional analysis of a large medical malpractice claims database was the first phase of a three-phase project to estimate the US burden of serious misdiagnosis-related harms.

Methods

We sought to identify diseases accounting for the majority of serious misdiagnosis-related harms (morbidity/mortality). Diagnostic error cases were identified from Controlled Risk Insurance Company (CRICO)’s Comparative Benchmarking System (CBS) database (2006–2015), representing 28.7% of all US malpractice claims. Diseases were grouped according to the Agency for Healthcare Research and Quality (AHRQ) Clinical Classifications Software (CCS) that aggregates the International Classification of Diseases diagnostic codes into clinically sensible groupings. We analyzed vascular events, infections, and cancers (the “Big Three”), including frequency, severity, and settings. High-severity (serious) harms were defined by scores of 6–9 (serious, permanent disability, or death) on the National Association of Insurance Commissioners (NAIC) Severity of Injury Scale.

Results

From 55,377 closed claims, we analyzed 11,592 diagnostic error cases [median age 49, interquartile range (IQR) 36–60; 51.7% female]. These included 7379 with high-severity harms (53.0% death). The Big Three diseases accounted for 74.1% of high-severity cases (vascular events 22.8%, infections 13.5%, and cancers 37.8%). In aggregate, the top five from each category (n = 15 diseases) accounted for 47.1% of high-severity cases. The most frequent disease in each category, respectively, was stroke, sepsis, and lung cancer. Causes were disproportionately clinical judgment factors (85.7%) across categories (range 82.0–88.8%).

Conclusions

The Big Three diseases account for about three-fourths of serious misdiagnosis-related harms. Initial efforts to improve diagnosis should focus on vascular events, infections, and cancers.

Migraine as a Stroke Mimic and as a Stroke Chameleon

PURPOSE OF REVIEW

This review details the frequency of and ways in which migraine can be both an ischemic stroke/transient ischemic attack mimic (false positive) and chameleon (false negative). We additionally seek to clarify the complex relationships between migraine and cerebrovascular diseases with regard to diagnostic error.

RECENT FINDINGS

Nearly 2% of all patients evaluated emergently for possible stroke have an ultimate diagnosis of migraine; approximately 18% of all stroke mimic patients treated with intravenous thrombolysis have a final diagnosis of migraine. Though the treatment of a patient with migraine with thrombolytics confers a low risk of complication, symptomatic intracerebral hemorrhage may occur. Three clinical prediction scores with high sensitivity and specificity exist that can aid in the diagnosis of acute cerebral ischemia. Differentiating between migraine aura and transient ischemic attacks remains challenging. On the other hand, migraine is a common incorrect diagnosis initially given to patients with stroke. Among patients discharged from an emergency visit to home with a diagnosis of a non-specific headache disorder, 0.5% were misdiagnosed. Further development of tools to quantify and understand sources of stroke misdiagnosis among patients who present with headache is warranted. Both failure to identify cerebral ischemia among patients with headache and overdiagnosis of ischemia can lead to patient harms. While some tools exist to help with acute diagnostic decision-making, additional strategies to improve diagnostic safety among patients with migraine and/or cerebral ischemia are needed.