Electronic health record-based surveillance of diagnostic errors in primary care

Computable phenotype for diagnostic error: developing the data schema for application of symptom-disease pair analysis of diagnostic error (SPADE)

OBJECTIVES

Diagnostic errors are the leading cause of preventable harm in clinical practice. Implementable tools to quantify and target this problem are needed. To address this gap, we aimed to generalize the Symptom-Disease Pair Analysis of Diagnostic Error (SPADE) framework by developing its computable phenotype and then demonstrated how that schema could be applied in multiple clinical contexts.

METHODS

We created an information model for the SPADE processes, then mapped data fields from electronic health records (EHR) and claims data in use to that model to create the SPADE information model (intention) and the SPADE computable phenotype (extension). Later we validated the computable phenotype and tested it in four case studies in three different health systems to demonstrate its utility.

RESULTS

We mapped and tested the SPADE computable phenotype in three different sites using four different case studies. We showed that data fields to compute an SPADE base measure are fully available in the EHR Data Warehouse for extraction and can operationalize the SPADE framework from provider and/or insurer perspective, and they could be implemented on numerous health systems for future work in monitor misdiagnosis-related harms.

CONCLUSIONS

Data for the SPADE base measure is readily available in EHR and administrative claims. The method of data extraction is potentially universally applicable, and the data extracted is conveniently available within a network system. Further study is needed to validate the computable phenotype across different settings with different data infrastructures.

Leveraging learning systems to improve quality and patient safety in allergen immunotherapy

Adverse events occur in all fields of medicine, including allergy-immunology, in which allergen immunotherapy medical errors can cause significant harm. Although difficult to experience, such errors constitute opportunities for improvement. Identifying system vulnerabilities can allow resolution of latent errors before they become active problems. We review key aspects and frameworks of the medical error response, acknowledging the fundamental responsibility of clinical teams to learn from harm. Adverse event response comprises 4 major phases: (1) event recognition and reporting, (2) investigation (for which root cause analysis can be helpful), (3) improvement (inclusive of the plan-do-study-act cycle), and (4) communication and resolution. Throughout the process, clinician wellness must be maintained. Adverse event prevention should be prioritized, and a human factors engineering approach can be useful. Quality improvement tools and approaches complement one another and together offer a meaningful avenue for error recovery and prevention.

Longitudinal Changes in Diagnostic Accuracy of a Differential Diagnosis List Developed by an AI-Based Symptom Checker: Retrospective Observational Study

BACKGROUND

Artificial intelligence (AI) symptom checker models should be trained using real-world patient data to improve their diagnostic accuracy. Given that AI-based symptom checkers are currently used in clinical practice, their performance should improve over time. However, longitudinal evaluations of the diagnostic accuracy of these symptom checkers are limited.

OBJECTIVE

This study aimed to assess the longitudinal changes in the accuracy of differential diagnosis lists created by an AI-based symptom checker used in the real world.

METHODS

This was a single-center, retrospective, observational study. Patients who visited an outpatient clinic without an appointment between May 1, 2019, and April 30, 2022, and who were admitted to a community hospital in Japan within 30 days of their index visit were considered eligible. We only included patients who underwent an AI-based symptom checkup at the index visit, and the diagnosis was finally confirmed during follow-up. Final diagnoses were categorized as common or uncommon, and all cases were categorized as typical or atypical. The primary outcome measure was the accuracy of the differential diagnosis list created by the AI-based symptom checker, defined as the final diagnosis in a list of 10 differential diagnoses created by the symptom checker. To assess the change in the symptom checker's diagnostic accuracy over 3 years, we used a chi-square test to compare the primary outcome over 3 periods: from May 1, 2019, to April 30, 2020 (first year); from May 1, 2020, to April 30, 2021 (second year); and from May 1, 2021, to April 30, 2022 (third year).

RESULTS

A total of 381 patients were included. Common diseases comprised 257 (67.5%) cases, and typical presentations were observed in 298 (78.2%) cases. Overall, the accuracy of the differential diagnosis list created by the AI-based symptom checker was 172 (45.1%), which did not differ across the 3 years (first year: 97/219, 44.3%; second year: 32/72, 44.4%; and third year: 43/90, 47.7%; P=.85). The accuracy of the differential diagnosis list created by the symptom checker was low in those with uncommon diseases (30/124, 24.2%) and atypical presentations (12/83, 14.5%). In the multivariate logistic regression model, common disease (P<.001; odds ratio 4.13, 95% CI 2.50-6.98) and typical presentation (P<.001; odds ratio 6.92, 95% CI 3.62-14.2) were significantly associated with the accuracy of the differential diagnosis list created by the symptom checker.

CONCLUSIONS

A 3-year longitudinal survey of the diagnostic accuracy of differential diagnosis lists developed by an AI-based symptom checker, which has been implemented in real-world clinical practice settings, showed no improvement over time. Uncommon diseases and atypical presentations were independently associated with a lower diagnostic accuracy. In the future, symptom checkers should be trained to recognize uncommon conditions.

Effect of contextual factors on the prevalence of diagnostic errors among patients managed by physicians of the same specialty: a single-centre retrospective observational study

BACKGROUND

There has been growing recognition that contextual factors influence the physician's cognitive processes. However, given that cognitive processes may depend on the physicians' specialties, the effects of contextual factors on diagnostic errors reported in previous studies could be confounded by difference in physicians.

OBJECTIVE

This study aimed to clarify whether contextual factors such as location and consultation type affect diagnostic accuracy.

METHODS

We reviewed the medical records of 1992 consecutive outpatients consulted by physicians from the Department of Diagnostic and Generalist Medicine in a university hospital between 1 January and 31 December 2019. Diagnostic processes were assessed using the Revised Safer Dx Instrument. Patients were categorised into three groups according to contextual factors (location and consultation type): (1) referred patients with scheduled visit to the outpatient department; (2) patients with urgent visit to the outpatient department; and (3) patients with emergency visit to the emergency room. The effect of the contextual factors on the prevalence of diagnostic errors was investigated using logistic regression analysis.

RESULTS

Diagnostic errors were observed in 12 of 534 referred patients with scheduled visit to the outpatient department (2.2%), 3 of 599 patients with urgent visit to the outpatient department (0.5%) and 13 of 859 patients with emergency visit to the emergency room (1.5%). Multivariable logistic regression analysis showed a significantly higher prevalence of diagnostic errors in referred patients with scheduled visit to the outpatient department than in patients with urgent visit to the outpatient department (OR 4.08, p=0.03), but no difference between patients with emergency and urgent visit to the emergency room and outpatient department, respectively.

CONCLUSION

Contextual factors such as consultation type may affect diagnostic errors; however, since the differences in the prevalence of diagnostic errors were small, the effect of contextual factors on diagnostic accuracy may be small in physicians working in different care settings.

Delayed diagnosis of serious paediatric conditions in 13 regional emergency departments

OBJECTIVE

To evaluate rates, risk factors and outcomes of delayed diagnosis of seven serious paediatric conditions.

METHODS

This was a retrospective, cross-sectional study of children under 21 years old visiting 13 community and tertiary emergency departments (EDs) with appendicitis, bacterial meningitis, intussusception, mastoiditis, ovarian torsion, sepsis or testicular torsion. Delayed diagnosis was defined as having a previous ED encounter within 1 week in which the condition was present per case review. Patients with delayed diagnosis were each matched to four control patients without delay by condition, facility and age. Conditional logistic regression models evaluated risk factors of delay. Complications were compared between by delayed diagnosis status.

RESULTS

Among 14 972 children, delayed diagnosis occurred in 1.1% (range 0.3% for sepsis to 2.6% for ovarian torsion). Hispanic (matched OR 2.71, 95% CI 1.69 to 4.35) and non-Hispanic black (OR 2.40, 95% CI 1.21 to 4.79) race/ethnicity were associated with delayed diagnosis, whereas Asian and other race/ethnicity were not. Public (OR 2.21, 95% CI 1.42 to 3.44) and other (OR 2.43, 95% CI 1.50 to 3.93) insurance were also associated with delay. Non-English language was associated with delay (OR 1.65, 95% CI 1.02 to 2.69). Abnormal vital signs were associated with a lower likelihood of delay (OR 0.15, 95% CI 0.09 to 0.25). In an adjusted model, Hispanic race/ethnicity, other insurance, abnormal vital signs and complex chronic conditions (CCCs) were associated with delay. The odds of a complication were 2.5-fold (95% CI 1.6 to 3.8) higher among patients with a delay.

CONCLUSION

Delayed diagnosis was uncommon across 13 regional EDs but was more likely among children with Hispanic ethnicity, CCCs or normal vital signs. Delays were associated with a higher risk of complications.

Emergency Department Volume and Delayed Diagnosis of Pediatric Appendicitis: A Retrospective Cohort Study

OBJECTIVE

To determine the association of emergency department (ED) volume of children and delayed diagnosis of appendicitis.

BACKGROUND

Delayed diagnosis of appendicitis is common in children. The association between ED volume and delayed diagnosis is uncertain, but diagnosis-specific experience might improve diagnostic timeliness.

METHODS

Using Healthcare Cost and Utilization Project 8-state data from 2014 to 2019, we studied all children with appendicitis <18 years old in all EDs. The main outcome was probable delayed diagnosis: >75% likelihood that a delay occurred based on a previously validated measure. Hierarchical models tested associations between ED volumes and delay, adjusting for age, sex, and chronic conditions. We compared complication rates by delayed diagnosis occurrence.

RESULTS

Among 93,136 children with appendicitis, 3,293 (3.5%) had delayed diagnosis. Each 2-fold increase in ED volume was associated with a 6.9% (95% CI: 2.2, 11.3) decreased odds of delayed diagnosis. Each 2-fold increase in appendicitis volume was associated with a 24.1% (95% CI: 21.0, 27.0) decreased odds of delay. Those with delayed diagnosis were more likely to receive intensive care [odds ratio (OR): 1.81, 95% CI: 1.48, 2.21], have perforated appendicitis (OR: 2.81, 95% CI: 2.62, 3.02), undergo abdominal abscess drainage (OR: 2.49, 95% CI: 2.16, 2.88), have multiple abdominal surgeries (OR: 2.56, 95% CI: 2.13, 3.07), or develop sepsis (OR: 2.02, 95% CI: 1.61, 2.54).

CONCLUSIONS

Higher ED volumes were associated with a lower risk of delayed diagnosis of pediatric appendicitis. Delay was associated with complications.

Validation of a Reduced Set of High-Performance Triggers for Identifying Patient Safety Incidents with Harm in Primary Care: TriggerPrim Project

OBJECTIVE

The aim of the study was to construct and validate a reduced set of high-performance triggers for identifying adverse events (AEs) via electronic medical records (EMRs) review in primary care (PC).

METHODS

This was a cross-sectional descriptive study for validating a diagnostic test. The study included all 262 PC centers of Madrid region (Spain). Patients were older than 18 years who attended their PC center over the last quarter of 2018. The randomized sample was n = 1797. Main measurements were as follows: ( a ) presence of each of 19 specific computer-identified triggers in the EMR and ( b ) occurrence of an AE. To collect data, EMR review was conducted by 3 doctor-nurse teams. Triggers with statistically significant odds ratios for identifying AEs were selected for the final set after adjusting for age and sex using logistic regression.

RESULTS

The sensitivity (SS) and specificity (SP) for the selected triggers were: ≥3 appointments in a week at the PC center (SS = 32.3% [95% confidence interval {CI}, 22.8%-41.8%]; SP = 92.8% [95% CI, 91.6%-94.0%]); hospital admission (SS = 19.4% [95% CI, 11.4%-27.4%]; SP = 97.2% [95% CI, 96.4%-98.0%]); hospital emergency department visit (SS = 31.2% [95% CI, 21.8%-40.6%]; SP = 90.8% [95% CI, 89.4%-92.2%]); major opioids prescription (SS = 2.2% [95% CI, 0.0%-5.2%]; SP = 99.8% [95% CI, 99.6%-100%]); and chronic benzodiazepine treatment in patients 75 years or older (SS = 14.0% [95% CI, 6.9%-21.1%]; SP = 95.5% [95% CI, 94.5%-96.5%]).The following values were obtained in the validation of this trigger set (the occurrence of at least one of these triggers in the EMR): SS = 60.2% (95% CI, 50.2%-70.1%), SP = 80.8% (95% CI, 78.8%-82.6%), positive predictive value = 14.6% (95% CI, 11.0%-18.1%), negative predictive value = 97.4% (95% CI, 96.5%-98.2%), positive likelihood ratio = 3.13 (95% CI, 2.3-4.2), and negative likelihood ratio = 0.49 (95% CI, 0.3-0.7).

CONCLUSIONS

The set containing the 5 selected triggers almost triples the efficiency of EMR review in detecting AEs. This suggests that this set is easily implementable and of great utility in risk-management practice.

Achieving diagnostic excellence through prevention and teamwork (ADEPT) study protocol: A multicenter, prospective quality and safety program to improve diagnostic processes in medical inpatients

BACKGROUND

Few hospitals have built surveillance for diagnostic errors into usual care or used comparative quantitative and qualitative data to understand their diagnostic processes and implement interventions designed to reduce these errors.

OBJECTIVES

To build surveillance for diagnostic errors into usual care, benchmark diagnostic performance across sites, pilot test interventions, and evaluate the program's impact on diagnostic error rates.

METHODS AND ANALYSIS

Achieving diagnostic excellence through prevention and teamwork (ADEPT) is a multicenter, real-world quality and safety program utilizing interrupted time-series techniques to evaluate outcomes. Study subjects will be a randomly sampled population of medical patients hospitalized at 16 US hospitals who died, were transferred to intensive care, or had a rapid response during the hospitalization. Surveillance for diagnostic errors will occur on 10 events per month per site using a previously established two-person adjudication process. Concurrent reviews of patients who had a qualifying event in the previous week will allow for surveys of clinicians to better understand contributors to diagnostic error, or conversely, examples of diagnostic excellence, which cannot be gleaned from medical record review alone. With guidance from national experts in quality and safety, sites will report and benchmark diagnostic error rates, share lessons regarding underlying causes, and design, implement, and pilot test interventions using both Safety I and Safety II approaches aimed at patients, providers, and health systems. Safety II approaches will focus on cases where diagnostic error did not occur, applying theories of how people and systems are able to succeed under varying conditions. The primary outcome will be the number of diagnostic errors per patient, using segmented multivariable regression to evaluate change in y-intercept and change in slope after initiation of the program.

ETHICS AND DISSEMINATION

The study has been approved by the University of California, San Francisco Institutional Review Board (IRB), which is serving as the single IRB. Intervention toolkits and study findings will be disseminated through partners including Vizient, The Joint Commission, and Press-Ganey, and through national meetings, scientific journals, and publications aimed at the general public.

Multicenter evaluation of a method to identify delayed diagnosis of diabetic ketoacidosis and sepsis in administrative data

OBJECTIVES

To derive a method of automated identification of delayed diagnosis of two serious pediatric conditions seen in the emergency department (ED): new-onset diabetic ketoacidosis (DKA) and sepsis.

METHODS

Patients under 21 years old from five pediatric EDs were included if they had two encounters within 7 days, the second resulting in a diagnosis of DKA or sepsis. The main outcome was delayed diagnosis based on detailed health record review using a validated rubric. Using logistic regression, we derived a decision rule evaluating the likelihood of delayed diagnosis using only characteristics available in administrative data. Test characteristics at a maximal accuracy threshold were determined.

RESULTS

Delayed diagnosis was present in 41/46 (89 %) of DKA patients seen twice within 7 days. Because of the high rate of delayed diagnosis, no characteristic we tested added predictive power beyond the presence of a revisit. For sepsis, 109/646 (17 %) of patients were deemed to have a delay in diagnosis. Fewer days between ED encounters was the most important characteristic associated with delayed diagnosis. In sepsis, our final model had a sensitivity for delayed diagnosis of 83.5 % (95 % confidence interval 75.2-89.9) and specificity of 61.3 % (95 % confidence interval 56.0-65.4).

CONCLUSIONS

Children with delayed diagnosis of DKA can be identified by having a revisit within 7 days. Many children with delayed diagnosis of sepsis may be identified using this approach with low specificity, indicating the need for manual case review.

A taxonomy of early diagnosis research to guide study design and funding prioritisation

Researchers and research funders aiming to improve diagnosis seek to identify if, when, where, and how earlier diagnosis is possible. This has led to the propagation of research studies using a wide range of methodologies and data sources to explore diagnostic processes. Many such studies use electronic health record data and focus on cancer diagnosis. Based on this literature, we propose a taxonomy to guide the design and support the synthesis of early diagnosis research, focusing on five key questions: Do healthcare use patterns suggest earlier diagnosis could be possible? How does the diagnostic process begin? How do patients progress from presentation to diagnosis? How long does the diagnostic process take? Could anything have been done differently to reach the correct diagnosis sooner? We define families of diagnostic research study designs addressing each of these questions and appraise their unique or complementary contributions and limitations. We identify three further questions on relationships between the families and their relevance for examining patient group inequalities, supported with examples from the cancer literature. Although exemplified through cancer as a disease model, we recognise the framework is also applicable to non-neoplastic disease. The proposed framework can guide future study design and research funding prioritisation.

Diagnostic error experiences of patients and families with limited English-language health literacy or disadvantaged socioeconomic position in a cross-sectional US population-based survey

BACKGROUND

Language barrier, reduced self-advocacy, lower health literacy or biased care may hinder the diagnostic process. Data on how patients/families with limited English-language health literacy (LEHL) or disadvantaged socioeconomic position (dSEP) experience diagnostic errors are sparse.

METHOD

We compared patient-reported diagnostic errors, contributing factors and impacts between respondents with LEHL or dSEP and their counterparts in the 2017 Institute for Healthcare Improvement US population-based survey, using contingency analysis and multivariable logistic regression models for the analyses.

RESULTS

596 respondents reported a diagnostic error; among these, 381 reported LEHL or dSEP. After adjusting for sex, race/ethnicity and physical health, individuals with LEHL/dSEP were more likely than their counterparts to report unique contributing factors: "(No) qualified translator or healthcare provider that spoke (the patient's) language" (OR and 95% CI 4.4 (1.3 to 14.9)); "not understanding the follow-up plan" (1.9 (1.1 to 3.1)); "too many providers… but no clear leader" (1.8 (1.2 to 2.7)); "not able to keep follow-up appointments" (1.9 (1.1 to 3.2)); "not being able to pay for necessary medical care" (2.5 (1.4 to 4.4)) and "out-of-date or incorrect medical records" (2.6 (1.4 to 4.8)). Participants with LEHL/dSEP were more likely to report long-term emotional, financial and relational impacts, compared with their counterparts. Subgroup analysis (LEHL-only and dSEP-only participants) showed similar results.

CONCLUSIONS

Individuals with LEHL or dSEP identified unique and actionable contributing factors to diagnostic errors. Interpreter access should be viewed as a diagnostic safety imperative, social determinants affecting care access/affordability should be routinely addressed as part of the diagnostic process and patients/families should be encouraged to access and update their medical records. The frequent and disproportionate long-term impacts from self-reported diagnostic error among LEHL/dSEP patients/families raises urgency for greater prevention and supportive efforts.

Physicians' Perspectives on a Multi-Dimensional Model for the Roles of Electronic Health Records in Approaching a Proper Differential Diagnosis

Many healthcare organizations have adopted Electronic Health Records (EHRs) to improve the quality of care and help physicians make proper clinical decisions. The vital roles of EHRs can support the accuracy of diagnosis, suggest, and rationalize the provided care to patients. This study aims to understand the roles of EHRs in approaching proper differential diagnosis and optimizing patient safety. This study utilized a cross-sectional survey-based descriptive research design to assess physicians' perceptions of the roles of EHRs on diagnosis quality and safety. Physicians working in tertiary hospitals in Saudi Arabia were surveyed. Three hundred and fifty-one participants were included in the study, of which 61% were male. The main participants were family/general practice (22%), medicine, general (14%), and OB/GYN (12%). Overall, 66% of the participants ranked themselves as IT competent, most of the participants underwent IT self-guided learning, and 65% of the participants always used the system. The results generally reveal positive physicians' perceptions toward the roles of the EHR system on diagnosis quality and safety. There was a statistically significant relationship between user characteristics and the roles of the EHR by enhancing access to care, patient-physician encounter, clinical reasoning, diagnostic testing and consultation, follow-up, and diagnostic safety functionality. The study participants demonstrate positive perceptions of physicians toward the roles of the EHR system in approaching differential diagnosis. Yet, areas of improvement in the design and using EHRs are emphasized.

A comprehensive framework to estimate the frequency, duration, and risk factors for diagnostic delays using bootstrapping-based simulation methods

BACKGROUND

The incidence of diagnostic delays is unknown for many diseases and specific healthcare settings. Many existing methods to identify diagnostic delays are resource intensive or difficult to apply to different diseases or settings. Administrative and other real-world data sources may offer the ability to better identify and study diagnostic delays for a range of diseases.

METHODS

We propose a comprehensive framework to estimate the frequency of missed diagnostic opportunities for a given disease using real-world longitudinal data sources. We provide a conceptual model of the disease-diagnostic, data-generating process. We then propose a bootstrapping method to estimate measures of the frequency of missed diagnostic opportunities and duration of delays. This approach identifies diagnostic opportunities based on signs and symptoms occurring prior to an initial diagnosis, while accounting for expected patterns of healthcare that may appear as coincidental symptoms. Three different bootstrapping algorithms are described along with estimation procedures to implement the resampling. Finally, we apply our approach to the diseases of tuberculosis, acute myocardial infarction, and stroke to estimate the frequency and duration of diagnostic delays for these diseases.

RESULTS

Using the IBM MarketScan Research databases from 2001 to 2017, we identified 2,073 cases of tuberculosis, 359,625 cases of AMI, and 367,768 cases of stroke. Depending on the simulation approach that was used, we estimated that 6.9-8.3% of patients with stroke, 16.0-21.3% of patients with AMI and 63.9-82.3% of patients with tuberculosis experienced a missed diagnostic opportunity. Similarly, we estimated that, on average, diagnostic delays lasted 6.7-7.6 days for stroke, 6.7-8.2 days for AMI, and 34.3-44.5 days for tuberculosis. Estimates for each of these measures was consistent with prior literature; however, specific estimates varied across the different simulation algorithms considered.

CONCLUSIONS

Our approach can be easily applied to study diagnostic delays using longitudinal administrative data sources. Moreover, this general approach can be customized to fit a range of diseases to account for specific clinical characteristics of a given disease. We summarize how the choice of simulation algorithm may impact the resulting estimates and provide guidance on the statistical considerations for applying our approach to future studies.

Types of diagnostic errors reported by paediatric emergency providers in a global paediatric emergency care research network

Background

Diagnostic errors, reframed as missed opportunities for improving diagnosis (MOIDs), are poorly understood in the paediatric emergency department (ED) setting. We investigated the clinical experience, harm and contributing factors related to MOIDs reported by physicians working in paediatric EDs.

Methods

We developed a web-based survey in which physicians participating in the international Paediatric Emergency Research Network representing five out of six WHO regions, described examples of MOIDs involving their own or a colleague’s patients. Respondents provided case summaries and answered questions regarding harm and factors contributing to the event.

Results

Of 1594 physicians surveyed, 412 (25.8%) responded (mean age=43 years (SD=9.2), 42.0% female, mean years in practice=12 (SD=9.0)). Patient presentations involving MOIDs had common undifferentiated symptoms at initial presentation, including abdominal pain (21.1%), fever (17.2%) and vomiting (16.5%). Patients were discharged from the ED with commonly reported diagnoses, including acute gastroenteritis (16.7%), viral syndrome (10.2%) and constipation (7.0%). Most reported MOIDs (65%) were detected on ED return visits (46% within 24 hours and 76% within 72 hours). The most common reported MOID was appendicitis (11.4%), followed by brain tumour (4.4%), meningitis (4.4%) and non-accidental trauma (4.1%). More than half (59.1%) of the reported MOIDs involved the patient/parent–provider encounter (eg, misinterpreted/ignored history or an incomplete/inadequate physical examination). Types of MOIDs and contributing factors did not differ significantly between countries. More than half of patients had either moderate (48.7%) or major (10%) harm due to the MOID.

Conclusions

An international cohort of paediatric ED physicians reported several MOIDs, often in children who presented to the ED with common undifferentiated symptoms. Many of these were related to patient/parent–provider interaction factors such as suboptimal history and physical examination. Physicians’ personal experiences offer an underexplored source for investigating and mitigating diagnostic errors in the paediatric ED.

Identification of delayed diagnosis of paediatric appendicitis in administrative data: a multicentre retrospective validation study

OBJECTIVE

To derive and validate a tool that retrospectively identifies delayed diagnosis of appendicitis in administrative data with high accuracy.

DESIGN

Cross-sectional study.

SETTING

Five paediatric emergency departments (EDs).

PARTICIPANTS

669 patients under 21 years old with possible delayed diagnosis of appendicitis, defined as two ED encounters within 7 days, the second with appendicitis.

OUTCOME

Delayed diagnosis was defined as appendicitis being present but not diagnosed at the first ED encounter based on standardised record review. The cohort was split into derivation (2/3) and validation (1/3) groups. We derived a prediction rule using logistic regression, with covariates including variables obtainable only from administrative data. The resulting trigger tool was applied to the validation group to determine area under the curve (AUC). Test characteristics were determined at two predicted probability thresholds.

RESULTS

Delayed diagnosis occurred in 471 (70.4%) patients. The tool had an AUC of 0.892 (95% CI 0.858 to 0.925) in the derivation group and 0.859 (95% CI 0.806 to 0.912) in the validation group. The positive predictive value (PPV) for delay at a maximal accuracy threshold was 84.7% (95% CI 78.2% to 89.8%) and identified 87.3% of delayed cases. The PPV at a stricter threshold was 94.9% (95% CI 87.4% to 98.6%) and identified 46.8% of delayed cases.

CONCLUSIONS

This tool accurately identified delayed diagnosis of appendicitis. It may be used to screen for potential missed diagnoses or to specifically identify a cohort of children with delayed diagnosis.

A structured approach to EHR surveillance of diagnostic error in acute care: an exploratory analysis of two institutionally-defined case cohorts

OBJECTIVES

To test a structured electronic health record (EHR) case review process to identify diagnostic errors (DE) and diagnostic process failures (DPFs) in acute care.

METHODS

We adapted validated tools (Safer Dx, Diagnostic Error Evaluation Research [DEER] Taxonomy) to assess the diagnostic process during the hospital encounter and categorized 13 postulated e-triggers. We created two test cohorts of all preventable cases (n=28) and an equal number of randomly sampled non-preventable cases (n=28) from 365 adult general medicine patients who expired and underwent our institution's mortality case review process. After excluding patients with a length of stay of more than one month, each case was reviewed by two blinded clinicians trained in our process and by an expert panel. Inter-rater reliability was assessed. We compared the frequency of DE contributing to death in both cohorts, as well as mean DPFs and e-triggers for DE positive and negative cases within each cohort.

RESULTS

Twenty-seven (96.4%) preventable and 24 (85.7%) non-preventable cases underwent our review process. Inter-rater reliability was moderate between individual reviewers (Cohen's kappa 0.41) and substantial with the expert panel (Cohen's kappa 0.74). The frequency of DE contributing to death was significantly higher for the preventable compared to the non-preventable cohort (56% vs. 17%, OR 6.25 [1.68, 23.27], p<0.01). Mean DPFs and e-triggers were significantly and non-significantly higher for DE positive compared to DE negative cases in each cohort, respectively.

CONCLUSIONS

We observed substantial agreement among final consensus and expert panel reviews using our structured EHR case review process. DEs contributing to death associated with DPFs were identified in institutionally designated preventable and non-preventable cases. While e-triggers may be useful for discriminating DE positive from DE negative cases, larger studies are required for validation. Our approach has potential to augment institutional mortality case review processes with respect to DE surveillance.

The e-Autopsy/e-Biopsy: a systematic chart review to increase safety and diagnostic accuracy

Solving diagnostic errors is difficult and progress on preventing those errors has been slow since the 2015 National Academy of Medicine report. There are several methods used to improve diagnostic and other errors including voluntary reporting; malpractice claims; patient complaints; physician surveys, random quality reviews and audits, and peer review data which usually evaluates single cases and not the systems that allowed the error. Additionally, manual review of charts is often labor intensive and reviewer dependent. In 2010 we developed an e-Autopsy/e-Biopsy (eA/eB) methodology to aggregate cases with quality/safety/diagnostic issues, focusing on a specific population of patients and conditions. By performing a hybrid review process (cases are first filtered using administrative data followed by standardized manual chart reviews) we can efficiently identify patterns of medical and diagnostic error leading to opportunities for system improvements that have improved care for future patients. We present a detailed methodology for eA/eB studies and describe results from three successful studies on different diagnoses (ectopic pregnancy, abdominal aortic aneurysms, and advanced colon cancer) that illustrate our eA/eB process and how it reveals insights into creating systems that reduce diagnostic and other errors. The eA/eB process is innovative and transferable to other healthcare organizations and settings to identify trends in diagnostic error and other quality issues resulting in improved systems of care.

Frequency of diagnostic errors in the neonatal intensive care unit: a retrospective cohort study

OBJECTIVE

To determine the frequency and etiology of diagnostic errors during the first 7 days of admission for inborn neonatal intensive care unit (NICU) patients.

STUDY DESIGN

We conducted a retrospective cohort study of 600 consecutive inborn admissions. A physician used the "Safer Dx NICU Instrument" to review the electronic health record for the first 7 days of admission, and categorized cases as "yes," "unclear," or "no" for diagnostic error. A secondary reviewer evaluated all "yes" charts plus a random sample of charts in the other categories. Subsequently, all secondary reviewers reviewed records with discordance between primary and secondary review to arrive at consensus.

RESULTS

We identified 37 diagnostic errors (6.2% of study patients) with "substantial agreement" between reviewers (κ = 0.66). The most common diagnostic process breakdown was missed maternal history (51%).

CONCLUSION

The frequency of diagnostic error in inborn NICU patients during the first 7 days of admission is 6.2%.

Use of e-triggers to identify diagnostic errors in the paediatric ED

BACKGROUND

Diagnostic errors (DxEs) are an understudied source of patient harm in children rarely captured in current adverse event reporting systems. Applying electronic triggers (e-triggers) to electronic health records shows promise in identifying DxEs but has not been used in the emergency department (ED) setting.

OBJECTIVES

To assess the performance of an e-trigger and subsequent manual screening for identifying probable DxEs among children with unplanned admission following a prior ED visit and to compare performance to existing incident reporting systems.

DESIGN/METHODS

Retrospective single-centre cohort study of children ages 0-22 admitted within 14 days of a previous ED visit between 1 January 2018 and 31 December 2019. Subjects were identified by e-trigger, screened to identify cases where index visit and hospital discharge diagnoses were potentially related but pathophysiologically distinct, and then these screened-in cases were reviewed for DxE using the SaferDx Instrument. Cases of DxE identified by e-trigger were cross-referenced against existing institutional incident reporting systems.

RESULTS

An e-trigger identified 1915 unplanned admissions (7.7% of 24 849 total admissions) with a preceding index visit. 453 (23.7%) were screened in and underwent review using SaferDx. 92 cases were classified as likely DxEs, representing 0.4% of all hospital admissions, 4.8% among those selected by e-trigger and 20.3% among those screened in for review. Half of cases were reviewed by two reviewers using SaferDx with substantial inter-rater reliability (Cohen's κ=0.65 (95% CI 0.54 to 0.75)). Six (6.5%) cases had been reported elsewhere: two to the hospital's incident reporting system and five to the ED case review team (one reported to both).

CONCLUSION

An e-trigger coupled with manual screening enriched a cohort of patients at risk for DxEs. Fewer than 10% of DxEs were identified through existing surveillance systems, suggesting that they miss a large proportion of DxEs. Further study is required to identify specific clinical presentations at risk of DxEs.

Will Any Road Get You There? Examining Warranted and Unwarranted Variation in Medical Education

Undergraduate and graduate medical education have long embraced uniqueness and variability in curricular and assessment approaches. Some of this variability is justified (warranted or necessary variation), but a substantial portion represents unwarranted variation. A primary tenet of outcomes-based medical education is ensuring that all learners acquire essential competencies to be publicly accountable to meet societal needs. Unwarranted variation in curricular and assessment practices contributes to suboptimal and variable educational outcomes and, by extension, risks graduates delivering suboptimal health care quality. Medical education can use lessons from the decades of study on unwarranted variation in health care as part of efforts to continuously improve the quality of training programs. To accomplish this, medical educators will first need to recognize the difference between warranted and unwarranted variation in both clinical care and educational practices. Addressing unwarranted variation will require cooperation and collaboration between multiple levels of the health care and educational systems using a quality improvement mindset. These efforts at improvement should acknowledge that some aspects of variability are not scientifically informed and do not support desired outcomes or societal needs. This perspective examines the correlates of unwarranted variation of clinical care in medical education and the need to address the interdependency of unwarranted variation occurring between clinical and educational practices. The authors explore the challenges of variation across multiple levels: community, institution, program, and individual faculty members. The article concludes with recommendations to improve medical education by embracing the principles of continuous quality improvement to reduce the harmful effect of unwarranted variation.

Inviting patients to identify diagnostic concerns through structured evaluation of their online visit notes

BACKGROUND

The 21st Century Cures Act mandates patients' access to their electronic health record (EHR) notes. To our knowledge, no previous work has systematically invited patients to proactively report diagnostic concerns while documenting and tracking their diagnostic experiences through EHR-based clinician note review.

OBJECTIVE

To test if patients can identify concerns about their diagnosis through structured evaluation of their online visit notes.

METHODS

In a large integrated health system, patients aged 18-85 years actively using the patient portal and seen between October 2019 and February 2020 were invited to respond to an online questionnaire if an EHR algorithm detected any recent unexpected return visit following an initial primary care consultation ("at-risk" visit). We developed and tested an instrument (Safer Dx Patient Instrument) to help patients identify concerns related to several dimensions of the diagnostic process based on notes review and recall of recent "at-risk" visits. Additional questions assessed patients' trust in their providers and their general feelings about the visit. The primary outcome was a self-reported diagnostic concern. Multivariate logistic regression tested whether the primary outcome was predicted by instrument variables.

RESULTS

Of 293 566 visits, the algorithm identified 1282 eligible patients, of whom 486 responded. After applying exclusion criteria, 418 patients were included in the analysis. Fifty-one patients (12.2%) identified a diagnostic concern. Patients were more likely to report a concern if they disagreed with statements "the care plan the provider developed for me addressed all my medical concerns" [odds ratio (OR), 2.65; 95% confidence interval [CI], 1.45-4.87) and "I trust the provider that I saw during my visit" (OR, 2.10; 95% CI, 1.19-3.71) and agreed with the statement "I did not have a good feeling about my visit" (OR, 1.48; 95% CI, 1.09-2.01).

CONCLUSION

Patients can identify diagnostic concerns based on a proactive online structured evaluation of visit notes. This surveillance strategy could potentially improve transparency in the diagnostic process.

Diagnostic Error in Pediatrics: A Narrative Review

A priority topic for patient safety research is diagnostic errors. However, despite the significant growth in awareness of their unacceptably high incidence and associated harm, a relative paucity of large, high-quality studies of diagnostic error in pediatrics exists. In this narrative review, we present what is known about the incidence and epidemiology of diagnostic error in pediatrics as well as the established research methods for identifying, evaluating, and reducing diagnostic errors, including their strengths and weaknesses. Additionally, we highlight that pediatric diagnostic error remains an area in need of both innovative research and quality improvement efforts to apply learnings from a rapidly growing evidence base. We propose several key research questions aimed at addressing persistent gaps in the pediatric diagnostic error literature that focus on the foundational knowledge needed to inform effective interventions to reduce the incidence of diagnostic errors and their associated harm. Additional research is needed to better establish the epidemiology of diagnostic error in pediatrics, including identifying high-risk clinical scenarios, patient populations, and groups of diagnoses. A critical need exists for validated measures of both diagnostic errors and diagnostic processes that can be adapted for different clinical settings and standardized for use across varying institutions. Pediatric researchers will need to work collaboratively on large-scale, high-quality studies to accomplish the ultimate goal of reducing diagnostic errors and their associated harm in children by addressing these fundamental gaps in knowledge.

[Diagnostic error and overdiagnosis in Primary Care. Proposals for the improvement of clinical practice family medicine]

Family doctors see a wide range of patients, with a wide range of complexity, in a short time and with few diagnostic resources. This situation makes primary care professionals more vulnerable to diagnostic errors. For this reason, an adequate clinical reasoning process is the most powerful tool family doctors have to safely guide the patient care process. Considering these errors as missed opportunities for a correct diagnosis, which may cause harm to the patient, leads us as professionals to review how to improve this process. The review includes, among other aspects, identifying cognitive biases, analysing the ways in which work is organised in primary care teams, and situations in the care context that may contribute to such errors. In this article we describe the most frequent diagnostic errors and their causal factors in primary care, the impact of cognitive process failures, situations of overdiagnosis and the diagnostic and therapeutic cascades associated with them. Finally, we propose a set of tools to improve decision-making in the diagnostic process in primary care.

Harnessing the electronic health care record to optimise patient safety in primary care: a framework for evaluating “electronic safety netting” tools (Preprint)

The management of diagnostic uncertainty is part of every primary care physician’s role. e–Safety-netting tools help health care professionals to manage diagnostic uncertainty. Using software in addition to verbal or paper based safety-netting methods could make diagnostic delays and errors less likely. There are an increasing number of software products that have been identified as e–safety-netting tools, particularly since the start of the COVID-19 pandemic. e–Safety-netting tools can have a variety of functions, such as sending clinician alerts, facilitating administrative tasking, providing decision support, and sending reminder text messages to patients. However, these tools have not been evaluated by using robust research designs for patient safety interventions. We present an emergent framework of criteria for effective e–safety-netting tools that can be used to support the development of software. The framework is based on validated frameworks for electronic health record development and patient safety. There are currently no tools available that meet all of the criteria in the framework. We hope that the framework will stimulate clinical and public conversations about e–safety-netting tools. In the future, a validated framework would drive audits and improvements. We outline key areas for future research both in primary care and within integrated care systems.

Incidence, origins and avoidable harm of missed opportunities in diagnosis: longitudinal patient record review in 21 English general practices

BACKGROUND

Diagnostic error is a global patient safety priority.

OBJECTIVES

To estimate the incidence, origins and avoidable harm of diagnostic errors in English general practice. Diagnostic errors were defined as missed opportunities to make a correct or timely diagnosis based on the evidence available (missed diagnostic opportunities, MDOs).

METHOD

Retrospective medical record reviews identified MDOs in 21 general practices. In each practice, two trained general practitioner reviewers independently conducted case note reviews on 100 randomly selected adult consultations performed during 2013-2014. Consultations where either reviewer identified an MDO were jointly reviewed.

RESULTS

Across 2057 unique consultations, reviewers agreed that an MDO was possible, likely or certain in 89 cases or 4.3% (95% CI 3.6% to 5.2%) of reviewed consultations. Inter-reviewer agreement was higher than most comparable studies (Fleiss' kappa=0.63). Sixty-four MDOs (72%) had two or more contributing process breakdowns. Breakdowns involved problems in the patient-practitioner encounter such as history taking, examination or ordering tests (main or secondary factor in 61 (68%) cases), performance and interpretation of diagnostic tests (31; 35%) and follow-up and tracking of diagnostic information (43; 48%). 37% of MDOs were rated as resulting in moderate to severe avoidable patient harm.

CONCLUSIONS

Although MDOs occurred in fewer than 5% of the investigated consultations, the high numbers of primary care contacts nationally suggest that several million patients are potentially at risk of avoidable harm from MDOs each year. Causes of MDOs were frequently multifactorial, suggesting the need for development and evaluation of multipronged interventions, along with policy changes to support them.

Tackling Ambulatory Safety Risks Through Patient Engagement: What 10,000 Patients and Families Say About Safety-Related Knowledge, Behaviors, and Attitudes After Reading Visit Notes

BACKGROUND

Ambulatory safety risks including delayed diagnoses or missed abnormal test results are difficult for clinicians to see, because they often occur in the space between visits. Experts advocate greater patient engagement to improve safety, but strategies are limited. Patient access to clinical notes ("OpenNotes") may help close the safety gap between visits.

METHODS

We surveyed patients and families who logged on to the patient portal and had at least one ambulatory note available in the past 12 months at two academic hospitals during June to September 2016, focusing on patient-reported effects of OpenNotes on safety knowledge, behaviors, and attitudes.

RESULTS

A total of 6913 (28%) of 24,722 patients at an adult hospital and 3672 (17%) of 21,579 participants at the children's hospital submitted surveys. Approximately 75% of patients and parents each reported that reading notes helped them understand the reason for both tests and referrals, and approximately 50% felt that it helped them complete tests and referrals. Roughly 75% of participants were more likely to check and understand test results. Overall, 97% of participants reported that trust in the provider, activation, patient-provider goal alignment, and teamwork were each better or the same after reading 1 note or more. Nonwhite participants and those with high school education or less were 30% to 50% more likely to report that reading notes helped them complete tests compared with white and more educated respondents, respectively.

CONCLUSIONS

Overall, the majority of more than 10,000 patients and parents reported reading notes helped them understand and follow through on tests and referrals. As information transparency spreads, OpenNotes can help activate patients and families, facilitate safety behaviors, and forge stronger partnerships with clinicians.

An Operational Framework to Study Diagnostic Errors in Emergency Departments: Findings From A Consensus Panel

OBJECTIVE

To create an operational definition and framework to study diagnostic error in the emergency department setting.

METHODS

We convened a 17-member multidisciplinary panel with expertise in general and pediatric emergency medicine, nursing, patient safety, informatics, cognitive psychology, social sciences, human factors, and risk management and a patient/caregiver advocate. We used a modified nominal group technique to develop a shared understanding to operationally define diagnostic errors in emergency care and modify the National Academies of Sciences, Engineering, and Medicine's conceptual process framework to this setting.

RESULTS

The expert panel defined diagnostic errors as "a divergence from evidence-based processes that increases the risk of poor outcomes despite the availability of sufficient information to provide a timely and accurate explanation of the patient's health problem(s)." Diagnostic processes include tasks related to (a) acuity recognition, information and synthesis, evaluation coordination, and (b) communication with patients/caregivers and other diagnostic team members. The expert panel also modified the National Academies of Sciences, Engineering, and Medicine's diagnostic process framework to incorporate influence of mode of arrival, triage level, and interventions during emergency care and underscored the importance of outcome feedback to emergency department providers to promote learning and improvement related to diagnosis.

CONCLUSIONS

The proposed operational definition and modified diagnostic process framework can potentially inform the development of measurement tools and strategies to study the epidemiology and interventions to improve emergency care diagnosis.

Incidence of diagnostic errors in unplanned hospitalized patients using an automated medical history-taking system with differential diagnosis generator: retrospective observational study (Preprint)

Background

Automated medical history–taking systems that generate differential diagnosis lists have been suggested to contribute to improved diagnostic accuracy. However, the effect of these systems on diagnostic errors in clinical practice remains unknown.

Objective

This study aimed to assess the incidence of diagnostic errors in an outpatient department, where an artificial intelligence (AI)–driven automated medical history–taking system that generates differential diagnosis lists was implemented in clinical practice.

Methods

We conducted a retrospective observational study using data from a community hospital in Japan. We included patients aged 20 years and older who used an AI-driven, automated medical history–taking system that generates differential diagnosis lists in the outpatient department of internal medicine for whom the index visit was between July 1, 2019, and June 30, 2020, followed by unplanned hospitalization within 14 days. The primary endpoint was the incidence of diagnostic errors, which were detected using the Revised Safer Dx Instrument by at least two independent reviewers. To evaluate the effect of differential diagnosis lists from the AI system on the incidence of diagnostic errors, we compared the incidence of these errors between a group where the AI system generated the final diagnosis in the differential diagnosis list and a group where the AI system did not generate the final diagnosis in the list; the Fisher exact test was used for comparison between these groups. For cases with confirmed diagnostic errors, further review was conducted to identify the contributing factors of these errors via discussion among three reviewers, using the Safer Dx Process Breakdown Supplement as a reference.

Results

A total of 146 patients were analyzed. A final diagnosis was confirmed for 138 patients and was observed in the differential diagnosis list from the AI system for 69 patients. Diagnostic errors occurred in 16 out of 146 patients (11.0%, 95% CI 6.4%-17.2%). Although statistically insignificant, the incidence of diagnostic errors was lower in cases where the final diagnosis was included in the differential diagnosis list from the AI system than in cases where the final diagnosis was not included in the list (7.2% vs 15.9%, P=.18).

Conclusions

The incidence of diagnostic errors among patients in the outpatient department of internal medicine who used an automated medical history–taking system that generates differential diagnosis lists seemed to be lower than the previously reported incidence of diagnostic errors. This result suggests that the implementation of an automated medical history–taking system that generates differential diagnosis lists could be beneficial for diagnostic safety in the outpatient department of internal medicine.

Diagnostic Errors Are Common in Acute Pediatric Respiratory Disease: A Prospective, Single-Blinded Multicenter Diagnostic Accuracy Study in Australian Emergency Departments

Background: Diagnostic errors are a global health priority and a common cause of preventable harm. There is limited data available for the prevalence of misdiagnosis in pediatric acute-care settings. Respiratory illnesses, which are particularly challenging to diagnose, are the most frequent reason for presentation to pediatric emergency departments. Objective: To evaluate the diagnostic accuracy of emergency department clinicians in diagnosing acute childhood respiratory diseases, as compared with expert panel consensus (reference standard). Methods: Prospective, multicenter, single-blinded, diagnostic accuracy study in two well-resourced pediatric emergency departments in a large Australian city. Between September 2016 and August 2018, a convenience sample of children aged 29 days to 12 years who presented with respiratory symptoms was enrolled. The emergency department discharge diagnoses were reported by clinicians based upon standard clinical diagnostic definitions. These diagnoses were compared against consensus diagnoses given by an expert panel of pediatric specialists using standardized disease definitions after they reviewed all medical records. Results: For 620 participants, the sensitivity and specificity (%, [95% CI]) of the emergency department compared with the expert panel diagnoses were generally poor: isolated upper respiratory tract disease (64.9 [54.6, 74.4], 91.0 [88.2, 93.3]), croup (76.8 [66.2, 85.4], 97.9 [96.2, 98.9]), lower respiratory tract disease (86.6 [83.1, 89.6], 92.9 [87.6, 96.4]), bronchiolitis (66.9 [58.6, 74.5], 94.3 [80.8, 99.3]), asthma/reactive airway disease (91.0 [85.8, 94.8], 93.0 [90.1, 95.3]), clinical pneumonia (63·9 [50.6, 75·8], 95·0 [92·8, 96·7]), focal (consolidative) pneumonia (54·8 [38·7, 70·2], 86.2 [79.3, 91.5]). Only 59% of chest x-rays with consolidation were correctly identified. Between 6.9 and 14.5% of children were inappropriately prescribed based on their eventual diagnosis. Conclusion: In well-resourced emergency departments, we have identified a previously unrecognized high diagnostic error rate for acute childhood respiratory disorders, particularly in pneumonia and bronchiolitis. These errors lead to the potential of avoidable harm and the administration of inappropriate treatment.

Automated identification of diagnostic labelling errors in medicine

Objectives

Identification of diagnostic error is complex and mostly relies on expert ratings, a severely limited procedure. We developed a system that allows to automatically identify diagnostic labelling error from diagnoses coded according to the international classification of diseases (ICD), often available as routine health care data.

Methods

The system developed (index test) was validated against rater based classifications taken from three previous studies of diagnostic labeling error (reference standard). The system compares pairs of diagnoses through calculation of their distance within the ICD taxonomy. Calculation is based on four different algorithms. To assess the concordance between index test and reference standard, we calculated the area under the receiver operating characteristics curve (AUROC) and corresponding confidence intervals. Analysis were conducted overall and separately per algorithm and type of available dataset.

Results

Diagnoses of 1,127 cases were analyzed. Raters previously classified 24.58% of cases as diagnostic labelling errors (ranging from 12.3 to 87.2% in the three datasets). AUROC ranged between 0.821 and 0.837 overall, depending on the algorithm used to calculate the index test (95% CIs ranging from 0.8 to 0.86). Analyzed per type of dataset separately, the highest AUROC was 0.924 (95% CI 0.887–0.962).

Conclusions

The trigger system to automatically identify diagnostic labeling error from routine health care data performs excellent, and is unaffected by the reference standards’ limitations. It is however only applicable to cases with pairs of diagnoses, of which one must be more accurate or otherwise superior than the other, reflecting a prevalent definition of a diagnostic labeling error.

Validation of an electronic trigger to measure missed diagnosis of stroke in emergency departments

OBJECTIVE

Diagnostic errors are major contributors to preventable patient harm. We validated the use of an electronic health record (EHR)-based trigger (e-trigger) to measure missed opportunities in stroke diagnosis in emergency departments (EDs).

METHODS

Using two frameworks, the Safer Dx Trigger Tools Framework and the Symptom-disease Pair Analysis of Diagnostic Error Framework, we applied a symptom-disease pair-based e-trigger to identify patients hospitalized for stroke who, in the preceding 30 days, were discharged from the ED with benign headache or dizziness diagnoses. The algorithm was applied to Veteran Affairs National Corporate Data Warehouse on patients seen between 1/1/2016 and 12/31/2017. Trained reviewers evaluated medical records for presence/absence of missed opportunities in stroke diagnosis and stroke-related red-flags, risk factors, neurological examination, and clinical interventions. Reviewers also estimated quality of clinical documentation at the index ED visit.

RESULTS

We applied the e-trigger to 7,752,326 unique patients and identified 46,931 stroke-related admissions, of which 398 records were flagged as trigger-positive and reviewed. Of these, 124 had missed opportunities (positive predictive value for "missed" = 31.2%), 93 (23.4%) had no missed opportunity (non-missed), 162 (40.7%) were miscoded, and 19 (4.7%) were inconclusive. Reviewer agreement was high (87.3%, Cohen's kappa = 0.81). Compared to the non-missed group, the missed group had more stroke risk factors (mean 3.2 vs 2.6), red flags (mean 0.5 vs 0.2), and a higher rate of inadequate documentation (66.9% vs 28.0%).

CONCLUSION

In a large national EHR repository, a symptom-disease pair-based e-trigger identified missed diagnoses of stroke with a modest positive predictive value, underscoring the need for chart review validation procedures to identify diagnostic errors in large data sets.

Development of a rubric for assessing delayed diagnosis of appendicitis, diabetic ketoacidosis and sepsis

OBJECTIVES

Using case review to determine whether a patient experienced a delayed diagnosis is challenging. Measurement would be more accurate if case reviewers had access to multi-expert consensus on grading the likelihood of delayed diagnosis. Our objective was to use expert consensus to create a guide for objectively grading the likelihood of delayed diagnosis of appendicitis, new-onset diabetic ketoacidosis (DKA), and sepsis.

METHODS

Case vignettes were constructed for each condition. In each vignette, a patient has the condition and had a previous emergency department (ED) visit within 7 days. Condition-specific multi-specialty expert Delphi panels reviewed the case vignettes and graded the likelihood of a delayed diagnosis on a five-point scale. Delayed diagnosis was defined as the condition being present during the previous ED visit. Consensus was defined as ≥75% agreement. In each Delphi round, panelists were given the scores from the previous round and asked to rescore. A case scoring guide was created from the consensus scores.

RESULTS

Eighteen expert panelists participated. Consensus was achieved within three Delphi rounds for all appendicitis and sepsis vignettes. We reached consensus on 23/30 (77%) DKA vignettes. A case review guide was created from the consensus scores.

CONCLUSIONS

Multi-specialty expert reviewers can agree on the likelihood of a delayed diagnosis for cases of appendicitis and sepsis, and for most cases of DKA. We created a guide that can be used by researchers and quality improvement specialists to allow for objective case review to determine when delayed diagnoses have occurred for appendicitis, DKA, and sepsis.

Use of patient complaints to identify diagnosis-related safety concerns: a mixed-method evaluation

Background

Patient complaints are associated with adverse events and malpractice claims but underused in patient safety improvement.

Objective

To systematically evaluate the use of patient complaint data to identify safety concerns related to diagnosis as an initial step to using this information to facilitate learning and improvement.

Methods

We reviewed patient complaints submitted to Geisinger, a large healthcare organisation in the USA, from August to December 2017 (cohort 1) and January to June 2018 (cohort 2). We selected complaints more likely to be associated with diagnostic concerns in Geisinger’s existing complaint taxonomy. Investigators reviewed all complaint summaries and identified cases as ‘concerning’ for diagnostic error using the National Academy of Medicine’s definition of diagnostic error. For all ‘concerning’ cases, a clinician-reviewer evaluated the associated investigation report and the patient’s medical record to identify any missed opportunities in making a correct or timely diagnosis. In cohort 2, we selected a 10% sample of ‘concerning’ cases to test this smaller pragmatic sample as a proof of concept for future organisational monitoring.

Results

In cohort 1, we reviewed 1865 complaint summaries and identified 177 (9.5%) concerning reports. Review and analysis identified 39 diagnostic errors. Most were categorised as ‘Clinical Care issues’ (27, 69.2%), defined as concerns/questions related to the care that is provided by clinicians in any setting. In cohort 2, we reviewed 2423 patient complaint summaries and identified 310 (12.8%) concerning reports. The 10% sample (n=31 cases) contained five diagnostic errors. Qualitative analysis of cohort 1 cases identified concerns about return visits for persistent and/or worsening symptoms, interpersonal issues and diagnostic testing.

Conclusions

Analysis of patient complaint data and corresponding medical record review identifies patterns of failures in the diagnostic process reported by patients and families. Health systems could systematically analyse available data on patient complaints to monitor diagnostic safety concerns and identify opportunities for learning and improvement.

Increasing Physician Reporting of Diagnostic Learning Opportunities

BACKGROUND

An estimated 10% of Americans experience a diagnostic error annually, yet little is known about pediatric diagnostic errors. Physician reporting is a promising method for identifying diagnostic errors. However, our pediatric hospital medicine (PHM) division had only 1 diagnostic-related safety report in the preceding 4 years. We aimed to improve attending physician reporting of suspected diagnostic errors from 0 to 2 per 100 PHM patient admissions within 6 months.

METHODS

Our improvement team used the Model for Improvement, targeting the PHM service. To promote a safe reporting culture, we used the term diagnostic learning opportunity (DLO) rather than diagnostic error, defined as a "potential opportunity to make a better or more timely diagnosis." We developed an electronic reporting form and encouraged its use through reminders, scheduled reflection time, and monthly progress reports. The outcome measure, the number of DLO reports per 100 patient admissions, was tracked on an annotated control chart to assess the effect of our interventions over time. We evaluated DLOs using a formal 2-reviewer process.

RESULTS

Over the course of 13 weeks, there was an increase in the number of reports filed from 0 to 1.6 per 100 patient admissions, which met special cause variation, and was subsequently sustained. Most events (66%) were true diagnostic errors and were found to be multifactorial after formal review.

CONCLUSIONS

We used quality improvement methodology, focusing on psychological safety, to increase physician reporting of DLOs. This growing data set has generated nuanced learnings that will guide future improvement work.

Machine learning in primary care: potential to improve public health

It is estimated that missed opportunities for diagnosis occur in 1 in 20 primary care appointments. This is not only detrimental to individual patients, but also to the healthcare system as health outcomes are affected and healthcare expenditure inevitably increases. There are many potential solutions to limit the number of missed opportunities for diagnosis and management, one of which is through the use of artificial intelligence. Artificial intelligence and machine learning research and capabilities have exponentially grown in the past decades, with their applications in medicine showing great promise. As such, this review aims to discuss the possible uses of machine learning in primary care to maximise the quality of care provided.

Identifying trigger concepts to screen emergency department visits for diagnostic errors

OBJECTIVES

The diagnostic process is a vital component of safe and effective emergency department (ED) care. There are no standardized methods for identifying or reliably monitoring diagnostic errors in the ED, impeding efforts to enhance diagnostic safety. We sought to identify trigger concepts to screen ED records for diagnostic errors and describe how they can be used as a measurement strategy to identify and reduce preventable diagnostic harm.

METHODS

We conducted a literature review and surveyed ED directors to compile a list of potential electronic health record (EHR) trigger (e-triggers) and non-EHR based concepts. We convened a multidisciplinary expert panel to build consensus on trigger concepts to identify and reduce preventable diagnostic harm in the ED.

RESULTS

Six e-trigger and five non-EHR based concepts were selected by the expert panel. E-trigger concepts included: unscheduled ED return to ED resulting in hospital admission, death following ED visit, care escalation, high-risk conditions based on symptom-disease dyads, return visits with new diagnostic/therapeutic interventions, and change of treating service after admission. Non-EHR based signals included: cases from mortality/morbidity conferences, risk management/safety office referrals, ED medical director case referrals, patient complaints, and radiology/laboratory misreads and callbacks. The panel suggested further refinements to aid future research in defining diagnostic error epidemiology in ED settings.

CONCLUSIONS

We identified a set of e-trigger concepts and non-EHR based signals that could be developed further to screen ED visits for diagnostic safety events. With additional evaluation, trigger-based methods can be used as tools to monitor and improve ED diagnostic performance.

Multimorbidity and patient-reported diagnostic errors in the primary care setting: multicentre cross-sectional study in Japan

OBJECTIVES

There is lack of evidence for the association between multimorbidity and diagnostic errors. Information on diagnostic errors from patients' perspectives is crucial to improve the diagnostic process. In this study, we aimed to investigate patient-reported diagnostic errors and to examine the relationship between multimorbidity and patient-reported diagnostic errors in the primary care setting.

DESIGN

Multicentre cross-sectional study.

SETTING

A primary care practice-based research network in Japan (25 primary care facilities).

PARTICIPANTS

Adult outpatients filled out a standardised questionnaire.

PRIMARY OUTCOME MEASURE

Patient-reported diagnostic errors.

RESULTS

Data collected from 1474 primary care outpatients were analysed. The number of participants who reported diagnostic errors was 57 (3.9%). Most of the missed diagnoses were common conditions in primary care, such as cancer, dermatitis and hypertension. After adjustment for possible confounders and clustering within facilities, multimorbidity was positively associated with patient-reported diagnostic errors (adjusted OR=1.83, 95% CI 1.01 to 3.31). The results of the sensitivity analysis were consistent with those of the primary analysis.

CONCLUSIONS

The present study showed a lower proportion of patients reporting experiences of diagnostic errors in primary care than those reported in previous studies in other countries. However, patients with multimorbidity are more likely to report diagnostic errors in primary care; thus, further research is necessary to improve the diagnostic process for patients with multimorbidity.

Developing Health Care Organizations That Pursue Learning and Exploration of Diagnostic Excellence: An Action Plan

Reducing errors in diagnosis is the next big challenge for patient safety. Diagnostic safety improvement efforts should become a priority for health care organizations, payers, and accrediting bodies; however, external incentives, policies, and practical guidance to develop these efforts are largely absent. In this Perspective, the authors highlight ways in which health care organizations can pursue learning and exploration of diagnostic excellence (LEDE). Building on current evidence and their recent experiences in developing such a learning organization at Geisinger, the authors propose a 5-point action plan and corresponding policy levers to support development of LEDE organizations. These recommendations, which are applicable to many health care organizations, include (1) implementing a virtual hub to coordinate organizational activities for improving diagnosis, such as identifying risks and prioritizing interventions that cross intra-institutional silos while promoting a culture of learning and safety; (2) participating in novel scientific initiatives to generate and translate evidence, given the rapidly evolving "basic science" of diagnostic excellence; (3) avoiding the "tyranny of metrics" by focusing on measurement for improvement rather than using measures to reward or punish; (4) engaging clinicians in activities for improving diagnosis and framing missed opportunities positively as learning opportunities rather than negatively as errors; and (5) developing an accountable culture of engaging and learning from patients, who are often underexplored sources of information. The authors also outline specific policy actions to support organizations in implementing these recommendations. They suggest this action plan can stimulate scientific, practice, and policy progress needed for achieving diagnostic excellence and reducing preventable patient harm.

Operational measurement of diagnostic safety: state of the science

Reducing the incidence of diagnostic errors is increasingly a priority for government, professional, and philanthropic organizations. Several obstacles to measurement of diagnostic safety have hampered progress toward this goal. Although a coordinated national strategy to measure diagnostic safety remains an aspirational goal, recent research has yielded practical guidance for healthcare organizations to start using measurement to enhance diagnostic safety. This paper, concurrently published as an Issue Brief by the Agency for Healthcare Research and Quality, issues a "call to action" for healthcare organizations to begin measurement efforts using data sources currently available to them. Our aims are to outline the state of the science and provide practical recommendations for organizations to start identifying and learning from diagnostic errors. Whether by strategically leveraging current resources or building additional capacity for data gathering, nearly all organizations can begin their journeys to measure and reduce preventable diagnostic harm.

Assessing diagnostic error in cerebral venous thrombosis via detailed chart review

Background Diagnostic error in cerebral venous thrombosis (CVT) has been understudied despite the harm associated with misdiagnosis of other cerebrovascular diseases as well as the known challenges of evaluating non-specific neurological symptoms in clinical practice. Methods We conducted a retrospective cohort study of CVT patients hospitalized at a single center. Two independent reviewers used a medical record review tool, the Safer Dx Instrument, to identify diagnostic errors. Demographic and clinical factors were abstracted. We compared subjects with and without a diagnostic error using the t-test for continuous variables and the chi-square (χ2) test or Fisher's exact test for categorical variables; an alpha of 0.05 was the cutoff for significance. Results A total of 72 CVT patients initially met study inclusion criteria; 19 were excluded due to incomplete medical records. Of the 53 patients included in the final analysis, the mean age was 48 years and 32 (60.4%) were women. Diagnostic error occurred in 11 cases [20.8%; 95% confidence interval (CI) 11.8-33.6%]. Subjects with diagnostic errors were younger (42 vs. 49 years, p = 0.13), more often women (81.8% vs. 54.8%, p = 0.17), and were significantly more likely to have a past medical history of a headache disorder prior to the index CVT visit (7.1% vs. 36.4%, p = 0.03). Conclusions Nearly one in five patients with complete medical records experienced a diagnostic error. Prior history of headache was the only evaluated clinical factor that was more common among those with an error in diagnosis. Future work on distinguishing primary from secondary headaches to improve diagnostic accuracy in acute neurological disease is warranted.

Measures to Improve Diagnostic Safety in Clinical Practice

Timely and accurate diagnosis is foundational to good clinical practice and an essential first step to achieving optimal patient outcomes. However, a recent Institute of Medicine report concluded that most of us will experience at least one diagnostic error in our lifetime. The report argues for efforts to improve the reliability of the diagnostic process through better measurement of diagnostic performance. The diagnostic process is a dynamic team-based activity that involves uncertainty, plays out over time, and requires effective communication and collaboration among multiple clinicians, diagnostic services, and the patient. Thus, it poses special challenges for measurement. In this paper, we discuss how the need to develop measures to improve diagnostic performance could move forward at a time when the scientific foundation needed to inform measurement is still evolving. We highlight challenges and opportunities for developing potential measures of "diagnostic safety" related to clinical diagnostic errors and associated preventable diagnostic harm. In doing so, we propose a starter set of measurement concepts for initial consideration that seem reasonably related to diagnostic safety and call for these to be studied and further refined. This would enable safe diagnosis to become an organizational priority and facilitate quality improvement. Health-care systems should consider measurement and evaluation of diagnostic performance as essential to timely and accurate diagnosis and to the reduction of preventable diagnostic harm.

Isolating red flags to enhance diagnosis (I-RED): An experimental vignette study

OBJECTIVE

To investigate effects of a cognitive intervention based on isolation of red flags (I-RED) on diagnostic accuracy of 'do-not-miss diagnoses.'

DESIGN

A 2 × 2 randomized case vignette-based experiment with manipulation of I-RED strategy between subjects and case complexity within subjects.

SETTING

Two university-based residency programs.

PARTICIPANTS

One-hundred and nine pediatric residents from all levels of training.

INTERVENTIONS

Participants were randomly assigned to the I-RED vs. control group, and within each group, they were further randomized to the order in which they saw simple and complex cases. The I-RED strategy involved an instruction to look for a constellation of symptoms, signs, clinical data or circumstances that should heighten suspicion for a serious condition.

MAIN OUTCOME MEASURES

Primary outcome was diagnostic accuracy, scored as 1 if any of the three differentials given by participants included the correct diagnosis, and 0 if not. We analyzed effects of I-RED strategy on diagnostic accuracy using logistic regression.

RESULTS

I-RED strategy did not yield statistically higher diagnostic accuracy compared to controls (62 vs. 48%, respectively; odd ratio = 2.07 [95% confidence interval, 0.78-5.5], P = 0.14) although participants reported higher decision confidence compared to controls (7.00 vs. 5.77 on a scale of 1 to 10, P < 0.02) in simple but not complex cases. I-RED strategy significantly shortened time to decision (460 vs. 657 s, P < 0.001) and increased the number of red flags generated (3.04 vs. 2.09, P < 0.001).

CONCLUSIONS

A cognitive strategy of prompting red flag isolation prior to differential diagnosis did not improve diagnostic accuracy of 'do-not-miss diagnoses.' Given the paucity of evidence-based solutions to reduce diagnostic error and the intervention's potential effect on confidence, findings warrant additional exploration.

Recommendations for using the Revised Safer Dx Instrument to help measure and improve diagnostic safety

The medical record continues to be one of the most useful and accessible sources of information to examine the diagnostic process. However, medical record review studies of diagnostic errors have often used subjective judgments and found low inter-rater agreement among reviewers when determining the presence or absence of diagnostic error. In our previous work, we developed a structured data-collection instrument, called the Safer Dx Instrument, consisting of objective criteria to improve the accuracy of assessing diagnostic errors in primary care. This paper proposes recommendations on how clinicians and health care organizations could use the Revised Safer Dx Instrument in identifying and understanding missed opportunities to make correct and timely diagnoses. The instrument revisions addressed both methodological and implementation issues identified during initial use and included refinements to the instrument to allow broader application across all health care settings. In addition to leveraging knowledge from piloting the instrument in several health care settings, we gained insights from multiple researchers who had used the instrument in studies involving emergency care, inpatient care and intensive care unit settings. This allowed us to enhance and extend the scope of this previously validated data collection instrument. In this paper, we describe the refinement process and provide recommendations for application and use of the Revised Safer Dx Instrument across a broad range of health care settings. The instrument can help users identify potential diagnostic errors in a standardized way for further analysis and safety improvement efforts as well as provide data for clinician feedback and reflection. With wider adoption and use by clinicians and health systems, the Revised Safer Dx Instrument could help propel the science of measuring and reducing diagnostic errors forward.

Payment Innovations To Improve Diagnostic Accuracy And Reduce Diagnostic Error

Diagnostic accuracy is essential for treatment decisions but is largely unaccounted for by payers, including in fee-for-service Medicare and proposed Alternative Payment Models (APMs). We discuss three payment-related approaches to reducing diagnostic error. First, coding changes in the Medicare Physician Fee Schedule could facilitate the more effective use of teamwork and information technology in the diagnostic process and better support the cognitive work and time commitment that physicians make in the quest for diagnostic accuracy, especially in difficult or uncertain cases. Second, new APMs could be developed to focus on improving diagnostic accuracy in challenging cases and make available support resources for diagnosis, including condition-specific centers of diagnostic expertise or general diagnostic centers of excellence that provide second (or even third) opinions. Performing quality improvement activities that promote safer diagnosis should be a part of the accountability of APM recipients. Third, the accuracy of diagnoses that trigger APM payments and establish payment amounts should be confirmed by APM recipients. Implementation of these multipronged approaches can make current payment models more accountable for addressing diagnostic error and position diagnostic performance as a critical component of quality-based payment.

Electronic Health Record-Related Events in Medical Malpractice Claims

BACKGROUND

There is widespread agreement that the full potential of health information technology (health IT) has not yet been realized and of particular concern are the examples of unintended consequences of health IT that detract from the safety of health care or from the use of health IT itself. The goal of this project was to obtain additional information on these health IT-related problems, using a mixed methods (qualitative and quantitative) analysis of electronic health record-related harm in cases submitted to a large database of malpractice suits and claims.

METHODS

Cases submitted to the CRICO claims database and coded during 2012 and 2013 were analyzed. A total of 248 cases (<1%) involving health IT were identified and coded using a proprietary taxonomy that identifies user- and system-related sociotechnical factors. Ambulatory care accounted for most of the cases (146 cases). Cases were most typically filed as a result of an error involving medications (31%), diagnosis (28%), or a complication of treatment (31%). More than 80% of cases involved moderate or severe harm, although lethal cases were less likely in cases from ambulatory settings. Etiologic factors spanned all of the sociotechnical dimensions, and many recurring patterns of error were identified.

CONCLUSIONS

Adverse events associated with health IT vulnerabilities can cause extensive harm and are encountered across the continuum of health care settings and sociotechnical factors. The recurring patterns provide valuable lessons that both practicing clinicians and health IT developers could use to reduce the risk of harm in the future. The likelihood of harm seems to relate more to a patient's particular situation than to any one class of error.

"Closing the loop": a mixed-methods study about resident learning from outcome feedback after patient handoffs

Background Learning patient outcomes is recognized as crucial for ongoing refinement of clinical decision-making, but is often difficult in fragmented care with frequent handoffs. Data on resident habits of seeking outcome feedback after handoffs are lacking. Methods We performed a mixed-methods study including (1) an analysis of chart re-access rates after handoffs performed using access logs of the electronic health record (EHR); and (2) a web-based survey sent to internal medicine (IM) and emergency medicine (EM) residents about their habits of and barriers to learning the outcomes of patients after they have handed them off to other teams. Results Residents on ward rotations were often able to re-access charts of patients after handoffs, but those on EM or night admitting rotations did so <5% of the time. Among residents surveyed, only a minority stated that they frequently find out the outcomes of patients they have handed off, although learning outcomes was important to both their education and job satisfaction. Most were not satisfied with current systems of learning outcomes of patients after handoffs, citing too little time and lack of reliable patient tracking systems as the main barriers. Conclusions Despite perceived importance of learning outcomes after handoffs, residents cite difficulty with obtaining such information. Systematically providing feedback on patient outcomes would meet a recognized need among physicians in training.

Application of electronic trigger tools to identify targets for improving diagnostic safety

Progress in reducing diagnostic errors remains slow partly due to poorly defined methods to identify errors, high-risk situations, and adverse events. Electronic trigger (e-trigger) tools, which mine vast amounts of patient data to identify signals indicative of a likely error or adverse event, offer a promising method to efficiently identify errors. The increasing amounts of longitudinal electronic data and maturing data warehousing techniques and infrastructure offer an unprecedented opportunity to implement new types of e-trigger tools that use algorithms to identify risks and events related to the diagnostic process. We present a knowledge discovery framework, the Safer Dx Trigger Tools Framework, that enables health systems to develop and implement e-trigger tools to identify and measure diagnostic errors using comprehensive electronic health record (EHR) data. Safer Dx e-trigger tools detect potential diagnostic events, allowing health systems to monitor event rates, study contributory factors and identify targets for improving diagnostic safety. In addition to promoting organisational learning, some e-triggers can monitor data prospectively and help identify patients at high-risk for a future adverse event, enabling clinicians, patients or safety personnel to take preventive actions proactively. Successful application of electronic algorithms requires health systems to invest in clinical informaticists, information technology professionals, patient safety professionals and clinicians, all of who work closely together to overcome development and implementation challenges. We outline key future research, including advances in natural language processing and machine learning, needed to improve effectiveness of e-triggers. Integrating diagnostic safety e-triggers in institutional patient safety strategies can accelerate progress in reducing preventable harm from diagnostic errors.

Diagnostic Decision-Making in the Emergency Department

Emergency medicine requires diagnosing unfamiliar patients with undifferentiated acute presentations. This requires hypothesis generation and questioning, examination, and testing. Balancing patient load, care across the severity spectrum, and frequent interruptions create time pressures that predispose humans to fast thinking or cognitive shortcuts, including cognitive biases. Diagnostic error is the failure to establish an accurate and timely explanation of the problem or communicate that to the patient, often contributing to physical, emotional, or financial harm. Methods for monitoring diagnostic error in the emergency department are needed to establish frequency and serve as a foundation for future interventions.