Operational measurement of diagnostic safety: state of the science

Evaluation of Measure Dx, a Resource to Accelerate Diagnostic Safety Learning and Improvement

BACKGROUND

Several strategies have been developed to detect diagnostic errors for organizational learning and improvement. However, few health care organizations (HCOs) have integrated these strategies into routine operations. To address this gap, the Agency for Healthcare Research and Quality released "Measure Dx: A Resource To Identify, Analyze, and Learn From Diagnostic Safety Events" in 2022.

OBJECTIVE

We conducted an evaluation of Measure Dx to measure feasibility of implementation and effects on short-term and intermediate outcomes related to diagnostic safety.

DESIGN

Prospective observational study.

PARTICIPANTS

Teams from 11 HCOs, primarily academic medical centers.

INTERVENTIONS

Participants were asked to use Measure Dx over approximately 6 months and attend monthly virtual learning collaborative sessions to share and discuss approaches to measuring diagnostic safety.

MAIN MEASURES

Descriptive outcomes were gathered at the HCO level and included uptake of different case-finding strategies and the number of cases reviewed and confirmed to have diagnostic safety improvement opportunities. We collected information on organizational practices related to diagnostic safety at each HCO at baseline and at the conclusion of the project.

KEY RESULTS

The 11 HCOs completed all requirements for the evaluation. Each of the four diagnostic safety case finding strategies outlined in Measure Dx were used by at least three HCOs. Across the cohort, participants reviewed 703 cases using a standardized data collection instrument. Of those cases, 224 (31.8%) were identified as diagnostic safety events with improvement opportunities. Unexpectedly, self-ratings on the checklist assessment declined for several organizations.

CONCLUSIONS

Use of Measure Dx can help accelerate implementation of systematic approaches to diagnostic error measurement and learning across a variety of HCOs, while potentially enabling HCOs to identify opportunities to improve diagnostic safety practices.

Performance evaluation of ChatGPT in detecting diagnostic errors and their contributing factors: an analysis of 545 case reports of diagnostic errors

BACKGROUND

Manual chart review using validated assessment tools is a standardised methodology for detecting diagnostic errors. However, this requires considerable human resources and time. ChatGPT, a recently developed artificial intelligence chatbot based on a large language model, can effectively classify text based on suitable prompts. Therefore, ChatGPT can assist manual chart reviews in detecting diagnostic errors.

OBJECTIVE

This study aimed to clarify whether ChatGPT could correctly detect diagnostic errors and possible factors contributing to them based on case presentations.

METHODS

We analysed 545 published case reports that included diagnostic errors. We imputed the texts of case presentations and the final diagnoses with some original prompts into ChatGPT (GPT-4) to generate responses, including the judgement of diagnostic errors and contributing factors of diagnostic errors. Factors contributing to diagnostic errors were coded according to the following three taxonomies: Diagnosis Error Evaluation and Research (DEER), Reliable Diagnosis Challenges (RDC) and Generic Diagnostic Pitfalls (GDP). The responses on the contributing factors from ChatGPT were compared with those from physicians.

RESULTS

ChatGPT correctly detected diagnostic errors in 519/545 cases (95%) and coded statistically larger numbers of factors contributing to diagnostic errors per case than physicians: DEER (median 5 vs 1, p<0.001), RDC (median 4 vs 2, p<0.001) and GDP (median 4 vs 1, p<0.001). The most important contributing factors of diagnostic errors coded by ChatGPT were 'failure/delay in considering the diagnosis' (315, 57.8%) in DEER, 'atypical presentation' (365, 67.0%) in RDC, and 'atypical presentation' (264, 48.4%) in GDP.

CONCLUSION

ChatGPT accurately detects diagnostic errors from case presentations. ChatGPT may be more sensitive than manual reviewing in detecting factors contributing to diagnostic errors, especially for 'atypical presentation'.

Partnership as a Pathway to Diagnostic Excellence: The Challenges and Successes of Implementing the Safer Dx Learning Lab

BACKGROUND

Learning health system (LHS) approaches could potentially help health care organizations (HCOs) identify and address diagnostic errors. However, few such programs exist, and their implementation is poorly understood.

METHODS

The authors conducted a qualitative evaluation of the Safer Dx Learning Lab, a partnership between a health system and a research team, to identify and learn from diagnostic errors and improve diagnostic safety at an organizational level. The research team conducted virtual interviews to solicit participant feedback regarding experiences with the lab, focusing specifically on implementation and sustainment issues.

RESULTS

Interviews of 25 members associated with the lab identified the following successes: learning and professional growth, improved workflow related to streamlining the process of reporting error cases, and a psychologically safe culture for identifying and reporting diagnostic errors. However, multiple barriers also emerged: competing priorities between clinical responsibilities and research, time-management issues related to a lack of protected time, and inadequate guidance to disseminate findings. Lessons learned included understanding the importance of obtaining buy-in from leadership and interested stakeholders, creating a psychologically safe environment for reporting cases, and the need for more protected time for clinicians to review and learn from cases.

CONCLUSION

Findings suggest that a learning health systems approach using partnerships between researchers and a health system affected organizational culture by prioritizing learning from diagnostic errors and encouraging clinicians to be more open to reporting. The study findings can help organizations overcome barriers to engage clinicians and inform future implementation and sustainment of similar initiatives.

Effect of contextual factors on the prevalence of diagnostic errors among patients managed by physicians of the same specialty: a single-centre retrospective observational study

BACKGROUND

There has been growing recognition that contextual factors influence the physician's cognitive processes. However, given that cognitive processes may depend on the physicians' specialties, the effects of contextual factors on diagnostic errors reported in previous studies could be confounded by difference in physicians.

OBJECTIVE

This study aimed to clarify whether contextual factors such as location and consultation type affect diagnostic accuracy.

METHODS

We reviewed the medical records of 1992 consecutive outpatients consulted by physicians from the Department of Diagnostic and Generalist Medicine in a university hospital between 1 January and 31 December 2019. Diagnostic processes were assessed using the Revised Safer Dx Instrument. Patients were categorised into three groups according to contextual factors (location and consultation type): (1) referred patients with scheduled visit to the outpatient department; (2) patients with urgent visit to the outpatient department; and (3) patients with emergency visit to the emergency room. The effect of the contextual factors on the prevalence of diagnostic errors was investigated using logistic regression analysis.

RESULTS

Diagnostic errors were observed in 12 of 534 referred patients with scheduled visit to the outpatient department (2.2%), 3 of 599 patients with urgent visit to the outpatient department (0.5%) and 13 of 859 patients with emergency visit to the emergency room (1.5%). Multivariable logistic regression analysis showed a significantly higher prevalence of diagnostic errors in referred patients with scheduled visit to the outpatient department than in patients with urgent visit to the outpatient department (OR 4.08, p=0.03), but no difference between patients with emergency and urgent visit to the emergency room and outpatient department, respectively.

CONCLUSION

Contextual factors such as consultation type may affect diagnostic errors; however, since the differences in the prevalence of diagnostic errors were small, the effect of contextual factors on diagnostic accuracy may be small in physicians working in different care settings.

An electronic trigger to detect telemedicine-related diagnostic errors

INTRODUCTION

The COVID-19 pandemic advanced the use of telehealth-facilitated care. However, little is known about how to measure safety of clinical diagnosis made through telehealth-facilitated primary care.

METHODS

We used the seven-step Safer Dx Trigger Tool framework to develop an electronic trigger (e-trigger) tool to identify potential missed opportunities for more timely diagnosis during primary care telehealth visits at a large Department of Veterans Affairs facility. We then applied the e-trigger algorithm to electronic health record data related to primary care visits during a 1-year period (1 April 2020-31 March 2021). The algorithm identified patients with unexpected visits within 10 days of an index telemedicine visit and classified such records as e-trigger positive. We then validated the e-trigger's ability to detect missed opportunities in diagnosis using chart reviews based on a structured data collection instrument (the Revised Safer Dx instrument).

RESULTS

We identified 128,761 telehealth visits (32,459 unique patients), of which 434 visits led to subsequent unplanned emergency department (ED), hospital, or primary care visits within 10 days of the index visit. Of these, 116 were excluded for clinical reasons (trauma, injury, or childbirth), leaving 318 visits (240 unique patients) needing further evaluation. From these, 100 records were randomly selected for review, of which four were falsely flagged due to invalid data (visits by non-providers or those incorrectly flagged as completed telehealth visits). Eleven patients had a missed opportunity in diagnosis, yielding a positive predictive value of 11%.

DISCUSSION

Electronic triggers that identify missed opportunities for additional evaluation could help advance the understanding of safety of clinical diagnosis made in telehealth-enabled care. Better measurement can help determine which patients can safely be cared for via telemedicine versus traditional in-person visits.

Guideline concordance for timely chest imaging after new presentations of dyspnoea or haemoptysis in primary care: a retrospective cohort study

BACKGROUND

Guidelines recommend urgent chest X-ray for newly presenting dyspnoea or haemoptysis but there is little evidence about their implementation.

METHODS

We analysed linked primary care and hospital imaging data for patients aged 30+ years newly presenting with dyspnoea or haemoptysis in primary care during April 2012 to March 2017. We examined guideline-concordant management, defined as General Practitioner-ordered chest X-ray/CT carried out within 2 weeks of symptomatic presentation, and variation by sociodemographic characteristic and relevant medical history using logistic regression. Additionally, among patients diagnosed with cancer we described time to diagnosis, diagnostic route and stage at diagnosis by guideline-concordant status.

RESULTS

In total, 22 560/162 161 (13.9%) patients with dyspnoea and 4022/8120 (49.5%) patients with haemoptysis received guideline-concordant imaging within the recommended 2-week period. Patients with recent chest imaging pre-presentation were much less likely to receive imaging (adjusted OR 0.16, 95% CI 0.14-0.18 for dyspnoea, and adjusted OR 0.09, 95% CI 0.06-0.11 for haemoptysis). History of chronic obstructive pulmonary disease/asthma was also associated with lower odds of guideline concordance (dyspnoea: OR 0.234, 95% CI 0.225-0.242 and haemoptysis: 0.88, 0.79-0.97). Guideline-concordant imaging was lower among dyspnoea presenters with prior heart failure; current or ex-smokers; and those in more socioeconomically disadvantaged groups.The likelihood of lung cancer diagnosis within 12 months was greater among the guideline-concordant imaging group (dyspnoea: 1.1% vs 0.6%; haemoptysis: 3.5% vs 2.7%).

CONCLUSION

The likelihood of receiving urgent imaging concords with the risk of subsequent cancer diagnosis. Nevertheless, large proportions of dyspnoea and haemoptysis presenters do not receive prompt chest imaging despite being eligible, indicating opportunities for earlier lung cancer diagnosis.

The PRIDx framework to engage payers in reducing diagnostic errors in healthcare

OBJECTIVES

No framework currently exists to guide how payers and providers can collaboratively develop and implement incentives to improve diagnostic safety. We conducted a literature review and interviews with subject matter experts to develop a multi-component 'Payer Relationships for Improving Diagnoses (PRIDx)' framework, that could be used to engage payers in diagnostic safety efforts.

CONTENT

The PRIDx framework, 1) conceptualizes diagnostic safety links to care provision, 2) illustrates ways to promote payer and provider engagement in the design and adoption of accountability mechanisms, and 3) explicates the use of data analytics. Certain approaches suggested by PRIDx were refined by subject matter expert interviewee perspectives.

SUMMARY

The PRIDx framework can catalyze public and private payers to take specific actions to improve diagnostic safety.

OUTLOOK

Implementation of the PRIDx framework requires new types of partnerships, including external support from public and private payer organizations, and requires creation of strong provider incentives without undermining providers' sense of professionalism and autonomy. PRIDx could help facilitate collaborative payer-provider approaches to improve diagnostic safety and generate research concepts, policy ideas, and potential innovations for engaging payers in diagnostic safety improvement activities.

Developing electronic clinical quality measures to assess the cancer diagnostic process

OBJECTIVE

Measures of diagnostic performance in cancer are underdeveloped. Electronic clinical quality measures (eCQMs) to assess quality of cancer diagnosis could help quantify and improve diagnostic performance.

MATERIALS AND METHODS

We developed 2 eCQMs to assess diagnostic evaluation of red-flag clinical findings for colorectal (CRC; based on abnormal stool-based cancer screening tests or labs suggestive of iron deficiency anemia) and lung (abnormal chest imaging) cancer. The 2 eCQMs quantified rates of red-flag follow-up in CRC and lung cancer using electronic health record data repositories at 2 large healthcare systems. Each measure used clinical data to identify abnormal results, evidence of appropriate follow-up, and exclusions that signified follow-up was unnecessary. Clinicians reviewed 100 positive and 20 negative randomly selected records for each eCQM at each site to validate accuracy and categorized missed opportunities related to system, provider, or patient factors.

RESULTS

We implemented the CRC eCQM at both sites, while the lung cancer eCQM was only implemented at the VA due to lack of structured data indicating level of cancer suspicion on most chest imaging results at Geisinger. For the CRC eCQM, the rate of appropriate follow-up was 36.0% (26 746/74 314 patients) in the VA after removing clinical exclusions and 41.1% at Geisinger (1009/2461 patients; P < .001). Similarly, the rate of appropriate evaluation for lung cancer in the VA was 61.5% (25 166/40 924 patients). Reviewers most frequently attributed missed opportunities at both sites to provider factors (84 of 157).

CONCLUSIONS

We implemented 2 eCQMs to evaluate the diagnostic process in cancer at 2 large health systems. Health care organizations can use these eCQMs to monitor diagnostic performance related to cancer.

Improving the diagnostic process for patients with possible bladder and kidney cancer: a mixed-methods study to identify potential missed diagnostic opportunities

BACKGROUND

Patients with bladder and kidney cancer may experience diagnostic delays.

AIM

To identify patterns of suboptimal care and contributors of potential missed diagnostic opportunities (MDOs).

DESIGN AND SETTING

Prospective, mixed-methods study recruiting participants from nine general practices in Eastern England between June 2018 and October 2019.

METHOD

Patients with possible bladder and kidney cancer were identified using eligibility criteria based on National Institute for Health and Care Excellence (NICE) guidelines for suspected cancer. Primary care records were reviewed at recruitment and at 1 year for data on symptoms, tests, referrals, and diagnosis. Referral predictors were examined using logistic regression. Semi-structured interviews were undertaken with 15 patients to explore their experiences of the diagnostic process, and these were analysed thematically.

RESULTS

Participants (n = 940) were mostly female (n = 657, 69.9%), with a median age of 71 years (interquartile range 64-77 years). In total, 268 (28.5%) received a referral and 465 (48.5%) had a final diagnosis of urinary tract infection (UTI). There were 33 (3.5%) patients who were diagnosed with cancer, including prostate (n = 17), bladder (n = 7), and upper urothelial tract (n = 1) cancers. Among referred patients, those who had a final diagnosis of UTI had the longest time to referral (median 81.5 days). Only one-third of patients with recurrent UTIs were referred despite meeting NICE referral guidelines. Qualitative findings revealed barriers during the diagnostic process, including inadequate clinical examination, female patients given repeated antibiotics without clinical reviews, and suboptimal communication of test results to patients.

CONCLUSION

Older females with UTIs might be at increased risk of MDOs for cancer. Targeting barriers during the initial diagnostic assessment and follow-up might improve quality of diagnosis.

How does work environment relate to diagnostic quality? A prospective, mixed methods study in primary care

OBJECTIVES

The quest to measure and improve diagnosis has proven challenging; new approaches are needed to better understand and measure key elements of the diagnostic process in clinical encounters. The aim of this study was to develop a tool assessing key elements of the diagnostic assessment process and apply it to a series of diagnostic encounters examining clinical notes and encounters' recorded transcripts. Additionally, we aimed to correlate and contextualise these findings with measures of encounter time and physician burnout.

DESIGN

We audio-recorded encounters, reviewed their transcripts and associated them with their clinical notes and findings were correlated with concurrent Mini Z Worklife measures and physician burnout.

SETTING

Three primary urgent-care settings.

PARTICIPANTS

We conducted in-depth evaluations of 28 clinical encounters delivered by seven physicians.

RESULTS

Comparing encounter transcripts with clinical notes, in 24 of 28 (86%) there was high note/transcript concordance for the diagnostic elements on our tool. Reliably included elements were red flags (92% of notes/encounters), aetiologies (88%), likelihood/uncertainties (71%) and follow-up contingencies (71%), whereas psychosocial/contextual information (35%) and mentioning common pitfalls (7%) were often missing. In 22% of encounters, follow-up contingencies were in the note, but absent from the recorded encounter. There was a trend for higher burnout scores being associated with physicians less likely to address key diagnosis items, such as psychosocial history/context.

CONCLUSIONS

A new tool shows promise as a means of assessing key elements of diagnostic quality in clinical encounters. Work conditions and physician reactions appear to correlate with diagnostic behaviours. Future research should continue to assess relationships between time pressure and diagnostic quality.

A comprehensive framework to estimate the frequency, duration, and risk factors for diagnostic delays using bootstrapping-based simulation methods

BACKGROUND

The incidence of diagnostic delays is unknown for many diseases and specific healthcare settings. Many existing methods to identify diagnostic delays are resource intensive or difficult to apply to different diseases or settings. Administrative and other real-world data sources may offer the ability to better identify and study diagnostic delays for a range of diseases.

METHODS

We propose a comprehensive framework to estimate the frequency of missed diagnostic opportunities for a given disease using real-world longitudinal data sources. We provide a conceptual model of the disease-diagnostic, data-generating process. We then propose a bootstrapping method to estimate measures of the frequency of missed diagnostic opportunities and duration of delays. This approach identifies diagnostic opportunities based on signs and symptoms occurring prior to an initial diagnosis, while accounting for expected patterns of healthcare that may appear as coincidental symptoms. Three different bootstrapping algorithms are described along with estimation procedures to implement the resampling. Finally, we apply our approach to the diseases of tuberculosis, acute myocardial infarction, and stroke to estimate the frequency and duration of diagnostic delays for these diseases.

RESULTS

Using the IBM MarketScan Research databases from 2001 to 2017, we identified 2,073 cases of tuberculosis, 359,625 cases of AMI, and 367,768 cases of stroke. Depending on the simulation approach that was used, we estimated that 6.9-8.3% of patients with stroke, 16.0-21.3% of patients with AMI and 63.9-82.3% of patients with tuberculosis experienced a missed diagnostic opportunity. Similarly, we estimated that, on average, diagnostic delays lasted 6.7-7.6 days for stroke, 6.7-8.2 days for AMI, and 34.3-44.5 days for tuberculosis. Estimates for each of these measures was consistent with prior literature; however, specific estimates varied across the different simulation algorithms considered.

CONCLUSIONS

Our approach can be easily applied to study diagnostic delays using longitudinal administrative data sources. Moreover, this general approach can be customized to fit a range of diseases to account for specific clinical characteristics of a given disease. We summarize how the choice of simulation algorithm may impact the resulting estimates and provide guidance on the statistical considerations for applying our approach to future studies.

Effects of a computerised diagnostic decision support tool on diagnostic quality in emergency departments: study protocol of the DDx-BRO multicentre cluster randomised cross-over trial

INTRODUCTION

Computerised diagnostic decision support systems (CDDS) suggesting differential diagnoses to physicians aim to improve clinical reasoning and diagnostic quality. However, controlled clinical trials investigating their effectiveness and safety are absent and the consequences of its use in clinical practice are unknown. We aim to investigate the effect of CDDS use in the emergency department (ED) on diagnostic quality, workflow, resource consumption and patient outcomes.

METHODS AND ANALYSIS

This is a multicentre, outcome assessor and patient-blinded, cluster-randomised, multiperiod crossover superiority trial. A validated differential diagnosis generator will be implemented in four EDs and randomly allocated to a sequence of six alternating intervention and control periods. During intervention periods, the treating ED physician will be asked to consult the CDDS at least once during diagnostic workup. During control periods, physicians will not have access to the CDDS and diagnostic workup will follow usual clinical care. Key inclusion criteria will be patients' presentation to the ED with either fever, abdominal pain, syncope or a non-specific complaint as chief complaint. The primary outcome is a binary diagnostic quality risk score composed of presence of an unscheduled medical care after discharge, change in diagnosis or death during time of follow-up or an unexpected upscale in care within 24 hours after hospital admission. Time of follow-up is 14 days. At least 1184 patients will be included. Secondary outcomes include length of hospital stay, diagnostics and data regarding CDDS usage, physicians' confidence calibration and diagnostic workflow. Statistical analysis will use general linear mixed modelling methods.

ETHICS AND DISSEMINATION

Approved by the cantonal ethics committee of canton Berne (2022-D0002) and Swissmedic, the Swiss national regulatory authority on medical devices. Study results will be disseminated through peer-reviewed journals, open repositories and the network of investigators and the expert and patients advisory board.

TRIAL REGISTRATION NUMBER

NCT05346523.

A clinically-guided unsupervised clustering approach to recommend symptoms of disease associated with diagnostic opportunities

OBJECTIVES

A first step in studying diagnostic delays is to select the signs, symptoms and alternative diseases that represent missed diagnostic opportunities. Because this step is labor intensive requiring exhaustive literature reviews, we developed machine learning approaches to mine administrative data sources and recommend conditions for consideration. We propose a methodological approach to find diagnostic codes that exhibit known patterns of diagnostic delays and apply this to the diseases of tuberculosis and appendicitis.

METHODS

We used the IBM MarketScan Research Databases, and consider the initial symptoms of cough before tuberculosis and abdominal pain before appendicitis. We analyze diagnosis codes during healthcare visits before the index diagnosis, and use k-means clustering to recommend conditions that exhibit similar trends to the initial symptoms provided. We evaluate the clinical plausibility of the recommended conditions and the corresponding number of possible diagnostic delays based on these diseases.

RESULTS

For both diseases of interest, the clustering approach suggested a large number of clinically-plausible conditions to consider (e.g., fever, hemoptysis, and pneumonia before tuberculosis). The recommended conditions had a high degree of precision in terms of clinical plausibility: >70% for tuberculosis and >90% for appendicitis. Including these additional clinically-plausible conditions resulted in more than twice the number of possible diagnostic delays identified.

CONCLUSIONS

Our approach can mine administrative datasets to detect patterns of diagnostic delay and help investigators avoid under-identifying potential missed diagnostic opportunities. In addition, the methods we describe can be used to discover less-common presentations of diseases that are frequently misdiagnosed.

Diagnostic errors in Dentistry, opinions of egyptian dental teaching staff, a cross-sectional study

BACKGROUND

Diagnostic errors is a known problem in healthcare practice. Data on diagnostic errors in the dental field are extremely lacking. The objective of the study is to explore the perception of dental teaching staff about the prevalence of dental diagnostic errors in Egypt, identify the most commonly misdiagnosed dental conditions and point out the contributing factors and levels of patient harm.

METHODS

A cross-sectional questionnaire-based study was conducted on 151 dental teaching staff of Egyptian governmental and private universities. The questionnaire was distributed electronically via social media and messaging apps to dental staff members with at least five years of clinical experience to assess their opinion regarding the study objectives. Results were collected and statistically analyzed.

RESULTS

94.7% of participants believed that diagnostic errors represent an urgent problem, lecturers believed by 2.703 folds more than professors that diagnostic errors are an urgent problem The percentage of diagnostic errors was estimated to be < 20% and 20-40% by more than 90% of participants. The most commonly misdiagnosed conditions were oral mucosal lesions (83.4%), followed by temporomandibular joint and periodontal conditions (58.9%) for each. More than half of the participants (60.9%) believe that medical education methodology is one of the factors that lead to dental diagnosis errors. For the impact of errors on patients, 53% of participants reported moderate impacts followed by minor impact (37.7%) while 4.6% reported no impact and the same percentage reported major impact.

CONCLUSION

This study with statistically significant results reported that dental diagnostic errors are frequent and need to be approached. Oral mucosal lesions, periodontal and temporomandibular joint diseases represent areas that include the most commonly seen errors. Further, besides the lack of resources, the dental education system and lack of proper training are the main causes of this problem.

Common things are common, but what is common? Incorporating probability information into differential diagnosis

The well-known clinical axiom declaring that 'common things are common' attests to the pivotal role of probability in diagnosis. Despite the popularity of this and related axioms, there is no operationalized definition of a common disease, and no practicable way of incorporating actual disease frequencies into differential diagnosis. In this essay, we aim to disambiguate the definition of a common (or rare) disease and show that incidence-not prevalence-is the proper metric of disease frequency for differential diagnosis. We explore how numerical estimates of disease frequencies based on incidence can be incorporated into differential diagnosis as well as the inherent limitations of this method. These concepts have important implications for diagnostic decision making and medical education, and hold promise as a method to improve diagnostic accuracy.

Creating a Learning Health System for Improving Diagnostic Safety: Pragmatic Insights from US Health Care Organizations

OBJECTIVE

To identify challenges and pragmatic strategies for improving diagnostic safety at an organizational level using concepts from learning health systems METHODS: We interviewed 32 safety leaders across the USA on how their organizations approach diagnostic safety. Participants were recruited through email and represented geographically diverse academic and non-academic settings. The interview included questions on culture of reporting and learning from diagnostic errors; data gathering and analysis activities; diagnostic training and educational activities; and engagement of clinical leadership, staff, patients, and families in diagnostic safety activities. We conducted an inductive content analysis of interview transcripts and two reviewers coded all data.

RESULTS

Of 32 participants, 12 reported having a specific program to address diagnostic errors. Multiple barriers to implement diagnostic safety activities emerged: serious concerns about psychological safety associated with diagnostic error; lack of infrastructure for measurement, monitoring, and improvement activities related to diagnosis; lack of leadership investment, which was often diverted to competing priorities related to publicly reported measures or other incentives; and lack of dedicated teams to work on diagnostic safety. Participants provided several strategies to overcome barriers including adapting trigger tools to identify safety events, engaging patients in diagnostic safety, and appointing dedicated diagnostic safety champions.

CONCLUSIONS

Several foundational building blocks related to learning health systems could inform organizational efforts to reduce diagnostic error. Promoting an organizational culture specific to diagnostic safety, using science and informatics to improve measurement and analysis, leadership incentives to build institutional capacity to address diagnostic errors, and patient engagement in diagnostic safety activities can enable progress.

Measure Dx: Implementing pathways to discover and learn from diagnostic errors

Despite the high frequency of diagnostic errors, multiple barriers, including measurement, make it difficult learn from these events. This article discusses Measure Dx, a new resource from the Agency for Healthcare Research and Quality that translates knowledge from diagnostic safety measurement research into actionable recommendations. Measure Dx guides healthcare organizations to detect, analyze, and learn from diagnostic safety events as part of a continuous learning and feedback cycle. Wider adoption of Measure Dx, along with the implementation of solutions that result, can advance new frontiers in reducing preventable diagnostic harm to patients.

Translating electronic health record-based patient safety algorithms from research to clinical practice at multiple sites

Introduction

Researchers are increasingly developing algorithms that impact patient care, but algorithms must also be implemented in practice to improve quality and safety.

Objective

We worked with clinical operations personnel at two US health systems to implement algorithms to proactively identify patients without timely follow-up of abnormal test results that warrant diagnostic evaluation for colorectal or lung cancer. We summarise the steps involved and lessons learned.

Methods

Twelve sites were involved across two health systems. Implementation involved extensive software documentation, frequent communication with sites and local validation of results. Additionally, we used automated edits of existing code to adapt it to sites’ local contexts.

Results

All sites successfully implemented the algorithms. Automated edits saved sites significant work in direct code modification. Documentation and communication of changes further aided sites in implementation.

Conclusion

Patient safety algorithms developed in research projects were implemented at multiple sites to monitor for missed diagnostic opportunities. Automated algorithm translation procedures can produce more consistent results across sites.

Diagnostic Error: Why Now?

Diagnostic errors remain relatively understudied and underappreciated. They are particularly concerning in the intensive care unit, where they are more likely to result in harm to patients. There is a lack of consensus on the definition of diagnostic error, and current methods to quantify diagnostic error have numerous limitations as noted in the sentinel report by the National Academy of Medicine. Although definitive definition and measurement remain elusive goals, increasing our understanding of diagnostic error is crucial if we are to make progress in reducing the incidence and harm caused by errors in diagnosis.

Filling a gap in safety metrics: development of a patient-centred framework to identify and categorise patient-reported breakdowns related to the diagnostic process in ambulatory care

BACKGROUND

Patients and families are important contributors to the diagnostic team, but their perspectives are not reflected in current diagnostic measures. Patients/families can identify some breakdowns in the diagnostic process beyond the clinician's view. We aimed to develop a framework with patients/families to help organisations identify and categorise patient-reported diagnostic process-related breakdowns (PRDBs) to inform organisational learning.

METHOD

A multi-stakeholder advisory group including patients, families, clinicians, and experts in diagnostic error, patient engagement and safety, and user-centred design, co-developed a framework for PRDBs in ambulatory care. We tested the framework using standard qualitative analysis methods with two physicians and one patient coder, analysing 2165 patient-reported ambulatory errors in two large surveys representing 25 425 US respondents. We tested intercoder reliability of breakdown categorisation using the Gwet's AC1 and Cohen's kappa statistic. We considered agreement coefficients 0.61-0.8=good agreement and 0.81-1.00=excellent agreement.

RESULTS

The framework describes 7 patient-reported breakdown categories (with 40 subcategories), 19 patient-identified contributing factors and 11 potential patient-reported impacts. Patients identified breakdowns in each step of the diagnostic process, including missing or inaccurate main concerns and symptoms; missing/outdated test results; and communication breakdowns such as not feeling heard or misalignment between patient and provider about symptoms, events, or their significance. The frequency of PRDBs was 6.4% in one dataset and 6.9% in the other. Intercoder reliability showed good-to-excellent reliability in each dataset: AC1 0.89 (95% CI 0.89 to 0.90) to 0.96 (95% CI 0.95 to 0.97); kappa 0.64 (95% CI 0.62, to 0.66) to 0.85 (95% CI 0.83 to 0.88).

CONCLUSIONS

The PRDB framework, developed in partnership with patients/families, can help organisations identify and reliably categorise PRDBs, including some that are invisible to clinicians; guide interventions to engage patients and families as diagnostic partners; and inform whole organisational learning.

Validation of an electronic trigger to measure missed diagnosis of stroke in emergency departments

OBJECTIVE

Diagnostic errors are major contributors to preventable patient harm. We validated the use of an electronic health record (EHR)-based trigger (e-trigger) to measure missed opportunities in stroke diagnosis in emergency departments (EDs).

METHODS

Using two frameworks, the Safer Dx Trigger Tools Framework and the Symptom-disease Pair Analysis of Diagnostic Error Framework, we applied a symptom-disease pair-based e-trigger to identify patients hospitalized for stroke who, in the preceding 30 days, were discharged from the ED with benign headache or dizziness diagnoses. The algorithm was applied to Veteran Affairs National Corporate Data Warehouse on patients seen between 1/1/2016 and 12/31/2017. Trained reviewers evaluated medical records for presence/absence of missed opportunities in stroke diagnosis and stroke-related red-flags, risk factors, neurological examination, and clinical interventions. Reviewers also estimated quality of clinical documentation at the index ED visit.

RESULTS

We applied the e-trigger to 7,752,326 unique patients and identified 46,931 stroke-related admissions, of which 398 records were flagged as trigger-positive and reviewed. Of these, 124 had missed opportunities (positive predictive value for "missed" = 31.2%), 93 (23.4%) had no missed opportunity (non-missed), 162 (40.7%) were miscoded, and 19 (4.7%) were inconclusive. Reviewer agreement was high (87.3%, Cohen's kappa = 0.81). Compared to the non-missed group, the missed group had more stroke risk factors (mean 3.2 vs 2.6), red flags (mean 0.5 vs 0.2), and a higher rate of inadequate documentation (66.9% vs 28.0%).

CONCLUSION

In a large national EHR repository, a symptom-disease pair-based e-trigger identified missed diagnoses of stroke with a modest positive predictive value, underscoring the need for chart review validation procedures to identify diagnostic errors in large data sets.

Bringing the clinical laboratory into the strategy to advance diagnostic excellence

OBJECTIVES

Clinical laboratory testing provides essential data for making medical diagnoses. Generating accurate and timely test results clearly communicated to the treating clinician, and ultimately the patient, is a critical component that supports diagnostic excellence. On the other hand, failure to achieve this can lead to diagnostic errors that manifest in missed, delayed and wrong diagnoses.

CONTENT

Innovations that support diagnostic excellence address: 1) test utilization, 2) leveraging clinical and laboratory data, 3) promoting the use of credible information resources, 4) enhancing communication among laboratory professionals, health care providers and the patient, and 5) advancing the use of diagnostic management teams. Integrating evidence-based laboratory and patient-care quality management approaches may provide a strategy to support diagnostic excellence. Professional societies, government agencies, and healthcare systems are actively engaged in efforts to advance diagnostic excellence. Leveraging clinical laboratory capabilities within a healthcare system can measurably improve the diagnostic process and reduce diagnostic errors.

SUMMARY

An expanded quality management approach that builds on existing processes and measures can promote diagnostic excellence and provide a pathway to transition innovative concepts to practice.

OUTLOOK

There are increasing opportunities for clinical laboratory professionals and organizations to be part of a strategy to improve diagnoses.

Diagnostic Errors in Pediatric Critical Care: A Systematic Review

OBJECTIVES

To summarize the literature on prevalence, impact, and contributing factors related to diagnostic error in the PICU.

DATA SOURCES

Search of PubMed, EMBASE, and the Cochrane Library up to December 2019.

STUDY SELECTION

Studies on diagnostic error and the diagnostic process in pediatric critical care were included. Non-English studies with no translation, case reports/series, studies providing no information on diagnostic error, studies focused on non-PICU populations, and studies focused on a single condition/disease or a single diagnostic test/tool were excluded.

DATA EXTRACTION

Data on research design, objectives, study sample, and results pertaining to the prevalence, impact, and factors associated with diagnostic error were abstracted from each study.

DATA SYNTHESIS

Using independent tiered review, 396 abstracts were screened, and 17 studies (14 full-text, 3 abstracts) were ultimately included. Fifteen of 17 studies (88%) had an observational research design. Autopsy studies (autopsy rates were 20-47%) showed a 10-23% rate of missed major diagnoses; 5-16% of autopsy-discovered diagnostic errors had a potential adverse impact on survival and would have changed management. Retrospective record reviews reported varying rates of diagnostic error from 8% in a general PICU population to 12% among unexpected critical admissions and 21-25% of patients discussed at PICU morbidity and mortality conferences. Cardiovascular, infectious, congenital, and neurologic conditions were most commonly misdiagnosed. Systems factors (40-67%), cognitive factors (20-3%), and both systems and cognitive factors (40%) were associated with diagnostic error. Limited information was available on the impact of misdiagnosis.

CONCLUSIONS

Knowledge of diagnostic errors in the PICU is limited. Future work to understand diagnostic errors should involve a balanced focus between studying the diagnosis of individual diseases and uncovering common system- and process-related determinants of diagnostic error.

Use of patient complaints to identify diagnosis-related safety concerns: a mixed-method evaluation

Background

Patient complaints are associated with adverse events and malpractice claims but underused in patient safety improvement.

Objective

To systematically evaluate the use of patient complaint data to identify safety concerns related to diagnosis as an initial step to using this information to facilitate learning and improvement.

Methods

We reviewed patient complaints submitted to Geisinger, a large healthcare organisation in the USA, from August to December 2017 (cohort 1) and January to June 2018 (cohort 2). We selected complaints more likely to be associated with diagnostic concerns in Geisinger’s existing complaint taxonomy. Investigators reviewed all complaint summaries and identified cases as ‘concerning’ for diagnostic error using the National Academy of Medicine’s definition of diagnostic error. For all ‘concerning’ cases, a clinician-reviewer evaluated the associated investigation report and the patient’s medical record to identify any missed opportunities in making a correct or timely diagnosis. In cohort 2, we selected a 10% sample of ‘concerning’ cases to test this smaller pragmatic sample as a proof of concept for future organisational monitoring.

Results

In cohort 1, we reviewed 1865 complaint summaries and identified 177 (9.5%) concerning reports. Review and analysis identified 39 diagnostic errors. Most were categorised as ‘Clinical Care issues’ (27, 69.2%), defined as concerns/questions related to the care that is provided by clinicians in any setting. In cohort 2, we reviewed 2423 patient complaint summaries and identified 310 (12.8%) concerning reports. The 10% sample (n=31 cases) contained five diagnostic errors. Qualitative analysis of cohort 1 cases identified concerns about return visits for persistent and/or worsening symptoms, interpersonal issues and diagnostic testing.

Conclusions

Analysis of patient complaint data and corresponding medical record review identifies patterns of failures in the diagnostic process reported by patients and families. Health systems could systematically analyse available data on patient complaints to monitor diagnostic safety concerns and identify opportunities for learning and improvement.

Prolonged Diagnostic Intervals as Marker of Missed Diagnostic Opportunities in Bladder and Kidney Cancer Patients with Alarm Features: A Longitudinal Linked Data Study

BACKGROUND

In England, patients who meet National Institute for Health and Care Excellence (NICE) guideline criteria for suspected cancer should receive a specialist assessment within 14 days. We examined how quickly bladder and kidney cancer patients who met fast-track referral criteria were actually diagnosed.

METHODS

We used linked primary care and cancer registration data on bladder and kidney cancer patients who met fast-track referral criteria and examined the time from their first presentation with alarm features to diagnosis. Using logistic regression we examined factors most likely to be associated with non-timely diagnosis (defined as intervals exceeding 90 days), adjusting for age, sex and cancer type, positing that such occurrences represent missed opportunity for timely referral, possibly due to sub-optimal guideline adherence.

RESULTS

28%, 42% and 31% of all urological cancer patients reported no, one or two or more relevant symptoms respectively in the year before diagnosis. Of the 2105 patients with alarm features warranting fast-track assessment, 1373 (65%) presented with unexplained haematuria, 382 (18%) with recurrent urinary tract infections (UTIs), 303 (14%) with visible haematuria, and 45 (2%) with an abdominal mass. 27% overall, and 24%, 45%, 18% and 27% of each group respectively, had a non-timely diagnosis. Presentation with recurrent UTI was associated with longest median diagnostic interval (median 83 days, IQR 43-151) and visible haematuria with the shortest (median 50 days, IQR 30-79). After adjustment, presentation with recurrent UTIs, being in the youngest or oldest age group, female sex, and diagnosis of kidney and upper tract urothelial cancer, were associated with greater odds of non-timely diagnosis.

CONCLUSION

More than a quarter of patients presenting with fast-track referral features did not achieve a timely diagnosis, suggesting inadequate guideline adherence for some patients. The findings highlight a substantial number of opportunities for expediting the diagnosis of patients with bladder or kidney cancers.

Identifying trigger concepts to screen emergency department visits for diagnostic errors

OBJECTIVES

The diagnostic process is a vital component of safe and effective emergency department (ED) care. There are no standardized methods for identifying or reliably monitoring diagnostic errors in the ED, impeding efforts to enhance diagnostic safety. We sought to identify trigger concepts to screen ED records for diagnostic errors and describe how they can be used as a measurement strategy to identify and reduce preventable diagnostic harm.

METHODS

We conducted a literature review and surveyed ED directors to compile a list of potential electronic health record (EHR) trigger (e-triggers) and non-EHR based concepts. We convened a multidisciplinary expert panel to build consensus on trigger concepts to identify and reduce preventable diagnostic harm in the ED.

RESULTS

Six e-trigger and five non-EHR based concepts were selected by the expert panel. E-trigger concepts included: unscheduled ED return to ED resulting in hospital admission, death following ED visit, care escalation, high-risk conditions based on symptom-disease dyads, return visits with new diagnostic/therapeutic interventions, and change of treating service after admission. Non-EHR based signals included: cases from mortality/morbidity conferences, risk management/safety office referrals, ED medical director case referrals, patient complaints, and radiology/laboratory misreads and callbacks. The panel suggested further refinements to aid future research in defining diagnostic error epidemiology in ED settings.

CONCLUSIONS

We identified a set of e-trigger concepts and non-EHR based signals that could be developed further to screen ED visits for diagnostic safety events. With additional evaluation, trigger-based methods can be used as tools to monitor and improve ED diagnostic performance.