Application of electronic trigger tools to identify targets for improving diagnostic safety

Performance evaluation of ChatGPT in detecting diagnostic errors and their contributing factors: an analysis of 545 case reports of diagnostic errors

BACKGROUND

Manual chart review using validated assessment tools is a standardised methodology for detecting diagnostic errors. However, this requires considerable human resources and time. ChatGPT, a recently developed artificial intelligence chatbot based on a large language model, can effectively classify text based on suitable prompts. Therefore, ChatGPT can assist manual chart reviews in detecting diagnostic errors.

OBJECTIVE

This study aimed to clarify whether ChatGPT could correctly detect diagnostic errors and possible factors contributing to them based on case presentations.

METHODS

We analysed 545 published case reports that included diagnostic errors. We imputed the texts of case presentations and the final diagnoses with some original prompts into ChatGPT (GPT-4) to generate responses, including the judgement of diagnostic errors and contributing factors of diagnostic errors. Factors contributing to diagnostic errors were coded according to the following three taxonomies: Diagnosis Error Evaluation and Research (DEER), Reliable Diagnosis Challenges (RDC) and Generic Diagnostic Pitfalls (GDP). The responses on the contributing factors from ChatGPT were compared with those from physicians.

RESULTS

ChatGPT correctly detected diagnostic errors in 519/545 cases (95%) and coded statistically larger numbers of factors contributing to diagnostic errors per case than physicians: DEER (median 5 vs 1, p<0.001), RDC (median 4 vs 2, p<0.001) and GDP (median 4 vs 1, p<0.001). The most important contributing factors of diagnostic errors coded by ChatGPT were 'failure/delay in considering the diagnosis' (315, 57.8%) in DEER, 'atypical presentation' (365, 67.0%) in RDC, and 'atypical presentation' (264, 48.4%) in GDP.

CONCLUSION

ChatGPT accurately detects diagnostic errors from case presentations. ChatGPT may be more sensitive than manual reviewing in detecting factors contributing to diagnostic errors, especially for 'atypical presentation'.

Partnership as a Pathway to Diagnostic Excellence: The Challenges and Successes of Implementing the Safer Dx Learning Lab

BACKGROUND

Learning health system (LHS) approaches could potentially help health care organizations (HCOs) identify and address diagnostic errors. However, few such programs exist, and their implementation is poorly understood.

METHODS

The authors conducted a qualitative evaluation of the Safer Dx Learning Lab, a partnership between a health system and a research team, to identify and learn from diagnostic errors and improve diagnostic safety at an organizational level. The research team conducted virtual interviews to solicit participant feedback regarding experiences with the lab, focusing specifically on implementation and sustainment issues.

RESULTS

Interviews of 25 members associated with the lab identified the following successes: learning and professional growth, improved workflow related to streamlining the process of reporting error cases, and a psychologically safe culture for identifying and reporting diagnostic errors. However, multiple barriers also emerged: competing priorities between clinical responsibilities and research, time-management issues related to a lack of protected time, and inadequate guidance to disseminate findings. Lessons learned included understanding the importance of obtaining buy-in from leadership and interested stakeholders, creating a psychologically safe environment for reporting cases, and the need for more protected time for clinicians to review and learn from cases.

CONCLUSION

Findings suggest that a learning health systems approach using partnerships between researchers and a health system affected organizational culture by prioritizing learning from diagnostic errors and encouraging clinicians to be more open to reporting. The study findings can help organizations overcome barriers to engage clinicians and inform future implementation and sustainment of similar initiatives.

Management of acute exacerbations of COPD in the emergency department and its associations with clinical variables

Acute exacerbation of chronic obstructive pulmonary disease (AECOPD) is a common cause for emergency department (ED) visits. Still, large scale studies that assess the management of AECOPD in the ED are limited. Our aim was to evaluate treatment characteristics of AE-COPD in the ED on a national scale. A prospective study as part of the COPD Israeli survey, conducted between 2017 and 2019, in 13 medical centers. Patients hospitalized with AECOPD were included and interviewed. Clinical data related to their ED and hospital stay were collected. 344 patients were included, 38% females, mean age of 70 ± 11 years. Median (IQR) time to first ED treatment was 59 (23-125) minutes and to admission 293 (173-490) minutes. Delayed ED treatment (> 1 h) was associated with older age (p = 0.01) and lack of a coded diagnosis of COPD in hospital records (p = 0.01). Long ED length-of-stay (> 5 h) was linked with longer hospitalizations (p = 0.01). Routine ED care included inhalations of short-acting bronchodilators (246 patients, 72%) and systemic steroids (188 patients, 55%). Receiving routine ED care was associated with its continuation during hospitalization (p < 0.001). In multivariate analysis, predictors for patients not receiving routine care were obesity (adjusted odds ratio 0.5, 95% CI 0.3-0.8, p = 0.01) and fever (AOR 0.3, 95% CI 0.1-0.6, p < 0.01), while oxygen saturation < 91% was an independent predictor for ED routine treatment (AOR 3.6, 95% CI 2.1-6.3, p < 0.01). Our findings highlight gaps in the treatment of AECOPD in the ED on a national scale, with specific predictors for their occurrence.

An electronic trigger to detect telemedicine-related diagnostic errors

INTRODUCTION

The COVID-19 pandemic advanced the use of telehealth-facilitated care. However, little is known about how to measure safety of clinical diagnosis made through telehealth-facilitated primary care.

METHODS

We used the seven-step Safer Dx Trigger Tool framework to develop an electronic trigger (e-trigger) tool to identify potential missed opportunities for more timely diagnosis during primary care telehealth visits at a large Department of Veterans Affairs facility. We then applied the e-trigger algorithm to electronic health record data related to primary care visits during a 1-year period (1 April 2020-31 March 2021). The algorithm identified patients with unexpected visits within 10 days of an index telemedicine visit and classified such records as e-trigger positive. We then validated the e-trigger's ability to detect missed opportunities in diagnosis using chart reviews based on a structured data collection instrument (the Revised Safer Dx instrument).

RESULTS

We identified 128,761 telehealth visits (32,459 unique patients), of which 434 visits led to subsequent unplanned emergency department (ED), hospital, or primary care visits within 10 days of the index visit. Of these, 116 were excluded for clinical reasons (trauma, injury, or childbirth), leaving 318 visits (240 unique patients) needing further evaluation. From these, 100 records were randomly selected for review, of which four were falsely flagged due to invalid data (visits by non-providers or those incorrectly flagged as completed telehealth visits). Eleven patients had a missed opportunity in diagnosis, yielding a positive predictive value of 11%.

DISCUSSION

Electronic triggers that identify missed opportunities for additional evaluation could help advance the understanding of safety of clinical diagnosis made in telehealth-enabled care. Better measurement can help determine which patients can safely be cared for via telemedicine versus traditional in-person visits.

Medical Home Implementation and Follow-Up of Cancer-Related Abnormal Test Results in the Veterans Health Administration

Importance

Lack of timely follow-up of cancer-related abnormal test results can lead to delayed or missed diagnoses, adverse cancer outcomes, and substantial cost burden for patients. Care delivery models, such as the Veterans Affairs' (VA) Patient-Aligned Care Team (PACT), which aim to improve patient-centered care coordination, could potentially also improve timely follow-up of abnormal test results. PACT was implemented nationally in the VA between 2010 and 2012.

Objective

To evaluate the long-term association between PACT implementation and timely follow-up of abnormal test results related to the diagnosis of 5 different cancers.

Design, Setting, and Participants

This multiyear retrospective cohort study used 14 years of VA data (2006-2019), which were analyzed using panel data-based random-effects linear regressions. The setting included all VA clinics and facilities. The participants were adult patients who underwent diagnostic testing related to 5 different cancers and had abnormal test results. Data extraction and statistical analyses were performed from September 2021 to December 2023.

Exposure

Calendar years denoting preperiods and postperiods of PACT implementation, and the PACT Implementation Progress Index Score denoting the extent of implementation in each VA clinic and facility.

Main Outcome and Measure

Percentage of potentially missed timely follow-ups of abnormal test results.

Results

This study analyzed 6 data sets representing 5 different types of cancers. During the initial years of PACT implementation (2010 to 2013), percentage of potentially missed timely follow-ups decreased between 3 to 7 percentage points for urinalysis suggestive of bladder cancer, 12 to 14 percentage points for mammograms suggestive of breast cancer, 19 to 22 percentage points for fecal tests suggestive of colorectal cancer, and 6 to 13 percentage points for iron deficiency anemia laboratory tests suggestive of colorectal cancer, with no statistically significant changes for α-fetoprotien tests and lung cancer imaging. However, these beneficial reductions were not sustained over time. Better PACT implementation scores were associated with a decrease in potentially missed timely follow-up percentages for urinalysis (0.3-percentage point reduction [95% CI, -0.6 to -0.1] with 1-point increase in the score), and laboratory tests suggestive of iron deficiency anemia (0.5-percentage point reduction [95% CI,-0.8 to -0.2] with 1-point increase in the score).

Conclusions and Relevance

This cohort study found that implementation of PACT in the VA was associated with a potential short-term improvement in the quality of follow-up for certain test results. Additional multifaceted sustained interventions to reduce missed test results are required to prevent care delays.

The PRIDx framework to engage payers in reducing diagnostic errors in healthcare

OBJECTIVES

No framework currently exists to guide how payers and providers can collaboratively develop and implement incentives to improve diagnostic safety. We conducted a literature review and interviews with subject matter experts to develop a multi-component 'Payer Relationships for Improving Diagnoses (PRIDx)' framework, that could be used to engage payers in diagnostic safety efforts.

CONTENT

The PRIDx framework, 1) conceptualizes diagnostic safety links to care provision, 2) illustrates ways to promote payer and provider engagement in the design and adoption of accountability mechanisms, and 3) explicates the use of data analytics. Certain approaches suggested by PRIDx were refined by subject matter expert interviewee perspectives.

SUMMARY

The PRIDx framework can catalyze public and private payers to take specific actions to improve diagnostic safety.

OUTLOOK

Implementation of the PRIDx framework requires new types of partnerships, including external support from public and private payer organizations, and requires creation of strong provider incentives without undermining providers' sense of professionalism and autonomy. PRIDx could help facilitate collaborative payer-provider approaches to improve diagnostic safety and generate research concepts, policy ideas, and potential innovations for engaging payers in diagnostic safety improvement activities.

Characteristics and contributing factors of diagnostic error in surgery: analysis of closed medico-legal cases and complaints in Canada

BACKGROUND

Diagnostic errors lead to patient harm; however, most research has been conducted in nonsurgical disciplines. We sought to characterize diagnostic error in the pre-, intra-, and postoperative surgical phases, describe their contributing factors, and quantify their impact related to patient harm.

METHODS

We performed a retrospective analysis of closed medico-legal cases and complaints using a database representing more than 95% of all Canadian physicians. We included cases if they involved a legal action or complaint that closed between 2014 and 2018 and involved a diagnostic error assigned by peer expert review to a surgeon.

RESULTS

We identified 387 surgical cases that involved a diagnostic error. The surgical specialties most often associated with diagnostic error were general surgery (n = 151, 39.0%), gynecology (n = 71, 18.3%), and orthopedic surgery (n = 48, 12.4%), but most surgical specialties were represented. Errors occurred more often in the postoperative phase (n = 171, 44.2%) than in the pre- (n = 127, 32.8%) or intra-operative (n = 120, 31.0%) phases of surgical care. More than 80% of the contributing factors for diagnostic errors were related to providers, with clinical decision-making being the principal contributing factor. Half of the contributing factors were related to the health care team (n = 194, 50.1%), the most common of which was communication breakdown. More than half of patients involved in a surgical diagnostic error experienced at least moderate harm, with 1 in 7 cases resulting in death.

CONCLUSION

In our cohort, diagnostic errors occurred in most surgical disciplines and across all surgical phases of care; contributing factors were commonly attributed to provider clinical decision-making and communication breakdown. Surgical patient safety efforts should include diagnostic errors with a focus on understanding and reducing errors in surgical clinical decision-making and improving communication.

A taxonomy of early diagnosis research to guide study design and funding prioritisation

Researchers and research funders aiming to improve diagnosis seek to identify if, when, where, and how earlier diagnosis is possible. This has led to the propagation of research studies using a wide range of methodologies and data sources to explore diagnostic processes. Many such studies use electronic health record data and focus on cancer diagnosis. Based on this literature, we propose a taxonomy to guide the design and support the synthesis of early diagnosis research, focusing on five key questions: Do healthcare use patterns suggest earlier diagnosis could be possible? How does the diagnostic process begin? How do patients progress from presentation to diagnosis? How long does the diagnostic process take? Could anything have been done differently to reach the correct diagnosis sooner? We define families of diagnostic research study designs addressing each of these questions and appraise their unique or complementary contributions and limitations. We identify three further questions on relationships between the families and their relevance for examining patient group inequalities, supported with examples from the cancer literature. Although exemplified through cancer as a disease model, we recognise the framework is also applicable to non-neoplastic disease. The proposed framework can guide future study design and research funding prioritisation.

Development of Electronic Health Record-Based Machine Learning Models to Predict Barrett's Esophagus and Esophageal Adenocarcinoma Risk

INTRODUCTION

Screening for Barrett's esophagus (BE) is suggested in those with risk factors, but remains underutilized. BE/esophageal adenocarcinoma (EAC) risk prediction tools integrating multiple risk factors have been described. However, accuracy remains modest (area under the receiver-operating curve [AUROC] ≤0.7), and clinical implementation has been challenging. We aimed to develop machine learning (ML) BE/EAC risk prediction models from an electronic health record (EHR) database.

METHODS

The Clinical Data Analytics Platform, a deidentified EHR database of 6 million Mayo Clinic patients, was used to predict BE and EAC risk. BE and EAC cases and controls were identified using International Classification of Diseases codes and augmented curation (natural language processing) techniques applied to clinical, endoscopy, laboratory, and pathology notes. Cases were propensity score matched to 5 independent randomly selected control groups. An ensemble transformer-based ML model architecture was used to develop predictive models.

RESULTS

We identified 8,476 BE cases, 1,539 EAC cases, and 252,276 controls. The BE ML transformer model had an overall sensitivity, specificity, and AUROC of 76%, 76%, and 0.84, respectively. The EAC ML transformer model had an overall sensitivity, specificity, and AUROC of 84%, 70%, and 0.84, respectively. Predictors of BE and EAC included conventional risk factors and additional novel factors, such as coronary artery disease, serum triglycerides, and electrolytes.

DISCUSSION

ML models developed on an EHR database can predict incident BE and EAC risk with improved accuracy compared with conventional risk factor-based risk scores. Such a model may enable effective implementation of a minimally invasive screening technology.

Partnering with Patients and Families to Improve Diagnostic Safety through the OurDX Tool: Effects of Race, Ethnicity, and Language Preference

BACKGROUND

Patients and families at risk for health disparities may also be at higher risk for diagnostic errors but less likely to report them.

OBJECTIVES

This study aimed to explore differences in race, ethnicity, and language preference associated with patient and family contributions and concerns using an electronic previsit tool designed to engage patients and families in the diagnostic process (DxP).

METHODS

Cross-sectional study of 5,731 patients and families presenting to three subspecialty clinics at an urban pediatric hospital May to December 2021 who completed a previsit tool, codeveloped and tested with patients and families. Prior to each visit, patients/families were invited to share visit priorities, recent histories, and potential diagnostic concerns. We used logistic regression to determine factors associated with patient-reported diagnostic concerns. We conducted chart review on a random subset of visits to review concerns and determine whether patient/family contributions were included in the visit note.

RESULTS

Participants provided a similar mean number of contributions regardless of patient race, ethnicity, or language preference. Compared with patients self-identifying as White, those self-identifying as Black (odds ratio [OR]: 1.70; 95% confidence interval [CI]: [1.18, 2.43]) or "other" race (OR: 1.48; 95% CI: [1.08, 2.03]) were more likely to report a diagnostic concern. Participants who preferred a language other than English were more likely to report a diagnostic concern than English-preferring patients (OR: 2.53; 95% CI: [1.78, 3.59]. There were no significant differences in physician-verified diagnostic concerns or in integration of patient contributions into the note based on race, ethnicity, or language preference.

CONCLUSION

Participants self-identifying as Black or "other" race, or those who prefer a language other than English were 1.5 to 2.5 times more likely than their counterparts to report potential diagnostic concerns when proactively asked to provide this information prior to a visit. Actively engaging patients and families in the DxP may uncover opportunities to reduce the risk of diagnostic errors and potential safety disparities.

Patient and family contributions to improve the diagnostic process through the OurDX electronic health record tool: a mixed method analysis

BACKGROUND

Accurate and timely diagnosis relies on sharing perspectives among team members and avoiding information asymmetries. Patients/Families hold unique diagnostic process (DxP) information, including knowledge of diagnostic safety blindspots-information that patients/families know, but may be invisible to clinicians. To improve information sharing, we co-developed with patients/families an online tool called 'Our Diagnosis (OurDX)'. We aimed to characterise patient/family contributions in OurDX and how they differed between individuals with and without diagnostic concerns.

METHOD

We implemented OurDX in two academic organisations serving patients/families living with chronic conditions in three subspecialty clinics and one primary care clinic. Prior to each visit, patients/families were invited to contribute visit priorities, recent histories and potential diagnostic concerns. Responses were available in the electronic health record and could be incorporated by clinicians into visit notes. We randomly sampled OurDX reports with and without diagnostic concerns for chart review and used inductive and deductive qualitative analysis to assess patient/family contributions.

RESULTS

7075 (39%) OurDX reports were submitted at 18 129 paediatric subspecialty clinic visits and 460 (65%) reports were submitted among 706 eligible adult primary care visits. Qualitative analysis of OurDX reports in the chart review sample (n=450) revealed that participants contributed DxP information across 10 categories, most commonly: clinical symptoms/medical history (82%), tests/referrals (54%) and diagnosis/next steps (51%). Participants with diagnostic concerns were more likely to contribute information on DxP risks including access barriers, recent visits for the same problem, problems with tests/referrals or care coordination and communication breakdowns, some of which may represent diagnostic blindspots.

CONCLUSION

Partnering with patients and families living with chronic conditions through OurDX may help clinicians gain a broader perspective of the DxP, including unique information to coproduce diagnostic safety.

Developing electronic clinical quality measures to assess the cancer diagnostic process

OBJECTIVE

Measures of diagnostic performance in cancer are underdeveloped. Electronic clinical quality measures (eCQMs) to assess quality of cancer diagnosis could help quantify and improve diagnostic performance.

MATERIALS AND METHODS

We developed 2 eCQMs to assess diagnostic evaluation of red-flag clinical findings for colorectal (CRC; based on abnormal stool-based cancer screening tests or labs suggestive of iron deficiency anemia) and lung (abnormal chest imaging) cancer. The 2 eCQMs quantified rates of red-flag follow-up in CRC and lung cancer using electronic health record data repositories at 2 large healthcare systems. Each measure used clinical data to identify abnormal results, evidence of appropriate follow-up, and exclusions that signified follow-up was unnecessary. Clinicians reviewed 100 positive and 20 negative randomly selected records for each eCQM at each site to validate accuracy and categorized missed opportunities related to system, provider, or patient factors.

RESULTS

We implemented the CRC eCQM at both sites, while the lung cancer eCQM was only implemented at the VA due to lack of structured data indicating level of cancer suspicion on most chest imaging results at Geisinger. For the CRC eCQM, the rate of appropriate follow-up was 36.0% (26 746/74 314 patients) in the VA after removing clinical exclusions and 41.1% at Geisinger (1009/2461 patients; P < .001). Similarly, the rate of appropriate evaluation for lung cancer in the VA was 61.5% (25 166/40 924 patients). Reviewers most frequently attributed missed opportunities at both sites to provider factors (84 of 157).

CONCLUSIONS

We implemented 2 eCQMs to evaluate the diagnostic process in cancer at 2 large health systems. Health care organizations can use these eCQMs to monitor diagnostic performance related to cancer.

Comparing rates of adverse events detected in incident reporting and the Global Trigger Tool: a systematic review

Many hospitals continue to use incident reporting systems (IRSs) as their primary patient safety data source. The information IRSs collect on the frequency of harm to patients [adverse events (AEs)] is generally of poor quality, and some incident types (e.g. diagnostic errors) are under-reported. Other methods of collecting patient safety information using medical record review, such as the Global Trigger Tool (GTT), have been developed. The aim of this study was to undertake a systematic review to empirically quantify the gap between the percentage of AEs detected using the GTT to those that are also detected via IRSs. The review was conducted in adherence to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement. Studies published in English, which collected AE data using the GTT and IRSs, were included. In total, 14 studies met the inclusion criteria. All studies were undertaken in hospitals and were published between 2006 and 2022. The studies were conducted in six countries, mainly in the USA (nine studies). Studies reviewed 22 589 medical records using the GTT across 107 institutions finding 7166 AEs. The percentage of AEs detected using the GTT that were also detected in corresponding IRSs ranged from 0% to 37.4% with an average of 7.0% (SD 9.1; median 3.9 and IQR 5.2). Twelve of the fourteen studies found <10% of the AEs detected using the GTT were also found in corresponding IRSs. The >10-fold gap between the detection rates of the GTT and IRSs is strong evidence that the rate of AEs collected in IRSs in hospitals should not be used to measure or as a proxy for the level of safety of a hospital. IRSs should be recognized for their strengths which are to detect rare, serious, and new incident types and to enable analysis of contributing and contextual factors to develop preventive and corrective strategies. Health systems should use multiple patient safety data sources to prioritize interventions and promote a cycle of action and improvement based on data rather than merely just collecting and analysing information.

Advancing leadership in surgery: a realist review of interventions and strategies to promote evidence-based leadership in healthcare

BACKGROUND

Healthcare systems invest in leadership development of surgeons, surgical trainees, and teams. However, there is no agreement on how interventions should be designed, or what components they must contain to be successful. The objective of this realist review was to generate a programme theory explaining in which context and for whom surgical leadership interventions work and why.

METHODS

Five databases were systematically searched, and articles screened against inclusion considering their relevance. Context-mechanism-outcome configurations (CMOCs) and fragments of CMOCs were identified. Gaps in the CMOCs were filled through deliberation with the research team and stakeholder feedback. We identified patterns between CMOCs and causal relationships to create a programme theory.

RESULTS

Thirty-three studies were included and 19 CMOCs were developed. Findings suggests that interventions for surgeons and surgical teams improve leadership if timely feedback is delivered on multiple occasions and by trusted and respected people. Negative feedback is best provided privately. Feedback from senior-to-junior or peer-to-peer should be delivered directly, whereas feedback from junior-to-senior is preferred when delivered anonymously. Leadership interventions were shown to be most effective for those with awareness of the importance of leadership, those with confidence in their technical surgical skills, and those with identified leadership deficits. For interventions to improve leadership in surgery, they need to be delivered in an intimate learning environment, consider implementing a speak-up culture, provide a variety of interactive learning activities, show a genuine investment in the intervention, and be customised to the needs of surgeons. Leadership of surgical teams can be best developed by enabling surgical teams to train together.

CONCLUSIONS

The programme theory provides evidence-based guidance for those who are designing, developing and implementing leadership interventions in surgery. Adopting the recommendations will help to ensure interventions are acceptable to the surgical community and successful in improving surgical leadership.

TRIAL REGISTRATION

The review protocol is registered with PROSPERO (CRD42021230709).

Using Natural Language Processing to Automatically Identify Dysplasia in Pathology Reports for Patients With Barrett's Esophagus

BACKGROUND & AIMS

Identifying dysplasia of Barrett's esophagus (BE) in the electronic medical record (EMR) requires manual abstraction of unstructured data. Natural language processing (NLP) creates structure to unstructured free text. We aimed to develop and validate an NLP algorithm to identify dysplasia in BE patients on histopathology reports with varying report formats in a large integrated EMR system.

METHODS

We randomly selected 600 pathology reports for NLP development and 400 reports for validation from patients with suspected BE in the national Veterans Affairs databases. BE and dysplasia were verified by manual review of the pathology reports. We used NLP software (Clinical Language Annotation, Modeling, and Processing Toolkit; Melax Tech, Houston, TX) to develop an algorithm to identify dysplasia using findings. The algorithm performance characteristics were calculated as recall, precision, accuracy, and F-measure.

RESULTS

In the development set of 600 patients, 457 patients had confirmed BE (60 with dysplasia). The NLP identified dysplasia with 98.0% accuracy, 91.7% recall, and 93.2% precision, with an F-measure of 92.4%. All 7 patients with confirmed high-grade dysplasia were classified by the algorithm as having dysplasia. Among the 400 patients in the validation cohort, 230 had confirmed BE (39 with dysplasia). Compared with manual review, the NLP algorithm identified dysplasia with 98.7% accuracy, 92.3% recall, and 100.0% precision, with an F-measure of 96.0%.

CONCLUSIONS

NLP yielded a high degree of sensitivity and accuracy for identifying dysplasia from diverse types of pathology reports for patients with BE. The application of this algorithm would facilitate research and clinical care in an EMR system with text reports in large data repositories.

Physicians' Perspectives on a Multi-Dimensional Model for the Roles of Electronic Health Records in Approaching a Proper Differential Diagnosis

Many healthcare organizations have adopted Electronic Health Records (EHRs) to improve the quality of care and help physicians make proper clinical decisions. The vital roles of EHRs can support the accuracy of diagnosis, suggest, and rationalize the provided care to patients. This study aims to understand the roles of EHRs in approaching proper differential diagnosis and optimizing patient safety. This study utilized a cross-sectional survey-based descriptive research design to assess physicians' perceptions of the roles of EHRs on diagnosis quality and safety. Physicians working in tertiary hospitals in Saudi Arabia were surveyed. Three hundred and fifty-one participants were included in the study, of which 61% were male. The main participants were family/general practice (22%), medicine, general (14%), and OB/GYN (12%). Overall, 66% of the participants ranked themselves as IT competent, most of the participants underwent IT self-guided learning, and 65% of the participants always used the system. The results generally reveal positive physicians' perceptions toward the roles of the EHR system on diagnosis quality and safety. There was a statistically significant relationship between user characteristics and the roles of the EHR by enhancing access to care, patient-physician encounter, clinical reasoning, diagnostic testing and consultation, follow-up, and diagnostic safety functionality. The study participants demonstrate positive perceptions of physicians toward the roles of the EHR system in approaching differential diagnosis. Yet, areas of improvement in the design and using EHRs are emphasized.

Partnering with patients and families living with chronic conditions to coproduce diagnostic safety through OurDX: a previsit online engagement tool

OBJECTIVE

Patients and families are key partners in diagnosis, but methods to routinely engage them in diagnostic safety are lacking. Policy mandating patient access to electronic health information presents new opportunities. We tested a new online tool ("OurDX") that was codesigned with patients and families, to determine the types and frequencies of potential safety issues identified by patients/families with chronic health conditions and whether their contributions were integrated into the visit note.

METHODS

Patients/families at 2 US healthcare sites were invited to contribute, through an online previsit survey: (1) visit priorities, (2) recent medical history/symptoms, and (3) potential diagnostic concerns. Two physicians reviewed patient-reported diagnostic concerns to verify and categorize diagnostic safety opportunities (DSOs). We conducted a chart review to determine whether patient contributions were integrated into the note. We used descriptive statistics to report implementation outcomes, verification of DSOs, and chart review findings.

RESULTS

Participants completed OurDX reports in 7075 of 18 129 (39%) eligible pediatric subspecialty visits (site 1), and 460 of 706 (65%) eligible adult primary care visits (site 2). Among patients reporting diagnostic concerns, 63% were verified as probable DSOs. In total, probable DSOs were identified by 7.5% of pediatric and adult patients/families with underlying health conditions, respectively. The most common types of DSOs were patients/families not feeling heard; problems/delays with tests or referrals; and problems/delays with explanation or next steps. In chart review, most clinician notes included all or some patient/family priorities and patient-reported histories.

CONCLUSIONS

OurDX can help engage patients and families living with chronic health conditions in diagnosis. Participating patients/families identified DSOs and most of their OurDX contributions were included in the visit note.

Automated Detection of Patient Harm: Implementation and Prospective Evaluation of a Real-Time Broad-Spectrum Surveillance Application in a Hospital With Limited Resources

OBJECTIVES

This study aimed to prospectively validate an application that automates the detection of broad categories of hospital adverse events (AEs) extracted from a basic hospital information system, and to efficiently mobilize resources to reduce the level of acquired patient harm.

METHODS

Data were collected from an internally designed software, extracting results from 14 triggers indicative of patient harm, querying clinical and administrative databases including all inpatient admissions (n = 8760) from October 2019 to June 2020. Representative samples of the triggered cases were clinically validated using chart review by a consensus expert panel. The positive predictive value (PPV) of each trigger was evaluated, and the detection sensitivity of the surveillance system was estimated relative to incidence ranges in the literature.

RESULTS

The system identified 394 AEs among 946 triggered cases, associated with 291 patients, yielding an overall PPV of 42%. Variability was observed among the trigger PPVs and among the estimated detection sensitivities across the harm categories, the highest being for the healthcare-associated infections. The median length of stay of patients with an AE showed to be significantly higher than the median for the overall patient population.

CONCLUSIONS

This application was able to identify AEs across a broad spectrum of harm categories, in a real-time manner, while reducing the use of resources required by other harm detection methods. Such a system could serve as a promising patient safety tool for AE surveillance, allowing for timely, targeted, and resource-efficient interventions, even for hospitals with limited resources.

Diagnostic delays among COVID-19 patients with a second concurrent diagnosis

BACKGROUND

Little is known about the effect of a new pandemic on diagnostic errors.

OBJECTIVE

We aimed to identify delayed second diagnoses among patients presenting to the emergency department (ED) with COVID-19.

DESIGNS

An observational cohort Study.

SETTINGS AND PARTICIPANTS

Consecutive hospitalized adult patients presenting to the ED of a tertiary referral center with COVID-19 during the Delta and Omicron variant surges. Included patients had evidence of a second diagnosis during their ED stay.

MAIN OUTCOME AND MEASURES

The primary outcome was delayed diagnosis (without documentation or treatment in the ED). Contributing factors were assessed using two logistic regression models.

RESULTS

Among 1249 hospitalized COVID-19 patients, 216 (17%) had evidence of a second diagnosis in the ED. The second diagnosis of 73 patients (34%) was delayed, with a mean (SD) delay of 1.5 (0.8) days. Medical treatment was deferred in 63 patients (86%) and interventional therapy in 26 (36%). The probability of an ED diagnosis was the lowest for Infection-related diagnoses (56%) and highest for surgical-related diagnoses (89%). Evidence for the second diagnosis by physical examination (adjusted odds ratios [AOR] 2.35, 95% confidence interval [CI] 1.20-4.68) or by imaging (AOR 2.10, 95% CI 1.16-3.79) were predictors for ED diagnosis. Low oxygen saturation (AOR 0.38, 95% CI 0.18-0.79) and cough or dyspnea (AOR 0.48, 95% CI 0.25-0.94) in the ED were predictors of a delayed second diagnosis.

A Customized Triggers Program: A Children's Hospital's Experience in Improving Trigger Usability

BACKGROUND

Despite the growth of patient safety programs across the United States, errors and adverse events remain a source of patient harm. Many hospitals rely on retrospective voluntary reporting systems; however, there are opportunities to improve patient safety using novel tools like trigger programs.

METHODS

Children's National Hospital developed a unique pediatric triggers program that offers customized, near real-time reports of potential safety events. Our team defined a measure to quantify clinical utility of triggers, termed "trigger signal," as the percentage of cases that represent true adverse or near-miss events (numerator) per total triggers activated (denominator). Our key driver diagram focused on unifying the program structure, increasing data analytics, promoting organizational awareness, and supporting multidisciplinary end user engagement. Using the model for improvement, we aimed to double overall trigger signal from 8% to 16% and sustain for 12 months.

RESULTS

The trigger signal increased from 8% to 41% and sustained during the coronavirus disease 2019 pandemic. A balancing measure of time to implement a new trigger decreased. Key interventions to increase trigger signal were change in the program structure, increasing stakeholder engagement, and development of self-service reports for end users.

CONCLUSIONS

Children's National Hospital's triggers program highlights successful evolution of an iterative, customized approach to increase clinical utility that hospitals can implement to impact real-time patient care. This triggers program requires an iterative, customized approach rather than a "1-size-fits-all," static paradigm to add a new dimension to current patient safety programs.

Cancer Evaluations During the COVID-19 Pandemic: An Observational Study Using National Veterans Affairs Data

INTRODUCTION

Fewer cancer diagnoses have been made during the COVID-19 pandemic. Pandemic-related delays in cancer diagnosis could occur from limited access to care or patient evaluation delays (e.g., delayed testing after abnormal results). Follow-up of abnormal test results warranting evaluation for cancer was examined before and during the pandemic.

METHODS

Electronic trigger algorithms were applied to the Department of Veterans Affairs electronic health record data to assess follow-up of abnormal test results before (March 10, 2019-March 7, 2020) and during (March 8, 2020-March 6, 2021) the pandemic.

RESULTS

Electronic triggers were applied to 8,021,406 veterans' electronic health records to identify follow-up delays for abnormal results warranting evaluation for 5 cancers: bladder (urinalysis with high-grade hematuria), breast (abnormal mammograms), colorectal (positive fecal occult blood tests/fecal immunochemical tests or results consistent with iron deficiency anemia), liver (elevated alpha-fetoprotein), and lung (chest imaging suggestive of malignancy) cancers. Between prepandemic and pandemic periods, test quantities decreased by 12.6%-27.8%, and proportions of abnormal results lacking follow-up decreased for urinalyses (-0.8%), increased for fecal occult blood tests/fecal immunochemical test (+2.3%) and chest imaging (+1.8%), and remained constant for others. Follow-up times decreased for most tests; however, control charts suggested increased delays at 2 stages: early (pandemic beginning) for urinalyses, mammograms, fecal occult blood tests/fecal immunochemical test, iron deficiency anemia, and chest imaging and late (30-45 weeks into pandemic) for mammograms, fecal occult blood tests/fecal immunochemical test, and iron deficiency anemia.

CONCLUSIONS

Although early pandemic delays in follow-up may have led to reduced cancer rates, the significant decrease in tests performed is likely a large driver of these reductions. Future emergency preparedness efforts should bolster essential follow-up and testing procedures to facilitate timely cancer diagnosis.

A structured approach to EHR surveillance of diagnostic error in acute care: an exploratory analysis of two institutionally-defined case cohorts

OBJECTIVES

To test a structured electronic health record (EHR) case review process to identify diagnostic errors (DE) and diagnostic process failures (DPFs) in acute care.

METHODS

We adapted validated tools (Safer Dx, Diagnostic Error Evaluation Research [DEER] Taxonomy) to assess the diagnostic process during the hospital encounter and categorized 13 postulated e-triggers. We created two test cohorts of all preventable cases (n=28) and an equal number of randomly sampled non-preventable cases (n=28) from 365 adult general medicine patients who expired and underwent our institution's mortality case review process. After excluding patients with a length of stay of more than one month, each case was reviewed by two blinded clinicians trained in our process and by an expert panel. Inter-rater reliability was assessed. We compared the frequency of DE contributing to death in both cohorts, as well as mean DPFs and e-triggers for DE positive and negative cases within each cohort.

RESULTS

Twenty-seven (96.4%) preventable and 24 (85.7%) non-preventable cases underwent our review process. Inter-rater reliability was moderate between individual reviewers (Cohen's kappa 0.41) and substantial with the expert panel (Cohen's kappa 0.74). The frequency of DE contributing to death was significantly higher for the preventable compared to the non-preventable cohort (56% vs. 17%, OR 6.25 [1.68, 23.27], p<0.01). Mean DPFs and e-triggers were significantly and non-significantly higher for DE positive compared to DE negative cases in each cohort, respectively.

CONCLUSIONS

We observed substantial agreement among final consensus and expert panel reviews using our structured EHR case review process. DEs contributing to death associated with DPFs were identified in institutionally designated preventable and non-preventable cases. While e-triggers may be useful for discriminating DE positive from DE negative cases, larger studies are required for validation. Our approach has potential to augment institutional mortality case review processes with respect to DE surveillance.

The e-Autopsy/e-Biopsy: a systematic chart review to increase safety and diagnostic accuracy

Solving diagnostic errors is difficult and progress on preventing those errors has been slow since the 2015 National Academy of Medicine report. There are several methods used to improve diagnostic and other errors including voluntary reporting; malpractice claims; patient complaints; physician surveys, random quality reviews and audits, and peer review data which usually evaluates single cases and not the systems that allowed the error. Additionally, manual review of charts is often labor intensive and reviewer dependent. In 2010 we developed an e-Autopsy/e-Biopsy (eA/eB) methodology to aggregate cases with quality/safety/diagnostic issues, focusing on a specific population of patients and conditions. By performing a hybrid review process (cases are first filtered using administrative data followed by standardized manual chart reviews) we can efficiently identify patterns of medical and diagnostic error leading to opportunities for system improvements that have improved care for future patients. We present a detailed methodology for eA/eB studies and describe results from three successful studies on different diagnoses (ectopic pregnancy, abdominal aortic aneurysms, and advanced colon cancer) that illustrate our eA/eB process and how it reveals insights into creating systems that reduce diagnostic and other errors. The eA/eB process is innovative and transferable to other healthcare organizations and settings to identify trends in diagnostic error and other quality issues resulting in improved systems of care.

Creating a Learning Health System for Improving Diagnostic Safety: Pragmatic Insights from US Health Care Organizations

OBJECTIVE

To identify challenges and pragmatic strategies for improving diagnostic safety at an organizational level using concepts from learning health systems METHODS: We interviewed 32 safety leaders across the USA on how their organizations approach diagnostic safety. Participants were recruited through email and represented geographically diverse academic and non-academic settings. The interview included questions on culture of reporting and learning from diagnostic errors; data gathering and analysis activities; diagnostic training and educational activities; and engagement of clinical leadership, staff, patients, and families in diagnostic safety activities. We conducted an inductive content analysis of interview transcripts and two reviewers coded all data.

RESULTS

Of 32 participants, 12 reported having a specific program to address diagnostic errors. Multiple barriers to implement diagnostic safety activities emerged: serious concerns about psychological safety associated with diagnostic error; lack of infrastructure for measurement, monitoring, and improvement activities related to diagnosis; lack of leadership investment, which was often diverted to competing priorities related to publicly reported measures or other incentives; and lack of dedicated teams to work on diagnostic safety. Participants provided several strategies to overcome barriers including adapting trigger tools to identify safety events, engaging patients in diagnostic safety, and appointing dedicated diagnostic safety champions.

CONCLUSIONS

Several foundational building blocks related to learning health systems could inform organizational efforts to reduce diagnostic error. Promoting an organizational culture specific to diagnostic safety, using science and informatics to improve measurement and analysis, leadership incentives to build institutional capacity to address diagnostic errors, and patient engagement in diagnostic safety activities can enable progress.

Use of e-triggers to identify diagnostic errors in the paediatric ED

BACKGROUND

Diagnostic errors (DxEs) are an understudied source of patient harm in children rarely captured in current adverse event reporting systems. Applying electronic triggers (e-triggers) to electronic health records shows promise in identifying DxEs but has not been used in the emergency department (ED) setting.

OBJECTIVES

To assess the performance of an e-trigger and subsequent manual screening for identifying probable DxEs among children with unplanned admission following a prior ED visit and to compare performance to existing incident reporting systems.

DESIGN/METHODS

Retrospective single-centre cohort study of children ages 0-22 admitted within 14 days of a previous ED visit between 1 January 2018 and 31 December 2019. Subjects were identified by e-trigger, screened to identify cases where index visit and hospital discharge diagnoses were potentially related but pathophysiologically distinct, and then these screened-in cases were reviewed for DxE using the SaferDx Instrument. Cases of DxE identified by e-trigger were cross-referenced against existing institutional incident reporting systems.

RESULTS

An e-trigger identified 1915 unplanned admissions (7.7% of 24 849 total admissions) with a preceding index visit. 453 (23.7%) were screened in and underwent review using SaferDx. 92 cases were classified as likely DxEs, representing 0.4% of all hospital admissions, 4.8% among those selected by e-trigger and 20.3% among those screened in for review. Half of cases were reviewed by two reviewers using SaferDx with substantial inter-rater reliability (Cohen's κ=0.65 (95% CI 0.54 to 0.75)). Six (6.5%) cases had been reported elsewhere: two to the hospital's incident reporting system and five to the ED case review team (one reported to both).

CONCLUSION

An e-trigger coupled with manual screening enriched a cohort of patients at risk for DxEs. Fewer than 10% of DxEs were identified through existing surveillance systems, suggesting that they miss a large proportion of DxEs. Further study is required to identify specific clinical presentations at risk of DxEs.

Medication adverse events in the ambulatory setting: A mixed-methods analysis

DISCLAIMER

In an effort to expedite the publication of articles, AJHP is posting manuscripts online as soon as possible after acceptance. Accepted manuscripts have been peer-reviewed and copyedited, but are posted online before technical formatting and author proofing. These manuscripts are not the final version of record and will be replaced with the final article (formatted per AJHP style and proofed by the authors) at a later time.

PURPOSE

To characterize ambulatory care adverse drug events reported to the Collaborative Healthcare Patient Safety Organization (CHPSO), a network of 400 hospitals across the United States, and identify addressable contributing factors.

METHODS

We abstracted deidentified ambulatory care CHPSO reports compiled from May 2012 to October 2018 that included medication-related adverse events to identify implicated medications and contributing factors. We dual-coded 20% of the sample. We quantitatively calculated co-occurring frequent item sets of contributing factors and then applied a qualitative thematic analysis of co-occurring sets of contributing factors for each drug class using an inductive analytic approach to develop formal themes.

RESULTS

Of 1,244 events in the sample, 208 were medication related. The most commonly implicated medication classes were anticoagulants (n = 97, or 46% of events), antibiotics (n = 24, 11%), hypoglycemics (n = 19, 9%), and opioids (n = 17, 8%). For anticoagulants, timely follow-up on supratherapeutic international normalized ratio (INR) values often occurred before the development of symptoms. Incident reports citing antibiotics often described prescribing errors and failure to review clinical contraindications. Reports citing hypoglycemic drugs often described low blood sugar events due to a lack of patient education or communication. Reports citing opioids often described drug-drug interactions, commonly involving benzodiazepines.

CONCLUSION

Ambulatory care prescribing clinicians and community pharmacists have the potential to mitigate harm related to anticoagulants, antibiotics, hypoglycemics, and opioids. Recommendations include increased follow-up for subtherapeutic INRs, improved medical record integration and chart review for antibiotic prescriptions, enhanced patient education regarding hypoglycemics, and alerts to dissuade coprescription of opioids and benzodiazepines.

Understanding barriers to diagnosis in a rare, genetic disease: Delays and errors in diagnosing schwannomatosis

Diagnosis of rare, genetic diseases is challenging, but conceptual frameworks of the diagnostic process can guide quality improvement initiatives. Using the National Academy of Medicine diagnostic framework, we assessed the extent of, and reasons for diagnostic delays and diagnostic errors in schwannomatosis, a neurogenetic syndrome characterized by nerve sheath tumors and chronic pain. We reviewed the medical records of 97 people with confirmed or probable schwannomatosis seen in two US tertiary care clinics. Time-to-event analysis revealed a median time from first symptom to diagnosis of 16.7 years (95% CI, 7.5-26.0 years) and median time from first medical consultation to diagnosis of 9.8 years (95% CI, 3.5-16.2 years). Factors associated with longer times to diagnosis included initial signs/symptoms that were intermittent, non-specific, or occurred at younger ages (p < 0.05). Thirty-six percent of patients were misdiagnosed; misdiagnoses were of underlying genetic condition (18.6%), pain etiology (16.5%), and nerve sheath tumor presence/pathology (11.3%) (non-mutually exclusive categories). One-fifth (19.6%) of patients had a clear missed opportunity for genetics workup that could have led to an earlier schwannomatosis diagnosis. These results suggest that interventions in clinician education, genetic testing availability, expert review of pathology findings, and automatic triggers for genetics referrals may improve diagnosis of schwannomatosis.

Translating electronic health record-based patient safety algorithms from research to clinical practice at multiple sites

Introduction

Researchers are increasingly developing algorithms that impact patient care, but algorithms must also be implemented in practice to improve quality and safety.

Objective

We worked with clinical operations personnel at two US health systems to implement algorithms to proactively identify patients without timely follow-up of abnormal test results that warrant diagnostic evaluation for colorectal or lung cancer. We summarise the steps involved and lessons learned.

Methods

Twelve sites were involved across two health systems. Implementation involved extensive software documentation, frequent communication with sites and local validation of results. Additionally, we used automated edits of existing code to adapt it to sites’ local contexts.

Results

All sites successfully implemented the algorithms. Automated edits saved sites significant work in direct code modification. Documentation and communication of changes further aided sites in implementation.

Conclusion

Patient safety algorithms developed in research projects were implemented at multiple sites to monitor for missed diagnostic opportunities. Automated algorithm translation procedures can produce more consistent results across sites.

Single-Vendor Electronic Health Record Use Is Associated With Greater Opportunities for Organizational and Clinical Care Improvements

Objective

To compare how hospitals that use single-vendor vs best-of-breed electronic health record (EHR) vendors utilize clinical and organizational evaluation capabilities.

Methods

Data from the 2018 (June 1, 2016, to December 31, 2017) American Hospital Association Information Technology Supplement Survey and Medicare Final Rule Standardizing File were used. Multinomial logistic regression analysis of hospitals (n=1902) was conducted to identify hospital characteristics associated with the use of EHRs for (1) clinical care evaluation capabilities and (2) organizational evaluation capabilities.

Results

Single-vendor EHR hospitals were more likely (relative risk ratio, 3.37; 95% confidence interval, 1.97-5.76) to use EHRs for clinical care and organizational evaluation capabilities. Not-for-profit hospitals were more likely to use EHRs for all organizational evaluation capabilities than government nonfederal hospitals. For-profit hospitals were less likely to use EHRs for organizational or clinical evaluation capabilities than government nonfederal hospitals.

Conclusion

Hospitals using the single-vendor EHR system were more likely to engage in clinical care and organizational evaluation than hospitals using best-of-breed EHR systems.

Monitoring Preventable Adverse Events and Near Misses: Number and Type Identified Differ Depending on Method Used

OBJECTIVES

This study aimed to investigate how many preventable adverse events (PAEs) and near misses are identified through the methods structured record review, Web-based incident reporting (IR), and daily safety briefings, and to distinguish the type of events identified by each method.

METHODS

One year of retrospective data from 2017 were collected from one patient cohort in a 422-bed acute care hospital. Preventable adverse events and near misses were collected from the hospital's existing resources and presented descriptively as number per 1000 patient-days.

RESULTS

The structured record review identified 19.9 PAEs; the IR system, 3.4 PAEs; and daily safety briefings, 5.4 PAEs per 1000 patient-days. The most common PAEs identified by the record review method were drug-related PAEs, pressure ulcers, and hospital-acquired infections. The most common PAEs identified by the IR system and daily safety briefings were fall injury and pressure ulcers, followed by skin/superficial vessel injuries for the IR system and hospital-acquired infections for the daily safety briefings. Incident reporting and daily safety briefings identified 7.8 and 31.9 near misses per 1000 patient-days, respectively. The most common near misses were related to how care is organized.

CONCLUSIONS

The different methods identified different amounts and types of PAEs and near misses. The study supports that health care organizations should adopt multiple methods to get a comprehensive review of the number and type of events occurring in their setting. Daily safety briefings seem to be a particularly suitable method for assessing an organization's inherent security and may foster a nonpunitive culture.

Inviting patients to identify diagnostic concerns through structured evaluation of their online visit notes

BACKGROUND

The 21st Century Cures Act mandates patients' access to their electronic health record (EHR) notes. To our knowledge, no previous work has systematically invited patients to proactively report diagnostic concerns while documenting and tracking their diagnostic experiences through EHR-based clinician note review.

OBJECTIVE

To test if patients can identify concerns about their diagnosis through structured evaluation of their online visit notes.

METHODS

In a large integrated health system, patients aged 18-85 years actively using the patient portal and seen between October 2019 and February 2020 were invited to respond to an online questionnaire if an EHR algorithm detected any recent unexpected return visit following an initial primary care consultation ("at-risk" visit). We developed and tested an instrument (Safer Dx Patient Instrument) to help patients identify concerns related to several dimensions of the diagnostic process based on notes review and recall of recent "at-risk" visits. Additional questions assessed patients' trust in their providers and their general feelings about the visit. The primary outcome was a self-reported diagnostic concern. Multivariate logistic regression tested whether the primary outcome was predicted by instrument variables.

RESULTS

Of 293 566 visits, the algorithm identified 1282 eligible patients, of whom 486 responded. After applying exclusion criteria, 418 patients were included in the analysis. Fifty-one patients (12.2%) identified a diagnostic concern. Patients were more likely to report a concern if they disagreed with statements "the care plan the provider developed for me addressed all my medical concerns" [odds ratio (OR), 2.65; 95% confidence interval [CI], 1.45-4.87) and "I trust the provider that I saw during my visit" (OR, 2.10; 95% CI, 1.19-3.71) and agreed with the statement "I did not have a good feeling about my visit" (OR, 1.48; 95% CI, 1.09-2.01).

CONCLUSION

Patients can identify diagnostic concerns based on a proactive online structured evaluation of visit notes. This surveillance strategy could potentially improve transparency in the diagnostic process.

Checklists to reduce diagnostic error: a systematic review of the literature using a human factors framework

OBJECTIVES

To apply a human factors framework to understand whether checklists reduce clinical diagnostic error have (1) gaps in composition; and (2) components that may be more likely to reduce errors.

DESIGN

Systematic review.

DATA SOURCES

PubMed, EMBASE, Scopus and Web of Science were searched through 15 February 2022.

ELIGIBILITY CRITERIA

Any article that included a clinical checklist aimed at improving the diagnostic process. Checklists were defined as any structured guide intended to elicit additional thinking regarding diagnosis.

DATA EXTRACTION AND SYNTHESIS

Two authors independently reviewed and selected articles based on eligibility criteria. Each extracted unique checklist was independently characterised according to the well-established human factors framework: Systems Engineering Initiative for Patient Safety 2.0 (SEIPS 2.0). If reported, checklist efficacy in reducing diagnostic error (eg, diagnostic accuracy, number of errors or any patient-related outcomes) was outlined. Risk of study bias was independently evaluated using standardised quality assessment tools in accordance with Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

RESULTS

A total of 30 articles containing 25 unique checklists were included. Checklists were characterised within the SEIPS 2.0 framework as follows: Work Systems subcomponents of Tasks (n=13), Persons (n=2) and Internal Environment (n=3); Processes subcomponents of Cognitive (n=20) and Social and Behavioural (n=2); and Outcomes subcomponents of Professional (n=2). Other subcomponents, such as External Environment or Patient outcomes, were not addressed. Fourteen checklists examined effect on diagnostic outcomes: seven demonstrated improvement, six were without improvement and one demonstrated mixed results. Importantly, Tasks-oriented studies more often demonstrated error reduction (n=5/7) than those addressing the Cognitive process (n=4/10).

CONCLUSIONS

Most diagnostic checklists incorporated few human factors components. Checklists addressing the SEIPS 2.0 Tasks subcomponent were more often associated with a reduction in diagnostic errors. Studies examining less explored subcomponents and emphasis on Tasks, rather than the Cognitive subcomponents, may be warranted to prevent diagnostic errors.

Impact of subspecialty consultations on diagnosis in the pediatric intensive care unit

OBJECTIVES

Intensivists and subspecialists often collaborate in diagnosing patients in the pediatric intensive care unit (PICU). Our objectives were to characterize critically ill children for whom subspecialty consultations were requested, describe consultation characteristics, and determine consultations' impact on PICU diagnosis.

METHODS

We performed a retrospective study using chart review in a single tertiary referral PICU including children admitted for acute illness. We collected data on patients with and without subspecialty consultations within the first three days of PICU admission and determined changes in PICU clinicians' diagnostic evaluation or treatment after consultations.

RESULTS

PICU clinicians requested 152 subspecialty consultations for 87 of 101 (86%) patients. Consultations were requested equally for assistance in diagnosis (65%) and treatment (66%). Eighteen of 87 (21%) patients with consultations had a change in diagnosis from PICU admission to discharge, 11 (61%) attributed to subspecialty input. Thirty-nine (45%) patients with consultations had additional imaging and/or laboratory testing and 48 (55%) had medication changes and/or a procedure performed immediately after consultation.

CONCLUSIONS

Subspecialty consultations were requested during a majority of PICU admissions. Consultations can influence the diagnosis and treatment of critically ill children. Future research should investigate PICU interdisciplinary collaborations, which are essential for teamwork in diagnosis.

Diagnostic Error in Pediatrics: A Narrative Review

A priority topic for patient safety research is diagnostic errors. However, despite the significant growth in awareness of their unacceptably high incidence and associated harm, a relative paucity of large, high-quality studies of diagnostic error in pediatrics exists. In this narrative review, we present what is known about the incidence and epidemiology of diagnostic error in pediatrics as well as the established research methods for identifying, evaluating, and reducing diagnostic errors, including their strengths and weaknesses. Additionally, we highlight that pediatric diagnostic error remains an area in need of both innovative research and quality improvement efforts to apply learnings from a rapidly growing evidence base. We propose several key research questions aimed at addressing persistent gaps in the pediatric diagnostic error literature that focus on the foundational knowledge needed to inform effective interventions to reduce the incidence of diagnostic errors and their associated harm. Additional research is needed to better establish the epidemiology of diagnostic error in pediatrics, including identifying high-risk clinical scenarios, patient populations, and groups of diagnoses. A critical need exists for validated measures of both diagnostic errors and diagnostic processes that can be adapted for different clinical settings and standardized for use across varying institutions. Pediatric researchers will need to work collaboratively on large-scale, high-quality studies to accomplish the ultimate goal of reducing diagnostic errors and their associated harm in children by addressing these fundamental gaps in knowledge.

The Trigger Tool Method for Routine Pharmacovigilance: A Retrospective Cohort Study of the Medical Records of Hospitalized Geriatric Patients

OBJECTIVE

The main objective was to assess the feasibility of the trigger tool method for the retrospective detection of adverse drug reactions (ADRs) in the Rennes University Hospital. The secondary objective was to describe the performance of the method in terms of positive predictive values (PPVs) and severity or preventability of ADRs.

METHODS

Using the Rennes University Hospital clinical data warehouse, pharmacovigilance experts performed a retrospective review of a random sample of 30 inpatient hospital medical records per month using the triggers "fall" and "delirium" to identify related ADRs among patients 65 years and older in 2018 in the geriatrics department. Using the Z test, we compared the proportion of medical records with a positive (identified) trigger related to an ADR, which were reviewed within 20 minutes using the reference of 50% reviewed within 20 minutes.

RESULTS

Among the 355 medical records reviewed, 222 had at least 1 trigger and 98 at least 1 related ADR. Among the 222 positive trigger medical records, 99.6% were reviewed in under 20 minutes (P < 0.001). The pharmacovigilance assessment took 3 months. The PPVs reached 53.9% (46.0%-61.7%) for falls and 21.0% (14.3%-27.5%) for delirium. Among the ADRs, 80% were serious and 53% were preventable.

CONCLUSIONS

Given the low PPV of the triggers used and the considerable need for technical and human resources, the trigger tool method cannot be used as a routine tool at the pharmacovigilance center. However, it could be implemented occasionally for specific purposes such as monitoring the impact of risk minimization measures to prevent ADRs.

Characteristics of Disease-Specific and Generic Diagnostic Pitfalls: A Qualitative Study

IMPORTANCE

Progress in understanding and preventing diagnostic errors has been modest. New approaches are needed to help clinicians anticipate and prevent such errors. Delineating recurring diagnostic pitfalls holds potential for conceptual and practical ways for improvement.

OBJECTIVES

To develop the construct and collect examples of "diagnostic pitfalls," defined as clinical situations and scenarios vulnerable to errors that may lead to missed, delayed, or wrong diagnoses.

DESIGN, SETTING, AND PARTICIPANTS

This qualitative study used data from January 1, 2004, to December 31, 2016, from retrospective analysis of diagnosis-related patient safety incident reports, closed malpractice claims, and ambulatory morbidity and mortality conferences, as well as specialty focus groups. Data analyses were conducted between January 1, 2017, and December 31, 2019.

MAIN OUTCOMES AND MEASURES

From each data source, potential diagnostic error cases were identified, and the following information was extracted: erroneous and correct diagnoses, presenting signs and symptoms, and areas of breakdowns in the diagnostic process (using Diagnosis Error Evaluation and Research and Reliable Diagnosis Challenges taxonomies). From this compilation, examples were collected of disease-specific pitfalls; this list was used to conduct a qualitative analysis of emerging themes to derive a generic taxonomy of diagnostic pitfalls.

RESULTS

A total of 836 relevant cases were identified among 4325 patient safety incident reports, 403 closed malpractice claims, 24 ambulatory morbidity and mortality conferences, and 355 focus groups responses. From these, 661 disease-specific diagnostic pitfalls were identified. A qualitative review of these disease-specific pitfalls identified 21 generic diagnostic pitfalls categories, which included mistaking one disease for another disease (eg, aortic dissection is misdiagnosed as acute myocardial infarction), failure to appreciate test result limitations, and atypical disease presentations.

CONCLUSIONS AND RELEVANCE

Recurring types of pitfalls were identified and collected from diagnostic error cases. Clinicians could benefit from knowledge of both disease-specific and generic cross-cutting pitfalls. Study findings can potentially inform educational and quality improvement efforts to anticipate and prevent future errors.

Why Test Results Are Still Getting "Lost" to Follow-up: a Qualitative Study of Implementation Gaps

BACKGROUND

Lack of timely follow-up of abnormal test results is common and has been implicated in missed or delayed diagnosis, resulting in potential for patient harm.

OBJECTIVE

As part of a larger project to implement change strategies to improve follow-up of diagnostic test results, this study sought to identify specifically where implementation gaps exist, as well as possible solutions identified by front-line staff.

DESIGN

We used a semi-structured interview guide to collect qualitative data from Veterans Affairs (VA) facility staff who had experience with test results management and patient safety.

SETTING

Twelve VA facilities across the USA.

PARTICIPANTS

Facility staff members (n = 27), including clinicians, lab and imaging professionals, nursing staff, patient safety professionals, and leadership.

APPROACH

We conducted a content analysis of interview transcripts to identify perceived barriers and high-risk areas for effective test result management, as well as recommendations for improvement.

RESULTS

We identified seven themes to guide further development of interventions to improve test result follow-up. Themes related to trainees, incidental findings, tracking systems for electronic health record notifications, outdated contact information, referrals, backup or covering providers, and responsibility for test results pending at discharge. Participants provided recommendations for improvement within each theme.

CONCLUSIONS

Perceived barriers and recommendations for improving test result follow-up often reflected previously known problems and their corresponding solutions, which have not been consistently implemented in practice. Better policy solutions and improvement methods, such as quality improvement collaboratives, may bridge the implementation gaps between knowledge and practice.

A Research Agenda for Diagnostic Excellence in Critical Care Medicine

Diagnosing critically ill patients in the intensive care unit is difficult. As a result, diagnostic errors in the intensive care unit are common and have been shown to cause harm. Research to improve diagnosis in critical care medicine has accelerated in past years. However, much work remains to fully elucidate the diagnostic process in critical care. To achieve diagnostic excellence, interdisciplinary research is needed, adopting a balanced strategy of continued biomedical discovery while addressing the complex care delivery systems underpinning the diagnosis of critical illness.

Harnessing the electronic health care record to optimise patient safety in primary care: a framework for evaluating “electronic safety netting” tools (Preprint)

The management of diagnostic uncertainty is part of every primary care physician’s role. e–Safety-netting tools help health care professionals to manage diagnostic uncertainty. Using software in addition to verbal or paper based safety-netting methods could make diagnostic delays and errors less likely. There are an increasing number of software products that have been identified as e–safety-netting tools, particularly since the start of the COVID-19 pandemic. e–Safety-netting tools can have a variety of functions, such as sending clinician alerts, facilitating administrative tasking, providing decision support, and sending reminder text messages to patients. However, these tools have not been evaluated by using robust research designs for patient safety interventions. We present an emergent framework of criteria for effective e–safety-netting tools that can be used to support the development of software. The framework is based on validated frameworks for electronic health record development and patient safety. There are currently no tools available that meet all of the criteria in the framework. We hope that the framework will stimulate clinical and public conversations about e–safety-netting tools. In the future, a validated framework would drive audits and improvements. We outline key areas for future research both in primary care and within integrated care systems.

Learning from Missed Opportunities Through Reflective Practice

Identification of diagnostic errors is difficult but is not alone sufficient for performance improvement. Instead, cases must be reflected on to identify ways to improve decision-making in the future. There are many tools and modalities to retrospectively reflect on action to study medical decisions and outcomes and improve future performance. Reflection in action-in which diagnostic decisions are considered in real-time-may also improve medical decision-making especially through strategies such as structured reflection. Ongoing regular feedback can normalize the discussion about improving decision-making, enable reflective practice, and improve decision making.

Automated identification of diagnostic labelling errors in medicine

Objectives

Identification of diagnostic error is complex and mostly relies on expert ratings, a severely limited procedure. We developed a system that allows to automatically identify diagnostic labelling error from diagnoses coded according to the international classification of diseases (ICD), often available as routine health care data.

Methods

The system developed (index test) was validated against rater based classifications taken from three previous studies of diagnostic labeling error (reference standard). The system compares pairs of diagnoses through calculation of their distance within the ICD taxonomy. Calculation is based on four different algorithms. To assess the concordance between index test and reference standard, we calculated the area under the receiver operating characteristics curve (AUROC) and corresponding confidence intervals. Analysis were conducted overall and separately per algorithm and type of available dataset.

Results

Diagnoses of 1,127 cases were analyzed. Raters previously classified 24.58% of cases as diagnostic labelling errors (ranging from 12.3 to 87.2% in the three datasets). AUROC ranged between 0.821 and 0.837 overall, depending on the algorithm used to calculate the index test (95% CIs ranging from 0.8 to 0.86). Analyzed per type of dataset separately, the highest AUROC was 0.924 (95% CI 0.887–0.962).

Conclusions

The trigger system to automatically identify diagnostic labeling error from routine health care data performs excellent, and is unaffected by the reference standards’ limitations. It is however only applicable to cases with pairs of diagnoses, of which one must be more accurate or otherwise superior than the other, reflecting a prevalent definition of a diagnostic labeling error.

Filling a gap in safety metrics: development of a patient-centred framework to identify and categorise patient-reported breakdowns related to the diagnostic process in ambulatory care

BACKGROUND

Patients and families are important contributors to the diagnostic team, but their perspectives are not reflected in current diagnostic measures. Patients/families can identify some breakdowns in the diagnostic process beyond the clinician's view. We aimed to develop a framework with patients/families to help organisations identify and categorise patient-reported diagnostic process-related breakdowns (PRDBs) to inform organisational learning.

METHOD

A multi-stakeholder advisory group including patients, families, clinicians, and experts in diagnostic error, patient engagement and safety, and user-centred design, co-developed a framework for PRDBs in ambulatory care. We tested the framework using standard qualitative analysis methods with two physicians and one patient coder, analysing 2165 patient-reported ambulatory errors in two large surveys representing 25 425 US respondents. We tested intercoder reliability of breakdown categorisation using the Gwet's AC1 and Cohen's kappa statistic. We considered agreement coefficients 0.61-0.8=good agreement and 0.81-1.00=excellent agreement.

RESULTS

The framework describes 7 patient-reported breakdown categories (with 40 subcategories), 19 patient-identified contributing factors and 11 potential patient-reported impacts. Patients identified breakdowns in each step of the diagnostic process, including missing or inaccurate main concerns and symptoms; missing/outdated test results; and communication breakdowns such as not feeling heard or misalignment between patient and provider about symptoms, events, or their significance. The frequency of PRDBs was 6.4% in one dataset and 6.9% in the other. Intercoder reliability showed good-to-excellent reliability in each dataset: AC1 0.89 (95% CI 0.89 to 0.90) to 0.96 (95% CI 0.95 to 0.97); kappa 0.64 (95% CI 0.62, to 0.66) to 0.85 (95% CI 0.83 to 0.88).

CONCLUSIONS

The PRDB framework, developed in partnership with patients/families, can help organisations identify and reliably categorise PRDBs, including some that are invisible to clinicians; guide interventions to engage patients and families as diagnostic partners; and inform whole organisational learning.

Validation of an electronic trigger to measure missed diagnosis of stroke in emergency departments

OBJECTIVE

Diagnostic errors are major contributors to preventable patient harm. We validated the use of an electronic health record (EHR)-based trigger (e-trigger) to measure missed opportunities in stroke diagnosis in emergency departments (EDs).

METHODS

Using two frameworks, the Safer Dx Trigger Tools Framework and the Symptom-disease Pair Analysis of Diagnostic Error Framework, we applied a symptom-disease pair-based e-trigger to identify patients hospitalized for stroke who, in the preceding 30 days, were discharged from the ED with benign headache or dizziness diagnoses. The algorithm was applied to Veteran Affairs National Corporate Data Warehouse on patients seen between 1/1/2016 and 12/31/2017. Trained reviewers evaluated medical records for presence/absence of missed opportunities in stroke diagnosis and stroke-related red-flags, risk factors, neurological examination, and clinical interventions. Reviewers also estimated quality of clinical documentation at the index ED visit.

RESULTS

We applied the e-trigger to 7,752,326 unique patients and identified 46,931 stroke-related admissions, of which 398 records were flagged as trigger-positive and reviewed. Of these, 124 had missed opportunities (positive predictive value for "missed" = 31.2%), 93 (23.4%) had no missed opportunity (non-missed), 162 (40.7%) were miscoded, and 19 (4.7%) were inconclusive. Reviewer agreement was high (87.3%, Cohen's kappa = 0.81). Compared to the non-missed group, the missed group had more stroke risk factors (mean 3.2 vs 2.6), red flags (mean 0.5 vs 0.2), and a higher rate of inadequate documentation (66.9% vs 28.0%).

CONCLUSION

In a large national EHR repository, a symptom-disease pair-based e-trigger identified missed diagnoses of stroke with a modest positive predictive value, underscoring the need for chart review validation procedures to identify diagnostic errors in large data sets.