Tracking Progress in Improving Diagnosis: A Framework for Defining Undesirable Diagnostic Events

Diagnostic quality model (DQM): an integrated framework for the assessment of diagnostic quality when using AI/ML

BACKGROUND

Laboratory medicine has reached the era where promises of artificial intelligence and machine learning (AI/ML) seem palpable. Currently, the primary responsibility for risk-benefit assessment in clinical practice resides with the medical director. Unfortunately, there is no tool or concept that enables diagnostic quality assessment for the various potential AI/ML applications. Specifically, we noted that an operational definition of laboratory diagnostic quality - for the specific purpose of assessing AI/ML improvements - is currently missing.

METHODS

A session at the 3rd Strategic Conference of the European Federation of Laboratory Medicine in 2022 on "AI in the Laboratory of the Future" prompted an expert roundtable discussion. Here we present a conceptual diagnostic quality framework for the specific purpose of assessing AI/ML implementations.

RESULTS

The presented framework is termed diagnostic quality model (DQM) and distinguishes AI/ML improvements at the test, procedure, laboratory, or healthcare ecosystem level. The operational definition illustrates the nested relationship among these levels. The model can help to define relevant objectives for implementation and how levels come together to form coherent diagnostics. The affected levels are referred to as scope and we provide a rubric to quantify AI/ML improvements while complying with existing, mandated regulatory standards. We present 4 relevant clinical scenarios including multi-modal diagnostics and compare the model to existing quality management systems.

CONCLUSIONS

A diagnostic quality model is essential to navigate the complexities of clinical AI/ML implementations. The presented diagnostic quality framework can help to specify and communicate the key implications of AI/ML solutions in laboratory diagnostics.

Identifying and prioritizing educational content from a malpractice claims database for clinical reasoning education in the vocational training of general practitioners

Diagnostic reasoning is an important topic in General Practitioners' (GPs) vocational training. Interestingly, research has paid little attention to the content of the cases used in clinical reasoning education. Malpractice claims of diagnostic errors represent cases that impact patients and that reflect potential knowledge gaps and contextual factors. With this study, we aimed to identify and prioritize educational content from a malpractice claims database in order to improve clinical reasoning education in GP training. With input from various experts in clinical reasoning and diagnostic error, we defined five priority criteria that reflect educational relevance. Fifty unique medical conditions from a malpractice claims database were scored on those priority criteria by stakeholders in clinical reasoning education in 2021. Subsequently, we calculated the mean total priority score for each condition. Mean total priority score (min 5-max 25) for all fifty diagnoses was 17,11 with a range from 13,89 to 19,61. We identified and described the fifteen highest scoring diseases (with priority scores ranging from 18,17 to 19,61). The prioritized conditions involved complex common (e.g., cardiovascular diseases, renal insufficiency and cancer), complex rare (e.g., endocarditis, ectopic pregnancy, testicular torsion) and more straightforward common conditions (e.g., tendon rupture/injury, eye infection). The claim cases often demonstrated atypical presentations or complex contextual factors. Including those malpractice cases in GP vocational training could enrich the illness scripts of diseases that are at high risk of errors, which may reduce diagnostic error and related patient harm.

Diagnosis Code and Health Care Utilization Patterns Associated With Diagnostic Uncertainty

BACKGROUND AND OBJECTIVES

Diagnostic uncertainty is challenging to identify and study in clinical practice. This study compares differences in diagnosis code and health care utilization between a unique cohort of hospitalized children with uncertain diagnoses (UD) and matched controls.

PATIENTS AND METHODS

This case-control study was conducted at Cincinnati Children's Hospital Medical Center. Cases were defined as patients admitted to the pediatric hospital medicine service and having UDs during their hospitalization. Control patients were matched on age strata, biological sex, and time of year. Outcomes included type of diagnosis codes used (ie, disease- or nondisease-based) and change in code from admission to discharge. Differences in diagnosis codes were evaluated using conditional logistic regression. Health care utilization outcomes included hospital length of stay (LOS), hospital transfer, consulting service utilization, rapid response team activations, escalation to intensive care, and 30-day health care reutilization. Differences in health care utilization were assessed using bivariate statistics.

RESULTS

Our final cohort included 240 UD cases and 911 matched controls. Compared with matched controls, UD cases were 8 times more likely to receive a nondisease-based diagnosis code (odds ratio [OR], 8.0; 95% confidence interval [CI], 5.7-11.2) and 2.5 times more likely to have a change in their primary International Classification of Disease, 10th revision, diagnosis code between admission and discharge (OR, 2.5; 95% CI, 1.9-3.4). UD cases had a longer average LOS and higher transfer rates to our main hospital campus, consulting service use, and 30-day readmission rates.

CONCLUSIONS

Hospitalized children with UDs have meaningfully different patterns of diagnosis code use and increased health care utilization compared with matched controls.

Defining Diagnostic Error: A Scoping Review to Assess the Impact of the National Academies' Report Improving Diagnosis in Health Care

BACKGROUND

Standards for accurate and timely diagnosis are ill-defined. In 2015, the National Academies of Science, Engineering, and Medicine (NASEM) committee published a landmark report, Improving Diagnosis in Health Care , and proposed a new definition of diagnostic error, "the failure to ( a ) establish an accurate and timely explanation of the patient's health problem(s) or ( b ) communicate that explanation to the patient."

OBJECTIVE

This study aimed to explore how researchers operationalize the NASEM's definition of diagnostic error with relevance to accuracy, timeliness, and/or communication in peer-reviewed published literature.

METHODS

Using the Arskey and O'Malley's framework framework, we identified published literature from October 2015 to February 2021 using Medline and Google Scholar. We also conducted subject matter expert interviews with researchers.

RESULTS

Of 34 studies identified, 16 were analyzed and abstracted to determine how diagnostic error was operationalized and measured. Studies were grouped by theme: epidemiology, patient focus, measurement/surveillance, and clinician focus. Nine studies indicated using the NASEM definition. Of those, 5 studies also operationalized with existing definitions proposed before the NASEM report. Four studies operationalized the components of the NASEM definition and did not cite existing definitions. Three studies operationalized error using existing definitions only. Subject matter experts indicated that the NASEM definition functions as foundation for researchers to conceptualize diagnostic error.

CONCLUSIONS

The NASEM report produced a common understanding of diagnostic error that includes accuracy, timeliness, and communication. In recent peer-reviewed literature, most researchers continue to use pre-NASEM report definitions to operationalize accuracy and timeliness. The report catalyzed the use of patient-centered concepts in the definition, resulting in emerging studies focused on examining errors related to communicating diagnosis to patients.

Diagnostic Error in Pediatrics: A Narrative Review

A priority topic for patient safety research is diagnostic errors. However, despite the significant growth in awareness of their unacceptably high incidence and associated harm, a relative paucity of large, high-quality studies of diagnostic error in pediatrics exists. In this narrative review, we present what is known about the incidence and epidemiology of diagnostic error in pediatrics as well as the established research methods for identifying, evaluating, and reducing diagnostic errors, including their strengths and weaknesses. Additionally, we highlight that pediatric diagnostic error remains an area in need of both innovative research and quality improvement efforts to apply learnings from a rapidly growing evidence base. We propose several key research questions aimed at addressing persistent gaps in the pediatric diagnostic error literature that focus on the foundational knowledge needed to inform effective interventions to reduce the incidence of diagnostic errors and their associated harm. Additional research is needed to better establish the epidemiology of diagnostic error in pediatrics, including identifying high-risk clinical scenarios, patient populations, and groups of diagnoses. A critical need exists for validated measures of both diagnostic errors and diagnostic processes that can be adapted for different clinical settings and standardized for use across varying institutions. Pediatric researchers will need to work collaboratively on large-scale, high-quality studies to accomplish the ultimate goal of reducing diagnostic errors and their associated harm in children by addressing these fundamental gaps in knowledge.

Confidence-competence alignment and the role of self-confidence in medical education: A conceptual review

CONTEXT

There have been significant advances in competency-based medical education (CBME) within health professions education. While most of the efforts have focused on competency, less attention has been paid to the role of confidence as a factor in preparing for practice. This paper seeks to address this deficit by exploring the role of confidence and the calibration of confidence with regard to competence.

METHODS

This paper presents a conceptual review of confidence and the calibration of confidence in different medical education contexts. Building from an initial literature review, the authors engaged in iterative discussions exploring divergent and convergent perspectives, which were then supplemented with targeted literature reviews. Finally, a stakeholder consultation was conducted to situate and validate the provisional findings.

RESULTS

A series of axioms were developed to guide perceptions and responses to different states of confidence in health professionals: (a) confidence can shape how we act and is optimised when it closely corresponds to reality; (b) self-confidence is task-specific, but also inextricably influenced by the individual self-conceptualisation, the surrounding system and society; (c) confidence is shaped by many external factors and the context of the situation; (d) confidence must be considered in conjunction with competence and (e) the confidence-competence ratio (CCR) changes over time. It is important to track learners' CCRs and work with them to maintain balance.

CONCLUSION

Confidence is expressed in different ways and is shaped by a variety of modifiers. While CBME primarily focuses on competency, proportional confidence is an integral component in ensuring safe and professional practice. As such, it is important to consider both confidence and competence, as well as their relationship in CBME. The CCR can serve as a key construct in developing mindful and capable health professionals. Future research should evaluate strategies for assessing CCR, identify best practices for teaching confidence and guiding self-calibration of CCR and explore the role of CCR in continuing professional development for individuals and teams.

Thresholds, rules and defensive strategies: how physicians learn from their prior diagnosis-related experiences

Background Health professionals are encouraged to learn from their errors. Determining how primary care physicians (PCPs) react to a case, in which their original diagnosis differed from the final outcome, could provide new insights on how they learn from experiences. We explored how PCPs altered their diagnostic evaluation of future patients after cases where the originally assumed diagnosis turned out to be wrong. Methods We asked German PCPs to complete an online survey where they described how the patient concerned originally presented, the subsequent course of events and whether they would change their diagnostic work-up of future patients. Qualitative methods were used to analyze narrative text obtained by this survey. Results A total of 29 PCPs submitted cases, most of which were ultimately found to be more severe than originally assumed. PCPs (n = 27) reflected on changes to their subsequent clinical decisions in the form of general maxims (n = 20) or more specific rules (n = 11). Most changes would have resulted in a lower threshold for investigations, referral and/or a more extensive collection of diagnostic information. PCPs decided not only to listen more often to their intuition (gut feelings), but to also practice more analytical reasoning. Participants felt the need for change of practice even if no clinical standards had been violated in the diagnosis of that case. Some decided to resort to defensive strategies in the future. Conclusions We describe mechanisms by which physicians calibrate their decision thresholds, as well as their cognitive mode (intuitive vs. analytical). PCPs reported the need for change in clinical practice despite the absence of error in some cases.

Development of a rubric for assessing delayed diagnosis of appendicitis, diabetic ketoacidosis and sepsis

OBJECTIVES

Using case review to determine whether a patient experienced a delayed diagnosis is challenging. Measurement would be more accurate if case reviewers had access to multi-expert consensus on grading the likelihood of delayed diagnosis. Our objective was to use expert consensus to create a guide for objectively grading the likelihood of delayed diagnosis of appendicitis, new-onset diabetic ketoacidosis (DKA), and sepsis.

METHODS

Case vignettes were constructed for each condition. In each vignette, a patient has the condition and had a previous emergency department (ED) visit within 7 days. Condition-specific multi-specialty expert Delphi panels reviewed the case vignettes and graded the likelihood of a delayed diagnosis on a five-point scale. Delayed diagnosis was defined as the condition being present during the previous ED visit. Consensus was defined as ≥75% agreement. In each Delphi round, panelists were given the scores from the previous round and asked to rescore. A case scoring guide was created from the consensus scores.

RESULTS

Eighteen expert panelists participated. Consensus was achieved within three Delphi rounds for all appendicitis and sepsis vignettes. We reached consensus on 23/30 (77%) DKA vignettes. A case review guide was created from the consensus scores.

CONCLUSIONS

Multi-specialty expert reviewers can agree on the likelihood of a delayed diagnosis for cases of appendicitis and sepsis, and for most cases of DKA. We created a guide that can be used by researchers and quality improvement specialists to allow for objective case review to determine when delayed diagnoses have occurred for appendicitis, DKA, and sepsis.

Measuring and Improving Diagnostic Safety in Primary Care: Addressing the "Twin" Pandemics of Diagnostic Error and Clinician Burnout

Diagnostic errors are a source of unacceptable harm in health care. However, improvement efforts have been hampered by the lack of valid measures reflecting the quality of the diagnostic process. At the same time, it has become apparent that the healthcare work system, particularly in primary care, is chaotic and stressful, leading to clinician burnout and patient harm. We propose a new construct that health systems and researchers can use to measure the quality and safety of the diagnostic process that is sensitive to the context of the health care work system. This model focuses on three measurable practices: considering "don't miss" diagnoses, looking for red flags, and ensuring that clinicians avoid common diagnostic pitfalls. We believe that the performance of clinicians with respect to these factors is sensitive to the health care work system, allowing for context-dependent measurement and improvement of the diagnostic process. Such process measures will enable more rapid improvements rather than exclusively measuring outcomes related to "correct" or "incorrect" diagnoses.

Controversies in diagnosis: contemporary debates in the diagnostic safety literature

Since the 2015 publication of the National Academy of Medicine's (NAM) Improving Diagnosis in Health Care (Improving Diagnosis in Health Care. In: Balogh EP, Miller BT, Ball JR, editors. Improving Diagnosis in Health Care. Washington (DC): National Academies Press, 2015.), literature in diagnostic safety has grown rapidly. This update was presented at the annual international meeting of the Society to Improve Diagnosis in Medicine (SIDM). We focused our literature search on articles published between 2016 and 2018 using keywords in Pubmed and the Agency for Healthcare Research and Quality (AHRQ)'s Patient Safety Network's running bibliography of diagnostic error literature (Diagnostic Errors Patient Safety Network: Agency for Healthcare Research and Quality; Available from: https://psnet.ahrq.gov/search?topic=Diagnostic-Errors&f_topicIDs=407). Three key topics emerged from our review of recent abstracts in diagnostic safety. First, definitions of diagnostic error and related concepts are evolving since the NAM's report. Second, medical educators are grappling with new approaches to teaching clinical reasoning and diagnosis. Finally, the potential of artificial intelligence (AI) to advance diagnostic excellence is coming to fruition. Here we present contemporary debates around these three topics in a pro/con format.

Regret among primary care physicians: a survey of diagnostic decisions

BACKGROUND

Experienced and anticipated regret influence physicians' decision-making. In medicine, diagnostic decisions and diagnostic errors can have a severe impact on both patients and physicians. Little empirical research exists on regret experienced by physicians when they make diagnostic decisions in primary care that later prove inappropriate or incorrect. The aim of this study was to explore the experience of regret following diagnostic decisions in primary care.

METHODS

In this qualitative study, we used an online questionnaire on a sample of German primary care physicians. We asked participants to report on cases in which the final diagnosis differed from their original opinion, and in which treatment was at the very least delayed, possibly resulting in harm to the patient. We asked about original and final diagnoses, illness trajectories, and the reactions of other physicians, patients and relatives. We used thematic analysis to assess the data, supported by MAXQDA 11 and Microsoft Excel 2016.

RESULTS

29 GPs described one case each (14 female/15 male patients, aged 1.5-80 years, response rate < 1%). In 26 of 29 cases, the final diagnosis was more serious than the original diagnosis. In two cases, the diagnoses were equally serious, and in one case less serious. Clinical trajectories and the reactions of patients and relatives differed widely. Although only one third of cases involved preventable harm to patients, the vast majority (27 of 29) of physicians expressed deep feelings of regret.

CONCLUSION

Even if harm to patients is unavoidable, regret following diagnostic decisions can be devastating for clinicians, making them 'second victims'. Procedures and tools are needed to analyse cases involving undesirable diagnostic events, so that 'true' diagnostic errors, in which harm could have been prevented, can be distinguished from others. Further studies should also explore how physicians can be supported in dealing with such events in order to prevent them from practicing defensive medicine.

Impact on 30-d readmissions for cirrhotic patients with ascites after an educational intervention: A pilot study

BACKGROUND

A low proportion of patients admitted to hospital with cirrhosis receive quality care with timely paracentesis an important target for improvement. We hypothesized that a medical educational intervention, delivered to medical residents caring for patients with cirrhosis, would improve quality of care.

AIM

To determine if an educational intervention can improve quality of care in cirrhotic patients admitted to hospital with ascites.

METHODS

We performed a pilot prospective cohort study with time-based randomization over six months at a large teaching hospital. Residents rotating on hospital medicine teams received an educational intervention while residents rotating on hospital medicine teams on alternate months comprised the control group. The primary outcome was provision of quality care- defined as adherence to all quality-based indicators derived from evidence-based practice guidelines- in admissions for patients with cirrhosis and ascites. Patient clinical outcomes- including length of hospital stay (LOS); 30-d readmission; in-hospital mortality and overall mortality- and resident educational outcomes were also evaluated.

RESULTS

Eighty-five admissions (60 unique patients) met inclusion criteria over the study period-46 admissions in the intervention group and 39 admissions in the control group. Thirty-seven admissions were female patients, and 44 admissions were for alcoholic liver disease. Mean model for end-stage liver disease (MELD)-Na score at admission was 25.8. Forty-seven (55.3%) admissions received quality care. There was no difference in the provision of quality care (56.41% vs 54.35%, P = 0.9) between the two groups. 30-d readmission was lower in the intervention group (35% vs 52.78%, P = 0.1) and after correction for age, gender and MELD-Na score [RR = 0.62 (0.39, 1.00), P = 0.05]. No significant differences were seen for LOS, complications, in-hospital mortality or overall mortality between the two groups. Resident medical knowledge and self-efficacy with paracentesis improved after the educational intervention.

CONCLUSION

Medical education has the potential to improve clinical outcomes in patients admitted to hospital with cirrhosis and ascites.

Determining qualitative effect size ratings using a likelihood ratio scatter matrix in diagnostic test accuracy systematic reviews

Background Diagnostic test accuracy (DTA) systematic reviews (SRs) characterize a test's potential for diagnostic quality and safety. However, interpreting DTA measures in the context of SRs is challenging. Further, some evidence grading methods (e.g. Centers for Disease Control and Prevention, Division of Laboratory Systems Laboratory Medicine Best Practices method) require determination of qualitative effect size ratings as a contributor to practice recommendations. This paper describes a recently developed effect size rating approach for assessing a DTA evidence base. Methods A likelihood ratio scatter matrix will plot positive and negative likelihood ratio pairings for DTA studies. Pairings are graphed as single point estimates with confidence intervals, positioned in one of four quadrants derived from established thresholds for test clinical validity. These quadrants support defensible judgments on "substantial", "moderate", or "minimal" effect size ratings for each plotted study. The approach is flexible in relation to a priori determinations of the relative clinical importance of false positive and false negative test results. Results and conclusions This qualitative effect size rating approach was operationalized in a recent SR that assessed effectiveness of test practices for the diagnosis of Clostridium difficile. Relevance of this approach to other methods of grading evidence, and efforts to measure diagnostic quality and safety are described. Limitations of the approach arise from understanding that a diagnostic test is not an isolated element in the diagnostic process, but provides information in clinical context towards diagnostic quality and safety.

Application of electronic trigger tools to identify targets for improving diagnostic safety

Progress in reducing diagnostic errors remains slow partly due to poorly defined methods to identify errors, high-risk situations, and adverse events. Electronic trigger (e-trigger) tools, which mine vast amounts of patient data to identify signals indicative of a likely error or adverse event, offer a promising method to efficiently identify errors. The increasing amounts of longitudinal electronic data and maturing data warehousing techniques and infrastructure offer an unprecedented opportunity to implement new types of e-trigger tools that use algorithms to identify risks and events related to the diagnostic process. We present a knowledge discovery framework, the Safer Dx Trigger Tools Framework, that enables health systems to develop and implement e-trigger tools to identify and measure diagnostic errors using comprehensive electronic health record (EHR) data. Safer Dx e-trigger tools detect potential diagnostic events, allowing health systems to monitor event rates, study contributory factors and identify targets for improving diagnostic safety. In addition to promoting organisational learning, some e-triggers can monitor data prospectively and help identify patients at high-risk for a future adverse event, enabling clinicians, patients or safety personnel to take preventive actions proactively. Successful application of electronic algorithms requires health systems to invest in clinical informaticists, information technology professionals, patient safety professionals and clinicians, all of who work closely together to overcome development and implementation challenges. We outline key future research, including advances in natural language processing and machine learning, needed to improve effectiveness of e-triggers. Integrating diagnostic safety e-triggers in institutional patient safety strategies can accelerate progress in reducing preventable harm from diagnostic errors.

Diagnostic Discordance, Health Information Exchange, and Inter-Hospital Transfer Outcomes: a Population Study

BACKGROUND

Studying diagnostic error at the population level requires an understanding of how diagnoses change over time.

OBJECTIVE

To use inter-hospital transfers to examine the frequency and impact of changes in diagnosis on patient risk, and whether health information exchange can improve patient safety by enhancing diagnostic accuracy.

DESIGN

Diagnosis coding before and after hospital transfer was merged with responses from the American Hospital Association Annual Survey for a cohort of patients transferred between hospitals to identify predictors of mortality.

PARTICIPANTS

Patients (180,337) 18 years or older transferred between 473 acute care hospitals from NY, FL, IA, UT, and VT from 2011 to 2013.

MAIN MEASURES

We identified discordant Elixhauser comorbidities before and after transfer to determine the frequency and developed a weighted score of diagnostic discordance to predict mortality. This was included in a multivariate model with inpatient mortality as the dependent variable. We investigated whether health information exchange (HIE) functionality adoption as reported by hospitals improved diagnostic discordance and inpatient mortality.

KEY RESULTS

Discordance in diagnoses occurred in 85.5% of all patients. Seventy-three percent of patients gained a new diagnosis following transfer while 47% of patients lost a diagnosis. Diagnostic discordance was associated with increased adjusted inpatient mortality (OR 1.11 95% CI 1.10-1.11, p < 0.001) and allowed for improved mortality prediction. Bilateral hospital HIE participation was associated with reduced diagnostic discordance index (3.69 vs. 1.87%, p < 0.001) and decreased inpatient mortality (OR 0.88, 95% CI 0.89-0.99, p < 0.001).

CONCLUSIONS

Diagnostic discordance commonly occurred during inter-hospital transfers and was associated with increased inpatient mortality. Health information exchange adoption was associated with decreased discordance and improved patient outcomes.