Creating a Learning Health System for Improving Diagnostic Safety: Pragmatic Insights from US Health Care Organizations

Diagnostic Safety: Needs Assessment and Informed Curriculum at an Academic Children's Hospital

Background

Diagnostic excellence is central to healthcare quality and safety. Prior literature identified a lack of psychological safety and time as barriers to diagnostic reasoning education. We performed a needs assessment to inform the development of diagnostic safety education.

Methods

To evaluate existing educational programming and identify opportunities for content delivery, surveys were emailed to 155 interprofessional educational leaders and 627 clinicians at our hospital. Educational leaders and learners were invited to participate in focus groups to further explore beliefs, perceptions, and recommendations about diagnostic reasoning. The study team analyzed data using directed content analysis to identify themes.

Results

Of the 57 education leaders who responded to our survey, only 2 (5%) reported having formal training on diagnostic reasoning in their respective departments. The learner survey had a response rate of 47% (293/627). Learners expressed discomfort discussing diagnostic uncertainty and preferred case-based discussions and bedside learning as avenues for learning about the topic. Focus groups, including 7 educators and 16 learners, identified the following as necessary precursors to effective teaching about diagnostic safety: (1) faculty development, (2) institutional culture change, and (3) improved reporting of missed diagnoses. Participants preferred mandatory sessions integrated into existing educational programs.

Conclusions

Our needs assessment identified a broad interest in education regarding medical diagnosis and potential barriers to implementation. Respondents highlighted the need to develop communication skills regarding diagnostic errors and uncertainty across professions and care areas. Study findings informed a pilot diagnostic reasoning curriculum for faculty and trainees.

Evaluation of Measure Dx, a Resource to Accelerate Diagnostic Safety Learning and Improvement

BACKGROUND

Several strategies have been developed to detect diagnostic errors for organizational learning and improvement. However, few health care organizations (HCOs) have integrated these strategies into routine operations. To address this gap, the Agency for Healthcare Research and Quality released "Measure Dx: A Resource To Identify, Analyze, and Learn From Diagnostic Safety Events" in 2022.

OBJECTIVE

We conducted an evaluation of Measure Dx to measure feasibility of implementation and effects on short-term and intermediate outcomes related to diagnostic safety.

DESIGN

Prospective observational study.

PARTICIPANTS

Teams from 11 HCOs, primarily academic medical centers.

INTERVENTIONS

Participants were asked to use Measure Dx over approximately 6 months and attend monthly virtual learning collaborative sessions to share and discuss approaches to measuring diagnostic safety.

MAIN MEASURES

Descriptive outcomes were gathered at the HCO level and included uptake of different case-finding strategies and the number of cases reviewed and confirmed to have diagnostic safety improvement opportunities. We collected information on organizational practices related to diagnostic safety at each HCO at baseline and at the conclusion of the project.

KEY RESULTS

The 11 HCOs completed all requirements for the evaluation. Each of the four diagnostic safety case finding strategies outlined in Measure Dx were used by at least three HCOs. Across the cohort, participants reviewed 703 cases using a standardized data collection instrument. Of those cases, 224 (31.8%) were identified as diagnostic safety events with improvement opportunities. Unexpectedly, self-ratings on the checklist assessment declined for several organizations.

CONCLUSIONS

Use of Measure Dx can help accelerate implementation of systematic approaches to diagnostic error measurement and learning across a variety of HCOs, while potentially enabling HCOs to identify opportunities to improve diagnostic safety practices.

Patient-Clinician Diagnostic Concordance upon Hospital Admission

OBJECTIVES

 This study aimed to pilot an application-based patient diagnostic questionnaire (PDQ) and assess the concordance of the admission diagnosis reported by the patient and entered by the clinician.

METHODS

 Eligible patients completed the PDQ assessing patients' understanding of and confidence in the diagnosis 24 hours into hospitalization either independently or with assistance. Demographic data, the hospital principal problem upon admission, and International Classification of Diseases 10th Revision (ICD-10) codes were retrieved from the electronic health record (EHR). Two physicians independently rated concordance between patient-reported diagnosis and clinician-entered principal problem as full, partial, or no. Discrepancies were resolved by consensus. Descriptive statistics were used to report demographics for concordant (full) and nonconcordant (partial or no) outcome groups. Multivariable logistic regressions of PDQ questions and a priori selected EHR data as independent variables were conducted to predict nonconcordance.

RESULTS

 A total of 157 (77.7%) questionnaires were completed by 202 participants; 77 (49.0%), 46 (29.3%), and 34 (21.7%) were rated fully concordant, partially concordant, and not concordant, respectively. Cohen's kappa for agreement on preconsensus ratings by independent reviewers was 0.81 (0.74, 0.88). In multivariable analyses, patient-reported lack of confidence and undifferentiated symptoms (ICD-10 "R-code") for the principal problem were significantly associated with nonconcordance (partial or no concordance ratings) after adjusting for other PDQ questions (3.43 [1.30, 10.39], p = 0.02) and in a model using selected variables (4.02 [1.80, 9.55], p < 0.01), respectively.

CONCLUSION

 About one-half of patient-reported diagnoses were concordant with the clinician-entered diagnosis on admission. An ICD-10 "R-code" entered as the principal problem and patient-reported lack of confidence may predict patient-clinician nonconcordance early during hospitalization via this approach.

The effectiveness of checklists and error reporting systems in enhancing patient safety and reducing medical errors in hospital settings: A narrative review

Objectives

This narrative review aimed to explore the impact of checklists and error reporting systems on hospital patient safety and medical errors.

Methods

A systematic search of academic databases from 2013 to 2023 was conducted, and peer-reviewed studies meeting inclusion criteria were assessed for methodological rigor. The review highlights evidence supporting the efficacy of checklists in reducing medication errors, surgical complications, and other adverse events. Error reporting systems foster transparency, encouraging professionals to report incidents and identify systemic vulnerabilities.

Results

Checklists and error reporting systems are interconnected. Interprofessional collaboration is emphasized in checklist implementation. In this review, limitations arise due to the different methodologies used in the articles and potential publication bias. In addition, language restrictions may exclude valuable non-English research. While positive impacts are evident, success depends on organizational culture and resources.

Conclusions

This review contributes to patient safety knowledge by examining the relevant literature, emphasizing the importance of interventions, and calling for further research into their effectiveness across diverse healthcare and cultural settings. Understanding these dynamics is crucial for healthcare providers to optimize patient safety outcomes.

Patient-reported outcome and experience domains for diagnostic excellence: a scoping review to inform future measure development

PURPOSE

"Diagnostic excellence," as a relatively new construct centered on the diagnostic process and its health-related outcomes, can be refined by patient reporting and its measurement. We aimed to explore the scope of patient-reported outcome (PRO) and patient-reported experience (PRE) domains that are diagnostically relevant, regardless of the future diagnosed condition, and to review the state of measurement of these patient-reported domains.

METHODS

We conducted an exploratory analysis to identify these domains by employing a scoping review supplemented with internal expert consultations, 24-member international expert convening, additional environmental scans, and the validation of the domains' diagnostic relevance via mapping these onto patient diagnostic journeys. We created a narrative bibliography of the domains illustrating them with existing measurement examples.

RESULTS

We identified 41 diagnostically relevant PRO and PRE domains. We classified 10 domains as PRO, 28 as PRE, and three as mixed PRO/PRE. Among these domains, 19 were captured in existing instruments, and 20 were captured only in qualitative studies. Two domains were conceptualized during this exploratory analysis with no examples identified of capturing these domains. For 27 domains, patients and care partners report on a specific encounter; for 14 domains, reporting relates to an entire diagnostic journey over time, which presents particular measurement opportunities and challenges.

CONCLUSION

The multitude of PRO and PRE domains, if measured rigorously, would allow the diagnostic excellence construct to evolve further and in a manner that is patient-centered, prospectively focused, and concentrates on effectiveness and efficiency of diagnostic care on patients' well-being.

Performance evaluation of ChatGPT in detecting diagnostic errors and their contributing factors: an analysis of 545 case reports of diagnostic errors

BACKGROUND

Manual chart review using validated assessment tools is a standardised methodology for detecting diagnostic errors. However, this requires considerable human resources and time. ChatGPT, a recently developed artificial intelligence chatbot based on a large language model, can effectively classify text based on suitable prompts. Therefore, ChatGPT can assist manual chart reviews in detecting diagnostic errors.

OBJECTIVE

This study aimed to clarify whether ChatGPT could correctly detect diagnostic errors and possible factors contributing to them based on case presentations.

METHODS

We analysed 545 published case reports that included diagnostic errors. We imputed the texts of case presentations and the final diagnoses with some original prompts into ChatGPT (GPT-4) to generate responses, including the judgement of diagnostic errors and contributing factors of diagnostic errors. Factors contributing to diagnostic errors were coded according to the following three taxonomies: Diagnosis Error Evaluation and Research (DEER), Reliable Diagnosis Challenges (RDC) and Generic Diagnostic Pitfalls (GDP). The responses on the contributing factors from ChatGPT were compared with those from physicians.

RESULTS

ChatGPT correctly detected diagnostic errors in 519/545 cases (95%) and coded statistically larger numbers of factors contributing to diagnostic errors per case than physicians: DEER (median 5 vs 1, p<0.001), RDC (median 4 vs 2, p<0.001) and GDP (median 4 vs 1, p<0.001). The most important contributing factors of diagnostic errors coded by ChatGPT were 'failure/delay in considering the diagnosis' (315, 57.8%) in DEER, 'atypical presentation' (365, 67.0%) in RDC, and 'atypical presentation' (264, 48.4%) in GDP.

CONCLUSION

ChatGPT accurately detects diagnostic errors from case presentations. ChatGPT may be more sensitive than manual reviewing in detecting factors contributing to diagnostic errors, especially for 'atypical presentation'.

Partnership as a Pathway to Diagnostic Excellence: The Challenges and Successes of Implementing the Safer Dx Learning Lab

BACKGROUND

Learning health system (LHS) approaches could potentially help health care organizations (HCOs) identify and address diagnostic errors. However, few such programs exist, and their implementation is poorly understood.

METHODS

The authors conducted a qualitative evaluation of the Safer Dx Learning Lab, a partnership between a health system and a research team, to identify and learn from diagnostic errors and improve diagnostic safety at an organizational level. The research team conducted virtual interviews to solicit participant feedback regarding experiences with the lab, focusing specifically on implementation and sustainment issues.

RESULTS

Interviews of 25 members associated with the lab identified the following successes: learning and professional growth, improved workflow related to streamlining the process of reporting error cases, and a psychologically safe culture for identifying and reporting diagnostic errors. However, multiple barriers also emerged: competing priorities between clinical responsibilities and research, time-management issues related to a lack of protected time, and inadequate guidance to disseminate findings. Lessons learned included understanding the importance of obtaining buy-in from leadership and interested stakeholders, creating a psychologically safe environment for reporting cases, and the need for more protected time for clinicians to review and learn from cases.

CONCLUSION

Findings suggest that a learning health systems approach using partnerships between researchers and a health system affected organizational culture by prioritizing learning from diagnostic errors and encouraging clinicians to be more open to reporting. The study findings can help organizations overcome barriers to engage clinicians and inform future implementation and sustainment of similar initiatives.

The PRIDx framework to engage payers in reducing diagnostic errors in healthcare

OBJECTIVES

No framework currently exists to guide how payers and providers can collaboratively develop and implement incentives to improve diagnostic safety. We conducted a literature review and interviews with subject matter experts to develop a multi-component 'Payer Relationships for Improving Diagnoses (PRIDx)' framework, that could be used to engage payers in diagnostic safety efforts.

CONTENT

The PRIDx framework, 1) conceptualizes diagnostic safety links to care provision, 2) illustrates ways to promote payer and provider engagement in the design and adoption of accountability mechanisms, and 3) explicates the use of data analytics. Certain approaches suggested by PRIDx were refined by subject matter expert interviewee perspectives.

SUMMARY

The PRIDx framework can catalyze public and private payers to take specific actions to improve diagnostic safety.

OUTLOOK

Implementation of the PRIDx framework requires new types of partnerships, including external support from public and private payer organizations, and requires creation of strong provider incentives without undermining providers' sense of professionalism and autonomy. PRIDx could help facilitate collaborative payer-provider approaches to improve diagnostic safety and generate research concepts, policy ideas, and potential innovations for engaging payers in diagnostic safety improvement activities.

Developing LHS scholars' competency around reducing burnout and moral injury

Despite the known benefits of supportive work environments for promoting patient quality and safety and healthcare worker retention, there is no clear mandate for improving work environments within Learning Health Systems (LHS) nor an LHS wellness competency. Striking rises in burnout levels among healthcare workers provide urgency for this topic.

Methods

We brought three experts on moral injury, burnout prevention, and ethics to a recurring, interactive LHS training program "Design Shop" session, harnessing scholars' ideas prior to the meeting. Generally following SQUIRE 2.0 guidelines, we evaluated the prework and discussion via informal content analysis to develop a set of pathways for developing moral injury and burnout prevention programs. Along these lines, we developed a new competency for moral injury and burnout prevention within LHS training programs.

Results

In preparation for the session, scholars differentiated moral injury from burnout, highlighted the profound impact of COVID-19 on moral injury, and proposed testable interventions to reduce injury. Scholar and expert input was then merged into developing the new competency in moral injury and burnout prevention. In particular, the competency focuses on preparing scholars to (1) demonstrate knowledge of moral injury and burnout, (2) measure burnout, moral injury, and their remediable predictors, (3) use methods for improving burnout, (4) structure training programs with supportive work environments, and (5) embed burnout and moral injury prevention into LHS structures.

Conclusions

Burnout and moral injury prevention have been largely omitted in LHS training. A competency related to burnout and moral injury reduction can potentially bring sustainable work lives for scholars and their colleagues, better incorporation of their science into clinical practice, and better outcomes for patients.

Medication safety climate from the perspectives of healthcare providers: A cross-sectional study

BACKGROUND

Medication safety as an indicator of care quality is the measures taken by healthcare team members to prevent or adjust adverse drug events at the time of medication administration. This study was conducted to investigate the medication safety climate from healthcare providers' perspectives.

MATERIALS AND METHODS

This cross-sectional descriptive study was conducted in a selected educational hospital affiliated with the Isfahan University of Medical Sciences, Isfahan, Iran, in 2021. Participants were healthcare providers who are involved in the medication process. The sampling was done using the quota method. The study instruments were a demographic questionnaire and the Medication Safety Climate (MSC).

RESULTS

The total mean of positive responses to MSC items was 64.11%, denoting a moderate-level MSC. Collected data were managed using the SPSS software (v. 16.0) and were summarized using the measures of descriptive statistics, namely mean, standard deviation, frequency, and percentage. The lowest and the highest dimensional mean scores were related to the management support for medication safety dimension (mean: 48.42%) and the Teamwork dimension (mean: 80.43%), respectively.

CONCLUSION

Managers' inattention and insufficient understanding of safety provide the basis for medication errors and threaten patient safety. Healthcare team members are highly motivated to provide quality and safe care by observing the managers' positive performance regarding patient safety. To improve the medication safety climate, healthcare team members are required to work in a safe workplace and have sufficient job satisfaction. Health center managers need to employ a proactive approach to prevent errors.

Developing the Safer Dx Checklist of Ten Safety Recommendations for Health Care Organizations to Address Diagnostic Errors

BACKGROUND

Most health care organizations (HCOs) find diagnostic errors hard to address. The research team developed a checklist (the Safer Dx Checklist) of 10 high-priority safety practices HCOs can use to conduct a proactive risk assessment to address diagnostic error.

METHODS

First, the team identified potential practices based on reviews of recent literature, reports by national and international organizations, and interviews with quality/safety leaders. Then a Delphi panel was conducted, followed by an online expert panel, to prioritize 10 practices. The prioritization process considered impact on safety and feasibility of practice implementation within a one- to three-year time frame. Finally, cognitive walkthroughs were conducted for a face-validity check with end users. The team also conducted content analysis in each step to look for themes that influenced prioritization or checklist implementation.

RESULTS

A total of 71 practices for prioritization were identified through the Delphi panel of 28 experts; 65% of participants reached consensus on 28 practices. A multidisciplinary panel of 10 experts helped prioritize and refine the top 10 practices, which were then developed into a checklist paired with implementation guidance. Practices included themes related to creating organizational and leadership accountability for improving diagnosis, including patients in diagnostic safety work, and developing and implementing organizational infrastructure for measurement and improvement activities. Qualitative analysis revealed insights for implementation. End users at three different HCOs helped refine implementation guidance for the checklist.

CONCLUSION

The researchers identified 10 safety practices to help organizations conduct a proactive, systematic assessment of risks to timely and accurate diagnosis. The Safer Dx Checklist can enable HCOs to begin implementing strategies to address diagnostic error.