Evidence-based decision support for pediatric rheumatology reduces diagnostic errors

Effect on diagnostic accuracy of cognitive reasoning tools for the workplace setting: systematic review and meta-analysis

Background

Preventable diagnostic errors are a large burden on healthcare. Cognitive reasoning tools, that is, tools that aim to improve clinical reasoning, are commonly suggested interventions. However, quantitative estimates of tool effectiveness have been aggregated over both workplace-oriented and educational-oriented tools, leaving the impact of workplace-oriented cognitive reasoning tools alone unclear. This systematic review and meta-analysis aims to estimate the effect of cognitive reasoning tools on improving diagnostic performance among medical professionals and students, and to identify factors associated with larger improvements.

Methods

Controlled experimental studies that assessed whether cognitive reasoning tools improved the diagnostic accuracy of individual medical students or professionals in a workplace setting were included. Embase.com, Medline ALL via Ovid, Web of Science Core Collection, Cochrane Central Register of Controlled Trials and Google Scholar were searched from inception to 15 October 2021, supplemented with handsearching. Meta-analysis was performed using a random-effects model.

Results

The literature search resulted in 4546 articles of which 29 studies with data from 2732 participants were included for meta-analysis. The pooled estimate showed considerable heterogeneity (I2=70%). This was reduced to I2=38% by removing three studies that offered training with the tool before the intervention effect was measured. After removing these studies, the pooled estimate indicated that cognitive reasoning tools led to a small improvement in diagnostic accuracy (Hedges’ g=0.20, 95% CI 0.10 to 0.29, p<0.001). There were no significant subgroup differences.

Conclusion

Cognitive reasoning tools resulted in small but clinically important improvements in diagnostic accuracy in medical students and professionals, although no factors could be distinguished that resulted in larger improvements. Cognitive reasoning tools could be routinely implemented to improve diagnosis in practice, but going forward, more large-scale studies and evaluations of these tools in practice are needed to determine how these tools can be effectively implemented.

PROSPERO registration number

CRD42020186994.

Role of Artificial Intelligence Applications in Real-Life Clinical Practice: Systematic Review

BACKGROUND

Artificial intelligence (AI) applications are growing at an unprecedented pace in health care, including disease diagnosis, triage or screening, risk analysis, surgical operations, and so forth. Despite a great deal of research in the development and validation of health care AI, only few applications have been actually implemented at the frontlines of clinical practice.

OBJECTIVE

The objective of this study was to systematically review AI applications that have been implemented in real-life clinical practice.

METHODS

We conducted a literature search in PubMed, Embase, Cochrane Central, and CINAHL to identify relevant articles published between January 2010 and May 2020. We also hand searched premier computer science journals and conferences as well as registered clinical trials. Studies were included if they reported AI applications that had been implemented in real-world clinical settings.

RESULTS

We identified 51 relevant studies that reported the implementation and evaluation of AI applications in clinical practice, of which 13 adopted a randomized controlled trial design and eight adopted an experimental design. The AI applications targeted various clinical tasks, such as screening or triage (n=16), disease diagnosis (n=16), risk analysis (n=14), and treatment (n=7). The most commonly addressed diseases and conditions were sepsis (n=6), breast cancer (n=5), diabetic retinopathy (n=4), and polyp and adenoma (n=4). Regarding the evaluation outcomes, we found that 26 studies examined the performance of AI applications in clinical settings, 33 studies examined the effect of AI applications on clinician outcomes, 14 studies examined the effect on patient outcomes, and one study examined the economic impact associated with AI implementation.

CONCLUSIONS

This review indicates that research on the clinical implementation of AI applications is still at an early stage despite the great potential. More research needs to assess the benefits and challenges associated with clinical AI applications through a more rigorous methodology.

The accuracy of computer-based diagnostic tools for the identification of concurrent genetic disorders

The increasing use of next-generation sequencing, especially clinical exome sequencing, has revealed that individuals having two coexisting genetic conditions are not uncommon occurrences. This pilot study evaluates the efficacy of two methodologically distinct computational differential diagnosis generating tools-FindZebra and SimulConsult-in identifying multiple genetic conditions in a single patient. Clinical query terms were generated for each of 15 monogenic disorders that were effective in resulting in the top 10 list of differential diagnoses for each of the 15 monogenic conditions when entered into these bioinformatics tools. Then, the terms of over 125 pairings of these conditions were entered using each tool and the resulting list of diagnoses evaluated to determine how often both diagnoses of a pair were represented in that list. Neither tool was successful in identifying both members of a pair of conditions in greater than 40% of test cases. Disorder detection sensitivity was not homogeneous within a tool, with each tool favoring the identification of a subset of genetic conditions. In view of recent exome sequencing data showing an unexpectedly high prevalence of coexistent monogenic conditions, the results from this pilot study highlight a need for the development of computational tools designed to effectively generate differential diagnoses with consideration of the possibility of coexisting conditions.

Experience with Integrating Diagnostic Decision Support Software with Electronic Health Records: Benefits versus Risks of Information Sharing

Introduction:

Reducing misdiagnosis has long been a goal of medical informatics. Current thinking has focused on achieving this goal by integrating diagnostic decision support into electronic health records.

Methods:

A diagnostic decision support system already in clinical use was integrated into electronic health record systems at two large health systems, after clinician input on desired capabilities. The decision support provided three outputs: editable text for use in a clinical note, a summary including the suggested differential diagnosis with a graphical representation of probability, and a list of pertinent positive and pertinent negative findings (with onsets).

Results:

Structured interviews showed widespread agreement that the tool was useful and that the integration improved workflow. There was disagreement among various specialties over the risks versus benefits of documenting intermediate diagnostic thinking. Benefits were most valued by specialists involved in diagnostic testing, who were able to use the additional clinical context for richer interpretation of test results. Risks were most cited by physicians making clinical diagnoses, who expressed concern that a process that generated diagnostic possibilities exposed them to legal liability.

Discussion and Conclusion:

Reconciling the preferences of the various groups could include saving only the finding list as a patient-wide resource, saving intermediate diagnostic thinking only temporarily, or adoption of professional guidelines to clarify the role of decision support in diagnosis.