Experience with Integrating Diagnostic Decision Support Software with Electronic Health Records: Benefits versus Risks of Information Sharing

The implementation of NILS: A web-based artificial neural network decision support tool for noninvasive lymph node staging in breast cancer

Objective

To implement artificial neural network (ANN) algorithms for noninvasive lymph node staging (NILS) to a decision support tool and facilitate the option to omit surgical axillary staging in breast cancer patients with low-risk of nodal metastasis.

Methods

The NILS tool is a further development of an ANN prototype for the prediction of nodal status. Training and internal validation of the original algorithm included 15 clinical and tumor-related variables from a consecutive cohort of 800 breast cancer cases. The updated NILS tool included 10 top-ranked input variables from the original prototype. A workflow with four ANN pathways was additionally developed to allow different combinations of missing preoperative input values. Predictive performances were assessed by area under the receiver operating characteristics curves (AUC) and sensitivity/specificity values at defined cut-points. Clinical utility was presented by estimating possible sentinel lymph node biopsy (SLNB) reduction rates. The principles of user-centered design were applied to develop an interactive web-interface to predict the patient's probability of healthy lymph nodes. A technical validation of the interface was performed using data from 100 test patients selected to cover all combinations of missing histopathological input values.

Results

ANN algorithms for the prediction of nodal status have been implemented into the web-based NILS tool for personalized, noninvasive nodal staging in breast cancer. The estimated probability of healthy lymph nodes using the interface showed a complete concordance with estimations from the reference algorithm except in two cases that had been wrongly included (ineligible for the technical validation). NILS predictive performance to distinguish node-negative from node-positive disease, also with missing values, displayed AUC ranged from 0.718 (95% CI, 0.687-0.748) to 0.735 (95% CI, 0.704-0.764), with good calibration. Sensitivity 90% and specificity 34% were demonstrated. The potential to abstain from axillary surgery was observed in 26% of patients using the NILS tool, acknowledging a false negative rate of 10%, which is clinically accepted for the standard SLNB technique.

Conclusions

The implementation of NILS into a web-interface are expected to provide the health care with decision support and facilitate preoperative identification of patients who could be good candidates to avoid unnecessary surgical axillary staging.

Acceptance, barriers and facilitators to implementing AI-based decision support systems in emergency departments: a quantitative and qualitative evaluation (Preprint)

Background

Despite the increasing availability of clinical decision support systems (CDSSs) and rising expectation for CDSSs based on artificial intelligence (AI), little is known about the acceptance of AI-based CDSS by physicians and its barriers and facilitators in emergency care settings.

Objective

We aimed to evaluate the acceptance, barriers, and facilitators to implementing AI-based CDSSs in the emergency care setting through the opinions of physicians on our newly developed, real-time AI-based CDSS, which alerts ED physicians by predicting aortic dissection based on numeric and text information from medical charts, by using the Unified Theory of Acceptance and Use of Technology (UTAUT; for quantitative evaluation) and the Consolidated Framework for Implementation Research (CFIR; for qualitative evaluation) frameworks.

Methods

This mixed methods study was performed from March to April 2021. Transitional year residents (n=6), emergency medicine residents (n=5), and emergency physicians (n=3) from two community, tertiary care hospitals in Japan were included. We first developed a real-time CDSS for predicting aortic dissection based on numeric and text information from medical charts (eg, chief complaints, medical history, vital signs) with natural language processing. This system was deployed on the internet, and the participants used the system with clinical vignettes of model cases. Participants were then involved in a mixed methods evaluation consisting of a UTAUT-based questionnaire with a 5-point Likert scale (quantitative) and a CFIR-based semistructured interview (qualitative). Cronbach α was calculated as a reliability estimate for UTAUT subconstructs. Interviews were sampled, transcribed, and analyzed using the MaxQDA software. The framework analysis approach was used during the study to determine the relevance of the CFIR constructs.

Results

All 14 participants completed the questionnaires and interviews. Quantitative analysis revealed generally positive responses for user acceptance with all scores above the neutral score of 3.0. In addition, the mixed methods analysis identified two significant barriers (System Performance, Compatibility) and two major facilitators (Evidence Strength, Design Quality) for implementation of AI-based CDSSs in emergency care settings.

Conclusions

Our mixed methods evaluation based on theoretically grounded frameworks revealed the acceptance, barriers, and facilitators of implementation of AI-based CDSS. Although the concern of system failure and overtrusting of the system could be barriers to implementation, the locality of the system and designing an intuitive user interface could likely facilitate the use of optimal AI-based CDSS. Alleviating and resolving these factors should be key to achieving good user acceptance of AI-based CDSS.

User testing of a diagnostic decision support system with machine-assisted chart review to facilitate clinical genomic diagnosis

Objectives

There is a need in clinical genomics for systems that assist in clinical diagnosis, analysis of genomic information and periodic reanalysis of results, and can use information from the electronic health record to do so. Such systems should be built using the concepts of human-centred design, fit within clinical workflows and provide solutions to priority problems.

Methods

We adapted a commercially available diagnostic decision support system (DDSS) to use extracted findings from a patient record and combine them with genomic variant information in the DDSS interface. Three representative patient cases were created in a simulated clinical environment for user testing. A semistructured interview guide was created to illuminate factors relevant to human factors in CDS design and organisational implementation.

Results

Six individuals completed the user testing process. Tester responses were positive and noted good fit with real-world clinical genetics workflow. Technical issues related to interface, interaction and design were minor and fixable. Testers suggested solving issues related to terminology and usability through training and infobuttons. Time savings was estimated at 30%–50% and additional uses such as in-house clinical variant analysis were suggested for increase fit with workflow and to further address priority problems.

Conclusion

This study provides preliminary evidence for usability, workflow fit, acceptability and implementation potential of a modified DDSS that includes machine-assisted chart review. Continued development and testing using principles from human-centred design and implementation science are necessary to improve technical functionality and acceptability for multiple stakeholders and organisational implementation potential to improve the genomic diagnosis process.

Drug Alert Experience and Salience during Medical Residency at Two Healthcare Institutions

BACKGROUND

Drug alerts are clinical decision support tools intended to prevent medication misadministration. In teaching hospitals, residents encounter the majority of the drug alerts while learning under variable workloads and responsibilities that may have an impact on drug-alert response rates.

OBJECTIVES

This study was aimed to explore drug-alert experience and salience among postgraduate year 1 (PGY-1), postgraduate year 2 (PGY-2), and postgraduate year 3 (PGY-3) internal medicine resident physicians at two different institutions.

METHODS

Drug-alert information was queried from the electronic health record (EHR) for 47 internal medicine residents at the University of Pennsylvania Medical Center (UPMC) Pinnacle in Pennsylvania, and 79 internal medicine residents at the MetroHealth System (MHS) in Ohio from December 2018 through February 2019. Salience was defined as the percentage of drug alerts resulting in removal or modification of the triggering order. Comparisons were made across institutions, residency training year, and alert burden.

RESULTS

A total of 126 residents were exposed to 52,624 alerts over a 3-month period. UPMC Pinnacle had 15,574 alerts with 47 residents and MHS had 37,050 alerts with 79 residents. At MHS, salience was 8.6% which was lower than UPMC Pinnacle with 15%. The relatively lower salience (42% lower) at MHS corresponded to a greater number of alerts-per-resident (41% higher) compared with UPMC Pinnacle. Overall, salience was 11.6% for PGY-1, 10.5% for PGY-2, and 8.9% for PGY-3 residents.

CONCLUSION

Our results are suggestive of long-term drug-alert desensitization during progressive residency training. A higher number of alerts-per-resident correlating with a lower salience suggests alert fatigue; however, other factors should also be considered including differences in workload and culture.

Role of Artificial Intelligence Applications in Real-Life Clinical Practice: Systematic Review

BACKGROUND

Artificial intelligence (AI) applications are growing at an unprecedented pace in health care, including disease diagnosis, triage or screening, risk analysis, surgical operations, and so forth. Despite a great deal of research in the development and validation of health care AI, only few applications have been actually implemented at the frontlines of clinical practice.

OBJECTIVE

The objective of this study was to systematically review AI applications that have been implemented in real-life clinical practice.

METHODS

We conducted a literature search in PubMed, Embase, Cochrane Central, and CINAHL to identify relevant articles published between January 2010 and May 2020. We also hand searched premier computer science journals and conferences as well as registered clinical trials. Studies were included if they reported AI applications that had been implemented in real-world clinical settings.

RESULTS

We identified 51 relevant studies that reported the implementation and evaluation of AI applications in clinical practice, of which 13 adopted a randomized controlled trial design and eight adopted an experimental design. The AI applications targeted various clinical tasks, such as screening or triage (n=16), disease diagnosis (n=16), risk analysis (n=14), and treatment (n=7). The most commonly addressed diseases and conditions were sepsis (n=6), breast cancer (n=5), diabetic retinopathy (n=4), and polyp and adenoma (n=4). Regarding the evaluation outcomes, we found that 26 studies examined the performance of AI applications in clinical settings, 33 studies examined the effect of AI applications on clinician outcomes, 14 studies examined the effect on patient outcomes, and one study examined the economic impact associated with AI implementation.

CONCLUSIONS

This review indicates that research on the clinical implementation of AI applications is still at an early stage despite the great potential. More research needs to assess the benefits and challenges associated with clinical AI applications through a more rigorous methodology.

Expansion of the Human Phenotype Ontology (HPO) knowledge base and resources

The Human Phenotype Ontology (HPO)—a standardized vocabulary of phenotypic abnormalities associated with 7000+ diseases—is used by thousands of researchers, clinicians, informaticians and electronic health record systems around the world. Its detailed descriptions of clinical abnormalities and computable disease definitions have made HPO the de facto standard for deep phenotyping in the field of rare disease. The HPO’s interoperability with other ontologies has enabled it to be used to improve diagnostic accuracy by incorporating model organism data. It also plays a key role in the popular Exomiser tool, which identifies potential disease-causing variants from whole-exome or whole-genome sequencing data. Since the HPO was first introduced in 2008, its users have become both more numerous and more diverse. To meet these emerging needs, the project has added new content, language translations, mappings and computational tooling, as well as integrations with external community data. The HPO continues to collaborate with clinical adopters to improve specific areas of the ontology and extend standardized disease descriptions. The newly redesigned HPO website (www.human-phenotype-ontology.org) simplifies browsing terms and exploring clinical features, diseases, and human genes.

An overview of clinical decision support systems: benefits, risks, and strategies for success

Computerized clinical decision support systems, or CDSS, represent a paradigm shift in healthcare today. CDSS are used to augment clinicians in their complex decision-making processes. Since their first use in the 1980s, CDSS have seen a rapid evolution. They are now commonly administered through electronic medical records and other computerized clinical workflows, which has been facilitated by increasing global adoption of electronic medical records with advanced capabilities. Despite these advances, there remain unknowns regarding the effect CDSS have on the providers who use them, patient outcomes, and costs. There have been numerous published examples in the past decade(s) of CDSS success stories, but notable setbacks have also shown us that CDSS are not without risks. In this paper, we provide a state-of-the-art overview on the use of clinical decision support systems in medicine, including the different types, current use cases with proven efficacy, common pitfalls, and potential harms. We conclude with evidence-based recommendations for minimizing risk in CDSS design, implementation, evaluation, and maintenance.

Development of a computer-aided design software for the quantitative evaluation of aesthetic damage

Concerns associated with the assessment of aesthetic damage or injury raise critical difficulties, such as the scarcity of methodology and standardization that may result in fundamental precepts to establish impartial forms of compensation and aiming the total reparation of bodily injury. The complexity of the aesthetic damage evaluation is associated with the confluence of legal and technical perspectives and expert subjectivity while conducting examination and writing a report. Experts face additional difficulties associated with the objectivity while assessing aesthetic damage, independently on its location or expert skills, due to complex details observed in these lesions. Another situation in the clinical area, doctors (mainly plastic surgeons) and dentists could show the improvement or not, of the aesthetic condition to the patients. In health related areas, the use of information technology has contributed to increase the number of appropriate diagnoses, besides promoting quality, efficiency and satisfaction to health care providers. In order to make this assessment more objective, a technological tool was developed to aid experts in the evaluation of aesthetic damage and report elaboration. The objective was to develop computer-aided design software for aesthetic damage quantification/evaluation that is accessible via internet to be applied as a complementary report on body aesthetic damage. The software uses as a parameter the AIPE method, translated transculturally from Spanish to Portuguese and English. The present study allowed the construction of open access auxiliary software for the evaluation of corporal aesthetic damage. Its use is facilitated by intuitive and interactive filling, and the text may be customized by the user. It transforms the report into PDF and saves all evaluations already done in its own file. Information is encrypted for added security and confidentiality. The software is available on website at https://www.aestheticdamage.com.