Evaluation Framework for Successful Artificial Intelligence-Enabled Clinical Decision Support Systems: Mixed Methods Study

Clinician Perspectives on Decision Support and AI-based Decision Support in a Pediatric ED

BACKGROUND

Clinical decision support (CDS) systems offer the potential to improve pediatric care through enhanced test ordering, prescribing, and standardization of care. Its augmentation with artificial intelligence (AI-CDS) may help address current limitations with CDS implementation regarding alarm fatigue and accuracy of recommendations. We sought to evaluate strengths and perceptions of CDS, with a focus on AI-CDS, through semistructured interviews of clinician partners.

METHODS

We conducted a qualitative study using semistructured interviews of physicians, nurse practitioners, and nurses at a single quaternary-care pediatric emergency department to evaluate clinician perceptions of CDS and AI-CDS. We used reflexive thematic analysis to identify themes and purposive sampling to complete recruitment with the goal of reaching theoretical sufficiency.

RESULTS

We interviewed 20 clinicians. Participants demonstrated a variable understanding of CDS and AI, with some lacking a clear definition. Most recognized the potential benefits of AI-CDS in clinical contexts, such as data summarization and interpretation. Identified themes included the potential of AI-CDS to improve diagnostic accuracy, standardize care, and improve efficiency, while also providing educational benefits to clinicians. Participants raised concerns about the ability of AI-based tools to appreciate nuanced pediatric care, accurately interpret data, and about tensions between AI recommendations and clinician autonomy.

CONCLUSIONS

AI-CDS tools have a promising role in pediatric emergency medicine but require careful integration to address clinicians' concerns about autonomy, nuance recognition, and interpretability. A collaborative approach to development and implementation, informed by clinicians' insights and perspectives, will be pivotal for their successful adoption and efficacy in improving patient care.

Recent advances in artificial intelligent strategies for tissue engineering and regenerative medicine

BACKGROUND

Tissue engineering and regenerative medicine (TERM) aim to repair or replace damaged or lost tissues or organs due to accidents, diseases, or aging, by applying different sciences. For this purpose, an essential part of TERM is the designing, manufacturing, and evaluating of scaffolds, cells, tissues, and organs. Artificial intelligence (AI) or the intelligence of machines or software can be effective in all areas where computers play a role.

METHODS

The "artificial intelligence," "machine learning," "tissue engineering," "clinical evaluation," and "scaffold" keywords used for searching in various databases and articles published from 2000 to 2024 were evaluated.

RESULTS

The combination of tissue engineering and AI has created a new generation of technological advancement in the biomedical industry. Experience in TERM has been refined using advanced design and manufacturing techniques. Advances in AI, particularly deep learning, offer an opportunity to improve scientific understanding and clinical outcomes in TERM.

CONCLUSION

The findings of this research show the high potential of AI, machine learning, and robots in the selection, design, and fabrication of scaffolds, cells, tissues, or organs, and their analysis, characterization, and evaluation after their implantation. AI can be a tool to accelerate the introduction of tissue engineering products to the bedside.

HIGHLIGHTS

The capabilities of artificial intelligence (AI) can be used in different ways in all the different stages of TERM and not only solve the existing limitations, but also accelerate the processes, increase efficiency and precision, reduce costs, and complications after transplantation. ML predicts which technologies have the most efficient and easiest path to enter the market and clinic. The use of AI along with these imaging techniques can lead to the improvement of diagnostic information, the reduction of operator errors when reading images, and the improvement of image analysis (such as classification, localization, regression, and segmentation).

Prescription for Digital Evolution: Transformative Recommendations for Pharmacy Practice in the Digital Age

Increased administrative tasks, evolving expectations of care and advancement in practice scope have rapidly advanced digital health. Health policy has acknowledged the need for evaluation to determine the technological needs of clinicians, including pharmacists, to practice to full and top of scope. There is an emergent need for recommendations to address the technological transformation to enable community pharmacists' practice. This study aimed to develop digital health recommendations, through expert consensus, for the government, pharmacy professional associations, pharmacy enterprises and software vendors, to facilitate community pharmacists' practice. A modified Delphi survey was conducted online in February-March 2024. Pharmacists with digital health expertise were purposively recruited. Participants were asked to rate their level of agreement with the initial 24 research-derived statements in round 1. Consensus was defined a priori as ≥80% of participants strongly agreeing or agreeing with a statement and a standard deviation of ≤1.00. Review of participants' free-text comments progressively reduced and refined the statements. All 22 participants completed the modified Delphi study in 3 survey rounds. Participants represented every Australian jurisdiction. Eighteen participants had more than 10 years of professional experience. Sixteen recommendations emerged: 6 for government, 2 for pharmacy professional associations, 4 for pharmacy enterprises and 4 for software vendors. The majority of recommendations require financial investment and harmonization of legislation across jurisdictions. Adoption of these recommendations, with significant investment across partners in the healthcare system and technology providers, will enable pharmacists to more effectively and safely practice utilizing technology solutions.

Evaluation of a Computer-Aided Clinical Decision Support System for Point-of-Care Use in Low-Resource Primary Care Settings: Acceptability Evaluation Study

BACKGROUND

A clinical decision support system (CDSS) based on the logic and philosophy of clinical pathways is critical for managing the quality of health care and for standardizing care processes. Using such a system at a point-of-care setting is becoming more frequent these days. However, in a low-resource setting (LRS), such systems are frequently overlooked.

OBJECTIVE

The purpose of the study was to evaluate the user acceptance of a CDSS in LRSs.

METHODS

The CDSS evaluation was carried out at the Jimma Health Center and the Jimma Higher Two Health Center, Jimma, Ethiopia. The evaluation was based on 22 parameters organized into 6 categories: ease of use, system quality, information quality, decision changes, process changes, and user acceptance. A Mann-Whitney U test was used to investigate whether the difference between the 2 health centers was significant (2-tailed, 95% CI; α=.05). Pearson correlation and partial least squares structural equation modeling (PLS-SEM) was used to identify the relationship and factors influencing the overall acceptance of the CDSS in an LRS.

RESULTS

On the basis of 116 antenatal care, pregnant patient care, and postnatal care cases, 73 CDSS evaluation responses were recorded. We found that the 2 health centers did not differ significantly on 16 evaluation parameters. We did, however, detect a statistically significant difference in 6 parameters (P<.05). PLS-SEM results showed that the coefficient of determination, R2, of perceived user acceptance was 0.703. More precisely, the perceived ease of use (β=.015, P=.91) and information quality (β=.149, P=.25) had no positive effect on CDSS acceptance but, rather, on the system quality and perceived benefits of the CDSS, with P<.05 and β=.321 and β=.486, respectively. Furthermore, the perceived ease of use was influenced by information quality and system quality, with an R2 value of 0.479, indicating that the influence of information quality on the ease of use is significant but the influence of system quality on the ease of use is not, with β=.678 (P<.05) and β=.021(P=.89), respectively. Moreover, the influence of decision changes (β=.374, P<.05) and process changes (β=.749, P<.05) both was significant on perceived benefits (R2=0.983).

CONCLUSIONS

This study concludes that users are more likely to accept and use a CDSS at the point of care when it is easy to grasp the perceived benefits and system quality in terms of health care professionals' needs. We believe that the CDSS acceptance model developed in this study reveals specific factors and variables that constitute a step toward the effective adoption and deployment of a CDSS in LRSs.

Machine Learning Model for Readmission Prediction of Patients With Heart Failure Based on Electronic Health Records: Protocol for a Quasi-Experimental Study for Impact Assessment

BACKGROUND

Care for patients with heart failure (HF) causes a substantial load on health care systems where a prominent challenge is the elevated rate of readmissions within 30 days following initial discharge. Clinical professionals face high levels of uncertainty and subjectivity in the decision-making process on the optimal timing of discharge. Unwanted hospital stays generate costs and cause stress to patients and potentially have an impact on care outcomes. Recent studies have aimed to mitigate the uncertainty by developing and testing risk assessment tools and predictive models to identify patients at risk of readmission, often using novel methods such as machine learning (ML).

OBJECTIVE

This study aims to investigate how a developed clinical decision support (CDS) tool alters the decision-making processes of health care professionals in the specific context of discharging patients with HF, and if so, in which ways. Additionally, the aim is to capture the experiences of health care practitioners as they engage with the system's outputs to analyze usability aspects and obtain insights related to future implementation.

METHODS

A quasi-experimental design with randomized crossover assessment will be conducted with health care professionals on HF patients' scenarios in a region located in the South of Sweden. In total, 12 physicians and nurses will be randomized into control and test groups. The groups shall be provided with 20 scenarios of purposefully sampled patients. The clinicians will be asked to take decisions on the next action regarding a patient. The test group will be provided with the 10 scenarios containing patient data from electronic health records and an outcome from an ML-based CDS model on the risk level for readmission of the same patients. The control group will have 10 other scenarios without the CDS model output and containing only the patients' data from electronic medical records. The groups will switch roles for the next 10 scenarios. This study will collect data through interviews and observations. The key outcome measures are decision consistency, decision quality, work efficiency, perceived benefits of using the CDS model, reliability, validity, and confidence in the CDS model outcome, integrability in the routine workflow, ease of use, and intention to use. This study will be carried out in collaboration with Cambio Healthcare Systems.

RESULTS

The project is part of the Center for Applied Intelligent Systems Research Health research profile, funded by the Knowledge Foundation (2021-2028). Ethical approval for this study was granted by the Swedish ethical review authority (2022-07287-02). The recruitment process of the clinicians and the patient scenario selection will start in September 2023 and last till March 2024.

CONCLUSIONS

This study protocol will contribute to the development of future formative evaluation studies to test ML models with clinical professionals.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/52744.

The need to strengthen the evaluation of the impact of Artificial Intelligence-based decision support systems on healthcare provision

Despite the renewed interest in Artificial Intelligence-based clinical decision support systems (AI-CDS), there is still a lack of empirical evidence supporting their effectiveness. This underscores the need for rigorous and continuous evaluation and monitoring of processes and outcomes associated with the introduction of health information technology. We illustrate how the emergence of AI-CDS has helped to bring to the fore the critical importance of evaluation principles and action regarding all health information technology applications, as these hitherto have received limited attention. Key aspects include assessment of design, implementation and adoption contexts; ensuring systems support and optimise human performance (which in turn requires understanding clinical and system logics); and ensuring that design of systems prioritises ethics, equity, effectiveness, and outcomes. Going forward, information technology strategy, implementation and assessment need to actively incorporate these dimensions. International policy makers, regulators and strategic decision makers in implementing organisations therefore need to be cognisant of these aspects and incorporate them in decision-making and in prioritising investment. In particular, the emphasis needs to be on stronger and more evidence-based evaluation surrounding system limitations and risks as well as optimisation of outcomes, whilst ensuring learning and contextual review. Otherwise, there is a risk that applications will be sub-optimally embodied in health systems with unintended consequences and without yielding intended benefits.

Evaluating a clinical decision support point of care instrument in low resource setting

BACKGROUND

Clinical pathways are one of the main tools to manage the health care's quality and concerned with the standardization of care processes. They have been used to help frontline healthcare workers by presenting summarized evidence and generating clinical workflows involving a series of tasks performed by various people within and between work environments to deliver care. Integrating clinical pathways into Clinical Decision Support Systems (CDSSs) is a common practice today. However, in a low-resource setting (LRS), this kind of decision support systems is often not readily accessible or even not available. To fill this gap, we developed a computer aided CDSS that swiftly identifies which cases require a referral and which ones may be managed locally. The computer aided CDSS is designed primarily for use in primary care settings for maternal and childcare services, namely for pregnant patients, antenatal and postnatal care. The purpose of this paper is to assess the user acceptance of the computer aided CDSS at the point of care in LRSs.

METHODS

For evaluation, we used a total of 22 parameters structured in to six major categories, namely "ease of use, system quality, information quality, decision changes, process changes, and user acceptance." Based on these parameters, the caregivers from Jimma Health Center's Maternal and Child Health Service Unit evaluated the acceptability of a computer aided CDSS. The respondents were asked to express their level of agreement using 22 parameters in a think-aloud approach. The evaluation was conducted in the caregiver's spare-time after the clinical decision. It was based on eighteen cases over the course of two days. The respondents were then asked to score their level of agreement with some statements on a five-point scale: strongly disagree, disagree, neutral, agree, and strongly agree.

RESULTS

The CDSS received a favorable agreement score in all six categories by obtaining primarily strongly agree and agree responses. In contrast, a follow-up interview revealed a variety of reasons for disagreement based on the neutral, disagree, and strongly disagree responses.

CONCLUSIONS

Though the study had a positive outcome, it was limited to the Jimma Health Center Maternal and Childcare Unit, and hence a wider scale evaluation and longitudinal measurements, including computer aided CDSS usage frequency, speed of operation and impact on intervention time are needed.

Clinical research staff perceptions on a natural language processing-driven tool for eligibility prescreening: An iterative usability assessment

BACKGROUND

Participant recruitment is a barrier to successful clinical research. One strategy to improve recruitment is to conduct eligibility prescreening, a resource-intensive process where clinical research staff manually reviews electronic health records data to identify potentially eligible patients. Criteria2Query (C2Q) was developed to address this problem by capitalizing on natural language processing to generate queries to identify eligible participants from clinical databases semi-autonomously.

OBJECTIVE

We examined the clinical research staff's perceived usability of C2Q for clinical research eligibility prescreening.

METHODS

Twenty clinical research staff evaluated the usability of C2Q using a cognitive walkthrough with a think-aloud protocol and a Post-Study System Usability Questionnaire. On-screen activity and audio were recorded and transcribed. After every-five evaluators completed an evaluation, usability problems were rated by informatics experts and prioritized for system refinement. There were four iterations of system refinement based on the evaluation feedback. Guided by the Organizational Framework for Intuitive Human-computer Interaction, we performed a directed deductive content analysis of the verbatim transcriptions.

RESULTS

Evaluators aged from 24 to 46 years old (33.8; SD: 7.32) demonstrated high computer literacy (6.36; SD:0.17); female (75 %), White (35 %), and clinical research coordinators (45 %). C2Q demonstrated high usability during the final cycle (2.26 out of 7 [lower scores are better], SD: 0.74). The number of unique usability issues decreased after each refinement. Fourteen subthemes emerged from three themes: seeking user goals, performing well-learned tasks, and determining what to do next.

CONCLUSIONS

The cognitive walkthrough with a think-aloud protocol informed iterative system refinement and demonstrated the usability of C2Q by clinical research staff. Key recommendations for system development and implementation include improving system intuitiveness and overall user experience through comprehensive consideration of user needs and requirements for task completion.

Health information communication technology evaluation frameworks for pharmacist prescribing: A systematic scoping review

BACKGROUND

Information communication technology (ICT) is instrumental in pharmacists' current practice and emerging roles. One such role is prescribing, which requires the use of clinical guidelines and documentation of decision-making, commonly via ICT. Development and refinement of ICT should be guided by evaluation frameworks that describe or measure features of ICT and its implementation. In the context of pharmacist prescribing, these evaluation frameworks should be specific to health stakeholders and the pharmacy setting.

OBJECTIVES

To identify ICT evaluation frameworks from health-related literature and review frameworks relevant to the development, implementation, and evaluation of pharmacist prescribing.

METHODS

A database search of CINAHL, Cochrane Library, EMBASE, Medline (Ovid), ProQuest, Scopus, Web of Science and grey literature was conducted, using combinations of keywords relating to 'ICT', 'utilization', 'usability', and 'evaluation framework'. Abstracts and titles were screened according to inclusion criteria. Identified evaluation frameworks were critiqued for relevance to pharmacy practice.

RESULTS

Twenty-two articles were identified, describing the development or application of 20 evaluation frameworks. None of the frameworks was developed specifically for pharmacy practice. The Technology Acceptance Model (TAM), describing use behavior, behavior intention, perceived usefulness, and perceived ease of use, was the most widely utilized framework. The Information System Success (ISS) and Human-Organization and Technology Fit (HOT-fit) are notable evaluation frameworks that address user and organizational influences in health ICT utility, and factors of both can address the limitation of TAM.

CONCLUSIONS

The findings call for development of an agile evaluation framework for the system under review; however, this can prove difficult due to the heterogenicity and complexity of the healthcare system, particularly contemporary pharmacy practice. While the TAM appears useful to evaluate user attitudes and intentions towards ICT, its relevance to ICT in contemporary community pharmacy practice requires exploration.

Demystifying Supervised Learning in Healthcare 4.0: A New Reality of Transforming Diagnostic Medicine

The global healthcare sector continues to grow rapidly and is reflected as one of the fastest-growing sectors in the fourth industrial revolution (4.0). The majority of the healthcare industry still uses labor-intensive, time-consuming, and error-prone traditional, manual, and manpower-based methods. This review addresses the current paradigm, the potential for new scientific discoveries, the technological state of preparation, the potential for supervised machine learning (SML) prospects in various healthcare sectors, and ethical issues. The effectiveness and potential for innovation of disease diagnosis, personalized medicine, clinical trials, non-invasive image analysis, drug discovery, patient care services, remote patient monitoring, hospital data, and nanotechnology in various learning-based automation in healthcare along with the requirement for explainable artificial intelligence (AI) in healthcare are evaluated. In order to understand the potential architecture of non-invasive treatment, a thorough study of medical imaging analysis from a technical point of view is presented. This study also represents new thinking and developments that will push the boundaries and increase the opportunity for healthcare through AI and SML in the near future. Nowadays, SML-based applications require a lot of data quality awareness as healthcare is data-heavy, and knowledge management is paramount. Nowadays, SML in biomedical and healthcare developments needs skills, quality data consciousness for data-intensive study, and a knowledge-centric health management system. As a result, the merits, demerits, and precautions need to take ethics and the other effects of AI and SML into consideration. The overall insight in this paper will help researchers in academia and industry to understand and address the future research that needs to be discussed on SML in the healthcare and biomedical sectors.

Conformity assessment of a computer vision-based posture analysis system for the screening of postural deformation

Background

This study evaluates the conformity of using a computer vision-based posture analysis system as a screening assessment for postural deformity detection in the spine that is easily applicable to clinical practice.

Methods

One hundred forty participants were enrolled for screening of the postural deformation. Factors that determine the presence or absence of spinal deformation, such as shoulder height difference (SHD), pelvic height difference (PHD), and leg length mismatch (LLD), were used as parameters for the clinical decision support system (CDSS) using a commercial computer vision-based posture analysis system. For conformity analysis, the probability of postural deformation provided by CDSS, the Cobb angle, the PHD, and the SHD was compared and analyzed between the system and radiographic parameters. A principal component analysis (PCA) of the CDSS and correlation analysis were conducted.

Results

The Cobb angles of the 140 participants ranged from 0° to 61°, with an average of 6.16° ± 8.50°. The postural deformation of CDSS showed 94% conformity correlated with radiographic assessment. The conformity assessment results were more accurate in the participants of postural deformation with normal (0–9°) and mild (10–25°) ranges of scoliosis. The referenced SHD and the SHD of the CDSS showed statistical significance (p < 0.001) on a paired t-test. SHD and PHD for PCA were the predominant factors (PC1 SHD for 79.97%, PC2 PHD for 19.86%).

Conclusion

The CDSS showed 94% conformity for the screening of postural spinal deformity. The main factors determining diagnostic suitability were two main variables: SHD and PHD. In conclusion, a computer vision-based posture analysis system can be utilized as a safe, efficient, and convenient CDSS for early diagnosis of spinal posture deformation, including scoliosis.

The Interplay of Work, Digital Health Usage, and the Perceived Effects of Digitalization on Physicians' Work: Network Analysis Approach

Background

In health care, the benefits of digitalization need to outweigh the risks, but there is limited knowledge about the factors affecting this balance in the work environment of physicians. To achieve the benefits of digitalization, a more comprehensive understanding of this complex phenomenon related to the digitalization of physicians’ work is needed.

Objective

The aim of this study was to examine physicians’ perceptions of the effects of health care digitalization on their work and to analyze how these perceptions are associated with multiple factors related to work and digital health usage.

Methods

A representative sample of 4630 (response rate 24.46%) Finnish physicians (2960/4617, 64.11% women) was used. Statements measuring the perceived effects of digitalization on work included the patients’ active role, preventive work, interprofessional cooperation, decision support, access to patient information, and faster consultations. Network analysis of the perceived effects of digitalization and factors related to work and digital health usage was conducted using mixed graphical modeling. Adjusted and standardized regression coefficients are denoted by b. Centrality statistics were examined to evaluate the relative influence of each variable in terms of node strength.

Results

Nearly half of physicians considered that digitalization has promoted an active role for patients in their own care (2104/4537, 46.37%) and easier access to patient information (1986/4551, 43.64%), but only 1 in 10 (445/4529, 9.82%) felt that the impact has been positive on consultation times with patients. Almost half of the respondents estimated that digitalization has neither increased nor decreased the possibilities for preventive work (2036/4506, 45.18%) and supportiveness of clinical decision support systems (1941/4458, 43.54%). When all variables were integrated into the network, the most influential variables were purpose of using health information systems, employment sector, and specialization status. However, the grade given to the electronic health record (EHR) system that was primarily used had the strongest direct links to faster consultations (b=0.32) and facilitated access to patient information (b=0.28). At least 6 months of use of the main EHR was associated with facilitated access to patient information (b=0.18).

Conclusions

The results highlight the complex interdependence of multiple factors associated with the perceived effects of digitalization on physicians’ work. It seems that a high-quality EHR system is critical for promoting smooth clinical practice. In addition, work-related factors may influence other factors that affect digital health success. These factors should be considered when developing and implementing new digital health technologies or services for physicians’ work. The adoption of digital health is not just a technological project but a project that changes existing work practices.

Artificial intelligence and thyroid disease management: considerations for thyroid function tests

Artificial intelligence (AI) is transforming healthcare and offers new tools in clinical research, personalized medicine, and medical diagnostics. Thyroid function tests represent an important asset for physicians in the diagnosis and monitoring of pathologies. Artificial intelligence tools can clearly assist physicians and specialists in laboratory medicine to optimize test prescription, tests interpretation, decision making, process optimization, and assay design. Our article is reviewing several of these aspects. As thyroid AI models rely on large data sets, which often requires distributed learning from multi-center contributions, this article also briefly discusses this issue.