Technology Acceptance of a Machine Learning Algorithm Predicting Delirium in a Clinical Setting: a Mixed-Methods Study

Understanding machine learning applications in dementia research and clinical practice: a review for biomedical scientists and clinicians

Several (inter)national longitudinal dementia observational datasets encompassing demographic information, neuroimaging, biomarkers, neuropsychological evaluations, and muti-omics data, have ushered in a new era of potential for integrating machine learning (ML) into dementia research and clinical practice. ML, with its proficiency in handling multi-modal and high-dimensional data, has emerged as an innovative technique to facilitate early diagnosis, differential diagnosis, and to predict onset and progression of mild cognitive impairment and dementia. In this review, we evaluate current and potential applications of ML, including its history in dementia research, how it compares to traditional statistics, the types of datasets it uses and the general workflow. Moreover, we identify the technical barriers and challenges of ML implementations in clinical practice. Overall, this review provides a comprehensive understanding of ML with non-technical explanations for broader accessibility to biomedical scientists and clinicians.

A Pilot Study Toward Development of the Digital Literacy, Usability, and Acceptability of Technology Instrument for Healthcare

Electronic health record proficiency is critical for health professionals to deliver and document patient care. There is scarce research on this topic within undergraduate nursing student populations. The purpose of this study is to describe the psychometric evaluation of the Digital Literacy, Usability, and Acceptability of Technology Instrument for Healthcare. A cross-sectional pilot study for psychometric evaluation of the instrument was conducted using data collected through an emailed survey. Exploratory factor analysis, inter-item and adjusted item-total correlations, and Cronbach's α calculated subscale reliability. A total of 297 nursing students completed the survey. A seven-factor structure best fit the data: technology use-engagement, technology use-confidence, technology use-history, electronic health record-ease of use, electronic health record-comparability, and electronic health record-burden. Cronbach's α indicated good to very good internal consistency (α = .68 to .89). The instrument effectively measured digital literacy, acceptance, and usability of an electronic health record and may be implemented with good to very good reliability across varied healthcare simulation and training experiences.

Beyond technology acceptance-a focused ethnography on the implementation, acceptance and use of new nursing technology in a German hospital

Introduction

Hospitalised patients could benefit from the emergence of novel technologies for nursing care. There are numerous technical products available, but these rarely find their way into practice. Further knowledge is required about the circumstances under which technology in nursing is accepted and used. In the research project "Centre for Implementing Nursing Care Innovations", technical innovations are implemented on a trauma surgery inpatient ward in Germany. After implementation, it was investigated: Which implemented technologies are accepted/rejected, and which factors influence the acceptance/rejection of technology for nurses?

Material and methods

A focused ethnography was used, containing two approaches: First, participant observation was conducted to examine nurses' and patients' interaction with technologies. Observations were fixed in a field research diary and analysed using evaluative qualitative content analysis. Second, a questionnaire was used by nurses to provide information about the use frequency and technology suitability. The results of the study were consolidated and analysed using the UTAUT model.

Results

Seven studied technologies can be summarised in four result categories: (1) A Mobilising mattress, a Special projector and a Sound pillow are accepted and used by nurses and patients, because they offer a way to provide high quality care with little additional effort. (2) A Fall prevention system is consistently used in patient care as a work obligation, but since nurses consider the system error-prone, acceptance is low. (3) An Interactive therapy ball is accepted but nurses cannot use it due to the high workload. (4) An App for nurse-patient communication and a work-equipment tracking system are not used or accepted because nurses do not see a practical benefit in the systems.

Discussion

Acceptance or rejection of a product does not necessarily equate to use or non-use of the technology. Before implementation, technology acceptance among users occurs as prejudice-when users are given time to experiment with technology, intention-to-use can stabilize into sustained use. Accepted and used technologies can serve to mask problems (such as staff shortages) and encourage problematic developments, such as the reduction of contact time at the bedside. Therefore, technology acceptance should be qualified in asking to what accepted technology contributes.

Trust and acceptability of data-driven clinical recommendations in everyday practice: A scoping review

BACKGROUND

Increasing attention is being given to the analysis of large health datasets to derive new clinical decision support systems (CDSS). However, few data-driven CDSS are being adopted into clinical practice. Trust in these tools is believed to be fundamental for acceptance and uptake but to date little attention has been given to defining or evaluating trust in clinical settings.

OBJECTIVES

A scoping review was conducted to explore how and where acceptability and trustworthiness of data-driven CDSS have been assessed from the health professional's perspective.

METHODS

Medline, Embase, PsycInfo, Web of Science, Scopus, ACM Digital, IEEE Xplore and Google Scholar were searched in March 2022 using terms expanded from: "data-driven" AND "clinical decision support" AND "acceptability". Included studies focused on healthcare practitioner-facing data-driven CDSS, relating directly to clinical care. They included trust or a proxy as an outcome, or in the discussion. The preferred reporting items for systematic reviews and meta-analyses extension for scoping reviews (PRISMA-ScR) is followed in the reporting of this review.

RESULTS

3291 papers were screened, with 85 primary research studies eligible for inclusion. Studies covered a diverse range of clinical specialisms and intended contexts, but hypothetical systems (24) outnumbered those in clinical use (18). Twenty-five studies measured trust, via a wide variety of quantitative, qualitative and mixed methods. A further 24 discussed themes of trust without it being explicitly evaluated, and from these, themes of transparency, explainability, and supporting evidence were identified as factors influencing healthcare practitioner trust in data-driven CDSS.

CONCLUSION

There is a growing body of research on data-driven CDSS, but few studies have explored stakeholder perceptions in depth, with limited focused research on trustworthiness. Further research on healthcare practitioner acceptance, including requirements for transparency and explainability, should inform clinical implementation.

Factors influencing clinician and patient interaction with machine learning-based risk prediction models: a systematic review

Machine learning (ML)-based risk prediction models hold the potential to support the health-care setting in several ways; however, use of such models is scarce. We aimed to review health-care professional (HCP) and patient perceptions of ML risk prediction models in published literature, to inform future risk prediction model development. Following database and citation searches, we identified 41 articles suitable for inclusion. Article quality varied with qualitative studies performing strongest. Overall, perceptions of ML risk prediction models were positive. HCPs and patients considered that models have the potential to add benefit in the health-care setting. However, reservations remain; for example, concerns regarding data quality for model development and fears of unintended consequences following ML model use. We identified that public views regarding these models might be more negative than HCPs and that concerns (eg, extra demands on workload) were not always borne out in practice. Conclusions are tempered by the low number of patient and public studies, the absence of participant ethnic diversity, and variation in article quality. We identified gaps in knowledge (particularly views from under-represented groups) and optimum methods for model explanation and alerts, which require future research.

A Data-Driven Paradigm for a Resilient and Sustainable Integrated Health Information Systems for Health Care Applications

Introduction

Many transformations and uncertainties, such as the fourth industrial revolution and pandemics, have propelled healthcare acceptance and deployment of health information systems (HIS). External and internal determinants aligning with the global course influence their deployments. At the epic is digitalization, which generates endless data that has permeated healthcare. The continuous proliferation of complex and dynamic healthcare data is the digitalization frontier in healthcare that necessitates attention.

Objective

This study explores the existing body of information on HIS for healthcare through the data lens to present a data-driven paradigm for healthcare augmentation paramount to attaining a sustainable and resilient HIS.

Method

Preferred Reporting Items for Systematic Reviews and Meta-Analyses: PRISMA-compliant in-depth literature review was conducted systematically to synthesize and analyze the literature content to ascertain the value disposition of HIS data in healthcare delivery.

Results

This study details the aspects of a data-driven paradigm for robust and sustainable HIS for health care applications. Data source, data action and decisions, data sciences techniques, serialization of data sciences techniques in the HIS, and data insight implementation and application are data-driven features expounded. These are essential data-driven paradigm building blocks that need iteration to succeed.

Discussions

Existing literature considers insurgent data in healthcare challenging, disruptive, and potentially revolutionary. This view echoes the current healthcare quandary of good and bad data availability. Thus, data-driven insights are essential for building a resilient and sustainable HIS. People, technology, and tasks dominated prior HIS frameworks, with few data-centric facets. Improving healthcare and the HIS requires identifying and integrating crucial data elements.

Conclusion

The paper presented a data-driven paradigm for a resilient and sustainable HIS. The findings show that data-driven track and components are essential to improve healthcare using data analytics insights. It provides an integrated footing for data analytics to support and effectively assist health care delivery.

Effects of machine learning-based clinical decision support systems on decision-making, care delivery, and patient outcomes: a scoping review

OBJECTIVE

This study aims to summarize the research literature evaluating machine learning (ML)-based clinical decision support (CDS) systems in healthcare settings.

MATERIALS AND METHODS

We conducted a review in accordance with the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta Analyses extension for Scoping Review). Four databases, including PubMed, Medline, Embase, and Scopus were searched for studies published from January 2016 to April 2021 evaluating the use of ML-based CDS in clinical settings. We extracted the study design, care setting, clinical task, CDS task, and ML method. The level of CDS autonomy was examined using a previously published 3-level classification based on the division of clinical tasks between the clinician and CDS; effects on decision-making, care delivery, and patient outcomes were summarized.

RESULTS

Thirty-two studies evaluating the use of ML-based CDS in clinical settings were identified. All were undertaken in developed countries and largely in secondary and tertiary care settings. The most common clinical tasks supported by ML-based CDS were image recognition and interpretation (n = 12) and risk assessment (n = 9). The majority of studies examined assistive CDS (n = 23) which required clinicians to confirm or approve CDS recommendations for risk assessment in sepsis and for interpreting cancerous lesions in colonoscopy. Effects on decision-making, care delivery, and patient outcomes were mixed.

CONCLUSION

ML-based CDS are being evaluated in many clinical areas. There remain many opportunities to apply and evaluate effects of ML-based CDS on decision-making, care delivery, and patient outcomes, particularly in resource-constrained settings.

An integrative review on the acceptance of artificial intelligence among healthcare professionals in hospitals

Artificial intelligence (AI) in the domain of healthcare is increasing in prominence. Acceptance is an indispensable prerequisite for the widespread implementation of AI. The aim of this integrative review is to explore barriers and facilitators influencing healthcare professionals' acceptance of AI in the hospital setting. Forty-two articles met the inclusion criteria for this review. Pertinent elements to the study such as the type of AI, factors influencing acceptance, and the participants' profession were extracted from the included studies, and the studies were appraised for their quality. The data extraction and results were presented according to the Unified Theory of Acceptance and Use of Technology (UTAUT) model. The included studies revealed a variety of facilitating and hindering factors for AI acceptance in the hospital setting. Clinical decision support systems (CDSS) were the AI form included in most studies (n = 21). Heterogeneous results with regard to the perceptions of the effects of AI on error occurrence, alert sensitivity and timely resources were reported. In contrast, fear of a loss of (professional) autonomy and difficulties in integrating AI into clinical workflows were unanimously reported to be hindering factors. On the other hand, training for the use of AI facilitated acceptance. Heterogeneous results may be explained by differences in the application and functioning of the different AI systems as well as inter-professional and interdisciplinary disparities. To conclude, in order to facilitate acceptance of AI among healthcare professionals it is advisable to integrate end-users in the early stages of AI development as well as to offer needs-adjusted training for the use of AI in healthcare and providing adequate infrastructure.

Data Science Implementation Trends in Nursing Practice: A Review of the 2021 Literature

OBJECTIVES

The goal of this work was to provide a review of the implementation of data science-driven applications focused on structural or outcome-related nurse-sensitive indicators in the literature in 2021. By conducting this review, we aim to inform readers of trends in the nursing indicators being addressed, the patient populations and settings of focus, and lessons and challenges identified during the implementation of these tools.

METHODS

We conducted a rigorous descriptive review of the literature to identify relevant research published in 2021. We extracted data on model development, implementation-related strategies and measures, lessons learned, and challenges and stakeholder involvement. We also assessed whether reports of data science application implementations currently follow the guidelines of the Developmental and Exploratory Clinical Investigations of DEcision support systems driven by AI (DECIDE-AI) framework.

RESULTS

Of 4,943 articles found in PubMed (NLM) and CINAHL (EBSCOhost), 11 were included in the final review and data extraction. Systems leveraging data science were developed for adult patient populations and were primarily deployed in hospital settings. The clinical domains targeted included mortality/deterioration, utilization/resource allocation, and hospital-acquired infections/COVID-19. The composition of development teams and types of stakeholders involved varied. Research teams more frequently reported on implementation methods than implementation results. Most studies provided lessons learned that could help inform future implementations of data science systems in health care.

CONCLUSION

In 2021, very few studies report on the implementation of data science-driven applications focused on structural- or outcome-related nurse-sensitive indicators. This gap in the sharing of implementation strategies needs to be addressed in order for these systems to be successfully adopted in health care settings.

Implemented machine learning tools to inform decision-making for patient care in hospital settings: a scoping review

OBJECTIVES

To identify ML tools in hospital settings and how they were implemented to inform decision-making for patient care through a scoping review. We investigated the following research questions: What ML interventions have been used to inform decision-making for patient care in hospital settings? What strategies have been used to implement these ML interventions?

DESIGN

A scoping review was undertaken. MEDLINE, Embase, Cochrane Central Register of Controlled Trials (CENTRAL) and the Cochrane Database of Systematic Reviews (CDSR) were searched from 2009 until June 2021. Two reviewers screened titles and abstracts, full-text articles, and charted data independently. Conflicts were resolved by another reviewer. Data were summarised descriptively using simple content analysis.

SETTING

Hospital setting.

PARTICIPANT

Any type of clinician caring for any type of patient.

INTERVENTION

Machine learning tools used by clinicians to inform decision-making for patient care, such as AI-based computerised decision support systems or "'model-based'" decision support systems.

PRIMARY AND SECONDARY OUTCOME MEASURES

Patient and study characteristics, as well as intervention characteristics including the type of machine learning tool, implementation strategies, target population. Equity issues were examined with PROGRESS-PLUS criteria.

RESULTS

After screening 17 386 citations and 3474 full-text articles, 20 unique studies and 1 companion report were included. The included articles totalled 82 656 patients and 915 clinicians. Seven studies reported gender and four studies reported PROGRESS-PLUS criteria (race, health insurance, rural/urban). Common implementation strategies for the tools were clinician reminders that integrated ML predictions (44.4%), facilitated relay of clinical information (17.8%) and staff education (15.6%). Common barriers to successful implementation of ML tools were time (11.1%) and reliability (11.1%), and common facilitators were time/efficiency (13.6%) and perceived usefulness (13.6%).

CONCLUSIONS

We found limited evidence related to the implementation of ML tools to assist clinicians with patient healthcare decisions in hospital settings. Future research should examine other approaches to integrating ML into hospital clinician decisions related to patient care, and report on PROGRESS-PLUS items.

FUNDING

Canadian Institutes of Health Research (CIHR) Foundation grant awarded to SES and the CIHR Strategy for Patient Oriented-Research Initiative (GSR-154442). SCOPING REVIEW REGISTRATION: https://osf.io/e2mna.

Evaluation of a Machine Learning-Based Dysphagia Prediction Tool in Clinical Routine: A Prospective Observational Cohort Study

Based on a large number of pre-existing documented electronic health records (EHR), we developed a machine learning (ML) algorithm for detection of dysphagia and aspiration pneumonia. The aim of our study was to prospectively apply this algorithm in two large patient cohorts. The tool was integrated in the hospital information system of a secondary care hospital in Austria. Based on existing data such as diagnoses, laboratory, and medication, dysphagia risk was predicted automatically, and patients were stratified into three risk groups. Patients' risk groups and risk factors were visualized in a web application. Prospective predictions of 1270 admissions to geriatric or internal medicine departments were compared with the occurrence of dysphagia or aspiration pneumonia of routinely documented events. The discriminative performance for internal medicine patients (n = 885) was excellent with an AUROC of 0.841, a sensitivity of 74.2%, and a specificity of 84.1%. For the smaller geriatric cohort (n = 221), the AUROC was 0.758, sensitivity 44.4%, and specificity 93.0%. For both cohorts, calibration plots showed a slight overestimation of the risk. This is the first study to evaluate the performance of a ML-based prediction tool for dysphagia in a prospective clinical setting. Future studies should validate the predictions on data of systematic dysphagia screening by specialists and evaluate user satisfaction and acceptance. The ML-based dysphagia prediction tool achieved an excellent performance in the internal medicine cohort. More data are needed to determine the performance in geriatric patients.

Inclusion of Clinicians in the Development and Evaluation of Clinical Artificial Intelligence Tools: A Systematic Literature Review

The application of machine learning (ML) and artificial intelligence (AI) in healthcare domains has received much attention in recent years, yet significant questions remain about how these new tools integrate into frontline user workflow, and how their design will impact implementation. Lack of acceptance among clinicians is a major barrier to the translation of healthcare innovations into clinical practice. In this systematic review, we examine when and how clinicians are consulted about their needs and desires for clinical AI tools. Forty-five articles met criteria for inclusion, of which 24 were considered design studies. The design studies used a variety of methods to solicit and gather user feedback, with interviews, surveys, and user evaluations. Our findings show that tool designers consult clinicians at various but inconsistent points during the design process, and most typically at later stages in the design cycle (82%, 19/24 design studies). We also observed a smaller amount of studies adopting a human-centered approach and where clinician input was solicited throughout the design process (22%, 5/24). A third (15/45) of all studies reported on clinician trust in clinical AI algorithms and tools. The surveyed articles did not universally report validation against the “gold standard” of clinical expertise or provide detailed descriptions of the algorithms or computational methods used in their work. To realize the full potential of AI tools within healthcare settings, our review suggests there are opportunities to more thoroughly integrate frontline users’ needs and feedback in the design process.

Impact of Face-Recognition-Based Access Control System on College Students' Sense of School Identity and Belonging During COVID-19 Pandemic

In the context of coronavirus pandemic (COVID-19), the face-recognition-based access control system (FACS) has been intensively adopted to protect students’ and teachers’ health and safety in school. However, the impact of FACS, as a new technology, on students’ attitude toward accepting FACS has remained unknown from the psychological halo effect. Drawn on “halo effect” theory where psychological effects affect the sense of social identity and belonging, the present study explored college students’ sense of school identity and belonging in using FACS during COVID-19 based on the technology acceptance model (TAM). Data collected from 391 college students was analyzed using SEM to verify the relationship among perceived usefulness (PU), perceived ease of use (PEU), intention to use (IU), school identity, and school belonging. The results show that PU and PEU can positively predict IU, and consequentially can positively predict school identity and school belonging. Our study expands the application of halo effect theory to study FACS acceptance based on TAM, and provides strong evidence to support the effect of school FACS during the pandemic. The findings of this study also suggest that FACS acceptance can enhance students’ sense of school identity and belonging.