Assessment of Health Information Technology Interventions in Evidence-Based Medicine: A Systematic Review by Adopting a Methodological Evaluation Framework

Validation of a Questionnaire to Assess the Usability of and User Experience with Mobile Health Applications

BACKGROUND/OBJECTIVES

The growing use of mobile health (mHealth) applications needs reliable tools to assess their usability and user experience in clinical practice to improve the digital health (eHealth) interventions and ensure engagement, as higher engagement is often linked to increased efficacy of healthcare interventions. This study aimed to validate the patient Satisfaction and Usability with APPs questionnaire (pSUAPP), a multidimensional tool designed for the comprehensive assessment of mHealth applications, particularly for the integrated follow-up of patients with chronic diseases.

METHODS

A validation study was conducted between August and December 2022 with 85 participants from two hospitals in Spain, who completed the pSUAPP questionnaire, comprising 27 Likert-like items across four dimensions: first contact, registration, features and overall experience, and 1 open question. The questionnaire was validated by a panel of 11 experts and further assessed for psychometric properties.

RESULTS

The mean pSUAPP score was 79.0 (SD = 12.0), indicating high usability and positive user experience, with the highest scores in the 'features' dimension. The pSUAPP demonstrated moderate correlation with the System Usability Scale (SUS) and high reliability (Cronbach's alpha and omega t > 0.9). A reproducibility analysis showed negligible changes between repeated measures.

CONCLUSIONS

The pSUAPP questionnaire was found to be a robust tool for evaluating mHealth app usability and user experience, with potential application across various clinical settings.

The Impact of a Digital Referral Platform to Improve Access to Child and Adolescent Mental Health Services: A Prospective Observational Study with Real-World Data

(1) Background: In the UK, mental health needs for children and young people (CYP) are rising, whilst access to care is declining, particularly in the North of England and post-COVID-19. However, Health Information Technologies (HITs) can simplify access to Child and Adolescent Mental Health Services (CAMHS), reduce waiting times, and provide anonymous support and reliable information. (2) Methods: A single-centre prospective observational study examined the impact of "CYP as One"-a digital referral point to CAMHS-on waiting times and referral rejection rates. (3) Results: Waiting times during the first 12 months of "CYP as One" implementation were compared to the 12 months prior using non-parametric tests. "CYP as One" demonstrated an increase of 1314 referrals, with self-referrals rising by 71%. Initial implementation showed an increase in waiting times by 16.13 days (53.89 days) compared to pre-implementation (37.76 days) (p < 0.001). However, months 10 (M = 16.18, p < 0.001), 11 (M = 17.45, p < 0.001), and 12 (M = 31.45, p < 0.001) implementation showed reduced waiting times. Rejection rates rose due to a 108% increase in referral volume. "CYP as One" improved access and reduced waiting times after the initial phase. (4) Conclusions: Further research is needed to assess its long-term impact and cost-effectiveness, particularly regarding specific mental health conditions and staff time.

Development of an evaluation framework for health information communication technology in contemporary pharmacy practice

Background

Health information communication technology (ICT) has rapidly evolved in contemporary pharmacy practice worldwide. The Australian healthcare system is experiencing a paradigm shift to real-time interconnectivity for practitioners and consumers and interoperable digital health. With these developments come a need to evaluate use of technologies specifically in pharmacy practice to optimize their clinical functionality. There are no published frameworks for evaluating ICT needs or implementation in pharmacy practice.

Objective

This paper proposes a theoretical framework for evaluating health ICT in pharmacy.

Methods

Development of the evaluation framework was informed by a systematic scoping review and health informatics literature. Specifically, the framework drew upon critical appraisal and concept mapping of the TAM, ISS and HOT-fit validated models, with respect to health ICT in contemporary pharmacy practice.

Results

The proposed model was named the Technology Evaluation Key (TEK). The TEK comprises of 10 domains; healthcare system, organization, practitioner, user interface, ICT, use, operational outcomes, system outcomes, clinical outcomes and timely access to care.

Conclusions

This is the first published proposed evaluation framework developed for health ICT specifically in contemporary pharmacy practice. TEK represents a pragmatic way to ensure the development, refinement and implementation of new and existing technologies in contemporary pharmacy practice to keep pace with the clinical and professional requirements of community pharmacists. Operational, clinical and system outcomes should be evaluated as coexisting factors that may impact implementation. Validation research utilizing Design Science Research Methodology will enhance utility for end users and ensure the relevance and application of the TEK to contemporary pharmacy practice.

Impacts on Context Aware Systems in Evidence-Based Health Informatics: A Review

BACKGROUND

The application of Context Aware Computing (CAC) can be an effective, useful, feasible, and acceptable way to advance medical research and provide health services.

METHODS

This review was conducted in accordance with the principles of the development of a mixed methods review and existing knowledge in the field via the Synthesis Framework for the Assessment of Health Information Technology to evaluate CAC implemented by Evidence-Based Health Informatics (EBHI). A systematic search of the literature was performed during 18 November 2021-22 January 2022 in Cochrane Library, IEEE Xplore, PUBMED, Scopus and in the clinical registry platform Clinicaltrials.gov. The author included the articles in the review if they were implemented by EBHI and concerned with CAC technologies.

RESULTS

29 articles met the inclusion criteria and refer to 26 trials published between 2011 and 2022. The author noticed improvements in healthcare provision using EBHI in the findings of CAC application. She also confirmed that CAC systems are a valuable and reliable method in health care provision.

CONCLUSIONS

The use of CAC systems in healthcare is a promising new area of research and development. The author presented that the evaluation of CAC systems in EBHI presents positive effects on the state of health and the management of long-term diseases. These implications are presented in this article in a detailed, clear, and reliable manner.

Evaluation in Life Cycle of Information Technology (ELICIT) framework: Supporting the innovation life cycle from business case assessment to summative evaluation

OBJECTIVE

Our objective was to develop an evaluation framework for electronic health record (EHR)-integrated innovations to support evaluation activities at each of four information technology (IT) life cycle phases: planning, development, implementation, and operation.

METHODS

The evaluation framework was developed based on a review of existing evaluation frameworks from health informatics and other domains (human factors engineering, software engineering, and social sciences); expert consensus; and real-world testing in multiple EHR-integrated innovation studies.

RESULTS

The resulting Evaluation in Life Cycle of IT (ELICIT) framework covers four IT life cycle phases and three measure levels (society, user, and IT). The ELICIT framework recommends 12 evaluation steps: (1) business case assessment; (2) stakeholder requirements gathering; (3) technical requirements gathering; (4) technical acceptability assessment; (5) user acceptability assessment; (6) social acceptability assessment; (7) social implementation assessment; (8) initial user satisfaction assessment; (9) technical implementation assessment; (10) technical portability assessment; (11) long-term user satisfaction assessment; and (12) social outcomes assessment.

DISCUSSION

Effective evaluation requires a shared understanding and collaboration across disciplines throughout the entire IT life cycle. In contrast with previous evaluation frameworks, the ELICIT framework focuses on all phases of the IT life cycle across the society, user, and IT levels. Institutions seeking to establish evaluation programs for EHR-integrated innovations could use our framework to create such shared understanding and justify the need to invest in evaluation.

CONCLUSION

As health care undergoes a digital transformation, it will be critical for EHR-integrated innovations to be systematically evaluated. The ELICIT framework can facilitate these evaluations.

Safety and Acceptability of a Natural Language Artificial Intelligence Assistant to Deliver Clinical Follow-up to Cataract Surgery Patients: Proposal

Background

Due to an aging population, the demand for many services is exceeding the capacity of the clinical workforce. As a result, staff are facing a crisis of burnout from being pressured to deliver high-volume workloads, driving increasing costs for providers. Artificial intelligence (AI), in the form of conversational agents, presents a possible opportunity to enable efficiency in the delivery of care.

Objective

This study aims to evaluate the effectiveness, usability, and acceptability of Dora agent: Ufonia’s autonomous voice conversational agent, an AI-enabled autonomous telemedicine call for the detection of postoperative cataract surgery patients who require further assessment. The objectives of this study are to establish Dora’s efficacy in comparison with an expert clinician, determine baseline sensitivity and specificity for the detection of true complications, evaluate patient acceptability, collect evidence for cost-effectiveness, and capture data to support further development and evaluation.

Methods

Using an implementation science construct, the interdisciplinary study will be a mixed methods phase 1 pilot establishing interobserver reliability of the system, usability, and acceptability. This will be done using the following scales and frameworks: the system usability scale; assessment of Health Information Technology Interventions in Evidence-Based Medicine Evaluation Framework; the telehealth usability questionnaire; and the Non-Adoption, Abandonment, and Challenges to the Scale-up, Spread and Suitability framework.

Results

The evaluation is expected to show that conversational technology can be used to conduct an accurate assessment and that it is acceptable to different populations with different backgrounds. In addition, the results will demonstrate how successfully the system can be delivered in organizations with different clinical pathways and how it can be integrated with their existing platforms.

Conclusions

The project’s key contributions will be evidence of the effectiveness of AI voice conversational agents and their associated usability and acceptability.

International Registered Report Identifier (IRRID)

PRR1-10.2196/27227

Using ActionADE to create information continuity to reduce re-exposures to harmful medications: study protocol for a randomized controlled trial

BACKGROUND

Repeat exposures to culprit medications are a common cause of preventable adverse drug events. Health information technologies have the potential to reduce repeat adverse drug events by improving information continuity. However, they rarely interoperate to ensure providers can view adverse drug events documented in other systems. We designed ActionADE to enable rapid documentation of adverse drug events and communication of standardized information across health sectors by integrating with legacy systems. We will leverage ActionADE's implementation to conduct two parallel, randomized trials: patients with adverse drug reactions in the main trial and those diagnosed with non-adherence in a secondary trial. Primary objective of the main trial is to evaluate the effects of providing information continuity about adverse drug reactions on culprit medication re-dispensations over 12 months. Primary objective of the secondary trial is to evaluate the effect of providing information continuity on adherence over 12 months.

METHODS

We will conduct two parallel group, triple-blind randomized controlled trials in participating hospitals in British Columbia, Canada. We will enroll adults presenting to hospital with an adverse drug event to prescribed outpatient medication. Clinicians will document the adverse drug event in ActionADE. The software will use an algorithm to determine patient eligibility and allocate eligible patients to experimental or control. In the experimental arm, ActionADE will transmit information to PharmaNet, where adverse drug event information will be displayed in community pharmacies when re-dispensations are attempted. In the control arm, ActionADE will retain information in the local record. We will enroll 3600 adults with an adverse drug reaction into the main trial. The main trial's primary outcome is re-dispensation of a culprit or same-class medication within 12 months; the secondary trial's primary outcome will be adherence to culprit medication. Secondary outcomes include health services utilization and mortality.

DISCUSSION

These studies have the potential to guide policy decisions and investments needed to drive health information technology integrations to prevent repeat adverse drug events. We present an example of how a health information technology implementation can be leveraged to conduct pragmatic randomized controlled trials.

TRIAL REGISTRATION

ClinicalTrials.gov NCT04568668 , NCT04574648 . Registered on 1 October 2020.

The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review

BACKGROUND

The high demand for health care services and the growing capability of artificial intelligence have led to the development of conversational agents designed to support a variety of health-related activities, including behavior change, treatment support, health monitoring, training, triage, and screening support. Automation of these tasks could free clinicians to focus on more complex work and increase the accessibility to health care services for the public. An overarching assessment of the acceptability, usability, and effectiveness of these agents in health care is needed to collate the evidence so that future development can target areas for improvement and potential for sustainable adoption.

OBJECTIVE

This systematic review aims to assess the effectiveness and usability of conversational agents in health care and identify the elements that users like and dislike to inform future research and development of these agents.

METHODS

PubMed, Medline (Ovid), EMBASE (Excerpta Medica dataBASE), CINAHL (Cumulative Index to Nursing and Allied Health Literature), Web of Science, and the Association for Computing Machinery Digital Library were systematically searched for articles published since 2008 that evaluated unconstrained natural language processing conversational agents used in health care. EndNote (version X9, Clarivate Analytics) reference management software was used for initial screening, and full-text screening was conducted by 1 reviewer. Data were extracted, and the risk of bias was assessed by one reviewer and validated by another.

RESULTS

A total of 31 studies were selected and included a variety of conversational agents, including 14 chatbots (2 of which were voice chatbots), 6 embodied conversational agents (3 of which were interactive voice response calls, virtual patients, and speech recognition screening systems), 1 contextual question-answering agent, and 1 voice recognition triage system. Overall, the evidence reported was mostly positive or mixed. Usability and satisfaction performed well (27/30 and 26/31), and positive or mixed effectiveness was found in three-quarters of the studies (23/30). However, there were several limitations of the agents highlighted in specific qualitative feedback.

CONCLUSIONS

The studies generally reported positive or mixed evidence for the effectiveness, usability, and satisfactoriness of the conversational agents investigated, but qualitative user perceptions were more mixed. The quality of many of the studies was limited, and improved study design and reporting are necessary to more accurately evaluate the usefulness of the agents in health care and identify key areas for improvement. Further research should also analyze the cost-effectiveness, privacy, and security of the agents.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR2-10.2196/16934.

A design and evaluation framework for digital health interventions

Digital health interventions (DHIs) have the potential to help the growing number of chronic disease patients better manage their everyday lives. However, guidelines for the systematic development of DHIs are still scarce. The current work has, therefore, the objective to propose a framework for the design and evaluation of DHIs (DEDHI). The DEDHI framework is meant to support both researchers and practitioners alike from early conceptual DHI models to large-scale implementations of DHIs in the healthcare market.