Is the LITE version of the usability metric for user experience (UMUX-LITE) a reliable tool to support rapid assessment of new healthcare technology?

Adapting and Evaluating an AI-Based Chatbot Through Patient and Stakeholder Engagement to Provide Information for Different Health Conditions: Master Protocol for an Adaptive Platform Trial (the MARVIN Chatbots Study)

BACKGROUND

Artificial intelligence (AI)-based chatbots could help address some of the challenges patients face in acquiring information essential to their self-health management, including unreliable sources and overburdened health care professionals. Research to ensure the proper design, implementation, and uptake of chatbots is imperative. Inclusive digital health research and responsible AI integration into health care require active and sustained patient and stakeholder engagement, yet corresponding activities and guidance are limited for this purpose.

OBJECTIVE

In response, this manuscript presents a master protocol for the development, testing, and implementation of a chatbot family in partnership with stakeholders. This protocol aims to help efficiently translate an initial chatbot intervention (MARVIN) to multiple health domains and populations.

METHODS

The MARVIN chatbots study has an adaptive platform trial design consisting of multiple parallel individual chatbot substudies with four common objectives: (1) co-construct a tailored AI chatbot for a specific health care setting, (2) assess its usability with a small sample of participants, (3) measure implementation outcomes (usability, acceptability, appropriateness, adoption, and fidelity) within a large sample, and (4) evaluate the impact of patient and stakeholder partnerships on chatbot development. For objective 1, a needs assessment will be conducted within the setting, involving four 2-hour focus groups with 5 participants each. Then, a co-construction design committee will be formed with patient partners, health care professionals, and researchers who will participate in 6 workshops for chatbot development, testing, and improvement. For objective 2, a total of 30 participants will interact with the prototype for 3 weeks and assess its usability through a survey and 3 focus groups. Positive usability outcomes will lead to the initiation of objective 3, whereby the public will be able to access the chatbot for a 12-month real-world implementation study using web-based questionnaires to measure usability, acceptability, and appropriateness for 150 participants and meta-use data to inform adoption and fidelity. After each objective, for objective 4, focus groups will be conducted with the design committee to better understand their perspectives on the engagement process.

RESULTS

From July 2022 to October 2023, this master protocol led to four substudies conducted at the McGill University Health Centre or the Centre hospitalier de l'Université de Montréal (both in Montreal, Quebec, Canada): (1) MARVIN for HIV (large-scale implementation expected in mid-2024), (2) MARVIN-Pharma for community pharmacists providing HIV care (usability study planned for mid-2024), (3) MARVINA for breast cancer, and (4) MARVIN-CHAMP for pediatric infectious conditions (both in preparation, with development to begin in early 2024).

CONCLUSIONS

This master protocol offers an approach to chatbot development in partnership with patients and health care professionals that includes a comprehensive assessment of implementation outcomes. It also contributes to best practice recommendations for patient and stakeholder engagement in digital health research.

TRIAL REGISTRATION

ClinicalTrials.gov NCT05789901; https://classic.clinicaltrials.gov/ct2/show/NCT05789901.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/54668.

Advances in prosthetic technology: a perspective on ethical considerations for development and clinical translation

Technological advancements of prostheses in recent years, such as haptic feedback, active power, and machine learning for prosthetic control, have opened new doors for improved functioning, satisfaction, and overall quality of life. However, little attention has been paid to ethical considerations surrounding the development and translation of prosthetic technologies into clinical practice. This article, based on current literature, presents perspectives surrounding ethical considerations from the authors' multidisciplinary views as prosthetists (HG, AM, CLM, MGF), as well as combined research experience working directly with people using prostheses (AM, CLM, MGF), wearable technologies for rehabilitation (MGF, BN), machine learning and artificial intelligence (BN, KKQ), and ethics of advanced technologies (KKQ). The target audience for this article includes developers, manufacturers, and researchers of prosthetic devices and related technology. We present several ethical considerations for current advances in prosthetic technology, as well as topics for future research, that may inform product and policy decisions and positively influence the lives of those who can benefit from advances in prosthetic technology.

The NORDeHEALTH 2022 Patient Survey: Cross-Sectional Study of National Patient Portal Users in Norway, Sweden, Finland, and Estonia

BACKGROUND

Although many surveys have been conducted on patients accessing their own health records in recent years, there is a limited amount of nationwide cross-country data available on patients' views and preferences. To address this gap, an international survey of patient users was conducted in the Nordic eHealth project, NORDeHEALTH.

OBJECTIVE

We aimed to investigate the sociodemographic characteristics and experiences of patients who accessed their electronic health records (EHRs) through national patient portals in Norway, Sweden, Finland, and Estonia.

METHODS

A cross-sectional web-based survey was distributed using the national online health portals. The target participants were patients who accessed the national patient portals at the start of 2022 and who were aged ≥15 years. The survey included a mixture of close-ended and free-text questions about participant sociodemographics, usability experience, experiences with health care and the EHR, reasons for reading health records online, experience with errors, omissions and offense, opinions about security and privacy, and the usefulness of portal functions. In this paper, we summarized the data on participant demographics, past experience with health care, and the patient portal through descriptive statistics.

RESULTS

In total, 29,334 users completed the survey, of which 9503 (32.40%) were from Norway, 13,008 (44.35%) from Sweden, 4713 (16.07%) from Finland, and 2104 (7.17%) from Estonia. National samples were comparable according to reported gender, with about two-thirds identifying as women (19,904/29,302, 67.93%). Age distributions were similar across the countries, but Finland had older users while Estonia had younger users. The highest attained education and presence of health care education varied among the national samples. In all 4 countries, patients most commonly rated their health as "fair" (11,279/29,302, 38.48%). In Estonia, participants were more often inclined to rate their health positively, whereas Norway and Sweden had the highest proportion of negative health ratings. Across the whole sample, most patients received some care in the last 2 years (25,318/29,254, 86.55%). Mental health care was more common (6214/29,254, 21.24%) than oncological care (3664/29,254, 12.52%). Overall, most patients had accessed their health record "2 to 9 times" (11,546/29,306, 39.4%), with the most frequent users residing in Sweden, where about one-third of patients accessed it "more than 20 times" (4571/13,008, 35.14%).

CONCLUSIONS

This is the first large-scale international survey to compare patient users' sociodemographics and experiences with accessing their EHRs. Although the countries are in close geographic proximity and demonstrate similar advancements in giving their residents online records access, patient users in this survey differed. We will continue to investigate patients' experiences and opinions about national patient-accessible EHRs through focused analyses of the national and combined data sets from the NORDeHEALTH 2022 Patient Survey.

Patients' Experiences of Web-Based Access to Electronic Health Records in Finland: Cross-sectional Survey

Background

Patient portals that provide access to electronic health records offer a means for patients to better understand and self-manage their health. Yet, patient access to electronic health records raises many concerns among physicians, and little is known about the use practices and experiences of patients who access their electronic health records via a mature patient portal that has been available for citizens for over five years.

Objective

We aimed to identify patients’ experiences using a national patient portal to access their electronic health records. In particular, we focused on understanding usability-related perceptions and the benefits and challenges of reading clinical notes written by health care professionals.

Methods

Data were collected from 3135 patient users of the Finnish My Kanta patient portal through a web-based survey in June 2021 (response rate: 0.7%). Patients received an invitation to complete the questionnaire when they logged out of the patient portal. Respondents were asked to rate the usability of the patient portal, and the ratings were used to calculate approximations of the System Usability Scale score. Patients were also asked about the usefulness of features, and whether they had discussed the notes with health professionals. Open-ended questions were used to ask patients about their experiences of the benefits and challenges related to reading health professionals’ notes.

Results

Overall, patient evaluations of My Kanta were positive, and its usability was rated as good (System Usability Scale score approximation: mean 72.7, SD 15.9). Patients found the portal to be the most useful for managing prescriptions and viewing the results of examinations and medical notes. Viewing notes was the most frequent reason (978/3135, 31.2%) for visiting the portal. Benefits of reading the notes mentioned by patients included remembering and understanding what was said by health professionals and the instructions given during an appointment, the convenience of receiving information about health and care, the capability to check the accuracy of notes, and using the information to support self-management. However, there were challenges related to difficulty in understanding medical terminology, incorrect or inadequate notes, missing notes, and usability.

Conclusions

Patients actively used medical notes to receive information to follow professionals' instructions to take care of their health, and patient access to electronic health records can support self-management. However, for the benefits to be realized, improvements in the quality and availability of medical professionals’ notes are necessary. Providing a standard information structure could help patients find the information they need. Furthermore, linking notes to vocabularies and other information sources could also improve the understandability of medical terminology; patient agency could be supported by allowing them to add comments to their notes, and patient trust of the system could be improved by allowing them to control the visibility of the professionals’ notes.

Usability and User Experience of Cognitive Intervention Technologies for Elderly People With MCI or Dementia: A Systematic Review

Introduction

Incorporating technology in cognitive interventions represents an innovation, making them more accessible, flexible, and cost-effective. This will not be feasible without adequate user-technology fit. Bearing in mind the importance of developing cognitive interventions whose technology is appropriate for elderly people with cognitive impairment, the objective of this systematic review was to find evidence about usability and user experience (UX) measurements and features of stimulation, training, and cognitive rehabilitation technologies for older adults with mild cognitive impairment (MCI) or dementia.

Method

The Medline, PubMed, Scopus, ScienceDirect, and PsycINFO databases were searched for literature published in the last 10 years (2009–2019), and three researchers independently reviewed potentially eligible studies, following specific inclusion criteria. A systematic review of the studies was conducted, presenting a qualitative synthesis of usability and UX measures with their outcomes, study characteristics and features of the cognitive intervention technologies.

Results

Ten studies were selected: five were cognitive stimulation and five were cognitive training. Most of them (60%) were computer-based programs with a serious game format. Efficiency and effectiveness were the most frequent measurements used for collecting objective usability data, showing that elderly people with cognitive impairment require more time (45%) and help (40%) but can complete tasks (60%). Regarding UX or subjective usability data, questionnaires and scales were the most used methods, reporting positive experience despite certain difficulties with the interface in five studies.

Conclusion

Measuring usability and UX in cognitive intervention technologies for older adults with MCI or dementia provides an integrated view that can contribute to their development according to the needs and characteristics of the target population. More research is required to include this population group in usability and UX studies, as well as standardized tools and consensus on the relationship of these terms to guarantee the future effectiveness of cognitive intervention technologies.

Review registration

This review was registered in the PROSPERO (CRD42020158147) International Register of Systematic Review Protocols.

Remembering Ben-Tzion Karsh's scholarship, impact, and legacy

Dr. Ben-Tzion (Bentzi) Karsh was a mentor, collaborator, colleague, and friend who profoundly impacted the fields of human factors and ergonomics (HFE), medical informatics, patient safety, and primary care, among others. In this paper we honor his contributions by reflecting on his scholarship, impact, and legacy in three ways: first, through an updated simplified bibliometric analysis in 2020, highlighting the breadth of his scholarly impact from the perspective of the number and types of communities and collaborators with which and whom he engaged; second, through targeted reflections on the history and impact of Dr. Karsh's most cited works, commenting on the particular ways they impacted our academic community; and lastly, through quotes from collaborators and mentees, illustrating Dr. Karsh's long-lasting impact on his contemporaries and students.

Inside pandora's box: a systematic review of the assessment of the perceived quality of chatbots for people with disabilities or special needs

Introduction: People with disabilities or special needs can benefit from AI-based conversational agents (i.e., chatbots) that are used for competence training and well-being management. Assessing the quality of interactions with these chatbots is key to being able to reduce dissatisfaction with them and to understanding their potential long-term benefit. This in turn will help to increase adherence to their use, thereby improving the quality of life of the large population of end-users that they are able to serve.Methods: Following Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) methodology, we systematically reviewed the literature on methods of assessing the perceived quality of interactions with chatbots using the from Scopus and the Web of Science electronic databases. Using the Boolean operators (AND/OR) the keywords chatbot*, conversational agent*, special needs, disability were combined.Results: Revealed that only 15 of 192 papers on this topic included people with disabilities or special needs in their assessments. The results also highlighted the lack of a shared theoretical framework for assessing the perceived quality of interactions with chatbots.Conclusion: Systematic procedures based on reliable and valid methodologies continue to be needed in this field. The current lack of reliable tools and systematic methods to assess chatbots for people with disabilities and special needs is concerning, and ultimately, it may also lead to unreliable systems entering the market with disruptive consequences for people.Implications for rehabilitationChatbots applied in rehabilitation are mainly tested in terms of clinical effectiveness and validity with a minimal focus on measuring the quality of the interactionThe usability and interactive properties of chatbots applied in rehabilitation are not comparable as each tool is measured in different wayThe lack of a common framework to assess chatbots exposes people with disability and special needs to the risk of using unreliable tools.

How to Ease the Pain of Taking a Diagnostic Point of Care Test to the Market: A Framework for Evidence Development

Bringing a diagnostic point of care test (POCT) to a healthcare market can be a painful experience as it requires the manufacturer to meet considerable technical, financial, managerial, and regulatory challenges. In this opinion article we propose a framework for developing the evidence needed to support product development, marketing, and adoption. We discuss each step in the evidence development pathway from the invention phase to the implementation of a new POCT in the healthcare system. We highlight the importance of articulating the value propositions and documenting the care pathway. We provide guidance on how to conduct care pathway analysis as little has been published on this. We summarize the clinical, economic and qualitative studies to be considered for developing evidence, and provide useful links to relevant software, on-line applications, websites, and give practical advice. We also provide advice on patient and public involvement and engagement (PPIE), and on product management. Our aim is to help device manufacturers to understand the concepts and terminology used in evaluation of in vitro diagnostics (IVDs) so that they can communicate effectively with evaluation methodologists, statisticians, and health economists. Manufacturers of medical tests and devices can use the proposed framework to plan their evidence development strategy in alignment with device development, applications for regulatory approval, and publication.