A Tablet Based Embodied Conversational Agent to Promote Smoking Cessation among Veterans: A Feasibility Study

Roles, Users, Benefits, and Limitations of Chatbots in Health Care: Rapid Review

BACKGROUND

Chatbots, or conversational agents, have emerged as significant tools in health care, driven by advancements in artificial intelligence and digital technology. These programs are designed to simulate human conversations, addressing various health care needs. However, no comprehensive synthesis of health care chatbots' roles, users, benefits, and limitations is available to inform future research and application in the field.

OBJECTIVE

This review aims to describe health care chatbots' characteristics, focusing on their diverse roles in the health care pathway, user groups, benefits, and limitations.

METHODS

A rapid review of published literature from 2017 to 2023 was performed with a search strategy developed in collaboration with a health sciences librarian and implemented in the MEDLINE and Embase databases. Primary research studies reporting on chatbot roles or benefits in health care were included. Two reviewers dual-screened the search results. Extracted data on chatbot roles, users, benefits, and limitations were subjected to content analysis.

RESULTS

The review categorized chatbot roles into 2 themes: delivery of remote health services, including patient support, care management, education, skills building, and health behavior promotion, and provision of administrative assistance to health care providers. User groups spanned across patients with chronic conditions as well as patients with cancer; individuals focused on lifestyle improvements; and various demographic groups such as women, families, and older adults. Professionals and students in health care also emerged as significant users, alongside groups seeking mental health support, behavioral change, and educational enhancement. The benefits of health care chatbots were also classified into 2 themes: improvement of health care quality and efficiency and cost-effectiveness in health care delivery. The identified limitations encompassed ethical challenges, medicolegal and safety concerns, technical difficulties, user experience issues, and societal and economic impacts.

CONCLUSIONS

Health care chatbots offer a wide spectrum of applications, potentially impacting various aspects of health care. While they are promising tools for improving health care efficiency and quality, their integration into the health care system must be approached with consideration of their limitations to ensure optimal, safe, and equitable use.

Development of a Virtual Human for Supporting Tobacco Cessation During the COVID-19 Pandemic

People who consume tobacco are at greater risk of developing severe COVID-19. Unfortunately, the COVID-19 pandemic reduced the accessibility of tobacco cessation services as a result of necessary social restrictions. Innovations were urgently needed to support tobacco cessation during the pandemic. Virtual humans are artificially intelligent computer agents with a realistic, humanlike appearance. Virtual humans could be a scalable and engaging way to deliver tobacco cessation information and support. Florence, a virtual human health worker, was developed in collaboration with the World Health Organization to remotely support people toward tobacco cessation during the COVID-19 pandemic. Florence delivers evidence-based information, assists with making quit plans, and directs people to World Health Organization-recommended cessation services in their country. In this viewpoint, we describe the process of developing Florence. The development was influenced by a formative evaluation of data from 115 early users of Florence from 49 countries. In general, Florence was positively perceived; however, changes were requested to aspects of her design and content. In addition, areas for new content were identified (eg, for nonsmoker support persons). Virtual health workers could expand the reach of evidence-based tobacco cessation information and personalized support. However, as they are a new innovation in tobacco cessation, their efficacy, feasibility, and acceptability in this application needs to be evaluated, including in diverse populations.

Factors Influencing the Acceptability, Acceptance, and Adoption of Conversational Agents in Health Care: Integrative Review

BACKGROUND

Conversational agents (CAs), also known as chatbots, are digital dialog systems that enable people to have a text-based, speech-based, or nonverbal conversation with a computer or another machine based on natural language via an interface. The use of CAs offers new opportunities and various benefits for health care. However, they are not yet ubiquitous in daily practice. Nevertheless, research regarding the implementation of CAs in health care has grown tremendously in recent years.

OBJECTIVE

This review aims to present a synthesis of the factors that facilitate or hinder the implementation of CAs from the perspectives of patients and health care professionals. Specifically, it focuses on the early implementation outcomes of acceptability, acceptance, and adoption as cornerstones of later implementation success.

METHODS

We performed an integrative review. To identify relevant literature, a broad literature search was conducted in June 2021 with no date limits and using all fields in PubMed, Cochrane Library, Web of Science, LIVIVO, and PsycINFO. To keep the review current, another search was conducted in March 2022. To identify as many eligible primary sources as possible, we used a snowballing approach by searching reference lists and conducted a hand search. Factors influencing the acceptability, acceptance, and adoption of CAs in health care were coded through parallel deductive and inductive approaches, which were informed by current technology acceptance and adoption models. Finally, the factors were synthesized in a thematic map.

RESULTS

Overall, 76 studies were included in this review. We identified influencing factors related to 4 core Unified Theory of Acceptance and Use of Technology (UTAUT) and Unified Theory of Acceptance and Use of Technology 2 (UTAUT2) factors (performance expectancy, effort expectancy, facilitating conditions, and hedonic motivation), with most studies underlining the relevance of performance and effort expectancy. To meet the particularities of the health care context, we redefined the UTAUT2 factors social influence, habit, and price value. We identified 6 other influencing factors: perceived risk, trust, anthropomorphism, health issue, working alliance, and user characteristics. Overall, we identified 10 factors influencing acceptability, acceptance, and adoption among health care professionals (performance expectancy, effort expectancy, facilitating conditions, social influence, price value, perceived risk, trust, anthropomorphism, working alliance, and user characteristics) and 13 factors influencing acceptability, acceptance, and adoption among patients (additionally hedonic motivation, habit, and health issue).

CONCLUSIONS

This review shows manifold factors influencing the acceptability, acceptance, and adoption of CAs in health care. Knowledge of these factors is fundamental for implementation planning. Therefore, the findings of this review can serve as a basis for future studies to develop appropriate implementation strategies. Furthermore, this review provides an empirical test of current technology acceptance and adoption models and identifies areas where additional research is necessary.

TRIAL REGISTRATION

PROSPERO CRD42022343690; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=343690.

Effectiveness and Acceptability of Conversational Agents for Smoking Cessation: A Systematic Review and Meta-analysis

INTRODUCTION

Conversational agents (CAs; computer programs that use artificial intelligence to simulate a conversation with users through natural language) have evolved considerably in recent years to support healthcare by providing autonomous, interactive, and accessible services, making them potentially useful for supporting smoking cessation. We performed a systematic review and meta-analysis to provide an overarching evaluation of their effectiveness and acceptability to inform future development and adoption.

AIMS AND METHODS

PsycInfo, Web of Science, ACM Digital Library, IEEE Xplore, Medline, EMBASE, Communication and Mass Media Complete, and CINAHL Complete were searched for studies examining the use of CAs for smoking cessation. Data from eligible studies were extracted and used for random-effects meta-analyses.

RESULTS

The search yielded 1245 publications with 13 studies eligible for systematic review (total N = 8236) and six studies for random-effects meta-analyses. All studies reported positive effects on cessation-related outcomes. A meta-analysis with randomized controlled trials reporting on abstinence yielded a sample-weighted odds ratio of 1.66 (95% CI = 1.33% to 2.07%, p < .001), favoring CAs over comparison groups. A narrative synthesis of all included studies showed overall high acceptability, while some barriers were identified from user feedback. Overall, included studies were diverse in design with mixed quality, and evidence of publication bias was identified. A lack of theoretical foundations was noted, as well as a clear need for relational communication in future designs.

CONCLUSIONS

The effectiveness and acceptability of CAs for smoking cessation are promising. However, standardization of reporting and designing of the agents is warranted for a more comprehensive evaluation.

IMPLICATIONS

This is the first systematic review to provide insight into the use of CAs to support smoking cessation. Our findings demonstrated initial promise in the effectiveness and user acceptability of these agents. We also identified a lack of theoretical and methodological limitations to improve future study design and intervention delivery.

Design, development, and use of conversational agents in rehabilitation for adults with brain-related neurological conditions: a scoping review

OBJECTIVE

The objective of this review was to identify how conversational agents are designed and used in rehabilitation for adults with brain-related neurological conditions.

INTRODUCTION

Adults with brain-related neurological conditions experience varied cognitive and functional challenges that can persist long term. However, rehabilitation services are time- and resource-limited, and novel rehabilitation approaches are warranted. Conversational agents provide a human-computer interface with which the user can converse. A conversational agent can be designed to meet specific user needs, such as rehabilitation and support.

INCLUSION CRITERIA

Studies focused on the design and use of conversational agents for rehabilitation for people aged 18 years or older with brain-related neurological conditions were considered for inclusion. Eligible publication types included peer-reviewed publications (quantitative, qualitative, and/or mixed methods study designs; research protocols; peer-reviewed expert opinion papers; clinical studies, including pilot trials; systematic or scoping reviews), full conference papers, and master's or PhD theses. Eligible types of research included prototype development, feasibility testing, and clinical trials.

METHODS

Online databases, including MEDLINE, Scopus, ProQuest (all databases), Web of Science, and gray literature sources were searched with no date limit. Only English publications were considered due to a lack of resourcing available for translations. Title and abstract screening and full-text review were conducted by two independent reviewers. Data extraction was shared by three independent reviewers. The data extraction instrument was iteratively refined to meet the requirements of all included papers, and covered details for technological aspects and the clinical context. Results are presented narratively and in tabular format, with emphasis on participants, concept and context, and data extraction instrument components.

RESULTS

Eleven papers were included in the review, which represented seven distinct conversational agent prototypes. Methodologies included technology description (n = 9) and initial user testing (n = 6). The intended clinical cohorts for the reported conversational agents were people with dementia (n = 5), Parkinson disease (n = 2), stroke (n = 1), traumatic brain injury (n = 1), mixed dementia and mild cognitive impairment (n = 1), and mixed dementia and Parkinson disease (n = 1). Two studies included participants who were healthy or otherwise from the general community. The design of the conversational agents considered technology aspects and clinical purposes. Two conversational agent prototypes incorporated a speaking humanoid avatar as reported in five of the papers. Topics of conversation focused on subjects enjoyable to the user (life history, hobbies, where they lived). The clinical purposes reported in the 11 papers were to increase the amount of conversation the user has each day (n = 4), reminiscence (n = 2), and one study each for anxiety management and education, Parkinson disease education, to obtain and analyze a recording of the user's voice, to monitor well-being, and to build rapport before providing daily task prompts. One study reported clinician oversight of the conversational agent use. The studies had low sample sizes (range: 1-33). No studies undertook effectiveness testing. Outcome measures focused on usability, language detection and production, and technological performance. No health-related outcomes were measured. No adverse events were reported, and only two studies reported safety considerations.

CONCLUSIONS

Current literature reporting the design and use of conversational agents for rehabilitation for adults with brain-related neurological conditions is heterogeneous and represents early stages of conversational agent development and testing. We recommend, as per our customized data extraction instrument, that studies of conversational agents for this population clearly define technical aspects, methodology for developing the conversation content, recruitment methods, safety issues, and requirements for clinician oversight.

Chatbots for Smoking Cessation: Scoping Review

BACKGROUND

Despite significant progress in reducing tobacco use over the past 2 decades, tobacco still kills over 8 million people every year. Digital interventions, such as text messaging, have been found to help people quit smoking. Chatbots, or conversational agents, are new digital tools that mimic instantaneous human conversation and therefore could extend the effectiveness of text messaging.

OBJECTIVE

This scoping review aims to assess the extent of research in the chatbot literature for smoking cessation and provide recommendations for future research in this area.

METHODS

Relevant studies were identified through searches conducted in Embase, MEDLINE, APA PsycINFO, Google Scholar, and Scopus, as well as additional searches on JMIR, Cochrane Library, Lancet Digital Health, and Digital Medicine. Studies were considered if they were conducted with tobacco smokers, were conducted between 2000 and 2021, were available in English, and included a chatbot intervention.

RESULTS

Of 323 studies identified, 10 studies were included in the review (3 framework articles, 1 study protocol, 2 pilot studies, 2 trials, and 2 randomized controlled trials). Most studies noted some benefits related to smoking cessation and participant engagement; however, outcome measures varied considerably. The quality of the studies overall was low, with methodological issues and low follow-up rates.

CONCLUSIONS

More research is needed to make a firm conclusion about the efficacy of chatbots for smoking cessation. Researchers need to provide more in-depth descriptions of chatbot functionality, mode of delivery, and theoretical underpinnings. Consistency in language and terminology would also assist in reviews of what approaches work across the field.

Factors Influencing Adherence to mHealth Apps for Prevention or Management of Noncommunicable Diseases: Systematic Review

Background

Mobile health (mHealth) apps show vast potential in supporting patients and health care systems with the increasing prevalence and economic costs of noncommunicable diseases (NCDs) worldwide. However, despite the availability of evidence-based mHealth apps, a substantial proportion of users do not adhere to them as intended and may consequently not receive treatment. Therefore, understanding the factors that act as barriers to or facilitators of adherence is a fundamental concern in preventing intervention dropouts and increasing the effectiveness of digital health interventions.

Objective

This review aimed to help stakeholders develop more effective digital health interventions by identifying factors influencing the continued use of mHealth apps targeting NCDs. We further derived quantified adherence scores for various health domains to validate the qualitative findings and explore adherence benchmarks.

Methods

A comprehensive systematic literature search (January 2007 to December 2020) was conducted on MEDLINE, Embase, Web of Science, Scopus, and ACM Digital Library. Data on intended use, actual use, and factors influencing adherence were extracted. Intervention-related and patient-related factors with a positive or negative influence on adherence are presented separately for the health domains of NCD self-management, mental health, substance use, nutrition, physical activity, weight loss, multicomponent lifestyle interventions, mindfulness, and other NCDs. Quantified adherence measures, calculated as the ratio between the estimated intended use and actual use, were derived for each study and compared with the qualitative findings.

Results

The literature search yielded 2862 potentially relevant articles, of which 99 (3.46%) were included as part of the inclusion criteria. A total of 4 intervention-related factors indicated positive effects on adherence across all health domains: personalization or tailoring of the content of mHealth apps to the individual needs of the user, reminders in the form of individualized push notifications, user-friendly and technically stable app design, and personal support complementary to the digital intervention. Social and gamification features were also identified as drivers of app adherence across several health domains. A wide variety of patient-related factors such as user characteristics or recruitment channels further affects adherence. The derived adherence scores of the included mHealth apps averaged 56.0% (SD 24.4%).

Conclusions

This study contributes to the scarce scientific evidence on factors that positively or negatively influence adherence to mHealth apps and is the first to quantitatively compare adherence relative to the intended use of various health domains. As underlying studies mostly have a pilot character with short study durations, research on factors influencing adherence to mHealth apps is still limited. To facilitate future research on mHealth app adherence, researchers should clearly outline and justify the app’s intended use; report objective data on actual use relative to the intended use; and, ideally, provide long-term use and retention data.

“I don't Know what you Mean by ‘I am Anxious’”: A New Method for Evaluating Conversational Agent Responses to Standardized Mental Health Inputs for Anxiety and Depression

Conversational agents (CAs) are increasingly ubiquitous and are now commonly used to access medical information. However, we lack systematic data about the quality of advice such agents provide. This paper evaluates CA advice for mental health (MH) questions, a pressing issue given that we are undergoing a mental health crisis. Building on prior work, we define a new method to systematically evaluate mental health responses from CAs. We develop multi-utterance conversational probes derived from two widely used mental health diagnostic surveys, the PHQ-9 (Depression) and the GAD-7 (Anxiety). We evaluate the responses of two text-based chatbots and four voice assistants to determine whether CAs provide relevant responses and treatments. Evaluations were conducted both by clinicians and immersively by trained raters, yielding consistent results across all raters. Although advice and recommendations were generally low quality, they were better for Crisis probes and for probes concerning symptoms of Anxiety rather than Depression. Responses were slightly improved for text versus speech-based agents, and when CAs had access to extended dialogue context. Design implications include suggestions for improved responses through clarification sub-dialogues. Responses may also be improved by the incorporation of empathy although this needs to be combined with effective treatments or advice.

Automatic topic selection for long-term interaction with embodied conversational agents in health coaching: A micro-randomized trial

INTRODUCTION

Embodied Conversational Agents (ECAs) can be included in health coaching applications as virtual coaches. The engagement with these virtual coaches could be improved by presenting users with tailored coaching dialogues. In this article, we investigate if the suggestion of an automatically tailored topic by an ECA leads to higher engagement by the user and thus longer sessions of interaction.

METHODS

A Micro-Randomized Trial (MRT) was conducted in which two types of interaction with an ECA were compared: (a) the coach suggests a relevant topic to discuss, and (b) the coach asks the user to select a topic from a set of options. Every time the user would interact with the ECA, one of those conditions would be randomly selected. Participants interacted in their daily life with the ECA that was part of a multi-agent health coaching application for 4-8 weeks.

RESULTS

In two rounds, 82 participants interacted with the micro-randomized coach a total of 1011 times. Interactions in which the coach took the initiative were found to be of equal length as interactions in which the user was allowed to choose the topic, and the acceptance of topic suggestions was high (71.1% overall, 75.8% for coaching topics).

CONCLUSION

Tailoring coaching conversations with ECAs by letting the coach automatically suggest a topic that is tailored to the user is perceived as a natural variation in the flow of interaction. Future research could focus on improving the novel coaching engine component that supports the topic selection process for these suggestions or on investigating how the amount of initiative and coaching approach by the ECA could be tailored.

A Review of Virtual Coaching Systems in Healthcare: Closing the Loop With Real-Time Feedback

This review focuses on virtual coaching systems that were designed to enhance healthcare interventions, combining the available sensing and system-user interaction technologies. In total, more than 1,200 research papers have been retrieved and evaluated for the purposes of this review, which were obtained from three online databases (i.e.,PubMed, Scopus and IEEE Xplore) using an extensive set of search keywords. After applying exclusion criteria, the remaining 41 research papers were used to evaluate the status of virtual coaching systems over the past 10 years and assess current and future trends in this field. The results suggest that in home coaching systems were mainly focused in promoting physical activity and a healthier lifestyle, while a wider range of medical domains was considered in systems that were evaluated in lab environment. In home patient monitoring with IoT devices and sensors was mostly limited to activity trackers, pedometers and heart rate monitoring. Real-time evaluations and personalized patient feedback was also found to be rather lacking in home coaching systems and this is the most alarming find of this analysis. Feasibility studies in controlled environment and an ongoing active research on Horizon 2020 funded projects, show that the future trends in this field are aiming to close the loop with automated patient monitoring, real-time evaluations and more precise interventions.

Technical Metrics Used to Evaluate Health Care Chatbots: Scoping Review

BACKGROUND

Dialog agents (chatbots) have a long history of application in health care, where they have been used for tasks such as supporting patient self-management and providing counseling. Their use is expected to grow with increasing demands on health systems and improving artificial intelligence (AI) capability. Approaches to the evaluation of health care chatbots, however, appear to be diverse and haphazard, resulting in a potential barrier to the advancement of the field.

OBJECTIVE

This study aims to identify the technical (nonclinical) metrics used by previous studies to evaluate health care chatbots.

METHODS

Studies were identified by searching 7 bibliographic databases (eg, MEDLINE and PsycINFO) in addition to conducting backward and forward reference list checking of the included studies and relevant reviews. The studies were independently selected by two reviewers who then extracted data from the included studies. Extracted data were synthesized narratively by grouping the identified metrics into categories based on the aspect of chatbots that the metrics evaluated.

RESULTS

Of the 1498 citations retrieved, 65 studies were included in this review. Chatbots were evaluated using 27 technical metrics, which were related to chatbots as a whole (eg, usability, classifier performance, speed), response generation (eg, comprehensibility, realism, repetitiveness), response understanding (eg, chatbot understanding as assessed by users, word error rate, concept error rate), and esthetics (eg, appearance of the virtual agent, background color, and content).

CONCLUSIONS

The technical metrics of health chatbot studies were diverse, with survey designs and global usability metrics dominating. The lack of standardization and paucity of objective measures make it difficult to compare the performance of health chatbots and could inhibit advancement of the field. We suggest that researchers more frequently include metrics computed from conversation logs. In addition, we recommend the development of a framework of technical metrics with recommendations for specific circumstances for their inclusion in chatbot studies.

Effectiveness of Conversational Agents (Virtual Assistants) in Health Care: Protocol for a Systematic Review

Background

Conversational agents (also known as chatbots) have evolved in recent decades to become multimodal, multifunctional platforms with potential to automate a diverse range of health-related activities supporting the general public, patients, and physicians. Multiple studies have reported the development of these agents, and recent systematic reviews have described the scope of use of conversational agents in health care. However, there is scarce research on the effectiveness of these systems; thus, their viability and applicability are unclear.

Objective

The objective of this systematic review is to assess the effectiveness of conversational agents in health care and to identify limitations, adverse events, and areas for future investigation of these agents.

Methods

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocols will be used to structure this protocol. The focus of the systematic review is guided by a population, intervention, comparator, and outcome framework. A systematic search of the PubMed (Medline), EMBASE, CINAHL, and Web of Science databases will be conducted. Two authors will independently screen the titles and abstracts of the identified references and select studies according to the eligibility criteria. Any discrepancies will then be discussed and resolved. Two reviewers will independently extract and validate data from the included studies into a standardized form and conduct quality appraisal.

Results

As of January 2020, we have begun a preliminary literature search and piloting of the study selection process.

Conclusions

This systematic review aims to clarify the effectiveness, limitations, and future applications of conversational agents in health care. Our findings may be useful to inform the future development of conversational agents and promote the personalization of patient care.

International Registered Report Identifier (IRRID)

PRR1-10.2196/16934

Technical Metrics Used to Evaluate Health Care Chatbots: Scoping Review (Preprint).. [DOI: 10.2196/preprints.18301] [Citation(s) in RCA: 0] [Impact Index Per Article: 0]

BACKGROUND

Dialog agents (chatbots) have a long history of application in health care, where they have been used for tasks such as supporting patient self-management and providing counseling. Their use is expected to grow with increasing demands on health systems and improving artificial intelligence (AI) capability. Approaches to the evaluation of health care chatbots, however, appear to be diverse and haphazard, resulting in a potential barrier to the advancement of the field.

OBJECTIVE

This study aims to identify the technical (nonclinical) metrics used by previous studies to evaluate health care chatbots.

METHODS

Studies were identified by searching 7 bibliographic databases (eg, MEDLINE and PsycINFO) in addition to conducting backward and forward reference list checking of the included studies and relevant reviews. The studies were independently selected by two reviewers who then extracted data from the included studies. Extracted data were synthesized narratively by grouping the identified metrics into categories based on the aspect of chatbots that the metrics evaluated.

RESULTS

Of the 1498 citations retrieved, 65 studies were included in this review. Chatbots were evaluated using 27 technical metrics, which were related to chatbots as a whole (eg, usability, classifier performance, speed), response generation (eg, comprehensibility, realism, repetitiveness), response understanding (eg, chatbot understanding as assessed by users, word error rate, concept error rate), and esthetics (eg, appearance of the virtual agent, background color, and content).

CONCLUSIONS

The technical metrics of health chatbot studies were diverse, with survey designs and global usability metrics dominating. The lack of standardization and paucity of objective measures make it difficult to compare the performance of health chatbots and could inhibit advancement of the field. We suggest that researchers more frequently include metrics computed from conversation logs. In addition, we recommend the development of a framework of technical metrics with recommendations for specific circumstances for their inclusion in chatbot studies.

Promoting Relational Agent for Health Behavior Change in Low and Middle - Income Countries (LMICs): Issues and Approaches

The use of contemporary technologies in healthcare systems to improve quality of care and to promote behavioral healthcare outcomes are prevalent in high-income countries. However, low and middle-income countries (LMICs) are not receiving the same advantages of technology, which may be due to inadequate technological infrastructure and financial resources, lack of interest among policy makers and healthcare service providers, lack of skills and capacity among healthcare professionals in using technology based interventions, and resistance of the public to the use of technologies for healthcare or health promotion activities. Technology-based interventions offer considerable promise to develop entirely new models of healthcare both within and outside of formal systems of care and offer the opportunity to have a large public health impact. Such technology-based interventions could be used to address targeted global health problems in LMICs, including the chronic non-communicable diseases (NCDs) - a growing health system burden in LMICs. Major preventable behavioral risk factors of chronic NCDs are increasing in LMICs, and innovative interventions are essential to address these risk factors. Computer-based or mobile-based virtual coaches or Relational Agents (RAs) are increasingly being explored for counseling patients to change their health behavior in high-income countries; however, the use of RAs in LMICs has not been studied. In this paper, we summarize the growing application of RA technology in behavior change interventions in high-income countries and describe the potential of its use in LMICs. Finally, we review the potential barriers and challenges in promoting RAs in LMICs.