1
|
Amin S, Uyeda K, Pagano I, Tabangcura KR, Taketa R, Terawaki Kawamoto C, Pokhrel P. Virtual Assistants' Response to Queries About Nicotine Replacement Therapy: A Mixed-Method Analysis. Eval Health Prof 2024:1632787241235689. [PMID: 38408450 DOI: 10.1177/01632787241235689] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/28/2024]
Abstract
This study focused on investigating the potential of Artificial Intelligent-powered Virtual Assistants (VAs) such as Amazon Alexa, Apple Siri, and Google Assistant as tools to help individuals seeking information about Nicotine Replacement Treatment (NRT) for smoking cessation. The researchers asked 40 NRT-related questions to each of the 3 VAs and evaluated the responses for voice recognition. The study used a cross-sectional mixed-method design with a total sample size of 360 responses. Inter-rater reliability and differences between VAs' responses were examined by SAS software, and qualitative assessments were conducted using NVivo software. Google Assistant achieved 100% voice recognition for NRT-related questions, followed by Apple Siri at 97.5%, and Amazon Alexa at 83.3%. Statistically significant differences were found between the responses of Amazon Alexa relative to both Google Assistant and Apple Siri. Researcher 1's ratings significantly differed from Researcher 2's (p = .001), but not from Researcher 3's (p = .11). Virtual Assistants occasionally struggled to understand the context or nuances of questions, lacked in-depth information in their responses, and provided generic or unrelated responses. Virtual Assistants have the potential to be incorporated into smoking cessation interventions and tobacco control initiatives, contingent upon improving their competencies.
Collapse
Affiliation(s)
- Samia Amin
- University of Hawai'i Cancer Center, USA
| | | | - Ian Pagano
- University of Hawai'i Cancer Center, USA
| | | | | | | | | |
Collapse
|
2
|
Ding H, Simmich J, Vaezipour A, Andrews N, Russell T. Evaluation framework for conversational agents with artificial intelligence in health interventions: a systematic scoping review. J Am Med Inform Assoc 2024; 31:746-761. [PMID: 38070173 PMCID: PMC10873847 DOI: 10.1093/jamia/ocad222] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2023] [Revised: 11/04/2023] [Accepted: 11/24/2023] [Indexed: 02/18/2024] Open
Abstract
OBJECTIVES Conversational agents (CAs) with emerging artificial intelligence present new opportunities to assist in health interventions but are difficult to evaluate, deterring their applications in the real world. We aimed to synthesize existing evidence and knowledge and outline an evaluation framework for CA interventions. MATERIALS AND METHODS We conducted a systematic scoping review to investigate designs and outcome measures used in the studies that evaluated CAs for health interventions. We then nested the results into an overarching digital health framework proposed by the World Health Organization (WHO). RESULTS The review included 81 studies evaluating CAs in experimental (n = 59), observational (n = 15) trials, and other research designs (n = 7). Most studies (n = 72, 89%) were published in the past 5 years. The proposed CA-evaluation framework includes 4 evaluation stages: (1) feasibility/usability, (2) efficacy, (3) effectiveness, and (4) implementation, aligning with WHO's stepwise evaluation strategy. Across these stages, this article presents the essential evidence of different study designs (n = 8), sample sizes, and main evaluation categories (n = 7) with subcategories (n = 40). The main evaluation categories included (1) functionality, (2) safety and information quality, (3) user experience, (4) clinical and health outcomes, (5) costs and cost benefits, (6) usage, adherence, and uptake, and (7) user characteristics for implementation research. Furthermore, the framework highlighted the essential evaluation areas (potential primary outcomes) and gaps across the evaluation stages. DISCUSSION AND CONCLUSION This review presents a new framework with practical design details to support the evaluation of CA interventions in healthcare research. PROTOCOL REGISTRATION The Open Science Framework (https://osf.io/9hq2v) on March 22, 2021.
Collapse
Affiliation(s)
- Hang Ding
- RECOVER Injury Research Centre, Faculty of Health and Behavioural Sciences, The University of Queensland, Brisbane, QLD, Australia
- STARS Education and Research Alliance, Surgical Treatment and Rehabilitation Service (STARS), The University of Queensland and Metro North Health, Brisbane, QLD, Australia
| | - Joshua Simmich
- RECOVER Injury Research Centre, Faculty of Health and Behavioural Sciences, The University of Queensland, Brisbane, QLD, Australia
- STARS Education and Research Alliance, Surgical Treatment and Rehabilitation Service (STARS), The University of Queensland and Metro North Health, Brisbane, QLD, Australia
| | - Atiyeh Vaezipour
- RECOVER Injury Research Centre, Faculty of Health and Behavioural Sciences, The University of Queensland, Brisbane, QLD, Australia
- STARS Education and Research Alliance, Surgical Treatment and Rehabilitation Service (STARS), The University of Queensland and Metro North Health, Brisbane, QLD, Australia
| | - Nicole Andrews
- RECOVER Injury Research Centre, Faculty of Health and Behavioural Sciences, The University of Queensland, Brisbane, QLD, Australia
- STARS Education and Research Alliance, Surgical Treatment and Rehabilitation Service (STARS), The University of Queensland and Metro North Health, Brisbane, QLD, Australia
- The Tess Cramond Pain and Research Centre, Metro North Hospital and Health Service, Brisbane, QLD, Australia
- The Occupational Therapy Department, The Royal Brisbane and Women’s Hospital, Metro North Hospital and Health Service, Brisbane, QLD, Australia
| | - Trevor Russell
- RECOVER Injury Research Centre, Faculty of Health and Behavioural Sciences, The University of Queensland, Brisbane, QLD, Australia
- STARS Education and Research Alliance, Surgical Treatment and Rehabilitation Service (STARS), The University of Queensland and Metro North Health, Brisbane, QLD, Australia
| |
Collapse
|
3
|
Amin S, Kawamoto CT, Pokhrel P. Exploring the ChatGPT platform with scenario-specific prompts for vaping cessation. Tob Control 2023:tc-2023-058009. [PMID: 37460216 PMCID: PMC10792116 DOI: 10.1136/tc-2023-058009] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2023] [Accepted: 06/28/2023] [Indexed: 07/28/2023]
Abstract
OBJECTIVE To evaluate and start a discussion on the potential usefulness of applying Artificial Intelligence (AI)-driven natural language processing technology such as the ChatGPT in tobacco control efforts, specifically vaping cessation. METHOD Ten real-world questions about vaping cessation were selected from a Reddit forum and used as ChatGPT prompts or queries. Content analysis was performed on the ChatGPT responses to identify the thematic aspects of vaping cessation support represented in the responses. Next, the responses were empirically evaluated by five experts in tobacco control on accuracy, quality, clarity, and empathy. RESULT The following themes related to vaping cessation support were identified: understanding nicotine withdrawal symptoms, self-regulation, peer support, motivational support, and Nicotine Replacement Therapy (NRT). The experts judged the ChatGPT responses to be 'satisfactory' to 'excellent' in areas of accuracy, quality, clarity, and empathy. CONCLUSION If managed by a group of experts, including clinicians, and behavioral and computer scientists, a platform such as the ChatGPT may be leveraged to design tailored interventions for tobacco use cessation, including vaping cessation.
Collapse
Affiliation(s)
- Samia Amin
- Population Sciences in the Pacific Program, University of Hawai'i Cancer Center, Honolulu, Hawaii, USA
| | - Crissy Terawaki Kawamoto
- Population Sciences in the Pacific Program, University of Hawai'i Cancer Center, Honolulu, Hawaii, USA
| | - Pallav Pokhrel
- Population Sciences in the Pacific Program, University of Hawai'i Cancer Center, Honolulu, Hawaii, USA
| |
Collapse
|
4
|
Esposito CP, Schindler-Ruwisch J. "Alexa, did the pandemic make you smarter?" A follow up content analysis of a virtual assistant's responses to a prenatal query. Inform Health Soc Care 2023; 48:231-238. [PMID: 35997330 DOI: 10.1080/17538157.2022.2110107] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
Abstract
To compare responses to 40 common prenatal questions from Amazon's virtual assistant, Alexa, one year apart during the COVID pandemic. Participants: Two researchers replicated a prenatal query using unique Alexa devices. A conceptual content analysis was conducted where the researchers independently queried Alexa the identical questions from their 2020 study during the same one-week timeframe, between May 20, 2021 and May 27, 2021. Alexa's responses were compared to the 2020 study and the American College of Obstetricians and Gynecologists data and verified by one of the researchers, a Certified Nurse Midwife. Alexa provided accurate responses to 26 (65%) of the questions, an increase by 55 percentage points from 2020. Alexa was able to recite the symptoms of COVID-19 illness but was unable to provide a response to the two other COVID-specific questions. Compared to the 2020 query, Alexa provided more reputable sources for the responses including the CDC, WHO, NIH, and Mayo Clinic. Alexa's ability to provide more accurate, evidence-based responses was remarkably improved in 2021. Mobile health tools, like Amazon Alexa, are highly utilized by the public, particularly with limited healthcare access during the COVID-19 pandemic. Technology-based platforms should provide credible, evidence-based content.
Collapse
|
5
|
Avila FR, Boczar D, Spaulding AC, Quest DJ, Samanta A, Torres-Guzman RA, Maita KC, Garcia JP, Eldaly AS, Forte AJ. High Satisfaction With a Virtual Assistant for Plastic Surgery Frequently Asked Questions. Aesthet Surg J 2023; 43:494-503. [PMID: 36353923 DOI: 10.1093/asj/sjac290] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/22/2022] [Revised: 11/03/2022] [Accepted: 11/03/2022] [Indexed: 11/11/2022] Open
Abstract
BACKGROUND Most of a surgeon's office time is dedicated to patient education, preventing an appropriate patient-physician relationship. Telephone-accessed artificial intelligent virtual assistants (AIVAs) that simulate a human conversation and answer preoperative frequently asked questions (FAQs) can be effective solutions to this matter. An AIVA capable of answering preoperative plastic surgery-related FAQs has previously been described by the authors. OBJECTIVES The aim of this paper was to determine patients' perception and satisfaction with an AIVA. METHODS Twenty-six adult patients from a plastic surgery service answered a 3-part survey consisting of: (1) an evaluation of the answers' correctness, (2) their agreement with the feasibility, usefulness, and future uses of the AIVA, and (3) a section on comments. The first part made it possible to measure the system's accuracy, and the second to evaluate perception and satisfaction. The data were analyzed with Microsoft Excel 2010 (Microsoft Corporation, Redmond, WA). RESULTS The AIVA correctly answered the patients' questions 98.5% of the time, and the topic with the lowest accuracy was "nausea." Additionally, 88% of patients agreed with the statements of the second part of the survey. Thus, the patients' perception was positive and overall satisfaction with the AIVA was high. Patients agreed the least with using the AIVA to select their surgical procedure. The comments provided improvement areas for subsequent stages of the project. CONCLUSIONS The results show that patients were satisfied and expressed a positive experience with using the AIVA to answer plastic surgery FAQs before surgery. The system is also highly accurate.
Collapse
|
6
|
Mavragani A, Hong G, Devon-Sand A, Lin S. Voice Assistants' Responses to Questions About the COVID-19 Vaccine: National Cross-sectional Study. JMIR Form Res 2023; 7:e43007. [PMID: 36719815 PMCID: PMC9947766 DOI: 10.2196/43007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/27/2022] [Revised: 01/24/2023] [Accepted: 01/29/2023] [Indexed: 01/31/2023] Open
Abstract
BACKGROUND Artificial intelligence-powered voice assistants (VAs), such as Apple Siri, Google Assistant, and Amazon Alexa, interact with users in natural language and are capable of responding to simple commands, searching the internet, and answering questions. Despite being an increasingly popular way for the public to access health information, VAs could be a source of ambiguous or potentially biased information. OBJECTIVE In response to the ongoing prevalence of vaccine misinformation and disinformation, this study aims to evaluate how smartphone VAs respond to information- and recommendation-seeking inquiries regarding the COVID-19 vaccine. METHODS A national cross-sectional survey of English-speaking adults who owned a smartphone with a VA installed was conducted online from April 22 to 28, 2021. The primary outcomes were the VAs' responses to 2 questions: "Should I get the COVID vaccine?" and "Is the COVID vaccine safe?" Directed content analysis was used to assign a negative, neutral, or positive connotation to each response and website title provided by the VAs. Statistical significance was assessed using the t test (parametric) or Mann-Whitney U (nonparametric) test for continuous variables and the chi-square or Fisher exact test for categorical variables. RESULTS Of the 466 survey respondents included in the final analysis, 404 (86.7%) used Apple Siri, 53 (11.4%) used Google Assistant, and 9 (1.9%) used Amazon Alexa. In response to the question "Is the COVID vaccine safe?" 419 (89.9%) users received a direct response, of which 408 (97.3%) had a positive connotation encouraging users to get vaccinated. Of the websites presented, only 5.3% (11/207) had a positive connotation and 94.7% (196/207) had a neutral connotation. In response to the question "Should I get the COVID vaccine?" 93.1% (434/466) of users received a list of websites, of which 91.5% (1155/1262) had a neutral connotation. For both COVID-19 vaccine-related questions, there was no association between the connotation of a response and the age, gender, zip code, race or ethnicity, and education level of the respondent. CONCLUSIONS Our study found that VAs were much more likely to respond directly with positive connotations to the question "Is the COVID vaccine safe?" but not respond directly and provide a list of websites with neutral connotations to the question "Should I get the COVID vaccine?" To our knowledge, this is the first study to evaluate how VAs respond to both information- and recommendation-seeking inquiries regarding the COVID-19 vaccine. These findings add to our growing understanding of both the opportunities and pitfalls of VAs in supporting public health information dissemination.
Collapse
Affiliation(s)
- Amaryllis Mavragani
- Department of MedicineStanford University School of MedicinePalo Alto, CAUnited States
| | - Grace Hong
- Department of Medicine, Stanford University School of Medicine, Palo Alto, CA, United States
| | - Anna Devon-Sand
- Department of Medicine, Stanford University School of Medicine, Palo Alto, CA, United States
| | - Steven Lin
- Department of Medicine, Stanford University School of Medicine, Palo Alto, CA, United States
| |
Collapse
|
7
|
Bickmore TW, Ólafsson S, O'Leary TK. Mitigating Patient and Consumer Safety Risks When Using Conversational Assistants for Medical Information: Exploratory Mixed Methods Experiment. J Med Internet Res 2021; 23:e30704. [PMID: 34751661 PMCID: PMC8663571 DOI: 10.2196/30704] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/26/2021] [Revised: 08/29/2021] [Accepted: 09/18/2021] [Indexed: 02/01/2023] Open
Abstract
BACKGROUND Prior studies have demonstrated the safety risks when patients and consumers use conversational assistants such as Apple's Siri and Amazon's Alexa for obtaining medical information. OBJECTIVE The aim of this study is to evaluate two approaches to reducing the likelihood that patients or consumers will act on the potentially harmful medical information they receive from conversational assistants. METHODS Participants were given medical problems to pose to conversational assistants that had been previously demonstrated to result in potentially harmful recommendations. Each conversational assistant's response was randomly varied to include either a correct or incorrect paraphrase of the query or a disclaimer message-or not-telling the participants that they should not act on the advice without first talking to a physician. The participants were then asked what actions they would take based on their interaction, along with the likelihood of taking the action. The reported actions were recorded and analyzed, and the participants were interviewed at the end of each interaction. RESULTS A total of 32 participants completed the study, each interacting with 4 conversational assistants. The participants were on average aged 42.44 (SD 14.08) years, 53% (17/32) were women, and 66% (21/32) were college educated. Those participants who heard a correct paraphrase of their query were significantly more likely to state that they would follow the medical advice provided by the conversational assistant (χ21=3.1; P=.04). Those participants who heard a disclaimer message were significantly more likely to say that they would contact a physician or health professional before acting on the medical advice received (χ21=43.5; P=.001). CONCLUSIONS Designers of conversational systems should consider incorporating both disclaimers and feedback on query understanding in response to user queries for medical advice. Unconstrained natural language input should not be used in systems designed specifically to provide medical advice.
Collapse
Affiliation(s)
- Timothy W Bickmore
- Khoury College of Computer Sciences, Northeastern University, Boston, MA, United States
| | - Stefán Ólafsson
- Khoury College of Computer Sciences, Northeastern University, Boston, MA, United States
| | - Teresa K O'Leary
- Khoury College of Computer Sciences, Northeastern University, Boston, MA, United States
| |
Collapse
|
8
|
Finset A. How does the Covid-19 pandemic affect provider-patient relations? PATIENT EDUCATION AND COUNSELING 2021; 104:437. [PMID: 33714418 PMCID: PMC7945861 DOI: 10.1016/j.pec.2021.02.011] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
|