"Alexa, Am I pregnant?": A content analysis of a virtual assistant's responses to prenatal health questions during the COVID-19 pandemic

Virtual Assistants' Response to Queries About Nicotine Replacement Therapy: A Mixed-Method Analysis

This study focused on investigating the potential of Artificial Intelligent-powered Virtual Assistants (VAs) such as Amazon Alexa, Apple Siri, and Google Assistant as tools to help individuals seeking information about Nicotine Replacement Treatment (NRT) for smoking cessation. The researchers asked 40 NRT-related questions to each of the 3 VAs and evaluated the responses for voice recognition. The study used a cross-sectional mixed-method design with a total sample size of 360 responses. Inter-rater reliability and differences between VAs' responses were examined by SAS software, and qualitative assessments were conducted using NVivo software. Google Assistant achieved 100% voice recognition for NRT-related questions, followed by Apple Siri at 97.5%, and Amazon Alexa at 83.3%. Statistically significant differences were found between the responses of Amazon Alexa relative to both Google Assistant and Apple Siri. Researcher 1's ratings significantly differed from Researcher 2's (p = .001), but not from Researcher 3's (p = .11). Virtual Assistants occasionally struggled to understand the context or nuances of questions, lacked in-depth information in their responses, and provided generic or unrelated responses. Virtual Assistants have the potential to be incorporated into smoking cessation interventions and tobacco control initiatives, contingent upon improving their competencies.

Evaluation framework for conversational agents with artificial intelligence in health interventions: a systematic scoping review

OBJECTIVES

Conversational agents (CAs) with emerging artificial intelligence present new opportunities to assist in health interventions but are difficult to evaluate, deterring their applications in the real world. We aimed to synthesize existing evidence and knowledge and outline an evaluation framework for CA interventions.

MATERIALS AND METHODS

We conducted a systematic scoping review to investigate designs and outcome measures used in the studies that evaluated CAs for health interventions. We then nested the results into an overarching digital health framework proposed by the World Health Organization (WHO).

RESULTS

The review included 81 studies evaluating CAs in experimental (n = 59), observational (n = 15) trials, and other research designs (n = 7). Most studies (n = 72, 89%) were published in the past 5 years. The proposed CA-evaluation framework includes 4 evaluation stages: (1) feasibility/usability, (2) efficacy, (3) effectiveness, and (4) implementation, aligning with WHO's stepwise evaluation strategy. Across these stages, this article presents the essential evidence of different study designs (n = 8), sample sizes, and main evaluation categories (n = 7) with subcategories (n = 40). The main evaluation categories included (1) functionality, (2) safety and information quality, (3) user experience, (4) clinical and health outcomes, (5) costs and cost benefits, (6) usage, adherence, and uptake, and (7) user characteristics for implementation research. Furthermore, the framework highlighted the essential evaluation areas (potential primary outcomes) and gaps across the evaluation stages.

DISCUSSION AND CONCLUSION

This review presents a new framework with practical design details to support the evaluation of CA interventions in healthcare research.

PROTOCOL REGISTRATION

The Open Science Framework (https://osf.io/9hq2v) on March 22, 2021.

Exploring the ChatGPT platform with scenario-specific prompts for vaping cessation

OBJECTIVE

To evaluate and start a discussion on the potential usefulness of applying Artificial Intelligence (AI)-driven natural language processing technology such as the ChatGPT in tobacco control efforts, specifically vaping cessation.

METHOD

Ten real-world questions about vaping cessation were selected from a Reddit forum and used as ChatGPT prompts or queries. Content analysis was performed on the ChatGPT responses to identify the thematic aspects of vaping cessation support represented in the responses. Next, the responses were empirically evaluated by five experts in tobacco control on accuracy, quality, clarity, and empathy.

RESULT

The following themes related to vaping cessation support were identified: understanding nicotine withdrawal symptoms, self-regulation, peer support, motivational support, and Nicotine Replacement Therapy (NRT). The experts judged the ChatGPT responses to be 'satisfactory' to 'excellent' in areas of accuracy, quality, clarity, and empathy.

CONCLUSION

If managed by a group of experts, including clinicians, and behavioral and computer scientists, a platform such as the ChatGPT may be leveraged to design tailored interventions for tobacco use cessation, including vaping cessation.

"Alexa, did the pandemic make you smarter?" A follow up content analysis of a virtual assistant's responses to a prenatal query

To compare responses to 40 common prenatal questions from Amazon's virtual assistant, Alexa, one year apart during the COVID pandemic. Participants: Two researchers replicated a prenatal query using unique Alexa devices. A conceptual content analysis was conducted where the researchers independently queried Alexa the identical questions from their 2020 study during the same one-week timeframe, between May 20, 2021 and May 27, 2021. Alexa's responses were compared to the 2020 study and the American College of Obstetricians and Gynecologists data and verified by one of the researchers, a Certified Nurse Midwife. Alexa provided accurate responses to 26 (65%) of the questions, an increase by 55 percentage points from 2020. Alexa was able to recite the symptoms of COVID-19 illness but was unable to provide a response to the two other COVID-specific questions. Compared to the 2020 query, Alexa provided more reputable sources for the responses including the CDC, WHO, NIH, and Mayo Clinic. Alexa's ability to provide more accurate, evidence-based responses was remarkably improved in 2021. Mobile health tools, like Amazon Alexa, are highly utilized by the public, particularly with limited healthcare access during the COVID-19 pandemic. Technology-based platforms should provide credible, evidence-based content.

High Satisfaction With a Virtual Assistant for Plastic Surgery Frequently Asked Questions

BACKGROUND

Most of a surgeon's office time is dedicated to patient education, preventing an appropriate patient-physician relationship. Telephone-accessed artificial intelligent virtual assistants (AIVAs) that simulate a human conversation and answer preoperative frequently asked questions (FAQs) can be effective solutions to this matter. An AIVA capable of answering preoperative plastic surgery-related FAQs has previously been described by the authors.

OBJECTIVES

The aim of this paper was to determine patients' perception and satisfaction with an AIVA.

METHODS

Twenty-six adult patients from a plastic surgery service answered a 3-part survey consisting of: (1) an evaluation of the answers' correctness, (2) their agreement with the feasibility, usefulness, and future uses of the AIVA, and (3) a section on comments. The first part made it possible to measure the system's accuracy, and the second to evaluate perception and satisfaction. The data were analyzed with Microsoft Excel 2010 (Microsoft Corporation, Redmond, WA).

RESULTS

The AIVA correctly answered the patients' questions 98.5% of the time, and the topic with the lowest accuracy was "nausea." Additionally, 88% of patients agreed with the statements of the second part of the survey. Thus, the patients' perception was positive and overall satisfaction with the AIVA was high. Patients agreed the least with using the AIVA to select their surgical procedure. The comments provided improvement areas for subsequent stages of the project.

CONCLUSIONS

The results show that patients were satisfied and expressed a positive experience with using the AIVA to answer plastic surgery FAQs before surgery. The system is also highly accurate.

Voice Assistants' Responses to Questions About the COVID-19 Vaccine: National Cross-sectional Study

BACKGROUND

Artificial intelligence-powered voice assistants (VAs), such as Apple Siri, Google Assistant, and Amazon Alexa, interact with users in natural language and are capable of responding to simple commands, searching the internet, and answering questions. Despite being an increasingly popular way for the public to access health information, VAs could be a source of ambiguous or potentially biased information.

OBJECTIVE

In response to the ongoing prevalence of vaccine misinformation and disinformation, this study aims to evaluate how smartphone VAs respond to information- and recommendation-seeking inquiries regarding the COVID-19 vaccine.

METHODS

A national cross-sectional survey of English-speaking adults who owned a smartphone with a VA installed was conducted online from April 22 to 28, 2021. The primary outcomes were the VAs' responses to 2 questions: "Should I get the COVID vaccine?" and "Is the COVID vaccine safe?" Directed content analysis was used to assign a negative, neutral, or positive connotation to each response and website title provided by the VAs. Statistical significance was assessed using the t test (parametric) or Mann-Whitney U (nonparametric) test for continuous variables and the chi-square or Fisher exact test for categorical variables.

RESULTS

Of the 466 survey respondents included in the final analysis, 404 (86.7%) used Apple Siri, 53 (11.4%) used Google Assistant, and 9 (1.9%) used Amazon Alexa. In response to the question "Is the COVID vaccine safe?" 419 (89.9%) users received a direct response, of which 408 (97.3%) had a positive connotation encouraging users to get vaccinated. Of the websites presented, only 5.3% (11/207) had a positive connotation and 94.7% (196/207) had a neutral connotation. In response to the question "Should I get the COVID vaccine?" 93.1% (434/466) of users received a list of websites, of which 91.5% (1155/1262) had a neutral connotation. For both COVID-19 vaccine-related questions, there was no association between the connotation of a response and the age, gender, zip code, race or ethnicity, and education level of the respondent.

CONCLUSIONS

Our study found that VAs were much more likely to respond directly with positive connotations to the question "Is the COVID vaccine safe?" but not respond directly and provide a list of websites with neutral connotations to the question "Should I get the COVID vaccine?" To our knowledge, this is the first study to evaluate how VAs respond to both information- and recommendation-seeking inquiries regarding the COVID-19 vaccine. These findings add to our growing understanding of both the opportunities and pitfalls of VAs in supporting public health information dissemination.

Mitigating Patient and Consumer Safety Risks When Using Conversational Assistants for Medical Information: Exploratory Mixed Methods Experiment

BACKGROUND

Prior studies have demonstrated the safety risks when patients and consumers use conversational assistants such as Apple's Siri and Amazon's Alexa for obtaining medical information.

OBJECTIVE

The aim of this study is to evaluate two approaches to reducing the likelihood that patients or consumers will act on the potentially harmful medical information they receive from conversational assistants.

METHODS

Participants were given medical problems to pose to conversational assistants that had been previously demonstrated to result in potentially harmful recommendations. Each conversational assistant's response was randomly varied to include either a correct or incorrect paraphrase of the query or a disclaimer message-or not-telling the participants that they should not act on the advice without first talking to a physician. The participants were then asked what actions they would take based on their interaction, along with the likelihood of taking the action. The reported actions were recorded and analyzed, and the participants were interviewed at the end of each interaction.

RESULTS

A total of 32 participants completed the study, each interacting with 4 conversational assistants. The participants were on average aged 42.44 (SD 14.08) years, 53% (17/32) were women, and 66% (21/32) were college educated. Those participants who heard a correct paraphrase of their query were significantly more likely to state that they would follow the medical advice provided by the conversational assistant (χ²₁=3.1; P=.04). Those participants who heard a disclaimer message were significantly more likely to say that they would contact a physician or health professional before acting on the medical advice received (χ²₁=43.5; P=.001).

CONCLUSIONS

Designers of conversational systems should consider incorporating both disclaimers and feedback on query understanding in response to user queries for medical advice. Unconstrained natural language input should not be used in systems designed specifically to provide medical advice.