Physicians' Perceptions of the Use of a Chatbot for Information Seeking: Qualitative Study

Needs-Assessment for an Artificial Intelligence-Based Chatbot for Pharmacists in HIV Care: Results from a Knowledge-Attitudes-Practices Survey

BACKGROUND

Pharmacists need up-to-date knowledge and decision-making support in HIV care. We aim to develop MARVIN-Pharma, an adapted artificial intelligence-based chatbot initially for people with HIV, to assist pharmacists in considering evidence-based needs.

METHODS

From December 2022 to December 2023, an online needs-assessment survey evaluated Québec pharmacists' knowledge, attitudes, involvement, and barriers relative to HIV care, alongside perceptions relevant to the usability of MARVIN-Pharma. Recruitment involved convenience and snowball sampling, targeting National HIV and Hepatitis Mentoring Program affiliates.

RESULTS

Forty-one pharmacists (28 community, 13 hospital-based) across 15 Québec municipalities participated. Participants perceived their HIV knowledge as moderate (M = 3.74/6). They held largely favorable attitudes towards providing HIV care (M = 4.02/6). They reported a "little" involvement in the delivery of HIV care services (M = 2.08/5), most often ART adherence counseling, refilling, and monitoring. The most common barriers reported to HIV care delivery were a lack of time, staff resources, clinical tools, and HIV information/training, with pharmacists at least somewhat agreeing that they experienced each (M ≥ 4.00/6). On average, MARVIN-Pharma's acceptability and compatibility were in the 'undecided' range (M = 4.34, M = 4.13/7, respectively), while pharmacists agreed to their self-efficacy to use online health services (M = 5.6/7).

CONCLUSION

MARVIN-Pharma might help address pharmacists' knowledge gaps and barriers to HIV treatment and care, but pharmacist engagement in the chatbot's development seems vital for its future uptake and usability.

Implementation of Chatbot Technology in Health Care: Protocol for a Bibliometric Analysis

BACKGROUND

Chatbots have the potential to increase people's access to quality health care. However, the implementation of chatbot technology in the health care system is unclear due to the scarce analysis of publications on the adoption of chatbot in health and medical settings.

OBJECTIVE

This paper presents a protocol of a bibliometric analysis aimed at offering the public insights into the current state and emerging trends in research related to the use of chatbot technology for promoting health.

METHODS

In this bibliometric analysis, we will select published papers from the databases of CINAHL, IEEE Xplore, PubMed, Scopus, and Web of Science that pertain to chatbot technology and its applications in health care. Our search strategy includes keywords such as "chatbot," "virtual agent," "virtual assistant," "conversational agent," "conversational AI," "interactive agent," "health," and "healthcare." Five researchers who are AI engineers and clinicians will independently review the titles and abstracts of selected papers to determine their eligibility for a full-text review. The corresponding author (ZN) will serve as a mediator to address any discrepancies and disputes among the 5 reviewers. Our analysis will encompass various publication patterns of chatbot research, including the number of annual publications, their geographic or institutional distribution, and the number of annual grants supporting chatbot research, and further summarize the methodologies used in the development of health-related chatbots, along with their features and applications in health care settings. Software tool VOSViewer (version 1.6.19; Leiden University) will be used to construct and visualize bibliometric networks.

RESULTS

The preparation for the bibliometric analysis began on December 3, 2021, when the research team started the process of familiarizing themselves with the software tools that may be used in this analysis, VOSViewer and CiteSpace, during which they consulted 3 librarians at the Yale University regarding search terms and tentative results. Tentative searches on the aforementioned databases yielded a total of 2340 papers. The official search phase started on July 27, 2023. Our goal is to complete the screening of papers and the analysis by February 15, 2024.

CONCLUSIONS

Artificial intelligence chatbots, such as ChatGPT (OpenAI Inc), have sparked numerous discussions within the health care industry regarding their impact on human health. Chatbot technology holds substantial promise for advancing health care systems worldwide. However, developing a sophisticated chatbot capable of precise interaction with health care consumers, delivering personalized care, and providing accurate health-related information and knowledge remain considerable challenges. This bibliometric analysis seeks to fill the knowledge gap in the existing literature on health-related chatbots, entailing their applications, the software used in their development, and their preferred functionalities among users.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/54349.

Factors Influencing the Acceptability, Acceptance, and Adoption of Conversational Agents in Health Care: Integrative Review

BACKGROUND

Conversational agents (CAs), also known as chatbots, are digital dialog systems that enable people to have a text-based, speech-based, or nonverbal conversation with a computer or another machine based on natural language via an interface. The use of CAs offers new opportunities and various benefits for health care. However, they are not yet ubiquitous in daily practice. Nevertheless, research regarding the implementation of CAs in health care has grown tremendously in recent years.

OBJECTIVE

This review aims to present a synthesis of the factors that facilitate or hinder the implementation of CAs from the perspectives of patients and health care professionals. Specifically, it focuses on the early implementation outcomes of acceptability, acceptance, and adoption as cornerstones of later implementation success.

METHODS

We performed an integrative review. To identify relevant literature, a broad literature search was conducted in June 2021 with no date limits and using all fields in PubMed, Cochrane Library, Web of Science, LIVIVO, and PsycINFO. To keep the review current, another search was conducted in March 2022. To identify as many eligible primary sources as possible, we used a snowballing approach by searching reference lists and conducted a hand search. Factors influencing the acceptability, acceptance, and adoption of CAs in health care were coded through parallel deductive and inductive approaches, which were informed by current technology acceptance and adoption models. Finally, the factors were synthesized in a thematic map.

RESULTS

Overall, 76 studies were included in this review. We identified influencing factors related to 4 core Unified Theory of Acceptance and Use of Technology (UTAUT) and Unified Theory of Acceptance and Use of Technology 2 (UTAUT2) factors (performance expectancy, effort expectancy, facilitating conditions, and hedonic motivation), with most studies underlining the relevance of performance and effort expectancy. To meet the particularities of the health care context, we redefined the UTAUT2 factors social influence, habit, and price value. We identified 6 other influencing factors: perceived risk, trust, anthropomorphism, health issue, working alliance, and user characteristics. Overall, we identified 10 factors influencing acceptability, acceptance, and adoption among health care professionals (performance expectancy, effort expectancy, facilitating conditions, social influence, price value, perceived risk, trust, anthropomorphism, working alliance, and user characteristics) and 13 factors influencing acceptability, acceptance, and adoption among patients (additionally hedonic motivation, habit, and health issue).

CONCLUSIONS

This review shows manifold factors influencing the acceptability, acceptance, and adoption of CAs in health care. Knowledge of these factors is fundamental for implementation planning. Therefore, the findings of this review can serve as a basis for future studies to develop appropriate implementation strategies. Furthermore, this review provides an empirical test of current technology acceptance and adoption models and identifies areas where additional research is necessary.

TRIAL REGISTRATION

PROSPERO CRD42022343690; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=343690.

Trialling a Large Language Model (ChatGPT) in General Practice With the Applied Knowledge Test: Observational Study Demonstrating Opportunities and Limitations in Primary Care

BACKGROUND

Large language models exhibiting human-level performance in specialized tasks are emerging; examples include Generative Pretrained Transformer 3.5, which underlies the processing of ChatGPT. Rigorous trials are required to understand the capabilities of emerging technology, so that innovation can be directed to benefit patients and practitioners.

OBJECTIVE

Here, we evaluated the strengths and weaknesses of ChatGPT in primary care using the Membership of the Royal College of General Practitioners Applied Knowledge Test (AKT) as a medium.

METHODS

AKT questions were sourced from a web-based question bank and 2 AKT practice papers. In total, 674 unique AKT questions were inputted to ChatGPT, with the model's answers recorded and compared to correct answers provided by the Royal College of General Practitioners. Each question was inputted twice in separate ChatGPT sessions, with answers on repeated trials compared to gauge consistency. Subject difficulty was gauged by referring to examiners' reports from 2018 to 2022. Novel explanations from ChatGPT-defined as information provided that was not inputted within the question or multiple answer choices-were recorded. Performance was analyzed with respect to subject, difficulty, question source, and novel model outputs to explore ChatGPT's strengths and weaknesses.

RESULTS

Average overall performance of ChatGPT was 60.17%, which is below the mean passing mark in the last 2 years (70.42%). Accuracy differed between sources (P=.04 and .06). ChatGPT's performance varied with subject category (P=.02 and .02), but variation did not correlate with difficulty (Spearman ρ=-0.241 and -0.238; P=.19 and .20). The proclivity of ChatGPT to provide novel explanations did not affect accuracy (P>.99 and .23).

CONCLUSIONS

Large language models are approaching human expert-level performance, although further development is required to match the performance of qualified primary care physicians in the AKT. Validated high-performance models may serve as assistants or autonomous clinical tools to ameliorate the general practice workforce crisis.

Experiences and perceptions of final-year nursing students of using a chatbot in a simulated emergency situation: A qualitative study

AIM

The aim of this study is to explore the experiences and perceptions of final-year nursing students on the acceptability and feasibility of using a chatbot for clinical decision-making and patient safety.

BACKGROUND

The effective and inclusive use of new technologies such as conversational agents or chatbots could support nurses in increasing evidence-based care and decreasing low-quality services.

METHODS

A descriptive qualitative study was used through focus group interviews. The data analysis was conducted using a thematic analysis.

RESULTS

This study included 114 participants. After our data analysis, two main themes emerged: (i) experiences in the use of a chatbot service for clinical decision-making and and (ii) integrating conversational agents into the organizational safety culture.

CONCLUSIONS

The findings of our study provide preliminary support for the acceptability and feasibility of adopting SafeBot, a chatbot for clinical decision-making and patient safety. Our results revealed substantial recommendations for refining navigation, layout and content, as well as useful insights to support its acceptance in real nursing practice.

IMPLICATIONS FOR NURSING MANAGEMENT

Leaders and managers may well see artificial intelligence-based conversational agents like SafeBot as a potential solution in modern nursing practice for effective problem-solving resolution, innovative staffing and nursing care delivery models at the bedside and criteria for measuring and ensure quality and patient safety.