Patient Perspectives on the Usefulness of an Artificial Intelligence-Assisted Symptom Checker: Cross-Sectional Survey Study

Consumer opinion on the use of machine learning in healthcare settings: A qualitative systematic review

Introduction

Given the increasing number of artificial intelligence and machine learning (AI/ML) tools in healthcare, we aimed to gain an understanding of consumer perspectives on the use of AI/ML tools for healthcare diagnostics.

Methods

We conducted a qualitative systematic review, following established standardized methods, of the existing literature indexed in the following databases up to 4 April 2022: OVID MEDLINE, OVID EMBASE, Scopus and Web of Science.

Results

Fourteen studies were identified as appropriate for inclusion in the meta-synthesis and systematic review. Most studies (n = 12) were conducted in high-income countries, with data extracted from both mixed methods (42.9%) and qualitative (57.1%) studies. The meta-synthesis identified four overarching themes across the included studies: (1) Trust, fear, and uncertainty; (2) Data privacy and ML governance; (3) Impact on healthcare delivery and access; and (4) Consumers want to be engaged.

Conclusion

The current evidence demonstrates consumers' understandings of AI/ML for medical diagnosis are complex. Consumers express a complex combination of both hesitancy and support towards AI/ML in healthcare diagnosis. Importantly, their views of the use of AI/ML in medical diagnosis are influenced by the perceived trustworthiness of their healthcare providers who use these AI/ML tools. Consumers recognize the potential for AI/ML tools to improve diagnostic accuracy, efficiency and access, and express a strong interest to be engaged in the development and implementation process of AI/ML into routine healthcare.

The RepVig framework for designing use-case specific representative vignettes and evaluating triage accuracy of laypeople and symptom assessment applications

Most studies evaluating symptom-assessment applications (SAAs) rely on a common set of case vignettes that are authored by clinicians and devoid of context, which may be representative of clinical settings but not of situations where patients use SAAs. Assuming the use case of self-triage, we used representative design principles to sample case vignettes from online platforms where patients describe their symptoms to obtain professional advice and compared triage performance of laypeople, SAAs (e.g., WebMD or NHS 111), and Large Language Models (LLMs, e.g., GPT-4 or Claude) on representative versus standard vignettes. We found performance differences in all three groups depending on vignette type: When using representative vignettes, accuracy was higher (OR = 1.52 to 2.00, p < .001 to .03 in binary decisions, i.e., correct or incorrect), safety was higher (OR = 1.81 to 3.41, p < .001 to .002 in binary decisions, i.e., safe or unsafe), and the inclination to overtriage was also higher (OR = 1.80 to 2.66, p < .001 to p = .035 in binary decisions, overtriage or undertriage error). Additionally, we found changed rankings of best-performing SAAs and LLMs. Based on these results, we argue that our representative vignette sampling approach (that we call the RepVig Framework) should replace the practice of using a fixed vignette set as standard for SAA evaluation studies.

Assessing the impact of information on patient attitudes toward artificial intelligence-based clinical decision support (AI/CDS): a pilot web-based SMART vignette study

BACKGROUND

It is increasingly recognised that the success of artificial intelligence-based clinical decision support (AI/CDS) tools will depend on physician and patient trust, but factors impacting patients' views on clinical care reliant on AI have been less explored.

OBJECTIVE

This pilot study explores whether, and in what contexts, detail of explanation provided about AI/CDS tools impacts patients' attitudes toward the tools and their clinical care.

METHODS

We designed a Sequential Multiple Assignment Randomized Trial vignette web-based survey. Participants recruited through Amazon Mechanical Turk were presented with hypothetical vignettes describing health concerns and were sequentially randomised along three factors: (1) the level of detail of explanation regarding an AI/CDS tool; (2) the AI/CDS result; and (3) the physician's level of agreement with the AI/CDS result. We compared mean ratings of comfort and confidence by the level of detail of explanation using t-tests. Regression models were fit to confirm conditional effects of detail of explanation.

RESULTS

The detail of explanation provided regarding the AI/CDS tools was positively related to respondents' comfort and confidence in the usage of the tools and their perception of the physician's final decision. The effects of detail of explanation on their perception of the physician's final decision were different given the AI/CDS result and the physician's agreement or disagreement with the result.

CONCLUSIONS

More information provided by physicians regarding the use of AI/CDS tools may improve patient attitudes toward healthcare involving AI/CDS tools in general and in certain contexts of the AI/CDS result and physician agreement.

Exploring Self-Reported Symptoms for Developing and Evaluating Digital Symptom Checkers for Polycystic Ovarian Syndrome, Endometriosis, and Uterine Fibroids: Exploratory Survey Study

BACKGROUND

Reproductive health conditions such as polycystic ovary syndrome (PCOS), endometriosis, and uterine fibroids pose a significant burden to people who menstruate, health care systems, and economies. Despite clinical guidelines for each condition, prolonged delays in diagnosis are commonplace, resulting in an increase to health care costs and risk of health complications. Symptom checker apps have the potential to significantly reduce time to diagnosis by providing users with health information and tools to better understand their symptoms.

OBJECTIVE

This study aims to study the prevalence and predictive importance of self-reported symptoms of PCOS, endometriosis, and uterine fibroids, and to explore the efficacy of 3 symptom checkers (developed by Flo Health UK Limited) that use self-reported symptoms when screening for each condition.

METHODS

Flo's symptom checkers were transcribed into separate web-based surveys for PCOS, endometriosis, and uterine fibroids, asking respondents their diagnostic history for each condition. Participants were aged 18 years or older, female, and living in the United States. Participants either had a confirmed diagnosis (condition-positive) and reported symptoms retrospectively as experienced at the time of diagnosis, or they had not been examined for the condition (condition-negative) and reported their current symptoms as experienced at the time of surveying. Symptom prevalence was calculated for each condition based on the surveys. Least absolute shrinkage and selection operator regression was used to identify key symptoms for predicting each condition. Participants' symptoms were processed by Flo's 3 single-condition symptom checkers, and accuracy was assessed by comparing the symptom checker output with the participant's condition designation.

RESULTS

A total of 1317 participants were included with 418, 476, and 423 in the PCOS, endometriosis, and uterine fibroids groups, respectively. The most prevalent symptoms for PCOS were fatigue (92%), feeling anxious (87%), BMI over 25 (84%); for endometriosis: very regular lower abdominal pain (89%), fatigue (85%), and referred lower back pain (80%); for uterine fibroids: fatigue (76%), bloating (69%), and changing sanitary protection often (68%). Symptoms of anovulation and amenorrhea (long periods, irregular cycles, and absent periods), and hyperandrogenism (excess hair on chin and abdomen, scalp hair loss, and BMI over 25) were identified as the most predictive symptoms for PCOS, while symptoms related to abdominal pain and the effect pain has on life, bleeding, and fertility complications were among the most predictive symptoms for both endometriosis and uterine fibroids. Symptom checker accuracy was 78%, 73%, and 75% for PCOS, endometriosis, and uterine fibroids, respectively.

CONCLUSIONS

This exploratory study characterizes self-reported symptomatology and identifies the key predictive symptoms for 3 reproductive conditions. The Flo symptom checkers were evaluated using real, self-reported symptoms and demonstrated high levels of accuracy.

Can Patients With Urogenital Cancer Rely on Artificial Intelligence Chatbots for Treatment Decisions?

OBJECTIVES

In the era of artificial intelligence, almost half of the patients use the internet to get information about their diseases. Our study aims to demonstrate the reliability of the information provided by artificial intelligence chatbots (AICs) about urogenital cancer treatments.

METHODS

The most frequently searched keyword about prostate, bladder, kidney, and testicular cancer treatment via Google Trends was asked to 3 different AICs (ChatGPT, Gemini, Copilot). The answers were evaluated by 5 different examiners in terms of readability, understandability, actionability, reliability, and transparency.

RESULTS

The DISCERN score evaluation indicates that ChatGPT and Gemini provided moderate quality information, while Copilot's quality was low. (Total DISCERN scores; 41, 42, 35, respectively). PEMAT-P Understandability scores were low (40%) and PEMAT-P Actionability scores were moderate only for Gemini (60%) and low for the others (40%). Their readability according to the Coleman-Liau index was above the college level (16.9, 17.2, 16, respectively).

CONCLUSIONS

In the era of artificial intelligence, patients will inevitably use AICs due to their easy and fast accessibility. However, patients need to recognize that AICs do not provide stage-specific treatment options, but only moderate-quality, low-reliability information about the disease, as well as information that is very difficult to read.

Patients Facing Large Language Models in Oncology: A Narrative Review

The integration of large language models (LLMs) into oncology is transforming patients' journeys through education, diagnosis, treatment monitoring, and follow-up. This review examines the current landscape, potential benefits, and associated ethical and regulatory considerations of the application of LLMs for patients in the oncologic domain.

Triage Accuracy and the Safety of User-Initiated Symptom Assessment With an Electronic Symptom Checker in a Real-Life Setting: Instrument Validation Study

BACKGROUND

Previous studies have evaluated the accuracy of the diagnostics of electronic symptom checkers (ESCs) and triage using clinical case vignettes. National Omaolo digital services (Omaolo) in Finland consist of an ESC for various symptoms. Omaolo is a medical device with a Conformité Européenne marking (risk class: IIa), based on Duodecim Clinical Decision Support, EBMEDS.

OBJECTIVE

This study investigates how well triage performed by the ESC nurse triage within the chief symptom list available in Omaolo (anal region symptoms, cough, diarrhea, discharge from the eye or watery or reddish eye, headache, heartburn, knee symptom or injury, lower back pain or injury, oral health, painful or blocked ear, respiratory tract infection, sexually transmitted disease, shoulder pain or stiffness or injury, sore throat or throat symptom, and urinary tract infection). In addition, the accuracy, specificity, sensitivity, and safety of the Omaolo ESC were assessed.

METHODS

This is a clinical validation study in a real-life setting performed at multiple primary health care (PHC) centers across Finland. The included units were of the walk-in model of primary care, where no previous phone call or contact was required. Upon arriving at the PHC center, users (patients) answered the ESC questions and received a triage recommendation; a nurse then assessed their triage. Findings on 877 patients were analyzed by matching the ESC recommendations with triage by the triage nurse.

RESULTS

Safe assessments by the ESC accounted for 97.6% (856/877; 95% CI 95.6%-98.0%) of all assessments made. The mean of the exact match for all symptom assessments was 53.7% (471/877; 95% CI 49.2%-55.9%). The mean value of the exact match or overly conservative but suitable for all (ESC's assessment was 1 triage level higher than the nurse's triage) symptom assessments was 66.6% (584/877; 95% CI 63.4%-69.7%). When the nurse concluded that urgent treatment was needed, the ESC's exactly matched accuracy was 70.9% (244/344; 95% CI 65.8%-75.7%). Sensitivity for the Omaolo ESC was 62.6% and specificity of 69.2%. A total of 21 critical assessments were identified for further analysis: there was no indication of compromised patient safety.

CONCLUSIONS

The primary objectives of this study were to evaluate the safety and to explore the accuracy, specificity, and sensitivity of the Omaolo ESC. The results indicate that the ESC is safe in a real-life setting when appraised with assessments conducted by triage nurses. Furthermore, the Omaolo ESC exhibits the potential to guide patients to appropriate triage destinations effectively, helping them to receive timely and suitable care.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR2-10.2196/41423.

Does an App a Day Keep the Doctor Away? AI Symptom Checker Applications, Entrenched Bias, and Professional Responsibility

The growing prominence of artificial intelligence (AI) in mobile health (mHealth) has given rise to a distinct subset of apps that provide users with diagnostic information using their inputted health status and symptom information-AI-powered symptom checker apps (AISympCheck). While these apps may potentially increase access to health care, they raise consequential ethical and legal questions. This paper will highlight notable concerns with AI usage in the health care system, further entrenchment of preexisting biases in the health care system and issues with professional accountability. To provide an in-depth analysis of the issues of bias and complications of professional obligations and liability, we focus on 2 mHealth apps as examples-Babylon and Ada. We selected these 2 apps as they were both widely distributed during the COVID-19 pandemic and make prominent claims about their use of AI for the purpose of assessing user symptoms. First, bias entrenchment often originates from the data used to train AI systems, causing the AI to replicate these inequalities through a "garbage in, garbage out" phenomenon. Users of these apps are also unlikely to be demographically representative of the larger population, leading to distorted results. Second, professional accountability poses a substantial challenge given the vast diversity and lack of regulation surrounding the reliability of AISympCheck apps. It is unclear whether these apps should be subject to safety reviews, who is responsible for app-mediated misdiagnosis, and whether these apps ought to be recommended by physicians. With the rapidly increasing number of apps, there remains little guidance available for health professionals. Professional bodies and advocacy organizations have a particularly important role to play in addressing these ethical and legal gaps. Implementing technical safeguards within these apps could mitigate bias, AIs could be trained with primarily neutral data, and apps could be subject to a system of regulation to allow users to make informed decisions. In our view, it is critical that these legal concerns are considered throughout the design and implementation of these potentially disruptive technologies. Entrenched bias and professional responsibility, while operating in different ways, are ultimately exacerbated by the unregulated nature of mHealth.

Evaluating the Diagnostic Performance of Symptom Checkers: Clinical Vignette Study

BACKGROUND

Medical self-diagnostic tools (or symptom checkers) are becoming an integral part of digital health and our daily lives, whereby patients are increasingly using them to identify the underlying causes of their symptoms. As such, it is essential to rigorously investigate and comprehensively report the diagnostic performance of symptom checkers using standard clinical and scientific approaches.

OBJECTIVE

This study aims to evaluate and report the accuracies of a few known and new symptom checkers using a standard and transparent methodology, which allows the scientific community to cross-validate and reproduce the reported results, a step much needed in health informatics.

METHODS

We propose a 4-stage experimentation methodology that capitalizes on the standard clinical vignette approach to evaluate 6 symptom checkers. To this end, we developed and peer-reviewed 400 vignettes, each approved by at least 5 out of 7 independent and experienced primary care physicians. To establish a frame of reference and interpret the results of symptom checkers accordingly, we further compared the best-performing symptom checker against 3 primary care physicians with an average experience of 16.6 (SD 9.42) years. To measure accuracy, we used 7 standard metrics, including M1 as a measure of a symptom checker's or a physician's ability to return a vignette's main diagnosis at the top of their differential list, F1-score as a trade-off measure between recall and precision, and Normalized Discounted Cumulative Gain (NDCG) as a measure of a differential list's ranking quality, among others.

RESULTS

The diagnostic accuracies of the 6 tested symptom checkers vary significantly. For instance, the differences in the M1, F1-score, and NDCG results between the best-performing and worst-performing symptom checkers or ranges were 65.3%, 39.2%, and 74.2%, respectively. The same was observed among the participating human physicians, whereby the M1, F1-score, and NDCG ranges were 22.8%, 15.3%, and 21.3%, respectively. When compared against each other, physicians outperformed the best-performing symptom checker by an average of 1.2% using F1-score, whereas the best-performing symptom checker outperformed physicians by averages of 10.2% and 25.1% using M1 and NDCG, respectively.

CONCLUSIONS

The performance variation between symptom checkers is substantial, suggesting that symptom checkers cannot be treated as a single entity. On a different note, the best-performing symptom checker was an artificial intelligence (AI)-based one, shedding light on the promise of AI in improving the diagnostic capabilities of symptom checkers, especially as AI keeps advancing exponentially.

Recommendation endpoints and safety of an online self-triage for depression symptoms

INTRODUCTION

Online symptom checkers are a way to address patient concerns and potentially offload a burdened healthcare system. However, safety outcomes of self-triage are unknown, so we reviewed triage recommendations and outcomes of our institution's depression symptom checker.

METHODS

We examined endpoint recommendations and follow-up encounters seven days afterward during 2 December 2021 to 13 December 2022. Patients with an emergency department visit or hospitalization within seven days of self-triaging had a manual review of the electronic health record to determine if the visit was related to depression, suicidal ideation, or suicide attempt. Charts were reviewed for deaths within seven days of self-triage.

RESULTS

There were 287 unique encounters from 263 unique patients. In 86.1% (247/287), the endpoint was an instruction to call nurse triage; in 3.1% of encounters (9/287), instruction was to seek emergency care. Only 20.2% (58/287) followed the recommendations given. Of the 229 patients that did not follow the endpoint recommendations, 121 (52.8%) had some type of follow-up within seven days. Nearly 11% (31/287) were triaged to endpoints not requiring urgent contact and 9.1% (26/287) to an endpoint that would not need any healthcare team input. No patients died in the study period.

CONCLUSIONS

Most patients did not follow the recommendations for follow-up care although ultimately most patients did receive care within seven days. Self-triage appears to appropriately sort patients with depressed mood to emergency care. On-line self-triaging tools for depression have the potential to safely offload some work from clinic personnel.

AI and Aphasia in the Digital Age: A Critical Review

Aphasiology has a long and rich tradition of contributing to understanding how culture, language, and social environment contribute to brain development and function. Recent breakthroughs in AI can transform the role of aphasiology in the digital age by leveraging speech data in all languages to model how damage to specific brain regions impacts linguistic universals such as grammar. These tools, including generative AI (ChatGPT) and natural language processing (NLP) models, could also inform practitioners working with clinical populations in the assessment and treatment of aphasia using AI-based interventions such as personalized therapy and adaptive platforms. Although these possibilities have generated enthusiasm in aphasiology, a rigorous interrogation of their limitations is necessary before AI is integrated into practice. We explain the history and first principles of reciprocity between AI and aphasiology, highlighting how lesioning neural networks opened the black box of cognitive neurolinguistic processing. We then argue that when more data from aphasia across languages become digitized and available online, deep learning will reveal hitherto unreported patterns of language processing of theoretical interest for aphasiologists. We also anticipate some problems using AI, including language biases, cultural, ethical, and scientific limitations, a misrepresentation of marginalized languages, and a lack of rigorous validation of tools. However, as these challenges are met with better governance, AI could have an equitable impact.

Walking forward or on hold: Could the ChatGPT be applied for seeking health information in neurosurgical settings?

Self-management is important for patients suffering from cerebrovascular events after neurosurgical procedures. An increasing number of artificial intelligence (AI)-assisted tools have been used in postoperative health management. ChatGPT is a new trend dialog-based chatbot that could be used as a supplemental tool for seeking health information. Responses from ChatGPT version 3.5 and 4.0 toward 13 questions raised by experienced neurosurgeons were evaluated in this exploratory study for their consistency and appropriateness blindly by the other three neurosurgeons. The readability of response text was investigated quantitively by word count and the Gunning Fog and Flesch-Kincaid indices. Results showed that the chatbot could provide relatively stable output between the two versions on consistency and appropriateness (χ² = 0.348). As for readability, there was a higher demand for readers to comprehend the output text in the 4.0 version (more counts of words; lower Flesch-Kincaid reading ease score; and higher Flesch-Kincaid grade level). In general, the capacity of ChatGPT to deliver effective health information is still under debate.

Testing Two Online Symptom Checkers With Vulnerable Groups: Usability Study to Improve Cognitive Accessibility of eHealth Services

BACKGROUND

The popularity of eHealth services has surged significantly, underscoring the importance of ensuring their usability and accessibility for users with diverse needs, characteristics, and capabilities. These services can pose cognitive demands, especially for individuals who are unwell, fatigued, or experiencing distress. Additionally, numerous potentially vulnerable groups, including older adults, are susceptible to digital exclusion and may encounter cognitive limitations related to perception, attention, memory, and language comprehension. Regrettably, many studies overlook the preferences and needs of user groups likely to encounter challenges associated with these cognitive aspects.

OBJECTIVE

This study primarily aims to gain a deeper understanding of cognitive accessibility in the practical context of eHealth services. Additionally, we aimed to identify the specific challenges that vulnerable groups encounter when using eHealth services and determine key considerations for testing these services with such groups.

METHODS

As a case study of eHealth services, we conducted qualitative usability testing on 2 online symptom checkers used in Finnish public primary care. A total of 13 participants from 3 distinct groups participated in the study: older adults, individuals with mild intellectual disabilities, and nonnative Finnish speakers. The primary research methods used were the thinking-aloud method, questionnaires, and semistructured interviews.

RESULTS

We found that potentially vulnerable groups encountered numerous issues with the tested services, with similar problems observed across all 3 groups. Specifically, clarity and the use of terminology posed significant challenges. The services overwhelmed users with excessive information and choices, while the terminology consisted of numerous complex medical terms that were difficult to understand. When conducting tests with vulnerable groups, it is crucial to carefully plan the sessions to avoid being overly lengthy, as these users often require more time to complete tasks. Additionally, testing with vulnerable groups proved to be quite efficient, with results likely to benefit a wider audience as well.

CONCLUSIONS

Based on the findings of this study, it is evident that older adults, individuals with mild intellectual disability, and nonnative speakers may encounter cognitive challenges when using eHealth services, which can impede or slow down their use and make the services more difficult to navigate. In the worst-case scenario, these challenges may lead to errors in using the services. We recommend expanding the scope of testing to include a broader range of eHealth services with vulnerable groups, incorporating users with diverse characteristics and capabilities who are likely to encounter difficulties in cognitive accessibility.

Implications of Online Self-Diagnosis in Psychiatry

Online self-diagnosis of psychiatric disorders by the general public is increasing. The reasons for the increase include the expansion of Internet technologies and the use of social media, the rapid growth of direct-to-consumer e-commerce in healthcare, and the increased emphasis on patient involvement in decision making. The publicity given to artificial intelligence (AI) has also contributed to the increased use of online screening tools by the general public. This paper aims to review factors contributing to the expansion of online self-diagnosis by the general public, and discuss both the risks and benefits of online self-diagnosis of psychiatric disorders. A narrative review was performed with examples obtained from the scientific literature and commercial articles written for the general public. Online self-diagnosis of psychiatric disorders is growing rapidly. Some people with a positive result on a screening tool will seek professional help. However, there are many potential risks for patients who self-diagnose, including an incorrect or dangerous diagnosis, increased patient anxiety about the diagnosis, obtaining unfiltered advice on social media, using the self-diagnosis to self-treat, including online purchase of medications without a prescription, and technical issues including the loss of privacy. Physicians need to be aware of the increase in self-diagnosis by the general public and the potential risks, both medical and technical. Psychiatrists must recognize that the general public is often unaware of the challenging medical and technical issues involved in the diagnosis of a mental disorder, and be ready to treat patients who have already obtained an online self-diagnosis.

"That's just Future Medicine" - a qualitative study on users' experiences of symptom checker apps

BACKGROUND

Symptom checker apps (SCAs) are mobile or online applications for lay people that usually have two main functions: symptom analysis and recommendations. SCAs ask users questions about their symptoms via a chatbot, give a list with possible causes, and provide a recommendation, such as seeing a physician. However, it is unclear whether the actual performance of a SCA corresponds to the users' experiences. This qualitative study investigates the subjective perspectives of SCA users to close the empirical gap identified in the literature and answers the following main research question: How do individuals (healthy users and patients) experience the usage of SCA, including their attitudes, expectations, motivations, and concerns regarding their SCA use?

METHODS

A qualitative interview study was chosen to clarify the relatively unknown experience of SCA use. Semi-structured qualitative interviews with SCA users were carried out by two researchers in tandem via video call. Qualitative content analysis was selected as methodology for the data analysis.

RESULTS

Fifteen interviews with SCA users were conducted and seven main categories identified: (1) Attitudes towards findings and recommendations, (2) Communication, (3) Contact with physicians, (4) Expectations (prior to use), (5) Motivations, (6) Risks, and (7) SCA-use for others.

CONCLUSIONS

The aspects identified in the analysis emphasise the specific perspective of SCA users and, at the same time, the immense scope of different experiences. Moreover, the study reveals ethical issues, such as relational aspects, that are often overlooked in debates on mHealth. Both empirical and ethical research is more needed, as the awareness of the subjective experience of those affected is an essential component in the responsible development and implementation of health apps such as SCA.

TRIAL REGISTRATION

German Clinical Trials Register (DRKS): DRKS00022465. 07/08/2020.

Only the anxious ones? Identifying characteristics of symptom checker app users: a cross-sectional survey

BACKGROUND

Symptom checker applications (SCAs) may help laypeople classify their symptoms and receive recommendations on medically appropriate actions. Further research is necessary to estimate the influence of user characteristics, attitudes and (e)health-related competencies.

OBJECTIVE

The objective of this study is to identify meaningful predictors for SCA use considering user characteristics.

METHODS

An explorative cross-sectional survey was conducted to investigate German citizens' demographics, eHealth literacy, hypochondria, self-efficacy, and affinity for technology using German language-validated questionnaires. A total of 869 participants were eligible for inclusion in the study. As n = 67 SCA users were assessed and matched 1:1 with non-users, a sample of n = 134 participants were assessed in the main analysis. A four-step analysis was conducted involving explorative predictor selection, model comparisons, and parameter estimates for selected predictors, including sensitivity and post hoc analyses.

RESULTS

Hypochondria and self-efficacy were identified as meaningful predictors of SCA use. Hypochondria showed a consistent and significant effect across all analyses OR: 1.24-1.26 (95% CI: 1.1-1.4). Self-efficacy OR: 0.64-0.93 (95% CI: 0.3-1.4) showed inconsistent and nonsignificant results, leaving its role in SCA use unclear. Over half of the SCA users in our sample met the classification for hypochondria (cut-off on the WI of 5).

CONCLUSIONS

Hypochondria has emerged as a significant predictor of SCA use with a consistently stable effect, yet according to the literature, individuals with this trait may be less likely to benefit from SCA despite their greater likelihood of using it. These users could be further unsettled by risk-averse triage and unlikely but serious diagnosis suggestions.

TRIAL REGISTRATION

The study was registered in the German Clinical Trials Register (DRKS) DRKS00022465, DERR1- https://doi.org/10.2196/34026 .

Empowering the digital health workforce in Latin America in the context of the COVID-19 pandemic: the Peruvian case

The COVID-19 pandemic has exposed significant gaps in healthcare access, quality, and the urgent need for enhancing the capacity of digital health human resources, particularly in Latin America. During the pandemic, online courses and telehealth initiatives supported by governmental agencies, the Pan American Health Organization, and other public and private resources, have played a crucial role in meeting training demands. This article discusses the role of capacity building programs in digital health within the context of Latin America, with a specific focus on the Peruvian case. We highlight the development of digital health competencies and related policies, while also describing selected experiences related to capacity building in this field. Additionally, we discuss the pivotal role of collaborative partnerships among institutions and countries, emphasizing the importance of culturally relevant training programs in digital health. These initiatives have the potential to accelerate training and research opportunities in Latin America, drawing on the involvement of government agencies, non-governmental organizations, industry, universities, professional societies, and communities.

Beliefs, experiences and concerns of using artificial intelligence in healthcare: A qualitative synthesis

Objective

Artificial intelligence (AI) is a developing field in the context of healthcare. As this technology continues to be implemented in patient care, there is a growing need to understand the thoughts and experiences of stakeholders in this area to ensure that future AI development and implementation is successful. The aim of this study was to conduct a literature search of qualitative studies exploring the opinions of stakeholders such as clinicians, patients, and technology experts in order to establish the most common themes and ideas that have been presented in this research.

Methods

A literature search was conducted of existing qualitative research on stakeholder beliefs about the use of AI use in healthcare. Twenty-one papers were selected and analysed resulting in the development of four key themes relating to patient care, patient-doctor relationships, lack of education and resources, and the need for regulations.

Results

Overall, patients and healthcare workers are open to the use of AI in care and appear positive about potential benefits. However, concerns were raised relating to the lack of empathy in interactions of AI tools, and potential risks that may arise from the data collection needed for AI use and development. Stakeholders in the healthcare, technology, and business sectors all stressed that there was a lack of appropriate education, funding, and guidelines surrounding AI, and these concerns needed to be addressed to ensure future implementation is safe and suitable for patient care.

Conclusion

Ultimately, the results found in this study highlighted that there was a need for communication between stakeholder in order for these concerns to be addressed, mitigate potential risks, and maximise benefits for patients and clinicians alike. The results also identified a need for further qualitative research in this area to further understand stakeholder experiences as AI use continues to develop.

'Better see a doctor?' Status quo of symptom checker apps in Germany: A cross-sectional survey with a mixed-methods design (CHECK.APP)

Background

Symptom checker apps (SCAs) offer symptom classification and low-threshold self-triage for laypeople. They are already in use despite their poor accuracy and concerns that they may negatively affect primary care. This study assesses the extent to which SCAs are used by medical laypeople in Germany and which software is most popular. We examined associations between satisfaction with the general practitioner (GP) and SCA use as well as the number of GP visits and SCA use. Furthermore, we assessed the reasons for intentional non-use.

Methods

We conducted a survey comprising standardised and open-ended questions. Quantitative data were weighted, and open-ended responses were examined using thematic analysis.

Results

This study included 850 participants. The SCA usage rate was 8%, and approximately 50% of SCA non-users were uninterested in trying SCAs. The most commonly used SCAs were NetDoktor and Ada. Surprisingly, SCAs were most frequently used in the age group of 51-55 years. No significant associations were found between SCA usage and satisfaction with the GP or the number of GP visits and SCA usage. Thematic analysis revealed skepticism regarding the results and recommendations of SCAs and discrepancies between users' requirements and the features of apps.

Conclusion

SCAs are still widely unknown in the German population and have been sparsely used so far. Many participants were not interested in trying SCAs, and we found no positive or negative associations of SCAs and primary care.

Assessing usability of intelligent guidance chatbots in Chinese hospitals: Cross-sectional study

Objective

This study aimed to assessing usability of intelligent guidance chatbots (IGCs) in Chinese hospitals.

Methods

A cross-sectional study based on expert survey was conducted between August to December 2023. The survey assessed the usability of chatbots in 590 Chinese hospitals. One-way ANOVA was used to analyze the impact of the number of functions, human-like characteristics, number of outpatients, and staff size on the usability of the IGCs.

Results

The results indicate that there are 273 (46.27%) hospitals scoring above 45 points. In terms of function development, 581(98.47%) hospitals have set the number of functions between 1 and 5. Besides, 350 hospitals have excellent function implementation, accounting for 59.32%. In terms of the IGC's human-like characteristic, 220 hospitals have both an avatar and a nickname. Results of One-way ANOVA show that, the number of functions(F = 202.667, P < 0.001), human-like characteristics(F = 372.29, P < 0.001), staff size(F = 9.846, P < 0.001), and the number of outpatients(F = 5.709, P = 0.004) have significant impact on the usability of hospital IGCs.

Conclusions

This study found that the differences in the usability of hospital IGCs at various levels of the number of functions, human-like characteristics, number of outpatients, and staff size. These findings provide insights for deploying hospital IGCs and can inform improvements in patient's experience and adoption of chatbots.

Experiences of using artificial intelligence in healthcare: a qualitative study of UK clinician and key stakeholder perspectives

OBJECTIVES

Artificial intelligence (AI) is a rapidly developing field in healthcare, with tools being developed across various specialties to support healthcare professionals and reduce workloads. It is important to understand the experiences of professionals working in healthcare to ensure that future AI tools are acceptable and effectively implemented. The aim of this study was to gain an in-depth understanding of the experiences and perceptions of UK healthcare workers and other key stakeholders about the use of AI in the National Health Service (NHS).

DESIGN

A qualitative study using semistructured interviews conducted remotely via MS Teams. Thematic analysis was carried out.

SETTING

NHS and UK higher education institutes.

PARTICIPANTS

Thirteen participants were recruited, including clinical and non-clinical participants working for the NHS and researchers working to develop AI tools for healthcare settings.

RESULTS

Four core themes were identified: positive perceptions of AI; potential barriers to using AI in healthcare; concerns regarding AI use and steps needed to ensure the acceptability of future AI tools. Overall, we found that those working in healthcare were generally open to the use of AI and expected it to have many benefits for patients and facilitate access to care. However, concerns were raised regarding the security of patient data, the potential for misdiagnosis and that AI could increase the burden on already strained healthcare staff.

CONCLUSION

This study found that healthcare staff are willing to engage with AI research and incorporate AI tools into care pathways. Going forward, the NHS and AI developers will need to collaborate closely to ensure that future tools are suitable for their intended use and do not negatively impact workloads or patient trust. Future AI studies should continue to incorporate the views of key stakeholders to improve tool acceptability.

TRIAL REGISTRATION NUMBER

NCT05028179; ISRCTN15113915; IRAS ref: 293515.

Assessment of a Digital Symptom Checker Tool's Accuracy in Suggesting Reproductive Health Conditions: Clinical Vignettes Study

BACKGROUND

Reproductive health conditions such as endometriosis, uterine fibroids, and polycystic ovary syndrome (PCOS) affect a large proportion of women and people who menstruate worldwide. Prevalence estimates for these conditions range from 5% to 40% of women of reproductive age. Long diagnostic delays, up to 12 years, are common and contribute to health complications and increased health care costs. Symptom checker apps provide users with information and tools to better understand their symptoms and thus have the potential to reduce the time to diagnosis for reproductive health conditions.

OBJECTIVE

This study aimed to evaluate the agreement between clinicians and 3 symptom checkers (developed by Flo Health UK Limited) in assessing symptoms of endometriosis, uterine fibroids, and PCOS using vignettes. We also aimed to present a robust example of vignette case creation, review, and classification in the context of predeployment testing and validation of digital health symptom checker tools.

METHODS

Independent general practitioners were recruited to create clinical case vignettes of simulated users for the purpose of testing each condition symptom checker; vignettes created for each condition contained a mixture of condition-positive and condition-negative outcomes. A second panel of general practitioners then reviewed, approved, and modified (if necessary) each vignette. A third group of general practitioners reviewed each vignette case and designated a final classification. Vignettes were then entered into the symptom checkers by a fourth, different group of general practitioners. The outcomes of each symptom checker were then compared with the final classification of each vignette to produce accuracy metrics including percent agreement, sensitivity, specificity, positive predictive value, and negative predictive value.

RESULTS

A total of 24 cases were created per condition. Overall, exact matches between the vignette general practitioner classification and the symptom checker outcome were 83% (n=20) for endometriosis, 83% (n=20) for uterine fibroids, and 88% (n=21) for PCOS. For each symptom checker, sensitivity was reported as 81.8% for endometriosis, 84.6% for uterine fibroids, and 100% for PCOS; specificity was reported as 84.6% for endometriosis, 81.8% for uterine fibroids, and 75% for PCOS; positive predictive value was reported as 81.8% for endometriosis, 84.6% for uterine fibroids, 80% for PCOS; and negative predictive value was reported as 84.6% for endometriosis, 81.8% for uterine fibroids, and 100% for PCOS.

CONCLUSIONS

The single-condition symptom checkers have high levels of agreement with general practitioner classification for endometriosis, uterine fibroids, and PCOS. Given long delays in diagnosis for many reproductive health conditions, which lead to increased medical costs and potential health complications for individuals and health care providers, innovative health apps and symptom checkers hold the potential to improve care pathways.

Multi-stakeholder preferences for the use of artificial intelligence in healthcare: A systematic review and thematic analysis

INTRODUCTION

Despite the proliferation of Artificial Intelligence (AI) technology over the last decade, clinician, patient, and public perceptions of its use in healthcare raise a number of ethical, legal and social questions. We systematically review the literature on attitudes towards the use of AI in healthcare from patients, the general public and health professionals' perspectives to understand these issues from multiple perspectives.

METHODOLOGY

A search for original research articles using qualitative, quantitative, and mixed methods published between 1 Jan 2001 to 24 Aug 2021 was conducted on six bibliographic databases. Data were extracted and classified into different themes representing views on: (i) knowledge and familiarity of AI, (ii) AI benefits, risks, and challenges, (iii) AI acceptability, (iv) AI development, (v) AI implementation, (vi) AI regulations, and (vii) Human - AI relationship.

RESULTS

The final search identified 7,490 different records of which 105 publications were selected based on predefined inclusion/exclusion criteria. While the majority of patients, the general public and health professionals generally had a positive attitude towards the use of AI in healthcare, all groups indicated some perceived risks and challenges. Commonly perceived risks included data privacy; reduced professional autonomy; algorithmic bias; healthcare inequities; and greater burnout to acquire AI-related skills. While patients had mixed opinions on whether healthcare workers suffer from job loss due to the use of AI, health professionals strongly indicated that AI would not be able to completely replace them in their professions. Both groups shared similar doubts about AI's ability to deliver empathic care. The need for AI validation, transparency, explainability, and patient and clinical involvement in the development of AI was emphasised. To help successfully implement AI in health care, most participants envisioned that an investment in training and education campaigns was necessary, especially for health professionals. Lack of familiarity, lack of trust, and regulatory uncertainties were identified as factors hindering AI implementation. Regarding AI regulations, key themes included data access and data privacy. While the general public and patients exhibited a willingness to share anonymised data for AI development, there remained concerns about sharing data with insurance or technology companies. One key domain under this theme was the question of who should be held accountable in the case of adverse events arising from using AI.

CONCLUSIONS

While overall positivity persists in attitudes and preferences toward AI use in healthcare, some prevalent problems require more attention. There is a need to go beyond addressing algorithm-related issues to look at the translation of legislation and guidelines into practice to ensure fairness, accountability, transparency, and ethics in AI.

Artificial Intelligence in Head and Neck Cancer: A Systematic Review of Systematic Reviews

INTRODUCTION

Several studies have emphasized the potential of artificial intelligence (AI) and its subfields, such as machine learning (ML), as emerging and feasible approaches to optimize patient care in oncology. As a result, clinicians and decision-makers are faced with a plethora of reviews regarding the state of the art of applications of AI for head and neck cancer (HNC) management. This article provides an analysis of systematic reviews on the current status, and of the limitations of the application of AI/ML as adjunctive decision-making tools in HNC management.

METHODS

Electronic databases (PubMed, Medline via Ovid, Scopus, and Web of Science) were searched from inception until November 30, 2022. The study selection, searching and screening processes, inclusion, and exclusion criteria followed the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) guidelines. A risk of bias assessment was conducted using a tailored and modified version of the Assessment of Systematic Review (AMSTAR-2) tool and quality assessment using the Risk of Bias in Systematic Reviews (ROBIS) guidelines.

RESULTS

Of the 137 search hits retrieved, 17 fulfilled the inclusion criteria. This analysis of systematic reviews revealed that the application of AI/ML as a decision aid in HNC management can be thematized as follows: (1) detection of precancerous and cancerous lesions within histopathologic slides; (2) prediction of the histopathologic nature of a given lesion from various sources of medical imaging; (3) prognostication; (4) extraction of pathological findings from imaging; and (5) different applications in radiation oncology. In addition, the challenges in implementation of AI/ML models for clinical evaluations include the lack of standardized methodological guidelines for the collection of clinical images, development of these models, reporting of their performance, external validation procedures, and regulatory frameworks.

CONCLUSION

At present, there is a paucity of evidence to suggest the adoption of these models in clinical practice due to the aforementioned limitations. Therefore, this manuscript highlights the need for development of standardized guidelines to facilitate the adoption and implementation of these models in the daily clinical practice. In addition, adequately powered, prospective, randomized controlled trials are urgently needed to further assess the potential of AI/ML models in real-world clinical settings for the management of HNC.

Characteristics of Users and Nonusers of Symptom Checkers in Germany: Cross-Sectional Survey Study

BACKGROUND

Previous studies have revealed that users of symptom checkers (SCs, apps that support self-diagnosis and self-triage) are predominantly female, are younger than average, and have higher levels of formal education. Little data are available for Germany, and no study has so far compared usage patterns with people's awareness of SCs and the perception of usefulness.

OBJECTIVE

We explored the sociodemographic and individual characteristics that are associated with the awareness, usage, and perceived usefulness of SCs in the German population.

METHODS

We conducted a cross-sectional online survey among 1084 German residents in July 2022 regarding personal characteristics and people's awareness and usage of SCs. Using random sampling from a commercial panel, we collected participant responses stratified by gender, state of residence, income, and age to reflect the German population. We analyzed the collected data exploratively.

RESULTS

Of all respondents, 16.3% (177/1084) were aware of SCs and 6.5% (71/1084) had used them before. Those aware of SCs were younger (mean 38.8, SD 14.6 years, vs mean 48.3, SD 15.7 years), were more often female (107/177, 60.5%, vs 453/907, 49.9%), and had higher formal education levels (eg, 72/177, 40.7%, vs 238/907, 26.2%, with a university/college degree) than those unaware. The same observation applied to users compared to nonusers. It disappeared, however, when comparing users to nonusers who were aware of SCs. Among users, 40.8% (29/71) considered these tools useful. Those considering them useful reported higher self-efficacy (mean 4.21, SD 0.66, vs mean 3.63, SD 0.81, on a scale of 1-5) and a higher net household income (mean EUR 2591.63, SD EUR 1103.96 [mean US $2798.96, SD US $1192.28], vs mean EUR 1626.60, SD EUR 649.05 [mean US $1756.73, SD US $700.97]) than those who considered them not useful. More women considered SCs unhelpful (13/44, 29.5%) compared to men (4/26, 15.4%).

CONCLUSIONS

Concurring with studies from other countries, our findings show associations between sociodemographic characteristics and SC usage in a German sample: users were on average younger, of higher socioeconomic status, and more commonly female compared to nonusers. However, usage cannot be explained by sociodemographic differences alone. It rather seems that sociodemographics explain who is or is not aware of the technology, but those who are aware of SCs are equally likely to use them, independently of sociodemographic differences. Although in some groups (eg, people with anxiety disorder), more participants reported to know and use SCs, they tended to perceive them as less useful. In other groups (eg, male participants), fewer respondents were aware of SCs, but those who used them perceived them to be more useful. Thus, SCs should be designed to fit specific user needs, and strategies should be developed to help reach individuals who could benefit but are not aware of SCs yet.

Evaluation of a Medical Interview-Assistance System Using Artificial Intelligence for Resident Physicians Interviewing Simulated Patients: A Crossover, Randomized, Controlled Trial

Medical interviews are expected to undergo a major transformation through the use of artificial intelligence. However, artificial intelligence-based systems that support medical interviews are not yet widespread in Japan, and their usefulness is unclear. A randomized, controlled trial to determine the usefulness of a commercial medical interview support system using a question flow chart-type application based on a Bayesian model was conducted. Ten resident physicians were allocated to two groups with or without information from an artificial intelligence-based support system. The rate of correct diagnoses, amount of time to complete the interviews, and number of questions they asked were compared between the two groups. Two trials were conducted on different dates, with a total of 20 resident physicians participating. Data for 192 differential diagnoses were obtained. There was a significant difference in the rate of correct diagnosis between the two groups for two cases and for overall cases (0.561 vs. 0.393; p = 0.02). There was a significant difference in the time required between the two groups for overall cases (370 s (352-387) vs. 390 s (373-406), p = 0.04). Artificial intelligence-assisted medical interviews helped resident physicians make more accurate diagnoses and reduced consultation time. The widespread use of artificial intelligence systems in clinical settings could contribute to improving the quality of medical care.

Triage and Diagnostic Accuracy of Online Symptom Checkers: Systematic Review

BACKGROUND

In the context of a deepening global shortage of health workers and, in particular, the COVID-19 pandemic, there is growing international interest in, and use of, online symptom checkers (OSCs). However, the evidence surrounding the triage and diagnostic accuracy of these tools remains inconclusive.

OBJECTIVE

This systematic review aimed to summarize the existing peer-reviewed literature evaluating the triage accuracy (directing users to appropriate services based on their presenting symptoms) and diagnostic accuracy of OSCs aimed at lay users for general health concerns.

METHODS

Searches were conducted in MEDLINE, Embase, CINAHL, Health Management Information Consortium (HMIC), and Web of Science, as well as the citations of the studies selected for full-text screening. We included peer-reviewed studies published in English between January 1, 2010, and February 16, 2022, with a controlled and quantitative assessment of either or both triage and diagnostic accuracy of OSCs directed at lay users. We excluded tools supporting health care professionals, as well as disease- or specialty-specific OSCs. Screening and data extraction were carried out independently by 2 reviewers for each study. We performed a descriptive narrative synthesis.

RESULTS

A total of 21,296 studies were identified, of which 14 (0.07%) were included. The included studies used clinical vignettes, medical records, or direct input by patients. Of the 14 studies, 6 (43%) reported on triage and diagnostic accuracy, 7 (50%) focused on triage accuracy, and 1 (7%) focused on diagnostic accuracy. These outcomes were assessed based on the diagnostic and triage recommendations attached to the vignette in the case of vignette studies or on those provided by nurses or general practitioners, including through face-to-face and telephone consultations. Both diagnostic accuracy and triage accuracy varied greatly among OSCs. Overall diagnostic accuracy was deemed to be low and was almost always lower than that of the comparator. Similarly, most of the studies (9/13, 69 %) showed suboptimal triage accuracy overall, with a few exceptions (4/13, 31%). The main variables affecting the levels of diagnostic and triage accuracy were the severity and urgency of the condition, the use of artificial intelligence algorithms, and demographic questions. However, the impact of each variable differed across tools and studies, making it difficult to draw any solid conclusions. All included studies had at least one area with unclear risk of bias according to the revised Quality Assessment of Diagnostic Accuracy Studies-2 tool.

CONCLUSIONS

Although OSCs have potential to provide accessible and accurate health advice and triage recommendations to users, more research is needed to validate their triage and diagnostic accuracy before widescale adoption in community and health care settings. Future studies should aim to use a common methodology and agreed standard for evaluation to facilitate objective benchmarking and validation.

TRIAL REGISTRATION

PROSPERO CRD42020215210; https://tinyurl.com/3949zw83.

Impacts of Symptom Checkers for Laypersons' Self-diagnosis on Physicians in Primary Care: Scoping Review

BACKGROUND

Symptom checkers (SCs) for laypersons' self-assessment and preliminary self-diagnosis are widely used by the public. Little is known about the impact of these tools on health care professionals (HCPs) in primary care and their work. This is relevant to understanding how technological changes might affect the working world and how this is linked to work-related psychosocial demands and resources for HCPs.

OBJECTIVE

This scoping review aimed to systematically explore the existing publications on the impacts of SCs on HCPs in primary care and to identify knowledge gaps.

METHODS

We used the Arksey and O'Malley framework. We based our search string on the participant, concept, and context scheme and searched PubMed (MEDLINE) and CINAHL in January and June 2021. We performed a reference search in August 2021 and a manual search in November 2021. We included publications of peer-reviewed journals that focused on artificial intelligence- or algorithm-based self-diagnosing apps and tools for laypersons and had primary care or nonclinical settings as a relevant context. The characteristics of these studies were described numerically. We used thematic analysis to identify core themes. We followed the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) checklist to report the study.

RESULTS

Of the 2729 publications identified through initial and follow-up database searches, 43 full texts were screened for eligibility, of which 9 were included. Further 8 publications were included through manual search. Two publications were excluded after receiving feedback in the peer-review process. Fifteen publications were included in the final sample, which comprised 5 (33%) commentaries or nonresearch publications, 3 (20%) literature reviews, and 7 (47%) research publications. The earliest publications stemmed from 2015. We identified 5 themes. The theme finding prediagnosis comprised the comparison between SCs and physicians. We identified the performance of the diagnosis and the relevance of human factors as topics. In the theme layperson-technology relationship, we identified potentials for laypersons' empowerment and harm through SCs. Our analysis showed potential disruptions of the physician-patient relationship and uncontested roles of HCPs in the theme (impacts on) physician-patient relationship. In the theme impacts on HCPs' tasks, we described the reduction or increase in HCPs' workload. We identified potential transformations of HCPs' work and impacts on the health care system in the theme future role of SCs in health care.

CONCLUSIONS

The scoping review approach was suitable for this new field of research. The heterogeneity of technologies and wordings was challenging. We identified research gaps in the literature regarding the impact of artificial intelligence- or algorithm-based self-diagnosing apps or tools on the work of HCPs in primary care. Further empirical studies on HCPs' lived experiences are needed, as the current literature depicts expectations rather than empirical findings.

Artificial intelligence for secondary prevention of myocardial infarction: A qualitative study of patient and health professional perspectives

BACKGROUND

Artificial intelligence (AI) has potential to improve self-management of several chronic conditions. However, the perspective of patients and healthcare professionals regarding AI-enabled health management programs, which are key to successful implementation, remains poorly understood.

PURPOSE

To explore the opinions of people with a history of myocardial infarction (PHMI) and health professionals on the use of AI for secondary prevention of MI.

PROCEDURE

Three rounds of focus groups were conducted via videoconferencing with 38 participants: 22 PHMI and 16 health professionals.

FINDINGS

We identified 21 concepts stemming from participants' views, which we classified into five categories: Trust; Expected Functions; Adoption; Concerns; and Perceived Benefits. Trust covered the credibility of information and safety to believe health advice. Expected Functions covered tailored feedback and personalised advice. Adoption included usability features and overall interest in AI. Concerns originated from previous negative experience with AI. Perceived Benefits included the usefulness of AI to provide advice when regular contact with healthcare services is not feasible. Health professionals were more optimistic than PHMI about the usefulness of AI for improving health behaviour.

CONCLUSIONS

Altogether, our findings provide key insights from end-users to improve the likelihood of successful implementation and adoption of AI-enabled systems in the context of MI, as an exemplar of broader applications in chronic disease management.

User Experience of Symptom Checkers: A Systematic Review

This review reports the user experience of symptom checkers, aiming to characterize users studied in the existing literature, identify the aspects of user experience of symptom checkers that have been studied, and offer design suggestions. Our literature search resulted in 31 publications. We found that (1) most symptom checker users are relatively young; (2) eight relevant aspects of user experience have been explored, including motivation, trust, acceptability, satisfaction, accuracy, usability, safety/security, and functionality; (3) future symptom checkers should improve their accuracy, safety, and usability. Although many facets of user experience have been explored, methodological challenges exist and some important aspects of user experience remain understudied. Further research should be conducted to explore users' needs and the context of use. More qualitative and mixed-method studies are needed to understand actual users' experiences in the future.

Preparing for an Artificial Intelligence-Enabled Future: Patient Perspectives on Engagement and Health Care Professional Training for Adopting Artificial Intelligence Technologies in Health Care Settings

BACKGROUND

As new technologies emerge, there is a significant shift in the way care is delivered on a global scale. Artificial intelligence (AI) technologies have been rapidly and inexorably used to optimize patient outcomes, reduce health system costs, improve workflow efficiency, and enhance population health. Despite the widespread adoption of AI technologies, the literature on patient engagement and their perspectives on how AI will affect clinical care is scarce. Minimal patient engagement can limit the optimization of these novel technologies and contribute to suboptimal use in care settings.

OBJECTIVE

We aimed to explore patients' views on what skills they believe health care professionals should have in preparation for this AI-enabled future and how we can better engage patients when adopting and deploying AI technologies in health care settings.

METHODS

Semistructured interviews were conducted from August 2020 to December 2021 with 12 individuals who were a patient in any Canadian health care setting. Interviews were conducted until thematic saturation occurred. A thematic analysis approach outlined by Braun and Clarke was used to inductively analyze the data and identify overarching themes.

RESULTS

Among the 12 patients interviewed, 8 (67%) were from urban settings and 4 (33%) were from rural settings. A majority of the participants were very comfortable with technology (n=6, 50%) and somewhat familiar with AI (n=7, 58%). In total, 3 themes emerged: cultivating patients' trust, fostering patient engagement, and establishing data governance and validation of AI technologies.

CONCLUSIONS

With the rapid surge of AI solutions, there is a critical need to understand patient values in advancing the quality of care and contributing to an equitable health system. Our study demonstrated that health care professionals play a synergetic role in the future of AI and digital technologies. Patient engagement is vital in addressing underlying health inequities and fostering an optimal care experience. Future research is warranted to understand and capture the diverse perspectives of patients with various racial, ethnic, and socioeconomic backgrounds.

Exploratory study: Evaluation of a symptom checker effectiveness for providing a diagnosis and evaluating the situation emergency compared to emergency physicians using simulated and standardized patients

BACKGROUND

The overloading of health care systems is an international problem. In this context, new tools such as symptom checker (SC) are emerging to improve patient orientation and triage. This SC should be rigorously evaluated and we can take a cue from the way we evaluate medical students, using objective structured clinical examinations (OSCE) with simulated patients.

OBJECTIVE

The main objective of this study was to evaluate the efficiency of a symptom checker versus emergency physicians using OSCEs as an assessment method.

METHODS

We explored a method to evaluate the ability to set a diagnosis and evaluate the emergency of a situation with simulation. A panel of medical experts wrote 220 simulated patients cases. Each situation was played twice by an actor trained to the role: once for the SC, then for an emergency physician. Like a teleconsultation, only the patient's voice was accessible. We performed a prospective non-inferiority study. If primary analysis had failed to detect non-inferiority, we have planned a superiority analysis.

RESULTS

The SC established only 30% of the main diagnosis as the emergency physician found 81% of these. The emergency physician was also superior compared to the SC in the suggestion of secondary diagnosis (92% versus 52%). In the matter of patient triage (vital emergency or not), there is still a medical superiority (96% versus 71%). We prove a non-inferiority of the SC compared to the physician in terms of interviewing time.

CONCLUSIONS AND RELEVANCE

We should use simulated patients instead of clinical cases in order to evaluate the effectiveness of SCs.

Artificial intelligence (AI) acceptance in primary care during the coronavirus pandemic: What is the role of patients' gender, age and health awareness? A two-phase pilot study

Background

Artificial intelligence (AI) is steadily entering and transforming the health care and Primary Care (PC) domains. AI-based applications assist physicians in disease detection, medical advice, triage, clinical decision-making, diagnostics and digital public health. Recent literature has explored physicians' perspectives on the potential impact of digital public health on key tasks in PC. However, limited attention has been given to patients' perspectives of AI acceptance in PC, specifically during the coronavirus pandemic. Addressing this research gap, we administered a pilot study to investigate criteria for patients' readiness to use AI-based PC applications by analyzing key factors affecting the adoption of digital public health technology.

Methods

The pilot study utilized a two-phase mixed methods approach. First, we conducted a qualitative study with 18 semi-structured interviews. Second, based on the Technology Readiness and Acceptance Model (TRAM), we conducted an online survey (n = 447).

Results

The results indicate that respondents who scored high on innovativeness had a higher level of readiness to use AI-based technology in PC during the coronavirus pandemic. Surprisingly, patients' health awareness and sociodemographic factors, such as age, gender and education, were not significant predictors of AI-based technology acceptance in PC.

Conclusions

This paper makes two major contributions. First, we highlight key social and behavioral determinants of acceptance of AI-enabled health care and PC applications. Second, we propose that to increase the usability of digital public health tools and accelerate patients' AI adoption, in complex digital public health care ecosystems, we call for implementing adaptive, population-specific promotions of AI technologies and applications.

A scoping review on the use and usefulness of online symptom checkers and triage systems: How to proceed?

Background

Patients are increasingly turning to the Internet for health information. Numerous online symptom checkers and digital triage tools are currently available to the general public in an effort to meet this need, simultaneously acting as a demand management strategy to aid the overburdened health care system. The implementation of these services requires an evidence-based approach, warranting a review of the available literature on this rapidly evolving topic.

Objective

This scoping review aims to provide an overview of the current state of the art and identify research gaps through an analysis of the strengths and weaknesses of the presently available literature.

Methods

A systematic search strategy was formed and applied to six databases: Cochrane library, NICE, DARE, NIHR, Pubmed, and Web of Science. Data extraction was performed by two researchers according to a pre-established data charting methodology allowing for a thematic analysis of the results.

Results

A total of 10,250 articles were identified, and 28 publications were found eligible for inclusion. Users of these tools are often younger, female, more highly educated and technologically literate, potentially impacting digital divide and health equity. Triage algorithms remain risk-averse, which causes challenges for their accuracy. Recent evolutions in algorithms have varying degrees of success. Results on impact are highly variable, with potential effects on demand, accessibility of care, health literacy and syndromic surveillance. Both patients and healthcare providers are generally positive about the technology and seem amenable to the advice given, but there are still improvements to be made toward a more patient-centered approach. The significant heterogeneity across studies and triage systems remains the primary challenge for the field, limiting transferability of findings.

Conclusion

Current evidence included in this review is characterized by significant variability in study design and outcomes, highlighting the significant challenges for future research.An evolution toward more homogeneous methodologies, studies tailored to the intended setting, regulation and standardization of evaluations, and a patient-centered approach could benefit the field.

Ethical, legal, and social aspects of symptom checker applications: a scoping review

Symptom Checker Applications (SCA) are mobile applications often designed for the end-user to assist with symptom assessment and self-triage. SCA are meant to provide the user with easily accessible information about their own health conditions. However, SCA raise questions regarding ethical, legal, and social aspects (ELSA), for example, regarding fair access to this new technology. The aim of this scoping review is to identify the ELSA of SCA in the scientific literature. A scoping review was conducted to identify the ELSA of SCA. Ten databases (e.g., Web of Science and PubMed) were used. Studies on SCA that address ELSA, written in English or German, were included in the review. The ELSA of SCA were extracted and synthesized using qualitative content analysis. A total of 25,061 references were identified, of which 39 were included in the analysis. The identified aspects were allotted to three main categories: (1) Technology; (2) Individual Level; and (3) Healthcare system. The results show that there are controversial debates in the literature on the ethical and social challenges of SCA usage. Furthermore, the debates are characterised by a lack of a specific legal perspective and empirical data. The review provides an overview on the spectrum of ELSA regarding SCA. It offers guidance to stakeholders in the healthcare system, for example, patients, healthcare professionals, and insurance providers and could be used in future empirical research to investigate the perspectives of those affected, such as users.

A Sociotechnical Systems Framework for the Application of Artificial Intelligence in Health Care Delivery

In the coming years, artificial intelligence (AI) will pervade almost every aspect of the health care delivery system. AI has the potential to improve patient safety (e.g. diagnostic accuracy) as well as reduce the burden on clinicians (e.g. documentation-related workload); however, these benefits are yet to be realized. AI is only one element of a larger sociotechnical system that needs to be considered for effective AI application. In this paper, we describe the current challenges of integrating AI into clinical care and propose a sociotechnical systems (STS) approach for AI design and implementation. We demonstrate the importance of an STS approach through a case study on the design and implementation of a clinical decision support (CDS). In order for AI to reach its potential, the entire work system as well as clinical workflow must be systematically considered throughout the design of AI technology.

Utilization patterns and efficiency gains from use of a fully EHR-integrated COVID-19 self-triage and self-scheduling tool: a retrospective analysis

OBJECTIVE

Symptom checkers can help address high demand for SARS-CoV2 (COVID-19) testing and care by providing patients with self-service access to triage recommendations. However, health systems may be hesitant to invest in these tools, as their associated efficiency gains have not been studied. We aimed to quantify the operational efficiency gains associated with use of an online COVID-19 symptom checker as an alternative to a telephone hotline.

METHODS

In our health system, ambulatory patients can either use an online symptom checker or a telephone hotline to be triaged and connected to COVID-19 care. We performed a retrospective analysis of adults who used either method between October 20, 2021 and January 10, 2022, using call logs, electronic health record data, and local wages to calculate labor costs.

RESULTS

Of the 15 549 total COVID-19 triage encounters, 1820 (11.7%) used only the telephone hotline and 13 729 (88.3%) used the symptom checker. Only 271 (2%) of the patients who used the symptom checker also called the hotline. Hotline encounters required more clinician time compared to those involving the symptom checker (17.8 vs 0.4 min/encounter), resulting in higher average labor costs ($24.21 vs $0.55 per encounter). The symptom checker resulted in over 4200 clinician labor hours saved.

CONCLUSION

When given the option, most patients completed COVID-19 triage and visit scheduling online, resulting in substantial efficiency gains. These benefits may encourage health system investment in such tools.

Online Symptom Checkers are Poor Tools for Diagnosing Men's Health Conditions

OBJECTIVE

To analyze the accuracy of the four most commonly used online symptom checkers (OSCs) in diagnosing erectile dysfunction (ED), scrotal pain (SP), Peyronie's disease (PD), and low testosterone (LT).

METHODS AND OUTCOMES

One-hundred and sixty artificial vignettes were created by de-identifying recent initial outpatient consults presenting to discuss ED (40), SP (40), PD (40), and LT (40). The vignettes were entered into the 4 most frequently used OSCs (WebMD, MedicineNet, EverydayHealth, and SutterHealth) as determined by web traffic analysis tools. The top 5 conditions listed in the OSC differential diagnosis were recorded and scored.

RESULTS

WebMD's accuracy for ED, SP, PD, and LT vignettes was 0%, 22.5%, 0%, and 95%, respectively. EverydayHealth was only able to diagnose SP 20% of the time, and failed to diagnose ED, PD, or LT on all occasions. MedicineNet diagnosed ED, PD, SP, and LT in 100%, 98%, 27.5%, and 0% of vignettes, respectively. SutterHealth correctly diagnosed ED, SP, and LT in 100%, 20%, and 80% of patients, respectively. Cumulatively, the OSCs were most accurate in diagnosing ED and least accurate in diagnosing SP when using the Top 1 (37.5% vs. 6.9%) and Top 5 (50% vs. 24.5%) of the suggested conditions.

CONCLUSIONS

No OSC could accurately diagnose all the conditions tested. The OSCs, on average, were poor at suggesting precise diagnoses for ED, PD, LT, SP. Patients and practitioners should be cautioned regarding the accuracy of OSCs.

Comparison of physician and artificial intelligence-based symptom checker diagnostic accuracy

Symptom checkers are increasingly used to assess new symptoms and navigate the health care system. The aim of this study was to compare the accuracy of an artificial intelligence (AI)-based symptom checker (Ada) and physicians regarding the presence/absence of an inflammatory rheumatic disease (IRD). In this survey study, German-speaking physicians with prior rheumatology working experience were asked to determine IRD presence/absence and suggest diagnoses for 20 different real-world patient vignettes, which included only basic health and symptom-related medical history. IRD detection rate and suggested diagnoses of participants and Ada were compared to the gold standard, the final rheumatologists’ diagnosis, reported on the discharge summary report. A total of 132 vignettes were completed by 33 physicians (mean rheumatology working experience 8.8 (SD 7.1) years). Ada’s diagnostic accuracy (IRD) was significantly higher compared to physicians (70 vs 54%, p = 0.002) according to top diagnosis. Ada listed the correct diagnosis more often compared to physicians (54 vs 32%, p < 0.001) as top diagnosis as well as among the top 3 diagnoses (59 vs 42%, p < 0.001). Work experience was not related to suggesting the correct diagnosis or IRD status. Confined to basic health and symptom-related medical history, the diagnostic accuracy of physicians was lower compared to an AI-based symptom checker. These results highlight the potential of using symptom checkers early during the patient journey and importance of access to complete and sufficient patient information to establish a correct diagnosis.

User initiated symptom assessment with an electronic symptom checker. Study protocol for mixed-methods validation. (Preprint)

BACKGROUND

The national Omaolo digital social welfare and health care service of Finland provides a symptom checker, Omaolo, which is a medical device (based on Duodecim Clinical Decision Support EBMEDS software) with a CE marking (risk class IIa), manufactured by the government-owned DigiFinland Oy. Users of this service can perform their triage by using the questions in the symptom checker. By completing the symptom checker, the user receives a recommendation for action and a service assessment with appropriate guidance regarding their health problems on the basis of a selected specific symptom in the symptom checker. This allows users to be provided with appropriate health care services, regardless of time and place.

OBJECTIVE

This study describes the protocol for the mixed methods validation process of the symptom checker available in Omaolo digital services.

METHODS

This is a mixed methods study using quantitative and qualitative methods, which will be part of the clinical validation process that takes place in primary health care centers in Finland. Each organization provides a space where the study and the nurse triage can be done in order to include an unscreened target population of users. The primary health care units provide walk-in model services, where no prior phone call or contact is required. For the validation of the Omaolo symptom checker, case vignettes will be incorporated to supplement the triage accuracy of rare and acute cases that cannot be tested extensively in real-life settings. Vignettes are produced from a variety of clinical sources, and they test the symptom checker in different triage levels by using 1 standardized patient case example.

RESULTS

This study plan underwent an ethics review by the regional permission, which was requested from each organization participating in the research, and an ethics committee statement was requested and granted from Pirkanmaa hospital district's ethics committee, which is in accordance with the University of Tampere's regulations. Of 964 clinical user-filled symptom checker assessments, 877 cases were fully completed with a triage result, and therefore, they met the requirements for clinical validation studies. The goal for sufficient data has been reached for most of the chief symptoms. Data collection was completed in September 2019, and the first feasibility and patient experience results were published by the end of 2020. Case vignettes have been identified and are to be completed before further testing the symptom checker. The analysis and reporting are estimated to be finalized in 2024.

CONCLUSIONS

The primary goals of this multimethod electronic symptom checker study are to assess safety and to provide crucial information regarding the accuracy and usability of the Omaolo electronic symptom checker. To our knowledge, this will be the first study to include real-life clinical cases along with case vignettes.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/41423.

Effects and utility of an online forward triage tool during the SARS-CoV-2 pandemic: a mixed method study and patient perspectives, Switzerland

OBJECTIVE

To assess the effects (quantitatively) and the utility (qualitatively) of a COVID-19 online forward triage tool (OFTT) in a pandemic context.

DESIGN

A mixed method sequential explanatory study was employed. Quantitative data of all OFTT users, between 2 March 2020 and 12 May 2020, were collected. Second, qualitative data were collected through key informant interviews (n=19) to explain the quantitative findings, explore tool utility, user experience and elicit recommendations.

SETTING

The working group e-emergency medicine at the emergency department developed an OFTT, which was made available online.

PARTICIPANTS

Participants included all users above the age of 18 that used the OFTT between 2 March 2020 and 12 May 2020.

INTERVENTION

An OFTT that displayed the current test recommendations of the Federal Office of Public Health on whether someone needed testing for COVID-19 or not. No diagnosis was provided.

RESULTS

In the study period, 6272 users consulted our OFTT; 40.2% (1626/4049) would have contacted a healthcare provider had the tool not existed. 560 participants consented to a follow-up survey and provided a valid email address. 31.4% (176/560) participants returned a complete follow-up questionnaire. 84.7% (149/176) followed the recommendations given. 41.5% (73/176) reported that their fear was allayed after using the tool. Qualitatively, seven overarching themes emerged namely (1) accessibility of tool, (2) user-friendliness of tool, (3) utility of tool as an information source, (4) utility of tool in allaying fear and anxiety, (5) utility of tool in medical decision-making (6) utility of tool in reducing the potential for onward transmissions and (7) utility of tool in reducing health system burden.

CONCLUSION

Our findings demonstrated that a COVID-19 OFTT does not only reduce the health system burden but can also serve as an information source, reduce anxiety and fear, reduce potential for cross infections and facilitate medical decision-making.

Analyzing the determinants to accept a virtual assistant and use cases among cancer patients: a mixed methods study

Background

Technological progress in artificial intelligence has led to the increasing popularity of virtual assistants, i.e., embodied or disembodied conversational agents that allow chatting with a technical system in a natural language. However, only little comprehensive research is conducted about patients' perceptions and possible applications of virtual assistant in healthcare with cancer patients. This research aims to investigate the key acceptance factors and value-adding use cases of a virtual assistant for patients diagnosed with cancer.

Methods

Qualitative interviews with eight former patients and four doctors of a Dutch radiotherapy institute were conducted to determine what acceptance factors they find most important for a virtual assistant and gain insights into value-adding applications. The unified theory of acceptance and use of technology (UTAUT) was used to structure perceptions and was inductively modified as a result of the interviews. The subsequent research model was triangulated via an online survey with 127 respondents diagnosed with cancer. A structural equation model was used to determine the relevance of acceptance factors. Through a multigroup analysis, differences between sample subgroups were compared.

Results

The interviews found support for all factors of the UTAUT: performance expectancy, effort expectancy, social influence and facilitating conditions. Additionally, self-efficacy, trust, and resistance to change, were added as an extension of the UTAUT. Former patients found a virtual assistant helpful in receiving information about logistic questions, treatment procedures, side effects, or scheduling appointments. The quantitative study found that the constructs performance expectancy (ß = 0.399), effort expectancy (ß = 0.258), social influence (ß = 0.114), and trust (ß = 0.210) significantly influenced behavioral intention to use a virtual assistant, explaining 80% of its variance. Self-efficacy (ß = 0.792) acts as antecedent of effort expectancy. Facilitating conditions and resistance to change were not found to have a significant relationship with user intention.

Conclusions

Performance and effort expectancy are the leading determinants of virtual assistant acceptance. The latter is dependent on a patient’s self-efficacy. Therefore, including patients during the development and introduction of a VA in cancer treatment is important. The high relevance of trust indicates the need for a reliable, secure service that should be promoted as such. Social influence suggests using doctors in endorsing the VA.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-022-08189-7.

Diagnostic accuracy of an artificial intelligence online engine in migraine: A multi‐center study

Objective

This study assesses the concordance in migraine diagnosis between an online, self‐administered, Computer‐based, Diagnostic Engine (CDE) and semi‐structured interview (SSI) by a headache specialist, both using International Classification of Headache Disorders, 3rd edition (ICHD‐3) criteria.

Background

Delay in accurate diagnosis is a major barrier to headache care. Accurate computer‐based algorithms may help reduce the need for SSI‐based encounters to arrive at correct ICHD‐3 diagnosis.

Methods

Between March 2018 and August 2019, adult participants were recruited from three academic headache centers and the community via advertising to our cross‐sectional study. Participants completed two evaluations: phone interview conducted by headache specialists using the SSI and a web‐based expert questionnaire and analytics, CDE. Participants were randomly assigned to either the SSI followed by the web‐based questionnaire or the web‐based questionnaire followed by the SSI. Participants completed protocols a few minutes apart. The concordance in migraine/probable migraine (M/PM) diagnosis between SSI and CDE was measured using Cohen’s kappa statistics. The diagnostic accuracy of CDE was assessed using the SSI as reference standard.

Results

Of the 276 participants consented, 212 completed both SSI and CDE (study completion rate = 77%; median age = 32 years [interquartile range: 28–40], female:male ratio = 3:1). Concordance in M/PM diagnosis between SSI and CDE was: κ = 0.83 (95% confidence interval [CI]: 0.75–0.91). CDE diagnostic accuracy: sensitivity = 90.1% (118/131), 95% CI: 83.6%–94.6%; specificity = 95.8% (68/71), 95% CI: 88.1%–99.1%. Positive and negative predictive values = 97.0% (95% CI: 91.3%–99.0%) and 86.6% (95% CI: 79.3%–91.5%), respectively, using identified migraine prevalence of 60%. Assuming a general migraine population prevalence of 10%, positive and negative predictive values were 70.3% (95% CI: 43.9%–87.8%) and 98.9% (95% CI: 98.1%–99.3%), respectively.

Conclusion

The SSI and CDE have excellent concordance in diagnosing M/PM. Positive CDE helps rule in M/PM, through high specificity and positive likelihood ratio. A negative CDE helps rule out M/PM through high sensitivity and low negative likelihood ratio. CDE that mimics SSI logic is a valid tool for migraine diagnosis.

Ethical, Legal, and Social Implications of Symptom Checker Apps in Primary Health Care (CHECK.APP): Protocol for an Interdisciplinary Mixed Methods Study

Background

Symptom checker apps (SCAs) are accessible tools that provide early symptom assessment for users. The ethical, legal, and social implications of SCAs and their impact on the patient-physician relationship, the health care providers, and the health care system have sparsely been examined. This study protocol describes an approach to investigate the possible impacts and implications of SCAs on different levels of health care provision. It considers the perspectives of the users, nonusers, general practitioners (GPs), and health care experts.

Objective

We aim to assess a comprehensive overview of the use of SCAs and address problematic issues, if any. The primary outcomes of this study are empirically informed multi-perspective recommendations for different stakeholders on the ethical, legal, and social implications of SCAs.

Methods

Quantitative and qualitative methods will be used in several overlapping and interconnected study phases. In study phase 1, a comprehensive literature review will be conducted to assess the ethical, legal, social, and systemic impacts of SCAs. Study phase 2 comprises a survey that will be analyzed with a logistic regression. It aims to assess the user degree of SCAs in Germany as well as the predictors for SCA usage. Study phase 3 will investigate self-observational diaries and user interviews, which will be analyzed as integrated cases to assess user perspectives, usage pattern, and arising problems. Study phase 4 will comprise GP interviews to assess their experiences, perspectives, self-image, and concepts and will be analyzed with the basic procedure by Kruse. Moreover, interviews with health care experts will be conducted in study phase 3 and will be analyzed by using the reflexive thematical analysis approach of Braun and Clark.

Results

Study phase 1 will be completed in November 2021. We expect the results of study phase 2 in December 2021 and February 2022. In study phase 3, interviews are currently being conducted. The final study endpoint will be in February 2023.

Conclusions

The possible ethical, legal, social, and systemic impacts of a widespread use of SCAs that affect stakeholders and stakeholder groups on different levels of health care will be identified. The proposed methodological approach provides a multifaceted and diverse empirical basis for a broad discussion on these implications.

Trial Registration

German Clinical Trials Register (DRKS) DRKS00022465; https://tinyurl.com/yx53er67

International Registered Report Identifier (IRRID)

DERR1-10.2196/34026

Triage Accuracy of Symptom Checker Apps: 5-Year Follow-up Evaluation

BACKGROUND

Symptom checkers are digital tools assisting laypersons in self-assessing the urgency and potential causes of their medical complaints. They are widely used but face concerns from both patients and health care professionals, especially regarding their accuracy. A 2015 landmark study substantiated these concerns using case vignettes to demonstrate that symptom checkers commonly err in their triage assessment.

OBJECTIVE

This study aims to revisit the landmark index study to investigate whether and how symptom checkers' capabilities have evolved since 2015 and how they currently compare with laypersons' stand-alone triage appraisal.

METHODS

In early 2020, we searched for smartphone and web-based applications providing triage advice. We evaluated these apps on the same 45 case vignettes as the index study. Using descriptive statistics, we compared our findings with those of the index study and with publicly available data on laypersons' triage capability.

RESULTS

We retrieved 22 symptom checkers providing triage advice. The median triage accuracy in 2020 (55.8%, IQR 15.1%) was close to that in 2015 (59.1%, IQR 15.5%). The apps in 2020 were less risk averse (odds 1.11:1, the ratio of overtriage errors to undertriage errors) than those in 2015 (odds 2.82:1), missing >40% of emergencies. Few apps outperformed laypersons in either deciding whether emergency care was required or whether self-care was sufficient. No apps outperformed the laypersons on both decisions.

CONCLUSIONS

Triage performance of symptom checkers has, on average, not improved over the course of 5 years. It decreased in 2 use cases (advice on when emergency care is required and when no health care is needed for the moment). However, triage capability varies widely within the sample of symptom checkers. Whether it is beneficial to seek advice from symptom checkers depends on the app chosen and on the specific question to be answered. Future research should develop resources (eg, case vignette repositories) to audit the capabilities of symptom checkers continuously and independently and provide guidance on when and to whom they should be recommended.

Health Care Professionals' Experiences of Web-Based Symptom Checkers for Triage: Cross-sectional Survey Study

Background

Web-based symptom checkers are promising tools that provide help to patients seeking guidance on health problems. Many health organizations have started using them to enhance triage. Patients use the symptom checker to report their symptoms online and submit the report to the health care center through the system. Health care professionals (registered nurse, practical nurse, general physician, physiotherapist, etc) receive patient inquiries with urgency rating, decide on actions to be taken, and communicate these to the patients. The success of the adoption, however, depends on whether the tools can efficiently support health care professionals’ workflow and achieve their support.

Objective

This study explores the factors influencing health care professionals’ support for a web-based symptom checker for triage.

Methods

Data were collected through a web-based survey of 639 health care professionals using either of the two most used web-based symptom checkers in the Finnish public primary care. Linear regression models were fitted to study the associations between the study variables and health care professionals’ support for the symptom checkers. In addition, the health care professionals’ comments collected via survey were qualitatively analyzed to elicit additional insights about the benefits and challenges of the clinical use of symptom checkers.

Results

Results show that the perceived beneficial influence of the symptom checkers on health care professionals’ work and the perceived usability of the tools were positively associated with professionals’ support. The perceived benefits to patients and organizational support for use were positively associated, and threat to professionals’ autonomy was negatively associated with health care professionals’ support. These associations were, however, not independent of other factors included in the models. The influences on professionals’ work were both positive and negative; the tools streamlined work by providing preliminary information on patients and reduced the number of phone calls, but they also created extra work as the professionals needed to call patients and ask clarifying questions. Managing time between the use of symptom checkers and other tasks was also challenging. Meanwhile, according to health care professionals’ experience, the symptom checkers benefited patients as they received help quickly with a lower threshold for care.

Conclusions

The efficient use of symptom checkers for triage requires usable solutions that support health care professionals’ work. High-quality information about the patients’ conditions and an efficient way of communicating with patients are needed. Using a new eHealth tool also requires that health organizations and teams reorganize their workflows and work distributions to support clinical processes.

Determinants of Laypersons' Trust in Medical Decision Aids: Randomized Controlled Trial

BACKGROUND

Symptom checker apps are patient-facing decision support systems aimed at providing advice to laypersons on whether, where, and how to seek health care (disposition advice). Such advice can improve laypersons' self-assessment and ultimately improve medical outcomes. Past research has mainly focused on the accuracy of symptom checker apps' suggestions. To support decision-making, such apps need to provide not only accurate but also trustworthy advice. To date, only few studies have addressed the question of the extent to which laypersons trust symptom checker app advice or the factors that moderate their trust. Studies on general decision support systems have shown that framing automated systems (anthropomorphic or emphasizing expertise), for example, by using icons symbolizing artificial intelligence (AI), affects users' trust.

OBJECTIVE

This study aims to identify the factors influencing laypersons' trust in the advice provided by symptom checker apps. Primarily, we investigated whether designs using anthropomorphic framing or framing the app as an AI increases users' trust compared with no such framing.

METHODS

Through a web-based survey, we recruited 494 US residents with no professional medical training. The participants had to first appraise the urgency of a fictitious patient description (case vignette). Subsequently, a decision aid (mock symptom checker app) provided disposition advice contradicting the participants' appraisal, and they had to subsequently reappraise the vignette. Participants were randomized into 3 groups: 2 experimental groups using visual framing (anthropomorphic, 160/494, 32.4%, vs AI, 161/494, 32.6%) and a neutral group without such framing (173/494, 35%).

RESULTS

Most participants (384/494, 77.7%) followed the decision aid's advice, regardless of its urgency level. Neither anthropomorphic framing (odds ratio 1.120, 95% CI 0.664-1.897) nor framing as AI (odds ratio 0.942, 95% CI 0.565-1.570) increased behavioral or subjective trust (P=.99) compared with the no-frame condition. Even participants who were extremely certain in their own decisions (ie, 100% certain) commonly changed it in favor of the symptom checker's advice (19/34, 56%). Propensity to trust and eHealth literacy were associated with increased subjective trust in the symptom checker (propensity to trust b=0.25; eHealth literacy b=0.2), whereas sociodemographic variables showed no such link with either subjective or behavioral trust.

CONCLUSIONS

Contrary to our expectation, neither the anthropomorphic framing nor the emphasis on AI increased trust in symptom checker advice compared with that of a neutral control condition. However, independent of the interface, most participants trusted the mock app's advice, even when they were very certain of their own assessment. Thus, the question arises as to whether laypersons use such symptom checkers as substitutes rather than as aids in their own decision-making. With trust in symptom checkers already high at baseline, the benefit of symptom checkers depends on interface designs that enable users to adequately calibrate their trust levels during usage.

TRIAL REGISTRATION

Deutsches Register Klinischer Studien DRKS00028561; https://tinyurl.com/rv4utcfb (retrospectively registered).

Patient's Perception of Digital Symptom Assessment Technologies in Rheumatology: Results From a Multicentre Study

Introduction

An increasing number of digital tools, including dedicated diagnostic decision support systems (DDSS) exist to better assess new symptoms and understand when and where to seek medical care. The aim of this study was to evaluate patient's previous online assessment experiences and to compare the acceptability, usability, usefulness and potential impact of artificial intelligence (AI)-based symptom checker (Ada) and an online questionnaire-based self-referral tool (Rheport).

Materials and Methods

Patients newly presenting to three German secondary rheumatology outpatient clinics were randomly assigned in a 1:1 ratio to complete consecutively Ada or Rheport in a prospective non-blinded multicentre controlled crossover randomized trial. DDSS completion time was recorded by local study personnel and perceptions on DDSS and previous online assessment were collected through a self-completed study questionnaire, including usability measured with the validated System Usability Scale (SUS).

Results

600 patients (median age 52 years, 418 women) were included. 277/600 (46.2%) of patients used an online search engine prior to the appointment. The median time patients spent assessing symptoms was 180, 7, and 8 min, respectively using online using search engines, Ada and Rheport. 111/275 (40.4%), 266/600 (44.3%) and 395/600 (65.8%) of patients rated the respective symptom assessment as very helpful or helpful, using online search engines, Ada and Rheport, respectively. Usability of both diagnostic decision support systems (DDSS) was “good” with a significantly higher mean SUS score (SD) of Rheport 77.1/100 (16.0) compared to Ada 74.4/100 (16.8), (p < 0.0001). In male patients, usability of Rheport was rated higher than Ada (p = 0.02) and the usability rating of older (52 years ≥) patients of both DDSS was lower than in younger participants (p = 0.005). Both effects were independent of each other. 440/600 (73.3%) and 475/600 (79.2%) of the patients would recommend Ada and Rheport to friends and other patients, respectively.

Conclusion

In summary, patients increasingly assess their symptoms independently online, however only a minority used dedicated symptom assessment websites or DDSS. DDSS, such as Ada an Rheport are easy to use, well accepted among patients with musculoskeletal complaints and could replace online search engines for patient symptom assessment, potentially saving time and increasing helpfulness.

Triage Capability of Laypersons: Retrospective, Exploratory Analysis (Preprint)

Background

Although medical decision-making may be thought of as a task involving health professionals, many decisions, including critical health–related decisions are made by laypersons alone. Specifically, as the first step to most care episodes, it is the patient who determines whether and where to seek health care (triage). Overcautious self-assessments (ie, overtriaging) may lead to overutilization of health care facilities and overcrowded emergency departments, whereas imprudent decisions (ie, undertriaging) constitute a risk to the patient’s health. Recently, patient-facing decision support systems, commonly known as symptom checkers, have been developed to assist laypersons in these decisions.

Objective

The purpose of this study is to identify factors influencing laypersons’ ability to self-triage and their risk averseness in self-triage decisions.

Methods

We analyzed publicly available data on 91 laypersons appraising 45 short fictitious patient descriptions (case vignettes; N=4095 appraisals). Using signal detection theory and descriptive and inferential statistics, we explored whether the type of medical decision laypersons face, their confidence in their decision, and sociodemographic factors influence their triage accuracy and the type of errors they make. We distinguished between 2 decisions: whether emergency care was required (decision 1) and whether self-care was sufficient (decision 2).

Results

The accuracy of detecting emergencies (decision 1) was higher (mean 82.2%, SD 5.9%) than that of deciding whether any type of medical care is required (decision 2, mean 75.9%, SD 5.25%; t_>90=8.4; P<.001; Cohen d=0.9). Sensitivity for decision 1 was lower (mean 67.5%, SD 16.4%) than its specificity (mean 89.6%, SD 8.6%) whereas sensitivity for decision 2 was higher (mean 90.5%, SD 8.3%) than its specificity (mean 46.7%, SD 15.95%). Female participants were more risk averse and overtriaged more often than male participants, but age and level of education showed no association with participants’ risk averseness. Participants’ triage accuracy was higher when they were certain about their appraisal (2114/3381, 62.5%) than when being uncertain (378/714, 52.9%). However, most errors occurred when participants were certain of their decision (1267/1603, 79%). Participants were more commonly certain of their overtriage errors (mean 80.9%, SD 23.8%) than their undertriage errors (mean 72.5%, SD 30.9%; t_>89=3.7; P<.001; d=0.39).

Conclusions

Our study suggests that laypersons are overcautious in deciding whether they require medical care at all, but they miss identifying a considerable portion of emergencies. Our results further indicate that women are more risk averse than men in both types of decisions. Layperson participants made most triage errors when they were certain of their own appraisal. Thus, they might not follow or even seek advice (eg, from symptom checkers) in most instances where advice would be useful.