Beyond Dr. Google: the evidence on consumer-facing digital tools for diagnosis

Contrasting rule and machine learning based digital self triage systems in the USA

Patient smart access and self-triage systems have been in development for decades. As of now, no LLM for processing self-reported patient data has been published by health systems. Many expert systems and computational models have been released to millions. This review is the first to summarize progress in the field including an analysis of the exact self-triage solutions available on the websites of 647 health systems in the USA.

Accuracy of symptom checker for the diagnosis of sexually transmitted infections using machine learning and Bayesian network algorithms

BACKGROUND

A significant proportion of individuals with symptoms of sexually transmitted infection (STI) delay or avoid seeking healthcare, and digital diagnostic tools may prompt them to seek healthcare earlier. Unfortunately, none of the currently available tools fully mimic clinical assessment or cover a wide range of STIs.

METHODS

We prospectively invited attendees presenting with STI-related symptoms at Melbourne Sexual Health Centre to answer gender-specific questionnaires covering the symptoms of 12 common STIs using a computer-assisted self-interviewing system between 2015 and 2018. Then, we developed an online symptom checker (iSpySTI.org) using Bayesian networks. In this study, various machine learning algorithms were trained and evaluated for their ability to predict these STI and anogenital conditions. We used the Z-test to compare their average area under the ROC curve (AUC) scores with the Bayesian networks for diagnostic accuracy.

RESULTS

The study population included 6,162 men (median age 30, IQR: 26-38; approximately 40% of whom had sex with men in the past 12 months) and 4,358 women (median age 27, IQR: 24-31). Non-gonococcal urethritis (NGU) (23.6%, 1447/6121), genital warts (11.7%, 718/6121) and balanitis (8.9%, 546/6121) were the most common conditions in men. Candidiasis (16.6%, 722/4538) and bacterial vaginosis (16.2%, 707/4538) were the most common conditions in women. During evaluation with unseen datasets, machine learning models performed well for most male conditions, with the AUC ranging from 0.81 to 0.95, except for urinary tract infections (UTI) (AUC 0.72). Similarly, the models achieved AUCs ranging from 0.75 to 0.95 for female conditions, except for cervicitis (AUC 0.58). Urethral discharge and other urinary symptoms were important features for predicting urethral gonorrhoea, NGU and UTIs. Similarly, participants selected skin images that were similar to their own lesions, and the location of the anogenital skin lesions were also strong predictors. The vaginal discharge (odour, colour) and itchiness were important predictors for bacterial vaginosis and candidiasis. The performance of the machine learning models was significantly better than Bayesian models for male balanitis, molluscum contagiosum and genital warts (P < 0.05) but was similar for the other conditions.

CONCLUSIONS

Both machine learning and Bayesian models could predict correct diagnoses with reasonable accuracy using prospectively collected data for 12 STIs and other common anogenital conditions. Further work should expand the number of anogenital conditions and seek ways to improve the accuracy, potentially using patient collected images to supplement questionnaire data.

How Useful are Current Chatbots Regarding Urology Patient Information? Comparison of the Ten Most Popular Chatbots' Responses About Female Urinary Incontinence

This research evaluates the readability and quality of patient information material about female urinary incontinence (fUI) in ten popular artificial intelligence (AI) supported chatbots. We used the most recent versions of 10 widely-used chatbots, including OpenAI's GPT-4, Claude-3 Sonnet, Grok 1.5, Mistral Large 2, Google Palm 2, Meta's Llama 3, HuggingChat v0.8.4, Microsoft's Copilot, Gemini Advanced, and Perplexity. Prompts were created to generate texts about UI, stress type UI, urge type UI, and mix type UI. The modified Ensuring Quality Information for Patients (EQIP) technique and QUEST (Quality Evaluating Scoring Tool) were used to assess the quality, and the average of 8 well-known readability formulas, which is Average Reading Level Consensus (ARLC), were used to evaluate readability. When comparing the average scores, there were significant differences in the mean mQEIP and QUEST scores across ten chatbots (p = 0.049 and p = 0.018). Gemini received the greatest mean scores for mEQIP and QUEST, whereas Grok had the lowest values. The chatbots exhibited significant differences in mean ARLC, word count, and sentence count (p = 0.047, p = 0.001, and p = 0.001, respectively). For readability, Grok is the easiest to read, while Mistral is highly complex to understand. AI-supported chatbot technology needs to be improved in terms of readability and quality of patient information regarding female UI.

Evolutionary Trends in the Adoption, Adaptation, and Abandonment of Mobile Health Technologies: Viewpoint Based on 25 Years of Research

Over the past quarter-century, mobile health (mHealth) technologies have experienced significant changes in adoption rates, adaptation strategies, and instances of abandonment. Understanding the underlying factors driving these trends is essential for optimizing the design, implementation, and sustainability of interventions using these technologies. The evolution of mHealth adoption has followed a progressive trajectory, starting with cautious exploration and later accelerating due to technological advancements, increased smartphone penetration, and growing acceptance of digital health solutions by both health care providers and patients. However, alongside widespread adoption, challenges related to usability, interoperability, privacy concerns, and socioeconomic disparities have emerged, necessitating ongoing adaptation efforts. While many mHealth initiatives have successfully adapted to address these challenges, technology abandonment remains common, often due to unsustainable business models, inadequate user engagement, and insufficient evidence of effectiveness. This paper utilizes the Nonadoption, Abandonment, Scale-Up, Spread, and Sustainability (NASSS) framework to examine the interplay between the academic and industry sectors in patterns of adoption, adaptation, and abandonment, using 3 major mHealth innovations as examples: health-related SMS text messaging, mobile apps and wearables, and social media for health communication. Health SMS text messaging has demonstrated significant potential as a tool for health promotion, disease management, and patient engagement. The proliferation of mobile apps and devices has facilitated a shift from in-person and in-clinic practices to mobile- and wearable-centric solutions, encompassing everything from simple activity trackers to advanced health monitoring devices. Social media, initially characterized by basic text-based interactions in chat rooms and online forums, underwent a paradigm shift with the emergence of platforms such as MySpace and Facebook. This transition ushered in an era of mass communication through social media. The rise of microblogging and visually focused platforms such as Twitter(now X), Instagram, Snapchat, and TikTok, along with the integration of live streaming and augmented reality features, exemplifies the ongoing innovation within the social media landscape. Over the past 25 years, there have been remarkable strides in the adoption and adaptation of mHealth technologies, driven by technological innovation and a growing recognition of their potential to revolutionize health care delivery. Each mobile technology uniquely enhances public health and health care by catering to different user needs. SMS text messaging offers wide accessibility and proven effectiveness, while mobile apps and wearables provide comprehensive functionalities for more in-depth health management. Social media platforms amplify these efforts with their vast reach and community-building potential, making it essential to select the right tool for specific health interventions to maximize impact and engagement. Nevertheless, continued efforts are needed to address persistent challenges and mitigate instances of abandonment, ensuring that mHealth interventions reach their full potential in improving health outcomes and advancing equitable access to care.

Evaluation of the accuracy and readability of ChatGPT-4 and Google Gemini in providing information on retinal detachment: a multicenter expert comparative study

BACKGROUND

Large language models (LLMs) such as ChatGPT-4 and Google Gemini show potential for patient health education, but concerns about their accuracy require careful evaluation. This study evaluates the readability and accuracy of ChatGPT-4 and Google Gemini in answering questions about retinal detachment.

METHODS

Comparative study analyzing responses from ChatGPT-4 and Google Gemini to 13 retinal detachment questions, categorized by difficulty levels (D1, D2, D3). Masked responses were reviewed by ten vitreoretinal specialists and rated on correctness, errors, thematic accuracy, coherence, and overall quality grading. Analysis included Flesch Readability Ease Score, word and sentence counts.

RESULTS

Both Artificial Intelligence tools required college-level understanding for all difficulty levels. Google Gemini was easier to understand (p = 0.03), while ChatGPT-4 provided more correct answers for the more difficult questions (p = 0.0005) with fewer serious errors. ChatGPT-4 scored highest on most challenging questions, showing superior thematic accuracy (p = 0.003). ChatGPT-4 outperformed Google Gemini in 8 of 13 questions, with higher overall quality grades in the easiest (p = 0.03) and hardest levels (p = 0.0002), showing a lower grade as question difficulty increased.

CONCLUSIONS

ChatGPT-4 and Google Gemini effectively address queries about retinal detachment, offering mostly accurate answers with few critical errors, though patients require higher education for comprehension. The implementation of AI tools may contribute to improving medical care by providing accurate and relevant healthcare information quickly.

Evaluation of the Current Status of Artificial Intelligence for Endourology Patient Education: A Blind Comparison of ChatGPT and Google Bard Against Traditional Information Resources

Introduction: Artificial intelligence (AI) platforms such as ChatGPT and Bard are increasingly utilized to answer patient health care questions. We present the first study to blindly evaluate AI-generated responses to common endourology patient questions against official patient education materials. Methods: Thirty-two questions and answers spanning kidney stones, ureteral stents, benign prostatic hyperplasia (BPH), and upper tract urothelial carcinoma were extracted from official Urology Care Foundation (UCF) patient education documents. The same questions were input into ChatGPT 4.0 and Bard, limiting responses to within ±10% of the word count of the corresponding UCF response to ensure fair comparison. Six endourologists blindly evaluated responses from each platform using Likert scales for accuracy, clarity, comprehensiveness, and patient utility. Reviewers identified which response they believed was not AI generated. Finally, Flesch-Kincaid Reading Grade Level formulas assessed the readability of each platform response. Ratings were compared using analysis of variance (ANOVA) and chi-square tests. Results: ChatGPT responses were rated the highest across all categories, including accuracy, comprehensiveness, clarity, and patient utility, while UCF answers were consistently scored the lowest, all p < 0.01. A subanalysis revealed that this trend was consistent across question categories (i.e., kidney stones, BPH, etc.). However, AI-generated responses were more likely to be classified at an advanced reading level, while UCF responses showed improved readability (college or higher reading level: ChatGPT = 100%, Bard = 66%, and UCF = 19%), p < 0.001. When asked to identify which answer was not AI generated, 54.2% of responses indicated ChatGPT, 26.6% indicated Bard, and only 19.3% correctly identified it as the UCF response. Conclusions: In a blind evaluation, AI-generated responses from ChatGPT and Bard surpassed the quality of official patient education materials in endourology, suggesting that current AI platforms are already a reliable resource for basic urologic care information. AI-generated responses do, however, tend to require a higher reading level, which may limit their applicability to a broader audience.

Evolution of publicly available large language models for complex decision-making in breast cancer care

PURPOSE

This study investigated the concordance of five different publicly available Large Language Models (LLM) with the recommendations of a multidisciplinary tumor board regarding treatment recommendations for complex breast cancer patient profiles.

METHODS

Five LLM, including three versions of ChatGPT (version 4 and 3.5, with data access until September 3021 and January 2022), Llama2, and Bard were prompted to produce treatment recommendations for 20 complex breast cancer patient profiles. LLM recommendations were compared to the recommendations of a multidisciplinary tumor board (gold standard), including surgical, endocrine and systemic treatment, radiotherapy, and genetic testing therapy options.

RESULTS

GPT4 demonstrated the highest concordance (70.6%) for invasive breast cancer patient profiles, followed by GPT3.5 September 2021 (58.8%), GPT3.5 January 2022 (41.2%), Llama2 (35.3%) and Bard (23.5%). Including precancerous lesions of ductal carcinoma in situ, the identical ranking was reached with lower overall concordance for each LLM (GPT4 60.0%, GPT3.5 September 2021 50.0%, GPT3.5 January 2022 35.0%, Llama2 30.0%, Bard 20.0%). GPT4 achieved full concordance (100%) for radiotherapy. Lowest alignment was reached in recommending genetic testing, demonstrating a varying concordance (55.0% for GPT3.5 January 2022, Llama2 and Bard up to 85.0% for GPT4).

CONCLUSION

This early feasibility study is the first to compare different LLM in breast cancer care with regard to changes in accuracy over time, i.e., with access to more data or through technological upgrades. Methodological advancement, i.e., the optimization of prompting techniques, and technological development, i.e., enabling data input control and secure data processing, are necessary in the preparation of large-scale and multicenter studies to provide evidence on their safe and reliable clinical application. At present, safe and evidenced use of LLM in clinical breast cancer care is not yet feasible.

Evaluation of information accuracy and clarity: ChatGPT responses to the most frequently asked questions about premature ejaculation

Background

Premature ejaculation (PE) is the most prevalent sexual dysfunction in men, and like many diseases and conditions, patients use Internet sources like ChatGPT, which is a popular artificial intelligence-based language model, for queries about this andrological disorder.

Aim

The objective of this research was to evaluate the quality, readability, and understanding of texts produced by ChatGPT in response to frequently requested inquiries on PE.

Methods

In this study we used Google Trends to identify the most frequently searched phrases related to PE. Subsequently, the discovered keywords were methodically entered into ChatGPT, and the resulting replies were assessed for quality using the Ensuring Quality Information for Patients (EQIP) program. The produced texts were assessed for readability using the Flesch-Kincaid Grade Level (FKGL), Flesch Reading Ease Score (FRES), and DISCERN metrics.

Outcomes

This investigation has identified substantial concerns about the quality of texts produced by ChatGPT, highlighting severe problems with reading and understanding.

Results

The mean EQIP score for the texts was determined to be 45.93 ± 4.34, while the FRES was 15.8 ± 8.73. Additionally, the FKGL score was computed to be 15.68 ± 1.67 and the DISCERN score was 38.1 ± 3.78. The comparatively low average EQIP and DISCERN scores suggest that improvements are required to increase the quality and dependability of the presented information. In addition, the FKGL scores indicate a significant degree of linguistic intricacy, requiring a level of knowledge comparable to about 14 to 15 years of formal schooling in order to understand. The texts about treatment, which are the most frequently searched items, are more difficult to understand compared to other texts about other categories.

Clinical Implications

The results of this research suggest that compared to texts on other topics the PE texts produced by ChatGPT exhibit a higher degree of complexity, which exceeds the recommended reading threshold for effective health communication. Currently, ChatGPT is cannot be considered a substitute for comprehensive medical consultations.

Strengths and Limitations

This study is to our knowledge the first reported research investigating the quality and comprehensibility of information generated by ChatGPT in relation to frequently requested queries about PE. The main limitation is that the investigation included only the first 25 popular keywords in English.

Conclusion

ChatGPT is incapable of replacing the need for thorough medical consultations.

Artificial intelligence insights into osteoporosis: assessing ChatGPT's information quality and readability

Accessible, accurate information, and readability play crucial role in empowering individuals managing osteoporosis. This study showed that the responses generated by ChatGPT regarding osteoporosis had serious problems with quality and were at a level of complexity that that necessitates an educational background of approximately 17 years.

PURPOSE

The use of artificial intelligence (AI) applications as a source of information in the field of health is increasing. Readable and accurate information plays a critical role in empowering patients to make decisions about their disease. The aim was to examine the quality and readability of responses provided by ChatGPT, an AI chatbot, to commonly asked questions regarding osteoporosis, representing a major public health problem.

METHODS

"Osteoporosis," "female osteoporosis," and "male osteoporosis" were identified by using Google trends for the 25 most frequently searched keywords on Google. A selected set of 38 keywords was sequentially inputted into the chat interface of the ChatGPT. The responses were evaluated with tools of the Ensuring Quality Information for Patients (EQIP), the Flesch-Kincaid Grade Level (FKGL), and the Flesch-Kincaid Reading Ease (FKRE).

RESULTS

The EQIP score of the texts ranged from a minimum of 36.36 to a maximum of 61.76 with a mean value of 48.71 as having "serious problems with quality." The FKRE scores spanned from 13.71 to 56.06 with a mean value of 28.71 and the FKGL varied between 8.48 and 17.63, with a mean value of 13.25. There were no statistically significant correlations between the EQIP score and the FKGL or FKRE scores.

CONCLUSIONS

Although ChatGPT is easily accessible for patients to obtain information about osteoporosis, its current quality and readability fall short of meeting comprehensive healthcare standards.

Information Quality and Readability: ChatGPT's Responses to the Most Common Questions About Spinal Cord Injury

OBJECTIVE

This study aimed to assess the quality, readability, and comprehension of texts generated by ChatGPT in response to commonly asked questions about spinal cord injury (SCI).

METHODS

The study utilized Google Trends to identify the most frequently searched keywords related to SCI. The identified keywords were sequentially inputted into ChatGPT, and the resulting responses were assessed for quality using the Ensuring Quality Information for Patients (EQIP) tool. The readability of the texts was analyzed using the Flesch-Kincaid grade level and the Flesch-Kincaid reading ease parameters.

RESULTS

The mean EQIP score of the texts was determined to be 43.02 ± 6.37, the Flesch-Kincaid reading ease score to be 26.24 ± 13.81, and the Flesch-Kincaid grade level was determined to be 14.84 ± 1.79. The analysis revealed significant concerns regarding the quality of texts generated by ChatGPT, indicating serious problems with readability and comprehension. The mean EQIP score was low, suggesting a need for improvement in the accuracy and reliability of the information provided. The Flesch-Kincaid grade level indicated a high linguistic complexity, requiring a level of education equivalent to approximately 14 to 15 years of formal education for comprehension.

CONCLUSIONS

The results of this study show heightened complexity in ChatGPT-generated SCI texts, surpassing optimal health communication readability. ChatGPT currently cannot substitute comprehensive medical consultations. Enhancing text quality could be attainable through dependence on credible sources, the establishment of a scientific board, and collaboration with expert teams. Addressing these concerns could improve text accessibility, empowering patients and facilitating informed decision-making in SCI.

Feasibility of using an artificially intelligent chatbot to increase access to information and sexual and reproductive health services

Background

Following the US Supreme Court decision overturning Roe v. Wade, there is evidence of limitations in access to safe abortion care. Artificially intelligent (AI)-enabled conversational chatbots are becoming an appealing option to support access to care, but generative AI systems can misinform and hallucinate and risk reinforcing problematic bias and stigma related to sexual and reproductive healthcare.

Method

A single arm pilot study describing the development of an AI chatbot focused on sexual and reproductive health and its deployment in a clinic setting and community-based organization over a nine-month period.

Results

We adjusted chatbot content based on feedback from the medical director and clients of organizations where the system was deployed given updated medical guidelines and preferred language related to gender-affirming care. We deployed the system in two organizations and tracked use over nine months. In that time, there were 1749 queries from 425 unique users. One-tenth of users of the clinic based chatbot went on to schedule an appointment for care.

Conclusions

Ongoing challenges in accessing sexual and reproductive health suggest having diverse mechanisms to facilitate access to accurate and updated medical information is warranted. Using an AI chatbot is feasible to accomplish this goal and shows promise in increasing opportunities to access care.

Assessment of a Digital Symptom Checker Tool's Accuracy in Suggesting Reproductive Health Conditions: Clinical Vignettes Study

BACKGROUND

Reproductive health conditions such as endometriosis, uterine fibroids, and polycystic ovary syndrome (PCOS) affect a large proportion of women and people who menstruate worldwide. Prevalence estimates for these conditions range from 5% to 40% of women of reproductive age. Long diagnostic delays, up to 12 years, are common and contribute to health complications and increased health care costs. Symptom checker apps provide users with information and tools to better understand their symptoms and thus have the potential to reduce the time to diagnosis for reproductive health conditions.

OBJECTIVE

This study aimed to evaluate the agreement between clinicians and 3 symptom checkers (developed by Flo Health UK Limited) in assessing symptoms of endometriosis, uterine fibroids, and PCOS using vignettes. We also aimed to present a robust example of vignette case creation, review, and classification in the context of predeployment testing and validation of digital health symptom checker tools.

METHODS

Independent general practitioners were recruited to create clinical case vignettes of simulated users for the purpose of testing each condition symptom checker; vignettes created for each condition contained a mixture of condition-positive and condition-negative outcomes. A second panel of general practitioners then reviewed, approved, and modified (if necessary) each vignette. A third group of general practitioners reviewed each vignette case and designated a final classification. Vignettes were then entered into the symptom checkers by a fourth, different group of general practitioners. The outcomes of each symptom checker were then compared with the final classification of each vignette to produce accuracy metrics including percent agreement, sensitivity, specificity, positive predictive value, and negative predictive value.

RESULTS

A total of 24 cases were created per condition. Overall, exact matches between the vignette general practitioner classification and the symptom checker outcome were 83% (n=20) for endometriosis, 83% (n=20) for uterine fibroids, and 88% (n=21) for PCOS. For each symptom checker, sensitivity was reported as 81.8% for endometriosis, 84.6% for uterine fibroids, and 100% for PCOS; specificity was reported as 84.6% for endometriosis, 81.8% for uterine fibroids, and 75% for PCOS; positive predictive value was reported as 81.8% for endometriosis, 84.6% for uterine fibroids, 80% for PCOS; and negative predictive value was reported as 84.6% for endometriosis, 81.8% for uterine fibroids, and 100% for PCOS.

CONCLUSIONS

The single-condition symptom checkers have high levels of agreement with general practitioner classification for endometriosis, uterine fibroids, and PCOS. Given long delays in diagnosis for many reproductive health conditions, which lead to increased medical costs and potential health complications for individuals and health care providers, innovative health apps and symptom checkers hold the potential to improve care pathways.

ChatGPT and most frequent urological diseases: analysing the quality of information and potential risks for patients

PURPOSE

Artificial intelligence (AI) is a set of systems or combinations of algorithms, which mimic human intelligence. ChatGPT is software with artificial intelligence which was recently developed by OpenAI. One of its potential uses could be to consult the information about pathologies and treatments. Our objective was to assess the quality of the information provided by AI like ChatGPT and establish if it is a secure source of information for patients.

METHODS

Questions about bladder cancer, prostate cancer, renal cancer, benign prostatic hypertrophy (BPH), and urinary stones were queried through ChatGPT 4.0. Two urologists analysed the responses provided by ChatGPT using DISCERN questionary and a brief instrument for evaluating the quality of informed consent documents.

RESULTS

The overall information provided in all pathologies was well-balanced. In each pathology was explained its anatomical location, affected population and a description of the symptoms. It concluded with the established risk factors and possible treatment. All treatment answers had a moderate quality score with DISCERN (3 of 5 points). The answers about surgical options contain the recovery time, type of anaesthesia, and potential complications. After analysing all the responses related to each disease, all pathologies except BPH achieved a DISCERN score of 4.

CONCLUSIONS

ChatGPT information should be used with caution since the chatbot does not disclose the sources of information and may contain bias even with simple questions related to the basics of urologic diseases.

Comparison of Diagnostic and Triage Accuracy of Ada Health and WebMD Symptom Checkers, ChatGPT, and Physicians for Patients in an Emergency Department: Clinical Data Analysis Study

BACKGROUND

Diagnosis is a core component of effective health care, but misdiagnosis is common and can put patients at risk. Diagnostic decision support systems can play a role in improving diagnosis by physicians and other health care workers. Symptom checkers (SCs) have been designed to improve diagnosis and triage (ie, which level of care to seek) by patients.

OBJECTIVE

The aim of this study was to evaluate the performance of the new large language model ChatGPT (versions 3.5 and 4.0), the widely used WebMD SC, and an SC developed by Ada Health in the diagnosis and triage of patients with urgent or emergent clinical problems compared with the final emergency department (ED) diagnoses and physician reviews.

METHODS

We used previously collected, deidentified, self-report data from 40 patients presenting to an ED for care who used the Ada SC to record their symptoms prior to seeing the ED physician. Deidentified data were entered into ChatGPT versions 3.5 and 4.0 and WebMD by a research assistant blinded to diagnoses and triage. Diagnoses from all 4 systems were compared with the previously abstracted final diagnoses in the ED as well as with diagnoses and triage recommendations from three independent board-certified ED physicians who had blindly reviewed the self-report clinical data from Ada. Diagnostic accuracy was calculated as the proportion of the diagnoses from ChatGPT, Ada SC, WebMD SC, and the independent physicians that matched at least one ED diagnosis (stratified as top 1 or top 3). Triage accuracy was calculated as the number of recommendations from ChatGPT, WebMD, or Ada that agreed with at least 2 of the independent physicians or were rated "unsafe" or "too cautious."

RESULTS

Overall, 30 and 37 cases had sufficient data for diagnostic and triage analysis, respectively. The rate of top-1 diagnosis matches for Ada, ChatGPT 3.5, ChatGPT 4.0, and WebMD was 9 (30%), 12 (40%), 10 (33%), and 12 (40%), respectively, with a mean rate of 47% for the physicians. The rate of top-3 diagnostic matches for Ada, ChatGPT 3.5, ChatGPT 4.0, and WebMD was 19 (63%), 19 (63%), 15 (50%), and 17 (57%), respectively, with a mean rate of 69% for physicians. The distribution of triage results for Ada was 62% (n=23) agree, 14% unsafe (n=5), and 24% (n=9) too cautious; that for ChatGPT 3.5 was 59% (n=22) agree, 41% (n=15) unsafe, and 0% (n=0) too cautious; that for ChatGPT 4.0 was 76% (n=28) agree, 22% (n=8) unsafe, and 3% (n=1) too cautious; and that for WebMD was 70% (n=26) agree, 19% (n=7) unsafe, and 11% (n=4) too cautious. The unsafe triage rate for ChatGPT 3.5 (41%) was significantly higher (P=.009) than that of Ada (14%).

CONCLUSIONS

ChatGPT 3.5 had high diagnostic accuracy but a high unsafe triage rate. ChatGPT 4.0 had the poorest diagnostic accuracy, but a lower unsafe triage rate and the highest triage agreement with the physicians. The Ada and WebMD SCs performed better overall than ChatGPT. Unsupervised patient use of ChatGPT for diagnosis and triage is not recommended without improvements to triage accuracy and extensive clinical evaluation.

The promise of digital healthcare technologies

Digital health technologies have been in use for many years in a wide spectrum of healthcare scenarios. This narrative review outlines the current use and the future strategies and significance of digital health technologies in modern healthcare applications. It covers the current state of the scientific field (delineating major strengths, limitations, and applications) and envisions the future impact of relevant emerging key technologies. Furthermore, we attempt to provide recommendations for innovative approaches that would accelerate and benefit the research, translation and utilization of digital health technologies.

Triage and Diagnostic Accuracy of Online Symptom Checkers: Systematic Review

BACKGROUND

In the context of a deepening global shortage of health workers and, in particular, the COVID-19 pandemic, there is growing international interest in, and use of, online symptom checkers (OSCs). However, the evidence surrounding the triage and diagnostic accuracy of these tools remains inconclusive.

OBJECTIVE

This systematic review aimed to summarize the existing peer-reviewed literature evaluating the triage accuracy (directing users to appropriate services based on their presenting symptoms) and diagnostic accuracy of OSCs aimed at lay users for general health concerns.

METHODS

Searches were conducted in MEDLINE, Embase, CINAHL, Health Management Information Consortium (HMIC), and Web of Science, as well as the citations of the studies selected for full-text screening. We included peer-reviewed studies published in English between January 1, 2010, and February 16, 2022, with a controlled and quantitative assessment of either or both triage and diagnostic accuracy of OSCs directed at lay users. We excluded tools supporting health care professionals, as well as disease- or specialty-specific OSCs. Screening and data extraction were carried out independently by 2 reviewers for each study. We performed a descriptive narrative synthesis.

RESULTS

A total of 21,296 studies were identified, of which 14 (0.07%) were included. The included studies used clinical vignettes, medical records, or direct input by patients. Of the 14 studies, 6 (43%) reported on triage and diagnostic accuracy, 7 (50%) focused on triage accuracy, and 1 (7%) focused on diagnostic accuracy. These outcomes were assessed based on the diagnostic and triage recommendations attached to the vignette in the case of vignette studies or on those provided by nurses or general practitioners, including through face-to-face and telephone consultations. Both diagnostic accuracy and triage accuracy varied greatly among OSCs. Overall diagnostic accuracy was deemed to be low and was almost always lower than that of the comparator. Similarly, most of the studies (9/13, 69 %) showed suboptimal triage accuracy overall, with a few exceptions (4/13, 31%). The main variables affecting the levels of diagnostic and triage accuracy were the severity and urgency of the condition, the use of artificial intelligence algorithms, and demographic questions. However, the impact of each variable differed across tools and studies, making it difficult to draw any solid conclusions. All included studies had at least one area with unclear risk of bias according to the revised Quality Assessment of Diagnostic Accuracy Studies-2 tool.

CONCLUSIONS

Although OSCs have potential to provide accessible and accurate health advice and triage recommendations to users, more research is needed to validate their triage and diagnostic accuracy before widescale adoption in community and health care settings. Future studies should aim to use a common methodology and agreed standard for evaluation to facilitate objective benchmarking and validation.

TRIAL REGISTRATION

PROSPERO CRD42020215210; https://tinyurl.com/3949zw83.

Impacts of Symptom Checkers for Laypersons' Self-diagnosis on Physicians in Primary Care: Scoping Review

BACKGROUND

Symptom checkers (SCs) for laypersons' self-assessment and preliminary self-diagnosis are widely used by the public. Little is known about the impact of these tools on health care professionals (HCPs) in primary care and their work. This is relevant to understanding how technological changes might affect the working world and how this is linked to work-related psychosocial demands and resources for HCPs.

OBJECTIVE

This scoping review aimed to systematically explore the existing publications on the impacts of SCs on HCPs in primary care and to identify knowledge gaps.

METHODS

We used the Arksey and O'Malley framework. We based our search string on the participant, concept, and context scheme and searched PubMed (MEDLINE) and CINAHL in January and June 2021. We performed a reference search in August 2021 and a manual search in November 2021. We included publications of peer-reviewed journals that focused on artificial intelligence- or algorithm-based self-diagnosing apps and tools for laypersons and had primary care or nonclinical settings as a relevant context. The characteristics of these studies were described numerically. We used thematic analysis to identify core themes. We followed the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) checklist to report the study.

RESULTS

Of the 2729 publications identified through initial and follow-up database searches, 43 full texts were screened for eligibility, of which 9 were included. Further 8 publications were included through manual search. Two publications were excluded after receiving feedback in the peer-review process. Fifteen publications were included in the final sample, which comprised 5 (33%) commentaries or nonresearch publications, 3 (20%) literature reviews, and 7 (47%) research publications. The earliest publications stemmed from 2015. We identified 5 themes. The theme finding prediagnosis comprised the comparison between SCs and physicians. We identified the performance of the diagnosis and the relevance of human factors as topics. In the theme layperson-technology relationship, we identified potentials for laypersons' empowerment and harm through SCs. Our analysis showed potential disruptions of the physician-patient relationship and uncontested roles of HCPs in the theme (impacts on) physician-patient relationship. In the theme impacts on HCPs' tasks, we described the reduction or increase in HCPs' workload. We identified potential transformations of HCPs' work and impacts on the health care system in the theme future role of SCs in health care.

CONCLUSIONS

The scoping review approach was suitable for this new field of research. The heterogeneity of technologies and wordings was challenging. We identified research gaps in the literature regarding the impact of artificial intelligence- or algorithm-based self-diagnosing apps or tools on the work of HCPs in primary care. Further empirical studies on HCPs' lived experiences are needed, as the current literature depicts expectations rather than empirical findings.

More than just a stye: identifying seasonal patterns using google trends, and a review of infodemiological literature in ophthalmology

PURPOSE

We aim to evaluate the utility of internet search query data in ophthalmology by: (1) Evaluating trends in searches for styes in the United States and worldwide, and (2) Performing a review of literature of infodemiological data in ophthalmology.

METHODS

Google Trends search data for "stye" was analyzed from January 2004 to January 2020 in the United States and worldwide. Spearman's correlation coefficient and sinusoidal modeling were performed to assess the significance and seasonality of trends. Review of literature included searches for "ophthalmology Google trends," "ophthalmology twitter trends," "ophthalmology infodemiology," "eye google trends," and "social media ophthalmology."

RESULTS

Searches for styes were cyclical in the United States and globally with a steady increase from 2004 to 2020 (sum-of-squares F-test for sinusoidal model: p < .0001, r² = 0.96). Peak search volume index (SVI) months were 7.9 months in the United States and 6.8 months worldwide. U.S. temperature and SVI for stye were correlated in the United States at the state, divisional, and country-wide levels (p < .005; p < .005; p < .01 respectively). Seven articles met our literature review inclusion criteria.

CONCLUSIONS

We present a novel finding of seasonality with global and U.S. searches for stye, and association of searches with temperature in the United States. Within ophthalmology, infodemiological literature has been used to track trends and identify seasonal disease patterns, perform disease surveillance, improve resource optimization by identifying regional hotspots, tailor marketing, and monitor institutional reputation. Future research into this domain may help identify further trends, improve prevention efforts, and reduce medical costs.

www.coronabambini.ch: Development and usage of an online decision support tool for paediatric COVID-19-testing in Switzerland: a cross-sectional analysis

OBJECTIVES

To describe the development and usage of www.coronabambini.ch as an example of a paediatric electronic public health application and to explore its potential and limitations in providing information on disease epidemiology and public health policy implementation.

DESIGN

We developed and maintained a non-commercial online decision support tool, www.coronabambini.ch, to translate the Swiss Federal Office of Public Health (FOPH) paediatric (age 0-18 years) COVID-19 guidelines around testing and school/daycare attendance for caregivers, teachers and healthcare personnel. We analysed the online decision tool as well as a voluntary follow-up survey from October 2020 to September 2021 to explore its potential as a surveillance tool for public health policy and epidemiology.

PARTICIPANTS

68 269 users accessed and 52 726 filled out the complete online decision tool. 3% (1399/52 726) filled out a voluntary follow-up. 92% (18 797/20 330) of users were parents.

RESULTS

Certain dynamics of the pandemic and changes in testing strategies were reflected in the data captured by www.coronabambini.ch, for example, in terms of disease epidemiology, gastrointestinal symptoms were reported more frequently in younger age groups (13% (3308/26 180) in children 0-5 years vs 9% (3934/42 089) in children ≥6 years, χ²=184, p≤0.001). As a reflection of public health policy, the proportion of users consulting the tool for a positive contact without symptoms in children 6-12 years increased from 4% (1415/32 215) to 6% (636/9872) after the FOPH loosened testing criteria in this age group, χ²=69, p≤0.001. Adherence to the recommendation was generally high (84% (1131/1352)) but differed by the type of recommendation: 89% (344/385) for 'stay at home and observe', 75% (232/310) for 'school attendance'.

CONCLUSIONS

Usage of www.coronabambini.ch was generally high in areas where it was developed and promoted. Certain patterns in epidemiology and adherence to public health policy could be depicted but selection bias was difficult to measure showing the potential and challenges of digital decision support as public health tools.

Empowerment through health self-testing apps? Revisiting empowerment as a process

Empowerment, an already central concept in public health, has gained additional relevance through the expansion of mobile health (mHealth). Especially direct-to-consumer self-testing app companies mobilise the term to advertise their products, which allow users to self-test for various medical conditions independent of healthcare professionals. This article first demonstrates the absence of empowerment conceptualisations in the context of self-testing apps by engaging with empowerment literature. It then contrasts the service these apps provide with two widely cited empowerment definitions by the WHO, which describe the term as a process that, broadly, leads to knowledge and control of health decisions. We conclude that self-testing apps can only partly empower their users, as they, we argue, do not provide the type of knowledge and control the WHO definitions describe. More importantly, we observe that this shortcoming stems from the fact that in the literature on mHealth and in self-testing marketing, empowerment is understood as a goal rather than a process. This characterises a shift in the meaning of empowerment in the context of self-testing and mHealth, one that reveals a lack of awareness for relational and contextual factors that contribute to empowerment. We argue that returning to a process-understanding of empowerment helps to identify these apps' deficits, and we conclude the article by briefly suggesting several strategies to increase self-testing apps' empowerment function.

Big data in corneal diseases and cataract: Current applications and future directions

The accelerated growth in electronic health records (EHR), Internet-of-Things, mHealth, telemedicine, and artificial intelligence (AI) in the recent years have significantly fuelled the interest and development in big data research. Big data refer to complex datasets that are characterized by the attributes of "5 Vs"-variety, volume, velocity, veracity, and value. Big data analytics research has so far benefitted many fields of medicine, including ophthalmology. The availability of these big data not only allow for comprehensive and timely examinations of the epidemiology, trends, characteristics, outcomes, and prognostic factors of many diseases, but also enable the development of highly accurate AI algorithms in diagnosing a wide range of medical diseases as well as discovering new patterns or associations of diseases that are previously unknown to clinicians and researchers. Within the field of ophthalmology, there is a rapidly expanding pool of large clinical registries, epidemiological studies, omics studies, and biobanks through which big data can be accessed. National corneal transplant registries, genome-wide association studies, national cataract databases, and large ophthalmology-related EHR-based registries (e.g., AAO IRIS Registry) are some of the key resources. In this review, we aim to provide a succinct overview of the availability and clinical applicability of big data in ophthalmology, particularly from the perspective of corneal diseases and cataract, the synergistic potential of big data, AI technologies, internet of things, mHealth, and wearable smart devices, and the potential barriers for realizing the clinical and research potential of big data in this field.

A scoping review on the use and usefulness of online symptom checkers and triage systems: How to proceed?

Background

Patients are increasingly turning to the Internet for health information. Numerous online symptom checkers and digital triage tools are currently available to the general public in an effort to meet this need, simultaneously acting as a demand management strategy to aid the overburdened health care system. The implementation of these services requires an evidence-based approach, warranting a review of the available literature on this rapidly evolving topic.

Objective

This scoping review aims to provide an overview of the current state of the art and identify research gaps through an analysis of the strengths and weaknesses of the presently available literature.

Methods

A systematic search strategy was formed and applied to six databases: Cochrane library, NICE, DARE, NIHR, Pubmed, and Web of Science. Data extraction was performed by two researchers according to a pre-established data charting methodology allowing for a thematic analysis of the results.

Results

A total of 10,250 articles were identified, and 28 publications were found eligible for inclusion. Users of these tools are often younger, female, more highly educated and technologically literate, potentially impacting digital divide and health equity. Triage algorithms remain risk-averse, which causes challenges for their accuracy. Recent evolutions in algorithms have varying degrees of success. Results on impact are highly variable, with potential effects on demand, accessibility of care, health literacy and syndromic surveillance. Both patients and healthcare providers are generally positive about the technology and seem amenable to the advice given, but there are still improvements to be made toward a more patient-centered approach. The significant heterogeneity across studies and triage systems remains the primary challenge for the field, limiting transferability of findings.

Conclusion

Current evidence included in this review is characterized by significant variability in study design and outcomes, highlighting the significant challenges for future research.An evolution toward more homogeneous methodologies, studies tailored to the intended setting, regulation and standardization of evaluations, and a patient-centered approach could benefit the field.

Predictors of sustained use of mobile health applications: Content analysis of user perspectives from a fever management app

Objectives

Mobile health applications could be means of educating and changing behaviours of their users. Their features and qualities determine the sustainability of use. The FeverApp with two main features of information and documentation is a research-based app. In this observational cohort study, to evaluate the influential predictors of use, users' feedback on the FeverApp, were analyzed.

Methods

Feedback is given using a structured questionnaire, four Likert items and two open questions regarding positive and negative impressions, available via app menu. Conventional content analysis (inductive approach) on the two open questions was performed. Comments were grouped into 12 codes. These codes were grouped hierarchically in an iterative process into nine subcategories and lastly into two main categories 'format' and 'content'. Descriptive and quantitative analysis were performed.

Results

Out of 8243 users, 1804 of them answered the feedback questionnaire. The features of the app (N  =  344), followed by the information aspect (N  =  330) were most frequently mentioned. Documentation process (N  =  226), request for new features or improvement of the current ones (N  =  193), and functioning (N  =  132) were also highlighted in users' feedback. App's ease of use, design and being informative were important for the users. The first impression of the app seems important as the majority of feedback were given during the first month of using the app.

Conclusion

In-app feedback function could highlight shortcomings and strengths of mobile health apps. Taking users' feedback into consideration could increase the chance of sustained use. Besides ease of use and clear, likeable designs, users want apps that serve their needs while saving time.

'I understood the texting process well'. Participant perspectives on usability and acceptability of SMS-based telehealth follow-up after voluntary medical male circumcision in South Africa

Background

Voluntary medical male circumcision (MC) is a biomedical HIV prevention method that requires post-operative follow-up for healing confirmation. Recent research found that a two-way texting (2wT) app providing SMS-based telehealth for MC patients was safe and reduced provider workload. We evaluated 2wT usability among MC clients in South Africa assigned the 2wT intervention within a larger randomized controlled trial (RCT) of 2wT safety and workload.

Methods

This quantitative usability study is within an RCT where 547 men used 2wT to interact with an MC provider via SMS. The sub-study involved the first 100 men assigned to 2wT who completed a usability survey 14 days after surgery. Acceptability was assessed through 2wT response rates of the 547 men. Regression models analyzed associations between age, wage, location, potential adverse events (AEs), and 2wT responses.

Results

Men assigned to 2wT found it safe, comfortable, and convenient, reporting time and cost savings. High response rates (88%) to daily messages indicated acceptability. Age, wage, and location didn't affect text responses or potential AEs.

Conclusion

2wT for post-MC follow-up was highly usable and acceptable, suggesting its viability as an alternative to in-person visits. It enhanced confidence in wound self-management. This SMS-based telehealth can enhance MC care quality and be adapted to similar contexts for independent healing support, particularly for men.

Ethical, legal, and social aspects of symptom checker applications: a scoping review

Symptom Checker Applications (SCA) are mobile applications often designed for the end-user to assist with symptom assessment and self-triage. SCA are meant to provide the user with easily accessible information about their own health conditions. However, SCA raise questions regarding ethical, legal, and social aspects (ELSA), for example, regarding fair access to this new technology. The aim of this scoping review is to identify the ELSA of SCA in the scientific literature. A scoping review was conducted to identify the ELSA of SCA. Ten databases (e.g., Web of Science and PubMed) were used. Studies on SCA that address ELSA, written in English or German, were included in the review. The ELSA of SCA were extracted and synthesized using qualitative content analysis. A total of 25,061 references were identified, of which 39 were included in the analysis. The identified aspects were allotted to three main categories: (1) Technology; (2) Individual Level; and (3) Healthcare system. The results show that there are controversial debates in the literature on the ethical and social challenges of SCA usage. Furthermore, the debates are characterised by a lack of a specific legal perspective and empirical data. The review provides an overview on the spectrum of ELSA regarding SCA. It offers guidance to stakeholders in the healthcare system, for example, patients, healthcare professionals, and insurance providers and could be used in future empirical research to investigate the perspectives of those affected, such as users.

WtsWrng Interim Comparative Effectiveness Evaluation and Description of the Challenges to Develop, Assess, and Introduce This Novel Digital Application in a Traditional Health System

Science and technology have evolved quickly during the two decades of the 21st century, but healthcare systems are grounded in last century's structure and processes. Changes in the way health care is provided are demanded; digital transformation is a key driver making healthcare systems more accessible, agile, efficient, and citizen-centered. Nevertheless, the way healthcare systems function challenges the development (Innovation + Development and regulatory requirements), assessment (methodological guidance weaknesses), and adoption of digital applications (DAs). WtsWrng (WW), an innovative DA which uses images to interact with citizens for symptom triage and monitoring, is used as an example to show the challenges faced in its development and clinical validation and how these are being overcome. To prove WW's value from inception, novel approaches for evidence generation that allows for an agile and patient-centered development have been applied. Early scientific advice from NICE (UK) was sought for study design, an iterative development and interim analysis was performed, and different statistical parameters (Kappa, B statistic) were explored to face development and assessment challenges. WW triage accuracy at cutoff time ranged from 0.62 to 0.94 for the most frequent symptoms attending the Emergency Department (ED), with the observed concordance for the 12 most frequent diagnostics at hospital discharge fluctuating between 0.4 to 0.97; 8 of the diagnostics had a concordance greater than 0.8. This experience should provoke reflective thinking for DA developers, digital health scientists, regulators, health technology assessors, and payers.

A Novel Bayesian General Medical Diagnostic Assistant Achieves Superior Accuracy With Sparse History

Online AI symptom checkers and diagnostic assistants (DAs) have tremendous potential to reduce misdiagnosis and cost, while increasing the quality, convenience, and availability of healthcare, but only if they can perform with high accuracy. We introduce a novel Bayesian DA designed to improve diagnostic accuracy by addressing key weaknesses of Bayesian Network implementations for clinical diagnosis. We compare the performance of our prototype DA (MidasMed) to that of physicians and six other publicly accessible DAs (Ada, Babylon, Buoy, Isabel, Symptomate, and WebMD) using a set of 30 publicly available case vignettes, and using only sparse history (no exam findings or tests). Our results demonstrate superior performance of the MidasMed DA, with the correct diagnosis being the top ranked disorder in 93% of cases, and in the top 3 in 96% of cases.

Effects and utility of an online forward triage tool during the SARS-CoV-2 pandemic: a mixed method study and patient perspectives, Switzerland

OBJECTIVE

To assess the effects (quantitatively) and the utility (qualitatively) of a COVID-19 online forward triage tool (OFTT) in a pandemic context.

DESIGN

A mixed method sequential explanatory study was employed. Quantitative data of all OFTT users, between 2 March 2020 and 12 May 2020, were collected. Second, qualitative data were collected through key informant interviews (n=19) to explain the quantitative findings, explore tool utility, user experience and elicit recommendations.

SETTING

The working group e-emergency medicine at the emergency department developed an OFTT, which was made available online.

PARTICIPANTS

Participants included all users above the age of 18 that used the OFTT between 2 March 2020 and 12 May 2020.

INTERVENTION

An OFTT that displayed the current test recommendations of the Federal Office of Public Health on whether someone needed testing for COVID-19 or not. No diagnosis was provided.

RESULTS

In the study period, 6272 users consulted our OFTT; 40.2% (1626/4049) would have contacted a healthcare provider had the tool not existed. 560 participants consented to a follow-up survey and provided a valid email address. 31.4% (176/560) participants returned a complete follow-up questionnaire. 84.7% (149/176) followed the recommendations given. 41.5% (73/176) reported that their fear was allayed after using the tool. Qualitatively, seven overarching themes emerged namely (1) accessibility of tool, (2) user-friendliness of tool, (3) utility of tool as an information source, (4) utility of tool in allaying fear and anxiety, (5) utility of tool in medical decision-making (6) utility of tool in reducing the potential for onward transmissions and (7) utility of tool in reducing health system burden.

CONCLUSION

Our findings demonstrated that a COVID-19 OFTT does not only reduce the health system burden but can also serve as an information source, reduce anxiety and fear, reduce potential for cross infections and facilitate medical decision-making.

Determinants of Laypersons' Trust in Medical Decision Aids: Randomized Controlled Trial

BACKGROUND

Symptom checker apps are patient-facing decision support systems aimed at providing advice to laypersons on whether, where, and how to seek health care (disposition advice). Such advice can improve laypersons' self-assessment and ultimately improve medical outcomes. Past research has mainly focused on the accuracy of symptom checker apps' suggestions. To support decision-making, such apps need to provide not only accurate but also trustworthy advice. To date, only few studies have addressed the question of the extent to which laypersons trust symptom checker app advice or the factors that moderate their trust. Studies on general decision support systems have shown that framing automated systems (anthropomorphic or emphasizing expertise), for example, by using icons symbolizing artificial intelligence (AI), affects users' trust.

OBJECTIVE

This study aims to identify the factors influencing laypersons' trust in the advice provided by symptom checker apps. Primarily, we investigated whether designs using anthropomorphic framing or framing the app as an AI increases users' trust compared with no such framing.

METHODS

Through a web-based survey, we recruited 494 US residents with no professional medical training. The participants had to first appraise the urgency of a fictitious patient description (case vignette). Subsequently, a decision aid (mock symptom checker app) provided disposition advice contradicting the participants' appraisal, and they had to subsequently reappraise the vignette. Participants were randomized into 3 groups: 2 experimental groups using visual framing (anthropomorphic, 160/494, 32.4%, vs AI, 161/494, 32.6%) and a neutral group without such framing (173/494, 35%).

RESULTS

Most participants (384/494, 77.7%) followed the decision aid's advice, regardless of its urgency level. Neither anthropomorphic framing (odds ratio 1.120, 95% CI 0.664-1.897) nor framing as AI (odds ratio 0.942, 95% CI 0.565-1.570) increased behavioral or subjective trust (P=.99) compared with the no-frame condition. Even participants who were extremely certain in their own decisions (ie, 100% certain) commonly changed it in favor of the symptom checker's advice (19/34, 56%). Propensity to trust and eHealth literacy were associated with increased subjective trust in the symptom checker (propensity to trust b=0.25; eHealth literacy b=0.2), whereas sociodemographic variables showed no such link with either subjective or behavioral trust.

CONCLUSIONS

Contrary to our expectation, neither the anthropomorphic framing nor the emphasis on AI increased trust in symptom checker advice compared with that of a neutral control condition. However, independent of the interface, most participants trusted the mock app's advice, even when they were very certain of their own assessment. Thus, the question arises as to whether laypersons use such symptom checkers as substitutes rather than as aids in their own decision-making. With trust in symptom checkers already high at baseline, the benefit of symptom checkers depends on interface designs that enable users to adequately calibrate their trust levels during usage.

TRIAL REGISTRATION

Deutsches Register Klinischer Studien DRKS00028561; https://tinyurl.com/rv4utcfb (retrospectively registered).

Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients

BACKGROUND

Digital technologies have become a common starting point for health-related information-seeking. Web- or app-based symptom checkers aim to provide rapid and accurate condition suggestions and triage advice but have not yet been investigated for mental disorders in routine health care settings.

OBJECTIVE

This study aims to test the diagnostic performance of a widely available symptom checker in the context of formal diagnosis of mental disorders when compared with therapists' diagnoses based on structured clinical interviews.

METHODS

Adult patients from an outpatient psychotherapy clinic used the app-based symptom checker Ada-check your health (ADA; Ada Health GmbH) at intake. Accuracy was assessed as the agreement of the first and 1 of the first 5 condition suggestions of ADA with at least one of the interview-based therapist diagnoses. In addition, sensitivity, specificity, and interrater reliabilities (Gwet first-order agreement coefficient [AC1]) were calculated for the 3 most prevalent disorder categories. Self-reported usability (assessed using the System Usability Scale) and acceptance of ADA (assessed using an adapted feedback questionnaire) were evaluated.

RESULTS

A total of 49 patients (30/49, 61% women; mean age 33.41, SD 12.79 years) were included in this study. Across all patients, the interview-based diagnoses matched ADA's first condition suggestion in 51% (25/49; 95% CI 37.5-64.4) of cases and 1 of the first 5 condition suggestions in 69% (34/49; 95% CI 55.4-80.6) of cases. Within the main disorder categories, the accuracy of ADA's first condition suggestion was 0.82 for somatoform and associated disorders, 0.65 for affective disorders, and 0.53 for anxiety disorders. Interrater reliabilities ranged from low (AC1=0.15 for anxiety disorders) to good (AC1=0.76 for somatoform and associated disorders). The usability of ADA was rated as high in the System Usability Scale (mean 81.51, SD 11.82, score range 0-100). Approximately 71% (35/49) of participants would have preferred a face-to-face over an app-based diagnostic.

CONCLUSIONS

Overall, our findings suggest that a widely available symptom checker used in the formal diagnosis of mental disorders could provide clinicians with a list of condition suggestions with moderate-to-good accuracy. However, diagnostic performance was heterogeneous between disorder categories and included low interrater reliability. Although symptom checkers have some potential to complement the diagnostic process as a screening tool, the diagnostic performance should be tested in larger samples and in comparison with further diagnostic instruments.

Improving the Quality of Life of Patients with an Underactive Thyroid Through mHealth: A Patient-Centered Approach

Background: Three hundred fifty million people worldwide suffer from underactive thyroid conditions, which can lead to infertility, obesity, heart disease, and impaired mental health when poorly managed. Although mobile health (mHealth) applications can be a useful solution for self-managing one's condition, the impact of digital solutions for improving the health of thyroid patients remains unknown. Methods: We used a mixed methods analysis to assess the ways in which a digital approach might benefit thyroid patients. A cross-sectional study was conducted among users of BOOST Thyroid, an mHealth application for patients with an underactive thyroid. We collected data using a modified Short Form 36 Health Survey Questionnaire to measure the impact of in the app on participants' perceived health and quality of life. Participants were asked to (1) score their quality of life before and after using the app, and (2) describe whether and how using the app helped them. Results: We enrolled 406 users (380 females and 26 males), aged 18-78 years. Most participants (95.8%) reported using the app was helpful; of which 68% reported it improved their quality of life and 70.8% reported it had a positive impact on their health. Participants who found the app useful experienced less symptoms and a lower intensity of remaining symptoms. A key factor reported by these participants as helping with managing their health is the information provided in the app. Conclusions: The results support the idea that a patient-centered treatment would benefit from including mHealth tools for a daily self-management of underactive thyroid condition, as it can increase health literacy and improve both one's health status and quality of life.

Goldilocks and the Three Bears: A Just-Right Hybrid Model to Synthesize the Growing Landscape of Publicly Available Health-Related Mobile Apps

Mobile health (mHealth) technologies have provided an innovative platform for the deployment of health care diagnostics, symptom monitoring, and prevention and intervention programs. Such health-related smartphone apps are universally accepted by patients and providers with over 50 million users worldwide. Despite the rise in popularity and accessibility among consumers, the evidence base in support of health-related apps has fallen well behind the rapid pace of industry development. To bridge this evidence gap, researchers are beginning to consider how to best apply evidence-based research standards to the systematic synthesis of the mHealth consumer market. In this viewpoint, we argue for the adoption of a "hybrid model" that combines a traditional systematic review with a systematic search of mobile app download platforms for health sciences researchers interested in synthesizing the state of the science of consumer apps. This approach, which we have successfully executed in a recent review, maximizes the benefits of traditional and novel approaches to address the essential question of whether popular consumer mHealth apps work.

Reducing anticholinergic medication exposure among older adults using consumer technology: Protocol for a randomized clinical trial

INTRODUCTION

A growing body of scientific evidence points to the potentially harmful cognitive effects of anticholinergic medications among older adults. Most interventions designed to promote deprescribing of anticholinergics have directly targeted healthcare professionals and have had mixed results. Consumer-facing technologies may provide a unique benefit by empowering patients and can complement existing healthcare professional-centric efforts.

METHODS

We initiated a randomized clinical trial to evaluate the effectiveness of a patient-facing mobile application (Brain Safe app) compared to an attention control medication list app in reducing anticholinergic exposure among community-dwelling older adults. Study participants are adults aged 60 years and above, currently using at least one prescribed strong anticholinergic, and receiving primary care. The trial plans to enroll a total of 700 participants, randomly allocated in 1:1 proportion to the two study arms. Participants will have the Brain Safe app (intervention arm) or attention control medication list app (control arm) loaded onto a smartphone (study provided or personal device). All participants will be followed for 12 months and will have data collected at baseline, at 6 months, and 12 months by blinded outcome assessors. The primary outcome of the study is anticholinergic exposure measured as total standard daily dose (TSDD) computed from medication prescription electronic records. Secondary outcomes of the study are cognitive function and health-related quality of life.

DISCUSSION

A consumer-facing intervention to promote deprescribing of potentially high-risk medications can be part of a multi-pronged approach to reduce inappropriate medication use among older adult patients. Delivering a deprescribing intervention via a mobile app is a novel approach and may hold great promise to accelerate deployment of medication safety initiatives across diverse patient populations.

CLINICAL TRIAL REGISTRATION

Registered at ClinicalTrials.gov on October 10, 2019. Identifier number: NCT04121858.

Artificial Intelligence Techniques That May Be Applied to Primary Care Data to Facilitate Earlier Diagnosis of Cancer: Systematic Review

BACKGROUND

More than 17 million people worldwide, including 360,000 people in the United Kingdom, were diagnosed with cancer in 2018. Cancer prognosis and disease burden are highly dependent on the disease stage at diagnosis. Most people diagnosed with cancer first present in primary care settings, where improved assessment of the (often vague) presenting symptoms of cancer could lead to earlier detection and improved outcomes for patients. There is accumulating evidence that artificial intelligence (AI) can assist clinicians in making better clinical decisions in some areas of health care.

OBJECTIVE

This study aimed to systematically review AI techniques that may facilitate earlier diagnosis of cancer and could be applied to primary care electronic health record (EHR) data. The quality of the evidence, the phase of development the AI techniques have reached, the gaps that exist in the evidence, and the potential for use in primary care were evaluated.

METHODS

We searched MEDLINE, Embase, SCOPUS, and Web of Science databases from January 01, 2000, to June 11, 2019, and included all studies providing evidence for the accuracy or effectiveness of applying AI techniques for the early detection of cancer, which may be applicable to primary care EHRs. We included all study designs in all settings and languages. These searches were extended through a scoping review of AI-based commercial technologies. The main outcomes assessed were measures of diagnostic accuracy for cancer.

RESULTS

We identified 10,456 studies; 16 studies met the inclusion criteria, representing the data of 3,862,910 patients. A total of 13 studies described the initial development and testing of AI algorithms, and 3 studies described the validation of an AI algorithm in independent data sets. One study was based on prospectively collected data; only 3 studies were based on primary care data. We found no data on implementation barriers or cost-effectiveness. Risk of bias assessment highlighted a wide range of study quality. The additional scoping review of commercial AI technologies identified 21 technologies, only 1 meeting our inclusion criteria. Meta-analysis was not undertaken because of the heterogeneity of AI modalities, data set characteristics, and outcome measures.

CONCLUSIONS

AI techniques have been applied to EHR-type data to facilitate early diagnosis of cancer, but their use in primary care settings is still at an early stage of maturity. Further evidence is needed on their performance using primary care data, implementation barriers, and cost-effectiveness before widespread adoption into routine primary care clinical practice can be recommended.

The Digital Marshmallow Test (DMT) Diagnostic and Monitoring Mobile Health App for Impulsive Behavior: Development and Validation Study

Background

The classic Marshmallow Test, where children were offered a choice between one small but immediate reward (eg, one marshmallow) or a larger reward (eg, two marshmallows) if they waited for a period of time, instigated a wealth of research on the relationships among impulsive responding, self-regulation, and clinical and life outcomes. Impulsivity is a hallmark feature of self-regulation failures that lead to poor health decisions and outcomes, making understanding and treating impulsivity one of the most important constructs to tackle in building a culture of health. Despite a large literature base, impulsivity measurement remains difficult due to the multidimensional nature of the construct and limited methods of assessment in daily life. Mobile devices and the rise of mobile health (mHealth) have changed our ability to assess and intervene with individuals remotely, providing an avenue for ambulatory diagnostic testing and interventions. Longitudinal studies with mobile devices can further help to understand impulsive behaviors and variation in state impulsivity in daily life.

Objective

The aim of this study was to develop and validate an impulsivity mHealth diagnostics and monitoring app called Digital Marshmallow Test (DMT) using both the Apple and Android platforms for widespread dissemination to researchers, clinicians, and the general public.

Methods

The DMT app was developed using Apple’s ResearchKit (iOS) and Android’s ResearchStack open source frameworks for developing health research study apps. The DMT app consists of three main modules: self-report, ecological momentary assessment, and active behavioral and cognitive tasks. We conducted a study with a 21-day assessment period (N=116 participants) to validate the novel measures of the DMT app.

Results

We used a semantic differential scale to develop self-report trait and momentary state measures of impulsivity as part of the DMT app. We identified three state factors (inefficient, thrill seeking, and intentional) that correlated highly with established measures of impulsivity. We further leveraged momentary semantic differential questions to examine intraindividual variability, the effect of daily life, and the contextual effect of mood on state impulsivity and daily impulsive behaviors. Our results indicated validation of the self-report sematic differential and related results, and of the mobile behavioral tasks, including the Balloon Analogue Risk Task and Go-No-Go task, with relatively low validity of the mobile Delay Discounting task. We discuss the design implications of these results to mHealth research.

Conclusions

This study demonstrates the potential for assessing different facets of trait and state impulsivity during everyday life and in clinical settings using the DMT mobile app. The DMT app can be further used to enhance our understanding of the individual facets that underlie impulsive behaviors, as well as providing a promising avenue for digital interventions.

Trial Registration

ClinicalTrials.gov NCT03006653; https://www.clinicaltrials.gov/ct2/show/NCT03006653

A Survey of Parental Knowledge of Respiratory Syncytial Virus and Other Respiratory Infections in Preterm Infants

PURPOSE

Evaluate parental knowledge of respiratory syncytial virus (RSV) and other respiratory infections in preterm infants.

DESIGN

Survey.

SAMPLE

Five hundred and eighty-three parents of preterm infants with generalized, Canadian provincial representation.

MAIN OUTCOME

Knowledge of RSV infection, sources of information, and parental understanding of disease risk.

RESULTS

97.9 percent (571/583) of the parents had heard about RSV, since they all had a preterm infant. Sixty-one percent reported having good knowledge of RSV; 19.4 percent had very good knowledge; 19.7 percent had little or no awareness of RSV-related infection. Most (86.3 percent) believed that RSV illness was a very serious condition; 13 percent recognized that it could be a major problem for their child. Principal sources of information were the nurse, doctor and pamphlets. Over 480 participants cited 3 or more sources of additional information-Internet, social media platforms, and educational sessions. Respiratory syncytial virus prophylaxis was a priority, but knowledge regarding the eligibility criteria for prophylaxis is essential.

How accurate are digital symptom assessment apps for suggesting conditions and urgency advice? A clinical vignettes comparison to GPs

OBJECTIVES

To compare breadth of condition coverage, accuracy of suggested conditions and appropriateness of urgency advice of eight popular symptom assessment apps.

DESIGN

Vignettes study.

SETTING

200 primary care vignettes.

INTERVENTION/COMPARATOR

For eight apps and seven general practitioners (GPs): breadth of coverage and condition-suggestion and urgency advice accuracy measured against the vignettes' gold-standard.

PRIMARY OUTCOME MEASURES

(1) Proportion of conditions 'covered' by an app, that is, not excluded because the user was too young/old or pregnant, or not modelled; (2) proportion of vignettes with the correct primary diagnosis among the top 3 conditions suggested; (3) proportion of 'safe' urgency advice (ie, at gold standard level, more conservative, or no more than one level less conservative).

RESULTS

Condition-suggestion coverage was highly variable, with some apps not offering a suggestion for many users: in alphabetical order, Ada: 99.0%; Babylon: 51.5%; Buoy: 88.5%; K Health: 74.5%; Mediktor: 80.5%; Symptomate: 61.5%; Your.MD: 64.5%; WebMD: 93.0%. Top-3 suggestion accuracy was GPs (average): 82.1%±5.2%; Ada: 70.5%; Babylon: 32.0%; Buoy: 43.0%; K Health: 36.0%; Mediktor: 36.0%; Symptomate: 27.5%; WebMD: 35.5%; Your.MD: 23.5%. Some apps excluded certain user demographics or conditions and their performance was generally greater with the exclusion of corresponding vignettes. For safe urgency advice, tested GPs had an average of 97.0%±2.5%. For the vignettes with advice provided, only three apps had safety performance within 1 SD of the GPs-Ada: 97.0%; Babylon: 95.1%; Symptomate: 97.8%. One app had a safety performance within 2 SDs of GPs-Your.MD: 92.6%. Three apps had a safety performance outside 2 SDs of GPs-Buoy: 80.0% (p<0.001); K Health: 81.3% (p<0.001); Mediktor: 87.3% (p=1.3×10^-3).

CONCLUSIONS

The utility of digital symptom assessment apps relies on coverage, accuracy and safety. While no digital tool outperformed GPs, some came close, and the nature of iterative improvements to software offers scalable improvements to care.

Online forward triage during the COVID-19 outbreak

Health systems face major challenges during the COVID-19 pandemic with new information and challenges emerging daily and frequently changing guidelines. Online forward triage tools (OFTTs) provide useful information, direct patients and free physician resources.We implemented an OFTT targeted at the current pandemic, adapted the content and goals and assessed its effects. The OFTT was implemented on 2 March 2020 and modified regularly based on the revised testing criteria issued by the Swiss Federal Office of Public Health. After testing criteria liberalised, a chatbot tool was set up on 9 April 2020 to assess urgency of testing, referral to available testing sites and need for emergency care.In the first 40 days of the OFTT, there were more than 17 300 visitors and 69.8% indicated they would have contacted the healthcare system if the online test had not been available. During the initial week of operation, using the conservative testing strategy, 9.1% of visitors received recommendations to be tested, which increased to 36.0% of visitors after a change in testing criteria on 9 March 2020. Overall, since the implementation of the tool, 26.27% of all users of the site have been directed to obtain testing. The Chatbot tool has had approximately 50 consults/day.Setting up an OFTT should be considered as part of local strategies to cope with the COVID-19 pandemic. It may ease the burden on the healthcare system, reassure patients and inform authorities. To account for the dynamic development of the pandemic, frequent adaptation of the tool is of great importance. Further research on clinical outcomes of OFTT is urgently needed.

A Comparison of Artificial Intelligence and Human Doctors for the Purpose of Triage and Diagnosis

AI virtual assistants have significant potential to alleviate the pressure on overly burdened healthcare systems by enabling patients to self-assess their symptoms and to seek further care when appropriate. For these systems to make a meaningful contribution to healthcare globally, they must be trusted by patients and healthcare professionals alike, and service the needs of patients in diverse regions and segments of the population. We developed an AI virtual assistant which provides patients with triage and diagnostic information. Crucially, the system is based on a generative model, which allows for relatively straightforward re-parameterization to reflect local disease and risk factor burden in diverse regions and population segments. This is an appealing property, particularly when considering the potential of AI systems to improve the provision of healthcare on a global scale in many regions and for both developing and developed countries. We performed a prospective validation study of the accuracy and safety of the AI system and human doctors. Importantly, we assessed the accuracy and safety of both the AI and human doctors independently against identical clinical cases and, unlike previous studies, also accounted for the information gathering process of both agents. Overall, we found that the AI system is able to provide patients with triage and diagnostic information with a level of clinical accuracy and safety comparable to that of human doctors. Through this approach and study, we hope to start building trust in AI-powered systems by directly comparing their performance to human doctors, who do not always agree with each other on the cause of patients’ symptoms or the most appropriate triage recommendation.

Limited evidence of benefits of patient operated intelligent primary care triage tools: findings of a literature review

Introduction

There is consistent evidence that the workload in general practices is substantially increasing. The digitalisation of healthcare including the use of artificial intelligence has been suggested as a solution to this problem. We wanted to explore the features of intelligent online triage tools in primary care by conducting a literature review.

Method

A systematic literature search strategy was formulated and conducted in the PubMed database and the Cochrane Library. Articles were selected according to inclusion/exclusion criteria. Results and data were systematically extracted and thematically analysed. 17 articles of that reported large multimethod studies or smaller diagnostic accuracy tests on clinical vignettes were included. Reviews and expert opinions were also considered.

Results

There was limited evidence on the actual effects and performance of triage tools in primary care. Several aspects can guide further development: concepts of system design, system implementation and diagnostic performance. The most important findings were: a need to formulate evaluation guidelines and regulations; their assumed potential has not yet been met; a risk of increased or redistribution of workloads and the available symptom checker systems seem overly risk averse and should be tested in real-life settings.

Conclusion

This review identified several features associated with the design and implementation of intelligent online triage tools in a primary care context, although most of the investigated systems seemed underdeveloped and offered limited benefits. Current online triage systems should not be used by individuals who have reasonable access to traditional healthcare. Systems used should be strictly evaluated and regulated like other medical products.

Design and testing of a mobile health application rating tool

Mobile health applications ("apps") have rapidly proliferated, yet their ability to improve outcomes for patients remains unclear. A validated tool that addresses apps' potentially important dimensions has not been available to patients and clinicians. The objective of this study was to develop and preliminarily assess a usable, valid, and open-source rating tool to objectively measure the risks and benefits of health apps. We accomplished this by using a Delphi process, where we constructed an app rating tool called THESIS that could promote informed app selection. We used a systematic process to select chronic disease apps with ≥4 stars and <4-stars and then rated them with THESIS to examine the tool's interrater reliability and internal consistency. We rated 211 apps, finding they performed fair overall (3.02 out of 5 [95% CI, 2.96-3.09]), but especially poorly for privacy/security (2.21 out of 5 [95% CI, 2.11-2.32]), interoperability (1.75 [95% CI, 1.59-1.91]), and availability in multiple languages (1.43 out of 5 [95% CI, 1.30-1.56]). Ratings using THESIS had fair interrater reliability (κ = 0.3-0.6) and excellent scale reliability (ɑ = 0.85). Correlation with traditional star ratings was low (r = 0.24), suggesting THESIS captures issues beyond general user acceptance. Preliminary testing of THESIS suggests apps that serve patients with chronic disease could perform much better, particularly in privacy/security and interoperability. THESIS warrants further testing and may guide software and policymakers to further improve app performance, so apps can more consistently improve patient outcomes.

Assessment of Mobile Health Apps Using Built-In Smartphone Sensors for Diagnosis and Treatment: Systematic Survey of Apps Listed in International Curated Health App Libraries

Background

More than a million health and well-being apps are available from the Apple and Google app stores. Some apps use built-in mobile phone sensors to generate health data. Clinicians and patients can find information regarding safe and effective mobile health (mHealth) apps in third party–curated mHealth app libraries.

Objective

These independent Web-based repositories guide app selection from trusted lists, but do they offer apps using ubiquitous, low-cost smartphone sensors to improve health? This study aimed to identify the types of built-in mobile phone sensors used in apps listed on curated health app libraries, the range of health conditions these apps address, and the cross-platform availability of the apps.

Methods

This systematic survey reviewed three such repositories (National Health Service Apps Library, AppScript, and MyHealthApps), assessing the availability of apps using built-in mobile phone sensors for the diagnosis or treatment of health conditions.

Results

A total of 18 such apps were identified and included in this survey, representing 1.1% (8/699) to 3% (2/76) of all apps offered by the respective libraries examined. About one-third (7/18, 39%) of the identified apps offered cross-platform Apple and Android versions, with a further 50% (9/18) only dedicated to Apple and 11% (2/18) to Android. About one-fourth (4/18, 22%) of the identified apps offered dedicated diagnostic functions, with a majority featuring therapeutic (9/18, 50%) or combined functionality (5/18, 28%). Cameras, touch screens, and microphones were the most frequently used built-in sensors. Health concerns addressed by these apps included respiratory, dermatological, neurological, and anxiety conditions.

Conclusions

Diligent mHealth app library curation, medical device regulation constraints, and cross-platform differences in mobile phone sensor architectures may all contribute to the observed limited availability of mHealth apps using built-in phone sensors in curated mHealth app libraries. However, more efforts are needed to increase the number of such apps on curated lists, as they offer easily accessible low-cost options to assist people in managing clinical conditions.

Patient Perspectives on the Usefulness of an Artificial Intelligence-Assisted Symptom Checker: Cross-Sectional Survey Study

Background

Patients are increasingly seeking Web-based symptom checkers to obtain diagnoses. However, little is known about the characteristics of the patients who use these resources, their rationale for use, and whether they find them accurate and useful.

Objective

The study aimed to examine patients’ experiences using an artificial intelligence (AI)–assisted online symptom checker.

Methods

An online survey was administered between March 2, 2018, through March 15, 2018, to US users of the Isabel Symptom Checker within 6 months of their use. User characteristics, experiences of symptom checker use, experiences discussing results with physicians, and prior personal history of experiencing a diagnostic error were collected.

Results

A total of 329 usable responses was obtained. The mean respondent age was 48.0 (SD 16.7) years; most were women (230/304, 75.7%) and white (271/304, 89.1%). Patients most commonly used the symptom checker to better understand the causes of their symptoms (232/304, 76.3%), followed by for deciding whether to seek care (101/304, 33.2%) or where (eg, primary or urgent care: 63/304, 20.7%), obtaining medical advice without going to a doctor (48/304, 15.8%), and understanding their diagnoses better (39/304, 12.8%). Most patients reported receiving useful information for their health problems (274/304, 90.1%), with half reporting positive health effects (154/302, 51.0%). Most patients perceived it to be useful as a diagnostic tool (253/301, 84.1%), as a tool providing insights leading them closer to correct diagnoses (231/303, 76.2%), and reported they would use it again (278/304, 91.4%). Patients who discussed findings with their physicians (103/213, 48.4%) more often felt physicians were interested (42/103, 40.8%) than not interested in learning about the tool’s results (24/103, 23.3%) and more often felt physicians were open (62/103, 60.2%) than not open (21/103, 20.4%) to discussing the results. Compared with patients who had not previously experienced diagnostic errors (missed or delayed diagnoses: 123/304, 40.5%), patients who had previously experienced diagnostic errors (181/304, 59.5%) were more likely to use the symptom checker to determine where they should seek care (15/123, 12.2% vs 48/181, 26.5%; P=.002), but they less often felt that physicians were interested in discussing the tool’s results (20/34, 59% vs 22/69, 32%; P=.04).

Conclusions

Despite ongoing concerns about symptom checker accuracy, a large patient-user group perceived an AI-assisted symptom checker as useful for diagnosis. Formal validation studies evaluating symptom checker accuracy and effectiveness in real-world practice could provide additional useful information about their benefit.

What is the clinical value of mHealth for patients?

Despite growing interest from both patients and healthcare providers, there is little clinical guidance on how mobile apps should be utilized to add value to patient care. We categorize apps according to their functionality (e.g. preventative behavior change, digital self-management of a specific condition, diagnostic) and discuss evidence for effectiveness from published systematic reviews and meta-analyses and the relevance to patient care. We discuss the limitations of the current literature describing clinical outcomes from mHealth apps, what FDA clearance means now (510(k)/de novo FDA clearance) and in the future. We discuss data security and privacy as a major concern for patients when using mHealth apps. Patients are often not involved in the development of mobile health guidelines, and professionals' views regarding high-quality health apps may not reflect patients' views. We discuss efforts to develop guidelines for the development of safe and effective mHealth apps in the US and elsewhere and the role of independent app reviews sites in identifying mHealth apps for patient care. There are only a small number of clinical scenarios where published evidence suggests that mHealth apps may improve patient outcomes.