Digital and online symptom checkers and health assessment/triage services for urgent health problems: systematic review

Impact of the clinically oriented roles of a general practice receptionist: a systematic review with narrative synthesis

BACKGROUND

Modern general practice is characterised by increased demand and growing multidisciplinarity, including ring-fenced funding for additional non-clinical roles. For practice receptionists, however, training has remained unchanged for decades despite primary care being under greater pressure than ever, with receptionists becoming a growing focal point for abuse and unprecedented numbers leaving the role.

AIM

To present the evidence of the range of tasks that receptionists continue to perform, describing their impact on primary care delivery and how the role might be better supported.

DESIGN AND SETTING

Systematic review of research conducted in the UK.

METHOD

A systematic review of evidence contained in the major medical databases (MEDLINE/PubMed, CINAHL, ASSIA, Cochrane Library, and Embase) from January 2000 to March 2024 was conducted, including hand searches of the bibliographies of included studies.

RESULTS

In total, 29 studies were identified that grouped into three themes: service delivery, patient attitudes, and receptionist experience. The theme 'service delivery' confirms the continuing role of receptionists in providing administrative support alongside the clinical tasks of prioritising patients for consultations, facilitating repeat prescriptions, and communicating blood test results. The theme 'patient attitudes' describes how patients lacked trust in receptionists, who were viewed as unqualified and unnecessarily obstructive. Finally, in considering receptionist experience, the contrast between their confidence in performing administrative roles and the anxiety induced from the clinically related tasks was described, particularly the mounting pressure from patients to meet their preferences for clinician appointments.

CONCLUSION

Although confident performing administrative tasks, receptionists described uncertainty and anxiety when providing clinically oriented support or managing patients when their requests for appointments could not be met. More appropriate training or professionalisation might improve staff retainment.

Empowering nurses - a practical guide to artificial intelligence tools in healthcare settings: discussion paper

BACKGROUND

The rapid growth of artificial intelligence in healthcare is transforming how nurses deliver care and make clinical decisions. From supporting diagnostics to providing virtual health assistants, artificial intelligence offers new ways to enhance patient outcomes and streamline healthcare processes. However, these advancements also bring challenges, particularly around ethics, potential biases, and ensuring technology complements rather than replaces human expertise.

METHODS

A discussion paper designed to break down key artificial intelligence terms and demonstrate real-world applications to guide nurses to develop the skills needed to navigate this evolving technological landscape.

FINDINGS

This discussion emphasises the importance of maintaining the critical role of human clinical judgment, highlighting that artificial intelligence should support nurses' expertise rather than diminish it. The need for continuous education to keep nurses equipped with the knowledge to effectively integrate artificial intelligence into their practice is argued. With an inclusive approach, artificial intelligence has the potential to become a powerful tool that supports nurses in improving patient care while preserving the essential human touch in healthcare.

Enhancing access to primary care is critical to the future of an equitable health service: using process visualisation to understand the impact of national policy in the UK

Access to UK general practice is complicated by the need to provide equitable and universal care within a system adapting to workforce challenges, digital innovation, and unprecedented demand. Despite the importance of accessing primary care in meeting the overall aim of delivering equitable care, this is the first time the direct and indirect influence of policies intended to facilitate access have been systematically explored. Further consideration by policymakers is needed to accommodate the difference between what patients need and what patients want when accessing primary care, and the differences in their ability to utilise digital options. The designation of care was hindered by long-standing issues of reliable data and variations in the interpretation of local and national protocols and guidelines.

Accuracy of Artificial Intelligence Based Chatbots in Analyzing Orthopedic Pathologies: An Experimental Multi-Observer Analysis

Background and Objective: The rapid development of artificial intelligence (AI) is impacting the medical sector by offering new possibilities for faster and more accurate diagnoses. Symptom checker apps show potential for supporting patient decision-making in this regard. Whether the AI-based decision-making of symptom checker apps shows better performance in diagnostic accuracy and urgency assessment compared to physicians remains unclear. Therefore, this study aimed to investigate the performance of existing symptom checker apps in orthopedic and traumatology cases compared to physicians in the field. Methods: 30 fictitious case vignettes of common conditions in trauma surgery and orthopedics were retrospectively examined by four orthopedic and traumatology specialists and four different symptom checker apps for diagnostic accuracy and the recommended urgency of measures. Based on the estimation provided by the doctors and the individual symptom checker apps, the percentage of correct diagnoses and appropriate assessments of treatment urgency was calculated in mean and standard deviation [SD] in [%]. Data were analyzed statistically for accuracy and correlation between the apps and physicians using a nonparametric Spearman's correlation test (p < 0.05). Results: The physicians provided the correct diagnosis in 84.4 ± 18.4% of cases (range: 53.3 to 96.7%), and the symptom checker apps in 35.8 ± 1.0% of cases (range: 26.7 to 54.2%). The agreement in the accuracy of the diagnoses varied from low to high (Physicians vs. Physicians: Spearman's ρ: 0.143 to 0.538; Physicians vs. Apps: Spearman's ρ: 0.007 to 0.358) depending on the different physicians and apps. In relation to the whole population, the physicians correctly assessed the urgency level in 70.0 ± 4.7% (range: 66.7 to 73.3%) and the apps in 20.6 ± 5.6% (range: 10.8 to 37.5%) of cases. The agreement on the accuracy of estimating urgency levels was moderate to high between and within physicians and individual apps. Conclusions: AI-based symptom checker apps for diagnosis in orthopedics and traumatology do not yet provide a more accurate analysis regarding diagnosis and urgency evaluation than physicians. However, there is a broad variation in the accuracy between different digital tools. Altogether, this field of AI application shows excellent potential and should be further examined in future studies.

The Doctors, Their Patients, and the Symptom Checker App: Qualitative Interview Study With General Practitioners in Germany

BACKGROUND

Symptom checkers are designed for laypeople and promise to provide a preliminary diagnosis, a sense of urgency, and a suggested course of action.

OBJECTIVE

We used the international symptom checker app (SCA) Ada App as an example to answer the following question: How do general practitioners (GPs) experience the SCA in relation to the macro, meso, and micro level of their daily work, and how does this interact with work-related psychosocial resources and demands?

METHODS

We conducted 8 semistructured interviews with GPs in Germany between December 2020 and February 2022. We analyzed the data using the integrative basic method, an interpretative-reconstructive method, to identify core themes and modes of thematization.

RESULTS

Although most GPs in this study were open to digitization in health care and their practice, only one was familiar with the SCA. GPs considered the SCA as part of the "unorganized stage" of patients' searching about their conditions. Some preferred it to popular search engines. They considered it relevant to their work as soon as the SCA would influence patients' decisions to see a doctor. Some wanted to see the results of the SCA in advance in order to decide on the patient's next steps. GPs described the diagnostic process as guided by shared decision-making, with the GP taking the lead and the patient deciding. They saw diagnosis as an act of making sense of data, which the SCA would not be able to do, despite the huge amounts of data.

CONCLUSIONS

GPs took a techno-pragmatic view of SCA. They operate in a health care system of increasing scarcity. They saw the SCA as a potential work-related resource if it helped them to reduce administrative tasks and unnecessary patient contacts. The SCA was seen as a potential work-related demand if it increased workload, for example, if it increased patients' anxiety, was too risk-averse, or made patients more insistent on their own opinions.

Precision emergency medicine

BACKGROUND

Precision health is a burgeoning scientific discipline that aims to incorporate individual variability in biological, behavioral, and social factors to develop personalized health solutions. To date, emergency medicine has not deeply engaged in the precision health movement. However, rapid advances in health technology, data science, and medical informatics offer new opportunities for emergency medicine to realize the promises of precision health.

METHODS

In this article, we conceptualize precision emergency medicine as an emerging paradigm and identify key drivers of its implementation into current and future clinical practice. We acknowledge important obstacles to the specialty-wide adoption of precision emergency medicine and offer solutions that conceive a successful path forward.

RESULTS

Precision emergency medicine is defined as the use of information and technology to deliver acute care effectively, efficiently, and authentically to individual patients and their communities. Key drivers and opportunities include leveraging human data, capitalizing on technology and digital tools, providing deliberate access to care, advancing population health, and reimagining provider education and roles. Overcoming challenges in equity, privacy, and cost is essential for success. We close with a call to action to proactively incorporate precision health into the clinical practice of emergency medicine, the training of future emergency physicians, and the research agenda of the specialty.

CONCLUSIONS

Precision emergency medicine leverages new technology and data-driven artificial intelligence to advance diagnostic testing, individualize patient care plans and therapeutics, and strategically refine the convergence of the health system and the community.

E-participation in policy-making for health: a scoping review protocol

INTRODUCTION

For the general public, e-participation represents a potential solution to the challenges associated with in-person participation in health policy-making processes. By fostering democratic engagement, e-participation can enhance civic legitimacy and trust in public institutions. However, despite its importance, there is currently a gap in the literature regarding a comprehensive synthesis of studies on various aspects of e-participation in the health policy domain. These aspects include levels of participation, underlying mechanisms, barriers, facilitators, values and outcomes. To address this gap, our proposed scoping review aims to systematically investigate and classify the available literature related to e-participation in policy-making for health.

METHODS AND ANALYSIS

We will employ the Population, Concept and Context framework developed by Arksey and O'Malley (2005). Our population of interest will consist of participants involved in policy-making for health, including both government organisers of e-participation and participating citizens (the governed). To identify relevant studies, we will systematically search databases such as CINAHL (EBSCO), Academic Search Premier (EBSCO), Social Services Abstracts (ProQuest), Scopus (Elsevier), EMBASE (Elsevier), The Cochrane Database of Systematic Reviews, Campbell Collaboration, JBI Evidence Synthesis and PubMed using a predefined search strategy. Two independent reviewers will conduct a three-tiered screening process for identified articles, with a third reviewer resolving any discrepancies. Data extraction will follow a predefined yet flexible form. The results will be summarised in a narrative format, presented either in tabular or diagrammatic form.

ETHICS AND DISSEMINATION

The National Institute of Health Research of the Islamic Republic of Iran's ethics committee has approved this review study. Our findings will be disseminated through peer-reviewed publications, conference presentations and targeted knowledge-sharing sessions with relevant stakeholders.

Evaluation of AI-Driven LabTest Checker for Diagnostic Accuracy and Safety: Prospective Cohort Study

Background

In recent years, the implementation of artificial intelligence (AI) in health care is progressively transforming medical fields, with the use of clinical decision support systems (CDSSs) as a notable application. Laboratory tests are vital for accurate diagnoses, but their increasing reliance presents challenges. The need for effective strategies for managing laboratory test interpretation is evident from the millions of monthly searches on test results' significance. As the potential role of CDSSs in laboratory diagnostics gains significance, however, more research is needed to explore this area.

Objective

The primary objective of our study was to assess the accuracy and safety of LabTest Checker (LTC), a CDSS designed to support medical diagnoses by analyzing both laboratory test results and patients' medical histories.

Methods

This cohort study embraced a prospective data collection approach. A total of 101 patients aged ≥18 years, in stable condition, and requiring comprehensive diagnosis were enrolled. A panel of blood laboratory tests was conducted for each participant. Participants used LTC for test result interpretation. The accuracy and safety of the tool were assessed by comparing AI-generated suggestions to experienced doctor (consultant) recommendations, which are considered the gold standard.

Results

The system achieved a 74.3% accuracy and 100% sensitivity for emergency safety and 92.3% sensitivity for urgent cases. It potentially reduced unnecessary medical visits by 41.6% (42/101) and achieved an 82.9% accuracy in identifying underlying pathologies.

Conclusions

This study underscores the transformative potential of AI-based CDSSs in laboratory diagnostics, contributing to enhanced patient care, efficient health care systems, and improved medical outcomes. LTC's performance evaluation highlights the advancements in AI's role in laboratory medicine.

Pharmacist-led primary care interventions to promote medicines optimisation and reduce overprescribing: a systematic review of UK studies and initiatives

OBJECTIVES

To systematically review and synthesise evidence on the effectiveness and implementation barriers/facilitators of pharmacist-led interventions to promote medicines optimisation and reduce overprescribing in UK primary care.

DESIGN

Systematic review.

SETTING

UK primary care.

METHODS

We searched MEDLINE, Embase, CINAHL PsycINFO and The Cochrane Library for UK-based studies published between January 2013 and February 2023. Targeted searches for grey literature were conducted in May 2023. Quantitative and qualitative studies (including conference abstracts and grey literature) that addressed a relevant intervention and reported a primary outcome related to changes in prescribing were eligible for inclusion. Quality of included studies was assessed using the Multiple Methods Appraisal Tool. We performed a narrative synthesis, grouping studies by publication status, setting and type of data reported (effectiveness or implementation).

RESULTS

We included 14 peer-reviewed journal articles and 11 conference abstracts, together with 4 case study reports. The journal articles reported 10 different interventions, 5 delivered in general practice, 4 in care homes and 1 in community pharmacy. The quality of evidence was higher in general practice than in care home settings. It was consistently reported that the intervention improved outcomes related to prescribing, although the limited number of studies and wide range of outcomes reported made it difficult to estimate the size of any effect. Implementation was strongly influenced by relationships between pharmacists and other health and care professionals, especially general practitioners. Implementation in care homes appeared to be more complex than in general practice because of differences in systems and 'culture' between health and social care.

CONCLUSIONS

Pharmacist-led interventions have been reported to reduce overprescribing in primary care settings in the UK but a shortage of high-quality evidence means that more rigorous studies using high-quality designs are needed. More research is also needed in community pharmacy settings; to assess intervention effects on patient outcomes other than prescribing and to investigate how reducing overprescribing can impact health inequalities.

PROSPERO REGISTRATION NUMBER

CRD42023396366.

Evaluation of a Musculoskeletal Digital Assessment Routing Tool (DART): Crossover Noninferiority Randomized Pilot Trial

BACKGROUND

Musculoskeletal conditions account for 16% of global disability, resulting in a negative effect on patients and increasing demand for health care use. Triage directing patients to appropriate level intervention improving health outcomes and efficiency has been prioritized. We developed a musculoskeletal digital assessment routing tool (DART) mobile health (mHealth) system, which requires evaluation prior to implementation. Such innovations are rarely rigorously tested in clinical trials-considered the gold standard for evaluating safety and efficacy. This pilot study is a precursor to a trial assessing DART performance with a physiotherapist-led triage assessment.

OBJECTIVE

The study aims to evaluate trial design, assess procedures, and collect exploratory data to establish the feasibility of delivering an adequately powered, definitive randomized trial, assessing DART safety and efficacy in an NHS primary care setting.

METHODS

A crossover, noninferiority pilot trial using an integrated knowledge translation approach within a National Health Service England primary care setting. Participants were patients seeking assessment for a musculoskeletal condition, completing a DART assessment and the history-taking element of a face-to-face physiotherapist-led triage in a randomized order. The primary outcome was agreement between DART and physiotherapist triage recommendation. Data allowed analysis of participant recruitment and retention, randomization, blinding, study burden, and potential barriers to intervention delivery. Participant satisfaction was measured using the System Usability Scale.

RESULTS

Over 8 weeks, 129 patients were invited to participate. Of these, 92% (119/129) proceeded to eligibility assessment, with 60% (78/129) meeting the inclusion criteria and being randomized into each intervention arm (39/39). There were no dropouts and data were analyzed for all 78 participants. Agreement between physiotherapist and DART across all participants and all primary triage outcomes was 41% (32/78; 95% CI 22-45), intraclass correlation coefficient 0.37 (95% CI 0.16-0.55), indicating that the reliability of DART was poor to moderate. Feedback from the clinical service team led to an adjusted analysis yielding of 78% (61/78; 95% CI 47-78) and an intraclass correlation coefficient of 0.57 (95% CI 0.40-0.70). Participant satisfaction was measured quantitively using amalgamated System Usability Scale scores (n=78; mean score 84.0; 90% CI +2.94 to -2.94), equating to an "excellent" system. There were no study incidents, and the trial burden was acceptable.

CONCLUSIONS

Physiotherapist-DART agreement of 78%, with no adverse triage decisions and high patient satisfaction, was sufficient to conclude DART had the potential to improve the musculoskeletal pathway. Study validity was enhanced by the recruitment of real-world patients and using an integrated knowledge translation approach. Completion of a context-specific consensus process is recommended to provide definitive definitions of safety criteria, range of appropriateness, noninferiority margin, and sample size. This pilot demonstrated an adequately powered definitive trial is feasible, which would provide evidence of DART safety and efficacy, ultimately informing potential for DART implementation.

TRIAL REGISTRATION

ClinicalTrials.gov NCT04904029; http://clinicaltrials.gov/ct2/show/NCT04904029.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR2-10.2196/31541.

Efficacy and safety of a digital check-in and triage kiosk in emergency departments: a systematic review protocol

INTRODUCTION

Increasing demand for healthcare services worldwide has led to unprecedented challenges in managing patient flow and delivering timely care in emergency care settings. Overcrowding, prolonged waiting times, reduced patient satisfaction and increased mortality are some of the consequences of this increased demand. To address this issue, some healthcare providers have turned to digital systems, such as self-check-in kiosks, for efficient patient triage and prioritisation. While digital triage systems hold promise for efficient patient prioritisation, reduced data duplication, shorter waiting times, improved patient satisfaction, the impact on workflow, the accuracy of triage and staff workload require further exploration for successful implementation in emergency care settings. This systematic review aims to assess the efficacy and safety of digital check-in and triage kiosk implementation within emergency departments.

METHODS AND ANALYSIS

A systematic review will be conducted in MEDLINE (Ovid), Web of Science, Scopus and Science Direct and will include quantitative and mixed method studies with a significant quantitative component, related to self-service kiosk implementation in emergency departments. The outcomes of interest will focus on the efficacy and safety of digital triage, including triage time, workflow, the diagnostic accuracy of triage and adverse events. Risk of bias will be assessed using the Cochrane Risk of Bias Tool. A narrative synthesis will be used to summarise the findings of the included studies.

ETHICS AND DISSEMINATION

This review is exempt from ethical approval because it will be analysing published studies containing non-identifiable data. The findings will be disseminated through peer-reviewed publications.

PROSPERO REGISTRATION NUMBER

CRD42024481506.

Controlling Inputter Variability in Vignette Studies Assessing Web-Based Symptom Checkers: Evaluation of Current Practice and Recommendations for Isolated Accuracy Metrics

BACKGROUND

The rapid growth of web-based symptom checkers (SCs) is not matched by advances in quality assurance. Currently, there are no widely accepted criteria assessing SCs' performance. Vignette studies are widely used to evaluate SCs, measuring the accuracy of outcome. Accuracy behaves as a composite metric as it is affected by a number of individual SC- and tester-dependent factors. In contrast to clinical studies, vignette studies have a small number of testers. Hence, measuring accuracy alone in vignette studies may not provide a reliable assessment of performance due to tester variability.

OBJECTIVE

This study aims to investigate the impact of tester variability on the accuracy of outcome of SCs, using clinical vignettes. It further aims to investigate the feasibility of measuring isolated aspects of performance.

METHODS

Healthily's SC was assessed using 114 vignettes by 3 groups of 3 testers who processed vignettes with different instructions: free interpretation of vignettes (free testers), specified chief complaints (partially free testers), and specified chief complaints with strict instruction for answering additional symptoms (restricted testers). κ statistics were calculated to assess agreement of top outcome condition and recommended triage. Crude and adjusted accuracy was measured against a gold standard. Adjusted accuracy was calculated using only results of consultations identical to the vignette, following a review and selection process. A feasibility study for assessing symptom comprehension of SCs was performed using different variations of 51 chief complaints across 3 SCs.

RESULTS

Intertester agreement of most likely condition and triage was, respectively, 0.49 and 0.51 for the free tester group, 0.66 and 0.66 for the partially free group, and 0.72 and 0.71 for the restricted group. For the restricted group, accuracy ranged from 43.9% to 57% for individual testers, averaging 50.6% (SD 5.35%). Adjusted accuracy was 56.1%. Assessing symptom comprehension was feasible for all 3 SCs. Comprehension scores ranged from 52.9% and 68%.

CONCLUSIONS

We demonstrated that by improving standardization of the vignette testing process, there is a significant improvement in the agreement of outcome between testers. However, significant variability remained due to uncontrollable tester-dependent factors, reflected by varying outcome accuracy. Tester-dependent factors, combined with a small number of testers, limit the reliability and generalizability of outcome accuracy when used as a composite measure in vignette studies. Measuring and reporting different aspects of SC performance in isolation provides a more reliable assessment of SC performance. We developed an adjusted accuracy measure using a review and selection process to assess data algorithm quality. In addition, we demonstrated that symptom comprehension with different input methods can be feasibly compared. Future studies reporting accuracy need to apply vignette testing standardization and isolated metrics.

Longitudinal Changes in Diagnostic Accuracy of a Differential Diagnosis List Developed by an AI-Based Symptom Checker: Retrospective Observational Study

BACKGROUND

Artificial intelligence (AI) symptom checker models should be trained using real-world patient data to improve their diagnostic accuracy. Given that AI-based symptom checkers are currently used in clinical practice, their performance should improve over time. However, longitudinal evaluations of the diagnostic accuracy of these symptom checkers are limited.

OBJECTIVE

This study aimed to assess the longitudinal changes in the accuracy of differential diagnosis lists created by an AI-based symptom checker used in the real world.

METHODS

This was a single-center, retrospective, observational study. Patients who visited an outpatient clinic without an appointment between May 1, 2019, and April 30, 2022, and who were admitted to a community hospital in Japan within 30 days of their index visit were considered eligible. We only included patients who underwent an AI-based symptom checkup at the index visit, and the diagnosis was finally confirmed during follow-up. Final diagnoses were categorized as common or uncommon, and all cases were categorized as typical or atypical. The primary outcome measure was the accuracy of the differential diagnosis list created by the AI-based symptom checker, defined as the final diagnosis in a list of 10 differential diagnoses created by the symptom checker. To assess the change in the symptom checker's diagnostic accuracy over 3 years, we used a chi-square test to compare the primary outcome over 3 periods: from May 1, 2019, to April 30, 2020 (first year); from May 1, 2020, to April 30, 2021 (second year); and from May 1, 2021, to April 30, 2022 (third year).

RESULTS

A total of 381 patients were included. Common diseases comprised 257 (67.5%) cases, and typical presentations were observed in 298 (78.2%) cases. Overall, the accuracy of the differential diagnosis list created by the AI-based symptom checker was 172 (45.1%), which did not differ across the 3 years (first year: 97/219, 44.3%; second year: 32/72, 44.4%; and third year: 43/90, 47.7%; P=.85). The accuracy of the differential diagnosis list created by the symptom checker was low in those with uncommon diseases (30/124, 24.2%) and atypical presentations (12/83, 14.5%). In the multivariate logistic regression model, common disease (P<.001; odds ratio 4.13, 95% CI 2.50-6.98) and typical presentation (P<.001; odds ratio 6.92, 95% CI 3.62-14.2) were significantly associated with the accuracy of the differential diagnosis list created by the symptom checker.

CONCLUSIONS

A 3-year longitudinal survey of the diagnostic accuracy of differential diagnosis lists developed by an AI-based symptom checker, which has been implemented in real-world clinical practice settings, showed no improvement over time. Uncommon diseases and atypical presentations were independently associated with a lower diagnostic accuracy. In the future, symptom checkers should be trained to recognize uncommon conditions.

Evaluating the Diagnostic Performance of Symptom Checkers: Clinical Vignette Study

BACKGROUND

Medical self-diagnostic tools (or symptom checkers) are becoming an integral part of digital health and our daily lives, whereby patients are increasingly using them to identify the underlying causes of their symptoms. As such, it is essential to rigorously investigate and comprehensively report the diagnostic performance of symptom checkers using standard clinical and scientific approaches.

OBJECTIVE

This study aims to evaluate and report the accuracies of a few known and new symptom checkers using a standard and transparent methodology, which allows the scientific community to cross-validate and reproduce the reported results, a step much needed in health informatics.

METHODS

We propose a 4-stage experimentation methodology that capitalizes on the standard clinical vignette approach to evaluate 6 symptom checkers. To this end, we developed and peer-reviewed 400 vignettes, each approved by at least 5 out of 7 independent and experienced primary care physicians. To establish a frame of reference and interpret the results of symptom checkers accordingly, we further compared the best-performing symptom checker against 3 primary care physicians with an average experience of 16.6 (SD 9.42) years. To measure accuracy, we used 7 standard metrics, including M1 as a measure of a symptom checker's or a physician's ability to return a vignette's main diagnosis at the top of their differential list, F1-score as a trade-off measure between recall and precision, and Normalized Discounted Cumulative Gain (NDCG) as a measure of a differential list's ranking quality, among others.

RESULTS

The diagnostic accuracies of the 6 tested symptom checkers vary significantly. For instance, the differences in the M1, F1-score, and NDCG results between the best-performing and worst-performing symptom checkers or ranges were 65.3%, 39.2%, and 74.2%, respectively. The same was observed among the participating human physicians, whereby the M1, F1-score, and NDCG ranges were 22.8%, 15.3%, and 21.3%, respectively. When compared against each other, physicians outperformed the best-performing symptom checker by an average of 1.2% using F1-score, whereas the best-performing symptom checker outperformed physicians by averages of 10.2% and 25.1% using M1 and NDCG, respectively.

CONCLUSIONS

The performance variation between symptom checkers is substantial, suggesting that symptom checkers cannot be treated as a single entity. On a different note, the best-performing symptom checker was an artificial intelligence (AI)-based one, shedding light on the promise of AI in improving the diagnostic capabilities of symptom checkers, especially as AI keeps advancing exponentially.

Recommendation endpoints and safety of an online self-triage for depression symptoms

INTRODUCTION

Online symptom checkers are a way to address patient concerns and potentially offload a burdened healthcare system. However, safety outcomes of self-triage are unknown, so we reviewed triage recommendations and outcomes of our institution's depression symptom checker.

METHODS

We examined endpoint recommendations and follow-up encounters seven days afterward during 2 December 2021 to 13 December 2022. Patients with an emergency department visit or hospitalization within seven days of self-triaging had a manual review of the electronic health record to determine if the visit was related to depression, suicidal ideation, or suicide attempt. Charts were reviewed for deaths within seven days of self-triage.

RESULTS

There were 287 unique encounters from 263 unique patients. In 86.1% (247/287), the endpoint was an instruction to call nurse triage; in 3.1% of encounters (9/287), instruction was to seek emergency care. Only 20.2% (58/287) followed the recommendations given. Of the 229 patients that did not follow the endpoint recommendations, 121 (52.8%) had some type of follow-up within seven days. Nearly 11% (31/287) were triaged to endpoints not requiring urgent contact and 9.1% (26/287) to an endpoint that would not need any healthcare team input. No patients died in the study period.

CONCLUSIONS

Most patients did not follow the recommendations for follow-up care although ultimately most patients did receive care within seven days. Self-triage appears to appropriately sort patients with depressed mood to emergency care. On-line self-triaging tools for depression have the potential to safely offload some work from clinic personnel.

Testing Two Online Symptom Checkers With Vulnerable Groups: Usability Study to Improve Cognitive Accessibility of eHealth Services

BACKGROUND

The popularity of eHealth services has surged significantly, underscoring the importance of ensuring their usability and accessibility for users with diverse needs, characteristics, and capabilities. These services can pose cognitive demands, especially for individuals who are unwell, fatigued, or experiencing distress. Additionally, numerous potentially vulnerable groups, including older adults, are susceptible to digital exclusion and may encounter cognitive limitations related to perception, attention, memory, and language comprehension. Regrettably, many studies overlook the preferences and needs of user groups likely to encounter challenges associated with these cognitive aspects.

OBJECTIVE

This study primarily aims to gain a deeper understanding of cognitive accessibility in the practical context of eHealth services. Additionally, we aimed to identify the specific challenges that vulnerable groups encounter when using eHealth services and determine key considerations for testing these services with such groups.

METHODS

As a case study of eHealth services, we conducted qualitative usability testing on 2 online symptom checkers used in Finnish public primary care. A total of 13 participants from 3 distinct groups participated in the study: older adults, individuals with mild intellectual disabilities, and nonnative Finnish speakers. The primary research methods used were the thinking-aloud method, questionnaires, and semistructured interviews.

RESULTS

We found that potentially vulnerable groups encountered numerous issues with the tested services, with similar problems observed across all 3 groups. Specifically, clarity and the use of terminology posed significant challenges. The services overwhelmed users with excessive information and choices, while the terminology consisted of numerous complex medical terms that were difficult to understand. When conducting tests with vulnerable groups, it is crucial to carefully plan the sessions to avoid being overly lengthy, as these users often require more time to complete tasks. Additionally, testing with vulnerable groups proved to be quite efficient, with results likely to benefit a wider audience as well.

CONCLUSIONS

Based on the findings of this study, it is evident that older adults, individuals with mild intellectual disability, and nonnative speakers may encounter cognitive challenges when using eHealth services, which can impede or slow down their use and make the services more difficult to navigate. In the worst-case scenario, these challenges may lead to errors in using the services. We recommend expanding the scope of testing to include a broader range of eHealth services with vulnerable groups, incorporating users with diverse characteristics and capabilities who are likely to encounter difficulties in cognitive accessibility.

'Better see a doctor?' Status quo of symptom checker apps in Germany: A cross-sectional survey with a mixed-methods design (CHECK.APP)

Background

Symptom checker apps (SCAs) offer symptom classification and low-threshold self-triage for laypeople. They are already in use despite their poor accuracy and concerns that they may negatively affect primary care. This study assesses the extent to which SCAs are used by medical laypeople in Germany and which software is most popular. We examined associations between satisfaction with the general practitioner (GP) and SCA use as well as the number of GP visits and SCA use. Furthermore, we assessed the reasons for intentional non-use.

Methods

We conducted a survey comprising standardised and open-ended questions. Quantitative data were weighted, and open-ended responses were examined using thematic analysis.

Results

This study included 850 participants. The SCA usage rate was 8%, and approximately 50% of SCA non-users were uninterested in trying SCAs. The most commonly used SCAs were NetDoktor and Ada. Surprisingly, SCAs were most frequently used in the age group of 51-55 years. No significant associations were found between SCA usage and satisfaction with the GP or the number of GP visits and SCA usage. Thematic analysis revealed skepticism regarding the results and recommendations of SCAs and discrepancies between users' requirements and the features of apps.

Conclusion

SCAs are still widely unknown in the German population and have been sparsely used so far. Many participants were not interested in trying SCAs, and we found no positive or negative associations of SCAs and primary care.

An introduction to digital determinants of health

In recent years, technology has been increasingly incorporated within healthcare for the provision of safe and efficient delivery of services. Although this can be attributed to the benefits that can be harnessed, digital technology has the potential to exacerbate and reinforce preexisting health disparities. Previous work has highlighted how sociodemographic, economic, and political factors affect individuals' interactions with digital health systems and are termed social determinants of health [SDOH]. But, there is a paucity of literature addressing how the intrinsic design, implementation, and use of technology interact with SDOH to influence health outcomes. Such interactions are termed digital determinants of health [DDOH]. This paper will, for the first time, propose a definition of DDOH and provide a conceptual model characterizing its influence on healthcare outcomes. Specifically, DDOH is implicit in the design of artificial intelligence systems, mobile phone applications, telemedicine, digital health literacy [DHL], and other forms of digital technology. A better appreciation of DDOH by the various stakeholders at the individual and societal levels can be channeled towards policies that are more digitally inclusive. In tandem with ongoing work to minimize the digital divide caused by existing SDOH, further work is necessary to recognize digital determinants as an important and distinct entity.

"You're only a receptionist, what do you want to know for?": Street-level bureaucracy on the front line of primary care in the United Kingdom

Introduction

In care settings across the globe non-clinical staff are involved in filtering patients to the most appropriate source of care. This includes primary care where general practice receptionists are key in facilitating access to individual surgeries and the wider National Health Service. Despite the complexity and significance of their role little is known of how the decision-making behaviors of receptionists impact policy implementation and service delivery. By combining the agent-based implementation theory of street-level bureaucracy with a tri-level analytical framework this work acknowledges the impact of the decisions made by receptionists as street-level bureaucrats and demonstrates the benefits of using the novel framework to provide practical insight of the factors influencing those decisions.

Methods

A secondary analysis of qualitative data gathered from a series of semi-structured interviews conducted with 19 receptionists in the United Kingdom in 2019 was used to populate a tri-level framework: the micro-level relates to influences on decision making acting at an individual level, the meso-level influences at group and organizational levels, and the macro-level influences at a societal or policy level.

Results

At the micro-level we determined how receptionists are influenced by the level of rapport developed with patients and would use common sense to interpret urgency. At the meso-level, influences included their position at the forefront of premises, the culture of the workplace, and the processes and protocols used by their practice. At the macro-level, participants described the impact of limited health service capacity, the lack of mandatory training, and the growth in the use of digital technologies.

Conclusions

Street-level bureaucracy, complemented with a tri-level contextual analysis, is a useful theoretical framework to understand how health workers, such as receptionists, attempt to provide universality without sufficient resource, and could potentially be applied to other kinds of public service workers in this way. This theoretical framework also benefits from being an accessible foundation on which to base practice and policy changes.

Redesigning Primary Care: The Emergence of Artificial-Intelligence-Driven Symptom Diagnostic Tools

Modern healthcare is facing a juxtaposition of increasing patient demands owing to an aging population and a decreasing general practitioner workforce, leading to strained access to primary care. The coronavirus disease 2019 pandemic has emphasized the potential for alternative consultation methods, highlighting opportunities to minimize unnecessary care. This article discusses the role of artificial-intelligence-driven symptom checkers, particularly their efficiency, utility, and challenges in primary care. Based on a study conducted in Italian general practices, insights from both physicians and patients were gathered regarding this emergent technology, highlighting differences in perceived utility, user satisfaction, and potential challenges. While symptom checkers are seen as potential tools for addressing healthcare challenges, concerns regarding their accuracy and the potential for misdiagnosis persist. Patients generally viewed them positively, valuing their ease of use and the empowerment they provide in managing health. However, some general practitioners perceive these tools as challenges to their expertise. This article proposes that artificial-intelligence-based symptom checkers can optimize medical-history taking for the benefit of both general practitioners and patients, with potential enhancements in complex diagnostic tasks rather than routine diagnoses. It underscores the importance of carefully integrating digital innovations while preserving the essential human touch in healthcare. Symptom checkers offer promising solutions; ensuring their accuracy, reliability, and effective integration into primary care requires rigorous research, clinical guidance, and an understanding of varied user perceptions. Collaboration among technologists, clinicians, and patients is paramount for the successful evolution of digital tools in healthcare.

Supporting primary care through symptom checking artificial intelligence: a study of patient and physician attitudes in Italian general practice

BACKGROUND

Rapid advancements in artificial intelligence (AI) have led to the adoption of AI-driven symptom checkers in primary care. This study aimed to evaluate both patients' and physicians' attitudes towards these tools in Italian general practice settings, focusing on their perceived utility, user satisfaction, and potential challenges.

METHODS

This feasibility study involved ten general practitioners (GPs) and patients visiting GP offices. The patients used a chatbot-based symptom checker before their medical visit and conducted anamnestic screening for COVID-19 and a medical history algorithm concerning the current medical problem. The entered data were forwarded to the GP as medical history aid. After the medical visit, both physicians and patients evaluated their respective symptoms. Additionally, physicians performed a final overall evaluation of the symptom checker after the conclusion of the practice phase.

RESULTS

Most patients did not use symptom checkers. Overall, 49% of patients and 27% of physicians reported being rather or very satisfied with the symptom checker. The most frequent patient-reported reasons for satisfaction were ease of use, precise and comprehensive questions, perceived time-saving potential, and encouragement of self-reflection. Every other patient would consider at-home use of the symptom checker for the first appraisal of health problems to save time, reduce unnecessary visits, and/or as an aid for the physician. Patients' attitudes towards the symptom checker were not significantly associated with age, sex, or level of education. Most patients (75%) and physicians (84%) indicated that the symptom checker had no effect on the duration of the medical visit. Only a few participants found the use of the symptom checker to be disruptive to the medical visit or its quality.

CONCLUSIONS

The findings suggest a positive reception of the symptom checker, albeit with differing focus between patients and physicians. With the potential to be integrated further into primary care, these tools require meticulous clinical guidance to maximize their benefits.

TRIAL REGISTRATION

The study was not registered, as it did not include direct medical intervention on human participants.

Can clinical decision support systems be an asset in medical education? An experimental approach

BACKGROUND

Diagnostic accuracy is one of the major cornerstones of appropriate and successful medical decision-making. Clinical decision support systems (CDSSs) have recently been used to facilitate physician's diagnostic considerations. However, to date, little is known about the potential assets of CDSS for medical students in an educational setting. The purpose of our study was to explore the usefulness of CDSSs for medical students assessing their diagnostic performances and the influence of such software on students' trust in their own diagnostic abilities.

METHODS

Based on paper cases students had to diagnose two different patients using a CDSS and conventional methods such as e.g. textbooks, respectively. Both patients had a common disease, in one setting the clinical presentation was a typical one (tonsillitis), in the other setting (pulmonary embolism), however, the patient presented atypically. We used a 2x2x2 between- and within-subjects cluster-randomised controlled trial to assess the diagnostic accuracy in medical students, also by changing the order of the used resources (CDSS first or second).

RESULTS

Medical students in their 4th and 5th year performed equally well using conventional methods or the CDSS across the two cases (t(164) = 1,30; p = 0.197). Diagnostic accuracy and trust in the correct diagnosis were higher in the typical presentation condition than in the atypical presentation condition (t(85) = 19.97; p < .0001 and t(150) = 7.67; p < .0001).These results refute our main hypothesis that students diagnose more accurately when using conventional methods compared to the CDSS.

CONCLUSIONS

Medical students in their 4th and 5th year performed equally well in diagnosing two cases of common diseases with typical or atypical clinical presentations using conventional methods or a CDSS. Students were proficient in diagnosing a common disease with a typical presentation but underestimated their own factual knowledge in this scenario. Also, students were aware of their own diagnostic limitations when presented with a challenging case with an atypical presentation for which the use of a CDSS seemingly provided no additional insights.

Effects of Combinational Use of Additional Differential Diagnostic Generators on the Diagnostic Accuracy of the Differential Diagnosis List Developed by an Artificial Intelligence-Driven Automated History-Taking System: Pilot Cross-Sectional Study

BACKGROUND

Low diagnostic accuracy is a major concern in automated medical history-taking systems with differential diagnosis (DDx) generators. Extending the concept of collective intelligence to the field of DDx generators such that the accuracy of judgment becomes higher when accepting an integrated diagnosis list from multiple people than when accepting a diagnosis list from a single person may be a possible solution.

OBJECTIVE

The purpose of this study is to assess whether the combined use of several DDx generators improves the diagnostic accuracy of DDx lists.

METHODS

We used medical history data and the top 10 DDx lists (index DDx lists) generated by an artificial intelligence (AI)-driven automated medical history-taking system from 103 patients with confirmed diagnoses. Two research physicians independently created the other top 10 DDx lists (second and third DDx lists) per case by imputing key information into the other 2 DDx generators based on the medical history generated by the automated medical history-taking system without reading the index lists generated by the automated medical history-taking system. We used the McNemar test to assess the improvement in diagnostic accuracy from the index DDx lists to the three types of combined DDx lists: (1) simply combining DDx lists from the index, second, and third lists; (2) creating a new top 10 DDx list using a 1/n weighting rule; and (3) creating new lists with only shared diagnoses among DDx lists from the index, second, and third lists. We treated the data generated by 2 research physicians from the same patient as independent cases. Therefore, the number of cases included in analyses in the case using 2 additional lists was 206 (103 cases × 2 physicians' input).

RESULTS

The diagnostic accuracy of the index lists was 46% (47/103). Diagnostic accuracy was improved by simply combining the other 2 DDx lists (133/206, 65%, P<.001), whereas the other 2 combined DDx lists did not improve the diagnostic accuracy of the DDx lists (106/206, 52%, P=.05 in the collective list with the 1/n weighting rule and 29/206, 14%, P<.001 in the only shared diagnoses among the 3 DDx lists).

CONCLUSIONS

Simply adding each of the top 10 DDx lists from additional DDx generators increased the diagnostic accuracy of the DDx list by approximately 20%, suggesting that the combinational use of DDx generators early in the diagnostic process is beneficial.

Advancing AI in healthcare: A comprehensive review of best practices

Artificial Intelligence (AI) and Machine Learning (ML) are powerful tools shaping the healthcare sector. This review considers twelve key aspects of AI in clinical practice: 1) Ethical AI; 2) Explainable AI; 3) Health Equity and Bias in AI; 4) Sponsorship Bias; 5) Data Privacy; 6) Genomics and Privacy; 7) Insufficient Sample Size and Self-Serving Bias; 8) Bridging the Gap Between Training Datasets and Real-World Scenarios; 9) Open Source and Collaborative Development; 10) Dataset Bias and Synthetic Data; 11) Measurement Bias; 12) Reproducibility in AI Research. These categories represent both the challenges and opportunities of AI implementation in healthcare. While AI holds significant potential for improving patient care, it also presents risks and challenges, such as ensuring privacy, combating bias, and maintaining transparency and ethics. The review underscores the necessity of developing comprehensive best practices for healthcare organizations and fostering a diverse dialogue involving data scientists, clinicians, patient advocates, ethicists, economists, and policymakers. We are at the precipice of significant transformation in healthcare powered by AI. By continuing to reassess and refine our approach, we can ensure that AI is implemented responsibly and ethically, maximizing its benefit to patient care and public health.

Characteristics of Users and Nonusers of Symptom Checkers in Germany: Cross-Sectional Survey Study

BACKGROUND

Previous studies have revealed that users of symptom checkers (SCs, apps that support self-diagnosis and self-triage) are predominantly female, are younger than average, and have higher levels of formal education. Little data are available for Germany, and no study has so far compared usage patterns with people's awareness of SCs and the perception of usefulness.

OBJECTIVE

We explored the sociodemographic and individual characteristics that are associated with the awareness, usage, and perceived usefulness of SCs in the German population.

METHODS

We conducted a cross-sectional online survey among 1084 German residents in July 2022 regarding personal characteristics and people's awareness and usage of SCs. Using random sampling from a commercial panel, we collected participant responses stratified by gender, state of residence, income, and age to reflect the German population. We analyzed the collected data exploratively.

RESULTS

Of all respondents, 16.3% (177/1084) were aware of SCs and 6.5% (71/1084) had used them before. Those aware of SCs were younger (mean 38.8, SD 14.6 years, vs mean 48.3, SD 15.7 years), were more often female (107/177, 60.5%, vs 453/907, 49.9%), and had higher formal education levels (eg, 72/177, 40.7%, vs 238/907, 26.2%, with a university/college degree) than those unaware. The same observation applied to users compared to nonusers. It disappeared, however, when comparing users to nonusers who were aware of SCs. Among users, 40.8% (29/71) considered these tools useful. Those considering them useful reported higher self-efficacy (mean 4.21, SD 0.66, vs mean 3.63, SD 0.81, on a scale of 1-5) and a higher net household income (mean EUR 2591.63, SD EUR 1103.96 [mean US $2798.96, SD US $1192.28], vs mean EUR 1626.60, SD EUR 649.05 [mean US $1756.73, SD US $700.97]) than those who considered them not useful. More women considered SCs unhelpful (13/44, 29.5%) compared to men (4/26, 15.4%).

CONCLUSIONS

Concurring with studies from other countries, our findings show associations between sociodemographic characteristics and SC usage in a German sample: users were on average younger, of higher socioeconomic status, and more commonly female compared to nonusers. However, usage cannot be explained by sociodemographic differences alone. It rather seems that sociodemographics explain who is or is not aware of the technology, but those who are aware of SCs are equally likely to use them, independently of sociodemographic differences. Although in some groups (eg, people with anxiety disorder), more participants reported to know and use SCs, they tended to perceive them as less useful. In other groups (eg, male participants), fewer respondents were aware of SCs, but those who used them perceived them to be more useful. Thus, SCs should be designed to fit specific user needs, and strategies should be developed to help reach individuals who could benefit but are not aware of SCs yet.

Triage and Diagnostic Accuracy of Online Symptom Checkers: Systematic Review

BACKGROUND

In the context of a deepening global shortage of health workers and, in particular, the COVID-19 pandemic, there is growing international interest in, and use of, online symptom checkers (OSCs). However, the evidence surrounding the triage and diagnostic accuracy of these tools remains inconclusive.

OBJECTIVE

This systematic review aimed to summarize the existing peer-reviewed literature evaluating the triage accuracy (directing users to appropriate services based on their presenting symptoms) and diagnostic accuracy of OSCs aimed at lay users for general health concerns.

METHODS

Searches were conducted in MEDLINE, Embase, CINAHL, Health Management Information Consortium (HMIC), and Web of Science, as well as the citations of the studies selected for full-text screening. We included peer-reviewed studies published in English between January 1, 2010, and February 16, 2022, with a controlled and quantitative assessment of either or both triage and diagnostic accuracy of OSCs directed at lay users. We excluded tools supporting health care professionals, as well as disease- or specialty-specific OSCs. Screening and data extraction were carried out independently by 2 reviewers for each study. We performed a descriptive narrative synthesis.

RESULTS

A total of 21,296 studies were identified, of which 14 (0.07%) were included. The included studies used clinical vignettes, medical records, or direct input by patients. Of the 14 studies, 6 (43%) reported on triage and diagnostic accuracy, 7 (50%) focused on triage accuracy, and 1 (7%) focused on diagnostic accuracy. These outcomes were assessed based on the diagnostic and triage recommendations attached to the vignette in the case of vignette studies or on those provided by nurses or general practitioners, including through face-to-face and telephone consultations. Both diagnostic accuracy and triage accuracy varied greatly among OSCs. Overall diagnostic accuracy was deemed to be low and was almost always lower than that of the comparator. Similarly, most of the studies (9/13, 69 %) showed suboptimal triage accuracy overall, with a few exceptions (4/13, 31%). The main variables affecting the levels of diagnostic and triage accuracy were the severity and urgency of the condition, the use of artificial intelligence algorithms, and demographic questions. However, the impact of each variable differed across tools and studies, making it difficult to draw any solid conclusions. All included studies had at least one area with unclear risk of bias according to the revised Quality Assessment of Diagnostic Accuracy Studies-2 tool.

CONCLUSIONS

Although OSCs have potential to provide accessible and accurate health advice and triage recommendations to users, more research is needed to validate their triage and diagnostic accuracy before widescale adoption in community and health care settings. Future studies should aim to use a common methodology and agreed standard for evaluation to facilitate objective benchmarking and validation.

TRIAL REGISTRATION

PROSPERO CRD42020215210; https://tinyurl.com/3949zw83.

Impacts of Symptom Checkers for Laypersons' Self-diagnosis on Physicians in Primary Care: Scoping Review

BACKGROUND

Symptom checkers (SCs) for laypersons' self-assessment and preliminary self-diagnosis are widely used by the public. Little is known about the impact of these tools on health care professionals (HCPs) in primary care and their work. This is relevant to understanding how technological changes might affect the working world and how this is linked to work-related psychosocial demands and resources for HCPs.

OBJECTIVE

This scoping review aimed to systematically explore the existing publications on the impacts of SCs on HCPs in primary care and to identify knowledge gaps.

METHODS

We used the Arksey and O'Malley framework. We based our search string on the participant, concept, and context scheme and searched PubMed (MEDLINE) and CINAHL in January and June 2021. We performed a reference search in August 2021 and a manual search in November 2021. We included publications of peer-reviewed journals that focused on artificial intelligence- or algorithm-based self-diagnosing apps and tools for laypersons and had primary care or nonclinical settings as a relevant context. The characteristics of these studies were described numerically. We used thematic analysis to identify core themes. We followed the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) checklist to report the study.

RESULTS

Of the 2729 publications identified through initial and follow-up database searches, 43 full texts were screened for eligibility, of which 9 were included. Further 8 publications were included through manual search. Two publications were excluded after receiving feedback in the peer-review process. Fifteen publications were included in the final sample, which comprised 5 (33%) commentaries or nonresearch publications, 3 (20%) literature reviews, and 7 (47%) research publications. The earliest publications stemmed from 2015. We identified 5 themes. The theme finding prediagnosis comprised the comparison between SCs and physicians. We identified the performance of the diagnosis and the relevance of human factors as topics. In the theme layperson-technology relationship, we identified potentials for laypersons' empowerment and harm through SCs. Our analysis showed potential disruptions of the physician-patient relationship and uncontested roles of HCPs in the theme (impacts on) physician-patient relationship. In the theme impacts on HCPs' tasks, we described the reduction or increase in HCPs' workload. We identified potential transformations of HCPs' work and impacts on the health care system in the theme future role of SCs in health care.

CONCLUSIONS

The scoping review approach was suitable for this new field of research. The heterogeneity of technologies and wordings was challenging. We identified research gaps in the literature regarding the impact of artificial intelligence- or algorithm-based self-diagnosing apps or tools on the work of HCPs in primary care. Further empirical studies on HCPs' lived experiences are needed, as the current literature depicts expectations rather than empirical findings.

User Experience of Symptom Checkers: A Systematic Review

This review reports the user experience of symptom checkers, aiming to characterize users studied in the existing literature, identify the aspects of user experience of symptom checkers that have been studied, and offer design suggestions. Our literature search resulted in 31 publications. We found that (1) most symptom checker users are relatively young; (2) eight relevant aspects of user experience have been explored, including motivation, trust, acceptability, satisfaction, accuracy, usability, safety/security, and functionality; (3) future symptom checkers should improve their accuracy, safety, and usability. Although many facets of user experience have been explored, methodological challenges exist and some important aspects of user experience remain understudied. Further research should be conducted to explore users' needs and the context of use. More qualitative and mixed-method studies are needed to understand actual users' experiences in the future.

Mapping Inequities in Digital Health Technology Within the World Health Organization's European Region Using PROGRESS PLUS: Scoping Review

BACKGROUND

The use of digital technologies within health care rapidly increased as services transferred to web-based platforms during the COVID-19 pandemic. Inequalities in digital health across the domains of equity are not routinely examined; yet, the long-term integration of digitally delivered services needs to consider such inequalities to ensure equitable benefits.

OBJECTIVE

This scoping review aimed to map inequities in access, use, and engagement with digital health technologies across equity domains.

METHODS

We searched 4 electronic databases (MEDLINE, ASSIA, PsycINFO, and Scopus) for quantitative and mixed methods reviews and meta-analyses published between January 2016 and May 2022. Reviews were limited to those that included studies from the World Health Organization's European region. Extracted data were mapped against Cochrane's PROGRESS PLUS (place of residence, race, ethnicity, culture, and language, occupation, gender and sex, religion, education, socioeconomic status, social capital, and other characteristics) dimensions of equity.

RESULTS

In total, 404 unique citations were identified from the searches, and 2 citations were identified from other sources. After eligibility assessment, 22 reviews were included. Consistent evidence was found showing higher access to digital health technologies among patients who were of White ethnicity, were English speaking, and had no disability. There were no reviews that explored differences in access to digital health care by age, gender and sex, occupation, education, or homeless or substance misuse. Higher use of digital health technologies was observed among populations that were White, English speaking, younger, with a higher level of education, of higher economic status, and residents in urban areas. No clear evidence of differences in the use of digital technologies by occupation, gender and sex, disability, or homeless or substance misuse was found, nor was clear evidence found in the included reviews on inequalities in the engagement with digital technologies. Finally, no reviews were identified that explored differences by place of residence.

CONCLUSIONS

Despite awareness of the potential impact of inequalities in digital health, there are important evidence gaps across multiple equity domains. The development of a common framework for evaluating digital health equity in new health initiatives and consistency in reporting findings is needed.

Engagement With the Centers for Disease Control and Prevention Coronavirus Self-Checker and Guidance Provided to Users in the United States From March 23, 2020, to April 19, 2021: Thematic and Trend Analysis

BACKGROUND

In 2020, at the onset of the COVID-19 pandemic, the United States experienced surges in healthcare needs, which challenged capacity throughout the healthcare system. Stay-at-home orders in many jurisdictions, cancellation of elective procedures, and closures of outpatient medical offices disrupted patient access to care. To inform symptomatic persons about when to seek care and potentially help alleviate the burden on the healthcare system, Centers for Disease Control and Prevention (CDC) and partners developed the CDC Coronavirus Self-Checker ("Self-Checker"). This interactive tool assists individuals seeking information about COVID-19 to determine the appropriate level of care by asking demographic, clinical, and nonclinical questions during an online "conversation."

OBJECTIVE

This paper describes user characteristics, trends in use, and recommendations delivered by the Self-Checker between March 23, 2020, and April 19, 2021, for pursuing appropriate levels of medical care depending on the severity of user symptoms.

METHODS

User characteristics and trends in completed conversations that resulted in a care message were analyzed. Care messages delivered by the Self-Checker were manually classified into three overarching conversation themes: (1) seek care immediately; (2) take no action, or stay home and self-monitor; and (3) conversation redirected. Trends in 7-day averages of conversations and COVID-19 cases were examined with development and marketing milestones that potentially impacted Self-Checker user engagement.

RESULTS

Among 16,718,667 completed conversations, the Self-Checker delivered recommendations for 69.27% (n=11,580,738) of all conversations to "take no action, or stay home and self-monitor"; 28.8% (n=4,822,138) of conversations to "seek care immediately"; and 1.89% (n=315,791) of conversations were redirected to other resources without providing any care advice. Among 6.8 million conversations initiated for self-reported sick individuals without life-threatening symptoms, 59.21% resulted in a recommendation to "take no action, or stay home and self-monitor." Nearly all individuals (99.8%) who were not sick were also advised to "take no action, or stay home and self-monitor."

CONCLUSIONS

The majority of Self-Checker conversations resulted in advice to take no action, or stay home and self-monitor. This guidance may have reduced patient volume on the medical system; however, future studies evaluating patients' satisfaction, intention to follow the care advice received, course of action, and care modality pursued could clarify the impact of the Self-Checker and similar tools during future public health emergencies.

Rapid response nursing triage outcomes for COVID-19: factors associated with patient's participation in triage recommendations

BACKGROUND

COVID-19 is an ongoing global health crisis with prevention and treatment recommendations rapidly changing. Rapid response telephone triage and advice services are critical in providing timely care during pandemics. Understanding patient participation with triage recommendations and factors associated with patient participation can assist in developing sensitive and timely interventions for receiving the treatment to prevent adverse health effects of COVID-19.

METHODS

This cohort study aimed to assess patient participation (percentage of patients who followed nursing triage suggestions from the COVID hotline) and identify factors associated with patient participation in four quarterly electronic health records from March 2020 to March 2021 (Phase 1: 14 March 2020-6 June 2020; Phase 2: 17 June 2020-16 September 2020; Phase 3: 17 September 2020-16 December 2020; Phase 4: 17 December 2020-16 March 2021). All callers who provided their symptoms (including asymptomatic with exposure to COVID) and received nursing triage were included in the study. Factors associated with patient participation were identified using multivariable logistic regression analyses, including demographic variables, comorbidity variables, health behaviors, and COVID-19-related symptoms.

RESULTS

The aggregated data included 9849 encounters/calls from 9021 unique participants. Results indicated: (1) 72.5% of patient participation rate; (2) participants advised to seek emergency department care had the lowest patient participation rate (43.4%); (3) patient participation was associated with older age, a lower comorbidity index, a lack of unexplained muscle aches, and respiratory symptoms. The absence of respiratory symptoms was the only factor significantly associated with patient participation in all four phases (OR = 0.75, 0.60, 0.64, 0.52, respectively). Older age was associated with higher patient participation in three out of four phases (OR = 1.01-1.02), and a lower Charlson comorbidity index was associated with higher patient participation in phase 3 and phase 4 (OR = 0.83, 0.88).

CONCLUSION

Public participation in nursing triage during the COVID pandemic requires attention. This study supports using a nurse-led telehealth intervention and reveals crucial factors associated with patient participation. It highlighted the importance of timely follow-up in high-risk groups and the benefit of a telehealth intervention led by nurses serving as healthcare navigators during the COVID-19 pandemic.

Exploratory study: Evaluation of a symptom checker effectiveness for providing a diagnosis and evaluating the situation emergency compared to emergency physicians using simulated and standardized patients

BACKGROUND

The overloading of health care systems is an international problem. In this context, new tools such as symptom checker (SC) are emerging to improve patient orientation and triage. This SC should be rigorously evaluated and we can take a cue from the way we evaluate medical students, using objective structured clinical examinations (OSCE) with simulated patients.

OBJECTIVE

The main objective of this study was to evaluate the efficiency of a symptom checker versus emergency physicians using OSCEs as an assessment method.

METHODS

We explored a method to evaluate the ability to set a diagnosis and evaluate the emergency of a situation with simulation. A panel of medical experts wrote 220 simulated patients cases. Each situation was played twice by an actor trained to the role: once for the SC, then for an emergency physician. Like a teleconsultation, only the patient's voice was accessible. We performed a prospective non-inferiority study. If primary analysis had failed to detect non-inferiority, we have planned a superiority analysis.

RESULTS

The SC established only 30% of the main diagnosis as the emergency physician found 81% of these. The emergency physician was also superior compared to the SC in the suggestion of secondary diagnosis (92% versus 52%). In the matter of patient triage (vital emergency or not), there is still a medical superiority (96% versus 71%). We prove a non-inferiority of the SC compared to the physician in terms of interviewing time.

CONCLUSIONS AND RELEVANCE

We should use simulated patients instead of clinical cases in order to evaluate the effectiveness of SCs.

Telehealth for COVID-19: A Conceptual Framework

The world has been going through the global crisis of the coronavirus (COVID-19). It is a challenging situation for every country to tackle its healthcare system. COVID-19 spreads through physical contact with COVID-positive patients and causes potential damage to the country's health and economy system. Therefore, to overcome the chance of spreading the disease, the only preventive measure is to maintain social distancing. In this vulnerable situation, virtual resources have been utilized in order to maintain social distance, i.e., the telehealth system has been proposed and developed to access healthcare services remotely and manage people's health conditions. The telehealth system could become a regular part of our healthcare system, and during any calamity or natural disaster, it could be used as an emergency response to deal with the catastrophe. For this purpose, we proposed a conceptual telehealth framework in response to COVID-19. We focused on identifying critical issues concerning the use of telehealth in healthcare setups. Furthermore, the factors influencing the implementation of the telehealth system have been explored in detail. The proposed telehealth system utilizes artificial intelligence and data science to regulate and maintain the system efficiently. Before implementing the telehealth system, it is required that prearrangements be made, such as appropriate funding measures, the skills to know technological usage, training sessions, and staff endorsement. The barriers and influencing factors provided in this article can be helpful for future developments in telehealth systems and for making fruitful progress in fighting pandemics like COVID-19. At the same time, the same approach can be used to save the lives of many frontline workers.

A multinational survey of patient utilization of and value conveyed through virtual symptom triage and healthcare referral

Objective

To describe the use patterns, impact and derived patient-user value of a mobile web-based virtual triage/symptom checker.

Methods

Online survey of 2,113 web-based patient-users of a virtual triage/symptom checker was completed over an 8-week period. Questions focused on triage and care objectives, pre- and post-triage care intent, frequency of use, value derived and satisfaction with virtual triage. Responses were analyzed and stratified to characterize patient-user pre-triage and post-triage intent relative to triage engine output.

Results

Seventy-eight percent of virtual triage users were female, and 37% were 18-24 years old or younger, 28% were 25-44, 16% were 45-54, and 19% were 55 years or older; 41.2% completed the survey from the U.S., 12.5% from the U.K., 9.1% from Canada, 5.6% from India, 3.8% from South Africa. Motivations were to determine need to consult a physician (44.2%), to secure medical advice without visiting a physician (21.0%), and to confirm a diagnosis received (14.2%). Forty-three percent were first time users of virtual triage, 36.6% utilized a triage engine at least once every few months or more often. Pre-triage, 40.5% did not know what level of healthcare they were planning to utilize, 33.9% stated they intended to seek a physician consultation, 23.7% engage self-care and 1.8% seek emergency care. Virtual triage recommended 56.8% of patient-users consult a physician, 33.8% seek emergency care and 9.4% engage self-care. In three-fourths, virtual triage helped users decide level of care to pursue. Among 74.1%, triage recommended care different than pre-triage intentions. Post-triage, those who remained uncertain of their care path decreased by 25.4%. Patient-user experience and satisfaction with virtual triage was high, with 80.1% stating that they were highly likely or likely to use it again, and interest in and willingness to use telemedicine doubled.

Conclusion

Virtual triage successfully redirected patient-users who initially planned to seek an inappropriate level of care acuity, reduced patient uncertainty of care path, and doubled the percentage of patients amenable to telemedicine and virtual health engagement. Patient-users were highly satisfied with virtual triage and the virtual triage patient experience, and a large majority will use virtual triage recurrently in the future.

A scoping review on the use and usefulness of online symptom checkers and triage systems: How to proceed?

Background

Patients are increasingly turning to the Internet for health information. Numerous online symptom checkers and digital triage tools are currently available to the general public in an effort to meet this need, simultaneously acting as a demand management strategy to aid the overburdened health care system. The implementation of these services requires an evidence-based approach, warranting a review of the available literature on this rapidly evolving topic.

Objective

This scoping review aims to provide an overview of the current state of the art and identify research gaps through an analysis of the strengths and weaknesses of the presently available literature.

Methods

A systematic search strategy was formed and applied to six databases: Cochrane library, NICE, DARE, NIHR, Pubmed, and Web of Science. Data extraction was performed by two researchers according to a pre-established data charting methodology allowing for a thematic analysis of the results.

Results

A total of 10,250 articles were identified, and 28 publications were found eligible for inclusion. Users of these tools are often younger, female, more highly educated and technologically literate, potentially impacting digital divide and health equity. Triage algorithms remain risk-averse, which causes challenges for their accuracy. Recent evolutions in algorithms have varying degrees of success. Results on impact are highly variable, with potential effects on demand, accessibility of care, health literacy and syndromic surveillance. Both patients and healthcare providers are generally positive about the technology and seem amenable to the advice given, but there are still improvements to be made toward a more patient-centered approach. The significant heterogeneity across studies and triage systems remains the primary challenge for the field, limiting transferability of findings.

Conclusion

Current evidence included in this review is characterized by significant variability in study design and outcomes, highlighting the significant challenges for future research.An evolution toward more homogeneous methodologies, studies tailored to the intended setting, regulation and standardization of evaluations, and a patient-centered approach could benefit the field.

Dr. Google: Physicians-The Web-Patients Triangle: Digital Skills and Attitudes towards e-Health Solutions among Physicians in South Eastern Poland-A Cross-Sectional Study in a Pre-COVID-19 Era

The investment in digital e-health services is a priority direction in the development of global healthcare systems. While people are increasingly using the Web for health information, it is not entirely clear what physicians' attitudes are towards digital transformation, as well as the acceptance of new technologies in healthcare. The aim of this cross-sectional survey study was to investigate physicians' self-digital skills and their opinions on obtaining online health knowledge by patients, as well as the recognition of physicians' attitudes towards e-health solutions. Principal component analysis (PCA) was performed to emerge the variables from self-designed questionnaire and cross-sectional analysis, comparing descriptive statistics and correlations for dependent variables using the one-way ANOVA (F-test). A total of 307 physicians participated in the study, reported as using the internet mainly several times a day (66.8%). Most participants (70.4%) were familiar with new technologies and rated their e-health literacy high, although 84.0% reported the need for additional training in this field and reported a need to introduce a larger number of subjects shaping digital skills (75.9%). 53.4% of physicians perceived Internet-sourced information as sometimes reliable and, in general, assessed the effects of its use by their patients negatively (41.7%). Digital skills increased significantly with frequency of internet use (F = 13.167; p = 0.0001) and decreased with physicians' age and the need for training. Those who claimed that patients often experienced health benefits from online health showed higher digital skills (-1.06). Physicians most often recommended their patients to obtain laboratory test results online (32.2%) and to arrange medical appointments via the Internet (27.0%). Along with the deterioration of physicians' digital skills, the recommendation of e-health solutions decreased (r = 0.413) and lowered the assessment of e-health solutions for the patient (r = 0.449). Physicians perceive digitization as a sign of the times and frequently use its tools in daily practice. The evaluation of Dr. Google's phenomenon and online health is directly related to their own e-health literacy skills, but there is still a need for practical training to deal with the digital revolution.

How suitable are clinical vignettes for the evaluation of symptom checker apps? A test theoretical perspective

Objective

To evaluate the ability of case vignettes to assess the performance of symptom checker applications and to suggest refinements to the methodology used in case vignette-based audit studies.

Methods

We re-analyzed the publicly available data of two prominent case vignette-based symptom checker audit studies by calculating common metrics of test theory. Furthermore, we developed a new metric, the Capability Comparison Score (CCS), which compares symptom checker capability while controlling for the difficulty of the set of cases each symptom checker evaluated. We then scrutinized whether applying test theory and the CCS altered the performance ranking of the investigated symptom checkers.

Results

In both studies, most symptom checkers changed their rank order when adjusting the triage capability for item difficulty (ID) with the CCS. The previously reported triage accuracies commonly overestimated the capability of symptom checkers because they did not account for the fact that symptom checkers tend to selectively appraise easier cases (i.e., with high ID values). Also, many case vignettes in both studies showed insufficient (very low and even negative) values of item-total correlation (ITC), suggesting that individual items or the composition of item sets are of low quality.

Conclusions

A test-theoretic perspective helps identify previously undetected threats to the validity of case vignette-based symptom checker assessments and provides guidance and specific metrics to improve the quality of case vignettes, in particular by controlling for the difficulty of the vignettes an app was (not) able to evaluate correctly. Such measures might prove more meaningful than accuracy alone for the competitive assessment of symptom checkers. Our approach helps elaborate and standardize the methodology used for appraising symptom checker capability, which, ultimately, may yield more reliable results.

The place of digital triage in a complex healthcare system: An interview study with key stakeholders in Australia's national provider

Background

Digital triage tools such as telephone advice and online symptom checkers are now commonplace in health systems internationally. Research has focused on consumers' adherence to advice, health outcomes, satisfaction, and the degree to which these services manage demand for general practice or emergency departments. Such studies have had mixed findings, leaving equivocal the role of these services in healthcare.

Objective

We examined stakeholders' perspectives on Healthdirect, Australia's national digital triage provider, focusing on its role in the health system, and barriers to operation, in the context of the COVID-19 pandemic.

Methods

Key stakeholders took part in semi-structured interviews conducted online in the third quarter of 2021. Transcripts were coded and thematically analysed.

Results

Participants (n  =  41) were Healthdirect staff (n  =  13), employees of Primary Health Networks (PHNs; n  =  12), clinicians (n  =  9), shareholder representatives (n  =  4), consumer representatives (n  =  2) and other policymakers (n  =  1). Eight themes emerged from the analysis: (1) information and guidance in navigating the system, (2) efficiency through appropriate care, (3) value for consumers? (4) the difficulties in triage at a distance, (5) competition and the unfulfilled promise of integration, (6) challenges in promoting Healthdirect, (7) monitoring and evaluating digital triage services and (8) rapid change, challenge and opportunity from COVID-19.

Conclusion

Stakeholders varied in their views of the purpose of Healthdirect's digital triage services. They identified challenges in lack of integration, competition, and the limited public profile of the services, issues largely reflective of the complexity of the policy and health system landscape. There was acknowledgement of the value of the services during the COVID-19 pandemic, and an expectation of them realising greater potential in the wake of the rapid uptake of telehealth.

Online Self-Triage of Ear or Hearing Concerns in a Patient Portal: Comparison of Subsequent Diagnoses and Hospitalizations to National Emergency Department and National Ambulatory Ear or Hearing Visits

Background

Although online self-triage is easily accessible, little is known about the patients who use self-triage or their subsequent diagnoses. We compared ear/hearing self-triage subsequent diagnoses to ear/hearing visit diagnoses in emergency departments (ED) and ambulatory clinics across the United States.

Methods

We compared International Classification of Diseases version 10 (ICD10) coded diagnoses following online self-triage for ear/hearing concerns with those from national ED and ambulatory clinic samples. We used data from the Centers for Disease Control (CDC) National Hospital Ambulatory Medical Care Survey (NHAMCS) and National Ambulatory Medical Care Survey (NAMCS) for comparison. Using matched ear/hearing diagnostic categories for those aged 1 and over, we compared self-triage diagnosis frequencies with national ED and ambulatory diagnosis frequencies.

Results

Following ear/hearing self-triage, there were 1092 subsequent office visits with a primary diagnosis code. For five frequently diagnosed ear/hearing conditions (i.e., suppurative and nonsuppurative otitis media [OM], otalgia, otitis externa, and cerumen impaction), there was a strong correlation between diagnosis counts made following self-triage and estimated counts of national ED visit diagnoses (r = 0.94; CI 95% [0.37 to 0.99]; p = .016, adjusted r2 = 0.85). Seven diagnoses were available to compare with the national ambulatory sample; correlation was r = 0.79; CI 95% [0.08 to 0.97]; p = .037, adjusted r2 = 0.54. For ages 1 and over, estimated hospital admissions from the national ED visits for ear/hearing were 0.76%, CI 95% [0.28-2.1%]; estimated total national ear/hearing ED visits were 7.5 million (for 4 years, 2016 through 2019).

Conclusion

The strong correlation of ear-related self-triage diagnoses with national ED diagnoses and the low hospitalization risk for these diagnoses suggests that there is an opportunity for self-triage of ear/hearing concerns to decrease ED visits for these symptoms.

Challenges in evaluating the accuracy of AI-containing digital triage systems: A systematic review

INTRODUCTION

Patient-operated digital triage systems with AI components are becoming increasingly common. However, previous reviews have found a limited amount of research on such systems' accuracy. This systematic review of the literature aimed to identify the main challenges in determining the accuracy of patient-operated digital AI-based triage systems.

METHODS

A systematic review was designed and conducted in accordance with PRISMA guidelines in October 2021 using PubMed, Scopus and Web of Science. Articles were included if they assessed the accuracy of a patient-operated digital triage system that had an AI-component and could triage a general primary care population. Limitations and other pertinent data were extracted, synthesized and analysed. Risk of bias was not analysed as this review studied the included articles' limitations (rather than results). Results were synthesized qualitatively using a thematic analysis.

RESULTS

The search generated 76 articles and following exclusion 8 articles (6 primary articles and 2 reviews) were included in the analysis. Articles' limitations were synthesized into three groups: epistemological, ontological and methodological limitations. Limitations varied with regards to intractability and the level to which they can be addressed through methodological choices. Certain methodological limitations related to testing triage systems using vignettes can be addressed through methodological adjustments, whereas epistemological and ontological limitations require that readers of such studies appraise the studies with limitations in mind.

DISCUSSION

The reviewed literature highlights recurring limitations and challenges in studying the accuracy of patient-operated digital triage systems with AI components. Some of these challenges can be addressed through methodology whereas others are intrinsic to the area of inquiry and involve unavoidable trade-offs. Future studies should take these limitations in consideration in order to better address the current knowledge gaps in the literature.

"Nothing works without the doctor:" Physicians' perception of clinical decision-making and artificial intelligence

Introduction

Artificial intelligence-driven decision support systems (AI-DSS) have the potential to help physicians analyze data and facilitate the search for a correct diagnosis or suitable intervention. The potential of such systems is often emphasized. However, implementation in clinical practice deserves continuous attention. This article aims to shed light on the needs and challenges arising from the use of AI-DSS from physicians' perspectives.

Methods

The basis for this study is a qualitative content analysis of expert interviews with experienced nephrologists after testing an AI-DSS in a straightforward usage scenario.

Results

The results provide insights on the basics of clinical decision-making, expected challenges when using AI-DSS as well as a reflection on the test run.

Discussion

While we can confirm the somewhat expectable demand for better explainability and control, other insights highlight the need to uphold classical strengths of the medical profession when using AI-DSS as well as the importance of broadening the view of AI-related challenges to the clinical environment, especially during treatment. Our results stress the necessity for adjusting AI-DSS to shared decision-making. We conclude that explainability must be context-specific while fostering meaningful interaction with the systems available.

Ethical, legal, and social aspects of symptom checker applications: a scoping review

Symptom Checker Applications (SCA) are mobile applications often designed for the end-user to assist with symptom assessment and self-triage. SCA are meant to provide the user with easily accessible information about their own health conditions. However, SCA raise questions regarding ethical, legal, and social aspects (ELSA), for example, regarding fair access to this new technology. The aim of this scoping review is to identify the ELSA of SCA in the scientific literature. A scoping review was conducted to identify the ELSA of SCA. Ten databases (e.g., Web of Science and PubMed) were used. Studies on SCA that address ELSA, written in English or German, were included in the review. The ELSA of SCA were extracted and synthesized using qualitative content analysis. A total of 25,061 references were identified, of which 39 were included in the analysis. The identified aspects were allotted to three main categories: (1) Technology; (2) Individual Level; and (3) Healthcare system. The results show that there are controversial debates in the literature on the ethical and social challenges of SCA usage. Furthermore, the debates are characterised by a lack of a specific legal perspective and empirical data. The review provides an overview on the spectrum of ELSA regarding SCA. It offers guidance to stakeholders in the healthcare system, for example, patients, healthcare professionals, and insurance providers and could be used in future empirical research to investigate the perspectives of those affected, such as users.

Assessing data gathering of chatbot based symptom checkers - a clinical vignettes study

BACKGROUND

The burden on healthcare systems is mounting continuously owing to population growth and aging, overuse of medical services, and the recent COVID-19 pandemic. This overload is also causing reduced healthcare quality and outcomes. One solution gaining momentum is the integration of intelligent self-assessment tools, known as symptom-checkers, into healthcare-providers' systems. To the best of our knowledge, no study so far has investigated the data-gathering capabilities of these tools, which represent a crucial resource for simulating doctors' skills in medical-interviews.

OBJECTIVES

The goal of this study was to evaluate the data-gathering function of currently available chatbot symptom-checkers.

METHODS

We evaluated 8 symptom-checkers using 28 clinical vignettes from the repository of MSD-Manual case studies. The mean number of predefined pertinent findings for each case was 31.8 ± 6.8. The vignettes were entered into the platforms by 3 medical students who simulated the role of the patient. For each conversation, we obtained the number of pertinent findings retrieved and the number of questions asked. We then calculated the recall-rates (pertinent-findings retrieved out of all predefined pertinent-findings), and efficiency-rates (pertinent-findings retrieved out of the number of questions asked) of data-gathering, and compared them between the platforms.

RESULTS

The overall recall rate for all symptom-checkers was 0.32(2,280/7,112;95 %CI 0.31-0.33) for all pertinent findings, 0.37(1,110/2,992;95 %CI 0.35-0.39) for present findings, and 0.28(1140/4120;95 %CI 0.26-0.29) for absent findings. Among the symptom-checkers, Kahun platform had the highest recall rate with 0.51(450/889;95 %CI 0.47-0.54). Out of 4,877 questions asked overall, 2,280 findings were gathered, yielding an efficiency rate of 0.46(95 %CI 0.45-0.48) across all platforms. Kahun was the most efficient tool 0.74 (95 %CI 0.70-0.77) without a statistically significant difference from Your.MD 0.69(95 %CI 0.65-0.73).

CONCLUSION

The data-gathering performance of currently available symptom checkers is questionable. From among the tools available, Kahun demonstrated the best overall performance.

Utilization patterns and efficiency gains from use of a fully EHR-integrated COVID-19 self-triage and self-scheduling tool: a retrospective analysis

OBJECTIVE

Symptom checkers can help address high demand for SARS-CoV2 (COVID-19) testing and care by providing patients with self-service access to triage recommendations. However, health systems may be hesitant to invest in these tools, as their associated efficiency gains have not been studied. We aimed to quantify the operational efficiency gains associated with use of an online COVID-19 symptom checker as an alternative to a telephone hotline.

METHODS

In our health system, ambulatory patients can either use an online symptom checker or a telephone hotline to be triaged and connected to COVID-19 care. We performed a retrospective analysis of adults who used either method between October 20, 2021 and January 10, 2022, using call logs, electronic health record data, and local wages to calculate labor costs.

RESULTS

Of the 15 549 total COVID-19 triage encounters, 1820 (11.7%) used only the telephone hotline and 13 729 (88.3%) used the symptom checker. Only 271 (2%) of the patients who used the symptom checker also called the hotline. Hotline encounters required more clinician time compared to those involving the symptom checker (17.8 vs 0.4 min/encounter), resulting in higher average labor costs ($24.21 vs $0.55 per encounter). The symptom checker resulted in over 4200 clinician labor hours saved.

CONCLUSION

When given the option, most patients completed COVID-19 triage and visit scheduling online, resulting in substantial efficiency gains. These benefits may encourage health system investment in such tools.

Online Symptom Checkers: Recommendations for a Vignette-Based Clinical Evaluation Standard

The use of patient-facing online symptom checkers (OSCs) has expanded in recent years, but their accuracy, safety, and impact on patient behaviors and health care systems remain unclear. The lack of a standardized process of clinical evaluation has resulted in significant variation in approaches to OSC validation and evaluation. The aim of this paper is to characterize a set of congruent requirements for a standardized vignette-based clinical evaluation process of OSCs. Discrepancies in the findings of comparative studies to date suggest that different steps in OSC evaluation methodology can significantly influence outcomes. A standardized process with a clear specification for vignette-based clinical evaluation is urgently needed to guide developers and facilitate the objective comparison of OSCs. We propose 15 recommendation requirements for an OSC evaluation standard. A third-party evaluation process and protocols for prospective real-world evidence studies should also be prioritized to quality assure OSC assessment.

WtsWrng Interim Comparative Effectiveness Evaluation and Description of the Challenges to Develop, Assess, and Introduce This Novel Digital Application in a Traditional Health System

Science and technology have evolved quickly during the two decades of the 21st century, but healthcare systems are grounded in last century's structure and processes. Changes in the way health care is provided are demanded; digital transformation is a key driver making healthcare systems more accessible, agile, efficient, and citizen-centered. Nevertheless, the way healthcare systems function challenges the development (Innovation + Development and regulatory requirements), assessment (methodological guidance weaknesses), and adoption of digital applications (DAs). WtsWrng (WW), an innovative DA which uses images to interact with citizens for symptom triage and monitoring, is used as an example to show the challenges faced in its development and clinical validation and how these are being overcome. To prove WW's value from inception, novel approaches for evidence generation that allows for an agile and patient-centered development have been applied. Early scientific advice from NICE (UK) was sought for study design, an iterative development and interim analysis was performed, and different statistical parameters (Kappa, B statistic) were explored to face development and assessment challenges. WW triage accuracy at cutoff time ranged from 0.62 to 0.94 for the most frequent symptoms attending the Emergency Department (ED), with the observed concordance for the 12 most frequent diagnostics at hospital discharge fluctuating between 0.4 to 0.97; 8 of the diagnostics had a concordance greater than 0.8. This experience should provoke reflective thinking for DA developers, digital health scientists, regulators, health technology assessors, and payers.

Evaluation of Diagnostic and Triage Accuracy and Usability of a Symptom Checker in an Emergency Department: Observational Study

Background

Symptom checkers are clinical decision support apps for patients, used by tens of millions of people annually. They are designed to provide diagnostic and triage advice and assist users in seeking the appropriate level of care. Little evidence is available regarding their diagnostic and triage accuracy with direct use by patients for urgent conditions.

Objective

The aim of this study is to determine the diagnostic and triage accuracy and usability of a symptom checker in use by patients presenting to an emergency department (ED).

Methods

We recruited a convenience sample of English-speaking patients presenting for care in an urban ED. Each consenting patient used a leading symptom checker from Ada Health before the ED evaluation. Diagnostic accuracy was evaluated by comparing the symptom checker’s diagnoses and those of 3 independent emergency physicians viewing the patient-entered symptom data, with the final diagnoses from the ED evaluation. The Ada diagnoses and triage were also critiqued by the independent physicians. The patients completed a usability survey based on the Technology Acceptance Model.

Results

A total of 40 (80%) of the 50 participants approached completed the symptom checker assessment and usability survey. Their mean age was 39.3 (SD 15.9; range 18-76) years, and they were 65% (26/40) female, 68% (27/40) White, 48% (19/40) Hispanic or Latino, and 13% (5/40) Black or African American. Some cases had missing data or a lack of a clear ED diagnosis; 75% (30/40) were included in the analysis of diagnosis, and 93% (37/40) for triage. The sensitivity for at least one of the final ED diagnoses by Ada (based on its top 5 diagnoses) was 70% (95% CI 54%-86%), close to the mean sensitivity for the 3 physicians (on their top 3 diagnoses) of 68.9%. The physicians rated the Ada triage decisions as 62% (23/37) fully agree and 24% (9/37) safe but too cautious. It was rated as unsafe and too risky in 22% (8/37) of cases by at least one physician, in 14% (5/37) of cases by at least two physicians, and in 5% (2/37) of cases by all 3 physicians. Usability was rated highly; participants agreed or strongly agreed with the 7 Technology Acceptance Model usability questions with a mean score of 84.6%, although “satisfaction” and “enjoyment” were rated low.

Conclusions

This study provides preliminary evidence that a symptom checker can provide acceptable usability and diagnostic accuracy for patients with various urgent conditions. A total of 14% (5/37) of symptom checker triage recommendations were deemed unsafe and too risky by at least two physicians based on the symptoms recorded, similar to the results of studies on telephone and nurse triage. Larger studies are needed of diagnosis and triage performance with direct patient use in different clinical environments.

Understanding the invisible workforce: lessons for general practice from a survey of receptionists

Introduction

The significance of the role of receptionists during the recent shift to remote triage has been widely recognised and they will have a significant role to play in UK general practice as it continues to cope with a huge increase in demand exacerbated by the COVID-19 pandemic. To maximise their contribution, it is important the social and occupational characteristics of the modern receptionist are understood, alongside their attitudes towards the role and their perceptions of the support and training they receive .

Methods

We used convenience and cross-sectional sampling to survey the demographic characteristics of receptionists and various aspects of their role and responsibilities. This included the training received, specific tasks performed, job satisfaction, the importance of the role, and their interaction with clinical and non-clinical colleagues. We also captured data on the characteristics of their practice including the number of GPs and location.

Results

A total of 70 participants completed the survey (16 postal and 54 online responses) of whom the majority were white (97.2%), female (98.6%), and aged 40 and over (56.7%). The majority of the training focussed on customer service (72.9%), telephone (64.3%), and medical administration skills (58.6%). Just over a quarter had received training in basic triage (25.7%). A standard multiple regression model revealed that the strongest predictor of satisfaction was support from practice GPs (β = .65, p <.001) there were also significant positive correlations between satisfaction and appreciation from GPs, r(68) = .609, p < .001.

Conclusion

This study has provided a much-needed update on the demographics, duties, and job satisfaction of GP receptionists. The need for diversification of the workforce to reflect the range of primary care patients warrants consideration in light of continuing variation in access along lines of gender andethnicity. Training continues to focus on administrative duties not on the clinically relevant aspects of their role such as triage.

The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review

Digital and online symptom checkers are an increasingly adopted class of health technologies that enable patients to input their symptoms and biodata to produce a set of likely diagnoses and associated triage advice. However, concerns regarding the accuracy and safety of these symptom checkers have been raised. This systematic review evaluates the accuracy of symptom checkers in providing diagnoses and appropriate triage advice. MEDLINE and Web of Science were searched for studies that used either real or simulated patients to evaluate online or digital symptom checkers. The primary outcomes were the diagnostic and triage accuracy of the symptom checkers. The QUADAS-2 tool was used to assess study quality. Of the 177 studies retrieved, 10 studies met the inclusion criteria. Researchers evaluated the accuracy of symptom checkers using a variety of medical conditions, including ophthalmological conditions, inflammatory arthritides and HIV. A total of 50% of the studies recruited real patients, while the remainder used simulated cases. The diagnostic accuracy of the primary diagnosis was low across included studies (range: 19–37.9%) and varied between individual symptom checkers, despite consistent symptom data input. Triage accuracy (range: 48.8–90.1%) was typically higher than diagnostic accuracy. Overall, the diagnostic and triage accuracy of symptom checkers are variable and of low accuracy. Given the increasing push towards adopting this class of technologies across numerous health systems, this study demonstrates that reliance upon symptom checkers could pose significant patient safety hazards. Large-scale primary studies, based upon real-world data, are warranted to demonstrate the adequate performance of these technologies in a manner that is non-inferior to current best practices. Moreover, an urgent assessment of how these systems are regulated and implemented is required.