Performance of national COVID-19 'symptom checkers': a comparative case simulation study

Predictors of COVID-19 From a Statewide Digital Symptom and Risk Assessment Tool: Cross-Sectional Study

BACKGROUND

Some of the most vexing issues with the COVID-19 pandemic were the inability of facilities and events, such as schools and work areas, to track symptoms to mitigate the spread of the disease. To combat these challenges, many turned to the implementation of technology. Technology solutions to mitigate repercussions of the COVID-19 pandemic include tools that provide guidelines and interfaces to influence behavior, reduce exposure to the disease, and enable policy-driven avenues to return to a sense of normalcy. This paper presents the implementation and early evaluation of a return-to-work COVID-19 symptom and risk assessment tool. The system was implemented across 34 institutions of health and education in Alabama, including more than 174,000 users with over 4 million total uses and more than 86,000 reports of exposure risk between July 2020 and April 2021.

OBJECTIVE

This study aimed to explore the usage of technology, specifically a COVID-19 symptom and risk assessment tool, to mitigate exposure to COVID-19 within public spaces. More specifically, the objective was to assess the relationship between user-reported symptoms and exposure via a mobile health app, with confirmed COVID-19 cases reported by the Alabama Department of Public Health (ADPH).

METHODS

This cross-sectional study evaluated the relationship between confirmed COVID-19 cases and user-reported COVID-19 symptoms and exposure reported through the Healthcheck web-based mobile application. A dependent variable for confirmed COVID-19 cases in Alabama was obtained from ADPH. Independent variables (ie, health symptoms and exposure) were collected through Healthcheck survey data and included measures assessing COVID-19-related risk levels and symptoms. Multiple linear regression was used to examine the relationship between ADPH-confirmed diagnosis of COVID-19 and self-reported health symptoms and exposure via Healthcheck that were analyzed across the state population but not connected at the individual patient level.

RESULTS

Regression analysis showed that the self-reported information collected by Healthcheck significantly affects the number of COVID-19-confirmed cases. The results demonstrate that the average number of confirmed COVID-19 cases increased by 5 (high risk: β=5.10; P=.001), decreased by 24 (sore throat: β=-24.03; P=.001), and increased by 21 (nausea or vomiting: β=21.67; P=.02) per day for every additional self-report of symptoms by Healthcheck survey respondents. Congestion or runny nose was the most frequently reported symptom. Sore throat, low risk, high risk, nausea, or vomiting were all statistically significant factors.

CONCLUSIONS

The use of technology allowed organizations to remotely track a population as it is related to COVID-19. Healthcheck was a platform that aided in symptom tracking, risk assessment, and evaluation of status for admitting individuals into public spaces for people in the Alabama area. The confirmed relationship between symptom and exposure self-reporting using an app and population-wide confirmed cases suggests that further investigation is needed to determine the opportunity for such apps to mitigate disease spread at a community and individual level.

A Digital First Healthcare Approach to Managing Pandemics: A Scoping Review of Pandemic Self-Triage Tools

BACKGROUND

During the COVID-19 pandemic, many patient-facing digital self-triage tools were designed and deployed to alleviate demand for pandemic virus triage in hospitals and doctors' offices by providing a way for people to self-assess health status and get advice on whether to seek care. These tools, provided via websites, apps, or patient portals allow people to answer questions, e.g., about symptoms and contact history, and receive guidance to appropriate care, which might be self-care.

OBJECTIVE

The purpose of this scoping review was to explore the state of literature on digital self-triage tools that direct or advise care for adults during a pandemic, and to explore what has been learned about intended purpose, use, and quality of guidance, tool usability, impact on providers, and ability to forecast health outcomes or care demand.

METHODS

A literature search was conducted in July 2021 using MEDLINE, Embase, Scopus, PsycINFO, CINAHL and Cochrane databases. Using Covidence, 1227 titles and abstracts were screened by two researchers, with 83 reviewed via full text screening. 22 articles met inclusion criteria: they allowed adults to self-assess for pandemic virus and directed to care. Using Microsoft Excel, we extracted and charted the following data: authors, publication year and country, country the tool was used in, whether the tool was integrated into a healthcare system, research question/purpose, direction of care provided, and key findings.

RESULTS

All but two studies reported on tools developed since early 2020 during the COVID-19 pandemic. Studies reported on tools were developed in 17 countries. Direction of care advice included directing to an ER, seek urgent care, contact/see a doctor, be tested, or to stay at home/self-isolate. Only two studies evaluated tool usability. One used 52 use-cases to evaluate quality of advice by tools in four countries, finding advice varied, e.g., tools in the US and UK often advising staying home when clinical assessment was warranted, while tools in Japan and Singapore advised seeking care. No study demonstrated that the tools reduce demand on the health care system, although at least one suggests data can predict demand for care and data allows monitoring public health.

CONCLUSIONS

While self-triage tools developed and used around the world have similarities in directing to care (ER, physician, self-care), they also differ in important ways. Some collect data to predict healthcare demand. Some are for use when concerned about health status; others are intended to be used repeatedly by users to monitor public health. Quality of triage may vary. The high use of such tools during the COVID-19 pandemic suggests research is needed to assess and ensure quality of advice given by self-triage tools, and to assess intended or unintended consequences on public health and health care systems.

CLINICALTRIAL

Utilization patterns and efficiency gains from use of a fully EHR-integrated COVID-19 self-triage and self-scheduling tool: a retrospective analysis

OBJECTIVE

Symptom checkers can help address high demand for SARS-CoV2 (COVID-19) testing and care by providing patients with self-service access to triage recommendations. However, health systems may be hesitant to invest in these tools, as their associated efficiency gains have not been studied. We aimed to quantify the operational efficiency gains associated with use of an online COVID-19 symptom checker as an alternative to a telephone hotline.

METHODS

In our health system, ambulatory patients can either use an online symptom checker or a telephone hotline to be triaged and connected to COVID-19 care. We performed a retrospective analysis of adults who used either method between October 20, 2021 and January 10, 2022, using call logs, electronic health record data, and local wages to calculate labor costs.

RESULTS

Of the 15 549 total COVID-19 triage encounters, 1820 (11.7%) used only the telephone hotline and 13 729 (88.3%) used the symptom checker. Only 271 (2%) of the patients who used the symptom checker also called the hotline. Hotline encounters required more clinician time compared to those involving the symptom checker (17.8 vs 0.4 min/encounter), resulting in higher average labor costs ($24.21 vs $0.55 per encounter). The symptom checker resulted in over 4200 clinician labor hours saved.

CONCLUSION

When given the option, most patients completed COVID-19 triage and visit scheduling online, resulting in substantial efficiency gains. These benefits may encourage health system investment in such tools.

Evaluation of Diagnostic and Triage Accuracy and Usability of a Symptom Checker in an Emergency Department: Observational Study

Background

Symptom checkers are clinical decision support apps for patients, used by tens of millions of people annually. They are designed to provide diagnostic and triage advice and assist users in seeking the appropriate level of care. Little evidence is available regarding their diagnostic and triage accuracy with direct use by patients for urgent conditions.

Objective

The aim of this study is to determine the diagnostic and triage accuracy and usability of a symptom checker in use by patients presenting to an emergency department (ED).

Methods

We recruited a convenience sample of English-speaking patients presenting for care in an urban ED. Each consenting patient used a leading symptom checker from Ada Health before the ED evaluation. Diagnostic accuracy was evaluated by comparing the symptom checker’s diagnoses and those of 3 independent emergency physicians viewing the patient-entered symptom data, with the final diagnoses from the ED evaluation. The Ada diagnoses and triage were also critiqued by the independent physicians. The patients completed a usability survey based on the Technology Acceptance Model.

Results

A total of 40 (80%) of the 50 participants approached completed the symptom checker assessment and usability survey. Their mean age was 39.3 (SD 15.9; range 18-76) years, and they were 65% (26/40) female, 68% (27/40) White, 48% (19/40) Hispanic or Latino, and 13% (5/40) Black or African American. Some cases had missing data or a lack of a clear ED diagnosis; 75% (30/40) were included in the analysis of diagnosis, and 93% (37/40) for triage. The sensitivity for at least one of the final ED diagnoses by Ada (based on its top 5 diagnoses) was 70% (95% CI 54%-86%), close to the mean sensitivity for the 3 physicians (on their top 3 diagnoses) of 68.9%. The physicians rated the Ada triage decisions as 62% (23/37) fully agree and 24% (9/37) safe but too cautious. It was rated as unsafe and too risky in 22% (8/37) of cases by at least one physician, in 14% (5/37) of cases by at least two physicians, and in 5% (2/37) of cases by all 3 physicians. Usability was rated highly; participants agreed or strongly agreed with the 7 Technology Acceptance Model usability questions with a mean score of 84.6%, although “satisfaction” and “enjoyment” were rated low.

Conclusions

This study provides preliminary evidence that a symptom checker can provide acceptable usability and diagnostic accuracy for patients with various urgent conditions. A total of 14% (5/37) of symptom checker triage recommendations were deemed unsafe and too risky by at least two physicians based on the symptoms recorded, similar to the results of studies on telephone and nurse triage. Larger studies are needed of diagnosis and triage performance with direct patient use in different clinical environments.

Patient Experience and Feedback after Use of an EHR-integrated COVID-19 Symptom Checker

BACKGROUND

Symptom checkers have been widely used during the COVID-19 pandemic to alleviate strain on health systems and offer patients a 24/7 self-service triage option. Although studies suggest that users may positively perceive online symptom checkers, no studies have quantified user feedback after use of an electronic health record (EHR)-integrated COVID-19 symptom checker with self-scheduling functionality.

OBJECTIVE

We aimed to understand user experience, user satisfaction, and user-reported alternatives to use of a COVID-19 symptom checker with self-triage and self-scheduling functionality.

METHODS

We launched a patient-portal based self-triage and self-scheduling tool in March 2020 for patients with COVID-19 symptoms, exposures, or questions. We made an optional, anonymous Qualtrics survey available to patients immediately after they completed the symptom checker.

RESULTS

Between December 16th, 2021 and March 28th, 2022, there were 395 unique responses to the survey. Overall, respondents reported high satisfaction across all demographics, with a median rating of 8 out of 10, and 47.6% of respondents giving a rating of 9 or 10 out of 10. User satisfaction scores were not associated with any demographic factors. The most common user-reported alternatives had the online tool not been available were calling the COVID-19 telephone hotline and sending a patient-portal message to their physician for advice. The ability to schedule a test online was the most important symptom checker feature for respondents. The most common categories of user feedback were regarding other COVID-19 services (e.g. telephone hotline), policies or procedures, or requesting additional features or functionality.

CONCLUSIONS

This analysis suggests that COVID-19 symptom checkers with self-triage and self-scheduling functionality may have high overall user satisfaction, regardless of user demographics. By allowing users to self-triage and self-schedule tests and visits, tools like this may prevent unnecessary calls and messages to clinicians. Individual feedback suggested that the user experience for this type of tool is highly dependent on the organization's operational workflows for COVID-19 testing and care. The study provides insight for the implementation and improvement of COVID-19 symptom checkers to ensure high user satisfaction.  .

COVID-19 in Brazil-Preliminary Analysis of Response Supported by Artificial Intelligence in Municipalities

The novel coronavirus disease (COVID-19) forced rapid adaptations in the way healthcare is delivered and coordinated by health systems. Brazil has a universal public health system (Sistema Unico de Saúde-SUS), being the main source of care for 75% of the population. Therefore, a saturation of the system was foreseen with the continuous increase of cases. The use of Artificial Intelligence (AI) to empower telehealth could help to tackle this by increasing a coordinated patient access to the health system. In the present study we describe a descriptive case report analyzing the use of Laura Digital Emergency Room-an AI-powered telehealth platform-in three different cities. It was computed around 130,000 interactions made by the chatbot and 24,162 patients completed the digital triage. Almost half (44.8%) of the patients were classified as having mild symptoms, 33.6% were classified as moderate and only 14.2% were classified as severe. The implementation of an AI-powered telehealth to increase accessibility while maintaining safety and leveraging value amid the unprecedent impact of the COVID-19 pandemic was feasible in Brazil and may reduce healthcare overload. New efforts to yield sustainability of affordable and scalable solutions are needed to truly leverage value in health care systems, particularly in the context of middle-low-income countries.

Reliability of COVID-19 symptom checkers as national triage tools: an international case comparison study

Objectives

Triage is a critical component of the pandemic response. It affects morbidity, mortality and how effectively the available healthcare resources are used. In a number of nations the pandemic has sponsored the adoption of novel, online, patient-led triage systems—often referred to as COVID-19 symptom checkers. The current safety and reliability of these new automated triage systems remain unknown.

Methods

We tested six symptom checkers currently in use as triage tools at a national level against 52 cases simulating COVID-19 of various severities to determine if the symptom checkers appropriately triage time-critical cases onward to healthcare contact. We further analysed and compared each symptom checker to determine the discretionary aspects of triage decision-making that govern the automated advice generated.

Results

Of the 52 clinical presentations, the absolute rate of onward referral to any form of healthcare contact was: Singapore 100%, the USA 67%, Wales 65%, England 62%, Scotland 54% and Northern Ireland 46%. Triage decisions were broadly based on either estimates of ‘risk’ or ‘disease severity’. Risk-based symptom checkers were more reliable, with severity-based symptom checkers often triaging time-critical cases to stay home without clinical contact or follow-up.

Conclusion

The COVID-19 symptom checkers analysed here were unable to reliably discriminate between mild and severe COVID-19. Risk-based symptom checkers may hold some promise of contributing to pandemic case management, while severity-based symptom checkers—the CDC and NHS 111 versions—confer too much risk to both public and healthcare services to be deemed a viable option for COVID-19 triage.