Estimating the efficacy of symptom-based screening for COVID-19

The Stanford Medicine data science ecosystem for clinical and translational research

Objective

To describe the infrastructure, tools, and services developed at Stanford Medicine to maintain its data science ecosystem and research patient data repository for clinical and translational research.

Materials and Methods

The data science ecosystem, dubbed the Stanford Data Science Resources (SDSR), includes infrastructure and tools to create, search, retrieve, and analyze patient data, as well as services for data deidentification, linkage, and processing to extract high-value information from healthcare IT systems. Data are made available via self-service and concierge access, on HIPAA compliant secure computing infrastructure supported by in-depth user training.

Results

The Stanford Medicine Research Data Repository (STARR) functions as the SDSR data integration point, and includes electronic medical records, clinical images, text, bedside monitoring data and HL7 messages. SDSR tools include tools for electronic phenotyping, cohort building, and a search engine for patient timelines. The SDSR supports patient data collection, reproducible research, and teaching using healthcare data, and facilitates industry collaborations and large-scale observational studies.

Discussion

Research patient data repositories and their underlying data science infrastructure are essential to realizing a learning health system and advancing the mission of academic medical centers. Challenges to maintaining the SDSR include ensuring sufficient financial support while providing researchers and clinicians with maximal access to data and digital infrastructure, balancing tool development with user training, and supporting the diverse needs of users.

Conclusion

Our experience maintaining the SDSR offers a case study for academic medical centers developing data science and research informatics infrastructure.

Machine learning for optimal test admission in the presence of resource constraints

Developing rapid tools for early detection of viral infection is crucial for pandemic containment. This is particularly crucial when testing resources are constrained and/or there are significant delays until the test results are available - as was quite common in the early days of Covid-19 pandemic. We show how predictive analytics methods using machine learning algorithms can be combined with optimal pre-test screening mechanisms, greatly increasing test efficiency (i.e., rate of true positives identified per test), as well as to allow doctors to initiate treatment before the test results are available. Our optimal test admission policies account for imperfect accuracy of both the medical test and the model prediction mechanism. We derive the accuracy required for the optimized admission policies to be effective. We also show how our policies can be extended to re-testing high-risk patients, as well as combined with pool testing approaches. We illustrate our techniques by applying them to a large data reported by the Israeli Ministry of Health for RT-PCR tests from March to September 2020. Our results demonstrate that in the context of the Covid-19 pandemic a pre-test probability screening tool with conventional RT-PCR testing could have potentially increased efficiency by several times, compared to random admission control.

Towards an ML-based semantic IoT for pandemic management: A survey of enabling technologies for COVID-19

The connection between humans and digital technologies has been documented extensively in the past decades but needs to be evaluated through the current global pandemic. Artificial Intelligence(AI), with its two strands, Machine Learning (ML) and Semantic Reasoning, has proven to be a great solution to provide efficient ways to prevent, diagnose and limit the spread of COVID-19. IoT solutions have been widely proposed for COVID-19 disease monitoring, infection geolocation, and social applications. In this paper, we investigate the usage of the three technologies for handling the COVID-19 pandemic. For this purpose, we surveyed the existing ML applications and algorithms proposed during the pandemic to detect COVID-19 disease using symptom factors and image processing. The survey includes existing approaches including semantic technologies and IoT systems for COVID-19. Based on the survey result, we classified the main challenges and the solutions that could solve them. The study proposes a conceptual framework for pandemic management and discusses challenges and trends for future research.

Developing and Evaluating a Measure of the Willingness to Use Pandemic-Related mHealth Tools Using National Probability Samples in the United States: Quantitative Psychometric Analyses and Tests of Sociodemographic Group Differences

BACKGROUND

There are no psychometrically validated measures of the willingness to engage in public health screening and prevention efforts, particularly mobile health (mHealth)-based tracking, that can be adapted to future crises post-COVID-19.

OBJECTIVE

The psychometric properties of a novel measure of the willingness to participate in pandemic-related screening and tracking, including the willingness to use pandemic-related mHealth tools, were tested.

METHODS

Data were from a cross-sectional, national probability survey deployed in 3 cross-sectional stages several weeks apart to adult residents of the United States (N=6475; stage 1 n=2190, 33.82%; stage 2 n=2238, 34.56%; and stage 3 n=2047, 31.62%) from the AmeriSpeak probability-based research panel covering approximately 97% of the US household population. Five items asked about the willingness to use mHealth tools for COVID-19-related screening and tracking and provide biological specimens for COVID-19 testing.

RESULTS

In the first, exploratory sample, 3 of 5 items loaded onto 1 underlying factor, the willingness to use pandemic-related mHealth tools, based on exploratory factor analysis (EFA). A 2-factor solution, including the 3-item factor, fit the data (root mean square error of approximation [RMSEA]=0.038, comparative fit index [CFI]=1.000, standardized root mean square residual [SRMR]=0.005), and the factor loadings for the 3 items ranged from 0.849 to 0.893. In the second, validation sample, the reliability of the 3-item measure was high (Cronbach α=.90), and 1 underlying factor for the 3 items was confirmed using confirmatory factor analysis (CFA): RMSEA=0, CFI=1.000, SRMR=0 (a saturated model); factor loadings ranged from 1.000 to 0.962. The factor was independently associated with COVID-19-preventive behaviors (eg, "worn a face mask": r=0.313, SE=0.041, P<.001; "kept a 6-foot distance from those outside my household": r=0.282, SE=0.050, P<.001) and the willingness to provide biological specimens for COVID-19 testing (ie, swab to cheek or nose: r=0.709, SE=0.017, P<.001; small blood draw: r=0.684, SE=0.019, P<.001). In the third, multiple-group sample, the measure was invariant, or measured the same thing in the same way (ie, difference in CFI [ΔCFI]<0.010 across all grouping categories), across age groups, gender, racial/ethnic groups, education levels, US geographic region, and population density (ie, rural, suburban, urban). When repeated across different samples, factor-analytic findings were essentially the same. Additionally, there were mean differences (ΔM) in the willingness to use mHealth tools across samples, mainly based on race or ethnicity and population density. For example, in SD units, suburban (ΔM=-0.30, SE=0.13, P=.001) and urban (ΔM=-0.42, SE=0.12, P<.001) adults showed less willingness to use mHealth tools than rural adults in the third sample collected on May 30-June 8, 2020, but no differences were detected in the first sample collected on April 20-26, 2020.

CONCLUSIONS

Findings showed that the screener is psychometrically valid. It can also be adapted to future public health crises. Racial and ethnic minority adults showed a greater willingness to use mHealth tools than White adults. Rural adults showed more mHealth willingness than suburban and urban adults. Findings have implications for public health screening and tracking and understanding digital health inequities, including lack of uptake.

Wearable technology for early detection of COVID-19: A systematic scoping review

Wearable technology is an emerging method for the early detection of coronavirus disease 2019 (COVID-19) infection. This scoping review explored the types, mechanisms, and accuracy of wearable technology for the early detection of COVID-19. This review was conducted according to the five-step framework of Arksey and O'Malley. Studies published between December 31, 2019 and December 15, 2021 were obtained from 10 electronic databases, namely, PubMed, Embase, Cochrane, CINAHL, PsycINFO, ProQuest, Scopus, Web of Science, IEEE Xplore, and Taylor & Francis Online. Grey literature, reference lists, and key journals were also searched. All types of articles describing wearable technology for the detection of COVID-19 infection were included. Two reviewers independently screened the articles against the eligibility criteria and extracted the data using a data charting form. A total of 40 articles were included in this review. There are 22 different types of wearable technology used to detect COVID-19 infections early in the existing literature and are categorized as smartwatches or fitness trackers (67%), medical devices (27%), or others (6%). Based on deviations in physiological characteristics, anomaly detection models that can detect COVID-19 infection early were built using artificial intelligence or statistical analysis techniques. Reported area-under-the-curve values ranged from 75% to 94.4%, and sensitivity and specificity values ranged from 36.5% to 100% and 73% to 95.3%, respectively. Further research is necessary to validate the effectiveness and clinical dependability of wearable technology before healthcare policymakers can mandate its use for remote surveillance.

Estimating COVID-19 cases in Puerto Rico using an automated surveillance system

Due to concerns regarding limited testing and accuracy of estimation of COVID-19 cases, we created an automated surveillance system called “Puerto Rico Epidemiological Evaluation and Prevention of COVID-19 and Influenza” (PREPCOVI) to evaluate COVID-19 incidence and time trends across Puerto Rico. Automated text message invitations were sent to random phone numbers with Puerto Rican area codes. In addition to reported COVID-19 test results, we used a published model to classify cases from specific symptoms (loss of smell and taste, severe persistent cough, severe fatigue, and skipped meals). Between 18 November 2020, and 24 June 2021, we sent 1,427,241 messages, 26.8% were reached, and 6,975 participants answered questions about the last 30 days. Participants were aged 21–93 years and represented 97.4% of the municipalities. PREPCOVI total COVID-19 cases were higher among women and people aged between 21 and 40 years and in the Arecibo and Bayamón regions. COVID-19 was confirmed, and probable cases decreased over the study period. Confirmed COVID-19 cases ranged from 1.6 to 0.2% monthly, although testing rates only ranged from 30 to 42%. Test positivity decreased from 13.2% in November to 6.4% in March, increased in April (11.1%), and decreased in June (1.5%). PREPCOVI total cases (6.5%) were higher than cases reported by the Puerto Rico Department of Health (5.3%) for similar time periods, but time trends were similar. Automated surveillance systems and symptom-based models are useful in estimating COVID-19 cases and time trends, especially when testing is limited.

Safety Nets Work Both Ways: The Influence of Available Paid Leave on Employee Risk Taking During the COVID-19 Pandemic

BACKGROUND

During the COVID-19 pandemic, use of symptom-screening tools to limit attendance of infected workers has been widespread. However, it remains unknown how the reliability of responses to these tools may be compromised by individual and social factors. We aimed to determine whether personal concern over lost wages impacts responses to COVID-19 symptom-screening questionnaires making them less useful in limiting person-to-person transmission.

METHODS

We utilized an anonymous online questionnaire, administered through personal social media networks and those of two U.S. private colleges between September 16, 2020 and November 2, 2020 and distributed to currently or recently employed individuals 18 years of age or older. Participants considered ambiguous hypothetical scenarios involving possible COVID-19 symptoms or exposure and responded to a COVID-19 symptom screen (N = 219).

FINDINGS

In response to symptom-related scenarios (i.e., elevated temperature or slight cough), respondents lacking access to paid sick leave were 2.2 to 2.7 times more likely to attend work than those with access to paid leave (p < .05). This was not true for contact-related scenarios. Pay type and income level also significantly influenced screening responses.

CONCLUSION/APPLICATION TO PRACTICE

Risk of acute wage loss and overall financial stability appear to influence work-attendance decisions with regard to COVID-19 symptom screens. Broadened availability of paid leave and additional specificity within screening questionnaires would likely improve symptom-screen reliability.

Data science approaches to confronting the COVID-19 pandemic: a narrative review

During the COVID-19 pandemic, more than ever, data science has become a powerful weapon in combating an infectious disease epidemic and arguably any future infectious disease epidemic. Computer scientists, data scientists, physicists and mathematicians have joined public health professionals and virologists to confront the largest pandemic in the century by capitalizing on the large-scale 'big data' generated and harnessed for combating the COVID-19 pandemic. In this paper, we review the newly born data science approaches to confronting COVID-19, including the estimation of epidemiological parameters, digital contact tracing, diagnosis, policy-making, resource allocation, risk assessment, mental health surveillance, social media analytics, drug repurposing and drug development. We compare the new approaches with conventional epidemiological studies, discuss lessons we learned from the COVID-19 pandemic, and highlight opportunities and challenges of data science approaches to confronting future infectious disease epidemics. This article is part of the theme issue 'Data science approaches to infectious disease surveillance'.

Implementation of a COVID-19 screening tool in a southern Nigerian tertiary health facility

Screening for coronavirus disease 2019 (COVID-19) in emergency rooms of health facilities during outbreaks prevents nosocomial transmission. However, effective tools adapted for use in African countries are lacking. This study appraised an indigenous screening and triage tool for COVID-19 deployed at the medical emergency room of a Nigerian tertiary facility and determined the predictors of a positive molecular diagnostic test for COVID-19. A cross-sectional study of all patients seen between May and July 2020 at the Accident and Emergency of the University of Benin Teaching Hospital was conducted. Patients with any one of the inputs- presence of COVID-19 symptoms, history of international travel, age 60 years and above, presence of comorbidities and oxygen saturation < 94%- were stratified as high-risk and subjected to molecular testing for Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2). Data was obtained from the screening record book patterned after a modified screening tool for COVID-19, deidentified and entered into IBM-SPSS version 25.0. Binary logistic regression was conducted to determine significant predictors of a positive SARS-CoV-2 test. The level of significance was set at p < 0.05. In total, 1,624 patients were screened. Mean age (standard deviation) was 53.9±18.0 years and 651 (40.1%) were 60 years and above. One or more symptoms of COVID-19 were present in 586 (36.1%) patients. Overall, 1,116 (68.7%) patients were designated high risk and tested for SARS-CoV-2, of which 359 (32.2%) were positive. Additional inputs, besides symptoms, increased COVID-19 detection by 108%. Predictors of a positive test were elderly age [AOR = 1.545 (1.127-2.116)], co-morbidity [AOR = 1.811 (1.296-2.530)] and oxygen saturation [AOR = 3.427 (2.595-4.528)]. This protocol using additional inputs such as oxygen saturation improved upon symptoms-based screening for COVID-19. Models incorporating identified predictors will be invaluable in resource limited settings.

Dried blood spot specimens for SARS-CoV-2 antibody testing: A multi-site, multi-assay comparison

The true severity of infection due to COVID-19 is under-represented because it is based on only those who are tested. Although nucleic acid amplifications tests (NAAT) are the gold standard for COVID-19 diagnostic testing, serological assays provide better population-level SARS-CoV-2 prevalence estimates. Implementing large sero-surveys present several logistical challenges within Canada due its unique geography including rural and remote communities. Dried blood spot (DBS) sampling is a practical solution but comparative performance data on SARS-CoV-2 serological tests using DBS is currently lacking. Here we present test performance data from a well-characterized SARS-CoV-2 DBS panel sent to laboratories across Canada representing 10 commercial and 2 in-house developed tests for SARS-CoV-2 antibodies. Three commercial assays identified all positive and negative DBS correctly corresponding to a sensitivity, specificity, positive predictive value, and negative predictive value of 100% (95% CI = 72.2, 100). Two in-house assays also performed equally well. In contrast, several commercial assays could not achieve a sensitivity greater than 40% or a negative predictive value greater than 60%. Our findings represent the foundation for future validation studies on DBS specimens that will play a central role in strengthening Canada's public health policy in response to COVID-19.

Detection of COVID-19 Using Heart Rate and Blood Pressure: Lessons Learned from Patients with ARDS

The world has been affected by COVID-19 coronavirus. At the time of this study, the number of infected people in the United States is the highest globally (31.2 million infections). Within the infected population, patients diagnosed with acute respiratory distress syndrome (ARDS) are in more life-threatening circumstances, resulting in severe respiratory system failure. Various studies have investigated the infections to COVID-19 and ARDS by monitoring laboratory metrics and symptoms. Unfortunately, these methods are merely limited to clinical settings, and symptom-based methods are shown to be ineffective. In contrast, vital signs (e.g., heart rate) have been utilized to early-detect different respiratory diseases in ubiquitous health monitoring. We posit that such biomarkers are informative in identifying ARDS patients infected with COVID-19. In this study, we investigate the behavior of COVID-19 on ARDS patients by utilizing simple vital signs. We analyze the long-term daily logs of blood pressure (BP) and heart rate (HR) associated with 150 ARDS patients admitted to five University of California academic health centers (containing 77,972 samples for each vital sign) to distinguish subjects with COVID-19 positive and negative test results. In addition to the statistical analysis, we develop a deep neural network model to extract features from the longitudinal data. Our deep learning model is able to achieve 0.81 area under the curve (AUC) to classify the vital signs of ARDS patients infected with COVID-19 versus other ARDS diagnosed patients. Since our proposed model uses only the BP and HR, it would be possible to review data prior to the first reported cases in the U.S. to validate the presence or absence of COVID-19 in our communities prior to January 2020. In addition, by utilizing wearable devices, and monitoring vital signs of subjects in everyday settings it is possible to early-detect COVID-19 without visiting a hospital or a care site.

Coronavirus Disease 2019 Symptoms and Severe Acute Respiratory Syndrome Coronavirus 2 Antibody Positivity in a Large Survey of First Responders and Healthcare Personnel, May-July 2020

A SARS-CoV-2 serosurvey among first responder/healthcare personnel showed that loss of taste/smell was most predictive of seropositivity; percent seropositivity increased with number of COVID-19 symptoms. However, 22.9% with nine symptoms were seronegative, and 8.3% with no symptoms were seropositive. These findings demonstrate limitations of symptom-based surveillance and importance of testing.

The Evolution of Clinical Knowledge During COVID-19: Towards a Global Learning Health System

OBJECTIVES

We examine the knowledge ecosystem of COVID-19, focusing on clinical knowledge and the role of health informatics as enabling technology. We argue for commitment to the model of a global learning health system to facilitate rapid knowledge translation supporting health care decision making in the face of emerging diseases.

METHODS AND RESULTS

We frame the evolution of knowledge in the COVID-19 crisis in terms of learning theory, and present a view of what has occurred during the pandemic to rapidly derive and share knowledge as an (underdeveloped) instance of a global learning health system. We identify the key role of information technologies for electronic data capture and data sharing, computational modelling, evidence synthesis, and knowledge dissemination. We further highlight gaps in the system and barriers to full realisation of an efficient and effective global learning health system.

CONCLUSIONS

The need for a global knowledge ecosystem supporting rapid learning from clinical practice has become more apparent than ever during the COVID-19 pandemic. Continued effort to realise the vision of a global learning health system, including establishing effective approaches to data governance and ethics to support the system, is imperative to enable continuous improvement in our clinical care.

Artificial Intelligence in Action: Addressing the COVID-19 Pandemic with Natural Language Processing

The COVID-19 (coronavirus disease 2019) pandemic has had a significant impact on society, both because of the serious health effects of COVID-19 and because of public health measures implemented to slow its spread. Many of these difficulties are fundamentally information needs; attempts to address these needs have caused an information overload for both researchers and the public. Natural language processing (NLP)-the branch of artificial intelligence that interprets human language-can be applied to address many of the information needs made urgent by the COVID-19 pandemic. This review surveys approximately 150 NLP studies and more than 50 systems and datasets addressing the COVID-19 pandemic. We detail work on four core NLP tasks: information retrieval, named entity recognition, literature-based discovery, and question answering. We also describe work that directly addresses aspects of the pandemic through four additional tasks: topic modeling, sentiment and emotion analysis, caseload forecasting, and misinformation detection. We conclude by discussing observable trends and remaining challenges.

Use of connected digital products in clinical research following the COVID-19 pandemic: a comprehensive analysis of clinical trials

OBJECTIVES

In an effort to mitigate COVID-19 related challenges for clinical research, the US Food and Drug Administration (FDA) issued new guidance for the conduct of 'virtual' clinical trials in late March 2020. This study documents trends in the use of connected digital products (CDPs), tools that enable remote patient monitoring and telehealth consultation, in clinical trials both before and after the onset of the pandemic.

DESIGN

We applied a comprehensive text search algorithm to clinical trial registry data to identify trials that use CDPs for remote monitoring or telehealth. We compared CDP use in the months before and after the issuance of FDA guidance facilitating virtual clinical trials.

SETTING

All trials registered on ClinicalTrials.gov with start dates from May 2019 through February 2021.

OUTCOME MEASURES

The primary outcome measure was the overall percentage of CDP use in clinical trials started in the 10 months prior to the pandemic onset (May 2019-February 2020) compared with the 10 months following (May 2020-February 2021). Secondary outcome measures included CDP usage by trial type (interventional, observational), funder type (industry, non-industry) and diagnoses (COVID-19 or non-COVID-19 participants).

RESULTS

CDP usage in clinical trials increased by only 1.65 percentage points, from 14.19% (n=23 473) of all trials initiated in the 10 months prior to the pandemic onset to 15.84% (n=26 009) of those started in the 10 months following (p<0.01). The increase occurred primarily in observational studies and non-industry funded trials and was driven entirely by CDP usage in trials for COVID-19.

CONCLUSIONS

These findings suggest that in the short-term, new options created by regulatory guidance to stimulate telehealth and remote monitoring were not widely incorporated into clinical research. In the months immediately following the pandemic onset, CDP adoption increased primarily in observational and non-industry funded studies where virtual protocols are likely medically necessary due to the participants' COVID-19 diagnosis.

DP, Sanchez A, El Saddik A, Petriu E. A Machine Learning Approach as an Aid for Early COVID-19 Detection

The novel coronavirus SARS-CoV-2 that causes the disease COVID-19 has forced us to go into our homes and limit our physical interactions with others. Economies around the world have come to a halt, with non-essential businesses being forced to close in order to prevent further propagation of the virus. Developing countries are having more difficulties due to their lack of access to diagnostic resources. In this study, we present an approach for detecting COVID-19 infections exclusively on the basis of self-reported symptoms. Such an approach is of great interest because it is relatively inexpensive and easy to deploy at either an individual or population scale. Our best model delivers a sensitivity score of 0.752, a specificity score of 0.609, and an area under the curve for the receiver operating characteristic of 0.728. These are promising results that justify continuing research efforts towards a machine learning test for detecting COVID-19.

A Caveat to Using Wearable Sensor Data for COVID-19 Detection: The Role of Behavioral Change after Receipt of Test Results

Recent studies indicate that wearable sensors have the potential to capture subtle within-person changes that signal SARS-CoV-2 infection. However, it remains unclear the extent to which observed discriminative performance is attributable to behavioral change after receiving test results. We conducted a retrospective study in a sample of medical interns who received COVID-19 test results from March to December 2020. Our data confirmed that sensor data were able to differentiate between symptomatic COVID-19 positive and negative individuals with good accuracy (area under the curve (AUC) = 0.75). However, removing post-result data substantially reduced discriminative capacity (0.75 to 0.63; delta= -0.12, p=0.013). Removing data in the symptomatic period prior to receipt of test results did not produce similar reductions in discriminative capacity. These findings suggest a meaningful proportion of the discriminative capacity of wearable sensor data for SARS-CoV-2 infection may be due to behavior change after receiving test results.

Ontology-driven weak supervision for clinical entity classification in electronic health records

In the electronic health record, using clinical notes to identify entities such as disorders and their temporality (e.g. the order of an event relative to a time index) can inform many important analyses. However, creating training data for clinical entity tasks is time consuming and sharing labeled data is challenging due to privacy concerns. The information needs of the COVID-19 pandemic highlight the need for agile methods of training machine learning models for clinical notes. We present Trove, a framework for weakly supervised entity classification using medical ontologies and expert-generated rules. Our approach, unlike hand-labeled notes, is easy to share and modify, while offering performance comparable to learning from manually labeled training data. In this work, we validate our framework on six benchmark tasks and demonstrate Trove's ability to analyze the records of patients visiting the emergency department at Stanford Health Care for COVID-19 presenting symptoms and risk factors.

Understanding On-Campus Interactions With a Semiautomated, Barcode-Based Platform to Augment COVID-19 Contact Tracing: App Development and Usage

BACKGROUND

The COVID-19 pandemic has forced drastic changes to daily life, from the implementation of stay-at-home orders to mandating facial coverings and limiting in-person gatherings. While the relaxation of these control measures has varied geographically, it is widely agreed that contact tracing efforts will play a major role in the successful reopening of businesses and schools. As the volume of positive cases has increased in the United States, it has become clear that there is room for digital health interventions to assist in contact tracing.

OBJECTIVE

The goal of this study was to evaluate the use of a mobile-friendly app designed to supplement manual COVID-19 contact tracing efforts on a university campus. Here, we present the results of a development and validation study centered around the use of the MyCOVIDKey app on the Vanderbilt University campus during the summer of 2020.

METHODS

We performed a 6-week pilot study in the Stevenson Center Science and Engineering Complex on Vanderbilt University's campus in Nashville, TN. Graduate students, postdoctoral fellows, faculty, and staff >18 years who worked in Stevenson Center and had access to a mobile phone were eligible to register for a MyCOVIDKey account. All users were encouraged to complete regular self-assessments of COVID-19 risk and to key in to sites by scanning a location-specific barcode.

RESULTS

Between June 17, 2020, and July 29, 2020, 45 unique participants created MyCOVIDKey accounts. These users performed 227 self-assessments and 1410 key-ins. Self-assessments were performed by 89% (n=40) of users, 71% (n=32) of users keyed in, and 48 unique locations (of 71 possible locations) were visited. Overall, 89% (202/227) of assessments were determined to be low risk (ie, asymptomatic with no known exposures), and these assessments yielded a CLEAR status. The remaining self-assessments received a status of NOT CLEAR, indicating either risk of exposure or symptoms suggestive of COVID-19 (7.5% [n=17] and 3.5% [n=8] of self-assessments indicated moderate and high risk, respectively). These 25 instances came from 8 unique users, and in 19 of these instances, the at-risk user keyed in to a location on campus.

CONCLUSIONS

Digital contact tracing tools may be useful in assisting organizations to identify persons at risk of COVID-19 through contact tracing, or in locating places that may need to be cleaned or disinfected after being visited by an index case. Incentives to continue the use of such tools can improve uptake, and their continued usage increases utility to both organizational and public health efforts. Parameters of digital tools, including MyCOVIDKey, should ideally be optimized to supplement existing contact tracing efforts. These tools represent a critical addition to manual contact tracing efforts during reopening and sustained regular activity.

App-based symptom tracking to optimize SARS-CoV-2 testing strategy using machine learning

BACKGROUND

Tests are scarce resources, especially in low and middle-income countries, and the optimization of testing programs during a pandemic is critical for the effectiveness of the disease control. Hence, we aim to use the combination of symptoms to build a predictive model as a screening tool to identify people and areas with a higher risk of SARS-CoV-2 infection to be prioritized for testing.

MATERIALS AND METHODS

We performed a retrospective analysis of individuals registered in "Dados do Bem," a Brazilian app-based symptom tracker. We applied machine learning techniques and provided a SARS-CoV-2 infection risk map of Rio de Janeiro city.

RESULTS

From April 28 to July 16, 2020, 337,435 individuals registered their symptoms through the app. Of these, 49,721 participants were tested for SARS-CoV-2 infection, being 5,888 (11.8%) positive. Among self-reported symptoms, loss of smell (OR[95%CI]: 4.6 [4.4-4.9]), fever (2.6 [2.5-2.8]), and shortness of breath (2.1 [1.6-2.7]) were independently associated with SARS-CoV-2 infection. Our final model obtained a competitive performance, with only 7% of false-negative users predicted as negatives (NPV = 0.93). The model was incorporated by the "Dados do Bem" app aiming to prioritize users for testing. We developed an external validation in the city of Rio de Janeiro. We found that the proportion of positive results increased significantly from 14.9% (before using our model) to 18.1% (after the model).

CONCLUSIONS

Our results showed that the combination of symptoms might predict SARS-Cov-2 infection and, therefore, can be used as a tool by decision-makers to refine testing and disease control strategies.

Evaluating evaluation frameworks: a scoping review of frameworks for assessing health apps

OBJECTIVES

Despite an estimated 300 000 mobile health apps on the market, there remains no consensus around helping patients and clinicians select safe and effective apps. In 2018, our team drew on existing evaluation frameworks to identify salient categories and create a new framework endorsed by the American Psychiatric Association (APA). We have since created a more expanded and operational framework Mhealth Index and Navigation Database (MIND) that aligns with the APA categories but includes objective and auditable questions (105). We sought to survey the existing space, conducting a review of all mobile health app evaluation frameworks published since 2018, and demonstrate the comprehensiveness of this new model by comparing it to existing and emerging frameworks.

DESIGN

We conducted a scoping review of mobile health app evaluation frameworks.

DATA SOURCES

References were identified through searches of PubMed, EMBASE and PsychINFO with publication date between January 2018 and October 2020.

ELIGIBILITY CRITERIA

Papers were selected for inclusion if they meet the predetermined eligibility criteria-presenting an evaluation framework for mobile health apps with patient, clinician or end user-facing questions.

DATA EXTRACTION AND SYNTHESIS

Two reviewers screened the literature separately and applied the inclusion criteria. The data extracted from the papers included: author and dates of publication, source affiliation, country of origin, name of framework, study design, description of framework, intended audience/user and framework scoring system. We then compiled a collection of more than 1701 questions across 79 frameworks. We compared and grouped these questions using the MIND framework as a reference. We sought to identify the most common domains of evaluation while assessing the comprehensiveness and flexibility-as well as any potential gaps-of MIND.

RESULTS

New app evaluation frameworks continue to emerge and expand. Since our 2019 review of the app evaluation framework space, more frameworks include questions around privacy (43) and clinical foundation (57), reflecting an increased focus on issues of app security and evidence base. The majority of mapped frameworks overlapped with at least half of the MIND categories. The results of this search have informed a database (apps.digitalpsych.org) that users can access today.

CONCLUSION

As the number of app evaluation frameworks continues to rise, it is becoming difficult for users to select both an appropriate evaluation tool and to find an appropriate health app. This review provides a comparison of what different app evaluation frameworks are offering, where the field is converging and new priorities for improving clinical guidance.

Lyme Disease in the Era of COVID-19: A Delayed Diagnosis and Risk for Complications

We describe a patient with fever and myalgia who did not have COVID-19 but instead had Lyme disease. We propose that the co-occurrence of COVID-19 and Lyme disease during the spring of 2020 resulted in a delayed diagnosis of Lyme disease due to COVID-19 pandemic-related changes in healthcare workflow and diagnostic reasoning. This delayed diagnosis of Lyme disease in the patient we describe resulted in disseminated infection and sixth nerve palsy. We present the use of telemedicine to aid in the diagnosis of Lyme disease and to provide prompt access to diagnosis and care during the ongoing COVID-19 pandemic and in the future.

How the clinical research community responded to the COVID-19 pandemic: An analysis of the COVID-19 clinical studies in ClinicalTrials.gov

OBJECTIVE

The novel coronavirus disease (COVID-19), broke out in December 2019, and is now a global pandemic. In the past few months, a large number of clinical studies have been initiated worldwide to find effective therapeutics, vaccines, and preventive strategies for COVID-19. In this study, we aim to understand the landscape of COVID-19 clinical research and identify the gaps such as the lack of population representativeness and issues that may cause recruitment difficulty.

MATERIALS AND METHODS

We analyzed 3,765 COVID-19 studies registered in the largest public registry - ClinicalTrials.gov, leveraging natural language processing and using descriptive, association, and clustering analyses. We first characterized COVID-19 studies by study features such as phase and tested intervention. We then took a deep dive and analyzed their eligibility criteria to understand whether these studies: (1) considered the reported underlying health conditions that may lead to severe illnesses, and (2) excluded older adults, either explicitly or implicitly, which may reduce the generalizability of these studies to the older adults population.

RESULTS

Most trials did not have an upper age limit and did not exclude patients with common chronic conditions such as hypertension and diabetes that are more prevalent in older adults. However, known risk factors that may lead to severe illnesses have not been adequately considered.

CONCLUSIONS

A careful examination of existing COVID-19 studies can inform future COVID-19 trial design towards balanced internal validity and generalizability.

Virus database annotations assist in tracing information on patients infected with emerging pathogens

The global pandemic of SARS-CoV-2 has disrupted human social activities. In restarting economic activities, successive outbreaks by new variants are concerning. Here, we evaluated the applicability of public database annotations to estimate the virulence, transmission trends and origins of emerging SARS-CoV-2 variants. Among the detectable multiple mutations, we retraced the mutation in the spike protein. With the aid of the protein database, structural modelling yielded a testable scientific hypothesis on viral entry to host cells. Simultaneously, annotations for locations and collection dates suggested that the variant virus emerged somewhere in the world in approximately February 2020, entered the USA and propagated nationwide with periodic sampling fluctuation likely due to an approximately 5-day incubation delay. Thus, public database annotations are useful for automated elucidation of the early spreading patterns in relation to human behaviours, which should provide objective reference for local governments for social decision making to contain emerging substrains. We propose that additional annotations for past paths and symptoms of the patients should further assist in characterizing the exact virulence and origins of emerging pathogens.

Ontology-driven weak supervision for clinical entity classification in electronic health records

In the electronic health record, using clinical notes to identify entities such as disorders and their temporality (e.g. the order of an event relative to a time index) can inform many important analyses. However, creating training data for clinical entity tasks is time consuming and sharing labeled data is challenging due to privacy concerns. The information needs of the COVID-19 pandemic highlight the need for agile methods of training machine learning models for clinical notes. We present Trove, a framework for weakly supervised entity classification using medical ontologies and expert-generated rules. Our approach, unlike hand-labeled notes, is easy to share and modify, while offering performance comparable to learning from manually labeled training data. In this work, we validate our framework on six benchmark tasks and demonstrate Trove's ability to analyze the records of patients visiting the emergency department at Stanford Health Care for COVID-19 presenting symptoms and risk factors.