Practical comparison of aberration detection algorithms for biosurveillance systems

Toxicology Test Results for Public Health Surveillance of the Opioid Epidemic: Retrospective Analysis

Background

Addressing the opioid epidemic requires timely insights into population-level factors, such as trends in prevalence of legal and illegal substances, overdoses, and deaths.

Objective

This study aimed to examine whether toxicology test results of living individuals from a variety of sources could be useful in surveilling the opioid epidemic.

Methods

A retrospective analysis standardized, merged, and linked toxicology results from 24 laboratories in Marion County, Indiana, United States, from September 1, 2018, to August 31, 2019. The data set consisted of 33,787 Marion County residents and their 746,681 results. We related the data to general Marion County demographics and compared alerts generated by toxicology results to opioid overdose-related emergency department visits. Nineteen domain experts helped prototype analytical visualizations. Main outcome measures included test positivity in the county and by ZIP code; selected demographics of individuals with toxicology results; and correlation of toxicology results with opioid overdose-related emergency department visits.

Results

Four percent of Marion County residents had at least 1 toxicology result. Test positivity rates ranged from 3% to 19% across ZIP codes. Males were underrepresented in the data set. Age distribution resembled that of Marion County. Alerts for opioid toxicology results were not correlated with opioid overdose-related emergency department visits.

Conclusions

Analyzing toxicology results at scale was impeded by varying data formats, completeness, and representativeness; changes in data feeds; and patient matching difficulties. In this study, toxicology results did not predict spikes in opioid overdoses. Larger, more rigorous and well-controlled studies are needed to assess the utility of toxicology tests in predicting opioid overdose spikes.

Evaluation of the application of sequence data to the identification of outbreaks of disease using anomaly detection methods

Anomaly detection methods have a great potential to assist the detection of diseases in animal production systems. We used sequence data of Porcine Reproductive and Respiratory Syndrome (PRRS) to define the emergence of new strains at the farm level. We evaluated the performance of 24 anomaly detection methods based on machine learning, regression, time series techniques and control charts to identify outbreaks in time series of new strains and compared the best methods using different time series: PCR positives, PCR requests and laboratory requests. We introduced synthetic outbreaks of different size and calculated the probability of detection of outbreaks (POD), sensitivity (Se), probability of detection of outbreaks in the first week of appearance (POD1w) and background alarm rate (BAR). The use of time series of new strains from sequence data outperformed the other types of data but POD, Se, POD1w were only high when outbreaks were large. The methods based on Long Short-Term Memory (LSTM) and Bayesian approaches presented the best performance. Using anomaly detection methods with sequence data may help to identify the emergency of cases in multiple farms, but more work is required to improve the detection with time series of high variability. Our results suggest a promising application of sequence data for early detection of diseases at a production system level. This may provide a simple way to extract additional value from routine laboratory analysis. Next steps should include validation of this approach in different settings and with different diseases.

The effectiveness of syndromic surveillance for the early detection of waterborne outbreaks: a systematic review

BACKGROUND

Waterborne outbreaks are still a risk in high-income countries, and their early detection is crucial to limit their societal consequences. Although syndromic surveillance is widely used for the purpose of detecting outbreaks days earlier than traditional surveillance systems, evidence of the effectiveness of such systems is lacking. Thus, our objective was to conduct a systematic review of the effectiveness of syndromic surveillance to detect waterborne outbreaks.

METHOD

We searched the Cochrane Library, Medline/PubMed, EMBASE, Scopus, and Web of Science for relevant published articles using a combination of the keywords 'drinking water', 'surveillance', and 'waterborne disease' for the period of 1990 to 2018. The references lists of the identified articles for full-text record assessment were screened, and searches in Google Scholar using the same key words were conducted. We assessed the risk of bias in the included articles using the ROBINS-I tool and PRECEPT for the cumulative body of evidence.

RESULTS

From the 1959 articles identified, we reviewed 52 articles, of which 18 met the eligibility criteria. Twelve were descriptive/analytical studies, whereas six were simulation studies. There is no clear evidence for syndromic surveillance in terms of the ability to detect waterborne outbreaks (low sensitivity and high specificity). However, one simulation study implied that multiple sources of signals combined with spatial information may increase the timeliness in detecting a waterborne outbreak and reduce false alarms.

CONCLUSION

This review demonstrates that there is no conclusive evidence on the effectiveness of syndromic surveillance for the detection of waterborne outbreaks, thus suggesting the need to focus on primary prevention measures to reduce the risk of waterborne outbreaks. Future studies should investigate methods for combining health and environmental data with an assessment of needed financial and human resources for implementing such surveillance systems. In addition, a more critical thematic narrative synthesis on the most promising sources of data, and an assessment of the basis for arguments that joint analysis of different data or dimensions of data (e.g. spatial and temporal) might perform better, should be carried out.

TRIAL REGISTRATION

PROSPERO: International prospective register of systematic reviews. 2019. CRD42019122332 .

Exploiting Scanning Surveillance Data to Inform Future Strategies for the Control of Endemic Diseases: The Example of Sheep Scab

Scanning surveillance facilitates the monitoring of many endemic diseases of livestock in Great Britain, including sheep scab, an ectoparasitic disease of major welfare and economic burden. There is, however, a drive to improve the cost-effectiveness of animal health surveillance, for example by thoroughly exploiting existing data sources. By analysing the Veterinary Investigation Diagnosis Analysis (VIDA) database, this study aimed to enhance the use of existing scanning surveillance data for sheep scab to identify current trends, highlighting geographical "hotspots" for targeted disease control measures, and identifying a denominator to aid the interpretation of the diagnostic count data. Furthermore, this study collated and assessed the impact of past targeted disease control initiatives using a temporal aberration detection algorithm, the Farrington algorithm, to provide an evidence base towards developing cost-effective disease control strategies. A total of 2,401 positive skin scrapes were recorded from 2003 to 2018. A statistically significant decline in the number of positive skin scrapes diagnosed (p < 0.001) occurred across the study period, and significant clustering was observed in Wales, with a maximum of 47 positive scrapes in Ceredigion in 2007. Scheduled ectoparasite tests was also identified as a potential denominator for the interpretation of positive scrapes by stakeholders. Across the study period, 11 national disease control initiatives occurred: four in Wales, three in England, and four in Scotland. The majority (n = 8) offered free diagnostic testing while the remainder involved knowledge transfer either combined with free testing or skills training and the introduction of the Sheep Scab (Scotland) Order 2010. The Farrington algorithm raised 20 alarms of which 11 occurred within a period of free testing in Wales and one following the introduction of the Sheep Scab (Scotland) Order 2010. In summary, our analysis of the VIDA database has greatly enhanced our knowledge of sheep scab in Great Britain, firstly by identifying areas for targeted action and secondly by offering a framework to measure the impact of future disease control initiatives. Importantly this framework could be applied to inform future strategies for the control of other endemic diseases.

Alarm Thresholds for Pertussis Outbreaks in Iran: National Data Analysis

Objectives

The purpose of the current study was to determine the upper threshold number of cases for which pertussis infection would reach an outbreak level nationally in Iran.

Methods

Data on suspected cases of pertussis from the 25^th February 2012 to the 23^rd March 2018 from the Center for Disease Control and Prevention in Iran was used. The national upper threshold level was estimated using the exponentially weighted moving average (EWMA) method and the Poisson regression method.

Results

In total, 2,577 (33.6%) and 1,714 (22.3%) cases were reported in the Spring and Summer respectively. There were 1,417 (18.5%) and 1,971 (25.6%) cases reported in the Autumn and Winter, respectively. The overall upper threshold using the EWMA and the Poisson regression methods, was estimated as a daily occurrence of 8 (7.55) and 7.50 (4.48–11.06) suspected cases per 10,000,000 people, respectively. The daily seasonal thresholds estimated by the EWMA and the Poisson regression methods were 10, 7, 6, 8 cases and 10, 7, 7, 9 cases for the Spring, Summer, Autumn, and Winter, respectively.

Conclusion

The overall and seasonal estimated thresholds by the 2 methods were similar. Therefore, the estimated thresholds of 6–10 cases in a day, per 10,000,000 people could be used to detect pertussis outbreaks and epidemics by health policymakers.

Simulation Based Evaluation of Time Series for Syndromic Surveillance of Cattle in Switzerland

Choosing the syndrome time series to monitor in a syndromic surveillance system is not a straight forward process. Defining which syndromes to monitor in order to maximize detection performance has been recently identified as one of the research priorities in Syndromic surveillance. Estimating the minimum size of an epidemic that could potentially be detected in a specific syndrome could be used as a criteria for comparing the performance of different syndrome time series, and could provide some guidance for syndrome selection. The aim of our study was to estimate the potential value of different time series for building a national syndromic surveillance system for cattle in Switzerland. Simulations were used to produce outbreaks of different size and shape and to estimate the ability of each time series and aberration detection algorithm to detect them with high sensitivity, specificity and timeliness. Two temporal aberration detection algorithms were also compared: Holt-Winters generalized exponential smoothing (HW) and Exponential Weighted Moving Average (EWMA). Our results indicated that a specific aberration detection algorithm should be used for each time series. In addition, time series with high counts per unit of time had good overall detection performance, but poor detection performance for small epidemics making them of limited use for an early detection system. Estimating the minimum size of simulated epidemics that could potentially be detected in syndrome TS-event detection pairs can help surveillance system designers choosing the most appropriate syndrome TS to include in their early epidemic surveillance system.

Methods for detecting seasonal influenza epidemics using a school absenteeism surveillance system

Background

School absenteeism data have been collected daily by the public health unit in Wellington-Dufferin-Guelph, Ontario since 2008. To date, a threshold-based approach has been implemented to raise alerts for community-wide and within-school illness outbreaks. We investigate several statistical modelling approaches to using school absenteeism for influenza surveillance at the regional level, and compare their performances using two metrics.

Methods

Daily absenteeism percentages from elementary and secondary schools, and report dates for influenza cases, were obtained from Wellington-Dufferin-Guelph Public Health. Several absenteeism data aggregations were explored, including using the average across all schools or only using schools of one type. A 10% absence threshold, exponentially weighted moving average model, logistic regression with and without seasonality terms, day of week indicators, and random intercepts for school year, and generalized estimating equations were used as epidemic detection methods for seasonal influenza. In the regression models, absenteeism data with various lags were used as predictor variables, and missing values in the datasets used for parameter estimation were handled either by deletion or linear interpolation. The epidemic detection methods were compared using a false alarm rate (FAR) as well as a metric for alarm timeliness.

Results

All model-based epidemic detection methods were found to decrease the FAR when compared to the 10% absence threshold. Regression models outperformed the exponentially weighted moving average model and including seasonality terms and a random intercept for school year generally resulted in fewer false alarms. The best-performing model, a seasonal logistic regression model with random intercept for school year and a day of week indicator where parameters were estimated using absenteeism data that had missing values linearly interpolated, produced a FAR of 0.299, compared to the pre-existing threshold method which at best gave a FAR of 0.827.

Conclusions

School absenteeism can be a useful tool for alerting public health to upcoming influenza epidemics in Wellington-Dufferin-Guelph. Logistic regression with seasonality terms and a random intercept for school year was effective at maximizing true alarms while minimizing false alarms on historical data from this region.

A systematic review of aberration detection algorithms used in public health surveillance

The algorithms used for detecting anomalies have evolved substantially over the last decade to take advantage of advances in informatics and to accommodate changes in surveillance data. We identified 145 studies since 2007 that evaluated statistical methods used to detect aberrations in public health surveillance data. For each study, we classified the analytic methods and reviewed the evaluation metrics. We also summarized the practical usage of the detection algorithms in public health surveillance systems worldwide. Traditional methods (e.g., control charts, linear regressions) were the focus of most evaluation studies and continue to be used commonly in practice. There was, however, an increase in the number of studies using forecasting methods and studies applying machine learning methods, hidden Markov models, and Bayesian framework to multivariate datasets. Evaluation studies demonstrated improved accuracy with more sophisticated methods, but these methods do not appear to be used widely in public health practice.

Equine syndromic surveillance in Colorado using veterinary laboratory testing order data

Introduction

The Risk Identification Unit (RIU) of the US Dept. of Agriculture’s Center for Epidemiology and Animal Health (CEAH) conducts weekly surveillance of national livestock health data and routine coordination with agricultural stakeholders. As part of an initiative to increase the number of species, health issues, and data sources monitored, CEAH epidemiologists are building a surveillance system based on weekly syndromic counts of laboratory test orders in consultation with Colorado State University laboratorians and statistical analysts from the Johns Hopkins University Applied Physics Laboratory. Initial efforts focused on 12 years of equine test records from three state labs. Trial syndrome groups were formed based on RIU experience and published literature. Exploratory analysis, stakeholder input, and laboratory workflow details were needed to modify these groups and filter the corresponding data to eliminate alerting bias. Customized statistical detection methods were sought for effective monitoring based on specialized laboratory information characteristics and on the likely presentation and animal health significance of diseases associated with each syndrome.

Methods

Data transformation and syndrome formation focused on test battery type, test name, submitter source organization, and specimen type. We analyzed time series of weekly counts of tests included in candidate syndrome groups and conducted an iterative process of data analysis and veterinary consultation for syndrome refinement and record filters. This process produced a rule set in which records were directly classified into syndromes using only test name when possible, and otherwise, the specimen type or related body system was used with test name to determine the syndrome. Test orders associated with government regulatory programs, veterinary teaching hospital testing protocols, or research projects, rather than clinical concerns, were excluded. We constructed a testbed for sets of 1000 statistical trials and applied a stochastic injection process assuming lognormally distributed incubation periods to choose an alerting algorithm with the syndrome-required sensitivity and an alert rate within the specified acceptable range for each resulting syndrome. Alerting performance of the EARS C3 algorithm traditionally used by CEAH was compared to modified C2, CuSUM, and EWMA methods, with and without outlier removal and adjustments for the total weekly number of non-mandatory tests.

Results

The equine syndrome groups adopted for monitoring were abortion/reproductive, diarrhea/GI, necropsy, neurological, respiratory, systemic fungal, and tickborne. Data scales, seasonality, and variance differed widely among the weekly time series. Removal of mandatory and regulatory tests reduced weekly observed counts significantly—by >80% for diarrhea/GI syndrome. The RIU group studied outcomes associated with each syndrome and called for detection of single-week signals for most syndromes with expected false-alert intervals >8 and <52 weeks, 8-week signals for neurological and tickborne monitoring (requiring enhanced sensitivity), 6-week signals for respiratory, and 4-week signals for systemic fungal. From the test-bed trials, recommended methods, settings and thresholds were derived.

Conclusions

Understanding of laboratory submission sources, laboratory workflow, and of syndrome-related outcomes are crucial to form syndrome groups for routine monitoring without artifactual alerting. Choices of methods, parameters, and thresholds varied by syndrome and depended strongly on veterinary epidemiologist-specified performance requirements.

Choosing the best algorithm for event detection based on the intended application: A conceptual framework for syndromic surveillance

There is an extensive list of methods available for the early detection of an epidemic signal in syndromic surveillance data. However, there is no commonly accepted classification system for the statistical methods used for event detection in syndromic surveillance. Comparing and choosing appropriate event detection algorithms is an increasingly challenging task. Although lists of selection criteria, and statistical methods used for signal detection have been reported, selection criteria are rarely linked to a specific set of appropriate statistical methods. The paper presents a practical approach for guiding surveillance practitioners to make an informed choice from among the most popular event detection algorithms based on the intended application of the algorithm. We developed selection criteria by mapping the assumptions and performance characteristics of event detection algorithms directly to important characteristics of the time series used in syndromic surveillance. We also considered types of epidemics that may be expected and other characteristics of the surveillance system. These guidelines will provide decisions makers, data analysts, public health practitioners, and researchers with a comprehensive but practical overview of the domain, which may reduce the technical barriers to the development and implementation of syndromic surveillance systems in animal and human health. The classification scheme was restricted to univariate and temporal methods because they are the most commonly used algorithms in syndromic surveillance.

A methodological framework for the evaluation of syndromic surveillance systems: a case study of England

BACKGROUND

Syndromic surveillance complements traditional public health surveillance by collecting and analysing health indicators in near real time. The rationale of syndromic surveillance is that it may detect health threats faster than traditional surveillance systems permitting more timely, and hence potentially more effective public health action. The effectiveness of syndromic surveillance largely relies on the methods used to detect aberrations. Very few studies have evaluated the performance of syndromic surveillance systems and consequently little is known about the types of events that such systems can and cannot detect.

METHODS

We introduce a framework for the evaluation of syndromic surveillance systems that can be used in any setting based upon the use of simulated scenarios. For a range of scenarios this allows the time and probability of detection to be determined and uncertainty is fully incorporated. In addition, we demonstrate how such a framework can model the benefits of increases in the number of centres reporting syndromic data and also determine the minimum size of outbreaks that can or cannot be detected. Here, we demonstrate its utility using simulations of national influenza outbreaks and localised outbreaks of cryptosporidiosis.

RESULTS

Influenza outbreaks are consistently detected with larger outbreaks being detected in a more timely manner. Small cryptosporidiosis outbreaks (<1000 symptomatic individuals) are unlikely to be detected. We also demonstrate the advantages of having multiple syndromic data streams (e.g. emergency attendance data, telephone helpline data, general practice consultation data) as different streams are able to detect different outbreak types with different efficacy (e.g. emergency attendance data are useful for the detection of pandemic influenza but not for outbreaks of cryptosporidiosis). We also highlight that for any one disease, the utility of data streams may vary geographically, and that the detection ability of syndromic surveillance varies seasonally (e.g. an influenza outbreak starting in July is detected sooner than one starting later in the year). We argue that our framework constitutes a useful tool for public health emergency preparedness in multiple settings.

CONCLUSIONS

The proposed framework allows the exhaustive evaluation of any syndromic surveillance system and constitutes a useful tool for emergency preparedness and response.

Comparing the historical limits method with regression models for weekly monitoring of national notifiable diseases reports

To compare the performance of the standard Historical Limits Method (HLM), with a modified HLM (MHLM), the Farrington-like Method (FLM), and the Serfling-like Method (SLM) in detecting simulated outbreak signals. We used weekly time series data from 12 infectious diseases from the U.S. Centers for Disease Control and Prevention's National Notifiable Diseases Surveillance System (NNDSS). Data from 2006 to 2010 were used as baseline and from 2011 to 2014 were used to test the four detection methods. MHLM outperformed HLM in terms of background alert rate, sensitivity, and alerting delay. On average, SLM and FLM had higher sensitivity than MHLM. Among the four methods, the FLM had the highest sensitivity and lowest background alert rate and alerting delay. Revising or replacing the standard HLM may improve the performance of aberration detection for NNDSS standard weekly reports.

A Practitioner-Driven Research Agenda for Syndromic Surveillance

Syndromic surveillance has expanded since 2001 in both scope and geographic reach and has benefited from research studies adapted from numerous disciplines. The practice of syndromic surveillance continues to evolve rapidly. The International Society for Disease Surveillance solicited input from its global surveillance network on key research questions, with the goal of improving syndromic surveillance practice. A workgroup of syndromic surveillance subject matter experts was convened from February to June 2016 to review and categorize the proposed topics. The workgroup identified 12 topic areas in 4 syndromic surveillance categories: informatics, analytics, systems research, and communications. This article details the context of each topic and its implications for public health. This research agenda can help catalyze the research that public health practitioners identified as most important.