Assessing surveillance using sensitivity, specificity and timeliness

Comparison of statistical methods for the early detection of disease outbreaks in small population settings

Objectives

This study examines the performance of 6 aberration detection algorithms for the early detection of disease outbreaks in small population settings using syndrome-based early warning surveillance data collected by the Pacific Syndromic Surveillance System (PSSS). Although previous studies have proposed statistical methods for detecting aberrations in larger datasets, there is limited knowledge about how these perform in the presence of small numbers of background cases.

Methods

To address this gap a simulation model was developed to test and compare the performance of the 6 algorithms in detecting outbreaks of different magnitudes, durations, and case distributions.

Results

The study found that while the Early Aberration Reporting System-C1 algorithm developed by Hutwagner et al. outperformed others, no single approach provided reliable monitoring across all outbreak types. Furthermore, aberration detection approaches could only detect very large and acute outbreaks with any reliability.

Conclusion

The findings of this study suggest that algorithm-based approaches to outbreak signal detection perform poorly when applied to settings with small numbers of background cases and should not be relied upon in these contexts. This highlights the need for alternative approaches for accurate and timely outbreak detection in small population settings, particularly those that are resource-constrained.

ATQ: alert time quality, an evaluation metric for assessing timely epidemic detection models within a school absenteeism-based surveillance system

BACKGROUND

Wellington-Dufferin-Guelph Public Health (WDGPH) has conducted an absenteeism-based influenza surveillance program in the WDG region of Ontario, Canada since 2008, using a 10% absenteeism threshold to raise an alert for the implementation of mitigating measures. A recent study indicated that model-based alternatives, such as distributed lag seasonal logistic regression models, provided improved alerts for detecting an upcoming epidemic. However model evaluation and selection was primarily based on alert accuracy, measured by the false alert rate (FAR), and failed to optimize timeliness. Here, a new metric that simultaneously evaluates epidemic alert accuracy and timeliness is proposed. The alert time quality (ATQ) metric is investigated as a model selection criterion on both a simulated and real data set.

METHODS

The ATQ assessed alerts on a gradient, where alerts raised incrementally before or after an optimal day were considered informative, but were penalized for lack of timeliness. Summary statistics of ATQ, average alert time quality (AATQ) and first alert time quality (FATQ), were used for model evaluation and selection. Alerts raised by ATQ and FAR selected models were compared. Daily elementary school absenteeism and laboratory-confirmed influenza case data collected by WDGPH were used for demonstration and evaluation of the proposed metric. A simulation study that mimicked the WDG population and influenza demographics was conducted for further evaluation of the proposed metric.

RESULTS

The FATQ-selected model raised acceptable first alerts most frequently, while the AATQ-selected model raised first alerts within the ideal range most frequently.

CONCLUSIONS

Models selected by either FATQ or AATQ would more effectively predict community influenza activity with the local community than those selected by FAR.

Using decision fusion methods to improve outbreak detection in disease surveillance

BACKGROUND

When outbreak detection algorithms (ODAs) are considered individually, the task of outbreak detection can be seen as a classification problem and the ODA as a sensor providing a binary decision (outbreak yes or no) for each day of surveillance. When they are considered jointly (in cases where several ODAs analyze the same surveillance signal), the outbreak detection problem should be treated as a decision fusion (DF) problem of multiple sensors.

METHODS

This study evaluated the benefit for a decisions support system of using DF methods (fusing multiple ODA decisions) compared to using a single method of outbreak detection. For each day, we merged the decisions of six ODAs using 5 DF methods (two voting methods, logistic regression, CART and Bayesian network - BN). Classical metrics of accuracy, prediction and timelines were used during the evaluation steps.

RESULTS

In our results, we observed the greatest gain (77%) in positive predictive value compared to the best ODA if we used DF methods with a learning step (BN, logistic regression, and CART).

CONCLUSIONS

To identify disease outbreaks in systems using several ODAs to analyze surveillance data, we recommend using a DF method based on a Bayesian network. This method is at least equivalent to the best of the algorithms considered, regardless of the situation faced by the system. For those less familiar with this kind of technique, we propose that logistic regression be used when a training dataset is available.

A Syndrome-Based Surveillance System for Infectious Diseases Among Asylum Seekers in Austrian Reception Centers, 2015-2018: Analysis of Reported Data

Background

Austria has been among the main European countries hosting incoming asylum seekers since 2015. Consequently, there was an urgent need to predict any public health threats associated with the arriving asylum seekers. The Department of Surveillance and Infectious Disease Epidemiology at the Austrian Agency for Health and Food Safety (AGES) was mandated to implement a national syndrome-based surveillance system in the 7 reception centers by the Austrian Ministry of Interior and Ministry of Health.

Objective

We aimed to analyze the occurrence and spread of infectious diseases among asylum seekers using data reported by reception centers through the syndrome-based surveillance system from September 2015 through February 2018.

Methods

We deployed a daily data collection system for 13 syndromes: rash with fever; rash without fever; acute upper respiratory tract infection; acute lower respiratory tract infection; meningitis or encephalitis; fever and bleeding; nonbloody gastroenteritis or watery diarrhea; bloody diarrhea; acute jaundice; skin, soft tissue, or bone abnormalities; acute flaccid paralysis; high fever with no other signs; and unexplained death. General practitioners, the first professionals to consult for health problems at reception centers in Austria, sent the tally sheets on identified syndromes daily to the AGES.

Results

We identified a total of 2914 cases, presenting 8 of the 13 syndromes. A total of 405 signals were triggered, and 6.4% (26/405) of them generated alerts. Suspected acute upper respiratory tract infection (1470/2914, 50.45% of cases), rash without fever (1174/2914, 40.29% of cases), suspected acute lower respiratory tract infection (159/2914, 5.46% of cases), watery diarrhea (73/2914, 2.51% of cases), and skin, soft tissue, or bone abnormalities (32/2914, 1.10% of cases) were the top 5 syndromes.

Conclusions

The cooperation of the AGES with reception center health care staff, supported by the 2 involved ministries, was shown to be useful for syndromic surveillance of infectious diseases among asylum seekers. None of the identified alerts escalated to an outbreak.

A Systematic Review of Risk Analysis Tools for Differentiating Unnatural From Natural Epidemics

Introduction: In the era of genetic engineering of pathogens, distinguishing unnatural epidemics from natural ones is a challenge. Successful identification of unnatural infectious disease events can assist in rapid response, which relies on a sensitive risk assessment tool used for the early detection of deliberate attacks (i.e., bioterrorism). Methods: A systematic review was conducted according to the outline of Preferred Reporting Items for Systematic Reviews. Published papers related to the detection of unnatural diseases were searched in MEDLINE (January 1927–April 2016), EMBASE (January 1937–March 2016), and Web of Science (January 1978–March 2016). Full texts were reviewed for the selection of studies on scoring systems specially designed to discern between unnatural and natural outbreaks. Results: A total of 1,753 papers were reviewed, of which we identified the following five scoring systems specifically designed for detecting unnatural outbreaks: (1) the Grunow–Finke epidemiological assessment tool, (2) potential epidemiological clues to a deliberate epidemic, (3) bioterrorism risk assessment scoring, (4) and (5) two modified scoring systems based on (3). Various criteria ranging from the information on perpetrators, type of agents, spatial distribution, and intelligence of deliberate release were involved. Of these systems, the Grunow–Finke assessment tool remains the most widely used, but has low sensitivity for correctly identifying unnatural epidemics when tested against actual historical outbreaks. Others were applied into a few scenarios but provided different perspectives for bioterrorism detection and bio-preparedness. Conclusion: There are few risk assessment tools for differentiating unnatural from natural epidemics. These tools are increasingly necessary and valuable, but improved scoring systems with higher sensitivity, specificity, timeliness, and wider application to biological attacks must be developed.

Performances of statistical methods for the detection of seasonal influenza epidemics using a consensus-based gold standard

Influenza epidemics are monitored using influenza-like illness (ILI) data reported by health-care professionals. Timely detection of the onset of epidemics is often performed by applying a statistical method on weekly ILI incidence estimates with a large range of methods used worldwide. However, performance evaluation and comparison of these algorithms is hindered by: (1) the absence of a gold standard regarding influenza epidemic periods and (2) the absence of consensual evaluation criteria. As of now, performance evaluations metrics are based only on sensitivity, specificity and timeliness of detection, since definitions are not clear for time-repeated measurements such as weekly epidemic detection. We aimed to evaluate several epidemic detection methods by comparing their alerts to a gold standard determined by international expert consensus. We introduced new performance metrics that meet important objective of influenza surveillance in temperate countries: to detect accurately the start of the single epidemic period each year. Evaluations are presented using ILI incidence in France between 1995 and 2011. We found that the two performance metrics defined allowed discrimination between epidemic detection methods. In the context of performance detection evaluation, other metrics used commonly than the standard could better achieve the needs of real-time influenza surveillance.

Evaluation and comparison of statistical methods for early temporal detection of outbreaks: A simulation-based study

The objective of this paper is to evaluate a panel of statistical algorithms for temporal outbreak detection. Based on a large dataset of simulated weekly surveillance time series, we performed a systematic assessment of 21 statistical algorithms, 19 implemented in the R package surveillance and two other methods. We estimated false positive rate (FPR), probability of detection (POD), probability of detection during the first week, sensitivity, specificity, negative and positive predictive values and F₁-measure for each detection method. Then, to identify the factors associated with these performance measures, we ran multivariate Poisson regression models adjusted for the characteristics of the simulated time series (trend, seasonality, dispersion, outbreak sizes, etc.). The FPR ranged from 0.7% to 59.9% and the POD from 43.3% to 88.7%. Some methods had a very high specificity, up to 99.4%, but a low sensitivity. Methods with a high sensitivity (up to 79.5%) had a low specificity. All methods had a high negative predictive value, over 94%, while positive predictive values ranged from 6.5% to 68.4%. Multivariate Poisson regression models showed that performance measures were strongly influenced by the characteristics of time series. Past or current outbreak size and duration strongly influenced detection performances.

Optimal sampling strategies for detecting zoonotic disease epidemics

The early detection of disease epidemics reduces the chance of successful introductions into new locales, minimizes the number of infections, and reduces the financial impact. We develop a framework to determine the optimal sampling strategy for disease detection in zoonotic host-vector epidemiological systems when a disease goes from below detectable levels to an epidemic. We find that if the time of disease introduction is known then the optimal sampling strategy can switch abruptly between sampling only from the vector population to sampling only from the host population. We also construct time-independent optimal sampling strategies when conducting periodic sampling that can involve sampling both the host and the vector populations simultaneously. Both time-dependent and -independent solutions can be useful for sampling design, depending on whether the time of introduction of the disease is known or not. We illustrate the approach with West Nile virus, a globally-spreading zoonotic arbovirus. Though our analytical results are based on a linearization of the dynamical systems, the sampling rules appear robust over a wide range of parameter space when compared to nonlinear simulation models. Our results suggest some simple rules that can be used by practitioners when developing surveillance programs. These rules require knowledge of transition rates between epidemiological compartments, which population was initially infected, and of the cost per sample for serological tests.

Outbreaks of zoonoses can have large costs to society through public health and agricultural impacts. Because many zoonoses co-occur in multiple animal populations simultaneously, detection of zoonotic outbreaks can be especially difficult. We evaluated how to design sampling strategies for the early detection of disease outbreaks of vector-borne diseases. We built a framework to integrate epidemiological dynamical models with a sampling process that accounts for budgetary constraints, such as those faced by many management agencies. We illustrate our approach using West Nile virus, a globally-spreading zoonotic arbovirus that has significantly affected North American bird populations. Our results suggest that simple formulas can often make robust predictions about the proper sampling procedure, though we also illustrate how computational methods can be used to extend our framework to more realistic modeling scenarios when these simple predictions break down.

Comparing the accuracy of syndrome surveillance systems in detecting influenza-like illness: GUARDIAN vs. RODS vs. electronic medical record reports

BACKGROUND

A highly sensitive real-time syndrome surveillance system is critical to detect, monitor, and control infectious disease outbreaks, such as influenza. Direct comparisons of diagnostic accuracy of various surveillance systems are scarce.

OBJECTIVE

To statistically compare sensitivity and specificity of multiple proprietary and open source syndrome surveillance systems to detect influenza-like illness (ILI).

METHODS

A retrospective, cross-sectional study was conducted utilizing data from 1122 patients seen during November 1–7, 2009 in the emergency department of a single urban academic medical center. The study compared the Geographic Utilization of Artificial Intelligence in Real-time for Disease Identification and Alert Notification (GUARDIAN) system to the Complaint Coder (CoCo) of the Real-time Outbreak Detection System (RODS), the Symptom Coder (SyCo) of RODS, and to a standardized report generated via a proprietary electronic medical record (EMR) system. Sensitivity, specificity, and accuracy of each classifier's ability to identify ILI cases were calculated and compared to a manual review by a board-certified emergency physician. Chi-square and McNemar's tests were used to evaluate the statistical difference between the various surveillance systems.ResultsThe performance of GUARDIAN in detecting ILI in terms of sensitivity, specificity, and accuracy, as compared to a physician chart review, was 95.5%, 97.6%, and 97.1%, respectively. The EMR-generated reports were the next best system at identifying disease activity with a sensitivity, specificity, and accuracy of 36.7%, 99.3%, and 83.2%, respectively. RODS (CoCo and SyCo) had similar sensitivity (35.3%) but slightly different specificity (CoCo = 98.9%; SyCo = 99.3%). The GUARDIAN surveillance system with its multiple data sources performed significantly better compared to CoCo (χ2 = 130.6, p < 0.05), SyCo (χ2 = 125.2, p < 0.05), and EMR-based reports (χ2 = 121.3, p < 0.05). In addition, similar significant improvements in the accuracy (>12%) and sensitivity (>47%) were observed for GUARDIAN with only chief complaint data as compared to RODS (CoCo and SyCo) and EMR-based reports.

CONCLUSION

In our study population, the GUARDIAN surveillance system, with its ability to utilize multiple data sources from patient encounters and real-time automaticity, demonstrated a more robust performance when compared to standard EMR-based reports and the RODS systems in detecting ILI. More large-scale studies are needed to validate the study findings, and to compare the performance of GUARDIAN in detecting other infectious diseases.

Forecasting influenza epidemics from multi-stream surveillance data in a subtropical city of China

Background

Influenza has been associated with heavy burden of mortality and morbidity in subtropical regions. However, timely forecast of influenza epidemic in these regions has been hindered by unclear seasonality of influenza viruses. In this study, we developed a forecasting model by integrating multiple sentinel surveillance data to predict influenza epidemics in a subtropical city Shenzhen, China.

Methods

Dynamic linear models with the predictors of single or multiple surveillance data for influenza-like illness (ILI) were adopted to forecast influenza epidemics from 2006 to 2012 in Shenzhen. Temporal coherence of these surveillance data with laboratory-confirmed influenza cases was evaluated by wavelet analysis and only the coherent data streams were entered into the model. Timeliness, sensitivity and specificity of these models were also evaluated to compare their performance.

Results

Both influenza virology data and ILI consultation rates in Shenzhen demonstrated a significant annual seasonal cycle (p<0.05) during the entire study period, with occasional deviations observed in some data streams. The forecasting models that combined multi-stream ILI surveillance data generally outperformed the models with single-stream ILI data, by providing more timely, sensitive and specific alerts.

Conclusions

Forecasting models that combine multiple sentinel surveillance data can be considered to generate timely alerts for influenza epidemics in subtropical regions like Shenzhen.

in silico surveillance: evaluating outbreak detection with simulation models

Background

Detecting outbreaks is a crucial task for public health officials, yet gaps remain in the systematic evaluation of outbreak detection protocols. The authors’ objectives were to design, implement, and test a flexible methodology for generating detailed synthetic surveillance data that provides realistic geographical and temporal clustering of cases and use to evaluate outbreak detection protocols.

Methods

A detailed representation of the Boston area was constructed, based on data about individuals, locations, and activity patterns. Influenza-like illness (ILI) transmission was simulated, producing 100 years of in silico ILI data. Six different surveillance systems were designed and developed using gathered cases from the simulated disease data. Performance was measured by inserting test outbreaks into the surveillance streams and analyzing the likelihood and timeliness of detection.

Results

Detection of outbreaks varied from 21% to 95%. Increased coverage did not linearly improve detection probability for all surveillance systems. Relaxing the decision threshold for signaling outbreaks greatly increased false-positives, improved outbreak detection slightly, and led to earlier outbreak detection.

Conclusions

Geographical distribution can be more important than coverage level. Detailed simulations of infectious disease transmission can be configured to represent nearly any conceivable scenario. They are a powerful tool for evaluating the performance of surveillance systems and methods used for outbreak detection.

Performance of cancer cluster Q-statistics for case-control residential histories

Few investigations of health event clustering have evaluated residential mobility, though causative exposures for chronic diseases such as cancer often occur long before diagnosis. Recently developed Q-statistics incorporate human mobility into disease cluster investigations by quantifying space- and time-dependent nearest neighbor relationships. Using residential histories from two cancer case-control studies, we created simulated clusters to examine Q-statistic performance. Results suggest the intersection of cases with significant clustering over their life course, Q(i), with cases who are constituents of significant local clusters at given times, Q(it), yielded the best performance, which improved with increasing cluster size. Upon comparison, a larger proportion of true positives were detected with Kulldorf's spatial scan method if the time of clustering was provided. We recommend using Q-statistics to identify when and where clustering may have occurred, followed by the scan method to localize the candidate clusters. Future work should investigate the generalizability of these findings.

Evaluation of animal and public health surveillance systems: a systematic review

Disease surveillance programmes ought to be evaluated regularly to ensure they provide valuable information in an efficient manner. Evaluation of human and animal health surveillance programmes around the world is currently not standardized and therefore inconsistent. The aim of this systematic review was to review surveillance system attributes and the methods used for their assessment, together with the strengths and weaknesses of existing frameworks for evaluating surveillance in animal health, public health and allied disciplines. Information from 99 articles describing the evaluation of 101 surveillance systems was examined. A wide range of approaches for assessing 23 different system attributes was identified although most evaluations addressed only one or two attributes and comprehensive evaluations were uncommon. Surveillance objectives were often not stated in the articles reviewed and so the reasons for choosing certain attributes for assessment were not always apparent. This has the potential to introduce misleading results in surveillance evaluation. Due to the wide range of system attributes that may be assessed, methods should be explored which collapse these down into a small number of grouped characteristics by focusing on the relationships between attributes and their links to the objectives of the surveillance system and the evaluation. A generic and comprehensive evaluation framework could then be developed consisting of a limited number of common attributes together with several sets of secondary attributes which could be selected depending on the disease or range of diseases under surveillance and the purpose of the surveillance. Economic evaluation should be an integral part of the surveillance evaluation process. This would provide a significant benefit to decision-makers who often need to make choices based on limited or diminishing resources.

Bayesian hierarchical Poisson models with a hidden Markov structure for the detection of influenza epidemic outbreaks

Considerable effort has been devoted to the development of statistical algorithms for the automated monitoring of influenza surveillance data. In this article, we introduce a framework of models for the early detection of the onset of an influenza epidemic which is applicable to different kinds of surveillance data. In particular, the process of the observed cases is modelled via a Bayesian Hierarchical Poisson model in which the intensity parameter is a function of the incidence rate. The key point is to consider this incidence rate as a normal distribution in which both parameters (mean and variance) are modelled differently, depending on whether the system is in an epidemic or non-epidemic phase. To do so, we propose a hidden Markov model in which the transition between both phases is modelled as a function of the epidemic state of the previous week. Different options for modelling the rates are described, including the option of modelling the mean at each phase as autoregressive processes of order 0, 1 or 2. Bayesian inference is carried out to provide the probability of being in an epidemic state at any given moment. The methodology is applied to various influenza data sets. The results indicate that our methods outperform previous approaches in terms of sensitivity, specificity and timeliness.

Veterinary syndromic surveillance: Current initiatives and potential for development

This paper reviews recent progress in the development of syndromic surveillance systems for veterinary medicine. Peer-reviewed and grey literature were searched in order to identify surveillance systems that explicitly address outbreak detection based on systematic monitoring of animal population data, in any phase of implementation. The review found that developments in veterinary syndromic surveillance are focused not only on animal health, but also on the use of animals as sentinels for public health, representing a further step towards One Medicine. The main sources of information are clinical data from practitioners and laboratory data, but a number of other sources are being explored. Due to limitations inherent in the way data on animal health is collected, the development of veterinary syndromic surveillance initially focused on animal health data collection strategies, analyzing historical data for their potential to support systematic monitoring, or solving problems of data classification and integration. Systems based on passive notification or data transfers are now dealing with sustainability issues. Given the ongoing barriers in availability of data, diagnostic laboratories appear to provide the most readily available data sources for syndromic surveillance in animal health. As the bottlenecks around data source availability are overcome, the next challenge is consolidating data standards for data classification, promoting the integration of different animal health surveillance systems, and also the integration to public health surveillance. Moreover, the outputs of systems for systematic monitoring of animal health data must be directly connected to real-time decision support systems which are increasingly being used for disease management and control.

A method for selecting and monitoring medication sales for surveillance of gastroenteritis

PURPOSE

Monitoring appropriate categories of medication sales can provide early warning of certain disease outbreaks. This paper presents a methodology for choosing and monitoring medication sales relevant for the surveillance of gastroenteritis and assesses the operational characteristics of the selected medications for early warning.

METHODS

Acute diarrhoea incidences in mainland France were obtained from the Sentinelles network surveillance system for the period 2000-2009. Medication sales grouped by therapeutic classes were obtained on the same period. Hierarchical clustering was used to select therapeutic classes correlating with disease incidence over the period. Alert thresholds were defined for the selected therapeutic classes. Single and multiple voter algorithms were investigated for outbreak detection based on sales crossing the thresholds. Sensitivity and specificity were calculated respective to known outbreaks periods.

RESULTS

Four therapeutic classes were found to cluster with acute diarrhoea incidence. The therapeutic class other antiemetic and antinauseants had the best sensitivity (100%) and timeliness (1.625 weeks before official alerts), for a false alarm rate of 5%. Multiple voter algorithm was the most efficient with the rule: 'Emit an outbreak alert when at least three therapeutic classes are over their threshold' (sensitivity 100%, specificity 95%, timeliness 1.750 weeks before official alerts).

CONCLUSIONS

The presented method allowed selection of relevant therapeutic classes for surveillance of a specific condition. Multiple voter algorithm based on several therapeutic classes performed slightly better than the best therapeutic class alone, while improving robustness against abrupt changes occurring in a single therapeutic class.

Use of outcomes to evaluate surveillance systems for bioterrorist attacks

Background

Syndromic surveillance systems can potentially be used to detect a bioterrorist attack earlier than traditional surveillance, by virtue of their near real-time analysis of relevant data. Receiver operator characteristic (ROC) curve analysis using the area under the curve (AUC) as a comparison metric has been recommended as a practical evaluation tool for syndromic surveillance systems, yet traditional ROC curves do not account for timeliness of detection or subsequent time-dependent health outcomes.

Methods

Using a decision-analytic approach, we predicted outcomes, measured in lives, quality adjusted life years (QALYs), and costs, for a series of simulated bioterrorist attacks. We then evaluated seven detection algorithms applied to syndromic surveillance data using outcomes-weighted ROC curves compared to simple ROC curves and timeliness-weighted ROC curves. We performed sensitivity analyses by varying the model inputs between best and worst case scenarios and by applying different methods of AUC calculation.

Results

The decision analytic model results indicate that if a surveillance system was successful in detecting an attack, and measures were immediately taken to deliver treatment to the population, the lives, QALYs and dollars lost could be reduced considerably. The ROC curve analysis shows that the incorporation of outcomes into the evaluation metric has an important effect on the apparent performance of the surveillance systems. The relative order of performance is also heavily dependent on the choice of AUC calculation method.

Conclusions

This study demonstrates the importance of accounting for mortality, morbidity and costs in the evaluation of syndromic surveillance systems. Incorporating these outcomes into the ROC curve analysis allows for more accurate identification of the optimal method for signaling a possible bioterrorist attack. In addition, the parameters used to construct an ROC curve should be given careful consideration.

Hepatitis A disease following the implementation of universal vaccination: who is at risk?

The incidence of acute hepatitis A in Israel has decreased 25 folds in less than a decade, following the introduction of a two-dose universal toddler's hepatitis A immunization in July 1999. This retrospective study describes demographic data and behavioural determinants of hepatitis A patients following the implementation of a vaccination programme. All records of hepatitis A patients reported to the Ministry of Health during the years 2003 through 2005 were reviewed, and an epidemiological investigation was conducted. During the study period, 420 hepatitis A patients were reported, representing an average annual incidence of two per 100,000 population. Case fatality rate was 0.5%. The majority of the patients were younger than 15 years of age, males and non-Jewish. The highest incidence was recorded in east Jerusalem, where vaccine coverage is relatively low. After exclusion of 165 east Jerusalem patients, 133 (52.2%) patients were available for an interview. Of those, 16 (6%) had possible occupational exposure, 37 (27.8%) travelled to endemic areas, 44 (17%) were contacts of hepatitis A cases, and 3 male patients had sex with men. No known risk determinant was identified in 33 (24.8%) patients. Four patients (3%) were previously immunized with one dose, and none had two doses. The introduction of universal toddler hepatitis A vaccination decreased morbidity. Most of the patients who were detected 4-6 years after the implementation of the vaccination programme could be classified into one of the known risk groups for hepatitis A infection or living in a partly vaccinated community.

Disease surveillance using a hidden Markov model

Background

Routine surveillance of disease notification data can enable the early detection of localised disease outbreaks. Although hidden Markov models (HMMs) have been recognised as an appropriate method to model disease surveillance data, they have been rarely applied in public health practice. We aimed to develop and evaluate a simple flexible HMM for disease surveillance which is suitable for use with sparse small area count data and requires little baseline data.

Methods

A Bayesian HMM was designed to monitor routinely collected notifiable disease data that are aggregated by residential postcode. Semi-synthetic data were used to evaluate the algorithm and compare outbreak detection performance with the established Early Aberration Reporting System (EARS) algorithms and a negative binomial cusum.

Results

Algorithm performance varied according to the desired false alarm rate for surveillance. At false alarm rates around 0.05, the cusum-based algorithms provided the best overall outbreak detection performance, having similar sensitivity to the HMMs and a shorter average time to detection. At false alarm rates around 0.01, the HMM algorithms provided the best overall outbreak detection performance, having higher sensitivity than the cusum-based Methods and a generally shorter time to detection for larger outbreaks. Overall, the 14-day HMM had a significantly greater area under the receiver operator characteristic curve than the EARS C3 and 7-day negative binomial cusum algorithms.

Conclusion

Our findings suggest that the HMM provides an effective method for the surveillance of sparse small area notifiable disease data at low false alarm rates. Further investigations are required to evaluation algorithm performance across other diseases and surveillance contexts.

Understanding detection performance in public health surveillance: modeling aberrancy-detection algorithms

OBJECTIVE

Statistical aberrancy-detection algorithms play a central role in automated public health systems, analyzing large volumes of clinical and administrative data in real-time with the goal of detecting disease outbreaks rapidly and accurately. Not all algorithms perform equally well in terms of sensitivity, specificity, and timeliness in detecting disease outbreaks and the evidence describing the relative performance of different methods is fragmented and mainly qualitative.

DESIGN

We developed and evaluated a unified model of aberrancy-detection algorithms and a software infrastructure that uses this model to conduct studies to evaluate detection performance. We used a task-analytic methodology to identify the common features and meaningful distinctions among different algorithms and to provide an extensible framework for gathering evidence about the relative performance of these algorithms using a number of evaluation metrics. We implemented our model as part of a modular software infrastructure (Biological Space-Time Outbreak Reasoning Module, or BioSTORM) that allows configuration, deployment, and evaluation of aberrancy-detection algorithms in a systematic manner.

MEASUREMENT

We assessed the ability of our model to encode the commonly used EARS algorithms and the ability of the BioSTORM software to reproduce an existing evaluation study of these algorithms.

RESULTS

Using our unified model of aberrancy-detection algorithms, we successfully encoded the EARS algorithms, deployed these algorithms using BioSTORM, and were able to reproduce and extend previously published evaluation results.

CONCLUSION

The validated model of aberrancy-detection algorithms and its software implementation will enable principled comparison of algorithms, synthesis of results from evaluation studies, and identification of surveillance algorithms for use in specific public health settings.

Optimizing use of multistream influenza sentinel surveillance data

We applied time-series methods to multivariate sentinel surveillance data recorded in Hong Kong during 1998–2007. Our study demonstrates that simultaneous monitoring of multiple streams of influenza surveillance data can improve the accuracy and timeliness of alerts compared with monitoring of aggregate data or of any single stream alone.

Assessing the utility of public health surveillance using specificity, sensitivity, and lives saved

In modern surveillance of public health, data may be reported in a timely fashion and include spatial data on cases in addition to the time of their occurrence. This has lead to many recent developments in statistical methods to detect events of public health importance. However, there has been relatively little work about how to compare such methods. One powerful rationale for performing surveillance is earlier detection of events of public health significance; previous evaluation tools have focused on metrics that include the timeliness of detection in addition to sensitivity and specificity. However, such metrics have not accounted for the number of persons affected by the events. We re-examine the rationale for this surveillance and conclude that earlier detection is preferred because it can prevent additional morbidity and mortality. On the basis this observation, we propose evaluating the number of cases prevented by each detection method, and include this information in assessing the value of different detection methods. Using this approach incorporates more information about the events and the detection and provides a sound basis for making decisions about which detection methods to employ.

Applying cusum-based methods for the detection of outbreaks of Ross River virus disease in Western Australia

BACKGROUND

The automated monitoring of routinely collected disease surveillance data has the potential to ensure that important changes in disease incidence are promptly recognised. However, few studies have established whether the signals produced by automated monitoring methods correspond with events considered by epidemiologists to be of public health importance. This study investigates the correspondence between retrospective epidemiological evaluation of notifications of Ross River virus (RRv) disease in Western Australia, and the signals produced by two cumulative sum (cusum)-based automated monitoring methods.

METHODS

RRv disease case notification data between 1991 and 2004 were assessed retrospectively by two experienced epidemiologists, and the timing of identified outbreaks was compared with signals generated from two different types of cusum-based automated monitoring algorithms; the three Early Aberration Reporting System (EARS) cusum algorithms (C1, C2 and C3), and a negative binomial cusum.

RESULTS

We found the negative binomial cusum to have a significantly greater area under the receiver operator characteristic curve when compared with the EARS algorithms, suggesting that the negative binomial cusum has a greater level of agreement with epidemiological opinion than the EARS algorithms with respect to the existence of outbreaks of RRv disease, particularly at low false alarm rates. However, the performance of individual EARS and negative binomial cusum algorithms were not significantly different when timeliness was also incorporated into the area under the curve analyses.

CONCLUSION

Our retrospective analysis of historical data suggests that, compared with the EARS algorithms, the negative binomial cusum provides greater sensitivity for the detection of outbreaks of RRv disease at low false alarm levels, and decreased timeliness early in the outbreak period. Prospective studies are required to investigate the potential usefulness of these algorithms in practice.