Predictive accuracy of particle filtering in dynamic models supporting outbreak projections

Real-Time Epidemiology and Acute Care Need Monitoring and Forecasting for COVID-19 via Bayesian Sequential Monte Carlo-Leveraged Transmission Models

COVID-19 transmission models have conferred great value in informing public health understanding, planning, and response. However, the pandemic also demonstrated the infeasibility of basing public health decision-making on transmission models with pre-set assumptions. No matter how favourably evidenced when built, a model with fixed assumptions is challenged by numerous factors that are difficult to predict. Ongoing planning associated with rolling back and re-instituting measures, initiating surge planning, and issuing public health advisories can benefit from approaches that allow state estimates for transmission models to be continuously updated in light of unfolding time series. A model being continuously regrounded by empirical data in this way can provide a consistent, integrated depiction of the evolving underlying epidemiology and acute care demand, offer the ability to project forward such a depiction in a fashion suitable for triggering the deployment of acute care surge capacity or public health measures, and support quantitative evaluation of tradeoffs associated with prospective interventions in light of the latest estimates of the underlying epidemiology. We describe here the design, implementation, and multi-year daily use for public health and clinical support decision-making of a particle-filtered COVID-19 compartmental model, which served Canadian federal and provincial governments via regular reporting starting in June 2020. The use of the Bayesian sequential Monte Carlo algorithm of particle filtering allows the model to be regrounded daily and adapt to new trends within daily incoming data-including test volumes and positivity rates, endogenous and travel-related cases, hospital census and admissions flows, daily counts of dose-specific vaccinations administered, measured concentration of SARS-CoV-2 in wastewater, and mortality. Important model outputs include estimates (via sampling) of the count of undiagnosed infectives, the count of individuals at different stages of the natural history of frankly and pauci-symptomatic infection, the current force of infection, effective reproductive number, and current and cumulative infection prevalence. Following a brief description of the model design, we describe how the machine learning algorithm of particle filtering is used to continually reground estimates of the dynamic model state, support a probabilistic model projection of epidemiology and health system capacity utilization and service demand, and probabilistically evaluate tradeoffs between potential intervention scenarios. We further note aspects of model use in practice as an effective reporting tool in a manner that is parameterized by jurisdiction, including the support of a scripting pipeline that permits a fully automated reporting pipeline other than security-restricted new data retrieval, including automated model deployment, data validity checks, and automatic post-scenario scripting and reporting. As demonstrated by this multi-year deployment of the Bayesian machine learning algorithm of particle filtering to provide industrial-strength reporting to inform public health decision-making across Canada, such methods offer strong support for evidence-based public health decision-making informed by ever-current articulated transmission models whose probabilistic state and parameter estimates are continually regrounded by diverse data streams.

Estimating the instantaneous reproduction number (Rt) by using particle filter

Background

Monitoring the transmission of coronavirus disease 2019 (COVID-19) requires accurate estimation of the effective reproduction number (R t ). However, existing methods for calculating R t may yield biased estimates if important real-world factors, such as delays in confirmation, pre-symptomatic transmissions, or imperfect data observation, are not considered.

Method

To include real-world factors, we expanded the susceptible-exposed-infectious-recovered (SEIR) model by incorporating pre-symptomatic (P) and asymptomatic (A) states, creating the SEPIAR model. By utilizing both stochastic and deterministic versions of the model, and incorporating predetermined time series of R t , we generated simulated datasets that simulate real-world challenges in estimating R t . We then compared the performance of our proposed particle filtering method for estimating R t with the existing EpiEstim approach based on renewal equations.

Results

The particle filtering method accurately estimated R t even in the presence of data with delays, pre-symptomatic transmission, and imperfect observation. When evaluating via the root mean square error (RMSE) metric, the performance of the particle filtering method was better in general and was comparable to the EpiEstim approach if perfectly deconvolved infection time series were provided, and substantially better when R t exhibited short-term fluctuations and the data was right truncated.

Conclusions

The SEPIAR model, in conjunction with the particle filtering method, offers a reliable tool for predicting the transmission trend of COVID-19 and assessing the impact of intervention strategies. This approach enables enhanced monitoring of COVID-19 transmission and can inform public health policies aimed at controlling the spread of the disease.

Investigating effective testing strategies for the control of Johne's disease in western Canadian cow-calf herds using an agent-based simulation model

Johne's disease is an insidious infectious disease of ruminants caused by Mycobacterium avium subspecies paratuberculosis (MAP). Johne's disease can have important implications for animal welfare and risks causing economic losses in affected herds due to reduced productivity, premature culling and replacement, and veterinary costs. Despite the limited accuracy of diagnostic tools, testing and culling is the primary option for controlling Johne's disease in beef herds. However, evidence to inform specific test and cull strategies is lacking. In this study, a stochastic, continuous-time agent-based model was developed to investigate Johne's disease and potential control options in a typical western Canadian cow-calf herd. The objective of this study was to compare different testing and culling scenarios that included varying the testing method and frequency as well as the number and risk profile of animals targeted for testing using the model. The relative effectiveness of each testing scenario was determined by the simulated prevalence of cattle shedding MAP after a 10-year testing period. A second objective was to compare the direct testing costs of each scenario to identify least-cost options that are the most effective at reducing within-herd disease prevalence. Whole herd testing with individual PCR at frequencies of 6 or 12 months were the most effective options for reducing disease prevalence. Scenarios that were also effective at reducing prevalence but with the lowest total testing costs included testing the whole herd with individual PCR every 24 months and testing the whole herd with pooled PCR every 12 months. The most effective method with the lowest annual testing cost per unit of prevalence reduction was individual PCR on the whole herd every 24 months. Individual PCR testing only cows that had not already been tested 4 times also ranked well when considering both final estimated prevalence at 10 years and cost per unit of gain. A more in-depth economic analysis is needed to compare the cost of testing to the cost of disease, taking into account costs of culling, replacements and impacts on calf crops, and to determine if testing is an economically attractive option for commercial cow-calf operations.

Sound Decision Making in Uncertain Times: Can Systems Modelling Be Useful for Informing Policy and Planning for Suicide Prevention?

The COVID-19 pandemic demonstrated the significant value of systems modelling in supporting proactive and effective public health decision making despite the complexities and uncertainties that characterise an evolving crisis. The same approach is possible in the field of mental health. However, a commonly levelled (but misguided) criticism prevents systems modelling from being more routinely adopted, namely, that the presence of uncertainty around key model input parameters renders a model useless. This study explored whether radically different simulated trajectories of suicide would result in different advice to decision makers regarding the optimal strategy to mitigate the impacts of the pandemic on mental health. Using an existing system dynamics model developed in August 2020 for a regional catchment of Western Australia, four scenarios were simulated to model the possible effect of the COVID-19 pandemic on levels of psychological distress. The scenarios produced a range of projected impacts on suicide deaths, ranging from a relatively small to a dramatic increase. Discordance in the sets of best-performing intervention scenarios across the divergent COVID-mental health trajectories was assessed by comparing differences in projected numbers of suicides between the baseline scenario and each of 286 possible intervention scenarios calculated for two time horizons; 2026 and 2041. The best performing intervention combinations over the period 2021–2041 (i.e., post-suicide attempt assertive aftercare, community support programs to increase community connectedness, and technology enabled care coordination) were highly consistent across all four COVID-19 mental health trajectories, reducing suicide deaths by between 23.9–24.6% against the baseline. However, the ranking of best performing intervention combinations does alter depending on the time horizon under consideration due to non-linear intervention impacts. These findings suggest that systems models can retain value in informing robust decision making despite uncertainty in the trajectories of population mental health outcomes. It is recommended that the time horizon under consideration be sufficiently long to capture the full effects of interventions, and efforts should be made to achieve more timely tracking and access to key population mental health indicators to inform model refinements over time and reduce uncertainty in mental health policy and planning decisions.

Modeling Effects of Spatial Heterogeneities and Layered Exposure Interventions on the Spread of COVID-19 across New Jersey

COVID-19 created an unprecedented global public health crisis during 2020-2021. The severity of the fast-spreading infection, combined with uncertainties regarding the physical and biological processes affecting transmission of SARS-CoV-2, posed enormous challenges to healthcare systems. Pandemic dynamics exhibited complex spatial heterogeneities across multiple scales, as local demographic, socioeconomic, behavioral and environmental factors were modulating population exposures and susceptibilities. Before effective pharmacological interventions became available, controlling exposures to SARS-CoV-2 was the only public health option for mitigating the disease; therefore, models quantifying the impacts of heterogeneities and alternative exposure interventions on COVID-19 outcomes became essential tools informing policy development. This study used a stochastic SEIR framework, modeling each of the 21 New Jersey counties, to capture important heterogeneities of COVID-19 outcomes across the State. The models were calibrated using confirmed daily deaths and SQMC optimization and subsequently applied in predictive and exploratory modes. The predictions achieved good agreement between modeled and reported death data; counterfactual analysis was performed to assess the effectiveness of layered interventions on reducing exposures to SARS-CoV-2 and thereby fatality of COVID-19. The modeling analysis of the reduction in exposures to SARS-CoV-2 achieved through concurrent social distancing and face-mask wearing estimated that 357 [IQR (290, 429)] deaths per 100,000 people were averted.

Environment, Business, and Health Care Prevail: A Comprehensive, Systematic Review of System Dynamics Application Domains

System dynamics, as a methodology for analyzing and understanding various types of systems, has been applied in research for several decades. We undertook a review to identify the latest application domains and map the realm of system dynamics. The systematic review was conducted according to the PRISMA methodology. We analyzed and categorized 212 articles and found that the vast majority of studies belong to the fields of business administration, health, and environmental research. Altogether, 20 groups of modeling and simulation topics can be recognized. System dynamics is occasionally supported by other modeling methodologies such as the agent-based modeling approach. There are issues related to published studies mostly associated with testing of validity and reasonability of models, leading to the development of predictions that are not grounded in verified models. This study contributes to the development of system dynamics as a methodology that can offer new ideas, highlight limitations, or provide analogies for further research in various research disciplines.

Systems Dynamics Modeling of Pandemic Influenza for Strategic Policy Development: a Simulation-Based Analysis of the COVID-19 Case

The novel coronavirus disease 2019 (COVID-19) is a truly wicked problem which has remained a stubborn issue plaguing multiple countries worldwide. The continuously increasing number of infections and deaths has driven several countries to implement control and response strategies including community lockdowns, physical distancing, and travel bans with different levels of success. However, a disease outbreak and the corresponding policies can cause disastrous economic consequences due to business closures and risk minimization behaviors. This paper develops a system dynamics framework of a disease outbreak system covering various policies to evaluate their effectiveness in mitigating transmission and the resulting economic burden. The system dynamics modeling approach captures the relationships, feedbacks, and delays in such a system, revealing meaningful insights on the dynamics of several response strategies.

Policy Development for Pandemic Response Using System Dynamics: a Case Study on COVID-19

The coronavirus disease 2019 (COVID-19) outbreak has burdened several countries. Its high transmissibility and mortality rate have caused devastating impacts on human lives. This has led countries to implement control strategies, such as social distancing, travel bans, and community lockdowns, with varying levels of success. However, a disease outbreak can cause significant economic disruption from business closures and risk avoidance behaviors. This paper raises policy recommendations through a system dynamics modeling approach. The developed model captures relationships, feedbacks, and delays present in a disease transmission system. The dynamics of several policies are analyzed and assessed based on effectiveness in mitigating infection and the resulting economic strain.

Social Media Surveillance for Outbreak Projection via Transmission Models: Longitudinal Observational Study

Background

Although dynamic models are increasingly used by decision makers as a source of insight to guide interventions in order to control communicable disease outbreaks, such models have long suffered from a risk of rapid obsolescence due to failure to keep updated with emerging epidemiological evidence. The application of statistical filtering algorithms to high-velocity data streams has recently demonstrated effectiveness in allowing such models to be automatically regrounded by each new set of incoming observations. The attractiveness of such techniques has been enhanced by the emergence of a new generation of geospatially specific, high-velocity data sources, including daily counts of relevant searches and social media posts. The information available in such electronic data sources complements that of traditional epidemiological data sources.

Objective

This study aims to evaluate the degree to which the predictive accuracy of pandemic projection models regrounded via machine learning in daily clinical data can be enhanced by extending such methods to leverage daily search counts.

Methods

We combined a previously published influenza A (H1N1) pandemic projection model with the sequential Monte Carlo technique of particle filtering, to reground the model bu using confirmed incident case counts and search volumes. The effectiveness of particle filtering was evaluated using a norm discrepancy metric via predictive and dataset-specific cross-validation.

Results

Our results suggested that despite the data quality limitations of daily search volume data, the predictive accuracy of dynamic models can be strongly elevated by inclusion of such data in filtering methods.

Conclusions

The predictive accuracy of dynamic models can be notably enhanced by tapping a readily accessible, publicly available, high-velocity data source. This work highlights a low-cost, low-burden avenue for strengthening model-based outbreak intervention response planning using low-cost public electronic datasets.