Trade-offs between individual and ensemble forecasts of an emerging infectious disease

Modeling zoonotic and vector-borne viruses

The 2013-2016 Ebola virus disease epidemic and the coronavirus disease 2019 pandemic galvanized tremendous growth in models for emerging zoonotic and vector-borne viruses. Therefore, we have reviewed the main goals and methods of models to guide scientists and decision-makers. The elements of models for emerging viruses vary across spectrums: from understanding the past to forecasting the future, using data across space and time, and using statistical versus mechanistic methods. Hybrid/ensemble models and artificial intelligence offer new opportunities for modeling. Despite this progress, challenges remain in translating models into actionable decisions, particularly in areas at highest risk for viral disease outbreaks. To address this issue, we must identify gaps in models for specific viruses, strengthen validation, and involve policymakers in model development.

Evaluating the forecasting performance of ensemble sub-epidemic frameworks and other time series models for the 2022-2023 mpox epidemic

During the 2022-2023 unprecedented mpox epidemic, near real-time short-term forecasts of the epidemic's trajectory were essential in intervention implementation and guiding policy. However, as case levels have significantly decreased, evaluating model performance is vital to advancing the field of epidemic forecasting. Using laboratory-confirmed mpox case data from the Centers for Disease Control and Prevention and Our World in Data teams, we generated retrospective sequential weekly forecasts for Brazil, Canada, France, Germany, Spain, the United Kingdom, the United States and at the global scale using an auto-regressive integrated moving average (ARIMA) model, generalized additive model, simple linear regression, Facebook's Prophet model, as well as the sub-epidemic wave and n-sub-epidemic modelling frameworks. We assessed forecast performance using average mean squared error, mean absolute error, weighted interval scores, 95% prediction interval coverage, skill scores and Winkler scores. Overall, the n-sub-epidemic modelling framework outcompeted other models across most locations and forecasting horizons, with the unweighted ensemble model performing best most frequently. The n-sub-epidemic and spatial-wave frameworks considerably improved in average forecasting performance relative to the ARIMA model (greater than 10%) for all performance metrics. Findings further support sub-epidemic frameworks for short-term forecasting epidemics of emerging and re-emerging infectious diseases.

Chimeric Forecasting: An experiment to leverage human judgment to improve forecasts of infectious disease using simulated surveillance data

Forecasts of infectious agents provide public health officials advanced warning about the intensity and timing of the spread of disease. Past work has found that accuracy and calibration of forecasts is weakest when attempting to predict an epidemic peak. Forecasts from a mechanistic model would be improved if there existed accurate information about the timing and intensity of an epidemic. We presented 3000 humans with simulated surveillance data about the number of incident hospitalizations from a current and two past seasons, and asked that they predict the peak time and intensity of the underlying epidemic. We found that in comparison to two control models, a model including human judgment produced more accurate forecasts of peak time and intensity of hospitalizations during an epidemic. Chimeric models have the potential to improve our ability to predict targets of public health interest which may in turn reduce infectious disease burden.

Nowcasting and forecasting the 2022 U.S. mpox outbreak: Support for public health decision making and lessons learned

In June of 2022, the U.S. Centers for Disease Control and Prevention (CDC) Mpox Response wanted timely answers to important epidemiological questions which can now be answered more effectively through infectious disease modeling. Infectious disease models have shown to be valuable tools for decision making during outbreaks; however, model complexity often makes communicating the results and limitations of models to decision makers difficult. We performed nowcasting and forecasting for the 2022 mpox outbreak in the United States using the R package EpiNow2. We generated nowcasts/forecasts at the national level, by Census region, and for jurisdictions reporting the greatest number of mpox cases. Modeling results were shared for situational awareness within the CDC Mpox Response and publicly on the CDC website. We retrospectively evaluated forecast predictions at four key phases (early, exponential growth, peak, and decline) during the outbreak using three metrics, the weighted interval score, mean absolute error, and prediction interval coverage. We compared the performance of EpiNow2 with a naïve Bayesian generalized linear model (GLM). The EpiNow2 model had less probabilistic error than the GLM during every outbreak phase except for the early phase. We share our experiences with an existing tool for nowcasting/forecasting and highlight areas of improvement for the development of future tools. We also reflect on lessons learned regarding data quality issues and adapting modeling results for different audiences.

Using neural ordinary differential equations to predict complex ecological dynamics from population density data

Simple models have been used to describe ecological processes for over a century. However, the complexity of ecological systems makes simple models subject to modelling bias due to simplifying assumptions or unaccounted factors, limiting their predictive power. Neural ordinary differential equations (NODEs) have surged as a machine-learning algorithm that preserves the dynamic nature of the data (Chen et al. 2018 Adv. Neural Inf. Process. Syst.). Although preserving the dynamics in the data is an advantage, the question of how NODEs perform as a forecasting tool of ecological communities is unanswered. Here, we explore this question using simulated time series of competing species in a time-varying environment. We find that NODEs provide more precise forecasts than autoregressive integrated moving average (ARIMA) models. We also find that untuned NODEs have a similar forecasting accuracy to untuned long-short term memory neural networks and both are outperformed in accuracy and precision by empirical dynamical modelling . However, we also find NODEs generally outperform all other methods when evaluating with the interval score, which evaluates precision and accuracy in terms of prediction intervals rather than pointwise accuracy. We also discuss ways to improve the forecasting performance of NODEs. The power of a forecasting tool such as NODEs is that it can provide insights into population dynamics and should thus broaden the approaches to studying time series of ecological communities.

Challenges of COVID-19 Case Forecasting in the US, 2020-2021

During the COVID-19 pandemic, forecasting COVID-19 trends to support planning and response was a priority for scientists and decision makers alike. In the United States, COVID-19 forecasting was coordinated by a large group of universities, companies, and government entities led by the Centers for Disease Control and Prevention and the US COVID-19 Forecast Hub (https://covid19forecasthub.org). We evaluated approximately 9.7 million forecasts of weekly state-level COVID-19 cases for predictions 1-4 weeks into the future submitted by 24 teams from August 2020 to December 2021. We assessed coverage of central prediction intervals and weighted interval scores (WIS), adjusting for missing forecasts relative to a baseline forecast, and used a Gaussian generalized estimating equation (GEE) model to evaluate differences in skill across epidemic phases that were defined by the effective reproduction number. Overall, we found high variation in skill across individual models, with ensemble-based forecasts outperforming other approaches. Forecast skill relative to the baseline was generally higher for larger jurisdictions (e.g., states compared to counties). Over time, forecasts generally performed worst in periods of rapid changes in reported cases (either in increasing or decreasing epidemic phases) with 95% prediction interval coverage dropping below 50% during the growth phases of the winter 2020, Delta, and Omicron waves. Ideally, case forecasts could serve as a leading indicator of changes in transmission dynamics. However, while most COVID-19 case forecasts outperformed a naïve baseline model, even the most accurate case forecasts were unreliable in key phases. Further research could improve forecasts of leading indicators, like COVID-19 cases, by leveraging additional real-time data, addressing performance across phases, improving the characterization of forecast confidence, and ensuring that forecasts were coherent across spatial scales. In the meantime, it is critical for forecast users to appreciate current limitations and use a broad set of indicators to inform pandemic-related decision making.

Spatial spread of COVID-19 during the early pandemic phase in Italy

Quantifying the potential spatial spread of an infectious pathogen is key to defining effective containment and control strategies. The aim of this study is to estimate the risk of SARS-CoV-2 transmission at different distances in Italy before the first regional lockdown was imposed, identifying important sources of national spreading. To do this, we leverage on a probabilistic model applied to daily symptomatic cases retrospectively ascertained in each Italian municipality with symptom onset between January 28 and March 7, 2020. Results are validated using a multi-patch dynamic transmission model reproducing the spatiotemporal distribution of identified cases. Our results show that the contribution of short-distance ( ≤ 10 k m ) transmission increased from less than 40% in the last week of January to more than 80% in the first week of March 2020. On March 7, 2020, that is the day before the first regional lockdown was imposed, more than 200 local transmission foci were contributing to the spread of SARS-CoV-2 in Italy. At the time, isolation measures imposed only on municipalities with at least ten ascertained cases would have left uncontrolled more than 75% of spillover transmission from the already affected municipalities. In early March, national-wide restrictions were required to curb short-distance transmission of SARS-CoV-2 in Italy.

District-Level Forecast of Achieving Trachoma Elimination as a Public Health Problem By 2030: An Ensemble Modelling Approach

Assessing the feasibility of 2030 as a target date for global elimination of trachoma, and identification of districts that may require enhanced treatment to meet World Health Organization (WHO) elimination criteria by this date are key challenges in operational planning for trachoma programmes. Here we address these challenges by prospectively evaluating forecasting models of trachomatous inflammation-follicular (TF) prevalence, leveraging ensemble-based approaches. Seven candidate probabilistic models were developed to forecast district-wise TF prevalence in 11 760 districts, trained using district-level data on the population prevalence of TF in children aged 1-9 years from 2004 to 2022. Geographical location, history of mass drug administration treatment, and previously measured prevalence data were included in these models as key predictors. The best-performing models were included in an ensemble, using weights derived from their relative likelihood scores. To incorporate the inherent stochasticity of disease transmission and challenges of population-level surveillance, we forecasted probability distributions for the TF prevalence in each geographic district, rather than predicting a single value. Based on our probabilistic forecasts, 1.46% (95% confidence interval [CI]: 1.43-1.48%) of all districts in trachoma-endemic countries, equivalent to 172 districts, will exceed the 5% TF control threshold in 2030 with the current interventions. Global elimination of trachoma as a public health problem by 2030 may require enhanced intervention and/or surveillance of high-risk districts.

Mapping Potential Malaria Vector Larval Habitats for Larval Source Management in Western Kenya: Introduction to Multimodel Ensembling Approaches

Identification and mapping of larval sources are a prerequisite for effective planning and implementing mosquito larval source management (LSM). Ensemble modeling is increasingly used for prediction modeling, but it lacks standard procedures. We proposed a detailed framework to predict potential malaria vector larval habitats by using multimodel ensemble modeling, which includes selection of models, ensembling method, and predictors, evaluation of variable importance, prediction of potential larval habitats, and assessment of prediction uncertainty. The models were built and validated based on multisite, multiyear field observations and climatic/environmental variables. Model performance was tested using independent field observations. Overall, we found that the ensembled model predicted larval habitats with about 20% more accuracy than the average of the individual models ensembled. Key larval habitat predictors in western Kenya were elevation, geomorphon class, and precipitation for the 2 months prior. Additional predictors may be required to increase the predictive accuracy of the larva-positive habitats. This is the first study to provide a detailed framework for the process of multimodel ensemble modeling for malaria vector habitats. Mapping of potential habitats will be helpful in LSM planning.

ePyDGGA: automatic configuration for fitting epidemic curves

Many epidemiological models and algorithms are used to fit the parameters of a given epidemic curve. On many occasions, fitting algorithms are interleaved with the actual epidemic models, which yields combinations of model-parameters that are hard to compare among themselves. Here, we provide a model-agnostic framework for epidemic parameter fitting that can (fairly) compare different epidemic models without jeopardizing the quality of the fitted parameters. Briefly, we have developed a Python framework that expects a Python function (epidemic model) and epidemic data and performs parameter fitting using automatic configuration. Our framework is capable of fitting parameters for any type of epidemic model, as long as it is provided as a Python function (or even in a different programming language). Moreover, we provide the code for different types of models, as well as the implementation of 4 concrete models with data to fit them. Documentation, code and examples can be found at https://ulog.udl.cat/static/doc/epidemic-gga/html/index.html .

Retrospective evaluation of short-term forecast performance of ensemble sub-epidemic frameworks and other time-series models: The 2022-2023 mpox outbreak across multiple geographical scales, July 14th, 2022, through February 26th, 2023

In May 2022, public health officials noted an unprecedented surge in mpox cases in non-endemic countries worldwide. As the epidemic accelerated, multi-model forecasts of the epidemic's trajectory were critical in guiding the implementation of public health interventions and determining policy. As the case levels have significantly decreased as of early September 2022, evaluating model performance is essential to advance the growing field of epidemic forecasting. Using laboratory-confirmed mpox case data from the Centers for Disease Control and Prevention (CDC) and Our World in Data (OWID) teams through the week of January 26th, 2023, we generated retrospective sequential weekly forecasts (e.g., 1-4-weeks) for Brazil, Canada, France, Germany, Spain, the United Kingdom, the USA, and at the global scale using models that require minimal input data including the auto-regressive integrated moving average (ARIMA), general additive model (GAM), simple linear regression (SLR), Facebook's Prophet model, as well as the sub-epidemic wave (spatial-wave) and n -sub-epidemic modeling frameworks. We assess forecast performance using average mean squared error (MSE), mean absolute error (MAE), weighted interval score (WIS), 95% prediction interval coverage (95% PI coverage), and skill scores. Average Winkler scores were used to calculate skill scores for 95% PI coverage. Overall, the n -sub-epidemic modeling framework outcompeted other models across most locations and forecasting horizons, with the unweighted ensemble model performing best across all forecasting horizons for most locations regarding average MSE, MAE, WIS, and 95% PI coverage. However, many locations had multiple models performing equally well for the average 95% PI coverage. The n -sub-epidemic and spatial-wave frameworks improved considerably in average MSE, MAE, and WIS, and Winkler scores (95% PI coverage) relative to the ARIMA model. Findings lend further support to sub-epidemic frameworks for short-term forecasting epidemics of emerging and re-emerging infectious diseases.

Call detail record aggregation methodology impacts infectious disease models informed by human mobility

This paper demonstrates how two different methods used to calculate population-level mobility from Call Detail Records (CDR) produce varying predictions of the spread of epidemics informed by these data. Our findings are based on one CDR dataset describing inter-district movement in Ghana in 2021, produced using two different aggregation methodologies. One methodology, "all pairs," is designed to retain long distance network connections while the other, "sequential" methodology is designed to accurately reflect the volume of travel between locations. We show how the choice of methodology feeds through models of human mobility to the predictions of a metapopulation SEIR model of disease transmission. We also show that this impact varies depending on the location of pathogen introduction and the transmissibility of infections. For central locations or highly transmissible diseases, we do not observe significant differences between aggregation methodologies on the predicted spread of disease. For less transmissible diseases or those introduced into remote locations, we find that the choice of aggregation methodology influences the speed of spatial spread as well as the size of the peak number of infections in individual districts. Our findings can help researchers and users of epidemiological models to understand how methodological choices at the level of model inputs may influence the results of models of infectious disease transmission, as well as the circumstances in which these choices do not alter model predictions.

Linking mathematical models and trap data to infer the proliferation, abundance, and control of Aedes aegypti

Aedes aegypti is one of the most dominant mosquito species in the urban areas of Miami-Dade County, Florida, and is responsible for the local arbovirus transmissions. Since August 2016, mosquito traps have been placed throughout the county to improve surveillance and guide mosquito control and arbovirus outbreak response. In this paper, we develop a deterministic mosquito population model, estimate model parameters by using local entomological and temperature data, and use the model to calibrate the mosquito trap data from 2017 to 2019. We further use the model to compare the Ae. aegypti population and evaluate the impact of rainfall intensity in different urban built environments. Our results show that rainfall affects the breeding sites and the abundance of Ae. aegypti more significantly in tourist areas than in residential places. In addition, we apply the model to quantitatively assess the effectiveness of vector control strategies in Miami-Dade County.

Context-dependent representation of within- and between-model uncertainty: aggregating probabilistic predictions in infectious disease epidemiology

Probabilistic predictions support public health planning and decision making, especially in infectious disease emergencies. Aggregating outputs from multiple models yields more robust predictions of outcomes and associated uncertainty. While the selection of an aggregation method can be guided by retrospective performance evaluations, this is not always possible. For example, if predictions are conditional on assumptions about how the future will unfold (e.g. possible interventions), these assumptions may never materialize, precluding any direct comparison between predictions and observations. Here, we summarize literature on aggregating probabilistic predictions, illustrate various methods for infectious disease predictions via simulation, and present a strategy for choosing an aggregation method when empirical validation cannot be used. We focus on the linear opinion pool (LOP) and Vincent average, common methods that make different assumptions about between-prediction uncertainty. We contend that assumptions of the aggregation method should align with a hypothesis about how uncertainty is expressed within and between predictions from different sources. The LOP assumes that between-prediction uncertainty is meaningful and should be retained, while the Vincent average assumes that between-prediction uncertainty is akin to sampling error and should not be preserved. We provide an R package for implementation. Given the rising importance of multi-model infectious disease hubs, our work provides useful guidance on aggregation and a deeper understanding of the benefits and risks of different approaches.

Comparing sources of mobility for modelling the epidemic spread of Zika virus in Colombia

Timely, accurate, and comparative data on human mobility is of paramount importance for epidemic preparedness and response, but generally not available or easily accessible. Mobile phone metadata, typically in the form of Call Detail Records (CDRs), represents a powerful source of information on human movements at an unprecedented scale. In this work, we investigate the potential benefits of harnessing aggregated CDR-derived mobility to predict the 2015-2016 Zika virus (ZIKV) outbreak in Colombia, when compared to other traditional data sources. To simulate the spread of ZIKV at sub-national level in Colombia, we employ a stochastic metapopulation epidemic model for vector-borne diseases. Our model integrates detailed data on the key drivers of ZIKV spread, including the spatial heterogeneity of the mosquito abundance, and the exposure of the population to the virus due to environmental and socio-economic factors. Given the same modelling settings (i.e. initial conditions and epidemiological parameters), we perform in-silico simulations for each mobility network and assess their ability in reproducing the local outbreak as reported by the official surveillance data. We assess the performance of our epidemic modelling approach in capturing the ZIKV outbreak both nationally and sub-nationally. Our model estimates are strongly correlated with the surveillance data at the country level (Pearson’s r = 0.92 for the CDR-informed network). Moreover, we found strong performance of the model estimates generated by the CDR-informed mobility networks in reproducing the local outbreak observed at the sub-national level. Compared to the CDR-informed networks, the performance of the other mobility networks is either comparatively similar or substantially lower, with no added value in predicting the local epidemic. This suggests that mobile phone data captures a better picture of human mobility patterns. This work contributes to the ongoing discussion on the value of aggregated mobility estimates from CDRs data that, with appropriate data protection and privacy safeguards, can be used for social impact applications and humanitarian action.

An ensemble model based on early predictors to forecast COVID-19 health care demand in France

Short-term forecasting of the COVID-19 pandemic is required to facilitate the planning of COVID-19 health care demand in hospitals. Here, we evaluate the performance of 12 individual models and 19 predictors to anticipate French COVID-19-related health care needs from September 7, 2020, to March 6, 2021. We then build an ensemble model by combining the individual forecasts and retrospectively test this model from March 7, 2021, to July 6, 2021. We find that the inclusion of early predictors (epidemiological, mobility, and meteorological predictors) can halve the rms error for 14-d–ahead forecasts, with epidemiological and mobility predictors contributing the most to the improvement. On average, the ensemble model is the best or second-best model, depending on the evaluation metric. Our approach facilitates the comparison and benchmarking of competing models through their integration in a coherent analytical framework, ensuring that avenues for future improvements can be identified.