An open challenge to advance probabilistic forecasting for dengue epidemics

Integrating information from historical data into mechanistic models for influenza forecasting

Seasonal influenza causes significant annual morbidity and mortality worldwide. In France, it is estimated that, on average, 2 million individuals consult their GP for influenza-like-illness (ILI) every year. Traditionally, mathematical models used for epidemic forecasting can either include parameters capturing the infection process (mechanistic or compartmental models) or rely on time series analysis approaches that do not make mechanistic assumptions (statistical or phenomenological models). While the latter make extensive use of past epidemic data, mechanistic models are usually independently initialized in each season. As a result, forecasts from such models can contain trajectories that are vastly different from past epidemics. We developed a mechanistic model that takes into account epidemic data from training seasons when producing forecasts. The parameters of the model are estimated via a first particle filter running on the observed data. A second particle filter is then used to produce forecasts compatible with epidemic trajectories from the training set. The model was calibrated and tested on 35 years' worth of surveillance data from the French Sentinelles Network, representing the weekly number of patients consulting for ILI over the period 1985-2019. Our results show that the new method improves upon standard mechanistic approaches. In particular, when retrospectively tested on the available data, our model provides increased accuracy for short-term forecasts (from one to four weeks into the future) and peak timing and intensity. Our new approach for epidemic forecasting allows the integration of key strengths of the statistical approach into the mechanistic modelling framework and represents an attempt to provide accurate forecasts by making full use of the rich surveillance dataset collected in France since 1985.

Perspectives and challenges in developing and implementing integrated dengue surveillance tools and technology in Thailand: a qualitative study

BACKGROUND

Dengue remains a persistent public health concern, especially in tropical and sub-tropical countries like Thailand. The development and utilization of quantitative tools and information technology show significant promise for enhancing public health policy decisions in integrated dengue control. However, the effective implementation of these tools faces multifaceted challenges and barriers that are relatively underexplored.

METHODS

This qualitative study employed in-depth interviews to gain a better understanding of the experiences and challenges of quantitative tool development and implementation with key stakeholders involved in dengue control in Thailand, using a phenomenological framework. A diverse range of participants, including public health workers and dengue control experts, participated in these interviews. The collected interview data were systematically managed and investigated using thematic analysis to extract meaningful insights.

RESULTS

The ability to collect dengue surveillance data and conduct ongoing analyses were contingent upon the availability of individuals possessing essential digital literacy and analytical skills, which were often in short supply. Furthermore, effective space-time early warning and precise data collection were hindered by the absence of user-friendly tools, efficient reporting systems, and complexities in data integration. Additionally, the study underscored the importance of the crucial role of community involvement and collaboration among organizations involved in integrated dengue surveillance, control and quantitative tool development.

CONCLUSIONS

This study employed a qualitative approach to gain a deeper understanding of the contextual intricacies surrounding the development and implementation of quantitative tools, which, despite their potential for strengthening public health policy decisions in dengue control, remain relatively unexplored in the Thai context. The findings yield valuable insights and recommendations for the development and utilization of quantitative tools to support dengue control in Thailand. This information also has the potential to support use of such tools to exert impact beyond dengue to a broader spectrum of diseases.

Title evaluation of FluSight influenza forecasting in the 2021-22 and 2022-23 seasons with a new target laboratory-confirmed influenza hospitalizations

Accurate forecasts can enable more effective public health responses during seasonal influenza epidemics. For the 2021-22 and 2022-23 influenza seasons, 26 forecasting teams provided national and jurisdiction-specific probabilistic predictions of weekly confirmed influenza hospital admissions for one-to-four weeks ahead. Forecast skill is evaluated using the Weighted Interval Score (WIS), relative WIS, and coverage. Six out of 23 models outperform the baseline model across forecast weeks and locations in 2021-22 and 12 out of 18 models in 2022-23. Averaging across all forecast targets, the FluSight ensemble is the 2nd most accurate model measured by WIS in 2021-22 and the 5th most accurate in the 2022-23 season. Forecast skill and 95% coverage for the FluSight ensemble and most component models degrade over longer forecast horizons. In this work we demonstrate that while the FluSight ensemble was a robust predictor, even ensembles face challenges during periods of rapid change.

Evaluating the forecasting performance of ensemble sub-epidemic frameworks and other time series models for the 2022-2023 mpox epidemic

During the 2022-2023 unprecedented mpox epidemic, near real-time short-term forecasts of the epidemic's trajectory were essential in intervention implementation and guiding policy. However, as case levels have significantly decreased, evaluating model performance is vital to advancing the field of epidemic forecasting. Using laboratory-confirmed mpox case data from the Centers for Disease Control and Prevention and Our World in Data teams, we generated retrospective sequential weekly forecasts for Brazil, Canada, France, Germany, Spain, the United Kingdom, the United States and at the global scale using an auto-regressive integrated moving average (ARIMA) model, generalized additive model, simple linear regression, Facebook's Prophet model, as well as the sub-epidemic wave and n-sub-epidemic modelling frameworks. We assessed forecast performance using average mean squared error, mean absolute error, weighted interval scores, 95% prediction interval coverage, skill scores and Winkler scores. Overall, the n-sub-epidemic modelling framework outcompeted other models across most locations and forecasting horizons, with the unweighted ensemble model performing best most frequently. The n-sub-epidemic and spatial-wave frameworks considerably improved in average forecasting performance relative to the ARIMA model (greater than 10%) for all performance metrics. Findings further support sub-epidemic frameworks for short-term forecasting epidemics of emerging and re-emerging infectious diseases.

hubEnsembles: Ensembling Methods in R

Combining predictions from multiple models into an ensemble is a widely used practice across many fields with demonstrated performance benefits. The R package hubEnsembles provides a flexible framework for ensembling various types of predictions, including point estimates and probabilistic predictions. A range of common methods for generating ensembles are supported, including weighted averages, quantile averages, and linear pools. The hubEnsembles package fits within a broader framework of open-source software and data tools called the "hubverse", which facilitates the development and management of collaborative modelling exercises.

Masks and respirators for prevention of respiratory infections: a state of the science review

SUMMARYThis narrative review and meta-analysis summarizes a broad evidence base on the benefits-and also the practicalities, disbenefits, harms and personal, sociocultural and environmental impacts-of masks and masking. Our synthesis of evidence from over 100 published reviews and selected primary studies, including re-analyzing contested meta-analyses of key clinical trials, produced seven key findings. First, there is strong and consistent evidence for airborne transmission of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) and other respiratory pathogens. Second, masks are, if correctly and consistently worn, effective in reducing transmission of respiratory diseases and show a dose-response effect. Third, respirators are significantly more effective than medical or cloth masks. Fourth, mask mandates are, overall, effective in reducing community transmission of respiratory pathogens. Fifth, masks are important sociocultural symbols; non-adherence to masking is sometimes linked to political and ideological beliefs and to widely circulated mis- or disinformation. Sixth, while there is much evidence that masks are not generally harmful to the general population, masking may be relatively contraindicated in individuals with certain medical conditions, who may require exemption. Furthermore, certain groups (notably D/deaf people) are disadvantaged when others are masked. Finally, there are risks to the environment from single-use masks and respirators. We propose an agenda for future research, including improved characterization of the situations in which masking should be recommended or mandated; attention to comfort and acceptability; generalized and disability-focused communication support in settings where masks are worn; and development and testing of novel materials and designs for improved filtration, breathability, and environmental impact.

Scenario design for infectious disease projections: Integrating concepts from decision analysis and experimental design

Across many fields, scenario modeling has become an important tool for exploring long-term projections and how they might depend on potential interventions and critical uncertainties, with relevance to both decision makers and scientists. In the past decade, and especially during the COVID-19 pandemic, the field of epidemiology has seen substantial growth in the use of scenario projections. Multiple scenarios are often projected at the same time, allowing important comparisons that can guide the choice of intervention, the prioritization of research topics, or public communication. The design of the scenarios is central to their ability to inform important questions. In this paper, we draw on the fields of decision analysis and statistical design of experiments to propose a framework for scenario design in epidemiology, with relevance also to other fields. We identify six different fundamental purposes for scenario designs (decision making, sensitivity analysis, situational awareness, horizon scanning, forecasting, and value of information) and discuss how those purposes guide the structure of scenarios. We discuss other aspects of the content and process of scenario design, broadly for all settings and specifically for multi-model ensemble projections. As an illustrative case study, we examine the first 17 rounds of scenarios from the U.S. COVID-19 Scenario Modeling Hub, then reflect on future advancements that could improve the design of scenarios in epidemiological settings.

When do we need multiple infectious disease models? Agreement between projection rank and magnitude in a multi-model setting

Mathematical models are useful for public health planning and response to infectious disease threats. However, different models can provide differing results, which can hamper decision making if not synthesized appropriately. To address this challenge, multi-model hubs convene independent modeling groups to generate ensembles, known to provide more accurate predictions of future outcomes. Yet, these hubs are resource intensive, and how many models are sufficient in a hub is not known. Here, we compare the benefit of predictions from multiple models in different contexts: (1) decision settings that depend on predictions of quantitative outcomes (e.g., hospital capacity planning), where assessments of the benefits of multi-model ensembles have largely focused; and (2) decisions settings that require the ranking of alternative epidemic scenarios (e.g., comparing outcomes under multiple possible interventions and biological uncertainties). We develop a mathematical framework to mimic a multi-model prediction setting, and use this framework to quantify how frequently predictions from different models agree. We further explore multi-model agreement using real-world, empirical data from 14 rounds of U.S. COVID-19 Scenario Modeling Hub projections. Our results suggest that the value of multiple models could be different in different decision contexts, and if only a few models are available, focusing on the rank of alternative epidemic scenarios could be more robust than focusing on quantitative outcomes. Although additional exploration of the sufficient number of models for different contexts is still needed, our results indicate that it may be possible to identify decision contexts where it is robust to rely on fewer models, a finding that can inform the use of modeling resources during future public health crises.

Redefining pandemic preparedness: Multidisciplinary insights from the CERP modelling workshop in infectious diseases, workshop report

In July 2023, the Center of Excellence in Respiratory Pathogens organized a two-day workshop on infectious diseases modelling and the lessons learnt from the Covid-19 pandemic. This report summarizes the rich discussions that occurred during the workshop. The workshop participants discussed multisource data integration and highlighted the benefits of combining traditional surveillance with more novel data sources like mobility data, social media, and wastewater monitoring. Significant advancements were noted in the development of predictive models, with examples from various countries showcasing the use of machine learning and artificial intelligence in detecting and monitoring disease trends. The role of open collaboration between various stakeholders in modelling was stressed, advocating for the continuation of such partnerships beyond the pandemic. A major gap identified was the absence of a common international framework for data sharing, which is crucial for global pandemic preparedness. Overall, the workshop underscored the need for robust, adaptable modelling frameworks and the integration of different data sources and collaboration across sectors, as key elements in enhancing future pandemic response and preparedness.

Modeling the Regional Distribution of International Travelers in Spain to Estimate Imported Cases of Dengue and Malaria: Statistical Inference and Validation Study

BACKGROUND

Understanding the patterns of disease importation through international travel is paramount for effective public health interventions and global disease surveillance. While global airline network data have been used to assist in outbreak prevention and effective preparedness, accurately estimating how these imported cases disseminate locally in receiving countries remains a challenge.

OBJECTIVE

This study aimed to describe and understand the regional distribution of imported cases of dengue and malaria upon arrival in Spain via air travel.

METHODS

We have proposed a method to describe the regional distribution of imported cases of dengue and malaria based on the computation of the "travelers' index" from readily available socioeconomic data. We combined indicators representing the main drivers for international travel, including tourism, economy, and visits to friends and relatives, to measure the relative appeal of each region in the importing country for travelers. We validated the resulting estimates by comparing them with the reported cases of malaria and dengue in Spain from 2015 to 2019. We also assessed which motivation provided more accurate estimates for imported cases of both diseases.

RESULTS

The estimates provided by the best fitted model showed high correlation with notified cases of malaria (0.94) and dengue (0.87), with economic motivation being the most relevant for imported cases of malaria and visits to friends and relatives being the most relevant for imported cases of dengue.

CONCLUSIONS

Factual descriptions of the local movement of international travelers may substantially enhance the design of cost-effective prevention policies and control strategies, and essentially contribute to decision-support systems. Our approach contributes in this direction by providing a reliable estimate of the number of imported cases of nonendemic diseases, which could be generalized to other applications. Realistic risk assessments will be obtained by combining this regional predictor with the observed local distribution of vectors.

Informing Public Health Policies with Models for Disease Burden, Impact Evaluation, and Economic Evaluation

Conducting real-world public health experiments is often costly, time-consuming, and ethically challenging, so mathematical models have a long-standing history of being used to inform policy. Applications include estimating disease burden, performing economic evaluation of interventions, and responding to health emergencies such as pandemics. Models played a pivotal role during the COVID-19 pandemic, providing early detection of SARS-CoV-2's pandemic potential and informing subsequent public health measures. While models offer valuable policy insights, they often carry limitations, especially when they depend on assumptions and incomplete data. Striking a balance between accuracy and timely decision-making in rapidly evolving situations such as disease outbreaks is challenging. Modelers need to explore the extent to which their models deviate from representing the real world. The uncertainties inherent in models must be effectively communicated to policy makers and the public. As the field becomes increasingly influential, it needs to develop reporting standards that enable rigorous external scrutiny.

Combining models to generate consensus medium-term projections of hospital admissions, occupancy and deaths relating to COVID-19 in England

Mathematical modelling has played an important role in offering informed advice during the COVID-19 pandemic. In England, a cross government and academia collaboration generated medium-term projections (MTPs) of possible epidemic trajectories over the future 4-6 weeks from a collection of epidemiological models. In this article, we outline this collaborative modelling approach and evaluate the accuracy of the combined and individual model projections against the data over the period November 2021-December 2022 when various Omicron subvariants were spreading across England. Using a number of statistical methods, we quantify the predictive performance of the model projections for both the combined and individual MTPs, by evaluating the point and probabilistic accuracy. Our results illustrate that the combined MTPs, produced from an ensemble of heterogeneous epidemiological models, were a closer fit to the data than the individual models during the periods of epidemic growth or decline, with the 90% confidence intervals widest around the epidemic peaks. We also show that the combined MTPs increase the robustness and reduce the biases associated with a single model projection. Learning from our experience of ensemble modelling during the COVID-19 epidemic, our findings highlight the importance of developing cross-institutional multi-model infectious disease hubs for future outbreak control.

Challenges of COVID-19 Case Forecasting in the US, 2020-2021

During the COVID-19 pandemic, forecasting COVID-19 trends to support planning and response was a priority for scientists and decision makers alike. In the United States, COVID-19 forecasting was coordinated by a large group of universities, companies, and government entities led by the Centers for Disease Control and Prevention and the US COVID-19 Forecast Hub (https://covid19forecasthub.org). We evaluated approximately 9.7 million forecasts of weekly state-level COVID-19 cases for predictions 1-4 weeks into the future submitted by 24 teams from August 2020 to December 2021. We assessed coverage of central prediction intervals and weighted interval scores (WIS), adjusting for missing forecasts relative to a baseline forecast, and used a Gaussian generalized estimating equation (GEE) model to evaluate differences in skill across epidemic phases that were defined by the effective reproduction number. Overall, we found high variation in skill across individual models, with ensemble-based forecasts outperforming other approaches. Forecast skill relative to the baseline was generally higher for larger jurisdictions (e.g., states compared to counties). Over time, forecasts generally performed worst in periods of rapid changes in reported cases (either in increasing or decreasing epidemic phases) with 95% prediction interval coverage dropping below 50% during the growth phases of the winter 2020, Delta, and Omicron waves. Ideally, case forecasts could serve as a leading indicator of changes in transmission dynamics. However, while most COVID-19 case forecasts outperformed a naïve baseline model, even the most accurate case forecasts were unreliable in key phases. Further research could improve forecasts of leading indicators, like COVID-19 cases, by leveraging additional real-time data, addressing performance across phases, improving the characterization of forecast confidence, and ensuring that forecasts were coherent across spatial scales. In the meantime, it is critical for forecast users to appreciate current limitations and use a broad set of indicators to inform pandemic-related decision making.

Forecasting weekly dengue incidence in Sri Lanka: Modified Autoregressive Integrated Moving Average modeling approach

Dengue poses a significant and multifaceted public health challenge in Sri Lanka, encompassing both preventive and curative aspects. Accurate dengue incidence forecasting is pivotal for effective surveillance and disease control. To address this, we developed an Autoregressive Integrated Moving Average (ARIMA) model tailored for predicting weekly dengue cases in the Colombo district. The modeling process drew on comprehensive weekly dengue fever data from the Weekly Epidemiological Reports (WER), spanning January 2015 to August 2020. Following rigorous model selection, the ARIMA (2,1,0) model, augmented with an autoregressive component (AR) of order 16, emerged as the best-fitted model. It underwent initial calibration and fine-tuning using data from January 2015 to August 2020, and was validated against independent 2000 data. Selection criteria included parameter significance, the Akaike Information Criterion (AIC), and Schwarz Bayesian Information Criterion (SBIC). Importantly, the residuals of the ARIMA model conformed to the assumptions of randomness, constant variance, and normality affirming its suitability. The forecasts closely matched observed dengue incidence, offering a valuable tool for public health decision-makers. However, an increased percentage error was noted in late 2020, likely attributed to factors including potential underreporting due to COVID-19-related disruptions amid rising dengue cases. This research contributes to the critical task of managing dengue outbreaks and underscores the dynamic challenges posed by external influences on disease surveillance.

The US COVID-19 and Influenza Scenario Modeling Hubs: Delivering long-term projections to guide policy

Between December 2020 and April 2023, the COVID-19 Scenario Modeling Hub (SMH) generated operational multi-month projections of COVID-19 burden in the US to guide pandemic planning and decision-making in the context of high uncertainty. This effort was born out of an attempt to coordinate, synthesize and effectively use the unprecedented amount of predictive modeling that emerged throughout the COVID-19 pandemic. Here we describe the history of this massive collective research effort, the process of convening and maintaining an open modeling hub active over multiple years, and attempt to provide a blueprint for future efforts. We detail the process of generating 17 rounds of scenarios and projections at different stages of the COVID-19 pandemic, and disseminating results to the public health community and lay public. We also highlight how SMH was expanded to generate influenza projections during the 2022-23 season. We identify key impacts of SMH results on public health and draw lessons to improve future collaborative modeling efforts, research on scenario projections, and the interface between models and policy.

Ensemble2: Scenarios ensembling for communication and performance analysis

Throughout the COVID-19 pandemic, scenario modeling played a crucial role in shaping the decision-making process of public health policies. Unlike forecasts, scenario projections rely on specific assumptions about the future that consider different plausible states-of-the-world that may or may not be realized and that depend on policy interventions, unpredictable changes in the epidemic outlook, etc. As a consequence, long-term scenario projections require different evaluation criteria than the ones used for traditional short-term epidemic forecasts. Here, we propose a novel ensemble procedure for assessing pandemic scenario projections using the results of the Scenario Modeling Hub (SMH) for COVID-19 in the United States (US). By defining a "scenario ensemble" for each model and the ensemble of models, termed "Ensemble2", we provide a synthesis of potential epidemic outcomes, which we use to assess projections' performance, bypassing the identification of the most plausible scenario. We find that overall the Ensemble2 models are well-calibrated and provide better performance than the scenario ensemble of individual models. The ensemble procedure accounts for the full range of plausible outcomes and highlights the importance of scenario design and effective communication. The scenario ensembling approach can be extended to any scenario design strategy, with potential refinements including weighting scenarios and allowing the ensembling process to evolve over time.

ePyDGGA: automatic configuration for fitting epidemic curves

Many epidemiological models and algorithms are used to fit the parameters of a given epidemic curve. On many occasions, fitting algorithms are interleaved with the actual epidemic models, which yields combinations of model-parameters that are hard to compare among themselves. Here, we provide a model-agnostic framework for epidemic parameter fitting that can (fairly) compare different epidemic models without jeopardizing the quality of the fitted parameters. Briefly, we have developed a Python framework that expects a Python function (epidemic model) and epidemic data and performs parameter fitting using automatic configuration. Our framework is capable of fitting parameters for any type of epidemic model, as long as it is provided as a Python function (or even in a different programming language). Moreover, we provide the code for different types of models, as well as the implementation of 4 concrete models with data to fit them. Documentation, code and examples can be found at https://ulog.udl.cat/static/doc/epidemic-gga/html/index.html .

Evaluation of the US COVID-19 Scenario Modeling Hub for informing pandemic response under uncertainty

Our ability to forecast epidemics far into the future is constrained by the many complexities of disease systems. Realistic longer-term projections may, however, be possible under well-defined scenarios that specify the future state of critical epidemic drivers. Since December 2020, the U.S. COVID-19 Scenario Modeling Hub (SMH) has convened multiple modeling teams to make months ahead projections of SARS-CoV-2 burden, totaling nearly 1.8 million national and state-level projections. Here, we find SMH performance varied widely as a function of both scenario validity and model calibration. We show scenarios remained close to reality for 22 weeks on average before the arrival of unanticipated SARS-CoV-2 variants invalidated key assumptions. An ensemble of participating models that preserved variation between models (using the linear opinion pool method) was consistently more reliable than any single model in periods of valid scenario assumptions, while projection interval coverage was near target levels. SMH projections were used to guide pandemic response, illustrating the value of collaborative hubs for longer-term scenario projections.

Effects of boosting and waning in highly exposed populations on dengue epidemic dynamics

Sequential infection with multiple dengue virus (DENV) serotypes is thought to induce enduring protection against dengue disease. However, long-term antibody waning has been observed after repeated DENV infection. Here, we provide evidence that highly immune Nicaraguan children and adults (n = 4478) experience boosting and waning of antibodies during and after major Zika and dengue epidemics. We develop a susceptible-infected-recovered-susceptible (SIRS-type) model that tracks immunity by titer rather than number of infections to show that boosts in highly immune individuals can contribute to herd immunity, delaying their susceptibility to transmissible infection. In contrast, our model of lifelong immunity in highly immune individuals, as previously assumed, results in complete disease eradication after introduction. Periodic epidemics under this scenario can only be sustained with a constant influx of infected individuals into the population or a high basic reproductive number. We also find that Zika virus infection can boost DENV immunity and produce delays and then surges in dengue epidemics, as observed with real epidemiological data. This work provides insight into factors shaping periodicity in dengue incidence and may inform vaccine efforts to maintain population immunity.

Retrospective evaluation of real-time estimates of global COVID-19 transmission trends and mortality forecasts

Since 8th March 2020 up to the time of writing, we have been producing near real-time weekly estimates of SARS-CoV-2 transmissibility and forecasts of deaths due to COVID-19 for all countries with evidence of sustained transmission, shared online. We also developed a novel heuristic to combine weekly estimates of transmissibility to produce forecasts over a 4-week horizon. Here we present a retrospective evaluation of the forecasts produced between 8th March to 29th November 2020 for 81 countries. We evaluated the robustness of the forecasts produced in real-time using relative error, coverage probability, and comparisons with null models. During the 39-week period covered by this study, both the short- and medium-term forecasts captured well the epidemic trajectory across different waves of COVID-19 infections with small relative errors over the forecast horizon. The model was well calibrated with 56.3% and 45.6% of the observations lying in the 50% Credible Interval in 1-week and 4-week ahead forecasts respectively. The retrospective evaluation of our models shows that simple transmission models calibrated using routine disease surveillance data can reliably capture the epidemic trajectory in multiple countries. The medium-term forecasts can be used in conjunction with the short-term forecasts of COVID-19 mortality as a useful planning tool as countries continue to relax public health measures.

Retrospective evaluation of short-term forecast performance of ensemble sub-epidemic frameworks and other time-series models: The 2022-2023 mpox outbreak across multiple geographical scales, July 14th, 2022, through February 26th, 2023

In May 2022, public health officials noted an unprecedented surge in mpox cases in non-endemic countries worldwide. As the epidemic accelerated, multi-model forecasts of the epidemic's trajectory were critical in guiding the implementation of public health interventions and determining policy. As the case levels have significantly decreased as of early September 2022, evaluating model performance is essential to advance the growing field of epidemic forecasting. Using laboratory-confirmed mpox case data from the Centers for Disease Control and Prevention (CDC) and Our World in Data (OWID) teams through the week of January 26th, 2023, we generated retrospective sequential weekly forecasts (e.g., 1-4-weeks) for Brazil, Canada, France, Germany, Spain, the United Kingdom, the USA, and at the global scale using models that require minimal input data including the auto-regressive integrated moving average (ARIMA), general additive model (GAM), simple linear regression (SLR), Facebook's Prophet model, as well as the sub-epidemic wave (spatial-wave) and n -sub-epidemic modeling frameworks. We assess forecast performance using average mean squared error (MSE), mean absolute error (MAE), weighted interval score (WIS), 95% prediction interval coverage (95% PI coverage), and skill scores. Average Winkler scores were used to calculate skill scores for 95% PI coverage. Overall, the n -sub-epidemic modeling framework outcompeted other models across most locations and forecasting horizons, with the unweighted ensemble model performing best across all forecasting horizons for most locations regarding average MSE, MAE, WIS, and 95% PI coverage. However, many locations had multiple models performing equally well for the average 95% PI coverage. The n -sub-epidemic and spatial-wave frameworks improved considerably in average MSE, MAE, and WIS, and Winkler scores (95% PI coverage) relative to the ARIMA model. Findings lend further support to sub-epidemic frameworks for short-term forecasting epidemics of emerging and re-emerging infectious diseases.

Scenario Design for Infectious Disease Projections: Integrating Concepts from Decision Analysis and Experimental Design

Across many fields, scenario modeling has become an important tool for exploring long-term projections and how they might depend on potential interventions and critical uncertainties, with relevance to both decision makers and scientists. In the past decade, and especially during the COVID-19 pandemic, the field of epidemiology has seen substantial growth in the use of scenario projections. Multiple scenarios are often projected at the same time, allowing important comparisons that can guide the choice of intervention, the prioritization of research topics, or public communication. The design of the scenarios is central to their ability to inform important questions. In this paper, we draw on the fields of decision analysis and statistical design of experiments to propose a framework for scenario design in epidemiology, with relevance also to other fields. We identify six different fundamental purposes for scenario designs (decision making, sensitivity analysis, value of information, situational awareness, horizon scanning, and forecasting) and discuss how those purposes guide the structure of scenarios. We discuss other aspects of the content and process of scenario design, broadly for all settings and specifically for multi-model ensemble projections. As an illustrative case study, we examine the first 17 rounds of scenarios from the U.S. COVID-19 Scenario Modeling Hub, then reflect on future advancements that could improve the design of scenarios in epidemiological settings.

Disentangling local and global climate drivers in the population dynamics of mosquito-borne infections

Identifying climate drivers is essential to understand and predict epidemics of mosquito-borne infections whose population dynamics typically exhibit seasonality and multiannual cycles. Which climate covariates to consider varies across studies, from local factors such as temperature to remote drivers such as the El Niño-Southern Oscillation. With partial wavelet coherence, we present a systematic investigation of nonstationary associations between mosquito-borne disease incidence and a given climate factor while controlling for another. Analysis of almost 200 time series of dengue and malaria around the globe at different geographical scales shows a systematic effect of global climate drivers on interannual variability and of local ones on seasonality. This clear separation of time scales of action enhances detection of climate drivers and indicates those best suited for building early-warning systems.

Decision-support tools to build climate resilience against emerging infectious diseases in Europe and beyond

Climate change is one of several drivers of recurrent outbreaks and geographical range expansion of infectious diseases in Europe. We propose a framework for the co-production of policy-relevant indicators and decision-support tools that track past, present, and future climate-induced disease risks across hazard, exposure, and vulnerability domains at the animal, human, and environmental interface. This entails the co-development of early warning and response systems and tools to assess the costs and benefits of climate change adaptation and mitigation measures across sectors, to increase health system resilience at regional and local levels and reveal novel policy entry points and opportunities. Our approach involves multi-level engagement, innovative methodologies, and novel data streams. We take advantage of intelligence generated locally and empirically to quantify effects in areas experiencing rapid urban transformation and heterogeneous climate-induced disease threats. Our goal is to reduce the knowledge-to-action gap by developing an integrated One Health-Climate Risk framework.

Public health impact of the U.S. Scenario Modeling Hub

Beginning in December 2020, the COVID-19 Scenario Modeling Hub has provided quantitative scenario-based projections for cases, hospitalizations, and deaths, aggregated across up to nine modeling groups. Projections spanned multiple months into the future and provided timely information on potential impacts of epidemiological uncertainties and interventions. Projections results were shared with the public, public health partners, and the Centers for Disease Control COVID-19 Response Team. The projections provided insights on situational awareness and informed decision-making to mitigate COVID-19 disease burden (e.g., vaccination strategies). By aggregating projections from multiple modeling teams, the Scenario Modeling Hub provided rapidly synthesized information in times of great uncertainty and conveyed possible trajectories in the presence of emerging variants. Here we detail several use cases of these projections in public health practice and communication, including assessments of whether modeling results directly or indirectly informed public health communication or guidance. These include multiple examples where comparisons of projected COVID-19 disease outcomes under different vaccination scenarios were used to inform Advisory Committee for Immunization Practices recommendations. We also describe challenges and lessons learned during this highly beneficial collaboration.

Density forecasting of conjunctivitis burden using high-dimensional environmental time series data

As one of the most common eye conditions being presented at clinics, acute conjunctivitis puts substantial strain on primary health resources. To reduce this public health burden, it is important to forecast and provide forward guidance to policymakers by estimating conjunctivitis trends, taking into account factors which influence transmission. Using a high-dimensional set of ambient air pollution and meteorological data, this study describes new approaches to point and probabilistic forecasting of conjunctivitis burden which can be readily translated to other infectious diseases. Over the period of 2012 - 2022, we show that simple models without environmental data provided better point forecasts but the more complex models which optimized predictive accuracy and combined multiple predictors demonstrated superior density forecast performance. These results were shown to be consistent over periods with and without structural breaks in transmission. Furthermore, ecological analysis using post-selection inference showed that increases in SO2, O3 surface concentration and total precipitation were associated to increased conjunctivitis attendance. The methods proposed can provide rich and informative forward guidance for outbreak preparedness and help guide healthcare resource planning in both stable periods of transmission and periods where structural breaks in data occur.

Scoring epidemiological forecasts on transformed scales

Forecast evaluation is essential for the development of predictive epidemic models and can inform their use for public health decision-making. Common scores to evaluate epidemiological forecasts are the Continuous Ranked Probability Score (CRPS) and the Weighted Interval Score (WIS), which can be seen as measures of the absolute distance between the forecast distribution and the observation. However, applying these scores directly to predicted and observed incidence counts may not be the most appropriate due to the exponential nature of epidemic processes and the varying magnitudes of observed values across space and time. In this paper, we argue that transforming counts before applying scores such as the CRPS or WIS can effectively mitigate these difficulties and yield epidemiologically meaningful and easily interpretable results. Using the CRPS on log-transformed values as an example, we list three attractive properties: Firstly, it can be interpreted as a probabilistic version of a relative error. Secondly, it reflects how well models predicted the time-varying epidemic growth rate. And lastly, using arguments on variance-stabilizing transformations, it can be shown that under the assumption of a quadratic mean-variance relationship, the logarithmic transformation leads to expected CRPS values which are independent of the order of magnitude of the predicted quantity. Applying a transformation of log(x + 1) to data and forecasts from the European COVID-19 Forecast Hub, we find that it changes model rankings regardless of stratification by forecast date, location or target types. Situations in which models missed the beginning of upward swings are more strongly emphasised while failing to predict a downturn following a peak is less severely penalised when scoring transformed forecasts as opposed to untransformed ones. We conclude that appropriate transformations, of which the natural logarithm is only one particularly attractive option, should be considered when assessing the performance of different models in the context of infectious disease incidence.

Epicasting: An Ensemble Wavelet Neural Network for forecasting epidemics

Infectious diseases remain among the top contributors to human illness and death worldwide, among which many diseases produce epidemic waves of infection. The lack of specific drugs and ready-to-use vaccines to prevent most of these epidemics worsens the situation. These force public health officials and policymakers to rely on early warning systems generated by accurate and reliable epidemic forecasters. Accurate forecasts of epidemics can assist stakeholders in tailoring countermeasures, such as vaccination campaigns, staff scheduling, and resource allocation, to the situation at hand, which could translate to reductions in the impact of a disease. Unfortunately, most of these past epidemics exhibit nonlinear and non-stationary characteristics due to their spreading fluctuations based on seasonal-dependent variability and the nature of these epidemics. We analyze various epidemic time series datasets using a maximal overlap discrete wavelet transform (MODWT) based autoregressive neural network and call it Ensemble Wavelet Neural Network (EWNet) model. MODWT techniques effectively characterize non-stationary behavior and seasonal dependencies in the epidemic time series and improve the nonlinear forecasting scheme of the autoregressive neural network in the proposed ensemble wavelet network framework. From a nonlinear time series viewpoint, we explore the asymptotic stationarity of the proposed EWNet model to show the asymptotic behavior of the associated Markov Chain. We also theoretically investigate the effect of learning stability and the choice of hidden neurons in the proposal. From a practical perspective, we compare our proposed EWNet framework with twenty-two statistical, machine learning, and deep learning models for fifteen real-world epidemic datasets with three test horizons using four key performance indicators. Experimental results show that the proposed EWNet is highly competitive compared to the state-of-the-art epidemic forecasting methods.

Informing pandemic response in the face of uncertainty. An evaluation of the U.S. COVID-19 Scenario Modeling Hub

Our ability to forecast epidemics more than a few weeks into the future is constrained by the complexity of disease systems, our limited ability to measure the current state of an epidemic, and uncertainties in how human action will affect transmission. Realistic longer-term projections (spanning more than a few weeks) may, however, be possible under defined scenarios that specify the future state of critical epidemic drivers, with the additional benefit that such scenarios can be used to anticipate the comparative effect of control measures. Since December 2020, the U.S. COVID-19 Scenario Modeling Hub (SMH) has convened multiple modeling teams to make 6-month ahead projections of the number of SARS-CoV-2 cases, hospitalizations and deaths. The SMH released nearly 1.8 million national and state-level projections between February 2021 and November 2022. SMH performance varied widely as a function of both scenario validity and model calibration. Scenario assumptions were periodically invalidated by the arrival of unanticipated SARS-CoV-2 variants, but SMH still provided projections on average 22 weeks before changes in assumptions (such as virus transmissibility) invalidated scenarios and their corresponding projections. During these periods, before emergence of a novel variant, a linear opinion pool ensemble of contributed models was consistently more reliable than any single model, and projection interval coverage was near target levels for the most plausible scenarios (e.g., 79% coverage for 95% projection interval). SMH projections were used operationally to guide planning and policy at different stages of the pandemic, illustrating the value of the hub approach for long-term scenario projections.

Comparing trained and untrained probabilistic ensemble forecasts of COVID-19 cases and deaths in the United States

The U.S. COVID-19 Forecast Hub aggregates forecasts of the short-term burden of COVID-19 in the United States from many contributing teams. We study methods for building an ensemble that combines forecasts from these teams. These experiments have informed the ensemble methods used by the Hub. To be most useful to policymakers, ensemble forecasts must have stable performance in the presence of two key characteristics of the component forecasts: (1) occasional misalignment with the reported data, and (2) instability in the relative performance of component forecasters over time. Our results indicate that in the presence of these challenges, an untrained and robust approach to ensembling using an equally weighted median of all component forecasts is a good choice to support public health decision-makers. In settings where some contributing forecasters have a stable record of good performance, trained ensembles that give those forecasters higher weight can also be helpful.

Inferring a Causal Relationship between Environmental Factors and Respiratory Infections Using Convergent Cross-Mapping

The incidence of respiratory infections in the population is related to many factors, among which environmental factors such as air quality, temperature, and humidity have attracted much attention. In particular, air pollution has caused widespread discomfort and concern in developing countries. Although the correlation between respiratory infections and air pollution is well known, establishing causality between them remains elusive. In this study, by conducting theoretical analysis, we updated the procedure of performing the extended convergent cross-mapping (CCM, a method of causal inference) to infer the causality between periodic variables. Consistently, we validated this new procedure on the synthetic data generated by a mathematical model. For real data in Shaanxi province of China in the period of 1 January 2010 to 15 November 2016, we first confirmed that the refined method is applicable by investigating the periodicity of influenza-like illness cases, an air quality index, temperature, and humidity through wavelet analysis. We next illustrated that air quality (quantified by AQI), temperature, and humidity affect the daily influenza-like illness cases, and, in particular, the respiratory infection cases increased progressively with increased AQI with a time delay of 11 days.

Applying a multi-strain dengue model to epidemics data

Dengue disease transmission is a complex vector-borne disease, mainly due to the co-circulation of four serotypes of the virus. Mathematical models have proved to be a useful tool to understand the complexity of this disease. In this work, we extend the model studied by Esteva et al., 2003, originally proposed for two serotypes, to four circulating serotypes. Using epidemic data of dengue fever in Iquitos (Peru) and San Juan (Puerto Rico), we estimate numerically the co-circulation parameter values for selected outbreaks using a bootstrap method, and we also obtained the Basic Reproduction Number, R₀, for each serotype, using both analytical calculations and numerical simulations. Our results indicate that the impact of co-circulation of serotypes in population dynamics of dengue infection is such that there is a reduced effect from DENV-3 to DENV-4 in comparison to no-cross effect for epidemics in Iquitos. Concerning San Juan epidemics, also comparing to no-cross effect, we also observed a reduced effect from the predominant serotype DENV-3 to both DENV-2 and DENV-1 epidemics neglecting the very small number of cases of DENV-4.

Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations

Background

Short-term forecasts of infectious disease burden can contribute to situational awareness and aid capacity planning. Based on best practice in other fields and recent insights in infectious disease epidemiology, one can maximise the predictive performance of such forecasts if multiple models are combined into an ensemble. Here, we report on the performance of ensembles in predicting COVID-19 cases and deaths across Europe between 08 March 2021 and 07 March 2022.

Methods

We used open-source tools to develop a public European COVID-19 Forecast Hub. We invited groups globally to contribute weekly forecasts for COVID-19 cases and deaths reported by a standardised source for 32 countries over the next 1-4 weeks. Teams submitted forecasts from March 2021 using standardised quantiles of the predictive distribution. Each week we created an ensemble forecast, where each predictive quantile was calculated as the equally-weighted average (initially the mean and then from 26th July the median) of all individual models' predictive quantiles. We measured the performance of each model using the relative Weighted Interval Score (WIS), comparing models' forecast accuracy relative to all other models. We retrospectively explored alternative methods for ensemble forecasts, including weighted averages based on models' past predictive performance.

Results

Over 52 weeks, we collected forecasts from 48 unique models. We evaluated 29 models' forecast scores in comparison to the ensemble model. We found a weekly ensemble had a consistently strong performance across countries over time. Across all horizons and locations, the ensemble performed better on relative WIS than 83% of participating models' forecasts of incident cases (with a total N=886 predictions from 23 unique models), and 91% of participating models' forecasts of deaths (N=763 predictions from 20 models). Across a 1-4 week time horizon, ensemble performance declined with longer forecast periods when forecasting cases, but remained stable over 4 weeks for incident death forecasts. In every forecast across 32 countries, the ensemble outperformed most contributing models when forecasting either cases or deaths, frequently outperforming all of its individual component models. Among several choices of ensemble methods we found that the most influential and best choice was to use a median average of models instead of using the mean, regardless of methods of weighting component forecast models.

Conclusions

Our results support the use of combining forecasts from individual models into an ensemble in order to improve predictive performance across epidemiological targets and populations during infectious disease epidemics. Our findings further suggest that median ensemble methods yield better predictive performance more than ones based on means. Our findings also highlight that forecast consumers should place more weight on incident death forecasts than incident case forecasts at forecast horizons greater than 2 weeks.

Funding

AA, BH, BL, LWa, MMa, PP, SV funded by National Institutes of Health (NIH) Grant 1R01GM109718, NSF BIG DATA Grant IIS-1633028, NSF Grant No.: OAC-1916805, NSF Expeditions in Computing Grant CCF-1918656, CCF-1917819, NSF RAPID CNS-2028004, NSF RAPID OAC-2027541, US Centers for Disease Control and Prevention 75D30119C05935, a grant from Google, University of Virginia Strategic Investment Fund award number SIF160, Defense Threat Reduction Agency (DTRA) under Contract No. HDTRA1-19-D-0007, and respectively Virginia Dept of Health Grant VDH-21-501-0141, VDH-21-501-0143, VDH-21-501-0147, VDH-21-501-0145, VDH-21-501-0146, VDH-21-501-0142, VDH-21-501-0148. AF, AMa, GL funded by SMIGE - Modelli statistici inferenziali per governare l'epidemia, FISR 2020-Covid-19 I Fase, FISR2020IP-00156, Codice Progetto: PRJ-0695. AM, BK, FD, FR, JK, JN, JZ, KN, MG, MR, MS, RB funded by Ministry of Science and Higher Education of Poland with grant 28/WFSN/2021 to the University of Warsaw. BRe, CPe, JLAz funded by Ministerio de Sanidad/ISCIII. BT, PG funded by PERISCOPE European H2020 project, contract number 101016233. CP, DL, EA, MC, SA funded by European Commission - Directorate-General for Communications Networks, Content and Technology through the contract LC-01485746, and Ministerio de Ciencia, Innovacion y Universidades and FEDER, with the project PGC2018-095456-B-I00. DE., MGu funded by Spanish Ministry of Health / REACT-UE (FEDER). DO, GF, IMi, LC funded by Laboratory Directed Research and Development program of Los Alamos National Laboratory (LANL) under project number 20200700ER. DS, ELR, GG, NGR, NW, YW funded by National Institutes of General Medical Sciences (R35GM119582; the content is solely the responsibility of the authors and does not necessarily represent the official views of NIGMS or the National Institutes of Health). FB, FP funded by InPresa, Lombardy Region, Italy. HG, KS funded by European Centre for Disease Prevention and Control. IV funded by Agencia de Qualitat i Avaluacio Sanitaries de Catalunya (AQuAS) through contract 2021-021OE. JDe, SMo, VP funded by Netzwerk Universitatsmedizin (NUM) project egePan (01KX2021). JPB, SH, TH funded by Federal Ministry of Education and Research (BMBF; grant 05M18SIA). KH, MSc, YKh funded by Project SaxoCOV, funded by the German Free State of Saxony. Presentation of data, model results and simulations also funded by the NFDI4Health Task Force COVID-19 (https://www.nfdi4health.de/task-force-covid-19-2) within the framework of a DFG-project (LO-342/17-1). LP, VE funded by Mathematical and Statistical modelling project (MUNI/A/1615/2020), Online platform for real-time monitoring, analysis and management of epidemic situations (MUNI/11/02202001/2020); VE also supported by RECETOX research infrastructure (Ministry of Education, Youth and Sports of the Czech Republic: LM2018121), the CETOCOEN EXCELLENCE (CZ.02.1.01/0.0/0.0/17-043/0009632), RECETOX RI project (CZ.02.1.01/0.0/0.0/16-013/0001761). NIB funded by Health Protection Research Unit (grant code NIHR200908). SAb, SF funded by Wellcome Trust (210758/Z/18/Z).

Challenges in Forecasting Antimicrobial Resistance

Antimicrobial resistance is a major threat to human health. Since the 2000s, computational tools for predicting infectious diseases have been greatly advanced; however, efforts to develop real-time forecasting models for antimicrobial-resistant organisms (AMROs) have been absent. In this perspective, we discuss the utility of AMRO forecasting at different scales, highlight the challenges in this field, and suggest future research priorities. We also discuss challenges in scientific understanding, access to high-quality data, model calibration, and implementation and evaluation of forecasting models. We further highlight the need to initiate research on AMRO forecasting using currently available data and resources to galvanize the research community and address initial practical questions.

Differences in Longevity and Temperature-Driven Extrinsic Incubation Period Correlate with Varying Dengue Risk in the Arizona-Sonora Desert Region

Dengue transmission is determined by a complex set of interactions between the environment, Aedes aegypti mosquitoes, dengue viruses, and humans. Emergence in new geographic areas can be unpredictable, with some regions having established mosquito populations for decades without locally acquired transmission. Key factors such as mosquito longevity, temperature-driven extrinsic incubation period (EIP), and vector-human contact can strongly influence the potential for disease transmission. To assess how these factors interact at the edge of the geographical range of dengue virus transmission, we conducted mosquito sampling in multiple urban areas located throughout the Arizona-Sonora desert region during the summer rainy seasons from 2013 to 2015. Mosquito population age structure, reflecting mosquito survivorship, was measured using a combination of parity analysis and relative gene expression of an age-related gene, SCP-1. Bloodmeal analysis was conducted on field collected blood-fed mosquitoes. Site-specific temperature was used to estimate the EIP, and this predicted EIP combined with mosquito age were combined to estimate the abundance of "potential" vectors (i.e., mosquitoes old enough to survive the EIP). Comparisons were made across cities by month and year. The dengue endemic cities Hermosillo and Ciudad Obregon, both in the state of Sonora, Mexico, had higher abundance of potential vectors than non-endemic Nogales, Sonora, Mexico. Interestingly, Tucson, Arizona consistently had a higher estimated abundance of potential vectors than dengue endemic regions of Sonora, Mexico. There were no observed city-level differences in species composition of blood meals. Combined, these data offer insights into the critical factors required for dengue transmission at the ecological edge of the mosquito's range. However, further research is needed to integrate an understanding of how social and additional environmental factors constrain and enhance dengue transmission in emerging regions.

Hindcasts and forecasts of suicide mortality in US: A modeling study

Deaths by suicide, as well as suicidal ideations, plans and attempts, have been increasing in the US for the past two decades. Deployment of effective interventions would require timely, geographically well-resolved estimates of suicide activity. In this study, we evaluated the feasibility of a two-step process for predicting suicide mortality: a) generation of hindcasts, mortality estimates for past months for which observational data would not have been available if forecasts were generated in real-time; and b) generation of forecasts with observational data augmented with hindcasts. Calls to crisis hotline services and online queries to the Google search engine for suicide-related terms were used as proxy data sources to generate hindcasts. The primary hindcast model (auto) is an Autoregressive Integrated Moving average model (ARIMA), trained on suicide mortality rates alone. Three regression models augment hindcast estimates from auto with call rates (calls), GHT search rates (ght) and both datasets together (calls_ght). The 4 forecast models used are ARIMA models trained with corresponding hindcast estimates. All models were evaluated against a baseline random walk with drift model. Rolling monthly 6-month ahead forecasts for all 50 states between 2012 and 2020 were generated. Quantile score (QS) was used to assess the quality of the forecast distributions. Median QS for auto was better than baseline (0.114 vs. 0.21. Median QS of augmented models were lower than auto, but not significantly different from each other (Wilcoxon signed-rank test, p > .05). Forecasts from augmented models were also better calibrated. Together, these results provide evidence that proxy data can address delays in release of suicide mortality data and improve forecast quality. An operational forecast system of state-level suicide risk may be feasible with sustained engagement between modelers and public health departments to appraise data sources and methods as well as to continuously evaluate forecast accuracy.

A systematic review of dengue outbreak prediction models: Current scenario and future directions

Dengue is among the fastest-spreading vector-borne infectious disease, with outbreaks often overwhelm the health system and result in huge morbidity and mortality in its endemic populations in the absence of an efficient warning system. A large number of prediction models are currently in use globally. As such, this study aimed to systematically review the published literature that used quantitative models to predict dengue outbreaks and provide insights about the current practices. A systematic search was undertaken, using the Ovid MEDLINE, EMBASE, Scopus and Web of Science databases for published citations, without time or geographical restrictions. Study selection, data extraction and management process were devised in accordance with the 'Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies' ('CHARMS') framework. A total of 99 models were included in the review from 64 studies. Most models sourced climate (94.7%) and climate change (77.8%) data from agency reports and only 59.6% of the models adjusted for reporting time lag. All included models used climate predictors; 70.7% of them were built with only climate factors. Climate factors were used in combination with climate change factors (13.4%), both climate change and demographic factors (3.1%), vector factors (6.3%), and demographic factors (5.2%). Machine learning techniques were used for 39.4% of the models. Of these, random forest (15.4%), neural networks (23.1%) and ensemble models (10.3%) were notable. Among the statistical (60.6%) models, linear regression (18.3%), Poisson regression (18.3%), generalized additive models (16.7%) and time series/autoregressive models (26.7%) were notable. Around 20.2% of the models reported no validation at all and only 5.2% reported external validation. The reporting of methodology and model performance measures were inadequate in many of the existing prediction models. This review collates plausible predictors and methodological approaches, which will contribute to robust modelling in diverse settings and populations.

Evaluation of an open forecasting challenge to assess skill of West Nile virus neuroinvasive disease prediction

BACKGROUND

West Nile virus (WNV) is the leading cause of mosquito-borne illness in the continental USA. WNV occurrence has high spatiotemporal variation, and current approaches to targeted control of the virus are limited, making forecasting a public health priority. However, little research has been done to compare strengths and weaknesses of WNV disease forecasting approaches on the national scale. We used forecasts submitted to the 2020 WNV Forecasting Challenge, an open challenge organized by the Centers for Disease Control and Prevention, to assess the status of WNV neuroinvasive disease (WNND) prediction and identify avenues for improvement.

METHODS

We performed a multi-model comparative assessment of probabilistic forecasts submitted by 15 teams for annual WNND cases in US counties for 2020 and assessed forecast accuracy, calibration, and discriminatory power. In the evaluation, we included forecasts produced by comparison models of varying complexity as benchmarks of forecast performance. We also used regression analysis to identify modeling approaches and contextual factors that were associated with forecast skill.

RESULTS

Simple models based on historical WNND cases generally scored better than more complex models and combined higher discriminatory power with better calibration of uncertainty. Forecast skill improved across updated forecast submissions submitted during the 2020 season. Among models using additional data, inclusion of climate or human demographic data was associated with higher skill, while inclusion of mosquito or land use data was associated with lower skill. We also identified population size, extreme minimum winter temperature, and interannual variation in WNND cases as county-level characteristics associated with variation in forecast skill.

CONCLUSIONS

Historical WNND cases were strong predictors of future cases with minimal increase in skill achieved by models that included other factors. Although opportunities might exist to specifically improve predictions for areas with large populations and low or high winter temperatures, areas with high case-count variability are intrinsically more difficult to predict. Also, the prediction of outbreaks, which are outliers relative to typical case numbers, remains difficult. Further improvements to prediction could be obtained with improved calibration of forecast uncertainty and access to real-time data streams (e.g. current weather and preliminary human cases).

Context-dependent representation of within- and between-model uncertainty: aggregating probabilistic predictions in infectious disease epidemiology

Probabilistic predictions support public health planning and decision making, especially in infectious disease emergencies. Aggregating outputs from multiple models yields more robust predictions of outcomes and associated uncertainty. While the selection of an aggregation method can be guided by retrospective performance evaluations, this is not always possible. For example, if predictions are conditional on assumptions about how the future will unfold (e.g. possible interventions), these assumptions may never materialize, precluding any direct comparison between predictions and observations. Here, we summarize literature on aggregating probabilistic predictions, illustrate various methods for infectious disease predictions via simulation, and present a strategy for choosing an aggregation method when empirical validation cannot be used. We focus on the linear opinion pool (LOP) and Vincent average, common methods that make different assumptions about between-prediction uncertainty. We contend that assumptions of the aggregation method should align with a hypothesis about how uncertainty is expressed within and between predictions from different sources. The LOP assumes that between-prediction uncertainty is meaningful and should be retained, while the Vincent average assumes that between-prediction uncertainty is akin to sampling error and should not be preserved. We provide an R package for implementation. Given the rising importance of multi-model infectious disease hubs, our work provides useful guidance on aggregation and a deeper understanding of the benefits and risks of different approaches.

Better null models for assessing predictive accuracy of disease models

Null models provide a critical baseline for the evaluation of predictive disease models. Many studies consider only the grand mean null model (i.e. R2) when evaluating the predictive ability of a model, which is insufficient to convey the predictive power of a model. We evaluated ten null models for human cases of West Nile virus (WNV), a zoonotic mosquito-borne disease introduced to the United States in 1999. The Negative Binomial, Historical (i.e. using previous cases to predict future cases) and Always Absent null models were the strongest overall, and the majority of null models significantly outperformed the grand mean. The length of the training timeseries increased the performance of most null models in US counties where WNV cases were frequent, but improvements were similar for most null models, so relative scores remained unchanged. We argue that a combination of null models is needed to evaluate the forecasting performance of predictive models for infectious diseases and the grand mean is the lowest bar.

A critical assessment of the detailed Aedes aegypti simulation model Skeeter Buster 2 using field experiments of indoor insecticidal control in Iquitos, Peru

The importance of mosquitoes in human pathogen transmission has motivated major research efforts into mosquito biology in pursuit of more effective vector control measures. Aedes aegypti is a particular concern in tropical urban areas, where it is the primary vector of numerous flaviviruses, including the yellow fever, Zika, and dengue viruses. With an anthropophilic habit, Ae. aegypti prefers houses, human blood meals, and ovipositioning in water-filled containers. We hypothesized that this relatively simple ecological niche should allow us to predict the impacts of insecticidal control measures on mosquito populations. To do this, we use Skeeter Buster 2 (SB2), a stochastic, spatially explicit, mechanistic model of Ae. aegypti population biology. SB2 builds on Skeeter Buster, which reproduced equilibrium dynamics of Ae. aegypti in Iquitos, Peru. Our goal was to validate SB2 by predicting the response of mosquito populations to perturbations by indoor insecticidal spraying and widespread destructive insect surveys. To evaluate SB2, we conducted two field experiments in Iquitos, Peru: a smaller pilot study in 2013 (S-2013) followed by a larger experiment in 2014 (L-2014). Here, we compare model predictions with (previously reported) empirical results from these experiments. In both simulated and empirical populations, repeated spraying yielded substantial yet temporary reductions in adult densities. The proportional effects of spraying were broadly comparable between simulated and empirical results, but we found noteworthy differences. In particular, SB2 consistently over-estimated the proportion of nulliparous females and the proportion of containers holding immature mosquitoes. We also observed less temporal variation in simulated surveys of adult abundance relative to corresponding empirical observations. Our results indicate the presence of ecological heterogeneities or sampling processes not effectively represented by SB2. Although additional empirical research could further improve the accuracy and precision of SB2, our results underscore the importance of non-linear dynamics in the response of Ae. aegypti populations to perturbations, and suggest general limits to the fine-grained predictability of its population dynamics over space and time.

National and subnational short-term forecasting of COVID-19 in Germany and Poland during early 2021

BACKGROUND

During the COVID-19 pandemic there has been a strong interest in forecasts of the short-term development of epidemiological indicators to inform decision makers. In this study we evaluate probabilistic real-time predictions of confirmed cases and deaths from COVID-19 in Germany and Poland for the period from January through April 2021.

METHODS

We evaluate probabilistic real-time predictions of confirmed cases and deaths from COVID-19 in Germany and Poland. These were issued by 15 different forecasting models, run by independent research teams. Moreover, we study the performance of combined ensemble forecasts. Evaluation of probabilistic forecasts is based on proper scoring rules, along with interval coverage proportions to assess calibration. The presented work is part of a pre-registered evaluation study.

RESULTS

We find that many, though not all, models outperform a simple baseline model up to four weeks ahead for the considered targets. Ensemble methods show very good relative performance. The addressed time period is characterized by rather stable non-pharmaceutical interventions in both countries, making short-term predictions more straightforward than in previous periods. However, major trend changes in reported cases, like the rebound in cases due to the rise of the B.1.1.7 (Alpha) variant in March 2021, prove challenging to predict.

CONCLUSIONS

Multi-model approaches can help to improve the performance of epidemiological forecasts. However, while death numbers can be predicted with some success based on current case and hospitalization data, predictability of case numbers remains low beyond quite short time horizons. Additional data sources including sequencing and mobility data, which were not extensively used in the present study, may help to improve performance.

An ensemble n-sub-epidemic modeling framework for short-term forecasting epidemic trajectories: Application to the COVID-19 pandemic in the USA

We analyze an ensemble of n-sub-epidemic modeling for forecasting the trajectory of epidemics and pandemics. These ensemble modeling approaches, and models that integrate sub-epidemics to capture complex temporal dynamics, have demonstrated powerful forecasting capability. This modeling framework can characterize complex epidemic patterns, including plateaus, epidemic resurgences, and epidemic waves characterized by multiple peaks of different sizes. We systematically assess their calibration and short-term forecasting performance in short-term forecasts for the COVID-19 pandemic in the USA from late April 2020 to late February 2022. We compare their performance with two commonly used statistical ARIMA models. The best fit sub-epidemic model and three ensemble models constructed using the top-ranking sub-epidemic models consistently outperformed the ARIMA models in terms of the weighted interval score (WIS) and the coverage of the 95% prediction interval across the 10-, 20-, and 30-day short-term forecasts. In our 30-day forecasts, the average WIS ranged from 377.6 to 421.3 for the sub-epidemic models, whereas it ranged from 439.29 to 767.05 for the ARIMA models. Across 98 short-term forecasts, the ensemble model incorporating the top four ranking sub-epidemic models (Ensemble(4)) outperformed the (log) ARIMA model 66.3% of the time, and the ARIMA model, 69.4% of the time in 30-day ahead forecasts in terms of the WIS. Ensemble(4) consistently yielded the best performance in terms of the metrics that account for the uncertainty of the predictions. This framework can be readily applied to investigate the spread of epidemics and pandemics beyond COVID-19, as well as other dynamic growth processes found in nature and society that would benefit from short-term predictions.

How heterogeneous is the dengue transmission profile in Brazil? A study in six Brazilian states

Dengue is a vector-borne disease present in most tropical countries, infecting an average of 50 to 100 million people per year. Socioeconomic, demographic, and environmental factors directly influence the transmission cycle of the dengue virus (DENV). In Brazil, these factors vary between regions producing different profiles of dengue transmission and challenging the epidemiological surveillance of the disease. In this article, we aimed at classifying the profiles of dengue transmission in 1,823 Brazilian municipalities, covering different climates, from 2010 to 2019. Time series data of dengue cases were obtained from six states: Ceará and Maranhão in the semiarid Northeast, Minas Gerais in the countryside, Espírito Santo and Rio de Janeiro in the tropical Atlantic coast, and Paraná in the subtropical region. To describe the time series, we proposed a set of epi-features of the magnitude and duration of the dengue epidemic cycles, totaling 13 indicators. Using these epi-features as inputs, a multivariate cluster algorithm was employed to classify the municipalities according to their dengue transmission profile. Municipalities were classified into four distinct dengue transmission profiles: persistent transmission (7.8%), epidemic (21.3%), episodic/epidemic (43.2%), and episodic transmission (27.6%). Different profiles were associated with the municipality’s population size and climate. Municipalities with higher incidence and larger populations tended to be classified as persistent transmission, suggesting the existence of critical community size. This association, however, varies depending on the state, indicating the importance of other factors. The proposed classification is useful for developing more specific and precise surveillance protocols for regions with different dengue transmission profiles, as well as more precise public policies for dengue prevention.

Dengue is one of the fastest-growing vector-borne diseases in the world. Currently, vaccines are experimental and are not very effective, so prevention depends on the control of the mosquito Aedes aegypti. Health promotion campaigns aimed at encouraging people to reduce mosquito breeding sites have limited effect. In addition, the heterogeneity of the territories that have dengue becomes a major challenge for the epidemiological surveillance of the disease. Brazil has a territory of continental size, and single standardized surveillance is not very effective for monitoring this arbovirus. Classifying types of dengue dynamics based on features of the epidemiological cycle in each location has the potential to increase the precision of surveillance and control strategies. In our study, we were able to classify areas according to different dengue transmission profiles, ranging from episodic to persistent transmission. These results can provide tools to guide actions aimed at achieving the World Health Organization’s goals of eliminating neglected tropical diseases in countries that have the virus.

Stochastic modelling of African swine fever in wild boar and domestic pigs: Epidemic forecasting and comparison of disease management strategies

African swine fever (ASF), caused by the African swine fever virus (ASFV), is highly virulent in domestic pigs and wild boar (Sus scrofa), causing up to 100% mortality. The recent epidemic of ASF in Europe has had a serious economic impact and poses a threat to global food security. Unfortunately, there is no effective treatment or vaccine against ASFV, limiting the available disease management strategies. Mathematical models allow us to further our understanding of infectious disease dynamics and evaluate the efficacy of disease management strategies. The ASF Challenge, organised by the French National Research Institute for Agriculture, Food, and the Environment, aimed to expand the development of ASF transmission models to inform policy makers in a timely manner. Here, we present the model and associated projections produced by our team during the challenge. We developed a stochastic model combining transmission between wild boar and domestic pigs, which was calibrated to synthetic data corresponding to different phases describing the epidemic progression. The model was then used to produce forward projections describing the likely temporal evolution of the epidemic under various disease management scenarios. Despite the interventions implemented, long-term projections forecasted persistence of ASFV in wild boar, and hence repeated outbreaks in domestic pigs. A key finding was that it is important to consider the timescale over which different measures are evaluated: interventions that have only limited effectiveness in the short term may yield substantial long-term benefits. Our model has several limitations, partly because it was developed in real-time. Nonetheless, it can inform understanding of the likely development of ASF epidemics and the efficacy of disease management strategies, should the virus continue its spread in Europe.

Comparing human and model-based forecasts of COVID-19 in Germany and Poland

Forecasts based on epidemiological modelling have played an important role in shaping public policy throughout the COVID-19 pandemic. This modelling combines knowledge about infectious disease dynamics with the subjective opinion of the researcher who develops and refines the model and often also adjusts model outputs. Developing a forecast model is difficult, resource- and time-consuming. It is therefore worth asking what modelling is able to add beyond the subjective opinion of the researcher alone. To investigate this, we analysed different real-time forecasts of cases of and deaths from COVID-19 in Germany and Poland over a 1-4 week horizon submitted to the German and Polish Forecast Hub. We compared crowd forecasts elicited from researchers and volunteers, against a) forecasts from two semi-mechanistic models based on common epidemiological assumptions and b) the ensemble of all other models submitted to the Forecast Hub. We found crowd forecasts, despite being overconfident, to outperform all other methods across all forecast horizons when forecasting cases (weighted interval score relative to the Hub ensemble 2 weeks ahead: 0.89). Forecasts based on computational models performed comparably better when predicting deaths (rel. WIS 1.26), suggesting that epidemiological modelling and human judgement can complement each other in important ways.

The African swine fever modelling challenge: Model comparison and lessons learnt

Robust epidemiological knowledge and predictive modelling tools are needed to address challenging objectives, such as: understanding epidemic drivers; forecasting epidemics; and prioritising control measures. Often, multiple modelling approaches can be used during an epidemic to support effective decision making in a timely manner. Modelling challenges contribute to understanding the pros and cons of different approaches and to fostering technical dialogue between modellers. In this paper, we present the results of the first modelling challenge in animal health - the ASF Challenge - which focused on a synthetic epidemic of African swine fever (ASF) on an island. The modelling approaches proposed by five independent international teams were compared. We assessed their ability to predict temporal and spatial epidemic expansion at the interface between domestic pigs and wild boar, and to prioritise a limited number of alternative interventions. We also compared their qualitative and quantitative spatio-temporal predictions over the first two one-month projection phases of the challenge. Top-performing models in predicting the ASF epidemic differed according to the challenge phase, host species, and in predicting spatial or temporal dynamics. Ensemble models built using all team-predictions outperformed any individual model in at least one phase. The ASF Challenge demonstrated that accounting for the interface between livestock and wildlife is key to increasing our effectiveness in controlling emerging animal diseases, and contributed to improving the readiness of the scientific community to face future ASF epidemics. Finally, we discuss the lessons learnt from model comparison to guide decision making.

Predictable Chikungunya Infection Dynamics in Brazil

Chikungunya virus (CHIKV) was first imported into the Caribbean in 2013 and subsequently spread across the Americas. It has infected millions in the region and Brazil has become the hub of ongoing transmission. Using Seasonal Autoregressive Integrated Moving Average (SARIMA) models trained and validated on Brazilian data from the Ministry of Health’s notifiable diseases information system, we tested the hypothesis that transmission in Brazil had transitioned from sporadic and explosive to become more predictable. Consistency weighted, population standardized kernel density estimates were used to identify municipalities with the most consistent inter-annual transmission rates. Spatial clustering was assessed per calendar month for 2017−2021 inclusive using Moran’s I. SARIMA models were validated on 2020−2021 data and forecasted 106,162 (95%CI 27,303−200,917) serologically confirmed cases and 339,907 (95%CI 35,780−1035,449) total notifications for 2022−2023 inclusive, with >90% of cases in the Northeast and Southeast regions. Comparing forecasts for the first five months of 2022 to the most up-to-date ECDC report (published 2 June 2022) showed remarkable accuracy: the models predicted 92,739 (95%CI 20,685−195,191) case notifications during which the ECDC reported 92,349 case notifications. Hotspots of consistent transmission were identified in the states of Para and Tocantins (North region); Rio Grande do Norte, Paraiba and Pernambuco (Northeast region); and Rio de Janeiro and eastern Minas Gerais (Southeast region). Significant spatial clustering peaked during late summer/early autumn. This analysis highlights how CHIKV transmission in Brazil has transitioned, making it more predictable and thus enabling improved control targeting and site selection for trialing interventions.