Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States

Wastewater-based surveillance of SARS-CoV-2: Short-term projection (forecasting), smoothing and outlier identification using Bayesian smoothing

BACKGROUND

Day-to-day variation in the measurement of SARS-CoV-2 in wastewater can challenge public health interpretation. We assessed a Bayesian smoothing and forecasting method previously used for surveillance and short-term projection of COVID-19 cases, hospitalizations, and deaths.

METHODS

SARS-CoV-2 viral measurement from the sewershed in Ottawa, Canada, sampled at the municipal wastewater treatment plant from July 1, 2020, to February 15, 2022, was used to assess and internally validate measurement averaging and prediction. External validation was performed using viral measurement data from influent wastewater samples from 15 wastewater treatment plants and municipalities across Ontario.

RESULTS

Plots of SARS-CoV-2 viral measurement over time using Bayesian smoothing visually represented distinct COVID-19 "waves" described by case and hospitalization data in both initial (Ottawa) and external validation in 15 Ontario communities. The time-varying growth rate of viral measurement in wastewater samples approximated the growth rate observed for cases and hospitalization. One-week predicted viral measurement approximated the observed viral measurement throughout the assessment period from December 23, 2020, to August 8, 2022. An uncalibrated model showed underprediction during rapid increases in viral measurement (positive growth) and overprediction during rapid decreases. After recalibration, the model showed a close approximation between observed and predicted estimates.

CONCLUSION

Bayesian smoothing of wastewater surveillance data of SARS-CoV-2 allows for accurate estimates of COVID-19 growth rates and one- and two-week forecasting of SARS-CoV-2 in wastewater for 16 municipalities in Ontario, Canada. Further assessment is warranted in other communities representing different sewersheds and environmental conditions.

An adaptive weight ensemble approach to forecast influenza activity in an irregular seasonality context

Forecasting influenza activity in tropical and subtropical regions, such as Hong Kong, is challenging due to irregular seasonality and high variability. We develop a diverse set of statistical, machine learning, and deep learning approaches to forecast influenza activity in Hong Kong 0 to 8 weeks ahead, leveraging a unique multi-year surveillance record spanning 32 epidemics from 1998 to 2019. We consider a simple average ensemble (SAE) of the top two individual models, and develop an adaptive weight blending ensemble (AWBE) that dynamically updates model contribution. All models outperform the baseline constant incidence model, reducing the root mean square error (RMSE) by 23%-29% and weighted interval score (WIS) by 25%-31% for 8-week ahead forecasts. The SAE model performed similarly to individual models, while the AWBE model reduces RMSE by 52% and WIS by 53%, outperforming individual models for forecasts in different epidemic trends (growth, plateau, decline) and during both winter and summer seasons. Using the post-COVID data (2023-2024) as another test period, the AWBE model still reduces RMSE by 39% and WIS by 45%. Our framework contributes to comparing and benchmarking models in ensemble forecasts, enhancing evidence for synthesizing multiple models in disease forecasting for geographies with irregular influenza seasonality.

Expanding the use of mathematical modeling in healthcare epidemiology and infection prevention and control

During the coronavirus disease 2019 pandemic, mathematical modeling has been widely used to understand epidemiological burden, trends, and transmission dynamics, to facilitate policy decisions, and, to a lesser extent, to evaluate infection prevention and control (IPC) measures. This review highlights the added value of using conventional epidemiology and modeling approaches to address the complexity of healthcare-associated infections (HAI) and antimicrobial resistance. It demonstrates how epidemiological surveillance data and modeling can be used to infer transmission dynamics in healthcare settings and to forecast healthcare impact, how modeling can be used to improve the validity of interpretation of epidemiological surveillance data, how modeling can be used to estimate the impact of IPC interventions, and how modeling can be used to guide IPC and antimicrobial treatment and stewardship decision-making. There are several priority areas for expanding the use of modeling in healthcare epidemiology and IPC. Importantly, modeling should be viewed as complementary to conventional healthcare epidemiological approaches, and this requires collaboration and active coordination between IPC, healthcare epidemiology, and mathematical modeling groups.

Optimizing Disease Outbreak Forecast Ensembles

On the basis of historical influenza and COVID-19 forecasts, we found that more than 3 forecast models are needed to ensure robust ensemble accuracy. Additional models can improve ensemble performance, but with diminishing accuracy returns. This understanding will assist with the design of current and future collaborative infectious disease forecasting efforts.

Inference of epidemic dynamics in the COVID-19 era and beyond

The COVID-19 pandemic demonstrated the key role that epidemiology and modelling play in analysing infectious threats and supporting decision making in real-time. Motivated by the unprecedented volume and breadth of data generated during the pandemic, we review modern opportunities for analysis to address questions that emerge during a major modern epidemic. Following the broad chronology of insights required - from understanding initial dynamics to retrospective evaluation of interventions, we describe the theoretical foundations of each approach and the underlying intuition. Through a series of case studies, we illustrate real life applications, and discuss implications for future work.

Title evaluation of FluSight influenza forecasting in the 2021-22 and 2022-23 seasons with a new target laboratory-confirmed influenza hospitalizations

Accurate forecasts can enable more effective public health responses during seasonal influenza epidemics. For the 2021-22 and 2022-23 influenza seasons, 26 forecasting teams provided national and jurisdiction-specific probabilistic predictions of weekly confirmed influenza hospital admissions for one-to-four weeks ahead. Forecast skill is evaluated using the Weighted Interval Score (WIS), relative WIS, and coverage. Six out of 23 models outperform the baseline model across forecast weeks and locations in 2021-22 and 12 out of 18 models in 2022-23. Averaging across all forecast targets, the FluSight ensemble is the 2nd most accurate model measured by WIS in 2021-22 and the 5th most accurate in the 2022-23 season. Forecast skill and 95% coverage for the FluSight ensemble and most component models degrade over longer forecast horizons. In this work we demonstrate that while the FluSight ensemble was a robust predictor, even ensembles face challenges during periods of rapid change.

Learning dynamical systems from data: An introduction to physics-guided deep learning

Modeling complex physical dynamics is a fundamental task in science and engineering. Traditional physics-based models are first-principled, explainable, and sample-efficient. However, they often rely on strong modeling assumptions and expensive numerical integration, requiring significant computational resources and domain expertise. While deep learning (DL) provides efficient alternatives for modeling complex dynamics, they require a large amount of labeled training data. Furthermore, its predictions may disobey the governing physical laws and are difficult to interpret. Physics-guided DL aims to integrate first-principled physical knowledge into data-driven methods. It has the best of both worlds and is well equipped to better solve scientific problems. Recently, this field has gained great progress and has drawn considerable interest across discipline Here, we introduce the framework of physics-guided DL with a special emphasis on learning dynamical systems. We describe the learning pipeline and categorize state-of-the-art methods under this framework. We also offer our perspectives on the open challenges and emerging opportunities.

Evaluating the forecasting performance of ensemble sub-epidemic frameworks and other time series models for the 2022-2023 mpox epidemic

During the 2022-2023 unprecedented mpox epidemic, near real-time short-term forecasts of the epidemic's trajectory were essential in intervention implementation and guiding policy. However, as case levels have significantly decreased, evaluating model performance is vital to advancing the field of epidemic forecasting. Using laboratory-confirmed mpox case data from the Centers for Disease Control and Prevention and Our World in Data teams, we generated retrospective sequential weekly forecasts for Brazil, Canada, France, Germany, Spain, the United Kingdom, the United States and at the global scale using an auto-regressive integrated moving average (ARIMA) model, generalized additive model, simple linear regression, Facebook's Prophet model, as well as the sub-epidemic wave and n-sub-epidemic modelling frameworks. We assessed forecast performance using average mean squared error, mean absolute error, weighted interval scores, 95% prediction interval coverage, skill scores and Winkler scores. Overall, the n-sub-epidemic modelling framework outcompeted other models across most locations and forecasting horizons, with the unweighted ensemble model performing best most frequently. The n-sub-epidemic and spatial-wave frameworks considerably improved in average forecasting performance relative to the ARIMA model (greater than 10%) for all performance metrics. Findings further support sub-epidemic frameworks for short-term forecasting epidemics of emerging and re-emerging infectious diseases.

hubEnsembles: Ensembling Methods in R

Combining predictions from multiple models into an ensemble is a widely used practice across many fields with demonstrated performance benefits. The R package hubEnsembles provides a flexible framework for ensembling various types of predictions, including point estimates and probabilistic predictions. A range of common methods for generating ensembles are supported, including weighted averages, quantile averages, and linear pools. The hubEnsembles package fits within a broader framework of open-source software and data tools called the "hubverse", which facilitates the development and management of collaborative modelling exercises.

The disutility of compartmental model forecasts during the COVID-19 pandemic

During the COVID-19 pandemic, several forecasting models were released to predict the spread of the virus along variables vital for public health policymaking. Of these, the susceptible-infected-recovered (SIR) compartmental model was the most common. In this paper, we investigated the forecasting performance of The University of Texas COVID-19 Modeling Consortium SIR model. We considered the following daily outcomes: hospitalizations, ICU patients, and deaths. We evaluated the overall forecasting performance, highlighted some stark forecast biases, and considered forecast errors conditional on different pandemic regimes. We found that this model tends to overforecast over the longer horizons and when there is a surge in viral spread. We bolstered these findings by linking them to faults with the SIR framework itself.

SpatialWavePredict: a tutorial-based primer and toolbox for forecasting growth trajectories using the ensemble spatial wave sub-epidemic modeling framework

BACKGROUND

Dynamical mathematical models defined by a system of differential equations are typically not easily accessible to non-experts. However, forecasts based on these types of models can help gain insights into the mechanisms driving the process and may outcompete simpler phenomenological growth models. Here we introduce a friendly toolbox, SpatialWavePredict, to characterize and forecast the spatial wave sub-epidemic model, which captures diverse wave dynamics by aggregating multiple asynchronous growth processes and has outperformed simpler phenomenological growth models in short-term forecasts of various infectious diseases outbreaks including SARS, Ebola, and the early waves of the COVID-19 pandemic in the US.

RESULTS

This tutorial-based primer introduces and illustrates a user-friendly MATLAB toolbox for fitting and forecasting time-series trajectories using an ensemble spatial wave sub-epidemic model based on ordinary differential equations. Scientists, policymakers, and students can use the toolbox to conduct real-time short-term forecasts. The five-parameter epidemic wave model in the toolbox aggregates linked overlapping sub-epidemics and captures a rich spectrum of epidemic wave dynamics, including oscillatory wave behavior and plateaus. An ensemble strategy aims to improve forecasting performance by combining the resulting top-ranked models. The toolbox provides a tutorial for forecasting time-series trajectories, including the full uncertainty distribution derived through parametric bootstrapping, which is needed to construct prediction intervals and evaluate their accuracy. Functions are available to assess forecasting performance, estimation methods, error structures in the data, and forecasting horizons. The toolbox also includes functions to quantify forecasting performance using metrics that evaluate point and distributional forecasts, including the weighted interval score.

CONCLUSIONS

We have developed the first comprehensive toolbox to characterize and forecast time-series data using an ensemble spatial wave sub-epidemic wave model. As an epidemic situation or contagion occurs, the tools presented in this tutorial can facilitate policymakers to guide the implementation of containment strategies and assess the impact of control interventions. We demonstrate the functionality of the toolbox with examples, including a tutorial video, and is illustrated using daily data on the COVID-19 pandemic in the USA.

Scenario design for infectious disease projections: Integrating concepts from decision analysis and experimental design

Across many fields, scenario modeling has become an important tool for exploring long-term projections and how they might depend on potential interventions and critical uncertainties, with relevance to both decision makers and scientists. In the past decade, and especially during the COVID-19 pandemic, the field of epidemiology has seen substantial growth in the use of scenario projections. Multiple scenarios are often projected at the same time, allowing important comparisons that can guide the choice of intervention, the prioritization of research topics, or public communication. The design of the scenarios is central to their ability to inform important questions. In this paper, we draw on the fields of decision analysis and statistical design of experiments to propose a framework for scenario design in epidemiology, with relevance also to other fields. We identify six different fundamental purposes for scenario designs (decision making, sensitivity analysis, situational awareness, horizon scanning, forecasting, and value of information) and discuss how those purposes guide the structure of scenarios. We discuss other aspects of the content and process of scenario design, broadly for all settings and specifically for multi-model ensemble projections. As an illustrative case study, we examine the first 17 rounds of scenarios from the U.S. COVID-19 Scenario Modeling Hub, then reflect on future advancements that could improve the design of scenarios in epidemiological settings.

Characterising information gains and losses when collecting multiple epidemic model outputs

BACKGROUND

Collaborative comparisons and combinations of epidemic models are used as policy-relevant evidence during epidemic outbreaks. In the process of collecting multiple model projections, such collaborations may gain or lose relevant information. Typically, modellers contribute a probabilistic summary at each time-step. We compared this to directly collecting simulated trajectories. We aimed to explore information on key epidemic quantities; ensemble uncertainty; and performance against data, investigating potential to continuously gain information from a single cross-sectional collection of model results.

METHODS

We compared projections from the European COVID-19 Scenario Modelling Hub. Five teams modelled incidence in Belgium, the Netherlands, and Spain. We compared July 2022 projections by incidence, peaks, and cumulative totals. We created a probabilistic ensemble drawn from all trajectories, and compared to ensembles from a median across each model's quantiles, or a linear opinion pool. We measured the predictive accuracy of individual trajectories against observations, using this in a weighted ensemble. We repeated this sequentially against increasing weeks of observed data. We evaluated these ensembles to reflect performance with varying observed data.

RESULTS

By collecting modelled trajectories, we showed policy-relevant epidemic characteristics. Trajectories contained a right-skewed distribution well represented by an ensemble of trajectories or a linear opinion pool, but not models' quantile intervals. Ensembles weighted by performance typically retained the range of plausible incidence over time, and in some cases narrowed this by excluding some epidemic shapes.

CONCLUSIONS

We observed several information gains from collecting modelled trajectories rather than quantile distributions, including potential for continuously updated information from a single model collection. The value of information gains and losses may vary with each collaborative effort's aims, depending on the needs of projection users. Understanding the differing information potential of methods to collect model projections can support the accuracy, sustainability, and communication of collaborative infectious disease modelling efforts.

SubEpiPredict: A tutorial-based primer and toolbox for fitting and forecasting growth trajectories using the ensemble n-sub-epidemic modeling framework

An ensemble n-sub-epidemic modeling framework that integrates sub-epidemics to capture complex temporal dynamics has demonstrated powerful forecasting capability in previous works. This modeling framework can characterize complex epidemic patterns, including plateaus, epidemic resurgences, and epidemic waves characterized by multiple peaks of different sizes. In this tutorial paper, we introduce and illustrate SubEpiPredict, a user-friendly MATLAB toolbox for fitting and forecasting time series data using an ensemble n-sub-epidemic modeling framework. The toolbox can be used for model fitting, forecasting, and evaluation of model performance of the calibration and forecasting periods using metrics such as the weighted interval score (WIS). We also provide a detailed description of these methods including the concept of the n-sub-epidemic model, constructing ensemble forecasts from the top-ranking models, etc. For the illustration of the toolbox, we utilize publicly available daily COVID-19 death data at the national level for the United States. The MATLAB toolbox introduced in this paper can be very useful for a wider group of audiences, including policymakers, and can be easily utilized by those without extensive coding and modeling backgrounds.

When do we need multiple infectious disease models? Agreement between projection rank and magnitude in a multi-model setting

Mathematical models are useful for public health planning and response to infectious disease threats. However, different models can provide differing results, which can hamper decision making if not synthesized appropriately. To address this challenge, multi-model hubs convene independent modeling groups to generate ensembles, known to provide more accurate predictions of future outcomes. Yet, these hubs are resource intensive, and how many models are sufficient in a hub is not known. Here, we compare the benefit of predictions from multiple models in different contexts: (1) decision settings that depend on predictions of quantitative outcomes (e.g., hospital capacity planning), where assessments of the benefits of multi-model ensembles have largely focused; and (2) decisions settings that require the ranking of alternative epidemic scenarios (e.g., comparing outcomes under multiple possible interventions and biological uncertainties). We develop a mathematical framework to mimic a multi-model prediction setting, and use this framework to quantify how frequently predictions from different models agree. We further explore multi-model agreement using real-world, empirical data from 14 rounds of U.S. COVID-19 Scenario Modeling Hub projections. Our results suggest that the value of multiple models could be different in different decision contexts, and if only a few models are available, focusing on the rank of alternative epidemic scenarios could be more robust than focusing on quantitative outcomes. Although additional exploration of the sufficient number of models for different contexts is still needed, our results indicate that it may be possible to identify decision contexts where it is robust to rely on fewer models, a finding that can inform the use of modeling resources during future public health crises.

Chimeric Forecasting: An experiment to leverage human judgment to improve forecasts of infectious disease using simulated surveillance data

Forecasts of infectious agents provide public health officials advanced warning about the intensity and timing of the spread of disease. Past work has found that accuracy and calibration of forecasts is weakest when attempting to predict an epidemic peak. Forecasts from a mechanistic model would be improved if there existed accurate information about the timing and intensity of an epidemic. We presented 3000 humans with simulated surveillance data about the number of incident hospitalizations from a current and two past seasons, and asked that they predict the peak time and intensity of the underlying epidemic. We found that in comparison to two control models, a model including human judgment produced more accurate forecasts of peak time and intensity of hospitalizations during an epidemic. Chimeric models have the potential to improve our ability to predict targets of public health interest which may in turn reduce infectious disease burden.

Projecting the future impact of emerging SARS-CoV-2 variants under uncertainty: Modeling the initial Omicron outbreak

Over the past several years, the emergence of novel SARS-CoV-2 variants has led to multiple waves of increased COVID-19 incidence. When the Omicron variant emerged, there was considerable concern about its potential impact in the winter of 2021-2022 due to its increased fitness. However, there was also considerable uncertainty regarding its likely impact due to questions about its relative transmissibility, severity, and degree of immune escape. We sought to evaluate the ability of an agent-based model to forecast incidence in the context of this emerging pathogen variant. To project COVID-19 cases and deaths in Indiana, we calibrated our model to COVID-19 hospitalizations, deaths, and test-positivity rates through November 2021, and then projected COVID-19 incidence through April 2022 under four different scenarios that covered the plausible ranges of Omicron's severity, transmissibility, and degree of immune escape. Our initial projections from December 2021 through March 2022 indicated that under a pessimistic scenario with high disease severity, the peak in weekly COVID-19 deaths in Indiana would be larger than the previous peak in December 2020. However, retrospective analyses indicate that Omicron's severity was closer to the optimistic scenario, and even though cases and hospitalizations reached a new peak, fewer deaths occurred than during the previous peak. According to our results, Omicron's rapid spread was consistent with a combination of higher transmissibility and immune escape relative to earlier variants. Our updated projections starting in January 2022 accurately predicted that cases would peak in mid-January and decline rapidly over the next several months. The performance of our projections shows that following the emergence of a new pathogen variant, models can help quantify the potential range of outbreak magnitudes and trajectories. Agent-based models are particularly useful in these scenarios because they can efficiently track individual vaccination and infection histories with multiple variants with varying degrees of cross-protection.

flepiMoP: The evolution of a flexible infectious disease modeling pipeline during the COVID-19 pandemic

The COVID-19 pandemic led to an unprecedented demand for projections of disease burden and healthcare utilization under scenarios ranging from unmitigated spread to strict social distancing policies. In response, members of the Johns Hopkins Infectious Disease Dynamics Group developed flepiMoP (formerly called the COVID Scenario Modeling Pipeline), a comprehensive open-source software pipeline designed for creating and simulating compartmental models of infectious disease transmission and inferring parameters through these models. The framework has been used extensively to produce short-term forecasts and longer-term scenario projections of COVID-19 at the state and county level in the US, for COVID-19 in other countries at various geographic scales, and more recently for seasonal influenza. In this paper, we highlight how the flepiMoP has evolved throughout the COVID-19 pandemic to address changing epidemiological dynamics, new interventions, and shifts in policy-relevant model outputs. As the framework has reached a mature state, we provide a detailed overview of flepiMoP's key features and remaining limitations, thereby distributing flepiMoP and its documentation as a flexible and powerful tool for researchers and public health professionals to rapidly build and deploy large-scale complex infectious disease models for any pathogen and demographic setup.

Reporting delays: A widely neglected impact factor in COVID-19 forecasts

Epidemic forecasts are only as good as the accuracy of epidemic measurements. Is epidemic data, particularly COVID-19 epidemic data, clean, and devoid of noise? The complexity and variability inherent in data collection and reporting suggest otherwise. While we cannot evaluate the integrity of the COVID-19 epidemic data in a holistic fashion, we can assess the data for the presence of reporting delays. In our work, through the analysis of the first COVID-19 wave, we find substantial reporting delays in the published epidemic data. Motivated by the desire to enhance epidemic forecasts, we develop a statistical framework to detect, uncover, and remove reporting delays in the infectious, recovered, and deceased epidemic time series. Using our framework, we expose and analyze reporting delays in eight regions significantly affected by the first COVID-19 wave. Further, we demonstrate that removing reporting delays from epidemic data by using our statistical framework may decrease the error in epidemic forecasts. While our statistical framework can be used in combination with any epidemic forecast method that intakes infectious, recovered, and deceased data, to make a basic assessment, we employed the classical SIRD epidemic model. Our results indicate that the removal of reporting delays from the epidemic data may decrease the forecast error by up to 50%. We anticipate that our framework will be indispensable in the analysis of novel COVID-19 strains and other existing or novel infectious diseases.

Nowcasting and forecasting the 2022 U.S. mpox outbreak: Support for public health decision making and lessons learned

In June of 2022, the U.S. Centers for Disease Control and Prevention (CDC) Mpox Response wanted timely answers to important epidemiological questions which can now be answered more effectively through infectious disease modeling. Infectious disease models have shown to be valuable tools for decision making during outbreaks; however, model complexity often makes communicating the results and limitations of models to decision makers difficult. We performed nowcasting and forecasting for the 2022 mpox outbreak in the United States using the R package EpiNow2. We generated nowcasts/forecasts at the national level, by Census region, and for jurisdictions reporting the greatest number of mpox cases. Modeling results were shared for situational awareness within the CDC Mpox Response and publicly on the CDC website. We retrospectively evaluated forecast predictions at four key phases (early, exponential growth, peak, and decline) during the outbreak using three metrics, the weighted interval score, mean absolute error, and prediction interval coverage. We compared the performance of EpiNow2 with a naïve Bayesian generalized linear model (GLM). The EpiNow2 model had less probabilistic error than the GLM during every outbreak phase except for the early phase. We share our experiences with an existing tool for nowcasting/forecasting and highlight areas of improvement for the development of future tools. We also reflect on lessons learned regarding data quality issues and adapting modeling results for different audiences.

An inaugural forum on epidemiological modeling for public health stakeholders in Arizona

Epidemiological models-which help us understand and forecast the spread of infectious disease-can be valuable tools for public health. However, barriers exist that can make it difficult to employ epidemiological models routinely within the repertoire of public health planning. These barriers include technical challenges associated with constructing the models, challenges in obtaining appropriate data for model parameterization, and problems with clear communication of modeling outputs and uncertainty. To learn about the unique barriers and opportunities within the state of Arizona, we gathered a diverse set of 48 public health stakeholders for a day-and-a-half forum. Our research group was motivated specifically by our work building software for public health-relevant modeling and by our earnest desire to collaborate closely with stakeholders to ensure that our software tools are practical and useful in the face of evolving public health needs. Here we outline the planning and structure of the forum, and we highlight as a case study some of the lessons learned from breakout discussions. While unique barriers exist for implementing modeling for public health, there is also keen interest in doing so across diverse sectors of State and Local government, although issues of equal and fair access to modeling knowledge and technologies remain key issues for future development. We found this forum to be useful for building relationships and informing our software development, and we plan to continue such meetings annually to create a continual feedback loop between academic molders and public health practitioners.

Predicting COVID-19 and respiratory illness: results of the 2022-2023 Armed Forces Health Surveillance Division forecasting challenge

Since 2019, the Integrated Biosurveillance Branch of the Armed Forces Health Surveillance Division has conducted an annual forecasting challenge during influenza season to predict short-term respiratory disease activity among Military Health System beneficiaries. Weekly case and encounter observed data were used to generate 1- through 4-week advanced forecasts of disease activity. To create unified combinations of model inputs for evaluation across multiple spatial resolutions, 8 individual models were used to calculate 3 ensemble models. Forecast accuracy compared to the observed activity for each model was evaluated by calculating a weighted interval score. Weekly 1- through 4-week ahead forecasts for each ensemble model were generally higher than observed data, especially during periods of peak activity, with peaks in forecasted activity occurring later than observed peaks. The larger the forecasting horizon, the more pronounced the gap between forecasted peak and observed peak. The results showed that several models accurately predicted COVID-19 cases and respiratory encounters with enough lead time for public health response by senior leaders.

Country-report pattern corrections of new cases allow accurate 2-week predictions of COVID-19 evolution with the Gompertz model

Accurate short-term predictions of COVID-19 cases with empirical models allow Health Officials to prepare for hospital contingencies in a two-three week window given the delay between case reporting and the admission of patients in a hospital. We investigate the ability of Gompertz-type empiric models to provide accurate prediction up to two and three weeks to give a large window of preparation in case of a surge in virus transmission. We investigate the stability of the prediction and its accuracy using bi-weekly predictions during the last trimester of 2020 and 2021. Using data from 2020, we show that understanding and correcting for the daily reporting structure of cases in the different countries is key to accomplish accurate predictions. Furthermore, we found that filtering out predictions that are highly unstable to changes in the parameters of the model, which are roughly 20%, reduces strongly the number of predictions that are way-off. The method is then tested for robustness with data from 2021. We found that, for this data, only 1-2% of the one-week predictions were off by more than 50%. This increased to 3% for two-week predictions, and only for three-week predictions it reached 10%.

Combining models to generate consensus medium-term projections of hospital admissions, occupancy and deaths relating to COVID-19 in England

Mathematical modelling has played an important role in offering informed advice during the COVID-19 pandemic. In England, a cross government and academia collaboration generated medium-term projections (MTPs) of possible epidemic trajectories over the future 4-6 weeks from a collection of epidemiological models. In this article, we outline this collaborative modelling approach and evaluate the accuracy of the combined and individual model projections against the data over the period November 2021-December 2022 when various Omicron subvariants were spreading across England. Using a number of statistical methods, we quantify the predictive performance of the model projections for both the combined and individual MTPs, by evaluating the point and probabilistic accuracy. Our results illustrate that the combined MTPs, produced from an ensemble of heterogeneous epidemiological models, were a closer fit to the data than the individual models during the periods of epidemic growth or decline, with the 90% confidence intervals widest around the epidemic peaks. We also show that the combined MTPs increase the robustness and reduce the biases associated with a single model projection. Learning from our experience of ensemble modelling during the COVID-19 epidemic, our findings highlight the importance of developing cross-institutional multi-model infectious disease hubs for future outbreak control.

Challenges of COVID-19 Case Forecasting in the US, 2020-2021

During the COVID-19 pandemic, forecasting COVID-19 trends to support planning and response was a priority for scientists and decision makers alike. In the United States, COVID-19 forecasting was coordinated by a large group of universities, companies, and government entities led by the Centers for Disease Control and Prevention and the US COVID-19 Forecast Hub (https://covid19forecasthub.org). We evaluated approximately 9.7 million forecasts of weekly state-level COVID-19 cases for predictions 1-4 weeks into the future submitted by 24 teams from August 2020 to December 2021. We assessed coverage of central prediction intervals and weighted interval scores (WIS), adjusting for missing forecasts relative to a baseline forecast, and used a Gaussian generalized estimating equation (GEE) model to evaluate differences in skill across epidemic phases that were defined by the effective reproduction number. Overall, we found high variation in skill across individual models, with ensemble-based forecasts outperforming other approaches. Forecast skill relative to the baseline was generally higher for larger jurisdictions (e.g., states compared to counties). Over time, forecasts generally performed worst in periods of rapid changes in reported cases (either in increasing or decreasing epidemic phases) with 95% prediction interval coverage dropping below 50% during the growth phases of the winter 2020, Delta, and Omicron waves. Ideally, case forecasts could serve as a leading indicator of changes in transmission dynamics. However, while most COVID-19 case forecasts outperformed a naïve baseline model, even the most accurate case forecasts were unreliable in key phases. Further research could improve forecasts of leading indicators, like COVID-19 cases, by leveraging additional real-time data, addressing performance across phases, improving the characterization of forecast confidence, and ensuring that forecasts were coherent across spatial scales. In the meantime, it is critical for forecast users to appreciate current limitations and use a broad set of indicators to inform pandemic-related decision making.

Parameter estimation and forecasting with quantified uncertainty for ordinary differential equation models using QuantDiffForecast: A MATLAB toolbox and tutorial

Mathematical models based on systems of ordinary differential equations (ODEs) are frequently applied in various scientific fields to assess hypotheses, estimate key model parameters, and generate predictions about the system's state. To support their application, we present a comprehensive, easy-to-use, and flexible MATLAB toolbox, QuantDiffForecast, and associated tutorial to estimate parameters and generate short-term forecasts with quantified uncertainty from dynamical models based on systems of ODEs. We provide software ( https://github.com/gchowell/paramEstimation_forecasting_ODEmodels/) and detailed guidance on estimating parameters and forecasting time-series trajectories that are characterized using ODEs with quantified uncertainty through a parametric bootstrapping approach. It includes functions that allow the user to infer model parameters and assess forecasting performance for different ODE models specified by the user, using different estimation methods and error structures in the data. The tutorial is intended for a diverse audience, including students training in dynamic systems, and will be broadly applicable to estimate parameters and generate forecasts from models based on ODEs. The functions included in the toolbox are illustrated using epidemic models with varying levels of complexity applied to data from the 1918 influenza pandemic in San Francisco. A tutorial video that demonstrates the functionality of the toolbox is included.

Responding to the Return of Influenza in the United States by Applying Centers for Disease Control and Prevention Surveillance, Analysis, and Modeling to Inform Understanding of Seasonal Influenza

We reviewed the tools that have been developed to characterize and communicate seasonal influenza activity in the United States. Here we focus on systematic surveillance and applied analytics, including seasonal burden and disease severity estimation, short-term forecasting, and longer-term modeling efforts. For each set of activities, we describe the challenges and opportunities that have arisen because of the COVID-19 pandemic. In conclusion, we highlight how collaboration and communication have been and will continue to be key components of reliable and actionable influenza monitoring, forecasting, and modeling activities.

Building Environmental and Sociological Predictive Intelligence to Understand the Seasonal Threat of SARS-CoV-2 in Human Populations

Current modeling practices for environmental and sociological modulated infectious diseases remain inadequate to forecast the risk of outbreak(s) in human populations, partly due to a lack of integration of disciplinary knowledge, limited availability of disease surveillance datasets, and overreliance on compartmental epidemiological modeling methods. Harvesting data knowledge from virus transmission (aerosols) and detection (wastewater) of SARS-CoV-2, a heuristic score-based environmental predictive intelligence system was developed that calculates the risk of COVID-19 in the human population. Seasonal validation of the algorithm was uniquely associated with wastewater surveillance of the virus, providing a lead time of 7-14 days before a county-level outbreak. Using county-scale disease prevalence data from the United States, the algorithm could predict COVID-19 risk with an overall accuracy ranging between 81% and 98%. Similarly, using wastewater surveillance data from Illinois and Maryland, the SARS-CoV-2 detection rate was greater than 80% for 75% of the locations during the same time the risk was predicted to be high. Results suggest the importance of a holistic approach across disciplinary boundaries that can potentially allow anticipatory decision-making policies of saving lives and maximizing the use of available capacity and resources.

The US COVID-19 and Influenza Scenario Modeling Hubs: Delivering long-term projections to guide policy

Between December 2020 and April 2023, the COVID-19 Scenario Modeling Hub (SMH) generated operational multi-month projections of COVID-19 burden in the US to guide pandemic planning and decision-making in the context of high uncertainty. This effort was born out of an attempt to coordinate, synthesize and effectively use the unprecedented amount of predictive modeling that emerged throughout the COVID-19 pandemic. Here we describe the history of this massive collective research effort, the process of convening and maintaining an open modeling hub active over multiple years, and attempt to provide a blueprint for future efforts. We detail the process of generating 17 rounds of scenarios and projections at different stages of the COVID-19 pandemic, and disseminating results to the public health community and lay public. We also highlight how SMH was expanded to generate influenza projections during the 2022-23 season. We identify key impacts of SMH results on public health and draw lessons to improve future collaborative modeling efforts, research on scenario projections, and the interface between models and policy.

Projecting COVID-19 intensive care admissions for policy advice, the Netherlands, February 2020 to January 2021

BackgroundModel projections of coronavirus disease 2019 (COVID-19) incidence help policymakers about decisions to implement or lift control measures. During the pandemic, policymakers in the Netherlands were informed on a weekly basis with short-term projections of COVID-19 intensive care unit (ICU) admissions.AimWe aimed at developing a model on ICU admissions and updating a procedure for informing policymakers.MethodThe projections were produced using an age-structured transmission model. A consistent, incremental update procedure integrating all new surveillance and hospital data was conducted weekly. First, up-to-date estimates for most parameter values were obtained through re-analysis of all data sources. Then, estimates were made for changes in the age-specific contact rates in response to policy changes. Finally, a piecewise constant transmission rate was estimated by fitting the model to reported daily ICU admissions, with a changepoint analysis guided by Akaike's Information Criterion.ResultsThe model and update procedure allowed us to make weekly projections. Most 3-week prediction intervals were accurate in covering the later observed numbers of ICU admissions. When projections were too high in March and August 2020 or too low in November 2020, the estimated effectiveness of the policy changes was adequately adapted in the changepoint analysis based on the natural accumulation of incoming data.ConclusionThe model incorporates basic epidemiological principles and most model parameters were estimated per data source. Therefore, it had potential to be adapted to a more complex epidemiological situation with the rise of new variants and the start of vaccination.

Ensemble2: Scenarios ensembling for communication and performance analysis

Throughout the COVID-19 pandemic, scenario modeling played a crucial role in shaping the decision-making process of public health policies. Unlike forecasts, scenario projections rely on specific assumptions about the future that consider different plausible states-of-the-world that may or may not be realized and that depend on policy interventions, unpredictable changes in the epidemic outlook, etc. As a consequence, long-term scenario projections require different evaluation criteria than the ones used for traditional short-term epidemic forecasts. Here, we propose a novel ensemble procedure for assessing pandemic scenario projections using the results of the Scenario Modeling Hub (SMH) for COVID-19 in the United States (US). By defining a "scenario ensemble" for each model and the ensemble of models, termed "Ensemble2", we provide a synthesis of potential epidemic outcomes, which we use to assess projections' performance, bypassing the identification of the most plausible scenario. We find that overall the Ensemble2 models are well-calibrated and provide better performance than the scenario ensemble of individual models. The ensemble procedure accounts for the full range of plausible outcomes and highlights the importance of scenario design and effective communication. The scenario ensembling approach can be extended to any scenario design strategy, with potential refinements including weighting scenarios and allowing the ensembling process to evolve over time.

Predicting Covid-19 pandemic waves with biologically and behaviorally informed universal differential equations

During the COVID-19 pandemic, it became clear that pandemic waves and population responses were locked in a mutual feedback loop in a classic example of a coupled behavior-disease system. We demonstrate for the first time that universal differential equation (UDE) models are able to extract this interplay from data. We develop a UDE model for COVID-19 and test its ability to make predictions of second pandemic waves. We find that UDEs are capable of learning coupled behavior-disease dynamics and predicting second waves in a variety of populations, provided they are supplied with learning biases describing simple assumptions about disease transmission and population response. Though not yet suitable for deployment as a policy-guiding tool, our results demonstrate potential benefits, drawbacks, and useful techniques when applying universal differential equations to coupled systems.

Predicting subnational incidence of COVID-19 cases and deaths in EU countries

BACKGROUND

Recurring COVID-19 waves highlight the need for tools able to quantify transmission risk, and identify geographical areas at risk of outbreaks. Local outbreak risk depends on complex immunity patterns resulting from previous infections, vaccination, waning and immune escape, alongside other factors (population density, social contact patterns). Immunity patterns are spatially and demographically heterogeneous, and are challenging to capture in country-level forecast models.

METHODS

We used a spatiotemporal regression model to forecast subnational case and death counts and applied it to three EU countries as test cases: France, Czechia, and Italy. Cases in local regions arise from importations or local transmission. Our model produces age-stratified forecasts given age-stratified data, and links reported case counts to routinely collected covariates (e.g. test number, vaccine coverage). We assessed the predictive performance of our model up to four weeks ahead using proper scoring rules and compared it to the European COVID-19 Forecast Hub ensemble model. Using simulations, we evaluated the impact of variations in transmission on the forecasts. We developed an open-source RShiny App to visualise the forecasts and scenarios.

RESULTS

At a national level, the median relative difference between our median weekly case forecasts and the data up to four weeks ahead was 25% (IQR: 12-50%) over the prediction period. The accuracy decreased as the forecast horizon increased (on average 24% increase in the median ranked probability score per added week), while the accuracy of death forecasts was more stable. Beyond two weeks, the model generated a narrow range of likely transmission dynamics. The median national case forecasts showed similar accuracy to forecasts from the European COVID-19 Forecast Hub ensemble model, but the prediction interval was narrower in our model. Generating forecasts under alternative transmission scenarios was therefore key to capturing the range of possible short-term transmission dynamics.

DISCUSSION

Our model captures changes in local COVID-19 outbreak dynamics, and enables quantification of short-term transmission risk at a subnational level. The outputs of the model improve our ability to identify areas where outbreaks are most likely, and are available to a wide range of public health professionals through the Shiny App we developed.

Use of Telehealth Information for Early Detection: Insights From the COVID-19 Pandemic

Objectives. To examine whether the addition of telehealth data to existing surveillance infrastructure can improve forecasts of cases and mortality. Methods. In this observational study, we compared accuracy of 14-day forecasts using real-time data available to the National Syndromic Surveillance Program (standard forecasts) to forecasts that also included telehealth information (telehealth forecasts). The study was performed in a national telehealth service provider in 2020 serving 50 US states and the District of Columbia. Results. Among 10.5 million telemedicine encounters, 169 672 probable COVID-19 cases were diagnosed by 5050 clinicians, with a rate between 0.79 and 47.8 probable cases per 100 000 encounters per day (mean = 8.37; SD = 10.75). Publicly reported case counts ranged from 0.5 to 237 916 (mean: 53 913; SD = 47 466) and 0 to 2328 deaths (mean = 1035; SD = 550) per day. Telehealth-based forecasts improved 14-day case forecasting accuracy by 1.8 percentage points to 30.9% (P = .06) and mortality forecasting by 6.4 percentage points to 26.9% (P < .048). Conclusions. Modest improvements in forecasting can be gained from adding telehealth data to syndromic surveillance infrastructure. (Am J Public Health. 2024;114(2):218-225. https://doi.org/10.2105/AJPH.2023.307499).

GrowthPredict: A toolbox and tutorial-based primer for fitting and forecasting growth trajectories using phenomenological growth models

Simple dynamic modeling tools can help generate real-time short-term forecasts with quantified uncertainty of the trajectory of diverse growth processes unfolding in nature and society, including disease outbreaks. An easy-to-use and flexible toolbox for this purpose is lacking. This tutorial-based primer introduces and illustrates GrowthPredict, a user-friendly MATLAB toolbox for fitting and forecasting time-series trajectories using phenomenological dynamic growth models based on ordinary differential equations. This toolbox is accessible to a broad audience, including students training in mathematical biology, applied statistics, and infectious disease modeling, as well as researchers and policymakers who need to conduct short-term forecasts in real-time. The models included in the toolbox capture exponential and sub-exponential growth patterns that typically follow a rising pattern followed by a decline phase, a common feature of contagion processes. Models include the 1-parameter exponential growth model and the 2-parameter generalized-growth model, which have proven useful in characterizing and forecasting the ascending phase of epidemic outbreaks. It also includes the 2-parameter Gompertz model, the 3-parameter generalized logistic-growth model, and the 3-parameter Richards model, which have demonstrated competitive performance in forecasting single peak outbreaks. We provide detailed guidance on forecasting time-series trajectories and available software ( https://github.com/gchowell/forecasting_growthmodels ), including the full uncertainty distribution derived through parametric bootstrapping, which is needed to construct prediction intervals and evaluate their accuracy. Functions are available to assess forecasting performance across different models, estimation methods, error structures in the data, and forecasting horizons. The toolbox also includes functions to quantify forecasting performance using metrics that evaluate point and distributional forecasts, including the weighted interval score. This tutorial and toolbox can be broadly applied to characterizing and forecasting time-series data using simple phenomenological growth models. As a contagion process takes off, the tools presented in this tutorial can help create forecasts to guide policy regarding implementing control strategies and assess the impact of interventions. The toolbox functionality is demonstrated through various examples, including a tutorial video, and the examples use publicly available data on the monkeypox (mpox) epidemic in the USA.

A simplicial epidemic model for COVID-19 spread analysis

Networks allow us to describe a wide range of interaction phenomena that occur in complex systems arising in such diverse fields of knowledge as neuroscience, engineering, ecology, finance, and social sciences. Until very recently, the primary focus of network models and tools has been on describing the pairwise relationships between system entities. However, increasingly more studies indicate that polyadic or higher-order group relationships among multiple network entities may be the key toward better understanding of the intrinsic mechanisms behind the functionality of complex systems. Such group interactions can be, in turn, described in a holistic manner by simplicial complexes of graphs. Inspired by these recently emerging results on the utility of the simplicial geometry of complex networks for contagion propagation and armed with a large-scale synthetic social contact network (also known as a digital twin) of the population in the U.S. state of Virginia, in this paper, we aim to glean insights into the role of higher-order social interactions and the associated varying social group determinants on COVID-19 propagation and mitigation measures.

Application of COVIDTracer to Inform Public Health Workforce Planning in Alaska During the COVID-19 Pandemic

The COVID-19 pandemic has caused social and economic disruption worldwide and spurred numerous mitigation strategies, including state investments in training a large contact tracing and case investigation workforce. A team at the University of Alaska Anchorage evaluated implementation of the COVID-19 contact tracing and case investigation program of the State of Alaska Department of Health and Social Services, Division of Public Health, Section of Public Health Nursing. As part of that evaluation, the team used COVIDTracer, a spreadsheet modeling tool. COVIDTracer generated projections of COVID-19 case counts that informed estimates of workforce needs and case prioritization strategies. Case count projections approximated the reported epidemiologic curve with a median 7% difference in the first month. The accuracy of case count predictions declined after 1 month with a median difference of 80% in the second month. COVIDTracer inputs included previous case counts, the average length of time for telephone calls to cases and outreach to identified contacts, and the average number of contacts per case. As each variable increased, so too did estimated workforce needs. Decreasing the average time from exposure to outreach from 10 to 5 days reduced case counts estimated by COVIDTracer by approximately 93% during a 5-month period. COVIDTracer estimates informed Alaska's workforce planning and decisions about prioritizing case investigation during the pandemic. Lessons learned included the importance of being able to rapidly scale up and scale down workforce to adjust to a dynamic crisis and the limitations of prediction modeling (eg, that COVIDTracer was accurate for only about 1 month into the future). These findings may be useful for future pandemic preparedness planning and other public health emergency response activities.

Predicting the spread of SARS-CoV-2 variants: An artificial intelligence enabled early detection

During more than 3 years since its emergence, SARS-CoV-2 has shown great ability to mutate rapidly into diverse variants, some of which turned out to be very infectious and have spread throughout the world causing waves of infections. At this point, many countries have already experienced up to six waves of infections. Extensive academic work has focused on the development of models to predict the pandemic trajectory based on epidemiological data, but none has focused on predicting variant-specific spread. Moreover, important scientific literature analyzes the genetic evolution of SARS-CoV-2 variants and how it might functionally affect their infectivity. However, genetic attributes have not yet been incorporated into existing epidemiological modeling that aims to capture infection trajectory. Thus, this study leverages variant-specific genetic characteristics together with epidemiological information to systematically predict the future spread trajectory of newly detected variants. The study describes the analysis of 9.0 million SARS-CoV-2 genetic sequences in 30 countries and identifies temporal characteristic patterns of SARS-CoV-2 variants that caused significant infection waves. Using this descriptive analysis, a machine-learning-enabled risk assessment model has been developed to predict, as early as 1 week after their first detection, which variants are likely to constitute the new wave of infections in the following 3 months. The model's out-of-sample area under the curve (AUC) is 86.3% for predictions after 1 week and 90.8% for predictions after 2 weeks. The methodology described in this paper could contribute more broadly to the development of improved predictive models for variants of other infectious viruses.

Application progress of ensemble forecast technology in influenza forecast based on infectious disease model

To comprehensively understand the application progress of ensemble forecast technology in influenza forecast based on infectious disease model, so as to provide scientific references for further research. In this study, two keywords of "influenza" and "ensemble forecast" are selected to search and select the relevant literatures, which are then outlined and summarized. It is found that: In recent years, some studies about ensemble forecast technology for influenza have been reported in the literature, and some well-performed influenza ensemble forecast systems have already been operationally implemented and provide references for scientific prevention and control. In general, ensemble forecast can well represent various uncertainties in forecasting influenza cases based on infectious disease models, and can achieve more accurate forecasts and more valuable information than single deterministic forecast. However, there are still some shortcomings in the current studies, it is suggested that scientists engaged in influenza forecast based on infectious disease models strengthen cooperation with scholars in the field of numerical weather forecast, which is expected to further improve the skills and application level of ensemble forecast for influenza.

Evaluation of FluSight influenza forecasting in the 2021-22 and 2022-23 seasons with a new target laboratory-confirmed influenza hospitalizations

Accurate forecasts can enable more effective public health responses during seasonal influenza epidemics. Forecasting teams were asked to provide national and jurisdiction-specific probabilistic predictions of weekly confirmed influenza hospital admissions for one through four weeks ahead for the 2021-22 and 2022-23 influenza seasons. Across both seasons, 26 teams submitted forecasts, with the submitting teams varying between seasons. Forecast skill was evaluated using the Weighted Interval Score (WIS), relative WIS, and coverage. Six out of 23 models outperformed the baseline model across forecast weeks and locations in 2021-22 and 12 out of 18 models in 2022-23. Averaging across all forecast targets, the FluSight ensemble was the 2nd most accurate model measured by WIS in 2021-22 and the 5th most accurate in the 2022-23 season. Forecast skill and 95% coverage for the FluSight ensemble and most component models degraded over longer forecast horizons and during periods of rapid change. Current influenza forecasting efforts help inform situational awareness, but research is needed to address limitations, including decreased performance during periods of changing epidemic dynamics.

The variations of SIkJalpha model for COVID-19 forecasting and scenario projections

We proposed the SIkJalpha model at the beginning of the COVID-19 pandemic (early 2020). Since then, as the pandemic evolved, more complexities were added to capture crucial factors and variables that can assist with projecting desired future scenarios. Throughout the pandemic, multi-model collaborative efforts have been organized to predict short-term outcomes (cases, deaths, and hospitalizations) of COVID-19 and long-term scenario projections. We have been participating in five such efforts. This paper presents the evolution of the SIkJalpha model and its many versions that have been used to submit to these collaborative efforts since the beginning of the pandemic. Specifically, we show that the SIkJalpha model is an approximation of a class of epidemiological models. We demonstrate how the model can be used to incorporate various complexities, including under-reporting, multiple variants, waning of immunity, and contact rates, and to generate probabilistic outputs.

Assessing the utility of COVID-19 case reports as a leading indicator for hospitalization forecasting in the United States

Identifying data streams that can consistently improve the accuracy of epidemiological forecasting models is challenging. Using models designed to predict daily state-level hospital admissions due to COVID-19 in California and Massachusetts, we investigated whether incorporating COVID-19 case data systematically improved forecast accuracy. Additionally, we considered whether using case data aggregated by date of test or by date of report from a surveillance system made a difference to the forecast accuracy. Evaluating forecast accuracy in a test period, after first having selected the best-performing methods in a validation period, we found that overall the difference in accuracy between approaches was small, especially at forecast horizons of less than two weeks. However, forecasts from models using cases aggregated by test date showed lower accuracy at longer horizons and at key moments in the pandemic, such as the peak of the Omicron wave in January 2022. Overall, these results highlight the challenge of finding a modeling approach that can generate accurate forecasts of outbreak trends both during periods of relative stability and during periods that show rapid growth or decay of transmission rates. While COVID-19 case counts seem to be a natural choice to help predict COVID-19 hospitalizations, in practice any benefits we observed were small and inconsistent.

Reflections On Epidemiological Modeling To Inform Policy During The COVID-19 Pandemic In Western Europe, 2020-23

We reflect on epidemiological modeling conducted throughout the COVID-19 pandemic in Western Europe, specifically in Belgium, France, Italy, the Netherlands, Portugal, Switzerland, and the United Kingdom. Western Europe was initially one of the worst-hit regions during the COVID-19 pandemic. Western European countries deployed a range of policy responses to the pandemic, which were often informed by mathematical, computational, and statistical models. Models differed in terms of temporal scope, pandemic stage, interventions modeled, and analytical form. This diversity was modulated by differences in data availability and quality, government interventions, societal responses, and technical capacity. Many of these models were decisive to policy making at key junctures, such as during the introduction of vaccination and the emergence of the Alpha, Delta, and Omicron variants. However, models also faced intense criticism from the press, other scientists, and politicians around their accuracy and appropriateness for decision making. Hence, evaluating the success of models in terms of accuracy and influence is an essential task. Modeling needs to be supported by infrastructure for systems to collect and share data, model development, and collaboration between groups, as well as two-way engagement between modelers and both policy makers and the public.

The importance of investing in data, models, experiments, team science, and public trust to help policymakers prepare for the next pandemic

The COVID-19 pandemic has brought about valuable insights regarding models, data, and experiments. In this narrative review, we summarised the existing literature on these three themes, exploring the challenges of providing forecasts, the requirement for real-time linkage of health-related datasets, and the role of 'experimentation' in evaluating interventions. This literature review encourages us to broaden our perspective for the future, acknowledging the significance of investing in models, data, and experimentation, but also to invest in areas that are conceptually more abstract: the value of 'team science', the need for public trust in science, and in establishing processes for using science in policy. Policy-makers rely on model forecasts early in a pandemic when there is little data, and it is vital to communicate the assumptions, limitations, and uncertainties (theme 1). Linked routine data can provide critical information, for example, in establishing risk factors for adverse outcomes but are often not available quickly enough to make a real-time impact. The interoperability of data resources internationally is required to facilitate sharing across jurisdictions (theme 2). Randomised controlled trials (RCTs) provided timely evidence on the efficacy and safety of vaccinations and pharmaceuticals but were largely conducted in higher income countries, restricting generalisability to low- and middle-income countries (LMIC). Trials for non-pharmaceutical interventions (NPIs) were almost non-existent which was a missed opportunity (theme 3). Building on these themes from the narrative review, we underscore the importance of three other areas that need investment for effective evidence-driven policy-making. The COVID-19 response relied on strong multidisciplinary research infrastructures, but funders and academic institutions need to do more to incentivise team science (4). To enhance public trust in the use of scientific evidence for policy, researchers and policy-makers must work together to clearly communicate uncertainties in current evidence and any need to change policy as evidence evolves (5). Timely policy decisions require an established two-way process between scientists and policy makers to make the best use of evidence (6). For effective preparedness against future pandemics, it is essential to establish models, data, and experiments as fundamental pillars, complemented by efforts in planning and investment towards team science, public trust, and evidence-based policy-making across international communities. The paper concludes with a 'call to actions' for both policy-makers and researchers.

Evaluation of the US COVID-19 Scenario Modeling Hub for informing pandemic response under uncertainty

Our ability to forecast epidemics far into the future is constrained by the many complexities of disease systems. Realistic longer-term projections may, however, be possible under well-defined scenarios that specify the future state of critical epidemic drivers. Since December 2020, the U.S. COVID-19 Scenario Modeling Hub (SMH) has convened multiple modeling teams to make months ahead projections of SARS-CoV-2 burden, totaling nearly 1.8 million national and state-level projections. Here, we find SMH performance varied widely as a function of both scenario validity and model calibration. We show scenarios remained close to reality for 22 weeks on average before the arrival of unanticipated SARS-CoV-2 variants invalidated key assumptions. An ensemble of participating models that preserved variation between models (using the linear opinion pool method) was consistently more reliable than any single model in periods of valid scenario assumptions, while projection interval coverage was near target levels. SMH projections were used to guide pandemic response, illustrating the value of collaborative hubs for longer-term scenario projections.

Comparison of combination methods to create calibrated ensemble forecasts for seasonal influenza in the U.S

The characteristics of influenza seasons vary substantially from year to year, posing challenges for public health preparation and response. Influenza forecasting is used to inform seasonal outbreak response, which can in turn potentially reduce the impact of an epidemic. The United States Centers for Disease Control and Prevention, in collaboration with external researchers, has run an annual prospective influenza forecasting exercise, known as the FluSight challenge. Uniting theoretical results from the forecasting literature with domain-specific forecasts from influenza outbreaks, we applied parametric forecast combination methods that simultaneously optimize model weights and calibrate the ensemble via a beta transformation and made adjustments to the methods to reduce their complexity. We used the beta-transformed linear pool, the finite beta mixture model, and their equal weight adaptations to produce ensemble forecasts retrospectively for the 2016/2017, 2017/2018, and 2018/2019 influenza seasons in the U.S. We compared their performance to methods that were used in the FluSight challenge to produce the FluSight Network ensemble, namely the equally weighted linear pool and the linear pool. Ensemble forecasts produced from methods with a beta transformation were shown to outperform those from the equally weighted linear pool and the linear pool for all week-ahead targets across in the test seasons based on average log scores. We observed improvements in overall accuracy despite the beta-transformed linear pool or beta mixture methods' modest under-prediction across all targets and seasons. Combination techniques that explicitly adjust for known calibration issues in linear pooling should be considered to improve probabilistic scores in outbreak settings.

Deep learning framework for epidemiological forecasting: A study on COVID-19 cases and deaths in the Amazon state of Pará, Brazil

Modeling time series has been a particularly challenging aspect due to the need for constant adjustments in a rapidly changing environment, data uncertainty, dependencies between variables, volatile fluctuations, and the need to identify ideal hyperparameters. The present study presents a Framework capable of making projections from time series related to cases and deaths by COVID-19 in the Amazonian state of Pará, in Brazil. For the first time, deep learning models such as TCN, TRANSFORMER, TFT, N-BEATS, and N-HiTS were assessed for this purpose. The ARIMA statistical model was also used in post-processing for residual adjustment and short-term smoothing of the generated forecasts. The Framework generates probabilistic forecasts, with multivariate support, considering the following variables: daily cases per day of the first symptom, cases published daily, the occurrence of deaths, deaths published daily, and percentage of daily vaccination. The generated predictions are statistically evaluated by determining the best model for 7-day moving average projections using evaluating metrics such as MSE, RMSE, MAPE, sMAPE, r2, Coefficient of Variation, and residual analysis. As a result, the generated projections showed an average error of 5.4% for Cases Publication, 8.0% for Cases Symptoms, 11.12% for Deaths Publication, and 4.6% for Deaths Occurrence, with the N-HiTS and N-BEATS models obtaining better results. In general terms, the use of deep learning models to predict cases and deaths from COVID-19 has proven to be a valuable practice for analyzing the spread of the virus, which allows health managers to better understand and respond to this kind of pandemic outbreak.

The Epidemiological and Economic Impact of COVID-19 in Kazakhstan: An Agent-Based Modeling

BACKGROUND

Our study aimed to assess how effective the preventative measures taken by the state authorities during the pandemic were in terms of public health protection and the rational use of material and human resources.

MATERIALS AND METHODS

We utilized a stochastic agent-based model for COVID-19's spread combined with the WHO-recommended COVID-ESFT version 2.0 tool for material and labor cost estimation.

RESULTS

Our long-term forecasts (up to 50 days) showed satisfactory results with a steady trend in the total cases. However, the short-term forecasts (up to 10 days) were more accurate during periods of relative stability interrupted by sudden outbreaks. The simulations indicated that the infection's spread was highest within families, with most COVID-19 cases occurring in the 26-59 age group. Government interventions resulted in 3.2 times fewer cases in Karaganda than predicted under a "no intervention" scenario, yielding an estimated economic benefit of 40%.

CONCLUSION

The combined tool we propose can accurately forecast the progression of the infection, enabling health organizations to allocate specialists and material resources in a timely manner.