Public health impact of the U.S. Scenario Modeling Hub

Best practices for estimating and reporting epidemiological delay distributions of infectious diseases

Epidemiological delays are key quantities that inform public health policy and clinical practice. They are used as inputs for mathematical and statistical models, which in turn can guide control strategies. In recent work, we found that censoring, right truncation, and dynamical bias were rarely addressed correctly when estimating delays and that these biases were large enough to have knock-on impacts across a large number of use cases. Here, we formulate a checklist of best practices for estimating and reporting epidemiological delays. We also provide a flowchart to guide practitioners based on their data. Our examples are focused on the incubation period and serial interval due to their importance in outbreak response and modeling, but our recommendations are applicable to other delays. The recommendations, which are based on the literature and our experience estimating epidemiological delay distributions during outbreak responses, can help improve the robustness and utility of reported estimates and provide guidance for the evaluation of estimates for downstream use in transmission models or other analyses.

Expanding the use of mathematical modeling in healthcare epidemiology and infection prevention and control

During the coronavirus disease 2019 pandemic, mathematical modeling has been widely used to understand epidemiological burden, trends, and transmission dynamics, to facilitate policy decisions, and, to a lesser extent, to evaluate infection prevention and control (IPC) measures. This review highlights the added value of using conventional epidemiology and modeling approaches to address the complexity of healthcare-associated infections (HAI) and antimicrobial resistance. It demonstrates how epidemiological surveillance data and modeling can be used to infer transmission dynamics in healthcare settings and to forecast healthcare impact, how modeling can be used to improve the validity of interpretation of epidemiological surveillance data, how modeling can be used to estimate the impact of IPC interventions, and how modeling can be used to guide IPC and antimicrobial treatment and stewardship decision-making. There are several priority areas for expanding the use of modeling in healthcare epidemiology and IPC. Importantly, modeling should be viewed as complementary to conventional healthcare epidemiological approaches, and this requires collaboration and active coordination between IPC, healthcare epidemiology, and mathematical modeling groups.

Inference of epidemic dynamics in the COVID-19 era and beyond

The COVID-19 pandemic demonstrated the key role that epidemiology and modelling play in analysing infectious threats and supporting decision making in real-time. Motivated by the unprecedented volume and breadth of data generated during the pandemic, we review modern opportunities for analysis to address questions that emerge during a major modern epidemic. Following the broad chronology of insights required - from understanding initial dynamics to retrospective evaluation of interventions, we describe the theoretical foundations of each approach and the underlying intuition. Through a series of case studies, we illustrate real life applications, and discuss implications for future work.

hubEnsembles: Ensembling Methods in R

Combining predictions from multiple models into an ensemble is a widely used practice across many fields with demonstrated performance benefits. The R package hubEnsembles provides a flexible framework for ensembling various types of predictions, including point estimates and probabilistic predictions. A range of common methods for generating ensembles are supported, including weighted averages, quantile averages, and linear pools. The hubEnsembles package fits within a broader framework of open-source software and data tools called the "hubverse", which facilitates the development and management of collaborative modelling exercises.

Projecting the future impact of emerging SARS-CoV-2 variants under uncertainty: Modeling the initial Omicron outbreak

Over the past several years, the emergence of novel SARS-CoV-2 variants has led to multiple waves of increased COVID-19 incidence. When the Omicron variant emerged, there was considerable concern about its potential impact in the winter of 2021-2022 due to its increased fitness. However, there was also considerable uncertainty regarding its likely impact due to questions about its relative transmissibility, severity, and degree of immune escape. We sought to evaluate the ability of an agent-based model to forecast incidence in the context of this emerging pathogen variant. To project COVID-19 cases and deaths in Indiana, we calibrated our model to COVID-19 hospitalizations, deaths, and test-positivity rates through November 2021, and then projected COVID-19 incidence through April 2022 under four different scenarios that covered the plausible ranges of Omicron's severity, transmissibility, and degree of immune escape. Our initial projections from December 2021 through March 2022 indicated that under a pessimistic scenario with high disease severity, the peak in weekly COVID-19 deaths in Indiana would be larger than the previous peak in December 2020. However, retrospective analyses indicate that Omicron's severity was closer to the optimistic scenario, and even though cases and hospitalizations reached a new peak, fewer deaths occurred than during the previous peak. According to our results, Omicron's rapid spread was consistent with a combination of higher transmissibility and immune escape relative to earlier variants. Our updated projections starting in January 2022 accurately predicted that cases would peak in mid-January and decline rapidly over the next several months. The performance of our projections shows that following the emergence of a new pathogen variant, models can help quantify the potential range of outbreak magnitudes and trajectories. Agent-based models are particularly useful in these scenarios because they can efficiently track individual vaccination and infection histories with multiple variants with varying degrees of cross-protection.

flepiMoP: The evolution of a flexible infectious disease modeling pipeline during the COVID-19 pandemic

The COVID-19 pandemic led to an unprecedented demand for projections of disease burden and healthcare utilization under scenarios ranging from unmitigated spread to strict social distancing policies. In response, members of the Johns Hopkins Infectious Disease Dynamics Group developed flepiMoP (formerly called the COVID Scenario Modeling Pipeline), a comprehensive open-source software pipeline designed for creating and simulating compartmental models of infectious disease transmission and inferring parameters through these models. The framework has been used extensively to produce short-term forecasts and longer-term scenario projections of COVID-19 at the state and county level in the US, for COVID-19 in other countries at various geographic scales, and more recently for seasonal influenza. In this paper, we highlight how the flepiMoP has evolved throughout the COVID-19 pandemic to address changing epidemiological dynamics, new interventions, and shifts in policy-relevant model outputs. As the framework has reached a mature state, we provide a detailed overview of flepiMoP's key features and remaining limitations, thereby distributing flepiMoP and its documentation as a flexible and powerful tool for researchers and public health professionals to rapidly build and deploy large-scale complex infectious disease models for any pathogen and demographic setup.

Redefining pandemic preparedness: Multidisciplinary insights from the CERP modelling workshop in infectious diseases, workshop report

In July 2023, the Center of Excellence in Respiratory Pathogens organized a two-day workshop on infectious diseases modelling and the lessons learnt from the Covid-19 pandemic. This report summarizes the rich discussions that occurred during the workshop. The workshop participants discussed multisource data integration and highlighted the benefits of combining traditional surveillance with more novel data sources like mobility data, social media, and wastewater monitoring. Significant advancements were noted in the development of predictive models, with examples from various countries showcasing the use of machine learning and artificial intelligence in detecting and monitoring disease trends. The role of open collaboration between various stakeholders in modelling was stressed, advocating for the continuation of such partnerships beyond the pandemic. A major gap identified was the absence of a common international framework for data sharing, which is crucial for global pandemic preparedness. Overall, the workshop underscored the need for robust, adaptable modelling frameworks and the integration of different data sources and collaboration across sectors, as key elements in enhancing future pandemic response and preparedness.

Mechanistic models for West Nile virus transmission: a systematic review of features, aims and parametrization

Mathematical models within the Ross-Macdonald framework increasingly play a role in our understanding of vector-borne disease dynamics and as tools for assessing scenarios to respond to emerging threats. These threats are typically characterized by a high degree of heterogeneity, introducing a range of possible complexities in models and challenges to maintain the link with empirical evidence. We systematically identified and analysed a total of 77 published papers presenting compartmental West Nile virus (WNV) models that use parameter values derived from empirical studies. Using a set of 15 criteria, we measured the dissimilarity compared with the Ross-Macdonald framework. We also retrieved the purpose and type of models and traced the empirical sources of their parameters. Our review highlights the increasing refinements in WNV models. Models for prediction included the highest number of refinements. We found uneven distributions of refinements and of evidence for parameter values. We identified several challenges in parametrizing such increasingly complex models. For parameters common to most models, we also synthesize the empirical evidence for their values and ranges. The study highlights the potential to improve the quality of WNV models and their applicability for policy by establishing closer collaboration between mathematical modelling and empirical work.

The US COVID-19 and Influenza Scenario Modeling Hubs: Delivering long-term projections to guide policy

Between December 2020 and April 2023, the COVID-19 Scenario Modeling Hub (SMH) generated operational multi-month projections of COVID-19 burden in the US to guide pandemic planning and decision-making in the context of high uncertainty. This effort was born out of an attempt to coordinate, synthesize and effectively use the unprecedented amount of predictive modeling that emerged throughout the COVID-19 pandemic. Here we describe the history of this massive collective research effort, the process of convening and maintaining an open modeling hub active over multiple years, and attempt to provide a blueprint for future efforts. We detail the process of generating 17 rounds of scenarios and projections at different stages of the COVID-19 pandemic, and disseminating results to the public health community and lay public. We also highlight how SMH was expanded to generate influenza projections during the 2022-23 season. We identify key impacts of SMH results on public health and draw lessons to improve future collaborative modeling efforts, research on scenario projections, and the interface between models and policy.

Covid19Vaxplorer: A free, online, user-friendly COVID-19 vaccine allocation comparison tool

There are many COVID-19 vaccines currently available, however, Low- and middle-income countries (LMIC) still have large proportions of their populations unvaccinated. Decision-makers must decide how to effectively allocate available vaccines (e.g. boosters or primary series vaccination, which age groups to target) but LMIC often lack the resources to undergo quantitative analyses of vaccine allocation, resulting in ad-hoc policies. We developed Covid19Vaxplorer (https://covid19vaxplorer.fredhutch.org/), a free, user-friendly online tool that simulates region-specific COVID-19 epidemics in conjunction with vaccination with the purpose of providing public health officials worldwide with a tool for vaccine allocation planning and comparison. We developed an age-structured mathematical model of SARS-CoV-2 transmission and COVID-19 vaccination. The model considers vaccination with up to three different vaccine products, primary series and boosters. We simulated partial immunity derived from waning of natural infection and vaccination. The model is embedded in an online tool, Covid19Vaxplorer that was optimized for its ease of use. By prompting users to fill information through several windows to input local parameters (e.g. cumulative and current prevalence), epidemiological parameters (e.g basic reproduction number, current social distancing interventions), vaccine parameters (e.g. vaccine efficacy, duration of immunity) and vaccine allocation (both by age groups and by vaccination status). Covid19Vaxplorer connects the user to the mathematical model and simulates, in real time, region-specific epidemics. The tool then produces key outcomes including expected numbers of deaths, hospitalizations and cases, with the possibility of simulating several scenarios of vaccine allocation at once for a side-by-side comparison. We provide two usage examples of Covid19Vaxplorer for vaccine allocation in Haiti and Afghanistan, which had as of Spring 2023, 2% and 33% of their populations vaccinated, and show that for these particular examples, using available vaccine as primary series vaccinations prevents more deaths than using them as boosters.

Covid19Vaxplorer: a free, online, user-friendly COVID-19 Vaccine Allocation Comparison Tool

Background

There are many COVID-19 vaccines currently available, however, Low- and middle-income countries (LMIC) still have large proportions of their populations unvaccinated. Decision-makers must decide how to effectively allocate available vaccines (e.g. boosters or primary series vaccination, which age groups to target) but LMIC often lack the resources to undergo quantitative analyses of vaccine allocation, resulting in ad-hoc policies. We developed Covid19Vaxplorer (https://covid19vaxplorer.fredhutch.org/), a free, user-friendly online tool that simulates region-specific COVID-19 epidemics in conjunction with vaccination with the purpose of providing public health officials worldwide with a tool for vaccine allocation planning and comparison.

Methods

We developed an age-structured mathematical model of SARS-CoV-2 transmission and COVID-19 vaccination. The model considers vaccination with up to three different vaccine products, primary series and boosters. We simulated partial immunity derived from waning of natural infection and vaccination. The model is embedded in an online tool, Covid19Vaxplorer that was optimized for its ease of use. By prompting users to fill information through several windows to input local parameters (e.g. cumulative and current prevalence), epidemiological parameters (e.g basic reproduction number, current social distancing interventions), vaccine parameters (e.g. vaccine efficacy, duration of immunity) and vaccine allocation (both by age groups and by vaccination status). Covid19Vaxplorer connects the user to the mathematical model and simulates, in real time, region-specific epidemics. The tool then produces key outcomes including expected numbers of deaths, hospitalizations and cases, with the possibility of simulating several scenarios of vaccine allocation at once for a side-by-side comparison.

Results

We provide two usage examples of Covid19Vaxplorer for vaccine allocation in Haiti and Afghanistan, which had as of Spring 2023 2% and 33% of their populations vaccinated, and show that for these particular examples, using available vaccine as primary series vaccinations prevents more deaths than using them as boosters. Covid19Vaxplorer allows users in 183 regions in the world to compare several vaccination strategies simultaneously, adjusting parameters to their local epidemics, infrastructure and logistics. Covid19Vaxplorer is an online, free, user-friendly tool that facilitates evidence-based decision making for vaccine distribution.