Improving Pandemic Response: Employing Mathematical Modeling to Confront COVID-19

A Sequential Calibration Approach to Address Challenges of Repeated Calibration of a COVID-19 Model

BACKGROUND

Mathematical models served a critical role in COVID-19 decision making throughout the pandemic. Model calibration is an essential, but often computationally burdensome, step in model development that provides estimates for difficult-to-measure parameters and establishes an up-to-date modeling platform for scenario analysis. In the evolving COVID-19 pandemic, frequent recalibration was necessary to provide ongoing support to decision makers. In this study, we address the computational challenges of frequent recalibration with a new calibration approach.

METHODS

We calibrated and recalibrated an age-stratified dynamic compartmental model of COVID-19 in Minnesota to statewide COVID-19 cumulative mortality and prevalent age-specific hospitalizations from March 22, 2020 through August 20, 2021. This period was divided into 10 calibration periods, reflecting significant changes in policies, messaging, and/or epidemiological conditions in Minnesota. When recalibrating the model from one period to the next, we employed a sequential calibration approach that leveraged calibration results from previous periods and adjusted only parameters most relevant to the calibration target data of the new calibration period to improve computational efficiency. We compared computational burden and performance of the sequential calibration approach to a more traditional calibration method, in which all parameters were readjusted with each recalibration.

RESULTS

Both calibration methods identified parameter sets closely reproducing prevalent hospitalizations and cumulative deaths over time. By the last calibration period, both approaches converged to similar parameter values. However, the sequential calibration approach identified parameter sets that more tightly fit calibration targets and required substantially less computation time than traditional calibration.

CONCLUSIONS

Sequential calibration is an efficient approach to maintaining up-to-date models with evolving, time-varying parameters and potentially identifies better-fitting parameter sets than traditional calibration.

HIGHLIGHTS

This study used a sequential calibration approach, which takes advantage of previous calibration results to reduce the number of parameters to be estimated in each round of calibration, improving computational efficiency and algorithm convergence to best-fitting parameter values.Both sequential and traditional calibration approaches were able to identify parameter sets that closely reproduced calibration targets. However, the sequential calibration approach generated parameter sets that yielded tighter fits and was less computationally burdensome.Sequential calibration is an efficient approach to maintaining up-to-date models with evolving, time-varying parameters.

Forecasting the relative abundance of Aedes vector populations to enhance situational awareness for mosquito control operations

Aedes-borne diseases represent a major public health threat and mosquito control operations represent a key line of defense. Improving the real-time awareness of mosquito control authorities by providing reliable forecasts of the relative abundance of mosquito vectors could greatly enhance control efforts. To this aim, we developed an analytical tool that forecasts Aedes aegypti relative abundance 1 to 4 weeks ahead. Forecasts were validated against mosquito surveillance data (2,760 data points) collected over multiple years in four jurisdictions in the US. The symmetric absolute percentage error was in the range 0.43-0.69, and the 90% interquantile range of the forecasts had a coverage of 83-92%. Our forecasts consistently outperformed a reference "naïve" model for all analyzed study sites, forecasting horizon, and for periods with medium/high Ae. aegypti activity. The developed tool can be instrumental to address the need for evidence-based decision making.

Expanding the use of mathematical modeling in healthcare epidemiology and infection prevention and control

During the coronavirus disease 2019 pandemic, mathematical modeling has been widely used to understand epidemiological burden, trends, and transmission dynamics, to facilitate policy decisions, and, to a lesser extent, to evaluate infection prevention and control (IPC) measures. This review highlights the added value of using conventional epidemiology and modeling approaches to address the complexity of healthcare-associated infections (HAI) and antimicrobial resistance. It demonstrates how epidemiological surveillance data and modeling can be used to infer transmission dynamics in healthcare settings and to forecast healthcare impact, how modeling can be used to improve the validity of interpretation of epidemiological surveillance data, how modeling can be used to estimate the impact of IPC interventions, and how modeling can be used to guide IPC and antimicrobial treatment and stewardship decision-making. There are several priority areas for expanding the use of modeling in healthcare epidemiology and IPC. Importantly, modeling should be viewed as complementary to conventional healthcare epidemiological approaches, and this requires collaboration and active coordination between IPC, healthcare epidemiology, and mathematical modeling groups.

Canada's provincial COVID-19 pandemic modelling efforts: A review of mathematical models and their impacts on the responses

SETTING

Mathematical modelling played an important role in the public health response to COVID-19 in Canada. Variability in epidemic trajectories, modelling approaches, and data infrastructure across provinces provides a unique opportunity to understand the factors that shaped modelling strategies.

INTERVENTION

Provinces implemented stringent pandemic interventions to mitigate SARS-CoV-2 transmission, considering evidence from epidemic models. This study aimed to summarize provincial COVID-19 modelling efforts. We identified modelling teams working with provincial decision-makers, through referrals and membership in Canadian modelling networks. Information on models, data sources, and knowledge translation were abstracted using standardized instruments.

OUTCOMES

We obtained information from six provinces. For provinces with sustained community transmission, initial modelling efforts focused on projecting epidemic trajectories and healthcare demands, and evaluating impacts of proposed interventions. In provinces with low community transmission, models emphasized quantifying importation risks. Most of the models were compartmental and deterministic, with projection horizons of a few weeks. Models were updated regularly or replaced by new ones, adapting to changing local epidemic dynamics, pathogen characteristics, vaccines, and requests from public health. Surveillance datasets for cases, hospitalizations and deaths, and serological studies were the main data sources for model calibration. Access to data for modelling and the structure for knowledge translation differed markedly between provinces.

IMPLICATION

Provincial modelling efforts during the COVID-19 pandemic were tailored to local contexts and modulated by available resources. Strengthening Canadian modelling capacity, developing and sustaining collaborations between modellers and governments, and ensuring earlier access to linked and timely surveillance data could help improve pandemic preparedness.

Title evaluation of FluSight influenza forecasting in the 2021-22 and 2022-23 seasons with a new target laboratory-confirmed influenza hospitalizations

Accurate forecasts can enable more effective public health responses during seasonal influenza epidemics. For the 2021-22 and 2022-23 influenza seasons, 26 forecasting teams provided national and jurisdiction-specific probabilistic predictions of weekly confirmed influenza hospital admissions for one-to-four weeks ahead. Forecast skill is evaluated using the Weighted Interval Score (WIS), relative WIS, and coverage. Six out of 23 models outperform the baseline model across forecast weeks and locations in 2021-22 and 12 out of 18 models in 2022-23. Averaging across all forecast targets, the FluSight ensemble is the 2nd most accurate model measured by WIS in 2021-22 and the 5th most accurate in the 2022-23 season. Forecast skill and 95% coverage for the FluSight ensemble and most component models degrade over longer forecast horizons. In this work we demonstrate that while the FluSight ensemble was a robust predictor, even ensembles face challenges during periods of rapid change.

Scenario design for infectious disease projections: Integrating concepts from decision analysis and experimental design

Across many fields, scenario modeling has become an important tool for exploring long-term projections and how they might depend on potential interventions and critical uncertainties, with relevance to both decision makers and scientists. In the past decade, and especially during the COVID-19 pandemic, the field of epidemiology has seen substantial growth in the use of scenario projections. Multiple scenarios are often projected at the same time, allowing important comparisons that can guide the choice of intervention, the prioritization of research topics, or public communication. The design of the scenarios is central to their ability to inform important questions. In this paper, we draw on the fields of decision analysis and statistical design of experiments to propose a framework for scenario design in epidemiology, with relevance also to other fields. We identify six different fundamental purposes for scenario designs (decision making, sensitivity analysis, situational awareness, horizon scanning, forecasting, and value of information) and discuss how those purposes guide the structure of scenarios. We discuss other aspects of the content and process of scenario design, broadly for all settings and specifically for multi-model ensemble projections. As an illustrative case study, we examine the first 17 rounds of scenarios from the U.S. COVID-19 Scenario Modeling Hub, then reflect on future advancements that could improve the design of scenarios in epidemiological settings.

Data-driven mechanistic framework with stratified immunity and effective transmissibility for COVID-19 scenario projections

Scenario-based modeling frameworks have been widely used to support policy-making at state and federal levels in the United States during the COVID-19 response. While custom-built models can be used to support one-off studies, sustained updates to projections under changing pandemic conditions requires a robust, integrated, and adaptive framework. In this paper, we describe one such framework, UVA-adaptive, that was built to support the CDC-aligned Scenario Modeling Hub (SMH) across multiple rounds, as well as weekly/biweekly projections to Virginia Department of Health (VDH) and US Department of Defense during the COVID-19 response. Building upon an existing metapopulation framework, PatchSim, UVA-adaptive uses a calibration mechanism relying on adjustable effective transmissibility as a basis for scenario definition while also incorporating real-time datasets on case incidence, seroprevalence, variant characteristics, and vaccine uptake. Through the pandemic, our framework evolved by incorporating available data sources and was extended to capture complexities of multiple strains and heterogeneous immunity of the population. Here we present the version of the model that was used for the recent projections for SMH and VDH, describe the calibration and projection framework, and demonstrate that the calibrated transmissibility correlates with the evolution of the pathogen as well as associated societal dynamics.

When do we need multiple infectious disease models? Agreement between projection rank and magnitude in a multi-model setting

Mathematical models are useful for public health planning and response to infectious disease threats. However, different models can provide differing results, which can hamper decision making if not synthesized appropriately. To address this challenge, multi-model hubs convene independent modeling groups to generate ensembles, known to provide more accurate predictions of future outcomes. Yet, these hubs are resource intensive, and how many models are sufficient in a hub is not known. Here, we compare the benefit of predictions from multiple models in different contexts: (1) decision settings that depend on predictions of quantitative outcomes (e.g., hospital capacity planning), where assessments of the benefits of multi-model ensembles have largely focused; and (2) decisions settings that require the ranking of alternative epidemic scenarios (e.g., comparing outcomes under multiple possible interventions and biological uncertainties). We develop a mathematical framework to mimic a multi-model prediction setting, and use this framework to quantify how frequently predictions from different models agree. We further explore multi-model agreement using real-world, empirical data from 14 rounds of U.S. COVID-19 Scenario Modeling Hub projections. Our results suggest that the value of multiple models could be different in different decision contexts, and if only a few models are available, focusing on the rank of alternative epidemic scenarios could be more robust than focusing on quantitative outcomes. Although additional exploration of the sufficient number of models for different contexts is still needed, our results indicate that it may be possible to identify decision contexts where it is robust to rely on fewer models, a finding that can inform the use of modeling resources during future public health crises.

Chimeric Forecasting: An experiment to leverage human judgment to improve forecasts of infectious disease using simulated surveillance data

Forecasts of infectious agents provide public health officials advanced warning about the intensity and timing of the spread of disease. Past work has found that accuracy and calibration of forecasts is weakest when attempting to predict an epidemic peak. Forecasts from a mechanistic model would be improved if there existed accurate information about the timing and intensity of an epidemic. We presented 3000 humans with simulated surveillance data about the number of incident hospitalizations from a current and two past seasons, and asked that they predict the peak time and intensity of the underlying epidemic. We found that in comparison to two control models, a model including human judgment produced more accurate forecasts of peak time and intensity of hospitalizations during an epidemic. Chimeric models have the potential to improve our ability to predict targets of public health interest which may in turn reduce infectious disease burden.

Projecting the future impact of emerging SARS-CoV-2 variants under uncertainty: Modeling the initial Omicron outbreak

Over the past several years, the emergence of novel SARS-CoV-2 variants has led to multiple waves of increased COVID-19 incidence. When the Omicron variant emerged, there was considerable concern about its potential impact in the winter of 2021-2022 due to its increased fitness. However, there was also considerable uncertainty regarding its likely impact due to questions about its relative transmissibility, severity, and degree of immune escape. We sought to evaluate the ability of an agent-based model to forecast incidence in the context of this emerging pathogen variant. To project COVID-19 cases and deaths in Indiana, we calibrated our model to COVID-19 hospitalizations, deaths, and test-positivity rates through November 2021, and then projected COVID-19 incidence through April 2022 under four different scenarios that covered the plausible ranges of Omicron's severity, transmissibility, and degree of immune escape. Our initial projections from December 2021 through March 2022 indicated that under a pessimistic scenario with high disease severity, the peak in weekly COVID-19 deaths in Indiana would be larger than the previous peak in December 2020. However, retrospective analyses indicate that Omicron's severity was closer to the optimistic scenario, and even though cases and hospitalizations reached a new peak, fewer deaths occurred than during the previous peak. According to our results, Omicron's rapid spread was consistent with a combination of higher transmissibility and immune escape relative to earlier variants. Our updated projections starting in January 2022 accurately predicted that cases would peak in mid-January and decline rapidly over the next several months. The performance of our projections shows that following the emergence of a new pathogen variant, models can help quantify the potential range of outbreak magnitudes and trajectories. Agent-based models are particularly useful in these scenarios because they can efficiently track individual vaccination and infection histories with multiple variants with varying degrees of cross-protection.

A multiscale modeling framework for Scenario Modeling: Characterizing the heterogeneity of the COVID-19 epidemic in the US

The Scenario Modeling Hub (SMH) initiative provides projections of potential epidemic scenarios in the United States (US) by using a multi-model approach. Our contribution to the SMH is generated by a multiscale model that combines the global epidemic metapopulation modeling approach (GLEAM) with a local epidemic and mobility model of the US (LEAM-US), first introduced here. The LEAM-US model consists of 3142 subpopulations each representing a single county across the 50 US states and the District of Columbia, enabling us to project state and national trajectories of COVID-19 cases, hospitalizations, and deaths under different epidemic scenarios. The model is age-structured, and multi-strain. It integrates data on vaccine administration, human mobility, and non-pharmaceutical interventions. The model contributed to all 17 rounds of the SMH, and allows for the mechanistic characterization of the spatio-temporal heterogeneities observed during the COVID-19 pandemic. Here we describe the mathematical and computational structure of our model, and present the results concerning the emergence of the SARS-CoV-2 Alpha variant (lineage designation B.1.1.7) as a case study. Our findings show considerable spatial and temporal heterogeneity in the introduction and diffusion of the Alpha variant, both at the level of individual states and combined statistical areas, as it competes against the ancestral lineage. We discuss the key factors driving the time required for the Alpha variant to rise to dominance within a population, and quantify the impact that the emergence of the Alpha variant had on the effective reproduction number at the state level. Overall, we show that our multiscale modeling approach is able to capture the complexity and heterogeneity of the COVID-19 pandemic response in the US.

Nowcasting and forecasting the 2022 U.S. mpox outbreak: Support for public health decision making and lessons learned

In June of 2022, the U.S. Centers for Disease Control and Prevention (CDC) Mpox Response wanted timely answers to important epidemiological questions which can now be answered more effectively through infectious disease modeling. Infectious disease models have shown to be valuable tools for decision making during outbreaks; however, model complexity often makes communicating the results and limitations of models to decision makers difficult. We performed nowcasting and forecasting for the 2022 mpox outbreak in the United States using the R package EpiNow2. We generated nowcasts/forecasts at the national level, by Census region, and for jurisdictions reporting the greatest number of mpox cases. Modeling results were shared for situational awareness within the CDC Mpox Response and publicly on the CDC website. We retrospectively evaluated forecast predictions at four key phases (early, exponential growth, peak, and decline) during the outbreak using three metrics, the weighted interval score, mean absolute error, and prediction interval coverage. We compared the performance of EpiNow2 with a naïve Bayesian generalized linear model (GLM). The EpiNow2 model had less probabilistic error than the GLM during every outbreak phase except for the early phase. We share our experiences with an existing tool for nowcasting/forecasting and highlight areas of improvement for the development of future tools. We also reflect on lessons learned regarding data quality issues and adapting modeling results for different audiences.

Combining models to generate consensus medium-term projections of hospital admissions, occupancy and deaths relating to COVID-19 in England

Mathematical modelling has played an important role in offering informed advice during the COVID-19 pandemic. In England, a cross government and academia collaboration generated medium-term projections (MTPs) of possible epidemic trajectories over the future 4-6 weeks from a collection of epidemiological models. In this article, we outline this collaborative modelling approach and evaluate the accuracy of the combined and individual model projections against the data over the period November 2021-December 2022 when various Omicron subvariants were spreading across England. Using a number of statistical methods, we quantify the predictive performance of the model projections for both the combined and individual MTPs, by evaluating the point and probabilistic accuracy. Our results illustrate that the combined MTPs, produced from an ensemble of heterogeneous epidemiological models, were a closer fit to the data than the individual models during the periods of epidemic growth or decline, with the 90% confidence intervals widest around the epidemic peaks. We also show that the combined MTPs increase the robustness and reduce the biases associated with a single model projection. Learning from our experience of ensemble modelling during the COVID-19 epidemic, our findings highlight the importance of developing cross-institutional multi-model infectious disease hubs for future outbreak control.

Challenges of COVID-19 Case Forecasting in the US, 2020-2021

During the COVID-19 pandemic, forecasting COVID-19 trends to support planning and response was a priority for scientists and decision makers alike. In the United States, COVID-19 forecasting was coordinated by a large group of universities, companies, and government entities led by the Centers for Disease Control and Prevention and the US COVID-19 Forecast Hub (https://covid19forecasthub.org). We evaluated approximately 9.7 million forecasts of weekly state-level COVID-19 cases for predictions 1-4 weeks into the future submitted by 24 teams from August 2020 to December 2021. We assessed coverage of central prediction intervals and weighted interval scores (WIS), adjusting for missing forecasts relative to a baseline forecast, and used a Gaussian generalized estimating equation (GEE) model to evaluate differences in skill across epidemic phases that were defined by the effective reproduction number. Overall, we found high variation in skill across individual models, with ensemble-based forecasts outperforming other approaches. Forecast skill relative to the baseline was generally higher for larger jurisdictions (e.g., states compared to counties). Over time, forecasts generally performed worst in periods of rapid changes in reported cases (either in increasing or decreasing epidemic phases) with 95% prediction interval coverage dropping below 50% during the growth phases of the winter 2020, Delta, and Omicron waves. Ideally, case forecasts could serve as a leading indicator of changes in transmission dynamics. However, while most COVID-19 case forecasts outperformed a naïve baseline model, even the most accurate case forecasts were unreliable in key phases. Further research could improve forecasts of leading indicators, like COVID-19 cases, by leveraging additional real-time data, addressing performance across phases, improving the characterization of forecast confidence, and ensuring that forecasts were coherent across spatial scales. In the meantime, it is critical for forecast users to appreciate current limitations and use a broad set of indicators to inform pandemic-related decision making.

Responding to the Return of Influenza in the United States by Applying Centers for Disease Control and Prevention Surveillance, Analysis, and Modeling to Inform Understanding of Seasonal Influenza

We reviewed the tools that have been developed to characterize and communicate seasonal influenza activity in the United States. Here we focus on systematic surveillance and applied analytics, including seasonal burden and disease severity estimation, short-term forecasting, and longer-term modeling efforts. For each set of activities, we describe the challenges and opportunities that have arisen because of the COVID-19 pandemic. In conclusion, we highlight how collaboration and communication have been and will continue to be key components of reliable and actionable influenza monitoring, forecasting, and modeling activities.

The US COVID-19 and Influenza Scenario Modeling Hubs: Delivering long-term projections to guide policy

Between December 2020 and April 2023, the COVID-19 Scenario Modeling Hub (SMH) generated operational multi-month projections of COVID-19 burden in the US to guide pandemic planning and decision-making in the context of high uncertainty. This effort was born out of an attempt to coordinate, synthesize and effectively use the unprecedented amount of predictive modeling that emerged throughout the COVID-19 pandemic. Here we describe the history of this massive collective research effort, the process of convening and maintaining an open modeling hub active over multiple years, and attempt to provide a blueprint for future efforts. We detail the process of generating 17 rounds of scenarios and projections at different stages of the COVID-19 pandemic, and disseminating results to the public health community and lay public. We also highlight how SMH was expanded to generate influenza projections during the 2022-23 season. We identify key impacts of SMH results on public health and draw lessons to improve future collaborative modeling efforts, research on scenario projections, and the interface between models and policy.

Ensemble2: Scenarios ensembling for communication and performance analysis

Throughout the COVID-19 pandemic, scenario modeling played a crucial role in shaping the decision-making process of public health policies. Unlike forecasts, scenario projections rely on specific assumptions about the future that consider different plausible states-of-the-world that may or may not be realized and that depend on policy interventions, unpredictable changes in the epidemic outlook, etc. As a consequence, long-term scenario projections require different evaluation criteria than the ones used for traditional short-term epidemic forecasts. Here, we propose a novel ensemble procedure for assessing pandemic scenario projections using the results of the Scenario Modeling Hub (SMH) for COVID-19 in the United States (US). By defining a "scenario ensemble" for each model and the ensemble of models, termed "Ensemble2", we provide a synthesis of potential epidemic outcomes, which we use to assess projections' performance, bypassing the identification of the most plausible scenario. We find that overall the Ensemble2 models are well-calibrated and provide better performance than the scenario ensemble of individual models. The ensemble procedure accounts for the full range of plausible outcomes and highlights the importance of scenario design and effective communication. The scenario ensembling approach can be extended to any scenario design strategy, with potential refinements including weighting scenarios and allowing the ensembling process to evolve over time.

Examining the Influence of Imbalanced Social Contact Matrices in Epidemic Models

Transmissible infections such as those caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) spread according to who contacts whom. Therefore, many epidemic models incorporate contact patterns through contact matrices. Contact matrices can be generated from social contact survey data. However, the resulting matrices are often imbalanced, such that the total number of contacts reported by group A with group B do not match those reported by group B with group A. We examined the theoretical influence of imbalanced contact matrices on the estimated basic reproduction number (R0). We then explored how imbalanced matrices may bias model-based epidemic projections using an illustrative simulation model of SARS-CoV-2 with 2 age groups (<15 and ≥15 years). Models with imbalanced matrices underestimated the initial spread of SARS-CoV-2, had later time to peak incidence, and had smaller peak incidence. Imbalanced matrices also influenced cumulative infections observed per age group, as well as the estimated impact of an age-specific vaccination strategy. Stratified transmission models that do not consider contact balancing may generate biased projections of epidemic trajectory and the impact of targeted public health interventions. Therefore, modeling studies should implement and report methods used to balance contact matrices for stratified transmission models.

Evaluation of FluSight influenza forecasting in the 2021-22 and 2022-23 seasons with a new target laboratory-confirmed influenza hospitalizations

Accurate forecasts can enable more effective public health responses during seasonal influenza epidemics. Forecasting teams were asked to provide national and jurisdiction-specific probabilistic predictions of weekly confirmed influenza hospital admissions for one through four weeks ahead for the 2021-22 and 2022-23 influenza seasons. Across both seasons, 26 teams submitted forecasts, with the submitting teams varying between seasons. Forecast skill was evaluated using the Weighted Interval Score (WIS), relative WIS, and coverage. Six out of 23 models outperformed the baseline model across forecast weeks and locations in 2021-22 and 12 out of 18 models in 2022-23. Averaging across all forecast targets, the FluSight ensemble was the 2nd most accurate model measured by WIS in 2021-22 and the 5th most accurate in the 2022-23 season. Forecast skill and 95% coverage for the FluSight ensemble and most component models degraded over longer forecast horizons and during periods of rapid change. Current influenza forecasting efforts help inform situational awareness, but research is needed to address limitations, including decreased performance during periods of changing epidemic dynamics.

A computational approach to identifiability analysis for a model of the propagation and control of COVID-19 in Chile

A computational approach is adapted to analyze the parameter identifiability of a compartmental model. The model is intended to describe the progression of the COVID-19 pandemic in Chile during the initial phase in early 2020 when government declared quarantine measures. The computational approach to analyze the structural and practical identifiability is applied in two parts, one for synthetic data and another for some Chilean regional data. The first part defines the identifiable parameter sets when these recover the true parameters used to create the synthetic data. The second part compares the results derived from synthetic data, estimating the identifiable parameter sets from regional Chilean epidemic data. Experiments provide evidence of the loss of identifiability if some initial conditions are estimated, the period of time used to fit is before the peak, and if a significant proportion of the population is involved in quarantine periods.

Evaluation of the US COVID-19 Scenario Modeling Hub for informing pandemic response under uncertainty

Our ability to forecast epidemics far into the future is constrained by the many complexities of disease systems. Realistic longer-term projections may, however, be possible under well-defined scenarios that specify the future state of critical epidemic drivers. Since December 2020, the U.S. COVID-19 Scenario Modeling Hub (SMH) has convened multiple modeling teams to make months ahead projections of SARS-CoV-2 burden, totaling nearly 1.8 million national and state-level projections. Here, we find SMH performance varied widely as a function of both scenario validity and model calibration. We show scenarios remained close to reality for 22 weeks on average before the arrival of unanticipated SARS-CoV-2 variants invalidated key assumptions. An ensemble of participating models that preserved variation between models (using the linear opinion pool method) was consistently more reliable than any single model in periods of valid scenario assumptions, while projection interval coverage was near target levels. SMH projections were used to guide pandemic response, illustrating the value of collaborative hubs for longer-term scenario projections.

Scenario Design for Infectious Disease Projections: Integrating Concepts from Decision Analysis and Experimental Design

Across many fields, scenario modeling has become an important tool for exploring long-term projections and how they might depend on potential interventions and critical uncertainties, with relevance to both decision makers and scientists. In the past decade, and especially during the COVID-19 pandemic, the field of epidemiology has seen substantial growth in the use of scenario projections. Multiple scenarios are often projected at the same time, allowing important comparisons that can guide the choice of intervention, the prioritization of research topics, or public communication. The design of the scenarios is central to their ability to inform important questions. In this paper, we draw on the fields of decision analysis and statistical design of experiments to propose a framework for scenario design in epidemiology, with relevance also to other fields. We identify six different fundamental purposes for scenario designs (decision making, sensitivity analysis, value of information, situational awareness, horizon scanning, and forecasting) and discuss how those purposes guide the structure of scenarios. We discuss other aspects of the content and process of scenario design, broadly for all settings and specifically for multi-model ensemble projections. As an illustrative case study, we examine the first 17 rounds of scenarios from the U.S. COVID-19 Scenario Modeling Hub, then reflect on future advancements that could improve the design of scenarios in epidemiological settings.

Development and Validation of the US Diabetes, Obesity, Cardiovascular Disease Microsimulation (DOC-M) Model: Health Disparity and Economic Impact Model

BACKGROUND

Few simulation models have incorporated the interplay of diabetes, obesity, and cardiovascular disease (CVD); their upstream lifestyle and biological risk factors; and their downstream effects on health disparities and economic consequences.

METHODS

We developed and validated a US Diabetes, Obesity, Cardiovascular Disease Microsimulation (DOC-M) model that incorporates demographic, clinical, and lifestyle risk factors to jointly predict overall and racial-ethnic groups-specific obesity, diabetes, CVD, and cause-specific mortality for the US adult population aged 40 to 79 y at baseline. An individualized health care cost prediction model was further developed and integrated. This model incorporates nationally representative data on baseline demographics, lifestyle, health, and cause-specific mortality; dynamic changes in modifiable risk factors over time; and parameter uncertainty using probabilistic distributions. Validation analyses included assessment of 1) population-level risk calibration and 2) individual-level risk discrimination. To illustrate the application of the DOC-M model, we evaluated the long-term cost-effectiveness of a national produce prescription program.

RESULTS

Comparing the 15-y model-predicted population risk of primary outcomes among the 2001-2002 National Health and Nutrition Examination Survey (NHANES) cohort with the observed prevalence from age-matched cross-sectional 2003-2016 NHANES cohorts, calibration performance was strong based on observed-to-expected ratio and calibration plot analysis. In most cases, Brier scores fell below 0.0004, indicating a low overall prediction error. Using the Multi-Ethnic Study of Atherosclerosis cohorts, the c-statistics for assessing individual-level risk discrimination were 0.85 to 0.88 for diabetes, 0.93 to 0.95 for obesity, 0.74 to 0.76 for CVD history, and 0.78 to 0.81 for all-cause mortality, both overall and in three racial-ethnic groups. Open-source code for the model was posted at https://github.com/food-price/DOC-M-Model-Development-and-Validation.

CONCLUSIONS

The validated DOC-M model can be used to examine health, equity, and the economic impact of health policies and interventions on behavioral and clinical risk factors for obesity, diabetes, and CVD.

HIGHLIGHTS

We developed a novel microsimula'tion model for obesity, diabetes, and CVD, which intersect together and - critically for prevention and treatment interventions - share common lifestyle, biologic, and demographic risk factors.Validation analyses, including assessment of (1) population-level risk calibration and (2) individual-level risk discrimination, showed strong performance across the overall population and three major racial-ethnic groups for 6 outcomes (obesity, diabetes, CVD, and all-cause mortality, CVD- and DM-cause mortality)This paper provides a thorough explanation and documentation of the development and validation process of a novel microsimulation model, along with the open-source code (https://github.com/food-price/ DOCM_validation) for public use, to serve as a guide for future simulation model assessments, validation, and implementation.

Informing pandemic response in the face of uncertainty. An evaluation of the U.S. COVID-19 Scenario Modeling Hub

Our ability to forecast epidemics more than a few weeks into the future is constrained by the complexity of disease systems, our limited ability to measure the current state of an epidemic, and uncertainties in how human action will affect transmission. Realistic longer-term projections (spanning more than a few weeks) may, however, be possible under defined scenarios that specify the future state of critical epidemic drivers, with the additional benefit that such scenarios can be used to anticipate the comparative effect of control measures. Since December 2020, the U.S. COVID-19 Scenario Modeling Hub (SMH) has convened multiple modeling teams to make 6-month ahead projections of the number of SARS-CoV-2 cases, hospitalizations and deaths. The SMH released nearly 1.8 million national and state-level projections between February 2021 and November 2022. SMH performance varied widely as a function of both scenario validity and model calibration. Scenario assumptions were periodically invalidated by the arrival of unanticipated SARS-CoV-2 variants, but SMH still provided projections on average 22 weeks before changes in assumptions (such as virus transmissibility) invalidated scenarios and their corresponding projections. During these periods, before emergence of a novel variant, a linear opinion pool ensemble of contributed models was consistently more reliable than any single model, and projection interval coverage was near target levels for the most plausible scenarios (e.g., 79% coverage for 95% projection interval). SMH projections were used operationally to guide planning and policy at different stages of the pandemic, illustrating the value of the hub approach for long-term scenario projections.

Forecasting COVID-19 activity in Australia to support pandemic response: May to October 2020

As of January 2021, Australia had effectively controlled local transmission of COVID-19 despite a steady influx of imported cases and several local, but contained, outbreaks in 2020. Throughout 2020, state and territory public health responses were informed by weekly situational reports that included an ensemble forecast of daily COVID-19 cases for each jurisdiction. We present here an analysis of one forecasting model included in this ensemble across the variety of scenarios experienced by each jurisdiction from May to October 2020. We examine how successfully the forecasts characterised future case incidence, subject to variations in data timeliness and completeness, showcase how we adapted these forecasts to support decisions of public health priority in rapidly-evolving situations, evaluate the impact of key model features on forecast skill, and demonstrate how to assess forecast skill in real-time before the ground truth is known. Conditioning the model on the most recent, but incomplete, data improved the forecast skill, emphasising the importance of developing strong quantitative models of surveillance system characteristics, such as ascertainment delay distributions. Forecast skill was highest when there were at least 10 reported cases per day, the circumstances in which authorities were most in need of forecasts to aid in planning and response.

How does framing influence preference for multiple solutions to societal problems?

Solutions to environmental and social problems are often framed in dichotomous ways, which can be counterproductive. Instead, multiple solutions are often needed to fully address these problems. Here we examine how framing influences people's preference for multiple solutions. In a pre-registered experiment, participants (N = 1,432) were randomly assigned to one of four framing conditions. In the first three conditions, participants were presented with a series of eight problems, each framed with multiple causes, multiple impacts, or multiple solutions to the problem. The control condition did not present any framing information. Participants indicated their preferred solution, perceived severity and urgency of the problem, and their dichotomous thinking tendency. Pre-registered analyses showed that none of the three frames had a significant impact on preference for multiple solutions, perceived severity, perceived urgency, or dichotomous thinking. However, exploratory analyses showed that perceived severity and urgency of the problem were positively correlated with people's preference for multiple solutions, while dichotomous thinking was negatively correlated. These findings showed no demonstrable impact of framing on multi-solution preference. Future interventions should focus on addressing perceived severity and urgency, or decreasing dichotomous thinking to encourage people to adopt multiple solutions to address complex environmental and social problems.

Evidence-Based Policy Making during the Coronavirus Disease 2019 Pandemic: A Systematic Review

INTRODUCTION

The aim of this systematic review was to collect evidence and recommendations for the applicability of the concept of evidence-based policy making (EBPM) during the coronavirus disease 2019 (COVID-19) pandemic and to discuss the implementation of this concept from a medical science perspective.

METHODS

This study was performed according to the guidelines, checklist, and flow diagram of Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020. An electronic literature search was conducted on September 20, 2022 using PubMed, Web of Science, Cochrane Library, and CINAHL databases with the following search terms: "evidence based policy making" and "infectious disease." Study eligibility assessment was performed based on the flow diagram of PRISMA 2020, and risk of bias assessment was performed using The Critical Appraisal Skills Program.

RESULTS

Eleven eligible articles were included in this review and divided into three groups as follows: early, middle, and late stages of the COVID-19 pandemic. Basics of COVID-19 control were suggested in the early stage. The articles published in the middle stage discussed the importance of the collection and analysis of evidence of COVID-19 from around the world for the establishment of EBPM in the COVID-19 pandemic. The articles published in the late stage discussed the collection of large amounts of high-quality data and the development of methods to analyze them, as well as emerging issues related to the COVID-19 pandemic.

CONCLUSIONS

This study revealed that the concept of EBPM applicable to emerging infectious disease pandemics changed between the early, middle, and late stages of the pandemic. The concept of EBPM will play an important role in medicine in the future.

Barriers to conducting independent quantitative research in low-income countries: A cross-sectional study of public health graduate students in Liberia

INTRODUCTION

During recent disease outbreaks, quantitative research has been used to investigate intervention scenarios while accounting for local epidemiological, social, and clinical context. Despite the value of such work, few documented research efforts have been observed to originate from low-income countries. This study aimed to assess barriers that may be limiting the awareness and conduct of quantitative research among Liberian public health graduate students.

METHODS

A semi-structured questionnaire was administered September-November 2021 to Master's in Public Health (MPH) students in Liberia. Potential barriers around technology access, understanding of quantitative science, and availability of mentorship were interrogated. Associations between barriers and self-reported likelihood of conducting quantitative research within six months of the investigation period were evaluated using ordinal logistic regression.

RESULTS

Among 120 participating MPH students, 86% reported owning a personal computer, but 18.4% and 39.4% had machines with malfunctioning hardware and/or with battery power lasting ≤2 hours, respectively. On average, students reported having poor internet network 3.4 days weekly. 47% reported never using any computer software for analysis, and 46% reported no specific knowledge on statistical analysis. Students indicated spending a median 30 minutes per week reading scientific articles. Moreover, 50% had no access to quantitative research mentors. Despite barriers, 59% indicated they were very likely to undertake quantitative research in the next 6 months; only 7% indicated they were not at all likely. Computer ownership was found to be statistically significantly associated with higher likelihood of conducting quantitative research in the multivariable analysis (aOR: 4.90,95% CI: 1.54-16.3).

CONCLUSION

The high likelihood of conducting quantitative research among MPH students contrasts with limitations around computing capacity, awareness of research tools/methods, and access to mentorship. To promote rigorous analytical research in Liberia, there is a need for systematic measures to enhance capacity for diverse quantitative methods through efforts sensitive to the local research environment.

Context-dependent representation of within- and between-model uncertainty: aggregating probabilistic predictions in infectious disease epidemiology

Probabilistic predictions support public health planning and decision making, especially in infectious disease emergencies. Aggregating outputs from multiple models yields more robust predictions of outcomes and associated uncertainty. While the selection of an aggregation method can be guided by retrospective performance evaluations, this is not always possible. For example, if predictions are conditional on assumptions about how the future will unfold (e.g. possible interventions), these assumptions may never materialize, precluding any direct comparison between predictions and observations. Here, we summarize literature on aggregating probabilistic predictions, illustrate various methods for infectious disease predictions via simulation, and present a strategy for choosing an aggregation method when empirical validation cannot be used. We focus on the linear opinion pool (LOP) and Vincent average, common methods that make different assumptions about between-prediction uncertainty. We contend that assumptions of the aggregation method should align with a hypothesis about how uncertainty is expressed within and between predictions from different sources. The LOP assumes that between-prediction uncertainty is meaningful and should be retained, while the Vincent average assumes that between-prediction uncertainty is akin to sampling error and should not be preserved. We provide an R package for implementation. Given the rising importance of multi-model infectious disease hubs, our work provides useful guidance on aggregation and a deeper understanding of the benefits and risks of different approaches.

Crowdsourced Perceptions of Human Behavior to Improve Computational Forecasts of US National Incident Cases of COVID-19: Survey Study

BACKGROUND

Past research has shown that various signals associated with human behavior (eg, social media engagement) can benefit computational forecasts of COVID-19. One behavior that has been shown to reduce the spread of infectious agents is compliance with nonpharmaceutical interventions (NPIs). However, the extent to which the public adheres to NPIs is difficult to measure and consequently difficult to incorporate into computational forecasts of infectious diseases. Soliciting judgments from many individuals (ie, crowdsourcing) can lead to surprisingly accurate estimates of both current and future targets of interest. Therefore, asking a crowd to estimate community-level compliance with NPIs may prove to be an accurate and predictive signal of an infectious disease such as COVID-19.

OBJECTIVE

We aimed to show that crowdsourced perceptions of compliance with NPIs can be a fast and reliable signal that can predict the spread of an infectious agent. We showed this by measuring the correlation between crowdsourced perceptions of NPIs and US incident cases of COVID-19 1-4 weeks ahead, and evaluating whether incorporating crowdsourced perceptions improves the predictive performance of a computational forecast of incident cases.

METHODS

For 36 weeks from September 2020 to April 2021, we asked 2 crowds 21 questions about their perceptions of community adherence to NPIs and public health guidelines, and collected 10,120 responses. Self-reported state residency was compared to estimates from the US census to determine the representativeness of the crowds. Crowdsourced NPI signals were mapped to 21 mean perceived adherence (MEPA) signals and analyzed descriptively to investigate features, such as how MEPA signals changed over time and whether MEPA time series could be clustered into groups based on response patterns. We investigated whether MEPA signals were associated with incident cases of COVID-19 1-4 weeks ahead by (1) estimating correlations between MEPA and incident cases, and (2) including MEPA into computational forecasts.

RESULTS

The crowds were mostly geographically representative of the US population with slight overrepresentation in the Northeast. MEPA signals tended to converge toward moderate levels of compliance throughout the survey period, and an unsupervised analysis revealed signals clustered into 4 groups roughly based on the type of question being asked. Several MEPA signals linearly correlated with incident cases of COVID-19 1-4 weeks ahead at the US national level. Including questions related to social distancing, testing, and limiting large gatherings increased out-of-sample predictive performance for probabilistic forecasts of incident cases of COVID-19 1-3 weeks ahead when compared to a model that was trained on only past incident cases.

CONCLUSIONS

Crowdsourced perceptions of nonpharmaceutical adherence may be an important signal to improve forecasts of the trajectory of an infectious agent and increase public health situational awareness.

Aggregating Human Judgment Probabilistic Predictions of Coronavirus Disease 2019 Transmission, Burden, and Preventive Measures

Aggregated human judgment forecasts for coronavirus disease 2019 (COVID-19) targets of public health importance are accurate, often outperforming computational models. Our work shows that aggregated human judgment forecasts for infectious agents are timely, accurate, and adaptable, and can be used as a tool to aid public health decision making during outbreaks.

Aggregating human judgment probabilistic predictions of COVID-19 transmission, burden, and preventative measures

Aggregated human judgment forecasts for COVID-19 targets of public health importance are accurate, often outperforming computational models. Our work shows aggregated human judgment forecasts for infectious agents are timely, accurate, and adaptable, and can be used as tool to aid public health decision making during outbreaks.