Recommended reporting items for epidemic forecasting and prediction research: The EPIFORGE 2020 guidelines

Incorporating endogenous human behavior in models of COVID-19 transmission: A systematic scoping review

Background

During the COVID-19 pandemic there was a plethora of dynamical forecasting models created, but their ability to effectively describe future trajectories of disease was mixed. A major challenge in evaluating future case trends was forecasting the behavior of individuals. When behavior was incorporated into models, it was primarily incorporated exogenously (e.g., fitting to cellphone mobility data). Fewer models incorporated behavior endogenously (e.g., dynamically changing a model parameter throughout the simulation).

Methods

This review aimed to qualitatively characterize models that included an adaptive (endogenous) behavioral element in the context of COVID-19 transmission. We categorized studies into three approaches: 1) feedback loops, 2) game theory/utility theory, and 3) information/opinion spread.

Findings

Of the 92 included studies, 72% employed a feedback loop, 27% used game/utility theory, and 9% used a model if information/opinion spread. Among all studies, 89% used a compartmental model alone or in combination with other model types. Similarly, 15% used a network model, 11% used an agent-based model, 7% used a system dynamics model, and 1% used a Markov chain model. Descriptors of behavior change included mask-wearing, social distancing, vaccination, and others. Sixty-eight percent of studies calibrated their model to observed data and 25% compared simulated forecasts to observed data. Forty-one percent of studies compared versions of their model with and without endogenous behavior. Models with endogenous behavior tended to show a smaller and delayed initial peak with subsequent periodic waves.

Interpretation

While many COVID-19 models incorporated behavior exogenously, these approaches may fail to capture future adaptations in human behavior, resulting in under- or overestimates of disease burden. By incorporating behavior endogenously, the next generation of infectious disease models could more effectively predict outcomes so that decision makers can better prepare for and respond to epidemics.

Funding

This study was funded in-part by Centers for Disease Control and Prevention (CDC) MInD-Healthcare Program (1U01CK000536), the National Science Foundation (NSF) Modeling Dynamic Disease-Behavior Feedbacks for Improved Epidemic Prediction and Response grant (2229996), and the NSF PIPP Phase I: Evaluating the Effectiveness of Messaging and Modeling during Pandemics grant (2200256).

Prognosis prediction models for post-stroke depression: a protocol for systematic review, meta-analysis, and critical appraisal

INTRODUCTION

Post-stroke depression (PSD) is a prevalent complication that has been shown to have a negative impact on rehabilitation outcomes and quality of life and poses a significant risk for suicidal intention. However, models for discriminating and predicting PSD in stroke survivors for effective secondary prevention strategies are inadequate as the pathogenesis of PSD remains unknown. Prognostic prediction models that exhibit greater rule-in capacity have the potential to mitigate the issue of underdiagnosis and undertreatment of PSD. Thus, the planned study aims to systematically review and critically evaluate published studies on prognostic prediction models for PSD.

METHODS AND ANALYSIS

A systematic literature search will be conducted in PubMed and Embase through Ovid. Two reviewers will complete study screening, data extraction, and quality assessment utilizing appropriate tools. Qualitative data on the characteristics of the included studies, methodological quality, and the appraisal of the clinical applicability of models will be summarized in the form of narrative comments and tables or figures. The predictive performance of the same model involving multiple studies will be synthesized with a random effects meta-analysis model or meta-regression, taking into account heterogeneity.

ETHICS AND DISSEMINATION

Ethical approval is considered not applicable for this systematic review. Findings will be shared through dissemination at academic conferences and/or publication in peer-reviewed academic journals.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42023388548.

Informing Public Health Policies with Models for Disease Burden, Impact Evaluation, and Economic Evaluation

Conducting real-world public health experiments is often costly, time-consuming, and ethically challenging, so mathematical models have a long-standing history of being used to inform policy. Applications include estimating disease burden, performing economic evaluation of interventions, and responding to health emergencies such as pandemics. Models played a pivotal role during the COVID-19 pandemic, providing early detection of SARS-CoV-2's pandemic potential and informing subsequent public health measures. While models offer valuable policy insights, they often carry limitations, especially when they depend on assumptions and incomplete data. Striking a balance between accuracy and timely decision-making in rapidly evolving situations such as disease outbreaks is challenging. Modelers need to explore the extent to which their models deviate from representing the real world. The uncertainties inherent in models must be effectively communicated to policy makers and the public. As the field becomes increasingly influential, it needs to develop reporting standards that enable rigorous external scrutiny.

Challenges of COVID-19 Case Forecasting in the US, 2020-2021

During the COVID-19 pandemic, forecasting COVID-19 trends to support planning and response was a priority for scientists and decision makers alike. In the United States, COVID-19 forecasting was coordinated by a large group of universities, companies, and government entities led by the Centers for Disease Control and Prevention and the US COVID-19 Forecast Hub (https://covid19forecasthub.org). We evaluated approximately 9.7 million forecasts of weekly state-level COVID-19 cases for predictions 1-4 weeks into the future submitted by 24 teams from August 2020 to December 2021. We assessed coverage of central prediction intervals and weighted interval scores (WIS), adjusting for missing forecasts relative to a baseline forecast, and used a Gaussian generalized estimating equation (GEE) model to evaluate differences in skill across epidemic phases that were defined by the effective reproduction number. Overall, we found high variation in skill across individual models, with ensemble-based forecasts outperforming other approaches. Forecast skill relative to the baseline was generally higher for larger jurisdictions (e.g., states compared to counties). Over time, forecasts generally performed worst in periods of rapid changes in reported cases (either in increasing or decreasing epidemic phases) with 95% prediction interval coverage dropping below 50% during the growth phases of the winter 2020, Delta, and Omicron waves. Ideally, case forecasts could serve as a leading indicator of changes in transmission dynamics. However, while most COVID-19 case forecasts outperformed a naïve baseline model, even the most accurate case forecasts were unreliable in key phases. Further research could improve forecasts of leading indicators, like COVID-19 cases, by leveraging additional real-time data, addressing performance across phases, improving the characterization of forecast confidence, and ensuring that forecasts were coherent across spatial scales. In the meantime, it is critical for forecast users to appreciate current limitations and use a broad set of indicators to inform pandemic-related decision making.

Wastewater-based epidemiology for COVID-19 using dynamic artificial neural networks

Global efforts in vaccination have led to a decrease in COVID-19 mortality but a high circulation of SARS-CoV-2 is still observed in several countries, resulting in some cases of severe lockdowns. In this sense, wastewater-based epidemiology remains a powerful tool for supporting regional health administrations in assessing risk levels and acting accordingly. In this work, a dynamic artificial neural network (DANN) has been developed for predicting the number of COVID-19 hospitalized patients in hospitals in Valladolid (Spain). This model takes as inputs a wastewater epidemiology indicator for COVID-19 (concentration of RNA from SARS-CoV-2 N1 gene reported from Valladolid Wastewater Treatment Plant), vaccination coverage, and past data of hospitalizations. The model considered both the instantaneous values of these variables and their historical evolution. Two study periods were selected (from May 2021 until September 2022 and from September 2022 to July 2023). During the first period, accurate predictions of hospitalizations (with an overall range between 6 and 171) were favored by the correlation of this indicator with N1 concentrations in wastewater (r = 0.43, p < 0.05), showing accurate forecasting for 1 day ahead and 5 days ahead. The second period's retraining strategy maintained the overall accuracy of the model despite lower hospitalizations. Furthermore, risk levels were assigned to each 1 day ahead prediction during the first and second periods, showing agreement with the level measured and reported by regional health authorities in 95 % and 93 % of cases, respectively. These results evidenced the potential of this novel DANN model for predicting COVID-19 hospitalizations based on SARS-CoV-2 wastewater concentrations at a regional scale. The model architecture herein developed can support regional health authorities in COVID-19 risk management based on wastewater-based epidemiology.

Mechanistic models for West Nile virus transmission: a systematic review of features, aims and parametrization

Mathematical models within the Ross-Macdonald framework increasingly play a role in our understanding of vector-borne disease dynamics and as tools for assessing scenarios to respond to emerging threats. These threats are typically characterized by a high degree of heterogeneity, introducing a range of possible complexities in models and challenges to maintain the link with empirical evidence. We systematically identified and analysed a total of 77 published papers presenting compartmental West Nile virus (WNV) models that use parameter values derived from empirical studies. Using a set of 15 criteria, we measured the dissimilarity compared with the Ross-Macdonald framework. We also retrieved the purpose and type of models and traced the empirical sources of their parameters. Our review highlights the increasing refinements in WNV models. Models for prediction included the highest number of refinements. We found uneven distributions of refinements and of evidence for parameter values. We identified several challenges in parametrizing such increasingly complex models. For parameters common to most models, we also synthesize the empirical evidence for their values and ranges. The study highlights the potential to improve the quality of WNV models and their applicability for policy by establishing closer collaboration between mathematical modelling and empirical work.

Reproducibility of COVID-era infectious disease models

Infectious disease modelling has been prominent throughout the COVID-19 pandemic, helping to understand the virus' transmission dynamics and inform response policies. Given their potential importance and translational impact, we evaluated the computational reproducibility of infectious disease modelling articles from the COVID era. We found that four out of 100 randomly sampled studies released between January 2020 and August 2022 could be completely computationally reproduced using the resources provided (e.g., code, data, instructions) whilst a further eight were partially reproducible. For the 100 most highly cited articles from the same period we found that 11 were completely reproducible with a further 22 partially reproducible. Reflecting on our experience, we discuss common issues affecting computational reproducibility and how these might be addressed.

Time series analysis of malaria cases to assess the impact of various interventions over the last three decades and forecasting malaria in India towards the 2030 elimination goals

BACKGROUND

Despite the progress made in this decade towards malaria elimination, it remains a significant public health concern in India and many other countries in South Asia and Asia Pacific region. Understanding the historical trends of malaria incidence in relation to various commodity and policy interventions and identifying the factors associated with its occurrence can inform future intervention strategies for malaria elimination goals.

METHODS

This study analysed historical malaria cases in India from 1990 to 2022 to assess the annual trends and the impact of key anti-malarial interventions on malaria incidence. Factors associated with malaria incidence were identified using univariate and multivariate linear regression analyses. Generalized linear, smoothing, autoregressive integrated moving averages (ARIMA) and Holt's models were used to forecast malaria cases from 2023 to 2030.

RESULTS

The reported annual malaria cases in India during 1990-2000 were 2.38 million, which dropped to 0.73 million cases annually during 2011-2022. The overall reduction from 1990 (2,018,783) to 2022 (176,522) was 91%. The key interventions of the Enhanced Malaria Control Project (EMCP), Intensified Malaria Control Project (IMCP), use of bivalent rapid diagnostic tests (RDT-Pf/Pv), artemisinin-based combination therapy (ACT), and involvement of the Accredited Social Health Activists (ASHAs) as front-line workers were found to result in the decline of malaria significantly. The ARIMA and Holt's models projected a continued decline in cases with the potential for reaching zero indigenous cases by 2027-2028. Important factors influencing malaria incidence included tribal population density, literacy rate, health infrastructure, and forested and hard-to-reach areas.

CONCLUSIONS

Studies aimed at assessing the impact of major commodity and policy interventions on the incidence of disease and studies of disease forecasting will inform programmes and policymakers of steps needed during the last mile phase to achieve malaria elimination. It is proposed that these time series and disease forecasting studies should be performed periodically using granular (monthly) and meteorological data to validate predictions of prior studies and suggest any changes needed for elimination efforts at national and sub-national levels.

Bridging the gap - estimation of 2022/2023 SARS-CoV-2 healthcare burden in Germany based on multidimensional data from a rapid epidemic panel

OBJECTIVES

Throughout the SARS-CoV-2 pandemic, Germany like other countries lacked adaptive population-based panels to monitor the spread of epidemic diseases.

METHODS

To fill a gap in population-based estimates needed for winter 2022/23 we resampled in the German SARS-CoV-2 cohort study MuSPAD in mid-2022, including characterization of systemic cellular and humoral immune responses by interferon-γ-release assay (IGRA) and CLIA/IVN assay. We were able to confirm categorization of our study population into four groups with differing protection levels against severe COVID-19 courses based on literature synthesis. Using these estimates, we assessed potential healthcare burden for winter 2022/23 in different scenarios with varying assumptions on transmissibility, pathogenicity, new variants, and vaccine booster campaigns in ordinary differential equation models.

RESULTS

We included 9921 participants from eight German regions. While 85% of individuals were located in one of the two highest protection categories, hospitalization estimates from scenario modeling were highly dependent on viral variant characteristics ranging from 30-300% compared to the 02/2021 peak. Our results were openly communicated and published to an epidemic panel network and a newly established modeling network.

CONCLUSIONS

We demonstrate feasibility of a rapid epidemic panel to provide complex immune protection levels for inclusion in dynamic disease burden modeling scenarios.

Assessing the utility of COVID-19 case reports as a leading indicator for hospitalization forecasting in the United States

Identifying data streams that can consistently improve the accuracy of epidemiological forecasting models is challenging. Using models designed to predict daily state-level hospital admissions due to COVID-19 in California and Massachusetts, we investigated whether incorporating COVID-19 case data systematically improved forecast accuracy. Additionally, we considered whether using case data aggregated by date of test or by date of report from a surveillance system made a difference to the forecast accuracy. Evaluating forecast accuracy in a test period, after first having selected the best-performing methods in a validation period, we found that overall the difference in accuracy between approaches was small, especially at forecast horizons of less than two weeks. However, forecasts from models using cases aggregated by test date showed lower accuracy at longer horizons and at key moments in the pandemic, such as the peak of the Omicron wave in January 2022. Overall, these results highlight the challenge of finding a modeling approach that can generate accurate forecasts of outbreak trends both during periods of relative stability and during periods that show rapid growth or decay of transmission rates. While COVID-19 case counts seem to be a natural choice to help predict COVID-19 hospitalizations, in practice any benefits we observed were small and inconsistent.

Evaluation of the US COVID-19 Scenario Modeling Hub for informing pandemic response under uncertainty

Our ability to forecast epidemics far into the future is constrained by the many complexities of disease systems. Realistic longer-term projections may, however, be possible under well-defined scenarios that specify the future state of critical epidemic drivers. Since December 2020, the U.S. COVID-19 Scenario Modeling Hub (SMH) has convened multiple modeling teams to make months ahead projections of SARS-CoV-2 burden, totaling nearly 1.8 million national and state-level projections. Here, we find SMH performance varied widely as a function of both scenario validity and model calibration. We show scenarios remained close to reality for 22 weeks on average before the arrival of unanticipated SARS-CoV-2 variants invalidated key assumptions. An ensemble of participating models that preserved variation between models (using the linear opinion pool method) was consistently more reliable than any single model in periods of valid scenario assumptions, while projection interval coverage was near target levels. SMH projections were used to guide pandemic response, illustrating the value of collaborative hubs for longer-term scenario projections.

Assessing the impact of COVID-19 passes and mandates on disease transmission, vaccination intention, and uptake: a scoping review

PURPOSE

Policymakers have struggled to maintain SARS-CoV-2 transmission at levels that are manageable to contain the COVID-19 disease burden while enabling a maximum of societal and economic activities. One of the tools that have been used to facilitate this is the so-called "COVID-19 pass". We aimed to document current evidence on the effectiveness of COVID-19 passes, distinguishing their indirect effects by improving vaccination intention and uptake from their direct effects on COVID-19 transmission measured by the incidence of cases, hospitalizations, and deaths.

METHODS

We performed a scoping review on the scientific literature of the proposed topic covering the period January 2021 to September 2022, in accordance with the PRISMA-ScR guidelines for scoping reviews.

RESULTS

Out of a yield of 4,693 publications, 45 studies from multiple countries were retained for full-text review. The results suggest that implementing COVID-19 passes tends to reduce the incidence of cases, hospitalizations, and deaths due to COVID-19. The use of COVID-19 passes was also shown to improve overall vaccination uptake and intention, but not in people who hold strong anti-COVID-19 vaccine beliefs.

CONCLUSION

The evidence from the literature we reviewed tends to indicate positive direct and indirect effects from the use of COVID-19 passes. A major limitation to establishing this firmly is the entanglement of individual effects of multiple measures being implemented simultaneously.

Correction: Recommended reporting items for epidemic forecasting and prediction research: The EPIFORGE 2020 guidelines

[This corrects the article DOI: 10.1371/journal.pmed.1003793.].

Inter-rater reliability of the infectious disease modeling reproducibility checklist (IDMRC) as applied to COVID-19 computational modeling research

BACKGROUND

Infectious disease computational modeling studies have been widely published during the coronavirus disease 2019 (COVID-19) pandemic, yet they have limited reproducibility. Developed through an iterative testing process with multiple reviewers, the Infectious Disease Modeling Reproducibility Checklist (IDMRC) enumerates the minimal elements necessary to support reproducible infectious disease computational modeling publications. The primary objective of this study was to assess the reliability of the IDMRC and to identify which reproducibility elements were unreported in a sample of COVID-19 computational modeling publications.

METHODS

Four reviewers used the IDMRC to assess 46 preprint and peer reviewed COVID-19 modeling studies published between March 13th, 2020, and July 30th, 2020. The inter-rater reliability was evaluated by mean percent agreement and Fleiss' kappa coefficients (κ). Papers were ranked based on the average number of reported reproducibility elements, and average proportion of papers that reported each checklist item were tabulated.

RESULTS

Questions related to the computational environment (mean κ = 0.90, range = 0.90-0.90), analytical software (mean κ = 0.74, range = 0.68-0.82), model description (mean κ = 0.71, range = 0.58-0.84), model implementation (mean κ = 0.68, range = 0.39-0.86), and experimental protocol (mean κ = 0.63, range = 0.58-0.69) had moderate or greater (κ > 0.41) inter-rater reliability. Questions related to data had the lowest values (mean κ = 0.37, range = 0.23-0.59). Reviewers ranked similar papers in the upper and lower quartiles based on the proportion of reproducibility elements each paper reported. While over 70% of the publications provided data used in their models, less than 30% provided the model implementation.

CONCLUSIONS

The IDMRC is the first comprehensive, quality-assessed tool for guiding researchers in reporting reproducible infectious disease computational modeling studies. The inter-rater reliability assessment found that most scores were characterized by moderate or greater agreement. These results suggest that the IDMRC might be used to provide reliable assessments of the potential for reproducibility of published infectious disease modeling publications. Results of this evaluation identified opportunities for improvement to the model implementation and data questions that can further improve the reliability of the checklist.

A systematic review of the data, methods and environmental covariates used to map Aedes-borne arbovirus transmission risk

BACKGROUND

Aedes (Stegomyia)-borne diseases are an expanding global threat, but gaps in surveillance make comprehensive and comparable risk assessments challenging. Geostatistical models combine data from multiple locations and use links with environmental and socioeconomic factors to make predictive risk maps. Here we systematically review past approaches to map risk for different Aedes-borne arboviruses from local to global scales, identifying differences and similarities in the data types, covariates, and modelling approaches used.

METHODS

We searched on-line databases for predictive risk mapping studies for dengue, Zika, chikungunya, and yellow fever with no geographical or date restrictions. We included studies that needed to parameterise or fit their model to real-world epidemiological data and make predictions to new spatial locations of some measure of population-level risk of viral transmission (e.g. incidence, occurrence, suitability, etc.).

RESULTS

We found a growing number of arbovirus risk mapping studies across all endemic regions and arboviral diseases, with a total of 176 papers published 2002-2022 with the largest increases shortly following major epidemics. Three dominant use cases emerged: (i) global maps to identify limits of transmission, estimate burden and assess impacts of future global change, (ii) regional models used to predict the spread of major epidemics between countries and (iii) national and sub-national models that use local datasets to better understand transmission dynamics to improve outbreak detection and response. Temperature and rainfall were the most popular choice of covariates (included in 50% and 40% of studies respectively) but variables such as human mobility are increasingly being included. Surprisingly, few studies (22%, 31/144) robustly tested combinations of covariates from different domains (e.g. climatic, sociodemographic, ecological, etc.) and only 49% of studies assessed predictive performance via out-of-sample validation procedures.

CONCLUSIONS

Here we show that approaches to map risk for different arboviruses have diversified in response to changing use cases, epidemiology and data availability. We identify key differences in mapping approaches between different arboviral diseases, discuss future research needs and outline specific recommendations for future arbovirus mapping.

The coming Omicron waves and factors affecting its spread after China reopening borders

The Chinese government relaxed the Zero-COVID policy on Dec 15, 2022, and reopened the border on Jan 8, 2023. Therefore, COVID prevention in China is facing new challenges. Though there are plenty of prior studies on COVID, none is regarding the predictions on daily confirmed cases, and medical resources needs after China reopens its borders. To fill this gap, this study innovates a combination of the Erdos Renyl network, modified computational model [Formula: see text], and python code instead of only mathematical formulas or computer simulations in the previous studies. The research background in this study is Shanghai, a representative city in China. Therefore, the results in this study also demonstrate the situation in other regions of China. According to the population distribution and migration characteristics, we divided Shanghai into six epidemic research areas. We built a COVID spread model of the Erodos Renyl network. And then, we use python code to simulate COVID spread based on modified [Formula: see text] model. The results demonstrate that the second and third waves will occur in July-September and Oct-Dec, respectively. At the peak of the epidemic in 2023, the daily confirmed cases will be 340,000, and the cumulative death will be about 31,500. Moreover, 74,000 hospital beds and 3,700 Intensive Care Unit (ICU) beds will be occupied in Shanghai. Therefore, Shanghai faces a shortage of medical resources. In this simulation, daily confirmed cases predictions significantly rely on transmission, migration, and waning immunity rate. The study builds a mixed-effect model to verify further the three parameters' effect on the new confirmed cases. The results demonstrate that migration and waning immunity rates are two significant parameters in COVID spread and daily confirmed cases. This study offers theoretical evidence for the government to prevent COVID after China opened its borders.

Informing pandemic response in the face of uncertainty. An evaluation of the U.S. COVID-19 Scenario Modeling Hub

Our ability to forecast epidemics more than a few weeks into the future is constrained by the complexity of disease systems, our limited ability to measure the current state of an epidemic, and uncertainties in how human action will affect transmission. Realistic longer-term projections (spanning more than a few weeks) may, however, be possible under defined scenarios that specify the future state of critical epidemic drivers, with the additional benefit that such scenarios can be used to anticipate the comparative effect of control measures. Since December 2020, the U.S. COVID-19 Scenario Modeling Hub (SMH) has convened multiple modeling teams to make 6-month ahead projections of the number of SARS-CoV-2 cases, hospitalizations and deaths. The SMH released nearly 1.8 million national and state-level projections between February 2021 and November 2022. SMH performance varied widely as a function of both scenario validity and model calibration. Scenario assumptions were periodically invalidated by the arrival of unanticipated SARS-CoV-2 variants, but SMH still provided projections on average 22 weeks before changes in assumptions (such as virus transmissibility) invalidated scenarios and their corresponding projections. During these periods, before emergence of a novel variant, a linear opinion pool ensemble of contributed models was consistently more reliable than any single model, and projection interval coverage was near target levels for the most plausible scenarios (e.g., 79% coverage for 95% projection interval). SMH projections were used operationally to guide planning and policy at different stages of the pandemic, illustrating the value of the hub approach for long-term scenario projections.

Joint COVID-19 and influenza-like illness forecasts in the United States using internet search information

BACKGROUND

As the prolonged COVID-19 pandemic continues, severe seasonal Influenza (flu) may happen alongside COVID-19. This could cause a "twindemic", in which there are additional burdens on health care resources and public safety compared to those occurring in the presence of a single infection. Amidst the raising trend of co-infections of the two diseases, forecasting both Influenza-like Illness (ILI) outbreaks and COVID-19 waves in a reliable and timely manner becomes more urgent than ever. Accurate and real-time joint prediction of the twindemic aids public health organizations and policymakers in adequate preparation and decision making. However, in the current pandemic, existing ILI and COVID-19 forecasting models face shortcomings under complex inter-disease dynamics, particularly due to the similarities in symptoms and healthcare-seeking patterns of the two diseases.

METHODS

Inspired by the interconnection between ILI and COVID-19 activities, we combine related internet search and bi-disease time series information for the U.S. national level and state level forecasts. Our proposed ARGOX-Joint-Ensemble adopts a new ensemble framework that integrates ILI and COVID-19 disease forecasting models to pool the information between the two diseases and provide joint multi-resolution and multi-target predictions. Through a winner-takes-all ensemble fashion, our framework is able to adaptively select the most predictive COVID-19 or ILI signals.

RESULTS

In the retrospective evaluation, our model steadily outperforms alternative benchmark methods, and remains competitive with other publicly available models in both point estimates and probabilistic predictions (including intervals).

CONCLUSIONS

The success of our approach illustrates that pooling information between the ILI and COVID-19 leads to improved forecasting models than individual models for either of the disease.

Inter-rater reliability of the Infectious Disease Modeling Reproducibility Checklist (IDMRC) as applied to COVID-19 computational modeling research

Background

Infectious disease computational modeling studies have been widely published during the coronavirus disease 2019 (COVID-19) pandemic, yet they have limited reproducibility. Developed through an iterative testing process with multiple reviewers, the Infectious Disease Modeling Reproducibility Checklist (IDMRC) enumerates the minimal elements necessary to support reproducible infectious disease computational modeling publications. The primary objective of this study was to assess the reliability of the IDMRC and to identify which reproducibility elements were unreported in a sample of COVID-19 computational modeling publications.

Methods

Four reviewers used the IDMRC to assess 46 preprint and peer reviewed COVID-19 modeling studies published between March 13th, 2020, and July 31st, 2020. The inter-rater reliability was evaluated by mean percent agreement and Fleiss' kappa coefficients (κ). Papers were ranked based on the average number of reported reproducibility elements, and average proportion of papers that reported each checklist item were tabulated.

Results

Questions related to the computational environment (mean κ = 0.90, range = 0.90-0.90), analytical software (mean κ = 0.74, range = 0.68-0.82), model description (mean κ = 0.71, range = 0.58-0.84), model implementation (mean κ = 0.68, range = 0.39-0.86), and experimental protocol (mean κ = 0.63, range = 0.58-0.69) had moderate or greater (κ > 0.41) inter-rater reliability. Questions related to data had the lowest values (mean κ = 0.37, range = 0.23-0.59). Reviewers ranked similar papers in the upper and lower quartiles based on the proportion of reproducibility elements each paper reported. While over 70% of the publications provided data used in their models, less than 30% provided the model implementation.

Conclusions

The IDMRC is the first comprehensive, quality-assessed tool for guiding researchers in reporting reproducible infectious disease computational modeling studies. The inter-rater reliability assessment found that most scores were characterized by moderate or greater agreement. These results suggests that the IDMRC might be used to provide reliable assessments of the potential for reproducibility of published infectious disease modeling publications. Results of this evaluation identified opportunities for improvement to the model implementation and data questions that can further improve the reliability of the checklist.

Assessing the utility of COVID-19 case reports as a leading indicator for hospitalization forecasting in the United States

Identifying data streams that can consistently improve the accuracy of epidemiological forecasting models is challenging. Using models designed to predict daily state-level hospital admissions due to COVID-19 in California and Massachusetts, we investigated whether incorporating COVID-19 case data systematically improved forecast accuracy. Additionally, we considered whether using case data aggregated by date of test or by date of report from a surveillance system made a difference to the forecast accuracy. Evaluating forecast accuracy in a test period, after first having selected the best-performing methods in a validation period, we found that overall the difference in accuracy between approaches was small, especially at forecast horizons of less than two weeks. However, forecasts from models using cases aggregated by test date showed lower accuracy at longer horizons and at key moments in the pandemic, such as the peak of the Omicron wave in January 2022. Overall, these results highlight the challenge of finding a modeling approach that can generate accurate forecasts of outbreak trends both during periods of relative stability and during periods that show rapid growth or decay of transmission rates. While COVID-19 case counts seem to be a natural choice to help predict COVID-19 hospitalizations, in practice any benefits we observed were small and inconsistent.

How to avoid a local epidemic becoming a global pandemic

Here, we combine international air travel passenger data with a standard epidemiological model of the initial 3 mo of the COVID-19 pandemic (January through March 2020; toward the end of which the entire world locked down). Using the information available during this initial phase of the pandemic, our model accurately describes the main features of the actual global development of the pandemic demonstrated by the high degree of coherence between the model and global data. The validated model allows for an exploration of alternative policy efficacies (reducing air travel and/or introducing different degrees of compulsory immigration quarantine upon arrival to a country) in delaying the global spread of SARS-CoV-2 and thus is suggestive of similar efficacy in anticipating the spread of future global disease outbreaks. We show that a lesson from the recent pandemic is that reducing air travel globally is more effective in reducing the global spread than adopting immigration quarantine. Reducing air travel out of a source country has the most important effect regarding the spreading of the disease to the rest of the world. Based upon our results, we propose a digital twin as a further developed tool to inform future pandemic decision-making to inform measures intended to control the spread of disease agents of potential future pandemics. We discuss the design criteria for such a digital twin model as well as the feasibility of obtaining access to the necessary online data on international air travel.

An implementation framework to improve the transparency and reproducibility of computational models of infectious diseases

Computational models of infectious diseases have become valuable tools for research and the public health response against epidemic threats. The reproducibility of computational models has been limited, undermining the scientific process and possibly trust in modeling results and related response strategies, such as vaccination. We translated published reproducibility guidelines from a wide range of scientific disciplines into an implementation framework for improving reproducibility of infectious disease computational models. The framework comprises 22 elements that should be described, grouped into 6 categories: computational environment, analytical software, model description, model implementation, data, and experimental protocol. The framework can be used by scientific communities to develop actionable tools for sharing computational models in a reproducible way.

Incorporating variant frequencies data into short-term forecasting for COVID-19 cases and deaths in the USA: a deep learning approach

BACKGROUND

Since the US reported its first COVID-19 case on January 21, 2020, the science community has been applying various techniques to forecast incident cases and deaths. To date, providing an accurate and robust forecast at a high spatial resolution has proved challenging, even in the short term.

METHOD

Here we present a novel multi-stage deep learning model to forecast the number of COVID-19 cases and deaths for each US state at a weekly level for a forecast horizon of 1-4 weeks. The model is heavily data driven, and relies on epidemiological, mobility, survey, climate, demographic, and SARS-CoV-2 variant frequencies data. We implement a rigorous and robust evaluation of our model-specifically we report on weekly performance over a one-year period based on multiple error metrics, and explicitly assess how our model performance varies over space, chronological time, and different outbreak phases.

FINDINGS

The proposed model is shown to consistently outperform the CDC ensemble model for all evaluation metrics in multiple spatiotemporal settings, especially for the longer-term (3 and 4 weeks ahead) forecast horizon. Our case study also highlights the potential value of variant frequencies data for use in short-term forecasting to identify forthcoming surges driven by new variants.

INTERPRETATION

Based on our findings, the proposed forecasting framework improves upon the available state-of-the-art forecasting tools currently used to support public health decision making with respect to COVID-19 risk.

FUNDING

This work was funded the NSF Rapid Response Research (RAPID) grant Award ID 2108526 and the CDC Contract #75D30120C09570.

Crowdsourced Perceptions of Human Behavior to Improve Computational Forecasts of US National Incident Cases of COVID-19: Survey Study

BACKGROUND

Past research has shown that various signals associated with human behavior (eg, social media engagement) can benefit computational forecasts of COVID-19. One behavior that has been shown to reduce the spread of infectious agents is compliance with nonpharmaceutical interventions (NPIs). However, the extent to which the public adheres to NPIs is difficult to measure and consequently difficult to incorporate into computational forecasts of infectious diseases. Soliciting judgments from many individuals (ie, crowdsourcing) can lead to surprisingly accurate estimates of both current and future targets of interest. Therefore, asking a crowd to estimate community-level compliance with NPIs may prove to be an accurate and predictive signal of an infectious disease such as COVID-19.

OBJECTIVE

We aimed to show that crowdsourced perceptions of compliance with NPIs can be a fast and reliable signal that can predict the spread of an infectious agent. We showed this by measuring the correlation between crowdsourced perceptions of NPIs and US incident cases of COVID-19 1-4 weeks ahead, and evaluating whether incorporating crowdsourced perceptions improves the predictive performance of a computational forecast of incident cases.

METHODS

For 36 weeks from September 2020 to April 2021, we asked 2 crowds 21 questions about their perceptions of community adherence to NPIs and public health guidelines, and collected 10,120 responses. Self-reported state residency was compared to estimates from the US census to determine the representativeness of the crowds. Crowdsourced NPI signals were mapped to 21 mean perceived adherence (MEPA) signals and analyzed descriptively to investigate features, such as how MEPA signals changed over time and whether MEPA time series could be clustered into groups based on response patterns. We investigated whether MEPA signals were associated with incident cases of COVID-19 1-4 weeks ahead by (1) estimating correlations between MEPA and incident cases, and (2) including MEPA into computational forecasts.

RESULTS

The crowds were mostly geographically representative of the US population with slight overrepresentation in the Northeast. MEPA signals tended to converge toward moderate levels of compliance throughout the survey period, and an unsupervised analysis revealed signals clustered into 4 groups roughly based on the type of question being asked. Several MEPA signals linearly correlated with incident cases of COVID-19 1-4 weeks ahead at the US national level. Including questions related to social distancing, testing, and limiting large gatherings increased out-of-sample predictive performance for probabilistic forecasts of incident cases of COVID-19 1-3 weeks ahead when compared to a model that was trained on only past incident cases.

CONCLUSIONS

Crowdsourced perceptions of nonpharmaceutical adherence may be an important signal to improve forecasts of the trajectory of an infectious agent and increase public health situational awareness.

A meta-epidemiological assessment of transparency indicators of infectious disease models

Mathematical models have become very influential, especially during the COVID-19 pandemic. Data and code sharing are indispensable for reproducing them, protocol registration may be useful sometimes, and declarations of conflicts of interest (COIs) and of funding are quintessential for transparency. Here, we evaluated these features in publications of infectious disease-related models and assessed whether there were differences before and during the COVID-19 pandemic and for COVID-19 models versus models for other diseases. We analysed all PubMed Central open access publications of infectious disease models published in 2019 and 2021 using previously validated text mining algorithms of transparency indicators. We evaluated 1338 articles: 216 from 2019 and 1122 from 2021 (of which 818 were on COVID-19); almost a six-fold increase in publications within the field. 511 (39.2%) were compartmental models, 337 (25.2%) were time series, 279 (20.9%) were spatiotemporal, 186 (13.9%) were agent-based and 25 (1.9%) contained multiple model types. 288 (21.5%) articles shared code, 332 (24.8%) shared data, 6 (0.4%) were registered, and 1197 (89.5%) and 1109 (82.9%) contained COI and funding statements, respectively. There was no major changes in transparency indicators between 2019 and 2021. COVID-19 articles were less likely to have funding statements and more likely to share code. Further validation was performed by manual assessment of 10% of the articles identified by text mining as fulfilling transparency indicators and of 10% of the articles lacking them. Correcting estimates for validation performance, 26.0% of papers shared code and 41.1% shared data. On manual assessment, 5/6 articles identified as registered had indeed been registered. Of articles containing COI and funding statements, 95.8% disclosed no conflict and 11.7% reported no funding. Transparency in infectious disease modelling is relatively low, especially for data and code sharing. This is concerning, considering the nature of this research and the heightened influence it has acquired.

An ensemble n-sub-epidemic modeling framework for short-term forecasting epidemic trajectories: Application to the COVID-19 pandemic in the USA

We analyze an ensemble of n-sub-epidemic modeling for forecasting the trajectory of epidemics and pandemics. These ensemble modeling approaches, and models that integrate sub-epidemics to capture complex temporal dynamics, have demonstrated powerful forecasting capability. This modeling framework can characterize complex epidemic patterns, including plateaus, epidemic resurgences, and epidemic waves characterized by multiple peaks of different sizes. We systematically assess their calibration and short-term forecasting performance in short-term forecasts for the COVID-19 pandemic in the USA from late April 2020 to late February 2022. We compare their performance with two commonly used statistical ARIMA models. The best fit sub-epidemic model and three ensemble models constructed using the top-ranking sub-epidemic models consistently outperformed the ARIMA models in terms of the weighted interval score (WIS) and the coverage of the 95% prediction interval across the 10-, 20-, and 30-day short-term forecasts. In our 30-day forecasts, the average WIS ranged from 377.6 to 421.3 for the sub-epidemic models, whereas it ranged from 439.29 to 767.05 for the ARIMA models. Across 98 short-term forecasts, the ensemble model incorporating the top four ranking sub-epidemic models (Ensemble(4)) outperformed the (log) ARIMA model 66.3% of the time, and the ARIMA model, 69.4% of the time in 30-day ahead forecasts in terms of the WIS. Ensemble(4) consistently yielded the best performance in terms of the metrics that account for the uncertainty of the predictions. This framework can be readily applied to investigate the spread of epidemics and pandemics beyond COVID-19, as well as other dynamic growth processes found in nature and society that would benefit from short-term predictions.

An evaluation of prospective COVID-19 modelling studies in the USA: from data to science translation

Infectious disease modelling can serve as a powerful tool for situational awareness and decision support for policy makers. However, COVID-19 modelling efforts faced many challenges, from poor data quality to changing policy and human behaviour. To extract practical insight from the large body of COVID-19 modelling literature available, we provide a narrative review with a systematic approach that quantitatively assessed prospective, data-driven modelling studies of COVID-19 in the USA. We analysed 136 papers, and focused on the aspects of models that are essential for decision makers. We have documented the forecasting window, methodology, prediction target, datasets used, and geographical resolution for each study. We also found that a large fraction of papers did not evaluate performance (25%), express uncertainty (50%), or state limitations (36%). To remedy some of these identified gaps, we recommend the adoption of the EPIFORGE 2020 model reporting guidelines and creating an information-sharing system that is suitable for fast-paced infectious disease outbreak science.

Aggregating Human Judgment Probabilistic Predictions of Coronavirus Disease 2019 Transmission, Burden, and Preventive Measures

Aggregated human judgment forecasts for coronavirus disease 2019 (COVID-19) targets of public health importance are accurate, often outperforming computational models. Our work shows that aggregated human judgment forecasts for infectious agents are timely, accurate, and adaptable, and can be used as a tool to aid public health decision making during outbreaks.

Comparing trained and untrained probabilistic ensemble forecasts of COVID-19 cases and deaths in the United States

The U.S. COVID-19 Forecast Hub aggregates forecasts of the short-term burden of COVID-19 in the United States from many contributing teams. We study methods for building an ensemble that combines forecasts from these teams. These experiments have informed the ensemble methods used by the Hub. To be most useful to policy makers, ensemble forecasts must have stable performance in the presence of two key characteristics of the component forecasts: (1) occasional misalignment with the reported data, and (2) instability in the relative performance of component forecasters over time. Our results indicate that in the presence of these challenges, an untrained and robust approach to ensembling using an equally weighted median of all component forecasts is a good choice to support public health decision makers. In settings where some contributing forecasters have a stable record of good performance, trained ensembles that give those forecasters higher weight can also be helpful.

An ensemble n-sub-epidemic modeling framework for short-term forecasting epidemic trajectories: Application to the COVID-19 pandemic in the USA.. [PMID: 35794886 PMCID: PMC9258290 DOI: 10.1101/2022.06.19.22276608] [Citation(s) in RCA: 0] [Impact Index Per Article: 0]

We analyze an ensemble of n-sub-epidemic modeling for forecasting the trajectory of epidemics and pandemics. These ensemble modeling approaches, and models that integrate sub-epidemics to capture complex temporal dynamics, have demonstrated powerful forecasting capability. This modeling framework can characterize complex epidemic patterns, including plateaus, epidemic resurgences, and epidemic waves characterized by multiple peaks of different sizes. We systematically assess their calibration and short-term forecasting performance in short-term forecasts for the COVID-19 pandemic in the USA from late April 2020 to late February 2022. We compare their performance with two commonly used statistical ARIMA models. The best fit sub-epidemic model and three ensemble models constructed using the top-ranking sub-epidemic models consistently outperformed the ARIMA models in terms of the weighted interval score (WIS) and the coverage of the 95% prediction interval across the 10-, 20-, and 30-day short-term forecasts. In the 30-day forecasts, the average WIS ranged from 377.6 to 421.3 for the sub-epidemic models, whereas it ranged from 439.29 to 767.05 for the ARIMA models. Across 98 short-term forecasts, the ensemble model incorporating the top four ranking sub-epidemic models (Ensemble(4)) outperformed the (log) ARIMA model 66.3% of the time, and the ARIMA model 69.4% of the time in 30-day ahead forecasts in terms of the WIS. Ensemble(4) consistently yielded the best performance in terms of the metrics that account for the uncertainty of the predictions. This framework could be readily applied to investigate the spread of epidemics and pandemics beyond COVID-19, as well as other dynamic growth processes found in nature and society that would benefit from short-term predictions.

The COVID-19 pandemic has highlighted the urgent need to develop reliable tools to forecast the trajectory of epidemics and pandemics in near real-time. We describe and apply an ensemble n-sub-epidemic modeling framework for forecasting the trajectory of epidemics and pandemics. We systematically assess its calibration and short-term forecasting performance in weekly 10–30 days ahead forecasts for the COVID-19 pandemic in the USA from late April 2020 to late February 2022 and compare its performance with two different statistical ARIMA models. This framework demonstrated reliable forecasting performance and substantially outcompeted the ARIMA models. The forecasting performance was consistently best for the ensemble sub-epidemic models incorporating a higher number of top-ranking sub-epidemic models. The ensemble model incorporating the top four ranking sub-epidemic models consistently yielded the best performance, particularly in terms of the coverage rate of the 95% prediction interval and the weighted interval score. This framework can be applied to forecast other growth processes found in nature and society including the spread of information through social media.

Analyzing historical and future acute neurosurgical demand using an AI-enabled predictive dashboard

Characterizing acute service demand is critical for neurosurgery and other emergency-dominant specialties in order to dynamically distribute resources and ensure timely access to treatment. This is especially important in the post-Covid 19 pandemic period, when healthcare centers are grappling with a record backlog of pending surgical procedures and rising acute referral numbers. Healthcare dashboards are well-placed to analyze this data, making key information about service and clinical outcomes available to staff in an easy-to-understand format. However, they typically provide insights based on inference rather than prediction, limiting their operational utility. We retrospectively analyzed and prospectively forecasted acute neurosurgical referrals, based on 10,033 referrals made to a large volume tertiary neurosciences center in London, U.K., from the start of the Covid-19 pandemic lockdown period until October 2021 through the use of a novel AI-enabled predictive dashboard. As anticipated, weekly referral volumes significantly increased during this period, largely owing to an increase in spinal referrals (p < 0.05). Applying validated time-series forecasting methods, we found that referrals were projected to increase beyond this time-point, with Prophet demonstrating the best test and computational performance. Using a mixed-methods approach, we determined that a dashboard approach was usable, feasible, and acceptable among key stakeholders.

Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States

Short-term probabilistic forecasts of the trajectory of the COVID-19 pandemic in the United States have served as a visible and important communication channel between the scientific modeling community and both the general public and decision-makers. Forecasting models provide specific, quantitative, and evaluable predictions that inform short-term decisions such as healthcare staffing needs, school closures, and allocation of medical supplies. Starting in April 2020, the US COVID-19 Forecast Hub (https://covid19forecasthub.org/) collected, disseminated, and synthesized tens of millions of specific predictions from more than 90 different academic, industry, and independent research groups. A multimodel ensemble forecast that combined predictions from dozens of groups every week provided the most consistently accurate probabilistic forecasts of incident deaths due to COVID-19 at the state and national level from April 2020 through October 2021. The performance of 27 individual models that submitted complete forecasts of COVID-19 deaths consistently throughout this year showed high variability in forecast skill across time, geospatial units, and forecast horizons. Two-thirds of the models evaluated showed better accuracy than a naïve baseline model. Forecast accuracy degraded as models made predictions further into the future, with probabilistic error at a 20-wk horizon three to five times larger than when predicting at a 1-wk horizon. This project underscores the role that collaboration and active coordination between governmental public-health agencies, academic modeling teams, and industry partners can play in developing modern modeling capabilities to support local, state, and federal response to outbreaks.

Aggregating human judgment probabilistic predictions of COVID-19 transmission, burden, and preventative measures

Aggregated human judgment forecasts for COVID-19 targets of public health importance are accurate, often outperforming computational models. Our work shows aggregated human judgment forecasts for infectious agents are timely, accurate, and adaptable, and can be used as tool to aid public health decision making during outbreaks.