1
|
Lopez VK, Cramer EY, Pagano R, Drake JM, O’Dea EB, Adee M, Ayer T, Chhatwal J, Dalgic OO, Ladd MA, Linas BP, Mueller PP, Xiao J, Bracher J, Castro Rivadeneira AJ, Gerding A, Gneiting T, Huang Y, Jayawardena D, Kanji AH, Le K, Mühlemann A, Niemi J, Ray EL, Stark A, Wang Y, Wattanachit N, Zorn MW, Pei S, Shaman J, Yamana TK, Tarasewicz SR, Wilson DJ, Baccam S, Gurung H, Stage S, Suchoski B, Gao L, Gu Z, Kim M, Li X, Wang G, Wang L, Wang Y, Yu S, Gardner L, Jindal S, Marshall M, Nixon K, Dent J, Hill AL, Kaminsky J, Lee EC, Lemaitre JC, Lessler J, Smith CP, Truelove S, Kinsey M, Mullany LC, Rainwater-Lovett K, Shin L, Tallaksen K, Wilson S, Karlen D, Castro L, Fairchild G, Michaud I, Osthus D, Bian J, Cao W, Gao Z, Lavista Ferres J, Li C, Liu TY, Xie X, Zhang S, Zheng S, Chinazzi M, Davis JT, Mu K, Pastore y Piontti A, Vespignani A, Xiong X, Walraven R, Chen J, Gu Q, Wang L, Xu P, Zhang W, Zou D, Gibson GC, Sheldon D, Srivastava A, Adiga A, Hurt B, Kaur G, Lewis B, Marathe M, Peddireddy AS, Porebski P, Venkatramanan S, Wang L, Prasad PV, Walker JW, Webber AE, Slayton RB, Biggerstaff M, Reich NG, Johansson MA. Challenges of COVID-19 Case Forecasting in the US, 2020-2021. PLoS Comput Biol 2024; 20:e1011200. [PMID: 38709852 PMCID: PMC11098513 DOI: 10.1371/journal.pcbi.1011200] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2023] [Revised: 05/16/2024] [Accepted: 04/01/2024] [Indexed: 05/08/2024] Open
Abstract
During the COVID-19 pandemic, forecasting COVID-19 trends to support planning and response was a priority for scientists and decision makers alike. In the United States, COVID-19 forecasting was coordinated by a large group of universities, companies, and government entities led by the Centers for Disease Control and Prevention and the US COVID-19 Forecast Hub (https://covid19forecasthub.org). We evaluated approximately 9.7 million forecasts of weekly state-level COVID-19 cases for predictions 1-4 weeks into the future submitted by 24 teams from August 2020 to December 2021. We assessed coverage of central prediction intervals and weighted interval scores (WIS), adjusting for missing forecasts relative to a baseline forecast, and used a Gaussian generalized estimating equation (GEE) model to evaluate differences in skill across epidemic phases that were defined by the effective reproduction number. Overall, we found high variation in skill across individual models, with ensemble-based forecasts outperforming other approaches. Forecast skill relative to the baseline was generally higher for larger jurisdictions (e.g., states compared to counties). Over time, forecasts generally performed worst in periods of rapid changes in reported cases (either in increasing or decreasing epidemic phases) with 95% prediction interval coverage dropping below 50% during the growth phases of the winter 2020, Delta, and Omicron waves. Ideally, case forecasts could serve as a leading indicator of changes in transmission dynamics. However, while most COVID-19 case forecasts outperformed a naïve baseline model, even the most accurate case forecasts were unreliable in key phases. Further research could improve forecasts of leading indicators, like COVID-19 cases, by leveraging additional real-time data, addressing performance across phases, improving the characterization of forecast confidence, and ensuring that forecasts were coherent across spatial scales. In the meantime, it is critical for forecast users to appreciate current limitations and use a broad set of indicators to inform pandemic-related decision making.
Collapse
Affiliation(s)
- Velma K. Lopez
- COVID-19 Response, Centers for Disease Control and Prevention, Atlanta, Georgia, United States of America
| | - Estee Y. Cramer
- University of Massachusetts, Amherst, Amherst, Massachusetts, United States of America
| | - Robert Pagano
- Unaffiliated, Tucson, Arizona, United States of America
| | - John M. Drake
- University of Georgia, Athens, Georgia, United States of America
| | - Eamon B. O’Dea
- University of Georgia, Athens, Georgia, United States of America
| | - Madeline Adee
- Massachusetts General Hospital, Boston, Massachusetts, United States of America
| | - Turgay Ayer
- Georgia Institute of Technology, Atlanta, Georgia, United States of America
| | - Jagpreet Chhatwal
- Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts, United States of America
| | - Ozden O. Dalgic
- Value Analytics Labs, Boston, Massachusetts, United States of America
| | - Mary A. Ladd
- Massachusetts General Hospital, Boston, Massachusetts, United States of America
| | - Benjamin P. Linas
- Boston University School of Medicine, Boston, Massachusetts, United States of America
| | - Peter P. Mueller
- Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts, United States of America
| | - Jade Xiao
- Georgia Institute of Technology, Atlanta, Georgia, United States of America
| | - Johannes Bracher
- Chair of Econometrics and Statistics, Karlsruhe Institute of Technology, Karlsruhe, Germany
| | | | - Aaron Gerding
- University of Massachusetts, Amherst, Amherst, Massachusetts, United States of America
| | - Tilmann Gneiting
- Heidelberg Institute for Theoretical Studies, Heidelberg, Germany
| | - Yuxin Huang
- University of Massachusetts, Amherst, Amherst, Massachusetts, United States of America
| | - Dasuni Jayawardena
- University of Massachusetts, Amherst, Amherst, Massachusetts, United States of America
| | - Abdul H. Kanji
- University of Massachusetts, Amherst, Amherst, Massachusetts, United States of America
| | - Khoa Le
- University of Massachusetts, Amherst, Amherst, Massachusetts, United States of America
| | - Anja Mühlemann
- Institute of Mathematical Statistics and Actuarial Science, University of Bern, Bern, Switzerland
| | - Jarad Niemi
- Iowa State University, Ames, Iowa, United States of America
| | - Evan L. Ray
- University of Massachusetts, Amherst, Amherst, Massachusetts, United States of America
| | - Ariane Stark
- University of Massachusetts, Amherst, Amherst, Massachusetts, United States of America
| | - Yijin Wang
- University of Massachusetts, Amherst, Amherst, Massachusetts, United States of America
| | - Nutcha Wattanachit
- University of Massachusetts, Amherst, Amherst, Massachusetts, United States of America
| | - Martha W. Zorn
- University of Massachusetts, Amherst, Amherst, Massachusetts, United States of America
| | - Sen Pei
- Mailman School of Public Health, Columbia University, New York, New York, United States of America
| | - Jeffrey Shaman
- Mailman School of Public Health, Columbia University, New York, New York, United States of America
| | - Teresa K. Yamana
- Mailman School of Public Health, Columbia University, New York, New York, United States of America
| | - Samuel R. Tarasewicz
- Federal Reserve Bank of San Francisco, San Francisco, California, United States of America
| | - Daniel J. Wilson
- Federal Reserve Bank of San Francisco, San Francisco, California, United States of America
| | - Sid Baccam
- IEM, Bel Air, Maryland, United States of America
| | - Heidi Gurung
- IEM, Bel Air, Maryland, United States of America
| | - Steve Stage
- IEM, Baton Rouge, Louisiana, United States of America
| | | | - Lei Gao
- George Mason University, Fairfax, Virginia, United States of America
| | - Zhiling Gu
- Iowa State University, Ames, Iowa, United States of America
| | - Myungjin Kim
- Kyungpook National University, Bukgu, Daegu, Republic of Korea
| | - Xinyi Li
- Clemson University, Clemson, South Carolina, United States of America
| | - Guannan Wang
- College of William & Mary, Williamsburg, Virginia, United States of America
| | - Lily Wang
- George Mason University, Fairfax, Virginia, United States of America
| | - Yueying Wang
- Amazon, Seattle, Washington, United States of America
| | - Shan Yu
- University of Virginia, Charlottesville, Virginia, United States of America
| | - Lauren Gardner
- Johns Hopkins University, Baltimore, Maryland, United States of America
| | - Sonia Jindal
- Johns Hopkins University, Baltimore, Maryland, United States of America
| | | | - Kristen Nixon
- Johns Hopkins University, Baltimore, Maryland, United States of America
| | - Juan Dent
- Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, United States of America
| | - Alison L. Hill
- Johns Hopkins University, Baltimore, Maryland, United States of America
| | - Joshua Kaminsky
- Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, United States of America
| | - Elizabeth C. Lee
- Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, United States of America
| | | | - Justin Lessler
- Gillings School of Global Public Health, University of North Carolina at Chapel Hill, Chapel Hill, North Carolina, United States of America
| | - Claire P. Smith
- Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, United States of America
| | - Shaun Truelove
- Johns Hopkins Bloomberg School of Public Health, Baltimore, Maryland, United States of America
| | - Matt Kinsey
- Johns Hopkins University Applied Physics Lab, Baltimore, Maryland, United States of America
| | - Luke C. Mullany
- Johns Hopkins University Applied Physics Lab, Baltimore, Maryland, United States of America
| | | | - Lauren Shin
- Johns Hopkins University Applied Physics Lab, Baltimore, Maryland, United States of America
| | - Katharine Tallaksen
- Johns Hopkins University Applied Physics Lab, Baltimore, Maryland, United States of America
| | - Shelby Wilson
- Johns Hopkins University Applied Physics Lab, Baltimore, Maryland, United States of America
| | - Dean Karlen
- University of Victoria and TRIUMF, Victoria, British Columbia, Canada
| | - Lauren Castro
- Los Alamos National Laboratory, Los Alamos, New Mexico, United States of America
| | - Geoffrey Fairchild
- Los Alamos National Laboratory, Los Alamos, New Mexico, United States of America
| | - Isaac Michaud
- Los Alamos National Laboratory, Los Alamos, New Mexico, United States of America
| | - Dave Osthus
- Los Alamos National Laboratory, Los Alamos, New Mexico, United States of America
| | - Jiang Bian
- Microsoft, Redmond, Washington, United States of America
| | - Wei Cao
- Microsoft, Redmond, Washington, United States of America
| | - Zhifeng Gao
- Microsoft, Redmond, Washington, United States of America
| | | | - Chaozhuo Li
- Microsoft, Redmond, Washington, United States of America
| | - Tie-Yan Liu
- Microsoft, Redmond, Washington, United States of America
| | - Xing Xie
- Microsoft, Redmond, Washington, United States of America
| | - Shun Zhang
- Microsoft, Redmond, Washington, United States of America
| | - Shun Zheng
- Microsoft, Redmond, Washington, United States of America
| | - Matteo Chinazzi
- Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, Massachusetts, United States of America
| | - Jessica T. Davis
- Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, Massachusetts, United States of America
| | - Kunpeng Mu
- Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, Massachusetts, United States of America
| | - Ana Pastore y Piontti
- Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, Massachusetts, United States of America
| | - Alessandro Vespignani
- Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, Massachusetts, United States of America
| | - Xinyue Xiong
- Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, Massachusetts, United States of America
| | | | - Jinghui Chen
- University of California, Los Angeles, Los Angeles, California, United States of America
| | - Quanquan Gu
- University of California, Los Angeles, Los Angeles, California, United States of America
| | - Lingxiao Wang
- University of California, Los Angeles, Los Angeles, California, United States of America
| | - Pan Xu
- University of California, Los Angeles, Los Angeles, California, United States of America
| | - Weitong Zhang
- University of California, Los Angeles, Los Angeles, California, United States of America
| | - Difan Zou
- University of California, Los Angeles, Los Angeles, California, United States of America
| | - Graham Casey Gibson
- Los Alamos National Laboratory, Los Alamos, New Mexico, United States of America
| | - Daniel Sheldon
- University of Massachusetts, Amherst, Amherst, Massachusetts, United States of America
| | - Ajitesh Srivastava
- University of Southern California, Los Angeles, California, United States of America
| | - Aniruddha Adiga
- University of Virginia, Charlottesville, Virginia, United States of America
| | - Benjamin Hurt
- University of Virginia, Charlottesville, Virginia, United States of America
| | - Gursharn Kaur
- University of Virginia, Charlottesville, Virginia, United States of America
| | - Bryan Lewis
- University of Virginia, Charlottesville, Virginia, United States of America
| | - Madhav Marathe
- University of Virginia, Charlottesville, Virginia, United States of America
| | | | | | | | - Lijing Wang
- New Jersey Institute of Technology, Newark, New Jersey, United States of America
| | - Pragati V. Prasad
- COVID-19 Response, Centers for Disease Control and Prevention, Atlanta, Georgia, United States of America
| | - Jo W. Walker
- COVID-19 Response, Centers for Disease Control and Prevention, Atlanta, Georgia, United States of America
| | - Alexander E. Webber
- COVID-19 Response, Centers for Disease Control and Prevention, Atlanta, Georgia, United States of America
| | - Rachel B. Slayton
- COVID-19 Response, Centers for Disease Control and Prevention, Atlanta, Georgia, United States of America
| | - Matthew Biggerstaff
- COVID-19 Response, Centers for Disease Control and Prevention, Atlanta, Georgia, United States of America
| | - Nicholas G. Reich
- University of Massachusetts, Amherst, Amherst, Massachusetts, United States of America
| | - Michael A. Johansson
- COVID-19 Response, Centers for Disease Control and Prevention, Atlanta, Georgia, United States of America
| |
Collapse
|
2
|
Mathis SM, Webber AE, León TM, Murray EL, Sun M, White LA, Brooks LC, Green A, Hu AJ, McDonald DJ, Rosenfeld R, Shemetov D, Tibshirani RJ, Kandula S, Pei S, Shaman J, Yaari R, Yamana TK, Agarwal P, Balusu S, Gururajan G, Kamarthi H, Prakash BA, Raman R, Rodríguez A, Zhao Z, Meiyappan A, Omar S, Baccam P, Gurung HL, Stage SA, Suchoski BT, Ajelli M, Kummer AG, Litvinova M, Ventura PC, Wadsworth S, Niemi J, Carcelen E, Hill AL, Jung SM, Lemaitre JC, Lessler J, Loo SL, McKee CD, Sato K, Smith C, Truelove S, McAndrew T, Ye W, Bosse N, Hlavacek WS, Lin YT, Mallela A, Chen Y, Lamm SM, Lee J, Posner RG, Perofsky AC, Viboud C, Clemente L, Lu F, Meyer AG, Santillana M, Chinazzi M, Davis JT, Mu K, Piontti APY, Vespignani A, Xiong X, Ben-Nun M, Riley P, Turtle J, Hulme-Lowe C, Jessa S, Nagraj VP, Turner SD, Williams D, Basu A, Drake JM, Fox SJ, Gibson GC, Suez E, Thommes EW, Cojocaru MG, Cramer EY, Gerding A, Stark A, Ray EL, Reich NG, Shandross L, Wattanachit N, Wang Y, Zorn MW, Al Aawar M, Srivastava A, Meyers LA, Adiga A, Hurt B, Kaur G, Lewis BL, Marathe M, Venkatramanan S, Butler P, Farabow A, Muralidhar N, Ramakrishnan N, Reed C, Biggerstaff M, Borchering RK. Evaluation of FluSight influenza forecasting in the 2021-22 and 2022-23 seasons with a new target laboratory-confirmed influenza hospitalizations. medRxiv 2023:2023.12.08.23299726. [PMID: 38168429 PMCID: PMC10760285 DOI: 10.1101/2023.12.08.23299726] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/05/2024]
Abstract
Accurate forecasts can enable more effective public health responses during seasonal influenza epidemics. Forecasting teams were asked to provide national and jurisdiction-specific probabilistic predictions of weekly confirmed influenza hospital admissions for one through four weeks ahead for the 2021-22 and 2022-23 influenza seasons. Across both seasons, 26 teams submitted forecasts, with the submitting teams varying between seasons. Forecast skill was evaluated using the Weighted Interval Score (WIS), relative WIS, and coverage. Six out of 23 models outperformed the baseline model across forecast weeks and locations in 2021-22 and 12 out of 18 models in 2022-23. Averaging across all forecast targets, the FluSight ensemble was the 2nd most accurate model measured by WIS in 2021-22 and the 5th most accurate in the 2022-23 season. Forecast skill and 95% coverage for the FluSight ensemble and most component models degraded over longer forecast horizons and during periods of rapid change. Current influenza forecasting efforts help inform situational awareness, but research is needed to address limitations, including decreased performance during periods of changing epidemic dynamics.
Collapse
Affiliation(s)
- Sarabeth M Mathis
- Centers for Disease Control and Prevention, Atlanta, Georgia, 30329, USA
| | - Alexander E Webber
- Centers for Disease Control and Prevention, Atlanta, Georgia, 30329, USA
| | - Tomás M León
- California Department of Public Health, Richmond, CA, 95899
| | - Erin L Murray
- California Department of Public Health, Richmond, CA, 95899
| | - Monica Sun
- California Department of Public Health, Richmond, CA, 95899
| | - Lauren A White
- California Department of Public Health, Richmond, CA, 95899
| | - Logan C Brooks
- Carnegie Mellon University, Pittsburgh, PA, 15213
- University of California, Berkeley, Berkeley, CA 94720
| | - Alden Green
- Carnegie Mellon University, Pittsburgh, PA, 15213
| | - Addison J Hu
- Carnegie Mellon University, Pittsburgh, PA, 15213
| | | | | | | | - Ryan J Tibshirani
- Carnegie Mellon University, Pittsburgh, PA, 15213
- University of California, Berkeley, Berkeley, CA 94720
| | | | - Sen Pei
- Columbia University, New York, NY, 10032
| | - Jeffrey Shaman
- Columbia University, New York, NY, 10032
- Columbia University School of Climate, New York, NY 10025
| | - Rami Yaari
- Columbia University, New York, NY, 10032
| | | | | | | | | | | | | | - Rishi Raman
- Georgia Institute of Technology, Atlanta, GA, 30318
| | | | - Zhiyuan Zhao
- Georgia Institute of Technology, Atlanta, GA, 30318
| | | | - Shalina Omar
- Guidehouse Advisory and Consulting Services, McClean VA, 22102
| | | | | | | | | | - Marco Ajelli
- Indiana University School of Public Health, Bloomington, IN, 47405
| | | | - Maria Litvinova
- Indiana University School of Public Health, Bloomington, IN, 47405
| | - Paulo C Ventura
- Indiana University School of Public Health, Bloomington, IN, 47405
| | | | | | | | | | - Sung-Mok Jung
- University of North Carolina at Chapel Hill, Chapel Hill, NC
| | | | - Justin Lessler
- University of North Carolina at Chapel Hill, Chapel Hill, NC
| | - Sara L Loo
- Johns Hopkins University, Baltimore, MD, 21205
| | | | - Koji Sato
- Johns Hopkins University, Baltimore, MD, 21205
| | | | | | | | | | - Nikos Bosse
- London School of Health and Tropical Medicine, London, UK, WC1E 7HT
| | | | - Yen Ting Lin
- Los Alamos National Laboratory, Los Alamos, NM, 87545
| | | | - Ye Chen
- Northern Arizona University, Flagstaff, AZ, 86011
| | | | - Jaechoul Lee
- Northern Arizona University, Flagstaff, AZ, 86011
| | | | - Amanda C Perofsky
- Fogarty International Center, National Institutes of Health, Bethesda, MD, 20892
| | - Cécile Viboud
- Fogarty International Center, National Institutes of Health, Bethesda, MD, 20892
| | | | - Fred Lu
- Northeastern University, Boston, MA, 02115
| | | | | | | | | | - Kunpeng Mu
- Northeastern University, Boston, MA, 02115
| | | | | | | | | | - Pete Riley
- Predictive Science Inc, San Diego, CA 92121
| | | | | | | | - V P Nagraj
- Signature Science, LLC, Charlottesville, VA, 22911
| | | | | | | | | | | | | | - Ehsan Suez
- University of Georgia, Athens, GA, 30609
| | - Edward W Thommes
- University of Guelph, Guelph, ON N1G 2W1, Canada
- Sanofi, Toronto, ON, M2R 3T4
| | | | | | - Aaron Gerding
- University of Massachusetts Amherst, Amherst, MA, 01003
| | - Ariane Stark
- University of Massachusetts Amherst, Amherst, MA, 01003
| | - Evan L Ray
- University of Massachusetts Amherst, Amherst, MA, 01003
| | | | - Li Shandross
- University of Massachusetts Amherst, Amherst, MA, 01003
| | | | - Yijin Wang
- University of Massachusetts Amherst, Amherst, MA, 01003
| | - Martha W Zorn
- University of Massachusetts Amherst, Amherst, MA, 01003
| | - Majd Al Aawar
- University of Southern California, Los Angeles, CA, 90089
| | | | | | | | | | | | | | | | | | | | | | | | | | - Carrie Reed
- Centers for Disease Control and Prevention, Atlanta, Georgia, 30329, USA
| | | | | |
Collapse
|