1
|
McAndrew T, Gibson GC, Braun D, Srivastava A, Brown K. Chimeric Forecasting: An experiment to leverage human judgment to improve forecasts of infectious disease using simulated surveillance data. Epidemics 2024; 47:100756. [PMID: 38452456 DOI: 10.1016/j.epidem.2024.100756] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/18/2023] [Revised: 12/06/2023] [Accepted: 02/26/2024] [Indexed: 03/09/2024] Open
Abstract
Forecasts of infectious agents provide public health officials advanced warning about the intensity and timing of the spread of disease. Past work has found that accuracy and calibration of forecasts is weakest when attempting to predict an epidemic peak. Forecasts from a mechanistic model would be improved if there existed accurate information about the timing and intensity of an epidemic. We presented 3000 humans with simulated surveillance data about the number of incident hospitalizations from a current and two past seasons, and asked that they predict the peak time and intensity of the underlying epidemic. We found that in comparison to two control models, a model including human judgment produced more accurate forecasts of peak time and intensity of hospitalizations during an epidemic. Chimeric models have the potential to improve our ability to predict targets of public health interest which may in turn reduce infectious disease burden.
Collapse
Affiliation(s)
- Thomas McAndrew
- Department of Community and Population Health, College of Health, Lehigh University, Bethlehem PA, United States of America.
| | - Graham C Gibson
- Statistical Sciences, Los Alamos National Laboratory, Los Alamos, NM, United States of America
| | - David Braun
- Department of Psychology College of Arts and Science, Lehigh University, Bethlehem PA, United States of America
| | - Abhishek Srivastava
- P.C. Rossin College of Engineering & Applied Science, Lehigh University, Bethlehem PA, United States of America
| | - Kate Brown
- Department of Community and Population Health, College of Health, Lehigh University, Bethlehem PA, United States of America
| |
Collapse
|
2
|
McAndrew T, Codi A, Cambeiro J, Besiroglu T, Braun D, Chen E, De Cèsaris LEU, Luk D. Chimeric forecasting: combining probabilistic predictions from computational models and human judgment. BMC Infect Dis 2022; 22:833. [PMID: 36357829 PMCID: PMC9648897 DOI: 10.1186/s12879-022-07794-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/30/2022] [Accepted: 10/12/2022] [Indexed: 11/12/2022] Open
Abstract
Forecasts of the trajectory of an infectious agent can help guide public health decision making. A traditional approach to forecasting fits a computational model to structured data and generates a predictive distribution. However, human judgment has access to the same data as computational models plus experience, intuition, and subjective data. We propose a chimeric ensemble-a combination of computational and human judgment forecasts-as a novel approach to predicting the trajectory of an infectious agent. Each month from January, 2021 to June, 2021 we asked two generalist crowds, using the same criteria as the COVID-19 Forecast Hub, to submit a predictive distribution over incident cases and deaths at the US national level either two or three weeks into the future and combined these human judgment forecasts with forecasts from computational models submitted to the COVID-19 Forecasthub into a chimeric ensemble. We find a chimeric ensemble compared to an ensemble including only computational models improves predictions of incident cases and shows similar performance for predictions of incident deaths. A chimeric ensemble is a flexible, supportive public health tool and shows promising results for predictions of the spread of an infectious agent.
Collapse
Affiliation(s)
| | - Allison Codi
- College of Health, Lehigh University, Bethlehem, PA, USA
| | - Juan Cambeiro
- Metaculus, Santa Cruz, CA, USA
- Department of Epidemiology, Mailman School of Public Health, Columbia University, New York, USA
| | - Tamay Besiroglu
- Metaculus, Santa Cruz, CA, USA
- Massachusetts Institute of Technology, Cambridge, MA, USA
| | - David Braun
- Department of Psychology, Lehigh University, Bethlehem, PA, USA
| | - Eva Chen
- Good Judgment Inc., New York, NY, USA
| | | | - Damon Luk
- College of Health, Lehigh University, Bethlehem, PA, USA
| |
Collapse
|
3
|
McAndrew T, Majumder MS, Lover AA, Venkatramanan S, Bocchini P, Besiroglu T, Codi A, Braun D, Dempsey G, Abbott S, Chevalier S, Bosse NI, Cambeiro J. Early human judgment forecasts of human monkeypox, May 2022. Lancet Digit Health 2022; 4:e569-e571. [PMID: 35811294 PMCID: PMC9534486 DOI: 10.1016/s2589-7500(22)00127-3] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/02/2022] [Revised: 06/20/2022] [Accepted: 06/27/2022] [Indexed: 01/25/2023]
Affiliation(s)
- Thomas McAndrew
- Department of Community and Population Health, College of Health, Lehigh University, Bethlehem, PA 18015, USA.
| | - Maimuna S Majumder
- Boston Children's Hospital, Boston, MA, USA; Harvard Medical School, Boston, MA, USA
| | - Andrew A Lover
- Department of Biostatistics and Epidemiology, School of Public Health and Health Sciences, University of Massachusetts Amherst, Amherst, MA, USA
| | - Srini Venkatramanan
- Biocomplexity Institute and Initiative, University of Virginia, Charlottesville, VA, USA
| | - Paolo Bocchini
- Department of Civil and Environmental Engineering, P C Rossin College of Engineering and Applied Science, Lehigh University, Bethlehem, PA 18015, USA
| | - Tamay Besiroglu
- MIT Computer Science and Artificial Intelligence Laboratory, Cambridge, MA, USA; Metaculus, Santa Cruz, CA, USA
| | - Allison Codi
- Department of Community and Population Health, College of Health, Lehigh University, Bethlehem, PA 18015, USA
| | - David Braun
- Department of Psychology, College of Arts and Sciences, Lehigh University, Bethlehem, PA 18015, USA
| | | | - Sam Abbott
- Centre for Mathematical Modelling of Infectious Diseases, London School of Hygiene & Tropical Medicine, London, UK
| | | | - Nikos I Bosse
- Metaculus, Santa Cruz, CA, USA; Centre for Mathematical Modelling of Infectious Diseases, London School of Hygiene & Tropical Medicine, London, UK
| | - Juan Cambeiro
- Metaculus, Santa Cruz, CA, USA; Department of Epidemiology, Mailman School of Public Health, Columbia University, New York, NY, USA
| |
Collapse
|
4
|
Codi A, Luk D, Braun D, Cambeiro J, Besiroglu T, Chen E, de Cesaris LEU, Bocchini P, McAndrew T. Aggregating Human Judgment Probabilistic Predictions of Coronavirus Disease 2019 Transmission, Burden, and Preventive Measures. Open Forum Infect Dis 2022; 9:ofac354. [PMID: 35937647 PMCID: PMC9348614 DOI: 10.1093/ofid/ofac354] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/23/2022] [Indexed: 12/04/2022] Open
Abstract
Aggregated human judgment forecasts for coronavirus disease 2019 (COVID-19) targets of public health importance are accurate, often outperforming computational models. Our work shows that aggregated human judgment forecasts for infectious agents are timely, accurate, and adaptable, and can be used as a tool to aid public health decision making during outbreaks.
Collapse
Affiliation(s)
- Allison Codi
- College of Health, Lehigh University, Bethlehem, Pennsylvania, USA
| | - Damon Luk
- College of Health, Lehigh University, Bethlehem, Pennsylvania, USA
| | - David Braun
- Department of Psychology, College of Arts and Sciences, Lehigh University, Bethlehem, Pennsylvania, USA
| | - Juan Cambeiro
- Department of Epidemiology, Mailman School of Public Health, Columbia University, New York, USA
- Metaculus, Santa Cruz, California, USA
| | - Tamay Besiroglu
- Department of Epidemiology, Mailman School of Public Health, Columbia University, New York, USA
- Computer Science and AI Lab, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA
| | - Eva Chen
- Good Judgment Inc, New York, New York, USA
| | | | - Paolo Bocchini
- Department of Civil and Environmental Engineering, P.C. Rossin College of Engineering and Applied Science, Lehigh University, Bethlehem, Pennsylvania, USA
| | - Thomas McAndrew
- College of Health, Lehigh University, Bethlehem, Pennsylvania, USA
| |
Collapse
|