Applying Machine Learning Models with An Ensemble Approach for Accurate Real-Time Influenza Forecasting in Taiwan: Development and Validation Study

Multisite development and validation of machine learning models to predict severe outcomes and guide decision-making for emergency department patients with influenza

Objective

Millions of Americans are infected by influenza annually. A minority seek care in the emergency department (ED) and, of those, only a limited number experience severe disease or death. ED clinicians must distinguish those at risk for deterioration from those who can be safely discharged.

Methods

We developed random forest machine learning (ML) models to estimate needs for critical care within 24 h and inpatient care within 72 h in ED patients with influenza. Predictor data were limited to those recorded prior to ED disposition decision: demographics, ED complaint, medical problems, vital signs, supplemental oxygen use, and laboratory results. Our study population was comprised of adults diagnosed with influenza at one of five EDs in our university health system between January 1, 2017 and May 18, 2022; visits were divided into two cohorts to facilitate model development and validation. Prediction performance was assessed by the area under the receiver operating characteristic curve (AUC) and the Brier score.

Results

Among 8032 patients with laboratory-confirmed influenza, incidence of critical care needs was 6.3% and incidence of inpatient care needs was 19.6%. The most common reasons for ED visit were symptoms of respiratory tract infection, fever, and shortness of breath. Model AUCs were 0.89 (95% CI 0.86-0.93) for prediction of critical care and 0.90 (95% CI 0.88-0.93) for inpatient care needs; Brier scores were 0.026 and 0.042, respectively. Importantpredictors included shortness of breath, increasing respiratory rate, and a high number of comorbid diseases.

Conclusions

ML methods can be used to accurately predict clinical deterioration in ED patients with influenza and have potential to support ED disposition decision-making.

Predicting the incidence of infectious diarrhea with symptom surveillance data using a stacking-based ensembled model

BACKGROUND

Infectious diarrhea remains a major public health problem worldwide. This study used stacking ensemble to developed a predictive model for the incidence of infectious diarrhea, aiming to achieve better prediction performance.

METHODS

Based on the surveillance data of infectious diarrhea cases, relevant symptoms and meteorological factors of Guangzhou from 2016 to 2021, we developed four base prediction models using artificial neural networks (ANN), Long Short-Term Memory networks (LSTM), support vector regression (SVR) and extreme gradient boosting regression trees (XGBoost), which were then ensembled using stacking to obtain the final prediction model. All the models were evaluated with three metrics: mean absolute percentage error (MAPE), root mean square error (RMSE), and mean absolute error (MAE).

RESULTS

Base models that incorporated symptom surveillance data and weekly number of infectious diarrhea cases were able to achieve lower RMSEs, MAEs, and MAPEs than models that added meteorological data and weekly number of infectious diarrhea cases. The LSTM had the best prediction performance among the four base models, and its RMSE, MAE, and MAPE were: 84.85, 57.50 and 15.92%, respectively. The stacking ensembled model outperformed the four base models, whose RMSE, MAE, and MAPE were 75.82, 55.93, and 15.70%, respectively.

CONCLUSIONS

The incorporation of symptom surveillance data could improve the predictive accuracy of infectious diarrhea prediction models, and symptom surveillance data was more effective than meteorological data in enhancing model performance. Using stacking to combine multiple prediction models were able to alleviate the difficulty in selecting the optimal model, and could obtain a model with better performance than base models.

Multi-step influenza forecasting through singular value decomposition and kernel ridge regression with MARCOS-guided gradient-based optimization

This research delves into the significance of influenza outbreaks in public health, particularly the importance of accurate forecasts using weekly Influenza-like illness (ILI) rates. The present work develops a novel hybrid machine-learning model by combining singular value decomposition with kernel ridge regression (SKRR). In this context, a novel hybrid model known as H-SKRR is developed by combining two robust forecasting approaches, SKRR and ridge regression, which aims to improve multi-step-ahead predictions for weekly ILI rates in Southern and Northern China. The study begins with feature selection via XGBoost in the preprocessing phase, identifying optimal precursor information guided by importance factors. It decomposes the original signal using multivariate variational mode decomposition (MVMD) to address non-stationarity and complexity. H-SKRR is implemented by incorporating significant lagged-time components across sub-components. The aggregated forecasted values from these sub-components generate ILI values for two horizons (i.e., 4-and 7-weekly ahead). Employing the gradient-based optimization (GBO) algorithm fine-tunes model parameters. Furthermore, the deep random vector functional link (dRVFL), Ridge regression, and gated recurrent unit neural network (GRU) models were employed to validate the MVMD-H-SKRR-GBO paradigm's effectiveness. The outcomes, assessed using the MARCOS (Measurement of alternatives and ranking according to compromise solution) method as a multi-criteria decision-making method, highlight the superior accuracy of the MVMD-H-SKRR-GBO model in predicting ILI rates. The results clearly highlight the exceptional performance of the MVMD-H-SKRR-GBO model, with outstanding precision demonstrated by impressive R, RMSE, IA, and U95 % values of 0.946, 0.388, 0.970, and 1.075, respectively, at t + 7.

Forecasting influenza hospital admissions within English sub-regions using hierarchical generalised additive models

BACKGROUND

Seasonal influenza places a substantial burden annually on healthcare services. Policies during the COVID-19 pandemic limited the transmission of seasonal influenza, making the timing and magnitude of a potential resurgence difficult to ascertain and its impact important to forecast.

METHODS

We have developed a hierarchical generalised additive model (GAM) for the short-term forecasting of hospital admissions with a positive test for the influenza virus sub-regionally across England. The model incorporates a multi-level structure of spatio-temporal splines, weekly cycles in admissions, and spatial correlation. Using multiple performance metrics including interval score, coverage, bias, and median absolute error, the predictive performance is evaluated for the 2022-2023 seasonal wave. Performance is measured against autoregressive integrated moving average (ARIMA) and Prophet time series models.

RESULTS

Across the epidemic phases the hierarchical GAM shows improved performance, at all geographic scales relative to the ARIMA and Prophet models. Temporally, the hierarchical GAM has overall an improved performance at 7 and 14 day time horizons. The performance of the GAM is most sensitive to the flexibility of the smoothing function that measures the national epidemic trend.

CONCLUSIONS

This study introduces an approach to short-term forecasting of hospital admissions for the influenza virus using hierarchical, spatial, and temporal components. The methodology was designed for the real time forecasting of epidemics. This modelling framework was used across the 2022-2023 winter for healthcare operational planning by the UK Health Security Agency and the National Health Service in England.

Comparison of statistical approaches to predicting norovirus laboratory reports before and during COVID-19: insights to inform public health surveillance

Social distancing interrupted transmission patterns of contact-driven infectious agents such as norovirus during the Covid-19 pandemic. Since routine surveillance of norovirus was additionally disrupted during the pandemic, traditional naïve forecasts that rely only on past public health surveillance data may not reliably represent norovirus activity. This study investigates the use of statistical modelling to predict the number of norovirus laboratory reports in England 4-weeks ahead of time before and during Covid-19 pandemic thus providing insights to inform existing practices in norovirus surveillance in England. We compare the predictive performance from three forecasting approaches that assume different underlying structure of the norovirus data and utilized various external data sources including mobility, air temperature and relative internet searches (Time Series and Regularized Generalized Linear Model, and Quantile Regression Forest). The performance of each approach was evaluated using multiple metrics, including a relative prediction error against the traditional naive forecast of a five-season mean. Our data suggest that all three forecasting approaches improve predictive performance over the naïve forecasts, especially in the 2020/21 season (30-45% relative improvement) when the number of norovirus reports reduced. The improvement ranged from 7 to 22% before the pandemic. However, performance varied: regularized regression incorporating internet searches showed the best forecasting score pre-pandemic and the time series approach achieved the best results post pandemic onset without external data. Overall, our results demonstrate that there is a significant value for public health in considering the adoption of more sophisticated forecasting tools, moving beyond traditional naïve methods, and utilizing available software to enhance the precision and timeliness of norovirus surveillance in England.

Supervised Machine Learning Methods for Seasonal Influenza Diagnosis

Influenza has been a stationary disease in Mexico since 2009, and this causes a high cost for the national public health system, including its detection using RT-qPCR tests, treatments, and absenteeism in the workplace. Despite influenza's relevance, the main clinical features to detect the disease defined by international institutions like the World Health Organization (WHO) and the United States Centers for Disease Control and Prevention (CDC) do not follow the same pattern in all populations. The aim of this work is to find a machine learning method to facilitate decision making in the clinical differentiation between positive and negative influenza patients, based on their symptoms and demographic features. The research sample consisted of 15480 records, including clinical and demographic data of patients with a positive/negative RT-qPCR influenza tests, from 2010 to 2020 in the public healthcare institutions of Mexico City. The performance of the methods for classifying influenza cases were evaluated with indices like accuracy, specificity, sensitivity, precision, the f1-measure and the area under the curve (AUC). Results indicate that random forest and bagging classifiers were the best supervised methods; they showed promise in supporting clinical diagnosis, especially in places where performing molecular tests might be challenging or not feasible.

Determining meteorologically-favorable zones for seasonal influenza activity in Hong Kong

Investigations of simple and accurate meteorology classification systems for influenza epidemics, particularly in subtropical regions, are limited. To assist in preparing for potential upsurges in the demand on healthcare facilities during influenza seasons, our study aims to develop a set of meteorologically-favorable zones for epidemics of influenza A and B, defined as the intervals of meteorological variables with prediction performance optimized. We collected weekly detection rates of laboratory-confirmed influenza cases from four local major hospitals in Hong Kong between 2004 and 2019. Meteorological and air quality records for hospitals were collected from their closest monitoring stations. We employed classification and regression trees to identify zones that optimize the prediction performance of meteorological data in influenza epidemics, defined as a weekly rate > 50^th percentile over a year. According to the results, a combination of temperature > 25.1℃ and relative humidity > 79% was favorable to epidemics in hot seasons, whereas either temperature < 16.4℃ or a combination of < 20.4℃ and relative humidity > 76% was favorable to epidemics in cold seasons. The area under the receiver operating characteristic curve (AUC) in model training achieved 0.80 (95% confidence interval [CI], 0.76-0.83) and was kept at 0.71 (95%CI, 0.65-0.77) in validation. The meteorologically-favorable zones for predicting influenza A or A and B epidemics together were similar, but the AUC for predicting influenza B epidemics was comparatively lower. In conclusion, we established meteorologically-favorable zones for influenza A and B epidemics with a satisfactory prediction performance, even though the influenza seasonality in this subtropical setting was weak and type-specific.

Forecasting the Potential Number of Influenza-like Illness Cases by Fusing Internet Public Opinion

As influenza viruses mutate rapidly, a prediction model for potential outbreaks of influenza-like illnesses helps detect the spread of the illnesses in real time. In order to create a better prediction model, in this study, in addition to using the traditional hydrological and atmospheric data, features, such as popular search keywords on Google Trends, public holiday information, population density, air quality indices, and the numbers of COVID-19 confirmed cases, were also used to train the model in this research. Furthermore, Random Forest and XGBoost were combined and used in the proposed prediction model to increase the prediction accuracy. The training data used in this research were the historical data taken from 2016 to 2021. In our experiments, different combinations of features were tested. The results show that features, such as popular search keywords on Google Trends, the numbers of COVID-19 confirmed cases, and air quality indices can improve the outcome of the prediction model. The evaluation results showed that the error rate between the predicted results and the actual number of influenza-like cases form Week 15 to Week 18 fell to less than 5%. The outbreak of COVID-19 in Taiwan began in Week 19 and resulted in a sharp rise in the number of clinic or hospital visits by patients of influenza-like illnesses. After that, from Week 21 to Week 26, the error rate between the predicted and actual numbers of influenza-like cases in the later period dropped down to 13%. It can be confirmed from the actual experimental results in this research that the use of the ensemble learning prediction model proposed in this research can accurately predict the trend of influenza-like cases.

A Decision-Level Fusion Method for COVID-19 Patient Health Prediction

With the continuous attempts to develop effective machine learning methods, information fusion approaches play an important role in integrating data from multiple sources and improving these methods' performance. Among the different fusion techniques, decision-level fusion has unique advantages to fuse the decisions of various classifiers and getting an effective outcome. In this paper, we propose a decision-level fusion method that combines three well-calibrated ensemble classifiers, namely, a random forest (RF), gradient boosting (GB), and extreme gradient boosting (XGB) methods. It is used to predict the COVID-19 patient health for early monitoring and efficient treatment. A soft voting technique is used to generate the final decision result from the predictions of these calibrated classifiers. The method uses the COVID-19 patient's health information, travel demographic, and geographical data to predict the possible outcome of the COVID-19 case, recovered, or death. A different set of experiments is conducted on a public novel Corona Virus 2019 dataset using a different ratio of test sets. The experimental results show that the proposed fusion method achieved an accuracy of 97.24% and an F1-score of 0.97, which is higher than the current related work that has an accuracy of 94% and an F1-score 0.86, on 20% test set taken from the dataset.

A Comparison of Infectious Disease Forecasting Methods across Locations, Diseases, and Time

Accurate infectious disease forecasting can inform efforts to prevent outbreaks and mitigate adverse impacts. This study compares the performance of statistical, machine learning (ML), and deep learning (DL) approaches in forecasting infectious disease incidences across different countries and time intervals. We forecasted three diverse diseases: campylobacteriosis, typhoid, and Q-fever, using a wide variety of features (n = 46) from public datasets, e.g., landscape, climate, and socioeconomic factors. We compared autoregressive statistical models to two tree-based ML models (extreme gradient boosted trees [XGB] and random forest [RF]) and two DL models (multi-layer perceptron and encoder–decoder model). The disease models were trained on data from seven different countries at the region-level between 2009–2017. Forecasting performance of all models was assessed using mean absolute error, root mean square error, and Poisson deviance across Australia, Israel, and the United States for the months of January through August of 2018. The overall model results were compared across diseases as well as various data splits, including country, regions with highest and lowest cases, and the forecasted months out (i.e., nowcasting, short-term, and long-term forecasting). Overall, the XGB models performed the best for all diseases and, in general, tree-based ML models performed the best when looking at data splits. There were a few instances where the statistical or DL models had minutely smaller error metrics for specific subsets of typhoid, which is a disease with very low case counts. Feature importance per disease was measured by using four tree-based ML models (i.e., XGB and RF with and without region name as a feature). The most important feature groups included previous case counts, region name, population counts and density, mortality causes of neonatal to under 5 years of age, sanitation factors, and elevation. This study demonstrates the power of ML approaches to incorporate a wide range of factors to forecast various diseases, regardless of location, more accurately than traditional statistical approaches.

Time series analysis of hemorrhagic fever with renal syndrome in mainland China by using an XGBoost forecasting model

BACKGROUND

Hemorrhagic fever with renal syndrome (HFRS) is still attracting public attention because of its outbreak in various cities in China. Predicting future outbreaks or epidemics disease based on past incidence data can help health departments take targeted measures to prevent diseases in advance. In this study, we propose a multistep prediction strategy based on extreme gradient boosting (XGBoost) for HFRS as an extension of the one-step prediction model. Moreover, the fitting and prediction accuracy of the XGBoost model will be compared with the autoregressive integrated moving average (ARIMA) model by different evaluation indicators.

METHODS

We collected HFRS incidence data from 2004 to 2018 of mainland China. The data from 2004 to 2017 were divided into training sets to establish the seasonal ARIMA model and XGBoost model, while the 2018 data were used to test the prediction performance. In the multistep XGBoost forecasting model, one-hot encoding was used to handle seasonal features. Furthermore, a series of evaluation indices were performed to evaluate the accuracy of the multistep forecast XGBoost model.

RESULTS

There were 200,237 HFRS cases in China from 2004 to 2018. A long-term downward trend and bimodal seasonality were identified in the original time series. According to the minimum corrected akaike information criterion (CAIC) value, the optimal ARIMA (3, 1, 0) × (1, 1, 0)12 model is selected. The index ME, RMSE, MAE, MPE, MAPE, and MASE indices of the XGBoost model were higher than those of the ARIMA model in the fitting part, whereas the RMSE of the XGBoost model was lower. The prediction performance evaluation indicators (MAE, MPE, MAPE, RMSE and MASE) of the one-step prediction and multistep prediction XGBoost model were all notably lower than those of the ARIMA model.

CONCLUSIONS

The multistep XGBoost prediction model showed a much better prediction accuracy and model stability than the multistep ARIMA prediction model. The XGBoost model performed better in predicting complicated and nonlinear data like HFRS. Additionally, Multistep prediction models are more practical than one-step prediction models in forecasting infectious diseases.

Self-attention-based Deep Learning Network for Regional Influenza Forecasting

Early prediction of influenza plays an important role in minimizing the damage caused, as it provides the resources and time needed to formulate preventive measures. Compared to traditional mechanistic approach, deep/machine learning-based models have demonstrated excellent forecasting performance by efficiently handling various data such as weather and internet data. However, due to the limited availability and reliability of such data, many forecasting models use only historical occurrence data and formulate the influenza forecasting as a multivariate time-series task. Recently, attention mechanisms have been exploited to deal with this issue by selecting valuable data in the input data and giving them high weights. Particularly, self-attention has shown its potential in various forecasting tasks by utilizing the predictive relationship between objects from the input data describing target objects. Hence, in this study, we propose a forecasting model based on self-attention for regional influenza forecasting, called SAIFlu-Net. The model exploits a long short-term memory network for extracting time-series patterns of each region and the self-attention mechanism to find the similarities between the occurrence patterns. To evaluate its performance, we conducted extensive experiments with existing forecasting models using weekly regional influenza datasets. The results show that the proposed model outperforms other models in terms of root mean square error and Pearson correlation coefficient.

An aberration detection-based approach for sentinel syndromic surveillance of COVID-19 and other novel influenza-like illnesses

Coronavirus Disease 2019 has emerged as a significant global concern, triggering harsh public health restrictions in a successful bid to curb its exponential growth. As discussion shifts towards relaxation of these restrictions, there is significant concern of second-wave resurgence. The key to managing these outbreaks is early detection and intervention, and yet there is a significant lag time associated with usage of laboratory confirmed cases for surveillance purposes. To address this, syndromic surveillance can be considered to provide a timelier alternative for first-line screening. Existing syndromic surveillance solutions are however typically focused around a known disease and have limited capability to distinguish between outbreaks of individual diseases sharing similar syndromes. This poses a challenge for surveillance of COVID-19 as its active periods tend to overlap temporally with other influenza-like illnesses. In this study we explore performing sentinel syndromic surveillance for COVID-19 and other influenza-like illnesses using a deep learning-based approach. Our methods are based on aberration detection utilizing autoencoders that leverages symptom prevalence distributions to distinguish outbreaks of two ongoing diseases that share similar syndromes, even if they occur concurrently. We first demonstrate that this approach works for detection of outbreaks of influenza, which has known temporal boundaries. We then demonstrate that the autoencoder can be trained to not alert on known and well-managed influenza-like illnesses such as the common cold and influenza. Finally, we applied our approach to 2019-2020 data in the context of a COVID-19 syndromic surveillance task to demonstrate how implementation of such a system could have provided early warning of an outbreak of a novel influenza-like illness that did not match the symptom prevalence profile of influenza and other known influenza-like illnesses.