Assessment of Time-Series Machine Learning Methods for Forecasting Hospital Discharge Volume

A systematic literature review of predicting patient discharges using statistical methods and machine learning

Discharge planning is integral to patient flow as delays can lead to hospital-wide congestion. Because a structured discharge plan can reduce hospital length of stay while enhancing patient satisfaction, this topic has caught the interest of many healthcare professionals and researchers. Predicting discharge outcomes, such as destination and time, is crucial in discharge planning by helping healthcare providers anticipate patient needs and resource requirements. This article examines the literature on the prediction of various discharge outcomes. Our review discovered papers that explore the use of prediction models to forecast the time, volume, and destination of discharged patients. Of the 101 reviewed papers, 49.5% looked at the prediction with machine learning tools, and 50.5% focused on prediction with statistical methods. The fact that knowing discharge outcomes in advance affects operational, tactical, medical, and administrative aspects is a frequent theme in the papers studied. Furthermore, conducting system-wide optimization, predicting the time and destination of patients after discharge, and addressing the primary causes of discharge delay in the process are among the recommendations for further research in this field.

Forecasting hospital discharges for respiratory conditions in Costa Rica using climate and pollution data

Respiratory diseases represent one of the most significant economic burdens on healthcare systems worldwide. The variation in the increasing number of cases depends greatly on climatic seasonal effects, socioeconomic factors, and pollution. Therefore, understanding these variations and obtaining precise forecasts allows health authorities to make correct decisions regarding the allocation of limited economic and human resources. We aimed to model and forecast weekly hospitalizations due to respiratory conditions in seven regional hospitals in Costa Rica using four statistical learning techniques (Random Forest, XGboost, Facebook's Prophet forecasting model, and an ensemble method combining the above methods), along with 22 climate change indices and aerosol optical depth as an indicator of pollution. Models were trained using data from 2000 to 2018 and were evaluated using data from 2019 as testing data. During the training period, we set up 2-year sliding windows and a 1-year assessment period, along with the grid search method to optimize hyperparameters for each model. The best model for each region was selected using testing data, based on predictive precision and to prevent overfitting. Prediction intervals were then computed using conformal inference. The relative importance of all climatic variables was computed for the best model, and similar patterns in some of the seven regions were observed based on the selected model. Finally, reliable predictions were obtained for each of the seven regional hospitals.

The promise of a model-based psychiatry: building computational models of mental ill health

Computational models have great potential to revolutionise psychiatry research and clinical practice. These models are now used across multiple subfields, including computational psychiatry and precision psychiatry. Their goals vary from understanding mechanisms underlying disorders to deriving reliable classification and personalised predictions. Rapid growth of new tools and data sources (eg, digital data, gamification, and social media) requires an understanding of the constraints and advantages of different modelling approaches in psychiatry. In this Series paper, we take a critical look at the range of computational models that are used in psychiatry and evaluate their advantages and disadvantages for different purposes and data sources. We describe mechanism-driven and mechanism-agnostic computational models and discuss how interpretability of models is crucial for clinical translation. Based on these evaluations, we provide recommendations on how to build computational models that are clinically useful.

Comparison of Natural Language Processing of Clinical Notes With a Validated Risk-Stratification Tool to Predict Severe Maternal Morbidity

IMPORTANCE

Risk-stratification tools are routinely used in obstetrics to assist care teams in assessing and communicating risk associated with delivery. Electronic health record data and machine learning methods may offer a novel opportunity to improve and automate risk assessment.

OBJECTIVE

To compare the predictive performance of natural language processing (NLP) of clinician documentation with that of a previously validated tool to identify individuals at high risk for maternal morbidity.

DESIGN, SETTING, AND PARTICIPANTS

This retrospective diagnostic study was conducted at Brigham and Women's Hospital and Massachusetts General Hospital, Boston, Massachusetts, and included individuals admitted for delivery at the former institution from July 1, 2016, to February 29, 2020. A subset of these encounters (admissions from February to December 2018) was part of a previous prospective validation study of the Obstetric Comorbidity Index (OB-CMI), a comorbidity-weighted score to stratify risk of severe maternal morbidity (SMM).

EXPOSURES

Natural language processing of clinician documentation and OB-CMI scores.

MAIN OUTCOMES AND MEASURES

Natural language processing of clinician-authored admission notes was used to predict SMM in individuals delivering at the same institution but not included in the prospective OB-CMI study. The NLP model was then compared with the OB-CMI in the subset with a known OB-CMI score. Model discrimination between the 2 approaches was compared using the DeLong test. Sensitivity and positive predictive value for the identification of individuals at highest risk were prioritized as the characteristics of interest.

RESULTS

This study included 19 794 individuals; 4034 (20.4%) were included in the original prospective validation study of the OB-CMI (testing set), and the remaining 15 760 (79.6%) composed the training set. Mean (SD) age was 32.3 (5.2) years in the testing cohort and 32.2 (5.2) years in the training cohort. A total of 115 individuals in the testing cohort (2.9%) and 468 in the training cohort (3.0%) experienced SMM. The NLP model was built from a pruned vocabulary of 2783 unique words that occurred within the 15 760 admission notes from individuals in the training set. The area under the receiver operating characteristic curve of the NLP-based model for the prediction of SMM was 0.76 (95% CI, 0.72-0.81) and was comparable with that of the OB-CMI model (0.74; 95% CI, 0.70-0.79) in the testing set (P = .53). Sensitivity (NLP, 28.7%; OB-CMI, 24.4%) and positive predictive value (NLP, 19.4%; OB-CMI, 17.6%) were comparable between the NLP and OB-CMI high-risk designations for the prediction of SMM.

CONCLUSIONS AND RELEVANCE

In this study, the NLP method and a validated risk-stratification tool had a similar ability to identify patients at high risk of SMM. Future prospective research is needed to validate the NLP approach in clinical practice and determine whether it could augment or replace tools requiring manual user input.

Forecasting admissions in psychiatric hospitals before and during Covid-19: a retrospective study with routine data

The COVID-19 pandemic has strong effects on most health care systems. Forecasting of admissions can help for the efficient organisation of hospital care. We aimed to forecast the number of admissions to psychiatric hospitals before and during the COVID-19 pandemic and we compared the performance of machine learning models and time series models. This would eventually allow to support timely resource allocation for optimal treatment of patients. We used admission data from 9 psychiatric hospitals in Germany between 2017 and 2020. We compared machine learning models with time series models in weekly, monthly and yearly forecasting before and during the COVID-19 pandemic. A total of 90,686 admissions were analysed. The models explained up to 90% of variance in hospital admissions in 2019 and 75% in 2020 with the effects of the COVID-19 pandemic. The best models substantially outperformed a one-step seasonal naïve forecast (seasonal mean absolute scaled error (sMASE) 2019: 0.59, 2020: 0.76). The best model in 2019 was a machine learning model (elastic net, mean absolute error (MAE): 7.25). The best model in 2020 was a time series model (exponential smoothing state space model with Box-Cox transformation, ARMA errors and trend and seasonal components, MAE: 10.44). Models forecasting admissions one week in advance did not perform better than monthly and yearly models in 2019 but they did in 2020. The most important features for the machine learning models were calendrical variables. Model performance did not vary much between different modelling approaches before the COVID-19 pandemic and established forecasts were substantially better than one-step seasonal naïve forecasts. However, weekly time series models adjusted quicker to the COVID-19 related shock effects. In practice, multiple individual forecast horizons could be used simultaneously, such as a yearly model to achieve early forecasts for a long planning period and weekly models to adjust quicker to sudden changes.

Natural language processing of admission notes to predict severe maternal morbidity during the delivery encounter

BACKGROUND

Severe maternal morbidity and mortality remain public health priorities in the United States, given their high rates relative to other high-income countries and the notable racial and ethnic disparities that exist. In general, accurate risk stratification methods are needed to help patients, providers, hospitals, and health systems plan for and potentially avert adverse outcomes.

OBJECTIVE

Our objective was to understand if machine learning methods with natural language processing of history and physical notes could identify a group of patients at high risk of maternal morbidity on admission for delivery without relying on any additional patient information (eg, demographics and diagnosis codes).

STUDY DESIGN

This was a retrospective study of people admitted for delivery at 2 hospitals (hospitals A and B) in a single healthcare system between July 1, 2016, and June 30, 2020. The primary outcome was severe maternal morbidity, as defined by the Centers for Disease Control and Prevention; furthermore, we examined nontransfusion severe maternal morbidity. Clinician documents designated as history and physical notes were extracted from the electronic health record for processing and analysis. A bag-of-words approach was used for this natural language processing analysis (ie, each history or physical note was converted into a matrix of counts of individual words (or phrases) that occurred within the document). The least absolute shrinkage and selection operator models were used to generate prediction probabilities for severe maternal morbidity and nontransfusion severe maternal morbidity for each note. Model discrimination was assessed via the area under the receiver operating curve. Discrimination was compared between models using the DeLong test. Calibration plots were generated to assess model calibration. Moreover, the natural language processing models with history and physical note texts were compared with validated obstetrical comorbidity risk scores based on diagnosis codes.

RESULTS

There were 13,572 delivery encounters with history and physical notes from hospital A, split between training (A_train, n=10,250) and testing (A_test, n=3,322) datasets for model derivation and internal validation. There were 23,397 delivery encounters with history and physical notes from hospital B (B_valid) used for external validation. For the outcome of severe maternal morbidity, the natural language processing model had an area under the receiver operating curve of 0.67 (95% confidence interval, 0.63-0.72) and 0.72 (95% confidence interval, 0.70-0.74) in the A_test and B_valid datasets, respectively. For the outcome of nontransfusion severe maternal morbidity, the area under the receiver operating curve was 0.72 (95% confidence interval, 0.65-0.80) and 0.76 (95% confidence interval, 0.73-0.79) in the A_test and B_valid datasets, respectively. The calibration plots demonstrated the bag-of-words model's ability to distinguish a group of individuals at a substantially higher risk of severe maternal morbidity and nontransfusion severe maternal morbidity, notably those in the top deciles of predicted risk. Areas under the receiver operating curve in the natural language processing-based models were similar to those generated using a validated, retrospectively derived, diagnosis code-based comorbidity score.

CONCLUSION

In this practical application of machine learning, we demonstrated the capabilities of natural language processing for the prediction of severe maternal morbidity based on provider documentation inherently generated at the time of admission. This work should serve as a catalyst for providers, hospitals, and electronic health record systems to explore ways that artificial intelligence can be incorporated into clinical practice and evaluated rigorously for their ability to improve health.

Longitudinal validation of an electronic health record delirium prediction model applied at admission in COVID-19 patients

OBJECTIVE

To validate a previously published machine learning model of delirium risk in hospitalized patients with coronavirus disease 2019 (COVID-19).

METHOD

Using data from six hospitals across two academic medical networks covering care occurring after initial model development, we calculated the predicted risk of delirium using a previously developed risk model applied to diagnostic, medication, laboratory, and other clinical features available in the electronic health record (EHR) at time of hospital admission. We evaluated the accuracy of these predictions against subsequent delirium diagnoses during that admission.

RESULTS

Of the 5102 patients in this cohort, 716 (14%) developed delirium. The model's risk predictions produced a c-index of 0.75 (95% CI, 0.73-0.77) with 27.7% of cases occurring in the top decile of predicted risk scores. Model calibration was diminished compared to the initial COVID-19 wave.

CONCLUSION

This EHR delirium risk prediction model, developed during the initial surge of COVID-19 patients, produced consistent discrimination over subsequent larger waves; however, with changing cohort composition and delirium occurrence rates, model calibration decreased. These results underscore the importance of calibration, and the challenge of developing risk models for clinical contexts where standard of care and clinical populations may shift.

Heart Rate Modeling and Prediction Using Autoregressive Models and Deep Learning

Physiological time series are affected by many factors, making them highly nonlinear and nonstationary. As a consequence, heart rate time series are often considered difficult to predict and handle. However, heart rate behavior can indicate underlying cardiovascular and respiratory diseases as well as mood disorders. Given the importance of accurate modeling and reliable predictions of heart rate fluctuations for the prevention and control of certain diseases, it is paramount to identify models with the best performance in such tasks. The objectives of this study were to compare the results of three different forecasting models (Autoregressive Model, Long Short-Term Memory Network, and Convolutional Long Short-Term Memory Network) trained and tested on heart rate beats per minute data obtained from twelve heterogeneous participants and to identify the architecture with the best performance in terms of modeling and forecasting heart rate behavior. Heart rate beats per minute data were collected using a wearable device over a period of 10 days from twelve different participants who were heterogeneous in age, sex, medical history, and lifestyle behaviors. The goodness of the results produced by the models was measured using both the mean absolute error and the root mean square error as error metrics. Despite the three models showing similar performance, the Autoregressive Model gave the best results in all settings examined. For example, considering one of the participants, the Autoregressive Model gave a mean absolute error of 2.069 (compared to 2.173 of the Long Short-Term Memory Network and 2.138 of the Convolutional Long Short-Term Memory Network), achieving an improvement of 5.027% and 3.335%, respectively. Similar results can be observed for the other participants. The findings of the study suggest that regardless of an individual's age, sex, and lifestyle behaviors, their heart rate largely depends on the pattern observed in the previous few minutes, suggesting that heart rate can be reasonably regarded as an autoregressive process. The findings also suggest that minute-by-minute heart rate prediction can be accurately performed using a linear model, at least in individuals without pathologies that cause heartbeat irregularities. The findings also suggest many possible applications for the Autoregressive Model, in principle in any context where minute-by-minute heart rate prediction is required (arrhythmia detection and analysis of the response to training, among others).

Exploiting Causality for Improved Prediction of Patient Volumes by Gaussian Processes

Estimating and surveillance volumes of patients are of great importance for public health and resource allocation. In many situations, the change of these volumes is correlated with many factors, e.g., seasonal environmental variables, medicine sales, and patient medical claims. It is often of interest to predict patient volumes and to that end, discovering causalities can improve the prediction accuracy. Correlations do not imply causations and they can be spurious, which in turn may entail deterioration of prediction performance if the prediction is based on them. By contrast, in this paper, we propose an approach for prediction based on causalities discovered by Gaussian processes. Our interest is in estimating volumes of patients that suffer from allergy and where the model and the results are highly interpretable. In selecting features, instead of only using correlation, we take causal information into account. Specifically, we adopt the Gaussian processes-based convergent cross mapping framework for causal discovery which is proven to be more reliable than the Granger causality when time series are coupled. Moreover, we introduce a novel method for selecting the history or look-back length of features from the perspective of a dynamical system in a principled manner. The quasi-periodicities that commonly exist in observations of volumes of patients and environment variables can readily be accommodated. Further, the proposed method performs well even in cases when the data are scarce. Also, the approach can be modified without much difficulty to forecast other types of patient volumes. We validate the method with synthetic and real-world datasets.

How New Mexico Leveraged a COVID-19 Case Forecasting Model to Preemptively Address the Health Care Needs of the State: Quantitative Analysis

BACKGROUND

Prior to the COVID-19 pandemic, US hospitals relied on static projections of future trends for long-term planning and were only beginning to consider forecasting methods for short-term planning of staffing and other resources. With the overwhelming burden imposed by COVID-19 on the health care system, an emergent need exists to accurately forecast hospitalization needs within an actionable timeframe.

OBJECTIVE

Our goal was to leverage an existing COVID-19 case and death forecasting tool to generate the expected number of concurrent hospitalizations, occupied intensive care unit (ICU) beds, and in-use ventilators 1 day to 4 weeks in the future for New Mexico and each of its five health regions.

METHODS

We developed a probabilistic model that took as input the number of new COVID-19 cases for New Mexico from Los Alamos National Laboratory's COVID-19 Forecasts Using Fast Evaluations and Estimation tool, and we used the model to estimate the number of new daily hospital admissions 4 weeks into the future based on current statewide hospitalization rates. The model estimated the number of new admissions that would require an ICU bed or use of a ventilator and then projected the individual lengths of hospital stays based on the resource need. By tracking the lengths of stay through time, we captured the projected simultaneous need for inpatient beds, ICU beds, and ventilators. We used a postprocessing method to adjust the forecasts based on the differences between prior forecasts and the subsequent observed data. Thus, we ensured that our forecasts could reflect a dynamically changing situation on the ground.

RESULTS

Forecasts made between September 1 and December 9, 2020, showed variable accuracy across time, health care resource needs, and forecast horizon. Forecasts made in October, when new COVID-19 cases were steadily increasing, had an average accuracy error of 20.0%, while the error in forecasts made in September, a month with low COVID-19 activity, was 39.7%. Across health care use categories, state-level forecasts were more accurate than those at the regional level. Although the accuracy declined as the forecast was projected further into the future, the stated uncertainty of the prediction improved. Forecasts were within 5% of their stated uncertainty at the 50% and 90% prediction intervals at the 3- to 4-week forecast horizon for state-level inpatient and ICU needs. However, uncertainty intervals were too narrow for forecasts of state-level ventilator need and all regional health care resource needs.

CONCLUSIONS

Real-time forecasting of the burden imposed by a spreading infectious disease is a crucial component of decision support during a public health emergency. Our proposed methodology demonstrated utility in providing near-term forecasts, particularly at the state level. This tool can aid other stakeholders as they face COVID-19 population impacts now and in the future.

Quantifying paediatric intensive care unit staffing levels at a paediatric academic medical centre: A mixed-methods approach

AIM

To identify, simulate and evaluate the formal and informal patient-level and unit-level factors that nurse managers use to determine the number of nurses for each shift.

BACKGROUND

Nurse staffing schedules are commonly set based on metrics such as midnight census that do not account for seasonality or midday turnover, resulting in last-minute adjustments or inappropriate staffing levels.

METHODS

Staffing schedules at a paediatric intensive care unit (PICU) were simulated based on nurse-to-patient assignment rules from interviews with nursing management. Multivariate regression modelled the discrepancies between scheduled and historical staffing levels and constructed rules to reduce these discrepancies. The primary outcome was the median difference between simulated and historical staffing levels.

RESULTS

Nurse-to-patient ratios underestimated staffing by a median of 1.5 nurses per shift. Multivariate regression identified patient turnover as the primary factor accounting for this difference and subgroup analysis revealed that patient age and weight were also important. New rules reduced the difference to a median of 0.07 nurses per shift.

CONCLUSION

Measurable, predictable indicators of patient acuity and historical trends may allow for schedules that better match demand.

IMPLICATIONS FOR NURSING MANAGEMENT

Data-driven methods can quantify what drives unit demand and generate nurse schedules that require fewer last-minute adjustments.

A Comparison of Time-Series Predictions for Healthcare Emergency Department Indicators and the Impact of COVID-19

Across the world, healthcare systems are under stress and this has been hugely exacerbated by the COVID pandemic. Key Performance Indicators (KPIs), usually in the form of time-series data, are used to help manage that stress. Making reliable predictions of these indicators, particularly for emergency departments (ED), can facilitate acute unit planning, enhance quality of care and optimise resources. This motivates models that can forecast relevant KPIs and this paper addresses that need by comparing the Autoregressive Integrated Moving Average (ARIMA) method, a purely statistical model, to Prophet, a decomposable forecasting model based on trend, seasonality and holidays variables, and to the General Regression Neural Network (GRNN), a machine learning model. The dataset analysed is formed of four hourly valued indicators from a UK hospital: Patients in Department; Number of Attendances; Unallocated Patients with a DTA (Decision to Admit); Medically Fit for Discharge. Typically, the data exhibit regular patterns and seasonal trends and can be impacted by external factors such as the weather or major incidents. The COVID pandemic is an extreme instance of the latter and the behaviour of sample data changed dramatically. The capacity to quickly adapt to these changes is crucial and is a factor that shows better results for GRNN in both accuracy and reliability.

Machine learning in patient flow: a review

This work is a review of the ways in which machine learning has been used in order to plan, improve or aid the problem of moving patients through healthcare services. We decompose the patient flow problem into four subcategories: prediction of demand on a healthcare institution, prediction of the demand and resource required to transfer patients from the emergency department to the hospital, prediction of potential resource required for the treatment and movement of inpatients and prediction of length-of-stay and discharge timing. We argue that there are benefits to both approaches of considering the healthcare institution as a whole as well as the patient by patient case and that ideally a combination of these would be best for improving patient flow through hospitals. We also argue that it is essential for there to be a shared dataset that will allow researchers to benchmark their algorithms on and thereby allow future researchers to build on that which has already been done. We conclude that machine learning for the improvement of patient flow is still a young field with very few papers tailor-making machine learning methods for the problem being considered. Future works should consider the need to transfer algorithms trained on a dataset to multiple hospitals and allowing for dynamic algorithms which will allow real-time decision-making to help clinical staff on the shop floor.

Development and External Validation of a Delirium Prediction Model for Hospitalized Patients With Coronavirus Disease 2019

Background

The coronavirus disease 2019 pandemic has placed unprecedented stress on health systems and has been associated with elevated risk for delirium. The convergence of pandemic resource limitation and clinical demand associated with delirium requires careful risk stratification for targeted prevention efforts.

Objectives

To develop an incident delirium predictive model among coronavirus disease 2019 patients.

Methods

We applied supervised machine learning to electronic health record data for inpatients with coronavirus disease 2019 at three hospitals to build an incident delirium diagnosis prediction model. We validated this model in three different hospitals. Both hospital cohorts included academic and community settings.

Results

Among 2907 patients across 6 hospitals, 488 (16.8%) developed delirium. Applying the predictive model in the external validation cohort of 755 patients, the c-index was 0.75 (0.71–0.79) and the lift in the top quintile was 2.1. At a sensitivity of 80%, the specificity was 56%, negative predictive value 92%, and positive predictive value 30%. Equivalent model performance was observed in subsamples stratified by age, sex, race, need for critical care and care at community vs. academic hospitals.

Conclusion

Machine learning applied to electronic health records available at the time of inpatient admission can be used to risk-stratify patients with coronavirus disease 2019 for incident delirium. Delirium is common among patients with coronavirus disease 2019, and resource constraints during a pandemic demand careful attention to the optimal application of predictive models.

Development and Validation of a Machine Learning Model to Aid Discharge Processes for Inpatient Surgical Care

IMPORTANCE

Inpatient overcrowding is associated with delays in care, including the deferral of surgical care until beds are available to accommodate postoperative patients. Timely patient discharge is critical to address inpatient overcrowding and requires coordination among surgeons, nurses, case managers, and others. This is difficult to achieve without early identification and systemwide transparency of discharge candidates and their respective barriers to discharge.

OBJECTIVE

To validate the performance of a clinically interpretable feedforward neural network model that could improve the discharge process by predicting which patients would be discharged within 24 hours and their clinical and nonclinical barriers.

DESIGN, SETTING, AND PARTICIPANTS

This prognostic study included adult patients discharged from inpatient surgical care from May 1, 2016, to August 31, 2017, at a quaternary care teaching hospital. Model performance was assessed with standard cross-validation techniques. The model's performance was compared with a baseline model using historical procedure median length of stay to predict discharges. In prospective cohort analysis, the feedforward neural network model was used to make predictions on general surgical care floors with 63 beds. If patients were not discharged when predicted, the causes of delay were recorded.

MAIN OUTCOMES AND MEASURES

The primary outcome was the out-of-sample area under the receiver operating characteristic curve of the model. Secondary outcomes included the causes of discharge delay and the number of avoidable bed-days.

RESULTS

The model was trained on 15 201 patients (median [interquartile range] age, 60 [46-70] years; 7623 [50.1%] men) discharged from inpatient surgical care. The estimated out-of-sample area under the receiver operating characteristic curve of the model was 0.840 (SD, 0.008; 95% CI, 0.839-0.844). Compared with the baseline model, the neural network model had higher sensitivity (52.5% vs 56.6%) and specificity (51.7% vs 82.6%). The neural network model identified 65 barriers to discharge. In the prospective study of 605 patients, causes of delays included clinical barriers (41 patients [30.1%]), variation in clinical practice (30 patients [22.1%]), and nonclinical reasons (65 patients [47.8%]). Summing patients who were not discharged owing to variation in clinical practice and nonclinical reasons, 128 bed-days, or 1.2 beds per day, were classified as avoidable.

CONCLUSIONS AND RELEVANCE

This cohort study found that a neural network model could predict daily inpatient surgical care discharges and their barriers. The model identified systemic causes of discharge delays. Such models should be studied for their ability to increase the timeliness of discharges.