Designing risk prediction models for ambulatory no-shows across different specialties and clinics

Public perspectives on the use of different data types for prediction in healthcare

OBJECTIVE

Understand public comfort with the use of different data types for predictive models.

MATERIALS AND METHODS

We analyzed data from a national survey of US adults (n = 1436) fielded from November to December 2021. For three categories of data (identified using factor analysis), we use descriptive statistics to capture comfort level.

RESULTS

Public comfort with data use for prediction is low. For 13 of 15 data types, most respondents were uncomfortable with that data being used for prediction. In factor analysis, 15 types of data grouped into three categories based on public comfort: (1) personal characteristic data, (2) health-related data, and (3) sensitive data. Mean comfort was highest for health-related data (2.45, SD 0.84, range 1-4), followed by personal characteristic data (2.36, SD 0.94), and sensitive data (1.88, SD 0.77). Across these categories, we observe a statistically significant positive relationship between trust in health systems' use of patient information and comfort with data use for prediction.

DISCUSSION

Although public trust is recognized as important for the sustainable expansion of predictive tools, current policy does not reflect public concerns. Low comfort with data use for prediction should be addressed in order to prevent potential negative impacts on trust in healthcare.

CONCLUSION

Our results provide empirical evidence on public perspectives, which are important for shaping the use of predictive models. Findings demonstrate a need for realignment of policy around the sensitivity of non-clinical data categories.

Evaluating the Relationship between Neighborhood-Level Social Vulnerability and Patient Adherence to Ophthalmology Appointments

PURPOSE

To examine the association between neighborhood-level social vulnerability and adherence to scheduled ophthalmology appointments.

METHODS

In this retrospective cohort study, records of all patients ≥18 years scheduled for an ophthalmology appointment between September 12, 2020, and February 8, 2021, were reviewed. Primary exposure is neighborhood-level Social Vulnerability Index (SVI) based on the patient's residential location. SVI is a rank score of 15 social factors into four themes (socioeconomic status, household composition/disability, minority status/language, and housing type/transportation), ranging from 0 to 1.0, with higher ranks indicating greater social vulnerability. The overall SVI score and each theme were analyzed separately as the primary exposure of interest in multivariable logistic regression models that controlled for age, sex, appointment status (new or established), race, and distance from clinic. The primary outcome, non-adherence, was defined as missing more than 25% of scheduled appointments.

RESULTS

Of 8,322 patients (41% non-Hispanic Black, 24% Hispanic, 22% non-Hispanic White) with scheduled appointments, 28% were non-adherent. Non-adherence was associated with greater social vulnerability (adjusted odds ratio [aOR] per 0.01 increase in overall SVI = 2.46 [95% confidence interval, 1.99, 3.06]) and each SVI theme (socioeconomic status: aOR = 2.38 [1.94, 2.91]; household composition/disability: aOR = = 1.51 [1.26, 1.81]; minority status/language: aOR = 2.03 [1.55, 2.68]; housing type/transportation: aOR = 1.41 [1.16, 1.73]).

CONCLUSION

Neighborhood-level social vulnerability is associated with greater risk of non-adherence to scheduled ophthalmology appointments, controlling for individual characteristics. Multi-level intervention strategies that incorporate neighborhood-level vulnerabilities are needed to reduce disparities in access to ophthalmology care.

Decision analysis framework for predicting no-shows to appointments using machine learning algorithms

BACKGROUND

No-show to medical appointments has significant adverse effects on healthcare systems and their clients. Using machine learning to predict no-shows allows managers to implement strategies such as overbooking and reminders targeting patients most likely to miss appointments, optimizing the use of resources.

METHODS

In this study, we proposed a detailed analytical framework for predicting no-shows while addressing imbalanced datasets. The framework includes a novel use of z-fold cross-validation performed twice during the modeling process to improve model robustness and generalization. We also introduce Symbolic Regression (SR) as a classification algorithm and Instance Hardness Threshold (IHT) as a resampling technique and compared their performance with that of other classification algorithms, such as K-Nearest Neighbors (KNN) and Support Vector Machine (SVM), and resampling techniques, such as Random under Sampling (RUS), Synthetic Minority Oversampling Technique (SMOTE) and NearMiss-1. We validated the framework using two attendance datasets from Brazilian hospitals with no-show rates of 6.65% and 19.03%.

RESULTS

From the academic perspective, our study is the first to propose using SR and IHT to predict the no-show of patients. Our findings indicate that SR and IHT presented superior performances compared to other techniques, particularly IHT, which excelled when combined with all classification algorithms and led to low variability in performance metrics results. Our results also outperformed sensitivity outcomes reported in the literature, with values above 0.94 for both datasets.

CONCLUSION

This is the first study to use SR and IHT methods to predict patient no-shows and the first to propose performing z-fold cross-validation twice. Our study highlights the importance of avoiding relying on few validation runs for imbalanced datasets as it may lead to biased results and inadequate analysis of the generalization and stability of the models obtained during the training stage.

How patients distinguish between clinical and administrative predictive models in health care

OBJECTIVES

To understand patient perceptions of specific applications of predictive models in health care.

STUDY DESIGN

Original, cross-sectional national survey.

METHODS

We conducted a national online survey of US adults with the National Opinion Research Center from November to December 2021. Measures of internal consistency were used to identify how patients differentiate between clinical and administrative predictive models. Multivariable logistic regressions were used to identify relationships between comfort with various types of predictive models and patient demographics, perceptions of privacy protections, and experiences in the health care system.

RESULTS

A total of 1541 respondents completed the survey. After excluding observations with missing data for the variables of interest, the final analytic sample was 1488. We found that patients differentiate between clinical and administrative predictive models. Comfort with prediction of bill payment and missed appointments was especially low (21.6% and 36.6%, respectively). Comfort was higher with clinical predictive models, such as predicting stroke in an emergency (55.8%). Experiences of discrimination were significant negative predictors of comfort with administrative predictive models. Health system transparency around privacy policies was a significant positive predictor of comfort with both clinical and administrative predictive models.

CONCLUSIONS

Patients are more comfortable with clinical applications of predictive models than administrative ones. Privacy protections and transparency about how health care systems protect patient data may facilitate patient comfort with these technologies. However, larger inequities and negative experiences in health care remain important for how patients perceive administrative applications of prediction.

Reducing Disparities in No Show Rates Using Predictive Model-Driven Live Appointment Reminders for At-Risk Patients: a Randomized Controlled Quality Improvement Initiative

BACKGROUND

Appointment no shows are prevalent in safety-net healthcare systems. The efficacy and equitability of using predictive algorithms to selectively add resource-intensive live telephone outreach to standard automated reminders in such a setting is not known.

OBJECTIVE

To determine if adding risk-driven telephone outreach to standard automated reminders can improve in-person primary care internal medicine clinic no show rates without worsening racial and ethnic show-rate disparities.

DESIGN

Randomized controlled quality improvement initiative.

PARTICIPANTS

Adult patients with an in-person appointment at a primary care internal medicine clinic in a safety-net healthcare system from 1/1/2022 to 8/24/2022.

INTERVENTIONS

A random forest model that leveraged electronic health record data to predict appointment no show risk was internally trained and validated to ensure fair performance. Schedulers leveraged the model to place reminder calls to patients in the augmented care arm who had a predicted no show rate of 15% or higher.

MAINE MEASURES

The primary outcome was no show rate stratified by race and ethnicity.

KEY RESULTS

There were 5840 appointments with a predicted no show rate of 15% or higher. A total of 2858 had been randomized to the augmented care group and 2982 randomized to standard care. The augmented care group had a significantly lower no show rate than the standard care group (33% vs 36%, p < 0.01). There was a significant reduction in no show rates for Black patients (36% vs 42% respectively, p < 0.001) not reflected in white, non-Hispanic patients.

CONCLUSIONS

In this randomized controlled quality improvement initiative, adding model-driven telephone outreach to standard automated reminders was associated with a significant reduction of in-person no show rates in a diverse primary care clinic. The initiative reduced no show disparities by predominantly improving access for Black patients.

Development of an evidence-based model for predicting patient, provider, and appointment factors that influence no-shows in a rural healthcare system

BACKGROUND

No-show appointments pose a significant challenge for healthcare providers, particularly in rural areas. In this study, we developed an evidence-based predictive model for patient no-shows at the Marshfield Clinic Health System (MCHS) rural provider network in Wisconsin, with the aim of improving overbooking approaches in outpatient settings and reducing the negative impact of no-shows in our underserved rural patient populations.

METHODS

Retrospective data (2021) were obtained from the MCHS scheduling system, which included 1,260,083 total appointments from 263,464 patients, as well as their demographic, appointment, and insurance information. We used descriptive statistics to associate variables with show or no-show status, logistic regression, and random forests utilized, and eXtreme Gradient Boosting (XGBoost) was chosen to develop the final model, determine cut-offs, and evaluate performance. We also used the model to predict future no-shows for appointments from 2022 and onwards.

RESULTS

The no-show rate was 6.0% in both the train and test datasets. The train and test datasets both yielded 5.98. Appointments scheduled further in advance (> 60 days of lead time) had a higher (7.7%) no-show rate. Appointments for patients aged 21-30 had the highest no-show rate (11.8%), and those for patients over 60 years of age had the lowest (2.9%). The model predictions yielded an Area Under Curve (AUC) of 0.84 for the train set and 0.83 for the test set. With the cut-off set to 0.4, the sensitivity was 0.71 and the positive predictive value was 0.18. Model results were used to recommend 1 overbook for every 6 at-risk appointments per provider per day.

CONCLUSIONS

Our findings demonstrate the feasibility of developing a predictive model based on administrative data from a predominantly rural healthcare system. Our new model distinguished between show and no-show appointments with high performance, and 1 overbook was advised for every 6 at-risk appointments. This data-driven approach to mitigating the impact of no-shows increases treatment availability in rural areas by overbooking appointment slots on days with an elevated risk of no-shows.

Associations between Socioeconomic Factors and Visit Adherence among Patients with Glaucoma in the All of Us Research Program

PURPOSE

To identify socioeconomic factors associated with visit adherence among patients with glaucoma in a nationwide cohort.

DESIGN

Cross-sectional study.

SUBJECTS

All subjects were participants in the National Institutes of Health All of Us Research Program. This study cohort consists of participants who were diagnosed with glaucoma and who answered the question on the Health Care Access and Utilization Survey regarding whether they have seen an eye care provider in the last 12 months.

METHODS

Descriptive analyses were conducted based on participant age, gender, race/ethnicity, insurance status, level of education, and income bracket. Multivariable logistic regression adjusting for these factors was used to generate odds ratios (ORs) for the association between socioeconomic factors and visit adherence.

MAIN OUTCOME MEASURE

Visit adherence, defined as reporting seeing an eye care provider in the last 12 months.

RESULTS

Among 4517 patients with glaucoma, 730 (16.3%) indicated that they had not seen or spoken to an eye doctor in the last 12 months. In multivariable models, those with some college education (OR: 1.91; 95% confidence interval [CI]: 1.19-3.04) and those with a college degree or advanced degree (OR: 2.25; 95% CI: 1.39-3.60) and those with the highest annual income of ≥ $200 000 (OR: 1.64; 95% CI: 1.10-2.45) were more likely to have seen an eye doctor in the past year compared with those in the lowest education and income categories, respectively.

CONCLUSION

Lower income and education levels were significantly associated with lower odds of seeing an eye doctor in the past year among all patients with glaucoma in All of Us. This highlights an important health disparity and may inform subsequent interventions to promote improved adherence to clinical guidelines regarding eye care for glaucoma monitoring and management.

FINANCIAL DISCLOSURE(S)

Proprietary or commercial disclosure may be found after the references.

Predicting and explaining absenteeism risk in hospital patients before and during COVID-19

In order to address one of the most challenging problems in hospital management - patients' absenteeism without prior notice - this study analyses the risk factors associated with this event. To this end, through real data from a hospital located in the North of Portugal, a prediction model previously validated in the literature is used to infer absenteeism risk factors, and an explainable model is proposed, based on a modified CART algorithm. The latter intends to generate a human-interpretable explanation for patient absenteeism, and its implementation is described in detail. Furthermore, given the significant impact, the COVID-19 pandemic had on hospital management, a comparison between patients' profiles upon absenteeism before and during the COVID-19 pandemic situation is performed. Results obtained differ between hospital specialities and time periods meaning that patient profiles on absenteeism change during pandemic periods and within specialities.

Social determinants of health and the prediction of missed breast imaging appointments

BACKGROUND

Predictive models utilizing social determinants of health (SDH), demographic data, and local weather data were trained to predict missed imaging appointments (MIA) among breast imaging patients at the Boston Medical Center (BMC). Patients were characterized by many different variables, including social needs, demographics, imaging utilization, appointment features, and weather conditions on the date of the appointment.

METHODS

This HIPAA compliant retrospective cohort study was IRB approved. Informed consent was waived. After data preprocessing steps, the dataset contained 9,970 patients and 36,606 appointments from 1/1/2015 to 12/31/2019. We identified 57 potentially impactful variables used in the initial prediction model and assessed each patient for MIA. We then developed a parsimonious model via recursive feature elimination, which identified the 25 most predictive variables. We utilized linear and non-linear models including support vector machines (SVM), logistic regression (LR), and random forest (RF) to predict MIA and compared their performance.

RESULTS

The highest-performing full model is the nonlinear RF, achieving the highest Area Under the ROC Curve (AUC) of 76% and average F1 score of 85%. Models limited to the most predictive variables were able to attain AUC and F1 scores comparable to models with all variables included. The variables most predictive of missed appointments included timing, prior appointment history, referral department of origin, and socioeconomic factors such as household income and access to caregiving services.

CONCLUSIONS

Prediction of MIA with the data available is inherently limited by the complex, multifactorial nature of MIA. However, the algorithms presented achieved acceptable performance and demonstrated that socioeconomic factors were useful predictors of MIA. In contrast with non-modifiable demographic factors, we can address SDH to decrease the incidence of MIA.

Predicting no-shows for dental appointments

Patient no-shows is a significant problem in healthcare, reaching up to 80% of booked appointments and costing billions of dollars. Predicting no-shows for individual patients empowers clinics to implement better mitigation strategies. Patients' no-show behavior varies across health clinics and the types of appointments, calling for fine-grained studies to uncover these variations in no-show patterns. This article focuses on dental appointments because they are notably longer than regular medical appointments due to the complexity of dental procedures. We leverage machine learning techniques to develop predictive models for dental no-shows, with the best model achieving an Area Under the Curve (AUC) of 0.718 and an F1 score of 66.5%. Additionally, we propose and evaluate a novel method to represent no-show history as a binary sequence of events, enabling the predictive models to learn the associated future no-show behavior with these patterns. We discuss the utility of no-show predictions to improve the scheduling of dental appointments, such as reallocating appointments and reducing their duration.

Modelling the no-show of patients to exam appointments of computed tomography

BACKGROUND

Patients' no-shows negatively impact healthcare systems, leading to resources' underutilisation, efficiency loss, and cost increase. Predicting no-shows is key to developing strategies that counteract their effects. In this paper, we propose a model to predict the no-show of ambulatory patients to exam appointments of computed tomography at the Radiology department of a large Brazilian public hospital.

METHODS

We carried out a retrospective study on 8382 appointments to computed tomography (CT) exams between January and December 2017. Penalised logistic regression and multivariate logistic regression were used to model the influence of 15 candidate variables on patients' no-shows. The predictive capabilities of the models were evaluated by analysing the Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC).

RESULTS

The no-show rate in computerised tomography exams appointments was 6.65%. The two models performed similarly in terms of AUC. The penalised logistic regression model was selected using the parsimony criterion, with 8 of the 15 variables analysed appearing as significant. One of the variables included in the model (number of exams scheduled in the previous year) had not been previously reported in the related literature.

CONCLUSIONS

Our findings may be used to guide the development of strategies to reduce the no-show of patients to exam appointments.

Reducing non-attendance in outpatient appointments: predictive model development, validation, and clinical assessment

Background

Non-attendance to scheduled hospital outpatient appointments may compromise healthcare resource planning, which ultimately reduces the quality of healthcare provision by delaying assessments and increasing waiting lists. We developed a model for predicting non-attendance and assessed the effectiveness of an intervention for reducing non-attendance based on the model.

Methods

The study was conducted in three stages: (1) model development, (2) prospective validation of the model with new data, and (3) a clinical assessment with a pilot study that included the model as a stratification tool to select the patients in the intervention. Candidate models were built using retrospective data from appointments scheduled between January 1, 2015, and November 30, 2018, in the dermatology and pneumology outpatient services of the Hospital Municipal de Badalona (Spain). The predictive capacity of the selected model was then validated prospectively with appointments scheduled between January 7 and February 8, 2019. The effectiveness of selective phone call reminders to patients at high risk of non-attendance according to the model was assessed on all consecutive patients with at least one appointment scheduled between February 25 and April 19, 2019. We finally conducted a pilot study in which all patients identified by the model as high risk of non-attendance were randomly assigned to either a control (no intervention) or intervention group, the last receiving phone call reminders one week before the appointment.

Results

Decision trees were selected for model development. Models were trained and selected using 33,329 appointments in the dermatology service and 21,050 in the pneumology service. Specificity, sensitivity, and accuracy for the prediction of non-attendance were 79.90%, 67.09%, and 73.49% for dermatology, and 71.38%, 57.84%, and 64.61% for pneumology outpatient services. The prospective validation showed a specificity of 78.34% (95%CI 71.07, 84.51) and balanced accuracy of 70.45% for dermatology; and 69.83% (95%CI 60.61, 78.00) for pneumology, respectively. The effectiveness of the intervention was assessed on 1,311 individuals identified as high risk of non-attendance according to the selected model. Overall, the intervention resulted in a significant reduction in the non-attendance rate to both the dermatology and pneumology services, with a decrease of 50.61% (p<0.001) and 39.33% (p=0.048), respectively.

Conclusions

The risk of non-attendance can be adequately estimated using patient information stored in medical records. The patient stratification according to the non-attendance risk allows prioritizing interventions, such as phone call reminders, to effectively reduce non-attendance rates.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-022-07865-y.

Combining adult with pediatric patient data to develop a clinical decision support tool intended for children: leveraging machine learning to model heterogeneity

Background

Clinical decision support (CDS) tools built using adult data do not typically perform well for children. We explored how best to leverage adult data to improve the performance of such tools. This study assesses whether it is better to build CDS tools for children using data from children alone or to use combined data from both adults and children.

Methods

Retrospective cohort using data from 2017 to 2020. Participants include all individuals (adults and children) receiving an elective surgery at a large academic medical center that provides adult and pediatric services. We predicted need for mechanical ventilation or admission to the intensive care unit (ICU). Predictor variables included demographic, clinical, and service utilization factors known prior to surgery. We compared predictive models built using machine learning to regression-based methods that used a pediatric or combined adult-pediatric cohort. We compared model performance based on Area Under the Receiver Operator Characteristic.

Results

While we found that adults and children have different risk factors, machine learning methods are able to appropriately model the underlying heterogeneity of each population and produce equally accurate predictive models whether using data only from pediatric patients or combined data from both children and adults. Results from regression-based methods were improved by the use of pediatric-specific data.

Conclusions

CDS tools for children can successfully use combined data from adults and children if the model accounts for underlying heterogeneity, as in machine learning models.

No-show after extracorporeal shock wave lithotripsy treatment in endourology clinic: Can we build a typical patient profile?

Objectives

Patients “no‐show” in outpatient clinics is a worldwide challenge. Healthcare providers and patients suffer from negative impacts that include increased expenditure, clinical management ineffectiveness, and decreased access to care. This study aims to evaluate no‐show rate among extracorporeal shock wave lithotripsy patients visiting endourology clinic and to identify the demographic and clinical predictors of no‐show.

Methods

A cross‐sectional and historical cohort study using electronic medical records. We included 790 patients aged >18 years old referred for endourology clinic following shock wave lithotripsy during 2010–2017 at Hadassah Medical Center in Israel. We predicted no‐show rate following shock wave lithotripsy by various patient characteristics by a multivariate logistic regression model.

Results

Overall, 291 (36.8%) patients did not arrive for postoperative clinic. Of these, 91 (11.52%) patients referred to Emergency Department. Patients who were younger in age (odds ratio 1.49, 95% confidence interval 1.08–2.04), patients who underwent hospitalization ≥3 days (odds ratio 1.63, 95% confidence interval 1.11–2.41) and patients who had undergone a stent‐free shock wave lithotripsy (odds ratio 5.71, 95% confidence interval 2.40–13.57) were significantly associated with higher no‐show rate. Larger stone size was associated with reduction in no‐show rate with every millimeter increase of stone diameter was associated with a reduction of 6.1% probability for no‐show (odds ratio 0.94, 95% confidence interval 0.89–0.99).

Conclusions

Predicting patients' characteristics and no‐show patterns is necessary to improve clinical management efficiency, access to care, and costs. We showed that patients who were younger, patients who underwent stent‐free shock wave lithotripsy, patients who had a smaller stone, and patients who underwent a longer hospitalization were more prone to miss their appointment. Paying attention to the characteristics of individual patients may assist in implementing intervening program of patient scheduling.

Efficient Prediction of Missed Clinical Appointment Using Machine Learning

Public health and its related facilities are crucial for thriving cities and societies. The optimum utilization of health resources saves money and time, but above all, it saves precious lives. It has become even more evident in the present as the pandemic has overstretched the existing medical resources. Specific to patient appointment scheduling, the casual attitude of missing medical appointments (no-show-ups) may cause severe damage to a patient's health. In this paper, with the help of machine learning, we analyze six million plus patient appointment records to predict a patient's behaviors/characteristics by using ten different machine learning algorithms. For this purpose, we first extracted meaningful features from raw data using data cleaning. We applied Synthetic Minority Oversampling Technique (SMOTE), Adaptive Synthetic Sampling Method (Adasyn), and random undersampling (RUS) to balance our data. After balancing, we applied ten different machine learning algorithms, namely, random forest classifier, decision tree, logistic regression, XG Boost, gradient boosting, Adaboost Classifier, Naive Bayes, stochastic gradient descent, multilayer perceptron, and Support Vector Machine. We analyzed these results with the help of six different metrics, i.e., recall, accuracy, precision, F1-score, area under the curve, and mean square error. Our study has achieved 94% recall, 86% accuracy, 83% precision, 87% F1-score, 92% area under the curve, and 0.106 minimum mean square error. Effectiveness of presented data cleaning and feature selection is confirmed by better results in all training algorithms. Notably, recall is greater than 75%, accuracy is greater than 73%, F1-score is more significant than 75%, MSE is lesser than 0.26, and AUC is greater than 74%. The research shows that instead of individual features, combining different features helps make better predictions of a patient's appointment status.

Prevalence and predictors of no-shows to physical therapy for musculoskeletal conditions

OBJECTIVES

Chronic pain affects 50 million Americans and is often treated with non-pharmacologic approaches like physical therapy. Developing a no-show prediction model for individuals seeking physical therapy care for musculoskeletal conditions has several benefits including enhancement of workforce efficiency without growing the existing provider pool, delivering guideline adherent care, and identifying those that may benefit from telehealth. The objective of this paper was to quantify the national prevalence of no-shows for patients seeking physical therapy care and to identify individual and organizational factors predicting whether a patient will be a no-show when seeking physical therapy care.

DESIGN

Retrospective cohort study.

SETTING

Commercial provider of physical therapy within the United States with 828 clinics across 26 states.

PARTICIPANTS

Adolescent and adult patients (age cutoffs: 14-117 years) seeking non-pharmacological treatment for musculoskeletal conditions from January 1, 2016, to December 31, 2017 (n = 542,685). Exclusion criteria were a primary complaint not considered an MSK condition or improbable values for height, weight, or body mass index values. The study included 444,995 individuals.

PRIMARY AND SECONDARY OUTCOME MEASURES

Prevalence of no-shows for musculoskeletal conditions and predictors of patient no-show.

RESULTS

In our population, 73% missed at least 1 appointment for a given physical therapy care episode. Our model had moderate discrimination for no-shows (c-statistic:0.72, all appointments; 0.73, first 7 appointments) and was well calibrated, with predicted and observed no-shows in good agreement. Variables predicting higher no-show rates included insurance type; smoking-status; higher BMI; and more prior cancellations, time between visit and scheduling date, and between current and previous visit.

CONCLUSIONS

The high prevalence of no-shows when seeking care for musculoskeletal conditions from physical therapists highlights an inefficiency that, unaddressed, could limit delivery of guideline-adherent care that advocates for earlier use of non-pharmacological treatments for musculoskeletal conditions and result in missed opportunities for using telehealth to deliver physical therapy.

A novel model to label delirium in an intensive care unit from clinician actions

BACKGROUND

In the intensive care unit (ICU), delirium is a common, acute, confusional state associated with high risk for short- and long-term morbidity and mortality. Machine learning (ML) has promise to address research priorities and improve delirium outcomes. However, due to clinical and billing conventions, delirium is often inconsistently or incompletely labeled in electronic health record (EHR) datasets. Here, we identify clinical actions abstracted from clinical guidelines in electronic health records (EHR) data that indicate risk of delirium among intensive care unit (ICU) patients. We develop a novel prediction model to label patients with delirium based on a large data set and assess model performance.

METHODS

EHR data on 48,451 admissions from 2001 to 2012, available through Medical Information Mart for Intensive Care-III database (MIMIC-III), was used to identify features to develop our prediction models. Five binary ML classification models (Logistic Regression; Classification and Regression Trees; Random Forests; Naïve Bayes; and Support Vector Machines) were fit and ranked by Area Under the Curve (AUC) scores. We compared our best model with two models previously proposed in the literature for goodness of fit, precision, and through biological validation.

RESULTS

Our best performing model with threshold reclassification for predicting delirium was based on a multiple logistic regression using the 31 clinical actions (AUC 0.83). Our model out performed other proposed models by biological validation on clinically meaningful, delirium-associated outcomes.

CONCLUSIONS

Hurdles in identifying accurate labels in large-scale datasets limit clinical applications of ML in delirium. We developed a novel labeling model for delirium in the ICU using a large, public data set. By using guideline-directed clinical actions independent from risk factors, treatments, and outcomes as model predictors, our classifier could be used as a delirium label for future clinically targeted models.

Application of Machine Learning to Predict Patient No-Shows in an Academic Pediatric Ophthalmology Clinic

Patient "no-shows" are missed appointments resulting in clinical inefficiencies, revenue loss, and discontinuity of care. Using secondary electronic health record (EHR) data, we used machine learning to predict patient no-shows in follow-up and new patient visits in pediatric ophthalmology and to evaluate features for importance. The best model, XGBoost, had an area under the receiver operating characteristics curve (AUC) score of 0.90 for predicting no-shows in follow-up visits. The key findings from this study are: (1) secondary use of EHR data can be used to build datasets for predictive modeling and successfully predict patient no-shows in pediatric ophthalmology, (2) models predicting no-shows for follow-up visits are more accurate than those for new patient visits, and (3) the performance of predictive models is more robust in predicting no-shows compared to individual important features. We hope these models will be used for more effective interventions to mitigate the impact ofpatient no-shows.

Development and Performance of a Clinical Decision Support Tool to Inform Resource Utilization for Elective Operations

IMPORTANCE

Hospitals ceased most elective procedures during the height of coronavirus disease 2019 (COVID-19) infections. As hospitals begin to recommence elective procedures, it is necessary to have a means to assess how resource intensive a given case may be.

OBJECTIVE

To evaluate the development and performance of a clinical decision support tool to inform resource utilization for elective procedures.

DESIGN, SETTING, AND PARTICIPANTS

In this prognostic study, predictive modeling was used on retrospective electronic health records data from a large academic health system comprising 1 tertiary care hospital and 2 community hospitals of patients undergoing scheduled elective procedures from January 1, 2017, to March 1, 2020. Electronic health records data on case type, patient demographic characteristics, service utilization history, comorbidities, and medications were and abstracted and analyzed. Data were analyzed from April to June 2020.

MAIN OUTCOMES AND MEASURES

Predicitons of hospital length of stay, intensive care unit length of stay, need for mechanical ventilation, and need to be discharged to a skilled nursing facility. These predictions were generated using the random forests algorithm. Predicted probabilities were turned into risk classifications designed to give assessments of resource utilization risk.

RESULTS

Data from the electronic health records of 42 199 patients from 3 hospitals were abstracted for analysis. The median length of stay was 2.3 days (range, 1.3-4.2 days), 6416 patients (15.2%) were admitted to the intensive care unit, 1624 (3.8%) received mechanical ventilation, and 2843 (6.7%) were discharged to a skilled nursing facility. Predictive performance was strong with an area under the receiver operator characteristic ranging from 0.76 to 0.93. Sensitivity of the high-risk and medium-risk groupings was set at 95%. The negative predictive value of the low-risk grouping was 99%. We integrated the models into a daily refreshing Tableau dashboard to guide decision-making.

CONCLUSIONS AND RELEVANCE

The clinical decision support tool is currently being used by surgical leadership to inform case scheduling. This work shows the importance of a learning health care environment in surgical care, using quantitative modeling to guide decision-making.

Using machine learning methods to predict in-hospital mortality of sepsis patients in the ICU

Background

Early and accurate identification of sepsis patients with high risk of in-hospital death can help physicians in intensive care units (ICUs) make optimal clinical decisions. This study aimed to develop machine learning-based tools to predict the risk of hospital death of patients with sepsis in ICUs.

Methods

The source database used for model development and validation is the medical information mart for intensive care (MIMIC) III. We identified adult sepsis patients using the new sepsis definition Sepsis-3. A total of 86 predictor variables consisting of demographics, laboratory tests and comorbidities were used. We employed the least absolute shrinkage and selection operator (LASSO), random forest (RF), gradient boosting machine (GBM) and the traditional logistic regression (LR) method to develop prediction models. In addition, the prediction performance of the four developed models was evaluated and compared with that of an existent scoring tool – simplified acute physiology score (SAPS) II – using five different performance measures: the area under the receiver operating characteristic curve (AUROC), Brier score, sensitivity, specificity and calibration plot.

Results

The records of 16,688 sepsis patients in MIMIC III were used for model training and test. Amongst them, 2949 (17.7%) patients had in-hospital death. The average AUROCs of the LASSO, RF, GBM, LR and SAPS II models were 0.829, 0.829, 0.845, 0.833 and 0.77, respectively. The Brier scores of the LASSO, RF, GBM, LR and SAPS II models were 0.108, 0.109, 0.104, 0.107 and 0.146, respectively. The calibration plots showed that the GBM, LASSO and LR models had good calibration; the RF model underestimated high-risk patients; and SAPS II had the poorest calibration.

Conclusion

The machine learning-based models developed in this study had good prediction performance. Amongst them, the GBM model showed the best performance in predicting the risk of in-hospital death. It has the potential to assist physicians in the ICU to perform appropriate clinical interventions for critically ill sepsis patients and thus may help improve the prognoses of sepsis patients in the ICU.

A Probabilistic Patient Scheduling Model with Time Variable Slots

One of the current challenges faced by health centers is to reduce the number of patients who do not attend their appointments. The existence of these patients causes the underutilization of the center's services, which reduces their income and extends patient's access time. In order to reduce these negative effects, several appointment scheduling systems have been developed. With the recent availability of electronic health records, patient scheduling systems that incorporate the patient's no-show prediction are being developed. However, the benefits of including a personalized individual variable time slot for each patient in those probabilistic systems have not been yet analyzed. In this article, we propose a scheduling system based on patients' no-show probabilities with variable time slots and a dynamic priority allocation scheme. The system is based on the solution of a mixed-integer programming model that aims at maximizing the expected profits of the clinic, accounting for first and follow-up visits. We validate our findings by performing an extensive simulation study based on real data and specific scheduling requirements provided by a Spanish hospital. The results suggest potential benefits with the implementation of the proposed allocation system with variable slot times. In particular, the proposed model increases the annual cumulated profit in more than 50% while decreasing the waiting list and waiting times by 30% and 50%, respectively, with respect to the actual appointment scheduling system.

It's how you say it: Systematic A/B testing of digital messaging cut hospital no-show rates

Failure to attend hospital appointments has a detrimental impact on care quality. Documented efforts to address this challenge have only modestly decreased no-show rates. Behavioral economics theory has suggested that more effective messages may lead to increased responsiveness. In complex, real-world settings, it has proven difficult to predict the optimal message composition. In this study, we aimed to systematically compare the effects of several pre-appointment message formats on no-show rates. We randomly assigned members from Clalit Health Services (CHS), the largest payer-provider healthcare organization in Israel, who had scheduled outpatient clinic appointments in 14 CHS hospitals, to one of nine groups. Each individual received a pre-appointment SMS text reminder five days before the appointment, which differed by group. No-show and advanced cancellation rates were compared between the eight alternative messages, with the previously used generic message serving as the control. There were 161,587 CHS members who received pre-appointment reminder messages who were included in this study. Five message frames significantly differed from the control group. Members who received a reminder designed to evoke emotional guilt had a no-show rates of 14.2%, compared with 21.1% in the control group (odds ratio [OR]: 0.69, 95% confidence interval [CI]: 0.67, 0.76), and an advanced cancellation rate of 26.3% compared with 17.2% in the control group (OR: 1.2, 95% CI: 1.19, 1.21). Four additional reminder formats demonstrated significantly improved impact on no-show rates, compared to the control, though not as effective as the best performing message format. Carefully selecting the narrative of pre-appointment SMS reminders can lead to a marked decrease in no-show rates. The process of a/b testing, selecting, and adopting optimal messages is a practical example of implementing the learning healthcare system paradigm, which could prevent up to one-third of the 352,000 annually unattended appointments in Israel.

Patient No-Show Prediction: A Systematic Literature Review

Nowadays, across the most important problems faced by health centers are those caused by the existence of patients who do not attend their appointments. Among others, these patients cause loss of revenue to the health centers and increase the patients’ waiting list. In order to tackle these problems, several scheduling systems have been developed. Many of them require predicting whether a patient will show up for an appointment. However, obtaining these estimates accurately is currently a challenging problem. In this work, a systematic review of the literature on predicting patient no-shows is conducted aiming at establishing the current state-of-the-art. Based on a systematic review following the PRISMA methodology, 50 articles were found and analyzed. Of these articles, 82% were published in the last 10 years and the most used technique was logistic regression. In addition, there is significant growth in the size of the databases used to build the classifiers. An important finding is that only two studies achieved an accuracy higher than the show rate. Moreover, a single study attained an area under the curve greater than the 0.9 value. These facts indicate the difficulty of this problem and the need for further research.

Development, Implementation, and Evaluation of an In-Hospital Optimized Early Warning Score for Patient Deterioration

Background. Identification of patients at risk of deteriorating during their hospitalization is an important concern. However, many off-shelf scores have poor in-center performance. In this article, we report our experience developing, implementing, and evaluating an in-hospital score for deterioration. Methods. We abstracted 3 years of data (2014–2016) and identified patients on medical wards that died or were transferred to the intensive care unit. We developed a time-varying risk model and then implemented the model over a 10-week period to assess prospective predictive performance. We compared performance to our currently used tool, National Early Warning Score. In order to aid clinical decision making, we transformed the quantitative score into a three-level clinical decision support tool. Results. The developed risk score had an average area under the curve of 0.814 (95% confidence interval = 0.79–0.83) versus 0.740 (95% confidence interval = 0.72–0.76) for the National Early Warning Score. We found the proposed score was able to respond to acute clinical changes in patients’ clinical status. Upon implementing the score, we were able to achieve the desired positive predictive value but needed to retune the thresholds to get the desired sensitivity. Discussion. This work illustrates the potential for academic medical centers to build, refine, and implement risk models that are targeted to their patient population and work flow.

Untapped potential of multicenter studies: a review of cardiovascular risk prediction models revealed inappropriate analyses and wide variation in reporting

BACKGROUND

Clinical prediction models are often constructed using multicenter databases. Such a data structure poses additional challenges for statistical analysis (clustered data) but offers opportunities for model generalizability to a broad range of centers. The purpose of this study was to describe properties, analysis, and reporting of multicenter studies in the Tufts PACE Clinical Prediction Model Registry and to illustrate consequences of common design and analyses choices.

METHODS

Fifty randomly selected studies that are included in the Tufts registry as multicenter and published after 2000 underwent full-text screening. Simulated examples illustrate some key concepts relevant to multicenter prediction research.

RESULTS

Multicenter studies differed widely in the number of participating centers (range 2 to 5473). Thirty-nine of 50 studies ignored the multicenter nature of data in the statistical analysis. In the others, clustering was resolved by developing the model on only one center, using mixed effects or stratified regression, or by using center-level characteristics as predictors. Twenty-three of 50 studies did not describe the clinical settings or type of centers from which data was obtained. Four of 50 studies discussed neither generalizability nor external validity of the developed model.

CONCLUSIONS

Regression methods and validation strategies tailored to multicenter studies are underutilized. Reporting on generalizability and potential external validity of the model lacks transparency. Hence, multicenter prediction research has untapped potential.

REGISTRATION

This review was not registered.