Development and validation of a patient no-show predictive model at a primary care setting in Southern Brazil

Enhancing Health Equity by Predicting Missed Appointments in Health Care: Machine Learning Study

BACKGROUND

The phenomenon of patients missing booked appointments without canceling them-known as Did Not Show (DNS), Did Not Attend (DNA), or Failed To Attend (FTA)-has a detrimental effect on patients' health and results in massive health care resource wastage.

OBJECTIVE

Our objective was to develop machine learning (ML) models and evaluate their performance in predicting the likelihood of DNS for hospital outpatient appointments at the MidCentral District Health Board (MDHB) in New Zealand.

METHODS

We sourced 5 years of MDHB outpatient records (a total of 1,080,566 outpatient visits) to build the ML prediction models. We developed 3 ML models using logistic regression, random forest, and Extreme Gradient Boosting (XGBoost). Subsequently, 10-fold cross-validation and hyperparameter tuning were deployed to minimize model bias and boost the algorithms' prediction strength. All models were evaluated against accuracy, sensitivity, specificity, and area under the receiver operating characteristic (AUROC) curve metrics.

RESULTS

Based on 5 years of MDHB data, the best prediction classifier was XGBoost, with an area under the curve (AUC) of 0.92, sensitivity of 0.83, and specificity of 0.85. The patients' DNS history, age, ethnicity, and appointment lead time significantly contributed to DNS prediction. An ML system trained on a large data set can produce useful levels of DNS prediction.

CONCLUSIONS

This research is one of the very first published studies that use ML technologies to assist with DNS management in New Zealand. It is a proof of concept and could be used to benchmark DNS predictions for the MDHB and other district health boards. We encourage conducting additional qualitative research to investigate the root cause of DNS issues and potential solutions. Addressing DNS using better strategies potentially can result in better utilization of health care resources and improve health equity.

Decision analysis framework for predicting no-shows to appointments using machine learning algorithms

BACKGROUND

No-show to medical appointments has significant adverse effects on healthcare systems and their clients. Using machine learning to predict no-shows allows managers to implement strategies such as overbooking and reminders targeting patients most likely to miss appointments, optimizing the use of resources.

METHODS

In this study, we proposed a detailed analytical framework for predicting no-shows while addressing imbalanced datasets. The framework includes a novel use of z-fold cross-validation performed twice during the modeling process to improve model robustness and generalization. We also introduce Symbolic Regression (SR) as a classification algorithm and Instance Hardness Threshold (IHT) as a resampling technique and compared their performance with that of other classification algorithms, such as K-Nearest Neighbors (KNN) and Support Vector Machine (SVM), and resampling techniques, such as Random under Sampling (RUS), Synthetic Minority Oversampling Technique (SMOTE) and NearMiss-1. We validated the framework using two attendance datasets from Brazilian hospitals with no-show rates of 6.65% and 19.03%.

RESULTS

From the academic perspective, our study is the first to propose using SR and IHT to predict the no-show of patients. Our findings indicate that SR and IHT presented superior performances compared to other techniques, particularly IHT, which excelled when combined with all classification algorithms and led to low variability in performance metrics results. Our results also outperformed sensitivity outcomes reported in the literature, with values above 0.94 for both datasets.

CONCLUSION

This is the first study to use SR and IHT methods to predict patient no-shows and the first to propose performing z-fold cross-validation twice. Our study highlights the importance of avoiding relying on few validation runs for imbalanced datasets as it may lead to biased results and inadequate analysis of the generalization and stability of the models obtained during the training stage.

Predicting ophthalmic clinic non-attendance using machine learning: Development and validation of models using nationwide data

BACKGROUND

Ophthalmic clinic non-attendance in New Zealand is associated with poorer health outcomes, marked inequities and costs NZD$30 million per annum. Initiatives to improve attendance typically involve expensive and ineffective brute-force strategies. The aim was to develop machine learning models to accurately predict ophthalmic clinic non-attendance.

METHODS

This multicentre, retrospective observational study developed and validated predictive models of clinic non-attendance. Attendance data for 3.1 million appointments from all New Zealand government-funded ophthalmology clinics from 2009 to 2018 were aggregated for analysis. Repeated ten-fold cross validation was used to train and optimise XGBoost and logistic regression models on several demographic and clinic-related variables. Models developed using the entire training set were compared with those restricted to regional subsets of the data.

RESULTS

In the testing data set from 2019, there were 407 574 appointments (median [range] age, 66 [0-105] years; 210 365 [51.6%] female) with a non-attendance rate of 5.7% (n = 23 309 missed appointments), XGBoost models trained on each region's data achieved the highest mean AUROC of 0.764 (SD 0.058) and mean AUPRC of 0.157 (SD 0.072). XGBoost performed better than logistic regression (mean AUROC = 0.756, p = 0.002). Training individual XGBoost models for each region led to better performance than training a single model on the complete nationwide dataset (mean AUROC = 0.754, p = 0.04).

CONCLUSION

Machine learning algorithms can predict ophthalmic clinic non-attendance with relatively basic demographic and clinic data. These findings suggest further research examining implementation of such algorithms in scheduling systems or public health interventions may be useful.

What contributes to Primary Health Care effectiveness? Integrative literature review, 2010-2020

Primary Health Care (PHC) intends to rearrange services to make it more effective. Nevertheless, effectiveness in PHC is quite a challenge. This study reviews several articles regarding the effectiveness improvements in PHC between 2010 and 2020. Ninety out of 8,369 articles found in PubMed and the Virtual Health Library databases search were selected for thematic analysis using the Atlas.ti® 9.0 software. There were four categories identified: strategies for monitoring and evaluating health services, organizational arrangements, models and technologies applied to PHC. Studies concerning the sensitive conditions indicators were predominant. Institutional assessment programs, PHC as a structuring policy, appropriate workforce, measures to increase access and digital technologies showed positive effects. However, payment for performance is still controversial. The expressive number of Brazilian publications reveals the broad diffusion of PHC in the country and the concern on its performance. These findings reassure well-known aspects, but it also points to the need for a logical model to better define what is intended as effectiveness within primary health care as well as clarify the polysemy that surrounds the concept. We also suggest substituting the term "resolvability", commonly used in Brazil, for "effectiveness".

Application of a Machine Learning Algorithm to Develop and Validate a Prediction Model for Ambulatory Non-Arrivals

BACKGROUND

Non-arrivals to scheduled ambulatory visits are common and lead to a discontinuity of care, poor health outcomes, and increased subsequent healthcare utilization. Reducing non-arrivals is important given their association with poorer health outcomes and cost to health systems.

OBJECTIVE

To develop and validate a prediction model for ambulatory non-arrivals.

DESIGN

Retrospective cohort study.

PATIENTS OR SUBJECTS

Patients at an integrated health system who had an outpatient visit scheduled from January 1, 2020, to February 28, 2022.

MAIN MEASURES

Non-arrivals to scheduled appointments.

KEY RESULTS

There were over 4.3 million ambulatory appointments from 1.2 million adult patients. Patients with appointment non-arrivals were more likely to be single, racial/ethnic minorities, and not having an established primary care provider compared to those who arrived at their appointments. A prediction model using the XGBoost machine learning algorithm had the highest AUC value (0.768 [0.767-0.770]). Using SHAP values, the most impactful features in the model include rescheduled appointments, lead time (number of days from scheduled to appointment date), appointment provider, number of days since last appointment with the same department, and a patient's prior appointment status within the same department. Scheduling visits close to an appointment date is predicted to be less likely to result in a non-arrival. Overall, the prediction model calibrated well for each department, especially over the operationally relevant probability range of 0 to 40%. Departments with fewer observations and lower non-arrival rates generally had a worse calibration.

CONCLUSIONS

Using a machine learning algorithm, we developed a prediction model for non-arrivals to scheduled ambulatory appointments usable for all medical specialties. The proposed prediction model can be deployed within an electronic health system or integrated into other dashboards to reduce non-arrivals. Future work will focus on the implementation and application of the model to reduce non-arrivals.

Improving Hospital Outpatient Clinics Appointment Schedules by Prediction Models

Patient no-shows and suboptimal patient appointment length scheduling reduce clinical efficiency and impair the clinic's quality of service. The main objective of this study is to improve appointment scheduling in hospital outpatient clinics. We developed generic supervised machine learning models to predict patient no-shows and patient's length of appointment (LOA). We performed a retrospective study using more than 100,000 records of patient appointments in a hospital outpatient clinic. Several machine learning algorithms were used for the development of our prediction models. We trained our models on a dataset that contained patients', physicians', and appointments' characteristics. Our feature set combines both unstudied features and features adopted from previous studies. In addition, we identified the influential features for predicting LOA and no-show. Our LOA model's performance was 6.92 in terms of MAE, and our no-show model's performance was 92.1% in terms of F-score. We compared our models' performance to the performance of previous research models by applying their methods to our dataset; our models demonstrated better performance. We show that the major effector of such differences is the use of our novel features. To evaluate the effect of our prediction results on the quality of schedules produced by appointment systems (AS), we developed an interface layer between our prediction models and the AS, where prediction results comprise the AS input. Using our prediction models, there was an 80% improvement in the daily cumulative patient waiting time and a 33% reduction in the daily cumulative physician idle time.

Predicting no-shows for dental appointments

Patient no-shows is a significant problem in healthcare, reaching up to 80% of booked appointments and costing billions of dollars. Predicting no-shows for individual patients empowers clinics to implement better mitigation strategies. Patients' no-show behavior varies across health clinics and the types of appointments, calling for fine-grained studies to uncover these variations in no-show patterns. This article focuses on dental appointments because they are notably longer than regular medical appointments due to the complexity of dental procedures. We leverage machine learning techniques to develop predictive models for dental no-shows, with the best model achieving an Area Under the Curve (AUC) of 0.718 and an F1 score of 66.5%. Additionally, we propose and evaluate a novel method to represent no-show history as a binary sequence of events, enabling the predictive models to learn the associated future no-show behavior with these patterns. We discuss the utility of no-show predictions to improve the scheduling of dental appointments, such as reallocating appointments and reducing their duration.

Modelling the no-show of patients to exam appointments of computed tomography

BACKGROUND

Patients' no-shows negatively impact healthcare systems, leading to resources' underutilisation, efficiency loss, and cost increase. Predicting no-shows is key to developing strategies that counteract their effects. In this paper, we propose a model to predict the no-show of ambulatory patients to exam appointments of computed tomography at the Radiology department of a large Brazilian public hospital.

METHODS

We carried out a retrospective study on 8382 appointments to computed tomography (CT) exams between January and December 2017. Penalised logistic regression and multivariate logistic regression were used to model the influence of 15 candidate variables on patients' no-shows. The predictive capabilities of the models were evaluated by analysing the Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC).

RESULTS

The no-show rate in computerised tomography exams appointments was 6.65%. The two models performed similarly in terms of AUC. The penalised logistic regression model was selected using the parsimony criterion, with 8 of the 15 variables analysed appearing as significant. One of the variables included in the model (number of exams scheduled in the previous year) had not been previously reported in the related literature.

CONCLUSIONS

Our findings may be used to guide the development of strategies to reduce the no-show of patients to exam appointments.

Comparison Between Short Text Messages and Phone Calls to Reduce No-Show Rates in Outpatient Medical Appointments: A Randomized Trial

The objective of this study was to evaluate the impact of telephone calls and short text messages (SMS) on no-show rates regarding scheduled appointments with a general practitioner. In a prospective, intervention-controlled, and randomized study, we divided 306 patients into 3 groups: telephone call, SMS, and no intervention. We compared no-show rates, as well as variables that influenced it. The lowest percentage of no-show (9.5%) occurred in the telephone call group, while the SMS group presented at 21% and the no-intervention group at 22.8% (P = .025). Telephone calls proved to be a superior strategy to text messaging.

Modeling Patient No-Show History and Predicting Future Appointment Behavior at the Veterans Administration's Outpatient Mental Health Clinics: NIRMO-2

INTRODUCTION

No-shows are detrimental to both patients' health and health care systems. Literature documents no-show rates ranging from 10% in primary care clinics to over 60% in mental health clinics. Our model predicts the probability that a mental health clinic outpatient appointment will not be completed and identifies actionable variables associated with lowering the probability of no-show.

MATERIALS AND METHODS

We were granted access to de-identified administrative data from the Veterans Administration Corporate Data Warehouse related to appointments at 13 Veterans Administration Medical Centers. Our modeling data set included 1,206,271 unique appointment records scheduled to occur between January 1, 2013 and February 28, 2017. The training set included 846,668 appointment records scheduled between January 1, 2013 and December 31, 2015. The testing set included 359,603 appointment records scheduled between January 1, 2016 and February 28, 2017. The dependent binary variable was whether the appointment was completed or not. Independent variables were categorized into seven clusters: patient's demographics, appointment characteristics, patient's attendance history, alcohol use screening score, medications and medication possession ratios, prior diagnoses, and past utilization of Veterans Health Administration services. We used a forward stepwise selection, based on the likelihood ratio, to choose the variables in the model. The predictive model was built using the SAS HPLOGISTIC procedure.

RESULTS

The best indicator of whether someone will miss an appointment is their historical attendance behavior. The top three variables associated with higher probabilities of a no-show were: the no-show rate over the previous 2 years before the current appointment, the no-show probability derived from the Markov model, and the age of the appointment. The top three variables that decrease the chance of no-showing were: the appointment was a new consult, the appointment was an overbook, and the patient had multiple appointments on the same day. The average of the areas under the receiver operating characteristic curves was 0.7577 for the training dataset, and 0.7513 for the test set.

CONCLUSIONS

The National Initiative to Reduce Missed Opportunities-2 confirmed findings that previous patient attendance is one of the key predictors of a future attendance and provides an additional layer of complexity for analyzing the effect of a patient's past behavior on future attendance. The National Initiative to Reduce Missed Opportunities-2 establishes that appointment attendance is related to medication adherence, particularly for medications used for treatment of mood disorders or to block the effects of opioids. However, there is no way to confirm whether a patient is actually taking medications as prescribed. Thus, a low medication possession ratio is an informative, albeit not a perfect, measure. It is our intention to further explore how diagnosis and medications can be better captured and used in predictive modeling of no-shows. Our findings on the effects of different factors on no-show rates can be used to predict individual no-show probabilities, and to identify patients who are high risk for missing appointments. The ability to predict a patient's risk of missing an appointment would allow for both advanced interventions to decrease no-shows and for more efficient scheduling.

Application of Machine Learning to Predict Patient No-Shows in an Academic Pediatric Ophthalmology Clinic

Patient "no-shows" are missed appointments resulting in clinical inefficiencies, revenue loss, and discontinuity of care. Using secondary electronic health record (EHR) data, we used machine learning to predict patient no-shows in follow-up and new patient visits in pediatric ophthalmology and to evaluate features for importance. The best model, XGBoost, had an area under the receiver operating characteristics curve (AUC) score of 0.90 for predicting no-shows in follow-up visits. The key findings from this study are: (1) secondary use of EHR data can be used to build datasets for predictive modeling and successfully predict patient no-shows in pediatric ophthalmology, (2) models predicting no-shows for follow-up visits are more accurate than those for new patient visits, and (3) the performance of predictive models is more robust in predicting no-shows compared to individual important features. We hope these models will be used for more effective interventions to mitigate the impact ofpatient no-shows.

<p>Who Misses Appointments Made Online? Retrospective Analysis of the Outpatient Department of a General Hospital in Jinan, Shandong Province, China</p>

Purpose

Missed appointments in outpatient registration pose challenges for hospital administrators, especially in the context of China’s shortage of medical resources. Previous studies have identified factors that affect healthcare access via traditional appointment systems. Few studies, however, have specifically investigated Internet appointment systems. Therefore, this study explored the key factors related to missed appointments made on the Internet appointment system of a general hospital in Jinan, Shandong Province.

Methods

Online appointment data were collected from the outpatient department of a general hospital in Jinan from September 2017 to February 2018. Logistic regression was used to analyze the relative importance of eight variables: gender, age, interval between scheduling and appointment, day of the week, physician’s academic rank, appointment fee, previous missed appointments, and clinical department.

Results

A total of 48,777 online appointment records were collected, which included a 15% no-show rate. The key factors associated with no-shows included age, interval between scheduling and appointment, previous missed appointments, and clinical department. No significant relationships were found between no-shows and gender, day of the week, and appointment fee.

Conclusion

No-show rates were influenced by many factors. Based on this study’s findings, targeted measures can be taken to decrease no-show frequency and improve medical efficiency.

Patient No-Show Prediction: A Systematic Literature Review

Nowadays, across the most important problems faced by health centers are those caused by the existence of patients who do not attend their appointments. Among others, these patients cause loss of revenue to the health centers and increase the patients’ waiting list. In order to tackle these problems, several scheduling systems have been developed. Many of them require predicting whether a patient will show up for an appointment. However, obtaining these estimates accurately is currently a challenging problem. In this work, a systematic review of the literature on predicting patient no-shows is conducted aiming at establishing the current state-of-the-art. Based on a systematic review following the PRISMA methodology, 50 articles were found and analyzed. Of these articles, 82% were published in the last 10 years and the most used technique was logistic regression. In addition, there is significant growth in the size of the databases used to build the classifiers. An important finding is that only two studies achieved an accuracy higher than the show rate. Moreover, a single study attained an area under the curve greater than the 0.9 value. These facts indicate the difficulty of this problem and the need for further research.