Evaluation of Machine-Learning Algorithms for Predicting Opioid Overdose Risk Among Medicare Beneficiaries With Opioid Prescriptions

Prediction of sustained opioid use in children and adolescents using machine learning

BACKGROUND

Opioid misuse in the paediatric population is understudied. This study aimed to develop a machine learning classifier to differentiate between occasional and sustained opioid users among children and adolescents in outpatient settings.

METHODS

Data for 29,335 patients under 19 yr with recorded opioid purchases were collected from medical records. Machine learning methods were applied to predict sustained opioid use within 1, 2, or 3 yr after first opioid use, using sociodemographic information, medical history, and healthcare usage variables collected near the time of first prescription fulfilment. The models' performance was evaluated with classification and calibration metrics, and a decision curve analysis. An online tool was deployed for model self-exploration and visualisation.

RESULTS

The models demonstrated good performance, with a 1-yr follow-up model achieving a sensitivity of 0.772, a specificity of 0.703, and an ROC-AUC of 0.792 on an independent test set, with calibration intercept and slope of 0.00 and 1.02, respectively. Decision curve analysis revealed the clinical benefit of using the model relative to other strategies. SHAP analysis (SHapley Additive exPlanations) identified influential variables, including the number of diagnoses, medical images, laboratory tests, and type of opioid used.

CONCLUSIONS

Our model showed promising performance in predicting sustained opioid use among paediatric patients. The online risk prediction tool can facilitate compliance to such tools by clinicians. This study presents the potential of machine learning in identifying at-risk paediatric populations for sustained opioid use, potentially contributing to secondary prevention of opioid abuse.

Deep learning predicts postoperative opioids refills in a multi-institutional cohort of surgical patients

BACKGROUND

To combat the opioid epidemic, several strategies were implemented to limit the unnecessary prescription of opioids in the postoperative period. However, this leaves a subset of patients who genuinely require additional opioids with inadequate pain control. Deep learning models are powerful tools with great potential of optimizing health care delivery through a patient-centered focus. We sought to investigate whether deep learning models can be used to predict patients who would require additional opioid prescription refills in the postoperative period after elective surgery.

METHODS

This is a retrospective study of patients who received elective surgical intervention at the Mayo Clinic. Adult English-speaking patients ≥18 years old, who underwent an elective surgical procedure between 2013 and 2019, were eligible for inclusion. Machine learning models, including deep learning, random forest, and eXtreme Gradient Boosting, were designed to predict patients who require opioid refills after discharge from hospital.

RESULTS

A total of 9,731 patients with mean age of 62.1 years (51.4% female) were included in the study. Deep learning and random forest models predicted patients who required opioid refills with high accuracy, 0.79 ± 0.07 and 0.78 ± 0.08, respectively. Procedure performed, highest pain score recorded during hospitalization, and total oral morphine milligram equivalents prescribed at discharge were the top 3 predictors for requiring opioid refills after discharge.

CONCLUSION

Deep learning models can be used to predict patients who require postoperative opioid prescription refills with high accuracy. Other machine learning models, such as random forest, can perform equal to deep learning, increasing the applicability of machine learning for combating the opioid epidemic.

An explainable machine learning framework for predicting the risk of buprenorphine treatment discontinuation for opioid use disorder among commercially insured individuals

OBJECTIVES

Buprenorphine is an effective evidence-based medication for opioid use disorder (OUD). Yet premature discontinuation undermines treatment effectiveness, increasing the risk of mortality and overdose. We developed and evaluated a machine learning (ML) framework for predicting buprenorphine care discontinuity within 12 months following treatment initiation.

METHODS

This retrospective study used United States (US) 2018-2021 MarketScan commercial claims data of insured individuals aged 18-64 who initiated buprenorphine between July 2018 and December 2020 with no buprenorphine prescriptions in the previous six months. We measured buprenorphine prescription discontinuation gaps of ≥30 days within 12 months of initiating treatment. We developed predictive models employing logistic regression, decision tree classifier, random forest, extreme gradient boosting, Adaboost, and random forest-extreme gradient boosting ensemble. We applied recursive feature elimination with cross-validation to reduce dimensionality and identify the most predictive features while maintaining model robustness. For model validation, we used several statistics to evaluate performance, such as C-statistics and precision-recall curves. We focused on two distinct treatment stages: at the time of treatment initiation and one and three months after treatment initiation. We employed SHapley Additive exPlanations (SHAP) analysis that helped us explain the contributions of different features in predicting buprenorphine discontinuation. We stratified patients into risk subgroups based on their predicted likelihood of treatment discontinuation, dividing them into decile subgroups. Additionally, we used a calibration plot to analyze the reliability of the models.

RESULTS

A total of 30,373 patients initiated buprenorphine and 14.98% (4551) discontinued treatment. C-statistic varied between 0.56 and 0.76 for the first-stage models including patient-level demographic and clinical variables. Inclusion of proportion of days covered (PDC) measured after one month and three months following treatment initiation significantly increased the models' discriminative power (C-statistics: 0.60 to 0.82). Random forest (C-statistics: 0.76, 0.79 and 0.82 with baseline predictors, one-month PDC and three-months PDC, respectively) outperformed other ML models in discriminative performance in all stages (C-statistics: 0.56 to 0.77). Most influential risk factors of discontinuation included early stage medication adherence, age, and initial days of supply.

CONCLUSION

ML algorithms demonstrated a good discriminative power in identifying patients at higher risk of buprenorphine care discontinuity. The proposed framework may help healthcare providers optimize treatment strategies and deliver targeted interventions to improve buprenorphine care continuity.

Development and validation of machine-learning algorithms predicting retention, overdoses, and all-cause mortality among US military veterans treated with buprenorphine for opioid use disorder

BACKGROUND

Buprenorphine for opioid use disorder (B-MOUD) is essential to improving patient outcomes; however, retention is essential.

OBJECTIVE

To develop and validate machine-learning algorithms predicting retention, overdoses, and all-cause mortality among US military veterans initiating B-MOUD.

METHODS

Veterans initiating B-MOUD from fiscal years 2006-2020 were identified. Veterans' B-MOUD episodes were randomly divided into training (80%;n = 45,238) and testing samples (20%;n = 11,309). Candidate algorithms [multiple logistic regression, least absolute shrinkage and selection operator regression, random forest (RF), gradient boosting machine (GBM), and deep neural network (DNN)] were used to build and validate classification models to predict six binary outcomes: 1) B-MOUD retention, 2) any overdose, 3) opioid-related overdose, 4) overdose death, 5) opioid overdose death, and 6) all-cause mortality. Model performance was assessed using standard classification statistics [e.g., area under the receiver operating characteristic curve (AUC-ROC)].

RESULTS

Episodes in the training sample were 93.0% male, 78.0% White, 72.3% unemployed, and 48.3% had a concurrent drug use disorder. The GBM model slightly outperformed others in predicting B-MOUD retention (AUC-ROC = 0.72). RF models outperformed others in predicting any overdose (AUC-ROC = 0.77) and opioid overdose (AUC-ROC = 0.77). RF and GBM outperformed other models for overdose death (AUC-ROC = 0.74 for both), and RF and DNN outperformed other models for opioid overdose death (RF AUC-ROC = 0.79; DNN AUC-ROC = 0.78). RF and GBM also outperformed other models for all-cause mortality (AUC-ROC = 0.76 for both). No single predictor accounted for >3% of the model's variance.

CONCLUSIONS

Machine-learning algorithms can accurately predict OUD-related outcomes with moderate predictive performance; however, prediction of these outcomes is driven by many characteristics.

Advancing precision rheumatology: applications of machine learning for rheumatoid arthritis management

Rheumatoid arthritis (RA) is an autoimmune disease causing progressive joint damage. Early diagnosis and treatment is critical, but remains challenging due to RA complexity and heterogeneity. Machine learning (ML) techniques may enhance RA management by identifying patterns within multidimensional biomedical data to improve classification, diagnosis, and treatment predictions. In this review, we summarize the applications of ML for RA management. Emerging studies or applications have developed diagnostic and predictive models for RA that utilize a variety of data modalities, including electronic health records, imaging, and multi-omics data. High-performance supervised learning models have demonstrated an Area Under the Curve (AUC) exceeding 0.85, which is used for identifying RA patients and predicting treatment responses. Unsupervised learning has revealed potential RA subtypes. Ongoing research is integrating multimodal data with deep learning to further improve performance. However, key challenges remain regarding model overfitting, generalizability, validation in clinical settings, and interpretability. Small sample sizes and lack of diverse population testing risks overestimating model performance. Prospective studies evaluating real-world clinical utility are lacking. Enhancing model interpretability is critical for clinician acceptance. In summary, while ML shows promise for transforming RA management through earlier diagnosis and optimized treatment, larger scale multisite data, prospective clinical validation of interpretable models, and testing across diverse populations is still needed. As these gaps are addressed, ML may pave the way towards precision medicine in RA.

A Machine Learning Application to Classify Patients at Differing Levels of Risk of Opioid Use Disorder: Clinician-Based Validation Study

Background

Despite restrictive opioid management guidelines, opioid use disorder (OUD) remains a major public health concern. Machine learning (ML) offers a promising avenue for identifying and alerting clinicians about OUD, thus supporting better clinical decision-making regarding treatment.

Objective

This study aimed to assess the clinical validity of an ML application designed to identify and alert clinicians of different levels of OUD risk by comparing it to a structured review of medical records by clinicians.

Methods

The ML application generated OUD risk alerts on outpatient data for 649,504 patients from 2 medical centers between 2010 and 2013. A random sample of 60 patients was selected from 3 OUD risk level categories (n=180). An OUD risk classification scheme and standardized data extraction tool were developed to evaluate the validity of the alerts. Clinicians independently conducted a systematic and structured review of medical records and reached a consensus on a patient's OUD risk level, which was then compared to the ML application's risk assignments.

Results

A total of 78,587 patients without cancer with at least 1 opioid prescription were identified as follows: not high risk (n=50,405, 64.1%), high risk (n=16,636, 21.2%), and suspected OUD or OUD (n=11,546, 14.7%). The sample of 180 patients was representative of the total population in terms of age, sex, and race. The interrater reliability between the ML application and clinicians had a weighted kappa coefficient of 0.62 (95% CI 0.53-0.71), indicating good agreement. Combining the high risk and suspected OUD or OUD categories and using the review of medical records as a gold standard, the ML application had a corrected sensitivity of 56.6% (95% CI 48.7%-64.5%) and a corrected specificity of 94.2% (95% CI 90.3%-98.1%). The positive and negative predictive values were 93.3% (95% CI 88.2%-96.3%) and 60.0% (95% CI 50.4%-68.9%), respectively. Key themes for disagreements between the ML application and clinician reviews were identified.

Conclusions

A systematic comparison was conducted between an ML application and clinicians for identifying OUD risk. The ML application generated clinically valid and useful alerts about patients' different OUD risk levels. ML applications hold promise for identifying patients at differing levels of OUD risk and will likely complement traditional rule-based approaches to generating alerts about opioid safety issues.

Using decision tree models and comprehensive statewide data to predict opioid overdoses following prison release

PURPOSE

Identifying predictors of opioid overdose following release from prison is critical for opioid overdose prevention.

METHODS

We leveraged an individually linked, state-wide database from 2015-2020 to predict the risk of opioid overdose within 90 days of release from Massachusetts state prisons. We developed two decision tree modeling schemes: a model fit on all individuals with a single weight for those that experienced an opioid overdose and models stratified by race/ethnicity. We compared the performance of each model using several performance measures and identified factors that were most predictive of opioid overdose within racial/ethnic groups and across models.

RESULTS

We found that out of 44,246 prison releases in Massachusetts between 2015-2020, 2237 (5.1%) resulted in opioid overdose in the 90 days following release. The performance of the two predictive models varied. The single weight model had high sensitivity (79%) and low specificity (56%) for predicting opioid overdose and was more sensitive for White non-Hispanic individuals (sensitivity = 84%) than for racial/ethnic minority individuals.

CONCLUSIONS

Stratified models had better balanced performance metrics for both White non-Hispanic and racial/ethnic minority groups and identified different predictors of overdose between racial/ethnic groups. Across racial/ethnic groups and models, involuntary commitment (involuntary treatment for alcohol/substance use disorder) was an important predictor of opioid overdose.

Prediction of naloxone dose in opioids toxicity based on machine learning techniques (artificial intelligence)

BACKGROUND

Treatment management for opioid poisoning is critical and, at the same time, requires specialized knowledge and skills. This study was designed to develop and evaluate machine learning algorithms for predicting the maintenance dose and duration of hospital stay in opioid poisoning, in order to facilitate appropriate clinical decision-making.

METHOD AND RESULTS

This study used artificial intelligence technology to predict the maintenance dose and duration of administration by selecting clinical and paraclinical features that were selected by Pearson correlation (filter method) (Stage 1) and then the (wrapper method) Recursive Feature Elimination Cross-Validated (RFECV) (Stage2). The duration of administration was divided into two categories: A (which includes a duration of less than or equal to 24 h of infusion) and B (more than 24 h of naloxone infusion). XGBoost algorithm model with an accuracy rate of 91.04%, a prediction rate of 91.34%, and a sensitivity rate of 91.04% and area under the Curve (AUC) 0.97 was best model for classification patients. Also, the best maintenance dose of naloxone was obtained with XGBoost algorithm with R2 = 0.678. Based on the selected algorithm, the most important features for classifying patients for the duration of treatment were bicarbonate, respiration rate, physical sign, The partial pressure of carbon dioxide (PCO2), diastolic blood pressure, pulse rate, naloxone bolus dose, Blood Creatinine(Cr), Body temperature (T). The most important characteristics for determining the maintenance dose of naloxone were physical signs, bolus dose of 4.5 mg/kg, Glasgow Coma Scale (GCS), Creatine Phosphokinase (CPK) and intensive care unit (ICU) add.

CONCLUSION

A predictive model can significantly enhance the decision-making and clinical care provided by emergency physicians in hospitals and medical settings. XGBoost was found to be the superior model.

Predicting Postoperative Pain and Opioid Use with Machine Learning Applied to Longitudinal Electronic Health Record and Wearable Data

BACKGROUND

 Managing acute postoperative pain and minimizing chronic opioid use are crucial for patient recovery and long-term well-being.

OBJECTIVES

 This study explored using preoperative electronic health record (EHR) and wearable device data for machine-learning models that predict postoperative acute pain and chronic opioid use.

METHODS

 The study cohort consisted of approximately 347 All of Us Research Program participants who underwent one of eight surgical procedures and shared EHR and wearable device data. We developed four machine learning models and used the Shapley additive explanations (SHAP) technique to identify the most relevant predictors of acute pain and chronic opioid use.

RESULTS

 The stacking ensemble model achieved the highest accuracy in predicting acute pain (0.68) and chronic opioid use (0.89). The area under the curve score for severe pain versus other pain was highest (0.88) when predicting acute postoperative pain. Values of logistic regression, random forest, extreme gradient boosting, and stacking ensemble ranged from 0.74 to 0.90 when predicting postoperative chronic opioid use. Variables from wearable devices played a prominent role in predicting both outcomes.

CONCLUSION

 SHAP detection of individual risk factors for severe pain can help health care providers tailor pain management plans. Accurate prediction of postoperative chronic opioid use before surgery can help mitigate the risk for the outcomes we studied. Prediction can also reduce the chances of opioid overuse and dependence. Such mitigation can promote safer and more effective pain control for patients during their recovery.

Development and validation of a medication-based risk prediction model for acute kidney injury in older outpatients

BACKGROUND

Older adults are at an increased risk of acute kidney injury (AKI), particularly in community settings, often due to medications. Effective prevention hinges on identifying high-risk patients, yet existing models for predicting AKI risk in older outpatients are scarce, particularly those incorporating medication variables. We aimed to develop an AKI risk prediction model that included medication-related variables for older outpatients.

METHODS

We constructed a cohort of 2,272,257 outpatients aged ≥65 years using a national claims database. This cohort was split into a development (70%) and validation (30%) groups. Our primary goal was to identify newly diagnosed AKI within one month of cohort entry in an outpatient context. We screened 170 variables and developed a risk prediction model using logistic regression.

RESULTS

The final model integrated 12 variables: 2 demographic, 4 comorbid, and 6 medication-related. It showed good performance with acceptable calibration. In the validation cohort, the area under the receiver operating characteristic curve value was 0.720 (95% confidence interval, 0.692-0.748). Sensitivity and specificity were 69.9% and 61.9%, respectively. Notably, the model identified high-risk patients as having a 27-fold increased AKI risk compared with low-risk individuals.

CONCLUSION

We have developed a new AKI risk prediction model for older outpatients, incorporating critical medication-related variables with good discrimination. This tool may be useful in identifying and targeting patients who may require interventions to prevent AKI in an outpatient setting.

Predictive Models to Assess Risk of Persistent Opioid Use, Opioid Use Disorder, and Overdose

BACKGROUND

This systematic review summarizes the development, accuracy, quality, and clinical utility of predictive models to assess the risk of opioid use disorder (OUD), persistent opioid use, and opioid overdose.

METHODS

In accordance with Preferred Reporting Items for a Systematic Review and Meta-analysis guidelines, 8 electronic databases were searched for studies on predictive models and OUD, overdose, or persistent use in adults until June 25, 2023. Study selection and data extraction were completed independently by 2 reviewers. Risk of bias of included studies was assessed independently by 2 reviewers using the Prediction model Risk of Bias ASsessment Tool (PROBAST).

RESULTS

The literature search yielded 3130 reports; after removing 199 duplicates, excluding 2685 studies after abstract review, and excluding 204 studies after full-text review, the final sample consisted of 41 studies that developed more than 160 predictive models. Primary outcomes included opioid overdose (31.6% of studies), OUD (41.4%), and persistent opioid use (17%). The most common modeling approach was regression modeling, and the most common predictors included age, sex, mental health diagnosis history, and substance use disorder history. Most studies reported model performance via the c statistic, ranging from 0.507 to 0.959; gradient boosting tree models and neural network models performed well in the context of their own study. One study deployed a model in real time. Risk of bias was predominantly high; concerns regarding applicability were predominantly low.

CONCLUSIONS

Models to predict opioid-related risks are developed using diverse data sources and predictors, with a wide and heterogenous range of accuracy metrics. There is a need for further research to improve their accuracy and implementation.

Association between trajectories of prescription opioid use and risk of opioid use disorder and overdose among US nonmetastatic breast cancer survivors

PURPOSE

To examine the association between prescription opioid use trajectories and risk of opioid use disorder (OUD) or overdose among nonmetastatic breast cancer survivors by treatment type.

METHODS

This retrospective cohort study included female nonmetastatic breast cancer survivors with at least 1 opioid prescription fill in 2010-2019 Surveillance, Epidemiology and End Results linked Medicare data. Opioid mean daily morphine milligram equivalents (MME) calculated within 1.5 years after initiating active breast cancer therapy. Group-based trajectory models identified distinct opioid use trajectory patterns. Risk of time to first OUD/overdose event within 1 year after the trajectory period was calculated for distinct trajectory groups using Cox proportional hazards models. Analyses were stratified by treatment type.

RESULTS

Four opioid use trajectories were identified for each treatment group. For 38,030 survivors with systemic endocrine therapy, 3 trajectories were associated with increased OUD/overdose risk compared with early discontinuation: minimal dose (< 5 MME; adjusted hazard ratio [aHR] = 1.73 [95% CI  1.43-2.09]), very low dose (5-25 MME; 2.67 [2.05-3.48]), and moderate dose (51-90 MME; 6.20 [4.69-8.19]). For 9477 survivors with adjuvant chemotherapy, low-dose opioid use was associated with higher OUD/overdose risk (aHR = 7.33 [95% CI 2.52-21.31]) compared with early discontinuation. For 3513 survivors with neoadjuvant chemotherapy, the differences in OUD/OD risks across the 4 trajectories were not significant.

CONCLUSIONS

Among Medicare nonmetastatic breast cancer survivors receiving systemic endocrine therapy or adjuvant chemotherapy, compared with early discontinuation, low-dose or moderate-dose opioid use were associated with six- to sevenfold higher OUD/overdose risk. Breast cancer survivors at high-risk of OUD/overdose may benefit from targeted interventions (e.g., pain clinic referral).

Evaluating methods for risk prediction of Covid-19 mortality in nursing home residents before and after vaccine availability: a retrospective cohort study

BACKGROUND

SARS-CoV-2 vaccines are effective in reducing hospitalization, COVID-19 symptoms, and COVID-19 mortality for nursing home (NH) residents. We sought to compare the accuracy of various machine learning models, examine changes to model performance, and identify resident characteristics that have the strongest associations with 30-day COVID-19 mortality, before and after vaccine availability.

METHODS

We conducted a population-based retrospective cohort study analyzing data from all NH facilities across Ontario, Canada. We included all residents diagnosed with SARS-CoV-2 and living in NHs between March 2020 and July 2021. We employed five machine learning algorithms to predict COVID-19 mortality, including logistic regression, LASSO regression, classification and regression trees (CART), random forests, and gradient boosted trees. The discriminative performance of the models was evaluated using the area under the receiver operating characteristic curve (AUC) for each model using 10-fold cross-validation. Model calibration was determined through evaluation of calibration slopes. Variable importance was calculated by repeatedly and randomly permutating the values of each predictor in the dataset and re-evaluating the model's performance.

RESULTS

A total of 14,977 NH residents and 20 resident characteristics were included in the model. The cross-validated AUCs were similar across algorithms and ranged from 0.64 to 0.67. Gradient boosted trees and logistic regression had an AUC of 0.67 pre- and post-vaccine availability. CART had the lowest discrimination ability with an AUC of 0.64 pre-vaccine availability, and 0.65 post-vaccine availability. The most influential resident characteristics, irrespective of vaccine availability, included advanced age (≥ 75 years), health instability, functional and cognitive status, sex (male), and polypharmacy.

CONCLUSIONS

The predictive accuracy and discrimination exhibited by all five examined machine learning algorithms were similar. Both logistic regression and gradient boosted trees exhibit comparable performance and display slight superiority over other machine learning algorithms. We observed consistent model performance both before and after vaccine availability. The influence of resident characteristics on COVID-19 mortality remained consistent across time periods, suggesting that changes to pre-vaccination screening practices for high-risk individuals are effective in the post-vaccination era.

PROVIDENT: Development and Validation of a Machine Learning Model to Predict Neighborhood-level Overdose Risk in Rhode Island

BACKGROUND

Drug overdose persists as a leading cause of death in the United States, but resources to address it remain limited. As a result, health authorities must consider where to allocate scarce resources within their jurisdictions. Machine learning offers a strategy to identify areas with increased future overdose risk to proactively allocate overdose prevention resources. This modeling study is embedded in a randomized trial to measure the effect of proactive resource allocation on statewide overdose rates in Rhode Island (RI).

METHODS

We used statewide data from RI from 2016 to 2020 to develop an ensemble machine learning model predicting neighborhood-level fatal overdose risk. Our ensemble model integrated gradient boosting machine and super learner base models in a moving window framework to make predictions in 6-month intervals. Our performance target, developed a priori with the RI Department of Health, was to identify the 20% of RI neighborhoods containing at least 40% of statewide overdose deaths, including at least one neighborhood per municipality. The model was validated after trial launch.

RESULTS

Our model selected priority neighborhoods capturing 40.2% of statewide overdose deaths during the test periods and 44.1% of statewide overdose deaths during validation periods. Our ensemble outperformed the base models during the test periods and performed comparably to the best-performing base model during the validation periods.

CONCLUSIONS

We demonstrated the capacity for machine learning models to predict neighborhood-level fatal overdose risk to a degree of accuracy suitable for practitioners. Jurisdictions may consider predictive modeling as a tool to guide allocation of scarce resources.

Leveraging artificial intelligence to advance implementation science: potential opportunities and cautions

BACKGROUND

The field of implementation science was developed to address the significant time delay between establishing an evidence-based practice and its widespread use. Although implementation science has contributed much toward bridging this gap, the evidence-to-practice chasm remains a challenge. There are some key aspects of implementation science in which advances are needed, including speed and assessing causality and mechanisms. The increasing availability of artificial intelligence applications offers opportunities to help address specific issues faced by the field of implementation science and expand its methods.

MAIN TEXT

This paper discusses the many ways artificial intelligence can address key challenges in applying implementation science methods while also considering potential pitfalls to the use of artificial intelligence. We answer the questions of "why" the field of implementation science should consider artificial intelligence, for "what" (the purpose and methods), and the "what" (consequences and challenges). We describe specific ways artificial intelligence can address implementation science challenges related to (1) speed, (2) sustainability, (3) equity, (4) generalizability, (5) assessing context and context-outcome relationships, and (6) assessing causality and mechanisms. Examples are provided from global health systems, public health, and precision health that illustrate both potential advantages and hazards of integrating artificial intelligence applications into implementation science methods. We conclude by providing recommendations and resources for implementation researchers and practitioners to leverage artificial intelligence in their work responsibly.

CONCLUSIONS

Artificial intelligence holds promise to advance implementation science methods ("why") and accelerate its goals of closing the evidence-to-practice gap ("purpose"). However, evaluation of artificial intelligence's potential unintended consequences must be considered and proactively monitored. Given the technical nature of artificial intelligence applications as well as their potential impact on the field, transdisciplinary collaboration is needed and may suggest the need for a subset of implementation scientists cross-trained in both fields to ensure artificial intelligence is used optimally and ethically.

Impact of 30-day prescribed opioid dose trajectory on fatal overdose risk: A population-based, statewide cohort study

BACKGROUND

Both increases and decreases in patients' prescribed daily opioid dose have been linked to increased overdose risk, but associations between 30-day dose trajectories and subsequent overdose risk have not been systematically examined.

OBJECTIVE

To examine the associations between 30-day prescribed opioid dose trajectories and fatal opioid overdose risk during the subsequent 15 days.

DESIGN

Statewide cohort study using linked prescription drug monitoring program and death certificate data. We constructed a multivariable Cox proportional hazards model that accounted for time-varying prescription-, prescriber-, and pharmacy-level factors.

PARTICIPANTS

All patients prescribed an opioid analgesic in California from March to December, 2013 (5,326,392 patients).

MAIN MEASURES

Dependent variable: fatal drug overdose involving opioids. Primary independent variable: a 16-level variable denoting all possible opioid dose trajectories using the following categories for current and 30-day previously prescribed daily dose: 0-29, 30-59, 60-89, or ≥90 milligram morphine equivalents (MME).

KEY RESULTS

Relative to patients prescribed a stable daily dose of 0-29 MME, large (≥2 categories) dose increases and having a previous or current dose ≥60 MME per day were associated with significantly greater 15-day overdose risk. Patients whose dose decreased from ≥90 to 0-29 MME per day had significantly greater overdose risk compared to both patients prescribed a stable daily dose of ≥90 MME (aHR 3.56, 95%CI 2.24-5.67) and to patients prescribed a stable daily dose of 0-29 MME (aHR 7.87, 95%CI 5.49-11.28). Patients prescribed benzodiazepines also had significantly greater overdose risk; being prescribed Z-drugs, carisoprodol, or psychostimulants was not associated with overdose risk.

CONCLUSIONS

Large (≥2 categories) 30-day dose increases and decreases were both associated with increased risk of fatal opioid overdose, particularly for patients taking ≥90 MME whose opioids were abruptly stopped. Results align with 2022 CDC guidelines that urge caution when reducing opioid doses for patients taking long-term opioid for chronic pain.

Drug overdose risk with benzodiazepine treatment in young adults: Comparative analysis in privately and publicly insured individuals

BACKGROUND AND AIMS

Benzodiazepines (BZDs) carry a risk for drug overdose and are prescribed alone or simultaneously with selective-serotonin reuptake inhibitors (SSRIs) for the treatment of anxiety and depression in young adults. We aimed to measure risks of drug overdose following BZD treatment initiation, and simultaneous BZD and SSRI initiation, compared with SSRI treatment alone in young adults with depression or anxiety.

DESIGN, SETTING, PARTICIPANTS

The cohort study used administrative databases covering privately (MarketScan, 1/1/2009-12/31/2018) and publicly (Medicaid, 1/1/2015-12/31/2016) insured young adults (18-29 years) in the United States. Those with depression or anxiety diagnoses newly initiating BZD or SSRI treatment (without BZD or SSRI prescriptions in prior year) were included. Simultaneous "BZD + SSRI" initiation was defined as starting BZD and SSRI treatment on the same day. The cohorts included 604 664 privately insured young adults (BZD = 22%, BZD + SSRI = 10%, SSRI = 68%) and 110 493 publicly insured young adults (BZD = 23%, BZD + SSRI = 5%, SSRI = 72%).

MEASUREMENTS

Incident medically treated drug overdose events were identified from emergency department and inpatient encounters (ICD poisoning codes) within 6 months of treatment initiation. Crude and propensity-score adjusted cumulative incidence and hazard ratios (HR) were estimated. Sub-analyses evaluated drug overdose intent.

FINDINGS

Adjusted HRs of drug overdose for BZD vs. SSRI treatment was 1.36 (95% confidence interval [CI]:1.23-1.51) in privately and 1.59 (95%CI:1.37-1.83) in publicly insured young adults. The adjusted HRs of drug overdose for BZD + SSRI treatment vs. SSRI treatment were 1.99 (95%CI:1.77-2.25) in privately and 1.98 (95%CI:1.47-2.68) in publicly insured young adults.

CONCLUSIONS

Among young adults in the United States, initiating benzodiazepine treatment for anxiety and depression, alone or simultaneously with selective-serotonin reuptake inhibitors (SSRI), appears to have an increased risk of medically treated drug overdose compared with SSRI treatment alone. These associations were observed in publicly and privately insured individuals.

Ensemble Machine Learning Model Incorporating Radiomics and Body Composition for Predicting Intraoperative HDI in PPGL

CONTEXT

Intraoperative hemodynamic instability (HDI) can lead to cardiovascular and cerebrovascular complications during surgery for pheochromocytoma/paraganglioma (PPGL).

OBJECTIVES

We aimed to assess the risk of intraoperative HDI in patients with PPGL to improve surgical outcome.

METHODS

A total of 199 consecutive patients with PPGL confirmed by surgical pathology were retrospectively included in this study. This cohort was separated into 2 groups according to intraoperative systolic blood pressure, the HDI group (n = 101) and the hemodynamic stability (HDS) group (n = 98). It was also divided into 2 subcohorts for predictive modeling: the training cohort (n = 140) and the validation cohort (n = 59). Prediction models were developed with both the ensemble machine learning method (EL model) and the multivariate logistic regression model using body composition parameters on computed tomography, tumor radiomics, and clinical data. The efficiency of the models was evaluated with discrimination, calibration, and decision curves.

RESULTS

The EL model showed good discrimination between the HDI group and HDS group, with an area under the curve of (AUC) of 96.2% (95% CI, 93.5%-99.0%) in the training cohort, and an AUC of 93.7% (95% CI, 88.0%-99.4%) in the validation cohort. The AUC values from the EL model were significantly higher than the logistic regression model, which had an AUC of 74.4% (95% CI, 66.1%-82.6%) in the training cohort and an AUC of 74.2% (95% CI, 61.1%-87.3%) in the validation cohort. Favorable calibration performance and clinical applicability of the EL model were observed.

CONCLUSION

The EL model combining preoperative computed tomography-based body composition, tumor radiomics, and clinical data could potentially help predict intraoperative HDI in patients with PPGL.

Development and validation of an artificial neural network prediction model for postpartum hemorrhage with placenta previa

BACKGROUND

Postpartum hemorrhage (PPH) is a leading cause of maternal morbidity worldwide and placenta previa is one of the major risk factors for PPH in overall population. However, the clinical prediction of PPH remains challenging. This study aimed to investigate an ideal machine learning-based prediction model for PPH in placenta previa parturients with cesarean section.

METHODS

The clinical data of 223 placenta previa parturients who underwent cesarean delivery in our hospital from 2016 to 2019 were retrospectively collected for analysis. An artificial neural network model was designed to predict PPH, defined as blood loss exceeding 1000 mL with 24h after delivery. Twenty clinical variables were selected as predictors. We also applied six conventional machine learning methods as reference models, including support vector machine, decision tree, random forest, gradient boosting decision tree, adaboost and logistic regression. All the models were validated using 5-fold cross-validation. The area under the receiver operating characteristic curve (AUC), precision, recall and the prediction accuracy of each model were reported.

RESULTS

A total of 223 pregnant women were enrolled in this study, including 101 cases (45.29%) experienced PPH. The proposed model achieved superior prediction performance with an AUC of 0.917, an accuracy of 0.851, a precision score of 0.829 and a recall score of 0.851, which outperformed other six conventional machine learning methods.

CONCLUSIONS

Compared to the conventional machine learning approaches, artificial neural network model shows discriminative ability in identifying women's risk of PPH with placenta previa during cesarean section.

Predictors of fatal and nonfatal overdose after prescription of opioids for chronic pain: a systematic review and meta-analysis of observational studies

BACKGROUND

Higher doses of opioids, mental health comorbidities, co-prescription of sedatives, lower socioeconomic status and a history of opioid overdose have been reported as risk factors for opioid overdose; however, the magnitude of these associations and their credibility are unclear. We sought to identify predictors of fatal and nonfatal overdose from prescription opioids.

METHODS

We systematically searched MEDLINE, Embase, CINAHL, PsycINFO and Web of Science up to Oct. 30, 2022, for observational studies that explored predictors of opioid overdose after their prescription for chronic pain. We performed random-effects meta-analyses for all predictors reported by 2 or more studies using odds ratios (ORs) and 95% confidence intervals (CIs).

RESULTS

Twenty-eight studies (23 963 716 patients) reported the association of 103 predictors with fatal or nonfatal opioid overdose. Moderate- to high-certainty evidence supported large relative associations with history of overdose (OR 5.85, 95% CI 3.78-9.04), higher opioid dose (OR 2.57, 95% CI 2.08-3.18 per 90-mg increment), 3 or more prescribers (OR 4.68, 95% CI 3.57-6.12), 4 or more dispensing pharmacies (OR 4.92, 95% CI 4.35-5.57), prescription of fentanyl (OR 2.80, 95% CI 2.30-3.41), current substance use disorder (OR 2.62, 95% CI 2.09-3.27), any mental health diagnosis (OR 2.12, 95% CI 1.73-2.61), depression (OR 2.22, 95% CI 1.57-3.14), bipolar disorder (OR 2.07, 95% CI 1.77-2.41) or pancreatitis (OR 2.00, 95% CI 1.52-2.64), with absolute risks among patients with the predictor ranging from 2-6 per 1000 for fatal overdose and 4-12 per 1000 for nonfatal overdose.

INTERPRETATION

We identified 10 predictors that were strongly associated with opioid overdose. Awareness of these predictors may facilitate shared decision-making regarding prescribing opioids for chronic pain and inform harm-reduction strategies SYSTEMATIC REVIEW REGISTRATION: Open Science Framework (https://osf.io/vznxj/).

Translating Predictive Analytics for Public Health Practice: A Case Study of Overdose Prevention in Rhode Island

Prior applications of machine learning to population health have relied on conventional model assessment criteria, limiting the utility of models as decision support tools for public health practitioners. To facilitate practitioners' use of machine learning as a decision support tool for area-level intervention, we developed and applied 4 practice-based predictive model evaluation criteria (implementation capacity, preventive potential, health equity, and jurisdictional practicalities). We used a case study of overdose prevention in Rhode Island to illustrate how these criteria could inform public health practice and health equity promotion. We used Rhode Island overdose mortality records from January 2016-June 2020 (n = 1,408) and neighborhood-level US Census data. We employed 2 disparate machine learning models, Gaussian process and random forest, to illustrate the comparative utility of our criteria to guide interventions. Our models predicted 7.5%-36.4% of overdose deaths during the test period, illustrating the preventive potential of overdose interventions assuming 5%-20% statewide implementation capacities for neighborhood-level resource deployment. We describe the health equity implications of use of predictive modeling to guide interventions along the lines of urbanicity, racial/ethnic composition, and poverty. We then discuss considerations to complement predictive model evaluation criteria and inform the prevention and mitigation of spatially dynamic public health problems across the breadth of practice. This article is part of a Special Collection on Mental Health.

Predicting 30-day risk from benzodiazepine/Z-drug dispensations in older adults using administrative data: A prognostic machine learning approach

OBJECTIVE

To develop a machine-learning (ML) model using administrative data to estimate risk of adverse outcomes within 30-days of a benzodiazepine (BZRA) dispensation in older adults for use by health departments/regulators.

DESIGN, SETTING AND PARTICIPANTS

This study was conducted in Alberta, Canada during 2018-2019 in Albertans 65 years of age and older. Those with any history of malignancy or palliative care were excluded.

EXPOSURE

Each BZRA dispensation from a community pharmacy served as the unit of analysis.

MAIN OUTCOMES AND MEASURES

ML algorithms were developed on 2018 administrative data to predict risk of any-cause hospitalization, emergency department visit or death within 30-days of a BZRA dispensation. Validation on 2019 administrative data was done using XGBoost to evaluate discrimination, calibration and other relevant metrics on ranked predictions. Daily and quarterly predictions were simulated on 2019 data.

RESULTS

65,063 study participants were included which represented 633,333 BZRA dispensation during 2018-2019. The validation set had 314,615 dispensations linked to 55,928 all-cause outcomes representing a pre-test probability of 17.8%. C-statistic for the XGBoost model was 0.75. Measuring risk at the end of 2019, the top 0.1 percentile of predicted risk had a LR + of 40.31 translating to a post-test probability of 90%. Daily and quarterly classification simulations resulted in uninformative predictions with positive likelihood ratios less than 10 in all risk prediction categories. Previous history of admissions was ranked highest in variable importance.

CONCLUSION

Developing ML models using only administrative health data may not provide health regulators with sufficient informative predictions to use as decision aids for potential interventions, especially if considering daily or quarterly classifications of BZRA risks in older adults. ML models may be informative for this context if yearly classifications are preferred. Health regulators should have access to other types of data to improve ML prediction.

Imaging Artificial Intelligence: A Framework for Radiologists to Address Health Equity, From the AJR Special Series on DEI

Artificial intelligence (AI) holds promise for helping patients access new and individualized health care pathways while increasing efficiencies for health care practitioners. Radiology has been at the forefront of this technology in medicine; many radiology practices are implementing and trialing AI-focused products. AI also holds great promise for reducing health disparities and promoting health equity. Radiology is ideally positioned to help reduce disparities given its central and critical role in patient care. The purposes of this article are to discuss the potential benefits and pitfalls of deploying AI algorithms in radiology, specifically highlighting the impact of AI on health equity; to explore ways to mitigate drivers of inequity; and to enhance pathways for creating better health care for all individuals, centering on a practical framework that helps radiologists address health equity during deployment of new tools.

Machine-learning-based adverse drug event prediction from observational health data: A review

Adverse drug events (ADEs) are responsible for a significant number of hospital admissions and fatalities. Machine learning models have been developed to assess the individual patient risk of having an ADE. In this article, we have reviewed studies addressing the prediction of ADEs in observational health data with machine learning. The field of individualised ADE prediction is rapidly emerging through the increasing availability of additional data modalities (e.g., genetic data, screening data, wearables data) and advanced deep learning models such as transformers. Consequently, personalised adverse drug event predictions are becoming more feasible and tangible.

Update of a Multivariable Opioid Overdose Risk Prediction Model to Enhance Clinical Care for Long-term Opioid Therapy Patients

BACKGROUND

Clinical opioid overdose risk prediction models can be useful tools to reduce the risk of overdose in patients prescribed long-term opioid therapy (LTOT). However, evolving overdose risk environments and clinical practices in addition to potential harmful model misapplications require careful assessment prior to widespread implementation into clinical care. Models may need to be tailored to meet local clinical operational needs and intended applications in practice.

OBJECTIVE

To update and validate an existing opioid overdose risk model, the Kaiser Permanente Colorado Opioid Overdose (KPCOOR) Model, in patients prescribed LTOT for implementation in clinical care.

DESIGN, SETTING, AND PARTICIPANTS

The retrospective cohort study consisted of 33, 625 patients prescribed LTOT between January 2015 and June 2019 at Kaiser Permanente Colorado, with follow-up through June 2021.

MAIN MEASURES

The outcome consisted of fatal opioid overdoses identified from vital records and non-fatal opioid overdoses from emergency department and inpatient settings. Predictors included demographics, medication dispensings, substance use disorder history, mental health history, and medical diagnoses. Cox proportional hazards regressions were used to model 2-year overdose risk.

KEY RESULTS

During follow-up, 65 incident opioid overdoses were observed (111.4 overdoses per 100,000 person-years) in the study cohort, of which 11 were fatal. The optimal risk model needed to risk-stratify patients and to be easily interpreted by clinicians. The original 5-variable model re-validated on the new study cohort had a bootstrap-corrected C-statistic of 0.73 (95% CI, 0.64-0.85) compared to a C-statistic of 0.80 (95% CI, 0.70-0.88) in the updated model and 0.77 (95% CI, 0.66-0.87) in the final adapted 7-variable model, which was also well-calibrated.

CONCLUSIONS

Updating and adapting predictors for opioid overdose in the KPCOOR Model with input from clinical partners resulted in a parsimonious and clinically relevant model that was poised for integration in clinical care.

An Automatically Adaptive Digital Health Intervention to Decrease Opioid-Related Risk While Conserving Counselor Time: Quantitative Analysis of Treatment Decisions Based on Artificial Intelligence and Patient-Reported Risk Measures

BACKGROUND

Some patients prescribed opioid analgesic (OA) medications for pain experience serious side effects, including dependence, sedation, and overdose. As most patients are at low risk for OA-related harms, risk reduction interventions requiring multiple counseling sessions are impractical on a large scale.

OBJECTIVE

This study evaluates whether an intervention based on reinforcement learning (RL), a field of artificial intelligence, learned through experience to personalize interactions with patients with pain discharged from the emergency department (ED) and decreased self-reported OA misuse behaviors while conserving counselors' time.

METHODS

We used data representing 2439 weekly interactions between a digital health intervention ("Prescription Opioid Wellness and Engagement Research in the ED" [PowerED]) and 228 patients with pain discharged from 2 EDs who reported recent opioid misuse. During each patient's 12 weeks of intervention, PowerED used RL to select from 3 treatment options: a brief motivational message delivered via an interactive voice response (IVR) call, a longer motivational IVR call, or a live call from a counselor. The algorithm selected session types for each patient each week, with the goal of minimizing OA risk, defined in terms of a dynamic score reflecting patient reports during IVR monitoring calls. When a live counseling call was predicted to have a similar impact on future risk as an IVR message, the algorithm favored IVR to conserve counselor time. We used logit models to estimate changes in the relative frequency of each session type as PowerED gained experience. Poisson regression was used to examine the changes in self-reported OA risk scores over calendar time, controlling for the ordinal session number (1st to 12th).

RESULTS

Participants on average were 40 (SD 12.7) years of age; 66.7% (152/228) were women and 51.3% (117/228) were unemployed. Most participants (175/228, 76.8%) reported chronic pain, and 46.2% (104/225) had moderate to severe depressive symptoms. As PowerED gained experience through interactions over a period of 142 weeks, it delivered fewer live counseling sessions than brief IVR sessions (P=.006) and extended IVR sessions (P<.001). Live counseling sessions were selected 33.5% of the time in the first 5 weeks of interactions (95% CI 27.4%-39.7%) but only for 16.4% of sessions (95% CI 12.7%-20%) after 125 weeks. Controlling for each patient's changes during the course of treatment, this adaptation of treatment-type allocation led to progressively greater improvements in self-reported OA risk scores (P<.001) over calendar time, as measured by the number of weeks since enrollment began. Improvement in risk behaviors over time was especially pronounced among patients with the highest risk at baseline (P=.02).

CONCLUSIONS

The RL-supported program learned which treatment modalities worked best to improve self-reported OA risk behaviors while conserving counselors' time. RL-supported interventions represent a scalable solution for patients with pain receiving OA prescriptions.

TRIAL REGISTRATION

Clinicaltrials.gov NCT02990377; https://classic.clinicaltrials.gov/ct2/show/NCT02990377.

A machine learning evaluation of patient characteristics associated with prescribing of guideline-directed medical therapy for heart failure

Introduction/background

Patients with heart failure and reduced ejection fraction (HFrEF) are consistently underprescribed guideline-directed medications. Although many barriers to prescribing are known, identification of these barriers has relied on traditional a priori hypotheses or qualitative methods. Machine learning can overcome many limitations of traditional methods to capture complex relationships in data and lead to a more comprehensive understanding of the underpinnings driving underprescribing. Here, we used machine learning methods and routinely available electronic health record data to identify predictors of prescribing.

Methods

We evaluated the predictive performance of machine learning algorithms to predict prescription of four types of medications for adults with HFrEF: angiotensin converting enzyme inhibitor/angiotensin receptor blocker (ACE/ARB), angiotensin receptor-neprilysin inhibitor (ARNI), evidence-based beta blocker (BB), or mineralocorticoid receptor antagonist (MRA). The models with the best predictive performance were used to identify the top 20 characteristics associated with prescribing each medication type. Shapley values were used to provide insight into the importance and direction of the predictor relationships with medication prescribing.

Results

For 3,832 patients meeting the inclusion criteria, 70% were prescribed an ACE/ARB, 8% an ARNI, 75% a BB, and 40% an MRA. The best-predicting model for each medication type was a random forest (area under the curve: 0.788-0.821; Brier score: 0.063-0.185). Across all medications, top predictors of prescribing included prescription of other evidence-based medications and younger age. Unique to prescribing an ARNI, the top predictors included lack of diagnoses of chronic kidney disease, chronic obstructive pulmonary disease, or hypotension, as well as being in a relationship, nontobacco use, and alcohol use.

Discussion/conclusions

We identified multiple predictors of prescribing for HFrEF medications that are being used to strategically design interventions to address barriers to prescribing and to inform further investigations. The machine learning approach used in this study to identify predictors of suboptimal prescribing can also be used by other health systems to identify and address locally relevant gaps and solutions to prescribing.

Outcome class imbalance and rare events: An underappreciated complication for overdose risk prediction modeling

BACKGROUND AND AIMS

Low outcome prevalence, often observed with opioid-related outcomes, poses an underappreciated challenge to accurate predictive modeling. Outcome class imbalance, where non-events (i.e. negative class observations) outnumber events (i.e. positive class observations) by a moderate to extreme degree, can distort measures of predictive accuracy in misleading ways, and make the overall predictive accuracy and the discriminatory ability of a predictive model appear spuriously high. We conducted a simulation study to measure the impact of outcome class imbalance on predictive performance of a simple SuperLearner ensemble model and suggest strategies for reducing that impact.

DESIGN, SETTING, PARTICIPANTS

Using a Monte Carlo design with 250 repetitions, we trained and evaluated these models on four simulated data sets with 100 000 observations each: one with perfect balance between events and non-events, and three where non-events outnumbered events by an approximate factor of 10:1, 100:1, and 1000:1, respectively.

MEASUREMENTS

We evaluated the performance of these models using a comprehensive suite of measures, including measures that are more appropriate for imbalanced data.

FINDINGS

Increasing imbalance tended to spuriously improve overall accuracy (using a high threshold to classify events vs non-events, overall accuracy improved from 0.45 with perfect balance to 0.99 with the most severe outcome class imbalance), but diminished predictive performance was evident using other metrics (corresponding positive predictive value decreased from 0.99 to 0.14).

CONCLUSION

Increasing reliance on algorithmic risk scores in consequential decision-making processes raises critical fairness and ethical concerns. This paper provides broad guidance for analytic strategies that clinical investigators can use to remedy the impacts of outcome class imbalance on risk prediction tools.

Evaluation of a non-pharmacological interdisciplinary pain rehabilitation and functional restoration program for chronic pain in veterans

OBJECTIVE

To conduct a quality improvement evaluation of the Empower Veterans Program (EVP), an interdisciplinary pain rehabilitation/functional restoration program option for functional restoration for high-impact chronic pain, offered in a large metro-area Veterans Health Administration (VHA) system.

DATA SOURCES

VHA Corporate Data Warehouse electronic medical record data for patients treated by EVP between 2015 and 2019.

EVALUATION DESIGN

This retrospective design first compared EVP patients considered engaged or not engaged in completing treatment in terms of demographic characteristics and post-treatment changes in clinical measures related to opioid use and mental health. We then compared mortality risk between matched groups of treated and untreated patients with chronic pain and concurrent opioid prescriptions using propensity score matching and Cox proportional hazards methods. "Treated" in the matched groups was defined as any level of EVP participation (i.e., both engaged and not engaged).

DATA COLLECTION/EXTRACTION METHODS

We first identified 1053 EVP patients with 1 year of pre-and post-treatment follow-time and determined their engagement level. From those with chronic pain and prescription opioids, we matched 237 EVP patients to 375 untreated patients.

PRINCIPAL FINDINGS

Engaged patients (57.4% of treated patients), were somewhat older than the non-engaged (mean age 57.1 vs. 53.7, Cohen's D = 0.30), and achieved lower mean PHQ9 depression scores in the post-treatment year (9.2 vs. 10.6, Cohen's D = 0.20). Participation in EVP was associated with a 65% lower mortality risk among Veterans with chronic pain and opioid use when compared to the untreated patients: (HR: 0.35, 95% CI: 0.17, 0.75).

CONCLUSIONS

EVP was associated with a large reduction in mortality risk for Veterans with both chronic pain and opioid use. This result could inform the decision process in a VA station or region when considering providing or expanding access to an interdisciplinary rehabilitation/functional restoration program for chronic pain.

Predicting benzodiazepine prescriptions: A proof-of-concept machine learning approach

Introduction

Benzodiazepines are the most commonly prescribed psychotropic medications, but they may place users at risk of serious adverse effects. Developing a method to predict benzodiazepine prescriptions could assist in prevention efforts.

Methods

The present study applies machine learning methods to de-identified electronic health record data, in order to develop algorithms for predicting benzodiazepine prescription receipt (yes/no) and number of benzodiazepine prescriptions (0, 1, 2+) at a given encounter. Support-vector machine (SVM) and random forest (RF) approaches were applied to outpatient psychiatry, family medicine, and geriatric medicine data from a large academic medical center. The training sample comprised encounters taking place between January 2020 and December 2021 (N = 204,723 encounters); the testing sample comprised data from encounters taking place between January and March 2022 (N = 28,631 encounters). The following empirically-supported features were evaluated: anxiety and sleep disorders (primary anxiety diagnosis, any anxiety diagnosis, primary sleep diagnosis, any sleep diagnosis), demographic characteristics (age, gender, race), medications (opioid prescription, number of opioid prescriptions, antidepressant prescription, antipsychotic prescription), other clinical variables (mood disorder, psychotic disorder, neurocognitive disorder, prescriber specialty), and insurance status (any insurance, type of insurance). We took a step-wise approach to developing a prediction model, wherein Model 1 included only anxiety and sleep diagnoses, and each subsequent model included an additional group of features.

Results

For predicting benzodiazepine prescription receipt (yes/no), all models showed good to excellent overall accuracy and area under the receiver operating characteristic curve (AUC) for both SVM (Accuracy = 0.868-0.883; AUC = 0.864-0.924) and RF (Accuracy = 0.860-0.887; AUC = 0.877-0.953). Overall accuracy was also high for predicting number of benzodiazepine prescriptions (0, 1, 2+) for both SVM (Accuracy = 0.861-0.877) and RF (Accuracy = 0.846-0.878).

Discussion

Results suggest SVM and RF algorithms can accurately classify individuals who receive a benzodiazepine prescription and can separate patients by the number of benzodiazepine prescriptions received at a given encounter. If replicated, these predictive models could inform system-level interventions to reduce the public health burden of benzodiazepines.

Opioid death projections with AI-based forecasts using social media language

Targeting of location-specific aid for the U.S. opioid epidemic is difficult due to our inability to accurately predict changes in opioid mortality across heterogeneous communities. AI-based language analyses, having recently shown promise in cross-sectional (between-community) well-being assessments, may offer a way to more accurately longitudinally predict community-level overdose mortality. Here, we develop and evaluate, TROP (Transformer for Opiod Prediction), a model for community-specific trend projection that uses community-specific social media language along with past opioid-related mortality data to predict future changes in opioid-related deaths. TOP builds on recent advances in sequence modeling, namely transformer networks, to use changes in yearly language on Twitter and past mortality to project the following year's mortality rates by county. Trained over five years and evaluated over the next two years TROP demonstrated state-of-the-art accuracy in predicting future county-specific opioid trends. A model built using linear auto-regression and traditional socioeconomic data gave 7% error (MAPE) or within 2.93 deaths per 100,000 people on average; our proposed architecture was able to forecast yearly death rates with less than half that error: 3% MAPE and within 1.15 per 100,000 people.

Development and validation of medical record-based logistic regression and machine learning models to diagnose diabetic retinopathy

PURPOSES

Many factors were reported to be associated with diabetic retinopathy (DR); however, their contributions remained unclear. We aimed to evaluate the prognostic and diagnostic accuracy of logistic regression and three machine learning models based on various medical records.

METHODS

This was a cross-sectional study. We investigated the prevalence and associations of DR among 757 participants aged 40 years or older in the 2005-2006 National Health and Nutrition Examination Survey (NHANES). We trained the models to predict if the participants had DR with 15 predictor variables. Area under the receiver operating characteristic (AUROC) and mean squared error (MSE) of each algorithm were compared in the external validation dataset using a replicate cohort from NHANES 2007-2008.

RESULTS

Among the 757 participants, 53 (7.00%) subjects had DR, the mean (standard deviation, SD) age was 57.7 (13.04), and 78.0% were male (n = 42). Logistic regression revealed that female gender (OR = 4.130, 95% CI: 1.820-9.380; P < 0.05), HbA1c (OR = 1.665, 95% CI: 1.197-2.317; P < 0.05), serum creatine level (OR = 2.952, 95% CI: 1.274-6.851; P < 0.05), and eGFR level (OR = 1.009, 95% CI: 1.000-1.014, P < 0.05) increased the risk of DR. The average performance obtained from internal validation was similar in all models (AUROC ≥ 0.945), and k-nearest neighbors (KNN) had the highest value with an AUROC of 0.984. In external validation, they remained robust or with modest reductions in discrimination with AUROC still ≥ 0.902, and KNN also performed the best with an AUROC of 0.982. Both logistic regression and machine learning models had good performance in the clinical diagnosis of DR.

CONCLUSIONS

This study highlights the utility of comparing traditional logistic regression to machine learning models. We found that logistic regression performed as well as optimized machine learning methods when classifying DR patients.

Predicting Hospitalization among Medicaid Home- and Community-Based Services Users Using Machine Learning Methods

We compare multiple machine learning algorithms and develop models to predict future hospitalization among Home- and Community-Based Services (HCBS) Users. Furthermore, we calculate feature importance, the score of input variables based on their importance to predict the outcome, to identify the most relevant variables to predict hospitalization. We use the 2012 national Medicaid Analytic eXtract data and Medicare Provider Analysis and Review data. Predicting any hospitalization, Random Forest appears to be the most robust approach, though XGBoost achieved similar predictive performance. While the importance of features varies by algorithm, chronic conditions, previous hospitalizations, as well as use of services for ambulance, personal care, and durable medical equipment were generally found to be important predictors of hospitalization. Utilizing prediction models to identify those who are prone to hospitalization could be useful in developing early interventions to improve outcomes among HCBS users.

Predicting Patient Satisfaction With Medications for Treating Opioid Use Disorder: Case Study Applying Natural Language Processing to Reviews of Methadone and Buprenorphine/Naloxone on Health-Related Social Media

Background

Medication-assisted treatment (MAT) is an effective method for treating opioid use disorder (OUD), which combines behavioral therapies with one of three Food and Drug Administration-approved medications: methadone, buprenorphine, and naloxone. While MAT has been shown to be effective initially, there is a need for more information from the patient perspective about the satisfaction with medications. Existing research focuses on patient satisfaction with the entirety of the treatment, making it difficult to determine the unique role of medication and overlooking the views of those who may lack access to treatment due to being uninsured or concerns over stigma. Studies focusing on patients' perspectives are also limited by the lack of scales that can efficiently collect self-reports across domains of concerns.

Objective

A broad survey of patients' viewpoints can be obtained through social media and drug review forums, which are then assessed using automated methods to discover factors associated with medication satisfaction. Because the text is unstructured, it may contain a mix of formal and informal language. The primary aim of this study was to use natural language processing methods on text posted on health-related social media to detect patients' satisfaction with two well-studied OUD medications: methadone and buprenorphine/naloxone.

Methods

We collected 4353 patient reviews of methadone and buprenorphine/naloxone from 2008 to 2021 posted on WebMD and Drugs.com. To build our predictive models for detecting patient satisfaction, we first employed different analyses to build four input feature sets using the vectorized text, topic models, duration of treatment, and biomedical concepts by applying MetaMap. We then developed six prediction models: logistic regression, Elastic Net, least absolute shrinkage and selection operator, random forest classifier, Ridge classifier, and extreme gradient boosting to predict patients' satisfaction. Lastly, we compared the prediction models' performance over different feature sets.

Results

Topics discovered included oral sensation, side effects, insurance, and doctor visits. Biomedical concepts included symptoms, drugs, and illnesses. The F-score of the predictive models across all methods ranged from 89.9% to 90.8%. The Ridge classifier model, a regression-based method, outperformed the other models.

Conclusions

Assessment of patients' satisfaction with opioid dependency treatment medication can be predicted using automated text analysis. Adding biomedical concepts such as symptoms, drug name, and illness, along with the duration of treatment and topic models, had the most benefits for improving the prediction performance of the Elastic Net model compared to other models. Some of the factors associated with patient satisfaction overlap with domains covered in medication satisfaction scales (eg, side effects) and qualitative patient reports (eg, doctors' visits), while others (insurance) are overlooked, thereby underscoring the value added from processing text on online health forums to better understand patient adherence.

Using Machine Learning to Predict Treatment Adherence in Patients on Medication for Opioid Use Disorder

OBJECTIVE

Patients receiving medication for opioid use disorder (MOUD) may continue using nonprescribed drugs or have trouble with medication adherence, and it is difficult to predict which patients will continue to do so. In this study, we develop and validate an automated risk-modeling framework to predict opioid abstinence and medication adherence at a patient's next attended appointment and evaluate the predictive performance of machine-learning algorithms versus logistic regression.

METHODS

Urine drug screen and attendance records from 40,005 appointments drawn from 2742 patients at a multilocation office-based MOUD program were used to train logistic regression, logistic ridge regression, and XGBoost models to predict a composite indicator of treatment adherence (opioid-negative and norbuprenorphine-positive urine, no evidence of urine adulteration) at next attended appointment.

RESULTS

The XGBoost model had similar accuracy and discriminative ability (accuracy, 88%; area under the receiver operating curve, 0.87) to the two logistic regression models (accuracy, 88%; area under the receiver operating curve, 0.87). The XGBoost model had nearly perfect calibration in independent validation data; the logistic and ridge regression models slightly overestimated adherence likelihood. Historical treatment adherence, attendance rate, and fentanyl-positive urine at current appointment were the strongest contributors to treatment adherence at next attended appointment.

DISCUSSION

There is a need for risk prediction tools to improve delivery of MOUD. This study presents an automated and portable risk-modeling framework to predict treatment adherence at each patient's next attended appointment. The XGBoost algorithm appears to provide similar classification accuracy to logistic regression models; however, XGBoost may offer improved calibration of risk estimates compared with logistic regression.

Individualized Prospective Prediction of Opioid Use Disorder

OBJECTIVE

Opioid use disorder (OUD) is a chronic relapsing disorder with a problematic pattern of opioid use, affecting nearly 27 million people worldwide. Machine learning (ML)-based prediction of OUD may lead to early detection and intervention. However, most ML prediction studies were not based on representative data sources and prospective validations, limiting their potential to predict future new cases. In the current study, we aimed to develop and prospectively validate an ML model that could predict individual OUD cases based on representative large-scale health data.

METHOD

We present an ensemble machine-learning model trained on a cross-linked Canadian administrative health data set from 2014 to 2018 (n  =  699,164), with validation of model-predicted OUD cases on a hold-out sample from 2014 to 2018 (n  =  174,791) and prospective prediction of OUD cases on a non-overlapping sample from 2019 (n  =  316,039). We used administrative records of OUD diagnosis for each subject based on International Classification of Diseases (ICD) codes.

RESULTS

With 6409 OUD cases in 2019 (mean [SD], 45.34 [14.28], 3400 males), our model prospectively predicted OUD cases at a high accuracy (balanced accuracy, 86%, sensitivity, 93%; specificity 79%). In accord with prior findings, the top risk factors for OUD in this model were opioid use indicators and a history of other substance use disorders.

CONCLUSION

Our study presents an individualized prospective prediction of OUD cases by applying ML to large administrative health datasets. Such prospective predictions based on ML would be essential for potential future clinical applications in the early detection of OUD.

Using machine learning to study the effect of medication adherence in Opioid Use Disorder

BACKGROUND

Opioid Use Disorder (OUD) and opioid overdose (OD) impose huge social and economic burdens on society and health care systems. Research suggests that Medication for Opioid Use Disorder (MOUD) is effective in the treatment of OUD. We use machine learning to investigate the association between patient's adherence to prescribed MOUD along with other risk factors in patients diagnosed with OUD and potential OD following the treatment.

METHODS

We used longitudinal Medicaid claims for two selected US states to subset a total of 26,685 patients with OUD diagnosis and appropriate Medicaid coverage between 2015 and 2018. We considered patient age, sex, region level socio-economic data, past comorbidities, MOUD prescription type and other selected prescribed medications along with the Proportion of Days Covered (PDC) as a proxy for adherence to MOUD as predictive variables for our model, and overdose events as the dependent variable. We applied four different machine learning classifiers and compared their performance, focusing on the importance and effect of PDC as a variable. We also calculated results based on risk stratification, where our models separate high risk individuals from low risk, to assess usefulness in clinical decision-making.

RESULTS

Among the selected classifiers, the XGBoost classifier has the highest AUC (0.77) closely followed by the Logistic Regression (LR). The LR has the best stratification result: patients in the top 10% of risk scores account for 35.37% of overdose events over the next 12 month observation period. PDC score calculated over the treatment window is one of the most important features, with better PDC lowering risk of OD, as expected. In terms of risk stratification results, of the 35.37% of overdose events that the predictive model could detect within the top 10% of risk scores, 72.3% of these cases were non-adherent in terms of their medication (PDC <0.8). Targeting the top 10% outcome of the predictive model could decrease the total number of OD events by 10.4%.

CONCLUSIONS

The best performing models allow identification of, and focus on, those at high risk of opioid overdose. With MOUD being included for the first time as a factor of interest, and being identified as a significant factor, outreach activities related to MOUD can be targeted at those at highest risk.

Development and Validation of a Machine Learning Model to Estimate Risk of Adverse Outcomes Within 30 Days of Opioid Dispensation

IMPORTANCE

Machine learning approaches can assist opioid stewardship by identifying high-risk opioid prescribing for potential interventions.

OBJECTIVE

To develop a machine learning model for deployment that can estimate the risk of adverse outcomes within 30 days of an opioid dispensation as a potential component of prescription drug monitoring programs using access to real-world data.

DESIGN, SETTING, AND PARTICIPANTS

This prognostic study used population-level administrative health data to construct a machine learning model. This study took place in Alberta, Canada (from January 1, 2018, to December 31, 2019), and included all patients 18 years and older who received at least 1 opioid dispensation from a community pharmacy within the province.

EXPOSURES

Each opioid dispensation served as the unit of analysis.

MAIN OUTCOMES AND MEASURES

Opioid-related adverse outcomes were identified from administrative data sets. An XGBoost model was developed on 2018 data to estimate the risk of hospitalization, an emergency department visit, or mortality within 30 days of an opioid dispensation; validation on 2019 data was done to evaluate model performance. Model discrimination, calibration, and other relevant metrics are reported using daily and weekly predictions on both ranked predictions and predicted probability thresholds using all data from 2019.

RESULTS

A total of 853 324 participants represented 6 181 025 opioid dispensations, with 145 016 outcome events reported (2.3%); 46.4% of the participants were men and 53.6% were women, with a mean (SD) age of 49.1 (15.6) years for men and 51.0 (18.0) years for women. Of the outcome events, 77 326 (2.6% pretest probability) occurred within 30 days of a dispensation in the validation set (XGBoost C statistic, 0.82 [95% CI, 0.81-0.82]). The top 0.1 percentile of estimated risk had a positive likelihood ratio (LR) of 28.7, which translated to a posttest probability of 43.1%. In our simulations, the weekly measured predictions had higher positive LRs in both the highest-risk dispensations and percentiles of estimated risk compared with predictions measured daily. Net benefit analysis showed that using machine learning prediction may not add additional benefit over the entire range of probability thresholds.

CONCLUSIONS AND RELEVANCE

These findings suggest that prescription drug monitoring programs can use machine learning classifiers to identify patients at risk of opioid-related adverse outcomes and intervene on high-risk ranked predictions. Better access to available administrative and clinical data could improve the prediction performance of machine learning classifiers and thus expand opioid stewardship efforts.

New opioid use and risk of opioid-related adverse events among adults with intellectual and developmental disabilities in Ontario, Canada

BACKGROUND

Individuals with intellectual and developmental disability (IDD) can have a high prevalence of pain, which can be managed with prescription opioids. However, the prevalence of substance use disorder is also high in this population, raising concern about opioid-related adverse events.

AIMS

To assess the risk of opioid-related adverse events following opioid initiation among adults with versus without IDD.

METHOD

We conducted a population-based, propensity score matched cohort study on all adults starting prescription opioid therapy in Ontario, Canada, between January 2013 and December 2018. The outcomes of interest were opioid toxicity, new opioid use disorder (OUD) diagnosis and dose escalation (≥90 mg morphine or equivalent) in the year after opioid initiation. We used Cox proportional hazards models to assess the association between IDD diagnosis and each outcome.

RESULTS

The hazards of opioid toxicity and OUD were significantly higher in those with IDD compared with those without IDD in unmatched analyses (opioid toxicity hazard ratio 3.19, 95% CI 2.81-5.18; OUD hazard ratio 2.36, 95% CI 2.10-2.65), whereas the hazard of dose escalation was significantly lower (hazard ratio 0.76, 95% CI 0.66-0.88). Findings were no longer significant in propensity score matched models for opioid toxicity and dose escalation, whereas the hazard of OUD diagnosis was attenuated substantially in those with IDD (hazard ratio 0.79, 95% CI 0.68-0.91).

CONCLUSIONS

IDD diagnosis is not a driver of opioid-related harm. The increased risk we observed is likely driven by various risk factors often present in this population.

Decision tree model based prediction of the efficacy of acupuncture in methadone maintenance treatment

Background

Patients with MMT often face difficulties such as sleep disturbance, headaches, and difficulty in complete abstinence from drugs. Research has shown that acupuncture can mitigate side effects while attenuating methadone dependence. It also has a synergistic and attenuated effect on methadone maintenance treatment (MMT). Exploring the predictors of the efficacy of acupuncture intervention in MMT might help clinicians and patients promote acupuncture-assisted participation in MMT, and improve clinical treatment strategies for MMT.

Objective

To describe the effect of potential predictors on MMT after acupuncture intervention by building a decision-tree model of data from A Clinical Study of Acupuncture-assisted MMT.

Design, setting, and participants

In this randomized controlled trial, 135 patients with MMT underwent acupuncture at the Substance Dependence Department of Guangzhou Huiai Hospital in Guangzhou, Guangdong Province, China.

Intervention

A total of 135 patients were 1:1 randomly assigned to either an acupuncture plus routine care group (acupuncture plus methadone) or a routine group (methadone only) for 6 weeks, and followed up for 10 weeks. Sex, age, education level, route of previous opioid use, years of opioid use, and MMT time were recorded before the trial.

Outcome measurements and statistical analysis

All analyses were based on the intention-to-treat (ITT) population. The two decision tree models used the change of methadone dosage and the VAS score for opioid desire as response variables, respectively, and the evaluation criteria were positive effect (decreased by ≥20%) and no effect (decreased by <20%, or increased). We generated the respective feature weights for the decision tree and evaluated the model's accuracy and performance by Precision-Recall.

Results

The overall accuracy of methadone reduction and psychological craving VAS scoring decision trees were 0.63 and 0.74, respectively. The Methadone Dosage Efficacy decision tree identified years of opioid use (weight = 0.348), acupuncture (weight = 0.346), and route of previous opioid use (weight = 0.162) as key features. For the VAS Score decision tree, acupuncture (weight = 0.618), MMT time (weight = 0.235), and age (weight = 0.043) were the important features.

Conclusion

Exploratory decision tree analysis showed that acupuncture, years of opioid use, route of previous opioid use, MMT time, and age were key predictors of the MMT treatment. Thus, acupuncture-assisted MMT strategy should consider the relevant influencing factors mentioned above.

Patient summary

Understanding patient characteristics and the impact of acupuncture regimens on methadone dosage reduction in MMT patients may help physicians determine the best treatment regimen for patients. An analysis of data from our clinical trial showed that acupuncture, years of opioid use, route of previous opioid use, age, and MMT time were key predictors of progressive recovery in patients with MMT. Eligible patients may benefit most from the MMT rehabilitation that reduces consumption and psychological cravings for methadone.

Clinical trial registration

http://www.chictr.org.cn/index.aspx, identifier: ChiCTR1900026357.

Use of machine learning to examine disparities in completion of substance use disorder treatment

The objective of this work is to examine disparities in the completion of substance use disorder treatment in the U.S. Our data is from the Treatment Episode Dataset Discharge (TEDS-D) datasets from the U.S. Substance Abuse and Mental Health Services Administration (SAMHSA) for 2017–2019. We apply a two-stage virtual twins model (random forest + decision tree) where, in the first stage (random forest), we determine differences in treatment completion probability associated with race/ethnicity, income source, no co-occurrence of mental health disorders, gender (biological), no health insurance, veteran status, age, and primary substance (alcohol or opioid). In the second stage (decision tree), we identify subgroups associated with probability differences, where such subgroups are more or less likely to complete treatment. We find the subgroups most likely to complete substance use disorder treatment, when the subgroup represents more than 1% of the sample, are those with no mental health condition co-occurrence (4.8% more likely when discharged from an ambulatory outpatient treatment program, representing 62% of the sample; and 10% more likely for one of the more specifically defined subgroups representing 10% of the sample), an income source of job-related wages/salary (4.3% more likely when not having used in the 30 days primary to discharge and when primary substance is not alcohol only, representing 28% of the sample), and white non-Hispanics (2.7% more likely when discharged from residential long-term treatment, representing 9% of the sample). Important implications are that: 1) those without a co-occurring mental health condition are the most likely to complete treatment, 2) those with job related wages or income are more likely to complete treatment, and 3) racial/ethnicity disparities persist in favor of white non-Hispanic individuals seeking to complete treatment. Thus, additional resources may be needed to combat such disparities.

Prescription quantity and duration predict progression from acute to chronic opioid use in opioid-naïve Medicaid patients

Opiates used for acute pain are an established risk factor for chronic opioid use (COU). Patient characteristics contribute to progression from acute opioid use to COU, but most are not clinically modifiable. To develop and validate machine-learning algorithms that use claims data to predict progression from acute to COU in the Medicaid population, Adult opioid naïve Medicaid patients from 6 anonymized states who received an opioid prescription between 2015 and 2019 were included. Five machine learning (ML) Models were developed, and model performance assessed by area under the receiver operating characteristic curve (auROC), precision and recall. In the study, 29.9% (53820/180000) of patients transitioned from acute opioid use to COU. Initial opioid prescriptions in COU patients had increased morphine milligram equivalents (MME) (33.2 vs. 23.2), tablets per prescription (45.6 vs. 36.54), longer prescriptions (26.63 vs 24.69 days), and higher proportions of tramadol (16.06% vs. 13.44%) and long acting oxycodone (0.24% vs 0.04%) compared to non- COU patients. The top performing model was XGBoost that achieved average precision of 0.87 and auROC of 0.63 in testing and 0.55 and 0.69 in validation, respectively. Top-ranking prescription-related features in the model included quantity of tablets per prescription, prescription length, and emergency department claims. In this study, the Medicaid population, opioid prescriptions with increased tablet quantity and days supply predict increased risk of progression from acute to COU in opioid-naïve patients. Future research should evaluate the effects of modifying these risk factors on COU incidence.

Predictive Modeling for Suicide-Related Outcomes and Risk Factors among Patients with Pain Conditions: A Systematic Review

Suicide is a leading cause of death in the US. Patients with pain conditions have higher suicidal risks. In a systematic review searching observational studies from multiple sources (e.g., MEDLINE) from 1 January 2000–12 September 2020, we evaluated existing suicide prediction models’ (SPMs) performance and identified risk factors and their derived data sources among patients with pain conditions. The suicide-related outcomes included suicidal ideation, suicide attempts, suicide deaths, and suicide behaviors. Among the 87 studies included (with 8 SPM studies), 107 suicide risk factors (grouped into 27 categories) were identified. The most frequently occurring risk factor category was depression and their severity (33%). Approximately 20% of the risk factor categories would require identification from data sources beyond structured data (e.g., clinical notes). For 8 SPM studies (only 2 performing validation), the reported prediction metrics/performance varied: C-statistics (n = 3 studies) ranged 0.67–0.84, overall accuracy(n = 5): 0.78–0.96, sensitivity(n = 2): 0.65–0.91, and positive predictive values(n = 3): 0.01–0.43. Using the modified Quality in Prognosis Studies tool to assess the risk of biases, four SPM studies had moderate-to-high risk of biases. This systematic review identified a comprehensive list of risk factors that may improve predicting suicidal risks for patients with pain conditions. Future studies need to examine reasons for performance variations and SPM’s clinical utility.

Changes in predicted opioid overdose risk over time in a state Medicaid program: a group-based trajectory modeling analysis

BACKGROUND AND AIMS

The time lag encountered when accessing health-care data is one major barrier to implementing opioid overdose prediction measures in practice. Little is known regarding how one's opioid overdose risk changes over time. We aimed to identify longitudinal patterns of individual predicted overdose risks among Medicaid beneficiaries after initiation of opioid prescriptions.

DESIGN, SETTING AND PARTICIPANTS

A retrospective cohort study in Pennsylvania, USA among Pennsylvania Medicaid beneficiaries aged 18-64 years who initiated opioid prescriptions between July 2017 and September 2018 (318 585 eligible beneficiaries (mean age = 39 ± 12 years, female = 65.7%, White = 62.2% and Black = 24.9%).

MEASUREMENTS

We first applied a previously developed and validated machine-learning algorithm to obtain risk scores for opioid overdose emergency room or hospital visits in 3-month intervals for each beneficiary who initiated opioid therapy, until disenrollment from Medicaid, death or the end of observation (December 2018). We performed group-based trajectory modeling to identify trajectories of these predicted overdose risk scores over time.

FINDINGS

Among eligible beneficiaries, 0.61% had one or more occurrences of opioid overdose in a median follow-up of 15 months. We identified five unique opioid overdose risk trajectories: three trajectories (accounting for 92% of the cohort) had consistent overdose risk over time, including consistent low-risk (63%), consistent medium-risk (25%) and consistent high-risk (4%) groups; another two trajectories (accounting for 8%) had overdose risks that substantially changed over time, including a group that transitioned from high- to medium-risk (3%) and another group that increased from medium- to high-risk over time (5%).

CONCLUSIONS

More than 90% of Medicaid beneficiaries in Pennsylvania USA with one or more opioid prescriptions had consistent, predicted opioid overdose risks over 15 months. Applying opioid prediction algorithms developed from historical data may not be a major barrier to implementation in practice for the large majority of individuals.

Dosing profiles of concurrent opioid and benzodiazepine use associated with overdose risk among US Medicare beneficiaries: group-based multi-trajectory models

BACKGROUND AND AIMS

One-third of opioid (OPI) overdose deaths involve concurrent benzodiazepine (BZD) use. Little is known about concurrent opioid and benzodiazepine use (OPI-BZD) most associated with overdose risk. We aimed to examine associations between OPI-BZD dose and duration trajectories, and subsequent OPI or BZD overdose in US Medicare.

DESIGN

Retrospective cohort study.

SETTING

US Medicare.

PARTICIPANTS

Using a 5% national Medicare data sample (2013-16) of fee-for-service beneficiaries without cancer initiating OPI prescriptions, we identified 37 879 beneficiaries (age ≥ 65 = 59.3%, female = 71.9%, white = 87.6%, having OPI overdose = 0.3%).

MEASUREMENTS

During the 6 months following OPI initiation (i.e. trajectory period), we identified OPI-BZD dose and duration patterns using group-based multi-trajectory models, based on average daily morphine milligram equivalents (MME) for OPIs and diazepam milligram equivalents (DME) for BZDs. To label dose levels in each trajectory, we defined OPI use as very low (< 25 MME), low (25-50 MME), moderate (51-90 MME), high (91-150 MME) and very high (>150 MME) dose. Similarly, we defined BZD use as very low (< 10 DME), low (10-20 DME), moderate (21-40 DME), high (41-60 DME) and very high (> 60 DME) dose. Our primary analysis was to estimate the risk of time to first hospital or emergency department visit for OPI overdose within 6 months following the trajectory period using inverse probability of treatment-weighted Cox proportional hazards models.

FINDINGS

We identified nine distinct OPI-BZD trajectories: group A: very low OPI (early discontinuation)-very low declining BZD (n = 10 598; 28.0% of the cohort); B: very low OPI (early discontinuation)-very low stable BZD (n = 4923; 13.0%); C: very low OPI (early discontinuation)-medium BZD (n = 4997; 13.2%); D: low OPI-low BZD (n = 5083; 13.4%); E: low OPI-high BZD (n = 3906; 10.3%); F: medium OPI-low BZD (n = 3948; 10.4%); G: very high OPI-high BZD (n = 1371; 3.6%); H: very high OPI-very high BZD (n = 957; 2.5%); and I: very high OPI-low BZD (n = 2096; 5.5%). Compared with group A, five trajectories (32.3% of the study cohort) were associated with increased 6-month OPI overdose risks: E: low OPI-high BZD [hazard ratio (HR) = 3.27, 95% confidence interval (CI) = 1.61-6.63]; F: medium OPI-low BZD (HR = 4.04, 95% CI = 2.06-7.95); G: very high OPI-high BZD (HR = 6.98, 95% CI = 3.11-15.64); H: very high OPI-very high BZD (HR = 4.41, 95% CI = 1.51-12.85); and I: very high OPI-low BZD (HR = 6.50, 95% CI = 3.15-13.42).

CONCLUSIONS

Patterns of concurrent opioid and benzodiazepine use most associated with overdose risk among fee-for-service US Medicare beneficiaries initiating opioid prescriptions include very high-dose opioid use (MME > 150), high-dose benzodiazepine use (DME > 40) or medium-dose opioid with low-dose benzodiazepine use.

Developing and validating a machine-learning algorithm to predict opioid overdose in Medicaid beneficiaries in two US states: a prognostic modelling study

BACKGROUND

Little is known about whether machine-learning algorithms developed to predict opioid overdose using earlier years and from a single state will perform as well when applied to other populations. We aimed to develop a machine-learning algorithm to predict 3-month risk of opioid overdose using Pennsylvania Medicaid data and externally validated it in two data sources (ie, later years of Pennsylvania Medicaid data and data from a different state).

METHODS

This prognostic modelling study developed and validated a machine-learning algorithm to predict overdose in Medicaid beneficiaries with one or more opioid prescription in Pennsylvania and Arizona, USA. To predict risk of hospital or emergency department visits for overdose in the subsequent 3 months, we measured 284 potential predictors from pharmaceutical and health-care encounter claims data in 3-month periods, starting 3 months before the first opioid prescription and continuing until loss to follow-up or study end. We developed and internally validated a gradient-boosting machine algorithm to predict overdose using 2013-16 Pennsylvania Medicaid data (n=639 693). We externally validated the model using (1) 2017-18 Pennsylvania Medicaid data (n=318 585) and (2) 2015-17 Arizona Medicaid data (n=391 959). We reported several prediction performance metrics (eg, C-statistic, positive predictive value). Beneficiaries were stratified into risk-score subgroups to support clinical use.

FINDINGS

A total of 8641 (1·35%) 2013-16 Pennsylvania Medicaid beneficiaries, 2705 (0·85%) 2017-18 Pennsylvania Medicaid beneficiaries, and 2410 (0·61%) 2015-17 Arizona beneficiaries had one or more overdose during the study period. C-statistics for the algorithm predicting 3-month overdoses developed from the 2013-16 Pennsylvania training dataset and validated on the 2013-16 Pennsylvania internal validation dataset, 2017-18 Pennsylvania external validation dataset, and 2015-17 Arizona external validation dataset were 0·841 (95% CI 0·835-0·847), 0·828 (0·822-0·834), and 0·817 (0·807-0·826), respectively. In external validation datasets, 71 361 (22·4%) of 318 585 2017-18 Pennsylvania beneficiaries were in high-risk subgroups (positive predictive value of 0·38-4·08%; capturing 73% of overdoses in the subsequent 3 months) and 40 041 (10%) of 391 959 2015-17 Arizona beneficiaries were in high-risk subgroups (positive predictive value of 0·19-1·97%; capturing 55% of overdoses). Lower risk subgroups in both validation datasets had few individuals (≤0·2%) with an overdose.

INTERPRETATION

A machine-learning algorithm predicting opioid overdose derived from Pennsylvania Medicaid data performed well in external validation with more recent Pennsylvania data and with Arizona Medicaid data. The algorithm might be valuable for overdose risk prediction and stratification in Medicaid beneficiaries.

FUNDING

National Institute of Health, National Institute on Drug Abuse, National Institute on Aging.

Development and multimodal validation of a substance misuse algorithm for referral to treatment using artificial intelligence (SMART-AI): a retrospective deep learning study

Background

Substance misuse is a heterogeneous and complex set of behavioural conditions that are highly prevalent in hospital settings and frequently co-occur. Few hospital-wide solutions exist to comprehensively and reliably identify these conditions to prioritise care and guide treatment. The aim of this study was to apply natural language processing (NLP) to clinical notes collected in the electronic health record (EHR) to accurately screen for substance misuse.

Methods

The model was trained and developed on a reference dataset derived from a hospital-wide programme at Rush University Medical Center (RUMC), Chicago, IL, USA, that used structured diagnostic interviews to manually screen admitted patients over 27 months (between Oct 1, 2017, and Dec 31, 2019; n=54 915). The Alcohol Use Disorder Identification Test and Drug Abuse Screening Tool served as reference standards. The first 24 h of notes in the EHR were mapped to standardised medical vocabulary and fed into single-label, multilabel, and multilabel with auxillary-task neural network models. Temporal validation of the model was done using data from the subsequent 12 months on a subset of RUMC patients (n=16 917). External validation was done using data from Loyola University Medical Center, Chicago, IL, USA between Jan 1, 2007, and Sept 30, 2017 (n=1991 adult patients). The primary outcome was discrimination for alcohol misuse, opioid misuse, or non-opioid drug misuse. Discrimination was assessed by the area under the receiver operating characteristic curve (AUROC). Calibration slope and intercept were measured with the unreliability index. Bias assessments were performed across demographic subgroups.

Findings

The model was trained on a cohort that had 3·5% misuse (n=1 921) with any type of substance. 220 (11%) of 1921 patients with substance misuse had more than one type of misuse. The multilabel convolutional neural network classifier had a mean AUROC of 0·97 (95% CI 0·96–0·98) during temporal validation for all types of substance misuse. The model was well calibrated and showed good face validity with model features containing explicit mentions of aberrant drug-taking behaviour. A false-negative rate of 0·18–0·19 and a false-positive rate of 0·03 between non-Hispanic Black and non-Hispanic White groups occurred. In external validation, the AUROCs for alcohol and opioid misuse were 0·88 (95% CI 0·86–0·90) and 0·94 (0·92–0·95), respectively.

Interpretation

We developed a novel and accurate approach to leveraging the first 24 h of EHR notes for screening multiple types of substance misuse.

Funding

National Institute On Drug Abuse, National Institutes of Health.

Practical foundations of machine learning for addiction research. Part II. Workflow and use cases

In a continuum with applied statistics, machine learning offers a wide variety of tools to explore, analyze, and understand addiction data. These tools include algorithms that can leverage useful information from data to build models; these models can solve particular tasks to answer addiction scientific questions. In this second part of a two-part review on machine learning, we explain how to apply machine learning methods to addiction research. Like other analytical tools, machine learning methods require a careful implementation to carry out a reproducible and transparent research process with reliable results. This review describes a workflow to guide the application of machine learning in addiction research, detailing study design, data collection, data pre-processing, modeling, and results communication. How to train, validate, and test a model, detect and characterize overfitting, and determine an adequate sample size are some of the key issues when applying machine learning. We also illustrate the process and particular nuances with examples of how researchers in addiction have applied machine learning techniques with different goals, study designs, or data sources as well as explain the main limitations of machine learning approaches and how to best address them. A good use of machine learning enriches the addiction research toolkit.

Practical foundations of machine learning for addiction research. Part I. Methods and techniques

Machine learning assembles a broad set of methods and techniques to solve a wide range of problems, such as identifying individuals with substance use disorders (SUD), finding patterns in neuroimages, understanding SUD prognostic factors and their association, or determining addiction genetic underpinnings. However, the addiction research field underuses machine learning. This two-part narrative review focuses on machine learning tools and concepts, providing an introductory insight into their capabilities to facilitate their understanding and acquisition by addiction researchers. This first part presents supervised and unsupervised methods such as linear models, naive Bayes, support vector machines, artificial neural networks, and k-means. We illustrate each technique with examples of its use in current addiction research. We also present some open-source programming tools and methodological good practices that facilitate using these techniques. Throughout this work, we emphasize a continuum between applied statistics and machine learning, we show their commonalities, and provide sources for further reading to deepen the understanding of these methods. This two-part review is a primer for the next generation of addiction researchers incorporating machine learning in their projects. Researchers will find a bridge between applied statistics and machine learning, ways to expand their analytical toolkit, recommendations to incorporate well-established good practices in addiction data analysis (e.g., stating the rationale for using newer analytical tools, calculating sample size, improving reproducibility), and the vocabulary to enhance collaboration between researchers who do not conduct data analyses and those who do.