Development and validation of an overdose risk prediction tool using prescription drug monitoring program data

OBJECTIVES

To develop and validate a machine-learning algorithm to predict fatal overdose using Pennsylvania Prescription Drug Monitoring Program (PDMP) data.

METHODS

The training/testing (n = 3020,748) and validation (n = 2237,701) cohorts included Pennsylvania residents with a prescription dispensing from February 2018-September 2021. Potential predictors (n = 222) were measured in the 6 months prior to a random index date. Using a gradient boosting machine, we developed a 20-variable model to predict risk of fatal drug overdose in the 6 months after the index date.

RESULTS

Beneficiaries in the training (n = 1,812,448), testing (n = 1,208,300), and validation (n = 2,237,701) samples had similar age, with low rates of fatal overdose during 6-month follow up (0.12%, 0.12%, 0.04%, respectively). The validation c-statistic was 0.86 for predicting fatal overdose using 20 PDMP variables. When ranking individuals based on risk score, the prediction model more accurately identified fatal overdose at 6 months compared to using opioid dosage or opioid/benzodiazepine overlap, although the percentage of individuals in the highest risk percentile who died at 6 months was less than 1%.

CONCLUSIONS AND POLICY IMPLICATIONS

A gradient boosting machine algorithm predicting fatal overdose derived from twenty variables performed well in discriminating risk across testing and validation samples, improving on single factor risk measures like opioid dosage.

Changes in predicted opioid overdose risk over time in a state Medicaid program: a group-based trajectory modeling analysis

BACKGROUND AND AIMS

The time lag encountered when accessing health-care data is one major barrier to implementing opioid overdose prediction measures in practice. Little is known regarding how one's opioid overdose risk changes over time. We aimed to identify longitudinal patterns of individual predicted overdose risks among Medicaid beneficiaries after initiation of opioid prescriptions.

DESIGN, SETTING AND PARTICIPANTS

A retrospective cohort study in Pennsylvania, USA among Pennsylvania Medicaid beneficiaries aged 18-64 years who initiated opioid prescriptions between July 2017 and September 2018 (318 585 eligible beneficiaries (mean age = 39 ± 12 years, female = 65.7%, White = 62.2% and Black = 24.9%).

MEASUREMENTS

We first applied a previously developed and validated machine-learning algorithm to obtain risk scores for opioid overdose emergency room or hospital visits in 3-month intervals for each beneficiary who initiated opioid therapy, until disenrollment from Medicaid, death or the end of observation (December 2018). We performed group-based trajectory modeling to identify trajectories of these predicted overdose risk scores over time.

FINDINGS

Among eligible beneficiaries, 0.61% had one or more occurrences of opioid overdose in a median follow-up of 15 months. We identified five unique opioid overdose risk trajectories: three trajectories (accounting for 92% of the cohort) had consistent overdose risk over time, including consistent low-risk (63%), consistent medium-risk (25%) and consistent high-risk (4%) groups; another two trajectories (accounting for 8%) had overdose risks that substantially changed over time, including a group that transitioned from high- to medium-risk (3%) and another group that increased from medium- to high-risk over time (5%).

CONCLUSIONS

More than 90% of Medicaid beneficiaries in Pennsylvania USA with one or more opioid prescriptions had consistent, predicted opioid overdose risks over 15 months. Applying opioid prediction algorithms developed from historical data may not be a major barrier to implementation in practice for the large majority of individuals.

Developing and validating a machine-learning algorithm to predict opioid overdose in Medicaid beneficiaries in two US states: a prognostic modelling study

BACKGROUND

Little is known about whether machine-learning algorithms developed to predict opioid overdose using earlier years and from a single state will perform as well when applied to other populations. We aimed to develop a machine-learning algorithm to predict 3-month risk of opioid overdose using Pennsylvania Medicaid data and externally validated it in two data sources (ie, later years of Pennsylvania Medicaid data and data from a different state).

METHODS

This prognostic modelling study developed and validated a machine-learning algorithm to predict overdose in Medicaid beneficiaries with one or more opioid prescription in Pennsylvania and Arizona, USA. To predict risk of hospital or emergency department visits for overdose in the subsequent 3 months, we measured 284 potential predictors from pharmaceutical and health-care encounter claims data in 3-month periods, starting 3 months before the first opioid prescription and continuing until loss to follow-up or study end. We developed and internally validated a gradient-boosting machine algorithm to predict overdose using 2013-16 Pennsylvania Medicaid data (n=639 693). We externally validated the model using (1) 2017-18 Pennsylvania Medicaid data (n=318 585) and (2) 2015-17 Arizona Medicaid data (n=391 959). We reported several prediction performance metrics (eg, C-statistic, positive predictive value). Beneficiaries were stratified into risk-score subgroups to support clinical use.

FINDINGS

A total of 8641 (1·35%) 2013-16 Pennsylvania Medicaid beneficiaries, 2705 (0·85%) 2017-18 Pennsylvania Medicaid beneficiaries, and 2410 (0·61%) 2015-17 Arizona beneficiaries had one or more overdose during the study period. C-statistics for the algorithm predicting 3-month overdoses developed from the 2013-16 Pennsylvania training dataset and validated on the 2013-16 Pennsylvania internal validation dataset, 2017-18 Pennsylvania external validation dataset, and 2015-17 Arizona external validation dataset were 0·841 (95% CI 0·835-0·847), 0·828 (0·822-0·834), and 0·817 (0·807-0·826), respectively. In external validation datasets, 71 361 (22·4%) of 318 585 2017-18 Pennsylvania beneficiaries were in high-risk subgroups (positive predictive value of 0·38-4·08%; capturing 73% of overdoses in the subsequent 3 months) and 40 041 (10%) of 391 959 2015-17 Arizona beneficiaries were in high-risk subgroups (positive predictive value of 0·19-1·97%; capturing 55% of overdoses). Lower risk subgroups in both validation datasets had few individuals (≤0·2%) with an overdose.

INTERPRETATION

A machine-learning algorithm predicting opioid overdose derived from Pennsylvania Medicaid data performed well in external validation with more recent Pennsylvania data and with Arizona Medicaid data. The algorithm might be valuable for overdose risk prediction and stratification in Medicaid beneficiaries.

FUNDING

National Institute of Health, National Institute on Drug Abuse, National Institute on Aging.

Predicting Mortality Risk After a Hospital or Emergency Department Visit for Nonfatal Opioid Overdose

BACKGROUND

Survivors of opioid overdose have substantially increased mortality risk, although this risk is not evenly distributed across individuals. No study has focused on predicting an individual's risk of death after a nonfatal opioid overdose.

OBJECTIVE

To predict risk of death after a nonfatal opioid overdose.

DESIGN AND PARTICIPANTS

This retrospective cohort study included 9686 Pennsylvania Medicaid beneficiaries with an emergency department or inpatient claim for nonfatal opioid overdose in 2014-2016. The index date was the first overdose claim during this period.

EXPOSURES, MAIN OUTCOME, AND MEASURES

Predictor candidates were measured in the 180 days before the index overdose. Primary outcome was 180-day all-cause mortality. Using a gradient boosting machine model, we classified beneficiaries into six subgroups according to their risk of mortality (< 25th percentile of the risk score, 25th to < 50th, 50th to < 75th, 75th to < 90th, 90th to < 98th, ≥ 98th). We then measured receipt of medication for opioid use disorder (OUD), risk mitigation interventions (e.g., prescriptions for naloxone), and prescription opioids filled in the 180 days after the index overdose, by risk subgroup.

KEY RESULTS

Of eligible beneficiaries, 347 (3.6%) died within 180 days after the index overdose. The C-statistic of the mortality prediction model was 0.71. In the highest risk subgroup, the observed 180-day mortality rate was 20.3%, while in the lowest risk subgroup, it was 1.5%. Medication for OUD and risk mitigation interventions after overdose were more commonly seen in lower risk groups, while opioid prescriptions were more likely to be used in higher risk groups (both p trends < .001).

CONCLUSIONS

A risk prediction model performed well for classifying mortality risk after a nonfatal opioid overdose. This prediction score can identify high-risk subgroups to target interventions to improve outcomes among overdose survivors.

Using machine learning to predict risk of incident opioid use disorder among fee-for-service Medicare beneficiaries: A prognostic study

OBJECTIVE

To develop and validate a machine-learning algorithm to improve prediction of incident OUD diagnosis among Medicare beneficiaries with ≥1 opioid prescriptions.

METHODS

This prognostic study included 361,527 fee-for-service Medicare beneficiaries, without cancer, filling ≥1 opioid prescriptions from 2011-2016. We randomly divided beneficiaries into training, testing, and validation samples. We measured 269 potential predictors including socio-demographics, health status, patterns of opioid use, and provider-level and regional-level factors in 3-month periods, starting from three months before initiating opioids until development of OUD, loss of follow-up or end of 2016. The primary outcome was a recorded OUD diagnosis or initiating methadone or buprenorphine for OUD as proxy of incident OUD. We applied elastic net, random forests, gradient boosting machine, and deep neural network to predict OUD in the subsequent three months. We assessed prediction performance using C-statistics and other metrics (e.g., number needed to evaluate to identify an individual with OUD [NNE]). Beneficiaries were stratified into subgroups by risk-score decile.

RESULTS

The training (n = 120,474), testing (n = 120,556), and validation (n = 120,497) samples had similar characteristics (age ≥65 years = 81.1%; female = 61.3%; white = 83.5%; with disability eligibility = 25.5%; 1.5% had incident OUD). In the validation sample, the four approaches had similar prediction performances (C-statistic ranged from 0.874 to 0.882); elastic net required the fewest predictors (n = 48). Using the elastic net algorithm, individuals in the top decile of risk (15.8% [n = 19,047] of validation cohort) had a positive predictive value of 0.96%, negative predictive value of 99.7%, and NNE of 104. Nearly 70% of individuals with incident OUD were in the top two deciles (n = 37,078), having highest incident OUD (36 to 301 per 10,000 beneficiaries). Individuals in the bottom eight deciles (n = 83,419) had minimal incident OUD (3 to 28 per 10,000).

CONCLUSIONS

Machine-learning algorithms improve risk prediction and risk stratification of incident OUD in Medicare beneficiaries.

Effective Care Practices in Patients Receiving Prolonged Mechanical Ventilation. An Ethnographic Study

Rationale: Patients receiving prolonged mechanical ventilation experience low survival rates and incur high healthcare costs. However, little is known about how to optimally organize and manage their care.Objectives: To identify a set of effective care practices for patients receiving prolonged mechanical ventilation.Methods: We performed a focused ethnographic evaluation at eight long-term acute care hospitals in the United States ranking in either the lowest or highest quartile of risk-adjusted mortality in at least four of the five years between 2007 and 2011.Measurements and Main Results: We conducted 329 hours of direct observation, 196 interviews, and 39 episodes of job shadowing. Data were analyzed using thematic content analysis and a positive-negative deviance approach. We found that high- and low-performing hospitals differed substantially in their approach to care. High-performing hospitals actively promoted interdisciplinary communication and coordination using a range of organizational practices, including factors related to leadership (e.g., leaders who communicate a culture of quality improvement), staffing (e.g., lower nurse-to-patient ratios and ready availability of psychologists and spiritual care providers), care protocols (e.g., specific yet flexible respiratory therapy-driven weaning protocols), team meetings (e.g., interdisciplinary meetings that include direct care providers), and the physical plant (e.g., large workstations that allow groups to interact). These practices were believed to facilitate care that is simultaneously goal directed and responsive to individual patient needs, leading to more successful liberation from mechanical ventilation and improved survival.Conclusions: High-performing long-term acute care hospitals employ several organizational practices that may be helpful in improving care for patients receiving prolonged mechanical ventilation.

Sponsorship and Funding for Gene Therapy Trials in the United States

This study characterizes government, academia, and private funding for gene therapy trials in the United States by technology type and therapeutic and disease area.

Determinants of Intensive Care Unit Telemedicine Effectiveness. An Ethnographic Study

RATIONALE

Telemedicine is an increasingly common care delivery strategy in the ICU. However, ICU telemedicine programs vary widely in their clinical effectiveness, with some studies showing a large mortality benefit and others showing no benefit or even harm.

OBJECTIVES

To identify the organizational factors associated with ICU telemedicine effectiveness.

METHODS

We performed a focused ethnographic evaluation of 10 ICU telemedicine programs using site visits, interviews, and focus groups in both facilities providing remote care and the target ICUs. Programs were selected based on their change in risk-adjusted mortality after adoption (decreased mortality, no change in mortality, and increased mortality). We used a constant comparative approach to guide data collection and analysis.

MEASUREMENTS AND MAIN RESULTS

We conducted 460 hours of direct observation, 222 interviews, and 18 focus groups across six telemedicine facilities and 10 target ICUs. Data analysis revealed three domains that influence ICU telemedicine effectiveness: 1) leadership (i.e., the decisions related to the role of the telemedicine, conflict resolution, and relationship building), 2) perceived value (i.e., expectations of availability and impact, staff satisfaction, and understanding of operations), and 3) organizational characteristics (i.e., staffing models, allowed involvement of the telemedicine unit, and new hire orientation). In the most effective telemedicine programs these factors led to services that are viewed as appropriate, integrated, responsive, and consistent.

CONCLUSIONS

The effectiveness of ICU telemedicine programs may be influenced by several potentially modifiable factors within the domains of leadership, perceived value, and organizational structure.

Evaluation of Machine-Learning Algorithms for Predicting Opioid Overdose Risk Among Medicare Beneficiaries With Opioid Prescriptions

IMPORTANCE

Current approaches to identifying individuals at high risk for opioid overdose target many patients who are not truly at high risk.

OBJECTIVE

To develop and validate a machine-learning algorithm to predict opioid overdose risk among Medicare beneficiaries with at least 1 opioid prescription.

DESIGN, SETTING, AND PARTICIPANTS

A prognostic study was conducted between September 1, 2017, and December 31, 2018. Participants (n = 560 057) included fee-for-service Medicare beneficiaries without cancer who filled 1 or more opioid prescriptions from January 1, 2011, to December 31, 2015. Beneficiaries were randomly and equally divided into training, testing, and validation samples.

EXPOSURES

Potential predictors (n = 268), including sociodemographics, health status, patterns of opioid use, and practitioner-level and regional-level factors, were measured in 3-month windows, starting 3 months before initiating opioids until loss of follow-up or the end of observation.

MAIN OUTCOMES AND MEASURES

Opioid overdose episodes from inpatient and emergency department claims were identified. Multivariate logistic regression (MLR), least absolute shrinkage and selection operator-type regression (LASSO), random forest (RF), gradient boosting machine (GBM), and deep neural network (DNN) were applied to predict overdose risk in the subsequent 3 months after initiation of treatment with prescription opioids. Prediction performance was assessed using the C statistic and other metrics (eg, sensitivity, specificity, and number needed to evaluate [NNE] to identify one overdose). The Youden index was used to identify the optimized threshold of predicted score that balanced sensitivity and specificity.

RESULTS

Beneficiaries in the training (n = 186 686), testing (n = 186 685), and validation (n = 186 686) samples had similar characteristics (mean [SD] age of 68.0 [14.5] years, and approximately 63% were female, 82% were white, 35% had disabilities, 41% were dual eligible, and 0.60% had at least 1 overdose episode). In the validation sample, the DNN (C statistic = 0.91; 95% CI, 0.88-0.93) and GBM (C statistic = 0.90; 95% CI, 0.87-0.94) algorithms outperformed the LASSO (C statistic = 0.84; 95% CI, 0.80-0.89), RF (C statistic = 0.80; 95% CI, 0.75-0.84), and MLR (C statistic = 0.75; 95% CI, 0.69-0.80) methods for predicting opioid overdose. At the optimized sensitivity and specificity, DNN had a sensitivity of 92.3%, specificity of 75.7%, NNE of 542, positive predictive value of 0.18%, and negative predictive value of 99.9%. The DNN classified patients into low-risk (76.2% [142 180] of the cohort), medium-risk (18.6% [34 579] of the cohort), and high-risk (5.2% [9747] of the cohort) subgroups, with only 1 in 10 000 in the low-risk subgroup having an overdose episode. More than 90% of overdose episodes occurred in the high-risk and medium-risk subgroups, although positive predictive values were low, given the rare overdose outcome.

CONCLUSIONS AND RELEVANCE

Machine-learning algorithms appear to perform well for risk prediction and stratification of opioid overdose, especially in identifying low-risk subgroups that have minimal risk of overdose.

Perceptions of rounding checklists in the intensive care unit: a qualitative study

BACKGROUND

Rounding checklists are an increasingly common quality improvement tool in the intensive care unit (ICU). However, effectiveness studies have shown conflicting results. We sought to understand ICU providers' perceptions of checklists, as well as barriers and facilitators to effective utilisation of checklists during daily rounds.

OBJECTIVES

To understand how ICU providers perceive rounding checklists and develop a framework for more effective rounding checklist implementation.

METHODS

We performed a qualitative study in 32 ICUs within 14 hospitals in a large integrated health system in the USA. We used two complementary data collection methods: direct observation of daily rounds and semistructured interviews with ICU clinicians. Observations and interviews were thematically coded and primary themes were identified using a combined inductive and deductive approach.

RESULTS

We conducted 89 interviews and performed 114 hours of observation. Among study ICUs, 12 used checklists and 20 did not. Participants described the purpose of rounding checklists as a daily reminder for evidence-based practices, a tool for increasing shared understanding of patient care across care providers and a way to increase the efficiency of rounds. Checklists were perceived as not helpful when viewed as overstandardising care and when they are not relevant to a particular ICU's needs. Strategies to improve checklist implementation include attention to the brevity and relevance of the checklist to the particular ICU, consistent use over time, and integration with daily work flow.

CONCLUSION

Our results provide potential insights about why ICU rounding checklists frequently fail to improve outcomes and offer a framework for effective checklist implementation through greater feedback and accountability.

Hospital Perceptions of Medicare's Sepsis Quality Reporting Initiative

BACKGROUND

In October 2015, the Centers for Medicare and Medicaid Services (CMS) implemented the Sepsis CMS Core Measure (SEP-1) program, requiring hospitals to report data on the quality of care for their patients with sepsis.

OBJECTIVE

We sought to understand hospital perceptions of and responses to the SEP-1 program.

DESIGN

A thematic content analysis of semistructured interviews with hospital quality officials.

SETTING

A stratified random sample of short-stay, nonfederal, general acute care hospitals in the United States.

PATIENTS

Hospital quality officers, including nurses and physicians.

MEASUREMENTS

We completed 29 interviews before reaching content saturation.

RESULTS

Hospitals reported a variety of actions in response to SEP-1, including new efforts to collect data, improve sepsis diagnosis and treatment, and manage clinicians' attitudes toward SEP-1. These efforts frequently required dedicated resources to meet the program's requirements for treatment and documentation, which were thought to be complex and not consistently linked to patient-centered outcomes. Most respondents felt that SEP-1 was likely to improve sepsis outcomes. At the same time, they described specific changes that could improve its effectiveness, including allowing hospitals to focus on the treatment processes most directly associated with improved patient outcomes and better aligning the measure's sepsis definitions with current clinical definitions.

CONCLUSIONS

Hospitals are responding to the SEP-1 program across a number of domains and in ways that consistently require dedicated resources. Hospitals are interested in further revisions to the program to alleviate the burden of the reporting requirements and help them optimize the effectiveness of their investments in quality-improvement efforts.

Identifying Strategies for Effective Telemedicine Use in Intensive Care Units: The ConnECCT Study Protocol

Telemedicine, the use of audiovisual technology to provide health care from a remote location, is increasingly used in intensive care units (ICUs). However, studies evaluating the impact of ICU telemedicine show mixed results, with some studies demonstrating improved patient outcomes, while others show limited benefit or even harm. Little is known about the mechanisms that influence variation in ICU telemedicine effectiveness, leaving providers without guidance on how to best use this potentially transformative technology. The Contributors to Effective Critical Care Telemedicine (ConnECCT) study aims to fill this knowledge gap by identifying the clinical and organizational factors associated with variation in ICU telemedicine effectiveness, as well as exploring the clinical contexts and provider perceptions of ICU telemedicine use and its impact on patient outcomes, using a range of qualitative methods. In this report, we describe the study protocol, data collection methods, and planned future analyses of the ConnECCT study. Over the course of 1 year, the study team visited purposefully sampled health systems across the United States that have adopted telemedicine. Data collection methods included direct observations, interviews, focus groups, and artifact collection. Data were collected at the ICUs that provide in-person critical care as well as at the supporting telemedicine units. Iterative thematic content analysis will be used to identify and define key constructs related to telemedicine effectiveness and describe the relationship between them. Ultimately, the study results will provide a framework for more effective implementation of ICU telemedicine, leading to improved clinical outcomes for critically ill patients.