Advancing Quality Improvement with Regression Discontinuity Designs

Effectiveness of an Artificial Intelligence-Enabled Intervention for Detecting Clinical Deterioration

Importance

Inpatient clinical deterioration is associated with substantial morbidity and mortality but may be easily missed by clinicians. Early warning scores have been developed to alert clinicians to patients at high risk of clinical deterioration, but there is limited evidence for their effectiveness.

Objective

To evaluate the effectiveness of an artificial intelligence deterioration model-enabled intervention to reduce the risk of escalations in care among hospitalized patients using a study design that facilitates stronger causal inference.

Design, Setting, and Participants

This cohort study used a regression discontinuity design that controlled for confounding and was based on Epic Deterioration Index (EDI; Epic Systems Corporation) prediction model scores. Compared with other observational research, the regression discontinuity design facilitates causal analysis. Hospitalized adults were included from 4 general internal medicine units in 1 academic hospital from January 17, 2021, through November 16, 2022.

Exposure

An artificial intelligence deterioration model-enabled intervention, consisting of alerts based on an EDI score threshold with an associated collaborative workflow among nurses and physicians.

Main Outcomes and Measures

The primary outcome was escalations in care, including rapid response team activation, transfer to the intensive care unit, or cardiopulmonary arrest during hospitalization.

Results

During the study, 9938 patients were admitted to 1 of the 4 units, with 963 patients (median [IQR] age, 76.1 [64.2-86.2] years; 498 males [52.3%]) included within the primary regression discontinuity analysis. The median (IQR) Elixhauser Comorbidity Index score in the primary analysis cohort was 10 (0-24). The intervention was associated with a -10.4-percentage point (95% CI, -20.1 to -0.8 percentage points; P = .03) absolute risk reduction in the primary outcome for patients at the EDI score threshold. There was no evidence of a discontinuity in measured confounders at the EDI score threshold.

Conclusions and Relevance

Using a regression discontinuity design, this cohort study found that the implementation of an artificial intelligence deterioration model-enabled intervention was associated with a significantly decreased risk of escalations in care among inpatients. These results provide evidence for the effectiveness of this intervention and support its further expansion and testing in other care settings.

Relevance and use of health policy, health systems and health services research for strengthening rehabilitation in real-life settings: methodological considerations

Research on health policy, systems, and services (HPSSR) has seen significant growth in recent decades and received increasing attention in the field of rehabilitation. This growth is driven by the imperative to effectively address real-life challenges in complex healthcare settings. A recent resolution on 'Strengthening rehabilitation in health systems' adopted by the World Health Assembly emphasizes the need to support societal health goals related to rehabilitation, particularly to promote high-quality rehabilitation research, including HPSSR. This conceptual paper, discussed with the participants in the 5th Cochrane Rehabilitation Methodological Meeting held in Milan on September 2023, outlines study designs at diverse levels at which HPSSR studies can be conducted: the macro, meso, and micro levels. It categorizes research questions into four types: those framed from the perspective of policies, healthcare delivery organizations or systems, defined patient or provider populations, and important data sources or research methods. Illustrative examples of appropriate methodologies are provided for each type of research question, demonstrating the potential of HPSSR in shaping policies, improving healthcare delivery, and addressing patient and provider perspectives. The paper concludes by discussing the applicability, usefulness, and implementation of HPSSR findings, and the importance of knowledge translation strategies, drawing insights from implementation science. The goal is to facilitate the integration of research findings into everyday clinical practice to bridge the gap between research and practice in rehabilitation.

The Cystic Fibrosis Learning Network: A mixed methods evaluation of program goals, attributes, and impact

Introduction

The Cystic Fibrosis (CF) Foundation sponsored the design, pilot testing, and implementation of the CF Learning Network (CFLN) to explore how the Foundation's Care Center Network (CCN) could become a learning health system. Six years after the design, the Foundation commissioned a formative mixed methods evaluation of the CFLN to assess: CFLN participants' understanding of program goals, attributes, and perceptions of current and future impact.

Methods

We performed semi-structured interviews with CFLN participants to identify perceived goals, attributes, and impact of the network. Following thematic analyses, we developed and distributed a survey to CFLN members and a matched sample of CCN programs to understand whether the themes were unique to the CFLN.

Results

Interviews with 24 CFLN participants were conducted. Interviewees identified the primary CFLN goal as improving outcomes for people living with CF, with secondary goals of providing training in quality improvement (QI), creating a learning community, engaging all stakeholders in improvement, and spreading best practices to the CCN. Project management, use of data, common QI methods, and the learning community were seen as critical to success. Survey responses were collected from 103 CFLN members and 25 CCN members. The data revealed that CFLN respondents were more likely than CCN respondents to connect with other CF programs, routinely use data for QI, and engage patient and family partners in QI.

Conclusions

Our study suggests that the CFLN provides value beyond that achieved by the CCN. Key questions remain about whether spread of the CFLN could improve outcomes for more people living with CF.

Delivery of Lung-protective Ventilation for Acute Respiratory Distress Syndrome: A Hybrid Implementation-Effectiveness Trial

Rationale: Lung-protective ventilation (LPV) improves outcomes for patients with acute respiratory distress syndrome (ARDS), but adherence remains inadequate. Objectives: To measure the process and clinical impacts of implementation of a science-based intervention to improve LPV adherence for patients with ARDS, in part by increased use of clinical decision support (CDS). Methods: We conducted a type III hybrid implementation/effectiveness pilot trial enrolling adult patients with ARDS admitted to three hospitals before and after the launch of a multimodal implementation intervention to increase the use of mechanical ventilation CDS and improve LPV adherence. The primary outcome was patients' percentage of time adherent to low tidal volume (⩽6.5 ml/kg predicted body weight) ventilation (LTVV). Secondary outcomes included adherence to prescribed oxygenation settings, the use of the CDS tool's independent oxygenation and ventilation components, ventilator-free days, and mortality. Analyses employed multivariable regression to compare adjusted pre- versus postintervention outcomes after the exclusion of a postintervention wash-in period. A sensitivity analysis measured process outcomes' level and trend change postintervention using segmented regression. Results: The 446 included patients had a mean age of 60 years, and 43% were female. Demographic and clinical characteristics were similar pre- versus postintervention. The adjusted proportion of adherent time increased postintervention for LTVV (9.2%; 95% confidence interval [CI], 3.8-14.5%) and prescribed oxygenation settings (11.9%; 95% CI, 7.2-16.5%), as did the probability patients spent ⩾90% of ventilated time on LTVV (adjusted odds ratio [aOR] 2.58; 95% CI, 1.64-4.10) and use of ventilation CDS (aOR, 41.3%; 95% CI, 35.9-46.7%) and oxygenation CDS (aOR, 54.3%; 95% CI, 50.9-57.7%). Ventilator-free days (aOR, 1.15; 95% CI, 0.81-1.62) and 28-day mortality (aOR, 0.78; 95% CI, 0.50-1.20) did not change significantly after intervention. Segmented regression analysis supported a causal relationship between the intervention and improved CDS usage but suggested trends before intervention rather than the studied intervention could explain increased LPV adherence after the intervention. Conclusions: In this pilot trial, a multimodal implementation intervention was associated with increased use of ventilator management CDS for patients with ARDS but was not associated with differences in clinical outcomes and may not have independently caused the observed postintervention improvements in LPV adherence. Clinical trial registered with www.clinicaltrials.gov (NCT03984175).

Role of artificial intelligence in the diagnosis and management of kidney disease: applications to chronic kidney disease and acute kidney injury

PURPOSE OF REVIEW

Chronic kidney disease (CKD) and acute kidney injury (AKI) are global public health problems associated with a significant burden of morbidity, healthcare resource use, and all-cause mortality. This review explores recently published studies that take a machine learning approach to the diagnosis, management, and prognostication in patients with AKI or CKD.

RECENT FINDINGS

The release of novel therapeutics for CKD has highlighted the importance of accurately identifying patients at the highest risk of progression. Many models have been constructed with reasonable predictive accuracy but have not been extensively externally validated and peer reviewed. Similarly, machine learning models have been developed for prediction of AKI and have found sufficient accuracy. There are issues to implementing these models, however, with conflicting results with respect to the relationship between prediction of an AKI outcome and improvements in the occurrence of other adverse events, and in some circumstances potential harm.

SUMMARY

Artificial intelligence models can help guide management of CKD and AKI, but it is important to ensure that they are broadly applicable and generalizable to various settings and associated with improved clinical decision-making and outcomes.

Learning healthcare systems and rapid learning in radiation oncology: Where are we and where are we going?

Learning health systems and rapid-learning are well developed at the conceptual level. The promise of rapidly generating and applying evidence where conventional clinical trials would not usually be practical is attractive in principle. The connectivity of modern digital healthcare information systems and the increasing volumes of data accrued through patients' care pathways offer an ideal platform for the concepts. This is particularly true in radiotherapy where modern treatment planning and image guidance offers a precise digital record of the treatment planned and delivered. The vision is of real-world data, accrued by patients during their routine care, being used to drive programmes of continuous clinical improvement as part of standard practice. This vision, however, is not yet a reality in radiotherapy departments. In this article we review the literature to explore why this is not the case, identify barriers to its implementation, and suggest how wider clinical application might be achieved.

Evaluation of an intervention targeted with predictive analytics to prevent readmissions in an integrated health system: observational study

OBJECTIVES

To determine the associations between a care coordination intervention (the Transitions Program) targeted to patients after hospital discharge and 30 day readmission and mortality in a large, integrated healthcare system.

DESIGN

Observational study.

SETTING

21 hospitals operated by Kaiser Permanente Northern California.

PARTICIPANTS

1 539 285 eligible index hospital admissions corresponding to 739 040 unique patients from June 2010 to December 2018. 411 507 patients were discharged post-implementation of the Transitions Program; 80 424 (19.5%) of these patients were at medium or high predicted risk and were assigned to receive the intervention after discharge.

INTERVENTION

Patients admitted to hospital were automatically assigned to be followed by the Transitions Program in the 30 days post-discharge if their predicted risk of 30 day readmission or mortality was greater than 25% on the basis of electronic health record data.

MAIN OUTCOME MEASURES

Non-elective hospital readmissions and all cause mortality in the 30 days after hospital discharge.

RESULTS

Difference-in-differences estimates indicated that the intervention was associated with significantly reduced odds of 30 day non-elective readmission (adjusted odds ratio 0.91, 95% confidence interval 0.89 to 0.93; absolute risk reduction 95% confidence interval -2.5%, -3.1% to -2.0%) but not with the odds of 30 day post-discharge mortality (1.00, 0.95 to 1.04). Based on the regression discontinuity estimate, the association with readmission was of similar magnitude (absolute risk reduction -2.7%, -3.2% to -2.2%) among patients at medium risk near the risk threshold used for enrollment. However, the regression discontinuity estimate of the association with post-discharge mortality (-0.7% -1.4% to -0.0%) was significant and suggested benefit in this subgroup of patients.

CONCLUSIONS

In an integrated health system, the implementation of a comprehensive readmissions prevention intervention was associated with a reduction in 30 day readmission rates. Moreover, there was no association with 30 day post-discharge mortality, except among medium risk patients, where some evidence for benefit was found. Altogether, the study provides evidence to suggest the effectiveness of readmission prevention interventions in community settings, but further research might be required to confirm the findings beyond this setting.

Awake Prone Positioning Strategy for Nonintubated Hypoxic Patients with COVID-19: A Pilot Trial with Embedded Implementation Evaluation

Rationale: Prone positioning is an appealing therapeutic strategy for nonintubated hypoxic patients with coronavirus disease (COVID-19), but its effectiveness remains to be established in randomized controlled trials.

Objectives: To identify contextual factors relevant to the conduct of a definitive clinical trial evaluating a prone positioning strategy for nonintubated hypoxic patients with COVID-19.

Methods: We conducted a cluster randomized pilot trial at a quaternary care teaching hospital. Five inpatient medical service teams were randomly allocated to two treatment arms: 1) usual care (UC), consisting of current, standard management of hypoxia and COVID-19; or 2) the Awake Prone Positioning Strategy (APPS) plus UC. Included patients had positive severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) testing or suspected COVID-19 pneumonia and oxygen saturation less than 93% or new oxygen requirement of 3 L per minute or greater and no contraindications to prone positioning. Oxygenation measures were collected within 48 hours of eligibility and included nadir oxygen saturation to fraction of inspired oxygen (S/F) ratio and time spent with S/F ratio less than 315. Concurrently, we conducted an embedded implementation evaluation using semistructured interviews with clinician and patient participants to determine contextual factors relevant to the successful conduct of a future clinical trial. The primary outcomes were drawn from an implementation science framework including acceptability, adoption, appropriateness, effectiveness, equity, feasibility, fidelity, and penetration.

Results: Forty patients were included in the cluster randomized trial. Patients in the UC group (n = 13) had a median nadir S/F ratio over the 48-hour study period of 216 (95% confidence interval [95% CI], 95–303) versus 253 (95% CI, 197–267) in the APPS group (n = 27). Patients in the UC group spent 42 hours (95% CI, 13–47) of the 48-hour study period with an S/F ratio below 315 versus 20 hours (95% CI, 6–39) for patients in the APPS group. Mixed-methods analyses uncovered several barriers relevant to the conduct of a successful definitive randomized controlled trial, including low adherence to prone positioning, large differences between physician-recommended and patient-tolerated prone durations, and diffusion of prone positioning into usual care.

Conclusions: A definitive trial evaluating the effect of prone positioning in nonintubated patients with COVID-19 is warranted, but several barriers must be addressed to ensure that the results of such a trial are informative and readily translated into practice.

Institution of an emergency department "swarming" care model and sepsis door-to-antibiotic time: A quasi-experimental retrospective analysis

Background

Prompt sepsis treatment is associated with improved outcomes but requires a complex series of actions by multiple clinicians. We investigated whether simply reorganizing emergency department (ED) care to expedite patients’ initial evaluation was associated with shorter sepsis door-to-antibiotic times.

Methods

Patients eligible for this retrospective study received IV antibiotics and demonstrated acute organ failure after presenting to one of three EDs in Utah. On May 1, 2016, the intervention ED instituted “swarming” as the default model for initial evaluation of all mid- and low-acuity patients. Swarming involved simultaneous patient evaluation by the ED physician, nurse, and technician followed by a team discussion of the initial care plan. Care was unchanged at the two control EDs. A 30-day wash-in period separated the baseline (May 16, 2015 to April 15, 2016) and post-intervention (May 16, 2016 to November 15, 2016) analysis periods. We conducted a quasi-experimental analysis comparing door-to-antibiotic time for sepsis patients at the intervention ED after versus before care reorganization, applying difference-in-differences methods to control for trends in door-to-antibiotic time unrelated to the studied intervention and multivariable regression to adjust for patient characteristics.

Results

The analysis included 3,230 ED sepsis patients, including 1,406 from the intervention ED. Adjusted analyses using difference-in-differences methods to control for temporal trends unrelated to the studied intervention revealed no significant change in door-to-antibiotic time after care reorganization (-7 minutes, 95% CI -20 to 6 minutes, p = 0.29). Multivariable pre/post analyses using data only from the intervention ED overestimated the magnitude and statistical significance of outcome changes associated with ED care reorganization.

Conclusions

Implementation of an ED care model involving parallel multidisciplinary assessment and early team discussion of the care plan was not associated with improvements in mid- and low-acuity sepsis patients’ door-to-antibiotic time after accounting for changes in the outcome unrelated to the studied intervention.

Novel tools for a learning health system: a combined difference-in-difference/regression discontinuity approach to evaluate effectiveness of a readmission reduction initiative

Current methods used to evaluate the effects of healthcare improvement efforts have limitations. Designs with strong causal inference—such as individual patient or cluster randomisation—can be inappropriate and infeasible to use in single-centre settings. Simpler designs—such as prepost studies—are unable to infer causal relationships between improvement interventions and outcomes of interest, often leading to spurious conclusions regarding programme success. Other designs, such as regression discontinuity or difference-in-difference (DD) approaches alone, require multiple assumptions that are often unable to be met in real world improvement settings. We present a case study of a novel design in improvement and implementation research—a hybrid regression discontinuity/DD design—that leverages risk-targeted improvement interventions within a hospital readmission reduction programme. We demonstrate how the hybrid regression discontinuity-DD approach addresses many of the limitations of either method alone, and represents a useful method to evaluate the effects of multiple, simultaneous heath system improvement activities—a necessary capacity of a learning health system. Finally, we discuss some of the limitations of the hybrid regression discontinuity-DD approach, including the need to assign patients to interventions based upon a continuous measure, the need for large sample sizes, and potential susceptibility of risk-based intervention assignment to gaming.