Integrating the Practical Robust Implementation and Sustainability Model With Best Practices in Clinical Decision Support Design: Implementation Science Approach

A Qualitative Assessment of South Africa's Central Chronic Medication Dispensing and Distribution Program for Differentiated Antiretroviral Therapy Delivery in Umlazi Township, South Africa: Client Perspectives after 12 Months of Participation

South Africa's Central Chronic Medicine Dispensing and Distribution (CCMDD) program provides community-based medication delivery for clinically stable people with HIV (PWH) on antiretroviral therapy (ART). To evaluate CCMDD implementation, we conducted semi-structured interviews with 60 PWH enrolled in CCMDD for at least 12 months. In a directed content analysis based on the Practical, Robust Implementation and Sustainability Model (PRISM) implementation science framework, key themes were compared with qualitative data collected from PWH enrolling in CCMDD at an earlier time-point. Results indicated consistently positive views of CCMDD, primarily attributed to convenient and smooth medication pick-up. At the later (vs. earlier) time-point, participants discussed less clinic crowding, mentioned few medication errors, and noted improved communication around refills. Community HIV stigma was a persistent challenge, as was nurses' judgmental communication style. To ensure CCMDD's success, continued focus is needed on decreasing HIV stigma beyond the clinic context and improving provider-patient relationships.

Interventions and implementation considerations for reducing pre-treatment loss to follow-up in adults with pulmonary tuberculosis: A scoping review

Background

Tuberculosis (TB) is a leading cause of death worldwide with over 90% of reported cases occurring in low- and middle-income countries (LMICs). Pre-treatment loss to follow-up (PTLFU) is a key contributor to TB mortality and infection transmission.

Objectives

We performed a scoping review to map available evidence on interventions to reduce PTLFU in adults with pulmonary TB, identify gaps in existing knowledge, and develop a conceptual framework to guide intervention implementation.

Methods

We searched eight electronic databases up to February 6 2024, medRxiv for pre-prints, and reference lists of included studies. Two review authors independently selected studies and extracted data using a predesigned form. We analysed data descriptively, presented findings in a narrative summary and developed a conceptual framework based on the Practical, Robust Implementation, and Sustainability Model to map the factors for effective intervention implementation.

Results

We reviewed 1262 records and included 17 studies. Most studies were randomized controlled trials (8/17, 47%). Intervention barriers included stigma and inadequate resources; enablers included mobile phones and TB testing and results on the same day. We identified eight interventions that reduced PTLFU: treatment support groups; mobile notifications; community health workers; integrated HIV/TB services; Xpert MTB/RIF as the initial diagnostic test; computer-aided detection with chest radiography screening; active linkage to care; and multi-component strategies.

Conclusion

Given the variation of healthcare settings, TB programs should consider contextual factors such as user acceptability, political commitment, resources, and infrastructure before adopting an intervention. Future research should utilize qualitative study designs, be people-centred, and include social and economic factors affecting PTLFU.

A scoping review, novel taxonomy and catalogue of implementation frameworks for clinical decision support systems

BACKGROUND

The primary aim of this scoping review was to synthesise key domains and sub-domains described in existing clinical decision support systems (CDSS) implementation frameworks into a novel taxonomy and demonstrate most-studied and least-studied areas. Secondary objectives were to evaluate the frequency and manner of use of each framework, and catalogue frameworks by implementation stage.

METHODS

A scoping review of Pubmed, Scopus, Web of Science, PsychInfo and Embase was conducted on 12/01/2022, limited to English language, including 2000-2021. Each framework was categorised as addressing one or multiple stages of implementation: design and development, evaluation, acceptance and integration, and adoption and maintenance. Key parts of each framework were grouped into domains and sub-domains.

RESULTS

Of 3550 titles identified, 58 papers were included. The most-studied implementation stage was acceptance and integration, while the least-studied was design and development. The three main framework uses were: for evaluating adoption, for understanding attitudes toward implementation, and for framework validation. The most frequently used framework was the Consolidated Framework for Implementation Research.

CONCLUSIONS

Many frameworks have been published to overcome barriers to CDSS implementation and offer guidance towards successful adoption. However, for co-developers, choosing relevant frameworks may be a challenge. A taxonomy of domains addressed by CDSS implementation frameworks is provided, as well as a description of their use, and a catalogue of frameworks listed by the implementation stages they address. Future work should ensure best practices for CDSS design are adequately described, and existing frameworks are well-validated. An emphasis on collaboration between clinician and non-clinician affected parties may help advance the field.

Applying the Non-Adoption, Abandonment, Scale-up, Spread, and Sustainability Framework Across Implementation Stages to Identify Key Strategies to Facilitate Clinical Decision Support System Integration Within a Large Metropolitan Health Service: Interview and Focus Group Study

BACKGROUND

Computerized clinical decision support systems (CDSSs) enhance patient care through real-time, evidence-based guidance for health care professionals. Despite this, the effective implementation of these systems for health services presents multifaceted challenges, leading to inappropriate use and abandonment over the course of time. Using the Non-Adoption, Abandonment, Scale-Up, Spread, and Sustainability (NASSS) framework, this qualitative study examined CDSS adoption in a metropolitan health service, identifying determinants across implementation stages to optimize CDSS integration into health care practice.

OBJECTIVE

This study aims to identify the theory-informed (NASSS) determinants, which included multiple CDSS interventions across a 2-year period, both at the health-service level and at the individual hospital setting, that either facilitate or hinder the application of CDSSs within a metropolitan health service. In addition, this study aimed to map these determinants onto specific stages of the implementation process, thereby developing a system-level understanding of CDSS application across implementation stages.

METHODS

Participants involved in various stages of the implementation process were recruited (N=30). Participants took part in interviews and focus groups. We used a hybrid inductive-deductive qualitative content analysis and a framework mapping approach to categorize findings into barriers, enablers, or neutral determinants aligned to NASSS framework domains. These determinants were also mapped to implementation stages using the Active Implementation Framework stages approach.

RESULTS

Participants comprised clinical adopters (14/30, 47%), organizational champions (5/30, 16%), and those with roles in organizational clinical informatics (5/30, 16%). Most determinants were mapped to the organization level, technology, and adopter subdomains. However, the study findings also demonstrated a relative lack of long-term implementation planning. Consequently, determinants were not uniformly distributed across the stages of implementation, with 61.1% (77/126) identified in the exploration stage, 30.9% (39/126) in the full implementation stage, and 4.7% (6/126) in the installation stages. Stakeholders engaged in more preimplementation and full-scale implementation activities, with fewer cycles of monitoring and iteration activities identified.

CONCLUSIONS

These findings addressed a substantial knowledge gap in the literature using systems thinking principles to identify the interdependent dynamics of CDSS implementation. A lack of sustained implementation strategies (ie, training and longer-term, adopter-level championing) weakened the sociotechnical network between developers and adopters, leading to communication barriers. More rigorous implementation planning, encompassing all 4 implementation stages, may, in a way, help in addressing the barriers identified and enhancing enablers.

Applying Implementation Science to Advance Electronic Health Record-Driven Learning Health Systems: Case Studies, Challenges, and Recommendations

With the widespread implementation of electronic health records (EHRs), there has been significant progress in developing learning health systems (LHSs) aimed at improving health and health care delivery through rapid and continuous knowledge generation and translation. To support LHSs in achieving these goals, implementation science (IS) and its frameworks are increasingly being leveraged to ensure that LHSs are feasible, rapid, iterative, reliable, reproducible, equitable, and sustainable. However, 6 key challenges limit the application of IS to EHR-driven LHSs: barriers to team science, limited IS experience, data and technology limitations, time and resource constraints, the appropriateness of certain IS approaches, and equity considerations. Using 3 case studies from diverse health settings and 1 IS framework, we illustrate these challenges faced by LHSs and offer solutions to overcome the bottlenecks in applying IS and utilizing EHRs, which often stymie LHS progress. We discuss the lessons learned and provide recommendations for future research and practice, including the need for more guidance on the practical application of IS methods and a renewed emphasis on generating and accessing inclusive data.

Bridging clinical informatics and implementation science to improve cancer symptom management in ambulatory oncology practices: experiences from the IMPACT consortium

Objectives

To report lessons from integrating the methods and perspectives of clinical informatics (CI) and implementation science (IS) in the context of Improving the Management of symPtoms during and following Cancer Treatment (IMPACT) Consortium pragmatic trials.

Materials and Methods

IMPACT informaticists, trialists, and implementation scientists met to identify challenges and solutions by examining robust case examples from 3 Research Centers that are deploying systematic symptom assessment and management interventions via electronic health records (EHRs). Investigators discussed data collection and CI challenges, implementation strategies, and lessons learned.

Results

CI implementation strategies and EHRs systems were utilized to collect and act upon symptoms and impairments in functioning via electronic patient-reported outcomes (ePRO) captured in ambulatory oncology settings. Limited EHR functionality and data collection capabilities constrained the ability to address IS questions. Collecting ePRO data required significant planning and organizational champions adept at navigating ambiguity.

Discussion

Bringing together CI and IS perspectives offers critical opportunities for monitoring and managing cancer symptoms via ePROs. Discussions between CI and IS researchers identified and addressed gaps between applied informatics implementation and theory-based IS trial and evaluation methods. The use of common terminology may foster shared mental models between CI and IS communities to enhance EHR design to more effectively facilitate ePRO implementation and clinical responses.

Conclusion

Implementation of ePROs in ambulatory oncology clinics benefits from common understanding of the concepts, lexicon, and incentives between CI implementers and IS researchers to facilitate and measure the results of implementation efforts.

Artificial Intelligence and Machine Learning for Inborn Errors of Immunity: Current State and Future Promise

Artificial intelligence (AI) and machine learning (ML) research within medicine has exponentially increased over the last decade, with studies showcasing the potential of AI/ML algorithms to improve clinical practice and outcomes. Ongoing research and efforts to develop AI-based models have expanded to aid in the identification of inborn errors of immunity (IEI). The use of larger electronic health record data sets, coupled with advances in phenotyping precision and enhancements in ML techniques, has the potential to significantly improve the early recognition of IEI, thereby increasing access to equitable care. In this review, we provide a comprehensive examination of AI/ML for IEI, covering the spectrum from data preprocessing for AI/ML analysis to current applications within immunology, and address the challenges associated with implementing clinical decision support systems to refine the diagnosis and management of IEI.

Implementation and evaluation of a proactive consultation-liaison model on an inpatient obstetric unit

OBJECTIVE

Perinatal mental and anxiety disorders (PMADs) contribute to adverse health outcomes, though they are underrecognized and undertreated. Inpatient obstetric settings represent a unique opportunity for behavioral health engagement, including screening, brief treatment, and referrals for outpatient care. The proactive consultation-liaison (CL) model has proven effective in general hospital settings but is not well-studied in obstetric settings. This article describes the implementation and evaluation of a proactive CL model in an inpatient obstetric unit within a tertiary medical center.

METHODS

We implemented a multidisciplinary, proactive CL model in an inpatient obstetric unit with the purpose of identifying patients at risk for or experiencing PMADs and providing intervention and/or referral to treatment. Systematic screening of 7322 admitted patients was performed over a 17-month period to identify eligible patients for behavioral health consultation. Consultation data was retrospectively extracted from the electronic medical record. Key implementation outcomes were assessed using a RE-AIM measures (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework.

RESULTS

1589 initial consults were conducted by the multidisciplinary team, yielding a consult rate of 21.7 %. The majority of consults (94 %) were completed by a social worker or psychologist, with most patients identified for consultation at multidisciplinary rounds (60.7 %). The most common indications for consultation with a psychiatrist included medication management, history of bipolar disorder, and history of anxiety. All invited staff and providers participated in the model. Alternative funding sources agreed to cover the salaries of the multidisciplinary team following conclusion of pilot grant funding.

CONCLUSIONS

A proactive CL model implemented in an inpatient obstetric unit led to a higher consult rate (21.7 %) than is observed with traditional CL services. A multidisciplinary proactive CL model shows promise in identifying people at-risk for PMADs and providing targeted interventions to prevent PMADs and treat those with active symptoms.

The Application and Evolution of the Practical, Robust Implementation and Sustainability Model (PRISM): History and Innovations

Implementation science theories, models, and frameworks (TMF) should help users understand complex issues in translating research into practice, guide selection of appropriate implementation strategies, and evaluate implementation outcomes. They should also be sensitive to evidence from projects that apply the framework, evolve based on those experiences, and be accessible to a range of users. This paper describes these issues as they relate to the Practical, Robust Implementation and Sustainability Model (PRISM). PRISM was created to assess key multilevel contextual factors related to the reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) outcomes of health interventions. We describe key aspects of PRISM and how it has been applied, evolved, and adapted across settings, time, and content areas. Since its development in 2008 PRISM has been used in over 200 publications, with increased use in recent years. It has been used for a wide variety of purposes and more recent applications have focused on increasing its accessibility for non-researcher groups and more rapid and iterative application for use in learning heath systems. PRISM has been applied to address health equity issues including representation, representativeness, and co-creation activities in both US and non-US settings. We describe common types of adaptations made by implementation teams when applying PRISM to fit with the resources and priorities of diverse and low-resource settings. We conclude by summarizing lessons learned and providing recommendations for future research and practice using PRISM.

"Following the data": perceptions of and willingness to use clinical decision support tools to inform HIV care among Haitian clinicians

Background

Clinical decision support (CDS) tools can support HIV care, including through case tracking, treatment and medication monitoring, and promoting provider compliance with care guidelines. There has been limited research into the technical, organizational, and behavioral factors that impact perceptions of and willingness to use CDS tools at scale in resource-limited settings, including in Haiti.

Methods

Our sample included fifteen purposively chosen Haitian HIV program experts, including active clinicians and HIV program managers. Participants completed structured quantitative surveys and one-on-one qualitative semi-structured interviews.

Results

Study participants had high levels of familiarity and experience with CDS tools. The primary motivator for CDS tool use was a perceived benefit to quality of care, including improved provider time use, efficiency, and decision-making ability, and patient outcomes. Participants highlighted decision-making autonomy and how CDS tools could support provider decision making but should not supplant provider knowledge and experience. Participants highlighted the need for sufficient provider training/sensitization, inclusion of providers in the system design process, and prioritization of tool user-friendliness as key mechanisms to drive tool use and impact. Some participants noted that systemic issues, such as limited laboratory capacity, may reduce the usefulness of CDS alerts, particularly concerning differentiated care and priority viral load testing.

Conclusion

Respondents had largely positive perceptions of EMRs and CDS tools, particularly due to perceived improvements in quality of care. To improve tool use, stakeholders should prioritize tool user-friendliness and provider training. Addressing systemic health system issues is necessary to unlock the full potential of these tools.

Use of the iPRISM webtool in a learning community to assess implementation context and fit of a novel clinical decision support tool for physical therapy triage in acute care hospitals

BACKGROUND

The iPRISM webtool is an interactive tool designed to aid the process of applying the Practical, Robust Implementation and Sustainability Model (PRISM) for the assessment of and fit with context. A learning community (LC) is a multidisciplinary group of partners addressing a complex problem. Our LC coproduced the Physical TheraPy frEqueNcy Clinical decIsion support tooL (PT-PENCIL) to guide the use of physical therapist services in acute care hospitals.

OBJECTIVE

To describe our LC's activities to co-produce the PT-PENCIL, use of the iPRISM webtool to assess its preimplementation context and fit, and develop a multicomponent implementation strategy for the PT-PENCIL.

DESIGN

A descriptive research design.

SETTING

Three tertiary care hospitals.

PARTICIPANTS

Thirteen LC partners: six clinical physical therapists, three rehabilitation managers, three researchers, and a bioinformaticist.

INTERVENTIONS

Not applicable.

OUTCOME MEASURES

Using the iPRISM webtool, expected fit of the PT-PENCIL was rated 1 (not aligned) to 6 (well aligned) for each PRISM domain and expected reach, effectiveness, adoption, implementation, and maintenance were rated 1 (not likely at all) to 6 (very likely). Discrete implementation strategies were identified from the Expert Recommendations for Implementing Change.

RESULTS

The process spanned 18 meetings over 8 months. Ten LC partners completed the iPRISM webtool. PRISM domains with the lowest expected alignment were the "implementation and sustainability infrastructure" (mean = 4.7 out of 6; range = 3-6) and the "external environment" (mean = 4.9 of 6; range = 4-6). Adoption was the outcome with the lowest expected likelihood (mean = 4.5 out of 6; range = 1-6). Six discrete implementation strategies were identified and combined into a multicomponent strategy.

CONCLUSIONS

Within a LC, we used existing implementation science resources to co-produce a novel clinical decision support tool for acute care physical therapists and develop a strategy for its implementation. Our methodology can be replicated for similar projects given the public availability of each resource used.

Reach, Adoption, and Implementation Strategies of a Telehealth Fall Prevention Program: Perspectives From Francophone Communities Across Canada

Introduction. A fall may impact a person's physical, emotional, and psychological well-being. Fall prevention programs are being implemented to reduce these negative outcomes. However, linguistic barriers in health services may reduce access to such prevention programs. A telehealth fall prevention program was designed to increase access to such programs in French for Francophone minority communities in Canada. This capacity-building project aimed to support community partners to deliver this telehealth program and document strategies used to reach, adopt, and implement the program within various Francophone and Acadian Minority Communities. Methods. A sequential explanatory mixed methodology was used to document reach, adoption, and implementation strategies and describe the lived experiences of program facilitators and organization representatives. Reach, adoption, and implementation were documented and analyzed descriptively, while lived experiences were analyzed using content analysis following the Consortium Framework for Implementation Research. Results. Twelve organization representatives or program facilitators from eight organizations operating in four different provinces participated in the study. Three themes emerged from the qualitative data on reach and adoption: external context, internal context, and capacity building. Four themes were identified as barriers and facilitators to implementation: level of preparation and time management, interpersonal relations and telepresence, exercise facilitation and safety, and technological problem-solving. Conclusion. Using tailored reach and adoption strategies such as prioritizing provinces with higher proportions of needs and training local community program facilitators may lead to the successful implementation of a new telehealth fall prevention program. Results from this study could potentially inform other primary prevention programs or telehealth program implementation.

Clinical Decision Support Requirements for Ventricular Tachycardia Diagnosis Within the Frameworks of Knowledge and Practice: Survey Study

BACKGROUND

Ventricular tachycardia (VT) diagnosis is challenging due to the similarity between VT and some forms of supraventricular tachycardia, complexity of clinical manifestations, heterogeneity of underlying diseases, and potential for life-threatening hemodynamic instability. Clinical decision support systems (CDSSs) have emerged as promising tools to augment the diagnostic capabilities of cardiologists. However, a requirements analysis is acknowledged to be vital for the success of a CDSS, especially for complex clinical tasks such as VT diagnosis.

OBJECTIVE

The aims of this study were to analyze the requirements for a VT diagnosis CDSS within the frameworks of knowledge and practice and to determine the clinical decision support (CDS) needs.

METHODS

Our multidisciplinary team first conducted semistructured interviews with seven cardiologists related to the clinical challenges of VT and expected decision support. A questionnaire was designed by the multidisciplinary team based on the results of interviews. The questionnaire was divided into four sections: demographic information, knowledge assessment, practice assessment, and CDS needs. The practice section consisted of two simulated cases for a total score of 10 marks. Online questionnaires were disseminated to registered cardiologists across China from December 2022 to February 2023. The scores for the practice section were summarized as continuous variables, using the mean, median, and range. The knowledge and CDS needs sections were assessed using a 4-point Likert scale without a neutral option. Kruskal-Wallis tests were performed to investigate the relationship between scores and practice years or specialty.

RESULTS

Of the 687 cardiologists who completed the questionnaire, 567 responses were eligible for further analysis. The results of the knowledge assessment showed that 383 cardiologists (68%) lacked knowledge in diagnostic evaluation. The overall average score of the practice assessment was 6.11 (SD 0.55); the etiological diagnosis section had the highest overall scores (mean 6.74, SD 1.75), whereas the diagnostic evaluation section had the lowest scores (mean 5.78, SD 1.19). A majority of cardiologists (344/567, 60.7%) reported the need for a CDSS. There was a significant difference in practice competency scores between general cardiologists and arrhythmia specialists (P=.02).

CONCLUSIONS

There was a notable deficiency in the knowledge and practice of VT among Chinese cardiologists. Specific knowledge and practice support requirements were identified, which provide a foundation for further development and optimization of a CDSS. Moreover, it is important to consider clinicians' specialization levels and years of practice for effective and personalized support.

Improving Health Equity in Living Donor Kidney Transplant: Application of an Implementation Science Framework

BACKGROUND

Interventions to improve racial equity in access to living donor kidney transplants (LDKT) have focused primarily on patients, ignoring the contributions of clinicians, transplant centers, and health system factors. Obtaining access to LDKT is a complex, multi-step process involving patients, their families, clinicians, and health system functions. An implementation science framework can help elucidate multi-level barriers to achieving racial equity in LDKT and guide the implementation of interventions targeted at all levels.

METHODS

We adopted the Pragmatic Robust Implementation and Sustainability Model (PRISM), an implementation science framework for racial equity in LDKT. The purpose was to provide a guide for assessment, inform intervention design, and support planning for the implementation of interventions.

RESULTS

We applied 4 main PRISM domains to racial equity in LDKT: Organizational Characteristics, Program Components, External Environment, and Patient Characteristics. We specified elements within each domain that consider perspectives of the health system, transplant center, clinical staff, and patients.

CONCLUSION

The applied PRISM framework provides a foundation for the examination of multi-level influences across the entirety of LDKT care. Researchers, quality improvement staff, and clinicians can use the applied PRISM framework to guide the assessment of inequities, support collaborative intervention development, monitor intervention implementation, and inform resource allocation to improve equity in access to LDKT.

Management of Fall Risk Among Older Adults in Diverse Primary Care Settings

OBJECTIVES

Falls are persistent among community-dwelling older adults despite existing prevention guidelines. We described how urban and rural primary care staff and older adults manage fall risk and factors important to integration of computerized clinical decision support (CCDS).

METHODS

Interviews, contextual inquiries, and workflow observations were analyzed using content analysis and synthesized into a journey map. Sociotechnical and PRISM domains were applied to identify workflow factors important to sustainable CCDS integration.

RESULTS

Participants valued fall prevention and described similar approaches. Available resources differed between rural and urban locations. Participants wanted evidence-based guidance integrated into workflows to bridge skills gaps.

DISCUSSION

Sites described similar clinical approaches with differences in resource availability. This implies that a single intervention would need to be flexible to environments with differing resources. Electronic Health Record's inherent ability to provide tailored CCDS is limited. However, CCDS middleware could integrate into different settings and increase evidence use.

Sustained Effect of Clinical Decision Support for Heart Failure: A Natural Experiment Using Implementation Science

OBJECTIVES

In a randomized controlled trial, we found that applying implementation science (IS) methods and best practices in clinical decision support (CDS) design to create a locally customized, "enhanced" CDS significantly improved evidence-based prescribing of β blockers (BB) for heart failure compared with an unmodified commercially available CDS. At trial conclusion, the enhanced CDS was expanded to all sites. The purpose of this study was to evaluate the real-world sustained effect of the enhanced CDS compared with the commercial CDS.

METHODS

In this natural experiment of 28 primary care clinics, we compared clinics exposed to the commercial CDS (preperiod) to clinics exposed to the enhanced CDS (both periods). The primary effectiveness outcome was the proportion of alerts resulting in a BB prescription. Secondary outcomes included patient reach and clinician adoption (dismissals).

RESULTS

There were 367 alerts for 183 unique patients and 171 unique clinicians (pre: March 2019-August 2019; post: October 2019-March 2020). The enhanced CDS increased prescribing by 26.1% compared with the commercial (95% confidence interval [CI]: 17.0-35.1%), which is consistent with the 24% increase in the previous study. The odds of adopting the enhanced CDS was 81% compared with 29% with the commercial (odds ratio: 4.17, 95% CI: 1.96-8.85). The enhanced CDS adoption and effectiveness rates were 62 and 14% in the preperiod and 92 and 10% in the postperiod.

CONCLUSION

Applying IS methods with CDS best practices was associated with improved and sustained clinician adoption and effectiveness compared with a commercially available CDS tool.

Comparative effectiveness of generic commercial versus locally customized clinical decision support tools to reduce prescription of nonsteroidal anti-inflammatory drugs for patients with heart failure

OBJECTIVE

To compare the effectiveness of 2 clinical decision support (CDS) tools to avoid prescription of nonsteroidal anti-inflammatory drugs (NSAIDs) in patients with heart failure (HF): a "commercial" and a locally "customized" alert.

METHODS

We conducted a retrospective cohort study of 2 CDS tools implemented within a large integrated health system. The commercial CDS tool was designed according to third-party drug content and EHR vendor specifications. The customized CDS tool underwent a user-centered design process informed by implementation science principles, with input from a cross disciplinary team. The customized CDS tool replaced the commercial CDS tool. Data were collected from the electronic health record via analytic reports and manual chart review. The primary outcome was effectiveness, defined as whether the clinician changed their behavior and did not prescribe an NSAID.

RESULTS

A random sample of 366 alerts (183 per CDS tool) was evaluated that represented 355 unique patients. The commercial CDS tool was effective for 7 of 172 (4%) patients, while the customized CDS tool was effective for 81 of 183 (44%) patients. After adjusting for age, chronic kidney disease, ejection fraction, NYHA class, concurrent prescription of an opioid or acetaminophen, visit type (inpatient or outpatient), and clinician specialty, the customized alerts were at 24.3 times greater odds of effectiveness compared to the commercial alerts (OR: 24.3 CI: 10.20-58.06).

CONCLUSION

Investing additional resources to customize a CDS tool resulted in a CDS tool that was more effective at reducing the total number of NSAID orders placed for patients with HF compared to a commercially available CDS tool.

Examining Care Planning Efficiency and Clinical Decision Support Adoption in a System Tailoring to Nurses' Graph Literacy: National, Web-Based Randomized Controlled Trial

BACKGROUND

The proliferation of health care data in electronic health records (EHRs) is fueling the need for clinical decision support (CDS) that ensures accuracy and reduces cognitive processing and documentation burden. The CDS format can play a key role in achieving the desired outcomes. Building on our laboratory-based pilot study with 60 registered nurses (RNs) from 1 Midwest US metropolitan area indicating the importance of graph literacy (GL), we conducted a fully powered, innovative, national, and web-based randomized controlled trial with 203 RNs.

OBJECTIVE

This study aimed to compare care planning time (CPT) and the adoption of evidence-based CDS recommendations by RNs randomly assigned to 1 of 4 CDS format groups: text only (TO), text+table (TT), text+graph (TG), and tailored (based on the RN's GL score). We hypothesized that the tailored CDS group will have faster CPT (primary) and higher adoption rates (secondary) than the 3 nontailored CDS groups.

METHODS

Eligible RNs employed in an adult hospital unit within the past 2 years were recruited randomly from 10 State Board of Nursing lists representing the 5 regions of the United States (Northeast, Southeast, Midwest, Southwest, and West) to participate in a randomized controlled trial. RNs were randomly assigned to 1 of 4 CDS format groups-TO, TT, TG, and tailored (based on the RN's GL score)-and interacted with the intervention on their PCs. Regression analysis was performed to estimate the effect of tailoring and the association between CPT and RN characteristics.

RESULTS

The differences between the tailored (n=46) and nontailored (TO, n=55; TT, n=54; and TG, n=48) CDS groups were not significant for either the CPT or the CDS adoption rate. RNs with low GL had longer CPT interacting with the TG CDS format than the TO CDS format (P=.01). The CPT in the TG CDS format was associated with age (P=.02), GL (P=.02), and comfort with EHRs (P=.047). Comfort with EHRs was also associated with CPT in the TT CDS format (P<.001).

CONCLUSIONS

Although tailoring based on GL did not improve CPT or adoption, the study reinforced previous pilot findings that low GL is associated with longer CPT when graphs were included in care planning CDS. Higher GL, younger age, and comfort with EHRs were associated with shorter CPT. These findings are robust based on our new innovative testing strategy in which a diverse national sample of RN participants (randomly derived from 10 State Board of Nursing lists) interacted on the web with the intervention on their PCs. Future studies applying our innovative methodology are recommended to cost-effectively enhance the understanding of how the RN's GL, combined with additional factors, can inform the development of efficient CDS for care planning and other EHR components before use in practice.

Around the EQUATOR with clinician-scientists transdisciplinary aging research (Clin-STAR) principles: Implementation science challenges and opportunities

The Institute of Medicine and the National Institute on Aging increasingly understand that knowledge alone is necessary but insufficient to improve healthcare outcomes. Adapting the behaviors of clinicians, patients, and stakeholders to new standards of evidence-based clinical practice is often significantly delayed. In response, over the past twenty years, Implementation Science has developed as the study of methods and strategies that facilitate the uptake of evidence-based practice into regular use by practitioners and policymakers. One important advance in Implementation Science research was the development of Standards for Reporting Implementation Studies (StaRI), which provided a 27-item checklist for researchers to consistently report essential elements of the implementation and intervention strategies. Using StaRI as a framework, this review discusses specific Implementation Science challenges for research with older adults, provides solutions for those obstacles, and opportunities to improve the value of this evolving approach to reduce the knowledge translation losses that exist between published research and clinical practice.

How Dissemination and Implementation Science Can Contribute to the Advancement of Learning Health Systems

Many health systems are working to become learning health systems (LHSs), which aim to improve the value of health care by rapidly, continuously generating evidence to apply to practice. However, challenges remain to advance toward the aspirational goal of becoming a fully mature LHS. While some important challenges have been well described (i.e., building system-level supporting infrastructure and the accessibility of inclusive, integrated, and actionable data), other key challenges are underrecognized, including balancing evaluation rapidity with rigor, applying principles of health equity and classic ethics, focusing on external validity and reproducibility (generalizability), and designing for sustainability. Many LHSs focus on continuous learning cycles, but with limited consideration of issues related to the rapidity of these learning cycles, as well as the sustainability or generalizability of solutions. Some types of data have been consistently underrepresented, including patient-reported outcomes and preferences, social determinants, and behavioral and environmental data, the absence of which can exacerbate health disparities. A promising approach to addressing many challenges that LHSs face may be found in dissemination and implementation (D&I) science. With an emphasis on multilevel dynamic contextual factors, representation of implementation partner engagement, pragmatic research, sustainability, and generalizability, D&I science methods can assist in overcoming many of the challenges facing LHSs. In this article, the authors describe the current state of LHSs and challenges to becoming a mature LHS, propose solutions to current challenges, focusing on the contributions of D&I science with other methods, and propose key components and characteristics of a mature LHS model that others can use to plan and develop their LHSs.

Research and Scholarly Methods: Implementation Science Studies

Traditional research focuses on efficacy or effectiveness of interventions but lacks evaluation of strategies needed for equitable uptake, scalable implementation, and sustainable evidence-based practice transformation. The purpose of this introductory review is to describe key implementation science (IS) concepts as they apply to medication management and pharmacy practice, and to provide guidance on literature review with an IS lens. There are five key ingredients of IS, including: (1) evidence-based intervention; (2) implementation strategies; (3) IS theory, model, or framework; (4) IS outcomes and measures; and (5) stakeholder engagement, which is key to a successful implementation. These key ingredients apply across the three stages of IS research: (1) pre-implementation; (2) implementation; and (3) sustainment. A case example using a combination of IS models, PRISM (Practical, Robust Implementation and Sustainability model) and RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance), is included to describe how an IS study is designed and conducted. This case is a cluster randomized trial comparing two clinical decision support tools to improve guideline-concordant prescribing for patients with heart failure and reduced ejection fraction. The review also includes information on the Standards for Reporting Implementation Studies (StaRI), which is used for literature review and reporting of IS studies,as well as IS-related learning resources.

A Personalized Ontology-Based Decision Support System for Complex Chronic Patients: Retrospective Observational Study

BACKGROUND

Due to an increase in life expectancy, the prevalence of chronic diseases is also on the rise. Clinical practice guidelines (CPGs) provide recommendations for suitable interventions regarding different chronic diseases, but a deficiency in the implementation of these CPGs has been identified. The PITeS-TiiSS (Telemedicine and eHealth Innovation Platform: Information Communications Technology for Research and Information Challenges in Health Services) tool, a personalized ontology-based clinical decision support system (CDSS), aims to reduce variability, prevent errors, and consider interactions between different CPG recommendations, among other benefits.

OBJECTIVE

The aim of this study is to design, develop, and validate an ontology-based CDSS that provides personalized recommendations related to drug prescription. The target population is older adult patients with chronic diseases and polypharmacy, and the goal is to reduce complications related to these types of conditions while offering integrated care.

METHODS

A study scenario about atrial fibrillation and treatment with anticoagulants was selected to validate the tool. After this, a series of knowledge sources were identified, including CPGs, PROFUND index, LESS/CHRON criteria, and STOPP/START criteria, to extract the information. Modeling was carried out using an ontology, and mapping was done with Health Level 7 Fast Healthcare Interoperability Resources (HL7 FHIR) and Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT; International Health Terminology Standards Development Organisation). Once the CDSS was developed, validation was carried out by using a retrospective case study.

RESULTS

This project was funded in January 2015 and approved by the Virgen del Rocio University Hospital ethics committee on November 24, 2015. Two different tasks were carried out to test the functioning of the tool. First, retrospective data from a real patient who met the inclusion criteria were used. Second, the analysis of an adoption model was performed through the study of the requirements and characteristics that a CDSS must meet in order to be well accepted and used by health professionals. The results are favorable and allow the proposed research to continue to the next phase.

CONCLUSIONS

An ontology-based CDSS was successfully designed, developed, and validated. However, in future work, validation in a real environment should be performed to ensure the tool is usable and reliable.

Sustainability of a Clinical Decision Support Intervention for Outpatient Care for Emergency Department Patients With Acute Pulmonary Embolism

IMPORTANCE

Physicians commonly hospitalize patients presenting to the emergency department (ED) with acute pulmonary embolism (PE), despite eligibility for safe outpatient management. Risk stratification using electronic health record-embedded clinical decision support systems can aid physician site-of-care decision-making and increase safe outpatient management. The long-term sustainability of early improvements after the cessation of trial-based, champion-led promotion is uncertain.

OBJECTIVE

To evaluate the sustainability of recommended site-of-care decision-making support 4 years after initial physician champion-led interventions to increase outpatient management for patients with acute PE.

DESIGN, SETTING, AND PARTICIPANTS

This retrospective cohort study was conducted in 21 US community hospitals in an integrated health system. Participants included adult patients presenting to the ED with acute PE. Study sites had participated in an original decision-support intervention trial 4 years prior to the current study period: 10 sites were intervention sites, 11 sites were controls. In that trial, decision support with champion promotion resulted in significantly higher outpatient management at intervention sites compared with controls. After trial completion, all study sites were given continued access to a modified decision-support tool without further champion-led outreach. Data were analyzed from January 2019 to February 2020.

EXPOSURES

ED treatment with a modified clinical decision support tool.

MAIN OUTCOMES AND MEASURES

The main outcome was frequency of outpatient management, defined as discharge home directly from the ED, stratified by the PE Severity Index. The safety measure of outpatient care was 7-day PE-related hospitalization.

RESULTS

This study included 1039 patients, including 533 (51.3%) women, with a median (IQR) age of 65 (52-74) years. Nearly half (474 patients [45.6%]) were rated lower risk on the PE Severity Index. Overall, 278 patients (26.8%) were treated as outpatients, with only four 7-day PE-related hospitalizations (1.4%; 95% CI, 0.4%-3.6%). The practice gap in outpatient management created by the earlier trial persisted in the outpatient management for patients with lower risk: 109 of 236 patients (46.2%) at former intervention sites vs 81 of 238 patients (34.0%) at former control sites (difference, 12.2; [95% CI, 3.4-20.9] percentage points; P = .007), with wide interfacility variation (range, 7.1%-47.1%).

CONCLUSIONS AND RELEVANCE

In this cohort study, a champion-led, decision-support intervention to increase outpatient management for patients presenting to the ED with acute pulmonary embolism was associated with sustained higher rates of outpatient management 4 years later. The application of our findings to improving sustainability of practice change for other clinical conditions warrants further study.

Evaluation in Life Cycle of Information Technology (ELICIT) framework: Supporting the innovation life cycle from business case assessment to summative evaluation

OBJECTIVE

Our objective was to develop an evaluation framework for electronic health record (EHR)-integrated innovations to support evaluation activities at each of four information technology (IT) life cycle phases: planning, development, implementation, and operation.

METHODS

The evaluation framework was developed based on a review of existing evaluation frameworks from health informatics and other domains (human factors engineering, software engineering, and social sciences); expert consensus; and real-world testing in multiple EHR-integrated innovation studies.

RESULTS

The resulting Evaluation in Life Cycle of IT (ELICIT) framework covers four IT life cycle phases and three measure levels (society, user, and IT). The ELICIT framework recommends 12 evaluation steps: (1) business case assessment; (2) stakeholder requirements gathering; (3) technical requirements gathering; (4) technical acceptability assessment; (5) user acceptability assessment; (6) social acceptability assessment; (7) social implementation assessment; (8) initial user satisfaction assessment; (9) technical implementation assessment; (10) technical portability assessment; (11) long-term user satisfaction assessment; and (12) social outcomes assessment.

DISCUSSION

Effective evaluation requires a shared understanding and collaboration across disciplines throughout the entire IT life cycle. In contrast with previous evaluation frameworks, the ELICIT framework focuses on all phases of the IT life cycle across the society, user, and IT levels. Institutions seeking to establish evaluation programs for EHR-integrated innovations could use our framework to create such shared understanding and justify the need to invest in evaluation.

CONCLUSION

As health care undergoes a digital transformation, it will be critical for EHR-integrated innovations to be systematically evaluated. The ELICIT framework can facilitate these evaluations.

Implementation of South Africa's Central Chronic Medicine Dispensing and Distribution Program for HIV Treatment: A Qualitative Evaluation

We used the Practical, Robust Implementation and Sustainability Model to evaluate implementation of South Africa’s Central Chronic Medicine Dispensing and Distribution (CCMDD) program, a differentiated service delivery program which allows clinically stable HIV-positive patients to receive antiretroviral therapy refills at clinic- or community-based pick-up points. Across ten clinics, we conducted 109 semi-structured interviews with stakeholders (pick-up point staff, CCMDD service providers and administrators) and 16 focus groups with 138 patients. Participants had highly favorable attitudes and said CCMDD decreased stigma concerns. Patient-level barriers included inadequate education about CCMDD and inability to get refills on designated dates. Organizational-level barriers included challenges with communication and transportation, errors in medication packaging and tracking, rigid CCMDD rules, and inadequate infrastructure. Recommendations included: (1) provide patient education and improve communication around refills (at the patient level); (2) provide dedicated space and staff, and ongoing training (at the organizational/clinic level); and (3) allow for prescription renewal at pick-up points and less frequent refills, and provide feedback to clinics (at the CCMDD program level).

Development and implementation of a clinical decision support-based initiative to drive intravenous fluid prescribing

OBJECTIVE

Studies suggest superior outcomes with use of intravenous (IV) balanced fluids compared to normal saline (NS). However, significant fluid prescribing variability persists, highlighting the knowledge-to-practice gap. We sought to identify contributors to prescribing variation and utilize a clinical decision support system (CDSS) to increase institutional balanced fluid prescribing.

MATERIALS AND METHODS

This single-center informatics-enabled quality improvement initiative for patients hospitalized or treated in the emergency department included stepwise interventions of 1) identification of design factors within the computerized provider order entry (CPOE) of our electronic health record (EHR) that contribute to preferential NS ordering, 2) clinician education, 3) fluid stocking modifications, 4) re-design and implementation of a CDSS-integrated IV fluid ordering panel, and 5) comparison of fluid prescribing before and after the intervention. EHR-derived prescribing data was analyzed via single interrupted time series.

RESULTS

Pre-intervention (3/2019-9/2019), balanced fluids comprised 33% of isotonic fluid orders, with gradual uptake (1.4%/month) of balanced fluid prescribing. Clinician education (10/2019-2/2020) yielded a modest (4.4%/month, 95% CI 1.6-7.2, p = 0.01) proportional increase in balanced fluid prescribing, while CPOE redesign (3/2020) yielded an immediate (20.7%, 95% CI 17.7-23.6, p < 0.0001) and sustained increase (72% of fluid orders in 12/2020). The intervention proved most effective among those with lower baseline balanced fluids utilization, including emergency medicine (57% increase, 95% CI 0.7-1.8, p < 0.0001) and internal medicine/subspecialties (18% increase, 95% CI 14.4-21.3, p < 0.0001) clinicians and substantially reduced institutional prescribing variation.

CONCLUSION

Integration of CDSS into an EHR yielded a robust and sustained increase in balanced fluid prescribing. This impact far exceeded that of clinician education highlighting the importance of CDSS.

Evaluation Framework for Successful Artificial Intelligence-Enabled Clinical Decision Support Systems: Mixed Methods Study

Background

Clinical decision support systems are designed to utilize medical data, knowledge, and analysis engines and to generate patient-specific assessments or recommendations to health professionals in order to assist decision making. Artificial intelligence–enabled clinical decision support systems aid the decision-making process through an intelligent component. Well-defined evaluation methods are essential to ensure the seamless integration and contribution of these systems to clinical practice.

Objective

The purpose of this study was to develop and validate a measurement instrument and test the interrelationships of evaluation variables for an artificial intelligence–enabled clinical decision support system evaluation framework.

Methods

An artificial intelligence–enabled clinical decision support system evaluation framework consisting of 6 variables was developed. A Delphi process was conducted to develop the measurement instrument items. Cognitive interviews and pretesting were performed to refine the questions. Web-based survey response data were analyzed to remove irrelevant questions from the measurement instrument, to test dimensional structure, and to assess reliability and validity. The interrelationships of relevant variables were tested and verified using path analysis, and a 28-item measurement instrument was developed. Measurement instrument survey responses were collected from 156 respondents.

Results

The Cronbach α of the measurement instrument was 0.963, and its content validity was 0.943. Values of average variance extracted ranged from 0.582 to 0.756, and values of the heterotrait-monotrait ratio ranged from 0.376 to 0.896. The final model had a good fit (χ₂₆²=36.984; P=.08; comparative fit index 0.991; goodness-of-fit index 0.957; root mean square error of approximation 0.052; standardized root mean square residual 0.028). Variables in the final model accounted for 89% of the variance in the user acceptance dimension.

Conclusions

User acceptance is the central dimension of artificial intelligence–enabled clinical decision support system success. Acceptance was directly influenced by perceived ease of use, information quality, service quality, and perceived benefit. Acceptance was also indirectly influenced by system quality and information quality through perceived ease of use. User acceptance and perceived benefit were interrelated.

Applying Clinical Decision Support Design Best Practices With the Practical Robust Implementation and Sustainability Model Versus Reliance on Commercially Available Clinical Decision Support Tools: Randomized Controlled Trial

BACKGROUND

Limited consideration of clinical decision support (CDS) design best practices, such as a user-centered design, is often cited as a key barrier to CDS adoption and effectiveness. The application of CDS best practices is resource intensive; thus, institutions often rely on commercially available CDS tools that are created to meet the generalized needs of many institutions and are not user centered. Beyond resource availability, insufficient guidance on how to address key aspects of implementation, such as contextual factors, may also limit the application of CDS best practices. An implementation science (IS) framework could provide needed guidance and increase the reproducibility of CDS implementations.

OBJECTIVE

This study aims to compare the effectiveness of an enhanced CDS tool informed by CDS best practices and an IS framework with a generic, commercially available CDS tool.

METHODS

We conducted an explanatory sequential mixed methods study. An IS-enhanced and commercial CDS alert were compared in a cluster randomized trial across 28 primary care clinics. Both alerts aimed to improve beta-blocker prescribing for heart failure. The enhanced alert was informed by CDS best practices and the Practical, Robust, Implementation, and Sustainability Model (PRISM) IS framework, whereas the commercial alert followed vendor-supplied specifications. Following PRISM, the enhanced alert was informed by iterative, multilevel stakeholder input and the dynamic interactions of the internal and external environment. Outcomes aligned with PRISM's evaluation measures, including patient reach, clinician adoption, and changes in prescribing behavior. Clinicians exposed to each alert were interviewed to identify design features that might influence adoption. The interviews were analyzed using a thematic approach.

RESULTS

Between March 15 and August 23, 2019, the enhanced alert fired for 61 patients (106 alerts, 87 clinicians) and the commercial alert fired for 26 patients (59 alerts, 31 clinicians). The adoption and effectiveness of the enhanced alert were significantly higher than those of the commercial alert (62% vs 29% alerts adopted, P<.001; 14% vs 0% changed prescribing, P=.006). Of the 21 clinicians interviewed, most stated that they preferred the enhanced alert.

CONCLUSIONS

The results of this study suggest that applying CDS best practices with an IS framework to create CDS tools improves implementation success compared with a commercially available tool.

TRIAL REGISTRATION

ClinicalTrials.gov NCT04028557; http://clinicaltrials.gov/ct2/show/NCT04028557.

Assessing Prescriber Behavior with a Clinical Decision Support Tool to Prevent Drug-Induced Long QT Syndrome

OBJECTIVE

Clinical decision support (CDS) alerts built into the electronic health record (EHR) have the potential to reduce the risk of drug-induced long QT syndrome (diLQTS) in susceptible patients. However, the degree to which providers incorporate this information into prescription behavior and the impact on patient outcomes is often unknown.

METHODS

We examined provider response data over a period from October 8, 2016 until November 8, 2018 for a CDS alert deployed within the EHR from a 13-hospital integrated health care system that fires when a patient with a QTc ≥ 500 ms within the past 14 days is prescribed a known QT-prolonging medication. We used multivariate generalized estimating equations to analyze the impact of therapeutic alternatives, relative risk of diLQTS for specific medications, and patient characteristics on provider response to the CDS and overall patient mortality.

RESULTS

The CDS alert fired 15,002 times for 7,510 patients for which the most common response (51.0%) was to override the alert and order the culprit medication. In multivariate models, we found that patient age, relative risk of diLQTS, and presence of alternative agents were significant predictors of adherence to the CDS alerts and that nonadherence itself was a predictor of mortality. Risk of diLQTS and presence of an alternative agent are major factors in provider adherence to a CDS to prevent diLQTS; however, provider nonadherence was associated with a decreased risk of mortality.

CONCLUSION

Surrogate endpoints, such as provider adherence, can be useful measures of CDS value but attention to hard outcomes, such as mortality, is likely needed.

Blending an implementation science framework with principles of proportionate universalism to support physical activity promotion in primary healthcare while addressing health inequities

Globally, insufficient physical activity (PA) is one of the main risk factors for premature mortality. Although insufficient PA is prevalent in nearly every demographic, people with socio-economic disadvantage participate in lower levels of PA than those who are more affluent, and this contributes to widening health inequities. PA promotion interventions in primary healthcare are effective and cost effective, however they are not widely implemented in practice. Further, current approaches that adopt a ‘universal’ approach to PA promotion do not consider or address the additional barriers experienced by people who experience socioeconomic disadvantages. To address the research to policy and practice gap, and taking Australia as a case study, this commentary proposes a novel model which blends an implementation science framework with the principles of proportionate universalism. Proportionate universalism is a principle suggesting that health interventions and policies need to be universal, not targeted, but with intensity and scale proportionate to the level of social need and/or disadvantage. Within this model, we propose interrelated and multi-level evidence-based policies and strategies to support PA promotion in primary healthcare while addressing health inequities. The principles outlined in the new model which blends proportionate (Pro) universalism principles and Practical, Robust Implementation and Sustainability Model (PRISM), ‘ProPRISM’ can be applied to the implementation of PA promotion interventions in health care settings in other high-income countries. Future studies should test the model and provide evidence of its effectiveness in improving implementation and patient health outcomes and cost-effectiveness. There is potential to expand the proposed model to other health sectors (e.g., secondary and tertiary care) and to address other chronic disease risk factors such as unhealthy diet, smoking, and alcohol consumption. Therefore, this approach has the potential to transform the delivery of health care to a prevention-focused health service model, which could reduce the prevalence and burden of chronic disease and health care costs in high-income countries.