Fidelity is not easy! Challenges and guidelines for assessing fidelity in complex interventions

Comparison of Researcher-Led versus Teacher-Led effectiveness and fidelity: A Hybrid Type 1 study of Kids SIPsmartER in Appalachia middle schools

The implementation of school-based obesity-prevention programs is understudied. Kids SIPsmartER is a 6-month, school-based, behavioral intervention for Appalachian middle school students and includes a teacher implementation strategy. Kids SIPsmartER effectively reduced students' sugar-sweetened beverages (SSB) when Researcher-Led. However, Teacher-Led effectiveness and fidelity are unknown. To explore the relative SSB effects when Kids SIPsmartER was Researcher-Led versus Teacher-Led and to examine teacher fidelity. This study of secondary outcomes used a quasi-experimental analytic approach of a Hybrid Type 1 effectiveness-implementation and cluster randomized controlled tria (RCT) of Kids SIPsmartER. Student SSB behaviors and teacher self-rated fidelity were assessed, respectively, with the validated Beverage Intake Questionnaire (BEVQ-15) and lesson-specific checklists. Analyses included descriptive statistics and modified two-part models with time-fixed effects and school-year cohort cluster controls. The analytic sample included students from six control schools (n = 220), six Researcher-Led intervention schools (n = 306), and five Teacher-Led intervention schools (n = 218), as well as eight teachers. Teacher-Led intervention students decreased SSB by -14.3 ounces/day (95% confidence interval = -15.4, -13.2; P < .001). Relative to control and to Researcher-Led intervention, the Teacher-Led treatment effect among students was -11.6 ounces SSB/day (P < .001, effect size = 0.75) and -4.3 (P = .004, effect size = 0.25), respectively. Teachers returned fidelity checklists for 90% of planned lessons. Fidelity averaged 94% (SD = 4.0%) among returned forms and 85% (SD = 18.9%) when missing forms were counted as zeros. Teachers can implement Kids SIPsmartER with high fidelity and produce statistically and clinically meaningful improvements in students' SSB behaviors. Findings have implications for the sustained implementation of Kids SIPsmartER and other school-based obesity-prevention programs. Clinical Trial information: NCT03740113.

Evaluating Real-World Implementation of INFORM (Improving Nursing Home Care through Feedback on Performance Data): An Improvement Initiative in Canadian Nursing Homes

BACKGROUND

INFORM (Improving Nursing Home Care through Feedback on Performance Data) was a research intervention that equipped nursing home managers with skills to conduct local improvement projects and supported them in improving performance through modifiable elements in their units. Prior reports have found positive and sustained outcomes from INFORM intervention. In this article, the authors report findings from a formative service evaluation of INFORM as modified for implementation in real-world settings.

METHODS

INFORM was transformed for real-world implementation with an initial cohort of 26 nursing homes in British Columbia, Canada (INFORM BC). Three stakeholder groups were involved: nursing home teams, an academic team that modified INFORM for implementation, and a BC team that implemented INFORM and coached participating nursing home teams in applying it locally. Service evaluation was conducted drawing on participants from all three stakeholder groups, using convenience sampling, with numbers varying by data source. Using a mixed methods design, outcome data included qualitative and quantitative assessment of surveys, discussions, observations, and a review of documents and resources.

RESULTS

The majority of nursing home teams reported positive outcomes relative to the usefulness and relevance of the initiative for local needs despite a number of operational challenges during implementation. A key factor in their success was combining targeted external support with the opportunity to set goals and measure success locally. Challenges included a lack of time at the nursing home level, COVID-19-related disruptions, and issues with role clarity and alignment of expectations among the academic and BC teams.

CONCLUSION

INFORM BC advanced the processes of change planning and transferable learning among nursing home managers and their local teams. Success was facilitated externally but defined and achieved locally. Future iterations should probe outcome sustainability and how nursing home teams adapt the INFORM approach in practice.

Developing an international consensus Reporting guideline for intervention Fidelity in Non-Drug, non-surgical trials: The ReFiND protocol

BACKGROUND

Inadequate reporting of fidelity to interventions in trials limits the transparency and interpretation of trial findings. Despite this, most trials of non-drug, non-surgical interventions lack comprehensive reporting of fidelity. If fidelity is poorly reported, it is unclear which intervention components were tested or implemented within the trial, which also hinders research reproducibility. This protocol describes the development process of a reporting guideline for fidelity of non-drug, non-surgical interventions (ReFiND) in the context of trials.

METHODS

The ReFiND guideline will be developed in six stages. Stage one: a guideline development group has been formed to oversee the guideline methodology. Stage two: a scoping review will be conducted to identify and summarize existing guidance documents on the fidelity of non-drug, non-surgical interventions. Stage three: a Delphi study will be conducted to reach consensus on reporting items. Stage four: a consensus meeting will be held to consolidate the reporting items and discuss the wording and structure of the guideline. Stage five: a guidance statement, an elaboration and explanation document, and a reporting checklist will be developed. Stage six: different strategies will be used to disseminate and implement the ReFiND guideline.

DISCUSSION

The ReFiND guideline will provide a set of items developed through international consensus to improve the reporting of intervention fidelity in trials of non-drug, non-surgical interventions. This reporting guideline will enhance transparency and reproducibility in future non-drug, non-surgical intervention research.

The Development and Pilot Testing of a Fidelity Checklist for a Family-Centered Telehealth Intervention for Parents of Children with Motor Delay

This multi-methods study describes the development of a pediatric rehabilitation telehealth intervention fidelity checklist, estimates its inter-rater reliability, and documents raters' implementation experience. A literature scan and expert consultation identified eighteen key behaviors and categorized them into three subdomains, measured using a 5-point measurement system. To estimate the checklist's inter-rater reliability, three raters scored 33 video recordings. A Shrout and Fleiss Class 1,1 intraclass correlation (ICC)) and 95% confidence intervals (CI) calculated ICCs = 0.5 (CI: 0, 0.9) for both the Therapist and the Parent-Therapists subdomains, and the Parent subdomain an ICC = 0.3 (CI: 0, 0.8). In the implementation surveys, raters reported high levels of satisfaction (100%), ease of use (84% to 88%), and confidence in their video ratings (87% to 100%). Changes in procedures and scoring were recommended. Capturing raters' implementation experiences is crucial in the early evaluation of the fidelity checklists for telehealth.

Fidelity to the acupuncture intervention protocol in the ACUpuncture In The EmergencY department for pain management (ACUITY) trial: Expanding the gold standard of STRICTA and CONSORT guidelines

Background

Acupuncture shows promise as an effective nonpharmacologic option for reduction of acute pain in the emergency department (ED). Following CONSORT and STRICTA guidelines, randomized controlled trials (RCTs) generally report intervention details and acupoint options, but fidelity to acupuncture interventions, critical to reliability in intervention research, is rarely reported.

Methods

ACUITY is an NCCIH-funded, multi-site feasibility RCT of acupuncture in 3 EDs (Cleveland, Nashville, and San Diego). ACUITY acupuncturists were trained in study design, responsive acupuncture manualization protocol, logistics and real-time recording of session details via REDCap forms created to track fidelity.

Results

Across 3 recruiting sites, 79 participants received acupuncture: 51 % women, 43 % Black/African American, with heterogeneous acute pain sites at baseline: 32 % low back, 22 % extremity, 20 % abdominal, 10 % head. Pragmatically, participants were treated in ED common areas (52 %), private rooms (39 %), and semi-private rooms (9 %). Objective tracking found 98 % adherence to the six components of the acupuncture manualization protocol: staging, number of insertion points (M = 13.2, range 2-22), needle retention time (M = 23.5 min, range 4-52), session length (M = 40.3 min, range 20-66), whether general recommendations were provided and completion of the session form.

Conclusion

To the best of our knowledge, this is the first RCT to assess and report fidelity to an acupuncture protocol. Fidelity monitoring will be fundamental for ACUITY2, which would be a future definitive, multi-site RCT. Furthermore, we recommend that fidelity to acupuncture interventions be added to CONSORT and STRICTA reporting guidelines in future RCTs.

Protocol registration

The protocol of this study is registered at clinicaltrials.gov: NCT04880733.

Documenting the implementation processes and effects of the data use initiatives in primary health care settings in Tanzania: A before-after mixed methods study protocol

BACKGROUND

The use of data in decision making and planning in primary health care settings is critical for improving efficiency and health outcomes for patients and communities. Implementation research can be used to fully understand the effects, context, challenges, and facilitators of data use, as well as how to scale up data use interventions. However, in the context of low resource settings, little is known about how implementation research can be employed to assess the implementation and impact of data use interventions.

METHODS

We will conduct a hybrid type 2 effectiveness-implementation study employing a mixed method controlled before and after design to measure the effects of data use interventions while simultaneously understanding the implementation of those initiatives. The controlled before and after entails measurement of the effects of the interventions at baseline and end line in a matched intervention and control health facilities using structured questionnaire to health workers (n = 440) and existing patients (n = 422) while also extracting selected health outcome variable from routine data in all participating health facilities (n = 80). The mixed methods component entails measuring the implementation outcomes (adoption, acceptability, fidelity and maintenance) and their moderators entails the integration of both quantitative and qualitative data collection, analysis, and interpretation (i.e. mixed methods) approach by using a structured questionnaire to implementers (health workers and managers) (n = 400). Experiential dimensions of implementation processes and moderators will be explored using qualitative interviews. Guided by implementation research theories and frameworks, a theory of change (TOC) is developed first to guide the evaluation of implementation processes and effects of the interventions. Descriptive and inferential statistics will be employed to analyze quantitative data whereas thematic analysis approach will be employed for qualitative data.

DISCUSSION

This study is one of the first to test the simultaneous measurement of effects and implementation processes of data use interventions in the primary health care settings. Findings will support efforts to improve quality of services by optimizing scale up and sustainability of the data use initiatives in primary health care settings.

Methodological challenges in systematic reviews of mHealth interventions: Survey and consensus-based recommendations

OBJECTIVE

Mobile Health (mHealth) refers to using mobile devices to support health. This study aimed to identify specific methodological challenges in systematic reviews (SRs) of mHealth interventions and to develop guidance for addressing selected challenges.

STUDY DESIGN AND SETTING

Two-phase participatory research project. First, we sent an online survey to corresponding authors of SRs of mHealth interventions. On a five-category scale, survey respondents rated how challenging they found 24 methodological aspects in SRs of mHealth interventions compared to non-mHealth intervention SRs. Second, a subset of survey respondents participated in an online workshop to discuss recommendations to address the most challenging methodological aspects identified in the survey. Finally, consensus-based recommendations were developed based on the workshop discussion and subsequent interaction via email with the workshop participants and two external mHealth SR authors.

RESULTS

We contacted 953 corresponding authors of mHealth intervention SRs, of whom 50 (5 %) completed the survey. All the respondents identified at least one methodological aspect as more or much more challenging in mHealth intervention SRs than in non-mHealth SRs. A median of 11 (IQR 7.25-15) out of 24 aspects (46 %) were rated as more or much more challenging. Those most frequently reported were: defining intervention intensity and components (85 %), extracting mHealth intervention details (71 %), dealing with dynamic research with evolving interventions (70 %), assessing intervention integrity (69 %), defining the intervention (66 %) and maintaining an updated review (65 %). Eleven survey respondents participated in the workshop (five had authored more than three mHealth SRs). Eighteen consensus-based recommendations were developed to address issues related to mHealth intervention integrity and to keep mHealth SRs up to date.

CONCLUSION

mHealth SRs present specific methodological challenges compared to non-mHealth interventions, particularly related to intervention integrity and keeping SRs current. Our recommendations for addressing these challenges can improve mHealth SRs.

A RCT to explore the effectiveness of supporting adherence to nebuliser medication in adults with cystic fibrosis: fidelity assessment of study interventions

BACKGROUND

A multi-component self-management intervention 'CFHealthHub' was developed to reduce pulmonary exacerbations in adults with Cystic Fibrosis (CF) by supporting adherence to nebuliser medication. It was evaluated in a randomized controlled trial (RCT) involving 19 CF centres, with 32 interventionists, 305 participants in the intervention group, and 303 participants in the standard care arm. Ensuring treatment fidelity of intervention delivery was crucial to ensure that the intervention produced the expected outcomes.

METHODS

Fidelity of the CFHealthHub intervention and standard care was assessed using different methods for each of the five fidelity domains defined by the Borrelli framework: study design, training, treatment delivery, receipt, and enactment. Study design ensured that the groups received the intended intervention or standard care. Interventionists underwent training and competency assessments to be deemed certified to deliver the intervention. Audio-recorded intervention sessions were assessed for fidelity drift. Receipt was assessed by identifying whether participants set Action and Coping Plans, while enactment was assessed using click analytics on the CFHealthHub digital platform.

RESULTS

Design: There was reasonable agreement (74%, 226/305) between the expected versus actual intervention dose received by participants in the CFHealthHub intervention group. The standard care group did not include focused adherence support for most centres and participants. Training: All interventionists were trained. Treatment delivery: The trial demonstrated good fidelity (overall fidelity by centre ranged from 79 to 97%), with only one centre falling below the mean threshold (> 80%) on fidelity drift assessments. Receipt: Among participants who completed the 12-month intervention, 77% (205/265) completed at least one action plan, and 60% (160/265) completed at least one coping plan. Enactment: 88% (268/305) of participants used web/app click analytics outside the intervention sessions. The mean (SD) number of web/app click analytics per participant was 31.2 (58.9). Additionally, 64% (195/305) of participants agreed to receive notifications via the mobile application, with an average of 53.6 (14.9) notifications per participant.

CONCLUSIONS

The study demonstrates high fidelity throughout the RCT, and the CFHealthHub intervention was delivered as intended. This provides confidence that the results of the RCT are a valid reflection of the effectiveness of the CFHealthHub intervention compared to standard care.

TRIAL REGISTRATION

ISRCTN registry: ISRCTN55504164 (date of registration: 12/10/2017).

Alternative community-led intervention to improve uptake of cataract surgery services in rural Tanzania-The Dodoma Community Cataract Acceptance Trial (DoCCAT): a protocol for intervention co-designing and implementation in a cluster-randomized controlled trial

Age-related lens opacification (cataract) remains the leading cause of visual impairment and blindness worldwide. In low- and middle-income countries, utilization of cataract surgical services is often limited despite community-based outreach programmes. Community-led research, whereby researchers and community members collaboratively co-design intervention is an approach that ensures the interventions are locally relevant and that their implementation is feasible and socially accepted in the targeted contexts. Community-led interventions have the potential to increase cataract surgery uptake if done appropriately. In this study, once the intervention is co-designed it will be implemented through a cluster-randomized controlled trial (cRCT) with ward as a unit of randomization. This study will utilise both the qualitative methods for co-designing the intervention and the quantitative methods for effective assessment of the developed community-led intervention through a cRCT in 80 rural wards of Dodoma region, Tanzania (40 Intervention). The 'intervention package' will be developed through participatory community meetings and ongoing evaluation and modification of the intervention based on its impact on service utilization. Leask's four stages of intervention co-creation will guide the development within Rifkin's CHOICE framework. The primary outcomes are two: the number of patients attending eye disease screening camps, and the number of patients accepting cataract surgery. NVivo version 12 will be used for qualitative data analysis and Stata version 12 for quantitative data. Independent and paired t-tests will be performed to make comparisons between and within groups. P-values less than 0.05 will be considered statistically significant.

Implementation of Parenting Programs in Real-World Community Settings: A Scoping Review

Implementing parenting programs in real-world community settings is fundamental to making effective programs widely available and consequently improving the lives of children and their families. Despite the literature acknowledging that the high-quality implementation of parenting programs is particularly challenging in real-world community settings, little is known about how the programs are implemented in these settings. This scoping review followed the methodological framework described by the Joanna Briggs Institute to map evidence on how evidence-based parenting programs have been implemented under real-world conditions. A systematic search of 12 scientific databases, gray literature, and the reference lists of the included studies identified 1918 records, of which 145 were included in the review. Fifty-three parenting programs were identified in studies documenting implementation in real-world community settings worldwide. Most studies included families in psychosocial risk engaged with family-support agencies. The qualitative synthesis identified several implementation outcomes, adaptations, barriers, and facilitators. Most studies reported a maximum of two implementation outcomes, mainly fidelity and acceptability. Providers frequently made adaptations, mainly to bring down barriers and to tailor the program to improve its fit. Findings highlight the need for a more detailed description of the implementation of programs, with greater consistency in terminology, operationalization, and measurement of implementation outcomes across studies. This will promote a more transparent, consistent, and accurate evaluation and reporting of implementation and increase the public health impact of parenting programs. Future studies should also assess the impact of adaptations and the cost-effectiveness and sustainability of programs in real-world community settings.

Assessment of description and implementation fidelity of clinical trials involving exercise-based treatment in individuals with rotator cuff tears: a scoping review

BACKGROUND

For evidence-based practice, clinicians and researchers can rely on well-conducted randomized clinical trials that exhibit good methodological quality, provide adequate intervention descriptions, and implementation fidelity.

OBJECTIVE

To assess the description and implementation fidelity of exercise-based interventions in clinical trials for individuals with rotator cuff tears.

METHODS

A systematic search was conducted in PubMed, Embase, CINAHL, LILACS, Cochrane Library, Web of Science, SCOPUS and SciELO. Randomized clinical trials that assessed individuals with rotator cuff tears confirmed by imaging exam were included. All individuals must have received an exercise-based treatment. The methodological quality was scored with the Physiotherapy Evidence Database (PEDro) scale. The Template for Intervention Description and Replication (TIDieR) checklist and the National Institutes of Health Behaviour Change Consortium (NIHBCC) were used to assess intervention description and implementation fidelity, respectively.

RESULTS

A total of 13 studies were included. Despite their adequate methodological quality, the description of the intervention was poor with TIDieR scores ranging from 6 to 15 out of 24 total points. The TIDieR highest-scoring item was item 1 (brief name) that was reported in all studies. Considering fidelity, only one of the five domains of NIHBCC (i.e., treatment design) reached just over 50%.

CONCLUSION

Exercise-based interventions used in studies for individuals with rotator cuff tears are poorly reported. The description and fidelity of the intervention need to be better reported to assist clinical decision-making and support evidence-based practice.

How to Measure Human-Dog Interaction in Dog Assisted Interventions? A Scoping Review

Human-dog interaction is the working tool through which the therapeutic, educational and recreational goals of dog-assisted interventions (DAIs) are achieved. A better understanding of the characteristics of this interaction could improve the effectiveness of DAIs. This scoping review addresses the question: how has the human-dog connection been measured in the context of DAIs? After searching the Web of Science and Scopus platforms, only peer-reviewed, primary research studies reporting measures of therapy dog-human interaction, relationship and bond were included. A total of 70 included articles provided information on what was measured (interaction, relationship or bond) and how, as well as the general context (DAIs or experimental situations with therapy dogs). While the majority of the articles identified use behavioural analysis methods to analyse the interaction between the participant/recipient and the therapy dog during DAIs, it was possible to identify some more structured tools that assess the participant/recipient's interaction, relationship or bond with the therapy dog, as well as tools that consider the animal's perspective or focus on the dog-handler dyad, indicating growing areas of research. The tools and methods identified can be used by both practitioners and researchers to further explore aspects of human-dog interaction in the field of DAIs.

Development and validation of the Overall Fidelity Enactment Scale for Complex Interventions (OFES-CI)

BACKGROUND

In many quality improvement (QI) and other complex interventions, assessing the fidelity with which participants 'enact' intervention activities (ie, implement them as intended) is underexplored. Adapting the evaluative approach used in objective structured clinical examinations, we aimed to develop and validate a practical approach to assessing fidelity enactment-the Overall Fidelity Enactment Scale for Complex Interventions (OFES-CI).

METHODS

We developed the OFES-CI to evaluate enactment of the SCOPE QI intervention, which teaches nursing home teams to use plan-do-study-act (PDSA) cycles. The OFES-CI was piloted and revised early in SCOPE with good inter-rater reliability, so we proceeded with a single rater. An intraclass correlation coefficient (ICC) was used to assess inter-rater reliability. For 27 SCOPE teams, we used ICC to compare two methods for assessing fidelity enactment: (1) OFES-CI ratings provided by one of five trained experts who observed structured 6 min PDSA progress presentations made at the end of SCOPE, (2) average rating of two coders' deductive content analysis of qualitative process evaluation data collected during the final 3 months of SCOPE (our gold standard).

RESULTS

Using Cicchetti's classification, inter-rater reliability between two coders who derived the gold standard enactment score was 'excellent' (ICC=0.93, 95% CI=0.85 to 0.97). Inter-rater reliability between the OFES-CI and the gold standard was good (ICC=0.71, 95% CI=0.46 to 0.86), after removing one team where open-text comments were discrepant with the rating. Rater feedback suggests the OFES-CI has strong face validity and positive implementation qualities (acceptability, easy to use, low training requirements).

CONCLUSIONS

The OFES-CI provides a promising novel approach for assessing fidelity enactment in QI and other complex interventions. It demonstrates good reliability against our gold standard assessment approach and addresses the practicality problem in fidelity assessment by virtue of its suitable implementation qualities. Steps for adapting the OFES-CI to other complex interventions are offered.

Developing the Workforce of the Digital Future: mHealth Competency and Fidelity Measurement in Community-Based Care

Integrating mobile health (mHealth) interventions into settings that serve diverse patient populations requires that prerequisite professional competencies are delineated and that standards for clinical quality assurance can be pragmatically assessed. Heretofore, proposed mHealth competencies have been broad and have lacked a framework to support specific applications. We outline the meta-competencies identified in the literature relevant to mHealth interventions and demonstrate how these meta-competencies can be integrated with population- and intervention-related competencies to help guide a pragmatic approach to competency assessment. We present a use case based on FOCUS-an evidence-based mHealth intervention designed for individuals with serious mental illness and currently being implemented in geographically and demographically diverse community behavioral health settings. Subsequent to identifying the cross-cutting competencies relevant to the target population (outpatients experiencing psychotic symptoms), substratal intervention (Cognitive Behavioral Therapy for psychosis), and treatment modality (mHealth), we detail the development process of an mHealth fidelity monitoring system (mHealth-FMS). We adhered to a published sequential 5-step process to design a fidelity monitoring system that aligns with our integrated mHealth competency framework and that was guided by best practices prescribed by the Treatment Fidelity Workgroup of the National Institutes of Health Behavior Change Consortium. The mHealth-FMS is intended to enhance both clinical and implementation outcomes by grounding the mHealth interventionist and the system of care in which they operate in the core functions, tasks, knowledge, and competencies associated with system-integrated mHealth delivery. Future research will explore acceptability and feasibility of the mHealth-FMS.

Treatment fidelity in clinical trials

In the context of clinical trials, treatment fidelity (TF) has traditionally referred to the extent to which an intervention or treatment is implemented by the clinicians as intended by the researchers who designed the trial. Updated definitions of TF have included an appropriate design of the intervention that was performed in a way that is known to be therapeutically beneficial. This requires careful attention to three key components: (1) protocol and dosage adherence, (2) quality of delivery, and (3) participant adherence. In this viewpoint, we describe several cases in which TF was lacking in clinical trials and give opportunities to improve the deficits encountered in those trials. We feel that along with quality, risk of bias, and certainty of evidence, TF should be considered an essential element of the veracity of clinical trial.

Developing and Implementing a Culturally Consonant Treatment Fidelity Support Plan with the Apsáalooke Nation

Treatment fidelity remains underreported in health intervention research, particularly among Indigenous communities. One explanation for this gap is the lack of culturally consonant strategies listed in the National Institutes of Health (NIH) Behavior Change Consortium (BCC) treatment fidelity framework, the gold standard for understanding and measuring fidelity. This paper focuses on the development and implementation of a culturally consonant treatment fidelity support plan across two of the five BCC fidelity areas, provider training and treatment delivery, within a chronic illness self-management program for the Apsáalooke (Crow) Nation. Our team selected and adapted strategies from, and added strategies to, the BCC framework, that centered on relational accountability and the Apsáalooke culture. To be culturally consonant, we approached treatment fidelity as supporting Aakbaabaaniilea (Apsáalooke program facilitators) rather than monitoring them. This resulted in the development of a fifth treatment fidelity area: building and fostering relationships. We propose that fidelity to relational accountability is the foundation of successful programs in Indigenous communities. This suggests an important shift from tracking what was conducted in an intervention to prioritizing how things were conducted. We encourage others to view the BCC framework as a starting point in developing fidelity strategies that are consonant with local cultures.

Lack of Publicly Available Documentation Limits Spread of Integrated Care Innovations in Canada

As healthcare in Canada is provincially operated, the program innovations in one jurisdiction may not be readily known in other jurisdictions. We examine the availability of implementation-specific data for 30 innovative Canadian programs designed to integrate health and social services for patients with complex needs. Using publicly available data and key informant interviews, we were able to populate only ∼50% of our data collection tool (on average). Formal program evaluations were available for only ∼30% of programs. Multiple barriers exist to the compilation and verification of healthcare programs' implementation data across Canada, limiting cross-jurisdictional learning and making a comparison of programs challenging.

Understanding implementation of a complex intervention in a stroke rehabilitation research trial: A qualitative evaluation using Normalisation Process Theory

BACKGROUND

The Implicit Learning in Stroke study was a pilot cluster randomised controlled trial, investigating the use of different motor learning strategies in acute stroke rehabilitation. Participating Stroke Units (n = 8) were from the South East/West regions of the UK, with the experimental intervention (implicit learning) being delivered by clinical teams. It required therapists to change how they gave instructions and feedback to patients during rehabilitation. This paper reports the processes underpinning implementation of the implicit learning intervention. The evaluation aimed to i) understand how therapists made sense of, engaged with and interpreted the effects of the intervention; ii) compare this to the experience reported by patients; iii) extrapolate learning of broader relevance to the design and conduct of research involving complex interventions in stroke rehabilitation.

METHODS

Qualitative evaluation, with data collected through focus groups with clinical staff (n = 20) and semi structured interviews with people with stroke (n = 19). Mixed inductive and theory driven analysis, underpinned by Normalisation Process Theory.

RESULTS

How therapists made sense of and experienced the intervention impacted how it was implemented. The intervention was delivered by individual therapists, and was influenced by their individual values, beliefs and concerns. However, how teams worked together to build a shared (team) understanding, also played a key role. Teams with a more "flexible" interpretation, reported the view that the intervention could have benefits in a wide range of scenarios. Those with a more fixed, "rule based" interpretation, found it harder to implement, and perceived the benefits to be more limited. Therapists' concerns that the intervention may impair therapeutic relationships and patient learning were not reflected in how patients experienced it.

CONCLUSIONS

Changing practice, whether in a research study or in the "real world", is complex. Understanding the process of implementation is crucial to effective research delivery. Implementation frameworks facilitate understanding, and subsequently the systematic and iterative development of strategies for this to be addressed. How teams (rather than individuals) work together is central to how complex interventions are understood and implemented. It is possible that new complex interventions work best in contexts where there are 'flexible' cultures. Researchers should consider, and potentially measure this, before they can effectively implement and evaluate an intervention.

TRIAL REGISTRATION

Clinical Trials - NCT03792126.

Implementation of a multicomponent family support intervention in adult intensive care units: study protocol for an embedded mixed-methods multiple case study (FICUS implementation study)

BACKGROUND

The implementation of complex interventions is considered challenging, particularly in multi-site clinical trials and dynamic clinical settings. This study protocol is part of the family intensive care units (FICUS) hybrid effectiveness-implementation study. It aims to understand the integration of a multicomponent family support intervention in the real-world context of adult intensive care units (ICUs). Specifically, the study will assess implementation processes and outcomes of the study intervention, including fidelity, and will enable explanation of the clinical effectiveness outcomes of the trial.

METHODS AND ANALYSIS

This mixed-methods multiple case study is guided by two implementation theories, the Normalisation Process Theory and the Consolidated Framework for Implementation Research. Participants are key clinical partners and healthcare professionals of eight ICUs allocated to the intervention group of the FICUS trial in the German-speaking part of Switzerland. Data will be collected at four timepoints over the 18-month active implementation and delivery phase using qualitative (small group interviews, observation, focus group interviews) and quantitative data collection methods (surveys, logs). Descriptive statistics and parametric and non-parametric tests will be used according to data distribution to analyse within and between cluster differences, similarities and factors associated with fidelity and the level of integration over time. Qualitative data will be analysed using a pragmatic rapid analysis approach and content analysis.

ETHICS AND DISSEMINATION

Ethics approval was obtained from the Cantonal Ethics Committee of Zurich BASEC ID 2021-02300 (8 February 2022). Study findings will provide insights into implementation and its contribution to intervention outcomes, enabling understanding of the usefulness of applied implementation strategies and highlighting main barriers that need to be addressed for scaling the intervention to other healthcare contexts. Findings will be disseminated in peer-reviewed journals and conferences.

PROTOCOL REGISTRATION NUMBER

Open science framework (OSF) https://osf.io/8t2ud Registered on 21 December 2022.

Effectiveness of cognitive behavioural therapy-based interventions for maternal perinatal depression: a systematic review and meta-analysis

BACKGROUND

Depression during the perinatal period (during pregnancy and the year after childbirth) is common and associated with a range of negative effects for mothers, infants, family members, and wider society. Although existing evidence suggests cognitive behavioral therapy (CBT) based interventions are effective for perinatal depression, less is known about the effect of CBT-based interventions on important secondary outcomes, and a number of potential clinical and methodological moderators have not been examined.

METHODS

A systematic review and meta-analysis primarily examined the effectiveness of CBT-based interventions for perinatal depression on symptoms of depression. Secondary aims examined the effectiveness of CBT-based interventions for perinatal depression on symptoms of anxiety, stress, parenting, perceived social support, and perceived parental competence; and explored clinical and methodological moderators potentially associated with effectiveness. A systematic search of electronic databases and other sources was performed up to November 2021. We included randomized controlled trials comparing CBT-based interventions for perinatal depression with control conditions allowing for the isolation of the effects of CBT.

RESULTS

In total, 31 studies (5291 participants) were included in the systematic review and 26 studies (4658 participants) were included in the meta-analysis. The overall effect size was medium (hedges g = - 0.53 [95% CI - 0.65 to - 0.40]); with high heterogeneity. Significant effects were also found for anxiety, individual stress, and perceived social support, however few studies examined secondary outcomes. Subgroup analysis identified type of control, type of CBT, and type of health professional as significant moderators of the main effect (symptoms of depression). Some concerns of risk of bias were present in the majority of studies and one study had a high risk of bias.

CONCLUSIONS

CBT-based interventions for depression during the perinatal period appear effective, however results should be interpreted with caution given high levels of heterogeneity and low quality of included studies. There is a need to further investigate possibly important clinical moderators of effect, including the type of health professional delivering interventions. Further, results indicate a need to establish a minimum core data set to improve the consistency of secondary outcome collection across trials and to design and conduct trials with longer-term follow-up periods.

TRIAL REGISTRATION

CRD42020152254 .

Characterizing modifications to a comparative effectiveness research study: the OPTIMIZE trial-using the Framework for Reporting Adaptations and Modifications to Evidence-based Interventions (FRAME)

BACKGROUND

The OPTIMIZE trial is a multi-site, comparative effectiveness research (CER) study that uses a Sequential Multiple Assessment Randomized Trial (SMART) designed to examine the effectiveness of complex health interventions (cognitive behavioral therapy, physical therapy, and mindfulness) for adults with chronic low back pain. Modifications are anticipated when implementing complex interventions in CER. Disruptions due to COVID have created unanticipated challenges also requiring modifications. Recent methodologic standards for CER studies emphasize that fully characterizing modifications made is necessary to interpret and implement trial results. The purpose of this paper is to outline the modifications made to the OPTIMIZE trial using the Framework for Reporting Adaptations and Modifications to Evidence-Based Interventions (FRAME) to characterize modifications to the OPTIMIZE trial in response to the COVID pandemic and other challenges encountered.

METHODS

The FRAME outlines a strategy to identify and report modifications to evidence-based interventions or implementation strategies, whether planned or unplanned. We use the FRAME to characterize the process used to modify the aspects of the OPTIMIZE trial. Modifications were made to improve lower-than-anticipated rates of treatment initiation and COVID-related restrictions. Contextual modifications were made to permit telehealth delivery of treatments originally designed for in-person delivery. Training modifications were made with study personnel to provide more detailed information to potential participants, use motivational interviewing communication techniques to clarify potential participants' motivation and possible barriers to initiating treatment, and provide greater assistance with scheduling of assigned treatments.

RESULTS

Modifications were developed with input from the trial's patient and stakeholder advisory panels. The goals of the modifications were to improve trial feasibility without compromising the interventions' core functions. Modifications were approved by the study funder and the trial steering committee.

CONCLUSIONS

Full and transparent reporting of modifications to clinical trials, whether planned or unplanned, is critical for interpreting the trial's eventual results and considering future implementation efforts.

TRIAL REGISTRATION

ClinicalTrials.gov NCT03859713. Registered on March 1, 2019.

Evaluating the implementation fidelity to a successful nurse-led model (INTERCARE) which reduced nursing home unplanned hospitalisations

BACKGROUND

Implementation fidelity assesses the degree to which an intervention is delivered as it should be. Fidelity helps to determine if the outcome(s) of an intervention are attributed to the intervention itself or to a failure of its implementation. Little is known about how fidelity impacts the intended outcome(s) and what elements or moderators can affect the fidelity trajectory over time. We exemplify the meaning of implementation fidelity with INTERCARE, a nurse-led care model that was implemented in eleven Swiss nursing homes (NHs) and showed effectiveness in reducing unplanned hospital transfers. INTERCARE comprises six core elements, including advance care planning and tools to support inter- and interprofessional communication, which were introduced with carefully developed implementation strategies.

METHODS

A mixed-methods convergent/triangulation design was used to investigate the influence of implementation fidelity on unplanned transfers. A fidelity questionnaire measuring the degree of fidelity to INTERCARE's core components was fielded at four time points in the participating NHs. Two-monthly meetings were conducted with NHs (September 2018-January 2020) and structured notes were used to determine moderators affecting fidelity (e.g., participant responsiveness). We used the fidelity scores and generalized linear mixed models to analyze the quantitative data. The Framework method was used for the qualitative analysis. The quantitative and qualitative findings were integrated using triangulation.

RESULTS

A higher overall fidelity score showed a decreasing rate of unplanned hospital transfers post-intervention (OR: 0.65 (CI = 0.43-0.99), p = 0.047). A higher fidelity score to advance care planning was associated with lower unplanned transfers (OR = 0.24 (CI 0.13-0.44), p = < 0.001) and a lower fidelity score for communication tools (e.g., ISBAR) to higher rates in unplanned transfers (OR = 1.69 (CI 1.30-2.19), p = < 0.003). In-house physicians with a collaborative approach and staff's perceived need for nurses working in extended roles, were important moderators to achieve and sustain high fidelity.

CONCLUSION

Implementation fidelity is challenging to measure and report, especially in complex interventions, yet is crucial to better understand how such interventions may be tailored for scale-up. This study provides both a detailed description of how fidelity can be measured and which ingredients highly contributed to reducing unplanned NH transfers.

TRIAL REGISTRATION

The INTERCARE study was registered at clinicaltrials.gov Protocol Record NCT03590470.

Challenges of conducting of online educational programs for family caregivers of people with dementia living at home: An integrative review

Objectives

This integrative review aimed to understand the challenges of conducting online educational programs for family caregivers of people with dementia by focusing on the components and design of them.

Methods

Following Whittemore & Knafl's five-step method, seven databases were systematically searched. The Mixed Methods Appraisal Tool was used to evaluate the quality of the studies.

Results

Of the 25,256 articles identified, 49 studies were included. Limitations in components (including useless or repetitive information, incomplete access to dementia-related information, the impact of components related to culture or ethnicity or gender) and in the format of delivered information (including less interaction, time schedule limitations and preference for traditional forms of delivery of information) make it more challenging to conduct online educational programs. Additionally, implementation constraints such as technical problems, poor computer literacy, and fidelity assessment are challenges that cannot be ignored.

Conclusions

Insight into the challenges of online educational programs for family caregivers of people with dementia can help guide researchers in constructing the optimal online educational program. Incorporating cultural specificity, considering structured construction strategies, optimizing interaction design, and increasing fidelity assessment may contribute to the conduct of online educational programs.

Around the EQUATOR with clinician-scientists transdisciplinary aging research (Clin-STAR) principles: Implementation science challenges and opportunities

The Institute of Medicine and the National Institute on Aging increasingly understand that knowledge alone is necessary but insufficient to improve healthcare outcomes. Adapting the behaviors of clinicians, patients, and stakeholders to new standards of evidence-based clinical practice is often significantly delayed. In response, over the past twenty years, Implementation Science has developed as the study of methods and strategies that facilitate the uptake of evidence-based practice into regular use by practitioners and policymakers. One important advance in Implementation Science research was the development of Standards for Reporting Implementation Studies (StaRI), which provided a 27-item checklist for researchers to consistently report essential elements of the implementation and intervention strategies. Using StaRI as a framework, this review discusses specific Implementation Science challenges for research with older adults, provides solutions for those obstacles, and opportunities to improve the value of this evolving approach to reduce the knowledge translation losses that exist between published research and clinical practice.

Measurement and assessment of fidelity and competence in nonspecialist-delivered, evidence-based behavioral and mental health interventions: A systematic review

Nonspecialists have increasingly been used to deliver evidence-based, mental health and behavioral interventions in lower resource settings where there is a dearth of specialized providers and a corresponding gap in service delivery. Recent literature acknowledges that nonspecialist-delivered interventions are shown to be effective. However, few studies report on the fidelity (the degree to which an intervention was implemented as intended) and/or competence (general skills of nonspecialists), key concepts that measure quality of evidence-based intervention delivery. This study seeks to understand how both fidelity and competence have been assessed in nonspecialist-delivered, evidence-based interventions with an intended social or psychological behavior-change outcome. Our search results originally yielded 2317 studies, and ultimately, 16 were included in our final analysis. Generally, results from a narrative synthesis indicated that tools used in the studies demonstrated sufficient inter-rater reliability and intra-class correlation components. Included studies used and described a range of fidelity and competence tools. However, the ENhancing Assessment of Common Therapeutic factors tool was the most commonly used tool that measures competence of nonspecialists, and has been adapted to several other settings. The roles of supervisors in mentoring, monitoring, and supervising nonspecialists emerged as a key ingredient for ensuring fidelity. Most studies assessing fidelity were limited by small sample sizes due to low numbers of nonspecialists implementing interventions, however, more advanced statistical methods may not be needed and may actually impede community-based organizations from assessing fidelity data. Our results suggest interventions can share resources, tools, and compare findings regardless with proper supervision. While the two terms "fidelity" and "competence" are often used interchangeably, their differences are noteworthy. Ultimately, both competency and fidelity are critical for delivering evidence-based interventions, and nonspecialists are most effective when they can be evaluated and mentored on both throughout the course of the intervention.

A protocol for monitoring fidelity of a preconception-life course intervention in a middle-income setting: the Healthy Life Trajectories Initiative (HeLTI), South Africa

INTRODUCTION

Despite the importance of intervention fidelity in interpreting the outcomes of complex public health interventions, there is a lack of both reporting fidelity trial protocols and uniformity. In evaluating complex, adaptable/pragmatic interventions in resource-strapped settings with systemic issues, unique challenges to intervention adherence and monitoring are introduced, increasing the importance of a fidelity protocol. We aim to describe the intervention fidelity and monitoring protocol for the Healthy Life Trajectories Initiative (HeLTI) South Africa, a complex four-phase intervention set in urban Soweto, starting preconceptionally and continuing through to pregnancy, infancy, and early childhood to improve the health of young women and reduce the intergenerational risk of obesity.

METHODS

The HeLTI SA fidelity protocol was based on the NIH Behaviour Change Consortium (NIH BCC) Treatment Fidelity Framework, outlining the following components of intervention fidelity: study design, provider training, intervention delivery, intervention receipt, and intervention enactment. Context-specific fidelity challenges were identified. The intervention fidelity components and associated monitoring strategies were developed to align with HeLTI SA. Strategies for fidelity monitoring include, amongst others, qualitative process evaluation methods, reviewing observed and recorded intervention sessions, monitoring of activity logs, standardized training, and intervention session checklists. Possible challenges to fidelity and fidelity monitoring include high provider turnover, lack of qualification amongst providers, difficulty tracing participants for follow-up sessions, participant health literacy levels, and the need to prioritize participants' non-health-related challenges. Solutions proposed include adapting intervention delivery methods, recruitment methods, and provider training methods.

DISCUSSION

The NIH BCC Treatment Fidelity Framework provided a solid foundation for reporting intervention fidelity across settings to improve intervention validity, ability to assess intervention effectiveness, and transparency. However, context-specific challenges to fidelity (monitoring) were identified, and transparency around such challenges and possible solutions in low- and middle-income settings could help foster solutions to improve adherence, reporting, and monitoring of intervention fidelity in this setting.

TRIAL REGISTRATION

Pan African Clinical Trials Registry PACTR201903750173871 . Registered on 27 March 2019.

Association of Interprofessional Discharge Planning Using an Electronic Health Record Tool With Hospital Length of Stay Among Patients with Multimorbidity: A Nonrandomized Controlled Trial

IMPORTANCE

Whether interprofessional collaboration is effective and safe in decreasing hospital length of stay remains controversial.

OBJECTIVE

To evaluate the outcomes and safety associated with an electronic interprofessional-led discharge planning tool vs standard discharge planning to safely reduce length of stay among medical inpatients with multimorbidity.

DESIGN, SETTING, AND PARTICIPANTS

This multicenter prospective nonrandomized controlled trial used interrupted time series analysis to examine medical acute hospitalizations at 82 hospitals in Switzerland. It was conducted from February 2017 through January 2019. Data analysis was conducted from March 2021 to July 2022.

INTERVENTION

After a 12-month preintervention phase (February 2017 through January 2018), an electronic interprofessional-led discharge planning tool was implemented in February 2018 in 7 intervention hospitals in addition to standard discharge planning.

MAIN OUTCOMES AND MEASURES

Mixed-effects segmented regression analyses were used to compare monthly changes in trends of length of stay, hospital readmission, in-hospital mortality, and facility discharge after the implementation of the tool with changes in trends among control hospitals.

RESULTS

There were 54 695 hospitalizations at intervention hospitals, with 27 219 in the preintervention period (median [IQR] age, 72 [59-82] years; 14 400 [52.9%] men) and 27 476 in the intervention phase (median [IQR] age, 72 [59-82] years; 14 448 [52.6%] men) and 438 791 at control hospitals, with 216 261 in the preintervention period (median [IQR] age, 74 [60-83] years; 109 770 [50.8%] men) and 222 530 in the intervention phase (median [IQR] age, 74 [60-83] years; 113 053 [50.8%] men). The mean (SD) length of stay in the preintervention phase was 7.6 (7.1) days for intervention hospitals and 7.5 (7.4) days for control hospitals. During the preintervention phase, population-averaged length of stay decreased by -0.344 hr/mo (95% CI, -0.599 to -0.090 hr/mo) in control hospitals; however, no change in trend was observed among intervention hospitals (-0.034 hr/mo; 95% CI, -0.646 to 0.714 hr/mo; difference in slopes, P = .09). Over the intervention phase (February 2018 through January 2019), length of stay remained unchanged in control hospitals (slope, -0.011 hr/mo; 95% CI, -0.281 to 0.260 hr/mo; change in slope, P = .03), but decreased steadily among intervention hospitals by -0.879 hr/mo (95% CI, -1.607 to -0.150 hr/mo; change in slope, P = .04, difference in slopes, P = .03). Safety analyses showed no change in trends of hospital readmission, in-hospital mortality, or facility discharge over the whole study time.

CONCLUSIONS AND RELEVANCE

In this nonrandomized controlled trial, the implementation of an electronic interprofessional-led discharge planning tool was associated with a decline in length of stay without an increase in hospital readmission, in-hospital mortality, or facility discharge.

TRIAL REGISTRATION

isrctn.org Identifier: ISRCTN83274049.

Delivery fidelity of the REACT (REtirement in ACTion) physical activity and behaviour maintenance intervention for community dwelling older people with mobility limitations

BACKGROUND

Fidelity assessment of behaviour change interventions is vital to understanding trial outcomes. This study assesses the delivery fidelity of behaviour change techniques used in the Retirement in ACTion (REACT) randomised controlled trial. REACT is a community-based physical activity (PA) and behaviour maintenance intervention to prevent decline of physical functioning in older adults (≥ 65 years) at high risk of mobility-related disability in the UK.

METHODS

The delivery fidelity of intervention behaviour change techniques and delivery processes were assessed using multi-observer coding of purposively sampled in-vivo audio recordings (n = 25) of health behaviour maintenance sessions over 12-months. Delivery fidelity was scored using a modified Dreyfus scale (scores 0-5) to assess competence and completeness of delivery for each technique and delivery process. "Competent delivery" was defined as a score of 3 points or more for each item. Examples of competent intervention delivery were identified to inform recommendations for future programme delivery and training.

RESULTS

The mean intervention fidelity score was 2.5 (SD 0.45) with delivery fidelity varying between techniques/processes and intervention groups. Person-centred delivery, Facilitating Enjoyment and Promoting Autonomy were delivered competently (scoring 3.0 or more). There was scope for improvement (score 2.0-2.9) in Monitoring Progress (Acknowledging and Reviewing), Self-Monitoring, Monitoring Progress (Eliciting Benefits of Physical Activity), Goal Setting and Action Planning, Modelling, Supporting Self-Efficacy for Physical Activity and Supporting Relatedness. Managing Setbacks and Problem Solving was delivered with low fidelity. Numerous examples of both good and sub-optimal practice were identified.

CONCLUSIONS

This study highlights successes and improvements needed to enhance delivery fidelity in future implementation of the behavioural maintenance programme of the REACT intervention. Future training of REACT session leaders and assessment of delivery fidelity needs to focus on the delivery of Goal setting and Action Planning, Modelling, Supporting Relatedness, Supporting Self-Efficacy for Physical Activity, and Managing Setbacks/ Problem Solving.

SCOPE: safer care for older persons (in residential) environments-a pilot study to enhance care aide-led quality improvement in nursing homes

Background

Nursing home residents require daily support. While care aides provide most of this support they are rarely empowered to lead quality improvement (QI) initiatives. Researchers have shown that care aide-led teams can successfully participate in a QI intervention called Safer Care for Older Persons in Residential Care Environments (SCOPE). In preparation for a large-scale study, we conducted a 1-year pilot to evaluate how well coaching strategies helped teams to enact this intervention. Secondarily, we measured if improvements in team cohesion and communication, and resident quality of care, occurred.

Methods

This study was conducted using a prospective single-arm study design, on 7 nursing homes in Winnipeg Manitoba belonging to the Translating Research in Elder Care research program. One QI team was selected per site, led by care aides who partnered with other front-line staff. Each team received facilitated coaching to enact SCOPE during three learning sessions, and additional support from quality advisors between these sessions. Researchers developed a rubric to evaluate how well teams enacted their interventions (i.e., created actionable aim statements, implemented interventions using plan-do-study-act cycles, and used measurement to guide decision-making). Team cohesion and communication were measured using surveys, and changes in unit-level quality indicators were measured using Resident Assessment Instrument-Minimum Data Set data.

Results

Most teams successfully enacted their interventions. Five of 7 teams created adequate-to-excellent aim statements. While 6 of 7 teams successfully implemented plan-do-study-act cycles, only 2 reported spreading their change ideas to other residents and staff on their unit. Three of 7 teams explicitly stated how measurement was used to guide intervention decisions. Teams scored high in cohesion and communication at baseline, and hence improved minimally. Indicators of resident quality care improved in 4 nursing home units; teams at 3 of these sites were scored as ‘excellent’ in two or more enactment areas, versus 1 of the 3 remaining teams.

Conclusions

Our coaching strategies helped most care aide-led teams to enact SCOPE. Coaching modifications are needed to help teams more effectively use measurement. Refinements to our evaluation rubric are also recommended.

Supplementary Information

The online version contains supplementary material available at 10.1186/s40814-022-00975-8.