Operationalizing a model to quantify implementation of a multi-component intervention in a stepped-wedge trial

A participatory and comprehensive intervention to improve violence prevention in two high-risk occupations: effect and process evaluation of a stepped wedge cluster randomised trial

BACKGROUND

Work-related violence committed by clients, patients, and customers represents a major occupational health risk for employees that needs to be reduced.

METHODS

We tested a comprehensive violence prevention intervention involving active participation of both employees and managers in the Prison and Probation Service (PPS) and on psychiatric wards in Denmark. We used a stepped wedge cluster randomised controlled trial design. We measured the degree of implementation of the intervention by registration of fidelity, reach, and dose and used a mixed-effects regression analysis to estimate the effects of the intervention.

RESULTS

We recruited 16 work units for the intervention, but three work units dropped out. The average implementation rate was 73%. In the psychiatric wards, the intervention led to statistically significant improvements in the primary outcome (an increase in the degree to which managers and employees continuously work on violence prevention practices based on their registration and experiences), but none statistically significant improvements in any of the secondary outcomes. In the PPS units, the intervention did not lead to a statistically significant improvement in the primary outcome, but to statistically significant improvements in three secondary outcomes.

CONCLUSION

Most work units were able to carry out the intervention as planned. The intervention showed mixed results regarding the primary outcome. Nevertheless, the results indicate improvements also in the sector where a change in the primary outcome was not achieved. The results point at that a participatory and comprehensive approach could be a viable way of working with violence prevention in high-risk workplaces.

TRIAL REGISTRATION

ISRCTN86993466: 20/12/2017.

What do we want to get out of this? a critical interpretive synthesis of the value of process evaluations, with a practical planning framework

BACKGROUND

Process evaluations aim to understand how complex interventions bring about outcomes by examining intervention mechanisms, implementation, and context. While much attention has been paid to the methodology of process evaluations in health research, the value of process evaluations has received less critical attention. We aimed to unpack how value is conceptualised in process evaluations by identifying and critically analysing 1) how process evaluations may create value and 2) what kind of value they may create.

METHODS

We systematically searched for and identified published literature on process evaluation, including guidance, opinion pieces, primary research, reviews, and discussion of methodological and practical issues. We conducted a critical interpretive synthesis and developed a practical planning framework.

RESULTS

We identified and included 147 literature items. From these we determined three ways in which process evaluations may create value or negative consequences: 1) through the socio-technical processes of 'doing' the process evaluation, 2) through the features/qualities of process evaluation knowledge, and 3) through using process evaluation knowledge. We identified 15 value themes. We also found that value varies according to the characteristics of individual process evaluations, and is subjective and context dependent.

CONCLUSION

The concept of value in process evaluations is complex and multi-faceted. Stakeholders in different contexts may have very different expectations of process evaluations and the value that can and should be obtained from them. We propose a planning framework to support an open and transparent process to plan and create value from process evaluations and negotiate trade-offs. This will support the development of joint solutions and, ultimately, generate more value from process evaluations to all.

Implementation fidelity in a multifaceted program to foster rational antibiotics use in primary care: an observational study

BACKGROUND

The ARena study (Sustainable Reduction of Antimicrobial Resistance in German Ambulatory Care) is a three-arm, cluster randomized trial to evaluate a multifaceted implementation program in a German primary care setting. In the context of a prospective process evaluation conducted alongside ARena, this study aimed to document and explore fidelity of the implementation program.

METHODS

This observational study is based on data generated in a three-wave survey of 312 participating physicians in the ARena program and attendance documentation. Measures concerned persistence of participation in the ARena program and adherence to intervention components (thematic quality circles, e-learning, basic expenditure reimbursements, additional bonus payments and a computerized decision support system). Participants' views on five domains of the implementation were also measured. Binary logistic and multiple linear regression analyses were used to explore which views on the implementation were associated with participants' adherence to quality circles and use of additional bonus compensation.

RESULTS

The analysis of fidelity showed overall high persistence of participation in the intervention components across the three intervention arms (90,1%; 97,9%; 92,9%). 96.4% of planned quality circles were delivered to study participants and, across waves, 30.4% to 93% of practices participated; 56.1% of physicians attended the maximum of four quality circles. 84% of the practices (n = 158) with a minimum of one index patient received a performance-based additional bonus payment at least once. In total, bonus compensation was triggered for 51.8% of affected patients. Participation rate for e-learning (a prerequisite for reimbursement of project-related expenditure) covered 90.8% of practices across all intervention arms, with the highest rate in arm II (96.5%). Uptake of expenditure reimbursement was heterogeneous across study arms, with a mean rate of 86.5% (89.1% in arm I, 96.4% in arm II and 74.1% in arm III). Participants' views regarding participant responsiveness (OR = 2.298) 95% CI [1.598, 3.305] and Context (OR = 2.146) 95% CI [1.135, 4.055] affected additional bonus payment. Participants' views on participant responsiveness (Beta = 0.718) 95% CI [0.479, 0.957], Context (Beta = 0.323) 95% CI [0.055, 0.590] and Culture of shared decision-making (Beta = -0.334) 95% CI [-0.614, -0.053] affected quality circle attendance.

CONCLUSION

This study showed an overall high fidelity to the implementation program. Participants' views on the implementation were associated with degree of intervention fidelity.

TRIAL REGISTRATION

ISRCTN, ISRCTN58150046.

Is level of implementation linked with intervention outcomes? Process evaluation of the TransformUs intervention to increase children’s physical activity and reduce sedentary behaviour

Background

TransformUs was a four-arm school-based intervention to increase physical activity and reduce sedentary behaviour among primary school children. Pedagogical and environmental strategies targeted the classroom, school grounds and family setting. The aims of this study were to evaluate program fidelity, dose, appropriateness, satisfaction and sustainability, and associations between implementation level and outcomes among the three intervention arms.

Methods

At baseline, 18-months (mid-intervention) and 30-months (post-intervention), teachers, parents and children completed surveys, and children wore GT3X ActiGraph accelerometers for 8 days at each time point to determine physical activity and sedentary time. Implementation data were pooled across the three intervention groups and teachers were categorised by level of implementation: (i) ‘Low’ (< 33% delivered); (ii) ‘Moderate’ (33–67% delivered); and (iii) ‘High’ (> 67% delivered). Linear and logistic mixed models examined between group differences in implementation, and the association with children’s physical activity and sedentary time outcomes. Qualitative survey data were analysed thematically.

Results

Among intervention recipients, 52% (n = 85) of teachers, 29% (n = 331) of parents and 92% (n = 407) of children completed baseline evaluation surveys. At 18-months, teachers delivered on average 70% of the key messages, 65% set active/standing homework, 30% reported delivering > 1 standing lesson/day, and 56% delivered active breaks per day. The majority of teachers (96%) made activity/sports equipment available during recess and lunch, and also used this equipment in class (81%). Fidelity and dose of key messages and active homework reduced over time, whilst fidelity of standing lessons, active breaks and equipment use increased. TransformUs was deemed appropriate for the school setting and positively received. Implementation level and child behavioural outcomes were not associated. Integration of TransformUs into existing practices, children’s enjoyment, and teachers’ awareness of program benefits all facilitated delivery and sustainability.

Conclusions

This study demonstrated that intervention dose and fidelity increased over time, and that children’s enjoyment, senior school leadership and effective integration of interventions into school practices facilitated improved intervention delivery and sustainability. Teacher implementation level and child behavioural outcomes were unrelated, suggesting intervention efficacy was achieved irrespective of implementation variability. The potential translatability of TransformUs into practice contexts may therefore be increased. Findings have informed scale-up of TransformUs across Victoria, Australia.

Trial registration

International Standard Randomized Controlled Trial Number ISRCTN83725066; Australian New Zealand Clinical Trials Registry Number ACTRN12609000715279. Registered 19 August 2009. Available at: https://www.anzctr.org.au/Trial/Registration/TrialReview.aspx?id=308387&isReview=true

Supplementary Information

The online version contains supplementary material available at 10.1186/s12966-022-01354-5.

Reducing smoking in youth by a smoke-free school environment: A stratified cluster randomized controlled trial of Focus, a multicomponent program for alternative high schools

Youth smoking remains a major challenge for public health. Socioeconomic position influences the initiation and maintenance of smoking, and alternative high school students are at particularly high risk. The school environment is an important setting to promote health, however there is a lack of evidence-based school intervention programs. This article presents the Focus study, which aims to test the implementation and effectiveness of a school-based intervention integrating¹ a comprehensive school smoking policy [i.e. smoke-free school hours (SFSH)]², a course for school staff in short motivational conversations³, school class-based teaching material⁴, an edutainment session5, a class-based competition, and6 access to smoking cessation support. Together these intervention components address students' acceptability of smoking, social influences, attitudes, motivation, and opportunities for smoking. The setting is alternative high schools across Denmark, and the evaluation design is based on a stratified cluster randomized controlled trial comparing the intervention group to a control group. Outcome data is collected at baseline, midway, and at the end of the intervention period. Moreover, a detailed process evaluation, using qualitative and quantitative methods, is conducted among students, teachers, and school principals. The results from this trial will provide important knowledge on the effectiveness of a smoke-free school environment. The findings will lead to a better understanding of which policies, environments, and cognitions, contribute to preventing and reducing cigarette use among young people in a diverse and high-risk school setting, and illuminate which complementary factors are significant to achieve success when implementing SFSH.

Implementation of the acutely presenting older patient (APOP) screening program in routine emergency department care : A before-after study

OBJECTIVE

The aim of this study was to evaluate the effects of implementation of the acutely presenting older patient (APOP) screening program for older patients in routine emergency department (ED) care shortly after implementation.

METHODS

We conducted an implementation study with before-after design, using the plan-do-study-act (PDSA) model for quality improvement, in the ED of a Dutch academic hospital. All consecutive patients ≥ 70 years during 2 months before and after implementation were included. The APOP program comprises screening for risk of functional decline, mortality and cognitive impairment, targeted interventions for high-risk patients and education of professionals. Outcome measures were compliance with interventions and impact on ED process, length of stay (LOS) and hospital admission rate.

RESULTS

Two comparable groups of patients (median age 77 years) were included before (n = 920) and after (n = 953) implementation. After implementation 560 (59%) patients were screened of which 190 (34%) were high-risk patients. Some of the program interventions for high-risk patients in the ED were adhered to, some were not. More hospitalized patients received comprehensive geriatric assessment (CGA) after implementation (21% before vs. 31% after; p = 0.002). In 89% of high-risk patients who were discharged to home, telephone follow-up was initiated. Implementation did not influence median ED LOS (202 min before vs. 196 min after; p = 0.152) or hospital admission rate (40% before vs. 39% after; p = 0.410).

CONCLUSION

Implementation of the APOP screening program in routine ED care did not negatively impact the ED process and resulted in an increase of CGA and telephone follow-up in older patients. Future studies should investigate whether sustainable changes in management and patient outcomes occur after more PDSA cycles.

Effectiveness of a structured, framework-based approach to implementation: the Researching Effective Approaches to Cleaning in Hospitals (REACH) Trial

Background

Implementing sustainable practice change in hospital cleaning has proven to be an ongoing challenge in reducing healthcare associated infections. The purpose of this study was to develop a reliable framework-based approach to implement and quantitatively evaluate the implementation of evidence-based practice change in hospital cleaning.

Design/methods

The Researching Effective Approaches to Cleaning in Hospitals (REACH) trial was a pragmatic, stepped-wedge randomised trial of an environmental cleaning bundle implemented in 11 Australian hospitals from 2016 to 2017. Using a structured multi-step approach, we adapted the integrated Promoting Action on Research Implementation in Health Services (i-PARIHS) framework to support rigorous and tailored implementation of the cleaning bundle intervention in eleven diverse and complex settings. To evaluate the effectiveness of this strategy we examined post-intervention cleaning bundle alignment calculated as a score (an implementation measure) and cleaning performance audit data collected using ultraviolet (UV) gel markers (an outcome measure).

Results

We successfully implemented the bundle and observed improvements in cleaning practice and performance, regardless of hospital size, intervention duration and contextual issues such as staff and organisational readiness at baseline. There was a positive association between bundle alignment scores and cleaning performance at baseline. This diminished over the duration of the intervention, as hospitals with lower baseline scores were able to implement practice change successfully.

Conclusion

Using a structured framework-based approach allows for pragmatic and successful implementation of clinical trials across diverse settings, and assists with quantitative evaluation of practice change.

Trial registration

Australia New Zealand Clinical Trial Registry ACTRN12615000325505, registered on 4 September 2015.

Can childcare work be designed to promote moderate and vigorous physical activity, cardiorespiratory fitness and health? Study protocol for the Goldilocks-childcare randomised controlled trial

BACKGROUND

Despite extensive efforts, issues like obesity and poor physical capacity remain challenges for a healthy work life in several occupations. The Goldilocks work principle offers a new approach, encouraging design of productive work to promote physical capacity and health. This paper presents the protocol for the Goldilocks-childcare study, a randomised controlled intervention trial aiming to evaluate the effectiveness of implementing the Goldilocks work principle in childcare. The primary aim of the intervention is to increase time in moderate to vigorous physical activity (MVPA) by having the childcare workers act as active role models for children in daily playful physical activities, and thereby improve cardiorespiratory fitness and health of the workers.

METHODS

The study is a cluster-randomised trial with a usual-practice wait-list control group. The 10-week intervention consists of two phases. In the first, the childcare workers will participate in two participatory workshops aiming to a) develop playful physical activities ('Goldilocks-games') for children in which childcare workers participate as active role models at MVPA intensity, and b) develop action plans for implementation of the Goldilocks-games in daily work routines. In the second phase, childcare institutions will implement the Goldilocks-games. The primary outcome is working time spent in MVPA, and secondary outcomes are cardiorespiratory fitness, sleeping heart rate, perceived need for recovery, and productivity. Primary outcome and process evaluation will be based on direct measurements of physical activity and heart rate, determination of cardiorespiratory fitness, and questionnaires.

DISCUSSION

If proven effective, the Goldilocks work principle has a large potential for promoting sustainable health and working lives of childcare workers.

TRIAL REGISTRATION

ISRCTN, ISRCTN15644757, Registered 25th December 2019.

A theory-based, task-oriented, outdoor walking programme for older adults with difficulty walking outdoors: protocol for the Getting Older Adults Outdoors (GO-OUT) randomised controlled trial

INTRODUCTION

A theory-based, task-oriented, community walking programme can increase outdoor walking activity among older adults to optimise functional independence, social participation and well-being. The study objective is to determine if there is a difference in the change in outdoor walking activity from baseline to 10 weeks, 5.5 months and 12 months after receiving a 1-day interactive workshop and outdoor walking programme (Getting Older Adults Outdoors (GO-OUT)) compared with the workshop and weekly reminders (WR) in older adults with difficulty walking outdoors.

METHODS AND ANALYSIS

A randomised controlled trial is being conducted in four urban Canadian communities. We will stratify 240 individuals by site and participant type (ie, individual vs spousal/friend pair) and randomise to either the GO-OUT or WR intervention. The GO-OUT intervention involves a 1-day workshop, where participants complete eight interactive stations to build knowledge and skills to walk outside, followed by a 10-week group outdoor walking programme (two 1-hour sessions/week) led by a physiotherapist or kinesiologist in parks. The WR intervention consists of the same workshop and 10 weekly telephone reminders to facilitate outdoor walking. The primary outcome measure is mean outdoor walking time in minutes/week derived from accelerometry and global positioning system data. GO-OUT is powered to detect an effect size of 0.4, given α=0.05, β=0.20, equal number of participants/group and a 20% attrition rate. Secondary outcomes include physical activity, lifespace mobility, participation, health-related quality of life, balance, leg strength, walking self-efficacy, walking speed, walking distance/endurance and mood.

ETHICS AND DISSEMINATION

GO-OUT has received ethics approval at all sites. A Data Safety Monitoring Board will monitor adverse events. We will disseminate findings through lay summaries, conference presentations and journal articles.

TRIAL REGISTRATION NUMBER

NCT03292510 (Pre-results).

A stepped wedge cluster randomised trial of nurse-delivered Teach-Back in a consumer telehealth service

OBJECTIVE

To evaluate the impact of Teach-Back on communication quality in a national telephone-based telehealth service, for callers varying in health literacy.

DESIGN

Cross-sectional stepped wedge cluster randomised trial with continuous recruitment, short (fixed) exposure and blinded outcome assessors. Nurses were stratified by hours worked and randomised into training groups using a computer generated sequence.

SETTING

An Australian national pregnancy and parenting telephone helpline.

INTERVENTION

Complex intervention involving a single 2-hour group Teach-Back training session, combined with ongoing nurse self-reflection on their communication following each call and each shift.

PARTICIPANTS

Participants were 637 callers to the telephone helpline aged 18-75 (87% female), of whom 127 (13%) had inadequate health literacy (measured with the Single Item Literacy Screener); and 15 maternal and child health nurses with 15 years' experience on average.

MEASURES

Primary outcome was a modified subscale of the Health Literacy Questionnaire, 'having sufficient information to manage health'. Secondary caller outcomes included caller confidence, perceived actionability of information and nurse effort to listen and understand. Nurse outcomes were perceptions of their communication effectiveness.

RESULTS

Over a 7 week period, 376 surveyed callers received usual care and 261 Teach-Back. Ratings on the primary outcome increased over time (OR 1.17, CI 1.01 to 1.32, p = 0.03) but no independent Teach-Back effect was observed. A consistent pattern suggests that, compared with usual care, Teach-Back helps callers with inadequate health literacy feel listened to (OR 2.3, CI 0.98 to 5.42, p = 0.06), confident to act (OR 2.44, CI 1.00 to 5.98, p = 0.06), and know what steps to take (OR 2.68, CI 1.00 to 7.17, p = 0.06). Nurse perceptions of both their own communication effectiveness (OR = 2.31; CI 1.38 to 3.86, p<0.0001), and caller understanding (OR = 2.56; CI 1.52 to 4.30, p<0.001) both increased with Teach-Back. No harms were reported.

CONCLUSIONS

Teach-Back appears to benefit telephone health service users with inadequate health literacy, but the extent of this is unclear due to smaller numbers of lower literacy participants. Improving caller ratings over time are likely due to increasing nurse communication competence.