Developing strategies to improve fidelity of delivery of, and engagement with, a complex intervention to improve independence in dementia: a mixed methods study

Consolidated guidance for behavioral intervention pilot and feasibility studies

BACKGROUND

In the behavioral sciences, conducting pilot and/or feasibility studies (PFS) is a key step that provides essential information used to inform the design, conduct, and implementation of a larger-scale trial. There are more than 160 published guidelines, reporting checklists, frameworks, and recommendations related to PFS. All of these publications offer some form of guidance on PFS, but many focus on one or a few topics. This makes it difficult for researchers wanting to gain a broader understanding of all the relevant and important aspects of PFS and requires them to seek out multiple sources of information, which increases the risk of missing key considerations to incorporate into their PFS. The purpose of this study was to develop a consolidated set of considerations for the design, conduct, implementation, and reporting of PFS for interventions conducted in the behavioral sciences.

METHODS

To develop this consolidation, we undertook a review of the published guidance on PFS in combination with expert consensus (via a Delphi study) from the authors who wrote such guidance to inform the identified considerations. A total of 161 PFS-related guidelines, checklists, frameworks, and recommendations were identified via a review of recently published behavioral intervention PFS and backward/forward citation tracking of a well-known PFS literature (e.g., CONSORT Ext. for PFS). Authors of all 161 PFS publications were invited to complete a three-round Delphi survey, which was used to guide the creation of a consolidated list of considerations to guide the design, conduct, and reporting of PFS conducted by researchers in the behavioral sciences.

RESULTS

A total of 496 authors were invited to take part in the three-round Delphi survey (round 1, N = 46; round 2, N = 24; round 3, N = 22). A set of twenty considerations, broadly categorized into six themes (intervention design, study design, conduct of trial, implementation of intervention, statistical analysis, and reporting) were generated from a review of the 161 PFS-related publications as well as a synthesis of feedback from the three-round Delphi process. These 20 considerations are presented alongside a supporting narrative for each consideration as well as a crosswalk of all 161 publications aligned with each consideration for further reading.

CONCLUSION

We leveraged expert opinion from researchers who have published PFS-related guidelines, checklists, frameworks, and recommendations on a wide range of topics and distilled this knowledge into a valuable and universal resource for researchers conducting PFS. Researchers may use these considerations alongside the previously published literature to guide decisions about all aspects of PFS, with the hope of creating and disseminating interventions with broad public health impact.

Development of fidelity of delivery and enactment measures for interventions in communication disorders

OBJECTIVES

This study was part of a process evaluation for a single-blind, randomized controlled pilot study comparing Better Conversations with Primary Progressive Aphasia (BCPPA), an approach to communication partner training, with no speech and language therapy treatment. It was necessary to explore fidelity of delivery (delivery of intervention components) and intervention enactment (participants' use of intervention skills in the form of conversation behaviours comprising facilitators, that enhance the conversational flow, and barriers, that impeded the flow of conversation). This study aimed to: (1) Outline an adapted methodological process that uses video observation, to measure both fidelity of delivery and enactment. (2) Measure the extent to which the BCPPA pilot study was delivered as planned, and enacted.

DESIGN

Observational methods were used alongside statistical analysis to explore the fidelity of intervention and enactment using video recordings obtained from the BCPPA pilot study.

METHODS

A 5-step methodology, was developed to measure fidelity of delivery and enactment for the BCPPA study using video-recorded data. To identify delivery of intervention components, a random sample of eight video recorded and transcribed BCPPA intervention sessions was coded. To examine the enactment of conversation behaviours, 108 transcribed 10 -min-video recorded conversations were coded from 18 participants across the control and intervention group.

RESULTS

Checklists and guidelines for measurement of fidelity of treatment delivery and coding spreadsheets and guidelines for measurement of enactment are presented. Local collaborators demonstrated 87.2% fidelity to the BCPPA protocol. Participants in the BCPPA treatment group increased their use of facilitator behaviours enacted in conversation from a mean of 13.5 pre-intervention to 14.2 post-intervention, whilst control group facilitators decreased from a mean of 15.5 to 14.4, over the same timescale.

CONCLUSIONS

This study proposes a novel and robust methods, using video recorded intervention sessions and conversation samples, to measure both fidelity of intervention delivery and enactment. The learnings from this intervention are transferable to other communication interventions.

Expert Perspectives on Pilot and Feasibility Studies: A Delphi Study and Consolidation of Considerations for Behavioral Interventions

Background

In the behavioral sciences, conducting pilot and/or feasibility studies (PFS) is a key step that provides essential information used to inform the design, conduct, and implementation of a larger-scale trial. There are more than 160 published guidelines, reporting checklists, frameworks, and recommendations related to PFS. All of these publications offer some form of guidance on PFS, but many focus on one or a few topics. This makes it difficult for researchers wanting to gain a broader understanding of all the relevant and important aspects of PFS and requires them to seek out multiple sources of information, which increases the risk of missing key considerations to incorporate into their PFS. The purpose of this study was to develop a consolidated set of considerations for the design, conduct, implementation, and reporting of PFS for interventions conducted in the behavioral sciences.

Methods

To develop this consolidation, we undertook a review of the published guidance on PFS in combination with expert consensus (via a Delphi study) from the authors who wrote such guidance to inform the identified considerations. A total of 161 PFS-related guidelines, checklists, frameworks, and recommendations were identified via a review of recently published behavioral intervention PFS and backward/forward citation tracking of well-know PFS literature (e.g., CONSORT Ext. for PFS). Authors of all 161 PFS publications were invited to complete a three-round Delphi survey, which was used to guide the creation of a consolidated list of considerations to guide the design, conduct, and reporting of PFS conducted by researchers in the behavioral sciences.

Results

A total of 496 authors were invited to take part in the Delphi survey, 50 (10.1%) of which completed all three rounds, representing 60 (37.3%) of the 161 identified PFS-related guidelines, checklists, frameworks, and recommendations. A set of twenty considerations, broadly categorized into six themes (Intervention Design, Study Design, Conduct of Trial, Implementation of Intervention, Statistical Analysis and Reporting) were generated from a review of the 161 PFS-related publications as well as a synthesis of feedback from the three-round Delphi process. These 20 considerations are presented alongside a supporting narrative for each consideration as well as a crosswalk of all 161 publications aligned with each consideration for further reading.

Conclusion

We leveraged expert opinion from researchers who have published PFS-related guidelines, checklists, frameworks, and recommendations on a wide range of topics and distilled this knowledge into a valuable and universal resource for researchers conducting PFS. Researchers may use these considerations alongside the previously published literature to guide decisions about all aspects of PFS, with the hope of creating and disseminating interventions with broad public health impact.

A rapid mixed-methods evaluation of remote home monitoring models during the COVID-19 pandemic in England

Background

Remote home monitoring services were developed and implemented for patients with COVID-19 during the pandemic. Patients monitored blood oxygen saturation and other readings (e.g. temperature) at home and were escalated as necessary.

Objective

To evaluate effectiveness, costs, implementation, and staff and patient experiences (including disparities and mode) of COVID-19 remote home monitoring services in England during the COVID-19 pandemic (waves 1 and 2).

Methods

A rapid mixed-methods evaluation, conducted in two phases. Phase 1 (July-August 2020) comprised a rapid systematic review, implementation and economic analysis study (in eight sites). Phase 2 (January-June 2021) comprised a large-scale, multisite, mixed-methods study of effectiveness, costs, implementation and patient/staff experience, using national data sets, surveys (28 sites) and interviews (17 sites).

Results

Phase 1 Findings from the review and empirical study indicated that these services have been implemented worldwide and vary substantially. Empirical findings highlighted that communication, appropriate information and multiple modes of monitoring facilitated implementation; barriers included unclear referral processes, workforce availability and lack of administrative support. Phase 2 We received surveys from 292 staff (39% response rate) and 1069 patients/carers (18% response rate). We conducted interviews with 58 staff, 62 patients/carers and 5 national leads. Despite national roll-out, enrolment to services was lower than expected (average enrolment across 37 clinical commissioning groups judged to have completed data was 8.7%). There was large variability in implementation of services, influenced by patient (e.g. local population needs), workforce (e.g. workload), organisational (e.g. collaboration) and resource (e.g. software) factors. We found that for every 10% increase in enrolment to the programme, mortality was reduced by 2% (95% confidence interval: 4% reduction to 1% increase), admissions increased by 3% (-1% to 7%), in-hospital mortality fell by 3% (-8% to 3%) and lengths of stay increased by 1.8% (-1.2% to 4.9%). None of these results are statistically significant. We found slightly longer hospital lengths of stay associated with virtual ward services (adjusted incidence rate ratio 1.05, 95% confidence interval 1.01 to 1.09), and no statistically significant impact on subsequent COVID-19 readmissions (adjusted odds ratio 0.95, 95% confidence interval 0.89 to 1.02). Low patient enrolment rates and incomplete data may have affected chances of detecting possible impact. The mean running cost per patient varied for different types of service and mode; and was driven by the number and grade of staff. Staff, patients and carers generally reported positive experiences of services. Services were easy to deliver but staff needed additional training. Staff knowledge/confidence, NHS resources/workload, dynamics between multidisciplinary team members and patients' engagement with the service (e.g. using the oximeter to record and submit readings) influenced delivery. Patients and carers felt services and human contact received reassured them and were easy to engage with. Engagement was conditional on patient, support, resource and service factors. Many sites designed services to suit the needs of their local population. Despite adaptations, disparities were reported across some patient groups. For example, older adults and patients from ethnic minorities reported more difficulties engaging with the service. Tech-enabled models helped to manage large patient groups but did not completely replace phone calls.

Limitations

Limitations included data completeness, inability to link data on service use to outcomes at a patient level, low survey response rates and under-representation of some patient groups.

Future work

Further research should consider the long-term impact and cost-effectiveness of these services and the appropriateness of different models for different groups of patients.

Conclusions

We were not able to find quantitative evidence that COVID-19 remote home monitoring services have been effective. However, low enrolment rates, incomplete data and varied implementation reduced our chances of detecting any impact that may have existed. While services were viewed positively by staff and patients, barriers to implementation, delivery and engagement should be considered.

Study registration

This study is registered with the ISRCTN (14962466).

Funding

This project was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme (RSET: 16/138/17; BRACE: 16/138/31) and NHSEI and will be published in full in Health and Social Care Delivery Research; Vol. 11, No. 13. See the NIHR Journals Library website for further project information. The views expressed in this publication are those of the authors and not necessarily those of the National Institute for Health and Care Research or the Department of Health and Social Care.

A systematic review of process evaluations for psychosocial interventions designed to improve the wellbeing and quality of life of community-dwelling people with dementia and their carers

BACKGROUND

Psychosocial interventions improve the wellbeing and quality of life of People Living with Dementia (PLWD) and their family carers; but due to their complexity it can be challenging to identify mechanisms of action. We reviewed process evaluations that have sought to elucidate how these interventions work, to inform their implementation.

METHOD

We systematically reviewed process evaluations of studies evaluating psychosocial interventions for PLWD in their own home and/or their family carers. We rated study quality using the Mixed Methods Appraisal Tool. We described, with reference to Medical Research Council (2015) process evaluation guidance, how implementation, mechanisms of impact and contextual factors were investigated; and describe commonalities in the mechanisms of action identified across studies.

RESULTS

Twenty four included studies evaluated the processes of 22 interventions. These studies collectively applied five frameworks; almost all frameworks' advised evaluations were theory-based and used mixed-methods analyses, but only 5/24 evaluation designs were informed by the intervention theory and 8/24 used mixed methods. 8/24 evaluations considered contextual factors in their design, though 20/24 cited contextual factors in findings. Interventions were more successful where PLWD were motivated and aware of potential benefits, and when carers could support engagement and were themselves supported by the intervention. How the intervention aligned with participants' current needs and stage of dementia were key influencing factors.

CONCLUSION

Knowing how interventions can influence change for community-dwelling people with dementia and their family carer's will improve translation of trial findings into practice. Robust, theory-driven process evaluations can enable this.

Patients' experiences of, and engagement with, remote home monitoring services for COVID-19 patients: A rapid mixed-methods study

INTRODUCTION

Remote home monitoring models were implemented during the COVID-19 pandemic to shorten hospital length of stay, reduce unnecessary hospital admission, readmission and infection and appropriately escalate care. Within these models, patients are asked to take and record readings and escalate care if advised. There is limited evidence on how patients and carers experience these services. This study aimed to evaluate patient experiences of, and engagement with, remote home monitoring models for COVID-19.

METHODS

A rapid mixed-methods study was carried out in England (conducted from March to June 2021). We remotely conducted a cross-sectional survey and semi-structured interviews with patients and carers. Interview findings were summarized using rapid assessment procedures sheets and data were grouped into themes (using thematic analysis). Survey data were analysed using descriptive statistics.

RESULTS

We received 1069 surveys (18% response rate) and conducted interviews with patients (n = 59) or their carers (n = 3). 'Care' relied on support from staff members and family/friends. Patients and carers reported positive experiences and felt that the service and human contact reassured them and was easy to engage with. Yet, some patients and carers identified problems with engagement (e.g., hesitancy to self-escalate care). Engagement was influenced by patient factors such as health and knowledge, support from family/friends and staff, availability and ease of use of informational and material resources (e.g., equipment) and service factors.

CONCLUSION

Remote home monitoring models place responsibility on patients to self-manage symptoms in partnership with staff; yet, many patients required support and preferred human contact (especially for identifying problems). Caring burden and experiences of those living alone and barriers to engagement should be considered when designing and implementing remote home monitoring services.

PATIENT OR PUBLIC CONTRIBUTION

The study team met with service users and public members of the evaluation teams throughout the project in a series of workshops. Workshops informed study design, data collection tools and data interpretation and were conducted to also discuss study dissemination. Public patient involvement (PPI) members helped to pilot patient surveys and interview guides with the research team. Some members of the public also piloted the patient survey. Members of the PPI group were given the opportunity to comment on the manuscript, and the manuscript was amended accordingly.

Parallel randomized controlled feasibility trials of the "Active Brains" digital intervention to protect cognitive health in adults aged 60-85

Introduction

Multidomain interventions to address modifiable risk factors for dementia are promising, but require more cost-effective, scalable delivery. This study investigated the feasibility of the "Active Brains" digital behavior change intervention and its trial procedures.

Materials and methods

Active Brains aims to reduce cognitive decline by promoting physical activity, healthy eating, and online cognitive training. We conducted 12-month parallel-design randomized controlled feasibility trials of "Active Brains" amongst "lower cognitive scoring" (n = 180) and "higher cognitive scoring" (n = 180) adults aged 60-85.

Results

We collected 67.2 and 76.1% of our 12-month primary outcome (Baddeley verbal reasoning task) data for the "lower cognitive score" and "higher cognitive score" groups, respectively. Usage of "Active Brains" indicated overall feasibility and satisfactory engagement with the physical activity intervention content (which did not require sustained online engagement), but engagement with online cognitive training was limited. Uptake of the additional brief telephone support appeared to be higher in the "lower cognitive score" trial. Preliminary descriptive trends in the primary outcome data might indicate a protective effect of Active Brains against cognitive decline, but further investigation in fully-powered trials is required to answer this definitively.

Discussion

Whilst initial uptake and engagement with the online intervention was modest, it was in line with typical usage of other digital behavior change interventions, and early indications from the descriptive analysis of the primary outcome and behavioral data suggest that further exploration of the potential protective benefits of Active Brains are warranted. The study also identified minor modifications to procedures, particularly to improve online primary-outcome completion. Further investigation of Active Brains will now seek to determine its efficacy in protecting cognitive performance amongst adults aged 60-85 with varied levels of existing cognitive performance.

Design of a mindfulness-based intervention to support teachers’ emotional regulation behaviors

Addressing teachers’ ability to better navigate stress and emotional responses has the potential to improve classroom behaviors, student learning as well as teachers’ wellbeing. While there is growing interest in mindfulness-based interventions (MBIs), there is need for greater attention to design aligned with relevant theories and transparent reporting to improve the rigor and translation of research. While utilized widely in health and behavior fields, behavior change theory has not been applied extensively in education or MBI research. This descriptive methodological paper articulates the intervention mapping and design process of an MBI to improve teachers’ stress and emotional regulation. To improve reporting, we use a post hoc application of a well-established behavioral model. Application of the Behavior Change Wheel (BCW) as part of an intervention mapping approach for an MBI to align teacher’s needs with intervention elements most likely to achieve desired behavioral outcomes. The paper illustrates an approach which includes defining the problem in behavioral terms, identification of target behaviors and behavioral change techniques, and selection of strategies and modes of intervention delivery. A systematic intervention mapping approach applying the BCW can yield a holistic MBI for the desired behavior change, enhanced emotion and stress regulation in teachers. This example of applying the BCW to the design of an MBI demonstrates how intervention designers and researchers can make decisions about which techniques and delivery components to include in their interventions, and how to systematically describe them. Such approaches have the potential to improve the reporting and methodological rigor of future MBI research and the successful application of interventions in practice.

Mediation Analysis of a Theory-Based Culture and Age-Appropriate HIV/STI Prevention

Few studies have investigated mediator effects of HIV prevention interventions on adolescents in sub-Saharan Africa. Herein, we report on a secondary analysis of an intervention that increased intentions to use condoms, abstain from sexual intercourse, and seek safe male circumcision among adolescents in Botswana. In a study conducted in Botswana, 806 grade 9 students from 21 public Junior Secondary Schools were randomly assigned to either the OWN THE FUTURE: Pulling Together We Will" ( PTWW) intervention group or a health promotion control group. Both conditions consisted of 12 1-h modules, with two modules delivered during each of the six sessions on six consecutive school days. The students in both groups completed confidential computer-based surveys at several time points: pre-, immediately post-, 3, 6, and 12 months post-intervention. Mediation was assessed using the product-of-coefficients approach in a generalized estimating equations (GEE) framework. The analyses showed that condom use beliefs were significant mediators of the intervention effect on the intention to consistently use condoms over time. Also, negative socio-cultural beliefs, prevention beliefs, and HIV/STI knowledge were significant mediators of the intervention's effects on the intention to abstain from sex. Additionally, normative beliefs, prevention beliefs, parental negotiation, and circumcision knowledge were significant mediators of the intervention's effect on intention to seek safe male circumcision. The mediation analysis delineated a theoretical model and isolated activities that positively impact condom use, abstinence from sex, and circumcision intentions of Batswana middle school adolescents.

Fidelity assessment of nurse-led non-pharmacological package of care for knee pain in the package development phase of a feasibility randomised controlled trial based in secondary care: a mixed methods study

OBJECTIVES

To evaluate fidelity of delivery of a nurse-led non-pharmacological complex intervention for knee pain.

SETTING

Secondary care. Single-centre study.

STUDY DESIGN

Mixed methods study.

PARTICIPANTS

Eighteen adults with chronic knee pain.

INCLUSION CRITERIA

Age >40 years, knee pain present for longer than 3 months, knee pain for most days of the previous month, at least moderate pain in two of the five domains of Western Ontario and McMaster Universities Osteoarthritis Index pain scale.

INTERVENTIONS

Nurse-led non-pharmacological intervention comprising assessment, education, exercise, use of hot/cold treatments, footwear modification, walking aids and weight-loss advice (if required).

OUTCOMES

Primary: fidelity of delivery of intervention, secondary: nurses' experience of delivering intervention.

METHODS

Each intervention session with every participant was video recorded and formed part of fidelity assessment. Fidelity checklists were completed by the research nurse after each session and by an independent researcher, after viewing the video-recordings blinded to nurse ratings. Fidelity scores (%), percentage agreement and 95% Confidence Intervals (CI) were calculated. Two semi-structured interviews were conducted with the research nurse.

RESULTS

Fourteen participants completed all visits. 62 treatment sessions took place. Nurse self-report and assessor video rating scores for all 62 treatment sessions were included in fidelity assessment. Overall fidelity was higher on nurse self-report (97.7%) than on objective video-rating (84.2%). Percentage agreement between nurse self-report and video-rating was 73.3% (95% CI 71.3 to 75.3). Fidelity was lowest for advice on footwear and walking aids. The nurse reported difficulty advising on thermal treatments, footwear and walking aids, and did not feel confident negotiating achievable and realistic goals with participants.

CONCLUSIONS

A trained research nurse can deliver most components of a non-pharmacological intervention for knee pain to a high degree of fidelity. Future research should assess intervention fidelity in a routine clinical setting, and examine its clinical and cost-effectiveness.

TRIAL REGISTRATION NUMBER

NCT03670706.

Fitness facility staff demonstrate high fidelity when implementing an evidence-based diabetes prevention program

Translating evidence-based diabetes prevention programs into the community is needed to make promising interventions accessible to individuals at-risk of type 2 diabetes. To increase the likelihood of successful translation, implementation evaluations should be conducted to understand program outcomes and provide feedback for future scale-up sites. The purpose of this research was to examine the delivery of, and engagement with, an evidence-based diet and exercise diabetes prevention program when delivered by fitness facility staff within a community organization. Ten staff from a community organization were trained to deliver the diabetes prevention program. Between August 2019-March 2020, 26 clients enrolled in the program and were assigned to one of the ten staff. Three fidelity components were accessed. First, staff completed session-specific fidelity checklists (n = 156). Second, two audio-recorded counseling sessions from all clients underwent an independent coder fidelity check (n = 49). Third, staff recorded client goals on session-specific fidelity checklists and all goals were independently assessed for (a) staff goal-setting fidelity, (b) client intervention receipt, and (c) client goal enactment by two coders (n = 285). Average self-reported fidelity was 90% for all six sessions. Independent coder scores for both counseling sessions were 83% and 81%. Overall staff helped clients create goals in line with program content and had a goal achievement of 78%. The program was implemented with high fidelity by staff at a community organization and clients engaged with the program. Findings increase confidence that program effects are due to the intervention itself and provide feedback to refine implementation strategies to support future scale-up efforts.