Process evaluations in neurological rehabilitation: a mixed-evidence systematic review and recommendations for future research

Process Evaluations of Diabetes Self-Management Programs: A Systematic Review of the Literature

OBJECTIVE

To conduct a systematic review of process evaluations (PEs) of diabetes self-management programs (DSMPs).

DATA SOURCE

An electronic search using Medline (Ovid), Embase (Ovid), CINAHL (Ensco), Academic Search (Ebsco), and APA PsycInfo (Ebsco).

STUDY INCLUSION AND EXCLUSION CRITERIA

Peer-reviewed, empirical quantitative, qualitative, or mixed-method studies were included if they (1) were a traditional, group-based DSMP, (2) involved adults at least 18 years with T1DM or T2DM, (3) were a stand-alone or embedded PE, and (4) published in English.

DATA EXTRACTION

The following process evaluation outcomes were extracted: fidelity, dose delivered, dose received, reach, recruitment, retention, and context. Additional items were extracted, (eg, process evaluation type, data collection methods; theories; frameworks or conceptual models used to guide the process evaluation, and etc).

DATA SYNTHESIS

Due to heterogeneity across studies, studies were synthesized qualitatively (narratively).

RESULTS

Sixty-eight studies (k) in 78 articles (n) (k = 68; n = 78) were included. Most were mixed methods of low quality. Studies were typically integrated into outcome evaluations vs being stand-alone, lacked theoretical approaches to guide them, and incorporated limited outcomes such as dose received, reach, and retention.

CONCLUSION

Future research should 1) implement stand-alone theoretically grounded PE studies and 2) provide a shared understanding of standardized guidelines to conduct PEs. This will allow public health practitioners and researchers to assess and compare the quality of different programs to be implemented.

OptiCogs: feasibility of a multicomponent intervention to rehabilitate people with cognitive impairment post-stroke

BACKGROUND

Stroke is a leading cause of death and disability worldwide. Despite the prevalence and associated burden of cognitive impairment post-stroke, there is uncertainty regarding optimal cognitive rehabilitation for people post-stroke. This study aimed to assess whether a multicomponent intervention, called OptiCogs, is feasible, acceptable, and safe for people with cognitive impairment post-stroke. A secondary aim was to explore changes in cognitive function, fatigue, quality of life, physical function, and occupational performance, from pre-intervention to post-intervention.

METHODS

A feasibility study was conducted where people post-stroke with cognitive impairment enrolled in a 6-week multicomponent intervention. The primary outcomes recorded included response rate, recruitment rate, retention rate, adherence to the intervention protocol, adverse events, and acceptability of the intervention to people post-stroke. Secondary outcomes included (i) change in cognitive functioning using the Addenbrooke's Cognitive Examination III, (ii) fatigue using the Fatigue Severity scale, (iii) quality of life using the Stroke Specific Quality of Life scale (iv) physical function using the patient-reported outcomes measurement information system, and (v) patient-reported occupational performance using the Canadian Occupational Performance Measure. The Consolidated Standards of Reporting Trials extension reporting guidelines were followed, for pilot and feasibility studies, to standardize the conduct and reporting of this study.

RESULTS

The response rate was 10.9%. Nine eligible participants were enrolled during the 4-month recruitment period, with eight participants completing the entire 6-week intervention, as well as the pre- and post-intervention outcome measures. There were no reported adverse events. Participants were satisfied with the intervention and found it acceptable overall. Results of the secondary outcomes were promising for cognitive function (ACE III, pre: 63.3 ± 23.9 to post: 69 ± 24.6), fatigue (FSS, pre: 52.5 ± 7.3 to post: 45.6 ± 7.2), quality of life (SSQoL, pre: 131.0 ± 26.3 to post: 169.9 ± 15.3), physical function (PROMIS-PF, pre: 15.5 ± 6.3 to post: 15.8 ± 5.3), and occupational performance (COPM performance, pre: 9.3 ± 2.3 to post: 22.9 ± 4.2) and COPM satisfaction, pre: 9.9 ± 2.1 to post: 22.7 ± 3.5).

CONCLUSION

Preliminary results suggest low-modest recruitment and high retention rates for the OptiCogs intervention. Changes in cognitive function, fatigue, quality of life, and self-reported occupational performance show improvement from pre- to post-intervention. These potential benefits require further testing in a larger pilot trial.

TRIAL REGISTRATION

NCT05414539.

Preliminary efficacy and predictors of response to a remotely-delivered symptom self-management program for persistent symptoms after concussion

BACKGROUND

More than a quarter of adults with concussion endure prolonged symptoms of >3 months. We developed the Concussion Education Self-Management program to help people manage persisting symptoms. Here, we assess feasibility, preliminary efficacy, and correlates of response.

METHODS

N = 80 adults participated in the program; ages ranged from 18 to 65 years and time post-injury ranged from 6 months to 18 years. Weekly sessions, delivered remotely and in groups, comprised education and strategies for management of cognitive, emotional, and physical symptoms. Primary outcome: Confidence to self-manage symptoms. Secondary outcomes: Quality of life; mood/anxiety/stress. Predictors of response: Self-reported cognitive, emotional and physical symptoms at intake.

RESULTS

Pre- to post-program improvements were observed in confidence to self-manage, p < 0.03; quality of life, p < 0.001; depression, p < 0.001; anxiety, p < 0.001; and stress, p < 0.001. Considering confidence to self-manage, those with fewer cognitive and physical symptoms benefitted more (p's < 0.0005 and p < 0.01, respectively).

DISCUSSION

This program shows promise for improving self-management of prolonged symptoms. Those with high symptom burden may need extra sessions to benefit. This is a cost-effective and scalable program that can reach people regardless of geographic location or impediments to travel.

Early and Intensive Motor Training for people with spinal cord injuries (the SCI-MT Trial): protocol of the process evaluation

INTRODUCTION

People with spinal cord injury receive physical rehabilitation to promote neurological recovery. Physical rehabilitation commences as soon as possible when a person is medically stable. One key component of physical rehabilitation is motor training. There is initial evidence to suggest that motor training can enhance neurological recovery if it is provided soon after injury and in a high dosage. The Early and Intensive Motor Training Trial is a pragmatic randomised controlled trial to determine whether 10 weeks of intensive motor training enhances neurological recovery for people with spinal cord injury. This pragmatic randomised controlled trial will recruit 220 participants from 15 spinal injury units in Australia, Scotland, Italy, Norway, England, Belgium and the Netherlands. This protocol paper describes the process evaluation that will run alongside the Early and Intensive Motor Training Trial. This process evaluation will help to explain the trial results and explore the potential facilitators and barriers to the possible future rollout of the trial intervention.

METHODS AND ANALYSIS

The UK Medical Research Council process evaluation framework and the Implementation Research Logic Model will be used to explain the trial outcomes and inform future implementation. Key components of the context, implementation and mechanism of impact, as well as the essential elements of the intervention and outcomes, will be identified and analysed. Qualitative and quantitative data will be collected and triangulated with the results of the Early and Intensive Motor Training Trial to strengthen the findings of this process evaluation.

ETHICS AND DISSEMINATION

Ethical approval for the Early and Intensive Motor Training Trial and process evaluation has been obtained from the Human Research Ethics Committee at the Northern Sydney Local Health District (New South Wales) in Australia (project identifier: 2020/ETH02540). All participants are required to provide written consent after being informed about the trial and the process evaluation. The results of this process evaluation will be published in peer-reviewed journals.

TRIAL REGISTRATION NUMBER

Australian New Zealand Clinical Trial Registry (ACTRN12621000091808); Universal Trial Number (U1111-1264-1689).

Implementation of a Treatment Program for Individuals Imprisoned for Sex Offenses in Uruguay: Achievements, Problems and Challenges

Treatment for individuals convicted of sex offenses has substantially improved in developed countries in recent decades, providing practitioners with an extensive literature to guide the implementation of effective programs to reduce sexual reoffending. Nevertheless, sexual offending rehabilitation is still in its infancy in Latin American countries such as Uruguay, so little is known about the transference and implementation of evidence-based programs. The current study examines the strengths, barriers, and challenges of implementing a sex offenses treatment program in Uruguay. The findings suggest some achievements of the program, but also several problems with implementation. Some problems are universal among different countries (e.g., scarce resources and facilities, insufficiently trained staff, and unexpected changes in the organization), but others were particularly relevant in the Uruguayan context (e.g., government policy alien to a rehabilitation approach, lack of appropriate prison facilities, lack of training for therapists from a cognitive-behavioral perspective). All these difficulties must be anticipated and solved for successful generalizability of rehabilitation programs to different correctional systems.

What do we want to get out of this? a critical interpretive synthesis of the value of process evaluations, with a practical planning framework

BACKGROUND

Process evaluations aim to understand how complex interventions bring about outcomes by examining intervention mechanisms, implementation, and context. While much attention has been paid to the methodology of process evaluations in health research, the value of process evaluations has received less critical attention. We aimed to unpack how value is conceptualised in process evaluations by identifying and critically analysing 1) how process evaluations may create value and 2) what kind of value they may create.

METHODS

We systematically searched for and identified published literature on process evaluation, including guidance, opinion pieces, primary research, reviews, and discussion of methodological and practical issues. We conducted a critical interpretive synthesis and developed a practical planning framework.

RESULTS

We identified and included 147 literature items. From these we determined three ways in which process evaluations may create value or negative consequences: 1) through the socio-technical processes of 'doing' the process evaluation, 2) through the features/qualities of process evaluation knowledge, and 3) through using process evaluation knowledge. We identified 15 value themes. We also found that value varies according to the characteristics of individual process evaluations, and is subjective and context dependent.

CONCLUSION

The concept of value in process evaluations is complex and multi-faceted. Stakeholders in different contexts may have very different expectations of process evaluations and the value that can and should be obtained from them. We propose a planning framework to support an open and transparent process to plan and create value from process evaluations and negotiate trade-offs. This will support the development of joint solutions and, ultimately, generate more value from process evaluations to all.

Process evaluations undertaken alongside randomised controlled trials in the hospital setting: A scoping review

Background

There is increasing recognition of the importance of undertaking process evaluations alongside implementation of health interventions by examining mechanisms of impact and contextual factors. However, a comprehensive synthesis of process evaluations undertaken alongside clinical trials in hospital settings is lacking. We undertook a scoping review to address this gap.

Methods

This review was guided by the methodological framework for scoping studies. Studies were identified using four databases; Ovid Medline, EBSCO CINAHL, EMBASE and Scopus. Two authors independently screened all titles and available abstracts, with a third author available to adjudicate. Studies were eligible for inclusion if they described a process evaluation undertaken alongside a randomised controlled trial in the hospital setting. Data were abstracted by one author and checked by two others and analysed both descriptively and using inductive content analysis.

Results

Data were extracted from 30 articles reporting on 15 trials, most of which were cluster randomised trials (c-RTs) (n = 12). The most common data collection methods used in process evaluations were interviews, questionnaires or surveys, and records or logs. Data analysis revealed three themes relative to how authors: use process data to interpret, understand and explain trial outcomes; evaluate responses to the intervention; and consider the implementation context.

Conclusions

Findings from this review demonstrate the complex nature of intervention implementation in the hospital setting. Overall, there is need for standardised reporting of process evaluations and more explicit descriptions of how authors use frameworks to guide their evaluation.

Implementation of highly challenging balance training for Parkinson's disease in clinical practice: a process evaluation

Background

Process evaluations provide contextual insight into the way in which interventions are delivered. This information is essential when designing strategies to implement programs into wider clinical practice. We performed a process evaluation of the HiBalance effectiveness trial investigating the effects of a 10-week of highly challenging and progressive balance training for mild-moderate Parkinson’s disease (PD). Study aims were to investigate i) the quality and quantity of intervention delivery and ii) barriers and facilitators for implementation.

Methods

Process outcomes included; Fidelity; Dose (delivered and received) Recruitment and Reach. Investigation of barriers and facilitators was guided by the Consolidated Framework for Implementation Research. Program delivery was assessed across four neurological rehabilitation sites during a two-year period. Data collection was mixed-methods in nature and quantitative and qualitative data were merged during the analysis phase.

Results

Thirteen program trainers delivered the intervention to 12 separate groups during 119 training sessions. Trainer fidelity to program core components was very high in 104 (87%) of the sessions. Participant responsiveness to the core components was generally high, although adherence to the home exercise program was low (50%). No significant context-specific differences were observed across sites in terms of fidelity, dose delivered/ received or participant characteristics, despite varying recruitment methods. Facilitators to program delivery were; PD-specificity, high training frequency and professional autonomy. Perceived barriers included; cognitive impairment, absent reactional balance among participants, as well a heterogeneous group in relation to balance capacity.

Conclusion

These findings provide corroborating evidence for outcome evaluation results and valuable information for the further adaptation and implementation of this program. Important lessons can also be learned for researchers and clinicians planning to implement challenging exercise training programs for people with mild-moderate PD.

Trial registration

ClinicalTrials.gov, NCT02727478, registered 30 march, 2016 − Retrospectively registered.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12877-021-02031-1.

Process evaluation within pragmatic randomised controlled trials: what is it, why is it done, and can we find it?-a systematic review

BACKGROUND

Process evaluations are increasingly conducted within pragmatic randomised controlled trials (RCTs) of health services interventions and provide vital information to enhance understanding of RCT findings. However, issues pertaining to process evaluation in this specific context have been little discussed. We aimed to describe the frequency, characteristics, labelling, value, practical conduct issues, and accessibility of published process evaluations within pragmatic RCTs in health services research.

METHODS

We used a 2-phase systematic search process to (1) identify an index sample of journal articles reporting primary outcome results of pragmatic RCTs published in 2015 and then (2) identify all associated publications. We used an operational definition of process evaluation based on the Medical Research Council's process evaluation framework to identify both process evaluations reported separately and process data reported in the trial results papers. We extracted and analysed quantitative and qualitative data to answer review objectives.

RESULTS

From an index sample of 31 pragmatic RCTs, we identified 17 separate process evaluation studies. These had varied characteristics and only three were labelled 'process evaluation'. Each of the 31 trial results papers also reported process data, with a median of five different process evaluation components per trial. Reported barriers and facilitators related to real-world collection of process data, recruitment of participants to process evaluations, and health services research regulations. We synthesised a wide range of reported benefits of process evaluations to interventions, trials, and wider knowledge. Visibility was often poor, with 13/17 process evaluations not mentioned in the trial results paper and 12/16 process evaluation journal articles not appearing in the trial registry.

CONCLUSIONS

In our sample of reviewed pragmatic RCTs, the meaning of the label 'process evaluation' appears uncertain, and the scope and significance of the term warrant further research and clarification. Although there were many ways in which the process evaluations added value, they often had poor visibility. Our findings suggest approaches that could enhance the planning and utility of process evaluations in the context of pragmatic RCTs.

TRIAL REGISTRATION

Not applicable for PROSPERO registration.

Development of the Better Balance Program for People with Multiple Sclerosis: A Complex Fall-Prevention Intervention

Background

Approximately 56% of people with multiple sclerosis (MS) will fall in any 3-month period, with the potential for physical, psychological, and social consequences. Fall-prevention research for people with MS is in its infancy, with a timely need to develop theory-based interventions that reflect the complexity of falls. The clear articulation of the development of any complex intervention is paramount to its future evaluation, usability, and effectiveness. Our aim was to describe how the development of Better Balance, a complex multicomponent fall-prevention intervention for people with MS, was guided by the Medical Research Council framework for the development of complex interventions.

Methods

Sources of information included existing literature, original research, clinician interviews, and views of people with MS. These sources were synthesized and refined through an iterative process of intervention development involving researchers, clinicians, and people with MS.

Results

The resulting intervention is outlined through a variety of key tasks supplementing the original Medical Research Council framework. Use of this framework resulted in a theoretically based and user-informed complex intervention designed to address the physiological, personal, and behavioral risk factors associated with falls in people with MS.

Conclusions

The articulation of the systematic process used to develop Better Balance will inform the future evaluation and usability of the intervention.

Process evaluations of primary care interventions addressing chronic disease: a systematic review

OBJECTIVE

Process evaluations (PEs) alongside randomised controlled trials of complex interventions are valuable because they address questions of for whom, how and why interventions had an impact. We synthesised the methods used in PEs of primary care interventions, and their main findings on implementation barriers and facilitators.

DESIGN

Systematic review using the UK Medical Research Council guidance for PE as a guide.

DATA SOURCES

Academic databases (MEDLINE, SCOPUS, PsycINFO, Cumulative Index to Nursing and Allied Health Literature, EMBASE and Global Health) were searched from 1998 until June 2018.

ELIGIBILITY CRITERIA

We included PE alongside randomised controlled trials of primary care interventions which aimed to improve outcomes for patients with non-communicable diseases.

DATA EXTRACTION AND SYNTHESIS

Two independent reviewers screened and conducted the data extraction and synthesis, with a third reviewer checking a sample for quality assurance.

RESULTS

69 studies were included. There was an overall lack of consistency in how PEs were conducted and reported. The main weakness is that only 30 studies were underpinned by a clear intervention theory often facilitated by the use of existing theoretical frameworks. The main strengths were robust sampling strategies, and the triangulation of qualitative and quantitative data to understand an intervention's mechanisms. Findings were synthesised into three key themes: (1) a fundamental mismatch between what the intervention was designed to achieve and local needs; (2) the required roles and responsibilities of key actors were often not clearly understood; and (3) the health system context-factors such as governance, financing structures and workforce-if unanticipated could adversely impact implementation.

CONCLUSION

Greater consistency is needed in the reporting and the methods of PEs, in particular greater use of theoretical frameworks to inform intervention theory. More emphasis on formative research in designing interventions is needed to align the intervention with the needs of local stakeholders, and to minimise unanticipated consequences due to context-specific barriers.

PROSPERO REGISTRATION NUMBER

CRD42016035572.

Process evaluation of complex interventions tested in randomised controlled trials in musculoskeletal disorders: a systematic review protocol

INTRODUCTION

The effectiveness of complex interventions for the management of musculoskeletal disorders has been estimated in many randomised clinical trials (RCTs). These trials inform which interventions are the most effective, however they do not always inform how an intervention achieved its clinical outcomes, nor how and what elements of an intervention were delivered to patients. Such information is useful for translating findings into clinical practice. A few process evaluation studies have been conducted alongside RCTs and a variety of methods have been used. To gain a better understanding of current practices of process evaluation in RCTs in musculoskeletal disorders, this systematic review is designed to answer the following research question: How are process evaluation of complex interventions tested in RCTs in musculoskeletal disorders being conducted?

METHODS AND ANALYSIS

We will systematically search seven electronic databases (MEDLINE, SCOPUS, CINAHL, PsycINFO, EMBASE, Web of Science and Cochrane database) from the date of inception to August 2018 for studies on process evaluation of RCTs on non-surgical and non-pharmacological management of musculoskeletal disorders. We will include qualitative and quantitative studies conducted alongside RCTs, reported with the RCTs or separate studies that assessed interventions for musculoskeletal disorders. Two reviewers will screen abstracts and apply prespecified inclusion criteria to identify relevant studies, extract the data and assess the risk of bias within included studies. We will follow recommendations from the 'Cochrane Qualitative and Implementation Methods Group Guidance Series' when assessing methodological strengths and limitations of included studies. We will use a narrative synthesis to describe findings.

ETHICS AND DISSEMINATION

Ethical approval is not required as this review will not collect original data. Findings from this systematic review will be presented at a scientific conference and published in a peer reviewed journal.

PROSPERO REGISTRATION NUMBER

CRD42018109600.

The generation of consensus guidelines for carrying out process evaluations in rehabilitation research

BACKGROUND

Although in recent years there has been a strong increase in published research on theories (e.g. realist evaluation, normalization process theory) driving and guiding process evaluations of complex interventions, there is limited guidance to help rehabilitation researchers design and carry out process evaluations. This can lead to the risk of process evaluations being unsystematic. This paper reports on the development of new consensus guidelines that address the specific challenges of conducting process evaluations alongside clinical trials of rehabilitation interventions.

METHODS

A formal consensus process was carried out based on a modified nominal group technique, which comprised two phases. Phase I was informed by the findings of a systematic review, and included a nominal group meeting with an expert panel of participants to rate and discuss the proposed statements. Phase II was an in depth semi-structured telephone interviews with expert panel participants in order to further discuss the structure and contents of the revised guidelines. Frequency of rating responses to each statement was calculated and thematic analysis was carried out on all qualitative data.

RESULTS

The guidelines for carrying out process evaluations within complex intervention rehabilitation research were produced by combining findings from Phase I and Phase II. The consensus guidelines include recommendations that are grouped in seven sections. These sections are theoretical work, design and methods, context, recruitment and retention, intervention staff, delivery of the intervention and results. These sections represent different aspects or stages of the evaluation process.

CONCLUSION

The consensus guidelines here presented can play a role at assisting rehabilitation researchers at the time of designing and conducting process evaluations alongside trials of complex interventions. The guidelines break new ground in terms of concepts and theory and works towards a consensus in regards to how rehabilitation researchers should go about carrying out process evaluations and how this evaluation should be linked into the proposed trials. These guidelines may be used, adapted and tested by rehabilitation researchers depending on the research stage or study design (e.g. feasibility trial, pilot trial, etc.).

An EEG Tool for Monitoring Patient Engagement during Stroke Rehabilitation: A Feasibility Study

Objective

Patient engagement is of major significance in neural rehabilitation. We developed a real-time EEG marker for attention, the Brain Engagement Index (BEI). In this work we investigate the relation between the BEI and temporary functional change during a rehabilitation session.

Methods

First part: 13 unimpaired controls underwent BEI monitoring during motor exercise of varying levels of difficulty. Second part: 18 subacute stroke patients underwent standard motor rehabilitation with and without use of real-time BEI feedback regarding their level of engagement. Single-session temporary functional changes were evaluated based on videos taken before and after training on a given task. Two assessors, blinded to feedback use, assessed the change following single-session treatments.

Results

First part: a relation between difficulty of exercise and BEI was identified. Second part: temporary functional change was associated with BEI level regardless of the use of feedback.

Conclusions

This study provides preliminary evidence that when BEI is higher, the temporary functional change induced by the treatment session is better. Further work is required to expand this preliminary study and to evaluate whether such temporary functional change can be harnessed to improve clinical outcome.

Clinical Trial Registration

Registered with clinicaltrials.gov, unique identifier: NCT02603718 (retrospectively registered 10/14/2015).