Using Normalization Process Theory in feasibility studies and process evaluations of complex healthcare interventions: a systematic review

BACKGROUND

Normalization Process Theory (NPT) identifies, characterises and explains key mechanisms that promote and inhibit the implementation, embedding and integration of new health techniques, technologies and other complex interventions. A large body of literature that employs NPT to inform feasibility studies and process evaluations of complex healthcare interventions has now emerged. The aims of this review were to review this literature; to identify and characterise the uses and limits of NPT in research on the implementation and integration of healthcare interventions; and to explore NPT's contribution to understanding the dynamics of these processes.

METHODS

A qualitative systematic review was conducted. We searched Web of Science, Scopus and Google Scholar for articles with empirical data in peer-reviewed journals that cited either key papers presenting and developing NPT, or the NPT Online Toolkit ( www.normalizationprocess.org ). We included in the review only articles that used NPT as the primary approach to collection, analysis or reporting of data in studies of the implementation of healthcare techniques, technologies or other interventions. A structured data extraction instrument was used, and data were analysed qualitatively.

RESULTS

Searches revealed 3322 citations. We show that after eliminating 2337 duplicates and broken or junk URLs, 985 were screened as titles and abstracts. Of these, 101 were excluded because they did not fit the inclusion criteria for the review. This left 884 articles for full-text screening. Of these, 754 did not fit the inclusion criteria for the review. This left 130 papers presenting results from 108 identifiable studies to be included in the review. NPT appears to provide researchers and practitioners with a conceptual vocabulary for rigorous studies of implementation processes. It identifies, characterises and explains empirically identifiable mechanisms that motivate and shape implementation processes. Taken together, these mean that analyses using NPT can effectively assist in the explanation of the success or failure of specific implementation projects. Ten percent of papers included critiques of some aspect of NPT, with those that did mainly focusing on its terminology. However, two studies critiqued NPT emphasis on agency, and one study critiqued NPT for its normative focus.

CONCLUSIONS

This review demonstrates that researchers found NPT useful and applied it across a wide range of interventions. It has been effectively used to aid intervention development and implementation planning as well as evaluating and understanding implementation processes themselves. In particular, NPT appears to have offered a valuable set of conceptual tools to aid understanding of implementation as a dynamic process.

Methods used to address fidelity of receipt in health intervention research: a citation analysis and systematic review

BACKGROUND

The American Behaviour Change Consortium (BCC) framework acknowledges patients as active participants and supports the need to investigate the fidelity with which they receive interventions, i.e. receipt. According to this framework, addressing receipt consists in using strategies to assess or enhance participants' understanding and/or performance of intervention skills. This systematic review aims to establish the frequency with which receipt is addressed as defined in the BCC framework in health research, and to describe the methods used in papers informed by the BCC framework and in the wider literature.

METHODS

A forward citation search on papers presenting the BCC framework was performed to determine the frequency with which receipt as defined in this framework was addressed. A second electronic database search, including search terms pertaining to fidelity, receipt, health and process evaluations was performed to identify papers reporting on receipt in the wider literature and irrespective of the framework used. These results were combined with forward citation search results to review methods to assess receipt. Eligibility criteria and data extraction forms were developed and applied to papers. Results are described in a narrative synthesis.

RESULTS

19.6% of 33 studies identified from the forward citation search to report on fidelity were found to address receipt. In 60.6% of these, receipt was assessed in relation to understanding and in 42.4% in relation to performance of skill. Strategies to enhance these were present in 12.1% and 21.1% of studies, respectively. Fifty-five studies were included in the review of the wider literature. Several frameworks and operationalisations of receipt were reported, but the latter were not always consistent with the guiding framework. Receipt was most frequently operationalised in relation to intervention content (16.4%), satisfaction (14.5%), engagement (14.5%), and attendance (14.5%). The majority of studies (90.0%) included subjective assessments of receipt. These relied on quantitative (76.0%) rather than qualitative (42.0%) methods and studies collected data on intervention recipients (50.0%), intervention deliverers (28.0%), or both (22.0%). Few studies (26.0%) reported on the reliability or validity of methods used.

CONCLUSIONS

Receipt is infrequently addressed in health research and improvements to methods of assessment and reporting are required.

Feasibility and preliminary efficacy of the 'HEYMAN' healthy lifestyle program for young men: a pilot randomised controlled trial

Background

In young men, unhealthy lifestyle behaviours can be detrimental to their physical and/or mental health and set them on a negative health trajectory into adulthood. Despite this, there is a lack of evidence to guide development of effective health behaviour change interventions for young men. This study assessed the feasibility and preliminary efficacy of the ‘HEYMAN’ (Harnessing Ehealth to enhance Young men’s Mental health, Activity and Nutrition) healthy lifestyle program for young men.

Methods

A pilot RCT with 50 young men aged 18–25 years randomised to the HEYMAN intervention (n = 26) or waitlist control (n = 24). HEYMAN was a 3-month intervention, targeted for young men to improve eating habits, activity levels and well-being. Intervention development was informed by a participatory research model (PRECEDE-PROCEED). Intervention components included eHealth support (website, wearable device, Facebook support group), face-to-face sessions (group and individual), a personalised food and nutrient report, home-based resistance training equipment and a portion control tool. Outcomes included: feasibility of research procedures (recruitment, randomisation, data collection and retention) and of intervention components. Generalized linear mixed models estimated the treatment effect at 3-months for the primary outcomes: pedometer steps/day, diet quality, well-being and several secondary outcomes.

Results

A 7-week recruitment period was required to enrol 50 young men. A retention rate of 94% was achieved at 3-months post-intervention. Retained intervention participants (n = 24) demonstrated reasonable usage levels for most program components and also reported reasonable levels of program component acceptability for attractiveness, comprehension, usability, support, satisfaction and ability to persuade, with scores ranging from 3.0 to 4.6 (maximum 5). No significant intervention effects were observed for the primary outcomes of steps/day (1012.7, 95% CI = −506.2, 2531.6, p = 0.191, d = 0.36), diet quality score (3.6, 95% CI = −0.4, 7.6, p = 0.081, d = 0.48) or total well-being score (0.4, 95% CI = −1.6, 2.5, p = 0.683, d = 0.11). Significant intervention effects were found for daily vegetable servings, energy-dense, nutrient-poor foods, MVPA, weight, BMI, fat mass, waist circumference and cholesterol (all p < 0.05).

Conclusions

The HEYMAN program demonstrated feasibility in assisting young men to make some positive lifestyle changes. This provides support for the conduct of a larger, fully-powered RCT, but with minor amendments to research procedures and intervention components required.

Trial registration

Australian New Zealand Clinical Trials Registry ACTRN12616000350426.

Electronic supplementary material

The online version of this article (doi:10.1186/s12937-017-0227-8) contains supplementary material, which is available to authorized users.

A nurse-led interdisciplinary primary care approach to prevent disability among community-dwelling frail older people: a large-scale process evaluation

BACKGROUND

The complex healthcare needs of frail older people and their increased risk of disability require an integrated and proactive approach. In the Netherlands, an interdisciplinary primary care approach has recently been developed, involving individualized assessment and interventions (tailor-made care), case management and long-term follow-up. The practice nurse as part of a general practice is case manager and plans, organizes and monitors the care process and facilitates cooperation between professionals. The approach has shown positive indications regarding its feasibility in a small pilot, but its implementation on a large scale had not hitherto been investigated.

OBJECTIVES

To examine the extent to which the interdisciplinary care approach is implemented as planned and to gain insight into healthcare professionals' and frail older people's experiences regarding the benefits, burden, stimulating factors and barriers.

DESIGN

A process evaluation was conducted using a mixed methods design.

SETTINGS

Six GP practices in the south of the Netherlands.

PARTICIPANTS

Practice nurses (n=7), GPs (n=12), occupational therapists (n=6) and physical therapists (n=20) participated in the process evaluation. Furthermore, 194 community-dwelling frail older people (≥ 70 years) were included using the Groningen Frailty Indicator. People who were terminally ill, were confined to bed, had severe cognitive or psychological impairments or were unable to communicate in Dutch were excluded.

METHODS

Quantitative data (logbooks and evaluation forms) were collected from all the participating frail older people and 13 semi-structured interviews with a selection of them were conducted. In addition, data from healthcare professionals were collected through 12 semi-structured interviews and four focus group discussions.

RESULTS

Although some parts of the protocol were insufficiently executed, healthcare professionals and frail older people were satisfied with the care approach, as it provided a useful structure for the delivery of geriatric primary care and increased the attention to preventive treatment. Frail older people felt acknowledged by healthcare professionals and experienced support in handling their problems and fulfilling their wishes.

CONCLUSIONS

The findings of the study revealed several positive aspects of the interdisciplinary primary care approach. Given its complexity, the implementation of the nurse-led interdisciplinary care approach is challenging and some parts of the protocol need special attention.

Peer engagement in harm reduction strategies and services: a critical case study and evaluation framework from British Columbia, Canada

Background

Engaging people with drug use experience, or ‘peers,’ in decision-making helps to ensure harm reduction services reflect current need. There is little published on the implementation, evaluation, and effectiveness of meaningful peer engagement. This paper aims to describe and evaluate peer engagement in British Columbia from 2010–2014.

Methods

A process evaluation framework specific to peer engagement was developed and used to assess progress made, lessons learned, and future opportunities under four domains: supportive environment, equitable participation, capacity building and empowerment, and improved programming and policy. The evaluation was conducted by reviewing primary and secondary qualitative data including focus groups, formal documents, and meeting minutes.

Results

Peer engagement was an iterative process that increased and improved over time as a consequence of reflexive learning. Practical ways to develop trust, redress power imbalances, and improve relationships were crosscutting themes. Lack of support, coordination, and building on existing capacity were factors that could undermine peer engagement. Peers involved across the province reviewed and provided feedback on these results.

Conclusion

Recommendations from this evaluation can be applied to other peer engagement initiatives in decision-making settings to improve relationships between peers and professionals and to ensure programs and policies are relevant and equitable.

Understanding community-based participatory research through a social movement framework: a case study of the Kahnawake Schools Diabetes Prevention Project

BACKGROUND

A longstanding challenge of community-based participatory research (CBPR) has been to anchor evaluation and practice in a relevant theoretical framework of community change, which articulates specific and concrete evaluative benchmarks. Social movement theories provide a broad range of theoretical tools to understand and facilitate social change processes, such as those involved in CBPR. Social movement theories have the potential to provide a coherent representation of how mobilization and collective action is gradually developed and leads to systemic change in the context of CBPR. The current study builds on a social movement perspective to assess the processes and intermediate outcomes of a longstanding health promotion CBPR project with an Indigenous community, the Kahnawake Schools Diabetes Prevention Project (KDSPP).

METHODS

This research uses a case study design layered on a movement-building evaluation framework, which allows progress to be tracked over time. Data collection strategies included document (scientific and organizational) review (n = 51) and talking circles with four important community stakeholder groups (n = 24).

RESULTS

Findings provide an innovative and chronological perspective of the evolution of KSDPP as seen through a social movement lens, and identify intermediate outcomes associated with different dimensions of movement building achieved by the project over time (mobilization, leadership, vision and frames, alliance and partnerships, as well as advocacy and action strategies). It also points to areas of improvement for KSDPP in building its potential for action.

CONCLUSION

While this study's results are directly relevant and applicable to the local context of KSDPP, they also highlight useful lessons and conclusions for the planning and evaluation of other long-standing and sustainable CBPR initiatives. The conceptual framework provides meaningful benchmarks to track evidence of progress in the context of CBPR. Findings from the study offer new ways of thinking about the evaluation of CBPR projects and their progress by drawing on frameworks that guide other forms of collective action.

Implementation of internet-delivered cognitive behavior therapy within community mental health clinics: a process evaluation using the consolidated framework for implementation research

BACKGROUND

Depression and anxiety are prevalent and under treated conditions that create enormous burden for the patient and the health system. Internet-delivered cognitive behavior therapy (ICBT) improves patient access to treatment by providing therapeutic information via the Internet, presented in sequential lessons, accompanied by brief weekly therapist support. While there is growing research supporting ICBT, use of ICBT within community mental health clinics is limited. In a recent trial, an external unit specializing in ICBT facilitated use of ICBT in community mental health clinics in one Canadian province (ISRCTN42729166; registered November 5, 2013). Patient outcomes were very promising and uptake was encouraging. This paper reports on a parallel process evaluation designed to understand facilitators and barriers impacting the uptake and implementation of ICBT.

METHODS

Therapists (n = 22) and managers (n = 11) from seven community mental health clinics dispersed across one Canadian province who were involved in implementing ICBT over ~2 years completed an online survey (including open and closed-ended questions) about ICBT experiences. The questions were based on the Consolidated Framework for Implementation Research (CFIR), which outlines diverse constructs that have the potential to impact program implementation.

RESULTS

Analyses suggested ICBT implementation was perceived to be most prominently facilitated by intervention characteristics (namely the relative advantages of ICBT compared to face-to-face therapy, the quality of the ICBT program that was delivered, and evidence supporting ICBT) and implementation processes (namely the use of an external facilitation unit that aided with engaging patients, therapists, and managers and ICBT implementation). The inner setting was identified as the most significant barrier to implementation as a result of limited resources for ICBT combined with greater priority given to face-to-face care.

CONCLUSIONS

The results contribute to understanding facilitators and barriers to using ICBT within community mental health clinics and serve to identify recommendations for improving uptake and implementation of ICBT in clinic settings.

A salutogenic program to enhance sense of coherence and quality of life for older people in the community: A feasibility randomized controlled trial and process evaluation

OBJECTIVE

To determine the feasibility of a salutogenesis-based self-care program on quality of life, sense of coherence, activation and resilience among older community dwellers.

METHODS

This is a feasibility randomized controlled trial. Sixty-four older community-dwellers were recruited from a Singapore senior activity center and randomly assigned to intervention and control groups. The intervention group attended a 12-week Resource Enhancement and Activation Program. The outcomes were assessed with the Chinese versions of World Health Organization Quality of Life Scale, Sense of Coherence, Patient Activation Measure, and Connor-Davidson Resilience Scale. Process evaluation was conducted using focus groups with the intervention group.

RESULTS

At the end of the program, the intervention group showed significant improvement in the Sense of Coherence scale and the psychological subscale of the WHO Quality of Life scale compared with the control group. Three themes emerged from the process evaluation: participation in the program, reflection on the experience, and improving the experience.

CONCLUSION

A salutogenic self-care approach could be a potential health promotion strategy for older people.

PRACTICE IMPLICATIONS

With improved sense of coherence and psychological aspect of quality of life, older people's self-care ability may improve, leading to better health and better quality of life.

Development and validation of a generic nontarget method based on liquid chromatography - high resolution mass spectrometry analysis for the evaluation of different wastewater treatment options

A comprehensive workflow for using nontarget approaches as process evaluation tools was implemented, including data acquisition based on a LC-HRMS (QTOF) system using direct injection and data post-processing for the peak recognition in "full scan" data. Both parts of the approach were not only developed and validated in a conventional way using the suspected analysis of a set of spiked known micropollutants but also the nontarget analysis of a wastewater treatment plant (WWTP) effluent itself was utilized to consider a more environmental relevant range of analytes. Hereby, special focus was laid on the minimization of false positive results (FPs) during the peak recognition. The optimized data post-processing procedure reduced the percentage of FPs from 42% to 10-15%. Furthermore, the choice of a suitable chromatography for biological treated wastewater systems was also discussed during the method development. The workflow paid also attention to differences in the performance levels of the LC-HRMS system by implementation of an adaption system for intensity variations comparing different measurements dates or different instruments. The application of this workflow on wastewater samples from a municipal WWTP revealed that more than 91% compounds were eliminated by the biological treatment step and that the received effluent contained 55% newly formed potential transformation products.

A cluster randomised controlled trial of the Wellbeing in Secondary Education (WISE) Project - an intervention to improve the mental health support and training available to secondary school teachers: protocol for an integrated process evaluation

BACKGROUND

Secondary school teachers have low levels of wellbeing and high levels of depression compared with the general population. Teachers are in a key position to support students, but poor mental health may be a barrier to doing so effectively. The Wellbeing in Secondary Education (WISE) project is a cluster randomised controlled trial (RCT) of an intervention to improve the mental health support and training available to secondary school teachers through delivery of the training package Mental Health First Aid and a staff peer support service. We will conduct a process evaluation as part of the WISE trial to support the interpretation of trial outcomes and refine intervention theory. The domains assessed will be: the extent to which the hypothesised mechanisms of change are activated; system level influences on these mechanisms; programme differentiation and usual practice; intervention implementation, including any adaptations; intervention acceptability; and intervention sustainability.

METHODS

Research questions will be addressed via quantitative and qualitative methods. All study schools (n = 25) will provide process evaluation data, with more detailed focus group, interview and observation data being collected from a subsample of case study schools (4 intervention and 4 control). Mechanisms of change, as outlined in a logic model, will be measured via teacher and student surveys and focus groups. School context will be explored via audits of school practice that relate to mental health and wellbeing, combined with stakeholder interviews and focus groups. Implementation of the training and peer support service will be assessed via training observations, training participant evaluation forms, focus groups with participants, interviews with trainers and peer support service users, and peer supporter logs recording help provided. Acceptability and sustainability will be examined via interviews with funders, head teachers, trainers and peer support services users, and focus groups with training participants.

DISCUSSION

The process evaluation embedded within the WISE cluster RCT will illuminate how and why the intervention was effective, ineffective or conferred iatrogenic effects. It will contribute to the refinement of the theory underpinning the intervention, and will help to inform any future implementation.

TRIAL REGISTRATION

International Standard Randomised Controlled Trial Number: ISRCTN95909211 registered on 24 March 2016.

Peer-support to increase uptake of screening for diabetic retinopathy: process evaluation of the DURE cluster randomized trial

BACKGROUND

There is limited evidence on how implementation of peer support interventions influences effectiveness, particularly for individuals with diabetes. We conducted a cluster randomized controlled trial to compare the effectiveness of a peer-led health education package versus usual care to increase uptake of screening for diabetic retinopathy (DR).

METHODS

Our process evaluation used a mixed-method design to investigate the recruitment and retention, reach, dose, fidelity, acceptability, and context of implementation, and was guided by the Consolidated Framework for Implementation Research (CFIR). We reviewed trial documents, conducted semi-structured interviews with key informants (n = 10) and conducted four focus group discussions with participants in both arms of the trial. Three analysts undertook CFIR theory-driven content analysis of the qualitative data. Quantitative data was analyzed to provide descriptive statistics relevant to the objectives of the process evaluation.

RESULTS

The trial had positive implementation outcomes, 100% retention of clusters and 96% retention for participants, 83% adherence to delivery of content of group talks (fidelity), and 78% attendance (reach) to at least 50% (3/6) of the group talks (dose). The data revealed that intervention characteristics, outer setting, inner setting, individual characteristics, and process (all the constructs of CFIR) influenced the implementation. There were more facilitators than barriers to the implementation. Facilitators included the relative advantage of the intervention compared with current practice (intervention characteristics); awareness of the growing prioritization of diabetes in the national health policy framework (outer setting); tension for change due to the realization of the vulnerability to vision loss from DR (inner setting); a strong collective sense of accountability of peer supporters to implement the intervention (individual characteristics); and regular feedback on the progress with implementation (process). Potential barriers included the need to queue at the eye clinic (intervention characteristic), travel inconveniences (inner setting), and socio-political disruption (outer setting).

CONCLUSIONS

The intervention was implemented with high retention, reach, fidelity, and dose. The CFIR provided a valuable framework for evaluating contextual factors that influenced implementation and helped to understand what adaptations may be needed during scale up.

TRIAL REGISTRATION

Pan African Clinical Trials Registry: PACTR201707002430195 registered 15 July 2017.

Systematic review of the use of process evaluations in knowledge translation research

BACKGROUND

Experimental designs for evaluating knowledge translation (KT) interventions can provide strong estimates of effectiveness but offer limited insight into how the intervention worked. Consequently, process evaluations have been used to explore the causal mechanisms at work; however, there are limited standards to guide this work. This study synthesizes current evidence of KT process evaluations to provide future methodological recommendations.

METHODS

Peer-reviewed search strategies were developed by a health research librarian. Studies had to be in English, published since 1996, and were not excluded based on design. Studies had to (1) be a process evaluation of a KT intervention study in primary health, (2) be a primary research study, and (3) include a licensed healthcare professional delivering or receiving the intervention. A two-step, two-person hybrid screening approach was used for study inclusion with inter-rater reliability ranging from 94 to 95%. Data on study design, data collection, theoretical influences, and approaches used to evaluate the KT intervention, analysis, and outcomes were extracted by two reviewers. Methodological quality was assessed with the Mixed Methods Appraisal Tool (MMAT).

RESULTS

Of the 20,968 articles screened, 226 studies fit our inclusion criteria. The majority of process evaluations used qualitative forms of data collection (43.4%) and individual interviews as the predominant data collection method. 72.1% of studies evaluated barriers and/or facilitators to implementation. 59.7% of process evaluations were stand-alone evaluations. The timing of data collection varied widely with post-intervention data collection being the most frequent (46.0%). Only 38.1% of the studies were informed by theory. Furthermore, 38.9% of studies had MMAT scores of 50 or less indicating poor methodological quality.

CONCLUSIONS

There is widespread acceptance that the generalizability of quantitative trials of KT interventions would be significantly enhanced through complementary process evaluations. However, this systematic review found that process evaluations are of mixed quality and lack theoretical guidance. Most process evaluation data collection occurred post-intervention undermining the ability to evaluate the process of implementation. Strong science and methodological guidance is needed to underpin and guide the design and execution of process evaluations in KT science.

REGISTRATION

This study is not registered with PROSPERO.

Process evaluation of the systematic medical appraisal, referral and treatment (SMART) mental health project in rural India

BACKGROUND

Availability of basic mental health services is limited in rural areas of India. Health system and individual level factors such as lack of mental health professionals and infrastructure, poor awareness about mental health, stigma related to help seeking, are responsible for poor awareness and use of mental health services. We implemented a mental health services delivery model that leveraged technology and task sharing to facilitate identification and treatment of common mental disorders (CMDs) such as stress, depression, anxiety and suicide risk in rural areas of the state of Andhra Pradesh, India. The intervention was delivered by lay village health workers (Accredited Social Health Activists - ASHAs) and primary care doctors. An anti-stigma campaign was implemented prior to this activity. This paper reports the process evaluation of the intervention using mixed methods.

METHODS

A mixed methods pre-post evaluation assessed the intervention using quantitative service usage analytics from the server, and qualitative interviews with different stakeholders. Barriers and facilitators in implementing the intervention were identified.

RESULTS

Health service use increased significantly at post-intervention, ASHAs could followup 78.6% of those who had screened positive, and 78.6% of the 1243 Interactive Voice Response System calls made, were successful. Most respondents were aware of the intervention. They indicated that knowledge received through the intervention empowered them to approach ASHAs and share their mental health symptoms. ASHAs and doctors opined that EDSS was useful and easy to use. Medical camps organized in villages to increase access to the doctor were received positively by all. However, some aspects or facilitators of the intervention need to be improved, including network connectivity, booster training, anti-stigma campaigns, quality of mental health services provided by doctors, provision of psychotropic medications at primary health centers and frequency of health camps.

CONCLUSION

The respondents' views helped to understand the barriers and facilitators for improving the likely effectiveness of the intervention using Andersen's Modified Behavioral Model of Health Services Use, and identify the mechanisms by which those factors affected mental health services uptake in the community.

TRIAL REGISTRATION

The study is registered with Clinical Trials Registry India (Applied - 16/07/14-Ref2014/07/007256; registration received - 04/10/17-CTRI/2017/10/009992 ).

Peer support for people with severe mental illness versus usual care in high-, middle- and low-income countries: study protocol for a pragmatic, multicentre, randomised controlled trial (UPSIDES-RCT)

BACKGROUND

Peer support is an established intervention involving a person recovering from mental illness supporting others with mental illness. Peer support is an under-used resource in global mental health. Building upon comprehensive formative research, this study will rigorously evaluate the impact of peer support at multiple levels, including service user outcomes (psychosocial and clinical), peer support worker outcomes (work role and empowerment), service outcomes (cost-effectiveness and return on investment), and implementation outcomes (adoption, sustainability and organisational change).

METHODS

UPSIDES-RCT is a pragmatic, parallel-group, multicentre, randomised controlled trial assessing the effectiveness of using peer support in developing empowering mental health services (UPSIDES) at four measurement points over 1 year (baseline, 4-, 8- and 12-month follow-up), with embedded process evaluation and cost-effectiveness analysis. Research will take place in a range of high-, middle- and low-income countries (Germany, UK, Israel, India, Uganda and Tanzania). The primary outcome is social inclusion of service users with severe mental illness (N = 558; N = 93 per site) at 8-month follow-up, measured with the Social Inclusion Scale. Secondary outcomes include empowerment (using the Empowerment Scale), hope (using the HOPE scale), recovery (using Stages of Recovery) and health and social functioning (using the Health of the Nations Outcome Scales). Mixed-methods process evaluation will investigate mediators and moderators of effect and the implementation experiences of four UPSIDES stakeholder groups (service users, peer support workers, mental health workers and policy makers). A cost-effectiveness analysis examining cost-utility and health budget impact will estimate the value for money of UPSIDES peer support.

DISCUSSION

The UPSIDES-RCT will explore the essential components necessary to create a peer support model in mental health care, while providing the evidence required to sustain and eventually scale-up the intervention in different cultural, organisational and resource settings. By actively involving and empowering service users, UPSIDES will move mental health systems toward a recovery orientation, emphasising user-centredness, community participation and the realisation of mental health as a human right.

TRIAL REGISTRATION

ISRCTN, ISRCTN26008944. Registered on 30 October 2019.

Nontarget analysis: A new tool for the evaluation of wastewater processes

Strategies to determine the removal efficiency of micropollutants in wastewater treatment plants (WWTPs) are widely discussed. Especially the evaluation of the potential benefit of further advanced treatment steps such as an additional tertiary treatment based on ozonation or activated carbon have come into focus. Such evaluation strategies are often based on the removal behavior of known micropollutants via target or suspected analysis. The utilization of nontarget analysis is considered to lead to a more comprehensive picture as also unknown or not expected micropollutants are analyzed. Here, the results of an evaluation via target and nontarget analysis were compared for biological treatment (BT) processes of eleven full-scale WWTPs and three different post-treatments (PTs): one sand filter (SF) and two granular activated carbon (GAC) filters. The similarity of the determined removals from target and nontarget analysis of the BTs increased significantly by excluding easily degradable "features" from the nontarget evaluation. A similar ranking of the removal trends for the BTs could also be achieved by comparing this new subset of nontarget features with a set of nine readily to moderately biodegradable micropollutants. This observation suggests that a performance ranking of BTs based either on target or nontarget analysis is plausible. In contrast to the BTs, the evaluation of the three PTs revealed that the difference of feature removal between SF and the two GACs was small, but large for the target analytes with substantially higher removal effciencies for the GACs compared to the SF. In addition to the removal behavior, the nontarget analysis provided further information about the number and quantity of transformation products (TPs) in the effluent from the BTs. For all BTs more than half (55-67%) of the features detected in the effluent were not found in the influent. A comparable proportion of TPs was also detected after GAC and sand filtration due to their microbial activities.

Process evaluation within pragmatic randomised controlled trials: what is it, why is it done, and can we find it?-a systematic review

BACKGROUND

Process evaluations are increasingly conducted within pragmatic randomised controlled trials (RCTs) of health services interventions and provide vital information to enhance understanding of RCT findings. However, issues pertaining to process evaluation in this specific context have been little discussed. We aimed to describe the frequency, characteristics, labelling, value, practical conduct issues, and accessibility of published process evaluations within pragmatic RCTs in health services research.

METHODS

We used a 2-phase systematic search process to (1) identify an index sample of journal articles reporting primary outcome results of pragmatic RCTs published in 2015 and then (2) identify all associated publications. We used an operational definition of process evaluation based on the Medical Research Council's process evaluation framework to identify both process evaluations reported separately and process data reported in the trial results papers. We extracted and analysed quantitative and qualitative data to answer review objectives.

RESULTS

From an index sample of 31 pragmatic RCTs, we identified 17 separate process evaluation studies. These had varied characteristics and only three were labelled 'process evaluation'. Each of the 31 trial results papers also reported process data, with a median of five different process evaluation components per trial. Reported barriers and facilitators related to real-world collection of process data, recruitment of participants to process evaluations, and health services research regulations. We synthesised a wide range of reported benefits of process evaluations to interventions, trials, and wider knowledge. Visibility was often poor, with 13/17 process evaluations not mentioned in the trial results paper and 12/16 process evaluation journal articles not appearing in the trial registry.

CONCLUSIONS

In our sample of reviewed pragmatic RCTs, the meaning of the label 'process evaluation' appears uncertain, and the scope and significance of the term warrant further research and clarification. Although there were many ways in which the process evaluations added value, they often had poor visibility. Our findings suggest approaches that could enhance the planning and utility of process evaluations in the context of pragmatic RCTs.

TRIAL REGISTRATION

Not applicable for PROSPERO registration.

Process evaluation in the field: global learnings from seven implementation research hypertension projects in low-and middle-income countries

BACKGROUND

Process evaluation is increasingly recognized as an important component of effective implementation research and yet, there has been surprisingly little work to understand what constitutes best practice. Researchers use different methodologies describing causal pathways and understanding barriers and facilitators to implementation of interventions in diverse contexts and settings. We report on challenges and lessons learned from undertaking process evaluation of seven hypertension intervention trials funded through the Global Alliance of Chronic Diseases (GACD).

METHODS

Preliminary data collected from the GACD hypertension teams in 2015 were used to inform a template for data collection. Case study themes included: (1) description of the intervention, (2) objectives of the process evaluation, (3) methods including theoretical basis, (4) main findings of the study and the process evaluation, (5) implications for the project, policy and research practice and (6) lessons for future process evaluations. The information was summarized and reported descriptively and narratively and key lessons were identified.

RESULTS

The case studies were from low- and middle-income countries and Indigenous communities in Canada. They were implementation research projects with intervention arm. Six theoretical approaches were used but most comprised of mixed-methods approaches. Each of the process evaluations generated findings on whether interventions were implemented with fidelity, the extent of capacity building, contextual factors and the extent to which relationships between researchers and community impacted on intervention implementation. The most important learning was that although process evaluation is time consuming, it enhances understanding of factors affecting implementation of complex interventions. The research highlighted the need to initiate process evaluations early on in the project, to help guide design of the intervention; and the importance of effective communication between researchers responsible for trial implementation, process evaluation and outcome evaluation.

CONCLUSION

This research demonstrates the important role of process evaluation in understanding implementation process of complex interventions. This can help to highlight a broad range of system requirements such as new policies and capacity building to support implementation. Process evaluation is crucial in understanding contextual factors that may impact intervention implementation which is important in considering whether or not the intervention can be translated to other contexts.

Facilitated interprofessional implementation of a physical rehabilitation guideline for stroke in inpatient settings: process evaluation of a cluster randomized trial

Background

The Stroke Canada Optimization of Rehabilitation by Evidence-Implementation Trial (SCORE-IT) showed that a facilitated knowledge translation (KT) approach to implementing a stroke rehabilitation guideline was more likely than passive strategies to improve functional walking capacity, but not gross manual dexterity, among patients in rehabilitation hospitals. This paper presents the results of a planned process evaluation designed to assess whether the type and number of recommended treatments implemented by stroke teams in each group would help to explain the results related to patient outcomes.

Methods

As part of a cluster randomized trial, 20 rehabilitation units were stratified by language and allocated to a facilitated or passive KT intervention group. Sites in the facilitated group received the guideline with treatment protocols and funding for a part-time nurse and therapist facilitator who attended a 2-day training workshop and promoted guideline implementation for 16 months. Sites in the passive group received the guideline excluding treatment protocols. As part of a process evaluation, nurses, and occupational and physical therapists, blinded to study hypotheses, were asked to record their implementation of 18 recommended treatments targeting motor function, postural control and mobility using individualized patient checklists after treatment sessions for 2 weeks pre- and post-intervention. The percentage of patients receiving each treatment pre- and post-intervention and between groups was compared after adjusting for clustering and covariates in a random-effects logistic regression analysis.

Results

Data on treatment implementation from nine and eight sites in the facilitated and passive KT group, respectively, were available for analysis. The facilitated KT intervention was associated with improved implementation of sit-to-stand (p = 0.028) and walking (p = 0.043) training while the passive KT intervention was associated with improved implementation of standing balance training (p = 0.037), after adjusting for clustering at patient and provider levels and covariates.

Conclusions

Despite multiple strategies and resources, the facilitated KT intervention was unsuccessful in improving integration of 18 treatments concurrently. The facilitated approach may not have adequately addressed barriers to integrating numerous treatments simultaneously and complex treatments that were unfamiliar to providers.

Trial registration

Unique identifier-NCT00359593

Electronic supplementary material

The online version of this article (doi:10.1186/s13012-017-0631-7) contains supplementary material, which is available to authorized users.

Collaborative Process Evaluation: Enhancing Community Relevance and Cultural Appropriateness in HIV Prevention

Process evaluations provide multiple opportunities to improve prevention and intervention programs that benefit communities experiencing oppression and marginalization. In order to thwart the negative effects of power and privilege, it is essential that collaborative partnerships with community-based organizations (CBOs) incorporate the voices, experiences, and skills of community members. This article presents a process evaluation that was collaboratively developed and implemented by a team consisting of members from an HIV/AIDS-related CBO and a university. The process evaluation was guided by an empowerment evaluation theoretical framework and utilized elements of narrative ethnography. The methods for the evaluation were developed to ensure cultural appropriateness, community sensitivity, and scientific rigor. Modifications made to the CBO's HIV prevention programs for Latina female adolescents and gay/bisexual/questioning Latino and African-American male adolescents incorporated specific social, cultural, and environmental factors that impact HIV risk. The multiple benefits gained by those involved in the evaluation are detailed and emphasize the importance of conducting collaborative process evaluations when developing interventions for oppressed and marginalized communities.

Cochrane Qualitative and Implementation Methods Group guidance series-paper 4: methods for assessing evidence on intervention implementation

OBJECTIVES

This article provides reviewers with guidance on methods for identifying and processing evidence to understand intervention implementation.

STUDY DESIGN AND SETTING

Strategies, tools, and methods are applied to the systematic review process to illustrate how process and implementation can be addressed using quantitative, qualitative, and other sources of evidence (i.e., descriptive textual and nonempirical).

RESULTS

Reviewers can take steps to navigate the heterogeneity and level of uncertainty present in the concepts, measures, and methods used to assess implementation. Activities can be undertaken in advance of a Cochrane quantitative review to develop program theory and logic models that situate implementation in the causal chain. Four search strategies are offered to retrieve process and implementation evidence. Recommendations are made for addressing rigor or risk of bias in process evaluation or implementation evidence. Strategies are recommended for locating and extracting data from primary studies. The basic logic is presented to assist reviewers to make initial review-level judgments about implementation failure and theory failure.

CONCLUSION

Although strategies, tools, and methods can assist reviewers to address process and implementation using quantitative, qualitative, and other forms of evidence, few exemplar reviews exist. There is a need for further methodological development and trialing of proposed approaches.

Process evaluation for complex interventions in health services research: analysing context, text trajectories and disruptions

Background

Process evaluations assess the implementation and sustainability of complex healthcare interventions within clinical trials, with well-established theoretical models available for evaluating intervention delivery within specific contexts. However, there is a need to translate conceptualisations of context into analytical tools which enable the dynamic relationship between context and intervention implementation to be captured and understood.

Methods

In this paper I propose an alternative approach to the design, implementation and analysis of process evaluations for complex health interventions through a consideration of trial protocols as textual documents, distributed and enacted at multiple contextual levels. As an example, I conduct retrospective analysis of a sample of field notes and transcripts collected during the ESTEEM study - a cluster randomised controlled trial of primary care telephone triage. I draw on theoretical perspectives associated with Linguistic Ethnography to examine the delivery of ESTEEM through staff orientation to different texts. In doing so I consider what can be learned from examining the flow and enactment of protocols for notions of implementation and theoretical fidelity (i.e. intervention delivered as intended and whether congruent with the intervention theory).

Results

Implementation of the triage intervention required staff to integrate essential elements of the protocol within everyday practice, seen through the adoption and use of different texts that were distributed across staff and within specific events. Staff were observed deploying texts in diverse ways (e.g. reinterpreting scripts, deviating from standard operating procedures, difficulty completing decision support software), providing numerous instances of disruption to maintaining intervention fidelity. Such observations exposed tensions between different contextual features in which the trial was implemented, offering theoretical explanations for the main trial findings.

Conclusions

The value of following how trial protocols produce new texts is that we can observe the flow of ‘the intervention as intended’ across a series of events which are enacted to meet specific demands of intervention delivery. Such observations are not solely premised on identifying routines or practices of implementation, but where ‘protocols as intended’ breaks down. In doing so, I discuss whether it is here where we might expose the ‘active ingredients’ of interventions in action.

Electronic supplementary material

The online version of this article (doi:10.1186/s12913-016-1651-8) contains supplementary material, which is available to authorized users.

Process evaluation of the Walk Well study: a cluster-randomised controlled trial of a community based walking programme for adults with intellectual disabilities

BACKGROUND

Walking interventions can be effective in encouraging sedentary populations to become more active; however, limited research has explored the effectiveness of walking interventions for adults with intellectual disabilities. This process evaluation explored the delivery of a community based walking intervention for adults with intellectual disabilities.

METHODS

Walk Well was a single-blind cluster randomised controlled trial of a 12-week physical activity consultation-led walking intervention. 102 participants were randomised to the Walk Well intervention or a waiting list control group. Participants in the intervention group received three physical activity consultations with a walking advisor at baseline, 6 & 12-weeks. They were encouraged to use a pedometer to set goals and monitor their daily step count. Primary outcome was change in daily step count at 12-weeks. Process evaluation measures included qualitative interviews with key stakeholders (n = 6) and quantifiable data collected as part of the intervention. Additional process data were extracted from a sub-set of qualitative interviews with participants and carers (n = 20). Data were analysed for process information related to context, recruitment and retention, reach, implementation, and fidelity.

RESULTS

Walk Well was not effective in significantly increasing levels of physical activity. The process evaluation did, however, highlight several important areas for consideration in future studies, including: a successful recruitment and retention strategy reaching a representative sample of adults with intellectual disabilities in the community; feasible and (for most) enjoyable methods of engaging adults with intellectual disabilities in activities to support behaviour change; potential need for greater intervention duration and frequency of contact; advantages and disadvantages of using pedometers as a behaviour change tool; the need for strategies which engage carers in supporting participants; and the complex issue of 'freedom of choice' in relation to lifestyle behaviours and study participation.

CONCLUSIONS

Walking interventions for adults with intellectual disabilities can be feasibly delivered in the community in relation to reach, recruitment, retention and intervention fidelity. More intensive intervention methods need to be explored as well as strategies to engage and motivate carers in their support of participants.

TRIAL REGISTRATION

Current Controlled Trials ISRCTN50494254 (3(rd) April 2012).

Mesophilic batch anaerobic co-digestion of fruit-juice industrial waste and municipal waste sludge: process and cost-benefit analysis

The feasibility of anaerobic co-digestion of two juice-based beverage industrial wastes, screen cake (SC) and thickened waste activated sludge (TWAS), along with municipal sludge cake (MC) was investigated. Experiments were conducted in twenty mesophilic batch 160 ml serum bottles with no inhibition occurred. The statistical analysis proved that the substrate type had statistically significant effect on both ultimate biogas and methane yields (P=0.0003<0.05). The maximum and minimum ultimate cumulative methane yields were 890.90 and 308.34 mL/g-VSremoved from the digesters containing only TWAS and SC as substrate. First-order reaction model well described VS utilization in all digesters. The first 2-day and 10-day specific biodegradation rate constants were statistically higher in the digesters containing SC (P=0.004<0.05) and MC (P=0.0005<0.05), respectively. The cost-benefit analysis showed that the capital, operating and total costs can be decreased by 21.5%, 29.8% and 27.6%, respectively using a co-digester rather than two separate digesters.

Worldwide use of the first set of physical activity Country Cards: The Global Observatory for Physical Activity - GoPA!

Background

The work of The Global Observatory for Physical Activity-GoPA! is the first global effort to compile standardized country-level surveillance, policy and research data for physical activity in order to better understand how countries and regions address promoting physical activity. GoPA! developed standardized country-specific physical activity profiles (“Country Cards”) to summarize country-level data through 2013. The aim of this study was to assess use of the Country Cards, identify the factors associated with their use, and develop recommendations for supporting country-level physical activity promotion.

Methods

Cross sectional internet-based survey conducted between August–October 2016. Target study participants were national physical activity leaders and advocates in academia, government and practice from the GoPA! countries, and members of the International Society of Physical Activity and Health. A Country Card use composite score was created based on the diversity and frequency of use. Statistical analyses on the associations between the composite score and respondent characteristics, country characteristics, barriers and opinions were conducted (including descriptive analyses and a logistic regression with robust standard errors).

Results

One hundred forty three participants from 68 countries completed the survey. Use of the Country Cards was associated with being part of the GoPA! network, knowing about the Country Cards, and on the stage of country capacity for physical activity promotion. Country Card knowledge varied by country income group, region and the country specific context. More diverse and frequent use of the cards (highest tertile of the composite score for use) was associated with: 1. Being a country contact vs general participant (OR 18.32–95% CI 5.63–59.55, p = 0.002), and 2. Collaborating with a government representative working in NCDs on a monthly or more frequent contact vs less frequent contact (OR 3.39–95% CI 1.00–11.54, P < 0.05).

Conclusions

For the Country Cards to have a broader impact, GoPA! will need to widen its reach beyond the academic sector. With further refinement of the cards, and training in their implementation, they could be an important tool for advancing country capacity for contextually-relevant strategies, actions and timelines for PA promotion.

Electronic supplementary material

The online version of this article (10.1186/s12966-018-0663-7) contains supplementary material, which is available to authorized users.

Can we Build on Social Movement Theories to Develop and Improve Community-Based Participatory Research? A Framework Synthesis Review

A long-standing challenge in community-based participatory research (CBPR) has been to anchor practice and evaluation in a relevant and comprehensive theoretical framework of community change. This study describes the development of a multidimensional conceptual framework that builds on social movement theories to identify key components of CBPR processes. Framework synthesis was used as a general literature search and analysis strategy. An initial conceptual framework was developed from the theoretical literature on social movement. A literature search performed to identify illustrative CBPR projects yielded 635 potentially relevant documents, from which eight projects (corresponding to 58 publications) were retained after record and full-text screening. Framework synthesis was used to code and organize data from these projects, ultimately providing a refined framework. The final conceptual framework maps key concepts of CBPR mobilization processes, such as the pivotal role of the partnership; resources and opportunities as necessary components feeding the partnership's development; the importance of framing processes; and a tight alignment between the cause (partnership's goal), the collective action strategy, and the system changes targeted. The revised framework provides a context-specific model to generate a new, innovative understanding of CBPR mobilization processes, drawing on existing theoretical foundations.