Measurement of fidelity of implementation of evidence‐based practices: Case example of the IPS Fidelity Scale

Protocol of a randomized controlled trial on the effectiveness and cost-effectiveness of the PLACES intervention: a supported employment intervention aimed at enhancing work participation of unemployed and/or work-disabled cancer survivors

BACKGROUND

Approximately onethird of cancer survivors encounter challenges reintegrating into the workforce, often experiencing involuntary unemployment and/or partial or full work disability following diagnosis and treatment. Returning to paid employment presents evident challenges due to uncertainties regarding work ability, perceived employer discrimination, and a lack of support, thereby risking social exclusion. However, interventions addressing return to paid employment among unemployed and/or work-disabled cancer survivors are scarce. Here, we describe the protocol of a randomized controlled trial (RCT), including a process and economic evaluation, evaluating the effectiveness and cost-effectiveness of the PLACES (unemPLoyed cAnCEr survivors Support) intervention aimed at supporting unemployed and/or work-disabled cancer survivors returning to paid employment.

METHODS

A two-armed RCT with a 12-month follow-up period will be conducted. Eligible participants: (1) are of working age (18-65 years), (2) are diagnosed with cancer between 6 months and 10 years ago, (3) are unemployed and/or partially or fully work-disabled, (4) have completed cancer treatment, and (5) are seeking paid employment and are motivated to initiate work immediately. Participants will primarily be identified through the Dutch Social Security Agency and the Netherlands Cancer Registry and recruited via healthcare professionals. Participants randomly allocated to the intervention group (n = 82) will receive the PLACES intervention: a tailored supported employment intervention based on the principles of Individual Placement and Support (IPS). This includes support in seeking, returning to, and maintaining paid employment. Participants allocated to the control group (n = 82) will receive care as usual. All participants will be asked to complete questionnaires, at baseline (T0), and after 3 (T1), 6 (T2), and 12 (T3) months of follow-up. The primary outcome is paid employment [yes/no]. Secondary outcomes are time until paid employment, change in working hours, work ability, quality of (working) life, and self-efficacy regarding return to work. Additionally, process and economic evaluations will be conducted.

DISCUSSION

We hypothesize that the PLACES intervention will be effective in obtaining paid employment, enhancing work ability, and improving quality of life. In addition, we expect the intervention to be cost-effective. If proven effective and cost-effective, actions should be taken to implement the intervention in usual care.

TRIAL REGISTRATION

NCT06028048.

Employment support for Black people with long-term health conditions: a systematic narrative review of UK studies

BACKGROUND

Black people in the United Kingdom disproportionately acquire long-term health conditions and are marginalised from the labour market compared with other groups. These conditions interact and reinforce high rates of unemployment among Black people with long-term health conditions.

AIMS

To examine the efficacy, and experience, of employment support interventions in meeting the needs of Black service users in Britain.

METHODS

A systematic literature search was conducted focusing on peer-reviewed literature featuring samples drawn from the United Kingdom.

RESULTS

The literature search revealed a paucity of articles that include analysis of Black people's outcomes or experiences. Six articles met the selection criteria of the review, of which five focused on mental health impairments. No firm conclusions could be drawn from the systematic review; however, the evidence suggests that Black people are less likely than their White counterparts to secure competitive employment and that Individual Placement and Support (IPS) may be less effective for Black participants.

CONCLUSIONS

We argue for a greater focus on ethnic differences in employment support outcomes with an emphasis on how such services may remediate racial differences in employment outcomes. We conclude by foregrounding how structural racism may explain the dearth of empirical evidence in this review.

Exploring the Use of the Picture Exchange Communication System (PECS) in Special Education Settings

The Picture Exchange Communication System (PECS) is an Augmentative and Alternative Communication (AAC) system which is widely used to support children with developmental disabilities. In the present study, we surveyed individuals responsible for implementing PECS in special educational settings in the United Kingdom (N=283). We explored knowledge of and adherence to the  intervention, with a view to identifying training and support needs. Specifically, we examined participants' knowledge, implementation accuracy, training experiences, access to resources, and attitudes towards PECS. We developed hierarchical logistic regression models to explore the association between training experience and both knowledge and use of PECS. We pre-registered our methods, predictions and the analysis plan on the Open Science Framework (OSF).We found considerable variation in practitioner knowledge and implementation of PECS. Formal training predicted greater knowledge and more accurate implementation when practitioner role and the degree of setting support were accounted for. While PECS was rated by a large majority to be effective and practical, many participants identified that time and the availability of resources were barriers to implementation. We also found that the purpose of PECS was not always fully understood by practitioners, and we identified some consistent gaps in knowledge and implementation.  This study contributes new information regarding the real-world  use of PECS in educational settings and offers new insights for supporting practitioners.

Assessing the Fidelity of an Affirmative Cognitive Behavioral Group Intervention

PURPOSE

Support implementation fidelity in intervention research with lesbian, gay, bisexual, transgender, queer, and sexual and gender diverse (LGBTQ+) populations, this study explores the systematic development of a fidelity process for AFFIRM, an evidence-based, affirmative cognitive behavioral therapy group intervention for LGBTQ+ youth and adults.

METHOD

As part of a clinical trial, the AFFIRM fidelity checklist was designed to assess clinician adherence. A total of 151 audio-recorded group sessions were coded by four trained raters.

RESULTS

Adherence was high with a mean fidelity score of 84.13 (SD = 12.50). Inter-rater reliability was 81%, suggesting substantial agreement. Qualitative thematic analysis of low-rated sessions identified deviations from the manual and difficulties in group facilitation, while high-rated sessions specified affirmative and effective clinical responses.

DISCUSSION

Findings were integrated into clinical training and coaching. The fidelity process provides insights into the challenges of implementing social work interventions effectively with LGBTQ+ populations in community settings.

Protocol for the Feasibility Randomised Controlled Trial of Being a Parent- Enjoying Family Life (BAP-EFL): A peer-led group intervention for parents and caregivers with significant emotional and interpersonal difficulties

Group-format parenting interventions are well-established at reducing challenging child behavior and risk for psychopathology. However, there is significantly less evidence about the performance of these interventions for parents with significant emotional and interpersonal difficulties, including personality disorder. This protocol presents the rationale and design of a two-arm parallel group feasibility Randomised Controlled Trial and nested process evaluation of Being a Parent (BaP)- Enjoying Family Life, a novel peer-led intervention. The trial compares BaP-Enjoying Family Life to the well-established Empowering Parents Empowering Communities-Being a Parent (EPEC-Being a Parent) in a sample of parents who experience significant emotional and interpersonal difficulties and who are concerned about their child's, aged 2-11 years, behavior. 72 parents will be recruited and randomised to receive either BaP-Enjoying Family Life or EPEC-Being a Parent group-format interventions. The primary aim of this study is to examine the feasibility and acceptability of BaP-Enjoying Family Life and the proposed trial methods. Secondary clinical outcomes include child behavioral difficulties, parenting, parental reflective function, parent wellbeing, satisfaction and self-efficacy. An observational assessment of parent and index child will also assess changes in the home environment. Outcome measures will be collected pre-intervention, post-intervention and at 6-month follow up. A parallel process evaluation will use qualitative data from interviews to assess parents' experience of the intervention delivery and trial methods. Findings will be evaluated against pre-determined feasibility criteria. The results will be used to determine the planning of a definitive clinical trial. The wider methodological and clinical implications are also discussed.

Evaluating Educational and Employment Services for Young People With Psychiatric Conditions: A Systematic Review

OBJECTIVE

The authors conducted a systematic review of studies evaluating vocational interventions for young people with psychiatric conditions to determine the extent to which services were adapted for young people and whether services promoted gains in postsecondary education and employment.

METHODS

Five databases (PubMed, PsycINFO, Web of Science, Academic Search Premier, and ERIC) were searched. Sources eligible for inclusion were controlled studies published between 2000 and mid-2020 that evaluated a vocational intervention and examined postsecondary educational or employment outcomes for youths or young adults (ages 14-35 years) with psychiatric conditions.

RESULTS

Ten studies met the inclusion criteria. Several of the studies evaluated services that were adapted for young people, including through the incorporation of educational supports. The most consistent finding was that services based on the individual placement and support (IPS) model improved employment outcomes more effectively than did comparison approaches and treatment as usual. Fewer studies assessed educational outcomes, and they yielded mixed results; however, recent findings from a controlled trial indicate that an enhanced IPS intervention that included well-specified supported education and skills training led to significantly superior outcomes in both education and employment.

CONCLUSIONS

These results provide support for the efficacy of IPS-based services to improve employment outcomes among young people with psychiatric conditions and suggest that adapting IPS to include comprehensive educational supports and skills training may be important for efforts to improve postsecondary educational outcomes. Additional well-controlled intervention studies that examine educational and longer-term outcomes should further inform the development and delivery of vocational services for this population.

The Active Recovery Triad monitor: evaluation of a model fidelity scale for recovery-oriented care in long-term mental health care settings

OBJECTIVE

The Active Recovery Triad (ART) model is a recently developed care model for people who are admitted to an institutional setting for several years and receive 24-h mental health care and support. This study focuses on the ART monitor, a model fidelity scale that measures the degree of compliance with the ART model. Our aim is to evaluate the psychometric properties of the ART monitor and to further improve the instrument.

METHODS

Fifteen teams at the start (n = 7, group 1) or in the process (6 months to three years) of implementing care according to the ART model (n = 8, group 2) were audited using the ART monitor. Auditors were trained care workers, peer workers, and family peer workers. Auditors and team members provided feedback on the instrument. The content validity, construct validity and inter-rater reliability of the ART monitor were investigated. Based on the outcomes of these psychometric properties, the ART monitor was finalized.

RESULTS

Regarding content validity, auditors and teams indicated that they perceived the ART monitor to be a useful instrument. In terms of construct validity, a significant difference (t(13) = 2.53, p < 0.05) was found between teams at the start of the implementation process (group 1, average score of 2.42 (SD = 0.44)) and teams with a longer duration of implementation (group 2, average score of 2.95 (SD = 0.37)). When allowing for a one-point difference in scores, 88% of the items had an inter-rater agreement over 65%. Items with a relatively low inter-rater reliability, in combination with feedback from auditors and teams regarding content validity, provided direction for further improvement and revision of the instrument.

CONCLUSIONS

We concluded that the revised ART monitor is feasible and useful in mental health care practice. However, further evaluation of its psychometric properties will be needed.

The case for prioritizing implementation strategy fidelity measurement: benefits and challenges

Implementation strategies are systematic approaches to improve the uptake and sustainability of evidence-based interventions. They frequently focus on changing provider behavior through the provision of interventions such as training, coaching, and audit-and-feedback. Implementation strategies often impact intermediate behavioral outcomes like provider guideline adherence, in turn improving patient outcomes. Fidelity of implementation strategy delivery is defined as the extent to which an implementation strategy is carried out as it was designed. Implementation strategy fidelity measurement is under-developed and under-reported, with the quality of reporting decreasing over time. Benefits of fidelity measurement include the exploration of the extent to which observed effects are moderated by fidelity, and critical information about Type-III research errors, or the likelihood that null findings result from implementation strategy fidelity failure. Reviews of implementation strategy efficacy often report wide variation across studies, commonly calling for increased implementation strategy fidelity measurement to help explain variations. Despite the methodological benefits of rigorous fidelity measurement, implementation researchers face multi-level challenges and complexities. Challenges include the measurement of a complex variable, multiple data collection modalities with varying precision and costs, and the need for fidelity measurement to change in-step with adaptations. In this position paper, we weigh these costs and benefits and ultimately contend that implementation strategy fidelity measurement and reporting should be improved in trials of implementation strategies. We offer pragmatic solutions for researchers to make immediate improvements like the use of mixed methods or innovative data collection and analysis techniques, the inclusion of implementation strategy fidelity assessment in reporting guidelines, and the staged development of fidelity tools across the evolution of an implementation strategy. We also call for additional research into the barriers and facilitators of implementation strategy fidelity measurement to further clarify the best path forward.

The Forensic High and Intensive Care Monitor: Measurement Properties of a Model Fidelity Scale for Contact-Based Care in Forensic Psychiatry

Forensic High and Intensive Care (FHIC) has recently been developed as a new care model in Dutch forensic psychiatry. FHIC aims to provide contact-based care. To support Dutch forensic care institutions in the implementation of the model, a model fidelity scale was developed called the FHIC monitor. The aim of this study was to assess the inter-rater reliability, content validity, and construct validity of the FHIC monitor. A multi-methods design was used, combining qualitative and quantitative research. To collect data, audits and focus group meetings were organized to score care at individual wards with the monitor and get feedback from auditors and audit receiving teams about the quality of the monitor. In total, fifteen forensic mental healthcare institutions participated. The instrument showed acceptable inter-rater reliability and content validity, and a significant difference between expected high and low scoring institutions, supporting construct validity. The instrument can be used as a valid instrument to measure the level of implementation of the FHIC model on forensic psychiatric wards in the Netherlands.

Rethinking Management Information Systems for Scaling up Employment Outcomes

People with intellectual disability, autism, and other developmental disabilities have successfully demonstrated their ability to work in the general labor market. Yet, their employment rate remains substantially lower compared with the general population without disabilities. Scaling up the employment outcomes of this population requires improving the effectiveness and efficiency of employment support systems. To this end, this article recommends supporting employment providers rethink how they use their management information systems (MIS): from primary tools that automate billing and compliance, to tools that track metrics for continuous quality improvement. As federal and state funding policies are the main factors shaping how MIS are currently used, this article recommends that federal and state policy guidelines be issued that enable employment providers to leverage their MIS to improve effectiveness and efficiency, and thus scale up employment outcomes of job seekers.

Understanding the value of adhering to or adapting evidence-based interventions: a study protocol of a discrete choice experiment

Background

Whereas the value of an evidence-based intervention (EBI) is often determined by its effect on clinical outcomes, the value of implementing and using EBIs in practice is broader, reflecting qualities such as appropriateness, equity, costs, and impact. Reconciling these value conflicts involves a complicated decision process that has received very limited scholarly attention. Inspired by studies on decision-making, the objective of this project is to explore how practitioners appraise the values of different outcomes and to test how this appraisal influences their decisions surrounding the so-called fidelity–adaptation dilemma. This dilemma is related to the balance between using an EBI as it was designed (to ensure its effectiveness) and making appropriate adaptations (to ensure alignment with constraints and possibilities in the local context).

Methods

This project consists of three sub-studies. The participants will be professionals leading evidence-based parental programs in Sweden and, in Sub-study 1, parents and decision-makers. Sub-study 1 will use sequential focus groups and individual interviews to explore parameters that influence fidelity and adaptation decisions—the dilemmas encountered, available options, how outcomes are valued by practitioners as well as other stakeholders, and value trade-offs. Sub-study 2 is a discrete choice experiment that will test how value appraisals influence decision-making using data from Sub-study 1 as input. Sub-study 3 uses a mixed-method design, with findings from the two preceding sub-studies as input in focus group interviews to investigate how practitioners make sense of findings from optimal decision situations (experiment) and constrained, real-world decision situations.

Discussion

The project will offer unique insights into decision-making processes that influence how EBIs are used in practice. Such knowledge is needed for a more granular understanding of how practitioners manage the fidelity–adaptation dilemma and thus, ultimately, how the value of EBI implementation can be optimized. This study contributes to our knowledge of what happens once EBIs are adopted—that is, the gap between the way in which EBIs are intended to be used and the way in which they are used in practice.

International application of standards for health care quality, access and evaluation of services for early intervention in psychotic disorders

AIM

Standards for health care quality, access and evaluation of early intervention in psychosis services are required to assess implementation, provide accountability to service users and funders and support quality assurance. The aim of this article is to review the application of standards in Europe and North America.

METHODS

Descriptive methods will be used to illustrate the organizational context in which standards are being applied and used, specific measures being applied and results so far.

RESULTS

Both fidelity scales and quality indicators of health care are being used. Fidelity scales are being applied in Australia, Canada, Denmark, Italy and United States. In England, quality indicators derived from the National Institute for Health and Care Excellence guidance are being used.

CONCLUSION

In the last 4 years, significant progress has been made in the development and application of measures that assess quality and access to evidence-based practices for early intervention in psychosis services. This represents an important step towards providing accountability, improving outcomes and service user experience. The methods used allow for comparison between the services that are assessed with the same methods, but there is a need to compare the different methods. Further research is also required to explore links between quality of care and outcomes for community mental health services that deliver early intervention in psychotic disorders.

[Implementation of the principle of supported employment in Germany : Position paper of a task force of the DGPPN]

The effects of mental diseases on the employment and working situation can be substantial. They are one of the main reasons for inability to work and reduced earning capacity. Against this background the question arises about suitable occupational reintegration measures for people with severe mental illnesses. In recent years, the principle of supported employment has been internationally shown to be increasingly more successful. In this context mentally ill people are primarily placed at a position of the first employment market and supported on-site by a job coach. This concept is inclusive, individual and evidence based. Despite proven effectiveness, it has so far been insufficiently implemented in German-speaking regions. In the future it will be a matter of considering the individual needs for assistance of mentally ill people more intensively than previously and to respond with functional and in a best-case scenario, multiprofessional and flexible offers.

Disseminating TF-CBT: A Mixed Methods Investigation of Clinician Perspectives and the Impact of Training Format and Formalized Problem-Solving Approaches on Implementation Outcomes

BACKGROUND

Trauma Focused-Cognitive Behavioural Therapy (TF-CBT) has been established as an evidence-based treatment for youth with traumatic stress symptoms. The versatility of TF-CBT in conjunction with its established effectiveness has led to its widespread dissemination. However, dissemination efforts have not always translated into sustainability, which has prompted a more thorough investigation into those factors that impact implementation and encouraged the development of strategies that promote sustainability. Toward this end, the aims of this study were to: (1) determine which components of TF-CBT clinicians found the most difficult to implement; (2) explore clinicians' perceptions as to why these components were challenging; and (3) examine whether the use of formalized problem-solving approaches (FPSAs) or training format was associated with implementation outcomes.

METHODS

A mixed methods design was utilized to analyse survey data from mental health professionals (N = 85) who received TF-CBT training. Thematic analysis explored responses to questions concerning those TF-CBT components respondents found most difficult to implement, while bivariate analyses helped determine whether training format or the use of FPSAs was associated with training engagement, confidence in delivering TF-CBT, fidelity or sustainability.

RESULTS

Clinicians perceived the same three components of TF-CBT as most difficult regardless of the type of training they received, and provider, youth, caregiver, and organizational-related themes emerged from thematic analysis. Bivariate analyses indicated more extended training formats and the use of FPSAs were associated with greater implementation success.

CONCLUSION

Findings suggest that perceived difficulty of TF-CBT components did not vary by training format, but more extended formats and the use of FPSAs were associated with more favourable implementation outcomes. Implementers should consider ways to utilize FPSAs within training programs, as well as focus on content identified by clinicians as difficult, as this may assist clinicians in developing skills and managing implementation barriers.

A structured mixed method process evaluation of a randomized controlled trial of Individual Placement and Support (IPS)

Background

Individual Placement and Support (IPS) is an evidence-based work rehabilitation program helping people with moderate to severe mental illness to obtain ordinary employment. Although IPS has proven superior to other work rehabilitation programs, in many studies, the majority of the participants remain unemployed. Structured process evaluations of IPS that use mixed methods are scarce, although they could identify implementation aspects that may enhance its effect. The aim of the current study is to assess reach, fidelity, and identify barriers and facilitators to implement IPS.

Methods

The process evaluation was conducted alongside a randomized controlled trial including six IPS centers, comparing IPS with treatment as usual in a population of patients in treatment for moderate to severe mental illness. Mixed methods were used in the process evaluation, including focus group interviews with service providers, individual interviews and survey data from participants, and fidelity reviews using the validated IPS Fidelity Scale.

Results

The intervention reached the intended target group. All centers reached fair to good fidelity according to the IPS Fidelity Scale within the project period (range 97–109, SD 8.1) (see Table 5). Certain fidelity items indicated implementation issues related to employer contact, community-based services, and integration with health services. Survey data showed that less than half of the participants regarded their illness as a barrier for participating in IPS and that freedom of disclosure was important. Participant interviews gave further insight into the role of the IPS specialist, emphasizing their availability and consistent job focus.

Conclusions

Indications of implementation challenges across centers during the first year suggest special attention should be given to these aspects in an early phase to ensure higher fidelity from the start and thus enhance the effectiveness of IPS. The IPS specialist played an important role for participants and was described as positive, pushing in a positive way, and encouraging. More knowledge on the characteristics of successful IPS specialists could further enhance the effectiveness of the intervention.

Trial registration

The study was registered on clinicaltrials.gov prior to the inclusion period (reg.no: NCT01964092, registered 17/07/2013).

Supplementary information

Supplementary information accompanies this paper at 10.1186/s43058-020-00083-9.

Psychometric Properties of the General Organizational Index (GOI): A Measure of Individualization and Quality Improvement to Complement Program Fidelity

To assess the implementation of effective practices, mental health programs need standardized measures. The General Organizational Index (GOI), although widely used for this purpose, has received minimal psychometric research. For this study, we assessed psychometric properties of the GOI scale administered four times over 18 months during the implementation of a new program in 11 sites. The GOI scale demonstrated high levels of interrater reliability (.97), agreement between assessors on item ratings (86% overall), internal consistency (.77-.80 at three time points), sensitivity to change, and feasibility. We conclude that the GOI scale has acceptable psychometric properties, and its use may enhance implementation and research on evidence-based mental health practices.Trial registration: REK2015/2169. ClinicalTrials.gov Identifier: NCT03271242.

The Family Psychoeducation Fidelity Scale: Psychometric Properties

This study examined psychometric properties and feasibility of the Family Psychoeducation (FPE) Fidelity Scale. Fidelity assessors conducted reviews using the FPE fidelity scale four times over 18 months at five sites in Norway. After completing fidelity reviews, assessors rated feasibility of the fidelity review process. The FPE fidelity scale showed excellent interrater reliability (.99), interrater item agreement (88%), and internal consistency (mean = .84 across four time points). By the 18-month follow-up, all five sites increased fidelity and three reached adequate fidelity. Fidelity assessors rated feasibility as excellent. The FPE fidelity scale has good psychometric properties and is feasible for evaluating the implementation of FPE programs. Trial registration ClinicalTrials.gov Identifier: NCT03271242.

Expanding Individual Placement and Support to Populations With Conditions and Disorders Other Than Serious Mental Illness

OBJECTIVE

A systematic review of studies of individual placement and support (IPS) for populations other than those with serious mental illness was conducted.

METHODS

The authors searched three electronic databases (PubMed, Web of Science, and Scopus) for studies of IPS and modified IPS. Eligibility criteria for the systematic review included randomized controlled trials with prospective data collection on competitive employment rate and at least 10 study participants from a well-defined population other than people with serious mental illness. Results were compiled for competitive employment rates, IPS fidelity, and other outcomes.

RESULTS

Three clinical groups other than people with serious mental illness have been studied: people with psychiatric disorders other than serious mental illness, people with substance use disorders, and people with musculoskeletal or neurological disorders. Nine controlled trials with a total of 2,902 participants included six trials with people who had psychiatric disorders other than serious mental illness, two with people who had substance use disorders, and one with people who had spinal cord injuries. In eight studies, results for competitive employment rates significantly favored IPS. Meta-analysis yielded an overall weighted odds ratio of 2.23 (95% confidence interval=1.53-3.24, p<.001). Findings for other employment outcomes also favored IPS, but findings on symptom reduction and quality of life were inconsistent. The strongest (and only replicated) findings were for veterans with posttraumatic stress disorder (PTSD). Methodological limitations included small samples, major modifications to IPS fidelity, and short follow-up periods.

CONCLUSIONS

IPS, often with modifications, is a promising employment intervention for several populations in addition to people with serious mental illnesses. The strongest evidence pertains to veterans with PTSD. IPS should be offered to these veterans. Research on other populations, including people with anxiety, depression, substance use disorder, musculoskeletal or neurological conditions, or pain syndromes, needs development, amplification, and replication.

What is important for the sustained implementation of evidence-based brief psychotherapy interventions in psychiatric care? A quantitative evaluation of a real-world programme

PURPOSE

Behavioural activation and motivational interviewing, both evidence-based treatments (EBTs), were implemented in secondary psychiatric care. This longitudinal evaluation of a real-world programme focused on the penetration of EBT adoption and its associations with therapist-related and perceived intervention-related variables. The implementation plan was also compared to sub-processes of Normalization Process Theory.

MATERIAL AND METHODS

Six participating units employed 72 therapists regularly and they comprise the target group. Due to staff turnover, a total of 84 therapists were trained stepwise. Three survey points (q1, q2, q3) were set for a four-year cycle beginning a year after the initial training and completed 4-5 months after closing patient recruitment. The implementation plan included two workshop days, one for each EBT, and subsequent case consultation groups and other more general strategies.

RESULTS

Fifty-seven (68%) of programme-trained therapists responded to one or more of three questionnaires. The self-reported penetration covers about a third of the target group a few months after the completion of the programme. Therapists' favourable perceptions of the EBTs regarding relative advantage, compatibility and complexity were associated with their sustained adoption. Therapists' background factors (e.g. work experience) and positive adoption intention at q1 did not predict the actual adoption of the EBTs at q3. No specific sustainment strategies were included in the implementation plan.

CONCLUSION

Brief but multi-faceted training with subsequent case consultations promoted the adoption of EBTs in a real-world setting. Adding specific sustainment strategies to the implementation plan is proposed to ensure the long-term survival of the implementation outcomes.

Implementation and fidelity assessment of the NAVIGATE treatment program for first episode psychosis in a multi-site study

The NAVIGATE program was developed for the Recovery After Initial Schizophrenia Episode-Early Treatment Program (RAISE-ETP) study, which compared NAVIGATE to usual Community Care in a cluster randomized design involving 34 sites and 404 patients. This article describes the approach to training and implementing the NAVIGATE program at the 17 sites (including 134 practitioners) randomized to provide it, and to evaluating the fidelity of service delivery to the NAVIGATE model. Fidelity was evaluated to five different components of the program, all of which were standardized in manuals in advance of implementation. The components included four interventions (Individualized Resiliency Training, Family Education Program, Supported Employment and Education, Personalized Medication Management) and the overall organization (staffing and structure) of the NAVIGATE team. Most of the sites demonstrated acceptable or higher levels of fidelity in their implementation of the four interventions and the organization of the program, with all 17 sites demonstrating at least acceptable overall fidelity to the NAVIGATE program. The results indicate that the NAVIGATE program can be implemented with good fidelity to the treatment model in a diverse array of community mental health care settings serving persons with a first episode psychosis.

Training professionals in a recovery-oriented methodology: a mixed method evaluation

BACKGROUND

Several studies have reported difficulties concerning the implementation of recovery-oriented interventions. In this study, the effect of training in the Comprehensive Approach to Rehabilitation (CARe) on daily practice was evaluated. Additionally, we aimed to acquire insight into the experiences with the implementation process involving professionals, management and trainers.

METHODS

Fourteen teams for sheltered and supported housing in the Netherlands participated in this study. As part of a cluster-randomised controlled trial (RCT) design, eight teams received training in the CARe methodology. Model fidelity (using the CARe fidelity audit) and professionals' knowledge of recovery (using the Recovery Knowledge Inventory) were measured for all teams until 20 months after the start. Afterwards, an evaluation meeting with participating stakeholders was organised in which barriers and facilitators of the implementation of the CARe methodology were inventoried.

RESULTS

Ten months after the training, the intervention teams scored higher than the control teams on the fidelity subscales: 'recovery', 'strengths orientation' and 'amount of training and coaching'. Twenty months after the training, only the effect of 'amount of coaching and training' remained. Additionally, 'methodological working' clearly differed between the groups after 20 months in favour of the intervention teams. In all teams, model fidelity was moderate at both measurements. The knowledge of recovery of the trained teams was slightly and significantly higher at 10 and 20 months after training. Although professionals were positive about recovery and strength-oriented working, they experienced several organisational and societal barriers.

CONCLUSION

Training in the CARe methodology improved the fidelity and knowledge of recovery among professionals. However, the differences were small, and fidelity decreased over time. More in-depth knowledge is needed on which barriers professionals experience in practice so that tailored training and implementation strategies can be developed. Furthermore, more attention is needed for professional development and the translation of theory into practice.

Predictors of Acquisition of Competitive Employment for People Enrolled in Supported Employment Programs

This study aims at assessing the relative contribution of employment specialist competencies working in supported employment (SE) programs and client variables in determining the likelihood of obtaining competitive employment. A total of 489 persons with a severe mental illness and 97 employment specialists working in 24 SE programs across three Canadian provinces were included in the study. Overall, 43% of the sample obtained competitive work. Both client variables and employment specialist competencies, while controlling for the quality of SE programs implementation, predicted job acquisition. Multilevel analyses further indicated that younger client age, shorter duration of unemployment, and client use of job search strategies, as well as the working alliance perceived by the employment specialist, were the strongest predictors of competitive employment for people with severe mental illness, with 51% of variance explained. For people with severe mental illness seeking employment, active job search behaviors, relational abilities, and employment specialist competencies are central contributors to acquisition of competitive employment.

Development and Pilot Evaluation of a Tablet-Based Application to Improve Quality of Care in Child Mental Health Treatment

Background

Children need access to high quality mental health care. Effective treatments now exist for a wide range of mental health conditions. However, these interventions are delivered with variable effectiveness in traditional mental health service settings. Innovative solutions are needed to improve treatment delivery quality and effectiveness.

Objective

The aim of this study was to develop a scalable, sustainable technology-based approach to improve the quality of care in child mental health treatment.

Methods

A tablet-based resource was developed with input from mental health training experts, mental health providers, and patients. A series of qualitative data collection phases (ie, expert interviews, patient and provider focus groups, usability testing) guided the initial concept and design of the resource, and then its refinement. The result was an iPad-based “e-workbook” designed to improve child engagement and provider fidelity in implementation of a best-practice treatment. We are currently conducting a small scale randomized controlled trial to evaluate the feasibility of e-workbook facilitated child mental health treatment with 10 providers and 20 families recruited from 4 local community-based mental health clinics.

Results

Usability and focus group testing yielded a number of strong, favorable reactions from providers and families. Recommendations for refining the e-workbook also were provided, and these guided several improvements to the resource prior to initiating the feasibility trial, which is currently underway.

Conclusions

This study aimed to develop and preliminarily evaluate a tablet-based application to improve provider fidelity and child engagement in child mental health treatment. If successful, this approach may serve as a key step toward making best-practice treatment more accessible to children and families. As various technologies continue to increase in popularity worldwide and within the health care field more specifically, it is essential to rigorously test the usability, feasibility, acceptability, and effectiveness of novel health technology solutions. It is also essential to ensure that patients and providers drive decision making that supports the development of these resources to ensure that they can be seamlessly integrated into practice.

Trial Registration

Clinicaltrials.gov NCT01915160; https://clinicaltrials.gov/ct2/show/NCT01915160 (Archived by WebCite at http://www.webcitation.org/6cPIiQDpu)

Supported Employment for the Reintegration of Disability Pensioners with Mental Illnesses: A Randomized Controlled Trial

Work is beneficial for the recovery from mental illness. Although the approach of individual placement and support (IPS) has been shown to be effective in Europe, it has not yet been widely implemented in European health care systems. The aim of this randomized controlled trial was to assess the effectiveness of IPS for disability pensioners with mental illnesses new on disability benefits in Switzerland. In the study at hand, 250 participants were randomly assigned to either the control or the intervention group. The participants in the intervention group received job coaching according to IPS during 2 years. The control group received no structured support. Both groups were interviewed at baseline and followed up every 6 months (baseline, 6, 12, 16, 18, 24 months) for 2 years. Primary outcome was to obtain a job in the competitive employment. IPS was more effective for the reintegration into the competitive employment market for disability pensioners than the control condition. Thirty-two percent of the participants of the intervention group and 12% of the control group obtained new jobs in the competitive employment. IPS is also effective for the reintegration into competitive employment of people with mental illness receiving disability pensions.

Developing an instrument for assessing fidelity of motivational care planning: The Aboriginal and Islander Mental health initiative adherence scale

Purpose

The aim of this study was to design and trial an Adherence Scale to measure fidelity of Motivational Care Planning (MCP) within a clinical trial. This culturally adapted therapy MCP uses a client centered holistic approach that emphasises family and culture to motivate healthy life style changes.

Methods

The Motivational Care Planning-Adherence Scale (MCP-AS) was developed through consultation with Aboriginal and Islander Mental Health Initiative (AIMhi) Indigenous and non-Indigenous trainers, and review of MCP training resources. The resultant ten-item scale incorporates a 9-Point Likert Scale with a supporting protocol manual and uses objective, behaviourally anchored criteria for each scale point. A fidelity assessor piloted the tool through analysis of four audio-recordings of MCP (conducted by Indigenous researchers within a study in remote communities in Northern Australia). File audits of the remote therapy sessions were utilised as an additional source of information. A Gold Standard Motivational Care Planning training video was also assessed using the MCP-AS.

Results

The Motivational Care Planning-Adherence Scale contains items measuring both process and content of therapy sessions. This scale was used successfully to assess therapy through observation of audio or video-recorded sessions and review of clinical notes. Treatment fidelity measured by the MCP-AS within the pilot study indicated high fidelity ratings. Ratings were high across the three domains of rapport, motivation, and self-management with especially high ratings for positive feedback and engagement, review of stressors and goal setting.

Conclusions

The Motivational Care Planning-Adherence Scale has the potential to provide a measure of quality of delivery of Motivation Care Planning. The pilot findings suggest that despite challenges within the remote Indigenous community setting, Indigenous therapists delivered therapy that was of high fidelity. While developed as a research tool, the scale has the potential to support fidelity of delivery of Motivation Care Planning in clinical, supervision and training settings. Larger studies are needed to establish inter-rater reliability and internal and external validity.

Supported employment for persons with serious mental illness: current status and future directions

The individual placement and supported (IPS) model of supported employment is the most empirically validated model of vocational rehabilitation for persons with schizophrenia or another serious mental illness. Over 18 randomized controlled trials have been conducted throughout the world demonstrating the effectiveness of supported employment at improving competitive work compared to other vocational programs: IPS supported employment is defined by the following principles: 1) inclusion of all clients who want to work; 2) integration of vocational and clinical services; 3) focus on competitive employment; 4) rapid job search and no required prevocational skills training; 5) job development by the employment specialist; 6) attention to client preferences about desired work and disclosure of mental illness to prospective employers; 7) benefits counseling; and 8) follow-along supports after a job is obtained. Supported employment has been successfully implemented in a wide range of cultural and clinical populations, although challenges to implementation are also encountered. Common challenges are related to problems such as the failure to access technical assistance, system issues, negative beliefs and attitudes of providers, funding restrictions, and poor leadership. These challenges can be overcome by tapping expertise in IPS supported employment, including standardized and tested models of training and consultation. Efforts are underway to increase the efficiency of training methods for supported employment and the overall program, and to improve its effectiveness for those clients who do not benefit. Progress in IPS supported employment offers people with a serious mental illness realistic hope for achieving their work goals, and taking greater control over their lives.

Health and well-being of homeless veterans participating in transitional and supported employment: Six-month outcomes

Supported employment, specifically individual placement and support (IPS), improves competitive employment (CE) rates for individuals with serious mental illness, but has not shown greater improvement in non-vocational outcomes than other rehabilitation approaches. The Department of Veterans Affairs offers two types of vocational services, IPS and transitional work experience (TWE), but no study has compared the effectiveness of these approaches. This secondary analysis of data from a study of homeless veterans compared 6 mo improvement in diverse outcomes for five employment patterns: never worked, worked only in TWE, worked in TWE followed by CE, worked in CE without IPS, and worked in CE with IPS referral. Veterans referred to IPS were more likely to be competitively employed. Those who worked in CE (whether following TWE or with or without IPS referral) showed the greatest increase in days worked, employment income, and total income and the greatest decrease in public support income when compared with those who worked only in TWE or not at all. Veterans in TWE showed the greatest increase in residential treatment days, but there were no other differences in non-vocational outcomes between groups. There are multiple paths to CE, but few differences in non-vocational outcomes across employment experiences.

Defining and Evaluating Fidelity at the Program Level in Psychosocial Treatments

One challenge to research on the implementation of effective psychosocial treatments is how to define and measure fidelity at the program level. The purpose of this study was to evaluate an approach to defining, measuring, and observing over time fidelity at the program level for Multisystemic Therapy (MST). For this study, program fidelity was conceptualized as a program’s performance on several key areas identified in the literature as contributing to sustainability. A composite index, the MST Program Performance Index (PPI), was developed using data from the MST quality assurance system. The PPI included indicators of treatment adherence, treatment completion, program operations, program capacity, clinical supervisor leadership, and stakeholder relationships. A PPI score was calculated for 496 MST teams every 6 months for a two year period, during which time, the teams served over 25,000 young people. The predictive validity of the PPI score was supported by both client- and team-level outcomes. Specifically, youth treated by teams with lower PPI scores were more likely to be rearrested during their course of treatment. Likewise, teams with lower PPI scores were more likely to close during the time period covered by the study. Analysis of scores over the two year period showed substantial within-team variability in the PPI; however, the scores did not follow a linear pattern of change. In summary, preliminary evidence suggests the PPI may be a useful tool to index program-level fidelity for comparative purposes and as an additional tool for a decision support system.

Effectiveness of supported employment for veterans with spinal cord injuries: results from a randomized multisite study

OBJECTIVE

To examine whether supported employment (SE) is more effective than treatment as usual (TAU) in returning veterans to competitive employment after spinal cord injury (SCI).

DESIGN

Prospective, randomized, controlled, multisite trial of SE versus TAU for vocational issues with 12 months of follow-up data.

SETTING

SCI centers in the Veterans Health Administration.

PARTICIPANTS

Subjects (N=201) were enrolled and completed baseline interviews. In interventional sites, subjects were randomly assigned to the SE condition (n=81) or the TAU condition (treatment as usual-interventional site [TAU-IS], n=76). In observational sites where the SE program was not available, 44 subjects were enrolled in a nonrandomized TAU condition (treatment as usual-observational site [TAU-OS]).

INTERVENTIONS

The intervention consisted of an SE vocational rehabilitation program called the Spinal Cord Injury Vocational Integration Program, which adhered as closely as possible to principles of SE as developed and described in the individual placement and support model of SE for persons with mental illness.

MAIN OUTCOME MEASURES

The primary study outcome measurement was competitive employment in the community.

RESULTS

Subjects in the SE group were 2.5 times more likely than the TAU-IS group and 11.4 times more likely than the TAU-OS group to obtain competitive employment.

CONCLUSIONS

To the best of our knowledge, this is the first and only controlled study of a specific vocational rehabilitation program to report improved employment outcomes for persons with SCI. SE, a well-prescribed method of integrated vocational care, was superior to usual practices in improving employment outcomes for veterans with SCI.

Enhancing work-focused supports for people with severe mental illnesses in australia

Persons with severe mental illness (SMI) have reduced workforce participation, which leads to significant economic and social disadvantage. This theoretical review introduces the strategies that have been implemented to address this issue. These include Individual Placement and Support (IPS) services, the most widely researched form of supported employment, to which cognitive remediation has more recently been recognised in the USA, as an intervention to improve employment outcomes by addressing the cognitive impairments often experienced by people with SMI. The authors review the international literature and discuss specifically the Australian context. They suggest that Australia is in a prime position to engage clients in such a dual intervention, having had recent success with increasing access to supported employment programs and workforce reentry, through implementation of the Health Optimisation Program for Employment (HOPE). Such programs assist with gaining and maintaining employment. However, they do not address the cognitive issues that often prevent persons with SMI from effectively participating in work. Thus, optimising current interventions, with work-focused cognitive skills development is critical to enhancing employment rates that remain low for persons with SMI.

Generalizability of the Individual Placement and Support (IPS) model of supported employment outside the US

While reviews of controlled studies of the Individual Placement and Support (IPS) model of supported employment for clients with severe mental illness have documented its effectiveness in the US, its generalizability to other countries has not been systematically evaluated. This is the first review to compare US to non-US studies. We identified 15 randomized controlled trials of IPS programs, 9 in the US and 6 outside the US. We examined competitive employment outcomes, including employment rate, days to first job, weeks worked during follow-up, and hours worked. We also considered noncompetitive employment, program retention, and nonvocational outcomes. IPS programs had significantly better outcomes across a range of competitive employment indicators and higher retention in services than control groups. The overall competitive employment rate for IPS clients in US studies was significantly higher than in non-US studies (62% vs. 47%). The consistently positive competitive employment outcomes strongly favoring IPS over a range of comparison programs in a group of international studies suggest that IPS is an evidence-based practice that may transport well into new settings as long as programs achieve high fidelity to the IPS model, but further research is needed on international adaptations.

[New parameters in community treatment practices: the contribution of community and alternative mental health groups in Québec]

This article examines the origin and the context of the implementation project of community treatment according to the perspective of community and alternative mental health groups in Québec. The author presents results of a research on practices of these groups through a specific array that integrates the essential components of the community treatment model. The author examines six parameters characterizing devices, approaches and practices: 1) approach in supportive care and attention; 2) voluntary relation with the organization 3) the place of biomedical treatment; 4) flexibility in length and intensity of treatment; 5) the individual and collective dimension of supportive care; 6) and, participation as an active citizen.

It's a Bird, It's A Plane, It's … Fidelity Measurement In the Real World

In psychotherapy research, fidelity instruments were originally developed as manipulation checks in experimental tests of treatment efficacy. The purposes of fidelity measurement are expanding as consumers, administrators, and payers seek to determine the extent to which the interventions purchased are actually received. Emerging purposes for fidelity measurement are described, as are challenges to developing a single instrument that can adequately meet multiple purposes, and that is both effective (psychometrically sound) and efficient (feasibly used in routine care). Examples are provided of efforts to balance these attributes of fidelity measurement, to measure fidelity at multiple levels of the practice context, and to index and evaluate the effects of additional program parameters on client outcomes in routine care.

On the Topic of Treatment Integrity

Treatment integrity, also known as treatment fidelity, is integral for empirical testing of intervention efficacy as it allows for unambiguous interpretations of the obtained results. Assuring treatment integrity is also important for dissemination of evidence-based practices and quality improvement of services. However, in the examination of the relationship between treatment integrity and treatment outcome it is important to consider that treatment integrity may be a proxy variable for other variables impacting therapeutic change (e.g., characteristics of intervention, clients, setting, and therapist). Considerations on examining the association between integrity and outcome are discussed. Further, recommendations on the level to which treatment integrity needs to be addressed in psychotherapy research and clinical practice are provided.