The Family Psychoeducation Fidelity Scale: Psychometric Properties

Implementing Clinical Guidelines for the Treatment of Psychosis: The Frontline Leaders' Point of View. A Qualitative Study

Background

Despite the large amount of leadership and implementation theories and recommendations, healthcare services continue to struggle with efficiently incorporating new knowledge. The questioning of conventional leadership approaches in healthcare organizations prompted us to investigate how frontline leaders comprehend their own implementation intentions and actions, and how these intentions and actions may impact the implementation of clinical guidelines in mental healthcare in Norway.

Methods

Employing a theory-driven qualitative design, we conducted nine semi-structured interviews with frontline leaders who had recently led implementation of clinical guidelines for the treatment of psychosis in mental health. We employed Systematic Text Condensation, informed by Normalization Process Theory, to structure and analyze the data and used fidelity scales to measure the degree of implementation and distinguish between leaders' levels of success in implementation.

Results

Frontline leaders in units that achieved high success in implementation described their intentions and actions differently, from those with less success. The former group's actions aligned more closely with the constructs of the Normalization Process Theory compared to the latter group when describing their actions. Frontline leaders leading units with a high degree of implementation success describe relation-orientation, trust, and providing adaptive space for staff members to take initiative. In contrast, those leading units with less implementation success describe more control and guidance of co-operators and place more emphasize on information and knowledge.

Conclusion

Differences in how frontline leaders describe their actions and intentions to achieve clinical guideline implementation suggest that the leadership approach of these frontline leaders is an important factor to consider when planning and conducting implementation. To better understand the implementation process, it is important to pay attention to how frontline leaders customize their leadership approaches to the dynamics of complex organizations, and how they interact with their team and superiors.

Implementation of Guidelines on Family Involvement for Persons with Psychotic Disorders (IFIP): A Cluster Randomised Controlled Trial

Family involvement is part of the evidence-based treatment for persons with psychotic disorders, yet is under-implemented despite guideline recommendations. This study assessed whether an implementation support programme increased the adherence to guidelines on family involvement, compared to guideline/manual only. In a cluster randomised design, community mental health centre units in South-East Norway went through stratified allocation to the experimental (n = 7) or control (n = 7) arm. Experimental clusters received an implementation support programme including clinical training and supervision, appointing a family coordinator and an implementation team, a toolkit, and fidelity measurements at baseline, 12, 18, and 24 months with on-site feedback and supervision. Control clusters received no such support and had fidelity measurements at baseline and 24 months without feedback. During fidelity measurements, adherence to the guidelines was measured with the basic family involvement and support scale, the general organizational index, and the family psychoeducation fidelity scale, the latter being the primary outcome. The scales consist of 12-14 items rated from 1 to 5. Data was analysed with an independent samples t-test, linear mixed models, and a tobit regression model. At 24 months, the mean scores were 4.00 or higher on all scales in the experimental arm, and the increase in adherence to the guidelines was significantly greater than in the control arm with p-values < 0.001. Large-scale implementation of guidelines on family involvement for persons with psychotic disorders in community mental health centres may be accomplished, with substantial implementation support.Trial Registration: ClinicalTrials.gov Identifier NCT03869177. Registered 11.03.19.

Development and validation of a fidelity instrument for Cognitive Adaptation Training

Purpose

Cognitive Adaptation Training (CAT) is a psychosocial intervention with demonstrated effectiveness. However, no validated fidelity instrument is available. In this study, a CAT Fidelity Scale was developed and its psychometric properties, including interrater reliability and internal consistency, were evaluated.

Methods

The fidelity scale was developed in a multidisciplinary collaboration between international research groups using the Delphi method. Four Delphi rounds were organized to reach consensus for the items included in the scale. To examine the psychometric properties of the scale, data from a large cluster randomized controlled trial evaluating the implementation of CAT in clinical practice was used. Fidelity assessors conducted 73 fidelity reviews at four mental health institutions in the Netherlands.

Results

After three Delphi rounds, consensus was reached on a 44-item CAT Fidelity Scale. After administration of the scale, 24 items were removed in round four resulting in a 20-item fidelity scale. Psychometric properties of the 20-item CAT Fidelity Scale shows a fair interrater reliability and an excellent internal consistency.

Conclusions

The CAT fidelity scale in its current form is useful for both research purposes as well as for individual health professionals to monitor their own adherence to the protocol. Future research needs to focus on improvement of items and formulating qualitative anchor point to the items to increase generalizability and psychometric properties of the scale. The described suggestions for improvement provide a good starting point for further development.

Can systematic implementation support improve programme fidelity by improving care providers' perceptions of implementation factors? A cluster randomized trial

BACKGROUND

Investigations of implementation factors (e.g., collegial support and sense of coherence) are recommended to better understand and address inadequate implementation outcomes. Little is known about the relationship between implementation factors and outcomes, especially in later phases of an implementation effort. The aims of this study were to assess the association between implementation success (measured by programme fidelity) and care providers' perceptions of implementation factors during an implementation process and to investigate whether these perceptions are affected by systematic implementation support.

METHODS

Using a cluster-randomized design, mental health clinics were drawn to receive implementation support for one (intervention) and not for another (control) of four evidence-based practices. Programme fidelity and care providers' perceptions (Implementation Process Assessment Tool questionnaire) were scored for both intervention and control groups at baseline, 6-, 12- and 18-months. Associations and group differences were tested by means of descriptive statistics (mean, standard deviation and confidence interval) and linear mixed effect analysis.

RESULTS

Including 33 mental health centres or wards, we found care providers' perceptions of a set of implementation factors to be associated with fidelity but not at baseline. After 18 months of implementation effort, fidelity and care providers' perceptions were strongly correlated (B (95% CI) = .7 (.2, 1.1), p = .004). Care providers perceived implementation factors more positively when implementation support was provided than when it was not (t (140) = 2.22, p = .028).

CONCLUSIONS

Implementation support can facilitate positive perceptions among care providers, which is associated with higher programme fidelity. To improve implementation success, we should pay more attention to how care providers constantly perceive implementation factors during all phases of the implementation effort. Further research is needed to investigate the validity of our findings in other settings and to improve our understanding of ongoing decision-making among care providers, i.e., the mechanisms of sustaining the high fidelity of recommended practices.

TRIAL REGISTRATION

ClinicalTrials.gov Identifier: NCT03271242 (registration date: 05.09.2017).

Family involvement practices for persons with psychotic disorders in community mental health centres - a cross-sectional fidelity-based study

BACKGROUND

Family involvement for persons with psychotic disorders is supported by scientific evidence, as well as legal and ethical considerations, and recommended in clinical practice guidelines. This article reports a cross-sectional measurement of the level of implementation of such guidelines in fifteen community mental health centre units in Norway, and presents a novel fidelity scale to measure basic family involvement and support. The aim was to investigate current family involvement practices comprehensively, as a basis for targeted quality improvement.

METHODS

We employed three fidelity scales, with 12-14 items, to measure family involvement practices. Items were scored from 1 to 5, where 1 equals no implementation and 5 equals full implementation. Data was analysed using descriptive statistics, a non-parametric test, and calculation of interrater reliability for the scales.

RESULTS

The mean score was 2.33 on the fidelity scale measuring basic family involvement and support. Among patients with psychotic disorders, only 4% had received family psychoeducation. On the family psychoeducation fidelity assessment scale, measuring practice and content, the mean score was 2.78. Among the eight units who offered family psychoeducation, it was 4.34. On the general organizational index scale, measuring the organisation and implementation of family psychoeducation, the mean score was 1.78. Among the units who offered family psychoeducation, it was 2.46. As a measure of interrater reliability, the intra-class correlation coefficient was 0.99 for the basic family involvement and support scale, 0.93 for the family psychoeducation fidelity assessment scale and 0.96 for the general organizational index scale.

CONCLUSIONS

The implementation level of the national guidelines on family involvement for persons with psychotic disorders was generally poor. The quality of family psychoeducation was high, but few patients had received this evidence-based treatment. Our novel fidelity scale shows promising psychometric properties and may prove a useful tool to improve the quality of health services. There is a need to increase the implementation of family involvement practices in Norway, to reach a larger percentage of patients and relatives.

TRIAL REGISTRATION

ClinicalTrials.gov Identifier NCT03869177 . Registered 11.03.19.

The Effect of Intensive Implementation Support on Fidelity for Four Evidence-Based Psychosis Treatments: A Cluster Randomized Trial

PURPOSE

Service providers need effective strategies to implement evidence-based practices (EBPs) with high fidelity. This study aimed to evaluate an intensive implementation support strategy to increase fidelity to EBP standards in treatment of patients with psychosis.

METHODS

The study used a cluster randomized design with pairwise assignment of practices within each of 39 Norwegian mental health clinics. Each site chose two of four practices for implementation: physical health care, antipsychotic medication management, family psychoeducation, illness management and recovery. One practice was assigned to the experimental condition (toolkits, clinical training, implementation facilitation, data-based feedback) and the other to the control condition (manual only). The outcome measure was fidelity to the EBP, measured at baseline and after 6, 12, and 18 months, analyzed using linear mixed models and effect sizes.

RESULTS

The increase in fidelity scores (within a range 1-5) from baseline to 18 months was significantly greater for experimental sites than for control sites for the combined four practices, with mean difference in change of 0.86 with 95% CI (0.21; 1.50), p = 0.009). Effect sizes for increase in group difference of mean fidelity scores were 2.24 for illness management and recovery, 0.68 for physical health care, 0.71 for antipsychotic medication management, and 0.27 for family psychoeducation. Most improvements occurred during the first 12 months.

CONCLUSIONS

Intensive implementation strategies (toolkits, clinical training, implementation facilitation, data-based feedback) over 12 months can facilitate the implementation of EBPs for psychosis treatment. The approach may be more effective for some practices than for others.

Implementation of guidelines on family involvement for persons with psychotic disorders in community mental health centres (IFIP): protocol for a cluster randomised controlled trial

BACKGROUND

Family involvement for persons with psychotic disorders is under-implemented in mental health care, despite its firm scientific, economic, legal and moral basis. This appears to be the case in Norway, despite the presence of national guidelines providing both general recommendations on family involvement and support in the health- and care services, and specific guidance on family interventions for patients with psychotic disorders. The aim of this project is to improve mental health services and the psychosocial health of persons with psychotic disorders and their relatives, by implementing selected recommendations from the national guidelines in community mental health centres, and to evaluate this process.

METHODS

The trial is cluster randomised, where 14 outpatient clusters from community mental health centres undergo stratified randomisation with an allocation ratio of 1:1. The seven intervention clusters will receive implementation support for 18 months, whereas the control clusters will receive the same support after this implementation period. The intervention consists of: 1. A basic level of family involvement and support. 2. Family psychoeducation in single-family groups. 3. Training and guidance of health care personnel. 4. A family coordinator and 5. Other implementation measures. Fidelity to the intervention will be measured four times in the intervention arm and two times in the control arm, and the differences in fidelity changes between the arms constitute the primary outcomes. In each arm, we aim to include 161 patients with psychotic disorders and their closest relative to fill in questionnaires at inclusion, 6 months and 12 months, measuring psychosocial health and satisfaction with services. Clinicians will contribute clinical data about patients at inclusion and 12 months. Use of health and welfare services and work participation, for both patients and relatives, will be retrieved from national registries. We will also perform qualitative interviews with patients, relatives, health care personnel and leaders. Finally, we will conduct a cost-effectiveness analysis and a political economy analysis.

DISCUSSION

This project, with its multilevel and mixed methods approach, may contribute valuable knowledge to the fields of family involvement, mental health service research and implementation science.

TRIAL REGISTRATION

ClinicalTrials.gov Identifier NCT03869177 . Registered 11.03.19.

Measuring Fidelity to Evidence-Based Practices: Psychometrics

This special section presents the psychometric properties of fidelity scales used in a national mental health services project in Norway to improve the quality of care of people with psychoses. Across Norway, 39 clinical units in six health trusts participated. The project provided education, implementation support and fidelity assessments. The papers in the section address the psychometrics of fidelity measurement for the specific evidence-based practices: illness management and recovery, family psychoeducation, physical healthcare and antipsychotic medication management. Another paper analyzes the psychometrics of a scale measuring individualization and quality improvement that may be used in conjunction with fidelity scales for specific evidence-based practices. The first paper in the section presents the development and field of fidelity scales, and the two final papers with comments add some additional perspectives and discuss fidelity scales in a wider context. The psychometrics of the five scales were good to excellent. Fidelity assessment is a necessary and effective strategy for quality improvement.

Fidelity Scale: From Black Box to Holy Grail

Fidelity scales are indispensable in the pursuit of evidence-based mental healthcare. Without fidelity checks, treatment remains a mysterious black box. The aim of this article is to comment on the studies in this special section, and to discuss some general issues with regard to fidelity assessment. Despite all of their supposed benefits, resistance to fidelity scales persists among mental health practitioners. One way to overcome this resistance is to conduct fidelity assessments in the context of a well-guided learning community. The predictive validity of fidelity scales is considered the single most valuable attribute of these instruments. Research on predictive validity requires large sample sizes, which is difficult to achieve. It should nevertheless not prevent us from rigorously searching for this Holy Grail of fidelity assessment. In addition, fidelity assessment should be placed in a broader perspective. The quality of care for people with severe mental illness cannot be assessed conclusively according to the extent to which separate interventions have been applied with good fidelity. These individuals need access to high-quality treatment and support systems within the community, which can enable them to live their lives as valued citizens. In conclusion, fidelity assessment, both at the level of interventions and systems, contributes to a highly desirable transparency in practice variations within the field of mental healthcare.