Development of a fidelity scale for Danish specialized early interventions service

Development of an early intervention in psychosis services fidelity questionnaire

PURPOSE

We describe the development of an online fidelity questionnaire for early intervention in psychosis (EIP) services, to be used in population-level research, and that can be completed using self-reports from EIP staff.

METHODS

A review of key literature sources on the components of EIP services was used to identify those components eligible for inclusion in the questionnaire. A modified Delphi approach, using experts in EIP services, was used to select the most important components to include in the questionnaire. To pilot test the questionnaire, two EIP staff members completed one fidelity questionnaire each, and a third questionnaire was completed by an external rater. Responses from the three sources were compared and used to revise the fidelity questionnaire.

RESULTS

Twenty-two experts from England and Canada responded to two Delphi rounds, identifying the top 25 most important EIP service components. Some evidence-based components were not rated as highly as some non-evidence-based components. Pilot testing showed that the EIP staff rated fidelity higher than the external rater. Several questions were removed and/or revised based on the pilot study findings.

CONCLUSIONS

Fidelity instruments are limited by the available evidence and the personal experiences of experts used to develop them. As such, fidelity instruments and EIP services should continually be updated to reflect new knowledge. The online fidelity questionnaire was a simple and efficient way to collect data. Future evaluations of the fidelity questionnaire need to ensure that externally collected fidelity data are comprehensive and accurate.

Preliminary evaluation of a questionnaire for assessing fidelity of early intervention for psychosis services

PURPOSE

Fast, easy, and cost-effective methods are needed for fidelity assessment, quality improvement initiatives, and population-based studies in Early Intervention for Psychosis (EIP) services. Having an online questionnaire assessing the fidelity of EIP services, completed by staff self-reports, and having evidence of reliability and validity, could fill that gap. We assess the reliability and validity of the Early Intervention for Psychosis Services Fidelity Questionnaire (EIPS-FQ), developed in Part I of this set of papers.

METHODS

A convenience sample of 10 EIP teams in England was used. Two staff members completed online questionnaires assessing recent and past fidelity. An external rater completed the same questionnaire for the two time periods, using a random sample of patient medical records, program documentation, and interviews with staff. The intra-class correlation coefficient (ICC) was calculated to assess inter-rater reliability. Validity was assessed using Bland-Altman plots, absolute mean differences, and the ICC.

RESULTS

The fidelity score measuring recent fidelity ranged from 54.2 to 82.7, out of a possible 100. The ICC assessing reliability of the fidelity score was 0.40 (95% CI: 0.0-0.81). The ICCs for the fidelity sub-category scores ranged from 0 to 0.76. Two sub-categories, comprehensive assessments and family involvement and intervention, had low ICCs, regardless of period examined.

CONCLUSIONS

This first attempt at validating the EIPS-FQ has demonstrated that the reliability of the EIPS-FQ is moderate/low, and therefore requires modification prior to use. The next iteration of the fidelity questionnaire will clarify or remove items which had very low reliability and add evidence-based components not identified in the Delphi exercise.

The utility of a fidelity measure to monitor implementation of new early psychosis services across Australia

AIM

Early psychosis delivery models have proliferated worldwide, but there is limited research into establishing model fidelity. In this context, this article aims to describe the development and implementation of a fidelity tool in a national network of early psychosis services across Australia-the headspace Early Psychosis program.

METHODS

Following a detailed consultation process, and based on the Australian Early Psychosis model, an 80-item Early Psychosis Prevention and Intervention Centre Model Integrity Tool (EMIT) was developed along with predefined thresholds for fidelity. The tool was used to assess adherence to the model in six clusters of service sites across Australia. Ratings on the EMIT were informed by interviews with site staff and young people receiving the service, routinely collected data and site policies and procedures.

RESULTS

All six clusters of headspace Early Psychosis programs participated in five fidelity assessments across a period of two and a half years. In the initial two visits, the average fidelity score was in the 'low' fidelity range (i.e., <75%). By the fifth fidelity visit, the network average improved to 92.35%, reflecting 'superior' fidelity.

CONCLUSIONS

Results of the longitudinal fidelity assessments indicate the successful implementation of the Australian Early Psychosis model across the headspace Early Psychosis program. Utilisation of ongoing fidelity assessments has proved an effective method to improve and maintain adherence to the model.

International application of standards for health care quality, access and evaluation of services for early intervention in psychotic disorders

AIM

Standards for health care quality, access and evaluation of early intervention in psychosis services are required to assess implementation, provide accountability to service users and funders and support quality assurance. The aim of this article is to review the application of standards in Europe and North America.

METHODS

Descriptive methods will be used to illustrate the organizational context in which standards are being applied and used, specific measures being applied and results so far.

RESULTS

Both fidelity scales and quality indicators of health care are being used. Fidelity scales are being applied in Australia, Canada, Denmark, Italy and United States. In England, quality indicators derived from the National Institute for Health and Care Excellence guidance are being used.

CONCLUSION

In the last 4 years, significant progress has been made in the development and application of measures that assess quality and access to evidence-based practices for early intervention in psychosis services. This represents an important step towards providing accountability, improving outcomes and service user experience. The methods used allow for comparison between the services that are assessed with the same methods, but there is a need to compare the different methods. Further research is also required to explore links between quality of care and outcomes for community mental health services that deliver early intervention in psychotic disorders.

Reliability and Feasibility of the First-Episode Psychosis Services Fidelity Scale-Revised for Remote Assessment

OBJECTIVE

The authors sought to evaluate the interrater reliability and feasibility of the First-Episode Psychosis Services Fidelity Scale-Revised (FEPS-FS-R) for remote assessment of first-episode psychosis programs according to the coordinated specialty care model.

METHODS

The authors used the FEPS-FS-R to assess the fidelity of 36 first-episode psychosis program sites in the United States with information from three sources: administrative data, health record review, and phone interviews with staff. Four raters independently conducted fidelity assessments of five program sites by listening to each of the staff interviews and independently rating the two other data sources from each site. To calculate interrater reliability, the authors used intraclass correlation coefficients (ICCs) for each of the five sites and across the total scores for each site.

RESULTS

Total interrater reliability was in the good to excellent range, with a mean ICC of 0.91 (95% confidence interval = 0.72-0.99, p<0.001). Two first-episode psychosis program sites (6%) achieved excellent fidelity, 25 (69%) good fidelity, and nine (25%) fair fidelity. Of the 32 distinct items on the FEPS-FS-R, 23 (72%) were used with good or excellent fidelity. Most sites achieved high fidelity on most items, but five items received ratings indicating low-fidelity use at most sites. The fidelity assessment proved feasible, and sites required on average 10.5 hours for preparing and conducting the fidelity review.

CONCLUSIONS

The FEPS-FS-R has high interrater reliability and can differentiate high-, moderate-, and low-fidelity sites. Most sites had good overall fidelity, but the FEPS-FS-R identified some services that were challenging to implement at many sites.

Où en sommes-nous? An Overview of Successes and Challenges after 30 Years of Early Intervention Services for Psychosis in Quebec: Où en sommes-nous? Un aperçu des réussites et des problèmes après 30 ans de services d'intervention précoce pour la psychose au Québec

INTRODUCTION

Over the last 30 years, early intervention services (EIS) for first-episode psychosis (FEP) were gradually implemented in the province of Quebec. Such implementation occurred without provincial standards/guidelines and policy commitment to EIS until 2017. Although the literature highlights essential elements for EIS, studies conducted elsewhere reveal that important EIS components are often missing. No thorough review of Quebec EIS practices has ever been conducted, a gap we sought to address.

METHODS

Adopting a cross-sectional descriptive study design, an online survey was distributed to 18 EIS that existed in Quebec in 2016 to collect data on clinical, administrative, training, and research variables. Survey responses were compared with existing EIS service delivery recommendations.

RESULTS

Half of Quebec's population had access to EIS, with some regions having no programs. Most programs adhered to essential components of EIS. However, divergence from expert recommendations occurred with respect to variables such as open referral processes and patient-clinician ratio. Nonurban EIS encountered additional challenges related to their geography and lower population densities, which impacted their team size/composition and intensity of follow-up.

CONCLUSIONS

Most Quebec EIS offer adequate services but lack resources and organizational support to adhere to some core components. Recently, the provincial government has created EIS guidelines, invested in the development of new programs and offered implementation support from the National Centre of Excellence in Mental Health. These changes, along with continued mentoring and networking of clinicians and researchers, can help all Quebec EIS to attain and maintain recommended quality standards.

Which components of specialized early intervention for psychosis do senior providers see as most important?

AIM

Specialized early interventions improve outcomes in early psychosis (EP). Experts have proposed a number of essential treatment components. However, it is unclear whether these reflect the views of senior clinic staff charged with implementing this model in practice.

METHOD

Twenty-Five senior EP clinic staff across California completed a survey indicating which features of EP treatment they considered most important.

RESULTS

Components related to the service structure and the need for a prompt, comprehensive assessment and care planning were considered most important, despite the limited evidence base evaluating these aspects of care. Administration of clozapine to treatment-refractory patients and weight gain interventions were considered the least important, despite the relatively strong evidence base supporting these treatment components.

CONCLUSION

The findings suggest a bi-directional dissemination gap, where components considered most important by senior providers receive limited research attention, while some areas with supporting evidence may be underappreciated in clinical practice.