The State Health Authority Yardstick (SHAY)

State Mental Health Authority Level of Involvement in Coordinated Specialty Care Clinics and Client Outcomes

OBJECTIVE

State mental health authorities (SMHAs) in all U.S. states and territories administer the Mental Health Block Grant (MHBG) set-aside funding for first-episode psychosis. Funds support implementation of coordinated specialty care (CSC) programs. The authors investigated the relationship between the level of SMHA involvement with CSC programs and clinical outcomes of clients in these programs.

METHODS

As part of a mixed-methods study of 34 CSC programs, SMHAs from 21 states and one U.S. territory associated with the 34 CSC programs participated in a 1-hour interview (between November 2018 and May 2019) focused on SMHA involvement in administration of MHBG set-aside funds and the SMHA's ongoing relationship with funded CSC programs. SMHA involvement was rated on a scale of 1 to 5, with 5 indicating the highest involvement. Client outcome data were collected at the 34 study sites over an 18-month period. Multilevel random-effect modeling was used, controlling for response propensity (propensity score), client demographic variables, and program-level covariates (i.e., fidelity score, staff turnover rates, service area urbanicity, and number of clients enrolled).

RESULTS

Clients in CSC programs with SMHAs that were the most involved (level 5) had significantly improved symptoms, social functioning, and role functioning, compared with clients in programs with which SMHAs were least involved (level 1).

CONCLUSIONS

The findings suggest that increased SMHA involvement in CSC programs is relevant for positive client outcomes. Levels of first-episode psychosis funding doubled in 2021 and 2022, and it is important to identify how SMHAs affect the success of CSC programs and the individuals served.

Evaluation of measures of sustainability and sustainability determinants for use in community, public health, and clinical settings: a systematic review

BACKGROUND

Sustainability is concerned with the long-term delivery and subsequent benefits of evidence-based interventions. To further this field, we require a strong understanding and thus measurement of sustainability and what impacts sustainability (i.e., sustainability determinants). This systematic review aimed to evaluate the quality and empirical application of measures of sustainability and sustainability determinants for use in clinical, public health, and community settings.

METHODS

Seven electronic databases, reference lists of relevant reviews, online repositories of implementation measures, and the grey literature were searched. Publications were included if they reported on the development, psychometric evaluation, or empirical use of a multi-item, quantitative measure of sustainability, or sustainability determinants. Eligibility was not restricted by language or date. Eligibility screening and data extraction were conducted independently by two members of the research team. Content coverage of each measure was assessed by mapping measure items to relevant constructs of sustainability and sustainability determinants. The pragmatic and psychometric properties of included measures was assessed using the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). The empirical use of each measure was descriptively analyzed.

RESULTS

A total of 32,782 articles were screened from the database search, of which 37 were eligible. An additional 186 publications were identified from the grey literature search. The 223 included articles represented 28 individual measures, of which two assessed sustainability as an outcome, 25 covered sustainability determinants and one explicitly assessed both. The psychometric and pragmatic quality was variable, with PAPERS scores ranging from 14 to 35, out of a possible 56 points. The Provider Report of Sustainment Scale had the highest PAPERS score and measured sustainability as an outcome. The School-wide Universal Behaviour Sustainability Index-School Teams had the highest PAPERS score (score=29) of the measure of sustainability determinants.

CONCLUSIONS

This review can be used to guide selection of the most psychometrically robust, pragmatic, and relevant measure of sustainability and sustainability determinants. It also highlights that future research is needed to improve the psychometric and pragmatic quality of current measures in this field.

TRIAL REGISTRATION

This review was prospectively registered with Research Registry (reviewregistry1097), March 2021.

Understanding the implementation of evidence-informed policies and practices from a policy perspective: a critical interpretive synthesis

BACKGROUND

The fields of implementation science and knowledge translation have evolved somewhat independently from the field of policy implementation research, despite calls for better integration. As a result, implementation theory and empirical work do not often reflect the implementation experience from a policy lens nor benefit from the scholarship in all three fields. This means policymakers, researchers, and practitioners may find it challenging to draw from theory that adequately reflects their implementation efforts.

METHODS

We developed an integrated theoretical framework of the implementation process from a policy perspective by combining findings from these fields using the critical interpretive synthesis method. We began with the compass question: How is policy currently described in implementation theory and processes and what aspects of policy are important for implementation success? We then searched 12 databases as well as gray literature and supplemented these documents with other sources to fill conceptual gaps. Using a grounded and interpretive approach to analysis, we built the framework constructs, drawing largely from the theoretical literature and then tested and refined the framework using empirical literature.

RESULTS

A total of 11,434 documents were retrieved and assessed for eligibility and 35 additional documents were identified through other sources. Eighty-six unique documents were ultimately included in the analysis. Our findings indicate that policy is described as (1) the context, (2) a focusing lens, (3) the innovation itself, (4) a lever of influence, (5) an enabler/facilitator or barrier, or (6) an outcome. Policy actors were also identified as important participants or leaders of implementation. Our analysis led to the development of a two-part conceptual framework, including process and determinant components.

CONCLUSIONS

This framework begins to bridge the divide between disciplines and provides a new perspective about implementation processes at the systems level. It offers researchers, policymakers, and implementers a new way of thinking about implementation that better integrates policy considerations and can be used for planning or evaluating implementation efforts.

Understanding the implementation of evidence-informed policies and practices from a policy perspective: a critical interpretive synthesis

BACKGROUND

The fields of implementation science and knowledge translation have evolved somewhat independently from the field of policy implementation research, despite calls for better integration. As a result, implementation theory and empirical work do not often reflect the implementation experience from a policy lens nor benefit from the scholarship in all three fields. This means policymakers, researchers, and practitioners may find it challenging to draw from theory that adequately reflects their implementation efforts.

METHODS

We developed an integrated theoretical framework of the implementation process from a policy perspective by combining findings from these fields using the critical interpretive synthesis method. We began with the compass question: How is policy currently described in implementation theory and processes and what aspects of policy are important for implementation success? We then searched 12 databases as well as gray literature and supplemented these documents with other sources to fill conceptual gaps. Using a grounded and interpretive approach to analysis, we built the framework constructs, drawing largely from the theoretical literature and then tested and refined the framework using empirical literature.

RESULTS

A total of 11,434 documents were retrieved and assessed for eligibility and 35 additional documents were identified through other sources. Eighty-six unique documents were ultimately included in the analysis. Our findings indicate that policy is described as (1) the context, (2) a focusing lens, (3) the innovation itself, (4) a lever of influence, (5) an enabler/facilitator or barrier, or (6) an outcome. Policy actors were also identified as important participants or leaders of implementation. Our analysis led to the development of a two-part conceptual framework, including process and determinant components.

CONCLUSIONS

This framework begins to bridge the divide between disciplines and provides a new perspective about implementation processes at the systems level. It offers researchers, policymakers, and implementers a new way of thinking about implementation that better integrates policy considerations and can be used for planning or evaluating implementation efforts.

A Tale of Four States: Factors Influencing the Statewide Adoption of IPS

Evidence-based supported employment has become a core community mental health service in much of the U.S. Although a national learning community has facilitated progress in about half of the states, other states have tried to implement evidence-based supported employment on their own. Many studies have examined site-level factors influencing implementation of supported employment, but few have focused on the role of state agency policies and actions. This study examined four states that have not joined the learning community, comparing two that have implemented with success (adopting states) and two that have faced challenges (non-adopting states). This comparative case study approach compared barriers, facilitators, and strategies in two states adopting IPS to two states that did not. The authors examined quantitative data from public records and conducted content analysis of qualitative and quantitative data from key informant interviews. The two non-adopting states lacked model clarity, funding, focus on people with serious mental illness, and collaboration between state mental health and vocational rehabilitation agencies. The two successful states experienced similar barriers but overcame them following lawsuit settlements that required implementation of evidence-based supported employment. Key strategies for successful implementation were funding, fidelity monitoring, technical assistance, and collaboration between state mental health and vocational rehabilitation agencies. With legal settlements serving as the catalyst, states facing challenges to implementing evidence-based supported employment can achieve success using standard implementation strategies to fund and ensure the quality of services.

Patterns and predictors of compliance with utilization management guidelines supporting a state policy to improve the quality of youth mental health services

Despite a need to improve community mental health services for youths, little is known about compliance with state policies created to improve the quality of services in these settings. This study examined rates, patterns, and predictors of compliance with utilization management guidelines developed by the state of Texas to support a public health policy based on empirical evidence of effective mental health services (i.e., an evidence-based policy). Compliance was defined as authorizing policy-recommended service packages, whereas policy "overrides" occurred when recommended service packages were not authorized. The study sample consisted of 688 youths from ethnically and economically diverse backgrounds. Clinics reported that forty-six percent of youths were not authorized the policy-recommended service package. Overrides were primarily based on level of intensity. Most often, authorized services were less intensive than those recommended by the state guidelines. Higher severity at intake across multiple indicators was associated with authorizing less intensive services than what the policy guidelines recommended. Future studies evaluating system-level efforts such as state mental health policies should pay close attention to levels of service intensity, and their relation to the needs of youth in community settings.

A mixed-methods study of system-level sustainability of evidence-based practices in 12 large-scale implementation initiatives

Background

In recent decades, evidence-based practices (EBPs) have been broadly promoted in community behavioural health systems in the United States of America, yet reported EBP penetration rates remain low. Determining how to systematically sustain EBPs in complex, multi-level service systems has important implications for public health. This study examined factors impacting the sustainability of parent-child interaction therapy (PCIT) in large-scale initiatives in order to identify potential predictors of sustainment.

Methods

A mixed-methods approach to data collection was used. Qualitative interviews and quantitative surveys examining sustainability processes and outcomes were completed by participants from 12 large-scale initiatives.

Results

Sustainment strategies fell into nine categories, including infrastructure, training, marketing, integration and building partnerships. Strategies involving integration of PCIT into existing practices and quality monitoring predicted sustainment, while financing also emerged as a key factor.

Conclusions

The reported factors and strategies impacting sustainability varied across initiatives; however, integration into existing practices, monitoring quality and financing appear central to high levels of sustainability of PCIT in community-based systems. More detailed examination of the progression of specific activities related to these strategies may aide in identifying priorities to include in strategic planning of future large-scale initiatives.

Trial registration

ClinicalTrials.gov ID NCT02543359; Protocol number PRO12060529.

Electronic supplementary material

The online version of this article (doi:10.1186/s12961-017-0230-8) contains supplementary material, which is available to authorized users.

New directions for psychiatric rehabilitation in the USA

American researchers have led the world in developing, evaluating, and disseminating evidence-based psychiatric rehabilitation practices for people with serious mental illness. Paradoxically, however, the USA lags behind most industrialized nations in providing access to high-quality mental health and psychiatric services. This essay examines several evidence-based practices developed in the USA, the spread of these practices, the barriers to ensuring availability to people who could benefit from these services, and some promising directions for overcoming the barriers. Factors influencing the growth and sustainment of effective client-centred practices include the availability of adequate and stable funding, committed leadership, and the influence of vested interests. Two strategies for promoting the spread and sustainment of well-implemented evidence-based practices are the adoption of fidelity scales and learning communities.

Prevalence and Quality of Individual Placement and Support (IPS) Supported Employment in the United States

The individual placement and support (IPS) model of supported employment for people with serious mental illness is an evidence-based practice. Factors including a national learning community promoting IPS and enforcement of the Supreme Court's Olmstead decision have spurred the growth of IPS nationwide. In this study we first evaluated the national prevalence and quality of IPS programs. We then evaluated the impact of learning community membership and Olmstead settlements on IPS program penetration and quality across the United States. We interviewed representatives from 48 state behavioral health agencies and 51 state vocational rehabilitation agencies. Survey questions examined the number of IPS programs in each state, the presence of an Olmstead settlement mandating employment services for people with serious mental illness, and the presence of three indicators of quality in IPS programs: collaboration between state behavioral health and vocational rehabilitation agencies, regular, independent fidelity monitoring, and technical assistance and training for IPS programs. Respondents from 38 (75%) states, including 19 states in the IPS Learning Community and 19 outside the learning community, reported a total of 523 IPS programs nationwide (M = 14, SD = 16). The state IPS program penetration rate (number of IPS programs per 1,000,000 people) ranged from 0.05 to 16.62 (M = 3.61, SD = 3.62) among states with IPS. The penetration rate was similar for learning community and non-learning community states with IPS, but learning community states were much more likely than non-learning community states with IPS to report the presence of each of three quality indicators. Eleven states reported Olmstead or other settlements that positively impacted employment services for people with serious mental illness, but among the 38 states with IPS programs, Olmstead states did not differ from non-Olmstead states in IPS program penetration or on the quality indicators. Nationally, most states provide IPS programs, but the within-state penetration rate and quality of implementation vary widely. While learning community and non-learning community states with IPS do not differ in the prevalence of IPS programs, learning community states are much more likely to report key quality indicators, which may enhance these states' potential for sustaining and expanding IPS. Olmstead settlements have not yet shown a direct impact on the penetration and quality of IPS, but as the Department of Justice continues to enforce the Supreme Court's Olmstead decision, their significance may increase.

Measures for Predictors of Innovation Adoption

Building on a narrative synthesis of adoption theories by Wisdom et al. (2013), this review identifies 118 measures associated with the 27 adoption predictors in the synthesis. The distribution of measures is uneven across the predictors and predictors vary in modifiability. Multiple dimensions and definitions of predictors further complicate measurement efforts. For state policymakers and researchers, more effective and integrated measurement can advance the adoption of complex innovations such as evidence-based practices.

Sustaining and Expanding Evidence-Based Supported Employment: The Role of State Leaders Participating in a Learning Community

State leaders often promote implementation of evidence-based practices but have difficulty sustaining and expanding them over time. This paper examines the activities of leaders in 13 states that have successfully implemented, sustained, and expanded evidence-based supported employment, known as Individual Placement and Support (IPS), for 4 to 12 years. We interviewed state leaders from 13 states participating in a learning community regarding the composition of their leadership team, participation in the learning community, interagency collaboration, state policy alignment, financing, training, and monitoring of fidelity and outcome. To assess state-level performance in implementing, sustaining, and expanding IPS services, we obtained measures of sustainment, expansion, program fidelity, and employment in the subsequent year and compared them to a priori benchmarks. The majority of states (between 69 % and 77 %) met benchmarks for sustainment, expansion, fidelity, and employment. States varied widely in specific actions to advance IPS, but all had established leadership teams, participated in the national learning community, and built an infrastructure supporting IPS. Leaders in 13 states participating in a learning community have adopted and maintained multiple strategies to sustain and expand evidence-based supported employment at a high level of fidelity with good employment outcomes.

Research, Data, and Evidence-Based Treatment Use in State Behavioral Health Systems, 2001-2012

OBJECTIVE

Empirical study of public behavioral health systems' use of data and their investment in evidence-based treatments (EBTs) is limited. This study describes trends in state-level EBT investment and research supports from 2001 to 2012.

METHODS

Data were from National Association for State Mental Health Program Directors Research Institute (NRI) surveys, which were completed by representatives of state mental health authorities (SMHAs). Multilevel models examined change over time related to state adoption of EBTs, numbers of clients served, and penetration rates for six behavioral health EBTs for adults and children: supported housing, supported employment, assertive community treatment, therapeutic foster care, multisystemic therapy, and functional family therapy. State supports related to research, evaluation, and information management were also examined.

RESULTS

Increasing percentages of states reported funding an external research center, promoting the adoption of EBTs through provider contracts, and providing financial incentives for EBTs. Decreasing percentages of states reported promoting EBT adoption through stakeholder mobilization, monitoring fidelity, and specific budget requests. There was greater reported use of adult-focused EBTs (65%-80%) compared with youth-focused EBTs (25%-50%). Overall penetration rates of EBTs were low (1%-3%) and EBT adoption by states showed flat or declining trends. SMHAs' investment in data systems and use of research showed little change.

CONCLUSIONS

SMHA investment in EBTs, implementation infrastructure, and use of research has declined. More systematic measurement and examination of these metrics may provide a useful approach for setting priorities, evaluating success of health reform efforts, and making future investments.

Role of state mental health leaders in supporting evidence-based practices over time

State administrative supports of evidence-based practices (EBPs) may critically affect sustainability. As part of a larger follow-up to the multistate National Implementing Evidence-Based Practices Project, 13 leaders from eight States reported on recent statewide policies and resources that influenced the maintenance of 49 EBP programs 6 years after implementation. Nearly three-quarters (71 %) of the programs were currently operational and adhering to State fidelity standards. Most of the programs had level or increased funding over the last few years, and most received other State support and resources, such as training and technical assistance. Steady State funding as well as State-sponsored training and expertise appear critical to long-term EBP program maintenance.

A Longitudinal Study of State Strategies and Policies to Accelerate Evidence-Based Practices in the Context of Systems Transformation

OBJECTIVE

To profile state agency efforts to promote implementation of three evidence-based practices (EBPs): screening and brief intervention (SBIRT), psychosocial interventions, and medication-assisted treatment (MAT).

DATA SOURCES/STUDY SETTING

Primary data collected from representatives of 50 states and the District of Columbia's Single State Authorities from 2007 to 2009.

STUDY DESIGN/DATA COLLECTION

The study used mixed methods, in-depth, semistructured interviews and quantitative surveys. Interviews assessed state and provider strategies to accelerate implementation of EBPs.

PRINCIPAL FINDINGS

Statewide implementation of psychosocial interventions and MAT increased significantly over 3 years. In the first two assessments, states that contracted directly with providers were more likely to link use of EBPs to reimbursement, and states with indirect contract, through counties and other entities, increased recommendations, and some requirements for provision of specific EBPs. The number of states using legislation as a policy lever to promote EBPs was unchanged.

CONCLUSIONS

Health care reform and implementation of parity in coverage increases access to treatment for alcohol and drug use. Science-based substance abuse treatment will become even more crucial as payers seek consistent quality of care. This study provides baseline data on service delivery, contracting, and financing as state agencies and treatment providers prepare for implementation of the Affordable Care Act.

Organizational fidelity to a medication management evidence-based practice in the treatment of schizophrenia

Organizational support is essential for successful implementation of evidence-based practice (EBP) in clinical settings. This 3-year study used a mixed qualitative and quantitative design to implement a medication management EBP in the treatment of schizophrenia in six community mental health clinics in a south-central state of the United States. Findings from organizational fidelity assessments indicate that support for EBP implementation was moderate. Organizational support was highest for prescriber access to relevant patient information at each medication visit, scheduling flexibility for patients' urgent problems, and availability of medication guidelines. Organizational support was lowest for medication availability and identification of treatment refractory patients. Findings suggest that leadership is essential to support successful implementation. Nurse educators can incorporate implementation research and leadership training into graduate nursing programs to facilitate successful EBP implementation in practice settings.

Mental Health Disability

Mental health disability imposes an enormous cost to patients, families, and society. Across free market countries, the rate of mental health disability is growing and now constitutes about one third of all disability claimants. Living on disability benefits begets demoralization and passivity. This article reviews (a) the current status of mental health disability and mental health disability systems, (b) the encouraging findings related to research on evidence-based treatments and supported employment, and (c) policy changes that might ameliorate the problem of rising rates (and costs) of mental health disability. Mental health treatments, employment services, and disability policies powerfully shape the disability experience. Evidence-based mental health and employment services can ameliorate disability and could potentially prevent disability if applied as early interventions. Yet evidence-based mental health treatments and supported employment services are rarely available. Furthermore, current disability policies tend to trap people in a “disability benefits culture.” Policy changes should recognize that people with mental health disabilities usually want to work, that they are often able to work when provided with appropriate mental health and vocational services, and that employment is highly therapeutic. Disability policies need to strengthen work incentives for beneficiaries, providers, and employers.

Employing policy and purchasing levers to increase the use of evidence-based practices in community-based substance abuse treatment settings: reports from single state authorities

State public health authorities are critical to the successful implementation of science based addiction treatment practices by community-based providers. The literature to date, however, lacks examples of state level policy strategies that promote evidence-based practices (EBPs). This mixed-methods study documents changes in two critical state-to-provider strategies aimed at accelerating use of evidence-based practices: purchasing levers (financial incentives and mechanisms) and policy or regulatory levers. A sample of 51 state representatives was interviewed. Single State Authorities for substance abuse treatment (SSAs) that fund providers directly or through managed care were significantly more likely to have contracts that required or encouraged evidence-based interventions, as compared to SSAs that fund providers indirectly through sub-state entities. Policy levers included EBP-related legislation, language in rules and regulations, and evidence-based criteria in state plans and standards. These differences in state policy are likely to result in significant state level variations regarding both the extent to which EBPs are implemented by community-based treatment providers and the quality of implementation.

Evidence-based practice implementation in Kansas

Over the last 8 years, Kansas has been successful in the implementation of evidence-based practices. This paper describes the strategies used at multiple levels of the mental health system including: state policy, provider agency management, fidelity and outcome monitoring, supervisor training and support, and practitioner training. The challenges going forth are described.

The 2009 schizophrenia PORT psychosocial treatment recommendations and summary statements

The Schizophrenia Patient Outcomes Research Team (PORT) psychosocial treatment recommendations provide a comprehensive summary of current evidence-based psychosocial treatment interventions for persons with schizophrenia. There have been 2 previous sets of psychosocial treatment recommendations (Lehman AF, Steinwachs DM. Translating research into practice: the Schizophrenia Patient Outcomes Research Team (PORT) treatment recommendations. Schizophr Bull. 1998;24:1-10 and Lehman AF, Kreyenbuhl J, Buchanan RW, et al. The Schizophrenia Patient Outcomes Research Team (PORT): updated treatment recommendations 2003. Schizophr Bull. 2004;30:193-217). This article reports the third set of PORT recommendations that includes updated reviews in 7 areas as well as adding 5 new areas of review. Members of the psychosocial Evidence Review Group conducted reviews of the literature in each intervention area and drafted the recommendation or summary statement with supporting discussion. A Psychosocial Advisory Committee was consulted in all aspects of the review, and an expert panel commented on draft recommendations and summary statements. Our review process produced 8 treatment recommendations in the following areas: assertive community treatment, supported employment, cognitive behavioral therapy, family-based services, token economy, skills training, psychosocial interventions for alcohol and substance use disorders, and psychosocial interventions for weight management. Reviews of treatments focused on medication adherence, cognitive remediation, psychosocial treatments for recent onset schizophrenia, and peer support and peer-delivered services indicated that none of these treatment areas yet have enough evidence to merit a treatment recommendation, though each is an emerging area of interest. This update of PORT psychosocial treatment recommendations underscores both the expansion of knowledge regarding psychosocial treatments for persons with schizophrenia at the same time as the limitations in their implementation in clinical practice settings.

Implementing evidence-based practices for people with schizophrenia

Over the last decade, a consensus has emerged regarding a set of evidence-based practices for schizophrenia that address symptom management and psychosocial functioning. Yet, surveys suggest that the great majority of the population of individuals with schizophrenia do not receive evidence-based care. In this article, we review the empirical literature on implementation of evidence-based practices for schizophrenia patients. We first examine lessons learned from implementation studies in general medicine. We then summarize the implementation literature specific to schizophrenia, including medication practices, psychosocial interventions, information technology, and state- and federal-level interventions. We conclude with recommendations for future directions.

Individualization and quality improvement: two new scales to complement measurement of program fidelity

Fidelity scales have been widely used to assess program adherence to the principles of an evidence-based practice, but they do not measure important aspects of quality of care. Pragmatic scales measuring clinical quality of services are needed to complement fidelity scales measuring structural aspects of program implementation. As part of the instrumentation developed for the National Implementing Evidence-Based Practices Project, we piloted a new instrument with two 5-item quality scales, Individualization (a client-level quality scale) and Quality Improvement (an organizational-level quality scale). Pairs of independent fidelity assessors conducted fidelity reviews in 49 sites in 8 states at baseline and at four subsequent 6-month intervals over a 2-year follow-up period. The assessors followed a standardized protocol to administer these quality scales during daylong site visits; during these same visits they assessed programs on fidelity to the evidence-based practice that the site was seeking to implement. Assessors achieved acceptable interrater reliability for both Individualization and Quality Improvement. Principal components factor analysis confirmed the 2-scale structure. The two scales were modestly correlated with each other and with the evidence-based practice fidelity scales. Over the first year, Individualization and Quality Improvement improved, but showed little or no improvement during the last year of follow-up. The two newly developed scales showed adequate psychometric properties in this preliminary study, but further research is needed to assess their validity and utility in routine clinical practice.