Development of a measure to assess the implementation of children's systems of care: the Systems of Care Implementation Survey (SOCIS)

State-Community System of Care Development: an Exploratory Longitudinal Review

The system of care (SOC) philosophy evolved into a framework to support access to effective behavioral health services for children. This study explored the use of the System of Care Implementation Survey (SOCIS) to monitor SOC development during one Midwestern state's federal planning and expansion grants. Utilizing a translational framework, results showed that despite fluctuations in SOC factor implementation over time, state and local SOCs had mid-level development. Further, inferential analysis of select factors indicated that outreach and access to services and the skilled provider network were significantly worse over time. Significant improvement in the treatment quality was documented but not sustained. Changes for Family Choice and Voice and Collaboration were not statistically significant. Variability in survey participation limited the evaluation. Refinement of evaluation methods is needed to monitor progress and to manage SOC development.

Psychometric properties of implementation measures for public health and community settings and mapping of constructs against the Consolidated Framework for Implementation Research: a systematic review

BACKGROUND

Recent reviews have synthesised the psychometric properties of measures developed to examine implementation science constructs in healthcare and mental health settings. However, no reviews have focussed primarily on the properties of measures developed to assess innovations in public health and community settings. This review identified quantitative measures developed in public health and community settings, examined their psychometric properties, and described how the domains of each measure align with the five domains and 37 constructs of the Consolidated Framework for Implementation Research (CFIR).

METHODS

MEDLINE, PsycINFO, EMBASE, and CINAHL were searched to identify publications describing the development of measures to assess implementation science constructs in public health and community settings. The psychometric properties of each measure were assessed against recommended criteria for validity (face/content, construct, criterion), reliability (internal consistency, test-retest), responsiveness, acceptability, feasibility, and revalidation and cross-cultural adaptation. Relevant domains were mapped against implementation constructs defined by the CFIR.

RESULTS

Fifty-one measures met the inclusion criteria. The majority of these were developed in schools, universities, or colleges and other workplaces or organisations. Overall, most measures did not adequately assess or report psychometric properties. Forty-six percent of measures using exploratory factor analysis reported >50 % of variance was explained by the final model; none of the measures assessed using confirmatory factor analysis reported root mean square error of approximation (<0.06) or comparative fit index (>0.95). Fifty percent of measures reported Cronbach's alpha of <0.70 for at least one domain; 6 % adequately assessed test-retest reliability; 16 % of measures adequately assessed criterion validity (i.e. known-groups); 2 % adequately assessed convergent validity (r > 0.40). Twenty-five percent of measures reported revalidation or cross-cultural validation. The CFIR constructs most frequently assessed by the included measures were relative advantage, available resources, knowledge and beliefs, complexity, implementation climate, and other personal resources (assessed by more than ten measures). Five CFIR constructs were not addressed by any measure.

CONCLUSIONS

This review highlights gaps in the range of implementation constructs that are assessed by existing measures developed for use in public health and community settings. Moreover, measures with robust psychometric properties are lacking. Without rigorous tools, the factors associated with the successful implementation of innovations in these settings will remain unknown.

Community Readiness Within Systems of Care: The Validity and Reliability of the System of Care Readiness and Implementation Measurement Scale (SOC-RIMS)

Establishing a system of care requires communities to identify ways to successfully implement strategies and support positive outcomes for children and their families. Such community transformation is complex and communities vary in terms of their readiness for implementing sustainable community interventions. Assessing community readiness and guiding implementation, specifically for the funded communities implementing a system of care, requires a well-designed tool with sound psychometric properties. This scale development study used the results of a previously published concept mapping study to create, administer, and assess the psychometric characteristics of the System of Care Readiness and Implementation Measurement Scale (SOC-RIMS). The results indicate the SOC-RIMS possesses excellent internal consistency characteristics, measures clearly discernible dimensions of community readiness, and demonstrates the target constructs exist within a broad network of content. The SOC-RIMS can be a useful part of a comprehensive assessment in communities where system of care practices, principles, and philosophies are implemented and evaluated.

Community characteristics and implementation factors associated with effective systems of care

How are characteristics of communities associated with the implementation of the principles of systems of care (SOC)? This study uses multilevel modeling with a stratified random sample (N = 225) of US counties to explore community-level predictors of the implementation factors of the System of Care Implementation Survey. A model composed of community-level social indicators fits well with 5 of 14 factors identified as relevant for effective SOCs. As hypothesized, community disadvantage was negatively and residential stability positively associated with the implementation of SOC principles. Designation as a mental health professional shortage area was positively related to some implementation scores, as was the percentage of minority residents, while rurality was not significantly associated with any of the factors. Given the limitations of the study, the results should be interpreted with caution, but suggest that further research is merited to clarify these relationships that could inform efforts directed at promoting SOCs.

Multilevel confirmatory factor analysis of the Systems of Care Implementation Survey (SOCIS)

A major impediment to obtaining national information on systems of care implementation has been the lack of a psychometrically sound large-scale survey instrument. The present study provided information on the factorial and concurrent validity of the Systems of Care Implementation Survey scales. Multilevel confirmatory factor analysis and multilevel regression analysis were used to test these indicators of internal and external validity. Two hundred twenty-five counties were randomly selected and stratified by population size and poverty level. Nine hundred ten informants responded to the survey questionnaire, M = 4.04 informants per county (SD = 3.17). Results indicated that all models had at least adequate fit to the data, with nine of the 14 factor models having excellent fit. Overall, 11 of the 14 factors had some indication that receiving federal funding to create systems of care was associated with higher scores on the factors. Implications for future research were discussed.